Something was off in the reference path tracer I am implementing (heavily based on the fantastic one in Ray Tracing Gems II), and eventually figured out my view vector was negated, causing all kinds of funny stuff :P
Now to figure out why Resampled Importance Sampling causes spot lights to look off.