Unity's baked lighting isn't very friendly to toon shading, but I've managed to make it work smoothly no matter the lighting conditions, without sacrificing looks.
It's not the most obvious thing from the outside, but getting it working this well was the biggest struggle when making the shader.