Btw, I noticed one peculiar issue. It seems the RSA is cached in RsaSecurityKey. If you re-create a new instance, it will fail during validation. Here is a simple example.
If I disable the CacheSignatureProviders, then works each time. Not sure why is cached, but finding the root cause was tedious.