If someone tries to shame you for "blocking anyone who disagrees with you", it's 99.9% likely to be someone who deserves to be blocked and deplatformed.
Listen to your gut. Block fascists and bad faith actors into the void.
And if possible and required, report them and warn others to do so.