I think you're assigning way more power and malice to individual non voters. Non voters are not a monolith, they aren't in positions of power, they're disenfranchised and ignored, and most of them are the main target of trump's actions. Further, they're not "withholding votes"