If the racists do the math, they also see that if you keep population constant, they will be a dwindling percentage of the population. They don't want to be treated the way they've treated other minorities. And, they don't want to be replaced, like the Picts and the neanderthals.