Without getting Zinn about it, the Daniel Immerwahr book How to Hide an Empire was interesting on this, he came to a similar conclusion to you - that you couldn’t be a multiracial empire unless you had universal franchise, and if you have the universal franchise you probably aren’t an empire anymore