We experiment on a smaller 130B token setting, where we show EMO subsets also match or outperform memory-matched models trained from scratch.
Instead of training many separate small models for fixed memory budgets, one EMO model can provide many domain-specific expert subsets.
