Online data mixing reduces training costs for foundation models, but faces challenges:
⚠️ Human-defined domains miss semantic nuances
⚠️ Limited eval accessibility
⚠️ Poor scalability
Introducing 🎵R&B: first regroup data, then dynamically reweight domains during training!
