Arthur Douillard @douillard.bsky.social · 25/11/2024distributed learning for LLM? recently, @primeintellect.bsky.social have announced finishing their 10B distributed learning, trained across the world. what is it exactly? 🧵 1256
Arthur Douillard @douillard.bsky.social · 25/11/2024The bitter lesson is that putting more scale, w.r.t # params, flops, and data will make a better model. Usual distributed methods train across multiple GPUs, but require a tight synchronization! Where communication is done at every step! This is fine when bandwidth is high, but otherwise? 140
Arthur Douillard @douillard.bsky.social · 25/11/2024You don't want to wait for communication, GPUs must be 100% burning! Federated Learning (FL, arxiv.org/abs/1602.05629) aims to train a model across a fleet of phones. Each phone will perform multiple training steps (forward/backward & update) independently from each other. 110
