Leandro von Werra @lvwerra.bsky.social · 06/01/2025Distributed training is notoriously hard to learn - knowledge is scattered across papers and complex codebases. Enter picotron: implementing all 4D parallelism concepts in separate, readable files totaling just 1988 LoC! 240
Thomas Capelle @capetorch.bsky.social · 06/01/2025This could be a good test bed to debug wandb 4d logging. Cc @morgymcg.bsky.social 010
Leandro von Werra @lvwerra.bsky.social · 06/01/2025Checkout the code and super detailed video walkthrough! Code: github.com/huggingface/... Video: youtube.com/playlist?lis... Work lead by Haojun Zhao and Ferdinand Mom!github.comGitHub - huggingface/picotron: Minimalistic 4D-parallelism distributed training framework for education purposeMinimalistic 4D-parallelism distributed training framework for education purpose - huggingface/picotron 011
