iiinova — where your digital self keeps evolving

Zach Mueller @muellerzr.bsky.social · 05/10/2025

On November 1st we're running it again! Come join over 300 students in learning about what makes distributed training tick, common bottlenecks, and more. maven.com/walk-with-c...

maven.com

Scratch to Scale: Large-Scale Training in the Modern World by Zachary Mueller on Maven

Learn the techniques used today to take your model training from Colab to Clusters

020

Zach Mueller @muellerzr.bsky.social · 05/10/2025

Dispatching works by keeping the dataset on one process and then sending the batches to the other workers throughout training. This incurs a memory cost since this is a GPU -> GPU transfer, however many find this to be more appealing than other alternatives.

120

Zach Mueller @muellerzr.bsky.social · 05/10/2025

DataLoader Dispatching When constrained by a variety of reasons to where you can't include multiple copies (or mmaps) of datasets in memory, be it too many concurrent streams, low resource availability, or a slow CPU, dispatching is here to help.

150

Zach Mueller @muellerzr.bsky.social · 03/10/2025

On November 1st starts the second cohort of Scratch to Scale! Come learn the major tricks and algorithms used when single-GPU training hits failure points.

100

Zach Mueller @muellerzr.bsky.social · 03/10/2025

This ensures consistent randomness and avoids redundant computation, making it well-suited for more complex sampling strategies.

100

Zach Mueller @muellerzr.bsky.social · 03/10/2025

Batch sampler sharding is especially efficient when using non-trivial sampling methods (weighted, balanced, temperature-based, etc.), since the sampling logic runs once globally rather than being duplicated per worker.

100

Zach Mueller @muellerzr.bsky.social · 03/10/2025

Unlike iterable dataset sharding, where each worker directly pulls raw data in its __iter__, batch sampler sharding first generates indices that specify which items to fetch. These indices are grouped into batches, which are then collated into tensors and moved to CUDA for training.

100

Zach Mueller @muellerzr.bsky.social · 02/10/2025

On November 1st starts the second cohort of Scratch to Scale! Come learn the major tricks and algorithms used when single-GPU training hits failure points. Also get access to the prior cohort's guest speakers too! (And get in for 35% off) maven.com/walk-with-c...

maven.com

Scratch to Scale: Large-Scale Training in the Modern World by Zachary Mueller on Maven

Learn the techniques used today to take your model training from Colab to Clusters

000

Zach Mueller @muellerzr.bsky.social · 02/10/2025

The main benefit to doing things this way is it's extremely fast (there are no communications involved since each process can figure out its own slice of the data), however it's more RAM heavy since the entire dataset needs to live in memory.

100

Zach Mueller @muellerzr.bsky.social · 02/10/2025

Afterwards, all that's needed is to move the data to the device and you're ready to perform distributed data parallelism!

100

Zach Mueller @muellerzr.bsky.social · 02/10/2025

For this to work, every process contains the entire dataset (so uses more RAM, but RAM is cheap). During the Dataset's __iter__() call each process will grab a pre-determined "chunk" of a global batch of items from the dataset in the __getitem__().

100

Zach Mueller @muellerzr.bsky.social · 02/10/2025

Dataset Sharding When performing distributed data parallelism, we split the dataset every batch so every device sees a different chunk of the data. There are different methods for doing so. One example is sharding at the *dataset* level, shown here.

120

Zach Mueller @muellerzr.bsky.social · 23/06/2025

Reach out to me if you need financial assistance/cost is too much. Happy to try and work within budgets (or company education stipends): scratchtoscale@gmail.com

020

Zach Mueller @muellerzr.bsky.social · 23/06/2025

Hi all 👋 Back very briefly to mention I’m working on a new course, and there’s a star-studded set of guest speakers 🎉 From Scratch to Scale: Distributed Training (from the ground up). From now until I’m done writing the course material, it’s 25% off :) maven.com/walk-with-co...

140

Reposted by Zach Mueller

kat cosgrove @kat.lol · 10/03/2025

AAAHHHHHHHHH BE NICE TO OPEN SOURCE MAINTAINERS OH MY GOD. SOME OF YOU ARE SO RUDE, WHO RAISED YOU

1455484

Zach Mueller @muellerzr.bsky.social · 11/03/2025

JK STRONG is still awesome. Solution (which they were already aware of this) was literally logout -> login! 11/10 customer support

000

Zach Mueller @muellerzr.bsky.social · 11/03/2025

I specifically mean my gym tracking app, STRONG. I’ve used it for 5 years, have a lifetime membership, and they decided that there needs to be internet access to check for premium to add warmup sets. Which is a bit insane. (It is a premium feature, but there’s better ways to do this)

110

Zach Mueller @muellerzr.bsky.social · 11/03/2025

Enshittification has come to my gym app. Lord help us all 😭

120

Zach Mueller @muellerzr.bsky.social · 08/03/2025

And I’ll try wearing it not sleeping only :)

010

Zach Mueller @muellerzr.bsky.social · 08/03/2025

(As a result I no longer wear mine and track my sleep using other methods that don’t have that psychological affect)

120

Zach Mueller @muellerzr.bsky.social · 08/03/2025

As someone who did Oura for 2 years, I can say it got to a point where my sleep depended on the Oura score, not the Oura score providing insight. So be weary of that long-term otherwise full agree 💯

120

Zach Mueller @muellerzr.bsky.social · 02/03/2025

Recently did my first lifting competition, overall quite happy where things ended up (bar bench) and was a ton of fun. Might wait a year and give it another go, but I did get first in the weight class! 145kg squat 180kg deadlift 90kg bench

2130

Zach Mueller @muellerzr.bsky.social · 26/02/2025

It comes up and down sporadically for me 🫠

000

Zach Mueller @muellerzr.bsky.social · 26/02/2025

I guess work is done for the day, everyone can go home 🤷‍♂️

100

Reposted by Zach Mueller

Alt National Park Service @altnps.bsky.social · 24/02/2025

A federal judge in Maryland issued a temporary restraining order on Monday, blocking members of Elon Musk’s so-called Department of Government Efficiency team and anyone “working on the DOGE agenda at the Department of Education” from accessing sensitive data until March 10.

106109251195

Zach Mueller @muellerzr.bsky.social · 24/02/2025

Nothing humbles you more than a front squat. Going from 260lbs back squat to 115lbs front squat was *lovely*

230

Reposted by Zach Mueller

Qian Li @qianli.dev · 23/02/2025

Interesting discussions between John Ousterhout and "Uncle Bob" on software design: method length, comments, and test-driven development. It's a great example of "agree to disagree". There is no silver bullet in software design. Let's focus on writing easy-to-parse code 👀 github.com/johnousterho...

Social preview of https://github.com/johnousterhout/aposd-vs-clean-code/blob/main/README.md

1225

Zach Mueller @muellerzr.bsky.social · 22/02/2025

Doesn’t this really translate as `x -= -1`? And just the whitespace makes it weird? (In a language that cares about whitespace… for the why…)

110

Zach Mueller @muellerzr.bsky.social · 19/02/2025

Estimated read time: 2-3 days Who’s going to be the first to start a study group?

060

Zach Mueller @muellerzr.bsky.social · 19/02/2025

Congrats all! Super impressive 🤩🤩🤩

020

Reposted by Zach Mueller

Thomas Wolf @thomwolf.bsky.social · 19/02/2025

After 6+ months in the making and over a year of GPU compute, we're excited to release the "Ultra-Scale Playbook": hf.co/spaces/nanot... A book to learn all about 5D parallelism, ZeRO, CUDA kernels, how/why overlap compute & coms with theory, motivation, interactive plots and 4000+ experiments!

hf.co

The Ultra-Scale Playbook - a Hugging Face Space by nanotron

The ultimate guide to training LLM on large GPU Clusters

217952

Zach Mueller @muellerzr.bsky.social · 14/02/2025

Anyone in my bubble built a lack rack before? 👀 Eyeballing doing this for the new place

110

Zach Mueller @muellerzr.bsky.social · 13/02/2025

0/10 can’t recommend: * Selling your house * Buying a house (*after* putting your old house on the market) * Lifting competition All in the same week

010

Zach Mueller @muellerzr.bsky.social · 12/02/2025

Quarto? :) (if that’s too techie, oop sorry)

110

Zach Mueller @muellerzr.bsky.social · 10/02/2025

If people want the source lmk, might upload in the morning if so

050

Zach Mueller @muellerzr.bsky.social · 10/02/2025

Update: Ran the equivalent of this space locally via fastapi + hooked into the LLM commit message generator got the job done, and very fast (since it’s a T5 model under the hood) huggingface.co/spaces/mamik...

huggingface.co

Commit Message Generator - a Hugging Face Space by mamiksik

Discover amazing ML apps made by the community

250

Zach Mueller @muellerzr.bsky.social · 10/02/2025

Me post seeing SmolLM2 quant is fast enough to run on my tiny laptop

1100