How mixture-of-experts LLMs work, both in theory and in real working code based on 'Build a Large Language Model (from Scratch)'.

I'd not been putting my JAX models on the Hugging Face Hub because it seemed hard, but then realised I could just upload the tensors converted to PyTorch

I wanted to have an easier way to put diagrams on this blog. D2 seems to work pretty well!

Switching from the hand-rolled GELU module I'd been using to PyTorch's built-in one unsurprisingly made training models faster -- but I was surprised to see it was 20% faster.

Having overtrained two models, I decided to use an ~equivalent number of FLOPs to train Chinchilla-optimal models. Would the rule hold up?

A point-in-time snapshot of how I'm currently using AI to help with my experiments, and with blogging about them

The original GPT-2 small weights are much better than my own models at a specific instruction-following task. Time to look into why!

A friend asked me how many tokens per second I could get on Qwen 3.6 35B MoE on an RTX 3090. I probably dug into it more than he expected.

Explore how embedding width and layer count shape a GPT-2 model's parameter count.

Repurposing an old SFF PC as a dedicated LLM training box: a second-hand RTX 3090, an accidental 11-day train on a GTX 1660, and one dead CPU fan.

The capstone of my LLM from scratch journey: building up from a bigram-style model to GPT-2 small in JAX, watching the loss fall as each component goes in.

I'm going to build an LLM from scratch in JAX, but I wanted to get a training loop working first so that I could put it together piece by piece.

A quick way to help with debugging plumbing issues in JAX training loops.

As predicted by several people, one of my Marvell-based SFP+ modules had overheating problems. I've switched it over to a Broadcom-based one, and it looks better, though I can't be certain.

JAX memory management is tricky; data can be moved around unexpectedly if you don't make sure it's committed to a device.

Some basic notes on how to get JAX to load data to a particular device.

Using Safetensors to save Flax models isn't all that hard, so long as you know the trick.

Some (well, quite a lot of) initial thoughts about JAX, having spent a short time playing with it.

I already had 2.5Gb/s working. Here's how I upgraded the whole house to 10Gb/s Ethernet -- including real iperf3 numbers, MikroTik switches, scary thermals, and what actually worked.

Wired networking in the home and small offices has been pretty stagnant for ages, so upgrading my home from 2.5Gb ethernet to 10Gb meant I had to (re)learn a few things.

I've finished the main body of Raschka's book -- here are my thoughts on the appendices, and why I'm glad I read them after learning things the hard way.

Wrapping up my 'Interventions' mini-series: what I've learned, what I've achieved, and what's next!

I wanted to revisit the instruction fine-tuning tests that I'd put on hold, and try them with my new models. I found that loss predicts real-world usefulness much less than I would have thought!

As an LLM is trained, it gradually learns how to generate increasingly coherent text. Here's an example.

Having worked out which combination of interventions into my model and training run improved loss the most, it was time to see how well that worked with a local run, which meant I had to learn about g...

Now that I've tried a number of interventions into my model and training run, and some of them seem to improve the model, how do we stack them together, and what are the results?

How much of the variation in my training runs is real signal, and how much is random noise? I trained seven more models to find out.

I've been using PyTorch's Automated Mixed Precision (AMP) and lower-precision matrix multiplication logic for my training runs so far for larger batch sizes and faster training. Does doing that lead ...

Weight tying is apparently not used in modern LLMs, and intuitively would worsen performance in general. Does it?

What is weight decay, and what is the right value to set it to in order to get the best possible training run for our model?

The learning rate is an essential hyperparameter for training models with gradient descent, but what actual values should you use, and what does it mean to schedule it?

Having bias terms for the query, key, and value attention weights is apparently no longer used because it doesn't help. Let's check that it really doesn't for our model!

Does removing dropout improve our baseline model's test loss? Yes, absolutely, and even more than gradient clipping did.

Does adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected.

I want to try a bunch of interventions like gradient clipping and removing dropout to see if my models get better. I need a baseline train without them so that I can be sure of the results.

A worked example of packaging a from-scratch GPT-2-style model for the Hugging Face Hub so it loads via from_pretrained, runs with pipeline, and trains with Trainer -- with notes on tokeniser gotchas.

I've trained seven models using the GPT-2 architecture: let's share them on Hugging Face!

I was unhappy with the LLM-as-a-judge instruction fine-tuning results I got when comparing my various base models. Could I make them any better?

Having trained a base model from scratch on my own machine over 48 hours, I wanted to make it faster by training with multiple GPUs in the cloud.

I felt like it should be possible to train a GPT-2 small level model on my own hardware using modern tools and open datasets from scratch. It was!

Having finished the main body of 'Build an LLM (from scratch)', it's time to think about what I need to do to treat this project as fully done

Coming to the end of 'Build an LLM (from scratch)'! We evaluate the quality of the responses our model produces.

Some notes on the first part of chapter 7 of 'Build an LLM (from scratch)': instruction fine-tuning

Back when I started playing with LLMs, I found that you could build a (very basic) chatbot with a base model -- no instruction fine-tuning at all! Does that work with GPT-2?

Revisiting Karpathy’s text-generating RNNs with PyTorch’s built-in LSTM class — a practical look at why training sequence models is so different from Transformers.

After all the hard work, chapter 6 in 'Build an LLM (from scratch)' is a nice easy one -- how do we take a next-token predictor and turn it into a classifier?

Finally, we train an LLM! The final part of Chapter 5 of Build an LLM (from Scratch) runs the model on real text, then loads OpenAI’s GPT-2 weights for comparison.

Andrej Karpathy's 2015 blog post 'The Unreasonable Effectiveness of Recurrent Neural Networks' went viral in its day, for good reason. How does it read ten years later?

Raschka calls out perplexity in a sidebar, but I wanted to understand it in a little more depth

Starting training our LLM requires a loss function, which is called cross entropy loss. What is this and why does it work?