Does improving the training data quality help me close the gap between my models and the original GPT-2 small on my IFT eval?

Does improving the training data quality help me close the gap between my models and the original GPT-2 small on my IFT eval?

How mixture-of-experts LLMs work, both in theory and in real working code based on 'Build a Large Language Model (from Scratch)'.

I'd not been putting my JAX models on the Hugging Face Hub because it seemed hard, but then realised I could just upload the tensors converted to PyTorch

Might whether or not I use dropout while doing multiple epochs of fine-tuning hold the key to the variation in the results?

I wanted to have an easier way to put diagrams on this blog. D2 seems to work pretty well!

Switching from the hand-rolled GELU module I'd been using to PyTorch's built-in one unsurprisingly made training models faster -- but I was surprised to see it was 20% faster.

Having overtrained two models, I decided to use an ~equivalent number of FLOPs to train Chinchilla-optimal models. Would the rule hold up?

A point-in-time snapshot of how I'm currently using AI to help with my experiments, and with blogging about them

Does deliberately overtraining my own GPT-2 small style models match the original models' instruction-following abilities?

Before I get started on my attempts to make my models as good as the original GPT-2 ones at instruction-following, there was a bug I needed to fix.

The original GPT-2 small weights are much better than my own models at a specific instruction-following task. Time to look into why!

A friend asked me how many tokens per second I could get on Qwen 3.6 35B MoE on an RTX 3090. I probably dug into it more than he expected.

Explore how embedding width and layer count shape a GPT-2 model's parameter count.

Repurposing an old SFF PC as a dedicated LLM training box: a second-hand RTX 3090, an accidental 11-day train on a GTX 1660, and one dead CPU fan.

The capstone of my LLM from scratch journey: building up from a bigram-style model to GPT-2 small in JAX, watching the loss fall as each component goes in.

The capstone of my LLM from scratch journey: building up from a bigram-style model to GPT-2 small in JAX, watching the loss fall as each component goes in.

I'm going to build an LLM from scratch in JAX, but I wanted to get a training loop working first so that I could put it together piece by piece.

I'm going to build an LLM from scratch in JAX, but I wanted to get a training loop working first so that I could put it together piece by piece.

A quick way to help with debugging plumbing issues in JAX training loops.

As predicted by several people, one of my Marvell-based SFP+ modules had overheating problems. I've switched it over to a Broadcom-based one, and it looks better, though I can't be certain.

JAX memory management is tricky; data can be moved around unexpectedly if you don't make sure it's committed to a device.

Some basic notes on how to get JAX to load data to a particular device.

Using Safetensors to save Flax models isn't all that hard, so long as you know the trick.

Some (well, quite a lot of) initial thoughts about JAX, having spent a short time playing with it.

A quick update to show the (smallish) benefit of adding mini-heatsinks to a 10GBASE-T SFP+ module

I already had 2.5Gb/s working. Here's how I upgraded the whole house to 10Gb/s Ethernet -- including real iperf3 numbers, MikroTik switches, scary thermals, and what actually worked.

I already had 2.5Gb/s working. Here's how I upgraded the whole house to 10Gb/s Ethernet -- including real iperf3 numbers, MikroTik switches, scary thermals, and what actually worked.

Wired networking in the home and small offices has been pretty stagnant for ages, so upgrading my home from 2.5Gb ethernet to 10Gb meant I had to (re)learn a few things.

I've finished the main body of Raschka's book -- here are my thoughts on the appendices, and why I'm glad I read them after learning things the hard way.

I've finished the main body of Raschka's book -- here are my thoughts on the appendices, and why I'm glad I read them after learning things the hard way.

Wrapping up my 'Interventions' mini-series: what I've learned, what I've achieved, and what's next!

Wrapping up my 'Interventions' mini-series: what I've learned, what I've achieved, and what's next!

I wanted to revisit the instruction fine-tuning tests that I'd put on hold, and try them with my new models. I found that loss predicts real-world usefulness much less than I would have thought!

I wanted to revisit the instruction fine-tuning tests that I'd put on hold, and try them with my new models. I found that loss predicts real-world usefulness much less than I would have thought!

As an LLM is trained, it gradually learns how to generate increasingly coherent text. Here's an example.

Having worked out which combination of interventions into my model and training run improved loss the most, it was time to see how well that worked with a local run, which meant I had to learn about g...

Having worked out which combination of interventions into my model and training run improved loss the most, it was time to see how well that worked with a local run, which meant I had to learn about g...

Now that I've tried a number of interventions into my model and training run, and some of them seem to improve the model, how do we stack them together, and what are the results?

Now that I've tried a number of interventions into my model and training run, and some of them seem to improve the model, how do we stack them together, and what are the results?

How much of the variation in my training runs is real signal, and how much is random noise? I trained seven more models to find out.

How much of the variation in my training runs is real signal, and how much is random noise? I trained seven more models to find out.

I've been using PyTorch's Automated Mixed Precision (AMP) and lower-precision matrix multiplication logic for my training runs so far for larger batch sizes and faster training. Does doing that lead ...

I've been using PyTorch's Automated Mixed Precision (AMP) and lower-precision matrix multiplication logic for my training runs so far for larger batch sizes and faster training. Does doing that lead ...

Weight tying is apparently not used in modern LLMs, and intuitively would worsen performance in general. Does it?

Weight tying is apparently not used in modern LLMs, and intuitively would worsen performance in general. Does it?

What is weight decay, and what is the right value to set it to in order to get the best possible training run for our model?

What is weight decay, and what is the right value to set it to in order to get the best possible training run for our model?