linkedin.com
#opensource | Marouane Khoukh
Andrej Karpathy said you could train an #opensource ChatGPT-like model from scratch for under $100.
I wanted to see if that actually holds, so I tried the full NanoChat (https://lnkd.in/dTfWtFv2) on Nebius.
⚙️ I spun up an 8×H100 instance on Nebius, cloned karpathy/nanochat, and ran the full pipeline end-to-end.
The steps:
• Train a tokenizer on FineWeb-EDU (65K vocab)
• Pretrain a 560M-parameter Transformer to predict the next token
• Mid-train on conversational and reasoning data (SmolTalk, MMLU, GSM8K)
• Supervised Finetuning — aligning it to chat format
The results:
⏱️ The whole thing took a bit more than three hours.
💰 Cost: around $80 on Nebius .
📉 Validation bpb ≈ 0.81 📊 CORE ≈ 0.20 ⚡ MFU ≈ 48 %
So yes, it really does train a ChatGPT-style model under $100.
After the run, I opened the web UI and chatted with my model, and that’s the video below.
It’s pretty wild that you can now train, fine-tune, and talk to your own LLM in an afternoon.
Curious how the pipeli
