Reposted by Iska
NanoGPT pretraining runs now take 39.9 seconds for a 124M model(!!)
now, 124M is *tiny* so it might not seem relevant. But much of the gains in LLM pretraining are data quality and a great way to find quality data is to train on it & measure the lift
