Fern @fernbear.bsky.social · 03/02/2025This is a classic example of _why_ choose-one-of-n datasets need to have large-scale, crowd-sourced statistics and should use the KL-divergence instead of cross-entropy. Reviewers will be more biased than a crowd, it's a high variance+bias estimator, it can harm research. 120
Fern @fernbear.bsky.social · 25/11/2024Thanks for 100 followers, y'all! Happened so fast and can't wait to put out more research on here! 😊❤️ 170
Fern @fernbear.bsky.social · 25/11/2024New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks 2333