Ai2 @ai2.bsky.social · 16/09/2025We apply Fluid Benchmarking in the context of pretraining & see how the evaluated items completely change over the course of training—easy items at the beginning, difficult items at the end. 130
Ai2 @ai2.bsky.social · 16/09/2025📈 Stable signals: adaptive item selection cuts step-to-step variance & delays saturation. 🧼 Cleaner data: fewer mislabeled items than random sampling at the same budget. ➕ Results generalize better across benchmarks targeting the same capability. ⚡️ 140
Ai2 @ai2.bsky.social · 16/09/2025On MMLU, Fluid Benchmarking leads to lower variance with ~50× fewer questions than standard evals + increased validity. 130
