Ai2 @ai2.bsky.social · 16/09/2025🚀 Introducing Fluid Benchmarking—an adaptive way to evaluate LLMs. Inspired by psychometrics, it tailors which questions to ask based on each model’s capability, making evals more efficient & reliable. 🧵 1207
Ai2 @ai2.bsky.social · 16/09/2025🎯 Core idea: not every question is equally informative when evaluating an LLM. Using Item Response Theory (IRT), we iteratively estimate ability and select highly informative questions—similar to computerized adaptive testing in education. 🧠📋 130
Ai2 @ai2.bsky.social · 16/09/2025We apply Fluid Benchmarking in the context of pretraining & see how the evaluated items completely change over the course of training—easy items at the beginning, difficult items at the end. 130
