Reposted by Chaitanya MalaviyaAi2 @ai2.bsky.social · 18/08/2025LLMs power research, decision‑making, and exploration—but most benchmarks don’t test how well they stitch together evidence across dozens (or hundreds) of sources. Meet MoNaCo, our new eval for question-answering cross‑source reasoning. 👇 1166
Chaitanya Malaviya @cmalaviya.bsky.social · 30/07/2025People at #ACL2025, come drop by our poster today & chat with me about how context matters for reliable language model evaluations! Jul 30, 11:00-12:30 at Hall 4X, board 424. 041
Reposted by Chaitanya MalaviyaKyle Lo @ ICML2026 🇰🇷 @kylelo.bsky.social · 22/07/2025issues w preference LM benchmarks: 🐡data contains cases where the "bad" response is just as good as chosen one 🐟model rankings can feel off (claude ranks lower than expected) led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025 2144
Chaitanya Malaviya @cmalaviya.bsky.social · 22/07/2025Context is an overlooked aspect of language model evaluations. Check out how to incorporate context into evaluations in our TACL paper, how it changes evaluation conclusions and makes evaluation more reliable! 001
Chaitanya Malaviya @cmalaviya.bsky.social · 06/06/2025Ever wondered what makes language models generate overly verbose, vague, or sycophantic responses? Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓ 1103
Reposted by Chaitanya MalaviyaManya Wadhwa @manyawadhwa.bsky.social · 22/04/2025Evaluating language model responses on open-ended tasks is hard! 🤔 We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️. EvalAgent identifies 👩🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇 1214
Chaitanya Malaviya @cmalaviya.bsky.social · 13/11/2024Excited to share ✨ Contextualized Evaluations ✨! Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓ 192