Sign in

Chaitanya Malaviya

@cmalaviya.bsky.social
253 followers 94 following 20 posts

Senior research scientist @ GoogleDeepMind | benchmarking and evaluation | prev @upenn.edu @ai2.bsky.social, and @ltiatcmu.bsky.social‬ chaitanyamalaviya.github.io

PostsRepliesMedia
Reposted by Chaitanya Malaviya
Ai2 @ai2.bsky.social · 18/08/2025
LLMs power research, decision‑making, and exploration—but most benchmarks don’t test how well they stitch together evidence across dozens (or hundreds) of sources. Meet MoNaCo, our new eval for question-answering cross‑source reasoning. 👇
1166
Chaitanya Malaviya @cmalaviya.bsky.social · 30/07/2025
People at #ACL2025, come drop by our poster today & chat with me about how context matters for reliable language model evaluations! Jul 30, 11:00-12:30 at Hall 4X, board 424.
041
Reposted by Chaitanya Malaviya
Kyle Lo @ ICML2026 🇰🇷 @kylelo.bsky.social · 22/07/2025
issues w preference LM benchmarks: 🐡data contains cases where the "bad" response is just as good as chosen one 🐟model rankings can feel off (claude ranks lower than expected) led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025
2144
Chaitanya Malaviya @cmalaviya.bsky.social · 22/07/2025
Context is an overlooked aspect of language model evaluations. Check out how to incorporate context into evaluations in our TACL paper, how it changes evaluation conclusions and makes evaluation more reliable!
001
Chaitanya Malaviya @cmalaviya.bsky.social · 06/06/2025
Ever wondered what makes language models generate overly verbose, vague, or sycophantic responses? Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓
1103
Reposted by Chaitanya Malaviya
Manya Wadhwa @manyawadhwa.bsky.social · 22/04/2025
Evaluating language model responses on open-ended tasks is hard! 🤔 We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️. EvalAgent identifies 👩‍🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇
1214
Chaitanya Malaviya @cmalaviya.bsky.social · 13/11/2024
Excited to share ✨ Contextualized Evaluations ✨! Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓
192