iiinova — where your digital self keeps evolving

EvalEval Coalition @eval-eval.bsky.social · 10/12/2025

It's a wrap on EvalEval in San Diego! A jam packed day of learning, making new friends, critically examining the field of evals, and walking away with renewed energy and new collaborations! We have a lot of announcements coming, but first: EvalEval will be back for #ACL2026!

151

EvalEval Coalition @eval-eval.bsky.social · 13/11/2025

Continued.. 📉 Reporting on social impact dimensions has steadily declined, both in frequency and detail, across major providers 🧑‍💻 Sensitive content gets the most attention, as it’s easier to define and measure 🛡️Solution? Standardized reporting & safety policies (6/7)

100

EvalEval Coalition @eval-eval.bsky.social · 13/11/2025

Key Takeaways: ⛔️ First-party reporting is often sparse & superficial, with many reporting NO social impact evals 📉 On average, first-party scores are far lower than third-party evals (0.72 vs 2.62/3) 🎯 Third parties provide some complementary coverage (GPT-4 and LLaMA) (5/7)

110

EvalEval Coalition @eval-eval.bsky.social · 13/11/2025

🚨 AI keeps scaling, but social impact evaluations aren’t–and the data proves it 🚨 Our new paper, 📎“Who Evaluates AI’s Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations,” analyzes hundreds of evaluation reports and reveals major blind spots ‼️🧵 (1/7)

1113

EvalEval Coalition @eval-eval.bsky.social · 31/10/2025

📊Results & Findings 🧪 Experiments across 6 LLMs and 6 major benchmarks: 🏃Fluid Benchmarking outperforms all baselines across all four evaluation dimensions: efficiency, validity, variance, and saturation. ⚡️It achieves lower variance with up to 50× fewer items needed!!

110

EvalEval Coalition @eval-eval.bsky.social · 31/10/2025

🔍How to address this? 🤔 🧩Fluid Benchmarking: This work proposes a framework inspired by psychometrics that uses Item Response Theory (IRT) and adaptive item selection to dynamically tailor benchmark evaluations to each model’s capability level. Continued...👇

110

EvalEval Coalition @eval-eval.bsky.social · 31/10/2025

✨ Weekly AI Evaluation Paper Spotlight ✨ 🤔Is it time to move beyond static tests and toward more dynamic, adaptive, and model-aware evaluation? 🖇️ "Fluid Language Model Benchmarking" by @valentinhofmann.bsky.social et. al introduces a dynamic benchmarking method for evaluating language models

130

EvalEval Coalition @eval-eval.bsky.social · 24/10/2025

📊Key insights 🗳️Popular leaderboards (e.g., ChatArena, MTEB) can be exploited to distribute poisoned LLMs at scale 🔐Derivative models (finetuned, quantized, “abliterated”) are easy backdoor vectors. For instance, unsafe LLM variants often get downloaded as much as originals! Continued...

100

EvalEval Coalition @eval-eval.bsky.social · 24/10/2025

🔍 Method: 🧮Introduces TrojanClimb, a framework showing how attackers can: ⌨️ Simulate leaderboard attacks where malicious models achieve high test scores while embedding harmful pay loads (4 modalities) 🔒 Leverage stylistic watermarks/tags to game voting-based leaderboards

100

EvalEval Coalition @eval-eval.bsky.social · 24/10/2025

🌟 Weekly AI Evaluation Spotlight 🌟 🤖 Did you know malicious actors can exploit trust in AI leaderboards to promote poisoned models in the community? This week's paper 📜"Exploiting Leaderboards for Large-Scale Distribution of Malicious Models" by @iamgroot42.bsky.social explores this!

152

EvalEval Coalition @eval-eval.bsky.social · 17/10/2025

📊Key insights: ‼️Noise in benchmarks is substantial! For some datasets, up to 90% of reported “model errors” actually stem from *bad data* instead of model failures. 🧠 After benchmark cleaning, even top LLMs fail on simple, unambiguous platinum benchmark tasks. Continued...

143

EvalEval Coalition @eval-eval.bsky.social · 17/10/2025

🔍 Method: 🧹 Revise & clean 15 popular LLM benchmarks across 6 domains to create *platinum* benchmarks. 🤖 Use multiple LLMs to flag inconsistent samples via disagreement. ⚠️ Bad” questions fall into 4 types: mislabeled, contradictory, ambiguous, or ill-posed. Example 👇

130

EvalEval Coalition @eval-eval.bsky.social · 17/10/2025

✨Weekly AI Evaluation Paper Spotlight✨ 🕵️ Is benchmark noise and label errors masking the true fragility of LLMs? 🖇️"Do Large Language Model Benchmarks Test Reliability?" - This paper by @joshvendrow.bsky.social provides insights!

171