iiinova — where your digital self keeps evolving

Aarash Feizi @aarashfeizi.bsky.social · 27/02/2025

🧵 6/7 ✅ Beyond benchmarking, PairBench can be used during VLM training & fine-tuning to detect biases early and improve evaluation methods! This could lead to more trustworthy, consistent AI systems for real-world tasks. 🚀

120

Aarash Feizi @aarashfeizi.bsky.social · 27/02/2025

🧵 5/7 ✅ PairBench correlates strongly with existing benchmarks, meaning it can serve as a low-cost alternative to expensive human-annotated benchmarks! This makes it easier to compare and rank models efficiently—without excessive computational costs.

100

Aarash Feizi @aarashfeizi.bsky.social · 27/02/2025

🧵 4/7 Instead of blindly picking a judge model, we should ask: 🔹 What task is being evaluated? 🔹 What metric matters most? ✅ PairBench helps match the right VLM to the right task, improving fairness & reliability in auto-evaluation.

100

Aarash Feizi @aarashfeizi.bsky.social · 27/02/2025

🧵 3/7 🚨 No single VLM is the best! Models vary drastically across PairBench metrics. Although some align well with human judgements, they may struggle at symmetry, smoothness, or controllability—making their scores unreliable! 📄 More failure cases in our paper’s appendix!

100

Aarash Feizi @aarashfeizi.bsky.social · 27/02/2025

🧵 2/7 ✅ Surprising (and concerning) result: Most VLMs lack symmetry! 🤯 In theory, sim(A, B) = sim(B, A)—but in practice? Many models fail! For example, simply swapping the order of the input images makes GPT-4o and Gemini 1.5 Pro change their decision and scores drastically. 🔄

110

Aarash Feizi @aarashfeizi.bsky.social · 27/02/2025

🧵 1/7 Vision language models (VLMs) are widely used as automated evaluators, but can they actually compare data reliably? 🤔 ✅ PairBench systematically tests how well VLMs judge similarity across modalities, revealing key strengths & weaknesses in their decisions.

100

Aarash Feizi @aarashfeizi.bsky.social · 27/02/2025

🚨 Excited to introduce PairBench! 🚨 💡 TL;DR: VLM-judges can fail at data comparison! ✅ PairBench helps you pick the right one by testing alignment, symmetry, smoothness & controllability—ensuring reliable auto-evaluation. 📄 Paper: arxiv.org/abs/2502.15210 🧵 Thread: 👇

112