iiinova — where your digital self keeps evolving

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 09/07/2026

We vibe-tested our own paper, but the reviewers gave us the actual metrics: "From Feelings to Metrics" has been accepted to #COLM2026! 🎉 We turn "this model just feels better" into structured, user-aware evaluation! #COLM people - let’s grab a coffee and vibe-test some research in person! ☕️🌴

031

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 23/04/2026

Results: standard evaluation mask model quality. 🎭 Top models like GPT-5.1 were initially less preferred by beginners, but their win rate skyrocketed on personalized tasks (e.g., 9% to 94%). This shows how superior models are better for all users when evaluation is *user-aware*. 📈

100

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 23/04/2026

Can we scale it? We built a pipeline mirroring vibe-testing structure: 👤 Profile: Turns user descriptions into structured profiles. ✍️ Rewrite: Personalizes prompts for specific contexts. ⚖️ Judge: Compares models using user-defined criteria. Automation meets personalization. 🤖✨

100

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 23/04/2026

The anatomy of a vibe-test: 👉 Input: Users adapt task complexity and context to mimic their daily life. 👉 Output: The judge is the user. What counts as "clear" or "good tone" is defined by the individual’s perspective, not a static definition.

100

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 23/04/2026

Why do we "vibe-test" and ignore leaderboards? We ran a survey to find out. Our findings: ❌ 86% said they’ve used a model that "felt" significantly better (or worse) than its reported scores. ✅ 82% of you are "vibe-testing" models through direct interaction.

100

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 23/04/2026

Ever used a top-ranked LLM that just... felt wrong for you? You’re not alone. Instead of leaderboards, many of us turn to "vibe-testing" - manually comparing models to our own needs. But can we turn these feelings into a structured evaluation? New paper: "From Feelings to Metrics" 🧵

271

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 23/04/2026

In Rio for #ICLR2026 🇧🇷 and already had my first açaí! 🍧 Come chat LLM safety and evaluation, and stop by our ManagerBench poster (w/ @adisimhi)! - Tomorrow (Friday) @ 10:30 - Poster Session 3, Pavilion 4

050

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 13/10/2025

Had a blast at CoLM! It really was as good as everyone says, congrats to the organizers 🎉 This week I’ll be in New York giving talks at NYU, Yale, and Cornell Tech. If you’re around and want to chat about LLM behavior, safety, interpretability, or just say hi - DM me!

010

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 27/07/2025

In Vienna for #ACL2025, and already had my first (vegan) Austrian sausage! Now hungry for discussing: – LLMs behavior – Interpretability – Biases & Hallucinations – Why eval is so hard (but so fun) Come say hi if that’s your vibe too!

031

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 15/07/2025

🔄 Step 2: Cross-tuning. We swap instruction datasets between models with different pretraining. Result: Biases follow the pretrained model! PCA clearly shows models group by pretraining base, not by instruction. The bias “signature” stays intact, no matter the finetuning!

100

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 15/07/2025

🎲 Step 1: Training randomness. We finetune the same model 3× with different seeds. Result: Some variation in bias scores, but behavior patterns stay stable compared to MMLU variance. ✅ Aggregating across seeds reveals consistent trends.

130

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 15/07/2025

🧪 We introduce a two-step causal framework to disentangle the effects of: - Pretraining - Instruction tuning - Training randomness - 🍁 Bottom line - pretraining is the origin of bias. Finetuning? Just the messenger #CausalInference #TrustworthyAI #NLP

110

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 15/07/2025

🚨New paper alert🚨 🧠 Instruction-tuned LLMs show amplified cognitive biases — but are these new behaviors, or pretraining ghosts resurfacing? Excited to share our new paper, accepted to CoLM 2025🎉! See thread below 👇 #BiasInAI #LLMs #MachineLearning #NLProc

151