iiinova — where your digital self keeps evolving

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 09/07/2026

We vibe-tested our own paper, but the reviewers gave us the actual metrics: "From Feelings to Metrics" has been accepted to #COLM2026! 🎉 We turn "this model just feels better" into structured, user-aware evaluation! #COLM people - let’s grab a coffee and vibe-test some research in person! ☕️🌴

031

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 23/04/2026

Ever used a top-ranked LLM that just... felt wrong for you? You’re not alone. Instead of leaderboards, many of us turn to "vibe-testing" - manually comparing models to our own needs. But can we turn these feelings into a structured evaluation? New paper: "From Feelings to Metrics" 🧵

271

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 23/04/2026

In Rio for #ICLR2026 🇧🇷 and already had my first açaí! 🍧 Come chat LLM safety and evaluation, and stop by our ManagerBench poster (w/ @adisimhi)! - Tomorrow (Friday) @ 10:30 - Poster Session 3, Pavilion 4

050

Reposted by Itay Itzhak @ COLM 🍁

Multilingual Representation Workshop @ EMNLP 2026 @mrl-workshop.bsky.social · 29/10/2025

Introducing Global PIQA, a new multilingual benchmark for 100+ languages. This benchmark is the outcome of this year’s MRL shared task, in collaboration with 300+ researchers from 65 countries. This dataset evaluates physical commonsense reasoning in culturally relevant contexts.

12210

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 13/10/2025

Had a blast at CoLM! It really was as good as everyone says, congrats to the organizers 🎉 This week I’ll be in New York giving talks at NYU, Yale, and Cornell Tech. If you’re around and want to chat about LLM behavior, safety, interpretability, or just say hi - DM me!

010

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 08/10/2025

Thrilled to be part of this work led by @adisimhi.bsky.social ! ManagerBench reveals a critical problem: ✅ LLMs can recognize harm ❌ But often choose it anyway to meet goals 🤖 Or overcorrect and become ineffective We need better balance! A must-read for safety folks!

040

Reposted by Itay Itzhak @ COLM 🍁

Yonatan Belinkov @boknilev.bsky.social · 07/10/2025

Traveling to #COLM2025 this week, and here's some work from our group and collaborators: Cognitive biases, hidden knowledge, CoT faithfulness, model editing, and LM4Science See the thread for details and reach out if you'd like to discuss more!

161

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 30/07/2025

At #ACL2025 and not sure what to do next? GEM 💎² is the place to be for awesome talks on the future of LLM evaluation. Come hear @GabiStanovsky, @EliyaHabba, @LChoshen and others rethink what it means to actually evaluate LLMs beyond accuracy and vibes. Thursday @ Hall C!

000

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 27/07/2025

In Vienna for #ACL2025, and already had my first (vegan) Austrian sausage! Now hungry for discussing: – LLMs behavior – Interpretability – Biases & Hallucinations – Why eval is so hard (but so fun) Come say hi if that’s your vibe too!

031

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 15/07/2025

🚨New paper alert🚨 🧠 Instruction-tuned LLMs show amplified cognitive biases — but are these new behaviors, or pretraining ghosts resurfacing? Excited to share our new paper, accepted to CoLM 2025🎉! See thread below 👇 #BiasInAI #LLMs #MachineLearning #NLProc

151

Reposted by Itay Itzhak @ COLM 🍁

Fazl Barez @fbarez.bsky.social · 01/07/2025

Excited to share our paper: "Chain-of-Thought Is Not Explainability"! We unpack a critical misconception in AI: models explaining their steps (CoT) aren't necessarily revealing their true reasoning. Spoiler: the transparency can be an illusion. (1/9) 🧵

28531

Reposted by Itay Itzhak @ COLM 🍁

Sebastian Gehrmann @sebgehr.bsky.social · 24/03/2025

Are you recovering from your @colmweb.org abstract submission? GEM has a non-archival track that allows you to submit a two-page abstract in parallel? Our workshop deadline is soon, please consider submitting your evaluation paper! You can find our call for papers at gem-benchmark.com/workshop

011

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 17/03/2025

New paper alert! Curious how small prompt tweaks impact LLM accuracy but don’t want to run endless inferences? We got you. Meet DOVE - a dataset built to uncover these sensitivities. Use DOVE for your analysis or contribute samples -we're growing and welcome you aboard!

041

Reposted by Itay Itzhak @ COLM 🍁

Tal Haklay @talhaklay.bsky.social · 06/03/2025

1/13 LLM circuits tell us where the computation happens inside the model—but the computation varies by token position, a key detail often ignored! We propose a method to automatically find position-aware circuits, improving faithfulness while keeping circuits compact. 🧵👇

1268

Reposted by Itay Itzhak @ COLM 🍁

Martin Tutek @mtutek.bsky.social · 21/02/2025

🚨🚨 New preprint 🚨🚨 Ever wonder whether verbalized CoTs correspond to the internal reasoning process of the model? We propose a novel parametric faithfulness approach, which erases information contained in CoT steps from the model parameters to assess CoT faithfulness. arxiv.org/abs/2502.14829

arxiv.org

Measuring Faithfulness of Chains of Thought by Unlearning Reasoning Steps

When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. However, despite mu...

24813

Itay Itzhak @ COLM 🍁 @itay-itzhak.bsky.social · 19/02/2025

We usually blame hallucinations on uncertainty or missing knowledge. But what if I told you that LLMs hallucinate even when they *know* the correct answer - and they do it with *high certainty* 🤯? Check out our new paper that challenges assumptions on AI trustworthiness! 🧵👇

020

Reposted by Itay Itzhak @ COLM 🍁

Sebastian Gehrmann @sebgehr.bsky.social · 12/02/2025

GEM is so back! Our workshop for Generation, Evaluation, and Metrics is coming to an ACL near you. Evaluation in the world of GenAI is more important than ever, so please consider submitting your amazing work. CfP can be found at gem-benchmark.com/workshop

095