iiinova — where your digital self keeps evolving

We tried very hard to get this right, and have spent the last 5 months working carefully to ensure rigor. If you made it this far, take a look at the full 68 pages: arxiv.org/abs/2504.20879 Any feedback or corrections are of course very welcome.

0120

Very proud of this work that we led by Shivalika Singh and @mziizm.bsky.social with Yiyang Nan, Alex Wang, Daniel D'Souza, @sayash.bsky.social, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, @shaynelongpre.bsky.social @nlpnoah.bsky.social @beyzaermis.bsky.social

This was an uncomfortable paper to work on because it asks us to look in the mirror as a community. As scientists, we must do better. As a community, I hope we can demand better. We make very clear the 5 changes needed.

161

Overall, our work suggests that engagement from a handful of providers and preferential policies from Arena towards the same small group have created conditions to overfit to Arena-specific dynamics rather than general model quality.

120

We show that access to Chatbot Arena data yields substantial benefits. While using Arena-style data in training boosts win rates by 112%, this improvement doesn't transfer to tasks like MMLU, indicating overfitting to Arena's quirks rather than general performance gains.

These data differences stem from some key policies that benefit a handful of providers: 1) proprietary models sampled at higher rates to appear in battles 📶 2) open-weights + open-source models removed from Arena more often 🚮 3) How many private variants 🔍

110

We also observe large differences in Arena Data Access Chatbot Arena is a open community resource that provides free feedback but 61.3% of all data goes to proprietary model providers.

120

We even do real world private testing using Aya Vision models to show the gains you can expect. Even when you test identical checkpoints we see gains. This is the most conservative case where quality is identical.

There is no reasonable scientific justification for this practice. Being able to choose the best score to disclose enables systematic gaming of Arena score. This advantage increases with number of variants and if all other providers don’t know they can also private test.

171

There is an unspoken policy of hidden testing that benefits a small subset of providers. Providers can choose what score to disclose and retract all others. At an extreme, we see testing of up to 27 models in lead up to releases.

140

We spent 5 months analyzing 2.8M battles on the Arena, covering 238 models across 43 providers. We show that preferential policies engaged in by a handful of providers lead to overfitting to Arena-specific metrics rather than genuine AI progress.

240

Marzieh Fadaee @mziizm.bsky.social · 30/04/2025

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

19569

Reposted by Sara Hooker

1/ Science is only as strong as the benchmarks it relies on. So how fair—and scientifically rigorous—is today’s most widely used evaluation benchmark? We took a deep dive into Chatbot Arena to find out. 🧵

1286

Reposted by Sara Hooker

Jonathan Wenger @jwenger.bsky.social · 16/04/2025

This has been a topic close to my heart for a long time. We have an awesome lineup of speakers who have made deep contributions to open-source in ML, e.g. @sarahooker.bsky.social , @chrisrackauckas.bsky.social, Matt Johnson, Tri Dao, @stellaathena.bsky.social, Evan Shelhamer.

0102

Reposted by Sara Hooker

Isra Salazar @israsalazar.bsky.social · 10/04/2025

Today we are releasing Kaleidoscope 🎉 A comprehensive multimodal & multilingual benchmark for VLMs! It contains real questions from exams in different languages. 🌍 20,911 questions and 18 languages 📚 14 subjects (STEM → Humanities) 📸 55% multimodal questions

1266

Sara Hooker @sarahooker.bsky.social · 19/03/2025

It is rare I get to completely disconnect. Very grateful for this week in Patagonia.

2330

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 05/03/2025

We're particularly proud to release Aya Vision 8B - it's compact 🐭 and efficient 🐎, outperforming models up to 11x its size 📈. Releasing open weights helps to make breakthroughs in VLMs accessible to the research community.

1144

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 06/03/2025

Just 2 days after launch, Aya Vision is trending on @hf.co 🔥🔥 We launched open-weights with the goal of making VLM breakthroughs accessible to the research community - so exciting to see such a positive response. huggingface.co/CohereForAI/...

072

Reposted by Sara Hooker

(((Steve Chapman))) @stevechapman.bsky.social · 03/03/2025

Love this post by @sarahooker.bsky.social on that other platform: "The first step of any meaningful pursuit is to severely underestimate its difficulty."

051

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 04/03/2025

Introducing ✨ Aya Vision ✨ - an open-weights model to connect our world through language and vision Aya Vision adds breakthrough multimodal capabilities to our state-of-the-art multilingual 8B and 32B models. 🌿

183

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 25/02/2025

An important topic in AI is the climate impacts of the energy-intensive computing hardware needed to train and deploy AI models ⚡ Our policy primer explores ways to move towards more sustainable AI. 🌱 📜 cohere.com/research/pap...

021

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 11/02/2025

Does more compute equate with greater risk?⚡️What is our track record predicting what risks emerge with scale? 📈 In this work led by Sara Hooker, we seek to understand the viability of compute thresholds ⚖️ as a way to mitigate risk. 🦺 arxiv.org/abs/2407.05694

011

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 18/02/2025

In this work, we ask "How does model merging stack up when optimizing language models for diverse multitask learning?" 📚🧩 📜https://arxiv.org/abs/2410.10801

051

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 22/01/2025

Aya Expanse, our open-weight 32B model, outperforms drastically larger models including Claude, Mistral Large 2, & Llama 405B on Scale's Private Multilingual Protocol. We are proud to work on global AI that is efficient and accessible 🔥

163

Reposted by Sara Hooker

Jekaterina Novikova @j-novikova-nlp.bsky.social · 23/01/2025

Our paper is accepted to ICLR! INCLUDE: Evaluating Multilingual LLMs with Regional Knowledge (arxiv.org/abs/2411.19799) A benchmark of ~200k QA pairs across 44 languages, capturing real-world cultural nuances. A collaborative effort led by @cohereforai.bsky.social, with contributors worldwide. /1

1114

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 12/02/2025

In this cross-institutional work, we introduce technical governance for AI and 100+ 🔢 open technical problems 🔧. We provide a taxonomy of open problem areas in TAIG organized by governance capacities and governance targets. 📜https://arxiv.org/pdf/2407.14981

022

Reposted by Sara Hooker

Cohere Labs @cohereforai.bsky.social · 13/02/2025

The C4AI Research Grant program is proud to have supported a project focused on building LLM tools for teachers 🧑‍🏫 This project focused on adapting educational materials to students’ skill levels, ensuring more effective and responsible AI integration in classrooms.

141

The next year matters for this. Understanding the dynamics between national efforts and the progress of science will be important. I am very grateful I was invited to participate in these rooms, and to be part of several panels of forums.

070

... but we should be careful that an emphasis on sovereign AI doesn't undermine our shared experience. At the Elysee dinner, Macron put this tension well --saying it would be unfortunate if we end up with a Chinese, French, US model – biased towards our differences.

160

For me, this captures the complexity of this summit. Critical topics were on the agenda for the first time: global frontier AI, multilingual, sustainability. And most countries now want to embark on national AI policies -- likely improve how technology serves our world...

140

… in parallel there a clear movement towards equating AI progress with National capabilities. Much of the summit centered on national champions. There was also talk about an AI hub in Europe similar to CERN for Europe.

No-one could have foreseen JD starting his speech saying “I'm not here this morning to talk about AI safety” However, this was part of a wider trend – even the EU seems at pains to say that safety should be balanced with innovation.

140

A few years ago at the first summit, the idea of global governance and coordination of AI seemed daunting but feasible. At this summit, far more of the globe was represented, but the overall trend was one of realism and fault lines fell along national boundaries

151

What didn’t go well: This is an example of a summit where despite best efforts, current events overtook the agenda. Summits are planned for months in advance and can’t anticipate rapid changes in geopolitics.

The panels at the actual summit did not say much – many people simply read from prepared remarks. But follow-up day at Station F felt like a rockstar concert, complete with long lines. It bottled the energy we all feel in Paris -- of new tech scene.

Work on multilingual global AI 🌎, access, AI efficiency 🌿 was centered and deemed critical at this summit. I think this is part of a broader reality of grappling w the widening divide frontier AI introduces – in terms of compute access and geographical disparities.

There was meaningful policy announced with compute investments + new foundation. Could be very impactful if executed well. Number of gpus is rarely the only ingredient, so many national datacenters are underutilized because of cumbersome bureaucracy to proportion resources

France created right mix of invitees + executed incredibly well. I could find the same people that were at bletchley summit, but more global influence. India was co-host, Africa, China, Korea, Japan, Singapore represented. Will be a lasting legacy, of setting a global lens.