iiinova — where your digital self keeps evolving

Reposted by Wayne

Chris Donahue @chrisdonahue.com · 05/03/2025

Inaugurating new acct to share work from my PhD student! Wayne et al have been running a live eval platform Copilot Arena - a VSCode extension serving code completions from AI systems to real developers. See 🧵 for findings and preprint Excited to be evaluating human-AI *workflows* holistically!

0103

Wayne @waynechi.bsky.social · 05/03/2025

Full Paper with additional analyses: arxiv.org/abs/2502.09328 Code: github.com/lmarena/copi... w/ Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, @chrisdonahue.com , @atalwalkar.bsky.social

arxiv.org

100

Wayne @waynechi.bsky.social · 05/03/2025

Our paper analyzes human preferences across 10 SOTA coding models, but we continue to add more models to the live Copilot Arena leaderboard on lmarena.ai!

100

Wayne @waynechi.bsky.social · 05/03/2025

Different data slices affect user preferences disproportionally. There is a drastic difference in relative model performance between real-world tasks such as frontend or backend development versus leetcode style coding challenges but little difference between programming languages.

100

Wayne @waynechi.bsky.social · 05/03/2025

We attribute these differences to a significant shift in our data distribution. Compared to previous benchmarks, Copilot Arena observes more programming languages (PL), natural languages (NL), longer context lengths, multiple task types, and various code structures.

100

Wayne @waynechi.bsky.social · 05/03/2025

Our leaderboard differs from existing evaluations. In particular, smaller models over perform in static benchmarks compared to real development workflows.

100

Wayne @waynechi.bsky.social · 05/03/2025

We evaluate models in a developer's IDE by presenting pairs of code completions generated by two different models. This workflow evaluates human preferences across models with real users and tasks in their native environment.

100

Wayne @waynechi.bsky.social · 05/03/2025

What do developers 𝘳𝘦𝘢𝘭𝘭𝘺 think of AI coding assistants? In October, we launched Copilot Arena to collect user preferences on real dev workflows. After months of live service, we’re here to share our findings in our recent preprint. Here's what we have learned /🧵

110

Wayne @waynechi.bsky.social · 26/02/2025

Got to test out InceptionAILab's newest model, Mercury Coder Mini, on Copilot Arena! Mercury Coder Mini is blazing fast and overtakes Codestral as the fastest coding model out there (0.24s end-to-end latency) while boasting similar performance (joint #2). Congrats to InceptionAILabs! 📸

010

Wayne @waynechi.bsky.social · 05/01/2025

I had the same problem. I only use cursor for newer, small projects. I use Copilot Arena's edit feature for projects in VSCode (but obviously I'm biased)

010

Wayne @waynechi.bsky.social · 31/12/2024

Deepseek v3 (FiM) is now available in Copilot Arena for free! Download at lmarena.ai/copilot

000

Wayne @waynechi.bsky.social · 27/12/2024

These lists are better than most "2024's best games" lists

000

Wayne @waynechi.bsky.social · 23/12/2024

Copilot Arena's leaderboard is now live on lmarena.ai/leaderboard! We've collected over 15k votes on 11 models (2 new models since our last blogpost release). Congrats @deepseek.bsky.social🥇and @anthropic.com🥇!

lmarena.ai

Chatbot Arena (formerly LMSYS): Free AI Chat to Compare & Test Best AI Chatbots

000

Wayne @waynechi.bsky.social · 13/12/2024

I'm not physically at NeurIPS, but my good friend @naveenraman.bsky.social will be presenting in my stead. In this work, we found that UI element ordering significantly affected GUI agent performance. Come check out the poster (and quiz Naveen) at the Workshop on Open-World Agents (OWA-2024)!

000

Wayne @waynechi.bsky.social · 05/12/2024

We've open sourced CopilotArena’s server code! Check out how we handle code completions and share your ideas for new system prompts! Github: github.com/lmarena/copi... Technical details in the blog: blog.lmarena.ai/blog/2024/co... Download Copilot now at: lmarena.ai/copilot

000

Wayne @waynechi.bsky.social · 20/11/2024

Trying out Bluesky. Will mostly be posting about Copilot Arena!

000