iiinova — where your digital self keeps evolving

Wayne

@waynechi.bsky.social

36 followers 170 following 17 posts

CS Ph.D. at CMU. Building Copilot Arena. Editor at blog.ml.cmu.edu

Posts Replies Media

Wayne @waynechi.bsky.social · 05/03/2025

Our paper analyzes human preferences across 10 SOTA coding models, but we continue to add more models to the live Copilot Arena leaderboard on lmarena.ai!

100

Wayne @waynechi.bsky.social · 05/03/2025

We attribute these differences to a significant shift in our data distribution. Compared to previous benchmarks, Copilot Arena observes more programming languages (PL), natural languages (NL), longer context lengths, multiple task types, and various code structures.

100

Wayne @waynechi.bsky.social · 05/03/2025

Our leaderboard differs from existing evaluations. In particular, smaller models over perform in static benchmarks compared to real development workflows.

100

Wayne @waynechi.bsky.social · 05/03/2025

We evaluate models in a developer's IDE by presenting pairs of code completions generated by two different models. This workflow evaluates human preferences across models with real users and tasks in their native environment.

100

Wayne @waynechi.bsky.social · 05/03/2025

What do developers 𝘳𝘦𝘢𝘭𝘭𝘺 think of AI coding assistants? In October, we launched Copilot Arena to collect user preferences on real dev workflows. After months of live service, we’re here to share our findings in our recent preprint. Here's what we have learned /🧵

110

Wayne @waynechi.bsky.social · 26/02/2025

Got to test out InceptionAILab's newest model, Mercury Coder Mini, on Copilot Arena! Mercury Coder Mini is blazing fast and overtakes Codestral as the fastest coding model out there (0.24s end-to-end latency) while boasting similar performance (joint #2). Congrats to InceptionAILabs! 📸

010

Wayne @waynechi.bsky.social · 13/12/2024

I'm not physically at NeurIPS, but my good friend @naveenraman.bsky.social will be presenting in my stead. In this work, we found that UI element ordering significantly affected GUI agent performance. Come check out the poster (and quiz Naveen) at the Workshop on Open-World Agents (OWA-2024)!

000

Wayne @waynechi.bsky.social · 10/12/2024

Bruh what... 💀

000

Wayne @waynechi.bsky.social · 05/12/2024

We've open sourced CopilotArena’s server code! Check out how we handle code completions and share your ideas for new system prompts! Github: github.com/lmarena/copi... Technical details in the blog: blog.lmarena.ai/blog/2024/co... Download Copilot now at: lmarena.ai/copilot

000