Sign in

klieret.bsky.social

@klieret.bsky.social
14 followers 17 following 81 posts
PostsRepliesMedia
klieret.bsky.social @klieret.bsky.social · 8h
Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do. SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%.
100
klieret.bsky.social @klieret.bsky.social · 29/07/2026
Thanks to AGI house for hosting me for a research spotlight! In the talk I look at agentic coding benchmarks, starting with SWE-bench and then covering CodeClash and ProgramBench youtu.be/wOtS8W-_QKs
youtu.be
Research Spotlight: AI Coding Benchmarks Are Saturating. What Comes Next? | Kilian Lieret
YouTube video by AGI House
000
klieret.bsky.social @klieret.bsky.social · 27/05/2026
DeepSWE finds that mini-swe-agent significantly outperforms ClaudeCode and Codex on the benchmark. The simpler the system, the better it generalizes (and mini's core agent class is just ~150 lines of code)
100
klieret.bsky.social @klieret.bsky.social · 10/04/2026
SWE-ReX was downloaded 190M times, mostly from a strong peak last year. One of the first & most simple ways to run agents at scale in cloud containers (modal, fargate, etc.)
100
klieret.bsky.social @klieret.bsky.social · 06/03/2026
LMs building gcc or web renderers is super impressive but let's not forget 1) the criteria for success were extremely clear 2) LMs were able to self-test if they were successful. If things get more ambiguous or more niche, things look quite differently
100
klieret.bsky.social @klieret.bsky.social · 04/03/2026
Bloated patches: LM generated solutions of SWE-bench tasks are consistently longer than human-written gold solutions (and it's not just comments) 🧵
111
klieret.bsky.social @klieret.bsky.social · 24/02/2026
Our new SWE-bench Multilingual leaderboard compares software engineering performance across 9 different languages as evaluated with mini-SWE-agent v2. Model rankings are significantly different between languages. Detailed stats & browsable trajectories in 🧵
110
klieret.bsky.social @klieret.bsky.social · 12/02/2026
Launching mini-SWE-agent 2.0, the simplest coding agent. Near SoTA performance, with the agent/model/environment only ~100 lines each. Powering benchmarks and RL training at NVIDIA, Anyscale, Stanford and many more!
100
klieret.bsky.social @klieret.bsky.social · 26/01/2026
New comparison tools on the main SWE-bench leaderboard. Using the same mini-SWE-agent setup for all models makes behavioral differences between models easy to compare. Thanks to Albert for adding the latest plot!
100
klieret.bsky.social @klieret.bsky.social · 02/12/2025
Updates to the official SWE-bench leaderboard: Minimax M2 is best open source model (but expensive!). Deepseek v3.2 reasoning close behind, very cheap, but very slow. GLM 4.6 reaches good performance fast and cheap. All independently evaluated with mini-swe-agent. Details in 🧵
110
klieret.bsky.social @klieret.bsky.social · 02/12/2025
At #NeurIPS till Monday! Stop by the SWE-smith poster (Wed 11am-2pm, C/D/E #113, Spotlight) for synthetic data for software engineering agents or AlgoTune (Thu 4:30pm-7:30pm, C/D/E #2514) for saturation-free benchmarks for high performance code
100
klieret.bsky.social @klieret.bsky.social · 26/11/2025
Very cool to see featured in the litellm docs! Litellm has been our default framework to support various models & get accurate cost tracking per API call.
100
klieret.bsky.social @klieret.bsky.social · 26/11/2025
Snapshot of the SWE-bench leaderboard: Opus 4.5 reclaims the top, though only narrowly ahead of Gemini 3 which is a lot cheaper. Both models require at least 100 agent steps to get to max performance.
100
klieret.bsky.social @klieret.bsky.social · 19/11/2025
Gemini 3 Pro sets new record on SWE-bench verified: 74%! (evaluated with minimal agent) Costs are 1.6x of GPT-5, but still cheaper than Sonnet 4.5. Gemini iterates longer than everyone; run your agent with a step limit of >100 for max performance. Details & full agent logs in 🧵
100
klieret.bsky.social @klieret.bsky.social · 30/09/2025
We evaluated Anthropic's Sonnet 4.5 with our minimal agent. New record on SWE-bench verified: 70.6%! Same price/token as Sonnet 4, but takes more steps, ending up being more expensive. Cost analysis details & link to full trajectories in 🧵
100
klieret.bsky.social @klieret.bsky.social · 11/09/2025
Very cool to see anyscale use mini-swe-agent in their large scale agent runs " because it is extremely simple and hackable and also gives good performance on software engineering problems without extra complexity" www.anyscale.com/blog/massive...
anyscale.com
Massively Parallel Agentic Simulations with Ray | Anyscale
Powered by Ray, Anyscale empowers AI builders to run and scale all ML and AI workloads on any cloud and on-prem.
110
klieret.bsky.social @klieret.bsky.social · 21/08/2025
Deepseek v3.1 chat scores 53.8% on SWE-bench verified with mini-SWE-agent. Tends to take more steps to solve problems than others (flattens out after some 125 steps). As a result effective cost is somewhere near GPT-5 mini. Details in 🧵
100
klieret.bsky.social @klieret.bsky.social · 20/08/2025
What if your agent uses a different LM at every turn? We let mini-SWE-agent randomly switch between GPT-5 and Sonnet 4 and it scored higher on SWE-bench than with either model separately. Read more in the SWE-bench blog 🧵
100
klieret.bsky.social @klieret.bsky.social · 08/08/2025
We evaluated the new GPT models with a minimal agent on SWE-bench verified. GPT-5 scores 65%, mini 60%, nano 35%. Still behind Opus 5 (68%), on par with Sonnet 4 (65%). But a lot cheaper, especially mini! Complete cost breakdown + details in 🧵
SWE-bench scores (same as text)
101
klieret.bsky.social @klieret.bsky.social · 08/08/2025
gpt-5-mini delivers software engineering for very cheap! We're seeing 60% on SWE-bench verified with just $18 total using our bare-bones 100 line agent. That's for solving 299/500 GitHub issues! Very fast, too! (1.5h total with 10 workers)
100
klieret.bsky.social @klieret.bsky.social · 07/08/2025
Play with gpt-5 in our minimal agent (guide in the 🧵)! gpt-5 really wants to solve anything in one shot, so some prompting adjustments are needed to have it behave like a proper agent. Still likes to cram in a lot into a single step. Full evals tomorrow!
100