iiinova — where your digital self keeps evolving

Reposted by Sebastian Schuster

tddl @bachmannwettbewerb.bsky.social · 28/06/2026

Wir gratulieren Magdalena Schrefel zum 3sat Preis! 🥳 #tddl

Magdalena Schrefel mit kurzen, dunklen Haaren lächelt in die Kamera.

0457

Reposted by Sebastian Schuster

Gaurav Kamath @grvkamath.bsky.social · 04/03/2026

🚨New Paper!🚨 How do reasoning LLMs handle inferences that have no deterministic answer? We find that they diverge from humans in some significant ways, and fail to reflect human uncertainty… 🧵(1/10)

35820

Reposted by Sebastian Schuster

Gaurav Kamath @grvkamath.bsky.social · 29/07/2025

Our new paper in #PNAS (bit.ly/4fcWfma) presents a surprising finding—when words change meaning, older speakers rapidly adopt the new usage; inter-generational differences are often minor. w/ Michelle Yang, ‪@sivareddyg.bsky.social‬ , @msonderegger.bsky.social‬ and @dallascard.bsky.social‬👇(1/12)

33317

Reposted by Sebastian Schuster

Najoung Kim @najoung.bsky.social · 25/07/2025

👾 Full-time research assistant position (1 year) with @sebschu.bsky.social and me! 👾 We're looking for someone to join the research agent evaluation team, starting Fall 2025. Application link to be available soon, but feel free to send us your CV and/or come talk to us at #ACL2025. 🧵

1114

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

This work was done by our amazing team: @nedwards99.bsky.social, @yukyunglee.bsky.social, Yujun (Audrey) Mao, and Yulu Qin. And as always, it was super fun co-directing this with @najoung.bsky.social. We also thank Max Nadeau and Ajeya Cotra for initial advice and support.

110

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

Think your agent can do better? Check out the paper, download the data, and submit your agent to our leaderboard: 🌐Website: rexbench.com 📄Paper: arxiv.org/abs/2506.22598

rexbench.com

RExBench: A benchmark of machine learning research extensions for evaluating coding agents

110

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

We note that the current set of RexBench tasks is NOT extremely challenging for a PhD student-level domain expert. We hope to release a more challenging set of tasks in the near future, and would be excited about community contributions, so please reach out if you are interested! 🫵

110

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

What makes an extension difficult for agents? Statistically, tasks with more lines of change in the gold solution were harder. Meanwhile, repo size and popularity had marginal effects. Qualitatively, the performance aligned poorly with human-expert perceived difficulty!

130

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

What if we give them hints? We provided two levels of human-written hints. L1: information localization (e.g., files to edit) & L2: step-by-step guidance. With hints, the best agent’s performance improves to 39%, showing that substantial human guidance is still needed.

Figure showing the comparison of results, depending on level of hints for each agent.

110

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

Results! All agents we tested struggled on RExBench. The best-performing agents (OpenHands + Claude 3.7 Sonnet and Claude Code) only had a 25% average success rate across 3 runs. But we were still impressed that the top agents achieved end-to-end success on several tasks!Res

Result plot showing final success rate, execution success rate, and file recall for each of the agents. Final success rate was still only around 25% for the best agent.

120

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

The execution outcomes are evaluated against expert implementations of the extensions. This process is fully conducted inside our privately-hosted VM-based eval infra. This eval design and the target being novel extensions make RexBench highly resistant to data contamination.

110

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

We created 12 realistic extensions of existing AI research and tested 9 agents built upon aider, Claude Code (‪@anthropic.com‬) and OpenHands. The agents get papers, code, & extension hypotheses as inputs and produce code edits. The edited code is then executed.

110

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

Why do we focus on extensions? New research builds on prior work, so understanding existing research & building upon it is a key capacity for autonomous research agents. Many research coding benchmarks focus on replication, but we wanted to target *novel* research extensions.

110

Sebastian Schuster @sebschu.bsky.social · 02/07/2025

Can coding agents autonomously implement AI research extensions? We introduce RExBench, a benchmark that tests if a coding agent can implement a novel experiment based on existing research and code. Finding: Most agents we tested had a low success rate, but there is promise!

Screenshot of the RExBench preprint title page.

1134