pivot-to-ai.com
AI coding bot vendors tell you to use a context file with instructions for the chatbot on how to edit your project. Claude Code wants a CLAUDE.md, or there’s AGENTS.md in general. [_Anthropic_]
But does your AGENTS.md do anything? A team at ETH Zurich tested AGENTS files. They ran the bots over some test projects with and without AGENTS.md: [_arXiv_ _, PDF;__presentation_ _, video_]
> Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average. This observation holds across different LLMs, coding agents, and for both LLM-generated and developer-committed context files.
A chatbot-generated AGENTS file makes the coding agent succeed at coding tasks about as well as having _no_ AGENTS file — or slightly worse. A human-written AGENTS file gives slightly better success rates, but not much better.
Having an AGENTS.md increases the token cost quite a lot — whether the file is chatbot-generated or human-written. (Anthropic probably considers this a feature.) The bot churns through more stuff, but it’s no more likely to finish your task successfully.
Overviews of the file structure of the code repository are not useful to the agent. They don’t help the bot get on with editing the listed files any faster.
Does the AGENTS file do anything? If you mention particular tools in the AGENTS file, the coding agent will use them more often. That’s about it.
So when Anthropic recommended CLAUDE.md, they clearly didn’t measure if the instruction file actually does anything. They just chucked it in as a magical incantation you can invoke so you can hope the robot gets it right. This time. At least you might feel like you’re doing something.
This is very like how Anthropic writes Claude Code, which is mostly a pile of magical incantations telling the robot not to screw up yet again. Anthropic doesn’t know any other way to steer the bot.
The researchers’ recommendation: keep the instructions short. Be minimal. Or, of course, you could write a nice long file and chug through those tokens!
The closest anyone can come to a use for the AGENTS.md file is so you clarify your own thinking on how the project works, One developer on Reddit said: [_Reddit_ _,__archive_]
> The actual token-level context it provides matters less than the fact that writing it forces you to articulate things about your codebase that were previously just in your head.
Yeah, that’s what we used to call _writing_. Be right back, just starting the project wiki. I call it HUMANS.md.
* _Video_ _—_ _Podcast_
### Share this post:
* Share on Facebook (Opens in new window) Facebook
* Share on LinkedIn (Opens in new window) LinkedIn
* Share on Reddit (Opens in new window) Reddit
* Share on X (Opens in new window) X
* Share on Mastodon (Opens in new window) Mastodon
* Share on Bluesky (Opens in new window) Bluesky
* Email a link to a friend (Opens in new window) Email
*
### Like this:
Like Loading…
### _Related_
