went to lobste.rs after a while. I think they are still not fans of AI.
atharva
@atharvaraykar.com
117 followers 198 following 81 posts
i write at atharvaraykar.com i work @nilenso.com yes-anding the world.
i write at atharvaraykar.com i work @nilenso.com yes-anding the world.
![Our tasks typically use environments that do not significantly change unless directly acted upon by the agent. In contrast, real tasks often occur in the context of a changing environment.
[…]
Similarly, very few of our tasks are punishing of single mistakes. This is in part to reduce the expected cost of collecting human baselines.
This is not at all like the tasks I am doing.
METR acknowledges the messiness of the real world. They have come up with a “messiness rating” for their tasks, and the “mean messiness” of their tasks is 3.2/16.
By METR’s definitions, the kind of software engineering work that I’m mostly exposed to would score at least around 7-8, given that software engineering projects are path-dependent, dynamic and without clear counterfactuals. I have worked on problems that get to around 13/16 levels of messiness.
An increase in task messiness by 1 point reduces mean success rates by roughly 8.1%
Extrapolating from METR’s measured effect of messiness, GPT-5 would go from 70% to around 40% success rate for 2-hour tasks. This maps to my experienced reality.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:hadgpxog6eietjqixstnkdts/bafkreibcparvw3u6ipem5sbufp6jkmafhax7rantpynhs2n4pu5ynru2ne)
