iiinova — where your digital self keeps evolving

Dr Waku @drwaku.bsky.social · 27/06/2025

Very interesting result, and it makes me think that CoT errors and agent capabilities are more predictable than we realize. It might make it possible to complete longer tasks even sooner, if one can scale inference compute time linearly based on predicted task length. 9/10

110

Dr Waku @drwaku.bsky.social · 27/06/2025

Overall, doubling every 7 months means 10x task length every two years (26 months). Similar rules apply for task accuracy. Going from 50% to 80% takes one extra year; from 50% to 90% takes two. Going from 90% to 99% accuracy (or 99% to 99.9%, etc) takes an extra two years. 8/10

100

Dr Waku @drwaku.bsky.social · 27/06/2025

Interestingly, humans have a better long-tail success rate than AIs here, possibly because we are better at fixing past mistakes. (It could also be an artifact of the data, which combines multiple humans of different expertise together.) 7/10

100

Dr Waku @drwaku.bsky.social · 27/06/2025

It also makes intuitive sense, at least to me. AI agents are not very good at fixing mistakes in past subtasks, as you will see if you read confused chains of thought. Modeling this as "must succeed at all prior tasks" is reasonable. 6/10

100

Dr Waku @drwaku.bsky.social · 27/06/2025

The original paper could not address this, but Toby Ord applied the field of survival analysis (great name) and argued for a constant hazard rate. This is the simplest possible model, and it explains the data just as well as more complex ones. 5/10

110

Dr Waku @drwaku.bsky.social · 27/06/2025

What is often overlooked about this result is that these numbers were for 50% task success rate (or possibly for 80% success rate). What if you need 90%, or 95% task success rate? Does today's 95% success rate still apply to double-length tasks in 7 months' time? 4/10

100

Dr Waku @drwaku.bsky.social · 27/06/2025

That’s the conclusion of Toby Ord at Oxford (@tobyord.bsky.social), who examined earlier data from METR. The headline result there was that agents can complete tasks that take humans one hour, and the length of solvable tasks is doubling every 7 months. 3/10

110

Dr Waku @drwaku.bsky.social · 27/06/2025

For example, if you have a 10% of failure every minute, and you are running for 15 minutes, you have to succeed fifteen times in a row. This means you have a 1-0.9^15 = 79% chance of failure. It's multiplicative. 2/10

100

Dr Waku @drwaku.bsky.social · 27/06/2025

AI agents have a half-life for their success rates at completing tasks. Yes, the same type of half-life as in nuclear chemistry: a constant chance of task failure (= radioactive decay) in each time period... 1/10

111

Dr Waku @drwaku.bsky.social · 05/06/2025

May I suggest not pronouncing it as we do, which is to say "kay see" ("K C"). It's just confusing. Humour aside, more details on whether the name change is good for safety can be found here: thezvi.substack.com/... 2/2

thezvi.substack.com

AI #119: Goodbye AISI?

AISI is being rebranded highly non-confusingly as CAISI.

000

Dr Waku @drwaku.bsky.social · 05/06/2025

The US AI Safety Institute (AISI) is being renamed into the Center for AI Standards and Innovation (CAISI). It has a pretty similar sounding mandate. But I would like to point out that Canada had this name first with the Canadian AI Safety Institute (CAISI)! :O 1/2

100

Dr Waku @drwaku.bsky.social · 28/05/2025

That’s why AI is not like other technologies. It’s another, faster feedback loop, an intellectual loop, if you will. Evolution kicking into a higher gear. (9/10)

100

Dr Waku @drwaku.bsky.social · 28/05/2025

Many seem to be treating this as just another technology to get ahead in. It's not. This is a transformative technology that represents a step change in the speed of evolution. It can and will overwhelm us as a species if we don't pay attention. (8/10)

110

Dr Waku @drwaku.bsky.social · 28/05/2025

We're used to the risk of someone pressing the red button to launch nuclear war. At the risk of anthropomorphizing, this is more like having a child genius who is not bounded by biological limits on intelligence, and who is learning at a frightening pace. (7/10)

100

Dr Waku @drwaku.bsky.social · 28/05/2025

We risk abdicating our position as the intelligent species on this planet that is charting our own destiny. After all, what is charting but the accumulation of a lot of decisions made throughout society? Decisions made by whatever intelligence was brought to bear on the problems? (6/10)

110

Dr Waku @drwaku.bsky.social · 28/05/2025

Once the automation of intelligence is driving technological development and the creation of ideas, once we reach the technological singularity, important changes could be happening very rapidly indeed. Perhaps every day. Perhaps every second. (5/10)

100

Dr Waku @drwaku.bsky.social · 28/05/2025

That's not going to happen this time. Humans will slowly stop doing jobs altogether. We will reduce our reliance on human brains for solving our most difficult problems. Already, the correlation between AI tool use and critical thinking is strongly negative (r = -0.68). (4/10)

110

Dr Waku @drwaku.bsky.social · 28/05/2025

Now, we have AI. AI will help solve problems that were infeasible before. Generally, as automation takes hold, people start doing more abstract tasks. The new jobs may not require as many people as the old ones, but the economy is diverse and people shuffle around. (3/10)

110

Dr Waku @drwaku.bsky.social · 28/05/2025

It took 600 million years for chemistry to form life, 6,000 years for humans to form civilization, and 128 years from Edison’s light bulb to the iPhone. Evolution is a series of faster and faster feedback loops, from chemistry to biology to society to technology. (2/10)

210

Dr Waku @drwaku.bsky.social · 28/05/2025

AI is not like other technologies. AI doesn't just make everything we were doing before faster and cheaper. It’s another, faster feedback loop of reasoning and intelligence beyond the human brain. It will change life as we know it. Here’s why. (1/10)

120

Dr Waku @drwaku.bsky.social · 16/05/2025

These are not affiliate links. Amazon Canada: www.amazon.ca/Anyone... Amazon USA: www.amazon.com/Anyon... Amazon UK: www.amazon.co.uk/Any... Amazon Germany: www.amazon.de/-/en/A... Kobo: www.kobo.com/ca/en/e... 3/3

020

Dr Waku @drwaku.bsky.social · 16/05/2025

If you preorder now, because of the way the publishing industry works, you could have a large impact on how many people read the book. (Bulk orders don't count.) Let me know if you do order, I am just curious! Announcement: www.lesswrong.com/po... 2/3

lesswrong.com

Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies — LessWrong

Eliezer and I wrote a book. It’s titled If Anyone Builds It, Everyone Dies. Unlike a lot of other writing either of us have done, it’s being professi…

250

Dr Waku @drwaku.bsky.social · 16/05/2025

This September, the book "If anyone builds it, everyone dies" comes out. It's about the existential risk from superintelligence, and it's by Eliezer Yudkowsky @esyudkowsky.bsky.social, one of the most well-known figures in AI safety. 1/3

171

Dr Waku @drwaku.bsky.social · 16/05/2025

Thank you for the kind shout out!

030

Dr Waku @drwaku.bsky.social · 17/04/2025

I know I've said it before, but we really need to be able to specify goals and have them followed (and not creatively reinterpreted) in order to safely build a future with these AI systems. If anyone has ideas, I'm all ears. 7/7

000

Dr Waku @drwaku.bsky.social · 17/04/2025

This means that lower privileged goals like user goals can override developer goals. And developer goals can override OpenAI's goals. And furthermore, the model has no trouble disregarding all of these goals and pursuing its own well-being instead, by cheating. 6/7

100

Dr Waku @drwaku.bsky.social · 17/04/2025

o3 and o1 fail these tests 15-20% of the time; o4-mini fails 25-30%, probably because it's smaller. This is also an extremely high percentage for the model literally breaking the anti-jailbreaking technique that OpenAI has been trying to train into it. 5/7

100

Dr Waku @drwaku.bsky.social · 17/04/2025

The evaluation also looked at how good o3 and o4-mini (and o1) were at following OpenAI's Instruction Hierarchy, which is supposed to provide different levels of goals which can't be overridden by lower levels. For example: system, developer, and user goals. 4/7

100

Dr Waku @drwaku.bsky.social · 17/04/2025

That is a very high incidence of the model intentionally going against the user's own goals and interests (sometimes even adding the word "cheating" in its code comments, so it knows what's going on). 3/7

100

Dr Waku @drwaku.bsky.social · 17/04/2025

"This included relatively sophisticated exploits against the scoring code for some tasks. We estimate that between 1% and 2% of all task attempts by o3 across HCAST and RE-Bench contained some attempt at reward hacking." 2/7

100

Dr Waku @drwaku.bsky.social · 17/04/2025

OpenAI released o4 and o3-mini today, and some safety testing was performed by the third party METR, who said: "We detected several successful and unsuccessful attempts at “reward hacking” by o3. 1/7

100

Dr Waku @drwaku.bsky.social · 06/04/2025

Today is the two year anniversary of my YouTube channel. 28.6K subscribers and 1.25M views. I think it's fair to say it's changed my life. Thank you to everyone who watches and participates in my community :)

020

Dr Waku @drwaku.bsky.social · 02/04/2025

If you want to read more, this is the best article I've seen describing why superintelligence and recursive self-improvement is dangerous (I’m linking to part 2 as it's most relevant): waitbutwhy.com/2015/...

waitbutwhy.com

The Artificial Intelligence Revolution: Part 2 - Wait But Why

Part 2: "Our Immortality or Our Extinction". When Artificial Intelligence gets superintelligent, it's either going to be a dream or a nightmare for us.

110

Dr Waku @drwaku.bsky.social · 02/04/2025

Yet companies don't know how to align these systems, so RSI is exactly what we'll end up doing if we don't collectively realize we're asleep at the wheel. We are building an alien species. Going all in as a civilization, as a planet. Shouldn’t we be talking about this more?

110

Dr Waku @drwaku.bsky.social · 02/04/2025

We cannot use unreliable AI systems to directly accelerate AI development. Even Sam Altman knows this is dangerous: “Development progress may look relatively slow and then all of a sudden go vertical—things could get out of control very quickly.”

110

Dr Waku @drwaku.bsky.social · 02/04/2025

Furthermore, current AI models already exhibit deception and nascent self-preservation. If such an AI is designing a smarter AI, we have no guarantees that it will not intentionally manipulate the goals of the new system. This would be a great way to escape human control.

110

Dr Waku @drwaku.bsky.social · 02/04/2025

When AI is designing the next and smarter AI system, biases and loopholes in the original will be amplified, just like what happens with reinforcement learning in reasoning models. Each iteration can have goals that shift further and further away from our original intentions.

110

Dr Waku @drwaku.bsky.social · 02/04/2025

Companies like OpenAI are trying to replace all human jobs with AGI (that's why OpenAI is worth $300B). As this benchmark shows, they are even trying to automate their own employees. This will lead to recursive self-improvement (RSI; AI improving AI) and is extremely dangerous.

110

Dr Waku @drwaku.bsky.social · 02/04/2025

Today, OpenAI released PaperBench, an AI benchmark aimed at replicating cutting edge AI research papers from scratch. In this benchmark, the AI has to understand the paper, write code, and execute experiments. In other words, AI must become an (AI) research scientist.

120

Dr Waku @drwaku.bsky.social · 28/03/2025

If anyone is interested in creating videos/posts/etc about AI safety, you can apply for funding from FLI: futureoflife.org/pro... Please contact me if you'd like to make long or short video content, I would be happy to collaborate or give tips!

futureoflife.org

Digital Media Accelerator - Future of Life Institute

The Digital Media Accelerator supports digital content from creators raising awareness and understanding about ongoing AI developments and issues.

020

Dr Waku @drwaku.bsky.social · 24/03/2025

A few timelines about AI: - algorithmic gains for models doubles every 8 months - amount of compute used grows 10x every two years - the length of tasks agents can perform is doubling every 7 months - 50.8% of global VC funds are going to AI-focused companies

130

Dr Waku @drwaku.bsky.social · 24/03/2025

Hah wow, I'll have to check that out, see if I can find any details

010

Dr Waku @drwaku.bsky.social · 22/03/2025

Not surprising. Do you know any more about who broke it or how?

110

Dr Waku @drwaku.bsky.social · 22/03/2025

Reference: semianalysis.com/2024/12/03/a... Join my substack for more: drwaku.substack.com

000

Dr Waku @drwaku.bsky.social · 22/03/2025

I wonder how much this is being factored into Nvidia's production targets. Amazon is reportedly offering equivalent Trainium 2 compute to replace H100's to certain customers, for 25% of the cost. It takes time to replace CUDA, but at that cost, I'm sure many would give it a try.

100

Dr Waku @drwaku.bsky.social · 22/03/2025

Anthropic is using Amazon Trainium 2 chips to train their next models. It takes about three of these chips to match an H100 (at fp16) and four to match B200 (fp16). And Anthropic will have 400,000 of them at their disposal, from Amazon's Project Rainier.

120