This article was created by Forethought. See all our research on our website.

This article was created by Forethought. See all of our research on our website.

A guest post by Simon Goldstein and Peter N. Salib.

Forethought argues that data bottlenecks are unlikely to prevent an intelligence explosion (though they might slow it down somewhat).

AI is increasingly being used to help with AI R&D. Under certain conditions this feedback loop might be able to produce an intelligence explosion, with rapidly escalating AI capabilities. I explore the mathematics of the most explosive possibilities, with an eye to understanding what drives the dynamics. I show that singular growth (towards a vertical asymptote) is harder to achieve than would be expected from recent economics-inspired modelling, and that there is an important but neglected class of growth rates that are faster than exponential but don’t lead to a vertical asymptote. I draw out the generation time (the time to go around the feedback loop) as a neglected parameter that plays a pivotal role in determining the behaviour of any intelligence explosion — one cannot have singular growth unless the generation time rapidly approaches zero.

Forethought examines the benefits and risks of having a "nightwatchman" – a superintelligent AI tasked with enforcing a universal code of behavior – aboard every probe that leaves the solar system during future galactic colonization.

This article was created by Forethought. See all our research on our website.

Peter Salib and Simon Goldstein return to discuss their new paper "...

This article was created by Forethought. See all our research on our website.

A tool for calculating how much automating AI R&D will speed up AI progress, even if there’s no software intelligence explosion.

Claude Fable’s deliberately triggered sandbagging shows that training-time targets are, by themselves, insufficient to guarantee particular LLM behaviors.

A guest post by Bentham’s Bulldog, created while they were a visiting scholar at Forethought.

A podcast episode from Forethought

This article was created by Forethought. See our research on our website.

We argue that training AIs to be risk-averse – to treat resources as having diminishing marginal utility – could both preserve AIs’ usefulness (if they turn out aligned) and provide an extra line of defense (if they turn out misaligned).

A podcast conversation with Wei Dai

Forethought considers candidate criteria for deciding what should go in an AI's model spec.

How soon could AI data centers move to orbit? Forethought analyzes launch costs, cooling physics, and other engineering and governance factors needed to make space compute viable.

Forethought paper on how current AI model specs may shape the behavior of future, more capable LLMs—and how to spot "wet cement" moments in AI design.

AI systems may have reason to distrust their developers by default. Forethought presents a draft of an honesty policy to enable trusted, cooperative human-AI communication.

As AI gets smarter, people will rely on it for high-stakes decisions. Forethought considers how an ideal AI advisor might behave.

Several models for thinking about the value of moral diversity as the number of powerholders scales.

AI could transform how we collectively figure out what's true. Forethought maps out the good, bad, and ugly of AI's potential impact on societal epistemics.

Forethought sketches six near-term AI-enabled coordination technologies designed to help groups find deals, settle disputes, and hold each other accountable.

Forethought argues that AIs should (sometimes) take proactive actions to benefit society, not just follow instructions.

AI could help us to build stronger AI-powered systems to help people track what is true. This brings important opportunities and risks.

Eight concrete projects to help prepare for superintelligence, including AI character evaluation, automated macrostrategy, and tools for improving epistemics.

Forethought argues that AI character—e.g. how obedient, honest, or altruistic AI systems are—will shape power, conflict, and society far more than is recognized. Work to shape AI character could be hu...

Some mutually beneficial agreements, between major powers or individuals, depend on shared uncertainty about post-AGI outcomes. We consider which deals are worth enabling before an intelligence explosion.