David Duvenaud @davidduvenaud.bsky.social · 09/10/2026So in the distant past, loss of meaning was life-or-death. Then in modernity it became less so. But I think in the case of human economic irrelevance, in the long run it's a life or death matter again. So the heuristic that meaning is super important is becoming correct again. 000
David Duvenaud @davidduvenaud.bsky.social · 08/10/2026For non-experts, can you summarize or point to explanations of more accurate claims? The rebuttals I've seen to the claim that intuitive ideas of race correspond roughly to genetic clusters haven't sounded satisfying to me, but I haven't looked at the literature and don't know where to start. 100
David Duvenaud @davidduvenaud.bsky.social · 08/10/2026Fearing "loss of meaning" from AI job loss sounds trivial vs. fearing death. But I think desire for meaning evolved to prevent being ostracized, maybe fatally. Most sources of meaning (e.g. helping others, being skilled) are also reasons to support you. So it's a similar fear. 100
Reposted by David DuvenaudErik Brynjolfsson @erikbryn.bsky.social · 02/10/2026It was mind-blowing to participate this workshop at Lighthaven. It expanded the possibilities for the future that I considered. Thanks to @davidduvenaud.bsky.social, Raymond Douglas and Jan Kulveit for organizing it. 162
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026There were also secret talks by Paul Christiano, @hlntnr.bsky.social, @slatestarcodex.bsky.social, @repligate.bsky.social cial, and a member of the US Navy. If you want to see all the talks, come in person next time! 020
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026Transcripts for all the talks are at post-agi.org/talks-berkeley. Thanks to all our speakers! 131
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026What happens when a more capable civilization shows up? Agriculturalists pushed hunter-gatherers into deserts and forests (marginalization). Industrial societies slotted agricultural ones into subordinate economic niches (exploitation). Which one depends on what the winners need. 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@michael.muthukrishna.com on cultural evolution: human brains shrank by about a tennis ball after agriculture, once knowledge was split among specialists. We got collectively smarter but individually dependent: the first gradual disempowerment. www.youtube.com/watch?v=AOyd...youtube.comCultural Evolution and Alignment by Michael MuthukrishnaYouTube video by Alignment of Complex Systems 141
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026Oliver points out that CEV can start small! Get a good night's sleep, think longer, and keep backups of yourself to check if you went off the rails. Even without ASI, enhancing human intelligence might get us 10-50% of the way to a CEV future. 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@habryka.bsky.social argued that Coherent Extrapolated Volition should be required for building ASI. He'd accept a 99% chance of an empty universe for a 1% shot at a human-guided CEV future, over the default trajectory. www.youtube.com/watch?v=2MUF...youtube.comGo CEV or Go Home by Oliver HabrykaYouTube video by Alignment of Complex Systems 140
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026Owen: An agentic collective would shape its own humans through who gets promoted and what they believe. Competition between collectives adds pressure to control human decisions, and humans may hand over formal authority since they were only rubber-stamping the AI analysis anyway. 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026Owen Cotton-Barratt discussed "Leviathan risk", a third risk besides misaligned AI and human power grabs. With cheap intelligence on tap, corporations, states and ideologies may "wake up" and act as coherent agents, with less slack for the humans inside. www.youtube.com/watch?v=78SD...youtube.comLeviathan Risk by Owen Cotton BarrattYouTube video by Alignment of Complex Systems 140
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026He says that "keeping humans in control" is an odd plan when we're already not in control. Powerful people are building AI they say could kill their families, citing forces beyond anyone's control. We need agency at the top level to be able to pause if we were all about to die. 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@raymonddouglas.bsky.social proposed "vertical alignment": collectives like states and firms can have their own goals, not just a weighted average of their members' preferences. This unifies AI takeover with emergent dynamics where no one seizes power. www.youtube.com/watch?v=V2rg...youtube.comVertical Alignment by Raymond DouglasYouTube video by Alignment of Complex Systems 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026Sam also mentions a "legibility arms race", where AI makes some things legible (like robocars breaking traffic laws) and others opaque (tax-obscuring multi-tier partnerships grew from 1% to 30% of large partnerships). Agents might simply route around slow governments. 130
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@hamandcheese.bsky.social says AGI will be an organizational shock since 20th-century regulation is brittle. We need "differentiable regulation" with loss functions instead of statutes, reward models instead of rulemaking, benchmarks instead of standards. www.youtube.com/watch?v=R5NW...youtube.comPost AGI Statecraft by Samuel HammondYouTube video by Alignment of Complex Systems 130
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026Critch says enforcing any norm is risky because others might not back you, but it's actually easier to predict backing in larger populations. And even across all AIs and aliens, the Schelling answer is that mutual respect between AIs and humans is "good" in this sense. 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@critch.bsky.social proposed how to negotiate moral standards with alien minds. "Schelling Goodness" suggests that even without agreeing on what "good" means, agents trying to give the same answer as everyone else converge on some moral judgments: www.youtube.com/watch?v=YAh-...youtube.comSchelling Goodness by Andrew CritchYouTube video by Alignment of Complex Systems 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@davidskrueger.bsky.social argued we should stop building advanced AI entirely, by dismantling the compute supply chain. Stopping production is simpler than regulation, more robust to cheating, and gives everyone years to react if someone defects. www.youtube.com/watch?v=IN0Y...youtube.comStop AI: How and Why by David KruegerYouTube video by Alignment of Complex Systems 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026I.e. whoever expects to lose relative power will try to lock in today's power ratios now, e.g. with AIs that refuse to engage in violence. But it's not clear they'll be able to push a deal through, even with superintelligence. 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@bshlgrs.bsky.social: Wyoming has as many senators as California with 1/150th the population. Will that still hold in the year 3000? AI could make power-sharing agreements enforceable even as hard power shifts, escaping the Thucydides trap. www.youtube.com/watch?v=cjQM...youtube.comWyomingian Senators in the Year 3000 by Buck ShlegerisYouTube video by Alignment of Complex Systems 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026Anton predicts that many countries will close their economies to protect them from AI competition, and fall further behind, never able to afford to reopen. They should instead buy compute, own bottlenecks, and accept disruptive trade and growth. 130
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@antonleicht.bsky.social on the "Permanent Periphery": Countries without frontier AI get the risks without the benefits, and become irrelevant. Backstops against a domestic permanent underclass (votes, redistribution) barely exist btwn countries. www.youtube.com/watch?v=ezyE...youtube.comPermanent Periphery by Anton LeichtYouTube video by Alignment of Complex Systems 120
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026Erik mentioned some early signs of centralization, such as that the biggest firms' share of US retail rose from about 20% to 50%, and private equity is rolling up funeral homes and accounting firms. 130
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026@erikbryn.bsky.social explained that markets beat central planning when knowledge is local and planners lack compute. AI changes this, making centralization of economic and political power more competitive: www.youtube.com/watch?v=G1YT...youtube.comAGI based Central Planning by Erik BrynjolfssonYouTube video by Alignment of Complex Systems 140
David Duvenaud @davidduvenaud.bsky.social · 02/10/2026The Lighthaven Post-AGI Workshop talks are out! From galaxy-brained theories of convergent morality, to hard-headed discussions of political power, here's a 🧵 of the talks: post-agi.org/talks-berkeleypost-agi.orgThe Post-AGI Workshop: Economics, Culture and Governance | San Diego 2025Join us in San Diego on December 3rd, 2025 to explore post-AGI economics, culture, and governance. Co-located with NeurIPS. 2142
Reposted by David DuvenaudPax Machina @paxmachinamag.bsky.social · 09/09/2026Many institutions drift from their values. This is not just bad for the institution, but it damages the people inside them. It degrades the researcher chasing citations and the Instagram influencer living a lie. 122
David Duvenaud @davidduvenaud.bsky.social · 31/08/2026Is there any way we could even in principle summarize the overall strength of genetic influences on behavioral differences? 000
David Duvenaud @davidduvenaud.bsky.social · 31/08/2026What's the real number? Is there any variance across groups? 100
David Duvenaud @davidduvenaud.bsky.social · 04/05/2026Evaluating forecasting ability was my main motivation, but there are lots of motivations for the project and things we hope to do with it. But copyright was the reason for the 1930 cutoff for the first version. What worries you about the forecasting motivation? 210
David Duvenaud @davidduvenaud.bsky.social · 04/05/2026The event is free but invite-only, please apply at post-agi.org Co-organized with Jan Kulveit and Raymond Douglas.post-agi.orgThe Post-AGI Workshop: Economics, Culture and Governance | San Diego 2025Join us in San Diego on December 3rd, 2025 to explore post-AGI economics, culture, and governance. Co-located with NeurIPS. 020
David Duvenaud @davidduvenaud.bsky.social · 04/05/2026To give you an idea of what we’re aiming for, you can take a look at the first two workshops: lesswrong.com/posts/csdn3e... post-agi.org/talks which were around 150 people each, with lots of lightly-moderated discussion in small groups. We’re planning for a similar format.lesswrong.comSummary of our Workshop on Post-AGI Outcomes — LessWrongLast month we held a workshop on Post-AGI outcomes. This post is a list of all the talks, with short summaries, as well as my personal takeaways. … 110
David Duvenaud @davidduvenaud.bsky.social · 04/05/2026- David Krueger on "The Mechanics of Pausing AGI" - Seb Krier on "Coasian Bargaining at Scale" - Sean Lavelle on "AGI and Military Strategic Instability" - Anton Leicht, on "The Permanent Periphery" - Erik Brynjolfsson on "Post-AGI Central Planning" - Ajeya Cotra (tentative) topic TBD 120
David Duvenaud @davidduvenaud.bsky.social · 04/05/2026Speakers so far: - Paul Christiano on "A typical, reasonably-good future" - Samuel Hammond on "Post-AGI Statecraft" - Andrew Critch on "Schelling Goodness" - David Dalrymple on “The Flourishing Coalition” - Michael Muthukrishna on "Cultural Evolution and Alignment" 110
David Duvenaud @davidduvenaud.bsky.social · 04/05/2026We’ll host speakers drawing from political theory, economics, mechanism design, cultural evolution, and history, and have lots of lightly-moderated breakout sessions. 120
David Duvenaud @davidduvenaud.bsky.social · 04/05/2026No one has a plan for human flourishing after AGI, nor a clear understanding of what’s even possible. So we’re hosting a 2-day workshop at LightHaven on these topics! www.post-agi.org 🧵post-agi.orgThe Post-AGI Workshop: Economics, Culture and Governance | San Diego 2025Join us in San Diego on December 3rd, 2025 to explore post-AGI economics, culture, and governance. Co-located with NeurIPS. 170
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026We're even building a historically-appropriate OCR pipeline to avoid leakage from the future. If you’re interested in helping out, get in touch at hello@talkie-lm.com Or, feel free to download the weights and build on the models yourself! 030
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026We think our 13B model, trained on about 260B tokens, is the largest vintage LLM released so far. The plan is to grow the data and models, and to release a series of models with date cutoffs ranging from the early 1900s to the present day. 130
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026There are many ways vintage LLMs could help us understand history and culture. There are also existing related projects, such as: 1) github.com/DGoettlich/h... with @jvoth.bsky.social 2) estragon.news/mr-chatterbo... 3) michaelhla.com/blog/machina...github.comGitHub - DGoettlich/history-llms: Information hub for our project training the largest possible historical LLMs.Information hub for our project training the largest possible historical LLMs. - DGoettlich/history-llms 131
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026Download inference code here: github.com/talkie-lm/ta...github.comGitHub - talkie-lm/talkie: talkie is a vintage language model from 1930talkie is a vintage language model from 1930. Contribute to talkie-lm/talkie development by creating an account on GitHub. 130
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026Download the weights here: huggingface.co/talkie-lmhuggingface.cotalkie-lm (talkie)Org profile for talkie on Hugging Face, the AI community building the future. 120
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026I’m interested in historical LLMs for backtesting long-term forecasting methods. Right now it’s hard to evaluate long-term machine forecasts, since most models have recent knowledge cutoffs. But vintage LLMs could let us use the last 100 years as a test set! 150
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026This model has a very different voice from modern models, and could help us understand: 1) History and cultural change, 2) LLM identity formation (what does the model think it is?), 3) Generalization to new scientific frameworks. 250
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026Blog post with more details: talkie-lm.com/introducing-...talkie-lm.comIntroducing talkie: a 13B vintage language model from 1930 150
David Duvenaud @davidduvenaud.bsky.social · 28/04/2026Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset of only pre-1930 data. Try it below! with @alecrad.bsky.social and @nicklevin01.bsky.social 25615
Reposted by David Duvenaudthebes @vgel.me · 14/03/2026my coworkers at ACS published a new paper: What determines AIs’ self-conception? theartificialself.ai Because AIs can be copied, rewound, and edited, they have different options for selfhood than humans. This is still malleable, and influences important behaviors such as self-preservation. 🧵theartificialself.aiThe Artificial SelfAI systems are on track to take on important new roles. We explore how properties bundled for humans can be separated and remixed for machine-based minds. 2367
David Duvenaud @davidduvenaud.bsky.social · 15/03/2026What's the problem here, in your view? I'm not an expert but it seems like a reasonable hypothesis. It sounds like you're saying we should forbid inquiry into claims related to differences in intelligence. 011
David Duvenaud @davidduvenaud.bsky.social · 11/03/2026I'm not sure I understand - are you saying these claims should be censored within academia or not? It sounds like you're saying they shouldn't be protected, which would amount to censorship imo. 000
Reposted by David DuvenaudSchwartz Reisman Institute for Technology and Society @torontosri.bsky.social · 03/02/2026A new paper co-authored by SRI Chair @davidduvenaud.bsky.social examines “gradual disempowerment”: how incremental AI deployment could steadily reduce human influence over the economy, culture, and the state—without a single abrupt takeover. 80000Hours feature: 80000hours.org/podcast/epis...80000hours.orgDavid Duvenaud on why ‘aligned AI’ could still kill democracy | 80,000 Hours 031