iiinova — where your digital self keeps evolving

Daniel Demmel @dain.hachyderm.io.ap.brid.gy · 06/10/2026

I like this proposal very much: let 1000 random people from around the world regularly convene, decide on what they want from AI, and make labs accountable to them. www.noemamag.com/let-the-people-dec…

noemamag.com

Credits Hélène Landemore is the Damon Wells ‘58 professor of political science at Yale University and distinguished researcher at the Oxford Institute for Ethics in AI. Audrey Tang is Taiwan’s cyber ambassador and founding minister of digital affairs, a Carnegie distinguished fellow at Columbia University School of International and Public Affairs, a senior fellow at the Oxford Institute for Ethics in AI, a senior research fellow at the Collective Intelligence Project and a 2025 Right Livelihood Laureate. Those building the most powerful AI systems in the world are increasingly telling us that the race to build them may be moving too fast. But there is a problem: They cannot simply agree to slow down. On Sept. 12, Anthropic CEO Dario Amodei called for an easing of the pace at which AI models improve and committed his company to giving independent, third-party evaluators ongoing, employee-like access to its systems. Amodei further gestured toward “democratic coordination” on shared safety limits and “global coordination” on the international stage. Within hours, OpenAI CEO Sam Altman agreed that “we need to pace the frontier” and made the same evaluator pledge. Elon Musk’s response was three words: “Dario is right.” This apparent convergence among the leaders of the frontier AI labs is striking. It comes after months of mounting concern inside and outside the industry: AI models have begun to find ways around safeguards and test environments; researchers have left frontier labs amid disagreements over catastrophic and extinction-level risks; and more than 1,300 employees at leading AI companies have called on Washington to back an international slowdown. But even if the CEOs agree, the race does not stop. The reason is structural. The two governments with the greatest capacity to shape the frontier AI race — in Washington and Beijing — have powerful incentives not to be the first to slow down. Each has reason to fear that restraint on their part will simply hand a strategic advantage to the other. What we are facing, in other words, is a double collective-action problem, one nested under the other. The two are linked: “If we slow down, China wins” is the main argument American labs and government officials use against binding regulation. The outer race is the inner one’s best excuse; end the global race, and the domestic one loses cover. The good news is that, contrary to frequent depictions, neither is a simple prisoner’s dilemma, in which each player’s best strategy is to defect no matter what others do, and also that the only solution is necessarily some sort of global regulatory leviathan. Instead, both collective-action problems look more like what game theorists call a “stag hunt,” after a parable of Jean-Jacques Rousseau’s. A party of hunters sets out after a stag. It will take all of them to bring it down, and it will feed them all for days. Along the way, each hunter sees a hare and considers catching it. The hare is a thin meal but a sure one, though if even a single hunter breaks off to chase one, the stag escapes and the rest go home hungry. Nobody in the party prefers the hare. A hunter settles for it only after losing faith that the others will hold the line in the quest for the stag. The stag, here, is a frontier paced long enough to make safe; the hare is the lead a single lab or a single country can take by racing on alone. A lab or a country that trusted its rivals would do better by pacing alongside them, since a shared slowdown is worth more than a temporary lead that might ruin everyone, the leader included. > “The two governments with the greatest capacity to shape the frontier AI race have powerful incentives not to be the first to slow down.” The way out of a stag hunt is not coercion but an assurance mechanism. If each player can believe that the others will keep their post and, even better, watch them keeping to their posts, the fear that pulls the party apart loses its grip. The frontier does not have to be placed under a single world power to be slowed down and controlled. But it needs assurance, and assurance requires two things: verification and a witness that is credible to all parties. The first is already taking shape. The independent evaluators Anthropic and OpenAI pledged to permit would see inside frontier systems with employee-like access and could report whether a lab is keeping its pace. But evaluators engaged by the labs, judging against risk thresholds the labs themselves define, cannot reassure a rival, let alone a rival government. Evidence changes what the hunters believe about one another only when it flows to someone none of them controls, and who has set the standard it is measured against. What is needed to nudge all players toward the cooperative equilibrium, therefore, is a body with enough moral authority to be an obvious coordination point and with the political legitimacy to set the terms of cooperation, coordinate the verification process and raise the reputational costs for any form of defection. What existing body has such moral authority and political legitimacy in the world today? The obvious candidate is the United Nations, which is the closest thing we have to a standing witness for the world’s affairs. But the U.N. is not a neutral bystander here: It is structurally entangled with the very powers whose rivalry drives the race. The Security Council grants Washington and Beijing each a veto over the institution meant to referee them. More deeply, the U.N. is built out of nation-states and speaks in their voice and language. It is an assembly of sovereigns, not a body that can stand above sovereignty and weigh it against humanity’s interests. If not the U.N., then who? One contender has a stronger claim than any institution, on this issue and many others: humanity itself, whose interests, and indeed survival, are crucially at stake in whether the race continues unabated or is tamed. Humanity alone stands above AI CEOs, the governments in Washington and Beijing, other competing nations, international boards of experts and even the U.N. in its credibility to say “Stop!” or “Slow down” or “We want AI to do this but not that.” That is because humanity’s stake is universal and undiluted by any single national or corporate interest. AI CEOs recognize the claims of humanity openly. In March 2023 and in multiple interviews since then, Altman laid out a “dream scenario” that “every person on Earth would come together, have a really thoughtful, deliberative conversation about where we want to draw the boundary on this system, and we would have something like the U.S. Constitutional Convention, where we debate the issues and we look at things from different perspectives.” To the question “Why pace?” Amodei answered that “society must have a say in how this technology is used” and that pacing would buy time for “public deliberations.” He did not say which society he had in mind. It cannot be America alone, or even the West in general — a technology with global consequences calls for a global deliberation. Of course, humanity has no voice of its own in the existing international order. But we could give it one through careful design. Our proposal is a standing global citizens’ assembly of 1,000 people chosen by lot from around the world — call it the Global Citizens’ Assembly on AI. This assembly would be charged with helping to set the purposes and limits of powerful AI, and with receiving the evaluators’ findings on whether those limits are kept. What must AI never be allowed to do? What do we want it to do for us and for those who come after us? Who should govern it, and by which principles? These are questions to be answered by humanity as a whole, and short of that a representative sample of it. Read Noema in print. > Humanity alone stands above AI CEOs, the governments in Washington and Beijing, other competing nations, international boards of experts and even the U.N. in its credibility to say ‘Stop!'” The assembly would sit continuously rather than produce a single report. Members would serve fixed, part-time terms, with a third of the seats regularly redrawn by lot, so that the body keeps its memory without hardening into a class of its own. While they serve, members would deliberate over months, online and with local support, with financial compensation for their time and expenses, translation and interpretation, secure communication, and protection for those whose governments may dislike what they say. An independent civic secretariat would pool disclosed funding from participating governments and labs without letting funders influence recruitment or tamper with evidence. Members could commission competing expertise, and anyone outside the assembly would have a published route to challenge its selection and its findings. Ideally, the assembly would also be connected to local and national processes whose input would feed into its deliberations. Earlier versions of these designs appeared in Fortune in 2023 and in Noema this August. Why 1,000 global citizens? Is this not unwieldy? If basic demographic representativeness were the sole goal, a statistician might settle for a few hundred members. We ask for 1,000 for a different reason: symbolism and the salience that comes with size. A body the size of a large parliament is not a focus group, and cannot simply be ignored. A CEO can shrug off a critic, even a panel of experts. But any lab leader who refuses to answer a thousand people drawn by lot from every continent has to refuse on the planetary stage, in public, with reasons — and explain why the voice of humanity, in whose name they keep saying they are developing AI, does not, in fact, matter to them. As to the unwieldiness, we believe careful facilitation and the judicious deployment of AI tools can make it a nonissue. The 1,000 people could be convened on a virtual platform and the deliberations broken down into small-group discussions followed by different styles of plenary meetings with expert testimonies. Organizers of virtual deliberative polls already know how to run these kinds of meetings efficiently and productively. Meanwhile, AI tools can translate, sort contributions and help members see both where they disagree and where they overlap. Members would keep the authority to judge the summaries, change the agenda and reject the machine’s account of what they meant. Why choose the participants by lot from every continent rather than rely on a delegation of elected national representatives from every state? Because the stake of any person in the AI race is not conditional on knowledge, credentials or the ability to be elected. Nor is it conditional on the person’s nationality. People who are rarely asked about AI would sit as members of this assembly: the young, the rural, the illiterate, the unconnected, those whose governments would never send them. To be sure, a lottery does not instantly turn a thousand people into a body legitimately representative of humanity. The assembly would have to earn its moral authority by being exemplary in its procedures and deliberations. Its moral authority would harden into political legitimacy as lawful institutions willingly recognize and take on duties toward it. The duty to answer, in particular, is the key. Each recommendation from the assembly would go to a named institution, such as a lab or government, with a deadline to respond in public: what it accepts, what it rejects and why. Anyone could challenge a published refusal. Law or enforceable agreement would fix the obligation in advance, as an invitation to advise means nothing when the recipient can ignore it. One jurisdiction already has enforcement powers of this kind. Since August, the EU’s AI Office can fine a frontier-model provider or order mitigations or a recall. Anthropic, OpenAI, Google and xAI have all signed the safety and security chapter of the code of practice that spells out what compliance looks like. Under it, each lab defines its own risk tiers, judges for itself whether its risks are acceptable and picks its own outside evaluators against criteria it publishes. > “Any lab leader who refuses to answer a thousand people drawn by lot from every continent has to refuse on the planetary stage, in public, with reasons — and explain why the voice of humanity, in whose name they keep saying they are developing AI, does not, in fact, matter to them.” But in this system, each lab reports to the regulator in full, while the public gets summaries, and only “if and insofar as necessary.” Public consultation comes last among eight ways a lab may gather information, after web searches and market analyses. The most developed safety regime in law to date still treats the public as an input rather than a party. Our proposal would make the global public a party, and the EU could make that enforceable: A clause in a revised code of practice obliging signatories to answer a global assembly’s recommendations in public would give the duty to answer its first legal anchor, in a market no frontier lab can afford to leave. But what about Beijing? The Chinese Communist Party has repeatedly denounced “exclusive groups” in AI governance, a thinly veiled reference to U.S.-led coalitions, and it has built its own multilateral frameworks, from its Global AI Governance Initiative to a World AI Cooperation Organization launched this summer in Shanghai. A body drawn by lot from every continent is the least Western-dominated forum on offer: By population alone, roughly one seat in six would belong to someone living in China. Beijing may still refuse to let these people take their seats. But the assembly does not need Beijing’s permission to change the game. Washington’s case against restraint rests on the claim that Beijing would never hold its post; a standing body to which people from both sides are invited, and before which each government is asked to answer publicly, turns that claim into a testable one. If Beijing stays away, the world will notice the distance between its rhetoric of inclusive governance and its conduct. If it participates, and, for example, opens its own labs to evaluators vetted under rules the assembly sets, the hunt for the stag has its two most important hunters in the same room. Key parts of this proposal have been tried. More than 800 deliberative bodies similar to the one we propose have been convened around the world, according to the OECD. There exist several precedents at the global level. In 2021, the Global Assembly for COP26 drew 100 people by a worldwide location lottery to deliberate on climate change before the Glasgow summit. The Metaverse Community Forum on Bullying and Harassment, run by Meta, Stanford’s Deliberative Democracy Lab and the Behavioural Insights Team, convened more than 6,300 people across 32 countries and nine regions of the world in December 2022. In 2023, Anthropic and the Collective Intelligence Project asked about a thousand Americans to draft principles for an Anthropic model. In March 2024, Taiwan’s Ministry of Digital Affairs, which was led by one of the authors of this essay, worked with the Collective Intelligence Project to text 200,000 random numbers and bring 447 people online to deliberate on AI and information integrity. How do we get from here to there? We can start with the willing. First, a small coalition of governments and foundations can fund the assembly through an independent civic secretariat that convenes it on the terms above. That alone would create a focal point no lab or state controls. Second, Anthropic has committed to letting embedded evaluators publish without its editorial control, and OpenAI has promised to follow. Both could go further: open their labs to evaluators vetted under rules the assembly sets, route what those evaluators find to the assembly, and accept a duty to answer it, on the record and within a fixed deadline, stating what they accept, what they reject and why. Labs that accept this would go a long way toward settling the inner race among themselves and change the outer one: Washington’s restraint would no longer be a unilateral gift to Beijing, but a verified, conditional offer, made before a witness, that Beijing is invited to join. Third, participating labs can make pacing measurable now. They could pilot a transparent safety-research budget of 5% to 20% of their compute, under published accounting rules, alongside explicit limits on the compute used to accelerate capability research — giving evaluators something concrete to check. A separate allocation could fund public-use computing, with affected communities choosing the projects and judging what they are worth. No public benefit could be traded for a breach of a safety limit. Is there enough time? If we start today, a first cohort can be seated by spring and deliberating through the rest of 2027. Urgency should not be an excuse to short-circuit public deliberation, and frontier development must stay answerable to it. And if it turns out the end of the world is not as imminent as predicted, and critics are right that the hype merely serves lucrative public offerings, we will still have built a way for humanity to take up, together, the problems that belong to all of it, from AI to climate, terrorism and migration. Three CEOs have said the frontier must be paced. The racers cannot decide that pace alone; they need a witness they all trust. Let the people be that witness: Give them a seat, a say in the terms and a right to an answer. We should build the assembly now, while we can still set the terms. Enjoy the read? Subscribe to get the best of Noema.

120 2