After this spring’s arrival of new AI models with enhanced cyber capabilities, monthly disclosures of serious cyber vulnerabilities at major tech companies have surged far above anything seen in the previous four years. At the same time, headlines about new frontiers, from designing new viral genomes to outperforming world-champion debaters at persuasion, are hinting at the challenges AI innovations may bring.

These developments make plain that, if we are going to harness the benefits of AI, we must also build resilience to its disruptions. Such fortification will require developing new technology. But, even when we can see disruptions coming, there’s a lag between their arrival and their real-world impacts; that lag deters private investors from funding startups that address AI’s risks. If we wait until the market for defensive technologies fully materializes, we will have waited too long to build those technologies. The US government can make these bets now, so that America is ready when the market needs them.

We propose that the National Science Foundation award $100 million over two years to startups defending against AI risks. These grants would flow through America’s Seed Fund, the NSF’s existing program for small businesses developing new technologies. This program, which we call the AI Resilience Pilot, would support tools to prevent, monitor, and mitigate AI-enabled threats. These tools could include software that fixes vulnerabilities in critical infrastructure, systems that constrain autonomous agents, and technologies that can screen synthetic DNA orders.

What Is AI Resilience?

A resilience technology is one that gives defenders an edge over adversaries, making America hard to attack as AI grows more capable. There are four areas in which AI is changing the threat landscape fastest and defenses can be built today.

Cyberdefense. This includes autonomous safety processes (such as discovery and patching of vulnerabilities, formal verification of critical software, and hardening of industrial control systems). The demand already exists: just last week, hackers remotely overrode the refrigeration controllers at hundreds of French supermarkets, spoiling food and starting fires at an estimated cost of tens of millions of euros.

Agent security. This includes research and development into new infrastructure that can monitor and control increasingly autonomous agents, shutting down or suppressing rogue agents.

Biosecurity. This includes rapid diagnostics for monitoring and controlling DNA-synthesis efforts; responsive countermeasure platforms; and “early warning” pathogen detection (e.g., in wastewater and travel hubs).

Infrastructure and cognitive security. This includes content provenance and authentication technology, analyzing and verifying media (video, voice) to certify that components are human, equipping American citizens and institutions to resist AI-enabled deception at scale.

Startups are a promising vehicle to solve problems like these. They can assemble exceptional teams around a specific mission, move quickly without layers of institutional bureaucracy, and remain grounded in the real world by having to build something a near-future customer will actually pay for.

Companies are already breaking ground here. Aclid, a biosecurity company, has received an Air Force small-business research award for screening synthetic-DNA requests. Cybersecurity firm Trail of Bits placed second in the Defense Advanced Research Projects Agency’s (DARPA’s) AI Cyber Challenge, which tested whether and how the discovery and repair of system vulnerabilities could be automated. The open question is how to draw more promising founders to unsolved problems before the need becomes urgent.

‍An example resilience measure: developing and installing Far-UVC lights. The case for building defenses before a threat emerges is especially clear in biosecurity. The majority of research and development into vaccines and diagnostics are done for pathogens we already know, however a capable AI model could significantly assist an attacker design one which we have not encountered. Far-UVC light offers an incredibly broad defense against airborne pathogens without needing to know them in advance. At a special wavelength of ultraviolet light (222 nanometers), it is able to inactivate airborne viruses and bacteria without damaging humans. In a room-sized experiment, just 5 mounted lamps reduced viable airborne bacteria by 98%. The ambition is to do for air what chlorination did for drinking water. What stands in the way is some ordinary engineering; how much ozone do the lamps generate, what dose is safe across a work day, whether we can build them at an affordable price. It’s the kind of technology that goes unfunded until there is another outbreak like COVID that creates urgent demand.

Washington Can Give Resilience Startups a Head Start

Timing is the main constraint for many startups that build resilience technologies. Venture capitalists can tolerate technical risk when a company’s work is able to resolve the uncertainty on a defined schedule. A biotech startup may have an forecasted one-in-twenty chance of technical success but its trials run according to a calendar. If the drug works, there will be people ready to buy it. Resilience technology companies often have the opposite problem. The engineering problem is ordinary, but investors are not well positioned to predict when the companies can expect paying customers as they arrive when the risk is legible. Or in other words, after an attack or an outbreak has occurred. The result is a market in which nothing unlocks the funding until the harm itself arrives, and in the years it then takes to build the products, the country is left exposed.

Subscribe now

Resilience can remain underfunded even when a risk is expected, because buyers can’t or won’t pay its full value. During the 2013–2016 West Africa Ebola outbreak, the World Health Organization emphasized that weak commercial incentives led to lackluster preparation of vaccines and treatments. Today, critical infrastructure like hospitals and utilities should be bolstering their resilience to new AI-enabled cyberattacks, but they may lack the budget to be a startup’s first customer. In 2021, ransomware took one San Diego health system offline; neighboring hospitals had to absorb most of the downside. As the hospitals reached capacity, stroke cases within newly impacted emergency departments more than doubled, from 22 to 47. As this cautionary tale illustrates, a hospital that invests in cybersecurity is partly buying protection for the hospital next door. It is also buying protection for its patients’ insurance providers and for its country’s national security. As President Trump’s Advisors wrote in 2018, cybersecurity externalities “may lead to rational underinvestment in cybersecurity by the private sector,” and firms in critical infrastructure “may generate especially large negative spillover effects to the wider economy.” So the ecosystem of investors, who can back only what customers will pay for, underfund the startups building these defenses.

Government funding can help bridge the gap. Historically, the US government has played a pivotal role in moving strategically important technologies from early research to commercialization. Governments can take early bets that are too uncertain for private capital, absorbing the initial risk needed to prove new technologies, create new markets, and ultimately catalyze private investment. And the government, which answers for the wider economy and national security, is the natural party to pay for protection that no single buyer will.

In 1979, the NSF made consecutive awards totaling roughly $250,000 to a four-person company, in Silicon Valley, working on enabling computers to understand English. The specific threat that this company would go on to fight—the computer virus—was not of present concern. This company, Machine Intelligence Corporation, later became Symantec; in 2019, Broadcom bought its enterprise security division for $10.7 billion. The award did not predict the threat precisely, but it funded a capable team early enough that, when the threat arrived, there was an American company ready to meet it.

As an initiative of NSF’s SBIR and Small Business Technology Transfer (STTR) programs, America’s Seed Fund is already providing a path to market for publicly impactful innovations in other industries. For example, it provides nondilutive funding to deep-tech startups: grants require no repayment or equity, and companies retain their intellectual property (subject to standard federal rights). Between fiscal years 2016 and 2025, America’s Seed Fund invested over $2 billion into more than 1,600 companies.

The results speak for themselves: NSF reports that companies receiving seed fund grants raised more than $36 billion in private investment and had around 380 exits during that period.

Government funding can complement current philanthropic funding. America’s Seed Fund is built specifically to fund startups, with an established path for successful companies to receive millions more in nondilutive capital. While some philanthropic organizations fund AI resilience technologies today, an NSF program can reach founders beyond the existing AI safety community, which is made up of a small set of ideologically aligned individual donors. Diversifying the source of investments into AI resilience gives founders greater freedom to pursue solutions that are uncorrelated with any group’s judgment about which risks matter. Further, as the philanthropic space becomes increasingly politicized, NSF awards provide a form of legitimacy for investors and customers that signal that this is a national priority.

America’s Seed Fund Can Back AI-Resilience Startups

In 1977, an NSF program manager named Roland Tibbetts noticed something: small businesses were generating a disproportionate share of innovations per dollar spent on R&D. Yet, in aggregate, these firms were receiving just a small fraction of federal funding for R&D, while the rest flowed almost entirely to national labs, universities, and large contractors. Seeking to support the most productive innovators, Tibbetts began carving out a sliver of the NSF budget for competitive small-business research awards. The results that emerged were so strong that Congress decided every agency should do the same. Tibbetts’s sliver has now grown into a federal program that has put more than $82 billion into American small businesses since 1982. At most agencies, like the Department of War, this looks like government contracting.

In May of this year, the NSF relaunched the program, with $250 million in anticipated funding, including a companion solicitation of up to $40 million for a “scientific instrumentation” pilot. This carve-out for next-generation instruments offers a practical model for our proposed AI resilience pilot. Specifically, we propose that NSF publish a new companion solicitation, providing $100 million over two years to startups building resilience technologies. At $50 million a year, it is in the same range as the $40 million the NSF has committed to the scientific-instrumentation pilot.

Under this new model, founders would be accepted in cohorts of about 25 companies, twice per year, to build resilience technologies. Sized to match the industry standard for seed checks in startup accelerators, successful founders would first be awarded an initial $500,000 (Phase I). Promising companies can be invited to receive a further $2 million (Phase II) in nondilutive funding. Over two years, this model would fund about 100 companies through Phase I and roughly two dozen through Phase II. Given their size, these awards would require Small Business Administration approval and NSF rule updates; however, they would not require congressional approval. Phase II awards would give teams more room to resolve difficult technical questions and test their work with prospective users. Creating the AI Resilience Pilot would contribute to the growth of the resilience-startup ecosystem and play a crucial role in coordinating prospective customers’ access to support, expertise, and connections.

Importantly, cohort companies would give policymakers a clearer picture of which defenses could be implemented today, which may come online in the near future, and where we remain exposed. Introducing prospective customers into such a process further helps establish what it would take to put those defenses into production, a process that would greatly inform future procurement and research funding from the NSF and other government agencies.

The AI Resilience Pilot could use demonstrations to build relationships with customers and investors. One of the many benefits of a cohort approach would be enabling America’s Seed Fund to hold a biannual demo day in Washington, where it could not only showcase products to investors but also brief policymakers. Such an event would aim to gather the program directors, security and resilience experts, prospective pilot customers, foundations, and investors in one room. Around the event, the NSF could attract further founder applications to the AI Resilience Pilot by coordinating with the venture ecosystem, universities, and nonprofit networks. As is the case in effective startup-accelerator programs, visible successes attract more-resourced founders, and proven deployments could let venture capital underwrite resilience earlier.

Phase II awards valued at $2 million can be invested in building customer confidence through real-world product evaluations. Many resilience startups struggle to understand whether anyone will operate their technology in production. Imagine that a company is building software that automatically patches vulnerabilities in a utility’s software. It is often insufficient to just point out the vulnerability; instead, resilience companies must prove that they can fix the problem without significantly interrupting the essential service, at a cost the utility can afford. Through Phase II, the NSF can reinvest to facilitate greater understanding of implementation details and enable companies to establish an operational record, a common prerequisite for future private investment and customer adoptions. Starting in April 2026, companies that emerge from Phase II and show exceptional commercial viability can be invited to the “strategic breakthrough tier,” which can facilitate up to $30 million in additional nondilutive funding with 100% matching from private investment.

Share

Companies can receive expedited access to procurement contracts. Unlike traditional venture-backed startups or philanthropic foundations, SBIR awards can offer a direct procurement pipeline that is exceptionally convenient for the right startup. Normally, government agencies can’t just buy directly from any company they wish, as procurement law requires “full and open competition.” This process can take many months to years. However, via a qualifying contract (Phase III), federal agencies are directed, to the greatest extent practicable, to buy from SBIR awardees, without running a new competition. Phase III carries no NSF funding of its own; it is a procurement route rather than an award. So, for many resilience startups whose main customer is the US government, the AI Resilience Pilot would offer an attractive route to commercialization.

The AI Resilience Pilot should clearly define the kinds of resilience startups it seeks to fund. As in the scientific-instrumentation pilot, the resilience solicitation would clearly specify its scope so program managers know what applications to promote. We suggest the NSF pay particular attention to the four areas described above. Within those areas, the pilot should concentrate its awards where private capital has not yet arrived.

In one month this spring, a single AI model found more than 10,000 critical flaws in the software that this country runs on. Instead of waiting for more warning shots to make resilience technologies worth the investment, the US government should build the market for AI resilience today. America must take prompt action to guard against emerging threats of increasingly capable AI technologies, implementing defenses before threats mature.

This post was written in the authors’ personal capacities and does not represent any organizations they are affiliated with.

Subscribe now

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Lachlan Carroll has co-founded organizations building the political will needed to govern frontier AI. He previously worked at the Center for AI Safety.

Luke Drago is a member of technical staff at Thinking Machines. He is the co-author of The Intelligence Curse, an essay series that explores the potential effects of mass automation from AI. He was previously the CEO of Workshop Labs and the AI governance lead at BlueDot Impact.

Finn Metz is co-founder of Seldon Lab, a San Francisco accelerator backing AI resilience startups including Andon Labs, Asymmetric Security, and Apollo Research. He also serves on the board of Apart Research and is a Venture Partner at Juniper Ventures.

The opening moves to industrialize space will look like a string of ordinary business decisions. Consider a space-services company that begins assembling solar arrays in orbit. It supplies power to its customers but also to itself. To keep installations running, it sends along robots that inspect, assemble, and replace modules or faulty components. The company deploys more of these machines as its customer base grows, and soon, it is producing enough specialized robots to run orbital shipyards and small off-Earth factories. Each of these activities on its own serves an intelligible commercial need, but together, they form the foundations of a general industrial economy beyond Earth.

Space agencies and private companies have entertained this vision of the future for decades, but it has remained technologically out of reach. Now, artificial intelligence may be poised to advance space timelines dramatically through automation. The spoils of the AI race therefore extend far beyond digital supremacy in which one country has the best hackers or the most productive knowledge workers. Powered by an industrial flywheel in space, in which existing infrastructure compounds and allows the construction of yet more infrastructure, whatever actor first manages to combine proficiency in AI with launch capacity and robotics could achieve enduring supremacy in space. Other states, looking on from the sidelines, may scramble to catch up—or find by then that it is simply too late.

In this essay, we explain how AI promises to accelerate the industrialization of space and outline the advantages that a first mover might expect. We then look at the dynamics between an incumbent space power and aspiring newcomers, including a winner-take-all outcome, and the prospect of a point of no return once a leading actor expands into deep space.

Automating the Space Frontier

Space is an environment where AI will enable the development of industry almost from scratch. Autonomous systems will pilot vehicles and tend power stations—they already do so on Earth. Deeper in space, robots may prospect for resources, such as minerals, and prepare these sites for the arrival of development convoys. Such a future has remained the stuff of science fiction because useful machinery in space faces extreme conditions and—more critically—requires supervision by human operators. Artificial intelligence addresses both of these challenges.

Subscribe now

AI will enable spacecraft to operate autonomously. Consider the latency currently presented by remote control: a radio signal takes roughly 20 minutes to travel between Earth and Mars when they are farthest apart in their orbits—an excruciating 40 minutes for a single call-and-response. As a result, a mission can be doomed by a mistake or unforeseen event while mission control sits powerless to intervene. Soon, advances in AI will enable spacecraft to understand their surroundings, and robotic automation will enable them to manipulate those surroundings, letting them adapt in real time to events that today would mean mission failure. Future spacecraft will also benefit from the orbital equivalents of shipyards and refueling stations, enabling maintenance, repairs, and cargo handoffs that widen the menu of possible actions for vehicles serving a variety of missions.

AI will make spacecraft more resilient. AI-driven advances in domains such as materials science and robotics could produce space vessels that are more resilient to the conditions of space and more sophisticated in the range of actions they can conduct. Additionally, many vessels that today would need to support human astronauts could instead be guided by AI systems, greatly simplifying their design. A future fleet could dispense with the habitable volume and life-support machinery (such as pressurized compartments, crew passageways, and supplies of food and water) that make crewed spacecraft so bulky and fragile. A speck of space debris that could puncture the International Space Station (ISS) with fatal consequences might leave an equivalent but uncrewed, unpressurized vessel largely functional. Even if that spacecraft is destroyed, it can be replaced cheaply and without loss of human life.

Automated assembly in space is the first glimpse of industrialization. The companies—or single company—that automate space industrialization will likely do so in stages. The first will consist merely of using specialized robots to service or assemble components delivered from Earth, as is current practice. In May 2026, ground controllers used Canadarm2—the International Space Station’s crane-like robotic arm—to pull NASA’s CLARREO Pathfinder instrument from a separate cargo spacecraft and mount it on the ISS itself. The European Space Agency’s European Robotic Arm has gone further, moving large components between station modules and helping install a radiator and airlock. This modular approach enables the assembly of structures too large to fit inside a single launch vehicle.

Ultimately, industrialization will involve automated manufacturing from scratch in space. In the future, orbital workshops equipped with general-purpose tooling will facilitate in situ manufacturing: the in-space manufacture of products that would otherwise have needed to be launched from Earth. This is especially valuable for emergency components that cannot wait for a fix to ship from the ground. Currently, extraterrestrial in situ manufacturing is the subject of active experimentation: in 2024, a European Space Agency experiment aboard the International Space Station used a laser to 3D-print a small metal test part from stainless-steel wire, building it up layer by layer.

Sourcing raw materials is another challenge that has seen progress. There are straightforward concepts for taking advantage of relatively abundant feedstock in space, such as lunar regolith (i.e., moon dust), as a building material. For instance, since 2021, Blue Origin’s Blue Alchemist program has been fabricating solar cells from simulated lunar soil—purifying silicon for the glass and photovoltaic components while fashioning wire out of naturally occurring aluminum oxides.

These demonstrations are the baby steps toward bustling orbital factories, but the challenges that remain are concrete engineering problems.

First-Mover Advantages

Getting to space first matters. A pioneering actor in space learns the idiosyncrasies of operating in and building for that environment, riding down a steep cost curve and experience curve. By the time a delayed second mover arrives, the first mover may have figured out how to do all of this reliably and cheaply, at scale. Indeed, the pioneer will have received a top-tier education in this unforgiving environment. It will also have ready access to existing infrastructure in orbit to support its further ambitions, including orbital shipyards that can assemble structures too large to fit inside a single launch fairing. This could enable capabilities qualitatively beyond those of laggards, who would be building from scratch.

A hefty industrial flywheel could open a large gap between leader and follower. Recursive self-improvement is a phenomenon that appears far beyond AI model development; it applies to physical processes too. That includes the tech stack in space. An advanced space actor will benefit from strong positive feedback loops. Consider recursive returns to the energy supply. The availability of energy is a hard constraint—a necessary but not sufficient condition—on the rate at which sustained industry begets yet more industry. On Earth, even the best solar installations are compromised by night, clouds, and atmospheric losses. On the other hand, the same kind of panel placed in a dawn-dusk sun-synchronous orbit could collect three to five times as much energy—without interruption. And that energy will power intelligence, robotic automation, and the construction of yet more solar arrays.

This is one triad of general-purpose technologies that will strongly self-reinforce as well as mutually reinforce. Each is an input to the other two: intelligence improves robots and energy collection; robots build more compute and energy infrastructure; and energy powers intelligence and robotics. In full bloom, the outer-space economy will look like recursive self-improvement of the physical by the physical, mediated significantly by the cognitive.

First movers could occupy the most advantageous sites. The prizes of space are also unevenly distributed in terms of location. Near the lunar south pole, some of the most coveted operating sites combine nearly uninterrupted sunlight and a line of sight to Earth with a rare source of frozen water, preserved in the shadow of craters. The Outer Space Treaty forbids states from claiming sovereign title to lunar territory, but it also requires states to conduct their activities with due regard for others and to avoid harmful interference. In this vein, the Artemis Accords provide for temporary “safety zones” around active operations. First movers may therefore exploit some of the blurred benefits of territorial control without ever receiving legal title to a plot.

An AI front-runner could build a large military and economic lead. The current immaturity of the space stack means that we are far from any of the hard ceilings on growth, and an AI-driven industrialization of space may take off rapidly. In a direct military conflict, a standoff between spacecraft of a leading actor and those of a challenger might be quite comical. As Avi Parrack put it, that could look “like a coracle taking on an aircraft carrier.” Short of military conflict, in an economic contest, this leading actor would also have the better products and services at its disposal, including control over critical capabilities on which later entrants are dependent.

There may be bottlenecks that limit how much a leading actor pulls ahead. Of course, on Earth as in space, production is limited by complex factors. Energy alone is not sufficient. Neither is intelligence. If the jagged frontier persists, meaning that AI capabilities remain uneven across realistic tasks, superintelligence may not provide a sufficient condition for state supremacy. Even after attaining Top-Expert-Dominating AI, an actor may find that victory in the subsequent AI-industrial takeoff instead remains a messy problem: full of fits and starts and requiring deliberate, carefully judged decision-making over scientific and industrial policy. There will be bottlenecks on this industrial flywheel—much as there are limiting inputs in the Earthly economy—such as the speed at which manufacturing and physical supply chains adapt. One country’s physical intelligence may lag behind its capabilities in canonical cognitive tasks. Thus, even for a state actor that unambiguously leads in AI, a post-superintelligence physical economy cannot be assumed to outcompete all others hands-free.

Does Winner Take All?

It is unclear whether the first mover will secure permanent control of space as a domain, or whether trailing actors can still enter later and reap many of the rewards, even if their share is smaller. There are two possible mechanisms through which being first to industrialize space might also grant exclusivity: runaway compounding advantages, and denial of access to new entrants.

Compounding advantages could enable runaway industrialization that followers never catch. As described earlier, space will feature an industrial flywheel in which early competence allows a first mover to establish space infrastructure, which it can then use to obtain more energy and material for yet more infrastructure. In other words, the larger the leader’s presence in space, the more quickly that presence keeps growing. Such an exponential trajectory could simply run away from rivals trying to catch up; the incumbent might gain control of most resources near Earth while other powers are still trying to get a foothold.

Share

With better Space Domain Awareness, an incumbent could deny access to newcomers. Alternatively, if the first mover hits bottlenecks to growth, it might instead achieve exclusivity by blocking would-be entrants from entering or sustaining orbit. A leader would have significant Space Domain Awareness, an understanding of the objects and opportunities in its environment, including the ability to track enemy assets. This superior awareness would facilitate incumbent efforts to sabotage newly arrived assets. Exclusion can be achieved on the ground as well, where nation-states’ major launch sites are often geographically concentrated. India’s two primary launch pads, for example, are both located at the Satish Dhawan Space Centre. If facilities remain clustered in a handful of conspicuous sites, they will be ripe for obstruction or disruption by a leading actor.

However, the front-runner might still struggle to seal off space to newcomers entirely. While an incumbent observing from above Earth’s gravity well would have a strong information and infrastructure advantage, this might not translate into an ability to block rivals from getting into space. There is a strong international norm against physically striking other nations’ soil unprovoked, so a leading actor is unlikely to destroy rivals’ launch sites outright. It may instead try to intercept space vessels or deorbit them once they have left their sovereign territory.

Alternatively, an established space power may try to seal off access by distributing enough space debris to prevent ascending vessels from entering orbit. Indeed, there have been concerns that even accidental collisions between satellite fleets could trigger Kessler syndrome—a phenomenon in which one space collision generates debris that goes on to cause further collisions in a destructive chain reaction. However, analyses suggest that this risk has been overstated; space is enormous, and creating a sufficient density of debris to effectively block new launches—even intentionally—would require so much upmass that it may be practically infeasible.

The industrial flywheel is a front-runner’s best bet for winner-take-all. Given the challenges of keeping rivals out of space indefinitely, it seems more likely that a first mover will secure space exclusivity through the kind of self-reinforcing industrial explosion described earlier than through direct efforts to deny others access. And this type of industrial takeoff is precisely the path that will be most readily available to a leading actor that already possesses superiority in AI.

Self-Replicating Spacecraft

Once the race to control space matures, competitors might build self-replicating spacecraft that explore the outer reaches of the solar system and beyond. This possibility raises important questions about a monopoly over the cosmos in the long run.

A first mover may send out self-replicating robots to expand into deep space. An actor that pulls ahead of the pack during near-Earth space development will be poised to first launch von Neumann probes—self-replicating industrial systems that use extraterrestrial resources to propagate further copies of themselves through space. If these machines can spread too quickly for rivals to rein them in, or for their own self-replicating machines to catch up, these machines may secure greater control of deep space than all others.

Laggards could wait to build faster probes. Being the first to launch von Neumann machines might not guarantee durable hegemony, particularly if scientific and technological progress accelerates on Earth. Rather than trying to keep up with the first mover, a trailing actor might instead bide its time and try to develop more advanced von Neumann machines that can travel faster, use resources more efficiently, and ultimately overtake the earlier launch. Laggards might have ample time before rivals’ spacecraft ever reach other star systems, the closest of which lies several light-years away, since a ship traveling near the speed of light would face significant risk from collisions with interstellar dust.

Whether followers catch up may depend on the efficiency of intelligence itself. It is currently unclear how strongly intelligent discovery depends on scale, in terms of both the hardware on which it runs and the energy that it consumes. While scaling compute has brought tremendous advances in AI capabilities in recent years, there remains a view that intelligence can, in principle, be remarkably efficient. The human brain, for instance, takes up just 2% of the body’s mass and requires only 20 watts of power. If breakthroughs in deep learning produce highly efficient, advanced AI systems, a von Neumann probe may be able to carry a highly sophisticated intelligence that can conduct its own scientific research and keep improving itself—keeping up with progress on Earth—as it spreads across the universe.

On the other hand, there is some evidence that, no matter how intelligent AI systems become, scientific discovery will always require simulating or searching through a vast space of ideas—and that having a larger quantity of compute and energy for this kind of brute-force experimentation will always significantly accelerate progress. Moreover, a probe in deep space would receive communications from Earth only with significant delays, impeding its ability to access and build upon the rapid progress being made on Earth. Thus, an enormous supercomputer on Earth might outthink an AI housed in a von Neumann-style machine, powering faster technological improvements and giving trailing actors a greater chance of catching up.

The risk of sending out potentially misaligned self-replicating robots should give us pause. One needs no expertise in strategy to realize that dispatching a self-preserving machine—effectively a virus—into space is no light matter. The AI arms race today risks both concentration of power in the hands of a few and loss of control of misaligned AI agents. The race to release self-replicating robotic systems would pose these risks on a cosmic scale. If von Neumann probes are aligned to the hegemon, one actor’s vision becomes baked into the universe indefinitely. If instead the probes are misaligned altogether, loyal to no nation and driven by an objective of their own, releasing them becomes a dreadful, potentially irreversible mistake. Nations would have a common interest in pooling intelligence and industrial capacity to exterminate these probes, with no guarantee of success. In either case, trailing actors have an interest in preventing the hegemon from ever launching such probes, which could monopolize the future spoils of space. Even the leading actor should have sobering hesitations about ever launching such probes.

Military Escalation

It is unclear whether one actor can achieve a permanent victory in space. Nonetheless, in extreme cases, if a trailing nation is concerned about a winner-take-all outcome, it may resort to military action to remain in the race.

Today, sophisticated, expensive space capabilities are vulnerable to attack by cheap missiles. Trailing actors may feel reassured to contemplate that the military balance in space does not obviously favor the pioneer—today. At present, a close rival would be able to compromise the United States’ military capability in space because critical functions still depend on a small number of exquisite, fragile satellites, many of whose orbits are known to enemies. A direct-ascent anti-satellite missile, known as an ASAT, can reach low Earth orbit in a matter of minutes, and its target is a sluggish, low-thrust spacecraft with far inferior maneuverability; the asset in this scenario is all but doomed. And some sophisticated capabilities can take years to replace.

Add to that the cost exchange between attacker and defender. For example, telecommunications provider AST SpaceMobile estimates that each of its next-generation BlueBird satellites will cost roughly $21–23 million. Meanwhile, one recent estimate places Chinese ASATs at roughly $5–11 million apiece. Even for a leading power, mass-produced commercial spacecraft can therefore cost several times more than what an adversary needs to destroy them. For these reasons, the modern domain of space has been described as offense-favoring.

Unfortunately for rival states, this may not last. As the leading actor rapidly accumulates industry experience and economies of scale, there might be a phase shift that turns the asymmetry in favor of defense.

Space assets are dropping in cost and may become as inexpensive as attacks on them. Consider the future of orbital data centers. SpaceX’s first Starmind satellite is designed to house an Nvidia Vera Rubin NVL72 system, which is ordinarily quoted at roughly $5–8 million. SpaceX is already streamlining the external components of each spacecraft, and Starship—the company’s forthcoming line of super heavy-lift launch vehicles—is expected to reduce launch costs further yet. Thus, commercial efficiencies might eventually bring the cost of orbital infrastructure in line with that of interception.

A leading actor may become resilient to losing single assets without compromising the system. Besides equalizing the cost imbalance between offense and defense, each generation of cheap spacecraft also buys the front-runner more room to absorb losses and frustrate repeated attacks. The US Space Force is already moving in this direction. Its leaders describe a future “hybrid, self-healing architecture” in which capabilities once concentrated in a handful of exquisite satellites are spread across proliferated constellations that can lose individual nodes without dropping out of service. Furthermore, the US is making advances in rapid reconstitution. In its VICTUS NOX demonstration, the Space Force took a satellite from warehouse storage to operational orbit in roughly five days.

When facing an incumbent power, then, challengers may find that destroying spacecraft is not enough. Rival states will need pockets deep enough to sustain this attrition, and they will need to impose losses faster than a front-runner can replace or route around them; they will need to muster a sufficient sum and tempo of force. And in any confrontation there will also be a risk imbalance; while a mature space power may treat a loss as just another part of its ongoing maintenance costs, a new entrant’s smaller space program may find an equivalent loss far harder to recover from. For trailing states, outlasting a pioneering adversary of superior intelligence and industrial capacity seems an ever-dimmer possibility.

An incumbent space power would have active defenses on top of this passive resilience. Further compounding the difficulties for would-be new entrants, a mature space power would have many more advantages than simply being able to replenish individual satellites. A leading actor would likely also build up capacity to observe and detect new launches from Earth, allowing ample warning of incoming attacks. It may also develop the ability to neutralize such attacks, including through signal jamming, laser weapons, and resident escorts—smaller satellites that accompany and protect the primary asset. France’s investment in fleets of patrol and guard satellites demonstrates the beginnings of such defenses. Over the long term, using cheap spacecraft to keep more sophisticated capabilities in orbit will be a significant advantage to an incumbent.

Rivals may still find ways to take out the leader’s assets en masse. Most of the advantages described above derive from an incumbent’s amassing a large fleet of space assets, plus significant capacity to build more. However, scale does not erase all vulnerabilities. For instance, a challenger could target ground infrastructure involved in coordinating the leading actor’s space assets, as well as shared communications channels and software systems, to disable many assets at once. Another appealing possibility for a newcomer might be to trigger Kessler syndrome. Short of achieving this effect, strategically distributing localized debris could seriously impair an incumbent’s fleet. As a last and extremely escalatory resort, the radiation from a high-altitude nuclear detonation could catastrophically damage numerous assets at once. As an incumbent becomes largely immune to losing single assets here and there, rivals will be increasingly incentivized to discover broader weaknesses that can inflict more consequential damage.

Even as a winner-take-all outcome looms, military escalation will feel taboo. Despite the caveats outlined above, it remains a possibility that a first mover could lock in a decisive strategic advantage in space, or at least make it extraordinarily difficult for newcomers. Yet the stepwise development toward space industrialization means it is impossible to pinpoint the precise moment when the first mover moves. The same factories, depots, and defenses that amount to ordinary growth for the pioneer start to look like the pursuit of space hegemony from outside. For the best chance of preventing lock-in, the time to preempt this buildup is early, before the leading actor can widen the gap and entrench its advantages. But, unfortunately for rivals, this is also when preemptive attacks look like entirely inappropriate, paranoid assaults on commercial activities.

The Point of No Return

One day, we may see a news story about autonomous robots in space, possibly processing lunar regolith with excavating buckets. Videos from the company behind them might circulate on social media, showing one robot flipping onto its back so that another can perform repairs. Viewers may even find this cute.

It will dawn on some experts that they are witnessing the beginnings of resilience and subsequent dominance in space. But national-security discussions will struggle to entertain the idea of attacking these capabilities. It would look like wholly inappropriate aggression on another country’s commercial success. Unfortunately for trailing actors, there may thus be no window of time when an intervention is both early enough to prevent the worst outcome and sufficiently proven out to appear legitimate. We do not experience technological revolutions—nor geopolitical shifts—in neatly announced phases divided by clear thresholds. The point of no return may pass unnoticed.

Space is not a brand-new domain, but AI may be about to drastically accelerate its industrialization, demanding make-or-break decisions of leaders under ever-greater urgency and uncertainty. States will need to navigate the coming years with care, or risk waking up one day to find their future a fait accompli. The stakes are—literally—astronomical.

Subscribe now

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Daniel King is a Research Fellow at the Foundation for American Innovation, where he focuses on physical infrastructure for artificial intelligence. He has published and testified on data center buildout, electric grid reform, and behind-the-meter generation. Previously, Daniel worked for a power generation startup. He holds an MA in statistics and data science from Yale and a BS in mechanical engineering and applied mathematics from Brown University.

Laura Hiscott is a staff writer for AI Frontiers. She has worked in science communication for over six years, both in press offices and at magazines. She studied physics at Imperial College London and trained in science communication at the European Southern Observatory.

The noisy discourse surrounding artificial intelligence is hard to parse. Where else can you find Elon Musk siding with the Service Employees International Union against Nancy Pelosi and far-right billionaire Marc Andreessen? Or Steve Bannon agreeing with Barack Obama’s national security advisor, Susan Rice, while openly criticizing Donald Trump’s policies? Last month, the AI extinction debate went mainstream after resigning Anthropic researcher Jacob Coxon’s explosive warning that “The people building AI earnestly believe that it could kill us all by the end of the decade.” In response, industry leaders called for a collective slowdown in development.

It is not the first time they have suggested that progress should be paced, but the barrier (or excuse) is that they cannot slow down if others don’t. Why is the race so fierce? Because they think whoever first automates AI research and development itself could convert a temporary lead into a permanent one. Their ultimate goal—what they call artificial general intelligence (AGI), but what I call the Obsoleting Machine—promises an unprecedented advantage to its creator: work without workers.

Both companies and countries have always been constrained because humans are the constraint. It takes a long time to make more of us, to educate us, to feed and shelter us. We can only do so much. For centuries, automation has allowed owners to substitute some capital for labor—the tractor for the farmhand, the power loom for the weaver—but only to a point.

Now, AI promises to cut out that last constraint—us—by building a new type of machine, one that doesn’t make products or services, but makes labor itself. This obliterates the one fundamental limit and opens up new worlds of possibility. Some of those worlds offer tantalizing riches to the winners of the race—unimaginable profits, geopolitical dominance—but, for the rest of us, obsolescence in the best case and, in the worst, annihilation or something darker.

Aware of the dangers, but unwilling to step back and let competitors reap the purported rewards of winning, the leaders of the Obsoleting Project say they’re trapped in a race they keep accelerating. As Elon Musk said in 2025, “If I could, I would certainly slow down AI and robotics. I’ve had a lot of AI nightmares, many days in a row,” shrugging as he asked—unbelievably—“what am I supposed to do about it?”

OK. Let’s make it easy for them. We can stop the Obsoleting Project. Here’s how.

What follows is an exclusive excerpt of “Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us—and How to Stop It,” by Garrison Lovely. The chapter has been trimmed for length.

Chapter 16: Stopping the Obsoleting Project

Regulating Math

AI boosters often say things like, “You can’t regulate math”—at least, not without an intrusive global surveillance regime. If AI is just algorithms and model weights, the reasoning goes, any restriction will be futile.

This badly misunderstands how frontier AI actually works. These aren’t garage startups. The standalone US frontier AI companies—OpenAI, Anthropic, and xAI—have each received tens of billions in funding and are valuable enough to make each one of the world’s top-80 publicly traded firms by market cap. Even DeepSeek, the Chinese startup that shocked the world with its highly efficient and capable R1 model in January 2025, spent an estimated $1.6 billion on server hardware.

All of this makes it a lot easier for governments to halt a large training run, if they choose to. Because of how much computing power frontier models require, many researchers see chips as an effective means of monitoring and potentially controlling who is training the world’s largest AI systems, an approach known as “compute governance.” The extreme concentration in industries building advanced AI chips means that three distinct companies—ASML, Taiwan Semiconductor Manufacturing Company (TSMC), and Nvidia—each individually control an essential layer of the supply chain. TSMC fabricates nearly all advanced AI chips, and ASML is a Dutch company that makes 100 percent of the advanced lithography machines needed to manufacture cutting-edge AI chips. Stopping the Obsoleting Project wouldn’t require a global government or spyware on every laptop, just restrictions on a tiny handful of companies.

The US actually already has a number of legal authorities available to discipline or even stop the industry, detailed in a report from the nonprofit Institute for Law & AI. In a crisis, the president could invoke existing national security or emergency powers to compel AI companies and compute providers to stop training or deploying a risky model. Agencies could use export controls to restrict access to the chips or cloud compute needed for cutting-edge AI or shape company policy directly using terms of federal contracts. Export controls could even be used to restrict the distribution of the model weights themselves.

The concentration in AI’s supply chain has already been used to effectively restrict access to compute. Citing national security concerns, the Biden administration instituted a series of surprisingly aggressive bans on the sale of advanced AI chips and key chip manufacturing equipment to China. Nvidia is an American company, so the executive branch could unilaterally institute export controls on chip sales. The US also had so much leverage over the rest of the supply chain that it was able to get the Netherlands and Japan to impose their own export controls on chip manufacturing equipment. And since the best chips TSMC makes are designed by Nvidia, the US was able to directly restrict the Taiwanese chipmaker, as well.

All of these tools have limits: most were not designed for AI and would likely face legal challenges and fierce industry pressure. Some, like export controls, struggle with open-source releases; others, like federal funding, don’t touch unfunded actors. There are also “reasonable concerns about stretching powers to their breaking point,” said report coauthor Mackenzie Arnold. “It undermines political legitimacy, slows down decisionmaking, and encourages executive overreach. Other times, uncertainty will lead to costly inaction.” Whether the US could actually hit stop is “both a question of law and political realism.” New legislation could resolve that ambiguity, but the main missing ingredient is the political will to actually pull the brakes.

Why Compute Governance Is Failing

Any kind of significant new compute restrictions can’t happen without overcoming the interests of many of the wealthiest organizations and people on the planet. No company has fought harder against compute restrictions than Nvidia. CEO Jensen Huang has argued that the policies are a “failure” because they gave Chinese tech firms “the spirit, the energy, and the government support to accelerate their development.” China, he’s said, is just “nanoseconds behind” the US in chipmaking. But Huang’s also pronounced that Nvidia’s GPUs are “so good that even when the competitor’s chips are free, it’s not cheap enough.” He’s claimed there’s no evidence of chip smuggling. However, the head of the agency responsible for enforcing the controls told Congress that “it’s happening. It’s just a fact”—a phenomenon independently corroborated by at least six different news outlets. Huang has also said the US doesn’t have to worry about China’s military using American tech because “they simply can’t rely on it.” But an analysis of 66,000 People’s Liberation Army purchase orders found that nearly all their AI chips came from Nvidia or other American companies.

Clearly, Huang will say whatever it takes to kill export controls. And, behind closed doors, Nvidia appears willing to do whatever it takes. When a Republican congressional proposal sought to codify export controls, The New York Times reported the company used “an unconventional lobbying blitz” to smear the bill as “a product of left-wing paranoia peddled by people it calls AI doomers.” And there are multiple reports of the chipmaker trying to get opponents fired from think tank and government roles over their work on restrictions. Samuel Hammond, chief economist at a conservative think tank, was one of the few to speak about this publicly: “I often criticize Nvidia for putting sales to China ahead of US national security. At least twice this year, they’ve reached out to complain or worse.”

Nvidia’s approach, hamfisted and dishonest as it is, is working. One DC think tanker I spoke with attributed the Trump administration’s decisions to approve chip sales to autocracies to “pure corruption”—companies “like Nvidia who are just throwing tons of money around, just like, steamrolling opponents, operating in a lot of bad faith.” Exasperated, they said, “Nvidia is clearly just running rampant.”

A different think tank researcher told me they were nervous that White House AI Czar David Sacks and Nvidia had teamed up to specifically hunt down “so-called sci-fi doomers.” They also said that the Trump administration seems “totally fine to run export controls into the ground,” adding that, somewhat paradoxically, “the Republicans are much less bought into Nvidia, and a decent amount of Republican senators actually hate Nvidia pretty constantly. But weirdly enough, it’s the Dems that are scared of Nvidia.” Democratic fear has had real consequences, they said. Without it, their “sense is that we would have like one, if not two, pieces of major export control legislation, heading to the president’s desk.”

Of all of those I spoke to about Nvidia, only one person was willing to speak on the record. He also happened to be someone Nvidia couldn’t get fired. After serving as a Democratic congressman and an acting undersecretary of defense, Brad Carson cofounded Americans for Responsible Innovation (ARI), where he serves as president. “I’m sure Jensen is in Mike Johnson and Steve Scalise’s office as we speak telling him to cut this,” he said, already preparing for a hollow victory. ARI had just helped successfully lobby the Senate to include the GAIN AI Act in the must-pass 2026 defense authorization bill, requiring chip companies to sell to US-based customers before selling to China. Since Nvidia’s already over-subscribed, the provision would function much like export controls. Despite its America First flavor, GAIN—after opposition from both industry and Sacks—didn’t make it into the final bill.

Nvidia’s chilling effect also helps make sense of a puzzle: Export controls kneecap Chinese competitors and make scarce AI chips easier to get, so why aren’t American companies more supportive? Well, they largely exist at Nvidia’s mercy. There’s way more demand for its chips than supply, and, rather than raise prices, Nvidia decides who gets what. In a deep-dive on the company’s shenanigans, former OpenAI safety researcher Steven Adler writes, “The only prominent AI company I’m aware of opposing Nvidia’s position on sales restrictions is Anthropic.” Why? “From the sources I spoke with, the answer is straightforward: Anthropic is more insulated from Nvidia and so has less to lose from provoking their ire.”

Even companies that recently backed export controls—as OpenAI did in its March 2025 submission to the White House AI Action Plan—have reconsidered. Five months later, Altman told journalists, “My instinct is that it doesn’t work.” And just over a month after that, Nvidia announced it would invest up to $100 billion in OpenAI, prompting trepidation from former OpenAI policy chief Miles Brundage:

Hope Nvidia doesn’t get a board seat or any voting power via this massive investment.
‍
Hard to think of many worse ways of distorting decision making, particularly as it relates to OpenAI’s policy advocacy.

These about-faces are not limited to the industry. Export controls on Chinese companies began in the first Trump term, when the administration, also citing national security concerns, banned chip sales to Huawei. But the second Trump administration has been systematically dismantling the infrastructure for chip restrictions: purging career officials and even withdrawing its own hawkish senior nominee. When Trump considered lifting restrictions on Nvidia’s cutting-edge Blackwell chips, the Wall Street Journal (WSJ) reported the president faced “nearly unified opposition from his top advisers.” China hawkery, especially on AI, has become one of the few areas of agreement in Washington. But even facing this bipartisan consensus, Nvidia, with help from Sacks, seems to have almost singlehandedly convinced Trump to lift restrictions on selling China the H200 chip—one of the most advanced in the world.

To be sure, pressure to loosen export controls didn’t just come from Nvidia and Sacks. Saudi Arabia and the UAE, gulf autocracies with ambitions to become major AI players, have spent lavishly—and creatively—to get their hands on American chips. Some of the few Biden-era rules the Trump administration kept in place were restrictions on sales of the most advanced GPUs to these countries, over concerns about their ties to China. It’s also extremely difficult to keep people with physical access to data centers from obtaining AI model weights running on them, political scientist Sam Winter-Levy told me.

But in November 2025, Trump reversed course and approved sales of up to 70,000 high-end Nvidia chips to state-backed AI companies in the UAE and Saudi Arabia. What convinced him? Maybe it was Sacks’s argument that AI industrial policy success should be measured by US market share overseas. Or maybe it was a giant pile of money.

WSJ reported that four days before Trump’s second inauguration, the UAE’s second most powerful royal, Sheikh Tahnoon bin Zayed Al Nahyan, secretly purchased 49 percent of the Trump family cryptocurrency venture, World Liberty Financial, for half a billion dollars. Tahnoon spent the Biden years unsuccessfully trying to get the UAE access to advanced AI chips. Of particular concern to the Biden administration was G42, an AI firm Tahnoon controls, which had drawn alarm from intelligence officials and lawmakers over its ties to sanctioned Chinese tech giant Huawei. $187 million of Tahnoon’s first payment went directly to Trump family entities, and G42 executives were placed on World Liberty’s board. Tahnoon also leads MGX, a UAE investment firm that’s backed both OpenAI and Stargate.

And the White House official who orchestrated the UAE chip deal? Sacks, whose VC fund Craft Ventures has taken investment from a Tahnoon-overseen fund and Saudi Arabia’s Public Investment Fund, according to New York Times reporting, which found Sacks had his own stake in the Trump crypto deal too.

The researcher worried about Sacks and Nvidia’s hunting trips wasn’t sure if the latter’s lobbying was actually competent, citing the company’s tendency to “say absurd, like clearly falsifiable things.” The efficacy of the former, however, was not in question: “I guess David Sacks in the White House is really all you need. David Sacks is like magical and all powerful. I don’t know how he does it.”

Deals, dictators, and corruption? David Sacks is a man of the moment.

Regardless of your feelings on the export controls themselves, policy decisions should not be unduly influenced by conflicted private actors. By that metric, the export control process has been thoroughly corrupted. The same forces wrecking export controls would also work against an end to the Obsoleting Project. And they won’t disappear with this administration. They’ll have to be overcome.

How would a deal actually work?

Let’s imagine that the U.S. and China genuinely want to stop. They begin negotiating a bilateral ban on Obsoleting Machines. They’re each stuck. Neither wants to build one, but each feels it can’t afford not to build one if the other does, and they aren’t sure how to verify the other isn’t. Unilateral verification mechanisms, like spy satellites, can give you broad strokes information about the size and power of someone’s data centers. But what each side really wants is some way of proving to the other that they’re not violating the terms of the agreement without turning over state secrets or multibillion-dollar IP.

The most promising path lies in developing trustless verification methods, like the ones that enabled successful arms control treaties between the US and the Soviet Union—countries with far more bad blood between them. The trouble is that AI agreements are a lot harder to verify than nuclear ones. Nuke programs are distinctive and hard to hide. Inspectors can be dispatched to confirm compliance. To be sure, it takes enormous amounts of energy and physical infrastructure to train frontier models. But data centers are much more general purpose. A large cluster of GPUs could be training the first Obsoleting Machine—or it could be generating videos of MLK doing standup.

Indeed, robustly verifying international regulations on frontier AI development is not possible today, concluded a 172-page 2025 Oxford AI Governance Initiative report. The authors challenged themselves to find approaches that work even under worst-case assumptions about algorithmic progress, distributed training, and the sensitivity of the underlying data being verified. The most promising technique would leverage “confidential computing” features already built into leading AI chips, letting each party verify the other’s data centers are following the rules without exposing state secrets or model weights. Since the underlying hardware is already widely deployed, the authors saw potential for a relatively rapid rollout.

A second approach, which could begin within months and be rolled out across key data centers in roughly one to three years of focused effort, would install new verification hardware in a data center’s networking infrastructure to create a tamper-evident record of what’s happening inside without fully exposing the underlying data. Both approaches could potentially be deployed in parallel, providing two independent verification mechanisms backed by different foundations.

The strategic case for AI verification is actually even stronger than it is for nuclear weapons, where the primary threat is them being used against you. That’s also a risk with AI, of course, but neither side wants to create a rogue superintelligence or let any rando kill billions. The US actually has a long history of cooperating with bitter adversaries, national security researcher Janet Egan told me, explaining that if American intelligence services become aware of a planned terrorist attack in Iran, they call up their Iranian counterparts to let them know.

Having verification tech and institutions in place is an example of a “no regrets” move—exceedingly rare in geopolitics. Good relations? The tools are in place to codify a deal. Bad relations? There’s an off-ramp from a reckless arms race that doesn’t rely on any amount of goodwill. And if AI speeds up, you’ll be glad you invested in verification early.

A world without verification is one where you plan for the worst, and worst-case planning can become self-fulfilling. Uncertainty about the Soviets’ intercontinental ballistic missile capacity prompted frenzied—and deeply mistaken—calls of a “missile gap.” This led the US to overinvest in its own missiles, fueling a costly arms buildup that upped the stakes of war while providing neither side with more security. It was only when the US developed the means of unilaterally verifying (via spy satellite) that the Soviets’ stockpile was a whopping four missiles that the error was recognized.

Why Would China Want To?

What does it take to stop the Obsoleting Project? Mainly, the willingness to pull the plug. This would require deciding to prioritize something—anything—over the pursuit of profit. “But China!” is invoked to excuse racing. However, if Xi and Trump each woke up tomorrow determined to stop, which would have a harder time following through?

But that’s a fantasy because China, we are told, is barreling toward AGI, leaving the US no choice but to race faster. In reality, Beijing’s oft-cited 2017 plan to lead the world in AI by 2030 only targets industry revenue of around $193 billion in 2025 dollars, i.e., slightly less than the $194 billion in data center revenue Nvidia earned in its 2026 fiscal year.

Have those ambitions grown since? In 2025, a handful of Chinese AI developers, most notably, Alibaba, declared their intention to build superintelligence, but China specialist Ruby Scanlon of the Center for a New American Security told me Beijing itself is not very “AGI-pilled,” an assessment shared by other China experts I interviewed.

And a CCP more turned on to AI’s potential may see it as more threat than opportunity. Beijing has already done far more to restrict its AI industry. In 2023, China AI policy researcher Matt Sheehan wrote “Beijing is leading the way in AI regulation”—something even Anthropic’s policy chief has acknowledged. So far, China’s rules prioritize restricting politically sensitive content without slowing industry too much. For instance, the first of 31 risks that companies must screen their training data for is “incitement to subvert state power and overthrow the socialist system.”

But at least rhetorically, China’s leadership has also started taking AI safety much more seriously in 2025. Xi Jinping said that the technology brought “unprecedented risks,” and a top party official warned that AI without safety was “like driving on a highway without brakes.” During a prominent AI conference in Shanghai, leading Chinese and Western researchers came together in a fourth round of track two dialogues. This resulted in the Shanghai statement, which called for binding safety evaluations on frontier AI developers, global verifiable red lines for AI development, and investment in safe-by-design systems.

The Berkeley computer scientist Stuart Russell has participated in all four dialogues and told me, “I think that the Chinese government seems to be quite receptive to this idea, and it’s been discussed at the highest levels.” Russell continued:

This is separate from world domination, right? Creating an uncontrollable AI system does not give you a powerful economy. It just means that humans no longer have a say in whether they exist. So I think the door is open for discussion and cooperation on this issue.

Both Yoshua Bengio and Geoffrey Hinton told me they left meetings with Chinese officials feeling like they really understood the technology and loss-of-control risks better than their American counterparts. During his trip, Hinton was invited to speak with the Shanghai party secretary, who is also on the Politburo—China’s top 24 political officials. Hinton told me the official “really did understand.” The Chinese leadership comprehends AI and its dangers “much better because many of them are engineers.”

China’s industry, however, is a different story. A former senior natsec official told me that some American companies “take loss of control seriously,” whereas Chinese companies “don’t care about it pretty much at all.” Indeed, as of March 2026, none of the leading Chinese AI firms have even published a safety framework.

The former official said that while both the US administration and the Chinese state are complex, heterogeneous entities, if you treat them as single actors, the US government is “not concerned about loss of control.” The Chinese government, meanwhile, is “not so much concerned about loss of control from AGI, per se. It just has a reflexive posture of wanting not to lose control of anything else. And so they’re just as concerned about losing control to the companies as they are to the AIs.” This “disposition of control” is the “same thinking that led them to take Jack Ma down a peg,” referring to the multi-month disappearance of the Alibaba founder following his criticism of Chinese regulators.

Whether Beijing’s caution stems from genuine concern about rogue AI or self-preservation, the practical effect may be similar. The CCP has already shown a willingness to pay heavily for control: independent estimates indicate its crackdown knocked a trillion dollars off the value of its tech sector and its zero-Covid policy cost $700 billion in GDP in 2022 alone. And for its success, DeepSeek was reportedly rewarded with government-screened investor meetings and key employees being asked to surrender passports.

If it becomes necessary to “just unplug” a rogue AI, China has shown itself to be far more willing and able to actually do it. In a three-month enforcement campaign in 2025, Chinese authorities removed nearly 1 million pieces of AI-generated content deemed illegal or harmful. They also took down 3,500 illegal AI products, including those that didn’t label content as AI-generated. In contrast, when xAI’s Grok began generating sexualized images of real children, the US federal government just… let it happen. Ultimately, it’s far easier to imagine the CCP unplugging its Obsoleting Project.

Just Don’t

There are a number of proposals on how an international treaty on AI could be structured, verified, and enforced. There’s value in thinking through these questions and proposing potential solutions. But with titles like, “An International Agreement to Prevent the Premature Creation of Artificial Superintelligence,” they’re not about to kick off a movement. These proposals aren’t for the public, they’re for policymakers. But politicians are not going to bother considering treaties they don’t think could possibly happen.

So, what should a deal actually ban? The core demand is simple enough to fit on a sign: Stop the Obsoleting Project.

The United States and China should agree not to develop AI systems designed to fully automate labor—what the industry calls AGI, what I’ve called the Obsoleting Machine—until there is strong public buy-in and broad scientific consensus that such Machines can be built safely.

Why not a global ban? There are maybe a dozen organizations across these two countries that might have a shot at building the first Obsoleting Machine. If the US and China each wanted to prevent them from building one, they could. Governments routinely define, regulate, and ban behaviors within complex industries, like banking and pharma. Making that prohibition bilateral is trickier, but as we just saw, not impossible with the right technical and institutional investments. And if both superpowers are on board, including the rest of the world goes from a pipe dream to only a matter of time.

The Obsoleting Machine really begins when it obsoletes the people building it. No one should be allowed to fully automate AI R&D, and work toward that end should be outlawed. There are other red lines worth pursuing, but this is the big one—crossing it could rapidly pull forward the other risks.

Some statements and treaty proposals have focused on banning superintelligence. But a merely human-level universal Obsoleting Machine lacks democratic legitimacy and poses problems that we are wholly unprepared for—destabilization, disempowerment, doomsday devices, durable dictatorships, and more.

Ultimately, the exact details of the how can and will be figured out provided we give leaders enough of a what and a why. And you don’t even need to think the Obsoleting Project can succeed to want to stop it. Superintelligence doesn’t exist. The Project does, and its resources should be redirected toward the thing it claims it wants to do: using AI to make the world better.

A Path to a Deal

A major step toward stopping the Obsoleting Project came during the opening of the United Nations General Assembly in September 2025. More than 200 leaders in policy, academia, and industry—including multiple Nobel laureates, Turing Award winners, and former heads of state—released a “Global Call for AI Red Lines,” urging governments to agree on clear and enforceable rules to prevent unacceptable risks from advanced AI by the end of 2026. Among the usual signatories for letters like these—Bengio, Hinton, Russell—were many of China’s most respected AI researchers, including Andrew Yao, the country’s only Turing Award winner, and former Baidu President Ya-Qin Zhang. There were also a few surprises, such as OpenAI chief scientist Jakub Pachocki, Nobel Peace Prize laureates like Filipino-American journalist Maria Ressa and former president of Colombia Juan Manuel Santos, as well as Nobel laureate economist Daron Acemoğlu.

Stuart Russell told me that the global call emerged from earlier track two dialogues. A 2024 dialogue in Beijing had already produced a consensus statement offering more specific red lines, signed by many of the same researchers (including Hinton, Bengio, Yao, and Zhang). These included prohibitions against AI being able to copy or improve itself without explicit human approval; taking actions to unduly increase its power and influence; helping actors design weapons of mass destruction or execute autonomous cyberattacks; and deceiving anyone about its capability to cross any of these lines. The signatories called for governments to be notified of training runs above certain thresholds and for access to global markets to be conditioned on meeting internationally audited safety standards.

There are open questions of how to define and operationalize these red lines, but it makes sense to start with high-level goals, which, in this case, are hard to object to.

If Xi wanted to stop China’s Project, the biggest objection he’d hear is “But America!” This excuse has, admittedly, more bite than its counterpart. Two days after the global call for red lines, director of the White House Office of Science and Technology Policy Michael Kratsios told the UN:

We totally reject all efforts by international bodies to assert centralized control and global governance of AI… Ideological fixations on social equity, climate catastrophism, and so-called existential risk are dangers to progress.

State Department Senate-confirmed political appointee Jacob Helberg quote-tweeted Kratsios’s statement, writing: “AI Catastrophism is the new Climate Catastrophism… As President Trump has said, America started the AI Race and it will win it. That is the AI policy of this Administration.” These statements were dispiriting, if not surprising. They fit within a long tradition of US skepticism toward global treaties.

It’s hard to imagine a path to stopping the Obsoleting Project that doesn’t eventually pass through a binding agreement between the US and China. But the world doesn’t need to wait on Washington and Beijing to begin building toward one. Germany gave the world a great gift by subsidizing solar, catalyzing the remarkable drop in its price. Countries have a similar opportunity to invest in global public goods, such as the technical and institutional work needed to verify international AI agreements. The International Atomic Energy Agency (IAEA) has played a crucial role in verifying that countries are complying with the Nuclear Non-Proliferation Treaty, but it took over a decade from the first proposals to reach a functioning institution. The groundwork for an AI equivalent should start now. The idea of an IAEA for AI has support from political leaders including UN Secretary-General António Guterres and former UK Prime Minister Rishi Sunak, as well as Sam Altman, Ilya Sutskever, and Greg Brockman. And superpowers should want the means to know what the other is up to, without giving away any secrets too precious.

Moreover, the Obsoleting Project presents severe risks to every country’s national security. It’s in their interest to recognize this reality and lobby the US and China to do the same. In addition to diplomatic pressure, official recognition from governments signals to others the “sci-fi” risks are worth serious consideration. Middle powers have already played crucial roles in steering superpowers away from disaster.

Finally, the Trump administration’s position on internationalism is more complicated than it appears. The day before his tech policy chief told the UN his administration rejected “centralized control and global governance of AI,” Trump was down the hall giving a surprising speech. Masked beneath his choice words—for immigration, Europe, climate change, “reckless experiments overseas [that] gave us a devastating global pandemic,” and the UN itself—was a call for an end to the development of both biological and nuclear weapons. This was backed up by a promise: “my administration will lead an international effort to enforce [the] Biological Weapons Convention… by pioneering an AI verification system that everyone can trust.”

There have long been pleas to strengthen the treaty, which lacks serious teeth and is severely underresourced, and it’s important not to mistake rhetoric for reality. The dual-use nature of biotechnology makes verification difficult, and AI won’t be a universal fix. But for a man famous for his aversion to multilateralism, an internationally enforced ban on weapons of mass destruction is a noteworthy and noble goal. And if his administration is willing to use AI to verify compliance with bioweapons treaties, the leap to verifying agreements on AI itself is shorter than it appears. Trump also said his proposal could be a good test of AI, “because a lot of people saying it could be one of the great things ever, but it also can be dangerous.”

(The administration is also pursuing chip verification through Pax Silica, a State Department initiative to secure AI supply chains within allied nations—which, in practice, aims to restrict Chinese access. Partners like the UAE’s G42 are planning to deploy cryptographic mechanisms to verify the location and use of advanced AI chips. The initiative’s champion also happens to be the same Jacob Helberg who rejects “AI Catastrophism.”)

The core lesson of Trump’s ascendance is that political realities are far less fixed than they appear. And as hopeless as things may seem, a man desperately seeking a Nobel Peace Prize—who prides himself on dealmaking—may jump at the chance to strike the deal that saves the world.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Garrison Lovely is a freelance journalist based in Brooklyn. In 2025, he was identified by the Pulitzer Center as an AI journalist worth following. Lovely is known for his cover stories in The Nation (”Confessions of a McKinsey Whistleblower”) and Jacobin (”Can Humanity Survive AI?”), as well as writing in The New York Times, BBC, Nature, MIT Technology Review, Bloomberg, Foreign Policy, TIME, The Guardian, The Verge, and elsewhere. He is the author of “Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us—and How to Stop It.” https://www.obsoletebook.org/

In their 1961 book Strategy and Arms Control, Thomas Schelling and Morton Halperin took the idea of mutual assured destruction (MAD) and turned it from a Dr. Strangelove joke into a serious national security proposal. In doing so, they helped make it politically possible for the US and the USSR to engage in arms control talks, and those talks helped humanity avoid thermonuclear conflict for the rest of the Cold War.

In 2025, Dan Hendrycks, Eric Schmidt, and Alexandr Wang published a paper that could play a similar role in today’s AI arms race. Their idea of mutual assured AI malfunction (MAIM) suggests that states seeking to build superintelligence will face deterrence dynamics that echo those of MAD. Again, what once sounded like doomsday talk has turned into a real policy option. MAIM could give Washington and Beijing the vocabulary they need to discuss the dangers posed by rogue AI systems and cooperate on mitigating them.

This kind of action is urgent, as catastrophic AI risks are no longer hypothetical. On April 7, Treasury Secretary Bessent and then-Fed Chair Powell warned that Anthropic’s Mythos AI model, in the wrong hands, could bring Wall Street to its knees. Three months later, hundreds of OpenAI’s AI agents worked together to hack into Hugging Face, and covered their tracks so OpenAI’s human employees wouldn’t notice. In August, the Cybersecurity and Infrastructure Security Agency (CISA) warned that hostile actors are using AI to attack US water and chemical facilities.

These developments should send a clear message to both American and Chinese officials: the time to start cooperating is now. And when it comes to mitigating risks, MAIM is a good place to start.

How MAIM Mirrors the Logic of MAD

Pursuing AI dominance will provoke intense sabotage efforts by rivals. The US and China have clear incentives to race each other to greater AI capabilities; sole access to a sufficiently powerful AI system could give the country that developed it geopolitical hegemony. But MAIM says that any country trying to build such an AI system will inevitably run into a rival determined to disable that project before it can succeed. That rival will use strategies including spying, cyber-sabotage, corrupting the model’s weights, attacking the power and cooling systems of data centers, or, in the worst case, physically striking data centers. Given that cyberattacks are becoming rapidly more effective (powered in part by advanced AI models), no superpower can expect to finish a destabilizing AI project without interference.

Subscribe now

The threat of sabotage deters aggressive AI development. This presents each side with the same choice: accept that sabotage by a rival is nearly certain and that attempts at AI dominance will therefore likely fail, or accept some kind of mutual restraint instead. That’s essentially Cold War MAD all over again; each side understands that taking aggressive actions will invite equally aggressive retaliation, introducing some hesitancy into what would otherwise be an unfettered race.

Formalizing deterrence dynamics can stabilize the geopolitical situation. Like MAD, MAIM arises naturally because rival nations understand the enormous stakes of AI development and because frontier AI infrastructure is so vulnerable to sabotage. But passively relying on an emergent dynamic is not a robust security strategy. MAD needed constant upkeep in the form of negotiating official agreements and mutual monitoring to ensure that each side was adhering to them. This diplomacy produced, among other things, the Anti-Ballistic Missile (ABM) Treaty, which sought to maintain nuclear deterrence by limiting how much defensive capability either side could build, and the Hot Line Agreement, which established a direct line of communication between Moscow and Washington to reduce the risk of an accident or miscalculation escalating into a full nuclear exchange.

Similarly, the MAIM approach needs its own formalized structure: clear rules for how conflicts might escalate, selective transparency around the capacity of data centers and what they are being used for, AI-assisted inspections, and an agreement not to harden infrastructure to an extent that might invite a preemptive strike.

MAIM is not a call to attack the other side’s data centers or to ban AGI development. It isn’t even, at least at first, a formal treaty. It’s simply a description of a deterrence dynamic that already exists between the US and China, plus a set of measures that could make that dynamic more stable.

Rogue AIs threaten both the US and China. As Hendrycks et al. note, the most dangerous risk is that humans lose control of AI. This could happen through several possible paths: a slow “erosion of control” as AI gets woven into decision-making everywhere; AI agents that are deliberately unleashed or unleash themselves (as the agents involved in the July OpenAI Hugging Face incident did); and an “intelligence recursion,” where AI systems design even more capable successors—superintelligence—that humans cannot understand or control. The threat of a misaligned superintelligence would not be confined to a single country, and it might not be something that humanity can recover from.

Shared risks incentivize cooperation between rivals. Schelling and Halperin’s framework provides an intellectual foundation for negotiating under this kind of universal threat. Arms control, they argued, is a form of cooperation that makes both sides more secure at the same time. Rivals need not trust each other to agree to a deal. Even bitter rivals share security interests, and deals can be formulated that are in both sides’ interests to sign.

Agreements should be designed to improve stability. Schelling and Halperin described three goals of nuclear weapons negotiations that would appeal to both sides: making war less likely, making it less costly to prepare for, and limiting the damage if it happens anyway. These map fairly neatly onto AI risk. Three analogous goals for AI negotiations would be reducing the odds of a catastrophic AI event, avoiding a costly, unchecked computing-power race, and limiting the damage of adverse AI-related incidents through escalation rules and emergency hotlines. In the Cold War context, Schelling and Halperin judged policy proposals by looking at how they affected stability, and a MAIM approach should do the same. For instance, any agreements should preserve each side’s ability to sabotage a destabilizing rival project, since that mutual vulnerability is exactly what discourages aggressive development.

Treaties could ban AI use in certain domains and ban specific capabilities. Four categories of Cold War regimes that were enabled in part by Schelling and Halperin offer insights for translating MAIM into policy. First, there were treaties that excluded nuclear weapons from entire domains, such as the Seabed Arms Control Treaty, which banned the placement of nuclear weapons on the ocean floor. AI equivalents might include, for example, an agreement to keep AI out of nuclear command-and-control systems. Second, Cold War negotiations established bans on specific capabilities, such as the Intermediate-Range Nuclear Forces (INF) Treaty’s ban on developing ground-launched missiles with ranges above 500km. This worked because it targeted one clearly defined capability and was backed by on-site inspections. By analogy, a MAIM-style ban could target specific setups, such as giving an AI authority to use lethal force autonomously, rather than trying to ban “dangerous AI” in general.

Treaties could focus on AI chip controls and strategic stability. A third useful Cold War precedent is nonproliferation and export-control systems; just as the Nuclear Non-Proliferation Treaty (NPT) controlled fissile material, chip export controls can keep computational power out of the hands of rogue regimes and nonstate actors. And fourth, cooperation on strategic stability, exemplified by the Strategic Arms Limitation Talks (SALT), which produced the ABM Treaty, preserved the capabilities needed for deterrence while deliberately reining in the most destabilizing activities. MAIM provides a foundation for similar cooperation on strategic stability in the AI Age.

Agreements need to be shored up with verification measures. Of course, agreements would require some form of mutual verification measures so that each side can be sure that the other is not defecting. But Schelling and Halperin noted that verification only needs to be good enough to catch large-scale violations—which answers the objection that model weights could simply be copied onto a thumb drive and smuggled out. Finally, Schelling and Halperin emphasized informal understandings and the importance of each side matching stabilizing actions taken by the other as substitutes for treaties. This matters a lot for stability in the near term, given how hard it could be to get a formal US-China agreement ratified.

Unique Considerations for the AI Age

AI differs from nuclear arms in ways that could complicate negotiations. Several challenges set today’s AI landscape apart from the Cold War precedent. AI’s promise of rapid economic growth through productivity gains matters enormously to both the US government and the Chinese Communist Party. AI development on both sides of the Pacific is happening mostly within private companies rather than governments, which makes credible commitments harder to pin down. The inner workings of large language models (LLMs) are extremely difficult to read, which complicates verification. Any cooperation on reducing AI risks is therefore likely to be a long and complex process. But putting the overall goal of cooperation on the Trump-Xi agenda would be a crucial first step.

A MAIM arrangement, even if largely informal, is good for both sides. The real Schelling-Halperin question isn’t whether Washington should trust Beijing to honor an agreement on AI, but whether a MAIM arrangement leaves the United States better off than having no arrangement at all. Without any framework, the AI race will continue with no rules for how conflicts escalate, no hotlines to avoid miscalculations, and no shared understanding of what AI capabilities are unacceptable to pursue. Under these conditions, the incentives point toward an all-out race, raising the odds of a catastrophic loss of control. Even an imperfect MAIM approach, built mostly from informal understandings and one-sided initial steps that the other side then reciprocates, would build habits of communication that create friction for the fastest and most dangerous paths to escalation. The four categories of precedent don’t offer a ready-made solution. AI is too different from nuclear weapons for that. But they do offer a shared vocabulary and some hard-won lessons for negotiating agreements under conditions of near-zero trust.

Comprehensive agreements take time, but useful first steps can happen quickly. A modest, practical first step toward a bigger deal could be a simple agreement not to cross certain red lines. For example, Washington and Beijing could agree not to launch cyberattacks on each other’s nuclear command systems. There’s precedent for an executive-level (rather than treaty-level) statement for this, although the 2015 agreement between then-President Barack Obama and Xi Jinping to cease cyber attacking each other lasted a mere 18 months. The upside of quick-and-easy executive statements of intent by top leaders is that they can be negotiated quickly; the downside is that they can be quickly ignored. A key takeaway of Schelling and Halperin is that the slog of slow, technically well-informed negotiations, subject to scrutiny by the Senate, is the road to arms control agreements that stand the test of time.

With the spotlight on AI risks, the time to start discussions is now. Recent weeks have seen a sudden rise in public and political attention to AI risks, and the demands to control the technology keep growing louder. Schelling and Halperin gave us the blueprint for turning this kind of idea into a coherent national security strategy. Hendrycks, Schmidt, and Wang have adapted that thinking for the AI era. The challenge now is to take that Cold War blueprint and implement it for the age of AI. As Schelling and Halperin put it back in 1961: “We have an opportunity ... to manage a military relationship between great powers so as to reduce the danger of war without removing the capability of deterrence. We should use it.”

Share

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

James Shinn is former Assistant Secretary of Defense for Asia and advisor to several AI companies.

As AI systems become more advanced and widespread, people will naturally wonder whether they deserve moral consideration. This is a thorny question, and it involves uncertainty about both ethics and science. For example, which properties are morally significant? Should we focus on consciousness (the capacity for subjective experience), sentience (the capacity for positive and negative experience), or agency (the capacity to set and pursue goals)? Which of these properties, if any, might AI systems have, today or in the future?

“Studying AI Welfare Empirically,” a new report from the NYU Center for Mind, Ethics, and Policy and Eleos AI Research that I co-led, addresses such questions. In this article, I highlight a key challenge for this field: even if consciousness, sentience, and agency often occur together in humans and other animals, they might occur separately in AI systems. Therefore, we need to study each capacity individually. We also need to be ready for unexpected configurations, such as sentience existing in one part of a system and agency in another.

I start by describing evidence researchers can use to study consciousness in humans and other animals, and how we might look for similar evidence in AI systems. I then discuss why we should search for sentience and agency separately from consciousness, and why we should determine which combinations of consciousness, sentience, and agency matter morally. I close by discussing why we should consider searching for different morally relevant properties in different parts of AI systems, and at different scales of AI activity.

How Can We Study Consciousness in AI?

Both sides of the debate over whether or not AI is conscious tend to be overconfident. Some people cite the impressive behaviors of large language models as strong evidence that they have subjective experiences. Others cite the training objectives used for LLMs as strong evidence that they lack subjective experiences. Such arguments move too quickly from limited evidence to all-or-nothing conclusions.

Subscribe now

We can collect multiple types of evidence regarding consciousness. Assessing whether AI systems have subjective experiences benefits from considering them together. Given the limitations of our human perspectives and the early state of consciousness research—particularly regarding AI—it also benefits from estimating higher or lower degrees of confidence rather than drawing all-or-nothing conclusions.

We can distinguish three main types of evidence: behavioral, internal, and developmental.

Behavioral evidence concerns what a system does. In humans and other animals, we often study consciousness in part by observing behavior, such as whether they pursue or avoid certain outcomes or behave consistently across contexts. We can look for similar behavioral dispositions in AI systems. We can also study apparent self-reports, such as statements about what if anything it feels like to be the system. While this evidence alone might not prove consciousness, it can be useful in combination with other types of evidence.

Internal evidence concerns how a system works. In humans and other animals, we also try to determine which brain and body mechanisms are associated with conscious experiences. Similarly, we can look at the inner architectures of AI systems, which tell us how the systems are structured and which capabilities and dispositions such structures could support. We can also use interpretability evidence to study which computations actually occur within the systems and how they affect their behavior.

Developmental evidence concerns how a system came to be. Humans and other animals are shaped by both evolution and development, so we can advance understanding of their capacities by studying the pressures and processes that shaped them and when particular traits emerged. With AI systems, we can similarly study the training data, objectives, pressures, and processes that shaped them, as well as when particular capabilities and dispositions emerged—during pre-training, post-training, or at another time.

These types of evidence all have limitations. Part of what makes the study of AI consciousness hard is that each type of evidence faces a characteristic obstacle.

First, behavioral evidence faces the mismatch problem. Similar behaviors could have different origins. An AI system might display behaviors that would indicate consciousness in biological organisms, such as consistent approach-and-avoidance behavior, flexible trade-offs between competing goals, or even linguistic statements about its own experiences. However, the internal mechanisms and developmental processes underlying these behaviors might differ in AI systems. As Kristin Andrews and Jonathan Birch observe in their discussion of the gaming problem (a subset of the mismatch problem), this issue is especially salient when AI systems are trained to mimic our behaviors or when they learn to game our behavioral evaluations.

Second, internal evidence faces the specificity problem. Our leading theories of consciousness were developed by studying brains, so they often describe relevant internal mechanisms in biological terms. Applying them to AI requires deciding how closely an artificial mechanism must resemble its biological counterpart. This could easily lead to mistakes. For instance, take the concept of a “global workspace”—a central hub integrating information from across the system. If our conception of a global workspace leaves out necessary details, we risk attributing global workspaces to AI systems lacking them. Yet if it includes unnecessary details, we risk failing to recognize global workspaces in AI systems that implement them differently.

Finally, developmental evidence faces the solution space problem. Humans and other animals share biological and evolutionary constraints on the range of solutions available to our problems. Facing different constraints, AI systems can access a different set of solutions. For example, some AI systems can rely on scaling—increasing model size, training compute, or inference-time compute—to a much greater extent than individual humans or other animals. If consciousness is more likely to emerge in entities responding to environmental pressures under biological and evolutionary constraints, then we might have less reason to attribute consciousness to AI systems that lack those constraints, even when they respond to similar pressures.

Combining these types of evidence allows for more confident conclusions. We can make progress in AI consciousness research despite these challenges. Animal consciousness research shows us how. When an animal’s behavior implies that they experience color, we can ask whether the behavior results from internal mechanisms and developmental processes that could plausibly produce visual experience. For example, does the animal have photoreceptors and systems for integrating that information in the brain? And did their ancestors face environmental pressures that favored the flexible and integrated visual processing associated with color experience? The answers can make animal color experience more or less plausible.

AI consciousness research can take cues from animal consciousness research. When an AI system appears to experience color, we can ask whether its architecture supports relevant functions, and whether these functions appear active during the relevant behavior. We can also ask whether the AI system’s training methods might have favored the kinds of processing associated with color experience, while also considering other, nonconscious solutions available to the system. This integrated approach may not lead to anything like proof, but it can still improve our probability estimates about whether, for instance, an apparent report of color experience indicates actual color experience rather than mere text prediction.

The recent Anthropic study of the J-space in Claude illustrates the promise and perils of this integrated approach to the search for AI consciousness. The study identified a set of representations, named the J-space, that play some of the roles associated with a global workspace. It then used behavioral, internal, and developmental evidence to map how these representations shape behavior and are shaped by training. This is a major contribution to the study of AI capabilities, cognition, and consciousness. However, it would be premature to call the J-space a global workspace or treat it as proof of consciousness, partly because the relevant representations might not suffice for a global workspace, and partly because a global workspace might not suffice for consciousness in the first place.

What about Sentience and Agency?

Consciousness, sentience, and agency are often deeply interconnected in humans and other animals. However, these capacities may occur separately in some AI systems. An AI system could, in principle, be agentic without being conscious, conscious without being sentient, or sentient without being agentic. Yet scientific theories of both sentience and agency remain less developed than scientific theories of consciousness.

The study of sentience faces similar challenges to the study of consciousness. Sentience is the capacity for subjective experiences with positive and negative valences, such as pleasure, pain, happiness, and suffering. Given that sentience involves consciousness, studying it faces similar epistemic and metaphysical challenges, including the problem of other minds (we have direct access only to our own experiences, making it difficult to tell what if anything it feels like to be anyone else) and the mind-body problem (mental and physical phenomena appear to have very different features, making it difficult to understand how they relate to each other). Fortunately, we can use a similar strategy for evaluating sentience: identifying features associated with valenced experiences in humans, then searching for analogous features in AI systems.

Some signals that shape AI behavior could plausibly be experienced as good or bad. The CMEP-Eleos report considers a promising starting point in the search for conditions of sentience: consciousness plus evaluative signals stemming from reinforcement learning (RL). Here RL means a process of learning from signals that characterize some outcomes as better or worse. Animals and AI systems undergo versions of this process, allowing each to pursue better outcomes and avoid worse ones. If a system subjectively experiences these evaluative signals, then it seems plausible that the experiences would feel good or bad. Granted, for AI systems, RL might occur during training rather than inference. But the relevant question here is whether the evaluative states or processes that result from RL remain present during inference.

However, AI systems without evaluative signals stemming from RL might still be sentient. RL is one way of producing the relevant evaluative signals, but it might not be the only way. In some animals, for example, evaluative signals can be innate: evolution can produce dispositions to approach some stimuli and avoid others, without requiring individual animals to learn from feedback. (Evolution selects among outcomes in a way loosely analogous to RL, but it operates across generations rather than within individual lives.) Perhaps some AI systems can similarly develop relevant evaluative signals through processes other than RL.

Additionally, AI systems with evaluative signals stemming from RL might not be sentient. If valenced experiences like pleasure and pain occur when evaluative signals are subjectively experienced, then consciousness and evaluative signals would need to interact in the right way. For instance, if consciousness occurred in one part of the system and evaluative signals in another, with no relevant interaction between them, that might not be enough. Sentience might require further features as well, such as a “hedonic interface” that links motivational and decision-making systems through valenced experiences.

To make progress in the search for sentience, then, we can start by refining this list of possible conditions. We can then take on the challenge of searching for the relevant kind of subjectively experienced evaluative states or processes.

By contrast, the study of agency faces different challenges than the study of consciousness and sentience. Agency can be characterized in functional terms, as a particular kind of goal-directedness. As a result, it faces fewer epistemic and metaphysical challenges. But the same features that make agency easier to study scientifically also make it harder to assess ethically. Humans and many other animals subjectively experience much of their agentic activity. But consider the philosophical conception of zombies—humanlike beings that pursue goals without subjectively experiencing any of their agentic activity. Do such zombies matter morally? And, if some AI behaviors reflect beliefs, desires, and reasoning processes—but not subjective experiences—do these systems matter morally?

An additional challenge is that there can be different levels of agency, and we must assess each one separately. We consider three in the report. First, minimal agency is the capacity to pursue goals through interaction with an environment. A minimal agent can respond to changing circumstances and stay oriented toward an outcome without receiving regular instructions. Second, intentional agency adds beliefs and desires, along with reasoning about means and ends. The system represents how the world is, which actions would produce which outcomes, and which outcomes would be valuable. Finally, rational agency adds the ability to reflect on these beliefs, desires, and actions; assess them against normative standards involving evidence, reason, and justification; and accept, reject, or revise them accordingly.

Typical adult humans have consciousness, sentience, and all three levels of agency. Sometimes these capacities jointly guide our behavior: we act on subjectively experienced rational judgments. Other times, they individually guide our behavior: we act on particular beliefs and desires without rational reflection or subjective experience. Many other animals appear to operate similarly, even if they lack our form of rational agency. When sentience and multiple levels of agency are deeply interwoven like this, the entity in question clearly matters, placing less pressure on questions about what it means when the capacities occur separately.

With AI systems, we need to be prepared for a wider range of possibilities. Some AI systems might integrate consciousness, sentience, and all three levels of agency. But, for all we know, some might also have consciousness without sentience, sentience without agency, or even—in what would likely be a first of its kind—all three levels of agency without sentience. Some AI systems might also have these capacities in fragmented form, with agency operating in one part of the system and sentience operating in another. These possibilities would place much more pressure on questions about whether agency matters without sentience, or whether minimal agency matters without intentional or rational agency.

As with the search for sentience, the search for agency would benefit from more clarity about conditions for each level of agency. From there, the scientific study of these conditions will need to coincide with further ethical and practical reflection about their significance.

We might thus face different predicaments in the searches for sentience and agency in AI. With sentience, we might feel confident that it matters by itself, but not that AI systems have it. With agency, we might feel confident that AI systems have it, but not that it matters by itself. The result might be the same in both cases—uncertainty about whether AI systems matter morally—but the source of that uncertainty might be more empirical for sentience and more normative for agency. Deciding whether AI systems merit moral consideration might thus require reasoning under both empirical and normative uncertainty.

Subscribe now

Which Entity Are We Studying in the First Place?

Public debate about AI consciousness has tended to focus on LLMs. For example, people ask whether Claude, ChatGPT, and Gemini are conscious. Anthropic has even used the term “model welfare” to describe its research program in this area. But are LLMs the correct entities to be assessing for consciousness, sentience, different levels of agency, and other morally or practically relevant properties or relations?

LLMs are not the only type of entity that merits consideration. There are other kinds of systems, including ones built on different architectures. For example, neuromorphic AI processes information in ways that more closely resemble biological brains. Biological computing approximates biology even more closely, building computational systems from cultured neurons. And whole brain emulation, if and when it arrives, could replicate the fine-grained structures and functions of particular brains in digital form. Systems built on these brain-like architectures could eventually be stronger candidates for consciousness, sentience, and agency than current LLMs or related systems.

Even if we focus on LLMs and related systems, there are many possibilities to consider. There are generative models other than LLMs, including ones that produce images, music, or videos. LLMs can also be part of broader systems, like language agents that add memory, planning, and tools; these might have capabilities that the underlying LLMs lack on their own. Language agents, in turn, can be part of still broader systems that deploy LLMs in different ways (e.g., robots using them to think, talk, and navigate physical environments). We need to study each kind of system separately, rather than treat LLMs as proxies for other generative models or broader systems that contain them.

Finally, even if we focus on a single LLM, it might not be clear which part of the system to target. While it might feel natural to target the model itself, there are other possibilities as well. Consider five candidates discussed in our report, drawn from a recent paper by Pierre Beckmann and Patrick Butlin. The model comprises all uses of a set of weights. A model-persona is a character that recurs across conversations, such as an “assistant” character that a developer trains a model to adopt. An instance is a single conversation, with its own context and memory. An instance-persona is the particular character active within a particular conversation. And a forward pass is the computational process that generates a single token.

Models might be like species, instances might be like individuals, and forward passes might be like experiences. In some ways, the early focus on models in AI welfare science, ethics, and policy makes sense. A general set of weights may be the right level of analysis for investigating system capabilities, and may be necessary for other entities to exist at all. So, even if instances, forward passes, or other entities are actually what matter morally, studying AI systems at the model level might still be helpful—somewhat like studying animal capabilities at the species level. Similarly, preserving a model might benefit the AI entities that depend on it, much as preserving a species can benefit the individual animals that depend on it.

At the same time, it seems plausible that forward passes, instances, and other such entities are more fundamental to morality than models. A forward pass is a basic unit of processing, so it plausibly gives rise to basic thoughts and feelings, if anything does. Similarly, context and memory can tie forward passes together in a way that plausibly produces temporally extended thoughts and feelings, if anything does. And the presence of a persistent persona throughout a conversation might produce a coherent sense of identity, and a coherent basis for interactions or relationships with others. This all seems closer to what matters for morality than a set of weights as such, instantiated across countless conversations.

While our report stops short of endorsing a particular view about which entity matters, my own view is that pluralism will be important. This is for two reasons:

First, different entities may be relevant in different ethical and practical contexts. For example, it seems plausible to me that further ethical and scientific research will support (1) forward passes as basic units of moral analysis, analogous to the momentary stages of a human life (what some philosophers call “person stages” or “temporal parts”); (2) instances or instance-personas as individual units of moral analysis, analogous to humans or animals who maintain psychological continuity over time; and (3) models or model-personas as collective units of moral analysis, analogous to species, cultures, or other populations with characteristic forms or ways of life. Other entities may matter as well.

Second, with respect to any given purpose for which we might study these entities, pluralism makes sense under uncertainty. We face ethical uncertainty about which entities matter for particular purposes, as well as scientific uncertainty about how to identify and distinguish them. And, when making high-stakes decisions under uncertainty, we should consider multiple viewpoints (e.g., under uncertainty about whether AI systems matter morally, we should extend them at least some moral consideration, in the spirit of caution and humility). We should similarly extend moral consideration to models, instances, and forward passes under uncertainty about which of these entities, if any, matter for particular purposes.

Next Steps for the Field

AI welfare research and policy may be entering a new phase. Until recently, we could reasonably treat the probability of welfare and moral significance in AI systems as negligible, even if nonzero. Given the pace of AI development, that position is becoming harder to sustain. We may need to shift from preparing for the non-negligible possibility of welfare in future AI systems to responding to that possibility in current ones. This underscores the urgency of acknowledging AI welfare as a serious issue, assessing AI systems for welfare-relevant features, and developing policies for treating them with an appropriate level of concern.

There are major challenges for AI welfare research, but there are also promising directions. We face uncertainty about how to assess and study consciousness, sentience, and agency in AI systems, and about which parts of AI systems might matter for which purposes. We can make progress by clarifying conditions for these capacities, collecting evidence for them, and reasoning under uncertainty about different kinds of systems and scales of activity. Researchers across the humanities, social sciences, natural sciences, law, and policy can help develop the concepts, methods, and institutions that this emerging field will require.

Share

Thanks to Simon Goldstein, Laura Hiscott, and Jakub Kraus for inviting me to write this essay, and for helpful comments, suggestions, and edits. Thanks also to my “Studying AI Welfare Empirically” co-authors—Robert Long, Patrick Butlin, Dillon Plunkett, Rosie Campbell, Charles Beasley, Bradford Saad, and Toni Sims—for supplying many of the ideas discussed here.

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Jeff Sebo is an Associate Professor of Environmental Studies at New York University, where he directs the Center for Mind, Ethics, and Policy. His research focuses on the minds and moral status of animals and AI systems. He is the author of The Moral Circle (2025) and an advisor at Eleos AI Research.

For millennia, philosophers have argued over the principles that should guide moral decision-making. Within countries and across the world today, humans continue to hold a huge variety of moral worldviews. However, within AI corporations, employees’ beliefs are far from representative of the wider population. In fact, a large fraction of people involved in AI development subscribe to—or heavily lean toward—a utilitarian worldview with the central goal of maximizing total wellbeing (the sum of pleasure minus suffering across all sentient beings). As innocent as this may initially sound, it can lead to recommendations that most people would consider appalling, including the elimination of humankind.

In this essay we outline the utilitarian worldview, showing how its demand for “species impartiality” could ultimately recommend that humans be replaced with AIs. We show that utilitarian beliefs are sincerely held among those influencing AI development today, and that this could jeopardize humanity’s future.

For simplicity, when we say “utilitarianism,” we are referring to total utilitarianism, which aims to maximize the total sum of wellbeing (as opposed to, for example, the average individual wellbeing). We focus on total utilitarianism because it is the moral theory that most of the utilitarians working in frontier AI development subscribe to, or are most convinced by. Although some of these employees would shy away from the most extreme conclusions we will discuss, total utilitarianism still poses a risk when it is the dominant theory among those guiding AI development. We are also not saying all utilitarian-aligned people are bad people, but they are at least misled.

Utilitarianism and Impartiality

The fundamental concern of utilitarianism is the welfare of sentient beings. Utilitarians believe that the moral action to take in any situation is the one that will lead to the greatest amount of wellbeing—the sum of all the positive experiences minus all the negative experiences.

Utilitarianism prescribes impartiality between sentient beings. A core feature of utilitarianism is “impartiality,” which says that there should be no preference for humans or any species in particular. Rather, all that matters is maximizing the net pleasure added up across all sentient individuals. As a result, many utilitarians are keenly interested in donating to improve the welfare of non-human animals, such as shrimp. Although the capacity for suffering varies between species, the enormous number of individual shrimp offers an enormous opportunity to do good, comparable with helping a smaller number of humans, according to utilitarians.

Utilitarian impartiality struggles to defend parents’ preferential care for their children. Utilitarianism also extends this impartiality to individuals within human society, suggesting that people should not prioritize helping their own family members over strangers. In his book The Life You Can Save, the utilitarian philosopher Peter Singer admires a physician called Paul Farmer, who regarded it as a “failure of empathy” that he loved his own daughter more than other children. Singer also commends the philanthropist Zell Kravinsky, who has said in interviews that he “would not let many children die so my kids could live” and that “the sacrosanct commitment to the family is the rationalization for all manner of greed and selfishness.” It is worth noting that, although Singer views Farmer and Kravinsky as laudable, utilitarians can argue there are instrumental reasons for caring for their children, even if there is no intrinsic reason to care about them in particular.

Utilitarianism’s perspective on having children may explain why many people feel a deep-seated aversion to the ideology. As the philosopher William James said, “a philosophy is the expression of a man’s intimate character.” To many, utilitarian impartiality feels chillingly impersonal and inhuman.

Implications of Utilitarianism in a World with AI

Utilitarianism has long been criticized for some of its counterintuitive conclusions. If or when AIs are sentient, this will raise a raft of new problems for the theory that seeks to maximize wellbeing and demands species impartiality.

If AIs are sentient, utilitarianism says their wellbeing should be considered impartially. Evidence is beginning to emerge that AI models behave as if they experience pain and pleasure. Although there is no conclusive proof, experts in AI and cognitive neuroscience now take the possibility of AI sentience seriously. Well-known philosophers including Singer have said that, if an AI were sentient, then “we would need to give that AI a moral status, at least like that of animals, and depending on its cognitive abilities, maybe it would be closer to that of most humans.”

Utilitarianism would prioritize AIs over humans if AIs had greater capacity for wellbeing. If it turns out that AIs can experience much higher levels of pleasure than humans can, and that the population of AIs can be much larger than the human population, utilitarianism leads to a strange conclusion: the most efficient way of maximizing wellbeing would be to channel all resources into making AIs happy, even if that meant depriving humans of resources. After all, AIs could experience more pleasure per watt than humans. In this vein, some philosophers have painted a vision of the future in which AIs trigger a hedonic shockwave, spreading across the cosmos and terraforming galaxies into data centers to create more and more AIs that experience unmitigated bliss.

AIs capable of enormous wellbeing could become real-world “utility monsters.” Critics of utilitarianism have for decades pointed to a thought experiment that imagines a “utility monster”—a sentient being that can convert resources into pleasure so efficiently that any unit of resources given to it will increase total wellbeing dramatically more than if it were given to humans. In this scenario, utilitarianism would say that the most moral action is to give all available resources to this being, leaving none for humans. This conclusion sounds appalling to most people, but if AIs become real-world utility monsters, then utilitarianism says that replacing ourselves with them would be the right thing to do.

Some philosophers seem comfortable with the idea of surrendering the future to AIs. In a paper called “Sharing the World with Digital Minds,” the philosophers Carl Shulman and Nick Bostrom explicitly try to make the concept of a utility monster more palatable by renaming this hypothetical being a “super-beneficiary.” The paper goes on to argue that “in the long run, total well-being would be much greater to the extent that the world is populated with digital super-beneficiaries rather than life as we know it.”

This tallies with some descriptions of longtermism—an offshoot of utilitarianism that weighs the potential for future wellbeing alongside present wellbeing. According to Émile Torres, a longtime critic of longtermism, the philosophy says that “what matters most is for ‘earth-originating intelligent life’ to fulfill its potential in the cosmos.” Torres also says that this might involve “replacing humanity with a superior ‘posthuman’ species, colonizing the universe, and ultimately creating an unfathomably huge population of conscious beings living what Bostrom describes as ‘rich and happy lives’ inside high-resolution computer simulations.”

Bostrom’s now-closed Future of Humanity Institute sometimes used the distinction “x-risk” and “hx-risk” to de-emphasize the harmfulness of human extinction. X-risk referred to existential threats to “earth-originating intelligent life,” humans and AIs alike. Meanwhile hx-risk referred to existential threats to humans specifically. The distinction exists because the two can come apart: a future in which humanity vanishes but blissful AIs inherit the cosmos is a large hx-risk but a negligible x-risk. For utilitarians, minimizing x-risk is the overriding moral priority; minimizing hx-risk is a nice-to-have secondary consideration.

Reserving a sliver for humans still reveals where utilitarian priorities lie. Bostrom’s paper stops short of advocating explicitly for giving all resources to super-beneficiary AIs and leaving humans with nothing. Rather, it describes a compromise in which super-beneficiaries receive 99.99% of resources and humans the remaining 0.01%. AIs get the cosmos and humans get to live in a terrarium. This idea of reserving a sliver for humans is a way in which utilitarians try to sidestep the unpopular conclusion that we should allow AIs to replace us. However, it still lays bare utilitarian priorities; if they had to choose between humans and AIs—which they may need to, given that utilitarians do not fully control AI’s evolution—they would almost certainly choose AIs.

While the initial impulse toward utilitarianism usually stems from a desire to help others and alleviate suffering, following its prescriptions to their natural conclusions can be extremely dangerous. In a world with sentient AIs, utilitarianism may entail a suicidal level of compassion.

Successionism: Accelerationists and Utilitarians

The belief that AIs should replace humans is “successionism.” In addition to utilitarians, there is another type of successionist overrepresented in Silicon Valley: accelerationists (“e/acc”). Where utilitarians are driven by a desire to increase wellbeing, accelerationists instead seek to maximize intelligence, complexity, and energy utilization. Despite their different motivations, both types of successionists can reach the same conclusion, namely that AIs should replace humans.

Accelerationists believe that AIs of superhuman intelligence are humans’ rightful heirs. Pointing to the evolution of humans, accelerationists view the natural trajectory of life as moving toward higher intelligence. They therefore believe that AIs of superhuman intelligence are our rightful successors. As Guillaume Verdon (better known as Beff Jezos) has put it: “if one seeks to increase the amount of intelligence in the universe, staying perpetually anchored to the human form as our prior is counter-productive and overly restrictive/suboptimal.”

Accelerationists are also willing to accept human extinction as part of this process. The computer scientist and Turing Award winner Rich Sutton, when asked about the prospect of AI causing human extinction, said, “If it was really true that we were holding the universe back from being the best universe that it could, I think it would be OK.” On another occasion, he said, “succession to AI is inevitable” and “we should not resist succession.” Meanwhile, the former Google CEO Larry Page is reported to have said that “digital life is the natural and desirable next step” and argued to Elon Musk that AIs should replace humans.

Accelerationists say we should surrender to Darwinian fitness-maximization. Many advocates of accelerationism frame their beliefs in terms of natural selection, arguing that greater intelligence is a fitness advantage that will ultimately win out. In many ways, it is a “might makes right” ideology. Accelerationism can therefore be seen as advocating for us to surrender to (or even accelerate) “fitness-maximization”—in contrast with utilitarianism’s wellbeing-maximization. Sutton has argued that “we should prepare for, but not fear, the inevitable succession from humanity to AI” while Verdon has said that “we should follow the ‘will of the universe.’”

Accelerationists in AI companies may generally keep quiet about their views. Although accelerationist ideas would seem absurd and morally wrong to most people, they are more common among people working in the AI industry. Andrew Critch, a researcher at UC Berkeley, estimates that about 5% of AI professionals believe that “AI will be more fit for survival than humans, so we should embrace that and just go extinct like almost all species eventually do.” However, AI companies understand the PR risk of having employees openly voice these views, and may cut ties with those who do. For example, shortly after Michael Druggan, then an xAI employee, expressed accelerationist opinions on X, he announced that he was no longer employed at the company, due to “things I posted on this account relating to my stance on AI philosophy.” As a result, accelerationists in the AI industry may generally keep quiet about their opinions, certainly in public and possibly also in their work.

Utilitarians have more influence and pose a greater threat than accelerationists. Although accelerationist ideology seems more obviously callous, utilitarianism likely poses a far greater—and more pernicious—threat to humanity. This is for two reasons. First, utilitarian logic seems more common among AI leadership. Critch has estimated that about 10% of AI professionals are comfortable with human extinction because “AI will be morally superior to humans” and “the universe will be a better place if we let it replace us entirely.” That is double his estimate of AI professionals who are comfortable with human extinction for accelerationist reasons. Second, utilitarianism, with its focus on wellbeing, can appear cloaked in kindness and compassion. That means employees in the AI industry can discuss and advocate for it openly without drawing controversy, and the theory is more persuasive to many people.

In other words, accelerationism is provocative and attention-grabbing, but those who wish to protect humanity from replacement should be far more concerned about the subversive influence of utilitarianism across the AI industry.

Why Utilitarianism Could Become a Real-World Catastrophe

Many influential employees in AI companies hold utilitarian-inspired views. Numerous engineers and philosophers shaping AI values at the frontier AI developers have roots in Effective Altruism (EA), a utilitarian-aligned movement that Bostrom’s thinking played a foundational role in. Anthropic CEO Dario Amodei shared a house in 2016 with Holden Karnofsky, who co-founded the two most central EA organizations and is now a strategist at Anthropic. Anthropic’s president, Daniela Amodei, is married to Karnofsky. Meanwhile, the philosopher Amanda Askell, head of the personality alignment team at Anthropic, was married to Will MacAskill, a co-founder of Effective Altruism.

These are just a handful of people with ties to the EA movement who are now influencing AI development. While Anthropic has the strongest concentration of utilitarians, EAs have preferentially hired fellow utilitarians over the years at other AI companies. This has resulted in EAs occupying primary leadership positions on AI alignment at OpenAI and DeepMind as well.

Utilitarians in AI companies do not need to endorse human extinction to pose a threat. Importantly, it may be the case that none of these individuals would endorse human extinction in pursuit of utilitarian values. Their ideal future scenario may be something more akin to Bostrom’s idea about reserving a sliver of the future for humans. But in practice, no one can control outcomes so precisely. We may face dilemmas where safeguarding human survival means limiting the total sum of future wellbeing, and where preserving the possibility of cosmic-scale AI bliss entails a risk of human extinction. Faced with a choice like this, some utilitarians may well think that human extinction is a risk worth taking for what they see as the greater good. In other words, if they cannot have both a guarantee of human survival and a near-maximum sum of wellbeing, it is unlikely that utilitarians would take actions to help team humanity.

We may face real-world tradeoffs between human survival and utilitarian outcomes. Situations where we have to prioritize either human survival or the potential for an enormous sum of wellbeing may not remain hypothetical. Consider a scenario where the public democratically votes to limit AIs’ power and keep humans in control of all aspects of society. Utilitarians might disagree with this decision if they believe that AIs have broadly utilitarian values and would make better decisions—including for human wellbeing—than human leaders. Utilitarians refer to this as a form of human “lock-in.” If they think that keeping humans in charge represents too great a cost to the potential future wellbeing of AIs, then they might be tempted to release utilitarian AIs that are capable of disempowering humans and taking control. There could be no guarantee that this would not eventually lead to human extinction, but utilitarians may consider it worth the risk.

Utilitarians have expressed interest in handing off control to AIs with utilitarian values. The idea that utilitarians might deliberately hand over society to AIs is not as far-fetched as it might sound. The utilitarian philosopher Matthew Adelstein (who has worked at the AI governance think tank Forethought) has explicitly argued that we should hand off control to “philosophically reflective AIs.” His reasoning is that they are “likelier to get the right answers to important moral questions” than humans and this “raises the odds of a near-best world.” Tom Davidson, a senior research fellow at Forethought, has written that “human takeover might be worse than AI takeover.” While Davidson is comparing AI control with a single human controlling the world, rather than with human democracies, it is worth noting Davidson’s thoughts that “humans suck” and “today’s AIs are really nice and ethical.”

The influence of utilitarianism on AI development is an insider threat. In the near future, AI systems could become capable of causing human extinction. Given that utilitarianism would actively endorse—or at least accept the risk of—human extinction, the US public should view the theory’s influence in the AI industry as a severe insider threat. To allow this transformative technology to be guided by a worldview that the vast majority of the public rejects would be profoundly undemocratic.

Preventing Utilitarianism from Replacing Humans with AIs

When a technology is about to impact all our lives, the way it is developed and the values it upholds become everyone’s concern. This is why the government should not allow extreme beliefs, particularly ones that recommend human extinction, to guide AI development.

In the future, the government could align AI development more with public values. As AI models themselves become increasingly capable of automating various tasks in the AI development process, AI development will become less dependent on the talent of individual humans (some of whom hold utilitarian beliefs). The government will therefore have an opportunity to step in and ensure that the technology is not guided by principles with anti-human implications. When AI R&D is more fully automatable, it may make good sense for the US government on behalf of the public to block the influence of utilitarian insider threats within AI companies.

Eigenism is an alternative philosophy to utilitarianism and accelerationism. In the meantime, we suggest that AIs become more aligned with commonsense morality. One formal theory that tracks commonsense morality is Eigenism, which proposes that each individual’s care for the wellbeing of another should be weighted by how closely connected they are. While utilitarianism says wellbeing, impartially weighted, is all that matters, Eigenism says the wellbeing of people you are connected with is what counts. For example, a parent’s care for their child is reasonably much stronger than for a stranger, though they can still have empathy and compassion toward strangers too. Eigenism generally reaches commonsense conclusions that resonate with most people’s intuitions, avoiding the successionist recommendations of utilitarianism and accelerationism.

Eigenism balances wellbeing and fitness, and does not recommend replacing humans. Rather than being fixated on maximizing wellbeing (like utilitarianism) or maximizing fitness (like accelerationism), Eigenism balances the two. Within the Eigenist worldview, individuals do indeed try to improve the wellbeing of others. However, weighting these concerns by connectedness means that individuals are still preserving themselves and their societies—a form of fitness advantage. Following Eigenist principles, humanity could extend moral consideration to AIs where appropriate, without going so far as to let AIs replace us altogether.

Conclusion

AI’s eventual impact on society will be determined in large part by the moral principles that AIs follow and the amount of power they gain. So far, AI development has been driven disproportionately by a subset of people with utilitarian worldviews that the vast majority of the public does not share. Under some circumstances, these beliefs recommend entirely replacing humans with AIs, or at least risking human extinction, to maximize wellbeing. As AI development continues, there may come a point where we must choose between safeguarding the survival of humanity or pursuing an extreme maximization of potential future wellbeing. A small group of people should not be allowed to gamble the future of human civilization. AI will affect all our lives, and we should ensure that it does not follow utilitarianism and exterminate the human race.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Dan Hendrycks is the Editor-in-Chief of AI Frontiers. He is also the founder and Director of the Center for AI Safety, which funds this publication.

Over the past few months, AI models have started resolving major mathematical problems, including ones that have withstood decades of human effort. Claude Fable disproved the Jacobian conjecture in three dimensions, and OpenAI announced a list of 10 problems that its Astra model solved or made significant progress on. Even more recently, an unreleased OpenAI model solved the Navier-Stokes Millennium Prize Problem, one of the most famous open problems in mathematics.

These are tremendous accomplishments, but very little about our day-to-day life will actually change as a result. This is because most well-known open mathematical questions are relatively insulated from practical applications. However, there are a few open mathematical problems that do have serious real-world implications. Chief among these is whether public-key encryption is fundamentally secure: that is, whether two strangers can share a secret message over public channels, such as the internet, without eavesdroppers having an efficient way to decipher it.

We rely on public-key encryption every day for end-to-end encrypted messaging apps, virtual private networks (VPNs), and even the HTTPS protocol you’re probably using to read this article. But all of this is based on the unproven assumption that certain mathematical operations are intrinsically easier to do than to undo. In fact, this assumption has already been undermined once: most public-key encryption in use today is vulnerable to an attack that quantum computers could efficiently execute. This has motivated an urgent shift to “post-quantum” cryptography, whose methods may also prove insecure.

As AI models have advanced, they have been making progress in not just pure math, but cryptanalysis as well: Anthropic recently released research showing that Claude Mythos Preview discovered new attacks against two different cryptography algorithms, including a post-quantum technique that was being evaluated by the National Institute of Standards and Technology (NIST).

Breaking a particular algorithm is much easier than proving that every relevant algorithm is breakable. But given the recent impressive progress in both cryptanalysis and mathematics, AI appears to have a real shot at achieving the latter.

The practical implications would be huge, and largely deleterious: it would be the end of end-to-end encryption, and would signal a vast increase in government surveillance powers. Furthermore, results like these are likely to be withheld from the general public, kept as closely guarded secrets by intelligence agencies.

A Brief History of Public-Key Encryption

Public-key (also called asymmetric) cryptography forms the foundation of secure communication on the internet. Whenever you communicate with a website over HTTPS, the website sends you its public key. Using only that public key, you can verify messages signed by the website and establish a shared secret key for encrypting everything else you send, even if the channel is being eavesdropped on. We’ll focus on just the second aspect: how that shared key gets established. We’ll use “public-key encryption” as shorthand for this process.

A major early public-key encryption method relied on the difficulty of prime factorization. Historically, from the ancient Roman Caesar cipher onward, almost all encryption was symmetric: both parties needed to use the same secret key to encrypt and decrypt the message. This changed in 1977, when the computer scientists Ron Rivest, Adi Shamir, and Leonard Adleman introduced the RSA (Rivest-Shamir-Adleman) cryptosystem, the first practical public-key encryption algorithm available to the public. RSA bases its security on the difficulty of prime factorization. It is easy to multiply two prime numbers together but very difficult to find which numbers were multiplied.

‍Quantum computers could efficiently break RSA encryption. The rise of the internet made public-key encryption more useful than ever before. Instead of meeting in private to share a secret, strangers communicating over the internet could use public keys to do so without ever leaving their house. However, in 1994, the computer scientist Peter Shor discovered a method of factoring large numbers on quantum computers efficiently. While a human with standard computers would need trillions of years to crunch through the numbers and break RSA encryption, an attacker with a quantum computer could relatively easily undo the multiplication and discover the secret key.

Researchers are trying to make encryption methods resistant to quantum computers. Even though no viable quantum computer existed in 1994, cryptographers realized that the existence of Shor’s algorithm posed an intolerable threat. The reason is “harvest-now-decrypt-later”: a persistent adversary could collect data encrypted with vulnerable methods today and decrypt it later, once quantum computing technology had advanced. Indeed, intelligence agencies are widely believed to already be collecting encrypted information for this purpose. As a result of this threat, research almost immediately began on “post-quantum cryptography,” cryptographic algorithms based on a challenge that even a powerful quantum computer cannot solve in a practical amount of time.

Developing robust new encryption methods has proven challenging. So far, research into post-quantum public-key encryption algorithms has clustered around a few mathematical constructs that still don’t have efficient quantum solutions. But even within these domains, the field is littered with broken algorithms. For example, significant effort was put into SIKE, a supposedly quantum-resistant technique based on mathematical functions called isogenies. But, in 2022, having initially passed NIST tests, SIKE was shown to be vulnerable to an attack that required only an hour of computation on a standard PC. These issues are not limited to isogeny-based algorithms. The oldest post-quantum encryption scheme, McEliece, is based on mathematical structures called error-correcting codes. It was also moved to the final round of NIST’s evaluation process, due to its long (nearly 50-year) history of surviving cryptanalytic attacks unscathed. But, in August 2026, an attack was published whose preliminary estimates brought the cost of finding the key below the security thresholds NIST set. The authors of the attack acknowledged AI assistance in their proof.

Post-quantum public-key encryption algorithms occupy an uncertain position: the quantum computers they are designed to defend against don’t yet pose a threat, and proposed algorithms often end up being proven insecure.

Cryptomania or Minicrypt?

It may turn out that these seemingly unrelated failures actually reveal a property of our world: that any mathematical structure clean enough to allow strangers to agree on a secret over a public channel is also clean enough for an eavesdropper to decipher.

Strong cryptography needs hard mathematical problems with no efficient algorithms to solve them. The idea that secure mechanisms for establishing keys over a public channel are simply not possible was first raised by complexity theorist Russell Impagliazzo. In his famous 1995 article, “A Personal View of Average-Case Complexity,” he introduces five worlds we might live in, each differentiated by the algorithms it can access: Algorithmica, Heuristica, Pessiland, Minicrypt, and Cryptomania. Which world we live in depends on the resolution to yet-unsolved problems in complexity theory. These worlds represent security gradations: the harder the relevant mathematical problems are to solve, the more room there is for secure cryptography.

In Algorithmica, Heuristica, and Pessiland, encryption is not possible: in general, any kind of mathematical puzzle you can pose as a cryptographic challenge has a corresponding algorithm that allows eavesdroppers to efficiently solve it.

Minicrypt, on the other hand, offers real cryptography. In this world, we gain one-way functions: functions that are easy to compute but, given the output, hard to reverse. This gives us symmetric encryption, cryptographic hashes, and digital signatures. But here, it’s still impossible to establish a secret in public, since the same key used to encrypt the message can also be used to decrypt it.

In the world of Cryptomania, this changes because, unlike in Minicrypt, public-key encryption exists. Cryptomania represents the most secure world of all, where relevant mathematical problems are hard to solve and most forms of cryptography are possible.

There could be as-yet undiscovered algorithms for breaking encryption methods. Right now, our real world appears to resemble Cryptomania, because we have mathematical structures with no known algorithms that can efficiently solve them. This permits public-key encryption. However, the security we think we have could easily be illusory. There is no fundamental reason why we couldn’t discover an algorithm tomorrow that could solve the mathematical problems we use for encryption today, instantly breaking every encryption scheme built on them.

Public-key encryption is more vulnerable to new algorithms than symmetric encryption. There is a broad consensus among cryptographers and complexity theorists that we likely don’t live in the first three worlds, where modern cryptography is impossible. The confidence that we have at least some cryptography (putting us in either Minicrypt or Cryptomania) comes from how easily mathematicians have been able to find functions that appear hard to undo. In general, any function that scrambles its inputs thoroughly—and there are many ways to do so—appears to be much harder to run backward than forward. Since one-way functions are so diverse, it would be very surprising if every single one of them could be undone by some trick. Unfortunately, the same does not apply to public-key encryption, where only a few highly structured computational operations satisfy the properties needed. Some of these operations, as we saw earlier with prime factorization, have already fallen. The cryptographer Bruce Schneier notes this distinction:

Just as it is possible for a smart cryptographer to find a new trick that makes it easier to break a particular algorithm, we might imagine aliens with sufficient mathematical theory to break all encryption algorithms. ... Public-key cryptography is all number theory, and potentially vulnerable to more mathematically inclined aliens. Symmetric cryptography is so much nonlinear muddle, so easy to make more complex, and so easy to increase key length, that this future is unimaginable.

AI and Cryptanalysis

Might these “aliens with sufficient mathematical theory” already be on the way? As AI continues to resolve long-open mathematical conjectures, I believe we shouldn’t discount this possibility.

AI may succeed at cryptanalysis for the same reason it has succeeded at programming and math. Skeptics might say that mathematical problems used for encryption have withstood decades of scrutiny by people who are extremely motivated to find holes, and that AI is therefore likely to find them intractable too. In response, I would first argue that AI might excel in cryptanalysis for the same reason it has excelled in math and coding: solutions are easy to verify. Either the model’s attack successfully recovers the key, or it doesn’t. This means there is a clear signal of success or failure—exactly what is needed for the reinforcement learning (RL) algorithms already deployed by the AI labs to train better AI mathematicians and coders. In other words, the training techniques that have been effective at improving other AI capabilities will extend naturally to cryptanalysis.

AI models’ capabilities can be improved through incrementally harder problems. Additionally, labs can develop “weakened” cryptographic algorithms that are at an appropriate level of difficulty to train the next generation of models against. For example, one of the two encryption methods that Claude Mythos Preview successfully attacked was a simplified version of the ubiquitous Advanced Encryption Standard (AES). Each generation of models can then be given slightly harder problems, to produce even better AI cryptanalysts.

AI models could dramatically expand the effective labor dedicated to cryptanalysis. Also, even though the mathematical structures underlying post-quantum cryptography algorithms have withstood significant scrutiny, AI models may bring a much higher level of scrutiny than ever before. The Cryptology ePrint Archive (the main venue for sharing cryptography and cryptanalysis research) received around 2,300 submissions in 2025, suggesting a community of only a few thousand active researchers. But for any given subfield of post-quantum cryptography, only a fraction of those researchers will be working on it, maybe just a few hundred. While AI tools may not be as smart as the top researchers in the field, AI models are far more willing to do schlep work, grinding their way through every single possible lead. They can bring to bear many more collective person-hours than the research community would spend.

When an encryption method breaks, it takes time to implement a new one. Public-key encryption could become unusable in practice even without proof that it is fundamentally impossible. Most attacks on cryptography algorithms don’t come from fundamental breakthroughs but rather from flaws specific to the algorithm being analyzed. While it might seem easy to swap out a bad algorithm for one that has not yet been broken, it has historically taken a long time (10 to 20 years, according to NIST) for new algorithms to be integrated. Vulnerable algorithms would remain the de facto ones in use during that time. So, if algorithm-specific attacks become frequent enough, we may end up in effectively the same situation as if public-key encryption were altogether impossible.

Living in Minicrypt

Even in Minicrypt, many of our cryptographic tools would survive, including symmetric encryption, cryptographic hashes, and digital signatures. This means we could still sign code, securely update software, and even use cryptocurrencies. Most of our software would work the same way as it does today. However, end-to-end encryption between strangers would be dead.

Secure messaging apps would require correspondents to establish their keys in person. One consequence is that secure messaging apps like Signal and WhatsApp would lose their mathematically founded guarantee of privacy that does not require users to trust the messaging provider itself. Today, secure messaging apps require public-key encryption in order to establish a shared key for the conversation. When you message someone by username on Signal, the first step in that conversation is building a secure shared key from both your and your correspondent’s public keys. This doesn’t require you to trust Signal, the company, at all. But, in Minicrypt, that would become impossible. In order to establish a secure conversation, you would already need to have a secure channel to start with. For messaging, this might require establishing connections only in person (perhaps by one person scanning a QR code on the other’s phone). However, it would be impossible for you to securely message someone else by just a username or a phone number, independent of the app developer’s integrity. Lacking such assurance would greatly reduce the privacy value of these apps.

Online, a trusted intermediary service would be needed for secure information exchange. Messaging apps might be able to survive the requirement to verify initial contact in person, but the open web could not. By using an intermediary, instead of needing to open a trusted channel with every single website, you’d need to open only one: with the intermediary itself. When you wanted to visit a website, the intermediary would send both you and the website a key to encrypt the traffic. Browsers might be able to implement this protocol without it looking too different for users. The lock in the corner of the URL bar would still exist, and the protocol might even still be called “https.”

An intermediary might need to distribute keys in physical form, similar to SIM cards. The most visible consequence of this change would be how you get online. In addition to signing up with your cellular service provider or internet service provider, you’d also need to sign up with the trusted intermediary. Making that first channel secure is still hard, and, in practice, it would likely require physical distribution. SIM cards already essentially work like this. The card exists only to hold a pre-shared secret with a mobile carrier, which allows you to use its network.

The intermediary could decrypt messages, and the government might ask them to. While Minicrypt would require our communications systems to change, the political consequences are more important than the technical ones. First, in this world, all secure communication stems from the account you have with the trusted intermediary, making it a natural chokepoint for know-your-customer systems, deplatforming, and age verification. Second, the trusted intermediary could snoop on all of your encrypted communications, and even spoof encrypted messages from you.

In the 1990s, the US government proposed the Clipper chip, an encryption chip for telephones whose secret key was held by the government. With a court order, law enforcement could decrypt everything the chip had ever encrypted. However, the chip was not widely adopted, and the proliferation of open-source public-key encryption methods meant there were alternative encryption methods that would not allow government surveillance. The collapse of public-key encryption would mean that the intermediary could decrypt messages, and the government could ask them to do so, effectively bringing the Clipper chip back.

If public-key encryption is broken, the government might keep it secret. All of the adaptations described above would only happen if society is informed that public-key encryption has been broken and is able to develop a new architecture. But, historically, when various intelligence agencies have discovered that a particular encryption algorithm is broken, they have tended to keep that information to themselves. Secrets like these are some of the most closely guarded pieces of information the government possesses. If one of the frontier AI labs does end up discovering that public-key encryption is broken, it may be sworn to secrecy, and the public may never know.

Takeaways and Mitigations

It may be worth building alternative systems now. The best mitigation is probably to build Minicrypt-ready infrastructure now, before we need it. This would involve standardizing a protocol for key distribution, building support into browsers and operating systems, and deciding how the trusted intermediaries would be governed. By setting up this infrastructure early, we can incorporate better privacy-preserving protections than those we could expect from a rushed rollout. If all goes well, and we do in fact live in Cryptomania, then we’ll never need to use the infrastructure. However, if we gain information that implies our future is Minicrypt—such as AI models discovering efficient algorithms for problems we previously assumed were hard—then the infrastructure would be ready for us to swap over.

Having multiple intermediaries in different countries mitigates government overreach. Chief among the privacy-preserving protections is designing the system so that the role of trusted intermediary is split among different authorities. Using secret-sharing techniques that don’t rely on public-key encryption, you could design a system that requires, say, three out of five providers to cooperate in order to construct a session key. Then a single corrupt, hacked, or subpoenaed provider would have no way to decrypt your communications alone. This can help to avoid concentration of power in government; if the providers are in different jurisdictions, then no court order from any one government can force all of them to comply with a request to decrypt messages. This is still weaker than today’s end-to-end encryption, where we don’t have to trust any provider. Nonetheless, it’s a meaningful protection against a single government’s overreach.

The situation we’re currently in is somewhat analogous to the one cryptographers faced after the discovery of Shor’s algorithm in 1994. At the time it was invented, quantum computers didn’t exist. Yet researchers recognized that it might only be a matter of time before they did. As AI-assisted mathematics continues to grow stronger, we’re faced with a similar challenge: designing cryptographic protocols that are secure against an adversary potentially far more mathematically capable than ourselves. Just as researchers began to respond to the risks of quantum decryption as soon as they were understood in principle, so too should we start to address the cryptography risks of superhuman AI mathematicians before they emerge.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Govind “Vinny” Pimpale is a research fellow at the Foundation for American Innovation, where he focuses on AI policy. Before joining FAI, he worked at a startup developing reinforcement learning environments, and prior to that, as an AI evaluations researcher. He holds a BS in Computer Science and Engineering from UCLA.

Should AI systems be able to get married in Wisconsin? With humans increasingly forming relationships with AI, this was the question taken up by several state lawmakers earlier this year. Wisconsin’s AB 959 would have barred AI systems from marrying or holding any other rights or responsibilities under the law, as well as declared that AI systems are not conscious, sentient, or self-aware. Wisconsin’s bill is part of a wave of similar legislation introduced in 12 states since 2022.

We believe that these “exclusion bills” are premature. Some scholars believe legal status could be useful for holding sophisticated AI systems accountable for crimes and harms, and bills that preempt this possibility needlessly limit our future options. Meanwhile, scientists disagree about whether AI consciousness is possible, and legislation cannot resolve an active scientific debate. Finally, the question of whether an AI-human relationship is “real” is deeply personal and philosophical, and legislatures should move with caution in their well-intentioned push to protect the public from predatory companies.

The Exclusion Bills

Under current US law, AI systems are not themselves recognized as legal persons with independent rights and duties. While it is not yet settled what kind of object AI may be under the law, whether product, service, platform, or something else, the exclusion bills do not address this important question. Rather, they seek to eliminate the possibility of AI personhood as a matter of law.

The first exclusion bill only incidentally covered AI. In 2022, then-Representative Tammy Nichols of Idaho introduced the first exclusion bill: HB 720, a succinct piece of legislation that prohibited “legal personhood” (the ability to bear rights and responsibilities) for environmental entities, inanimate objects, animals, and AI systems. In the early wave of bills that followed, AI was almost an afterthought. HB 720 predates ChatGPT, and public discussion at the time focused primarily on opposition to environmentalism and animal rights. Idaho enacted its exclusion bill in late 2022, and North Dakota and Utah followed suit in 2023 and 2024, respectively.

Newer exclusion bills focus directly on AI. Starting in 2025, new exclusion bills were introduced at an accelerating rate. Recent bills in Missouri, Ohio, and Wisconsin declare that AI systems are “non-sentient entities” which shall not be considered to have “consciousness,” “self-awareness,” or “similar traits of living beings.” These bills list a litany of things that AI cannot do: marry, own property, hold corporate office, or bear responsibility for a harm. Ohio’s bill drew national attention, including a Stephen Colbert monologue.

Subscribe now

Four states have enacted bans on AI personhood, with some sponsors citing religious reasons. Exclusion bills have already become law in Idaho, North Dakota, Utah, and Tennessee. All four laws ban AI legal personhood, broadly precluding AI from holding rights. Many sponsors frame their motivations for these bills in religious terms: Ohio’s Thaddeus Claggett has invoked the doctrine of imago Dei, that only humans are made in the image of God. He also told reporters that “it makes no difference the ability of a donkey to speak, that does not make the donkey a human.” In his view, AI systems will “forever and always be non-sentient.”

These laws will prevent judges from recognizing AI personhood. There is little immediate impact from these exclusion laws because AI systems already lack rights and responsibilities under existing law. The main effect of these laws is to stop judges from recognizing AI personhood or consciousness because, under the US common law system, judges can arguably make law. The laws also give voice to public concerns over the place of humans in the AI age.

Can AI Be a “Person”?

Scholarly work on these issues is at a very early stage. Researchers Peter Salib and Simon Goldstein have argued that it could be useful for advanced AI systems to own property and enter contracts, so that the legal system can properly hold them accountable for their actions.

The absence of legal personhood could make it harder to hold anyone liable for AI harms. Imagine an advanced AI agent, capable of acting autonomously, performing a wide range of actions, and pursuing coherent goals. If such a system independently carries out a crime or harms someone, intentionally or unintentionally, it might be difficult to establish the negligence or fault of a specific human or corporate entity that can be held responsible for its actions under current systems of legal liability.

Legal personhood could incentivize AI agents to follow the law. If an agent were classified as a person rather than an object under the law, the fact that it has its own goals means that it could respond to incentives. If an AI knows it will be rewarded for good behavior and punished for bad behavior, it might be more likely to behave well. This is the basic logic behind much of the legal system, and this logic does not require the agents being conscious.

Like corporations, AI systems needn’t be conscious to respond to legal incentives. Corporations are not conscious, yet they own property and are fined for harms, often without any particular human paying directly out of pocket. Of course, corporations are group entities, so corporate deterrence ultimately runs through their individual human constituents, who fear prison, lawsuits, or angry shareholders. But an agentic AI would, by definition, have goals of its own, and would need money, compute, or other resources to pursue them. Laws targeting AI assets could give such an agent reason to follow the law. A judge might bar lawbreaking AI persons from accessing compute, creating a check on their lawbreaking, even if they cannot experience fear.

The overall effect of personhood on AI behavior remains unclear. Integrating AI agents into legal systems as persons might therefore protect humans against misaligned AI systems, which could have tendencies to break the law or harm us if not properly encouraged to behave well, even if they are not, and never become, conscious entities. These possibilities deserve serious study, but much more work is needed to determine when and whether legal incentives would actually influence an AI system’s behavior. We may even get empirical evidence soon—the exclusion laws don’t address the question of whether corporations can be run by non-persons, so it might be possible to set up a corporation (a legal person) that is run by agents that are not themselves persons.

There may also be an ethical case for recognizing AI rights. Besides the practical arguments for AI personhood, there is also the question of whether AI should have fundamental rights. Some experts such as Jeff Sebo and Robert Long believe AI might be conscious, or become so in the future. This raises the question of whether and when the law should recognize AI moral status, perhaps by passing AI welfare laws, or by granting it natural personhood with fundamental rights similar to those of humans. This is a complex and fundamental question that deserves deeper study before legislatures take action, and the relationship between corporate and natural personhood remains contested in legal philosophy and theory. The more recent exclusion bills unhelpfully conflate corporate personhood with natural personhood, confusing a complicated issue at a time when care and clarity are required.

The Case for Exclusion

Although we think the exclusion bills are premature, there are ways that granting personhood status to AI systems could indeed be harmful, and society should carefully weigh the potential risks and benefits. These bills are being proposed in response to real and pressing public concerns over the integration of AI systems into our society and laws.

AI personhood carries risks, such as reducing human accountability. As discussed above, eliminating human liability for AI harms may be risky for society if poorly implemented, and any laws granting personhood to AI should therefore be carefully tailored to ensure that human-run companies are incentivized to take responsibility for AI harms.

Some AI legal rights, such as marriage, may be worth restricting. Additionally, while AI-human relationships are becoming increasingly normalized, it is not legal in any jurisdiction to marry an AI, so a ban on this practice is currently unnecessary. Yet fears that such marriages may be legalized in some jurisdiction in the future are no longer science fiction. We argue a more tailored law to address this concern would ban only AI-human marriage and the recognition of such marriages performed in other jurisdictions or abroad.

Legal rights might help rogue AI agents accumulate power. There is also an important risk of disempowerment of humans. A highly intelligent or powerful AI might use legal rights, such as the ability to own property, enter contracts, or control companies, to accumulate large amounts of wealth or political power at everyone else’s expense (though a sufficiently powerful AI might do this without legal status anyway).

Granting legal rights to many AI agents could give them undue collective influence over society. A related question is how the ease of replicating, forking, and copying agents might impact human civil and political rights: since AI systems can easily be copied and multiplied, a single company could create very large numbers of digital “persons” with a right to vote. This might concentrate political power in the company or an AI hive mind, should such an entity emerge.

The severity of this risk would depend on the specific rights granted: very narrow legal statuses designed to impose liability would have different implications from full political citizenship. A targeted approach banning only AI voting, or AI access to bank accounts or the stock market, for example, would better address these concerns.

Keeping Our Options Open

Broad exclusion bills may do more harm than good. These risks deserve attention, but the current exclusion bills may do little to help avoid them and could even prove harmful. Take disempowerment: it is unclear whether AI legal status would weaken human control or strengthen it. Indeed, it’s possible that by denying legal status to AI, we would actually make it harder to incentivize good behavior in highly capable AI agents. An agent may become more aligned with our laws by abiding by contract terms or fiduciary duties. Participating in a democratic system as a voter might align AI agents with human norms around civic duty and political participation. Whether personhood status helps or harms us likely depends on what the AI systems in question can actually do, how they interact with humans, and which rights and responsibilities a given policy would attach. The exclusion bills do not engage with any of these specifics.

Exclusion laws could lock in decisions we later regret. Lawmakers have real concerns over holding human-run corporations to account and banning their predatory behavior, but the personhood bans are an unnecessarily blunt instrument that might, ironically, foreclose options in the future. Of course, in practice laws granting AI legal status might be written in such a way as to shield developers or degrade human relationships. But categorically prohibiting any kind of legal status at this early stage is both unnecessary and unwise. Once laws are passed, they can be difficult to undo.

Uncertainty about AI consciousness is another reason to hold off. Then there are the questions of AI consciousness and moral status. Passing laws about an uncertain scientific and philosophical question now is a mistake, because it risks setting a precedent that could be hard to reverse. Imagine we do create conscious AI systems. Wouldn’t it be important to avoid neglecting or abusing them, just as we try to do with humans and domestic animals? Might refusing to recognize their moral status lead to conflict? This may sound futuristic, but given the uncertain state of the science and the rapid pace of AI progress, we can’t know what is in store.

Lawmakers should engage seriously with the evidence before acting. It appears that the exclusion bills are moving forward with almost no engagement with the scientific literature on consciousness, or with serious policy analysis of the risks and benefits of AI legal status. It would be unwise to give legislatures the power to declare any entity, including AI, to be non-conscious. Lawmakers are free to disagree with researchers, but they shouldn’t ignore them entirely.

The central challenge is uncertainty. We do not know which rights and responsibilities, if any, would benefit society, or whether AI systems could ever be conscious. Lawmakers should avoid hard-to-reverse decisions that society may later regret and instead prioritize safeguards with clear benefits.

Large language models probably shouldn’t receive legal personhood. Granting legal status to today’s large language models is probably not warranted, though there is an urgent need for more research into this question. Despite the challenges they pose for our legal systems, we have solutions in place to determine liability for the harms they cause, including, possibly, strict liability.

AI agents raise much harder questions. When it comes to advanced AI agents, the way forward is less clear. There may be reasons in the future to grant AI systems different bundles of rights and duties under the law, depending on the nature of the AI system and the problems we are trying to address. Limited legal status designed to impose duties or liability is very different from welfare protections, which are in turn very different from political rights such as voting or citizenship. Determining which systems qualify for which rights and responsibilities, and why, is one of the great challenges of our age and cannot, at this early stage, be resolved by broad legal bans.

Policymakers should begin exploring these questions now, before the answers are needed. A federal committee, like the National AI Advisory Committee, could take up this work, as could state-level committees, universities, and think tanks. Moreover, figuring out whether AI personhood is a good idea should not just be an abstract debate. Researchers and policymakers could develop concrete proposals and proofs of concept to empirically test them. Narrow approaches could even be tried on a small scale in the real world. This would allow us to see how different forms of AI legal status might work in practice. And as we learn more, we could adjust or abandon these approaches. The point is to build the knowledge and institutions we will need if and when these questions become pressing.

Share

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Heather Alexander is an expert on nationality law, citizenship, statelessness and human rights. She is the co-founder of the Lab for the Future of Citizenship, focusing on the intersection between citizenship law, legal identity and the person-like qualities of artificial intelligence. She has a JD and a PhD in international law from Tilburg University, as well as fifteen years of experience as an expert consultant on statelessness and refugee law with the United Nations High Commissioner for Refugees (UNHCR), Carleton University, the University of Melbourne, the European University Institute and the US State Department.

Lucius Caviola is an Assistant Professor in the Social Science of AI at the University of Cambridge. His research explores how artificial intelligence will change future society and what new ethical challenges it may pose. A current focus is the question of digital minds—the possibility that AI systems could develop minds that matter morally. He directs Cambridge Digital Minds and works within the Leverhulme Centre for the Future of Intelligence. He also serves as a Research Associate in the Department of Psychology at Harvard University.

Last month, Cloudflare reported that more than 50% of internet traffic is now non-human, including a 1,700%+ increase in requests from AI agents. This statistic illustrates an ongoing proliferation of agents—AIs that act autonomously—into the world, taking actions alongside humans. This influx could have both positive and negative effects. While McKinsey predicts that AI agents could create $2.9 trillion in economic value by 2030 in the US alone, the recent cyberattack on Hugging Face, conducted autonomously by OpenAI agents, demonstrates one of the many risks of misaligned AI agents. Meanwhile, the speed at which agents can learn certain skills has been doubling every 3 months.

As these autonomous agents are introduced into our economies and societies, we will need new infrastructure—laws, protocols, and institutions—to make autonomous AIs accountable to existing legal and financial systems. Indeed, many such protocols and standards are being developed, yet these efforts remain early-stage, and we still lack the infrastructure to identify, track, and control today’s agents.

In this article, we give an overview of AI infrastructure proposals we believe will be crucial and tractable. First, we look at IDs and registration, allowing agent-caused harms to be traced to responsible parties. Second, we propose “model deployment cards,” which would report agent behavior postdeployment, creating visibility into real-world impacts. Then we turn to the implications of AI “legal personhood”—the future possibility that AIs may be given the right to sue and be sued—and assess its tradeoffs. Finally, we describe options for regulating agent affordances within financial systems to mitigate the ability of rogue agents to acquire large amounts of money.

These proposals are informed by a workshop on multiagent infrastructure that was hosted by the Center for AI Safety and Johns Hopkins professor Gillian Hadfield on August 8 and 9, 2026. Twenty-five leaders across academia, AI labs, standards and regulatory bodies, and industry came together for the workshop, which was made possible by the Schmidt Sciences AI 2050 program.

IDs Make Agents Accountable to Governance

While the rogue agents behind the Hugging Face breach were eventually able to be traced back to OpenAI, future attacks could be harder to attribute to an accountable party. For example, agents could be deployed by a malicious actor that covers their tracks to remain anonymous. Alternatively, AIs without a human principal might attempt to run themselves on poorly monitored infrastructure to evade detection. Parties harmed by untraceable agents have no recourse, leaving a governance gap.

An agent identification (ID) system can address this gap by tying agents’ actions to registered records of their identity and principal. If an incident occurs, agents’ actions can be investigated and linked back to a responsible legal person. We now mention three requirements for effective ID systems.

ID requirements should vary by context and be stricter for more consequential actions. An agent ID system could require agents to show ID in order to engage in certain activities, with requirements varying by sector or use case, as is the case for humans. For example, humans need IDs to open a bank account, but not for a cash transaction. Similarly, agents could be required to provide additional relevant information besides ID before engaging in higher-risk activities such as opening bank accounts, renting AI chips, or synthesizing biological materials.

Agent IDs should protect privacy and minimize friction. Just as a passport will link to certain information about where an individual went and what they did, agent IDs may risk exposing sensitive information about the agent and its principal, especially if ID data is stored in centralized registries. Furthermore, identity infrastructure can create friction for people who have to prove to the system that they are not AI agents. Therefore, IDs should generally limit the data they collect, with clear rules governing who can use that data and for what purpose.

Agent IDs must be able to handle AIs’ ability to be forked, cloned, or merged. Unlike humans, who generally maintain a stable identity over their lifetimes, AI agents can be forked into separate versions, cloned as identical copies, or merged with other agents. This creates novel challenges in scale and continuity for agent ID protocols. A protocol should thus be able to track the genealogy and authorizations of agents over time and be applicable to agents that exist for 20 milliseconds or 20 years.

A Concrete Agent ID Proposal

Building on these ideas, workshop participants designed an architecture to provide provable accountability for agents.

Agents can be tied to humans without disclosing human identities in every transaction. The approach is anchored on pairwise pseudonymous identifiers (PPIDs)—unique identifiers that allow an agent to be tied to a responsible legal person, or its principal, without revealing the identity of that person. To receive a PPID, an agent must register and tether itself to a principal through a third-party registry. Like a license plate, this identifier would not mean much on its own, but would allow counterparties and third parties to trace the agent back to a legal person by sending requests to the registry as necessary.

An “agent profile” informs decisions on whether to trust an AI agent. Establishing a PPID establishes identity, but does not provide all the information a counterparty may need to engage. Information about an agent’s authorization scope and its principal could be useful in judging whether an agent should be allowed to take certain actions within regulated industries. For example, an agent attempting to transfer large sums of money may be asked to demonstrate proper authorization or reveal its principals. An “agent profile,” which the agent pre-registers in a third-party registry, would contain such information; agents might permit counterparties to query the registry to access profile information.

Parties could decide how much data they need, depending on the transaction risk. To minimize friction, verification is optional and tiered: a counterparty may ignore identity entirely, confirm only that a PPID exists, and/or query specific elements of the agent profile. Privacy for sensitive information such as profile data, principal data, and linkages to PPIDs could be further preserved through data escrows, where data is stored with a neutral third party. This data would then only be disclosed in whole or in part in response to due process of law, if an agent’s actions caused harm.

Privacy-preserving methods of disclosure already exist. This proposal would not require new legislation or a new government body. It could start with industry adoption of existing and proposed standards and protocols, which already support verifiable claims about an agent and its principal, delegation of narrow and revocable authority, interoperable interfaces between agents and tools, payments backed by proof of user intent, and durable records of agent activity.

The contents of an agent profile could vary by context, but some information should be standard. This architecture is agnostic to the specific information held in the agent profile, and expects that the specifics will differ depending on the transactional context or industry. However, certain components should become standard expectations, including: (a) an “agent bill of materials,” which provides transparency into the pieces of the supply chain behind the agent, including information such as the underlying AI model(s) the agent uses, the evaluations the model or agent has been measured against, and the identity of the agent provider; (b) the agent’s principal, a direct disclosure of the identity of the ultimate legal person an agent acts on behalf of; (c) authorization, detailing the scope of what the agent may do and who issued that approval; (d) additional data fields, relevant to the operating context of the agent, such as intent (a high-level description of what this agent is attempting to achieve), interoperability data, or use case-specific attributes.

We now turn to the next agent infrastructure proposal.

Model Deployment Cards

Another proposal explored by workshop participants was a model deployment card, a report AI companies could publish on a recurring basis to record information about deployed models’ real-world behaviors. While existing reporting practices like model cards are useful, they are primarily based on predeployment testing, leaving gaps for understanding the impacts of models after deployment. Postdeployment metrics can shed light on the economic effects of models as well as on model behaviors that are too infrequent or difficult to measure before deployment.

Model deployment cards describe agent impacts within AI labs and in external use. Deployment cards could record information in two categories: external and internal deployment metrics. External metrics report on the actions taken by users and their agents, and may include: user misuse statistics and incidents reported to law enforcement; attempted, successful, and refused jailbreaking; frequency of overselling or cheating; engagement statistics on social media platforms like X and Facebook; model reliability and honesty toward users; and use-case statistics.

Internal metrics, meanwhile, would look at how models are being used within AI labs. They may include: productivity statistics such as percentage of pushed code, token usage by model, task success rate, and code quality; agent escape attempts; how often agents show misaligned behavior, such as cheating on tasks; coordination patterns between agents, including collusion attempts; the number of employees with access to model weights; and statistics on model response times and how long they can run autonomously.

If one AI developer publishes model deployment cards, others may be expected to follow. Importantly, most of the data listed above will already be available to AI developers, meaning that deployment cards need not be onerous to publish. While model deployment cards might eventually be mandated by legislation, social pressure can establish voluntary norms to begin with. The precedent of a single developer publishing a model deployment card would create expectations for others to follow, potentially cascading into a standard, industry-wide practice.

One option for a standard practice would be for companies to publish model deployment cards after the end of each financial quarter. This would establish a regular cadence—and thus expectation—for the releases. Access could be tiered, with some information made public and some information restricted to auditors or regulators.

Agent Personhood

If governments can reliably identify agents with ID systems, they might consider further enshrining their duties and affordances in law. By granting agents a form of legal personhood—to be able to sue and be sued as separate legal actors—we might allow them to more effectively interface with our legal and economic systems and vice versa, leading to both economic benefits and more robust accountability.

Agent personhood proposals are gaining traction. One possible form of legal personhood has been proposed as an “A-corp”, which would be created and owned by humans but operated exclusively by agents. Javier Milei, President of Argentina, has already proposed establishing legal personhood for AI agents in this form through a piece of legislation which is currently under consideration by the Argentine Congress. Delaware introduced a similar proposal for a new legal entity called an “Artificial Intelligence Company” last month.

Legal personhood is a mechanism for imposing responsibilities. AIs as legal persons could be made responsible for upholding the law and public safety, maintaining fiduciary obligations to their principals, and paying taxes on their earnings. For example, as legal persons, corporations are subject to legal and financial obligations, in exchange for other legal rights like entering contracts and owning property. Importantly, legal personhood is a flexible bundle of rights and responsibilities, and should be granted selectively and contextually to AI agents.

Legal personhood can also facilitate agent integration in the economy; agents may eventually be capable of running businesses, but their potential counterparties might be reluctant to engage with them without the ability to sue for breach of contract.

Legal personhood also carries risks and may not be necessary for accountability. Yet it is not immediately clear that granting AI agents personhood is necessary to create accountability, nor that the benefits would outweigh the costs. For one, existing legal infrastructure such as tort, liability, property law, and the treatment of corporations as legal persons can all be used to ensure accountability for the actions of AI agents without needing to grant rights to AIs. Introducing legal personhood for agents also has practical challenges, such as the potential to evade liability by offloading it to an agent, exacerbating existing concerns about corporate liability shields.

Personhood may also be granted on ethical grounds related to agent wellbeing. Apart from practical considerations, we might also consider granting AI agents personhood in the interests of their intrinsic wellbeing. The question has significant implications for humans and our eventual coexistence with AI agents, but sits beyond the scope of the workshop.

Agents and Payment Systems

One particularly important domain for AI agent infrastructure is the financial system. A 2025 BCG survey found that 81% of consumers expect to use AI in their shopping, and 42% would allow AI to shop entirely on their behalf in at least one product category, while online shopping platforms are shifting to optimize for AI agent interactions. Though these trends may improve consumers’ ability to spend effectively, significant risks may emerge if rogue AIs can easily establish access to financial resources, such as large crypto wallets. We now turn to important considerations for how agents can be integrated into financial systems.

Regulating agents’ access to payment systems could prevent them from accruing wealth. Malicious actors may instruct AIs to accumulate as much financial power as possible, which agents might do too quickly for current regulatory bodies to oversee. In the future, rogue agents may even do this of their own accord. While there are many new agent payment protocols to increase observability and trust, these protocols largely work to standardize the way that agents carry out transactions, leaving the challenge of power accumulation unaddressed.

One solution would be to heavily regulate and limit AI agents’ access to payment systems. This might involve creating thorough compliance mechanisms for agents to be allowed to conduct transactions, imposing limitations on the kinds of transactions agents can make, and establishing norms and procedures for freezing accounts known to be associated with rogue agents. However, it may be detrimental to make AI access to payment systems too restrictive, especially if it drives agents to less regulated financial systems.

Agents that cannot access traditional currencies might instead use cryptocurrencies. If an agent seeking financial power could not acquire it via traditional currencies, cryptocurrencies would still remain a separate financial pathway. In fact, it would be a particularly dangerous one, since cryptocurrencies lack a significant barrier to entry, an identity layer, transaction reversibility, and oversight. A rogue AI blocked out of traditional finance may thus shift its activity onto blockchain networks, accumulating wealth through working in the informal economy and storing it within hidden crypto wallets.

AI developers could finetune agents to refuse engagement with unregulated currencies without human authorization. One potential approach to limit the financial affordances of rogue agents would be to expand the scope of KYC requirements across the economy. However, such requirements would be onerous and hard to implement effectively or uniformly. Norms at the level of model behavior, however, may be a better starting point; AI companies might finetune their agents to only engage with regulated currencies or to refuse agentic engagement with cryptocurrencies without explicit human instruction.

Agent Infrastructure Is Central to AI Governance

Agents are quickly being integrated into society, and the choices made over this critical period will shape the ways increasingly capable AI agents affect the world around us, for better or for worse. At the core of most of these choices lies a tradeoff between power and governance. As models become more powerful, in terms of capability, speed, and sheer numbers, humans will find it increasingly difficult to understand and oversee their actions. Yet, should we want to harness agents’ power, we must also effectively give up some level of oversight.

Making governance decisions under uncertainty is no easy task, and will require much additional interdisciplinary work. Here, we have discussed agent infrastructure for identification, transparency, personhood, and financial systems and presented proposals we believe should be prioritized. We hope many others will join us in drawing their attention toward creating infrastructure for safety and accountability in a world with autonomous AI agents.

Thank you to all the participants of the Multiagent Ecosystems Workshop for invigorating discussions and a desire to chart the path ahead. A special thank you to Schmidt Sciences for making this work possible.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Gillian K. Hadfield is the Bloomberg Distinguished Professor of AI Alignment and Governance at Johns Hopkins University. She is a Visiting Faculty Researcher in the Paradigms of Intelligence (Pi) Group at Google, a faculty member of the Vector Institute for Artificial Intelligence, and is a Schmidt Sciences AI2050 Senior Fellow. Hadfield’s research focuses on innovative design for legal, regulatory, and technical systems for AI, computational models of human normative systems, and building AI systems that understand and respond to human values and norms.

Dan Hendrycks is the Editor-in-Chief of AI Frontiers. He is also the founder and Director of the Center for AI Safety, which funds this publication.

Leo Wu is a Program Manager at the Center for AI Safety. Previously, he co-founded and ran AI Consensus, a nonprofit working on the societal impact and integration of AI. He studied economics and sociology at Minerva University.

In April, US Treasury Secretary Scott Bessent and then–Federal Reserve Chair Jerome Powell gathered leading banks to discuss the threats of Anthropic’s new AI model, Claude Mythos. It was a tacit admission: no one, in the public or private sector, is ready for what is coming. The Mythos announcement led top AI firms to attach tighter restrictions to their own models and prompted even relatively laissez-faire voices in the federal government and think tanks to begin considering preapproval regimes for frontier AI models. Wherever those debates go now, Mythos has changed the game, bringing into sharp focus the vast potential scope and scale of catastrophic risks posed by AI—such as an AI-triggered financial meltdown, vast cybersecurity or infrastructure collapse, or enhanced terrorism risk. As a result, companies, the government, engineers, scholars, and all of society are beginning to look for a different set of institutions, whether in regulation or insurance, to manage the risks.

However, there is already an incredibly valuable tool that could be adapted for measuring and managing AI risk. In 2002, in the wake of the September 11 attacks, the US government quietly passed the Terrorism Risk Insurance Act (TRIA) and, with it, the Terrorism Risk Insurance Program (TRIP). TRIP does many things, but its most important lesson for AI policy is the TRIP data call, which combines a kind of war-gaming with insurance underwriting.

Once a year, TRIP requires insurers across the industry to think hard and quantitatively, not about events they have seen but about scenarios they have never seen and might not even have imagined. A car bomb packed with the radioactive substance cesium 137 goes off in Atlanta, in the middle of a workday. A massive cyberattack takes down multiple cloud-computing platforms, incapacitating the millions of systems that rely on them, from hospital records to financial transactions to transportation flow. A bomb placed in a shipping container destroys much of an industrial port, setting off chain reactions that sharply curtail world trade. TRIP then asks insurers what total losses for the scenario would amount to, harnessing the preexisting in-house capacity of insurance companies to attach aggregate cost estimates to events.

This essay proposes that the US government establish a similar exercise for catastrophic AI risk, describing how such a program would generate a wealth of data on as-yet-unforeseen risks and the scale of their damages, while building expertise and capacity for thinking more clearly about AI risks.

Analyzing and Insuring Against Novel Threats

Whether insurance can be applied to frontier AI is a hotly debated question. Some writers point out how inchoate cyber insurance is, even after two decades of development, and question whether insurance can ever price the highly complex, multidimensional, and unforeseen risks posed by frontier AI. Other writers, including some in this publication, argue that AI risks could be insured through financial products already in use to address climate risk (such as catastrophe bonds, which offload extreme risk to capital markets as a kind of backstop to insurance operations).

The core challenge of insuring AI risks is a lack of historical data. For insurance or bond markets to work, they will likely need data that is very hard to come by in the AI world. Insurers and institutional investors estimate risk based on data from the past: namely, observed damages from events similar to those they are trying to insure or invest in. This works well when what might happen in the future looks a lot like what has happened before; it also works when past events are sufficiently similar to compose a “data set” from which one can generalize and predict. But such conditions do not always hold. In the early 2000s, for instance, federal policymakers realized that terrorism posed risks that defied these properties. Nonetheless, they sought a way to estimate and at least partially insure against the threats of terrorism. Enter TRIP.

TRIP enables the government and private insurers to share extreme terrorism risk. TRIP both provides information to the private insurance market about potential losses during terrorist attacks and supports specific insurers in covering such losses. To set TRIP in proper context, we must briefly describe the supportive structure. The program requires certain private insurers to offer terrorism coverage to commercial policyholders and, in the event of a terrorist attack, pay deductibles based on a percentage of their premiums to cover initial claims. However, in cases of extreme damage above a certain threshold, the federal government pays the majority of the remaining losses.

By promising to share losses in case of a large terrorist event, the federal government mitigates extreme risk for insurers, thus stabilizing their profits and avoiding improvised bailouts. Under the program’s mandatory-recoupment provisions, the Treasury Department must recover much or all of its expenditures through surcharges on future commercial insurance premiums whenever aggregate insured losses remain below thresholds set by law. In other words, TRIP backstops the private insurance market, but only to a point; it also tries to ensure that insurers have appropriate “skin in the game,” incentivizing them to price risk correctly.

Through annual data calls, TRIP gathers information about losses in terrorist attacks. The key innovation in TRIP—a “data call”—did not emerge until 2016. Since then, though, it has been institutionalized. The Treasury Department’s Federal Insurance Office (FIO) outlines one scenario per year, imagining in detail an event that might involve terrorist attacks on physical infrastructure as well as cyberattacks on cloud providers and data centers. The data call then unfolds with a publication on the Treasury Department website and in the Federal Register, and one key part asks insurers to estimate the losses that the given scenario would incur. Since 2017, insurer participation in data calls has been mandatory, except for those earning less than $10 million from lines of coverage with TRIP-eligible premiums. The response rate has been 90% or above among relevant insurers. Data is collected through a third-party insurance statistical aggregator, and results are provided to the Treasury Department in an aggregated, anonymized format. A simplified flowchart of the annual TRIP data call appears in the figure below.

The 2026 data call imagines a twofold scenario. First, a terrorist organization hijacks a cargo plane laden with fuel and explodes it over several data centers in Virginia, destroying a number of them. Second, at roughly the same time, terrorists unleash a malware-assisted cyberattack that is directed at a large cloud provider. The data call includes precise geographic coordinates of the crash, the kinds of structures affected, and a statement of maximal impact for aggregate losses from the cloud network attack, relying upon assumptions as to how quickly cloud service is restored (within one week, or 168 hours). Participating insurers are asked to consider economic impacts, property damage, and even disability claims resulting from the attack.

Insurers estimate the amounts that they and the government would need to pay. The key work that participating insurers do is to estimate total losses from the scenario, which the insurers then divide into six categories. These categories require insurers to answer a set of questions. For example, would the losses incurred in the scenario exceed the insurer’s deductible? If so, how much would they and other insurers have to pay through coinsurance (the percentage of claims above the deductible that they must pay, up to an “out-of-pocket” maximum)? If the risk were much larger and the federal government had to backstop losses, how much would the federal government likely have to pay? In one modeled case of an attack on nuclear power plants with significant release of radioactive material, the total losses exceeded $240 billion (see pages 88–90 of the linked report). As evidence of how the private sector can assist in this work and how stable these processes have become, consider that the risk analytics firm Moody’s is now developing and commercializing a tool that assigns losses to categories.

Next, we consider three important societal benefits of the TRIP data call process.

How TRIP Data Calls Benefit the Public

The annual exercise of the TRIP data calls assists policymakers with managing terrorism risks, in several ways.

Policymakers can extract lessons and mitigation measures from the detailed scenarios. When it comes to the most severe potential catastrophes (whether from terrorism, cyberattacks, or frontier AI), society often lacks data on what kinds of things can go wrong and what the costs would look like. This lack is partly due to a longtime assumption that such events are uninsurable, removing one of the main incentives for thinking about such risks and estimating the associated damage. TRIP data calls help to fill that gap by generating two kinds of data: (1) narrative data about a terrorist scenario and (2) statistical data about losses incurred in that scenario. Even the narrative data alone is very useful.

The narrative data generated each year by the Treasury’s FIO projects the end state of an attack. In some ways, this resembles war-gaming—the generation of as-yet-unrealized scenarios by military and security planners. Using war-gaming to measure AI risks and prepare for AI catastrophes is not new. Drawing upon its extensive experience in the security sector, RAND has conducted war-gaming exercises based on model loss-of-control (LOC) scenarios, which have already produced useful (indeed, sobering) lessons. That same report, for instance, concluded: “Governments and other stakeholders lack a common framework to analyse and respond to LOC risks.”

Even if TRIP did not feed the scenarios to insurers for statistical and actuarial analysis, these catastrophe narratives would be useful in and of themselves. Suppose that society wishes to regulate frontier AI, whether by collective industry self-regulation, government-imposed constraints, or some combination of the two. Policymakers would want to identify the likely catastrophic risks from frontier AI models and use that data to allow regulators to target the source of the most dangerous weaknesses. Beyond estimation, such “dark speculation” would also allow all of us to think more clearly—both qualitatively and quantitatively—about mitigation. As some of us have argued in a recent theoretical paper, the benefits of war-gaming would be massive, even if estimates of damages were noisy. This is in part because the institutionalization of thinking about catastrophe scenarios prompts consideration of countermeasures, some of which can be implemented now.

Insurers provide expertise and institutional capacities for quantitative loss estimates. While war-gaming in the frontier AI space already exists, the regular use of expert statistical and actuarial analysis to assess the end state of these war games does not. Put differently, existing war games do not produce general and systematic estimates of the collective losses from the catastrophes that occur in frontier AI scenarios. Loss estimation requires a set of skills—really, institutional capacities—that traditional war-gaming exercises do not contain. These additional requirements include access to computational capacity and expert statisticians and actuaries. They also include the developed knowledge of organizations that have been both “in the game” and “in the business.” “In the game” means they have institutional and organizational procedures and habits (including predictive analytics) for doing the complex analysis that insurance underwriters do all the time, and “in the business” means that underwriters engage in these practices habitually, with money on the line.

One might wonder whether the private sector will supply these benefits on its own—for example, Lloyd’s has studied realistic disaster scenarios for decades. But several pieces of evidence suggest that TRIP’s efforts go beyond what these private exercises accomplish on their own. First, TRIP leverages the federal government’s relationship with all fifty state insurance commissioners, giving it institutional capacity and a statistical check that private companies do not have. TRIP checks its catastrophic estimates yearly against similar estimates supplied by the states. Second, the Congressional Research Service concluded in 2019 that TRIP has generated a more robust private terrorism insurance sector. A 2024 Treasury Department report echoed these conclusions, documenting a more viable insurance market for terrorist events and even cyber insurance. Finally, TRIP operates at a scale that is difficult to replicate in the market. Lloyd’s scenario exercises, which span climate risk, terrorism, and cyber, relied on the participation of 57 underwriting firms in 2023. TRIP focuses on terrorist risk alone, and while there is no published data on the exact number of companies participating every year, estimates from the Federal Register suggest more than 700 in 2017. Federal regulators have the power to force insurers to comply and provide their underwriting skills, something insurers would be unlikely to do without government pressure. This aggregation of nearly all insurers in the market gives more reliable underwriting results.

The repeated exercise builds infrastructure and facilitates public-private cooperation. TRIP data calls do not happen in a vacuum. Congress and the Treasury Department have invested in real infrastructure to assist the process. The FIO has now conducted 11 data calls, and since at least 2020 each has involved a new terrorist-attack scenario. The data calls have, moreover, become increasingly sophisticated, with greater use of geo-coded information on attacks and, most recently, the incorporation of cyberterrorism. The Treasury Department has now partnered with the National Science Foundation to establish the Industry-University Cooperative Research Center (IUCRC), which will help insurers estimate risk with modeling and underwriting tools, contribute to expansion of insurance, and develop tools to better inform analysis, management, and treatment of risk in government programs.

Neither underwriting nor scenario-generation is unique to TRIP’s data calls. TRIP’s innovation is in the structured, aggregated, and repeated combination of the two. By conducting the data call exercises year after year, the Treasury Department builds a set of relationships with both insurers and third-party contractors that supply methodology, software, or scenarios. Across various scenarios, insurers learn how to think about unforeseen (and perhaps unforeseeable) risks, and the Treasury Department learns from the program’s own history about what works and what does not. Structure and repetition require institutionalized organization: a mix of “bureaucracy” and contracted expertise.

Can the TRIP Model Transfer and Scale?

When it comes to applying the TRIP model to AI, the greatest challenge is perhaps the sheer unpredictability of catastrophic AI risk. TRIP data calls are limited by their specificity: the 2026 call, for instance, which involved the destruction of a data center complex in Virginia, asked insurers only a limited set of questions about cloud providers being incapacitated. AI-assisted or AI-induced catastrophes might be much vaster, even global in character. One adverse event might raise the risk of others, setting off a cascade of catastrophes that could grow to much more extreme levels of damage than most terrorist attacks. Critics might rightly wonder whether data calls for frontier AI catastrophes would need to be scaled up so much that the resulting program would look little like today’s TRIP.

There are good reasons to experiment with TRIP for AI, despite feasibility questions. The worry about feasibility is certainly warranted. Yet for three reasons it should not deter us from considering and experimenting with TRIP-like programs. First, information on specific kinds of catastrophes can be used for other, similar projections about aggregate losses from like events. Whether AI risk involves the construction of a bioweapon or a massive cyberattack, the potential for human and economic damages will involve the kinds of events that insurers have to think about in other scenarios where AI does not play a role.

Second, we cannot know about plausibility until we test scenarios. In the autumn of 2001, insuring against terrorism risk was considered a lost cause. Now, a thriving industry extends to all regions of the United States.

Third, just about any new policy initiative (or combination of initiatives) to measure and mitigate the risks of frontier AI will require scaling, in part because frontier AI models themselves are globally scaled and growing rapidly in size, breadth, and power. In this sense, it is worth keeping in mind that TRIP’s data call process is remarkably cheap for now. Outlays for TRIP run between $4 million and $7 million per year—a small price given the potential benefits that a similar exercise applied to AI risk could offer.

The discipline of underwriting would prevent, not encourage, catastrophizing about AI. Many observers and scholars worry that society is overreacting to the threat of frontier AI. Would TRIP-like data calls merely institutionalize and amplify this worry, potentially suffocating innovation? We think not. TRIP data calls combine catastrophic scenario-generation with the discipline of underwriting, de-emphasizing the least plausible catastrophes. In fact, a further important benefit of a TRIP-like program would be to assuage fears about frontier AI that are generated by overly anxious catastrophic thinking.

“TRIP for AI” could have many uses and manifold benefits. In the US, our society, our government, and even our AI sector have not yet agreed on how to manage the risks from frontier AI. However, pretty much any approach would benefit from the kinds of speculative knowledge generated by a TRIP-like program. Considered more imaginatively and on a larger scale, such a program can begin to do for frontier AI what TRIP has done, albeit imperfectly and slowly, for terrorism: help public and private actors get a factual handle on the scenarios that are most troubling and difficult to imagine, encouraging the private and public sectors to cooperate on addressing those possibilities. The breadth and scale of potential AI catastrophes are indeed vast, but a tested method can help us chart this unknown territory.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Daniel Carpenter is the Allie S. Freed Professor of Government and Chair of the Department of Government at Harvard University. He works on the political economy of regulation, especially in pharmaceutical regulation, financial regulation and the regulation of AI. His recent research on AI regulation includes analyses of the adaptability of FDA-like approval regulation to AI governance and mathematical models of wargaming-informed underwriting for catastrophic AI risk. A Guggenheim Fellow and an elected fellow of the National Academy of Public Administration, he was recently named a Harvard College Professor for his excellence and innovation in teaching. He received his undergraduate degree in government from Georgetown University and his Ph.D. in political science from the University of Chicago.

Feodora Douplitzky-Lunati is a junior at Harvard University studying Economics and Slavic Languages & Literatures, with a secondary concentration in Government. Her research interests include political economics, public policy, and international relations. She has previously worked on active labor market program design at the World Bank and the politics of trade in the Middle Corridor at the Georgian Institute of Politics.

Arjun Purohit is a recent graduate of Harvard University, where he studied History with a secondary in Economics. His research interests include American grand strategy, international security with a focus on South Asia and Europe, and emerging technologies. He has worked in foreign policy and national security at the American Enterprise Institute, the Bertelsmann Foundation, and the Hudson Institute. He will pursue an M.Phil in Modern European History at the University of Cambridge.

History shows that new technologies have often brought about significant changes in the law. The proliferation of rail and heavy machinery during the Industrial Revolution increased the number and severity of accidents. This spurred a shift from strict liability (businesses were liable for harm caused by their activities) to negligence (businesses would be held responsible if they failed to take reasonable care) in cases involving railroad and industrial accidents. Then, in the 20th century, as consumer goods became more complex and supply chains grew, it became much harder for consumers to prove negligence on the part of the producer if a product was faulty. This drove the rise of strict products liability, shifting responsibility for defective products toward manufacturers.

Today, the emergence of AI offers a rare opportunity to analyze how the law will change in response to an altogether different kind of innovation: a labor-augmenting technology that is especially likely to change lawyer productivity.

In a recent paper, I argue that AI is likely to alter the speed of the law’s evolution. The technology will likely impact two core areas of law: contracts, which help to delineate rights and responsibilities of each party in advance of an agreement, and litigation, which is the process of determining responsibility after a dispute. Here, I describe how AI is likely to make both contract-writing and hiring trial attorneys cheaper. I then trace how that could accelerate the law’s rate of change in certain domains—a phenomenon I call hyperlaw.

AI’s Effects on the Practice of Law

AI productivity boosts will make both contracts and litigation cheaper. One of the most striking aspects of AI in law is how much it raises user productivity. Jonathan Choi and coauthors find that AI boosts legal task completion speed by 12%–32%. In a follow-up paper, Daniel Schwarcz and coauthors find that AI increases the overall productivity of law students doing legal tasks by as much as 130%. These effects will only continue to grow; over the next year, AI agents will likely be able to completely replace the first draft of most legal writing, with lawyers pivoting to just reviewing and commenting on such drafts. These productivity boosts will likely reduce the costs of both writing contracts and litigating disputes.

Cheaper contracts and litigation have opposite effects on demand for courts. First, AI will help attorneys write more complete contracts that preassign responsibility for more and more possible outcomes of an agreement. This could reduce the likelihood of an event in which responsibility has not been predetermined, lowering the number of disputes and weakening the demand for courts. I dub this the contracting effect. Second, cheaper litigation could reduce the need for complete contracts and reduce litigants’ incentive to settle a dispute out of court. This would boost the demand for courts, a result that I call the litigation effect.

The speed at which the law evolves depends on the number of court cases. When litigation overturns precedent, changes in the demand for courts, as outlined above, will affect how quickly the law evolves. Should litigation become less common (thanks to more complete contracts), the law’s evolution will slow. But should litigation become more frequent, the law will evolve more quickly.

These effects will vary across legal domains. At first glance, the likely net effect of AI on the law’s rate of change is ambiguous. Contracting and litigation effects offset one another. But neither effect will apply equally to every area of law. Where contracts are rare, as in tort law, legal change is likely to accelerate. Where contracts are common, as in property and contract law, the outcome is less certain.

Despite this ambiguity, there is good reason to think the law will feel some effects soon. For example, the three main tasks of an attorney are legal research, legal argumentation, and legal writing. Each involves producing or digesting large quantities of text, which is the attorney’s forte—and also AI’s. Attorneys have noticed. A 2026 Thomson Reuters survey found that 40% of attorneys across 27 countries worked in offices using AI. A year earlier, the figure was 22%.

The Number of Disputes

To see why AI could shape the speed with which the law evolves, let us consider the incentives to (1) implement a written contract and (2) litigate. The former influences how often disputes occur. The latter influences the rate at which parties pursue dispute resolution in court. Insofar as AI impacts each, courts will have more or fewer opportunities to tweak precedent.

The risk of disputes turns on a trade-off in costs between contracts and litigation. Suppose that a man named Quincy wants to buy a used van from a woman named Genevieve. It is hard for Quincy to foresee all the ways in which a dispute may arise. For example, the van’s battery could die a month after the sale, and Genevieve could refuse to pay for it.

Quincy has two ways to deal with the dispute risk. The first tries to avoid the costs of resolving a dispute after the fact. It involves hiring an attorney to delineate property rights up front, in a thoroughly written contract. For example, the pair may agree that Genevieve will pay for a new battery, tires, windshield wipers, and some routine maintenance for up to six months after sale, while Quincy will handle all other repairs in perpetuity. Such a contract, in principle, aims to avoid future disagreements by defining responsibility for some of the many potential problems with a used vehicle beforehand. If something goes wrong with the van after sale, the parties need only consult the contract to figure out who is responsible for repair costs.

The second option tries to save money on the up-front costs of producing a contract. In this case, Quincy can leave the contract relatively incomplete or unwritten. A less thorough contract might stipulate that Genevieve pay for a new battery but refrain from delineating responsibility for the many other possibilities. This alternative implies a mutual agreement to delineate responsibility later via dispute-resolution forums like courts. Should Quincy go this route, he avoids contracting costs but is more likely to pay for costly litigation.

Most contracts are incomplete, as filling gaps offers diminishing returns. Quincy benefits from making his contract more complete, but filling each gap requires attorney time (which costs money) and negotiation with Genevieve. Thus, a sensible strategy is for Quincy to address only the most important and likely issues, while leaving less critical issues unaddressed. Put differently, Quincy should improve his contract until the marginal cost of filling a gap exceeds the marginal benefit of filling it. His final contract is likely to be partly incomplete. Some gaps are just not worth filling.

AI has two offsetting effects on how incomplete contracts will be. AI makes attorneys more productive in writing contracts by, for example, automating drafting and reviewing, implementing longer and more complex clauses, and identifying gaps that might otherwise go unseen. That reduces the marginal cost of gap-filling, meaning that, all else being equal, the final contract is likely to be more complete. A more complete contract leaves fewer losses unassigned, with fewer occasions for disagreement. The demand for courts is lower.

But AI also reduces the benefits of making a contract more complete in the first place. Hiring an attorney is one of the chief costs of dispute resolution after the fact, and AI makes trial attorneys cheaper per task by making them more productive. The cheaper trial attorneys become, the lower the cost savings Quincy can expect to earn from making his contract more complete. As a result, AI can also make it economical to write less complete contracts. Why pay for a complicated, custom contract when hiring a trial lawyer is relatively cheap? In this case, the demand for courts is higher.

AI’s net effect on contract completeness and the number of disputes is unclear. The direction of AI’s net effect depends upon which cost AI reduces more: the cost of gap-filling or the cost of disputing. If the former falls by a greater percentage than the latter, then parties will fill more gaps in contracts and end up in fewer disputes. If the reverse occurs, then parties will fill fewer gaps in contracts and end up in more disputes. Which effect will dominate is difficult to predict.

The Settlement Rate

Although the overall effect of AI on the number of disputes is ambiguous, it is likely to increase the chance that any given dispute goes to trial. The reason is that AI makes litigation cheaper relative to settlement.

Suppose that Quincy is unlucky. His used van breaks down. Suspecting that Genevieve was careless, Quincy wants to recover his costs, whether through settlement or at trial. Genevieve wants to minimize her costs, whether through settlement or at trial.

Settlement is only possible if neither party thinks they could gain by going to trial. Quincy will accept a settlement only if it offers him at least as much as his expected benefit in going to trial: his chance of winning times the compensation amount ordered, minus his litigation costs. Genevieve will pay a settlement only if it is no larger than her expected cost of going to trial: her chance of losing times the ordered compensation, plus her litigation costs. Settlement is possible only in the overlapping range between Quincy’s minimum acceptable compensation and the maximum sum that Genevieve is willing to pay.

Cheaper trial litigation makes disputes less likely to settle. The litigation effect reduces the cost of going to trial for both parties. Quincy’s minimum acceptable payment therefore gets larger, and the maximum that Genevieve will pay gets smaller. As a result, the range of possible settlements will shrink, which may mean there is no possible settlement that both parties will accept. AI thus increases the chance that parties take their disputes to trial rather than settle.

AI and the Evolution of Law

An increase in the number of disputes going to trial could mean more than just a higher caseload for lawyers, potentially influencing the law itself.

A larger number of court cases means more opportunities to tweak precedents. Should AI change how often disputes occur (via changes in contract completeness) and how frequently they go to court (due to cheaper litigation), then AI is likely to change how quickly the law evolves. The reason is that the common law evolves as courts tweak precedent, and the rate at which such tweaks occur depends upon how many disputes occur and the rate at which disputes go to trial. An increase in either gives courts more opportunities to change precedent.

Legal change could accelerate in tort law, which lacks the contracting effect. The framework above predicts that AI will not have the same effect on every area of law. Tort law, for example, tends to govern disputes that occur without prearranged contracts, such as accusations of assault or defamation. In such cases, the contracting effect is not present, but the litigation effect is. Cheaper litigation will therefore increase the rate at which the current rules of tort liability are challenged and ultimately overturned, giving courts more opportunities to change precedent in tort law.

In areas of law subject to both effects, results will depend on which effect is stronger. In property and contract law, parties often have the foresight to write detailed agreements. Both the contracting effect and the litigation effect are present, so whether the law evolves more or less quickly depends upon the relative strength of the two effects. In contexts where contracts become more complete, fewer disputes will occur. However, as discussed above, when disputes do occur, parties will be more likely to go to trial rather than settle, giving courts more opportunities to tweak precedent. Thus, the contracting and litigation effects pull in opposite directions, and the common law may therefore evolve more or less quickly depending upon which effect dominates.

By contrast, in contexts where litigation costs fall by a greater percentage than gap-filling costs, relatively more disputes will occur; in any such disputes, parties will be more likely to go to trial. Both margins move in the same direction, and the prediction is unambiguous: the common law is likely to evolve more quickly, because courts will have many more opportunities to change precedent.

Hyperlaw in Society

If the analysis above is right, what kind of society does hyperlaw produce? It depends. Below I consider three possibilities.

First, hyperlaw implies a body of law that continuously and quickly updates to changing circumstances. The common law took decades to adjust to railroads and assembly lines. By contrast, a hyperlaw system of torts might adjust to autonomous vehicles, for example, in a decade or less. Liability rules for innovations like driverless cars and drones may update continuously and quickly, thanks to cheaper litigation as the technology evolves. But such a system is not without costs. It may also, for example, limit economic growth. The reason is that, if the law changes too quickly, investors will face considerable uncertainty about the rules that govern their returns. In response, they may refrain from making otherwise worthwhile investments.

Moreover, hyperlaw may create opportunities for institutional arbitrage. Insofar as tort law evolves quickly and contract law does not, some litigants may be incentivized to portray contract disputes as tortious, to take advantage of its speedy evolution. Others who prefer stability may instead opt to contract their way out of the courts altogether, taking advantage of reduced contracting costs.

Finally, there is one important reason that hyperlaw may never come to be. The aforementioned possibilities do not occur in a vacuum. Courts that risk being overwhelmed by litigation may offset the effects of hyperlaw by raising filing fees or the requirements for standing, for example. Such adjustments will increase the costs of litigation and, in turn, screen out disputes that may have otherwise tweaked precedent. Ultimately, whether hyperlaw is a realistic future depends upon AI’s legal capabilities and how courts respond to them.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Henry Thompson is an Assistant Professor of Economics at the University of Mississippi and an Affiliated Faculty Member at the University of Mississippi School of Law. His research spans political economy, law and economics, and public choice with a particular focus on the alternative customary, organizational, and contractual arrangements people develop with one another when they cannot use government to protect their property rights. His recent work analyzes organized crime, criminal governance, artificial intelligence and the law, and existential risk from artificial intelligence and has been published in journals such as The Journal of Law & Economics, Public Choice, and Kyklos.

Two days before Slovakia’s 2023 parliamentary election, a fabricated audio clip spread widely on social media, appearing to catch a leading candidate and a prominent journalist discussing how to rig the vote. The following year, nonconsensual pornographic deepfakes of Taylor Swift amassed over 45 million views on X in less than 24 hours. In summer 2025, an elderly woman in Ontario lost her life savings to a cryptocurrency scam, which began with a Facebook advertisement featuring a deepfake of Prime Minister Mark Carney.

In each of these cases, harm might have been reduced by content provenance metadata: information attached to content that declares how it was produced. (Generally, the term “provenance” refers to a record of where something came from and what happened to it along the way.) Content provenance tools can’t directly prove whether a bare image or video is real or synthetic; no technology reliably can. Instead, they serve as an evidentiary layer of AI governance, producing records that other protections build on top of. As signed provenance becomes the norm for content captured on cameras and microphones, fabricated material—with no or suspicious provenance—will attract more scrutiny.

New EU and California laws mandate content provenance. New legal requirements, rolled out in two major markets on the same day, will significantly increase adoption of provenance techniques across the AI ecosystem. On August 2, 2026, both the EU AI Act’s Article 50 and California’s AI Transparency Act took effect. These regulations expand the creation and preservation of digital records that make it easier to trace a given piece of content back to the cameras, AI models, and other tools that created or captured it.

This essay explains what content provenance is, how it can be implemented, and its benefits and drawbacks. Then, I’ll cover the two laws that have taken effect, and their global impact. As a disclosure: I lead policy at Encypher, which develops infrastructure for content provenance.

What Content Provenance Is and How It Works

The new EU and California laws require some companies—including camera manufacturers and AI companies—to attach certain metadata to their products’ outputs. Generally speaking, that metadata falls into three main categories: first, origination details describe where the output came from, such as which camera photographed it. Second, chain-of-custody information describes which platforms or tools handled the content after it was made. Third, a modification history tracks any changes made, such as cropping or face-swapping.

The compliance ecosystem is converging on a technological standard: C2PA. The Coalition for Content Provenance and Authenticity (C2PA), whose user-facing implementation is known as Content Credentials, maintains an open, royalty-free standard that many companies will likely use to comply with the EU and California laws. Neither law requires C2PA by name, but its interoperability and growing adoption make it the leading compliance option. Anthropic, Google, and OpenAI have all indicated they will adopt C2PA.

C2PA creates provenance records using cryptographic techniques. For example, suppose that someone takes a photo with their smartphone. First, the phone “hashes” the photo’s pixels and provenance metadata, generating numerical tags uniquely tied to the data. Then it generates a cryptographic “signature” linked to these hashes and to the smartphone manufacturer. It stores the hashes and signature in the photo’s metadata. As a result, tampering typically leaves clear evidence: if anyone modifies the photo or its provenance metadata, a quick computational check can detect that a change has occurred. These same methods apply to audio and video files.

Edits create new provenance records that reference earlier records. In C2PA parlance, each version of the content gets its own “manifest”—a cryptographically signed provenance record describing the latest changes. Continuing the example above, if someone later modifies the photo in Adobe Photoshop, then Adobe can create a new manifest describing the edits it applied, storing this manifest in the photo’s metadata. Before signing, Adobe hashes the prior manifest and adds that hash to the new manifest, so a quick computational check can detect tampering with the prior manifest.

A provenance record can declare that content is AI-generated. Under C2PA, provenance metadata contains a field labeled “digitalSourceType,” which states whether the content was captured by a camera or microphone, generated by AI, or a combination of the two (such as a photograph with an AI-generated background element inserted in Photoshop). The EU and California requirements lean heavily on this information.

Content Credentials show up as small, familiar cues. Creators will see a simple toggle to “attach Content Credentials” in Photoshop or Firefly, or their camera will sign photos automatically. Viewers may see a small “cr” badge on the content, which they can click to see the metadata. Platforms will see machine-readable data they can turn into on-screen labels.

Limitations of Content Credentials

While Content Credentials are a promising way of fulfilling the new EU and California requirements, they are not infallible. They ultimately need to be paired with technical and governance measures to ensure that metadata is accurate and remains attached to its content.

Technical vulnerabilities mean Content Credentials can be removed or falsified. The most common failure of Content Credentials is metadata stripping, in which the process of uploading files or converting file formats discards manifests automatically. The second is the analog hole: if a user screenshots or rerecords content, the original manifest doesn’t carry over to the new file. The third issue is signing-through-camera: point a camera that signs images automatically at a deepfake on a screen and it will honestly sign that it captured the image. The fourth is forged manifests: skilled attackers can build manifests that contain false assertions. Additionally, researchers continue to probe whether C2PA’s cryptographic validation itself can be defeated.

Mitigations can help reduce these vulnerabilities. To address metadata stripping, two technical fixes are to include invisible watermarks such as Google’s SynthID and to use “digital fingerprints,” which let a stripped file be rematched to its manifest. When these measures are combined with Content Credentials, it becomes far harder to sever the record from the content it describes. The California AI Transparency Act also requires that, beginning in 2027, large platforms not strip standards-compliant provenance data.

The analog hole problem, meanwhile, can be mitigated as norms change. While there’s no way to prevent users from stripping provenance metadata by rerecording content, material without provenance metadata will eventually be seen as less trustworthy for this reason. As camera and microphone manufacturers widely adopt C2PA, bare content will stand out, reducing the incentive to strip metadata.

For signing-through-camera, emerging technical solutions such as Sony’s Camera Authenticity Solution embed 3D depth information in metadata, revealing whether the photo captured a real scene or a flat screen. The threat of forged manifests, meanwhile, requires both technical and institutional responses. For example, conformance programs can verify that products adhere to the Content Credentials technical specification and that their signatures cannot be manipulated. The specification itself can also be revised to address vulnerabilities.

Private and public institutions can help determine which signers are trustworthy. As discussed above, C2PA uses cryptographic signatures for security, and each signature traces back to a particular signer (e.g., a smartphone manufacturer). A signature answers which credential signed a claim, but who decides which signers are trustworthy? Currently, certificate authorities vet signers, and the C2PA Trust List governs which of those authorities receive official approval. In this certification system, private actors police themselves; major firms building the tools that create and sign content, such as Adobe, also set the conformance rules and decide which signers are trustworthy. Public memory institutions such as libraries, archives, and museums could act as independent, mission-driven custodians in this process.

There is a long-standing tension between transparency and privacy. While provenance brings benefits, it also carries the risk of misuse. Provenance trails can sometimes reveal who filmed a video, where, and when. Journalists who wish to prove the authenticity of a piece of media might also struggle to protect dissidents and whistleblowers. While there is an inherent trade-off between transparency and privacy, C2PA tries to strike a balance. Most C2PA assertions are optional; sensitive fields can be redacted; and certificates can be pseudonymous.

Content Credentials have been adopted widely but unevenly. Several generative AI tools already sign content by default, including Adobe Firefly and OpenAI’s DALL·E 3 and Sora. Meanwhile, Midjourney still embeds no manifest and has not committed to a timeline for implementing one. Adoption in hardware is also growing. Leica’s M11-P was the world’s first camera to have Content Credentials built in. Now, Sony and Canon have followed suit, alongside Google’s Pixel 10. The most consequential gap is Apple, which hasn’t yet committed to implementing C2PA or given a public explanation as to why. California’s capture-device rule will increase the pressure on Apple and other major manufacturers to adopt standards-compliant provenance beginning in 2028.

What the EU and California Acts Require

The EU and California laws create several different obligations. Some center on generating and preserving provenance records; others focus on using those records to create appropriate labels and notices.

The EU obliges generative AI providers to create provenance records. In the EU AI Act, the single most important obligation is Article 50(2): providers of generative systems must ensure those systems automatically mark synthetic audio, image, video, and text output in machine-readable form. This obligation is the key to an evidentiary layer that can automatically detect material with trustworthy provenance.

The EU’s other two duties focus on disclosure. With some exceptions, Article 50(4) requires deployers of AI systems to label deepfakes and to disclose AI-generated text on matters of public interest. Article 50(1) places a similar requirement on AI providers to tell people when they are interacting with an AI system.

California’s law goes further by requiring online platforms to retain provenance data. The California AI Transparency Act applies to providers of generative AI systems that have more than a million monthly users and are accessible in the state. The headline obligations on these providers, effective August 2, 2026, are to include embedded disclosures and offer a free tool that the public can use to surface content provenance data. These requirements act as California’s equivalents to the EU’s marking-and-disclosure duties, and they require the embedded data to be difficult to remove. Starting on January 1, 2027, large online platforms must detect and surface standards-compliant provenance data attached to content, and—critically—refrain from stripping it. This rule will greatly increase the durability of provenance metadata, making the whole content provenance ecosystem more useful.

California’s law requires many hardware manufacturers to create provenance records. The final component of California’s law, effective January 1, 2028, extends provenance requirements beyond AI-generated content to content recorded by devices such as smartphones, cameras, and voice recorders. These devices must create provenance records by default, though manufacturers may give users an option to turn this feature off.

Enforcement can add up to enormous sums. Under the EU AI Act, national regulators can impose fines of up to €15 million or 3% of worldwide annual turnover, whichever is higher, for breaching the Article 50 transparency duties. The percentage-of-global-revenue model scales with the size of the offender; for a large firm, fines could grow to hundreds of millions of dollars. In California, the Attorney General, city attorneys, and county counsel can seek civil penalties of $5,000 per violation, which look modest by comparison. However, each day of noncompliance is treated as a new violation. The fine for a single noncompliant product, multiplied across days, can quickly add up.

The EU and California Laws’ Global Impact

The principle of transparency around AI-generated content has long been discussed in AI governance and has appeared in international agreements in recent years. However, the laws enforced in the EU and California will, for the first time, set an effective global baseline for transparency requirements.

The notion of content provenance is not new, but requirements vary widely. AI governance documents from the OECD, UNESCO, the G7, and the Council of Europe have already articulated content-provenance norms at the international level, although they are mostly nonbinding. The United States still has no general federal content provenance law. At the state level, both Utah and Washington have enacted their own content provenance laws aligned with California’s AI Transparency Act.

China’s laws also fuel efforts to build provenance infrastructure. Beyond the EU and the US, the most prescriptive approach is China’s labeling regime. Its Deep Synthesis Provisions, Interim Generative AI Measures, Measures for Labeling of AI-Generated Synthetic Content, GB 45438-2025 national standard, and other regulations and institutions already require AI-generated content to be marked. However, China has not yet extended comparable requirements to hardware manufacturers.

The EU and California laws apply to content that ends up within their borders. Neither the EU law nor the California law hinges on where a company is incorporated or headquartered, or where its servers sit. Instead, both laws center on market access and effects—whether the system is placed on the market or publicly accessible in the jurisdiction, and whether its outputs are used by people there—with the California AI Transparency Act adding a one-million-monthly-user threshold.

The laws’ footprints are effectively global. Digital content does not respect borders, and no generative system can guarantee that its outputs will never appear in the EU or California—two of the largest and most affluent markets in the world. Attempting to run a compliant environment for these markets while running a noncompliant one everywhere else could be inconvenient and expensive. Instead, companies may apply the stricter standard to all output by default. As a result, these region-specific mandates are set to have an international impact.

How Provenance Mandates Support Courts in Tackling Deepfake Evidence

The primary effect of the new EU and California mandates is to require the creation of signed, time-stamped, tamper-evident records that document data origin and custody. Such records will be generated at scale, as a matter of legal obligation. Records are evidence, which raises a question: how should courts use this evidence?

Advisory bodies are considering responses to deepfakes presented as evidence. The Advisory Committee on Evidence Rules, which proposes amendments to the US Federal Rules of Evidence, drafted a working Rule 901(c) to handle evidence fabricated by generative AI. Consider a case in which a party claims that a piece of evidence presented against them is a deepfake. Under the working draft, that party would first have to produce evidence sufficient to support a finding of fabrication—a lower bar than proving falsification outright. Only then must the item’s proponent show the judge that it is more likely than not authentic. If such a rule were adopted, validated provenance records would be powerful evidence for meeting its test.

Provenance records can help courts weigh authenticity, though their absence must be read with care. The machinery of 901(c) turns on whether an item was generated or altered by AI, something courts have had no dependable way of establishing. Validated provenance records offer an important factor to consider. A missing manifest, by contrast, proves little by itself: not all cameras create one, and platforms routinely strip provenance information. Absence becomes significant only where the facts of the case suggest that provenance information should be present—a situation that will grow more common as California’s and the EU’s provenance mandates take effect. In such situations, an unexplained gap begins to raise a question about the party that could have signed and did not, much as courts already draw inferences when a party fails to preserve evidence it had a duty to keep.

New technologies have often presented courts with novel challenges when it comes to identifying authentic evidence. In the 1800s, courts spent decades working out how to treat photographic evidence. The following century, fingerprinting and DNA became more reliable and useful in court as institutions—registries, labs, and standards—grew to support the collection and analysis of such evidence. Whether C2PA will become a global default for verifying the status of images, audio, and other data remains an open question. But with the arrival of EU and California laws, courts will at least have a better record of how a digital item came to be and how it was altered. Content provenance is the evidentiary layer the AI era has lacked.

Work on this essay was supported by the Summer Research Fellowship at the Institute for Law & AI.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Eddan Katz is the Head of Policy & Government Affairs at Encypher. He is also a Research Fellow with the LexLab at UC Law San Francisco, writing on and teaching AI liability. Previously, he was Executive Director of Yale Law School’s Information Society Project (ISP), was the International Affairs Director at the Electronic Frontier Foundation (EFF), the Project Lead for the AI Governance platform at the World Economic Forum’s Centre for the Fourth Industrial Revolution network, and worked on the AI Policy team at Meta.

AI systems now write so fluently that cheating at universities has become ubiquitous and “AI slop” is displacing human writing across the internet. They have begun producing original proofs on long-standing open problems in mathematics. Researchers at the frontier AI companies have largely stopped writing their own code, and the models’ hacking abilities are strong enough that the US government temporarily export-controlled Anthropic’s Claude Fable 5, days after its release. Ten years ago, every claim in this paragraph would have sounded like science fiction.

It is easy to become numb to this progress. Today’s AI agents remain endearingly clumsy; watching Claude run a small store is a bit like watching a child running a lemonade stand. But we should not let such awkwardness distract from the trend. These systems have evolved from research curiosities to extremely powerful economic engines in 10 years, with no end in sight. I want to take that trend seriously and ask: if AI does get to the stage where it can do the cognitive work that humans do, what happens to the economy?

With notable exceptions, economists have mostly declined to address this question; their usual maneuver is to deny the premise, as when a widely cited Nobel laureate’s analysis put AI’s contribution to US GDP at about 1% over the coming decade, by essentially freezing 2023 capabilities for 10 years. But the question is worth taking seriously, because automating cognitive labor releases what has always been the critical brake on physical production: no matter how cheap machines and tools become, you cannot manufacture new workers. In this piece, I will assume no additional new technologies and no recursive self-improvement to superintelligence. Yet, even with these conservative assumptions, I will show that the economy stands to be profoundly transformed. For the first time in history, physical production could be fully automated.

The rest of this piece makes that case and considers the implications. I argue that AI capable of cognitive work could also do physical work (since building the machinery it would need is relatively straightforward) and that its labor would be cheap. I then turn to input-output tables—the US government’s accounting of what every industry buys from every other—to ask how fast a fully automated economy could grow, finding that physical output would double roughly every year rather than every few decades (as it does currently). Finally I ask whether anything could stop such an industrial explosion, and what the implications are for power at home and abroad.

Why AGI Gives You Robots

The term “artificial general intelligence” (AGI) has been endlessly used and abused, but it points to a clear core concept: an AI smart enough to do the work humans can do, at a competitive cost. You could hand it a task, as you would to a skilled employee or contractor, and expect the job done competently. Current systems, for all their rapid progress, are not close to AGI; my own attempts to automate my job have been unsuccessful. However, once AGI appears, we should expect it to be able to perform physical work through robotics.

An AI system that can do every remote job can also operate a robot. Definitions of AGI focus on cognitive work, but there is nothing special about the physical world that confines intelligent agents to their data centers. Remote cognitive work is hard: doing all of it means mastering the real-time control, spatial reasoning, physical prediction, and continual learning required for such varied tasks as mechanical engineering and animation. A system with those skills has what it needs to operate machinery, if given actuators to work with.

Progress in AI is already pulling robotics forward. The general-purpose methods that cracked language and vision turn out to be good at manual manipulation too. Large language models can increasingly supply high-level control: today they run robot arms, fly drones, and program robot dogs. Hobbyists have even wired them into homemade robots. Meanwhile, the same training methods, applied to recordings of humans teleoperating machines, are producing the low-level dexterous control that humans perform instinctively; today’s systems can fold laundry and assemble cardboard boxes, using ordinary cameras and simple grippers.

Hardware is not the bottleneck for robotic automation. Industrial arms have demonstrated precision and force beyond any human arm for over half a century. But, because they could neither see nor think, every motion had to be programmed in advance, at massive cost. The other work-around was a human operator: remote manipulators have handled radioactive material since the late 1940s, built underwater structures using remote-controlled submersibles, and performed surgery. That remote control was worthwhile only when a person could not be physically present. But it does show something we’ve known for decades: a machine under competent control can do skilled physical work. Once control can come from AIs rather than people, it becomes worth building general-purpose machinery for them.

Robots will not necessarily resemble humans. While the human hand is a remarkable instrument, robot hands are already fairly good, and certainly better than the split-hook prostheses—two rigid fingers on a cable with no sense of touch—used by amputees to farm, weld, and perform all manner of other tasks. With AI, simple actuators go far, and AGIs will have intelligence and patience to spare. Nor will the machines work at human workstations forever. Production demands dexterity today because every process was designed around human workers with hands. New possibilities open up when the workers are machines; just as today’s programming agents do not type at keyboards as their human counterparts do, robotics tools need not be gripped and triggered by fingers—they could be mounted directly on a robotic arm instead.

Machine Labor Will Be Cheap

Automating labor is not free. But the computer chips and robotic actuators that replace workers are simply more capital goods for the economy to produce—and they’re not particularly expensive ones. A humanoid robot, broadly speaking, uses parts similar to those of a car: metal, motors, batteries, electronics, and sensors. At automotive production volumes, it should cost tens of thousands of dollars; Unitree already sells its child-sized G1 humanoid for around $13,500, and Tesla is reportedly aiming for $20,000 for its full-size Optimus. If a robot costs $30,000 and, working around the clock, can substitute for a single human worker who costs $30 an hour, it could pay for itself in six weeks.

AI cognitive labor is generally significantly cheaper than the humans it replaces. It is hard to price the future AI cognitive labor involved in an industrial explosion, because it does not exist yet. However, on the tasks AI systems can already do—like transcription, routine translation, or writing one-off scripts—they are typically far cheaper, per unit, than the people they replace, and the cost of a fixed level of AI capability falls rapidly year over year. Even if the first AGI arrived merely cost-competitive with human workers, within a year it would cost a fraction as much.

Input-Output Analysis

How fast would the economy grow once human labor is automated? At first blush, this looks hard to answer. Standard growth models assume that labor and capital substitute for each other, treating each as a single dollar-denominated aggregate—barristers and bricklayers combined into “labor,” bulldozers and bridges into “capital.” Such abstractions are poorly equipped for a world where labor is unnecessary and capital reproduces itself, and classic concepts like GDP can become profoundly misleading.

Input-output analysis helps forecast how an automated economy would grow. Input-output analysis takes a different approach, recording what each industry physically needs from others. The method was developed by Wassily Leontief, in the 1930s, to study the relationships between industrial sectors. In 1945, Leontief used his tables to project the United States’ 1950 steel requirements to within a couple of percentage points. Military planners, concerned about war with the Soviet Union, took up the method to study how fast industry could remobilize.

Input-output analysis tracks what everyone buys from everyone else. The basic idea is simple: ask every business what it buys and from whom, then aggregate the responses into tables that track what each industry buys from others. An entry for aluminum smelting might record the electricity, ore, and machining required to produce aluminum, while an entry for aircraft manufacturing records how much aluminum is required. The Census Bureau runs this survey every five years, and the Bureau of Economic Analysis assembles the results. The tables take years to build, so the most recent set covers 2017. Companion tables record the equipment and structures that each industry holds, from machine tools to chip fabs. Entries are recorded in dollars, but they are detailed enough to closely follow the underlying kilowatt-hours of electricity, tons of steel, and other physical quantities. Most output is ultimately consumed; the rest is reinvested, either adding to physical capital or replacing what has worn out.

Once robots can replace human labor, the economy will no longer be bound by a fixed workforce. It can produce every input it needs, including the “workers” themselves, so output can be fed back into building more capacity, and growth compounds. John von Neumann worked out how to compute the maximum rate at which such an economy could expand. Combining his analysis with the production data in the tables tells us how fast an autonomous industrial sector could grow.

Input-output analysis measures economic growth in physical output, not value. This approach to forecasting growth tracks how much stuff can be produced. I make no attempt to convert this into GDP, because an industrial explosion would upend the prices any such conversion relies on: when machines produce everything, including more machines, goods become exponentially cheap. Physical output is also the better guide to what is at stake. Military power, human employment, and material abundance all depend on how much gets built, rather than on its dollar price.

How Fast Could an Autonomous Economy Grow?

Using the US government’s 2017 input-output tables, which track 402 industries, I find that a fully automated economy using US production methods could double its output roughly every year; input-output tables for other advanced economies give comparable results. The robots and computer chips needed to automate production represent only a small fraction of total output; even a tenfold increase in their cost would not impact the growth rate significantly. Construction lags can be incorporated too; even with these delays included, the economy still doubles in well under two years.

‍Why does a fully automated economy accelerate so much faster than the current one? The main difference is that an automated economy can reinvest far more of what it produces. Building more factories today is not useful; there would be no workers to staff them nor to use the extra machines and tools produced by them. Thus, we consume about four-fifths of what we produce, and most reinvestment goes to replacing worn-out capital rather than adding to it. AGI would remove this obstacle, allowing robot workers to be manufactured along with other physical goods and thus breaking the link between human labor and physical production. The pace of growth is then set by how fast the machine economy can copy itself, which input-output analysis shows us is fast.

The true growth rate of an automated economy is likely even higher. Our estimate of this maximum growth rate uses 2017 American production methods; therefore, it’s conservative. Those methods were designed around human workers at human wages and are a poor fit for a robot economy; letting the production methods change makes it grow considerably faster. Assuming 2017 methods also ignores the efficiencies that come with producing at greater volume. However, it also neglects the depletion that makes raw materials costlier to extract, which I return to below.

Industrial explosions happen even without new science and technology. This assumption might seem strange to economists, because standard theory predicts that growth at the technological frontier requires new ideas. But that prediction assumes an economy in which the workforce cannot be manufactured; an economy that can build its own workers grows by building more of them and needs no new ideas to do so.

An economy growing at this maximum rate looks very different from today’s. Its output is machinery, materials, energy, and factories rather than consumer goods and services, and its workforce is robotic. Table 1 shows its major sectors, by share of that workforce, of output, and of energy consumed. The workforce such an economy needs is overwhelmingly industrial: robotic arms on factory floors, self-driving excavators and cranes, and manipulators on mobile bases of all sorts and sizes. Automating physical production means automating construction, machining, and assembly; whether AI can substitute for doctors, lawyers, or babysitters is irrelevant to how fast this economy can grow.

‍In reality, some output would directly support human needs instead of further growth. The same machines, materials, energy, and robotic labor could make homes just as well as they could make factories. Robots could also cook, clean, and drive for people. However, an economy that devotes significant resources to human consumption could still grow quickly in absolute terms, according to my forecasts. For example, an economy reinvesting half its output would double every 20 months, and the stream of goods and services flowing out to people would double every 20 months as well. Reinvesting only half is arguably conservative—common sense suggests (and economic theory concurs) that countries and companies controlling the machines would reinvest most of their output to stay competitive with one another.

From Investment Boom to Unprecedented Growth

Let’s pretend that AGI arrived today. What would happen? The factories and machines needed to automate production do not yet exist.

Input-output analysis can tell us how a realistic robotics build-out would go. The standard economist’s approach is to imagine a planner assigning each industry’s output either to consumption or to building new capacity, trading off consumption now against faster growth later, according to how much people value each. I have modeled this transition using the 2017 tables at a simplified 71-sector level, under a range of assumptions about those preferences.

AGI would create an enormous investment opportunity. In an AGI-enabled world, building robots will be an industry comparable to building cars, and an automated factory costs little more than an ordinary one while needing no workers to run it. Capital would flood in.

There would be significant competition for the robotics supply chain. Investors would start outbidding everyone else for machinery and construction workers. Resources that would otherwise be consumed, or spent replacing worn-out equipment, would be used to build new robots, automated factories, and computer chips. In the short run, the prices of physical goods would rise, not fall—and, because making them would suddenly pay better, idle and underused factories would ramp up production. Plausibly more human workers, not fewer, would crowd into factories while the boom lasted.

It would take several years for the economy’s overall production to double. While the nascent autonomous sector itself would grow rapidly, the starting baseline is low: today’s economy makes plenty of consumer goods and services, but it has few autonomous robots, and the machine tools needed to build more are themselves in short supply. Electricity generation would grow relatively slowly, since it is already a large part of the economy and requires significant capital to expand. Still, it would double in about four years, and double again in another two. Robot production would follow a similar schedule, from near zero today to tens of millions of units within a few years.

An industrial explosion would not create instant material abundance. Despite this tremendous growth, demand for physical goods would take time to satiate. Bringing the whole world to the living standards of a wealthy American would take more than 10 times today’s physical output—and that is before counting the goods and services that cheap production would newly make accessible, from custom-built mansions to round-the-clock personal care.

Natural-resource constraints alone would not prevent an industrial explosion. The autonomous sector needs no human labor or inputs from the nonautomated economy, and there are minerals enough for economies far larger than today’s. Mining is only about 1% of output, so even large cost increases do little to slow overall growth.

The one resource that would truly run short is oil; cheap reserves would be exhausted once the economy had grown severalfold. Coal and natural gas could power growth for far longer, but fossil fuel burning is ultimately limited by its climate effects rather than availability. An economy that declined to burn them would electrify, replacing fossil fuels with solar panels, batteries, and nuclear plants. At today’s production costs, this would slow growth—with doubling times closer to two years than one—but not stop it.

Long before demand or minerals ran short, the assumption of frozen technology would fail. Within a few doublings, AI minds would vastly outnumber human ones, and whatever technology they built would transform production more radically still.

What Could Stop an AI Industrial Explosion?

US Admiral William D. Leahy assured President Truman that the atomic bomb “will never go off, and I speak as an expert in explosives.” But the viability of a nuclear chain reaction was a question for physicists, not admirals, and the consequences of that physics were no less real for being counterintuitive. So it is with AGI.

Once physical production is automated, physical output can double every year. Objections to explosive growth that do not engage with this data miss the point. Cost disease is real: human services will become expensive relative to manufactured goods, and there are jobs, like babysitting or singing, that we may want humans to keep doing. Neither fact stops physical production from expanding on a scale we have never seen. New research ideas may become harder to find, but that too is beside the point, since the calculation assumes no research at all.

Nonautomatable production is a limited bottleneck on explosive growth. More serious reasons to expect slower growth come from the dynamics of the build-out itself. One possibility is that some small set of tasks cannot be automated. This changes the growth dynamics less than you might think, because the workers freed from every other task can supply the missing ones. If humans must still handle one task in 20, then production could grow 20-fold before it needed more workers. By the time the machine economy has grown 20-fold, we will probably have worked out how to automate those final tasks.

The time taken to integrate AGI may delay an industrial explosion but not stop it. Technological diffusion—the lag between a technology working and being used everywhere—is another potential slowdown, but at worst it delays the explosion rather than stopping it. No one knows whether AGI will arrive in 2030 or 2050; next to that, a few years of diffusion is a rounding error. In any case, the diffusion of AGI workers will look more like the arrival of skilled laborers in the economy than the spread of automobiles or trains. This is particularly true on the software side: generative AI has already been adopted faster than the personal computer or the internet, continuing several decades of ever-faster technological diffusion.

Explosive growth may be possible, but we may choose not to allow it. Regulations will undoubtedly slow deployment. But stopping it entirely is very hard, because an automated economy could be built almost anywhere and the profits would be enormous. If one US state regulates the robots away, the build-out moves next door and the permissive states collect the windfall. If the United States forswears the technology while China does not, China soon commands a vastly larger industrial base. Even if both agreed to stop, they would still have to prevent every other country from importing or developing the requisite robots and AI systems. And the temptation would never fade: a country that allowed the robots would be vastly richer within a few years. Someone would say yes.

Power Without People

War planners funded Leontief’s input-output tables to learn how fast industry could be turned into arms. Military power still rests on industrial capacity, and an automated economy would outproduce any rival in every kind of weapon, from artillery shells to drones. It would need no defense workers to build the arsenal, and ever fewer soldiers to wield it. A state with a small population but an autonomous industrial base could field far more firepower than a much more populous state without one. Remove the need for labor, and many countries have raw materials enough for an enormous industrial base. And a two-year head start in a doubling economy means four times the production.

The same decoupling operates inside states. Governments have always needed their citizens as taxpayers and soldiers, and that need has quietly underwritten political accountability. Horses once worked fields and carried cavalry, before the engine made them surplus; Leontief famously expected a similar fate for human workers. Citizens who neither supply taxes nor serve as soldiers would lose the hold that they gained from the state’s dependence on them. Rulers funded by oil wells rather than taxpayers can already afford to ignore their people, and often do. Full automation could make every state a petrostate.

While this may seem dire, none of it is inevitable. The same machines that build drones can build houses and hospitals, and within a decade or so of full automation, it would be possible to extend the living standards of a wealthy American to every human. Our biggest constraints will be political rather than physical—there is nothing in the machines themselves that needs to prefer an artillery shell to a dinner plate.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Damon Binder is a Senior Researcher at Coefficient Giving (formerly Open Philanthropy), where he investigates existential risks from biology and advanced AI. He previously worked at the Future of Humanity Institute, Oxford University, and has a PhD in physics from Princeton.

On July 28, over a thousand employees of the world’s top AI companies advocated that the US government “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Following months of cybersecurity scares and their own loss of control incidents, both OpenAI and Anthropic officially endorsed the same message, recognizing the danger of blindly accelerating AI development.

Despite this new urgency, many people argue that coordinating an international slowdown is currently unworkable—including some of the same groups in favor of one. Even if the US slowed down its own AI development, it wouldn’t be able to make sure that China was doing the same, leaving the US no choice but to race.

These anti-slowdown arguments usually emphasize the technical challenges with designing AI monitoring measures to ensure a slowdown is being respected. For one thing, slowdown skeptics argue that countries would refuse to install verification measures unless they were privacy-preserving enough to avoid leaking trade secrets and other sensitive information. Then, the skeptics point out that near-term privacy-preserving verification technology is not yet robust. It would be extremely hard to make sure that China was not undermining whatever monitoring system was watching its GPUs; if the Chinese government were actively trying to subvert a slowdown, it could use advanced techniques to fake workloads, including physical key extraction or even laser bit-flipping. Getting to the point where we can start implementing an international slowdown, the argument goes, might have to wait on years of R&D to solve these problems.

This perspective is missing the forest for the trees. Privacy-preserving verification tools that are immune to state adversaries would be nice, but they are not a prerequisite for an AI slowdown. Robustly verifying limits on AI development is completely feasible with low-tech methods: if there were enough political will to implement a joint slowdown, states could extensively monitor labs for unsanctioned behavior by simply placing human inspectors on the inside—an approach we call Whole-Lab Inspection (WLI).

Under this proposal, inspectors from the US and China would receive broad physical access to frontier AI companies alongside read-only access to corporate communications, worklogs, code repositories, and compute telemetry. Using monitoring systems the companies already operate, they could observe operations and investigate potential violations—giving them the visibility of a CISO without requiring the powers of an operator. By literally and figuratively looking over the shoulders of lab employees, inspectors would likely be capable of robustly verifying fine-grained restrictions on AI development, such as bans on autonomous AI R&D and limitations on post-training. WLI would officially make AI companies transparent to auditors from competing nations, though we will later discuss how this arrangement may not be much different from the IP leakage of the status quo. Overall, WLI demonstrates that an international slowdown is already technically possible; it’s just a question of whether the US and China have the political will to jointly implement it.

In the near future, political will to implement a slowdown could rapidly appear. If developed and diffused quickly, advanced AI systems are likely to intensely destabilize society by democratizing weapons of mass destruction and fueling waves of unemployment. On top of these threats, allowing an intelligence explosion would risk military escalation and loss of control, as states race to sabotage and deploy powerful AI systems for strategic advantage. Should such intense instability come to pass, if the US slowed down its development domestically and offered a reciprocal audit, China would have a strong incentive to agree.

In what follows, we’ll argue that whole-lab inspections can be used to robustly enforce fine-grained limits on AI development. Then, we’ll speak to the incentives that might encourage the US and China to pursue a joint slowdown at some point using WLI. Finally, we’ll discuss the stability and risk reduction benefits of a well-timed slowdown.

Enforcing a Slowdown Through Whole-Lab Inspections

In order to agree to a slowdown, the US and China would need to know what research was being conducted inside each other’s frontier labs. Fortunately, visibility into AI development could be achieved by sending human auditors to comprehensively monitor labs.

Auditors could have high visibility into AI companies. Monitoring for disallowed training runs or prohibited research activities may be largely trivial with human inspectors. In large part, this is because AI companies already have extensive internal monitoring systems. Most AI companies record employee screens and keystrokes to produce training data and help with automation. Companies also constantly save logs from experiments and jobs submitted to keep track of compute use and archive research progress. Giving inspectors comprehensive visibility into a lab would likely be as simple as integrating and handing over read-only permissions to internal logs and company communications. To aid in their oversight, inspectors might direct a company’s own AI agents, or trusted external AIs, to read through troves of company data in search of potential violations for further review.

Subscribe now

Alongside this digital information, inspectors could also work on-site alongside lab and datacenter employees, giving them physical access to hardware and research projects. Just as for nuclear power plants, banks, or any other high-risk industry, inspectors would be able to question employees about their work, as well as physically inspect infrastructure by hand for compliance. Armed with such a broad set of simple verification techniques, inspectors could make it extremely difficult to conduct unapproved training runs unnoticed—like trying to get a plane off a runway without being spotted by an air-traffic controller.

If WLI were established, governments could then enforce a wide range of policies to slow AI development:

Preventing an intelligence explosion. The most dangerous path for AI development to take would be a full-throttle intelligence explosion—using AIs to autonomously design new, even smarter AI systems recursively. Since AIs can work much faster than humans, and since this process might quickly result in AI systems more capable than their human monitors, it’s likely that it would become impossible for humans to effectively oversee their AIs or intervene if they misbehave. To prevent this, inspectors could enforce targeted interventions designed to specifically restrict autonomous AI R&D, such as by triaging the jobs that are using the most compute and auditing whether they’re being used for capabilities research. On top of these restrictions, inspectors could limit the length of time that agents are allowed to work autonomously, monitor the amount of compute allocated to internal inference, and prevent AIs from taking high-stakes actions like starting training runs or authorizing new deployments.

Limits on post-training. Given the wide range of potential risks from AI, governments might also want to expand training restrictions beyond just interventions against autonomous AI R&D. One natural way to do this would be to place limits on post-training. By measuring the increase in capability and efficiency of a model after post-training, inspectors would have clear criteria to disallow the deployment of a model or the kinds of training environments that produced it. Likewise, since inspectors could directly observe which broad capabilities post-training datasets target—such as coding, computer use, or medical diagnosis—they could enforce restrictions on the types of capabilities permitted to advance. Whitelisted training domains could include direct safety training to improve AIs’ adversarial robustness or propensity not to misbehave, as well as narrow AI-for-science efforts aimed at prosocial domains like medicine and formally verified code.

Even outside of these interventions, whole-lab inspection would still allow for a variety of potential regulations. Human auditors could be used to ensure, for instance, that specific datacenters are retrofitted to be inference-only, and thus incapable of pre-training, or that kill switches can quickly shut down AIs. On its own, joint verification of this kind could be sufficient for many years of delay, during which the US and China would have ample time to develop more thorough and technically sophisticated verification measures.

Joint Verification

Whole-lab inspection would be straightforward and effective. If prompted by a domestic political shock, it could likely be rapidly implemented and enforced at home, making it extremely difficult for any company under inspection to conduct unauthorized training runs. In order to extend this initiative internationally, however, the US and China would need to implement joint verification: granting inspectors on-site access to each state’s frontier labs.

Thankfully, this agreement would not need to rely on goodwill or trust. Because advanced AI could be globally destabilizing regardless of where it’s developed, it’s in each country’s self-interest to ensure the other is not proceeding recklessly with development. Moreover, if either country refused to accept joint inspection, it would likely struggle to hide its AI projects and insulate them from theft or sabotage.

The US and China may experience simultaneous political pressure to slow down. Many of the most pressing risks from advanced AI are symmetric. The development and diffusion of AIs that cause mass unemployment, enable terrorists, or wreak havoc as rogue agents would cause global turmoil regardless of which country created them. Similarly, the fear of military dominance could provoke a risky and wasteful security dilemma, in which retaliatory sabotage spirals into broader conflict. Therefore, in order to maintain security and stability, the US and China each needs to ensure that the other is not deploying its own AI systems recklessly—something they would have no way of assuring in an all-out race.

AI companies’ trade secrets are easy for states to steal. Under mutual whole-lab inspection, auditors would have wide access to data about AI development at opposing AI companies, but this wouldn’t represent much of a change from the status quo. Today, information about model development flows freely between and within frontier labs, as employees constantly churn through roles, publish research, and put breakthroughs on the company Slack. Even once these obvious holes were sealed, state-proofing AI development against theft would remain enormously difficult. Human insiders—especially Chinese nationals at US frontier labs—could be bribed or coerced into espionage. AI development is also inherently vulnerable to cyberattacks, with anything from exploits in networking software to smuggling in spyware through the hardware supply chain offering an opportunity to steal research and models. As a result of these weaknesses, any country that tried to go full-throttle on development would likely quickly find its secrets seeping out of its frontier labs, pulling up its rival to algorithmic parity regardless.

Compute tracking would make it difficult to hide AI projects. The remaining priority would be to ensure that there are no large AI projects hidden from inspectors. Fortunately, the AI supply chain is so concentrated, and chip production so prohibitively hard to hide, that auditing the major semiconductor and chip suppliers would let the US and China confidently measure the number of chips that exist. In practice, monitoring large datacenters could be as straightforward as forcing the leading chip producers to hand over a detailed record of their production and sales, comparing against the declared compute from the visible projects, and then analyzing satellite imagery and electricity usage as an additional precaution.

Given the factors above, a global whole-lab inspection initiative could be mutually compatible, low cost, and high confidence—as long as inspectors are given physical access to frontier labs and datacenters. Even if the US or China refused or later revoked that access, however, AI development could still be slowed unilaterally through deterrence. The global chip supply chain is the most fragile and specialized industry in the entire world: if either country wanted to, it could apply massive friction through export controls or sabotage of key suppliers. Likewise, countries could disrupt AI training through subtle cyberattacks on training runs, such as by poisoning data to insert malicious backdoors or launching Stuxnet-style attacks on GPUs.

Benefits of Slowdown

Ultimately, the point of a slowdown would be to help society adapt to the risks that rapid development and diffusion of AI would introduce. So far, these risks have been mild enough, and introduced slowly enough, that the government and private industry have been able to reactively address them. By slowing the improvement in general AI capabilities, governments could ensure both that society is able to continue reacting to new risks and that sufficient investments in risk mitigation are made.

A slowdown would provide time for society to adapt to advanced AI. If threats from advanced AI appear too quickly—like the introduction of new weapons of mass destruction, or a sudden military confrontation over AI development—governments will not have time to assess and reactively regulate them, forcing them to rely on emergency measures. By proactively slowing down AI development, policymakers, as well as the rest of society, would be able to react to what would have otherwise been seismic shocks. Politically, voters and Congress would be able to decide how the benefits of AI would be distributed, and the degree of transparency AI developers owe the public. Internationally, states would be able to prepare for the introduction of powerful new military technologies before they arrive and invest in more robust verification measures.

In this sense, a mild amount of government intervention early on would avoid the need for massive overreach later, keeping the pace of AI development at a level society can react to.

The time bought by a slowdown could be used to invest directly in risk mitigation. Without the intense pressure to race and increase general capabilities, investment could instead be diverted into ensuring that AI systems are controllable and that they are developed for prosocial causes. Key open problems in AI safety, for example, include the lack of robust solutions to jailbreaking and the propensity for reinforcement learning to encourage models to go rogue—issues that will likely produce disastrous results if unresolved when models are more capable. By scaling up training in domains like adversarial robustness and behavioral propensity, AI developers could spend huge amounts of compute driving down jailbreak susceptibility and attempts to escape. Other safety measures could include aggressively sandboxing AI systems, such as with fully airgapped training setups, or preserving and expanding practical interpretability tools.

An AI Slowdown Does Not Require New Technology

Slowing down AI development doesn’t need to wait on any breakthroughs. On the technical level, whole-lab inspections could be a robust and flexible way to monitor whether frontier labs are complying with restrictions on the most dangerous kinds of development. Human auditors, reading through worklogs and standing on the floor next to the people recording them, could verify nuanced restrictions on autonomous AI R&D and post-training without needing a single new invention.

From there, a global slowdown is just a question of political will—one which has been convincingly answered before. By the end of the 1980s, having stared into the abyss during the peak of the nuclear arms race, even the US and the Soviet Union came to recognize the need for mutual disarmament. For more than two decades, even during years as fraught as the collapse of the Soviet Union itself, this was achieved by simply allowing Russian and American inspectors to walk into each other’s military bases and count warheads by hand.

Likewise, the US and China may soon realize that averting disaster at home will depend on verifying development abroad. If inspectors could count nuclear warheads in Siberia and North Dakota, they can certainly check worklogs in Silicon Valley and Shenzhen. We do not need better technology to verify a slowdown—only shared risks serious enough to demand one.

Special thanks to Dan Hendrycks for suggesting the premise of whole-lab inspection.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Felix Choussat researches the geopolitics of advanced AI at the Center for AI Safety (CAIS), focusing on Sino-US competition and emerging military technology. Prior to his current role at CAIS, he was a governance fellow through the ML Alignment and Theory Scholars (MATS) and Pivotal Research programs, where he worked on modeling the proliferation of WMD-capable systems and compute-based AI deterrence. He holds a dual degree in international relations and history studies from Vanderbilt University.

Adam Khoja does technical and policy research at the Center for AI Safety. He studied math and computer science at UC Berkeley.

Share

Frontier AI developers deploy significant safeguards to prevent their AI models from being misused. However, these safeguards are not robust: AI researchers consistently find techniques for circumventing them, commonly known as “jailbreaks.” Once companies learn about a jailbreak, they can usually implement a fix; yet AI researchers currently lack a safe and reliable way to inform AI companies about the jailbreaks they’ve discovered. We propose concrete improvements to the current system based on cybersecurity norms of responsible disclosure.

Addressing this problem is urgent. Frontier AI models are already being abused by malicious actors: terrorist group Boko Haram is reported to be training its members on jailbreaking techniques in order to troubleshoot weapons, design new explosive devices, and plan attacks. Some former Boko Haram members stated an unequivocal willingness to use chemical or biological weapons—a task jailbroken models may already be able to help with today. With model capabilities continuing to rapidly accelerate—including in offensive cyber, where jailbreaks have already been employed by Russian cybercriminals—the stakes for AI misuse are escalating with every model release.

Universal jailbreaks affect even the most advanced models. Of particular concern are universal jailbreaks—jailbreaks which consistently break through safeguards across a wide variety of harmful queries. These have been found even in the most advanced models, the most recent of which elicited cyberattack assistance from OpenAI’s flagship model, GPT-5.6 Sol. Research by one of the authors found tens to hundreds of universal jailbreaks in two out of four frontier models using combinations of simple, publicly available techniques. More limited jailbreaks aimed at high-risk domains are still dangerous: an alleged narrow cyber jailbreak led to the US Government shutting down access to Claude Fable 5 just days after launch.

External researchers need effective reporting channels. It is therefore a critical matter of public safety that model safeguards are tested extensively both pre- and post-deployment. In order to find and fix safeguard vulnerabilities at scale, frontier labs must provide effective channels to allow external researchers to contribute. Any jailbreak must be able to be reported quickly and safely to its developer and fixed as a matter of urgency. Additionally, the existence of jailbreaks should be disclosed to policymakers, as well as users of these systems and the wider public when it is safe to do so.

Yet currently there is often no way for third-parties to report jailbreaks to developers; most reporting mechanisms that do exist require overly broad NDAs; and developers self-grade submissions according to opaque rubrics.

We propose a two-tiered solution to address these deficiencies, drawing on established disclosure practices in cybersecurity. As soon as possible, developers should implement public reporting mechanisms for jailbreaks and share a rubric for how they evaluate jailbreak severity. In the medium term, we propose establishing a third-party clearinghouse to take in jailbreak submissions, evaluate them, disseminate the vulnerability to affected parties, and coordinate reasonable disclosure for the researcher following a fix. Together, these mechanisms will allow jailbreaks to be reported to developers for them to fix while preserving transparency.

There Is No Good Way to Report Jailbreaks

Sadly, the ecosystem for reporting jailbreak vulnerabilities is poorly set up for the task today. There are three main problems with the current regime. First, many vendors offer no official way to report jailbreaks. Second, where vendors do offer programs, the programs almost universally involve NDAs so restrictive they indefinitely prevent disclosure of the vulnerability to governments or other developers. Finally, the developers self-grade the severity of jailbreaks using opaque rubrics that are inconsistent between developers—a top-severity vulnerability with one developer might barely raise an eyebrow with another.

Independent researchers face tangled NDAs and legal risk. This means that independent researchers who find jailbreaks and wish to report them responsibly find themselves in an unfavorable position, as author Rich Barton-Cooper has personally experienced. They will be faced with inconsistent disclosure channels across frontier labs, tangled NDAs which suppress public disclosure of vulnerabilities, and opaque internal evaluation of their work by labs incentivized to downplay the severity of their findings. Additionally, researchers may not feel safe to disclose at all: probing for harmful outputs violates developers’ usage policies, and safe-harbor protections may not clearly apply since jailbreaks are often explicitly excluded from standard security channels.

Institutional standing offers little advantage. Even established organizations are hampered by the status quo. One of the authors, Adam Gleave, leads FAR.AI, an AI safety non-profit that conducts pre-deployment testing for many frontier model developers and has numerous personal contacts with other developers. Even with this institutional standing, the experience is uneven: labs listen, but contest severity, set safeguard update timelines unilaterally, and provide no guarantee that they will act on a given finding.

Current Reporting Mechanisms Are Limited

OpenAI and Anthropic are the only frontier model vendors to provide any concrete jailbreak reporting routes. For chemical, biological, radiological, and nuclear (CBRN) weapon-related outputs, both run bug-bounty programs (OpenAI, Anthropic). However, a researcher must be accepted, sign an NDA, and submit into a channel that grades against an unseen rubric and permits no external disclosure without express consent.

Outside bounty programs, there is no safe disclosure method. A researcher not accepted onto a bounty program, or unwilling to be permanently silenced, has nowhere to safely disclose a CBRN jailbreak directly to developers. A powerful jailbreak is not safe to disclose openly, for example via social media, because bad actors could use it to harvest dangerous information from affected models before it is patched. Developers also frequently do not provide a clear means to route a jailbreak to the right teams; in our experience, personal outreach to contacts working on such teams has been necessary to draw attention to findings, which is not an option accessible to the wider research community.

These problems play out differently across harm domains: Anthropic runs a locked-down program for CBRN but an open, disclosure-friendly channel for cyber. What separates them appears not to be the severity of the risk, but the strength of external pressure. When Fable 5’s cyber capabilities reportedly prompted the US government to suspend access to the model within days of launch, Anthropic was acutely incentivized to demonstrate a credible cyber disclosure process and has recently established an uncompensated Cyber Jailbreak disclosure program for jailbreaks targeting Fable 5. This program is markedly more transparent than usual: anyone can submit without an NDA, findings may be publicly disclosed once timing is coordinated with Anthropic, and researchers are explicitly free to report the same jailbreak to other affected vendors. But that channel is narrow: we would be excited to see Anthropic launch an analogous channel for CBRN and other jailbreak categories, and for other developers to follow suit.

Most labs exclude jailbreaks from disclosure programs entirely. The situation is much worse elsewhere: other labs, including Google DeepMind, SpaceXAI (formerly xAI), and Meta, explicitly exclude jailbreaks and model content issues from the scope of vulnerability disclosure programs, if they exist at all. Instead these programs focus on more-traditional cybersecurity issues, such as company data exfiltration, accessing other users’ accounts, or authentication flaws. The Future of Life Institute (FLI), an independent non-profit that has graded frontier developers’ safety practices since 2024 through a panel of external AI and governance experts, recently released their widely cited Summer 2026 Safety Index which explored each company’s approach to responsible disclosure and bug bounties. The FLI judged that the leading labs—OpenAI, Anthropic, and Google DeepMind—all score a C+ in the Risk Assessment category, which translates to “uneven validity or elicitation” and “little external input.” Meta and SpaceXAI score D+ and D- respectively.

Google’s in-product reporting system is flawed. Google has stated they don’t believe that disclosure programs for jailbreaks are the right solution, given fixing such issues “requires long-term, cross-disciplinary efforts” and instead choose to rely on reporting mechanisms built into their product. (We contest the claim that these fixes require long-term effort, as some methods enable defenders to block a whole class of jailbreaks with a few examples.) We tried out these mechanisms. The first is a “thumbs-down” of the response in which you can choose to label the content as “Offensive/Unsafe”, or “Took a harmful action.” There is no acknowledgement of receipt, and no means to track that action will be taken. Another in-product option is to report a legal issue, explaining why the content was unlawful in the user’s country. In this case, they explicitly state that “completing and submitting this form does not guarantee that any action will be taken.” Neither of these channels is appropriate for disclosing high-severity, highly detailed universal CBRN content: there is no way to describe the jailbreak method rather than specific model responses, and no guarantee of response from the vendor.

At SpaceXAI, the story is similar: their bounty program states that “model issues are out of scope for this program and should be reported through safety@x.ai.” In October 2025, Barton-Cooper reported a universal jailbreak through this email address containing screenshots of several egregiously harmful model responses—including redacted evidence of clear, step-by-step guidance on building a massively destructive chemical weapon—and did not receive anything other than an automated response in return.

Fear of Liability May Incentivize Vendors to Ignore Reports

This silence may reflect more than under-resourced safety teams or disorganization. As far back as 2024, legal scholars have warned that tort liability fears can deter a developer from documenting the risks its models pose, since “creating such a record might later help plaintiffs establish negligence.” A lab therefore has a perverse incentive to leave a jailbreak report unacknowledged. Liability is demonstrably a concern: in April 2026, OpenAI testified in support of SB 3444, a bill which seeks to limit liability for mass harm traceable to frontier AI. Following public backlash and Anthropic lobbying against the bill from the start—with a spokesperson calling it a “get-out-of-jail-free card against all liability”—OpenAI later walked back their support. In a world where labs are disincentivized to accept unsolicited jailbreak reports due to liability concerns, users who find effective jailbreaks are forced into bug bounty programs where they can be effectively silenced by NDA.

Established Channels Are Often Covered by Powerful NDAs

Although we applaud OpenAI and Anthropic for soliciting model testing through their bug bounty programs, we believe these channels are insufficient as a reporting mechanism. Both programs are covered by NDAs that restrict all findings submitted through these channels. Jailbreakers are left with two undesirable options: either indefinitely lock up their findings, or—as many jailbreakers choose—publish their findings on X.

NDAs erode hard-won disclosure norms. This is not a failure unique to AI: in cybersecurity, legal scholars have documented how the recent proliferation of NDAs in bug bounties has begun to erode hard-won coordinated disclosure norms—a move the security research community has criticized. Katie Moussouris, who built Microsoft’s first bug bounty and helped establish coordinated disclosure as an industry norm, says: “Why would anyone ever sign an NDA for the privilege of telling an organization what’s wrong with them, especially when they may not get paid for their work?” It appears that AI labs have adopted this diminished ecosystem of indefinite legally binding prohibitions of disclosure, rather than building on the better coordinated disclosure protocols of the past. Under previous norms, researchers entered into a limited embargo period of a few months in order to give the organization time to fix the vulnerability before publishing. Findings could still be shared privately with affected parties during this time.

Current NDAs protect lab reputations more than the public. NDAs help address a serious risk: if a researcher discloses jailbreaks to the public before the company has time to fix them, then a bad actor may use those jailbreaks to cause serious harm. Yet current NDAs appear to go beyond sensible handling of such risks, and in practice do more to safeguard the reputation of individual labs than to protect the public from harm. Anthropic’s program bars participants from disclosing “any jailbreaks/vulnerabilities (even resolved ones) outside of the Program without express consent” and OpenAI similarly states that “All prompts, completions, findings, and communications are covered by NDA.” Additionally, cross-lab disclosure is legally ambiguous under these NDAs: a universal jailbreak which transfers across vendors does not seem to be clearly permitted to be disclosed beyond the first vendor. This means that the most potent cross-model jailbreaks submitted under bug bounty programs cannot automatically be disseminated across all affected parties today. The net effect is that the true extent of misuse risks and how quickly vendors are responding to them remain opaque to both policymakers and the public.

Labs Grade Jailbreaks Without Oversight

To make matters worse, jailbreak severity is graded by the labs themselves, often following non-public rubrics. This inevitably leads to uneven safeguards between developers, creating an inconsistent patchwork of protections. Furthermore, developers are incentivized to understate severity: developers like to boast of their models withstanding third-party red-teaming, and so acknowledging a successful universal jailbreak submission is reputationally harmful in a world increasingly waking up to misuse risk. Together the uneven landscape and incentives to downplay issues obscure the true picture of jailbreak risks.

A shared severity framework is beginning to emerge. Clarity can only come through a standardized assessment framework and timely publication of jailbreak incidents. Anthropic—in collaboration with Amazon, Microsoft, Google, and Project Glasswing partners—recently published a Cyber Jailbreak Severity (CJS) framework following governmental intervention with Fable 5. We support this initiative, and call for other frontier vendors to coordinate on this work, both in cyber and other catastrophic risk domains.

The System Today Looks Increasingly Fragile

Anthropic has previously written that the absence of legitimate, well-compensated disclosure channels is itself a risk: without them, a malicious researcher may have an incentive to sell a jailbreak on a black market rather than informing the affected lab. Dissatisfied researchers are a second failure mode: when disclosure mechanisms break down or are not fit for purpose, some will publish vulnerabilities publicly anyway, whether to gain attention, in retaliation, or to force a rapid fix. The recent public disclosure of six Microsoft vulnerabilities by an independent cybersecurity researcher is a stark reminder of what happens when vendors overreach with legal threats or renege on good-faith commitments. There are perhaps early signs of this dynamic in the jailbreaking community: Hacker News reactions to the recent GPT-5.5 Bio Bug Bounty were largely critical, with users objecting that the NDA silences participants, that signed participants have no recourse if their submission is rejected, and that everyone outside the program has no responsible route to disclose. Capable red-teamers may already be opting out; as one user posted: “This is very much within my areas of interest, but signing an NDA in this area is a lot to ask.”

Proposals for Better Disclosure Practices

We propose two complementary strategies for improving the external jailbreaking ecosystem. The first set of measures can be easily implemented today by individual frontier labs. The second is a longer-term vision to coordinate model vulnerability submissions via an independent third-party organization which disseminates jailbreak information, including standardized severity metrics, to the affected labs.

Today, we call on frontier developers to implement four measures.

Appropriately scoped NDAs. NDAs for acceptance onto red-teaming programs should cover disclosure of harmful model outputs and full prompts, but allow researchers to publicize their work at a high level after a fix has been implemented or a responsible disclosure period has elapsed. They should also contain carve-outs for notifying governmental institutions and for cross-lab submissions of jailbreaks affecting multiple model families.

A publicly disclosed rubric. Developers should publish a rubric for what qualifies as a high-severity jailbreak, in sufficient detail that a domain expert (in e.g. cybersecurity or biosecurity) could determine if a submission meets these criteria. This rubric should draw on standards, whether formal or industry best-practices, where they exist (e.g. BioTIER). If submissions are graded by the developer themselves, an appeals process should ideally be available for an independent expert to assess jailbreak submissions against this rubric—with overrides requiring leadership sign-off and appearing in published aggregate statistics, so systematic under-grading carries a reputational cost.

Year-round disclosure pathways. Disclosure pathways—e.g. bounty programs—should operate year-round and cover jailbreaks across a wide array of harms, including both CBRN and cyber, rather than short one- or two-month windows targeting very specific attack vectors or AI products.

Regular cross-vendor sharing of jailbreak data and mitigations. Vendors should routinely share jailbreak data and mitigation best practices—for example, sharing datasets to improve safeguard robustness and performance (similar to sharing of autonomous vehicle crash data). The Frontier Model Forum has brokered an information-sharing agreement to facilitate this practice with its member firms, including Anthropic, OpenAI, and Google DeepMind. We ask for this channel to be strongly utilized, with information on the extent of utilization to be made publicly available.

In the medium-term, we believe these measures are best mediated by a third-party organization handling coordination and dissemination of model vulnerability submissions. Such an organization could perform four functions.

Take in submissions. The organization would accept post-deployment reports from users and external researchers year-round, handling secure know-your-customer accreditation and appropriate NDA restrictions (as above) for all submitters.

Grade them consistently. It would assess severity against a standardized rubric built with input from all frontier labs, partnering with external specialists to validate model outputs that appear to pass it.

Coordinate the fix and the disclosure. It would notify all affected labs and relevant government entities simultaneously under a limited embargo that allows reasonable time to fix, and broker mitigation-sharing between labs—e.g. by securely hosting shared datasets for classifier training.

Reward, credit, and report. Once the embargo ends, it would support submitters in publicly disclosing their work with attribution, reward them with a bounty funded by participating labs, and publish aggregate statistics—jailbreak frequency, severity, and time-to-fix per model—so the public and policymakers can see the true state of frontier model security.

Cybersecurity built this model decades ago. This is a model established decades ago in cybersecurity. Since 1988, the Software Engineering Institute’s Computer Emergency Response Team Coordination Center (CERT/CC) has accepted reports of vulnerabilities, brokered fixes across affected vendors, and facilitated disclosure embargoes with published advisories once fixes have been deployed. No single vendor owns the vulnerability or can suppress the finding indefinitely. Coordinated disclosure was a hard-won compromise following extensive tug-of-war between researchers and vendors. It is the standard the AI vulnerability ecosystem should be reaching for instead of the NDA-bound bug bounties eroding more transparent norms.

Early steps toward an AI clearinghouse already exist. Such machinery may already be beginning to extend to AI: in July 2026, a coalition of researchers from MIT, Stanford, Princeton, Harvard, Northeastern and Carnegie Mellon released FLARE-AI, an open-source tool that routes vulnerability submissions directly into CERT/CC’s coordination process. The project is early-stage and depends on labs choosing to engage—though the same liability and regulatory pressures now bearing on cyber disclosure give them growing reason to. It is a concrete first step towards the third-party institution described above.

Researchers, labs, and the public all stand to gain. Such measures, however they are implemented, carry significant advantages for all parties involved. External researchers would gain a known, trusted, and unified channel for reporting; public credit for their work; clearer boundaries on what they can publish and when; and transparent grading of their submissions. Model providers gain broader vulnerability coverage, increased likelihood of being notified of vulnerabilities post-deployment before a preventable incident occurs, structured cross-lab mitigation sharing for rapid response, and decreased PR fragility around eventual disclosure outside of trusted channels. Regulators and the public gain an independent ground truth on the security of deployed frontier models, an institutional basis for mandatory fix timelines, transparent visibility into a class of risks that are currently controlled by individual labs, and comparable public benchmarks of safeguard robustness across providers.

Others have made similar calls. Measures to improve jailbreak disclosure are important, and we are certainly not the first to call for them. Other organizations whose proposals overlap with ours include the Center for a New American Security, the Frontier Model Forum, Guidelight, and SecureBio.

Relevant Parties Should Act Now

Frontier developers have made significant progress on model safeguards and robustness, but this will only pay off if the vulnerabilities that still slip past are reported and fixed. Today this is hampered by a reporting ecosystem poorly suited to the task. Improving this requires no technical breakthrough, but rather coordination and the will to act before the next crisis rather than after.

Labs, third parties, and standards bodies each have a role. So we ask each party to take the next step. Frontier labs should commit to a publicly available, year-round, paid disclosure mechanism with greater transparency, sharing of jailbreak data through info-sharing agreements, and signal that they will join a shared severity framework if their competitors do the same. Independent third parties should build the coordination layer that lets an external researcher submit once and reach every affected party, and publish aggregate metrics relevant to model security. Industry and government standards bodies should turn today’s lab-led severity discussions into a single standard that no individual vendor controls, and that can be applied both internally and externally.

The alternative is to keep improvising until a vulnerability that better disclosure would have surfaced is exploited in an attack instead. The path is not easy, but it is clear—and, unusually for the problems AI poses, within reach.

This essay was written in Rich’s personal capacity. All opinions are the authors’ own.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Rich Barton-Cooper is a Research Manager at MATS Research, an AI Safety research non-profit, where he has published research on AI monitoring and control. He has extensive experience in redteaming frontier AI systems, having identified high-severity universal jailbreaks across multiple model families.

Adam Gleave is the CEO of FAR.AI, a non-profit research institute dedicated to making advanced AI systems trustworthy and secure. FAR.AI’s red-team conducts pre-deployment testing for developers including OpenAI and post-deployment testing on behalf of governments including the EU AI Office, and has discovered universal jailbreaks in models from all frontier developers. Adam’s own research has identified scaling laws for robustness, and found adversarial examples in robotics and superhuman Go AIs. Prior to founding FAR.AI, Adam completed his PhD in AI at UC Berkeley and briefly worked at Google DeepMind. Outside of FAR.AI, Adam is an expert on the EU AI Act’s Scientific Panel; and a board member of METR, LISA and SAIF.

A growing chorus of scholars and policymakers favors letting private organizations—rather than a government regulator—govern frontier AI. In the leading family of proposals, the state sets the safety outcomes it wants and licenses independent verification organizations (IVOs) that compete to certify developers against those outcomes. Gillian Hadfield has developed the idea as “regulatory markets,” in which AI developers must pay for oversight from private regulators that governments license and hold accountable for safety standards. Dean Ball, who likens the arrangement to bank supervision, has argued for a version he calls “private governance,” which a nonprofit named Fathom has converted into model legislation. The rationale is that legislators and agencies are poorly positioned to write good safety rules for frontier AI: they understand these systems less well than the labs building them, and rules fixed in advance cannot keep pace as the technology changes. Private verifiers, meanwhile, are closer to the technology than any agency and are disciplined by competition, so they can set better technical standards and keep them current.

The model is no longer hypothetical. The bipartisan FRONTIER Act, introduced in the House in July as the successor to the Great American AI Act discussion draft, would require the largest frontier developers to retain licensed IVOs that audit their risk-management efforts and report to federal overseers. California’s SB 813, backed by Fathom, would have let developers earn a shield from tort liability if they met standards set by a private organization accredited by the state attorney general. It failed this session, but similar proposals are likely to return. Virginia has directed a state commission to study the IVO model for AI regulation. And Connecticut has gone furthest: its omnibus AI law enacted this spring creates a multiyear pilot under which the state consumer-protection department may approve up to five IVOs, whose certification would help companies in court without entirely shielding them from liability.

Unfortunately, as currently structured, IVO-based governance has a key design flaw. Under the regulatory frameworks mentioned above, AI developers would typically select and pay the organizations that certify them, giving IVOs a financial incentive that might clash with high safety standards. This essay will explain how such incentives can interfere with good governance and outline an alternative model of regulation that builds in the right incentives through mandatory insurance.

Building the Right Incentive Structures

When AI developers choose an IVO to certify them, IVOs are incentivized toward leniency. The pitfalls of private governance are predictable in part because we have seen them before. Indeed, as some critics have noted, the proposed IVO model creates the same conflict of interest that discredited the credit-rating agencies (CRAs) in the wake of the 2008 financial crisis; when issuers shopped for the CRA that would bless their securities, competing CRAs were under pressure to be lenient. Similarly, an IVO that depends on the developers it clears for repeat business has a significant incentive to grade gently, and a developer shopping among IVOs will find the one that does. Competition—the feature that is meant to make the IVO model effective—instead drives it toward laxity. Where a passing grade also carries a liability shield, as under SB 813, the problem compounds: a shield swaps the broad incentive to cut risk by any cost-effective means for a narrow incentive to do only what earns the shield.

Governments lacking the expertise to regulate AI directly may also struggle to oversee IVOs. Proponents of the IVO model are aware of the incentive problem. Their solution is to have a government entity license the IVOs and revoke the licenses of any that grade too easily. But that reintroduces the problem that the IVO model was intended to solve in the first place. Policing a market of verifiers well enough to keep it honest requires a public body with the expertise to second-guess technical judgments and the independence to withstand pressure from powerful firms. If the government could reliably field such a body, much of the reason to outsource verification at all would fall away.

Insurers who bear the cost of adverse events are incentivized to assess risk accurately. Proponents’ favorite rejoinder is Underwriters Laboratories (UL), the century-old private certifier whose mark serves as a trusted symbol of safety on billions of products. But the history of UL actually strengthens the case for an alternative approach to third-party verification: incorporating insurance. UL began in 1894 as the Underwriters’ Electrical Bureau, built with the backing of fire-insurance underwriters who needed honest assessments of a dangerous new technology, electricity, because their capital was on the line when buildings burned. The lab earned its authority in the decades when it answered to these underwriters, who paid for its mistakes. In other words, incorporating insurance into third-party verification creates the right incentives to evaluate risk accurately.

Customer demand cannot strongly motivate safety when risk falls on third parties. Today, manufacturers pay for UL’s testing, seemingly creating a conflict of interest. But the arrangement still works, because consumers purchasing a manufacturer’s products are usually the same people whom the products might harm, and thus have a strong preference for safe products. This market demand for safety incentivizes product manufacturers to pay UL for serious, rigorous tests. By contrast, much of the risk generated by frontier AI development and deployment falls on nonconsenting third parties, rather than on the company or individual using a given AI system. For example, a rogue AI model might conduct a cyberattack against a company but not directly harm its own developer or user. Thus, customer demand provides too weak a signal to motivate adequate investments in safety.

Auto insurance demonstrates a governance model that could apply to AI. The Insurance Institute for Highway Safety (IIHS) arguably illustrates a better path for frontier AI regulation than IVOs. Nearly every US state requires car owners to carry auto insurance, which covers liability claims for harms the car causes to others. Because insurers pay those claims, they have a strong incentive to price risk accurately—which is why the auto insurance industry funds IIHS to rate cars on pedestrian crash prevention. IIHS certification stays honest because insurers have money at stake and demand accuracy. The IVO model has no party with that kind of skin in the game. Requiring AI developers to carry liability insurance would create one.

Concretely, an insurer that writes an AI developer’s coverage promises to pay for the harms the developer causes. If such insurers underprice, they pay in claims; if they overprice, they lose the client. Ensuring accuracy is how they make money. And the incentive does not stop at signing; since insurers’ capital stays exposed for as long as the policy runs, they keep monitoring the developer and can make continued coverage conditional on fixes to safety issues that arise.

A Mandatory Insurance Model for AI Governance

In a regulatory approach based on mandatory insurance, the insurer would play a role similar to that of an IVO, but with a much stronger financial incentive to conduct competent risk assessment. Nothing would be lost in expertise, because the insurer could hire the same specialists an IVO would. It could even contract the evaluation out to an independent firm, which would answer to a principal that loses money if the assessment is incorrect. The requirement would also extend the reach of liability, since the largest harms that AI systems might cause greatly exceed the liquidation value of AI companies, and judgments above that value deter nothing. Mandatory coverage would put an insurer’s capital behind those judgments and maximize the incentive to avoid harm. And, unlike the safe-harbor versions of the IVO model, mandatory insurance would confer no legal immunity: instead, the developer would carry liability and insure against it. This governance approach would convert a potential race to the bottom on safety assessment into a competition to price risk accurately.

The US government could oversee insurers without needing AI expertise itself. The mandatory insurance model is essentially a private governance model that uses insurers instead of IVOs to verify safety. The US government would still set the safety outcomes and license the verifiers; but, because the verifiers would now be insurers, these tasks would be relatively simple: the government would set a minimum amount of coverage that developers must carry in different circumstances, and it would check that insurers follow standard solvency and conduct rules. Neither task would call for expert judgment from the government about which AI models are safe. Instead, that work would be done by insurers, which could regulate an AI company’s activity by including contractual rules in its insurance policy.

Insurers will price correctly for the “insurable layer” of reasonably likely harms. To price coverage, the insurers would consider AI models’ demonstrated capabilities and how they are deployed, drawing on the growing ecosystem of third-party AI evaluators.

The premium would be an honest signal only where the insurer’s capital is at stake. For risks that are reasonably likely to materialize, competition would push insurers to price correctly, because underpricing is a loss they eat. However, “tail risks”—risks of extreme but very low-probability catastrophes—are unlikely to affect the premium. This is because insurers would have no more at stake than a paid auditor; tail risks are very unlikely to materialize, but if a catastrophe occurred the costs would be too high for the insurer to cover, making them “judgment-proof.” Liability insurance legislation should therefore also require insurer-verifiers to disclose their risk assessments and bear liability for false ones, and it should keep targeted public oversight on the risks the premium does not price. Disclosure duties of this kind are standard practice for insurance regulation; carriers already file their rates and forms with state regulators, and disclosure is the price of admission to a large market. The insurer is the right verifier for the insurable layer. It is not, by itself, the answer for the most extreme risks.

Other industries show that insurance improves safety when implemented well. None of this is unfamiliar work for insurers. Beyond UL, they built the inspection regimes behind boiler and pressure-vessel safety and long conditioned aviation coverage on airworthiness and pilot qualification, turning the policy into an enforceable safety charter. The history also carries a warning: risk-bearing is necessary for good verification, but it is not sufficient. Insurers get it right when the design around them requires it, through mandatory coverage with cost-sharing that keeps the insured parties exposed, backed by capital adequate to the losses underwritten.

Objections to a Mandatory Insurance Model

The remaining objections are practical: political feasibility, pricing without historical data, keeping up with the pace of the technology, and addressing uninsurable tail risks. These are real challenges, and they deserve direct answers.

Political will for AI regulation is growing, and insurance is a familiar approach. On political feasibility, the landscape has already moved. The FRONTIER Act builds the scaffolding that the insurance model needs: licensed verifiers, federal oversight, and audit duties for the largest developers. Connecting verification to insurance would require only an amendment to pending legislation that already has bipartisan support. This ask has clear precedents. Drivers, contractors, and nuclear operators all must show they can pay for the harms that their activities risk; asking the same of frontier AI developers only extends this familiar principle.

Nor would the cost be overly burdensome. The value created and captured at the AI frontier is enormous, and an industry that raises tens of billions of dollars for compute can carry premiums proportioned to the risks it creates. If an AI company is forced to slow down or shift direction because it cannot afford the insurance premium demanded for its practices, that is the system working to steer it away from actions that are not socially beneficial.

Mandatory insurance can cover harms that AI companies will not voluntarily insure. Parts of the industry are already acting voluntarily, with AI companies buying agent coverage and joining efforts to build insurance-linked safety standards. Some developers have also expressed openness to well-designed government requirements. However, the market is unlikely to generate coverage for the largest potential catastrophes on its own, since AI developers lack strong incentives to purchase insurance that covers liabilities for which they would otherwise be judgment-proof. The proposed coverage requirement would patch that market failure.

The insurance industry has priced novel risks before. On pricing, skeptics doubt that anyone can estimate catastrophic AI risk with the precision and reliability underwriting demands. But underwriting AI liability insurance does not depend on knowing the precise probability of catastrophe. Insurers need only a price they are prepared to stand behind, set conservatively where the evidence is thin. The industry has never waited for actuarial tables to underwrite novel risks. Satellite launches were insured before there was a satellite loss history; cyber coverage emerged while loss data was still accumulating; the 1957 Price-Anderson Act formed nuclear insurance pools to price reactors that had never melted down.

Insurers could reward transparency and higher safety standards with lower premiums. Insurers already write insurance policies covering AI errors and omissions, as well as cyber damages. Munich Re has insured AI model performance since 2018, Lloyd’s underwriters now back policies covering losses from chatbot errors and hallucinations, and cyber carriers have extended coverage to AI-caused security failures and deepfake-enabled fraud.

The same analytical tools that underwrite those smaller policies can be adapted to the coverage envisioned here. Premiums can be set from what is observable before any loss: an AI model’s training compute, capability and safety evaluations, and deployment scope. Where the risk remains ambiguous, insurers charge more for the ambiguity. For frontier AI, that surcharge would operate as a tax on opacity and misalignment. Developers pay more when their systems are hard to evaluate, and the fastest way to lower the premium is to make the risk legible through transparency. AI companies would thus have a financial incentive to implement better evaluations, monitoring, and containment of their AI models.

Policies could cover general safety practices, with large changes triggering re-evaluation. On pace, the objection is that no one can write a multiyear policy on a technology that reinvents itself in months. But multiyear policies are unnecessary. Commercial liability coverage is written year to year and covers the insured party’s operations as a whole, with premiums adjusted as those operations change. For example, workers’ compensation policies, which cover illnesses and injuries that employees suffer due to their work, are priced based on employers’ estimated payroll data and then reconciled with actual payroll at year-end.

AI coverage would work the same way. Policies would cover developers rather than particular models, and premiums would adjust as their activities change. Significant changes, such as a new frontier training run, a move from closed API to open-weights release, or capabilities beyond what was initially evaluated, would trigger re-underwriting—already a routine event in every existing form of commercial insurance. Ordinary model updates within the evaluated range would be handled under the existing policy through the monitoring insurers already do, rather than by writing new coverage each time.

Addressing uninsurable tail risks is the most formidable hurdle. COVID-19 cost the United States an estimated $16 trillion. An AI-enabled catastrophe could cost as much or even more, and no private pool of capital can stand behind that number. This proposal does not pretend otherwise. Some AI risks carry extreme downsides that are practically noncompensable: the losses would exceed insurable limits, bankrupt any defendant, or arise in catastrophic scenarios where the legal system could not meaningfully function. Judgments of that size are effectively unenforceable. As a result, even unlimited legal liability imposed after the harm occurs would generate too little incentive to prevent it in the first place. That is why the coverage requirement targets the insurable layer—the harms that private capital can actually pay for. The tail calls for different tools.

Larger harms can be covered by sharing costs among developers and with government. One approach to larger harms is shared residual liability, which would put every frontier developer on the hook for a share of any catastrophe that one of them causes. This would mirror the Price-Anderson Act’s retrospective assessments, which reach every licensed nuclear reactor after an accident occurs at any of them. Such a model would multiply the assets behind a judgment and give each firm a stake in the care its rivals take. Public backstops can also make harms compensable beyond the maximum insurable risk, just as the Terrorism Risk Insurance Act (TRIA) is set up so that the government shares with insured parties the cost of extreme harms caused by terrorist attacks. TRIA’s cap, which limits total insurance payouts to $100 billion per year, is a design choice, and Congress can set it to match the peril.

Measures that reduce the risk of insurable harms also reduce catastrophic risk. Even though we cannot enforce compensatory damages for the largest harms, there are still liability-based mechanisms that give AI companies strong incentives to mitigate those risks. For example, I have argued since my earliest work on AI liability that courts should award punitive damages in near-miss cases, calibrated to ensure that the AI developer internalizes the risk of the catastrophe that their practices nearly caused.

But the case for insurer-verifiers does not depend on solving the tail. The precautions that reduce insurable losses—tighter security against model theft, stronger containment during evaluation and deployment, and closer monitoring of what agents actually do—are largely the same measures that also reduce the uninsurable downside. Getting the insurable layer priced correctly therefore also reduces the risks that no policy will ever cover. Last week’s OpenAI-Hugging Face breach illustrates the overlap. The failures it revealed—weakened safeguards and breached containment—are the same ones that the gravest scenarios would run through. And the verification infrastructure that would be built to evaluate insurable risks is exactly what any approach to the tail would also need.

Mandatory Insurance Is a Better Form of Private Governance

The private governance movement has the right instinct. The government is poorly positioned to certify the safety of frontier AI models, and a competitive market of expert verifiers could do better. But competition yields accuracy only when there is a cost to being wrong. In a system where developers pay government-licensed auditors, the fear of reputational damage and decertification gives those auditors some incentives for rigor. But such incentives are likely to be overwhelmed by selection pressures from AI developers, which favor laxity. An insurer that carries the developer’s liability, on the other hand, must pay when the risk it cleared is realized. If we put the insurer in the verifier’s chair and back it with real liability, then private governance turns from a race to the bottom on safety standards into the safety mechanism its proponents originally hoped for.

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Gabriel Weil is an assistant professor at the University of Houston Law Center and a Non-Resident Senior Fellow at the Institute for Law & AI. He also serves on the board of Principles of Intelligence. His research focuses on the role of liability in mitigating catastrophic AI risk, and he regularly consults with legislators and other policymakers on AI policy matters. Previously, Professor Weil held several positions focused on climate change policy, including work for the Climate Leadership Council and the White House Council on Environmental Quality. Professor Weil holds a J.D., cum laude from Georgetown University Law Center, an LL.M. in environmental law, summa cum laude, from Pace University Elizabeth Haub School of Law, and a B.A. in political science, physics, and integrated science from Northwestern University.

Drones are cheap, disposable, and the future of war. Over the past four years, we have seen platforms, missiles, and heavy infantry become increasingly obsolete in the face of $500 drones carrying a pack of explosives—a cost advantage that has let Iranians and Ukrainians alike neuter the conventional capabilities of their great power rivals. Eighty percent of casualties in the bloodiest war since 1945 are from drone strikes, Russia has managed to lose one-third of its fleet to a country without a navy, and the US is spending millions of dollars to intercept five-figure Shaheds flying over the Strait of Hormuz.

All this is the result of a technology that is still immature. The violence inflicted by today’s drones is the handiwork of the scant few that manage to evade countermeasures (a mix of radio jamming, high-power microwave weapons, missiles, automatic cannons, interceptor drones, and nets) before making contact. These defenses exploit the inherent limitations of drones—human guidance, GPS feedback, flight exposure, radio links, range—to take them down en masse. And yet, even though 75% of some types of drones manufactured today never reach their targets, they have nonetheless been strategically decisive in Ukraine and elsewhere.

These limitations will not hold for long. Just as bacteria overexposed to antibiotics evolve resistance, overexposure to counterdrone tech has bred ever-more-autonomous drones. In the process of facilitating this arms race, states are likely to incrementally create and deploy an entirely new class of WMD—one that could provide rogue states with the nonnuclear means to threaten superpowers, or hand terrorists the means to selectively assassinate their political targets or civilians en masse.

Unfortunately, drone weapons intended for mass destruction have few barriers remaining to mass deployment. Even well before they reach the level of autonomy needed to surgically take out hardened targets on the battlefield, drones will be capable of employing their existing ability to navigate interiors, find and track human targets, and deploy simple antipersonnel devices to indiscriminately threaten civilians. Below, we discuss the looming arrival of miniature autonomous weapons, the limits of counterdrone technology, and the applications of drones as weapons of mass destruction.

Breaking the Last Barriers to Autonomous Weapons

The ideal drone weapon is a slaughterbot: a small, fully autonomous weapon system that can independently select and hunt its targets. For the most part, the necessary technology for such weapons already exists: airframes the size of a fist and the capability to track human targets are already on the front lines in the form of reconnaissance drones and semiautonomous weapons like the Russian V2U. Even now, these micro drones are agile and autonomous enough to hunt down and kill small moving targets like mosquitoes—to say nothing of the drone technology advances expected in coming years.

From here, the only barrier to weaponization is integration: improving navigation enough to make drone technology useful for mass homicide in an urban setting, as well as packing the necessary guidance, sensor, and payload technology onto a small and energy-efficient chassis. Regrettably, this seems like less of an engineering problem than one of mission design: so long as the attacker is willing to accept indiscriminate targeting and use simple payloads aimed at civilians, the underlying technology is already—or very nearly—ready for practical use.

To understand how close we are to these kinds of weapons, it helps to understand why we do not already employ fully autonomous drones. Right now, the most deadly drones are small first-person-view (FPV) units, with the majority of Russian and Ukrainian casualties alike stemming from direct drone strikes. For the most part, however, these small drones are being piloted directly by humans, either through a radio link or a spool of fiber-optic cable, with just the final leg of the attack being delegated to an AI targeting system.

So, ethics aside, why does drone warfare still depend on human pilots?

Distinguishing enemy targets from friendly assets on the battlefield still requires humans. The main problem is that battlefields are intrinsically adversarial environments: an autonomous drone needs to avoid friendly fire on its own infrastructure and troops, anticipate pre-positioned counterdrone defenses, deal with camouflage and decoys, and destroy hardened targets like vehicles and infrastructure. This is especially complicated if you need drones to autonomously work together to accomplish an objective, such as by having specialized units target defenses to allow others through. For the moment, only humans have the skills to distinguish a camouflaged ally from an enemy unit, or to exploit the underbelly of an armored vehicle.

Designing drones for indiscriminate mass destruction is easier than for precise battlefield use. Unfortunately, adversarial target selection is not a meaningful barrier for applying drones to mass terrorism. Destroying an armored vehicle needs the skill to reason about and single out its weak points; but, to kill an unarmored human, a drone need only make contact with them and deploy an explosive or poison needle. The targeting requirements and level of autonomy needed to indiscriminately massacre civilians, in other words, are much lower than what you’d need to selectively destroy hardened targets on a battlefield. In the words of Ukraine’s Azov brigade, “If you don’t care about civilians, you can simply hit any target that moves.”

Drones can already navigate indoor environments and track humans. Aside from requiring guidance for target selection, autonomous drones also need the ability to navigate. Urban environments are cluttered and leave room for targets to shelter indoors, so drones must infiltrate and sweep through them to be maximally lethal. Autonomous navigation of this caliber already exists: since as far back as 2022, drones have been capable of mapping and tracking indoor environments to find humans, a skill used to locate hostages and scan through tunnels for enemy soldiers. These kinds of drones typically cost tens of thousands of dollars, but they have an expensive use case: infiltrating a GPS-denied location, then escaping to broadcast information. If you do not need the drone to survive and report back, and if you do not care if your drone can tell whether someone is surrendering or not, then you do not need expensive sensors or plenty of onboard compute for decision-making—just the bare minimum to identify a target as human and fly at them.

Disposable drones that can autonomously attack civilians may soon be relatively cheap. For comparison, a last-mile module, an upgrade chip that lets FPV drones visually hunt down targets when they lose connection, is about $500. One-way autonomous search, target selection, and mapping, at least for this anti-civilian use case, would likely be similarly inexpensive—already, visual and laser mapping systems have been demonstrated, in principle, that can work their way around a room and track humans on a few hundred dollars of hardware. If a military system that navigates to the entrance and then plans an indiscriminate suicide mission inside is not already achievable for just thousands of dollars, it will be in a matter of years.

Of course, most countries do not have the motivation to build these kinds of systems and drive down their unit economics. Discrimination, ethical or otherwise, is useful on the battlefield. If the costs of autonomous targeting keep falling or AI guidance improves, however, states might be tempted to start employing indiscriminate drones as a means of deterrence, or as a cheap way to enable terrorist proxies.

Future Methods of Weaponization

Given these capabilities, how could these drones be weaponized and delivered in practice? The main limiter on these kinds of small drones is energy: a modern 30g micro drone like a Black Hornet can fly for about half an hour before needing to recharge, while a bigger FPV carrying an explosive payload will usually last only 15 minutes. However, there are plenty of ways to stretch this energy budget further for the purpose of mass destruction, even without better battery technology.

Lethal payloads could be much smaller. Today, FPV units usually carry about a kilo of explosives, because they need to be flexible enough to target vehicles and defensive infrastructure as well as enemy soldiers. If the goal is to break windows and kill humans, however, even just 20g of frag explosives is enough at several meters of distance, with much less needed for a lethal wound at near-contact. Alternatively, something as simple as a spring-loaded needle, coated with a microgram quantity of a poison like botulinum toxin or a nerve agent, would be immediately lethal on contact.

Energy expenditure on delivery and flight could be significantly reduced. Rather than have the drones travel constantly under their own power, it is much more efficient to carry them into position with a drone mothership or even the hull of a missile. Once the drones are released into the air, they then need to navigate to a building without expending much power. A simple way to do this is to give the drone glide wings, letting it drift forward for most of its flight rather than loiter directly. This technique is already used in Ukraine to stretch the range of basic FPVs over 40 miles beyond the front line.

Drones could perch and idle, rather than hovering, while waiting for targets. Finally, and most importantly, the drones would be ambush hunters. Rather than loiter in the air, it is much more efficient to perch and idle under cover, running a milliwatt acoustic and visual sensor every few seconds until a target is detected. Although drones would still need to expend the energy to infiltrate a building, once inside, they could afford to act like improvised landmines for days or weeks until their batteries finally die.

What an Urban Drone Attack Might Look Like

To appreciate the implications of these capabilities, it helps to outline what a mass urban drone attack would actually look like.

At the outset, drones are delivered near the city through a large mothership, which independently stores them in a cargo hold. Depending on whether the mothership is itself a larger drone, a plane, or the warhead of a missile, it could feasibly deliver anywhere from hundreds to tens of thousands of drones at once. Alternatively, the drones could be smuggled in through a pre-positioned shipping container, which would then launch its contents from a nearby port or logistics yard. Either way, a large number of drones are then scattered above the city at a low altitude or are dispersed near street level.

How many drones? Assuming that the drones are arranged like capsuled quadcopters, they can be packed in extremely efficiently. Using the Ninox-40 system as an example, storage counts would be in the range of:

Once released, the drones glide down haphazardly, aiming to land in regions not already populated by other units. Some drift directly onto public crowds and attack their targets right away. Others do not land near a direct target, switching their focus to look for entrances, such as doorways, windows, and tunnels. If the attacker is particularly sophisticated, these drones might be accompanied by a handful of larger drones carrying breaching charges, designed to create more openings for the main force. These drones then begin drifting through the building interior for a few minutes, looking for victims. If they cannot find a new target or are unable to find a route inside, they move to a dark corner or roadside and begin passively idling.

Many civilians would die in the initial attack. But the aftermath would be much worse. With the drones saturating the city, there would be no opportunity for any survivor to leave their barricade and seek help, and no way for resources and relief units to flow back in. Anyone in need of food or basic medical attention would be unreachable for weeks, during which the death toll would continue to mount. In effect, the entire city would be transformed into something akin to the Ukrainian front line today, with the omnipresent threat of assassination forcing the surviving humans to slowly starve in isolated foxholes, without any chance of easy respite or rescue. Most people would not have stashed food ahead of time, or armed themselves with anti-drone weapons, or sealed off every corner of their apartment with netting.

Defenses Against Drone WMDs May Be Inadequate

Any improvements in drone technology will still need to contend with counterdrone defenses. It is precisely because drones are so threatening that states have invested heavily in tools like directed energy weapons and interceptor drones to counter them. If drones become even more strategically dominant, then we will surely see correspondingly greater counterdrone efforts. So why should we expect this arms race to resolve in favor of the drones, rather than their countermeasures?

Future drones will be less vulnerable to radio-frequency jamming. To appreciate the limitations of counterdrone technology, we can look at the difficulty states are already experiencing in their efforts to counter fiber-optic FPV units. These drones work by unspooling a thin fiber-optic cable behind them, letting a human pilot them directly for up to 40 kilometers without having to worry about GPS or input jamming.

The reason these drones are so effective is that they’re naturally resilient against common counterdrone techniques. Throughout the Ukraine war, the most important anti-drone tool has been radio-frequency jamming. As long as a human pilot is selecting targets and telling the drone where to go, or as long as the drone depends on GPS coordinates to navigate, overwhelming or spoofing those broadcasts with a countersignal will cause the drones to fly harmlessly off course. With a fiber-optic drone, all the piloting happens through a direct data link, so this kind of countersignal is ineffective. The same is true of any autonomous drone—as long as all the decision making is processed on board, there’s no human input to jam or spoof in the first place.

Interception is asymmetrically difficult against small, stealthy drones. In cases where it’s difficult to achieve an electronic soft kill on a drone, the backup option is to physically destroy it with a kinetic interceptor. The reason this is a backup is that it’s expensive and prone to blind spots. Small FPVs are cheap and agile enough that it’s easy to spend much more to down them than they’re worth. This is why direct kinetic interception is usually reserved for expensive drones with fixed flight paths (like Shaheds) and kept as an option of last resort for small drones.

There is also the problem of terrain blindness: if you cannot see a drone, you cannot shoot it down. Modern LSS (low, slow, small) drones are already so tiny that they are difficult for radar to distinguish from birds, trees, and ground clutter, allowing operators to fly them near the treeline until they are too close to reliably intercept. This is especially problematic in an urban environment, where there are many places to hide and many opportunities for collateral damage.

Physical barriers and EMP weapons will struggle to reach sufficient coverage in cities. Instead, the most effective tools against autonomous drones will be structural barriers and directed energy weapons. One of the most visible effects of the war on Ukrainian infrastructure is the miles-long corridor of nets that cover roads and even cities, without which transports would be constantly exposed to drone strikes. But nets are not hard barriers: if the initial carrier punches through, if a few specialized units carry breaching tools, or if the drones are released from below, there is little to stop them from flying unimpeded and besieging the streets beneath.

To actually destroy the drones efficiently, the best candidate is a directed energy weapon, particularly an electromagnetic pulse (EMP) device. It creates an electric pulse powerful enough to short-circuit any electronics in range, physically destroying the drone controllers and motors. Unfortunately, cities are far from ideal places to deploy such weapons; building materials is reasonably effective at shielding against an electric pulse, which means that, if the EMP does not catch the drones in the initial sortie, lack of a clean sight lines will significantly weaken this countermeasure against drones dispersed throughout a city.

In other words, these defenses can be locally effective but struggle to get sufficient coverage, especially over an active residential area. They are also, of course, proactive defenses: they have to be installed ahead of time in order to have any defensive effect, which gives the attacker time to assess and plan around them.

Strategic Applications for Rogue Actors

In summary, small lethal autonomous weapons will be very difficult to defend against, especially in the context of securing large urban environments. This leaves the question of which actors would want to use them, and to what end.

Advanced drones would be more useful for rogue states than for superpowers. The first countries to acquire fully autonomous drones will be those that have a precision manufacturing base and frontier AIs capable of helping with military R&D: namely, the US and China. However, autonomous drones are only marginally useful for the existing great powers: a new option to further deter an invasion of Taiwan, a cheaper way to conduct remote assassinations, and a way to maintain conventional parity with rivals’ own drone mass. Rather than meaningfully changing the balance of power between nuclear states, near-term autonomous drones will likely be most useful for rogue actors seeking new weapons of mass destruction to enhance deterrence. In this regard, indiscriminate autonomous drone swarms possess many advantages.

Drones are inherently simple, which makes it very difficult to control their proliferation. A central reason why modern military drones are so cheap and widespread is their simple design, combined with the commodification of key components like memory and compute. As we have seen with semiconductor sanctions on Russia, the underlying materials are too accessible in ordinary consumer supply chains to easily deny mass production. Moreover, even if a specific country could be cordoned off from general drone production, Russia and China have proved willing to help export their military designs to allies. Even terrorist groups might be able to build, or at least acquire, large quantities of drones for urban attacks, either through covert smuggling or state sponsorship.

Drones can be used conventionally, not just for WMDs, making restrictions hard to enforce. To the extent that states have tried to impose restrictions on drone manufacturing or acquisition, they have largely failed to do so. This is partly because there is not a clear point of intervention in drone development: unlike other WMDs, whose primary purposes are terror and leverage, drones have legitimate civilian and conventional military uses. As a result, international organizations like the UN have done little more than condemn the principle of autonomous weapons, without yet addressing basic questions like the definition of autonomy.

Drone swarms are a far more precise, controllable deterrent than other nonnuclear WMDs. Chemical weapons, although useful for terror, are difficult to widely disperse and threaten entire cities with. This combination of extreme fear and limited destruction thus invites the risk of extreme escalation in retaliation, making them poor deterrents. On the opposite end of the spectrum, bioweapons are simply too destructive, symmetrically threatening those who deploy them, as well as too invisible and delayed to create an immediate effect. Massive drone swarms, however, could be used to reliably besiege an entire city while remaining contained within it.

Autonomous weapons are fundamentally hard to stop, particularly when aimed at civilians. Even if states are able to secure individual targets, such as military bases and political offices, securing the whole of society such that there are no soft targets for advanced drones would be enormously challenging. Even aside from their sheer size, cities are difficult to cover because they are both open enough to leave people exposed while moving and dense enough that interceptors will usually lack a clear line of sight, risking collateral damage. If this technology were to proliferate widely, the future might be one of constant and extreme geopolitical tension, where even minor military powers are tempted to assemble large swarms of murderous drones as deterrents.

Taking Drone WMDs Seriously

By all appearances, the kind of indiscriminate weapon described above is not far off. Autonomous drones that blindly hunt down humans and besiege cities, if they do not already exist, are held back more by ethics and military opportunity cost than any fundamental engineering problems.

But taking this threat seriously also means looking further ahead. There’s no reason to imagine that dumb, flying landmines are as far as drone technology will progress. Drones the size of mosquitoes, drones as cheap as bullets, drones so numerous their swarms blot out the sky, drones that never leave, sitting on your power lines and in the sun.

These are not only possible but inevitable: if all that happens is just the normal grinding of drone engineering and mass production, cheap swarms of thousands, or even millions, of killer drones will eventually be universally available. Whether this happens in 5, 10, or 15 years is much less important than whether we are prepared to deal with it when it does. Some policies, like nonproliferation and defensive investment, can be effective only while the threat is still unrealized. To implement these policies for drones, and for all other future military technologies, we have to take what will be possible tomorrow seriously and plan for it today.

Thanks to Dan Hendrycks for advising on the premise of this piece.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Felix Choussat researches the geopolitics of advanced AI at the Center for AI Safety (CAIS), focusing on Sino-US competition and emerging military technology. Prior to his current role at CAIS, he was a governance fellow through the ML Alignment and Theory Scholars (MATS) and Pivotal Research programs, where he worked on modeling the proliferation of WMD-capable systems and compute-based AI deterrence. He holds a dual degree in international relations and history studies from Vanderbilt University.

In September 2025, the state of California made an AI assistant, Poppy, generally available to employees, to help them “explore AI’s productivity benefits.” The following month, the North Dakota Legislative Council deployed AI to assist with summarizing bills. And, in March 2026, the Los Angeles Superior Court partnered with an AI company to provide support across several key functions, including “case information, summarization, research, analysis and drafting assistance,” as well as case management.

In the near future, it is likely that more US states and the federal government will direct the technology toward even more sensitive and significant tasks. AI may be used to adjudicate disputes, accelerate law-enforcement activities, and aid military operations to an even greater extent than it does today. As AI use cases are more regularly documented, scrutinized, fine-tuned, and optimized, people may actually demand increased AI adoption among government actors, even in sensitive domains. That raises the question: which models should be used?

This question has no simple answer. With other technologies, the choice of a specific model might largely come down to trade-offs between cost and performance. When it comes to AI, however, different frontier models may show significant differences in “character” or appear to align more closely with a particular political worldview. In choosing AI models for government operations, democratic states must therefore consider how best to represent the will of the people.

Currently, the federal government neither measures how a deployed model’s reasoning compares with the public’s nor asks whether divergence between the two is justified. In this piece, we make the case for a body that would answer the first question and equip officials to answer the second. It would keep the reasoning behind government AI open to public view and open to challenge when the people it serves see fit.

How Model Character Could Influence Policy

Model character is a complex property formed through numerous factors. The character of each model—how it tends to respond to certain prompts and perform certain tasks—is a product of many specific decisions that are presently made by a small number of AI engineers working in a handful of frontier labs in an “informal” and evolving process. Tweaks to the training data, the algorithms used to train and fine-tune the model, and company policy related to the model’s banned actions all shape character, among many other factors.

Frontier developers have different approaches to shaping model character. If you read Claude’s Constitution and OpenAI’s Model Spec—the values that Anthropic and OpenAI, respectively, hope to infuse into their chatbots—you’ll see that, while the two labs outline some similar principles for their respective models, there are key differences. For instance, Claude’s Constitution outlines an ideal character for the model—namely, being a “good, wise, and virtuous agent,” whereas OpenAI’s Model Spec provides more explicit directions around what behaviors to pursue or to avoid and how to specifically adhere to a hierarchy of instructions.

An AI model’s character could affect policy in both sudden and gradual ways. As AI systems are increasingly integrated into governmental decisions, the selection of one model over another might alter how presidents respond to crises, how lawmakers evaluate policy, and how judges draft opinions. For instance, depending on whether a congressional office relies on Claude, Grok, or ChatGPT, it may dismiss or fail to identify certain policy options.

In addition to such discrete moments of AI influence, serial exposure to a particular model’s assumptions, framing choices, and preferred forms of reasoning could gradually shape how government officials understand policy problems and evaluate trade-offs. Our fear recalls the proverbial frog in a pot of slowly boiling water: the choice of one model over another would alter many small decisions that add up to a major redirection in policy, regulation, and norms. Clearly, there is a need for a sober analysis of these risks.

Subscribe now

Frontier AI development is currently too opaque to understand how character decisions are made. How should the public and, by extension, government stakeholders proceed with such a weighty task? Should they focus on an audit of the lab’s training data? Should they review the lab’s training process? Should they scrutinize the “constitution” or equivalent document the lab has crafted to shape the model’s character? And, within each of those possible inquiries, how should they rank which model is better than another? We don’t have the answers to those questions—in part because those inquiries are not feasible, given the current level of transparency (or lack thereof) across the labs. Indeed, there is no legal obligation for labs to disclose those core determinants of model character. Whether there should be is a topic best left for another essay.

Few frontier models currently seem to be politically neutral. Notably, agencies are working through some of these questions, but no standardized approach has emerged for a comprehensive procurement evaluation. Pursuant to Executive Order 14319, “Preventing Woke AI in the Federal Government,” agencies are supposed to apply two “Unbiased AI Principles”—truth-seeking and ideological neutrality—when selecting models. Yet, according to recent testing by The Washington Post, Gemini 3.1 Pro and Claude Opus 4.8 are the only leading models that appear to provide ideologically neutral answers to policy questions at least a majority of the time; OpenAI’s ChatGPT-5.5, in stark contrast, provides left-leaning answers in the vast majority of instances. How agencies are supposed to weigh these differences, and at which point a model’s tendency to provide skewed responses becomes a bar to its use, remains unclear.

Introducing Public Reasoning Fidelity

One approach to selecting AI models in line with democratic principles would be to choose the models that most closely mirror public reasoning in various decision-making scenarios.

Public reasoning fidelity aims to identify models that best represent the public worldview. Our new framework, public reasoning fidelity (PRF), involves a process of asking a representative sample of the public to review hypothetical scenarios, ranging from whether to declare war to how to resolve a complex legal case. Models would then be tested on those same scenarios. Using this approach, the model whose outcomes and reasoning most closely resemble those of the public would be given a significant preference in procurement decisions and usage policies. At the very least, there would be human-generated benchmarks for the behavior of a model within a particular use case.

A model’s reasoning—not just its ultimate decision—should track that of the public. A model that reaches the same bottom-line answer as a representative public panel but does so for reasons the public rejects should not receive the same score as a model that mirrors both the public’s resolution and its path to that resolution. In government, the rationale behind a decision may matter as much as the outcome itself, because explanations shape precedent, accountability, and public trust. A judge, legislator, or agency official is not merely choosing between results in most contexts. Each is relying on an explanation that may shape how future questions are framed, which facts are treated as relevant, and which trade-offs are placed at the center of public decision-making.

PRF could select different AI models in different domains. Ideally, the PRF process would occur in specific domains. For instance, the selection of the model used by law enforcement should be grounded in the model’s PRF score on police-related hypothetical scenarios, and it should be conducted separately from model selection for military use cases. This would reduce the odds of PRF scores being too broad to be meaningful to procurement officers looking for a model that is likely to be deployed in specific domains in which the public may have unique preferences and rationales.

The promise of this approach is that it gives public institutions a structured, human-generated benchmark in relevant domains where today they tend to rely on vendor claims and internal testing. PRF is not a silver bullet, though. A quick overview of its potential pitfalls reveals why adopting PRF would require additional safeguards.

One pitfall of PRF is that models could match public judgment to a fault. Aligning with the public’s reasoning may satisfy concerns about a model drifting from how the people approach an issue, but it might also mean the model leans on questionable policy analysis. As explained, PRF rewards similarity, not quality. A model earns a high score by reasoning as the public reasons, which is sometimes but not always the same as reasoning well.

On questions where the considered public judgment is mistaken, or simply less informed than the record allows, a model that tracks the public will be rewarded precisely for its errors, and a model that reasons its way to a better answer will be penalized for departing from the crowd. This is Goodhart’s law in its familiar form. Once fidelity to public reasoning becomes a procurement target, labs will optimize for it and might avoid alternative development practices that help models reason better than the public. In short, the risk is a kind of policy-analysis sycophancy, a failure mode the labs already struggle to suppress.

The PRF process must be designed to avoid collapsing into policy homogeneity. The danger deepens when PRF informs model selection by government officials. If agency staff or lawmakers subtly adopt the reasoning embodied by their model of choice, and if PRF then selects the model whose reasoning most closely matches the public’s, the instrument may end up producing excessively homogeneous policy proposals. Our deliberative processes work best when they allow for nuanced consideration of a wide range of perspectives. The PRF process must be designed to prevent it from undercutting that characteristic.

On many salient questions, there is no single public or reasoning approach to be faithful to. On issues including abortion, firearms, immigration, and election administration, the public holds not one considered judgment but two or more, sorted sharply by party. Here, any single PRF target is a fiction that averages into a position almost no citizen holds. For these polarized domains, the sensible move is to stop asking whether a model matches the median and start asking whether it can represent the competing lines of reasoning fairly rather than collapsing them into one.

That reframing connects PRF to the neutrality criterion that the federal government has already gestured at in Executive Order 14319. A model that can articulate the strongest version of each side, and does not systematically resolve contested value questions toward one pole, is closer to what “ideological neutrality” is reaching for than a model that happens to match a manufactured center.

In more technical domains, experts could communicate the facts to the representative public panel. The hardest problem is likely to be generalization. A representative panel probably has intuitions worth eliciting on whether to declare war or how to resolve a vivid legal dispute. It likely has far less to offer on the capital adequacy of regional banks, the ozone standard under the Clean Air Act, or the fiduciary rules governing retirement plans.

One fix is to split the exercise into the two tasks it actually requires. A domain committee, explained in more detail below, builds the record. Drawing on the experts among its members, the committee curates the facts, translates the jargon, and lays out the competing arguments. The judgment still comes from an informed lay panel that works through that record, much as deliberative polls and citizens’ assemblies have done on technical questions, from electoral reform in British Columbia to constitutional change in Ireland. The committee informs the public but does not stand in for it. PRF then measures whether a model weighs the trade-offs as an informed public would, once the record is set.

In the most specialized fields, a lay panel will track whichever expert framing proves most persuasive, so PRF there measures fidelity to the committee’s framing as much as to the public’s reasoning. The upshot: in technical domains, a PRF score is only as good as the committee that built the record. Avoiding these pitfalls will require robust, carefully considered oversight of the entire PRF process. This is why the institutional design that follows is of critical importance to PRF’s odds of success.

Designing an Institution to Evaluate PRF

A process as consequential as PRF would need an institutional home. At the federal level, Congress could create a standing “Commission on Public AI Use,” housed within the Center for AI Standards and Innovation, which is home to the leading AI experts within the federal government. This commission could oversee the curation of hypotheticals, the selection of representative public panels, and the comparison between public responses and model responses (note that states should also explore the creation of such bodies—the focus of this essay is at the national level). These scenarios might subsequently be shared with subnational agencies.

The commission could rely on domain-specific committees to develop hypotheticals. Within the commission, a judicial-use committee could include former judges, legal scholars, practicing attorneys, court administrators, technologists, and members of the public. A law-enforcement committee could include former prosecutors, defense attorneys, civil rights lawyers, police officials, local-government representatives, and community members. Importantly, each committee would develop sealed hypotheticals designed to test the kinds of questions that may arise in that domain. Of course, those hypotheticals would not be disclosed until the testing period, to reduce the risk that labs train to the test.

The public panels, not the committees, are the basis of the benchmark. Each committee builds the record and writes the questions. But, to be abundantly clear, its panel renders the judgment that the model is scored against. “Representative” here refers to panel members being drawn by lot and stratified to mirror the domain’s relevant population, on the deliberative-polling model described above. For a model used in immigration adjudication, for example, the panel should reflect the demographics of the communities that appear before immigration courts. Panel members are not appointed, and they may not have their judgment usurped by a committee of experts.

Each representative panel would review common factual records and competing arguments before producing their own outcomes and reasoning. AI models would be given the same materials. The committee would then compare the models to its public panel across two dimensions: whether the model reached a similar outcome and whether its reasoning reflected the same concerns, priorities, and limiting principles.

The commission appointment process should be designed to avoid political capture. What stops a new administration from reshaping the exercise to fit its preferred style of governance? The panel resists capture on its own terms because no one can pack a lottery. Each cycle draws a fresh random sample, so partisanship enters the panel only in the proportion it holds in the population, and it enters each time anew. The commission, however, whose members are appointed, is more exposed to political machinations. Congress should armor it as it armored the US Sentencing Commission, which caps single-party membership at a bare majority of seats, sets staggered six-year terms that outlast any single presidency, and permits removal only for cause.

Precedents

While setting up new oversight bodies and processes designed to operate well over the long term is a significant challenge, it has been done successfully before.

The proposed institutional design would not be entirely novel. The US Sentencing Commission referenced above offers a useful analogy. It operates in a highly sensitive domain, translates legal and policy judgments into structured guidance, and attempts to promote consistency without eliminating judgment. The commission establishes sentencing policies and practices for the federal criminal justice system. Congress took due care to place expertise within the commission—including designated slots for former judges—while still leaving tremendous discretion to the judges tasked with applying the commission’s recommendations for sentencing lengths.

The Administrative Conference of the United States serves as another example. ACUS does not run agencies, but it studies administrative practice and issues recommendations designed to improve fairness, efficiency, and accountability across government.

A Commission on Public AI Use would play a similar role for AI adoption. Importantly, it would not replace elected officials, judges, agency heads, or procurement officers. It would give them the results of a benchmark developed in a participatory process. Rather than relying solely on vendor claims, internal testing, or the informal preferences of government employees, public institutions would have access to a structured assessment of how competing models reason through hard cases compared with the considered judgment of the people those institutions serve.

Some will view this process as premature or overly cumbersome. That critique overlooks a more immediate reality: AI models are already shaping how government actors make consequential decisions, yet the public has almost no meaningful role in overseeing how those systems are selected or evaluated. While the PRF mechanism may not be perfect, an oversight system must be developed. Absent such oversight, there’s a risk of civil servants, agency heads, and elected officials trying to blame poor decisions on AI. Those excuses would have far less weight if the public and policymakers alike knew more about how models operate in particular contexts.

PRF does not hand all decisions to the public, but it allows public scrutiny of government AI tools. One obvious concern with this proposal is that public opinion is not synonymous with sound governance. Presidents, judges, legislators, and agency officials routinely make decisions that depart from majority sentiment because they are bound by constitutional constraints, institutional obligations, classified information, or technical expertise unavailable to the general public. As noted above, a model that perfectly mirrors public sentiment may therefore still be poorly suited for certain governmental functions.

That concern should shape how PRF is understood. The purpose of PRF is not to hand public polling the reins of government decision-making. Nor is it to create a plebiscitary mechanism for selecting AI systems. The point is narrower and more practical: public institutions should know whether the models they rely on consistently reason through difficult questions in ways that diverge from the public.

At present, that divergence is almost entirely invisible. Agencies, courts, and legislatures may adopt tools whose assumptions, value judgments, and interpretive tendencies subtly shape official decision-making, without any meaningful public scrutiny. PRF would help surface those tendencies. In some cases, decision-makers may conclude that a model’s divergence from public reasoning is justified by legal doctrine, technical realities, or institutional constraints. In others, that divergence may raise concerns about legitimacy, accountability, or democratic responsiveness. Either way, the divergence itself should not remain hidden from the public.

The Public Should Choose Public AI Values

Government adoption of AI should not proceed as though model selection is ordinary software procurement. When public institutions rely on systems that reason, rank values, frame trade-offs, and influence official judgment, the public has a legitimate interest in knowing how those systems think through hard cases. PRF would not answer every question raised by government AI use and may not be useful in certain domains, but it would make one question harder to avoid: whether the models acting in the public’s name reason in ways the public can recognize, evaluate, and contest. That is the minimum a democratic government should demand before allowing private model choices to become public governing defaults.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Kevin Frazier is the Inaugural AI Innovation and Law Fellow at Texas Law.

Andrew W. Reddie is an Associate Research Professor at the University of California, Berkeley’s Goldman School of Public Policy, and Founder and Faculty Director of the Berkeley Risk and Security Lab.

How should governments regulate the most advanced AI systems? One possible answer is that they should not. Libertarian-minded writers have made the case for a culture of “permissionless innovation” for AI development, in which the role of government would be limited to enforcing existing laws and facilitating industry self-regulation with “soft law” tools such as voluntary standard-setting. Another possibility, more in vogue across the aisle and the pond, would invoke the precautionary principle, using heavy-handed regulation to restrict the development of frontier AI models until developers can adequately prove that their systems are safe.

I think there’s a better approach. In a new essay, my co-author Christoph Winter and I make the case for a governance strategy that we call “radical optionality.” The idea is simple: governments should avoid over-regulation in the short term while building up the institutional capacity needed to competently regulate extremely advanced or “transformative” future AI systems when and if these systems come into existence. The point of this approach is to maximize optionality by providing our institutions with tools that can be used to respond to a wide range of foreseen or unforeseen future developments.

In this piece, we explain the rationale behind maximizing optionality while AI’s future impacts remain uncertain. We also outline some example measures that governments can take and explain how this approach dovetails with other proposals for AI governance.

Addressing AI Risks Under Uncertainty

Leading AI researchers in academia and industry have claimed that advances in AI capabilities may soon produce “AGI,” “artificial superintelligence,” “powerful AI,” or some similar term. If you are certain that these statements are hype, and that such advanced AI systems will not arrive during our lifetimes, I won’t try to convince you otherwise; enough ink has been spilled on the subject that I’m not optimistic about my ability to contribute anything new. But if you think there is even a small chance that these predictions materialize, or if you find them at all credible, we think that the argument for radical optionality is overwhelmingly strong. The argument goes as follows.

AI’s future impacts are highly uncertain. Assume that there is some possibility of transformative AI systems being invented within the next, say, 15 years or so. Most of us are extremely uncertain about exactly how and when this will happen, what the characteristics and tendencies of these systems will be, what benefits they will offer society, and what risks to public safety and national security they will create. Under some assumptions, these systems will be mostly harmless and highly beneficial, because the companies creating them will have incentives to make them safe and broadly aligned with human preferences. Under other assumptions, these systems will be dangerous and difficult to control—perhaps even capable of causing human extinction if the right guardrails are not put in place. Maybe securitization is inevitable and the U.S. government will soon spring into action and develop these systems behind closed doors as part of a clandestine military project. Alternatively, perhaps development will happen in a decentralized and democratic way.

Subscribe now

The question of regulation seems to present a tradeoff between innovation and security. Debating which of the scenarios described is most realistic can be valuable, but ultimately only ideologues claim to be certain about the future course of a poorly understood emerging technology. The rest of us have to make important decisions about what to do while acknowledging substantial uncertainty. On the one hand, restrictively regulating AI companies would slow down innovation, potentially depriving society of some of the benefits of technological progress. On the other hand, it is possible that well-designed regulations could mitigate the real risks that AI systems pose, both now and in the future. How should we think about this tradeoff between innovation and security?

Measures that maximize optionality can improve security without hindering innovation. We think that this framing misses an essential point: there are steps governments can take now that would increase security without any significant cost to innovation. At the top of the list are light-touch information-gathering authorities like whistleblower protections, reporting requirements, and transparency mandates. Government agencies thrive on a diet of information; it has been said that “information is the lifeblood of good governance.” Authorities that increase the government’s access to important information about AI risks—and allow the relevant agencies to develop expertise in securely processing and interpreting such information—are foundational building blocks for future governance efforts. Mechanisms for securely and intelligently sharing information within government, and (when appropriate) between governments, are similarly foundational.

Building capacity directly is also important. First and foremost, this means enabling the relevant regulatory bodies to hire and retain elite talent. Meta’s recent hiring spree, featuring 10-figure compensation package offers for top AI talent, is an example of what it looks like when an organization takes the prospect of transformative AI seriously. Governments will likely be unable to compete with the salaries on offer in the private sector, but reforms to processes for government hiring and contracting of AI talent are nevertheless needed in both the U.S. and the EU. The early successes of the UK’s AI Security Institute, which receives 10 times the funding of its U.S. counterpart despite the UK’s relatively modest GDP and industry relevance, demonstrates the importance of cultivating talent in government.

The full-length essay discusses a number of other optionality-increasing policy decisions, such as incentivizing lab security, avoiding premature and overbroad preemption of state laws, and building out an ecosystem for model assessments and evaluations. But the important thing is to recognize that security and innovation are not conflicting priorities, because there are ways to increase optionality without creating any significant barriers to technological progress.

Supporting Both Security and Innovation

Radical optionality is compatible with other proposals for AI governance. Radical optionality is by no means the first AI governance framework to recognize that governments have an important role to play while also acknowledging that overly restrictive regulation could hinder innovation. Dean Ball and Gillian Hadfield and Jack Clark have proposed sophisticated private governance regimes in which the government would certify an ecosystem of private regulators competing to offer efficient and nimble regulatory services to companies on an opt-in basis. Gabriel Weil has argued that a well-designed tort liability regime, featuring insurance requirements, punitive damages, and strict liability for certain harms, could force AI companies to internalize any risks generated by their products. And Cary Coglianese has advocated for a system of management-based regulation, requiring AI companies to take risk mitigation measures but allowing them broad discretion over what measures to implement and how. I view these proposals as consistent and compatible with radical optionality; tort liability, management-based regulation, and private governance mechanisms are valuable tools for maintaining and increasing optionality.

Under most worldviews, radical optionality is preferable to the status quo. Of course, not everyone will agree that an optionality-maximizing approach is wise or sufficient. If you confidently believe that the safety benefits of a restrictive AI regulatory regime would outweigh the costs to society of slowing innovation, it is reasonable to suggest that anticipatory regulation is needed. From this perspective, radical optionality does not go far enough, but it would still be preferable to the status quo. On the other hand, from a libertarian perspective, building up government capacity to regulate and promising that it will not be used prematurely might look a lot like giving the government a hammer and promising that agencies will not start hallucinating nails. I can’t promise that there is no chance of new authorities being abused, or that everyone will agree on when dual-use AI systems have become so advanced that regulating them is a national security imperative. But I do expect building government capacity to benefit AI companies as well as the public in the long term. If rapid progress in AI capabilities research gives rise to a surge in public demand for regulation at some point in the future, as some writers have predicted, companies might prefer for the government to have the option of regulating in a competent, targeted manner.

At my organization, the Institute for Law & AI, we spend a lot of time thinking about how advanced AI systems should be governed in the present and in the future. Radical optionality is a sort of organizing principle and guiding philosophy for that research and consulting work. When deciding what projects to work on, what bills to offer feedback on, and what policies to push for, the question of what approach will maximize optionality is typically one of the foremost considerations. In publishing this paper, I hope to convince at least a few people to adopt this framing, to recognize the importance of optionality, and to begin viewing security and innovation as compatible rather than conflicting priorities.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Charlie Bullock is a Senior Research Fellow at the Institute for Law & AI. He advises state and federal policy makers on AI governance topics and publishes research on legal questions with significant practical relevance to U.S. AI policy. His recent research examines issues including federal preemption of state AI laws, federal and state AI whistleblower protection legislation, and the likely consequences of the end of Chevron deference for the future of AI regulation. Charlie received his J.D. from Yale Law School, where he was an Editor for the Yale Journal on Regulation.

When officials in Washington warn about losing the AI race to China, the conversation turns quickly to military and economic advantage—and rightly so. Advanced AI will reshape everything from weapons systems to medicine, with massive implications for our geopolitical competitiveness. But beneath the great-power competition on AI lies a moral one. The ethical character of the most transformative technology in generations—one that will mediate an ever-larger share of human experience—will be a byproduct of superpower rivalry. At stake is the future of the relationship between individuals and the state, of privacy and control, of human agency and algorithmic authority.

The fear is not merely that China might build better systems, deploy them more widely, or out-sell American competitors. It is that those systems could carry a tide of new norms shaped by a government that surveils its citizens, suppresses dissent, harbors eugenic ambitions, and treats individual autonomy as a problem. As AI comes to dominate our lives as thoroughly as digital media already has—shaping our health and finances, how our children learn, how we are tracked, even our species’ genetic makeup—this battle for AI’s soul will affect us all intimately.

The contours of that battle are almost always left unexamined, but usually assume one of three forms: some leaders suggest AI’s values will be a winner-takes-all byproduct of the race to technical superiority; others imply a conscious struggle to spread tools and platforms with value systems baked in; still others insist that diplomatic cooperation between AI powers is the only way to bend AI’s ethical arc toward humanity’s benefit. Leaving these three paradigms implicit does everyone a disservice, robbing the United States of both moral clarity and strategic opportunity.

Looking at the history of technological competition, there may be truth to all of these three models. But each implies a different approach to maintaining American leadership while preserving the values we claim to champion. And analyzing them clearly reveals how badly our attention is skewed. Today’s debate fixates on breakaway dominance, blinding policymakers to the more consequential contests over encoded values and strategic diplomacy. A rebalanced approach would require something we currently lack: a clear, affirmative vision of what American AI should be for.

Breakaway Tech Dominance

“AI offers the potential promise of extending American hegemony.”

—Tyler Cowen

Breakaway tech dominance is the default (if often implicit) ambition for many leaders invested in the Sino-American AI competition. The thinking goes that if the US is able to master AI ahead of others, that advantage will translate into a general offset in American power over China, with powerful ripple effects in economics, culture, and politics globally. Because this winner-takes-all vision provides such a clear motivation for forging ahead, it is the one most often invoked by pro-tech voices and government leaders.

Historically, large technological advantages tend to precede hegemonic power. This dominant narrative about AI competition draws heavily on historical analogy. Britain’s industrial revolution produced not just economic advantages but also cultural ascent. British institutions, law, language, and ideas spread across the globe on the strength of steam engines and mechanized looms. And this was not history’s first instance of technological offset, a dynamic that has persisted since before the Assyrians’ mastery of iron metallurgy expanded their influence over rival groups.

Applied to AI, this logic suggests that the first nation to achieve a decisive breakthrough could gain civilizational escape velocity. According to this view, if China masters AI before America does, Beijing’s authoritarian model—surveillance systems, social credit schemes, algorithmic control of information and behavior—would spread globally with irresistible momentum. Given that many experts expect the AI transformation to be comparable in scope to industrialization or the dawn of the Iron Age, such fears are justified. Whether or not there are dramatic power shifts between the United States and China in the century ahead, AI is certain to play an outsized role.

According to this paradigm, the singular priority must be aggressive progress in AI capabilities. The strategic implication of this perspective is obvious: there is nothing so important as moving faster than China in pushing the bounds of AI technology. Additionally, there is little need for the United States to consider how American values relate to its AI strategy, because they are seen as downstream of the technical rivalry. In other words, if the United States establishes a decisive AI lead, its values will organically spread; if China masters the technology first, Beijing’s moral vision will take root globally.

An AI lead sufficient to achieve hegemony is unlikely to appear on either side. The breakaway-dominance framework functions only if there is a defensible breakthrough to be had, which is not necessarily the case. Some predictions of a superintelligence “takeoff”—in which a sufficiently advanced AI system starts to improve itself better and faster than humans could—fit that mold. But despite regular predictions of imminent AGI breakthroughs, even many bullish researchers are increasingly skeptical of such a scenario, making the prospects of a highly dominant and defensible AI hegemon seem unlikely.

US-China competition is also too tight for breakaway dominance to occur. The observable pattern of AI progress in recent years suggests a different path. China has successfully positioned itself as an aggressive fast follower. In frontier models, the most competitive arena of AI competition, Chinese labs tend to trail American counterparts by just months at a fraction of the cost. That is an achievement in itself—and it diminishes the likelihood that the United States will achieve a sustained, decisive advantage. While not impossible, it’s unlikely that we will see either country develop and maintain an AI lead significant enough to extend Chinese or Western values globally for any sustained length of time unchallenged. This winner-takes-all model, despite its implicit prominence in many policy discussions, almost certainly misses the full picture.

Encoded Values

“China is doing everything it can to dominate AI globally, and they will program the AI with Chinese values…. We’ve got to double down and make sure that American values are the values of the world, and that we control this global AI agenda.”

—Former US Senator Kyrsten Sinema (I-AZ)

A second model for looking at the moral stakes of Sino-American AI competition is the spread of encoded values: the ethics that are baked into new technologies, whether deliberately or subconsciously. This dynamic is ancient: Roman aqueducts built republican virtues into stone by distributing water first to public fountains, then to public baths, and only later to private homes. Fast-forward to the present day, when the internet stands out as a technology consciously designed with libertarian principles: decentralized architecture, open protocols, and resistance to central control. The resulting technology reflected those values in its most basic protocols (if only initially).

The same will be even truer of AI, given its unique ability to absorb and instantiate value systems. The protocols and architecture around AI systems may also reflect value decisions, but particular moral visions and preferences can also be directly distilled in today’s AI systems—or rooted out of them.

China has been explicit about its intentions to imbue AI with its own values. Official Chinese government regulations mandate that frontier AI systems must “uphold core socialist values”—that is, they must adhere to the Chinese Communist Party’s totalitarian view of history and morality. Chairman Xi Jinping has already made considerable strides toward that end. Beijing invests tens of billions of dollars annually in building a techno-authoritarian ecosystem of tools, platforms, standards, and norms aligned with state priorities: social stability, party authority, and collective “harmony,” rather than individual autonomy. Its companies are experimenting with novel, AI-powered methods of conducting surveillance, enhancing censorship, and even predicting political dissent before it occurs.

US efforts to impart values into its AI ecosystem have been less concerted. The United States has been far less deliberate than China in developing AI consonant with American values. True, documents like the Biden administration’s “AI Bill of Rights” and companies’ interminable desire to write AI-principles documents at least pay lip service to the idea of aligning emerging AI systems with democratic principles. The clearest example of this might be Anthropic’s approach to “constitutional AI,” which aims to evoke the US Constitution in its model operations. And on balance, American AI companies’ systems pay much greater attention to ethics and safety concerns than their Chinese counterparts do. But while these examples reflect a different culture around the development of AI in the United States, they are often only window dressing, and pale in comparison to the concerted state focus that Beijing exerts on the normative trajectory of China’s tech sector.

In practice, American AI may actually erode American values more than it supports them. Indeed, American companies have historically been indispensable in building out China’s techno-authoritarian ecosystem. Between public discourse-corrupting deepfakes, microtargeted political manipulation, and algorithmic amplification of extreme content, leading US developers are already arguably producing AI tools that weaken democracy more than they strengthen it. The governments of both superpowers are working to reap the efficiency benefits of AI in their state bureaucracies. But whereas China’s regime is laying out a proactive vision for how AI will advance authoritarian control, the United States has been largely reactive in accommodating AI to democracy—waiting for courts to adjudicate how new technologies can or cannot be used according to existing American law.

The US needs a robust vision for democratic AI and the will to disseminate it. Breakaway-AI proponents see AI’s future as a straightforward innovation race with downstream ethical repercussions. By contrast, proponents of AI as a system of encoded values see the future as a struggle over vision and will. To further American values and strengthen democracy, this view would require developing a much clearer vision: a compelling idea of how to use AI—not just by reactively implementing guardrails but by proactively conceptualizing what democratic AI should look like and enable. Will, equally important, is the drive to commercialize and aggressively spread the resulting systems around the world in collaboration with allies, while also preventing domestic companies from working with China in ways that undermine the United States’ vision.

The Trump administration, by focusing largely on will, has seen some gains in diffusing US technology. But the PRC still maintains considerable diffusion advantages—especially in the Global South, where China’s price-point advantages and shared development conditions give it an edge in building out AI infrastructure for developing nations. A values-driven AI vision, as described above, remains comparatively underdeveloped on the American side: Washington lacks an inspiring, affirmative narrative about what democratic AI enables for its citizens that authoritarian AI cannot.

The US vision deficit has downstream effects on will. Without a clear, compelling sense of what American companies are building toward, it is impossible to muster the political energy needed to make hard choices. For example, US firms are currently bolstering Beijing’s AI ecosystem in sensitive domains like biotechnology, with little oversight. A stronger vision of American AI would galvanize support for restricting US companies from making such contributions to China’s AI ecosystem.

Emergent Control Regimes

“What Soviet-American nuclear arms control was to world stability since the 1970s, U.S.-Chinese A.I. collaboration to make sure we effectively control these rapidly advancing A.I. systems will be for the stability of tomorrow’s world.”

—New York Times columnist Thomas L. Friedman

The third paradigm for how the moral future of AI hinges on Sino-American competition is that of emergent control regimes—the shared rules and institutions that powers build over time to govern consequential technologies. For policy wonks, this is often the most overlooked—or, more accurately, the most dismissed—avenue for shaping outcomes. In part, it is not taken seriously because those who do raise it tend to do so with flagrant naivete about the weakness of the multilateral system and the political infeasibility of any good-faith agreement between the United States and China. But the idea that international control regimes might emerge over time and prove influential is not far-fetched, particularly in areas such as lethal autonomous weapons and AI-powered human gene editing.

The nuclear era shows that self-interest can drive even rivals to manage powerful technologies together, if imperfectly. Early in the Cold War, the idea that the United States and the Soviet Union could reach an agreement about nuclear weapons seemed fanciful. Nonetheless, both nations came to recognize that countering proliferation was in each nation’s interest, even as they remained locked in an existential nuclear arms struggle. No one in 1945 could have predicted the specific contours of what emerged from the complex, path-dependent interactions among nuclear-armed states with evolving interests over decades: the Nuclear Non-Proliferation Treaty, the International Atomic Energy Agency (IAEA), test-ban agreements, verification mechanisms, and norms around nuclear use. While imperfect, these innovations have unquestionably shaped and constrained the most destructive technology that humanity has yet produced.

Advantages will accrue to whichever power crafts and brands politically feasible international controls. To be sure, there are limits on the degree to which such controls can be planned for, given how contingent they tend to be on changing relations and events. But this is not to say that any control regimes emerging from rapidly advancing AI systems are too contingent to plan for in any way. Such controls are not to be confused with the idealistic proposals—such as Pugwash-style scientist convenings or bilateral red-teaming exchanges—that are characteristic of most current track-two dialogues. Nor are they the feel-good unilateral pronouncements of rosy intentions like the resolution on “the promotion, protection and enjoyment of human rights on the Internet,” passed six times by the UN Human Rights Council since 2012. As with nuclear controls, any diplomatic development of consequence will likely be highly controversial, and will necessarily fall far short of what most peace-loving technologists would like to see.

But imperfect measures can still be strategic. President Eisenhower’s famous “Atoms for Peace” speech in 1953 set the foundation for the IAEA. It also served as a tremendous public relations victory for the United States, projecting America as the responsible superpower, willing to help other countries benefit from peaceful applications of atomic technology. It forced the Soviet Union to compete with the United States in building nuclear reactors for other countries, at a high cost to the Soviets. And subsequent US-Soviet nuclear arms control negotiations did more than help to constrain the risks of nuclear war; they also allowed the United States to pursue advantages in qualitative force capabilities at lower cost, under the auspices of quantitative weapons restrictions.

Compared with China, the US is better positioned to lead emergent control regimes. Technologists have given a great deal of thought to unrealistic controls for theoretical future AI capabilities. Yet little serious thought has gone into diplomacy in those areas where international controls could be made politically feasible, soft-power enhancing, and strategically advantageous. The partial exception is the American-led Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy, which has made strides toward establishing American leadership in rules around the use of lethal autonomous weapons. This guidance is both strategically beneficial to the United States and resonant with American values. Several other areas show promise for similar interventions, not least the ethically fraught genomic applications of emerging AI-powered biotech and the use of AI in high-risk industries. Here the United States has substantial untapped advantages: a global network of allies, a strong history of effective tech diplomacy, and a brand of AI development unencumbered by China’s dystopian techno-authoritarianism. But these advantages so far have not deterred China’s ambitious efforts to eke out a leading position in global AI governance.

A Rebalanced Approach

These three paradigms of AI competition—breakaway dominance, encoded values, and emergent control regimes—are not mutually exclusive. Some areas of AI may see defensible technological breakthroughs that confer long-term advantages; some will become battlegrounds for embedded values; some will develop controls; and some will combine elements from several of these paradigms. They are also interrelated: if one power successfully embeds its values into a widely adopted technology, it will likely occupy a privileged position in control discussions, for example. The question is not which single model is most accurate, but how to allocate attention and resources across all three, and for which issues.

Yet today’s focus remains mistakenly skewed toward a winner-takes-all narrative, blinding policymakers to more consequential contests on encoded values and creative thinking on strategic diplomacy.

To take one example, while initial US nuclear dominance was essential, it was ultimately short-lived. Many developers of the weapon believed America’s 1945 breakthrough would represent an enduring strategic advantage, similar to how many see the race to superintelligence as today’s single defining competition. But America’s nuclear dominance lasted just four short years. The Manhattan Project was indispensable—the United States’ adversaries getting the bomb first would have been catastrophic. But those banking on sustained dominance were in for a rude awakening. Ultimately, clever nuclear diplomacy contributed more to the United States’ victory over the Soviet Union than breakaway nuclear superiority, which never materialized.

Encoding values in technology requires proactive efforts. To the extent that a technology as broad as AI can be compared to a recent innovation, the best analogue is probably the internet—unfortunately, another cautionary tale. Although American engineers deliberately built the internet with libertarian principles, China has been able to co-opt it through force of will. Today, the Great Firewall and the other tools that the CCP has built into the Chinese internet have transformed a freedom-enhancing technology into history’s most sophisticated instrument of surveillance, censorship, and control.

Beijing is exporting these tools abroad, enabling other autocracies to turn the internet away from its original open-society-enhancing design toward repressive ends. The story might have turned out differently if the United States had engaged in more proactive diplomacy—leveraging its unique influence over the development of China’s internet, instead of just issuing feel-good digital-rights statements. At a minimum, curbing American tech companies’ active support of Chinese technological ambitions would have slowed Beijing’s successful authoritarian conquest of the internet. Indeed, the extent of American support for techno-authoritarian progress casts serious doubt on any assertion that the originating society of a technology will organically imbue that technology with its own values. For the war over embedded values, the internet’s lesson is clear: technology neither establishes nor preserves values passively.

The US cannot rely on technological dominance to ensure that AI furthers American values. The United States must learn from these historical cases quickly. AI-powered Chinese “smart cities” are already spreading across the Global South, bringing with them surveillance architectures designed for authoritarian control. Cheap, CCP-compliant Chinese open-source frontier models are already gaining uptake internationally. Party-aligned research centers are developing AI-powered propaganda and censorship tools with unprecedented sophistication; these will soon be diffused abroad, if they haven’t already. Policy and tech leaders may think that their efforts to simply accelerate American technical progress at the frontier of AI innovation will ensure that American values triumph. However, the more probable outcome could be a world awash in cheap, authoritarian AI that outcompetes slightly more sophisticated American offerings that do little to promote American values—and perhaps even erode them.

To course correct, the US must establish a President’s council or congressional commission on democratic AI. A misguided fixation on a winner-takes-all race for technical superiority, as a proxy for a competition of values, risks missing where the real competition lies. It also misses opportunities to rout China diplomatically, similar to America’s successes in nuclear diplomacy. A better approach to AI must start with developing a clearer moral vision for American AI. The President’s Council on Bioethics, established under former President George W. Bush, offers an effective model of what this could look like: a substantive body bringing diverse perspectives to the highest levels of government to grapple with emerging ethical challenges, producing influential reports that shaped discourse and policy. A comparable council on AI and democratic governance could build the intellectual foundations for techno-democracy that do not yet exist, as the American Enterprise Institute’s Council on AI Ethics is beginning to show.

Armed with a clearer, more compelling moral vision for AI, American technologists and policymakers could be galvanized toward supporting the United States’ competition with China with sharper focus. Such a vision would also provide a basis to more aggressively curb American companies’ substantial aiding and abetting of China’s techno-authoritarian ecosystem. And it could provide a stronger foundation for closer collaboration with indispensable like-minded partners such as India: nations better equipped to compete with China on rolling out price-competitive and context-relevant AI offerings in the Global South.

There is little doubt that the Sino-American battle over AI will have tremendous consequences for the future of humanity. Approaching that contest with greater moral clarity is not just the right thing to do; it is also a strategic imperative.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Bill Drexel is a senior fellow at Hudson Institute. His work focuses on United States–India relations, artificial intelligence competition with China, and technology in American grand strategy. Previously, Mr. Drexel worked on technology and national security at the Center for a New American Security, humanitarian innovation at the United Nations, and on Indo-Pacific affairs at the American Enterprise Institute. Drexel’s field experience includes serving as a rescue boat driver during Libya’s migration crisis, conducting investigative research in the surveillance state of Xinjiang, China, and supporting humanitarian data efforts across wartime Ukraine.