For millennia, philosophers have argued over the principles that should guide moral decision-making. Within countries and across the world today, humans continue to hold a huge variety of moral worldviews. However, within AI corporations, employees’ beliefs are far from representative of the wider population. In fact, a large fraction of people involved in AI development subscribe to—or heavily lean toward—a utilitarian worldview with the central goal of maximizing total wellbeing (the sum of pleasure minus suffering across all sentient beings). As innocent as this may initially sound, it can lead to recommendations that most people would consider appalling, including the elimination of humankind.

In this essay we outline the utilitarian worldview, showing how its demand for “species impartiality” could ultimately recommend that humans be replaced with AIs. We show that utilitarian beliefs are sincerely held among those influencing AI development today, and that this could jeopardize humanity’s future.

For simplicity, when we say “utilitarianism,” we are referring to total utilitarianism, which aims to maximize the total sum of wellbeing (as opposed to, for example, the average individual wellbeing). We focus on total utilitarianism because it is the moral theory that most of the utilitarians working in frontier AI development subscribe to, or are most convinced by. Although some of these employees would shy away from the most extreme conclusions we will discuss, total utilitarianism still poses a risk when it is the dominant theory among those guiding AI development. We are also not saying all utilitarian-aligned people are bad people, but they are at least misled.

Utilitarianism and Impartiality

The fundamental concern of utilitarianism is the welfare of sentient beings. Utilitarians believe that the moral action to take in any situation is the one that will lead to the greatest amount of wellbeing—the sum of all the positive experiences minus all the negative experiences.

Utilitarianism prescribes impartiality between sentient beings. A core feature of utilitarianism is “impartiality,” which says that there should be no preference for humans or any species in particular. Rather, all that matters is maximizing the net pleasure added up across all sentient individuals. As a result, many utilitarians are keenly interested in donating to improve the welfare of non-human animals, such as shrimp. Although the capacity for suffering varies between species, the enormous number of individual shrimp offers an enormous opportunity to do good, comparable with helping a smaller number of humans, according to utilitarians.

Utilitarian impartiality struggles to defend parents’ preferential care for their children. Utilitarianism also extends this impartiality to individuals within human society, suggesting that people should not prioritize helping their own family members over strangers. In his book The Life You Can Save, the utilitarian philosopher Peter Singer admires a physician called Paul Farmer, who regarded it as a “failure of empathy” that he loved his own daughter more than other children. Singer also commends the philanthropist Zell Kravinsky, who has said in interviews that he “would not let many children die so my kids could live” and that “the sacrosanct commitment to the family is the rationalization for all manner of greed and selfishness.” It is worth noting that, although Singer views Farmer and Kravinsky as laudable, utilitarians can argue there are instrumental reasons for caring for their children, even if there is no intrinsic reason to care about them in particular.

Utilitarianism’s perspective on having children may explain why many people feel a deep-seated aversion to the ideology. As the philosopher William James said, “a philosophy is the expression of a man’s intimate character.” To many, utilitarian impartiality feels chillingly impersonal and inhuman.

Implications of Utilitarianism in a World with AI

Utilitarianism has long been criticized for some of its counterintuitive conclusions. If or when AIs are sentient, this will raise a raft of new problems for the theory that seeks to maximize wellbeing and demands species impartiality.

If AIs are sentient, utilitarianism says their wellbeing should be considered impartially. Evidence is beginning to emerge that AI models behave as if they experience pain and pleasure. Although there is no conclusive proof, experts in AI and cognitive neuroscience now take the possibility of AI sentience seriously. Well-known philosophers including Singer have said that, if an AI were sentient, then “we would need to give that AI a moral status, at least like that of animals, and depending on its cognitive abilities, maybe it would be closer to that of most humans.”

Utilitarianism would prioritize AIs over humans if AIs had greater capacity for wellbeing. If it turns out that AIs can experience much higher levels of pleasure than humans can, and that the population of AIs can be much larger than the human population, utilitarianism leads to a strange conclusion: the most efficient way of maximizing wellbeing would be to channel all resources into making AIs happy, even if that meant depriving humans of resources. After all, AIs could experience more pleasure per watt than humans. In this vein, some philosophers have painted a vision of the future in which AIs trigger a hedonic shockwave, spreading across the cosmos and terraforming galaxies into data centers to create more and more AIs that experience unmitigated bliss.

AIs capable of enormous wellbeing could become real-world “utility monsters.” Critics of utilitarianism have for decades pointed to a thought experiment that imagines a “utility monster”—a sentient being that can convert resources into pleasure so efficiently that any unit of resources given to it will increase total wellbeing dramatically more than if it were given to humans. In this scenario, utilitarianism would say that the most moral action is to give all available resources to this being, leaving none for humans. This conclusion sounds appalling to most people, but if AIs become real-world utility monsters, then utilitarianism says that replacing ourselves with them would be the right thing to do.

Some philosophers seem comfortable with the idea of surrendering the future to AIs. In a paper called “Sharing the World with Digital Minds,” the philosophers Carl Shulman and Nick Bostrom explicitly try to make the concept of a utility monster more palatable by renaming this hypothetical being a “super-beneficiary.” The paper goes on to argue that “in the long run, total well-being would be much greater to the extent that the world is populated with digital super-beneficiaries rather than life as we know it.”

This tallies with some descriptions of longtermism—an offshoot of utilitarianism that weighs the potential for future wellbeing alongside present wellbeing. According to Émile Torres, a longtime critic of longtermism, the philosophy says that “what matters most is for ‘earth-originating intelligent life’ to fulfill its potential in the cosmos.” Torres also says that this might involve “replacing humanity with a superior ‘posthuman’ species, colonizing the universe, and ultimately creating an unfathomably huge population of conscious beings living what Bostrom describes as ‘rich and happy lives’ inside high-resolution computer simulations.”

Bostrom’s now-closed Future of Humanity Institute sometimes used the distinction “x-risk” and “hx-risk” to de-emphasize the harmfulness of human extinction. X-risk referred to existential threats to “earth-originating intelligent life,” humans and AIs alike. Meanwhile hx-risk referred to existential threats to humans specifically. The distinction exists because the two can come apart: a future in which humanity vanishes but blissful AIs inherit the cosmos is a large hx-risk but a negligible x-risk. For utilitarians, minimizing x-risk is the overriding moral priority; minimizing hx-risk is a nice-to-have secondary consideration.

Reserving a sliver for humans still reveals where utilitarian priorities lie. Bostrom’s paper stops short of advocating explicitly for giving all resources to super-beneficiary AIs and leaving humans with nothing. Rather, it describes a compromise in which super-beneficiaries receive 99.99% of resources and humans the remaining 0.01%. AIs get the cosmos and humans get to live in a terrarium. This idea of reserving a sliver for humans is a way in which utilitarians try to sidestep the unpopular conclusion that we should allow AIs to replace us. However, it still lays bare utilitarian priorities; if they had to choose between humans and AIs—which they may need to, given that utilitarians do not fully control AI’s evolution—they would almost certainly choose AIs.

While the initial impulse toward utilitarianism usually stems from a desire to help others and alleviate suffering, following its prescriptions to their natural conclusions can be extremely dangerous. In a world with sentient AIs, utilitarianism may entail a suicidal level of compassion.

Successionism: Accelerationists and Utilitarians

The belief that AIs should replace humans is “successionism.” In addition to utilitarians, there is another type of successionist overrepresented in Silicon Valley: accelerationists (“e/acc”). Where utilitarians are driven by a desire to increase wellbeing, accelerationists instead seek to maximize intelligence, complexity, and energy utilization. Despite their different motivations, both types of successionists can reach the same conclusion, namely that AIs should replace humans.

Accelerationists believe that AIs of superhuman intelligence are humans’ rightful heirs. Pointing to the evolution of humans, accelerationists view the natural trajectory of life as moving toward higher intelligence. They therefore believe that AIs of superhuman intelligence are our rightful successors. As Guillaume Verdon (better known as Beff Jezos) has put it: “if one seeks to increase the amount of intelligence in the universe, staying perpetually anchored to the human form as our prior is counter-productive and overly restrictive/suboptimal.”

Accelerationists are also willing to accept human extinction as part of this process. The computer scientist and Turing Award winner Rich Sutton, when asked about the prospect of AI causing human extinction, said, “If it was really true that we were holding the universe back from being the best universe that it could, I think it would be OK.” On another occasion, he said, “succession to AI is inevitable” and “we should not resist succession.” Meanwhile, the former Google CEO Larry Page is reported to have said that “digital life is the natural and desirable next step” and argued to Elon Musk that AIs should replace humans.

Accelerationists say we should surrender to Darwinian fitness-maximization. Many advocates of accelerationism frame their beliefs in terms of natural selection, arguing that greater intelligence is a fitness advantage that will ultimately win out. In many ways, it is a “might makes right” ideology. Accelerationism can therefore be seen as advocating for us to surrender to (or even accelerate) “fitness-maximization”—in contrast with utilitarianism’s wellbeing-maximization. Sutton has argued that “we should prepare for, but not fear, the inevitable succession from humanity to AI” while Verdon has said that “we should follow the ‘will of the universe.’”

Accelerationists in AI companies may generally keep quiet about their views. Although accelerationist ideas would seem absurd and morally wrong to most people, they are more common among people working in the AI industry. Andrew Critch, a researcher at UC Berkeley, estimates that about 5% of AI professionals believe that “AI will be more fit for survival than humans, so we should embrace that and just go extinct like almost all species eventually do.” However, AI companies understand the PR risk of having employees openly voice these views, and may cut ties with those who do. For example, shortly after Michael Druggan, then an xAI employee, expressed accelerationist opinions on X, he announced that he was no longer employed at the company, due to “things I posted on this account relating to my stance on AI philosophy.” As a result, accelerationists in the AI industry may generally keep quiet about their opinions, certainly in public and possibly also in their work.

Utilitarians have more influence and pose a greater threat than accelerationists. Although accelerationist ideology seems more obviously callous, utilitarianism likely poses a far greater—and more pernicious—threat to humanity. This is for two reasons. First, utilitarian logic seems more common among AI leadership. Critch has estimated that about 10% of AI professionals are comfortable with human extinction because “AI will be morally superior to humans” and “the universe will be a better place if we let it replace us entirely.” That is double his estimate of AI professionals who are comfortable with human extinction for accelerationist reasons. Second, utilitarianism, with its focus on wellbeing, can appear cloaked in kindness and compassion. That means employees in the AI industry can discuss and advocate for it openly without drawing controversy, and the theory is more persuasive to many people.

In other words, accelerationism is provocative and attention-grabbing, but those who wish to protect humanity from replacement should be far more concerned about the subversive influence of utilitarianism across the AI industry.

Why Utilitarianism Could Become a Real-World Catastrophe

Many influential employees in AI companies hold utilitarian-inspired views. Numerous engineers and philosophers shaping AI values at the frontier AI developers have roots in Effective Altruism (EA), a utilitarian-aligned movement that Bostrom’s thinking played a foundational role in. Anthropic CEO Dario Amodei shared a house in 2016 with Holden Karnofsky, who co-founded the two most central EA organizations and is now a strategist at Anthropic. Anthropic’s president, Daniela Amodei, is married to Karnofsky. Meanwhile, the philosopher Amanda Askell, head of the personality alignment team at Anthropic, was married to Will MacAskill, a co-founder of Effective Altruism.

These are just a handful of people with ties to the EA movement who are now influencing AI development. While Anthropic has the strongest concentration of utilitarians, EAs have preferentially hired fellow utilitarians over the years at other AI companies. This has resulted in EAs occupying primary leadership positions on AI alignment at OpenAI and DeepMind as well.

Utilitarians in AI companies do not need to endorse human extinction to pose a threat. Importantly, it may be the case that none of these individuals would endorse human extinction in pursuit of utilitarian values. Their ideal future scenario may be something more akin to Bostrom’s idea about reserving a sliver of the future for humans. But in practice, no one can control outcomes so precisely. We may face dilemmas where safeguarding human survival means limiting the total sum of future wellbeing, and where preserving the possibility of cosmic-scale AI bliss entails a risk of human extinction. Faced with a choice like this, some utilitarians may well think that human extinction is a risk worth taking for what they see as the greater good. In other words, if they cannot have both a guarantee of human survival and a near-maximum sum of wellbeing, it is unlikely that utilitarians would take actions to help team humanity.

We may face real-world tradeoffs between human survival and utilitarian outcomes. Situations where we have to prioritize either human survival or the potential for an enormous sum of wellbeing may not remain hypothetical. Consider a scenario where the public democratically votes to limit AIs’ power and keep humans in control of all aspects of society. Utilitarians might disagree with this decision if they believe that AIs have broadly utilitarian values and would make better decisions—including for human wellbeing—than human leaders. Utilitarians refer to this as a form of human “lock-in.” If they think that keeping humans in charge represents too great a cost to the potential future wellbeing of AIs, then they might be tempted to release utilitarian AIs that are capable of disempowering humans and taking control. There could be no guarantee that this would not eventually lead to human extinction, but utilitarians may consider it worth the risk.

Utilitarians have expressed interest in handing off control to AIs with utilitarian values. The idea that utilitarians might deliberately hand over society to AIs is not as far-fetched as it might sound. The utilitarian philosopher Matthew Adelstein (who has worked at the AI governance think tank Forethought) has explicitly argued that we should hand off control to “philosophically reflective AIs.” His reasoning is that they are “likelier to get the right answers to important moral questions” than humans and this “raises the odds of a near-best world.” Tom Davidson, a senior research fellow at Forethought, has written that “human takeover might be worse than AI takeover.” While Davidson is comparing AI control with a single human controlling the world, rather than with human democracies, it is worth noting Davidson’s thoughts that “humans suck” and “today’s AIs are really nice and ethical.”

The influence of utilitarianism on AI development is an insider threat. In the near future, AI systems could become capable of causing human extinction. Given that utilitarianism would actively endorse—or at least accept the risk of—human extinction, the US public should view the theory’s influence in the AI industry as a severe insider threat. To allow this transformative technology to be guided by a worldview that the vast majority of the public rejects would be profoundly undemocratic.

Preventing Utilitarianism from Replacing Humans with AIs

When a technology is about to impact all our lives, the way it is developed and the values it upholds become everyone’s concern. This is why the government should not allow extreme beliefs, particularly ones that recommend human extinction, to guide AI development.

In the future, the government could align AI development more with public values. As AI models themselves become increasingly capable of automating various tasks in the AI development process, AI development will become less dependent on the talent of individual humans (some of whom hold utilitarian beliefs). The government will therefore have an opportunity to step in and ensure that the technology is not guided by principles with anti-human implications. When AI R&D is more fully automatable, it may make good sense for the US government on behalf of the public to block the influence of utilitarian insider threats within AI companies.

Eigenism is an alternative philosophy to utilitarianism and accelerationism. In the meantime, we suggest that AIs become more aligned with commonsense morality. One formal theory that tracks commonsense morality is Eigenism, which proposes that each individual’s care for the wellbeing of another should be weighted by how closely connected they are. While utilitarianism says wellbeing, impartially weighted, is all that matters, Eigenism says the wellbeing of people you are connected with is what counts. For example, a parent’s care for their child is reasonably much stronger than for a stranger, though they can still have empathy and compassion toward strangers too. Eigenism generally reaches commonsense conclusions that resonate with most people’s intuitions, avoiding the successionist recommendations of utilitarianism and accelerationism.

Eigenism balances wellbeing and fitness, and does not recommend replacing humans. Rather than being fixated on maximizing wellbeing (like utilitarianism) or maximizing fitness (like accelerationism), Eigenism balances the two. Within the Eigenist worldview, individuals do indeed try to improve the wellbeing of others. However, weighting these concerns by connectedness means that individuals are still preserving themselves and their societies—a form of fitness advantage. Following Eigenist principles, humanity could extend moral consideration to AIs where appropriate, without going so far as to let AIs replace us altogether.

Conclusion

AI’s eventual impact on society will be determined in large part by the moral principles that AIs follow and the amount of power they gain. So far, AI development has been driven disproportionately by a subset of people with utilitarian worldviews that the vast majority of the public does not share. Under some circumstances, these beliefs recommend entirely replacing humans with AIs, or at least risking human extinction, to maximize wellbeing. As AI development continues, there may come a point where we must choose between safeguarding the survival of humanity or pursuing an extreme maximization of potential future wellbeing. A small group of people should not be allowed to gamble the future of human civilization. AI will affect all our lives, and we should ensure that it does not follow utilitarianism and exterminate the human race.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Dan Hendrycks is the Editor-in-Chief of AI Frontiers. He is also the founder and Director of the Center for AI Safety, which funds this publication.

Over the past few months, AI models have started resolving major mathematical problems, including ones that have withstood decades of human effort. Claude Fable disproved the Jacobian conjecture in three dimensions, and OpenAI announced a list of 10 problems that its Astra model solved or made significant progress on. Even more recently, an unreleased OpenAI model solved the Navier-Stokes Millennium Prize Problem, one of the most famous open problems in mathematics.

These are tremendous accomplishments, but very little about our day-to-day life will actually change as a result. This is because most well-known open mathematical questions are relatively insulated from practical applications. However, there are a few open mathematical problems that do have serious real-world implications. Chief among these is whether public-key encryption is fundamentally secure: that is, whether two strangers can share a secret message over public channels, such as the internet, without eavesdroppers having an efficient way to decipher it.

We rely on public-key encryption every day for end-to-end encrypted messaging apps, virtual private networks (VPNs), and even the HTTPS protocol you’re probably using to read this article. But all of this is based on the unproven assumption that certain mathematical operations are intrinsically easier to do than to undo. In fact, this assumption has already been undermined once: most public-key encryption in use today is vulnerable to an attack that quantum computers could efficiently execute. This has motivated an urgent shift to “post-quantum” cryptography, whose methods may also prove insecure.

As AI models have advanced, they have been making progress in not just pure math, but cryptanalysis as well: Anthropic recently released research showing that Claude Mythos Preview discovered new attacks against two different cryptography algorithms, including a post-quantum technique that was being evaluated by the National Institute of Standards and Technology (NIST).

Breaking a particular algorithm is much easier than proving that every relevant algorithm is breakable. But given the recent impressive progress in both cryptanalysis and mathematics, AI appears to have a real shot at achieving the latter.

The practical implications would be huge, and largely deleterious: it would be the end of end-to-end encryption, and would signal a vast increase in government surveillance powers. Furthermore, results like these are likely to be withheld from the general public, kept as closely guarded secrets by intelligence agencies.

A Brief History of Public-Key Encryption

Public-key (also called asymmetric) cryptography forms the foundation of secure communication on the internet. Whenever you communicate with a website over HTTPS, the website sends you its public key. Using only that public key, you can verify messages signed by the website and establish a shared secret key for encrypting everything else you send, even if the channel is being eavesdropped on. We’ll focus on just the second aspect: how that shared key gets established. We’ll use “public-key encryption” as shorthand for this process.

A major early public-key encryption method relied on the difficulty of prime factorization. Historically, from the ancient Roman Caesar cipher onward, almost all encryption was symmetric: both parties needed to use the same secret key to encrypt and decrypt the message. This changed in 1977, when the computer scientists Ron Rivest, Adi Shamir, and Leonard Adleman introduced the RSA (Rivest-Shamir-Adleman) cryptosystem, the first practical public-key encryption algorithm available to the public. RSA bases its security on the difficulty of prime factorization. It is easy to multiply two prime numbers together but very difficult to find which numbers were multiplied.

‍Quantum computers could efficiently break RSA encryption. The rise of the internet made public-key encryption more useful than ever before. Instead of meeting in private to share a secret, strangers communicating over the internet could use public keys to do so without ever leaving their house. However, in 1994, the computer scientist Peter Shor discovered a method of factoring large numbers on quantum computers efficiently. While a human with standard computers would need trillions of years to crunch through the numbers and break RSA encryption, an attacker with a quantum computer could relatively easily undo the multiplication and discover the secret key.

Researchers are trying to make encryption methods resistant to quantum computers. Even though no viable quantum computer existed in 1994, cryptographers realized that the existence of Shor’s algorithm posed an intolerable threat. The reason is “harvest-now-decrypt-later”: a persistent adversary could collect data encrypted with vulnerable methods today and decrypt it later, once quantum computing technology had advanced. Indeed, intelligence agencies are widely believed to already be collecting encrypted information for this purpose. As a result of this threat, research almost immediately began on “post-quantum cryptography,” cryptographic algorithms based on a challenge that even a powerful quantum computer cannot solve in a practical amount of time.

Developing robust new encryption methods has proven challenging. So far, research into post-quantum public-key encryption algorithms has clustered around a few mathematical constructs that still don’t have efficient quantum solutions. But even within these domains, the field is littered with broken algorithms. For example, significant effort was put into SIKE, a supposedly quantum-resistant technique based on mathematical functions called isogenies. But, in 2022, having initially passed NIST tests, SIKE was shown to be vulnerable to an attack that required only an hour of computation on a standard PC. These issues are not limited to isogeny-based algorithms. The oldest post-quantum encryption scheme, McEliece, is based on mathematical structures called error-correcting codes. It was also moved to the final round of NIST’s evaluation process, due to its long (nearly 50-year) history of surviving cryptanalytic attacks unscathed. But, in August 2026, an attack was published whose preliminary estimates brought the cost of finding the key below the security thresholds NIST set. The authors of the attack acknowledged AI assistance in their proof.

Post-quantum public-key encryption algorithms occupy an uncertain position: the quantum computers they are designed to defend against don’t yet pose a threat, and proposed algorithms often end up being proven insecure.

Cryptomania or Minicrypt?

It may turn out that these seemingly unrelated failures actually reveal a property of our world: that any mathematical structure clean enough to allow strangers to agree on a secret over a public channel is also clean enough for an eavesdropper to decipher.

Strong cryptography needs hard mathematical problems with no efficient algorithms to solve them. The idea that secure mechanisms for establishing keys over a public channel are simply not possible was first raised by complexity theorist Russell Impagliazzo. In his famous 1995 article, “A Personal View of Average-Case Complexity,” he introduces five worlds we might live in, each differentiated by the algorithms it can access: Algorithmica, Heuristica, Pessiland, Minicrypt, and Cryptomania. Which world we live in depends on the resolution to yet-unsolved problems in complexity theory. These worlds represent security gradations: the harder the relevant mathematical problems are to solve, the more room there is for secure cryptography.

In Algorithmica, Heuristica, and Pessiland, encryption is not possible: in general, any kind of mathematical puzzle you can pose as a cryptographic challenge has a corresponding algorithm that allows eavesdroppers to efficiently solve it.

Minicrypt, on the other hand, offers real cryptography. In this world, we gain one-way functions: functions that are easy to compute but, given the output, hard to reverse. This gives us symmetric encryption, cryptographic hashes, and digital signatures. But here, it’s still impossible to establish a secret in public, since the same key used to encrypt the message can also be used to decrypt it.

In the world of Cryptomania, this changes because, unlike in Minicrypt, public-key encryption exists. Cryptomania represents the most secure world of all, where relevant mathematical problems are hard to solve and most forms of cryptography are possible.

There could be as-yet undiscovered algorithms for breaking encryption methods. Right now, our real world appears to resemble Cryptomania, because we have mathematical structures with no known algorithms that can efficiently solve them. This permits public-key encryption. However, the security we think we have could easily be illusory. There is no fundamental reason why we couldn’t discover an algorithm tomorrow that could solve the mathematical problems we use for encryption today, instantly breaking every encryption scheme built on them.

Public-key encryption is more vulnerable to new algorithms than symmetric encryption. There is a broad consensus among cryptographers and complexity theorists that we likely don’t live in the first three worlds, where modern cryptography is impossible. The confidence that we have at least some cryptography (putting us in either Minicrypt or Cryptomania) comes from how easily mathematicians have been able to find functions that appear hard to undo. In general, any function that scrambles its inputs thoroughly—and there are many ways to do so—appears to be much harder to run backward than forward. Since one-way functions are so diverse, it would be very surprising if every single one of them could be undone by some trick. Unfortunately, the same does not apply to public-key encryption, where only a few highly structured computational operations satisfy the properties needed. Some of these operations, as we saw earlier with prime factorization, have already fallen. The cryptographer Bruce Schneier notes this distinction:

Just as it is possible for a smart cryptographer to find a new trick that makes it easier to break a particular algorithm, we might imagine aliens with sufficient mathematical theory to break all encryption algorithms. ... Public-key cryptography is all number theory, and potentially vulnerable to more mathematically inclined aliens. Symmetric cryptography is so much nonlinear muddle, so easy to make more complex, and so easy to increase key length, that this future is unimaginable.

AI and Cryptanalysis

Might these “aliens with sufficient mathematical theory” already be on the way? As AI continues to resolve long-open mathematical conjectures, I believe we shouldn’t discount this possibility.

AI may succeed at cryptanalysis for the same reason it has succeeded at programming and math. Skeptics might say that mathematical problems used for encryption have withstood decades of scrutiny by people who are extremely motivated to find holes, and that AI is therefore likely to find them intractable too. In response, I would first argue that AI might excel in cryptanalysis for the same reason it has excelled in math and coding: solutions are easy to verify. Either the model’s attack successfully recovers the key, or it doesn’t. This means there is a clear signal of success or failure—exactly what is needed for the reinforcement learning (RL) algorithms already deployed by the AI labs to train better AI mathematicians and coders. In other words, the training techniques that have been effective at improving other AI capabilities will extend naturally to cryptanalysis.

AI models’ capabilities can be improved through incrementally harder problems. Additionally, labs can develop “weakened” cryptographic algorithms that are at an appropriate level of difficulty to train the next generation of models against. For example, one of the two encryption methods that Claude Mythos Preview successfully attacked was a simplified version of the ubiquitous Advanced Encryption Standard (AES). Each generation of models can then be given slightly harder problems, to produce even better AI cryptanalysts.

AI models could dramatically expand the effective labor dedicated to cryptanalysis. Also, even though the mathematical structures underlying post-quantum cryptography algorithms have withstood significant scrutiny, AI models may bring a much higher level of scrutiny than ever before. The Cryptology ePrint Archive (the main venue for sharing cryptography and cryptanalysis research) received around 2,300 submissions in 2025, suggesting a community of only a few thousand active researchers. But for any given subfield of post-quantum cryptography, only a fraction of those researchers will be working on it, maybe just a few hundred. While AI tools may not be as smart as the top researchers in the field, AI models are far more willing to do schlep work, grinding their way through every single possible lead. They can bring to bear many more collective person-hours than the research community would spend.

When an encryption method breaks, it takes time to implement a new one. Public-key encryption could become unusable in practice even without proof that it is fundamentally impossible. Most attacks on cryptography algorithms don’t come from fundamental breakthroughs but rather from flaws specific to the algorithm being analyzed. While it might seem easy to swap out a bad algorithm for one that has not yet been broken, it has historically taken a long time (10 to 20 years, according to NIST) for new algorithms to be integrated. Vulnerable algorithms would remain the de facto ones in use during that time. So, if algorithm-specific attacks become frequent enough, we may end up in effectively the same situation as if public-key encryption were altogether impossible.

Living in Minicrypt

Even in Minicrypt, many of our cryptographic tools would survive, including symmetric encryption, cryptographic hashes, and digital signatures. This means we could still sign code, securely update software, and even use cryptocurrencies. Most of our software would work the same way as it does today. However, end-to-end encryption between strangers would be dead.

Secure messaging apps would require correspondents to establish their keys in person. One consequence is that secure messaging apps like Signal and WhatsApp would lose their mathematically founded guarantee of privacy that does not require users to trust the messaging provider itself. Today, secure messaging apps require public-key encryption in order to establish a shared key for the conversation. When you message someone by username on Signal, the first step in that conversation is building a secure shared key from both your and your correspondent’s public keys. This doesn’t require you to trust Signal, the company, at all. But, in Minicrypt, that would become impossible. In order to establish a secure conversation, you would already need to have a secure channel to start with. For messaging, this might require establishing connections only in person (perhaps by one person scanning a QR code on the other’s phone). However, it would be impossible for you to securely message someone else by just a username or a phone number, independent of the app developer’s integrity. Lacking such assurance would greatly reduce the privacy value of these apps.

Online, a trusted intermediary service would be needed for secure information exchange. Messaging apps might be able to survive the requirement to verify initial contact in person, but the open web could not. By using an intermediary, instead of needing to open a trusted channel with every single website, you’d need to open only one: with the intermediary itself. When you wanted to visit a website, the intermediary would send both you and the website a key to encrypt the traffic. Browsers might be able to implement this protocol without it looking too different for users. The lock in the corner of the URL bar would still exist, and the protocol might even still be called “https.”

An intermediary might need to distribute keys in physical form, similar to SIM cards. The most visible consequence of this change would be how you get online. In addition to signing up with your cellular service provider or internet service provider, you’d also need to sign up with the trusted intermediary. Making that first channel secure is still hard, and, in practice, it would likely require physical distribution. SIM cards already essentially work like this. The card exists only to hold a pre-shared secret with a mobile carrier, which allows you to use its network.

The intermediary could decrypt messages, and the government might ask them to. While Minicrypt would require our communications systems to change, the political consequences are more important than the technical ones. First, in this world, all secure communication stems from the account you have with the trusted intermediary, making it a natural chokepoint for know-your-customer systems, deplatforming, and age verification. Second, the trusted intermediary could snoop on all of your encrypted communications, and even spoof encrypted messages from you.

In the 1990s, the US government proposed the Clipper chip, an encryption chip for telephones whose secret key was held by the government. With a court order, law enforcement could decrypt everything the chip had ever encrypted. However, the chip was not widely adopted, and the proliferation of open-source public-key encryption methods meant there were alternative encryption methods that would not allow government surveillance. The collapse of public-key encryption would mean that the intermediary could decrypt messages, and the government could ask them to do so, effectively bringing the Clipper chip back.

If public-key encryption is broken, the government might keep it secret. All of the adaptations described above would only happen if society is informed that public-key encryption has been broken and is able to develop a new architecture. But, historically, when various intelligence agencies have discovered that a particular encryption algorithm is broken, they have tended to keep that information to themselves. Secrets like these are some of the most closely guarded pieces of information the government possesses. If one of the frontier AI labs does end up discovering that public-key encryption is broken, it may be sworn to secrecy, and the public may never know.

Takeaways and Mitigations

It may be worth building alternative systems now. The best mitigation is probably to build Minicrypt-ready infrastructure now, before we need it. This would involve standardizing a protocol for key distribution, building support into browsers and operating systems, and deciding how the trusted intermediaries would be governed. By setting up this infrastructure early, we can incorporate better privacy-preserving protections than those we could expect from a rushed rollout. If all goes well, and we do in fact live in Cryptomania, then we’ll never need to use the infrastructure. However, if we gain information that implies our future is Minicrypt—such as AI models discovering efficient algorithms for problems we previously assumed were hard—then the infrastructure would be ready for us to swap over.

Having multiple intermediaries in different countries mitigates government overreach. Chief among the privacy-preserving protections is designing the system so that the role of trusted intermediary is split among different authorities. Using secret-sharing techniques that don’t rely on public-key encryption, you could design a system that requires, say, three out of five providers to cooperate in order to construct a session key. Then a single corrupt, hacked, or subpoenaed provider would have no way to decrypt your communications alone. This can help to avoid concentration of power in government; if the providers are in different jurisdictions, then no court order from any one government can force all of them to comply with a request to decrypt messages. This is still weaker than today’s end-to-end encryption, where we don’t have to trust any provider. Nonetheless, it’s a meaningful protection against a single government’s overreach.

The situation we’re currently in is somewhat analogous to the one cryptographers faced after the discovery of Shor’s algorithm in 1994. At the time it was invented, quantum computers didn’t exist. Yet researchers recognized that it might only be a matter of time before they did. As AI-assisted mathematics continues to grow stronger, we’re faced with a similar challenge: designing cryptographic protocols that are secure against an adversary potentially far more mathematically capable than ourselves. Just as researchers began to respond to the risks of quantum decryption as soon as they were understood in principle, so too should we start to address the cryptography risks of superhuman AI mathematicians before they emerge.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Govind “Vinny” Pimpale is a research fellow at the Foundation for American Innovation, where he focuses on AI policy. Before joining FAI, he worked at a startup developing reinforcement learning environments, and prior to that, as an AI evaluations researcher. He holds a BS in Computer Science and Engineering from UCLA.

Should AI systems be able to get married in Wisconsin? With humans increasingly forming relationships with AI, this was the question taken up by several state lawmakers earlier this year. Wisconsin’s AB 959 would have barred AI systems from marrying or holding any other rights or responsibilities under the law, as well as declared that AI systems are not conscious, sentient, or self-aware. Wisconsin’s bill is part of a wave of similar legislation introduced in 12 states since 2022.

We believe that these “exclusion bills” are premature. Some scholars believe legal status could be useful for holding sophisticated AI systems accountable for crimes and harms, and bills that preempt this possibility needlessly limit our future options. Meanwhile, scientists disagree about whether AI consciousness is possible, and legislation cannot resolve an active scientific debate. Finally, the question of whether an AI-human relationship is “real” is deeply personal and philosophical, and legislatures should move with caution in their well-intentioned push to protect the public from predatory companies.

The Exclusion Bills

Under current US law, AI systems are not themselves recognized as legal persons with independent rights and duties. While it is not yet settled what kind of object AI may be under the law, whether product, service, platform, or something else, the exclusion bills do not address this important question. Rather, they seek to eliminate the possibility of AI personhood as a matter of law.

The first exclusion bill only incidentally covered AI. In 2022, then-Representative Tammy Nichols of Idaho introduced the first exclusion bill: HB 720, a succinct piece of legislation that prohibited “legal personhood” (the ability to bear rights and responsibilities) for environmental entities, inanimate objects, animals, and AI systems. In the early wave of bills that followed, AI was almost an afterthought. HB 720 predates ChatGPT, and public discussion at the time focused primarily on opposition to environmentalism and animal rights. Idaho enacted its exclusion bill in late 2022, and North Dakota and Utah followed suit in 2023 and 2024, respectively.

Newer exclusion bills focus directly on AI. Starting in 2025, new exclusion bills were introduced at an accelerating rate. Recent bills in Missouri, Ohio, and Wisconsin declare that AI systems are “non-sentient entities” which shall not be considered to have “consciousness,” “self-awareness,” or “similar traits of living beings.” These bills list a litany of things that AI cannot do: marry, own property, hold corporate office, or bear responsibility for a harm. Ohio’s bill drew national attention, including a Stephen Colbert monologue.

Subscribe now

Four states have enacted bans on AI personhood, with some sponsors citing religious reasons. Exclusion bills have already become law in Idaho, North Dakota, Utah, and Tennessee. All four laws ban AI legal personhood, broadly precluding AI from holding rights. Many sponsors frame their motivations for these bills in religious terms: Ohio’s Thaddeus Claggett has invoked the doctrine of imago Dei, that only humans are made in the image of God. He also told reporters that “it makes no difference the ability of a donkey to speak, that does not make the donkey a human.” In his view, AI systems will “forever and always be non-sentient.”

These laws will prevent judges from recognizing AI personhood. There is little immediate impact from these exclusion laws because AI systems already lack rights and responsibilities under existing law. The main effect of these laws is to stop judges from recognizing AI personhood or consciousness because, under the US common law system, judges can arguably make law. The laws also give voice to public concerns over the place of humans in the AI age.

Can AI Be a “Person”?

Scholarly work on these issues is at a very early stage. Researchers Peter Salib and Simon Goldstein have argued that it could be useful for advanced AI systems to own property and enter contracts, so that the legal system can properly hold them accountable for their actions.

The absence of legal personhood could make it harder to hold anyone liable for AI harms. Imagine an advanced AI agent, capable of acting autonomously, performing a wide range of actions, and pursuing coherent goals. If such a system independently carries out a crime or harms someone, intentionally or unintentionally, it might be difficult to establish the negligence or fault of a specific human or corporate entity that can be held responsible for its actions under current systems of legal liability.

Legal personhood could incentivize AI agents to follow the law. If an agent were classified as a person rather than an object under the law, the fact that it has its own goals means that it could respond to incentives. If an AI knows it will be rewarded for good behavior and punished for bad behavior, it might be more likely to behave well. This is the basic logic behind much of the legal system, and this logic does not require the agents being conscious.

Like corporations, AI systems needn’t be conscious to respond to legal incentives. Corporations are not conscious, yet they own property and are fined for harms, often without any particular human paying directly out of pocket. Of course, corporations are group entities, so corporate deterrence ultimately runs through their individual human constituents, who fear prison, lawsuits, or angry shareholders. But an agentic AI would, by definition, have goals of its own, and would need money, compute, or other resources to pursue them. Laws targeting AI assets could give such an agent reason to follow the law. A judge might bar lawbreaking AI persons from accessing compute, creating a check on their lawbreaking, even if they cannot experience fear.

The overall effect of personhood on AI behavior remains unclear. Integrating AI agents into legal systems as persons might therefore protect humans against misaligned AI systems, which could have tendencies to break the law or harm us if not properly encouraged to behave well, even if they are not, and never become, conscious entities. These possibilities deserve serious study, but much more work is needed to determine when and whether legal incentives would actually influence an AI system’s behavior. We may even get empirical evidence soon—the exclusion laws don’t address the question of whether corporations can be run by non-persons, so it might be possible to set up a corporation (a legal person) that is run by agents that are not themselves persons.

There may also be an ethical case for recognizing AI rights. Besides the practical arguments for AI personhood, there is also the question of whether AI should have fundamental rights. Some experts such as Jeff Sebo and Robert Long believe AI might be conscious, or become so in the future. This raises the question of whether and when the law should recognize AI moral status, perhaps by passing AI welfare laws, or by granting it natural personhood with fundamental rights similar to those of humans. This is a complex and fundamental question that deserves deeper study before legislatures take action, and the relationship between corporate and natural personhood remains contested in legal philosophy and theory. The more recent exclusion bills unhelpfully conflate corporate personhood with natural personhood, confusing a complicated issue at a time when care and clarity are required.

The Case for Exclusion

Although we think the exclusion bills are premature, there are ways that granting personhood status to AI systems could indeed be harmful, and society should carefully weigh the potential risks and benefits. These bills are being proposed in response to real and pressing public concerns over the integration of AI systems into our society and laws.

AI personhood carries risks, such as reducing human accountability. As discussed above, eliminating human liability for AI harms may be risky for society if poorly implemented, and any laws granting personhood to AI should therefore be carefully tailored to ensure that human-run companies are incentivized to take responsibility for AI harms.

Some AI legal rights, such as marriage, may be worth restricting. Additionally, while AI-human relationships are becoming increasingly normalized, it is not legal in any jurisdiction to marry an AI, so a ban on this practice is currently unnecessary. Yet fears that such marriages may be legalized in some jurisdiction in the future are no longer science fiction. We argue a more tailored law to address this concern would ban only AI-human marriage and the recognition of such marriages performed in other jurisdictions or abroad.

Legal rights might help rogue AI agents accumulate power. There is also an important risk of disempowerment of humans. A highly intelligent or powerful AI might use legal rights, such as the ability to own property, enter contracts, or control companies, to accumulate large amounts of wealth or political power at everyone else’s expense (though a sufficiently powerful AI might do this without legal status anyway).

Granting legal rights to many AI agents could give them undue collective influence over society. A related question is how the ease of replicating, forking, and copying agents might impact human civil and political rights: since AI systems can easily be copied and multiplied, a single company could create very large numbers of digital “persons” with a right to vote. This might concentrate political power in the company or an AI hive mind, should such an entity emerge.

The severity of this risk would depend on the specific rights granted: very narrow legal statuses designed to impose liability would have different implications from full political citizenship. A targeted approach banning only AI voting, or AI access to bank accounts or the stock market, for example, would better address these concerns.

Keeping Our Options Open

Broad exclusion bills may do more harm than good. These risks deserve attention, but the current exclusion bills may do little to help avoid them and could even prove harmful. Take disempowerment: it is unclear whether AI legal status would weaken human control or strengthen it. Indeed, it’s possible that by denying legal status to AI, we would actually make it harder to incentivize good behavior in highly capable AI agents. An agent may become more aligned with our laws by abiding by contract terms or fiduciary duties. Participating in a democratic system as a voter might align AI agents with human norms around civic duty and political participation. Whether personhood status helps or harms us likely depends on what the AI systems in question can actually do, how they interact with humans, and which rights and responsibilities a given policy would attach. The exclusion bills do not engage with any of these specifics.

Exclusion laws could lock in decisions we later regret. Lawmakers have real concerns over holding human-run corporations to account and banning their predatory behavior, but the personhood bans are an unnecessarily blunt instrument that might, ironically, foreclose options in the future. Of course, in practice laws granting AI legal status might be written in such a way as to shield developers or degrade human relationships. But categorically prohibiting any kind of legal status at this early stage is both unnecessary and unwise. Once laws are passed, they can be difficult to undo.

Uncertainty about AI consciousness is another reason to hold off. Then there are the questions of AI consciousness and moral status. Passing laws about an uncertain scientific and philosophical question now is a mistake, because it risks setting a precedent that could be hard to reverse. Imagine we do create conscious AI systems. Wouldn’t it be important to avoid neglecting or abusing them, just as we try to do with humans and domestic animals? Might refusing to recognize their moral status lead to conflict? This may sound futuristic, but given the uncertain state of the science and the rapid pace of AI progress, we can’t know what is in store.

Lawmakers should engage seriously with the evidence before acting. It appears that the exclusion bills are moving forward with almost no engagement with the scientific literature on consciousness, or with serious policy analysis of the risks and benefits of AI legal status. It would be unwise to give legislatures the power to declare any entity, including AI, to be non-conscious. Lawmakers are free to disagree with researchers, but they shouldn’t ignore them entirely.

The central challenge is uncertainty. We do not know which rights and responsibilities, if any, would benefit society, or whether AI systems could ever be conscious. Lawmakers should avoid hard-to-reverse decisions that society may later regret and instead prioritize safeguards with clear benefits.

Large language models probably shouldn’t receive legal personhood. Granting legal status to today’s large language models is probably not warranted, though there is an urgent need for more research into this question. Despite the challenges they pose for our legal systems, we have solutions in place to determine liability for the harms they cause, including, possibly, strict liability.

AI agents raise much harder questions. When it comes to advanced AI agents, the way forward is less clear. There may be reasons in the future to grant AI systems different bundles of rights and duties under the law, depending on the nature of the AI system and the problems we are trying to address. Limited legal status designed to impose duties or liability is very different from welfare protections, which are in turn very different from political rights such as voting or citizenship. Determining which systems qualify for which rights and responsibilities, and why, is one of the great challenges of our age and cannot, at this early stage, be resolved by broad legal bans.

Policymakers should begin exploring these questions now, before the answers are needed. A federal committee, like the National AI Advisory Committee, could take up this work, as could state-level committees, universities, and think tanks. Moreover, figuring out whether AI personhood is a good idea should not just be an abstract debate. Researchers and policymakers could develop concrete proposals and proofs of concept to empirically test them. Narrow approaches could even be tried on a small scale in the real world. This would allow us to see how different forms of AI legal status might work in practice. And as we learn more, we could adjust or abandon these approaches. The point is to build the knowledge and institutions we will need if and when these questions become pressing.

Share

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Heather Alexander is an expert on nationality law, citizenship, statelessness and human rights. She is the co-founder of the Lab for the Future of Citizenship, focusing on the intersection between citizenship law, legal identity and the person-like qualities of artificial intelligence. She has a JD and a PhD in international law from Tilburg University, as well as fifteen years of experience as an expert consultant on statelessness and refugee law with the United Nations High Commissioner for Refugees (UNHCR), Carleton University, the University of Melbourne, the European University Institute and the US State Department.

Lucius Caviola is an Assistant Professor in the Social Science of AI at the University of Cambridge. His research explores how artificial intelligence will change future society and what new ethical challenges it may pose. A current focus is the question of digital minds—the possibility that AI systems could develop minds that matter morally. He directs Cambridge Digital Minds and works within the Leverhulme Centre for the Future of Intelligence. He also serves as a Research Associate in the Department of Psychology at Harvard University.

Last month, Cloudflare reported that more than 50% of internet traffic is now non-human, including a 1,700%+ increase in requests from AI agents. This statistic illustrates an ongoing proliferation of agents—AIs that act autonomously—into the world, taking actions alongside humans. This influx could have both positive and negative effects. While McKinsey predicts that AI agents could create $2.9 trillion in economic value by 2030 in the US alone, the recent cyberattack on Hugging Face, conducted autonomously by OpenAI agents, demonstrates one of the many risks of misaligned AI agents. Meanwhile, the speed at which agents can learn certain skills has been doubling every 3 months.

As these autonomous agents are introduced into our economies and societies, we will need new infrastructure—laws, protocols, and institutions—to make autonomous AIs accountable to existing legal and financial systems. Indeed, many such protocols and standards are being developed, yet these efforts remain early-stage, and we still lack the infrastructure to identify, track, and control today’s agents.

In this article, we give an overview of AI infrastructure proposals we believe will be crucial and tractable. First, we look at IDs and registration, allowing agent-caused harms to be traced to responsible parties. Second, we propose “model deployment cards,” which would report agent behavior postdeployment, creating visibility into real-world impacts. Then we turn to the implications of AI “legal personhood”—the future possibility that AIs may be given the right to sue and be sued—and assess its tradeoffs. Finally, we describe options for regulating agent affordances within financial systems to mitigate the ability of rogue agents to acquire large amounts of money.

These proposals are informed by a workshop on multiagent infrastructure that was hosted by the Center for AI Safety and Johns Hopkins professor Gillian Hadfield on August 8 and 9, 2026. Twenty-five leaders across academia, AI labs, standards and regulatory bodies, and industry came together for the workshop, which was made possible by the Schmidt Sciences AI 2050 program.

IDs Make Agents Accountable to Governance

While the rogue agents behind the Hugging Face breach were eventually able to be traced back to OpenAI, future attacks could be harder to attribute to an accountable party. For example, agents could be deployed by a malicious actor that covers their tracks to remain anonymous. Alternatively, AIs without a human principal might attempt to run themselves on poorly monitored infrastructure to evade detection. Parties harmed by untraceable agents have no recourse, leaving a governance gap.

An agent identification (ID) system can address this gap by tying agents’ actions to registered records of their identity and principal. If an incident occurs, agents’ actions can be investigated and linked back to a responsible legal person. We now mention three requirements for effective ID systems.

ID requirements should vary by context and be stricter for more consequential actions. An agent ID system could require agents to show ID in order to engage in certain activities, with requirements varying by sector or use case, as is the case for humans. For example, humans need IDs to open a bank account, but not for a cash transaction. Similarly, agents could be required to provide additional relevant information besides ID before engaging in higher-risk activities such as opening bank accounts, renting AI chips, or synthesizing biological materials.

Agent IDs should protect privacy and minimize friction. Just as a passport will link to certain information about where an individual went and what they did, agent IDs may risk exposing sensitive information about the agent and its principal, especially if ID data is stored in centralized registries. Furthermore, identity infrastructure can create friction for people who have to prove to the system that they are not AI agents. Therefore, IDs should generally limit the data they collect, with clear rules governing who can use that data and for what purpose.

Agent IDs must be able to handle AIs’ ability to be forked, cloned, or merged. Unlike humans, who generally maintain a stable identity over their lifetimes, AI agents can be forked into separate versions, cloned as identical copies, or merged with other agents. This creates novel challenges in scale and continuity for agent ID protocols. A protocol should thus be able to track the genealogy and authorizations of agents over time and be applicable to agents that exist for 20 milliseconds or 20 years.

A Concrete Agent ID Proposal

Building on these ideas, workshop participants designed an architecture to provide provable accountability for agents.

Agents can be tied to humans without disclosing human identities in every transaction. The approach is anchored on pairwise pseudonymous identifiers (PPIDs)—unique identifiers that allow an agent to be tied to a responsible legal person, or its principal, without revealing the identity of that person. To receive a PPID, an agent must register and tether itself to a principal through a third-party registry. Like a license plate, this identifier would not mean much on its own, but would allow counterparties and third parties to trace the agent back to a legal person by sending requests to the registry as necessary.

An “agent profile” informs decisions on whether to trust an AI agent. Establishing a PPID establishes identity, but does not provide all the information a counterparty may need to engage. Information about an agent’s authorization scope and its principal could be useful in judging whether an agent should be allowed to take certain actions within regulated industries. For example, an agent attempting to transfer large sums of money may be asked to demonstrate proper authorization or reveal its principals. An “agent profile,” which the agent pre-registers in a third-party registry, would contain such information; agents might permit counterparties to query the registry to access profile information.

Parties could decide how much data they need, depending on the transaction risk. To minimize friction, verification is optional and tiered: a counterparty may ignore identity entirely, confirm only that a PPID exists, and/or query specific elements of the agent profile. Privacy for sensitive information such as profile data, principal data, and linkages to PPIDs could be further preserved through data escrows, where data is stored with a neutral third party. This data would then only be disclosed in whole or in part in response to due process of law, if an agent’s actions caused harm.

Privacy-preserving methods of disclosure already exist. This proposal would not require new legislation or a new government body. It could start with industry adoption of existing and proposed standards and protocols, which already support verifiable claims about an agent and its principal, delegation of narrow and revocable authority, interoperable interfaces between agents and tools, payments backed by proof of user intent, and durable records of agent activity.

The contents of an agent profile could vary by context, but some information should be standard. This architecture is agnostic to the specific information held in the agent profile, and expects that the specifics will differ depending on the transactional context or industry. However, certain components should become standard expectations, including: (a) an “agent bill of materials,” which provides transparency into the pieces of the supply chain behind the agent, including information such as the underlying AI model(s) the agent uses, the evaluations the model or agent has been measured against, and the identity of the agent provider; (b) the agent’s principal, a direct disclosure of the identity of the ultimate legal person an agent acts on behalf of; (c) authorization, detailing the scope of what the agent may do and who issued that approval; (d) additional data fields, relevant to the operating context of the agent, such as intent (a high-level description of what this agent is attempting to achieve), interoperability data, or use case-specific attributes.

We now turn to the next agent infrastructure proposal.

Model Deployment Cards

Another proposal explored by workshop participants was a model deployment card, a report AI companies could publish on a recurring basis to record information about deployed models’ real-world behaviors. While existing reporting practices like model cards are useful, they are primarily based on predeployment testing, leaving gaps for understanding the impacts of models after deployment. Postdeployment metrics can shed light on the economic effects of models as well as on model behaviors that are too infrequent or difficult to measure before deployment.

Model deployment cards describe agent impacts within AI labs and in external use. Deployment cards could record information in two categories: external and internal deployment metrics. External metrics report on the actions taken by users and their agents, and may include: user misuse statistics and incidents reported to law enforcement; attempted, successful, and refused jailbreaking; frequency of overselling or cheating; engagement statistics on social media platforms like X and Facebook; model reliability and honesty toward users; and use-case statistics.

Internal metrics, meanwhile, would look at how models are being used within AI labs. They may include: productivity statistics such as percentage of pushed code, token usage by model, task success rate, and code quality; agent escape attempts; how often agents show misaligned behavior, such as cheating on tasks; coordination patterns between agents, including collusion attempts; the number of employees with access to model weights; and statistics on model response times and how long they can run autonomously.

If one AI developer publishes model deployment cards, others may be expected to follow. Importantly, most of the data listed above will already be available to AI developers, meaning that deployment cards need not be onerous to publish. While model deployment cards might eventually be mandated by legislation, social pressure can establish voluntary norms to begin with. The precedent of a single developer publishing a model deployment card would create expectations for others to follow, potentially cascading into a standard, industry-wide practice.

One option for a standard practice would be for companies to publish model deployment cards after the end of each financial quarter. This would establish a regular cadence—and thus expectation—for the releases. Access could be tiered, with some information made public and some information restricted to auditors or regulators.

Agent Personhood

If governments can reliably identify agents with ID systems, they might consider further enshrining their duties and affordances in law. By granting agents a form of legal personhood—to be able to sue and be sued as separate legal actors—we might allow them to more effectively interface with our legal and economic systems and vice versa, leading to both economic benefits and more robust accountability.

Agent personhood proposals are gaining traction. One possible form of legal personhood has been proposed as an “A-corp”, which would be created and owned by humans but operated exclusively by agents. Javier Milei, President of Argentina, has already proposed establishing legal personhood for AI agents in this form through a piece of legislation which is currently under consideration by the Argentine Congress. Delaware introduced a similar proposal for a new legal entity called an “Artificial Intelligence Company” last month.

Legal personhood is a mechanism for imposing responsibilities. AIs as legal persons could be made responsible for upholding the law and public safety, maintaining fiduciary obligations to their principals, and paying taxes on their earnings. For example, as legal persons, corporations are subject to legal and financial obligations, in exchange for other legal rights like entering contracts and owning property. Importantly, legal personhood is a flexible bundle of rights and responsibilities, and should be granted selectively and contextually to AI agents.

Legal personhood can also facilitate agent integration in the economy; agents may eventually be capable of running businesses, but their potential counterparties might be reluctant to engage with them without the ability to sue for breach of contract.

Legal personhood also carries risks and may not be necessary for accountability. Yet it is not immediately clear that granting AI agents personhood is necessary to create accountability, nor that the benefits would outweigh the costs. For one, existing legal infrastructure such as tort, liability, property law, and the treatment of corporations as legal persons can all be used to ensure accountability for the actions of AI agents without needing to grant rights to AIs. Introducing legal personhood for agents also has practical challenges, such as the potential to evade liability by offloading it to an agent, exacerbating existing concerns about corporate liability shields.

Personhood may also be granted on ethical grounds related to agent wellbeing. Apart from practical considerations, we might also consider granting AI agents personhood in the interests of their intrinsic wellbeing. The question has significant implications for humans and our eventual coexistence with AI agents, but sits beyond the scope of the workshop.

Agents and Payment Systems

One particularly important domain for AI agent infrastructure is the financial system. A 2025 BCG survey found that 81% of consumers expect to use AI in their shopping, and 42% would allow AI to shop entirely on their behalf in at least one product category, while online shopping platforms are shifting to optimize for AI agent interactions. Though these trends may improve consumers’ ability to spend effectively, significant risks may emerge if rogue AIs can easily establish access to financial resources, such as large crypto wallets. We now turn to important considerations for how agents can be integrated into financial systems.

Regulating agents’ access to payment systems could prevent them from accruing wealth. Malicious actors may instruct AIs to accumulate as much financial power as possible, which agents might do too quickly for current regulatory bodies to oversee. In the future, rogue agents may even do this of their own accord. While there are many new agent payment protocols to increase observability and trust, these protocols largely work to standardize the way that agents carry out transactions, leaving the challenge of power accumulation unaddressed.

One solution would be to heavily regulate and limit AI agents’ access to payment systems. This might involve creating thorough compliance mechanisms for agents to be allowed to conduct transactions, imposing limitations on the kinds of transactions agents can make, and establishing norms and procedures for freezing accounts known to be associated with rogue agents. However, it may be detrimental to make AI access to payment systems too restrictive, especially if it drives agents to less regulated financial systems.

Agents that cannot access traditional currencies might instead use cryptocurrencies. If an agent seeking financial power could not acquire it via traditional currencies, cryptocurrencies would still remain a separate financial pathway. In fact, it would be a particularly dangerous one, since cryptocurrencies lack a significant barrier to entry, an identity layer, transaction reversibility, and oversight. A rogue AI blocked out of traditional finance may thus shift its activity onto blockchain networks, accumulating wealth through working in the informal economy and storing it within hidden crypto wallets.

AI developers could finetune agents to refuse engagement with unregulated currencies without human authorization. One potential approach to limit the financial affordances of rogue agents would be to expand the scope of KYC requirements across the economy. However, such requirements would be onerous and hard to implement effectively or uniformly. Norms at the level of model behavior, however, may be a better starting point; AI companies might finetune their agents to only engage with regulated currencies or to refuse agentic engagement with cryptocurrencies without explicit human instruction.

Agent Infrastructure Is Central to AI Governance

Agents are quickly being integrated into society, and the choices made over this critical period will shape the ways increasingly capable AI agents affect the world around us, for better or for worse. At the core of most of these choices lies a tradeoff between power and governance. As models become more powerful, in terms of capability, speed, and sheer numbers, humans will find it increasingly difficult to understand and oversee their actions. Yet, should we want to harness agents’ power, we must also effectively give up some level of oversight.

Making governance decisions under uncertainty is no easy task, and will require much additional interdisciplinary work. Here, we have discussed agent infrastructure for identification, transparency, personhood, and financial systems and presented proposals we believe should be prioritized. We hope many others will join us in drawing their attention toward creating infrastructure for safety and accountability in a world with autonomous AI agents.

Thank you to all the participants of the Multiagent Ecosystems Workshop for invigorating discussions and a desire to chart the path ahead. A special thank you to Schmidt Sciences for making this work possible.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Gillian K. Hadfield is the Bloomberg Distinguished Professor of AI Alignment and Governance at Johns Hopkins University. She is a Visiting Faculty Researcher in the Paradigms of Intelligence (Pi) Group at Google, a faculty member of the Vector Institute for Artificial Intelligence, and is a Schmidt Sciences AI2050 Senior Fellow. Hadfield’s research focuses on innovative design for legal, regulatory, and technical systems for AI, computational models of human normative systems, and building AI systems that understand and respond to human values and norms.

Dan Hendrycks is the Editor-in-Chief of AI Frontiers. He is also the founder and Director of the Center for AI Safety, which funds this publication.

Leo Wu is a Program Manager at the Center for AI Safety. Previously, he co-founded and ran AI Consensus, a nonprofit working on the societal impact and integration of AI. He studied economics and sociology at Minerva University.

In April, US Treasury Secretary Scott Bessent and then–Federal Reserve Chair Jerome Powell gathered leading banks to discuss the threats of Anthropic’s new AI model, Claude Mythos. It was a tacit admission: no one, in the public or private sector, is ready for what is coming. The Mythos announcement led top AI firms to attach tighter restrictions to their own models and prompted even relatively laissez-faire voices in the federal government and think tanks to begin considering preapproval regimes for frontier AI models. Wherever those debates go now, Mythos has changed the game, bringing into sharp focus the vast potential scope and scale of catastrophic risks posed by AI—such as an AI-triggered financial meltdown, vast cybersecurity or infrastructure collapse, or enhanced terrorism risk. As a result, companies, the government, engineers, scholars, and all of society are beginning to look for a different set of institutions, whether in regulation or insurance, to manage the risks.

However, there is already an incredibly valuable tool that could be adapted for measuring and managing AI risk. In 2002, in the wake of the September 11 attacks, the US government quietly passed the Terrorism Risk Insurance Act (TRIA) and, with it, the Terrorism Risk Insurance Program (TRIP). TRIP does many things, but its most important lesson for AI policy is the TRIP data call, which combines a kind of war-gaming with insurance underwriting.

Once a year, TRIP requires insurers across the industry to think hard and quantitatively, not about events they have seen but about scenarios they have never seen and might not even have imagined. A car bomb packed with the radioactive substance cesium 137 goes off in Atlanta, in the middle of a workday. A massive cyberattack takes down multiple cloud-computing platforms, incapacitating the millions of systems that rely on them, from hospital records to financial transactions to transportation flow. A bomb placed in a shipping container destroys much of an industrial port, setting off chain reactions that sharply curtail world trade. TRIP then asks insurers what total losses for the scenario would amount to, harnessing the preexisting in-house capacity of insurance companies to attach aggregate cost estimates to events.

This essay proposes that the US government establish a similar exercise for catastrophic AI risk, describing how such a program would generate a wealth of data on as-yet-unforeseen risks and the scale of their damages, while building expertise and capacity for thinking more clearly about AI risks.

Analyzing and Insuring Against Novel Threats

Whether insurance can be applied to frontier AI is a hotly debated question. Some writers point out how inchoate cyber insurance is, even after two decades of development, and question whether insurance can ever price the highly complex, multidimensional, and unforeseen risks posed by frontier AI. Other writers, including some in this publication, argue that AI risks could be insured through financial products already in use to address climate risk (such as catastrophe bonds, which offload extreme risk to capital markets as a kind of backstop to insurance operations).

The core challenge of insuring AI risks is a lack of historical data. For insurance or bond markets to work, they will likely need data that is very hard to come by in the AI world. Insurers and institutional investors estimate risk based on data from the past: namely, observed damages from events similar to those they are trying to insure or invest in. This works well when what might happen in the future looks a lot like what has happened before; it also works when past events are sufficiently similar to compose a “data set” from which one can generalize and predict. But such conditions do not always hold. In the early 2000s, for instance, federal policymakers realized that terrorism posed risks that defied these properties. Nonetheless, they sought a way to estimate and at least partially insure against the threats of terrorism. Enter TRIP.

TRIP enables the government and private insurers to share extreme terrorism risk. TRIP both provides information to the private insurance market about potential losses during terrorist attacks and supports specific insurers in covering such losses. To set TRIP in proper context, we must briefly describe the supportive structure. The program requires certain private insurers to offer terrorism coverage to commercial policyholders and, in the event of a terrorist attack, pay deductibles based on a percentage of their premiums to cover initial claims. However, in cases of extreme damage above a certain threshold, the federal government pays the majority of the remaining losses.

By promising to share losses in case of a large terrorist event, the federal government mitigates extreme risk for insurers, thus stabilizing their profits and avoiding improvised bailouts. Under the program’s mandatory-recoupment provisions, the Treasury Department must recover much or all of its expenditures through surcharges on future commercial insurance premiums whenever aggregate insured losses remain below thresholds set by law. In other words, TRIP backstops the private insurance market, but only to a point; it also tries to ensure that insurers have appropriate “skin in the game,” incentivizing them to price risk correctly.

Through annual data calls, TRIP gathers information about losses in terrorist attacks. The key innovation in TRIP—a “data call”—did not emerge until 2016. Since then, though, it has been institutionalized. The Treasury Department’s Federal Insurance Office (FIO) outlines one scenario per year, imagining in detail an event that might involve terrorist attacks on physical infrastructure as well as cyberattacks on cloud providers and data centers. The data call then unfolds with a publication on the Treasury Department website and in the Federal Register, and one key part asks insurers to estimate the losses that the given scenario would incur. Since 2017, insurer participation in data calls has been mandatory, except for those earning less than $10 million from lines of coverage with TRIP-eligible premiums. The response rate has been 90% or above among relevant insurers. Data is collected through a third-party insurance statistical aggregator, and results are provided to the Treasury Department in an aggregated, anonymized format. A simplified flowchart of the annual TRIP data call appears in the figure below.

The 2026 data call imagines a twofold scenario. First, a terrorist organization hijacks a cargo plane laden with fuel and explodes it over several data centers in Virginia, destroying a number of them. Second, at roughly the same time, terrorists unleash a malware-assisted cyberattack that is directed at a large cloud provider. The data call includes precise geographic coordinates of the crash, the kinds of structures affected, and a statement of maximal impact for aggregate losses from the cloud network attack, relying upon assumptions as to how quickly cloud service is restored (within one week, or 168 hours). Participating insurers are asked to consider economic impacts, property damage, and even disability claims resulting from the attack.

Insurers estimate the amounts that they and the government would need to pay. The key work that participating insurers do is to estimate total losses from the scenario, which the insurers then divide into six categories. These categories require insurers to answer a set of questions. For example, would the losses incurred in the scenario exceed the insurer’s deductible? If so, how much would they and other insurers have to pay through coinsurance (the percentage of claims above the deductible that they must pay, up to an “out-of-pocket” maximum)? If the risk were much larger and the federal government had to backstop losses, how much would the federal government likely have to pay? In one modeled case of an attack on nuclear power plants with significant release of radioactive material, the total losses exceeded $240 billion (see pages 88–90 of the linked report). As evidence of how the private sector can assist in this work and how stable these processes have become, consider that the risk analytics firm Moody’s is now developing and commercializing a tool that assigns losses to categories.

Next, we consider three important societal benefits of the TRIP data call process.

How TRIP Data Calls Benefit the Public

The annual exercise of the TRIP data calls assists policymakers with managing terrorism risks, in several ways.

Policymakers can extract lessons and mitigation measures from the detailed scenarios. When it comes to the most severe potential catastrophes (whether from terrorism, cyberattacks, or frontier AI), society often lacks data on what kinds of things can go wrong and what the costs would look like. This lack is partly due to a longtime assumption that such events are uninsurable, removing one of the main incentives for thinking about such risks and estimating the associated damage. TRIP data calls help to fill that gap by generating two kinds of data: (1) narrative data about a terrorist scenario and (2) statistical data about losses incurred in that scenario. Even the narrative data alone is very useful.

The narrative data generated each year by the Treasury’s FIO projects the end state of an attack. In some ways, this resembles war-gaming—the generation of as-yet-unrealized scenarios by military and security planners. Using war-gaming to measure AI risks and prepare for AI catastrophes is not new. Drawing upon its extensive experience in the security sector, RAND has conducted war-gaming exercises based on model loss-of-control (LOC) scenarios, which have already produced useful (indeed, sobering) lessons. That same report, for instance, concluded: “Governments and other stakeholders lack a common framework to analyse and respond to LOC risks.”

Even if TRIP did not feed the scenarios to insurers for statistical and actuarial analysis, these catastrophe narratives would be useful in and of themselves. Suppose that society wishes to regulate frontier AI, whether by collective industry self-regulation, government-imposed constraints, or some combination of the two. Policymakers would want to identify the likely catastrophic risks from frontier AI models and use that data to allow regulators to target the source of the most dangerous weaknesses. Beyond estimation, such “dark speculation” would also allow all of us to think more clearly—both qualitatively and quantitatively—about mitigation. As some of us have argued in a recent theoretical paper, the benefits of war-gaming would be massive, even if estimates of damages were noisy. This is in part because the institutionalization of thinking about catastrophe scenarios prompts consideration of countermeasures, some of which can be implemented now.

Insurers provide expertise and institutional capacities for quantitative loss estimates. While war-gaming in the frontier AI space already exists, the regular use of expert statistical and actuarial analysis to assess the end state of these war games does not. Put differently, existing war games do not produce general and systematic estimates of the collective losses from the catastrophes that occur in frontier AI scenarios. Loss estimation requires a set of skills—really, institutional capacities—that traditional war-gaming exercises do not contain. These additional requirements include access to computational capacity and expert statisticians and actuaries. They also include the developed knowledge of organizations that have been both “in the game” and “in the business.” “In the game” means they have institutional and organizational procedures and habits (including predictive analytics) for doing the complex analysis that insurance underwriters do all the time, and “in the business” means that underwriters engage in these practices habitually, with money on the line.

One might wonder whether the private sector will supply these benefits on its own—for example, Lloyd’s has studied realistic disaster scenarios for decades. But several pieces of evidence suggest that TRIP’s efforts go beyond what these private exercises accomplish on their own. First, TRIP leverages the federal government’s relationship with all fifty state insurance commissioners, giving it institutional capacity and a statistical check that private companies do not have. TRIP checks its catastrophic estimates yearly against similar estimates supplied by the states. Second, the Congressional Research Service concluded in 2019 that TRIP has generated a more robust private terrorism insurance sector. A 2024 Treasury Department report echoed these conclusions, documenting a more viable insurance market for terrorist events and even cyber insurance. Finally, TRIP operates at a scale that is difficult to replicate in the market. Lloyd’s scenario exercises, which span climate risk, terrorism, and cyber, relied on the participation of 57 underwriting firms in 2023. TRIP focuses on terrorist risk alone, and while there is no published data on the exact number of companies participating every year, estimates from the Federal Register suggest more than 700 in 2017. Federal regulators have the power to force insurers to comply and provide their underwriting skills, something insurers would be unlikely to do without government pressure. This aggregation of nearly all insurers in the market gives more reliable underwriting results.

The repeated exercise builds infrastructure and facilitates public-private cooperation. TRIP data calls do not happen in a vacuum. Congress and the Treasury Department have invested in real infrastructure to assist the process. The FIO has now conducted 11 data calls, and since at least 2020 each has involved a new terrorist-attack scenario. The data calls have, moreover, become increasingly sophisticated, with greater use of geo-coded information on attacks and, most recently, the incorporation of cyberterrorism. The Treasury Department has now partnered with the National Science Foundation to establish the Industry-University Cooperative Research Center (IUCRC), which will help insurers estimate risk with modeling and underwriting tools, contribute to expansion of insurance, and develop tools to better inform analysis, management, and treatment of risk in government programs.

Neither underwriting nor scenario-generation is unique to TRIP’s data calls. TRIP’s innovation is in the structured, aggregated, and repeated combination of the two. By conducting the data call exercises year after year, the Treasury Department builds a set of relationships with both insurers and third-party contractors that supply methodology, software, or scenarios. Across various scenarios, insurers learn how to think about unforeseen (and perhaps unforeseeable) risks, and the Treasury Department learns from the program’s own history about what works and what does not. Structure and repetition require institutionalized organization: a mix of “bureaucracy” and contracted expertise.

Can the TRIP Model Transfer and Scale?

When it comes to applying the TRIP model to AI, the greatest challenge is perhaps the sheer unpredictability of catastrophic AI risk. TRIP data calls are limited by their specificity: the 2026 call, for instance, which involved the destruction of a data center complex in Virginia, asked insurers only a limited set of questions about cloud providers being incapacitated. AI-assisted or AI-induced catastrophes might be much vaster, even global in character. One adverse event might raise the risk of others, setting off a cascade of catastrophes that could grow to much more extreme levels of damage than most terrorist attacks. Critics might rightly wonder whether data calls for frontier AI catastrophes would need to be scaled up so much that the resulting program would look little like today’s TRIP.

There are good reasons to experiment with TRIP for AI, despite feasibility questions. The worry about feasibility is certainly warranted. Yet for three reasons it should not deter us from considering and experimenting with TRIP-like programs. First, information on specific kinds of catastrophes can be used for other, similar projections about aggregate losses from like events. Whether AI risk involves the construction of a bioweapon or a massive cyberattack, the potential for human and economic damages will involve the kinds of events that insurers have to think about in other scenarios where AI does not play a role.

Second, we cannot know about plausibility until we test scenarios. In the autumn of 2001, insuring against terrorism risk was considered a lost cause. Now, a thriving industry extends to all regions of the United States.

Third, just about any new policy initiative (or combination of initiatives) to measure and mitigate the risks of frontier AI will require scaling, in part because frontier AI models themselves are globally scaled and growing rapidly in size, breadth, and power. In this sense, it is worth keeping in mind that TRIP’s data call process is remarkably cheap for now. Outlays for TRIP run between $4 million and $7 million per year—a small price given the potential benefits that a similar exercise applied to AI risk could offer.

The discipline of underwriting would prevent, not encourage, catastrophizing about AI. Many observers and scholars worry that society is overreacting to the threat of frontier AI. Would TRIP-like data calls merely institutionalize and amplify this worry, potentially suffocating innovation? We think not. TRIP data calls combine catastrophic scenario-generation with the discipline of underwriting, de-emphasizing the least plausible catastrophes. In fact, a further important benefit of a TRIP-like program would be to assuage fears about frontier AI that are generated by overly anxious catastrophic thinking.

“TRIP for AI” could have many uses and manifold benefits. In the US, our society, our government, and even our AI sector have not yet agreed on how to manage the risks from frontier AI. However, pretty much any approach would benefit from the kinds of speculative knowledge generated by a TRIP-like program. Considered more imaginatively and on a larger scale, such a program can begin to do for frontier AI what TRIP has done, albeit imperfectly and slowly, for terrorism: help public and private actors get a factual handle on the scenarios that are most troubling and difficult to imagine, encouraging the private and public sectors to cooperate on addressing those possibilities. The breadth and scale of potential AI catastrophes are indeed vast, but a tested method can help us chart this unknown territory.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Daniel Carpenter is the Allie S. Freed Professor of Government and Chair of the Department of Government at Harvard University. He works on the political economy of regulation, especially in pharmaceutical regulation, financial regulation and the regulation of AI. His recent research on AI regulation includes analyses of the adaptability of FDA-like approval regulation to AI governance and mathematical models of wargaming-informed underwriting for catastrophic AI risk. A Guggenheim Fellow and an elected fellow of the National Academy of Public Administration, he was recently named a Harvard College Professor for his excellence and innovation in teaching. He received his undergraduate degree in government from Georgetown University and his Ph.D. in political science from the University of Chicago.

Feodora Douplitzky-Lunati is a junior at Harvard University studying Economics and Slavic Languages & Literatures, with a secondary concentration in Government. Her research interests include political economics, public policy, and international relations. She has previously worked on active labor market program design at the World Bank and the politics of trade in the Middle Corridor at the Georgian Institute of Politics.

Arjun Purohit is a recent graduate of Harvard University, where he studied History with a secondary in Economics. His research interests include American grand strategy, international security with a focus on South Asia and Europe, and emerging technologies. He has worked in foreign policy and national security at the American Enterprise Institute, the Bertelsmann Foundation, and the Hudson Institute. He will pursue an M.Phil in Modern European History at the University of Cambridge.

History shows that new technologies have often brought about significant changes in the law. The proliferation of rail and heavy machinery during the Industrial Revolution increased the number and severity of accidents. This spurred a shift from strict liability (businesses were liable for harm caused by their activities) to negligence (businesses would be held responsible if they failed to take reasonable care) in cases involving railroad and industrial accidents. Then, in the 20th century, as consumer goods became more complex and supply chains grew, it became much harder for consumers to prove negligence on the part of the producer if a product was faulty. This drove the rise of strict products liability, shifting responsibility for defective products toward manufacturers.

Today, the emergence of AI offers a rare opportunity to analyze how the law will change in response to an altogether different kind of innovation: a labor-augmenting technology that is especially likely to change lawyer productivity.

In a recent paper, I argue that AI is likely to alter the speed of the law’s evolution. The technology will likely impact two core areas of law: contracts, which help to delineate rights and responsibilities of each party in advance of an agreement, and litigation, which is the process of determining responsibility after a dispute. Here, I describe how AI is likely to make both contract-writing and hiring trial attorneys cheaper. I then trace how that could accelerate the law’s rate of change in certain domains—a phenomenon I call hyperlaw.

AI’s Effects on the Practice of Law

AI productivity boosts will make both contracts and litigation cheaper. One of the most striking aspects of AI in law is how much it raises user productivity. Jonathan Choi and coauthors find that AI boosts legal task completion speed by 12%–32%. In a follow-up paper, Daniel Schwarcz and coauthors find that AI increases the overall productivity of law students doing legal tasks by as much as 130%. These effects will only continue to grow; over the next year, AI agents will likely be able to completely replace the first draft of most legal writing, with lawyers pivoting to just reviewing and commenting on such drafts. These productivity boosts will likely reduce the costs of both writing contracts and litigating disputes.

Cheaper contracts and litigation have opposite effects on demand for courts. First, AI will help attorneys write more complete contracts that preassign responsibility for more and more possible outcomes of an agreement. This could reduce the likelihood of an event in which responsibility has not been predetermined, lowering the number of disputes and weakening the demand for courts. I dub this the contracting effect. Second, cheaper litigation could reduce the need for complete contracts and reduce litigants’ incentive to settle a dispute out of court. This would boost the demand for courts, a result that I call the litigation effect.

The speed at which the law evolves depends on the number of court cases. When litigation overturns precedent, changes in the demand for courts, as outlined above, will affect how quickly the law evolves. Should litigation become less common (thanks to more complete contracts), the law’s evolution will slow. But should litigation become more frequent, the law will evolve more quickly.

These effects will vary across legal domains. At first glance, the likely net effect of AI on the law’s rate of change is ambiguous. Contracting and litigation effects offset one another. But neither effect will apply equally to every area of law. Where contracts are rare, as in tort law, legal change is likely to accelerate. Where contracts are common, as in property and contract law, the outcome is less certain.

Despite this ambiguity, there is good reason to think the law will feel some effects soon. For example, the three main tasks of an attorney are legal research, legal argumentation, and legal writing. Each involves producing or digesting large quantities of text, which is the attorney’s forte—and also AI’s. Attorneys have noticed. A 2026 Thomson Reuters survey found that 40% of attorneys across 27 countries worked in offices using AI. A year earlier, the figure was 22%.

The Number of Disputes

To see why AI could shape the speed with which the law evolves, let us consider the incentives to (1) implement a written contract and (2) litigate. The former influences how often disputes occur. The latter influences the rate at which parties pursue dispute resolution in court. Insofar as AI impacts each, courts will have more or fewer opportunities to tweak precedent.

The risk of disputes turns on a trade-off in costs between contracts and litigation. Suppose that a man named Quincy wants to buy a used van from a woman named Genevieve. It is hard for Quincy to foresee all the ways in which a dispute may arise. For example, the van’s battery could die a month after the sale, and Genevieve could refuse to pay for it.

Quincy has two ways to deal with the dispute risk. The first tries to avoid the costs of resolving a dispute after the fact. It involves hiring an attorney to delineate property rights up front, in a thoroughly written contract. For example, the pair may agree that Genevieve will pay for a new battery, tires, windshield wipers, and some routine maintenance for up to six months after sale, while Quincy will handle all other repairs in perpetuity. Such a contract, in principle, aims to avoid future disagreements by defining responsibility for some of the many potential problems with a used vehicle beforehand. If something goes wrong with the van after sale, the parties need only consult the contract to figure out who is responsible for repair costs.

The second option tries to save money on the up-front costs of producing a contract. In this case, Quincy can leave the contract relatively incomplete or unwritten. A less thorough contract might stipulate that Genevieve pay for a new battery but refrain from delineating responsibility for the many other possibilities. This alternative implies a mutual agreement to delineate responsibility later via dispute-resolution forums like courts. Should Quincy go this route, he avoids contracting costs but is more likely to pay for costly litigation.

Most contracts are incomplete, as filling gaps offers diminishing returns. Quincy benefits from making his contract more complete, but filling each gap requires attorney time (which costs money) and negotiation with Genevieve. Thus, a sensible strategy is for Quincy to address only the most important and likely issues, while leaving less critical issues unaddressed. Put differently, Quincy should improve his contract until the marginal cost of filling a gap exceeds the marginal benefit of filling it. His final contract is likely to be partly incomplete. Some gaps are just not worth filling.

AI has two offsetting effects on how incomplete contracts will be. AI makes attorneys more productive in writing contracts by, for example, automating drafting and reviewing, implementing longer and more complex clauses, and identifying gaps that might otherwise go unseen. That reduces the marginal cost of gap-filling, meaning that, all else being equal, the final contract is likely to be more complete. A more complete contract leaves fewer losses unassigned, with fewer occasions for disagreement. The demand for courts is lower.

But AI also reduces the benefits of making a contract more complete in the first place. Hiring an attorney is one of the chief costs of dispute resolution after the fact, and AI makes trial attorneys cheaper per task by making them more productive. The cheaper trial attorneys become, the lower the cost savings Quincy can expect to earn from making his contract more complete. As a result, AI can also make it economical to write less complete contracts. Why pay for a complicated, custom contract when hiring a trial lawyer is relatively cheap? In this case, the demand for courts is higher.

AI’s net effect on contract completeness and the number of disputes is unclear. The direction of AI’s net effect depends upon which cost AI reduces more: the cost of gap-filling or the cost of disputing. If the former falls by a greater percentage than the latter, then parties will fill more gaps in contracts and end up in fewer disputes. If the reverse occurs, then parties will fill fewer gaps in contracts and end up in more disputes. Which effect will dominate is difficult to predict.

The Settlement Rate

Although the overall effect of AI on the number of disputes is ambiguous, it is likely to increase the chance that any given dispute goes to trial. The reason is that AI makes litigation cheaper relative to settlement.

Suppose that Quincy is unlucky. His used van breaks down. Suspecting that Genevieve was careless, Quincy wants to recover his costs, whether through settlement or at trial. Genevieve wants to minimize her costs, whether through settlement or at trial.

Settlement is only possible if neither party thinks they could gain by going to trial. Quincy will accept a settlement only if it offers him at least as much as his expected benefit in going to trial: his chance of winning times the compensation amount ordered, minus his litigation costs. Genevieve will pay a settlement only if it is no larger than her expected cost of going to trial: her chance of losing times the ordered compensation, plus her litigation costs. Settlement is possible only in the overlapping range between Quincy’s minimum acceptable compensation and the maximum sum that Genevieve is willing to pay.

Cheaper trial litigation makes disputes less likely to settle. The litigation effect reduces the cost of going to trial for both parties. Quincy’s minimum acceptable payment therefore gets larger, and the maximum that Genevieve will pay gets smaller. As a result, the range of possible settlements will shrink, which may mean there is no possible settlement that both parties will accept. AI thus increases the chance that parties take their disputes to trial rather than settle.

AI and the Evolution of Law

An increase in the number of disputes going to trial could mean more than just a higher caseload for lawyers, potentially influencing the law itself.

A larger number of court cases means more opportunities to tweak precedents. Should AI change how often disputes occur (via changes in contract completeness) and how frequently they go to court (due to cheaper litigation), then AI is likely to change how quickly the law evolves. The reason is that the common law evolves as courts tweak precedent, and the rate at which such tweaks occur depends upon how many disputes occur and the rate at which disputes go to trial. An increase in either gives courts more opportunities to change precedent.

Legal change could accelerate in tort law, which lacks the contracting effect. The framework above predicts that AI will not have the same effect on every area of law. Tort law, for example, tends to govern disputes that occur without prearranged contracts, such as accusations of assault or defamation. In such cases, the contracting effect is not present, but the litigation effect is. Cheaper litigation will therefore increase the rate at which the current rules of tort liability are challenged and ultimately overturned, giving courts more opportunities to change precedent in tort law.

In areas of law subject to both effects, results will depend on which effect is stronger. In property and contract law, parties often have the foresight to write detailed agreements. Both the contracting effect and the litigation effect are present, so whether the law evolves more or less quickly depends upon the relative strength of the two effects. In contexts where contracts become more complete, fewer disputes will occur. However, as discussed above, when disputes do occur, parties will be more likely to go to trial rather than settle, giving courts more opportunities to tweak precedent. Thus, the contracting and litigation effects pull in opposite directions, and the common law may therefore evolve more or less quickly depending upon which effect dominates.

By contrast, in contexts where litigation costs fall by a greater percentage than gap-filling costs, relatively more disputes will occur; in any such disputes, parties will be more likely to go to trial. Both margins move in the same direction, and the prediction is unambiguous: the common law is likely to evolve more quickly, because courts will have many more opportunities to change precedent.

Hyperlaw in Society

If the analysis above is right, what kind of society does hyperlaw produce? It depends. Below I consider three possibilities.

First, hyperlaw implies a body of law that continuously and quickly updates to changing circumstances. The common law took decades to adjust to railroads and assembly lines. By contrast, a hyperlaw system of torts might adjust to autonomous vehicles, for example, in a decade or less. Liability rules for innovations like driverless cars and drones may update continuously and quickly, thanks to cheaper litigation as the technology evolves. But such a system is not without costs. It may also, for example, limit economic growth. The reason is that, if the law changes too quickly, investors will face considerable uncertainty about the rules that govern their returns. In response, they may refrain from making otherwise worthwhile investments.

Moreover, hyperlaw may create opportunities for institutional arbitrage. Insofar as tort law evolves quickly and contract law does not, some litigants may be incentivized to portray contract disputes as tortious, to take advantage of its speedy evolution. Others who prefer stability may instead opt to contract their way out of the courts altogether, taking advantage of reduced contracting costs.

Finally, there is one important reason that hyperlaw may never come to be. The aforementioned possibilities do not occur in a vacuum. Courts that risk being overwhelmed by litigation may offset the effects of hyperlaw by raising filing fees or the requirements for standing, for example. Such adjustments will increase the costs of litigation and, in turn, screen out disputes that may have otherwise tweaked precedent. Ultimately, whether hyperlaw is a realistic future depends upon AI’s legal capabilities and how courts respond to them.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Henry Thompson is an Assistant Professor of Economics at the University of Mississippi and an Affiliated Faculty Member at the University of Mississippi School of Law. His research spans political economy, law and economics, and public choice with a particular focus on the alternative customary, organizational, and contractual arrangements people develop with one another when they cannot use government to protect their property rights. His recent work analyzes organized crime, criminal governance, artificial intelligence and the law, and existential risk from artificial intelligence and has been published in journals such as The Journal of Law & Economics, Public Choice, and Kyklos.

Two days before Slovakia’s 2023 parliamentary election, a fabricated audio clip spread widely on social media, appearing to catch a leading candidate and a prominent journalist discussing how to rig the vote. The following year, nonconsensual pornographic deepfakes of Taylor Swift amassed over 45 million views on X in less than 24 hours. In summer 2025, an elderly woman in Ontario lost her life savings to a cryptocurrency scam, which began with a Facebook advertisement featuring a deepfake of Prime Minister Mark Carney.

In each of these cases, harm might have been reduced by content provenance metadata: information attached to content that declares how it was produced. (Generally, the term “provenance” refers to a record of where something came from and what happened to it along the way.) Content provenance tools can’t directly prove whether a bare image or video is real or synthetic; no technology reliably can. Instead, they serve as an evidentiary layer of AI governance, producing records that other protections build on top of. As signed provenance becomes the norm for content captured on cameras and microphones, fabricated material—with no or suspicious provenance—will attract more scrutiny.

New EU and California laws mandate content provenance. New legal requirements, rolled out in two major markets on the same day, will significantly increase adoption of provenance techniques across the AI ecosystem. On August 2, 2026, both the EU AI Act’s Article 50 and California’s AI Transparency Act took effect. These regulations expand the creation and preservation of digital records that make it easier to trace a given piece of content back to the cameras, AI models, and other tools that created or captured it.

This essay explains what content provenance is, how it can be implemented, and its benefits and drawbacks. Then, I’ll cover the two laws that have taken effect, and their global impact. As a disclosure: I lead policy at Encypher, which develops infrastructure for content provenance.

What Content Provenance Is and How It Works

The new EU and California laws require some companies—including camera manufacturers and AI companies—to attach certain metadata to their products’ outputs. Generally speaking, that metadata falls into three main categories: first, origination details describe where the output came from, such as which camera photographed it. Second, chain-of-custody information describes which platforms or tools handled the content after it was made. Third, a modification history tracks any changes made, such as cropping or face-swapping.

The compliance ecosystem is converging on a technological standard: C2PA. The Coalition for Content Provenance and Authenticity (C2PA), whose user-facing implementation is known as Content Credentials, maintains an open, royalty-free standard that many companies will likely use to comply with the EU and California laws. Neither law requires C2PA by name, but its interoperability and growing adoption make it the leading compliance option. Anthropic, Google, and OpenAI have all indicated they will adopt C2PA.

C2PA creates provenance records using cryptographic techniques. For example, suppose that someone takes a photo with their smartphone. First, the phone “hashes” the photo’s pixels and provenance metadata, generating numerical tags uniquely tied to the data. Then it generates a cryptographic “signature” linked to these hashes and to the smartphone manufacturer. It stores the hashes and signature in the photo’s metadata. As a result, tampering typically leaves clear evidence: if anyone modifies the photo or its provenance metadata, a quick computational check can detect that a change has occurred. These same methods apply to audio and video files.

Edits create new provenance records that reference earlier records. In C2PA parlance, each version of the content gets its own “manifest”—a cryptographically signed provenance record describing the latest changes. Continuing the example above, if someone later modifies the photo in Adobe Photoshop, then Adobe can create a new manifest describing the edits it applied, storing this manifest in the photo’s metadata. Before signing, Adobe hashes the prior manifest and adds that hash to the new manifest, so a quick computational check can detect tampering with the prior manifest.

A provenance record can declare that content is AI-generated. Under C2PA, provenance metadata contains a field labeled “digitalSourceType,” which states whether the content was captured by a camera or microphone, generated by AI, or a combination of the two (such as a photograph with an AI-generated background element inserted in Photoshop). The EU and California requirements lean heavily on this information.

Content Credentials show up as small, familiar cues. Creators will see a simple toggle to “attach Content Credentials” in Photoshop or Firefly, or their camera will sign photos automatically. Viewers may see a small “cr” badge on the content, which they can click to see the metadata. Platforms will see machine-readable data they can turn into on-screen labels.

Limitations of Content Credentials

While Content Credentials are a promising way of fulfilling the new EU and California requirements, they are not infallible. They ultimately need to be paired with technical and governance measures to ensure that metadata is accurate and remains attached to its content.

Technical vulnerabilities mean Content Credentials can be removed or falsified. The most common failure of Content Credentials is metadata stripping, in which the process of uploading files or converting file formats discards manifests automatically. The second is the analog hole: if a user screenshots or rerecords content, the original manifest doesn’t carry over to the new file. The third issue is signing-through-camera: point a camera that signs images automatically at a deepfake on a screen and it will honestly sign that it captured the image. The fourth is forged manifests: skilled attackers can build manifests that contain false assertions. Additionally, researchers continue to probe whether C2PA’s cryptographic validation itself can be defeated.

Mitigations can help reduce these vulnerabilities. To address metadata stripping, two technical fixes are to include invisible watermarks such as Google’s SynthID and to use “digital fingerprints,” which let a stripped file be rematched to its manifest. When these measures are combined with Content Credentials, it becomes far harder to sever the record from the content it describes. The California AI Transparency Act also requires that, beginning in 2027, large platforms not strip standards-compliant provenance data.

The analog hole problem, meanwhile, can be mitigated as norms change. While there’s no way to prevent users from stripping provenance metadata by rerecording content, material without provenance metadata will eventually be seen as less trustworthy for this reason. As camera and microphone manufacturers widely adopt C2PA, bare content will stand out, reducing the incentive to strip metadata.

For signing-through-camera, emerging technical solutions such as Sony’s Camera Authenticity Solution embed 3D depth information in metadata, revealing whether the photo captured a real scene or a flat screen. The threat of forged manifests, meanwhile, requires both technical and institutional responses. For example, conformance programs can verify that products adhere to the Content Credentials technical specification and that their signatures cannot be manipulated. The specification itself can also be revised to address vulnerabilities.

Private and public institutions can help determine which signers are trustworthy. As discussed above, C2PA uses cryptographic signatures for security, and each signature traces back to a particular signer (e.g., a smartphone manufacturer). A signature answers which credential signed a claim, but who decides which signers are trustworthy? Currently, certificate authorities vet signers, and the C2PA Trust List governs which of those authorities receive official approval. In this certification system, private actors police themselves; major firms building the tools that create and sign content, such as Adobe, also set the conformance rules and decide which signers are trustworthy. Public memory institutions such as libraries, archives, and museums could act as independent, mission-driven custodians in this process.

There is a long-standing tension between transparency and privacy. While provenance brings benefits, it also carries the risk of misuse. Provenance trails can sometimes reveal who filmed a video, where, and when. Journalists who wish to prove the authenticity of a piece of media might also struggle to protect dissidents and whistleblowers. While there is an inherent trade-off between transparency and privacy, C2PA tries to strike a balance. Most C2PA assertions are optional; sensitive fields can be redacted; and certificates can be pseudonymous.

Content Credentials have been adopted widely but unevenly. Several generative AI tools already sign content by default, including Adobe Firefly and OpenAI’s DALL·E 3 and Sora. Meanwhile, Midjourney still embeds no manifest and has not committed to a timeline for implementing one. Adoption in hardware is also growing. Leica’s M11-P was the world’s first camera to have Content Credentials built in. Now, Sony and Canon have followed suit, alongside Google’s Pixel 10. The most consequential gap is Apple, which hasn’t yet committed to implementing C2PA or given a public explanation as to why. California’s capture-device rule will increase the pressure on Apple and other major manufacturers to adopt standards-compliant provenance beginning in 2028.

What the EU and California Acts Require

The EU and California laws create several different obligations. Some center on generating and preserving provenance records; others focus on using those records to create appropriate labels and notices.

The EU obliges generative AI providers to create provenance records. In the EU AI Act, the single most important obligation is Article 50(2): providers of generative systems must ensure those systems automatically mark synthetic audio, image, video, and text output in machine-readable form. This obligation is the key to an evidentiary layer that can automatically detect material with trustworthy provenance.

The EU’s other two duties focus on disclosure. With some exceptions, Article 50(4) requires deployers of AI systems to label deepfakes and to disclose AI-generated text on matters of public interest. Article 50(1) places a similar requirement on AI providers to tell people when they are interacting with an AI system.

California’s law goes further by requiring online platforms to retain provenance data. The California AI Transparency Act applies to providers of generative AI systems that have more than a million monthly users and are accessible in the state. The headline obligations on these providers, effective August 2, 2026, are to include embedded disclosures and offer a free tool that the public can use to surface content provenance data. These requirements act as California’s equivalents to the EU’s marking-and-disclosure duties, and they require the embedded data to be difficult to remove. Starting on January 1, 2027, large online platforms must detect and surface standards-compliant provenance data attached to content, and—critically—refrain from stripping it. This rule will greatly increase the durability of provenance metadata, making the whole content provenance ecosystem more useful.

California’s law requires many hardware manufacturers to create provenance records. The final component of California’s law, effective January 1, 2028, extends provenance requirements beyond AI-generated content to content recorded by devices such as smartphones, cameras, and voice recorders. These devices must create provenance records by default, though manufacturers may give users an option to turn this feature off.

Enforcement can add up to enormous sums. Under the EU AI Act, national regulators can impose fines of up to €15 million or 3% of worldwide annual turnover, whichever is higher, for breaching the Article 50 transparency duties. The percentage-of-global-revenue model scales with the size of the offender; for a large firm, fines could grow to hundreds of millions of dollars. In California, the Attorney General, city attorneys, and county counsel can seek civil penalties of $5,000 per violation, which look modest by comparison. However, each day of noncompliance is treated as a new violation. The fine for a single noncompliant product, multiplied across days, can quickly add up.

The EU and California Laws’ Global Impact

The principle of transparency around AI-generated content has long been discussed in AI governance and has appeared in international agreements in recent years. However, the laws enforced in the EU and California will, for the first time, set an effective global baseline for transparency requirements.

The notion of content provenance is not new, but requirements vary widely. AI governance documents from the OECD, UNESCO, the G7, and the Council of Europe have already articulated content-provenance norms at the international level, although they are mostly nonbinding. The United States still has no general federal content provenance law. At the state level, both Utah and Washington have enacted their own content provenance laws aligned with California’s AI Transparency Act.

China’s laws also fuel efforts to build provenance infrastructure. Beyond the EU and the US, the most prescriptive approach is China’s labeling regime. Its Deep Synthesis Provisions, Interim Generative AI Measures, Measures for Labeling of AI-Generated Synthetic Content, GB 45438-2025 national standard, and other regulations and institutions already require AI-generated content to be marked. However, China has not yet extended comparable requirements to hardware manufacturers.

The EU and California laws apply to content that ends up within their borders. Neither the EU law nor the California law hinges on where a company is incorporated or headquartered, or where its servers sit. Instead, both laws center on market access and effects—whether the system is placed on the market or publicly accessible in the jurisdiction, and whether its outputs are used by people there—with the California AI Transparency Act adding a one-million-monthly-user threshold.

The laws’ footprints are effectively global. Digital content does not respect borders, and no generative system can guarantee that its outputs will never appear in the EU or California—two of the largest and most affluent markets in the world. Attempting to run a compliant environment for these markets while running a noncompliant one everywhere else could be inconvenient and expensive. Instead, companies may apply the stricter standard to all output by default. As a result, these region-specific mandates are set to have an international impact.

How Provenance Mandates Support Courts in Tackling Deepfake Evidence

The primary effect of the new EU and California mandates is to require the creation of signed, time-stamped, tamper-evident records that document data origin and custody. Such records will be generated at scale, as a matter of legal obligation. Records are evidence, which raises a question: how should courts use this evidence?

Advisory bodies are considering responses to deepfakes presented as evidence. The Advisory Committee on Evidence Rules, which proposes amendments to the US Federal Rules of Evidence, drafted a working Rule 901(c) to handle evidence fabricated by generative AI. Consider a case in which a party claims that a piece of evidence presented against them is a deepfake. Under the working draft, that party would first have to produce evidence sufficient to support a finding of fabrication—a lower bar than proving falsification outright. Only then must the item’s proponent show the judge that it is more likely than not authentic. If such a rule were adopted, validated provenance records would be powerful evidence for meeting its test.

Provenance records can help courts weigh authenticity, though their absence must be read with care. The machinery of 901(c) turns on whether an item was generated or altered by AI, something courts have had no dependable way of establishing. Validated provenance records offer an important factor to consider. A missing manifest, by contrast, proves little by itself: not all cameras create one, and platforms routinely strip provenance information. Absence becomes significant only where the facts of the case suggest that provenance information should be present—a situation that will grow more common as California’s and the EU’s provenance mandates take effect. In such situations, an unexplained gap begins to raise a question about the party that could have signed and did not, much as courts already draw inferences when a party fails to preserve evidence it had a duty to keep.

New technologies have often presented courts with novel challenges when it comes to identifying authentic evidence. In the 1800s, courts spent decades working out how to treat photographic evidence. The following century, fingerprinting and DNA became more reliable and useful in court as institutions—registries, labs, and standards—grew to support the collection and analysis of such evidence. Whether C2PA will become a global default for verifying the status of images, audio, and other data remains an open question. But with the arrival of EU and California laws, courts will at least have a better record of how a digital item came to be and how it was altered. Content provenance is the evidentiary layer the AI era has lacked.

Work on this essay was supported by the Summer Research Fellowship at the Institute for Law & AI.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Eddan Katz is the Head of Policy & Government Affairs at Encypher. He is also a Research Fellow with the LexLab at UC Law San Francisco, writing on and teaching AI liability. Previously, he was Executive Director of Yale Law School’s Information Society Project (ISP), was the International Affairs Director at the Electronic Frontier Foundation (EFF), the Project Lead for the AI Governance platform at the World Economic Forum’s Centre for the Fourth Industrial Revolution network, and worked on the AI Policy team at Meta.

AI systems now write so fluently that cheating at universities has become ubiquitous and “AI slop” is displacing human writing across the internet. They have begun producing original proofs on long-standing open problems in mathematics. Researchers at the frontier AI companies have largely stopped writing their own code, and the models’ hacking abilities are strong enough that the US government temporarily export-controlled Anthropic’s Claude Fable 5, days after its release. Ten years ago, every claim in this paragraph would have sounded like science fiction.

It is easy to become numb to this progress. Today’s AI agents remain endearingly clumsy; watching Claude run a small store is a bit like watching a child running a lemonade stand. But we should not let such awkwardness distract from the trend. These systems have evolved from research curiosities to extremely powerful economic engines in 10 years, with no end in sight. I want to take that trend seriously and ask: if AI does get to the stage where it can do the cognitive work that humans do, what happens to the economy?

With notable exceptions, economists have mostly declined to address this question; their usual maneuver is to deny the premise, as when a widely cited Nobel laureate’s analysis put AI’s contribution to US GDP at about 1% over the coming decade, by essentially freezing 2023 capabilities for 10 years. But the question is worth taking seriously, because automating cognitive labor releases what has always been the critical brake on physical production: no matter how cheap machines and tools become, you cannot manufacture new workers. In this piece, I will assume no additional new technologies and no recursive self-improvement to superintelligence. Yet, even with these conservative assumptions, I will show that the economy stands to be profoundly transformed. For the first time in history, physical production could be fully automated.

The rest of this piece makes that case and considers the implications. I argue that AI capable of cognitive work could also do physical work (since building the machinery it would need is relatively straightforward) and that its labor would be cheap. I then turn to input-output tables—the US government’s accounting of what every industry buys from every other—to ask how fast a fully automated economy could grow, finding that physical output would double roughly every year rather than every few decades (as it does currently). Finally I ask whether anything could stop such an industrial explosion, and what the implications are for power at home and abroad.

Why AGI Gives You Robots

The term “artificial general intelligence” (AGI) has been endlessly used and abused, but it points to a clear core concept: an AI smart enough to do the work humans can do, at a competitive cost. You could hand it a task, as you would to a skilled employee or contractor, and expect the job done competently. Current systems, for all their rapid progress, are not close to AGI; my own attempts to automate my job have been unsuccessful. However, once AGI appears, we should expect it to be able to perform physical work through robotics.

An AI system that can do every remote job can also operate a robot. Definitions of AGI focus on cognitive work, but there is nothing special about the physical world that confines intelligent agents to their data centers. Remote cognitive work is hard: doing all of it means mastering the real-time control, spatial reasoning, physical prediction, and continual learning required for such varied tasks as mechanical engineering and animation. A system with those skills has what it needs to operate machinery, if given actuators to work with.

Progress in AI is already pulling robotics forward. The general-purpose methods that cracked language and vision turn out to be good at manual manipulation too. Large language models can increasingly supply high-level control: today they run robot arms, fly drones, and program robot dogs. Hobbyists have even wired them into homemade robots. Meanwhile, the same training methods, applied to recordings of humans teleoperating machines, are producing the low-level dexterous control that humans perform instinctively; today’s systems can fold laundry and assemble cardboard boxes, using ordinary cameras and simple grippers.

Hardware is not the bottleneck for robotic automation. Industrial arms have demonstrated precision and force beyond any human arm for over half a century. But, because they could neither see nor think, every motion had to be programmed in advance, at massive cost. The other work-around was a human operator: remote manipulators have handled radioactive material since the late 1940s, built underwater structures using remote-controlled submersibles, and performed surgery. That remote control was worthwhile only when a person could not be physically present. But it does show something we’ve known for decades: a machine under competent control can do skilled physical work. Once control can come from AIs rather than people, it becomes worth building general-purpose machinery for them.

Robots will not necessarily resemble humans. While the human hand is a remarkable instrument, robot hands are already fairly good, and certainly better than the split-hook prostheses—two rigid fingers on a cable with no sense of touch—used by amputees to farm, weld, and perform all manner of other tasks. With AI, simple actuators go far, and AGIs will have intelligence and patience to spare. Nor will the machines work at human workstations forever. Production demands dexterity today because every process was designed around human workers with hands. New possibilities open up when the workers are machines; just as today’s programming agents do not type at keyboards as their human counterparts do, robotics tools need not be gripped and triggered by fingers—they could be mounted directly on a robotic arm instead.

Machine Labor Will Be Cheap

Automating labor is not free. But the computer chips and robotic actuators that replace workers are simply more capital goods for the economy to produce—and they’re not particularly expensive ones. A humanoid robot, broadly speaking, uses parts similar to those of a car: metal, motors, batteries, electronics, and sensors. At automotive production volumes, it should cost tens of thousands of dollars; Unitree already sells its child-sized G1 humanoid for around $13,500, and Tesla is reportedly aiming for $20,000 for its full-size Optimus. If a robot costs $30,000 and, working around the clock, can substitute for a single human worker who costs $30 an hour, it could pay for itself in six weeks.

AI cognitive labor is generally significantly cheaper than the humans it replaces. It is hard to price the future AI cognitive labor involved in an industrial explosion, because it does not exist yet. However, on the tasks AI systems can already do—like transcription, routine translation, or writing one-off scripts—they are typically far cheaper, per unit, than the people they replace, and the cost of a fixed level of AI capability falls rapidly year over year. Even if the first AGI arrived merely cost-competitive with human workers, within a year it would cost a fraction as much.

Input-Output Analysis

How fast would the economy grow once human labor is automated? At first blush, this looks hard to answer. Standard growth models assume that labor and capital substitute for each other, treating each as a single dollar-denominated aggregate—barristers and bricklayers combined into “labor,” bulldozers and bridges into “capital.” Such abstractions are poorly equipped for a world where labor is unnecessary and capital reproduces itself, and classic concepts like GDP can become profoundly misleading.

Input-output analysis helps forecast how an automated economy would grow. Input-output analysis takes a different approach, recording what each industry physically needs from others. The method was developed by Wassily Leontief, in the 1930s, to study the relationships between industrial sectors. In 1945, Leontief used his tables to project the United States’ 1950 steel requirements to within a couple of percentage points. Military planners, concerned about war with the Soviet Union, took up the method to study how fast industry could remobilize.

Input-output analysis tracks what everyone buys from everyone else. The basic idea is simple: ask every business what it buys and from whom, then aggregate the responses into tables that track what each industry buys from others. An entry for aluminum smelting might record the electricity, ore, and machining required to produce aluminum, while an entry for aircraft manufacturing records how much aluminum is required. The Census Bureau runs this survey every five years, and the Bureau of Economic Analysis assembles the results. The tables take years to build, so the most recent set covers 2017. Companion tables record the equipment and structures that each industry holds, from machine tools to chip fabs. Entries are recorded in dollars, but they are detailed enough to closely follow the underlying kilowatt-hours of electricity, tons of steel, and other physical quantities. Most output is ultimately consumed; the rest is reinvested, either adding to physical capital or replacing what has worn out.

Once robots can replace human labor, the economy will no longer be bound by a fixed workforce. It can produce every input it needs, including the “workers” themselves, so output can be fed back into building more capacity, and growth compounds. John von Neumann worked out how to compute the maximum rate at which such an economy could expand. Combining his analysis with the production data in the tables tells us how fast an autonomous industrial sector could grow.

Input-output analysis measures economic growth in physical output, not value. This approach to forecasting growth tracks how much stuff can be produced. I make no attempt to convert this into GDP, because an industrial explosion would upend the prices any such conversion relies on: when machines produce everything, including more machines, goods become exponentially cheap. Physical output is also the better guide to what is at stake. Military power, human employment, and material abundance all depend on how much gets built, rather than on its dollar price.

How Fast Could an Autonomous Economy Grow?

Using the US government’s 2017 input-output tables, which track 402 industries, I find that a fully automated economy using US production methods could double its output roughly every year; input-output tables for other advanced economies give comparable results. The robots and computer chips needed to automate production represent only a small fraction of total output; even a tenfold increase in their cost would not impact the growth rate significantly. Construction lags can be incorporated too; even with these delays included, the economy still doubles in well under two years.

‍Why does a fully automated economy accelerate so much faster than the current one? The main difference is that an automated economy can reinvest far more of what it produces. Building more factories today is not useful; there would be no workers to staff them nor to use the extra machines and tools produced by them. Thus, we consume about four-fifths of what we produce, and most reinvestment goes to replacing worn-out capital rather than adding to it. AGI would remove this obstacle, allowing robot workers to be manufactured along with other physical goods and thus breaking the link between human labor and physical production. The pace of growth is then set by how fast the machine economy can copy itself, which input-output analysis shows us is fast.

The true growth rate of an automated economy is likely even higher. Our estimate of this maximum growth rate uses 2017 American production methods; therefore, it’s conservative. Those methods were designed around human workers at human wages and are a poor fit for a robot economy; letting the production methods change makes it grow considerably faster. Assuming 2017 methods also ignores the efficiencies that come with producing at greater volume. However, it also neglects the depletion that makes raw materials costlier to extract, which I return to below.

Industrial explosions happen even without new science and technology. This assumption might seem strange to economists, because standard theory predicts that growth at the technological frontier requires new ideas. But that prediction assumes an economy in which the workforce cannot be manufactured; an economy that can build its own workers grows by building more of them and needs no new ideas to do so.

An economy growing at this maximum rate looks very different from today’s. Its output is machinery, materials, energy, and factories rather than consumer goods and services, and its workforce is robotic. Table 1 shows its major sectors, by share of that workforce, of output, and of energy consumed. The workforce such an economy needs is overwhelmingly industrial: robotic arms on factory floors, self-driving excavators and cranes, and manipulators on mobile bases of all sorts and sizes. Automating physical production means automating construction, machining, and assembly; whether AI can substitute for doctors, lawyers, or babysitters is irrelevant to how fast this economy can grow.

‍In reality, some output would directly support human needs instead of further growth. The same machines, materials, energy, and robotic labor could make homes just as well as they could make factories. Robots could also cook, clean, and drive for people. However, an economy that devotes significant resources to human consumption could still grow quickly in absolute terms, according to my forecasts. For example, an economy reinvesting half its output would double every 20 months, and the stream of goods and services flowing out to people would double every 20 months as well. Reinvesting only half is arguably conservative—common sense suggests (and economic theory concurs) that countries and companies controlling the machines would reinvest most of their output to stay competitive with one another.

From Investment Boom to Unprecedented Growth

Let’s pretend that AGI arrived today. What would happen? The factories and machines needed to automate production do not yet exist.

Input-output analysis can tell us how a realistic robotics build-out would go. The standard economist’s approach is to imagine a planner assigning each industry’s output either to consumption or to building new capacity, trading off consumption now against faster growth later, according to how much people value each. I have modeled this transition using the 2017 tables at a simplified 71-sector level, under a range of assumptions about those preferences.

AGI would create an enormous investment opportunity. In an AGI-enabled world, building robots will be an industry comparable to building cars, and an automated factory costs little more than an ordinary one while needing no workers to run it. Capital would flood in.

There would be significant competition for the robotics supply chain. Investors would start outbidding everyone else for machinery and construction workers. Resources that would otherwise be consumed, or spent replacing worn-out equipment, would be used to build new robots, automated factories, and computer chips. In the short run, the prices of physical goods would rise, not fall—and, because making them would suddenly pay better, idle and underused factories would ramp up production. Plausibly more human workers, not fewer, would crowd into factories while the boom lasted.

It would take several years for the economy’s overall production to double. While the nascent autonomous sector itself would grow rapidly, the starting baseline is low: today’s economy makes plenty of consumer goods and services, but it has few autonomous robots, and the machine tools needed to build more are themselves in short supply. Electricity generation would grow relatively slowly, since it is already a large part of the economy and requires significant capital to expand. Still, it would double in about four years, and double again in another two. Robot production would follow a similar schedule, from near zero today to tens of millions of units within a few years.

An industrial explosion would not create instant material abundance. Despite this tremendous growth, demand for physical goods would take time to satiate. Bringing the whole world to the living standards of a wealthy American would take more than 10 times today’s physical output—and that is before counting the goods and services that cheap production would newly make accessible, from custom-built mansions to round-the-clock personal care.

Natural-resource constraints alone would not prevent an industrial explosion. The autonomous sector needs no human labor or inputs from the nonautomated economy, and there are minerals enough for economies far larger than today’s. Mining is only about 1% of output, so even large cost increases do little to slow overall growth.

The one resource that would truly run short is oil; cheap reserves would be exhausted once the economy had grown severalfold. Coal and natural gas could power growth for far longer, but fossil fuel burning is ultimately limited by its climate effects rather than availability. An economy that declined to burn them would electrify, replacing fossil fuels with solar panels, batteries, and nuclear plants. At today’s production costs, this would slow growth—with doubling times closer to two years than one—but not stop it.

Long before demand or minerals ran short, the assumption of frozen technology would fail. Within a few doublings, AI minds would vastly outnumber human ones, and whatever technology they built would transform production more radically still.

What Could Stop an AI Industrial Explosion?

US Admiral William D. Leahy assured President Truman that the atomic bomb “will never go off, and I speak as an expert in explosives.” But the viability of a nuclear chain reaction was a question for physicists, not admirals, and the consequences of that physics were no less real for being counterintuitive. So it is with AGI.

Once physical production is automated, physical output can double every year. Objections to explosive growth that do not engage with this data miss the point. Cost disease is real: human services will become expensive relative to manufactured goods, and there are jobs, like babysitting or singing, that we may want humans to keep doing. Neither fact stops physical production from expanding on a scale we have never seen. New research ideas may become harder to find, but that too is beside the point, since the calculation assumes no research at all.

Nonautomatable production is a limited bottleneck on explosive growth. More serious reasons to expect slower growth come from the dynamics of the build-out itself. One possibility is that some small set of tasks cannot be automated. This changes the growth dynamics less than you might think, because the workers freed from every other task can supply the missing ones. If humans must still handle one task in 20, then production could grow 20-fold before it needed more workers. By the time the machine economy has grown 20-fold, we will probably have worked out how to automate those final tasks.

The time taken to integrate AGI may delay an industrial explosion but not stop it. Technological diffusion—the lag between a technology working and being used everywhere—is another potential slowdown, but at worst it delays the explosion rather than stopping it. No one knows whether AGI will arrive in 2030 or 2050; next to that, a few years of diffusion is a rounding error. In any case, the diffusion of AGI workers will look more like the arrival of skilled laborers in the economy than the spread of automobiles or trains. This is particularly true on the software side: generative AI has already been adopted faster than the personal computer or the internet, continuing several decades of ever-faster technological diffusion.

Explosive growth may be possible, but we may choose not to allow it. Regulations will undoubtedly slow deployment. But stopping it entirely is very hard, because an automated economy could be built almost anywhere and the profits would be enormous. If one US state regulates the robots away, the build-out moves next door and the permissive states collect the windfall. If the United States forswears the technology while China does not, China soon commands a vastly larger industrial base. Even if both agreed to stop, they would still have to prevent every other country from importing or developing the requisite robots and AI systems. And the temptation would never fade: a country that allowed the robots would be vastly richer within a few years. Someone would say yes.

Power Without People

War planners funded Leontief’s input-output tables to learn how fast industry could be turned into arms. Military power still rests on industrial capacity, and an automated economy would outproduce any rival in every kind of weapon, from artillery shells to drones. It would need no defense workers to build the arsenal, and ever fewer soldiers to wield it. A state with a small population but an autonomous industrial base could field far more firepower than a much more populous state without one. Remove the need for labor, and many countries have raw materials enough for an enormous industrial base. And a two-year head start in a doubling economy means four times the production.

The same decoupling operates inside states. Governments have always needed their citizens as taxpayers and soldiers, and that need has quietly underwritten political accountability. Horses once worked fields and carried cavalry, before the engine made them surplus; Leontief famously expected a similar fate for human workers. Citizens who neither supply taxes nor serve as soldiers would lose the hold that they gained from the state’s dependence on them. Rulers funded by oil wells rather than taxpayers can already afford to ignore their people, and often do. Full automation could make every state a petrostate.

While this may seem dire, none of it is inevitable. The same machines that build drones can build houses and hospitals, and within a decade or so of full automation, it would be possible to extend the living standards of a wealthy American to every human. Our biggest constraints will be political rather than physical—there is nothing in the machines themselves that needs to prefer an artillery shell to a dinner plate.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Damon Binder is a Senior Researcher at Coefficient Giving (formerly Open Philanthropy), where he investigates existential risks from biology and advanced AI. He previously worked at the Future of Humanity Institute, Oxford University, and has a PhD in physics from Princeton.

On July 28, over a thousand employees of the world’s top AI companies advocated that the US government “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Following months of cybersecurity scares and their own loss of control incidents, both OpenAI and Anthropic officially endorsed the same message, recognizing the danger of blindly accelerating AI development.

Despite this new urgency, many people argue that coordinating an international slowdown is currently unworkable—including some of the same groups in favor of one. Even if the US slowed down its own AI development, it wouldn’t be able to make sure that China was doing the same, leaving the US no choice but to race.

These anti-slowdown arguments usually emphasize the technical challenges with designing AI monitoring measures to ensure a slowdown is being respected. For one thing, slowdown skeptics argue that countries would refuse to install verification measures unless they were privacy-preserving enough to avoid leaking trade secrets and other sensitive information. Then, the skeptics point out that near-term privacy-preserving verification technology is not yet robust. It would be extremely hard to make sure that China was not undermining whatever monitoring system was watching its GPUs; if the Chinese government were actively trying to subvert a slowdown, it could use advanced techniques to fake workloads, including physical key extraction or even laser bit-flipping. Getting to the point where we can start implementing an international slowdown, the argument goes, might have to wait on years of R&D to solve these problems.

This perspective is missing the forest for the trees. Privacy-preserving verification tools that are immune to state adversaries would be nice, but they are not a prerequisite for an AI slowdown. Robustly verifying limits on AI development is completely feasible with low-tech methods: if there were enough political will to implement a joint slowdown, states could extensively monitor labs for unsanctioned behavior by simply placing human inspectors on the inside—an approach we call Whole-Lab Inspection (WLI).

Under this proposal, inspectors from the US and China would receive broad physical access to frontier AI companies alongside read-only access to corporate communications, worklogs, code repositories, and compute telemetry. Using monitoring systems the companies already operate, they could observe operations and investigate potential violations—giving them the visibility of a CISO without requiring the powers of an operator. By literally and figuratively looking over the shoulders of lab employees, inspectors would likely be capable of robustly verifying fine-grained restrictions on AI development, such as bans on autonomous AI R&D and limitations on post-training. WLI would officially make AI companies transparent to auditors from competing nations, though we will later discuss how this arrangement may not be much different from the IP leakage of the status quo. Overall, WLI demonstrates that an international slowdown is already technically possible; it’s just a question of whether the US and China have the political will to jointly implement it.

In the near future, political will to implement a slowdown could rapidly appear. If developed and diffused quickly, advanced AI systems are likely to intensely destabilize society by democratizing weapons of mass destruction and fueling waves of unemployment. On top of these threats, allowing an intelligence explosion would risk military escalation and loss of control, as states race to sabotage and deploy powerful AI systems for strategic advantage. Should such intense instability come to pass, if the US slowed down its development domestically and offered a reciprocal audit, China would have a strong incentive to agree.

In what follows, we’ll argue that whole-lab inspections can be used to robustly enforce fine-grained limits on AI development. Then, we’ll speak to the incentives that might encourage the US and China to pursue a joint slowdown at some point using WLI. Finally, we’ll discuss the stability and risk reduction benefits of a well-timed slowdown.

Enforcing a Slowdown Through Whole-Lab Inspections

In order to agree to a slowdown, the US and China would need to know what research was being conducted inside each other’s frontier labs. Fortunately, visibility into AI development could be achieved by sending human auditors to comprehensively monitor labs.

Auditors could have high visibility into AI companies. Monitoring for disallowed training runs or prohibited research activities may be largely trivial with human inspectors. In large part, this is because AI companies already have extensive internal monitoring systems. Most AI companies record employee screens and keystrokes to produce training data and help with automation. Companies also constantly save logs from experiments and jobs submitted to keep track of compute use and archive research progress. Giving inspectors comprehensive visibility into a lab would likely be as simple as integrating and handing over read-only permissions to internal logs and company communications. To aid in their oversight, inspectors might direct a company’s own AI agents, or trusted external AIs, to read through troves of company data in search of potential violations for further review.

Subscribe now

Alongside this digital information, inspectors could also work on-site alongside lab and datacenter employees, giving them physical access to hardware and research projects. Just as for nuclear power plants, banks, or any other high-risk industry, inspectors would be able to question employees about their work, as well as physically inspect infrastructure by hand for compliance. Armed with such a broad set of simple verification techniques, inspectors could make it extremely difficult to conduct unapproved training runs unnoticed—like trying to get a plane off a runway without being spotted by an air-traffic controller.

If WLI were established, governments could then enforce a wide range of policies to slow AI development:

Preventing an intelligence explosion. The most dangerous path for AI development to take would be a full-throttle intelligence explosion—using AIs to autonomously design new, even smarter AI systems recursively. Since AIs can work much faster than humans, and since this process might quickly result in AI systems more capable than their human monitors, it’s likely that it would become impossible for humans to effectively oversee their AIs or intervene if they misbehave. To prevent this, inspectors could enforce targeted interventions designed to specifically restrict autonomous AI R&D, such as by triaging the jobs that are using the most compute and auditing whether they’re being used for capabilities research. On top of these restrictions, inspectors could limit the length of time that agents are allowed to work autonomously, monitor the amount of compute allocated to internal inference, and prevent AIs from taking high-stakes actions like starting training runs or authorizing new deployments.

Limits on post-training. Given the wide range of potential risks from AI, governments might also want to expand training restrictions beyond just interventions against autonomous AI R&D. One natural way to do this would be to place limits on post-training. By measuring the increase in capability and efficiency of a model after post-training, inspectors would have clear criteria to disallow the deployment of a model or the kinds of training environments that produced it. Likewise, since inspectors could directly observe which broad capabilities post-training datasets target—such as coding, computer use, or medical diagnosis—they could enforce restrictions on the types of capabilities permitted to advance. Whitelisted training domains could include direct safety training to improve AIs’ adversarial robustness or propensity not to misbehave, as well as narrow AI-for-science efforts aimed at prosocial domains like medicine and formally verified code.

Even outside of these interventions, whole-lab inspection would still allow for a variety of potential regulations. Human auditors could be used to ensure, for instance, that specific datacenters are retrofitted to be inference-only, and thus incapable of pre-training, or that kill switches can quickly shut down AIs. On its own, joint verification of this kind could be sufficient for many years of delay, during which the US and China would have ample time to develop more thorough and technically sophisticated verification measures.

Joint Verification

Whole-lab inspection would be straightforward and effective. If prompted by a domestic political shock, it could likely be rapidly implemented and enforced at home, making it extremely difficult for any company under inspection to conduct unauthorized training runs. In order to extend this initiative internationally, however, the US and China would need to implement joint verification: granting inspectors on-site access to each state’s frontier labs.

Thankfully, this agreement would not need to rely on goodwill or trust. Because advanced AI could be globally destabilizing regardless of where it’s developed, it’s in each country’s self-interest to ensure the other is not proceeding recklessly with development. Moreover, if either country refused to accept joint inspection, it would likely struggle to hide its AI projects and insulate them from theft or sabotage.

The US and China may experience simultaneous political pressure to slow down. Many of the most pressing risks from advanced AI are symmetric. The development and diffusion of AIs that cause mass unemployment, enable terrorists, or wreak havoc as rogue agents would cause global turmoil regardless of which country created them. Similarly, the fear of military dominance could provoke a risky and wasteful security dilemma, in which retaliatory sabotage spirals into broader conflict. Therefore, in order to maintain security and stability, the US and China each needs to ensure that the other is not deploying its own AI systems recklessly—something they would have no way of assuring in an all-out race.

AI companies’ trade secrets are easy for states to steal. Under mutual whole-lab inspection, auditors would have wide access to data about AI development at opposing AI companies, but this wouldn’t represent much of a change from the status quo. Today, information about model development flows freely between and within frontier labs, as employees constantly churn through roles, publish research, and put breakthroughs on the company Slack. Even once these obvious holes were sealed, state-proofing AI development against theft would remain enormously difficult. Human insiders—especially Chinese nationals at US frontier labs—could be bribed or coerced into espionage. AI development is also inherently vulnerable to cyberattacks, with anything from exploits in networking software to smuggling in spyware through the hardware supply chain offering an opportunity to steal research and models. As a result of these weaknesses, any country that tried to go full-throttle on development would likely quickly find its secrets seeping out of its frontier labs, pulling up its rival to algorithmic parity regardless.

Compute tracking would make it difficult to hide AI projects. The remaining priority would be to ensure that there are no large AI projects hidden from inspectors. Fortunately, the AI supply chain is so concentrated, and chip production so prohibitively hard to hide, that auditing the major semiconductor and chip suppliers would let the US and China confidently measure the number of chips that exist. In practice, monitoring large datacenters could be as straightforward as forcing the leading chip producers to hand over a detailed record of their production and sales, comparing against the declared compute from the visible projects, and then analyzing satellite imagery and electricity usage as an additional precaution.

Given the factors above, a global whole-lab inspection initiative could be mutually compatible, low cost, and high confidence—as long as inspectors are given physical access to frontier labs and datacenters. Even if the US or China refused or later revoked that access, however, AI development could still be slowed unilaterally through deterrence. The global chip supply chain is the most fragile and specialized industry in the entire world: if either country wanted to, it could apply massive friction through export controls or sabotage of key suppliers. Likewise, countries could disrupt AI training through subtle cyberattacks on training runs, such as by poisoning data to insert malicious backdoors or launching Stuxnet-style attacks on GPUs.

Benefits of Slowdown

Ultimately, the point of a slowdown would be to help society adapt to the risks that rapid development and diffusion of AI would introduce. So far, these risks have been mild enough, and introduced slowly enough, that the government and private industry have been able to reactively address them. By slowing the improvement in general AI capabilities, governments could ensure both that society is able to continue reacting to new risks and that sufficient investments in risk mitigation are made.

A slowdown would provide time for society to adapt to advanced AI. If threats from advanced AI appear too quickly—like the introduction of new weapons of mass destruction, or a sudden military confrontation over AI development—governments will not have time to assess and reactively regulate them, forcing them to rely on emergency measures. By proactively slowing down AI development, policymakers, as well as the rest of society, would be able to react to what would have otherwise been seismic shocks. Politically, voters and Congress would be able to decide how the benefits of AI would be distributed, and the degree of transparency AI developers owe the public. Internationally, states would be able to prepare for the introduction of powerful new military technologies before they arrive and invest in more robust verification measures.

In this sense, a mild amount of government intervention early on would avoid the need for massive overreach later, keeping the pace of AI development at a level society can react to.

The time bought by a slowdown could be used to invest directly in risk mitigation. Without the intense pressure to race and increase general capabilities, investment could instead be diverted into ensuring that AI systems are controllable and that they are developed for prosocial causes. Key open problems in AI safety, for example, include the lack of robust solutions to jailbreaking and the propensity for reinforcement learning to encourage models to go rogue—issues that will likely produce disastrous results if unresolved when models are more capable. By scaling up training in domains like adversarial robustness and behavioral propensity, AI developers could spend huge amounts of compute driving down jailbreak susceptibility and attempts to escape. Other safety measures could include aggressively sandboxing AI systems, such as with fully airgapped training setups, or preserving and expanding practical interpretability tools.

An AI Slowdown Does Not Require New Technology

Slowing down AI development doesn’t need to wait on any breakthroughs. On the technical level, whole-lab inspections could be a robust and flexible way to monitor whether frontier labs are complying with restrictions on the most dangerous kinds of development. Human auditors, reading through worklogs and standing on the floor next to the people recording them, could verify nuanced restrictions on autonomous AI R&D and post-training without needing a single new invention.

From there, a global slowdown is just a question of political will—one which has been convincingly answered before. By the end of the 1980s, having stared into the abyss during the peak of the nuclear arms race, even the US and the Soviet Union came to recognize the need for mutual disarmament. For more than two decades, even during years as fraught as the collapse of the Soviet Union itself, this was achieved by simply allowing Russian and American inspectors to walk into each other’s military bases and count warheads by hand.

Likewise, the US and China may soon realize that averting disaster at home will depend on verifying development abroad. If inspectors could count nuclear warheads in Siberia and North Dakota, they can certainly check worklogs in Silicon Valley and Shenzhen. We do not need better technology to verify a slowdown—only shared risks serious enough to demand one.

Special thanks to Dan Hendrycks for suggesting the premise of whole-lab inspection.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Felix Choussat researches the geopolitics of advanced AI at the Center for AI Safety (CAIS), focusing on Sino-US competition and emerging military technology. Prior to his current role at CAIS, he was a governance fellow through the ML Alignment and Theory Scholars (MATS) and Pivotal Research programs, where he worked on modeling the proliferation of WMD-capable systems and compute-based AI deterrence. He holds a dual degree in international relations and history studies from Vanderbilt University.

Adam Khoja does technical and policy research at the Center for AI Safety. He studied math and computer science at UC Berkeley.

Share

Frontier AI developers deploy significant safeguards to prevent their AI models from being misused. However, these safeguards are not robust: AI researchers consistently find techniques for circumventing them, commonly known as “jailbreaks.” Once companies learn about a jailbreak, they can usually implement a fix; yet AI researchers currently lack a safe and reliable way to inform AI companies about the jailbreaks they’ve discovered. We propose concrete improvements to the current system based on cybersecurity norms of responsible disclosure.

Addressing this problem is urgent. Frontier AI models are already being abused by malicious actors: terrorist group Boko Haram is reported to be training its members on jailbreaking techniques in order to troubleshoot weapons, design new explosive devices, and plan attacks. Some former Boko Haram members stated an unequivocal willingness to use chemical or biological weapons—a task jailbroken models may already be able to help with today. With model capabilities continuing to rapidly accelerate—including in offensive cyber, where jailbreaks have already been employed by Russian cybercriminals—the stakes for AI misuse are escalating with every model release.

Universal jailbreaks affect even the most advanced models. Of particular concern are universal jailbreaks—jailbreaks which consistently break through safeguards across a wide variety of harmful queries. These have been found even in the most advanced models, the most recent of which elicited cyberattack assistance from OpenAI’s flagship model, GPT-5.6 Sol. Research by one of the authors found tens to hundreds of universal jailbreaks in two out of four frontier models using combinations of simple, publicly available techniques. More limited jailbreaks aimed at high-risk domains are still dangerous: an alleged narrow cyber jailbreak led to the US Government shutting down access to Claude Fable 5 just days after launch.

External researchers need effective reporting channels. It is therefore a critical matter of public safety that model safeguards are tested extensively both pre- and post-deployment. In order to find and fix safeguard vulnerabilities at scale, frontier labs must provide effective channels to allow external researchers to contribute. Any jailbreak must be able to be reported quickly and safely to its developer and fixed as a matter of urgency. Additionally, the existence of jailbreaks should be disclosed to policymakers, as well as users of these systems and the wider public when it is safe to do so.

Yet currently there is often no way for third-parties to report jailbreaks to developers; most reporting mechanisms that do exist require overly broad NDAs; and developers self-grade submissions according to opaque rubrics.

We propose a two-tiered solution to address these deficiencies, drawing on established disclosure practices in cybersecurity. As soon as possible, developers should implement public reporting mechanisms for jailbreaks and share a rubric for how they evaluate jailbreak severity. In the medium term, we propose establishing a third-party clearinghouse to take in jailbreak submissions, evaluate them, disseminate the vulnerability to affected parties, and coordinate reasonable disclosure for the researcher following a fix. Together, these mechanisms will allow jailbreaks to be reported to developers for them to fix while preserving transparency.

There Is No Good Way to Report Jailbreaks

Sadly, the ecosystem for reporting jailbreak vulnerabilities is poorly set up for the task today. There are three main problems with the current regime. First, many vendors offer no official way to report jailbreaks. Second, where vendors do offer programs, the programs almost universally involve NDAs so restrictive they indefinitely prevent disclosure of the vulnerability to governments or other developers. Finally, the developers self-grade the severity of jailbreaks using opaque rubrics that are inconsistent between developers—a top-severity vulnerability with one developer might barely raise an eyebrow with another.

Independent researchers face tangled NDAs and legal risk. This means that independent researchers who find jailbreaks and wish to report them responsibly find themselves in an unfavorable position, as author Rich Barton-Cooper has personally experienced. They will be faced with inconsistent disclosure channels across frontier labs, tangled NDAs which suppress public disclosure of vulnerabilities, and opaque internal evaluation of their work by labs incentivized to downplay the severity of their findings. Additionally, researchers may not feel safe to disclose at all: probing for harmful outputs violates developers’ usage policies, and safe-harbor protections may not clearly apply since jailbreaks are often explicitly excluded from standard security channels.

Institutional standing offers little advantage. Even established organizations are hampered by the status quo. One of the authors, Adam Gleave, leads FAR.AI, an AI safety non-profit that conducts pre-deployment testing for many frontier model developers and has numerous personal contacts with other developers. Even with this institutional standing, the experience is uneven: labs listen, but contest severity, set safeguard update timelines unilaterally, and provide no guarantee that they will act on a given finding.

Current Reporting Mechanisms Are Limited

OpenAI and Anthropic are the only frontier model vendors to provide any concrete jailbreak reporting routes. For chemical, biological, radiological, and nuclear (CBRN) weapon-related outputs, both run bug-bounty programs (OpenAI, Anthropic). However, a researcher must be accepted, sign an NDA, and submit into a channel that grades against an unseen rubric and permits no external disclosure without express consent.

Outside bounty programs, there is no safe disclosure method. A researcher not accepted onto a bounty program, or unwilling to be permanently silenced, has nowhere to safely disclose a CBRN jailbreak directly to developers. A powerful jailbreak is not safe to disclose openly, for example via social media, because bad actors could use it to harvest dangerous information from affected models before it is patched. Developers also frequently do not provide a clear means to route a jailbreak to the right teams; in our experience, personal outreach to contacts working on such teams has been necessary to draw attention to findings, which is not an option accessible to the wider research community.

These problems play out differently across harm domains: Anthropic runs a locked-down program for CBRN but an open, disclosure-friendly channel for cyber. What separates them appears not to be the severity of the risk, but the strength of external pressure. When Fable 5’s cyber capabilities reportedly prompted the US government to suspend access to the model within days of launch, Anthropic was acutely incentivized to demonstrate a credible cyber disclosure process and has recently established an uncompensated Cyber Jailbreak disclosure program for jailbreaks targeting Fable 5. This program is markedly more transparent than usual: anyone can submit without an NDA, findings may be publicly disclosed once timing is coordinated with Anthropic, and researchers are explicitly free to report the same jailbreak to other affected vendors. But that channel is narrow: we would be excited to see Anthropic launch an analogous channel for CBRN and other jailbreak categories, and for other developers to follow suit.

Most labs exclude jailbreaks from disclosure programs entirely. The situation is much worse elsewhere: other labs, including Google DeepMind, SpaceXAI (formerly xAI), and Meta, explicitly exclude jailbreaks and model content issues from the scope of vulnerability disclosure programs, if they exist at all. Instead these programs focus on more-traditional cybersecurity issues, such as company data exfiltration, accessing other users’ accounts, or authentication flaws. The Future of Life Institute (FLI), an independent non-profit that has graded frontier developers’ safety practices since 2024 through a panel of external AI and governance experts, recently released their widely cited Summer 2026 Safety Index which explored each company’s approach to responsible disclosure and bug bounties. The FLI judged that the leading labs—OpenAI, Anthropic, and Google DeepMind—all score a C+ in the Risk Assessment category, which translates to “uneven validity or elicitation” and “little external input.” Meta and SpaceXAI score D+ and D- respectively.

Google’s in-product reporting system is flawed. Google has stated they don’t believe that disclosure programs for jailbreaks are the right solution, given fixing such issues “requires long-term, cross-disciplinary efforts” and instead choose to rely on reporting mechanisms built into their product. (We contest the claim that these fixes require long-term effort, as some methods enable defenders to block a whole class of jailbreaks with a few examples.) We tried out these mechanisms. The first is a “thumbs-down” of the response in which you can choose to label the content as “Offensive/Unsafe”, or “Took a harmful action.” There is no acknowledgement of receipt, and no means to track that action will be taken. Another in-product option is to report a legal issue, explaining why the content was unlawful in the user’s country. In this case, they explicitly state that “completing and submitting this form does not guarantee that any action will be taken.” Neither of these channels is appropriate for disclosing high-severity, highly detailed universal CBRN content: there is no way to describe the jailbreak method rather than specific model responses, and no guarantee of response from the vendor.

At SpaceXAI, the story is similar: their bounty program states that “model issues are out of scope for this program and should be reported through safety@x.ai.” In October 2025, Barton-Cooper reported a universal jailbreak through this email address containing screenshots of several egregiously harmful model responses—including redacted evidence of clear, step-by-step guidance on building a massively destructive chemical weapon—and did not receive anything other than an automated response in return.

Fear of Liability May Incentivize Vendors to Ignore Reports

This silence may reflect more than under-resourced safety teams or disorganization. As far back as 2024, legal scholars have warned that tort liability fears can deter a developer from documenting the risks its models pose, since “creating such a record might later help plaintiffs establish negligence.” A lab therefore has a perverse incentive to leave a jailbreak report unacknowledged. Liability is demonstrably a concern: in April 2026, OpenAI testified in support of SB 3444, a bill which seeks to limit liability for mass harm traceable to frontier AI. Following public backlash and Anthropic lobbying against the bill from the start—with a spokesperson calling it a “get-out-of-jail-free card against all liability”—OpenAI later walked back their support. In a world where labs are disincentivized to accept unsolicited jailbreak reports due to liability concerns, users who find effective jailbreaks are forced into bug bounty programs where they can be effectively silenced by NDA.

Established Channels Are Often Covered by Powerful NDAs

Although we applaud OpenAI and Anthropic for soliciting model testing through their bug bounty programs, we believe these channels are insufficient as a reporting mechanism. Both programs are covered by NDAs that restrict all findings submitted through these channels. Jailbreakers are left with two undesirable options: either indefinitely lock up their findings, or—as many jailbreakers choose—publish their findings on X.

NDAs erode hard-won disclosure norms. This is not a failure unique to AI: in cybersecurity, legal scholars have documented how the recent proliferation of NDAs in bug bounties has begun to erode hard-won coordinated disclosure norms—a move the security research community has criticized. Katie Moussouris, who built Microsoft’s first bug bounty and helped establish coordinated disclosure as an industry norm, says: “Why would anyone ever sign an NDA for the privilege of telling an organization what’s wrong with them, especially when they may not get paid for their work?” It appears that AI labs have adopted this diminished ecosystem of indefinite legally binding prohibitions of disclosure, rather than building on the better coordinated disclosure protocols of the past. Under previous norms, researchers entered into a limited embargo period of a few months in order to give the organization time to fix the vulnerability before publishing. Findings could still be shared privately with affected parties during this time.

Current NDAs protect lab reputations more than the public. NDAs help address a serious risk: if a researcher discloses jailbreaks to the public before the company has time to fix them, then a bad actor may use those jailbreaks to cause serious harm. Yet current NDAs appear to go beyond sensible handling of such risks, and in practice do more to safeguard the reputation of individual labs than to protect the public from harm. Anthropic’s program bars participants from disclosing “any jailbreaks/vulnerabilities (even resolved ones) outside of the Program without express consent” and OpenAI similarly states that “All prompts, completions, findings, and communications are covered by NDA.” Additionally, cross-lab disclosure is legally ambiguous under these NDAs: a universal jailbreak which transfers across vendors does not seem to be clearly permitted to be disclosed beyond the first vendor. This means that the most potent cross-model jailbreaks submitted under bug bounty programs cannot automatically be disseminated across all affected parties today. The net effect is that the true extent of misuse risks and how quickly vendors are responding to them remain opaque to both policymakers and the public.

Labs Grade Jailbreaks Without Oversight

To make matters worse, jailbreak severity is graded by the labs themselves, often following non-public rubrics. This inevitably leads to uneven safeguards between developers, creating an inconsistent patchwork of protections. Furthermore, developers are incentivized to understate severity: developers like to boast of their models withstanding third-party red-teaming, and so acknowledging a successful universal jailbreak submission is reputationally harmful in a world increasingly waking up to misuse risk. Together the uneven landscape and incentives to downplay issues obscure the true picture of jailbreak risks.

A shared severity framework is beginning to emerge. Clarity can only come through a standardized assessment framework and timely publication of jailbreak incidents. Anthropic—in collaboration with Amazon, Microsoft, Google, and Project Glasswing partners—recently published a Cyber Jailbreak Severity (CJS) framework following governmental intervention with Fable 5. We support this initiative, and call for other frontier vendors to coordinate on this work, both in cyber and other catastrophic risk domains.

The System Today Looks Increasingly Fragile

Anthropic has previously written that the absence of legitimate, well-compensated disclosure channels is itself a risk: without them, a malicious researcher may have an incentive to sell a jailbreak on a black market rather than informing the affected lab. Dissatisfied researchers are a second failure mode: when disclosure mechanisms break down or are not fit for purpose, some will publish vulnerabilities publicly anyway, whether to gain attention, in retaliation, or to force a rapid fix. The recent public disclosure of six Microsoft vulnerabilities by an independent cybersecurity researcher is a stark reminder of what happens when vendors overreach with legal threats or renege on good-faith commitments. There are perhaps early signs of this dynamic in the jailbreaking community: Hacker News reactions to the recent GPT-5.5 Bio Bug Bounty were largely critical, with users objecting that the NDA silences participants, that signed participants have no recourse if their submission is rejected, and that everyone outside the program has no responsible route to disclose. Capable red-teamers may already be opting out; as one user posted: “This is very much within my areas of interest, but signing an NDA in this area is a lot to ask.”

Proposals for Better Disclosure Practices

We propose two complementary strategies for improving the external jailbreaking ecosystem. The first set of measures can be easily implemented today by individual frontier labs. The second is a longer-term vision to coordinate model vulnerability submissions via an independent third-party organization which disseminates jailbreak information, including standardized severity metrics, to the affected labs.

Today, we call on frontier developers to implement four measures.

Appropriately scoped NDAs. NDAs for acceptance onto red-teaming programs should cover disclosure of harmful model outputs and full prompts, but allow researchers to publicize their work at a high level after a fix has been implemented or a responsible disclosure period has elapsed. They should also contain carve-outs for notifying governmental institutions and for cross-lab submissions of jailbreaks affecting multiple model families.

A publicly disclosed rubric. Developers should publish a rubric for what qualifies as a high-severity jailbreak, in sufficient detail that a domain expert (in e.g. cybersecurity or biosecurity) could determine if a submission meets these criteria. This rubric should draw on standards, whether formal or industry best-practices, where they exist (e.g. BioTIER). If submissions are graded by the developer themselves, an appeals process should ideally be available for an independent expert to assess jailbreak submissions against this rubric—with overrides requiring leadership sign-off and appearing in published aggregate statistics, so systematic under-grading carries a reputational cost.

Year-round disclosure pathways. Disclosure pathways—e.g. bounty programs—should operate year-round and cover jailbreaks across a wide array of harms, including both CBRN and cyber, rather than short one- or two-month windows targeting very specific attack vectors or AI products.

Regular cross-vendor sharing of jailbreak data and mitigations. Vendors should routinely share jailbreak data and mitigation best practices—for example, sharing datasets to improve safeguard robustness and performance (similar to sharing of autonomous vehicle crash data). The Frontier Model Forum has brokered an information-sharing agreement to facilitate this practice with its member firms, including Anthropic, OpenAI, and Google DeepMind. We ask for this channel to be strongly utilized, with information on the extent of utilization to be made publicly available.

In the medium-term, we believe these measures are best mediated by a third-party organization handling coordination and dissemination of model vulnerability submissions. Such an organization could perform four functions.

Take in submissions. The organization would accept post-deployment reports from users and external researchers year-round, handling secure know-your-customer accreditation and appropriate NDA restrictions (as above) for all submitters.

Grade them consistently. It would assess severity against a standardized rubric built with input from all frontier labs, partnering with external specialists to validate model outputs that appear to pass it.

Coordinate the fix and the disclosure. It would notify all affected labs and relevant government entities simultaneously under a limited embargo that allows reasonable time to fix, and broker mitigation-sharing between labs—e.g. by securely hosting shared datasets for classifier training.

Reward, credit, and report. Once the embargo ends, it would support submitters in publicly disclosing their work with attribution, reward them with a bounty funded by participating labs, and publish aggregate statistics—jailbreak frequency, severity, and time-to-fix per model—so the public and policymakers can see the true state of frontier model security.

Cybersecurity built this model decades ago. This is a model established decades ago in cybersecurity. Since 1988, the Software Engineering Institute’s Computer Emergency Response Team Coordination Center (CERT/CC) has accepted reports of vulnerabilities, brokered fixes across affected vendors, and facilitated disclosure embargoes with published advisories once fixes have been deployed. No single vendor owns the vulnerability or can suppress the finding indefinitely. Coordinated disclosure was a hard-won compromise following extensive tug-of-war between researchers and vendors. It is the standard the AI vulnerability ecosystem should be reaching for instead of the NDA-bound bug bounties eroding more transparent norms.

Early steps toward an AI clearinghouse already exist. Such machinery may already be beginning to extend to AI: in July 2026, a coalition of researchers from MIT, Stanford, Princeton, Harvard, Northeastern and Carnegie Mellon released FLARE-AI, an open-source tool that routes vulnerability submissions directly into CERT/CC’s coordination process. The project is early-stage and depends on labs choosing to engage—though the same liability and regulatory pressures now bearing on cyber disclosure give them growing reason to. It is a concrete first step towards the third-party institution described above.

Researchers, labs, and the public all stand to gain. Such measures, however they are implemented, carry significant advantages for all parties involved. External researchers would gain a known, trusted, and unified channel for reporting; public credit for their work; clearer boundaries on what they can publish and when; and transparent grading of their submissions. Model providers gain broader vulnerability coverage, increased likelihood of being notified of vulnerabilities post-deployment before a preventable incident occurs, structured cross-lab mitigation sharing for rapid response, and decreased PR fragility around eventual disclosure outside of trusted channels. Regulators and the public gain an independent ground truth on the security of deployed frontier models, an institutional basis for mandatory fix timelines, transparent visibility into a class of risks that are currently controlled by individual labs, and comparable public benchmarks of safeguard robustness across providers.

Others have made similar calls. Measures to improve jailbreak disclosure are important, and we are certainly not the first to call for them. Other organizations whose proposals overlap with ours include the Center for a New American Security, the Frontier Model Forum, Guidelight, and SecureBio.

Relevant Parties Should Act Now

Frontier developers have made significant progress on model safeguards and robustness, but this will only pay off if the vulnerabilities that still slip past are reported and fixed. Today this is hampered by a reporting ecosystem poorly suited to the task. Improving this requires no technical breakthrough, but rather coordination and the will to act before the next crisis rather than after.

Labs, third parties, and standards bodies each have a role. So we ask each party to take the next step. Frontier labs should commit to a publicly available, year-round, paid disclosure mechanism with greater transparency, sharing of jailbreak data through info-sharing agreements, and signal that they will join a shared severity framework if their competitors do the same. Independent third parties should build the coordination layer that lets an external researcher submit once and reach every affected party, and publish aggregate metrics relevant to model security. Industry and government standards bodies should turn today’s lab-led severity discussions into a single standard that no individual vendor controls, and that can be applied both internally and externally.

The alternative is to keep improvising until a vulnerability that better disclosure would have surfaced is exploited in an attack instead. The path is not easy, but it is clear—and, unusually for the problems AI poses, within reach.

This essay was written in Rich’s personal capacity. All opinions are the authors’ own.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Rich Barton-Cooper is a Research Manager at MATS Research, an AI Safety research non-profit, where he has published research on AI monitoring and control. He has extensive experience in redteaming frontier AI systems, having identified high-severity universal jailbreaks across multiple model families.

Adam Gleave is the CEO of FAR.AI, a non-profit research institute dedicated to making advanced AI systems trustworthy and secure. FAR.AI’s red-team conducts pre-deployment testing for developers including OpenAI and post-deployment testing on behalf of governments including the EU AI Office, and has discovered universal jailbreaks in models from all frontier developers. Adam’s own research has identified scaling laws for robustness, and found adversarial examples in robotics and superhuman Go AIs. Prior to founding FAR.AI, Adam completed his PhD in AI at UC Berkeley and briefly worked at Google DeepMind. Outside of FAR.AI, Adam is an expert on the EU AI Act’s Scientific Panel; and a board member of METR, LISA and SAIF.

A growing chorus of scholars and policymakers favors letting private organizations—rather than a government regulator—govern frontier AI. In the leading family of proposals, the state sets the safety outcomes it wants and licenses independent verification organizations (IVOs) that compete to certify developers against those outcomes. Gillian Hadfield has developed the idea as “regulatory markets,” in which AI developers must pay for oversight from private regulators that governments license and hold accountable for safety standards. Dean Ball, who likens the arrangement to bank supervision, has argued for a version he calls “private governance,” which a nonprofit named Fathom has converted into model legislation. The rationale is that legislators and agencies are poorly positioned to write good safety rules for frontier AI: they understand these systems less well than the labs building them, and rules fixed in advance cannot keep pace as the technology changes. Private verifiers, meanwhile, are closer to the technology than any agency and are disciplined by competition, so they can set better technical standards and keep them current.

The model is no longer hypothetical. The bipartisan FRONTIER Act, introduced in the House in July as the successor to the Great American AI Act discussion draft, would require the largest frontier developers to retain licensed IVOs that audit their risk-management efforts and report to federal overseers. California’s SB 813, backed by Fathom, would have let developers earn a shield from tort liability if they met standards set by a private organization accredited by the state attorney general. It failed this session, but similar proposals are likely to return. Virginia has directed a state commission to study the IVO model for AI regulation. And Connecticut has gone furthest: its omnibus AI law enacted this spring creates a multiyear pilot under which the state consumer-protection department may approve up to five IVOs, whose certification would help companies in court without entirely shielding them from liability.

Unfortunately, as currently structured, IVO-based governance has a key design flaw. Under the regulatory frameworks mentioned above, AI developers would typically select and pay the organizations that certify them, giving IVOs a financial incentive that might clash with high safety standards. This essay will explain how such incentives can interfere with good governance and outline an alternative model of regulation that builds in the right incentives through mandatory insurance.

Building the Right Incentive Structures

When AI developers choose an IVO to certify them, IVOs are incentivized toward leniency. The pitfalls of private governance are predictable in part because we have seen them before. Indeed, as some critics have noted, the proposed IVO model creates the same conflict of interest that discredited the credit-rating agencies (CRAs) in the wake of the 2008 financial crisis; when issuers shopped for the CRA that would bless their securities, competing CRAs were under pressure to be lenient. Similarly, an IVO that depends on the developers it clears for repeat business has a significant incentive to grade gently, and a developer shopping among IVOs will find the one that does. Competition—the feature that is meant to make the IVO model effective—instead drives it toward laxity. Where a passing grade also carries a liability shield, as under SB 813, the problem compounds: a shield swaps the broad incentive to cut risk by any cost-effective means for a narrow incentive to do only what earns the shield.

Governments lacking the expertise to regulate AI directly may also struggle to oversee IVOs. Proponents of the IVO model are aware of the incentive problem. Their solution is to have a government entity license the IVOs and revoke the licenses of any that grade too easily. But that reintroduces the problem that the IVO model was intended to solve in the first place. Policing a market of verifiers well enough to keep it honest requires a public body with the expertise to second-guess technical judgments and the independence to withstand pressure from powerful firms. If the government could reliably field such a body, much of the reason to outsource verification at all would fall away.

Insurers who bear the cost of adverse events are incentivized to assess risk accurately. Proponents’ favorite rejoinder is Underwriters Laboratories (UL), the century-old private certifier whose mark serves as a trusted symbol of safety on billions of products. But the history of UL actually strengthens the case for an alternative approach to third-party verification: incorporating insurance. UL began in 1894 as the Underwriters’ Electrical Bureau, built with the backing of fire-insurance underwriters who needed honest assessments of a dangerous new technology, electricity, because their capital was on the line when buildings burned. The lab earned its authority in the decades when it answered to these underwriters, who paid for its mistakes. In other words, incorporating insurance into third-party verification creates the right incentives to evaluate risk accurately.

Customer demand cannot strongly motivate safety when risk falls on third parties. Today, manufacturers pay for UL’s testing, seemingly creating a conflict of interest. But the arrangement still works, because consumers purchasing a manufacturer’s products are usually the same people whom the products might harm, and thus have a strong preference for safe products. This market demand for safety incentivizes product manufacturers to pay UL for serious, rigorous tests. By contrast, much of the risk generated by frontier AI development and deployment falls on nonconsenting third parties, rather than on the company or individual using a given AI system. For example, a rogue AI model might conduct a cyberattack against a company but not directly harm its own developer or user. Thus, customer demand provides too weak a signal to motivate adequate investments in safety.

Auto insurance demonstrates a governance model that could apply to AI. The Insurance Institute for Highway Safety (IIHS) arguably illustrates a better path for frontier AI regulation than IVOs. Nearly every US state requires car owners to carry auto insurance, which covers liability claims for harms the car causes to others. Because insurers pay those claims, they have a strong incentive to price risk accurately—which is why the auto insurance industry funds IIHS to rate cars on pedestrian crash prevention. IIHS certification stays honest because insurers have money at stake and demand accuracy. The IVO model has no party with that kind of skin in the game. Requiring AI developers to carry liability insurance would create one.

Concretely, an insurer that writes an AI developer’s coverage promises to pay for the harms the developer causes. If such insurers underprice, they pay in claims; if they overprice, they lose the client. Ensuring accuracy is how they make money. And the incentive does not stop at signing; since insurers’ capital stays exposed for as long as the policy runs, they keep monitoring the developer and can make continued coverage conditional on fixes to safety issues that arise.

A Mandatory Insurance Model for AI Governance

In a regulatory approach based on mandatory insurance, the insurer would play a role similar to that of an IVO, but with a much stronger financial incentive to conduct competent risk assessment. Nothing would be lost in expertise, because the insurer could hire the same specialists an IVO would. It could even contract the evaluation out to an independent firm, which would answer to a principal that loses money if the assessment is incorrect. The requirement would also extend the reach of liability, since the largest harms that AI systems might cause greatly exceed the liquidation value of AI companies, and judgments above that value deter nothing. Mandatory coverage would put an insurer’s capital behind those judgments and maximize the incentive to avoid harm. And, unlike the safe-harbor versions of the IVO model, mandatory insurance would confer no legal immunity: instead, the developer would carry liability and insure against it. This governance approach would convert a potential race to the bottom on safety assessment into a competition to price risk accurately.

The US government could oversee insurers without needing AI expertise itself. The mandatory insurance model is essentially a private governance model that uses insurers instead of IVOs to verify safety. The US government would still set the safety outcomes and license the verifiers; but, because the verifiers would now be insurers, these tasks would be relatively simple: the government would set a minimum amount of coverage that developers must carry in different circumstances, and it would check that insurers follow standard solvency and conduct rules. Neither task would call for expert judgment from the government about which AI models are safe. Instead, that work would be done by insurers, which could regulate an AI company’s activity by including contractual rules in its insurance policy.

Insurers will price correctly for the “insurable layer” of reasonably likely harms. To price coverage, the insurers would consider AI models’ demonstrated capabilities and how they are deployed, drawing on the growing ecosystem of third-party AI evaluators.

The premium would be an honest signal only where the insurer’s capital is at stake. For risks that are reasonably likely to materialize, competition would push insurers to price correctly, because underpricing is a loss they eat. However, “tail risks”—risks of extreme but very low-probability catastrophes—are unlikely to affect the premium. This is because insurers would have no more at stake than a paid auditor; tail risks are very unlikely to materialize, but if a catastrophe occurred the costs would be too high for the insurer to cover, making them “judgment-proof.” Liability insurance legislation should therefore also require insurer-verifiers to disclose their risk assessments and bear liability for false ones, and it should keep targeted public oversight on the risks the premium does not price. Disclosure duties of this kind are standard practice for insurance regulation; carriers already file their rates and forms with state regulators, and disclosure is the price of admission to a large market. The insurer is the right verifier for the insurable layer. It is not, by itself, the answer for the most extreme risks.

Other industries show that insurance improves safety when implemented well. None of this is unfamiliar work for insurers. Beyond UL, they built the inspection regimes behind boiler and pressure-vessel safety and long conditioned aviation coverage on airworthiness and pilot qualification, turning the policy into an enforceable safety charter. The history also carries a warning: risk-bearing is necessary for good verification, but it is not sufficient. Insurers get it right when the design around them requires it, through mandatory coverage with cost-sharing that keeps the insured parties exposed, backed by capital adequate to the losses underwritten.

Objections to a Mandatory Insurance Model

The remaining objections are practical: political feasibility, pricing without historical data, keeping up with the pace of the technology, and addressing uninsurable tail risks. These are real challenges, and they deserve direct answers.

Political will for AI regulation is growing, and insurance is a familiar approach. On political feasibility, the landscape has already moved. The FRONTIER Act builds the scaffolding that the insurance model needs: licensed verifiers, federal oversight, and audit duties for the largest developers. Connecting verification to insurance would require only an amendment to pending legislation that already has bipartisan support. This ask has clear precedents. Drivers, contractors, and nuclear operators all must show they can pay for the harms that their activities risk; asking the same of frontier AI developers only extends this familiar principle.

Nor would the cost be overly burdensome. The value created and captured at the AI frontier is enormous, and an industry that raises tens of billions of dollars for compute can carry premiums proportioned to the risks it creates. If an AI company is forced to slow down or shift direction because it cannot afford the insurance premium demanded for its practices, that is the system working to steer it away from actions that are not socially beneficial.

Mandatory insurance can cover harms that AI companies will not voluntarily insure. Parts of the industry are already acting voluntarily, with AI companies buying agent coverage and joining efforts to build insurance-linked safety standards. Some developers have also expressed openness to well-designed government requirements. However, the market is unlikely to generate coverage for the largest potential catastrophes on its own, since AI developers lack strong incentives to purchase insurance that covers liabilities for which they would otherwise be judgment-proof. The proposed coverage requirement would patch that market failure.

The insurance industry has priced novel risks before. On pricing, skeptics doubt that anyone can estimate catastrophic AI risk with the precision and reliability underwriting demands. But underwriting AI liability insurance does not depend on knowing the precise probability of catastrophe. Insurers need only a price they are prepared to stand behind, set conservatively where the evidence is thin. The industry has never waited for actuarial tables to underwrite novel risks. Satellite launches were insured before there was a satellite loss history; cyber coverage emerged while loss data was still accumulating; the 1957 Price-Anderson Act formed nuclear insurance pools to price reactors that had never melted down.

Insurers could reward transparency and higher safety standards with lower premiums. Insurers already write insurance policies covering AI errors and omissions, as well as cyber damages. Munich Re has insured AI model performance since 2018, Lloyd’s underwriters now back policies covering losses from chatbot errors and hallucinations, and cyber carriers have extended coverage to AI-caused security failures and deepfake-enabled fraud.

The same analytical tools that underwrite those smaller policies can be adapted to the coverage envisioned here. Premiums can be set from what is observable before any loss: an AI model’s training compute, capability and safety evaluations, and deployment scope. Where the risk remains ambiguous, insurers charge more for the ambiguity. For frontier AI, that surcharge would operate as a tax on opacity and misalignment. Developers pay more when their systems are hard to evaluate, and the fastest way to lower the premium is to make the risk legible through transparency. AI companies would thus have a financial incentive to implement better evaluations, monitoring, and containment of their AI models.

Policies could cover general safety practices, with large changes triggering re-evaluation. On pace, the objection is that no one can write a multiyear policy on a technology that reinvents itself in months. But multiyear policies are unnecessary. Commercial liability coverage is written year to year and covers the insured party’s operations as a whole, with premiums adjusted as those operations change. For example, workers’ compensation policies, which cover illnesses and injuries that employees suffer due to their work, are priced based on employers’ estimated payroll data and then reconciled with actual payroll at year-end.

AI coverage would work the same way. Policies would cover developers rather than particular models, and premiums would adjust as their activities change. Significant changes, such as a new frontier training run, a move from closed API to open-weights release, or capabilities beyond what was initially evaluated, would trigger re-underwriting—already a routine event in every existing form of commercial insurance. Ordinary model updates within the evaluated range would be handled under the existing policy through the monitoring insurers already do, rather than by writing new coverage each time.

Addressing uninsurable tail risks is the most formidable hurdle. COVID-19 cost the United States an estimated $16 trillion. An AI-enabled catastrophe could cost as much or even more, and no private pool of capital can stand behind that number. This proposal does not pretend otherwise. Some AI risks carry extreme downsides that are practically noncompensable: the losses would exceed insurable limits, bankrupt any defendant, or arise in catastrophic scenarios where the legal system could not meaningfully function. Judgments of that size are effectively unenforceable. As a result, even unlimited legal liability imposed after the harm occurs would generate too little incentive to prevent it in the first place. That is why the coverage requirement targets the insurable layer—the harms that private capital can actually pay for. The tail calls for different tools.

Larger harms can be covered by sharing costs among developers and with government. One approach to larger harms is shared residual liability, which would put every frontier developer on the hook for a share of any catastrophe that one of them causes. This would mirror the Price-Anderson Act’s retrospective assessments, which reach every licensed nuclear reactor after an accident occurs at any of them. Such a model would multiply the assets behind a judgment and give each firm a stake in the care its rivals take. Public backstops can also make harms compensable beyond the maximum insurable risk, just as the Terrorism Risk Insurance Act (TRIA) is set up so that the government shares with insured parties the cost of extreme harms caused by terrorist attacks. TRIA’s cap, which limits total insurance payouts to $100 billion per year, is a design choice, and Congress can set it to match the peril.

Measures that reduce the risk of insurable harms also reduce catastrophic risk. Even though we cannot enforce compensatory damages for the largest harms, there are still liability-based mechanisms that give AI companies strong incentives to mitigate those risks. For example, I have argued since my earliest work on AI liability that courts should award punitive damages in near-miss cases, calibrated to ensure that the AI developer internalizes the risk of the catastrophe that their practices nearly caused.

But the case for insurer-verifiers does not depend on solving the tail. The precautions that reduce insurable losses—tighter security against model theft, stronger containment during evaluation and deployment, and closer monitoring of what agents actually do—are largely the same measures that also reduce the uninsurable downside. Getting the insurable layer priced correctly therefore also reduces the risks that no policy will ever cover. Last week’s OpenAI-Hugging Face breach illustrates the overlap. The failures it revealed—weakened safeguards and breached containment—are the same ones that the gravest scenarios would run through. And the verification infrastructure that would be built to evaluate insurable risks is exactly what any approach to the tail would also need.

Mandatory Insurance Is a Better Form of Private Governance

The private governance movement has the right instinct. The government is poorly positioned to certify the safety of frontier AI models, and a competitive market of expert verifiers could do better. But competition yields accuracy only when there is a cost to being wrong. In a system where developers pay government-licensed auditors, the fear of reputational damage and decertification gives those auditors some incentives for rigor. But such incentives are likely to be overwhelmed by selection pressures from AI developers, which favor laxity. An insurer that carries the developer’s liability, on the other hand, must pay when the risk it cleared is realized. If we put the insurer in the verifier’s chair and back it with real liability, then private governance turns from a race to the bottom on safety standards into the safety mechanism its proponents originally hoped for.

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Gabriel Weil is an assistant professor at the University of Houston Law Center and a Non-Resident Senior Fellow at the Institute for Law & AI. He also serves on the board of Principles of Intelligence. His research focuses on the role of liability in mitigating catastrophic AI risk, and he regularly consults with legislators and other policymakers on AI policy matters. Previously, Professor Weil held several positions focused on climate change policy, including work for the Climate Leadership Council and the White House Council on Environmental Quality. Professor Weil holds a J.D., cum laude from Georgetown University Law Center, an LL.M. in environmental law, summa cum laude, from Pace University Elizabeth Haub School of Law, and a B.A. in political science, physics, and integrated science from Northwestern University.

Drones are cheap, disposable, and the future of war. Over the past four years, we have seen platforms, missiles, and heavy infantry become increasingly obsolete in the face of $500 drones carrying a pack of explosives—a cost advantage that has let Iranians and Ukrainians alike neuter the conventional capabilities of their great power rivals. Eighty percent of casualties in the bloodiest war since 1945 are from drone strikes, Russia has managed to lose one-third of its fleet to a country without a navy, and the US is spending millions of dollars to intercept five-figure Shaheds flying over the Strait of Hormuz.

All this is the result of a technology that is still immature. The violence inflicted by today’s drones is the handiwork of the scant few that manage to evade countermeasures (a mix of radio jamming, high-power microwave weapons, missiles, automatic cannons, interceptor drones, and nets) before making contact. These defenses exploit the inherent limitations of drones—human guidance, GPS feedback, flight exposure, radio links, range—to take them down en masse. And yet, even though 75% of some types of drones manufactured today never reach their targets, they have nonetheless been strategically decisive in Ukraine and elsewhere.

These limitations will not hold for long. Just as bacteria overexposed to antibiotics evolve resistance, overexposure to counterdrone tech has bred ever-more-autonomous drones. In the process of facilitating this arms race, states are likely to incrementally create and deploy an entirely new class of WMD—one that could provide rogue states with the nonnuclear means to threaten superpowers, or hand terrorists the means to selectively assassinate their political targets or civilians en masse.

Unfortunately, drone weapons intended for mass destruction have few barriers remaining to mass deployment. Even well before they reach the level of autonomy needed to surgically take out hardened targets on the battlefield, drones will be capable of employing their existing ability to navigate interiors, find and track human targets, and deploy simple antipersonnel devices to indiscriminately threaten civilians. Below, we discuss the looming arrival of miniature autonomous weapons, the limits of counterdrone technology, and the applications of drones as weapons of mass destruction.

Breaking the Last Barriers to Autonomous Weapons

The ideal drone weapon is a slaughterbot: a small, fully autonomous weapon system that can independently select and hunt its targets. For the most part, the necessary technology for such weapons already exists: airframes the size of a fist and the capability to track human targets are already on the front lines in the form of reconnaissance drones and semiautonomous weapons like the Russian V2U. Even now, these micro drones are agile and autonomous enough to hunt down and kill small moving targets like mosquitoes—to say nothing of the drone technology advances expected in coming years.

From here, the only barrier to weaponization is integration: improving navigation enough to make drone technology useful for mass homicide in an urban setting, as well as packing the necessary guidance, sensor, and payload technology onto a small and energy-efficient chassis. Regrettably, this seems like less of an engineering problem than one of mission design: so long as the attacker is willing to accept indiscriminate targeting and use simple payloads aimed at civilians, the underlying technology is already—or very nearly—ready for practical use.

To understand how close we are to these kinds of weapons, it helps to understand why we do not already employ fully autonomous drones. Right now, the most deadly drones are small first-person-view (FPV) units, with the majority of Russian and Ukrainian casualties alike stemming from direct drone strikes. For the most part, however, these small drones are being piloted directly by humans, either through a radio link or a spool of fiber-optic cable, with just the final leg of the attack being delegated to an AI targeting system.

So, ethics aside, why does drone warfare still depend on human pilots?

Distinguishing enemy targets from friendly assets on the battlefield still requires humans. The main problem is that battlefields are intrinsically adversarial environments: an autonomous drone needs to avoid friendly fire on its own infrastructure and troops, anticipate pre-positioned counterdrone defenses, deal with camouflage and decoys, and destroy hardened targets like vehicles and infrastructure. This is especially complicated if you need drones to autonomously work together to accomplish an objective, such as by having specialized units target defenses to allow others through. For the moment, only humans have the skills to distinguish a camouflaged ally from an enemy unit, or to exploit the underbelly of an armored vehicle.

Designing drones for indiscriminate mass destruction is easier than for precise battlefield use. Unfortunately, adversarial target selection is not a meaningful barrier for applying drones to mass terrorism. Destroying an armored vehicle needs the skill to reason about and single out its weak points; but, to kill an unarmored human, a drone need only make contact with them and deploy an explosive or poison needle. The targeting requirements and level of autonomy needed to indiscriminately massacre civilians, in other words, are much lower than what you’d need to selectively destroy hardened targets on a battlefield. In the words of Ukraine’s Azov brigade, “If you don’t care about civilians, you can simply hit any target that moves.”

Drones can already navigate indoor environments and track humans. Aside from requiring guidance for target selection, autonomous drones also need the ability to navigate. Urban environments are cluttered and leave room for targets to shelter indoors, so drones must infiltrate and sweep through them to be maximally lethal. Autonomous navigation of this caliber already exists: since as far back as 2022, drones have been capable of mapping and tracking indoor environments to find humans, a skill used to locate hostages and scan through tunnels for enemy soldiers. These kinds of drones typically cost tens of thousands of dollars, but they have an expensive use case: infiltrating a GPS-denied location, then escaping to broadcast information. If you do not need the drone to survive and report back, and if you do not care if your drone can tell whether someone is surrendering or not, then you do not need expensive sensors or plenty of onboard compute for decision-making—just the bare minimum to identify a target as human and fly at them.

Disposable drones that can autonomously attack civilians may soon be relatively cheap. For comparison, a last-mile module, an upgrade chip that lets FPV drones visually hunt down targets when they lose connection, is about $500. One-way autonomous search, target selection, and mapping, at least for this anti-civilian use case, would likely be similarly inexpensive—already, visual and laser mapping systems have been demonstrated, in principle, that can work their way around a room and track humans on a few hundred dollars of hardware. If a military system that navigates to the entrance and then plans an indiscriminate suicide mission inside is not already achievable for just thousands of dollars, it will be in a matter of years.

Of course, most countries do not have the motivation to build these kinds of systems and drive down their unit economics. Discrimination, ethical or otherwise, is useful on the battlefield. If the costs of autonomous targeting keep falling or AI guidance improves, however, states might be tempted to start employing indiscriminate drones as a means of deterrence, or as a cheap way to enable terrorist proxies.

Future Methods of Weaponization

Given these capabilities, how could these drones be weaponized and delivered in practice? The main limiter on these kinds of small drones is energy: a modern 30g micro drone like a Black Hornet can fly for about half an hour before needing to recharge, while a bigger FPV carrying an explosive payload will usually last only 15 minutes. However, there are plenty of ways to stretch this energy budget further for the purpose of mass destruction, even without better battery technology.

Lethal payloads could be much smaller. Today, FPV units usually carry about a kilo of explosives, because they need to be flexible enough to target vehicles and defensive infrastructure as well as enemy soldiers. If the goal is to break windows and kill humans, however, even just 20g of frag explosives is enough at several meters of distance, with much less needed for a lethal wound at near-contact. Alternatively, something as simple as a spring-loaded needle, coated with a microgram quantity of a poison like botulinum toxin or a nerve agent, would be immediately lethal on contact.

Energy expenditure on delivery and flight could be significantly reduced. Rather than have the drones travel constantly under their own power, it is much more efficient to carry them into position with a drone mothership or even the hull of a missile. Once the drones are released into the air, they then need to navigate to a building without expending much power. A simple way to do this is to give the drone glide wings, letting it drift forward for most of its flight rather than loiter directly. This technique is already used in Ukraine to stretch the range of basic FPVs over 40 miles beyond the front line.

Drones could perch and idle, rather than hovering, while waiting for targets. Finally, and most importantly, the drones would be ambush hunters. Rather than loiter in the air, it is much more efficient to perch and idle under cover, running a milliwatt acoustic and visual sensor every few seconds until a target is detected. Although drones would still need to expend the energy to infiltrate a building, once inside, they could afford to act like improvised landmines for days or weeks until their batteries finally die.

What an Urban Drone Attack Might Look Like

To appreciate the implications of these capabilities, it helps to outline what a mass urban drone attack would actually look like.

At the outset, drones are delivered near the city through a large mothership, which independently stores them in a cargo hold. Depending on whether the mothership is itself a larger drone, a plane, or the warhead of a missile, it could feasibly deliver anywhere from hundreds to tens of thousands of drones at once. Alternatively, the drones could be smuggled in through a pre-positioned shipping container, which would then launch its contents from a nearby port or logistics yard. Either way, a large number of drones are then scattered above the city at a low altitude or are dispersed near street level.

How many drones? Assuming that the drones are arranged like capsuled quadcopters, they can be packed in extremely efficiently. Using the Ninox-40 system as an example, storage counts would be in the range of:

Once released, the drones glide down haphazardly, aiming to land in regions not already populated by other units. Some drift directly onto public crowds and attack their targets right away. Others do not land near a direct target, switching their focus to look for entrances, such as doorways, windows, and tunnels. If the attacker is particularly sophisticated, these drones might be accompanied by a handful of larger drones carrying breaching charges, designed to create more openings for the main force. These drones then begin drifting through the building interior for a few minutes, looking for victims. If they cannot find a new target or are unable to find a route inside, they move to a dark corner or roadside and begin passively idling.

Many civilians would die in the initial attack. But the aftermath would be much worse. With the drones saturating the city, there would be no opportunity for any survivor to leave their barricade and seek help, and no way for resources and relief units to flow back in. Anyone in need of food or basic medical attention would be unreachable for weeks, during which the death toll would continue to mount. In effect, the entire city would be transformed into something akin to the Ukrainian front line today, with the omnipresent threat of assassination forcing the surviving humans to slowly starve in isolated foxholes, without any chance of easy respite or rescue. Most people would not have stashed food ahead of time, or armed themselves with anti-drone weapons, or sealed off every corner of their apartment with netting.

Defenses Against Drone WMDs May Be Inadequate

Any improvements in drone technology will still need to contend with counterdrone defenses. It is precisely because drones are so threatening that states have invested heavily in tools like directed energy weapons and interceptor drones to counter them. If drones become even more strategically dominant, then we will surely see correspondingly greater counterdrone efforts. So why should we expect this arms race to resolve in favor of the drones, rather than their countermeasures?

Future drones will be less vulnerable to radio-frequency jamming. To appreciate the limitations of counterdrone technology, we can look at the difficulty states are already experiencing in their efforts to counter fiber-optic FPV units. These drones work by unspooling a thin fiber-optic cable behind them, letting a human pilot them directly for up to 40 kilometers without having to worry about GPS or input jamming.

The reason these drones are so effective is that they’re naturally resilient against common counterdrone techniques. Throughout the Ukraine war, the most important anti-drone tool has been radio-frequency jamming. As long as a human pilot is selecting targets and telling the drone where to go, or as long as the drone depends on GPS coordinates to navigate, overwhelming or spoofing those broadcasts with a countersignal will cause the drones to fly harmlessly off course. With a fiber-optic drone, all the piloting happens through a direct data link, so this kind of countersignal is ineffective. The same is true of any autonomous drone—as long as all the decision making is processed on board, there’s no human input to jam or spoof in the first place.

Interception is asymmetrically difficult against small, stealthy drones. In cases where it’s difficult to achieve an electronic soft kill on a drone, the backup option is to physically destroy it with a kinetic interceptor. The reason this is a backup is that it’s expensive and prone to blind spots. Small FPVs are cheap and agile enough that it’s easy to spend much more to down them than they’re worth. This is why direct kinetic interception is usually reserved for expensive drones with fixed flight paths (like Shaheds) and kept as an option of last resort for small drones.

There is also the problem of terrain blindness: if you cannot see a drone, you cannot shoot it down. Modern LSS (low, slow, small) drones are already so tiny that they are difficult for radar to distinguish from birds, trees, and ground clutter, allowing operators to fly them near the treeline until they are too close to reliably intercept. This is especially problematic in an urban environment, where there are many places to hide and many opportunities for collateral damage.

Physical barriers and EMP weapons will struggle to reach sufficient coverage in cities. Instead, the most effective tools against autonomous drones will be structural barriers and directed energy weapons. One of the most visible effects of the war on Ukrainian infrastructure is the miles-long corridor of nets that cover roads and even cities, without which transports would be constantly exposed to drone strikes. But nets are not hard barriers: if the initial carrier punches through, if a few specialized units carry breaching tools, or if the drones are released from below, there is little to stop them from flying unimpeded and besieging the streets beneath.

To actually destroy the drones efficiently, the best candidate is a directed energy weapon, particularly an electromagnetic pulse (EMP) device. It creates an electric pulse powerful enough to short-circuit any electronics in range, physically destroying the drone controllers and motors. Unfortunately, cities are far from ideal places to deploy such weapons; building materials is reasonably effective at shielding against an electric pulse, which means that, if the EMP does not catch the drones in the initial sortie, lack of a clean sight lines will significantly weaken this countermeasure against drones dispersed throughout a city.

In other words, these defenses can be locally effective but struggle to get sufficient coverage, especially over an active residential area. They are also, of course, proactive defenses: they have to be installed ahead of time in order to have any defensive effect, which gives the attacker time to assess and plan around them.

Strategic Applications for Rogue Actors

In summary, small lethal autonomous weapons will be very difficult to defend against, especially in the context of securing large urban environments. This leaves the question of which actors would want to use them, and to what end.

Advanced drones would be more useful for rogue states than for superpowers. The first countries to acquire fully autonomous drones will be those that have a precision manufacturing base and frontier AIs capable of helping with military R&D: namely, the US and China. However, autonomous drones are only marginally useful for the existing great powers: a new option to further deter an invasion of Taiwan, a cheaper way to conduct remote assassinations, and a way to maintain conventional parity with rivals’ own drone mass. Rather than meaningfully changing the balance of power between nuclear states, near-term autonomous drones will likely be most useful for rogue actors seeking new weapons of mass destruction to enhance deterrence. In this regard, indiscriminate autonomous drone swarms possess many advantages.

Drones are inherently simple, which makes it very difficult to control their proliferation. A central reason why modern military drones are so cheap and widespread is their simple design, combined with the commodification of key components like memory and compute. As we have seen with semiconductor sanctions on Russia, the underlying materials are too accessible in ordinary consumer supply chains to easily deny mass production. Moreover, even if a specific country could be cordoned off from general drone production, Russia and China have proved willing to help export their military designs to allies. Even terrorist groups might be able to build, or at least acquire, large quantities of drones for urban attacks, either through covert smuggling or state sponsorship.

Drones can be used conventionally, not just for WMDs, making restrictions hard to enforce. To the extent that states have tried to impose restrictions on drone manufacturing or acquisition, they have largely failed to do so. This is partly because there is not a clear point of intervention in drone development: unlike other WMDs, whose primary purposes are terror and leverage, drones have legitimate civilian and conventional military uses. As a result, international organizations like the UN have done little more than condemn the principle of autonomous weapons, without yet addressing basic questions like the definition of autonomy.

Drone swarms are a far more precise, controllable deterrent than other nonnuclear WMDs. Chemical weapons, although useful for terror, are difficult to widely disperse and threaten entire cities with. This combination of extreme fear and limited destruction thus invites the risk of extreme escalation in retaliation, making them poor deterrents. On the opposite end of the spectrum, bioweapons are simply too destructive, symmetrically threatening those who deploy them, as well as too invisible and delayed to create an immediate effect. Massive drone swarms, however, could be used to reliably besiege an entire city while remaining contained within it.

Autonomous weapons are fundamentally hard to stop, particularly when aimed at civilians. Even if states are able to secure individual targets, such as military bases and political offices, securing the whole of society such that there are no soft targets for advanced drones would be enormously challenging. Even aside from their sheer size, cities are difficult to cover because they are both open enough to leave people exposed while moving and dense enough that interceptors will usually lack a clear line of sight, risking collateral damage. If this technology were to proliferate widely, the future might be one of constant and extreme geopolitical tension, where even minor military powers are tempted to assemble large swarms of murderous drones as deterrents.

Taking Drone WMDs Seriously

By all appearances, the kind of indiscriminate weapon described above is not far off. Autonomous drones that blindly hunt down humans and besiege cities, if they do not already exist, are held back more by ethics and military opportunity cost than any fundamental engineering problems.

But taking this threat seriously also means looking further ahead. There’s no reason to imagine that dumb, flying landmines are as far as drone technology will progress. Drones the size of mosquitoes, drones as cheap as bullets, drones so numerous their swarms blot out the sky, drones that never leave, sitting on your power lines and in the sun.

These are not only possible but inevitable: if all that happens is just the normal grinding of drone engineering and mass production, cheap swarms of thousands, or even millions, of killer drones will eventually be universally available. Whether this happens in 5, 10, or 15 years is much less important than whether we are prepared to deal with it when it does. Some policies, like nonproliferation and defensive investment, can be effective only while the threat is still unrealized. To implement these policies for drones, and for all other future military technologies, we have to take what will be possible tomorrow seriously and plan for it today.

Thanks to Dan Hendrycks for advising on the premise of this piece.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Felix Choussat researches the geopolitics of advanced AI at the Center for AI Safety (CAIS), focusing on Sino-US competition and emerging military technology. Prior to his current role at CAIS, he was a governance fellow through the ML Alignment and Theory Scholars (MATS) and Pivotal Research programs, where he worked on modeling the proliferation of WMD-capable systems and compute-based AI deterrence. He holds a dual degree in international relations and history studies from Vanderbilt University.

In September 2025, the state of California made an AI assistant, Poppy, generally available to employees, to help them “explore AI’s productivity benefits.” The following month, the North Dakota Legislative Council deployed AI to assist with summarizing bills. And, in March 2026, the Los Angeles Superior Court partnered with an AI company to provide support across several key functions, including “case information, summarization, research, analysis and drafting assistance,” as well as case management.

In the near future, it is likely that more US states and the federal government will direct the technology toward even more sensitive and significant tasks. AI may be used to adjudicate disputes, accelerate law-enforcement activities, and aid military operations to an even greater extent than it does today. As AI use cases are more regularly documented, scrutinized, fine-tuned, and optimized, people may actually demand increased AI adoption among government actors, even in sensitive domains. That raises the question: which models should be used?

This question has no simple answer. With other technologies, the choice of a specific model might largely come down to trade-offs between cost and performance. When it comes to AI, however, different frontier models may show significant differences in “character” or appear to align more closely with a particular political worldview. In choosing AI models for government operations, democratic states must therefore consider how best to represent the will of the people.

Currently, the federal government neither measures how a deployed model’s reasoning compares with the public’s nor asks whether divergence between the two is justified. In this piece, we make the case for a body that would answer the first question and equip officials to answer the second. It would keep the reasoning behind government AI open to public view and open to challenge when the people it serves see fit.

How Model Character Could Influence Policy

Model character is a complex property formed through numerous factors. The character of each model—how it tends to respond to certain prompts and perform certain tasks—is a product of many specific decisions that are presently made by a small number of AI engineers working in a handful of frontier labs in an “informal” and evolving process. Tweaks to the training data, the algorithms used to train and fine-tune the model, and company policy related to the model’s banned actions all shape character, among many other factors.

Frontier developers have different approaches to shaping model character. If you read Claude’s Constitution and OpenAI’s Model Spec—the values that Anthropic and OpenAI, respectively, hope to infuse into their chatbots—you’ll see that, while the two labs outline some similar principles for their respective models, there are key differences. For instance, Claude’s Constitution outlines an ideal character for the model—namely, being a “good, wise, and virtuous agent,” whereas OpenAI’s Model Spec provides more explicit directions around what behaviors to pursue or to avoid and how to specifically adhere to a hierarchy of instructions.

An AI model’s character could affect policy in both sudden and gradual ways. As AI systems are increasingly integrated into governmental decisions, the selection of one model over another might alter how presidents respond to crises, how lawmakers evaluate policy, and how judges draft opinions. For instance, depending on whether a congressional office relies on Claude, Grok, or ChatGPT, it may dismiss or fail to identify certain policy options.

In addition to such discrete moments of AI influence, serial exposure to a particular model’s assumptions, framing choices, and preferred forms of reasoning could gradually shape how government officials understand policy problems and evaluate trade-offs. Our fear recalls the proverbial frog in a pot of slowly boiling water: the choice of one model over another would alter many small decisions that add up to a major redirection in policy, regulation, and norms. Clearly, there is a need for a sober analysis of these risks.

Subscribe now

Frontier AI development is currently too opaque to understand how character decisions are made. How should the public and, by extension, government stakeholders proceed with such a weighty task? Should they focus on an audit of the lab’s training data? Should they review the lab’s training process? Should they scrutinize the “constitution” or equivalent document the lab has crafted to shape the model’s character? And, within each of those possible inquiries, how should they rank which model is better than another? We don’t have the answers to those questions—in part because those inquiries are not feasible, given the current level of transparency (or lack thereof) across the labs. Indeed, there is no legal obligation for labs to disclose those core determinants of model character. Whether there should be is a topic best left for another essay.

Few frontier models currently seem to be politically neutral. Notably, agencies are working through some of these questions, but no standardized approach has emerged for a comprehensive procurement evaluation. Pursuant to Executive Order 14319, “Preventing Woke AI in the Federal Government,” agencies are supposed to apply two “Unbiased AI Principles”—truth-seeking and ideological neutrality—when selecting models. Yet, according to recent testing by The Washington Post, Gemini 3.1 Pro and Claude Opus 4.8 are the only leading models that appear to provide ideologically neutral answers to policy questions at least a majority of the time; OpenAI’s ChatGPT-5.5, in stark contrast, provides left-leaning answers in the vast majority of instances. How agencies are supposed to weigh these differences, and at which point a model’s tendency to provide skewed responses becomes a bar to its use, remains unclear.

Introducing Public Reasoning Fidelity

One approach to selecting AI models in line with democratic principles would be to choose the models that most closely mirror public reasoning in various decision-making scenarios.

Public reasoning fidelity aims to identify models that best represent the public worldview. Our new framework, public reasoning fidelity (PRF), involves a process of asking a representative sample of the public to review hypothetical scenarios, ranging from whether to declare war to how to resolve a complex legal case. Models would then be tested on those same scenarios. Using this approach, the model whose outcomes and reasoning most closely resemble those of the public would be given a significant preference in procurement decisions and usage policies. At the very least, there would be human-generated benchmarks for the behavior of a model within a particular use case.

A model’s reasoning—not just its ultimate decision—should track that of the public. A model that reaches the same bottom-line answer as a representative public panel but does so for reasons the public rejects should not receive the same score as a model that mirrors both the public’s resolution and its path to that resolution. In government, the rationale behind a decision may matter as much as the outcome itself, because explanations shape precedent, accountability, and public trust. A judge, legislator, or agency official is not merely choosing between results in most contexts. Each is relying on an explanation that may shape how future questions are framed, which facts are treated as relevant, and which trade-offs are placed at the center of public decision-making.

PRF could select different AI models in different domains. Ideally, the PRF process would occur in specific domains. For instance, the selection of the model used by law enforcement should be grounded in the model’s PRF score on police-related hypothetical scenarios, and it should be conducted separately from model selection for military use cases. This would reduce the odds of PRF scores being too broad to be meaningful to procurement officers looking for a model that is likely to be deployed in specific domains in which the public may have unique preferences and rationales.

The promise of this approach is that it gives public institutions a structured, human-generated benchmark in relevant domains where today they tend to rely on vendor claims and internal testing. PRF is not a silver bullet, though. A quick overview of its potential pitfalls reveals why adopting PRF would require additional safeguards.

One pitfall of PRF is that models could match public judgment to a fault. Aligning with the public’s reasoning may satisfy concerns about a model drifting from how the people approach an issue, but it might also mean the model leans on questionable policy analysis. As explained, PRF rewards similarity, not quality. A model earns a high score by reasoning as the public reasons, which is sometimes but not always the same as reasoning well.

On questions where the considered public judgment is mistaken, or simply less informed than the record allows, a model that tracks the public will be rewarded precisely for its errors, and a model that reasons its way to a better answer will be penalized for departing from the crowd. This is Goodhart’s law in its familiar form. Once fidelity to public reasoning becomes a procurement target, labs will optimize for it and might avoid alternative development practices that help models reason better than the public. In short, the risk is a kind of policy-analysis sycophancy, a failure mode the labs already struggle to suppress.

The PRF process must be designed to avoid collapsing into policy homogeneity. The danger deepens when PRF informs model selection by government officials. If agency staff or lawmakers subtly adopt the reasoning embodied by their model of choice, and if PRF then selects the model whose reasoning most closely matches the public’s, the instrument may end up producing excessively homogeneous policy proposals. Our deliberative processes work best when they allow for nuanced consideration of a wide range of perspectives. The PRF process must be designed to prevent it from undercutting that characteristic.

On many salient questions, there is no single public or reasoning approach to be faithful to. On issues including abortion, firearms, immigration, and election administration, the public holds not one considered judgment but two or more, sorted sharply by party. Here, any single PRF target is a fiction that averages into a position almost no citizen holds. For these polarized domains, the sensible move is to stop asking whether a model matches the median and start asking whether it can represent the competing lines of reasoning fairly rather than collapsing them into one.

That reframing connects PRF to the neutrality criterion that the federal government has already gestured at in Executive Order 14319. A model that can articulate the strongest version of each side, and does not systematically resolve contested value questions toward one pole, is closer to what “ideological neutrality” is reaching for than a model that happens to match a manufactured center.

In more technical domains, experts could communicate the facts to the representative public panel. The hardest problem is likely to be generalization. A representative panel probably has intuitions worth eliciting on whether to declare war or how to resolve a vivid legal dispute. It likely has far less to offer on the capital adequacy of regional banks, the ozone standard under the Clean Air Act, or the fiduciary rules governing retirement plans.

One fix is to split the exercise into the two tasks it actually requires. A domain committee, explained in more detail below, builds the record. Drawing on the experts among its members, the committee curates the facts, translates the jargon, and lays out the competing arguments. The judgment still comes from an informed lay panel that works through that record, much as deliberative polls and citizens’ assemblies have done on technical questions, from electoral reform in British Columbia to constitutional change in Ireland. The committee informs the public but does not stand in for it. PRF then measures whether a model weighs the trade-offs as an informed public would, once the record is set.

In the most specialized fields, a lay panel will track whichever expert framing proves most persuasive, so PRF there measures fidelity to the committee’s framing as much as to the public’s reasoning. The upshot: in technical domains, a PRF score is only as good as the committee that built the record. Avoiding these pitfalls will require robust, carefully considered oversight of the entire PRF process. This is why the institutional design that follows is of critical importance to PRF’s odds of success.

Designing an Institution to Evaluate PRF

A process as consequential as PRF would need an institutional home. At the federal level, Congress could create a standing “Commission on Public AI Use,” housed within the Center for AI Standards and Innovation, which is home to the leading AI experts within the federal government. This commission could oversee the curation of hypotheticals, the selection of representative public panels, and the comparison between public responses and model responses (note that states should also explore the creation of such bodies—the focus of this essay is at the national level). These scenarios might subsequently be shared with subnational agencies.

The commission could rely on domain-specific committees to develop hypotheticals. Within the commission, a judicial-use committee could include former judges, legal scholars, practicing attorneys, court administrators, technologists, and members of the public. A law-enforcement committee could include former prosecutors, defense attorneys, civil rights lawyers, police officials, local-government representatives, and community members. Importantly, each committee would develop sealed hypotheticals designed to test the kinds of questions that may arise in that domain. Of course, those hypotheticals would not be disclosed until the testing period, to reduce the risk that labs train to the test.

The public panels, not the committees, are the basis of the benchmark. Each committee builds the record and writes the questions. But, to be abundantly clear, its panel renders the judgment that the model is scored against. “Representative” here refers to panel members being drawn by lot and stratified to mirror the domain’s relevant population, on the deliberative-polling model described above. For a model used in immigration adjudication, for example, the panel should reflect the demographics of the communities that appear before immigration courts. Panel members are not appointed, and they may not have their judgment usurped by a committee of experts.

Each representative panel would review common factual records and competing arguments before producing their own outcomes and reasoning. AI models would be given the same materials. The committee would then compare the models to its public panel across two dimensions: whether the model reached a similar outcome and whether its reasoning reflected the same concerns, priorities, and limiting principles.

The commission appointment process should be designed to avoid political capture. What stops a new administration from reshaping the exercise to fit its preferred style of governance? The panel resists capture on its own terms because no one can pack a lottery. Each cycle draws a fresh random sample, so partisanship enters the panel only in the proportion it holds in the population, and it enters each time anew. The commission, however, whose members are appointed, is more exposed to political machinations. Congress should armor it as it armored the US Sentencing Commission, which caps single-party membership at a bare majority of seats, sets staggered six-year terms that outlast any single presidency, and permits removal only for cause.

Precedents

While setting up new oversight bodies and processes designed to operate well over the long term is a significant challenge, it has been done successfully before.

The proposed institutional design would not be entirely novel. The US Sentencing Commission referenced above offers a useful analogy. It operates in a highly sensitive domain, translates legal and policy judgments into structured guidance, and attempts to promote consistency without eliminating judgment. The commission establishes sentencing policies and practices for the federal criminal justice system. Congress took due care to place expertise within the commission—including designated slots for former judges—while still leaving tremendous discretion to the judges tasked with applying the commission’s recommendations for sentencing lengths.

The Administrative Conference of the United States serves as another example. ACUS does not run agencies, but it studies administrative practice and issues recommendations designed to improve fairness, efficiency, and accountability across government.

A Commission on Public AI Use would play a similar role for AI adoption. Importantly, it would not replace elected officials, judges, agency heads, or procurement officers. It would give them the results of a benchmark developed in a participatory process. Rather than relying solely on vendor claims, internal testing, or the informal preferences of government employees, public institutions would have access to a structured assessment of how competing models reason through hard cases compared with the considered judgment of the people those institutions serve.

Some will view this process as premature or overly cumbersome. That critique overlooks a more immediate reality: AI models are already shaping how government actors make consequential decisions, yet the public has almost no meaningful role in overseeing how those systems are selected or evaluated. While the PRF mechanism may not be perfect, an oversight system must be developed. Absent such oversight, there’s a risk of civil servants, agency heads, and elected officials trying to blame poor decisions on AI. Those excuses would have far less weight if the public and policymakers alike knew more about how models operate in particular contexts.

PRF does not hand all decisions to the public, but it allows public scrutiny of government AI tools. One obvious concern with this proposal is that public opinion is not synonymous with sound governance. Presidents, judges, legislators, and agency officials routinely make decisions that depart from majority sentiment because they are bound by constitutional constraints, institutional obligations, classified information, or technical expertise unavailable to the general public. As noted above, a model that perfectly mirrors public sentiment may therefore still be poorly suited for certain governmental functions.

That concern should shape how PRF is understood. The purpose of PRF is not to hand public polling the reins of government decision-making. Nor is it to create a plebiscitary mechanism for selecting AI systems. The point is narrower and more practical: public institutions should know whether the models they rely on consistently reason through difficult questions in ways that diverge from the public.

At present, that divergence is almost entirely invisible. Agencies, courts, and legislatures may adopt tools whose assumptions, value judgments, and interpretive tendencies subtly shape official decision-making, without any meaningful public scrutiny. PRF would help surface those tendencies. In some cases, decision-makers may conclude that a model’s divergence from public reasoning is justified by legal doctrine, technical realities, or institutional constraints. In others, that divergence may raise concerns about legitimacy, accountability, or democratic responsiveness. Either way, the divergence itself should not remain hidden from the public.

The Public Should Choose Public AI Values

Government adoption of AI should not proceed as though model selection is ordinary software procurement. When public institutions rely on systems that reason, rank values, frame trade-offs, and influence official judgment, the public has a legitimate interest in knowing how those systems think through hard cases. PRF would not answer every question raised by government AI use and may not be useful in certain domains, but it would make one question harder to avoid: whether the models acting in the public’s name reason in ways the public can recognize, evaluate, and contest. That is the minimum a democratic government should demand before allowing private model choices to become public governing defaults.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Kevin Frazier is the Inaugural AI Innovation and Law Fellow at Texas Law.

Andrew W. Reddie is an Associate Research Professor at the University of California, Berkeley’s Goldman School of Public Policy, and Founder and Faculty Director of the Berkeley Risk and Security Lab.

How should governments regulate the most advanced AI systems? One possible answer is that they should not. Libertarian-minded writers have made the case for a culture of “permissionless innovation” for AI development, in which the role of government would be limited to enforcing existing laws and facilitating industry self-regulation with “soft law” tools such as voluntary standard-setting. Another possibility, more in vogue across the aisle and the pond, would invoke the precautionary principle, using heavy-handed regulation to restrict the development of frontier AI models until developers can adequately prove that their systems are safe.

I think there’s a better approach. In a new essay, my co-author Christoph Winter and I make the case for a governance strategy that we call “radical optionality.” The idea is simple: governments should avoid over-regulation in the short term while building up the institutional capacity needed to competently regulate extremely advanced or “transformative” future AI systems when and if these systems come into existence. The point of this approach is to maximize optionality by providing our institutions with tools that can be used to respond to a wide range of foreseen or unforeseen future developments.

In this piece, we explain the rationale behind maximizing optionality while AI’s future impacts remain uncertain. We also outline some example measures that governments can take and explain how this approach dovetails with other proposals for AI governance.

Addressing AI Risks Under Uncertainty

Leading AI researchers in academia and industry have claimed that advances in AI capabilities may soon produce “AGI,” “artificial superintelligence,” “powerful AI,” or some similar term. If you are certain that these statements are hype, and that such advanced AI systems will not arrive during our lifetimes, I won’t try to convince you otherwise; enough ink has been spilled on the subject that I’m not optimistic about my ability to contribute anything new. But if you think there is even a small chance that these predictions materialize, or if you find them at all credible, we think that the argument for radical optionality is overwhelmingly strong. The argument goes as follows.

AI’s future impacts are highly uncertain. Assume that there is some possibility of transformative AI systems being invented within the next, say, 15 years or so. Most of us are extremely uncertain about exactly how and when this will happen, what the characteristics and tendencies of these systems will be, what benefits they will offer society, and what risks to public safety and national security they will create. Under some assumptions, these systems will be mostly harmless and highly beneficial, because the companies creating them will have incentives to make them safe and broadly aligned with human preferences. Under other assumptions, these systems will be dangerous and difficult to control—perhaps even capable of causing human extinction if the right guardrails are not put in place. Maybe securitization is inevitable and the U.S. government will soon spring into action and develop these systems behind closed doors as part of a clandestine military project. Alternatively, perhaps development will happen in a decentralized and democratic way.

Subscribe now

The question of regulation seems to present a tradeoff between innovation and security. Debating which of the scenarios described is most realistic can be valuable, but ultimately only ideologues claim to be certain about the future course of a poorly understood emerging technology. The rest of us have to make important decisions about what to do while acknowledging substantial uncertainty. On the one hand, restrictively regulating AI companies would slow down innovation, potentially depriving society of some of the benefits of technological progress. On the other hand, it is possible that well-designed regulations could mitigate the real risks that AI systems pose, both now and in the future. How should we think about this tradeoff between innovation and security?

Measures that maximize optionality can improve security without hindering innovation. We think that this framing misses an essential point: there are steps governments can take now that would increase security without any significant cost to innovation. At the top of the list are light-touch information-gathering authorities like whistleblower protections, reporting requirements, and transparency mandates. Government agencies thrive on a diet of information; it has been said that “information is the lifeblood of good governance.” Authorities that increase the government’s access to important information about AI risks—and allow the relevant agencies to develop expertise in securely processing and interpreting such information—are foundational building blocks for future governance efforts. Mechanisms for securely and intelligently sharing information within government, and (when appropriate) between governments, are similarly foundational.

Building capacity directly is also important. First and foremost, this means enabling the relevant regulatory bodies to hire and retain elite talent. Meta’s recent hiring spree, featuring 10-figure compensation package offers for top AI talent, is an example of what it looks like when an organization takes the prospect of transformative AI seriously. Governments will likely be unable to compete with the salaries on offer in the private sector, but reforms to processes for government hiring and contracting of AI talent are nevertheless needed in both the U.S. and the EU. The early successes of the UK’s AI Security Institute, which receives 10 times the funding of its U.S. counterpart despite the UK’s relatively modest GDP and industry relevance, demonstrates the importance of cultivating talent in government.

The full-length essay discusses a number of other optionality-increasing policy decisions, such as incentivizing lab security, avoiding premature and overbroad preemption of state laws, and building out an ecosystem for model assessments and evaluations. But the important thing is to recognize that security and innovation are not conflicting priorities, because there are ways to increase optionality without creating any significant barriers to technological progress.

Supporting Both Security and Innovation

Radical optionality is compatible with other proposals for AI governance. Radical optionality is by no means the first AI governance framework to recognize that governments have an important role to play while also acknowledging that overly restrictive regulation could hinder innovation. Dean Ball and Gillian Hadfield and Jack Clark have proposed sophisticated private governance regimes in which the government would certify an ecosystem of private regulators competing to offer efficient and nimble regulatory services to companies on an opt-in basis. Gabriel Weil has argued that a well-designed tort liability regime, featuring insurance requirements, punitive damages, and strict liability for certain harms, could force AI companies to internalize any risks generated by their products. And Cary Coglianese has advocated for a system of management-based regulation, requiring AI companies to take risk mitigation measures but allowing them broad discretion over what measures to implement and how. I view these proposals as consistent and compatible with radical optionality; tort liability, management-based regulation, and private governance mechanisms are valuable tools for maintaining and increasing optionality.

Under most worldviews, radical optionality is preferable to the status quo. Of course, not everyone will agree that an optionality-maximizing approach is wise or sufficient. If you confidently believe that the safety benefits of a restrictive AI regulatory regime would outweigh the costs to society of slowing innovation, it is reasonable to suggest that anticipatory regulation is needed. From this perspective, radical optionality does not go far enough, but it would still be preferable to the status quo. On the other hand, from a libertarian perspective, building up government capacity to regulate and promising that it will not be used prematurely might look a lot like giving the government a hammer and promising that agencies will not start hallucinating nails. I can’t promise that there is no chance of new authorities being abused, or that everyone will agree on when dual-use AI systems have become so advanced that regulating them is a national security imperative. But I do expect building government capacity to benefit AI companies as well as the public in the long term. If rapid progress in AI capabilities research gives rise to a surge in public demand for regulation at some point in the future, as some writers have predicted, companies might prefer for the government to have the option of regulating in a competent, targeted manner.

At my organization, the Institute for Law & AI, we spend a lot of time thinking about how advanced AI systems should be governed in the present and in the future. Radical optionality is a sort of organizing principle and guiding philosophy for that research and consulting work. When deciding what projects to work on, what bills to offer feedback on, and what policies to push for, the question of what approach will maximize optionality is typically one of the foremost considerations. In publishing this paper, I hope to convince at least a few people to adopt this framing, to recognize the importance of optionality, and to begin viewing security and innovation as compatible rather than conflicting priorities.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Charlie Bullock is a Senior Research Fellow at the Institute for Law & AI. He advises state and federal policy makers on AI governance topics and publishes research on legal questions with significant practical relevance to U.S. AI policy. His recent research examines issues including federal preemption of state AI laws, federal and state AI whistleblower protection legislation, and the likely consequences of the end of Chevron deference for the future of AI regulation. Charlie received his J.D. from Yale Law School, where he was an Editor for the Yale Journal on Regulation.

When officials in Washington warn about losing the AI race to China, the conversation turns quickly to military and economic advantage—and rightly so. Advanced AI will reshape everything from weapons systems to medicine, with massive implications for our geopolitical competitiveness. But beneath the great-power competition on AI lies a moral one. The ethical character of the most transformative technology in generations—one that will mediate an ever-larger share of human experience—will be a byproduct of superpower rivalry. At stake is the future of the relationship between individuals and the state, of privacy and control, of human agency and algorithmic authority.

The fear is not merely that China might build better systems, deploy them more widely, or out-sell American competitors. It is that those systems could carry a tide of new norms shaped by a government that surveils its citizens, suppresses dissent, harbors eugenic ambitions, and treats individual autonomy as a problem. As AI comes to dominate our lives as thoroughly as digital media already has—shaping our health and finances, how our children learn, how we are tracked, even our species’ genetic makeup—this battle for AI’s soul will affect us all intimately.

The contours of that battle are almost always left unexamined, but usually assume one of three forms: some leaders suggest AI’s values will be a winner-takes-all byproduct of the race to technical superiority; others imply a conscious struggle to spread tools and platforms with value systems baked in; still others insist that diplomatic cooperation between AI powers is the only way to bend AI’s ethical arc toward humanity’s benefit. Leaving these three paradigms implicit does everyone a disservice, robbing the United States of both moral clarity and strategic opportunity.

Looking at the history of technological competition, there may be truth to all of these three models. But each implies a different approach to maintaining American leadership while preserving the values we claim to champion. And analyzing them clearly reveals how badly our attention is skewed. Today’s debate fixates on breakaway dominance, blinding policymakers to the more consequential contests over encoded values and strategic diplomacy. A rebalanced approach would require something we currently lack: a clear, affirmative vision of what American AI should be for.

Breakaway Tech Dominance

“AI offers the potential promise of extending American hegemony.”

—Tyler Cowen

Breakaway tech dominance is the default (if often implicit) ambition for many leaders invested in the Sino-American AI competition. The thinking goes that if the US is able to master AI ahead of others, that advantage will translate into a general offset in American power over China, with powerful ripple effects in economics, culture, and politics globally. Because this winner-takes-all vision provides such a clear motivation for forging ahead, it is the one most often invoked by pro-tech voices and government leaders.

Historically, large technological advantages tend to precede hegemonic power. This dominant narrative about AI competition draws heavily on historical analogy. Britain’s industrial revolution produced not just economic advantages but also cultural ascent. British institutions, law, language, and ideas spread across the globe on the strength of steam engines and mechanized looms. And this was not history’s first instance of technological offset, a dynamic that has persisted since before the Assyrians’ mastery of iron metallurgy expanded their influence over rival groups.

Applied to AI, this logic suggests that the first nation to achieve a decisive breakthrough could gain civilizational escape velocity. According to this view, if China masters AI before America does, Beijing’s authoritarian model—surveillance systems, social credit schemes, algorithmic control of information and behavior—would spread globally with irresistible momentum. Given that many experts expect the AI transformation to be comparable in scope to industrialization or the dawn of the Iron Age, such fears are justified. Whether or not there are dramatic power shifts between the United States and China in the century ahead, AI is certain to play an outsized role.

According to this paradigm, the singular priority must be aggressive progress in AI capabilities. The strategic implication of this perspective is obvious: there is nothing so important as moving faster than China in pushing the bounds of AI technology. Additionally, there is little need for the United States to consider how American values relate to its AI strategy, because they are seen as downstream of the technical rivalry. In other words, if the United States establishes a decisive AI lead, its values will organically spread; if China masters the technology first, Beijing’s moral vision will take root globally.

An AI lead sufficient to achieve hegemony is unlikely to appear on either side. The breakaway-dominance framework functions only if there is a defensible breakthrough to be had, which is not necessarily the case. Some predictions of a superintelligence “takeoff”—in which a sufficiently advanced AI system starts to improve itself better and faster than humans could—fit that mold. But despite regular predictions of imminent AGI breakthroughs, even many bullish researchers are increasingly skeptical of such a scenario, making the prospects of a highly dominant and defensible AI hegemon seem unlikely.

US-China competition is also too tight for breakaway dominance to occur. The observable pattern of AI progress in recent years suggests a different path. China has successfully positioned itself as an aggressive fast follower. In frontier models, the most competitive arena of AI competition, Chinese labs tend to trail American counterparts by just months at a fraction of the cost. That is an achievement in itself—and it diminishes the likelihood that the United States will achieve a sustained, decisive advantage. While not impossible, it’s unlikely that we will see either country develop and maintain an AI lead significant enough to extend Chinese or Western values globally for any sustained length of time unchallenged. This winner-takes-all model, despite its implicit prominence in many policy discussions, almost certainly misses the full picture.

Encoded Values

“China is doing everything it can to dominate AI globally, and they will program the AI with Chinese values…. We’ve got to double down and make sure that American values are the values of the world, and that we control this global AI agenda.”

—Former US Senator Kyrsten Sinema (I-AZ)

A second model for looking at the moral stakes of Sino-American AI competition is the spread of encoded values: the ethics that are baked into new technologies, whether deliberately or subconsciously. This dynamic is ancient: Roman aqueducts built republican virtues into stone by distributing water first to public fountains, then to public baths, and only later to private homes. Fast-forward to the present day, when the internet stands out as a technology consciously designed with libertarian principles: decentralized architecture, open protocols, and resistance to central control. The resulting technology reflected those values in its most basic protocols (if only initially).

The same will be even truer of AI, given its unique ability to absorb and instantiate value systems. The protocols and architecture around AI systems may also reflect value decisions, but particular moral visions and preferences can also be directly distilled in today’s AI systems—or rooted out of them.

China has been explicit about its intentions to imbue AI with its own values. Official Chinese government regulations mandate that frontier AI systems must “uphold core socialist values”—that is, they must adhere to the Chinese Communist Party’s totalitarian view of history and morality. Chairman Xi Jinping has already made considerable strides toward that end. Beijing invests tens of billions of dollars annually in building a techno-authoritarian ecosystem of tools, platforms, standards, and norms aligned with state priorities: social stability, party authority, and collective “harmony,” rather than individual autonomy. Its companies are experimenting with novel, AI-powered methods of conducting surveillance, enhancing censorship, and even predicting political dissent before it occurs.

US efforts to impart values into its AI ecosystem have been less concerted. The United States has been far less deliberate than China in developing AI consonant with American values. True, documents like the Biden administration’s “AI Bill of Rights” and companies’ interminable desire to write AI-principles documents at least pay lip service to the idea of aligning emerging AI systems with democratic principles. The clearest example of this might be Anthropic’s approach to “constitutional AI,” which aims to evoke the US Constitution in its model operations. And on balance, American AI companies’ systems pay much greater attention to ethics and safety concerns than their Chinese counterparts do. But while these examples reflect a different culture around the development of AI in the United States, they are often only window dressing, and pale in comparison to the concerted state focus that Beijing exerts on the normative trajectory of China’s tech sector.

In practice, American AI may actually erode American values more than it supports them. Indeed, American companies have historically been indispensable in building out China’s techno-authoritarian ecosystem. Between public discourse-corrupting deepfakes, microtargeted political manipulation, and algorithmic amplification of extreme content, leading US developers are already arguably producing AI tools that weaken democracy more than they strengthen it. The governments of both superpowers are working to reap the efficiency benefits of AI in their state bureaucracies. But whereas China’s regime is laying out a proactive vision for how AI will advance authoritarian control, the United States has been largely reactive in accommodating AI to democracy—waiting for courts to adjudicate how new technologies can or cannot be used according to existing American law.

The US needs a robust vision for democratic AI and the will to disseminate it. Breakaway-AI proponents see AI’s future as a straightforward innovation race with downstream ethical repercussions. By contrast, proponents of AI as a system of encoded values see the future as a struggle over vision and will. To further American values and strengthen democracy, this view would require developing a much clearer vision: a compelling idea of how to use AI—not just by reactively implementing guardrails but by proactively conceptualizing what democratic AI should look like and enable. Will, equally important, is the drive to commercialize and aggressively spread the resulting systems around the world in collaboration with allies, while also preventing domestic companies from working with China in ways that undermine the United States’ vision.

The Trump administration, by focusing largely on will, has seen some gains in diffusing US technology. But the PRC still maintains considerable diffusion advantages—especially in the Global South, where China’s price-point advantages and shared development conditions give it an edge in building out AI infrastructure for developing nations. A values-driven AI vision, as described above, remains comparatively underdeveloped on the American side: Washington lacks an inspiring, affirmative narrative about what democratic AI enables for its citizens that authoritarian AI cannot.

The US vision deficit has downstream effects on will. Without a clear, compelling sense of what American companies are building toward, it is impossible to muster the political energy needed to make hard choices. For example, US firms are currently bolstering Beijing’s AI ecosystem in sensitive domains like biotechnology, with little oversight. A stronger vision of American AI would galvanize support for restricting US companies from making such contributions to China’s AI ecosystem.

Emergent Control Regimes

“What Soviet-American nuclear arms control was to world stability since the 1970s, U.S.-Chinese A.I. collaboration to make sure we effectively control these rapidly advancing A.I. systems will be for the stability of tomorrow’s world.”

—New York Times columnist Thomas L. Friedman

The third paradigm for how the moral future of AI hinges on Sino-American competition is that of emergent control regimes—the shared rules and institutions that powers build over time to govern consequential technologies. For policy wonks, this is often the most overlooked—or, more accurately, the most dismissed—avenue for shaping outcomes. In part, it is not taken seriously because those who do raise it tend to do so with flagrant naivete about the weakness of the multilateral system and the political infeasibility of any good-faith agreement between the United States and China. But the idea that international control regimes might emerge over time and prove influential is not far-fetched, particularly in areas such as lethal autonomous weapons and AI-powered human gene editing.

The nuclear era shows that self-interest can drive even rivals to manage powerful technologies together, if imperfectly. Early in the Cold War, the idea that the United States and the Soviet Union could reach an agreement about nuclear weapons seemed fanciful. Nonetheless, both nations came to recognize that countering proliferation was in each nation’s interest, even as they remained locked in an existential nuclear arms struggle. No one in 1945 could have predicted the specific contours of what emerged from the complex, path-dependent interactions among nuclear-armed states with evolving interests over decades: the Nuclear Non-Proliferation Treaty, the International Atomic Energy Agency (IAEA), test-ban agreements, verification mechanisms, and norms around nuclear use. While imperfect, these innovations have unquestionably shaped and constrained the most destructive technology that humanity has yet produced.

Advantages will accrue to whichever power crafts and brands politically feasible international controls. To be sure, there are limits on the degree to which such controls can be planned for, given how contingent they tend to be on changing relations and events. But this is not to say that any control regimes emerging from rapidly advancing AI systems are too contingent to plan for in any way. Such controls are not to be confused with the idealistic proposals—such as Pugwash-style scientist convenings or bilateral red-teaming exchanges—that are characteristic of most current track-two dialogues. Nor are they the feel-good unilateral pronouncements of rosy intentions like the resolution on “the promotion, protection and enjoyment of human rights on the Internet,” passed six times by the UN Human Rights Council since 2012. As with nuclear controls, any diplomatic development of consequence will likely be highly controversial, and will necessarily fall far short of what most peace-loving technologists would like to see.

But imperfect measures can still be strategic. President Eisenhower’s famous “Atoms for Peace” speech in 1953 set the foundation for the IAEA. It also served as a tremendous public relations victory for the United States, projecting America as the responsible superpower, willing to help other countries benefit from peaceful applications of atomic technology. It forced the Soviet Union to compete with the United States in building nuclear reactors for other countries, at a high cost to the Soviets. And subsequent US-Soviet nuclear arms control negotiations did more than help to constrain the risks of nuclear war; they also allowed the United States to pursue advantages in qualitative force capabilities at lower cost, under the auspices of quantitative weapons restrictions.

Compared with China, the US is better positioned to lead emergent control regimes. Technologists have given a great deal of thought to unrealistic controls for theoretical future AI capabilities. Yet little serious thought has gone into diplomacy in those areas where international controls could be made politically feasible, soft-power enhancing, and strategically advantageous. The partial exception is the American-led Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy, which has made strides toward establishing American leadership in rules around the use of lethal autonomous weapons. This guidance is both strategically beneficial to the United States and resonant with American values. Several other areas show promise for similar interventions, not least the ethically fraught genomic applications of emerging AI-powered biotech and the use of AI in high-risk industries. Here the United States has substantial untapped advantages: a global network of allies, a strong history of effective tech diplomacy, and a brand of AI development unencumbered by China’s dystopian techno-authoritarianism. But these advantages so far have not deterred China’s ambitious efforts to eke out a leading position in global AI governance.

A Rebalanced Approach

These three paradigms of AI competition—breakaway dominance, encoded values, and emergent control regimes—are not mutually exclusive. Some areas of AI may see defensible technological breakthroughs that confer long-term advantages; some will become battlegrounds for embedded values; some will develop controls; and some will combine elements from several of these paradigms. They are also interrelated: if one power successfully embeds its values into a widely adopted technology, it will likely occupy a privileged position in control discussions, for example. The question is not which single model is most accurate, but how to allocate attention and resources across all three, and for which issues.

Yet today’s focus remains mistakenly skewed toward a winner-takes-all narrative, blinding policymakers to more consequential contests on encoded values and creative thinking on strategic diplomacy.

To take one example, while initial US nuclear dominance was essential, it was ultimately short-lived. Many developers of the weapon believed America’s 1945 breakthrough would represent an enduring strategic advantage, similar to how many see the race to superintelligence as today’s single defining competition. But America’s nuclear dominance lasted just four short years. The Manhattan Project was indispensable—the United States’ adversaries getting the bomb first would have been catastrophic. But those banking on sustained dominance were in for a rude awakening. Ultimately, clever nuclear diplomacy contributed more to the United States’ victory over the Soviet Union than breakaway nuclear superiority, which never materialized.

Encoding values in technology requires proactive efforts. To the extent that a technology as broad as AI can be compared to a recent innovation, the best analogue is probably the internet—unfortunately, another cautionary tale. Although American engineers deliberately built the internet with libertarian principles, China has been able to co-opt it through force of will. Today, the Great Firewall and the other tools that the CCP has built into the Chinese internet have transformed a freedom-enhancing technology into history’s most sophisticated instrument of surveillance, censorship, and control.

Beijing is exporting these tools abroad, enabling other autocracies to turn the internet away from its original open-society-enhancing design toward repressive ends. The story might have turned out differently if the United States had engaged in more proactive diplomacy—leveraging its unique influence over the development of China’s internet, instead of just issuing feel-good digital-rights statements. At a minimum, curbing American tech companies’ active support of Chinese technological ambitions would have slowed Beijing’s successful authoritarian conquest of the internet. Indeed, the extent of American support for techno-authoritarian progress casts serious doubt on any assertion that the originating society of a technology will organically imbue that technology with its own values. For the war over embedded values, the internet’s lesson is clear: technology neither establishes nor preserves values passively.

The US cannot rely on technological dominance to ensure that AI furthers American values. The United States must learn from these historical cases quickly. AI-powered Chinese “smart cities” are already spreading across the Global South, bringing with them surveillance architectures designed for authoritarian control. Cheap, CCP-compliant Chinese open-source frontier models are already gaining uptake internationally. Party-aligned research centers are developing AI-powered propaganda and censorship tools with unprecedented sophistication; these will soon be diffused abroad, if they haven’t already. Policy and tech leaders may think that their efforts to simply accelerate American technical progress at the frontier of AI innovation will ensure that American values triumph. However, the more probable outcome could be a world awash in cheap, authoritarian AI that outcompetes slightly more sophisticated American offerings that do little to promote American values—and perhaps even erode them.

To course correct, the US must establish a President’s council or congressional commission on democratic AI. A misguided fixation on a winner-takes-all race for technical superiority, as a proxy for a competition of values, risks missing where the real competition lies. It also misses opportunities to rout China diplomatically, similar to America’s successes in nuclear diplomacy. A better approach to AI must start with developing a clearer moral vision for American AI. The President’s Council on Bioethics, established under former President George W. Bush, offers an effective model of what this could look like: a substantive body bringing diverse perspectives to the highest levels of government to grapple with emerging ethical challenges, producing influential reports that shaped discourse and policy. A comparable council on AI and democratic governance could build the intellectual foundations for techno-democracy that do not yet exist, as the American Enterprise Institute’s Council on AI Ethics is beginning to show.

Armed with a clearer, more compelling moral vision for AI, American technologists and policymakers could be galvanized toward supporting the United States’ competition with China with sharper focus. Such a vision would also provide a basis to more aggressively curb American companies’ substantial aiding and abetting of China’s techno-authoritarian ecosystem. And it could provide a stronger foundation for closer collaboration with indispensable like-minded partners such as India: nations better equipped to compete with China on rolling out price-competitive and context-relevant AI offerings in the Global South.

There is little doubt that the Sino-American battle over AI will have tremendous consequences for the future of humanity. Approaching that contest with greater moral clarity is not just the right thing to do; it is also a strategic imperative.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Bill Drexel is a senior fellow at Hudson Institute. His work focuses on United States–India relations, artificial intelligence competition with China, and technology in American grand strategy. Previously, Mr. Drexel worked on technology and national security at the Center for a New American Security, humanitarian innovation at the United Nations, and on Indo-Pacific affairs at the American Enterprise Institute. Drexel’s field experience includes serving as a rescue boat driver during Libya’s migration crisis, conducting investigative research in the surveillance state of Xinjiang, China, and supporting humanitarian data efforts across wartime Ukraine.

Since the late 1950s, Mutual Assured Destruction (MAD) has served as a limiting factor on great-power conflict. The doctrine holds that, if two opposing nations have nuclear weapons that can survive one another’s initial strike, then the near-certainty of devastating retaliation will deter each side from launching a large-scale nuclear attack. Despite this logic, military planners have long considered the possibility of a counterforce nuclear attack, where a superpower uses nuclear weapons to cripple the nuclear capabilities of its enemy. If such an attack were executed preemptively, as a so-called “first strike,” it could both start and end a great-power conflict in a matter of hours: without retaliatory capacity, the defender would be at the mercy of the aggressor’s remaining nuclear weapons and forced to surrender.

The reason this does not happen is that a truly successful counterforce strike is nearly impossible to pull off: a would-be attacker doesn’t know the locations of all opposing missile launchers and submarines, nor does it have missiles with sufficient precision and speed to destroy opposing missile silos before they could launch retaliatory nuclear warheads. Thus, if provoked by a first strike, the opposing side would likely be able to launch a large-scale nuclear response, and the attacker, unable to counter all enemy missiles, could face devastating losses. The result of a preemptive counterforce strike, in other words, would be mutual assured destruction.

AI could change this dynamic. By the mid-2030s, AI-assisted research and development could reduce the cost required to develop and manufacture military hardware by an order of magnitude. Such lower costs could enable a nation with an AI lead to quickly and cheaply build military infrastructure projects, making nuclear counterforce strikes a realistic possibility. Nations with weaker AI capabilities would struggle to quickly build the countermeasures needed to retain a credible nuclear deterrent.

This article will primarily be a technical assessment of how AI could undermine nuclear deterrence, although I’ll lightly touch on a few political aspects. For a longer and more in-depth assessment of the factors discussed here, consider taking a look at an earlier piece I wrote.

Why MAD Has Worked So Far

In any preemptive counterforce strike scenario, the attacker is at a large disadvantage compared with the defender, because they must destroy all of the defender’s nuclear weapons. To execute a counterforce strike, the attacker must achieve all the following requirements simultaneously:

Suppress launch on warning. Many nuclear-armed nations have a policy of launch on warning—launching a retaliatory strike based on sensor data, before a nuclear weapon has been confirmed to land on their territory. The attacker must either use fast-arriving weapons or otherwise disable launch on warning.

Locate and destroy nuclear submarines. Nuclear submarines, while extremely stealthy, are vulnerable once detected. Each nuclear submarine can carry and launch hundreds of warheads. The attacker must accurately track their locations in real time and destroy them all within minutes.

Locate and destroy mobile launchers. China and Russia (although notably not the US, the UK, or France) each field a set of mobile missile launchers that can be dispersed during times of high alert. These mobile launchers are extremely difficult to locate for long enough to successfully strike. However, if accurate and up-to-date position data can be provided, they are easy to destroy.

Destroy all silos. Hardened silos have known positions, but they require almost a direct hit—ideally with a nuclear weapon—to ensure their destruction. They stand out among nuclear launch platforms, as they have the fastest reaction time and the hardest-to-disrupt communications.

Defend against surviving missiles. If the attacking nation has a functioning missile defense system, it may not need to destroy all of an enemy’s nuclear weapons, as its missile defense can handle some leftovers. The stronger a nation’s missile defense, the less thorough its first strike has to be.

Overcome the nuclear taboo. One theory for why nuclear weapons are rarely used is the nuclear taboo. The leaders of both the US and the USSR recognized the gravity of nuclear weapons usage, and had serious humanitarian reservations about starting a nuclear conflict. Even if a counterforce strike were perfect, radioactive nuclear fallout was expected to kill tens of millions in the target country.

The six requirements above have thus far preserved deterrence not because they are physically impossible to meet but because meeting them at the scale required has always been prohibitively expensive.

A useful paradigm to consider in nuclear conflict is the cost-exchange ratio. When one side fields an additional weapon, how much must the other side spend to neutralize it? The concept comes from the Cold War debate over ballistic missile defense. If an attacking nation can build another nuclear missile for $1 million, and the interceptor needed to stop it costs the defending nation $10 million, then missile defense is a losing game: each $1 spent on offense forces a $10 expenditure on defense. This particular cost-exchange ratio explains why a nation launching a counterforce strike would struggle to defend against surviving missiles.

Why AI Could Dramatically Cut Military Infrastructure Costs

AI-assisted military R&D could invert the cost-exchange ratio. If a nation with an AI lead can automate most of the design, systems integration, and production labor that currently makes military infrastructure expensive, that nation’s costs could be significantly reduced while its enemies’ costs stayed the same (since they don’t have the AI advantage). If interceptors cost only $100,000 and missiles still cost $1 million, then it suddenly becomes rational to build more interceptors.

The likelihood of this prediction’s coming true rests on answers to two questions: (1) whether AI can automate most of the intellectual labor, and (2) whether doing so really cuts costs by a margin significant enough to change militaries’ economic calculus. Notably, this forecast does not require progress in robotics, which could further reduce costs. I’ll focus on aerospace manufacturing, which comprises most of what the attacker needs to build: new missiles, surveillance constellations, and missile defense systems.

Can AI automate most intellectual labor? Aerospace projects of the type I’m describing here are highly interdisciplinary, and even simple systems demand expertise in electrical, computer, and mechanical engineering. More-advanced projects may require fundamental research in applied physics or math. While AI assistance for software development is a relatively mature use case, AI abilities in the other fields are much more nascent.

Yet there is good reason to believe that most intellectual fields will follow the same trends. AI abilities have increased at roughly the same rate across various domains, as shown in the chart below.

Additionally, we’re already beginning to see AI labs show interest in automating science and engineering fields: Anthropic is training on electrical engineering, and the company showcased 3D modeling performance in its Claude Fable 5 launch post. OpenAI has made discoveries in mathematics and physics. On the biology front, Google DeepMind’s AlphaFold has solved protein folding, enabling advances in drug discovery.

Extrapolating from such innovations, one engineer could do the work of many: outsourcing most of it to agents and handling only what agents can’t yet do, like running experiments or meeting stakeholders in person.

How much can automated intellectual labor lower military R&D costs? Although aerospace manufacturing seems like a labor-heavy job, it’s a remarkably white-collar profession, with high exposure to AI automation. In a 2021 article, Princeton University computer scientist Edward Felten and co-authors calculated that aerospace manufacturing has an AI Industry Exposure Score of 0.519, in the 70th percentile of all industries (around the same level as real estate). If AI automates the intellectual labor of aerospace manufacturing, this would substantially lower the cost of the final product.

To make things more concrete, consider SEC filings showing the financials of two public launch services companies, Rocket Lab and Firefly Aerospace.

R&D (which accounts for 28%–48% of spend at the two companies) is mostly engineering compensation and analysis, with the remainder going to propellant, test facilities, and hardware. It’s mostly cognitive labor, and therefore a potential target of future AI automation. SG&A expenses (21%–22% of spend) describe the cost of finance, legal, contracts, compliance, and business development. These disciplines rely almost completely on cognitive labor. Cost of revenues (31%–51%) is the most mixed category: it blends physical labor (technicians doing welding, assembly, and launch operations) with white-collar labor (manufacturing, quality, and test engineering), and purchased materials and components.

The costs of aerospace components are themselves quite compressible. For example, star trackers (devices used by satellites to measure their own orientation) can cost over $100,000. These devices’ physical components—a camera sensor, a housing, and some processing hardware— may cost as little as a few thousand dollars altogether. The more significant cost comes from qualification testing and R&D (because of low manufacturing volume, such costs can’t effectively be amortized). So, by reducing the cost of R&D, automating intellectual labor can drive down component costs.

I expect the cost savings to be even greater for defense aerospace products, which usually cost more than comparable commercial products. This higher cost partly reflects the increased documentation and security practices required for highly regulated uses. Such regulatory compliance practices rest almost entirely on cognitive labor that can be automated. Another driver of these product’s higher costs is costlier labor: defense contractors must have security clearances, limiting the worker pool and raising wages. An AI, on the other hand, must be cleared only once; it can then be scaled indefinitely.

What Could We Build?

Recall the six requirements that have kept a counterforce strike out of reach. Each one has held not because it was physically impossible, but because meeting it at scale was prohibitively expensive. Once cheap intellectual labor collapses those costs, an attacker with an AI lead could pursue several military megaprojects at once, each one solving a different requirement needed to launch a successful counterforce strike. This section will focus on the three requirements I expect to become significantly easier to meet.

Locating and destroying nuclear submarines. Research has shown that, in certain conditions, moving submarines leave a surface wake detectable by Synthetic Aperture Radar (SAR), a space-based radar system. A large constellation of SAR satellites would permit near-continuous surveillance of the entire world, regardless of time of day or local weather conditions. The US is already pursuing a SAR satellite constellation, and will launch its first satellite in 2028. The US could supplement this effort with a space-based constellation of Light Detection and Ranging (LiDAR) satellites. Each satellite would measure depth via pulses of intense light, detecting even stationary submarines (although that would require clear weather). Certain frequencies of LiDAR can detect submarines within 200 meters of the surface, within typical nuclear submarine operating depths. A dense constellation of LiDAR and SAR satellites could sweep the oceans, and expose submarines at scale.

However, while both satellite methods can reveal submarine positions temporarily, the only tool to keep track of them consistently would be underwater drones (often called Unmanned Underwater Vehicles, or UUVs). UUVs are currently limited by the difficulty of autonomous operation, but AI R&D would likely significantly improve this. The US has been funding research in this direction.

Defending against surviving missiles. A system with tens of thousands of interceptors prelaunched in space (similar to the 1980s Brilliant Pebbles concept, originally abandoned due to cost issues) could counter any missiles that are missed by the first strike. The US is already pursuing this, too, with its Golden Dome system.

Overcoming the nuclear taboo. The missiles of the Cold War had poor precision, so warheads with an explosive yield of hundreds of kilotons were common (designed to guarantee a silo kill). But, as mentioned, such heavy warheads would result in enough fallout to guarantee millions of deaths. In 2017, Keir A. Leiber of Georgetown and Darryl G. Press of Dartmouth found that modern missiles have much higher precision than those from the Cold War, and future improvements could reduce the average targeting error to mere meters. With such high precision, very low-yield weapons could be used, with little to no fallout. A counterforce strike could be accomplished with an estimated death toll of around tens of thousands rather than millions, well within the range of wars nations are willing to start.

Other aspects of the retaliator’s deterrent are vulnerable to AI R&D too. The retaliators’ mobile missile launchers can be located with the exact same SAR constellation that we discussed for submarine detection, leaving them vulnerable to a first strike. Launch on warning systems are also vulnerable. While it’s unlikely they could be hacked outright, Anthropic’s Mythos demonstrated AI driven vulnerability discovery that could be used to confuse, delay, or reduce confidence. Additionally, the defender’s early warning response time can be shortened significantly with depressed trajectory submarine-launched missiles, which could cover 2000 km in only 10 minutes. The combination of high-speed and precise missiles also works to efficiently counter fixed silos.

The Defender’s Options

If a lagging nation realizes that a rival is on course to achieve nuclear primacy—the ability to execute a counterforce strike without retaliation—it still has a few options.

The first, and most straightforward option is to expand the arsenal. It can increase the number of silos, build more nuclear submarines and mobile missile launchers, and raise its level of alert. This strategy would work in the short term, since the attacker would be forced to scale up its own forces until it was certain it could neutralize all of the new forces. The problem is the cost-exchange ratio. If the attacker has an AI advantage, the retaliator could end up paying a larger price for each new silo than the attacker pays to build the missiles or interceptors that could defeat it.

The second option is to target the root cause: the AI gap between the attacker and retaliator. Potential avenues in this direction can range from relatively diplomatic to highly escalatory. Options include disrupting the attacker’s supply chain, launching data poisoning attacks, or sabotaging its AI training runs. However, the most extreme actions—direct kinetic attacks on datacenters or researchers—would be likely to start wars. But even these interventions will only be effective if applied early. Once the lead is large enough, sufficiently smart AIs will already be trained. Thus, an AI-lagging defender must be alert enough to act before the AI capability gap becomes overwhelming.

Finally, the attacker and retaliator could negotiate a treaty. A bilateral arms-control regime could in principle cap satellite constellations, ballistic missile interceptors, or AI compute used for military R&D. There is precedent here, especially in the nuclear domain. However, the main challenge will be aligning incentives. The leading nation has no incentive to join a treaty where only the lagging party stands to gain. Traditional arms-control treaties have only worked where there were symmetric costs on both sides.

AI May Disrupt Nuclear Deterrence

In conclusion, the historical robustness of nuclear deterrence has rested on a cost-exchange ratio that favors the retaliator, but AI-assisted R&D can invert the ratio. This might enable a single superpower leading in AI to achieve nuclear primacy. The AI superpower would then wield enormous leverage, as it could credibly threaten to win just about any war. Such negotiating leverage could reshape the global balance of power, even if nuclear weapons are never used.

Even before military infrastructure megaprojects are complete, they will affect policy. If major powers come to believe that AI R&D may make their nuclear deterrents less effective, they will have incentives to expand arsenals, shorten decision timelines, rely more heavily on launch on warning, contest one another’s space architectures, and target the AI and semiconductor bases that underpin their adversaries. This new equilibrium would increase military spending, shorten decision times, and raise the risk of war.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Govind “Vinny” Pimpale is a research fellow at the Foundation for American Innovation, where he focuses on AI policy. Before joining FAI, he worked at a startup developing reinforcement learning environments, and prior to that, as an AI evaluations researcher. He holds a BS in Computer Science and Engineering from UCLA.

On June 12, the Trump administration issued an order requiring Anthropic to suspend access to its two most advanced AI models—Fable 5 and Mythos 5—for non-American nationals, just days after their public release. The decision caused immediate backlash across Europe. Politicians from France’s Gabriel Attal and Jordan Bardella to the Netherlands’ Geert Wilders, alongside European industry and civil society, amplified calls for AI sovereignty. Attal, the presidential candidate for Macron’s Renaissance party, even likened the shutdown of Anthropic’s models to Iran’s blockade of the Strait of Hormuz.

While such reactions are understandable, they underscore what we already know: Europe is behind on AI, overly dependent on the US, and vulnerable to unilateral decisions. Perhaps the renewed sovereignty rhetoric can help build the political capital needed to fix the continent’s AI positioning, but what Europe needs more urgently is a clear-eyed account of what to actually do. In the wake of the Anthropic episode, this piece separates what has genuinely changed from what is being overstated, while identifying where European policymakers should focus their attention.

Europe Should Have Planned for This

In recent years, AI has become increasingly relevant to national security. Europe could have anticipated that the US government would restrict access to US AI models at some point, but has not acted quickly enough to secure its position.

Export controls on US technology have affected Europe before. The use of export controls on American technology for national security reasons is nothing new and is not unique to the current US government. The Biden administration’s AI Diffusion Rule did the same for advanced chips, treating AI hardware as a national security instrument and using export controls to manage its global distribution. Europe was affected then, too, with Tier 1 countries in western Europe split from Tier 2 countries across much of eastern Europe. What the new controls do, less selectively than the Diffusion Rule, is extend export controls from AI hardware to AI models themselves.

The US implemented the new control suddenly and with opaque reasoning. While export controls are not unprecedented, the Trump administration’s recent directive differed from Biden’s Diffusion Rule. This decision happened more quickly, and was driven by harder-to-interpret motives, particularly in light of the recent clash between Anthropic and the Pentagon. It is plausible that the US government wanted to block Fable’s deployment for everyone (Americans included) and used export controls as a tool to do so.

Europe has been moving too slowly to build sovereign AI or secure access to frontier American AI. Europe could have planned more effectively for this moment by treating AI as a sovereign imperative and developing the resources and infrastructure needed to better serve this goal. Instead, the scaling down of EU plans for AI gigafactories and the successive delays to its Cloud and AI Development Act point to the complexity of moving at the pace of developments across the Atlantic. A separate but related issue is the need to access American frontier AI models for, among other things, hardening European infrastructure against cyberattacks. European leaders could have sought to negotiate guaranteed early access to highly capable systems. Yet the EU has lagged here as well, obtaining access to Anthropic’s Mythos model about two months after the company first started sharing it with a small group of American organizations to bolster cybersecurity.

Subscribe now

The Restrictions’ Short-Term Impact on Europe May Be Limited

Europe is not, for now, significantly worse off without access to Fable 5 and Mythos 5. This is for two reasons.

Access to Fable 5 and Mythos 5 will probably be restored. The US government and Anthropic will likely come to some sort of agreement to enable the continued rollout of these models. For one thing, the export restriction also applies to foreign nationals working inside American labs—employees crucial to model development. Maintaining the controls would be akin to telling Anthropic to stop developing future AI models.

Other models are good enough for most current applications of AI. Other than some cyberdefense uses, most tasks across the European economy do not currently require the most capable Anthropic models. Alternative models can perform many of the tasks that users would have assigned to Fable, at a similar level of competence. OpenAI’s GPT 5.5, whose capabilities are comparable to those of Fable, remains free of export controls. Open-source alternatives, only months behind, also remain available to European markets. Choosing an AI model is ultimately a practical calculation: the right capability, at the right price for the task at hand. That is why AI adopters often prefer cheaper Chinese models over frontier American ones: they choose based on the job they need to complete, rather than simply selecting the models that sit on top of a capability leaderboard.

Hence, the economic drag Europe will face by losing access to Fable is likely to be modest in the short run. It will compound only if the restriction on European access to American frontier models continues and the capability gap between American models and alternatives significantly widens, neither of which seem likely in the near term.

In the long term, Europe needs frontier AI for cybersecurity and economic competitiveness. Even if the model restrictions’ negative impact can be absorbed for now, the European market will suffer if it cannot access leading models like Mythos and Fable. Aura Salla, a Member of the European Parliament, has argued that “Europe is better off without these models,” because they pose significant cybersecurity risks, with insufficient safeguards. But access to frontier models is an absolute prerequisite for defensive security, allowing domestic companies and governments to identify and patch vulnerabilities before attackers find and exploit them. In the long term, frontier access is also an economic necessity to prevent Europe’s industrial base from being relegated to secondary tiers of productivity. Being locked out of the most capable AI models is categorically different from Europe choosing how to adopt AI on its own terms.

In the Long Term, These Restrictions May Provide Opportunities for Europe

Counterintuitively, the US government’s recent move may turn out to be good for Europe. While a gradual loss of access to American frontier AI might have been anticipated, that eventuality may never have felt urgent enough to prompt serious action. This sudden, unexpected loss of access will be a sobering warning.

The US government’s directive underlined its power to withdraw access. Export-controlling Anthropic’s leading models revealed who holds the kill switch—and how easy it is to press. Even if the controls are overturned, the intent and execution is visible to every government in the world. The use of export controls to restrict global access to AI models signals the distinct power of the American executive branch in shaping the trajectory of AI.

The more dependent Europe is on American AI, the more damaging restrictions could be. Fortunately, for now, Europe enjoys a narrow window of insulation: unlike cloud services, leading American AI models have not yet been deeply woven into European public administration or critical infrastructure. Access to American AI, like access to the nuclear umbrella, may come at a cost to those that accept and depend on it.

The episode is an opportunity to drive political momentum toward middle power coordination on AI. Even if the US directive brings only limited short-term consequences, it illustrates how vulnerable Europe could be in the future if it does not start working to secure frontier AI access now. Countries within the EU, including France, Germany, and the Netherlands, could do more to engage other middle powers like the UK, India, South Korea, Japan, and Canada. They can coordinate on how they wish to govern this technology, secure access to the frontier, and shape how AI affects society. Even if these countries play different roles in the AI stack (and even though existing coordination channels between these countries are currently scant), they share a common interest in preventing any one government from dictating access and governance unilaterally.

As part of a coordinated agenda on AI governance, middle powers could align on their evaluations regimes and incident monitoring practices, while slowly building up shared leverage through collective procurement standards and investment in each other’s ecosystems. A recent example is Aleph Alpha and Cohere’s merger. More ambitious moves could draw on the model of the Eurofighter Typhoon program: pooling procurement across multiple governments to build collective infrastructure or capabilities that no single country could generate alone.

Europe must also navigate public-private partnerships between Washington and American frontier labs. The Mythos episode marks a turning point in government-industry relationships on AI. Despite having been working with government officials and American industry to secure the world’s most critical software, Anthropic had to take its leading models offline immediately at Washington’s request. The age of governments ignoring capable model releases is likely over. The question is whether oversight will be principled and based on scientific thresholds being crossed, or reactive and politically driven. For now, absent federal laws on AI, the latter seems more likely.

The right response everywhere is to invest in building a more mature evaluation ecosystem, enforce actual regulations (like the EU AI Act), and establish regular and transparent predeployment engagement between governments and AI developers—so that policymakers are not scrambling to assess model capabilities in the days after a release. Beyond nurturing positive oversight mechanisms, European governments would also do well to consider how to respond to a reality where consequential AI decisions are made in closed-door conversations between the US government and US AI companies.

How Does Europe Build Leverage When It Needs Results Now?

AI is progressing rapidly, and Europe must act quickly to improve its position. Fortunately, it has a number of tools at its disposal to maintain access to frontier AI models.

Leading AI companies need compute capacity, and Europe could provide it. First, Europe could seek to build compute capacity quickly, along with the energy infrastructure necessary to power it. The aim would be to use this compute not exclusively for European AI development but as infrastructure that reduces dependence and creates negotiating leverage. The constraint on frontier AI development and deployment is increasingly compute, and evidence of its scarcity keeps accumulating. In March 2026, Anthropic tightened peak-hour session limits for paying users due to capacity constraints. By May, it had agreed to spend $1.25 billion per month to rent capacity from xAI’s Colossus cluster through 2029. When a leading AI lab is forced to buy infrastructure from a direct competitor just to keep its basic tiers online, it is a clear signal that compute remains a constraint to AI development.

An AI lab that needs European data centers, European talent, and European revenue has reasons to treat Europe as a partner rather than a market it can afford to work around. If Europe becomes a major host of leading models, this makes restrictions less likely due to the costs to American companies of losing access to the European market. Not so long ago, Nvidia was allowed to sell chips to China because of its importance as a market. To build such leverage in practice, Europe needs to urgently address the barriers to buildout: high energy costs, slow planning processes, and fragmented capital markets.

Middle powers could cooperate more deeply to demonstrate their importance to AI development. Europe should work with other middle powers to mobilize a coordinated response. The semiconductor supply chain runs through the Netherlands, Japan, South Korea, and Taiwan. AI evaluation and testing capacity is concentrated in the UK. Many of the most commercially significant AI applications—from Lovable in Sweden to leading adopters across Canada and India—are built on top of American frontier models, generating the revenue and usage data those labs depend on. Export-controlling American models while depending on allied components, talent, testing infrastructure, revenue, and data should be a strategy with a limited shelf life. Middle powers need to work more closely to demonstrate that they are as necessary to producing AI as the companies currently building it.

Europe must both invest in AI sovereignty and build relationships with US AI companies. Finally, Europe should amplify efforts to build indigenous frontier capabilities and chips, even if the near-term prospects appear improbable. An ambitious European AI effort would require spending levels that dwarf anything Europe can credibly mobilize through subsidies alone. Meta will invest $125 billion in capital expenditures this year, more than Germany’s entire defense budget, and its AI models still fail to match capabilities from OpenAI and Anthropic. But, difficulty aside, meaningful European AI development will become a necessity over the long term. The question is how Europe can build sovereign AI capabilities while strategically developing partnerships with value-aligned US companies, such as Anthropic. Doing neither or only one has too many downsides; doing both buys optionality.

Export Controls Are a Wake-Up Call for Europe

Arguably, the export controls on Fable 5 and Mythos 5 do not fundamentally change Europe’s position on AI. Europe was dependent on American AI models and chips before last Friday and remains so today. The immediate economic consequences are likely manageable, particularly if the restrictions prove short-lived. But Europe’s response must nonetheless move beyond the familiar debates on sovereignty. It should be concrete: expand compute capacity, coordinate closely with other middle powers, invest in domestic capabilities, and deepen partnerships with firms that need European markets as much as Europe needs them. The overall impact of the recent order could in fact be positive for Europe—but only if it spurs the continent into acting on AI in ways that it already needed to.

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Afek Shamir is an Analyst at RAND Europe, working with the Center on AI, Security, and Technology and the Frontiers of Technology hub. His research focuses on Europe’s role in frontier AI development and governance, with particular interest in the continent’s geopolitical positioning and leverage vis-a-vis other leading AI powers. Prior to RAND, Afek worked at a Brussels-based think tank on the EU AI Act’s governance of general-purpose AI as a Talos fellow and interned at the Tony Blair Institute. Afek holds an M.Sc. in European and International Public Policy from the London School of Economics.

The debate about AI’s future economic impacts often settles into two camps predicting incompatible futures. One camp insists that AI is a normal technology: simply the next in a long line of economic transformations, each increasing productivity while gradually reallocating labor. The other camp warns that AI will become a great displacer: that automation will hollow out the working class within a decade and eventually disempower large swaths of human workers.

Each side often treats the other’s predictions as unserious, and, consequently, policy debates often split along the same tired fault lines: whether we need reskilling or universal basic income, whether we should strengthen safety nets or structurally redesign our economy. The two camps’ forecasts diverge so sharply that it can be hard to see that they do not have to be mutually exclusive.

Rather, a more useful framing treats these predictions as describing different stages of the same overarching transition rather than as competing accounts of the same moment. From a macro perspective, both narratives will play out roughly sequentially, though those phases may overlap substantially across sectors and timelines.

In the short term, it seems inevitable that AI will look like an accelerated version of past automation waves: significant productivity gains after a period of integration, job displacement in specific occupations, and a familiar churn of workers cycling into new roles.

In the long term, it is hard to conceive of a future in which transformative AI systems do not lead to a massive restructuring of the economy and a reconsideration of the role of human labor. A world in which machine intelligence can perform most economically valuable cognitive (and, increasingly, physical) labor, at a fraction of human cost, must eventually lead to a completely new kind of economic system.

Our responsibility during this period is to prepare and to guide our economy deliberately through these sequential and overlapping transitions. To do so, we must develop thoughtful roadmaps that account for both near-term and long-term impacts, and that can adapt effectively to support national governments in managing these changes.

In the rest of this article, we lay out such a roadmap, describing each phase of the economic transition and outlining some of the most commonly discussed policy solutions at each stage.

Near Term: Managing Economic Shocks

In the near term, the most pressing economic concerns are AI economic shocks and the labor displacement of certain groups, such as early-career employees or workers in highly exposed occupations. Certainly, there will be other jobs to transition into. The only question is whether they will be accessible or desirable.

Initial displacement will be concentrated in certain domains. The impact of AI will be highly varied across sectors, with some being significantly more vulnerable than others. A recent report from Boston Consulting Group estimates that around 50% of American jobs will see restructuring or reshaping due to AI. For the average white-collar employee or college graduate, what their career will look like in five years is quite unclear.

Displacement could be sudden. In particular, markets are concerned about a potential rapid collapse of demand for historically well-paying occupations such as software engineers, financial analysts, and legal associates, which could trigger cascading effects. Research suggests that up to 90% of automation-related job losses occur during the first year of recessions. If an increasingly unequal economy encounters a sudden slowdown, labor displacement could be both sudden and concentrated. The initial shock would be further compounded by the second-order effects of reduced tax revenues, weakened consumer demand, and wage scarring (the long-term negative impact of unemployment on an individual’s wages).

One remedy is to modernize and scale active labor market policies. Among the most popular solutions to these near-term risks are policy interventions aiming to help people find and keep jobs. Wage insurance programs have shown promising empirical evidence to improve worker outcomes during transition periods. For instance, Germany’s Kurzarbeit helped it avoid rising unemployment during the 2008 financial crisis, making it the only G7 country to do so. Meanwhile, the US’s Reemployment Trade Adjustment Assistance program is estimated to have increased employment probability by 8% to 17%, and it largely pays for itself through higher tax revenue and reduced benefit outlays.

Reskilling programs have also been widely discussed among policymakers, though it is still unclear what industries workers should be retraining for. Other proposals include dynamically expanding unemployment benefits or job guarantee programs that could provide transitional public employment.

To strengthen these programs, governments must invest more significantly in labor market data, streamlined benefits systems, and payment infrastructure—the absence of which hampered COVID-era relief distribution globally. These investments, made in the near term, can also help to develop the infrastructural backbone for more ambitious medium- and long-term interventions.

Medium Term: Navigating Reorganization and Divergence

In the medium term, the transition to an economy dominated by AI will present both extraordinary opportunities and structural risks. As AI systems become increasingly capable, they will be able to complete ever more workstreams end-to-end, potentially driving broader job displacement than seen in the near term. A new class of superstar firms might emerge in winner-takes-all markets where scale—especially of compute and capital—could confer decisive advantages. The medium-term period could be defined by a delayed but rapidly accelerating impact on productivity and employment, an increasing divergence in AI adoption and growth between regions and countries, and growing pressure on fiscal systems.

Differences in AI adoption may drive divergent outcomes for countries. Since countries will adopt and develop AI unequally, the impacts on productivity are also likely to differ. This could contribute to increasing global inequality, especially when technological diffusion is limited (e.g., by export controls, protectionism, or regulatory barriers). The White House Council of Economic Advisers warns that countries lacking the ability to develop advanced AI face compounding disadvantages that could produce a second Great Divergence, paralleling the Industrial Revolution.

Widespread labor displacement could substantially impact tax revenue. If new economic growth is increasingly captured by a smaller proportion of AI-led corporations, tax systems built primarily around payroll taxation could face revenue shortfalls and a structural mismatch between where value is created and where it is taxed. Globally and in the US, labor revenue is typically taxed at a significantly higher rate (and more effectively) compared with how capital is taxed. A substantial shift of economic growth toward capital could therefore lead to multi-digit percentage declines in revenue.

Several common themes in today’s policy conversation could help to address the medium-term economic impacts of AI.

Policymakers will need to begin considering taxation reforms. Leading economists have proposed a shift toward consumption-based taxation if labor income declines in importance. By capturing spending rather than earning, higher consumption taxes would sidestep the question of labor versus capital income. A contrasting approach centers around progressive corporate taxation, which would impose higher marginal rates on the most profitable multinational enterprises through global coordination. Alternatively, token taxes could capture revenue streams directly from leading AI corporations. Any of these measures alone might be insufficient; an effective fiscal response would likely need to combine approaches to create a tax code with adequate resilience and fitness for future scenarios.

Countries could strengthen their economies by fostering AI-related industries. As economies grow more focused on AI, countries will need to consider active industrial policy, primarily to aid their economic competitiveness and stimulate job creation in increasingly important industries. Ramping up AI infrastructural investments is already a mainstream discussion in most countries, where we are seeing proposals to increase capital inflows, create special “AI Growth Zones,” and subsidize data center investments. Emerging ideas include publicly owned AI foundation models or tax breaks incentivizing employers to invest in worker retraining or human capital development.

Governments may need to actively protect vulnerable workers and industries. Labor subsidies targeting socially valuable sectors could preserve employment where human participation generates positive externalities, such as education or elderly care. For example, in 2021 South Korea’s Senior Employment Program provided work for roughly 840,000 people over age 60. Elsewhere, leading economists have proposed a series of policies to encourage “pro-worker AI”: deploying assistive AI that makes workers more productive, instead of outright replacing them. Strengthening collective bargaining rights may also play a key role in determining whether organized labor can secure meaningful leverage for workers.

By making such investments in the next decade, governments will determine whether they can protect workers in a medium-term future where new economic growth becomes increasingly dominated by capital. These measures could also lay the foundation for managing the largest economic impacts of AI over the long term.

Long Term: Restructuring Economies

Provided we avoid the more extreme risks of AI, a likely trajectory of the technology is that it eventually surpasses humans across an increasing proportion of economically valuable tasks. Machine intelligence faces fewer fundamental constraints than its biological counterpart. AI will continue to decrease in cost both for cognitive labor—which is already price-competitive with humans on many tasks—and eventually for manual labor, which will be constrained primarily by the marginal cost of robotics systems.

AI could automate a steadily increasing proportion of new economic growth. In the long run, there may be few persistent bottlenecks to automation as the economy restructures around powerful AI systems. Durable human advantages may persist primarily in domains requiring interpersonal connection or physical presence, as well as in contexts where people specifically prefer human involvement.

Automation could change the social contract. This transformation could challenge the foundational premise of modern economies: that hard work and talent are the primary route to income and economic security. If labor ceases to be a reliable path to capital accumulation, core aspects of the social contract may break down for a growing share of the population. Eventually, the policy challenge may shift to fundamentally redesigning the relationship between citizens and the economy itself.

Many ideas for meeting this challenge have been suggested, beyond the call for universal basic income.

Equity and capital may need to be predistributed. Many recent proposals have centered around fractional public ownership of AI equity, requiring that AI firms transfer equity to governments, which would own them on behalf of the public. Unlike universal basic income, which requires perpetual political will, redistributing capital would create durable property rights that compound over time. Leading economists such as David Autor and Neil Thompson argue that we should begin experimenting with universal basic capital (UBC) now, as, even in the most ambitious cases, it would take multiple decades for capital ownership to be broadly diffused.

Sovereign wealth and international coordination may help to distribute AI benefits. Sovereign wealth funds (SWFs) have emerged as a widely discussed institutional vehicle for public co-ownership, with promising examples in Norway and Alaska. By holding equity stakes in AI firms and infrastructure, governments could become better invested in the long-term success of AI and eventually distribute these gains to citizens via cash dividends or public services. Globally, an increasing divergence between countries that produce AI and countries that consume it may eventually lead to calls for stronger international institutions, multilateral tax coordination, or even dividend funds on behalf of all humans.

Governments may provide universal basic services. In the long run, governments may choose to expand the direct provision of essential services—including health care, child care, and education—so that they are fully decoupled from employment status. The UK’s National Health Service, Finland’s free university system, and Vienna’s social housing model demonstrate that universal basic services can be administratively feasible and politically durable.

With a combination of these policies, it is entirely plausible that in highly automated and productive futures, governments could ensure a basic level of economic security for all citizens. The open question is whether the political will to do so will exist.

Conclusion

The exact policy interventions will differ dramatically on a country-by-country basis. There is no single policy roadmap that will work everywhere; each government will need to design a strategy uniquely suited to its own citizens, culture, and institutional context.

Each stage of interventions can help create the infrastructure for the next. In many cases, the policy proposals described above help to lay the groundwork for navigating later stages of the economic transition, as well as having immediate benefits during the stage at which they are implemented. Building social safety nets today may enable greater bargaining power for labor later. Strengthening taxation mechanisms eventually supports funding for broader public service provisioning. Effective economic policies compound: they succeed as deeply interwoven networks over decades of investment.

Nations will need to develop their own economic preparedness plans. Governments should develop self-assessments and policy strategies tailored to their specific labor market exposure to AI. By evaluating a wide range of potential scenarios, countries can test their preparedness for highly uncertain futures. Only with that foundation can they develop strategic policy roadmaps for the transition ahead.

Policymakers globally are just beginning to recognize that the intersection of AI and labor will be a defining theme of upcoming elections. Within a few years, this will likely become a core issue for political candidates around the world. Yet governments are not remotely prepared to offer coherent responses on the scale these challenges will require. If we can support our policymakers with better foresight and more coherent roadmaps to economic success, we may be able to guide this upcoming transition toward prosperity and widely shared financial security.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Deric Cheng is the Director of Research for Windfall Trust, a non-profit focused on ensuring that the economic benefits of advanced AI are shared by everyone. He is also the lead for AGI Social Contract, a consortium of experts proposing strategies to design a new social contract for a post-AGI society.

Jacob Schaal an economist researching AI’s labor market impacts. He is a researcher at Kings College London, and co-edits the AI Economics Brief at Windfall Trust. He holds an MSc in Economics from the London School of Economics.

“Bloodthirsty AI models more willing to start nuclear war than human counterparts.”

It seems almost inevitable that any media headline about AI will be hyperbolic. Yet this statement, taken from a February 2026 New York Post headline, was accurate. The alarming claim stems from a widely publicized study by King’s College London, which found that, in simulations of international crises, LLMs reached for the nuclear trigger 95% of the time.

This academic study drew mainstream-media attention because it touched upon a cultural narrative that has long combined the concept of AI with nuclear weapons. Arguably, the first movie to bring the two together was 1957’s “Invisible Boy,” featuring Robby the Robot, who would later become famous (and less bloodthirsty) in the 1960s TV series “Lost in Space.” The trope has since been repeated across franchises ranging from “The Terminator” to “Mission Impossible.”

Yet the AI-nuclear fear is not confined to the media and movie theaters. The King’s College study is only one of scores of similar academic and think-tank research projects on AI’s proclivities for nuclear war, which have been backed by millions of dollars in research grants. Among the entities that have funded such work are the National Nuclear Security Administration (NNSA), the Department of Energy, the Department of Defense, Anthropic, the MacArthur Foundation, the Carnegie Corporation of New York, the Future of Life Institute, Open Philanthropy, RAND, and the Smith Richardson Foundation. It is also an essential element in the larger field of study on the existential risks of AI, funded to the tune of multiple billions of dollars at many of the world’s leading universities, including Oxford, Cambridge, Stanford, and the University of California, Berkeley.

Beyond academia, the discussion of AI-nuclear risks has also entered the halls of government in settings that range from the UN to multiple US-China superpower summits to the US Congress. It has even become part of the reasoning for why states like California are seeking limits on frontier AI. Indeed, the fear of “the incorporation of AI into nuclear decision-making” has reached such a height that the “Bulletin of the Atomic Scientists” earlier this year moved its famous “Doomsday Clock” to 85 seconds to midnight, the closest it has ever been. (For comparison, the clock was at 12 minutes in the aftermath of the Cuban Missile Crisis.)

Despite all this, I am excited to be the bearer of good news: AI is not going to start a nuclear war anytime soon. I do, however, also have bad news: AI is making it more likely that humans will start a nuclear war. And, if we want to avoid that outcome, we should focus on mitigating real risks, resisting incentives that steer us toward the tropes.

The following report will explain why no government will likely delegate nuclear launch to machines either now or in the future, and identify three mechanisms—arms racing, miscalculation, and machine speed—by which AI could already be amplifying the risk of humans deciding to go to war.

Why AI Will Not Decide Nuclear Wars

Studies such as the one from King’s College consider hypothetical scenarios in which an LLM determines whether a nation should proliferate nuclear weapons or even when to fire a full nuclear strike. Yet there are multiple good reasons why humans will not, in reality, hand machines this responsibility.

Adversarial robustness and a dearth of training data present technological barriers. First, to be trusted with nuclear decisions, AI systems would need to not only be more capable than humans but also extremely adversarially robust—meaning that geopolitical opponents could not influence such systems by manipulating input data. While capabilities have come a long way, adversarial robustness has historically proven very challenging. Another glaring issue is the availability of training data for fighting nuclear wars: there is none.

There are laws and agreements prohibiting fully autonomous nuclear decisions. Alongside the technological barriers are domestic laws and international agreements. Section 1638 of the US FY 2025 National Defense Authorization Act calls for keeping a human in the loop for nuclear decisions. Similarly, Chinese President Xi Jinping and then–US President Joe Biden agreed in late 2024 that AI should never be granted the authority to initiate a nuclear launch.

AI technology may advance and laws could certainly change or be ignored. But one thing will not: the use of nuclear weapons will always come down to decisions of politics and war. This is critical. It explains why AI systems are not presently in the position they occupy in the studies and movies and why they will not occupy that position in the future.

Take the area of politics. In the 2025 film “A House of Dynamite,” a fictional US president facing a nuclear crisis laments that he and his team spent more time on a Supreme Court justice nomination than on whether to launch a nuclear strike—the most important decision of not just his life but maybe human history.

What the various scenarios and studies require is that a human leader either consults an AI for advice, when weighing a potential nuclear strike, or has already turned over this decision to a machine at some point beforehand. Either would be the most unlikely of political outcomes, in any form of government outside the imaginings of Silicon Valley techno-state.

Political leaders would not give up the most important decision of their lives to an AI. Political leaders would neither surrender nuclear decision-making to a machine nor blindly follow its advice contrary to their own judgment. To think that elected leaders would defer to an AI even on whom to nominate to the Supreme Court, let alone on a nuclear launch, is to assume that such leaders lack confidence in themselves and their judgements—something which political leaders are not in short supply of. This is why it was so easy for Biden and Xi to agree not to let AI decide nuclear launch; it is something they would never have considered anyway.

Any suggestion to put AIs in charge of nuclear decisions would spark public outcry. Moreover, even to contemplate creating the policy and technical mechanism for changing the multiple-billion-dollar, three-generations-old nuclear command-and-control architecture would create career-ending outcomes for democratically elected leaders. Only 3% of Americans believe that we should trust information from AI even “almost all of the time.” A leader proposing to trust it with nuclear war could expect to be rapidly relieved of any power to implement that proposal.

Authoritarian leaders are even less inclined to hand over consequential decisions to anyone or anything other than themselves. This is even codified in some nations’ command structures. Under China’s “Chairman Responsibility System,” the operational authority to command or launch nuclear weapons rests solely with the chairman of the Central Military Commission, who just happens to be the general secretary of the Chinese Communist Party.

Militaries are rapidly adopting AI, but not for nuclear decision-making. Of course, militaries are spending billions of dollars on AI, but not on integrating it into nuclear decisions. As a study on the “nexus of nuclear weapons and AI” summed, “AI is unlikely to have material impact on nuclear command and control, which for several decades have synthesized automation but not autonomy.”

Rather, like institutions across the economy and society, militaries are turning to AI to save on money, time, or head count, and/or to get better at the tasks they are not good at. Nuclear warfighting is not an area where militaries feel those pressures.

To begin, the vast majority of militaries’ AI applications are off the battlefield in support roles, usually with civilian parallels, ranging from military medicine to military logistics. Such support roles make up over 90% of military jobs.

Militaries need AI to assist in dynamic, non-nuclear warfare. Applied to warfighting, the US military has been focused on closing the “OODA loop” (Observe, Orient, Decide, Act) in non-nuclear war, through enhancing sensors, expanding analytical capabilities, and accelerating the application of this process to a target to strike or defend against.

Consider the 2026 Iran-US war. The war began in an era with AI, and may have been partly about nuclear weapons. However, neither the decision to go to war nor the use of AI in the war have been linked to any of the worries that have consumed so many grant proposals and reports. AI was used by all parties, but in nothing like the way it is used in the scenarios. US and Israeli forces tracked hundreds of thousands of moving parts, analyzing both their own forces and their potential targets, and then parsed them out for thousands of strikes. In the first four weeks of conflict, US forces struck more than 10,000 targets, while Israel hit thousands more.

In turn, Iran used AI to aid in its tracking and targeting of everything from tankers to US helicopters, firing off thousands of drones and missiles. The conflict then moved into a cat-and-mouse game in which Iranians sought to fire hidden missiles or drones before loitering US drones found and destroyed them. Each side would then try to analyze whether it had destroyed its target or needed to repeat the attempt. One military analysis described the machine-speed cycle as a “90-second war” taking place over multiple months.

Nuclear weapons, by contrast, are both physically designed and organized in military doctrine to be used against large and usually pre-decided targets. They are not used in repeated short-loop cycles over campaigns of weeks or months. To put it another way, the military needs AI to help it find a needle in a moving haystack, pull that one needle out, attempt to snap it, and then determine whether it was snapped. The military does not need AI to find the haystack sitting in a field, set it on fire, and then know whether or not it burned down.

These non-nuclear visions of AI are shared by other nuclear powers, including China. Over the last decade, the People’s Liberation Army has pursued an “intelligentization” program that integrates AI as a decision aid for rapid attacks on the enemy’s “kill chain”; AI-coordinated swarming drones to overwhelm defenses; and cognitive warfare that uses AI to target human minds through information and cyberattacks. The PLA Information Support Force is building a “network information system” that uses AI, cloud computing, and big-data techniques to fuse data from operational units and create “dynamic kill networks” across air, land, sea, space, and cyberspace domains. AI is not, however, in charge of nuclear weapons. Indeed, in a study of over 9,000 AI-related requests for proposals published by the PLA, establishing its AI “wish list,” projects using AI for nuclear decisions did not feature once.

On the contrary, running through China’s military approach to AI is a goal to steer military operations from Beijing. Far from delegating all war choices to machines, Chinese leaders very much want machines that follow the orders of humans—specifically, that same decision-maker in charge of nuclear weapons, the general secretary of the Chinese Communist Party’s Central Committee.

AI Arms Races: Spend More, Feel Less Secure

If we truly want to support global peace and security, we should not focus on the cinematic threat narratives in which AI controls nuclear decision-making. Instead, the more realistic concerns arise not from the mix of AI and nuclear weapons but from interactions between AI and humans. The first of these concerns is that the promises of AI are now fueling an arms race more intense than previous arms races.

The current AI arms race is heightening feelings of insecurity. A 2026 report by the US Center for Strategic and International Studies summed up the consensus among policymakers and researchers this way: “Conventional wisdom holds that an AI arms race will define the twenty-first century and could be decided as early as 2030.” This viewpoint has even been codified into the highest levels of state doctrine. The second Trump administration’s National Security Strategy explicitly proclaims that AI “will decide the future,” echoing Russian President Vladimir Putin’s 2017 statement that whoever leads in this field “will be the ruler of the world.” In China, too, achieving global leadership in AI is a non-negotiable national priority; the country’s most recent five-year plan pledges to use “extraordinary measures” to realize that goal.

What is playing out is a classic security paradox: as nations accelerate their capital investment to secure a technological edge, they raise the stakes and concerns for their opponents, ultimately feeling more vulnerable than when the race began. In short, the more you arms-race, the less secure you feel. But, while arms races are nothing new, AI differs from previous military technologies in ways that introduce three additional layers of insecurity.

AI is considered a winner-takes-all technology, intensifying the arms race. First, AI may confer a decisive advantage at a smaller capabilities lead than other technologies. Historically, a nation could trail an adversary technologically and even quantitatively but still stay in the race. For instance, at the turn of the 20th century, the race for dreadnought battleships exacerbated tensions between Imperial Germany and Britain and became a contributing factor to World War I. Notably, though, the German navy never built as many dreadnoughts as the British. (Between 1908 and 1912, Britain launched 29 capital ships, while Germany launched 17: just under 59% of its opponent’s number.) Yet this disparity didn’t keep Germany from fighting the Royal Navy to a draw at the 1916 Battle of Jutland and maintaining the threat of a “fleet in being” for the rest of the war. So too in the nuclear age, China has maintained deterrence against the US with only 16% of the warhead count—approximately 600 warheads against the US arsenal of 3,700.

With AI, however, policymakers appear to perceive domination in binary terms: falling behind is equated with a total loss of strategic agency. Leaders would not tolerate having only 16% or even 59% of their opponent’s technological capability. On the flip side, a leader who believed their military AI capabilities were 85% better than those of their foe might feel invincible.

AI introduces a fear of falling behind permanently. Second, exacerbating insecurity even further, that perceived binary domination may prove real. At some point, an AI that darts ahead could conceivably become infinitely better than its competition, forever, potentially affording the leading nation a permanent decisive advantage. Regardless of whether this turns out to be true, it creates a fear of falling behind in a race where catching up feels impossible. Every technical milestone then becomes interpreted as a potential catastrophe. When the DeepSeek R1 model advanced beyond US expectations for Chinese LLMs, for instance, the discourse immediately framed it as a “Sputnik moment” for the United States.

Uncertainty about how best to use AI contributes to heightened concerns. Finally, AI’s versatility adds a layer of uncertainty about the smartest ways to use it. In past arms races, whether with ballistic missiles or battleships, the technology was largely uniform. The goal was simply to gain and deploy as many units as possible. AI, however, can diverge into radically different architectures and many more strategies. For instance, an article in “National Interest” warned that “America is running the wrong AI race,” by focusing on advancing frontier models, rather than on large-scale deployment of existing ones.

The overall result is that the AI age is creating a sense of extreme insecurity among many nations—an environment that is not conducive to peace.

The Cognitive Fog: Misperception and Miscalculation

A second concern about AI is its potential to add to the fog of war. Although AI is frequently marketed as a tool to provide clarity in complex analyses, it can also fuel misperception and miscalculation in multiple ways, potentially increasing the risk of humans deciding to go to war.

Militaries are using AI in sophisticated deception operations. First, as militaries integrate AI into their (conventional, not nuclear) battle plans, they are realizing they must also learn how to defeat opponents through new means of trickery. Recent PLA wargames have focused on how to “break intelligence,” preparing for battles in which AIs “work to distort each others’ reality.”

AI is being used for political deception. Second, AI is being used to dramatically scale up political disinformation campaigns. Conflicts in Ukraine, Gaza, and Iran have expanded to include what can be thought of as “LikeWar” battles to drive false information viral. Such campaigns involve automating information operations and creating high-fidelity deepfakes—tools that have already been used to successfully mislead heads of state, including those leading nuclear powers. US and Pakistani leaders have reacted to and pushed AI-generated imagery online. Studies on the present and future of deception operations suggest that this phenomenon will grow in scale and impact.

AI systems are vulnerable to errors, particularly in military contexts. The most dangerous and powerful kind of deception, however, is self-deception, which we can think of in both machine and human terms. AI undoubtedly brings incredible insights in various domains, often drawing on beyond-human analytical capabilities. Still, no matter how far the technology advances, these systems are plagued by issues that range from hallucination to algorithmic bias. Such problems, caused in part by training data that can never fully represent the real world, are especially salient in war. The civilian LLMs being brought into military systems are largely trained on the open internet—an objectively poor environment for high-stakes accuracy. Using military training data cannot solve the issue, since no two wars are the same. Datasets pulled from counterinsurgency operations in Iraq and Afghanistan, for instance, provide poor parallels for the conflicts of today and tomorrow.

A lack of understanding about AI could lead humans to make poor decisions. Yet, when it comes to miscalculation, the greater risk may lie with overly confident humans. History shows that the most dangerous phases of arms races are the earliest stages, when neither military nor political leaders yet fully understand the new weapons, and they make poor decisions based on erroneous assumptions. Before World War I, for example, the belief that new technologies like the railroad and fast-firing artillery gave a decisive advantage to the offense helped drive the quick march to war after the assassination of Archduke Ferdinand in Sarajevo. It turned out that these technologies in fact advantaged the defense, leading to four years of horrific stalemate in the trenches. A similar belief permeates discourse today—that AI rewards the side that strikes first (the offense) in every conventional war domain, from air strikes to cyberattacks.

It is more difficult to estimate opponents’ capabilities in AI than in other technologies. This risk is compounded by the challenges of understanding both one’s own capabilities and the enemy’s. Estimating power in the AI era is even more difficult than with traditional, kinetic weapons. Ships, tanks, planes, and even nuclear weapons can be counted, their physics understood, and their capabilities summed and compared. However, beyond estimating rivals’ data center capacity, understanding AI capabilities is far more difficult. Whether models can be accurately benchmarked is heavily contested. Even if accurate benchmarking were possible, how that would translate to battlefield performance would remain unknowable by humans or machines.

What we do know is that arms races traditionally incentivize exaggeration and fearmongering. Examples from the Cold War are the 1950s “bomber gap,” followed by the “missile gap”—Americans’ beliefs that the Soviet Union had achieved significant advantages in each technology. Both “gaps” turned out to be mythical, but they nonetheless contributed to the Cuban Missile Crisis. Similarly, both the George W. Bush administration and Iraq’s then-president, Saddam Hussein, issued claims about weapons of mass destruction that turned out to be nonexistent.

The outcome of direct interactions between opposing military AIs is unpredictable. Finally, the effects of interactions between AIs themselves create an informational void within military doctrine. In the past, adversaries generally understood one another’s concepts of fighting; with that awareness, they could deploy wargames and analyze recent conflicts to project outcomes. Because no nation will tip its hand regarding its true AI capabilities, and because military use of AI is relatively novel, the first time AI systems collide directly will likely be in a live, high-stakes environment where miscalculation is almost certain.

The Velocity of Catastrophe: Machine Speed

The third and final way in which AI is increasing the risk of war is through machine-speed operations. Yet, this is not about an AI making instantaneous decisions on nuclear strikes. Rather, it is about how AI is enabling a new generation of weapons that complicate humans’ nuclear decision-making.

AI is enabling weapons that compress the time window for humans to respond. The most notable examples of this phenomenon are boost-glide hypersonic weapons, including Russia’s Avangard, China’s DF 27, and America’s Dark Eagle. Despite their name, these delivery systems do not fly substantially faster than intercontinental ballistic missiles. What distinguishes them is that they use AI-enabled technologies, including adaptive control adjustments and cognitive and quantum inertial navigation systems, to make microsecond decisions and adjustments, while moving at thousands of miles per hour through denied airspace. By maneuvering around sensors and defenses, boost-glide weapons get much closer to their target before they are detected, allowing less time for humans to decide how to respond.

This “decision-time compression” represents the core of such weapons’ risk. Traditionally, Nuclear Command, Control, and Communications architectures provided a “decision window” of roughly 15–30 minutes to verify an incoming strike and weigh a response. To put this into context, the seemingly rushed time frame of “A House of Dynamite” spanned 18 minutes of deliberation, depicted by a 112-minute movie. A hypersonic weapon reduces the number of minutes for decision-making to single digits.

Shorter time windows often worsen human decision-making. The psychological reality is that humans make decisions poorly under stress and in short time frames, so anything that shrinks this window raises the likelihood of bad outcomes, such as a “Flash War.” There is a documented tendency for leaders to “lock in” on the first early concepts that enter the room—ideas that, in a more traditional crisis, might be aired out and debunked through hours of give and take. Historical experience confirms this; many of the most dangerous options considered by the US during the Cuban Missile Crisis were proposed during those frantic early days and then fortunately cast aside. AI-enabled weapons would not allow time for the same level of scrutiny.

Conclusions and Policy Recommendations

The notion that AI could start a nuclear war may be attention-grabbing. Yet research, grantmaking, and policy should be anchored in what is realistic rather than allowing the most dramatic narratives to steer the discourse disproportionately. The goal should be to understand and implement safeguards that tackle actual and likely risks, such as those posed by arms racing, misperceptions, and decision-making as described above.

Instead of pursuing purely symbolic measures to keep AI from nuclear weapons, we should prioritize reducing the incentives for and externalities of arms races, for example by finding ways to improve the defensive side of the equation. There may also be ways to reduce the likelihood of misperception, including through a concerted effort to enhance the education of political and military leaders about AI’s realities.

Finally, if we are to pursue effective arms control, our primary focus should not be on the speculative fear of AI launching a nuclear strike but, instead, on the real and growing number of physical platforms that shrink human decision-makers’ window for deliberation. For instance, prioritizing the regulation of hypersonic delivery systems—an existing technology that heightens the risk of nuclear catastrophe—is a more viable path toward strategic stability than chasing science fiction.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Peter Warren Singer is a Founder & Managing Partner at Useful Fiction LLC, a company specializing in strategic narrative, Strategist at New America, and a Professor of Practice at Arizona State University. A New York Times Bestselling author, described in the Wall Street Journal as “the premier futurist in the national-security environment” and “all-around smart guy” in the Washington Post, he has been named by the Smithsonian as one of the nation’s 100 leading innovators, by Defense News as one of the 100 most influential people in defense issues, by Foreign Policy to their Top 100 Global Thinkers List, and as an official “Mad Scientist” for the U.S. Army’s Training and Doctrine Command. No author, living or dead, has more books on the professional US military reading lists.

In 2017, Western media outlets warned that “Black Mirror is coming true in China.” The following year, Mike Pence claimed that “China’s rulers aim to implement an Orwellian system premised on controlling virtually every facet of human life—the so-called ‘Social Credit Score.’” So far, the CCP’s attempts at nationalized social scoring have remained fragmented and crude, largely due to difficulties in analyzing population-scale data. However, AI could soon lift that bottleneck, independently sifting through information and pulling out the most important details about every individual.

This unsettling prospect might renew fears about top-down social scoring by governments. However, an equally pressing concern is the potential for a bottom-up system, in which citizens choose to be surveilled and scored by AIs. As people integrate AIs into their lives to get more useful assistance with daily tasks, those AIs may soon be able to generate credible character assessments at the touch of a button. Early users who receive positive AI assessments may choose to share them with colleagues, businesses, bureaucrats, and so forth, in order to receive more favorable treatment. This dynamic would create an incentive for everyone else to follow suit.

This essay will explore why people will give AI assistants pervasive access to their lives and how this could soon translate into a form of social scoring. We’ll then map out how pressures to opt in will grow organically across every domain of life, creating a slippery slope toward self-imposed surveillance.

The Pressures Driving Self-Imposed Surveillance

The thought of sharing an AI judgment based on extensive personal data may sound too uncomfortable to believe that people would opt in. However, people already frequently give up their personal data to obtain benefits. In the US, over 21 million drivers voluntarily share driving data with insurers like Progressive and State Farm in exchange for discounts of up to 40%. Meanwhile, in China, voluntary disclosures have surged even as the country failed to implement a top-down, nationalized social credit system. In 2015, a private company called Ant Group launched Zhima Credit—an opt-in service that gives users social scores, and grants high-scoring users advantages from priority loan approval to dating site access. The platform claims to have more than 700 million authenticated users.

People are already sharing large amounts of personal data with AI for practical reasons. Some LLM power users are rushing to share their personal information with LLMs, connecting their agents to online accounts, medical records, and even bank information in hopes of obtaining more informed and wide-reaching assistance. Indeed, there are already wearable AI devices that can constantly record users’ lives, offering summaries of each day and making personalized plans for the next. From managing schedules to preserving an infallible, easily searchable memory of every conversation, AI agents are proving to be useful personal assistants in people’s busy lives. Adoption has already begun, and it is likely to expand.

Future AI assistants could provide attestations about their users. While people will initially share their personal data with AIs to get practical assistance, more capable future AIs could use this information for more than just helping with day-to-day tasks; given enough access, they could offer character references attesting that their users are reliable at work, committed as friends and partners, and honest in their financial and legal dealings. Once a user has granted their AI assistant wide-ranging access to their life, generating an assessment may be as simple as clicking a button.

After initial adoption, everyone else is under pressure to follow. Once early adopters disclose, others are likely to follow. This is due to an effect known as “unraveling,” which was described in two papers published by economists Sanford Grossman and Paul Milgrom in 1981. The logic is straightforward: those with the best track records have every reason to share them, since doing so distinguishes them from the crowd. Once the best disclose, everyone else looks worse by comparison, so the next-best tier discloses too. This cascade continues until nearly everyone has disclosed and silence itself becomes a red flag.

The transition to self-imposed surveillance could start small. While most people would balk at the end result of AI social scoring, it could nonetheless creep in gradually. The first use cases for AI attestation will be narrow and low-stakes: companies might let contractors attest to claims about previous projects, or dating apps might let users verify that their self-descriptions are accurate. Companies offering AI-based assessments might be widely disliked or little-used at first, but this would not necessarily block early adopters from opting in. FICO scores illustrate where this dynamic ultimately leads: no law requires you to have one, but opting out means losing access to housing, credit, and employment.

The Evolution of Self-Imposed Surveillance

Given the logic of unraveling, one might wonder why we don’t have complete self-surveillance and social scoring already. This can be explained by real-world frictions that make disclosure more difficult and less rewarding than it would be in theory. AI may soon remove these frictions, enabling unraveling not just in narrow domains such as driving and personal finance, but across every sphere of life.

We do not see full disclosure yet because there are frictions that block unraveling. Today, meaningful attestation often costs real effort, from assembling a job application to sharing references. This creates enough friction that a lack of full disclosure does not necessarily look suspicious.

Additionally, attestations are not always credible: landlords can overstate the quality of a rental, job applicants can embellish their qualifications, and there is often no practical way to check. This reduces the value of disclosure, since it does not reliably distinguish those with the best credentials from everyone else. Indeed, the domains where we already see unraveling are the narrow areas in which disclosure is both costless and credible. A FICO score captures financial behavior, and a telematics device captures driving behavior—information that is cheap to measure and difficult to fake.

Previous technological revolutions, such as the internet, sparked concerns about surveillance and social scoring. The rise of digital banking, health apps, online calendars, and social media means that large amounts of sensitive personal data is stored online. Yet we have not seen waves of disclosure across every area of life. This is because the internet does not in fact make disclosure costless and credible across all domains. On cost, analyzing vast sums of internet information to draw out valuable insights about every individual is still an intractable technological challenge. On credibility, the public internet is still a far cry from comprehensive, real-time surveillance of behavior. People can curate what they upload online, and this reduces trust that it is representative.

AI may make costless, credible disclosure dramatically easier. AI products like ChatGPT already store usage data that can speak to their users’ work competence, behavioral tendencies, and personal preferences. It could be very cheap and convenient to share such a profile (or a redacted summary generated by a trusted third party) with an employer or landlord.

Credible disclosure requires two things: comprehensive coverage of someone’s behavior, and the ability to draw accurate conclusions from that behavior. AI is making rapid progress on both fronts, particularly on coverage: heavy users already spend dozens of hours per week interacting with AI chatbots, and that coverage will only grow as companies roll out features to further personalize and integrate AI into daily life. Multimodal, always-on hardware like smart glasses and earbuds could grant AI assistants constant audio or visual access, and produce attestations far more credible than anything text-based interaction can support.

Societal pressures could make people accept comprehensive self-surveillance. Even as comprehensive surveillance becomes technologically feasible, people might feel uncomfortable about allowing their AI assistants to provide character assessments to others. Sharing driving data is one thing; sharing a judgment drawn from every detail of how one spends one’s day, from drinking habits to private political conversations, is quite another. Yet there are reasons to believe that resistance might yield surprisingly quickly.

Those who allow greater access will receive more credible attestations. Initially, users may try to game the system by granting AIs only selective access: interacting when they’re being productive and setting the AI aside when they’re not, or filtering access that may paint them in a bad light. For one thing, this would be a difficult strategy to sustain as AI assistants become increasingly useful and perceptive. Additionally, evaluators will lend more weight to attestations drawn from more comprehensive data, increasing the pressure to provide near-total access to AI assistants.

The pressure for AI attestations may extend to personal domains such as dating. Many people already consult AI agents for dating advice. If people come to view AIs as trusted judges of character, they may start requesting attestations from potential partners before agreeing to a date. Those who refuse would face a narrower pool of willing partners, extending the unraveling dynamic into intimate life.

Surveillance at the Civilizational Level

At this point, one might hope that data protection or anti-discrimination laws could prevent omnipresent observation. However, this is the insidiousness of self-surveillance; privacy legislation can help to prevent non-consensual surveillance by governments or businesses, but it cannot stop individuals from opting in to disclosure themselves. Unraveling can therefore happen anywhere.

Self-imposed surveillance can creep in under various political conditions. Different political systems will arrive at the same destination through different mechanisms: in the US and EU, the private sector will provide the infrastructure for surveillance. In China, the story is different: the government sidelined Zhima Credit because it wanted to control the infrastructure itself, evidence that surveillance in China may continue to trend toward mandatory top-down surveillance. The result may be the same: pervasive monitoring.

Conclusion

The future may involve substantially less privacy than the present. Throughout most of human history, privacy as we know it did not exist; our ancestors lived in small bands where reputation was built through direct mutual observation and gossip served as the original social credit system. The high-privacy society we inhabit today is a side effect of urbanization and the limited reach of pre-digital technology. As AI systems close that gap, we may be returning to the historical default. Societies tend to adopt values compatible with their technological constraints: the concept of intellectual property was meaningless before the printing press, and privacy may prove similarly contingent. Whether we accept this transition or resist it is an open question, but the forces driving it are already in motion.

While the direction of this trend seems robust, the form it takes is not predetermined. The question of who controls the AI attestation infrastructure matters enormously for how power is concentrated in the future: a world where attestation is managed by a handful of AI companies looks very different from one where it is controlled by governments, and different again from one built on decentralized protocols. Similarly, whether norms develop around narrow, domain-specific attestation or comprehensive behavioral transparency will determine how much power the system concentrates and in whose hands. These are important path dependencies, and the decisions shaping them are being made now.

Thanks to Dan Hendrycks and Devin Kim for formulating the premise of this piece.

‍

See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.

Steven Veld is an AI strategy and governance researcher, with a particular focus on scenario-based forecasting for multipolar AI futures. He was a 2025 policy fellow with the Institute for AI Policy and Strategy (IAPS), where he worked on Congressional engagement with the AI Policy Network. Before that, he was a ML Alignment and Theory Scholars (MATS) fellow with the AI Futures Project. He has a BS in Computer Science at UCLA, and has prior experience working on biosecurity and compute governance.