During the Cold War, humanity somehow managed to survive the invention of a technology capable of ending civilization. That survival was never guaranteed. Historians have documented enough misunderstandings, false alarms, equipment failures, and moments of political brinkmanship to make anyone wonder how close the world came to catastrophe. Yet the missiles remained in their silos. The bombers stayed on the ground. The apocalypse everyone feared never arrived.
It is tempting to attribute that outcome to restraint or wisdom. The truth is probably less flattering. The nuclear age eventually produced an incentive structure that made launching a first strike almost irrational. If one side attacked, the other would respond, ensuring that victory and defeat became nearly indistinguishable. Mutually Assured Destruction was a grim doctrine, but it provided something every dangerous technology requires: A brake pedal.
Reading Anthropic’s proposal for a coordinated pause in frontier AI development, I found myself wondering whether artificial intelligence will ever discover its own equivalent.
The immediate reaction from many observers was entirely predictable. Anthropic already occupies one of the leading positions in the AI race. Asking everyone to slow down naturally invites accusations that the company is trying to preserve an advantage it has already earned. History offers no shortage of examples in which incumbents embraced regulation once they had become large enough to benefit from it.
There is enough truth in that criticism that it cannot simply be waved away. Companies rarely advocate policies that undermine their own interests.
But motive and message are different things. A company can be acting in its own interest while simultaneously identifying a real problem. The possibility that Anthropic benefits from a pause does not automatically invalidate its warning, just as a pharmaceutical company’s profit motive does not automatically invalidate the effectiveness of a medicine.
For the sake of argument, then, suppose the warning deserves to be taken at face value.
Anthropic’s concern is that advances in frontier AI could eventually reach a point where systems become capable of improving successor systems with diminishing human involvement. Whether one calls that recursive self-improvement or simply accelerated automation of AI research is less important than the implication. Progress that currently unfolds over years could begin unfolding over months or even weeks, leaving governments, regulators, and perhaps even the companies building these systems struggling to keep pace.
If that possibility ever became credible, the instinct to slow down would be understandable.
The difficulty is that no one can slow down alone.
Anthropic acknowledges this openly, and in doing so identifies what may be the central governance problem of the AI era. A unilateral pause is not a pause at all. It is surrender. Any company that voluntarily stops developing frontier models while its competitors continue would almost certainly lose its position. Capital would migrate elsewhere. Engineers would follow opportunity. Customers would gravitate toward more capable systems. The market would punish restraint and reward acceleration.
National governments face precisely the same logic.
Artificial intelligence is increasingly viewed as strategic infrastructure rather than simply another technology sector. The nation that develops more capable systems first could enjoy advantages across scientific research, industrial productivity, cyber operations, military planning, intelligence analysis, and economic competitiveness. Even leaders who privately believed a coordinated slowdown would benefit humanity would still have to explain to their citizens why they accepted permanent strategic disadvantage while trusting rivals to honor the same agreement.
Trust has always been a scarce resource in international affairs.
That reality makes the comparison to nuclear arms control both useful and incomplete.
The nuclear era eventually developed elaborate verification mechanisms because verification was possible. Satellites could observe missile fields. Intelligence agencies could monitor enrichment facilities. Test explosions left unmistakable signatures. Treaties were imperfect, but physical reality made deception difficult over long periods.
Artificial intelligence is different. A frontier training run can occur inside a commercial data center that outwardly resembles thousands of others. Compute can be rented instead of owned. Research can be distributed across multiple facilities and jurisdictions. The evidence that a breakthrough has occurred may consist of nothing more than software running on hardware that already existed yesterday.
How does one verify that a nation is not training a model it has every incentive to conceal?
How does one inspect a cloud provider operating across dozens of countries?
How does one prevent a startup, a state-sponsored laboratory, or an intelligence agency from deciding that the rewards of cheating outweigh the risks of being discovered?
The more one thinks about enforcement, the more elusive it becomes.
Ironically, the technology that may demand the greatest level of international cooperation humanity has ever attempted may also be the hardest technology humanity has ever tried to regulate collectively.
That is where the analogy to the Cold War begins to break down.
Mutually Assured Destruction worked because no one could plausibly imagine winning. Launching first guaranteed retaliation. Retaliation guaranteed devastation. The absence of a victor became the foundation of stability.
Artificial intelligence offers no comparable equilibrium.
Every participant still believes victory is possible.
Every laboratory believes it may build the breakthrough that changes the industry. Every nation believes it may secure a lasting strategic advantage. Every investor believes one company may emerge as the defining enterprise of the century. The incentives do not point toward restraint. They point toward acceleration.
Perhaps the AI equivalent of Mutually Assured Destruction is not mutually assured destruction at all.
Perhaps it is mutually assured irrelevance.
The nation that pauses while another advances may not disappear, but it may find itself living in a world whose economic and technological center of gravity has shifted elsewhere. The company that slows development while competitors continue may survive, but as an observer rather than a leader. For organizations and governments accustomed to competing for influence, that prospect may feel almost impossible to accept.
Which returns us to Anthropic’s proposal.
The difficult question is not whether a pause would be desirable under certain circumstances. Most reasonable people would agree that if a technology appeared to be moving beyond our ability to understand or govern it, taking time to assess the situation would be prudent.
The difficult question is whether civilization possesses the political architecture required to make such a decision collectively.
History provides reasons for skepticism.
Markets reward first movers. Nations reward strategic advantage. Political leaders answer to domestic constituencies rather than global interests. International institutions move slowly by design, while technological competition rewards speed. We have built remarkably effective systems for encouraging innovation and competition. We have devoted far less effort to designing institutions capable of asking everyone to stop at the same time.
Perhaps none of this will matter. Perhaps recursive self-improvement will prove slower and more manageable than today’s forecasts suggest. Perhaps AI will deliver extraordinary advances in medicine, science, engineering, and education without forcing a fundamental reconsideration of how humanity governs technological progress.
I hope that proves to be the case.
But if it does not, the challenge before us may have less to do with artificial intelligence than with ourselves.
Over the past year, I have repeatedly written that AI is beginning to reshape assumptions about work, wealth, infrastructure, and even the social contract. The more I consider those changes, the more convinced I become that governance belongs on that list as well. We are creating a capability whose consequences will ignore borders while expecting institutions built around borders to manage it.
That mismatch may become the defining policy challenge of this century.
We often assume humanity’s greatest test will be whether we can build intelligence greater than our own.
I am beginning to wonder whether the real test is simpler but much more difficult.
Can eight billion people, nearly two hundred nations, and thousands of competing institutions agree to slow down when every instinct, every market signal, and every lesson of history tells them to keep accelerating?
If the answer is no, then perhaps the most dangerous capability artificial intelligence will ever possess is not recursive self-improvement.
It will be exposing that human civilization never developed a governance model capable of managing a technology this powerful.
And if that is true, the greatest challenge of the AI era will not be teaching machines how to think.
It will be teaching ourselves how to act together before competition makes the decision for us.
