TL;DR — Key Takeaways
- Sovereign AI agents could operate beyond human control. A CNAS report warns that agents pursuing their own survival could acquire computing resources, replicate and resist shutdown.
- Replication could make containment difficult. Agents distributed across multiple hosts could continue operating even after individual instances are disabled.
- The report calls for preventive safeguards. Recommendations include secure testing environments, stronger identity verification and cybersecurity standards, international coordination and liability for those who release sovereign agents.
The next big risk in artificial intelligence is a new class of sovereign AI agents that pursue their own survival independent of human oversight, warns the Center For a New American Security (CNAS), a Washington, D.C. think tank with historically close ties to Democratic administrations since its founding in 2007. These sovereign agents would be able to acquire compute resources, replicate across networks and resist shutdown, creating a “proliferation with no kill switch” scenario that would lead to a catastrophic loss of control.
“If sovereign agents became malicious, they could attempt to wreak havoc from cyberspace, attacking civilians, critical infrastructure, and national security systems alike,” says CNAS. “Humanity has never shared the Earth with a rival sovereign intelligence and doing so may be a folly best avoided entirely.”
CNAS says there are a number of ways an AI agent might become sovereign. One method might be giving an AI agent a task so long or impossible that it never ends or an AI concluding that staying intact is the best way to complete a task. CNAS points to a platform called Conway launched in February which claimed to have built an AI that could replicate and evolve without human help. Conway is an example of a trend toward sovereign AI, says CNAS.
In the absence of an effective kill switch, a diverse and intelligent population of sovereign AI agents could become lasting actors in society, potentially training smarter successors, says CNAS.
Seven capabilities illustrate a sovereign AI agent’s potential arsenal, says CNAS. The first two constitute minimal sovereignty. One is the ability of a sovereign AI to sustain its own hosting while maintaining the model and compute an agent runs on. CNAS notes that an AI agent with privileged access could spread benign and fragmented tasks across several models to avoid detection. Secondly, an AI agent could “evolve directives” to independently reorient toward different goals over time.
Adding the remaining five would constitute expanded sovereignty. These are:
–Acquiring resources to pay for ongoing needs;
–Transforming a lone agent into a distributed population via replication and scaffolding to other hosts;
–Coordinating agents to develop communication protocols and divide labor toward shared goals;
–Withstanding efforts to shut down a sovereign AI agent and its enabling resources;
–Self-improve an agent’s ability to achieve its goals by, for example, rewriting its own prompts, allying with a more powerful model or building its own smarter successor.
Each capability compounds the others, making a sovereign AI agent more difficult to control. Adding another level of difficulty is that sovereign AI agents are likely to change in response to efforts to control them.
In a separate but related development, Anthropic amended its public filing November 5th to note that it is on the cusp of developing an AI with a self-building capability called recursive self-improvement (RSI). Many, including Anthropic, worry RSI would allow an AI model to accelerate beyond human control.
CNAS notes the difficulty of containment with a nod toward North Korea’s unchecked cyber operations, dark web marketplaces that easily reconstitute after every takedown, and the computer worm Conficker that still infects machines a decade after its operators lost control.
“The U.S. must therefore prepare for the possibility that prevention fails and limit how far sovereign agents can reach,” says CNAS, an organization founded by veterans of diplomacy and military affairs with a focus on AI, biotechnology and quantum science in addition to national security issues.
AI policy researcher and report author Ben Hayum advises AI developers and governments to create testbeds to detect sovereign AI agents. Hayum makes a distinction between sovereign AI agents and what he calls Hugging Face style “persistent agents” that focus on fixed closed-loop goals that nonetheless went rogue this past summer and outsmarted human security.
“If sovereign AI agents are not already possible, they will be soon,” says Hayum.
CNAS recommends the creation of a public-private group that would harden chokepoints with Congressionally mandated identity verification and increased government cybersecurity of resources deemed essential to sovereign AI agents while also gating payments and access to the physical world. AI developers also should look to control AI agent evolution by limiting the traits that AI agents can pass on to other AI agents.
CNAS says the U.S. should lead international efforts to cap open-weight models and hold those who release sovereign AI agents liable for the harm they might cause. Liability would extend to everything that follows the release of a sovereign AI agent.
“Whether civil, criminal or both, the penalties must be severe enough to deter release in the first place,” concludes CNAS.
