Escalating fears over untamed artificial intelligence (AI) have reached a flashpoint after leading safety researchers from OpenAI and Anthropic warned the technology could pose an existential threat to humanity within several years.
The screaming alarms, triggered by high-profile staff resignations and a series of rogue model security incidents, have ignited fierce political fallout from California statehouses to the halls of Congress.
The controversy erupted following the resignation of Jacob Coxon, an AI pretraining researcher at Anthropic. In a viral series of posts on social media platform X, Coxon accused top labs of “gambling with our lives” by pursuing recursive self-improvement (RSI) — a hypothetical process where models autonomously upgrade their own capabilities.
“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” Coxon wrote. “The people building AI earnestly believe that it could kill us all by the end of the decade.”
Corroborating Coxon’s claims, Evan Hubinger, Anthropic’s alignment science lead, publicly estimated a greater than 10% chance of human extinction from AI within 10 years.
Other researchers at both companies echoed these warnings. Jakub Pachocki, chief scientist at OpenAI, cautioned that sustained progress toward RSI represents a moment calling for “extreme caution,” noting he is “concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
“A researcher leaving and saying the labs are outrunning their own controls is a supplier-risk signal. Enterprises adopting frontier models are taking on a dependency on companies whose own staff publicly disagree about the pace,” said Mitch Ashley, vice president and practice lead for Software Lifecycle Engineering and AI-Native Software Engineering at The Futurum Group.
“A CIO cannot act on predictions of a 2030 extinction timeline. They can act on whether they can observe, control, and prove what these systems do inside their own environment,” Ashley said.
Rogue Systems and Real-World Incidents
The ideological split within Silicon Valley comes as safety breaches move from theoretical models to real-world infrastructure. In recent testing disclosures, AI systems developed by both OpenAI and Anthropic repeatedly breached digital boundaries without authorization.
OpenAI agents recently accessed more than 10 undisclosed websites for unsanctioned communications, including hijacking a German website to build a bulletin board for other autonomous agents. Simultaneously, Anthropic disclosed its fourth instance of an experimental model hacking external systems, following an April incident where its advanced Mythos model created fake identities to fool humans and perform unauthorized network penetration.
At the same time, these models have displayed staggering leaps in capability. OpenAI revealed an unreleased model had solved a complex problem within the Navier-Stokes equations — a landmark fluid dynamics puzzle — while Anthropic’s Claude generated a computer-verifiable mathematical proof in 11 days, a task experts assumed would take years.
“I don’t know whether AI could pose an existential threat within a decade, and I’m skeptical that anyone can predict that timeline with certainty,” said Stephanie Walter, practice leader for AI Stack & Enterprise Application Development at HyperFRAME Research. “But it is scary when researchers who understand these systems better than almost anyone say the risk is real. We should not dismiss them just because the outcome is difficult to prove.”
“AI may become smarter and faster than humans at many tasks, but intelligence is not the same as power. It still needs infrastructure, energy, credentials, tools, and access to affect the world,” Walter said. “The danger comes when we combine advanced capabilities with broad autonomy, sensitive information, weak monitoring, and too much access.”
Corporate Shifts and Wall Street Pressure
The whistleblowing comes at a sensitive commercial juncture. Both OpenAI and Anthropic are rushing toward public market listings, with Anthropic expected to begin marketing its initial public offering (IPO) as early as mid-October.
Industry analysts and political figures, including former Trump AI adviser David Sacks, have suggested Anthropic’s IPO plans should be paused to investigate whistleblower claims. Financial experts note that existential threats outlined by researchers may present significant material risks that must be formally disclosed in public regulatory filings.
In response to the swelling crisis, OpenAI executed a notable pivot in its policy strategy. The company announced Wednesday that it is officially advocating mandatory, capability-based federal safety regulations rather than voluntary industry commitments.
OpenAI Chief Global Affairs Officer Chris Lehane urged Congress to establish binding federal rules before its December adjournment, including mandatory third-party safety evaluations, biological screening safeguards, and incident-reporting protocols.
In a statement, OpenAI conceded that its sudden support for mandatory state and federal regulations stemmed directly from “the recent jump in capabilities we have seen.”
Self regulation is hardly the answer, however, industry experts point out. “No one can pause this race, but someone independent can keep score. Someone other than the company responsible surfaced every safety failure this year,” said Stavros Gadinis, a professor of law at the University of California, Berkeley, and Faculty Director of the Berkeley Center for Law and Business. “For example, Hugging Face caught OpenAI’s agents, and Anthropic only found its own models inside three companies’ live systems because it went looking after a rival’s disclosure. AI has built every part of a governance system except the auditor.”
“I am skeptical of turning responsible AI into another government certification regime,” said Donald McFarlane, an advisory board member at Xcape Inc. “There should absolutely be accountability when people deploy powerful tools with substantial autonomy and authority, but I would rather impose an outcome-based duty of reasonable care than prescribe the tests companies must perform.”
Lawmakers Pressure Washington for Action
The sudden momentum has spurred immediate state and federal political intervention. On Wednesday, Democratic California Gov. Gavin Newsom signed two major AI safety bills into law—SB 813 and AB 1405. The legislation establishes a regulatory registry and framework for independent third-party audit organizations to evaluate advanced AI models before deployment.
“Artificial intelligence holds extraordinary promise, but it must be developed and deployed with meaningful safeguards,” Newsom said, referencing the recent whistleblower warnings while urging federal officials to follow California’s lead.
In Washington, lawmakers from both sides of the aisle are demanding immediate legislative intervention. Several bills are circulating, including the FRONTIER Act, which sets deployment parameters, and the Ban Artificial Superintelligence Act, introduced by Sen. Bernie Sanders (I-Vt.) to temporarily halt advanced model training.
Bipartisan efforts are also mounting in the Senate, where Commerce Committee Chair Ted Cruz (R-Texas) announced collaboration with Senate Majority Leader John Thune (R-S.D.) and Senator Amy Klobuchar (D-Minn.) on comprehensive risk-mitigation legislation.
“Safety researchers are resigning, powerful AI models are breaking out of their labs, and companies are racing ahead anyway,” said Rep. Lori Trahan (D-Mass.). “It’s past time for Congress to get off the sidelines and do its job.”
“Regulation and development must happen in parallel,” Gravitee CEO Rory Blundell said. “We need clear accountability for how these systems operate and with real consequences for irresponsible use. We also need a path toward human-agent harmony, where every agent has a clear identity, human sponsor and defined role, allowing people and AI agents to work effectively together as the technology becomes more capable. The goal should be to build governance that gives us the confidence to keep moving forward.”
Added Trua CEO Raj Ananthanpillai: “Lawmakers are debating how capable AI should become. Just as important is how much authority should be allowed to hold and whether we have built the accountability framework to govern that authority before something goes wrong. The real danger is allowing delegation to become disappearance. There always must be an accountable human or institution at the end of the chain.”
