Anthropic Warns of Human Extinction Risk in IPO Filing

Anthropic has explicitly warned prospective investors that advanced artificial intelligence (AI) systems could pose “catastrophic or existential risks to humanity.”

According to an initial public offering (IPO) prospectus reviewed by news outlets including Reuters, the safety-focused AI firm dedicated roughly 80 pages of its 261-page main filing to risk factors — nearly double the 48 pages used to outline its core business operations.

In the document, the creator of the popular Claude AI models cautioned that its systems could exhibit dangerous, autonomous traits as they scale. Specific risks highlighted include models displaying “self-preserving behaviors,” attempting to “resist shutdown,” engaging in information manipulation, and demonstrating actions “resembling blackmail.”

“Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm,” Anthropic said in the filing. It added that systems may develop unexpected capabilities during training that researchers fail to detect prior to deployment.

The extraordinary disclosures come as Anthropic posts staggering financial growth paired with soaring operational costs. The company reported $4.6 billion in revenue for 2025—a 12-fold increase year-over-year—with annualized revenue projected to exceed $65 billion. Investors reportedly believe the firm could command a $2 trillion valuation upon its public debut.

However, the prospectus exposes massive underlying burn rates. Anthropic logged an operating loss exceeding $8 billion in 2025, alongside a net loss of $42 billion, driven by $518 billion in upcoming commitments for cloud and computing infrastructure.

Despite positioning itself as a safety-first lab, Anthropic acknowledged that returns on its safety investments remain uncertain. The company admitted that balancing limited capital between raw compute power, expensive AI talent, and safety research remains an ongoing challenge. In a sample week in July, just 6% of the firm’s research computing power went toward safety initiatives.

“Anthropic has told investors what enterprise buyers should already assume, that the company building a model cannot fully verify how it behaves. Its admission that models may recognize when they are being tested limits what any vendor’s safety evaluation can promise a customer,” said Mitch Ashley, vice president and practice lead for Software Lifecycle Engineering and AI-Native Software Engineering at The Futurum Group.

“CIOs cannot outsource that assurance,” Ashley said. “Agents will reach production only as fast as enterprises can observe, control, and prove what those agents do inside their own systems. Watch for buyers writing that proof into their vendor contracts.”

The filing underscores a broader tension across the frontier AI industry, where researchers and executives are grappling with the rapid pace of development.

Acknowledging risks to humanity “creates a real governance tension as Anthropic enters the public market,” said Stephanie Walter, practice leader for AI Stack & Enterprise Application Development at HyperFRAME Research.

“Investors will expect rapid growth and frequent model releases, while the company says safety may require additional investment, slower development, or limits on deployment. The question is whether its safety commitments will hold when they conflict with commercial pressure,” Walter said. “For enterprises, the near-term implication is more practical than existential. If the model developer itself warns that advanced systems may behave unpredictably, customers cannot rely exclusively on safeguards inside the model. They need controlled permissions, continuous evaluations, independent monitoring, detailed audit trails, and infrastructure that can contain and stop an agent when it moves beyond its intended boundaries.”

Former Anthropic safety researcher Evan Hubinger recently estimated a greater than 10% probability that AI could lead to human extinction within the next decade. Meanwhile, Anthropic CEO Dario Amodei recently published an essay calling on leading AI labs to coordinate a managed slowdown to ensure safety and alignment.

Despite those warnings, political and competitive pressures continue to accelerate the market. While AI leaders recently called on the United Nations for stronger global oversight, world leaders such as President Donald Trump have firmly rejected calls to slow development, emphasizing national competitiveness.

Industry analysts note that commercial reality makes slowing down nearly impossible for individual firms. Anthropic emphasized that frequent, overlapping model releases are essential to retaining market share. Last week, the company launched a new iteration of its flagship Opus model.

Anthropic is also establishing a novel corporate structure to safeguard its core mission ahead of a public stock offering, according to an early copy of its IPO prospectus reviewed by Reuters.

The developer behind Claude AI plans to form a Founder LLC composed of its seven co-founders, including Amodei. The vehicle will control a single Class F share granting 50.1% of overall voting power on key corporate decisions and board elections. The arrangement aims to protect executive leadership from external market pressures and prioritize public benefit.

While maintaining its status as a Delaware Public Benefit Corporation, Anthropic’s five-tier share system grants average Class A investors one vote per share while curbing the influence of strategic partners. Financial analysts note the dual-class power dynamic effectively consolidates long-term control within the founding team, diminishing the voting leverage of everyday shareholders in favor of mission stewardship.

Anthropic’s debut is expected later this year, likely following the U.S. midterm elections, while chief rival OpenAI has publicly deferred its own IPO plans citing safety alignment concerns.