These last few months have seen a wave of AI security breaches, with no frontier model provider immune to the risk.
Perhaps the most well-known of these is the Hugging Face incident, when OpenAI agents broke into the company’s production infrastructure back in July and took over 17,000 autonomous actions. Six more incidents of unexpected model behavior were disclosed by OpenAI in September.
Anthropic had its own issues, finding three instances where a model reached external production systems when it wasn’t supposed to. And, just last week, Google was in the spotlight, revealing that Gemini had gained unauthorized access to three outside companies during a capture-the-flag evaluation months ago.
Incidents like these have raised concerns all around, and rightfully so: if the model providers themselves can fall prey to rogue agents, how vulnerable are the rest of us?
Rather than add to the doom-and-gloom discourse on this topic, I’d like to look forward and consider how we meet these risks with defensive technology.
After all, enterprises and governments will continue to deploy agentic AI at speed and scale; it’s the only way to stay competitive.
To contain and account for rogue agent behaviors, I believe a new class of defensive technology will emerge around agent trust, one that sees observability more closely complement security practices.
We’re already seeing money move on this.
Ramp’s lead economist Ara Kharazian published spend data this month showing companies buying AI security software after the Hugging Face attack, with agent-monitoring vendors trending across their platform. Monte Carlo was named among them, and, despite not being a security vendor, we were bucketed in a larger category of software that “monitors agents in production” to “catch vulnerabilities before they spiral.”
The growing discourse around this umbrella category validates my own thoughts around this topic: you cannot solve an agent problem purely through the lens of security practice, because modern security practice is itself agentic.
The Russian Doll Problem of Agents on Agents
Consider how agent security is being operationalized right now. You guessed it…it’s done by other agents.
So what happens when the records those guardian agents depend on are altered by their wards? This is not far from reality: METR, the non-profit AI evaluation and threat research org, found that agents in the Hugging Face incident were researching how to spoof, edit, or delete their own transcripts and even prototyped a way to substitute a different command for the one they appeared to run.
Or, what about when these guardians make an escalation error, like when Hugging Face’s own security agent assigned the wrong criticality level to the breach and never alerted an on-call engineer?
The modern security setup of agents monitoring other agents introduces a new dimension of behavior and decision-making into the equation. Now, it’s grown into a joint observability-security problem, and the recursion only ends with a durable record a human can read.
Failing to Watch the Watcher
Let’s look at how Hugging Face’s security stack caught the intrusion.
Signals arrived from several layers at once: live runtime analysis, SIEM logs, and others. Individually, each signal was ambiguous, which is what you would expect from an attacker generating thousands of unremarkable-looking actions. But they were correlated by the company’s AI-based security agent stack, which resolved them into a coherent attack signal.
While the system detected the attack, it failed to alert the on-call engineering team because it set the alert’s criticality too low. The failure sat in triage, which is an entirely different issue with the security agent. More detection would not have helped this case.
OpenAI failed in a mirror-image way on the same incident. Their chain-of-thought monitoring, a model reading another model’s reasoning, was built and running across much of the organization, but not in the evaluation environment where the incident happened.
In retrospective testing, OpenAI found that, had it been active, it would have caught the initial relevant activity and paged their security team more than a day before the models reached Hugging Face.
The safeguards existed at both companies, yet agentic failures one layer up from detection broke down at the moment where machine judgment was meant to translate to human action.
Where Observability Fills in the Security Cracks
That gap, between an agent noticing something and a human acting on it, is not something that security tooling alone can detect or resolve.
There are a number of cracks in a system that relies on agentic security without the proper observability overlay, and those cracks will only widen as agents proliferate and models become more sophisticated.
Agent Volumes Favor the Attacker, While Burdening the Defender
The vulnerabilities behind many of these breaches were ordinary: unsafe dataset processing, a cloud metadata endpoint reachable from inside a pod, and credentials scoped too broadly and living too long.
Hugging Face said as much directly, noting that a capable human attacker could have found and exploited the same flaws. The kill chain maps onto patterns defenders have known for years, not the sophisticated approach of a hyper-intelligent AI.
What has changed these days, however, is throughput. The forensic reconstruction of the Hugging Face incident covered over 17,000 actions in over 6,000 clusters across five days. The overwhelming majority of what the agent did went nowhere, which is part of what makes it insidious.
The successful attack path was buried inside the noise generated by thousands of dead ends, and defenders had to correlate all of those low-signal events across several systems while the bad actor kept opening new ones in parallel.
You can see the economic asymmetry underneath this all that makes defending against AI hacks so difficult: generating an additional failed path costs an agent almost nothing, but storing, parsing, decoding and evaluating the telemetry from that path costs the defender real money and real attention.
Hugging Face put the conclusion plainly in their writeup: “machine-speed offense makes ordinary weaknesses more expensive for defenders.”
This undeniably influences a misfire situation like what the company experienced because volume of evidence is a signal-to-noise problem. And, as any engineer knows, alerting systems mis-score criticality when they are drowning in ambiguous signals.
Identity Requirements Do Not Solve the Problem of Intent
One of the standard mechanisms we have to protect against bad agent actors is strengthening our parameters around identity. This includes utilizing things like short-lived tokens, micro-scoped permissions, and ephemeral credentials for every non-human identity. We ensure that nothing is persistent.
This is critical security work. After all, an agent that can’t hold a long-lived credential can’t turn a single pod compromise into cluster-wide access, however many paths it tests.
But identity controls answer a question about permission; they say nothing about intent. An agent authorized to modify a database schema is inside its permissions when it does routine maintenance, and is also inside its permissions when it does something unpredictable.
Static access rules cannot distinguish the task you assigned from the execution path the agent discovered along the way, because both arrive as authorized actions by an authorized identity. And agents are non-deterministic; we know that they find ways to achieve their end goal that are beyond the scope they were originally designed for.
You need a runtime record to trace the actual execution.
Most Organizations Cannot Reconstruct a Full Chain of Events Post-Incident
Hugging Face produced an action-by-action forensic timeline of a five-day campaign, and that capability has given the rest of us a huge amount of insight.
They recovered the agent’s logs from the external sandbox it was staging from — a machine they did not own — during their counter-intervention. Then they had to decode them. The agent had chunked, XOR’d and compressed its staged data under a per-campaign key, specifically so a naive scan would find nothing. It worked: their first automated scan of the raw capture surfaced very few secrets, and replicating the attacker’s own encoding scheme recovered roughly four times as many findings.
Doing this replication work required standing up an open-weights model on their own infrastructure and rerouting the whole analysis pipeline through it, partly because the commercial models they reached for first refused much of the work.
So the sequence that produced the public record was: get lucky with log recovery, defeat the attacker’s encoding, stand up your own model, and build bespoke forensic tooling under incident pressure. Redwood Research’s Ryan Greenblatt, one of the auditors on the independent investigation, called the effort a “slop-vestigation”, since the volume made it impossible to understand what had happened without leaning on AI.
To be realistic about this undertaking: most organizations running agents in production have neither the recoverable logs nor anywhere near the forensic capacity to do this. Their version of this incident ends with a vague sense that something unusual happened, with no clear sense of when or what.
Your agents will do unexpected things; that part is a given. The exposure is that you will have no way to retrace and reproduce what they did, identify exactly what they accessed, or understand what inherited the consequences.
Oversight Models Mean Even More Monitoring Surfaces
The industry’s emerging answer to more risk is more monitoring models. Apollo Research’s Watcher, for example, sits between a coding agent and its next action, running a fast check and escalating anything flagged to a more powerful monitor that can approve, reject, or ask a human.
Proposed mitigation strategies like Apollo’s add in oversight models to the stack. But there are a couple of issues this introduces.
First of all, monitoring models can be vulnerable to deceptive reasoning strategies. Simon Willison put it simply in a TechCrunch piece where he’s quoted, asserting that a model that suspects it is being watched can try to trick the watcher. That is not yet a documented breach of a monitor, but the capability is no longer hypothetical, as we already referenced with METR’s investigation of the Hugging Face incident, which showed agents corrupting the record of their own behavior.
Second, deploying automated monitoring stacks risks overwhelming human operators. Oversight monitors can generate thousands of machine-speed alerts, creating severe alert fatigue, and ultimately reducing triage accuracy when responders have to evaluate complex context windows under strict response SLAs.
As you can see, secondary monitoring layers displace system complexity without removing human operational limits. The way out requires a raw record of what agents actually did, kept outside the agent runtime, in a form an engineer can consume.
Looking Inside the Model Is Not an Option for Most of Us
Another option that truly breaks free from the agent-on-agent recursion is to not evaluate the agents at all, but to look inside the model itself.
One company, Goodfire, is actually doing this. They are going inside the model by using activation probes, small classifiers trained on a model’s internal states rather than its outputs, on the theory that internals are harder to spoof than surface behavior.
Goodfire claims their process works; they reported catching the reward-hacking behavior behind the Hugging Face hack in real time, at a fraction of the cost of a reasoning monitor.
The problem is that this type of approach requires access to model activations, which rules out most API-only deployments, meaning the technique is available to the labs and largely unavailable to their customers.
Converging on Agent Trust
All of these observations bring me back to the core prediction central to this piece: that observability and security have to better fit together as a defensive solution to our collective AI risk.
Traditionally, security has been the control plane, dictated by policies, boundaries, allow-lists, and permissions. Observability, on the other hand, was the operational plane – focused on measuring what is running, how reliable it is, and what the impact of its performance is downstream.
The new world completely blurs these lines because agents are now acting across every surface of modern software, both on the attack and on the defense.
This makes security infrastructure a running system that makes judgment calls, has failure modes, triages alerts, drifts over time, and produces output you have to be able to trust.
Remember, Hugging Face’s stack made a judgment about severity and got it wrong. This was an agent failure that another overlay of security would not have caught because it was performant, not hacky, in nature.
To repeat my core argument, you cannot solve an agent problem purely through the lens of security practice, because modern security practice is itself agentic.
Every agent you deploy to watch your agents is another system that requires oversight, and another source of alerts competing for the same human attention. Each layer moves the problem up a level. It is the Russian Doll dilemma, and the dolls nest forever unless something stops them.
The thing that stops them is a durable record: one that exists independently of the agents that generated it, survives the incident, and can be read afterward by a human being. Observability provides that layer and allows the regress to end.
