If AI’s Black Box Can Be Opened, We Have Fewer Excuses

I would like Nick Cammarata to be right.

We are giving AI more responsibility in software development, security operations and business decisions. We are connecting agents to systems where they can take action. The prospect of understanding more of what happens inside those models is welcome news.

But I would also like us to resist turning a promising research argument into permission to declare the safety problem solved.

Cammarata, a researcher at Goodfire who previously worked at OpenAI, has set off a substantial debate with his essay, “Neural Networks Are Already Clean.” He argues that neural networks may be considerably more understandable than researchers, including himself, previously believed.

That willingness to reconsider his own work deserves credit. Scientists are supposed to follow the evidence, including when it leads away from conclusions they helped establish. It takes some humility to do that publicly.

The question for the rest of us is what follows. If we can understand more of these systems, what will we do differently? Will that knowledge produce stronger controls? Will buyers get better evidence about what they are deploying?

Or will “more understandable” become another way of saying “trust us”?

A Cat, a Car and a Different Explanation

The example at the center of Cammarata’s argument is easy enough to follow. Researchers studying an image-recognition model found a neuron that responded to both cats and cars. They interpreted that as evidence that the model was packing unrelated concepts together.

Cammarata revisited the example and arrived at a different explanation: the neuron was responding to a shared V-shaped feature. The apparent jumble had a pattern.

His broader contention is that researchers may have mistaken coherent functions for arbitrary mixtures. He also presents preliminary language-model observations, rather than limiting the argument to older image models. The essay remains explicitly labeled a working hypothesis.

This work falls within mechanistic interpretability, the effort to understand the internal computations that produce a model’s behavior. It goes deeper than asking a chatbot to explain its answer. A model can produce a convincing explanation without that explanation faithfully describing how it reached its conclusion.

Cammarata is offering a reason to think that inspecting the machinery itself may be more productive than many expected. That would be valuable. It could help researchers move from observing failures to investigating their causes.

Still, identifying a component’s apparent function does not explain everything the larger system does with it.

An Explanation Has to Survive a Test

Leo Gao’s response makes that distinction clear. He accepts that researchers have sometimes been too quick to dismiss neurons as mixtures of unrelated concepts. His objection is to how much understanding we can claim from finding a recognizable pattern.

Does the explanation predict responses to unfamiliar inputs? Does it account for how strongly a neuron activates? Are weaker responses carrying information that the simple description leaves out?

Gao argues for a demanding test: An explanation should reproduce a neuron’s behavior well enough that substituting its predictions does not substantially degrade the model. That is his proposed standard, and the disagreement over standards is part of the debate.

For an enterprise buyer, the underlying concern should sound familiar. A vendor can give you a compelling explanation of how something works. You still want to know what happens when you put it under conditions the demonstration did not cover.

The distinction matters especially when the claim moves from understanding selected mechanisms to assuring the behavior of an entire model. At our October 25 research check, we had not located the companion technical paper promised in Cammarata’s essay or an independent replication of its broad language-model findings.

That leaves plenty of room for serious interest. It also leaves work to do before the strongest conclusions are justified.

Useful Progress Does Not Require Complete Understanding

There is a practical story here that deserves attention alongside the scientific argument.

On October 8, Goodfire published research on cybersecurity monitors for Kimi K3 and GLM 5.3. The system reads internal model activity and flags suspicious exchanges for evaluation by another model. That can support intervention before an agent executes a tool call.

Goodfire reports 93% recall at a 5.5% false-positive rate at its chosen operating point, along with substantially lower monitoring costs. Those are company-reported results in the evaluated setting. They are not a promise that the monitor will catch every dangerous action in an unfamiliar deployment.

Nevertheless, this is the kind of application enterprises should examine. It connects an internal signal to a decision about whether an action should proceed.

It also stands on its own. Useful detection does not prove that we fully understand the mechanism producing a behavior, and this monitoring work does not independently validate Cammarata’s broader hypothesis.

Anthropic’s research on persona vectors provides another example. Researchers identified internal activity patterns associated with traits including sycophancy and hallucination in two studied open models. They demonstrated ways to monitor and influence those behaviors.

They also found trade-offs. Some interventions applied after training reduced undesirable traits while degrading general capabilities.

That is an engineering problem worth working on. We should expect adjustments to have consequences and measure them. A technique that reduces one failure while making a model less useful may still be appropriate in a particular setting. The buyer needs to understand that choice.

We do not need a complete explanation of every computation before a monitor or intervention can be useful. We need evidence for the particular claim being made, including where it stops holding up.

Understanding Gives People More Power

There is another complication that gets lost when interpretability is discussed as an automatic safety win.

Knowing how something works can make it easier to defeat.

A 2024 paper coauthored by Neel Nanda examined refusal behavior across 13 chat models. Researchers found an internal direction associated with that behavior and showed that changing it could suppress refusals of harmful requests.

The finding is limited to the studied mechanisms and models. It does not say that every safeguard in every system can be removed the same way. It does show why greater understanding and greater safety cannot be treated as interchangeable.

Researchers can use internal knowledge to strengthen safeguards. Someone else may use it to weaken them. Access, modification rights and independent auditing therefore belong in the discussion.

This is no argument for abandoning interpretability research. We would be poorly served by choosing ignorance as a defense. It is an argument for taking the consequences of that knowledge seriously.

The person holding the controls matters, too.

The Model is Part of the System

For business leaders, the most immediate responsibility remains the environment into which an agent is deployed.

Consider a hypothetical agent given credentials that allow it to delete production data. A more understandable model may offer better ways to detect a dangerous action. But the deployment still needs limits on what those credentials permit, approval requirements for consequential operations and a way to recover when something goes wrong.

An internal warning signal does not revoke a credential by itself. Somebody has to connect detection to enforcement.

Who receives the alert? Can the system stop the action before it executes? Who reviews a blocked request? What happens when a legitimate operation is repeatedly interrupted? What contains the damage when the monitor misses something?

Those questions determine whether an impressive research result becomes a useful operational control.

They also explain why containment remains relevant even as models improve. An organization can restrict an agent’s access to the data and systems required for its assignment. It can separate sensitive environments, preserve logs and require human approval where consequences warrant it.

These controls serve different purposes. Internal monitoring may identify a concerning pattern. A permission boundary limits what the agent can do regardless of whether that pattern is recognized. Recovery procedures address what happens after prevention fails.

There is no reason to wait for a complete theory of neural networks before doing this work. Nor should progress in that theory become an excuse to relax controls already available.

A company buying an AI service should ask what it can actually inspect and enforce. Does the provider offer meaningful monitoring information? Can the customer act on it? What changes when the underlying model is updated? How are failures investigated?

A vendor’s ability to examine its model does not automatically give a customer the same visibility.

Better Knowledge Should Raise Expectations

Cammarata’s work may turn out to be an important step toward making AI easier to understand. It may also require substantial revision as other researchers test its claims. Both possibilities are consistent with taking it seriously.

What we should resist is letting the discussion end with a declaration that the black box has been opened.

Researchers need reproducible methods and opportunities to challenge explanations. Vendors need to show which behaviors their controls address, under what conditions and with what limitations. Enterprises need to decide which actions they will authorize and which risks they are prepared to accept.

Nobody can reasonably demand perfect understanding or zero risk before using technology. But buyers can demand that claims match the evidence and that safeguards reflect the consequences of failure.

I am encouraged by work that gives us better ways to examine these systems. The possibility of catching a dangerous action earlier, diagnosing a failure more accurately or making a targeted improvement is worth pursuing.

The benefits will come through the decisions people make with that knowledge. They will require testing, operational investment and a willingness to stop an action when the evidence says it should not proceed.

If we can see more of what these systems are doing, we should expect more from the companies building them and the organizations putting them to work.