TL;DR — Key Takeaways
1. AI coding agents struggle with rules that require restraint. Research from Peking University found that agents can follow disclosure and verification instructions when prompted, but failed to reliably refuse prohibited work or hand control to humans.
2. Every AI model has its own security weaknesses. Secure Code Warrior’s AI Trust Index identifies repeatable vulnerability patterns across models and frameworks, demonstrating that stronger models and general security instructions do not guarantee safer code.
3. The AI harness is becoming a critical security boundary. Rather than relying on agents to obey written policies, organizations need enforced permissions, automated checks, restricted execution and human approval at critical decision points.
Most people at their mid-career point (or later) have dealt with their share of employees who were “ungovernable” to some level. When they’re good at their job but otherwise a pain in the butt, the kinder term is usually to refer to them as a “diva”. If they’re middling at the role and ignore most senior instructions to boot, those people tend to be moved on to avoid further damage.
Well, certainly in terms of disclosure, verification and predictable security outcomes, LLMs fall somewhere between “diva” and “frightening agent of chaos”. A recent study from researchers at Peking University of the flood of AI-generated contributions into open source communities confirmed that, even when contribution rules are written and implemented to regulate coding agents’ behavior, such as enforcing mandatory disclosure, verification gates and human sign-offs, it largely remains unknown whether coding agents read and follow those rules.
These findings correlate with our recently released AI Trust Index, a living benchmark that measures how often leading AI models generate code with security vulnerabilities. We found that, much like the findings from Peking University, the risk is predictable and repeatable by model and framework.
Why Are Some Rules So Much Harder for Coding Agents to Follow Than Others?
The Peking University study shows a clean split: Rules that ask an agent to add a step, like disclosing AI use or running verification checks, get followed reliably once the agent is steered toward them, reaching 77–100% compliance with a reminder or a single feedback message. Rules that ask an agent to stop or reverse: e.g., refusing banned work, or handing a gated step to a human, sit at 0% compliance across every model tested, even when the agent is handed the exact policy clause and directly told to comply. Generally, this is the expected outcome because extending completed work aligns with an agent’s built-in drive to finish the task, while abandoning or reversing that work runs directly counter to it.
This is where managing and orchestrating AI agents gets far more complex; it requires an intuitive, skilled operator who is both well-versed in security best practices and the nuances of how the models in use handle tasks and queries. A developer who takes everything at face value and assumes the model adheres to guidelines by default in all situations is likely to miss major security issues. Recent tests on our AI Trust Index, where we specifically prompted models to “follow all security best practices”, yield no improvement, but asking them more specific, tailored tasks, like instructing them not to hardcode secrets, API keys, or passwords in the code, does result in a better overall security outcome.
Model Security Performance is Not Random; It Has a Fingerprint
If you take a deep dive into the Peking University study, you quickly get the impression that there is simply no universal compliance IQ among the models. For instance, the paper describes significant inconsistencies, like a model’s Disclose rate and Verify rate not moving together (e.g., Sonnet beats GPT-5.5 on Disclose but trails it on Verify; DeepSeek out-verifies Sonnet).
Our AI Trust Index research uncovered a parallel finding: Each of the models we tested leaves its own repeatable mix of OWASP categories, or a “vulnerability fingerprint,” which has the benefit of letting teams anticipate rather than react to where a given model is likely to introduce risk. It’s essentially the same type of finding, just on a different axis; we’re focused on code content, whereas the Peking University study calls out process compliance.
These outcomes tend to leave CISOs and senior security personnel in somewhat of a quandary: the instinct or desire to “just pick the best model” doesn’t really work. No single model wins everywhere, and the overall frontrunners even shift by cost and framework. Sure, the most expensive and latest model is likely to give you the best results, but if the cost per token/task multiplies ten times tomorrow, will everyone still be able to afford it? Or will they fall back to cheaper/open-source/weight models?
Capability and price don’t buy obedience or security by default.
The AI Harness as the New Perimeter
There’s a buzzword making the rounds this year that’s surprisingly useful, unlike most: The harness. Coined by HashiCorp co-founder Mitchell Hashimoto, the framing is: Agent = Model + Harness. The implication is that the model thinks, while the harness encompasses all the hardcoded extra bits that can dramatically affect security outcomes: permissions, tooling, guardrails, and the parts of the system that decide what the model is even allowed to try to execute.
Security teams have started describing coding agents less like tools and more like actors, as in, working entities with their own credentials, access to repositories and cloud services, and a concerning pile of entitlements the longer they’re left to work unsupervised. Every task you hand an agent is, in effect, giving it “a little more rope”. That’s a different kind of risk than fretting over whether it will read CONTRIBUTING.md, but it’s related: in both cases, the fix is deciding, in advance, what the agent is structurally capable of doing at all, rather than assuming or hoping you’re using the model with the most sound judgment.Which gets at the real reason none of this is going away soon. Traditional, rule-based security assumes predictable execution, and agents have proven across multiple security studies that they don’t offer that. Their behavior is nondeterministic by design, which means the rules can, and will, eventually be side-stepped in favor of whatever big, hairy, audacious goal it has been set. You can’t patch nondeterminism with a stronger sentence in a policy doc.
So that leaves the harness as the only place to put the controls the model itself keeps failing to hold onto, like refusal and hand-off. You know, the stuff agents are, by design, bad at wanting to do.
Open Source Contribution Workflows Must Change (and Quickly)
Keeping a “skilled human in the loop” is the soft, somewhat dated answer, these days, because the approach is more complex and nuanced. It starts with mandatory human review at the merge gate, scoped to where it’s actually needed, backed by automated verification and disclosure checks for everything else.
Skilled development personnel need to orchestrate these elements, with an innate understanding of their codebase and contextual oversight of any AI-powered layers.
The near-term fix is a mandatory pre-execution step built into the agent’s harness that scans a repository for governance documents before any code is touched; one recent study found agents currently do this in only about 3.5% of sessions, not due to a bug but because their objective is resolving the task in front of them, not reading policy files outside that path. That would help with discovery, but it wouldn’t solve the harder half of the problem; even when agents in that same study were handed the exact policy wording, they still didn’t reliably refuse banned work or hand off gated steps to a human. Researchers argue that’s a training-level issue: today’s agents are never rewarded for stopping, only for finishing, and point to future reinforcement-learning approaches that treat refusal as a successful outcome, though that’s a proposed research direction, not a demonstrated fix. So, the honest answer is somewhat underwhelming: Better scaffolding can fix whether an agent finds the rule, but nobody has yet shown a reliable way to make it obey a rule that tells it to stop.
