The hardest question my teams have faced while building with AI agents concerns trust rather than capability: when to accept an agent’s output and when to keep a human in the loop. It came up across a wide range of enterprise workflows, from migrating code and data pipelines to reviewing and fixing bugs. More than half of the organizations surveyed in LangChain’s 2026 State of Agent Engineering report now run AI agents in production, and respondents named quality, ahead of cost, as the biggest barrier to deployment. That ordering tells a story: teams have learned to build agents faster than they have learned to judge whether an agent deserves their trust.
The instinct most teams bring to this problem comes from traditional software testing, where a given input produces a known output and any deviation counts as a bug. Agents break that model: they can take several legitimate paths to the same outcome, and it can take the same path twice and land somewhere different. Researchers at Sierra built the τ-bench benchmark to measure this, and the results should give any platform owner pause. State-of-the-art agents completed fewer than half of the benchmark’s customer service tasks, and when a task was attempted eight separate times, consistency in the retail domain fell below 25%.
My perspective on this comes from a somewhat unusual place. Before I worked on agents, I spent years building data governance for enterprise platforms, including classification, lineage, and access controls for some of the largest data estates in the world. Enterprises already solved a version of this problem. No compliance officer reads every row of a petabyte-scale warehouse, and regulated companies trust those systems anyway, because governance gives them properties they can verify, starting with who touched the data and under which policy. Agents call for the same shift in thinking. Certainty about every output is out of reach, so the goal becomes confidence in the boundaries. One evaluation made this concrete for me. An agent we were assessing ran a data analysis pipeline and returned the right answer. When the team sat down to compare its trajectories, we found that three separate runs had approached the solution in three distinct ways, and most of those paths looked nothing like how a human would have worked through the problem. Each run also carried its own performance profile and token cost. That review changed how my team approached agentic systems. A correct output told us less than we assumed, and measuring the trajectory itself, covering efficiency, security adherence, bias, and resource usage, proved just as important.
A workable evaluation framework grew out of those lessons. Success gets defined as an end state rather than a script. τ-bench does this by comparing the state of the system after the agent finishes against an annotated goal, which lets any legitimate path count. Policy boundaries come next, meaning the rules an agent must respect even while improvising, since an agent that reaches the right outcome by breaking policy has still failed. Reliability then gets measured across repeated trials, because an agent that succeeds nine times out of ten needs a different deployment posture than one that succeeds every time. Recent academic work on multi-dimensional evaluation for enterprise agents is formalizing these ideas, and the NIST AI Risk Management Framework supplies useful vocabulary, treating trustworthiness as a set of measurable properties.
Current practice falls well short of this. In the same LangChain survey, 89% of organizations reported having observability for their agents while barely half ran offline evaluations, which means most teams can watch an agent work without any structured way to judge the work. Gartner projects that more than 40% of agentic AI projects will be canceled by the end of 2027, with inadequate risk controls among the leading causes. Watching an agent turns out to be far easier than judging one, and the gap between those two capabilities is where deployments stall. On my own teams, we measured task success alongside efficiency across agents, skills, tools, and model versions, then layered in consistency, accuracy, bias, policy and security adherence, latency, and token consumption. No single metric determined production readiness, but taken together they told us how far an agent could be trusted.
Agents will stay unpredictable for the foreseeable future, and waiting for that to change means ceding ground to competitors willing to manage the uncertainty. The better path mirrors how enterprises learned to ship data platforms: inside boundaries you can verify, with autonomy that expands as the evidence accumulates. In practice, that means letting an agent earn its independence. Start with low-risk work, such as read-only tasks in a QA environment, before allowing an agent to make production changes. Define explicit guardrails that spell out which actions the agent can take on its own, which require approval, and which it must never attempt. Keep observing its behavior, from consistency and policy adherence to latency and cost, and let its autonomy grow only as quickly as the evidence supporting it.
