Platform teams now run AI agents from several sources at once: LangChain and LangGraph builds, custom frameworks, and vendor copilots inside observability and ITSM tools. Day 1 was getting those agents to produce answers. Day 2 is knowing whether the answers are right, and keeping them right while the cluster underneath them changes every week.
Most evaluation work targets the model or the final answer. Neither is a good unit for Day 2. You rarely control the model. A final-answer score tells you something failed but not what to change. The unit worth measuring is the skill.
What a Skill Is
A skill is the reusable procedure an agent follows for one kind of work: a prompt, a tool procedure, a set of queries, a runbook, or an MCP tool definition. A Kubernetes triage skill might specify which logs to read, which metrics to pull, in what order, and when to conclude.
Three properties make skills the practical unit of improvement. You own them; the model provider does not. They are text and configuration, so they can be versioned and diffed. They are shared: one triage skill may run inside four agents built on three frameworks, so one fix improves all four.
Start From the Traces You Already Emit
Evaluating skills does not require a new agent harness. Many agent frameworks support tracing, either natively or through instrumentation libraries, and the OpenTelemetry GenAI semantic conventions give those traces a common shape. The conventions define agent operations such as invoke_agent, plan and execute_tool, and attributes such as gen_ai.agent.name and gen_ai.tool.name. They are still in Development status, so expect names to change. Instrumentation libraries and open-source backends such as Langfuse support OpenTelemetry-based tracing.
The conventions also define attributes for Agent Skills, including gen_ai.skill.name, but these do not automatically identify every reusable procedure an agent follows. For consistent evaluation across frameworks, attach a skill identifier and version to every relevant span the skill produces, using custom attributes where necessary. That turns a pile of LLM and tool spans into per-skill evidence.
Run the evaluation out of band, on the trace stream. The evaluator should be read-only against both agents and telemetry, and it should never sit in the request path. An evaluator that adds latency or can block an agent will be switched off during the first incident.
Figure 1. The skill improvement loop. Traces feed an out-of-band evaluator; every change to a skill passes a review gate before it ships.
Score Each Skill on Four Dimensions
- Speed: Latency and steps the skill adds per use, including retries, dead-end tool calls and repeated queries.
- Accuracy: How often answers are right when this skill is in the loop.
- Correctness: Whether the skill queries the right sources, uses tools properly and claims only what the evidence supports.
- Reasoning: Whether each step advances the investigation, measured by hypothesis quality, evidence coverage and dead ends avoided.
Accuracy is the hard one. An LLM judge that reads the same context the agent read will share its blind spots. Reliable evaluation requires independently verified evidence, ideally including information the agent did not use: change records, topology and dependency data, and prior incidents with confirmed causes. An evaluator that sees no more than the agent can grade plausibility, but it cannot reliably determine whether the answer is right.
Attribute Every Miss to a Step
Consider an agent asked why checkout is slow. It queries error logs, notices that errors correlate with pod restarts, pulls six hours of memory metrics, and concludes that a memory leak calls for a pod restart. Each span looks reasonable. An evaluator with access to change history sees a deploy of the checkout service minutes before the errors began, and three earlier “leaks” on the same service that all traced back to deploys.
The final answer is wrong, but the useful finding is narrower. The triage skill never consults change history before it concludes. That is a one-step fix to one skill. Aggregate these attributions across hundreds of invocations and each skill gets a ranked list of the steps behind its misses. Ship the fix, then measure the skill’s accuracy on new invocations to confirm it worked.
Skills Drift With the Cluster
A skill that was right in June can be quietly wrong by September. Someone adds a service mesh, reorganizes namespaces or moves logs to a new backend. The skill still runs, but its dependency map is stale, and accuracy decays slowly enough that nobody notices.
Track accuracy per skill version over time and correlate declines with environment changes that touch the skill’s sources. When a drop lines up with a topology change, the needed revision is usually clear.
Traces also reveal missing skills. When investigations keep hand-building the same query or check, that procedure should exist as a skill. Draft it from the version that worked, with its sources, steps and guardrails, such as read-only queries and an escalation threshold.
Put Skill Changes Through the Same Gate as Code
Agents that rewrite their own procedures without review lose the trust of the teams that run them. Skill changes need the discipline platform teams already apply to infrastructure. Every tune, revision or new skill should be a versioned diff that carries its evidence and expected effect. A named owner approves it. After rollout, the measured result is compared with the projection, and any version can be rolled back. Teams that run GitOps can keep skills in the same repositories and review them through the same pull requests.
Include the Humans in the Loop
The same traces show how people work with agents. They can highlight approvals that add latency without apparent safety benefits, such as a class of fixes approved unchanged nine times out of ten. They show overrides the evidence did not support. They also show engineers whose corrections to agent output are good enough to fold back into a skill. Use this data to adjust approval policy and improve skills. Keep it out of individual performance reviews, or engineers will stop correcting the agents.
Where to Start
- Instrument agents with the OpenTelemetry GenAI conventions and export traces to a backend you control.
- Add skill name and version to every relevant span.
- Pick your noisiest agent and build a scorecard for each skill it uses.
- Take the skill with the most attributed misses and push one fix through a reviewed, versioned change.
- Measure the result on new traffic before moving to the next skill.
Many platform teams already have the tracing, version control and review processes this requires.
KubeCon + CloudNativeCon is taking place in Salt Lake City from November 9 to 12. Register Now!
