TL;DR — Key Takeaways
– AWS introduced CloudWatch Omni, a unified observability platform designed to help developers evaluate, monitor and troubleshoot AI applications and agents.
– The platform provides 17 built-in evaluators to assess AI agent performance across dimensions such as correctness, coherence, retrieval quality and tool selection.
– CloudWatch Omni enables developers to trace agent behavior, compare prompts, identify regressions and visualize complex agent workflows.
Amazon Web Services (AWS) this week unfurled CloudWatch Omni, a unified observability platform for evaluating and managing artificial intelligence (AI) applications and agents.
This addition to the AWS portfolio differs from the existing Amazon CloudWatch monitoring service in that it is designed to integrate directly within the integrated development environment (IDE) used to build an AI application or agent or through a separate web console.
Designed to capture every trace that an AI application or agent generates, CloudWatch Omni, among other capabilities, provides 17 built-in evaluations to assess, for example, correctness, coherence, retrieval quality, and tool selection. It also can be integrated with third-party evaluators, including AutoEval and DeepEval frameworks for assessing AI applications and agents.
Additionally, builders of AI applications and agents can compare prompts side by side in a playground, build test datasets from production traffic, run experiments across different configurations, and detect regressions automatically.
There is also a Prompt Management tool for tracking version and prompt configurations over time and a Session Explorer tool to review full conversation histories spanning multiple interactions. An Agent Topology, meanwhile, makes it possible to visualize the architecture of an agent workflow, including sub-agents, tools, and their interconnections.
Deployable across multiple cloud computing environments starting with AWS and Microsoft Azure, CloudWatch Omni also supports agent frameworks such as LangChain, LangGraph, CrewAI, OpenAI SDK, Strands, Vercel AI SDK, and others, running both Python and TypeScript. It also provides native observability for agents built with the Amazon Bedrock AgentCore service that provides additional observability tools. Finally, CloudWatch Omni supports open instrumentation frameworks such as OpenInference and AWS Distro for OpenTelemetry (ADOT).
The overall goal is to provide a unified experience for observing AI agents and applications regardless of how they were built, says Mukul Karnik, vice president of observability for AWS. “You can bring your own agent to Omni,” he says.
While traces as a type of log have been used for years now, the rise of AI applications and agents is making them essential for understanding behavior. Unlike traditional request-response systems, agents make multiple decisions per invocation. Without full trace visibility, diagnosing why an agent produced an incorrect answer or took an unexpected path becomes guesswork.
CloudWatch Omni instead records every step in a structured timeline to enable builders of AI applications and agents to pinpoint exactly where behavior diverged using a Trace Explorer tool that provides a detailed breakdown of every step an AI agent took to complete a task. Evaluators then score each response against quality dimensions to see what end users actually experienced and, just as importantly, identify regressions.
That capability is critical because AI agent behavior is, by definition, non-deterministic. A single prompt change can degrade response quality even when standard metrics show no errors. Without an ability to observe the behavior of an AI agent, an IT team would need to spend hours manually reviewing log data to determine the root cause of an issue.
Mitch Ashley, vice president and practice lead for software lifecycle engineering for the Futurum Group, said when an agent gives a bad answer, the cause can span everything from its reasoning to a stale data source or a slow database. Evaluating those issues needs to occur in production environments in order to understand how AI agent quality is degrading, he adds.
It’s not clear at what pace organizations are rushing to observe AI agent behavior, but as it becomes apparent that there will soon be thousands of AI agents deployed across the average enterprise, it’s now more a question of when, rather than if, organizations will need deeper visibility into AI agent behavior.
