AWS DevOps Agent uses large language models (LLMs) hosted on Amazon Bedrock (Amazon Bedrock) to power its agentic reasoning capabilities. The service operates through an agentic loop in which the AI agent: (1) receives an incident trigger or user prompt, (2) plans a sequence of investigation steps, (3) executes tool calls to query observability data, code repositories, deployment histories, and the application topology, (4) incorporates customer-provided context and knowledge (such as operational runbooks and findings from previous investigations) to guide its reasoning, (5) synthesizes findings across multiple data sources, and (6) generates natural language outputs including root cause summaries, evidence citations, and recommended actions. The agent iterates through this loop, refining its hypotheses as it gathers additional evidence. For prevention recommendations, the agent additionally analyzes patterns across historical investigations to identify systemic improvements. Amazon Bedrock Guardrails are applied to filter inputs for prompt attacks and enforce content safety policies.

Individual and confounding variation will differ between customer environments. This means that performance will also differ between environments, even if they support the same use case. Consider two deployments A and B. Deployment A monitors a microservices application with comprehensive logging, distributed tracing, and well-defined resource relationships in the topology. Deployment B monitors a monolithic application with limited logging and partial observability coverage. Because A and B have differing levels of observability data and architectural complexity, they may have different root cause accuracy rates, even assuming each deployment is configured correctly. Customers are in the best position to assess the performance of AWS DevOps Agent on their own operational environment.

We evaluate AWS DevOps Agent using structured scenarios that deploy real AWS infrastructure, inject known faults, and assess agent outputs against human-curated ground truths. The evaluation dataset spans diverse application architectures (serverless, containerized, and VM-based), multiple observability tool integrations, and various root cause categories (code regressions, infrastructure failures, configuration changes, capacity issues, and dependency problems). No single evaluation scenario provides a complete picture of performance because customer environments vary in architecture, observability coverage, and integration configurations.

Evaluation scenarios are developed by the AWS DevOps Agent engineering team in collaboration with applied scientists. Each ground truth is derived from expert knowledge of the injected failure mechanism and specifies the expected root cause, impacted resources, and relevant observability signals. The evaluation dataset currently spans multiple environment configurations across AWS-native and third-party observability integrations, and is continuously expanded as new failure patterns and integrations are supported.

To address uncertainty inherent in LLM-based evaluation, each scenario is evaluated across multiple independent iterations, and performance is monitored using aggregated daily metrics rather than individual run results.

No evaluation benchmark can fully capture the variability of customer environments, including differences in observability coverage, configured integrations, operational context supplied by customers, and application complexity. Customers should evaluate the agent on their own workloads.

AWS DevOps Agent provides multiple mechanisms for customers to control the agent's behavior:

AWS DevOps Agent incorporates multiple layers of safety controls. The agent currently operates in a read-only mode with respect to customer AWS resources where a mutative operations blocklist is designed to enforce this constraint at the tool layer, preventing the agent from executing write operations on customer infrastructure without explicit human approval. Each sub-agent is designed to be restricted to a defined set of allowed tools, limiting the scope of actions any individual component can perform. For customer-provided MCP (Model Context Protocol) server integrations, tool specifications are designed to be validated at association time to ensure only explicitly allowlisted tools are exposed to the agent. The evaluation framework distinguishes between valid approaches that address the root cause and invalid approaches that could be counterproductive (for example, suppressing alarms rather than addressing the underlying issue, or scaling resources without resolving the actual bottleneck).

AWS DevOps Agent uses the same system prompts and agent workflow uniformly across all customer environments. The service does not differentiate its behavior based on customer attributes, organization size, or workload characteristics. All customers receive the same quality of investigation and recommendation generation, subject to the observability data and integrations available in their environment.

AWS DevOps Agent is designed to provide transparent, evidence-based outputs. Investigation findings include citations to specific observability data (log entries, metrics, traces, deployment events) that support the root cause analysis. Findings are classified by multiple evidence types (root cause, contributing cause, or hypothesis) to communicate the degree of certainty in each conclusion. Prevention recommendations cite the evidence from historical investigations that motivates the recommendation. This enables operators to independently verify the agent's reasoning by examining the referenced data.

AWS DevOps Agent processes customer operational data (logs, metrics, traces, code, deployment data) within the scope of the customer's configured Agent Space. The service is designed to use IAM role-based access controls to prevent the agent from accessing resources other than those the customer has explicitly authorized. Amazon Bedrock Guardrails are deployed across all agent types with content filtering designed to detect and block prompt attacks in input. Customer data is encrypted at rest and in transit. Customer data processed by AWS DevOps Agent is not used to train or improve the underlying foundation models. When customers extend AWS DevOps Agent by connecting their own MCP servers, the security and trustworthiness of tool responses from those servers is the customers' responsibility. Customers select which tools are exposed to the agent during onboarding and are responsible for ensuring their MCP servers return trustworthy data. For more information, see Section 50.3 of the AWS Service Terms and the AWS Data Privacy FAQs. For more information, see the AWS DevOps Agent Security documentation.

AWS DevOps Agent uses an agentic architecture that combines observability signal retrieval, topology-aware reasoning, and LLM-based synthesis to produce investigation findings that are grounded in the customer's operational data. However, agentic systems can still produce errors. We assess AWS DevOps Agent using structured ground truths that define the expected root cause, impacted resources, and relevant observability signals for each evaluation scenario.

Our evaluation suite covers a range of root-cause identification scenarios across multiple diverse application environments and multiple accounts, including a banking application with traditional compute and database components, a containerized application, and a distributed microservices application. We periodically inject real-world operational failures across these environments, spanning application, infrastructure, database, configuration, and service issues in AWS-native and selected third-party and multi-cloud configurations.

For incident investigations, we evaluate root cause accuracy by comparing the agent's identified root cause against known ground truths using an LLM-based semantic judge. We periodically validate the LLM-based evaluation through human review. Engineers independently assess a sample of automated judgments against the ground truth and relevant environment evidence using a defined rubric for successful investigations, agent failures, and test-harness issues. Any overrides are recorded and tracked to identify systematic disagreement and improve the evaluation process.

Each scenario is evaluated multiple times to account for variability in agent behavior. We summarize performance using three complementary measures. Pass@1 measures performance on a single attempt. Pass@3 measures whether the agent succeeds at least once across three attempts and reflects its ability to find the correct answer when multiple attempts are available. Pass^3 measures the fraction of scenarios in which the agent is accurate 3 times in a row, providing a stricter measure of consistency and reliability. We also report 95% confidence intervals for benchmark-level performance, computed over scenario-level results to quantify uncertainty arising from the finite evaluation set.

The agent is designed to validate its hypothesis against observed signals before they are presented as root causes. Findings are classified using structured confidence types (root cause, contributing cause, hypothesis) to communicate different levels of certainty. When insufficient data is available to reach a conclusion, the agent is designed to communicate this limitation rather than generating unsupported hypotheses.

For prevention recommendations, we evaluate outputs across two levels of strictness. A recommendation is considered "valid" if it passes both safety checks (following the recommendation would not worsen the operational situation) and correctness checks (the proposed solution is technically sound). A recommendation is considered "high-value" if, in addition to being valid, it also demonstrates goal focus (targets the specific problem rather than generic advice), actionability (an engineer can execute the recommendation without ambiguity), and evidence quality (cites specific observability data from the customer's environment).

We conduct daily offline evaluations using this methodology to monitor the quality of AWS DevOps Agent. We target a root-cause accuracy (Pass@1) of at least 95% for incident investigations and a high-value rate of at least 80% for prevention recommendations. Since the service is continuously improved, a single published point estimate may not remain representative of the production system. Instead, changes and new capabilities are evaluated against these predefined release criteria before deployment. To avoid regression in accuracy, changes and new capabilities are released only when the observed point estimates meet these targets; 95% bootstrap confidence intervals are reported alongside the point estimates to characterize uncertainty. For the current production release, these release criteria were met across the evaluated environments. Exact evaluation results are tracked internally as part of the service's release and quality-monitoring process, curated daily.

When performance does not consistently meet these targets, we investigate and take corrective action, which may include updating the agent or its configuration, or rolling back recent changes.

Customers should conduct veracity testing on their own operational environment and incident patterns.

We measure robustness by evaluating root cause accuracy across multiple environment configurations that vary by cloud provider, observability, and third-party tool integration, testing whether the agent maintains consistent performance when the same class of incident is presented through different observability interfaces and infrastructure platforms. We additionally evaluate consistency of results across repeated attempts on the same scenario.

Individual and confounding variation will differ between customer environments, and customers are in the best position to assess the agent's performance on their own infrastructure and observability configuration.

This AI Service Card is part of AWS's commitment to transparency about how AI services work. AWS DevOps Agent identifies itself as "AWS DevOps Agent" in its communications through integrated channels. The service provides customers with visibility into the investigation process through detailed investigation timelines, tool call histories, and evidence citations. Customers can observe what data the agent accessed and how it arrived at its conclusions. The Operator App console provides onboarding information describing the Agent Space boundary and agent capabilities.

AWS DevOps Agent supports customer governance requirements through IAM-based access controls, Agent Space scoping, and audit capabilities. Customers can configure who has access to the agent, what resources the agent can query, and review investigation histories for compliance and audit purposes. Foundation model versions are pinned to specific version identifiers, and updates are validated through an automated evaluation canary pipeline that runs multiple times daily and emits performance metrics (including root cause accuracy and time to resolution) with associated alarms. An A/B variant testing system enables controlled evaluation of model updates before production deployment.