PromQL querying

When you ingest OpenTelemetry metrics into CloudWatch through the Metrics endpoint, the hierarchical OTLP data model is flattened into PromQL-compatible labels. This section describes the label structure, the PromQL syntax for querying these labels, and the UTF-8 support in PromQL.

When using PromQL in CloudWatch, the @ prefix convention distinguishes OTLP-scoped labels from standard Prometheus labels. Fields within each scope use a double-@ prefix (for example, @resource.@schema_url), while attributes use a single-@ scope prefix, for example, @resource.service.name. Datapoint attributes also support bare (un-prefixed) access for backward compatibility with standard PromQL queries, for example, {"http.server.active_requests"} and {"@datapoint.@name"="http.server.active_requests"} are equivalent.

A PromQL expression is enclosed in curly braces, specifying the metric name and an optional set of label matchers. The following example selects all time series for the http.server.active_requests metric:

{"http.server.active_requests"}

The following example selects all time series for the metric http.server.active_requests where the OpenTelemetry resource attribute service.name equals myservice:

{"http.server.active_requests", "@resource.service.name"="myservice"}

You can combine multiple label matchers in a single query. The following example selects all time series for the http.server.active_requests metric where the OpenTelemetry resource attribute service.name equals myservice across all US regions:

{"http.server.active_requests",
 "@resource.service.name"="myservice",
 "@aws.region"=~"us-.*"}

The following example shows a range query. It calculates the average value of all datapoints within a specified time range for each time series:

avg_over_time(
  {"http.server.active_requests",
   "@resource.service.name"="myservice"}[5m]
)

The following table summarizes the prefix conventions for each OTLP scope:

Querying vended AWS metrics with PromQL

To be able to query vended AWS metrics in PromQL, you first need to enable OTel enrichment of vended metrics. See: AWS vended metrics in OpenTelemetry format.

After you enable OTel enrichment, vended AWS metrics become queryable through PromQL with additional labels. The metric name is the same as the original CloudWatch metric name, and the original CloudWatch dimensions are available as datapoint attributes. The following labels are available (the example below is for an EC2 instance):

The following example selects Invocations for a specific Lambda function:

{Invocations, FunctionName="my-api-handler"}

The following example selects Lambda Errors for all functions tagged with a specific team:

{Errors, "@instrumentation.@name"="cloudwatch.aws/lambda", "@aws.tag.Team"="backend"}

The following example computes the total Lambda Invocations grouped by team:

sum by ("@aws.tag.Team")(
    {Invocations, "@instrumentation.@name"="cloudwatch.aws/lambda"}
)

The following example selects all time series for the EC2 CPUUtilization metric. The usage of "@instrumentation.@name"="cloudwatch.aws/ec2" is to exclusively match CPUUtilization from EC2 and not from other AWS services such as Amazon Relational Database Service:

histogram_avg({CPUUtilization, "@instrumentation.@name"="cloudwatch.aws/ec2"})

Querying histogram metrics

OpenTelemetry histogram metrics, such as request durations and latencies, are stored as histogram samples rather than float samples. Some AWS vended metrics are also stored as histograms. That's why the previous example uses histogram_avg. Histogram samples behave differently from float samples in PromQL, so a query pattern that works for one type can return an empty or incorrect result for the other.

To check whether a metric is a histogram, select its raw time series and inspect the __type__ label on the results:

{"http.server.request.duration"}

Delta and cumulative histograms

OpenTelemetry histograms use one of two aggregation temporalities:

The temporality determines which query pattern returns a correct result. CloudWatch reports a histogram's temporality in the __temporality__ label, whose value is either cumulative or delta. To check the temporality of a metric, select its raw time series and inspect the __temporality__ label on the results:

{"http.server.request.duration"}

The rate, increase, irate, and resets functions assume cumulative data. They measure growth between datapoints. CloudWatch applies this assumption to every histogram, including delta histograms. On a delta histogram, these functions return an incorrect result without a warning. For example, the common Prometheus pattern histogram_count(rate(histogram[5m])) returns the correct observation rate for a cumulative histogram. It doesn't return a correct rate for a delta histogram.

For a cumulative histogram, use rate. The following example returns the observation rate, in observations per second:

sum(histogram_count(rate({"http.server.request.duration"}[5m])))

For a delta histogram, add up the per-period totals with sum_over_time and divide by the length of the range. This example also returns the observation rate:

sum(histogram_count(sum_over_time({"http.server.request.duration"}[5m]))) / 5m

You can adapt the delta pattern as follows:

Getting statistics from a histogram

The min and max aggregation operators operate only on float samples. They skip histogram samples, so applying them to a histogram metric returns an empty result rather than an error. The response includes an informational annotation that contains the text ignored histogram.

To read a statistic from a histogram metric, use a histogram function:

PromQL doesn't provide an exact maximum for a histogram. The closest equivalent is a high quantile such as histogram_quantile(1, histogram). Because a quantile is estimated from bucket boundaries, the result can be slightly higher than the largest observed value.

For a delta histogram, apply the function directly to the metric. The following example returns the p99 duration for each export period:

histogram_quantile(0.99, {"http.server.request.duration"})

The preceding form returns the p99 for a single export period. To calculate the p99 over a longer window, first combine the per-period histograms with sum_over_time. You don't divide by the range, because a quantile is unaffected by scaling the bucket counts. The following example returns the p99 duration over the last five minutes:

histogram_quantile(0.99, sum_over_time({"http.server.request.duration"}[5m]))

For a cumulative histogram, applying the function directly returns a statistic over the whole life of the series. To get a statistic for a recent window, apply the function to rate. The following example returns the p99 duration over the last five minutes:

histogram_quantile(0.99, rate({"http.server.request.duration"}[5m]))

Keeping labels in the result

An aggregation operator such as avg or sum combines all matching time series into one. It removes every label that you don't group by. To keep one result per label value, add a by clause. The following example returns the observation rate of a delta histogram for each service:

sum by ("@resource.service.name")(
    histogram_count(sum_over_time({"http.server.request.duration"}[5m]))
) / 5m

Histogram functions such as histogram_quantile and histogram_avg are functions rather than aggregations, so they keep all labels and return one result for each time series.

Querying with MCP tools

The CloudWatch MCP Server provides Model Context Protocol (MCP) tools that let AI assistants and development tools query CloudWatch PromQL data on your behalf. The MCP tools handle authentication and request formatting automatically, so you can focus on writing PromQL queries rather than managing HTTP requests and SigV4 signing.

The following PromQL tools are available in the CloudWatch MCP Server:

For full details on parameters, configuration, and setup instructions, see Tools for CloudWatch PromQL in the CloudWatch MCP Server documentation.

Querying with the HTTP API

You can also query CloudWatch PromQL data programmatically by calling the Prometheus-compatible HTTP endpoints directly. Requests must be signed with AWS Signature Version 4 using monitoring as the service name.

The PromQL endpoint follows the pattern https://monitoring.AWS Region.amazonaws.com/api/v1/operation. For example, for the US East (N. Virginia) (us-east-1) Region, the endpoint for an instant query is https://monitoring.us-east-1.amazonaws.com/api/v1/query.

For the full API reference, including supported operations, request parameters, and response formats, see Prometheus-compatible APIs. For the list of AWS Regions where PromQL querying is available, see Supported AWS Regions. For the IAM actions required for each operation, see IAM permissions for PromQL.