View a markdown version of this page

Using the Prometheus data source - Amazon Managed Grafana
Prometheus settingsPrometheus query editorMetrics browserTemplatingAnnotations

Using the Prometheus data source

Note

The Amazon Managed Service for Prometheus data source is available starting in Amazon Managed Grafana version 12.

Prometheus settings

Name Description
Name The data source name. This is how you see the data source in panels and queries.
Url The URL of your Amazon Managed Service for Prometheus workspace; for example, https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-example1234/.
Authentication Provider Specify which AWS credentials chain to use.
Assume Role ARN Optional. Specifying the ARN of a role will ensure that the selected authentication provider is used to assume the role rather than the credentials directly.
External ID If you are assuming a role in another account, that has been created with an external ID, specify the external ID here.
Default Region Specify the region, such as for US West (Oregon) use us-west-2 as the region.
Service Provider Specify the AWS service to sign requests against (for example, aps for Prometheus).
HTTP headers Pass additional metadata with each request. Add a custom header name in the Header field and its Value.
Allowed cookies Specify cookies by name that should be forwarded to the data source. By default, the Grafana proxy deletes all forwarded cookies.
Timeout The HTTP request timeout, in seconds.
Manage alerts via Alerting UI Toggled on by default. Enables managing this data source's alerting rules from the Grafana Alerting UI. For Prometheus data sources, this supports viewing existing rules and alerts, which display as data source-managed.
Allow as recording rules target Toggled on by default. Allows this data source to be selected as a target destination for writing Grafana-managed recording rules.
Scrape interval Set this to the typical scrape and evaluation interval configured in Prometheus. Defaults to 15s.
Query timeout Sets the Prometheus query timeout. Defaults to 60s.
Default editor Sets the default query editor. Options are Builder (build queries using a visual interface) or Code (write PromQL directly). You can switch editors in the query editor UI.
Disable metrics lookup Checking this option will disable the metrics chooser and metric/label support in the query field's autocomplete. This helps if you have performance issues with bigger Prometheus instances.
Cache level Sets the browser caching level for editor queries. Options are Low, Medium, High, or None. Higher cache settings are recommended for high-cardinality data sources.
Incremental querying (beta) Toggle on to cache query results and fetch only new records on subsequent queries, instead of always requesting fresh data. This helps reduce database and network load. When enabled, use Query overlap window to specify a duration (for example, 10m) that is added to each incremental request to account for delayed data ingestion.
Disable recording rules (beta) Toggle on so that Grafana does not fetch and parse recording rules from Prometheus, improving dashboard performance by reducing processing overhead.
Custom Query Parameters Add custom parameters to the Prometheus query URL. For example timeout, partial_response, dedup, or max_source_resolution. Multiple parameters should be concatenated together with an "&".
HTTP method Select the POST or GET HTTP method used to query your data source. POST is recommended and selected by default, as it supports larger queries. Select GET if your network restricts POST requests.
Series limit The maximum number of series returned. The limit applies to metrics, labels, and values for both the series and labels endpoints. Leave empty to use the default limit (40000). Set to 0 to disable the limit, which may cause performance issues.
Query statistics Request query processing statistics to view the total queryable samples in Query Inspector. Because query cost scales with samples processed, use this to understand the expense of a query before creating an alert or adding it to a dashboard. This can increase the response payload size.
Query warning threshold Sets the warning threshold for Query Samples Processed (QSP). When queries hit this threshold, a warning message will be returned in the API response.
Query error threshold Sets the error threshold for Query Samples Processed (QSP). Queries that exceed this threshold will be rejected with an error and will not be charged. Used to prevent excessive query costs.
Use series endpoint Toggle on to use the series endpoint (/api/v1/series) with the match[] parameter instead of the label values endpoint. The label values endpoint is generally more performant, but the series endpoint supports the POST method.

Prometheus query editor

The following sections provide information and options for Prometheus query editor in the dashboard and in Explore.

Query editor in dashboards

Open a graph in edit mode by choosing the title and then choosing Edit (or by pressing e key while pausing on the panel).

Name Description
Query expression For more information about Prometheus query expressions, see the Prometheus documentation.
Legend format Controls the name of the time series, using name or pattern. For example {{hostname}} is replaced with the label value for the label hostname.
Min step An additional lower limit for the step parameter of Prometheus range queries and for the $__interval and $__rate_interval variables. The limit is absolute and not modified by the Resolution setting.
Resolution 1/1 sets both the $__interval variable and the step parameter of Prometheus range queries such that each pixel corresponds to one data point. For better performance, use lower resolutions. 1/2 only retrieves a data point for every other pixel, and 1/10 retrieves one data point per 10 pixels. Note that both Min time interval and Min step limit the final value of $__interval and step.
Metric lookup Search for metric names in this input field.
Format as Switch between Table, Time series, or Heatmap. Table works only in the table panel. Heatmap is suitable for displaying metrics of the histogram type on a heatmap panel. It converts cumulative histograms to regular ones and sorts series by the bucket bound.
Instant Perform an "instant" query, to return only the latest value that Prometheus has scraped for the requested time series. Instant queries return results much faster than normal range queries. Use them to look up label sets.
Min time interval This value multiplied by the denominator from the Resolution setting sets a lower limit to both the $__interval variable and the step parameter of Prometheus range queries. Defaults to Scrape interval as set in the data source options.
Note

Amazon Managed Grafana modifies the request dates for queries to align them with the dynamically calculated step. This ensures consistent display of metrics data, but it can result in a small gap of data at the right edge of a graph.

Instant queries in dashboards

The Prometheus data source allows you to run instant queries, which query only the latest value. You can visualize the results in a table panel to see all available labels of a time series.

Instant query results are made up of only one data point per series. They can be shown in the graph panel with the help of series overrides. To show them in the graph as a latest value point, add a series override and select Points > true. To show a horizontal line across the whole graph, add a series override and select Transform > constant For more information about series overrides, see Series overrides.

Query editor in Explore

Name Description
Query expression For more information about Prometheus query expression, see the Prometheus documentation.
Step Step parameter of Prometheus range queries. Time units can be used here, for example: 5s, 1m, 3h, 1d, 1y. Default unit if no unit specified is s (seconds).
Query type Range, Instant, or Both. When running Range query, the result of the query is displayed in graph and table. Instant query returns only the latest value that Prometheus has scraped for the requested time series and it is displayed in the table. When Both is selected, both instant query and range query is run. Result of range query is displayed in graph and the result of instant query is displayed in the table.

Metrics browser

The metrics browser allows you to quickly find metrics and select relevant labels to build basic queries. When you open the browser you will see all available metrics and labels. If supported by your Prometheus instance, each metric will show its HELP and TYPE as a tooltip.

When you select a metric, the browser narrows down the available labels to show only the ones applicable to the metric. You can then select one or more labels for which the available label values are shown in lists in the bottom section. Select one or more values for each label to tighten your query scope.

Note

If you do not remember a metric name to start with, you can also select a few labels first, to narrow down the list and then find relevant label values.

All lists in the metrics browser have a search field above them to quickly filter for metrics or labels that match a certain string. The values section only has one search field. Its filtering applies to all labels to help you find values across labels once they have been selected, for example, among your labels app, job, job_name only one might with the value you are looking for.

Once you are satisfied with your query, click "Use query" to run the query. The Use as rate query button adds a rate(...)[$__interval] around your query to help write queries for counter metrics. The "Validate selector" button will check with Prometheus how many time series are available for that selector.

Limitations

The metrics browser has a hard limit of 10,000 labels (keys) and 50,000 label values (including metric names). If your Prometheus instance returns more results, the browser will continue functioning. However, the result sets will be cut off above those maximum limits.

Templating

Instead of hardcoding things such as server, application and sensor name in your metric queries, you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. You can use these dropdown boxes to change the data being displayed in your dashboard.

For more information about templating and template variables, see Variables.

Query variable

Variable of the type Query allows you to query Prometheus for a list of metrics, labels, or label values. The Prometheus data source plugin provides the following functions you can use in the Query input field.

Name Description
label_names() Returns a list of label names.
label_values(label) Returns a list of label values for the label in every metric.
label_values(metric, label) Returns a list of label values for the label in the specified metric.
metrics(metric) Returns a list of metrics matching the specified metric regex.
query_result(query) Returns a list of Prometheus query result for the query.

For information about what metric names, label names and label values are, see the Prometheus documentation.

Using interval and range variables

Note

Support for $__range, $__range_s, and $__range_ms are available only from Grafana v5.3.

You can use some global variables in query variables: $__interval, $__interval_ms, $__range, $__range_s, and $__range_ms. For more information, see Global variables. These can be convenient to use with the query_result function when you must filter variable queries because the label_values function doesn't support queries.

To get the correct instances when changing the time range on the dashboard, make sure to set the variable's refresh trigger to be On Time Range Change.

The following code example shows how to populate a variable with the busiest five request instances based on average QPS over the time range shown in the dashboard. 


Query: query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
Regex: /"([^"]+)"/

The following code example shows how to populate a variable with the instances having a certain state over the time range shown in the dashboard, using $__range_s. 


Query: query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
Regex:

Using $__rate_interval variable

The $__rate_interval variable is meant to be used in the rate function. It is defined as max( $__interval + Scrape interval, 4 * Scrape interval). Scrape interval is the Min step setting (AKA query_interval, a setting per PromQL query), if any is set, and otherwise the Scrape interval as set in the Prometheus data source (but ignoring any Min interval setting in the panel, because the latter is modified by the resolution setting).

Using variables in queries

There are two syntaxes:

  • $<varname> Example: rate(http_requests_total{job=~"$job"}[5m])

  • [[varname]] Example: rate(http_requests_total{job=~"[[job]]"}[5m])

Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. When the Multi-value or Include all value options are enabled, Grafana converts the labels from plaintext to a regex compatible string. Which means you have to use =~ instead of =.

Annotations

You can use annotations to overlay rich event information on top of graphs. You add annotation queries using the Dashboard menu or Annotations view. For more information, see Annotate visualizations.

Prometheus supports two ways to query annotations.

The step option is useful to limit the number of events returned from your query.