Amazon Managed Service for Prometheus service quotas
The following two sections describe the quotas and limits associated with Amazon Managed Service for Prometheus.
Service quotas
Amazon Managed Service for Prometheus has the following quotas. Amazon Managed Service for Prometheus vends CloudWatch usage metrics to monitor Prometheus resource usage. Using the CloudWatch usage metrics alarm feature, you can monitor Prometheus resources and usage to prevent limit errors.
As your projects and workspaces grow, the most common quotas that you may need to monitor or request an increase for are: Active series per workspace, Ingestion rate per workspace, and Ingestion burst size per workspace.
For all adjustable quotas, you can request a quota increase by selecting the link
in the Adjustable column, or by requesting a
quota increase
The Active series per workspace limit is dynamically applied. For more information, see Active series default. The Ingestion rate per workspace and Ingestion burst size per workspace together control how quickly you can ingest data into your workspace. For more information see Ingestion throttling.
Note
Unless otherwise noted, these quotas are per workspace.
| Name | Default | Adjustable | Description |
|---|---|---|---|
| Active metrics with metadata per workspace | Each supported Region: 20,000 | No | The number of unique active metrics with metadata per workspace. Note: If the limit is reached, metric sample is recorded, but metadata over the limit is dropped. |
| Active series per workspace | Each supported Region: 10,000,000 per 2 hours |
Yes |
The number of unique active series per workspace. A series is active if a sample has been reported in the past 2 hours. Capacity from 2M to 10M is automatically adjusted based on the last 30 min of usage. |
| Alert aggregation group size in alert manager definition file | Each supported Region: 1,000 |
Yes |
The maximum size of an alert aggregation group in alert manager definition file. Each label value combination of group_by would create an aggregation group. |
| Alert manager definition file size | Each supported Region: 1 Megabytes | No | The maximum size of an alert manager definition file. |
| Alert payload size in Alert Manager | Each supported Region: 20 Megabytes | No | The maximum alert payload size of all Alert Manager alerts per workspace. Alert size is dependent on labels and annotations. |
| Alerts in Alert Manager | Each supported Region: 1,000 |
Yes |
The maximum number of concurrent Alert Manager alerts per workspace. |
| HA tracker clusters | Each supported Region: 500 | No | The maximum number of clusters that HA tracker will keep track of for ingested samples per workspace. |
| Ingestion burst size per workspace | Each supported Region: 1,000,000 |
Yes |
The maximum number samples that could be ingested per workspace in one burst per second. |
| Ingestion rate per workspace | Each supported Region: 170,000 |
Yes |
Metric sample ingestion rate per workspace per second. |
| Inhibition rules in alert manager definition file | Each supported Region: 100 |
Yes |
The maximum number of inhibition rules in alert manager definition file. |
| Label size | Each supported Region: 7 Kilobytes | No | The maximum combined size of all labels and label values accepted for a series. |
| Labels per metric series | Each supported Region: 70 |
Yes |
Number of labels per metric series. |
| Metadata length | Each supported Region: 1 Kilobytes | No | The maximum length accepted for metric metadata. Metadata refers to Metric Name, Type, Unit and Help Text. |
| Metadata per metric | Each supported Region: 10 | No | The number of metadata per metric. |
| Nodes in alert manager routing tree | Each supported Region: 100 |
Yes |
The maximum number of nodes in the alert manager routing tree. |
| Number of API operations per region in transactions per second | Each supported Region: 10 |
Yes |
The maximum number of API operations per second per region. This includes workspace CRUD APIs, tagging APIs, rule groups namespace CRUD APIs, and alert manager definition CRUD APIs. |
| Number of GetSeries, GetLabels and GetMetricMetadata API operations per workspace in transactions per second | Each supported Region: 10 | No | The maximum number of GetSeries, GetLabels and GetMetricMetadata Prometheus-compatible API operations per second per workspace. |
| Number of QueryMetrics API operations per workspace in transactions per second | Each supported Region: 300 | No | The maximum number of QueryMetrics Prometheus-compatible API operations per second per workspace. |
| Number of RemoteWrite API operations per workspace in transactions per second | Each supported Region: 3,000 | No | The maximum number of RemoteWrite Prometheus-compatible API operations per second per workspace. |
| Number of other Prometheus-compatible API operations per workspace in transactions per second | Each supported Region: 100 | No | The maximum number of API operations per second per workspace for all other Prometheus-compatible APIs including ListAlerts, ListRules, etc. |
| Query bytes for instant queries | Each supported Region: 5 Gigabytes | No | 750MB can be scanned by a single instant query. |
| Query bytes for range queries | Each supported Region: 5 Gigabytes | No | The maximum bytes that can be scanned per 24-hour interval in a single range query. |
| Query chunks fetched | Each supported Region: 20,000,000 | No | The maximum number of chunks that can be scanned during a single query. |
| Query samples | Each supported Region: 50,000,000 | No | The maximum number of samples that can be scanned during a single query. |
| Query series fetched | Each supported Region: 12,000,000 | No | The maximum number of series that can be scanned during a single query. |
| Query time range in days | Each supported Region: 32 | No | The maximum time range of QueryMetrics, GetSeries, and GetLabels APIs. |
| Request size | Each supported Region: 1 Megabytes | No | The maximum request size for ingestion or query. |
| Retention time for ingested data in days | Each supported Region: 150 |
Yes |
The number of days that data in a workspace is retained. Data older than this is deleted. You may request quota changes to increase or decrease this value. |
| Rule evaluation interval | Each supported Region: 30 Seconds |
Yes |
Minimum rule evaluation interval. |
| Rule group namespace definition file size | Each supported Region: 1 Megabytes | No | The maximum size of a rule group namespace definition file. |
| Rules per workspace | Each supported Region: 2,000 |
Yes |
The maximum number of rules per workspace. |
| Templates in alert manager definition file | Each supported Region: 100 |
Yes |
The maximum number of templates in the alert manager definition file. |
| Workspaces per region per account | Each supported Region: 25 |
Yes |
The maximum number of workspaces per region. |
Active series default
Amazon Managed Service for Prometheus allows you to use up to your quota of active time series by default.
Amazon Managed Service for Prometheus workspaces automatically adapt to your ingestion volume. As your usage increases, Amazon Managed Service for Prometheus will automatically increase your time series capacity to double your baseline usage, up to the default quota. For example, if your average active time series for the last 30 minutes is 3.5 million, you can use up to 7 million time series without throttling.
If you need more than double your previous baseline, Amazon Managed Service for Prometheus automatically allocates more capacity as your ingest volume increases, to help ensure your workload does not experience sustained throttling, up to your quota. However, throttling can occur if you exceed double your previous baseline computed over the last 30 minutes. To avoid throttling, Amazon Managed Service for Prometheus recommends gradually increasing ingestion when increasing to more than double your previous active time series.
Note
The minimum capacity for active time series is 2 million, there is no throttling when you have less than 2 million series.
To go beyond your default quota, you can request a quota increase.
Ingestion throttling
Amazon Managed Service for Prometheus throttles ingestion for each workspace, based on your current limits.
This helps maintain the
performance of the workspace. If you exceed the limit, you will see
DiscardedSamples in CloudWatch metrics (with the
rate_limited reason). You
can use Amazon CloudWatch to monitor
your ingestion, and to create an alarm to warn you when you are close to reaching
the throttling limits. For more information, see Use CloudWatch metrics to monitor Amazon Managed Service for Prometheus resources.
Amazon Managed Service for Prometheus uses the token
bucket algorithm
Each data sample ingested removes one token from the bucket. If your bucket size (Ingestion burst size per workspace) is 1,000,000, your workspace can ingest one million data samples in one second. If it exceeds one million samples to ingest, it will be throttled, and will not ingest any more records. Additional data samples will be discarded.
The bucket automatically refills at a set rate. If the bucket is below its maximum capacity, a set number of tokens is added back to it every second until it reaches its maximum capacity. If the bucket is full when the refill tokens arrive, they are discarded. The bucket can't hold more than its maximum number of tokens. The refill rate for sample ingestion is set by the Ingestion rate per workspace limit. If your Ingestion rate per workspace is set to 170,000, then the refill rate for the bucket is 170,000 tokens per second.
If your workspace ingests 1,000,000 data samples in a second, your bucket is immediately reduced to zero tokens. The bucket is then refilled by 170,000 tokens every second, until it reaches it's maximum capacity of 1,000,000 tokens. If there is no more ingestion, the previously empty bucket will return to it's maximum capacity in 6 seconds.
Note
Ingestion happens in batched requests. If you have 100 tokens available, and send a request with 101 samples, the entire request is rejected. Amazon Managed Service for Prometheus does not partially accept requests. If you are writing a collector, you can manage retries (with smaller batches or after some time has passed).
You do not need to wait for the bucket to be full before your workspace can ingest more data samples. You can use tokens as they are added to the bucket. If you immediately use the refill tokens, the bucket does not reach its maximum capacity. For example, if you deplete the bucket, you can continue to ingest 170,000 data samples per second. The bucket can refill to maximum capacity only if you ingest fewer than 170,000 data samples per second.
Additional limits on ingested data
Amazon Managed Service for Prometheus also has the following additional requirements for data ingested into the workspace. These are not adjustable.
-
Metric samples older than 1 hour are refused from being ingested.
-
Every sample and metadata must have a metric name.
