# Recommended CloudWatch alarms for Amazon MSK Provisioned clusters
<a name="bestpractices-cw-alarms"></a>

Monitor your Amazon MSK Provisioned clusters to detect issues before they affect your applications. CloudWatch alarms perform an action when a CloudWatch metric exceeds a specified value for some amount of time. For example, you might want to get an email notification if your partition count exceeds the recommended value for your broker instance size for longer than 15 minutes. The critical alarms in the following table are recommended, but they are not an exhaustive list of alarms you can create to monitor your cluster.

For more information about configuring alarms, see [Creating Amazon CloudWatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) in the *Amazon CloudWatch User Guide*.

The following table lists alarms that apply to both Standard and Express brokers.


| Alarm | Issue | 
| --- | --- | 
| `CPUUser` \+ `CPUSystem` Average >= 60 for 5 minutes, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | One or more brokers have average CPUUser\+CPUSystem over the recommended 60%. To learn more, see [Monitor CPU usage](bestpractices.md#bestpractices-monitor-cpu). | 
| `PartitionCount` Average >= *X* for 5 minutes, 3 consecutive times (*X* = Recommended partition count for broker instance size)<br />Dimensions: `Cluster Name`, `Broker ID` | One or more brokers have partitions higher than the recommended partition count limit. To learn more, see [Right-size your cluster: Number of partitions per Standard broker](bestpractices.md#partitions-per-broker) and [Express broker partition quota](limits.md#msk-express-broker-partition-quota). | 
| `SumOffsetLag` Average >= *X* for 5 minutes, 3 consecutive times (*X* is set for a consumer group and topic combination based on use case)<br />Dimensions: `Cluster Name`, `Consumer Group`, `Topic` | The aggregated offset lag for all the partitions in a topic is over *X*. For more information about the lag, you can use the partition-level `Offset` metric, which represents the lag for each partition, or use the Kafka command line tool to describe the consumer group. Review the processing speed of your consumer application relative to the producers to understand if the consumers are unable to keep up, and check if any consumer rebalances are slowing down the consumers. | 

The following table lists alarms that apply to Standard brokers only.


| Alarm | Issue | 
| --- | --- | 
| `OfflinePartitionsCount` Average >= 1 for 1 minute, 3 consecutive times<br />Dimensions: `Cluster Name` | One or more topic partitions are unavailable. When partitions are unavailable, produce and consume operations to those partitions fail. Offline partitions should not occur on a well-balanced, correctly sized, and correctly configured cluster. To learn more, see [Build highly available clusters](bestpractices.md#ensure-high-availability). | 
| `UnderMinIsrPartitionCount` Average >= 1 for 1 minute, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | One or more topics have partitions under the minimum configured in-sync replica set (ISR). When partitions fall below the minimum ISR, produce operations fail (with producer `acks=all`). To learn more, see [Build highly available clusters](bestpractices.md#ensure-high-availability). | 
| `KafkaDataLogsDiskUsed` Average >= 80 for 5 minutes, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | One or more brokers have data disk usage of 80% or more. To learn more, see [Monitor disk space](bestpractices.md#bestpractices-monitor-disk-space). | 
| `HeapMemoryAfterGC` Average >= 60 for 5 minutes, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | One or more brokers have 60% or more of total heap memory in use after garbage collection. To learn more, see [Monitor Apache Kafka memory](bestpractices.md#bestpractices-monitor-memory). | 
| (Sum(`VolumeReadBytes`) \+ Sum(`VolumeWriteBytes`)) / (5 \* 60 \* 1024 \* 1024) >= *X* MiB for 5 minutes, 3 consecutive times (*X* = 80% of available volume throughput)<br />Dimensions: `Cluster Name`, `Broker ID` | One or more brokers have underlying volume read and write activity using up 80% of its available volume throughput. To learn more, see [Provisioned storage throughput](https://docs.aws.amazon.com/msk/latest/developerguide/msk-provision-throughput-management.html). | 
| `CPUCreditBalance` Average <= 100 for 5 minutes, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | This is only relevant to the t3.small broker type. One or more brokers have depleted their CPU credit balance from a maximum of 576 to less than 100. When the balance reaches 0, the broker cannot exceed the 20% CPU baseline. To avoid CPU credit depletion, upgrade from a t3 broker instance type to an m7g instance type, which doesn't use CPU credits. | 
| `RequestHandlerAvgIdlePercent` Average <= 0.3 for 5 minutes, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | One or more brokers are seeing activity congestion on the threadpool responsible for serving requests (threadpool is less than 30% idle). Saturation here indicates slow requests, which can cause client-side timeouts. Also, review whether your clients are generating excessive requests. For example, unauthorized clients might be aggressively retrying requests that the brokers are denying. To learn more about optimizing cluster throughput, see [Optimize cluster throughput for m5.4xl, m7g.4xl or larger instances](bestpractices.md#optimize-broker-threads). | 
| `NetworkProcessorAvgIdlePercent` Average <= 0.3 for 5 minutes, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | One or more brokers are seeing activity congestion on the network connection threadpool (threadpool is less than 30% idle). Saturation here can cause timeouts. Also, review whether your clients are generating excessive requests. For example, unauthorized clients might be aggressively retrying requests that the brokers are denying. To learn more about optimizing cluster throughput, see [Optimize cluster throughput for m5.4xl, m7g.4xl or larger instances](bestpractices.md#optimize-broker-threads). | 
| `KafkaFileDescriptorsUsagePercent` > 80% for 5 minutes, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | The percentage of file descriptors in use on the broker. At 100% exhaustion, the Kafka broker might not be able to start. The file descriptors count increases with the number of partitions, number of log segments in each partition, and number of client connections. Review whether you have topics with low `segment.ms` values resulting in frequent log rolls, and consider reducing the number of client connections. | 
| `KafkaMemoryMappedFilesUsagePercent` > 80% for 5 minutes, 3 consecutive times<br />Dimensions: `Cluster Name`, `Broker ID` | The percentage of memory-mapped files in use on the broker. At 100% exhaustion, the Kafka broker might not be able to start. The memory-mapped files usage count increases with the number of partitions and number of log segments in each partition. Review whether you have topics with low `segment.ms` values resulting in frequent log rolls. | 

## IAM access control alarms
<a name="bestpractices-cw-alarms-iam"></a>

In addition to the preceding alarms, we recommend creating alarms for the following metrics specific to IAM access control. These alarms apply to both Standard and Express brokers with IAM authentication enabled. Amazon MSK places logical limits on IAM connections to protect the broker from IAM connection request overload. Breaching any of these limits results in client connection timeouts, which will impact your workload.


| Alarm | Issue | 
| --- | --- | 
| `ClientConnectionCount` Sum >= *X* for 1 minute, 3 consecutive times (*X* = 80% of Maximum TCP connections per broker)<br />Dimensions: `Cluster Name`, `Broker ID`, `Client Authentication` | One or more brokers have connection count equal to 80% of the connection limit. The default Maximum TCP connections per broker for IAM access control is 3000. This value can be changed. For more information, see [Amazon MSK Express broker quota](limits.md#msk-express-quota) for Express brokers and [Amazon MSK Standard broker quota](limits.md#msk-provisioned-quota) for Standard brokers. | 
| `ConnectionCreationRate` Sum >= *X* for 1 minute, 3 consecutive times (*X* = 80% of the connection creation rate limit for your instance size)<br />Dimensions: `Cluster Name`, `Broker ID` | One or more brokers have clients creating IAM connections at a rate equal to 80% of its connection creation rate limit. The Maximum TCP connections rate per broker for IAM access control depends on instance size. For more information, see [Amazon MSK Express broker quota](limits.md#msk-express-quota) for Express brokers and [Amazon MSK Standard broker quota](limits.md#msk-provisioned-quota) for Standard brokers. |