# Intelligent rebalancing for clusters
Amazon MSK provides intelligent rebalancing for all new MSK Provisioned clusters with Express brokers. This feature automatically manages partition distribution and cluster scaling operations, eliminating the need for third-party tools. Intelligent rebalancing automatically rebalances partitions when you scale clusters up or down. It also continuously monitors your cluster’s health for resource imbalance or overload and redistributes workload.
Intelligent rebalancing provides fast scaling operations that complete within 30 minutes and doesn’t impact cluster availability during scaling. It’s turned on by default for all new MSK Express-based Provisioned clusters and works with the recommended maximum partition limit of 20,000 partitions per broker. Additionally, this feature is available at no additional cost and doesn’t require any configurations.
Effective 20th Nov 2025, Intelligent Rebalancing is available in all AWS Regions where Amazon MSK Express brokers are supported.
**Topics**
+ [How intelligent rebalancing works](#how-intelligent-rebalancing-works)
+ [Monitoring intelligent rebalancing metrics](#intelligent-rebalancing-metrics)
+ [Considerations to use intelligent rebalancing](#intelligent-rebalancing-considerations)
+ [Scaling Amazon MSK clusters up and down with a single operation](intelligent-rebalancing-scaling-clusters.md)
+ [Steady state rebalancing for Amazon MSK clusters](intelligent-rebalancing-self-balancing-paritions.md)
## How intelligent rebalancing works
Intelligent rebalancing is turned on by default for all new MSK Provisioned clusters with Express brokers. It includes support for the following situations:
+ **Scaling up and down**: Lets you add or remove brokers to your MSK Express-based clusters with a single click. Once you specify the brokers to add or remove, intelligent rebalancing automatically redistributes partitions across the new cluster setup based on internal AWS best practices.
+ **Steady state rebalancing**: At steady state, this feature monitors your cluster’s health continuously and automatically rebalances partitions when:
+ Resource utilization becomes skewed across brokers.
+ Brokers become over-provisioned or under-utilized.
+ New brokers are added or existing brokers are removed.
**Note**
If intelligent rebalancing is turned on, you won’t be able to use third-party tools, such as Cruise Control, for partition rebalancing. You must first pause intelligent rebalancing to use the partition reassignment API provided by these third-party tools.
You can use this feature in the Amazon MSK console. You can also use this feature using the AWS CLI, Amazon MSK APIs or AWS SDK, and AWS CloudFormation. For more information, see [Scaling Amazon MSK clusters](intelligent-rebalancing-scaling-clusters.md) and [Steady state rebalancing](intelligent-rebalancing-self-balancing-paritions.md).
## Monitoring intelligent rebalancing metrics
You can monitor the status of ongoing and historical intelligent rebalancing operations using the following Amazon CloudWatch metrics:
+ `RebalanceInProgress`: This metric is published every minute with a value of 1 when rebalancing is ongoing and 0 otherwise.
+ `UnderProvisioned`: Indicates that a cluster is currently under provisioned and any partition rebalancing can’t be performed. You either need to add more brokers or scale-up your cluster’s instance type.
For information about monitoring an MSK Provisioned cluster, see [](monitoring.md) and [](cloudwatch-metrics.md).
## Considerations to use intelligent rebalancing
+ Support for intelligent rebalancing is only available for new MSK Provisioned clusters with Express brokers.
+ For automatic partition reassignment, maximum support for up to 20,000 partitions per broker is available.
+ You can’t use partition reassignment APIs or third-party rebalancing tools when intelligent rebalancing is enabled. To use such APIs or third-party tools, you must first pause intelligent rebalancing for your MSK Express-based cluster.
# Scaling Amazon MSK clusters up and down with a single operation
With intelligent rebalancing, you can scale your clusters up or down by editing the broker count in your clusters in a single action. You can do this in the Amazon MSK console, or by using the AWS CLI, Amazon MSK APIs or AWS SDK, and AWS CloudFormation. When you change the broker count, Amazon MSK does the following:
+ Automatically distributes partitions to new brokers.
+ Moves partitions from brokers being removed.
As you scale your clusters up and down, cluster availability for clients to produce and consume data remains unaffected.
**Topics**
------
#### [ Scaling clusters using AWS Management Console ]
1. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/home?region=us-east-1\$1/home/](https://console.aws.amazon.com/msk/home?region=us-east-1#/home/).
1. On the **Clusters** page, choose a newly created Express-based cluster. For information about creating a provisioned Express-based cluster, see [Step 1: Create an MSK Provisioned cluster](create-cluster.md).
1. On the **Actions** dropdown list, choose **Edit number of brokers**.
1. On the **Edit number of brokers per zone** page, do one of the following:
+ To add more brokers in your cluster, choose **Add brokers to each Availability Zone**, and then enter the number of brokers you want to add.
+ To remove brokers from your cluster, choose **Remove one broker from each Availability Zone**.
1. Choose **Save changes**.
------
#### [ Scaling clusters using AWS CLI ]
You can scale your clusters up or down by editing their broker count. To do this in the AWS CLI, use the [update-broker-count](https://docs.aws.amazon.com/cli/latest/reference/kafka/update-broker-count.html) command, as shown in the following example. In this command, specify the number of brokers you want in your cluster in the `target-broker-count` parameter.
```
aws msk update-broker-count --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/myCluster/abcd1234-5678-90ef-ghij-klmnopqrstuv-1 --current-version ABCDEF1GHIJK0L --target-broker-count 6
```
------
#### [ Scaling clusters using AWS SDK ]
You can scale your clusters up or down by programmatically editing the broker count. To do this using the AWS SDK, use the [UpdateBrokerCount](https://docs.aws.amazon.com/msk/1.0/apireference/clusters-clusterarn-nodes-count.html#UpdateBrokerCount) API, as shown in the following example. For the `TargetNumberOfBrokerNodes` parameter, specify the number of brokers you want in your cluster.
```
update_broker_count_response = client.update_broker_count(
ClusterArn='arn:aws:kafka:us-east-1:123456789012:cluster/myCluster/abcd1234-5678-90ef-ghij-klmnopqrstuv-1',
CurrentVersion='ABCDEF1GHIJK0L',
TargetNumberOfBrokerNodes=6
)
```
------
# Steady state rebalancing for Amazon MSK clusters
Steady state rebalancing is a part of the intelligent rebalancing feature, which is turned on by default for all new MSK Provisioned clusters with Express brokers. As you scale your clusters up or down, Amazon MSK automatically handles partition management by distributing partitions to new brokers and moving partitions from brokers due for removal. To ensure optimal distribution of workload across brokers, intelligent rebalancing uses Amazon MSK best practices to determine thresholds for automatically initiating rebalancing for your brokers.
You can pause and resume steady state rebalancing when needed. Steady state rebalancing continuously monitors your cluster and does the following:
+ Tracks broker resource usage (CPU, network, storage).
+ Adjusts partition placement automatically without any impact on data availability.
+ Completes rebalancing operations up to 180x faster for Express brokers as compared to Standard brokers.
+ Maintains cluster performance.
**Topics**
------
#### [ Pause and resume steady state rebalancing in AWS Management Console ]
1. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/home?region=us-east-1\$1/home/](https://console.aws.amazon.com/msk/home?region=us-east-1#/home/).
1. On the **Clusters** page, choose an Express-based cluster. For information about creating a provisioned Express-based cluster, see [Step 1: Create an MSK Provisioned cluster](create-cluster.md).
1. On the Cluster detail page, verify that the **Intelligent rebalancing** status is **Active**. If Intelligent rebalancing isn’t available or the status is **Paused**, create a new Express-based cluster.
1. On the **Actions** dropdown list, choose **Edit intelligent rebalancing**.
1. On the **Edit intelligent rebalancing** page, do the following:
1. Choose **Paused**.
1. Choose **Save changes**.
------
#### [ Pause and resume steady state rebalancing using AWS CLI ]
To set the rebalancing status of a cluster to **ACTIVE** using the AWS CLI, use the [update-rebalancing](https://docs.aws.amazon.com/cli/latest/reference/kafka/update-rebalancing.html) command, as shown in the following example. In this command, specify the status with the `rebalancing` parameter.
```
aws msk update-rebalancing --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/myCluster/abcd1234-5678-90ef-ghij-klmnopqrstuv-1 --current-version ABCDEF1GHIJK0L --rebalancing "{\"Rebalancing\":{\"Status\":\"ACTIVE\"}}"
```
------
#### [ Pause and resume steady state rebalancing using AWS SDK ]
You can also set the rebalancing status of a cluster using the [UpdateRebalancingRequest](https://docs.aws.amazon.com/msk/1.0/apireference/clusters-clusterarn-rebalancing.html#UpdateRebalancing) API to programmatically modify the broker count. The following examples show how to set the rebalancing status to **ACTIVE** and **PAUSED**.
```
final UpdateRebalancingRequest updateRebalancingRequest = new UpdateRebalancingRequest()
.withClusterArn(arn:aws:kafka:us-east-1:123456789012:cluster/myCluster/abcd1234-5678-90ef-ghij-klmnopqrstuv-1)
.withCurrentVersion(ABCDEF1GHIJK0L)
.withRebalancing(new Rebalancing().withStatus("ACTIVE"));
```
```
final UpdateRebalancingRequest updateRebalancingRequest = new UpdateRebalancingRequest()
.withClusterArn(arn:aws:kafka:us-east-1:123456789012:cluster/myCluster/abcd1234-5678-90ef-ghij-klmnopqrstuv-1)
.withCurrentVersion(ABCDEF1GHIJK0L)
.withRebalancing(new Rebalancing().withStatus("PAUSED"));
```
------