View a markdown version of this page

Migrate from non-MSK Apache Kafka clusters to Amazon MSK Express brokers - Amazon Managed Streaming for Apache Kafka

Migrate from non-MSK Apache Kafka clusters to Amazon MSK Express brokers

You can use MSK Replicator to migrate Apache Kafka workloads from self-managed environments to Amazon MSK Provisioned clusters with Express brokers. MSK Replicator supports data migration from Kafka deployments (Kafka version 2.8.1 or later) that have SASL/SCRAM authentication enabled.

Note

SASL/SCRAM authentication is required only for MSK Replicator to connect to your self-managed Kafka cluster. Your client applications can continue using their existing authentication mechanisms.

Prerequisites

Before you begin, ensure you have the following:

  1. Source Apache Kafka cluster running version 2.8.1 or later

  2. SASL/SCRAM authentication enabled on source cluster

  3. SSL encryption configured on source cluster

  4. Network connectivity via AWS Site-to-Site VPN or AWS Direct Connect

  5. VPC subnets configured for Secrets Manager access

For detailed instructions, see Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters.

Step 1: Create an Amazon MSK Express cluster

Create an MSK Provisioned cluster with Express brokers with IAM authentication enabled. Minimum three brokers across three AZs. See Prepare the target cluster.

Step 2: Create an IAM execution role

Attach AWSMSKReplicatorExecutionRole and AWSSecretsManagerClientReadOnlyAccess managed policies. Configure trust policy for kafka.amazonaws.com. See Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters.

Step 3: Configure SASL/SCRAM and SSL on self-managed cluster

Create dedicated SCRAM user with required ACL permissions. Configure SSL certificates. See Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters.

Step 4: Store credentials in AWS Secrets Manager

Create secret with username, password, and certificate key-value pairs. See Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters.

Step 5: Create the Replicator

Use CreateReplicator API with EARLIEST starting position, Identical topic name replication, and synchroniseConsumerGroupOffsets set to true. If you plan to set up bidirectional replication for rollback capability (Step 6), also set consumerGroupOffsetSyncMode to ENHANCED on both the forward and reverse Replicators. Allow approximately 30 minutes for the Replicator to reach RUNNING status. See CreateReplicator API examples for self-managed Kafka clusters.

Step 6: (Optional) Set up bidirectional replication

Create a reverse Replicator from the MSK Express cluster back to the self-managed cluster for rollback capabilities. See CreateReplicator API examples for self-managed Kafka clusters.

Step 7: Monitor replication progress

Monitor the following metrics:

  • MessageLag (should reach 0)

  • ReplicationLatency

  • ConsumerGroupOffsetSyncFailure (should be 0)

  • ConsumerGroupCount

  • OffsetLag (MSK Cluster) and OffsetLag (Non-MSK Cluster)

For more information, see Monitor replication.

Step 8: Migrate applications

Follow these steps to migrate your applications:

  1. Stop producers writing to self-managed cluster

  2. Reconfigure producers to MSK Express cluster with IAM authentication

  3. Monitor MessageLag until it reaches 0

  4. Stop consumers on self-managed cluster

  5. Reconfigure consumers to MSK Express cluster

Step 9: (Optional) Roll back to self-managed cluster

If bidirectional replication was configured, you can reverse the migration steps to roll back to the self-managed cluster. The reverse Replicator (MSK Express → External) will have been keeping the self-managed cluster in sync, so consumers can be redirected back without data loss.