View a markdown version of this page

Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters - Amazon Managed Streaming for Apache Kafka
Create an IAM execution roleConfigure SASL/SCRAM user and ACL permissionsConfigure SSL on self-managed clusterStore credentials in AWS Secrets ManagerConfigure network connectivityConfigure security groups

Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters

Create an IAM execution role

Create an IAM role with a trust policy for kafka.amazonaws.com. Attach the AWSMSKReplicatorExecutionRole and AWSSecretsManagerClientReadOnlyAccess managed policies.

Example trust policy:

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "kafka.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}

Configure SASL/SCRAM user and ACL permissions

Create a dedicated SCRAM user on your self-managed Kafka cluster. The following ACL permissions are required:

  1. Read, Describe on all topics

  2. Read, Describe on all consumer groups

  3. Describe on cluster resource

Example kafka-acls.sh commands:

# Grant Read and Describe on all topics
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Read --operation Describe \
  --topic '*'

# Grant Read and Describe on all consumer groups
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Read --operation Describe \
  --group '*'

# Grant Describe on cluster
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Describe --cluster

Configure SSL on self-managed cluster

Configure SSL listeners on your brokers. For publicly trusted certificates, no additional configuration is required. For private or self-signed certificates, include the full CA certificate chain in the secret stored in AWS Secrets Manager.

Store credentials in AWS Secrets Manager

Create a secret of type Other (not RDS/Redshift) in AWS Secrets Manager with the following key-value pairs:

  1. username — SCRAM username for the self-managed cluster

  2. password — SCRAM password for the self-managed cluster

  3. certificate — CA certificate chain (PEM format; required for private/self-signed certs)

Configure network connectivity

MSK Replicator requires network connectivity to your self-managed Kafka cluster. Supported options:

  • AWS Site-to-Site VPN — Connect on-premises networks to your VPC over the internet.

  • AWS Direct Connect — Establish a dedicated private network connection from your premises to AWS.

Configure security groups

Ensure security groups allow traffic between MSK Replicator and the self-managed cluster on the SASL_SSL port (typically 9096). Update both inbound rules on VPC security groups and outbound rules on the self-managed cluster firewall.