View a markdown version of this page

Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters - Amazon Managed Streaming for Apache Kafka
Create an IAM execution roleConfigure SASL/SCRAM user and ACL permissionsConfigure mTLS on self-managed clusterConfigure SSL on self-managed clusterStore credentials in AWS Secrets ManagerConfigure network connectivityConfigure security groups

Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters

Create an IAM execution role

Create an IAM role with a trust policy for kafka.amazonaws.com. Attach the AWSMSKReplicatorExecutionRole managed policy. The managed policy grants the cluster-, topic-, and consumer-group-level Kafka permissions the Replicator needs, but it does not include AWS Secrets Manager or AWS KMS permissions, which are required for SASL/SCRAM authentication and CMK-encrypted credentials. For the inline policy snippets to add, see Additional SER permissions for SASL/SCRAM, mTLS, and customer managed keys.

Example trust policy:

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "kafka.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}

Configure SASL/SCRAM user and ACL permissions

Create a dedicated SCRAM user on your self-managed Kafka cluster. The following ACL permissions are required:

  1. Read, Describe on all topics

  2. Read, Describe on all consumer groups

  3. Describe on cluster resource

Example kafka-acls.sh commands:

# Grant Read and Describe on all topics
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Read --operation Describe \
  --topic '*'

# Grant Read and Describe on all consumer groups
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Read --operation Describe \
  --group '*'

# Grant Describe on cluster
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Describe --cluster

Configure mTLS on self-managed cluster

Configure an SSL listener on your self-managed Kafka brokers with ssl.client.auth=required. The broker's truststore must contain the CA certificate that signed the client certificate you will use for MSK Replicator.

Grant ACL permissions to the Kafka principal derived from the client certificate's Distinguished Name (DN). The required permissions are: Read and Describe on all topics, Read and Describe on all consumer groups, and Describe on the cluster resource.

Configure SSL on self-managed cluster

Configure SSL listeners on your brokers. For publicly trusted certificates, no additional configuration is required. For private or self-signed certificates, include the full CA certificate chain in the secret stored in AWS Secrets Manager.

Store credentials in AWS Secrets Manager

Create a secret of type Other (not RDS/Redshift) in AWS Secrets Manager with the appropriate key-value pairs for your authentication type.

For SASL/SCRAM:

  1. username — SCRAM username for the self-managed cluster

  2. password — SCRAM password for the self-managed cluster

  3. certificate — CA certificate chain (PEM format; required for private/self-signed certs)

For mTLS:

  1. certificate — PEM-encoded client certificate chain

  2. privateKey — PEM-encoded private key

  3. privateKeyPassword — (Optional) Passphrase for the private key, required only for encrypted PKCS8 keys

Configure network connectivity

MSK Replicator requires network connectivity to your self-managed Kafka cluster. Supported options:

  • AWS Site-to-Site VPN — Connect on-premises networks to your VPC over the internet.

  • AWS Direct Connect — Establish a dedicated private network connection from your premises to AWS.

Configure security groups

Ensure security groups allow traffic between MSK Replicator and the self-managed cluster on the port used by your authentication listener. Update both inbound rules on VPC security groups and outbound rules on the self-managed cluster firewall.