How do I get started with encryption?
When creating an MSK cluster, you can specify encryption settings in JSON format. The following is an example.
{ "EncryptionAtRest": { "DataVolumeKMSKeyId": "arn:aws:kms:us-east-1:123456789012:key/abcdabcd-1234-abcd-1234-abcd123e8e8e" }, "EncryptionInTransit": { "InCluster": true, "ClientBroker": "TLS" } }
For DataVolumeKMSKeyId
, you can specify a customer managed key or the
AWS managed key for MSK in your account
(alias/aws/kafka
).
If you don't specify EncryptionAtRest
, Amazon MSK still encrypts your data at
rest under the AWS managed key. To determine which key your cluster is using, send a
GET
request or invoke the DescribeCluster
API operation.
For EncryptionInTransit
, the default value of InCluster
is
true, but you can set it to false if you don't want Amazon MSK to encrypt your data as it
passes between brokers.
To specify the encryption mode for data in transit between clients and brokers, set
ClientBroker
to one of three values: TLS
,
TLS_PLAINTEXT
, or PLAINTEXT
.
To specify encryption settings when creating a cluster
Save the contents of the previous example in a file and give the file any name that you want. For example, call it
encryption-settings.json
.-
Run the
create-cluster
command and use theencryption-info
option to point to the file where you saved your configuration JSON. The following is an example. Replace{YOUR MSK VERSION}
with a version that matches the Apache Kafka client version. For information on how to find your MSK cluster version, see To find the version of your MSK cluster. Be aware that using an Apache Kafka client version that is not the same as your MSK cluster version may lead to Apache Kafka data corruption, loss and down time.aws kafka create-cluster --cluster-name "ExampleClusterName" --broker-node-group-info file://brokernodegroupinfo.json --encryption-info file://encryptioninfo.json --kafka-version "
{YOUR MSK VERSION}
" --number-of-broker-nodes 3The following is an example of a successful response after running this command.
{ "ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/SecondTLSTest/abcdabcd-1234-abcd-1234-abcd123e8e8e", "ClusterName": "ExampleClusterName", "State": "CREATING" }
To test TLS encryption
Create a client machine following the guidance in Step 3: Create a client machine.
-
Install Apache Kafka on the client machine.
-
In this example we use the JVM truststore to talk to the MSK cluster. To do this, first create a folder named
/tmp
on the client machine. Then, go to thebin
folder of the Apache Kafka installation, and run the following command. (Your JVM path might be different.)cp /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64/jre/lib/security/cacerts /tmp/kafka.client.truststore.jks
-
While still in the
bin
folder of the Apache Kafka installation on the client machine, create a text file namedclient.properties
with the following contents.security.protocol=SSL ssl.truststore.location=/tmp/kafka.client.truststore.jks
-
Run the following command on a machine that has the AWS CLI installed, replacing
clusterARN
with the ARN of your cluster.aws kafka get-bootstrap-brokers --cluster-arn
clusterARN
A successful result looks like the following. Save this result because you need it for the next step.
{ "BootstrapBrokerStringTls": "a-1.example.g7oein.c2.kafka.us-east-1.amazonaws.com:0123,a-3.example.g7oein.c2.kafka.us-east-1.amazonaws.com:0123,a-2.example.g7oein.c2.kafka.us-east-1.amazonaws.com:0123" }
-
Run the following command, replacing
BootstrapBrokerStringTls
with one of the broker endpoints that you obtained in the previous step.<path-to-your-kafka-installation>
/bin/kafka-console-producer.sh --broker-listBootstrapBrokerStringTls
--producer.config client.properties --topic TLSTestTopic -
Open a new command window and connect to the same client machine. Then, run the following command to create a console consumer.
<path-to-your-kafka-installation>
/bin/kafka-console-consumer.sh --bootstrap-serverBootstrapBrokerStringTls
--consumer.config client.properties --topic TLSTestTopic -
In the producer window, type a text message followed by a return, and look for the same message in the consumer window. Amazon MSK encrypted this message in transit.
For more information about configuring Apache Kafka clients to work with encrypted data, see Configuring
Kafka Clients