# Get started using Amazon MSK This tutorial shows you an example of how you can create an MSK cluster, produce and consume data, and monitor the health of your cluster using metrics. This example doesn't represent all the options you can choose when you create an MSK cluster. In different parts of this tutorial, we choose default options for simplicity. This doesn't mean that they're the only options that work for setting up an MSK cluster or client instances. **Topics** + [Step 1: Create an MSK Provisioned cluster](create-cluster.md) + [Step 2: Create an IAM role granting access to create topics on the Amazon MSK cluster](create-client-iam-role.md) + [Step 3: Create a client machine](create-client-machine.md) + [Step 4: Create a topic in the Amazon MSK cluster](create-topic.md) + [Step 5: Produce and consume data](produce-consume.md) + [Step 6: Use Amazon CloudWatch to view Amazon MSK metrics](view-metrics.md) + [Step 7: Delete the AWS resources created for this tutorial](delete-cluster.md) # Step 1: Create an MSK Provisioned cluster In this step of [Getting Started Using Amazon MSK](getting-started.md), you create an Amazon MSK Provisioned cluster. You use the **Quick create** option in the AWS Management Console to create this cluster. **To create an Amazon MSK cluster using the AWS Management Console**Create a cluster using the AWS Management Console 1. Sign in to the AWS Management Console, and open the Amazon MSK console at [https://console.aws.amazon.com/msk/home?region=us-east-1\$1/home/](https://console.aws.amazon.com/msk/home?region=us-east-1#/home/). 1. Choose **Create cluster**. 1. For **Creation method**, leave the **Quick create** option selected. The **Quick create** option lets you create a cluster with default settings. 1. For **Cluster name**, enter a descriptive name for your cluster. For example, **MSKTutorialCluster**. 1. For **General cluster properties**, do the following: 1. For **Cluster type**, choose **Provisioned**. 1. Choose an **Apache Kafka version** to run on the brokers. Choose **View version compatibility** to see a comparison table. 1. For **Broker type**, choose either Standard or Express brokers. 1. Choose a **Broker size**. 1. From the table under **All cluster settings**, copy the values of the following settings and save them because you need them later in this tutorial: + VPC + Subnets + Security groups associated with VPC 1. Choose **Create cluster**. 1. Check the cluster **Status** on the **Cluster summary** page. The status changes from **Creating** to **Active** as Amazon MSK provisions the cluster. When the status is **Active**, you can connect to the cluster. For more information about cluster status, see [Understand MSK Provisioned cluster states](msk-cluster-states.md). **Next Step** [Step 2: Create an IAM role granting access to create topics on the Amazon MSK cluster](create-client-iam-role.md) # Step 2: Create an IAM role granting access to create topics on the Amazon MSK cluster In this step, you perform two tasks. The first task is to create an IAM policy that grants access to create topics on the cluster and to send data to those topics. The second task is to create an IAM role and associate this policy with it. In a later step, you create a client machine that assumes this role and uses it to create a topic on the cluster and to send data to that topic. **To create an IAM policy that makes it possible to create topics and write to them**Create an IAM policy 1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. On the navigation pane, choose **Policies**. 1. Choose **Create policy**. 1. In **Policy editor**, choose **JSON**, and then replace the JSON in the editor window with the following JSON. In the following example, replace the following: + *region* with the code of the AWS Region where you created your cluster. + Example account ID, *123456789012*, with your AWS account ID. + *MSKTutorialCluster* and *MSKTutorialCluster*/*7d7131e1-25c5-4e9a-9ac5-ea85bee4da11-14*, with the name of your cluster and its ID. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kafka-cluster:Connect", "kafka-cluster:AlterCluster", "kafka-cluster:DescribeCluster" ], "Resource": [ "arn:aws:kafka:us-east-1:123456789012:cluster/MSKTutorialCluster/7d7131e1-25c5-4e9a-9ac5-ea85bee4da11-14" ] }, { "Effect": "Allow", "Action": [ "kafka-cluster:*Topic*", "kafka-cluster:WriteData", "kafka-cluster:ReadData" ], "Resource": [ "arn:aws:kafka:us-east-1:123456789012:topic/MSKTutorialCluster/*" ] }, { "Effect": "Allow", "Action": [ "kafka-cluster:AlterGroup", "kafka-cluster:DescribeGroup" ], "Resource": [ "arn:aws:kafka:us-east-1:123456789012:group/MSKTutorialCluster/*" ] } ] } ``` ------ For instructions about how to write secure policies, see [IAM access control](iam-access-control.md). 1. Choose **Next**. 1. On the **Review and create** page, do the following: 1. For **Policy name**, enter a descriptive name, such as **msk-tutorial-policy**. 1. In **Permissions defined in this policy**, review and/or edit the permissions defined in your policy. 1. (Optional) To help identify, organize, or search for the policy, choose **Add new tag** to add tags as key-value pairs. For example, add a tag to your policy with the key-value pair of **Environment** and **Test**. For more information about using tags, see [Tags for AWS Identity and Access Management resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_tags.html) in the *IAM User Guide*. 1. Choose **Create policy**. **To create an IAM role and attach the policy to it** 1. On the navigation pane, choose **Roles**, and then choose **Create role**. 1. On the **Select trusted entity** page, do the following: 1. For **Trusted entity type**, choose **AWS service**. 1. For **Service or use case**, choose **EC2**. 1. Under **Use case**, choose **EC2**. 1. Choose **Next**. 1. On the **Add permissions** page, do the following: 1. In the search box under **Permissions policies**, enter the name of the policy that you previously created for this tutorial. Then, choose the box to the left of the policy name. 1. (Optional) Set a [permissions boundary](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html). This is an advanced feature that is available for service roles, but not service-linked roles. For information about setting a permissions boundary, see [Creating roles and attaching policies (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions_create-policies.html) in the *IAM User Guide*. 1. Choose **Next**. 1. On the **Name, review, and create** page, do the following: 1. For **Role name**, enter a descriptive name, such as **msk-tutorial-role**. **Important** When you name a role, note the following: Role names must be unique within your AWS account, and can't be made unique by case. For example, don't create roles named both **PRODROLE** and **prodrole**. When a role name is used in a policy or as part of an ARN, the role name is case sensitive, however when a role name appears to customers in the console, such as during the sign-in process, the role name is case insensitive. You can't edit the name of the role after it's created because other entities might reference the role. 1. (Optional) For **Description**, enter a description for the role. 1. (Optional) To edit the use cases and permissions for the role, in **Step 1: Select trusted entities** or **Step 2: Add permissions** sections, choose **Edit**. 1. (Optional) To help identify, organize, or search for the role, choose **Add new tag** to add tags as key-value pairs. For example, add a tag to your role with the key-value pair of **ProductManager** and **John**. For more information about using tags, see [Tags for AWS Identity and Access Management resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_tags.html) in the *IAM User Guide*. 1. Review the role, and then choose **Create role**. **Next Step** [Step 3: Create a client machine](create-client-machine.md) # Step 3: Create a client machine In this step of [Get Started Using Amazon MSK](getting-started.md), you create a client machine. You use this client machine to create a topic that produces and consumes data. For simplicity, you'll create this client machine in the VPC that is associated with the MSK cluster so that the client can easily connect to the cluster. **To create a client machine**Create a client machine 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. From the Amazon EC2 console dashboard, choose **Launch instance**. 1. Under **Name and tags**, for **Name**, enter a descriptive name for your client machine so that you can easily keep track of it. For example, **MSKTutorialClient**. 1. Under **Application and OS Images (Amazon Machine Image)**, for **Amazon Machine Image (AMI)**, choose **Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type**. 1. For **Instance type**, keep the default selection of **t2.micro**. 1. Under **Key pair (login)**, choose an existing key pair or create a new one. If you don't require a key pair to connect to your instance, you can choose **Proceed without a key pair (not recommended)**. To create a new key pair, do the following: 1. Choose **Create new key pair**. 1. For **Key pair name**, enter **MSKKeyPair**. 1. For **Key pair type** and **Private key file format**, keep the default selections. 1. Choose **Create key pair**. Alternatively, you can use an existing key pair. 1. Scroll down the page and expand the **Advanced details** section, then do the following: 1. For **IAM instance profile**, choose an IAM role that you want the client machine to assume. If you don't have an IAM role, do the following: 1. Choose **Create new IAM profile**. 1. Perform the steps mentioned in [Step 2: Create an IAM role](create-client-iam-role.md). 1. Choose **Launch instance**. 1. Choose **View Instances**. Then, in the **Security Groups** column, choose the security group that is associated with your new instance. Copy the ID of the security group, and save it for later. 1. Open the Amazon VPC console at [https://console.aws.amazon.com/vpc/](https://console.aws.amazon.com/vpc/). 1. In the navigation pane, choose **Security Groups**. Find the security group whose ID you saved in [Step 1: Create an MSK Provisioned cluster](create-cluster.md). 1. In the **Inbound Rules** tab, choose **Edit inbound rules**. 1. Choose **Add rule**. 1. In the new rule, choose **All traffic** in the **Type** column. In the second field in the **Source** column, select the security group of your client machine. This is the group whose name you saved after you launched the client machine instance. 1. Choose **Save rules**. Now the cluster's security group can accept traffic that comes from the client machine's security group. **Next Step** [Step 4: Create a topic in the Amazon MSK cluster](create-topic.md) # Step 4: Create a topic in the Amazon MSK cluster In this step of [Getting Started Using Amazon MSK](getting-started.md), you can create a topic using one of two approaches: using native AWS tools with the CreateTopic API, or using Apache Kafka AdminClient tools on a client machine. **Warning** When using AWS tools with the CreateTopic API, verify that your cluster meets the requirements. For details, see [Requirements for using topic APIs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-topic-operations-information.html#topic-operations-requirements). **Warning** When using the AdminClient approach, Apache Kafka version numbers used in this tutorial are examples only. We recommend that you use the same version of the client as your MSK cluster version. An older client version might be missing certain features and critical bug fixes. **Topics** + [Creating a topic using AWS tools](#create-topic-aws-tools) + [Determining your MSK cluster version](#find-msk-cluster-version) + [Creating a topic on the client machine](#create-topic-client-machine) ## Creating a topic using AWS tools You can create topics in your MSK cluster using AWS tools such as the AWS CLI, AWS SDKs, or the AWS Management Console. This approach provides a streamlined way to manage topics without requiring direct access to Kafka client tools. For detailed information about creating topics using the AWS tools, see the [CreateTopic API developer guide](https://docs.aws.amazon.com/msk/latest/developerguide/msk-create-topic.html). ## Determining your MSK cluster version 1. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/](https://console.aws.amazon.com/msk/). 1. In the navigation bar, choose the Region where you created the MSK cluster. 1. Choose the MSK cluster. 1. Note the version of Apache Kafka used on the cluster. 1. Replace instances of Amazon MSK version numbers in this tutorial with the version obtained in Step 3. ## Creating a topic on the client machine 1. **Connect to your client machine.** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. Then, select the check box beside the name of the client machine that you created in [Step 3: Create a client machine](create-client-machine.md). 1. Choose **Actions**, and then choose **Connect**. Follow the instructions in the console to connect to your client machine. 1. **Install Java and set up the Kafka version environment variable.** 1. Install Java on the client machine by running the following command. ``` sudo yum -y install java-11 ``` 1. Store the [Kafka version](#find-msk-cluster-version) of your MSK cluster in the environment variable, `KAFKA_VERSION`, as shown in the following command. You'll need this information throughout the setup. ``` export KAFKA_VERSION={KAFKA VERSION} ``` For example, if you're using version 3.6.0, use the following command. ``` export KAFKA_VERSION=3.6.0 ``` 1. **Download and extract Apache Kafka.** 1. Run the following command to download Apache Kafka. ``` wget https://archive.apache.org/dist/kafka/$KAFKA_VERSION/kafka_2.13-$KAFKA_VERSION.tgz ``` **Note** The following list presents some alternative Kafka download information that you can use, if you encounter any issues. If you encounter connectivity issues or want to use a mirror site, try using the Apache mirror selector, as shown in the following command. ``` wget https://www.apache.org/dyn/closer.cgi?path=/kafka/$KAFKA_VERSION/kafka_2.13-$KAFKA_VERSION.tgz ``` Download an appropriate version directly from the [Apache Kafka website](https://kafka.apache.org/downloads). 1. Run the following command in the directory where you downloaded the TAR file in the previous step. ``` tar -xzf kafka_2.13-$KAFKA_VERSION.tgz ``` 1. Store the full path to the newly created directory inside the `KAFKA_ROOT` environment variable. ``` export KAFKA_ROOT=$(pwd)/kafka_2.13-$KAFKA_VERSION ``` 1. **Set up authentication for your MSK cluster.** 1. [Find the latest version](https://github.com/aws/aws-msk-iam-auth/releases/latest) of the Amazon MSK IAM client library. This library allows your client machine to access the MSK cluster using IAM authentication. 1. Using the following commands, navigate to the `$KAFKA_ROOT/libs` directory and download the associated Amazon MSK IAM JAR that you found in the previous step. Make sure to replace *\$1LATEST VERSION\$1* with the actual version number you're downloading. ``` cd $KAFKA_ROOT/libs ``` ``` wget https://github.com/aws/aws-msk-iam-auth/releases/latest/download/aws-msk-iam-auth-{LATEST VERSION}-all.jar ``` **Note** Before running any Kafka commands that interact with your MSK cluster, you might need to add the Amazon MSK IAM JAR file to your Java classpath. Set the `CLASSPATH` environment variable, as shown in the following example. ``` export CLASSPATH=$KAFKA_ROOT/libs/aws-msk-iam-auth-{LATEST VERSION}-all.jar ``` This sets the `CLASSPATH` for your entire session, making the JAR available to all subsequent Kafka commands. 1. Go to the `$KAFKA_ROOT/config` directory to create the client configuration file. ``` cd $KAFKA_ROOT/config ``` 1. Copy the following property settings and paste them into a new file. Save the file as **client.properties**. ``` security.protocol=SASL_SSL sasl.mechanism=AWS_MSK_IAM sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler ``` 1. **(Optional) Adjust the Java heap size for Kafka tools. ** If you encounter any memory-related issues or you're working with a large number of topics or partitions, you can adjust the Java heap size. To do this, set the `KAFKA_HEAP_OPTS` environment variable before running Kafka commands. The following example sets both the maximum and initial heap size to 512 megabytes. Adjust these values according to your specific requirements and available system resources. ``` export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M" ``` 1. **Get your cluster connection information.** 1. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/](https://console.aws.amazon.com/msk/). 1. Wait for the status of your cluster to become **Active**. This might take several minutes. After the status becomes **Active**, choose the cluster name. This takes you to a page containing the cluster summary. 1. Choose **View client information**. 1. Copy the connection string for the private endpoint. You'll get three endpoints for each of the brokers. Store one of these connection strings in the environment variable `BOOTSTRAP_SERVER`, as shown in the following command. Replace ** with the actual value of the connection string. ``` export BOOTSTRAP_SERVER= ``` 1. **Run the following command to create the topic.** ``` $KAFKA_ROOT/bin/kafka-topics.sh --create --bootstrap-server $BOOTSTRAP_SERVER --command-config $KAFKA_ROOT/config/client.properties --replication-factor 3 --partitions 1 --topic MSKTutorialTopic ``` If you get a `NoSuchFileException` for the `client.properties` file, make sure that this file exists in the current working directory within the Kafka bin directory. **Note** If you prefer not to set the `CLASSPATH` environment variable for your entire session, you can alternatively prefix each Kafka command with the `CLASSPATH` variable. This approach applies the classpath only to that specific command. ``` CLASSPATH=$KAFKA_ROOT/libs/aws-msk-iam-auth-{LATEST VERSION}-all.jar \ $KAFKA_ROOT/bin/kafka-topics.sh --create \ --bootstrap-server $BOOTSTRAP_SERVER \ --command-config $KAFKA_ROOT/config/client.properties \ --replication-factor 3 \ --partitions 1 \ --topic MSKTutorialTopic ``` 1. **(Optional) Verify that the topic was created successfully.** 1. If the command succeeds, you should see the following message: `Created topic MSKTutorialTopic.` 1. List all topics to confirm your topic exists. ``` $KAFKA_ROOT/bin/kafka-topics.sh --list --bootstrap-server $BOOTSTRAP_SERVER --command-config $KAFKA_ROOT/config/client.properties ``` If the command is unsuccessful or you run into an error, see [Troubleshoot your Amazon MSK cluster](troubleshooting.md) for troubleshooting information. 1. **(Optional) Delete the environment variables you used in this tutorial.** If you want to keep your environment variables for the next steps in this tutorial, skip this step. Otherwise, you can unset these variables, as shown in the following example. ``` unset KAFKA_VERSION KAFKA_ROOT BOOTSTRAP_SERVER CLASSPATH KAFKA_HEAP_OPTS ``` **Next Step** [Step 5: Produce and consume data](produce-consume.md) # Step 5: Produce and consume data In this step of [Get Started Using Amazon MSK](getting-started.md), you produce and consume data. **To produce and consume messages**Produce and consume messages 1. Run the following command to start a console producer. ``` $KAFKA_ROOT/bin/kafka-console-producer.sh --broker-list $BOOTSTRAP_SERVER --producer.config $KAFKA_ROOT/config/client.properties --topic MSKTutorialTopic ``` 1. Enter any message that you want, and press **Enter**. Repeat this step two or three times. Every time you enter a line and press **Enter**, that line is sent to your Apache Kafka cluster as a separate message. 1. Keep the connection to the client machine open, and then open a second, separate connection to that machine in a new window. Because this is a new session, set the `KAFKA_ROOT` and `BOOTSTRAP_SERVER` environment variables again. For information about how to set these environment variables, see [Creating a topic on the client machine](create-topic.md#create-topic-client-machine). 1. Run the following command with your second connection string to the client machine to create a console consumer. ``` $KAFKA_ROOT/bin/kafka-console-consumer.sh --bootstrap-server $BOOTSTRAP_SERVER --consumer.config $KAFKA_ROOT/config/client.properties --topic MSKTutorialTopic --from-beginning ``` You should start seeing the messages you entered earlier when you used the console producer command. 1. Enter more messages in the producer window, and watch them appear in the consumer window. **Next Step** [Step 6: Use Amazon CloudWatch to view Amazon MSK metrics](view-metrics.md) # Step 6: Use Amazon CloudWatch to view Amazon MSK metrics In this step of [Getting Started Using Amazon MSK](getting-started.md), you look at the Amazon MSK metrics in Amazon CloudWatch. **To view Amazon MSK metrics in CloudWatch**View metrics in CloudWatch 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. Choose the **All metrics** tab, and then choose **AWS/Kafka**. 1. To view broker-level metrics, choose **Broker ID, Cluster Name**. For cluster-level metrics, choose **Cluster Name**. 1. (Optional) In the graph pane, select a statistic and a time period, and then create a CloudWatch alarm using these settings. **Next Step** [Step 7: Delete the AWS resources created for this tutorial](delete-cluster.md) # Step 7: Delete the AWS resources created for this tutorial In the final step of [Getting Started Using Amazon MSK](getting-started.md), you delete the MSK cluster and the client machine that you created for this tutorial. **To delete the resources using the AWS Management Console**Delete resources using the AWS Management Console 1. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/](https://console.aws.amazon.com/msk/). 1. Choose the name of your cluster. For example, **MSKTutorialCluster**. 1. Choose **Actions**, then choose **Delete**. 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. Choose the instance that you created for your client machine, for example, **MSKTutorialClient**. 1. Choose **Instance state**, then choose **Terminate instance**. **To delete the IAM policy and role**Delete IAM policy and role 1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. On the navigation pane, choose **Roles**. 1. In the search box, enter the name of the IAM role that you created for this tutorial. 1. Choose the role. Then choose **Delete role**, and confirm the deletion. 1. On the navigation pane, choose **Policies**. 1. In the search box, enter the name of the policy that you created for this tutorial. 1. Choose the policy to open its summary page. On the policy's **Summary** page, choose **Delete policy**. 1. Choose **Delete**.