# Use MSK Serverless clusters
This tutorial shows you an example of how you can create an MSK Serverless cluster, create a client machine that can access it, and use the client to create topics on the cluster and to write data to those topics. This exercise doesn't represent all the options that you can choose when you create a serverless cluster. In different parts of this exercise, we choose default options for simplicity. This doesn't mean that they're the only options that work for setting up a serverless cluster. You can also use the AWS CLI or the Amazon MSK API. For more information, see the [Amazon MSK API Reference 2.0](https://docs.aws.amazon.com/MSK/2.0/APIReference/what-is-msk.html).
**Topics**
+ [Create an MSK Serverless cluster](create-serverless-cluster.md)
+ [Create an IAM role for topics on MSK Serverless cluster](create-iam-role.md)
+ [Create a client machine to access MSK Serverless cluster](create-serverless-cluster-client.md)
+ [Create an Apache Kafka topic](msk-serverless-create-topic.md)
+ [Produce and consume data in MSK Serverless](msk-serverless-produce-consume.md)
+ [Delete resources that you created for MSK Serverless](delete-resources.md)
# Create an MSK Serverless cluster
In this step, you perform two tasks. First, you create an MSK Serverless cluster with default settings. Second, you gather information about the cluster. This is information that you need in later steps when you create a client that can send data to the cluster.
**To create a serverless cluster**
1. Sign in to the AWS Management Console, and open the Amazon MSK console at [https://console.aws.amazon.com/msk/home](https://console.aws.amazon.com/msk/home).
1. Choose **Create cluster**.
1. For **Creation method**, leave the **Quick create** option selected. The **Quick create** option lets you create a serverless cluster with default settings.
1. For **Cluster name**, enter a descriptive name, such as **msk-serverless-tutorial-cluster**.
1. For **General cluster properties**, choose **Serverless** as the **Cluster type**. Use the default values for the remaining **General cluster** properties.
1. Note the table under **All cluster settings**. This table lists the default values for important settings such as networking and availability, and indicates whether you can change each setting after you create the cluster. To change a setting before you create the cluster, you should choose the **Custom create** option under **Creation method**.
**Note**
You can connect clients from up to five different VPCs with MSK Serverless clusters. To help client applications switch over to another Availability Zone in the event of an outage, you must specify at least two subnets in each VPC.
1. Choose **Create cluster**.
**To gather information about the cluster**
1. In the **Cluster summary** section, choose **View client information**. This button remains grayed out until Amazon MSK finishes creating the cluster. You might need to wait a few minutes until the button becomes active so you can use it.
1. Copy the string under the label **Endpoint**. This is your bootstrap server string.
1. Choose the **Properties** tab.
1. Under the **Networking settings** section, copy the IDs of the subnets and the security group and save them because you need this information later to create a client machine.
1. Choose any of the subnets. This opens the Amazon VPC Console. Find the ID of the Amazon VPC that is associated with the subnet. Save this Amazon VPC ID for later use.
**Next Step**
[Create an IAM role for topics on MSK Serverless cluster](create-iam-role.md)
# Create an IAM role for topics on MSK Serverless cluster
In this step, you perform two tasks. The first task is to create an IAM policy that grants access to create topics on the cluster and to send data to those topics. The second task is to create an IAM role and associate this policy with it. In a later step, we create a client machine that assumes this role and uses it to create a topic on the cluster and to send data to that topic.
**To create an IAM policy that makes it possible to create topics and write to them**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. On the navigation pane, choose **Policies**.
1. Choose **Create Policy**.
1. Choose the **JSON** tab, then replace the JSON in the editor window with the following JSON.
In the following example, replace the following:
+ *region* with the code of the AWS Region where you created your cluster.
+ Example account ID, *123456789012*, with your AWS account ID.
+ *msk-serverless-tutorial-cluster*/*c07c74ea-5146-4a03-add1-9baa787a5b14-s3* and *msk-serverless-tutorial-cluster* with your serverless cluster ID and topic name.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kafka-cluster:Connect",
"kafka-cluster:DescribeCluster"
],
"Resource": [
"arn:aws:kafka:us-east-1:123456789012:cluster/msk-serverless-tutorial-cluster/c07c74ea-5146-4a03-add1-9baa787a5b14-s3"
]
},
{
"Effect": "Allow",
"Action": [
"kafka-cluster:CreateTopic",
"kafka-cluster:WriteData",
"kafka-cluster:DescribeTopic"
],
"Resource": [
"arn:aws:kafka:us-east-1:123456789012:topic/msk-serverless-tutorial-cluster/*"
]
}
]
}
```
------
For instructions about how to write secure policies, see [IAM access control](iam-access-control.md).
1. Choose **Next: Tags**.
1. Choose **Next: Review**.
1. For the policy name, enter a descriptive name, such as **msk-serverless-tutorial-policy**.
1. Choose **Create policy**.
**To create an IAM role and attach the policy to it**
1. On the navigation pane, choose **Roles**.
1. Choose **Create role**.
1. Under **Common use cases**, choose **EC2**, then choose **Next: Permissions**.
1. In the search box, enter the name of the policy that you previously created for this tutorial. Then select the box to the left of the policy.
1. Choose **Next: Tags**.
1. Choose **Next: Review**.
1. For the role name, enter a descriptive name, such as **msk-serverless-tutorial-role**.
1. Choose **Create role**.
**Next Step**
[Create a client machine to access MSK Serverless cluster](create-serverless-cluster-client.md)
# Create a client machine to access MSK Serverless cluster
In the step, you perform two tasks. The first task is to create an Amazon EC2 instance to use as an Apache Kafka client machine. The second task is to install Java and Apache Kafka tools on the machine.
**To create a client machine**
1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/).
1. Choose **Launch instance**.
1. Enter a descriptive **Name** for your client machine, such as **msk-serverless-tutorial-client**.
1. Leave the **Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type** selected for **Amazon Machine Image (AMI) type**.
1. Leave the **t2.micro** instance type selected.
1. Under **Key pair (login)**, choose **Create a new key pair**. Enter **MSKServerlessKeyPair** for **Key pair name**. Then choose **Download Key Pair**. Alternatively, you can use an existing key pair.
1. For **Network settings**, choose **Edit**.
1. Under **VPC**, enter the ID of the virtual private cloud (VPC) for your serverless cluster . This is the VPC based on the Amazon VPC service whose ID you saved after you created the cluster.
1. For **Subnet**, choose the subnet whose ID you saved after you created the cluster.
1. For **Firewall (security groups)**, select the security group associated with the cluster. This value works if that security group has an inbound rule that allows traffic from the security group to itself. With such a rule, members of the same security group can communicate with each other. For more information, see [Security group rules](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#SecurityGroupRules) in the Amazon VPC Developer Guide.
1. Expand the **Advanced details** section and choose the IAM role that you created in [Create an IAM role for topics on MSK Serverless cluster](create-iam-role.md).
1. Choose **Launch**.
1. In the left navigation pane, choose **Instances**. Then choose the check box in the row that represents your newly created Amazon EC2 instance. From this point forward, we call this instance the *client machine*.
1. Choose **Connect** and follow the instructions to connect to the client machine.
**To set up Apache Kafka client tools on the client machine**
1. To install Java, run the following command on the client machine:
```
sudo yum -y install java-11
```
1. To get the Apache Kafka tools that we need to create topics and send data, run the following commands:
```
wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz
```
```
tar -xzf kafka_2.12-2.8.1.tgz
```
**Note**
After extracting the Kafka archive, make sure that the scripts in the `bin` directory have proper execute permissions. To do this, run the following command.
```
chmod +x kafka_2.12-2.8.1/bin/*.sh
```
1. Go to the `kafka_2.12-2.8.1/libs` directory, then run the following command to download the Amazon MSK IAM JAR file. The Amazon MSK IAM JAR makes it possible for the client machine to access the cluster.
```
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v2.3.0/aws-msk-iam-auth-2.3.0-all.jar
```
Using this command, you can also [download other or newer versions](https://github.com/aws/aws-msk-iam-auth/releases) of Amazon MSK IAM JAR file.
1. Go to the `kafka_2.12-2.8.1/bin` directory. Copy the following property settings and paste them into a new file. Name the file `client.properties` and save it.
```
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```
**Next Step**
[Create an Apache Kafka topic](msk-serverless-create-topic.md)
# Create an Apache Kafka topic
In this step, you use the previously created client machine to create a topic on the serverless cluster.
**Topics**
+ [Setting up your environment for creating topics](#msk-serverless-create-topic-prerequisites)
+ [Creating a topic and writing data to it](#msk-serverless-create-topic-procedure)
## Setting up your environment for creating topics
+ Before creating a topic, make sure that you've downloaded the AWS MSK IAM JAR file to your Kafka installation's `libs/` directory. If you haven't done this yet, run the following command in your Kafka's `libs/` directory.
```
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v2.3.0/aws-msk-iam-auth-2.3.0-all.jar
```
This JAR file is required for IAM authentication with your MSK Serverless cluster.
+ When running Kafka commands, you might need to make sure the `classpath` includes the AWS MSK IAM JAR file. To do this, do one of the following:
+ Set the `CLASSPATH` environment variable to include your Kafka libraries as shown in the following example.
```
export CLASSPATH=/libs/*:/libs/aws-msk-iam-auth-2.3.0-all.jar
```
+ Run Kafka commands using the full Java command with explicit `classpath`, as shown in the following example.
```
java -cp "/libs/*:/libs/aws-msk-iam-auth-2.3.0-all.jar" org.apache.kafka.tools.TopicCommand --bootstrap-server $BS --command-config client.properties --create --topic msk-serverless-tutorial --partitions 6
```
## Creating a topic and writing data to it
1. In the following `export` command, replace *my-endpoint* with the bootstrap-server string you that you saved after you created the cluster. Then, go to the `kafka_2.12-2.8.1/bin` directory on the client machine and run the `export` command.
```
export BS=my-endpoint
```
1. Run the following command to create a topic called `msk-serverless-tutorial`.
```
/bin/kafka-topics.sh --bootstrap-server $BS --command-config client.properties --create --topic msk-serverless-tutorial --partitions 6
```
**Next Step**
[Produce and consume data in MSK Serverless](msk-serverless-produce-consume.md)
# Produce and consume data in MSK Serverless
In this step, you produce and consume data using the topic that you created in the previous step.
**To produce and consume messages**
1. Run the following command to create a console producer.
```
/bin/kafka-console-producer.sh --broker-list $BS --producer.config client.properties --topic msk-serverless-tutorial
```
1. Enter any message that you want, and press **Enter**. Repeat this step two or three times. Every time you enter a line and press **Enter**, that line is sent to your cluster as a separate message.
1. Keep the connection to the client machine open, and then open a second, separate connection to that machine in a new window.
1. Use your second connection to the client machine to create a console consumer with the following command. Replace *my-endpoint* with the bootstrap server string that you saved after you created the cluster.
```
/bin/kafka-console-consumer.sh --bootstrap-server my-endpoint --consumer.config client.properties --topic msk-serverless-tutorial --from-beginning
```
You start seeing the messages you entered earlier when you used the console producer command.
1. Enter more messages in the producer window, and watch them appear in the consumer window.
If you encounter `classpath` issues while running these commands, make sure that you're running them from the correct directory. Also, make sure that the AWS MSK IAM JAR is in the `libs` directory. Alternatively, you can run Kafka commands using the full Java command with explicit `classpath`, as shown in the following example.
```
java -cp "kafka_2.12-2.8.1/libs/*:kafka_2.12-2.8.1/libs/aws-msk-iam-auth-2.3.0-all.jar" org.apache.kafka.tools.ConsoleProducer —broker-list $BS —producer.config client.properties —topic msk-serverless-tutorial
```
**Next Step**
[Delete resources that you created for MSK Serverless](delete-resources.md)
# Delete resources that you created for MSK Serverless
In this step, you delete the resources that you created in this tutorial.
**To delete the cluster**
1. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/home](https://console.aws.amazon.com/msk/home).
1. In the list of clusters, choose the cluster that you created for this tutorial.
1. For **Actions**, choose **Delete cluster**.
1. Enter `delete` in the field, then choose **Delete**.
**To stop the client machine**
1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/).
1. In the list of Amazon EC2 instances, choose the client machine that you created for this tutorial.
1. Choose **Instance state**, then choose **Terminate instance**.
1. Choose **Terminate**.
**To delete the IAM policy and role**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. On the navigation pane, choose **Roles**.
1. In the search box, enter the name of the IAM role that you created for this tutorial.
1. Choose the role. Then choose **Delete role**, and confirm the deletion.
1. On the navigation pane, choose **Policies**.
1. In the search box, enter the name of the policy that you created for this tutorial.
1. Choose the policy to open its summary page. On the policy's **Summary** page, choose **Delete policy**.
1. Choose **Delete**.