Tutorial: Using an Amazon MSK event source mapping to invoke a Lambda function

Create a security group for your interface Amazon VPC endpoints, endpointSecurityGroup, that allows inbound TCP traffic on 443 from clusterSecurityGroups. Follow the procedure in Create a security group in the Amazon EC2 documentation to create a security group. Then, follow the procedure in Add rules to a security group in the Amazon EC2 documentation to add appropriate rules.

Create a security group with the following information:

When adding your inbound rules, create a rule for each security group in clusterSecurityGroups. For each rule:

Create an endpoint connecting the Lambda service to the Amazon VPC containing your Amazon MSK cluster. Follow the procedure in Create an interface endpoint.

Create an interface endpoint with the following information:

Create an endpoint connecting the AWS STS service to the Amazon VPC containing your Amazon MSK cluster. Follow the procedure in Create an interface endpoint.

Create an interface endpoint with the following information:

For each security group associated with your Amazon MSK cluster, that is, in clusterSecurityGroups, allow the following:

Some of this traffic is allowed by default security group rules, so if your cluster is attached to a single security group, and that group has default rules, additional rules are not necessary. To adjust security group rules, follow the procedures in Add rules to a security group in the Amazon EC2 documentation.

Add rules to your security groups with the following information:

Write an IAM policy document (a JSON document), clusterAuthPolicy, that allows Lambda to read from your Kafka topic in your Amazon MSK cluster using your Kafka consumer group. Lambda requires a Kafka consumer group to be set when reading.

Alter the following template to align with your prerequisites:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeClusterDynamicConfiguration"
            ],
            "Resource": [
                "arn:aws:kafka:region:account-id:cluster/mskClusterName/cluster-uuid",
                "arn:aws:kafka:region:account-id:topic/mskClusterName/cluster-uuid/mskTopicName",
                "arn:aws:kafka:region:account-id:group/mskClusterName/cluster-uuid/mskGroupName"
            ]
        }
    ]
}       

For more information, consult IAM role-based authentication. When writing your policy:

Identify the Amazon MSK, Amazon EC2 and CloudWatch permissions required for Lambda to discover and connect your Amazon MSK cluster, and log those events.

The AWSLambdaMSKExecutionRole managed policy permissively defines the required permissions. Use it in the following steps.

In a production environment, assess AWSLambdaMSKExecutionRole to restrict your execution role policy based on the principle of least privilege, then write a policy for your role that replaces this managed policy.

Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

In the navigation pane on the left, choose Policies.

Choose Create policy.

In the Policy editor section, choose the JSON option.

Paste clusterAuthPolicy.

When you are finished adding permissions to the policy, choose Next.

On the Review and create page, type a Policy Name and a Description (optional) for the policy that you are creating. Review Permissions defined in this policy to see the permissions that are granted by your policy.

Choose Create policy to save your new policy.

Open the Roles page in the IAM console.

Choose Create role.

Under Trusted entity type, choose AWS service.

Under Use case, choose Lambda.

Choose Next.

Select the following policies:

Choose Next.

For Role name, enter lambdaAuthRole and then choose Create role.

Open the Lambda console and select Create function from the header.

Select Author from scratch.

For Function name, provide an appropriate name of your choice.

For Runtime, choose the Latest supported version of Node.js to use the code provided in this tutorial.

Choose Change default execution role.

Select Use an existing role.

For Existing role, select lambdaAuthRole.

Navigate to your Lambda function's overview page.

In the function overview section, choose Add trigger on the bottom left.

In the Select a source dropdown, select Amazon MSK.

Don't set authentication.

For MSK cluster, select your cluster's name.

For Batch size, enter 1. This step makes this feature easier to test, and is not an ideal value in production.

For Topic name, provide the name of your Kafka topic.

For Consumer group ID, provide the id of your Kafka consumer group.

Navigate to your Lambda function's overview page.

Choose the Code tab.

Enter the provided code into the Code source IDE.

In the Code source navigation bar, choose Deploy.

Use your Kafka admin host to generate Kafka events using the kafka-console-producer CLI. For more information, see Write some events into the topic in the Kafka documentation. Send enough events to fill up the batch defined by batch size for your event source mapping defined in the previous step, or Lambda will wait for more information to invoke.

If your function runs, Lambda writes what happened to CloudWatch. In the console, navigate to your Lambda function's detail page.

Select the Configuration tab.

From the sidebar, select Monitoring and operations tools.

Identify the CloudWatch log group under Logging configuration. The log group should start with /aws/lambda. Choose the link to the log group.

In the CloudWatch console, inspect the Log events for the log events Lambda has sent to the log stream. Identify if there are log events containing the message from your Kafka event, as in the following image. If there are, you have successfully connected a Lambda function to Amazon MSK with a Lambda event source mapping.