"}}'`example:`aws dynamodb put-item --table-name SourceData_table --item '{"SourceDataId": {"S": "123"},"MessageData":{"S": "Hello World"}}'`By default, the `put-item` doesn't return any value as output if the operation succeeds. If the operation fails, it returns an error. The data is stored in DynamoDB and then sent to Kinesis Data Streams and Firehose. You use different approaches to add data into a DynamoDB table. For more information, see [Load data into tables](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SampleData.LoadData.html) in the DynamoDB documentation. | App developer |
| Verify that a new object is created in the S3 bucket. | Sign in to the AWS Management Console and monitor the S3 bucket to verify that a new object was created with the data that you sent. For more information, see [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) in the Amazon S3 documentation. | App developer, General AWS |
### Clean up resources
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up resources. | Run the `cdk destroy` command to delete all the resources used by this pattern. | App developer, General AWS |
## Related resources
+ [s3-static-site-stack.ts](https://github.com/awslabs/aws-solutions-constructs/blob/main/source/use_cases/aws-s3-static-website/lib/s3-static-site-stack.ts#L25) (GitHub repository)
+ [aws-apigateway-dynamodb module](https://github.com/awslabs/aws-solutions-constructs/tree/main/source/patterns/%40aws-solutions-constructs/aws-apigateway-dynamodb) (GitHub repository)
+ [aws-kinesisstreams-kinesisfirehose-s3 module](https://github.com/awslabs/aws-solutions-constructs/tree/main/source/patterns/%40aws-solutions-constructs/aws-kinesisstreams-kinesisfirehose-s3) (GitHub repository)
+ [Change data capture for DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) (DynamoDB documentation)
+ [Using Kinesis Data Streams to capture changes to DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html) (DynamoDB documentation)
# Implement path-based API versioning by using custom domains in Amazon API Gateway
*Corey Schnedl, Marcelo Barbosa, Mario Lopez Martinez, Anbazhagan Ponnuswamy, Gaurav Samudra, and Abhilash Vinod, Amazon Web Services*
## Summary
This pattern demonstrates how you can use the [API mappings](https://docs.aws.amazon.com/apigateway/latest/developerguide/rest-api-mappings.html) feature of [custom domains](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-custom-domains.html) to implement a path-based API versioning solution for Amazon API Gateway.
Amazon API Gateway is a fully managed service that you can use to create, publish, maintain, monitor, and secure APIs at any scale. By using the service’s custom domain feature, you can create custom domain names that are simpler with more intuitive URLs that you can provide to your API users. You can use API mappings to connect API stages to a custom domain name. After you create a domain name and configure DNS records, you use API mappings to send traffic to your APIs through your custom domain name.
After an API becomes publicly available, consumers use it. As a public API evolves, its service contract also evolves to reflect new features and capabilities. However, it’s unwise to change or remove existing features. Any breaking changes might impact the consumer’s applications and break them at runtime. API versioning is important to avoid breaking backward compatibility and breaking a contract.
You need a clear strategy for API versioning to help consumers adopt them. Versioning APIs by using path-based URLs is the most straightforward and commonly used approach. In this type of versioning, versions are explicitly defined as part of API URIs. The following example URLs show how a consumer can use the URI to specify an API version for their request:
`https://api.example.com/api/v1/orders `
`https://api.example.com/api/v2/orders `
`https://api.example.com/api/vX/orders`
This pattern uses the AWS Cloud Development Kit (AWS CDK) to build, deploy, and test a sample implementation of a scalable path-based versioning solution for your API. AWS CDK is an open source software development framework to model and provision your cloud application resources using familiar programming languages.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ Ownership of a domain is required to use this pattern’s sample repository and to use Amazon API Gateway custom domain functionality. You can use Amazon Route 53 to create and manage your domains for your organization. For information about how to register or transfer a domain with Route 53, see [Registering new domains](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-register-update.html) in the Route 53 documentation.
+ Before setting up a custom domain name for an API, you must have an [SSL/TLS certificate](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-specify-certificate-for-custom-domain-name.html) ready in AWS Certificate Manager.
+ You must create or update your DNS provider's resource record to map to your API endpoint. Without such a mapping, API requests bound for the custom domain name can’t reach API Gateway.
**Limitations **
+ A custom domain name must be unique within an AWS Region across all AWS accounts.
+ To configure API mappings with multiple levels, you must use a Regional custom domain name and use the TLS 1.2 security policy.
+ In an API mapping, the custom domain name and mapped APIs must be in the same AWS account.
+ API mappings must contain only letters, numbers, and the following characters: `$-_.+!*'()/`
+ The maximum length for the path in an API mapping is 300 characters.
+ You can have 200 API mappings with multiple levels for each domain name.
+ You can only map HTTP APIs to a Regional custom domain name with the TLS 1.2 security policy.
+ You can't map WebSocket APIs to the same custom domain name as an HTTP API or REST API.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
**Product versions**
+ This sample implementation uses [AWS CDK in TypeScript](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-typescript.html) version 2.149.0.
## Architecture
The following diagram shows the architecture workflow.
![\[Workflow using API mappings and custom domains to implement a path-based API versioning solution.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e1b32d2b-410f-4ace-967e-f0b8aaf0304c/images/fa9f04f1-efa6-4fb1-a541-ae3da4076b00.png)
The diagram illustrates the following:
1. The API user sends a request to Amazon API Gateway with a custom domain name.
1. API Gateway dynamically routes the user’s request to an appropriate instance and stage of API Gateway, based on the path indicated in the URL of the request. The following table shows an example of how the different URL-based paths can be routed to specific stages for different instances of API Gateway.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-path-based-api-versioning-by-using-custom-domains.html)
1. The destination API Gateway instance processes the request and returns the result to the user.
**Automation and scale**
We recommend that you use separate AWS CloudFormation stacks for each version of your API. With this approach, you can have complete isolation between the backend APIs that can be routed to by the custom domain API mapping feature. An advantage of this approach is that different versions of your API can be deployed or removed independently without introducing the risk of modifying another API. This approach increases resilience through isolation of CloudFormation stacks. Also, it provides you with different back-end options for your API such as AWS Lambda, AWS Fargate, HTTP endpoints, and actions of AWS services.
You can use Git branching strategies, such as [Gitflow](https://docs.aws.amazon.com/prescriptive-guidance/latest/choosing-git-branch-approach/gitflow-branching-strategy.html), in combination with isolated CloudFormation stacks to manage the source code that’s deployed to different versions of the API. By using this option, you can maintain different versions of your API without the need to duplicate the source code for new versions. With Gitflow, you can add tags to commits within your git repository as releases are performed. As a result, you have a complete snapshot of the source code related to a specific release. As updates need to be performed, you can check out the code from a specific release, make updates, and then deploy the updated source code to the CloudFormation stack that aligns with the corresponding major version. This approach reduces the risk of breaking another API version because each version of the API has isolated source code and is deployed to separate CloudFormation stacks.
## Tools
**AWS services**
+ [Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) helps you create, publish, maintain, monitor, and secure REST, HTTP, and WebSocket APIs at any scale.
+ [AWS Certificate Manager (ACM)](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html) helps you create, store, and renew public and private SSL/TLS X.509 certificates and keys that protect your AWS websites and applications.
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/v2/guide/home.html) is an open-source software development framework for defining your cloud infrastructure in code and provisioning it through CloudFormation. This pattern’s sample implementation uses the [AWS CDK in TypeScript](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-typescript.html). Working with the AWS CDK in TypeScript uses familiar tools, including the Microsoft TypeScript compiler (`tsc`), [Node.js](https://nodejs.org/), and the node package manager (`npm`). If you prefer, you can use [Yarn](https://yarnpkg.com/) although the examples in this pattern use `npm`. The modules that comprise the [AWS Construct Library](https://docs.aws.amazon.com/cdk/v2/guide/libraries.html#libraries-construct) are distributed through the `npm `repository, [npmjs.org](https://docs.npmjs.com/).
+ [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and AWS Regions.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) is a highly available and scalable DNS web service.
+ [AWS WAF](https://docs.aws.amazon.com/waf/latest/developerguide/what-is-aws-waf.html) is a web application firewall that helps you monitor HTTP and HTTPS requests that are forwarded to your protected web application resources.
**Other tools**
+ [Bruno](https://www.usebruno.com/) is an open source, git-friendly API testing client.
+ [cdk-nag](https://github.com/cdklabs/cdk-nag) is an open source utility that checks AWS CDK applications for best practices by using rule packs.
**Code repository**
The code for this pattern is available in the GitHub [path-based-versioning-with-api-gateway](https://github.com/aws-samples/path-based-versioning-with-api-gateway) repository.
## Best practices
+ Use a robust continuous integration and continuous delivery (CI/CD) pipeline to automate the testing and deployment of your CloudFormation stacks that are built with the AWS CDK. For more information related to this recommendation, see the [AWS Well-Architected DevOps Guidance](https://docs.aws.amazon.com/wellarchitected/latest/devops-guidance/devops-guidance.html).
+ AWS WAF is a managed firewall that easily integrates with services like Amazon API Gateway. Although AWS WAF isn’t a necessary component for this versioning pattern to work, we recommend as a security best practice to include AWS WAF with API Gateway.
+ Encourage API consumers to upgrade regularly to the latest version of your API so that older versions of your API can be deprecated and removed efficiently.
+ Before using this approach in a production setting, implement a firewall and authorization strategy for your API.
+ Implement access to the management of AWS resources of your AWS account by using the [least-privilege access model](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
+ To enforce best practices and security recommendations for applications built with the AWS CDK, we recommend that you use the [cdk-nag utility](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/check-aws-cdk-applications-or-cloudformation-templates-for-best-practices-by-using-cdk-nag-rule-packs.html).
## Epics
### Prepare your local environment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repository. | To clone the sample application repository, run the following command:git clone https://github.com/aws-samples/path-based-versioning-with-api-gateway
| App developer |
| Navigate to the cloned repository. | To navigate to the cloned repository folder location, run the following command: cd api-gateway-custom-domain-versioning
| App developer |
| Install the required dependencies. | To install the required dependencies, run the following command:npm install
| App developer |
### Deploy the CloudFormation routing stack
| Task | Description | Skills required |
| --- | --- | --- |
| Initiate deployment of the routing stack. | To initiate the deployment of the CloudFormation routing stack `CustomDomainRouterStack`, run the following command, replacing `example.com` with the name of the domain that you own:npx cdk deploy CustomDomainRouterStack --parameters PrerequisiteDomainName=example.com
The stack deployment will not succeed until the following domain DNS validation task is performed successfully. | App developer |
### Verify domain ownership
| Task | Description | Skills required |
| --- | --- | --- |
| Verify ownership of your domain. | The certificate will remain in a **Pending validation** status until you prove ownership of the associated domain. To prove ownership, add CNAME records to the hosted zone that is associated with the domain. For more information, see [DNS validation](https://docs.aws.amazon.com/acm/latest/userguide/dns-validation.html) in the AWS Certificate Manager documentation. Adding the appropriate records enables the `CustomDomainRouterStack` deployment to succeed. | App developer, AWS systems administrator, Network administrator |
| Create an alias record to point to your API Gateway custom domain. | After the certificate is issued and validated successfully, [create a DNS record](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-regional-api-custom-domain-create.html#apigateway-regional-api-custom-domain-dns-record) that points to your Amazon API Gateway custom domain URL. The custom domain URL is uniquely generated by the provisioning of the custom domain and is specified as a CloudFormation output parameter. Following is an [example of the record](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-values-basic.html): **Routing policy**: Simple routing**Record name**: `demo.api-gateway-custom-domain-versioning.example.com`**Alias**: Yes**Record type**: A DNS record of type "A" that points to an AWS resource**Value**: `d-xxxxxxxxxx.execute-api.xx-xxxx-x.amazonaws.com`**TTL (seconds)**: 300 | App developer, AWS systems administrator, Network administrator |
### Deploy CloudFormation stacks and invoke the API
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the `ApiStackV1` stack. | To deploy the `ApiStackV1` stack, use the following command:npm run deploy-v1
The following CDK code adds API mapping:var apiMapping = new CfnApiMapping(this, "ApiMapping", {
apiId: this.lambdaRestApi.restApiId,
domainName: props.customDomainName.domainName,
stage: "api",
apiMappingKey: "api/v1",
}); | App developer |
| Deploy the `ApiStackV2` stack. | To deploy the `ApiStackV2` stack, use the following command:npm run deploy-v2
| App developer |
| Invoke the API. | To invoke the API and test the API endpoints by using Bruno, see the instructions in [Additional information](#implement-path-based-api-versioning-by-using-custom-domains-additional). | App developer |
### Clean up resources
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up resources. | To destroy the resources associated with this sample application, use the following command:npx cdk destroy --all
Make sure that you clean up any Route 53 DNS records that were added manually for the domain ownership verification process. | App developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| The deployment of `CustomDomainRouterStack` times out because the certificate can’t be validated. | Make sure that you added the proper DNS validation CNAME records as described in the earlier task. Your new certificate might continue to display a status of **Pending validation** for up to 30 minutes after adding the DNS validation records. For more information, see [DNS validation](https://docs.aws.amazon.com/acm/latest/userguide/dns-validation.html) in the AWS Certificate Manager documentation. |
## Related resources
+ [Implementing header-based API Gateway versioning with Amazon CloudFront](https://aws.amazon.com/blogs/compute/implementing-header-based-api-gateway-versioning-with-amazon-cloudfront/) – This AWS Compute Blog post offers a header-based versioning strategy as an alternative to the path-based versioning strategy outlined in this pattern.
+ [AWS CDK Workshop](https://cdkworkshop.com/20-typescript.html) – This introductory workshop focuses on building and deploying applications on AWS by using the AWS Cloud Development Kit (AWS CDK). This workshop supports Go, Python, and TypeScript.
## Additional information
**Testing your API with Bruno**
We recommend that you use [Bruno](https://www.usebruno.com/), an open source API testing tool, to verify that the path-based routing is working properly for the sample application. This pattern provides a sample collection to facilitate testing your sample API.
To invoke and test your API, use the following steps:
1. [Install Bruno.](https://www.usebruno.com/downloads)
1. Open Bruno.
1. In this pattern’s [code repository](https://github.com/aws-samples/path-based-versioning-with-api-gateway), select **Bruno/Sample-API-Gateway-Custom-Domain-Versioning **and open the collection.
1. To see the **Environments** dropdown in the top right of the user interface (UI), select any request in the collection.
1. In the **Environments** dropdown, select **Configure**.
1. Replace the `REPLACE_ME_WITH_YOUR_DOMAIN` value with your custom domain.
1. Choose **Save**, and then close the **Configuration** section.
1. For **Sandbox Environment**,** **verify that the **Active** option is selected.
1. Invoke your API by using the **->** button for the selected request.
1. Take note on how validation (passing in non-number values) is handled in V1 compared to V2.
To see screenshots of example API invocation and comparison of V1 and V2 validation, see **Testing your sample API** in the `README.md` file in this pattern’s [code repository](https://github.com/aws-samples/path-based-versioning-with-api-gateway).
# Import the psycopg2 library to AWS Lambda to interact with your PostgreSQL database
*Louis Hourcade, Amazon Web Services*
## Summary
[Psycopg](https://www.psycopg.org/docs/) is a PostgresSQL database adapter for Python. Developers use the `psycopg2` library to write Python applications that interact with PostgreSQL databases.
On Amazon Web Services (AWS), developers also use [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) to run code for applications or backend services. Lambda is a serverless, event-driven compute service runs code without the need to provision or manage servers.
By default, when you create a new function that uses a [Python runtime that’s supported by Lambda](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html), the Lambda runtime environment is created from a [base image for Lambda](https://github.com/aws/aws-lambda-base-images) provided by AWS. Libraries, such as `pandas` or `psycopg2`, aren't included in the base image. To use a library, you need to bundle it in a custom package and attach it to Lambda.
There are multiple ways to bundle and attach a library, including the following:
+ Deploy your Lambda function from a [.zip file archive](https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-zip.html).
+ Deploy your Lambda function from a custom container image.
+ Create a [Lambda layer](https://docs.aws.amazon.com/lambda/latest/dg/chapter-layers.html#lambda-layer-versions), and attach it to your Lambda function.
This pattern demonstrates the first two options.
With a .zip deployment package, adding the `pandas` library to your Lambda function is relatively straightforward. Create a folder on your Linux machine, add the Lambda script together with the `pandas` library and the library's dependencies to the folder, zip the folder, and provide it as a source for your Lambda function.
Although using a .zip deployment package is a common practice, that approach doesn't work for the `psycopg2` library. This pattern first shows the error that you get if you use a .zip deployment package to add the `psycopg2` library to your Lambda function. The pattern then shows how to deploy Lambda from a Dockerfile and edit the Lambda image to make the `psycopg2` library work.
For information about the three resources that the pattern deploys, see the [Additional information](#import-psycopg2-library-lambda-additional) section.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account with sufficient permissions to deploy the AWS resources used by this pattern
+ AWS Cloud Development Kit (AWS CDK) installed globally by running `npm install -g aws-cdk`
+ A Git client
+ Python
+ Docker
**Limitations **
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service.
**Product versions**
+ Python runtime version that’s [supported by Lambda](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html)
+ Psycopg2 version 2.9.3
+ Pandas version 1.5.2
## Architecture
**Solution overview **
To illustrate the challenges that you might face when using the `psycopg2` library in Lambda, the pattern deploys two Lambda functions:
+ One Lambda function with the Python runtime created from a .zip file. The `psycopg2` and `pandas` libraries are installed in this .zip deployment package by using [pip](https://pypi.org/project/pip/).
+ One Lambda function with the Python runtime created from a Dockerfile. The Dockerfile installs the `psycopg2` and `pandas` libraries into the Lambda container image.
The first Lambda function installs the `pandas` library and its dependencies in a .zip file, and Lambda can use that library.
The second Lambda function demonstrates that by building a container image for your Lambda function, you can run the `pandas` and `psycopg2` libraries in Lambda.
## Tools
**AWS services**
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/v2/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
**Other tools**
+ [Docker](https://www.docker.com/) is a set of platform as a service (PaaS) products that use virtualization at the operating-system level to deliver software in containers.
+ [pandas](https://pandas.pydata.org/) is a Python-based open source tool for data analysis and manipulation.
+ [Psycopg](https://www.psycopg.org/docs/) is a PostgreSQL database adapter for the Python language that is designed for multithreaded applications. This pattern uses Psycopg 2.
+ [Python](https://www.python.org/) is a general-purpose computer programming language.
**Code repository**
The code for this pattern is available in the [import-psycopg2-in-lambda-to-interact-with-postgres-database](https://github.com/aws-samples/import-psycopg2-in-lambda-to-interact-with-postgres-database) repository on GitHub.
## Best practices
This pattern provides you with a working example of using AWS CDK to create a Lambda function from a Dockerfile. If you reuse this code in your application, make sure that the deployed resources meet all security requirements. Use tools such as [Checkov](https://www.checkov.io/), which scans cloud infrastructure configurations to find misconfiguration before the infrastructure is deployed.
## Epics
### Clone the repository and configure the deployment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repository. | To clone the GitHub repository on your local machine, run the following commands:git clone https://github.com/aws-samples/import-psycopg2-in-lambda-to-interact-with-postgres-database.git
cd AWS-lambda-psycopg2
| General AWS |
| Configure your deployment. | Edit the `app.py` file with information about your AWS account:aws_acccount = "AWS_ACCOUNT_ID"
region = "AWS_REGION"
# Select the CPU architecture you are using to build the image (ARM or X86)
architecture = "ARM"
| General AWS |
### Bootstrap your AWS account and deploy the application
| Task | Description | Skills required |
| --- | --- | --- |
| Bootstrap your AWS account. | If you haven't already [bootstrapped your AWS environment](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html), run the following commands with the AWS credentials of your AWS account:cdk bootstrap aws:///
| General AWS |
| Deploy the code. | To deploy the AWS CDK application, run the following command:cdk deploy AWSLambdaPyscopg2
| General AWS |
### Test the Lambda functions from the AWS Management Console
| Task | Description | Skills required |
| --- | --- | --- |
| Test the Lambda function created from the .zip file. | To test the Lambda function that was created from the .zip file, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/import-psycopg2-library-lambda.html)Because Lambda doesn't find the required PostgreSQL libraries in the default image, it can't use the `psycopg2` library. | General AWS |
| Test the Lambda function created from the Dockerfile. | To use the `psycopg2` library within your Lambda function, you must edit the Lambda Amazon Machine Image (AMI).To test the Lambda function that was created from the Dockerfile, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/import-psycopg2-library-lambda.html)The following code shows the Dockerfile that the AWS CDK template creates:# Start from lambda Python3.13 image
FROM public.ecr.aws/lambda/python:3.13
# Copy the lambda code, together with its requirements
COPY lambda/requirements.txt ${LAMBDA_TASK_ROOT}
COPY lambda/lambda_code.py ${LAMBDA_TASK_ROOT}
# Install postgresql-devel in your image
RUN yum install -y gcc postgresql-devel
# install the requirements for the Lambda code
RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
# Command can be overwritten by providing a different command in the template directly.
CMD ["lambda_code.handler"]
The Dockerfile takes the AWS provided Lambda image for the Python runtime and installs [postgresql-devel](https://yum-info.contradodigital.com/view-package/updates/postgresql-devel/), which contains the libraries needed to compile applications that directly interact with the PostgreSQL management server. The Dockerfile also installs the `pandas` and `psycopg2` libraries, which are indicated in the `requirements.txt` file. | General AWS |
## Related resources
+ [AWS CDK documentation](https://docs.aws.amazon.com/cdk/v2/guide/home.html)
+ [AWS Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html)
## Additional information
In this pattern, the AWS CDK template provides an AWS stack with three resources:
+ An [AWS Identity and Access Management (IAM) role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) for the Lambda functions.
+ A Lambda function with a Python runtime. The function is deployed from the `Constructs/lambda/lambda_deploy.zip` deployment package.
+ A Lambda function with a Python runtime. The function is deployed from the Dockerfile under the `Constructs` folder
The script for both Lambda functions checks whether the `pandas` and `psycopg2` libraries are successfully imported:
```
import pandas
print("pandas successfully imported")
import psycopg2
print("psycopg2 successfully imported")
def handler(event, context):
"""Function that checks whether psycopg2 and pandas are successfully imported or not"""
return {"Status": "psycopg2 and pandas successfully imported"}
```
The `lambda_deploy.zip` deployment package is built with the `Constructs/lambda/build.sh` bash script. This script creates a folder, copies the Lambda script, installs the `pandas` and `psycopg2` libraries, and generates the .zip file. To generate the .zip file yourself, run this bash script and redeploy the AWS CDK stack.
The Dockerfile starts with the AWS provided base image for Lambda with a Python runtime. The Dockerfile installs the `pandas` and `psycopg2` libraries on top of the default image.
# Integrate Amazon API Gateway with Amazon SQS to handle asynchronous REST APIs
*Natalia Colantonio Favero and Gustavo Martim, Amazon Web Services*
## Summary
When you deploy REST APIs, sometimes you need to expose a message queue that client applications can publish. For example, you might have problems with the latency of third-party APIs and delays in responses, or you might want to avoid the response time of database queries or avoid scaling the server when there are a large number of concurrent APIs. In these scenarios, the client applications that publish to the queue only need to know that the API received the data—not what happens after the data was received.
This pattern creates a REST API endpoint by using [Amazon API Gateway](https://aws.amazon.com/api-gateway/) to send a message to [Amazon Simple Queue Service (Amazon SQS)](https://aws.amazon.com/sqs/). It creates an easy-to-implement integration between the two services that avoids direct access to the SQS queue.
## Prerequisites and limitations
+ An [active AWS account](https://portal.aws.amazon.com/billing/signup/iam)
## Architecture
![\[Architecture for integrating API Gateway with Amazon SQS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/70984dee-e49f-4446-9d52-49ce826c3909/images/737ba0b2-da8f-4478-8c54-0a4835fd69f9.png)
The diagram illustrates these steps:
1. Request a POST REST API endpoint by using a tool such as Postman, another API, or other technologies.
1. API Gateway posts a message, which is received on the request's body, on the queue.
1. Amazon SQS receives the message and sends an answer to API Gateway with a success or failure code.
## Tools
+ [Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) helps you create, publish, maintain, monitor, and secure REST, HTTP, and WebSocket APIs at any scale.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Simple Queue Service (Amazon SQS)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.
## Epics
### Create an SQS queue
| Task | Description | Skills required |
| --- | --- | --- |
| Create a queue. | To create an SQS queue that receives the messages from the REST API:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer |
### Provide access to Amazon SQS
| Task | Description | Skills required |
| --- | --- | --- |
| Create an IAM role. | This IAM role gives API Gateway resources full access to Amazon SQS.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer, AWS administrator |
### Create a REST API
| Task | Description | Skills required |
| --- | --- | --- |
| Create a REST API. | This is the REST API that HTTP requests are sent to.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer |
| Connect API Gateway to Amazon SQS. | This step enables the message to flow from inside the HTTP request’s body to Amazon SQS.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer |
### Test the REST API
| Task | Description | Skills required |
| --- | --- | --- |
| Test the REST API. | Run a test to check for missing configuration:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer |
| Change the API integration to forward the request properly to Amazon SQS. | Complete the configuration to fix the integration error:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer |
| Test and validate the message in Amazon SQS. | Run a test to confirm that the test completed successfully:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer |
| Test API Gateway with a special character. | Run a test that includes special characters (such as &) that aren't acceptable in a message:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html)This is because special characters aren't supported by default in the message body. In the next step, you'll configure API Gateway to support special characters. For more information about content type conversions, see the [API Gateway documentation](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-workflow.html). | App developer |
| Change the API configuration to support special characters. | Adjust the configuration to accept special characters in the message:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html)The new message should include the special character. | App developer |
### Deploy the REST API
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the API. | To deploy the REST API:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer |
| Test with an external tool. | Run a test with an external tool to confirm that the message is received successfully:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-amazon-api-gateway-with-amazon-sqs-to-handle-asynchronous-rest-apis.html) | App developer |
### Clean Up
| Task | Description | Skills required |
| --- | --- | --- |
| Delete the API. | On the [API Gateway console](https://console.aws.amazon.com/apigateway/), choose the API you created, and then choose **Delete**. | App developer |
| Delete the IAM role. | On the [IAM console](https://console.aws.amazon.com/iam/), in the **Roles** pane, select **AWSGatewayRoleForSQS**, and then choose **Delete**. | App developer |
| Delete the SQS queue. | On the [Amazon SQS console](https://console.aws.amazon.com/sqs/), in the **Queues** pane, choose the SQS queue you created, and then choose **Delete**. | App developer |
## Related resources
+ [SQS-SendMessage](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-aws-services-reference.html#SQS-SendMessage) (API Gateway documentation)
+ [Content type conversions in API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-workflow.html) (API Gateway documentation)
+ [\$1util variables](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html#util-template-reference) (API Gateway documentation)
+ [How do I integrate an API Gateway REST API with Amazon SQS and resolve common errors?](https://repost.aws/knowledge-center/api-gateway-rest-api-sqs-errors) (AWS re:Post article)
# Process events asynchronously with Amazon API Gateway and AWS Lambda
*Andrea Meroni, Mariem Kthiri, Nadim Majed, and Michael Wallner, Amazon Web Services*
## Summary
[Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) is a fully managed service that developers can use to create, publish, maintain, monitor, and secure APIs at any scale. It handles the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls.
An important service quota of API Gateway is the integration timeout. The timeout is the maximum time in which a backend service must return a response before the REST API returns an error. The hard limit of 29 seconds is generally acceptable for synchronous workloads. However, that limit represents a challenge for those developers who want to use API Gateway with asynchronous workloads.
This pattern shows an example architecture to process events asynchronously using API Gateway and AWS Lambda. The architecture supports running processing jobs of duration up to 15 minutes, and it uses a basic REST API as the interface.
[Projen](https://pypi.org/project/projen/) is used to set up the local development environment and to deploy the example architecture to a target AWS account, in combination with the [AWS Cloud Development Kit (AWS CDK) Toolkit](https://docs.aws.amazon.com/cdk/v2/guide/cli.html), [Docker](https://docs.docker.com/get-docker/), and [Node.js](https://nodejs.org/en/download/). Projen automatically sets up a [Python](https://www.python.org/downloads/) virtual environment with [pre-commit](https://pre-commit.com/) and the tools that are used for code quality assurance, security scanning, and unit testing. For more information, see the [Tools](#process-events-asynchronously-with-amazon-api-gateway-and-aws-lambda-tools) section.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ The following tools installed on your workstation:
+ [AWS Cloud Development Kit (AWS CDK) Toolkit](https://docs.aws.amazon.com/cdk/v2/guide/cli.html) version 2.85.0
+ [Docker](https://docs.docker.com/get-docker/) version 20.10.21
+ [Node.js](https://nodejs.org/en/download/) version 18.13.0
+ [Projen](https://pypi.org/project/projen/) version 0.71.111
+ [Python](https://www.python.org/downloads/) version 3.9.16
**Limitations **
+ The maximum runtime of a job is limited by the maximum runtime for Lambda functions (15 minutes).
+ The maximum number of concurrent job requests is limited by the reserved concurrency of the Lambda function.
## Architecture
The following diagram shows the interaction of the jobs API with the event-processing and error-handling Lambda functions, with events stored in an Amazon EventBridge event archive.
![\[AWS Cloud architecture showing user interaction with jobs API, Lambda functions, and EventBridge.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e027130c-44c1-41ab-bbe9-f196a49bd9ac/images/3c437b65-48e3-477d-aeea-6ff938cc3285.png)
A typical workflow includes the following steps:
1. You authenticate against AWS Identity and Access Management (IAM) and obtain security credentials.
1. You send an HTTP `POST` request to the `/jobs` jobs API endpoint, specifying the job parameters in the request body.
1. The jobs API, which is an API Gateway REST API, returns to you an HTTP response that contains the job identifier.
1. The jobs API invokes asynchronously the event-processing Lambda function.
1. The event-processing function processes the event, and then it puts the job results in the jobs Amazon DynamoDB table
1. You send an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint, with the job identifier from step 3 as `{jobId}`.
1. The jobs API queries the `jobs` DynamoDB table to retrieve the job results.
1. The jobs API returns an HTTP response that contains the job results.
1. If the event processing fails, the event-processing function sends the event to the error-handling function.
1. The error-handling function puts the job parameters in the `jobs` DynamoDB table.
1. You can retrieve the job parameters by sending an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint.
1. If the error handling fails, the error-handling function sends the event to an EventBridge event archive.
You can replay the archived events by using EventBridge.
## Tools
**AWS services**
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/latest/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command-line shell.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.
+ [Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) is a serverless event bus service that helps you connect your applications with real-time data from a variety of sources. For example, Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
**Other tools**
+ [autopep8](https://github.com/hhatto/autopep8) automatically formats Python code based on the Python Enhancement Proposal (PEP) 8 style guide.
+ [Bandit](https://bandit.readthedocs.io/en/latest/) scans Python code to find common security issues.
+ [Commitizen](https://commitizen-tools.github.io/commitizen/) is a Git commit checker and `CHANGELOG` generator.
+ [cfn-lint](https://github.com/aws-cloudformation/cfn-lint) is an AWS CloudFormation linter
+ [Checkov](https://github.com/bridgecrewio/checkov) is a static code-analysis tool that checks infrastructure as code (IaC) for security and compliance misconfigurations.
+ [jq](https://stedolan.github.io/jq/download/) is a command-line tool for parsing JSON.
+ [Postman](https://www.postman.com/) is an API platform.
+ [pre-commit](https://pre-commit.com/) is a Git hooks manager.
+ [Projen](https://github.com/projen/projen) is a project generator.
+ [pytest](https://docs.pytest.org/en/7.2.x/index.html) is a Python framework for writing small, readable tests.
**Code repository**
This example architecture code can be found in the GitHub [Asynchronous Event Processing with API Gateway and Lambda](https://github.com/aws-samples/asynchronous-event-processing-api-gateway-lambda-cdk) repository.
## Best practices
+ This example architecture doesn't include monitoring of the deployed infrastructure. If your use case requires monitoring, evaluate adding [CDK Monitoring Constructs](https://constructs.dev/packages/cdk-monitoring-constructs) or another monitoring solution.
+ This example architecture uses [IAM permissions](https://docs.aws.amazon.com/apigateway/latest/developerguide/permissions.html) to control the access to the jobs API. Anyone authorized to assume the `JobsAPIInvokeRole` will be able to invoke the jobs API. As such, the access control mechanism is binary. If your use case requires a more complex authorization model, evaluate using a different [access control mechanism](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html).
+ When a user sends an HTTP `POST` request to the `/jobs` jobs API endpoint, the input data is validated at two different levels:
+ Amazon API Gateway is in charge of the first [request validation](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-method-request-validation.html).
+ The event processing function performs the second request.
No validation is performed when the user does an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint. If your use case requires additional input validation and an increased level of security, evaluate [using AWS WAF to protect your API](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-aws-waf.html).
## Epics
### Set up the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repository. | To clone the repository locally, run the following command:git clone https://github.com/aws-samples/asynchronous-event-processing-api-gateway-lambda-cdk.git
| DevOps engineer |
| Set up the project. | Change the directory to the repository root and set up the Python virtual environment and all the tools by using [Projen](https://github.com/projen/projen):cd asynchronous-event-processing-api-gateway-api-gateway-lambda-cdk
npx projen
| DevOps engineer |
| Install pre-commit hooks. | To install pre-commit hooks, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/process-events-asynchronously-with-amazon-api-gateway-and-aws-lambda.html) | DevOps engineer |
### Deploy the example architecture
| Task | Description | Skills required |
| --- | --- | --- |
| Bootstrap AWS CDK. | To bootstrap AWS CDK in your AWS account, run the following command:AWS_PROFILE=$YOUR_AWS_PROFILE npx projen bootstrap
| AWS DevOps |
| Deploy the example architecture. | To deploy the example architecture in your AWS account, run the following command:AWS_PROFILE=$YOUR_AWS_PROFILE npx projen deploy
| AWS DevOps |
### Test the architecture
| Task | Description | Skills required |
| --- | --- | --- |
| Install test prerequisites. | Install on your workstation the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [Postman](https://www.postman.com/downloads/), and [jq](https://jqlang.github.io/jq/).Using [Postman](https://www.postman.com/downloads/) to test this example architecture is suggested but not mandatory. If you choose an alternative API testing tool, make sure that it supports [AWS Signature Version 4 authentication](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html), and refer to the exposed API endpoints that can be inspected by [exporting the REST API](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-export-api.html). | DevOps engineer |
| Assume the `JobsAPIInvokeRole`. | [Assume](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sts/assume-role.html) the `JobsAPIInvokeRole` that was printed as output from the deploy command:CREDENTIALS=$(AWS_PROFILE=$ aws sts assume-role \
--no-cli-pager \
--role-arn $ \
--role-session-name JobsAPIInvoke)
export AWS_ACCESS_KEY_ID=$(cat $CREDENTIALS | jq ‘.Credentials’’.AccessKeyId’)
export AWS_SECRET_ACCESS_KEY=$(cat $CREDENTIALS | jq ‘.Credentials’’.SecretAccessKey’)
export AWS_SESSION_TOKEN==$(cat $CREDENTIALS | jq ‘.Credentials’’.SessionToken’)
| AWS DevOps |
| Configure Postman. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/process-events-asynchronously-with-amazon-api-gateway-and-aws-lambda.html) | AWS DevOps |
| Test the example architecture. | To test the example architecture, [send requests](https://learning.postman.com/docs/sending-requests/requests/#next-steps) to the jobs API. For more information, see the [Postman documentation](https://learning.postman.com/docs/getting-started/first-steps/sending-the-first-request/#send-an-api-request). | DevOps engineer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Destruction and subsequent redeployment of the example architecture fails because the [Amazon CloudWatch Logs log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) `/aws/apigateway/JobsAPIAccessLogs` already exists. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/process-events-asynchronously-with-amazon-api-gateway-and-aws-lambda.html) |
## Related resources
+ [API Gateway mapping template and access logging variable reference](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html)
+ [Set up asynchronous invocation of the backend Lambda function](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-integration-async.html)
# Process events asynchronously with Amazon API Gateway and Amazon DynamoDB Streams
*Andrea Meroni, Mariem Kthiri, Nadim Majed, Alessandro Trisolini, and Michael Wallner, Amazon Web Services*
## Summary
[Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) is a fully managed service that developers can use to create, publish, maintain, monitor, and secure APIs at any scale. It handles the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls.
An important service quota of API Gateway is the integration timeout. The timeout is the maximum time in which a backend service must return a response before the REST API returns an error. The hard limit of 29 seconds is generally acceptable for synchronous workloads. However, that limit represents a challenge for those developers who want to use API Gateway with asynchronous workloads.
This pattern shows an example architecture for processing events asynchronously using API Gateway, Amazon DynamoDB Streams, and AWS Lambda. The architecture supports running parallel processing jobs with the same input parameters, and it uses a basic REST API as the interface. In this example, using Lambda as the backend limits the duration of jobs to 15 minutes. You can avoid this limit by using an alternative service to process incoming events (for example, AWS Fargate).
[Projen](https://pypi.org/project/projen/) is used to set up the local development environment and to deploy the example architecture to a target AWS account, in combination with the [AWS Cloud Development Kit (AWS CDK) Toolkit](https://docs.aws.amazon.com/cdk/v2/guide/cli.html), [Docker](https://docs.docker.com/get-docker/) and [Node.js](https://nodejs.org/en/download/). Projen automatically sets up a [Python](https://www.python.org/downloads/) virtual environment with [pre-commit](https://pre-commit.com/) and the tools that are used for code quality assurance, security scanning, and unit testing. For more information, see the [Tools](#processing-events-asynchronously-with-amazon-api-gateway-and-amazon-dynamodb-streams-tools) section.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ The following tools installed on your workstation:
+ [AWS Cloud Development Kit (AWS CDK) Toolkit](https://docs.aws.amazon.com/cdk/v2/guide/cli.html) version 2.85.0 or later
+ [Docker](https://docs.docker.com/get-docker/) version 20.10.21 or later
+ [Node.js](https://nodejs.org/en/download/) version 18 or later
+ [Projen](https://pypi.org/project/projen/) version 0.71.111 or later
+ [Python](https://www.python.org/downloads/) version 3.9.16 or later
**Limitations **
+ The advised maximum number of readers for DynamoDB Streams is two to avoid throttling.
+ The maximum runtime of a job is limited by the maximum runtime for Lambda functions (15 minutes).
+ The maximum number of concurrent job requests is limited by the reserved concurrency of the Lambda functions.
## Architecture
**Architecture**
The following diagram shows the interaction of the jobs API with DynamoDB Streams and the event-processing and error-handling Lambda functions, with events stored in an Amazon EventBridge event archive.
![\[Diagram of architecture and process, with steps listed after the diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/68a46501-16e5-48e4-99c6-fc67a8b4133a/images/29fe6982-ad81-4099-9c65-08b17c96e78f.png)
A typical workflow includes the following steps:
1. You authenticate against AWS Identity and Access Management (IAM) and obtain security credentials.
1. You send an HTTP `POST` request to the `/jobs` jobs API endpoint, specifying the job parameters in the request body.
1. The jobs API returns to you an HTTP response that contains the job identifier.
1. The jobs API puts the job parameters in the `jobs_table` Amazon DynamoDB table.
1. The `jobs_table` DynamoDB table DynamoDB stream invokes the event-processing Lambda functions.
1. The event-processing Lambda functions process the event and then put the job results in the `jobs_table` DynamoDB table. To help ensure consistent results, the event-processing functions implement an [optimistic locking](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.OptimisticLocking.html) mechanism.
1. You send an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint, with the job identifier from step 3 as `{jobId}`.
1. The jobs API queries the `jobs_table` DynamoDB table to retrieve the job results.
1. The jobs API returns an HTTP response that contains the job results.
1. If the event processing fails, the event-processing function's source mapping sends the event to the error-handling Amazon Simple Notification Service (Amazon SNS) topic.
1. The error-handling SNS topic asynchronously pushes the event to the error-handling function.
1. The error-handling function puts the job parameters in the `jobs_table` DynamoDB table.
You can retrieve the job parameters by sending an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint.
1. If the error handling fails, the error-handling function sends the event to an Amazon EventBridge archive.
You can replay the archived events by using EventBridge.
## Tools
**AWS services**
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/v2/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.
+ [Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) is a serverless event bus service that helps you connect your applications with real-time data from a variety of sources. For example, AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Simple Notification Service (Amazon SNS)](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.
**Other tools**
+ [autopep8](https://github.com/hhatto/autopep8) automatically formats Python code based on the Python Enhancement Proposal (PEP) 8 style guide.
+ [Bandit](https://bandit.readthedocs.io/en/latest/) scans Python code to find common security issues.
+ [Commitizen](https://commitizen-tools.github.io/commitizen/) is a Git commit checker and `CHANGELOG` generator.
+ [cfn-lint](https://github.com/aws-cloudformation/cfn-lint) is an AWS CloudFormation linter
+ [Checkov](https://github.com/bridgecrewio/checkov) is a static code-analysis tool that checks infrastructure as code (IaC) for security and compliance misconfigurations.
+ [jq](https://stedolan.github.io/jq/download/) is a command-line tool for parsing JSON.
+ [Postman](https://www.postman.com/) is an API platform.
+ [pre-commit](https://pre-commit.com/) is a Git hooks manager.
+ [Projen](https://github.com/projen/projen) is a project generator.
+ [pytest](https://docs.pytest.org/en/7.2.x/index.html) is a Python framework for writing small, readable tests.
**Code repository**
This example architecture code can be found in the GitHub [Asynchronous Processing with API Gateway and DynamoDB Streams](https://github.com/aws-samples/asynchronous-event-processing-api-gateway-dynamodb-streams-cdk) repository.
## Best practices
+ This example architecture doesn't include monitoring of the deployed infrastructure. If your use case requires monitoring, evaluate adding [CDK Monitoring Constructs](https://constructs.dev/packages/cdk-monitoring-constructs) or another monitoring solution.
+ This example architecture uses [IAM permissions](https://docs.aws.amazon.com/apigateway/latest/developerguide/permissions.html) to control the access to the jobs API. Anyone authorized to assume the `JobsAPIInvokeRole` will be able to invoke the jobs API. As such, the access control mechanism is binary. If your use case requires a more complex authorization model, evaluate using a different [access control mechanism](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html).
+ When a user sends an HTTP `POST` request to the `/jobs` jobs API endpoint, the input data is validated at two different levels:
+ API Gateway is in charge of the first [request validation](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-method-request-validation.html).
+ The event processing function performs the second request.
No validation is performed when the user does an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint. If your use case requires additional input validation and an increased level of security, evaluate [using AWS WAF to protect your API](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-aws-waf.html).
+ To avoid throttling, the [DynamoDB Streams documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html#Streams.Processing) discourages users from reading with more than two consumers from the same stream’s shard. To scale out the number of consumers, we recommend using [Amazon Kinesis Data Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html).
+ [Optimistic locking](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.OptimisticLocking.html) has been used in this example to ensure consistent updates of items in the `jobs_table` DynamoDB table. Depending on the use-case requirement, you might need to implement more reliable locking mechanisms, such as pessimistic locking.
## Epics
### Set up the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repository. | To clone the repository locally, run the following command:git clone https://github.com/aws-samples/asynchronous-event-processing-api-gateway-dynamodb-streams-cdk.git
| DevOps engineer |
| Set up the project. | Change the directory to the repository root, and set up the Python virtual environment and all the tools by using [Projen](https://github.com/projen/projen):cd asynchronous-event-processing-api-gateway-api-gateway-dynamodb-streams-cdk
npx projen
| DevOps engineer |
| Install pre-commit hooks. | To install pre-commit hooks, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/processing-events-asynchronously-with-amazon-api-gateway-and-amazon-dynamodb-streams.html) | DevOps engineer |
### Deploy the example architecture
| Task | Description | Skills required |
| --- | --- | --- |
| Bootstrap AWS CDK. | To bootstrap [AWS CDK](https://aws.amazon.com/cdk/) in your AWS account, run the following command:AWS_PROFILE=$YOUR_AWS_PROFILE npx projen bootstrap
| AWS DevOps |
| Deploy the example architecture. | To deploy the example architecture in your AWS account, run the following command:AWS_PROFILE=$YOUR_AWS_PROFILE npx projen deploy
| AWS DevOps |
### Test the architecture
| Task | Description | Skills required |
| --- | --- | --- |
| Install test prerequisites. | Install on your workstation the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [Postman](https://www.postman.com/downloads/), and [jq](https://jqlang.github.io/jq/).Using [Postman](https://www.postman.com/downloads/) to test this example architecture is suggested but not mandatory. If you choose an alternative API testing tool, make sure that it supports [AWS Signature Version 4 authentication](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html), and refer to the exposed API endpoints that can be inspected by [exporting the REST API](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-export-api.html). | DevOps engineer |
| Assume the `JobsAPIInvokeRole`. | [Assume](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sts/assume-role.html) the `JobsAPIInvokeRole` that was printed as output from the `deploy` command:CREDENTIALS=$(AWS_PROFILE=$ aws sts assume-role \
--no-cli-pager \
--role-arn $ \
--role-session-name JobsAPIInvoke)
export AWS_ACCESS_KEY_ID=$(cat $CREDENTIALS | jq ‘.Credentials’’.AccessKeyId’)
export AWS_SECRET_ACCESS_KEY=$(cat $CREDENTIALS | jq ‘.Credentials’’.SecretAccessKey’)
export AWS_SESSION_TOKEN==$(cat $CREDENTIALS | jq ‘.Credentials’’.SessionToken’)
| AWS DevOps |
| Configure Postman. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/processing-events-asynchronously-with-amazon-api-gateway-and-amazon-dynamodb-streams.html) | AWS DevOps |
| Test the example architecture. | To test the example architecture, send requests to the jobs API. For more information, see the [Postman documentation](https://learning.postman.com/docs/getting-started/first-steps/sending-the-first-request/#send-an-api-request). | DevOps engineer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Destruction and subsequent redeployment of the example architecture fails because the [Amazon CloudWatch Logs log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) `/aws/apigateway/JobsAPIAccessLogs` already exists. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/processing-events-asynchronously-with-amazon-api-gateway-and-amazon-dynamodb-streams.html) |
## Related resources
+ [API Gateway mapping template and access logging variable reference](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html)
+ [Change data capture for DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html)
+ [Optimistic locking with version number](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.OptimisticLocking.html)
+ [Using Kinesis Data Streams to capture changes to DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html)
# Process events asynchronously with Amazon API Gateway, Amazon SQS, and AWS Fargate
*Andrea Meroni, Mariem Kthiri, Nadim Majed, Alessandro Trisolini, and Michael Wallner, Amazon Web Services*
## Summary
[Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) is a fully managed service that developers can use to create, publish, maintain, monitor, and secure APIs at any scale. It handles the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls.
An important service quota of API Gateway is the integration timeout. The timeout is the maximum time in which a backend service must return a response before the REST API returns an error. The hard limit of 29 seconds is generally acceptable for synchronous workloads. However, that limit represents a challenge for those developers who want to use API Gateway with asynchronous workloads.
This pattern shows an example architecture to process events asynchronously using API Gateway, Amazon Simple Queue Service (Amazon SQS) and AWS Fargate. The architecture supports running processing jobs without duration restrictions, and it uses a basic REST API as the interface.
[Projen](https://pypi.org/project/projen/) is used to set up the local development environment and to deploy the example architecture to a target AWS account, in combination with the [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/v2/guide/cli.html), [Docker](https://docs.docker.com/get-docker/), and [Node.js](https://nodejs.org/en/download/). Projen automatically sets up a [Python](https://www.python.org/downloads/) virtual environment with [pre-commit](https://pre-commit.com/) and the tools that are used for code quality assurance, security scanning, and unit testing. For more information, see the [Tools](#process-events-asynchronously-with-amazon-api-gateway-amazon-sqs-and-aws-fargate-tools) section.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ The following tools installed on your workstation:
+ [AWS Cloud Development Kit (AWS CDK) Toolkit](https://docs.aws.amazon.com/cdk/v2/guide/cli.html) version 2.85.0 or later
+ [Docker](https://docs.docker.com/get-docker/) version 20.10.21 or later
+ [Node.js](https://nodejs.org/en/download/) version 18 or later
+ [Projen](https://pypi.org/project/projen/) version 0.71.111 or later
+ [Python](https://www.python.org/downloads/) version 3.9.16 or later
**Limitations**
+ Concurrent jobs are limited to 500 tasks per minute, which is the maximum number of tasks that Fargate can provision.
## Architecture
The following diagram shows the interaction of the jobs API with the `jobs` Amazon DynamoDB table, the event-processing Fargate service, and the error-handling AWS Lambda function. Events are stored in an Amazon EventBridge event archive.
![\[Architecture diagram with description following the diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8a03149c-8f34-4593-84d5-accc1800a0a2/images/5e1071aa-4fbc-495c-bc22-8e62a32a136b.png)
A typical workflow includes the following steps:
1. You authenticate against AWS Identity and Access Management (IAM) and obtain security credentials.
1. You send an HTTP `POST` request to the `/jobs` jobs API endpoint, specifying the job parameters in the request body.
1. The jobs API, which is an API Gateway REST API, returns to you an HTTP response that contains the job identifier.
1. The jobs API sends a message to the SQS queue.
1. Fargate pulls the message from the SQS queue, processes the event, and then puts the job results in the `jobs` DynamoDB table.
1. You send an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint, with the job identifier from step 3 as `{jobId}`.
1. The jobs API queries the `jobs` DynamoDB table to retrieve the job results.
1. The jobs API returns an HTTP response that contains the job results.
1. If the event processing fails, the SQS queue sends the event to the dead-letter queue (DLQ).
1. An EventBridge event initiates the error-handling function.
1. The error-handling function puts the job parameters in the `jobs` DynamoDB table.
1. You can retrieve the job parameters by sending an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint.
1. If the error handling fails, the error-handling function sends the event to an EventBridge archive.
You can replay the archived events by using EventBridge.
## Tools
**AWS services**
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/v2/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.
+ [AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html) helps you run containers without needing to manage servers or Amazon Elastic Compute Cloud (Amazon EC2) instances. It’s used in conjunction with Amazon Elastic Container Service (Amazon ECS).
+ [Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) is a serverless event bus service that helps you connect your applications with real-time data from a variety of sources. For example, Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Simple Queue Service (Amazon SQS)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.
**Other tools**
+ [autopep8](https://github.com/hhatto/autopep8) automatically formats Python code based on the Python Enhancement Proposal (PEP) 8 style guide.
+ [Bandit](https://bandit.readthedocs.io/en/latest/) scans Python code to find common security issues.
+ [Commitizen](https://commitizen-tools.github.io/commitizen/) is a Git commit checker and `CHANGELOG` generator.
+ [cfn-lint](https://github.com/aws-cloudformation/cfn-lint) is an AWS CloudFormation linter
+ [Checkov](https://github.com/bridgecrewio/checkov) is a static code-analysis tool that checks infrastructure as code (IaC) for security and compliance misconfigurations.
+ [jq](https://stedolan.github.io/jq/download/) is a command-line tool for parsing JSON.
+ [Postman](https://www.postman.com/) is an API platform.
+ [pre-commit](https://pre-commit.com/) is a Git hooks manager.
+ [Projen](https://github.com/projen/projen) is a project generator.
+ [pytest](https://docs.pytest.org/en/7.2.x/index.html) is a Python framework for writing small, readable tests.
**Code repository**
This example architecture code can be found in the GitHub [Asynchronous Processing with API Gateway and SQS](https://github.com/aws-samples/asynchronous-event-processing-api-gateway-sqs-cdk) repository.
## Best practices
+ This example architecture doesn't include monitoring of the deployed infrastructure. If your use case requires monitoring, evaluate adding [CDK Monitoring Constructs](https://constructs.dev/packages/cdk-monitoring-constructs) or another monitoring solution.
+ This example architecture uses [IAM permissions](https://docs.aws.amazon.com/apigateway/latest/developerguide/permissions.html) to control the access to the jobs API. Anyone authorized to assume the `JobsAPIInvokeRole` will be able to invoke the jobs API. As such, the access control mechanism is binary. If your use case requires a more complex authorization model, evaluate using a different [access control mechanism](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html).
+ When a user sends an HTTP `POST` request to the `/jobs` jobs API endpoint, the input data is validated at two different levels:
+ API Gateway is in charge of the first [request validation](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-method-request-validation.html).
+ The event processing function performs the second request.
No validation is performed when the user does an HTTP `GET` request to the `/jobs/{jobId}` jobs API endpoint. If your use case requires additional input validation and an increased level of security, evaluate [using AWS WAF to protect your API](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-aws-waf.html).
## Epics
### Set up the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repository. | To clone the repository locally, run the following command:git clone https://github.com/aws-samples/asynchronous-event-processing-api-gateway-sqs-cdk.git
| DevOps engineer |
| Set up the project. | Change the directory to the repository root, and set up the Python virtual environment and all the tools by using [Projen](https://github.com/projen/projen):cd asynchronous-event-processing-api-gateway-api-gateway-sqs-cdk
npx projen
| DevOps engineer |
| Install pre-commit hooks. | To install pre-commit hooks, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/process-events-asynchronously-with-amazon-api-gateway-amazon-sqs-and-aws-fargate.html) | DevOps engineer |
### Deploy the example architecture
| Task | Description | Skills required |
| --- | --- | --- |
| Bootstrap AWS CDK. | To bootstrap [AWS CDK](https://aws.amazon.com/cdk/) in your AWS account, run the following command:AWS_PROFILE=$YOUR_AWS_PROFILE npx projen bootstrap
| AWS DevOps |
| Deploy the example architecture. | To deploy the example architecture in your AWS account, run the following command:AWS_PROFILE=$YOUR_AWS_PROFILE npx projen deploy
| AWS DevOps |
### Test the architecture
| Task | Description | Skills required |
| --- | --- | --- |
| Install test prerequisites. | Install on your workstation the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [Postman](https://www.postman.com/downloads/), and [jq](https://jqlang.github.io/jq/).Using [Postman](https://www.postman.com/downloads/) to test this example architecture is suggested but not mandatory. If you choose an alternative API testing tool, make sure that it supports [AWS Signature Version 4 authentication](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html), and refer to the exposed API endpoints that can be inspected by [exporting the REST API](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-export-api.html). | DevOps engineer |
| Assume the `JobsAPIInvokeRole`. | [Assume](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sts/assume-role.html) the `JobsAPIInvokeRole` that was printed as output from the `deploy` command:CREDENTIALS=$(AWS_PROFILE=$ aws sts assume-role \
--no-cli-pager \
--role-arn $ \
--role-session-name JobsAPIInvoke)
export AWS_ACCESS_KEY_ID=$(cat $CREDENTIALS | jq ‘.Credentials’’.AccessKeyId’)
export AWS_SECRET_ACCESS_KEY=$(cat $CREDENTIALS | jq ‘.Credentials’’.SecretAccessKey’)
export AWS_SESSION_TOKEN==$(cat $CREDENTIALS | jq ‘.Credentials’’.SessionToken’)
| AWS DevOps |
| Configure Postman. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/process-events-asynchronously-with-amazon-api-gateway-amazon-sqs-and-aws-fargate.html) | AWS DevOps |
| Test the example architecture. | To test the example architecture, send requests to the jobs API. For more information, see the [Postman documentation](https://learning.postman.com/docs/getting-started/first-steps/sending-the-first-request/#send-an-api-request). | DevOps engineer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Destruction and subsequent redeployment of the example architecture fails because the [Amazon CloudWatch Logs log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) `/aws/apigateway/JobsAPIAccessLogs` already exists. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/process-events-asynchronously-with-amazon-api-gateway-amazon-sqs-and-aws-fargate.html) |
| Destruction and subsequent redeployment of the example architecture fails because the [CloudWatch Logs log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) `/aws/ecs/EventProcessingServiceLogs` already exists. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/process-events-asynchronously-with-amazon-api-gateway-amazon-sqs-and-aws-fargate.html) |
## Related resources
+ [API Gateway mapping template and access logging variable reference](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html)
+ [How do I integrate an API Gateway REST API with Amazon SQS and resolve common errors?](https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-rest-api-sqs-errors/)
# Run AWS Systems Manager Automation tasks synchronously from AWS Step Functions
*Elie El khoury, Amazon Web Services*
## Summary
This pattern explains how to integrate AWS Step Functions with AWS Systems Manager. It uses AWS SDK service integrations to call the Systems Manager **startAutomationExecution** API with a task token from a state machine workflow, and pauses until the token returns with a success or failure call. To demonstrate the integration, this pattern implements an Automation document (runbook) wrapper around the `AWS-RunShellScript` or `AWS-RunPowerShellScript` document, and uses `.waitForTaskToken` to synchronously call `AWS-RunShellScript` or `AWS-RunPowerShellScript`. For more information about AWS SDK service integrations in Step Functions, see the [AWS Step Functions Developer Guide](https://docs.aws.amazon.com/step-functions/latest/dg/supported-services-awssdk.html).
Step Functions** **is a low-code, visual workflow service that you can use to build distributed applications, automate IT and business processes, and build data and machine learning pipelines by using AWS services. Workflows manage failures, retries, parallelization, service integrations, and observability so you can focus on higher-value business logic.
Automation, a capability of AWS Systems Manager, simplifies common maintenance, deployment, and remediation tasks for AWS services such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Service (Amazon RDS), Amazon Redshift, and Amazon Simple Storage Service (Amazon S3). Automation gives you granular control over the concurrency of your automations. For example, you can specify how many resources to target concurrently, and how many errors can occur before an automation is stopped.
For implementation details, including runbook steps, parameters, and examples, see the [Additional information](#run-aws-systems-manager-automation-tasks-synchronously-from-aws-step-functions-additional) section.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ AWS Identity and Access Management (IAM) permissions to access Step Functions and Systems Manager
+ An EC2 instance with Systems Manager Agent (SSM Agent) [installed](https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-ssm-agent.html) on the instance
+ [An IAM instance profile for Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-instance-profile.html) attached to the instance where you plan to run the runbook
+ A Step Functions role that has the following IAM permissions (which follow the principle of least privilege):
```
{
"Effect": "Allow",
"Action": "ssm:StartAutomationExecution",
"Resource": "*"
}
```
**Product versions**
+ SSM document schema version 0.3 or later
+ SSM Agent version 2.3.672.0 or later
## Architecture
**Target technology stack **
+ AWS Step Functions
+ AWS Systems Manager Automation
**Target architecture**
![\[Architecture for running Systems Manager automation tasks synchronously from Step Functions\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/47c19e4f-d68d-4f91-bb68-202098757529/images/2d248aae-d858-4565-8af2-593cde0da780.png)
**Automation and scale**
+ This pattern provides an AWS CloudFormation template that you can use to deploy the runbooks on multiple instances. (See the GitHub [Step Functions and Systems Manager implementation](https://github.com/aws-samples/amazon-stepfunctions-ssm-waitfortasktoken) repository.)
## Tools
**AWS services**
+ [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine AWS Lambda functions and other AWS services to build business-critical applications.
+ [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) helps you manage your applications and infrastructure running in the AWS Cloud. It simplifies application and resource management, shortens the time to detect and resolve operational problems, and helps you manage your AWS resources securely at scale.
**Code **
The code for this pattern is available in the GitHub [Step Functions and Systems Manager implementation](https://github.com/aws-samples/amazon-stepfunctions-ssm-waitfortasktoken) repository.
## Epics
### Create runbooks
| Task | Description | Skills required |
| --- | --- | --- |
| Download the CloudFormation template. | Download the `ssm-automation-documents.cfn.json` template from the `cloudformation `folder of the GitHub repository. | AWS DevOps |
| Create runbooks. | Sign in to the AWS Management Console, open the [CloudFormation console](https://console.aws.amazon.com/cloudformation/), and deploy the template. For more information about deploying CloudFormation templates, see [Creating a stack on the CloudFormation console](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html) in the CloudFormation documentation. The CloudFormation template deploys three resources:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-aws-systems-manager-automation-tasks-synchronously-from-aws-step-functions.html) | AWS DevOps |
### Create a sample state machine
| Task | Description | Skills required |
| --- | --- | --- |
| Create a test state machine. | Follow the instructions in the [AWS Step Functions Developer Guide](https://docs.aws.amazon.com/step-functions/latest/dg/getting-started-with-sfn.html) to create and run a state machine. For the definition, use the following code. Make sure to update the `InstanceIds` value with the ID of a valid Systems Manager-enabled instance in your account.{
"Comment": "A description of my state machine",
"StartAt": "StartAutomationWaitForCallBack",
"States": {
"StartAutomationWaitForCallBack": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:ssm:startAutomationExecution.waitForTaskToken",
"Parameters": {
"DocumentName": "SfnRunCommandByInstanceIds",
"Parameters": {
"InstanceIds": [
"i-1234567890abcdef0"
],
"taskToken.$": "States.Array($$.Task.Token)",
"workingDirectory": [
"/home/ssm-user/"
],
"Commands": [
"echo \"This is a test running automation waitForTaskToken\" >> automation.log",
"sleep 100"
],
"executionTimeout": [
"10800"
],
"deliveryTimeout": [
"30"
],
"shell": [
"Shell"
]
}
},
"End": true
}
}
}This code calls the runbook to run two commands that demonstrate the `waitForTaskToken` call to Systems Manager Automation.The `shell` parameter value (`Shell` or `PowerShell`) determines whether the Automation document runs `AWS-RunShellScript` or `AWS-RunPowerShellScript`.The task writes "This is a test running automation waitForTaskToken" into the `/home/ssm-user/automation.log` file, and then sleeps for 100 seconds before it responds with the task token and releases the next task in the workflow.If you want to call the `SfnRunCommandByTargets` runbook instead, replace the `Parameters` section of the previous code with the following:"Parameters": {
"Targets": [
{
"Key": "InstanceIds",
"Values": [
"i-02573cafcfEXAMPLE",
"i-0471e04240EXAMPLE"
]
}
], | AWS DevOps |
| Update the IAM role for the state machine. | The previous step automatically creates a dedicated IAM role for the state machine. However, it doesn’t grant permissions to call the runbook. Update the role by adding the following permissions:{
"Effect": "Allow",
"Action": "ssm:StartAutomationExecution",
"Resource": "*"
} | AWS DevOps |
| Validate the synchronous calls. | Run the state machine to validate the synchronous call between Step Functions and Systems Manager Automation. For sample output, see the [Additional information](#run-aws-systems-manager-automation-tasks-synchronously-from-aws-step-functions-additional) section. | AWS DevOps |
## Related resources
+ [Getting started with AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/getting-started-with-sfn.html) (*AWS Step Functions Developer Guide*)
+ [Wait for a callback with the task token](https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token) (*AWS Step Functions Developer Guide*, service integration patterns)
+ [send\$1task\$1success](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/stepfunctions/client/send_task_success.html) and [send\$1task\$1failure](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/stepfunctions/client/send_task_failure.html) API calls (Boto3 documentation)
+ [AWS Systems Manager Automation](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation.html) (*AWS Systems Manager User Guide*)
## Additional information
**Implementation details**
This pattern provides a CloudFormation template that deploys two Systems Manager runbooks:
+ `SfnRunCommandByInstanceIds` runs the `AWS-RunShellScript` or `AWS-RunPowerShellScript` command by using instance IDs.
+ `SfnRunCommandByTargets` runs the `AWS-RunShellScript` or `AWS-RunPowerShellScript` command by using targets.
Each runbook implements four steps to achieve a synchronous call when using the `.waitForTaskToken` option in Step Functions.
|
|
| Step | Action | Description |
| --- |--- |--- |
| **1** | `Branch` | Checks the `shell` parameter value (`Shell` or `PowerShell`) to decide whether to run `AWS-RunShellScript` for Linux or `AWS-RunPowerShellScript` for Windows. |
| **2** | `RunCommand_Shell` or `RunCommand_PowerShell` | Takes several inputs and runs the `RunShellScript` or `RunPowerShellScript` command. For more information, check the **Details** tab for the `RunCommand_Shell` or `RunCommand_PowerShell` Automation document on the Systems Manager console. |
| **3** | `SendTaskFailure` | Runs when step 2 is aborted or canceled. It calls the Step Functions [send\$1task\$1failure](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/stepfunctions/client/send_task_failure.html) API, which accepts three parameters as input: the token passed by the state machine, the failure error, and a description of the cause of the failure. |
| **4** | `SendTaskSuccess` | Runs when step 2 is successful. It calls the Step Functions [send\$1task\$1success](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/stepfunctions/client/send_task_success.html) API, which accepts the token passed by the state machine as input. |
**Runbook parameters**
`SfnRunCommandByInstanceIds` runbook:
|
|
| Parameter name | Type | Optional or required | Description |
| --- |--- |--- |--- |
| `shell` | String | Required | The instances shell to decide whether to run `AWS-RunShellScript` for Linux or `AWS-RunPowerShellScript` for Windows. |
| `deliveryTimeout` | Integer | Optional | The time, in seconds, to wait for a command to deliver to the SSM Agent on an instance. This parameter has a minimum value of 30 (0.5 minute) and a maximum value of 2592000 (720 hours). |
| `executionTimeout` | String | Optional | The time, in seconds, for a command to complete before it is considered to have failed. The default value is 3600 (1 hour). The maximum value is 172800 (48 hours). |
| `workingDirectory` | String | Optional | The path to the working directory on your instance. |
| `Commands` | StringList | Required | The shell script or command to run. |
| `InstanceIds` | StringList | Required | The IDs of the instances where you want to run the command. |
| `taskToken` | String | Required | The task token to use for callback responses. |
`SfnRunCommandByTargets` runbook:
|
|
| Name | Type | Optional or required | Description |
| --- |--- |--- |--- |
| `shell` | String | Required | The instances shell to decide whether to run `AWS-RunShellScript` for Linux or `AWS-RunPowerShellScript` for Windows. |
| `deliveryTimeout` | Integer | Optional | The time, in seconds, to wait for a command to deliver to the SSM Agent on an instance. This parameter has a minimum value of 30 (0.5 minute) and a maximum value of 2592000 (720 hours). |
| `executionTimeout` | Integer | Optional | The time, in seconds, for a command to complete before it is considered to have failed. The default value is 3600 (1 hour). The maximum value is 172800 (48 hours). |
| `workingDirectory` | String | Optional | The path to the working directory on your instance. |
| `Commands` | StringList | Required | The shell script or command to run. |
| `Targets` | MapList | Required | An array of search criteria that identifies instances by using key-value pairs that you specify. For example: `[{"Key":"InstanceIds","Values":["i-02573cafcfEXAMPLE","i-0471e04240EXAMPLE"]}]` |
| `taskToken` | String | Required | The task token to use for callback responses. |
**Sample output**
The following table provides sample output from the step function. It shows that the total run time is over 100 seconds between step 5 (`TaskSubmitted`) and step 6 (`TaskSucceeded`). This demonstrates that the step function waited for the `sleep 100` command to finish before moving to the next task in the workflow.
|
|
| ID | Type | Step | Resource | Elapsed Time (ms) | Timestamp |
| --- |--- |--- |--- |--- |--- |
| ** 1** | `ExecutionStarted` | | - | 0 | Mar 11, 2022 02:50:34.303 PM |
| ** 2** | `TaskStateEntered` | `StartAutomationWaitForCallBack` | - | 40 | Mar 11, 2022 02:50:34.343 PM |
| ** 3** | `TaskScheduled` | `StartAutomationWaitForCallBack` | - | 40 | Mar 11, 2022 02:50:34.343 PM |
| ** 4** | `TaskStarted` | `StartAutomationWaitForCallBack` | - | 154 | Mar 11, 2022 02:50:34.457 PM |
| ** 5** | `TaskSubmitted` | `StartAutomationWaitForCallBack` | - | 657 | Mar 11, 2022 02:50:34.960 PM |
| ** 6** | `TaskSucceeded` | `StartAutomationWaitForCallBack` | - | 103835 | Mar 11, 2022 02:52:18.138 PM |
| ** 7** | `TaskStateExited` | `StartAutomationWaitForCallBack` | - | 103860 | Mar 11, 2022 02:52:18.163 PM |
| ** 8** | `ExecutionSucceeded` | | - | 103897 | Mar 11, 2022 02:52:18.200 PM |
# Run parallel reads of S3 objects by using Python in an AWS Lambda function
*Eduardo Bortoluzzi, Amazon Web Services*
## Summary
You can use this pattern to retrieve and summarize a list of documents from Amazon Simple Storage Service (Amazon S3) buckets in real time. The pattern provides example code to parallel read objects from S3 buckets on Amazon Web Services (AWS). The pattern showcases how to efficiently run I/O bound tasks with AWS Lambda functions using Python.
A financial company used this pattern in an interactive solution to manually approve or reject correlated financial transactions in real time. The financial transaction documents were stored in an S3 bucket related to the market. An operator selected a list of documents from the S3 bucket, analyzed the total value of the transactions that the solution calculated, and decided to approve or reject the selected batch.
I/O bound tasks support multiple threads. In this example code, the [concurrent.futures.ThreadPoolExecutor](https://docs.python.org/3.13/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor) is used with a maximum of 30 simultaneous threads, even though Lambda functions support up to 1,024 threads (with one of those threads being your main process). This limit is because too many threads create latency issues due to context switching and utilization of computing resources. You also need to increase the maximum pool connections in `botocore` so that all threads can perform the S3 object download simultaneously.
The example code uses one 8.3 KB object, with JSON data, in an S3 bucket. The object is read multiple times. After the Lambda function reads the object, the JSON data is decoded to a Python object. In December 2024, the result after running this example was 1,000 reads processed in 2.3 seconds and 10,000 reads processed in 27 seconds using a Lambda function configured with 2,304 MB of memory. AWS Lambda supports memory configurations from 128 MB to 10,240 MB (10 GB), though increasing the Lambdamemory beyond 2,304 MB didn't help to decrease the time to run this particular I/O-bound task.
The [AWS Lambda Power Tuning](https://github.com/alexcasalboni/aws-lambda-power-tuning) tool was used to test different Lambda memory configurations and verify the best performance-to-cost ratio for the task. For test results, see the [Additional information](#run-parallel-reads-of-s3-objects-by-using-python-in-an-aws-lambda-function-additional) section.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ Proficiency with Python development
**Limitations **
+ A Lambda function can have at most [1,024 execution processes or threads](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#function-configuration-deployment-and-execution).
+ New AWS accounts have a Lambda memory limit of 3,008 MB. Adjust the AWS Lambda Power Tuning tool accordingly. For more information, see the [Troubleshooting](#run-parallel-reads-of-s3-objects-by-using-python-in-an-aws-lambda-function-troubleshooting) section.
+ Amazon S3 has a limit of [5,500 GET/HEAD requests per second per partitioned prefix](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html).
**Product versions**
+ Python 3.9 or later
+ AWS Cloud Development Kit (AWS CDK) v2
+ AWS Command Line Interface (AWS CLI) version 2
+ AWS Lambda Power Tuning 4.3.6 (optional)
## Architecture
**Target technology stack **
+ AWS Lambda
+ Amazon S3
+ AWS Step Functions (if AWS Lambda Power Tuning is deployed)
**Target architecture **
The following diagram shows a Lambda function that reads objects from an S3 bucket in parallel. The diagram also has a Step Functions workflow for the AWS Lambda Power Tuning tool to fine-tune the Lambda function memory. This fine-tuning helps to achieve a good balance between cost and performance.
![\[Diagram showing Lambda function, S3 bucket, and AWS Step Functions.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b46e9b16-9842-4291-adfa-3ef012b89aec/images/828696e2-6df7-4536-9205-951c99449f4e.png)
**Automation and scale**
The Lambda functions scale fast when required. To avoid 503 Slow Down errors from Amazon S3 during high demand, we recommend putting some limits on the scaling.
## Tools
**AWS services**
+ [AWS Cloud Development Kit (AWS CDK) v2](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code. The example infrastructure was created to be deployed with AWS CDK.
+ [AWS Command Line InterfaceAWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command-line shell. In this pattern, AWS CLI version 2 is used to upload an example JSON file.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Simple Storage Service Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine AWS Lambda functions and other AWS services to build business-critical applications.
**Other tools**
+ [Python](https://www.python.org/) is a general-purpose computer programming language. The [reuse of idle worker threads](https://docs.python.org/3.8/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor) was introduced in Python version 3.8, and the Lambda function code in this pattern was created for Python version 3.9 and later.
**Code repository**
The code for this pattern is available in the [aws-lambda-parallel-download](https://github.com/aws-samples/aws-lambda-parallel-download) GitHub repository.
## Best practices
+ This AWS CDK construct relies on your AWS account's user permissions to deploy the infrastructure. If you plan to use AWS CDK Pipelines or cross-account deployments, see [Stack synthesizers](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html#bootstrapping-synthesizers).
+ This example application doesn't have the access logs enabled at the S3 bucket. It's a best practice to enable access logs in production code.
## Epics
### Prepare the development environment
| Task | Description | Skills required |
| --- | --- | --- |
| Check the Python installed version. | This code has been tested specifically on Python 3.9 and Python 3.13, and it should work on all versions between these releases. To check your Python version, run `python3 -V` in your terminal, and install a newer version if needed.To verify that the required modules are installed, run `python3 -c "import pip, venv"`. No error message means the modules are properly installed and you're ready to run this example. | Cloud architect |
| Install AWS CDK. | To install the AWS CDK if it isn't already installed, follow the instructions at [Getting started with the AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). To confirm that the installed AWS CDK version is 2.0 or later, run `cdk –version`. | Cloud architect |
| Bootstrap your environment. | To bootstrap your environment, if it hasn’t already been done, follow the instructions at [Bootstrap your environment for use with the AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping-env.html). | Cloud architect |
### Clone the example repository
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repository. | To clone the latest version of the repository, run the following command:git clone --depth 1 --branch v1.2.0 \
git@github.com:aws-samples/aws-lambda-parallel-download.git
| Cloud architect |
| Change the working directory to the cloned repository. | Run the following command:cd aws-lambda-parallel-download
| Cloud architect |
| Create the Python virtual environment. | To create a Python virtual environment, run the following command:python3 -m venv .venv
| Cloud architect |
| Activate the virtual environment. | To activate the virtual environment, run the following command:source .venv/bin/activate
| Cloud architect |
| Install the dependencies. | To install the Python dependencies, run the `pip` command:pip install -r requirements.txt
| Cloud architect |
| Browse the code. | (Optional) The example code that downloads an object from the S3 bucket is at `resources/parallel.py`.The infrastructure code is in the `parallel_download` folder. | Cloud architect |
### Deploy and test the app
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the app. | Run `cdk deploy`.Write down the AWS CDK outputs:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-parallel-reads-of-s3-objects-by-using-python-in-an-aws-lambda-function.html) | Cloud architect |
| Upload an example JSON file. | The repository contains an example JSON file of about 9 KB. To upload the file to the S3 bucket of the created stack, run the following command:aws s3 cp sample.json s3://
Replace `` with the corresponding value from the AWS CDK output. | Cloud architect |
| Run the app. | To run the app, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-parallel-reads-of-s3-objects-by-using-python-in-an-aws-lambda-function.html) | Cloud architect |
| Add the number of downloads. | (Optional) To run 1,500 get object calls, use the following JSON in **Event JSON** of the `Test` parameter:{"repeat": 1500, "objectKey": "sample.json"} | Cloud architect |
### Optional: Run AWS Lambda Power Tuning
| Task | Description | Skills required |
| --- | --- | --- |
| Run the AWS Lambda Power Tuning tool. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-parallel-reads-of-s3-objects-by-using-python-in-an-aws-lambda-function.html)At the end of the run, the result will be on the **Execution input and output** tab. | Cloud architect |
| View the AWS Lambda Power Tuning results in a graph. | On the **Execution input and output** tab, copy the `visualization` property link, and paste it in a new browser tab. | Cloud architect |
### Clean up
| Task | Description | Skills required |
| --- | --- | --- |
| Remove the objects from the S3 bucket. | Before you destroy the deployed resources, you remove all the objects from the S3 bucket:aws s3 rm s3:// \
--recursive
Remember to replace `` with the value from the AWS CDK outputs. | Cloud architect |
| Destroy the resources. | To destroy all the resources that were created for this pilot, run the following command:cdk destroy
| Cloud architect |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| `'MemorySize' value failed to satisfy constraint: Member must have value less than or equal to 3008` | For new accounts, you might not be able to configure more than 3,008 MB in your Lambda functions. To test using AWS Lambda Power Tuning, add the following property at the input JSON when you are starting the Step Functions execution:"powerValues": [
512,
1024,
1536,
2048,
2560,
3008
]
|
## Related resources
+ [Python – concurrent.futures.ThreadPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor)
+ [Lambda quotas – Function configuration, deployment, and execution](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#function-configuration-deployment-and-execution)
+ [Working with the AWS CDK in Python](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-python.html)
+ [Profiling functions with AWS Lambda Power Tuning](https://docs.aws.amazon.com/lambda/latest/operatorguide/profile-functions.html)
## Additional information
**Code**
The following code snippet performs the parallel I/O processing:
```
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
for result in executor.map(a_function, (the_arguments)):
...
```
The `ThreadPoolExecutor` reuses the threads when they become available.
**Testing and results**
These tests were conducted in December 2024.
The first test processed 2,500 object reads, with the following result.
![\[Invocation time falling and invocation cost rising as memory increases.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b46e9b16-9842-4291-adfa-3ef012b89aec/images/f6743412-1e52-4c4c-a51c-ac0f75b3b998.png)
Starting at 3,009 MB, the processing-time level stayed almost the same for any memory increase, but the cost increased as the memory size increased.
Another test investigated the range between 1,536 MB and 3,072 MB of memory, using values that were multiples of 256 MB and processing 10,000 object reads, with the following results.
![\[Decreased difference between invocation time falling and invocation cost rising.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b46e9b16-9842-4291-adfa-3ef012b89aec/images/c75d4443-74d8-4b93-9b4d-b2640869381e.png)
The best performance-to-cost ratio was with the 2,304 MB memory Lambda configuration.
For comparison, a sequential process of 2,500 object reads took 47 seconds. The parallel process using the 2,304 MB Lambda configuration took 7 seconds, which is 85 percent less.
![\[Chart showing the decrease in time when switching from sequential to parallel processing.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b46e9b16-9842-4291-adfa-3ef012b89aec/images/f3dcc44d-ac20-4b75-897d-1d71f0d59781.png)
# Send telemetry data from AWS Lambda to OpenSearch for real-time analytics and visualization
*Tabby Ward, Guy Bachar, and David Kilzer, Amazon Web Services*
## Summary
Modern applications are becoming increasingly distributed and event-driven, which reinforces the need for real-time monitoring and observability. AWS Lambda is a serverless computing service that plays a crucial role in building scalable and event-driven architectures. However, monitoring and troubleshooting Lambda functions can be challenging if you rely solely on Amazon CloudWatch Logs, which can introduce latency and limited retention periods.
To address this challenge, AWS introduced the Lambda Telemetry API, which enables Lambda functions to send telemetry data directly to third-party monitoring and observability tools. This API supports real-time streaming of logs, metrics, and traces, and provides a comprehensive and timely view of the performance and health of your Lambda functions.
This pattern explains how to integrate the Lambda Telemetry API with [OpenSearch](https://opensearch.org/docs/latest/), which is an open-source, distributed search and analytics engine. OpenSearch offers a powerful and scalable platform for ingesting, storing, and analyzing large volumes of data, which makes it an ideal choice for Lambda telemetry data. Specifically, this pattern demonstrates how to send logs from a Lambda function that's written in Python directly to an OpenSearch cluster by using a Lambda extension that's provided by AWS. This solution is flexible and customizable, so you can create your own Lambda extension or alter the sample source code to change the output format as desired.
The pattern explains how to set up and configure the Lambda Telemetry API integration with OpenSearch, and includes best practices for security, cost optimization, and scalability. The objective is to help you gain deeper insights into your Lambda functions and enhance the overall observability of your serverless applications.
|
|
| Note: This pattern focuses on integrating the Lambda Telemetry API with managed OpenSearch. However, the principles and techniques discussed are also applicable to self-managed OpenSearch and Elasticsearch. |
| --- |
## Prerequisites and limitations
Before you begin the integration process, make sure that you have the following prerequisites in place:
**AWS account**: An active AWS account with appropriate permissions to create and manage the following AWS resources:
+ AWS Lambda
+ AWS Identity and Access Management (IAM)
+ Amazon OpenSearch Service (if you're using a managed OpenSearch cluster)
**OpenSearch cluster**:
+ You can use an existing self-managed OpenSearch cluster or a managed service such as OpenSearch Service.
+ If you're using OpenSearch Service, set up your OpenSearch cluster by following the instructions in [Getting started with Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gsg.html) in the OpenSearch Service documentation.
+ Make sure that the OpenSearch cluster is accessible from your Lambda function and is configured with the necessary security settings, such as access policies, encryption, and authentication.
+ Configure the OpenSearch cluster with the necessary index mappings and settings to ingest the Lambda telemetry data. For more information, see [Loading streaming data into Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/integrations.html) in the OpenSearch Service documentation.
**Network connectivity**:
+ Ensure that your Lambda function has the necessary network connectivity to access the OpenSearch cluster. For guidance on how to configure virtual private cloud (VPC) settings, see [Launching your Amazon OpenSearch Service domains within a VPC](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html) in the OpenSearch Service documentation.
**IAM roles and policies**:
+ Create an IAM role with the necessary permissions for your Lambda function to access the OpenSearch cluster and access your credentials stored in AWS Secrets Manager.
+ Attach the appropriate IAM policies to the role, such as the `AWSLambdaBasicExecutionRole` policy and any additional permissions required to interact with OpenSearch.
+ Verify that the IAM permissions granted to your Lambda function allow it to write data to the OpenSearch cluster. For information about managing IAM permissions, see [Defining Lambda function permissions with an execution role](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html) in the Lambda documentation.
**Programming language knowledge**:
+ You will need basic knowledge of Python (or the programming language of your choice) to understand and modify the sample code for the Lambda function and the Lambda extension.
**Development environment**:
+ Set up a local development environment with the necessary tools and dependencies for building and deploying Lambda functions and extensions.
**AWS CLI or AWS Management Console**:
+ Install and configure the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) or use the AWS Management Console with appropriate credentials to interact with the required AWS services.
**Monitoring and logging**:
+ Become familiar with monitoring and logging best practices on AWS, including services such as Amazon CloudWatch and AWS CloudTrail for monitoring and auditing purposes.
+ Check CloudWatch Logs for your Lambda function to identify any errors or exceptions related to the Lambda Telemetry API integration. For troubleshooting guidance, see the [Lambda Telemetry API documentation](https://docs.aws.amazon.com/lambda/latest/dg/telemetry-api.html).
## Architecture
This pattern uses OpenSearch Service to store logs and telemetry data that are generated by Lambda functions. This approach enables you to quickly stream logs directly to your OpenSearch cluster, which reduces the latency and costs associated with using CloudWatch Logs as an intermediary.
|
|
| Your Lambda extension code can push telemetry to OpenSearch Service, either by directly using the OpenSearch API or by using an [OpenSearch client library](https://opensearch.org/docs/latest/clients/index/). The Lambda extension can use the bulk operations supported by the OpenSearch API to batch telemetry events together and send them to OpenSearch Service in a single request. |
| --- |
The following workflow diagram illustrates the log workflow for Lambda functions when you use an OpenSearch cluster as the endpoint.
![\[Workflow for sending telemetry data to an OpenSearch cluster.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/57fe8796-9f36-46cf-8304-f506242b9f04/images/283ccdcd-a0e1-40a2-a95a-3bd046bfa8ca.png)
The architecture includes these components:
+ Lambda function: The serverless function that generates logs and telemetry data during execution.
+ Lambda extension: A Python-based extension that uses the Lambda Telemetry API to integrate directly with the OpenSearch cluster. This extension runs alongside the Lambda function in the same execution environment.
+ Lambda Telemetry API: The API that enables Lambda extensions to send telemetry data, including logs, metrics, and traces, directly to third-party monitoring and observability tools.
+ Amazon OpenSearch Service cluster: A managed OpenSearch cluster that's hosted on AWS. This cluster is responsible for ingesting, storing, and indexing the log data streamed from the Lambda function through the Lambda extension.
The workflow consists of these steps:
1. The Lambda function is called, and generates logs and telemetry data during its execution.
1. The Lambda extension runs alongside the function to capture the logs and telemetry data by using the Lambda Telemetry API.
1. The Lambda extension establishes a secure connection with the OpenSearch Service cluster and streams the log data in real time.
1. The OpenSearch Service cluster ingests, indexes, and stores the log data to make it available for search, analysis, and visualization through the use of tools such as Kibana or other compatible applications.
By circumventing CloudWatch Logs and sending log data directly to the OpenSearch cluster, this solution provides several benefits:
+ Real-time log streaming and analysis, enabling faster troubleshooting and improved observability.
+ Reduced latency and potential retention limitations associated with CloudWatch Logs.
+ Flexibility to customize the Lambda extension or create your own extension for specific output formats or additional processing.
+ Integration with the search, analytics, and visualization capabilities of OpenSearch Service for log analysis and monitoring.
The [Epics](#send-telemetry-data-from-lambda-to-opensearch-for-analytics-visualization-epics) section provides step-by-step instructions for setting up the Lambda extension, configuring the Lambda function, and integrating with the OpenSearch Service cluster. For security considerations, cost optimization strategies, and tips for monitoring and troubleshooting the solution, see the [Best practices](#send-telemetry-data-from-lambda-to-opensearch-for-analytics-visualization-best-practices) section.
## Tools
**AWS services**
+ [AWS Lambda](https://aws.amazon.com/lambda/) is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second.
+ [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/) is a fully managed service provided by AWS that makes it easy to deploy, operate, and scale OpenSearch clusters in the cloud.
+ [Lambda extensions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-extensions.html) extends the functionality of your Lambda functions by running custom code alongside them. You can use Lambda extensions to integrate Lambda with various monitoring, observability, security, and governance tools.
+ [AWS Lambda Telemetry API](https://docs.aws.amazon.com/lambda/latest/dg/telemetry-api.html) enables you to use extensions to capture enhanced monitoring and observability data directly from Lambda and send it to a destination of your choice.
+ [CloudFormation](https://aws.amazon.com/cloudformation/) helps you model and set up your AWS resources so that you can spend less time managing those resources and more time focusing on your applications.
**Code repositories**
+ [AWS Lambda Extensions](https://github.com/aws-samples/aws-lambda-extensions) includes demos and sample projects from AWS and AWS Partners to help you get started with building your own extensions.
+ [Example Lambda Telemetry Integrations for OpenSearch](https://github.com/aws-samples/aws-lambda-extensions/tree/main/python-example-telemetry-opensearch-extension) provides a sample Lambda extension that demonstrates how to send logs from a Lambda function to an OpenSearch cluster.
**Other tools**
+ [OpenSearch](https://opensearch.org/faq/) is an open-source distributed search and analytics engine that provides a powerful platform for ingesting, storing, and analyzing large volumes of data.
+ Kibana is an open-source data visualization and exploration tool that you can use with OpenSearch. Note that the implementation of visualization and analytics is beyond the scope of this pattern. For more information, see the [Kibana documentation](https://www.elastic.co/guide/en/kibana/current/index.html) and other resources.
## Best practices
When you integrate the Lambda Telemetry API with OpenSearch, consider the following best practices.
**Security and access control**
+ **Secure communication**: Encrypt all communications between your Lambda functions and the OpenSearch cluster by using HTTPS. Configure the necessary SSL/TLS settings in your Lambda extension and OpenSearch configuration.
+ **IAM permissions**:
+ Extensions run in the same execution environment as the Lambda function, so they inherit the same level of access to resources such as the file system, networking, and environment variables.
+ Grant the minimum necessary IAM permissions to your Lambda functions to access the Lambda Telemetry API and write data to the OpenSearch cluster. Use the [principle of least privilege](https://docs.aws.amazon.com/lambda/latest/operatorguide/least-privilege.html) to limit the scope of permissions.
+ **OpenSearch access control**: Implement fine-grained access control in your OpenSearch cluster to restrict access to sensitive data. Use the built-in security features, such as user authentication, role-based access control, and index-level permissions, in OpenSearch.
+ **Trusted extensions**: Always install extensions from a trusted source only. Use infrastructure as code (IaC) tools such as CloudFormation to simplify the process of attaching the same extension configuration, including IAM permissions, to multiple Lambda functions. IaC tools also provide an audit record of the extensions and versions used previously.
+ **Sensitive data handling**: When building extensions, avoid logging sensitive data. Sanitize payloads and metadata before logging or persisting them for audit purposes.
**Cost optimization**
+ **Monitoring and alerting**: Set up monitoring and alerting mechanisms to track the volume of data being sent to OpenSearch from your Lambda functions. This will help you identify and address any potential cost overruns.
+ **Data retention**: Carefully consider the appropriate data retention period for your Lambda telemetry data in OpenSearch. Longer retention periods can increase storage costs, so balance your observability needs with cost optimization.
+ **Compression and indexing**: Enable data compression and optimize your OpenSearch indexing strategy to reduce the storage footprint of your Lambda telemetry data.
+ **Reduced reliance on CloudWatch**: By integrating the Lambda Telemetry API directly with OpenSearch, you can potentially reduce your reliance on CloudWatch Logs, which can result in cost savings. This is because the Lambda Telemetry API enables you to send logs directly to OpenSearch, which bypasses the need to store and process the data in CloudWatch.
**Scalability and reliability**
+ **Asynchronous processing**: Use asynchronous processing patterns, such as Amazon Simple Queue Service (Amazon SQS) or Amazon Kinesis, to decouple the Lambda function execution from the OpenSearch data ingestion. This helps maintain the responsiveness of your Lambda functions and improves the overall reliability of the system.
+ **OpenSearch cluster scaling**: Monitor the performance and resource utilization of your OpenSearch cluster, and scale it up or down as needed to handle the increasing volume of Lambda telemetry data.
+ **Failover and disaster recovery**: Implement a robust disaster recovery strategy for your OpenSearch cluster, including regular backups and the ability to quickly restore data in the event of a failure.
**Observability and monitoring**
+ **Dashboards and visualizations**: Use Kibana or other dashboard tools to create custom dashboards and visualizations that provide insights into the performance and health of your Lambda functions based on the telemetry data in OpenSearch.
+ **Alerting and notifications**: Set up alerts and notifications to proactively monitor for anomalies, errors, or performance issues in your Lambda functions. Integrate these alerts and notifications with your existing incident management processes.
+ **Tracing and correlation**: Ensure that your Lambda telemetry data includes relevant tracing information, such as request IDs or correlation IDs, to enable end-to-end observability and troubleshooting across your distributed serverless applications.
By following these best practices, you can ensure that your integration of the Lambda Telemetry API with OpenSearch is secure, cost-effective, and scalable, and provides comprehensive observability for your serverless applications.
## Epics
### Build and deploy the Lambda extension layer
| Task | Description | Skills required |
| --- | --- | --- |
| Download the source code. | Download the sample extensions from the [AWS Lambda Extensions](https://github.com/aws-samples/aws-lambda-extensions) repository. | App developer, Cloud architect |
| Navigate to the `python-example-telemetry-opensearch-extension` folder. | The [AWS Lambda Extensions](https://github.com/aws-samples/aws-lambda-extensions) repository that you downloaded contains numerous examples for several use cases and language runtimes. Navigate to the [python-example-telemetry-opensearch-extension](https://github.com/aws-samples/aws-lambda-extensions/tree/main/python-example-telemetry-opensearch-extension) folder to use the Python OpenSearch extension, which sends logs to OpenSearch. | App developer, Cloud architect |
| Add permissions to execute the extension endpoint. | Run the following command to make the extension endpoint executable:chmod +x python-example-telemetry-opensearch-extension/extension.py
| App developer, Cloud architect |
| Install the extension dependencies locally. | Run the following command to install local dependencies for the Python code:pip3 install -r python-example-telemetry-opensearch-extension/requirements.txt -t ./python-example-telemetry-opensearch-extension/
These dependencies will be mounted along with the extension code. | App developer, Cloud architect |
| Create a .zip package for the extension to deploy it as a layer. | The extension .zip file should contain a root directory called `extensions/`, where the extension executable is located, and another root directory called `python-example-telemetry-opensearch-extension/`, where the core logic of the extension and its dependencies are located.Create the .zip package for the extension:chmod +x extensions/python-example-telemetry-opensearch-extension
zip -r extension.zip extensions python-example-telemetry-opensearch-extension
| App developer, Cloud architect |
| Deploy the extension as a Lambda layer. | Publish the layer by using your extension .zip file and the following command:aws lambda publish-layer-version \
--layer-name "python-example-telemetry-opensearch-extension" \
--zip-file "fileb://extension.zip"
| App developer, Cloud architect |
### Integrate the extension into your function
| Task | Description | Skills required |
| --- | --- | --- |
| Add the layer to your function. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-telemetry-data-from-lambda-to-opensearch-for-analytics-visualization.html)For more information about adding a layer to your Lambda function, see the [Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/adding-layers.html). | App developer, Cloud architect |
| Set the environment variables for the function. | On the function page, choose the **Configuration** tab and add the following environment variables to your function:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-telemetry-data-from-lambda-to-opensearch-for-analytics-visualization.html) | App developer, Cloud architect |
### Add logging statements and test your function
| Task | Description | Skills required |
| --- | --- | --- |
| Add logging statements to your function. | Add logging statements to your function by using one of the [built-in logging mechanisms](https://docs.aws.amazon.com/lambda/latest/dg/python-logging.html) or your logging module of choice. Here are examples of logging messages in Python:print("Your Log Message Here")
logger = logging.getLogger(__name__)
logger.info("Test Info Log.")
logger.error("Test Error Log.") | App developer, Cloud architect |
| Test your function. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-telemetry-data-from-lambda-to-opensearch-for-analytics-visualization.html)You should see **Executing function: succeeded** if everything works properly. | App developer, Cloud architect |
### View your logs in OpenSearch
| Task | Description | Skills required |
| --- | --- | --- |
| Query your indexes. | In OpenSearch, run the following command to query your indexes:SELECT * FROM index-name
Your logs should be displayed in the query results. | Cloud architect |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Connectivity issues | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-telemetry-data-from-lambda-to-opensearch-for-analytics-visualization.html) |
| Data ingestion errors | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-telemetry-data-from-lambda-to-opensearch-for-analytics-visualization.html) |
## Related resources
+ [Example Lambda Telemetry Integrations for OpenSearch](https://github.com/aws-samples/aws-lambda-extensions/tree/main/python-example-telemetry-opensearch-extension) (GitHub repository)
+ [Augment Lambda functions using Lambda extensions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-extensions.html) (Lambda documentation)
+ [Lambda Telemetry API](https://docs.aws.amazon.com/lambda/latest/dg/telemetry-api.html) (Lambda documentation)
+ [Introducing the AWS Lambda Telemetry API](https://aws.amazon.com/blogs/compute/introducing-the-aws-lambda-telemetry-api/) (AWS blog post)
+ [Integrating the AWS Lambda Telemetry API with Prometheus and OpenSearch](https://aws.amazon.com/blogs/opensource/integrating-the-aws-lambda-telemetry-api-with-prometheus-and-opensearch) (AWS blog post)
## Additional information
**Altering the log structure**
The extension sends logs as a nested document to OpenSearch by default. This allows you to perform nested queries to retrieve individual column values.
If the default log output doesn't meet your specific needs, you can customize it by modifying the source code of the Lambda extension that’s provided by AWS. AWS encourages customers to adapt the output to suit their business requirements. To change the log output, locate the `dispatch_to_opensearch` function in the `telemetry_dispatcher.py` file within the extension's source code and make the necessary alterations.
# Set up a serverless cell router for a cell-based architecture
*Mian Tariq and Ioannis Lioupras, Amazon Web Services*
## Summary
As the entry point to a global cell-based application's system, the cell router is responsible for efficiently assigning users to the appropriate cells and providing the endpoints to the users. The cell router handles functions such as storing user-to-cell mappings, monitoring cell capacity, and requesting new cells when needed. It's important to maintain cell-router functionality during potential disruptions.
The cell-router design framework in this pattern focuses on resilience, scalability, and overall performance optimization. The pattern uses static routing, where clients cache endpoints upon initial login and communicate directly with cells. This decoupling enhances system resilience by helping to ensure uninterrupted functionality of the cell-based application during a cell-router impairment.
This pattern uses an AWS CloudFormation template to deploy the architecture. For details about what the template deploys, or to deploy the same configuration by using the AWS Management Console, see the [Additional information](#serverless-cell-router-architecture-additional) section.
**Important**
The demonstration, the code, and the CloudFormation template presented in this pattern are intended for explanatory purposes only. The material provided is solely for the purpose of illustrating the design pattern and aiding in comprehension. The demo and code are not production-ready and should not be used for any live production activities. Any attempt to use the code or demo in a production environment is strongly discouraged and is at your own risk. We recommend consulting with appropriate professionals and performing thorough testing before implementing this pattern or any of its components in a production setting.
## Prerequisites and limitations
**Prerequisites**
+ An active Amazon Web Services (AWS) account
+ The latest version of [AWS Command Line Interface (AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html))
+ [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) with the necessary permissions to create the CloudFormation stack, AWS Lambda functions, and related resources
**Product versions**
+ Python 3.12
## Architecture
The following diagram shows a high-level design of the cell router.
![\[The five-step process of the cell router.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/fd2fbf9d-9ae4-4c27-bc32-cf117350137a/images/feb90b51-dd91-483b-b5a3-b0a5359686e3.png)
The diagram steps through the following workflow:
1. The user contacts Amazon API Gateway, which serves as the front for the cell-router API endpoints.
1. Amazon Cognito handles the authentication and authorization.
1. The AWS Step Functions workflow consists of the following components:
+ **Orchestrator** ‒ The `Orchestrator` uses AWS Step Functions to create a workflow, or state machine. The workflow is triggered by the cell router API. The `Orchestrator` executes Lambda functions based on the resource path.
+ **Dispatcher** ‒ The `Dispatcher` Lambda function identifies and assigns one static cell per registered new user. The function searches for the cell with the least number of users, assigns it to the user, and returns the endpoints.
+ **Mapper** ‒ The `Mapper` operation handles the user-to-cell mappings within the `RoutingDB` Amazon DynamoDB database that was created by the CloudFormation template. When triggered, the `Mapper` function provides the already assigned users with their endpoints.
+ **Scaler** ‒ The `Scaler` function keeps track of the cell occupancy and available capacity. When needed, the `Scaler` function can send a request through Amazon Simple Queue Service (Amazon SQS) to the Provision and Deploy layer to request new cells.
+ **Validator** ‒ The `Validator` function validates the cell endpoints and detects any potential issues.
1. The `RoutingDB` stores cell information and attributes (API endpoints, AWS Region, state, metrics).
1. When the available capacity of cells exceeds a threshold, the cell router requests provisioning and deployment services through Amazon SQS to create new cells.
When new cells are created, `RoutingDB` gets updated from the Provision and Deploy layer. However, that process is beyond the scope of this pattern. For an overview of cell-based architecture design premises and details about the cell-router design used in this pattern, see the [Additional information](#serverless-cell-router-architecture-additional) section.
## Tools
**AWS services**
+ [Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) helps you create, publish, maintain, monitor, and secure REST, HTTP, and WebSocket APIs at any scale.
+ [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and AWS Regions.
+ [Amazon Cognito](https://docs.aws.amazon.com/cognito/latest/developerguide/what-is-amazon-cognito.html) provides authentication, authorization, and user management for web and mobile apps.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Amazon Simple Queue Service (Amazon SQS)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine Lambda functions and other AWS services to build business-critical applications.
**Other tools**
+ [Python](https://www.python.org/) is a general-purpose computer programming language.
**Code repository**
The code for this pattern is available in the GitHub [Serverless-Cell-Router](https://github.com/aws-samples/Serverless-Cell-Router/) repository.
## Best practices
For best practices when building cell-based architectures, see the following AWS Well-Architected guidance:
+ [Reducing the Scope of Impact with Cell-Based Architecture](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/reducing-scope-of-impact-with-cell-based-architecture.html)
+ [AWS Well-Architected Framework Reliability Pillar: REL10-BP04 Use bulkhead architectures to limit scope of impact](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_fault_isolation_use_bulkhead.html)
## Epics
### Prepare source files
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the example code repository. | To clone the Serverless-Cell-Router GitHub repository to your computer, use the following command:git clone https://github.com/aws-samples/Serverless-Cell-Router/
| Developer |
| Set up AWS CLI temporary credentials. | Configure the AWS CLI with credentials for your AWS account. This walkthrough uses temporary credentials provided by the AWS IAM Identity Center **Command line or programmatic access** option. This sets the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` AWS environment variables with the appropriate credentials for use with the AWS CLI. | Developer |
| Create an S3 bucket. | Create an S3 bucket that will be used to store and access the Serverless-Cell-Router Lambda functions for deployment by the CloudFormation template. To create the S3 bucket, use following command: aws s3api create-bucket --bucket --region eu-central-1 --create-bucket-configuration LocationConstraint=eu-central-1
| Developer |
| Create .zip files. | Create one .zip file for each Lambda function located in the [Functions](https://github.com/aws-samples/Serverless-Cell-Router/tree/main/Functions) directory. These .zip files will be used to deploy the Lambda functions. On a Mac, use the following `zip` commands:zip -j mapper-scr.zip Functions/Mapper.py
zip -j dispatcher-scr.zip Functions/Dispatcher.py
zip -j scaler-scr.zip Functions/Scaler.py
zip -j cp validator-scr.zip Functions/Validator.py
zip -j dynamodbDummyData-scr.zip Functions/DynamodbDummyData.py
| Developer |
| Copy the .zip files to the S3 bucket. | To copy all the Lambda function .zip files to the S3 bucket, use the following commands:aws s3 cp mapper-scr.zip s3://
aws s3 cp dispatcher-scr.zip s3://
aws s3 cp scaler-scr.zip s3://
aws s3 cp validator-scr.zip s3://
aws s3 cp dynamodbDummyData-scr.zip s3://
| Developer |
### Create the CloudFormation stack
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the CloudFormation template. | To deploy the CloudFormation template, run the following AWS CLI command:aws cloudformation create-stack --stack-name serverless.cell-router \
--template-body file://Serverless-Cell-Router-Stack-v10.yaml \
--capabilities CAPABILITY_IAM \
--parameters ParameterKey=LambdaFunctionMapperS3KeyParameterSCR,ParameterValue=mapper-scr.zip \
ParameterKey=LambdaFunctionDispatcherS3KeyParameterSCR,ParameterValue=dispatcher-scr.zip \
ParameterKey=LambdaFunctionScalerS3KeyParameterSCR,ParameterValue=scaler-scr.zip \
ParameterKey=LambdaFunctionAddDynamoDBDummyItemsS3KeyParameterSCR,ParameterValue=dynamodbDummyData-scr.zip \
ParameterKey=LambdaFunctionsS3BucketParameterSCR,ParameterValue= \
ParameterKey=CognitoDomain,ParameterValue= \
--region
| Developer |
| Check progress. | Sign in to the AWS Management Console, open the CloudFormation console at [https://console.aws.amazon.com/cloudformation/](), and check the progress of stack development. When the status is `CREATE_COMPLETE`, the stack has been deployed successfully. | Developer |
### Assess and verify
| Task | Description | Skills required |
| --- | --- | --- |
| Assign cells to the user. | To initiate the `Orchestrator`, run the following curl command:curl -X POST \
-H "Authorization: Bearer {User id_token}" \
https://xxxxxx.execute-api.eu-central-1.amazonaws.com/Cell_Router_Development/cells
The `Orchestrator` triggers the execution of the `Dispatcher` function. The `Dispatcher`, in turn, verifies the existence of the user. If the user is found, the `Dispatcher` returns the associated cell ID and endpoint URLs. If the user isn't found, the `Dispatcher` allocates a cell to the user and sends the cell ID to the `Scaler` function for assessment of the assigned cell's residual capacity.The `Scaler` function's response is the following:`"cellID : cell-0002 , endPoint_1 : https://xxxxx.execute-api.eu-north-1.amazonaws.com/ , endPoint_2 : https://xxxxxxx.execute-api.eu-central-1.amazonaws.com/"` | Developer |
| Retrieve user cells. | To use the `Orchestrator` to execute the `Mapper` function, run the following command:curl -X POST \
-H "Authorization: Bearer {User id_token}" \
https://xxxxxxxxx.execute-api.eu-central-1.amazonaws.com/Cell_Router_Development/mapper
The `Orchestrator` searches for the cell assigned to the user and returns the cell ID and URLs in the following response:`"cellID : cell-0002 , endPoint_1 : https://xxxxx.execute-api.eu-north-1.amazonaws.com/ , endPoint_2 : https://xxxxxxx.execute-api.eu-central-1.amazonaws.com/"` | Developer |
### Clean up
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up the resources. | To avoid incurring additional charges in your account, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/serverless-cell-router-architecture.html) | App developer |
## Related resources
**References**
+ [Static stability using Availability Zones](https://aws.amazon.com/builders-library/static-stability-using-availability-zones/)
+ [AWS Fault Isolation Boundaries: Static stability](https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/static-stability.html)
**Video**
[Physalia: Cell-based Architecture to Provide Higher Availability on Amazon EBS](https://www.youtube.com/watch?v=6IknqRZMFic)
[https://www.youtube-nocookie.com/embed/6IknqRZMFic?controls=0](https://www.youtube-nocookie.com/embed/6IknqRZMFic?controls=0)
## Additional information
**Cell-based architecture design premises**
Although this pattern focuses on the cell router, it's important to understand the whole environment. The environment is structured into three discrete layers:
+ The Routing layer, or Thin layer, which contains the cell router
+ The Cell layer, comprising various cells
+ The Provision and Deploy Layer, which provisions cells and deploys the application
Each layer sustains functionality even in the event of impairments affecting other layers. AWS accounts serve as a fault isolation boundary.
The following diagram shows the layers at a high level. The Cell layer and the Provision and Deploy layer are outside the scope of this pattern.
![\[The Routing layer, the Cell layer with multiple cell accounts, and the Provision and Deploy layer.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/fd2fbf9d-9ae4-4c27-bc32-cf117350137a/images/137ac34d-43c3-42b6-95de-a365ff611ce8.png)
For more information about cell-based architecture, see [Reducing the Scope of Impact with Cell-Based Architecture: Cell routing](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/cell-routing.html).
**Cell-router design pattern**
The cell router is a shared component across cells. To mitigate potential impacts, it's important for the Routing layer to use a simplistic and horizontally scalable design that's as thin as possible. Serving as the system’s entry point, the Routing layer consists of only the components that are required to efficiently assign users to the appropriate cells. Components within this layer don't engage in the management or creation of cells.
This pattern uses static routing, which means that the client caches the endpoints at the initial login and subsequently establishes direct communication with the cell. Periodic interactions between the client and the cell router are initiated to confirm the current status or retrieve any updates. This intentional decoupling enables uninterrupted operations for existing users in the event of cell-router downtime, and it provides continued functionality and resilience within the system.
In this pattern, the cell router supports the following functionalities:
+ Retrieving cell data from the cell database in the Provision and Deploy layer and storing or updating the local database.
+ Assigning a cell to each new registered user of the application by using the cell assignment algorithm.
+ Storing the user-to-cells mapping in the local database.
+ Checking the capacity of the cells during user assignment and raising an event for the vending machine to the Provision and Deploy layer to create cells.
+ Using the cell creation criteria algorithm to provide this functionality.
+ Responding to the newly registered user requests by providing the URLs of the static cells. These URLs will be cached on the client with a time to live (TTL).
+ Responding to the existing user requests of an invalid URL by providing a new or updated URL.
To further understand the demonstration cell router that is set up by the CloudFormation template, review the following components and steps:
1. Set up and configure the Amazon Cognito user pool.
1. Set up and configure the API Gateway API for the cell router.
1. Create a DynamoDB table.
1. Create and configure an SQS queue.
1. Implement the `Orchestrator`.
1. Implement the Lambda functions: `Dispatcher`, `Scaler`, `Mapper`, `Validator`.
1. Asses and verify.
The presupposition is that the Provision and Deploy layer is already established. Its implementation details fall beyond the scope of this artifact.
Because these components are set up and configured by an CloudFormation template, the following steps are presented at a descriptive and high level. The assumption is that you have the required AWS skills to complete the setup and configuration.
*1. Setup and configure the Amazon Cognito user pool*
Sign in to the AWS Management Console, and open the Amazon Cognito console at [https://console.aws.amazon.com/cognito/](). Set up and configure an Amazon Cognito user pool named `CellRouterPool`, with app integration, hosted UI, and the necessary permissions.
*2. Set up and configure the API Gateway API for the cell router*
Open the API Gateway console at [https://console.aws.amazon.com/apigateway/](). Set up and configure an API named `CellRouter`, using an Amazon Cognito authorizer integrated with the Amazon Cognito user pool `CellRouterPool`. Implement the following elements:
+ `CellRouter` API resources, including `POST` methods
+ Integration with the Step Functions workflow implemented in step 5
+ Authorization through the Amazon Cognito authorizer
+ Integration request and response mappings
+ Allocation of necessary permissions
*3. Create a DynamoDB table*
Open the DynamoDB console at [https://console.aws.amazon.com/dynamodb/](), and create a standard DynamoDB table called `tbl_router` with the following configuration:
+ **Partition key** ‒ `marketId`
+ **Sort key** ‒ `cellId`
+ **Capacity mode** ‒ Provisioned
+ **Point-in-time recovery (PITR)** ‒ Off
On the **Indexes** tab, create a global secondary index called `marketId-currentCapacity-index`. The `Scaler` Lambda function will use the index to conduct efficient searches for the cell with the lowest number of assigned users.
Create the table structure with the following attributes:
+ `marketId` ‒ Europe
+ `cellId` ‒ cell-0002
+ `currentCapacity` ‒ 2
+ `endPoint_1` ‒
+ `endPoint_2` ‒
+ `IsHealthy` ‒ True
+ `maxCapacity` ‒ 10
+ `regionCode_1` ‒ `eu-north-1`
+ `regionCode_2` ‒ `eu-central-1`
+ `userIds` ‒
*4. Create and configure an SQS queue*
Open the Amazon SQS console at [https://console.aws.amazon.com/sqs/](), and create a standard SQS queue called `CellProvisioning` configured with **Amazon SQS key** encryption.
*5. Implement the Orchestrator*
Develop a Step Functions workflow to serve as the `Orchestrator` for the router. The workflow is callable through the cell router API. The workflow executes the designated Lambda functions based on the resource path. Integrate the step function with the API Gateway API for the cell router `CellRouter`, and configure the necessary permissions to invoke the Lambda functions.
The following diagram shows the workflow. The choice state invokes one of the Lambda functions. If the Lambda function is successful, the workflow ends. If the Lambda function fails, fail state is called.
![\[A diagram of the workflow with the four functions and ending in a fail state.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/fd2fbf9d-9ae4-4c27-bc32-cf117350137a/images/cfe8d029-6f30-49a1-aaad-cad503bdcbae.png)
*6. Implement the Lambda functions*
Implement the `Dispatcher`, `Mapper`, `Scaler`, and `Validator` functions. When you set up and configure each function in the demonstration, define a role for the function and assign the necessary permissions for performing required operations on the DynamoDB table `tbl_router`. Additionally, integrate each function into the workflow `Orchestrator`.
*Dispatcher function*
The `Dispatcher` function is responsible for identifying and assigning a single static cell for each new registered user. When a new user registers with the global application, the request goes to the `Dispatcher` function. The function processes the request by using predefined evaluation criteria such as the following:
1. **Region** ‒ Select the cell in the market where the user is located. For example, if the user is accessing the global application from Europe, select a cell that uses AWS Regions in Europe.
1. **Proximity or latency** ‒ Select the cell closest to the user For example, if the user is accessing the application from Holland, the function considers a cell that uses Frankfurt and Ireland. The decision regarding which cell is closest is based on metrics such as latency between the user's location and the cell Regions. For this example pattern, the information is statically fed from the Provision and Deploy layer.
1. **Health** ‒ The `Dispatcher` function checks whether the selected cell is healthy based on the provided cell state (Healthy = true or false).
1. **Capacity** ‒ The user distribution is based on *least number of users in a cell* logic, so the user is assigned to the cell that has least number of users.
**Note**
These criteria are presented to explain this example pattern only. For a real-life cell-router implementation, you can define more refined and use case‒based criteria.
The `Orchestrator` invokes the Dispatcher function to assign users to cells. In this demo function, the market value is a static parameter defined as `europe`.
The `Dispatcher` function assesses whether a cell is already assigned to the user. If the cell is already assigned, the `Dispatcher` function returns the cell's endpoints. If no cell is assigned to the user, the function searches for the cell with the least number of users, assigns it to the user, and returns the endpoints. The efficiency of the cell search query is optimized by using the global secondary index.
*Mapper function*
The `Mapper` function oversees the storage and maintenance of user-to-cell mappings in the database. A singular cell is allocated to each registered user. Each cell has two distinct URLs—one for each AWS Region. Serving as API endpoints hosted on API Gateway, these URLs function as inbound points to the global application.
When the `Mapper` function receives a request from the client application, it runs a query on the DynamoDB table `tbl_router` to retrieve the user-to-cell mapping that is associated with the provided email ID. If it finds an assigned cell, the `Mapper` function promptly provides the cell's two URLs. The `Mapper` function also actively monitors alterations to the cell URLs, and it initiates notifications or updates to user settings.
*Scaler function*
The `Scaler` function manages the residual capacity of the cell. For each new user-registration request, the `Scaler` function assesses the available capacity of the cell that the `Dispatcher` function assigned to the user. If the cell has reached its predetermined limit according to the specified evaluation criteria, the function initiates a request through an Amazon SQS queue to the Provision and Deploy layer, soliciting the provisioning and deployment of new cells. The scaling of cells can be executed based on a set of evaluation criteria such as the following:
1. **Maximum users** ‒ Each cell can have 500 maximum number of users.
1. **Buffer capacity** ‒ The buffer capacity of each cell is 20 percent, which means that each cell can be assigned to 400 users at any time. The remaining 20 percent buffer capacity is reserved for future use cases and handling of unexpected scenarios (for example, when cell creation and provisioning services are unavailable).
1. **Cell creation** ‒ As soon as an existing cell reaches 70 percent of capacity, a request is triggered to create an additional cell.
**Note**
These criteria are presented to explain this example pattern only. For a real-life cell-router implementation, you can define more refined and use case‒based criteria.
The demonstration `Scaler` code is executed by the `Orchestrator` after the `Dispatcher` successfully assigns a cell to the newly registered user. The `Scaler`, upon receipt of the cell ID from the `Dispatcher`, evaluates whether the designated cell has adequate capacity to accommodate additional users, based on predefined evaluation criteria. If the cell's capacity is insufficient, the `Scaler` function dispatches a message to the Amazon SQS service. This message is retrieved by the service within the Provision and Deploy layer, initiating the provisioning of a new cell.
**Validator function**
The `Validator` function identifies and resolves issues pertaining to cell access. When a user signs in to the global application, the application retrieves the cell's URLs from the user profile settings and routes user requests to one of the two assigned Regions within the cell. If the URLs are inaccessible, the application can dispatch a validate URL request to the cell router. The cell-router `Orchestrator` invokes the `Validator`. The `Validator` initiates the validation process. Validation might include, among other checks, the following:
+ Cross-referencing cell URLs in the request with URLs stored in database to identify and process potential updates
+ Running a deep health check (for example, an `HTTP GET` request to the cell's endpoint)
In conclusion, the `Validator` function delivers responses to client application requests, furnishing validation status along with any required remediation steps.
The `Validator` is designed to enhance user experience. Consider a scenario where certain users encounter difficulty accessing the global application because an incident causes cells to be temporarily unavailable. Instead of presenting generic errors, the `Validator` function can provide instructive remediation steps. These steps might include the following actions:
+ Inform users about the incident.
+ Provide an approximate wait time before service availability.
+ Provide a support contact number for obtaining additional information.
The demo code for the `Validator` function verifies that the user-supplied cell URLs in the request match the records stored in the `tbl_router` table. The `Validator` function also checks whether the cells are healthy.
# Set up private access to an Amazon S3 bucket through a VPC endpoint
*Martin Maritsch, Nicolas Jacob Baer, Gabriel Rodriguez Garcia, Shukhrat Khodjaev, Mohan Gowda Purushothama, and Joaquin Rinaudo, Amazon Web Services*
## Summary
In Amazon Simple Storage Service (Amazon S3), presigned URLs enable you to share files of arbitrary size with target users. By default, Amazon S3 presigned URLs are accessible from the internet within an expiration time window, which makes them convenient to use. However, corporate environments often require access to Amazon S3 presigned URLs to be limited to a private network only.
This pattern presents a serverless solution for securely interacting with S3 objects by using presigned URLs from a private network without internet traversal. In the architecture, users access an Application Load Balancer through an internal domain name. Traffic is routed internally through Amazon API Gateway and a virtual private cloud (VPC) endpoint for the S3 bucket. The AWS Lambda function generates presigned URLs for file downloads through the private VPC endpoint, which helps enhance security and privacy for sensitive data.
## Prerequisites and limitations
**Prerequisites**
+ A VPC that includes a subnet deployed in an AWS account that is connected to the corporate network (for example, through AWS Direct Connect).
**Limitations**
+ The S3 bucket must have the same name as the domain, so we recommend that you check [Amazon S3 bucket naming rules.](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html)
+ This sample architecture doesn't include monitoring features for the deployed infrastructure. If your use case requires monitoring, consider adding [AWS monitoring services](https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/welcome.html).
+ This sample architecture doesn't include input validation. If your use case requires input validation and an increased level of security, consider [using AWS WAF to protect your API](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-aws-waf.html).
+ This sample architecture doesn't include access logging with the Application Load Balancer. If your use case requires access logging, consider enabling [load balancer access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html).
**Versions**
+ Python version 3.11 or later
+ Terraform version 1.6 or later
## Architecture
**Target technology stack**
The following AWS services are used in the target technology stack:
+ **Amazon S3** is the core storage service used for uploading, downloading, and storing files securely.
+ **Amazon API Gateway** exposes resources and endpoints for interacting with the S3 bucket. This service plays a role in generating presigned URLs for downloading or uploading data.
+ **AWS Lambda** generates presigned URLs for downloading files from Amazon S3. The Lambda function is called by API Gateway.
+ **Amazon VPC** deploys resources within a VPC to provide network isolation. The VPC includes subnets and routing tables to control traffic flow.
+ **Application Load Balancer** routes incoming traffic either to API Gateway or to the VPC endpoint of the S3 bucket. It allows users from the corporate network to access resources internally.
+ **VPC endpoint for Amazon S3** enables direct, private communication between resources in the VPC and Amazon S3 without traversing the public internet.
+ **AWS Identity and Access Management (IAM)** controls access to AWS resources. Permissions are set up to ensure secure interactions with the API and other services.
**Target architecture**
![\[Setting up private access to an S3 bucket through a VPC endpoing\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/683ca6a1-789c-4444-bcbf-e4e80d253df3/images/1ca7ee17-d346-4eb9-bf61-ccf42528a401.png)
The diagram illustrates the following:
1. Users from the corporate network can access the Application Load Balancer through an internal domain name. We assume that a connection exists between the corporate network and the intranet subnet in the AWS account (for example, through a Direct Connect connection).
1. The Application Load Balancer routes incoming traffic either to API Gateway to generate presigned URLs to download or upload data to Amazon S3, or to the VPC endpoint of the S3 bucket. In both scenarios, requests are routed internally and do not need to traverse the internet.
1. API Gateway exposes resources and endpoints to interact with the S3 bucket. In this example, we provide an endpoint to download files from the S3 bucket, but this could be extended to provide upload functionality as well.
1. The Lambda function generates the presigned URL to download a file from Amazon S3 by using the domain name of the Application Load Balancer instead of the public Amazon S3 domain.
1. The user receives the presigned URL and uses it to download the file from Amazon S3 by using the Application Load Balancer. The load balancer includes a default route to send traffic that's not intended for the API toward the VPC endpoint of the S3 bucket.
1. The VPC endpoint routes the presigned URL with the custom domain name to the S3 bucket. The S3 bucket must have the same name as the domain.
**Automation and scale**
This pattern uses Terraform to deploy the infrastructure from the code repository into an AWS account.
## Tools
**Tools**
+ [Python](https://www.python.org/) is a general-purpose computer programming language.
+ [Terraform](https://www.terraform.io/) is an infrastructure as code (IaC) tool from HashiCorp that helps you create and manage cloud and on-premises resources.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command-line shell.
**Code repository**
The code for this pattern is available in a GitHub repository at [https://github.com/aws-samples/private-s3-vpce](https://github.com/aws-samples/private-s3-vpce).
## Best practices
The sample architecture for this pattern uses [IAM permissions](https://docs.aws.amazon.com/apigateway/latest/developerguide/permissions.html) to control access to the API. Anyone who has valid IAM credentials can call the API. If your use case requires a more complex authorization model, you might want to [use a different access control mechanism](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html).
## Epics
### Deploy the solution in an AWS account
| Task | Description | Skills required |
| --- | --- | --- |
| Obtain AWS credentials. | Review your AWS credentials and your access to your account. For instructions, see [Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) in the AWS CLI documentation. | AWS DevOps, General AWS |
| Clone the repository. | Clone the GitHub repository provided with this pattern:git clone https://github.com/aws-samples/private-s3-vpce
| AWS DevOps, General AWS |
| Configure variables. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-private-access-to-an-amazon-s3-bucket-through-a-vpc-endpoint.html) | AWS DevOps, General AWS |
| Deploy solution. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-private-access-to-an-amazon-s3-bucket-through-a-vpc-endpoint.html) | AWS DevOps, General AWS |
### Test the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Create a test file. | Upload a file to Amazon S3 to create a test scenario for the file download. You can use the [Amazon S3 console](https://console.aws.amazon.com/s3/) or the following AWS CLI command:aws s3 cp /path/to/testfile s3://your-bucket-name/testfile
| AWS DevOps, General AWS |
| Test presigned URL functionality. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-private-access-to-an-amazon-s3-bucket-through-a-vpc-endpoint.html) | AWS DevOps, General AWS |
| Clean up. | Make sure to remove the resources when they are no longer required:terraform destroy
| AWS DevOps, General AWS |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| S3 object key names with special characters such as number signs (\$1) break URL parameters and lead to errors. | Encode URL parameters properly, and make sure that the S3 object key name follows [Amazon S3 guidelines](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). |
## Related resources
Amazon S3:
+ [Sharing objects with presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html)
+ [Controlling access from VPC endpoints with bucket policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies-vpc-endpoint.html)
Amazon API Gateway:
+ [Use VPC endpoint policies for private APIs in API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-vpc-endpoint-policies.html)
Application Load Balancer:
+ [Hosting Internal HTTPS Static Websites with ALB, S3, and PrivateLink](https://aws.amazon.com/blogs/networking-and-content-delivery/hosting-internal-https-static-websites-with-alb-s3-and-privatelink/) (AWS blog post)
# Troubleshoot states in AWS Step Functions by using Amazon Bedrock
*Aniket Kurzadkar and Sangam Kushwaha, Amazon Web Services*
## Summary
AWS Step Functions error handling capabilities can help you see an error that occurs during a state in a [workflow](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-statemachines.html), but it can still be a challenge to find the root cause of an error and debug it. This pattern addresses that challenge and shows how Amazon Bedrock can help you resolve errors that occur during states in Step Functions.
Step Functions provides workflow orchestration, making it easier for developers to automate processes. Step Functions also provides error handling functionality that provides the following benefits:
+ Developers can create more resilient applications that don't fail completely when something goes wrong.
+ Workflows can include conditional logic to handle different types of errors differently.
+ The system can automatically retry failed operations, perhaps with exponential backoff.
+ Alternative execution paths can be defined for error scenarios, allowing the workflow to adapt and continue processing.
When an error occurs in a Step Functions workflow, this pattern shows how the error message and context can be sent to a foundation model (FM) like Claude 3 that’s supported by Step Functions. The FM can analyze the error, categorize it, and suggest potential remediation steps.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Basic understanding of [AWS Step Functions and workflows](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-statemachines.html)
+ Amazon Bedrock [API connectivity](https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started-api.html)
**Limitations**
+ You can use this pattern’s approach for various AWS services. However, the results might vary according to the prompt created by AWS Lambda that’s subsequently evaluated by Amazon Bedrock.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
## Architecture
The following diagram shows the workflow and architecture components for this pattern.
![\[Workflow for error handling and notification using Step Functions, Amazon Bedrock, and Amazon SNS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/78f86c74-c9de-4562-adcc-105b87a77a54/images/d8eda499-ea1d-45e5-8a36-e04a44ad5c4b.png)
The diagram shows the automated workflow for error handling and notification in a Step Functions state machine:
1. The developer starts a state machine’s execution.
1. The Step Functions state machine begins processing its states. There are two possible outcomes:
+ (a) If all states execute successfully, the workflow proceeds directly to Amazon SNS for an email success notification.
+ (b) If any state fails, the workflow moves to the error handling Lambda function.
1. In case of an error, the following occurs:
+ (a) The Lambda function (error handler) is triggered. The Lambda function extracts the error message from the event data that the Step Functions state machine passed to it. Then the Lambda function prepares a prompt based on this error message and sends the prompt to Amazon Bedrock. The prompt requests solutions and suggestions related to the specific error encountered.
+ (b) Amazon Bedrock, which hosts the generative AI model, processes the input prompt. (This pattern uses the Anthropic Claude 3 foundation model (FM), which is one of many FMs that Amazon Bedrock supports.) The AI model analyses the error context. Then the model generates a response that can include explanations of why the error occurred, potential solutions to resolve the error, and suggestions to avoid making the same mistakes in the future.
Amazon Bedrock returns its AI-generated response to the Lambda function. The Lambda function processes the response, potentially formatting it or extracting key information. Then the Lambda function sends the response to the state machine output.
1. After error handling or successful execution, the workflow concludes by triggering Amazon SNS to send an email notification.
## Tools
**AWS services**
+ [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models (FMs) from leading AI startups and Amazon available for your use through a unified API.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Simple Notification Service (Amazon SNS)](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine AWS Lambda functions and other AWS services to build business-critical applications.
## Best practices
+ Given that Amazon Bedrock is a generative AI model that learns from trained data, it also uses that data to train and generate context. As a best practice, conceal any private information that might lead to data leak problems.
+ Although generative AI can provide valuable insights, critical error-handling decisions should still involve human oversight, especially in production environments.
## Epics
### Create a state machine for your workflow
| Task | Description | Skills required |
| --- | --- | --- |
| Create a state machine. | To create a state machine that’s appropriate for your workflow, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/troubleshooting-states-in-aws-step-functions.html) | AWS DevOps |
### Create a Lambda function
| Task | Description | Skills required |
| --- | --- | --- |
| Create a Lambda function. | To create a Lambda function, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/troubleshooting-states-in-aws-step-functions.html) | AWS DevOps |
| Set up the required logic in the Lambda code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/troubleshooting-states-in-aws-step-functions.html)client = boto3.client(
service_name="bedrock-runtime", region_name="selected-region"
)
# Invoke Claude 3 with the text prompt
model_id = "your-model-id" # Select your Model ID, Based on the Model Id, Change the body format
try:
response = client.invoke_model(
modelId=model_id,
body=json.dumps(
{
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": [{"type": "text", "text": prompt}],
}
],
}
),
)
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/troubleshooting-states-in-aws-step-functions.html) | AWS DevOps |
### Integrate Step Functions with Lambda
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Lambda to handle errors in Step Functions. | To set up Step Functions to handle errors without disrupting the workflow, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/troubleshooting-states-in-aws-step-functions.html) | AWS DevOps |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Lambda cannot access the Amazon Bedrock API (Not authorized to perform) | This error occurs when the Lambda role doesn’t have permission to access the Amazon Bedrock API. To resolve this issue, add the `AmazonBedrockFullAccess` policy for the Lambda role. For more information, see [AmazonBedrockFullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonBedrockFullAccess.html) in the *AWS Managed Policy Reference Guide*. |
| Lambda timeout error | Sometimes it might take more than 30 seconds to generate a response and send it back, depending on the prompt. To resolve this issue, increase the configuration time. For more information, see [Configure Lambda function timeout](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonBedrockFullAccess.html) in the *AWS Lambda Developer Guide*. |
## Related resources
+ [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html)
+ [Amazon Bedrock API access](https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started-api.html)
+ [Create your first Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html)
+ [Developing workflows with Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/developing-workflows.html#development-run-debug)
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html)
# More patterns
**Topics**
+ [Access, query, and join Amazon DynamoDB tables using Athena](access-query-and-join-amazon-dynamodb-tables-using-athena.md)
+ [Automate Amazon CodeGuru reviews for AWS CDK Python applications by using GitHub Actions](automate-amazon-codeguru-reviews-for-aws-cdk-python-applications.md)
+ [Automate AWS resource assessment](automate-aws-resource-assessment.md)
+ [Automate deployment of nested applications using AWS SAM](automate-deployment-of-nested-applications-using-aws-sam.md)
+ [Automate AWS Supply Chain data lakes deployment in a multi-repository setup](automate-the-deployment-of-aws-supply-chain-data-lakes.md)
+ [Automate the replication of Amazon RDS instances across AWS accounts](automate-the-replication-of-amazon-rds-instances-across-aws-accounts.md)
+ [Automate the setup of inter-Region peering with AWS Transit Gateway](automate-the-setup-of-inter-region-peering-with-aws-transit-gateway.md)
+ [Automatically archive items to Amazon S3 using DynamoDB TTL](automatically-archive-items-to-amazon-s3-using-dynamodb-ttl.md)
+ [Automatically detect changes and initiate different CodePipeline pipelines for a monorepo in CodeCommit](automatically-detect-changes-and-initiate-different-codepipeline-pipelines-for-a-monorepo-in-codecommit.md)
+ [Build a multi-tenant serverless architecture in Amazon OpenSearch Service](build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.md)
+ [Build an advanced mainframe file viewer in the AWS Cloud](build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.md)
+ [Calculate value at risk (VaR) by using AWS services](calculate-value-at-risk-var-by-using-aws-services.md)
+ [Copy AWS Service Catalog products across different AWS accounts and AWS Regions](copy-aws-service-catalog-products-across-different-aws-accounts-and-aws-regions.md)
+ [Create dynamic CI pipelines for Java and Python projects automatically](create-dynamic-ci-pipelines-for-java-and-python-projects-automatically.md)
+ [Decompose monoliths into microservices by using CQRS and event sourcing](decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.md)
+ [Deploy a React-based single-page application to Amazon S3 and CloudFront](deploy-a-react-based-single-page-application-to-amazon-s3-and-cloudfront.md)
+ [Deploy an Amazon API Gateway API on an internal website using private endpoints and an Application Load Balancer](deploy-an-amazon-api-gateway-api-on-an-internal-website-using-private-endpoints-and-an-application-load-balancer.md)
+ [Deploy and manage a serverless data lake on the AWS Cloud by using infrastructure as code](deploy-and-manage-a-serverless-data-lake-on-the-aws-cloud-by-using-infrastructure-as-code.md)
+ [Deploy a RAG use case on AWS by using Terraform and Amazon Bedrock](deploy-rag-use-case-on-aws.md)
+ [Develop a fully automated chat-based assistant by using Amazon Bedrock agents and knowledge bases](develop-a-fully-automated-chat-based-assistant-by-using-amazon-bedrock-agents-and-knowledge-bases.md)
+ [Develop advanced generative AI chat-based assistants by using RAG and ReAct prompting](develop-advanced-generative-ai-chat-based-assistants-by-using-rag-and-react-prompting.md)
+ [Dynamically generate an IAM policy with IAM Access Analyzer by using Step Functions](dynamically-generate-an-iam-policy-with-iam-access-analyzer-by-using-step-functions.md)
+ [Embed Amazon Quick Sight visual components into web applications by using Amazon Cognito and IaC automation](embed-quick-sight-visual-components-into-web-apps-cognito-iac.md)
+ [Ensure Amazon EMR logging to Amazon S3 is enabled at launch](ensure-amazon-emr-logging-to-amazon-s3-is-enabled-at-launch.md)
+ [Estimate the cost of a DynamoDB table for on-demand capacity](estimate-the-cost-of-a-dynamodb-table-for-on-demand-capacity.md)
+ [Generate personalized and re-ranked recommendations using Amazon Personalize](generate-personalized-and-re-ranked-recommendations-using-amazon-personalize.md)
+ [Generate test data using an AWS Glue job and Python](generate-test-data-using-an-aws-glue-job-and-python.md)
+ [Implement SHA1 hashing for PII data when migrating from SQL Server to PostgreSQL](implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.md)
+ [Implement the serverless saga pattern by using AWS Step Functions](implement-the-serverless-saga-pattern-by-using-aws-step-functions.md)
+ [Improve operational performance by enabling Amazon DevOps Guru across multiple AWS Regions, accounts, and OUs with the AWS CDK](improve-operational-performance-by-enabling-amazon-devops-guru-across-multiple-aws-regions-accounts-and-ous-with-the-aws-cdk.md)
+ [Launch a CodeBuild project across AWS accounts using Step Functions and a Lambda proxy function](launch-a-codebuild-project-across-aws-accounts-using-step-functions-and-a-lambda-proxy-function.md)
+ [Migrate Apache Cassandra workloads to Amazon Keyspaces by using AWS Glue](migrate-apache-cassandra-workloads-to-amazon-keyspaces-by-using-aws-glue.md)
+ [Monitor use of a shared Amazon Machine Image across multiple AWS accounts](monitor-use-of-a-shared-amazon-machine-image-across-multiple-aws-accounts.md)
+ [Optimize multi-account serverless deployments by using the AWS CDK and GitHub Actions workflows](optimize-multi-account-serverless-deployments.md)
+ [Orchestrate an ETL pipeline with validation, transformation, and partitioning using AWS Step Functions](orchestrate-an-etl-pipeline-with-validation-transformation-and-partitioning-using-aws-step-functions.md)
+ [Query Amazon DynamoDB tables with SQL by using Amazon Athena](query-amazon-dynamodb-tables-sql-amazon-athena.md)
+ [Send custom attributes to Amazon Cognito and inject them into tokens](send-custom-attributes-cognito.md)
+ [Serve static content in an Amazon S3 bucket through a VPC by using Amazon CloudFront](serve-static-content-in-an-amazon-s3-bucket-through-a-vpc-by-using-amazon-cloudfront.md)
+ [Streamline Amazon Lex bot development and deployment by using an automated workflow](streamline-amazon-lex-bot-development-and-deployment-using-an-automated-workflow.md)
+ [Structure a Python project in hexagonal architecture using AWS Lambda](structure-a-python-project-in-hexagonal-architecture-using-aws-lambda.md)
+ [Translate natural language into query DSL for OpenSearch and Elasticsearch queries](translate-natural-language-query-dsl-opensearch-elasticsearch.md)
+ [Unload data from an Amazon Redshift cluster across accounts to Amazon S3](unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.md)
+ [Coordinate resource dependency and task execution by using the AWS Fargate WaitCondition hook construct](use-the-aws-fargate-waitcondition-hook-construct.md)
+ [Use Amazon Bedrock agents to automate creation of access entry controls in Amazon EKS through text-based prompts](using-amazon-bedrock-agents-to-automate-creation-of-access-entry-controls-in-amazon-eks.md)