Chain AWS services together using a serverless approach
Created by Aniket Braganza (AWS)
Environment: Production | Technologies: Serverless; CloudNative; DevelopmentAndTesting; DevOps; Modernization; Infrastructure | AWS services: Amazon S3; Amazon SNS; Amazon SQS; AWS Lambda |
Summary
This pattern demonstrates a scalable, serverless approach for processing an uploaded file by chaining together Amazon Simple Storage Service (Amazon S3), Amazon Simple Notification Service (Amazon SNS), Amazon Simple Queue Service (Amazon SQS), and AWS Lambda. The uploaded file example is for demonstration purposes. You can use a serverless approach to complete other tasks by chaining together the combination of AWS services that are necessary to meet your business goals. The serverless approach employs an asynchronous workflow that relies on event-driven notifications, resilient storage, and function as a service (FaaS) computing to process requests. You can use the serverless approach to scale to meet demand while minimizing costs.
Note: There are several options for chaining AWS services together through a serverless approach. For example, you can use an approach that combines Lambda with Amazon S3 instead of Amazon SNS and Amazon SQS. However, this pattern uses Amazon SNS and Amazon SQS because this approach makes it possible to add multiple integration points into the Lambda invocation process during an event notification and to extend the implementation to include multiple listeners in a serverless orchestration while minimizing the amount of processing overhead.
Prerequisites and limitations
Prerequisites
An active AWS account
Programmatic access to the AWS account. For more information, see:
Prerequisites in the AWS Cloud Development Kit (AWS CDK) documentation
Prerequisites in the AWS Command Line Interface (AWS CLI) documentation
AWS CDK, installed
AWS CLI, installed and configured
Product versions
AWS CDK 2.x
Python 3.9
Architecture
The following diagram illustrates how chained AWS services can enable a user to upload a file to an S3 bucket for processing.
![Workflow for uploading a file to an S3 bucket by using chained AWS services.](/images/prescriptive-guidance/latest/patterns/images/pattern-img/8b298cfc-509a-4502-97b2-f1af09914b99/images/f3ca9202-96eb-40cf-85cd-4bb569fcd083.png)
The diagram shows the following workflow:
A user uploads a file to the S3 bucket.
The upload initiates an S3 event that publishes a message to an SNS topic. The message contains the details of the S3 event.
The message published to the SNS topic is inserted into an SQS queue, which is subscribed to and receives notifications for that topic.
A Lambda function polls the SQS queue (as its event source) and waits for messages to process.
When the Lambda function receives messages from the SQS queue, it processes them and acknowledges receipt of those messages.
If a message isn’t processed by Lambda, then that message is returned to the SQS queue and is eventually transferred to an SQS dead-letter queue.
Technology stack
Amazon S3
Amazon SNS
Amazon SQS
AWS Lambda
Tools
AWS services
Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
Amazon Simple Notification Service (Amazon SNS) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.
Amazon Simple Queue Service (Amazon SQS) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.
AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
Other tools
AWS Cloud Development Kit (AWS CDK) is the primary tool for interacting with your AWS CDK app. It executes your app, interrogates the application model you defined, and produces and deploys the AWS CloudFormation templates generated by the AWS CDK.
AWS Command Line Interface (AWS CLI) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
Python
is a high-level, interpreted general purpose programming language.
Code
The code for this pattern is available in the GitHub Chaining S3 to SNS to SQS to Lambda
Epics
Task | Description | Skills required |
---|---|---|
Clone the repository. | Clone the repository | App developer |
Set up a virtual environment. |
| App developer |
Install dependencies. | Run the | App developer |
Task | Description | Skills required |
---|---|---|
Run unit tests. |
| App developer, Test engineer |
Task | Description | Skills required |
---|---|---|
Set up the bootstrap environment. | Follow the instructions in Bootstrapping in the AWS documentation to bootstrap the environment for AWS CDK deployment in each AWS Region where the CloudFormation stack will be deployed. Note: This step requires that you have credentials with programmatic access. | App developer, DevOps engineer, Data engineer |
Deploy the CloudFormation stack. | Run the | App developer, DevOps engineer, AWS DevOps |
Task | Description | Skills required |
---|---|---|
Delete the CloudFormation stack and remove associated resources. | To delete the CloudFormation stack that was created and remove all associated resources, run the | App developer |