Chain AWS services together using a serverless approach - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpics

Chain AWS services together using a serverless approach

Created by Aniket Braganza (AWS)

Environment: Production

Technologies: Serverless; CloudNative; DevelopmentAndTesting; DevOps; Modernization; Infrastructure

AWS services: Amazon S3; Amazon SNS; Amazon SQS; AWS Lambda

Summary

This pattern demonstrates a scalable, serverless approach for processing an uploaded file by chaining together Amazon Simple Storage Service (Amazon S3), Amazon Simple Notification Service (Amazon SNS), Amazon Simple Queue Service (Amazon SQS), and AWS Lambda. The uploaded file example is for demonstration purposes. You can use a serverless approach to complete other tasks by chaining together the combination of AWS services that are necessary to meet your business goals. The serverless approach employs an asynchronous workflow that relies on event-driven notifications, resilient storage, and function as a service (FaaS) computing to process requests. You can use the serverless approach to scale to meet demand while minimizing costs.

Note: There are several options for chaining AWS services together through a serverless approach. For example, you can use an approach that combines Lambda with Amazon S3 instead of Amazon SNS and Amazon SQS. However, this pattern uses Amazon SNS and Amazon SQS because this approach makes it possible to add multiple integration points into the Lambda invocation process during an event notification and to extend the implementation to include multiple listeners in a serverless orchestration while minimizing the amount of processing overhead.

Prerequisites and limitations

Prerequisites

Product versions

  • AWS CDK 2.x

  • Python 3.9

Architecture

The following diagram illustrates how chained AWS services can enable a user to upload a file to an S3 bucket for processing.

Workflow for uploading a file to an S3 bucket by using chained AWS services.

The diagram shows the following workflow:

  1. A user uploads a file to the S3 bucket.

  2. The upload initiates an S3 event that publishes a message to an SNS topic. The message contains the details of the S3 event.

  3. The message published to the SNS topic is inserted into an SQS queue, which is subscribed to and receives notifications for that topic.

  4. A Lambda function polls the SQS queue (as its event source) and waits for messages to process.

  5. When the Lambda function receives messages from the SQS queue, it processes them and acknowledges receipt of those messages.

  6. If a message isn’t processed by Lambda, then that message is returned to the SQS queue and is eventually transferred to an SQS dead-letter queue.

Technology stack

  • Amazon S3

  • Amazon SNS

  • Amazon SQS

  • AWS Lambda

Tools

AWS services

  • Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.

  • Amazon Simple Notification Service (Amazon SNS) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.

  • Amazon Simple Queue Service (Amazon SQS) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.

  • AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.

Other tools

  • AWS Cloud Development Kit (AWS CDK) is the primary tool for interacting with your AWS CDK app. It executes your app, interrogates the application model you defined, and produces and deploys the AWS CloudFormation templates generated by the AWS CDK.

  • AWS Command Line Interface (AWS CLI) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.

  • Python is a high-level, interpreted general purpose programming language.

Code

The code for this pattern is available in the GitHub Chaining S3 to SNS to SQS to Lambda repository.

Epics

TaskDescriptionSkills required

Clone the repository.

Clone the repository and navigate to the python/s3-sns-sqs-lambda-chain folder.

App developer

Set up a virtual environment.

  1. In the AWS CDK, run the python3 -m venv .venv command.

  2. Run the source .venv/bin/activate command on MacOS/Linux or .venv\Scripts\activate.bat on Windows.

App developer

Install dependencies.

Run the pip install -r requirements.txt command.

App developer
TaskDescriptionSkills required

Run unit tests.

  1. Run the pip install -r requirements-dev.txt command.

  2. (Optional) Run the cdk synth --no-staging > template.yml command to generate the CloudFormation stack. Important: You can inspect the stack, but avoid generating the staged resources and artifacts.

  3. Run the pytest command to run all unit tests.

  4. (Optional) Run the pytest tests/unit/<test_filename> command to run tests for a specific file.

App developer, Test engineer
TaskDescriptionSkills required

Set up the bootstrap environment.

Follow the instructions in Bootstrapping in the AWS documentation to bootstrap the environment for AWS CDK deployment in each AWS Region where the CloudFormation stack will be deployed.

Note: This step requires that you have credentials with programmatic access.

App developer, DevOps engineer, Data engineer

Deploy the CloudFormation stack.

Run the cdk deploy command to build and deploy the stack to the AWS account.

App developer, DevOps engineer, AWS DevOps
TaskDescriptionSkills required

Delete the CloudFormation stack and remove associated resources.

To delete the CloudFormation stack that was created and remove all associated resources, run the run cdk destroy command.

App developer