Queue-Based Architecture - High Performance Computing Lens

Queue-Based Architecture

Amazon SQS is a fully managed message queuing service that makes it easy to decouple pre-processing steps from compute steps and post-processing steps. Building applications from individual components that perform discrete functions improves scalability and reliability. Decoupling components is a best practice for designing modern applications. Amazon SQS frequently lies at the heart of cloud-native loosely coupled solutions.

Amazon SQS is often orchestrated with AWS CLI or AWS SDK scripted solutions for the deployment of applications from the desktop without users interacting with AWS components directly. A queue-based architecture with Amazon SQS and Amazon EC2 requires self-managed compute infrastructure, in contrast with a service-managed deployment, such as AWS Batch.

A queue-based architecture is best for loosely coupled workloads and can quickly become complex if applied to tightly coupled workloads.

Reference Architecture

AWS Cloud architecture showing user interaction with SDK, SQS, EC2 instances, and S3 storage within a VPC.

Figure 3: Amazon SQS deployed for a loosely coupled workload

Workflow steps:

  1. Multiple users submit jobs with the AWS CLI or SDK.

  2. The jobs are queued as messages in Amazon SQS.

  3. EC2 Instances poll the queue and start processing jobs.

  4. Amazon SQS emits metrics based on the number of messages (jobs) in the queue.

  5. An Amazon CloudWatch alarm is configured to notify Auto Scaling if the queue is longer than a specified length. Auto Scaling increases the number of EC2 instances.

  6. The EC2 instances pull source data and store result data in an S3 bucket.