Amazon Simple Queue Service Construct Library

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

Installation

Import to your project:

import aws_cdk.aws_sqs as sqs

Basic usage

Here’s how to add a basic queue to your application:

sqs.Queue(self, "Queue")

Encryption

By default queues are encrypted using SSE-SQS. If you want to change the encryption mode, set the encryption property. The following encryption modes are supported:

  • KMS key that SQS manages for you

  • KMS key that you can managed yourself

  • Server-side encryption managed by SQS (SSE-SQS)

  • Unencrypted

To learn more about SSE-SQS on Amazon SQS, please visit the Amazon SQS documentation.

# Use managed key
sqs.Queue(self, "Queue",
    encryption=sqs.QueueEncryption.KMS_MANAGED
)

# Use custom key
my_key = kms.Key(self, "Key")

sqs.Queue(self, "Queue",
    encryption=sqs.QueueEncryption.KMS,
    encryption_master_key=my_key
)

# Use SQS managed server side encryption (SSE-SQS)
sqs.Queue(self, "Queue",
    encryption=sqs.QueueEncryption.SQS_MANAGED
)

# Unencrypted queue
sqs.Queue(self, "Queue",
    encryption=sqs.QueueEncryption.UNENCRYPTED
)

Encryption in transit

If you want to enforce encryption of data in transit, set the enforceSSL property to true. A resource policy statement that allows only encrypted connections over HTTPS (TLS) will be added to the queue.

sqs.Queue(self, "Queue",
    enforce_ssl=True
)

First-In-First-Out (FIFO) queues

FIFO queues give guarantees on the order in which messages are dequeued, and have additional features in order to help guarantee exactly-once processing. For more information, see the SQS manual. Note that FIFO queues are not available in all AWS regions.

A queue can be made a FIFO queue by either setting fifo: true, giving it a name which ends in ".fifo", or by enabling a FIFO specific feature such as: content-based deduplication, deduplication scope or fifo throughput limit.

Dead letter source queues permission

You can configure the permission settings for queues that can designate the created queue as their dead-letter queue using the redriveAllowPolicy attribute.

By default, all queues within the same account and region are permitted as source queues.

# source_queue: sqs.IQueue


# Only the sourceQueue can specify this queue as the dead-letter queue.
queue1 = sqs.Queue(self, "Queue2",
    redrive_allow_policy=sqs.RedriveAllowPolicy(
        source_queues=[source_queue]
    )
)

# No source queues can specify this queue as the dead-letter queue.
queue2 = sqs.Queue(self, "Queue",
    redrive_allow_policy=sqs.RedriveAllowPolicy(
        redrive_permission=sqs.RedrivePermission.DENY_ALL
    )
)

Monitoring

SQS metrics are available as metric* methods on a queue; metric() returns any metric by name:

queue = sqs.Queue(self, "Queue")

queue.metric_approximate_age_of_oldest_message().create_alarm(self, "MessagesTooOld",
    threshold=Duration.minutes(15).to_seconds(),
    evaluation_periods=3
)

metricApproximateNumberOfMessagesOutstanding() returns ApproximateNumberOfMessagesVisible + ApproximateNumberOfMessagesNotVisible as a metric math expression: messages waiting to be picked up, plus messages received but not yet deleted.

Autoscaling consumers on queue depth

Scaling a worker fleet on queue depth needs a different metric in each direction:

  • Scale out on ApproximateNumberOfMessagesVisible — work nobody has started yet. An in-flight message is already owned by a consumer, so adding capacity for it produces an idle consumer.

  • Scale in on metricApproximateNumberOfMessagesOutstanding() — everything still owed. Receiving a message moves it from Visible to NotVisible, so a policy watching Visible alone cannot tell a consumer that just picked up work from one that finished it, and can terminate a consumer mid-message. The message reappears only after its visibility timeout, so the longer consumers hold messages, the longer that work stalls.

That means two one-sided policies; the change: 0 step keeps each from acting in the other direction:

# service: ecs.FargateService

queue = sqs.Queue(self, "Queue")

task_count = service.auto_scale_task_count(min_capacity=1, max_capacity=10)

task_count.scale_on_metric("ScaleOutOnWaitingWork",
    metric=queue.metric_approximate_number_of_messages_visible(period=Duration.minutes(1)),
    scaling_steps=[appscaling.ScalingInterval(upper=30, change=0), appscaling.ScalingInterval(lower=30, change=+1)
    ],
    adjustment_type=appscaling.AdjustmentType.CHANGE_IN_CAPACITY
)

# Remove a task only when nothing is outstanding, so it cannot be holding a message.
task_count.scale_on_metric("ScaleInOnOutstandingWork",
    metric=queue.metric_approximate_number_of_messages_outstanding(period=Duration.minutes(1)),
    scaling_steps=[appscaling.ScalingInterval(upper=0, change=-1), appscaling.ScalingInterval(lower=0, change=0)
    ],
    adjustment_type=appscaling.AdjustmentType.CHANGE_IN_CAPACITY
)

Caveats:

  • Target tracking rejects this metric. It accepts only direct metrics, so scaleToTrackCustomMetric() throws Only direct metrics are supported for Target Tracking (aws-cdk#20659).

  • ApproximateNumberOfMessagesNotVisible can briefly report non-zero on an empty queue if an SQS storage server is unavailable. Evaluate several consecutive datapoints, especially when scaling in to zero.

  • Neither term counts delayed messages. Add metricApproximateNumberOfMessagesDelayed() if you use delay queues or DelaySeconds.