Amazon SQS visibility timeout
When you receive a message from an Amazon SQS queue, it remains in the queue but becomes
temporarily invisible to other consumers. This invisibility is controlled by the visibility
timeout, which prevents other consumers from processing the same message while you're
working on it. Amazon SQS doesn't automatically delete messages; you must explicitly delete them
using the DeleteMessage action after processing.
Topics
Visibility timeout use cases
Manage long-running tasks – You can use SQS visibility timeout to handle long-running tasks effectively. Set an appropriate visibility timeout for messages that require extended processing time. This ensures that other consumers don't pick up the same message while it's being processed, preventing duplicate work and maintaining system efficiency.
Implement retry mechanisms – Leverage SQS visibility timeout to create robust retry mechanisms. If a task fails to complete within the initial visibility timeout, you can extend the timeout programmatically. This allows your system to retry processing the message without it becoming visible to other consumers, improving fault tolerance and reliability.
Coordinate distributed systems – Use SQS visibility timeout to coordinate tasks across distributed systems. Set visibility timeouts that align with your expected processing times for different components. This helps maintain consistency and prevents race conditions in complex, distributed architectures.
Optimize resource utilization – Adjust SQS visibility timeouts to optimize resource utilization in your application. By setting appropriate timeouts, you can ensure that messages are processed efficiently without tying up resources unnecessarily. This leads to better overall system performance and cost-effectiveness.
Setting the visibility timeout
The visibility timeout starts as soon as a message is delivered to you. During this period, you're expected to process and delete the message. If you don't delete it before the timeout expires, the message becomes visible again in the queue and can be retrieved by another consumer. The default visibility timeout for a queue is 30 seconds, but you can adjust this to match the time your application needs to process and delete a message. You can also set a specific visibility timeout for individual messages without changing the queue's overall setting.
In flight messages and quotas
In Amazon SQS, in-flight messages are those that have been received by a consumer but not
yet deleted. For standard queues, there's a limit of approximately 120,000 in-flight
messages, depending on queue traffic and message backlog. If you reach this limit, Amazon SQS
returns an OverLimit error, indicating that no additional messages can be
received until some in-flight messages are deleted.
-
When using short polling – If this limit is reached while using short polling, Amazon SQS will return an
OverLimiterror, indicating that no additional messages can be received until some in-flight messages are deleted. -
When using long polling – If you are using long polling, Amazon SQS does not return an error when the in-flight message limit is reached. Instead, it will not return any new messages until the number of in-flight messages drops below the limit.
To manage in-flight messages effectively:
-
Prompt deletion – Delete messages as soon as you've successfully processed them to reduce the in-flight count.
-
Monitor with CloudWatch – Use CloudWatch to monitor the number of in-flight messages and set alarms to alert you when approaching the limit.
-
Distribute load – If you're processing a high volume of messages, consider increasing the number of queues to distribute the load and prevent bottlenecks.
-
Request a quota increase – If necessary, submit a support request to AWS to increase the in-flight message quota for your application.
Understanding visibility timeout in standard and FIFO queues
In both standard and FIFO (First-In-First-Out) queues, the visibility timeout helps prevent multiple consumers from processing the same message simultaneously. However, due to the at-least-once delivery model of Amazon SQS, there's no absolute guarantee that a message won't be delivered more than once during the visibility timeout period.
-
Standard queues – The visibility timeout in standard queues prevents multiple consumers from processing the same message at the same time. However, because of the at-least-once delivery model, Amazon SQS doesn't guarantee that a message won’t be delivered more than once within the visibility timeout period.
-
FIFO queues – For FIFO queues, messages with the same message group ID are processed in a strict sequence. When a message with a message group ID is in-flight, subsequent messages in that group are not made available until the in-flight message is either deleted or the visibility timeout expires. However, this doesn’t "lock" the group indefinitely–each message is processed in sequence, and only when each message is deleted or becomes visible again will the next message in that group be available to consumers. This approach ensures ordered processing within the group without unnecessarily locking the group from delivering messages.
What happens if you fail to process a message
If you don't process and delete a message before the visibility timeout expires—due to application errors, crashes, or connectivity problems—the message becomes visible again in the queue. It can then be retrieved by the same or a different consumer for another processing attempt. This ensures that messages aren't lost even if the initial processing fails. However, setting the visibility timeout too high can delay the reappearance of unprocessed messages, potentially slowing down retries. It's crucial to set an appropriate visibility timeout based on the expected processing time for timely message handling.
Changing and terminating visibility timeout
You can change or terminate the visibility timeout using the
ChangeMessageVisibility action:
-
Changing the timeout – If the initial visibility timeout is insufficient, adjust it using the
ChangeMessageVisibilityaction. This allows you to either shorten or extend the timeout based on your processing needs. -
Terminating the timeout – If you decide not to process a received message, terminate its visibility timeout by setting the
VisibilityTimeoutto 0 seconds through theChangeMessageVisibilityaction. This immediately makes the message available for other consumers to process.
Best practices
Use the following best practices for managing visibility timeouts in Amazon SQS, including setting, adjusting, and extending timeouts, as well as handling unprocessed messages using Dead-Letter Queues (DLQs).
-
Setting and adjusting the timeout. Start by setting the visibility timeout to match the maximum time your application typically needs to process and delete a message. If you're unsure about the exact processing time, begin with a shorter timeout (for example, 2 minutes) and extend it as necessary. Implement a heartbeat mechanism to periodically extend the visibility timeout, ensuring the message remains invisible until processing is complete. This minimizes delays in reprocessing unhandled messages and prevents premature visibility.
-
Extending the timeout and handling the 12-Hour limit. If your processing time varies or may exceed the initially set timeout, use the
ChangeMessageVisibilityaction to extend the visibility timeout while processing the message. Keep in mind that the visibility timeout has a maximum limit of 12 hours from when the message is first received. Extending the timeout doesn't reset this 12-hour limit. If your processing requires more time than this limit, consider using AWS Step Functions or breaking the task into smaller steps. -
Handling unprocessed messages. To manage messages that fail multiple processing attempts, configure a Dead-Letter Queue (DLQ). This ensures that messages that can't be processed after several retries are captured separately for further analysis or handling, preventing them from repeatedly circulating in the main queue.
