SQS Delay and Visibility Timeout: When Your Data Needs a Coffee Break

Have you ever wondered how to make your message appear late? Yes, late and invisible. Welcome to the Amazon SQS (Simple Queue Service) delayed queues! In the fast-paced world of cloud computing, timing is important, and even a minor delay can mean the difference. SQS delay queues allow you to postpone message delivery by up to 15 minutes. Understanding and using SQS delays can increase the productivity and flexibility of your applications, whether you are dealing with complex workflows or simply giving your processing systems a break. So, grab a cup of coffee, sit back, and let’s look into the interesting topic of SQS message delays and visibility timeout.

SQS Delay Queues

Delay queues allow you to postpone the transmission of new messages to consumers for a certain number of seconds, such as when your consumer application requires more time to process messages. If you build a delay queue, any messages you submit to it are invisible to consumers throughout the wait time. The default (minimum) delay for a queue is 0 seconds and the limit is 15 minutes.

SQS Visibility Timeout

Visibility timeout, refers to the time when you receive the message and the time begins to count. If you reach the end of time, the message becomes accessible for receiving again. Visibility timeout stops all other consumers from receiving and processing the message for a certain amount of time. A message’s visible timeout defaults to 30 seconds, minimum to 0 sec and maximum to 12 hours.

Let’s understand it with an example:

SQS delay : The delay is the amount of time the message will wait before becoming available for the first time. For example, you send a message and need two minutes to ensure that data is present in the database which is inserted by another process parallelly. In this case you wait at sqs for 2min or the time which your service takes to complete the insert process.

Visibility timeout : Let’s continue with the above delay example, now once the delay time is completed and message is available to process, one of the consumers will get the message and start processing, and you set visibility timeout at 30s as your service will take 30s to process the message. So, for 30s no other consumer will be able to process the message. Now if your service processes the message successfully then it will by-default delete the message from the queue, but for an unusual case you experience an unhandled exception and you do not process the message in this time frame. After a minute the message will be available again to retry another time.

Some use cases where you can use these features.

SQS delay

Rate Limiting : If you want to control the rate of message processing to avoid overloading the system, SQS delay queues can help. By delaying messages, you can smooth out traffic disruptions and ensure that your processing system handles the load more effectively.

Workflow Sync : In complex workflows where certain tasks need to be delayed until a specific condition is met or a certain time has passed, SQS delay queues can be essential. For example, you might need to wait for a resource to be available or a specific event to occur before processing the next step in the workflow

Retry Logic : Delaying before retrying a failed operation can prevent failed services from being interrupted by repeated operations. SQS delay queues allow you to have a retry window, which gives you time to resolve temporary issues before trying again.

Visibility timeout:

Maintaining Order in FIFO Queues : For First-In-First-Out (FIFO) queues, visibility timeout ensures that messages are processed in the correct order.Maintains sequence control by preventing messages from being displayed before all previous messages have been processed.

Handling Long-Running Tasks : For tasks that take a significant amount of time to process, visibility timeout allows you to set a duration that matches the expected processing time. This ensures the task is not reassigned to another consumer before it is completed. If the processing time exceeds the initial timeout, the consumer can extend the visibility timeout as needed.

Deduplication : Visibility timeout ensures that a message is not processed by multiple consumers simultaneously. If a consumer retrieves a message, it becomes invisible to other consumers for a specified period, giving the original consumer time to process it completely. This prevents duplicate processing and ensures data consistency.

Conclusion

Amazon SQS delay queues are a versatile tool that can enhance the efficiency, reliability, and flexibility of your application by allowing you to introduce controlled delays in message processing. By leveraging this feature, you can optimise workflows, manage dependencies, and handle a variety of use cases that require timed delays. On the other hand, Visibility timeout is a powerful feature of Amazon SQS that ensures reliable and efficient message processing by preventing duplicate processing, managing long-running tasks, and enabling robust error handling.