Use Lambda recursive loop detection to prevent infinite loops

When you configure a Lambda function to output to the same service or resource that invokes the function, it's possible to create an infinite recursive loop. For example, a Lambda function might write a message to an Amazon Simple Queue Service (Amazon SQS) queue, which then invokes the same function. This invocation causes the function to write another message to the queue, which in turn invokes the function again.

Unintentional recursive loops can result in unexpected charges being billed to your AWS account. Loops can also cause Lambda to scale and use all of your account's available concurrency. To help reduce the impact of unintentional loops, Lambda detects certain types of recursive loops shortly after they occur. By default, when Lambda detects a recursive loop, it stops your function being invoked and notifies you. If your design intentionally uses recursive patterns, you can a change a function's default configuration to allow it to be invoked recursively. See Allowing a Lambda function to run in a recursive loop for more information.

Understanding recursive loop detection

Recursive loop detection in Lambda works by tracking events. Lambda is an event-driven compute service that runs your function code when certain events occur. For example, when an item is added to an Amazon SQS queue or Amazon Simple Notification Service (Amazon SNS) topic. Lambda passes events to your function as JSON objects, which contain information about the change in the system state. When an event causes your function to run, this is called an invocation.

To detect recursive loops, Lambda uses AWS X-Ray tracing headers. When AWS services that support recursive loop detection send events to Lambda, those events are automatically annotated with metadata. When your Lambda function writes one of these events to another supported AWS service using a supported version of an AWS SDK, it updates this metadata. The updated metadata includes a count of the number of times that the event has invoked the function.

A chain of requests is a sequence of Lambda invocations caused by the same triggering event. For example, imagine that an Amazon SQS queue invokes your Lambda function. Your Lambda function then sends the processed event back to the same Amazon SQS queue, which invokes your function again. In this example, each invocation of your function falls in the same chain of requests.

If your function is invoked approximately 16 times in the same chain of requests, then Lambda automatically stops the next function invocation in that request chain and notifies you. If your function is configured with multiple triggers, then invocations from other triggers aren't affected.

If you have an on-failure destination or dead-letter queue configured for your function, then Lambda also sends the event from the stopped invocation to your destination or dead-letter queue. When configuring a destination or dead-letter queue for your function, be sure not to use an Amazon SNS topic or Amazon SQS queue that your function also uses as an event trigger or event source mapping. If you send events to the same resource that invokes your function, then you can create another recursive loop.

Supported AWS services and SDKs

Lambda can detect only recursive loops that include certain supported AWS services. For recursive loops to be detected, your function must also use one of the supported AWS SDKs.

Supported AWS services

Lambda currently detects recursive loops between your functions, Amazon SQS, Amazon S3, and Amazon SNS. Lambda also detects loops comprised only of Lambda functions, which may invoke each other synchronously or asynchronously. The following diagrams show some examples of loops that Lambda can detect:

When another AWS service such as Amazon DynamoDB forms part of the loop, Lambda can't currently detect and stop it.

Because Lambda currently detects only recursive loops involving Amazon SQS, Amazon S3, and Amazon SNS, it's still possible that loops involving other AWS services can result in unintended usage of your Lambda functions.

To guard against unexpected charges being billed to your AWS account, we recommend that you configure Amazon CloudWatch alarms to alert you to unusual usage patterns. For example, you can configure CloudWatch to notify you about spikes in Lambda function concurrency or invocations. You can also configure a billing alarm to notify you when spending in your account exceeds a threshold that you specify. Or, you can use AWS Cost Anomaly Detection to alert you to unusual billing patterns.

Supported AWS SDKs

For Lambda to detect recursive loops, your function must use one of the following SDK versions or higher:

Some Lambda runtimes such as Python and Node.js include a version of the AWS SDK. If the SDK version included in your function's runtime is lower than the minimum required, then you can add a supported version of the SDK to your function's deployment package. You can also add a supported SDK version to your function using a Lambda layer. For a list of the SDKs included with each Lambda runtime, see Lambda runtimes.

Recursive loop notifications

When Lambda stops a recursive loop, you receive notifications through the AWS Health Dashboard and through email. You can also use CloudWatch metrics to monitor the number of recursive invocations that Lambda has stopped.

AWS Health Dashboard notifications

When Lambda stops a recursive invocation, the AWS Health Dashboard displays a notification on the Your account health page, under Open and recent issues. Note that it can take up to three hours after Lambda stops a recursive invocation before this notification is displayed. For more information about viewing account events in the AWS Health Dashboard, see Getting started with your AWS Health Dashboard – Your account health in the AWS Health User Guide.

Email alerts

When Lambda first stops a recursive invocation of your function, it sends you an email alert. Lambda sends a maximum of one email every 24 hours for each function in your AWS account. After Lambda sends an email notification, you won't receive any more emails for that function for another 24 hours, even if Lambda stops further recursive invocations of the function. Note that it can take up to three hours after Lambda stops a recursive invocation before you receive this email alert.

Lambda sends recursive loop email alerts to your AWS account's primary account contact and alternate operations contact. For information about viewing or updating the email addresses in your account, see Updating contact information in the AWS General Reference.

Amazon CloudWatch metrics

The CloudWatch metric RecursiveInvocationsDropped records the number of function invocations that Lambda has stopped because your function has been invoked more than approximately 16 times in a single chain of requests. Lambda emits this metric as soon as it stops a recursive invocation. To view this metric, follow the instructions for Viewing metrics on the CloudWatch console and choose the metric RecursiveInvocationsDropped.

Responding to recursive loop detection notifications

When your function is invoked more than approximately 16 times by the same triggering event, Lambda stops the next function invocation for that event to break the recursive loop. To prevent a reoccurrence of a recursive loop that Lambda has broken, do the following:

Additionally, if the event source for your Lambda function is an Amazon SQS queue, then consider configuring a dead-letter queue on the source queue.

If the event source is an Amazon SNS topic, then consider adding an on-failure destination for your function.

Allowing a Lambda function to run in a recursive loop

If your design intentionally uses a recursive loop, you can configure a Lambda function to allow it to be invoked recursively. We recommend that you avoid using recursive loops in your design. Implementation errors can lead to recursive invocations using all of your AWS account's available concurrency and to unexpected charges being billed to your account.

You can configure functions to allow recursive loops using the Lambda console, the AWS Command Line Interface (AWS CLI), and the PutFunctionRecursionConfig API. You can also configure a function's recursive loop detection setting in AWS SAM and AWS CloudFormation.

By default, Lambda detects and terminates recursive loops. Unless your design intentionally uses a recursive loop, we recommend that you don't change your functions' default configuration.

Note that when you configure a function to allow recursive loops, the CloudWatch metric RecursiveInvocationsDropped isn't emitted.

Console

To allow a function to run in a recursive loop (console)

1. Open the Functions page of the Lambda console.

2. Choose the name of your function to open the function detail page.

3. Choose the Configuration tab, then choose Concurrency and recursion detection.

4. Beside Recursive loop detection, choose Edit.

5. Select Allow recursive loops.

6. Choose Save.

AWS CLI

You can use the PutFunctionRecursionConfig API to allow your function to be invoked in a recursive loop. Specify Allow for the recursive loop parameter. For example, you can call this API with the put-function-recursion-config AWS CLI command:

aws lambda put-function-recursion-config --function-name yourFunctionName --recursive-loop Allow

You can change your function's configuration back to the default setting so that Lambda terminates recursive loops when it detects them. Edit your function's configuration using the Lambda console or the AWS CLI.

Console

To configure a function so that recursive loops are terminated (console)

1. Open the Functions page of the Lambda console.

2. Choose the name of your function to open the function detail page.

3. Choose the Configuration tab, then choose Concurrency and recursion detection.

4. Beside Recursive loop detection, choose Edit.

5. Select Terminate recursive loops.

6. Choose Save.

AWS CLI

You can use the PutFunctionRecursionConfig API to configure your function so that Lambda terminates recursive loops when it detects them. Specify Terminate for the recursive loop parameter. For example, you can call this API with the put-function-recursion-config AWS CLI command:

aws lambda put-function-recursion-config --function-name yourFunctionName --recursive-loop Terminate

Supported regions for Lambda recursive loop detection

Lambda recursive loop detection is supported in the following AWS Regions.