Visualize Lambda function invocations using AWS X-Ray

You can use AWS X-Ray to visualize the components of your application, identify performance bottlenecks, and troubleshoot requests that resulted in an error. Your Lambda functions send trace data to X-Ray, and X-Ray processes the data to generate a service map and searchable trace summaries.

If you've enabled X-Ray tracing in a service that invokes your function, Lambda sends traces to X-Ray automatically. The upstream service, such as Amazon API Gateway, or an application hosted on Amazon EC2 that is instrumented with the X-Ray SDK, samples incoming requests and adds a tracing header that tells Lambda to send traces or not. Traces from upstream message producers, such as Amazon SQS, are automatically linked to traces from downstream Lambda functions, creating an end-to-end view of the entire application. For more information, see Tracing event-driven applications in the AWS X-Ray Developer Guide.

To toggle active tracing on your Lambda function with the console, follow these steps:

Your function needs permission to upload trace data to X-Ray. When you activate tracing in the Lambda console, Lambda adds the required permissions to your function's execution role. Otherwise, add the AWSXRayDaemonWriteAccess policy to the execution role.

X-Ray doesn't trace all requests to your application. X-Ray applies a sampling algorithm to ensure that tracing is efficient, while still providing a representative sample of all requests. The sampling rate is 1 request per second and 5 percent of additional requests. You can't configure the X-Ray sampling rate for your functions.

Understanding X-Ray traces

In X-Ray, a trace records information about a request that is processed by one or more services. Lambda records 2 segments per trace, which creates two nodes on the service graph. The following image highlights these two nodes:

The first node on the left represents the Lambda service, which receives the invocation request. The second node represents your specific Lambda function.

The segment recorded for the Lambda service, AWS::Lambda, covers all the steps required to prepare the Lambda execution environment. This includes scheduling the MicroVM, creating or unfreezing an execution environment with the resources you have configured, as well as downloading your function code and all layers.

The AWS::Lambda::Function segment is for the work done by the function.

Old-style AWS X-Ray Lambda segment structure

The old-style X-Ray structure for the AWS::Lambda segment looks like the following:

In this format, the function segment has subsegments for Initialization, Invocation, and Overhead. For Lambda SnapStart only, there is also a Restore subsegment (not shown on this diagram).

The Initialization subsegment represents the init phase of the Lambda execution environment lifecycle. During this phase, Lambda initializes extensions, initializes the runtime, and runs the function's initialization code.

The Invocation subsegment represents the invoke phase where Lambda invokes the function handler. This begins with runtime and extension registration and it ends when the runtime is ready to send the response.

(Lambda SnapStart only) The Restore subsegment shows the time it takes for Lambda to restore a snapshot, load the runtime (JVM), and run any afterRestore runtime hooks. The process of restoring snapshots can include time spent on activities outside the MicroVM. This time is reported in the Restore subsegment. You aren't charged for the time spent outside the microVM to restore a snapshot.

The Overhead subsegment represents the phase that occurs between the time when the runtime sends the response and the signal for the next invoke. During this time, the runtime finishes all tasks related to an invoke and prepares to freeze the sandbox.

For more information about Lambda execution environment phases, see Understand the Lambda execution environment lifecycle.

An example trace using the old-style X-Ray structure is shown in the following diagram.

Note the two segments in the example. Both are named my-function, but one has an origin of AWS::Lambda and the other has an origin of AWS::Lambda::Function. If the AWS::Lambda segment shows an error, the Lambda service had an issue. If the AWS::Lambda::Function segment shows an error, your function had an issue.

New-style AWS X-Ray Lambda segment structure

The new-style X-Ray structure for the AWS::Lambda segment looks like the following:

In this new format, The Init subsegment represents the init phase of the Lambda execution environment lifecycle as before.

There is no invocation segment in the new format. Instead, customer subsegments are attached directly to the AWS::Lambda::Function segment. This segment contains the following metrics as annotations:

An example trace using the new-style X-Ray structure is shown in the following diagram.

Note the two segments in the example. Both are named my-function, but one has an origin of AWS::Lambda and the other has an origin of AWS::Lambda::Function. If the AWS::Lambda segment shows an error, the Lambda service had an issue. If the AWS::Lambda::Function segment shows an error, your function had an issue.

See the following topics for a language-specific introduction to tracing in Lambda:

For a full list of services that support active instrumentation, see Supported AWS services in the AWS X-Ray Developer Guide.

Execution role permissions

Lambda needs the following permissions to send trace data to X-Ray. Add them to your function's execution role.

These permissions are included in the AWSXRayDaemonWriteAccess managed policy.

The AWS X-Ray daemon

Instead of sending trace data directly to the X-Ray API, the X-Ray SDK uses a daemon process. The AWS X-Ray daemon is an application that runs in the Lambda environment and listens for UDP traffic that contains segments and subsegments. It buffers incoming data and writes it to X-Ray in batches, reducing the processing and memory overhead required to trace invocations.

The Lambda runtime allows the daemon to up to 3 percent of your function's configured memory or 16 MB, whichever is greater. If your function runs out of memory during invocation, the runtime terminates the daemon process first to free up memory.

The daemon process is fully managed by Lambda and cannot be configured by the user. All segments generated by function invocations are recorded in the same account as the Lambda function. The daemon cannot be configured to redirect them to any other account.

For more information, see The X-Ray daemon in the X-Ray Developer Guide.

Enabling active tracing with the Lambda API

To manage tracing configuration with the AWS CLI or AWS SDK, use the following API operations:

The following example AWS CLI command enables active tracing on a function named my-function.

aws lambda update-function-configuration --function-name my-function \
--tracing-config Mode=Active

Tracing mode is part of the version-specific configuration when you publish a version of your function. You can't change the tracing mode on a published version.

Enabling active tracing with AWS CloudFormation

To activate tracing on an AWS::Lambda::Function resource in an AWS CloudFormation template, use the TracingConfig property.

Resources:
  function:
    Type: AWS::Lambda::Function
    Properties:
      TracingConfig:
        Mode: Active
      ...

For an AWS Serverless Application Model (AWS SAM) AWS::Serverless::Function resource, use the Tracing property.

Resources:
  function:
    Type: AWS::Serverless::Function
    Properties:
      Tracing: Active
      ...