# Choosing an interface
AWS X-Ray can provide insights into how your application works and how well it interacts with other services and resources. After you instrument or configure your application, X-Ray collects trace data as your application serves requests. You can analyze this trace data to identify performance issues, troubleshoot errors, and optimization your resources. This guide shows you how to interact with X-Ray with the following guidelines:
+ Use an AWS Management Console if you want to get started quickly or can use pre-built visualizations to perform basic tasks.
+ Choose the Amazon CloudWatch console for the most updated user experience that contains all of the X-Ray console’s functionality.
+ Use the X-Ray console if you want a simpler interface or don’t want to change how you interact with X-Ray.
+ Use an SDK if you need more custom tracing, monitoring or logging capabilities than an AWS Management Console can provide.
+ Choose the ADOT SDK if you want a vendor-agnostic SDK based on the open source OpenTelemetry SDK with added layers of AWS security and optimization.
+ Choose the X-Ray SDK if you want a simpler SDK or don’t want to update your application code.
+ Use X-Ray API operations if an SDK does not support your application’s programming language.
The following diagram helps you choose how to interact with X-Ray:
![\[X-Ray displays detailed information about application requests.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/xray-choose-interface.png)
**Topics**
+ [Use an SDK](aws-xray-interface-sdk.md)
+ [Use a console](aws-xray-interface-console.md)
+ [Use the X-Ray API](xray-api.md)
# Use an SDK
**Note**
X-Ray SDK/Daemon Maintenance Notice – On February 25th, 2026, the AWS X-Ray SDKs/Daemon will enter maintenance mode, where AWS will limit X-Ray SDK and Daemon releases to address security issues only. For more information on the support timeline, see [X-Ray SDK and Daemon Support timeline](xray-sdk-daemon-timeline.md). We recommend to migrate to OpenTelemetry. For more information on migrating to OpenTelemetry, see [Migrating from X-Ray instrumentation to OpenTelemetry instrumentation ](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-migration.html).
Use an SDK if you want to use a command line interface or need more custom tracing, monitoring, or logging capabilities than what is available in an AWS Management Console. You can also use an AWS SDK to develop programs that use the X-Ray APIs. You can use either the AWS Distro for OpenTelemetry (ADOT) SDK or the X-Ray SDK.
If you use an SDK, you can add customizations to your workflow both when you instrument your application and when you configure your collector or agent. You can use an SDK to do the following tasks that you can’t do using an AWS Management Console:
+ Publish custom metrics – Sample metrics at high resolutions down to 1 second, use multiple dimensions to add information about a metric, and aggregate data points into a statistic set.
+ Customize your collector – Customize the configuration for any portion of a collector including the receiver, processor, exporter, and connector.
+ Customize your instrumentation – Customize segments and subsegments, add custom key-value pairs as attributes, and create custom metrics.
+ Create and update sampling rules programmatically.
Use the ADOT SDK if you want the flexibility of using a standardized OpenTelemetry SDK with added layers of AWS security and optimization. The AWS Distro for OpenTelemetry (ADOT) SDK is a vendor-agnostic package that allows for integration with back ends from other vendors and non-AWS services without having to reinstrument your code.
Use the X-Ray SDK if you are already using the X-Ray SDK, only integrate with AWS backends, and don’t want to change the way you interact with X-Ray or your application code.
For more information about each feature, see [Choosing between the AWS Distro for OpenTelemetry and X-Ray SDKs](xray-instrumenting-your-app.md#xray-instrumenting-choosing).
## Use the ADOT SDK
The ADOT SDK is a set of open source APIs, libraries and agents that send data to backend services. ADOT is supported by AWS, integrates with multiple backends and agents, and provides a large number of open source libraries maintained by the OpenTelemetry community. Use the ADOT SDK to instrument your application and collect logs, metadata, metrics and traces. You can also use ADOT to monitor services and set an alarm based on your metrics in CloudWatch.
If you are using the ADOT SDK, you have the following options, in combination with an agent:
+ Use the ADOT SDK with the [CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) – recommended.
+ Use the ADOT SDK with the [ADOT Collector](https://aws-otel.github.io/docs/getting-started/collector) – recommended if you want to use vendor agnostic software with AWS layers of security and optimization.
To use the ADOT SDK, do the following:
+ Instrument your application using the ADOT SDK. For more information, see the documentation for your programming language in the [ADOT technical documentation](https://aws-otel.github.io/docs/introduction).
+ Configure an ADOT collector to tell it where to send data that it collects.
After the ADOT collector receives your data, it sends it to the backend that you specify in the ADOT configuration. ADOT can send data to multiple backends, including to vendors outside of AWS, as shown in the following diagram:
![\[You can customize the ADOT Collector when you instrument your application and configure the collector.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/adot-sdk.png)
AWS regularly updates ADOT to add functionality and align with the [OpenTelemetry](https://opentelemetry.io/docs/) framework. Updates and future plans for developing ADOT are part of a [roadmap](https://github.com/orgs/aws-observability/projects/4) that is available to the public. ADOT supports several programming languages which include the following:
+ Go
+ Java
+ JavaScript
+ Python
+ .NET
+ Ruby
+ PHP
If you are using Python, ADOT can automatically instrument your application. To get started using ADOT, see [Introduction](https://aws-otel.github.io/docs/introduction) and [Getting Started with the AWS Distro for OpenTelemetry Collector](https://aws-otel.github.io/docs/getting-started/collector).
## Use the X-Ray SDK
The X-Ray SDK is a set of AWS APIs and libraries that send data to AWS backend services. Use the X-Ray SDK to instrument your application and collect trace data. You cannot use the X-Ray SDK to collect log or metric data.
If you are using the X-Ray SDK, you have the following options, in combination with an agent:
+ Use the X-Ray SDK with the [AWS X-Ray daemon](xray-daemon.md) – Use this if you don't want to update your application code.
+ Use the X-Ray SDK with the CloudWatch agent – (recommended) The CloudWatch agent is compatible with the X-Ray SDK.
To use the X-Ray SDK, do the following:
+ Instrument your application using the X-Ray SDK.
+ Configure a collector to tell it where to send data that it collects. You can use either the CloudWatch agent or the X-Ray daemon to collect your trace information.
After the collector or agent receives your data, it sends it to an AWS backend that you specify in the agent configuration. The X-Ray SDK can only send data to an AWS backend as shown in the following diagram:
![\[Use the X-Ray SDK with either the CloudWatch agent or the X-Ray daemon.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/xray-sdk.png)
If you are using Java, you can use the X-Ray SDK to automatically instrument your application. To get started using the X-Ray SDK, see the libraries associated with the following programming languages:
+ [Go](xray-go.md)
+ [Java](xray-java.md)
+ [Node.js](xray-nodejs.md)
+ [Python](xray-python.md)
+ [.NET](xray-dotnet.md)
+ [Ruby](xray-ruby.md)
# Use a console
Use a console if you want a graphical user interface (GUI) that requires minimal coding. Users that are new to X-Ray can get started quickly using pre-built visualizations, and performing basic tasks. You can do the following directly from the console:
+ Enable X-Ray.
+ View high-level summaries of your application's performance.
+ Check the health status of your applications.
+ Identify high-level errors.
+ View basic trace summaries.
You can use either the Amazon CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/) or the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home) to interact with X-Ray.
## Use the Amazon CloudWatch console
The CloudWatch console includes new X-Ray functionality that is redesigned from the X-Ray console to make it easier to use. If you use the CloudWatch console, you can view CloudWatch logs and metrics along with X-Ray trace data. Use the CloudWatch console to view and analyze data including the following:
+ X-Ray traces – View, analyze and filter traces associated with your application as it serves a request. Use these traces to find high latencies, debug errors, and optimize your application workflow. View a trace map and service map to see visual representations of your application workflow.
+ Logs – View, analyze and filter logs that your application produces. Use logs to troubleshoot errors and set up monitoring based on specific log values.
+ Metrics – Measure and monitor your application performance using metrics that your resources emit or create your own metrics. View these metrics in graphs and charts.
+ Monitoring networks and infrastructure – Monitor major networks for outages and the health and performance of your infrastructure including containerized applications, other AWS services, and clients.
+ All of the functionality from the X-Ray console listed in the following **Use the X-Ray console** section.
For more information about the CloudWatch console, see [Getting started with Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/GettingStarted.html).
Login the Amazon CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
## Use the X-Ray console
The X-Ray console offers distributed tracing for application requests. Use the X-Ray console if you want a simpler console experience or don’t want to update your application code. AWS is no longer developing the X-Ray console. The X-Ray console contains the following features for instrumented applications:
+ [Insights](xray-console-insights.md) – Automatically detect anomalies in your application’s performance and find the underlying causes. Insights are included in the CloudWatch console under **Insights**. For more information, see the **Use X-Ray Insights ** in [Use the X-Ray console](#xray-console).
+ Service map – View a graphical structure of your application and its connections with clients, resources, services, and dependencies.
+ Traces – See an overview of traces that are generated by your application as it serves a request. Use trace data to understand how your application performs against basic metrics including HTTP response and response time.
+ Analytics – Interpret, explore and analyze trace data using graphs for response time distribution.
+ Configuration – Create customized traces to change the default configurations for the following:
+ Sampling – Create a rule that defines how often to sample your application for trace information. For more information, see ** Configure sampling rules** in [Use the X-Ray console](#xray-console) .
+ [Encryption](xray-console-encryption.md) – Encrypt data at rest using a key that you can audit or disable using AWS Key Management Service.
+ Groups – Use a filter expression to define a group of traces with a common feature such as the name of a url or a response time. For more information, see [Configure groups](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray-interface-console.html#xray-console-groups).
Login the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home).
## Explore the X-Ray console
Use the X-Ray console to view a map of services and associated traces for requests that your applications serve, and to configure groups and sampling rules which affect how traces are sent to X-Ray.
**Note**
The X-Ray Service map and CloudWatch ServiceLens map have been combined into the X-Ray trace map within the Amazon CloudWatch console. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/) and choose **Trace Map** under **X-Ray traces** from the left navigation pane.
CloudWatch now includes [Application Signals](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html), which can discover and monitor your application services, clients, Synthetics canaries, and service dependencies. Use Application Signals to see a list or visual map of your services, view health metrics based on your service level objectives (SLOs), and drill down to see correlated X-Ray traces for more detailed troubleshooting.
The primary X-Ray console page is the trace map, which is a visual representation of the JSON service graph that X-Ray generates from the trace data generated by your applications. The map consists of service nodes for each application in your account that serves requests, upstream client nodes that represent the origins of the requests, and downstream service nodes that represent web services and resources used by an application while processing a request. There are additional pages for viewing traces and trace details, and configuring groups and sampling rules.
View the console experience for X-Ray and compare with the CloudWatch console in the following sections.
**Topics**
+ [Use the Amazon CloudWatch console](#aws-xray-interface-console-cw)
+ [Use the X-Ray console](#xray-console)
+ [Explore the X-Ray console](#xray-console-explore)
+ [Using the X-Ray trace map](xray-console-servicemap.md)
+ [Viewing traces and trace details](xray-console-traces.md)
+ [Using filter expressions](xray-console-filters.md)
+ [Cross-account tracing](xray-console-crossaccount.md)
+ [Tracing event-driven applications](xray-tracelinking.md)
+ [Using latency histograms](xray-console-histograms.md)
+ [Using X-Ray insights](xray-console-insights.md)
+ [Interacting with the Analytics console](xray-console-analytics.md)
+ [Configuring groups](xray-console-groups.md)
+ [Configuring sampling rules](xray-console-sampling.md)
+ [Configuring adaptive sampling](xray-adaptive-sampling.md)
+ [Console deep linking](xray-console-deeplinks.md)
# Using the X-Ray trace map
View the X-Ray trace map to identify services where errors are occurring, connections with high latency, or traces for requests that were unsuccessful.
**Note**
CloudWatch now includes [Application Signals](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html), which can discover and monitor your application services, clients, synthetics canaries, and service dependencies. Use Application Signals to see a list or visual map of your services, view health metrics based on your service level objectives (SLOs), and drill down to see correlated X-Ray traces for more detailed troubleshooting.
The X-Ray service map and CloudWatch ServiceLens map are combined into the X-Ray trace map within the Amazon CloudWatch console. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/) and choose **Trace Map** under **X-Ray traces** from the left navigation pane.
## Viewing the trace map
The trace map is a visual representation of the trace data that's generated by your applications. The map shows service nodes that serve requests, upstream client nodes that represent the origins of the requests, and downstream service nodes that represent web services and resources that are used by an application while processing a request.
The trace map displays a connected view of traces across event-driven applications that use Amazon SQS and Lambda. For more information, see [tracing event-driven applications](xray-tracelinking.md). The trace map also supports [cross-account tracing](xray-console-crossaccount.md), displaying nodes from multiple accounts in a single map.
------
#### [ CloudWatch console ]
**To view the trace map in the CloudWatch console**
1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/). Choose **Trace Map** under the **X-Ray Traces** section in the left navigation pane.
![\[CloudWatch console trace map page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-cw.png)
1. Choose a service node to view requests for that node, or an edge between two nodes to view requests that traveled that connection.
1. Additional information is displayed below the trace map, including tabs for metrics, alerts, and response time distribution. On the **Metrics** tab, select a range within each graph to drill down to view more detail, or choose **Faults** or **Errors** options to filter traces. On the **Response time distribution** tab, select a range within the graph to filter traces by response time.
![\[Dashboard showing latency, requests, and faults metrics for an ElasticBeanstalk environment.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-cw-servicemap-node-detail.png)
1. View traces by choosing **View traces**, or if a filter has been applied, choose **View filtered traces**.
1. Choose **View logs** to see CloudWatch logs associated with the selected node. Not all trace map nodes support viewing logs. See [troubleshooting CloudWatch logs](xray-troubleshooting.md#xray-troubleshooting-Nologs) for more information.
The trace map indicates issues within each node by outlining it with colors:
+ **Red** for server faults (500 series errors)
+ **Yellow** for client errors (400 series errors)
+ **Purple** for throttling errors (429 Too Many Requests)
If your trace map is large, use the on-screen controls or mouse to zoom in and out and move the map around.
------
#### [ X-Ray console ]
**To view the Service map**
1. Open the [X-Ray console](https://console.aws.amazon.com/xray/home#). The service map is displayed by default. You can also choose **Service Map** from the left navigation pane.
![\[X-Ray console service map page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-xray.png)
1. Choose a service node to view requests for that node, or an edge between two nodes to view requests that traveled that connection.
1. Use the response distribution [histogram](xray-console-histograms.md) to filter traces by duration, and select status codes for which you want to view traces. Then choose **View traces** to open the trace list with the filter expression applied.
![\[Response distribution graph showing latency peaks and service details for Scorekeep AWS ECS container.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-nodedetail-xray.png)
The service map indicates the health of each node by coloring it based on the ratio of successful calls to errors and faults:
+ **Green** for successful calls
+ **Red** for server faults (500 series errors)
+ **Yellow** for client errors (400 series errors)
+ **Purple** for throttling errors (429 Too Many Requests)
If your service map is large, use the on-screen controls or mouse to zoom in and out and move the map around.
------
**Note**
The X-Ray trace map can display up to 10,000 nodes. In rare scenarios where the total number of service nodes exceeds this limit, you may receive an error and be unable to display a complete trace map in the console.
## Filtering the trace map by group
Using a [filter expression](xray-console-filters.md), you can define criteria by which to include traces within a group. Use the following steps to then display that specific group in the trace map.
------
#### [ CloudWatch console ]
Choose a group name from the group filter on the top-left of the trace map.
![\[Search bar for filtering by X-Ray group, with "TestGroup" displayed as an option.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-groups-cw.png)
------
#### [ X-Ray console ]
Choose a group name from the drop-down menu to the left of the search bar.
![\[Drop-down menu showing Default, TestGroup, Create group, and Learn more options.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-select-console.png)
------
The service map will now be filtered to display traces that match the filter expression of the selected group.
## Trace map legend and options
The trace map includes a legend and several options for customizing the map display.
------
#### [ CloudWatch console ]
Choose the **Legend and options** drop-down at the top-right of the map. Choose what is displayed within nodes, including:
+ **Metrics** displays the average response time and number of traces sent per minute during the chosen time range.
+ **Nodes** displays the service icon within each node.
Choose additional map settings from the **Preferences** pane, which can be accessed via the gear icon at the top-right of the map. These settings include selecting which metric is used to determine the size of each node, and which canaries should be displayed on the map.
------
#### [ X-Ray console ]
Display the service map legend by choosing the **Map legend** link at the top-right of the map. Service map options can be chosen at the bottom-right of the trace map, including:
+ **Service Icons** toggles what is displayed within each node, displaying either the service icon, or the average response time and number of traces sent per minute during the chosen time range.
+ **Node sizing: None** sets all nodes to the same size.
+ **Node sizing: Health** sizes nodes by the number of impacted requests including errors, faults, or throttled requests.
+ **Node sizing: Traffic** sizes nodes by the total number of requests.
------
# Viewing traces and trace details
Use the **Traces** page in the X-Ray console to find traces by URL, response code, or other data from the trace summary. After selecting a trace from the trace list, the **Trace details** page displays a map of service nodes that are associated with the selected trace and a timeline of trace segments.
## Viewing traces
------
#### [ CloudWatch console ]
**To view traces in the CloudWatch console**
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the left navigation pane, choose **X-Ray traces**, then choose **Traces**. You can filter by group or enter a [filter expression](xray-console-filters.md). This filters the traces that are displayed in the **Traces** section at the bottom of the page.
Alternatively, you can use the service map to navigate to a specific service node, and then view traces. This opens the **Traces** page with a query already applied.
1. Refine your query in the **Query refiners** section. To filter traces by a common attribute, choose an option from the down arrow next to **Refine query by**. The options include the following:
+ Node – Filter traces by service node.
+ Resource ARN – Filter traces by a resource associated with a trace. Examples of these resources include Amazon Elastic Compute Cloud (Amazon EC2) instance, an AWS Lambda function, or an Amazon DynamoDB table.
+ User – Filter traces with a user ID.
+ Error root cause message – Filter traces by error root cause.
+ URL – Filter traces by a URL path used by your application.
+ HTTP status code – Filter traces by the HTTP status code returned by your application. You can specify a custom response code or select from the following:
+ `200` – The request was successful.
+ `401` – The request lacked valid authentication credentials.
+ `403` – The request lacked valid permissions.
+ `404` – The server could not find the requested resource.
+ `500` – The server encountered an unexpected condition and generated an internal error.
Choose one or more entries and then choose **Add to query** to add to the filter expression at the top of the page.
1. To find a single trace, enter a [trace ID](xray-api-sendingdata.md#xray-api-traceids) directly into the query field. You can use X-Ray format or World Wide Web Consortium (W3C) format. For example, a trace that's created using the [AWS Distro for OpenTelemetry](xray-instrumenting-your-app.md#xray-instrumenting-opentel) is in W3C format.
**Note**
When you query traces that are created with a W3C-format trace ID, the console displays the matching trace in X-Ray format. For example, if you query for `4efaaf4d1e8720b39541901950019ee5` in W3C format, the console displays the X-Ray equivalent: `1-4efaaf4d-1e8720b39541901950019ee5`.
1. Choose **Run query** at any time to display a list of matching traces within the **Traces** section at the bottom of the page.
1. To display the **Trace details** page for a single trace, select a trace ID from the list.
The following image shows a **Trace map** containing service nodes associated with the trace and edges between the nodes representing the path taken by segments that compose the trace. A **Trace summary** follows the **Trace Map**. The summary contains information about a sample `GET` operation, its **Response Code**, the **Duration** that the trace took to run, and the **Age** of the request. The **Segments Timeline** follows the **Trace Summary** that shows the duration of trace segments and subsegments.
![\[A trace map, summary and segments timeline details information about service nodes and the segments in the trace.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/xray-trace-details-cw.png)
If you have an event-driven application that uses Amazon SQS and Lambda, you can see a connected view of traces for each request in the **Trace map**. In the map, traces from message producers are linked to traces from AWS Lambda consumers and are displayed as a dashed-line edge. For more information about event-driven applications, see [Tracing event-driven applications](xray-tracelinking.md).
The **Traces** and **Trace details** pages also support [cross-account tracing](xray-console-crossaccount.md), which can list traces from multiple accounts in the trace list and inside a single trace map.
------
#### [ X-Ray console ]
**To view traces in the X-Ray console**
1. Open the [Traces](https://console.aws.amazon.com/xray/home#/traces) page in the X-Ray console. The **Trace overview** panel shows a list of traces that are grouped by common features including **Error root causes**, **ResourceARN**, and **InstanceId**.
1. To select a common feature to view a grouped set of traces, expand the down arrow next to **Group by**. The following illustration shows a trace overview of traces that are grouped by URL for the [AWS X-Ray sample application](xray-scorekeep.md), and a list of associated traces.
![\[Example trace overview grouped by URL, followed by a trace list with details including ID, Method, and Response.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-traces.png)
1. Choose the **ID** of a trace to view it under the **Trace list**. You can also choose **Service map** in the navigation pane to view traces for a specific service node. Then you can view traces that are associated with that node.
The **Timeline** tab shows the request flow for the trace, and includes the following:
+ A map of the path for each segment in the trace.
+ How long it took for the segment to reach a node in the trace map.
+ How many requests were made to the node in the trace map.
The following illustration shows an example **Trace Map** associated with a `GET` request made to a sample application. The arrows show the path that each segment took to complete the request. The service nodes show the number of requests made during the `GET` request.
![\[Trace map followed by a timeline with segments, their duration, origin, and end with respect to each other.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/xray-trace-details.png)
For more information about the **Timeline** tab, see the following **Exploring the trace timeline** section.
The **Raw data** tab shows information about the trace, and the segments and subsegments that compose the trace, in `JSON` format. This information may include the following:
+ Timestamps
+ Unique IDs
+ Resources associated with the segment or subsegment
+ The source, or origin, of the segment or subsegment
+ Additional information about the request to your application such as the response from an HTTP request
------
## Exploring the trace timeline
The **Timeline** section shows a hierarchy of segments and subsegments next to a horizontal bar that corresponds to time they used to complete their tasks. The first entry in the list is the segment, which represents all data recorded by the service for a single request. Subsegments are indented and listed following the segment. Columns contain information about each segment.
------
#### [ CloudWatch console ]
In the CloudWatch console, the **Segments Timeline** provides the following information:
+ The first column: Lists the segments and subsegments in the selected trace.
+ The **Segment status** column: Lists the status outcome of each segment and subsegment.
+ The **Response code** column: Lists an HTTP response status code to a browser request made by the segment or subsegment, when available.
+ The **Duration** column: Lists how long the segment or subsegment ran.
+ The **Hosted in** column: Lists the namespace or environment where the segment or subsegment is ran, if applicable. For more information, see [Dimensions collected and dimension combinations](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AppSignals-StandardMetrics.html#AppSignals-StandardMetrics-Dimensions).
+ The last column: Displays horizontal bars that correspond to the duration that the segment or subsegment ran, in relation to the other segments or subsegments in the timeline.
To group the list of segments and subsegments by service node, turn on **Group by nodes**.
------
#### [ X-Ray console ]
In the trace details page, choose the **Timeline** tab to see the timeline for each segment and subsegment that makes up a trace.
In the X-Ray console, the **Timeline** provides the following information:
+ The **Name** column: Lists the names of the segments and subsegments in the trace.
+ The **Res.** column: Lists an HTTP response status code to a browser request made by the segment or subsegment, when available.
+ The **Duration** column: Lists how long the segment or subsegment ran.
+ The **Status** column: Lists the outcome of the segment or subsegment status.
+ The last column: Displays horizontal bars that correspond to the duration that the segment or subsegment ran, in relation to the other segments or subsegments in the timeline.
To see the raw trace data that the console uses to generate the timeline, choose the **Raw data** tab. The raw data shows you information about the trace, and the segments and subsegments that compose the trace in `JSON` format. This information may include the following:
+ Timestamps
+ Unique IDs
+ Resources associated with the segment or subsegment
+ The source, or origin, of the segment or subsegment
+ Additional information about the request to your application such as the response from an HTTP request.
------
When you use an instrumented AWS SDK, HTTP, or SQL client to make calls to external resources, the X-Ray SDK records subsegments automatically. You can also use the X-Ray SDK to record custom subsegments for any function or block of code. Additional subsegments that are recorded while a custom subsegment are open become children of the custom subsegment.
## Viewing segment details
From the trace **Timeline**, choose the name of a segment to view its details.
The **Segment details** panel shows the **Overview**, **Resources**, **Annotations**, **Metadata**, **Exceptions**, and **SQL** tabs. The following apply:
+ The **Overview** tab shows information about the request and response. Information includes the name, start time, end time, duration, the request URL, request operation, request response code, and any errors and faults.
+ The **Resources** tab for a segment shows information from the X-Ray SDK and about the AWS resources running your application. Use the Amazon EC2, AWS Elastic Beanstalk, or Amazon ECS plugins for the X-Ray SDK to record service-specific resource information. For more information about plugins, see the **Service plugins** section in [Configuring the X-Ray SDK for Java](xray-sdk-java-configuration.md).
+ The remaining tabs show **Annotations**, **Metadata**, and **Exceptions** that are recorded for the segment. Exceptions are captured automatically when they are generated from an instrumented request. Annotations and metadata contain additional information that you record by using the operations that the X-Ray SDK provides. To add annotations or metadata to your segments, use the X-Ray SDK. For more information, see the language-specific link listed under Instrumenting your application with AWS X-Ray SDKs in [Instrumenting your application for AWS X-Ray](xray-instrumenting-your-app.md).
## Viewing subsegment details
From the trace timeline, choose the name of a subsegment to view its details:
+ The **Overview** tab contains information about the request and response. This includes the name, start time, end time, duration, the request URL, request operation, request response code, and any errors and faults. For subsegments generated with instrumented clients, the **Overview** tab contains information about the request and response from your application's point of view.
+ The **Resources** tab for a subsegment shows details about the AWS resources that were used to run the subsegment. For example, the resources tab may include an AWS Lambda function ARN, information about a DynamoDB table, any operation that is called, and request ID.
+ The remaining tabs show **Annotations**, **Metadata**, and **Exceptions** recorded on the subsegment. Exceptions are captured automatically when they are generated from an instrumented request. Annotations and metadata contain additional information that you record by using the operations that the X-Ray SDK provides. Use the X-Ray SDK to add annotations or metadata to your segments. For more information, see the language-specific link listed under **Instrumenting your application with AWS X-Ray SDKs** in [Instrumenting your application for AWS X-Ray](xray-instrumenting-your-app.md).
For custom subsegments, the **Overview** tab shows the name of the subsegment, which you can set to specify the area of the code or function that it records. For more information, see the language-specific link listed under **Instrumenting your application with AWS X-Ray SDKs** in [Generating custom subsegments with the X-Ray SDK for Java](xray-sdk-java-subsegments.md).
The following image shows the **Overview** tab for a custom subsegment. The overview contains the subsegment ID, parent ID, Name, start and end times, duration, status and errors or faults.
![\[Overview information about a subsegment including ID, parent ID, Name, times, errors, and faults.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-PUTrules-customsubsegment-overview.png)
The **Metadata** tab for a custom subsegment contains information in JSON format about resources used by that subsegment.
# Using filter expressions
Use *filter expressions* to view a trace map or traces for a specific request, service, connection between two services (an edge), or requests that satisfy a condition. X-Ray provides a filter expression language for filtering requests, services, and edges based on data in request headers, response status, and indexed fields on the original segments.
When you choose a time period of traces to view in the X-Ray console, you might get more results than the console can display. In the upper-right corner, the console shows the number of traces that it scanned and whether there are more traces available. You can use a filter expression to narrow the results to just the traces that you want to find.
**Topics**
+ [Filter expression details](#xray-console-filters-details)
+ [Using filter expressions with groups](#groups)
+ [Filter expression syntax](#console-filters-syntax)
+ [Boolean keywords](#console-filters-boolean)
+ [Number keywords](#console-filters-number)
+ [String keywords](#console-filters-string)
+ [Complex keywords](#console-filters-complex)
+ [id function](#console-filters-functions)
## Filter expression details
When you [choose a node in the trace map](xray-console-servicemap.md), the console constructs a filter expression based on the service name of the node, and the types of error present based on your selection. To find traces that show performance issues or that relate to specific requests, you can adjust the expression that the console provides or create your own. If you add annotations with the X-Ray SDK, you can also filter based on the presence of an annotation key or the value of a key.
**Note**
If you choose a relative time range in the trace map and choose a node, the console converts the time range to an absolute start and end time. To ensure that the traces for the node appear in the search results, and avoid scanning times when the node wasn't active, the time range only includes times when the node sent traces. To search relative to the current time, you can switch back to a relative time range in the traces page and scan again.
If there are still more results available than the console can show, the console shows you how many traces matched and the number of traces scanned. The percentage shown is the percentage of the selected time frame that was scanned. To ensure that you see all matching traces represented in the results, narrow your filter expression further, or choose a shorter time frame.
To get the freshest results first, the console starts scanning at the end of the time range and works backward. If there are a large number of traces, but few results, the console splits the time range into chunks and scans them in parallel. The progress bar shows the parts of the time range that have been scanned.
![\[Progress bar showing 52% of time range scanned, with 49 matching traces found.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-tracescan-parallel.png)
## Using filter expressions with groups
Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics.
Groups are identified by their name or an Amazon Resource Name (ARN), and contain a filter expression. The service compares incoming traces to the expression and stores them accordingly.
You can create and modify groups by using the dropdown menu to the left of the filter expression search bar.
**Note**
If the service encounters an error in qualifying a group, that group is no longer included in processing incoming traces and an error metric is recorded.
For more information about groups, see [Configuring groups](xray-console-groups.md).
## Filter expression syntax
Filter expressions can contain a *keyword*, a unary or binary *operator*, and a *value* for comparison.
```
keyword operator value
```
Different operators are available for different types of keyword. For example, `responsetime` is a number keyword and can be compared with operators related to numbers.
**Example – requests where response time was greater than 5 seconds**
```
responsetime > 5
```
You can combine multiple expressions in a compound expression by using the `AND` or `OR` operators.
**Example – requests where the total duration was 5–8 seconds**
```
duration >= 5 AND duration <= 8
```
Simple keywords and operators find issues only at the trace level. If an error occurs downstream, but is handled by your application and not returned to the user, a search for `error` will not find it.
To find traces with downstream issues, you can use the [complex keywords](#console-filters-complex) `service()` and `edge()`. These keywords let you apply a filter expression to all downstream nodes, a single downstream node, or an edge between two nodes. For more granularity, you can filter services and edges by type with [the id() function](#console-filters-functions).
## Boolean keywords
Boolean keyword values are either true or false. Use these keywords to find traces that resulted in errors.
**Boolean keywords**
+ `ok` – Response status code was 2XX Success.
+ `error` – Response status code was 4XX Client Error.
+ `throttle` – Response status code was 429 Too Many Requests.
+ `fault` – Response status code was 5XX Server Error.
+ `partial` – Request has incomplete segments.
+ `inferred` – Request has inferred segments.
+ `first` – Element is the first of an enumerated list.
+ `last` – Element is the last of an enumerated list.
+ `remote` – Root cause entity is remote.
+ `root` – Service is the entry point or root segment of a trace.
Boolean operators find segments where the specified key is `true` or `false`.
**Boolean operators**
+ none – The expression is true if the keyword is true.
+ `!` – The expression is true if the keyword is false.
+ `=`,`!=` – Compare the value of the keyword to the string `true` or `false`. These operators act the same as the other operators but are more explicit.
**Example – response status is 2XX OK**
```
ok
```
**Example – response status is not 2XX OK**
```
!ok
```
**Example – response status is not 2XX OK**
```
ok = false
```
**Example – last enumerated fault trace has error name "deserialize"**
```
rootcause.fault.entity { last and name = "deserialize" }
```
**Example – requests with remote segments where coverage is greater than 0.7 and the service name is "traces"**
```
rootcause.responsetime.entity { remote and coverage > 0.7 and name = "traces" }
```
**Example – requests with inferred segments where the service type is "AWS:DynamoDB"**
```
rootcause.fault.service { inferred and name = traces and type = "AWS::DynamoDB" }
```
**Example – requests that have a segment with the name "data-plane" as the root**
```
service("data-plane") {root = true and fault = true}
```
## Number keywords
Use number keywords to search for requests with a specific response time, duration, or response status.
**Number keywords**
+ `responsetime` – Time that the server took to send a response.
+ `duration` – Total request duration, including all downstream calls.
+ `http.status` – Response status code.
+ `index` – Position of an element in an enumerated list.
+ `coverage` – Decimal percentage of entity response time over root segment response time. Applicable only for response time root cause entities.
**Number operators**
Number keywords use standard equality and comparison operators.
+ `=`,`!=` – The keyword is equal to or not equal to a number value.
+ `<`,`<=`, `>`,`>=` – The keyword is less than or greater than a number value.
**Example – response status is not 200 OK**
```
http.status != 200
```
**Example – request where the total duration was 5–8 seconds**
```
duration >= 5 AND duration <= 8
```
**Example – requests that completed successfully in less than 3 seconds, including all downstream calls**
```
ok !partial duration <3
```
**Example – enumerated list entity that has an index greater than 5**
```
rootcause.fault.service { index > 5 }
```
**Example – requests where the last entity that has coverage greater than 0.8**
```
rootcause.responsetime.entity { last and coverage > 0.8 }
```
## String keywords
Use string keywords to find traces with specific text in the request headers, or specific user IDs.
**String keywords**
+ `http.url` – Request URL.
+ `http.method` – Request method.
+ `http.useragent` – Request user agent string.
+ `http.clientip` – Requestor's IP address.
+ `user` – Value of the user field on any segment in the trace.
+ `name` – The name of a service or exception.
+ `type` – Service type.
+ `message` – Exception message.
+ `availabilityzone` – Value of the availabilityzone field on any segment in the trace.
+ `instance.id` – Value of the instance ID field on any segment in the trace.
+ `resource.arn` – Value of the resource ARN field on any segment in the trace.
String operators find values that are equal to or contain specific text. Values must always be specified in quotation marks.
**String operators**
+ `=`,`!=` – The keyword is equal to or not equal to a number value.
+ `CONTAINS` – The keyword contains a specific string.
+ `BEGINSWITH` , `ENDSWITH` – The keyword begins or ends with a specific string.
**Example – http.url filter**
```
http.url CONTAINS "/api/game/"
```
To test if a field exists on a trace, regardless of its value, check to see if it contains the empty string.
**Example – user filter**
Find all traces with user IDs.
```
user CONTAINS ""
```
**Example – select traces with a fault root cause that includes a service named "Auth"**
```
rootcause.fault.service { name = "Auth" }
```
**Example – select traces with a response time root cause whose last service has a type of DynamoDB**
```
rootcause.responsetime.service { last and type = "AWS::DynamoDB" }
```
**Example – select traces with a fault root cause whose last exception has the message "access denied for account\$1id: 1234567890"**
```
rootcause.fault.exception { last and message = "Access Denied for account_id: 1234567890"
```
## Complex keywords
Use complex keywords to find requests based on service name, edge name, or annotation value. For services and edges, you can specify an additional filter expression that applies to the service or edge. For annotations, you can filter on the value of an annotation with a specific key using Boolean, number, or string operators.
**Complex keywords**
+ `annotation[key]` – Value of an annotation with field *key*. The value of an annotation can be a Boolean, number, or string, so you can use any of the comparison operators of those types. You can use this keyword in combination with the `service` or `edge` keyword. An annotation key that contains dots (periods) must be wrapped in square brackets (**[ ]**).
+ `edge(source, destination) {filter}` – Connection between services *source* and *destination*. Optional curly braces can contain a filter expression that applies to segments on this connection.
+ `group.name / group.arn` – The value of a group's filter expression, referenced by group name or group ARN.
+ `json` – JSON root cause object. See [Getting data from AWS X-Ray](xray-api-gettingdata.md) for steps to create JSON entities programmatically.
+ `service(name) {filter}` – Service with name *name*. Optional curly braces can contain a filter expression that applies to segments created by the service.
Use the service keyword to find traces for requests that hit a certain node on your trace map.
Complex keyword operators find segments where the specified key has been set, or not set.
**Complex keyword operators**
+ none – The expression is true if the keyword is set. If the keyword is of boolean type, it will evaluate to the boolean value.
+ `!` – The expression is true if the keyword is not set. If the keyword is of boolean type, it will evaluate to the boolean value.
+ `=`,`!=` – Compare the value of the keyword.
+ `edge(source, destination) {filter}` – Connection between services *source* and *destination*. Optional curly braces can contain a filter expression that applies to segments on this connection.
+ `annotation[key]` – Value of an annotation with field *key*. The value of an annotation can be a Boolean, number, or string, so you can use any of the comparison operators of those types. You can use this keyword in combination with the `service` or `edge` keyword.
+ `json` – JSON root cause object. See [Getting data from AWS X-Ray](xray-api-gettingdata.md) for steps to create JSON entities programmatically.
Use the service keyword to find traces for requests that hit a certain node on your trace map.
**Example – Service filter**
Requests that included a call to `api.example.com` with a fault (500 series error).
```
service("api.example.com") { fault }
```
You can exclude the service name to apply a filter expression to all nodes on your service map.
**Example – service filter**
Requests that caused a fault anywhere on your trace map.
```
service() { fault }
```
The edge keyword applies a filter expression to a connection between two nodes.
**Example – edge filter**
Request where the service `api.example.com` made a call to `backend.example.com` that failed with an error.
```
edge("api.example.com", "backend.example.com") { error }
```
You can also use the `!` operator with service and edge keywords to exclude a service or edge from the results of another filter expression.
**Example – service and request filter**
Request where the URL begins with `http://api.example.com/` and contains `/v2/` but does not reach a service named `api.example.com`.
```
http.url BEGINSWITH "http://api.example.com/" AND http.url CONTAINS "/v2/" AND !service("api.example.com")
```
**Example – service and response time filter**
Find traces where `http url` is set and response time is greater than 2 seconds.
```
http.url AND responseTime > 2
```
For annotations, you can call all traces where `annotation[key]` is set, or use the comparison operators that correspond to the type of value.
**Example – annotation with string value**
Requests with an annotation named `gameid` with string value `"817DL6VO"`.
```
annotation[gameid] = "817DL6VO"
```
**Example – annotation is set**
Requests with an annotation named `age` set.
```
annotation[age]
```
**Example – annotation is not set**
Requests without an annotation named `age` set.
```
!annotation[age]
```
**Example – annotation with number value**
Requests with annotation age with numerical value greater than 29.
```
annotation[age] > 29
```
**Example – annotation in combination with service or edge**
```
service { annotation[request.id] = "917DL6VO" }
```
```
edge { source.annotation[request.id] = "916DL6VO" }
```
```
edge { destination.annotation[request.id] = "918DL6VO" }
```
**Example – group with user**
Requests where traces meet the `high_response_time` group filter (e.g. `responseTime > 3`), and the user is named Alice.
```
group.name = "high_response_time" AND user = "alice"
```
**Example – JSON with root cause entity**
Requests with matching root cause entities.
```
rootcause.json = #[{ "Services": [ { "Name": "GetWeatherData", "EntityPath": [{ "Name": "GetWeatherData" }, { "Name": "get_temperature" } ] }, { "Name": "GetTemperature", "EntityPath": [ { "Name": "GetTemperature" } ] } ] }]
```
## id function
When you provide a service name to the `service` or `edge` keyword, you get results for all nodes that have that name. For more precise filtering, you can use the `id` function to specify a service type in addition to a name to distinguish between nodes with the same name.
Use the `account.id` function to specify a particular account for the service, when viewing traces from multiple accounts in a monitoring account.
```
id(name: "service-name", type:"service::type", account.id:"account-ID")
```
You can use the `id` function in place of a service name in service and edge filters.
```
service(id(name: "service-name", type:"service::type")) { filter }
```
```
edge(id(name: "service-one", type:"service::type"), id(name: "service-two", type:"service::type")) { filter }
```
For example, AWS Lambda functions result in two nodes in the trace map; one for the function invocation, and one for the Lambda service. The two nodes have the same name but different types. A standard service filter will find traces for both.
**Example – service filter**
Requests that include an error on any service named `random-name`.
```
service("random-name") { error }
```
Use the `id` function to narrow the search to errors on the function itself, excluding errors from the service.
**Example – service filter with id function**
Requests that include an error on a service named `random-name` with type `AWS::Lambda::Function`.
```
service(id(name: "random-name", type: "AWS::Lambda::Function")) { error }
```
To search for nodes by type, you can also exclude the name entirely.
**Example – service filter with id function and service type**
Requests that include an error on a service with type `AWS::Lambda::Function`.
```
service(id(type: "AWS::Lambda::Function")) { error }
```
To search for nodes for a particular AWS account, specify an account ID.
**Example – service filter with id function and account ID**
Requests that include a service within a specific account ID `AWS::Lambda::Function`.
```
service(id(account.id: "account-id"))
```
# Cross-account tracing
AWS X-Ray supports *cross-account observability*, enabling you to monitor and troubleshoot applications that span multiple accounts within an AWS Region. You can seamlessly search, visualize, and analyze your metrics, logs, and traces in any of the linked accounts as if you were operating in a single account. This provides a complete view of requests that travel across multiple accounts. You can view cross-account traces in the X-Ray trace map and traces pages within the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/).
The shared observability data can include any of the following types of telemetry:
+ Metrics in Amazon CloudWatch
+ Log groups in Amazon CloudWatch Logs
+ Traces in AWS X-Ray
+ Applications in Amazon CloudWatch Application Insights
## Configure cross-account observability
To turn on cross-account observability, set up one or more AWS *monitoring* accounts and link them with multiple *source* accounts. A monitoring account is a central AWS account that can view and interact with observability data that's generated from source accounts. A source account is an individual AWS account that generates observability data for the resources that it contains.
Source accounts share their observability data with monitoring accounts. Traces are copied from each source account to up to five monitoring accounts. Copies of traces from source accounts to the first monitoring account are free. Copies of traces sent to additional monitoring accounts are charged to each source account, based on standard pricing. For more information, see [AWS X-Ray pricing](https://aws.amazon.com/xray/pricing/) and [Amazon CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/).
To create links between monitoring accounts and source accounts, use the CloudWatch console or the new Observability Access Manager commands in the AWS CLI and API. For more information, see [CloudWatch cross-account observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html).
**Note**
X-Ray traces are billed to the AWS account where they're received. If a [sampled](xray-concepts.md#xray-concepts-sampling) request spans services across more than one AWS account, each account records a separate trace, and all traces share the same trace ID. To learn more about cross-account observability pricing, see [AWS X-Ray pricing](https://aws.amazon.com/xray/pricing/) and [Amazon CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/).
## Viewing cross-account traces
Cross-account traces are displayed in the monitoring account. Each source account displays only local traces for that specific account. The following sections assume that you're signed in to the monitoring account and have opened the Amazon CloudWatch console. On both the trace map and traces pages, a monitoring account badge is displayed in the upper-right corner.
![\[Monitoring account badge\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-monitoring-account.png)
### Trace map
In the CloudWatch console, choose **Trace Map** under **X-Ray traces** from the left navigation pane. By default, the trace map displays nodes for all source accounts that send traces to the monitoring account, and nodes for the monitoring account itself. On the trace map, choose **Filters** from the upper left to filter the trace map using the **Accounts** drop-down. After an account filter is applied, service nodes from accounts that don't match the current filter are grayed out.
![\[Filtered trace map\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-servicemap-account-filter.png)
When you choose a service node, the node details pane includes the service's account ID and label.
![\[Node detail pane\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-servicemap-node-detail.png)
In the upper-right corner of the trace map, choose **List view** to see a list of service nodes. The list of service nodes includes services from the monitoring account and all configured source accounts. Filter the list of nodes by **Account label** or **Account id** by choosing them from the **Nodes** filter.
![\[Filtered service list\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-servicelist-account-filter.png)
### Traces
View trace details for traces that span multiple accounts by opening the CloudWatch console from the monitoring account, and choosing **Traces** under **X-Ray traces** in the left navigation pane. You can also open this page by choosing a node in the X-Ray **Trace Map**, and then choosing **View traces** from the node details pane.
The **Traces** page supports querying by account ID. To get started, [enter a query](xray-console-filters.md) that includes one or more account IDs. The following example queries for traces that have passed through account ID X* *or *Y*:
```
service(id(account.id:"X")) OR service(id(account.id:"Y"))
```
![\[Query traces by account\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-traces-query-by-account.png)
Refine your query by **Account**. Select one or more accounts from the list, and choose **Add to query**.
![\[Refine trace query by account\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-traces-refine-by-account.png)
### Trace details
View details for a trace by choosing it from the **Traces** list at the bottom of the **Traces** page. The **Trace details** displays, including a trace details map with service nodes from across all accounts that the trace passed through. Choose a specific service node to see its corresponding account.
The **Segments timeline** section displays the account details for each segment in the timeline.
![\[Segments timeline\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-traces-segment-timeline.png)
# Tracing event-driven applications
AWS X-Ray supports tracing event-driven applications using Amazon SQS and AWS Lambda. Use the CloudWatch console to see a connected view of each request as it's queued with Amazon SQS and processed by one or more Lambda functions. Traces from upstream message producers are automatically linked to traces from downstream Lambda consumer nodes, creating an end-to-end view of the application.
**Note**
Each trace segment can be linked to up to 20 traces, while a trace can include a maximum of 100 links. In certain scenarios, linking additional traces may result in exceeding the [maximum trace document size](https://docs.aws.amazon.com/general/latest/gr/xray.html#limits_xray), causing a potentially incomplete trace. This can happen, for example, when a Lambda function with tracing enabled sends many SQS messages to a queue in a single invocation. If you encounter this issue, a mitigation is available which uses the X-Ray SDKs. See the X-Ray SDK for [Java](https://github.com/aws/aws-xray-sdk-java#oversampling-mitigation), [Node.js](https://github.com/aws/aws-xray-sdk-node/tree/master/packages/core#oversampling-mitigation), [Python](https://github.com/aws/aws-xray-sdk-python#oversampling-mitigation), [Go](https://github.com/aws/aws-xray-sdk-go#oversampling-mitigation), or [.NET](https://github.com/aws/aws-xray-sdk-dotnet#oversampling-mitigation) for more information.
## View linked traces in the trace map
Use the **Trace Map** page within the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/) to view a trace map with traces from message producers that are linked to traces from Lambda consumers. These links are displayed with a dashed-line edge that connects the Amazon SQS node and downstream Lambda consumer nodes.
![\[Edge between Amazon SQS and Lambda nodes.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-servicemap-linkededge.png)
Select a dashed-line edge to display a *received event age* histogram, which maps the spread of event age when it's received by consumers. The age is calculated each time an event is received.
![\[Edge with received event age histogram.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-linkededgedetails-cw.png)
## View linked trace details
**View trace details sent from a message producer, Amazon SQS queue, or Lambda consumer:**
1. Use the **Trace Map** to select a message producer, Amazon SQS, or Lambda consumer node.
1. Choose **View traces** from the node details pane to display a list of traces. You can also navigate directly to the **Traces** page within the CloudWatch console.
1. Choose a specific trace from the list to open the trace details page. The trace details page displays a message when the selected trace is part of a linked set of traces.
![\[Linked trace details\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-header.png)
The trace details map displays the current trace, along with upstream and downstream linked traces, each of which are contained within a box that indicates the bounds of each trace. If the currently selected trace is linked to multiple upstream or downstream traces, the nodes within the upstream or downstream linked traces are stacked, and a **Select trace** button is displayed.
![\[Multiple linked upstream traces\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-tracemap.png)
Beneath the trace details map, a timeline of trace segments displays, including upstream and downstream linked traces. If there are multiple upstream or downstream linked traces, their segment details can't be displayed. To view segment details for a single trace within a set of linked traces, [select a single trace](#xray-tracelinking-filterbatch) as described below.
![\[Segments timeline showing linked traces\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-timeline.png)
## Select a single trace within a set of linked traces
**Filter a linked set of traces to a single trace, to see segment details in the timeline.**
1. Choose **Select trace** underneath the linked traces on the trace details map. A list of traces displays.
![\[Linked trace list\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-tracelist.png)
1. Select the radio button next to a trace to view it within the trace details map.
1. Choose **Cancel trace selection** to view the entire set of linked traces.
![\[Single linked trace\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-filteredbatch.png)
# Using latency histograms
When you select a node or edge on an AWS X-Ray [trace map](xray-console-servicemap.md), the X-Ray console shows a latency distribution histogram.
## Latency
Latency is the amount of time between when a request starts and when it completes. A histogram shows a distribution of latencies. It shows duration on the x-axis, and the percentage of requests that match each duration on the y-axis.
This histogram shows a service that completes most requests in less than 300 ms. A small percentage of requests take up to 2 seconds, and a few outliers take more time.
![\[Latency histogram with duration on the x-axis and percentage of requests for each duration on the y-axis\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-servicemap-histogram.png)
## Interpreting service details
Service histograms and edge histograms provide a visual representation of latency from the viewpoint of a service or requester.
+ Choose a *service node* by clicking the circle. X-Ray shows a histogram for requests served by the service. The latencies are those recorded by the service, and don't include any network latency between the service and the requester.
+ Choose an *edge* by clicking the line or arrow tip of the edge between two services. X-Ray shows a histogram for requests from the requester that were served by the downstream service. The latencies are those recorded by the requester, and include latency in the network connection between the two services.
To interpret the **Service details** panel histogram, you can look for values that differ the most from the majority of values in the histogram. These *outliers* can be seen as peaks or spikes in the histogram, and you can view the traces for a specific area to investigate what's going on.
To view traces filtered by latency, select a range on the histogram. Click where you want to start the selection and drag from left to right to highlight a range of latencies to include in the trace filter.
![\[Select a range to view traces by clicking where to start and dragging left to right to create the range for the trace filter\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-servicemap-servicedetails-selection.png)
After selecting a range, you can choose **Zoom** to view just that portion of the histogram and refine your selection.
![\[Choose zoom to view the selected range in the histogram\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-servicemap-servicedetails-zoom.png)
Once you have the focus set to the area you're interested in, choose **View traces**.
# Using X-Ray insights
AWS X-Ray continuously analyzes trace data in your account to identify emergent issues in your applications. When fault rates exceed the expected range, it creates an *insight* that records the issue and tracks its impact until it's resolved. With insights, you can:
+ Identify where in your application issues are occurring, the root cause of the issue, and associated impact. The impact analysis provided by insights enables you to derive the severity and priority of an issue.
+ Receive notifications as the issue changes over time. Insights notifications can be integrated with your monitoring and alerting solution using Amazon EventBridge. This integration enables you to send automated emails or alerts based on the severity of the issue.
The X-Ray console identifies nodes with ongoing incidents in the trace map. To see a summary of the insight, choose the affected node. You can also view and filter insights by choosing **Insights** from the navigation pane on the left.
![\[Trace map node with insight summary.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-servicemap.png)
X-Ray creates an insight when it detects an *anomaly* in one or more nodes of the service map. The service uses statistical modeling to predict the expected fault rates of services in your application. In the preceding example, the anomaly is an increase in faults from AWS Elastic Beanstalk. The Elastic Beanstalk server experienced multiple API call timeouts, causing an anomaly in the downstream nodes.
## Enable insights in the X-Ray console
Insights must be enabled for each group you want to use insights features with. You can enable insights from the **Groups** page.
1. Open the [X-Ray console](https://console.aws.amazon.com/xray/home#).
1. Select an existing group or create a new one by choosing **Create group**, and then select **Enable Insights**. For more information about configuring groups in the X-Ray console, see [Configuring groups](xray-console-groups.md).
1. In the navigation pane on the left, choose **Insights**, and then choose an insight to view.
![\[List of insights in the X-Ray console.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights.png)
**Note**
X-Ray uses GetInsightSummaries, GetInsight, GetInsightEvents, and GetInsightImpactGraph API operations to retrieve data from insights.
For more information, see [How AWS X-Ray works with IAM](security_iam_service-with-iam.md).
## Enable insights notifications
With insights notifications, a notification is created for each insight event, such as when an insight is created, changes significantly, or is closed. Customers can receive these notifications through Amazon EventBridge events, and use conditional rules to take actions such as SNS notification, Lambda invocation, posting messages to an SQS queue, or any of the targets EventBridge supports. Insights notifications are emitted on a best-effort basis but are not guaranteed. For more information about targets, see [Amazon EventBridge Targets](https://docs.aws.amazon.com/eventbridge/latest/userguide/eventbridge-targets.html).
You can enable insights notifications for any insights enabled group from the **Groups** page.
**To enable notifications for an X-Ray group**
1. Open the [X-Ray console](https://console.aws.amazon.com/xray/home#).
1. Select an existing group or create a new one by choosing **Create group**, ensure that **Enable Insights** is selected, and then select **Enable Notifications**. For more information about configuring groups in the X-Ray console, see [Configuring groups](xray-console-groups.md).
**To configure Amazon EventBridge conditional rules**
1. Open the [Amazon EventBridge console](https://console.aws.amazon.com/events/home).
1. Navigate to **Rules** in the left navigation bar, and choose **Create rule**.
1. Provide a name and description for the rule.
1. Choose **Event pattern**, and then choose **Custom pattern**. Provide a pattern containing `"source": [ "aws.xray" ]` and `"detail-type": [ "AWS X-Ray Insight Update" ]`. The following are some examples of possible patterns.
+ Event pattern to match all incoming events from X-Ray insights:
```
{
"source": [ "aws.xray" ],
"detail-type": [ "AWS X-Ray Insight Update" ]
}
```
+ Event pattern to match a specified **state** and **category**:
```
{
"source": [ "aws.xray" ],
"detail-type": [ "AWS X-Ray Insight Update" ],
"detail": {
"State": [ "ACTIVE" ],
"Category": [ "FAULT" ]
}
}
```
1. Select and configure the targets that you would like to invoke when an event matches this rule.
1. (Optional) Provide tags to more easily identify and select this rule.
1. Choose **Create**.
**Note**
X-Ray insights notifications sends events to Amazon EventBridge, which does not currently support customer managed keys. For more information, see [Data protection in AWS X-Ray](xray-console-encryption.md).
## Insight overview
The overview page for an insight attempts to answer three key questions:
+ What is the underlying issue?
+ What is the root cause?
+ What is the impact?
The **Anomalous services** section shows a timeline for each service that illustrates the change in fault rates during the incident. The timeline shows the number of traces with faults overlaid on a solid band that indicates the expected number of faults based on the amount of traffic recorded. The duration of the insight is visualized by the *Incident window*. The incident window begins when X-Ray observes the metric becoming anomalous and persists while the insight is active.
The following example shows an increase in faults that caused an incident:
![\[Overview page of an X-Ray insight.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-overview.png)
The **Root cause** section shows a trace map focused on the root cause service and the impacted path. You may hide the unaffected nodes by selecting the eye icon in the top right of the Root cause map. The root cause service is the farthest downstream node where X-Ray identified an anomaly. It can represent a service that you instrumented or an external service that your service called with an instrumented client. For example, if you call Amazon DynamoDB with an instrumented AWS SDK client, an increase in faults from DynamoDB results in an insight with DynamoDB as the root cause.
To further investigate the root cause, select **View root cause details** on the root cause graph. You can use the **Analytics** page to investigate the root cause and related messages. For more information, see [Interacting with the Analytics console](xray-console-analytics.md).
![\[Overview page of an X-Ray insight.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-root-cause.png)
Faults that continue upstream in the map can impact multiple nodes and cause multiple anomalies. If a fault is passed all the way back to the user that made the request, the result is a *client fault*. This is a fault in the root node of the trace map. The **Impact** graph provides a timeline of the client experience for the entire group. This experience is calculated based on percentages of the following states: **Fault**, **Error**, **Throttle**, and **Okay**.
![\[Impact graph for an X-Ray incident.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-impact.png)
This example shows an increase in traces with a fault at the root node during the time of an incident. Incidents in downstream services don't always correspond to an increase in client errors.
Choosing **Analyze insight** opens the X-Ray Analytics console in a window where you can dive deep into the set of traces causing the insight. For more information, see [Interacting with the Analytics console](xray-console-analytics.md).
**Understanding impact**
AWS X-Ray measures the impact caused by an ongoing issue as part of generating insights and notifications. The impact is measured in two ways:
+ Impact to the X-Ray [group](xray-console-groups.md)
+ Impact on the root cause service
This impact is determined by the percentage of request that are failing or causing an error within a given time period. This impact analysis allows you to derive the severity and priority of the issue based on your particular scenario. This impact is available as part of the console experience in addition to insights notifications.
**Deduplication**
AWS X-Ray insights de-duplicates issues across multiple microservices. It uses anomaly detection to determine the service that is the root cause of an issue, determines if other related services are exhibiting anomalous behavior due to the same root cause, and records the result as a single insight.
## Review an insight's progress
X-Ray reevaluates insights periodically until they are resolved, and records each notable intermediate change as a [notification](#xray-console-insight-notifications), which can be sent as an Amazon EventBridge event. This enables you to build processes and workflows to determine how the issue has changed over time, and take appropriate actions such as sending an email or integrating with an alerting system using EventBridge.
You can review incident events in the **Impact Timeline** on the **Inspect** page. By default the timeline displays the most impacted service until you choose a different service.
![\[Inspect page with impact timeline.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-inspect.png)
To see a trace map and graphs for an event, choose it from the impact timeline. The trace map shows services in your application that are affected by the incident. Under **Impact analysis**, graphs show fault timelines for the selected node and for clients in the group.
![\[Impact analysis graph for an X-Ray insight.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-inspect-analysis.png)
To take a deeper look at the traces involved in an incident, choose **Analyze event** on the **Inspect** page. You can use the **Analytics** page to refine the list of traces and identify affected users. For more information, see [Interacting with the Analytics console](xray-console-analytics.md).
# Interacting with the Analytics console
The AWS X-Ray Analytics console is an interactive tool for interpreting trace data to quickly understand how your application and its underlying services are performing. The console enables you to explore, analyze, and visualize traces through interactive response time and time-series graphs.
When making selections in the Analytics console, the console constructs filters to reflect the selected subset of all traces. You can refine the active dataset with increasingly granular filters by clicking the graphs and the panels of metrics and fields that are associated with the current trace set.
**Topics**
+ [Console features](#xray-console-analytics-features)
+ [Response time distribution](#xray-console-analytics-response)
+ [Time series activity](#xray-console-analytics-time)
+ [Workflow examples](#xray-console-analytics-workflows)
+ [Observe faults on the service graph](#xray-console-analytics-observe-faults)
+ [Identify response time peaks](#xray-console-analytics-response-peaks)
+ [View all traces marked with a status code](#xray-console-analytics-response-peaks)
+ [View all items in a subgroup and associated to a user](#xray-console-analytics-subgroups)
+ [Compare two sets of traces with different criteria](#xray-console-analytics-compare)
+ [Identify a trace of interest and view its details](#xray-console-analytics-identify)
## Console features
The X-Ray Analytics console uses the following key features for grouping, filtering, comparing, and quantifying trace data.
### Features
| Feature | Description |
| --- | --- |
| **Groups** | The initial selected group is `Default`. To change the retrieved group, select a different group from the menu to the right of the main filter expression search bar. To learn more about groups see, [Using filter expressions with groups](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-filters.html#groups). |
| **Retrieved traces** | By default, the Analytics console generates graphs based on all traces in the selected group. Retrieved traces represent all traces in your working set. You can find the trace count in this tile. Filter expressions you apply to the main search bar refine and update the retrieved traces. |
| **Show in charts/Hide from charts** | A toggle to compare the active group against the retrieved traces. To compare the data related to the group against any active filters, choose **Show in charts**. To remove this view from the charts, choose **Hide from charts**. |
| **Filtered trace set A** | Through interactions with the graphs and tables, apply filters to create the criteria for **Filtered trace set A**. As the filters are applied, the number of applicable traces and the percentage of traces from the total that are retrieved are calculated within this tile. Filters populate as tags within the **Filtered trace set A** tile and can also be removed from the tile. |
| **Refine** | This function updates the set of retrieved traces based on the filters applied to trace set A. Refining the retrieved trace set refreshes the working set of all traces retrieved based on the filters for trace set A. The working set of retrieved traces is a sampled subset of all traces in the group. |
| **Filtered trace set B** | When created, **Filtered trace set B** is a copy of **Filtered trace set A**. To compare the two trace sets, make new filter selections that will apply to trace set B, while trace set A remains fixed. As the filters are applied, the number of applicable traces and the percentage of traces from the total retrieved are calculated within this tile. Filters populate as tags within the **Filtered trace set B** tile and can also be removed from the tile. |
| **Response time root cause entity paths** | A table of recorded entity paths. X-Ray determines which path in your trace is the most likely cause for the response time. The format indicates a hierarchy of entities that are encountered, ending in a response time root cause. Use these rows to filter for recurring response time faults. For more information about customizing a root cause filter and getting data through the API see, [Retrieving and refining root cause analytics](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-gettingdata.html#xray-api-analytics). |
| **Delta (**�**)** | A column that is added to the metrics tables when both trace set A and trace set B are active. The Delta column calculates the difference in percentage of traces between trace set A and trace set B. |
## Response time distribution
The X-Ray Analytics console generates two primary graphs to help you visualize traces: **Response Time Distribution** and **Time Series Activity**. This section and the following provide examples of each, and explain the basics of how to read the graphs.
The following are the colors associated with the response time line graph (the time series graph uses the same color scheme):
+ **All traces in the group** – gray
+ **Retrieved traces** – orange
+ **Filtered trace set A** – green
+ **Filtered trace set B** – blue
**Example – Response time distribution**
The response time distribution is a chart that shows the number of traces with a given response time. Click and drag to make selections within the response time distribution. This selects and creates a filter on the working trace set named `responseTime` for all traces within a specific response time.
![\[A chart that shows the response time distribution of traces.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/analytics-responseTime.png)
## Time series activity
The time series activity chart shows the number of traces at a given time period. The color indicators mirror the line graph colors of the response time distribution. The darker and fuller the color block within the activity series, the more traces are represented at the given time.
**Example – Time series activity**
Click and drag to make selections within the time series activity graph. This selects and creates a filter named `timerange` on the working trace set for all traces within a specific range of time.
![\[Make a selection and create filter\]](http://docs.aws.amazon.com/xray/latest/devguide/images/analytics-timeSeries.png)
## Workflow examples
The following examples show common use cases for the X-Ray Analytics console. Each example demonstrates a key function of the console experience. As a group, the examples follow a basic troubleshooting workflow. The steps walk through how to first spot unhealthy nodes, and then how to interact with the Analytics console to automatically generate comparative queries. Once you have narrowed the scope through queries, you will finally look at the details of traces of interest to determine what is damaging the health of your service.
## Observe faults on the service graph
The trace map indicates the health of each node by coloring it based on the ratio of successful calls to errors and faults. When you see a percentage of red on your node, it signals a fault. Use the X-Ray Analytics console to investigate it.
For more information about how to read the trace map, see [Viewing the trace map](https://docs.aws.amazon.com/xray/latest/devguide/xray-console.html#xray-console-servicemap).
![\[Observe a fault\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-gettingstarted-servicemap-before-2021.png)
## Identify response time peaks
Using the response time distribution, you can observe peaks in response time. By selecting the peak in response time, the tables below the graphs will update to expose all associated metrics, such as status codes.
When you click and drag, X-Ray selects and creates a filter. It's shown in a gray shadow on top of the graphed lines. You can now drag that shadow left and right along the distribution to update your selection and filter.
![\[Make a selection and create filter\]](http://docs.aws.amazon.com/xray/latest/devguide/images/analytics-showFilterf.png)
## View all traces marked with a status code
You can drill into traces within the selected peak by using the metrics tables below the graphs. By clicking a row in the **HTTP STATUS CODE** table, you automatically create a filter on the working dataset. For example, you could view all traces of status code 500. This creates a filter tag in the trace set tile named `http.status`.
## View all items in a subgroup and associated to a user
Drill into the error set based on user, URL, response time root cause, or other predefined attributes. For example, to additionally filter the set of traces with a 500 status code, select a row from the **USERS** table. This results in two filter tags in the trace set tile: `http.status`, as designated previously, and `user`.
## Compare two sets of traces with different criteria
Compare across various users and their POST requests to find other discrepancies and correlations. Apply your first set of filters. They are defined by a blue line in the response time distribution. Then select **Compare**. Initially, this creates a copy of the filters on trace set A.
To proceed, define a new set of filters to apply to trace set B. This second set is represented by a green line. The following example shows different lines according to the blue and green color scheme.
![\[Line graph comparison\]](http://docs.aws.amazon.com/xray/latest/devguide/images/analytics-compareLines.png)
## Identify a trace of interest and view its details
As you narrow your scope using the console filters, the trace list below the metrics tables becomes more meaningful. The trace list table combines information about **URL**, **USER**, and **STATUS CODE** into one view. For more insights, select a row from this table to open the trace's detail page and view its timeline and raw data.
# Configuring groups
Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics. You can use the AWS X-Ray console or X-Ray API to create and manage groups for your services. This topic describes how to create and manage groups by using the X-Ray console. For information about how to manage groups by using the X-Ray API, see [Groups](xray-api-configuration.md#xray-api-configuration-groups).
You can create groups of traces for trace maps, traces, or analytics. When you create a group, the group becomes available as a filter on the group dropdown menu on all three pages: **Trace Map**, **Traces**, and **Analytics**.
![\[Group menu\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-menu.png)
Groups are identified by their name or an Amazon Resource Name (ARN), and contain a filter expression. The service compares incoming traces to the expression and stores them accordingly. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md).
Updating a group's filter expression doesn't change data that's already recorded. The update applies only to subsequent traces. This can result in a merged graph of the new and old expressions. To avoid this, delete a current group and create a new one.
**Note**
Groups are billed by the number of retrieved traces that match the filter expression. For more information, see [AWS X-Ray pricing](https://aws.amazon.com/xray/pricing/).
**Topics**
+ [Create a group](#xray-console-group-create-console)
+ [Apply a group](#xray-console-group-apply)
+ [Edit a group](#xray-console-group-edit)
+ [Clone a group](#xray-console-group-clone)
+ [Delete a group](#xray-console-group-delete)
+ [View group metrics in Amazon CloudWatch](#xray-console-group-cloudwatch)
## Create a group
**Note**
You can now configure X-Ray groups from within the Amazon CloudWatch console. You can also continue to use the X-Ray console.
------
#### [ CloudWatch console ]
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. Choose **Settings** in the left navigation pane.
1. Choose **View settings** under **Groups** within the **X-Ray traces** section.
1. Choose **Create group** above the list of groups.
1. On the **Create group** page, enter a name for the group. A group name can have a maximum of 32 characters, and contain alphanumeric characters and dashes. Group names are case sensitive.
1. Enter a filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`. and requests to the service where the response time was greater than or equal to five seconds.
```
fault = true AND http.url CONTAINS "example/game" AND responsetime >= 5
```
1. In **Insights**, enable or disable insights access for the group. For more information about insights, see [Using X-Ray insights](xray-console-insights.md).
![\[Insights check boxes on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-insights-cw.png)
1. In **Tags**, choose **Add new tag** to enter a tag key, and optionally, a tag value. Continue to add additional tags as desired. Tag keys must be unique. To delete a tag, choose **Remove** underneath each tag. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md).
![\[Tag fields on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-tags-cw.png)
1. Choose **Create group**.
------
#### [ X-Ray console ]
1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home).
1. Open the **Create group** page from the **Groups** page in the left navigation pane, or from the group menu on one of the following pages: **Trace Map**, **Traces**, and **Analytics**.
1. On the **Create group** page, enter a name for the group. A group name can have a maximum of 32 characters, and contain alphanumeric characters and dashes. Group names are case sensitive.
1. Enter a filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`. and requests to the service where the response time was greater than or equal to five seconds.
```
fault = true AND http.url CONTAINS "example/game" AND responsetime >= 5
```
1. In **Insights**, enable or disable insights access for the group. For more information about insights, see [Using X-Ray insights](xray-console-insights.md).
![\[Insights check boxes on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-insights.png)
1. In **Tags**, enter a tag key, and optionally, a tag value. As you add a tag, a new line appears for you to enter another tag. Tag keys must be unique. To delete a tag, choose **X** at the end of the tag's row. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md).
![\[Tag fields on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-tags.png)
1. Choose **Create group**.
------
## Apply a group
------
#### [ CloudWatch console ]
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. Open one of the following pages from the navigation pane under **X-Ray traces**:
+ **Trace Map**
+ **Traces**
1. Enter a group name into the **Filter by X-Ray group** filter. The data shown on the page changes to match the filter expression set in the group.
------
#### [ X-Ray console ]
1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home).
1. Open one of the following pages from the navigation pane:
+ **Trace Map**
+ **Traces**
+ **Analytics**
1. On the group menu, choose the group that you created in [Create a group](#xray-console-group-create-console). The data shown on the page changes to match the filter expression set in the group.
------
## Edit a group
------
#### [ CloudWatch console ]
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. Choose **Settings** in the left navigation pane.
1. Choose **View settings** under **Groups** within the **X-Ray traces** section.
1. Choose a group from the **Groups** section and then choose **Edit**.
1. Although you can't rename a group, you can update the filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`, where the request URL address contains `example/game`, and response time for requests was greater than or equal to five seconds.
```
fault = true AND http.url CONTAINS "example/game" AND responsetime >= 5
```
1. In **Insights**, enable or disable insights access for the group. For more information about insights, see [Using X-Ray insights](xray-console-insights.md).
![\[Insights check boxes on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-insights-cw.png)
1. In **Tags**, choose **Add new tag** to enter a tag key, and optionally, a tag value. Continue to add additional tags as desired. Tag keys must be unique. To delete a tag, choose **Remove** underneath each tag. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md).
![\[Tag fields on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-tags-cw.png)
1. When you're finished updating the group, choose **Update group**.
------
#### [ X-Ray console ]
1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home).
1. Do one of the following to open the **Edit group** page.
1. On the **Groups** page, choose the name of a group to edit it.
1. On the group menu on one of the following pages, point to a group, and then choose **Edit**.
+ **Trace Map**
+ **Traces**
+ **Analytics**
1. Although you can't rename a group, you can update the filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`, where the request URL address contains `example/game`, and response time for requests was greater than or equal to five seconds.
```
fault = true AND http.url CONTAINS "example/game" AND responsetime >= 5
```
1. In **Insights**, enable or disable insights and insights notifications for the group. For more information about insights, see [Using X-Ray insights](xray-console-insights.md).
![\[Insights check boxes on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-insights.png)
1. In **Tags**, edit tag keys and values. Tag keys must be unique. Tag values are optional; you can delete values, if you want. To delete a tag, choose **X** at the end of the tag's row. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md).
![\[Tag fields on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-tags.png)
1. When you're finished updating the group, choose **Update group**.
------
## Clone a group
Cloning a group creates a new group that has the filter expression and tags of an existing group. When you clone a group, the new group has the same name as the group from which it's cloned, with `-clone` appended to the name.
------
#### [ CloudWatch console ]
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. Choose **Settings** in the left navigation pane.
1. Choose **View settings** under **Groups** within the **X-Ray traces** section.
1. Choose a group from the **Groups** section and then choose **Clone**.
1. On the **Create group** page, the name of the group is *group-name*`-clone`. Optionally, enter a new name for the group. A group name can have a maximum of 32 characters, and contain alphanumeric characters and dashes. Group names are case sensitive.
1. You can keep the filter expression from the existing group, or optionally, enter a new filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`. and requests to the service where the response time was greater than or equal to five seconds.
```
service("api.example.com") { fault = true OR responsetime >= 5 }
```
1. In **Tags**, edit tag keys and values, if needed. Tag keys must be unique. Tag values are optional; you can delete values if you want. To delete a tag, choose **X** at the end of the tag's row. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md).
1. Choose **Create group**.
------
#### [ X-Ray console ]
1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home).
1. Open the **Groups** page from the left navigation pane, and the choose the name of a group that you want to clone.
1. Choose **Clone group** from the **Actions** menu.
1. On the **Create group** page, the name of the group is *group-name*`-clone`. Optionally, enter a new name for the group. A group name can have a maximum of 32 characters, and contain alphanumeric characters and dashes. Group names are case sensitive.
1. You can keep the filter expression from the existing group, or optionally, enter a new filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`. and requests to the service where the response time was greater than or equal to five seconds.
```
service("api.example.com") { fault = true OR responsetime >= 5 }
```
1. In **Tags**, edit tag keys and values, if needed. Tag keys must be unique. Tag values are optional; you can delete values if you want. To delete a tag, choose **X** at the end of the tag's row. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md).
1. Choose **Create group**.
------
## Delete a group
Follow steps in this section to delete a group. You can't delete the **Default** group.
------
#### [ CloudWatch console ]
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. Choose **Settings** in the left navigation pane.
1. Choose **View settings** under **Groups** within the **X-Ray traces** section.
1. Choose a group from the **Groups** section and then choose **Delete**.
1. When you're prompted to confirm, choose **Delete**.
------
#### [ X-Ray console ]
1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home).
1. Open the **Groups** page from the left navigation pane, and the choose the name of a group that you want to delete.
1. On the **Actions** menu, choose **Delete group**.
1. When you're prompted to confirm, choose **Delete**.
------
## View group metrics in Amazon CloudWatch
After a group is created, incoming traces are checked against the group’s filter expression as they're stored in the X-Ray service. Metrics for the number of traces matching each criteria are published to Amazon CloudWatch every minute. Choosing **View metric** on the **Edit group** page opens the CloudWatch console to the **Metric** page. For more information about how to use CloudWatch metrics, see [Using Amazon CloudWatch Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html) in the *Amazon CloudWatch User Guide*.
------
#### [ CloudWatch console ]
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. Choose **Settings** in the left navigation pane.
1. Choose **View settings** under **Groups** within the **X-Ray traces** section.
1. Choose a group from the **Groups** section and then choose **Edit**.
1. On the **Edit group** page, choose **View metric**.
The CloudWatch console **Metrics** page opens in a new tab.
------
#### [ X-Ray console ]
1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home).
1. Open the **Groups** page from the left navigation pane, and the choose the name of a group that you want to view metrics for.
1. On the **Edit group** page, choose **View metric**.
The CloudWatch console **Metrics** page opens in a new tab.
------
# Configuring sampling rules
You can use the AWS X-Ray console to configure sampling rules for your services. The AWS Distro for OpenTelemetry, X-Ray SDK, and AWS services that support [active tracing](xray-services.md) with sampling configurations use sampling rules to determine which requests to record.
**Topics**
+ [Configuring sampling rules](#xray-console-config)
+ [Customizing sampling rules](#xray-console-custom)
+ [Sampling rule options](#xray-console-sampling-options)
+ [Sampling rule examples](#xray-console-sampling-examples)
+ [Configuring your service to use sampling rules](#xray-console-sampling-service)
+ [Viewing sampling results](#xray-console-sampling-results)
+ [Next steps](#xray-console-sampling-nextsteps)
## Configuring sampling rules
You can configure sampling for the following use cases:
+ **API Gateway Entrypoint** – API Gateway supports sampling and active tracing. To enable active tracing on an API stage, see [Amazon API Gateway active tracing support for AWS X-Ray](xray-services-apigateway.md).
+ **AWS AppSync** – AWS AppSync supports sampling and active tracing. To enable active tracing on AWS AppSync requests, see [Tracing with AWS X-Ray](https://docs.aws.amazon.com/appsync/latest/devguide/x-ray-tracing.html).
+ **AWS Step Functions** – AWS Step Functions supports sampling and active tracing. To enable active tracing on AWS Step Functions state machines, see [X-Ray tracing in Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-xray-tracing.html).
+ **Instrument AWS Distro for OpenTelemetry on compute platforms** – When using compute platforms such as Amazon EC2, Amazon ECS, or AWS Elastic Beanstalk, sampling is supported when the application has been instrumented with the latest AWS Distro for OpenTelemetry or X-Ray SDK.
## Customizing sampling rules
By customizing sampling rules, you can control the amount of data that you record. You can also modify sampling behavior without modifying or redeploying your code. Sampling rules tell the AWS Distro for OpenTelemetry (ADOT) or X-Ray SDK how many requests to record for a set of criteria. By default, the SDK records the first request each second, and five percent of any additional requests. One request per second is the *reservoir*. This ensures that at least one trace is recorded each second as long as the service is serving requests. Five percent is the *rate* at which additional requests beyond the reservoir size are sampled.
You can configure the X-Ray SDK to read sampling rules from a JSON document that you include with your code. However, when you run multiple instances of your service, each instance performs sampling independently. This causes the overall percentage of requests sampled to increase because the reservoirs of all of the instances are effectively added together. Additionally, to update local sampling rules, you must redeploy your code.
By defining sampling rules in the X-Ray console, and [configuring the SDK](#xray-console-sampling-service) to read rules from the X-Ray service, you can avoid both of these issues. The service manages the reservoir for each rule, and assigns quotas to each instance of your service to distribute the reservoir evenly, based on the number of instances that are running. The reservoir limit is calculated according to the rules you set. Because the rules are configured in the service, you can manage rules without making additional deployments.
**Note**
When configuring sampling rules, it is critical to understand that X-Ray sampling is "Parent-based." This means the sampling decision is made only once, typically by the first X-Ray-enabled service that handles the request (the "Root" service).
If a downstream service receives a request that already has a sampling decision from an upstream parent, it will honor that decision regardless of any of its own matching sampling rules.
When Rules Apply: Your custom sampling rules only take effect on services where a sampling decision has not yet been made. This usually applies to:
The entry point of your application (e.g., an API Gateway, a Load Balancer, or the first instrumented Microservice).
Asynchronous processes or workers that start a brand new trace.
Common Pitfall: If you create a strict sampling rule for "Service B," but "Service B" is always called by "Service A," your rule for Service B will likely never be applied because it is simply following the decision passed down by Service A. To change the sampling of traces for this workflow, you must configure the sampling rule to the Root service (Service A).
**Note**
X-Ray uses a best-effort approach in applying sampling rules, and in some cases the effective sampling rate may not exactly match the configured sampling rules. However, over time the number of requests sampled should be close to the configured percentage.
You can now configure X-Ray sampling rules from within the Amazon CloudWatch console.
**To configure sampling rules in the CloudWatch console**
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. Choose **Settings** in the left navigation pane underneath **Setup**.
1. Choose **View settings** under **Sampling rules** within the **X-Ray traces** section.
1. To create a rule, choose **Create sampling rule**.
To edit a rule, choose a rule and choose **Edit** to edit it.
To delete a rule, choose a rule and choose **Delete** to delete it.
## Sampling rule options
The following options are available for each rule. String values can use wildcards to match a single character (`?`) or zero or more characters (`*`).
**Sampling rule options**
+ **Rule name** (string) – A unique name for the rule.
+ **Priority** (integer between 1 and 9999) – The priority of the sampling rule. Services evaluate rules in ascending order of priority, and make a sampling decision with the first rule that matches.
+ **Reservoir** (non-negative integer) – A fixed number of matching requests to instrument per second, before applying the fixed rate. The reservoir is not used directly by services, but applies to all services using the rule collectively.
+ **Rate** (integer between 0 and 100) – The percentage of matching requests to instrument, after the reservoir is exhausted. When configuring a sampling rule in the console, choose a percentage between 0 and 100. When configuring a sampling rule in a client SDK using a JSON document, provide a percentage value between 0 and 1.
+ **Service name** (string) – The name of the instrumented service, as it appears in the trace map.
+ X-Ray SDK – The service name that you configure on the recorder.
+ Amazon API Gateway – `api-name/stage`.
+ **Service type** (string) – The service type, as it appears in the trace map. For the X-Ray SDK, set the service type by applying the appropriate plugin:
+ `AWS::ElasticBeanstalk::Environment` – An AWS Elastic Beanstalk environment (plugin).
+ `AWS::EC2::Instance` – An Amazon EC2 instance (plugin).
+ `AWS::ECS::Container` – An Amazon ECS container (plugin).
+ `AWS::EKS::Container` – An Amazon EKS container (plugin).
+ `AWS::APIGateway::Stage` – An Amazon API Gateway stage.
+ `AWS::AppSync::GraphQLAPI ` – An AWS AppSync API request.
+ `AWS::StepFunctions::StateMachine` – An AWS Step Functions state machine.
+ **Host** (string) – The hostname from the HTTP host header.
+ **HTTP method** (string) – The method of the HTTP request.
+ **URL path** (string) – The URL path of the request.
+ X-Ray SDK – The path portion of the HTTP request URL.
+ **Resource ARN** (string) – The ARN of the AWS resource running the service.
+ X-Ray SDK – Not supported. The SDK can only use rules with **Resource ARN** set to `*`.
+ Amazon API Gateway – The stage ARN.
+ (Optional) **Attributes** (key and value) – Segment attributes that are known when the sampling decision is made.
+ X-Ray SDK – Not supported. The SDK ignores rules that specify attributes.
+ Amazon API Gateway – Headers from the original HTTP request.
+ (Optional) **SamplingRateBoost** (object) – Controls anomaly-driven sampling boost behavior.
+ MaxRate (integer between 0 and 100) – Maximum sampling rate (0.0–1.0) X-Ray may increase to during anomalies
+ CooldownWindowMinutes (integer greater than 0) – Time window (minutes) in which only one boost can be triggered, preventing continuous boosts
## Sampling rule examples
**Example – Default rule with no reservoir and a low rate**
You can modify the default rule's reservoir and rate. The default rule applies to requests that don't match any other rule.
+ **Reservoir**: **0**
+ **Rate**: **5** (**0.05** if configured using a JSON document)
**Example – Debugging rule to trace all requests for a problematic route**
A high-priority rule applied temporarily for debugging.
+ **Rule name**: **DEBUG – history updates**
+ **Priority**: **1**
+ **Reservoir**: **1**
+ **Rate**: **100** (**1** if configured using a JSON document)
+ **Service name**: **Scorekeep**
+ **Service type**: **\$1**
+ **Host**: **\$1**
+ **HTTP method**: **PUT**
+ **URL path**: **/history/\$1**
+ **Resource ARN**: **\$1**
**Example – Higher minimum rate for POSTs**
+ **Rule name**: **POST minimum**
+ **Priority**: **100**
+ **Reservoir**: **10**
+ **Rate**: **10** (**.1** if configured using a JSON document)
+ **Service name**: **\$1**
+ **Service type**: **\$1**
+ **Host**: **\$1**
+ **HTTP method**: **POST**
+ **URL path**: **\$1**
+ **Resource ARN**: **\$1**
**Example Enable anomaly-driven boost**
Configure a rule that triggers a sampling boost up to 50% during anomalies, with a 10-minute cooldown.
+ **Rule name**: **MyAdaptiveRule**
+ **Priority**: **100**
+ **Reservoir**: **1**
+ **FixedRate**: **0.0510**
+ **Service name**: **\$1**
+ **Service type**: **\$1**
+ **Host**: **\$1**
+ **HTTP method**: **POST**
+ **URL path**: **\$1**
+ **maxRate**: **0.5**
+ **cooldownWindowMinutes**: **10**
## Configuring your service to use sampling rules
The AWS Distro for OpenTelemetry (ADOT) and X-Ray SDK requires additional configuration to use sampling rules that you configure in the console. See the configuration topic for your language for details on configuring a sampling strategy:
+ Java: [Using X-Ray Remote Sampling](https://aws-otel.github.io/docs/getting-started/java-sdk/auto-instr#using-x-ray-remote-sampling) with ADOT Java
+ Go: [Configuring sampling](https://aws-otel.github.io/docs/getting-started/go-sdk/manual-instr#configuring-sampling) with ADOT Go
+ Node.js: [Using X-Ray Remote Sampling](https://aws-otel.github.io/docs/getting-started/js-sdk/trace-metric-auto-instr#using-x-ray-remote-sampling) with ADOT JavaScript
+ Python: [Using X-Ray Remote Sampling](https://aws-otel.github.io/docs/getting-started/python-sdk/auto-instr#using-x-ray-remote-sampling) with ADOT Python
+ Ruby: [Sampling rules](xray-sdk-ruby-configuration.md#xray-sdk-ruby-configuration-sampling)
+ .NET: [Using X-Ray Remote Sampling](https://aws-otel.github.io/docs/getting-started/dotnet-sdk/auto-instr#using-x-ray-remote-sampling) with ADOT .NET
For API Gateway, see [Amazon API Gateway active tracing support for AWS X-Ray](xray-services-apigateway.md).
## Viewing sampling results
The X-Ray console **Sampling** page shows detailed information about how your services use each sampling rule.
The **Trend** column shows how the rule has been used in the last few minutes. Each column shows statistics for a 10-second window.
**Sampling statistics**
+ **Total matched rule**: The number of requests that matched this rule. This number doesn't include requests that could have matched this rule, but matched a higher-priority rule first.
+ **Total sampled**: The number of requests recorded.
+ **Sampled with fixed rate**: The number of requests sampled by applying the rule's fixed rate.
+ **Sampled with reservoir limit**: The number of requests sampled using a quota assigned by X-Ray.
+ **Borrowed from reservoir**: The number of requests sampled by borrowing from the reservoir. The first time a service matches a request to a rule, it has not yet been assigned a quota by X-Ray. However, if the reservoir is at least 1, the service borrows one trace per second until X-Ray assigns a quota.
For more information about sampling statistics and how services use sampling rules, see [Using sampling rules with the X-Ray API](xray-api-sampling.md).
## Next steps
You can use the X-Ray API to manage sampling rules. With the API, you can create and update rules programmatically on a schedule, or in response to alarms or notifications. See [Configuring sampling, groups, and encryption settings with the AWS X-Ray API](xray-api-configuration.md) for instructions and additional rule examples.
The AWS Distro for OpenTelemetry, X-Ray SDK and AWS services also use the X-Ray API to read sampling rules, report sampling results, and get sampling targets. Services must keep track of how often they apply each rule, evaluate rules based on priority, and borrow from the reservoir when a request matches a rule for which X-Ray has not yet assigned the service a quota. For more detail about how a service uses the API for sampling, see [Using sampling rules with the X-Ray API](xray-api-sampling.md).
When the AWS Distro for OpenTelemetry or the X-Ray SDK call sampling APIs, they use the CloudWatch agent as a proxy. If you already use TCP port 2000, you can configure the agent to run the proxy on a different port. See the [CloudWatch agent installation guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) for details.
# Configuring adaptive sampling
Missing critical traces during anomaly spikes can make root cause analysis difficult. However, maintaining high sampling rates is expensive. X-Ray adaptive sampling provides complete visibility into anomalies and controls cost during normal operations. With adaptive sampling, you set a maximum sampling rate, and X-Ray automatically adjusts within that limit. X-Ray calculates the minimum boost needed to capture error traces. If your baseline rate captures enough data, no boost occurs. You only pay for extra sampling when needed.
Benefits of using adaptive sampling:
+ Complete incident visibility – Get full traces during incidents without manual intervention. X-Ray automatically adjusts sampling rates to capture error traces, then returns to normal rates.
+ Root cause visibility – Always see the source of problems. X-Ray captures critical error data even when full trace sampling isn't triggered.
+ Optimize costs – Brief sampling boosts (up to 1 minute) and automatic cooldown periods prevent oversampling. You pay only for the data you need to diagnose issues.
**Topics**
+ [Supported SDKs and platforms](#adaptive-sampling-supported-sdks)
+ [Choose your adaptive sampling approach](#adaptive-sampling-features)
+ [Local SDK configuration](#local-sdk-configuration)
## Supported SDKs and platforms
**Supported SDK** – Adaptive sampling requires the latest version of the ADOT SDK.
**Supported language** – Java (version [v2.11.5](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/tag/v2.11.5) or higher)
Your application must be instrumented with the supported ADOT SDK and executed together with either the Amazon CloudWatch Agent or the OpenTelemetry Collector.
For example, Amazon EC2, Amazon ECS, and Amazon EKS are common platforms where AWS Application Signals provides guidance for enabling the ADOT SDK and Amazon CloudWatch Agent.
## Choose your adaptive sampling approach
Adaptive sampling supports two approaches, Sampling Boost and Anomaly Span Capture. These can be applied independently or can be combined together.
### Sampling boost
Adaptive sampling boost is based on sampling rules and works with the existing X-Ray head-based sampling model. Head-based sampling means that sampling decisions are made at the root service, and the sampling flag is passed downstream to all services in the call chain.
+ **Rule-based boosting** – Boosting is always tied to a specific X-Ray sampling rule. Each rule can define its own maximum boost rate and cool down behavior.
+ **Head-based sampling** – Sampling decisions are made at the root service, and the sampling flag is passed downstream to all services in the call chain.
+ **Anomaly-driven** – X-Ray relies on the SDK to report anomaly statistics. When X-Ray detects anomalies such as errors or high latency, it uses these statistics to calculate an appropriate boost rate (up to the configured maximum).
**Anomaly reporting**
Every application service in the call chain can emit anomaly statistics through the required SDK:
+ **Root service** – Must run on a supported SDK and platform to enable sampling boost. If the root service is not supported, no boost will occur.
+ **Downstream services** – Downstream services only report anomalies; they cannot make sampling decisions. When a downstream service is running a supported SDK, anomalies detected can trigger a sampling boost. When a downstream service is unsupported (for example, running an older SDK) , anomalies on that service will not trigger a boost. These services can still propagate the context downstream when they follow standard context propagation (such as W3C trace context and baggage). This ensures that supported SDKs in further downstream services can report anomalies that trigger a boost.
**Boost timing and scope**
+ **Trigger delay** – You can expect a sampling boost to begin as low as 10 seconds after X-Ray detects an anomaly.
+ **Boost period** – After X-Ray triggers a boost, it lasts up to 1 minute before returning back to the base sampling rate.
+ **Boost cool down** – After a boost occurs, X-Ray will not trigger another boost for the same rule until the cool down window has passed.
For example, when you set `cooldown` to 10 minutes, once a boost ends, no new boost can be triggered until the next 10 minutes window.
Special case: when you set `cooldown` to 1 minute, and since a boost itself can last up to 1 minute, boosts can effectively be triggered continuously if anomaly persist.
**Note**
Use supported SDKs and platforms for your root service. Sampling boost works only with supported SDKs and platforms. While sampling boost has a high probability of capturing anomaly traces, it may not capture every anomaly trace.
**Boost visibility**
When a sampling rule is configured with adaptive sampling boost, X-Ray automatically emits vended metrics that allow you to monitor boost activity.
+ **Metric name** – `SamplingRate`
+ **Dimension** – `RuleName` (set to the actual rule name)
Each rule with `SamplingRateBoost` enabled will publish its effective sampling rate, including both the baseline rate and any temporary boosts. This allows you to:
+ Track when boosts are triggered
+ Monitor the effective sampling rate for each rule
+ Correlate boosts with application anomalies (such as error spikes or latency events)
You can view these metrics in **Amazon CloudWatch Metrics, under AWS/X-Ray namespace. The metric value is a floating-point number between 0 and 1, representing the effective sampling rate**.
**Configure sampling boost using X-Ray sampling rules**
You can enable adaptive sampling directly in your existing X-Ray sampling rules by adding a new `SamplingRateBoost` field. For more information, see [Customizing sampling rules](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html#xray-console-custom). This provides a centralized way to enable adaptive sampling without modifying application code or applying application deployment. When you enable adaptive sampling, X-Ray automatically increases sampling during anomalies such as error spikes or latency outliers, while keeping sampling rates within your configured maximum. `SamplingRateBoost` can be applied to any custom sampling rule except the `Default` sampling rule.
The `SamplingRateBoost` field defines the upper bound and behavior for anomaly-driven sampling.
```
"SamplingRateBoost": {
"MaxRate": 0.25,
"CooldownWindowMinutes": 10
}
```
The `MaxRate` defines the maximum sampling rate X-Ray will apply when it detects anomalies. Value range is `0.0` to `1.0`. For example, `"MaxRate": 0.25` allows sampling to increase up to 25% of requests during an anomaly window. X-Ray determines the appropriate rate between your baseline and the maximum, depending on anomaly activity.
The `CooldownWindowMinutes` defines time window (in minutes) in which only one sampling rate boost can be triggered. After a boost occurs, no further boosts are allowed until the next window. The Value type is *integer (minutes)*.
**Example rule with adaptive sampling**
```
{
"RuleName": "MyAdaptiveRule",
"Priority": 1,
"ReservoirSize": 1,
"FixedRate": 0.05,
"ServiceName": "*",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"SamplingRateBoost": {
"MaxRate": 0.25,
"CooldownWindowMinutes": 10
}
}
```
In this example, baseline sampling is 5% (`FixedRate: 0.05`). During anomalies, X-Ray can increase sampling up to 25% (`MaxRate: 0.25`). Boost only once every 10 minutes.
**Anomaly condition configuration**
When no anomaly condition configuration is provided, the ADOT SDK uses **HTTP 5xx error codes** as the default anomaly condition to trigger sampling boost.
You can also fine tune anomaly conditions locally in the supported ADOT SDK using environment variables. For more information, see [Local SDK configuration](#local-sdk-configuration).
### Anomaly spans capture
Anomaly span capture ensures that critical spans representing anomalies are always recorded, even if the full trace is not sampled. This feature complements sampling boost by focusing on capturing the anomaly itself, rather than increasing sampling for future traces.
When the ADOT SDK detects an anomaly, it emits that span immediately, regardless of the sampling decision. Since the SDK emits only spans related to the anomaly, these traces are partial traces, not full end-to-end transactions.
Once the ADOT SDK detects an anomaly span, it attempts to emit as many spans from the same trace as possible. All spans emitted under this feature are tagged with the attribute, `aws.trace.flag.sampled = 0`. This allows you to easily distinguish partial traces (anomaly capture) from complete traces (normal sampling) in transaction search and analysis.
We recommend onboarding [Transaction Search](https://docs.aws.amazon.com/xray/latest/devguide/xray-transactionsearch.html) to view and query partial traces. The following example shows a Service page in [Application Signals](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html) console. ServiceC is configured with anomaly span capture, and it is part of a call chain where sampling boost applies. This configurations generates both complete and partial traces. You can use the `aws.trace.flag.sampled` attribute to distinguish between trace types.
![\[Anomaly spans capture\]](http://docs.aws.amazon.com/xray/latest/devguide/images/adaptive-sampling.png)
Anomaly spans capture can only be enabled or customized through the [Local SDK configuration](#local-sdk-configuration).
## Local SDK configuration
You can configure adaptive sampling features in the ADOT SDK by providing a YAML configuration through an environment variable. Local configuration provides fine-grained control over anomaly conditions, thresholds.
This is required for *anomaly span capture* and optional for customizing *sampling boost* conditions. The following is an example of the configuration:
```
version: 1.0
anomalyConditions:
- errorCodeRegex: "^5\\d\\d$"
usage: both
- operations:
- "/api"
errorCodeRegex: "^429|5\\d\\d$"
highLatencyMs: 300
usage: sampling-boost
- highLatencyMs: 1000
usage: anomaly-span-capture
anomalyCaptureLimit:
anomalyTracesPerSecond: 1
```
Field definitions are below:
+ `version` – Schema version for the configuration file
+ `anomalyConditions` – Defines the conditions under which anomalies are detected and how they are used
+ `errorCodeRegex` – Regular expression defining which HTTP status codes are considered anomalies
+ `operations` – List of operations or endpoints to which the condition applies
+ `highLatencyMs` – Latency threshold (in milliseconds) above which spans are treated as anomalies
+ `usage` – Defines which feature the condition applies to:
+ `both` – Applies to **sampling boost** and **anomaly span capture** (Default if usage is not specified)
+ `sampling-boost` – Used only for triggering sampling boosts
+ `anomaly-span-capture` – Used only for anomaly span capture
+ `anomalyCaptureLimit` – Defines limits on how many traces with anomaly spans are emitted.
`anomalyTracesPerSecond` – Maximum number of traces with anomaly spans captured per second, to prevent excessive span volume (Default value is 1 if anomalyCaptureLimit is not present).
**Note**
`AnomalyConditions` **overrides** the default anomaly condition for sampling boost (HTTP 5xx). If you want to retain the default condition while using local configuration, you must explicitly include it in any item of `AnomalyConditions`.
For each `anomalyConditions` item:
When the `operations` field is **omitted**, the condition applies to **all operations** (service level)
When the `operations` field is present but set to an **empty list**, the condition applies to **no operations**, making that item a no-op
When both `errorCodeRegex` and `highLatencyMs` are omitted, the condition has no anomaly criteria to evaluate, making that item a no-op
Logical relationships:
Between items in `anomalyConditions`, the relationship is **OR**.
Within a single item, multiple fields (for example, `errorCodeRegex` and `highLatencyMs`) are combined with **AND**.
For example:
```
errorCodeRegex: "^429|5\\d\\d$"
highLatencyMs: 300
```
This condition means, **status code matches 429 or 5xx AND latency ≥ 300 ms**.
### Apply the Local Configuration to ADOT SDK
You can apply the local configuration to the ADOT SDK by setting the environment variable `AWS_XRAY_ADAPTIVE_SAMPLING_CONFIG`. The value must be a valid YAML document (inline or nested).
For example, Amazon EC2 and Amazon ECS, set the environment variable directly:
```
AWS_XRAY_ADAPTIVE_SAMPLING_CONFIG="{version: 1.0, anomalyConditions: [{errorCodeRegex: \"^500$\", usage: \"sampling-boost\"}, {errorCodeRegex: \"^501$\", usage: \"anomaly-trace-capture\"}], anomalyCaptureLimit: {anomalyTracesPerSecond: 10}}"
```
For Amazon EKS, define the environment variable inside the pod spec as nested YAML:
```
apiVersion: v1
kind: Pod
metadata:
name: adot-sample
spec:
containers:
- name: adot-app
image: my-app:latest
env:
- name: AWS_XRAY_ADAPTIVE_SAMPLING_CONFIG
value: |
version: 1.0
anomalyConditions:
- errorCodeRegex: "^500$"
usage: sampling-boost
- errorCodeRegex: "^501$"
usage: anomaly-trace-capture
anomalyCaptureLimit:
anomalyTracesPerSecond: 10
```
# Console deep linking
You can use routes and queries to deep link into specific traces, or filtered views of traces and the trace map.
**Console pages**
+ Welcome page – [xray/home\$1/welcome](https://console.aws.amazon.com/xray/home#/welcome)
+ Getting started – [xray/home\$1/getting-started](https://console.aws.amazon.com/xray/home#/getting-started)
+ Trace map – [xray/home\$1/service-map](https://console.aws.amazon.com/xray/home#/service-map)
+ Traces – [xray/home\$1/traces](https://console.aws.amazon.com/xray/home#/traces)
## Traces
You can generate links for timeline, raw, and map views of individual traces.
**Trace timeline** – `xray/home#/traces/trace-id`
**Raw trace data** – `xray/home#/traces/trace-id/raw`
**Example – raw trace data**
```
https://console.aws.amazon.com/xray/home#/traces/1-57f5498f-d91047849216d0f2ea3b6442/raw
```
## Filter expressions
Link to a filtered list of traces.
**Filtered traces view** – `xray/home#/traces?filter=filter-expression`
**Example – filter expression**
```
https://console.aws.amazon.com/xray/home#/traces?filter=service("api.amazon.com") { fault = true OR responsetime > 2.5 } AND annotation.foo = "bar"
```
**Example – filter expression (URL encoded)**
```
https://console.aws.amazon.com/xray/home#/traces?filter=service(%22api.amazon.com%22)%20%7B%20fault%20%3D%20true%20OR%20responsetime%20%3E%202.5%20%7D%20AND%20annotation.foo%20%3D%20%22bar%22
```
For more information about filter expressions, see [Using filter expressions](xray-console-filters.md).
## Time range
Specify a length of time or start and end time in ISO8601 format. Time ranges are in UTC and can be up to 6 hours long.
**Length of time** – `xray/home#/page?timeRange=range-in-minutes`
**Example – trace map for the last hour**
```
https://console.aws.amazon.com/xray/home#/service-map?timeRange=PT1H
```
**Start and end time** – `xray/home#/page?timeRange=start~end`
**Example – time range accurate to seconds**
```
https://console.aws.amazon.com/xray/home#/traces?timeRange=2023-7-01T16:00:00~2023-7-01T22:00:00
```
**Example – time range accurate to minutes**
```
https://console.aws.amazon.com/xray/home#/traces?timeRange=2023-7-01T16:00~2023-7-01T22:00
```
## Region
Specify an AWS Region to link to pages in that Region. If you don't specify a Region, the console redirects you to the last visited Region.
**Region** – `xray/home?region=region#/page`
**Example – trace map in US West (Oregon) (us-west-2)**
```
https://console.aws.amazon.com/xray/home?region=us-west-2#/service-map
```
When you include a Region with other query parameters, the Region query goes before the hash, and the X-Ray-specific queries go after the page name.
**Example – trace map for the last hour in US West (Oregon) (us-west-2)**
```
https://console.aws.amazon.com/xray/home?region=us-west-2#/service-map?timeRange=PT1H
```
## Combined
**Example – recent traces with a duration filter**
```
https://console.aws.amazon.com/xray/home#/traces?timeRange=PT15M&filter=duration%20%3E%3D%205%20AND%20duration%20%3C%3D%208
```
**Output**
+ Page – Traces
+ Time Range – Last 15 minutes
+ Filter – duration >= 5 AND duration <= 8
# Use the X-Ray API
If the X-Ray SDK doesn’t support your programming language, you can use either the X-Ray APIs directly or the AWS Command Line Interface (AWS CLI) to call X-Ray API commands. Use the following guidance to choose how you interact with the API:
+ Use the AWS CLI for simpler syntax using pre-formatted commands or with options inside your request.
+ Use the X-Ray API directly for maximum flexibility and customization for requests that you make to X-Ray.
If you use the [X-Ray API](https://docs.aws.amazon.com/xray/latest/api/Welcome.html) directly instead of the AWS CLI, you must parametrize your request in the correct data format and may also have to configure authentication and error handling.
The following diagram shows guidance to choose how to interact with the X-Ray API:
![\[X-Ray displays detailed information about application requests.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/api-vs-cli.png)
Use the X-Ray API to send trace data to directly to X-Ray. The X-Ray API supports all functions available in the X-Ray SDK including the following common actions:
+ [PutTraceSegments](https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html) – Uploads segment documents to X-Ray.
+ [BatchGetTraces](https://docs.aws.amazon.com/xray/latest/api/API_BatchGetTraces.html) – Retrieves a list of traces in a list of trace IDs. Each retrieved trace is a collection of segment documents from a single request.
+ [GetTraceSummaries](https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSummaries.html) – Retrieves IDs and annotations for traces. You can specify a `FilterExpression` to retrieve a subset of trace summaries.
+ [GetTraceGraph](https://docs.aws.amazon.com/xray/latest/api/API_GetTraceGraph.html) – Retrieves a service graph for a specific trace ID.
+ [GetServiceGraph](https://docs.aws.amazon.com/xray/latest/api/API_GetServiceGraph.html) – Retrieves a JSON formatted document that describes services that process incoming requests and call downstream requests.
You can also use the AWS Command Line Interface (AWS CLI) inside your application code to programmatically interact with X-Ray. The AWS CLI supports all functions available in the X-Ray SDK including those for other AWS services. The following functions are versions of the API operations listed previously with a simpler format:
+ [put-trace-segments](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/xray/put-trace-segments.html) – Uploads segment documents to X-Ray.
+ [batch-get-traces](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/xray/batch-get-traces.html) – Retrieves a list of traces in a list of trace IDs. Each retrieved trace is a collection of segment documents from a single request.
+ [get-trace-summaries](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/xray/get-trace-summaries.html) – Retrieves IDs and annotations for traces. You can specify a `FilterExpression` to retrieve a subset of trace summaries.
+ [get-trace-graph](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/xray/get-trace-graph.html) – Retrieves a service graph for a specific trace ID.
+ [get-service-graph](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/xray/get-service-graph.html) – Retrieves a `JSON` formatted document that describes services that process incoming requests and call downstream requests.
To get started, you must install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) for your operating system. AWS supports Linux, macOS and Windows operating systems. For more information about the list of X-Ray commands, see the [AWS CLI Command Reference guide for X-Ray](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/xray/index.html).
**Topics**
+ [Using the AWS X-Ray API with the AWS CLI](xray-api-tutorial.md)
+ [Sending trace data to AWS X-Ray](xray-api-sendingdata.md)
+ [Getting data from AWS X-Ray](xray-api-gettingdata.md)
+ [Configuring sampling, groups, and encryption settings with the AWS X-Ray API](xray-api-configuration.md)
+ [Using sampling rules with the X-Ray API](xray-api-sampling.md)
+ [AWS X-Ray segment documents](xray-api-segmentdocuments.md)
# Using the AWS X-Ray API with the AWS CLI
The AWS CLI lets your access the X-Ray service directly and use the same APIs that the X-Ray console uses to retrieve the service graph and raw traces data. The sample application includes scripts that show how to use these APIs with the AWS CLI.
## Prerequisites
This tutorial uses the Scorekeep sample application and included scripts to generate tracing data and a service map. Follow the instructions in the [getting started tutorial](xray-gettingstarted.md) to launch the application.
This tutorial uses the AWS CLI to show basic use of the X-Ray API. The AWS CLI, [available for Windows, Linux, and OS-X](https://docs.aws.amazon.com/cli/latest/userguide/installing.html), provides command line access to the public APIs for all AWS services.
**Note**
You must verify that your AWS CLI is configured to the same Region that your Scorekeep sample application was created in.
Scripts included to test the sample application uses `cURL` to send traffic to the API and `jq` to parse the output. You can download the `jq` executable from [stedolan.github.io](https://stedolan.github.io/jq/), and the `curl` executable from [https://curl.haxx.se/download.html](https://curl.haxx.se/download.html). Most Linux and OS X installations include cURL.
## Generate trace data
The web app continues to generate traffic to the API every few seconds while the game is in-progress, but only generates one type of request. Use the `test-api.sh` script to run end to end scenarios and generate more diverse trace data while you test the API.
**To use the `test-api.sh` script**
1. Open the [Elastic Beanstalk console](https://console.aws.amazon.com/elasticbeanstalk).
1. Navigate to the [management console](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-console.html) for your environment.
1. Copy the environment **URL** from the page header.
1. Open `bin/test-api.sh` and replace the value for API with your environment's URL.
```
#!/bin/bash
API=scorekeep.9hbtbm23t2.us-west-2.elasticbeanstalk.com/api
```
1. Run the script to generate traffic to the API.
```
~/debugger-tutorial$ ./bin/test-api.sh
Creating users,
session,
game,
configuring game,
playing game,
ending game,
game complete.
{"id":"MTBP8BAS","session":"HUF6IT64","name":"tic-tac-toe-test","users":["QFF3HBGM","KL6JR98D"],"rules":"102","startTime":1476314241,"endTime":1476314245,"states":["JQVLEOM2","D67QLPIC","VF9BM9NC","OEAA6GK9","2A705O73","1U2LFTLJ","HUKIDD70","BAN1C8FI","G3UDJTUF","AB70HVEV"],"moves":["BS8F8LQ","4MTTSPKP","463OETES","SVEBCL3N","N7CQ1GHP","O84ONEPD","EG4BPROQ","V4BLIDJ3","9RL3NPMV"]}
```
## Use the X-Ray API
The AWS CLI provides commands for all of the API actions that X-Ray provides, including [https://docs.aws.amazon.com/xray/latest/api/API_GetServiceGraph.html](https://docs.aws.amazon.com/xray/latest/api/API_GetServiceGraph.html) and [https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSummaries.html](https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSummaries.html). See the [AWS X-Ray API Reference](https://docs.aws.amazon.com/xray/latest/api/Welcome.html) for more information on all of the supported actions and the data types that they use.
**Example bin/service-graph.sh**
```
EPOCH=$(date +%s)
aws xray get-service-graph --start-time $(($EPOCH-600)) --end-time $EPOCH
```
The script retrieves a service graph for the last 10 minutes.
```
~/eb-java-scorekeep$ ./bin/service-graph.sh | less
{
"StartTime": 1479068648.0,
"Services": [
{
"StartTime": 1479068648.0,
"ReferenceId": 0,
"State": "unknown",
"EndTime": 1479068651.0,
"Type": "client",
"Edges": [
{
"StartTime": 1479068648.0,
"ReferenceId": 1,
"SummaryStatistics": {
"ErrorStatistics": {
"ThrottleCount": 0,
"TotalCount": 0,
"OtherCount": 0
},
"FaultStatistics": {
"TotalCount": 0,
"OtherCount": 0
},
"TotalCount": 2,
"OkCount": 2,
"TotalResponseTime": 0.054000139236450195
},
"EndTime": 1479068651.0,
"Aliases": []
}
]
},
{
"StartTime": 1479068648.0,
"Names": [
"scorekeep.elasticbeanstalk.com"
],
"ReferenceId": 1,
"State": "active",
"EndTime": 1479068651.0,
"Root": true,
"Name": "scorekeep.elasticbeanstalk.com",
...
```
**Example bin/trace-urls.sh**
```
EPOCH=$(date +%s)
aws xray get-trace-summaries --start-time $(($EPOCH-120)) --end-time $(($EPOCH-60)) --query 'TraceSummaries[*].Http.HttpURL'
```
The script retrieves the URLs of traces generated between one and two minutes ago.
```
~/eb-java-scorekeep$ ./bin/trace-urls.sh
[
"http://scorekeep.elasticbeanstalk.com/api/game/6Q0UE1DG/5FGLM9U3/endtime/1479069438",
"http://scorekeep.elasticbeanstalk.com/api/session/KH4341QH",
"http://scorekeep.elasticbeanstalk.com/api/game/GLQBJ3K5/153AHDIA",
"http://scorekeep.elasticbeanstalk.com/api/game/VPDL672J/G2V41HM6/endtime/1479069466"
]
```
**Example bin/full-traces.sh**
```
EPOCH=$(date +%s)
TRACEIDS=$(aws xray get-trace-summaries --start-time $(($EPOCH-120)) --end-time $(($EPOCH-60)) --query 'TraceSummaries[*].Id' --output text)
aws xray batch-get-traces --trace-ids $TRACEIDS --query 'Traces[*]'
```
The script retrieves full traces generated between one and two minutes ago.
```
~/eb-java-scorekeep$ ./bin/full-traces.sh | less
[
{
"Segments": [
{
"Id": "3f212bc237bafd5d",
"Document": "{\"id\":\"3f212bc237bafd5d\",\"name\":\"DynamoDB\",\"trace_id\":\"1-5828d9f2-a90669393f4343211bc1cf75\",\"start_time\":1.479072242459E9,\"end_time\":1.479072242477E9,\"parent_id\":\"72a08dcf87991ca9\",\"http\":{\"response\":{\"content_length\":60,\"status\":200}},\"inferred\":true,\"aws\":{\"consistent_read\":false,\"table_name\":\"scorekeep-session-xray\",\"operation\":\"GetItem\",\"request_id\":\"QAKE0S8DD0LJM245KAOPMA746BVV4KQNSO5AEMVJF66Q9ASUAAJG\",\"resource_names\":[\"scorekeep-session-xray\"]},\"origin\":\"AWS::DynamoDB::Table\"}"
},
{
"Id": "309e355f1148347f",
"Document": "{\"id\":\"309e355f1148347f\",\"name\":\"DynamoDB\",\"trace_id\":\"1-5828d9f2-a90669393f4343211bc1cf75\",\"start_time\":1.479072242477E9,\"end_time\":1.479072242494E9,\"parent_id\":\"37f14ef837f00022\",\"http\":{\"response\":{\"content_length\":606,\"status\":200}},\"inferred\":true,\"aws\":{\"table_name\":\"scorekeep-game-xray\",\"operation\":\"UpdateItem\",\"request_id\":\"388GEROC4PCA6D59ED3CTI5EEJVV4KQNSO5AEMVJF66Q9ASUAAJG\",\"resource_names\":[\"scorekeep-game-xray\"]},\"origin\":\"AWS::DynamoDB::Table\"}"
}
],
"Id": "1-5828d9f2-a90669393f4343211bc1cf75",
"Duration": 0.05099987983703613
}
...
```
## Cleanup
Terminate your Elastic Beanstalk environment to shut down the Amazon EC2 instances, DynamoDB tables and other resources.
**To terminate your Elastic Beanstalk environment**
1. Open the [Elastic Beanstalk console](https://console.aws.amazon.com/elasticbeanstalk).
1. Navigate to the [management console](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-console.html) for your environment.
1. Choose **Actions**.
1. Choose **Terminate Environment**.
1. Choose **Terminate**.
Trace data is automatically deleted from X-Ray after 30 days.
# Sending trace data to AWS X-Ray
You can send trace data to X-Ray in the form of segment documents. A segment document is a JSON formatted string that contains information about the work that your application does in service of a request. Your application can record data about the work that it does itself in segments, or work that uses downstream services and resources in subsegments.
Segments record information about the work that your application does. A segment, at a minimum, records the time spent on a task, a name, and two IDs. The trace ID tracks the request as it travels between services. The segment ID tracks the work done for the request by a single service.
**Example Minimal complete segment**
```
{
"name" : "Scorekeep",
"id" : "70de5b6f19ff9a0a",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979",
"end_time" : 1.478293361449E9
}
```
When a request is received, you can send an in-progress segment as a placeholder until the request is completed.
**Example In-progress segment**
```
{
"name" : "Scorekeep",
"id" : "70de5b6f19ff9a0b",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979",
“in_progress”: true
}
```
You can send segments to X-Ray directly, with [`PutTraceSegments`](#xray-api-segments), or [through the X-Ray daemon](#xray-api-daemon).
Most applications call other services or access resources with the AWS SDK. Record information about downstream calls in *subsegments*. X-Ray uses subsegments to identify downstream services that don't send segments and create entries for them on the service graph.
A subsegment can be embedded in a full segment document, or sent separately. Send subsegments separately to asynchronously trace downstream calls for long-running requests, or to avoid exceeding the maximum segment document size (64 kB).
**Example Subsegment**
A subsegment has a `type` of `subsegment` and a `parent_id` that identifies the parent segment.
```
{
"name" : "www2.example.com",
"id" : "70de5b6f19ff9a0c",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979"
“end_time” : 1.478293361449E9,
“type” : “subsegment”,
“parent_id” : “70de5b6f19ff9a0b”
}
```
For more information on the fields and values that you can include in segments and subsegments, see [AWS X-Ray segment documents](xray-api-segmentdocuments.md).
**Topics**
+ [Generating trace IDs](#xray-api-traceids)
+ [Using PutTraceSegments](#xray-api-segments)
+ [Sending segment documents to the X-Ray daemon](#xray-api-daemon)
## Generating trace IDs
To send data to X-Ray, you must generate a unique trace ID for each request.
**X-Ray trace ID format**
An X-Ray `trace_id` consists of three numbers separated by hyphens. For example, `1-58406520-a006649127e371903a2de979`. This includes:
+ The version number, which is `1`.
+ The time of the original request in Unix epoch time using **8 hexadecimal digits**.
For example, 10:00AM December 1st, 2016 PST in epoch time is `1480615200` seconds or `58406520` in hexadecimal digits.
+ A globally unique 96-bit identifier for the trace in **24 hexadecimal digits**.
**Note**
X-Ray now supports trace IDs that are created using OpenTelemetry and any other framework that conforms with the [W3C Trace Context specification](https://www.w3.org/TR/trace-context/). A W3C trace ID must be formatted in X-Ray trace ID format when sending to X-Ray. For example, W3C trace ID `4efaaf4d1e8720b39541901950019ee5` should be formatted as `1-4efaaf4d-1e8720b39541901950019ee5` when sending to X-Ray. X-Ray trace IDs include the original request time stamp in Unix epoch time, but this isn't required when sending W3C trace IDs in X-Ray format.
You can write a script to generate X-Ray trace IDs for testing. Here are two examples.
**Python**
```
import time
import os
import binascii
START_TIME = time.time()
HEX=hex(int(START_TIME))[2:]
TRACE_ID="1-{}-{}".format(HEX, binascii.hexlify(os.urandom(12)).decode('utf-8'))
```
**Bash**
```
START_TIME=$(date +%s)
HEX_TIME=$(printf '%x\n' $START_TIME)
GUID=$(dd if=/dev/random bs=12 count=1 2>/dev/null | od -An -tx1 | tr -d ' \t\n')
TRACE_ID="1-$HEX_TIME-$GUID"
```
See the Scorekeep sample application for scripts that create trace IDs and send segments to the X-Ray daemon.
+ Python – [https://github.com/awslabs/eb-java-scorekeep/blob/xray/bin/xray_start.py](https://github.com/awslabs/eb-java-scorekeep/blob/xray/bin/xray_start.py)
+ Bash – [https://github.com/awslabs/eb-java-scorekeep/blob/xray/bin/xray_start.sh](https://github.com/awslabs/eb-java-scorekeep/blob/xray/bin/xray_start.sh)
## Using PutTraceSegments
You can upload segment documents with the [https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html](https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html) API. The API has a single parameter, `TraceSegmentDocuments`, that takes a list of JSON segment documents.
With the AWS CLI, use the `aws xray put-trace-segments` command to send segment documents directly to X-Ray.
```
$ DOC='{"trace_id": "1-5960082b-ab52431b496add878434aa25", "id": "6226467e3f845502", "start_time": 1498082657.37518, "end_time": 1498082695.4042, "name": "test.elasticbeanstalk.com"}'
$ aws xray put-trace-segments --trace-segment-documents "$DOC"
{
"UnprocessedTraceSegments": []
}
```
**Note**
Windows Command Processor and Windows PowerShell have different requirements for quoting and escaping quotes in JSON strings. See [Quoting Strings](https://docs.aws.amazon.com/cli/latest/userguide/cli-using-param.html#quoting-strings) in the AWS CLI User Guide for details.
The output lists any segments that failed processing. For example, if the date in the trace ID is too far in the past, you see an error like the following.
```
{
"UnprocessedTraceSegments": [
{
"ErrorCode": "InvalidTraceId",
"Message": "Invalid segment. ErrorCode: InvalidTraceId",
"Id": "6226467e3f845502"
}
]
}
```
You can pass multiple segment documents at the same time, separated by spaces.
```
$ aws xray put-trace-segments --trace-segment-documents "$DOC1" "$DOC2"
```
## Sending segment documents to the X-Ray daemon
Instead of sending segment documents to the X-Ray API, you can send segments and subsegments to the X-Ray daemon, which will buffer them and upload to the X-Ray API in batches. The X-Ray SDK sends segment documents to the daemon to avoid making calls to AWS directly.
**Note**
See [Running the X-Ray daemon locally](xray-daemon-local.md) for instructions on running the daemon.
Send the segment in JSON over UDP port 2000, prepended by the daemon header, `{"format": "json", "version": 1}\n`
```
{"format": "json", "version": 1}\n{"trace_id": "1-5759e988-bd862e3fe1be46a994272793", "id": "defdfd9912dc5a56", "start_time": 1461096053.37518, "end_time": 1461096053.4042, "name": "test.elasticbeanstalk.com"}
```
On Linux, you can send segment documents to the daemon from a Bash terminal. Save the header and segment document to a text file and pipe it to `/dev/udp` with `cat`.
```
$ cat segment.txt > /dev/udp/127.0.0.1/2000
```
**Example segment.txt**
```
{"format": "json", "version": 1}
{"trace_id": "1-594aed87-ad72e26896b3f9d3a27054bb", "id": "6226467e3f845502", "start_time": 1498082657.37518, "end_time": 1498082695.4042, "name": "test.elasticbeanstalk.com"}
```
Check the [daemon log](xray-daemon.md#xray-daemon-logging) to verify that it sent the segment to X-Ray.
```
2017-07-07T01:57:24Z [Debug] processor: sending partial batch
2017-07-07T01:57:24Z [Debug] processor: segment batch size: 1. capacity: 50
2017-07-07T01:57:24Z [Info] Successfully sent batch of 1 segments (0.020 seconds)
```
# Getting data from AWS X-Ray
AWS X-Ray processes the trace data that you send to it to generate full traces, trace summaries, and service graphs in JSON. You can retrieve the generated data directly from the API with the AWS CLI.
**Topics**
+ [Retrieving the service graph](#xray-api-servicegraph)
+ [Retrieving the service graph by group](#xray-api-servicegraphgroup)
+ [Retrieving traces](#xray-api-traces)
+ [Retrieving and refining root cause analytics](#xray-api-analytics)
## Retrieving the service graph
You can use the [https://docs.aws.amazon.com/xray/latest/api/API_GetServiceGraph.html](https://docs.aws.amazon.com/xray/latest/api/API_GetServiceGraph.html) API to retrieve the JSON service graph. The API requires a start time and end time, which you can calculate from a Linux terminal with the `date` command.
```
$ date +%s
1499394617
```
`date +%s` prints a date in seconds. Use this number as an end time and subtract time from it to get a start time.
**Example Script to retrieve a service graph for the last 10 minutes**
```
EPOCH=$(date +%s)
aws xray get-service-graph --start-time $(($EPOCH-600)) --end-time $EPOCH
```
The following example shows a service graph with 4 nodes, including a client node, an EC2 instance, a DynamoDB table, and an Amazon SNS topic.
**Example GetServiceGraph output**
```
{
"Services": [
{
"ReferenceId": 0,
"Name": "xray-sample.elasticbeanstalk.com",
"Names": [
"xray-sample.elasticbeanstalk.com"
],
"Type": "client",
"State": "unknown",
"StartTime": 1528317567.0,
"EndTime": 1528317589.0,
"Edges": [
{
"ReferenceId": 2,
"StartTime": 1528317567.0,
"EndTime": 1528317589.0,
"SummaryStatistics": {
"OkCount": 3,
"ErrorStatistics": {
"ThrottleCount": 0,
"OtherCount": 1,
"TotalCount": 1
},
"FaultStatistics": {
"OtherCount": 0,
"TotalCount": 0
},
"TotalCount": 4,
"TotalResponseTime": 0.273
},
"ResponseTimeHistogram": [
{
"Value": 0.005,
"Count": 1
},
{
"Value": 0.015,
"Count": 1
},
{
"Value": 0.157,
"Count": 1
},
{
"Value": 0.096,
"Count": 1
}
],
"Aliases": []
}
]
},
{
"ReferenceId": 1,
"Name": "awseb-e-dixzws4s9p-stack-StartupSignupsTable-4IMSMHAYX2BA",
"Names": [
"awseb-e-dixzws4s9p-stack-StartupSignupsTable-4IMSMHAYX2BA"
],
"Type": "AWS::DynamoDB::Table",
"State": "unknown",
"StartTime": 1528317583.0,
"EndTime": 1528317589.0,
"Edges": [],
"SummaryStatistics": {
"OkCount": 2,
"ErrorStatistics": {
"ThrottleCount": 0,
"OtherCount": 0,
"TotalCount": 0
},
"FaultStatistics": {
"OtherCount": 0,
"TotalCount": 0
},
"TotalCount": 2,
"TotalResponseTime": 0.12
},
"DurationHistogram": [
{
"Value": 0.076,
"Count": 1
},
{
"Value": 0.044,
"Count": 1
}
],
"ResponseTimeHistogram": [
{
"Value": 0.076,
"Count": 1
},
{
"Value": 0.044,
"Count": 1
}
]
},
{
"ReferenceId": 2,
"Name": "xray-sample.elasticbeanstalk.com",
"Names": [
"xray-sample.elasticbeanstalk.com"
],
"Root": true,
"Type": "AWS::EC2::Instance",
"State": "active",
"StartTime": 1528317567.0,
"EndTime": 1528317589.0,
"Edges": [
{
"ReferenceId": 1,
"StartTime": 1528317567.0,
"EndTime": 1528317589.0,
"SummaryStatistics": {
"OkCount": 2,
"ErrorStatistics": {
"ThrottleCount": 0,
"OtherCount": 0,
"TotalCount": 0
},
"FaultStatistics": {
"OtherCount": 0,
"TotalCount": 0
},
"TotalCount": 2,
"TotalResponseTime": 0.12
},
"ResponseTimeHistogram": [
{
"Value": 0.076,
"Count": 1
},
{
"Value": 0.044,
"Count": 1
}
],
"Aliases": []
},
{
"ReferenceId": 3,
"StartTime": 1528317567.0,
"EndTime": 1528317589.0,
"SummaryStatistics": {
"OkCount": 2,
"ErrorStatistics": {
"ThrottleCount": 0,
"OtherCount": 0,
"TotalCount": 0
},
"FaultStatistics": {
"OtherCount": 0,
"TotalCount": 0
},
"TotalCount": 2,
"TotalResponseTime": 0.125
},
"ResponseTimeHistogram": [
{
"Value": 0.049,
"Count": 1
},
{
"Value": 0.076,
"Count": 1
}
],
"Aliases": []
}
],
"SummaryStatistics": {
"OkCount": 3,
"ErrorStatistics": {
"ThrottleCount": 0,
"OtherCount": 1,
"TotalCount": 1
},
"FaultStatistics": {
"OtherCount": 0,
"TotalCount": 0
},
"TotalCount": 4,
"TotalResponseTime": 0.273
},
"DurationHistogram": [
{
"Value": 0.005,
"Count": 1
},
{
"Value": 0.015,
"Count": 1
},
{
"Value": 0.157,
"Count": 1
},
{
"Value": 0.096,
"Count": 1
}
],
"ResponseTimeHistogram": [
{
"Value": 0.005,
"Count": 1
},
{
"Value": 0.015,
"Count": 1
},
{
"Value": 0.157,
"Count": 1
},
{
"Value": 0.096,
"Count": 1
}
]
},
{
"ReferenceId": 3,
"Name": "SNS",
"Names": [
"SNS"
],
"Type": "AWS::SNS",
"State": "unknown",
"StartTime": 1528317583.0,
"EndTime": 1528317589.0,
"Edges": [],
"SummaryStatistics": {
"OkCount": 2,
"ErrorStatistics": {
"ThrottleCount": 0,
"OtherCount": 0,
"TotalCount": 0
},
"FaultStatistics": {
"OtherCount": 0,
"TotalCount": 0
},
"TotalCount": 2,
"TotalResponseTime": 0.125
},
"DurationHistogram": [
{
"Value": 0.049,
"Count": 1
},
{
"Value": 0.076,
"Count": 1
}
],
"ResponseTimeHistogram": [
{
"Value": 0.049,
"Count": 1
},
{
"Value": 0.076,
"Count": 1
}
]
}
]
}
```
## Retrieving the service graph by group
To call for a service graph based on the contents of a group, include a `groupName` or `groupARN`. The following example shows a service graph call to a group named Example1.
**Example Script to retrieve a service graph by name for group Example1**
```
aws xray get-service-graph --group-name "Example1"
```
## Retrieving traces
You can use the [https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSummaries.html](https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSummaries.html) API to get a list of trace summaries. Trace summaries include information that you can use to identify traces that you want to download in full, including annotations, request and response information, and IDs.
There are two `TimeRangeType` flags available when calling `aws xray get-trace-summaries`:
+ **TraceId** – The default `GetTraceSummaries` search uses TraceID time and returns traces started within the computed `[start_time, end_time)` range. This range of timestamps is calculated based on the encoding of the timestamp within the TraceId, or can be defined manually.
+ **Event time **– To search for events as they happen over the time, AWS X-Ray allows searching for traces using event timestamps. Event time returns traces active during the `[start_time, end_time)` range, regardless of when the trace began.
Use the `aws xray get-trace-summaries` command to get a list of trace summaries. The following commands get a list of trace summaries from between 1 and 2 minutes in the past using the default TraceId time.
**Example Script to get trace summaries**
```
EPOCH=$(date +%s)
aws xray get-trace-summaries --start-time $(($EPOCH-120)) --end-time $(($EPOCH-60))
```
**Example GetTraceSummaries output**
```
{
"TraceSummaries": [
{
"HasError": false,
"Http": {
"HttpStatus": 200,
"ClientIp": "205.255.255.183",
"HttpURL": "http://scorekeep.elasticbeanstalk.com/api/session",
"UserAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
"HttpMethod": "POST"
},
"Users": [],
"HasFault": false,
"Annotations": {},
"ResponseTime": 0.084,
"Duration": 0.084,
"Id": "1-59602606-a43a1ac52fc7ee0eea12a82c",
"HasThrottle": false
},
{
"HasError": false,
"Http": {
"HttpStatus": 200,
"ClientIp": "205.255.255.183",
"HttpURL": "http://scorekeep.elasticbeanstalk.com/api/user",
"UserAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
"HttpMethod": "POST"
},
"Users": [
{
"UserName": "5M388M1E"
}
],
"HasFault": false,
"Annotations": {
"UserID": [
{
"AnnotationValue": {
"StringValue": "5M388M1E"
}
}
],
"Name": [
{
"AnnotationValue": {
"StringValue": "Ola"
}
}
]
},
"ResponseTime": 3.232,
"Duration": 3.232,
"Id": "1-59602603-23fc5b688855d396af79b496",
"HasThrottle": false
}
],
"ApproximateTime": 1499473304.0,
"TracesProcessedCount": 2
}
```
Use the trace ID from the output to retrieve a full trace with the [https://docs.aws.amazon.com/xray/latest/api/API_BatchGetTraces.html](https://docs.aws.amazon.com/xray/latest/api/API_BatchGetTraces.html) API.
**Example BatchGetTraces command**
```
$ aws xray batch-get-traces --trace-ids 1-596025b4-7170afe49f7aa708b1dd4a6b
```
**Example BatchGetTraces output**
```
{
"Traces": [
{
"Duration": 3.232,
"Segments": [
{
"Document": "{\"id\":\"1fb07842d944e714\",\"name\":\"random-name\",\"start_time\":1.499473411677E9,\"end_time\":1.499473414572E9,\"parent_id\":\"0c544c1b1bbff948\",\"http\":{\"response\":{\"status\":200}},\"aws\":{\"request_id\":\"ac086670-6373-11e7-a174-f31b3397f190\"},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"origin\":\"AWS::Lambda\",\"resource_arn\":\"arn:aws:lambda:us-west-2:123456789012:function:random-name\"}",
"Id": "1fb07842d944e714"
},
{
"Document": "{\"id\":\"194fcc8747581230\",\"name\":\"Scorekeep\",\"start_time\":1.499473411562E9,\"end_time\":1.499473414794E9,\"http\":{\"request\":{\"url\":\"http://scorekeep.elasticbeanstalk.com/api/user\",\"method\":\"POST\",\"user_agent\":\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36\",\"client_ip\":\"205.251.233.183\"},\"response\":{\"status\":200}},\"aws\":{\"elastic_beanstalk\":{\"version_label\":\"app-abb9-170708_002045\",\"deployment_id\":406,\"environment_name\":\"scorekeep-dev\"},\"ec2\":{\"availability_zone\":\"us-west-2c\",\"instance_id\":\"i-0cd9e448944061b4a\"},\"xray\":{\"sdk_version\":\"1.1.2\",\"sdk\":\"X-Ray for Java\"}},\"service\":{},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"user\":\"5M388M1E\",\"origin\":\"AWS::ElasticBeanstalk::Environment\",\"subsegments\":[{\"id\":\"0c544c1b1bbff948\",\"name\":\"Lambda\",\"start_time\":1.499473411629E9,\"end_time\":1.499473414572E9,\"http\":{\"response\":{\"status\":200,\"content_length\":14}},\"aws\":{\"log_type\":\"None\",\"status_code\":200,\"function_name\":\"random-name\",\"invocation_type\":\"RequestResponse\",\"operation\":\"Invoke\",\"request_id\":\"ac086670-6373-11e7-a174-f31b3397f190\",\"resource_names\":[\"random-name\"]},\"namespace\":\"aws\"},{\"id\":\"071684f2e555e571\",\"name\":\"## UserModel.saveUser\",\"start_time\":1.499473414581E9,\"end_time\":1.499473414769E9,\"metadata\":{\"debug\":{\"test\":\"Metadata string from UserModel.saveUser\"}},\"subsegments\":[{\"id\":\"4cd3f10b76c624b4\",\"name\":\"DynamoDB\",\"start_time\":1.49947341469E9,\"end_time\":1.499473414769E9,\"http\":{\"response\":{\"status\":200,\"content_length\":57}},\"aws\":{\"table_name\":\"scorekeep-user\",\"operation\":\"UpdateItem\",\"request_id\":\"MFQ8CGJ3JTDDVVVASUAAJGQ6NJ82F738BOB4KQNSO5AEMVJF66Q9\",\"resource_names\":[\"scorekeep-user\"]},\"namespace\":\"aws\"}]}]}",
"Id": "194fcc8747581230"
},
{
"Document": "{\"id\":\"00f91aa01f4984fd\",\"name\":\"random-name\",\"start_time\":1.49947341283E9,\"end_time\":1.49947341457E9,\"parent_id\":\"1fb07842d944e714\",\"aws\":{\"function_arn\":\"arn:aws:lambda:us-west-2:123456789012:function:random-name\",\"resource_names\":[\"random-name\"],\"account_id\":\"123456789012\"},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"origin\":\"AWS::Lambda::Function\",\"subsegments\":[{\"id\":\"e6d2fe619f827804\",\"name\":\"annotations\",\"start_time\":1.499473413012E9,\"end_time\":1.499473413069E9,\"annotations\":{\"UserID\":\"5M388M1E\",\"Name\":\"Ola\"}},{\"id\":\"b29b548af4d54a0f\",\"name\":\"SNS\",\"start_time\":1.499473413112E9,\"end_time\":1.499473414071E9,\"http\":{\"response\":{\"status\":200}},\"aws\":{\"operation\":\"Publish\",\"region\":\"us-west-2\",\"request_id\":\"a2137970-f6fc-5029-83e8-28aadeb99198\",\"retries\":0,\"topic_arn\":\"arn:aws:sns:us-west-2:123456789012:awseb-e-ruag3jyweb-stack-NotificationTopic-6B829NT9V5O9\"},\"namespace\":\"aws\"},{\"id\":\"2279c0030c955e52\",\"name\":\"Initialization\",\"start_time\":1.499473412064E9,\"end_time\":1.499473412819E9,\"aws\":{\"function_arn\":\"arn:aws:lambda:us-west-2:123456789012:function:random-name\"}}]}",
"Id": "00f91aa01f4984fd"
},
{
"Document": "{\"id\":\"17ba309b32c7fbaf\",\"name\":\"DynamoDB\",\"start_time\":1.49947341469E9,\"end_time\":1.499473414769E9,\"parent_id\":\"4cd3f10b76c624b4\",\"inferred\":true,\"http\":{\"response\":{\"status\":200,\"content_length\":57}},\"aws\":{\"table_name\":\"scorekeep-user\",\"operation\":\"UpdateItem\",\"request_id\":\"MFQ8CGJ3JTDDVVVASUAAJGQ6NJ82F738BOB4KQNSO5AEMVJF66Q9\",\"resource_names\":[\"scorekeep-user\"]},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"origin\":\"AWS::DynamoDB::Table\"}",
"Id": "17ba309b32c7fbaf"
},
{
"Document": "{\"id\":\"1ee3c4a523f89ca5\",\"name\":\"SNS\",\"start_time\":1.499473413112E9,\"end_time\":1.499473414071E9,\"parent_id\":\"b29b548af4d54a0f\",\"inferred\":true,\"http\":{\"response\":{\"status\":200}},\"aws\":{\"operation\":\"Publish\",\"region\":\"us-west-2\",\"request_id\":\"a2137970-f6fc-5029-83e8-28aadeb99198\",\"retries\":0,\"topic_arn\":\"arn:aws:sns:us-west-2:123456789012:awseb-e-ruag3jyweb-stack-NotificationTopic-6B829NT9V5O9\"},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"origin\":\"AWS::SNS\"}",
"Id": "1ee3c4a523f89ca5"
}
],
"Id": "1-59602603-23fc5b688855d396af79b496"
}
],
"UnprocessedTraceIds": []
}
```
The full trace includes a document for each segment, compiled from all of the segment documents received with the same trace ID. These documents don't represent the data as it was sent to X-Ray by your application. Instead, they represent the processed documents generated by the X-Ray service. X-Ray creates the full trace document by compiling segment documents sent by your application, and removing data that doesn't comply with the [segment document schema](xray-api-segmentdocuments.md).
X-Ray also creates *inferred segments* for downstream calls to services that don't send segments themselves. For example, when you call DynamoDB with an instrumented client, the X-Ray SDK records a subsegment with details about the call from its point of view. However, DynamoDB doesn't send a corresponding segment. X-Ray uses the information in the subsegment to create an inferred segment to represent the DynamoDB resource in the trace map, and adds it to the trace document.
To get multiple traces from the API, you need a list of trace IDs, which you can extract from the output of `get-trace-summaries` with an [AWS CLI query](https://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html#controlling-output-filter). Redirect the list to the input of `batch-get-traces` to get full traces for a specific time period.
**Example Script to get full traces for a one minute period**
```
EPOCH=$(date +%s)
TRACEIDS=$(aws xray get-trace-summaries --start-time $(($EPOCH-120)) --end-time $(($EPOCH-60)) --query 'TraceSummaries[*].Id' --output text)
aws xray batch-get-traces --trace-ids $TRACEIDS --query 'Traces[*]'
```
## Retrieving and refining root cause analytics
Upon generating a trace summary with the [GetTraceSummaries API](https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSummaries.html) , partial trace summaries can be reused in their JSON format to create a refined filter expression based upon root causes. See the examples below for a walkthrough of the refinement steps.
**Example GetTraceSummaries output - response time root cause section**
```
{
"Services": [
{
"Name": "GetWeatherData",
"Names": ["GetWeatherData"],
"AccountId": 123456789012,
"Type": null,
"Inferred": false,
"EntityPath": [
{
"Name": "GetWeatherData",
"Coverage": 1.0,
'Remote": false
},
{
"Name": "get_temperature",
"Coverage": 0.8,
"Remote": false
}
]
},
{
"Name": "GetTemperature",
"Names": ["GetTemperature"],
"AccountId": 123456789012,
"Type": null,
"Inferred": false,
"EntityPath": [
{
"Name": "GetTemperature",
"Coverage": 0.7,
"Remote": false
}
]
}
]
}
```
By editing and making omissions to the above output, this JSON can become a filter for matched root cause entities. For every field present in the JSON, any candidate match must be exact, or the trace will not be returned. Removed fields become wildcard values, a format which is compatible with the filter expression query structure.
**Example Reformatted response time root cause**
```
{
"Services": [
{
"Name": "GetWeatherData",
"EntityPath": [
{
"Name": "GetWeatherData"
},
{
"Name": "get_temperature"
}
]
},
{
"Name": "GetTemperature",
"EntityPath": [
{
"Name": "GetTemperature"
}
]
}
]
}
```
This JSON is then used as part of a filter expression through a call to `rootcause.json = #[{}]`. Refer to the [Filter Expressions](xray-console-filters.md) chapter for more details about querying with filter expressions.
**Example JSON filter**
```
rootcause.json = #[{ "Services": [ { "Name": "GetWeatherData", "EntityPath": [{ "Name": "GetWeatherData" }, { "Name": "get_temperature" } ] }, { "Name": "GetTemperature", "EntityPath": [ { "Name": "GetTemperature" } ] } ] }]
```
# Configuring sampling, groups, and encryption settings with the AWS X-Ray API
AWS X-Ray provides APIs for configuring [sampling rules](xray-console-sampling.md), group rules, and [encryption settings](xray-console-encryption.md).
**Topics**
+ [Encryption settings](#xray-api-configuration-encryption)
+ [Sampling rules](#xray-api-configuration-sampling)
+ [Groups](#xray-api-configuration-groups)
## Encryption settings
Use [https://docs.aws.amazon.com/xray/latest/api/API_PutEncryptionConfig.html](https://docs.aws.amazon.com/xray/latest/api/API_PutEncryptionConfig.html) to specify an AWS Key Management Service (AWS KMS) key to use for encryption.
**Note**
X-Ray does not support asymmetric KMS keys.
```
$ aws xray put-encryption-config --type KMS --key-id alias/aws/xray
{
"EncryptionConfig": {
"KeyId": "arn:aws:kms:us-east-2:123456789012:key/c234g4e8-39e9-4gb0-84e2-b0ea215cbba5",
"Status": "UPDATING",
"Type": "KMS"
}
}
```
For the key ID, you can use an alias (as shown in the example), a key ID, or an Amazon Resource Name (ARN).
Use [https://docs.aws.amazon.com/xray/latest/api/API_GetEncryptionConfig.html](https://docs.aws.amazon.com/xray/latest/api/API_GetEncryptionConfig.html) to get the current configuration. When X-Ray finishes applying your settings, the status changes from `UPDATING` to `ACTIVE`.
```
$ aws xray get-encryption-config
{
"EncryptionConfig": {
"KeyId": "arn:aws:kms:us-east-2:123456789012:key/c234g4e8-39e9-4gb0-84e2-b0ea215cbba5",
"Status": "ACTIVE",
"Type": "KMS"
}
}
```
To stop using a KMS key and use default encryption, set the encryption type to `NONE`.
```
$ aws xray put-encryption-config --type NONE
{
"EncryptionConfig": {
"Status": "UPDATING",
"Type": "NONE"
}
}
```
## Sampling rules
You can manage the [sampling rules](xray-console-sampling.md) in your account with the X-Ray API. For more information about adding and managing tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md).
Get all sampling rules with [https://docs.aws.amazon.com/xray/latest/api/API_GetSamplingRules.html](https://docs.aws.amazon.com/xray/latest/api/API_GetSamplingRules.html).
```
$ aws xray get-sampling-rules
{
"SamplingRuleRecords": [
{
"SamplingRule": {
"RuleName": "Default",
"RuleARN": "arn:aws:xray:us-east-2:123456789012:sampling-rule/Default",
"ResourceARN": "*",
"Priority": 10000,
"FixedRate": 0.05,
"ReservoirSize": 1,
"ServiceName": "*",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 0.0,
"ModifiedAt": 1529959993.0
}
]
}
```
The default rule applies to all requests that don't match another rule. It is the lowest priority rule and cannot be deleted. You can, however, change the rate and reservoir size with [https://docs.aws.amazon.com/xray/latest/api/API_UpdateSamplingRule.html](https://docs.aws.amazon.com/xray/latest/api/API_UpdateSamplingRule.html).
**Example API input for [https://docs.aws.amazon.com/xray/latest/api/API_UpdateSamplingRule.html](https://docs.aws.amazon.com/xray/latest/api/API_UpdateSamplingRule.html) – 10000-default.json**
```
{
"SamplingRuleUpdate": {
"RuleName": "Default",
"FixedRate": 0.01,
"ReservoirSize": 0
}
}
```
The following example uses the previous file as input to change the default rule to one percent with no reservoir. Tags are optional. If you choose to add tags, a tag key is required, and tag values are optional. To remove existing tags from a sampling rule, use [UntagResource](https://docs.aws.amazon.com/xray/latest/api/API_UntagResource.html)
```
$ aws xray update-sampling-rule --cli-input-json file://1000-default.json --tags [{"Key": "key_name","Value": "value"},{"Key": "key_name","Value": "value"}]
{
"SamplingRuleRecords": [
{
"SamplingRule": {
"RuleName": "Default",
"RuleARN": "arn:aws:xray:us-east-2:123456789012:sampling-rule/Default",
"ResourceARN": "*",
"Priority": 10000,
"FixedRate": 0.01,
"ReservoirSize": 0,
"ServiceName": "*",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 0.0,
"ModifiedAt": 1529959993.0
},
```
Create additional sampling rules with [https://docs.aws.amazon.com/xray/latest/api/API_CreateSamplingRule.html](https://docs.aws.amazon.com/xray/latest/api/API_CreateSamplingRule.html). When you create a rule, most of the rule fields are required. The following example creates two rules. This first rule sets a base rate for the Scorekeep sample application. It matches all requests served by the API that don't match a higher priority rule.
**Example API input for [https://docs.aws.amazon.com/xray/latest/api/API_UpdateSamplingRule.html](https://docs.aws.amazon.com/xray/latest/api/API_UpdateSamplingRule.html) – 9000-base-scorekeep.json**
```
{
"SamplingRule": {
"RuleName": "base-scorekeep",
"ResourceARN": "*",
"Priority": 9000,
"FixedRate": 0.1,
"ReservoirSize": 5,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1
}
}
```
The second rule also applies to Scorekeep, but it has a higher priority and is more specific. This rule sets a very low sampling rate for polling requests. These are GET requests made by the client every few seconds to check for changes to the game state.
**Example API input for [https://docs.aws.amazon.com/xray/latest/api/API_UpdateSamplingRule.html](https://docs.aws.amazon.com/xray/latest/api/API_UpdateSamplingRule.html) – 5000-polling-scorekeep.json**
```
{
"SamplingRule": {
"RuleName": "polling-scorekeep",
"ResourceARN": "*",
"Priority": 5000,
"FixedRate": 0.003,
"ReservoirSize": 0,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "GET",
"URLPath": "/api/state/*",
"Version": 1
}
}
```
Tags are optional. If you choose to add tags, a tag key is required, and tag values are optional.
```
$ aws xray create-sampling-rule --cli-input-json file://5000-polling-scorekeep.json --tags [{"Key": "key_name","Value": "value"},{"Key": "key_name","Value": "value"}]
{
"SamplingRuleRecord": {
"SamplingRule": {
"RuleName": "polling-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1:123456789012:sampling-rule/polling-scorekeep",
"ResourceARN": "*",
"Priority": 5000,
"FixedRate": 0.003,
"ReservoirSize": 0,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "GET",
"URLPath": "/api/state/*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530574399.0,
"ModifiedAt": 1530574399.0
}
}
$ aws xray create-sampling-rule --cli-input-json file://9000-base-scorekeep.json
{
"SamplingRuleRecord": {
"SamplingRule": {
"RuleName": "base-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1:123456789012:sampling-rule/base-scorekeep",
"ResourceARN": "*",
"Priority": 9000,
"FixedRate": 0.1,
"ReservoirSize": 5,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530574410.0,
"ModifiedAt": 1530574410.0
}
}
```
To delete a sampling rule, use [https://docs.aws.amazon.com/xray/latest/api/API_DeleteSamplingRule.html](https://docs.aws.amazon.com/xray/latest/api/API_DeleteSamplingRule.html).
```
$ aws xray delete-sampling-rule --rule-name polling-scorekeep
{
"SamplingRuleRecord": {
"SamplingRule": {
"RuleName": "polling-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1:123456789012:sampling-rule/polling-scorekeep",
"ResourceARN": "*",
"Priority": 5000,
"FixedRate": 0.003,
"ReservoirSize": 0,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "GET",
"URLPath": "/api/state/*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530574399.0,
"ModifiedAt": 1530574399.0
}
}
```
## Groups
You can use the X-Ray API to manage groups in your account. Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics. See [Getting data from AWS X-Ray](xray-api-gettingdata.md) for more details about working with service graphs and metrics through the X-Ray API. For more information about groups, see [Configuring groups](xray-console-groups.md). For more information about adding and managing tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md).
Create a group with `CreateGroup`. Tags are optional. If you choose to add tags, a tag key is required, and tag values are optional.
```
$ aws xray create-group --group-name "TestGroup" --filter-expression "service(\"example.com\") {fault}" --tags [{"Key": "key_name","Value": "value"},{"Key": "key_name","Value": "value"}]
{
"GroupName": "TestGroup",
"GroupARN": "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID",
"FilterExpression": "service(\"example.com\") {fault OR error}"
}
```
Get all existing groups with `GetGroups`.
```
$ aws xray get-groups
{
"Groups": [
{
"GroupName": "TestGroup",
"GroupARN": "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID",
"FilterExpression": "service(\"example.com\") {fault OR error}"
},
{
"GroupName": "TestGroup2",
"GroupARN": "arn:aws:xray:us-east-2:123456789012:group/TestGroup2/UniqueID",
"FilterExpression": "responsetime > 2"
}
],
"NextToken": "tokenstring"
}
```
Update a group with `UpdateGroup`. Tags are optional. If you choose to add tags, a tag key is required, and tag values are optional. To remove existing tags from a group, use [UntagResource](https://docs.aws.amazon.com/xray/latest/api/API_UntagResource.html).
```
$ aws xray update-group --group-name "TestGroup" --group-arn "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID" --filter-expression "service(\"example.com\") {fault OR error}" --tags [{"Key": "Stage","Value": "Prod"},{"Key": "Department","Value": "QA"}]
{
"GroupName": "TestGroup",
"GroupARN": "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID",
"FilterExpression": "service(\"example.com\") {fault OR error}"
}
```
Delete a group with `DeleteGroup`.
```
$ aws xray delete-group --group-name "TestGroup" --group-arn "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID"
{
}
```
# Using sampling rules with the X-Ray API
The AWS X-Ray SDK uses the X-Ray API to get sampling rules, report sampling results, and get quotas. You can use these APIs to get a better understanding of how sampling rules work, or to implement sampling in a language that the X-Ray SDK doesn't support.
Start by getting all sampling rules with [https://docs.aws.amazon.com/xray/latest/api/API_GetSamplingRules.html](https://docs.aws.amazon.com/xray/latest/api/API_GetSamplingRules.html).
```
$ aws xray get-sampling-rules
{
"SamplingRuleRecords": [
{
"SamplingRule": {
"RuleName": "Default",
"RuleARN": "arn:aws:xray:us-east-1::sampling-rule/Default",
"ResourceARN": "*",
"Priority": 10000,
"FixedRate": 0.01,
"ReservoirSize": 0,
"ServiceName": "*",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 0.0,
"ModifiedAt": 1530558121.0
},
{
"SamplingRule": {
"RuleName": "base-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1::sampling-rule/base-scorekeep",
"ResourceARN": "*",
"Priority": 9000,
"FixedRate": 0.1,
"ReservoirSize": 2,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530573954.0,
"ModifiedAt": 1530920505.0
},
{
"SamplingRule": {
"RuleName": "polling-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1::sampling-rule/polling-scorekeep",
"ResourceARN": "*",
"Priority": 5000,
"FixedRate": 0.003,
"ReservoirSize": 0,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "GET",
"URLPath": "/api/state/*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530918163.0,
"ModifiedAt": 1530918163.0
}
]
}
```
The output includes the default rule and custom rules. See [Sampling rules](xray-api-configuration.md#xray-api-configuration-sampling) if you haven't yet created sampling rules.
Evaluate rules against incoming requests in ascending order of priority. When a rule matches, use the fixed rate and reservoir size to make a sampling decision. Record sampled requests and ignore (for tracing purposes) unsampled requests. Stop evaluating rules when a sampling decision is made.
A rules reservoir size is the target number of traces to record per second before applying the fixed rate. The reservoir applies across all services cumulatively, so you can't use it directly. However, if it is non-zero, you can borrow one trace per second from the reservoir until X-Ray assigns a quota. Before receiving a quota, record the first request each second, and apply the fixed rate to additional requests. The fixed rate is a decimal between 0 and 1.00 (100%).
The following example shows a call to [https://docs.aws.amazon.com/xray/latest/api/API_GetSamplingTargets.html](https://docs.aws.amazon.com/xray/latest/api/API_GetSamplingTargets.html) with details about sampling decisions made over the last 10 seconds.
```
$ aws xray get-sampling-targets --sampling-statistics-documents '[
{
"RuleName": "base-scorekeep",
"ClientID": "ABCDEF1234567890ABCDEF10",
"Timestamp": "2018-07-07T00:20:06",
"RequestCount": 110,
"SampledCount": 20,
"BorrowCount": 10
},
{
"RuleName": "polling-scorekeep",
"ClientID": "ABCDEF1234567890ABCDEF10",
"Timestamp": "2018-07-07T00:20:06",
"RequestCount": 10500,
"SampledCount": 31,
"BorrowCount": 0
}
]'
{
"SamplingTargetDocuments": [
{
"RuleName": "base-scorekeep",
"FixedRate": 0.1,
"ReservoirQuota": 2,
"ReservoirQuotaTTL": 1530923107.0,
"Interval": 10
},
{
"RuleName": "polling-scorekeep",
"FixedRate": 0.003,
"ReservoirQuota": 0,
"ReservoirQuotaTTL": 1530923107.0,
"Interval": 10
}
],
"LastRuleModification": 1530920505.0,
"UnprocessedStatistics": []
}
```
The response from X-Ray includes a quota to use instead of borrowing from the reservoir. In this example, the service borrowed 10 traces from the reservoir over 10 seconds, and applied the fixed rate of 10 percent to the other 100 requests, resulting in a total of 20 sampled requests. The quota is good for five minutes (indicated by the time to live) or until a new quota is assigned. X-Ray may also assign a longer reporting interval than the default, although it didn't here.
**Note**
The response from X-Ray might not include a quota the first time you call it. Continue borrowing from the reservoir until you are assigned a quota.
The other two fields in the response might indicate issues with the input. Check `LastRuleModification` against the last time you called [https://docs.aws.amazon.com/xray/latest/api/API_GetSamplingRules.html](https://docs.aws.amazon.com/xray/latest/api/API_GetSamplingRules.html). If it's newer, get a new copy of the rules. `UnprocessedStatistics` can include errors that indicate that a rule has been deleted, that the statistics document in the input was too old, or permissions errors.
# AWS X-Ray segment documents
A **trace segment** is a JSON representation of a request that your application serves. A trace segment records information about the original request, information about the work that your application does locally, and **subsegments** with information about downstream calls that your application makes to AWS resources, HTTP APIs, and SQL databases.
A **segment document** conveys information about a segment to X-Ray. A segment document can be up to 64 kB and contain a whole segment with subsegments, a fragment of a segment that indicates that a request is in progress, or a single subsegment that is sent separately. You can send segment documents directly to X-Ray by using the [https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html](https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html) API.
X-Ray compiles and processes segment documents to generate queryable **trace summaries** and **full traces** that you can access by using the [https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSummaries.html](https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSummaries.html) and [https://docs.aws.amazon.com/xray/latest/api/API_BatchGetTraces.html](https://docs.aws.amazon.com/xray/latest/api/API_BatchGetTraces.html) APIs, respectively. In addition to the segments and subsegments that you send to X-Ray, the service uses information in subsegments to generate **inferred segments** and adds them to the full trace. Inferred segments represent downstream services and resources in the trace map.
X-Ray provides a **JSON schema** for segment documents. You can download the schema here: [xray-segmentdocument-schema-v1.0.0](samples/xray-segmentdocument-schema-v1.0.0.zip). The fields and objects listed in the schema are described in more detail in the following sections.
A subset of segment fields are indexed by X-Ray for use with filter expressions. For example, if you set the `user` field on a segment to a unique identifier, you can search for segments associated with specific users in the X-Ray console or by using the `GetTraceSummaries` API. For more information, see [Using filter expressions](xray-console-filters.md).
When you instrument your application with the X-Ray SDK, the SDK generates segment documents for you. Instead of sending segment documents directly to X-Ray, the SDK transmits them over a local UDP port to the [X-Ray daemon](xray-daemon.md). For more information, see [Sending segment documents to the X-Ray daemon](xray-api-sendingdata.md#xray-api-daemon).
**Topics**
+ [Segment fields](#api-segmentdocuments-fields)
+ [Subsegments](#api-segmentdocuments-subsegments)
+ [HTTP request data](#api-segmentdocuments-http)
+ [Annotations](#api-segmentdocuments-annotations)
+ [Metadata](#api-segmentdocuments-metadata)
+ [AWS resource data](#api-segmentdocuments-aws)
+ [Errors and exceptions](#api-segmentdocuments-errors)
+ [SQL queries](#api-segmentdocuments-sql)
## Segment fields
A segment records tracing information about a request that your application serves. At a minimum, a segment records the name, ID, start time, trace ID, and end time of the request.
**Example Minimal complete segment**
```
{
"name" : "example.com",
"id" : "70de5b6f19ff9a0a",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979",
"end_time" : 1.478293361449E9
}
```
The following fields are required, or conditionally required, for segments.
**Note**
Values must be strings (up to 250 characters) unless noted otherwise.
**Required Segment Fields**
+ `name` – The logical name of the service that handled the request, up to **200 characters**. For example, your application's name or domain name. Names can contain Unicode letters, numbers, and whitespace, and the following symbols: `_`, `.`, `:`, `/`, `%`, `&`, `#`, `=`, `+`, `\`, `-`, `@`
+ `id` – A 64-bit identifier for the segment, unique among segments in the same trace, in **16 hexadecimal digits**.
+ `trace_id` – A unique identifier that connects all segments and subsegments originating from a single client request.
**X-Ray trace ID format**
An X-Ray `trace_id` consists of three numbers separated by hyphens. For example, `1-58406520-a006649127e371903a2de979`. This includes:
+ The version number, which is `1`.
+ The time of the original request in Unix epoch time using **8 hexadecimal digits**.
For example, 10:00AM December 1st, 2016 PST in epoch time is `1480615200` seconds or `58406520` in hexadecimal digits.
+ A globally unique 96-bit identifier for the trace in **24 hexadecimal digits**.
**Note**
X-Ray now supports trace IDs that are created using OpenTelemetry and any other framework that conforms with the [W3C Trace Context specification](https://www.w3.org/TR/trace-context/). A W3C trace ID must be formatted in X-Ray trace ID format when sending to X-Ray. For example, W3C trace ID `4efaaf4d1e8720b39541901950019ee5` should be formatted as `1-4efaaf4d-1e8720b39541901950019ee5` when sending to X-Ray. X-Ray trace IDs include the original request time stamp in Unix epoch time, but this isn't required when sending W3C trace IDs in X-Ray format.
**Trace ID Security**
Trace IDs are visible in [response headers](xray-concepts.md#xray-concepts-tracingheader). Generate trace IDs with a secure random algorithm to ensure that attackers cannot calculate future trace IDs and send requests with those IDs to your application.
+ `start_time` – **number** that is the time the segment was created, in floating point seconds in epoch time. For example, `1480615200.010` or `1.480615200010E9`. Use as many decimal places as you need. Microsecond resolution is recommended when available.
+ `end_time` – **number** that is the time the segment was closed. For example, `1480615200.090` or `1.480615200090E9`. Specify either an `end_time` or `in_progress`.
+ `in_progress` – **boolean**, set to `true` instead of specifying an `end_time` to record that a segment is started, but is not complete. Send an in-progress segment when your application receives a request that will take a long time to serve, to trace the request receipt. When the response is sent, send the complete segment to overwrite the in-progress segment. Only send one complete segment, and one or zero in-progress segments, per request.
**Service Names**
A segment's `name` should match the domain name or logical name of the service that generates the segment. However, this is not enforced. Any application that has permission to [https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html](https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html) can send segments with any name.
The following fields are optional for segments.
**Optional Segment Fields**
+ `service` – An object with information about your application.
+ `version` – A string that identifies the version of your application that served the request.
+ `user` – A string that identifies the user who sent the request.
+ `origin` – The type of AWS resource running your application.
**Supported Values**
+ `AWS::EC2::Instance` – An Amazon EC2 instance.
+ `AWS::ECS::Container` – An Amazon ECS container.
+ `AWS::ElasticBeanstalk::Environment` – An Elastic Beanstalk environment.
When multiple values are applicable to your application, use the one that is most specific. For example, a Multicontainer Docker Elastic Beanstalk environment runs your application on an Amazon ECS container, which in turn runs on an Amazon EC2 instance. In this case you would set the origin to `AWS::ElasticBeanstalk::Environment` as the environment is the parent of the other two resources.
+ `parent_id` – A subsegment ID you specify if the request originated from an instrumented application. The X-Ray SDK adds the parent subsegment ID to the [tracing header](xray-concepts.md#xray-concepts-tracingheader) for downstream HTTP calls. In the case of nested subsegments, a subsegment can have a segment or a subsegment as its parent.
+ `http` – [`http`](#api-segmentdocuments-http) objects with information about the original HTTP request.
+ `aws` – [`aws`](#api-segmentdocuments-aws) object with information about the AWS resource on which your application served the request.
+ `error`, `throttle`, `fault`, and `cause` – [error](#api-segmentdocuments-errors) fields that indicate an error occurred and that include information about the exception that caused the error.
+ `annotations` – [`annotations`](#api-segmentdocuments-annotations) object with key-value pairs that you want X-Ray to index for search.
+ `metadata` – [`metadata`](#api-segmentdocuments-metadata) object with any additional data that you want to store in the segment.
+ `subsegments` – **array** of [`subsegment`](#api-segmentdocuments-subsegments) objects.
## Subsegments
You can create subsegments to record calls to AWS services and resources that you make with the AWS SDK, calls to internal or external HTTP web APIs, or SQL database queries. You can also create subsegments to debug or annotate blocks of code in your application. Subsegments can contain other subsegments, so a custom subsegment that records metadata about an internal function call can contain other custom subsegments and subsegments for downstream calls.
A subsegment records a downstream call from the point of view of the service that calls it. X-Ray uses subsegments to identify downstream services that don't send segments and create entries for them on the service graph.
A subsegment can be embedded in a full segment document or sent independently. Send subsegments separately to asynchronously trace downstream calls for long-running requests, or to avoid exceeding the maximum segment document size.
**Example Segment with embedded subsegment**
An independent subsegment has a `type` of `subsegment` and a `parent_id` that identifies the parent segment.
```
{
"trace_id" : "1-5759e988-bd862e3fe1be46a994272793",
"id" : "defdfd9912dc5a56",
"start_time" : 1461096053.37518,
"end_time" : 1461096053.4042,
"name" : "www.example.com",
"http" : {
"request" : {
"url" : "https://www.example.com/health",
"method" : "GET",
"user_agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7",
"client_ip" : "11.0.3.111"
},
"response" : {
"status" : 200,
"content_length" : 86
}
},
"subsegments" : [
{
"id" : "53995c3f42cd8ad8",
"name" : "api.example.com",
"start_time" : 1461096053.37769,
"end_time" : 1461096053.40379,
"namespace" : "remote",
"http" : {
"request" : {
"url" : "https://api.example.com/health",
"method" : "POST",
"traced" : true
},
"response" : {
"status" : 200,
"content_length" : 861
}
}
}
]
}
```
For long-running requests, you can send an in-progress segment to notify X-Ray that the request was received, and then send subsegments separately to trace them before completing the original request.
**Example In-progress segment**
```
{
"name" : "example.com",
"id" : "70de5b6f19ff9a0b",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979",
"in_progress": true
}
```
**Example Independent subsegment**
An independent subsegment has a `type` of `subsegment`, a `trace_id`, and a `parent_id` that identifies the parent segment.
```
{
"name" : "api.example.com",
"id" : "53995c3f42cd8ad8",
"start_time" : 1.478293361271E9,
"end_time" : 1.478293361449E9,
"type" : "subsegment",
"trace_id" : "1-581cf771-a006649127e371903a2de979"
"parent_id" : "defdfd9912dc5a56",
"namespace" : "remote",
"http" : {
"request" : {
"url" : "https://api.example.com/health",
"method" : "POST",
"traced" : true
},
"response" : {
"status" : 200,
"content_length" : 861
}
}
}
```
When the request is complete, close the segment by resending it with an `end_time`. The complete segment overwrites the in-progress segment.
You can also send subsegments separately for completed requests that triggered asynchronous workflows. For example, a web API may return a `OK 200` response immediately prior to starting the work that the user requested. You can send a full segment to X-Ray as soon as the response is sent, followed by subsegments for work completed later. As with segments, you can also send a subsegment fragment to record that the subsegment has started, and then overwrite it with a full subsegment once the downstream call is complete.
The following fields are required, or are conditionally required, for subsegments.
**Note**
Values are strings up to 250 characters unless noted otherwise.
**Required Subsegment Fields**
+ `id` – A 64-bit identifier for the subsegment, unique among segments in the same trace, in **16 hexadecimal digits**.
+ `name` – The logical name of the subsegment. For downstream calls, name the subsegment after the resource or service called. For custom subsegments, name the subsegment after the code that it instruments (e.g., a function name).
+ `start_time` – **number** that is the time the subsegment was created, in floating point seconds in epoch time, accurate to milliseconds. For example, `1480615200.010` or `1.480615200010E9`.
+ `end_time` – **number** that is the time the subsegment was closed. For example, `1480615200.090` or `1.480615200090E9`. Specify an `end_time` or `in_progress`.
+ `in_progress` – **boolean** that is set to `true` instead of specifying an `end_time` to record that a subsegment is started, but is not complete. Only send one complete subsegment, and one or zero in-progress subsegments, per downstream request.
+ `trace_id` – Trace ID of the subsegment's parent segment. Required only if sending a subsegment separately.
**X-Ray trace ID format**
An X-Ray `trace_id` consists of three numbers separated by hyphens. For example, `1-58406520-a006649127e371903a2de979`. This includes:
+ The version number, which is `1`.
+ The time of the original request in Unix epoch time using **8 hexadecimal digits**.
For example, 10:00AM December 1st, 2016 PST in epoch time is `1480615200` seconds or `58406520` in hexadecimal digits.
+ A globally unique 96-bit identifier for the trace in **24 hexadecimal digits**.
**Note**
X-Ray now supports trace IDs that are created using OpenTelemetry and any other framework that conforms with the [W3C Trace Context specification](https://www.w3.org/TR/trace-context/). A W3C trace ID must be formatted in X-Ray trace ID format when sending to X-Ray. For example, W3C trace ID `4efaaf4d1e8720b39541901950019ee5` should be formatted as `1-4efaaf4d-1e8720b39541901950019ee5` when sending to X-Ray. X-Ray trace IDs include the original request time stamp in Unix epoch time, but this isn't required when sending W3C trace IDs in X-Ray format.
+ `parent_id` – Segment ID of the subsegment's parent segment. Required only if sending a subsegment separately. In the case of nested subsegments, a subsegment can have a segment or a subsegment as its parent.
+ `type` – `subsegment`. Required only if sending a subsegment separately.
The following fields are optional for subsegments.
**Optional Subsegment Fields**
+ `namespace` – `aws` for AWS SDK calls; `remote` for other downstream calls.
+ `http` – [`http`](#api-segmentdocuments-http) object with information about an outgoing HTTP call.
+ `aws` – [`aws`](#api-segmentdocuments-aws) object with information about the downstream AWS resource that your application called.
+ `error`, `throttle`, `fault`, and `cause` – [error](#api-segmentdocuments-errors) fields that indicate an error occurred and that include information about the exception that caused the error.
+ `annotations` – [`annotations`](#api-segmentdocuments-annotations) object with key-value pairs that you want X-Ray to index for search.
+ `metadata` – [`metadata`](#api-segmentdocuments-metadata) object with any additional data that you want to store in the segment.
+ `subsegments` – **array** of [`subsegment`](#api-segmentdocuments-subsegments) objects.
+ `precursor_ids` – **array** of subsegment IDs that identifies subsegments with the same parent that completed prior to this subsegment.
## HTTP request data
Use an HTTP block to record details about an HTTP request that your application served (in a segment) or that your application made to a downstream HTTP API (in a subsegment). Most of the fields in this object map to information found in an HTTP request and response.
**`http`**
All fields are optional.
+ `request` – Information about a request.
+ `method` – The request method. For example, `GET`.
+ `url` – The full URL of the request, compiled from the protocol, hostname, and path of the request.
+ `user_agent` – The user agent string from the requester's client.
+ `client_ip` – The IP address of the requester. Can be retrieved from the IP packet's `Source Address` or, for forwarded requests, from an `X-Forwarded-For` header.
+ `x_forwarded_for` – (segments only) **boolean** indicating that the `client_ip` was read from an `X-Forwarded-For` header and is not reliable as it could have been forged.
+ `traced` – (subsegments only) **boolean** indicating that the downstream call is to another traced service. If this field is set to `true`, X-Ray considers the trace to be broken until the downstream service uploads a segment with a `parent_id` that matches the `id` of the subsegment that contains this block.
+ `response` – Information about a response.
+ `status` – **integer** indicating the HTTP status of the response.
+ `content_length` – **integer** indicating the length of the response body in bytes.
When you instrument a call to a downstream web api, record a subsegment with information about the HTTP request and response. X-Ray uses the subsegment to generate an inferred segment for the remote API.
**Example Segment for HTTP call served by an application running on Amazon EC2**
```
{
"id": "6b55dcc497934f1a",
"start_time": 1484789387.126,
"end_time": 1484789387.535,
"trace_id": "1-5880168b-fd5158284b67678a3bb5a78c",
"name": "www.example.com",
"origin": "AWS::EC2::Instance",
"aws": {
"ec2": {
"availability_zone": "us-west-2c",
"instance_id": "i-0b5a4678fc325bg98"
},
"xray": {
"sdk_version": "2.11.0 for Java"
},
},
"http": {
"request": {
"method": "POST",
"client_ip": "78.255.233.48",
"url": "http://www.example.com/api/user",
"user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0",
"x_forwarded_for": true
},
"response": {
"status": 200
}
}
```
**Example Subsegment for a downstream HTTP call**
```
{
"id": "004f72be19cddc2a",
"start_time": 1484786387.131,
"end_time": 1484786387.501,
"name": "names.example.com",
"namespace": "remote",
"http": {
"request": {
"method": "GET",
"url": "https://names.example.com/"
},
"response": {
"content_length": -1,
"status": 200
}
}
}
```
**Example Inferred segment for a downstream HTTP call**
```
{
"id": "168416dc2ea97781",
"name": "names.example.com",
"trace_id": "1-62be1272-1b71c4274f39f122afa64eab",
"start_time": 1484786387.131,
"end_time": 1484786387.501,
"parent_id": "004f72be19cddc2a",
"http": {
"request": {
"method": "GET",
"url": "https://names.example.com/"
},
"response": {
"content_length": -1,
"status": 200
}
},
"inferred": true
}
```
## Annotations
Segments and subsegments can include an `annotations` object containing one or more fields that X-Ray indexes for use with filter expressions. Fields can have string, number, or Boolean values (no objects or arrays). X-Ray indexes up to 50 annotations per trace.
**Example Segment for HTTP call with annotations**
```
{
"id": "6b55dcc497932f1a",
"start_time": 1484789187.126,
"end_time": 1484789187.535,
"trace_id": "1-5880168b-fd515828bs07678a3bb5a78c",
"name": "www.example.com",
"origin": "AWS::EC2::Instance",
"aws": {
"ec2": {
"availability_zone": "us-west-2c",
"instance_id": "i-0b5a4678fc325bg98"
},
"xray": {
"sdk_version": "2.11.0 for Java"
},
},
"annotations": {
"customer_category" : 124,
"zip_code" : 98101,
"country" : "United States",
"internal" : false
},
"http": {
"request": {
"method": "POST",
"client_ip": "78.255.233.48",
"url": "http://www.example.com/api/user",
"user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0",
"x_forwarded_for": true
},
"response": {
"status": 200
}
}
```
Keys must be alphanumeric in order to work with filters. Underscore is allowed. Other symbols and whitespace are not allowed.
## Metadata
Segments and subsegments can include a `metadata` object containing one or more fields with values of any type, including objects and arrays. X-Ray does not index metadata, and values can be any size, as long as the segment document doesn't exceed the maximum size (64 kB). You can view metadata in the full segment document returned by the [https://docs.aws.amazon.com/xray/latest/api/API_BatchGetTraces.html](https://docs.aws.amazon.com/xray/latest/api/API_BatchGetTraces.html) API. Field keys (`debug` in the following example) starting with `AWS.` are reserved for use by AWS-provided SDKs and clients.
**Example Custom subsegment with metadata**
```
{
"id": "0e58d2918e9038e8",
"start_time": 1484789387.502,
"end_time": 1484789387.534,
"name": "## UserModel.saveUser",
"metadata": {
"debug": {
"test": "Metadata string from UserModel.saveUser"
}
},
"subsegments": [
{
"id": "0f910026178b71eb",
"start_time": 1484789387.502,
"end_time": 1484789387.534,
"name": "DynamoDB",
"namespace": "aws",
"http": {
"response": {
"content_length": 58,
"status": 200
}
},
"aws": {
"table_name": "scorekeep-user",
"operation": "UpdateItem",
"request_id": "3AIENM5J4ELQ3SPODHKBIRVIC3VV4KQNSO5AEMVJF66Q9ASUAAJG",
"resource_names": [
"scorekeep-user"
]
}
}
]
}
```
## AWS resource data
For segments, the `aws` object contains information about the resource on which your application is running. Multiple fields can apply to a single resource. For example, an application running in a multicontainer Docker environment on Elastic Beanstalk could have information about the Amazon EC2 instance, the Amazon ECS container running on the instance, and the Elastic Beanstalk environment itself.
**`aws` (Segments)**
All fields are optional.
+ `account_id` – If your application sends segments to a different AWS account, record the ID of the account running your application.
+ `cloudwatch_logs` – Array of objects that describe a single CloudWatch log group.
+ `log_group` – The CloudWatch Log Group name.
+ `arn` – The CloudWatch Log Group ARN.
+ `ec2` – Information about an Amazon EC2 instance.
+ `instance_id` – The instance ID of the EC2 instance.
+ `instance_size` – The type of EC2 instance.
+ `ami_id` – The Amazon Machine Image ID.
+ `availability_zone` – The Availability Zone in which the instance is running.
+ `ecs` – Information about an Amazon ECS container.
+ `container` – The hostname of your container.
+ `container_id` – The full container ID of your container.
+ `container_arn` – The ARN of your container instance.
+ `eks` – Information about an Amazon EKS cluster.
+ `pod` – The hostname of your EKS pod.
+ `cluster_name` – The EKS cluster name.
+ `container_id` – The full container ID of your container.
+ `elastic_beanstalk` – Information about an Elastic Beanstalk environment. You can find this information in a file named `/var/elasticbeanstalk/xray/environment.conf` on the latest Elastic Beanstalk platforms.
+ `environment_name` – The name of the environment.
+ `version_label` – The name of the application version that is currently deployed to the instance that served the request.
+ `deployment_id` – **number** indicating the ID of the last successful deployment to the instance that served the request.
+ `xray` – Metadata about the type and version of instrumentation used.
+ `auto_instrumentation` – Boolean indicating whether auto-instrumentation was used (for example, the Java Agent).
+ `sdk_version` – The version of SDK or agent being used.
+ `sdk` – The type of SDK.
**Example AWS block with plugins**
```
"aws":{
"elastic_beanstalk":{
"version_label":"app-5a56-170119_190650-stage-170119_190650",
"deployment_id":32,
"environment_name":"scorekeep"
},
"ec2":{
"availability_zone":"us-west-2c",
"instance_id":"i-075ad396f12bc325a",
"ami_id":
},
"cloudwatch_logs":[
{
"log_group":"my-cw-log-group",
"arn":"arn:aws:logs:us-west-2:012345678912:log-group:my-cw-log-group"
}
],
"xray":{
"auto_instrumentation":false,
"sdk":"X-Ray for Java",
"sdk_version":"2.8.0"
}
}
```
For subsegments, record information about the AWS services and resources that your application accesses. X-Ray uses this information to create inferred segments that represent the downstream services in your service map.
**`aws` (Subsegments)**
All fields are optional.
+ `operation` – The name of the API action invoked against an AWS service or resource.
+ `account_id` – If your application accesses resources in a different account, or sends segments to a different account, record the ID of the account that owns the AWS resource that your application accessed.
+ `region` – If the resource is in a region different from your application, record the region. For example, `us-west-2`.
+ `request_id` – Unique identifier for the request.
+ `queue_url` – For operations on an Amazon SQS queue, the queue's URL.
+ `table_name` – For operations on a DynamoDB table, the name of the table.
**Example Subsegment for a call to DynamoDB to save an item**
```
{
"id": "24756640c0d0978a",
"start_time": 1.480305974194E9,
"end_time": 1.4803059742E9,
"name": "DynamoDB",
"namespace": "aws",
"http": {
"response": {
"content_length": 60,
"status": 200
}
},
"aws": {
"table_name": "scorekeep-user",
"operation": "UpdateItem",
"request_id": "UBQNSO5AEM8T4FDA4RQDEB94OVTDRVV4K4HIRGVJF66Q9ASUAAJG",
}
}
```
## Errors and exceptions
When an error occurs, you can record details about the error and exceptions that it generated. Record errors in segments when your application returns an error to the user, and in subsegments when a downstream call returns an error.
**error types**
Set one or more of the following fields to `true` to indicate that an error occurred. Multiple types can apply if errors compound. For example, a `429 Too Many Requests` error from a downstream call may cause your application to return `500 Internal Server Error`, in which case all three types would apply.
+ `error` – **boolean** indicating that a client error occurred (response status code was 4XX Client Error).
+ `throttle` – **boolean** indicating that a request was throttled (response status code was *429 Too Many Requests*).
+ `fault` – **boolean** indicating that a server error occurred (response status code was 5XX Server Error).
Indicate the cause of the error by including a **cause** object in the segment or subsegment.
**`cause`**
A cause can be either a **16 character** exception ID or an object with the following fields:
+ `working_directory` – The full path of the working directory when the exception occurred.
+ `paths` – The **array** of paths to libraries or modules in use when the exception occurred.
+ `exceptions` – The **array** of **exception** objects.
Include detailed information about the error in one or more **exception** objects.
**`exception`**
All fields are optional.
+ `id` – A 64-bit identifier for the exception, unique among segments in the same trace, in **16 hexadecimal digits**.
+ `message` – The exception message.
+ `type` – The exception type.
+ `remote` – **boolean** indicating that the exception was caused by an error returned by a downstream service.
+ `truncated` – **integer** indicating the number of stack frames that are omitted from the `stack`.
+ `skipped` – **integer** indicating the number of exceptions that were skipped between this exception and its child, that is, the exception that it caused.
+ `cause` – Exception ID of the exception's parent, that is, the exception that caused this exception.
+ `stack` – **array** of **stackFrame** objects.
If available, record information about the call stack in **stackFrame** objects.
**`stackFrame`**
All fields are optional.
+ `path` – The relative path to the file.
+ `line` – The line in the file.
+ `label` – The function or method name.
## SQL queries
You can create subsegments for queries that your application makes to an SQL database.
**`sql`**
All fields are optional.
+ `connection_string` – For SQL Server or other database connections that don't use URL connection strings, record the connection string, excluding passwords.
+ `url` – For a database connection that uses a URL connection string, record the URL, excluding passwords.
+ `sanitized_query` – The database query, with any user provided values removed or replaced by a placeholder.
+ `database_type` – The name of the database engine.
+ `database_version` – The version number of the database engine.
+ `driver_version` – The name and version number of the database engine driver that your application uses.
+ `user` – The database username.
+ `preparation` – `call` if the query used a `PreparedCall`; `statement` if the query used a `PreparedStatement`.
**Example Subsegment with an SQL Query**
```
{
"id": "3fd8634e78ca9560",
"start_time": 1484872218.696,
"end_time": 1484872218.697,
"name": "ebdb@aawijb5u25wdoy.cpamxznpdoq8.us-west-2.rds.amazonaws.com",
"namespace": "remote",
"sql" : {
"url": "jdbc:postgresql://aawijb5u25wdoy.cpamxznpdoq8.us-west-2.rds.amazonaws.com:5432/ebdb",
"preparation": "statement",
"database_type": "PostgreSQL",
"database_version": "9.5.4",
"driver_version": "PostgreSQL 9.4.1211.jre7",
"user" : "dbuser",
"sanitized_query" : "SELECT * FROM customers WHERE customer_id=?;"
}
}
```