# Use a console Use a console if you want a graphical user interface (GUI) that requires minimal coding. Users that are new to X-Ray can get started quickly using pre-built visualizations, and performing basic tasks. You can do the following directly from the console: + Enable X-Ray. + View high-level summaries of your application's performance. + Check the health status of your applications. + Identify high-level errors. + View basic trace summaries. You can use either the Amazon CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/) or the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home) to interact with X-Ray. ## Use the Amazon CloudWatch console The CloudWatch console includes new X-Ray functionality that is redesigned from the X-Ray console to make it easier to use. If you use the CloudWatch console, you can view CloudWatch logs and metrics along with X-Ray trace data. Use the CloudWatch console to view and analyze data including the following: + X-Ray traces – View, analyze and filter traces associated with your application as it serves a request. Use these traces to find high latencies, debug errors, and optimize your application workflow. View a trace map and service map to see visual representations of your application workflow. + Logs – View, analyze and filter logs that your application produces. Use logs to troubleshoot errors and set up monitoring based on specific log values. + Metrics – Measure and monitor your application performance using metrics that your resources emit or create your own metrics. View these metrics in graphs and charts. + Monitoring networks and infrastructure – Monitor major networks for outages and the health and performance of your infrastructure including containerized applications, other AWS services, and clients. + All of the functionality from the X-Ray console listed in the following **Use the X-Ray console** section. For more information about the CloudWatch console, see [Getting started with Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/GettingStarted.html). Login the Amazon CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). ## Use the X-Ray console The X-Ray console offers distributed tracing for application requests. Use the X-Ray console if you want a simpler console experience or don’t want to update your application code. AWS is no longer developing the X-Ray console. The X-Ray console contains the following features for instrumented applications: + [Insights](xray-console-insights.md) – Automatically detect anomalies in your application’s performance and find the underlying causes. Insights are included in the CloudWatch console under **Insights**. For more information, see the **Use X-Ray Insights ** in [Use the X-Ray console](#xray-console). + Service map – View a graphical structure of your application and its connections with clients, resources, services, and dependencies. + Traces – See an overview of traces that are generated by your application as it serves a request. Use trace data to understand how your application performs against basic metrics including HTTP response and response time. + Analytics – Interpret, explore and analyze trace data using graphs for response time distribution. + Configuration – Create customized traces to change the default configurations for the following: + Sampling – Create a rule that defines how often to sample your application for trace information. For more information, see ** Configure sampling rules** in [Use the X-Ray console](#xray-console) . + [Encryption](xray-console-encryption.md) – Encrypt data at rest using a key that you can audit or disable using AWS Key Management Service. + Groups – Use a filter expression to define a group of traces with a common feature such as the name of a url or a response time. For more information, see [Configure groups](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray-interface-console.html#xray-console-groups). Login the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home). ## Explore the X-Ray console Use the X-Ray console to view a map of services and associated traces for requests that your applications serve, and to configure groups and sampling rules which affect how traces are sent to X-Ray. **Note** The X-Ray Service map and CloudWatch ServiceLens map have been combined into the X-Ray trace map within the Amazon CloudWatch console. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/) and choose **Trace Map** under **X-Ray traces** from the left navigation pane. CloudWatch now includes [Application Signals](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html), which can discover and monitor your application services, clients, Synthetics canaries, and service dependencies. Use Application Signals to see a list or visual map of your services, view health metrics based on your service level objectives (SLOs), and drill down to see correlated X-Ray traces for more detailed troubleshooting. The primary X-Ray console page is the trace map, which is a visual representation of the JSON service graph that X-Ray generates from the trace data generated by your applications. The map consists of service nodes for each application in your account that serves requests, upstream client nodes that represent the origins of the requests, and downstream service nodes that represent web services and resources used by an application while processing a request. There are additional pages for viewing traces and trace details, and configuring groups and sampling rules. View the console experience for X-Ray and compare with the CloudWatch console in the following sections. **Topics** + [ ## Use the Amazon CloudWatch console ](#aws-xray-interface-console-cw) + [ ## Use the X-Ray console ](#xray-console) + [ ## Explore the X-Ray console ](#xray-console-explore) + [ # Using the X-Ray trace map ](xray-console-servicemap.md) + [ # Viewing traces and trace details ](xray-console-traces.md) + [ # Using filter expressions ](xray-console-filters.md) + [ # Cross-account tracing ](xray-console-crossaccount.md) + [ # Tracing event-driven applications ](xray-tracelinking.md) + [ # Using latency histograms ](xray-console-histograms.md) + [ # Using X-Ray insights ](xray-console-insights.md) + [ # Interacting with the Analytics console ](xray-console-analytics.md) + [ # Configuring groups ](xray-console-groups.md) + [ # Configuring sampling rules ](xray-console-sampling.md) + [ # Configuring adaptive sampling ](xray-adaptive-sampling.md) + [ # Console deep linking ](xray-console-deeplinks.md) # Using the X-Ray trace map View the X-Ray trace map to identify services where errors are occurring, connections with high latency, or traces for requests that were unsuccessful. **Note** CloudWatch now includes [Application Signals](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html), which can discover and monitor your application services, clients, synthetics canaries, and service dependencies. Use Application Signals to see a list or visual map of your services, view health metrics based on your service level objectives (SLOs), and drill down to see correlated X-Ray traces for more detailed troubleshooting. The X-Ray service map and CloudWatch ServiceLens map are combined into the X-Ray trace map within the Amazon CloudWatch console. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/) and choose **Trace Map** under **X-Ray traces** from the left navigation pane. ## Viewing the trace map The trace map is a visual representation of the trace data that's generated by your applications. The map shows service nodes that serve requests, upstream client nodes that represent the origins of the requests, and downstream service nodes that represent web services and resources that are used by an application while processing a request. The trace map displays a connected view of traces across event-driven applications that use Amazon SQS and Lambda. For more information, see [tracing event-driven applications](xray-tracelinking.md). The trace map also supports [cross-account tracing](xray-console-crossaccount.md), displaying nodes from multiple accounts in a single map. ------ #### [ CloudWatch console ] **To view the trace map in the CloudWatch console** 1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/). Choose **Trace Map** under the **X-Ray Traces** section in the left navigation pane. ![\[CloudWatch console trace map page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-cw.png) 1. Choose a service node to view requests for that node, or an edge between two nodes to view requests that traveled that connection. 1. Additional information is displayed below the trace map, including tabs for metrics, alerts, and response time distribution. On the **Metrics** tab, select a range within each graph to drill down to view more detail, or choose **Faults** or **Errors** options to filter traces. On the **Response time distribution** tab, select a range within the graph to filter traces by response time. ![\[Dashboard showing latency, requests, and faults metrics for an ElasticBeanstalk environment.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-cw-servicemap-node-detail.png) 1. View traces by choosing **View traces**, or if a filter has been applied, choose **View filtered traces**. 1. Choose **View logs** to see CloudWatch logs associated with the selected node. Not all trace map nodes support viewing logs. See [troubleshooting CloudWatch logs](xray-troubleshooting.md#xray-troubleshooting-Nologs) for more information. The trace map indicates issues within each node by outlining it with colors: + **Red** for server faults (500 series errors) + **Yellow** for client errors (400 series errors) + **Purple** for throttling errors (429 Too Many Requests) If your trace map is large, use the on-screen controls or mouse to zoom in and out and move the map around. ------ #### [ X-Ray console ] **To view the Service map** 1. Open the [X-Ray console](https://console.aws.amazon.com/xray/home#). The service map is displayed by default. You can also choose **Service Map** from the left navigation pane. ![\[X-Ray console service map page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-xray.png) 1. Choose a service node to view requests for that node, or an edge between two nodes to view requests that traveled that connection. 1. Use the response distribution [histogram](xray-console-histograms.md) to filter traces by duration, and select status codes for which you want to view traces. Then choose **View traces** to open the trace list with the filter expression applied. ![\[Response distribution graph showing latency peaks and service details for Scorekeep AWS ECS container.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-nodedetail-xray.png) The service map indicates the health of each node by coloring it based on the ratio of successful calls to errors and faults: + **Green** for successful calls + **Red** for server faults (500 series errors) + **Yellow** for client errors (400 series errors) + **Purple** for throttling errors (429 Too Many Requests) If your service map is large, use the on-screen controls or mouse to zoom in and out and move the map around. ------ **Note** The X-Ray trace map can display up to 10,000 nodes. In rare scenarios where the total number of service nodes exceeds this limit, you may receive an error and be unable to display a complete trace map in the console. ## Filtering the trace map by group Using a [filter expression](xray-console-filters.md), you can define criteria by which to include traces within a group. Use the following steps to then display that specific group in the trace map. ------ #### [ CloudWatch console ] Choose a group name from the group filter on the top-left of the trace map. ![\[Search bar for filtering by X-Ray group, with "TestGroup" displayed as an option.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-groups-cw.png) ------ #### [ X-Ray console ] Choose a group name from the drop-down menu to the left of the search bar. ![\[Drop-down menu showing Default, TestGroup, Create group, and Learn more options.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-select-console.png) ------ The service map will now be filtered to display traces that match the filter expression of the selected group. ## Trace map legend and options The trace map includes a legend and several options for customizing the map display. ------ #### [ CloudWatch console ] Choose the **Legend and options** drop-down at the top-right of the map. Choose what is displayed within nodes, including: + **Metrics** displays the average response time and number of traces sent per minute during the chosen time range. + **Nodes** displays the service icon within each node. Choose additional map settings from the **Preferences** pane, which can be accessed via the gear icon at the top-right of the map. These settings include selecting which metric is used to determine the size of each node, and which canaries should be displayed on the map. ------ #### [ X-Ray console ] Display the service map legend by choosing the **Map legend** link at the top-right of the map. Service map options can be chosen at the bottom-right of the trace map, including: + **Service Icons** toggles what is displayed within each node, displaying either the service icon, or the average response time and number of traces sent per minute during the chosen time range. + **Node sizing: None** sets all nodes to the same size. + **Node sizing: Health** sizes nodes by the number of impacted requests including errors, faults, or throttled requests. + **Node sizing: Traffic** sizes nodes by the total number of requests. ------ # Viewing traces and trace details Use the **Traces** page in the X-Ray console to find traces by URL, response code, or other data from the trace summary. After selecting a trace from the trace list, the **Trace details** page displays a map of service nodes that are associated with the selected trace and a timeline of trace segments. ## Viewing traces ------ #### [ CloudWatch console ] **To view traces in the CloudWatch console** 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the left navigation pane, choose **X-Ray traces**, then choose **Traces**. You can filter by group or enter a [filter expression](xray-console-filters.md). This filters the traces that are displayed in the **Traces** section at the bottom of the page. Alternatively, you can use the service map to navigate to a specific service node, and then view traces. This opens the **Traces** page with a query already applied. 1. Refine your query in the **Query refiners** section. To filter traces by a common attribute, choose an option from the down arrow next to **Refine query by**. The options include the following: + Node – Filter traces by service node. + Resource ARN – Filter traces by a resource associated with a trace. Examples of these resources include Amazon Elastic Compute Cloud (Amazon EC2) instance, an AWS Lambda function, or an Amazon DynamoDB table. + User – Filter traces with a user ID. + Error root cause message – Filter traces by error root cause. + URL – Filter traces by a URL path used by your application. + HTTP status code – Filter traces by the HTTP status code returned by your application. You can specify a custom response code or select from the following: + `200` – The request was successful. + `401` – The request lacked valid authentication credentials. + `403` – The request lacked valid permissions. + `404` – The server could not find the requested resource. + `500` – The server encountered an unexpected condition and generated an internal error. Choose one or more entries and then choose **Add to query** to add to the filter expression at the top of the page. 1. To find a single trace, enter a [trace ID](xray-api-sendingdata.md#xray-api-traceids) directly into the query field. You can use X-Ray format or World Wide Web Consortium (W3C) format. For example, a trace that's created using the [AWS Distro for OpenTelemetry](xray-instrumenting-your-app.md#xray-instrumenting-opentel) is in W3C format. **Note** When you query traces that are created with a W3C-format trace ID, the console displays the matching trace in X-Ray format. For example, if you query for `4efaaf4d1e8720b39541901950019ee5` in W3C format, the console displays the X-Ray equivalent: `1-4efaaf4d-1e8720b39541901950019ee5`. 1. Choose **Run query** at any time to display a list of matching traces within the **Traces** section at the bottom of the page. 1. To display the **Trace details** page for a single trace, select a trace ID from the list. The following image shows a **Trace map** containing service nodes associated with the trace and edges between the nodes representing the path taken by segments that compose the trace. A **Trace summary** follows the **Trace Map**. The summary contains information about a sample `GET` operation, its **Response Code**, the **Duration** that the trace took to run, and the **Age** of the request. The **Segments Timeline** follows the **Trace Summary** that shows the duration of trace segments and subsegments. ![\[A trace map, summary and segments timeline details information about service nodes and the segments in the trace.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/xray-trace-details-cw.png) If you have an event-driven application that uses Amazon SQS and Lambda, you can see a connected view of traces for each request in the **Trace map**. In the map, traces from message producers are linked to traces from AWS Lambda consumers and are displayed as a dashed-line edge. For more information about event-driven applications, see [Tracing event-driven applications](xray-tracelinking.md). The **Traces** and **Trace details** pages also support [cross-account tracing](xray-console-crossaccount.md), which can list traces from multiple accounts in the trace list and inside a single trace map. ------ #### [ X-Ray console ] **To view traces in the X-Ray console** 1. Open the [Traces](https://console.aws.amazon.com/xray/home#/traces) page in the X-Ray console. The **Trace overview** panel shows a list of traces that are grouped by common features including **Error root causes**, **ResourceARN**, and **InstanceId**. 1. To select a common feature to view a grouped set of traces, expand the down arrow next to **Group by**. The following illustration shows a trace overview of traces that are grouped by URL for the [AWS X-Ray sample application](xray-scorekeep.md), and a list of associated traces. ![\[Example trace overview grouped by URL, followed by a trace list with details including ID, Method, and Response.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-traces.png) 1. Choose the **ID** of a trace to view it under the **Trace list**. You can also choose **Service map** in the navigation pane to view traces for a specific service node. Then you can view traces that are associated with that node. The **Timeline** tab shows the request flow for the trace, and includes the following: + A map of the path for each segment in the trace. + How long it took for the segment to reach a node in the trace map. + How many requests were made to the node in the trace map. The following illustration shows an example **Trace Map** associated with a `GET` request made to a sample application. The arrows show the path that each segment took to complete the request. The service nodes show the number of requests made during the `GET` request. ![\[Trace map followed by a timeline with segments, their duration, origin, and end with respect to each other.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/xray-trace-details.png) For more information about the **Timeline** tab, see the following **Exploring the trace timeline** section. The **Raw data** tab shows information about the trace, and the segments and subsegments that compose the trace, in `JSON` format. This information may include the following: + Timestamps + Unique IDs + Resources associated with the segment or subsegment + The source, or origin, of the segment or subsegment + Additional information about the request to your application such as the response from an HTTP request ------ ## Exploring the trace timeline The **Timeline** section shows a hierarchy of segments and subsegments next to a horizontal bar that corresponds to time they used to complete their tasks. The first entry in the list is the segment, which represents all data recorded by the service for a single request. Subsegments are indented and listed following the segment. Columns contain information about each segment. ------ #### [ CloudWatch console ] In the CloudWatch console, the **Segments Timeline** provides the following information: + The first column: Lists the segments and subsegments in the selected trace. + The **Segment status** column: Lists the status outcome of each segment and subsegment. + The **Response code** column: Lists an HTTP response status code to a browser request made by the segment or subsegment, when available. + The **Duration** column: Lists how long the segment or subsegment ran. + The **Hosted in** column: Lists the namespace or environment where the segment or subsegment is ran, if applicable. For more information, see [Dimensions collected and dimension combinations](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AppSignals-StandardMetrics.html#AppSignals-StandardMetrics-Dimensions). + The last column: Displays horizontal bars that correspond to the duration that the segment or subsegment ran, in relation to the other segments or subsegments in the timeline. To group the list of segments and subsegments by service node, turn on **Group by nodes**. ------ #### [ X-Ray console ] In the trace details page, choose the **Timeline** tab to see the timeline for each segment and subsegment that makes up a trace. In the X-Ray console, the **Timeline** provides the following information: + The **Name** column: Lists the names of the segments and subsegments in the trace. + The **Res.** column: Lists an HTTP response status code to a browser request made by the segment or subsegment, when available. + The **Duration** column: Lists how long the segment or subsegment ran. + The **Status** column: Lists the outcome of the segment or subsegment status. + The last column: Displays horizontal bars that correspond to the duration that the segment or subsegment ran, in relation to the other segments or subsegments in the timeline. To see the raw trace data that the console uses to generate the timeline, choose the **Raw data** tab. The raw data shows you information about the trace, and the segments and subsegments that compose the trace in `JSON` format. This information may include the following: + Timestamps + Unique IDs + Resources associated with the segment or subsegment + The source, or origin, of the segment or subsegment + Additional information about the request to your application such as the response from an HTTP request. ------ When you use an instrumented AWS SDK, HTTP, or SQL client to make calls to external resources, the X-Ray SDK records subsegments automatically. You can also use the X-Ray SDK to record custom subsegments for any function or block of code. Additional subsegments that are recorded while a custom subsegment are open become children of the custom subsegment. ## Viewing segment details From the trace **Timeline**, choose the name of a segment to view its details. The **Segment details** panel shows the **Overview**, **Resources**, **Annotations**, **Metadata**, **Exceptions**, and **SQL** tabs. The following apply: + The **Overview** tab shows information about the request and response. Information includes the name, start time, end time, duration, the request URL, request operation, request response code, and any errors and faults. + The **Resources** tab for a segment shows information from the X-Ray SDK and about the AWS resources running your application. Use the Amazon EC2, AWS Elastic Beanstalk, or Amazon ECS plugins for the X-Ray SDK to record service-specific resource information. For more information about plugins, see the **Service plugins** section in [Configuring the X-Ray SDK for Java](xray-sdk-java-configuration.md). + The remaining tabs show **Annotations**, **Metadata**, and **Exceptions** that are recorded for the segment. Exceptions are captured automatically when they are generated from an instrumented request. Annotations and metadata contain additional information that you record by using the operations that the X-Ray SDK provides. To add annotations or metadata to your segments, use the X-Ray SDK. For more information, see the language-specific link listed under Instrumenting your application with AWS X-Ray SDKs in [Instrumenting your application for AWS X-Ray](xray-instrumenting-your-app.md). ## Viewing subsegment details From the trace timeline, choose the name of a subsegment to view its details: + The **Overview** tab contains information about the request and response. This includes the name, start time, end time, duration, the request URL, request operation, request response code, and any errors and faults. For subsegments generated with instrumented clients, the **Overview** tab contains information about the request and response from your application's point of view. + The **Resources** tab for a subsegment shows details about the AWS resources that were used to run the subsegment. For example, the resources tab may include an AWS Lambda function ARN, information about a DynamoDB table, any operation that is called, and request ID. + The remaining tabs show **Annotations**, **Metadata**, and **Exceptions** recorded on the subsegment. Exceptions are captured automatically when they are generated from an instrumented request. Annotations and metadata contain additional information that you record by using the operations that the X-Ray SDK provides. Use the X-Ray SDK to add annotations or metadata to your segments. For more information, see the language-specific link listed under **Instrumenting your application with AWS X-Ray SDKs** in [Instrumenting your application for AWS X-Ray](xray-instrumenting-your-app.md). For custom subsegments, the **Overview** tab shows the name of the subsegment, which you can set to specify the area of the code or function that it records. For more information, see the language-specific link listed under **Instrumenting your application with AWS X-Ray SDKs** in [Generating custom subsegments with the X-Ray SDK for Java](xray-sdk-java-subsegments.md). The following image shows the **Overview** tab for a custom subsegment. The overview contains the subsegment ID, parent ID, Name, start and end times, duration, status and errors or faults. ![\[Overview information about a subsegment including ID, parent ID, Name, times, errors, and faults.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-PUTrules-customsubsegment-overview.png) The **Metadata** tab for a custom subsegment contains information in JSON format about resources used by that subsegment. # Using filter expressions Use *filter expressions* to view a trace map or traces for a specific request, service, connection between two services (an edge), or requests that satisfy a condition. X-Ray provides a filter expression language for filtering requests, services, and edges based on data in request headers, response status, and indexed fields on the original segments. When you choose a time period of traces to view in the X-Ray console, you might get more results than the console can display. In the upper-right corner, the console shows the number of traces that it scanned and whether there are more traces available. You can use a filter expression to narrow the results to just the traces that you want to find. **Topics** + [ ## Filter expression details ](#xray-console-filters-details) + [ ## Using filter expressions with groups ](#groups) + [ ## Filter expression syntax ](#console-filters-syntax) + [ ## Boolean keywords ](#console-filters-boolean) + [ ## Number keywords ](#console-filters-number) + [ ## String keywords ](#console-filters-string) + [ ## Complex keywords ](#console-filters-complex) + [ ## id function ](#console-filters-functions) ## Filter expression details When you [choose a node in the trace map](xray-console-servicemap.md), the console constructs a filter expression based on the service name of the node, and the types of error present based on your selection. To find traces that show performance issues or that relate to specific requests, you can adjust the expression that the console provides or create your own. If you add annotations with the X-Ray SDK, you can also filter based on the presence of an annotation key or the value of a key. **Note** If you choose a relative time range in the trace map and choose a node, the console converts the time range to an absolute start and end time. To ensure that the traces for the node appear in the search results, and avoid scanning times when the node wasn't active, the time range only includes times when the node sent traces. To search relative to the current time, you can switch back to a relative time range in the traces page and scan again. If there are still more results available than the console can show, the console shows you how many traces matched and the number of traces scanned. The percentage shown is the percentage of the selected time frame that was scanned. To ensure that you see all matching traces represented in the results, narrow your filter expression further, or choose a shorter time frame. To get the freshest results first, the console starts scanning at the end of the time range and works backward. If there are a large number of traces, but few results, the console splits the time range into chunks and scans them in parallel. The progress bar shows the parts of the time range that have been scanned. ![\[Progress bar showing 52% of time range scanned, with 49 matching traces found.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-tracescan-parallel.png) ## Using filter expressions with groups Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics. Groups are identified by their name or an Amazon Resource Name (ARN), and contain a filter expression. The service compares incoming traces to the expression and stores them accordingly. You can create and modify groups by using the dropdown menu to the left of the filter expression search bar. **Note** If the service encounters an error in qualifying a group, that group is no longer included in processing incoming traces and an error metric is recorded. For more information about groups, see [Configuring groups](xray-console-groups.md). ## Filter expression syntax Filter expressions can contain a *keyword*, a unary or binary *operator*, and a *value* for comparison. ``` keyword operator value ``` Different operators are available for different types of keyword. For example, `responsetime` is a number keyword and can be compared with operators related to numbers. **Example – requests where response time was greater than 5 seconds** ``` responsetime > 5 ``` You can combine multiple expressions in a compound expression by using the `AND` or `OR` operators. **Example – requests where the total duration was 5–8 seconds** ``` duration >= 5 AND duration <= 8 ``` Simple keywords and operators find issues only at the trace level. If an error occurs downstream, but is handled by your application and not returned to the user, a search for `error` will not find it. To find traces with downstream issues, you can use the [complex keywords](#console-filters-complex) `service()` and `edge()`. These keywords let you apply a filter expression to all downstream nodes, a single downstream node, or an edge between two nodes. For more granularity, you can filter services and edges by type with [the id() function](#console-filters-functions). ## Boolean keywords Boolean keyword values are either true or false. Use these keywords to find traces that resulted in errors. **Boolean keywords** + `ok` – Response status code was 2XX Success. + `error` – Response status code was 4XX Client Error. + `throttle` – Response status code was 429 Too Many Requests. + `fault` – Response status code was 5XX Server Error. + `partial` – Request has incomplete segments. + `inferred` – Request has inferred segments. + `first` – Element is the first of an enumerated list. + `last` – Element is the last of an enumerated list. + `remote` – Root cause entity is remote. + `root` – Service is the entry point or root segment of a trace. Boolean operators find segments where the specified key is `true` or `false`. **Boolean operators** + none – The expression is true if the keyword is true. + `!` – The expression is true if the keyword is false. + `=`,`!=` – Compare the value of the keyword to the string `true` or `false`. These operators act the same as the other operators but are more explicit. **Example – response status is 2XX OK** ``` ok ``` **Example – response status is not 2XX OK** ``` !ok ``` **Example – response status is not 2XX OK** ``` ok = false ``` **Example – last enumerated fault trace has error name "deserialize"** ``` rootcause.fault.entity { last and name = "deserialize" } ``` **Example – requests with remote segments where coverage is greater than 0.7 and the service name is "traces"** ``` rootcause.responsetime.entity { remote and coverage > 0.7 and name = "traces" } ``` **Example – requests with inferred segments where the service type is "AWS:DynamoDB"** ``` rootcause.fault.service { inferred and name = traces and type = "AWS::DynamoDB" } ``` **Example – requests that have a segment with the name "data-plane" as the root** ``` service("data-plane") {root = true and fault = true} ``` ## Number keywords Use number keywords to search for requests with a specific response time, duration, or response status. **Number keywords** + `responsetime` – Time that the server took to send a response. + `duration` – Total request duration, including all downstream calls. + `http.status` – Response status code. + `index` – Position of an element in an enumerated list. + `coverage` – Decimal percentage of entity response time over root segment response time. Applicable only for response time root cause entities. **Number operators** Number keywords use standard equality and comparison operators. + `=`,`!=` – The keyword is equal to or not equal to a number value. + `<`,`<=`, `>`,`>=` – The keyword is less than or greater than a number value. **Example – response status is not 200 OK** ``` http.status != 200 ``` **Example – request where the total duration was 5–8 seconds** ``` duration >= 5 AND duration <= 8 ``` **Example – requests that completed successfully in less than 3 seconds, including all downstream calls** ``` ok !partial duration <3 ``` **Example – enumerated list entity that has an index greater than 5** ``` rootcause.fault.service { index > 5 } ``` **Example – requests where the last entity that has coverage greater than 0.8** ``` rootcause.responsetime.entity { last and coverage > 0.8 } ``` ## String keywords Use string keywords to find traces with specific text in the request headers, or specific user IDs. **String keywords** + `http.url` – Request URL. + `http.method` – Request method. + `http.useragent` – Request user agent string. + `http.clientip` – Requestor's IP address. + `user` – Value of the user field on any segment in the trace. + `name` – The name of a service or exception. + `type` – Service type. + `message` – Exception message. + `availabilityzone` – Value of the availabilityzone field on any segment in the trace. + `instance.id` – Value of the instance ID field on any segment in the trace. + `resource.arn` – Value of the resource ARN field on any segment in the trace. String operators find values that are equal to or contain specific text. Values must always be specified in quotation marks. **String operators** + `=`,`!=` – The keyword is equal to or not equal to a number value. + `CONTAINS` – The keyword contains a specific string. + `BEGINSWITH` , `ENDSWITH` – The keyword begins or ends with a specific string. **Example – http.url filter** ``` http.url CONTAINS "/api/game/" ``` To test if a field exists on a trace, regardless of its value, check to see if it contains the empty string. **Example – user filter** Find all traces with user IDs. ``` user CONTAINS "" ``` **Example – select traces with a fault root cause that includes a service named "Auth"** ``` rootcause.fault.service { name = "Auth" } ``` **Example – select traces with a response time root cause whose last service has a type of DynamoDB** ``` rootcause.responsetime.service { last and type = "AWS::DynamoDB" } ``` **Example – select traces with a fault root cause whose last exception has the message "access denied for account\$1id: 1234567890"** ``` rootcause.fault.exception { last and message = "Access Denied for account_id: 1234567890" ``` ## Complex keywords Use complex keywords to find requests based on service name, edge name, or annotation value. For services and edges, you can specify an additional filter expression that applies to the service or edge. For annotations, you can filter on the value of an annotation with a specific key using Boolean, number, or string operators. **Complex keywords** + `annotation[key]` – Value of an annotation with field *key*. The value of an annotation can be a Boolean, number, or string, so you can use any of the comparison operators of those types. You can use this keyword in combination with the `service` or `edge` keyword. An annotation key that contains dots (periods) must be wrapped in square brackets (**[ ]**). + `edge(source, destination) {filter}` – Connection between services *source* and *destination*. Optional curly braces can contain a filter expression that applies to segments on this connection. + `group.name / group.arn` – The value of a group's filter expression, referenced by group name or group ARN. + `json` – JSON root cause object. See [Getting data from AWS X-Ray](xray-api-gettingdata.md) for steps to create JSON entities programmatically. + `service(name) {filter}` – Service with name *name*. Optional curly braces can contain a filter expression that applies to segments created by the service. Use the service keyword to find traces for requests that hit a certain node on your trace map. Complex keyword operators find segments where the specified key has been set, or not set. **Complex keyword operators** + none – The expression is true if the keyword is set. If the keyword is of boolean type, it will evaluate to the boolean value. + `!` – The expression is true if the keyword is not set. If the keyword is of boolean type, it will evaluate to the boolean value. + `=`,`!=` – Compare the value of the keyword. + `edge(source, destination) {filter}` – Connection between services *source* and *destination*. Optional curly braces can contain a filter expression that applies to segments on this connection. + `annotation[key]` – Value of an annotation with field *key*. The value of an annotation can be a Boolean, number, or string, so you can use any of the comparison operators of those types. You can use this keyword in combination with the `service` or `edge` keyword. + `json` – JSON root cause object. See [Getting data from AWS X-Ray](xray-api-gettingdata.md) for steps to create JSON entities programmatically. Use the service keyword to find traces for requests that hit a certain node on your trace map. **Example – Service filter** Requests that included a call to `api.example.com` with a fault (500 series error). ``` service("api.example.com") { fault } ``` You can exclude the service name to apply a filter expression to all nodes on your service map. **Example – service filter** Requests that caused a fault anywhere on your trace map. ``` service() { fault } ``` The edge keyword applies a filter expression to a connection between two nodes. **Example – edge filter** Request where the service `api.example.com` made a call to `backend.example.com` that failed with an error. ``` edge("api.example.com", "backend.example.com") { error } ``` You can also use the `!` operator with service and edge keywords to exclude a service or edge from the results of another filter expression. **Example – service and request filter** Request where the URL begins with `http://api.example.com/` and contains `/v2/` but does not reach a service named `api.example.com`. ``` http.url BEGINSWITH "http://api.example.com/" AND http.url CONTAINS "/v2/" AND !service("api.example.com") ``` **Example – service and response time filter** Find traces where `http url` is set and response time is greater than 2 seconds. ``` http.url AND responseTime > 2 ``` For annotations, you can call all traces where `annotation[key]` is set, or use the comparison operators that correspond to the type of value. **Example – annotation with string value** Requests with an annotation named `gameid` with string value `"817DL6VO"`. ``` annotation[gameid] = "817DL6VO" ``` **Example – annotation is set** Requests with an annotation named `age` set. ``` annotation[age] ``` **Example – annotation is not set** Requests without an annotation named `age` set. ``` !annotation[age] ``` **Example – annotation with number value** Requests with annotation age with numerical value greater than 29. ``` annotation[age] > 29 ``` **Example – annotation in combination with service or edge** ``` service { annotation[request.id] = "917DL6VO" } ``` ``` edge { source.annotation[request.id] = "916DL6VO" } ``` ``` edge { destination.annotation[request.id] = "918DL6VO" } ``` **Example – group with user** Requests where traces meet the `high_response_time` group filter (e.g. `responseTime > 3`), and the user is named Alice. ``` group.name = "high_response_time" AND user = "alice" ``` **Example – JSON with root cause entity** Requests with matching root cause entities. ``` rootcause.json = #[{ "Services": [ { "Name": "GetWeatherData", "EntityPath": [{ "Name": "GetWeatherData" }, { "Name": "get_temperature" } ] }, { "Name": "GetTemperature", "EntityPath": [ { "Name": "GetTemperature" } ] } ] }] ``` ## id function When you provide a service name to the `service` or `edge` keyword, you get results for all nodes that have that name. For more precise filtering, you can use the `id` function to specify a service type in addition to a name to distinguish between nodes with the same name. Use the `account.id` function to specify a particular account for the service, when viewing traces from multiple accounts in a monitoring account. ``` id(name: "service-name", type:"service::type", account.id:"account-ID") ``` You can use the `id` function in place of a service name in service and edge filters. ``` service(id(name: "service-name", type:"service::type")) { filter } ``` ``` edge(id(name: "service-one", type:"service::type"), id(name: "service-two", type:"service::type")) { filter } ``` For example, AWS Lambda functions result in two nodes in the trace map; one for the function invocation, and one for the Lambda service. The two nodes have the same name but different types. A standard service filter will find traces for both. **Example – service filter** Requests that include an error on any service named `random-name`. ``` service("random-name") { error } ``` Use the `id` function to narrow the search to errors on the function itself, excluding errors from the service. **Example – service filter with id function** Requests that include an error on a service named `random-name` with type `AWS::Lambda::Function`. ``` service(id(name: "random-name", type: "AWS::Lambda::Function")) { error } ``` To search for nodes by type, you can also exclude the name entirely. **Example – service filter with id function and service type** Requests that include an error on a service with type `AWS::Lambda::Function`. ``` service(id(type: "AWS::Lambda::Function")) { error } ``` To search for nodes for a particular AWS account, specify an account ID. **Example – service filter with id function and account ID** Requests that include a service within a specific account ID `AWS::Lambda::Function`. ``` service(id(account.id: "account-id")) ``` # Cross-account tracing AWS X-Ray supports *cross-account observability*, enabling you to monitor and troubleshoot applications that span multiple accounts within an AWS Region. You can seamlessly search, visualize, and analyze your metrics, logs, and traces in any of the linked accounts as if you were operating in a single account. This provides a complete view of requests that travel across multiple accounts. You can view cross-account traces in the X-Ray trace map and traces pages within the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/). The shared observability data can include any of the following types of telemetry: + Metrics in Amazon CloudWatch + Log groups in Amazon CloudWatch Logs + Traces in AWS X-Ray + Applications in Amazon CloudWatch Application Insights ## Configure cross-account observability To turn on cross-account observability, set up one or more AWS *monitoring* accounts and link them with multiple *source* accounts. A monitoring account is a central AWS account that can view and interact with observability data that's generated from source accounts. A source account is an individual AWS account that generates observability data for the resources that it contains. Source accounts share their observability data with monitoring accounts. Traces are copied from each source account to up to five monitoring accounts. Copies of traces from source accounts to the first monitoring account are free. Copies of traces sent to additional monitoring accounts are charged to each source account, based on standard pricing. For more information, see [AWS X-Ray pricing](https://aws.amazon.com/xray/pricing/) and [Amazon CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/). To create links between monitoring accounts and source accounts, use the CloudWatch console or the new Observability Access Manager commands in the AWS CLI and API. For more information, see [CloudWatch cross-account observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html). **Note** X-Ray traces are billed to the AWS account where they're received. If a [sampled](xray-concepts.md#xray-concepts-sampling) request spans services across more than one AWS account, each account records a separate trace, and all traces share the same trace ID. To learn more about cross-account observability pricing, see [AWS X-Ray pricing](https://aws.amazon.com/xray/pricing/) and [Amazon CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/). ## Viewing cross-account traces Cross-account traces are displayed in the monitoring account. Each source account displays only local traces for that specific account. The following sections assume that you're signed in to the monitoring account and have opened the Amazon CloudWatch console. On both the trace map and traces pages, a monitoring account badge is displayed in the upper-right corner. ![\[Monitoring account badge\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-monitoring-account.png) ### Trace map In the CloudWatch console, choose **Trace Map** under **X-Ray traces** from the left navigation pane. By default, the trace map displays nodes for all source accounts that send traces to the monitoring account, and nodes for the monitoring account itself. On the trace map, choose **Filters** from the upper left to filter the trace map using the **Accounts** drop-down. After an account filter is applied, service nodes from accounts that don't match the current filter are grayed out. ![\[Filtered trace map\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-servicemap-account-filter.png) When you choose a service node, the node details pane includes the service's account ID and label. ![\[Node detail pane\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-servicemap-node-detail.png) In the upper-right corner of the trace map, choose **List view** to see a list of service nodes. The list of service nodes includes services from the monitoring account and all configured source accounts. Filter the list of nodes by **Account label** or **Account id** by choosing them from the **Nodes** filter. ![\[Filtered service list\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-servicelist-account-filter.png) ### Traces View trace details for traces that span multiple accounts by opening the CloudWatch console from the monitoring account, and choosing **Traces** under **X-Ray traces** in the left navigation pane. You can also open this page by choosing a node in the X-Ray **Trace Map**, and then choosing **View traces** from the node details pane. The **Traces** page supports querying by account ID. To get started, [enter a query](xray-console-filters.md) that includes one or more account IDs. The following example queries for traces that have passed through account ID X* *or *Y*: ``` service(id(account.id:"X")) OR service(id(account.id:"Y")) ``` ![\[Query traces by account\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-traces-query-by-account.png) Refine your query by **Account**. Select one or more accounts from the list, and choose **Add to query**. ![\[Refine trace query by account\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-traces-refine-by-account.png) ### Trace details View details for a trace by choosing it from the **Traces** list at the bottom of the **Traces** page. The **Trace details** displays, including a trace details map with service nodes from across all accounts that the trace passed through. Choose a specific service node to see its corresponding account. The **Segments timeline** section displays the account details for each segment in the timeline. ![\[Segments timeline\]](http://docs.aws.amazon.com/xray/latest/devguide/images/crossaccount-traces-segment-timeline.png) # Tracing event-driven applications AWS X-Ray supports tracing event-driven applications using Amazon SQS and AWS Lambda. Use the CloudWatch console to see a connected view of each request as it's queued with Amazon SQS and processed by one or more Lambda functions. Traces from upstream message producers are automatically linked to traces from downstream Lambda consumer nodes, creating an end-to-end view of the application. **Note** Each trace segment can be linked to up to 20 traces, while a trace can include a maximum of 100 links. In certain scenarios, linking additional traces may result in exceeding the [maximum trace document size](https://docs.aws.amazon.com/general/latest/gr/xray.html#limits_xray), causing a potentially incomplete trace. This can happen, for example, when a Lambda function with tracing enabled sends many SQS messages to a queue in a single invocation. If you encounter this issue, a mitigation is available which uses the X-Ray SDKs. See the X-Ray SDK for [Java](https://github.com/aws/aws-xray-sdk-java#oversampling-mitigation), [Node.js](https://github.com/aws/aws-xray-sdk-node/tree/master/packages/core#oversampling-mitigation), [Python](https://github.com/aws/aws-xray-sdk-python#oversampling-mitigation), [Go](https://github.com/aws/aws-xray-sdk-go#oversampling-mitigation), or [.NET](https://github.com/aws/aws-xray-sdk-dotnet#oversampling-mitigation) for more information. ## View linked traces in the trace map Use the **Trace Map** page within the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/) to view a trace map with traces from message producers that are linked to traces from Lambda consumers. These links are displayed with a dashed-line edge that connects the Amazon SQS node and downstream Lambda consumer nodes. ![\[Edge between Amazon SQS and Lambda nodes.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-servicemap-linkededge.png) Select a dashed-line edge to display a *received event age* histogram, which maps the spread of event age when it's received by consumers. The age is calculated each time an event is received. ![\[Edge with received event age histogram.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-servicemap-linkededgedetails-cw.png) ## View linked trace details **View trace details sent from a message producer, Amazon SQS queue, or Lambda consumer:** 1. Use the **Trace Map** to select a message producer, Amazon SQS, or Lambda consumer node. 1. Choose **View traces** from the node details pane to display a list of traces. You can also navigate directly to the **Traces** page within the CloudWatch console. 1. Choose a specific trace from the list to open the trace details page. The trace details page displays a message when the selected trace is part of a linked set of traces. ![\[Linked trace details\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-header.png) The trace details map displays the current trace, along with upstream and downstream linked traces, each of which are contained within a box that indicates the bounds of each trace. If the currently selected trace is linked to multiple upstream or downstream traces, the nodes within the upstream or downstream linked traces are stacked, and a **Select trace** button is displayed. ![\[Multiple linked upstream traces\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-tracemap.png) Beneath the trace details map, a timeline of trace segments displays, including upstream and downstream linked traces. If there are multiple upstream or downstream linked traces, their segment details can't be displayed. To view segment details for a single trace within a set of linked traces, [select a single trace](#xray-tracelinking-filterbatch) as described below. ![\[Segments timeline showing linked traces\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-timeline.png) ## Select a single trace within a set of linked traces **Filter a linked set of traces to a single trace, to see segment details in the timeline.** 1. Choose **Select trace** underneath the linked traces on the trace details map. A list of traces displays. ![\[Linked trace list\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-tracelist.png) 1. Select the radio button next to a trace to view it within the trace details map. 1. Choose **Cancel trace selection** to view the entire set of linked traces. ![\[Single linked trace\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-batch-tracedetails-filteredbatch.png) # Using latency histograms When you select a node or edge on an AWS X-Ray [trace map](xray-console-servicemap.md), the X-Ray console shows a latency distribution histogram. ## Latency Latency is the amount of time between when a request starts and when it completes. A histogram shows a distribution of latencies. It shows duration on the x-axis, and the percentage of requests that match each duration on the y-axis. This histogram shows a service that completes most requests in less than 300 ms. A small percentage of requests take up to 2 seconds, and a few outliers take more time. ![\[Latency histogram with duration on the x-axis and percentage of requests for each duration on the y-axis\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-servicemap-histogram.png) ## Interpreting service details Service histograms and edge histograms provide a visual representation of latency from the viewpoint of a service or requester. + Choose a *service node* by clicking the circle. X-Ray shows a histogram for requests served by the service. The latencies are those recorded by the service, and don't include any network latency between the service and the requester. + Choose an *edge* by clicking the line or arrow tip of the edge between two services. X-Ray shows a histogram for requests from the requester that were served by the downstream service. The latencies are those recorded by the requester, and include latency in the network connection between the two services. To interpret the **Service details** panel histogram, you can look for values that differ the most from the majority of values in the histogram. These *outliers* can be seen as peaks or spikes in the histogram, and you can view the traces for a specific area to investigate what's going on. To view traces filtered by latency, select a range on the histogram. Click where you want to start the selection and drag from left to right to highlight a range of latencies to include in the trace filter. ![\[Select a range to view traces by clicking where to start and dragging left to right to create the range for the trace filter\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-servicemap-servicedetails-selection.png) After selecting a range, you can choose **Zoom** to view just that portion of the histogram and refine your selection. ![\[Choose zoom to view the selected range in the histogram\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-servicemap-servicedetails-zoom.png) Once you have the focus set to the area you're interested in, choose **View traces**. # Using X-Ray insights AWS X-Ray continuously analyzes trace data in your account to identify emergent issues in your applications. When fault rates exceed the expected range, it creates an *insight* that records the issue and tracks its impact until it's resolved. With insights, you can: + Identify where in your application issues are occurring, the root cause of the issue, and associated impact. The impact analysis provided by insights enables you to derive the severity and priority of an issue. + Receive notifications as the issue changes over time. Insights notifications can be integrated with your monitoring and alerting solution using Amazon EventBridge. This integration enables you to send automated emails or alerts based on the severity of the issue. The X-Ray console identifies nodes with ongoing incidents in the trace map. To see a summary of the insight, choose the affected node. You can also view and filter insights by choosing **Insights** from the navigation pane on the left. ![\[Trace map node with insight summary.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-servicemap.png) X-Ray creates an insight when it detects an *anomaly* in one or more nodes of the service map. The service uses statistical modeling to predict the expected fault rates of services in your application. In the preceding example, the anomaly is an increase in faults from AWS Elastic Beanstalk. The Elastic Beanstalk server experienced multiple API call timeouts, causing an anomaly in the downstream nodes. ## Enable insights in the X-Ray console Insights must be enabled for each group you want to use insights features with. You can enable insights from the **Groups** page. 1. Open the [X-Ray console](https://console.aws.amazon.com/xray/home#). 1. Select an existing group or create a new one by choosing **Create group**, and then select **Enable Insights**. For more information about configuring groups in the X-Ray console, see [Configuring groups](xray-console-groups.md). 1. In the navigation pane on the left, choose **Insights**, and then choose an insight to view. ![\[List of insights in the X-Ray console.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights.png) **Note** X-Ray uses GetInsightSummaries, GetInsight, GetInsightEvents, and GetInsightImpactGraph API operations to retrieve data from insights. For more information, see [How AWS X-Ray works with IAM](security_iam_service-with-iam.md). ## Enable insights notifications With insights notifications, a notification is created for each insight event, such as when an insight is created, changes significantly, or is closed. Customers can receive these notifications through Amazon EventBridge events, and use conditional rules to take actions such as SNS notification, Lambda invocation, posting messages to an SQS queue, or any of the targets EventBridge supports. Insights notifications are emitted on a best-effort basis but are not guaranteed. For more information about targets, see [Amazon EventBridge Targets](https://docs.aws.amazon.com/eventbridge/latest/userguide/eventbridge-targets.html). You can enable insights notifications for any insights enabled group from the **Groups** page. **To enable notifications for an X-Ray group** 1. Open the [X-Ray console](https://console.aws.amazon.com/xray/home#). 1. Select an existing group or create a new one by choosing **Create group**, ensure that **Enable Insights** is selected, and then select **Enable Notifications**. For more information about configuring groups in the X-Ray console, see [Configuring groups](xray-console-groups.md). **To configure Amazon EventBridge conditional rules** 1. Open the [Amazon EventBridge console](https://console.aws.amazon.com/events/home). 1. Navigate to **Rules** in the left navigation bar, and choose **Create rule**. 1. Provide a name and description for the rule. 1. Choose **Event pattern**, and then choose **Custom pattern**. Provide a pattern containing `"source": [ "aws.xray" ]` and `"detail-type": [ "AWS X-Ray Insight Update" ]`. The following are some examples of possible patterns. + Event pattern to match all incoming events from X-Ray insights: ``` { "source": [ "aws.xray" ], "detail-type": [ "AWS X-Ray Insight Update" ] } ``` + Event pattern to match a specified **state** and **category**: ``` { "source": [ "aws.xray" ], "detail-type": [ "AWS X-Ray Insight Update" ], "detail": { "State": [ "ACTIVE" ], "Category": [ "FAULT" ] } } ``` 1. Select and configure the targets that you would like to invoke when an event matches this rule. 1. (Optional) Provide tags to more easily identify and select this rule. 1. Choose **Create**. **Note** X-Ray insights notifications sends events to Amazon EventBridge, which does not currently support customer managed keys. For more information, see [Data protection in AWS X-Ray](xray-console-encryption.md). ## Insight overview The overview page for an insight attempts to answer three key questions: + What is the underlying issue? + What is the root cause? + What is the impact? The **Anomalous services** section shows a timeline for each service that illustrates the change in fault rates during the incident. The timeline shows the number of traces with faults overlaid on a solid band that indicates the expected number of faults based on the amount of traffic recorded. The duration of the insight is visualized by the *Incident window*. The incident window begins when X-Ray observes the metric becoming anomalous and persists while the insight is active. The following example shows an increase in faults that caused an incident: ![\[Overview page of an X-Ray insight.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-overview.png) The **Root cause** section shows a trace map focused on the root cause service and the impacted path. You may hide the unaffected nodes by selecting the eye icon in the top right of the Root cause map. The root cause service is the farthest downstream node where X-Ray identified an anomaly. It can represent a service that you instrumented or an external service that your service called with an instrumented client. For example, if you call Amazon DynamoDB with an instrumented AWS SDK client, an increase in faults from DynamoDB results in an insight with DynamoDB as the root cause. To further investigate the root cause, select **View root cause details** on the root cause graph. You can use the **Analytics** page to investigate the root cause and related messages. For more information, see [Interacting with the Analytics console](xray-console-analytics.md). ![\[Overview page of an X-Ray insight.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-root-cause.png) Faults that continue upstream in the map can impact multiple nodes and cause multiple anomalies. If a fault is passed all the way back to the user that made the request, the result is a *client fault*. This is a fault in the root node of the trace map. The **Impact** graph provides a timeline of the client experience for the entire group. This experience is calculated based on percentages of the following states: **Fault**, **Error**, **Throttle**, and **Okay**. ![\[Impact graph for an X-Ray incident.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-impact.png) This example shows an increase in traces with a fault at the root node during the time of an incident. Incidents in downstream services don't always correspond to an increase in client errors. Choosing **Analyze insight** opens the X-Ray Analytics console in a window where you can dive deep into the set of traces causing the insight. For more information, see [Interacting with the Analytics console](xray-console-analytics.md). **Understanding impact** AWS X-Ray measures the impact caused by an ongoing issue as part of generating insights and notifications. The impact is measured in two ways: + Impact to the X-Ray [group](xray-console-groups.md) + Impact on the root cause service This impact is determined by the percentage of request that are failing or causing an error within a given time period. This impact analysis allows you to derive the severity and priority of the issue based on your particular scenario. This impact is available as part of the console experience in addition to insights notifications. **Deduplication** AWS X-Ray insights de-duplicates issues across multiple microservices. It uses anomaly detection to determine the service that is the root cause of an issue, determines if other related services are exhibiting anomalous behavior due to the same root cause, and records the result as a single insight. ## Review an insight's progress X-Ray reevaluates insights periodically until they are resolved, and records each notable intermediate change as a [notification](#xray-console-insight-notifications), which can be sent as an Amazon EventBridge event. This enables you to build processes and workflows to determine how the issue has changed over time, and take appropriate actions such as sending an email or integrating with an alerting system using EventBridge. You can review incident events in the **Impact Timeline** on the **Inspect** page. By default the timeline displays the most impacted service until you choose a different service. ![\[Inspect page with impact timeline.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-inspect.png) To see a trace map and graphs for an event, choose it from the impact timeline. The trace map shows services in your application that are affected by the incident. Under **Impact analysis**, graphs show fault timelines for the selected node and for clients in the group. ![\[Impact analysis graph for an X-Ray insight.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/console-insights-inspect-analysis.png) To take a deeper look at the traces involved in an incident, choose **Analyze event** on the **Inspect** page. You can use the **Analytics** page to refine the list of traces and identify affected users. For more information, see [Interacting with the Analytics console](xray-console-analytics.md). # Interacting with the Analytics console The AWS X-Ray Analytics console is an interactive tool for interpreting trace data to quickly understand how your application and its underlying services are performing. The console enables you to explore, analyze, and visualize traces through interactive response time and time-series graphs. When making selections in the Analytics console, the console constructs filters to reflect the selected subset of all traces. You can refine the active dataset with increasingly granular filters by clicking the graphs and the panels of metrics and fields that are associated with the current trace set. **Topics** + [ ## Console features ](#xray-console-analytics-features) + [ ## Response time distribution ](#xray-console-analytics-response) + [ ## Time series activity ](#xray-console-analytics-time) + [ ## Workflow examples ](#xray-console-analytics-workflows) + [ ## Observe faults on the service graph ](#xray-console-analytics-observe-faults) + [ ## Identify response time peaks ](#xray-console-analytics-response-peaks) + [ ## View all traces marked with a status code ](#xray-console-analytics-response-peaks) + [ ## View all items in a subgroup and associated to a user ](#xray-console-analytics-subgroups) + [ ## Compare two sets of traces with different criteria ](#xray-console-analytics-compare) + [ ## Identify a trace of interest and view its details ](#xray-console-analytics-identify) ## Console features The X-Ray Analytics console uses the following key features for grouping, filtering, comparing, and quantifying trace data. ### Features | Feature | Description | | --- | --- | | **Groups** | The initial selected group is `Default`. To change the retrieved group, select a different group from the menu to the right of the main filter expression search bar. To learn more about groups see, [Using filter expressions with groups](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-filters.html#groups). | | **Retrieved traces** | By default, the Analytics console generates graphs based on all traces in the selected group. Retrieved traces represent all traces in your working set. You can find the trace count in this tile. Filter expressions you apply to the main search bar refine and update the retrieved traces. | | **Show in charts/Hide from charts** | A toggle to compare the active group against the retrieved traces. To compare the data related to the group against any active filters, choose **Show in charts**. To remove this view from the charts, choose **Hide from charts**. | | **Filtered trace set A** | Through interactions with the graphs and tables, apply filters to create the criteria for **Filtered trace set A**. As the filters are applied, the number of applicable traces and the percentage of traces from the total that are retrieved are calculated within this tile. Filters populate as tags within the **Filtered trace set A** tile and can also be removed from the tile. | | **Refine** | This function updates the set of retrieved traces based on the filters applied to trace set A. Refining the retrieved trace set refreshes the working set of all traces retrieved based on the filters for trace set A. The working set of retrieved traces is a sampled subset of all traces in the group. | | **Filtered trace set B** | When created, **Filtered trace set B** is a copy of **Filtered trace set A**. To compare the two trace sets, make new filter selections that will apply to trace set B, while trace set A remains fixed. As the filters are applied, the number of applicable traces and the percentage of traces from the total retrieved are calculated within this tile. Filters populate as tags within the **Filtered trace set B** tile and can also be removed from the tile. | | **Response time root cause entity paths** | A table of recorded entity paths. X-Ray determines which path in your trace is the most likely cause for the response time. The format indicates a hierarchy of entities that are encountered, ending in a response time root cause. Use these rows to filter for recurring response time faults. For more information about customizing a root cause filter and getting data through the API see, [Retrieving and refining root cause analytics](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-gettingdata.html#xray-api-analytics). | | **Delta (**�**)** | A column that is added to the metrics tables when both trace set A and trace set B are active. The Delta column calculates the difference in percentage of traces between trace set A and trace set B. | ## Response time distribution The X-Ray Analytics console generates two primary graphs to help you visualize traces: **Response Time Distribution** and **Time Series Activity**. This section and the following provide examples of each, and explain the basics of how to read the graphs. The following are the colors associated with the response time line graph (the time series graph uses the same color scheme): + **All traces in the group** – gray + **Retrieved traces** – orange + **Filtered trace set A** – green + **Filtered trace set B** – blue **Example – Response time distribution** The response time distribution is a chart that shows the number of traces with a given response time. Click and drag to make selections within the response time distribution. This selects and creates a filter on the working trace set named `responseTime` for all traces within a specific response time. ![\[A chart that shows the response time distribution of traces.\]](http://docs.aws.amazon.com/xray/latest/devguide/images/analytics-responseTime.png) ## Time series activity The time series activity chart shows the number of traces at a given time period. The color indicators mirror the line graph colors of the response time distribution. The darker and fuller the color block within the activity series, the more traces are represented at the given time. **Example – Time series activity** Click and drag to make selections within the time series activity graph. This selects and creates a filter named `timerange` on the working trace set for all traces within a specific range of time. ![\[Make a selection and create filter\]](http://docs.aws.amazon.com/xray/latest/devguide/images/analytics-timeSeries.png) ## Workflow examples The following examples show common use cases for the X-Ray Analytics console. Each example demonstrates a key function of the console experience. As a group, the examples follow a basic troubleshooting workflow. The steps walk through how to first spot unhealthy nodes, and then how to interact with the Analytics console to automatically generate comparative queries. Once you have narrowed the scope through queries, you will finally look at the details of traces of interest to determine what is damaging the health of your service. ## Observe faults on the service graph The trace map indicates the health of each node by coloring it based on the ratio of successful calls to errors and faults. When you see a percentage of red on your node, it signals a fault. Use the X-Ray Analytics console to investigate it. For more information about how to read the trace map, see [Viewing the trace map](https://docs.aws.amazon.com/xray/latest/devguide/xray-console.html#xray-console-servicemap). ![\[Observe a fault\]](http://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-gettingstarted-servicemap-before-2021.png) ## Identify response time peaks Using the response time distribution, you can observe peaks in response time. By selecting the peak in response time, the tables below the graphs will update to expose all associated metrics, such as status codes. When you click and drag, X-Ray selects and creates a filter. It's shown in a gray shadow on top of the graphed lines. You can now drag that shadow left and right along the distribution to update your selection and filter. ![\[Make a selection and create filter\]](http://docs.aws.amazon.com/xray/latest/devguide/images/analytics-showFilterf.png) ## View all traces marked with a status code You can drill into traces within the selected peak by using the metrics tables below the graphs. By clicking a row in the **HTTP STATUS CODE** table, you automatically create a filter on the working dataset. For example, you could view all traces of status code 500. This creates a filter tag in the trace set tile named `http.status`. ## View all items in a subgroup and associated to a user Drill into the error set based on user, URL, response time root cause, or other predefined attributes. For example, to additionally filter the set of traces with a 500 status code, select a row from the **USERS** table. This results in two filter tags in the trace set tile: `http.status`, as designated previously, and `user`. ## Compare two sets of traces with different criteria Compare across various users and their POST requests to find other discrepancies and correlations. Apply your first set of filters. They are defined by a blue line in the response time distribution. Then select **Compare**. Initially, this creates a copy of the filters on trace set A. To proceed, define a new set of filters to apply to trace set B. This second set is represented by a green line. The following example shows different lines according to the blue and green color scheme. ![\[Line graph comparison\]](http://docs.aws.amazon.com/xray/latest/devguide/images/analytics-compareLines.png) ## Identify a trace of interest and view its details As you narrow your scope using the console filters, the trace list below the metrics tables becomes more meaningful. The trace list table combines information about **URL**, **USER**, and **STATUS CODE** into one view. For more insights, select a row from this table to open the trace's detail page and view its timeline and raw data. # Configuring groups Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics. You can use the AWS X-Ray console or X-Ray API to create and manage groups for your services. This topic describes how to create and manage groups by using the X-Ray console. For information about how to manage groups by using the X-Ray API, see [Groups](xray-api-configuration.md#xray-api-configuration-groups). You can create groups of traces for trace maps, traces, or analytics. When you create a group, the group becomes available as a filter on the group dropdown menu on all three pages: **Trace Map**, **Traces**, and **Analytics**. ![\[Group menu\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-menu.png) Groups are identified by their name or an Amazon Resource Name (ARN), and contain a filter expression. The service compares incoming traces to the expression and stores them accordingly. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). Updating a group's filter expression doesn't change data that's already recorded. The update applies only to subsequent traces. This can result in a merged graph of the new and old expressions. To avoid this, delete a current group and create a new one. **Note** Groups are billed by the number of retrieved traces that match the filter expression. For more information, see [AWS X-Ray pricing](https://aws.amazon.com/xray/pricing/). **Topics** + [ ## Create a group ](#xray-console-group-create-console) + [ ## Apply a group ](#xray-console-group-apply) + [ ## Edit a group ](#xray-console-group-edit) + [ ## Clone a group ](#xray-console-group-clone) + [ ## Delete a group ](#xray-console-group-delete) + [ ## View group metrics in Amazon CloudWatch ](#xray-console-group-cloudwatch) ## Create a group **Note** You can now configure X-Ray groups from within the Amazon CloudWatch console. You can also continue to use the X-Ray console. ------ #### [ CloudWatch console ] 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Settings** in the left navigation pane. 1. Choose **View settings** under **Groups** within the **X-Ray traces** section. 1. Choose **Create group** above the list of groups. 1. On the **Create group** page, enter a name for the group. A group name can have a maximum of 32 characters, and contain alphanumeric characters and dashes. Group names are case sensitive. 1. Enter a filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`. and requests to the service where the response time was greater than or equal to five seconds. ``` fault = true AND http.url CONTAINS "example/game" AND responsetime >= 5 ``` 1. In **Insights**, enable or disable insights access for the group. For more information about insights, see [Using X-Ray insights](xray-console-insights.md). ![\[Insights check boxes on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-insights-cw.png) 1. In **Tags**, choose **Add new tag** to enter a tag key, and optionally, a tag value. Continue to add additional tags as desired. Tag keys must be unique. To delete a tag, choose **Remove** underneath each tag. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md). ![\[Tag fields on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-tags-cw.png) 1. Choose **Create group**. ------ #### [ X-Ray console ] 1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home). 1. Open the **Create group** page from the **Groups** page in the left navigation pane, or from the group menu on one of the following pages: **Trace Map**, **Traces**, and **Analytics**. 1. On the **Create group** page, enter a name for the group. A group name can have a maximum of 32 characters, and contain alphanumeric characters and dashes. Group names are case sensitive. 1. Enter a filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`. and requests to the service where the response time was greater than or equal to five seconds. ``` fault = true AND http.url CONTAINS "example/game" AND responsetime >= 5 ``` 1. In **Insights**, enable or disable insights access for the group. For more information about insights, see [Using X-Ray insights](xray-console-insights.md). ![\[Insights check boxes on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-insights.png) 1. In **Tags**, enter a tag key, and optionally, a tag value. As you add a tag, a new line appears for you to enter another tag. Tag keys must be unique. To delete a tag, choose **X** at the end of the tag's row. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md). ![\[Tag fields on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-tags.png) 1. Choose **Create group**. ------ ## Apply a group ------ #### [ CloudWatch console ] 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Open one of the following pages from the navigation pane under **X-Ray traces**: + **Trace Map** + **Traces** 1. Enter a group name into the **Filter by X-Ray group** filter. The data shown on the page changes to match the filter expression set in the group. ------ #### [ X-Ray console ] 1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home). 1. Open one of the following pages from the navigation pane: + **Trace Map** + **Traces** + **Analytics** 1. On the group menu, choose the group that you created in [Create a group](#xray-console-group-create-console). The data shown on the page changes to match the filter expression set in the group. ------ ## Edit a group ------ #### [ CloudWatch console ] 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Settings** in the left navigation pane. 1. Choose **View settings** under **Groups** within the **X-Ray traces** section. 1. Choose a group from the **Groups** section and then choose **Edit**. 1. Although you can't rename a group, you can update the filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`, where the request URL address contains `example/game`, and response time for requests was greater than or equal to five seconds. ``` fault = true AND http.url CONTAINS "example/game" AND responsetime >= 5 ``` 1. In **Insights**, enable or disable insights access for the group. For more information about insights, see [Using X-Ray insights](xray-console-insights.md). ![\[Insights check boxes on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-insights-cw.png) 1. In **Tags**, choose **Add new tag** to enter a tag key, and optionally, a tag value. Continue to add additional tags as desired. Tag keys must be unique. To delete a tag, choose **Remove** underneath each tag. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md). ![\[Tag fields on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-tags-cw.png) 1. When you're finished updating the group, choose **Update group**. ------ #### [ X-Ray console ] 1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home). 1. Do one of the following to open the **Edit group** page. 1. On the **Groups** page, choose the name of a group to edit it. 1. On the group menu on one of the following pages, point to a group, and then choose **Edit**. + **Trace Map** + **Traces** + **Analytics** 1. Although you can't rename a group, you can update the filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`, where the request URL address contains `example/game`, and response time for requests was greater than or equal to five seconds. ``` fault = true AND http.url CONTAINS "example/game" AND responsetime >= 5 ``` 1. In **Insights**, enable or disable insights and insights notifications for the group. For more information about insights, see [Using X-Ray insights](xray-console-insights.md). ![\[Insights check boxes on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-insights.png) 1. In **Tags**, edit tag keys and values. Tag keys must be unique. Tag values are optional; you can delete values, if you want. To delete a tag, choose **X** at the end of the tag's row. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md). ![\[Tag fields on Group page\]](http://docs.aws.amazon.com/xray/latest/devguide/images/group-tags.png) 1. When you're finished updating the group, choose **Update group**. ------ ## Clone a group Cloning a group creates a new group that has the filter expression and tags of an existing group. When you clone a group, the new group has the same name as the group from which it's cloned, with `-clone` appended to the name. ------ #### [ CloudWatch console ] 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Settings** in the left navigation pane. 1. Choose **View settings** under **Groups** within the **X-Ray traces** section. 1. Choose a group from the **Groups** section and then choose **Clone**. 1. On the **Create group** page, the name of the group is *group-name*`-clone`. Optionally, enter a new name for the group. A group name can have a maximum of 32 characters, and contain alphanumeric characters and dashes. Group names are case sensitive. 1. You can keep the filter expression from the existing group, or optionally, enter a new filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`. and requests to the service where the response time was greater than or equal to five seconds. ``` service("api.example.com") { fault = true OR responsetime >= 5 } ``` 1. In **Tags**, edit tag keys and values, if needed. Tag keys must be unique. Tag values are optional; you can delete values if you want. To delete a tag, choose **X** at the end of the tag's row. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md). 1. Choose **Create group**. ------ #### [ X-Ray console ] 1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home). 1. Open the **Groups** page from the left navigation pane, and the choose the name of a group that you want to clone. 1. Choose **Clone group** from the **Actions** menu. 1. On the **Create group** page, the name of the group is *group-name*`-clone`. Optionally, enter a new name for the group. A group name can have a maximum of 32 characters, and contain alphanumeric characters and dashes. Group names are case sensitive. 1. You can keep the filter expression from the existing group, or optionally, enter a new filter expression. For more information about how to build a filter expression, see [Using filter expressions](xray-console-filters.md). In the following example, the group filters for fault traces from the service `api.example.com`. and requests to the service where the response time was greater than or equal to five seconds. ``` service("api.example.com") { fault = true OR responsetime >= 5 } ``` 1. In **Tags**, edit tag keys and values, if needed. Tag keys must be unique. Tag values are optional; you can delete values if you want. To delete a tag, choose **X** at the end of the tag's row. For more information about tags, see [Tagging X-Ray sampling rules and groups](xray-tagging.md). 1. Choose **Create group**. ------ ## Delete a group Follow steps in this section to delete a group. You can't delete the **Default** group. ------ #### [ CloudWatch console ] 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Settings** in the left navigation pane. 1. Choose **View settings** under **Groups** within the **X-Ray traces** section. 1. Choose a group from the **Groups** section and then choose **Delete**. 1. When you're prompted to confirm, choose **Delete**. ------ #### [ X-Ray console ] 1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home). 1. Open the **Groups** page from the left navigation pane, and the choose the name of a group that you want to delete. 1. On the **Actions** menu, choose **Delete group**. 1. When you're prompted to confirm, choose **Delete**. ------ ## View group metrics in Amazon CloudWatch After a group is created, incoming traces are checked against the group’s filter expression as they're stored in the X-Ray service. Metrics for the number of traces matching each criteria are published to Amazon CloudWatch every minute. Choosing **View metric** on the **Edit group** page opens the CloudWatch console to the **Metric** page. For more information about how to use CloudWatch metrics, see [Using Amazon CloudWatch Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html) in the *Amazon CloudWatch User Guide*. ------ #### [ CloudWatch console ] 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Settings** in the left navigation pane. 1. Choose **View settings** under **Groups** within the **X-Ray traces** section. 1. Choose a group from the **Groups** section and then choose **Edit**. 1. On the **Edit group** page, choose **View metric**. The CloudWatch console **Metrics** page opens in a new tab. ------ #### [ X-Ray console ] 1. Sign in to the AWS Management Console and open the X-Ray console at [https://console.aws.amazon.com/xray/home](https://console.aws.amazon.com/xray/home). 1. Open the **Groups** page from the left navigation pane, and the choose the name of a group that you want to view metrics for. 1. On the **Edit group** page, choose **View metric**. The CloudWatch console **Metrics** page opens in a new tab. ------ # Configuring sampling rules You can use the AWS X-Ray console to configure sampling rules for your services. The AWS Distro for OpenTelemetry, X-Ray SDK, and AWS services that support [active tracing](xray-services.md) with sampling configurations use sampling rules to determine which requests to record. **Topics** + [ ## Configuring sampling rules ](#xray-console-config) + [ ## Customizing sampling rules ](#xray-console-custom) + [ ## Sampling rule options ](#xray-console-sampling-options) + [ ## Sampling rule examples ](#xray-console-sampling-examples) + [ ## Configuring your service to use sampling rules ](#xray-console-sampling-service) + [ ## Viewing sampling results ](#xray-console-sampling-results) + [ ## Next steps ](#xray-console-sampling-nextsteps) ## Configuring sampling rules You can configure sampling for the following use cases: + **API Gateway Entrypoint** – API Gateway supports sampling and active tracing. To enable active tracing on an API stage, see [Amazon API Gateway active tracing support for AWS X-Ray](xray-services-apigateway.md). + **AWS AppSync** – AWS AppSync supports sampling and active tracing. To enable active tracing on AWS AppSync requests, see [Tracing with AWS X-Ray](https://docs.aws.amazon.com/appsync/latest/devguide/x-ray-tracing.html). + **AWS Step Functions** – AWS Step Functions supports sampling and active tracing. To enable active tracing on AWS Step Functions state machines, see [X-Ray tracing in Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-xray-tracing.html). + **Instrument AWS Distro for OpenTelemetry on compute platforms** – When using compute platforms such as Amazon EC2, Amazon ECS, or AWS Elastic Beanstalk, sampling is supported when the application has been instrumented with the latest AWS Distro for OpenTelemetry or X-Ray SDK. ## Customizing sampling rules By customizing sampling rules, you can control the amount of data that you record. You can also modify sampling behavior without modifying or redeploying your code. Sampling rules tell the AWS Distro for OpenTelemetry (ADOT) or X-Ray SDK how many requests to record for a set of criteria. By default, the SDK records the first request each second, and five percent of any additional requests. One request per second is the *reservoir*. This ensures that at least one trace is recorded each second as long as the service is serving requests. Five percent is the *rate* at which additional requests beyond the reservoir size are sampled. You can configure the X-Ray SDK to read sampling rules from a JSON document that you include with your code. However, when you run multiple instances of your service, each instance performs sampling independently. This causes the overall percentage of requests sampled to increase because the reservoirs of all of the instances are effectively added together. Additionally, to update local sampling rules, you must redeploy your code. By defining sampling rules in the X-Ray console, and [configuring the SDK](#xray-console-sampling-service) to read rules from the X-Ray service, you can avoid both of these issues. The service manages the reservoir for each rule, and assigns quotas to each instance of your service to distribute the reservoir evenly, based on the number of instances that are running. The reservoir limit is calculated according to the rules you set. Because the rules are configured in the service, you can manage rules without making additional deployments. **Note** When configuring sampling rules, it is critical to understand that X-Ray sampling is "Parent-based." This means the sampling decision is made only once, typically by the first X-Ray-enabled service that handles the request (the "Root" service). If a downstream service receives a request that already has a sampling decision from an upstream parent, it will honor that decision regardless of any of its own matching sampling rules. When Rules Apply: Your custom sampling rules only take effect on services where a sampling decision has not yet been made. This usually applies to: The entry point of your application (e.g., an API Gateway, a Load Balancer, or the first instrumented Microservice). Asynchronous processes or workers that start a brand new trace. Common Pitfall: If you create a strict sampling rule for "Service B," but "Service B" is always called by "Service A," your rule for Service B will likely never be applied because it is simply following the decision passed down by Service A. To change the sampling of traces for this workflow, you must configure the sampling rule to the Root service (Service A). **Note** X-Ray uses a best-effort approach in applying sampling rules, and in some cases the effective sampling rate may not exactly match the configured sampling rules. However, over time the number of requests sampled should be close to the configured percentage. You can now configure X-Ray sampling rules from within the Amazon CloudWatch console. **To configure sampling rules in the CloudWatch console** 1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Settings** in the left navigation pane underneath **Setup**. 1. Choose **View settings** under **Sampling rules** within the **X-Ray traces** section. 1. To create a rule, choose **Create sampling rule**. To edit a rule, choose a rule and choose **Edit** to edit it. To delete a rule, choose a rule and choose **Delete** to delete it. ## Sampling rule options The following options are available for each rule. String values can use wildcards to match a single character (`?`) or zero or more characters (`*`). **Sampling rule options** + **Rule name** (string) – A unique name for the rule. + **Priority** (integer between 1 and 9999) – The priority of the sampling rule. Services evaluate rules in ascending order of priority, and make a sampling decision with the first rule that matches. + **Reservoir** (non-negative integer) – A fixed number of matching requests to instrument per second, before applying the fixed rate. The reservoir is not used directly by services, but applies to all services using the rule collectively. + **Rate** (integer between 0 and 100) – The percentage of matching requests to instrument, after the reservoir is exhausted. When configuring a sampling rule in the console, choose a percentage between 0 and 100. When configuring a sampling rule in a client SDK using a JSON document, provide a percentage value between 0 and 1. + **Service name** (string) – The name of the instrumented service, as it appears in the trace map. + X-Ray SDK – The service name that you configure on the recorder. + Amazon API Gateway – `api-name/stage`. + **Service type** (string) – The service type, as it appears in the trace map. For the X-Ray SDK, set the service type by applying the appropriate plugin: + `AWS::ElasticBeanstalk::Environment` – An AWS Elastic Beanstalk environment (plugin). + `AWS::EC2::Instance` – An Amazon EC2 instance (plugin). + `AWS::ECS::Container` – An Amazon ECS container (plugin). + `AWS::EKS::Container` – An Amazon EKS container (plugin). + `AWS::APIGateway::Stage` – An Amazon API Gateway stage. + `AWS::AppSync::GraphQLAPI ` – An AWS AppSync API request. + `AWS::StepFunctions::StateMachine` – An AWS Step Functions state machine. + **Host** (string) – The hostname from the HTTP host header. + **HTTP method** (string) – The method of the HTTP request. + **URL path** (string) – The URL path of the request. + X-Ray SDK – The path portion of the HTTP request URL. + **Resource ARN** (string) – The ARN of the AWS resource running the service. + X-Ray SDK – Not supported. The SDK can only use rules with **Resource ARN** set to `*`. + Amazon API Gateway – The stage ARN. + (Optional) **Attributes** (key and value) – Segment attributes that are known when the sampling decision is made. + X-Ray SDK – Not supported. The SDK ignores rules that specify attributes. + Amazon API Gateway – Headers from the original HTTP request. + (Optional) **SamplingRateBoost** (object) – Controls anomaly-driven sampling boost behavior. + MaxRate (integer between 0 and 100) – Maximum sampling rate (0.0–1.0) X-Ray may increase to during anomalies + CooldownWindowMinutes (integer greater than 0) – Time window (minutes) in which only one boost can be triggered, preventing continuous boosts ## Sampling rule examples **Example – Default rule with no reservoir and a low rate** You can modify the default rule's reservoir and rate. The default rule applies to requests that don't match any other rule. + **Reservoir**: **0** + **Rate**: **5** (**0.05** if configured using a JSON document) **Example – Debugging rule to trace all requests for a problematic route** A high-priority rule applied temporarily for debugging. + **Rule name**: **DEBUG – history updates** + **Priority**: **1** + **Reservoir**: **1** + **Rate**: **100** (**1** if configured using a JSON document) + **Service name**: **Scorekeep** + **Service type**: **\$1** + **Host**: **\$1** + **HTTP method**: **PUT** + **URL path**: **/history/\$1** + **Resource ARN**: **\$1** **Example – Higher minimum rate for POSTs** + **Rule name**: **POST minimum** + **Priority**: **100** + **Reservoir**: **10** + **Rate**: **10** (**.1** if configured using a JSON document) + **Service name**: **\$1** + **Service type**: **\$1** + **Host**: **\$1** + **HTTP method**: **POST** + **URL path**: **\$1** + **Resource ARN**: **\$1** **Example Enable anomaly-driven boost** Configure a rule that triggers a sampling boost up to 50% during anomalies, with a 10-minute cooldown. + **Rule name**: **MyAdaptiveRule** + **Priority**: **100** + **Reservoir**: **1** + **FixedRate**: **0.0510** + **Service name**: **\$1** + **Service type**: **\$1** + **Host**: **\$1** + **HTTP method**: **POST** + **URL path**: **\$1** + **maxRate**: **0.5** + **cooldownWindowMinutes**: **10** ## Configuring your service to use sampling rules The AWS Distro for OpenTelemetry (ADOT) and X-Ray SDK requires additional configuration to use sampling rules that you configure in the console. See the configuration topic for your language for details on configuring a sampling strategy: + Java: [Using X-Ray Remote Sampling](https://aws-otel.github.io/docs/getting-started/java-sdk/auto-instr#using-x-ray-remote-sampling) with ADOT Java + Go: [Configuring sampling](https://aws-otel.github.io/docs/getting-started/go-sdk/manual-instr#configuring-sampling) with ADOT Go + Node.js: [Using X-Ray Remote Sampling](https://aws-otel.github.io/docs/getting-started/js-sdk/trace-metric-auto-instr#using-x-ray-remote-sampling) with ADOT JavaScript + Python: [Using X-Ray Remote Sampling](https://aws-otel.github.io/docs/getting-started/python-sdk/auto-instr#using-x-ray-remote-sampling) with ADOT Python + Ruby: [Sampling rules](xray-sdk-ruby-configuration.md#xray-sdk-ruby-configuration-sampling) + .NET: [Using X-Ray Remote Sampling](https://aws-otel.github.io/docs/getting-started/dotnet-sdk/auto-instr#using-x-ray-remote-sampling) with ADOT .NET For API Gateway, see [Amazon API Gateway active tracing support for AWS X-Ray](xray-services-apigateway.md). ## Viewing sampling results The X-Ray console **Sampling** page shows detailed information about how your services use each sampling rule. The **Trend** column shows how the rule has been used in the last few minutes. Each column shows statistics for a 10-second window. **Sampling statistics** + **Total matched rule**: The number of requests that matched this rule. This number doesn't include requests that could have matched this rule, but matched a higher-priority rule first. + **Total sampled**: The number of requests recorded. + **Sampled with fixed rate**: The number of requests sampled by applying the rule's fixed rate. + **Sampled with reservoir limit**: The number of requests sampled using a quota assigned by X-Ray. + **Borrowed from reservoir**: The number of requests sampled by borrowing from the reservoir. The first time a service matches a request to a rule, it has not yet been assigned a quota by X-Ray. However, if the reservoir is at least 1, the service borrows one trace per second until X-Ray assigns a quota. For more information about sampling statistics and how services use sampling rules, see [Using sampling rules with the X-Ray API](xray-api-sampling.md). ## Next steps You can use the X-Ray API to manage sampling rules. With the API, you can create and update rules programmatically on a schedule, or in response to alarms or notifications. See [Configuring sampling, groups, and encryption settings with the AWS X-Ray API](xray-api-configuration.md) for instructions and additional rule examples. The AWS Distro for OpenTelemetry, X-Ray SDK and AWS services also use the X-Ray API to read sampling rules, report sampling results, and get sampling targets. Services must keep track of how often they apply each rule, evaluate rules based on priority, and borrow from the reservoir when a request matches a rule for which X-Ray has not yet assigned the service a quota. For more detail about how a service uses the API for sampling, see [Using sampling rules with the X-Ray API](xray-api-sampling.md). When the AWS Distro for OpenTelemetry or the X-Ray SDK call sampling APIs, they use the CloudWatch agent as a proxy. If you already use TCP port 2000, you can configure the agent to run the proxy on a different port. See the [CloudWatch agent installation guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) for details. # Configuring adaptive sampling Missing critical traces during anomaly spikes can make root cause analysis difficult. However, maintaining high sampling rates is expensive. X-Ray adaptive sampling provides complete visibility into anomalies and controls cost during normal operations. With adaptive sampling, you set a maximum sampling rate, and X-Ray automatically adjusts within that limit. X-Ray calculates the minimum boost needed to capture error traces. If your baseline rate captures enough data, no boost occurs. You only pay for extra sampling when needed. Benefits of using adaptive sampling: + Complete incident visibility – Get full traces during incidents without manual intervention. X-Ray automatically adjusts sampling rates to capture error traces, then returns to normal rates. + Root cause visibility – Always see the source of problems. X-Ray captures critical error data even when full trace sampling isn't triggered. + Optimize costs – Brief sampling boosts (up to 1 minute) and automatic cooldown periods prevent oversampling. You pay only for the data you need to diagnose issues. **Topics** + [ ## Supported SDKs and platforms ](#adaptive-sampling-supported-sdks) + [ ## Choose your adaptive sampling approach ](#adaptive-sampling-features) + [ ## Local SDK configuration ](#local-sdk-configuration) ## Supported SDKs and platforms **Supported SDK** – Adaptive sampling requires the latest version of the ADOT SDK. **Supported language** – Java (version [v2.11.5](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/tag/v2.11.5) or higher) Your application must be instrumented with the supported ADOT SDK and executed together with either the Amazon CloudWatch Agent or the OpenTelemetry Collector. For example, Amazon EC2, Amazon ECS, and Amazon EKS are common platforms where AWS Application Signals provides guidance for enabling the ADOT SDK and Amazon CloudWatch Agent. ## Choose your adaptive sampling approach Adaptive sampling supports two approaches, Sampling Boost and Anomaly Span Capture. These can be applied independently or can be combined together. ### Sampling boost Adaptive sampling boost is based on sampling rules and works with the existing X-Ray head-based sampling model. Head-based sampling means that sampling decisions are made at the root service, and the sampling flag is passed downstream to all services in the call chain. + **Rule-based boosting** – Boosting is always tied to a specific X-Ray sampling rule. Each rule can define its own maximum boost rate and cool down behavior. + **Head-based sampling** – Sampling decisions are made at the root service, and the sampling flag is passed downstream to all services in the call chain. + **Anomaly-driven** – X-Ray relies on the SDK to report anomaly statistics. When X-Ray detects anomalies such as errors or high latency, it uses these statistics to calculate an appropriate boost rate (up to the configured maximum). **Anomaly reporting** Every application service in the call chain can emit anomaly statistics through the required SDK: + **Root service** – Must run on a supported SDK and platform to enable sampling boost. If the root service is not supported, no boost will occur. + **Downstream services** – Downstream services only report anomalies; they cannot make sampling decisions. When a downstream service is running a supported SDK, anomalies detected can trigger a sampling boost. When a downstream service is unsupported (for example, running an older SDK) , anomalies on that service will not trigger a boost. These services can still propagate the context downstream when they follow standard context propagation (such as W3C trace context and baggage). This ensures that supported SDKs in further downstream services can report anomalies that trigger a boost. **Boost timing and scope** + **Trigger delay** – You can expect a sampling boost to begin as low as 10 seconds after X-Ray detects an anomaly. + **Boost period** – After X-Ray triggers a boost, it lasts up to 1 minute before returning back to the base sampling rate. + **Boost cool down** – After a boost occurs, X-Ray will not trigger another boost for the same rule until the cool down window has passed. For example, when you set `cooldown` to 10 minutes, once a boost ends, no new boost can be triggered until the next 10 minutes window. Special case: when you set `cooldown` to 1 minute, and since a boost itself can last up to 1 minute, boosts can effectively be triggered continuously if anomaly persist. **Note** Use supported SDKs and platforms for your root service. Sampling boost works only with supported SDKs and platforms. While sampling boost has a high probability of capturing anomaly traces, it may not capture every anomaly trace. **Boost visibility** When a sampling rule is configured with adaptive sampling boost, X-Ray automatically emits vended metrics that allow you to monitor boost activity. + **Metric name** – `SamplingRate` + **Dimension** – `RuleName` (set to the actual rule name) Each rule with `SamplingRateBoost` enabled will publish its effective sampling rate, including both the baseline rate and any temporary boosts. This allows you to: + Track when boosts are triggered + Monitor the effective sampling rate for each rule + Correlate boosts with application anomalies (such as error spikes or latency events) You can view these metrics in **Amazon CloudWatch Metrics, under AWS/X-Ray namespace. The metric value is a floating-point number between 0 and 1, representing the effective sampling rate**. **Configure sampling boost using X-Ray sampling rules** You can enable adaptive sampling directly in your existing X-Ray sampling rules by adding a new `SamplingRateBoost` field. For more information, see [Customizing sampling rules](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html#xray-console-custom). This provides a centralized way to enable adaptive sampling without modifying application code or applying application deployment. When you enable adaptive sampling, X-Ray automatically increases sampling during anomalies such as error spikes or latency outliers, while keeping sampling rates within your configured maximum. `SamplingRateBoost` can be applied to any custom sampling rule except the `Default` sampling rule. The `SamplingRateBoost` field defines the upper bound and behavior for anomaly-driven sampling. ``` "SamplingRateBoost": { "MaxRate": 0.25, "CooldownWindowMinutes": 10 } ``` The `MaxRate` defines the maximum sampling rate X-Ray will apply when it detects anomalies. Value range is `0.0` to `1.0`. For example, `"MaxRate": 0.25` allows sampling to increase up to 25% of requests during an anomaly window. X-Ray determines the appropriate rate between your baseline and the maximum, depending on anomaly activity. The `CooldownWindowMinutes` defines time window (in minutes) in which only one sampling rate boost can be triggered. After a boost occurs, no further boosts are allowed until the next window. The Value type is *integer (minutes)*. **Example rule with adaptive sampling** ``` { "RuleName": "MyAdaptiveRule", "Priority": 1, "ReservoirSize": 1, "FixedRate": 0.05, "ServiceName": "*", "ServiceType": "*", "Host": "*", "HTTPMethod": "*", "URLPath": "*", "SamplingRateBoost": { "MaxRate": 0.25, "CooldownWindowMinutes": 10 } } ``` In this example, baseline sampling is 5% (`FixedRate: 0.05`). During anomalies, X-Ray can increase sampling up to 25% (`MaxRate: 0.25`). Boost only once every 10 minutes. **Anomaly condition configuration** When no anomaly condition configuration is provided, the ADOT SDK uses **HTTP 5xx error codes** as the default anomaly condition to trigger sampling boost. You can also fine tune anomaly conditions locally in the supported ADOT SDK using environment variables. For more information, see [Local SDK configuration](#local-sdk-configuration). ### Anomaly spans capture Anomaly span capture ensures that critical spans representing anomalies are always recorded, even if the full trace is not sampled. This feature complements sampling boost by focusing on capturing the anomaly itself, rather than increasing sampling for future traces. When the ADOT SDK detects an anomaly, it emits that span immediately, regardless of the sampling decision. Since the SDK emits only spans related to the anomaly, these traces are partial traces, not full end-to-end transactions. Once the ADOT SDK detects an anomaly span, it attempts to emit as many spans from the same trace as possible. All spans emitted under this feature are tagged with the attribute, `aws.trace.flag.sampled = 0`. This allows you to easily distinguish partial traces (anomaly capture) from complete traces (normal sampling) in transaction search and analysis. We recommend onboarding [Transaction Search](https://docs.aws.amazon.com/xray/latest/devguide/xray-transactionsearch.html) to view and query partial traces. The following example shows a Service page in [Application Signals](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html) console. ServiceC is configured with anomaly span capture, and it is part of a call chain where sampling boost applies. This configurations generates both complete and partial traces. You can use the `aws.trace.flag.sampled` attribute to distinguish between trace types. ![\[Anomaly spans capture\]](http://docs.aws.amazon.com/xray/latest/devguide/images/adaptive-sampling.png) Anomaly spans capture can only be enabled or customized through the [Local SDK configuration](#local-sdk-configuration). ## Local SDK configuration You can configure adaptive sampling features in the ADOT SDK by providing a YAML configuration through an environment variable. Local configuration provides fine-grained control over anomaly conditions, thresholds. This is required for *anomaly span capture* and optional for customizing *sampling boost* conditions. The following is an example of the configuration: ``` version: 1.0 anomalyConditions: - errorCodeRegex: "^5\\d\\d$" usage: both - operations: - "/api" errorCodeRegex: "^429|5\\d\\d$" highLatencyMs: 300 usage: sampling-boost - highLatencyMs: 1000 usage: anomaly-span-capture anomalyCaptureLimit: anomalyTracesPerSecond: 1 ``` Field definitions are below: + `version` – Schema version for the configuration file + `anomalyConditions` – Defines the conditions under which anomalies are detected and how they are used + `errorCodeRegex` – Regular expression defining which HTTP status codes are considered anomalies + `operations` – List of operations or endpoints to which the condition applies + `highLatencyMs` – Latency threshold (in milliseconds) above which spans are treated as anomalies + `usage` – Defines which feature the condition applies to: + `both` – Applies to **sampling boost** and **anomaly span capture** (Default if usage is not specified) + `sampling-boost` – Used only for triggering sampling boosts + `anomaly-span-capture` – Used only for anomaly span capture + `anomalyCaptureLimit` – Defines limits on how many traces with anomaly spans are emitted. `anomalyTracesPerSecond` – Maximum number of traces with anomaly spans captured per second, to prevent excessive span volume (Default value is 1 if anomalyCaptureLimit is not present). **Note** `AnomalyConditions` **overrides** the default anomaly condition for sampling boost (HTTP 5xx). If you want to retain the default condition while using local configuration, you must explicitly include it in any item of `AnomalyConditions`. For each `anomalyConditions` item: When the `operations` field is **omitted**, the condition applies to **all operations** (service level) When the `operations` field is present but set to an **empty list**, the condition applies to **no operations**, making that item a no-op When both `errorCodeRegex` and `highLatencyMs` are omitted, the condition has no anomaly criteria to evaluate, making that item a no-op Logical relationships: Between items in `anomalyConditions`, the relationship is **OR**. Within a single item, multiple fields (for example, `errorCodeRegex` and `highLatencyMs`) are combined with **AND**. For example: ``` errorCodeRegex: "^429|5\\d\\d$" highLatencyMs: 300 ``` This condition means, **status code matches 429 or 5xx AND latency ≥ 300 ms**. ### Apply the Local Configuration to ADOT SDK You can apply the local configuration to the ADOT SDK by setting the environment variable `AWS_XRAY_ADAPTIVE_SAMPLING_CONFIG`. The value must be a valid YAML document (inline or nested). For example, Amazon EC2 and Amazon ECS, set the environment variable directly: ``` AWS_XRAY_ADAPTIVE_SAMPLING_CONFIG="{version: 1.0, anomalyConditions: [{errorCodeRegex: \"^500$\", usage: \"sampling-boost\"}, {errorCodeRegex: \"^501$\", usage: \"anomaly-trace-capture\"}], anomalyCaptureLimit: {anomalyTracesPerSecond: 10}}" ``` For Amazon EKS, define the environment variable inside the pod spec as nested YAML: ``` apiVersion: v1 kind: Pod metadata: name: adot-sample spec: containers: - name: adot-app image: my-app:latest env: - name: AWS_XRAY_ADAPTIVE_SAMPLING_CONFIG value: | version: 1.0 anomalyConditions: - errorCodeRegex: "^500$" usage: sampling-boost - errorCodeRegex: "^501$" usage: anomaly-trace-capture anomalyCaptureLimit: anomalyTracesPerSecond: 10 ``` # Console deep linking You can use routes and queries to deep link into specific traces, or filtered views of traces and the trace map. **Console pages** + Welcome page – [xray/home\$1/welcome](https://console.aws.amazon.com/xray/home#/welcome) + Getting started – [xray/home\$1/getting-started](https://console.aws.amazon.com/xray/home#/getting-started) + Trace map – [xray/home\$1/service-map](https://console.aws.amazon.com/xray/home#/service-map) + Traces – [xray/home\$1/traces](https://console.aws.amazon.com/xray/home#/traces) ## Traces You can generate links for timeline, raw, and map views of individual traces. **Trace timeline** – `xray/home#/traces/trace-id` **Raw trace data** – `xray/home#/traces/trace-id/raw` **Example – raw trace data** ``` https://console.aws.amazon.com/xray/home#/traces/1-57f5498f-d91047849216d0f2ea3b6442/raw ``` ## Filter expressions Link to a filtered list of traces. **Filtered traces view** – `xray/home#/traces?filter=filter-expression` **Example – filter expression** ``` https://console.aws.amazon.com/xray/home#/traces?filter=service("api.amazon.com") { fault = true OR responsetime > 2.5 } AND annotation.foo = "bar" ``` **Example – filter expression (URL encoded)** ``` https://console.aws.amazon.com/xray/home#/traces?filter=service(%22api.amazon.com%22)%20%7B%20fault%20%3D%20true%20OR%20responsetime%20%3E%202.5%20%7D%20AND%20annotation.foo%20%3D%20%22bar%22 ``` For more information about filter expressions, see [Using filter expressions](xray-console-filters.md). ## Time range Specify a length of time or start and end time in ISO8601 format. Time ranges are in UTC and can be up to 6 hours long. **Length of time** – `xray/home#/page?timeRange=range-in-minutes` **Example – trace map for the last hour** ``` https://console.aws.amazon.com/xray/home#/service-map?timeRange=PT1H ``` **Start and end time** – `xray/home#/page?timeRange=start~end` **Example – time range accurate to seconds** ``` https://console.aws.amazon.com/xray/home#/traces?timeRange=2023-7-01T16:00:00~2023-7-01T22:00:00 ``` **Example – time range accurate to minutes** ``` https://console.aws.amazon.com/xray/home#/traces?timeRange=2023-7-01T16:00~2023-7-01T22:00 ``` ## Region Specify an AWS Region to link to pages in that Region. If you don't specify a Region, the console redirects you to the last visited Region. **Region** – `xray/home?region=region#/page` **Example – trace map in US West (Oregon) (us-west-2)** ``` https://console.aws.amazon.com/xray/home?region=us-west-2#/service-map ``` When you include a Region with other query parameters, the Region query goes before the hash, and the X-Ray-specific queries go after the page name. **Example – trace map for the last hour in US West (Oregon) (us-west-2)** ``` https://console.aws.amazon.com/xray/home?region=us-west-2#/service-map?timeRange=PT1H ``` ## Combined **Example – recent traces with a duration filter** ``` https://console.aws.amazon.com/xray/home#/traces?timeRange=PT15M&filter=duration%20%3E%3D%205%20AND%20duration%20%3C%3D%208 ``` **Output** + Page – Traces + Time Range – Last 15 minutes + Filter – duration >= 5 AND duration <= 8