Use an AWS Management Console

View the X-Ray trace map to identify services where errors are occurring, connections with high latency, or traces for requests that were unsuccessful.

Viewing the trace map

The trace map is a visual representation of the trace data that's generated by your applications. The map shows service nodes that serve requests, upstream client nodes that represent the origins of the requests, and downstream service nodes that represent web services and resources that are used by an application while processing a request.

The trace map displays a connected view of traces across event-driven applications that use Amazon SQS and Lambda. For more information, see the following Trace event-driven applications section. The trace map also supports Cross-account tracing, displaying nodes from multiple accounts in a single map.

Filtering the trace map by group

Using a filter expression, you can define criteria by which to include traces within a group. For more information about filter expressions, see Use filter expressions. Next, use the following steps to then display that specific group in the trace map.

The service map will now be filtered to display traces that match the filter expression of the selected group.

Trace map legend and options

The trace map includes a legend and several options for customizing the map display.

Use the Traces page in the X-Ray console to find traces by URL, response code, or other data from the trace summary. After selecting a trace from the trace list, the Trace details page displays a map of service nodes that are associated with the selected trace and a timeline of trace segments.

Viewing traces

Exploring the trace timeline

The Timeline section shows a hierarchy of segments and subsegments next to a horizontal bar that corresponds to time they used to complete their tasks. The first entry in the list is the segment, which represents all data recorded by the service for a single request. Subsegments are indented and listed following the segment. Columns contain information about each segment.

When you use an instrumented AWS SDK, HTTP, or SQL client to make calls to external resources, the X-Ray SDK records subsegments automatically. You can also use the X-Ray SDK to record custom subsegments for any function or block of code. Additional subsegments that are recorded while a custom subsegment are open become children of the custom subsegment.

Viewing segment details

From the trace Timeline, choose the name of a segment to view its details.

The Segment details panel shows the Overview, Resources, Annotations, Metadata, Exceptions, and SQL tabs. The following apply:

Viewing subsegment details

From the trace timeline, choose the name of a subsegment to view its details:

For custom subsegments, the Overview tab shows the name of the subsegment, which you can set to specify the area of the code or function that it records. For more information, see the language-specific link listed under Instrumenting your application with AWS X-Ray SDKs in Generating custom subsegments with the X-Ray SDK for Java.

The following image shows the Overview tab for a custom subsegment. The overview contains the subsegment ID, parent ID, Name, start and end times, duration, status and errors or faults.

The Metadata tab for a custom subsegment contains information in JSON format about resources used by that subsegment.

Use filter expressions to view a trace map or traces for a specific request, service, connection between two services (an edge), or requests that satisfy a condition. X-Ray provides a filter expression language for filtering requests, services, and edges based on data in request headers, response status, and indexed fields on the original segments.

When you choose a time period of traces to view in the X-Ray console, you might get more results than the console can display. In the upper-right corner, the console shows the number of traces that it scanned and whether there are more traces available. You can use a filter expression to narrow the results to just the traces that you want to find.

When you choose a node in the trace map, the console constructs a filter expression based on the service name of the node, and the types of error present based on your selection. To find traces that show performance issues or that relate to specific requests, you can adjust the expression that the console provides or create your own. If you add annotations with the X-Ray SDK, you can also filter based on the presence of an annotation key or the value of a key.

Note

If you choose a relative time range in the trace map and choose a node, the console converts the time range to an absolute start and end time. To ensure that the traces for the node appear in the search results, and avoid scanning times when the node wasn't active, the time range only includes times when the node sent traces. To search relative to the current time, you can switch back to a relative time range in the traces page and scan again.

If there are still more results available than the console can show, the console shows you how many traces matched and the number of traces scanned. The percentage shown is the percentage of the selected time frame that was scanned. To ensure that you see all matching traces represented in the results, narrow your filter expression further, or choose a shorter time frame.

To get the freshest results first, the console starts scanning at the end of the time range and works backward. If there are a large number of traces, but few results, the console splits the time range into chunks and scans them in parallel. The progress bar shows the parts of the time range that have been scanned.

Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics.

Groups are identified by their name or an Amazon Resource Name (ARN), and contain a filter expression. The service compares incoming traces to the expression and stores them accordingly.

You can create and modify groups by using the dropdown menu to the left of the filter expression search bar.

Note

If the service encounters an error in qualifying a group, that group is no longer included in processing incoming traces and an error metric is recorded.

For more information about groups, see Configure groups.

Filter expressions can contain a keyword, a unary or binary operator, and a value for comparison.

keyword operator value

Different operators are available for different types of keyword. For example, responsetime is a number keyword and can be compared with operators related to numbers.

Example – requests where response time was greater than 5 seconds

responsetime > 5

You can combine multiple expressions in a compound expression by using the AND or OR operators.

Example – requests where the total duration was 5–8 seconds

duration >= 5 AND duration <= 8

Simple keywords and operators find issues only at the trace level. If an error occurs downstream, but is handled by your application and not returned to the user, a search for error will not find it.

To find traces with downstream issues, you can use the complex keywords service() and edge(). These keywords let you apply a filter expression to all downstream nodes, a single downstream node, or an edge between two nodes. For more about these keywords, see the following Complex keywords section. For more granularity, you can filter services and edges by type with the id() function. For more information see the following id function section.

Boolean keyword values are either true or false. Use these keywords to find traces that resulted in errors.

Boolean keywords

• ok – Response status code was 2XX Success.

• error – Response status code was 4XX Client Error.

• throttle – Response status code was 429 Too Many Requests.

• fault – Response status code was 5XX Server Error.

• partial – Request has incomplete segments.

• inferred – Request has inferred segments.

• first – Element is the first of an enumerated list.

• last – Element is the last of an enumerated list.

• remote – Root cause entity is remote.

• root – Service is the entry point or root segment of a trace.

Boolean operators find segments where the specified key is true or false.

Boolean operators

• none – The expression is true if the keyword is true.

• ! – The expression is true if the keyword is false.

• =,!= – Compare the value of the keyword to the string true or false. These operators act the same as the other operators but are more explicit.

Example – response status is 2XX OK

ok

Example – response status is not 2XX OK

!ok

Example – response status is not 2XX OK

ok = false

Example – last enumerated fault trace has error name "deserialize"

rootcause.fault.entity { last and name = "deserialize" }

Example – requests with remote segments where coverage is greater than 0.7 and the service name is "traces"

rootcause.responsetime.entity { remote and coverage > 0.7 and name = "traces" }

Example – requests with inferred segments where the service type is "AWS:DynamoDB"

rootcause.fault.service { inferred and name = traces and type = "AWS::DynamoDB" }

Example – requests that have a segment with the name "data-plane" as the root

service("data-plane") {root = true and fault = true}

Use number keywords to search for requests with a specific response time, duration, or response status.

Number keywords

• responsetime – Time that the server took to send a response.

• duration – Total request duration, including all downstream calls.

• http.status – Response status code.

• index – Position of an element in an enumerated list.

• coverage – Decimal percentage of entity response time over root segment response time. Applicable only for response time root cause entities.

Number operators

Number keywords use standard equality and comparison operators.

• =,!= – The keyword is equal to or not equal to a number value.

• <,<=, >,>= – The keyword is less than or greater than a number value.

Example – response status is not 200 OK

http.status != 200

Example – request where the total duration was 5–8 seconds

duration >= 5 AND duration <= 8

Example – requests that completed successfully in less than 3 seconds, including all downstream calls

ok !partial duration <3

Example – enumerated list entity that has an index greater than 5

rootcause.fault.service { index > 5 }

Example – requests where the last entity that has coverage greater than 0.8

rootcause.responsetime.entity { last and coverage > 0.8 }

Use string keywords to find traces with specific text in the request headers, or specific user IDs.

String keywords

• http.url – Request URL.

• http.method – Request method.

• http.useragent – Request user agent string.

• http.clientip – Requestor's IP address.

• user – Value of the user field on any segment in the trace.

• name – The name of a service or exception.

• type – Service type.

• message – Exception message.

• availabilityzone – Value of the availabilityzone field on any segment in the trace.

• instance.id – Value of the instance ID field on any segment in the trace.

• resource.arn – Value of the resource ARN field on any segment in the trace.

String operators find values that are equal to or contain specific text. Values must always be specified in quotation marks.

String operators

• =,!= – The keyword is equal to or not equal to a number value.

• CONTAINS – The keyword contains a specific string.

• BEGINSWITH , ENDSWITH – The keyword begins or ends with a specific string.

Example – http.url filter

http.url CONTAINS "/api/game/"

To test if a field exists on a trace, regardless of its value, check to see if it contains the empty string.

Example – user filter

Find all traces with user IDs.

user CONTAINS ""

Example – select traces with a fault root cause that includes a service named "Auth"

rootcause.fault.service { name = "Auth" }

Example – select traces with a response time root cause whose last service has a type of DynamoDB

rootcause.responsetime.service { last and type = "AWS::DynamoDB" }

Example – select traces with a fault root cause whose last exception has the message "access denied for account_id: 1234567890"

rootcause.fault.exception { last and message = "Access Denied for account_id: 1234567890" 

Use complex keywords to find requests based on service name, edge name, or annotation value. For services and edges, you can specify an additional filter expression that applies to the service or edge. For annotations, you can filter on the value of an annotation with a specific key using Boolean, number, or string operators.

Complex keywords

• annotation.key – Value of an annotation with field key. The value of an annotation can be a Boolean, number, or string, so you can use any of the comparison operators of those types. You can use this keyword in combination with the service or edge keyword.

• edge(source, destination) {filter} – Connection between services source and destination. Optional curly braces can contain a filter expression that applies to segments on this connection.

• group.name / group.arn – The value of a group's filter expression, referenced by group name or group ARN.

• json – JSON root cause object. See Getting data from AWS X-Ray for steps to create JSON entities programmatically.

• service(name) {filter} – Service with name name. Optional curly braces can contain a filter expression that applies to segments created by the service.

Use the service keyword to find traces for requests that hit a certain node on your trace map.

Complex keyword operators find segments where the specified key has been set, or not set.

Complex keyword operators

• none – The expression is true if the keyword is set. If the keyword is of boolean type, it will evaluate to the boolean value.

• ! – The expression is true if the keyword is not set. If the keyword is of boolean type, it will evaluate to the boolean value.

• =,!= – Compare the value of the keyword.

• edge(source, destination) {filter} – Connection between services source and destination. Optional curly braces can contain a filter expression that applies to segments on this connection.

• annotation.key – Value of an annotation with field key. The value of an annotation can be a Boolean, number, or string, so you can use any of the comparison operators of those types. You can use this keyword in combination with the service or edge keyword.

• json – JSON root cause object. See Getting data from AWS X-Ray for steps to create JSON entities programmatically.

Use the service keyword to find traces for requests that hit a certain node on your trace map.

Example – Service filter

Requests that included a call to api.example.com with a fault (500 series error).

service("api.example.com") { fault }

You can exclude the service name to apply a filter expression to all nodes on your service map.

Example – service filter

Requests that caused a fault anywhere on your trace map.

service() { fault }

The edge keyword applies a filter expression to a connection between two nodes.

Example – edge filter

Request where the service api.example.com made a call to backend.example.com that failed with an error.

edge("api.example.com", "backend.example.com") { error }

You can also use the ! operator with service and edge keywords to exclude a service or edge from the results of another filter expression.

Example – service and request filter

Request where the URL begins with http://api.example.com/ and contains /v2/ but does not reach a service named api.example.com.

http.url BEGINSWITH "http://api.example.com/" AND http.url CONTAINS "/v2/" AND !service("api.example.com")

Example – service and response time filter

Find traces where http url is set and response time is greater than 2 seconds.

http.url AND responseTime > 2

For annotations, you can call all traces where annotation.key is set, or use the comparison operators that correspond to the type of value.

Example – annotation with string value

Requests with an annotation named gameid with string value "817DL6VO".

annotation.gameid = "817DL6VO"

Example – annotation is set

Requests with an annotation named age set.

annotation.age

Example – annotation is not set

Requests without an annotation named age set.

!annotation.age

Example – annotation with number value

Requests with annotation age with numerical value greater than 29.

annotation.age > 29

Example – annotation in combination with service or edge

service { annotation.request_id = "917DL6VO" }

edge { source.annotation.request_id = "916DL6VO" }

edge { destination.annotation.request_id = "918DL6VO" }

Example – group with user

Requests where traces meet the high_response_time group filter (e.g. responseTime > 3), and the user is named Alice.

group.name = "high_response_time" AND user = "alice"

Example – JSON with root cause entity

Requests with matching root cause entities.

rootcause.json = #[{ "Services": [ { "Name": "GetWeatherData", "EntityPath": [{ "Name": "GetWeatherData" }, { "Name": "get_temperature" } ] }, { "Name": "GetTemperature", "EntityPath": [ { "Name": "GetTemperature" } ] } ] }]

When you provide a service name to the service or edge keyword, you get results for all nodes that have that name. For more precise filtering, you can use the id function to specify a service type in addition to a name to distinguish between nodes with the same name.

Use the account.id function to specify a particular account for the service, when viewing traces from multiple accounts in a monitoring account.

id(name: "service-name", type:"service::type", account.id:"account-ID")

You can use the id function in place of a service name in service and edge filters.

service(id(name: "service-name", type:"service::type")) { filter }

edge(id(name: "service-one", type:"service::type"), id(name: "service-two", type:"service::type")) { filter }

For example, AWS Lambda functions result in two nodes in the trace map; one for the function invocation, and one for the Lambda service. The two nodes have the same name but different types. A standard service filter will find traces for both.

Example – service filter

Requests that include an error on any service named random-name.

service("function-name") { error }

Use the id function to narrow the search to errors on the function itself, excluding errors from the service.

Example – service filter with id function

Requests that include an error on a service named random-name with type AWS::Lambda::Function.

service(id(name: "random-name", type: "AWS::Lambda::Function")) { error }

To search for nodes by type, you can also exclude the name entirely.

Example – service filter with id function and service type

Requests that include an error on a service with type AWS::Lambda::Function.

service(id(type: "AWS::Lambda::Function")) { error }

To search for nodes for a particular AWS account, specify an account ID.

Example – service filter with id function and account ID

Requests that include a service within a specific account ID AWS::Lambda::Function.

service(id(account.id: "account-id"))

AWS X-Ray supports cross-account observability, enabling you to monitor and troubleshoot applications that span multiple accounts within an AWS Region. You can seamlessly search, visualize, and analyze your metrics, logs, and traces in any of the linked accounts as if you were operating in a single account. This provides a complete view of requests that travel across multiple accounts. You can view cross-account traces in the X-Ray trace map and traces pages within the CloudWatch console.

The shared observability data can include any of the following types of telemetry:

Configure cross-account observability

To turn on cross-account observability, set up one or more AWS monitoring accounts and link them with multiple source accounts. A monitoring account is a central AWS account that can view and interact with observability data that's generated from source accounts. A source account is an individual AWS account that generates observability data for the resources that it contains.

Source accounts share their observability data with monitoring accounts. Traces are copied from each source account to up to five monitoring accounts. Copies of traces from source accounts to the first monitoring account are free. Copies of traces sent to additional monitoring accounts are charged to each source account, based on standard pricing. For more information, see AWS X-Ray pricing and Amazon CloudWatch pricing.

To create links between monitoring accounts and source accounts, use the CloudWatch console or the new Observability Access Manager commands in the AWS CLI and API. For more information, see CloudWatch cross-account observability.

Viewing cross-account traces

Cross-account traces are displayed in the monitoring account. Each source account displays only local traces for that specific account. The following sections assume that you're signed in to the monitoring account and have opened the Amazon CloudWatch console. On both the trace map and traces pages, a monitoring account badge is displayed in the upper-right corner.

Trace map

In the CloudWatch console, choose Trace Map under X-Ray traces from the left navigation pane. By default, the trace map displays nodes for all source accounts that send traces to the monitoring account, and nodes for the monitoring account itself. On the trace map, choose Filters from the upper left to filter the trace map using the Accounts drop-down. After an account filter is applied, service nodes from accounts that don't match the current filter are grayed out.

When you choose a service node, the node details pane includes the service's account ID and label.

In the upper-right corner of the trace map, choose List view to see a list of service nodes. The list of service nodes includes services from the monitoring account and all configured source accounts. Filter the list of nodes by Account label or Account id by choosing them from the Nodes filter.

Traces

View trace details for traces that span multiple accounts by opening the CloudWatch console from the monitoring account, and choosing Traces under X-Ray traces in the left navigation pane. You can also open this page by choosing a node in the X-Ray Trace Map, and then choosing View traces from the node details pane.

The Traces page supports querying by account ID. To get started, enter a query that includes one or more account IDs. For more information about queries, see Use filter expressions. The following example queries for traces that have passed through account ID X or Y:

service(id(account.id:"X")) OR service(id(account.id:"Y"))

Refine your query by Account. Select one or more accounts from the list, and choose Add to query.

Trace details

View details for a trace by choosing it from the Traces list at the bottom of the Traces page. The Trace details displays, including a trace details map with service nodes from across all accounts that the trace passed through. Choose a specific service node to see its corresponding account.

The Segments timeline section displays the account details for each segment in the timeline.

AWS X-Ray supports tracing event-driven applications using Amazon SQS and AWS Lambda. Use the CloudWatch console to see a connected view of each request as it's queued with Amazon SQS and processed by one or more Lambda functions. Traces from upstream message producers are automatically linked to traces from downstream Lambda consumer nodes, creating an end-to-end view of the application.

View linked traces in the trace map

Use the Trace Map page within the CloudWatch console to view a trace map with traces from message producers that are linked to traces from Lambda consumers. These links are displayed with a dashed-line edge that connects the Amazon SQS node and downstream Lambda consumer nodes.

Select a dashed-line edge to display a received event age histogram, which maps the spread of event age when it's received by consumers. The age is calculated each time an event is received.

View linked trace details

The trace details map displays the current trace, along with upstream and downstream linked traces, each of which are contained within a box that indicates the bounds of each trace. If the currently selected trace is linked to multiple upstream or downstream traces, the nodes within the upstream or downstream linked traces are stacked, and a Select trace button is displayed.

Beneath the trace details map, a timeline of trace segments displays, including upstream and downstream linked traces. If there are multiple upstream or downstream linked traces, their segment details can't be displayed. To view segment details for a single trace within a set of linked traces, select a single trace as described in the following section.

Select a single trace within a set of linked traces

When you select a node or edge on an trace map, the X-Ray console shows a latency distribution histogram.

Latency

Latency is the amount of time between when a request starts and when it completes. A histogram shows a distribution of latencies. It shows duration on the x-axis, and the percentage of requests that match each duration on the y-axis.

This histogram shows a service that completes most requests in less than 300 ms. A small percentage of requests take up to 2 seconds, and a few outliers take more time.

Interpreting service details

Service histograms and edge histograms provide a visual representation of latency from the viewpoint of a service or requester.

To interpret the Service details panel histogram, you can look for values that differ the most from the majority of values in the histogram. These outliers can be seen as peaks or spikes in the histogram, and you can view the traces for a specific area to investigate what's going on.

To view traces filtered by latency, select a range on the histogram. Click where you want to start the selection and drag from left to right to highlight a range of latencies to include in the trace filter.

After selecting a range, you can choose Zoom to view just that portion of the histogram and refine your selection.

Once you have the focus set to the area you're interested in, choose View traces.

AWS X-Ray continuously analyzes trace data in your account to identify emergent issues in your applications. When fault rates exceed the expected range, it creates an insight that records the issue and tracks its impact until it's resolved. With Insights, you can:

The X-Ray console identifies nodes with ongoing incidents in the trace map. To see a summary of the insight, choose the affected node. You can also view and filter Insights by choosing Insights from the navigation pane on the left.

X-Ray creates an insight when it detects an anomaly in one or more nodes of the service map. The service uses statistical modeling to predict the expected fault rates of services in your application. In the preceding example, the anomaly is an increase in faults from AWS Elastic Beanstalk. The Elastic Beanstalk server experienced multiple API call timeouts, causing an anomaly in the downstream nodes.

Enable Insights in the X-Ray console

Insights must be enabled for each group you want to use insights features with. You can enable insights from the Groups page.

          {
             "Version": "2012-10-17",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Action": [
                         "xray:GetInsightSummaries",
                         "xray:GetInsight",
                         "xray:GetInsightEvents",
                         "xray:GetInsightImpactGraph"
                     ],
                     "Resource": [
                         "*"
                     ]
                 }
             ]
         }
      

Enable insights notifications

With insights notifications, a notification is created for each insight event, such as when an insight is created, changes significantly, or is closed. Customers can receive these notifications through Amazon EventBridge events, and use conditional rules to take actions such as SNS notification, Lambda invocation, posting messages to an SQS queue, or any of the targets EventBridge supports. Insights notifications are emitted on a best-effort basis but are not guaranteed. For more information about targets, see Amazon EventBridge Targets.

You can enable insights notifications for any insights enabled group from the Groups page.

Insight overview

The overview page for an insight attempts to answer three key questions:

The Anomalous services section shows a timeline for each service that illustrates the change in fault rates during the incident. The timeline shows the number of traces with faults overlaid on a solid band that indicates the expected number of faults based on the amount of traffic recorded. The duration of the insight is visualized by the Incident window. The incident window begins when X-Ray observes the metric becoming anomalous and persists while the insight is active.

The following example shows an increase in faults that caused an incident:

The Root cause section shows a trace map focused on the root cause service and the impacted path. You may hide the unaffected nodes by selecting the eye icon in the top right of the Root cause map. The root cause service is the farthest downstream node where X-Ray identified an anomaly. It can represent a service that you instrumented or an external service that your service called with an instrumented client. For example, if you call Amazon DynamoDB with an instrumented AWS SDK client, an increase in faults from DynamoDB results in an insight with DynamoDB as the root cause.

To further investigate the root cause, select View root cause details on the root cause graph. You can use the Analytics page to investigate the root cause and related messages. For more information, see Interact with the Analytics console.

Faults that continue upstream in the map can impact multiple nodes and cause multiple anomalies. If a fault is passed all the way back to the user that made the request, the result is a client fault. This is a fault in the root node of the trace map. The Impact graph provides a timeline of the client experience for the entire group. This experience is calculated based on percentages of the following states: Fault, Error, Throttle, and Okay.

This example shows an increase in traces with a fault at the root node during the time of an incident. Incidents in downstream services don't always correspond to an increase in client errors.

Choosing Analyze insight opens the X-Ray Analytics console in a window where you can dive deep into the set of traces causing the insight. For more information, see Interact with the Analytics console.

Understanding impact

AWS X-Ray measures the impact caused by an ongoing issue as part of generating insights and notifications. The impact is measured in two ways:

This impact is determined by the percentage of request that are failing or causing an error within a given time period. This impact analysis allows you to derive the severity and priority of the issue based on your particular scenario. This impact is available as part of the console experience in addition to insights notifications.

Deduplication

AWS X-Ray insights de-duplicates issues across multiple microservices. It uses anomaly detection to determine the service that is the root cause of an issue, determines if other related services are exhibiting anomalous behavior due to the same root cause, and records the result as a single insight.

Review an insight's progress

X-Ray re-evaluates insights periodically until they are resolved, and records each notable intermediate change as a notification, which can be sent as an Amazon EventBridge event. This enables you to build processes and workflows to determine how the issue has changed over time, and take appropriate actions such as sending an email or integrating with an alerting system using EventBridge.

You can review incident events in the Impact Timeline on the Inspect page. By default the timeline displays the most impacted service until you choose a different service.

To see a trace map and graphs for an event, choose it from the impact timeline. The trace map shows services in your application that are affected by the incident. Under Impact analysis, graphs show fault timelines for the selected node and for clients in the group.

To take a deeper look at the traces involved in an incident, choose Analyze event on the Inspect page. You can use the Analytics page to refine the list of traces and identify affected users. For more information, see Interact with the Analytics console.

The AWS X-Ray Analytics console is an interactive tool for interpreting trace data to quickly understand how your application and its underlying services are performing. The console enables you to explore, analyze, and visualize traces through interactive response time and time-series graphs.

When making selections in the Analytics console, the console constructs filters to reflect the selected subset of all traces. You can refine the active dataset with increasingly granular filters by clicking the graphs and the panels of metrics and fields that are associated with the current trace set.

The X-Ray Analytics console uses the following key features for grouping, filtering, comparing, and quantifying trace data.

Features

Feature	Description
Groups	The initial selected group is Default. To change the retrieved group, select a different group from the menu to the right of the main filter expression search bar. To learn more about groups see Configure groups.
Retrieved traces	By default, the Analytics console generates graphs based on all traces in the selected group. Retrieved traces represent all traces in your working set. You can find the trace count in this tile. Filter expressions you apply to the main search bar refine and update the retrieved traces.
Show in charts/Hide from charts	A toggle to compare the active group against the retrieved traces. To compare the data related to the group against any active filters, choose Show in charts. To remove this view from the charts, choose Hide from charts.
Filtered trace set A	Through interactions with the graphs and tables, apply filters to create the criteria for Filtered trace set A. As the filters are applied, the number of applicable traces and the percentage of traces from the total that are retrieved are calculated within this tile. Filters populate as tags within the Filtered trace set A tile and can also be removed from the tile.
Refine	This function updates the set of retrieved traces based on the filters applied to trace set A. Refining the retrieved trace set refreshes the working set of all traces retrieved based on the filters for trace set A. The working set of retrieved traces is a sampled subset of all traces in the group.
Filtered trace set B	When created, Filtered trace set B is a copy of Filtered trace set A. To compare the two trace sets, make new filter selections that will apply to trace set B, while trace set A remains fixed. As the filters are applied, the number of applicable traces and the percentage of traces from the total retrieved are calculated within this tile. Filters populate as tags within the Filtered trace set B tile and can also be removed from the tile.
Response time root cause entity paths	A table of recorded entity paths. X-Ray determines which path in your trace is the most likely cause for the response time. The format indicates a hierarchy of entities that are encountered, ending in a response time root cause. Use these rows to filter for recurring response time faults. For more information about customizing a root cause filter and getting data through the API, see the Retrieving and refining root cause analytics section in Getting data from X-Ray.
Delta (�)	A column that is added to the metrics tables when both trace set A and trace set B are active. The Delta column calculates the difference in percentage of traces between trace set A and trace set B.

The X-Ray Analytics console generates two primary graphs to help you visualize traces: Response Time Distribution and Time Series Activity. This section and the following provide examples of each, and explain the basics of how to read the graphs.

The following are the colors associated with the response time line graph (the time series graph uses the same color scheme):

• All traces in the group – gray

• Retrieved traces – orange

• Filtered trace set A – green

• Filtered trace set B – blue

Example – Response time distribution

The response time distribution is a chart that shows the number of traces with a given response time. Click and drag to make selections within the response time distribution. This selects and creates a filter on the working trace set named responseTime for all traces within a specific response time.

The time series activity chart shows the number of traces at a given time period. The color indicators mirror the line graph colors of the response time distribution. The darker and fuller the color block within the activity series, the more traces are represented at the given time.

Example – Time series activity

Click and drag to make selections within the time series activity graph. This selects and creates a filter named timerange on the working trace set for all traces within a specific range of time.

The following examples show common use cases for the X-Ray Analytics console. Each example demonstrates a key function of the console experience. As a group, the examples follow a basic troubleshooting workflow. The steps walk through how to first spot unhealthy nodes, and then how to interact with the Analytics console to automatically generate comparative queries. Once you have narrowed the scope through queries, you will finally look at the details of traces of interest to determine what is damaging the health of your service.

The trace map indicates the health of each node by coloring it based on the ratio of successful calls to errors and faults. When you see a percentage of red on your node, it signals a fault. Use the X-Ray Analytics console to investigate it.

For more information about how to read the trace map, see Use the X-Ray trace map.

Using the response time distribution, you can observe peaks in response time. By selecting the peak in response time, the tables below the graphs will update to expose all associated metrics, such as status codes.

When you click and drag, X-Ray selects and creates a filter. It's shown in a gray shadow on top of the graphed lines. You can now drag that shadow left and right along the distribution to update your selection and filter.

You can drill into traces within the selected peak by using the metrics tables below the graphs. By clicking a row in the HTTP STATUS CODE table, you automatically create a filter on the working dataset. For example, you could view all traces of status code 500. This creates a filter tag in the trace set tile named http.status.

Drill into the error set based on user, URL, response time root cause, or other predefined attributes. For example, to additionally filter the set of traces with a 500 status code, select a row from the USERS table. This results in two filter tags in the trace set tile: http.status, as designated previously, and user.

Compare across various users and their POST requests to find other discrepancies and correlations. Apply your first set of filters. They are defined by a blue line in the response time distribution. Then select Compare. Initially, this creates a copy of the filters on trace set A.

To proceed, define a new set of filters to apply to trace set B. This second set is represented by a green line. The following example shows different lines according to the blue and green color scheme.

As you narrow your scope using the console filters, the trace list below the metrics tables becomes more meaningful. The trace list table combines information about URL, USER, and STATUS CODE into one view. For more insights, select a row from this table to open the trace's detail page and view its timeline and raw data.

Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics. You can use the AWS X-Ray console or X-Ray API to create and manage groups for your services. This topic describes how to create and manage groups by using the X-Ray console. For information about how to manage groups by using the X-Ray API, see Configuring sampling, groups, and encryption settings with the X-Ray API.

You can create groups of traces for trace maps, traces, or analytics. When you create a group, the group becomes available as a filter on the group dropdown menu on all three pages: Trace Map, Traces, and Analytics.

Groups are identified by their name or an Amazon Resource Name (ARN), and contain a filter expression. The service compares incoming traces to the expression and stores them accordingly. For more information about how to build a filter expression, see Use filter expressions.

Updating a group's filter expression doesn't change data that's already recorded. The update applies only to subsequent traces. This can result in a merged graph of the new and old expressions. To avoid this, delete a current group and create a new one.

Create a group

Apply a group

Edit a group

Clone a group

Cloning a group creates a new group that has the filter expression and tags of an existing group. When you clone a group, the new group has the same name as the group from which it's cloned, with -clone appended to the name.

Delete a group

Follow steps in this section to delete a group. You can't delete the Default group.

View group metrics in Amazon CloudWatch

After a group is created, incoming traces are checked against the group’s filter expression as they're stored in the X-Ray service. Metrics for the number of traces matching each criteria are published to Amazon CloudWatch every minute. Choosing View metric on the Edit group page opens the CloudWatch console to the Metric page. For more information about how to use CloudWatch metrics, see Using Amazon CloudWatch Metrics in the Amazon CloudWatch User Guide.

You can use the AWS X-Ray console to configure sampling rules for your services. The X-Ray SDK and AWS services that support active tracing with sampling configuration use sampling rules to determine which requests to record.

Configure sampling rules

You can configure sampling for the following use cases:

Customizing sampling rules

By customizing sampling rules, you can control the amount of data that you record. You can also modify sampling behavior without modifying or redeploying your code. Sampling rules tell the X-Ray SDK how many requests to record for a set of criteria. By default, the X-Ray SDK records the first request received at the beginning of each second, and five percent of any additional requests. One request per second is the reservoir. This ensures that at least one trace is recorded each second as long as the service is serving requests. Five percent is the rate at which additional requests beyond the reservoir size are sampled.

You can configure the X-Ray SDK to read sampling rules from a JSON document that you include with your code. However, when you run multiple instances of your service, each instance performs sampling independently. This causes the overall percentage of requests sampled to increase because the reservoirs of all of the instances are effectively added together. Additionally, to update local sampling rules, you must redeploy your code.

By defining sampling rules in the X-Ray console, and configuring the SDK to read rules from the X-Ray service, you can avoid both of these issues. The service manages the reservoir for each rule, and assigns quotas to each instance of your service to distribute the reservoir evenly, based on the number of instances that are running. The reservoir limit is calculated according to the rules you set. Because the rules are configured in the service, you can manage rules without making additional deployments. For more informaiton about the AWS SDK, see Use an SDK.

You can now configure X-Ray sampling rules from within the Amazon CloudWatch console. You can also continue to use the X-Ray console.

Sampling rule options

The following options are available for each rule. String values can use wildcards to match a single character (?) or zero or more characters (*).

Sampling rule examples

Configure your service to use sampling rules

The X-Ray SDK requires additional configuration to use sampling rules that you configure in the console. See the configuration topic for your language for details on configuring a sampling strategy:

For API Gateway, see Amazon API Gateway active tracing support for AWS X-Ray.

Viewing sampling results

The X-Ray console Sampling page shows detailed information about how your services use each sampling rule.

The Trend column shows how the rule has been used in the last few minutes. Each column shows statistics for a 10-second window.

For more information about sampling statistics and how services use sampling rules, see Using sampling rules with the X-Ray API.

Next steps

You can use the X-Ray API to manage sampling rules. With the API, you can create and update rules programmatically on a schedule, or in response to alarms or notifications. See Configuring sampling, groups, and encryption settings with the X-Ray API for instructions and additional rule examples.

The X-Ray SDK and AWS services also use the X-Ray API to read sampling rules, report sampling results, and get sampling targets. Services must keep track of how often they apply each rule, evaluate rules based on priority, and borrow from the reservoir when a request matches a rule for which X-Ray has not yet assigned the service a quota. For more details about how a service uses the API for sampling, see Using sampling rules with the X-Ray API.

When the X-Ray SDK calls sampling APIs, it uses the X-Ray daemon as a proxy. If you already use TCP port 2000, you can configure the daemon to run the proxy on a different port. See Configuring the AWS X-Ray daemon for details.

You can use routes and queries to deep link into specific traces, or filtered views of traces and the trace map.

Traces

You can generate links for timeline, raw, and map views of individual traces.

Trace timeline – xray/home#/traces/trace-id

Raw trace data – xray/home#/traces/trace-id/raw

https://console.aws.amazon.com/xray/home#/traces/1-57f5498f-d91047849216d0f2ea3b6442/raw

Filter expressions

Link to a filtered list of traces.

Filtered traces view – xray/home#/traces?filter=filter-expression

https://console.aws.amazon.com/xray/home#/traces?filter=service("api.amazon.com") { fault = true OR responsetime > 2.5 } AND annotation.foo = "bar"

https://console.aws.amazon.com/xray/home#/traces?filter=service(%22api.amazon.com%22)%20%7B%20fault%20%3D%20true%20OR%20responsetime%20%3E%202.5%20%7D%20AND%20annotation.foo%20%3D%20%22bar%22

For more information about filter expressions, see Use filter expressions.

Time range

Specify a length of time or start and end time in ISO8601 format. Time ranges are in UTC and can be up to 6 hours long.

Length of time – xray/home#/page?timeRange=range-in-minutes

https://console.aws.amazon.com/xray/home#/service-map?timeRange=PT1H

Start and end time – xray/home#/page?timeRange=start~end

https://console.aws.amazon.com/xray/home#/traces?timeRange=2023-7-01T16:00:00~2023-7-01T22:00:00

https://console.aws.amazon.com/xray/home#/traces?timeRange=2023-7-01T16:00~2023-7-01T22:00

Region

Specify an AWS Region to link to pages in that Region. If you don't specify a Region, the console redirects you to the last visited Region.

Region – xray/home?region=region#/page

https://console.aws.amazon.com/xray/home?region=us-west-2#/service-map

When you include a Region with other query parameters, the Region query goes before the hash, and the X-Ray-specific queries go after the page name.

https://console.aws.amazon.com/xray/home?region=us-west-2#/service-map?timeRange=PT1H

Combined

https://console.aws.amazon.com/xray/home#/traces?timeRange=PT15M&filter=duration%20%3E%3D%205%20AND%20duration%20%3C%3D%208