Use the X-Ray API
If the X-Ray SDK doesn’t support your programming language, you can use either the X-Ray APIs directly or the AWS Command Line Interface (AWS CLI) to call X-Ray API commands. Use the following guidance to choose how you interact with the API:
-
Use the AWS CLI for simpler syntax using pre-formatted commands or with options inside your request.
-
Use the X-Ray API directly for maximum flexibility and customization for requests that you make to X-Ray.
If you use the X-Ray API directly instead of the AWS CLI, you must parametrize your request in the correct data format and may also have to configure authentication and error handling.
The following diagram shows guidance to choose how to interact with the X-Ray API:
Use the X-Ray API to send trace data to directly to X-Ray. The X-Ray API supports all functions available in the X-Ray SDK including the following common actions:
-
PutTraceSegments – Uploads segment documents to X-Ray.
-
BatchGetTraces – Retrieves a list of traces in a list of trace IDs. Each retrieved trace is a collection of segment documents from a single request.
-
GetTraceSummaries – Retrieves IDs and annotations for traces. You can specify a
FilterExpression
to retrieve a subset of trace summaries. -
GetTraceGraph – Retrieves a service graph for a specific trace ID.
-
GetServiceGraph – Retrieves a JSON formatted document that describes services that process incoming requests and call downstream requests.
You can also use the AWS Command Line Interface (AWS CLI) inside your application code to programmatically interact with X-Ray. The AWS CLI supports all functions available in the X-Ray SDK including those for other AWS services. The following functions are versions of the API operations listed previously with a simpler format:
-
put-trace-segments
– Uploads segment documents to X-Ray. -
batch-get-traces
– Retrieves a list of traces in a list of trace IDs. Each retrieved trace is a collection of segment documents from a single request. -
get-trace-summaries
– Retrieves IDs and annotations for traces. You can specify a FilterExpression
to retrieve a subset of trace summaries. -
get-trace-graph
– Retrieves a service graph for a specific trace ID. -
get-service-graph
– Retrieves a JSON
formatted document that describes services that process incoming requests and call downstream requests.
To get started, you must install the AWS CLI for your
operating system. AWS supports Linux, macOS and
Windows operating systems. For more information about the list of
X-Ray commands, see the AWS CLI Command Reference guide for X-Ray
Explore the X-Ray API
The X-Ray API provides access to all X-Ray functionality through the AWS SDK, AWS Command Line Interface, or directly over HTTPS. The X-Ray API Reference documents input parameters for each API action, and the fields and data types that they return.
You can use the AWS SDK to develop programs that use the X-Ray API. The X-Ray console and X-Ray daemon both use the AWS SDK to communicate with X-Ray. The AWS SDK for each language has a reference document for classes and methods that map to X-Ray API actions and types.
AWS SDK References
-
Java – AWS SDK for Java
-
JavaScript – AWS SDK for JavaScript
-
.NET – AWS SDK for .NET
-
Ruby – AWS SDK for Ruby
-
Go – AWS SDK for Go
-
PHP – AWS SDK for PHP
-
Python – AWS SDK for Python (Boto)
The AWS Command Line Interface is a command line tool that uses the SDK for Python to call AWS APIs. When you are first learning an AWS API, the AWS CLI provides an easy way to explore the available parameters and view the service output in JSON or text form.
See the AWS CLI Command Reference for details on
aws xray
subcommands.
The AWS CLI lets your access the X-Ray service directly and use the same APIs that the X-Ray console uses to retrieve the service graph and raw traces data. The sample application includes scripts that show how to use these APIs with the AWS CLI.
Prerequisites
This tutorial uses the Scorekeep sample application and included scripts to generate tracing data and a service map. Follow the instructions in the sample application tutorial to launch the application.
This tutorial uses the AWS CLI to show basic use of the X-Ray API. The AWS CLI, available for Windows, Linux, and OS-X, provides command line access to the public APIs for all AWS services.
Note
You must verify that your AWS CLI is configured to the same region where your sample application was created.
Scripts included to test the sample application uses cURL
to send traffic to the API and
jq
to parse the output. You can download the jq
executable from stedolan.github.iocurl
executable from
https://curl.haxx.se/download.html
Generate trace data
The web app continues to generate traffic to the API every few seconds while the game is in-progress, but only
generates one type of request. Use the test-api.sh
script to run end to end scenarios and
generate more diverse trace data while you test the API.
To use the test-api.sh
script
Open the Elastic Beanstalk console
. Navigate to the management console for your environment.
-
Copy the environment URL from the page header.
-
Open
bin/test-api.sh
and replace the value for API with your environment's URL.#!/bin/bash API=
scorekeep.9hbtbm23t2
.us-west-2.elasticbeanstalk.com/api -
Run the script to generate traffic to the API.
~/debugger-tutorial$
./bin/test-api.sh
Creating users, session, game, configuring game, playing game, ending game, game complete. {"id":"MTBP8BAS","session":"HUF6IT64","name":"tic-tac-toe-test","users":["QFF3HBGM","KL6JR98D"],"rules":"102","startTime":1476314241,"endTime":1476314245,"states":["JQVLEOM2","D67QLPIC","VF9BM9NC","OEAA6GK9","2A705O73","1U2LFTLJ","HUKIDD70","BAN1C8FI","G3UDJTUF","AB70HVEV"],"moves":["BS8F8LQ","4MTTSPKP","463OETES","SVEBCL3N","N7CQ1GHP","O84ONEPD","EG4BPROQ","V4BLIDJ3","9RL3NPMV"]}
Use the X-Ray API
The AWS CLI provides commands for all of the API actions that X-Ray provides, including GetServiceGraph
and GetTraceSummaries
. See the AWS X-Ray API
Reference for more information on all of the supported actions and the data types that they use.
Example bin/service-graph.sh
EPOCH=$(date +%s)
aws xray get-service-graph --start-time $(($EPOCH-600)) --end-time $EPOCH
The script retrieves a service graph for the last 10 minutes.
~/eb-java-scorekeep$ ./bin/service-graph.sh
| less
{
"StartTime": 1479068648.0,
"Services": [
{
"StartTime": 1479068648.0,
"ReferenceId": 0,
"State": "unknown",
"EndTime": 1479068651.0,
"Type": "client",
"Edges": [
{
"StartTime": 1479068648.0,
"ReferenceId": 1,
"SummaryStatistics": {
"ErrorStatistics": {
"ThrottleCount": 0,
"TotalCount": 0,
"OtherCount": 0
},
"FaultStatistics": {
"TotalCount": 0,
"OtherCount": 0
},
"TotalCount": 2,
"OkCount": 2,
"TotalResponseTime": 0.054000139236450195
},
"EndTime": 1479068651.0,
"Aliases": []
}
]
},
{
"StartTime": 1479068648.0,
"Names": [
"scorekeep.elasticbeanstalk.com"
],
"ReferenceId": 1,
"State": "active",
"EndTime": 1479068651.0,
"Root": true,
"Name": "scorekeep.elasticbeanstalk.com",
...
Example bin/trace-urls.sh
EPOCH=$(date +%s)
aws xray get-trace-summaries --start-time $(($EPOCH-120)) --end-time $(($EPOCH-60)) --query 'TraceSummaries[*].Http.HttpURL'
The script retrieves the URLs of traces generated between one and two minutes ago.
~/eb-java-scorekeep$ ./bin/trace-urls.sh
[
"http://scorekeep.elasticbeanstalk.com/api/game/6Q0UE1DG/5FGLM9U3/endtime/1479069438",
"http://scorekeep.elasticbeanstalk.com/api/session/KH4341QH",
"http://scorekeep.elasticbeanstalk.com/api/game/GLQBJ3K5/153AHDIA",
"http://scorekeep.elasticbeanstalk.com/api/game/VPDL672J/G2V41HM6/endtime/1479069466"
]
Example bin/full-traces.sh
EPOCH=$(date +%s)
TRACEIDS=$(aws xray get-trace-summaries --start-time $(($EPOCH-120)) --end-time $(($EPOCH-60)) --query 'TraceSummaries[*].Id' --output text)
aws xray batch-get-traces --trace-ids $TRACEIDS --query 'Traces[*]'
The script retrieves full traces generated between one and two minutes ago.
~/eb-java-scorekeep$ ./bin/full-traces.sh
| less
[
{
"Segments": [
{
"Id": "3f212bc237bafd5d",
"Document": "{\"id\":\"3f212bc237bafd5d\",\"name\":\"DynamoDB\",\"trace_id\":\"1-5828d9f2-a90669393f4343211bc1cf75\",\"start_time\":1.479072242459E9,\"end_time\":1.479072242477E9,\"parent_id\":\"72a08dcf87991ca9\",\"http\":{\"response\":{\"content_length\":60,\"status\":200}},\"inferred\":true,\"aws\":{\"consistent_read\":false,\"table_name\":\"scorekeep-session-xray\",\"operation\":\"GetItem\",\"request_id\":\"QAKE0S8DD0LJM245KAOPMA746BVV4KQNSO5AEMVJF66Q9ASUAAJG\",\"resource_names\":[\"scorekeep-session-xray\"]},\"origin\":\"AWS::DynamoDB::Table\"}"
},
{
"Id": "309e355f1148347f",
"Document": "{\"id\":\"309e355f1148347f\",\"name\":\"DynamoDB\",\"trace_id\":\"1-5828d9f2-a90669393f4343211bc1cf75\",\"start_time\":1.479072242477E9,\"end_time\":1.479072242494E9,\"parent_id\":\"37f14ef837f00022\",\"http\":{\"response\":{\"content_length\":606,\"status\":200}},\"inferred\":true,\"aws\":{\"table_name\":\"scorekeep-game-xray\",\"operation\":\"UpdateItem\",\"request_id\":\"388GEROC4PCA6D59ED3CTI5EEJVV4KQNSO5AEMVJF66Q9ASUAAJG\",\"resource_names\":[\"scorekeep-game-xray\"]},\"origin\":\"AWS::DynamoDB::Table\"}"
}
],
"Id": "1-5828d9f2-a90669393f4343211bc1cf75",
"Duration": 0.05099987983703613
}
...
Cleanup
Terminate your Elastic Beanstalk environment to shut down the Amazon EC2 instances, DynamoDB tables and other resources.
To terminate your Elastic Beanstalk environment
Open the Elastic Beanstalk console
. Navigate to the management console for your environment.
-
Choose Actions.
-
Choose Terminate Environment.
-
Choose Terminate.
Trace data is automatically deleted from X-Ray after 30 days.
You can send trace data to X-Ray in the form of segment documents. A segment document is a JSON formatted string that contains information about the work that your application does in service of a request. Your application can record data about the work that it does itself in segments, or work that uses downstream services and resources in subsegments.
Segments record information about the work that your application does. A segment, at a minimum, records the time spent on a task, a name, and two IDs. The trace ID tracks the request as it travels between services. The segment ID tracks the work done for the request by a single service.
Example Minimal complete segment
{
"name" : "Scorekeep",
"id" : "70de5b6f19ff9a0a",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979",
"end_time" : 1.478293361449E9
}
When a request is received, you can send an in-progress segment as a placeholder until the request is completed.
Example In-progress segment
{
"name" : "Scorekeep",
"id" : "70de5b6f19ff9a0b",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979",
“in_progress”: true
}
You can send segments to X-Ray directly, with PutTraceSegments
, or
through the X-Ray daemon.
Most applications call other services or access resources with the AWS SDK. Record information about downstream calls in subsegments. X-Ray uses subsegments to identify downstream services that don't send segments and create entries for them on the service graph.
A subsegment can be embedded in a full segment document, or sent separately. Send subsegments separately to asynchronously trace downstream calls for long-running requests, or to avoid exceeding the maximum segment document size (64 kB).
Example Subsegment
A subsegment has a type
of subsegment
and a
parent_id
that identifies the parent segment.
{
"name" : "www2.example.com",
"id" : "70de5b6f19ff9a0c",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979"
“end_time” : 1.478293361449E9,
“type” : “subsegment”,
“parent_id” : “70de5b6f19ff9a0b”
}
For more information on the fields and values that you can include in segments and subsegments, see X-Ray segment documents.
To send data to X-Ray, you must generate a unique trace ID for each request.
X-Ray trace ID format
An X-Ray trace_id
consists of three numbers separated by hyphens. For example,
1-58406520-a006649127e371903a2de979
. This includes:
-
The version number, which is
1
. -
The time of the original request in Unix epoch time using 8 hexadecimal digits.
For example, 10:00AM December 1st, 2016 PST in epoch time is
1480615200
seconds or58406520
in hexadecimal digits. -
A globally unique 96-bit identifier for the trace in 24 hexadecimal digits.
Note
X-Ray now supports trace IDs that are created using OpenTelemetry and any other framework that conforms with the
W3C Trace Context specification4efaaf4d1e8720b39541901950019ee5
should be formatted as
1-4efaaf4d-1e8720b39541901950019ee5
when sending to X-Ray. X-Ray trace IDs include the original
request time stamp in Unix epoch time, but this isn't required when sending W3C trace IDs in X-Ray format.
You can write a script to generate X-Ray trace IDs for testing. Here are two examples.
Python
import time
import os
import binascii
START_TIME = time.time()
HEX=hex(int(START_TIME))[2:]
TRACE_ID="1-{}-{}".format(HEX, binascii.hexlify(os.urandom(12)).decode('utf-8'))
Bash
START_TIME=$(date +%s)
HEX_TIME=$(printf '%x\n' $START_TIME)
GUID=$(dd if=/dev/random bs=12 count=1 2>/dev/null | od -An -tx1 | tr -d ' \t\n')
TRACE_ID="1-$HEX_TIME-$GUID"
See the Scorekeep sample application for scripts that create trace IDs and send segments to the X-Ray daemon.
-
Python –
xray_start.py
-
Bash –
xray_start.sh
You can upload segment documents with the PutTraceSegments
API. The API has a
single parameter, TraceSegmentDocuments
, that takes a list of JSON segment
documents.
With the AWS CLI, use the aws xray put-trace-segments
command to send
segment documents directly to X-Ray.
$ DOC='{"trace_id": "1-5960082b-ab52431b496add878434aa25", "id": "6226467e3f845502", "start_time": 1498082657.37518, "end_time": 1498082695.4042, "name": "test.elasticbeanstalk.com"}'
$ aws xray put-trace-segments --trace-segment-documents "$DOC"
{
"UnprocessedTraceSegments": []
}
Note
Windows Command Processor and Windows PowerShell have different requirements for quoting and escaping quotes in JSON strings. See Quoting Strings in the AWS CLI User Guide for details.
The output lists any segments that failed processing. For example, if the date in the trace ID is too far in the past, you see an error like the following.
{
"UnprocessedTraceSegments": [
{
"ErrorCode": "InvalidTraceId",
"Message": "Invalid segment. ErrorCode: InvalidTraceId",
"Id": "6226467e3f845502"
}
]
}
You can pass multiple segment documents at the same time, separated by spaces.
$ aws xray put-trace-segments --trace-segment-documents "$DOC1" "$DOC2"
Instead of sending segment documents to the X-Ray API, you can send segments and subsegments to the X-Ray daemon, which will buffer them and upload to the X-Ray API in batches. The X-Ray SDK sends segment documents to the daemon to avoid making calls to AWS directly.
Note
See Running the X-Ray daemon locally for instructions on running the daemon.
Send the segment in JSON over UDP port 2000, prepended by the daemon header,
{"format": "json", "version": 1}\n
{"format": "json", "version": 1}\n{"trace_id": "1-5759e988-bd862e3fe1be46a994272793", "id": "defdfd9912dc5a56", "start_time": 1461096053.37518, "end_time": 1461096053.4042, "name": "test.elasticbeanstalk.com"}
On Linux, you can send segment documents to the daemon from a Bash terminal. Save the
header and segment document to a text file and pipe it to /dev/udp
with
cat
.
$ cat segment.txt > /dev/udp/127.0.0.1/2000
Example segment.txt
{"format": "json", "version": 1}
{"trace_id": "1-594aed87-ad72e26896b3f9d3a27054bb", "id": "6226467e3f845502", "start_time": 1498082657.37518, "end_time": 1498082695.4042, "name": "test.elasticbeanstalk.com"}
Check the daemon log to verify that it sent the segment to X-Ray.
2017-07-07T01:57:24Z [Debug] processor: sending partial batch
2017-07-07T01:57:24Z [Debug] processor: segment batch size: 1. capacity: 50
2017-07-07T01:57:24Z [Info] Successfully sent batch of 1 segments (0.020 seconds)
X-Ray processes the trace data that you send to it to generate full traces, trace summaries, and service graphs in JSON. You can retrieve the generated data directly from the API with the AWS CLI.
You can use the GetServiceGraph
API to retrieve the JSON service graph. The API
requires a start time and end time, which you can calculate from a Linux terminal with the
date
command.
$ date +%s
1499394617
date +%s
prints a date in seconds. Use this number as an end time and
subtract time from it to get a start time.
Example Script to retrieve a service graph for the last 10 minutes
EPOCH=$(date +%s) aws xray get-service-graph --start-time $(($EPOCH-600)) --end-time $EPOCH
The following example shows a service graph with 4 nodes, including a client node, an EC2 instance, a DynamoDB table, and an Amazon SNS topic.
Example GetServiceGraph output
{ "Services": [ { "ReferenceId": 0, "Name": "xray-sample.elasticbeanstalk.com", "Names": [ "xray-sample.elasticbeanstalk.com" ], "Type": "client", "State": "unknown", "StartTime": 1528317567.0, "EndTime": 1528317589.0, "Edges": [ { "ReferenceId": 2, "StartTime": 1528317567.0, "EndTime": 1528317589.0, "SummaryStatistics": { "OkCount": 3, "ErrorStatistics": { "ThrottleCount": 0, "OtherCount": 1, "TotalCount": 1 }, "FaultStatistics": { "OtherCount": 0, "TotalCount": 0 }, "TotalCount": 4, "TotalResponseTime": 0.273 }, "ResponseTimeHistogram": [ { "Value": 0.005, "Count": 1 }, { "Value": 0.015, "Count": 1 }, { "Value": 0.157, "Count": 1 }, { "Value": 0.096, "Count": 1 } ], "Aliases": [] } ] }, { "ReferenceId": 1, "Name": "awseb-e-dixzws4s9p-stack-StartupSignupsTable-4IMSMHAYX2BA", "Names": [ "awseb-e-dixzws4s9p-stack-StartupSignupsTable-4IMSMHAYX2BA" ], "Type": "AWS::DynamoDB::Table", "State": "unknown", "StartTime": 1528317583.0, "EndTime": 1528317589.0, "Edges": [], "SummaryStatistics": { "OkCount": 2, "ErrorStatistics": { "ThrottleCount": 0, "OtherCount": 0, "TotalCount": 0 }, "FaultStatistics": { "OtherCount": 0, "TotalCount": 0 }, "TotalCount": 2, "TotalResponseTime": 0.12 }, "DurationHistogram": [ { "Value": 0.076, "Count": 1 }, { "Value": 0.044, "Count": 1 } ], "ResponseTimeHistogram": [ { "Value": 0.076, "Count": 1 }, { "Value": 0.044, "Count": 1 } ] }, { "ReferenceId": 2, "Name": "xray-sample.elasticbeanstalk.com", "Names": [ "xray-sample.elasticbeanstalk.com" ], "Root": true, "Type": "AWS::EC2::Instance", "State": "active", "StartTime": 1528317567.0, "EndTime": 1528317589.0, "Edges": [ { "ReferenceId": 1, "StartTime": 1528317567.0, "EndTime": 1528317589.0, "SummaryStatistics": { "OkCount": 2, "ErrorStatistics": { "ThrottleCount": 0, "OtherCount": 0, "TotalCount": 0 }, "FaultStatistics": { "OtherCount": 0, "TotalCount": 0 }, "TotalCount": 2, "TotalResponseTime": 0.12 }, "ResponseTimeHistogram": [ { "Value": 0.076, "Count": 1 }, { "Value": 0.044, "Count": 1 } ], "Aliases": [] }, { "ReferenceId": 3, "StartTime": 1528317567.0, "EndTime": 1528317589.0, "SummaryStatistics": { "OkCount": 2, "ErrorStatistics": { "ThrottleCount": 0, "OtherCount": 0, "TotalCount": 0 }, "FaultStatistics": { "OtherCount": 0, "TotalCount": 0 }, "TotalCount": 2, "TotalResponseTime": 0.125 }, "ResponseTimeHistogram": [ { "Value": 0.049, "Count": 1 }, { "Value": 0.076, "Count": 1 } ], "Aliases": [] } ], "SummaryStatistics": { "OkCount": 3, "ErrorStatistics": { "ThrottleCount": 0, "OtherCount": 1, "TotalCount": 1 }, "FaultStatistics": { "OtherCount": 0, "TotalCount": 0 }, "TotalCount": 4, "TotalResponseTime": 0.273 }, "DurationHistogram": [ { "Value": 0.005, "Count": 1 }, { "Value": 0.015, "Count": 1 }, { "Value": 0.157, "Count": 1 }, { "Value": 0.096, "Count": 1 } ], "ResponseTimeHistogram": [ { "Value": 0.005, "Count": 1 }, { "Value": 0.015, "Count": 1 }, { "Value": 0.157, "Count": 1 }, { "Value": 0.096, "Count": 1 } ] }, { "ReferenceId": 3, "Name": "SNS", "Names": [ "SNS" ], "Type": "AWS::SNS", "State": "unknown", "StartTime": 1528317583.0, "EndTime": 1528317589.0, "Edges": [], "SummaryStatistics": { "OkCount": 2, "ErrorStatistics": { "ThrottleCount": 0, "OtherCount": 0, "TotalCount": 0 }, "FaultStatistics": { "OtherCount": 0, "TotalCount": 0 }, "TotalCount": 2, "TotalResponseTime": 0.125 }, "DurationHistogram": [ { "Value": 0.049, "Count": 1 }, { "Value": 0.076, "Count": 1 } ], "ResponseTimeHistogram": [ { "Value": 0.049, "Count": 1 }, { "Value": 0.076, "Count": 1 } ] } ] }
To call for a service graph based on the contents of a group, include a
groupName
or groupARN
. The following example shows a service graph
call to a group named Example1.
Example Script to retrieve a service graph by name for group Example1
aws xray get-service-graph --group-name "Example1"
You can use the GetTraceSummaries
API to get a list of trace summaries. Trace
summaries include information that you can use to identify traces that you want to download in
full, including annotations, request and response information, and IDs.
There are two TimeRangeType
flags available when calling aws xray
get-trace-summaries
:
-
TraceId – The default
GetTraceSummaries
search uses TraceID time and returns traces started within the computed[start_time, end_time)
range. This range of timestamps is calculated based on the encoding of the timestamp within the TraceId, or can be defined manually. -
Event time – To search for events as they happen over the time, AWS X-Ray allows searching for traces using event timestamps. Event time returns traces active during the
[start_time, end_time)
range, regardless of when the trace began.
Use the aws xray get-trace-summaries
command to get a list of trace
summaries. The following commands get a list of trace summaries from between 1 and 2 minutes
in the past using the default TraceId time.
Example Script to get trace summaries
EPOCH=$(date +%s) aws xray get-trace-summaries --start-time $(($EPOCH-120)) --end-time $(($EPOCH-60))
Example GetTraceSummaries output
{
"TraceSummaries": [
{
"HasError": false,
"Http": {
"HttpStatus": 200,
"ClientIp": "205.255.255.183",
"HttpURL": "http://scorekeep.elasticbeanstalk.com/api/session",
"UserAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
"HttpMethod": "POST"
},
"Users": [],
"HasFault": false,
"Annotations": {},
"ResponseTime": 0.084,
"Duration": 0.084,
"Id": "1-59602606-a43a1ac52fc7ee0eea12a82c",
"HasThrottle": false
},
{
"HasError": false,
"Http": {
"HttpStatus": 200,
"ClientIp": "205.255.255.183",
"HttpURL": "http://scorekeep.elasticbeanstalk.com/api/user",
"UserAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
"HttpMethod": "POST"
},
"Users": [
{
"UserName": "5M388M1E"
}
],
"HasFault": false,
"Annotations": {
"UserID": [
{
"AnnotationValue": {
"StringValue": "5M388M1E"
}
}
],
"Name": [
{
"AnnotationValue": {
"StringValue": "Ola"
}
}
]
},
"ResponseTime": 3.232,
"Duration": 3.232,
"Id": "1-59602603-23fc5b688855d396af79b496",
"HasThrottle": false
}
],
"ApproximateTime": 1499473304.0,
"TracesProcessedCount": 2
}
Use the trace ID from the output to retrieve a full trace with the BatchGetTraces
API.
Example BatchGetTraces command
$ aws xray batch-get-traces --trace-ids 1-596025b4-7170afe49f7aa708b1dd4a6b
Example BatchGetTraces output
{
"Traces": [
{
"Duration": 3.232,
"Segments": [
{
"Document": "{\"id\":\"1fb07842d944e714\",\"name\":\"random-name\",\"start_time\":1.499473411677E9,\"end_time\":1.499473414572E9,\"parent_id\":\"0c544c1b1bbff948\",\"http\":{\"response\":{\"status\":200}},\"aws\":{\"request_id\":\"ac086670-6373-11e7-a174-f31b3397f190\"},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"origin\":\"AWS::Lambda\",\"resource_arn\":\"arn:aws:lambda:us-west-2:123456789012:function:random-name\"}",
"Id": "1fb07842d944e714"
},
{
"Document": "{\"id\":\"194fcc8747581230\",\"name\":\"Scorekeep\",\"start_time\":1.499473411562E9,\"end_time\":1.499473414794E9,\"http\":{\"request\":{\"url\":\"http://scorekeep.elasticbeanstalk.com/api/user\",\"method\":\"POST\",\"user_agent\":\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36\",\"client_ip\":\"205.251.233.183\"},\"response\":{\"status\":200}},\"aws\":{\"elastic_beanstalk\":{\"version_label\":\"app-abb9-170708_002045\",\"deployment_id\":406,\"environment_name\":\"scorekeep-dev\"},\"ec2\":{\"availability_zone\":\"us-west-2c\",\"instance_id\":\"i-0cd9e448944061b4a\"},\"xray\":{\"sdk_version\":\"1.1.2\",\"sdk\":\"X-Ray for Java\"}},\"service\":{},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"user\":\"5M388M1E\",\"origin\":\"AWS::ElasticBeanstalk::Environment\",\"subsegments\":[{\"id\":\"0c544c1b1bbff948\",\"name\":\"Lambda\",\"start_time\":1.499473411629E9,\"end_time\":1.499473414572E9,\"http\":{\"response\":{\"status\":200,\"content_length\":14}},\"aws\":{\"log_type\":\"None\",\"status_code\":200,\"function_name\":\"random-name\",\"invocation_type\":\"RequestResponse\",\"operation\":\"Invoke\",\"request_id\":\"ac086670-6373-11e7-a174-f31b3397f190\",\"resource_names\":[\"random-name\"]},\"namespace\":\"aws\"},{\"id\":\"071684f2e555e571\",\"name\":\"## UserModel.saveUser\",\"start_time\":1.499473414581E9,\"end_time\":1.499473414769E9,\"metadata\":{\"debug\":{\"test\":\"Metadata string from UserModel.saveUser\"}},\"subsegments\":[{\"id\":\"4cd3f10b76c624b4\",\"name\":\"DynamoDB\",\"start_time\":1.49947341469E9,\"end_time\":1.499473414769E9,\"http\":{\"response\":{\"status\":200,\"content_length\":57}},\"aws\":{\"table_name\":\"scorekeep-user\",\"operation\":\"UpdateItem\",\"request_id\":\"MFQ8CGJ3JTDDVVVASUAAJGQ6NJ82F738BOB4KQNSO5AEMVJF66Q9\",\"resource_names\":[\"scorekeep-user\"]},\"namespace\":\"aws\"}]}]}",
"Id": "194fcc8747581230"
},
{
"Document": "{\"id\":\"00f91aa01f4984fd\",\"name\":\"random-name\",\"start_time\":1.49947341283E9,\"end_time\":1.49947341457E9,\"parent_id\":\"1fb07842d944e714\",\"aws\":{\"function_arn\":\"arn:aws:lambda:us-west-2:123456789012:function:random-name\",\"resource_names\":[\"random-name\"],\"account_id\":\"123456789012\"},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"origin\":\"AWS::Lambda::Function\",\"subsegments\":[{\"id\":\"e6d2fe619f827804\",\"name\":\"annotations\",\"start_time\":1.499473413012E9,\"end_time\":1.499473413069E9,\"annotations\":{\"UserID\":\"5M388M1E\",\"Name\":\"Ola\"}},{\"id\":\"b29b548af4d54a0f\",\"name\":\"SNS\",\"start_time\":1.499473413112E9,\"end_time\":1.499473414071E9,\"http\":{\"response\":{\"status\":200}},\"aws\":{\"operation\":\"Publish\",\"region\":\"us-west-2\",\"request_id\":\"a2137970-f6fc-5029-83e8-28aadeb99198\",\"retries\":0,\"topic_arn\":\"arn:aws:sns:us-west-2:123456789012:awseb-e-ruag3jyweb-stack-NotificationTopic-6B829NT9V5O9\"},\"namespace\":\"aws\"},{\"id\":\"2279c0030c955e52\",\"name\":\"Initialization\",\"start_time\":1.499473412064E9,\"end_time\":1.499473412819E9,\"aws\":{\"function_arn\":\"arn:aws:lambda:us-west-2:123456789012:function:random-name\"}}]}",
"Id": "00f91aa01f4984fd"
},
{
"Document": "{\"id\":\"17ba309b32c7fbaf\",\"name\":\"DynamoDB\",\"start_time\":1.49947341469E9,\"end_time\":1.499473414769E9,\"parent_id\":\"4cd3f10b76c624b4\",\"inferred\":true,\"http\":{\"response\":{\"status\":200,\"content_length\":57}},\"aws\":{\"table_name\":\"scorekeep-user\",\"operation\":\"UpdateItem\",\"request_id\":\"MFQ8CGJ3JTDDVVVASUAAJGQ6NJ82F738BOB4KQNSO5AEMVJF66Q9\",\"resource_names\":[\"scorekeep-user\"]},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"origin\":\"AWS::DynamoDB::Table\"}",
"Id": "17ba309b32c7fbaf"
},
{
"Document": "{\"id\":\"1ee3c4a523f89ca5\",\"name\":\"SNS\",\"start_time\":1.499473413112E9,\"end_time\":1.499473414071E9,\"parent_id\":\"b29b548af4d54a0f\",\"inferred\":true,\"http\":{\"response\":{\"status\":200}},\"aws\":{\"operation\":\"Publish\",\"region\":\"us-west-2\",\"request_id\":\"a2137970-f6fc-5029-83e8-28aadeb99198\",\"retries\":0,\"topic_arn\":\"arn:aws:sns:us-west-2:123456789012:awseb-e-ruag3jyweb-stack-NotificationTopic-6B829NT9V5O9\"},\"trace_id\":\"1-59602603-23fc5b688855d396af79b496\",\"origin\":\"AWS::SNS\"}",
"Id": "1ee3c4a523f89ca5"
}
],
"Id": "1-59602603-23fc5b688855d396af79b496"
}
],
"UnprocessedTraceIds": []
}
The full trace includes a document for each segment, compiled from all of the segment documents received with the same trace ID. These documents don't represent the data as it was sent to X-Ray by your application. Instead, they represent the processed documents generated by the X-Ray service. X-Ray creates the full trace document by compiling segment documents sent by your application, and removing data that doesn't comply with the segment document schema. For more information, see X-Ray segment documents.
X-Ray also creates inferred segments for downstream calls to services that don't send segments themselves. For example, when you call DynamoDB with an instrumented client, the X-Ray SDK records a subsegment with details about the call from its point of view. However, DynamoDB doesn't send a corresponding segment. X-Ray uses the information in the subsegment to create an inferred segment to represent the DynamoDB resource in the trace map, and adds it to the trace document.
To get multiple traces from the API, you need a list of trace IDs, which you can extract
from the output of get-trace-summaries
with an AWS CLI query.
Redirect the list to the input of batch-get-traces
to get full traces for a
specific time period.
Example Script to get full traces for a one minute period
EPOCH=$(date +%s) TRACEIDS=$(aws xray get-trace-summaries --start-time $(($EPOCH-120)) --end-time $(($EPOCH-60)) --query 'TraceSummaries[*].Id' --output text) aws xray batch-get-traces --trace-ids $TRACEIDS --query 'Traces[*]'
Upon generating a trace summary with the GetTraceSummaries API , partial trace summaries can be reused in their JSON format to create a refined filter expression based upon root causes. See the examples below for a walkthrough of the refinement steps.
Example GetTraceSummaries output - response time root cause section
{ "Services": [ { "Name": "GetWeatherData", "Names": ["GetWeatherData"], "AccountId": 123456789012, "Type": null, "Inferred": false, "EntityPath": [ { "Name": "GetWeatherData", "Coverage": 1.0, 'Remote": false }, { "Name": "get_temperature", "Coverage": 0.8, "Remote": false } ] }, { "Name": "GetTemperature", "Names": ["GetTemperature"], "AccountId": 123456789012, "Type": null, "Inferred": false, "EntityPath": [ { "Name": "GetTemperature", "Coverage": 0.7, "Remote": false } ] } ] }
By editing and making omissions to the above output, this JSON can become a filter for matched root cause entities. For every field present in the JSON, any candidate match must be exact, or the trace will not be returned. Removed fields become wildcard values, a format which is compatible with the filter expression query structure.
Example Reformatted response time root cause
{ "Services": [ { "Name": "GetWeatherData", "EntityPath": [ { "Name": "GetWeatherData" }, { "Name": "get_temperature" } ] }, { "Name": "GetTemperature", "EntityPath": [ { "Name": "GetTemperature" } ] } ] }
This JSON is then used as part of a filter expression through a call to
rootcause.json = #[{}]
. Refer to the Use filter expressions section in Explore the X-Ray console for more details about querying with filter
expressions.
Example JSON filter
rootcause.json = #[
{ "Services": [ { "Name": "GetWeatherData", "EntityPath": [{ "Name": "GetWeatherData" }, { "Name": "get_temperature" } ] }, { "Name": "GetTemperature", "EntityPath": [ { "Name": "GetTemperature" } ] } ] }
]
X-Ray provides APIs for configuring sampling rules, group rules, and encryption settings.
Encryption settings
Use PutEncryptionConfig
to specify an AWS Key Management Service (AWS KMS) key to use for
encryption.
Note
X-Ray does not support asymmetric KMS keys.
$ aws xray put-encryption-config --type KMS --key-id alias/aws/xray
{
"EncryptionConfig": {
"KeyId": "arn:aws:kms:us-east-2:123456789012:key/c234g4e8-39e9-4gb0-84e2-b0ea215cbba5",
"Status": "UPDATING",
"Type": "KMS"
}
}
For the key ID, you can use an alias (as shown in the example), a key ID, or an Amazon Resource Name (ARN).
Use GetEncryptionConfig
to get the current configuration. When X-Ray finishes
applying your settings, the status changes from UPDATING
to
ACTIVE
.
$ aws xray get-encryption-config
{
"EncryptionConfig": {
"KeyId": "arn:aws:kms:us-east-2:123456789012:key/c234g4e8-39e9-4gb0-84e2-b0ea215cbba5",
"Status": "ACTIVE",
"Type": "KMS"
}
}
To stop using a KMS key and use default encryption, set the encryption type to
NONE
.
$ aws xray put-encryption-config --type NONE
{
"EncryptionConfig": {
"Status": "UPDATING",
"Type": "NONE"
}
}
You can manage the sampling rules in your account with the X-Ray API. For more information about sampling, see Configure sampling rules. For more information about adding and managing tags, see Tagging X-Ray sampling rules and groups.
Get all sampling rules with GetSamplingRules
.
$ aws xray get-sampling-rules
{
"SamplingRuleRecords": [
{
"SamplingRule": {
"RuleName": "Default",
"RuleARN": "arn:aws:xray:us-east-2:123456789012:sampling-rule/Default",
"ResourceARN": "*",
"Priority": 10000,
"FixedRate": 0.05,
"ReservoirSize": 1,
"ServiceName": "*",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 0.0,
"ModifiedAt": 1529959993.0
}
]
}
The default rule applies to all requests that don't match another rule. It is the
lowest priority rule and cannot be deleted. You can, however, change the rate and
reservoir size with UpdateSamplingRule
.
Example API input for UpdateSamplingRule
– 10000-default.json
{ "SamplingRuleUpdate": { "RuleName": "Default", "FixedRate": 0.01, "ReservoirSize": 0 } }
The following example uses the previous file as input to change the default rule to one percent with no reservoir. Tags are optional. If you choose to add tags, a tag key is required, and tag values are optional. To remove existing tags from a sampling rule, use UntagResource.
$ aws xray update-sampling-rule --cli-input-json file://1000-default.json --tags [{"Key": "key_name
","Value": "value
"},{"Key": "key_name
","Value": "value
"}]
{
"SamplingRuleRecords": [
{
"SamplingRule": {
"RuleName": "Default",
"RuleARN": "arn:aws:xray:us-east-2:123456789012:sampling-rule/Default",
"ResourceARN": "*",
"Priority": 10000,
"FixedRate": 0.01,
"ReservoirSize": 0,
"ServiceName": "*",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 0.0,
"ModifiedAt": 1529959993.0
},
Create additional sampling rules with CreateSamplingRule
. When you create a
rule, most of the rule fields are required. The following example creates two rules.
This first rule sets a base rate for the Scorekeep sample application. It matches all
requests served by the API that don't match a higher priority rule.
Example API input for UpdateSamplingRule
– 9000-base-scorekeep.json
{ "SamplingRule": { "RuleName": "base-scorekeep", "ResourceARN": "*", "Priority": 9000, "FixedRate": 0.1, "ReservoirSize": 5, "ServiceName": "Scorekeep", "ServiceType": "*", "Host": "*", "HTTPMethod": "*", "URLPath": "*", "Version": 1 } }
The second rule also applies to Scorekeep, but it has a higher priority and is more specific. This rule sets a very low sampling rate for polling requests. These are GET requests made by the client every few seconds to check for changes to the game state.
Example API input for UpdateSamplingRule
–
5000-polling-scorekeep.json
{ "SamplingRule": { "RuleName": "polling-scorekeep", "ResourceARN": "*", "Priority": 5000, "FixedRate": 0.003, "ReservoirSize": 0, "ServiceName": "Scorekeep", "ServiceType": "*", "Host": "*", "HTTPMethod": "GET", "URLPath": "/api/state/*", "Version": 1 } }
Tags are optional. If you choose to add tags, a tag key is required, and tag values are optional.
$ aws xray create-sampling-rule --cli-input-json file://5000-polling-scorekeep.json --tags [{"Key": "key_name
","Value": "value
"},{"Key": "key_name
","Value": "value
"}]
{
"SamplingRuleRecord": {
"SamplingRule": {
"RuleName": "polling-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1:123456789012:sampling-rule/polling-scorekeep",
"ResourceARN": "*",
"Priority": 5000,
"FixedRate": 0.003,
"ReservoirSize": 0,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "GET",
"URLPath": "/api/state/*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530574399.0,
"ModifiedAt": 1530574399.0
}
}
$ aws xray create-sampling-rule --cli-input-json file://9000-base-scorekeep.json
{
"SamplingRuleRecord": {
"SamplingRule": {
"RuleName": "base-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1:123456789012:sampling-rule/base-scorekeep",
"ResourceARN": "*",
"Priority": 9000,
"FixedRate": 0.1,
"ReservoirSize": 5,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530574410.0,
"ModifiedAt": 1530574410.0
}
}
To delete a sampling rule, use DeleteSamplingRule
.
$ aws xray delete-sampling-rule --rule-name polling-scorekeep
{
"SamplingRuleRecord": {
"SamplingRule": {
"RuleName": "polling-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1:123456789012:sampling-rule/polling-scorekeep",
"ResourceARN": "*",
"Priority": 5000,
"FixedRate": 0.003,
"ReservoirSize": 0,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "GET",
"URLPath": "/api/state/*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530574399.0,
"ModifiedAt": 1530574399.0
}
}
You can use the X-Ray API to manage groups in your account. Groups are a collection of traces that are defined by a filter expression. You can use groups to generate additional service graphs and supply Amazon CloudWatch metrics. See Getting data from X-Ray for more details about working with service graphs and metrics through the X-Ray API. For more information about groups, see Configure groups. For more information about adding and managing tags, see Tagging X-Ray sampling rules and groups.
Create a group with CreateGroup
. Tags are optional. If you choose to add
tags, a tag key is required, and tag values are optional.
$ aws xray create-group --group-name "TestGroup" --filter-expression "service(\"example.com\") {fault}" --tags [{"Key": "key_name
","Value": "value
"},{"Key": "key_name
","Value": "value
"}]
{
"GroupName": "TestGroup",
"GroupARN": "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID",
"FilterExpression": "service(\"example.com\") {fault OR error}"
}
Get all existing groups with GetGroups
.
$ aws xray get-groups
{
"Groups": [
{
"GroupName": "TestGroup",
"GroupARN": "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID",
"FilterExpression": "service(\"example.com\") {fault OR error}"
},
{
"GroupName": "TestGroup2",
"GroupARN": "arn:aws:xray:us-east-2:123456789012:group/TestGroup2/UniqueID",
"FilterExpression": "responsetime > 2"
}
],
"NextToken": "tokenstring"
}
Update a group with UpdateGroup
. Tags are optional. If you choose to add
tags, a tag key is required, and tag values are optional. To remove existing tags from a
group, use UntagResource.
$ aws xray update-group --group-name "TestGroup" --group-arn "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID" --filter-expression "service(\"example.com\") {fault OR error}" --tags [{"Key": "Stage","Value": "Prod"},{"Key": "Department","Value": "QA"}]
{
"GroupName": "TestGroup",
"GroupARN": "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID",
"FilterExpression": "service(\"example.com\") {fault OR error}"
}
Delete a group with DeleteGroup
.
$ aws xray delete-group --group-name "TestGroup" --group-arn "arn:aws:xray:us-east-2:123456789012:group/TestGroup/UniqueID"
{
}
The X-Ray SDK uses the X-Ray API to get sampling rules, report sampling results, and get quotas. You can use these APIs to get a better understanding of how sampling rules work, or to implement sampling in a language that the X-Ray SDK doesn't support.
Start by getting all sampling rules with GetSamplingRules
.
$ aws xray get-sampling-rules
{
"SamplingRuleRecords": [
{
"SamplingRule": {
"RuleName": "Default",
"RuleARN": "arn:aws:xray:us-east-1::sampling-rule/Default",
"ResourceARN": "*",
"Priority": 10000,
"FixedRate": 0.01,
"ReservoirSize": 0,
"ServiceName": "*",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 0.0,
"ModifiedAt": 1530558121.0
},
{
"SamplingRule": {
"RuleName": "base-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1::sampling-rule/base-scorekeep",
"ResourceARN": "*",
"Priority": 9000,
"FixedRate": 0.1,
"ReservoirSize": 2,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "*",
"URLPath": "*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530573954.0,
"ModifiedAt": 1530920505.0
},
{
"SamplingRule": {
"RuleName": "polling-scorekeep",
"RuleARN": "arn:aws:xray:us-east-1::sampling-rule/polling-scorekeep",
"ResourceARN": "*",
"Priority": 5000,
"FixedRate": 0.003,
"ReservoirSize": 0,
"ServiceName": "Scorekeep",
"ServiceType": "*",
"Host": "*",
"HTTPMethod": "GET",
"URLPath": "/api/state/*",
"Version": 1,
"Attributes": {}
},
"CreatedAt": 1530918163.0,
"ModifiedAt": 1530918163.0
}
]
}
The output includes the default rule and custom rules. See Configuring sampling, groups, and encryption settings with the X-Ray API if you haven't yet created sampling rules.
Evaluate rules against incoming requests in ascending order of priority. When a rule matches, use the fixed rate and reservoir size to make a sampling decision. Record sampled requests and ignore (for tracing purposes) unsampled requests. Stop evaluating rules when a sampling decision is made.
A rules reservoir size is the target number of traces to record per second before applying the fixed rate. The reservoir applies across all services cumulatively, so you can't use it directly. However, if it is non-zero, you can borrow one trace per second from the reservoir until X-Ray assigns a quota. Before receiving a quota, record the first request each second, and apply the fixed rate to additional requests. The fixed rate is a decimal between 0 and 1.00 (100%).
The following example shows a call to GetSamplingTargets
with details about sampling decisions made over
the last 10 seconds.
$ aws xray get-sampling-targets --sampling-statistics-documents '[
{
"RuleName": "base-scorekeep",
"ClientID": "ABCDEF1234567890ABCDEF10",
"Timestamp": "2018-07-07T00:20:06",
"RequestCount": 110,
"SampledCount": 20,
"BorrowCount": 10
},
{
"RuleName": "polling-scorekeep",
"ClientID": "ABCDEF1234567890ABCDEF10",
"Timestamp": "2018-07-07T00:20:06",
"RequestCount": 10500,
"SampledCount": 31,
"BorrowCount": 0
}
]'
{
"SamplingTargetDocuments": [
{
"RuleName": "base-scorekeep",
"FixedRate": 0.1,
"ReservoirQuota": 2,
"ReservoirQuotaTTL": 1530923107.0,
"Interval": 10
},
{
"RuleName": "polling-scorekeep",
"FixedRate": 0.003,
"ReservoirQuota": 0,
"ReservoirQuotaTTL": 1530923107.0,
"Interval": 10
}
],
"LastRuleModification": 1530920505.0,
"UnprocessedStatistics": []
}
The response from X-Ray includes a quota to use instead of borrowing from the reservoir. In this example, the service borrowed 10 traces from the reservoir over 10 seconds, and applied the fixed rate of 10 percent to the other 100 requests, resulting in a total of 20 sampled requests. The quota is good for five minutes (indicated by the time to live) or until a new quota is assigned. X-Ray may also assign a longer reporting interval than the default, although it didn't here.
Note
The response from X-Ray might not include a quota the first time you call it. Continue borrowing from the reservoir until you are assigned a quota.
The other two fields in the response might indicate issues with the input. Check
LastRuleModification
against the last time you called GetSamplingRules
. If it's newer,
get a new copy of the rules. UnprocessedStatistics
can include errors that indicate that a rule has
been deleted, that the statistics document in the input was too old, or permissions errors.
A trace segment is a JSON representation of a request that your application serves. A trace segment records information about the original request, information about the work that your application does locally, and subsegments with information about downstream calls that your application makes to AWS resources, HTTP APIs, and SQL databases.
A segment document conveys information about a segment to
X-Ray. A segment document can be up to 64 kB and contain a whole segment with
subsegments, a fragment of a segment that indicates that a request is in progress, or a single
subsegment that is sent separately. You can send segment documents directly to X-Ray by using
the PutTraceSegments
API.
X-Ray compiles and processes segment documents to generate queryable trace summaries and full traces that you can
access by using the GetTraceSummaries
and BatchGetTraces
APIs, respectively. In
addition to the segments and subsegments that you send to X-Ray, the service uses information
in subsegments to generate inferred segments and adds them to
the full trace. Inferred segments represent downstream services and resources in the trace
map.
X-Ray provides a JSON schema for segment documents. You can download the schema here: xray-segmentdocument-schema-v1.0.0. The fields and objects listed in the schema are described in more detail in the following sections.
A subset of segment fields are indexed by X-Ray for use with filter expressions. For
example, if you set the user
field on a segment to a unique identifier, you can
search for segments associated with specific users in the X-Ray console or by using the
GetTraceSummaries
API. For more information, see Use filter expressions.
When you instrument your application with the X-Ray SDK, the SDK generates segment documents for you. Instead of sending segment documents directly to X-Ray, the SDK transmits them over a local UDP port to the X-Ray daemon. For more information, see Sending segment documents to the X-Ray daemon.
A segment records tracing information about a request that your application serves. At a minimum, a segment records the name, ID, start time, trace ID, and end time of the request.
Example Minimal complete segment
{
"name" : "example.com",
"id" : "70de5b6f19ff9a0a",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979",
"end_time" : 1.478293361449E9
}
The following fields are required, or conditionally required, for segments.
Note
Values must be strings (up to 250 characters) unless noted otherwise.
Required Segment Fields
-
name
– The logical name of the service that handled the request, up to 200 characters. For example, your application's name or domain name. Names can contain Unicode letters, numbers, and whitespace, and the following symbols:_
,.
,:
,/
,%
,&
,#
,=
,+
,\
,-
,@
-
id
– A 64-bit identifier for the segment, unique among segments in the same trace, in 16 hexadecimal digits. -
trace_id
– A unique identifier that connects all segments and subsegments originating from a single client request.X-Ray trace ID format
An X-Ray
trace_id
consists of three numbers separated by hyphens. For example,1-58406520-a006649127e371903a2de979
. This includes:-
The version number, which is
1
. -
The time of the original request in Unix epoch time using 8 hexadecimal digits.
For example, 10:00AM December 1st, 2016 PST in epoch time is
1480615200
seconds or58406520
in hexadecimal digits. -
A globally unique 96-bit identifier for the trace in 24 hexadecimal digits.
Note
X-Ray now supports trace IDs that are created using OpenTelemetry and any other framework that conforms with the W3C Trace Context specification
. A W3C trace ID must be formatted in X-Ray trace ID format when sending to X-Ray. For example, W3C trace ID 4efaaf4d1e8720b39541901950019ee5
should be formatted as1-4efaaf4d-1e8720b39541901950019ee5
when sending to X-Ray. X-Ray trace IDs include the original request time stamp in Unix epoch time, but this isn't required when sending W3C trace IDs in X-Ray format.Trace ID Security
Trace IDs are visible in response headers. Generate trace IDs with a secure random algorithm to ensure that attackers cannot calculate future trace IDs and send requests with those IDs to your application.
-
-
start_time
– number that is the time the segment was created, in floating point seconds in epoch time. For example,1480615200.010
or1.480615200010E9
. Use as many decimal places as you need. Microsecond resolution is recommended when available. -
end_time
– number that is the time the segment was closed. For example,1480615200.090
or1.480615200090E9
. Specify either anend_time
orin_progress
. -
in_progress
– boolean, set totrue
instead of specifying anend_time
to record that a segment is started, but is not complete. Send an in-progress segment when your application receives a request that will take a long time to serve, to trace the request receipt. When the response is sent, send the complete segment to overwrite the in-progress segment. Only send one complete segment, and one or zero in-progress segments, per request.
Service Names
A segment's name
should match the domain name or logical name of the service
that generates the segment. However, this is not enforced. Any application that has permission to
PutTraceSegments
can send segments with any name.
The following fields are optional for segments.
Optional Segment Fields
-
service
– An object with information about your application.-
version
– A string that identifies the version of your application that served the request.
-
-
user
– A string that identifies the user who sent the request. -
origin
– The type of AWS resource running your application.Supported Values
-
AWS::EC2::Instance
– An Amazon EC2 instance. -
AWS::ECS::Container
– An Amazon ECS container. -
AWS::ElasticBeanstalk::Environment
– An Elastic Beanstalk environment.
When multiple values are applicable to your application, use the one that is most specific. For example, a Multicontainer Docker Elastic Beanstalk environment runs your application on an Amazon ECS container, which in turn runs on an Amazon EC2 instance. In this case you would set the origin to
AWS::ElasticBeanstalk::Environment
as the environment is the parent of the other two resources. -
-
parent_id
– A subsegment ID you specify if the request originated from an instrumented application. The X-Ray SDK adds the parent subsegment ID to the tracing header for downstream HTTP calls. In the case of nested subsegments, a subsegment can have a segment or a subsegment as its parent. -
http
– http objects with information about the original HTTP request. -
aws
– aws object with information about the AWS resource on which your application served the request. -
error
,throttle
,fault
, andcause
– error fields that indicate an error occurred and that include information about the exception that caused the error. -
annotations
– annotations object with key-value pairs that you want X-Ray to index for search. -
metadata
– metadata object with any additional data that you want to store in the segment. -
subsegments
– array of subsegment objects.
You can create subsegments to record calls to AWS services and resources that you make with the AWS SDK, calls to internal or external HTTP web APIs, or SQL database queries. You can also create subsegments to debug or annotate blocks of code in your application. Subsegments can contain other subsegments, so a custom subsegment that records metadata about an internal function call can contain other custom subsegments and subsegments for downstream calls.
A subsegment records a downstream call from the point of view of the service that calls it. X-Ray uses subsegments to identify downstream services that don't send segments and create entries for them on the service graph.
A subsegment can be embedded in a full segment document or sent independently. Send subsegments separately to asynchronously trace downstream calls for long-running requests, or to avoid exceeding the maximum segment document size.
Example Segment with embedded subsegment
An independent subsegment has a type
of subsegment
and a
parent_id
that identifies the parent segment.
{
"trace_id" : "1-5759e988-bd862e3fe1be46a994272793",
"id" : "defdfd9912dc5a56",
"start_time" : 1461096053.37518,
"end_time" : 1461096053.4042,
"name" : "www.example.com",
"http" : {
"request" : {
"url" : "https://www.example.com/health",
"method" : "GET",
"user_agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7",
"client_ip" : "11.0.3.111"
},
"response" : {
"status" : 200,
"content_length" : 86
}
},
"subsegments" : [
{
"id" : "53995c3f42cd8ad8",
"name" : "api.example.com",
"start_time" : 1461096053.37769,
"end_time" : 1461096053.40379,
"namespace" : "remote",
"http" : {
"request" : {
"url" : "https://api.example.com/health",
"method" : "POST",
"traced" : true
},
"response" : {
"status" : 200,
"content_length" : 861
}
}
}
]
}
For long-running requests, you can send an in-progress segment to notify X-Ray that the request was received, and then send subsegments separately to trace them before completing the original request.
Example In-progress segment
{
"name" : "example.com",
"id" : "70de5b6f19ff9a0b",
"start_time" : 1.478293361271E9,
"trace_id" : "1-581cf771-a006649127e371903a2de979",
"in_progress": true
}
Example Independent subsegment
An independent subsegment has a type
of subsegment
, a
trace_id
, and a parent_id
that identifies the parent
segment.
{
"name" : "api.example.com",
"id" : "53995c3f42cd8ad8",
"start_time" : 1.478293361271E9,
"end_time" : 1.478293361449E9,
"type" : "subsegment",
"trace_id" : "1-581cf771-a006649127e371903a2de979"
"parent_id" : "defdfd9912dc5a56"
,
"namespace" : "remote",
"http" : {
"request" : {
"url" : "https://api.example.com/health",
"method" : "POST",
"traced" : true
},
"response" : {
"status" : 200,
"content_length" : 861
}
}
}
When the request is complete, close the segment by resending it with an
end_time
. The complete segment overwrites the in-progress segment.
You can also send subsegments separately for completed requests that triggered
asynchronous workflows. For example, a web API may return a OK 200
response
immediately prior to starting the work that the user requested. You can send a full segment to
X-Ray as soon as the response is sent, followed by subsegments for work completed later. As
with segments, you can also send a subsegment fragment to record that the subsegment has
started, and then overwrite it with a full subsegment once the downstream call is
complete.
The following fields are required, or are conditionally required, for subsegments.
Note
Values are strings up to 250 characters unless noted otherwise.
Required Subsegment Fields
-
id
– A 64-bit identifier for the subsegment, unique among segments in the same trace, in 16 hexadecimal digits. -
name
– The logical name of the subsegment. For downstream calls, name the subsegment after the resource or service called. For custom subsegments, name the subsegment after the code that it instruments (e.g., a function name). -
start_time
– number that is the time the subsegment was created, in floating point seconds in epoch time, accurate to milliseconds. For example,1480615200.010
or1.480615200010E9
. -
end_time
– number that is the time the subsegment was closed. For example,1480615200.090
or1.480615200090E9
. Specify anend_time
orin_progress
. -
in_progress
– boolean that is set totrue
instead of specifying anend_time
to record that a subsegment is started, but is not complete. Only send one complete subsegment, and one or zero in-progress subsegments, per downstream request. -
trace_id
– Trace ID of the subsegment's parent segment. Required only if sending a subsegment separately.X-Ray trace ID format
An X-Ray
trace_id
consists of three numbers separated by hyphens. For example,1-58406520-a006649127e371903a2de979
. This includes:-
The version number, which is
1
. -
The time of the original request in Unix epoch time using 8 hexadecimal digits.
For example, 10:00AM December 1st, 2016 PST in epoch time is
1480615200
seconds or58406520
in hexadecimal digits. -
A globally unique 96-bit identifier for the trace in 24 hexadecimal digits.
Note
X-Ray now supports trace IDs that are created using OpenTelemetry and any other framework that conforms with the W3C Trace Context specification
. A W3C trace ID must be formatted in X-Ray trace ID format when sending to X-Ray. For example, W3C trace ID 4efaaf4d1e8720b39541901950019ee5
should be formatted as1-4efaaf4d-1e8720b39541901950019ee5
when sending to X-Ray. X-Ray trace IDs include the original request time stamp in Unix epoch time, but this isn't required when sending W3C trace IDs in X-Ray format. -
-
parent_id
– Segment ID of the subsegment's parent segment. Required only if sending a subsegment separately. In the case of nested subsegments, a subsegment can have a segment or a subsegment as its parent. -
type
–subsegment
. Required only if sending a subsegment separately.
The following fields are optional for subsegments.
Optional Subsegment Fields
-
namespace
–aws
for AWS SDK calls;remote
for other downstream calls. -
http
– http object with information about an outgoing HTTP call. -
aws
– aws object with information about the downstream AWS resource that your application called. -
error
,throttle
,fault
, andcause
– error fields that indicate an error occurred and that include information about the exception that caused the error. -
annotations
– annotations object with key-value pairs that you want X-Ray to index for search. -
metadata
– metadata object with any additional data that you want to store in the segment. -
subsegments
– array of subsegment objects. -
precursor_ids
– array of subsegment IDs that identifies subsegments with the same parent that completed prior to this subsegment.
Use an HTTP block to record details about an HTTP request that your application served (in a segment) or that your application made to a downstream HTTP API (in a subsegment). Most of the fields in this object map to information found in an HTTP request and response.
http
All fields are optional.
-
request
– Information about a request.-
method
– The request method. For example,GET
. -
url
– The full URL of the request, compiled from the protocol, hostname, and path of the request. -
user_agent
– The user agent string from the requester's client. -
client_ip
– The IP address of the requester. Can be retrieved from the IP packet'sSource Address
or, for forwarded requests, from anX-Forwarded-For
header. -
x_forwarded_for
– (segments only) boolean indicating that theclient_ip
was read from anX-Forwarded-For
header and is not reliable as it could have been forged. -
traced
– (subsegments only) boolean indicating that the downstream call is to another traced service. If this field is set totrue
, X-Ray considers the trace to be broken until the downstream service uploads a segment with aparent_id
that matches theid
of the subsegment that contains this block.
-
-
response
– Information about a response.-
status
– integer indicating the HTTP status of the response. -
content_length
– integer indicating the length of the response body in bytes.
-
When you instrument a call to a downstream web api, record a subsegment with information about the HTTP request and response. X-Ray uses the subsegment to generate an inferred segment for the remote API.
Example Segment for HTTP call served by an application running on Amazon EC2
{
"id": "6b55dcc497934f1a",
"start_time": 1484789387.126,
"end_time": 1484789387.535,
"trace_id": "1-5880168b-fd5158284b67678a3bb5a78c",
"name": "www.example.com",
"origin": "AWS::EC2::Instance",
"aws": {
"ec2": {
"availability_zone": "us-west-2c",
"instance_id": "i-0b5a4678fc325bg98"
},
"xray": {
"sdk_version": "2.11.0 for Java"
},
},
"http": {
"request": {
"method": "POST",
"client_ip": "78.255.233.48",
"url": "http://www.example.com/api/user",
"user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0",
"x_forwarded_for": true
},
"response": {
"status": 200
}
}
Example Subsegment for a downstream HTTP call
{
"id": "004f72be19cddc2a",
"start_time": 1484786387.131,
"end_time": 1484786387.501,
"name": "names.example.com",
"namespace": "remote",
"http": {
"request": {
"method": "GET",
"url": "https://names.example.com/"
},
"response": {
"content_length": -1,
"status": 200
}
}
}
Example Inferred segment for a downstream HTTP call
{
"id": "168416dc2ea97781",
"name": "names.example.com",
"trace_id": "1-62be1272-1b71c4274f39f122afa64eab",
"start_time": 1484786387.131,
"end_time": 1484786387.501,
"parent_id": "004f72be19cddc2a",
"http": {
"request": {
"method": "GET",
"url": "https://names.example.com/"
},
"response": {
"content_length": -1,
"status": 200
}
},
"inferred": true
}
Segments and subsegments can include an annotations
object containing one or
more fields that X-Ray indexes for use with filter expressions. Fields can have string,
number, or Boolean values (no objects or arrays). X-Ray indexes up to 50
annotations per trace.
Example Segment for HTTP call with annotations
{
"id": "6b55dcc497932f1a",
"start_time": 1484789187.126,
"end_time": 1484789187.535,
"trace_id": "1-5880168b-fd515828bs07678a3bb5a78c",
"name": "www.example.com",
"origin": "AWS::EC2::Instance",
"aws": {
"ec2": {
"availability_zone": "us-west-2c",
"instance_id": "i-0b5a4678fc325bg98"
},
"xray": {
"sdk_version": "2.11.0 for Java"
},
},
"annotations": {
"customer_category" : 124,
"zip_code" : 98101,
"country" : "United States",
"internal" : false
},
"http": {
"request": {
"method": "POST",
"client_ip": "78.255.233.48",
"url": "http://www.example.com/api/user",
"user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0",
"x_forwarded_for": true
},
"response": {
"status": 200
}
}
Keys must be alphanumeric in order to work with filters. Underscore is allowed. Other symbols and whitespace are not allowed.
Segments and subsegments can include a metadata
object containing one or more
fields with values of any type, including objects and arrays. X-Ray does not index metadata,
and values can be any size, as long as the segment document doesn't exceed the maximum size
(64 kB). You can view metadata in the full segment document returned by the
BatchGetTraces
API. Field keys (debug
in the following example) starting
with AWS.
are reserved for use by AWS-provided SDKs and clients.
Example Custom subsegment with metadata
{
"id": "0e58d2918e9038e8",
"start_time": 1484789387.502,
"end_time": 1484789387.534,
"name": "## UserModel.saveUser",
"metadata": {
"debug": {
"test": "Metadata string from UserModel.saveUser"
}
},
"subsegments": [
{
"id": "0f910026178b71eb",
"start_time": 1484789387.502,
"end_time": 1484789387.534,
"name": "DynamoDB",
"namespace": "aws",
"http": {
"response": {
"content_length": 58,
"status": 200
}
},
"aws": {
"table_name": "scorekeep-user",
"operation": "UpdateItem",
"request_id": "3AIENM5J4ELQ3SPODHKBIRVIC3VV4KQNSO5AEMVJF66Q9ASUAAJG",
"resource_names": [
"scorekeep-user"
]
}
}
]
}
For segments, the aws
object contains information about the resource on which
your application is running. Multiple fields can apply to a single resource. For example, an
application running in a multicontainer Docker environment on Elastic Beanstalk could have information
about the Amazon EC2 instance, the Amazon ECS container running on the instance, and the Elastic Beanstalk
environment itself.
aws
(Segments)
All fields are optional.
-
account_id
– If your application sends segments to a different AWS account, record the ID of the account running your application. -
cloudwatch_logs
– Array of objects that describe a single CloudWatch log group.-
log_group
– The CloudWatch Log Group name. -
arn
– The CloudWatch Log Group ARN.
-
-
ec2
– Information about an Amazon EC2 instance.-
instance_id
– The instance ID of the EC2 instance. -
instance_size
– The type of EC2 instance. -
ami_id
– The Amazon Machine Image ID. -
availability_zone
– The Availability Zone in which the instance is running.
-
-
ecs
– Information about an Amazon ECS container.-
container
– The hostname of your container. -
container_id
– The full container ID of your container. -
container_arn
– The ARN of your container instance.
-
-
eks
– Information about an Amazon EKS cluster.-
pod
– The hostname of your EKS pod. -
cluster_name
– The EKS cluster name. -
container_id
– The full container ID of your container.
-
-
elastic_beanstalk
– Information about an Elastic Beanstalk environment. You can find this information in a file named/var/elasticbeanstalk/xray/environment.conf
on the latest Elastic Beanstalk platforms.-
environment_name
– The name of the environment. -
version_label
– The name of the application version that is currently deployed to the instance that served the request. -
deployment_id
– number indicating the ID of the last successful deployment to the instance that served the request.
-
-
xray
– Metadata about the type and version of instrumentation used.-
auto_instrumentation
– Boolean indicating whether auto-instrumentation was used (for example, the Java Agent). -
sdk_version
– The version of SDK or agent being used. -
sdk
– The type of SDK.
-
Example AWS block with plugins
"aws":{
"elastic_beanstalk":{
"version_label":"app-5a56-170119_190650-stage-170119_190650",
"deployment_id":32,
"environment_name":"scorekeep"
},
"ec2":{
"availability_zone":"us-west-2c",
"instance_id":"i-075ad396f12bc325a",
"ami_id":
},
"cloudwatch_logs":[
{
"log_group":"my-cw-log-group",
"arn":"arn:aws:logs:us-west-2:012345678912:log-group:my-cw-log-group"
}
],
"xray":{
"auto_instrumentation":false,
"sdk":"X-Ray for Java",
"sdk_version":"2.8.0"
}
}
For subsegments, record information about the AWS services and resources that your application accesses. X-Ray uses this information to create inferred segments that represent the downstream services in your service map.
aws
(Subsegments)
All fields are optional.
-
operation
– The name of the API action invoked against an AWS service or resource. -
account_id
– If your application accesses resources in a different account, or sends segments to a different account, record the ID of the account that owns the AWS resource that your application accessed. -
region
– If the resource is in a region different from your application, record the region. For example,us-west-2
. -
request_id
– Unique identifier for the request. -
queue_url
– For operations on an Amazon SQS queue, the queue's URL. -
table_name
– For operations on a DynamoDB table, the name of the table.
Example Subsegment for a call to DynamoDB to save an item
{
"id": "24756640c0d0978a",
"start_time": 1.480305974194E9,
"end_time": 1.4803059742E9,
"name": "DynamoDB",
"namespace": "aws",
"http": {
"response": {
"content_length": 60,
"status": 200
}
},
"aws": {
"table_name": "scorekeep-user",
"operation": "UpdateItem",
"request_id": "UBQNSO5AEM8T4FDA4RQDEB94OVTDRVV4K4HIRGVJF66Q9ASUAAJG",
}
}
When an error occurs, you can record details about the error and exceptions that it generated. Record errors in segments when your application returns an error to the user, and in subsegments when a downstream call returns an error.
error types
Set one or more of the following fields to true
to indicate that an error
occurred. Multiple types can apply if errors compound. For example, a 429 Too Many
Requests
error from a downstream call may cause your application to return
500 Internal Server Error
, in which case all three types would apply.
-
error
– boolean indicating that a client error occurred (response status code was 4XX Client Error). -
throttle
– boolean indicating that a request was throttled (response status code was 429 Too Many Requests). -
fault
– boolean indicating that a server error occurred (response status code was 5XX Server Error).
Indicate the cause of the error by including a cause object in the segment or subsegment.
cause
A cause can be either a 16 character exception ID or an object with the following fields:
-
working_directory
– The full path of the working directory when the exception occurred. -
paths
– The array of paths to libraries or modules in use when the exception occurred. -
exceptions
– The array of exception objects.
Include detailed information about the error in one or more exception objects.
exception
All fields are optional.
-
id
– A 64-bit identifier for the exception, unique among segments in the same trace, in 16 hexadecimal digits. -
message
– The exception message. -
type
– The exception type. -
remote
– boolean indicating that the exception was caused by an error returned by a downstream service. -
truncated
– integer indicating the number of stack frames that are omitted from thestack
. -
skipped
– integer indicating the number of exceptions that were skipped between this exception and its child, that is, the exception that it caused. -
cause
– Exception ID of the exception's parent, that is, the exception that caused this exception. -
stack
– array of stackFrame objects.
If available, record information about the call stack in stackFrame objects.
stackFrame
All fields are optional.
-
path
– The relative path to the file. -
line
– The line in the file. -
label
– The function or method name.
You can create subsegments for queries that your application makes to an SQL database.
sql
All fields are optional.
-
connection_string
– For SQL Server or other database connections that don't use URL connection strings, record the connection string, excluding passwords. -
url
– For a database connection that uses a URL connection string, record the URL, excluding passwords. -
sanitized_query
– The database query, with any user provided values removed or replaced by a placeholder. -
database_type
– The name of the database engine. -
database_version
– The version number of the database engine. -
driver_version
– The name and version number of the database engine driver that your application uses. -
user
– The database username. -
preparation
–call
if the query used aPreparedCall
;statement
if the query used aPreparedStatement
.
Example Subsegment with an SQL Query
{
"id": "3fd8634e78ca9560",
"start_time": 1484872218.696,
"end_time": 1484872218.697,
"name": "ebdb@aawijb5u25wdoy.cpamxznpdoq8.us-west-2.rds.amazonaws.com",
"namespace": "remote",
"sql" : {
"url": "jdbc:postgresql://aawijb5u25wdoy.cpamxznpdoq8.us-west-2.rds.amazonaws.com:5432/ebdb",
"preparation": "statement",
"database_type": "PostgreSQL",
"database_version": "9.5.4",
"driver_version": "PostgreSQL 9.4.1211.jre7",
"user" : "dbuser",
"sanitized_query" : "SELECT * FROM customers WHERE customer_id=?;"
}
}