# Transaction Search
Transaction Search is an interactive analytics experience you can use to get complete visibility of your application transaction spans. Spans are the fundamental units of operation in a distributed trace and represent specific actions or tasks in an application or system. Every span records details about a particular segment of the transaction. These details include start and end times, duration, and associated metadata, which can include business attributes like customer IDs and order IDs. Spans are arranged in a parent-child hierarchy. This hierarchy forms a complete trace, mapping the flow of a transaction across different components or services.
![\[View of the visual editor for spans\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/transactionsearch1.png)
**Topics**
+ [
## Benefits
](#w2aac28c21b9)
+ [
## How it works
](#w2aac28c21c11)
+ [
## Pricing
](#w2aac28c21c13)
+ [
# Enable transaction search
](Enable-TransactionSearch.md)
+ [
# Spans
](CloudWatch-Transaction-Search-ingesting-span-log-groups.md)
+ [
# Adding custom attributes
](CloudWatch-Transaction-Search-add-custom-attributes.md)
+ [
# Troubleshooting application issues
](CloudWatch-Transaction-Search-troubleshooting.md)
## Benefits
The following are benefits of using Transaction Search:
**Capture all spans**
Ingest 100 percent of spans as structured logs in CloudWatch to get complete visibility. This prevents broken traces and provides you the ability to view large traces containing up to 10,000 spans for detailed insights.
**Index spans as trace summaries**
Index a percentage of spans as trace summaries in X-Ray to unlock end to end trace search and analytics.
**Investigate transaction issues with free form analytics**
Search all span attributes in the visual editor to identify the cause of issues in application transactions. This helps you answer questions about application performance and the impact end-users make based on their application transactions.
**Send spans to the OpenTelemetry endpoint**
Send spans to the OpenTelemetry endpoint for X-Ray traces. These spans are stored in [the semantic convention format with W3C trace IDs](https://opentelemetry.io/docs/specs/semconv/general/trace/).
**Note**
X-Ray traces automatically convert to the semantic convention format before they're stored in a log group called `aws/spans`. For more information, see [The span log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-ingesting-span-log-groups.html).
**Use CloudWatch Logs with spans**
Use metric filters to extract custom metrics, subscription filters to forward data, and data masking to protect personally identifiable information.
**Troubleshoot application issues**
Access application dashboards, metrics, and topology when you enable Application Signals for all spans sent to CloudWatch.
## How it works
When you enable Transaction Search, you unlock multiple capabilities, including features in Application Signals and CloudWatch Logs.
![\[Overview of how Transaction Search works with other services\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/howitworks.png)
If you send traces to X-Ray, you can [get started by enabling Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html) in the console or with the API. If you don't send traces to X-Ray, you can use the [CloudWatch Application Signals](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Signals-Enable.html) that provides pre-packaged OpenTelemetry setup with AWS Distro fro OpenTelemetry (ADOT), CloudWatch Agent, or use OpenTelemetry directly.
When you enable Transaction Search, spans sent to X-Ray are ingested in a log group called `aws/spans`. CloudWatch uses these spans to generate a curated application performance monitoring (APM) experience in CloudWatch Application Signals. This provides you the ability to search and analyze spans, as well as use CloudWatch Logs capabilities like anomaly and pattern detection. You can even use custom metric extraction . CloudWatch Application Signals provides you with a unified, application-centric view of your applications, services, and dependencies. It also helps you monitor and triage application health.
You can also explore spans using the interactive search and analytics experience in CloudWatch to answer any questions related to application performance or end-user impact with Transaction Search. Detect the impact on end users, find transactions in context of those issues using relevant attributes, such as customer name or order number. You can correlate transactions to business events, such as failed payments, and dive into interactions between application components to establish a root cause. With CloudWatch, you get complete application transaction coverage with correlated insights, helping you to accelerate mean time to resolution.
## Pricing
For information about pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/).
# Enable transaction search
You can enable [Transaction Search](CloudWatch-Transaction-Search.md) through the console or by using an API. Transaction search is configured for the entire account and switches all spans ingestion through X-Ray into cost effective collection mode using [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). By default you will also index 1% of ingested spans for free as trace summary for analysis, which is typically sufficient given you already have full end-to-end trace visibility on all ingested spans through Transaction Search.
## Prerequisites
Before you can enable Transaction Search, you must create a role with the following permissions.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "TransactionSearchXRayPermissions",
"Effect": "Allow",
"Action": [
"xray:GetTraceSegmentDestination",
"xray:UpdateTraceSegmentDestination",
"xray:GetIndexingRules",
"xray:UpdateIndexingRule"
],
"Resource": "*"
},
{
"Sid": "TransactionSearchLogGroupPermissions",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutRetentionPolicy"
],
"Resource": [
"arn:aws:logs:*:*:log-group:/aws/application-signals/data:*",
"arn:aws:logs:*:*:log-group:aws/spans:*"
]
},
{
"Sid": "TransactionSearchLogsPermissions",
"Effect": "Allow",
"Action": [
"logs:PutResourcePolicy",
"logs:DescribeResourcePolicies"
],
"Resource": "*"
},
{
"Sid": "TransactionSearchApplicationSignalsPermissions",
"Effect": "Allow",
"Action": [
"application-signals:StartDiscovery"
],
"Resource": "*"
},
{
"Sid": "CloudWatchApplicationSignalsCreateServiceLinkedRolePermissions",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/application-signals.cloudwatch.amazonaws.com/AWSServiceRoleForCloudWatchApplicationSignals",
"Condition": {
"StringLike": {
"iam:AWSServiceName": "application-signals.cloudwatch.amazonaws.com"
}
}
},
{
"Sid": "CloudWatchApplicationSignalsGetRolePermissions",
"Effect": "Allow",
"Action": "iam:GetRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/application-signals.cloudwatch.amazonaws.com/AWSServiceRoleForCloudWatchApplicationSignals"
},
{
"Sid": "CloudWatchApplicationSignalsCloudTrailPermissions",
"Effect": "Allow",
"Action": [
"cloudtrail:CreateServiceLinkedChannel"
],
"Resource": "arn:aws:cloudtrail:*:*:channel/aws-service-channel/application-signals/*"
}
]
}
```
------
**Note**
To use Transaction Search and other CloudWatch features, add the [CloudWatchReadOnlyAccess policy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchReadOnlyAccess.html) to your role. For information about how to create a role, see [IAM role creation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html).
## Enabling Transaction Search in the console
The following procedure describes how to enable Transaction Search in the console.
**To enable Transaction Search in the CloudWatch console**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. From the navigation pane, under **Application Signals**, choose **Transaction Search**.
1. Choose **Enable Transaction Search**.
1. Select the box to ingest spans as structured logs, and enter a percentage of spans to be indexed. You can index spans at 1% for free and change the percentage later based on your requirements.
## Enabling Transaction Search using an API
The following procedure describes how to enable Transaction Search using an API.
### Step 1. Create a policy that grants access to ingest spans in CloudWatch Logs
When using the AWS CLI or SDK to enable Transaction Search, you must configure permissions using a resource-based policy with [https://docs.aws.amazon.com/xray/latest/api/API_PutResourcePolicy.html](https://docs.aws.amazon.com/xray/latest/api/API_PutResourcePolicy.html).
**Example policy**
The following example policy allows X-Ray to send traces to CloudWatch Logs
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "TransactionSearchXRayAccess",
"Effect": "Allow",
"Principal": {
"Service": "xray.amazonaws.com"
},
"Action": "logs:PutLogEvents",
"Resource": [
"arn:aws:logs:us-east-1:123456789012:log-group:aws/spans:*",
"arn:aws:logs:us-east-1:123456789012:log-group:/aws/application-signals/data:*"
],
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:xray:us-east-1:123456789012:*"
},
"StringEquals": {
"aws:SourceAccount": "123456789012"
}
}
}
]
}
```
------
**Example command**
The following example shows how to format your AWS CLI command with `PutResourcePolicy`.
```
aws logs put-resource-policy --policy-name MyResourcePolicy --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Sid": "TransactionSearchXRayAccess", "Effect": "Allow", "Principal": { "Service": "xray.amazonaws.com" }, "Action": "logs:PutLogEvents", "Resource": [ "arn:partition:logs:region:account-id:log-group:aws/spans:*", "arn:partition:logs:region:account-id:log-group:/aws/application-signals/data:*" ], "Condition": { "ArnLike": { "aws:SourceArn": "arn:partition:xray:region:account-id:*" }, "StringEquals": { "aws:SourceAccount": "account-id" } } } ]}'
```
### Step 2. Configure the destination of trace segments
Configure the ingestion of spans with [https://docs.aws.amazon.com/xray/latest/api/API_UpdateTraceSegmentDestination.html](https://docs.aws.amazon.com/xray/latest/api/API_UpdateTraceSegmentDestination.html).
**Example command**
The following example shows how to format your AWS CLI command with `UpdateTraceSegmentDestination`.
```
aws xray update-trace-segment-destination --destination CloudWatchLogs
```
### Step 3. Configure the amount of spans to index
Configure your desired sampling percentage with [https://docs.aws.amazon.com/xray/latest/api/API_UpdateIndexingRule.html](https://docs.aws.amazon.com/xray/latest/api/API_UpdateIndexingRule.html)
**Example command**
The following example shows how to format your AWS CLI command with `UpdateIndexingRule`.
```
aws xray update-indexing-rule --name "Default" --rule '{"Probabilistic": {"DesiredSamplingPercentage": number}}'
```
**Note**
After you enable Transaction Search, it can take ten minutes for spans to become available for search and analysis.
### Step 4. Verify spans are available for search and analysis
To verify spans are available for search and analysis, use [https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSegmentDestination.html](https://docs.aws.amazon.com/xray/latest/api/API_GetTraceSegmentDestination.html).
**Example commands**
The following example shows how to format your AWS CLI command with `GetTraceSegmentDestination`.
```
aws xray get-trace-segment-destination
```
**Example response**
The following example shows the response you can expect when Transaction Search is active.
```
{
"Destination": "CloudWatchLogs",
"Status": "ACTIVE"
}
```
# Using Transaction Search with CloudFormation
You can use CloudFormation to enable and configure X-Ray Transaction Search.
**Note**
To create a CloudFormation stack, see [Creating your first stack ](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/gettingstarted.walkthrough.html).
## Prerequisites
+ You must have access to an AWS account with an IAM user or role that has permissions to use Amazon EC2, Amazon S3, CloudFormation, or have administrative user access.
+ You must have a Virtual Private Cloud (VPC) that has access to the internet. To keep things simple, you can use the default VPC that comes with your account. The default VPC and default subnets are sufficient for this configuration.
+ Make sure Transaction Search is disabled before you enable using AWS CDK or CloudFormation.
## Enabling Transaction Search
To enable Transaction Search using CloudFormation, you need to create the following two resources.
+ `AWS::Logs::ResourcePolicy`
+ `AWS::XRay::TransactionSearchConfig`
1. **Create AWS::Logs::ResourcePolicy** – Create a resource policy that allows X-Ray to send traces to CloudWatch Logs
**YAML**
```
Resources:
LogsResourcePolicy:
Type: AWS::Logs::ResourcePolicy
Properties:
PolicyName: TransactionSearchAccess
PolicyDocument: !Sub >
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "TransactionSearchXRayAccess",
"Effect": "Allow",
"Principal": {
"Service": "xray.amazonaws.com"
},
"Action": "logs:PutLogEvents",
"Resource": [
"arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:aws/spans:*",
"arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/application-signals/data:*"
],
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:${AWS::Partition}:xray:${AWS::Region}:${AWS::AccountId}:*"
},
"StringEquals": {
"aws:SourceAccount": "${AWS::AccountId}"
}
}
}
]
}
```
**JSON**
```
{
"Resources": {
"LogsResourcePolicy": {
"Type": "AWS::Logs::ResourcePolicy",
"Properties": {
"PolicyName": "TransactionSearchAccess",
"PolicyDocument": {
"Fn::Sub": "{\n \"Version\": \"2012-10-17\", \n \"Statement\": [\n {\n \"Sid\": \"TransactionSearchXRayAccess\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Service\": \"xray.amazonaws.com\"\n },\n \"Action\": \"logs:PutLogEvents\",\n \"Resource\": [\n \"arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:aws/spans:*\",\n \"arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/application-signals/data:*\"\n ],\n \"Condition\": {\n \"ArnLike\": {\n \"aws:SourceArn\": \"arn:${AWS::Partition}:xray:${AWS::Region}:${AWS::AccountId}:*\"\n },\n \"StringEquals\": {\n \"aws:SourceAccount\": \"${AWS::AccountId}\"\n }\n }\n }\n ]\n}"
}
}
}
}
}
```
1. ** Create and Configure AWS::XRay::TransactionSearchConfig** – Create the `TransactionSearchConfig` resource to enable Transaction Search.
**YAML**
```
Resources:
XRayTransactionSearchConfig:
Type: AWS::XRay::TransactionSearchConfig
```
**JSON**
```
{
"Resources": {
"XRayTransactionSearchConfig": {
"Type": "AWS::XRay::TransactionSearchConfig"
}
}
}
```
1. (Optional) You can set the `IndexingPercentage` property to control the percentage of spans that will be indexed.
**YAML**
```
Resources:
XRayTransactionSearchConfig:
Type: AWS::XRay::TransactionSearchConfig
Properties:
IndexingPercentage: 50
```
**JSON**
```
{
"Resources": {
"XRayTransactionSearchConfig": {
"Type": "AWS::XRay::TransactionSearchConfig",
"Properties": {
"IndexingPercentage": 20
}
}
}
}
```
The IndexingPercentage value can be set between 0 and 100.
## Template examples
The following example includes both the resource policy and the TransactionSearchConfig.
**YAML**
```
Resources:
LogsResourcePolicy:
Type: AWS::Logs::ResourcePolicy
Properties:
PolicyName: TransactionSearchAccess
PolicyDocument: !Sub >
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "TransactionSearchXRayAccess",
"Effect": "Allow",
"Principal": {
"Service": "xray.amazonaws.com"
},
"Action": "logs:PutLogEvents",
"Resource": [
"arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:aws/spans:*",
"arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/application-signals/data:*"
],
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:${AWS::Partition}:xray:${AWS::Region}:${AWS::AccountId}:*"
},
"StringEquals": {
"aws:SourceAccount": "${AWS::AccountId}"
}
}
}
]
}
XRayTransactionSearchConfig:
Type: AWS::XRay::TransactionSearchConfig
Properties:
IndexingPercentage: 10
```
**JSON**
```
{
"Resources": {
"LogsResourcePolicy": {
"Type": "AWS::Logs::ResourcePolicy",
"Properties": {
"PolicyName": "TransactionSearchAccess",
"PolicyDocument": {
"Fn::Sub": "{\n \"Version\": \"2012-10-17\", \n \"Statement\": [\n {\n \"Sid\": \"TransactionSearchXRayAccess\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Service\": \"xray.amazonaws.com\"\n },\n \"Action\": \"logs:PutLogEvents\",\n \"Resource\": [\n \"arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:aws/spans:*\",\n \"arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/application-signals/data:*\"\n ],\n \"Condition\": {\n \"ArnLike\": {\n \"aws:SourceArn\": \"arn:${AWS::Partition}:xray:${AWS::Region}:${AWS::AccountId}:*\"\n },\n \"StringEquals\": {\n \"aws:SourceAccount\": \"${AWS::AccountId}\"\n }\n }\n }\n ]\n}"
}
}
},
"XRayTransactionSearchConfig": {
"Type": "AWS::XRay::TransactionSearchConfig",
"Properties": {
"IndexingPercentage": 20
}
}
}
}
```
Here is an example using AWS CDK in TypeScript.
**CDK**
```
import * as cdk from '@aws-cdk/core';
import * as logs from '@aws-cdk/aws-logs';
import * as xray from '@aws-cdk/aws-xray';
export class XRayTransactionSearchStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create the resource policy
const transactionSearchAccess = new logs.CfnResourcePolicy(this, 'XRayLogResourcePolicy', {
policyName: 'TransactionSearchAccess',
policyDocument: JSON.stringify({
Version: '2012-10-17',
Statement: [
{
Sid: 'TransactionSearchXRayAccess',
Effect: 'Allow',
Principal: {
Service: 'xray.amazonaws.com',
},
Action: 'logs:PutLogEvents',
Resource: [
`arn:${this.partition}:logs:${this.region}:${this.account}:log-group:aws/spans:*`,
`arn:${this.partition}:logs:${this.region}:${this.account}:log-group:/aws/application-signals/data:*`,
],
Condition: {
ArnLike: {
'aws:SourceArn': `arn:${this.partition}:xray:${this.region}:${this.account}:*`,
},
StringEquals: {
'aws:SourceAccount': this.account,
},
},
},
],
}),
});
// Create the TransactionSearchConfig with dependency
const transactionSearchConfig = new xray.CfnTransactionSearchConfig(this, 'XRayTransactionSearchConfig', {
indexingPercentage: 10,
});
// Add the dependency to ensure Resource Policy is created first
transactionSearchConfig.addDependsOn(transactionSearchAccess);
}
}
```
## Verifying the configuration
After deploying your CloudFormation stack, you can verify the configuration using the AWS CLI.
**aws xray get-trace-segment-destination**
A successful configuration will return the following.
```
{
"Destination": "CloudWatchLogs",
"Status": "ACTIVE"
}
```
# Spans
Spans sent to X-Ray are ingested and managed in a log group called `aws/spans`. This topic describes which CloudWatch Logs features are available for transaction spans.
**Available features**
The following CloudWatch Logs features are available for transaction spans.
+ [Metric filters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html) – Use metric filters to extract custom metrics from spans.
+ [Subscriptions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html) – Use subscriptions to access a real-time feed of span events from CloudWatch Logs.
+ [Log anomaly detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/LogsAnomalyDetection.html) – Use log anomaly detection to establish a baseline for spans sent to the `aws/spans` log group.
+ [Contributor Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContributorInsights.html) – Use Contributor Insights to analyze span data and create a time series displaying contributor data.
**Unsupported features**
The following are features not supported for transaction spans.
+ Spans cannot be sent to CloudWatch Logs with the `PutLogEvents` API.
+ Span data cannot be [enriched or transformed](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch-Logs-Transformation.html).
**Note**
Span ingestion is charged separately from log ingestion. For information about pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/).
# Searching and analyzing spans
Transaction Search provides you with a visual editor to search and analyze all ingested spans using attributes. You can use the visual editor to narrow down transaction spans and create interactive visualizations to troubleshoot issues in your distributed applications. You can also use the CloudWatch Logs Insights query language to analyze your spans. This topic describes how to access and use the visual editor.
## The visual editor
The following procedure describes how to access the visual editor.
**To access the visual editor**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. From the navigation pane, choose **Application Signals**, and then choose **Transaction Search**.
Use span attributes, such as service name, span duration, and span status to narrow down transaction spans quickly. You can access these filters and more on the right side of the visual editor under **Select filters**.
This visual editor suggests a list of attributes in the span. These attributes include attributes added through auto-instrumentation and custom attributes added through custom instrumentation.
![\[Filter spans by attributes\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/filter3.png)
Select a span key, and enter a value to refine span results. You can filter spans using various operations, such as "Equals," "Does Not Equal," and more.
![\[Filter spans with operators\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/filter4.png)
### Query formats
You can run queries in the visual editor using different formats. This section describes each of these formats.
#### List
View spans or span events in a list format, which displays information about each span. Use this type of analysis to analyze individual spans, understand specific transactions, or identify unique patterns in transaction events. Other use cases include the following:
**Use cases**
+ Troubleshoot customer support tickets
+ Locate APIs or dependencies, such as database queries taking longer than 1000 milliseconds to execute
+ Locate spans with errors
The following screenshots show how to troubleshoot a customer support ticket with this type of analysis.
**Example scenario**
In the visual editor, filter on all transaction spans with a particular customer issue. Before you run your query, choose **List** from the **Visualize as** dropdown.
![\[Locate spans with List\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/list1.png)
The results show a list of spans where you can choose a trace ID to get the end-to-end journey for the transaction and determine the root cause of the issue.
![\[List results\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/list2.png)
#### Timeseries
View spans or span events over time. Use this type of analysis to look at trends and spikes in transaction activity. Other use cases include the following:
+ Visualize latency
+ Visualize frequency of spans
+ Visualize performance
The following screenshots show how you can view p99 latency trends for an API with this type of analysis.
**Example scenario**
In the visual editor, filter on the service and API you want to analyze.
![\[Filtering on a service\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/timeseries1.png)
Before you run your query, choose **Time series** from the **Visualize as** dropdown. Choose **P99** for the duration statistic from the **Show span as** dropdown.
![\[Filtering\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/timeseries2.png)
The results show a latency trend for the service, with the x-axis of the graph being time and y-axis being p99 duration.
![\[Locate spans with time series\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/timeseries3.png)
You can choose a point on the chart to view correlated spans and span events.
![\[Time series results\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/timeseries4.png)
#### Group analysis
Aggregate spans or span events based on specific attributes, such as account IDs and status codes, to display statistical metrics. Use this type of analysis to analyze spans in clusters, compare different groups, and uncover trends at the macro level. Other use cases include the following:
**Use cases**
+ Identify top customers impacted by a service outage
+ Identify availability zones with the most errors
+ Identify the top slowest database queries
The following screenshots show how you can view the top customers impacted by a service outage with this type of analysis.
**Example scenario**
In the visual editor, you filter on the service experiencing issues.
![\[Filter by service issue\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/groupanalysis1.png)
Before you run your query, choose **Group Analysis** from the **Visualize as** dropdown. Group your query results by `account.id`, and limit the number of results to 10..
![\[Locate spans by group analysis\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/groupanalysis2.png)
The results show the top 10 customers who experienced the most number of errors.
![\[Group analysis results\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/groupanalysis3.png)
## CloudWatch Logs Insights
You can use [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html) to analyze your spans.
**Example query**
The following query shows the top five slowest database queries.
```
STATS pct(durationNano, 99) as `p99` by attributes.db.statement
| SORT p99 ASC
| LIMIT 5
| DISPLAY p99,attributes.db.statement
```
**Example query**
The following query shows which top five services are throwing errors.
```
FILTER `attributes.http.response.status_code` >= 500
| STATS count(*) as `count` by attributes.aws.local.service as service
| SORT count ASC
| LIMIT 5
| DISPLAY count,service
```
# Ingesting spans for complete visibility
Recording all transaction spans provides comprehensive visibility into application issues. It enables you to debug customer support tickets or troubleshoot rarely occurring p99 API latency spikes, which is crucial when identifying the root cause of issues in customer-facing and mission-critical applications. You can create a cost-effective strategy to start capturing 100% of trace spans in CloudWatch by configuring the head sampling rate and then adjusting a lower span indexing rate.
## Setting up head sampling
Head sampling is a tracing technique capturing requests at the beginning of a trace, which is based on a set rate or condition.
When the head sampling rate is set to 100%, it captures the beginning of every trace without skips, guaranteeing complete visibility into all incoming requests, and that no transaction data is missed.
You can configure head sampling if you're using X-Ray or AWS Distro for OpenTelemetry SDKs or the OpenTelemetry SDK.
**If you're using X-Ray or AWS Distro for OpenTelemetry SDKs**
Navigate to your sampling rules in the console, and set the fixed sampling rate to 100%. This guarantees all trace spans are captured and ingested into CloudWatch logs. For more information, see [Configuring sampling rules](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html#xray-console-config)
**If you're using the OpenTelemetry SDK**
To record 100% of spans and get complete visibility, set your sampling configuration to [https://opentelemetry.io/docs/languages/java/sdk/#sampler](https://opentelemetry.io/docs/languages/java/sdk/#sampler). For more information, see [Language APIs & SDKs]() on the OpenTelemetry website.
### Features unlocked with head sampling
When you enable Transaction Search, all spans collected from your application through head sampling are ingested as structured logs in CloudWatch. This provides you with the following features:
+ The ability to search span attributes and analyze span events in a visual editor.
+ The ability to visualize traces containing up to 10,000 spans.
+ Total support for OpenTelemetry, which includes the ability to embed business events into spans for analysis and use span links to define connections between traces for end-to-end viewing.
+ Access to application dashboards, metrics, and topology with CloudWatch Application Signals enabled for all spans sent to CloudWatch.
**Note**
Because spans are available in a log group called `aws/spans`, you can use CloudWatch Logs features with transaction spans. For more information, see [The span log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-ingesting-span-log-groups.html).
## Setting up span indexing with trace summaries
Trace summaries can help you debug transactions and are beneficial for asynchronous processes. You only need to index a small percentage of spans as trace summaries.
You configure span indexing when you enable Transaction Search in the console or with the API. To enable Transaction Search, see [Getting started with Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html).
### Features unlocked with trace summaries
The key features of X-Ray trace summaries include the following:
+ [Trace summary search](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-traces.html?icmpid=docs_xray_help_panel_traces#xray-console-traces-view) – Search and find traces from trace summaries.
+ [Trace summary analytics](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-analytics.html) – Interpret trace data.
+ [Trace insights](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-insights.html) – Analyze trace data to identify application issues.
# Monitoring spans across accounts
Spans sent to X-Ray are ingested and managed in a log group called `aws/spans`. To monitor spans across multiple accounts, you must [enable Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html) across all source and monitoring accounts and [enable cross-account observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html) for logs and traces. When you enable cross-account observability, you can search up to 10,000 accounts and get visibility into traces across accounts. This feature is provided at no extra cost for the `aws/spans` log group. If you enable cross-account observability for trace summaries, the first trace summary copy is free. For more information about pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/).
# Adding custom attributes
CloudWatch Application Signals utilizes OpenTelemetry to auto-instrument your applications and collect spans from popular libraries in different languages, such as Java, Python, and more.
Auto-instrumentation captures information, such as database queries, HTTP requests, cache accesses, and external service calls, which allows you to troubleshoot application performance issues.
You can add custom instrumentation to enrich spans with business-specific data or other information you wish to capture. This data can be recorded as a custom attribute or a span event, providing insights tailored to your troubleshooting needs.
**Note**
For information about adding custom attributes or span events in a different language, see [Language APIs and SDKS](https://opentelemetry.io/docs/languages/) in the *OpenTelemetry website*.
## Custom attributes
You can add business related attributes or any other attributes to your spans in all languages OpenTelemetry supports. The following is a Java code snippet that adds an order id and customer details to a span.
```
import io.opentelemetry.api.trace.Span;
public class OrderProcessor {
public void processOrder() {
Span span = Span.current();
span.setAttribute("order.id", "123456");
span.setAttribute("customer.name", "John Doe");
span.setAttribute("customer.id", "4343dfdd");
// Your order processing logic here
System.out.println("Order processed with custom attributes");
}
}
```
After these attributes have been added to the span, they become available to search and analyze in [the Transaction Search visual editor](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-search-analyze-spans.html).
## Span events
A span event is typically used to denote a meaningful, singular point in time during a span duration. Exceptions are auto-captured as span events through auto-instrumentation, but you can also add custom business events, such as payment-status or cart-abandonment. For more information, see [Span events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events) on the OpenTelemetry website.
You can embed span events to your spans in all the languages CloudWatch Application Signals and OpenTelemetry support. The following is a Java code snippet of adding a custom event to a span.
```
import io.opentelemetry.api.trace.Span;
public class OrderProcessor {
public void bookOrder() {
Span span = Span.current();
// Add a booking started event
span.addEvent("booking started");
// Add a payment succeeded event or failed event
span.addEvent("booking failed");
}
}
```
### Prerequisites for the CloudWatch agent
When using the CloudWatch agent to emit span events to X-Ray, you must turn on the ``transit_spans_in_otlp_format`` flag in your configuration.
```
{
"traces": {
...
"transit_spans_in_otlp_format": true
...
}
}
```
After you add these events, they become available in the [Transaction Search visual editor](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-search-analyze-spans.html).
## CloudWatch Logs queries
You can query span events in CloudWatch Logs to view advanced insights. The following example query commands show how to analyze exceptions thrown by your application:
```
fields jsonparse(@message) as js
| unnest js.events into event
| filter event.name = "exception"
| display event.attributes.`exception.stacktrace`
```
```
fields jsonparse(@message) as js
| unnest js.events into event
| filter event.name = "exception"
| stats count() by event.attributes.`exception.type`
```
# Troubleshooting application issues
With Application Signals, you can troubleshoot rarely occurring latency spikes in your applications. After you enable Transaction Search and configure a head sampling rate capturing 100% of spans, you get complete visibility into any application issue. The following scenario describes how Application Signals can be used with transaction spans to monitor your services and identify service quality issues.
## Example troubleshooting scenario
This scenario focuses on a pet clinic application composed of several micro-services calling third-party payment APIs. These calls have been intermittently slow, thus impacting revenue.
Jane opens the CloudWatch Application Signals console and notices a customer-service application responsible for registering customers is healthy and not breaching any SLOs.
![\[CloudWatch Application Signals console\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/troubleshooting1.png)
She opens the service to investigate any patterns of rarely occurring failures and notices the registration API experienced intermittent p99 latency spikes.
![\[Intermittent latency spikes\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/troubleshooting2.png)
Jane chooses a datapoint in the latency chart to view correlated spans. She groups the spans by customer ID to view all the customers who are impacted by the latency spikes.
![\[Customers impacted by latency spikes\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/troubleshooting3.png)
Jane selects one of the correlated spans with a fault status, which opens the trace detail page for the selected trace. She scrolls to the segments timeline section to follow the call path, where she notices that calls to the payment gateway have been failing and preventing customers from registering.
![\[Failing call payments\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/troubleshooting4.png)