# Getting started
To get started with OpenTelemetry in CloudWatch, you can use the pre-packaged OpenTelemetry setup that is available with the CloudWatch agent along with the AWS Distro for OpenTelemetry SDKs. This gives you the most integrated monitoring experience in CloudWatch.
**Note**
Make sure Transaction Search is enabled before you use the OTLP Endpoint for traces.
Alternatively, you have the flexibility to use the OpenTelemetry Collector or your own custom OpenTelemetry Collector to directly send telemetry to the OTLP endpoint. You can use the AWS Distro for OpenTelemetry to go collector-less and to send telemetry directly to the OTLP endpoint. Make an informed choice based on the feature support:
| Feature | OpenTelemetry Collector | Custom OpenTelemetry Collector | AWS Distro for OpenTelemetry |
| --- | --- | --- | --- |
| CloudWatch application signals (Application performance metrics, service discovery, and application map) | Yes | Yes | Yes |
| Search and analyze spans and trace summaries | Yes | Yes | Yes |
| Search and analyze logs summaries | Yes | Yes | Yes |
| Application performance monitoring telemetry enrichment with AWS infrastructure attributes that your application is hosted in. | No | Yes | Yes |
| Runtime metrics correlated with your application. For example, JVM metrics | No | Yes | No |
| AWS Support | Data received by AWS | Data received by AWS | Data received by AWS |
| Telemetry supported | Logs, Metrics, Traces | Logs, Traces, Metrics | Metrics, Traces |
**Topics**
+ [
# OpenTelemetry Collector
](CloudWatch-OTLPSimplesetup.md)
+ [
# Build your own custom OpenTelemetry Collector
](CloudWatch-OTLPAdvancedsetup.md)
+ [
# Exporting collector-less telemetry using AWS Distro for OpenTelemetry (ADOT) SDK
](CloudWatch-OTLP-UsingADOT.md)
+ [
# Enabling vended metrics in PromQL
](CloudWatch-OTelEnrichment.md)
# OpenTelemetry Collector
The OpenTelemetry Collector is an open source, vendor-agnostic agent that receives, processes, and exports telemetry data. It acts as a central pipeline between your applications and Amazon CloudWatch, collecting metrics, logs, and traces from multiple sources and sending them to CloudWatch using the OpenTelemetry Protocol (OTLP).
Using the OpenTelemetry Collector with CloudWatch provides the following benefits:
+ Collect telemetry from multiple applications and hosts through a single agent, reducing the number of connections to CloudWatch.
+ Process and filter telemetry before sending it to CloudWatch, including adding or removing attributes, batching data, and sampling traces.
+ Use the same collector configuration across AWS, on-premises, and other cloud environments, providing a consistent telemetry pipeline regardless of where your applications run.
+ Send metrics to CloudWatch with rich labels that are available to query using the Prometheus Query Language (PromQL) in CloudWatch Query Studio.
## Supported receivers
The OpenTelemetry Collector supports a wide range of receivers for ingesting telemetry data. You can use OpenTelemetry receivers such as the OTLP receiver for applications instrumented with OpenTelemetry SDKs, or Prometheus receivers to scrape metrics from existing Prometheus exporters. Common Prometheus receivers used with CloudWatch include:
+ Prometheus receiver, for scraping any Prometheus-compatible endpoint
+ Host Metrics receiver, for collecting system-level metrics from the host
+ Kubernetes Cluster receiver, for collecting cluster-level metrics from the Kubernetes API server
You can configure multiple receivers in a single collector, allowing you to collect both OpenTelemetry and Prometheus metrics and send them to CloudWatch through the same pipeline. For the full list of available receivers, see the OpenTelemetry Collector [repository](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver).
## Getting started
Prerequisite – If using the OTLP endpoint for tracing, ensure that Transaction Search is enabled.
Steps:
1. Download the latest release of the OpenTelemetry Collector distribution. For more information, see the OpenTelemetry Collector [releases](https://github.com/open-telemetry/opentelemetry-collector-releases/releases).
1. Install the OpenTelemetry Collector on your host. The collector runs on any operating system and platform. For more information, see [Install the Collector](https://opentelemetry.io/docs/collector/installation/).
1. Configure AWS credentials on your Amazon EC2 or on-premises host. The collector uses these credentials to authenticate with CloudWatch when sending telemetry data. See below for details.
------
#### [ Setup IAM permissions for Amazon EC2 ]
**Follow the below procedure to attach the `CloudWatchAgentServerPolicy` IAM policy to the IAM role of your Amazon EC2 instance.**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. Choose **Roles** and find and select the role used by your Amazon EC2 instance.
1. Under the **Permissions** tab, choose **Add permissions**, **Attach policies**.
1. Using the search box, search for `CloudWatchAgentServerPolicy` policy.
1. Select the **CloudWatchAgentServerPolicy** policy and choose **Add permissions**.
------
#### [ Setup IAM permissions for on-premise hosts ]
**You can create an IAM user that can be used to provide permissions to your on-premise hosts.**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. Choose **Users**, **Create User**.
1. Under **User details**,for **User name**, enter a name for the new IAM user. This is the sign-in name for AWS that will be used to authenticate your host.
1. Choose **Next**.
1. On the **Set permissions** page, under **Permissions options**, select **Attach policies directly**.
1. From the **Permissions policies** list, select the **CloudWatchAgentServerPolicy** policy to add to your user.
1. Choose **Next**.
1. On the **Review and create** page, ensure that you are satisfied with the user name and that the **CloudWatchAgentServerPolicy** policy is under the **Permissions summary**.
1. Choose **Create user**.
1. **Create and retrieve your AWS access key and secret key** – In the navigation pane in the IAM console, choose **Users** and then select the user name of the user that you created in the previous step.
1. On the user's page, choose the **Security credentials** tab.
1. Under the **Access keys** section, choose **Create access key**.
1. For **Create access key Step 1**, choose **Command Line Interface (CLI)**.
1. For **Create access key Step 2**, optionally enter a tag and then choose **Next**.
1. For **Create access key Step 3,** select **Download .csv file** to save a .csv file with your IAM user's access key and secret access key. You need this information for the next steps.
1. Choose **Done**.
1. Configure your AWS credentials in your on-premises host by entering the following command. Replace *ACCESS\$1KEY\$1ID * and *SECRET\$1ACCESS\$1ID * with your newly generated access key and secret access key from the .csv file that you downloaded in the previous step.
```
$ aws configure
AWS Access Key ID [None]: ACCESS_KEY_ID
AWS Secret Access Key [None]: SECRET_ACCESS_ID
Default region name [None]: MY_REGION
Default output format [None]: json
```
------
1. Setup AWS credentials for your Amazon EKS or Kubernetes clusters. The easiest way to get started with Amazon EKS is to use the EKS OTel Container Insights add-on. If you prefer to use the OpenTelemetry Collector directly, follow the procedure below to set up AWS credentials for your Amazon EKS or Kubernetes clusters to send telemetry to CloudWatch.
------
#### [ Setup IAM permissions for Amazon EKS ]
1. Create an IAM OIDC identity provider for your cluster using the following command.
```
eksctl utils associate-iam-oidc-provider --cluster ${CLUSTER_NAME} --region ${REGION} --approve
```
1. Assign IAM roles to Kubernetes service account for OTel Collector using the following command.
```
eksctl create iamserviceaccount \
--name ${COLLECTOR_SERVICE_ACCOUNT}\
--namespace ${NAMESPACE} \
--cluster ${CLUSTER_NAME} \
--region ${REGION} \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--approve \
--override-existing-serviceaccounts
```
------
#### [ Setup IAM permissions for Kubernetes ]
1. Configure your AWS credentials in your on-premises host by entering the following command. Replace *ACCESS\$1KEY\$1ID* and *SECRET\$1ACCESS\$1ID* with your newly generated access key and secret access key from the .csv file that you downloaded in the previous step. By default, the credential file is saved under */home/user/.aws/credentials.*.
```
aws configure
AWS Access Key ID [None]: ACCESS_KEY_ID
AWS Secret Access Key [None]: SECRET_ACCESS_ID
Default region name [None]: MY_REGION
Default output format [None]: json
```
1. Edit OpenTelemetry Collector resource to add the newly created AWS credentials secret by using the command: `kubectl edit OpenTelemetryCollector otel_collector`
1. Using the file editor, add the AWS credentials into the OpenTelemetryCollector container by adding the following configuration to the top of the deployment. Replace the path */home/user/.aws/credentials* with the location of your local AWS credentials file.
```
spec:
volumeMounts:
- mountPath: /rootfs
volumeMounts:
- name: aws-credentials
mountPath: /root/.aws
readOnly: true
volumes:
- hostPath:
path: /home/user/.aws/credentials
name: aws-credentials
```
------
1. Configure the OTLP exporter in your collector configuration to send telemetry to the CloudWatch endpoint. See examples below.
## Collector configuration examples
Copy and paste the content below to configure your collector to send logs and traces to the OTLP endpoints.
```
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlphttp/logs:
compression: gzip
logs_endpoint: logs_otlp_endpoint
headers:
x-aws-log-group: ency_log_group
x-aws-log-stream: default
auth:
authenticator: sigv4auth/logs
otlphttp/traces:
compression: gzip
traces_endpoint: traces_otlp_endpoint
auth:
authenticator: sigv4auth/traces
extensions:
sigv4auth/logs:
region: "region"
service: "logs"
sigv4auth/traces:
region: "region"
service: "xray"
service:
telemetry:
extensions: [sigv4auth/logs, sigv4auth/traces]
pipelines:
logs:
receivers: [otlp]
exporters: [otlphttp/logs]
traces:
receivers: [otlp]
exporters: [otlphttp/traces]
```
The following is an example to send logs and traces using sigv4 to us-east-1.
```
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlphttp/logs:
compression: gzip
logs_endpoint: https://logs.us-east-1.amazonaws.com/v1/logs
headers:
x-aws-log-group: MyApplicationLogs
x-aws-log-stream: default
auth:
authenticator: sigv4auth/logs
otlphttp/traces:
compression: gzip
traces_endpoint: https://xray.us-east-1.amazonaws.com/v1/traces
auth:
authenticator: sigv4auth/traces
extensions:
sigv4auth/logs:
region: "us-east-1"
service: "logs"
sigv4auth/traces:
region: "us-east-1"
service: "xray"
service:
telemetry:
extensions: [sigv4auth/logs, sigv4auth/traces]
pipelines:
logs:
receivers: [otlp]
exporters: [otlphttp/logs]
traces:
receivers: [otlp]
exporters: [otlphttp/traces]
```
**Note**
Configure your OpenTelemetry SDKs to *always\$1on* sampling configuration to reliably record 100% spans and get full visibility into your critical applications with CloudWatch Application Signals. For more information, see an [OpenTelemetry Java SDK sampler configuration](https://opentelemetry.io/docs/languages/java/sdk/#sampler) example. For an example on setting up OpenTelemetry Collector with X-Ray OTLP endpoint, see the [application signals demo](https://github.com/aws-observability/application-signals-demo/blob/main/scripts/opentelemetry/otel_simple_setup/opentelemetry.yaml) repository.
Copy and paste the content below to configure your collector to send metrics to the OTLP endpoints.
```
receivers:
otlp:
protocols:
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
send_batch_size: 200
timeout: 10s
exporters:
otlphttp:
tls:
insecure: false
endpoint: metrics_otlp_endpoint
auth:
authenticator: sigv4auth
extensions:
sigv4auth:
service: "monitoring"
region: "region"
service:
extensions: [sigv4auth]
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]
```
The following is an example to send metrics using sigv4 to us-east-1.
```
receivers:
otlp:
protocols:
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
send_batch_size: 200
timeout: 10s
exporters:
otlphttp:
tls:
insecure: false
endpoint: "https://monitoring.us-east-1.amazonaws.com/v1/metrics:443"
auth:
authenticator: sigv4auth
extensions:
sigv4auth:
service: "monitoring"
region: "us-east-1"
service:
extensions: [sigv4auth]
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]
```
# Build your own custom OpenTelemetry Collector
You can build your own custom OpenTelemetry Collector to get the best application observability experience in CloudWatch with OpenTelemetry. In this setup, you need to build your own OpenTelemetry Collector with open source CloudWatch components.
## Prerequisite
Make sure *Transaction Search* is enabled in CloudWatch. For more information, see [Transaction Search](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search.html).
## Build your own collector
You can build your own collector with the following configuration to monitor your application in CloudWatch with OpenTelemetry. For more information, see [Building a custom collector](https://opentelemetry.io/docs/collector/custom-collector/).
The common configuration for CloudWatch.
```
dist:
name: otelcol-dev
description: OTel Collector for sending telemetry to CloudWatch.
output_path: ./otelcol-dev
extensions:
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.111.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/awsproxy v0.113.0
exporters:
- gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.111.0
- gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.111.0
receivers:
- gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.111.0
```
Additional configuration for traces.
```
# Enable Tracing
dist:
name: otelcol-dev
description: OTel Collector for sending telemetry to CloudWatch.
output_path: ./otelcol-dev
extensions:
#Include common configurations and your custom extensions
exporters:
#Include common configurations and your custom extensions
receivers:
- gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.111.0
processors:
- gomod: github.com/amazon-contributing/opentelemetry-collector-contrib/processor/awsapplicationsignalsprocessor v0.113.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.113.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/metricstransformprocessor v0.113.0
replaces:
- github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/awsutil v0.113.0 => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/awsutil v0.113.0
- github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/cwlogs v0.113.0 => github.com/amazon-contributing/opentelemetry-collector-contrib/internal/aws/cwlogs v0.113.0
- github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter v0.113.0 => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awsemfexporter v0.113.0
- github.com/openshift/api v3.9.0+incompatible => github.com/openshift/api v0.0.0-20180801171038-322a19404e37
```
**Note**
Note the following:
After the collector is built, deploy and configure the custom collector in a host or kubernetes environment by following the procedure under [OpenTelemetry Collector](CloudWatch-OTLPSimplesetup.md).
For information on setting up custom OpenTelemetry collector with Application Signals Processor, see an [Application Signals custom configuration](https://github.com/aws-observability/application-signals-demo/blob/main/scripts/opentelemetry/appsignals_custom_otel_setup/custom-opentelemetry.yaml)example. Application Signals Processor only supports the latest versions of the OpenTelemetry Collectors for custom builds. For information on the supported versions, see [opentelemetry-collector-contrib repository.](https://github.com/amazon-contributing/opentelemetry-collector-contrib/tags)
# Exporting collector-less telemetry using AWS Distro for OpenTelemetry (ADOT) SDK
You can use the ADOT SDKs to go collector-less and to send metrics, traces and logs directly to the OTLP endpoints.
**Note**
Application Signals includes Transaction Search capabilities. To avoid duplicate costs, disable Application Signals in the ADOT SDK and keep `OTEL_AWS_APPLICATION_SIGNALS_ENABLED` set to false (default setting).
**Topics**
+ [
## Prerequisite
](#CloudWatch-OTLP-UsingADOT-Prerequisite)
+ [
## Set up IAM permissions for your role
](#setup-iam-permissions-role)
+ [
## Configure your credentials providers
](#configure-credentials-providers)
+ [
## Enabling ADOT SDKs
](#Enabling-ADOT)
## Prerequisite
If you are using traces, make sure Transaction Search is enabled to send spans to the X-Ray OTLP endpoint. For more information, see [Getting started with Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html).
## Set up IAM permissions for your role
Follow these steps to attach the necessary IAM policies to your role:
**Traces:**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. Choose **Roles** and find and select your configured role.
1. Under the **Permissions** tab, choose **Add permissions**, then **Attach policies**.
1. Using the search box, search for the `AWSXrayWriteOnlyPolicy`.
1. Select the `AWSXrayWriteOnlyPolicy` policy and choose **Add permissions**.
**Logs:**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. Choose **Roles** and find and select your configured role.
1. Under the **Permissions** tab, choose **Add permissions**, then **Create inline policy**.
1. Select **CloudWatch Logs** for the service and under **Actions allowed**, filter and select:
```
logs:PutLogEvents
logs:DescribeLogGroups
logs:DescribeLogStreams
```
1. The following is an example IAM policy that grants the required permissions:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "CloudWatchLogsAccess",
"Effect": "Allow",
"Action": [
"logs:PutLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:*:*:log-group:*"
]
}
]
}
```
------
## Configure your credentials providers
ADOT uses the AWS SDKs to automatically discover valid credentials required to export your telemetry data to AWS. For guidance on configuring credentials for your specific environment, [refer to the documentation on how the AWS SDKS retrieves credentials.](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html)
**Setting up IAM credentials for on-premise hosts:**
Configure your AWS credentials in your on-premises host by entering the following command. Replace **ACCESS\$1KEY\$1ID** and **SECRET\$1ACCESS\$1KEY** with the credentials for your configured IAM role or user.
```
$ aws configure
AWS Access Key ID [None]: ACCESS_KEY_ID
AWS Secret Access Key [None]: SECRET_ACCESS_ID
Default region name [None]: MY_REGION
Default output format [None]: json
```
## Enabling ADOT SDKs
You can enable logs and traces for your application to be sent directly to the OTLP endpoints from AWS Distro for OpenTelemetry (ADOT) SDK on Java, Node.js, Python, and .Net.
------
#### [ Java ]
**Note**
You must use **ADOT Java Agent version 2.11.2 or later** for these features to be available.
1. Download the latest version of the AWS Distro for OpenTelemetry Java auto-instrumentation agent. You can download the latest version by using this command:
```
curl -L -O https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest/download/aws-opentelemetry-agent.jar
```
You can view information about all released versions at [aws-otel-java-instrumentation Releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases).
1. To enable the different exporter that directly sends telemetry to the OTLP endpoints and to optimize benefits, use the following environment variables before you start your application:
**Traces:**
+ Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` to specify the X-Ray OTLP traces endpoint: `https://xray.[AWSRegion].amazonaws.com/v1/traces`
For example:
```
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://xray.us-west-2.amazonaws.com/v1/traces"
```
+ Set `OTEL_TRACES_EXPORTER` to `otlp` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` variable to `http/protobuf` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_RESOURCE_ATTRIBUTES` variable to specify the following information as key-value pairs. These environment variables are used by Application Signals, and is converted into X-Ray trace annotations and CloudWatch metric dimensions.:
+ (Optional) `service.name` sets the name of the service. This will be displayed as the service name for your application in Application Signals dashboards. If you don't provide a value for this key, the default of `UnknownService` is used.
+ (Optional) `deployment.environment` sets the environment that the application runs in. This will be displayed as the **Hosted In** environment of your application.
+ (Optional) To enable log correlation, in `OTEL_RESOURCE_ATTRIBUTES`, set an additional environment variable `aws.log.group.names` for the log groups of your application. By doing so, the traces and metrics from your application can be correlated with the relevant log entries from these log groups. For this variable, replace `$YOUR_APPLICATION_LOG_GROUP` with the log group names for your application. If you have multiple log groups, you can use an ampersand (`&`) to separate them as in this example: `aws.log.group.names=log-group-1&log-group-2`. To enable metric to log correlation, setting this current environmental variable is enough. For more information, see [Enable metric to log correlation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-MetricLogCorrelation.html). To enable trace to log correlation, you'll also need to change the logging configuration in your application. For more information, see [Enable trace to log correlation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-TraceLogCorrelation.html).
**Logs:**
+ Set `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` to specify the CloudWatch OTLP logs endpoint: `https://logs.[AWSRegion].amazonaws.com/v1/logs`
For example:
```
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://logs.us-west-2.amazonaws.com/v1/logs"
```
+ Set `OTEL_EXPORTER_OTLP_LOGS_HEADERS` to specify the log group and log stream (note: these must created **before** running ADOT) you want to export your logs to: `x-aws-log-group=[CW-LOG-GROUP-NAME],x-aws-log-stream=[CW-LOG-STREAM-NAME]`
For example:
```
export OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=MyLogGroup,x-aws-log-stream=default
```
+ Set `OTEL_LOGS_EXPORTER` to `otlp` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` variable to `http/protobuf` (this is optional and is the default value if this environment variable is not set).
1. Set `JAVA_TOOL_OPTIONS` to specify the path where the AWS Distro for OpenTelemetry Java auto-instrumentation agent is stored. For example:
```
export JAVA_TOOL_OPTIONS=" -javaagent:$AWS_ADOT_JAVA_INSTRUMENTATION_PATH"
export AWS_ADOT_JAVA_INSTRUMENTATION_PATH="./aws-opentelemetry-agent.jar"
```
1. Your application should now be running with ADOT Java instrumentation and will generate spans and logs. Spans will be stored in the `aws/spans` CloudWatch log group, while logs will be stored in the log group specified in the `OTEL_EXPORTER_OTLP_LOGS_HEADERS` header. You can also view the logs and metrics correlated with your spans in the CloudWatch Traces and Metrics Console.
1. Start your application with the environment variables listed in the previous step. The following is an example of a starting script.
```
JAVA_TOOL_OPTIONS=" -javaagent:$AWS_ADOT_JAVA_INSTRUMENTATION_PATH" \
OTEL_METRICS_EXPORTER=none \
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces \
OTEL_LOGS_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.us-east-1.amazonaws.com/v1/logs \
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=MyLogGroup,x-aws-log-stream=default
OTEL_RESOURCE_ATTRIBUTES="service.name=$YOUR_SVC_NAME" \
java -jar $MY_JAVA_APP.jar
```
------
#### [ Node.js ]
**Note**
You must use **ADOT JavaScript version 0.7.0 or later** for these features to be available.
1. Download the latest version of the AWS Distro for OpenTelemetry JavaScript auto-instrumentation. Install it by running the following command.
```
npm install @aws/aws-distro-opentelemetry-node-autoinstrumentation
```
You can view information about all released versions at [aws-otel-js-instrumentation Releases](https://github.com/aws-observability/aws-otel-js-instrumentation/releases).
1. To enable the different exporter that directly sends telemetry to the OTLP endpoints and to optimize benefits, use the following environment variables before you start your application:
**Traces:**
+ Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` to specify the X-Ray OTLP traces endpoint: `https://xray.[AWSRegion].amazonaws.com/v1/traces`
For example:
```
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://xray.us-west-2.amazonaws.com/v1/traces"
```
+ Set `OTEL_TRACES_EXPORTER` to `otlp` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` variable to `http/protobuf` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_RESOURCE_ATTRIBUTES` variable to specify the following information as key-value pairs. These environment variables are used by Application Signals, and is converted into X-Ray trace annotations and CloudWatch metric dimensions.:
+ (Optional) `service.name` sets the name of the service. This will be displayed as the service name for your application in Application Signals dashboards. If you don't provide a value for this key, the default of `UnknownService` is used.
+ (Optional) `deployment.environment` sets the environment that the application runs in. This will be diplayed as the **Hosted In** environment of your application.
+ (Optional) To enable log correlation, in `OTEL_RESOURCE_ATTRIBUTES`, set an additional environment variable `aws.log.group.names` for the log groups of your application. By doing so, the traces and metrics from your application can be correlated with the relevant log entries from these log groups. For this variable, replace `$YOUR_APPLICATION_LOG_GROUP` with the log group names for your application. If you have multiple log groups, you can use an ampersand (`&`) to separate them as in this example: `aws.log.group.names=log-group-1&log-group-2`. To enable metric to log correlation, setting this current environmental variable is enough. For more information, see [Enable metric to log correlation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-MetricLogCorrelation.html). To enable trace to log correlation, you'll also need to change the logging configuration in your application. For more information, see [Enable trace to log correlation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-TraceLogCorrelation.html).
**Logs:**
+ Set `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` to specify the CloudWatch OTLP logs endpoint: `https://logs.[AWSRegion].amazonaws.com/v1/logs`
For example:
```
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://logs.us-west-2.amazonaws.com/v1/logs"
```
+ Set `OTEL_EXPORTER_OTLP_LOGS_HEADERS` to specify the log group and log stream (note: these must created **before** running ADOT) you want to export your logs to: `x-aws-log-group=[CW-LOG-GROUP-NAME],x-aws-log-stream=[CW-LOG-STREAM-NAME]`
For example:
```
export OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=MyLogGroup,x-aws-log-stream=default
```
+ Set `OTEL_LOGS_EXPORTER` to `otlp` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` variable to `http/protobuf` (this is optional and is the default value if this environment variable is not set).
1. Your application should now be running with ADOT JavaScript instrumentation and will generate spans and logs. Spans will be stored in the `aws/spans` CloudWatch log group, while logs will be stored in the log group specified in the `OTEL_EXPORTER_OTLP_LOGS_HEADERS` header. You can also view the logs and metrics correlated with your spans in the CloudWatch Traces and Metrics Console.
1. Start your application with the environment variables listed in the previous step. The following is an example of a starting script.
Replace `$SVC_NAME` with your application name. This will be displayed as the name of the application.
```
OTEL_METRICS_EXPORTER=none \
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces \
OTEL_LOGS_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.us-east-1.amazonaws.com/v1/logs \
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=MyLogGroup,x-aws-log-stream=default
OTEL_RESOURCE_ATTRIBUTES="service.name=$YOUR_SVC_NAME" \
node --require '@aws/aws-distro-opentelemetry-node-autoinstrumentation/register' your-application.js
```
------
#### [ Python ]
**Note**
You must use **ADOT Python version 0.10.0 or later** and have **`botocore`** installed for these features to be available.
1. Download the latest version of the AWS Distro for OpenTelemetry Python auto-instrumentation. Install it by running the following command.
```
pip install aws-opentelemetry-distro
```
You can view information about all released versions at [aws-otel-python-instrumentation Releases](https://github.com/aws-observability/aws-otel-python-instrumentation/releases).
1. To enable the different exporter that directly sends telemetry to the OTLP endpoints and to optimize benefits, use the following environment variables before you start your application:
**For both traces and logs configurations, you must set the following environment variables:**
+ `OTEL_PYTHON_DISTRO` to `aws_distro`
+ `OTEL_PYTHON_CONFIGURATOR` to `aws_configurator`
**Traces:**
+ Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` to specify the X-Ray OTLP traces endpoint: `https://xray.[AWSRegion].amazonaws.com/v1/traces`
For example:
```
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://xray.us-west-2.amazonaws.com/v1/traces"
```
+ Set `OTEL_TRACES_EXPORTER` to `otlp` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` variable to `http/protobuf` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_RESOURCE_ATTRIBUTES` variable to specify the following information as key-value pairs. These environment variables are used by Application Signals, and is converted into X-Ray trace annotations and CloudWatch metric dimensions.:
+ (Optional) `service.name` sets the name of the service. This will be displayed as the service name for your application in Application Signals dashboards. If you don't provide a value for this key, the default of `UnknownService` is used.
+ (Optional) `deployment.environment` sets the environment that the application runs in. This will be diplayed as the **Hosted In** environment of your application.
+ (Optional) To enable log correlation, in `OTEL_RESOURCE_ATTRIBUTES`, set an additional environment variable `aws.log.group.names` for the log groups of your application. By doing so, the traces and metrics from your application can be correlated with the relevant log entries from these log groups. For this variable, replace `$YOUR_APPLICATION_LOG_GROUP` with the log group names for your application. If you have multiple log groups, you can use an ampersand (`&`) to separate them as in this example: `aws.log.group.names=log-group-1&log-group-2`. To enable metric to log correlation, setting this current environmental variable is enough. For more information, see [Enable metric to log correlation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-MetricLogCorrelation.html). To enable trace to log correlation, you'll also need to change the logging configuration in your application. For more information, see [Enable trace to log correlation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-TraceLogCorrelation.html).
**Logs:**
+ Set `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED` to `true`
+ Set `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` to specify the CloudWatch OTLP logs endpoint: `https://logs.[AWSRegion].amazonaws.com/v1/logs`
For example:
```
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://logs.us-west-2.amazonaws.com/v1/logs"
```
+ Set `OTEL_EXPORTER_OTLP_LOGS_HEADERS` to specify the log group and log stream (note: these must created **before** running ADOT) you want to export your logs to: `x-aws-log-group=[CW-LOG-GROUP-NAME],x-aws-log-stream=[CW-LOG-STREAM-NAME]`
For example:
```
export OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=MyLogGroup,x-aws-log-stream=default
```
+ Set `OTEL_LOGS_EXPORTER` to `otlp` (this is optional and is the default value if this environment variable is not set).
+ Set `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` variable to `http/protobuf` (this is optional and is the default value if this environment variable is not set).
1. Your application should now be running with ADOT Python instrumentation and will generate spans and logs. Spans will be stored in the `aws/spans` CloudWatch log group, while logs will be stored in the log group specified in the `OTEL_EXPORTER_OTLP_LOGS_HEADERS` header. You can also view the logs and metrics correlated with your spans in the CloudWatch Traces and Metrics Console.
1. Start your application with the environment variables listed in the previous step. The following is an example of a starting script.
Replace `$SVC_NAME` with your application name. This will be displayed as the name of the application.
Replace `$PYTHON_APP` with the location and name of your application.
```
OTEL_METRICS_EXPORTER=none \
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces \
OTEL_LOGS_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.us-east-1.amazonaws.com/v1/logs \
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=MyLogGroup,x-aws-log-stream=default
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
OTEL_RESOURCE_ATTRIBUTES="service.name=$YOUR_SVC_NAME" \
opentelemetry-instrument python $MY_PYTHON_APP.py
```
------
#### [ .Net ]
**Note**
**ADOT .NET version 1.8.0 or later** is required for these features to be available.
Compression is currently not supported.
Logs is currently not supported.
Download the latest version of the AWS Distro for OpenTelemetry .NET auto-instrumentation package. You can view information about all released versions at [aws-otel-dotnet-instrumentation Releases](https://github.com/aws-observability/aws-otel-dotnet-instrumentation/releases).
To enable the different exporters that directly sends telemetry to the OTLP endpoints and to optimize benefits, use the following environment variables before you start your application (replace `dotnet-service-name` in the `OTEL_RESOURCE_ATTRIBUTES` environment variable with the service name of your choice):
**Traces:**
+ You **MUST** set `OTEL_TRACES_EXPORTER` to `none`
+ You **MUST** set `OTEL_AWS_SIG_V4_ENABLED` to `true`
This feature is **not enabled automatically in .NET**. The environment variable is required specifically to identify this use case within the .NET instrumentation. This requirement is unique to .NET and does **not apply to other supported languages**.
The following is an example setup for Linux.
```
export INSTALL_DIR=OpenTelemetryDistribution
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}
export CORECLR_PROFILER_PATH=${INSTALL_DIR}/linux-x64/OpenTelemetry.AutoInstrumentation.Native.so
export DOTNET_ADDITIONAL_DEPS=${INSTALL_DIR}/AdditionalDeps
export DOTNET_SHARED_STORE=${INSTALL_DIR}/store
export DOTNET_STARTUP_HOOKS=${INSTALL_DIR}/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll
export OTEL_DOTNET_AUTO_HOME=${INSTALL_DIR}
export OTEL_DOTNET_AUTO_PLUGINS="AWS.Distro.OpenTelemetry.AutoInstrumentation.Plugin, AWS.Distro.OpenTelemetry.AutoInstrumentation"
export OTEL_TRACES_EXPORTER=none
export OTEL_AWS_SIG_V4_ENABLED=true
export OTEL_RESOURCE_ATTRIBUTES=service.name=dotnet-service-name
export OTEL_METRICS_EXPORTER=none
export OTEL_LOGS_EXPORTER=none
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces
```
The following is an example setup for Windows Server:
```
$env:INSTALL_DIR = "OpenTelemetryDistribution"
$env:CORECLR_ENABLE_PROFILING = 1
$env:CORECLR_PROFILER = "{918728DD-259F-4A6A-AC2B-B85E1B658318}"
$env:CORECLR_PROFILER_PATH = Join-Path $env:INSTALL_DIR "win-x64/OpenTelemetry.AutoInstrumentation.Native.dll"
$env:DOTNET_ADDITIONAL_DEPS = Join-Path $env:INSTALL_DIR "AdditionalDeps"
$env:DOTNET_SHARED_STORE = Join-Path $env:INSTALL_DIR "store"
$env:DOTNET_STARTUP_HOOKS = Join-Path $env:INSTALL_DIR "net/OpenTelemetry.AutoInstrumentation.StartupHook.dll"
$env:OTEL_DOTNET_AUTO_HOME = $env:INSTALL_DIR
$env:OTEL_DOTNET_AUTO_PLUGINS = "AWS.Distro.OpenTelemetry.AutoInstrumentation.Plugin, AWS.Distro.OpenTelemetry.AutoInstrumentation"
$env:OTEL_TRACES_EXPORTER=none
$env:OTEL_AWS_SIG_V4_ENABLED=true
$env:OTEL_RESOURCE_ATTRIBUTES=service.name=dotnet-service-name
$env:OTEL_METRICS_EXPORTER=none
$env:OTEL_LOGS_EXPORTER=none
$env:OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
$env:OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces
```
1. Start your application with the environment variables listed in the previous step.
1. (Optional) Alternatively, you can use the installation scripts provided to help installation and setup of AWS Distro for OpenTelemetry .NET auto-instrumentation package.
For Linux, download and install the Bash installation script from the GitHub releases page:
```
# Download and Install
curl -L -O https://github.com/aws-observability/aws-otel-dotnet-instrumentation/releases/latest/download/aws-otel-dotnet-install.sh
chmod +x ./aws-otel-dotnet-install.sh
./aws-otel-dotnet-install.sh
# Instrument
. $HOME/.otel-dotnet-auto/instrument.shexport OTEL_RESOURCE_ATTRIBUTES=service.name=dotnet-service-name
```
For Windows Server, download and install the PowerShell installation script from the GitHub releases page:
```
# Download and Install
$module_url = "https://github.com/aws-observability/aws-otel-dotnet-instrumentation/releases/latest/download/AWS.Otel.DotNet.Auto.psm1"
$download_path = Join-Path $env:temp "AWS.Otel.DotNet.Auto.psm1"
Invoke-WebRequest -Uri $module_url -OutFile $download_path
Import-Module $download_path
Install-OpenTelemetryCore
# Instrument
Import-Module $download_path
Register-OpenTelemetryForCurrentSession -OTelServiceName "dotnet-service-name"
Register-OpenTelemetryForIIS
```
You can find the NuGet package of the AWS Distro for OpenTelemetry .NET auto-instrumentation package in the [official NuGet repository](https://www.nuget.org/packages/AWS.Distro.OpenTelemetry.AutoInstrumentation). Be sure to check the [README file](https://github.com/aws-observability/aws-otel-dotnet-instrumentation/blob/main/src/AWS.Distro.OpenTelemetry.AutoInstrumentation/nuget-readme.md) for instructions.
------
# Enabling vended metrics in PromQL
You can enable OTel enrichment to make vended metrics for [supported AWS resources](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/UsingResourceTagsForTelemetry.html) queryable via PromQL. Once enabled, metrics that contain a resource identifier dimension (for example, EC2 CPUUtilization with an InstanceId dimension) are enriched with resource ARN and resource tag labels and become queryable using PromQL.
The enriched metric preserves the original metric name and CloudWatch dimensions, and adds:
+ **Resource attributes** – the resource ARN (`cloud.resource_id`), cloud provider, region, and account ID.
+ **Instrumentation scope** – identifies the source service and marks the metric as OTel-enriched.
+ **Resource tags** – any AWS resource tags associated with the resource, queryable as PromQL labels.
The original classic CloudWatch metric is not modified and remains available through existing CloudWatch APIs.
## Enabling OpenTelemetry enrichment for vended metrics
Before you start OTel enrichment, you must [enable resource tags on telemetry](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/EnableResourceTagsOnTelemetry.html) for your account.
You can enable OTel enrichment for your account in a specific region using the CloudWatch console, AWS CLI, or AWS SDK.
You will need permissions for the following operation: `cloudwatch:StartOTelEnrichment`
**To enable OTel enrichment for AWS metrics (CloudWatch Console)**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **CloudWatch**, then choose **Settings**.
1. In the **Enable OTel Enrichment for AWS Metrics** pane, toggle the feature On.
**AWS CLI**
```
aws cloudwatch start-o-tel-enrichment
```
To enable across multiple regions, invoke the API in each region of interest.
Once enrichment is enabled, you can start querying vended metrics via PromQL. See: [Querying vended AWS metrics with PromQL](CloudWatch-PromQL-Querying.md#CloudWatch-PromQL-Querying-Vended).