# Create the CloudWatch agent configuration file
Before running the CloudWatch agent on any servers, you must create one or more CloudWatch agent configuration files.
The agent configuration file is a JSON file that specifies the metrics, logs, and traces that the agent is to collect, including custom metrics. You can create it by using the wizard or by creating it yourself from scratch. You could also use the wizard to initially create the configuration file and then modify it manually. If you create or modify the file manually, the process is more complex, but you have more control over the metrics collected and can specify metrics not available through the wizard.
Any time you change the agent configuration file, you must then restart the agent to have the changes take effect. To restart the agent, follow the instructions in [(Optional) Modify the common configuration and named profile for CloudWatch agent](installing-cloudwatch-agent-ssm.md#CloudWatch-Agent-profile-instance-fleet).
After you have created a configuration file, you can save it manually as a JSON file and then use this file when installing the agent on your servers. Alternatively, you can store it in Systems Manager Parameter Store if you're going to use Systems Manager when you install the agent on servers.
The CloudWatch agent supports using multiple configuration files. For more information, see [Creating multiple CloudWatch agent configuration files](#CloudWatch-Agent-multiple-config-files).
Metrics, logs, and traces collected by the CloudWatch agent incur charges. For more information about pricing, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing).
**Topics**
+ [
# Create the CloudWatch agent configuration file with the wizard
](create-cloudwatch-agent-configuration-file-wizard.md)
+ [
## Creating multiple CloudWatch agent configuration files
](#CloudWatch-Agent-multiple-config-files)
+ [
# Manually create or edit the CloudWatch agent configuration file
](CloudWatch-Agent-Configuration-File-Details.md)
# Create the CloudWatch agent configuration file with the wizard
The agent configuration file wizard, `amazon-cloudwatch-agent-config-wizard`, asks a series of questions to help you configure the CloudWatch agent for your needs. This section describes the credentials required for the configuration file. It describes how to run the CloudWatch agent configuration wizard. It also describes the metrics that are predefined in the wizard.
## Required credentials
The wizard can autodetect the credentials and AWS Region to use if you have the AWS credentials and configuration files in place before you start the wizard. For more information about these files, see [ Configuration and Credential Files](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html) in the *AWS Systems Manager User Guide*.
In the AWS credentials file, the wizard checks for default credentials and also looks for an `AmazonCloudWatchAgent` section such as the following:
```
[AmazonCloudWatchAgent]
aws_access_key_id = my_access_key
aws_secret_access_key = my_secret_key
```
The wizard displays the default credentials, the credentials from the `AmazonCloudWatchAgent`, and an `Others` option. You can select which credentials to use. If you choose `Others`, you can input credentials.
For *my\$1access\$1key* and *my\$1secret\$1key*, use the keys from the IAM user that has the permissions to write to Systems Manager Parameter Store.
In the AWS configuration file, you can specify the Region that the agent sends metrics to if it's different than the `[default]` section. The default is to publish the metrics to the Region where the Amazon EC2 instance is located. If the metrics should be published to a different Region, specify the Region here. In the following example, the metrics are published to the `us-west-1` Region.
```
[AmazonCloudWatchAgent]
region = us-west-1
```
## Run the CloudWatch agent configuration wizard
**To create the CloudWatch agent configuration file**
1. Start the CloudWatch agent configuration wizard by entering the following:
```
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
```
On a server running Windows Server, run the following commands to launch the wizard:
```
cd "C:\Program Files\Amazon\AmazonCloudWatchAgent"
```
```
.\amazon-cloudwatch-agent-config-wizard.exe
```
1. Answer the questions to customize the configuration file for your server.
1. If you're storing the configuration file locally, the configuration file `config.json` is stored in `/opt/aws/amazon-cloudwatch-agent/bin/` on Linux servers, and is stored in `C:\Program Files\Amazon\AmazonCloudWatchAgent` on Windows Server. You can then copy this file to other servers where you want to install the agent.
If you're going to use Systems Manager to install and configure the agent, be sure to answer **Yes** when prompted whether to store the file in Systems Manager Parameter Store. You can also choose to store the file in Parameter Store even if you aren't using the SSM Agent to install the CloudWatch agent. To be able to store the file in Parameter Store, you must use an IAM role with sufficient permissions.
## CloudWatch agent predefined metric sets
The wizard is configured with predefined sets of metrics, with different detail levels. These sets of metrics are shown in the following tables. For more information about these metrics, see [Metrics collected by the CloudWatch agent](metrics-collected-by-CloudWatch-agent.md).
**Note**
Parameter Store supports parameters in Standard and Advanced tiers. These parameter tiers are not related to the Basic, Standard, and Advanced levels of metric details that are described in these tables.
**Amazon EC2 instances running Linux**
| Detail level | Metrics included |
| --- | --- |
| **Basic** | **Mem:** mem\$1used\$1percent **Disk:** disk\$1used\$1percent The `disk` metrics such as `disk_used_percent` have a dimension for `Partition`, which means that the number of custom metrics generated is dependent on the number of partitions associated with your instance. The number of disk partitions you have depends on which AMI you are using and the number of Amazon EBS volumes you attach to the server. |
| **Standard** | **CPU:** `cpu_usage_idle`, `cpu_usage_iowait`, `cpu_usage_user`, `cpu_usage_system` **Disk:** `disk_used_percent`, `disk_inodes_free` **Diskio:** `diskio_io_time` **Mem:** `mem_used_percent` **Swap:** `swap_used_percent` |
| **Advanced** | **CPU:** `cpu_usage_idle`, `cpu_usage_iowait`, `cpu_usage_user`, `cpu_usage_system` **Disk:** `disk_used_percent`, `disk_inodes_free` **Diskio:** `diskio_io_time`, `diskio_write_bytes`, `diskio_read_bytes`, `diskio_writes`, `diskio_reads` **Mem:** `mem_used_percent` **Netstat:** `netstat_tcp_established`, `netstat_tcp_time_wait` **Swap:** `swap_used_percent` |
**On-premises servers running Linux**
| Detail level | Metrics included |
| --- | --- |
| **Basic** | **Disk:** `disk_used_percent` **Diskio:** `diskio_write_bytes`, `diskio_read_bytes`, `diskio_writes`, `diskio_reads` **Mem:** `mem_used_percent` **Net:** `net_bytes_sent`, `net_bytes_recv`, `net_packets_sent`, `net_packets_recv` **Swap:** `swap_used_percent` |
| **Standard** | **CPU:** `cpu_usage_idle`, `cpu_usage_iowait` **Disk:** `disk_used_percent`, `disk_inodes_free` **Diskio:** `diskio_io_time`, `diskio_write_bytes`, `diskio_read_bytes`, `diskio_writes`, `diskio_reads` **Mem:** `mem_used_percent` **Net:** `net_bytes_sent`, `net_bytes_recv`, `net_packets_sent`, `net_packets_recv` **Swap:** `swap_used_percent` |
| **Advanced** | **CPU:** `cpu_usage_guest`, `cpu_usage_idle`, `cpu_usage_iowait`, `cpu_usage_steal`, `cpu_usage_user`, `cpu_usage_system` **Disk:** `disk_used_percent`, `disk_inodes_free` **Diskio:** `diskio_io_time`, `diskio_write_bytes`, `diskio_read_bytes`, `diskio_writes`, `diskio_reads` **Mem:** `mem_used_percent` **Net:** `net_bytes_sent`, `net_bytes_recv`, `net_packets_sent`, `net_packets_recv` **Netstat:** `netstat_tcp_established`, `netstat_tcp_time_wait` **Swap:** `swap_used_percent` |
**Amazon EC2 instances running Windows Server**
**Note**
The metric names listed in this table display how the metric appears when viewed in the console. The actual metric name might not include the first word. For example, the actual metric name for `LogicalDisk % Free Space` is just `% Free Space`.
| Detail level | Metrics included |
| --- | --- |
| **Basic** | **Memory:** `Memory % Committed Bytes In Use` **LogicalDisk:** `LogicalDisk % Free Space` |
| **Standard** | **Memory:** `Memory % Committed Bytes In Use` **Paging:** `Paging File % Usage` **Processor:** `Processor % Idle Time`, `Processor % Interrupt Time`, `Processor % User Time` **PhysicalDisk:** `PhysicalDisk % Disk Time` **LogicalDisk:** `LogicalDisk % Free Space` |
| **Advanced** | **Memory:** `Memory % Committed Bytes In Use` **Paging:** `Paging File % Usage` **Processor:** `Processor % Idle Time`, `Processor % Interrupt Time`, `Processor % User Time` **LogicalDisk:** `LogicalDisk % Free Space` **PhysicalDisk:** `PhysicalDisk % Disk Time`, `PhysicalDisk Disk Write Bytes/sec`, `PhysicalDisk Disk Read Bytes/sec`, `PhysicalDisk Disk Writes/sec`, `PhysicalDisk Disk Reads/sec` **TCP:** `TCPv4 Connections Established`, `TCPv6 Connections Established` |
**On-premises server running Windows Server**
**Note**
The metric names listed in this table display how the metric appears when viewed in the console. The actual metric name might not include the first word. For example, the actual metric name for `LogicalDisk % Free Space` is just `% Free Space`.
| Detail level | Metrics included |
| --- | --- |
| **Basic** | **Paging: **`Paging File % Usage` **Processor:** `Processor % Processor Time` **LogicalDisk:**`LogicalDisk % Free Space` **PhysicalDisk:** `PhysicalDisk Disk Write Bytes/sec`, `PhysicalDisk Disk Read Bytes/sec`, `PhysicalDisk Disk Writes/sec`, `PhysicalDisk Disk Reads/sec` **Memory:** `Memory % Committed Bytes In Use` **Network Interface:** `Network Interface Bytes Sent/sec`, `Network Interface Bytes Received/sec`, `Network Interface Packets Sent/sec`, `Network Interface Packets Received/sec` |
| **Standard** | **Paging:** `Paging File % Usage` **Processor:** `Processor % Processor Time`, `Processor % Idle Time`, `Processor % Interrupt Time` **LogicalDisk:** `LogicalDisk % Free Space` **PhysicalDisk:** `PhysicalDisk % Disk Time`, `PhysicalDisk Disk Write Bytes/sec`, `PhysicalDisk Disk Read Bytes/sec`, `PhysicalDisk Disk Writes/sec`, `PhysicalDisk Disk Reads/sec` **Memory:** `Memory % Committed Bytes In Use` **Network Interface:** `Network Interface Bytes Sent/sec`, `Network Interface Bytes Received/sec`, `Network Interface Packets Sent/sec`, `Network Interface Packets Received/sec` |
| **Advanced** | **Paging:**`Paging File % Usage` **Processor:** `Processor % Processor Time`, `Processor % Idle Time`, `Processor % Interrupt Time`, `Processor % User Time` **LogicalDisk:** `LogicalDisk % Free Space` **PhysicalDisk:** `PhysicalDisk % Disk Time`, `PhysicalDisk Disk Write Bytes/sec`, `PhysicalDisk Disk Read Bytes/sec`, `PhysicalDisk Disk Writes/sec`, `PhysicalDisk Disk Reads/sec` **Memory:** `Memory % Committed Bytes In Use` **Network Interface:** `Network Interface Bytes Sent/sec`, `Network Interface Bytes Received/sec`, `Network Interface Packets Sent/sec`, `Network Interface Packets Received/sec` **TCP:** `TCPv4 Connections Established`, `TCPv6 Connections Established` |
# Examples of configuration files
**Basic system metrics configuration**
```
{
"agent": {
"metrics_collection_interval": 60,
"region": "us-east-1"
},
"metrics": {
"namespace": "MySystem",
"metrics_collected": {
"cpu": {
"resources": ["*"],
"measurement": ["usage_active", "usage_system", "usage_user"]
},
"mem": {
"measurement": ["used_percent"]
},
"disk": {
"resources": ["/"],
"measurement": ["used_percent"]
}
},
"append_dimensions": {
"InstanceId": "${aws:InstanceId}"
}
}
}
```
**Web server monitoring configuration **
```
{
"agent": {
"metrics_collection_interval": 60,
"region": "us-east-1"
},
"metrics": {
"namespace": "WebServer",
"metrics_collected": {
"cpu": {
"resources": ["*"],
"measurement": ["usage_active"]
},
"mem": {
"measurement": ["used_percent"]
},
"net": {
"resources": ["eth0"],
"measurement": ["bytes_sent", "bytes_recv"]
}
}
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/apache2/access.log",
"log_group_name": "apache-access-logs",
"log_stream_name": "{instance_id}-access"
},
{
"file_path": "/var/log/apache2/error.log",
"log_group_name": "apache-error-logs",
"log_stream_name": "{instance_id}-error"
}
]
}
}
}
}
```
**Database server configuration **
```
{
"agent": {
"metrics_collection_interval": 30,
"region": "us-east-1"
},
"metrics": {
"namespace": "DatabaseServer",
"metrics_collected": {
"cpu": {
"resources": ["*"],
"measurement": ["usage_active", "usage_iowait"]
},
"mem": {
"measurement": ["used_percent", "available_percent"]
},
"disk": {
"resources": ["/", "/data"],
"measurement": ["used_percent", "inodes_free"]
},
"diskio": {
"resources": ["*"],
"measurement": ["read_bytes", "write_bytes", "io_time"]
}
},
"append_dimensions": {
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}",
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
}
}
}
```
## Creating multiple CloudWatch agent configuration files
On both Linux servers and Windows servers, you can set up the CloudWatch agent to use multiple configuration files. For example, you can use a common configuration file that collects a set of metrics, logs, and traces that you always want to collect from all servers in your infrastructure. You can then use additional configuration files that collect metrics from certain applications or in certain situations.
To set this up, first create the configuration files that you want to use. Any configuration files that will be used together on the same server must have different file names. You can store the configuration files on servers or in Parameter Store.
Start the CloudWatch agent using the `fetch-config` option and specify the first configuration file. To append the second configuration file to the running agent, use the same command but with the `append-config` option. All metrics, logs, and traces listed in either configuration file are collected. The following example commands illustrate this scenario using configurations stores as files. The first line starts the agent using the `infrastructure.json` configuration file, and the second line appends the `app.json` configuration file.
The following example commands are for Linux.
```
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/tmp/infrastructure.json
```
```
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:/tmp/app.json
```
The following example commands are for Windows Server.
```
& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\infrastructure.json"
```
```
& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a append-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\app.json"
```
The following example configuration files illustrate a use for this feature. The first configuration file is used for all servers in the infrastructure, and the second collects only logs from a certain application and is appended to servers running that application.
**infrastructure.json**
```
{
"metrics": {
"metrics_collected": {
"cpu": {
"resources": [
"*"
],
"measurement": [
"usage_active"
],
"totalcpu": true
},
"mem": {
"measurement": [
"used_percent"
]
}
}
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
"log_group_name": "amazon-cloudwatch-agent.log"
},
{
"file_path": "/var/log/messages",
"log_group_name": "/var/log/messages"
}
]
}
}
}
}
```
**app.json**
```
{
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/app/app.log*",
"log_group_name": "/app/app.log"
}
]
}
}
}
}
```
Any configuration files appended to the configuration must have different file names from each other and from the initial configuration file. If you use `append-config` with a configuration file with the same file name as a configuration file that the agent is already using, the append command overwrites the information from the first configuration file instead of appending to it. This is true even if the two configuration files with the same file name are on different file paths.
The preceding example shows the use of two configuration files, but there is no limit to the number of configuration files that you can append to the agent configuration. You can also mix the use of configuration files located on servers and configurations located in Parameter Store.
# Manually create or edit the CloudWatch agent configuration file
The CloudWatch agent configuration file is a JSON file with four sections: `agent`, `metrics`, `logs`, and `traces`.
+ The `agent` section includes fields for the overall configuration of the agent.
+ The `metrics` section specifies the custom metrics for collection and publishing to CloudWatch. If you're using the agent only to collect logs, you can omit the `metrics` section from the file.
+ The `logs` section specifies what log files are published to CloudWatch Logs. This can include events from the Windows Event Log if the server runs Windows Server.
+ The `traces` section specifies the sources for traces that are collected and sent to AWS X-Ray.
This section explains the structure and fields of the CloudWatch agent configuration file. You can view the schema definition for this configuration file. The schema definition is located at `installation-directory/doc/amazon-cloudwatch-agent-schema.json` on Linux servers and at `installation-directory/amazon-cloudwatch-agent-schema.json` on servers running Windows Server.
If you create or edit the agent configuration file manually, you can give it any name. For simplicity in troubleshooting, we recommend that you name it `/opt/aws/amazon-cloudwatch-agent/etc/cloudwatch-agent.json` on a Linux server and `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.json` on servers running Windows Server. After you have created the file, you can copy it to other servers where you want to install the agent.
When the agent is started, it creates a copy of each configuration file in `/opt/aws/amazon-cloudwatch/etc/amazon-cloudwatch-agent.d` directory, with the filename prefixed with either `file_` (for local file sources) or `ssm_` (for Systems Manager parameter store sources) to indicate the configuration origin.
**Note**
Metrics, logs, and traces collected by the CloudWatch agent incur charges. For more information about pricing, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing).
## CloudWatch agent configuration file: Agent section
The `agent` section can include the following fields. The wizard doesn't create an `agent` section. Instead, the wizard omits it and uses the default values for all fields in this section.
+ `metrics_collection_interval` – Optional. Specifies how often all metrics specified in this configuration file are to be collected. You can override this value for specific types of metrics.
This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
The default value is 60.
+ `region` – Specifies the Region to use for the CloudWatch endpoint when an Amazon EC2 instance is being monitored. The metrics collected are sent to this Region, such as `us-west-1`. If you omit this field, the agent sends metrics to the Region where the Amazon EC2 instance is located.
If you are monitoring an on-premises server, this field isn't used, and the agent reads the Region from the `AmazonCloudWatchAgent` profile of the AWS configuration file.
+ `credentials` – Specifies an IAM role to use when sending metrics, logs, and traces to a different AWS account. If specified, this field contains one parameter, `role_arn`.
+ `role_arn` – Specifies the Amazon Resource Name (ARN) of an IAM role to use for authentication when sending metrics, logs, and traces to a different AWS account. For more information, see [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account).
+ `debug` – Optional. Specifies running the CloudWatch agent with debug log messages. The default value is `false`.
+ `aws_sdk_log_level` – Optional. Supported only in versions 1.247350.0 and later of the CloudWatch agent.
You can specify this field to have the agent perform logging for AWS SDK endpoints. The value for this field can include one or more of the following options. Separate multiple options with the `|` character.
+ `LogDebug`
+ `LogDebugWithSigning`
+ `LogDebugWithHTTPBody`
+ `LogDebugRequestRetries`
+ `LogDebugWithEventStreamBody`
For more information about these options, see [LogLevelType](https://docs.aws.amazon.com/sdk-for-go/api/aws/#LogLevelType).
+ `logfile` – Specifies the location where the CloudWatch agent writes log messages. If you specify an empty string, the log goes to stderr. If you don't specify this option, the default locations are the following:
+ Linux: `/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log`
+ Windows Server: `c:\\ProgramData\\Amazon\\CloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log`
The CloudWatch agent automatically rotates the log file that it creates. A log file is rotated out when it reaches 100 MB in size. The agent keeps the rotated log files for up to seven days, and it keeps as many as five backup log files that have been rotated out. Backup log files have a timestamp appended to their filename. The timestamp shows the date and time that the file was rotated out: for example, `amazon-cloudwatch-agent-2018-06-08T21-01-50.247.log.gz`.
+ `omit_hostname` – Optional. By default, the hostname is published as a dimension of metrics that are collected by the agent, unless you are using the `append_dimensions` field in the `metrics` section. Set `omit_hostname ` to `true` to prevent the hostname from being published as a dimension even if you are not using `append_dimensions`. The default value is `false`.
+ `run_as_user` – Optional. Specifies a user to use to run the CloudWatch agent. If you don't specify this parameter, the root user is used. This option is valid only on Linux servers.
If you specify this option, the user must exist before you start the CloudWatch agent. For more information, see [Running the CloudWatch agent as a different user](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-run-as-user).
+ `user_agent` – Optional. Specifies the `user-agent` string that is used by the CloudWatch agent when it makes API calls to the CloudWatch backend. The default value is a string consisting of the agent version, the version of the Go programming language that was used to compile the agent, the runtime operating system and architecture, the build time, and the plugins enabled.
+ `usage_data` – Optional. By default, the CloudWatch agent sends health and performance data about itself to CloudWatch whenever it publishes metrics or logs to CloudWatch. This data incurs no costs to you. You can prevent the agent from sending this data by specifying `false` for `usage_data`. If you omit this parameter, the default of `true` is used and the agent sends the health and performance data.
If you set this value to `false`, you must stop and restart the agent for it to take effect.
+ `service.name` – Optional. Specifies the service name to be used to populate the entity for [finding related telemetry](ExploreRelated.md).
+ `deployment.environment` – Optional. Specifies the environment name to be used to populate the entity for [finding related telemetry](ExploreRelated.md).
+ `use_dualstack_endpoint` – Optional. If this is `true`, the CloudWatch agent will use [dual stack endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#dual-stack-endpoints) for all API calls.
The following is an example of an `agent` section.
```
"agent": {
"metrics_collection_interval": 60,
"region": "us-west-1",
"logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
"debug": false,
"run_as_user": "cwagent"
}
```
## CloudWatch agent configuration file: Metrics section
### Fields common to Linux and Windows
On servers running either Linux or Windows Server, the `metrics` section includes the following fields:
+ `namespace` – Optional. The namespace to use for the metrics collected by the agent. The default value is `CWAgent`. The maximum length is 255 characters. The following is an example:
```
{
"metrics": {
"namespace": "Development/Product1Metrics",
......
},
}
```
+ `append_dimensions` – Optional. Adds Amazon EC2 metric dimensions to all metrics collected by the agent. This also causes the agent to not publish the hostname as a dimension.
The only supported key-value pairs for `append_dimensions` are shown in the following list. Any other key-value pairs are ignored. The agent supports these key-value pairs exactly as they are shown in the following list. You can't change the key values to publish different dimension names for them.
+ `"ImageId":"${aws:ImageId}"` sets the instance's AMI ID as the value of the `ImageId` dimension.
+ `"InstanceId":"${aws:InstanceId}"` sets the instance's instance ID as the value of the `InstanceId` dimension.
+ `"InstanceType":"${aws:InstanceType}"` sets the instance's instance type as the value of the `InstanceType` dimension.
+ `"AutoScalingGroupName":"${aws:AutoScalingGroupName}"` sets the instance's Auto Scaling group name as the value of the `AutoScalingGroupName` dimension.
If you want to append dimensions to metrics with arbitrary key-value pairs, use the `append_dimensions` parameter in the field for that particular type of metric.
If you specify a value that depends on Amazon EC2 metadata and you use proxies, you must make sure that the server can access the endpoint for Amazon EC2. For more information about these endpoints, see [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region) in the *Amazon Web Services General Reference*.
+ `aggregation_dimensions` – Optional. Specifies the dimensions that collected metrics are to be aggregated on. For example, if you roll up metrics on the `AutoScalingGroupName` dimension, the metrics from all instances in each Auto Scaling group are aggregated and can be viewed as a whole.
You can roll up metrics along single or multiple dimensions. For example, specifying `[["InstanceId"], ["InstanceType"], ["InstanceId","InstanceType"]]` aggregates metrics for instance ID singly, instance type singly, and for the combination of the two dimensions.
You can also specify `[]` to roll up all metrics into one collection, disregarding all dimensions.
+ `endpoint_override` – Specifies a FIPS endpoint or private link to use as the endpoint where the agent sends metrics. Specifying this and setting a private link enables you to send the metrics to an Amazon VPC endpoint. For more information, see [What Is Amazon VPC?](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html).
The value of `endpoint_override` must be a string that is a URL.
For example, the following part of the metrics section of the configuration file sets the agent to use a VPC Endpoint when sending metrics.
```
{
"metrics": {
"endpoint_override": "vpce-XXXXXXXXXXXXXXXXXXXXXXXXX.monitoring.us-east-1.vpce.amazonaws.com",
......
},
}
```
+ `metrics_collected` – Required. Specifies which metrics are to be collected, including custom metrics collected through `StatsD` or `collectd`. This section includes several subsections.
The contents of the `metrics_collected` section depend on whether this configuration file is for a server running Linux or Windows Server.
+ `metrics_destinations` – Optional. Specifies one or more destinations for all metrics defined in `metrics_collected`. If specified here, it overrides the default destination of `cloudwatch`.
+ `cloudwatch` – Amazon CloudWatch.
+ `amp` – Amazon Managed Service for Prometheus.
+ `workspace_id` – The ID corresponding to the Amazon Managed Service for Prometheus workspace.
```
{
"metrics": {
"metrics_destinations": {
"cloudwatch": {},
"amp": {
"workspace_id": "ws-abcd1234-ef56-7890-ab12-example"
}
}
}
}
```
+ `force_flush_interval` – Specifies in seconds the maximum amount of time that metrics remain in the memory buffer before being sent to the server. No matter the setting for this, if the size of the metrics in the buffer reaches 1 MB or 1000 different metrics, the metrics are immediately sent to the server.
The default value is 60.
+ `credentials` – Specifies an IAM role to use when sending metrics to a different account. If specified, this field contains one parameter, `role_arn`.
+ `role_arn` – Specifies the ARN of an IAM role to use for authentication when sending metrics to a different account. For more information, see [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account). If specified here, this value overrides the `role_arn` specified in the `agent` section of the configuration file, if any.
+ `service.name` – Optional. Specifies the service name to be used to populate the entity for [finding related telemetry](ExploreRelated.md).
+ `deployment.environment` – Optional. Specifies the environment name to be used to populate the entity for [finding related telemetry](ExploreRelated.md).
### Linux section
On servers running Linux, the `metrics_collected` section of the configuration file can also contain the following fields.
Many of these fields can include a `measurement` sections that lists the metrics you want to collect for that resource. These `measurement` sections can either specify the complete metric name such as `swap_used`, or just the part of the metric name that will be appended to the type of resource. For example, specifying `reads` in the `measurement` section of the `diskio` section causes the `diskio_reads` metric to be collected.
+ `collectd` – Optional. Specifies that you want to retrieve custom metrics using the `collectd` protocol. You use `collectd` software to send the metrics to the CloudWatch agent. For more information about the configuration options available for collectd, see [Retrieve custom metrics with collectd](CloudWatch-Agent-custom-metrics-collectd.md).
+ `cpu` – Optional. Specifies that CPU metrics are to be collected. This section is valid only for Linux instances. You must include at least one of the `resources` and `totalcpu` fields for any CPU metrics to be collected. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `resources` – Optional. Specify this field with a value of `*` to cause per-cpu metrics to be collected. The only allowed value is `*`.
+ `totalcpu` – Optional. Specifies whether to report cpu metrics aggregated across all cpu cores. The default is true.
+ `measurement` – Specifies the array of cpu metrics to be collected. Possible values are `time_active`, `time_guest`, `time_guest_nice`, `time_idle`, `time_iowait`, `time_irq`, `time_nice`, `time_softirq`, `time_steal`, `time_system`, `time_user`, `usage_active`, `usage_guest`, `usage_guest_nice`, `usage_idle`, `usage_iowait`, `usage_irq`, `usage_nice`, `usage_softirq`, `usage_steal`, `usage_system`, and `usage_user`. This field is required if you include `cpu`.
By default, the unit for `cpu_usage_*` metrics is `Percent`, and `cpu_time_*` metrics don't have a unit.
Within the entry for each individual metric, you might optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `metrics_collection_interval` – Optional. Specifies how often to collect the cpu metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Additional dimensions to use for only the cpu metrics. If you specify this field, it's used in addition to dimensions specified in the global `append_dimensions` field that is used for all types of metrics that the agent collects.
+ `disk` – Optional. Specifies that disk metrics are to be collected. Collects metrics only for mounted volumes. This section is valid only for Linux instances. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `resources` – Optional. Specifies an array of disk mount points. This field limits CloudWatch to collect metrics from only the listed mount points. You can specify `*` as the value to collect metrics from all mount points. The default value is to collect metrics from all mount points.
+ `measurement` – Specifies the array of disk metrics to be collected. Possible values are `free`, `total`, `used`, `used_percent`, `inodes_free`, `inodes_used`, and `inodes_total`. This field is required if you include `disk`.
**Note**
The `disk` metrics have a dimension for `Partition`, which means that the number of custom metrics generated is dependent on the number of partitions associated with your instance. The number of disk partitions you have depends on which AMI you are using and the number of Amazon EBS volumes you attach to the server.
To see the default units for each `disk` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent).
Within the entry for each individual metric, you might optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `ignore_file_system_types` – Specifies file system types to exclude when collecting disk metrics. Valid values include `sysfs`, `devtmpfs`, and so on.
+ `drop_device` – Setting this to `true` causes `Device` to not be included as a dimension for disk metrics.
Preventing `Device` from being used as a dimension can be useful on instances that use the Nitro system because on those instances the device names change for each disk mount when the instance is rebooted. This can cause inconsistent data in your metrics and cause alarms based on these metrics to go to `INSUFFICIENT DATA` state.
The default is `false`.
+ `metrics_collection_interval` – Optional. Specifies how often to collect the disk metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Specify key-value pairs to use as additional dimensions for only the disk metrics. If you specify this field, it is used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent.
One key-value pair that you can use is the following. You can also specify other custom key-value pairs.
+ `"VolumeId":"${aws:VolumeId}"` adds a `VolumeId` dimension to your block device disk metrics. For Amazon EBS volumes, this will be the Amazon EBS Volume ID. For EC2 instance store, this will be the device serial. Using this requires the `drop_device` parameter to be set to `false`.
+ `diskio` – Optional. Specifies that disk i/o metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `resources` – Optional. If you specify an array of devices, CloudWatch collects metrics from only those devices. Otherwise, metrics for all devices are collected. You can also specify \$1 as the value to collect metrics from all devices.
+ `measurement` – Specifies the array of diskio and AWS NVMe driver metrics to be collected for Amazon EBS volumes and instance store volumes attached to Amazon EC2 instances. Possible diskio values are `reads`, `writes`, `read_bytes`, `write_bytes`, `read_time`, `write_time`, `io_time`, and `iops_in_progress`. For a list of the NVMe driver metrics for Amazon EBS volumes and Amazon EC2 instance store volumes, see [Collect Amazon EBS NVMe driver metrics](Container-Insights-metrics-EBS-Collect.md) and [Collect Amazon EC2 instance store volume NVMe driver metrics](Container-Insights-metrics-instance-store-Collect.md). This field is required if you include `diskio`.
Within the entry for each individual metric, you might optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
For information on default units and description of the metrics, see [Collect Amazon EBS NVMe driver metrics](Container-Insights-metrics-EBS-Collect.md).
+ `metrics_collection_interval` – Optional. Specifies how often to collect the diskio metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Additional dimensions to use for only the diskio metrics. If you specify this field, it is used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent.
+ `swap` – Optional. Specifies that swap memory metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `measurement` – Specifies the array of swap metrics to be collected. Possible values are `free`, `used`, and `used_percent`. This field is required if you include `swap`.
To see the default units for each `swap` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent).
Within the entry for each individual metric, you might optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `metrics_collection_interval` – Optional. Specifies how often to collect the swap metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Additional dimensions to use for only the swap metrics. If you specify this field, it is used in addition to dimensions specified in the global `append_dimensions` field that is used for all types of metrics collected by the agent. It's collected as a high-resolution metric.
+ `mem` – Optional. Specifies that memory metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `measurement` – Specifies the array of memory metrics to be collected. Possible values are `active`, `available`, `available_percent`, `buffered`, `cached`, `free`, `inactive`, `shared`, `total`, `used`, and `used_percent`. This field is required if you include `mem`.
To see the default units for each `mem` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent).
Within the entry for each individual metric, you might optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `metrics_collection_interval` – Optional. Specifies how often to collect the mem metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Additional dimensions to use for only the mem metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics that the agent collects.
+ `net` – Optional. Specifies that networking metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `resources` – Optional. If you specify an array of network interfaces, CloudWatch collects metrics from only those interfaces. Otherwise, metrics for all devices are collected. You can also specify `*` as the value to collect metrics from all interfaces.
+ `measurement` – Specifies the array of networking metrics to be collected. Possible values are `bytes_sent`, `bytes_recv`, `drop_in`, `drop_out`, `err_in`, `err_out`, `packets_sent`, and `packets_recv`. This field is required if you include `net`.
To see the default units for each `net` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent).
Within the entry for each individual metric, you might optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `metrics_collection_interval` – Optional. Specifies how often to collect the net metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Additional dimensions to use for only the net metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent.
+ `netstat` – Optional. Specifies that TCP connection state and UDP connection metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `measurement` – Specifies the array of netstat metrics to be collected. Possible values are `tcp_close`, `tcp_close_wait`, `tcp_closing`, `tcp_established`, `tcp_fin_wait1`, `tcp_fin_wait2`, `tcp_last_ack`, `tcp_listen`, `tcp_none`, `tcp_syn_sent`, `tcp_syn_recv`, `tcp_time_wait`, and `udp_socket`. This field is required if you include `netstat`.
To see the default units for each `netstat` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent).
Within the entry for each individual metric, you might optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `metrics_collection_interval` – Optional. Specifies how often to collect the netstat metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Additional dimensions to use for only the netstat metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent.
+ `processes` – Optional. Specifies that process metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `measurement` – Specifies the array of processes metrics to be collected. Possible values are `blocked`, `dead`, `idle`, `paging`, `running`, `sleeping`, `stopped`, `total`, `total_threads`, `wait`, and `zombies`. This field is required if you include `processes`.
For all `processes` metrics, the default unit is `None`.
Within the entry for each individual metric, you might optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `metrics_collection_interval` – Optional. Specifies how often to collect the processes metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Additional dimensions to use for only the process metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent.
+ `nvidia_gpu` – Optional. Specifies that NVIDIA GPU metrics are to be collected. This section is valid only for Linux instances on hosts that are configured with a NVIDIA GPU accelerator and have the NVIDIA System Management Interface (nvidia-smi) installed.
The NVIDIA GPU metrics that are collected are prefixed with the string `nvidia_smi_` to distinguish them from the metrics collected for other accelerator types. This section can include the following fields:
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
+ `measurement` – Specifies the array of NVIDIA GPU metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the table in [Collect NVIDIA GPU metrics](CloudWatch-Agent-NVIDIA-GPU.md).
Within the entry for each individual metric, you can optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `metrics_collection_interval` – Optional. Specifies how often to collect the NVIDIA GPU metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
+ `jmx` – Optional. Specifies that you want to retrieve Java Management Extensions (JMX) metrics from the instance. For more information about the parameters that you can use in this section and the metrics that you can collect, see [Collect Java Management Extensions (JMX) metrics](CloudWatch-Agent-JMX-metrics.md).
+ `otlp` – Optional. Specifies that you want to collect metrics from the OpenTelemetry SDK. For more information about the fields that you can use in this section, see [Collect metrics and traces with OpenTelemetry](CloudWatch-Agent-OpenTelemetry-metrics.md).
+ `procstat` – Optional. Specifies that you want to retrieve metrics from individual processes. For more information about the configuration options available for procstat, see [Collect process metrics with the procstat plugin](CloudWatch-Agent-procstat-process-metrics.md).
+ `statsd` – Optional. Specifies that you want to retrieve custom metrics using the `StatsD` protocol. The CloudWatch agent acts as a daemon for the protocol. You use any standard `StatsD` client to send the metrics to the CloudWatch agent. For more information about the configuration options available for StatsD, see [Retrieve custom metrics with StatsD](CloudWatch-Agent-custom-metrics-statsd.md).
+ `ethtool` – Optional. Specifies that you want to retrieve network metrics using the `ethtool` plugin. This plugin can import both the metrics collected by the standard ethtool utility, as well as network performance metrics from Amazon EC2 instances. For more information about the configuration options available for ethtool, see [Collect network performance metrics](CloudWatch-Agent-network-performance.md).
The following is an example of a `metrics` section for a Linux server. In this example, three CPU metrics, three netstat metrics, three process metrics, and one disk metric are collected, and the agent is set up to receive additional metrics from a `collectd` client.
```
"metrics": {
"aggregation_dimensions" : [["AutoScalingGroupName"], ["InstanceId", "InstanceType"],[]],
"metrics_collected": {
"collectd": {},
"cpu": {
"resources": [
"*"
],
"measurement": [
{"name": "cpu_usage_idle", "rename": "CPU_USAGE_IDLE", "unit": "Percent"},
{"name": "cpu_usage_nice", "unit": "Percent"},
"cpu_usage_guest"
],
"totalcpu": false,
"drop_original_metrics": [ "cpu_usage_guest" ],
"metrics_collection_interval": 10,
"append_dimensions": {
"test": "test1",
"date": "2017-10-01"
}
},
"netstat": {
"measurement": [
"tcp_established",
"tcp_syn_sent",
"tcp_close"
],
"metrics_collection_interval": 60
},
"disk": {
"measurement": [
"used_percent"
],
"resources": [
"*"
],
"drop_device": true
},
"processes": {
"measurement": [
"running",
"sleeping",
"dead"
]
}
},
"append_dimensions": {
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}",
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
}
}
```
### Windows Server
In the `metrics_collected` section for Windows Server, you can have subsections for each Windows performance object, such as `Memory`, `Processor`, and `LogicalDisk`. For information about what objects and counters are available, see [Performance Counters](https://learn.microsoft.com/en-us/windows/win32/perfctrs/performance-counters-portal) in the Microsoft Windows documentation.
Within the subsection for each object, you specify a `measurement` array of the counters to collect. The `measurement` array is required for each object that you specify in the configuration file. You can also specify a `resources` field to name the instances to collect metrics from. You can also specify `*` for `resources` to collect separate metrics for every instance. If you omit `resources` for counters that have instances, the data for all instances is aggregated into one set. If you omit `resources` for counters that don't have instances, the counters are not collected by the CloudWatch agent. To determine whether counters have instances, you can use one of the following commands.
Powershell:
```
Get-Counter -ListSet *
```
Command line (not Powershell):
```
TypePerf.exe –q
```
Within each object section, you can also specify the following optional fields:
+ `metrics_collection_interval` – Optional. Specifies how often to collect the metrics for this object, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
+ `append_dimensions` – Optional. Specifies additional dimensions to use for only the metrics for this object. If you specify this field, it's used in addition to dimensions specified in the global `append_dimensions` field that is used for all types of metrics collected by the agent.
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
Within each counter section, you can also specify the following optional fields:
+ `rename` – Specifies a different name to be used in CloudWatch for this metric.
+ `unit` – Specifies the unit to use for this metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
There are other optional sections that you can include in `metrics_collected`:
+ `statsd` – Enables you to retrieve custom metrics using the `StatsD` protocol. The CloudWatch agent acts as a daemon for the protocol. You use any standard `StatsD` client to send the metrics to the CloudWatch agent. For more information, see [Retrieve custom metrics with StatsD](CloudWatch-Agent-custom-metrics-statsd.md).
+ `procstat` – Enables you to retrieve metrics from individual processes. For more information, see [Collect process metrics with the procstat plugin](CloudWatch-Agent-procstat-process-metrics.md).
+ `jmx` – Optional. Specifies that you want to retrieve Java Management Extensions (JMX) metrics from the instance. For more information about the fields that you can use in this section and the metrics that you can collect, see [Collect Java Management Extensions (JMX) metrics](CloudWatch-Agent-JMX-metrics.md).
+ `otlp` – Optional. Specifies that you want to collect metrics from the OpenTelemetry SDK. For more information about the fields that you can use in this section, see [Collect metrics and traces with OpenTelemetry](CloudWatch-Agent-OpenTelemetry-metrics.md).
The following is an example `metrics` section for use on Windows Server. In this example, many Windows metrics are collected, and the computer is also set to receive additional metrics from a `StatsD` client.
```
"metrics": {
"metrics_collected": {
"statsd": {},
"Processor": {
"measurement": [
{"name": "% Idle Time", "rename": "CPU_IDLE", "unit": "Percent"},
"% Interrupt Time",
"% User Time",
"% Processor Time"
],
"resources": [
"*"
],
"append_dimensions": {
"d1": "win_foo",
"d2": "win_bar"
}
},
"LogicalDisk": {
"measurement": [
{"name": "% Idle Time", "unit": "Percent"},
{"name": "% Disk Read Time", "rename": "DISK_READ"},
"% Disk Write Time"
],
"resources": [
"*"
]
},
"Memory": {
"metrics_collection_interval": 5,
"measurement": [
"Available Bytes",
"Cache Faults/sec",
"Page Faults/sec",
"Pages/sec"
],
"append_dimensions": {
"d3": "win_bo"
}
},
"Network Interface": {
"metrics_collection_interval": 5,
"measurement": [
"Bytes Received/sec",
"Bytes Sent/sec",
"Packets Received/sec",
"Packets Sent/sec"
],
"resources": [
"*"
],
"append_dimensions": {
"d3": "win_bo"
}
},
"System": {
"measurement": [
"Context Switches/sec",
"System Calls/sec",
"Processor Queue Length"
],
"append_dimensions": {
"d1": "win_foo",
"d2": "win_bar"
}
}
},
"append_dimensions": {
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}",
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
},
"aggregation_dimensions" : [["ImageId"], ["InstanceId", "InstanceType"], ["d1"],[]]
}
}
```
## CloudWatch agent configuration file: Logs section
The `logs` section includes the following fields:
+ `service.name` – Optional. Specifies the service name to be used to populate the entity for [finding related telemetry](ExploreRelated.md).
+ `deployment.environment` – Optional. Specifies the environment name to be used to populate the entity for [finding related telemetry](ExploreRelated.md).
+ `backpressure_mode` – Optional. Specifies the behavior when the CloudWatch agent is ingesting logs faster than it can send to CloudWatch Logs, resulting in backpressure. Backpressure can happen from network issues, API throttling, or high log volume.
The agent supports the following values:
+ `fd_release` – Releases file descriptors for deleted files during backpressure conditions. This option can help prevent disk space exhaustion when external log rotation or cleanup processes remove files while the agent maintains open file descriptors. The `auto_removal` option takes precedence over the `backpressure_mode` option being set to `fd_release`. When `auto_removal` is enabled, the CloudWatch agent processes the file to completion without releasing the file descriptor.
**Important**
Using `fd_release` can result in the CloudWatch agent being unable to read log files to completion, causing log loss.
+ `concurrency` – Optional. Specifies the number of shared log publishers used to concurrently publish log files to CloudWatch Logs.
If you omit this field, each log file destination (log group, stream combination) has a single shared log publisher, which can lead to bottlenecks for large files or when writing multiple files to the same destination. Enabling concurrency can help with throughput.
+ `logs_collected` – Required if the `logs` section is included. Specifies which log files and Windows event logs are to be collected from the server. It can include two fields, `files` and `windows_events`.
+ `files` – Specifies which regular log files the CloudWatch agent is to collect. It contains one field, `collect_list`, which further defines these files.
+ `collect_list` – Required if `files` is included. Contains an array of entries, each of which specifies one log file to collect. Each of these entries can include the following fields:
+ `file_path` – Specifies the path of the log file to upload to CloudWatch Logs. Standard Unix glob matching rules are accepted, with the addition of `**` as a *super asterisk*. For example, specifying `/var/log/**.log` causes all `.log` files in the `/var/log` directory tree to be collected. For more examples, see [Glob Library](https://github.com/gobwas/glob).
You can also use the standard asterisk as a standard wildcard. For example, `/var/log/system.log*` matches files such as `system.log_1111`, `system.log_2222`, and so on in `/var/log`.
Only the latest file is pushed to CloudWatch Logs based on file modification time. We recommend that you use wildcards to specify a series of files of the same type, such as `access_log.2018-06-01-01` and `access_log.2018-06-01-02`, but not multiple kinds of files, such as `access_log_80` and `access_log_443`. To specify multiple kinds of files, add another log stream entry to the agent configuration file so that each kind of log file goes to a different log stream.
+ `auto_removal` – Optional. If this is `true`, the CloudWatch agent automatically deletes this log file after reading it and it has been rotated. Usually the log files are deleted after their entire contents are uploaded to CloudWatch Logs, but if the agent reaches the EOF (end of file) and also detects another newer log file that matches the same `file_path`, the agent deletes the OLD file, so you must make sure that you are done writing to the OLD file before creating the NEW file. The [RUST tracing library](https://docs.rs/tracing/latest/tracing/) has a known incompatibility because it will potentially create a NEW log file and then still attempt to write to the OLD log file.
The agent only removes complete files from logs that create multiple files, such as logs that create separate files for each date. If a log continuously writes to a single file, it is not removed.
If you already have a log file rotation or removal method in place, we recommend that you omit this field or set it to `false`.
If you omit this field, the default value of `false` is used.
+ `log_group_name` – Optional. Specifies what to use as the log group name in CloudWatch Logs.
We recommend that you use this field to specify a log group name to prevent confusion. If you omit `log_group_name`, the value of `file_path` up to the final dot is used as the log group name. For example, if the file path is `/tmp/TestLogFile.log.2017-07-11-14`, the log group name is `/tmp/TestLogFile.log`.
If you specify a log group name, you can use `{instance_id}`, `{hostname}`, `{local_hostname}`, and `{ip_address}` as variables within the name. `{hostname}` retrieves the hostname from the EC2 metadata, and `{local_hostname}` uses the hostname from the network configuration file.
If you use these variables to create many different log groups, keep in mind the limit of 1,000,000 log groups per Region per account.
Allowed characters include a–z, A–Z, 0–9, '\$1' (underscore), '-' (hyphen), '/' (forward slash), and '.' (period).
+ `log_group_class` – Optional. Specifies which log group class to use for the new log group. For more information about log group classes, see [ Log classes](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch_Logs_Log_Classes.html).
Valid values are `STANDARD` and `INFREQUENT_ACCESS`. If you omit this field, the default of `STANDARD` is used.
**Important**
After a log group is created, its class can't be changed.
+ `log_stream_name` – Optional. Specifies what to use as the log stream name in CloudWatch Logs. As part of the name, you can use `{instance_id}`, `{hostname}`, `{local_hostname}`, and `{ip_address}` as variables within the name. `{hostname}` retrieves the hostname from the EC2 metadata, and `{local_hostname}` uses the hostname from the network configuration file.
If you omit this field, the value of the `log_stream_name` parameter in the global `logs` section is used. If that is also omitted, the default value of `{instance_id}` is used.
If a log stream doesn't already exist, it's created automatically.
+ `retention_in_days` – Optional. Specifies the number of days to retain the log events in the specified log group.
+ If the agent is creating this log group now, and you omit this field, the retention of this new log group is set to never expire.
+ If this log group already exists and you specify this field, the new retention that you specify is used. If you omit this field for a log group that already exists, the log group's retention is not changed.
The CloudWatch agent wizard uses `-1` as the default value for this field when it is used to create the agent configuration file and you don't specify a value for log retention. This `-1` value set by the wizard specifies that the events in the log group will never expire. However, manually editing this value to `-1` has no effect.
Valid values are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, and 3653.
If you configure the agent to write multiple log streams to the same log group, specifying the `retention_in_days` in one place will set the log retention for the entire log group. If you specify `retention_in_days` for the same log group in multiple places, the retention is set if all of those values are equal. However, if different `retention_in_days` values are specified for the same log group in multiple places, the log retention will not be set and the agent will stop, returning an error.
**Note**
The agent's IAM role or IAM user must have the `logs:PutRetentionPolicy` for it to be able to set retention policies.
**Warning**
If you set `retention_in_days` for a log group that already exists, all logs in that log group that were published before the number of days that you specify are deleted. For example, setting it to 3 would cause all logs from 3 days ago and before to be deleted.
+ `filters` – Optional. Can contain an array of entries, each of which specifies a regular expression and a filter type to specify whether to publish or drop log entries that match the filter. If you omit this field, all logs in the log file are published to CloudWatch Logs. If you include this field, the agent processes each log message with all of the filters that you specify, and only the log events that pass all of the filters are published to CloudWatch Logs. The log entries that don’t pass all of the filters will still remain in the host's log file, but will not be sent to CloudWatch Logs.
Each entry in the filters array can include the following fields:
+ `type`– Denotes the type of filter. Valid values are `include` and `exclude`. With `include`, the log entry must match the expression to be published to CloudWatch Logs. With `exclude`, each log entry that matches the filter is not sent to CloudWatch Logs.
+ `expression`– A regular expression string that follows the [RE2 Syntax](https://github.com/google/re2/wiki/Syntax).
**Note**
The CloudWatch agent doesn't check the performance of any regular expression that you supply, or restrict the run time of the evaluation of the regular expressions. We recommend that you are careful not to write an expression that is expensive to evaluate. For more information about possible issues, see [Regular expression Denial of Service - ReDoS](https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS)
For example, the following excerpt of the CloudWatch agent configuration file publishes logs that are PUT and POST requests to CloudWatch Logs, but excluding logs that come from Firefox.
```
"collect_list": [
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/test.log",
"log_group_name": "test.log",
"log_stream_name": "test.log",
"filters": [
{
"type": "exclude",
"expression": "Firefox"
},
{
"type": "include",
"expression": "P(UT|OST)"
}
]
},
.....
]
```
**Note**
The order of the filters in the configuration file matters for performance. In the preceding example, the agent drops all the logs that match `Firefox` before it starts evaluating the second filter. To cause fewer log entries to be evaluated by more than one filter, put the filter that you expect to rule out more logs first in the configuration file.
+ `timezone` – Optional. Specifies the time zone to use when putting timestamps on log events. The valid values are `UTC` and `Local`. The default value is `Local`.
This parameter is ignored if you don't specify a value for `timestamp_format`.
+ `timestamp_format` – Optional. Specifies the timestamp format, using plaintext and special symbols that start with %. If you omit this field, the current time is used. If you use this field, you can use the symbols in the following list as part of the format.
**Note**
This parameter is not considered when the `file_path` is set to `amazon-cloudwatch-agent.log`
If a single log entry contains two time stamps that match the format, the first time stamp is used.
This list of symbols is different than the list used by the older CloudWatch Logs agent. For a summary of these differences, see [Timestamp differences between the CloudWatch agent and the earlier CloudWatch Logs agent](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-logs-timestamp-differences)
`%y`
Year without century as a zero-padded decimal number. For example, `19` to represent 2019.
`%Y`
Year with century as a decimal number. For example, `2019`.
`%b`
Month as the locale's abbreviated name
`%B`
Month as the locale's full name
`%m`
Month as a zero-padded decimal number
`%-m`
Month as a decimal number (not zero-padded)
`%d`
Day of the month as a zero-padded decimal number
`%-d`
Day of the month as a decimal number (not zero-padded)
`%A`
Full name of weekday, such as `Monday`
`%a`
Abbreviation of weekday, such as `Mon`
`%H`
Hour (in a 24-hour clock) as a zero-padded decimal number
`%I`
Hour (in a 12-hour clock) as a zero-padded decimal number
`%-I`
Hour (in a 12-hour clock) as a decimal number (not zero-padded)
`%p`
AM or PM
`%M`
Minutes as a zero-padded decimal number
`%-M`
Minutes as a decimal number (not zero-padded)
`%S`
Seconds as a zero-padded decimal number
`%-S`
Seconds as a decimal number (not zero padded)
`%f`
Fractional seconds as a decimal number (1-9 digits), zero-padded on the left.
`%Z`
Time zone, for example `PST`
`%z`
Time zone, expressed as the offset between the local time zone and UTC. For example, `-0700`. Only this format is supported. For example, `-07:00` isn't a valid format.
+ `multi_line_start_pattern` – Specifies the pattern for identifying the start of a log message. A log message is made of a line that matches the pattern and any subsequent lines that don't match the pattern.
If you omit this field, multi-line mode is disabled, and any line that begins with a non-whitespace character closes the previous log message and starts a new log message.
If you include this field, you can specify `{timestamp_format}` to use the same regular expression as your timestamp format. Otherwise, you can specify a different regular expression for CloudWatch Logs to use to determine the start lines of multi-line entries.
+ `encoding` – Specified the encoding of the log file so that it can be read correctly. If you specify an incorrect coding, there might be data loss because characters that can't be decoded are replaced with other characters.
The default value is `utf-8`. The following are all possible values:
`ascii, big5, euc-jp, euc-kr, gbk, gb18030, ibm866, iso2022-jp, iso8859-2, iso8859-3, iso8859-4, iso8859-5, iso8859-6, iso8859-7, iso8859-8, iso8859-8-i, iso8859-10, iso8859-13, iso8859-14, iso8859-15, iso8859-16, koi8-r, koi8-u, macintosh, shift_jis, utf-8, utf-16, utf-16le, UTF-16, UTF-16LE, windows-874, windows-1250, windows-1251, windows-1252, windows-1253, windows-1254, windows-1255, windows-1256, windows-1257, windows-1258, x-mac-cyrillic`
+ `service.name` – Optional. Specifies the service name to be used to populate the entity for [finding related telemetry](ExploreRelated.md).
+ `deployment.environment` – Optional. Specifies the environment name to be used to populate the entity for [finding related telemetry](ExploreRelated.md).
+ `trim_timestamp` – Optional. If this is true, the CloudWatch agent will remove the timestamp matched by `timestamp_format` from the line before sending it to CloudWatch Logs. The LogEvent will still contain the `timestamp` field.
If you omit this field, the default value of `false` is used.
+ The `windows_events` section specifies the type of Windows events to collect from servers running Windows Server. It includes the following fields:
+ `collect_list` – Required if `windows_events` is included. Specifies the types and levels of Windows events to be collected. Each log to be collected has an entry in this section, which can include the following fields:
+ `event_name` – Specifies the type of Windows events to log. This is equivalent to the Windows event log channel name: for example, `System`, `Security`, `Application`, and so on. This field is required for each type of Windows event to log.
**Note**
When CloudWatch retrieves messages from a Windows log channel, it looks up the log channel based on its `Full Name` property. Meanwhile, the Windows Event Viewer navigation pane displays the `Log Name` property of log channels. The `Full Name` and `Log Name` do not always match. To confirm the `Full Name` of a channel, right-click on it in the Windows Event viewer and open **Properties**.
+ `event_levels` – Optional. Specifies the levels of event to log. You must specify each level to log. Possible values include `INFORMATION`, `WARNING`, `ERROR`, `CRITICAL`, and `VERBOSE`. This field is optional for each type of Windows event to log. and can be used with other filtering option like `event_ids` and `filters`.
+ `event_ids` – Optional. Contains an array of Windows Event IDs to specify which events to collect from the Windows Event Log. When this field is excluded, all events from the specified event log are collected. When this field is included, the agent only collects events that match the specified Event IDs.
Each entry in the `event_ids` array should be a numeric Event ID value and can be used with other filtering options. See the third entry in the config sample below.
**Note**
Using `event_ids` for filtering is recommended over regex expressions when you need to filter by Event ID, as it provides better performance.
+ `filters` – Optional. Contains an array of entries. Each entry specifies a regular expression and a filter type to specify whether to publish or drop the log entries that match the filter. When the field is included, the agent processes each log message with all of the filters that you specify, and only the log events that pass all filters are published to CloudWatch Logs. The windows event logs that doesn’t pass all of the filters will be dropped and not sent to CloudWatch Logs. The filters section can also be used with other filtering mechanisms like event ids [4624, 4625] and system levels (Information, Error, or Critical) to effectively filter logs and push to CloudWatch.
Each entry in the filters array can include the following fields:
+ `type` – Specifies the type of filter. Valid values are `include` and `exclude`. With include, the windows events entry must match the expression to be published to CloudWatch Logs. With exclude, each windows event log entry that matches the filter is not sent to CloudWatch Logs.
+ `expression` – A regular expression string that follows the RE2 Syntax.
**Note**
The CloudWatch agent does not validate regular expressions that you provide. It also does not limit their evaluation time. Write your expressions carefully to avoid performance issues. For more information on security ricks, see [Regular expression Denial of Service - ReDoS ](https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS).
In the example agent configuration below:
For the first entry, the agent pushes logs that contain database failure messages, any authentication related activities, and all login events (both successful and failed attempts) to CloudWatch. Any log that doesn’t match this pattern is dropped.
In the second entry, initial filtering is done based on event ids for windows event subscription. The agent collects all logs that contain the string user, discarding logs that don't match these patterns. The agent then drops logs containing `successful` before sending the remaining logs to CloudWatch Logs. Every filter type is applied to each windows event log before sending to CloudWatch.
```
"collect_list": [
{
"event_name": "Application",
"log_group_name": "ApplicationEvents",
"log_stream_name": "ApplicationEvents",
"filters": [
{
"type": "include",
"expression": "Database.*failed|Authentication.*|login.*"
}
]
},
{
"event_name": "System",
"log_group_name": "SystemEvents",
"log_stream_name": "Logon-events",
"event_ids": [
4624,
4625
],
"filters": [
{
"type": "include",
"expression": ".*user.*"
},
{
"type": "exclude",
"expression": ".*successful.*"
}
]
}
.....
]
```
**Note**
The order of the filters in the configuration will impact the performance. In the second entry, the agent drops all the logs that does not match the user before it starts evaluating the second filter expression. For optimal performance, order filters from highest to lowest exclusion rate.
While you can filter out logs on event ids and system levels in the filter expression, it is recommended to use the `event_ids` and `log_level` as shown in the second entry for improved performance.
**Warning**
Even though all filtering mechanisms (event\$1levels, event\$1ids, filters) are optional, at least one is required during agent configuration to filter logs.
+ `log_group_name` – Required. Specifies what to use as the log group name in CloudWatch Logs.
+ `log_stream_name` – Optional. Specifies what to use as the log stream name in CloudWatch Logs. As part of the name, you can use `{instance_id}`, `{hostname}`, `{local_hostname}`, and `{ip_address}` as variables within the name. `{hostname}` retrieves the hostname from the EC2 metadata, and `{local_hostname}` uses the hostname from the network configuration file.
If you omit this field, the value of the `log_stream_name` parameter in the global `logs` section is used. If that is also omitted, the default value of `{instance_id}` is used.
If a log stream doesn't already exist, it's created automatically.
+ `event_format` – Optional. Specifies the format to use when storing Windows events in CloudWatch Logs. `xml` uses the XML format as in Windows Event Viewer. `text` uses the legacy CloudWatch Logs agent format.
+ `retention_in_days` – Optional. Specifies the number of days to retain the Windows events in the specified log group.
+ If the agent is creating this log group now, and you omit this field, the retention of this new log group is set to never expire.
+ If this log group already exists and you specify this field, the new retention that you specify is used. If you omit this field for a log group that already exists, the log group's retention is not changed.
The CloudWatch agent wizard uses `-1` as the default value for this field when it is used to create the agent configuration file and you don't specify a value for log retention. This `-1` value specifies set by the wizard specifies that the events in the log group don't expire. However, manually editing this value to `-1` has no effect.
Valid values are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, and 3653.
If you configure the agent to write multiple log streams to the same log group, specifying the `retention_in_days` in one place will set the log retention for the entire log group. If you specify `retention_in_days` for the same log group in multiple places, the retention is set if all of those values are equal. However, if different `retention_in_days` values are specified for the same log group in multiple places, the log retention will not be set and the agent will stop, returning an error.
**Note**
The agent's IAM role or IAM user must have the `logs:PutRetentionPolicy` for it to be able to set retention policies.
**Warning**
If you set `retention_in_days` for a log group that already exists, all logs in that log group that were published before the number of days that you specify are deleted. For example, setting it to 3 would cause all logs from 3 days ago and before to be deleted.
+ `log_stream_name` – Optional. Specifies the default log stream name to be used for any logs or Windows events that don't have individual log stream names defined in the `log_stream_name` parameter within their entry in `collect_list`.
+ `endpoint_override` – Specifies a FIPS endpoint or private link to use as the endpoint where the agent sends logs. Specifying this field and setting a private link enables you to send the logs to an Amazon VPC endpoint. For more information, see [What Is Amazon VPC?](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html).
The value of `endpoint_override` must be a string that is a URL.
For example, the following part of the logs section of the configuration file sets the agent to use a VPC Endpoint when sending logs.
```
{
"logs": {
"endpoint_override": "vpce-XXXXXXXXXXXXXXXXXXXXXXXXX.logs.us-east-1.vpce.amazonaws.com",
......
},
}
```
+ `force_flush_interval` – Specifies in seconds the maximum amount of time that logs remain in the memory buffer before being sent to the server. No matter the setting for this field, if the size of the logs in the buffer reaches 1 MB, the logs are immediately sent to the server. The default value is 5.
If you are using the agent to report high-resolution metrics in embedded metric format, and you are setting alarms on those metrics, keep this parameter set to the default value of 5. Otherwise, the metrics are reported with a delay that can cause alarming on partial or incomplete data.
+ `credentials` – Specifies an IAM role to use when sending logs to a different AWS account. If specified, this field contains one parameter, `role_arn`.
+ `role_arn` – Specifies the ARN of an IAM role to use for authentication when sending logs to a different AWS account. For more information, see [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account). If specified here, this overrides the `role_arn` specified in the `agent` section of the configuration file, if any.
+ `metrics_collected` – This field can contain sections to specify that the agent is to collect logs to enable use cases such as CloudWatch Application Signals and Container Insights with enhanced observability for Amazon EKS.
+ `application_signals` (Optional) Specifies that you want to enable [CloudWatch Application Signals](CloudWatch-Application-Monitoring-Sections.md) For more information about this configuration, see [Enable CloudWatch Application Signals](CloudWatch-Agent-Application_Signals.md).
+ `kubernetes` – This field can contain an `enhanced_container_insights` parameter, which you can use to enable Container Insights with enhanced observability for Amazon EKS.
+ `enhanced_container_insights` – Set this to `true` to enable Container Insights with enhanced observability for Amazon EKS. For more information, see [Container Insights with enhanced observability for Amazon EKS](container-insights-detailed-metrics.md).
+ `accelerated_compute_metrics` – Set this to `false` to opt out of collecting Nvidia GPU metrics on Amazon EKS clusters. For more information, see [NVIDIA GPU metrics](Container-Insights-metrics-enhanced-EKS.md#Container-Insights-metrics-EKS-GPU).
+ `emf` – To collect metrics embedded in logs, it is no longer necessary to add this `emf` field. This is a legacy field that specified that the agent is to collect logs that are in embedded metric format. You can generate metric data from these logs. For more information, see [Embedding metrics within logs](CloudWatch_Embedded_Metric_Format.md).
+ `otlp` – Optional. Specifies that you want to collect metrics from the OpenTelemetry SDK. For more information about the fields that you can use in this section, see [Collect metrics and traces with OpenTelemetry](CloudWatch-Agent-OpenTelemetry-metrics.md).
The following is an example of a `logs` section.
```
"logs":{
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log",
"log_group_name": "amazon-cloudwatch-agent.log",
"log_stream_name": "my_log_stream_name_1"
},
{
"file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\test.log",
"log_group_name": "test.log",
"log_stream_name": "my_log_stream_name_2"
}
]
},
"windows_events": {
"collect_list": [
{
"event_name": "System",
"event_ids": [
1001,
1008
],
"log_group_name": "System",
"log_stream_name": "System"
},
{
"event_name": "CustomizedName",
"event_levels": [
"INFORMATION",
"ERROR"
],
"log_group_name": "CustomizedLogGroup",
"log_stream_name": "CustomizedLogStream"
},
{
"event_name": "Application",
"event_levels": [
"INFORMATION",
"ERROR"
],
"event_ids":[
7369,
5624
],
"log_group_name": "CustomizedLogGroup",
"log_stream_name": "CustomizedLogStream"
}
]
}
},
"log_stream_name": "my_log_stream_name",
"metrics_collected": {
"kubernetes": {
"enhanced_container_insights": true
}
}
}
```
## CloudWatch agent configuration file: Traces section
By adding a `traces` section to the CloudWatch agent configuration file, you can enable CloudWatch Application Signals or collect traces from X-Ray and from the OpenTelemetry instrumentation SDK and send them to X-Ray.
**Important**
The agent's IAM role or IAM user must have the **AWSXrayWriteOnlyAccess** policy to send trace data to X-Ray.
For a quick start for collecting traces, you can add just the following to the CloudWatch agent configuration file.
```
"traces_collected": {
"xray": {
},
"otlp": {
}
}
```
If you add the previous section to the CloudWatch agent configuration file and restart the agent, this causes the agent to start collecting traces using the following default options and values. For more information about these parameters, see the parameter definitions later in this section.
```
"traces_collected": {
"xray": {
"bind_address": "127.0.0.1:2000",
"tcp_proxy": {
"bind_address": "127.0.0.1:2000"
}
},
"otlp": {
"grpc_endpoint": "127.0.0.1:4317",
"http_endpoint": "127.0.0.1:4318"
}
}
```
The `traces` section can include the following fields:
+ `traces_collected` – Required if the `traces` section is included. Specifies which SDKs to collect traces from. It can include the following fields:
+ `application_signals` – Optional. Specifies that you want to enable [CloudWatch Application Signals](CloudWatch-Application-Monitoring-Sections.md) For more information about this configuration, see [Enable CloudWatch Application Signals](CloudWatch-Agent-Application_Signals.md).
+ `xray` – Optional. Specifies that you want to collect traces from the X-Ray SDK. This section can include the following fields:
+ `bind_address` – Optional. Specifies the UDP address for the CloudWatch agent to use to listen for X-Ray traces. The format is `ip:port`. This address must match the address set in the X-Ray SDK.
If you omit this field, the default of `127.0.0.1:2000` is used.
+ `tcp_proxy` – Optional. Configures the address for a proxy used to support X-Ray remote sampling. For more information, see [ Configuring sampling rules](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html) in the X-Ray documentation.
This section can contain the following field.
+ `bind_address` – Optional. Specifies the TCP address to which the CloudWatch agent should set up the proxy. The format is `ip:port`. This address must match the address set in the X-Ray SDK.
If you omit this field, the default of `127.0.0.1:2000` is used.
+ `otlp` – Optional. Specifies that you want to collect traces from the OpenTelemetry SDK. For more information about the fields that you can use in this section, see [Collect metrics and traces with OpenTelemetry](CloudWatch-Agent-OpenTelemetry-metrics.md)). For more information about the AWS Distro for OpenTelemetry, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/). For more information about the AWS Distro for OpenTelemetry SDKs, see [Introduction](https://aws-otel.github.io/docs/introduction).
This section can include the following fields:
+ `grpc_endpoint` – Optional. Specifies the address for the CloudWatch agent to use to listen for OpenTelemetry traces sent using gRPC Remote Procedure Calls. The format is `ip:port`. This address must match the address set for the gRPC exporter in the OpenTelemetry SDK.
If you omit this field, the default of `127.0.0.1:4317` is used.
+ `http_endpoint` – Optional. Specifies the address for the CloudWatch agent to use to listen for OTLP traces sent over HTTP. The format is `ip:port`. This address must match the address set for the HTTP exporter in the OpenTelemetry SDK.
If you omit this field, the default of `127.0.0.1:4318` is used.
+ `concurrency` – Optional. Specifies the maximum number of concurrent calls to X-Ray that can be used to upload traces. The default value is `8`
+ `local_mode` – Optional. If `true`, the agent doesn't collect Amazon EC2 instance metadata. The default is `false`
+ `endpoint_override` – Optional. Specifies a FIPS endpoint or private link to use as the endpoint where the CloudWatch agent sends traces. Specifying this field and setting a private link enables you to send the traces to an Amazon VPC endpoint. For more information, see [ What is Amazon VPC](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html)
The value of `endpoint_override` must be a string that is a URL.
+ `region_override` – Optional. Specifies the Region to use for the X-Ray endpoint. The CloudWatch agent sends the traces to X-Ray in the specified Region. If you omit this field, the agent sends the traces to the Region where the Amazon EC2 instance is located.
If you specify a Region here, it takes precedence over the setting of the `region` parameter in the `agent` section of the configuration file.
+ `proxy_override` – Optional. Specifies the proxy server address for the CloudWatch agent to use when sending requests to X-Ray. The proxy server's protocol must be specified as part of this address.
+ `credentials` – Specifies an IAM role to use when sending traces to a different AWS account. If specified, this field contains one parameter, `role_arn`.
+ `role_arn` – Specifies the ARN of an IAM role to use for authentication when sending traces to a different AWS account. For more information, see [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account). If specified here, this overrides the `role_arn` specified in the `agent` section of the configuration file, if any.
+ `transit_spans_in_otlp_format` – Optional. If `true`, sends traces to X-Ray in the OpenTelemetry Protocol format, which supports span events in Transaction Search. For more information, see [Adding custom attributes](CloudWatch-Transaction-Search-add-custom-attributes.md). The default is `false`.
## CloudWatch agent configuration file: Complete examples
The following is an example of a complete CloudWatch agent configuration file for a Linux server.
The items listed in the `measurement` sections for the metrics you want to collect can either specify the complete metric name such or just the part of the metric name that will be appended to the type of resource. For example, specifying either `reads` or `diskio_reads` in the `measurement` section of the `diskio` section will cause the `diskio_reads` metric to be collected.
This example includes both ways of specifying metrics in the `measurement` section.
```
{
"agent": {
"metrics_collection_interval": 10,
"logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
},
"metrics": {
"namespace": "MyCustomNamespace",
"metrics_collected": {
"cpu": {
"resources": [
"*"
],
"measurement": [
{"name": "cpu_usage_idle", "rename": "CPU_USAGE_IDLE", "unit": "Percent"},
{"name": "cpu_usage_nice", "unit": "Percent"},
"cpu_usage_guest"
],
"totalcpu": false,
"metrics_collection_interval": 10,
"append_dimensions": {
"customized_dimension_key_1": "customized_dimension_value_1",
"customized_dimension_key_2": "customized_dimension_value_2"
}
},
"disk": {
"resources": [
"/",
"/tmp"
],
"measurement": [
{"name": "free", "rename": "DISK_FREE", "unit": "Gigabytes"},
"total",
"used"
],
"ignore_file_system_types": [
"sysfs", "devtmpfs"
],
"metrics_collection_interval": 60,
"append_dimensions": {
"customized_dimension_key_3": "customized_dimension_value_3",
"customized_dimension_key_4": "customized_dimension_value_4"
}
},
"diskio": {
"resources": [
"*"
],
"measurement": [
"reads",
"writes",
"read_time",
"write_time",
"io_time"
],
"metrics_collection_interval": 60
},
"swap": {
"measurement": [
"swap_used",
"swap_free",
"swap_used_percent"
]
},
"mem": {
"measurement": [
"mem_used",
"mem_cached",
"mem_total"
],
"metrics_collection_interval": 1
},
"net": {
"resources": [
"eth0"
],
"measurement": [
"bytes_sent",
"bytes_recv",
"drop_in",
"drop_out"
]
},
"netstat": {
"measurement": [
"tcp_established",
"tcp_syn_sent",
"tcp_close"
],
"metrics_collection_interval": 60
},
"processes": {
"measurement": [
"running",
"sleeping",
"dead"
]
}
},
"append_dimensions": {
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}",
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
},
"aggregation_dimensions" : [["ImageId"], ["InstanceId", "InstanceType"], ["d1"],[]],
"force_flush_interval" : 30
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
"log_group_name": "amazon-cloudwatch-agent.log",
"log_stream_name": "amazon-cloudwatch-agent.log",
"timezone": "UTC"
},
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/test.log",
"log_group_name": "test.log",
"log_stream_name": "test.log",
"timezone": "Local"
}
]
}
},
"log_stream_name": "my_log_stream_name",
"force_flush_interval" : 15,
"metrics_collected": {
"kubernetes": {
"enhanced_container_insights": true
}
}
}
}
```
The following is an example of a complete CloudWatch agent configuration file for a server running Windows Server.
```
{
"agent": {
"metrics_collection_interval": 60,
"logfile": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log"
},
"metrics": {
"namespace": "MyCustomNamespace",
"metrics_collected": {
"Processor": {
"measurement": [
{"name": "% Idle Time", "rename": "CPU_IDLE", "unit": "Percent"},
"% Interrupt Time",
"% User Time",
"% Processor Time"
],
"resources": [
"*"
],
"append_dimensions": {
"customized_dimension_key_1": "customized_dimension_value_1",
"customized_dimension_key_2": "customized_dimension_value_2"
}
},
"LogicalDisk": {
"measurement": [
{"name": "% Idle Time", "unit": "Percent"},
{"name": "% Disk Read Time", "rename": "DISK_READ"},
"% Disk Write Time"
],
"resources": [
"*"
]
},
"customizedObjectName": {
"metrics_collection_interval": 60,
"customizedCounterName": [
"metric1",
"metric2"
],
"resources": [
"customizedInstances"
]
},
"Memory": {
"metrics_collection_interval": 5,
"measurement": [
"Available Bytes",
"Cache Faults/sec",
"Page Faults/sec",
"Pages/sec"
]
},
"Network Interface": {
"metrics_collection_interval": 5,
"measurement": [
"Bytes Received/sec",
"Bytes Sent/sec",
"Packets Received/sec",
"Packets Sent/sec"
],
"resources": [
"*"
],
"append_dimensions": {
"customized_dimension_key_3": "customized_dimension_value_3"
}
},
"System": {
"measurement": [
"Context Switches/sec",
"System Calls/sec",
"Processor Queue Length"
]
}
},
"append_dimensions": {
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}",
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
},
"aggregation_dimensions" : [["ImageId"], ["InstanceId", "InstanceType"], ["d1"],[]]
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log",
"log_group_name": "amazon-cloudwatch-agent.log",
"timezone": "UTC"
},
{
"file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\test.log",
"log_group_name": "test.log",
"timezone": "Local"
}
]
},
"windows_events": {
"collect_list": [
{
"event_name": "System",
"event_levels": [
"INFORMATION",
"ERROR"
],
"log_group_name": "System",
"log_stream_name": "System",
"event_format": "xml"
},
{
"event_name": "CustomizedName",
"event_levels": [
"WARNING",
"ERROR"
],
"log_group_name": "CustomizedLogGroup",
"log_stream_name": "CustomizedLogStream",
"event_format": "xml"
}
]
}
},
"log_stream_name": "example_log_stream_name"
}
}
```
## Save the CloudWatch agent configuration file manually
If you create or edit the CloudWatch agent configuration file manually, you can give it any name. After you have created the file, you can copy it to other servers where you want to run the agent.
## Uploading the CloudWatch agent configuration file to Systems Manager Parameter Store
If you plan to use the SSM Agent to install the CloudWatch agent on servers, after you manually edit the CloudWatch agent configuration file, you can upload it to Systems Manager Parameter Store. To do so, use the Systems Manager `put-parameter` command.
To be able to store the file in Parameter Store, you must use an IAM role with sufficient permissions.
Use the following command, where *parameter name* is the name to be used for this file in Parameter Store and *configuration\$1file\$1pathname* is the path and file name of the configuration file that you edited.
```
aws ssm put-parameter --name "parameter name" --type "String" --value file://configuration_file_pathname
```
# Enable CloudWatch Application Signals
Use CloudWatch Application Signals to automatically instrument your applications on AWS so that you can track application performance against your business objectives. Application Signals provides you a unified, application-centric view of your Java applications, their dependencies, and their edges. For more information, see [Application Signals](CloudWatch-Application-Monitoring-Sections.md).
CloudWatch Application Signals leverages the CloudWatch agent to receive metrics and traces from your auto-instrumented applications, optionally apply rules to reduce high cardinality, and then publish the processed telemetry to CloudWatch. You can provide custom configuration to the CloudWatch agent specifically for Application Signals using the agent configuration file. To start with, the presence of an `application_signals` section under the `metrics_collected` section within the `logs` section of the agent configuration file specifies that the CloudWatch agent will receive metrics from your auto-instrumented applications. Similarly, the presence of an `application_signals` section under the `traces_collected` section within the `traces` section of the agent configuration file specifies that the CloudWatch agent is enabled to receive traces from your auto-instrumented applications. In addition, you can optionally pass in custom configuration rules to reduce publishing high-cardinality telemetry as outlined in this section.
+ For Amazon EKS clusters, when you install the [Amazon CloudWatch Observability](install-CloudWatch-Observability-EKS-addon.md) EKS add-on, the CloudWatch agent is by default enabled to receive both metrics and traces from your auto-instrumented applications. If you would like to optionally pass in custom configuration rules, you can do so by passing in a custom agent configuration to the Amazon EKS add-on when you create or update it by using additional configuration, as outlined in [(Optional) Additional configuration](install-CloudWatch-Observability-EKS-addon.md#install-CloudWatch-Observability-EKS-addon-configuration).
+ For RedHat for OpenShift on AWS (ROSA), when you install the CloudWatch agent operator using helm charts, the CloudWatch agent is by default enabled to receive both metrics and traces from your auto-instrumented applications. If you would like to optionally pass in custom configuration rules, you can do so by passing in a custom agent configuration by using the Helm chart, as outlined in [(Optional) Additional configuration](install-CloudWatch-Observability-EKS-addon.md#install-CloudWatch-Observability-EKS-addon-configuration).
+ For other supported platforms including Amazon EC2, you must start the CloudWatch agent with an agent configuration that enables Application Signals by specifying the `application_signals` sections and optionally any custom configuration rules as outlined later in this section.
The following is an overview of the fields in the CloudWatch agent configuration file that are related to CloudWatch Application Signals.
+ `logs`
+ `metrics_collected` – This field can contain sections to specify that the agent is to collect logs to enable use cases such as CloudWatch Application Signals and Container Insights with enhanced observability for Amazon EKS.
**Note**
Previously this section was also used to specify that the agent is to collect logs that are in embedded metric format. Those settings are no longer needed.
+ `application_signals` (Optional) Specifies that you want to enable CloudWatch Application Signals to receive metrics from your auto-instrumented applications to facilitate CloudWatch Application Signals.
+ `rules` (Optional) An array of rules to conditionally select metrics and traces and apply actions to handle high-cardinality scenarios. Each rule can contain the following fields:
+ `rule_name` (Optional) The name of the rule.
+ `selectors` (Optional) An array of metrics and traces dimension matchers. Each selector must provide the following fields:
+ `dimension` Required if `selectors` is not empty. This specifies the dimension of metrics and traces to use as a filter.
+ `match` Required if `selectors` is not empty. A wildcard pattern used for matching values of the specified dimension.
+ `action` (Optional) The action to be applied to metrics and traces that match the specified selectors. The value of `action` must be one of the following keywords:
+ `keep` Specifies to send only the metrics and traces to CloudWatch if matched by the `selectors`.
+ `drop` Specifies to drop the metric and traces that match the `selectors`.
+ `replace` Specifies to replace the dimensions of the metrics and traces that match `selectors`. They are replaced according to the `replacements` section.
+ `replacements` Required if `action` is `replace`. An array of dimension and value pairs that will be applied to metrics and traces that match the specified `selectors` when the `action` is `replace`. Each replacement must provide the following fields:
+ `target_dimension` Required if `replacements` is not empty. Specifies the dimension that needs to be replace.
+ `value` Required if `replacements` is not empty. The value to replace the original value of `target_dimension` with.
+ `limiter` (Optional) Use this section to limit how many metrics and dimensions Application Signals sends to CloudWatch, to optimize your costs.
+ `disabled` (Optional) If `true`, the metric limiting feature is disabled. The default is `false`
+ `drop_threshold` (Optional) The maximum number of distinct metrics per service in one rotation interval that can be exported by one CloudWatch agent. The default is 500.
+ `rotation_interval` (Optional) The interval at which the limiter resets the metric records for distinction counting. This is expressed as a string with a sequence of numbers and a unit suffix. Fractions are supported. The supported unit suffixes are `s`, `m`, `h`, `ms`, `us`, and `ns`. The default is `1h` for one hour.
+ `log_dropped_metrics` (Optional) Specifies whether the agent should write logs to the CloudWatch agent logs when Application Signals metrics are dropped. The default is `false`.
**Note**
To activate this logging, the `debug` parameter in the `agent` section must also be set to `true`.
+ `traces`
+ `traces_collected`
+ `application_signals` Optional. Specify this to enable the CloudWatch agent to receive traces from your auto-instrumented applications for facilitating CloudWatch Application Signals.
**Note**
Even though the custom `application_signals` rules are specified under the `metrics_collected` section that is contained in the `logs` section, they also implicitly apply to the `traces_collected` section as well. The same set of rules will apply to both metrics and traces.
When there are multiple rules with different actions, they apply in the following sequence: `keep`, then `drop`, then `replace`.
The following is an example of a full CloudWatch agent configuration file that applies custom rules.
```
{
"logs": {
"metrics_collected": {
"application_signals": {
"rules": [
{
"rule_name": "keep01",
"selectors": [
{
"dimension": "Service",
"match": "pet-clinic-frontend"
},
{
"dimension": "RemoteService",
"match": "customers-service"
}
],
"action": "keep"
},
{
"rule_name": "drop01",
"selectors": [
{
"dimension": "Operation",
"match": "GET /api/customer/owners/*"
}
],
"action": "drop"
},
{
"rule_name": "replace01",
"selectors": [
{
"dimension": "Operation",
"match": "PUT /api/customer/owners/*/pets/*"
},
{
"dimension": "RemoteOperation",
"match": "PUT /owners"
}
],
"replacements": [
{
"target_dimension": "Operation",
"value": "PUT /api/customer/owners/{ownerId}/pets{petId}"
}
],
"action": "replace"
}
]
}
}
},
"traces": {
"traces_collected": {
"application_signals": {}
}
}
}
```
For the previous example configuration file, the `rules` are processed as follows:
1. Rule `keep01` ensures that any metrics and traces with the dimension `Service` as `pet-clinic-frontend` and the dimension `RemoteService` as `customers-service` are kept.
1. For the processed metrics and traces after applying `keep01`, the `drop01` rule ensures that metrics and traces with the dimension `Operation` as `GET /api/customer/owners/*` are dropped.
1. For the processed metrics and traces after applying `drop01`, the `replace01` rule updates metrics and traces that have the dimension `Operation` as `PUT /api/customer/owners/*/pets/*` and the dimension `RemoteOperation` as `PUT /owners` such that their `Operation` dimension is now replaced to be `PUT /api/customer/owners/{ownerId}/pets{petId}`.
The following is a complete example of a CloudWatch configuration file that manages cardinality in Application Signals by changing the metric limit to 100, enabling the logging of dropped metrics, and setting the rotation interval to two hours.
```
{
"logs": {
"metrics_collected": {
"application_signals": {
"limiter": {
"disabled": false,
"drop_threshold": 100,
"rotation_interval": "2h",
"log_dropped_metrics": true
}
}
},
"traces": {
"traces_collected": {
"application_signals": {}
}
}
}
}
```
# Collect network performance metrics
EC2 instances running on Linux that use the Elastic Network Adapter (ENA) publish network performance metrics. Version 1.246396.0 and later of the CloudWatch agent enable you to import these network performance metrics into CloudWatch. When you import these network performance metrics into CloudWatch, they are charged as CloudWatch custom metrics.
For more information about the ENA driver, see [ Enabling enhanced networking with the Elastic Network Adapter (ENA) on Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html) and [ Enabling enhanced networking with the Elastic Network Adapter (ENA) on Windows instances](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/enhanced-networking-ena.html).
How you set up the collection of network performance metrics differs on Linux servers and Windows servers.
The following table lists these network performance metrics enabled by the ENA adapter. When the CloudWatch agent imports these metrics into CloudWatch from Linux instances, it prepends `ethtool_` at the beginning of each of these metric names.
| Metric | Description |
| --- | --- |
| Name on Linux servers: `bw_in_allowance_exceeded` Name on Windows servers: `Aggregate inbound BW allowance exceeded` | The number of packets queued and/or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None |
| Name on Linux servers: `bw_out_allowance_exceeded` Name on Windows servers: `Aggregate outbound BW allowance exceeded` | The number of packets queued and/or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None |
| Name on Linux servers:`conntrack_allowance_available` Name on Windows servers: `Available connection tracking allowance` | Reports the number of tracked connections that can be established by the instance before hitting the Connections Tracked allowance of that instance type. This metric is available only on Nitro-based EC2 instances using the Linux driver for Elastic Network Adapter (ENA) starting from version 2.8.1, and on computers using the Windows driver for Elastic Network Adapter (ENA) starting from version 2.6.0. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None |
| Name on Linux servers:`ena_srd_mode` Name on Windows servers: `ena srd mode` | Describes which ENA Express features are enabled. For more information about ENA Express, see [ Improve network performance with ENA Express on Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-express.html) Values are as follows: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-network-performance.html) |
| Name on Linux servers:`ena_srd_eligible_tx_pkts` Name on Windows servers: `ena srd eligible tx pkts` | The number of network packets sent within a given time period that meet AWS Scalable Reliable Datagram (SRD) requirements for eligibility, as follows: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-network-performance.html) |
| Name on Linux servers:`ena_srd_tx_pkts` Name on Windows servers: `ena srd tx pkts` | The number of SRD packets transmitted within a given time period. |
| Name on Linux servers:`ena_srd_rx_pkts` Name on Windows servers: `ena srd rx pkts` | The number of SRD packets received within a given time period. |
| Name on Linux servers:`ena_srd_resource_utilization` Name on Windows servers: `ena srd resource utilization` | The percentage of the maximum allowed memory utilization for concurrent SRD connections that the instance has consumed. |
| Name on Linux servers: `linklocal_allowance_exceeded` Name on Windows servers: `Link local packet rate allowance exceeded` | The number of packets dropped because the PPS of the traffic to local proxy services exceeded the maximum for the network interface. This impacts traffic to the DNS service, the Instance Metadata Service, and the Amazon Time Sync Service, but does not impact traffic to custom DNS resolvers. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None |
| Name on Linux servers: `pps_allowance_exceeded` Name on Windows servers: `PPS allowance exceeded` | The number of packets queued and/or dropped because the bidirectional PPS exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None |
## Linux setup
On Linux servers, the *ethtool plugin* enables you to import the network performance metrics into CloudWatch.
ethtool is a standard Linux utility that can collect statistics about Ethernet devices on Linux servers. The statistics it collects depend on the network device and driver. Examples of these statistics include `tx_cnt`, `rx_bytes`, `tx_errors`, and `align_errors`. When you use the ethtool plugin with the CloudWatch agent, you can also import these statistics into CloudWatch, along with the EC2 network performance metrics listed earlier in this section.
**Tip**
To find the statistics available on our operating system and network device, use the `ethtool –S` command.
When the CloudWatch agent imports metrics into CloudWatch, it adds an `ethtool_` prefix to the names of all imported metrics. So the standard ethtool statistic `rx_bytes` is called `ethtool_rx_bytes` in CloudWatch, and the EC2 network performance metric `bw_in_allowance_exceeded` is called `ethtool_bw_in_allowance_exceeded` in CloudWatch.
On Linux servers, to import ethtool metrics, add an `ethtool` section to the `metrics_collected` section of the CloudWatch agent configuration file. The `ethtool` section can include the following subsections:
+ **interface\$1include**— Including this section causes the agent to collect metrics from only the interfaces that have names listed in this section. If you omit this section, metrics are collected from all Ethernet interfaces that aren't listed in `interface_exclude`.
The default ethernet interface is `eth0`.
+ **interface\$1exclude**— If you include this section, list the Ethernet interfaces that you don't want to collect metrics from.
The ethtool plugin always ignores loopback interfaces.
+ **metrics\$1include**— This section lists the metrics to import into CloudWatch. It can include both standard statistics collected by ethtool and Amazon EC2 high-resolution network metrics.
The following example displays part of the CloudWatch agent configuration file. This configuration collects the standard ethtool metrics `rx_packets` and `tx_packets`, and the Amazon EC2 network performance metrics from only the `eth1` interface.
For more information about the CloudWatch agent configuration file, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md).
```
{
"metrics": {
"append_dimensions": {
"InstanceId": "${aws:InstanceId}"
},
"metrics_collected": {
"ethtool": {
"interface_include": [
"eth1"
],
"metrics_include": [
"bw_in_allowance_exceeded",
"bw_out_allowance_exceeded",
"conntrack_allowance_exceeded",
"linklocal_allowance_exceeded",
"pps_allowance_exceeded"
]
}
}
}
}
```
## Windows setup
On Windows servers, the network performance metrics are available through Windows Performance Counters, which the CloudWatch agent already collects metrics from. So you do not need a plugin to collect these metrics from Windows servers.
The following is a sample configuration file to collect network performance metrics from Windows. For more information about editing the CloudWatch agent configuration file, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md).
```
{
"metrics": {
"append_dimensions": {
"InstanceId": "${aws:InstanceId}"
},
"metrics_collected": {
"ENA Packets Shaping": {
"measurement": [
"Aggregate inbound BW allowance exceeded",
"Aggregate outbound BW allowance exceeded",
"Connection tracking allowance exceeded",
"Link local packet rate allowance exceeded",
"PPS allowance exceeded"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
}
}
}
}
```
## Viewing network performance metrics
After importing network performance metrics into CloudWatch, you can view these metrics as time series graphs, and create alarms that can watch these metrics and notify you if they breach a threshold that you specify. The following procedure shows how to view ethtool metrics as a time series graph. For more information about setting alarms, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md).
Because all of these metrics are aggregate counters, you can use CloudWatch metric math functions such as `RATE(METRICS())` to calculate the rate for these metrics in graphs or use them to set alarms. For more information about metric math functions, see [Using math expressions with CloudWatch metrics](using-metric-math.md).
**To view network performance metrics in the CloudWatch console**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**.
1. Choose the namespace for the metrics collected by the agent. By default, this is **CWAgent**, but you may have specified a different namespace in the CloudWatch agent configuration file.
1. Choose a metric dimension (for example, **Per-Instance Metrics**).
1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following:
1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table.
1. To sort the table, use the column heading.
1. To filter by resource, choose the resource ID, and then choose **Add to search**.
1. To filter by metric, choose the metric name, and then choose **Add to search**.
1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, and then choose **Add to dashboard**.
# Collect Amazon EBS NVMe driver metrics
For CloudWatch agent to collect AWS NVMe driver metrics for Amazon EBS volumes attached to an Amazon EC2 instance, add the `diskio` section inside the `metrics_collected` section of the CloudWatch agent configuration file.
Additionally, the CloudWatch agent binary requires `ioctl` permissions for NVMe driver devices to collect metrics from attached Amazon EBS volumes.
The following metrics can be collected.
| Metric | Metric name in CloudWatch | Description |
| --- | --- | --- |
| `ebs_total_read_ops` | `diskio_ebs_total_read_ops` | The total number of completed read operations. |
| `ebs_total_write_ops` | `diskio_ebs_total_write_ops` | The total number of completed write operations. |
| `ebs_total_read_bytes` | `diskio_ebs_total_read_bytes` | The total number of read bytes transferred. |
| `ebs_total_write_bytes` | `diskio_ebs_total_write_bytes` | The total number of write bytes transferred. |
| `ebs_total_read_time` | `diskio_ebs_total_read_time` | The total time spent, in microseconds, by all completed read operations. |
| `ebs_total_write_time` | `diskio_ebs_total_write_time` | The total time spent, in microseconds, by all completed write operations. |
| `ebs_volume_performance_exceeded_iops` | `diskio_ebs_volume_performance_exceeded_iops` | The total time, in microseconds, that IOPS demand exceeded the volume's provisioned IOPS performance. |
| `ebs_volume_performance_exceeded_tp` | `diskio_ebs_volume_performance_exceeded_tp` | The total time, in microseconds, that throughput demand exceeded the volume's provisioned throughput performance. |
| `ebs_ec2_instance_performance_exceeded_iops` | `diskio_ebs_ec2_instance_performance_exceeded_iops` | The total time, in microseconds, that the EBS volume exceeded the attached Amazon EC2 instance's maximum IOPS performance. |
| `ebs_ec2_instance_performance_exceeded_tp` | `diskio_ebs_ec2_instance_performance_exceeded_tp` | The total time, in microseconds, that the EBS volume exceeded the attached Amazon EC2 instance's maximum throughput performance. |
| `ebs_volume_queue_length` | `diskio_ebs_volume_queue_length` | The number of read and write operations waiting to be completed. |
# Collect Amazon EC2 instance store volume NVMe driver metrics
For CloudWatch agent to collect AWS NVMe driver metrics for instance store volumes attached to an Amazon EC2 instance, add the `diskio` section inside the `metrics_collected` section of the CloudWatch agent configuration file.
Additionally, the CloudWatch agent binary requires `ioctl` permissions for NVMe driver devices to collect metrics from attached instance store volumes.
The following metrics can be collected.
| Metric | Metric name in CloudWatch | Description |
| --- | --- | --- |
| `instance_store_total_read_ops` | `diskio_instance_store_total_read_ops` | The total number of completed read operations. |
| `instance_store_total_write_ops` | `diskio_instance_store_total_write_ops` | The total number of completed write operations. |
| `instance_store_total_read_bytes` | `diskio_instance_store_total_read_bytes` | The total number of read bytes transferred. |
| `instance_store_total_write_bytes` | `diskio_instance_store_total_write_bytes` | The total number of write bytes transferred. |
| `instance_store_total_read_time` | `diskio_instance_store_total_read_time` | The total time spent, in microseconds, by all completed read operations. |
| `instance_store_total_write_time` | `diskio_instance_store_total_write_time` | The total time spent, in microseconds, by all completed write operations. |
| `instance_store_performance_exceeded_iops` | `diskio_instance_store_performance_exceeded_iops` | The total time, in microseconds, that IOPS demand exceeded the volume's IOPS maximum performance. |
| `instance_store_performance_exceeded_tp` | `diskio_instance_store_performance_exceeded_tp` | The total time, in microseconds, that throughput demand exceeded the volume's maximum throughput performance. |
| `instance_store_volume_queue_length` | `diskio_instance_store_volume_queue_length` | The number of read and write operations waiting to be completed. |
# Collect NVIDIA GPU metrics
You can use the CloudWatch agent to collect NVIDIA GPU metrics from Linux servers. To set this up, add a `nvidia_gpu` section inside the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Linux section](CloudWatch-Agent-Configuration-File-Details.md#CloudWatch-Agent-Linux-section).
Additionally, the instance must have an NVIDIA driver installed. NVIDIA drivers on pre-installed on some Amazon Machine Images (AMIs). Otherwise, you can manually install the driver. For more information, see [ Install NVIDIA drivers on Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-nvidia-driver.html).
The following metrics can be collected. All of these metrics are collected with no CloudWatch `Unit`, but you can specify a unit for each metric by adding a parameter to the CloudWatch agent configuration file. For more information, see [Linux section](CloudWatch-Agent-Configuration-File-Details.md#CloudWatch-Agent-Linux-section).
| Metric | Metric name in CloudWatch | Description |
| --- | --- | --- |
| `utilization_gpu` | `nvidia_smi_utilization_gpu` | The percentage of time over the past sample period during which one or more kernals on the GPU was running. |
| `temperature_gpu` | `nvidia_smi_temperature_gpu` | The core GPU temperature in degrees Celsius. |
| `power_draw` | `nvidia_smi_power_draw` | The last measured power draw for the entire board, in watts. |
| `utilization_memory` | `nvidia_smi_utilization_memory` | The percentage of time over the past sample period during which global (device) memory was being read or written. |
| `fan_speed` | `nvidia_smi_fan_speed` | The percentage of maximum fan speed that the device's fan is currently intended to run at. |
| `memory_total` | `nvidia_smi_memory_total` | Reported total memory, in MB. |
| `memory_used` | `nvidia_smi_memory_used` | Memory used, in MB. |
| `memory_free` | `nvidia_smi_memory_free` | Memory free, in MB. |
| `pcie_link_gen_current` | `nvidia_smi_pcie_link_gen_current` | The current link generation. |
| `pcie_link_width_current` | `nvidia_smi_pcie_link_width_current` | The current link width. |
| `encoder_stats_session_count` | `nvidia_smi_encoder_stats_session_count` | Current number of encoder sessions. |
| `encoder_stats_average_fps` | `nvidia_smi_encoder_stats_average_fps` | The moving average of the encode frames per second. |
| `encoder_stats_average_latency` | `nvidia_smi_encoder_stats_average_latency` | The moving average of the encode latency in microseconds. |
| `clocks_current_graphics` | `nvidia_smi_clocks_current_graphics` | The current frequency of the graphics (shader) clock. |
| `clocks_current_sm` | `nvidia_smi_clocks_current_sm` | The current frequency of the Streaming Multiprocessor (SM) clock. |
| `clocks_current_memory` | `nvidia_smi_clocks_current_memory` | The current frequency of the memory clock. |
| `clocks_current_video` | `nvidia_smi_clocks_current_video` | The current frequency of the video (encoder plus decoder) clocks. |
All of these metrics are collected with the following dimensions:
| Dimension | Description |
| --- | --- |
| `index` | A unique identifier for the GPU on this server. Represents the NVIDIA Management Library (NVML) index of the device. |
| `name` | The type of GPU. For example, `NVIDIA Tesla A100` |
| `arch` | The server architecture. |
# Collect Java Management Extensions (JMX) metrics
You can use the CloudWatch agent to collect Java Management Extensions (JMX) metrics from your Java applications.
The CloudWatch agent supports collecting these metrics from the following versions:
+ JVM 8 and later
+ Kafka 0.8.2.x and later
+ Tomcat 9, 10.1, and 11 (beta)
------
#### [ Amazon EC2 ]
**To enable JMX in your JVM instance**
For the CloudWatch agent to be able to collect JMX metrics, your application’s JVM must bind to a port using the `com.sun.management.jmxremote.port` system property.
```
java -Dcom.sun.management.jmxremote.port=port-number -jar example.jar
```
For more information and other configurations, see the [JMX documentation](https://docs.oracle.com/en/java/javase/17/management/monitoring-and-management-using-jmx-technology.html).
------
#### [ Amazon EKS ]
**To enable JMX on your Java application pods**
When using the CloudWatch Observability EKS add-on, you can manage how JMX metrics are enabled with annotations. For more information, see [Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart](install-CloudWatch-Observability-EKS-addon.md). To enable JMX metrics collection from a workload, add the following annotations to the workload manifest file under the `PodTemplate` section:
+ `instrumentation.opentelemetry.io/inject-java: "true"`
+ One or more of the following:
+ For JVM metrics: `cloudwatch.aws.amazon.com/inject-jmx-jvm: "true"`
+ For Kafka broker metrics: `cloudwatch.aws.amazon.com/inject-jmx-kafka: "true"`
+ For Kafka consumer metrics: `cloudwatch.aws.amazon.com/inject-jmx-kafka-consumer: "true"`
+ For Kafka producer metrics: `cloudwatch.aws.amazon.com/inject-jmx-kafka-producer: "true"`
+ For Tomcat metrics: `cloudwatch.aws.amazon.com/inject-jmx-tomcat: "true"`
------
To start collecting JMX metrics, add a `jmx` section inside the `metrics_collected` section of the CloudWatch agent configuration file. The `jmx` section can contain the following fields.
+ `jvm` – Optional. Specifies that you want to retrieve Java Virtual Machine (JVM) metrics from the instance. For more information, see [Collect JVM metrics](#CloudWatch-Agent-JVM-metrics).
This section can include the following fields:
+ `measurement` – Specifies the array of JVM metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the table in [Collect JVM metrics](#CloudWatch-Agent-JVM-metrics).
Within the entry for each individual metric, you can optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `kafka` – Optional. Specifies that you want to retrieve Apache Kafka broker metrics from the instance. For more information, see [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics).
This section can include the following fields:
+ `measurement` – Specifies the array of Kafka broker metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the first table in [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics).
Within the entry for each individual metric, you can optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `kafka-consumer` – Optional. Specifies that you want to retrieve Apache Kafka consumer metrics from the instance. For more information, see [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics).
This section can include the following fields:
+ `measurement` – Specifies the array of Kafka broker metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the second metrics table in [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics).
Within the entry for each individual metric, you can optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `kafka-producer` – Optional. Specifies that you want to retrieve Apache Kafka producer metrics from the instance. For more information, see [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics).
This section can include the following fields:
+ `measurement` – Specifies the array of Kafka broker metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the third metrics table in [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics).
Within the entry for each individual metric, you can optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ `tomcat` – Optional. Specifies that you want to retrieve Tomcat metrics from the instance. For more information, see [Collect Tomcat metrics](#CloudWatch-Agent-Tomcat-metrics).
This section can include the following fields:
+ `measurement` – Specifies the array of Tomcat metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the table in [Collect Tomcat metrics](#CloudWatch-Agent-Tomcat-metrics).
Within the entry for each individual metric, you can optionally specify one or both of the following:
+ `rename` – Specifies a different name for this metric.
+ `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
The `jmx` section can also include the optional `append_dimensions` field:
+ `append_dimensions` – Optional. Additional dimensions to use for only the process metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent.
**The following fields are for Amazon EC2 only.**
+ `endpoint` – The address for the JMX client to connect to. The format is `ip:port`. If the endpoint is not the localhost, and password authentication and SSL must be enabled.
+ `metrics_collection_interval` – Optional. Specifies how often to collect the processes metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file.
This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes.
If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics).
If JMX was enabled with password authentication or SSL for remote access, you can use the following fields.
+ `password_file` – Optional. Specifies a Java properties file of keys to passwords. The file must be read-only and restricted to the user running the CloudWatch agent. If password authentication is enabled, this requires the same username and password pair as the entry in the JMX password file provided in the `com.sun.management.jmxremote.password.file` property. If SSL is enabled, it requires entries for `keystore` and `truststore` and corresponds to the `javax.net.ssl.keyStorePassword` and `javax.net.ssl.trustStorePassword` respectively.
+ `username` – If password authentication is enabled, specify the username that matches the username in the provided password file.
+ `keystore_path` – If SSL is enabled, specify the full path to the Java keystore, which consists of a private key and a certificate to the public key. Corresponds to the `javax.net.ssl.keyStore` property.
+ `keystore_type` – If SSL is enabled, specify the type of keystore being used. Corresponds to the `javax.net.ssl.keyStoreType` property.
+ `truststore_path` – If SSL is enabled, specify the full path to the Java truststore, which must contain the remote JMX server's public certificate. Corresponds to the `javax.net.ssl.trustStore` property.
+ `truststore_type` – If SSL is enabled, specify the type of truststore being used. Corresponds to the `javax.net.ssl.trustStoreType` property.
+ `remote_profile` – Optional. Supported JMX remote profiles are TLS in combination with SASL profiles: `SASL/PLAIN`, `SASL/DIGEST-MD5`, and `SASL/CRAM-MD5`. Should be one of:`SASL/PLAIN`, `SASL/DIGEST-MD5`, `SASL/CRAM-MD5`, `TLS SASL/PLAIN`, `TLS SASL/DIGEST-MD5`, or `TLS SASL/CRAM-MD5`
+ `realm` – Optional. The realm as required by the remote profile `SASL/DIGEST-MD5`.
+ `registry_ssl_enabled` – If RMI registry authentication is enabled. Set to true if the JVM was configured with `com.sun.management.jmxremote.registry.ssl=true`.
+ `insecure` Set to `true` to opt out of the validation required if the agent is configured for a non-localhost endpoint.
The following is an example of the `jmx` section of the CloudWatch agent configuration file.
```
{
"metrics": {
"metrics_collected": {
"jmx": [
{
"endpoint": "remotehost:1314",
"jvm": {
"measurement": [
"jvm.memory.heap.init",
"jvm.memory.nonheap.used"
]
},
"kafka": {
"measurement": [
"kafka.request.count",
{
"name": "kafka.message.count",
"rename": "KAFKA_MESSAGE_COUNT",
"unit": "Count"
}
]
},
"username": "cwagent",
"keystore_path": "/path/to/keystore",
"keystore_type": "PKCS12",
"truststore_path": "/path/to/truststore",
"truststore_type": "PKCS12"
},
{
"endpoint": "localhost:1315",
"kafka-producer": {
"measurement": [
"kafka.producer.request-rate"
]
},
"append_dimensions": {
"service.name": "kafka/1"
}
}
]
}
}
}
```
## Collect JVM metrics
You can use the CloudWatch agent to collect Java Virtual Machine (JVM) metrics. To set this up, add a `jvm` section inside the `jmx`section of the CloudWatch agent configuration file.
The following metrics can be collected.
| Metric | Dimensions | Description |
| --- | --- | --- |
| `jvm.classes.loaded` | [DEFAULT] | The total number of loaded classes. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.gc.collections.count` | [DEFAULT], `name` | The total number of garbage collections that have occurred. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.gc.collections.elapsed` | [DEFAULT], `name` | The approximate accumulated garbage collection elapsed time. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.heap.init` | [DEFAULT] | The initial amount of memory that the JVM requests from the operating system for the heap. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.heap.max` | [DEFAULT] | The maximum amount of memory that can be used for the heap. **Unit:** Bytes **Meaningful statistics:** Maximum |
| `jvm.memory.heap.used` | [DEFAULT] | The current heap memory usage. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.heap.committed` | [DEFAULT] | The amount of memory that is guaranteed to be available for the heap. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.nonheap.init` | [DEFAULT] | The initial amount of memory that the JVM requests from the operating system for non-heap purposes. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.nonheap.max` | [DEFAULT] | The maximum amount of memory that can be used for non-heap purposes. **Unit:** Bytes **Meaningful statistics:** Maximum |
| `jvm.memory.nonheap.used` | [DEFAULT] | The current non-heap memory usage. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.nonheap.committed` | [DEFAULT] | The amount of memory that is guaranteed to be available for non-heap purposes. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.pool.init` | [DEFAULT], `name` | The initial amount of memory that the JVM requests from the operating system for the memory pool. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.pool.max` | [DEFAULT], `name` | The maximum amount of memory that can be used for the memory pool. **Unit:** Bytes **Meaningful statistics:** Maximum |
| `jvm.memory.pool.used` | [DEFAULT], `name` | The current memory pool memory usage. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.memory.pool.committed` | [DEFAULT], `name` | The amount of memory that is guaranteed to be available for the memory pool. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `jvm.threads.count` | [DEFAULT] | The current number of threads. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
The JVM metrics are collected with the following dimensions:
| Dimension | Description |
| --- | --- |
| [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. |
| `name` | For `jvm.gc.collections` metrics, the value is the garbage collector name. For `jvm.memory.pool` metrics, the value is the memory pool name. |
## Collect Kafka metrics
You can use the CloudWatch agent to collect Apache Kafka metrics. To set this up, add one or more of the following subsections inside the `jmx` section of the CloudWatch agent configuration file.
+ Use a `kafka` section to collect Kafka broker metrics.
+ Use a `kafka-consumer` section to collect Kafka consumer metrics.
+ Use a `kafka-producer` section to collect Kafka producer metrics.
**Kafka broker metrics**
The following metrics can be collected for Kafka brokers.
| Metric | Dimensions | Description |
| --- | --- | --- |
| `kafka.message.count` | [DEFAULT] | The number of messages received by the Kafka broker. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.request.count` | [DEFAULT], `type` | The number of requests received by the Kafka broker. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.request.failed` | [DEFAULT], `type` | The number of requests to the Kafka broker that resulted in a failure. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.request.time.total` | [DEFAULT], `type` | The total time that the Kafka broker has taken to service requests. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.request.time.50p` | [DEFAULT], `type` | The 50th percentile time that the Kafka broker has taken to service requests. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.request.time.99p` | [DEFAULT], `type` | The 99th percentile time that the Kafka broker has taken to service requests. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.request.time.avg` | [DEFAULT], `type` | The average time that the Kafka broker has taken to service requests. **Unit:** Milliseconds **Meaningful statistics:** Average |
| `kafka.network.io` | [DEFAULT], `state` | The number of bytes received by or sent by the Kafka broker. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.purgatory.size` | [DEFAULT], `type` | The number of requests waiting in purgatory. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.partition.count` | [DEFAULT] | The number of partitions on the Kafka broker. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.partition.offline` | [DEFAULT] | The number of partitions that are offline. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.partition.under_replicated` | [DEFAULT] | The number of under-replicated partitions. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.isr.operation.count` | [DEFAULT], `operation` | The number of in-sync replica shrink and expand operations. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.max.lag` | [DEFAULT] | The maximum lag in messages between follower and leader replicas. **Unit:** None **Meaningful statistics:** Maximum |
| `kafka.controller.active.count` | [DEFAULT] | The number of active controllers on the broker. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.leader.election.rate` | [DEFAULT] | Leader election rate. If this increases, it indicates broker failures. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.unclean.election.rate` | [DEFAULT] | Unclean leader election rate. If this increases, it indicates broker failures. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.request.queue` | [DEFAULT] | The size of the request queue. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.logs.flush.time.count` | [DEFAULT] | The logs flush count. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.logs.flush.time.median` | [DEFAULT] | The 50th percentile value of the logs flush count. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.logs.flush.time.99p` | [DEFAULT] | The 99th percentile value of the logs flush count. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average |
The Kafka broker metrics are collected with the following dimensions:
| Dimension | Description |
| --- | --- |
| [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. |
| `type` | The request type. Possible values are `produce`, `fetch`, `fetchconsumer`, and `fetchfollower`. |
| `state` | The direction of network traffic. Possible values are `in` and `out`. |
| `operation` | The operation type for the in-sync replica. Possible values are `shrink` and `expand`. |
**Kafka consumer metrics**
The following metrics can be collected for Kafka consumers.
| Metric | Dimensions | Description |
| --- | --- | --- |
| `kafka.consumer.fetch-rate` | [DEFAULT], `client-id` | The number of fetch requests for all topics per second. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.consumer.records-lag-max` | [DEFAULT], `client-id` | The number of messages that the consumer lags behind the producer. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.consumer.total.bytes-consumed-rate` | [DEFAULT], `client-id` | The average number of bytes consumed for all topics per second. **Unit:** Bytes **Meaningful statistics:** Average |
| `kafka.consumer.total.fetch-size-avg` | [DEFAULT], `client-id` | The number of bytes fetched per request for all topics. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.consumer.total.records-consumed-rate` | [DEFAULT], `client-id` | The average number of records consumed for all topics per second. **Unit:** None **Meaningful statistics:** Average |
| `kafka.consumer.bytes-consumed-rate` | [DEFAULT], `client-id`, `topic` | The average number of bytes consumed per second. **Unit:** Bytes **Meaningful statistics:** Average |
| `kafka.consumer.fetch-size-avg` | [DEFAULT], `client-id`, `topic` | The number of bytes fetched per request. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.consumer.records-consumed-rate` | [DEFAULT], `client-id`, `topic` | The average number of records consumed per second. **Unit:** None **Meaningful statistics:** Average |
The Kafka consumer metrics are collected with the following dimensions:
| Dimension | Description |
| --- | --- |
| [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. |
| `client-id` | The ID of the client. |
| `topic` | The Kafka topic. |
**Kafka producer metrics**
The following metrics can be collected for Kafka producers.
| Metric | Dimensions | Description |
| --- | --- | --- |
| `kafka.producer.io-wait-time-ns-avg` | [DEFAULT], `client-id` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes. **Unit:** None **Meaningful statistics:** Average |
| `kafka.producer.outgoing-byte-rate` | [DEFAULT], `client-id` | The average number of outgoing bytes sent per second to all servers. **Unit:** Bytes **Meaningful statistics:** Average |
| `kafka.producer.request-latency-avg` | [DEFAULT], `client-id` | The average request latency. **Unit:** Milliseconds **Meaningful statistics:** Average |
| `kafka.producer.request-rate` | [DEFAULT], `client-id` | The average number of requests sent per second. **Unit:** None **Meaningful statistics:** Average |
| `kafka.producer.response-rate` | [DEFAULT], `client-id` | The number of responses received per second. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `kafka.producer.byte-rate` | [DEFAULT], `client-id`, `topic` | The average number of bytes sent per second for a topic. **Unit:** Bytes **Meaningful statistics:** Average |
| `kafka.producer.compression-rate` | [DEFAULT], `client-id`, `topic` | The average compression rate of record batches for a topic. **Unit:** None **Meaningful statistics:** Average |
| `kafka.producer.record-error-rate` | [DEFAULT], `client-id`, `topic` | The average per-second number of record sends that resulted in errors for a topic. **Unit:** None **Meaningful statistics:** Average |
| `kafka.producer.record-retry-rate` | [DEFAULT], `client-id`, `topic` | The average per-second number of retried record sends for a topic. **Unit:** None **Meaningful statistics:** Average |
| `kafka.producer.record-send-rate` | [DEFAULT], `client-id`, `topic` | The average number of records sent per second for a topic. **Unit:** None **Meaningful statistics:** Average |
Kafka producer metrics are collected with the following dimensions:
| Dimension | Description |
| --- | --- |
| [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. |
| `client-id` | The ID of the client. |
| `topic` | The Kafka topic. |
## Collect Tomcat metrics
You can use the CloudWatch agent to collect Apache Tomcat metrics. To set this up, add a `tomcat` section inside the `metrics_collected` section of the CloudWatch agent configuration file.
The following metrics can be collected.
| Metric | Dimensions | Description |
| --- | --- | --- |
| `tomcat.sessions` | [DEFAULT] | The number of active sessions. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `tomcat.errors` | [DEFAULT], `proto_handler` | The number of errors encountered. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average |
| `tomcat.processing_time` | [DEFAULT], `proto_handler` | The total processing time. **Unit:** Milliseconds **Meaningful statistics: **Minimum, Maximum, Average |
| `tomcat.traffic` | [DEFAULT], `proto_handler` | The number of bytes received and sent. **Unit:** Bytes **Meaningful statistics: **Minimum, Maximum, Average |
| `tomcat.threads` | [DEFAULT], `proto_handler` | The number of threads. **Unit:** None **Meaningful statistics: **Minimum, Maximum, Average |
| `tomcat.max_time` | [DEFAULT], `proto_handler`, `direction` | Maximum time to process a request. **Unit:** Milliseconds **Meaningful statistics: **Maximum |
| `tomcat.request_count` | [DEFAULT], `proto_handler` | The total requests. **Unit:** None **Meaningful statistics: **Minimum, Maximum, Average |
Tomcat metrics are collected with the following dimensions:
| Dimension | Description |
| --- | --- |
| [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. |
| `proto_handler` | The `proto_handler` is an identifier for a connector, which is provided in the `--` format (for example, `http-nio-8080`). |
| `direction` | The traffic direction. Possible values are `received` and `sent`. |
# Collect metrics and traces with OpenTelemetry
You can collect metrics and traces from your applications or services using the CloudWatch agent with the OpenTelemetry Protocol (OTLP), which is a popular open source solution. You can use any OpenTelemetry SDK to send metrics and traces to the CloudWatch agent. For more information about the available OpenTelemetry SDKs, see [ OpenTelemetry Supported Language APIs & SDKs.](https://opentelemetry.io/docs/languages/).
To collect OpenTelemetry metrics and traces, add an `otlp` section to the CloudWatch agent configuration file. The section has the following fields:
+ `grpc_endpoint` – Optional. Specifies the address for the CloudWatch agent to use to listen for OpenTelemetry metrics or traces sent using gRPC Remote Procedure Calls. The format is `ip:port`. This address must match the address set for the gRPC exporter in the OpenTelemetry SDK. If you omit this field, the default of `127.0.0.1:4317` is used.
+ `http_endpoint` – Optional. Specifies the address for the CloudWatch agent to use to listen for OpenTelemetry metrics or traces sent over HTTP. The format is `ip:port`. This address must match the address set for the HTTP exporter in the OpenTelemetry SDK. If you omit this field, the default of `127.0.0.1:4318` is used.
+ `tls` – Optional. Specifies that the server should be configured with TLS.
+ `cert_file` – Path to the TLS certificate to use for TLS required connections.
+ `key_file` – Path to the TLS key to use for TLS required connections.
The `otlp` section can be placed in multiple sections within the CloudWatch agent configuration file depending on how and where you want to send the metrics and traces.
**Important**
Each `otlp` section requires a unique endpoint and port. For detailed information about splitting the metrics and traces endpoints, see [OTLP Exporter Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/) in the OpenTelemetry SDK documentation.
To send metrics to CloudWatch or Amazon Managed Service for Prometheus, add the `otlp` section under `metrics_collected` within the `metrics` section. For more information about sending metrics to different destinations, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). The following example shows a configuration that sends metrics to CloudWatch:
**Note**
If you are running the agent in containerized environments and sending telemetry from outside the agent container’s network, make sure to specify the endpoint as `0.0.0.0` instead of the default endpoint `127.0.0.1`.
```
{
"metrics": {
"metrics_collected": {
"otlp": {
"grpc_endpoint": "127.0.0.1:4317",
"http_endpoint": "127.0.0.1:4318"
}
}
}
}
```
To send metrics to Amazon CloudWatch Logs using the Embedded metric format (EMF), add the `otlp` section under `metrics_collected` within the `logs` section. This sends the EMF logs by default to the `/aws/cwagent` log group and a generated log stream. The metrics are extracted into the `CWAgent` namespace by default. The following example shows a configuration that sends metrics as EMF logs to CloudWatch Logs:
```
{
"logs": {
"metrics_collected": {
"otlp": {
"grpc_endpoint": "127.0.0.1:4317",
"http_endpoint": "127.0.0.1:4318"
}
}
}
}
```
To send traces to AWS X-Ray, add the `otlp` section under `traces_collected` within the `traces` section. The following example shows a configuration that sends traces to X-Ray:
```
{
"traces": {
"traces_collected": {
"otlp": {
"grpc_endpoint": "127.0.0.1:4317",
"http_endpoint": "127.0.0.1:4318"
}
}
}
}
```
# Collect process metrics with the procstat plugin
The *procstat* plugin enables you to collect metrics from individual processes. The plugin is supported on Linux servers and on servers running supported version of Windows Server. This section describes how to configure the CloudWatch agent for procstat and view metrics the CloudWatch agent imports. It also lists the metrics that procstat collects.
**Note**
The `procstat` plugin is not supported for the Fargate launch type in Amazon ECS environments.
**Topics**
+ [
## Configure the CloudWatch agent for procstat
](#CloudWatch-Agent-procstat-configuration)
+ [
## Metrics collected by procstat
](#CloudWatch-Agent-procstat-process-metrics-collected)
+ [
## Viewing process metrics imported by the CloudWatch agent
](#CloudWatch-view-procstat-metrics)
## Configure the CloudWatch agent for procstat
To use the procstat plugin, add a `procstat` section in the `metrics_collected` section of the CloudWatch agent configuration file. There are three ways to specify the processes to monitor. You can use only one of these methods, but you can use that method to specify one or more processes to monitor.
+ `pid_file`: Selects processes by the names of the process identification number (PID) files they create.
+ `exe`: Selects the processes that have process names that match the string that you specify, using regular expression matching rules. The match is a "contains" match, meaning that if you specify `agent` as the term to match, processes with names like `cloudwatchagent` match the term. For more information, see [Syntax](https://github.com/google/re2/wiki/Syntax).
+ `pattern`: Selects processes by the command lines used to start the processes. All processes are selected that have command lines matching the specified string using regular expression matching rules. The entire command line is checked, including parameters and options used with the command.
The match is a "contains" match, meaning that if you specify `-c` as the term to match, processes with parameters like `-config` match the term.
The CloudWatch agent uses only one of these methods, even if you include more than one of the above sections. If you specify more than one section, the CloudWatch agent uses the `pid_file` section if it is present. If not, it uses the `exe` section.
On Linux servers, the strings that you specify in an `exe` or `pattern` section are evaluated as regular expressions. On servers running Windows Server, these strings are evaluated as WMI queries. An example would be `pattern: "%apache%"`. For more information, see [LIKE Operator](https://docs.microsoft.com/en-us/windows/desktop/WmiSdk/like-operator).
Whichever method you use, you can include an optional `metrics_collection_interval` parameter, which specifies how often in seconds to collect those metrics. If you omit this parameter, the default value of 60 seconds is used.
In the examples in the following sections, the `procstat` section is the only section included in the `metrics_collected` section of the agent configuration file. Actual configuration files can also include other sections in `metrics_collected`. For more information, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md).
### Configure with pid\$1file
The following example `procstat` section monitors the processes that create the PID files `example1.pid` and `example2.pid`. Different metrics are collected from each process. Metrics collected from the process that creates `example2.pid` are collected every 10 seconds, and the metrics collected from the `example1.pid` process are collected every 60 seconds, the default value.
```
{
"metrics": {
"metrics_collected": {
"procstat": [
{
"pid_file": "/var/run/example1.pid",
"measurement": [
"cpu_usage",
"memory_rss"
]
},
{
"pid_file": "/var/run/example2.pid",
"measurement": [
"read_bytes",
"read_count",
"write_bytes"
],
"metrics_collection_interval": 10
}
]
}
}
}
```
### Configuring with exe
The following example `procstat` section monitors all processes with names that match the strings `agent` or `plugin`. The same metrics are collected from each process.
```
{
"metrics": {
"metrics_collected": {
"procstat": [
{
"exe": "agent",
"measurement": [
"cpu_time",
"cpu_time_system",
"cpu_time_user"
]
},
{
"exe": "plugin",
"measurement": [
"cpu_time",
"cpu_time_system",
"cpu_time_user"
]
}
]
}
}
}
```
### Configuring with pattern
The following example `procstat` section monitors all processes with command lines that match the strings `config` or `-c`. The same metrics are collected from each process.
```
{
"metrics": {
"metrics_collected": {
"procstat": [
{
"pattern": "config",
"measurement": [
"rlimit_memory_data_hard",
"rlimit_memory_data_soft",
"rlimit_memory_stack_hard",
"rlimit_memory_stack_soft"
]
},
{
"pattern": "-c",
"measurement": [
"rlimit_memory_data_hard",
"rlimit_memory_data_soft",
"rlimit_memory_stack_hard",
"rlimit_memory_stack_soft"
]
}
]
}
}
}
```
## Metrics collected by procstat
The following table lists the metrics that you can collect with the `procstat` plugin.
The CloudWatch agent adds `procstat` to the beginning of the following metric names. There is a different syntax depending on whether it was collected from a Linux server or a server running Windows Server. For example, the `cpu_time` metric appears as `procstat_cpu_time` when collected from Linux and as `procstat cpu_time` when collected from Windows Server.
| Metric name | Available on | Description |
| --- | --- | --- |
| `cpu_time` | Linux | The amount of time that the process uses the CPU. This metric is measured in hundredths of a second. Unit: Count |
| `cpu_time_guest` | Linux | The amount of time that the process is in guest mode. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_guest_nice` | Linux | The amount of time that the process is running in a nice guest. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_idle` | Linux | The amount of time that the process is in idle mode. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_iowait` | Linux | The amount of time that the process is waiting for I/O operations to complete. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_irq` | Linux | The amount of time that the process is servicing interrupts. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_nice` | Linux | The amount of time that the process is in nice mode. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_soft_irq` | Linux | The amount of time that the process is servicing software interrupts. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_steal` | Linux | The amount of time spent running in other operating systems when running in a virtualized environment. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_stolen` | Linux, Windows Server | The amount of time that the process is in *stolen time*, which is time spent in other operating systems in a virtualized environment. This metric is measured in hundredths of a second. Type: Float Unit: None |
| `cpu_time_system` | Linux, Windows Server, macOS | The amount of time that the process is in system mode. This metric is measured in hundredths of a second. Type: Float Unit: Count |
| `cpu_time_user` | Linux, Windows Server, macOS | The amount of time that the process is in user mode. This metric is measured in hundredths of a second. Unit: Count |
| `cpu_usage` | Linux, Windows Server, macOS | The percentage of time that the process is active in any capacity. Unit: Percent |
| `memory_data` | Linux, macOS | The amount of memory that the process uses for data. Unit: Bytes |
| `memory_locked` | Linux, macOS | The amount of memory that the process has locked. Unit: Bytes |
| `memory_rss` | Linux, Windows Server, macOS | The amount of real memory (resident set) that the process is using. Unit: Bytes |
| `memory_stack` | Linux, macOS | The amount of stack memory that the process is using. Unit: Bytes |
| `memory_swap` | Linux, macOS | The amount of swap memory that the process is using. Unit: Bytes |
| `memory_vms` | Linux, Windows Server, macOS | The amount of virtual memory that the process is using. Unit: Bytes |
| `num_fds` | Linux | The number of file descriptors that this process has open. Unit: None |
| `num_threads` | Linux, Windows, macOS | The number of threads in this process. Unit: None |
| `pid` | Linux, Windows Server, macOS | Process identifier (ID). Unit: None |
| `pid_count` | Linux, Windows Server. macOS | The number of process IDs associated with the process. On Linux servers and macOS computers the full name of this metric is `procstat_lookup_pid_count` and on Windows Server it is `procstat_lookup pid_count`. Unit: None |
| `read_bytes` | Linux, Windows Server | The number of bytes that the process has read from disks. Unit: Bytes |
| `write_bytes` | Linux, Windows Server | The number of bytes that the process has written to disks. Unit: Bytes |
| `read_count` | Linux, Windows Server | The number of disk read operations that the process has executed. Unit: None |
| `rlimit_realtime_priority_hard` | Linux | The hard limit on the real-time priority that can be set for this process. Unit: None |
| `rlimit_realtime_priority_soft` | Linux | The soft limit on the real-time priority that can be set for this process. Unit: None |
| `rlimit_signals_pending_hard` | Linux | The hard limit on maximum number of signals that can be queued by this process. Unit: None |
| `rlimit_signals_pending_soft` | Linux | The soft limit on maximum number of signals that can be queued by this process. Unit: None |
| `rlimit_nice_priority_hard` | Linux | The hard limit on the maximum nice priority that can be set by this process. Unit: None |
| `rlimit_nice_priority_soft` | Linux | The soft limit on the maximum nice priority that can be set by this process. Unit: None |
| `rlimit_num_fds_hard` | Linux | The hard limit on the maximum number of file descriptors that this process can have open. Unit: None |
| `rlimit_num_fds_soft` | Linux | The soft limit on the maximum number of file descriptors that this process can have open. Unit: None |
| `write_count` | Linux, Windows Server | The number of disk write operations that the process has executed. Unit: None |
| `involuntary_context_switches` | Linux | The number of times that the process was involuntarily context-switched. Unit: None |
| `voluntary_context_switches` | Linux | The number of times that the process was context-switched voluntarily. Unit: None |
| `realtime_priority` | Linux | The current usage of real-time priority for the process. Unit: None |
| `nice_priority` | Linux | The current usage of nice priority for the process. Unit: None |
| `signals_pending` | Linux | The number of signals pending to be handled by the process. Unit: None |
| `rlimit_cpu_time_hard` | Linux | The hard CPU time resource limit for the process. Unit: None |
| `rlimit_cpu_time_soft` | Linux | The soft CPU time resource limit for the process. Unit: None |
| `rlimit_file_locks_hard` | Linux | The hard file locks resource limit for the process. Unit: None |
| `rlimit_file_locks_soft` | Linux | The soft file locks resource limit for the process. Unit: None |
| `rlimit_memory_data_hard` | Linux | The hard resource limit on the process for memory used for data. Unit: Bytes |
| `rlimit_memory_data_soft` | Linux | The soft resource limit on the process for memory used for data. Unit: Bytes |
| `rlimit_memory_locked_hard` | Linux | The hard resource limit on the process for locked memory. Unit: Bytes |
| `rlimit_memory_locked_soft` | Linux | The soft resource limit on the process for locked memory. Unit: Bytes |
| `rlimit_memory_rss_hard` | Linux | The hard resource limit on the process for physical memory. Unit: Bytes |
| `rlimit_memory_rss_soft` | Linux | The soft resource limit on the process for physical memory. Unit: Bytes |
| `rlimit_memory_stack_hard` | Linux | The hard resource limit on the process stack. Unit: Bytes |
| `rlimit_memory_stack_soft` | Linux | The soft resource limit on the process stack. Unit: Bytes |
| `rlimit_memory_vms_hard` | Linux | The hard resource limit on the process for virtual memory. Unit: Bytes |
| `rlimit_memory_vms_soft` | Linux | The soft resource limit on the process for virtual memory. Unit: Bytes |
## Viewing process metrics imported by the CloudWatch agent
After importing process metrics into CloudWatch, you can view these metrics as time series graphs, and create alarms that can watch these metrics and notify you if they breach a threshold that you specify. The following procedure shows how to view process metrics as a time series graph. For more information about setting alarms, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md).
**To view process metrics in the CloudWatch console**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**.
1. Choose the namespace for the metrics collected by the agent. By default, this is **CWAgent**, but you may have specified a different namespace in the CloudWatch agent configuration file.
1. Choose a metric dimension (for example, **Per-Instance Metrics**).
1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following:
1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table.
1. To sort the table, use the column heading.
1. To filter by resource, choose the resource ID and then choose **Add to search**.
1. To filter by metric, choose the metric name and then choose **Add to search**.
1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, **Add to dashboard**.
# Retrieve custom metrics with StatsD
You can retrieve additional custom metrics from your applications or services using the CloudWatch agent with the `StatsD` protocol. StatsD is a popular open-source solution that can gather metrics from a wide variety of applications. StatsD is especially useful for instrumenting your own metrics. For an example of using the CloudWatch agent and StatsD together, see [ How to better monitor your custom application metrics using Amazon CloudWatch Agent](https://aws.amazon.com/blogs/devops/new-how-to-better-monitor-your-custom-application-metrics-using-amazon-cloudwatch-agent/).
`StatsD` is supported on both Linux servers and servers running Windows Server. CloudWatch supports the following `StatsD` format:
```
MetricName:value|type|@sample_rate|#tag1:
value,tag1...
```
+ `MetricName` – A string with no colons, bars, \$1 characters, or @ characters.
+ `value` – This can be either integer or float.
+ `type` – Specify `c` for counter, `g` for gauge, `ms` for timer, `h` for histogram, or `s` for set.
+ `sample_rate` – (Optional) A float between 0 and 1, inclusive. Use only for counter, histogram, and timer metrics. The default value is 1 (sampling 100% of the time).
+ `tags` – (Optional) A comma-separated list of tags. `StatsD` tags are similar to dimensions in CloudWatch. Use colons for key/value tags, such as `env:prod`.
You can use any `StatsD` client that follows this format to send the metrics to the CloudWatch agent. For more information about some of the available `StatsD` clients, see the [StatsD client page on GitHub](https://github.com/etsy/statsd/wiki#client-implementations).
To collect these custom metrics, add a `"statsd": {}` line to the `metrics_collected` section of the agent configuration file. You can add this line manually. If you use the wizard to create the configuration file, it's done for you. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md).
The `StatsD` default configuration works for most users. There are optional fields that you can add to the **statsd** section of the agent configuration file as needed:
+ `service_address` – The service address to which the CloudWatch agent should listen. The format is `ip:port`. If you omit the IP address, the agent listens on all available interfaces. Only the UDP format is supported, so you don't need to specify a UDP prefix.
The default value is `:8125`.
+ `metrics_collection_interval` – How often in seconds that the `StatsD` plugin runs and collects metrics. The default value is 10 seconds. The range is 1–172,000.
+ `metrics_aggregation_interval` – How often in seconds CloudWatch aggregates metrics into single data points. The default value is 60 seconds.
For example, if `metrics_collection_interval` is 10 and `metrics_aggregation_interval` is 60, CloudWatch collects data every 10 seconds. After each minute, the six data readings from that minute are aggregated into a single data point, which is sent to CloudWatch.
The range is 0–172,000. Setting `metrics_aggregation_interval` to 0 disables the aggregation of `StatsD` metrics.
+ `allowed_pending_messages` – The number of UDP messages that are allowed to queue up. When the queue is full, the StatsD server starts dropping packets. The default value is 10000.
+ `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs.
The following is an example of the **statsd** section of the agent configuration file, using the default port and custom collection and aggregation intervals.
```
{
"metrics":{
"metrics_collected":{
"statsd":{
"service_address":":8125",
"metrics_collection_interval":60,
"metrics_aggregation_interval":300
}
}
}
}
```
## Viewing StatsD metrics imported by the CloudWatch agent
After importing StatsD metrics into CloudWatch, you can view these metrics as time series graphs, and create alarms that can watch these metrics and notify you if they breach a threshold that you specify. The following procedure shows how to view StatsD metrics as a time series graph. For more information about setting alarms, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md).
**To view StatsD metrics in the CloudWatch console**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**.
1. Choose the namespace for the metrics collected by the agent. By default, this is **CWAgent**, but you may have specified a different namespace in the CloudWatch agent configuration file.
1. Choose a metric dimension (for example, **Per-Instance Metrics**).
1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following:
1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table.
1. To sort the table, use the column heading.
1. To filter by resource, choose the resource ID and then choose **Add to search**.
1. To filter by metric, choose the metric name and then choose **Add to search**.
1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, **Add to dashboard**.
# Retrieve custom metrics with collectd
You can retrieve additional metrics from your applications or services using the CloudWatch agent with the collectd protocol, which is supported only on Linux servers. collectd is a popular open-source solution with plugins that can gather system statistics for a wide variety of applications. By combining the system metrics that the CloudWatch agent can already collect with the additional metrics from collectd, you can better monitor, analyze, and troubleshoot your systems and applications. For more information about collectd, see [collectd - The system statistics collection daemon](https://collectd.org/).
You use the collectd software to send the metrics to the CloudWatch agent. For the collectd metrics, the CloudWatch agent acts as the server while the collectd plugin acts as the client.
The collectd software is not installed automatically on every server. On a server running Amazon Linux 2, follow these steps to install collectd
```
sudo amazon-linux-extras install collectd
```
For information about installing collectd on other systems, see the [Download page for collectd.](https://www.collectd.org/download.html)
To collect these custom metrics, add a **"collectd": \$1\$1** line to the **metrics\$1collected** section of the agent configuration file. You can add this line manually. If you use the wizard to create the configuration file, it is done for you. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md).
Optional parameters are also available. If you are using collectd and you do not use `/etc/collectd/auth_file` as your **collectd\$1auth\$1file**, you must set some of these options.
+ **service\$1address:** The service address to which the CloudWatch agent should listen. The format is `"udp://ip:port`. The default is `udp://127.0.0.1:25826`.
+ **name\$1prefix:** A prefix to attach to the beginning of the name of each collectd metric. The default is `collectd_`. The maximum length is 255 characters.
+ **collectd\$1security\$1level:** Sets the security level for network communication. The default is **encrypt**.
**encrypt** specifies that only encrypted data is accepted. **sign** specifies that only signed and encrypted data is accepted. **none** specifies that all data is accepted. If you specify a value for **collectd\$1auth\$1file**, encrypted data is decrypted if possible.
For more information, see [Client setup](https://collectd.org/wiki/index.php/Networking_introduction#Client_setup) and [Possible interactions](https://collectd.org/wiki/index.php/Networking_introduction#Possible_interactions) in the collectd Wiki.
+ **collectd\$1auth\$1file** Sets a file in which user names are mapped to passwords. These passwords are used to verify signatures and to decrypt encrypted network packets. If given, signed data is verified and encrypted packets are decrypted. Otherwise, signed data is accepted without checking the signature and encrypted data cannot be decrypted.
The default is `/etc/collectd/auth_file`.
If **collectd\$1security\$1level** is set to **none**, this is optional. If you set **collectd\$1security\$1level** to `encrypt` or **sign**, you must specify **collectd\$1auth\$1file**.
For the format of the auth file, each line is a user name followed by a colon and any number of spaces followed by the password. For example:
`user1: user1_password`
`user2: user2_password`
+ **collectd\$1typesdb:** A list of one or more files that contain the dataset descriptions. The list must be surrounded by brackets, even if there is just one entry in the list. Each entry in the list must be surrounded by double quotes. If there are multiple entries, separate them with commas. The default on Linux servers is `["/usr/share/collectd/types.db"]`. The default on macOs computers depends on the version of collectd. For example, `["/usr/local/Cellar/collectd/5.12.0/share/collectd/types.db"]`.
For more information, see [https://www.collectd.org/documentation/manpages/types.db.html](https://www.collectd.org/documentation/manpages/types.db.html).
+ **metrics\$1aggregation\$1interval:** How often in seconds CloudWatch aggregates metrics into single data points. The default is 60 seconds. The range is 0 to 172,000. Setting it to 0 disables the aggregation of collectd metrics.
The following is an example of the collectd section of the agent configuration file.
```
{
"metrics":{
"metrics_collected":{
"collectd":{
"name_prefix":"My_collectd_metrics_",
"metrics_aggregation_interval":120
}
}
}
}
```
## Viewing collected metrics imported by the CloudWatch agent
After importing collectd metrics into CloudWatch, you can view these metrics as time series graphs, and create alarms that can watch these metrics and notify you if they breach a threshold that you specify. The following procedure shows how to view collectd metrics as a time series graph. For more information about setting alarms, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md).
**To view collectd metrics in the CloudWatch console**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**.
1. Choose the namespace for the metrics collected by the agent. By default, this is **CWAgent**, but you may have specified a different namespace in the CloudWatch agent configuration file.
1. Choose a metric dimension (for example, **Per-Instance Metrics**).
1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following:
1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table.
1. To sort the table, use the column heading.
1. To filter by resource, choose the resource ID and then choose **Add to search**.
1. To filter by metric, choose the metric name and then choose **Add to search**.
1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, **Add to dashboard**.
# Set up and configure Prometheus metrics collection on Amazon EC2 instances
The following sections explain how to install the CloudWatch agent with Prometheus monitoring on EC2 instances, and how to configure the agent to scrape additional targets. It also provides tutorials for setting up sample workloads to use testing with Prometheus monitoring.
Both Linux and Windows instances are supported.
For information about the operating systems supported by the CloudWatch agent, see [Collect metrics, logs, and traces using the CloudWatch agent](Install-CloudWatch-Agent.md)
**VPC security group requirements**
If you are using a VPC, the following requirements apply.
+ The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP.
+ The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP.
**Topics**
+ [
## Step 1: Install the CloudWatch agent
](#CloudWatch-Agent-PrometheusEC2-install)
+ [
## Step 2: Scrape Prometheus sources and import metrics
](#CloudWatch-Agent-PrometheusEC2-configure)
+ [
## Example: Set up Java/JMX sample workloads for Prometheus metric testing
](#CloudWatch-Agent-Prometheus-Java)
## Step 1: Install the CloudWatch agent
The first step is to install the CloudWatch agent on the EC2 instance. For instructions, see [Installing the CloudWatch agent](install-CloudWatch-Agent-on-EC2-Instance.md).
## Step 2: Scrape Prometheus sources and import metrics
The CloudWatch agent with Prometheus monitoring needs two configurations to scrape the Prometheus metrics. One is for the standard Prometheus configurations as documented in [ ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)in the Prometheus documentation. The other is for the CloudWatch agent configuration.
### Prometheus scrape configuration
The CloudWatch agent supports the standard Prometheus scrape configurations as documented in [ ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. You can edit this section to update the configurations that are already in this file, and add additional Prometheus scraping targets. A sample configuration file contains the following global configuration lines:
```
PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus.yaml
global:
scrape_interval: 1m
scrape_timeout: 10s
scrape_configs:
- job_name: MY_JOB
sample_limit: 10000
file_sd_configs:
- files: ["C:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\prometheus_sd_1.yaml", "C:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\prometheus_sd_2.yaml"]
```
The `global` section specifies parameters that are valid in all configuration contexts. They also serve as defaults for other configuration sections. It contains the following parameters:
+ `scrape_interval`— Defines how frequently to scrape targets.
+ `scrape_timeout`— Defines how long to wait before a scrape request times out.
The `scrape_configs` section specifies a set of targets and parameters that define how to scrape them. It contains the following parameters:
+ `job_name`— The job name assigned to scraped metrics by default.
+ `sample_limit`— Per-scrape limit on the number of scraped samples that will be accepted.
+ `file_sd_configs`— List of file service discovery configurations. It reads a set of files containing a list of zero or more static configs. The `file_sd_configs` section contains a `files` parameter which defines patterns for files from which target groups are extracted.
The CloudWatch agent supports the following service discovery configuration types.
**`static_config`** Allows specifying a list of targets and a common label set for them. It is the canonical way to specify static targets in a scrape configuration.
The following is a sample static config to scrape Prometheus metrics from a local host. Metrics can also be scraped from other servers if the Prometheus port is open to the server where the agent runs.
```
PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus_sd_1.yaml
- targets:
- 127.0.0.1:9404
labels:
key1: value1
key2: value2
```
This example contains the following parameters:
+ `targets`— The targets scraped by the static config.
+ `labels`— Labels assigned to all metrics that are scraped from the targets.
**`ec2_sd_config`** Allows retrieving scrape targets from Amazon EC2 instances. The following is a sample `ec2_sd_config` to scrape Prometheus metrics from a list of EC2 instances. The Prometheus ports of these instances have to open to the server where the CloudWatch agent runs. The IAM role for the EC2 instance where the CloudWatch agent runs must include the `ec2:DescribeInstance` permission. For example, you could attach the managed policy **AmazonEC2ReadOnlyAccess** to the instance running the CloudWatch agent.
```
PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus.yaml
global:
scrape_interval: 1m
scrape_timeout: 10s
scrape_configs:
- job_name: MY_JOB
sample_limit: 10000
ec2_sd_configs:
- region: us-east-1
port: 9404
filters:
- name: instance-id
values:
- i-98765432109876543
- i-12345678901234567
```
This example contains the following parameters:
+ `region`— The AWS Region where the target EC2 instance is. If you leave this blank, the Region from the instance metadata is used.
+ `port`— The port to scrape metrics from.
+ `filters`— Optional filters to use to filter the instance list. This example filters based on EC2 instance IDs. For more criteria that you can filter on, see [ DescribeInstances](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html).
### CloudWatch agent configuration for Prometheus
The CloudWatch agent configuration file includes `prometheus` sections under both `logs` and `metrics_collected`. It includes the following parameters.
+ **cluster\$1name**— specifies the cluster name to be added as a label in the log event. This field is optional.
+ **log\$1group\$1name**— specifies the log group name for the scraped Prometheus metrics.
+ **prometheus\$1config\$1path**— specifies the Prometheus scrape configuration file path.
+ **emf\$1processor**— specifies the embedded metric format processor configuration. For more information about embedded metric format, see [Embedding metrics within logs](CloudWatch_Embedded_Metric_Format.md).
The `emf_processor` section can contain the following parameters:
+ **metric\$1declaration\$1dedup**— It set to true, the de-duplication function for the embedded metric format metrics is enabled.
+ **metric\$1namespace**— Specifies the metric namespace for the emitted CloudWatch metrics.
+ **metric\$1unit**— Specifies the metric name:metric unit map. For information about supported metric units, see [ MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html).
+ **metric\$1declaration**— are sections that specify the array of logs with embedded metric format to be generated. There are `metric_declaration` sections for each Prometheus source that the CloudWatch agent imports from by default. These sections each include the following fields:
+ `source_labels` specifies the value of the labels that are checked by the `label_matcher` line.
+ `label_matcher` is a regular expression that checks the value of the labels listed in `source_labels`. The metrics that match are enabled for inclusion in the embedded metric format sent to CloudWatch.
+ `metric_selectors` is a regular expression that specifies the metrics to be collected and sent to CloudWatch.
+ `dimensions` is the list of labels to be used as CloudWatch dimensions for each selected metric.
The following is an example CloudWatch agent configuration for Prometheus.
```
{
"logs":{
"metrics_collected":{
"prometheus":{
"cluster_name":"prometheus-cluster",
"log_group_name":"Prometheus",
"prometheus_config_path":"C:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\prometheus.yaml",
"emf_processor":{
"metric_declaration_dedup":true,
"metric_namespace":"CWAgent-Prometheus",
"metric_unit":{
"jvm_threads_current": "Count",
"jvm_gc_collection_seconds_sum": "Milliseconds"
},
"metric_declaration":[
{
"source_labels":[
"job", "key2"
],
"label_matcher":"MY_JOB;^value2",
"dimensions":[
[
"key1", "key2"
],
[
"key2"
]
],
"metric_selectors":[
"^jvm_threads_current$",
"^jvm_gc_collection_seconds_sum$"
]
}
]
}
}
}
}
}
```
The previous example configures an embedded metric format section to be sent as a log event if the following conditions are met:
+ The value of the label `job` is `MY_JOB`
+ The value of the label `key2` is `value2`
+ The Prometheus metrics `jvm_threads_current` and `jvm_gc_collection_seconds_sum` contains both `job` and `key2` labels.
The log event that is sent includes the following highlighted section.
```
{
"CloudWatchMetrics": [
{
"Metrics": [
{
"Unit": "Count",
"Name": "jvm_threads_current"
},
{
"Unit": "Milliseconds",
"Name": "jvm_gc_collection_seconds_sum"
}
],
"Dimensions": [
[
"key1",
"key2"
],
[
"key2"
]
],
"Namespace": "CWAgent-Prometheus"
}
],
"ClusterName": "prometheus-cluster",
"InstanceId": "i-0e45bd06f196096c8",
"Timestamp": "1607966368109",
"Version": "0",
"host": "EC2AMAZ-PDDOIUM",
"instance": "127.0.0.1:9404",
"jvm_threads_current": 2,
"jvm_gc_collection_seconds_sum": 0.006000000000000002,
"prom_metric_type": "gauge",
...
}
```
## Example: Set up Java/JMX sample workloads for Prometheus metric testing
JMX Exporter is an official Prometheus exporter that can scrape and expose JMX mBeans as Prometheus metrics. For more information, see [prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter).
The CloudWatch agent can collect predefined Prometheus metrics from Java Virtual Machine (JVM), Hjava, and Tomcat (Catalina), from a JMX exporter on EC2 instances.
### Step 1: Install the CloudWatch agent
The first step is to install the CloudWatch agent on the EC2 instance. For instructions, see [Installing the CloudWatch agent](install-CloudWatch-Agent-on-EC2-Instance.md).
### Step 2: Start the Java/JMX workload
The next step is to start the Java/JMX workload.
First, download the latest JMX exporter jar file from the following location: [prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter).
**Use the jar for your sample application**
The example commands in the following sections use `SampleJavaApplication-1.0-SNAPSHOT.jar` as the jar file. Replace these parts of the commands with the jar for your application.
#### Prepare the JMX exporter configuration
The `config.yaml` file is the JMX exporter configuration file. For more information, see [Configuration](https://github.com/prometheus/jmx_exporter#Configuration) in the JMX exporter documentation.
Here is a sample configuration for Java and Tomcat.
```
---
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
- pattern: 'java.lang<>(FreePhysicalMemorySize|TotalPhysicalMemorySize|FreeSwapSpaceSize|TotalSwapSpaceSize|SystemCpuLoad|ProcessCpuLoad|OpenFileDescriptorCount|AvailableProcessors)'
name: java_lang_OperatingSystem_$1
type: GAUGE
- pattern: 'java.lang<>(TotalStartedThreadCount|ThreadCount)'
name: java_lang_threading_$1
type: GAUGE
- pattern: 'Catalina<>(\w+)'
name: catalina_globalrequestprocessor_$3_total
labels:
port: "$2"
protocol: "$1"
help: Catalina global $3
type: COUNTER
- pattern: 'Catalina<>(requestCount|maxTime|processingTime|errorCount)'
name: catalina_servlet_$3_total
labels:
module: "$1"
servlet: "$2"
help: Catalina servlet $3 total
type: COUNTER
- pattern: 'Catalina<>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount)'
name: catalina_threadpool_$3
labels:
port: "$2"
protocol: "$1"
help: Catalina threadpool $3
type: GAUGE
- pattern: 'Catalina<>(processingTime|sessionCounter|rejectedSessions|expiredSessions)'
name: catalina_session_$3_total
labels:
context: "$2"
host: "$1"
help: Catalina session $3 total
type: COUNTER
- pattern: ".*"
```
#### Start the Java application with the Prometheus exporter
Start the sample application. This will emit Prometheus metrics to port 9404. Be sure to replace the entry point `com.gubupt.sample.app.App` with the correct information for your sample java application.
On Linux, enter the following command.
```
$ nohup java -javaagent:./jmx_prometheus_javaagent-0.14.0.jar=9404:./config.yaml -cp ./SampleJavaApplication-1.0-SNAPSHOT.jar com.gubupt.sample.app.App &
```
On Windows, enter the following command.
```
PS C:\> java -javaagent:.\jmx_prometheus_javaagent-0.14.0.jar=9404:.\config.yaml -cp .\SampleJavaApplication-1.0-SNAPSHOT.jar com.gubupt.sample.app.App
```
#### Verify the Prometheus metrics emission
Verify that Prometheus metrics are being emitted.
On Linux, enter the following command.
```
$ curl localhost:9404
```
On Windows, enter the following command.
```
PS C:\> curl http://localhost:9404
```
Sample output on Linux:
```
StatusCode : 200
StatusDescription : OK
Content : # HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 2526.0
# HELP jvm_classes_loaded_total The total number of class...
RawContent : HTTP/1.1 200 OK
Content-Length: 71908
Content-Type: text/plain; version=0.0.4; charset=utf-8
Date: Fri, 18 Dec 2020 16:38:10 GMT
# HELP jvm_classes_loaded The number of classes that are currentl...
Forms : {}
Headers : {[Content-Length, 71908], [Content-Type, text/plain; version=0.0.4; charset=utf-8], [Date, Fri, 18
Dec 2020 16:38:10 GMT]}
Images : {}
InputFields : {}
Links : {}
ParsedHtml : System.__ComObject
RawContentLength : 71908
```
### Step 3: Configure the CloudWatch agent to scrape Prometheus metrics
Next, set up the Prometheus scrape configuration in the CloudWatch agent configuration file.
**To set up the Prometheus scrape configuration for the Java/JMX example**
1. Set up the configuration for `file_sd_config` and `static_config`.
On Linux, enter the following command.
```
$ cat /opt/aws/amazon-cloudwatch-agent/var/prometheus.yaml
global:
scrape_interval: 1m
scrape_timeout: 10s
scrape_configs:
- job_name: jmx
sample_limit: 10000
file_sd_configs:
- files: [ "/opt/aws/amazon-cloudwatch-agent/var/prometheus_file_sd.yaml" ]
```
On Windows, enter the following command.
```
PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus.yaml
global:
scrape_interval: 1m
scrape_timeout: 10s
scrape_configs:
- job_name: jmx
sample_limit: 10000
file_sd_configs:
- files: [ "C:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\prometheus_file_sd.yaml" ]
```
1. Set up the scrape targets configuration.
On Linux, enter the following command.
```
$ cat /opt/aws/amazon-cloudwatch-agent/var/prometheus_file_sd.yaml
- targets:
- 127.0.0.1:9404
labels:
application: sample_java_app
os: linux
```
On Windows, enter the following command.
```
PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus_file_sd.yaml
- targets:
- 127.0.0.1:9404
labels:
application: sample_java_app
os: windows
```
1. Set up the Prometheus scrape configuration by `ec2_sc_config`. Replace *your-ec2-instance-id* with the correct EC2 instance ID.
On Linux, enter the following command.
```
$ cat .\prometheus.yaml
global:
scrape_interval: 1m
scrape_timeout: 10s
scrape_configs:
- job_name: jmx
sample_limit: 10000
ec2_sd_configs:
- region: us-east-1
port: 9404
filters:
- name: instance-id
values:
- your-ec2-instance-id
```
On Windows, enter the following command.
```
PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus_file_sd.yaml
- targets:
- 127.0.0.1:9404
labels:
application: sample_java_app
os: windows
```
1. Set up the CloudWatch agent configuration. First, navigate to the correct directory. On Linux, it is `/opt/aws/amazon-cloudwatch-agent/var/cwagent-config.json`. On Windows, it is `C:\ProgramData\Amazon\AmazonCloudWatchAgent\cwagent-config.json`.
The following is a sample configuration with Java/JHX Prometheus metrics defined. Be sure to replace *path-to-Prometheus-Scrape-Configuration-file* with the correct path.
```
{
"agent": {
"region": "us-east-1"
},
"logs": {
"metrics_collected": {
"prometheus": {
"cluster_name": "my-cluster",
"log_group_name": "prometheus-test",
"prometheus_config_path": "path-to-Prometheus-Scrape-Configuration-file",
"emf_processor": {
"metric_declaration_dedup": true,
"metric_namespace": "PrometheusTest",
"metric_unit":{
"jvm_threads_current": "Count",
"jvm_classes_loaded": "Count",
"java_lang_operatingsystem_freephysicalmemorysize": "Bytes",
"catalina_manager_activesessions": "Count",
"jvm_gc_collection_seconds_sum": "Seconds",
"catalina_globalrequestprocessor_bytesreceived": "Bytes",
"jvm_memory_bytes_used": "Bytes",
"jvm_memory_pool_bytes_used": "Bytes"
},
"metric_declaration": [
{
"source_labels": ["job"],
"label_matcher": "^jmx$",
"dimensions": [["instance"]],
"metric_selectors": [
"^jvm_threads_current$",
"^jvm_classes_loaded$",
"^java_lang_operatingsystem_freephysicalmemorysize$",
"^catalina_manager_activesessions$",
"^jvm_gc_collection_seconds_sum$",
"^catalina_globalrequestprocessor_bytesreceived$"
]
},
{
"source_labels": ["job"],
"label_matcher": "^jmx$",
"dimensions": [["area"]],
"metric_selectors": [
"^jvm_memory_bytes_used$"
]
},
{
"source_labels": ["job"],
"label_matcher": "^jmx$",
"dimensions": [["pool"]],
"metric_selectors": [
"^jvm_memory_pool_bytes_used$"
]
}
]
}
}
},
"force_flush_interval": 5
}
}
```
1. Restart the CloudWatch agent by entering one of the following commands.
On Linux, enter the following command.
```
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/var/cwagent-config.json
```
On Windows, enter the following command.
```
& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:C:\ProgramData\Amazon\AmazonCloudWatchAgent\cwagent-config.json
```
### Viewing the Prometheus metrics and logs
You can now view the Java/JMX metrics being collected.
**To view the metrics for your sample Java/JMX workload**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the Region where your cluster is running, choose **Metrics** in the left navigation pane. Find the **PrometheusTest** namespace to see the metrics.
1. To see the CloudWatch Logs events, choose **Log groups** in the navigation pane. The events are in the log group **prometheus-test**.
# Configure CloudWatch agent service and environment names for related entities
The CloudWatch agent can send metrics and logs with entity data to support the [Explore related pane](ExploreRelated.md) in the CloudWatch console. The service name or environment name can be configured by the [CloudWatch Agent JSON configuration](CloudWatch-Agent-Configuration-File-Details.md).
**Note**
The agent configuration may be overridden. For details about how the agent decides what data to send for related entities, see [Using the CloudWatch agent with related telemetry](CloudWatch-Agent-RelatedEntities.md).
For metrics, it can be configured at the agent, metrics, or plugin level. For logs it can be configured at the agent, logs, or file level. The most specific configuration is always used. For example if the configuration exists at the agent level and metrics level, then metrics will use the metric configuration, and anything else (logs) will use the agent configuration. The following example shows different ways to configure the service name and environment name.
```
{
"agent": {
"service.name": "agent-level-service",
"deployment.environment": "agent-level-environment"
},
"metrics": {
"service.name": "metric-level-service",
"deployment.environment": "metric-level-environment",
"metrics_collected": {
"statsd": {
"service.name": "statsd-level-service",
"deployment.environment": "statsd-level-environment",
},
"collectd": {
"service.name": "collectdd-level-service",
"deployment.environment": "collectd-level-environment",
}
}
},
"logs": {
"service.name": "log-level-service",
"deployment.environment": "log-level-environment",
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
"log_group_name": "amazon-cloudwatch-agent.log",
"log_stream_name": "amazon-cloudwatch-agent.log",
"service.name": "file-level-service",
"deployment.environment": "file-level-environment"
}
]
}
}
}
}
```