# Collect metrics, logs, and traces using the CloudWatch agent CloudWatch agent is a software component that collects metrics, logs, and traces from your Amazon EC2 instances, on-premises servers, and containerized applications. It enables you to monitor your infrastructure and applications more comprehensively than the basic monitoring provided by default. **Key benefits** + Collect system-level metrics (CPU, memory, disk, network) + Gather custom metrics from your applications + Collect and centralize logs from various sources + Monitor both AWS and on-premises environments with a single tool + Set up alarms and notifications based on collected data The CloudWatch agent enables you to do the following: + Collect internal system-level metrics from Amazon EC2 instances across operating systems. The metrics can include in-guest metrics, in addition to the metrics for EC2 instances. The additional metrics that can be collected are listed in [Metrics collected by the CloudWatch agent](metrics-collected-by-CloudWatch-agent.md). + Collect system-level metrics from on-premises servers. These can include servers in a hybrid environment as well as servers not managed by AWS. + Retrieve custom metrics from your applications or services using the `StatsD` and `collectd` protocols. `StatsD` is supported on both Linux servers and servers running Windows Server. `collectd` is supported only on Linux servers. + Collect logs from Amazon EC2 instances and on-premises servers, running either Linux or Windows Server. **Note** The CloudWatch agent does not support collecting logs from FIFO pipes. + Send the metrics to either CloudWatch or Amazon Managed Service for Prometheus, or to both. The CloudWatch agent configuration file contains a `metrics_destinations` parameter in the `metrics` section. You can specify `cloudwatch`, `amp`, or both in this parameter. + Version 1.300031.0 and later can be used to enable CloudWatch Application Signals. For more information, see [Application Signals](CloudWatch-Application-Monitoring-Sections.md). + Version 1.300025.0 and later can collect traces from [OpenTelemetry](https://docs.aws.amazon.com/xray/latest/devguide/xray-instrumenting-your-app.html#xray-instrumenting-opentel) or [X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-instrumenting-your-app.html#xray-instrumenting-xray-sdk) client SDKs, and send them to X-Ray. Using the CloudWatch agent allows you to collect traces without needing to run a separate trace collection daemon, helping to reduce the number of agents that you run and manage. Metrics sent to CloudWatch can be viewed in CloudWatch just as any other CloudWatch metrics. The default CloudWatch namespace for metrics collected by the CloudWatch agent is `CWAgent`, although you can specify a different namespace when you configure the agent. The logs collected by the CloudWatch agent are processed and stored in Amazon CloudWatch Logs, just like logs collected by the older CloudWatch Logs agent. For information about CloudWatch Logs pricing, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing). Metrics collected by the CloudWatch agent are billed as custom metrics. For more information about CloudWatch metrics pricing, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing). The CloudWatch agent is open-source under the MIT license, and is [ hosted on GitHub](https://github.com/aws/amazon-cloudwatch-agent/). If you would like to build, customize or contribute to the CloudWatch agent, see the GitHub repository for the latest instructions. If you think you’ve found a potential security issue, do not post it on GitHub or any public forum. Instead, please follow the instructions at [ Vulnerability Reporting](https://aws.amazon.com/security/vulnerability-reporting/) or [ email AWS security directly](mailto:aws-security@amazon.com). You can download and install the CloudWatch agent manually using the command line, or you can integrate it with AWS Systems Manager. The general flow of installing the CloudWatch agent is as follows: 1. Create IAM roles or users that enable the agent to collect metrics from the server and optionally to integrate with AWS Systems Manager. 1. Download the agent package. 1. Modify the CloudWatch agent configuration file and specify the metrics that you want to collect. 1. Install and start the agent on your servers. **Topics** + [ # Supported operating systems ](supported-operating-systems.md) + [ # Prerequisites ](prerequisites.md) + [ # Download the CloudWatch agent package ](download-CloudWatch-Agent-on-EC2-Instance-commandline-first.md) + [ # Verifying the signature of the CloudWatch agent package ](verify-CloudWatch-Agent-Package-Signature.md) + [ # Installing the CloudWatch agent ](install-CloudWatch-Agent-on-EC2-Instance.md) + [ # Set up the CloudWatch agent with security-enhanced Linux (SELinux) ](CloudWatch-Agent-SELinux.md) + [ # Create the CloudWatch agent configuration file ](create-cloudwatch-agent-configuration-file.md) + [ # Starting the CloudWatch agent ](start-CloudWatch-Agent-on-premise-SSM-onprem.md) + [ # Metrics collected by the CloudWatch agent ](metrics-collected-by-CloudWatch-agent.md) + [ # Using the CloudWatch agent with related telemetry ](CloudWatch-Agent-RelatedEntities.md) + [ # Common scenarios with the CloudWatch agent ](CloudWatch-Agent-common-scenarios.md) + [ # CloudWatch agent credentials preference ](CloudWatch-Agent-Credentials-Preference.md) + [ # Troubleshooting the CloudWatch agent ](troubleshooting-CloudWatch-Agent.md) # Supported operating systems The CloudWatch agent is supported on the following operating systems: ## x86-64 architecture + Amazon Linux 2023 + Amazon Linux 2 + Ubuntu Server versions 25.04, 24.04, and 22.04 + Red Hat Enterprise Linux (RHEL) versions 10, 9, and 8 + Debian version 12 + SUSE Linux Enterprise Server (SLES) version 15 + Oracle Linux versions 9 and 8 + AlmaLinux versions 10, 9, and 8 + Rocky Linux versions 10, 9, and 8 + macOS computers: EC2 M1 Mac1 instances, and computers running macOS 14 (Sonoma), macOS 13 (Ventura), and macOS 12 (Monterey) + Windows Server 2025, Windows Server 2022, Windows Server 2019, and Windows Server 2016 + Windows 11 + 64-bit Windows 10 ## ARM64 architecture + Amazon Linux 2023 + Amazon Linux 2 + Ubuntu Server version 22.04 + Red Hat Enterprise Linux (RHEL) versions 9 and 8 + Debian version 12 + SUSE Linux Enterprise Server 15 + macOS computers: macOS 14 (Sonoma), macOS 13 (Ventura), and macOS 12 (Monterey) # Prerequisites Make sure the following prerequisites are completed before installing the CloudWatch agent for the first time. ## IAM roles and users for CloudWatch agent Access to AWS resources requires permissions. You create an IAM role, an IAM user, or both to grant permissions that the CloudWatch agent needs to write metrics to CloudWatch. ### Create an IAM role for Amazon EC2 instances If you're going to run the CloudWatch agent on Amazon EC2 instances, create an IAM role with the necessary permissions. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Roles** and then **Create role**. 1. Make sure that **AWS service** is selected under **Trusted entity type**. 1. For **Use case**, choose **EC2** under **Common use cases**. 1. Choose **Next**. 1. In the list of policies, select the check box next to **CloudWatchAgentServerPolicy**. If necessary, use the search box to find the policy. 1. Choose **Next**. 1. In **Role name**, enter a name for the role, such as `CloudWatchAgentServerRole`. Optionally give it a description. Then choose **Create role**. (Optional) If the agent is going to send logs to CloudWatch Logs and you want the agent to be able to set retention policies for these log groups, you need to add the `logs:PutRetentionPolicy` permission to the role. ### Create IAM user for on-premises servers If you're going to run the CloudWatch agent on on-premises servers, create an IAM user with the necessary permissions. **Note** This scenario requires IAM users with programmatic access and long-term credentials, which presents a security risk. To help mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Users** and then **Add users**. 1. Enter the user name for the new user. 1. Select **Access key - Programmatic access** and choose **Next: Permissions**. 1. Choose **Attach existing policies directly**. 1. In the list of policies, select the check box next to **CloudWatchAgentServerPolicy**. If necessary, use the search box to find the policy. 1. Choose **Next: Tags**. 1. Optionally create tags for the new IAM, and then choose **Next: Review**. 1. Confirm that the correct policy is listed, and choose **Create user**. 1. Next to the name of the new user, choose **Show**. Copy the access key and secret key to a file so that you can use them when installing the agent. Choose **Close**. ### Attaching an IAM role to an Amazon EC2 instance To enable the CloudWatch agent to send data from an Amazon EC2 instance, you must attach the IAM role you created to the instance. For more information on attaching an IAM role to an instance, see [Attaching an IAM Role to an Instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/iam-roles-for-amazon-ec2.html#attach-iam-role) in the Amazon Elastic Compute Cloud User Guide. ### Allowing the CloudWatch agent to set log retention policy You can configure the CloudWatch agent to set the retention policy for log groups that it sends log events to. If you do this, you must grant the `logs:PutRetentionPolicy` to the IAM role or user that the agent uses. The agent uses an IAM role to run on Amazon EC2 instances, and uses an IAM user for on-premises servers. **To grant the CloudWatch agent's IAM role permission to set log retention policies** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**. 1. In the search box, Type the beginning of the name of the CloudWatch agent's IAM role. You chose this name when you created the role. It might be named `CloudWatchAgentServerRole`. When you see the role, choose the name of the role. 1. In the **Permissions** tab, choose **Add permissions**, **Create inline policy**. 1. Choose the **JSON** tab and copy the following policy into the box, replacing the default JSON in the box: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "logs:PutRetentionPolicy", "Resource": "*" } ] } ``` ------ 1. Choose **Review policy**. 1. For **Name**, enter **CloudWatchAgentPutLogsRetention** or something similar, and choose **Create policy**. **To grant the CloudWatch agent's IAM user permission to set log retention policies** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Users**. 1. In the search box, Type the beginning of the name of the CloudWatch agent's IAM user. You chose this name when you created the user. When you see the user, choose the name of the user. 1. In the **Permissions** tab, choose **Add inline policy**. 1. Choose the **JSON** tab and copy the following policy into the box, replacing the default JSON in the box: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "logs:PutRetentionPolicy", "Resource": "*" } ] } ``` ------ 1. Choose **Review policy**. 1. For **Name**, enter **CloudWatchAgentPutLogsRetention** or something similar, and choose **Create policy**. ## Network requirements **Note** When the server is in a public subnet make sure there is access to an internet gateway. When the server is in a private subnet, access is through the NAT gateways or VPC Endpoint. For more information on NAT gateways, see [https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html). Your Amazon EC2 instances must have outbound internet access to send data to CloudWatch or CloudWatch Logs. For more information about how to configure internet access, see [Internet Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) in the Amazon VPC User Guide. ### Using VPC endpoints If you're using a VPC and want to use CloudWatch agent without public internet access, you can configure VPC endpoints for CloudWatch and CloudWatch Logs. The endpoints and ports to configure on your proxy are as follows: + If you're using the agent to collect metrics, you must add the CloudWatch endpoints for the appropriate Regions to the allow list. These endpoints are listed in [Amazon CloudWatch endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/cw_region.html). + If you're using the agent to collect logs, you must add the CloudWatch Logs endpoints for the appropriate Regions to the allow list. These endpoints are listed in [Amazon CloudWatch Logs endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/cwl_region.html). + If you're using Systems Manager to install the agent or Parameter Store to store your configuration file, you must add the Systems Manager endpoints for the appropriate Regions to the allow list. These endpoints are listed in [AWS Systems Manager endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/ssm.html). # Download the CloudWatch agent package **Note** To download the CloudWatch agent, your connection must use TLS 1.2 or later. The CloudWatch agent is available as a package in Amazon Linux 2023 and Amazon Linux 2. If you are using this operating system, you can install the package by entering the following command. You must also make sure that the IAM role attached to the instance has the **CloudWatchAgentServerPolicy** attached. ``` sudo yum install amazon-cloudwatch-agent ``` On all supported operating systems, you can download and install the CloudWatch agent using the command line. For each download link, there is a general link as well as links for each AWS Region. If you use the Region-specific links, replace the default Region (*us-east-1*) with the appropriate Region for your account. For example, for Amazon Linux 2023 and Amazon Linux 2 and the x86-64 architecture, three of the valid download links are: + `https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm` + `https://amazoncloudwatch-agent-us-east-1.s3.us-east-1.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm` + `https://amazoncloudwatch-agent-eu-central-1.s3.eu-central-1.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm` You can also download a README file about the latest changes to the agent, and a file that indicates the version number that is available for download. These files are in the following locations: **Note** If you use the Region-specific links, replace the default Region (*us-east-1*) with the appropriate Region for your account. + `https://amazoncloudwatch-agent.s3.amazonaws.com/info/latest/RELEASE_NOTES` or `https://amazoncloudwatch-agent-us-east-1.s3.us-east-1.amazonaws.com/info/latest/RELEASE_NOTES` + `https://amazoncloudwatch-agent.s3.amazonaws.com/info/latest/CWAGENT_VERSION` or `https://amazoncloudwatch-agent-us-east-1.s3.us-east-1.amazonaws.com/info/latest/CWAGENT_VERSION` | Architecture | Platform | Download link | Signature file link | | --- | --- | --- | --- | | x86-64 | Amazon Linux 2023 and Amazon Linux 2 | https://amazoncloudwatch-agent.s3.amazonaws.com/amazon\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/amazon\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/amazon\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/amazon\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | Centos | https://amazoncloudwatch-agent.s3.amazonaws.com/centos/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/centos/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/centos/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/centos/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | Redhat | https://amazoncloudwatch-agent.s3.amazonaws.com/redhat/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/redhat/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/redhat/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/redhat/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | SUSE | https://amazoncloudwatch-agent.s3.amazonaws.com/suse/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/suse/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/suse/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/suse/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | Debian | https://amazoncloudwatch-agent.s3.amazonaws.com/debian/amd64/latest/amazon-cloudwatch-agent.deb https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/debian/amd64/latest/amazon-cloudwatch-agent.deb | https://amazoncloudwatch-agent.s3.amazonaws.com/debian/amd64/latest/amazon-cloudwatch-agent.deb.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/debian/amd64/latest/amazon-cloudwatch-agent.deb.sig | | x86-64 | Ubuntu | https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb | https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb.sig | | x86-64 | Oracle | https://amazoncloudwatch-agent.s3.amazonaws.com/oracle\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/oracle\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/oracle\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/oracle\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | macOS | https://amazoncloudwatch-agent.s3.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg | https://amazoncloudwatch-agent.s3.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg.sig | | x86-64 | Windows | https://amazoncloudwatch-agent.s3.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi | https://amazoncloudwatch-agent.s3.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi.sig | | ARM64 | Amazon Linux 2023 and Amazon Linux 2 | https://amazoncloudwatch-agent.s3.amazonaws.com/amazon\$1linux/arm64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/amazon\$1linux/arm64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/amazon\$1linux/arm64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/amazon\$1linux/arm64/latest/amazon-cloudwatch-agent.rpm.sig | | ARM64 | Redhat | https://amazoncloudwatch-agent.s3.amazonaws.com/redhat/arm64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/redhat/arm64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/redhat/arm64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/redhat/arm64/latest/amazon-cloudwatch-agent.rpm.sig | | ARM64 | Ubuntu | https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb | https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb.sig | | ARM64 | Debian | https://amazoncloudwatch-agent.s3.amazonaws.com/debian/arm64/latest/amazon-cloudwatch-agent.deb https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/debian/arm64/latest/amazon-cloudwatch-agent.deb | https://amazoncloudwatch-agent.s3.amazonaws.com/debian/arm64/latest/amazon-cloudwatch-agent.deb.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/debian/arm64/latest/amazon-cloudwatch-agent.deb.sig | | ARM64 | SUSE | https://amazoncloudwatch-agent.s3.amazonaws.com/suse/arm64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/suse/arm64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/suse/arm64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/suse/arm64/latest/amazon-cloudwatch-agent.rpm.sig | | ARM64 | MacOS | https://amazoncloudwatch-agent.s3.amazonaws.com/darwin/arm64/latest/amazon-cloudwatch-agent.pkg https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/darwin/arm64/latest/amazon-cloudwatch-agent.pkg | https://amazoncloudwatch-agent.s3.amazonaws.com/darwin/arm64/latest/amazon-cloudwatch-agent.pkg.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/darwin/arm64/latest/amazon-cloudwatch-agent.pkg.sig | # Verifying the signature of the CloudWatch agent package GPG signature files are included for CloudWatch agent packages on Linux servers. You can use a public key to verify the agent download file is original and unmodified. For Windows Server, you can use the MSI to verify the signature. For macOS computers, the signature is included in the agent download package. To find the correct signature file, use the following table. For each architecture and operating system, you can see a general link and links for each Region. If you use the Region-specific links, replace the default Region (*us-east-1*) with the appropriate Region for your account. For example, for Amazon Linux 2023 and Amazon Linux 2 and the x86-64 architecture, three of the valid links are: + `https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig` + `https://amazoncloudwatch-agent-us-east-1.s3.us-east-1.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm` + `https://amazoncloudwatch-agent-eu-central-1.s3.eu-central-1.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm` **Note** To download the CloudWatch agent, your connection must use TLS 1.2 or later. | Architecture | Platform | Download link | Signature file link | | --- | --- | --- | --- | | x86-64 | Amazon Linux 2023 and Amazon Linux 2 | https://amazoncloudwatch-agent.s3.amazonaws.com/amazon\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/amazon\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/amazon\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/amazon\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | Centos | https://amazoncloudwatch-agent.s3.amazonaws.com/centos/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/centos/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/centos/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/centos/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | Redhat | https://amazoncloudwatch-agent.s3.amazonaws.com/redhat/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/redhat/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/redhat/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/redhat/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | SUSE | https://amazoncloudwatch-agent.s3.amazonaws.com/suse/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/suse/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/suse/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/suse/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | Debian | https://amazoncloudwatch-agent.s3.amazonaws.com/debian/amd64/latest/amazon-cloudwatch-agent.deb https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/debian/amd64/latest/amazon-cloudwatch-agent.deb | https://amazoncloudwatch-agent.s3.amazonaws.com/debian/amd64/latest/amazon-cloudwatch-agent.deb.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/debian/amd64/latest/amazon-cloudwatch-agent.deb.sig | | x86-64 | Ubuntu | https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb | https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb.sig | | x86-64 | Oracle | https://amazoncloudwatch-agent.s3.amazonaws.com/oracle\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/oracle\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/oracle\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/oracle\$1linux/amd64/latest/amazon-cloudwatch-agent.rpm.sig | | x86-64 | macOS | https://amazoncloudwatch-agent.s3.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg | https://amazoncloudwatch-agent.s3.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg.sig | | x86-64 | Windows | https://amazoncloudwatch-agent.s3.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi | https://amazoncloudwatch-agent.s3.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi.sig | | ARM64 | Amazon Linux 2023 and Amazon Linux 2 | https://amazoncloudwatch-agent.s3.amazonaws.com/amazon\$1linux/arm64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/amazon\$1linux/arm64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/amazon\$1linux/arm64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/amazon\$1linux/arm64/latest/amazon-cloudwatch-agent.rpm.sig | | ARM64 | Redhat | https://amazoncloudwatch-agent.s3.amazonaws.com/redhat/arm64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/redhat/arm64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/redhat/arm64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/redhat/arm64/latest/amazon-cloudwatch-agent.rpm.sig | | ARM64 | Ubuntu | https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb | https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb.sig | | ARM64 | Debian | https://amazoncloudwatch-agent.s3.amazonaws.com/debian/arm64/latest/amazon-cloudwatch-agent.deb https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/debian/arm64/latest/amazon-cloudwatch-agent.deb | https://amazoncloudwatch-agent.s3.amazonaws.com/debian/arm64/latest/amazon-cloudwatch-agent.deb.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/debian/arm64/latest/amazon-cloudwatch-agent.deb.sig | | ARM64 | SUSE | https://amazoncloudwatch-agent.s3.amazonaws.com/suse/arm64/latest/amazon-cloudwatch-agent.rpm https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/suse/arm64/latest/amazon-cloudwatch-agent.rpm | https://amazoncloudwatch-agent.s3.amazonaws.com/suse/arm64/latest/amazon-cloudwatch-agent.rpm.sig https://amazoncloudwatch-agent-*us-east-1*.s3.*us-east-1*.amazonaws.com/suse/arm64/latest/amazon-cloudwatch-agent.rpm.sig | **To verify the CloudWatch agent package on a Linux server** 1. Download the public key. ``` shell$ wget https://amazoncloudwatch-agent.s3.amazonaws.com/assets/amazon-cloudwatch-agent.gpg ``` 1. Import the public key into your keyring. ``` shell$ gpg --import amazon-cloudwatch-agent.gpg gpg: key 3B789C72: public key "Amazon CloudWatch Agent" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) ``` Make a note of the key value, as you need it in the next step. In the preceding example, the key value is `3B789C72`. 1. Verify the fingerprint by running the following command, replacing *key-value* with the value from the preceding step: ``` shell$ gpg --fingerprint key-value pub 2048R/3B789C72 2017-11-14 Key fingerprint = 9376 16F3 450B 7D80 6CBD 9725 D581 6730 3B78 9C72 uid Amazon CloudWatch Agent ``` The fingerprint string should be equal to the following: `9376 16F3 450B 7D80 6CBD 9725 D581 6730 3B78 9C72` If the fingerprint string doesn't match, don't install the agent. Contact Amazon Web Services. After you have verified the fingerprint, you can use it to verify the signature of the CloudWatch agent package. 1. Download the package signature file using **wget**. To determine the correct signature file, see the preceding table. ``` wget Signature File Link ``` 1. To verify the signature, run **gpg --verify**. ``` shell$ gpg --verify signature-filename agent-download-filename gpg: Signature made Wed 29 Nov 2017 03:00:59 PM PST using RSA key ID 3B789C72 gpg: Good signature from "Amazon CloudWatch Agent" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 9376 16F3 450B 7D80 6CBD 9725 D581 6730 3B78 9C72 ``` If the output includes the phrase `BAD signature`, check whether you performed the procedure correctly. If you continue to get this response, contact Amazon Web Services and avoid using the downloaded file. Note the warning about trust. A key is trusted only if you or someone who you trust has signed it. This doesn't mean that the signature is invalid, only that you have not verified the public key. **To verify the CloudWatch agent package on a server running Windows Server** 1. Download and install GnuPG for Windows from [https://gnupg.org/download/](https://gnupg.org/download/). When installing, include the **Shell Extension (GpgEx)** option. You can perform the remaining steps in Windows PowerShell. 1. Download the public key. ``` PS> wget https://amazoncloudwatch-agent.s3.amazonaws.com/assets/amazon-cloudwatch-agent.gpg -OutFile amazon-cloudwatch-agent.gpg ``` 1. Import the public key into your keyring. ``` PS> gpg --import amazon-cloudwatch-agent.gpg gpg: key 3B789C72: public key "Amazon CloudWatch Agent" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) ``` Make a note of the key value because you need it in the next step. In the preceding example, the key value is `3B789C72`. 1. Verify the fingerprint by running the following command, replacing *key-value* with the value from the preceding step: ``` PS> gpg --fingerprint key-value pub rsa2048 2017-11-14 [SC] 9376 16F3 450B 7D80 6CBD 9725 D581 6730 3B78 9C72 uid [ unknown] Amazon CloudWatch Agent ``` The fingerprint string should be equal to the following: `9376 16F3 450B 7D80 6CBD 9725 D581 6730 3B78 9C72` If the fingerprint string doesn't match, don't install the agent. Contact Amazon Web Services. After you have verified the fingerprint, you can use it to verify the signature of the CloudWatch agent package. 1. Download the package signature file using wget. To determine the correct signature file, see [CloudWatch Agent Download Links](download-CloudWatch-Agent-on-EC2-Instance-commandline-first.md#agent-download-link-table). 1. To verify the signature, run **gpg --verify**. ``` PS> gpg --verify sig-filename agent-download-filename gpg: Signature made 11/29/17 23:00:45 Coordinated Universal Time gpg: using RSA key D58167303B789C72 gpg: Good signature from "Amazon CloudWatch Agent" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 9376 16F3 450B 7D80 6CBD 9725 D581 6730 3B78 9C72 ``` If the output includes the phrase `BAD signature`, check whether you performed the procedure correctly. If you continue to get this response, contact Amazon Web Services and avoid using the downloaded file. Note the warning about trust. A key is trusted only if you or someone who you trust has signed it. This doesn't mean that the signature is invalid, only that you have not verified the public key. **To verify the CloudWatch agent package on a macOS computer** + There are two methods for signature verification on macOS. + Verify the fingerprint by running the following command. ``` pkgutil --check-signature amazon-cloudwatch-agent.pkg ``` You should see a result similar to the following. ``` Package "amazon-cloudwatch-agent.pkg": Status: signed by a developer certificate issued by Apple for distribution Signed with a trusted timestamp on: 2020-10-02 18:13:24 +0000 Certificate Chain: 1. Developer ID Installer: AMZN Mobile LLC (94KV3E626L) Expires: 2024-10-18 22:31:30 +0000 SHA256 Fingerprint: 81 B4 6F AF 1C CA E1 E8 3C 6F FB 9E 52 5E 84 02 6E 7F 17 21 8E FB 0C 40 79 13 66 8D 9F 1F 10 1C ------------------------------------------------------------------------ 2. Developer ID Certification Authority Expires: 2027-02-01 22:12:15 +0000 SHA256 Fingerprint: 7A FC 9D 01 A6 2F 03 A2 DE 96 37 93 6D 4A FE 68 09 0D 2D E1 8D 03 F2 9C 88 CF B0 B1 BA 63 58 7F ------------------------------------------------------------------------ 3. Apple Root CA Expires: 2035-02-09 21:40:36 +0000 SHA256 Fingerprint: B0 B1 73 0E CB C7 FF 45 05 14 2C 49 F1 29 5E 6E DA 6B CA ED 7E 2C 68 C5 BE 91 B5 A1 10 01 F0 24 ``` + Or, download and use the .sig file To use this method, follow these steps. 1. Install the GPG application to your macOS host by entering the following command. ``` brew install GnuPG ``` + Download the package signature file using curl. To determine the correct signature file, see [CloudWatch Agent Download Links](download-CloudWatch-Agent-on-EC2-Instance-commandline-first.md#agent-download-link-table). + To verify the signature, run **gpg --verify**. ``` PS> gpg --verify sig-filename agent-download-filename gpg: Signature made 11/29/17 23:00:45 Coordinated Universal Time gpg: using RSA key D58167303B789C72 gpg: Good signature from "Amazon CloudWatch Agent" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 9376 16F3 450B 7D80 6CBD 9725 D581 6730 3B78 9C72 ``` If the output includes the phrase `BAD signature`, check whether you performed the procedure correctly. If you continue to get this response, contact Amazon Web Services and avoid using the downloaded file. Note the warning about trust. A key is trusted only if you or someone who you trust has signed it. This doesn't mean that the signature is invalid, only that you have not verified the public key. # Installing the CloudWatch agent You can install the CloudWatch agent on your Amazon EC2 instances, on-premises servers, and in containerized environments to collect metrics, logs, and traces. The agent supports Linux, Windows, and macOS operating systems. There are several ways to install the agent, including using Systems Manager, from the CloudWatch console, installing from the command line, or installing using a configuration file. Before installing, make sure you have the required IAM permissions and network access configured. **Topics** + [ # Install and Configure Amazon CloudWatch Agent with Workload Detection in the CloudWatch console ](install-cloudwatch-agent-workload-detection.md) + [ # Manual installation on Amazon EC2 ](manual-installation.md) + [ # Install the CloudWatch agent using AWS Systems Manager ](installing-cloudwatch-agent-ssm.md) + [ # Install the CloudWatch agent on on-premises servers ](install-CloudWatch-Agent-on-premise.md) + [ # Install the CloudWatch agent on new instances using CloudFormation ](Install-CloudWatch-Agent-New-Instances-CloudFormation.md) + [ # Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart ](install-CloudWatch-Observability-EKS-addon.md) # Install and Configure Amazon CloudWatch Agent with Workload Detection in the CloudWatch console ## Introduction You can use the CloudWatch Getting Started console to install and configure the CloudWatch agent on Amazon EC2 instances. The Amazon CloudWatch agent is a lightweight software component that collects system-level metrics, logs, and traces from your Amazon EC2 instances. By automating the collection and delivery of monitoring data to CloudWatch, the agent enables you to gain actionable insights, optimize resource utilization, and ensure your applications are running smoothly with minimal configuration effort. Configure the CloudWatch agent with pre-defined, workload-specific configurations that leverage automatic workload detection to identify running applications and services on your instances. You can customize data collection with specific metrics, logs, and traces, enabling you to monitor application performance and troubleshoot issues effectively. ## How it Works The CloudWatch agent detects workloads running on your Amazon EC2 instances through automatic workload detection capabilities. This feature identifies running applications and services on your instances, enabling intelligent monitoring without manual configuration. Observability solutions provide pre-defined, workload-specific configurations tailored to common applications such as Apache Kafka, Apache Tomcat, Java Virtual Machines (JVM), NGINX, and NVIDIA GPU workloads. These solutions streamline the monitoring setup by automatically collecting the right metrics, logs, and traces specific to each detected workload, eliminating the need for manual instrumentation and configuration. When workload detection is enabled, the agent analyzes your instance environment and automatically selects relevant pre-configured monitoring templates. These configurations are optimized by AWS subject-matter experts to capture the most important telemetry data for each workload type, ensuring you have comprehensive observability from the start. ## Prerequisites ### SSM Agent Installation (Required) You must have AWS Systems Manager (SSM) agent installed on your Amazon EC2 instances. SSM agent comes pre-installed on most AWS-supplied Amazon Machine Images (AMIs), If you need to manually install or update the SSM agent, refer to the [Systems Manager documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html). **Note** Default Host Management Configuration (DHMC) is a Systems Manager feature that automatically grants Amazon EC2 instances permissions to connect to Systems Manager without requiring you to manually attach an IAM instance profile to each instance. If your Amazon EC2 instances are using DHMC and the CloudWatch agent installation process attaches the CloudWatch policy to your instances, it can take up to 30 minutes for the new policy to take effect. This delay can postpone the publication of metrics, logs, and traces to CloudWatch. To mitigate, you can create your Amazon EC2 instances with an IAM role that contains the [AmazonSSMManagedInstanceCore](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSSMManagedInstanceCore.html) policy. ### Workload Detection (Recommended) Workload detection is an opt-in feature that automatically identifies running applications and services on your instances. We recommend turning on workload detection to take advantage of pre-configured, workload-specific monitoring templates. You can enable workload detection in the [CloudWatch console settings](https://console.aws.amazon.com/cloudwatch/home#settings). ## Getting started Open the Getting Started with Amazon CloudWatch agent page in the Amazon CloudWatch console: [https://console.aws.amazon.com/cloudwatch/home\$1cloudwatch-agent](https://console.aws.amazon.com/cloudwatch/home#cloudwatch-agent) **Manual instance deployment for CloudWatch Agent** Manually select up to 50 instances for CloudWatch agent installation and configuration. This targeted approach allows you to enhance monitoring for specific Amazon EC2 instances. **Tag-based deployment for CloudWatch Agent** Use tag-based deployment to install and configure the CloudWatch agent on fleets of Amazon EC2 instances. This method applies to all current and future instances with matching tags. **Tag-based configuration** Tag-based configurations allow you to efficiently organize, view, and modify configurations, helping you to manage CloudWatch agent and its configuration across fleets of Amazon EC2 instances. **CloudWatch agent installation** Install the CloudWatch agent to collect metrics, logs, and traces from Amazon EC2 instances and on-premises hosts. This telemetry provides important health and performance data about your infrastructure and applications. **CloudWatch agent configuration** Configure the CloudWatch agent with pre-defined, workload-specific configurations. You can customize data collection with specific metrics, logs, and traces, enabling you to monitor application performance and troubleshoot issues effectively. ## Costs Additional metrics that you add during this process are billed as custom metrics. For more information about CloudWatch metrics pricing, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing). # Manual installation on Amazon EC2 **Note** Make sure the prerequisites are completed before installing the CloudWatch agent for the first time. **Topics** + [ ## Installing on Amazon Linux using the package manager ](#amazon-linux-package) + [ ## Installing on Amazon Linux using the command line ](#linux-manual-install) + [ ## Installing on Windows ](#windows-installation) + [ ## Installing on macOS ](#macos-installation) ## Installing on Amazon Linux using the package manager The CloudWatch agent is available as a package in Amazon Linux 2023 and Amazon Linux 2. If you are using one of these operating systems, you can install the package by entering the following command: ``` sudo yum install amazon-cloudwatch-agent ``` You must also make sure that the IAM role attached to the instance has the **CloudWatchAgentServerPolicy** attached. ## Installing on Amazon Linux using the command line On all supported Linux operating systems, you can download and install the CloudWatch agent using the command line. 1. Download the CloudWatch agent. For a Linux server, enter the following command. For `download-link`, use the appropriate download link from the table below. ``` wget download-link ``` [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/manual-installation.html) 1. After you have downloaded the package, you can optionally verify the package signature. For more information, see [Verifying the signature of the CloudWatch agent package](verify-CloudWatch-Agent-Package-Signature.md). 1. Install the package. If you downloaded an RPM package on a Linux server, change to the directory containing the package and enter the following: ``` sudo rpm -U ./amazon-cloudwatch-agent.rpm ``` If you downloaded a DEB package on a Linux server, change to the directory containing the package and enter the following: ``` sudo dpkg -i -E ./amazon-cloudwatch-agent.deb ``` ## Installing on Windows On Windows Server, you can download and install the CloudWatch agent using the command line. 1. Download the following file: ``` https://amazoncloudwatch-agent.s3.amazonaws.com/windows/amd64/latest/amazon-cloudwatch-agent.msi ``` 1. After you have downloaded the package, you can optionally verify the package signature. For more information, see [Verifying the signature of the CloudWatch agent package](verify-CloudWatch-Agent-Package-Signature.md). 1. Install the package. Change to the directory containing the package and enter the following: ``` msiexec /i amazon-cloudwatch-agent.msi ``` This command also works from within PowerShell. For more information about MSI command options, see [Command-Line Options](https://docs.microsoft.com/en-us/windows/desktop/Msi/command-line-options) in the Microsoft Windows documentation. ## Installing on macOS On macOS computers, you can download and install the CloudWatch agent using the command line. 1. Download the appropriate package for your architecture: ``` https://amazoncloudwatch-agent.s3.amazonaws.com/darwin/amd64/latest/amazon-cloudwatch-agent.pkg ``` For ARM64 architecture: ``` https://amazoncloudwatch-agent.s3.amazonaws.com/darwin/arm64/latest/amazon-cloudwatch-agent.pkg ``` 1. After you have downloaded the package, you can optionally verify the package signature. For more information, see [Verifying the signature of the CloudWatch agent package](verify-CloudWatch-Agent-Package-Signature.md). 1. Install the package. Change to the directory containing the package and enter the following: ``` sudo installer -pkg ./amazon-cloudwatch-agent.pkg -target / ``` # Install the CloudWatch agent using AWS Systems Manager Using AWS Systems Manager makes it easier to install the CloudWatch agent on a fleet of Amazon EC2 instances. You can download the agent into one server and create your CloudWatch agent configuration file for all servers in the fleet. Then you can use Systems Manager to install the agent on the other servers, using the configuration file that you created. Use the following topics to install and run the CloudWatch agent using AWS Systems Manager. **Topics** + [ ## Install or update the SSM Agent ](#update-SSM-Agent-EC2instance-first) + [ ## Verify Systems Manager prerequisites ](#install-CloudWatch-Agent-minimum-requirements-first) + [ ## Verify internet access ](#install-CloudWatch-Agent-internet-access-first) + [ ## Download the CloudWatch agent package to your first instance ](#install-CloudWatch-Agent-EC2-first) + [ ## Create and modify the agent configuration file ](#CW-Agent-Instance-Create-Configuration-File-first) + [ ## Install and start the CloudWatch agent on additional EC2 instances using your agent configuration ](#install-CloudWatch-Agent-on-EC2-Instance-fleet) + [ ## Install the CloudWatch agent on additional EC2 instances using your agent configuration ](#install-CloudWatch-Agent-on-EC2-Instance-fleet) + [ ## (Optional) Modify the common configuration and named profile for CloudWatch agent ](#CloudWatch-Agent-profile-instance-fleet) ## Install or update the SSM Agent On an Amazon EC2 instance, the CloudWatch agent requires that the instance is running version 2.2.93.0 or later of the SSM Agent. Before you install the CloudWatch agent, update or install SSM Agent on the instance if you haven't already done so. For information about installing or updating SSM Agent on an instance running Linux, see [ Installing and Configuring SSM Agent on Linux Instances](https://docs.aws.amazon.com/systems-manager/latest/userguide/manually-install-ssm-agent-linux.html) in the *AWS Systems Manager User Guide*. For information about installing or updating the SSM Agent, see [Working with SSM Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html) in the *AWS Systems Manager User Guide*. ## Verify Systems Manager prerequisites Before you use Systems Manager Run Command to install and configure the CloudWatch agent, verify that your instances meet the minimum Systems Manager requirements. For more information, see [Systems Manager Prerequisites](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-setting-up.html#systems-manager-prereqs) in the *AWS Systems Manager User Guide*. ## Verify internet access Your Amazon EC2 instances must be able to connect to CloudWatch endpoints. This can be by Internet Gateway, NAT gateway, or CloudWatch Interface VPC endpoints. For more information about how to configure internet access, see [Internet Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) in the *Amazon VPC User Guide*. The endpoints and ports to configure on your proxy are as follows: + If you're using the agent to collect metrics, you must allow list the CloudWatch endpoints for the appropriate Regions. These endpoints are listed in [Amazon CloudWatch](https://docs.aws.amazon.com/general/latest/gr/rande.html#cw_region) in the *Amazon Web Services General Reference*. + If you're using the agent to collect logs, you must allow list the CloudWatch Logs endpoints for the appropriate Regions. These endpoints are listed in [Amazon CloudWatch Logs](https://docs.aws.amazon.com/general/latest/gr/rande.html#cwl_region) in the *Amazon Web Services General Reference*. + If you're using Systems Manager to install the agent or Parameter Store to store your configuration file, you must allow list the Systems Manager endpoints for the appropriate Regions. These endpoints are listed in [AWS Systems Manager](https://docs.aws.amazon.com/general/latest/gr/rande.html#ssm_region) in the *Amazon Web Services General Reference*. ## Download the CloudWatch agent package to your first instance Use the following steps to download the CloudWatch agent package using Systems Manager. **To download the CloudWatch agent using Systems Manager** 1. Open the Systems Manager console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/). 1. In the navigation pane, choose **Run Command**. -or- If the AWS Systems Manager home page opens, scroll down and choose **Explore Run Command**. 1. Choose **Run command**. 1. In the **Command document** list, choose **AWS-ConfigureAWSPackage**. 1. In the **Targets** area, choose the instance to install the CloudWatch agent on. If you don't see a specific instance, it might not be configured as a managed instance for use with Systems Manager. For more information, see [Setting Up AWS Systems Manager for Hybrid Environments](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-managedinstances.html) in the *AWS Systems Manager User Guide*. 1. In the **Action** list, choose **Install**. 1. In the **Name** field, enter *AmazonCloudWatchAgent*. 1. Keep **Version** set to **latest** to install the latest version of the agent. 1. Choose **Run**. 1. Optionally, in the **Targets and outputs** areas, select the button next to an instance name and choose **View output**. Systems Manager should show that the agent was successfully installed. ## Create and modify the agent configuration file After you have downloaded the CloudWatch agent, you must create the configuration file before you start the agent on any servers. If you're going to save your agent configuration file in the Systems Manager Parameter Store, you must use an EC2 instance to save to the Parameter Store. Additionally, you must first attach to that instance the `CloudWatchAgentAdminRole` IAM role. For more information about attaching roles, see [Attaching an IAM Role to an Instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/iam-roles-for-amazon-ec2.html#attach-iam-role) in the *Amazon EC2 User Guide*. For more information about creating the CloudWatch agent configuration file, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md). ## Install and start the CloudWatch agent on additional EC2 instances using your agent configuration After you have a CloudWatch agent configuration saved in Parameter Store, you can use it when you install the agent on other servers. For each of these servers, follow the steps listed previously in this section to verify the Systems Manager prerequisites, the version of the SSM Agent, and internet access. Then use the following instructions to install the CloudWatch agent on the additional instances, using the CloudWatch agent configuration file that you have created. **Step 1: Download and install the CloudWatch agent** To be able to send the CloudWatch data to a different Region, make sure that the IAM role that you attached to this instance has permissions to write the CloudWatch data in that Region. Following is an example of using the `aws configure` command to create a named profile for the CloudWatch agent. This example assumes that you are using the default profile name of `AmazonCloudWatchAgent`. **To create the AmazonCloudWatchAgent profile for the CloudWatch agent** + On Linux servers, type the following command and follow the prompts: ``` sudo aws configure --profile AmazonCloudWatchAgent ``` On Windows Server, open PowerShell as an administrator, type the following command and follow the prompts. ``` aws configure --profile AmazonCloudWatchAgent ``` ## Install the CloudWatch agent on additional EC2 instances using your agent configuration After you have a CloudWatch agent configuration saved in Parameter Store, you can use it when you install the agent on other servers. For each of these servers, follow the steps listed previously in this section to verify the Systems Manager prerequisites, the version of the SSM Agent, and internet access. Then use the following instructions to install the CloudWatch agent on the additional instances, using the CloudWatch agent configuration file that you have created. **Step 1: Download and install the CloudWatch agent** You need to install the agent on each server where you will run the agent. The CloudWatch agent is available as a package in Amazon Linux 2023 and Amazon Linux 2. If you are using this operating system, you can use Systems Manager to install the package by using the following steps. **Note** You must also make sure that the IAM role attached to the instance has the **CloudWatchAgentServerPolicy** attached. For more information, see [Prerequisites](prerequisites.md). **To use Systems Manager to install the CloudWatch agent package** 1. Open the Systems Manager console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/). 1. In the navigation pane, choose **Run Command**. -or- If the AWS Systems Manager home page opens, scroll down and choose **Explore Run Command**. 1. Choose **Run command**. 1. In the **Command document** list, select **AWS-RunShellScript**. Then paste the following into **Command parameters**. ``` sudo yum install amazon-cloudwatch-agent ``` 1. Choose **Run** On all supported operating systems, you can download the CloudWatch agent package using either Systems Manager Run Command or an Amazon S3 download link. **Note** When you install or update the CloudWatch agent, only the **Uninstall and reinstall** option is supported. You can't use the **In-place update** option. Systems Manager Run Command enables you to manage the configuration of your instances. You specify a Systems Manager document, specify parameters, and execute the command on one or more instances. SSM Agent on the instance processes the command and configures the instance as specified. **To download the CloudWatch agent using Run Command** 1. Open the Systems Manager console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/). 1. In the navigation pane, choose **Run Command**. -or- If the AWS Systems Manager home page opens, scroll down and choose **Explore Run Command**. 1. Choose **Run command**. 1. In the **Command document** list, choose **AWS-ConfigureAWSPackage**. 1. In the **Targets** area, choose the instance on which to install the CloudWatch agent. If you do not see a specific instance, it might not be configured for Run Command. For more information, see [Setting Up AWS Systems Manager for Hybrid Environments](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-managedinstances.html) in the *AWS Systems Manager User Guide*. 1. In the **Action** list, choose **Install**. 1. In the **Name** box, enter *AmazonCloudWatchAgent*. 1. Keep **Version** set to **latest** to install the latest version of the agent. 1. Choose **Run**. 1. Optionally, in the **Targets and outputs** areas, select the button next to an instance name and choose **View output**. Systems Manager should show that the agent was successfully installed. **Step 2: Start the CloudWatch agent using your agent configuration file** Follow these steps to start the agent using Systems Manager Run Command. For information about setting up the agent on a system that has security-enhanced Linux (SELinux) enabled, see [Set up the CloudWatch agent with security-enhanced Linux (SELinux)](CloudWatch-Agent-SELinux.md). **To start the CloudWatch agent using Run Command** 1. Open the Systems Manager console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/). 1. In the navigation pane, choose **Run Command**. -or- If the AWS Systems Manager home page opens, scroll down and choose **Explore Run Command**. 1. Choose **Run command**. 1. In the **Command document** list, choose **AmazonCloudWatch-ManageAgent**. 1. In the **Targets** area, choose the instance where you installed the CloudWatch agent. 1. In the **Action** list, choose **configure**. 1. In the **Optional Configuration Source** list, choose **ssm**. 1. In the **Optional Configuration Location** box, enter the name of the Systems Manager parameter name of the agent configuration file that you created and saved to Systems Manager Parameter Store, as explained in [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md). 1. In the **Optional Restart** list, choose **yes** to start the agent after you have finished these steps. 1. Choose **Run**. 1. Optionally, in the **Targets and outputs** areas, select the button next to an instance name and choose **View output**. Systems Manager should show that the agent was successfully started. ## (Optional) Modify the common configuration and named profile for CloudWatch agent The CloudWatch agent includes a configuration file called `common-config.toml`. You can use this file to optionally specify proxy and Region information. On a server running Linux, this file is in the `/opt/aws/amazon-cloudwatch-agent/etc` directory. On a server running Windows Server, this file is in the `C:\ProgramData\Amazon\AmazonCloudWatchAgent` directory. The default `common-config.toml` is as follows: ``` # This common-config is used to configure items used for both ssm and cloudwatch access ## Configuration for shared credential. ## Default credential strategy will be used if it is absent here: ## Instance role is used for EC2 case by default. ## AmazonCloudWatchAgent profile is used for onPremise case by default. # [credentials] # shared_credential_profile = "{profile_name}" # shared_credential_file= "{file_name}" ## Configuration for proxy. ## System-wide environment-variable will be read if it is absent here. ## i.e. HTTP_PROXY/http_proxy; HTTPS_PROXY/https_proxy; NO_PROXY/no_proxy ## Note: system-wide environment-variable is not accessible when using ssm run-command. ## Absent in both here and environment-variable means no proxy will be used. # [proxy] # http_proxy = "{http_url}" # https_proxy = "{https_url}" # no_proxy = "{domain}" ``` All lines are commented out initially. To set the credential profile or proxy settings, remove the `#` from that line and specify a value. You can edit this file manually, or by using the `RunShellScript` Run Command in Systems Manager: + `shared_credential_profile` – For on-premises servers, this line specifies the IAM user credential profile to use to send data to CloudWatch. If you keep this line commented out, `AmazonCloudWatchAgent` is used. On an EC2 instance, you can use this line to have the CloudWatch agent send data from this instance to CloudWatch in a different AWS Region. To do so, specify a named profile that includes a `region` field specifying the name of the Region to send to. If you specify a `shared_credential_profile`, you must also remove the `#` from the beginning of the `[credentials]` line. + `shared_credential_file` – To have the agent look for credentials in a file located in a path other than the default path, specify that complete path and file name here. The default path is `/root/.aws` on Linux and is `C:\\Users\\Administrator\\.aws` on Windows Server. The first example below shows the syntax of a valid `shared_credential_file` line for Linux servers, and the second example is valid for Windows Server. On Windows Server, you must escape the \$1 characters. ``` shared_credential_file= "/usr/username/credentials" ``` ``` shared_credential_file= "C:\\Documents and Settings\\username\\.aws\\credentials" ``` If you specify a `shared_credential_file`, you must also remove the `#` from the beginning of the `[credentials]` line. + Proxy settings – If your servers use HTTP or HTTPS proxies to contact AWS services, specify those proxies in the `http_proxy` and `https_proxy` fields. If there are URLs that should be excluded from proxying, specify them in the `no_proxy` field, separated by commas. # Install the CloudWatch agent on on-premises servers If you downloaded the CloudWatch agent on a computer and created your agent configuration file, you can use that configuration file to install the agent in other on-premises servers. ## Download the CloudWatch agent on an on-premises server You can download the CloudWatch agent package using either Systems Manager Run Command or an Amazon S3 download link. ### Download using Systems Manager To use Systems Manager Run Command, you must register your on-premises server with Amazon EC2 Systems Manager. For more information, see [ Setting Up Systems Manager in Hybrid Environments](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-managedinstances.html) in the *AWS Systems Manager User Guide*. If you have already registered your server, update SSM Agent to the latest version. For information about updating SSM Agent on a server running Linux, see [Install SSM Agent for a Hybrid Environment (Linux)](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-managedinstances.html#sysman-install-managed-linux) in the *AWS Systems Manager User Guide*. For information about updating the SSM Agent on a server running Windows Server, see [ Install SSM Agent for a Hybrid Environment (Windows)](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-managedinstances.html#sysman-install-managed-win) in the *AWS Systems Manager User Guide*. **To use the SSM Agent to download the CloudWatch agent package on an on-premises server** 1. Open the Systems Manager console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/). 1. In the navigation pane, choose **Run Command**. -or- If the AWS Systems Manager home page opens, scroll down and choose **Explore Run Command**. 1. Choose **Run command**. 1. In the **Command document** list, select the button next to **AWS-ConfigureAWSPackage**. 1. In the **Targets** area, select the server to install the CloudWatch agent on. If you don't see a specific server, it might not be configured for Run Command. For more information, see [Setting Up AWS Systems Manager for Hybrid Environments](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-managedinstances.html) in the *AWS Systems Manager User Guide*. 1. In the **Action** list, choose **Install**. 1. In the **Name** box, enter *AmazonCloudWatchAgent*. 1. Keep **Version** blank to install the latest version of the agent. 1. Choose **Run**. The agent package is downloaded, and the next steps are to configure and start it. ## (Installing on an on-premises server) Specify IAM credentials and AWS Region To enable the CloudWatch agent to send data from an on-premises server, you must specify the access key and secret key of the IAM user that you created earlier. You also must specify the AWS Region to send the metrics to, using the `region` field. Following is an example of this file. ``` [AmazonCloudWatchAgent] aws_access_key_id=my_access_key aws_secret_access_key=my_secret_key region = us-west-1 ``` For *my\$1access\$1key* and *my\$1secret\$1key*, use the keys from the IAM user that doesn't have the permissions to write to Systems Manager Parameter Store. If you name this profile `AmazonCloudWatchAgent`, you don't need to do anything more. Optionally, you can give it a different name and specify that name as the value for `shared_credential_profile` in the` common-config.toml` file, which is explained in the following section. Following is an example of using the **aws configure** command to create a named profile for the CloudWatch agent. This example assumes that you're using the default profile name of `AmazonCloudWatchAgent`. **To create the AmazonCloudWatchAgent profile for the CloudWatch agent** 1. If you haven't already done so, install the AWS Command Line Interface on the server. For more information, see [ Installing the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html). 1. On Linux servers, enter the following command and follow the prompts: ``` sudo aws configure --profile AmazonCloudWatchAgent ``` On Windows Server, open PowerShell as an administrator, enter the following command, and follow the prompts. ``` aws configure --profile AmazonCloudWatchAgent ``` ## (Optional) Modifying the common configuration and named profile for CloudWatch agent The CloudWatch agent includes a configuration file called `common-config.toml`. You can optionally use this file to specify proxy and Region information. On a server running Linux, this file is in the `/opt/aws/amazon-cloudwatch-agent/etc` directory. On a server running Windows Server, this file is in the `C:\ProgramData\Amazon\AmazonCloudWatchAgent` directory. The default `common-config.toml` is as follows: ``` # This common-config is used to configure items used for both ssm and cloudwatch access ## Configuration for shared credential. ## Default credential strategy will be used if it is absent here: ## Instance role is used for EC2 case by default. ## AmazonCloudWatchAgent profile is used for onPremise case by default. # [credentials] # shared_credential_profile = "{profile_name}" # shared_credential_file= "{file_name}" ## Configuration for proxy. ## System-wide environment-variable will be read if it is absent here. ## i.e. HTTP_PROXY/http_proxy; HTTPS_PROXY/https_proxy; NO_PROXY/no_proxy ## Note: system-wide environment-variable is not accessible when using ssm run-command. ## Absent in both here and environment-variable means no proxy will be used. # [proxy] # http_proxy = "{http_url}" # https_proxy = "{https_url}" # no_proxy = "{domain}" ``` All lines are commented out initially. To set the credential profile or proxy settings, remove the `#` from that line and specify a value. You can edit this file manually, or by using the `RunShellScript` Run Command in Systems Manager: + `shared_credential_profile` – For on-premises servers, this line specifies the IAM user credential profile to use to send data to CloudWatch. If you keep this line commented out, `AmazonCloudWatchAgent` is used. For more information about creating this profile, see [(Installing on an on-premises server) Specify IAM credentials and AWS Region](#install-CloudWatch-Agent-iam_user-SSM-onprem). On an EC2 instance, you can use this line to have the CloudWatch agent send data from this instance to CloudWatch in a different AWS Region. To do so, specify a named profile that includes a `region` field specifying the name of the Region to send to. If you specify a `shared_credential_profile`, you must also remove the `#` from the beginning of the `[credentials]` line. + `shared_credential_file` – To have the agent look for credentials in a file located in a path other than the default path, specify that complete path and file name here. The default path is `/root/.aws` on Linux and is `C:\\Users\\Administrator\\.aws` on Windows Server. The first example below shows the syntax of a valid `shared_credential_file` line for Linux servers, and the second example is valid for Windows Server. On Windows Server, you must escape the \$1 characters. ``` shared_credential_file= "/usr/username/credentials" ``` ``` shared_credential_file= "C:\\Documents and Settings\\username\\.aws\\credentials" ``` If you specify a `shared_credential_file`, you must also remove the `#` from the beginning of the `[credentials]` line. + Proxy settings – If your servers use HTTP or HTTPS proxies to contact AWS services, specify those proxies in the `http_proxy` and `https_proxy` fields. If there are URLs that should be excluded from proxying, specify them in the `no_proxy` field, separated by commas. # Install the CloudWatch agent on new instances using CloudFormation This section describes how to install the CloudWatch agent on new Amazon EC2 instances using AWS CloudFormation. **Note** Amazon uploaded several CloudFormation templates to GitHub that can help you install and update the CloudWatch agent on new Amazon EC2 instances. For more information about using CloudFormation, see [ What is AWS CloudFormation?](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html). The template location is [ Deploy the Amazon CloudWatch agent to EC2 instances using CloudFormation](https://github.com/aws-cloudformation/aws-cloudformation-templates/tree/main/Solutions/AmazonCloudWatchAgent). This location includes both `inline` and `ssm` directories. Each of these directories contains templates for both Linux and Windows instances. + The templates in the `inline` directory have the CloudWatch agent configuration embedded into the CloudFormation template. By default, the Linux templates collect the metrics `mem_used_percent` and `swap_used_percent`, and the Windows templates collect `Memory % Committed Bytes In Use` and `Paging File % Usage`. To modify these templates to collect different metrics, modify the following section of the template. The following example is from the template for Linux servers. Follow the format and syntax of the agent configuration file to make these changes. For more information, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). ``` { "metrics":{ "append_dimensions":{ "AutoScalingGroupName":"${!aws:AutoScalingGroupName}", "ImageId":"${!aws:ImageId}", "InstanceId":"${!aws:InstanceId}", "InstanceType":"${!aws:InstanceType}" }, "metrics_collected":{ "mem":{ "measurement":[ "mem_used_percent" ] }, "swap":{ "measurement":[ "swap_used_percent" ] } } } } ``` **Note** In the inline templates, all placeholder variables must have an exclamation mark (\$1) before them as an escape character. You can see this in the example template. If you add other placeholder variables, be sure to add an exclamation mark before the name. + The templates in the `ssm` directory load an agent configuration file from Parameter Store. To use these templates, you must first create a configuration file and upload it to Parameter Store. You then provide the Parameter Store name of the file in the template. You can create the configuration file manually or by using the wizard. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md). You can use both types of templates for installing the CloudWatch agent and for updating the agent configuration. For information about setting up the agent on a system that has security-enhanced Linux (SELinux) enabled, see [Set up the CloudWatch agent with security-enhanced Linux (SELinux)](CloudWatch-Agent-SELinux.md). ## Tutorial: Install and configure the CloudWatch agent using an CloudFormation inline template This tutorial walks you through using CloudFormation to install the CloudWatch agent on a new Amazon EC2 instance. This tutorial installs on a new instance running Amazon Linux 2 using the inline templates, which don't require the use of the JSON configuration file or Parameter Store. The inline template includes the agent configuration in the template. In this tutorial, you use the default agent configuration contained in the template. After the procedure for installing the agent, the tutorial continues with how to update the agent. **To use CloudFormation to install the CloudWatch agent on a new instance** 1. Download the template from GitHub. In this tutorial, download the inline template for Amazon Linux 2 as follows: ``` curl -O https://raw.githubusercontent.com/aws-cloudformation/aws-cloudformation-templates/main/Solutions/AmazonCloudWatchAgent/inline/amazon_linux.yaml ``` 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. Choose **Create stack**. 1. For **Choose a template**, select **Upload a template to Amazon S3**, choose the downloaded template, and choose **Next**. 1. On the **Specify Details** page, fill out the following parameters and choose **Next**: + **Stack name**: Choose a stack name for your CloudFormation stack. + **IAMRole**: Choose an IAM role that has permissions to write CloudWatch metrics, logs, and traces. For more information, see [Prerequisites](prerequisites.md). + **InstanceAMI**: Choose an AMI that is valid in the Region where you're going to launch your stack. + **InstanceType**: Choose a valid instance type. + **KeyName**: To enable SSH access to the new instance, choose an existing Amazon EC2 key pair. If you don't already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For more information, see [Amazon EC2 Key Pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*. + **SSHLocation**: Specifies the IP address range that can be used to connect to the instance using SSH. The default allows access from any IP address. 1. On the **Options** page, you can choose to tag your stack resources. Choose **Next**. 1. On the **Review** page, review your information, acknowledge that the stack might create IAM resources, and then choose **Create**. If you refresh the console, you see that the new stack has the `CREATE_IN_PROGRESS` status. 1. When the instance is created, you can see it in the Amazon EC2 console. Optionally, you can connect to the host and check the progress. Use the following command to confirm that the agent is installed: ``` rpm -qa amazon-cloudwatch-agent ``` Use the following command to confirm that the agent is running: ``` ps aux | grep amazon-cloudwatch-agent ``` The next procedure demonstrates using CloudFormation to update the CloudWatch agent using an inline template. The default inline template collects the `mem_used_percent` metric. In this tutorial, you change the agent configuration to stop collecting that metric. **To use CloudFormation to update the CloudWatch agent** 1. In the template that you downloaded in the previous procedure, remove the following lines and then save the template: ``` "mem": { "measurement": [ "mem_used_percent" ] }, ``` 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. On the CloudFormation dashboard, select the stack that you created and choose **Update Stack**. 1. For **Select Template**, select **Upload a template to Amazon S3**, choose the template that you modified, and choose **Next**. 1. On the **Options** page, choose **Next** and then **Next**. 1. On the **Review** page, review your information and choose **Update**. After some time, you see `UPDATE_COMPLETE`. ## Tutorial: Install the CloudWatch agent using CloudFormation and Parameter Store This tutorial walks you through using CloudFormation to install the CloudWatch agent on a new Amazon EC2 instance. This tutorial installs on a new instance running Amazon Linux 2 using an agent configuration file that you create and save in Parameter Store. After the procedure for installing the agent, the tutorial continues with how to update the agent. **To use CloudFormation to install the CloudWatch agent on a new instance using a configuration from Parameter Store** 1. If you haven't done so already, download the CloudWatch agent package to one of your computers so that you can create the agent configuration file. For more information and downloading the agent using Parameter Store, see [Download the CloudWatch agent package](download-CloudWatch-Agent-on-EC2-Instance-commandline-first.md). 1. Create the agent configuration file and save it in Parameter Store. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md). 1. Download the template from GitHub as follows: ``` curl -O https://raw.githubusercontent.com/awslabs/aws-cloudformation-templates/master/aws/solutions/AmazonCloudWatchAgent/ssm/amazon_linux.template ``` 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. Choose **Create stack**. 1. For **Choose a template**, select **Upload a template to Amazon S3**, choose the template that you downloaded, and choose **Next**. 1. On the **Specify Details** page, fill out the following parameters accordingly and choose **Next**: + **Stack name**: Choose a stack name for your CloudFormation stack. + **IAMRole**: Choose an IAM role that has permissions to write CloudWatch metrics, logs, and traces. For more information, see [Prerequisites](prerequisites.md). + **InstanceAMI**: Choose an AMI that is valid in the Region where you're going to launch your stack. + **InstanceType**: Choose a valid instance type. + **KeyName**: To enable SSH access to the new instance, choose an existing Amazon EC2 key pair. If you don't already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For more information, see [Amazon EC2 Key Pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*. + **SSHLocation**: Specifies the IP address range that can be used to connect to the instance using SSH. The default allows access from any IP address. + **SSMKey**: Specifies the agent configuration file that you created and saved in Parameter Store. 1. On the **Options** page, you can choose to tag your stack resources. Choose **Next**. 1. On the **Review** page, review your information, acknowledge that the stack might create IAM resources, and then choose **Create**. If you refresh the console, you see that the new stack has the `CREATE_IN_PROGRESS` status. 1. When the instance is created, you can see it in the Amazon EC2 console. Optionally, you can connect to the host and check the progress. Use the following command to confirm that the agent is installed: ``` rpm -qa amazon-cloudwatch-agent ``` Use the following command to confirm that the agent is running: ``` ps aux | grep amazon-cloudwatch-agent ``` The next procedure demonstrates using CloudFormation to update the CloudWatch agent, using an agent configuration that you saved in Parameter Store. **To use CloudFormation to update the CloudWatch agent using a configuration in Parameter Store** 1. Change the agent configuration file stored in Parameter Store to the new configuration that you want. 1. In the CloudFormation template that you downloaded in the [Tutorial: Install the CloudWatch agent using CloudFormation and Parameter Store](#installing-CloudWatch-Agent-using-CloudFormation-Templates) topic, change the version number. For example, you might change `VERSION=1.0` to `VERSION=2.0`. 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. On the CloudFormation dashboard, select the stack that you created and choose **Update Stack**. 1. For **Select Template**, select **Upload a template to Amazon S3**, select the template that you just modified, and choose **Next**. 1. On the **Options** page, choose **Next** and then **Next**. 1. On the **Review** page, review your information and choose **Update**. After some time, you see `UPDATE_COMPLETE`. ## Troubleshooting installation of the CloudWatch agent with CloudFormation This section helps you troubleshoot issues with installing and updating the CloudWatch agent using CloudFormation. ### Detecting when an update fails If you use CloudFormation to update your CloudWatch agent configuration, and use an invalid configuration, the agent stops sending any metrics to CloudWatch. A quick way to check whether an agent configuration update succeeded is to look at the `cfn-init-cmd.log` file. On a Linux server, the file is located at `/var/log/cfn-init-cmd.log`. On a Windows instance, the file is located at `C:\cfn\log\cfn-init-cmd.log`. ### Metrics are missing If you don't see metrics that you expect to see after installing or updating the agent, confirm that the agent is configured to collect that metric. To do this, check the `amazon-cloudwatch-agent.json` file to make sure that the metric is listed, and check that you are looking in the correct metric namespace. For more information, see [CloudWatch agent files and locations](troubleshooting-CloudWatch-Agent.md#CloudWatch-Agent-files-and-locations). # Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart You can use either the Amazon CloudWatch Observability EKS add-on or the Amazon CloudWatch Observability Helm chart to install the CloudWatch Agent and the Fluent-bit agent on an Amazon EKS cluster. You can also use the Helm chart to install the CloudWatch Agent and the Fluent-bit agent on a Kubernetes cluster that is not hosted on Amazon EKS. Using either method on an Amazon EKS cluster enables both [Container Insights](ContainerInsights.md) with enhanced observability for Amazon EKS and [CloudWatch Application Signals](CloudWatch-Application-Monitoring-Sections.md) by default. Both features help you to collect infrastructure metrics, application performance telemetry, and container logs from the cluster. With version `v6.0.1-eksbuild.1` or later of the add-on, Container Insights with OpenTelemetry metrics is enabled, which collects metrics using the OpenTelemetry Protocol (OTLP) and supports PromQL queries. For more information, see [Container Insights with OpenTelemetry metrics for Amazon EKS](container-insights-otel-metrics.md). With Container Insights with enhanced observability for Amazon EKS, Container Insights metrics are charged per observation instead of being charged per metric stored or log ingested. For Application Signals, billing is based on inbound requests to your applications, outbound requests from your applications, and each configured service level objective (SLO). Each inbound request received generates one application signal, and each outbound request made generates one application signal. Every SLO creates two application signals per measurement period. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). Both methods enable Container Insights on both Linux and Windows worker nodes in the Amazon EKS cluster. To enable Container Insights on Windows, you must use version 1.5.0 or later of the Amazon EKS add-on or the Helm chart. Application Signals is not supported on Windows in Amazon EKS clusters. The Amazon CloudWatch Observability EKS add-on is supported on Amazon EKS clusters running with Kubernetes version 1.23 or later. When you install the add-on or the Helm chart, you must also grant IAM permissions to enable the CloudWatch agent to send metrics, logs, and traces to CloudWatch. There are two ways to do this: + Attach a policy to the IAM role of your worker nodes. This option grants permissions to worker nodes to send telemetry to CloudWatch. + Use an IAM role for service accounts for the agent pods, and attach the policy to this role. This works only for Amazon EKS clusters. This option gives CloudWatch access only to the appropriate agent pods. **Topics** + [ ## Option 1: Install using EKS Pod Identity ](#install-CloudWatch-Observability-EKS-pod-identity) + [ ## Option 2: Install with IAM permissions on worker nodes ](#install-CloudWatch-Observability-EKS-addon-workernodes) + [ ## Option 3: Install using IAM service account role (applies only to using the add-on) ](#install-CloudWatch-Observability-EKS-addon-serviceaccountrole) + [ ## (Optional) Additional configuration ](#install-CloudWatch-Observability-EKS-addon-configuration) + [ ## Collect Java Management Extensions (JMX) metrics ](#install-CloudWatch-Observability-EKS-addon-JMX-metrics) + [ ## Enable Kueue metrics ](#enable-Kueue-metrics) + [ ## Appending OpenTelemetry collector configuration files ](#install-CloudWatch-Observability-EKS-addon-OpenTelemetry) + [ ## Enabling APM through Application Signals for your Amazon EKS cluster ](#Container-Insights-setup-EKS-appsignalsconfiguration) + [ ## Considerations for large Kubernetes clusters ](#install-CloudWatch-Observability-EKS-addon-large-clusters) + [ ## Considerations for Amazon EKS Hybrid Nodes ](#install-CloudWatch-Observability-EKS-addon-hybrid) + [ ## Troubleshooting the Amazon CloudWatch Observability EKS add-on or the Helm chart ](#Container-Insights-setup-EKS-addon-troubleshoot) + [ ## Opt out of Application Signals ](#Opting-out-App-Signals) ## Option 1: Install using EKS Pod Identity If you use version 3.1.0 or later of the add-on, you can use EKS Pod Identity to grant the required permissions to the add-on. EKS Pod Identity is the recommended option and provides benefits such as least privilege, credential rotation, and auditability. Additionally, using EKS Pod Identity allows you to install the EKS add-on as part of the cluster creation itself. To use this method, first follow the [EKS Pod Identity association](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-association.html#pod-id-association-create/) steps to create the IAM role and set up the EKS Pod Identity agent. Then install the Amazon CloudWatch Observability EKS add-on. To install the add-on, you can use the AWS CLI, the Amazon EKS console, CloudFormation, or Terraform. ------ #### [ AWS CLI ] **To use the AWS CLI to install the Amazon CloudWatch Observability EKS add-on** Enter the following commands. Replace `my-cluster-name` with the name of your cluster and replace *111122223333* with your account ID. Replace *my-role* with the IAM role that you created in the EKS Pod Identity association step. ``` aws iam attach-role-policy \ --role-name my-role \ --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy aws eks create-addon \ --addon-name amazon-cloudwatch-observability \ --cluster-name my-cluster-name \ --pod-identity-associations serviceAccount=cloudwatch-agent,roleArn=arn:aws:iam::111122223333:role/my-role ``` ------ #### [ Amazon EKS console ] **To use the Amazon EKS console to add the Amazon CloudWatch Observability EKS add-on** 1. Open the Amazon EKS console at [https://console.aws.amazon.com/eks/home\$1/clusters](https://console.aws.amazon.com/eks/home#/clusters). 1. In the left navigation pane, choose **Clusters**. 1. Choose the name of the cluster that you want to configure the Amazon CloudWatch Observability EKS add-on for. 1. Choose the **Add-ons** tab. 1. Choose **Get more add-ons**. 1. On the **Select add-ons** page, do the following: 1. In the **Amazon EKS-addons** section, select the **Amazon CloudWatch Observability** check box. 1. Choose **Next**. 1. On the **Configure selected add-ons settings** page, do the following: 1. Select the **Version** you'd like to use. 1. For **Add-on access**, select **EKS Pod Identity** 1. If you don't have an IAM role configured, choose **Create recommended role**, then choose **Next** until you are at **Step 3 Name, review, and create**. You can change your role name if desired, otherwise, choose **Create Role**, and then return to the Add-on page and select the IAM role that you just created. 1. (Optional) You can expand the **Optional configuration settings**. If you select **Override** for the **Conflict resolution method**, one or more of the settings for the existing add-on can be overwritten with the Amazon EKS add-on settings. If you don't enable this option and there's a conflict with your existing settings, the operation fails. You can use the resulting error message to troubleshoot the conflict. Before selecting this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to self-manage. 1. Choose **Next**. 1. On the **Review and add** page, choose **Create**. After the add-on installation is complete, you see your installed add-on. ------ #### [ CloudFormation ] **To use CloudFormation to install the Amazon CloudWatch Observability EKS add-on** 1. First, run the following AWS CLI command to attach the necessary IAM policy to your IAM role. Replace *my-role* with the role that you created in the EKS Pod Identity association step. ``` aws iam attach-role-policy \ --role-name my-role \ --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy ``` 1. Then create the following resource. Replace `my-cluster-name` with the name of your cluster, replace *111122223333* with your account ID, and replace *my-role* with the IAM role created in EKS Pod Identity association step. For more information, see [ AWS::EKS::Addon](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-eks-addon.html). ``` { "Resources": { "EKSAddOn": { "Type": "AWS::EKS::Addon", "Properties": { "AddonName": "amazon-cloudwatch-observability", "ClusterName": "my-cluster-name", "PodIdentityAssociations": [ { "ServiceAccount": "cloudwatch-agent", "RoleArn": "arn:aws:iam::111122223333:role/my-role" } ] } } } } ``` ------ #### [ Terraform ] **To use Terraform to install the Amazon CloudWatch Observability EKS add-on** 1. Use the following. Replace `my-cluster-name` with the name of your cluster, replace *111122223333* with your account ID, and replace *my-service-account-role* with the IAM role created in EKS Pod Identity association step. For more information, see [ Resource: aws\$1eks\$1addon](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_addon) in the Terraform documentation. 1. ``` resource "aws_iam_role_policy_attachment" "CloudWatchAgentServerPolicy" { policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" role = "my-role" } resource "aws_eks_addon" "example" { cluster_name = "my-cluster-name" addon_name = "amazon-cloudwatch-observability" pod_identity_associations { roleArn = "arn:aws:iam::111122223333:role/my-role" serviceAccount = "cloudwatch-agent" } } ``` ------ ## Option 2: Install with IAM permissions on worker nodes To use this method, first attach the **CloudWatchAgentServerPolicy** IAM policy to your worker nodes by entering the following command. In this command, replace *my-worker-node-role* with the IAM role used by your Kubernetes worker nodes. ``` aws iam attach-role-policy \ --role-name my-worker-node-role \ --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy ``` Then install the Amazon CloudWatch Observability EKS add-on. To install the add-on, you can use the AWS CLI, the console, CloudFormation, or Terraform. ------ #### [ AWS CLI ] **To use the AWS CLI to install the Amazon CloudWatch Observability EKS add-on** Enter the following command. Replace `my-cluster-name` with the name of your cluster. ``` aws eks create-addon --addon-name amazon-cloudwatch-observability --cluster-name my-cluster-name ``` ------ #### [ Amazon EKS console ] **To use the Amazon EKS console to add the Amazon CloudWatch Observability EKS add-on** 1. Open the Amazon EKS console at [https://console.aws.amazon.com/eks/home\$1/clusters](https://console.aws.amazon.com/eks/home#/clusters). 1. In the left navigation pane, choose **Clusters**. 1. Choose the name of the cluster that you want to configure the Amazon CloudWatch Observability EKS add-on for. 1. Choose the **Add-ons** tab. 1. Choose **Get more add-ons**. 1. On the **Select add-ons** page, do the following: 1. In the **Amazon EKS-addons** section, select the **Amazon CloudWatch Observability** check box. 1. Choose **Next**. 1. On the **Configure selected add-ons settings** page, do the following: 1. Select the **Version** you'd like to use. 1. (Optional) You can expand the **Optional configuration settings**. If you select **Override** for the **Conflict resolution method**, one or more of the settings for the existing add-on can be overwritten with the Amazon EKS add-on settings. If you don't enable this option and there's a conflict with your existing settings, the operation fails. You can use the resulting error message to troubleshoot the conflict. Before selecting this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to self-manage. 1. Choose **Next**. 1. On the **Review and add** page, choose **Create**. After the add-on installation is complete, you see your installed add-on. ------ #### [ CloudFormation ] **To use CloudFormation to install the Amazon CloudWatch Observability EKS add-on** Replace `my-cluster-name` with the name of your cluster. For more information, see [ AWS::EKS::Addon](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-eks-addon.html). ``` { "Resources": { "EKSAddOn": { "Type": "AWS::EKS::Addon", "Properties": { "AddonName": "amazon-cloudwatch-observability", "ClusterName": "my-cluster-name" } } } } ``` ------ #### [ Helm chart ] **To use the `amazon-cloudwatch-observability` Helm chart** 1. You must have Helm installed to use this chart. For more information about installing Helm, see the [Helm documentation](https://helm.sh/docs/). 1. After you have installed Helm, enter the following commands. Replace *my-cluster-name* with the name of your cluster, and replace *my-cluster-region* with the Region that the cluster runs in. ``` helm repo add aws-observability https://aws-observability.github.io/helm-charts helm repo update aws-observability helm install --wait --create-namespace --namespace amazon-cloudwatch amazon-cloudwatch-observability aws-observability/amazon-cloudwatch-observability --set clusterName=my-cluster-name --set region=my-cluster-region ``` ------ #### [ Terraform ] **To use Terraform to install the Amazon CloudWatch Observability EKS add-on** Replace `my-cluster-name` with the name of your cluster. For more information, see [Resource: aws\$1eks\$1addon](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_ad). ``` resource "aws_eks_addon" "example" { addon_name = "amazon-cloudwatch-observability" cluster_name = "my-cluster-name" } ``` ------ ## Option 3: Install using IAM service account role (applies only to using the add-on) This method is valid only if you are using the Amazon CloudWatch Observability EKS add-on. Before using this method, verify the following prerequisites: + You have a functional Amazon EKS cluster with nodes attached in one of the AWS Regions that supports Container Insights. For the list of supported Regions, see [Container Insights](ContainerInsights.md). + You have `kubectl` installed and configured for the cluster. For more information, see [Installing `kubectl`](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) in the *Amazon EKS User Guide*. + You have `eksctl` installed. For more information, see [Installing or updating `eksctl`](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) in the *Amazon EKS User Guide*. ------ #### [ AWS CLI ] **To use the AWS CLI to install the Amazon CloudWatch Observability EKS add-on using the IAM service account role** 1. Enter the following command to create an OpenID Connect (OIDC) provider, if the cluster doesn't have one already. For more information, see [Configuring a Kubernetes service account to assume an IAM role](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) in the *Amazon EKS User Guide*. ``` eksctl utils associate-iam-oidc-provider --cluster my-cluster-name --approve ``` 1. Enter the following command to create the IAM role with the **CloudWatchAgentServerPolicy** policy attached, and configure the agent service account to assume that role using OIDC. Replace *my-cluster-name* with the name of your cluster, and replace *my-service-account-role* with the name of the role that you want to associate the service account with. If the role doesn't already exist, `eksctl` creates it for you. ``` eksctl create iamserviceaccount \ --name cloudwatch-agent \ --namespace amazon-cloudwatch --cluster my-cluster-name \ --role-name my-service-account-role \ --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \ --role-only \ --approve ``` 1. Install the add-on by entering the following command. Replace *my-cluster-name* with the name of your cluster, replace *111122223333* with your account ID, and replace *my-service-account-role* with the IAM role created in the previous step. ``` aws eks create-addon --addon-name amazon-cloudwatch-observability --cluster-name my-cluster-name --service-account-role-arn arn:aws:iam::111122223333:role/my-service-account-role ``` ------ #### [ Amazon EKS console ] **To use the console to install the Amazon CloudWatch Observability EKS add-on using the IAM service account role** 1. Open the Amazon EKS console at [https://console.aws.amazon.com/eks/home\$1/clusters](https://console.aws.amazon.com/eks/home#/clusters). 1. In the left navigation pane, choose **Clusters**. 1. Choose the name of the cluster that you want to configure the Amazon CloudWatch Observability EKS add-on for. 1. Choose the **Add-ons** tab. 1. Choose **Get more add-ons**. 1. On the **Select add-ons** page, do the following: 1. In the **Amazon EKS-addons** section, select the **Amazon CloudWatch Observability** check box. 1. Choose **Next**. 1. On the **Configure selected add-ons settings** page, do the following: 1. Select the **Version** you'd like to use. 1. For **Add-on access**, select **IAM roles for service accounts (IRSA)** 1. Select the IAM role in the **Add-on access** box. 1. (Optional) You can expand the **Optional configuration settings**. If you select **Override** for the **Conflict resolution method**, one or more of the settings for the existing add-on can be overwritten with the Amazon EKS add-on settings. If you don't enable this option and there's a conflict with your existing settings, the operation fails. You can use the resulting error message to troubleshoot the conflict. Before selecting this option, make sure that the Amazon EKS add-on doesn't manage settings that you need to self-manage. 1. Choose **Next**. 1. On the **Review and add** page, choose **Create**. After the add-on installation is complete, you see your installed add-on. ------ ## (Optional) Additional configuration **Topics** + [ ### Opt out of collecting container logs ](#CloudWatch-Observability-EKS-addon-OptOutContainerLogs) + [ ### Use a custom Fluent Bit configuration ](#CloudWatch-Observability-EKS-addon-CustomFluentBit) + [ ### Manage Kubernetes tolerations for the installed pod workloads ](#CloudWatch-Observability-EKS-addon-Tolerations) + [ ### Opt out of accelerated compute metrics collection ](#CloudWatch-Observability-EKS-addon-OptOutAccelerated) + [ ### Use a custom CloudWatch agent configuration ](#CloudWatch-Observability-EKS-addon-CustomAgentConfig) + [ ### Manage admission webhook TLS certificates ](#CloudWatch-Observability-EKS-addon-Webhook) + [ ### Collect Amazon EBS volume IDs ](#CloudWatch-Observability-EKS-addon-VolumeIDs) ### Opt out of collecting container logs By default, the add-on uses Fluent Bit to collect container logs from all pods and then sends the logs to CloudWatch Logs. For information about which logs are collected, see [Setting up Fluent Bit](Container-Insights-setup-logs-FluentBit.md#Container-Insights-FluentBit-setup). **Note** Neither the add-on or the Helm chart manage existing Fluentd or Fluent Bit resources in a cluster. You can delete the existing Fluentd or Fluent Bit resources before installing the add-on or Helm chart. Alternatively, to keep your existing setup and avoid having the add-on or the Helm chart from also installing Fluent Bit, you can disable it by following the instructions in this section. To opt out of the collection of container logs if you are using the Amazon CloudWatch Observability EKS add-on, pass the following option when you create or update the add-on: ``` --configuration-values '{ "containerLogs": { "enabled": false } }' ``` To opt out of the collection of container logs if you are using the Helm chart, pass the following option when you create or update the add-on: ``` --set containerLogs.enabled=false ``` ### Use a custom Fluent Bit configuration Starting with version 1.7.0 of the Amazon CloudWatch Observability EKS add-on, you can modify the Fluent Bit configuration when you create or update the add-on or Helm chart. You supply the custom Fluent Bit configuration in the `containerLogs` root level section of the advanced configuration of the add-on or the value overrides in the Helm chart. Within this section, you supply the custom Fluent Bit configuration in the `config` section (for Linux) or `configWindows` section (for Windows). The `config` is further broken down into the following sub-sections: + `service`– This section represents the `SERVICE` config to define the global behavior of the Fluent Bit engine. + `customParsers`– This section represents any global `PARSER`s that you want to include that are capable of taking unstructured log entries and giving them a structure to make it easier for processing and further filtering. + `extraFiles`– This section can be used to provide additional Fluent Bit `conf` files to be included. By default, the following 3 `conf` files are included:. + `application-log.conf`– A `conf` file for sending application logs from your cluster to the log group `/aws/containerinsights/my-cluster-name/application` in CloudWatch Logs. + `dataplane-log.conf`– A `conf` file for sending logs corresponding to your cluster’s data plane components including the CRI logs, kubelet logs, kube-proxy logs and Amazon VPC CNI logs to the log group `/aws/containerinsights/my-cluster-name/dataplane` in CloudWatch Logs. + `host-log.conf`– A `conf` for sending logs from `/var/log/dmesg`, `/var/log/messages`, and `/var/log/secure` on Linux, and System `winlogs` on Windows, to the log group `/aws/containerinsights/my-cluster-name/host` in CloudWatch. **Note** Provide the full configuration for each of these individual sections even if you are modifying only one field within a sub-section. We recommend that you use the default configuration provided below as a baseline and then modify it accordingly so that you don't disable functionality that is enabled by default. You can use the following YAML configuration when modifying the advanced config for the Amazon EKS add-on or when you supply value overrides for the Helm chart. To find the `config` section for your cluster, see [aws-observability / helm-charts](https://github.com/aws-observability/helm-charts/releases) on GitHub and find the release corresponding to the version of the add-on or Helm chart that you are installing. Then navigate to `/charts/amazon-cloudwatch-observability/values.yaml` to find the `config` section (for Linux) and `configWindows` section (for Windows) within the `fluentBit` section under `containerLogs`. As an example, the default Fluent Bit configuration for version 1.7.0 can be found [here](https://github.com/aws-observability/helm-charts/blob/v1.7.0/charts/amazon-cloudwatch-observability/values.yaml#L44)). We recommend that you provide the `config` as YAML when you supply it using the Amazon EKS add-on’s advanced config or when you supply it as value overrides for your Helm installation. Be sure that the YAML conforms to the following structure. ``` containerLogs: fluentBit: config: service: | ... customParsers: | ... extraFiles: application-log.conf: | ... dataplane-log.conf: | ... host-log.conf: | ... ``` The following example `config` changes the global setting for the flush interval to be 45 seconds. Even though the only modification is to the `Flush` field, you must still provide the full `SERVICE` definition for the service sub-section. Because this example didn't specify overrides for the other sub-sections, the defaults are used for them. ``` containerLogs: fluentBit: config: service: | [SERVICE] Flush 45 Grace 30 Log_Level error Daemon off Parsers_File parsers.conf storage.path /var/fluent-bit/state/flb-storage/ storage.sync normal storage.checksum off storage.backlog.mem_limit 5M ``` The following example configuration includes an extra Fluent bit `conf` file. In this example, we are adding a custom `my-service.conf` under `extraFiles` and it will be included in addition to the three default `extraFiles`. ``` containerLogs: fluentBit: config: extraFiles: my-service.conf: | [INPUT] Name tail Tag myservice.* Path /var/log/containers/*myservice*.log DB /var/fluent-bit/state/flb_myservice.db Mem_Buf_Limit 5MB Skip_Long_Lines On Ignore_Older 1d Refresh_Interval 10 [OUTPUT] Name cloudwatch_logs Match myservice.* region ${AWS_REGION} log_group_name /aws/containerinsights/${CLUSTER_NAME}/myservice log_stream_prefix ${HOST_NAME}- auto_create_group true ``` The next example removes an existing `conf` file entirely from `extraFiles`. This excludes the `application-log.conf` entirely by overriding it with an empty string. Simply omitting `application-log.conf` from `extraFiles` would instead imply to use the default, which is not what we are trying to achieve in this example. The same applies to removing any custom `conf` file that you might have previously added to `extraFiles`. ``` containerLogs: fluentBit: config: extraFiles: application-log.conf: "" ``` ### Manage Kubernetes tolerations for the installed pod workloads Starting with version 1.7.0 of the Amazon CloudWatch Observability EKS add-on, the add-on and the Helm chart by default set Kubernetes *tolerations* to tolerate all taints on the pod workloads that are installed by the add-on or the Helm chart. This ensures that daemonsets such as the CloudWatch agent and Fluent Bit can schedule pods on all nodes in your cluster by default. For more information about tolerations and taints, see [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) in the Kubernetes documentation. The default tolerations set by the add-on or the Helm chart are as follows: ``` tolerations: - operator: Exists ``` You can override the default tolerations by setting the `tolerations` field at the root level when using the add-on advanced config or when you install or upgrade the Helm chart with value overrides. An example would look like the following: ``` tolerations: - key: "key1" operator: "Exists" effect: "NoSchedule" ``` To omit tolerations completely, you can use a config that looks like the following: ``` tolerations: [] ``` Any changes to tolerations apply to all pod workloads that are installed by the add-on or the Helm chart. ### Opt out of accelerated compute metrics collection By default, Container Insights with enhanced observability collects metrics for Accelerated Compute monitoring, including NVIDIA GPU metrics, AWS Neuron metrics for AWS Trainium and AWS Inferentia, and AWS Elastic Fabric Adapter (EFA) metrics. NVIDIA GPU metrics from Amazon EKS workloads are collected by default beginning with version `v1.3.0-eksbuild.1` of the EKS add-on or the Helm chart and version `1.300034.0` of the CloudWatch agent. For a list of metrics collected and prerequisites, see [NVIDIA GPU metrics](Container-Insights-metrics-enhanced-EKS.md#Container-Insights-metrics-EKS-GPU). AWS Neuron metrics for AWS Trainium and AWS Inferentia accelerators are collected by default beginning with version `v1.5.0-eksbuild.1` of the EKS add-on or the Helm chart, and version `1.300036.0` of the CloudWatch agent. For a list of metrics collected and prerequisites, see [AWS Neuron metrics for AWS Trainium and AWS Inferentia](Container-Insights-metrics-enhanced-EKS.md#Container-Insights-metrics-EKS-Neuron). AWS Elastic Fabric Adapter (EFA) metrics from Linux nodes on Amazon EKS clusters are collected by default beginning with version `v1.5.2-eksbuild.1` of the EKS add-on or the Helm chart and version `1.300037.0` of the CloudWatch agent. For a list of metrics collected and prerequisites, see [AWS Elastic Fabric Adapter (EFA) metrics](Container-Insights-metrics-enhanced-EKS.md#Container-Insights-metrics-EFA). You can opt out of collecting these metrics by setting the `accelerated_compute_metrics` field in the CloudWatch agent configuration file to `false`. This field is in the `kubernetes` section of the `metrics_collected` section in the CloudWatch configuration file. The following is an example of an opt-out configuration. For more information about how to use custom CloudWatch agent configurations, see the following section, [Use a custom CloudWatch agent configuration](#CloudWatch-Observability-EKS-addon-CustomAgentConfig). ``` { "logs": { "metrics_collected": { "kubernetes": { "enhanced_container_insights": true, "accelerated_compute_metrics": false } } } } ``` ### Use a custom CloudWatch agent configuration To collect other metrics, logs or traces using the CloudWatch agent, you can specify a custom configuration while also keeping Container Insights and CloudWatch Application Signals enabled. To do so, embed the CloudWatch agent configuration file within the config key under the agent key of the advanced configuration that you can use when creating or updating the EKS add-on or the Helm chart. The following represents the default agent configuration when you do not provide any additional configuration. **Important** Any custom configuration that you provide using additional configuration settings overrides the default configuration used by the agent. Be cautious not to unintentionally disable functionality that is enabled by default, such as Container Insights with enhanced observability and CloudWatch Application Signals. In the scenario that you are required to provide a custom agent configuration, we recommend using the following default configuration as a baseline and then modifying it accordingly. + For using the Amazon CloudWatch observability EKS add-on ``` --configuration-values '{ "agent": { "config": { "logs": { "metrics_collected": { "application_signals": {}, "kubernetes": { "enhanced_container_insights": true } } }, "traces": { "traces_collected": { "application_signals": {} } } } } }' ``` + For using the Helm chart ``` --set agent.config='{ "logs": { "metrics_collected": { "application_signals": {}, "kubernetes": { "enhanced_container_insights": true } } }, "traces": { "traces_collected": { "application_signals": {} } } }' ``` The following example shows the default agent configuration for the CloudWatch agent on Windows. The CloudWatch agent on Windows does not support custom configuration. ``` { "logs": { "metrics_collected": { "kubernetes": { "enhanced_container_insights": true }, } } } ``` ### Manage admission webhook TLS certificates The Amazon CloudWatch Observability EKS add-on and the Helm chart leverage Kubernetes [ admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) to validate and mutate `AmazonCloudWatchAgent` and `Instrumentation` custom resource (CR) requests, and optionally Kubernetes pod requests on the cluster if CloudWatch Application Signals is enabled. In Kubernetes, webhooks require a TLS certificate that the API server is configured to trust in order to ensure secure communication. By default, the Amazon CloudWatch Observability EKS add-on and the Helm chart auto-generate a self-signed CA and a TLS certificate signed by this CA for securing the communication between the API server and the webhook server. This auto-generated certificate has a default expiry of 10 years and is not auto-renewed upon expiry. In addition, the CA bundle and the certificate are re-generated every time the add-on or Helm chart is upgraded or re-installed, thus resetting the expiry. If you want to change the default expiry of the auto-generated certificate, you can use the following additional configurations when creating or updating the add-on. Replace *expiry-in-days* with your desired expiry duration in days. + Use this for the Amazon CloudWatch Observability EKS add-on ``` --configuration-values '{ "admissionWebhooks": { "autoGenerateCert": { "expiryDays": expiry-in-days } } }' ``` + Use this for the Helm chart ``` --set admissionWebhooks.autoGenerateCert.expiryDays=expiry-in-days ``` For a more secure and feature-rich certificate authority solution, the add-on has opt-in support for [ cert-manager](https://cert-manager.io/docs/), a widely-adopted solution for TLS certificate management in Kubernetes that simplifies the process of obtaining, renewing, managing and using those certificates. It ensures that certificates are valid and up to date, and attempts to renew certificates at a configured time before expiry. cert-manager also facilitates issuing certificates from a variety of supported sources, including [AWS Certificate Manager Private Certificate Authority](https://aws.amazon.com/private-ca/). We recommend that you review best practices for management of TLS certificates on your clusters and advise you to opt in to cert-manager for production environments. Note that if you opt-in to enabling cert-manager for managing the admission webhook TLS certificates, you are required to pre-install cert-manager on your Amazon EKS cluster before you install the Amazon CloudWatch Observability EKS add-on or the Helm chart. For more information about available installation options, see [ cert-manager documentation](https://cert-manager.io/docs/installation/). After you install it, you can opt in to using cert-manager for managing the admission webhook TLS certificates using the following additional configuration. + If you are using the Amazon CloudWatch Observability EKS add-on ``` --configuration-values '{ "admissionWebhooks": { "certManager": { "enabled": true } } }' ``` + If you are using the Helm chart ``` --set admissionWebhooks.certManager.enabled=true ``` ``` --configuration-values '{ "admissionWebhooks": { "certManager": { "enabled": true } } }' ``` The advanced configuration discussed in this section will by default use a [ SelfSigned](https://cert-manager.io/docs/configuration/selfsigned/) issuer. ### Collect Amazon EBS volume IDs If you want to collect Amazon EBS volume IDs in the performance logs, you must add another policy to the IAM role that is attached to the worker nodes or to the service account. Add the following as an inline policy. For more information, see [ Adding and Removing IAM Identity Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "ec2:DescribeVolumes" ], "Resource": "*", "Effect": "Allow" } ] } ``` ------ ## Collect Java Management Extensions (JMX) metrics The CloudWatch agent supports Java Management Extensions (JMX) metrics collection on Amazon EKS. This allows you to collect additional metrics from Java applications running on Amazon EKS clusters enabling insight into performance, memory usage, traffic, and other critical metrics. For more information, see [Collect Java Management Extensions (JMX) metrics](CloudWatch-Agent-JMX-metrics.md). ## Enable Kueue metrics Beginning with version `v2.4.0-eksbuild.1` of the the CloudWatch Observability EKS add-on, Container Insights for Amazon EKS supports collecting Kueue metrics from Amazon EKS clusters. For more information about these metrics, see [Kueue metrics](Container-Insights-metrics-EKS.md#Container-Insights-metrics-Kueue). If you are using the Amazon SageMaker AI Hyperpod Task Governance EKS add-on, you can skip the steps in the **Prerequisites** section and just follow the steps in [Enable the configuration flag](#enable-Kueue-metrics-flag). ### Prerequisites Before you install Kueue in your Amazon EKS cluster, make the following updates in the manifest file: 1. Enable the optional cluster queue resource metrics for Kueue. To do this, modify the in-line `controller_manager_config.yaml` in the `kueue-system` ConfigMap. In the `metrics` section, add or uncomment the line `enableClusterQueueResources: true`. ``` apiVersion: v1 data: controller_manager_config.yaml: | apiVersion: config.kueue.x-k8s.io/v1beta1 kind: Configuration health: healthProbeBindAddress: :8081 metrics: bindAddress: :8080 enableClusterQueueResources: true <-- ADD/UNCOMMENT THIS LINE ``` 1. By default, all `k8s` services are available cluster-wide. Kueue creates a service `kueue-controller-manager-metrics-service` for exposing metrics. To prevent duplicate observations for metrics, modify this service to allow access only to the metrics service from the same node. To do this, add the line `internalTrafficPolicy: Local` to the `kueue-controller-manager-metrics-service` definition. ``` apiVersion: v1 kind: Service metadata: labels: ... name: kueue-controller-manager-metrics-service namespace: kueue-system spec: ports: - name: https port: 8443 protocol: TCP targetPort: https internalTrafficPolicy: Local <-- ADD THIS LINE selector: control-plane: controller-manager ``` 1. Lastly, the `kueue-controller-manager` pod creates a `kube-rbac-proxy` container. This container currently has a high level of logging verbosity, which causes the cluster's bearer token to be logged by that container when the metrics scraper accesses the `kueue-controller-manager-metrics-service`. We recommend that you decrease this logging verbosity. The default value in the manifest distributed by Kueue is 10, we recommend to change it to 0. ``` apiVersion: apps/v1 kind: Deployment metadata: labels: ... name: kueue-controller-manager namespace: kueue-system spec: ... template: ... spec: containers: ... - args: - --secure-listen-address=0.0.0.0:8443 - --upstream=http://127.0.0.1:8080/ - --logtostderr=true - --v=0 <-- CHANGE v=10 TO v=0 image: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0 name: kube-rbac-proxy ... ``` ### Enable the configuration flag To enable the Kueue metrics, you must enable `kueue_container_insights` in the add-on additional configuration. You can do this either by using the AWS CLI to set up the EKS Observability add-on, or by using the Amazon EKS console. After you have successfully installed the EKS Observability add-on with one of the following methods, you can view your Amazon EKS cluster metrics under the HyperPod console **Dashboard** tab. ------ #### [ AWS CLI ] **To enable Kueue metrics using the AWS CLI** + Enter the following AWS CLI command to install the add-on. ``` aws eks create-addon --cluster-name cluster-name --addon-name amazon-cloudwatch-observability --configuration-values "configuration_json_file" ``` The following is an example of the JSON file with the configuration values. ``` { "agent": { "config": { "logs": { "metrics_collected": { "kubernetes": { "kueue_container_insights": true, "enhanced_container_insights": true }, "application_signals": { } } }, "traces": { "traces_collected": { "application_signals": { } } } }, }, } ``` ------ #### [ Amazon EKS console ] **To enable Kueue metrics using the Amazon EKS console** 1. Open the Amazon EKS console at [https://console.aws.amazon.com/eks/home\$1/clusters](https://console.aws.amazon.com/eks/home#/clusters). 1. Choose the name of your cluster. 1. Choose **Add-ons**. 1. Find the **Amazon CloudWatch Observability** add-on in the list, and install it. When you do so, choose **Optional configuration** and include the following JSON configuration values. ``` { "agent": { "config": { "logs": { "metrics_collected": { "kubernetes": { "kueue_container_insights": true, "enhanced_container_insights": true }, "application_signals": { } } }, "traces": { "traces_collected": { "application_signals": { } } } }, }, } ``` ------ ## Appending OpenTelemetry collector configuration files The CloudWatch agent supports supplemental OpenTelemetry collector configuration files alongside its own configuration files. This feature allows you to use CloudWatch agent features such as CloudWatch Application Signals or Container Insights through the CloudWatch agent configuration and bring in your existing OpenTelemetry collector configuration with a single agent. To prevent merge conflicts with pipelines automatically created by CloudWatch agent, we recommend that you add a custom suffix to each of the components and pipelines in your OpenTelemetry collector configuration. This will prevent clashing and merge conflicts. + If you are using the Amazon CloudWatch Observability EKS add-on ``` --configuration-values file://values.yaml ``` or ``` --configuration-values ' agent: otelConfig: receivers: otlp/custom-suffix: protocols: http: {} exporters: awscloudwatchlogs/custom-suffix: log_group_name: "test-group" log_stream_name: "test-stream" service: pipelines: logs/custom-suffix: receivers: [otlp/custom-suffix] exporters: [awscloudwatchlogs/custom-suffix] ' ``` + If you are using the Helm chart ``` --set agent.otelConfig=' receivers: otlp/custom-suffix: protocols: http: {} exporters: awscloudwatchlogs/custom-suffix: log_group_name: "test-group" log_stream_name: "test-stream" service: pipelines: logs/custom-suffix: receivers: [otlp/custom-suffix] exporters: [awscloudwatchlogs/custom-suffix] ' ``` ## Enabling APM through Application Signals for your Amazon EKS cluster By default, OpenTelemetry (OTEL) based Application Performance Monitoring (APM) is enabled through Application Signals when installing either the CloudWatch Observability EKS add-on (V5.0.0 or greater) or the Helm chart. You can further customize specific settings using the advanced configuration for the Amazon EKS add-on or by overriding values with the Helm chart. **Note** If you use any OpenTelemetry (OTEL) based APM solution, enabling Application Signals affects your existing observability setup. Review your current implementation before proceeding. To maintain your existing APM setup after upgrading to V5.0.0 or later, see [Opt out of Application Signals](#Opting-out-App-Signals). **Application Signals Auto monitor** Version 5.0.0 of the CloudWatch Observability Amazon EKS add-on and Helm chart introduces new functionality. You can now automatically enable Application Signals for all or specific service workloads in your EKS cluster through the Auto monitor configuration. The following `autoMonitor` settings can be specified within the `applicationSignals` section under the `manager` section of the advanced configuration. + *monitorAllServices* – A boolean flag to enable (true) or disable (false) monitoring of all service workloads by Auto monitor. Defaults to true. Enabling this flag will ensure that all Kubernetes workloads (Deployments, DaemonSets, and StatefulSets) in the cluster that are mapped to a Kubernetes Service will be in scope for automatic enablement of Application Signals when they are brought up for the first time (or when restarted for existing workloads). The system excludes workloads in the `kube-system` and `amazon-cloudwatch` namespaces by default. + *languages * – A list of strings specifying the set of languages that Application Signals will try to automatically instrument your services with, when `monitorAllServices` is enabled. Defaults to all the [supported languages](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html). + *restartPods* – A boolean flag controls whether workloads restart after configuration changes. Defaults to false. Enabling this flag to `true` controls whether Kubernetes workloads within Auto monitor scope restart automatically when saving configuration changes. Any settings on your Kubernetes workloads that influence the restart of the pods such as `updateStrategy` will be considered. Consider that restarting may cause service downtime. + *customSelector* – Settings to select specific Kubernetes namespaces or workloads for Auto monitor. + *java* – Specify workloads to automatically instrument with Java + *python* – Specify workloads to automatically instrument with Python + *nodejs* – Specify workloads to automatically instrument with Node.js + *dotnet* – Specify workloads to automatically instrument with .NET For each of the languages above, the following fields can be configured. + *namespaces* – A list of strings specifying the namespaces to be selected. Defaults to an empty list, that is [] + *deployments* – A list of strings specifying the deployments to be selected. Specify in `namespace/deployment`format. Defaults to an empty list, that is [] + *daemonsets* – A list of strings specifying the daemonsets to be selected. Specify in `namespace/daemonset` format. Defaults to an empty list, that is [] + *statefulsets* – A list of strings specifying the statefulsets to be selected. Specify in `namespace/statefulset` format. Defaults to an empty list, that is [] + *exclude* – Settings to exclude specific Kubernetes namespaces or workloads from Auto monitor. Excluding a workload takes precedence when the same workload is also within scope of `monitorAllServices` or `customSelector`. + *java* – Specify workloads to exclude from being automatically instrumented with Java + *python* – Specify workloads to exclude from being automatically instrumented with Python + *nodejs* – Specify workloads to exclude from being automatically instrumented with Node.js + *dotnet* – Specify workloads to exclude from being automatically instrumented with .NET For each of the languages above, the following fields can be configured. + *namespaces* – A list of strings specifying the namespaces to be excluded. Defaults to an empty list, that is [] + *deployments* – A list of strings specifying the deployments to be excluded. Specify in `namespace/deployment`format. Defaults to an empty list, that is [] + *daemonsets* – A list of strings specifying the daemonsets to be excluded. Specify in `namespace/daemonset` format. Defaults to an empty list, that is [] + *statefulsets* – A list of strings specifying the statefulsets to be excluded. Specify in `namespace/statefulset` format. Defaults to an empty list, that is [] The following is an example configuration that automatically enables Application Signals for all existing and new service workloads on the cluster. ``` manager: applicationSignals: autoMonitor: monitorAllServices: true restartPods: true ``` The following is an example configuration that automatically enables Application Signals for any new service workload that is brought up and for any existing service workload that is explicitly restarted on the cluster. ``` manager: applicationSignals: autoMonitor: monitorAllServices: true ``` The following is an example configuration that automatically enables Application Signals with Java for all existing and new pods corresponding to a workload in the `pet-warehouse` namespace. ``` manager: applicationSignals: autoMonitor: restartPods: true customSelector: java: namespaces: ["pet-warehouse"] ``` The following is an example configuration that automatically enables Application Signals with Python for all existing and new service workloads in the cluster excluding the `pet-clinic` deployment. ``` manager: applicationSignals: autoMonitor: monitorAllServices: true languages: ["python"] restartPods: true exclude: python: deployments: ["pet-warehouse/pet-clinic"] ``` The following is an example configuration that automatically enables Application Signals with Java for all service workloads in the cluster except for the ones in the `python-apps` namespace and further enables Application Signals with Python specifically for the `sample-python-app` deployment in the `python-apps` namespace. ``` manager: applicationSignals: autoMonitor: monitorAllServices: true languages: ["java"] restartPods: true customSelector: python: deployments: ["python-apps/sample-python-app"] exclude: java: namespaces: ["python-apps"] ``` ## Considerations for large Kubernetes clusters If you run large Kubernetes clusters, you might need additional configuration to ensure the CloudWatch agent runs reliably. The following sections describe common issues and recommended configurations for large clusters. **Separate agent installations for cluster-level and node-level metrics** **Note** This section applies only if you have [Container Insights with enhanced observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/container-insights-detailed-metrics.html) enabled. If you use [Container Insights with OpenTelemetry metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/container-insights-otel-metrics.html) exclusively, the default installation already deploys separate agent installations. By default, the CloudWatch agent runs as a daemonset on each node in your cluster. One agent Pod is elected as a leader to collect cluster-level metrics. These metrics include control plane metrics and workload status metrics. On large clusters, this leader Pod may be terminated with an out-of-memory (OOM) error because it performs additional work while sharing the same resource limits as all other agent Pods. To determine if you are experiencing this issue, check for agent Pods in a failed state with exit code `137` and reason `OOMKilled`. A common symptom is that one agent Pod (the leader) crashes, a new leader is elected, and that Pod also crashes, creating a recurring cycle. To confirm this behavior, inspect the leader election ConfigMap by running the following command. ``` kubectl describe configmap cwagent-clusterleader -n amazon-cloudwatch ``` If the ConfigMap shows frequent changes to the leader entry, this indicates that a new leader is being elected repeatedly, which confirms the issue. To avoid this issue, you can separate the agent into two installations: + A daemonset for node-level metrics and Application Signals + A deployment for cluster-level metrics This separation lets you manage and scale each installation independently. To enable this configuration, use the advanced configuration to define two entries in the `agents` section: 1. Set the `CWAGENT_ROLE` environment variable to `NODE` on the daemonset agent. 1. Set the `CWAGENT_ROLE` environment variable to `LEADER` on the deployment agent. **Important** The deployment agent configuration must enable only Container Insights. Don't enable Application Signals on the deployment agent. Both installations run with `hostNetwork` by default, and enabling Application Signals on both causes port binding conflicts. The following example shows an advanced configuration that separates the agent into two installations. ``` agents: - name: cloudwatch-agent env: - name: CWAGENT_ROLE value: NODE - name: cloudwatch-agent-ci-leader mode: deployment config: { "logs": { "metrics_collected": { "kubernetes": { "enhanced_container_insights": true } } } } env: - name: CWAGENT_ROLE value: LEADER resources: limits: memory: 10Gi cpu: 2000m ``` In this example, besides setting the `CWAGENT_ROLE` environment variable on each agent installation, it overrides the default mode, configuration, and resource limits for the deployment agent only. The base `cloudwatch-agent` installation uses the defaults. You can further customize each agent installation by adding `nodeSelector`, `tolerations`, or `nodeAffinity` fields to schedule the leader Pod on a specific node. The resource limits shown in the preceding example are illustrative. Adjust these values based on your cluster size and the number of metrics that the leader Pod collects. ## Considerations for Amazon EKS Hybrid Nodes Node-level metrics aren't available for hybrid nodes because [Container Insights](ContainerInsights.md) depends on the availability of the [EC2 Instance Metadata Service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html) (IMDS) for node-level metrics. Cluster, workload, Pod, and container-level metrics are available for hybrid nodes. After you install the add-on, patch the `amazoncloudwatchagents` resource to add the `RUN_WITH_IRSA` environment variable so the agent runs successfully on hybrid nodes: ``` kubectl patch amazoncloudwatchagents cloudwatch-agent -n amazon-cloudwatch --type=json -p '[{"op":"add","path":"/spec/env/-","value":{"name":"RUN_WITH_IRSA","value":"True"}}]' ``` ## Troubleshooting the Amazon CloudWatch Observability EKS add-on or the Helm chart Use the following information to help troubleshoot problems with the Amazon CloudWatch Observability EKS add-on or the Helm chart **Topics** + [ ### Updating and deleting the Amazon CloudWatch Observability EKS add-on or the Helm chart ](#EKS-addon-troubleshoot-update) + [ ### Verify the version of the CloudWatch agent used by the Amazon CloudWatch Observability EKS add-on or the Helm chart ](#EKS-addon-troubleshoot-version) + [ ### Handling a ConfigurationConflict when managing the add-on or the Helm chart ](#EKS-addon-troubleshoot-conflict) ### Updating and deleting the Amazon CloudWatch Observability EKS add-on or the Helm chart For instructions about updating or deleting the Amazon CloudWatch Observability EKS add-on, see [Managing Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/managing-add-ons.html). Use `amazon-cloudwatch-observability` as the name of the add-on. To delete the Helm chart in a cluster, enter the following command. ``` helm delete amazon-cloudwatch-observability -n amazon-cloudwatch --wait ``` ### Verify the version of the CloudWatch agent used by the Amazon CloudWatch Observability EKS add-on or the Helm chart The Amazon CloudWatch Observability EKS add-on and the Helm chart installs a custom resource of kind `AmazonCloudWatchAgent` that controls the behavior of the CloudWatch agent daemonset on the cluster, including the version of the CloudWatch agent being used. You can get a list of all the `AmazonCloudWatchAgent` custom resources installed on your cluster u by entering the following command: ``` kubectl get amazoncloudwatchagent -A ``` In the output of this command, you should be able to check the version of the CloudWatch agent. Alternatively, you can also describe the `amazoncloudwatchagent` resource or one of the `cloudwatch-agent-*` pods running on your cluster to inspect the image being used. ### Handling a ConfigurationConflict when managing the add-on or the Helm chart When you install or update the Amazon CloudWatch Observability EKS add-on or the Helm chart, if you notice a failure caused by existing resources, it is likely because you already have the CloudWatch agent and its associated components such as the ServiceAccount, the ClusterRole and the ClusterRoleBinding installed on the cluster. The error displayed by the add-on will include `Conflicts found when trying to apply. Will not continue due to resolve conflicts mode`, The error displayed by the Helm chart will be similar to `Error: INSTALLATION FAILED: Unable to continue with install and invalid ownership metadata.`. When the add-on or the Helm chart tries to install the CloudWatch agent and its associated components, if it detects any change in the contents, it by default fails the installation or update to avoid overwriting the state of the resources on the cluster. If you are trying to onboard to the Amazon CloudWatch Observability EKS add-on and you see this failure, we recommend deleting an existing CloudWatch agent setup that you had previously installed on the cluster and then installing the EKS add-on or Helm chart. Be sure to back up any customizations you might have made to the original CloudWatch agent setup such as a custom agent configuration, and provide these to the add-on or Helm chart when you next install or update it. If you had previously installed the CloudWatch agent for onboarding to Container Insights, see [Deleting the CloudWatch agent and Fluent Bit for Container Insights](ContainerInsights-delete-agent.md) for more information. Alternatively, the add-on supports a conflict resolution configuration option that has the capability to specify `OVERWRITE`. You can use this option to proceed with installing or updating the add-on by overwriting the conflicts on the cluster. If you are using the Amazon EKS console, you'll find the **Conflict resolution method** when you choose the **Optional configuration settings** when you create or update the add-on. If you are using the AWS CLI, you can supply the `--resolve-conflicts OVERWRITE` to your command to create or update the add-on. ## Opt out of Application Signals Fine-tune your service monitoring preferences in the CloudWatch console or with the SDK. For versions before 5.0.0, to disable Application Signals auto-monitoring, follow the procedure below: **Using CLI or SDK** The following configuration can be applied either as an advanced configuration to the EKS add-on or as a values override when using the helm chart. ``` { "manager": { "applicationSignals": { "autoMonitor": { "monitorAllServices": false } } } } ``` Restart your services for the changes to take effect. **Using the console** Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, under **Application Signals (APM)**, choose **Services**. 1. Choose **Enable Application Signals** to view the enablement page. 1. Clear the **Auto-Monitor** checkbox for each service you don't want to monitor. 1. Restart your services for the changes to take effect. # Set up the CloudWatch agent with security-enhanced Linux (SELinux) If your system has security-enhanced Linux (SELinux) enabled, you must apply the appropriate security policies to ensure that the CloudWatch agent runs in a confined domain. ## Prerequisites Before you can configure SELinux for the agent, check the following **prerequisites**: **To complete the prerequisites for using the CloudWatch agent with SELinux** 1. If you haven't done so, install the following SELinux policy development packages: ``` sudo yum update sudo yum install -y selinux-policy-devel policycoreutils-devel rpm-build git ``` 1. Run the following command to check your system's SELinux status: ``` sestatus ``` Example output: ``` SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: permissive Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 ``` If you find that SELinux is currently disabled, do the following: 1. Open the SELinux file by entering the following command: ``` sudo vi /etc/selinux/config ``` 1. Set the `SELINUX` parameter to either `permissive` or `enforcing`. For example: ``` SELINUX=enforcing ``` 1. Save the file and reboot the system to apply the changes. ``` sudo reboot ``` 1. Ensure that the CloudWatch agent is running as a `systemd` service. This is required to use it within a confined SELinux domain. ``` sudo systemctl status amazon-cloudwatch-agent ``` If the agent is correctly configured, the output should indicate that it is `active (running)` and `enabled` at startup. ## Configure SELinux for the agent After you complete the prerequisites, you can configure SELinux. **To configure SELinux for the CloudWatch agent** 1. Clone the SELinux policy for the CloudWatch agent by entering the following command: ``` git clone https://github.com/aws/amazon-cloudwatch-agent-selinux.git ``` 1. Navigate to the cloned repository and then update the script permissions by entering the following commands: ``` cd amazon-cloudwatch-agent-selinux chmod +x amazon_cloudwatch_agent.sh ``` 1. Use `sudo` to run the SELinux policy installation script by entering the following command. During execution, the script prompts you to enter `y` or `n` to allow automatic restart. This restart ensures that the agent transitions into the correct SELinux domain. ``` sudo ./amazon_cloudwatch_agent.sh ``` 1. If the CloudWatch agent hasn't been restarted yet, restart it to ensure that it transitions to the correct SELinux domain: ``` sudo systemctl restart amazon-cloudwatch-agent ``` 1. Verify that CloudWatch Agent is running in the confined domain by entering the following command: ``` ps -efZ | grep amazon-cloudwatch-agent ``` If the agent is correctly confined, the output should indicate a SELinux-confined domain instead of `unconfined_service_t`. The following is an example of output when the agent is correctly confined. ``` system_u:system_r:confined_t:s0 root 1234 1 0 12:00 ? 00:00:10 /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent ``` After SELinux is configured, you can proceed to configure the agent to collect metrics, logs, and traces. For more information, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). # Create the CloudWatch agent configuration file Before running the CloudWatch agent on any servers, you must create one or more CloudWatch agent configuration files. The agent configuration file is a JSON file that specifies the metrics, logs, and traces that the agent is to collect, including custom metrics. You can create it by using the wizard or by creating it yourself from scratch. You could also use the wizard to initially create the configuration file and then modify it manually. If you create or modify the file manually, the process is more complex, but you have more control over the metrics collected and can specify metrics not available through the wizard. Any time you change the agent configuration file, you must then restart the agent to have the changes take effect. To restart the agent, follow the instructions in [(Optional) Modify the common configuration and named profile for CloudWatch agent](installing-cloudwatch-agent-ssm.md#CloudWatch-Agent-profile-instance-fleet). After you have created a configuration file, you can save it manually as a JSON file and then use this file when installing the agent on your servers. Alternatively, you can store it in Systems Manager Parameter Store if you're going to use Systems Manager when you install the agent on servers. The CloudWatch agent supports using multiple configuration files. For more information, see [Creating multiple CloudWatch agent configuration files](#CloudWatch-Agent-multiple-config-files). Metrics, logs, and traces collected by the CloudWatch agent incur charges. For more information about pricing, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing). **Topics** + [ # Create the CloudWatch agent configuration file with the wizard ](create-cloudwatch-agent-configuration-file-wizard.md) + [ ## Creating multiple CloudWatch agent configuration files ](#CloudWatch-Agent-multiple-config-files) + [ # Manually create or edit the CloudWatch agent configuration file ](CloudWatch-Agent-Configuration-File-Details.md) # Create the CloudWatch agent configuration file with the wizard The agent configuration file wizard, `amazon-cloudwatch-agent-config-wizard`, asks a series of questions to help you configure the CloudWatch agent for your needs. This section describes the credentials required for the configuration file. It describes how to run the CloudWatch agent configuration wizard. It also describes the metrics that are predefined in the wizard. ## Required credentials The wizard can autodetect the credentials and AWS Region to use if you have the AWS credentials and configuration files in place before you start the wizard. For more information about these files, see [ Configuration and Credential Files](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html) in the *AWS Systems Manager User Guide*. In the AWS credentials file, the wizard checks for default credentials and also looks for an `AmazonCloudWatchAgent` section such as the following: ``` [AmazonCloudWatchAgent] aws_access_key_id = my_access_key aws_secret_access_key = my_secret_key ``` The wizard displays the default credentials, the credentials from the `AmazonCloudWatchAgent`, and an `Others` option. You can select which credentials to use. If you choose `Others`, you can input credentials. For *my\$1access\$1key* and *my\$1secret\$1key*, use the keys from the IAM user that has the permissions to write to Systems Manager Parameter Store. In the AWS configuration file, you can specify the Region that the agent sends metrics to if it's different than the `[default]` section. The default is to publish the metrics to the Region where the Amazon EC2 instance is located. If the metrics should be published to a different Region, specify the Region here. In the following example, the metrics are published to the `us-west-1` Region. ``` [AmazonCloudWatchAgent] region = us-west-1 ``` ## Run the CloudWatch agent configuration wizard **To create the CloudWatch agent configuration file** 1. Start the CloudWatch agent configuration wizard by entering the following: ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard ``` On a server running Windows Server, run the following commands to launch the wizard: ``` cd "C:\Program Files\Amazon\AmazonCloudWatchAgent" ``` ``` .\amazon-cloudwatch-agent-config-wizard.exe ``` 1. Answer the questions to customize the configuration file for your server. 1. If you're storing the configuration file locally, the configuration file `config.json` is stored in `/opt/aws/amazon-cloudwatch-agent/bin/` on Linux servers, and is stored in `C:\Program Files\Amazon\AmazonCloudWatchAgent` on Windows Server. You can then copy this file to other servers where you want to install the agent. If you're going to use Systems Manager to install and configure the agent, be sure to answer **Yes** when prompted whether to store the file in Systems Manager Parameter Store. You can also choose to store the file in Parameter Store even if you aren't using the SSM Agent to install the CloudWatch agent. To be able to store the file in Parameter Store, you must use an IAM role with sufficient permissions. ## CloudWatch agent predefined metric sets The wizard is configured with predefined sets of metrics, with different detail levels. These sets of metrics are shown in the following tables. For more information about these metrics, see [Metrics collected by the CloudWatch agent](metrics-collected-by-CloudWatch-agent.md). **Note** Parameter Store supports parameters in Standard and Advanced tiers. These parameter tiers are not related to the Basic, Standard, and Advanced levels of metric details that are described in these tables. **Amazon EC2 instances running Linux** | Detail level | Metrics included | | --- | --- | | **Basic** | **Mem:** mem\$1used\$1percent **Disk:** disk\$1used\$1percent The `disk` metrics such as `disk_used_percent` have a dimension for `Partition`, which means that the number of custom metrics generated is dependent on the number of partitions associated with your instance. The number of disk partitions you have depends on which AMI you are using and the number of Amazon EBS volumes you attach to the server. | | **Standard** | **CPU:** `cpu_usage_idle`, `cpu_usage_iowait`, `cpu_usage_user`, `cpu_usage_system` **Disk:** `disk_used_percent`, `disk_inodes_free` **Diskio:** `diskio_io_time` **Mem:** `mem_used_percent` **Swap:** `swap_used_percent` | | **Advanced** | **CPU:** `cpu_usage_idle`, `cpu_usage_iowait`, `cpu_usage_user`, `cpu_usage_system` **Disk:** `disk_used_percent`, `disk_inodes_free` **Diskio:** `diskio_io_time`, `diskio_write_bytes`, `diskio_read_bytes`, `diskio_writes`, `diskio_reads` **Mem:** `mem_used_percent` **Netstat:** `netstat_tcp_established`, `netstat_tcp_time_wait` **Swap:** `swap_used_percent` | **On-premises servers running Linux** | Detail level | Metrics included | | --- | --- | | **Basic** | **Disk:** `disk_used_percent` **Diskio:** `diskio_write_bytes`, `diskio_read_bytes`, `diskio_writes`, `diskio_reads` **Mem:** `mem_used_percent` **Net:** `net_bytes_sent`, `net_bytes_recv`, `net_packets_sent`, `net_packets_recv` **Swap:** `swap_used_percent` | | **Standard** | **CPU:** `cpu_usage_idle`, `cpu_usage_iowait` **Disk:** `disk_used_percent`, `disk_inodes_free` **Diskio:** `diskio_io_time`, `diskio_write_bytes`, `diskio_read_bytes`, `diskio_writes`, `diskio_reads` **Mem:** `mem_used_percent` **Net:** `net_bytes_sent`, `net_bytes_recv`, `net_packets_sent`, `net_packets_recv` **Swap:** `swap_used_percent` | | **Advanced** | **CPU:** `cpu_usage_guest`, `cpu_usage_idle`, `cpu_usage_iowait`, `cpu_usage_steal`, `cpu_usage_user`, `cpu_usage_system` **Disk:** `disk_used_percent`, `disk_inodes_free` **Diskio:** `diskio_io_time`, `diskio_write_bytes`, `diskio_read_bytes`, `diskio_writes`, `diskio_reads` **Mem:** `mem_used_percent` **Net:** `net_bytes_sent`, `net_bytes_recv`, `net_packets_sent`, `net_packets_recv` **Netstat:** `netstat_tcp_established`, `netstat_tcp_time_wait` **Swap:** `swap_used_percent` | **Amazon EC2 instances running Windows Server** **Note** The metric names listed in this table display how the metric appears when viewed in the console. The actual metric name might not include the first word. For example, the actual metric name for `LogicalDisk % Free Space` is just `% Free Space`. | Detail level | Metrics included | | --- | --- | | **Basic** | **Memory:** `Memory % Committed Bytes In Use` **LogicalDisk:** `LogicalDisk % Free Space` | | **Standard** | **Memory:** `Memory % Committed Bytes In Use` **Paging:** `Paging File % Usage` **Processor:** `Processor % Idle Time`, `Processor % Interrupt Time`, `Processor % User Time` **PhysicalDisk:** `PhysicalDisk % Disk Time` **LogicalDisk:** `LogicalDisk % Free Space` | | **Advanced** | **Memory:** `Memory % Committed Bytes In Use` **Paging:** `Paging File % Usage` **Processor:** `Processor % Idle Time`, `Processor % Interrupt Time`, `Processor % User Time` **LogicalDisk:** `LogicalDisk % Free Space` **PhysicalDisk:** `PhysicalDisk % Disk Time`, `PhysicalDisk Disk Write Bytes/sec`, `PhysicalDisk Disk Read Bytes/sec`, `PhysicalDisk Disk Writes/sec`, `PhysicalDisk Disk Reads/sec` **TCP:** `TCPv4 Connections Established`, `TCPv6 Connections Established` | **On-premises server running Windows Server** **Note** The metric names listed in this table display how the metric appears when viewed in the console. The actual metric name might not include the first word. For example, the actual metric name for `LogicalDisk % Free Space` is just `% Free Space`. | Detail level | Metrics included | | --- | --- | | **Basic** | **Paging: **`Paging File % Usage` **Processor:** `Processor % Processor Time` **LogicalDisk:**`LogicalDisk % Free Space` **PhysicalDisk:** `PhysicalDisk Disk Write Bytes/sec`, `PhysicalDisk Disk Read Bytes/sec`, `PhysicalDisk Disk Writes/sec`, `PhysicalDisk Disk Reads/sec` **Memory:** `Memory % Committed Bytes In Use` **Network Interface:** `Network Interface Bytes Sent/sec`, `Network Interface Bytes Received/sec`, `Network Interface Packets Sent/sec`, `Network Interface Packets Received/sec` | | **Standard** | **Paging:** `Paging File % Usage` **Processor:** `Processor % Processor Time`, `Processor % Idle Time`, `Processor % Interrupt Time` **LogicalDisk:** `LogicalDisk % Free Space` **PhysicalDisk:** `PhysicalDisk % Disk Time`, `PhysicalDisk Disk Write Bytes/sec`, `PhysicalDisk Disk Read Bytes/sec`, `PhysicalDisk Disk Writes/sec`, `PhysicalDisk Disk Reads/sec` **Memory:** `Memory % Committed Bytes In Use` **Network Interface:** `Network Interface Bytes Sent/sec`, `Network Interface Bytes Received/sec`, `Network Interface Packets Sent/sec`, `Network Interface Packets Received/sec` | | **Advanced** | **Paging:**`Paging File % Usage` **Processor:** `Processor % Processor Time`, `Processor % Idle Time`, `Processor % Interrupt Time`, `Processor % User Time` **LogicalDisk:** `LogicalDisk % Free Space` **PhysicalDisk:** `PhysicalDisk % Disk Time`, `PhysicalDisk Disk Write Bytes/sec`, `PhysicalDisk Disk Read Bytes/sec`, `PhysicalDisk Disk Writes/sec`, `PhysicalDisk Disk Reads/sec` **Memory:** `Memory % Committed Bytes In Use` **Network Interface:** `Network Interface Bytes Sent/sec`, `Network Interface Bytes Received/sec`, `Network Interface Packets Sent/sec`, `Network Interface Packets Received/sec` **TCP:** `TCPv4 Connections Established`, `TCPv6 Connections Established` | # Examples of configuration files **Basic system metrics configuration** ``` { "agent": { "metrics_collection_interval": 60, "region": "us-east-1" }, "metrics": { "namespace": "MySystem", "metrics_collected": { "cpu": { "resources": ["*"], "measurement": ["usage_active", "usage_system", "usage_user"] }, "mem": { "measurement": ["used_percent"] }, "disk": { "resources": ["/"], "measurement": ["used_percent"] } }, "append_dimensions": { "InstanceId": "${aws:InstanceId}" } } } ``` **Web server monitoring configuration ** ``` { "agent": { "metrics_collection_interval": 60, "region": "us-east-1" }, "metrics": { "namespace": "WebServer", "metrics_collected": { "cpu": { "resources": ["*"], "measurement": ["usage_active"] }, "mem": { "measurement": ["used_percent"] }, "net": { "resources": ["eth0"], "measurement": ["bytes_sent", "bytes_recv"] } } }, "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "/var/log/apache2/access.log", "log_group_name": "apache-access-logs", "log_stream_name": "{instance_id}-access" }, { "file_path": "/var/log/apache2/error.log", "log_group_name": "apache-error-logs", "log_stream_name": "{instance_id}-error" } ] } } } } ``` **Database server configuration ** ``` { "agent": { "metrics_collection_interval": 30, "region": "us-east-1" }, "metrics": { "namespace": "DatabaseServer", "metrics_collected": { "cpu": { "resources": ["*"], "measurement": ["usage_active", "usage_iowait"] }, "mem": { "measurement": ["used_percent", "available_percent"] }, "disk": { "resources": ["/", "/data"], "measurement": ["used_percent", "inodes_free"] }, "diskio": { "resources": ["*"], "measurement": ["read_bytes", "write_bytes", "io_time"] } }, "append_dimensions": { "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}", "AutoScalingGroupName": "${aws:AutoScalingGroupName}" } } } ``` ## Creating multiple CloudWatch agent configuration files On both Linux servers and Windows servers, you can set up the CloudWatch agent to use multiple configuration files. For example, you can use a common configuration file that collects a set of metrics, logs, and traces that you always want to collect from all servers in your infrastructure. You can then use additional configuration files that collect metrics from certain applications or in certain situations. To set this up, first create the configuration files that you want to use. Any configuration files that will be used together on the same server must have different file names. You can store the configuration files on servers or in Parameter Store. Start the CloudWatch agent using the `fetch-config` option and specify the first configuration file. To append the second configuration file to the running agent, use the same command but with the `append-config` option. All metrics, logs, and traces listed in either configuration file are collected. The following example commands illustrate this scenario using configurations stores as files. The first line starts the agent using the `infrastructure.json` configuration file, and the second line appends the `app.json` configuration file. The following example commands are for Linux. ``` /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/tmp/infrastructure.json ``` ``` /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:/tmp/app.json ``` The following example commands are for Windows Server. ``` & "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\infrastructure.json" ``` ``` & "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a append-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\app.json" ``` The following example configuration files illustrate a use for this feature. The first configuration file is used for all servers in the infrastructure, and the second collects only logs from a certain application and is appended to servers running that application. **infrastructure.json** ``` { "metrics": { "metrics_collected": { "cpu": { "resources": [ "*" ], "measurement": [ "usage_active" ], "totalcpu": true }, "mem": { "measurement": [ "used_percent" ] } } }, "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log", "log_group_name": "amazon-cloudwatch-agent.log" }, { "file_path": "/var/log/messages", "log_group_name": "/var/log/messages" } ] } } } } ``` **app.json** ``` { "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "/app/app.log*", "log_group_name": "/app/app.log" } ] } } } } ``` Any configuration files appended to the configuration must have different file names from each other and from the initial configuration file. If you use `append-config` with a configuration file with the same file name as a configuration file that the agent is already using, the append command overwrites the information from the first configuration file instead of appending to it. This is true even if the two configuration files with the same file name are on different file paths. The preceding example shows the use of two configuration files, but there is no limit to the number of configuration files that you can append to the agent configuration. You can also mix the use of configuration files located on servers and configurations located in Parameter Store. # Manually create or edit the CloudWatch agent configuration file The CloudWatch agent configuration file is a JSON file with four sections: `agent`, `metrics`, `logs`, and `traces`. + The `agent` section includes fields for the overall configuration of the agent. + The `metrics` section specifies the custom metrics for collection and publishing to CloudWatch. If you're using the agent only to collect logs, you can omit the `metrics` section from the file. + The `logs` section specifies what log files are published to CloudWatch Logs. This can include events from the Windows Event Log if the server runs Windows Server. + The `traces` section specifies the sources for traces that are collected and sent to AWS X-Ray. This section explains the structure and fields of the CloudWatch agent configuration file. You can view the schema definition for this configuration file. The schema definition is located at `installation-directory/doc/amazon-cloudwatch-agent-schema.json` on Linux servers and at `installation-directory/amazon-cloudwatch-agent-schema.json` on servers running Windows Server. If you create or edit the agent configuration file manually, you can give it any name. For simplicity in troubleshooting, we recommend that you name it `/opt/aws/amazon-cloudwatch-agent/etc/cloudwatch-agent.json` on a Linux server and `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.json` on servers running Windows Server. After you have created the file, you can copy it to other servers where you want to install the agent. When the agent is started, it creates a copy of each configuration file in `/opt/aws/amazon-cloudwatch/etc/amazon-cloudwatch-agent.d` directory, with the filename prefixed with either `file_` (for local file sources) or `ssm_` (for Systems Manager parameter store sources) to indicate the configuration origin. **Note** Metrics, logs, and traces collected by the CloudWatch agent incur charges. For more information about pricing, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing). ## CloudWatch agent configuration file: Agent section The `agent` section can include the following fields. The wizard doesn't create an `agent` section. Instead, the wizard omits it and uses the default values for all fields in this section. + `metrics_collection_interval` – Optional. Specifies how often all metrics specified in this configuration file are to be collected. You can override this value for specific types of metrics. This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). The default value is 60. + `region` – Specifies the Region to use for the CloudWatch endpoint when an Amazon EC2 instance is being monitored. The metrics collected are sent to this Region, such as `us-west-1`. If you omit this field, the agent sends metrics to the Region where the Amazon EC2 instance is located. If you are monitoring an on-premises server, this field isn't used, and the agent reads the Region from the `AmazonCloudWatchAgent` profile of the AWS configuration file. + `credentials` – Specifies an IAM role to use when sending metrics, logs, and traces to a different AWS account. If specified, this field contains one parameter, `role_arn`. + `role_arn` – Specifies the Amazon Resource Name (ARN) of an IAM role to use for authentication when sending metrics, logs, and traces to a different AWS account. For more information, see [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account). + `debug` – Optional. Specifies running the CloudWatch agent with debug log messages. The default value is `false`. + `aws_sdk_log_level` – Optional. Supported only in versions 1.247350.0 and later of the CloudWatch agent. You can specify this field to have the agent perform logging for AWS SDK endpoints. The value for this field can include one or more of the following options. Separate multiple options with the `|` character. + `LogDebug` + `LogDebugWithSigning` + `LogDebugWithHTTPBody` + `LogDebugRequestRetries` + `LogDebugWithEventStreamBody` For more information about these options, see [LogLevelType](https://docs.aws.amazon.com/sdk-for-go/api/aws/#LogLevelType). + `logfile` – Specifies the location where the CloudWatch agent writes log messages. If you specify an empty string, the log goes to stderr. If you don't specify this option, the default locations are the following: + Linux: `/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log` + Windows Server: `c:\\ProgramData\\Amazon\\CloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log` The CloudWatch agent automatically rotates the log file that it creates. A log file is rotated out when it reaches 100 MB in size. The agent keeps the rotated log files for up to seven days, and it keeps as many as five backup log files that have been rotated out. Backup log files have a timestamp appended to their filename. The timestamp shows the date and time that the file was rotated out: for example, `amazon-cloudwatch-agent-2018-06-08T21-01-50.247.log.gz`. + `omit_hostname` – Optional. By default, the hostname is published as a dimension of metrics that are collected by the agent, unless you are using the `append_dimensions` field in the `metrics` section. Set `omit_hostname ` to `true` to prevent the hostname from being published as a dimension even if you are not using `append_dimensions`. The default value is `false`. + `run_as_user` – Optional. Specifies a user to use to run the CloudWatch agent. If you don't specify this parameter, the root user is used. This option is valid only on Linux servers. If you specify this option, the user must exist before you start the CloudWatch agent. For more information, see [Running the CloudWatch agent as a different user](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-run-as-user). + `user_agent` – Optional. Specifies the `user-agent` string that is used by the CloudWatch agent when it makes API calls to the CloudWatch backend. The default value is a string consisting of the agent version, the version of the Go programming language that was used to compile the agent, the runtime operating system and architecture, the build time, and the plugins enabled. + `usage_data` – Optional. By default, the CloudWatch agent sends health and performance data about itself to CloudWatch whenever it publishes metrics or logs to CloudWatch. This data incurs no costs to you. You can prevent the agent from sending this data by specifying `false` for `usage_data`. If you omit this parameter, the default of `true` is used and the agent sends the health and performance data. If you set this value to `false`, you must stop and restart the agent for it to take effect. + `service.name` – Optional. Specifies the service name to be used to populate the entity for [finding related telemetry](ExploreRelated.md). + `deployment.environment` – Optional. Specifies the environment name to be used to populate the entity for [finding related telemetry](ExploreRelated.md). + `use_dualstack_endpoint` – Optional. If this is `true`, the CloudWatch agent will use [dual stack endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#dual-stack-endpoints) for all API calls. The following is an example of an `agent` section. ``` "agent": { "metrics_collection_interval": 60, "region": "us-west-1", "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log", "debug": false, "run_as_user": "cwagent" } ``` ## CloudWatch agent configuration file: Metrics section ### Fields common to Linux and Windows On servers running either Linux or Windows Server, the `metrics` section includes the following fields: + `namespace` – Optional. The namespace to use for the metrics collected by the agent. The default value is `CWAgent`. The maximum length is 255 characters. The following is an example: ``` { "metrics": { "namespace": "Development/Product1Metrics", ...... }, } ``` + `append_dimensions` – Optional. Adds Amazon EC2 metric dimensions to all metrics collected by the agent. This also causes the agent to not publish the hostname as a dimension. The only supported key-value pairs for `append_dimensions` are shown in the following list. Any other key-value pairs are ignored. The agent supports these key-value pairs exactly as they are shown in the following list. You can't change the key values to publish different dimension names for them. + `"ImageId":"${aws:ImageId}"` sets the instance's AMI ID as the value of the `ImageId` dimension. + `"InstanceId":"${aws:InstanceId}"` sets the instance's instance ID as the value of the `InstanceId` dimension. + `"InstanceType":"${aws:InstanceType}"` sets the instance's instance type as the value of the `InstanceType` dimension. + `"AutoScalingGroupName":"${aws:AutoScalingGroupName}"` sets the instance's Auto Scaling group name as the value of the `AutoScalingGroupName` dimension. If you want to append dimensions to metrics with arbitrary key-value pairs, use the `append_dimensions` parameter in the field for that particular type of metric. If you specify a value that depends on Amazon EC2 metadata and you use proxies, you must make sure that the server can access the endpoint for Amazon EC2. For more information about these endpoints, see [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region) in the *Amazon Web Services General Reference*. + `aggregation_dimensions` – Optional. Specifies the dimensions that collected metrics are to be aggregated on. For example, if you roll up metrics on the `AutoScalingGroupName` dimension, the metrics from all instances in each Auto Scaling group are aggregated and can be viewed as a whole. You can roll up metrics along single or multiple dimensions. For example, specifying `[["InstanceId"], ["InstanceType"], ["InstanceId","InstanceType"]]` aggregates metrics for instance ID singly, instance type singly, and for the combination of the two dimensions. You can also specify `[]` to roll up all metrics into one collection, disregarding all dimensions. + `endpoint_override` – Specifies a FIPS endpoint or private link to use as the endpoint where the agent sends metrics. Specifying this and setting a private link enables you to send the metrics to an Amazon VPC endpoint. For more information, see [What Is Amazon VPC?](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html). The value of `endpoint_override` must be a string that is a URL. For example, the following part of the metrics section of the configuration file sets the agent to use a VPC Endpoint when sending metrics. ``` { "metrics": { "endpoint_override": "vpce-XXXXXXXXXXXXXXXXXXXXXXXXX.monitoring.us-east-1.vpce.amazonaws.com", ...... }, } ``` + `metrics_collected` – Required. Specifies which metrics are to be collected, including custom metrics collected through `StatsD` or `collectd`. This section includes several subsections. The contents of the `metrics_collected` section depend on whether this configuration file is for a server running Linux or Windows Server. + `metrics_destinations` – Optional. Specifies one or more destinations for all metrics defined in `metrics_collected`. If specified here, it overrides the default destination of `cloudwatch`. + `cloudwatch` – Amazon CloudWatch. + `amp` – Amazon Managed Service for Prometheus. + `workspace_id` – The ID corresponding to the Amazon Managed Service for Prometheus workspace. ``` { "metrics": { "metrics_destinations": { "cloudwatch": {}, "amp": { "workspace_id": "ws-abcd1234-ef56-7890-ab12-example" } } } } ``` + `force_flush_interval` – Specifies in seconds the maximum amount of time that metrics remain in the memory buffer before being sent to the server. No matter the setting for this, if the size of the metrics in the buffer reaches 1 MB or 1000 different metrics, the metrics are immediately sent to the server. The default value is 60. + `credentials` – Specifies an IAM role to use when sending metrics to a different account. If specified, this field contains one parameter, `role_arn`. + `role_arn` – Specifies the ARN of an IAM role to use for authentication when sending metrics to a different account. For more information, see [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account). If specified here, this value overrides the `role_arn` specified in the `agent` section of the configuration file, if any. + `service.name` – Optional. Specifies the service name to be used to populate the entity for [finding related telemetry](ExploreRelated.md). + `deployment.environment` – Optional. Specifies the environment name to be used to populate the entity for [finding related telemetry](ExploreRelated.md). ### Linux section On servers running Linux, the `metrics_collected` section of the configuration file can also contain the following fields. Many of these fields can include a `measurement` sections that lists the metrics you want to collect for that resource. These `measurement` sections can either specify the complete metric name such as `swap_used`, or just the part of the metric name that will be appended to the type of resource. For example, specifying `reads` in the `measurement` section of the `diskio` section causes the `diskio_reads` metric to be collected. + `collectd` – Optional. Specifies that you want to retrieve custom metrics using the `collectd` protocol. You use `collectd` software to send the metrics to the CloudWatch agent. For more information about the configuration options available for collectd, see [Retrieve custom metrics with collectd](CloudWatch-Agent-custom-metrics-collectd.md). + `cpu` – Optional. Specifies that CPU metrics are to be collected. This section is valid only for Linux instances. You must include at least one of the `resources` and `totalcpu` fields for any CPU metrics to be collected. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `resources` – Optional. Specify this field with a value of `*` to cause per-cpu metrics to be collected. The only allowed value is `*`. + `totalcpu` – Optional. Specifies whether to report cpu metrics aggregated across all cpu cores. The default is true. + `measurement` – Specifies the array of cpu metrics to be collected. Possible values are `time_active`, `time_guest`, `time_guest_nice`, `time_idle`, `time_iowait`, `time_irq`, `time_nice`, `time_softirq`, `time_steal`, `time_system`, `time_user`, `usage_active`, `usage_guest`, `usage_guest_nice`, `usage_idle`, `usage_iowait`, `usage_irq`, `usage_nice`, `usage_softirq`, `usage_steal`, `usage_system`, and `usage_user`. This field is required if you include `cpu`. By default, the unit for `cpu_usage_*` metrics is `Percent`, and `cpu_time_*` metrics don't have a unit. Within the entry for each individual metric, you might optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `metrics_collection_interval` – Optional. Specifies how often to collect the cpu metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Additional dimensions to use for only the cpu metrics. If you specify this field, it's used in addition to dimensions specified in the global `append_dimensions` field that is used for all types of metrics that the agent collects. + `disk` – Optional. Specifies that disk metrics are to be collected. Collects metrics only for mounted volumes. This section is valid only for Linux instances. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `resources` – Optional. Specifies an array of disk mount points. This field limits CloudWatch to collect metrics from only the listed mount points. You can specify `*` as the value to collect metrics from all mount points. The default value is to collect metrics from all mount points. + `measurement` – Specifies the array of disk metrics to be collected. Possible values are `free`, `total`, `used`, `used_percent`, `inodes_free`, `inodes_used`, and `inodes_total`. This field is required if you include `disk`. **Note** The `disk` metrics have a dimension for `Partition`, which means that the number of custom metrics generated is dependent on the number of partitions associated with your instance. The number of disk partitions you have depends on which AMI you are using and the number of Amazon EBS volumes you attach to the server. To see the default units for each `disk` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent). Within the entry for each individual metric, you might optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `ignore_file_system_types` – Specifies file system types to exclude when collecting disk metrics. Valid values include `sysfs`, `devtmpfs`, and so on. + `drop_device` – Setting this to `true` causes `Device` to not be included as a dimension for disk metrics. Preventing `Device` from being used as a dimension can be useful on instances that use the Nitro system because on those instances the device names change for each disk mount when the instance is rebooted. This can cause inconsistent data in your metrics and cause alarms based on these metrics to go to `INSUFFICIENT DATA` state. The default is `false`. + `metrics_collection_interval` – Optional. Specifies how often to collect the disk metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Specify key-value pairs to use as additional dimensions for only the disk metrics. If you specify this field, it is used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent. One key-value pair that you can use is the following. You can also specify other custom key-value pairs. + `"VolumeId":"${aws:VolumeId}"` adds a `VolumeId` dimension to your block device disk metrics. For Amazon EBS volumes, this will be the Amazon EBS Volume ID. For EC2 instance store, this will be the device serial. Using this requires the `drop_device` parameter to be set to `false`. + `diskio` – Optional. Specifies that disk i/o metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `resources` – Optional. If you specify an array of devices, CloudWatch collects metrics from only those devices. Otherwise, metrics for all devices are collected. You can also specify \$1 as the value to collect metrics from all devices. + `measurement` – Specifies the array of diskio and AWS NVMe driver metrics to be collected for Amazon EBS volumes and instance store volumes attached to Amazon EC2 instances. Possible diskio values are `reads`, `writes`, `read_bytes`, `write_bytes`, `read_time`, `write_time`, `io_time`, and `iops_in_progress`. For a list of the NVMe driver metrics for Amazon EBS volumes and Amazon EC2 instance store volumes, see [Collect Amazon EBS NVMe driver metrics](Container-Insights-metrics-EBS-Collect.md) and [Collect Amazon EC2 instance store volume NVMe driver metrics](Container-Insights-metrics-instance-store-Collect.md). This field is required if you include `diskio`. Within the entry for each individual metric, you might optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). For information on default units and description of the metrics, see [Collect Amazon EBS NVMe driver metrics](Container-Insights-metrics-EBS-Collect.md). + `metrics_collection_interval` – Optional. Specifies how often to collect the diskio metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Additional dimensions to use for only the diskio metrics. If you specify this field, it is used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent. + `swap` – Optional. Specifies that swap memory metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `measurement` – Specifies the array of swap metrics to be collected. Possible values are `free`, `used`, and `used_percent`. This field is required if you include `swap`. To see the default units for each `swap` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent). Within the entry for each individual metric, you might optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `metrics_collection_interval` – Optional. Specifies how often to collect the swap metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Additional dimensions to use for only the swap metrics. If you specify this field, it is used in addition to dimensions specified in the global `append_dimensions` field that is used for all types of metrics collected by the agent. It's collected as a high-resolution metric. + `mem` – Optional. Specifies that memory metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `measurement` – Specifies the array of memory metrics to be collected. Possible values are `active`, `available`, `available_percent`, `buffered`, `cached`, `free`, `inactive`, `shared`, `total`, `used`, and `used_percent`. This field is required if you include `mem`. To see the default units for each `mem` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent). Within the entry for each individual metric, you might optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `metrics_collection_interval` – Optional. Specifies how often to collect the mem metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Additional dimensions to use for only the mem metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics that the agent collects. + `net` – Optional. Specifies that networking metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `resources` – Optional. If you specify an array of network interfaces, CloudWatch collects metrics from only those interfaces. Otherwise, metrics for all devices are collected. You can also specify `*` as the value to collect metrics from all interfaces. + `measurement` – Specifies the array of networking metrics to be collected. Possible values are `bytes_sent`, `bytes_recv`, `drop_in`, `drop_out`, `err_in`, `err_out`, `packets_sent`, and `packets_recv`. This field is required if you include `net`. To see the default units for each `net` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent). Within the entry for each individual metric, you might optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `metrics_collection_interval` – Optional. Specifies how often to collect the net metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Additional dimensions to use for only the net metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent. + `netstat` – Optional. Specifies that TCP connection state and UDP connection metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `measurement` – Specifies the array of netstat metrics to be collected. Possible values are `tcp_close`, `tcp_close_wait`, `tcp_closing`, `tcp_established`, `tcp_fin_wait1`, `tcp_fin_wait2`, `tcp_last_ack`, `tcp_listen`, `tcp_none`, `tcp_syn_sent`, `tcp_syn_recv`, `tcp_time_wait`, and `udp_socket`. This field is required if you include `netstat`. To see the default units for each `netstat` metric, see [Metrics collected by the CloudWatch agent on Linux and macOS instances](metrics-collected-by-CloudWatch-agent.md#linux-metrics-enabled-by-CloudWatch-agent). Within the entry for each individual metric, you might optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `metrics_collection_interval` – Optional. Specifies how often to collect the netstat metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information about high-resolution metrics, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Additional dimensions to use for only the netstat metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent. + `processes` – Optional. Specifies that process metrics are to be collected. This section is valid only for Linux instances. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `measurement` – Specifies the array of processes metrics to be collected. Possible values are `blocked`, `dead`, `idle`, `paging`, `running`, `sleeping`, `stopped`, `total`, `total_threads`, `wait`, and `zombies`. This field is required if you include `processes`. For all `processes` metrics, the default unit is `None`. Within the entry for each individual metric, you might optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `metrics_collection_interval` – Optional. Specifies how often to collect the processes metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Additional dimensions to use for only the process metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent. + `nvidia_gpu` – Optional. Specifies that NVIDIA GPU metrics are to be collected. This section is valid only for Linux instances on hosts that are configured with a NVIDIA GPU accelerator and have the NVIDIA System Management Interface (nvidia-smi) installed. The NVIDIA GPU metrics that are collected are prefixed with the string `nvidia_smi_` to distinguish them from the metrics collected for other accelerator types. This section can include the following fields: + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. + `measurement` – Specifies the array of NVIDIA GPU metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the table in [Collect NVIDIA GPU metrics](CloudWatch-Agent-NVIDIA-GPU.md). Within the entry for each individual metric, you can optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit of `None` for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `metrics_collection_interval` – Optional. Specifies how often to collect the NVIDIA GPU metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. + `jmx` – Optional. Specifies that you want to retrieve Java Management Extensions (JMX) metrics from the instance. For more information about the parameters that you can use in this section and the metrics that you can collect, see [Collect Java Management Extensions (JMX) metrics](CloudWatch-Agent-JMX-metrics.md). + `otlp` – Optional. Specifies that you want to collect metrics from the OpenTelemetry SDK. For more information about the fields that you can use in this section, see [Collect metrics and traces with OpenTelemetry](CloudWatch-Agent-OpenTelemetry-metrics.md). + `procstat` – Optional. Specifies that you want to retrieve metrics from individual processes. For more information about the configuration options available for procstat, see [Collect process metrics with the procstat plugin](CloudWatch-Agent-procstat-process-metrics.md). + `statsd` – Optional. Specifies that you want to retrieve custom metrics using the `StatsD` protocol. The CloudWatch agent acts as a daemon for the protocol. You use any standard `StatsD` client to send the metrics to the CloudWatch agent. For more information about the configuration options available for StatsD, see [Retrieve custom metrics with StatsD](CloudWatch-Agent-custom-metrics-statsd.md). + `ethtool` – Optional. Specifies that you want to retrieve network metrics using the `ethtool` plugin. This plugin can import both the metrics collected by the standard ethtool utility, as well as network performance metrics from Amazon EC2 instances. For more information about the configuration options available for ethtool, see [Collect network performance metrics](CloudWatch-Agent-network-performance.md). The following is an example of a `metrics` section for a Linux server. In this example, three CPU metrics, three netstat metrics, three process metrics, and one disk metric are collected, and the agent is set up to receive additional metrics from a `collectd` client. ``` "metrics": { "aggregation_dimensions" : [["AutoScalingGroupName"], ["InstanceId", "InstanceType"],[]], "metrics_collected": { "collectd": {}, "cpu": { "resources": [ "*" ], "measurement": [ {"name": "cpu_usage_idle", "rename": "CPU_USAGE_IDLE", "unit": "Percent"}, {"name": "cpu_usage_nice", "unit": "Percent"}, "cpu_usage_guest" ], "totalcpu": false, "drop_original_metrics": [ "cpu_usage_guest" ], "metrics_collection_interval": 10, "append_dimensions": { "test": "test1", "date": "2017-10-01" } }, "netstat": { "measurement": [ "tcp_established", "tcp_syn_sent", "tcp_close" ], "metrics_collection_interval": 60 }, "disk": { "measurement": [ "used_percent" ], "resources": [ "*" ], "drop_device": true }, "processes": { "measurement": [ "running", "sleeping", "dead" ] } }, "append_dimensions": { "ImageId": "${aws:ImageId}", "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}", "AutoScalingGroupName": "${aws:AutoScalingGroupName}" } } ``` ### Windows Server In the `metrics_collected` section for Windows Server, you can have subsections for each Windows performance object, such as `Memory`, `Processor`, and `LogicalDisk`. For information about what objects and counters are available, see [Performance Counters](https://learn.microsoft.com/en-us/windows/win32/perfctrs/performance-counters-portal) in the Microsoft Windows documentation. Within the subsection for each object, you specify a `measurement` array of the counters to collect. The `measurement` array is required for each object that you specify in the configuration file. You can also specify a `resources` field to name the instances to collect metrics from. You can also specify `*` for `resources` to collect separate metrics for every instance. If you omit `resources` for counters that have instances, the data for all instances is aggregated into one set. If you omit `resources` for counters that don't have instances, the counters are not collected by the CloudWatch agent. To determine whether counters have instances, you can use one of the following commands. Powershell: ``` Get-Counter -ListSet * ``` Command line (not Powershell): ``` TypePerf.exe –q ``` Within each object section, you can also specify the following optional fields: + `metrics_collection_interval` – Optional. Specifies how often to collect the metrics for this object, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). + `append_dimensions` – Optional. Specifies additional dimensions to use for only the metrics for this object. If you specify this field, it's used in addition to dimensions specified in the global `append_dimensions` field that is used for all types of metrics collected by the agent. + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. Within each counter section, you can also specify the following optional fields: + `rename` – Specifies a different name to be used in CloudWatch for this metric. + `unit` – Specifies the unit to use for this metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). There are other optional sections that you can include in `metrics_collected`: + `statsd` – Enables you to retrieve custom metrics using the `StatsD` protocol. The CloudWatch agent acts as a daemon for the protocol. You use any standard `StatsD` client to send the metrics to the CloudWatch agent. For more information, see [Retrieve custom metrics with StatsD](CloudWatch-Agent-custom-metrics-statsd.md). + `procstat` – Enables you to retrieve metrics from individual processes. For more information, see [Collect process metrics with the procstat plugin](CloudWatch-Agent-procstat-process-metrics.md). + `jmx` – Optional. Specifies that you want to retrieve Java Management Extensions (JMX) metrics from the instance. For more information about the fields that you can use in this section and the metrics that you can collect, see [Collect Java Management Extensions (JMX) metrics](CloudWatch-Agent-JMX-metrics.md). + `otlp` – Optional. Specifies that you want to collect metrics from the OpenTelemetry SDK. For more information about the fields that you can use in this section, see [Collect metrics and traces with OpenTelemetry](CloudWatch-Agent-OpenTelemetry-metrics.md). The following is an example `metrics` section for use on Windows Server. In this example, many Windows metrics are collected, and the computer is also set to receive additional metrics from a `StatsD` client. ``` "metrics": { "metrics_collected": { "statsd": {}, "Processor": { "measurement": [ {"name": "% Idle Time", "rename": "CPU_IDLE", "unit": "Percent"}, "% Interrupt Time", "% User Time", "% Processor Time" ], "resources": [ "*" ], "append_dimensions": { "d1": "win_foo", "d2": "win_bar" } }, "LogicalDisk": { "measurement": [ {"name": "% Idle Time", "unit": "Percent"}, {"name": "% Disk Read Time", "rename": "DISK_READ"}, "% Disk Write Time" ], "resources": [ "*" ] }, "Memory": { "metrics_collection_interval": 5, "measurement": [ "Available Bytes", "Cache Faults/sec", "Page Faults/sec", "Pages/sec" ], "append_dimensions": { "d3": "win_bo" } }, "Network Interface": { "metrics_collection_interval": 5, "measurement": [ "Bytes Received/sec", "Bytes Sent/sec", "Packets Received/sec", "Packets Sent/sec" ], "resources": [ "*" ], "append_dimensions": { "d3": "win_bo" } }, "System": { "measurement": [ "Context Switches/sec", "System Calls/sec", "Processor Queue Length" ], "append_dimensions": { "d1": "win_foo", "d2": "win_bar" } } }, "append_dimensions": { "ImageId": "${aws:ImageId}", "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}", "AutoScalingGroupName": "${aws:AutoScalingGroupName}" }, "aggregation_dimensions" : [["ImageId"], ["InstanceId", "InstanceType"], ["d1"],[]] } } ``` ## CloudWatch agent configuration file: Logs section The `logs` section includes the following fields: + `service.name` – Optional. Specifies the service name to be used to populate the entity for [finding related telemetry](ExploreRelated.md). + `deployment.environment` – Optional. Specifies the environment name to be used to populate the entity for [finding related telemetry](ExploreRelated.md). + `backpressure_mode` – Optional. Specifies the behavior when the CloudWatch agent is ingesting logs faster than it can send to CloudWatch Logs, resulting in backpressure. Backpressure can happen from network issues, API throttling, or high log volume. The agent supports the following values: + `fd_release` – Releases file descriptors for deleted files during backpressure conditions. This option can help prevent disk space exhaustion when external log rotation or cleanup processes remove files while the agent maintains open file descriptors. The `auto_removal` option takes precedence over the `backpressure_mode` option being set to `fd_release`. When `auto_removal` is enabled, the CloudWatch agent processes the file to completion without releasing the file descriptor. **Important** Using `fd_release` can result in the CloudWatch agent being unable to read log files to completion, causing log loss. + `concurrency` – Optional. Specifies the number of shared log publishers used to concurrently publish log files to CloudWatch Logs. If you omit this field, each log file destination (log group, stream combination) has a single shared log publisher, which can lead to bottlenecks for large files or when writing multiple files to the same destination. Enabling concurrency can help with throughput. + `logs_collected` – Required if the `logs` section is included. Specifies which log files and Windows event logs are to be collected from the server. It can include two fields, `files` and `windows_events`. + `files` – Specifies which regular log files the CloudWatch agent is to collect. It contains one field, `collect_list`, which further defines these files. + `collect_list` – Required if `files` is included. Contains an array of entries, each of which specifies one log file to collect. Each of these entries can include the following fields: + `file_path` – Specifies the path of the log file to upload to CloudWatch Logs. Standard Unix glob matching rules are accepted, with the addition of `**` as a *super asterisk*. For example, specifying `/var/log/**.log` causes all `.log` files in the `/var/log` directory tree to be collected. For more examples, see [Glob Library](https://github.com/gobwas/glob). You can also use the standard asterisk as a standard wildcard. For example, `/var/log/system.log*` matches files such as `system.log_1111`, `system.log_2222`, and so on in `/var/log`. Only the latest file is pushed to CloudWatch Logs based on file modification time. We recommend that you use wildcards to specify a series of files of the same type, such as `access_log.2018-06-01-01` and `access_log.2018-06-01-02`, but not multiple kinds of files, such as `access_log_80` and `access_log_443`. To specify multiple kinds of files, add another log stream entry to the agent configuration file so that each kind of log file goes to a different log stream. + `auto_removal` – Optional. If this is `true`, the CloudWatch agent automatically deletes this log file after reading it and it has been rotated. Usually the log files are deleted after their entire contents are uploaded to CloudWatch Logs, but if the agent reaches the EOF (end of file) and also detects another newer log file that matches the same `file_path`, the agent deletes the OLD file, so you must make sure that you are done writing to the OLD file before creating the NEW file. The [RUST tracing library](https://docs.rs/tracing/latest/tracing/) has a known incompatibility because it will potentially create a NEW log file and then still attempt to write to the OLD log file. The agent only removes complete files from logs that create multiple files, such as logs that create separate files for each date. If a log continuously writes to a single file, it is not removed. If you already have a log file rotation or removal method in place, we recommend that you omit this field or set it to `false`. If you omit this field, the default value of `false` is used. + `log_group_name` – Optional. Specifies what to use as the log group name in CloudWatch Logs. We recommend that you use this field to specify a log group name to prevent confusion. If you omit `log_group_name`, the value of `file_path` up to the final dot is used as the log group name. For example, if the file path is `/tmp/TestLogFile.log.2017-07-11-14`, the log group name is `/tmp/TestLogFile.log`. If you specify a log group name, you can use `{instance_id}`, `{hostname}`, `{local_hostname}`, and `{ip_address}` as variables within the name. `{hostname}` retrieves the hostname from the EC2 metadata, and `{local_hostname}` uses the hostname from the network configuration file. If you use these variables to create many different log groups, keep in mind the limit of 1,000,000 log groups per Region per account. Allowed characters include a–z, A–Z, 0–9, '\$1' (underscore), '-' (hyphen), '/' (forward slash), and '.' (period). + `log_group_class` – Optional. Specifies which log group class to use for the new log group. For more information about log group classes, see [ Log classes](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch_Logs_Log_Classes.html). Valid values are `STANDARD` and `INFREQUENT_ACCESS`. If you omit this field, the default of `STANDARD` is used. **Important** After a log group is created, its class can't be changed. + `log_stream_name` – Optional. Specifies what to use as the log stream name in CloudWatch Logs. As part of the name, you can use `{instance_id}`, `{hostname}`, `{local_hostname}`, and `{ip_address}` as variables within the name. `{hostname}` retrieves the hostname from the EC2 metadata, and `{local_hostname}` uses the hostname from the network configuration file. If you omit this field, the value of the `log_stream_name` parameter in the global `logs` section is used. If that is also omitted, the default value of `{instance_id}` is used. If a log stream doesn't already exist, it's created automatically. + `retention_in_days` – Optional. Specifies the number of days to retain the log events in the specified log group. + If the agent is creating this log group now, and you omit this field, the retention of this new log group is set to never expire. + If this log group already exists and you specify this field, the new retention that you specify is used. If you omit this field for a log group that already exists, the log group's retention is not changed. The CloudWatch agent wizard uses `-1` as the default value for this field when it is used to create the agent configuration file and you don't specify a value for log retention. This `-1` value set by the wizard specifies that the events in the log group will never expire. However, manually editing this value to `-1` has no effect. Valid values are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, and 3653. If you configure the agent to write multiple log streams to the same log group, specifying the `retention_in_days` in one place will set the log retention for the entire log group. If you specify `retention_in_days` for the same log group in multiple places, the retention is set if all of those values are equal. However, if different `retention_in_days` values are specified for the same log group in multiple places, the log retention will not be set and the agent will stop, returning an error. **Note** The agent's IAM role or IAM user must have the `logs:PutRetentionPolicy` for it to be able to set retention policies. **Warning** If you set `retention_in_days` for a log group that already exists, all logs in that log group that were published before the number of days that you specify are deleted. For example, setting it to 3 would cause all logs from 3 days ago and before to be deleted. + `filters` – Optional. Can contain an array of entries, each of which specifies a regular expression and a filter type to specify whether to publish or drop log entries that match the filter. If you omit this field, all logs in the log file are published to CloudWatch Logs. If you include this field, the agent processes each log message with all of the filters that you specify, and only the log events that pass all of the filters are published to CloudWatch Logs. The log entries that don’t pass all of the filters will still remain in the host's log file, but will not be sent to CloudWatch Logs. Each entry in the filters array can include the following fields: + `type`– Denotes the type of filter. Valid values are `include` and `exclude`. With `include`, the log entry must match the expression to be published to CloudWatch Logs. With `exclude`, each log entry that matches the filter is not sent to CloudWatch Logs. + `expression`– A regular expression string that follows the [RE2 Syntax](https://github.com/google/re2/wiki/Syntax). **Note** The CloudWatch agent doesn't check the performance of any regular expression that you supply, or restrict the run time of the evaluation of the regular expressions. We recommend that you are careful not to write an expression that is expensive to evaluate. For more information about possible issues, see [Regular expression Denial of Service - ReDoS](https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS) For example, the following excerpt of the CloudWatch agent configuration file publishes logs that are PUT and POST requests to CloudWatch Logs, but excluding logs that come from Firefox. ``` "collect_list": [ { "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/test.log", "log_group_name": "test.log", "log_stream_name": "test.log", "filters": [ { "type": "exclude", "expression": "Firefox" }, { "type": "include", "expression": "P(UT|OST)" } ] }, ..... ] ``` **Note** The order of the filters in the configuration file matters for performance. In the preceding example, the agent drops all the logs that match `Firefox` before it starts evaluating the second filter. To cause fewer log entries to be evaluated by more than one filter, put the filter that you expect to rule out more logs first in the configuration file. + `timezone` – Optional. Specifies the time zone to use when putting timestamps on log events. The valid values are `UTC` and `Local`. The default value is `Local`. This parameter is ignored if you don't specify a value for `timestamp_format`. + `timestamp_format` – Optional. Specifies the timestamp format, using plaintext and special symbols that start with %. If you omit this field, the current time is used. If you use this field, you can use the symbols in the following list as part of the format. **Note** This parameter is not considered when the `file_path` is set to `amazon-cloudwatch-agent.log` If a single log entry contains two time stamps that match the format, the first time stamp is used. This list of symbols is different than the list used by the older CloudWatch Logs agent. For a summary of these differences, see [Timestamp differences between the CloudWatch agent and the earlier CloudWatch Logs agent](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-logs-timestamp-differences) `%y` Year without century as a zero-padded decimal number. For example, `19` to represent 2019. `%Y` Year with century as a decimal number. For example, `2019`. `%b` Month as the locale's abbreviated name `%B` Month as the locale's full name `%m` Month as a zero-padded decimal number `%-m` Month as a decimal number (not zero-padded) `%d` Day of the month as a zero-padded decimal number `%-d` Day of the month as a decimal number (not zero-padded) `%A` Full name of weekday, such as `Monday` `%a` Abbreviation of weekday, such as `Mon` `%H` Hour (in a 24-hour clock) as a zero-padded decimal number `%I` Hour (in a 12-hour clock) as a zero-padded decimal number `%-I` Hour (in a 12-hour clock) as a decimal number (not zero-padded) `%p` AM or PM `%M` Minutes as a zero-padded decimal number `%-M` Minutes as a decimal number (not zero-padded) `%S` Seconds as a zero-padded decimal number `%-S` Seconds as a decimal number (not zero padded) `%f` Fractional seconds as a decimal number (1-9 digits), zero-padded on the left. `%Z` Time zone, for example `PST` `%z` Time zone, expressed as the offset between the local time zone and UTC. For example, `-0700`. Only this format is supported. For example, `-07:00` isn't a valid format. + `multi_line_start_pattern` – Specifies the pattern for identifying the start of a log message. A log message is made of a line that matches the pattern and any subsequent lines that don't match the pattern. If you omit this field, multi-line mode is disabled, and any line that begins with a non-whitespace character closes the previous log message and starts a new log message. If you include this field, you can specify `{timestamp_format}` to use the same regular expression as your timestamp format. Otherwise, you can specify a different regular expression for CloudWatch Logs to use to determine the start lines of multi-line entries. + `encoding` – Specified the encoding of the log file so that it can be read correctly. If you specify an incorrect coding, there might be data loss because characters that can't be decoded are replaced with other characters. The default value is `utf-8`. The following are all possible values: `ascii, big5, euc-jp, euc-kr, gbk, gb18030, ibm866, iso2022-jp, iso8859-2, iso8859-3, iso8859-4, iso8859-5, iso8859-6, iso8859-7, iso8859-8, iso8859-8-i, iso8859-10, iso8859-13, iso8859-14, iso8859-15, iso8859-16, koi8-r, koi8-u, macintosh, shift_jis, utf-8, utf-16, utf-16le, UTF-16, UTF-16LE, windows-874, windows-1250, windows-1251, windows-1252, windows-1253, windows-1254, windows-1255, windows-1256, windows-1257, windows-1258, x-mac-cyrillic` + `service.name` – Optional. Specifies the service name to be used to populate the entity for [finding related telemetry](ExploreRelated.md). + `deployment.environment` – Optional. Specifies the environment name to be used to populate the entity for [finding related telemetry](ExploreRelated.md). + `trim_timestamp` – Optional. If this is true, the CloudWatch agent will remove the timestamp matched by `timestamp_format` from the line before sending it to CloudWatch Logs. The LogEvent will still contain the `timestamp` field. If you omit this field, the default value of `false` is used. + The `windows_events` section specifies the type of Windows events to collect from servers running Windows Server. It includes the following fields: + `collect_list` – Required if `windows_events` is included. Specifies the types and levels of Windows events to be collected. Each log to be collected has an entry in this section, which can include the following fields: + `event_name` – Specifies the type of Windows events to log. This is equivalent to the Windows event log channel name: for example, `System`, `Security`, `Application`, and so on. This field is required for each type of Windows event to log. **Note** When CloudWatch retrieves messages from a Windows log channel, it looks up the log channel based on its `Full Name` property. Meanwhile, the Windows Event Viewer navigation pane displays the `Log Name` property of log channels. The `Full Name` and `Log Name` do not always match. To confirm the `Full Name` of a channel, right-click on it in the Windows Event viewer and open **Properties**. + `event_levels` – Optional. Specifies the levels of event to log. You must specify each level to log. Possible values include `INFORMATION`, `WARNING`, `ERROR`, `CRITICAL`, and `VERBOSE`. This field is optional for each type of Windows event to log. and can be used with other filtering option like `event_ids` and `filters`. + `event_ids` – Optional. Contains an array of Windows Event IDs to specify which events to collect from the Windows Event Log. When this field is excluded, all events from the specified event log are collected. When this field is included, the agent only collects events that match the specified Event IDs. Each entry in the `event_ids` array should be a numeric Event ID value and can be used with other filtering options. See the third entry in the config sample below. **Note** Using `event_ids` for filtering is recommended over regex expressions when you need to filter by Event ID, as it provides better performance. + `filters` – Optional. Contains an array of entries. Each entry specifies a regular expression and a filter type to specify whether to publish or drop the log entries that match the filter. When the field is included, the agent processes each log message with all of the filters that you specify, and only the log events that pass all filters are published to CloudWatch Logs. The windows event logs that doesn’t pass all of the filters will be dropped and not sent to CloudWatch Logs. The filters section can also be used with other filtering mechanisms like event ids [4624, 4625] and system levels (Information, Error, or Critical) to effectively filter logs and push to CloudWatch. Each entry in the filters array can include the following fields: + `type` – Specifies the type of filter. Valid values are `include` and `exclude`. With include, the windows events entry must match the expression to be published to CloudWatch Logs. With exclude, each windows event log entry that matches the filter is not sent to CloudWatch Logs. + `expression` – A regular expression string that follows the RE2 Syntax. **Note** The CloudWatch agent does not validate regular expressions that you provide. It also does not limit their evaluation time. Write your expressions carefully to avoid performance issues. For more information on security ricks, see [Regular expression Denial of Service - ReDoS ](https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS). In the example agent configuration below: For the first entry, the agent pushes logs that contain database failure messages, any authentication related activities, and all login events (both successful and failed attempts) to CloudWatch. Any log that doesn’t match this pattern is dropped. In the second entry, initial filtering is done based on event ids for windows event subscription. The agent collects all logs that contain the string user, discarding logs that don't match these patterns. The agent then drops logs containing `successful` before sending the remaining logs to CloudWatch Logs. Every filter type is applied to each windows event log before sending to CloudWatch. ``` "collect_list": [ { "event_name": "Application", "log_group_name": "ApplicationEvents", "log_stream_name": "ApplicationEvents", "filters": [ { "type": "include", "expression": "Database.*failed|Authentication.*|login.*" } ] }, { "event_name": "System", "log_group_name": "SystemEvents", "log_stream_name": "Logon-events", "event_ids": [ 4624, 4625 ], "filters": [ { "type": "include", "expression": ".*user.*" }, { "type": "exclude", "expression": ".*successful.*" } ] } ..... ] ``` **Note** The order of the filters in the configuration will impact the performance. In the second entry, the agent drops all the logs that does not match the user before it starts evaluating the second filter expression. For optimal performance, order filters from highest to lowest exclusion rate. While you can filter out logs on event ids and system levels in the filter expression, it is recommended to use the `event_ids` and `log_level` as shown in the second entry for improved performance. **Warning** Even though all filtering mechanisms (event\$1levels, event\$1ids, filters) are optional, at least one is required during agent configuration to filter logs. + `log_group_name` – Required. Specifies what to use as the log group name in CloudWatch Logs. + `log_stream_name` – Optional. Specifies what to use as the log stream name in CloudWatch Logs. As part of the name, you can use `{instance_id}`, `{hostname}`, `{local_hostname}`, and `{ip_address}` as variables within the name. `{hostname}` retrieves the hostname from the EC2 metadata, and `{local_hostname}` uses the hostname from the network configuration file. If you omit this field, the value of the `log_stream_name` parameter in the global `logs` section is used. If that is also omitted, the default value of `{instance_id}` is used. If a log stream doesn't already exist, it's created automatically. + `event_format` – Optional. Specifies the format to use when storing Windows events in CloudWatch Logs. `xml` uses the XML format as in Windows Event Viewer. `text` uses the legacy CloudWatch Logs agent format. + `retention_in_days` – Optional. Specifies the number of days to retain the Windows events in the specified log group. + If the agent is creating this log group now, and you omit this field, the retention of this new log group is set to never expire. + If this log group already exists and you specify this field, the new retention that you specify is used. If you omit this field for a log group that already exists, the log group's retention is not changed. The CloudWatch agent wizard uses `-1` as the default value for this field when it is used to create the agent configuration file and you don't specify a value for log retention. This `-1` value specifies set by the wizard specifies that the events in the log group don't expire. However, manually editing this value to `-1` has no effect. Valid values are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, and 3653. If you configure the agent to write multiple log streams to the same log group, specifying the `retention_in_days` in one place will set the log retention for the entire log group. If you specify `retention_in_days` for the same log group in multiple places, the retention is set if all of those values are equal. However, if different `retention_in_days` values are specified for the same log group in multiple places, the log retention will not be set and the agent will stop, returning an error. **Note** The agent's IAM role or IAM user must have the `logs:PutRetentionPolicy` for it to be able to set retention policies. **Warning** If you set `retention_in_days` for a log group that already exists, all logs in that log group that were published before the number of days that you specify are deleted. For example, setting it to 3 would cause all logs from 3 days ago and before to be deleted. + `log_stream_name` – Optional. Specifies the default log stream name to be used for any logs or Windows events that don't have individual log stream names defined in the `log_stream_name` parameter within their entry in `collect_list`. + `endpoint_override` – Specifies a FIPS endpoint or private link to use as the endpoint where the agent sends logs. Specifying this field and setting a private link enables you to send the logs to an Amazon VPC endpoint. For more information, see [What Is Amazon VPC?](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html). The value of `endpoint_override` must be a string that is a URL. For example, the following part of the logs section of the configuration file sets the agent to use a VPC Endpoint when sending logs. ``` { "logs": { "endpoint_override": "vpce-XXXXXXXXXXXXXXXXXXXXXXXXX.logs.us-east-1.vpce.amazonaws.com", ...... }, } ``` + `force_flush_interval` – Specifies in seconds the maximum amount of time that logs remain in the memory buffer before being sent to the server. No matter the setting for this field, if the size of the logs in the buffer reaches 1 MB, the logs are immediately sent to the server. The default value is 5. If you are using the agent to report high-resolution metrics in embedded metric format, and you are setting alarms on those metrics, keep this parameter set to the default value of 5. Otherwise, the metrics are reported with a delay that can cause alarming on partial or incomplete data. + `credentials` – Specifies an IAM role to use when sending logs to a different AWS account. If specified, this field contains one parameter, `role_arn`. + `role_arn` – Specifies the ARN of an IAM role to use for authentication when sending logs to a different AWS account. For more information, see [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account). If specified here, this overrides the `role_arn` specified in the `agent` section of the configuration file, if any. + `metrics_collected` – This field can contain sections to specify that the agent is to collect logs to enable use cases such as CloudWatch Application Signals and Container Insights with enhanced observability for Amazon EKS. + `application_signals` (Optional) Specifies that you want to enable [CloudWatch Application Signals](CloudWatch-Application-Monitoring-Sections.md) For more information about this configuration, see [Enable CloudWatch Application Signals](CloudWatch-Agent-Application_Signals.md). + `kubernetes` – This field can contain an `enhanced_container_insights` parameter, which you can use to enable Container Insights with enhanced observability for Amazon EKS. + `enhanced_container_insights` – Set this to `true` to enable Container Insights with enhanced observability for Amazon EKS. For more information, see [Container Insights with enhanced observability for Amazon EKS](container-insights-detailed-metrics.md). + `accelerated_compute_metrics` – Set this to `false` to opt out of collecting Nvidia GPU metrics on Amazon EKS clusters. For more information, see [NVIDIA GPU metrics](Container-Insights-metrics-enhanced-EKS.md#Container-Insights-metrics-EKS-GPU). + `emf` – To collect metrics embedded in logs, it is no longer necessary to add this `emf` field. This is a legacy field that specified that the agent is to collect logs that are in embedded metric format. You can generate metric data from these logs. For more information, see [Embedding metrics within logs](CloudWatch_Embedded_Metric_Format.md). + `otlp` – Optional. Specifies that you want to collect metrics from the OpenTelemetry SDK. For more information about the fields that you can use in this section, see [Collect metrics and traces with OpenTelemetry](CloudWatch-Agent-OpenTelemetry-metrics.md). The following is an example of a `logs` section. ``` "logs":{ "logs_collected": { "files": { "collect_list": [ { "file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log", "log_group_name": "amazon-cloudwatch-agent.log", "log_stream_name": "my_log_stream_name_1" }, { "file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\test.log", "log_group_name": "test.log", "log_stream_name": "my_log_stream_name_2" } ] }, "windows_events": { "collect_list": [ { "event_name": "System", "event_ids": [ 1001, 1008 ], "log_group_name": "System", "log_stream_name": "System" }, { "event_name": "CustomizedName", "event_levels": [ "INFORMATION", "ERROR" ], "log_group_name": "CustomizedLogGroup", "log_stream_name": "CustomizedLogStream" }, { "event_name": "Application", "event_levels": [ "INFORMATION", "ERROR" ], "event_ids":[ 7369, 5624 ], "log_group_name": "CustomizedLogGroup", "log_stream_name": "CustomizedLogStream" } ] } }, "log_stream_name": "my_log_stream_name", "metrics_collected": { "kubernetes": { "enhanced_container_insights": true } } } ``` ## CloudWatch agent configuration file: Traces section By adding a `traces` section to the CloudWatch agent configuration file, you can enable CloudWatch Application Signals or collect traces from X-Ray and from the OpenTelemetry instrumentation SDK and send them to X-Ray. **Important** The agent's IAM role or IAM user must have the **AWSXrayWriteOnlyAccess** policy to send trace data to X-Ray. For a quick start for collecting traces, you can add just the following to the CloudWatch agent configuration file. ``` "traces_collected": { "xray": { }, "otlp": { } } ``` If you add the previous section to the CloudWatch agent configuration file and restart the agent, this causes the agent to start collecting traces using the following default options and values. For more information about these parameters, see the parameter definitions later in this section. ``` "traces_collected": { "xray": { "bind_address": "127.0.0.1:2000", "tcp_proxy": { "bind_address": "127.0.0.1:2000" } }, "otlp": { "grpc_endpoint": "127.0.0.1:4317", "http_endpoint": "127.0.0.1:4318" } } ``` The `traces` section can include the following fields: + `traces_collected` – Required if the `traces` section is included. Specifies which SDKs to collect traces from. It can include the following fields: + `application_signals` – Optional. Specifies that you want to enable [CloudWatch Application Signals](CloudWatch-Application-Monitoring-Sections.md) For more information about this configuration, see [Enable CloudWatch Application Signals](CloudWatch-Agent-Application_Signals.md). + `xray` – Optional. Specifies that you want to collect traces from the X-Ray SDK. This section can include the following fields: + `bind_address` – Optional. Specifies the UDP address for the CloudWatch agent to use to listen for X-Ray traces. The format is `ip:port`. This address must match the address set in the X-Ray SDK. If you omit this field, the default of `127.0.0.1:2000` is used. + `tcp_proxy` – Optional. Configures the address for a proxy used to support X-Ray remote sampling. For more information, see [ Configuring sampling rules](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html) in the X-Ray documentation. This section can contain the following field. + `bind_address` – Optional. Specifies the TCP address to which the CloudWatch agent should set up the proxy. The format is `ip:port`. This address must match the address set in the X-Ray SDK. If you omit this field, the default of `127.0.0.1:2000` is used. + `otlp` – Optional. Specifies that you want to collect traces from the OpenTelemetry SDK. For more information about the fields that you can use in this section, see [Collect metrics and traces with OpenTelemetry](CloudWatch-Agent-OpenTelemetry-metrics.md)). For more information about the AWS Distro for OpenTelemetry, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/). For more information about the AWS Distro for OpenTelemetry SDKs, see [Introduction](https://aws-otel.github.io/docs/introduction). This section can include the following fields: + `grpc_endpoint` – Optional. Specifies the address for the CloudWatch agent to use to listen for OpenTelemetry traces sent using gRPC Remote Procedure Calls. The format is `ip:port`. This address must match the address set for the gRPC exporter in the OpenTelemetry SDK. If you omit this field, the default of `127.0.0.1:4317` is used. + `http_endpoint` – Optional. Specifies the address for the CloudWatch agent to use to listen for OTLP traces sent over HTTP. The format is `ip:port`. This address must match the address set for the HTTP exporter in the OpenTelemetry SDK. If you omit this field, the default of `127.0.0.1:4318` is used. + `concurrency` – Optional. Specifies the maximum number of concurrent calls to X-Ray that can be used to upload traces. The default value is `8` + `local_mode` – Optional. If `true`, the agent doesn't collect Amazon EC2 instance metadata. The default is `false` + `endpoint_override` – Optional. Specifies a FIPS endpoint or private link to use as the endpoint where the CloudWatch agent sends traces. Specifying this field and setting a private link enables you to send the traces to an Amazon VPC endpoint. For more information, see [ What is Amazon VPC](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) The value of `endpoint_override` must be a string that is a URL. + `region_override` – Optional. Specifies the Region to use for the X-Ray endpoint. The CloudWatch agent sends the traces to X-Ray in the specified Region. If you omit this field, the agent sends the traces to the Region where the Amazon EC2 instance is located. If you specify a Region here, it takes precedence over the setting of the `region` parameter in the `agent` section of the configuration file. + `proxy_override` – Optional. Specifies the proxy server address for the CloudWatch agent to use when sending requests to X-Ray. The proxy server's protocol must be specified as part of this address. + `credentials` – Specifies an IAM role to use when sending traces to a different AWS account. If specified, this field contains one parameter, `role_arn`. + `role_arn` – Specifies the ARN of an IAM role to use for authentication when sending traces to a different AWS account. For more information, see [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account). If specified here, this overrides the `role_arn` specified in the `agent` section of the configuration file, if any. + `transit_spans_in_otlp_format` – Optional. If `true`, sends traces to X-Ray in the OpenTelemetry Protocol format, which supports span events in Transaction Search. For more information, see [Adding custom attributes](CloudWatch-Transaction-Search-add-custom-attributes.md). The default is `false`. ## CloudWatch agent configuration file: Complete examples The following is an example of a complete CloudWatch agent configuration file for a Linux server. The items listed in the `measurement` sections for the metrics you want to collect can either specify the complete metric name such or just the part of the metric name that will be appended to the type of resource. For example, specifying either `reads` or `diskio_reads` in the `measurement` section of the `diskio` section will cause the `diskio_reads` metric to be collected. This example includes both ways of specifying metrics in the `measurement` section. ``` { "agent": { "metrics_collection_interval": 10, "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log" }, "metrics": { "namespace": "MyCustomNamespace", "metrics_collected": { "cpu": { "resources": [ "*" ], "measurement": [ {"name": "cpu_usage_idle", "rename": "CPU_USAGE_IDLE", "unit": "Percent"}, {"name": "cpu_usage_nice", "unit": "Percent"}, "cpu_usage_guest" ], "totalcpu": false, "metrics_collection_interval": 10, "append_dimensions": { "customized_dimension_key_1": "customized_dimension_value_1", "customized_dimension_key_2": "customized_dimension_value_2" } }, "disk": { "resources": [ "/", "/tmp" ], "measurement": [ {"name": "free", "rename": "DISK_FREE", "unit": "Gigabytes"}, "total", "used" ], "ignore_file_system_types": [ "sysfs", "devtmpfs" ], "metrics_collection_interval": 60, "append_dimensions": { "customized_dimension_key_3": "customized_dimension_value_3", "customized_dimension_key_4": "customized_dimension_value_4" } }, "diskio": { "resources": [ "*" ], "measurement": [ "reads", "writes", "read_time", "write_time", "io_time" ], "metrics_collection_interval": 60 }, "swap": { "measurement": [ "swap_used", "swap_free", "swap_used_percent" ] }, "mem": { "measurement": [ "mem_used", "mem_cached", "mem_total" ], "metrics_collection_interval": 1 }, "net": { "resources": [ "eth0" ], "measurement": [ "bytes_sent", "bytes_recv", "drop_in", "drop_out" ] }, "netstat": { "measurement": [ "tcp_established", "tcp_syn_sent", "tcp_close" ], "metrics_collection_interval": 60 }, "processes": { "measurement": [ "running", "sleeping", "dead" ] } }, "append_dimensions": { "ImageId": "${aws:ImageId}", "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}", "AutoScalingGroupName": "${aws:AutoScalingGroupName}" }, "aggregation_dimensions" : [["ImageId"], ["InstanceId", "InstanceType"], ["d1"],[]], "force_flush_interval" : 30 }, "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log", "log_group_name": "amazon-cloudwatch-agent.log", "log_stream_name": "amazon-cloudwatch-agent.log", "timezone": "UTC" }, { "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/test.log", "log_group_name": "test.log", "log_stream_name": "test.log", "timezone": "Local" } ] } }, "log_stream_name": "my_log_stream_name", "force_flush_interval" : 15, "metrics_collected": { "kubernetes": { "enhanced_container_insights": true } } } } ``` The following is an example of a complete CloudWatch agent configuration file for a server running Windows Server. ``` { "agent": { "metrics_collection_interval": 60, "logfile": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log" }, "metrics": { "namespace": "MyCustomNamespace", "metrics_collected": { "Processor": { "measurement": [ {"name": "% Idle Time", "rename": "CPU_IDLE", "unit": "Percent"}, "% Interrupt Time", "% User Time", "% Processor Time" ], "resources": [ "*" ], "append_dimensions": { "customized_dimension_key_1": "customized_dimension_value_1", "customized_dimension_key_2": "customized_dimension_value_2" } }, "LogicalDisk": { "measurement": [ {"name": "% Idle Time", "unit": "Percent"}, {"name": "% Disk Read Time", "rename": "DISK_READ"}, "% Disk Write Time" ], "resources": [ "*" ] }, "customizedObjectName": { "metrics_collection_interval": 60, "customizedCounterName": [ "metric1", "metric2" ], "resources": [ "customizedInstances" ] }, "Memory": { "metrics_collection_interval": 5, "measurement": [ "Available Bytes", "Cache Faults/sec", "Page Faults/sec", "Pages/sec" ] }, "Network Interface": { "metrics_collection_interval": 5, "measurement": [ "Bytes Received/sec", "Bytes Sent/sec", "Packets Received/sec", "Packets Sent/sec" ], "resources": [ "*" ], "append_dimensions": { "customized_dimension_key_3": "customized_dimension_value_3" } }, "System": { "measurement": [ "Context Switches/sec", "System Calls/sec", "Processor Queue Length" ] } }, "append_dimensions": { "ImageId": "${aws:ImageId}", "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}", "AutoScalingGroupName": "${aws:AutoScalingGroupName}" }, "aggregation_dimensions" : [["ImageId"], ["InstanceId", "InstanceType"], ["d1"],[]] }, "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log", "log_group_name": "amazon-cloudwatch-agent.log", "timezone": "UTC" }, { "file_path": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\test.log", "log_group_name": "test.log", "timezone": "Local" } ] }, "windows_events": { "collect_list": [ { "event_name": "System", "event_levels": [ "INFORMATION", "ERROR" ], "log_group_name": "System", "log_stream_name": "System", "event_format": "xml" }, { "event_name": "CustomizedName", "event_levels": [ "WARNING", "ERROR" ], "log_group_name": "CustomizedLogGroup", "log_stream_name": "CustomizedLogStream", "event_format": "xml" } ] } }, "log_stream_name": "example_log_stream_name" } } ``` ## Save the CloudWatch agent configuration file manually If you create or edit the CloudWatch agent configuration file manually, you can give it any name. After you have created the file, you can copy it to other servers where you want to run the agent. ## Uploading the CloudWatch agent configuration file to Systems Manager Parameter Store If you plan to use the SSM Agent to install the CloudWatch agent on servers, after you manually edit the CloudWatch agent configuration file, you can upload it to Systems Manager Parameter Store. To do so, use the Systems Manager `put-parameter` command. To be able to store the file in Parameter Store, you must use an IAM role with sufficient permissions. Use the following command, where *parameter name* is the name to be used for this file in Parameter Store and *configuration\$1file\$1pathname* is the path and file name of the configuration file that you edited. ``` aws ssm put-parameter --name "parameter name" --type "String" --value file://configuration_file_pathname ``` # Enable CloudWatch Application Signals Use CloudWatch Application Signals to automatically instrument your applications on AWS so that you can track application performance against your business objectives. Application Signals provides you a unified, application-centric view of your Java applications, their dependencies, and their edges. For more information, see [Application Signals](CloudWatch-Application-Monitoring-Sections.md). CloudWatch Application Signals leverages the CloudWatch agent to receive metrics and traces from your auto-instrumented applications, optionally apply rules to reduce high cardinality, and then publish the processed telemetry to CloudWatch. You can provide custom configuration to the CloudWatch agent specifically for Application Signals using the agent configuration file. To start with, the presence of an `application_signals` section under the `metrics_collected` section within the `logs` section of the agent configuration file specifies that the CloudWatch agent will receive metrics from your auto-instrumented applications. Similarly, the presence of an `application_signals` section under the `traces_collected` section within the `traces` section of the agent configuration file specifies that the CloudWatch agent is enabled to receive traces from your auto-instrumented applications. In addition, you can optionally pass in custom configuration rules to reduce publishing high-cardinality telemetry as outlined in this section. + For Amazon EKS clusters, when you install the [Amazon CloudWatch Observability](install-CloudWatch-Observability-EKS-addon.md) EKS add-on, the CloudWatch agent is by default enabled to receive both metrics and traces from your auto-instrumented applications. If you would like to optionally pass in custom configuration rules, you can do so by passing in a custom agent configuration to the Amazon EKS add-on when you create or update it by using additional configuration, as outlined in [(Optional) Additional configuration](install-CloudWatch-Observability-EKS-addon.md#install-CloudWatch-Observability-EKS-addon-configuration). + For RedHat for OpenShift on AWS (ROSA), when you install the CloudWatch agent operator using helm charts, the CloudWatch agent is by default enabled to receive both metrics and traces from your auto-instrumented applications. If you would like to optionally pass in custom configuration rules, you can do so by passing in a custom agent configuration by using the Helm chart, as outlined in [(Optional) Additional configuration](install-CloudWatch-Observability-EKS-addon.md#install-CloudWatch-Observability-EKS-addon-configuration). + For other supported platforms including Amazon EC2, you must start the CloudWatch agent with an agent configuration that enables Application Signals by specifying the `application_signals` sections and optionally any custom configuration rules as outlined later in this section. The following is an overview of the fields in the CloudWatch agent configuration file that are related to CloudWatch Application Signals. + `logs` + `metrics_collected` – This field can contain sections to specify that the agent is to collect logs to enable use cases such as CloudWatch Application Signals and Container Insights with enhanced observability for Amazon EKS. **Note** Previously this section was also used to specify that the agent is to collect logs that are in embedded metric format. Those settings are no longer needed. + `application_signals` (Optional) Specifies that you want to enable CloudWatch Application Signals to receive metrics from your auto-instrumented applications to facilitate CloudWatch Application Signals. + `rules` (Optional) An array of rules to conditionally select metrics and traces and apply actions to handle high-cardinality scenarios. Each rule can contain the following fields: + `rule_name` (Optional) The name of the rule. + `selectors` (Optional) An array of metrics and traces dimension matchers. Each selector must provide the following fields: + `dimension` Required if `selectors` is not empty. This specifies the dimension of metrics and traces to use as a filter. + `match` Required if `selectors` is not empty. A wildcard pattern used for matching values of the specified dimension. + `action` (Optional) The action to be applied to metrics and traces that match the specified selectors. The value of `action` must be one of the following keywords: + `keep` Specifies to send only the metrics and traces to CloudWatch if matched by the `selectors`. + `drop` Specifies to drop the metric and traces that match the `selectors`. + `replace` Specifies to replace the dimensions of the metrics and traces that match `selectors`. They are replaced according to the `replacements` section. + `replacements` Required if `action` is `replace`. An array of dimension and value pairs that will be applied to metrics and traces that match the specified `selectors` when the `action` is `replace`. Each replacement must provide the following fields: + `target_dimension` Required if `replacements` is not empty. Specifies the dimension that needs to be replace. + `value` Required if `replacements` is not empty. The value to replace the original value of `target_dimension` with. + `limiter` (Optional) Use this section to limit how many metrics and dimensions Application Signals sends to CloudWatch, to optimize your costs. + `disabled` (Optional) If `true`, the metric limiting feature is disabled. The default is `false` + `drop_threshold` (Optional) The maximum number of distinct metrics per service in one rotation interval that can be exported by one CloudWatch agent. The default is 500. + `rotation_interval` (Optional) The interval at which the limiter resets the metric records for distinction counting. This is expressed as a string with a sequence of numbers and a unit suffix. Fractions are supported. The supported unit suffixes are `s`, `m`, `h`, `ms`, `us`, and `ns`. The default is `1h` for one hour. + `log_dropped_metrics` (Optional) Specifies whether the agent should write logs to the CloudWatch agent logs when Application Signals metrics are dropped. The default is `false`. **Note** To activate this logging, the `debug` parameter in the `agent` section must also be set to `true`. + `traces` + `traces_collected` + `application_signals` Optional. Specify this to enable the CloudWatch agent to receive traces from your auto-instrumented applications for facilitating CloudWatch Application Signals. **Note** Even though the custom `application_signals` rules are specified under the `metrics_collected` section that is contained in the `logs` section, they also implicitly apply to the `traces_collected` section as well. The same set of rules will apply to both metrics and traces. When there are multiple rules with different actions, they apply in the following sequence: `keep`, then `drop`, then `replace`. The following is an example of a full CloudWatch agent configuration file that applies custom rules. ``` { "logs": { "metrics_collected": { "application_signals": { "rules": [ { "rule_name": "keep01", "selectors": [ { "dimension": "Service", "match": "pet-clinic-frontend" }, { "dimension": "RemoteService", "match": "customers-service" } ], "action": "keep" }, { "rule_name": "drop01", "selectors": [ { "dimension": "Operation", "match": "GET /api/customer/owners/*" } ], "action": "drop" }, { "rule_name": "replace01", "selectors": [ { "dimension": "Operation", "match": "PUT /api/customer/owners/*/pets/*" }, { "dimension": "RemoteOperation", "match": "PUT /owners" } ], "replacements": [ { "target_dimension": "Operation", "value": "PUT /api/customer/owners/{ownerId}/pets{petId}" } ], "action": "replace" } ] } } }, "traces": { "traces_collected": { "application_signals": {} } } } ``` For the previous example configuration file, the `rules` are processed as follows: 1. Rule `keep01` ensures that any metrics and traces with the dimension `Service` as `pet-clinic-frontend` and the dimension `RemoteService` as `customers-service` are kept. 1. For the processed metrics and traces after applying `keep01`, the `drop01` rule ensures that metrics and traces with the dimension `Operation` as `GET /api/customer/owners/*` are dropped. 1. For the processed metrics and traces after applying `drop01`, the `replace01` rule updates metrics and traces that have the dimension `Operation` as `PUT /api/customer/owners/*/pets/*` and the dimension `RemoteOperation` as `PUT /owners` such that their `Operation` dimension is now replaced to be `PUT /api/customer/owners/{ownerId}/pets{petId}`. The following is a complete example of a CloudWatch configuration file that manages cardinality in Application Signals by changing the metric limit to 100, enabling the logging of dropped metrics, and setting the rotation interval to two hours. ``` { "logs": { "metrics_collected": { "application_signals": { "limiter": { "disabled": false, "drop_threshold": 100, "rotation_interval": "2h", "log_dropped_metrics": true } } }, "traces": { "traces_collected": { "application_signals": {} } } } } ``` # Collect network performance metrics EC2 instances running on Linux that use the Elastic Network Adapter (ENA) publish network performance metrics. Version 1.246396.0 and later of the CloudWatch agent enable you to import these network performance metrics into CloudWatch. When you import these network performance metrics into CloudWatch, they are charged as CloudWatch custom metrics. For more information about the ENA driver, see [ Enabling enhanced networking with the Elastic Network Adapter (ENA) on Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html) and [ Enabling enhanced networking with the Elastic Network Adapter (ENA) on Windows instances](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/enhanced-networking-ena.html). How you set up the collection of network performance metrics differs on Linux servers and Windows servers. The following table lists these network performance metrics enabled by the ENA adapter. When the CloudWatch agent imports these metrics into CloudWatch from Linux instances, it prepends `ethtool_` at the beginning of each of these metric names. | Metric | Description | | --- | --- | | Name on Linux servers: `bw_in_allowance_exceeded` Name on Windows servers: `Aggregate inbound BW allowance exceeded` | The number of packets queued and/or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None | | Name on Linux servers: `bw_out_allowance_exceeded` Name on Windows servers: `Aggregate outbound BW allowance exceeded` | The number of packets queued and/or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None | | Name on Linux servers:`conntrack_allowance_available` Name on Windows servers: `Available connection tracking allowance` | Reports the number of tracked connections that can be established by the instance before hitting the Connections Tracked allowance of that instance type. This metric is available only on Nitro-based EC2 instances using the Linux driver for Elastic Network Adapter (ENA) starting from version 2.8.1, and on computers using the Windows driver for Elastic Network Adapter (ENA) starting from version 2.6.0. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None | | Name on Linux servers:`ena_srd_mode` Name on Windows servers: `ena srd mode` | Describes which ENA Express features are enabled. For more information about ENA Express, see [ Improve network performance with ENA Express on Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-express.html) Values are as follows: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-network-performance.html) | | Name on Linux servers:`ena_srd_eligible_tx_pkts` Name on Windows servers: `ena srd eligible tx pkts` | The number of network packets sent within a given time period that meet AWS Scalable Reliable Datagram (SRD) requirements for eligibility, as follows: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-network-performance.html) | | Name on Linux servers:`ena_srd_tx_pkts` Name on Windows servers: `ena srd tx pkts` | The number of SRD packets transmitted within a given time period. | | Name on Linux servers:`ena_srd_rx_pkts` Name on Windows servers: `ena srd rx pkts` | The number of SRD packets received within a given time period. | | Name on Linux servers:`ena_srd_resource_utilization` Name on Windows servers: `ena srd resource utilization` | The percentage of the maximum allowed memory utilization for concurrent SRD connections that the instance has consumed. | | Name on Linux servers: `linklocal_allowance_exceeded` Name on Windows servers: `Link local packet rate allowance exceeded` | The number of packets dropped because the PPS of the traffic to local proxy services exceeded the maximum for the network interface. This impacts traffic to the DNS service, the Instance Metadata Service, and the Amazon Time Sync Service, but does not impact traffic to custom DNS resolvers. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None | | Name on Linux servers: `pps_allowance_exceeded` Name on Windows servers: `PPS allowance exceeded` | The number of packets queued and/or dropped because the bidirectional PPS exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](#CloudWatch-Agent-network-performance) Unit: None | ## Linux setup On Linux servers, the *ethtool plugin* enables you to import the network performance metrics into CloudWatch. ethtool is a standard Linux utility that can collect statistics about Ethernet devices on Linux servers. The statistics it collects depend on the network device and driver. Examples of these statistics include `tx_cnt`, `rx_bytes`, `tx_errors`, and `align_errors`. When you use the ethtool plugin with the CloudWatch agent, you can also import these statistics into CloudWatch, along with the EC2 network performance metrics listed earlier in this section. **Tip** To find the statistics available on our operating system and network device, use the `ethtool –S` command. When the CloudWatch agent imports metrics into CloudWatch, it adds an `ethtool_` prefix to the names of all imported metrics. So the standard ethtool statistic `rx_bytes` is called `ethtool_rx_bytes` in CloudWatch, and the EC2 network performance metric `bw_in_allowance_exceeded` is called `ethtool_bw_in_allowance_exceeded` in CloudWatch. On Linux servers, to import ethtool metrics, add an `ethtool` section to the `metrics_collected` section of the CloudWatch agent configuration file. The `ethtool` section can include the following subsections: + **interface\$1include**— Including this section causes the agent to collect metrics from only the interfaces that have names listed in this section. If you omit this section, metrics are collected from all Ethernet interfaces that aren't listed in `interface_exclude`. The default ethernet interface is `eth0`. + **interface\$1exclude**— If you include this section, list the Ethernet interfaces that you don't want to collect metrics from. The ethtool plugin always ignores loopback interfaces. + **metrics\$1include**— This section lists the metrics to import into CloudWatch. It can include both standard statistics collected by ethtool and Amazon EC2 high-resolution network metrics. The following example displays part of the CloudWatch agent configuration file. This configuration collects the standard ethtool metrics `rx_packets` and `tx_packets`, and the Amazon EC2 network performance metrics from only the `eth1` interface. For more information about the CloudWatch agent configuration file, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). ``` { "metrics": { "append_dimensions": { "InstanceId": "${aws:InstanceId}" }, "metrics_collected": { "ethtool": { "interface_include": [ "eth1" ], "metrics_include": [ "bw_in_allowance_exceeded", "bw_out_allowance_exceeded", "conntrack_allowance_exceeded", "linklocal_allowance_exceeded", "pps_allowance_exceeded" ] } } } } ``` ## Windows setup On Windows servers, the network performance metrics are available through Windows Performance Counters, which the CloudWatch agent already collects metrics from. So you do not need a plugin to collect these metrics from Windows servers. The following is a sample configuration file to collect network performance metrics from Windows. For more information about editing the CloudWatch agent configuration file, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). ``` { "metrics": { "append_dimensions": { "InstanceId": "${aws:InstanceId}" }, "metrics_collected": { "ENA Packets Shaping": { "measurement": [ "Aggregate inbound BW allowance exceeded", "Aggregate outbound BW allowance exceeded", "Connection tracking allowance exceeded", "Link local packet rate allowance exceeded", "PPS allowance exceeded" ], "metrics_collection_interval": 60, "resources": [ "*" ] } } } } ``` ## Viewing network performance metrics After importing network performance metrics into CloudWatch, you can view these metrics as time series graphs, and create alarms that can watch these metrics and notify you if they breach a threshold that you specify. The following procedure shows how to view ethtool metrics as a time series graph. For more information about setting alarms, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md). Because all of these metrics are aggregate counters, you can use CloudWatch metric math functions such as `RATE(METRICS())` to calculate the rate for these metrics in graphs or use them to set alarms. For more information about metric math functions, see [Using math expressions with CloudWatch metrics](using-metric-math.md). **To view network performance metrics in the CloudWatch console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. Choose the namespace for the metrics collected by the agent. By default, this is **CWAgent**, but you may have specified a different namespace in the CloudWatch agent configuration file. 1. Choose a metric dimension (for example, **Per-Instance Metrics**). 1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following: 1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table. 1. To sort the table, use the column heading. 1. To filter by resource, choose the resource ID, and then choose **Add to search**. 1. To filter by metric, choose the metric name, and then choose **Add to search**. 1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, and then choose **Add to dashboard**. # Collect Amazon EBS NVMe driver metrics For CloudWatch agent to collect AWS NVMe driver metrics for Amazon EBS volumes attached to an Amazon EC2 instance, add the `diskio` section inside the `metrics_collected` section of the CloudWatch agent configuration file. Additionally, the CloudWatch agent binary requires `ioctl` permissions for NVMe driver devices to collect metrics from attached Amazon EBS volumes. The following metrics can be collected. | Metric | Metric name in CloudWatch | Description | | --- | --- | --- | | `ebs_total_read_ops` | `diskio_ebs_total_read_ops` | The total number of completed read operations. | | `ebs_total_write_ops` | `diskio_ebs_total_write_ops` | The total number of completed write operations. | | `ebs_total_read_bytes` | `diskio_ebs_total_read_bytes` | The total number of read bytes transferred. | | `ebs_total_write_bytes` | `diskio_ebs_total_write_bytes` | The total number of write bytes transferred. | | `ebs_total_read_time` | `diskio_ebs_total_read_time` | The total time spent, in microseconds, by all completed read operations. | | `ebs_total_write_time` | `diskio_ebs_total_write_time` | The total time spent, in microseconds, by all completed write operations. | | `ebs_volume_performance_exceeded_iops` | `diskio_ebs_volume_performance_exceeded_iops` | The total time, in microseconds, that IOPS demand exceeded the volume's provisioned IOPS performance. | | `ebs_volume_performance_exceeded_tp` | `diskio_ebs_volume_performance_exceeded_tp` | The total time, in microseconds, that throughput demand exceeded the volume's provisioned throughput performance. | | `ebs_ec2_instance_performance_exceeded_iops` | `diskio_ebs_ec2_instance_performance_exceeded_iops` | The total time, in microseconds, that the EBS volume exceeded the attached Amazon EC2 instance's maximum IOPS performance. | | `ebs_ec2_instance_performance_exceeded_tp` | `diskio_ebs_ec2_instance_performance_exceeded_tp` | The total time, in microseconds, that the EBS volume exceeded the attached Amazon EC2 instance's maximum throughput performance. | | `ebs_volume_queue_length` | `diskio_ebs_volume_queue_length` | The number of read and write operations waiting to be completed. | # Collect Amazon EC2 instance store volume NVMe driver metrics For CloudWatch agent to collect AWS NVMe driver metrics for instance store volumes attached to an Amazon EC2 instance, add the `diskio` section inside the `metrics_collected` section of the CloudWatch agent configuration file. Additionally, the CloudWatch agent binary requires `ioctl` permissions for NVMe driver devices to collect metrics from attached instance store volumes. The following metrics can be collected. | Metric | Metric name in CloudWatch | Description | | --- | --- | --- | | `instance_store_total_read_ops` | `diskio_instance_store_total_read_ops` | The total number of completed read operations. | | `instance_store_total_write_ops` | `diskio_instance_store_total_write_ops` | The total number of completed write operations. | | `instance_store_total_read_bytes` | `diskio_instance_store_total_read_bytes` | The total number of read bytes transferred. | | `instance_store_total_write_bytes` | `diskio_instance_store_total_write_bytes` | The total number of write bytes transferred. | | `instance_store_total_read_time` | `diskio_instance_store_total_read_time` | The total time spent, in microseconds, by all completed read operations. | | `instance_store_total_write_time` | `diskio_instance_store_total_write_time` | The total time spent, in microseconds, by all completed write operations. | | `instance_store_performance_exceeded_iops` | `diskio_instance_store_performance_exceeded_iops` | The total time, in microseconds, that IOPS demand exceeded the volume's IOPS maximum performance. | | `instance_store_performance_exceeded_tp` | `diskio_instance_store_performance_exceeded_tp` | The total time, in microseconds, that throughput demand exceeded the volume's maximum throughput performance. | | `instance_store_volume_queue_length` | `diskio_instance_store_volume_queue_length` | The number of read and write operations waiting to be completed. | # Collect NVIDIA GPU metrics You can use the CloudWatch agent to collect NVIDIA GPU metrics from Linux servers. To set this up, add a `nvidia_gpu` section inside the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Linux section](CloudWatch-Agent-Configuration-File-Details.md#CloudWatch-Agent-Linux-section). Additionally, the instance must have an NVIDIA driver installed. NVIDIA drivers on pre-installed on some Amazon Machine Images (AMIs). Otherwise, you can manually install the driver. For more information, see [ Install NVIDIA drivers on Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-nvidia-driver.html). The following metrics can be collected. All of these metrics are collected with no CloudWatch `Unit`, but you can specify a unit for each metric by adding a parameter to the CloudWatch agent configuration file. For more information, see [Linux section](CloudWatch-Agent-Configuration-File-Details.md#CloudWatch-Agent-Linux-section). | Metric | Metric name in CloudWatch | Description | | --- | --- | --- | | `utilization_gpu` | `nvidia_smi_utilization_gpu` | The percentage of time over the past sample period during which one or more kernals on the GPU was running. | | `temperature_gpu` | `nvidia_smi_temperature_gpu` | The core GPU temperature in degrees Celsius. | | `power_draw` | `nvidia_smi_power_draw` | The last measured power draw for the entire board, in watts. | | `utilization_memory` | `nvidia_smi_utilization_memory` | The percentage of time over the past sample period during which global (device) memory was being read or written. | | `fan_speed` | `nvidia_smi_fan_speed` | The percentage of maximum fan speed that the device's fan is currently intended to run at. | | `memory_total` | `nvidia_smi_memory_total` | Reported total memory, in MB. | | `memory_used` | `nvidia_smi_memory_used` | Memory used, in MB. | | `memory_free` | `nvidia_smi_memory_free` | Memory free, in MB. | | `pcie_link_gen_current` | `nvidia_smi_pcie_link_gen_current` | The current link generation. | | `pcie_link_width_current` | `nvidia_smi_pcie_link_width_current` | The current link width. | | `encoder_stats_session_count` | `nvidia_smi_encoder_stats_session_count` | Current number of encoder sessions. | | `encoder_stats_average_fps` | `nvidia_smi_encoder_stats_average_fps` | The moving average of the encode frames per second. | | `encoder_stats_average_latency` | `nvidia_smi_encoder_stats_average_latency` | The moving average of the encode latency in microseconds. | | `clocks_current_graphics` | `nvidia_smi_clocks_current_graphics` | The current frequency of the graphics (shader) clock. | | `clocks_current_sm` | `nvidia_smi_clocks_current_sm` | The current frequency of the Streaming Multiprocessor (SM) clock. | | `clocks_current_memory` | `nvidia_smi_clocks_current_memory` | The current frequency of the memory clock. | | `clocks_current_video` | `nvidia_smi_clocks_current_video` | The current frequency of the video (encoder plus decoder) clocks. | All of these metrics are collected with the following dimensions: | Dimension | Description | | --- | --- | | `index` | A unique identifier for the GPU on this server. Represents the NVIDIA Management Library (NVML) index of the device. | | `name` | The type of GPU. For example, `NVIDIA Tesla A100` | | `arch` | The server architecture. | # Collect Java Management Extensions (JMX) metrics You can use the CloudWatch agent to collect Java Management Extensions (JMX) metrics from your Java applications. The CloudWatch agent supports collecting these metrics from the following versions: + JVM 8 and later + Kafka 0.8.2.x and later + Tomcat 9, 10.1, and 11 (beta) ------ #### [ Amazon EC2 ] **To enable JMX in your JVM instance** For the CloudWatch agent to be able to collect JMX metrics, your application’s JVM must bind to a port using the `com.sun.management.jmxremote.port` system property. ``` java -Dcom.sun.management.jmxremote.port=port-number -jar example.jar ``` For more information and other configurations, see the [JMX documentation](https://docs.oracle.com/en/java/javase/17/management/monitoring-and-management-using-jmx-technology.html). ------ #### [ Amazon EKS ] **To enable JMX on your Java application pods** When using the CloudWatch Observability EKS add-on, you can manage how JMX metrics are enabled with annotations. For more information, see [Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart](install-CloudWatch-Observability-EKS-addon.md). To enable JMX metrics collection from a workload, add the following annotations to the workload manifest file under the `PodTemplate` section: + `instrumentation.opentelemetry.io/inject-java: "true"` + One or more of the following: + For JVM metrics: `cloudwatch.aws.amazon.com/inject-jmx-jvm: "true"` + For Kafka broker metrics: `cloudwatch.aws.amazon.com/inject-jmx-kafka: "true"` + For Kafka consumer metrics: `cloudwatch.aws.amazon.com/inject-jmx-kafka-consumer: "true"` + For Kafka producer metrics: `cloudwatch.aws.amazon.com/inject-jmx-kafka-producer: "true"` + For Tomcat metrics: `cloudwatch.aws.amazon.com/inject-jmx-tomcat: "true"` ------ To start collecting JMX metrics, add a `jmx` section inside the `metrics_collected` section of the CloudWatch agent configuration file. The `jmx` section can contain the following fields. + `jvm` – Optional. Specifies that you want to retrieve Java Virtual Machine (JVM) metrics from the instance. For more information, see [Collect JVM metrics](#CloudWatch-Agent-JVM-metrics). This section can include the following fields: + `measurement` – Specifies the array of JVM metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the table in [Collect JVM metrics](#CloudWatch-Agent-JVM-metrics). Within the entry for each individual metric, you can optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `kafka` – Optional. Specifies that you want to retrieve Apache Kafka broker metrics from the instance. For more information, see [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics). This section can include the following fields: + `measurement` – Specifies the array of Kafka broker metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the first table in [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics). Within the entry for each individual metric, you can optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `kafka-consumer` – Optional. Specifies that you want to retrieve Apache Kafka consumer metrics from the instance. For more information, see [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics). This section can include the following fields: + `measurement` – Specifies the array of Kafka broker metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the second metrics table in [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics). Within the entry for each individual metric, you can optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `kafka-producer` – Optional. Specifies that you want to retrieve Apache Kafka producer metrics from the instance. For more information, see [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics). This section can include the following fields: + `measurement` – Specifies the array of Kafka broker metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the third metrics table in [Collect Kafka metrics](#CloudWatch-Agent-Kafka-metrics). Within the entry for each individual metric, you can optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + `tomcat` – Optional. Specifies that you want to retrieve Tomcat metrics from the instance. For more information, see [Collect Tomcat metrics](#CloudWatch-Agent-Tomcat-metrics). This section can include the following fields: + `measurement` – Specifies the array of Tomcat metrics to be collected. For a list of the possible values to use here, see the **Metric** column in the table in [Collect Tomcat metrics](#CloudWatch-Agent-Tomcat-metrics). Within the entry for each individual metric, you can optionally specify one or both of the following: + `rename` – Specifies a different name for this metric. + `unit` – Specifies the unit to use for this metric, overriding the default unit for the metric. The unit that you specify must be a valid CloudWatch metric unit, as listed in the `Unit` description in [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). The `jmx` section can also include the optional `append_dimensions` field: + `append_dimensions` – Optional. Additional dimensions to use for only the process metrics. If you specify this field, it's used in addition to dimensions specified in the `append_dimensions` field that is used for all types of metrics collected by the agent. **The following fields are for Amazon EC2 only.** + `endpoint` – The address for the JMX client to connect to. The format is `ip:port`. If the endpoint is not the localhost, and password authentication and SSL must be enabled. + `metrics_collection_interval` – Optional. Specifies how often to collect the processes metrics, overriding the global `metrics_collection_interval` specified in the `agent` section of the configuration file. This value is specified in seconds. For example, specifying 10 causes metrics to be collected every 10 seconds, and setting it to 300 specifies metrics to be collected every 5 minutes. If you set this value below 60 seconds, each metric is collected as a high-resolution metric. For more information, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). If JMX was enabled with password authentication or SSL for remote access, you can use the following fields. + `password_file` – Optional. Specifies a Java properties file of keys to passwords. The file must be read-only and restricted to the user running the CloudWatch agent. If password authentication is enabled, this requires the same username and password pair as the entry in the JMX password file provided in the `com.sun.management.jmxremote.password.file` property. If SSL is enabled, it requires entries for `keystore` and `truststore` and corresponds to the `javax.net.ssl.keyStorePassword` and `javax.net.ssl.trustStorePassword` respectively. + `username` – If password authentication is enabled, specify the username that matches the username in the provided password file. + `keystore_path` – If SSL is enabled, specify the full path to the Java keystore, which consists of a private key and a certificate to the public key. Corresponds to the `javax.net.ssl.keyStore` property. + `keystore_type` – If SSL is enabled, specify the type of keystore being used. Corresponds to the `javax.net.ssl.keyStoreType` property. + `truststore_path` – If SSL is enabled, specify the full path to the Java truststore, which must contain the remote JMX server's public certificate. Corresponds to the `javax.net.ssl.trustStore` property. + `truststore_type` – If SSL is enabled, specify the type of truststore being used. Corresponds to the `javax.net.ssl.trustStoreType` property. + `remote_profile` – Optional. Supported JMX remote profiles are TLS in combination with SASL profiles: `SASL/PLAIN`, `SASL/DIGEST-MD5`, and `SASL/CRAM-MD5`. Should be one of:`SASL/PLAIN`, `SASL/DIGEST-MD5`, `SASL/CRAM-MD5`, `TLS SASL/PLAIN`, `TLS SASL/DIGEST-MD5`, or `TLS SASL/CRAM-MD5` + `realm` – Optional. The realm as required by the remote profile `SASL/DIGEST-MD5`. + `registry_ssl_enabled` – If RMI registry authentication is enabled. Set to true if the JVM was configured with `com.sun.management.jmxremote.registry.ssl=true`. + `insecure` Set to `true` to opt out of the validation required if the agent is configured for a non-localhost endpoint. The following is an example of the `jmx` section of the CloudWatch agent configuration file. ``` { "metrics": { "metrics_collected": { "jmx": [ { "endpoint": "remotehost:1314", "jvm": { "measurement": [ "jvm.memory.heap.init", "jvm.memory.nonheap.used" ] }, "kafka": { "measurement": [ "kafka.request.count", { "name": "kafka.message.count", "rename": "KAFKA_MESSAGE_COUNT", "unit": "Count" } ] }, "username": "cwagent", "keystore_path": "/path/to/keystore", "keystore_type": "PKCS12", "truststore_path": "/path/to/truststore", "truststore_type": "PKCS12" }, { "endpoint": "localhost:1315", "kafka-producer": { "measurement": [ "kafka.producer.request-rate" ] }, "append_dimensions": { "service.name": "kafka/1" } } ] } } } ``` ## Collect JVM metrics You can use the CloudWatch agent to collect Java Virtual Machine (JVM) metrics. To set this up, add a `jvm` section inside the `jmx`section of the CloudWatch agent configuration file. The following metrics can be collected. | Metric | Dimensions | Description | | --- | --- | --- | | `jvm.classes.loaded` | [DEFAULT] | The total number of loaded classes. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.gc.collections.count` | [DEFAULT], `name` | The total number of garbage collections that have occurred. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.gc.collections.elapsed` | [DEFAULT], `name` | The approximate accumulated garbage collection elapsed time. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.heap.init` | [DEFAULT] | The initial amount of memory that the JVM requests from the operating system for the heap. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.heap.max` | [DEFAULT] | The maximum amount of memory that can be used for the heap. **Unit:** Bytes **Meaningful statistics:** Maximum | | `jvm.memory.heap.used` | [DEFAULT] | The current heap memory usage. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.heap.committed` | [DEFAULT] | The amount of memory that is guaranteed to be available for the heap. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.nonheap.init` | [DEFAULT] | The initial amount of memory that the JVM requests from the operating system for non-heap purposes. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.nonheap.max` | [DEFAULT] | The maximum amount of memory that can be used for non-heap purposes. **Unit:** Bytes **Meaningful statistics:** Maximum | | `jvm.memory.nonheap.used` | [DEFAULT] | The current non-heap memory usage. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.nonheap.committed` | [DEFAULT] | The amount of memory that is guaranteed to be available for non-heap purposes. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.pool.init` | [DEFAULT], `name` | The initial amount of memory that the JVM requests from the operating system for the memory pool. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.pool.max` | [DEFAULT], `name` | The maximum amount of memory that can be used for the memory pool. **Unit:** Bytes **Meaningful statistics:** Maximum | | `jvm.memory.pool.used` | [DEFAULT], `name` | The current memory pool memory usage. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.memory.pool.committed` | [DEFAULT], `name` | The amount of memory that is guaranteed to be available for the memory pool. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `jvm.threads.count` | [DEFAULT] | The current number of threads. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | The JVM metrics are collected with the following dimensions: | Dimension | Description | | --- | --- | | [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. | | `name` | For `jvm.gc.collections` metrics, the value is the garbage collector name. For `jvm.memory.pool` metrics, the value is the memory pool name. | ## Collect Kafka metrics You can use the CloudWatch agent to collect Apache Kafka metrics. To set this up, add one or more of the following subsections inside the `jmx` section of the CloudWatch agent configuration file. + Use a `kafka` section to collect Kafka broker metrics. + Use a `kafka-consumer` section to collect Kafka consumer metrics. + Use a `kafka-producer` section to collect Kafka producer metrics. **Kafka broker metrics** The following metrics can be collected for Kafka brokers. | Metric | Dimensions | Description | | --- | --- | --- | | `kafka.message.count` | [DEFAULT] | The number of messages received by the Kafka broker. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.request.count` | [DEFAULT], `type` | The number of requests received by the Kafka broker. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.request.failed` | [DEFAULT], `type` | The number of requests to the Kafka broker that resulted in a failure. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.request.time.total` | [DEFAULT], `type` | The total time that the Kafka broker has taken to service requests. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.request.time.50p` | [DEFAULT], `type` | The 50th percentile time that the Kafka broker has taken to service requests. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.request.time.99p` | [DEFAULT], `type` | The 99th percentile time that the Kafka broker has taken to service requests. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.request.time.avg` | [DEFAULT], `type` | The average time that the Kafka broker has taken to service requests. **Unit:** Milliseconds **Meaningful statistics:** Average | | `kafka.network.io` | [DEFAULT], `state` | The number of bytes received by or sent by the Kafka broker. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.purgatory.size` | [DEFAULT], `type` | The number of requests waiting in purgatory. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.partition.count` | [DEFAULT] | The number of partitions on the Kafka broker. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.partition.offline` | [DEFAULT] | The number of partitions that are offline. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.partition.under_replicated` | [DEFAULT] | The number of under-replicated partitions. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.isr.operation.count` | [DEFAULT], `operation` | The number of in-sync replica shrink and expand operations. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.max.lag` | [DEFAULT] | The maximum lag in messages between follower and leader replicas. **Unit:** None **Meaningful statistics:** Maximum | | `kafka.controller.active.count` | [DEFAULT] | The number of active controllers on the broker. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.leader.election.rate` | [DEFAULT] | Leader election rate. If this increases, it indicates broker failures. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.unclean.election.rate` | [DEFAULT] | Unclean leader election rate. If this increases, it indicates broker failures. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.request.queue` | [DEFAULT] | The size of the request queue. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.logs.flush.time.count` | [DEFAULT] | The logs flush count. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.logs.flush.time.median` | [DEFAULT] | The 50th percentile value of the logs flush count. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.logs.flush.time.99p` | [DEFAULT] | The 99th percentile value of the logs flush count. **Unit:** Milliseconds **Meaningful statistics:** Minimum, Maximum, Average | The Kafka broker metrics are collected with the following dimensions: | Dimension | Description | | --- | --- | | [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. | | `type` | The request type. Possible values are `produce`, `fetch`, `fetchconsumer`, and `fetchfollower`. | | `state` | The direction of network traffic. Possible values are `in` and `out`. | | `operation` | The operation type for the in-sync replica. Possible values are `shrink` and `expand`. | **Kafka consumer metrics** The following metrics can be collected for Kafka consumers. | Metric | Dimensions | Description | | --- | --- | --- | | `kafka.consumer.fetch-rate` | [DEFAULT], `client-id` | The number of fetch requests for all topics per second. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.consumer.records-lag-max` | [DEFAULT], `client-id` | The number of messages that the consumer lags behind the producer. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.consumer.total.bytes-consumed-rate` | [DEFAULT], `client-id` | The average number of bytes consumed for all topics per second. **Unit:** Bytes **Meaningful statistics:** Average | | `kafka.consumer.total.fetch-size-avg` | [DEFAULT], `client-id` | The number of bytes fetched per request for all topics. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.consumer.total.records-consumed-rate` | [DEFAULT], `client-id` | The average number of records consumed for all topics per second. **Unit:** None **Meaningful statistics:** Average | | `kafka.consumer.bytes-consumed-rate` | [DEFAULT], `client-id`, `topic` | The average number of bytes consumed per second. **Unit:** Bytes **Meaningful statistics:** Average | | `kafka.consumer.fetch-size-avg` | [DEFAULT], `client-id`, `topic` | The number of bytes fetched per request. **Unit:** Bytes **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.consumer.records-consumed-rate` | [DEFAULT], `client-id`, `topic` | The average number of records consumed per second. **Unit:** None **Meaningful statistics:** Average | The Kafka consumer metrics are collected with the following dimensions: | Dimension | Description | | --- | --- | | [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. | | `client-id` | The ID of the client. | | `topic` | The Kafka topic. | **Kafka producer metrics** The following metrics can be collected for Kafka producers. | Metric | Dimensions | Description | | --- | --- | --- | | `kafka.producer.io-wait-time-ns-avg` | [DEFAULT], `client-id` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes. **Unit:** None **Meaningful statistics:** Average | | `kafka.producer.outgoing-byte-rate` | [DEFAULT], `client-id` | The average number of outgoing bytes sent per second to all servers. **Unit:** Bytes **Meaningful statistics:** Average | | `kafka.producer.request-latency-avg` | [DEFAULT], `client-id` | The average request latency. **Unit:** Milliseconds **Meaningful statistics:** Average | | `kafka.producer.request-rate` | [DEFAULT], `client-id` | The average number of requests sent per second. **Unit:** None **Meaningful statistics:** Average | | `kafka.producer.response-rate` | [DEFAULT], `client-id` | The number of responses received per second. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `kafka.producer.byte-rate` | [DEFAULT], `client-id`, `topic` | The average number of bytes sent per second for a topic. **Unit:** Bytes **Meaningful statistics:** Average | | `kafka.producer.compression-rate` | [DEFAULT], `client-id`, `topic` | The average compression rate of record batches for a topic. **Unit:** None **Meaningful statistics:** Average | | `kafka.producer.record-error-rate` | [DEFAULT], `client-id`, `topic` | The average per-second number of record sends that resulted in errors for a topic. **Unit:** None **Meaningful statistics:** Average | | `kafka.producer.record-retry-rate` | [DEFAULT], `client-id`, `topic` | The average per-second number of retried record sends for a topic. **Unit:** None **Meaningful statistics:** Average | | `kafka.producer.record-send-rate` | [DEFAULT], `client-id`, `topic` | The average number of records sent per second for a topic. **Unit:** None **Meaningful statistics:** Average | Kafka producer metrics are collected with the following dimensions: | Dimension | Description | | --- | --- | | [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. | | `client-id` | The ID of the client. | | `topic` | The Kafka topic. | ## Collect Tomcat metrics You can use the CloudWatch agent to collect Apache Tomcat metrics. To set this up, add a `tomcat` section inside the `metrics_collected` section of the CloudWatch agent configuration file. The following metrics can be collected. | Metric | Dimensions | Description | | --- | --- | --- | | `tomcat.sessions` | [DEFAULT] | The number of active sessions. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `tomcat.errors` | [DEFAULT], `proto_handler` | The number of errors encountered. **Unit:** None **Meaningful statistics:** Minimum, Maximum, Average | | `tomcat.processing_time` | [DEFAULT], `proto_handler` | The total processing time. **Unit:** Milliseconds **Meaningful statistics: **Minimum, Maximum, Average | | `tomcat.traffic` | [DEFAULT], `proto_handler` | The number of bytes received and sent. **Unit:** Bytes **Meaningful statistics: **Minimum, Maximum, Average | | `tomcat.threads` | [DEFAULT], `proto_handler` | The number of threads. **Unit:** None **Meaningful statistics: **Minimum, Maximum, Average | | `tomcat.max_time` | [DEFAULT], `proto_handler`, `direction` | Maximum time to process a request. **Unit:** Milliseconds **Meaningful statistics: **Maximum | | `tomcat.request_count` | [DEFAULT], `proto_handler` | The total requests. **Unit:** None **Meaningful statistics: **Minimum, Maximum, Average | Tomcat metrics are collected with the following dimensions: | Dimension | Description | | --- | --- | | [DEFAULT] | On Amazon EC2 by default, the host is also published as a dimension of metrics that are collected by the CloudWatch agent, unless you are using the `append_dimensions` field in the `metrics` section. See `omit_hostname` in the agent section of [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md) for more information. On Amazon EKS by default, k8s related context is also published as dimensions of metrics (`k8s.container.name`, `k8s.deployment.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.pod.name`, and `k8s.replicaset.name`). These can be filtered down using the `aggregation_dimensions` field. | | `proto_handler` | The `proto_handler` is an identifier for a connector, which is provided in the `--` format (for example, `http-nio-8080`). | | `direction` | The traffic direction. Possible values are `received` and `sent`. | # Collect metrics and traces with OpenTelemetry You can collect metrics and traces from your applications or services using the CloudWatch agent with the OpenTelemetry Protocol (OTLP), which is a popular open source solution. You can use any OpenTelemetry SDK to send metrics and traces to the CloudWatch agent. For more information about the available OpenTelemetry SDKs, see [ OpenTelemetry Supported Language APIs & SDKs.](https://opentelemetry.io/docs/languages/). To collect OpenTelemetry metrics and traces, add an `otlp` section to the CloudWatch agent configuration file. The section has the following fields: + `grpc_endpoint` – Optional. Specifies the address for the CloudWatch agent to use to listen for OpenTelemetry metrics or traces sent using gRPC Remote Procedure Calls. The format is `ip:port`. This address must match the address set for the gRPC exporter in the OpenTelemetry SDK. If you omit this field, the default of `127.0.0.1:4317` is used. + `http_endpoint` – Optional. Specifies the address for the CloudWatch agent to use to listen for OpenTelemetry metrics or traces sent over HTTP. The format is `ip:port`. This address must match the address set for the HTTP exporter in the OpenTelemetry SDK. If you omit this field, the default of `127.0.0.1:4318` is used. + `tls` – Optional. Specifies that the server should be configured with TLS. + `cert_file` – Path to the TLS certificate to use for TLS required connections. + `key_file` – Path to the TLS key to use for TLS required connections. The `otlp` section can be placed in multiple sections within the CloudWatch agent configuration file depending on how and where you want to send the metrics and traces. **Important** Each `otlp` section requires a unique endpoint and port. For detailed information about splitting the metrics and traces endpoints, see [OTLP Exporter Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/) in the OpenTelemetry SDK documentation. To send metrics to CloudWatch or Amazon Managed Service for Prometheus, add the `otlp` section under `metrics_collected` within the `metrics` section. For more information about sending metrics to different destinations, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). The following example shows a configuration that sends metrics to CloudWatch: **Note** If you are running the agent in containerized environments and sending telemetry from outside the agent container’s network, make sure to specify the endpoint as `0.0.0.0` instead of the default endpoint `127.0.0.1`. ``` { "metrics": { "metrics_collected": { "otlp": { "grpc_endpoint": "127.0.0.1:4317", "http_endpoint": "127.0.0.1:4318" } } } } ``` To send metrics to Amazon CloudWatch Logs using the Embedded metric format (EMF), add the `otlp` section under `metrics_collected` within the `logs` section. This sends the EMF logs by default to the `/aws/cwagent` log group and a generated log stream. The metrics are extracted into the `CWAgent` namespace by default. The following example shows a configuration that sends metrics as EMF logs to CloudWatch Logs: ``` { "logs": { "metrics_collected": { "otlp": { "grpc_endpoint": "127.0.0.1:4317", "http_endpoint": "127.0.0.1:4318" } } } } ``` To send traces to AWS X-Ray, add the `otlp` section under `traces_collected` within the `traces` section. The following example shows a configuration that sends traces to X-Ray: ``` { "traces": { "traces_collected": { "otlp": { "grpc_endpoint": "127.0.0.1:4317", "http_endpoint": "127.0.0.1:4318" } } } } ``` # Collect process metrics with the procstat plugin The *procstat* plugin enables you to collect metrics from individual processes. The plugin is supported on Linux servers and on servers running supported version of Windows Server. This section describes how to configure the CloudWatch agent for procstat and view metrics the CloudWatch agent imports. It also lists the metrics that procstat collects. **Note** The `procstat` plugin is not supported for the Fargate launch type in Amazon ECS environments. **Topics** + [ ## Configure the CloudWatch agent for procstat ](#CloudWatch-Agent-procstat-configuration) + [ ## Metrics collected by procstat ](#CloudWatch-Agent-procstat-process-metrics-collected) + [ ## Viewing process metrics imported by the CloudWatch agent ](#CloudWatch-view-procstat-metrics) ## Configure the CloudWatch agent for procstat To use the procstat plugin, add a `procstat` section in the `metrics_collected` section of the CloudWatch agent configuration file. There are three ways to specify the processes to monitor. You can use only one of these methods, but you can use that method to specify one or more processes to monitor. + `pid_file`: Selects processes by the names of the process identification number (PID) files they create. + `exe`: Selects the processes that have process names that match the string that you specify, using regular expression matching rules. The match is a "contains" match, meaning that if you specify `agent` as the term to match, processes with names like `cloudwatchagent` match the term. For more information, see [Syntax](https://github.com/google/re2/wiki/Syntax). + `pattern`: Selects processes by the command lines used to start the processes. All processes are selected that have command lines matching the specified string using regular expression matching rules. The entire command line is checked, including parameters and options used with the command. The match is a "contains" match, meaning that if you specify `-c` as the term to match, processes with parameters like `-config` match the term. The CloudWatch agent uses only one of these methods, even if you include more than one of the above sections. If you specify more than one section, the CloudWatch agent uses the `pid_file` section if it is present. If not, it uses the `exe` section. On Linux servers, the strings that you specify in an `exe` or `pattern` section are evaluated as regular expressions. On servers running Windows Server, these strings are evaluated as WMI queries. An example would be `pattern: "%apache%"`. For more information, see [LIKE Operator](https://docs.microsoft.com/en-us/windows/desktop/WmiSdk/like-operator). Whichever method you use, you can include an optional `metrics_collection_interval` parameter, which specifies how often in seconds to collect those metrics. If you omit this parameter, the default value of 60 seconds is used. In the examples in the following sections, the `procstat` section is the only section included in the `metrics_collected` section of the agent configuration file. Actual configuration files can also include other sections in `metrics_collected`. For more information, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). ### Configure with pid\$1file The following example `procstat` section monitors the processes that create the PID files `example1.pid` and `example2.pid`. Different metrics are collected from each process. Metrics collected from the process that creates `example2.pid` are collected every 10 seconds, and the metrics collected from the `example1.pid` process are collected every 60 seconds, the default value. ``` { "metrics": { "metrics_collected": { "procstat": [ { "pid_file": "/var/run/example1.pid", "measurement": [ "cpu_usage", "memory_rss" ] }, { "pid_file": "/var/run/example2.pid", "measurement": [ "read_bytes", "read_count", "write_bytes" ], "metrics_collection_interval": 10 } ] } } } ``` ### Configuring with exe The following example `procstat` section monitors all processes with names that match the strings `agent` or `plugin`. The same metrics are collected from each process. ``` { "metrics": { "metrics_collected": { "procstat": [ { "exe": "agent", "measurement": [ "cpu_time", "cpu_time_system", "cpu_time_user" ] }, { "exe": "plugin", "measurement": [ "cpu_time", "cpu_time_system", "cpu_time_user" ] } ] } } } ``` ### Configuring with pattern The following example `procstat` section monitors all processes with command lines that match the strings `config` or `-c`. The same metrics are collected from each process. ``` { "metrics": { "metrics_collected": { "procstat": [ { "pattern": "config", "measurement": [ "rlimit_memory_data_hard", "rlimit_memory_data_soft", "rlimit_memory_stack_hard", "rlimit_memory_stack_soft" ] }, { "pattern": "-c", "measurement": [ "rlimit_memory_data_hard", "rlimit_memory_data_soft", "rlimit_memory_stack_hard", "rlimit_memory_stack_soft" ] } ] } } } ``` ## Metrics collected by procstat The following table lists the metrics that you can collect with the `procstat` plugin. The CloudWatch agent adds `procstat` to the beginning of the following metric names. There is a different syntax depending on whether it was collected from a Linux server or a server running Windows Server. For example, the `cpu_time` metric appears as `procstat_cpu_time` when collected from Linux and as `procstat cpu_time` when collected from Windows Server. | Metric name | Available on | Description | | --- | --- | --- | | `cpu_time` | Linux | The amount of time that the process uses the CPU. This metric is measured in hundredths of a second. Unit: Count | | `cpu_time_guest` | Linux | The amount of time that the process is in guest mode. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_guest_nice` | Linux | The amount of time that the process is running in a nice guest. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_idle` | Linux | The amount of time that the process is in idle mode. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_iowait` | Linux | The amount of time that the process is waiting for I/O operations to complete. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_irq` | Linux | The amount of time that the process is servicing interrupts. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_nice` | Linux | The amount of time that the process is in nice mode. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_soft_irq` | Linux | The amount of time that the process is servicing software interrupts. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_steal` | Linux | The amount of time spent running in other operating systems when running in a virtualized environment. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_stolen` | Linux, Windows Server | The amount of time that the process is in *stolen time*, which is time spent in other operating systems in a virtualized environment. This metric is measured in hundredths of a second. Type: Float Unit: None | | `cpu_time_system` | Linux, Windows Server, macOS | The amount of time that the process is in system mode. This metric is measured in hundredths of a second. Type: Float Unit: Count | | `cpu_time_user` | Linux, Windows Server, macOS | The amount of time that the process is in user mode. This metric is measured in hundredths of a second. Unit: Count | | `cpu_usage` | Linux, Windows Server, macOS | The percentage of time that the process is active in any capacity. Unit: Percent | | `memory_data` | Linux, macOS | The amount of memory that the process uses for data. Unit: Bytes | | `memory_locked` | Linux, macOS | The amount of memory that the process has locked. Unit: Bytes | | `memory_rss` | Linux, Windows Server, macOS | The amount of real memory (resident set) that the process is using. Unit: Bytes | | `memory_stack` | Linux, macOS | The amount of stack memory that the process is using. Unit: Bytes | | `memory_swap` | Linux, macOS | The amount of swap memory that the process is using. Unit: Bytes | | `memory_vms` | Linux, Windows Server, macOS | The amount of virtual memory that the process is using. Unit: Bytes | | `num_fds` | Linux | The number of file descriptors that this process has open. Unit: None | | `num_threads` | Linux, Windows, macOS | The number of threads in this process. Unit: None | | `pid` | Linux, Windows Server, macOS | Process identifier (ID). Unit: None | | `pid_count` | Linux, Windows Server. macOS | The number of process IDs associated with the process. On Linux servers and macOS computers the full name of this metric is `procstat_lookup_pid_count` and on Windows Server it is `procstat_lookup pid_count`. Unit: None | | `read_bytes` | Linux, Windows Server | The number of bytes that the process has read from disks. Unit: Bytes | | `write_bytes` | Linux, Windows Server | The number of bytes that the process has written to disks. Unit: Bytes | | `read_count` | Linux, Windows Server | The number of disk read operations that the process has executed. Unit: None | | `rlimit_realtime_priority_hard` | Linux | The hard limit on the real-time priority that can be set for this process. Unit: None | | `rlimit_realtime_priority_soft` | Linux | The soft limit on the real-time priority that can be set for this process. Unit: None | | `rlimit_signals_pending_hard` | Linux | The hard limit on maximum number of signals that can be queued by this process. Unit: None | | `rlimit_signals_pending_soft` | Linux | The soft limit on maximum number of signals that can be queued by this process. Unit: None | | `rlimit_nice_priority_hard` | Linux | The hard limit on the maximum nice priority that can be set by this process. Unit: None | | `rlimit_nice_priority_soft` | Linux | The soft limit on the maximum nice priority that can be set by this process. Unit: None | | `rlimit_num_fds_hard` | Linux | The hard limit on the maximum number of file descriptors that this process can have open. Unit: None | | `rlimit_num_fds_soft` | Linux | The soft limit on the maximum number of file descriptors that this process can have open. Unit: None | | `write_count` | Linux, Windows Server | The number of disk write operations that the process has executed. Unit: None | | `involuntary_context_switches` | Linux | The number of times that the process was involuntarily context-switched. Unit: None | | `voluntary_context_switches` | Linux | The number of times that the process was context-switched voluntarily. Unit: None | | `realtime_priority` | Linux | The current usage of real-time priority for the process. Unit: None | | `nice_priority` | Linux | The current usage of nice priority for the process. Unit: None | | `signals_pending` | Linux | The number of signals pending to be handled by the process. Unit: None | | `rlimit_cpu_time_hard` | Linux | The hard CPU time resource limit for the process. Unit: None | | `rlimit_cpu_time_soft` | Linux | The soft CPU time resource limit for the process. Unit: None | | `rlimit_file_locks_hard` | Linux | The hard file locks resource limit for the process. Unit: None | | `rlimit_file_locks_soft` | Linux | The soft file locks resource limit for the process. Unit: None | | `rlimit_memory_data_hard` | Linux | The hard resource limit on the process for memory used for data. Unit: Bytes | | `rlimit_memory_data_soft` | Linux | The soft resource limit on the process for memory used for data. Unit: Bytes | | `rlimit_memory_locked_hard` | Linux | The hard resource limit on the process for locked memory. Unit: Bytes | | `rlimit_memory_locked_soft` | Linux | The soft resource limit on the process for locked memory. Unit: Bytes | | `rlimit_memory_rss_hard` | Linux | The hard resource limit on the process for physical memory. Unit: Bytes | | `rlimit_memory_rss_soft` | Linux | The soft resource limit on the process for physical memory. Unit: Bytes | | `rlimit_memory_stack_hard` | Linux | The hard resource limit on the process stack. Unit: Bytes | | `rlimit_memory_stack_soft` | Linux | The soft resource limit on the process stack. Unit: Bytes | | `rlimit_memory_vms_hard` | Linux | The hard resource limit on the process for virtual memory. Unit: Bytes | | `rlimit_memory_vms_soft` | Linux | The soft resource limit on the process for virtual memory. Unit: Bytes | ## Viewing process metrics imported by the CloudWatch agent After importing process metrics into CloudWatch, you can view these metrics as time series graphs, and create alarms that can watch these metrics and notify you if they breach a threshold that you specify. The following procedure shows how to view process metrics as a time series graph. For more information about setting alarms, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md). **To view process metrics in the CloudWatch console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. Choose the namespace for the metrics collected by the agent. By default, this is **CWAgent**, but you may have specified a different namespace in the CloudWatch agent configuration file. 1. Choose a metric dimension (for example, **Per-Instance Metrics**). 1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following: 1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table. 1. To sort the table, use the column heading. 1. To filter by resource, choose the resource ID and then choose **Add to search**. 1. To filter by metric, choose the metric name and then choose **Add to search**. 1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, **Add to dashboard**. # Retrieve custom metrics with StatsD You can retrieve additional custom metrics from your applications or services using the CloudWatch agent with the `StatsD` protocol. StatsD is a popular open-source solution that can gather metrics from a wide variety of applications. StatsD is especially useful for instrumenting your own metrics. For an example of using the CloudWatch agent and StatsD together, see [ How to better monitor your custom application metrics using Amazon CloudWatch Agent](https://aws.amazon.com/blogs/devops/new-how-to-better-monitor-your-custom-application-metrics-using-amazon-cloudwatch-agent/). `StatsD` is supported on both Linux servers and servers running Windows Server. CloudWatch supports the following `StatsD` format: ``` MetricName:value|type|@sample_rate|#tag1: value,tag1... ``` + `MetricName` – A string with no colons, bars, \$1 characters, or @ characters. + `value` – This can be either integer or float. + `type` – Specify `c` for counter, `g` for gauge, `ms` for timer, `h` for histogram, or `s` for set. + `sample_rate` – (Optional) A float between 0 and 1, inclusive. Use only for counter, histogram, and timer metrics. The default value is 1 (sampling 100% of the time). + `tags` – (Optional) A comma-separated list of tags. `StatsD` tags are similar to dimensions in CloudWatch. Use colons for key/value tags, such as `env:prod`. You can use any `StatsD` client that follows this format to send the metrics to the CloudWatch agent. For more information about some of the available `StatsD` clients, see the [StatsD client page on GitHub](https://github.com/etsy/statsd/wiki#client-implementations). To collect these custom metrics, add a `"statsd": {}` line to the `metrics_collected` section of the agent configuration file. You can add this line manually. If you use the wizard to create the configuration file, it's done for you. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md). The `StatsD` default configuration works for most users. There are optional fields that you can add to the **statsd** section of the agent configuration file as needed: + `service_address` – The service address to which the CloudWatch agent should listen. The format is `ip:port`. If you omit the IP address, the agent listens on all available interfaces. Only the UDP format is supported, so you don't need to specify a UDP prefix. The default value is `:8125`. + `metrics_collection_interval` – How often in seconds that the `StatsD` plugin runs and collects metrics. The default value is 10 seconds. The range is 1–172,000. + `metrics_aggregation_interval` – How often in seconds CloudWatch aggregates metrics into single data points. The default value is 60 seconds. For example, if `metrics_collection_interval` is 10 and `metrics_aggregation_interval` is 60, CloudWatch collects data every 10 seconds. After each minute, the six data readings from that minute are aggregated into a single data point, which is sent to CloudWatch. The range is 0–172,000. Setting `metrics_aggregation_interval` to 0 disables the aggregation of `StatsD` metrics. + `allowed_pending_messages` – The number of UDP messages that are allowed to queue up. When the queue is full, the StatsD server starts dropping packets. The default value is 10000. + `drop_original_metrics` – Optional. If you are using the `aggregation_dimensions` field in the `metrics` section to roll up metrics into aggregated results, then by default the agent sends both the aggregated metrics and the original metrics that are separated for each value of the dimension. If you don't want the original metrics to be sent to CloudWatch, you can specify this parameter with a list of metrics. The metrics specified along with this parameter don't have their metrics by dimension reported to CloudWatch. Instead, only the aggregated metrics are reported. This reduces the number of metrics that the agent collects, reducing your costs. The following is an example of the **statsd** section of the agent configuration file, using the default port and custom collection and aggregation intervals. ``` { "metrics":{ "metrics_collected":{ "statsd":{ "service_address":":8125", "metrics_collection_interval":60, "metrics_aggregation_interval":300 } } } } ``` ## Viewing StatsD metrics imported by the CloudWatch agent After importing StatsD metrics into CloudWatch, you can view these metrics as time series graphs, and create alarms that can watch these metrics and notify you if they breach a threshold that you specify. The following procedure shows how to view StatsD metrics as a time series graph. For more information about setting alarms, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md). **To view StatsD metrics in the CloudWatch console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. Choose the namespace for the metrics collected by the agent. By default, this is **CWAgent**, but you may have specified a different namespace in the CloudWatch agent configuration file. 1. Choose a metric dimension (for example, **Per-Instance Metrics**). 1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following: 1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table. 1. To sort the table, use the column heading. 1. To filter by resource, choose the resource ID and then choose **Add to search**. 1. To filter by metric, choose the metric name and then choose **Add to search**. 1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, **Add to dashboard**. # Retrieve custom metrics with collectd You can retrieve additional metrics from your applications or services using the CloudWatch agent with the collectd protocol, which is supported only on Linux servers. collectd is a popular open-source solution with plugins that can gather system statistics for a wide variety of applications. By combining the system metrics that the CloudWatch agent can already collect with the additional metrics from collectd, you can better monitor, analyze, and troubleshoot your systems and applications. For more information about collectd, see [collectd - The system statistics collection daemon](https://collectd.org/). You use the collectd software to send the metrics to the CloudWatch agent. For the collectd metrics, the CloudWatch agent acts as the server while the collectd plugin acts as the client. The collectd software is not installed automatically on every server. On a server running Amazon Linux 2, follow these steps to install collectd ``` sudo amazon-linux-extras install collectd ``` For information about installing collectd on other systems, see the [Download page for collectd.](https://www.collectd.org/download.html) To collect these custom metrics, add a **"collectd": \$1\$1** line to the **metrics\$1collected** section of the agent configuration file. You can add this line manually. If you use the wizard to create the configuration file, it is done for you. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md). Optional parameters are also available. If you are using collectd and you do not use `/etc/collectd/auth_file` as your **collectd\$1auth\$1file**, you must set some of these options. + **service\$1address:** The service address to which the CloudWatch agent should listen. The format is `"udp://ip:port`. The default is `udp://127.0.0.1:25826`. + **name\$1prefix:** A prefix to attach to the beginning of the name of each collectd metric. The default is `collectd_`. The maximum length is 255 characters. + **collectd\$1security\$1level:** Sets the security level for network communication. The default is **encrypt**. **encrypt** specifies that only encrypted data is accepted. **sign** specifies that only signed and encrypted data is accepted. **none** specifies that all data is accepted. If you specify a value for **collectd\$1auth\$1file**, encrypted data is decrypted if possible. For more information, see [Client setup](https://collectd.org/wiki/index.php/Networking_introduction#Client_setup) and [Possible interactions](https://collectd.org/wiki/index.php/Networking_introduction#Possible_interactions) in the collectd Wiki. + **collectd\$1auth\$1file** Sets a file in which user names are mapped to passwords. These passwords are used to verify signatures and to decrypt encrypted network packets. If given, signed data is verified and encrypted packets are decrypted. Otherwise, signed data is accepted without checking the signature and encrypted data cannot be decrypted. The default is `/etc/collectd/auth_file`. If **collectd\$1security\$1level** is set to **none**, this is optional. If you set **collectd\$1security\$1level** to `encrypt` or **sign**, you must specify **collectd\$1auth\$1file**. For the format of the auth file, each line is a user name followed by a colon and any number of spaces followed by the password. For example: `user1: user1_password` `user2: user2_password` + **collectd\$1typesdb:** A list of one or more files that contain the dataset descriptions. The list must be surrounded by brackets, even if there is just one entry in the list. Each entry in the list must be surrounded by double quotes. If there are multiple entries, separate them with commas. The default on Linux servers is `["/usr/share/collectd/types.db"]`. The default on macOs computers depends on the version of collectd. For example, `["/usr/local/Cellar/collectd/5.12.0/share/collectd/types.db"]`. For more information, see [https://www.collectd.org/documentation/manpages/types.db.html](https://www.collectd.org/documentation/manpages/types.db.html). + **metrics\$1aggregation\$1interval:** How often in seconds CloudWatch aggregates metrics into single data points. The default is 60 seconds. The range is 0 to 172,000. Setting it to 0 disables the aggregation of collectd metrics. The following is an example of the collectd section of the agent configuration file. ``` { "metrics":{ "metrics_collected":{ "collectd":{ "name_prefix":"My_collectd_metrics_", "metrics_aggregation_interval":120 } } } } ``` ## Viewing collected metrics imported by the CloudWatch agent After importing collectd metrics into CloudWatch, you can view these metrics as time series graphs, and create alarms that can watch these metrics and notify you if they breach a threshold that you specify. The following procedure shows how to view collectd metrics as a time series graph. For more information about setting alarms, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md). **To view collectd metrics in the CloudWatch console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**. 1. Choose the namespace for the metrics collected by the agent. By default, this is **CWAgent**, but you may have specified a different namespace in the CloudWatch agent configuration file. 1. Choose a metric dimension (for example, **Per-Instance Metrics**). 1. The **All metrics** tab displays all metrics for that dimension in the namespace. You can do the following: 1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table. 1. To sort the table, use the column heading. 1. To filter by resource, choose the resource ID and then choose **Add to search**. 1. To filter by metric, choose the metric name and then choose **Add to search**. 1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, **Add to dashboard**. # Set up and configure Prometheus metrics collection on Amazon EC2 instances The following sections explain how to install the CloudWatch agent with Prometheus monitoring on EC2 instances, and how to configure the agent to scrape additional targets. It also provides tutorials for setting up sample workloads to use testing with Prometheus monitoring. Both Linux and Windows instances are supported. For information about the operating systems supported by the CloudWatch agent, see [Collect metrics, logs, and traces using the CloudWatch agent](Install-CloudWatch-Agent.md) **VPC security group requirements** If you are using a VPC, the following requirements apply. + The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP. + The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP. **Topics** + [ ## Step 1: Install the CloudWatch agent ](#CloudWatch-Agent-PrometheusEC2-install) + [ ## Step 2: Scrape Prometheus sources and import metrics ](#CloudWatch-Agent-PrometheusEC2-configure) + [ ## Example: Set up Java/JMX sample workloads for Prometheus metric testing ](#CloudWatch-Agent-Prometheus-Java) ## Step 1: Install the CloudWatch agent The first step is to install the CloudWatch agent on the EC2 instance. For instructions, see [Installing the CloudWatch agent](install-CloudWatch-Agent-on-EC2-Instance.md). ## Step 2: Scrape Prometheus sources and import metrics The CloudWatch agent with Prometheus monitoring needs two configurations to scrape the Prometheus metrics. One is for the standard Prometheus configurations as documented in [ ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)in the Prometheus documentation. The other is for the CloudWatch agent configuration. ### Prometheus scrape configuration The CloudWatch agent supports the standard Prometheus scrape configurations as documented in [ ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) in the Prometheus documentation. You can edit this section to update the configurations that are already in this file, and add additional Prometheus scraping targets. A sample configuration file contains the following global configuration lines: ``` PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus.yaml global: scrape_interval: 1m scrape_timeout: 10s scrape_configs: - job_name: MY_JOB sample_limit: 10000 file_sd_configs: - files: ["C:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\prometheus_sd_1.yaml", "C:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\prometheus_sd_2.yaml"] ``` The `global` section specifies parameters that are valid in all configuration contexts. They also serve as defaults for other configuration sections. It contains the following parameters: + `scrape_interval`— Defines how frequently to scrape targets. + `scrape_timeout`— Defines how long to wait before a scrape request times out. The `scrape_configs` section specifies a set of targets and parameters that define how to scrape them. It contains the following parameters: + `job_name`— The job name assigned to scraped metrics by default. + `sample_limit`— Per-scrape limit on the number of scraped samples that will be accepted. + `file_sd_configs`— List of file service discovery configurations. It reads a set of files containing a list of zero or more static configs. The `file_sd_configs` section contains a `files` parameter which defines patterns for files from which target groups are extracted. The CloudWatch agent supports the following service discovery configuration types. **`static_config`** Allows specifying a list of targets and a common label set for them. It is the canonical way to specify static targets in a scrape configuration. The following is a sample static config to scrape Prometheus metrics from a local host. Metrics can also be scraped from other servers if the Prometheus port is open to the server where the agent runs. ``` PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus_sd_1.yaml - targets: - 127.0.0.1:9404 labels: key1: value1 key2: value2 ``` This example contains the following parameters: + `targets`— The targets scraped by the static config. + `labels`— Labels assigned to all metrics that are scraped from the targets. **`ec2_sd_config`** Allows retrieving scrape targets from Amazon EC2 instances. The following is a sample `ec2_sd_config` to scrape Prometheus metrics from a list of EC2 instances. The Prometheus ports of these instances have to open to the server where the CloudWatch agent runs. The IAM role for the EC2 instance where the CloudWatch agent runs must include the `ec2:DescribeInstance` permission. For example, you could attach the managed policy **AmazonEC2ReadOnlyAccess** to the instance running the CloudWatch agent. ``` PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus.yaml global: scrape_interval: 1m scrape_timeout: 10s scrape_configs: - job_name: MY_JOB sample_limit: 10000 ec2_sd_configs: - region: us-east-1 port: 9404 filters: - name: instance-id values: - i-98765432109876543 - i-12345678901234567 ``` This example contains the following parameters: + `region`— The AWS Region where the target EC2 instance is. If you leave this blank, the Region from the instance metadata is used. + `port`— The port to scrape metrics from. + `filters`— Optional filters to use to filter the instance list. This example filters based on EC2 instance IDs. For more criteria that you can filter on, see [ DescribeInstances](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html). ### CloudWatch agent configuration for Prometheus The CloudWatch agent configuration file includes `prometheus` sections under both `logs` and `metrics_collected`. It includes the following parameters. + **cluster\$1name**— specifies the cluster name to be added as a label in the log event. This field is optional. + **log\$1group\$1name**— specifies the log group name for the scraped Prometheus metrics. + **prometheus\$1config\$1path**— specifies the Prometheus scrape configuration file path. + **emf\$1processor**— specifies the embedded metric format processor configuration. For more information about embedded metric format, see [Embedding metrics within logs](CloudWatch_Embedded_Metric_Format.md). The `emf_processor` section can contain the following parameters: + **metric\$1declaration\$1dedup**— It set to true, the de-duplication function for the embedded metric format metrics is enabled. + **metric\$1namespace**— Specifies the metric namespace for the emitted CloudWatch metrics. + **metric\$1unit**— Specifies the metric name:metric unit map. For information about supported metric units, see [ MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). + **metric\$1declaration**— are sections that specify the array of logs with embedded metric format to be generated. There are `metric_declaration` sections for each Prometheus source that the CloudWatch agent imports from by default. These sections each include the following fields: + `source_labels` specifies the value of the labels that are checked by the `label_matcher` line. + `label_matcher` is a regular expression that checks the value of the labels listed in `source_labels`. The metrics that match are enabled for inclusion in the embedded metric format sent to CloudWatch. + `metric_selectors` is a regular expression that specifies the metrics to be collected and sent to CloudWatch. + `dimensions` is the list of labels to be used as CloudWatch dimensions for each selected metric. The following is an example CloudWatch agent configuration for Prometheus. ``` { "logs":{ "metrics_collected":{ "prometheus":{ "cluster_name":"prometheus-cluster", "log_group_name":"Prometheus", "prometheus_config_path":"C:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\prometheus.yaml", "emf_processor":{ "metric_declaration_dedup":true, "metric_namespace":"CWAgent-Prometheus", "metric_unit":{ "jvm_threads_current": "Count", "jvm_gc_collection_seconds_sum": "Milliseconds" }, "metric_declaration":[ { "source_labels":[ "job", "key2" ], "label_matcher":"MY_JOB;^value2", "dimensions":[ [ "key1", "key2" ], [ "key2" ] ], "metric_selectors":[ "^jvm_threads_current$", "^jvm_gc_collection_seconds_sum$" ] } ] } } } } } ``` The previous example configures an embedded metric format section to be sent as a log event if the following conditions are met: + The value of the label `job` is `MY_JOB` + The value of the label `key2` is `value2` + The Prometheus metrics `jvm_threads_current` and `jvm_gc_collection_seconds_sum` contains both `job` and `key2` labels. The log event that is sent includes the following highlighted section. ``` { "CloudWatchMetrics": [ { "Metrics": [ { "Unit": "Count", "Name": "jvm_threads_current" }, { "Unit": "Milliseconds", "Name": "jvm_gc_collection_seconds_sum" } ], "Dimensions": [ [ "key1", "key2" ], [ "key2" ] ], "Namespace": "CWAgent-Prometheus" } ], "ClusterName": "prometheus-cluster", "InstanceId": "i-0e45bd06f196096c8", "Timestamp": "1607966368109", "Version": "0", "host": "EC2AMAZ-PDDOIUM", "instance": "127.0.0.1:9404", "jvm_threads_current": 2, "jvm_gc_collection_seconds_sum": 0.006000000000000002, "prom_metric_type": "gauge", ... } ``` ## Example: Set up Java/JMX sample workloads for Prometheus metric testing JMX Exporter is an official Prometheus exporter that can scrape and expose JMX mBeans as Prometheus metrics. For more information, see [prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter). The CloudWatch agent can collect predefined Prometheus metrics from Java Virtual Machine (JVM), Hjava, and Tomcat (Catalina), from a JMX exporter on EC2 instances. ### Step 1: Install the CloudWatch agent The first step is to install the CloudWatch agent on the EC2 instance. For instructions, see [Installing the CloudWatch agent](install-CloudWatch-Agent-on-EC2-Instance.md). ### Step 2: Start the Java/JMX workload The next step is to start the Java/JMX workload. First, download the latest JMX exporter jar file from the following location: [prometheus/jmx\$1exporter](https://github.com/prometheus/jmx_exporter). **Use the jar for your sample application** The example commands in the following sections use `SampleJavaApplication-1.0-SNAPSHOT.jar` as the jar file. Replace these parts of the commands with the jar for your application. #### Prepare the JMX exporter configuration The `config.yaml` file is the JMX exporter configuration file. For more information, see [Configuration](https://github.com/prometheus/jmx_exporter#Configuration) in the JMX exporter documentation. Here is a sample configuration for Java and Tomcat. ``` --- lowercaseOutputName: true lowercaseOutputLabelNames: true rules: - pattern: 'java.lang<>(FreePhysicalMemorySize|TotalPhysicalMemorySize|FreeSwapSpaceSize|TotalSwapSpaceSize|SystemCpuLoad|ProcessCpuLoad|OpenFileDescriptorCount|AvailableProcessors)' name: java_lang_OperatingSystem_$1 type: GAUGE - pattern: 'java.lang<>(TotalStartedThreadCount|ThreadCount)' name: java_lang_threading_$1 type: GAUGE - pattern: 'Catalina<>(\w+)' name: catalina_globalrequestprocessor_$3_total labels: port: "$2" protocol: "$1" help: Catalina global $3 type: COUNTER - pattern: 'Catalina<>(requestCount|maxTime|processingTime|errorCount)' name: catalina_servlet_$3_total labels: module: "$1" servlet: "$2" help: Catalina servlet $3 total type: COUNTER - pattern: 'Catalina<>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount)' name: catalina_threadpool_$3 labels: port: "$2" protocol: "$1" help: Catalina threadpool $3 type: GAUGE - pattern: 'Catalina<>(processingTime|sessionCounter|rejectedSessions|expiredSessions)' name: catalina_session_$3_total labels: context: "$2" host: "$1" help: Catalina session $3 total type: COUNTER - pattern: ".*" ``` #### Start the Java application with the Prometheus exporter Start the sample application. This will emit Prometheus metrics to port 9404. Be sure to replace the entry point `com.gubupt.sample.app.App` with the correct information for your sample java application. On Linux, enter the following command. ``` $ nohup java -javaagent:./jmx_prometheus_javaagent-0.14.0.jar=9404:./config.yaml -cp ./SampleJavaApplication-1.0-SNAPSHOT.jar com.gubupt.sample.app.App & ``` On Windows, enter the following command. ``` PS C:\> java -javaagent:.\jmx_prometheus_javaagent-0.14.0.jar=9404:.\config.yaml -cp .\SampleJavaApplication-1.0-SNAPSHOT.jar com.gubupt.sample.app.App ``` #### Verify the Prometheus metrics emission Verify that Prometheus metrics are being emitted. On Linux, enter the following command. ``` $ curl localhost:9404 ``` On Windows, enter the following command. ``` PS C:\> curl http://localhost:9404 ``` Sample output on Linux: ``` StatusCode : 200 StatusDescription : OK Content : # HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM # TYPE jvm_classes_loaded gauge jvm_classes_loaded 2526.0 # HELP jvm_classes_loaded_total The total number of class... RawContent : HTTP/1.1 200 OK Content-Length: 71908 Content-Type: text/plain; version=0.0.4; charset=utf-8 Date: Fri, 18 Dec 2020 16:38:10 GMT # HELP jvm_classes_loaded The number of classes that are currentl... Forms : {} Headers : {[Content-Length, 71908], [Content-Type, text/plain; version=0.0.4; charset=utf-8], [Date, Fri, 18 Dec 2020 16:38:10 GMT]} Images : {} InputFields : {} Links : {} ParsedHtml : System.__ComObject RawContentLength : 71908 ``` ### Step 3: Configure the CloudWatch agent to scrape Prometheus metrics Next, set up the Prometheus scrape configuration in the CloudWatch agent configuration file. **To set up the Prometheus scrape configuration for the Java/JMX example** 1. Set up the configuration for `file_sd_config` and `static_config`. On Linux, enter the following command. ``` $ cat /opt/aws/amazon-cloudwatch-agent/var/prometheus.yaml global: scrape_interval: 1m scrape_timeout: 10s scrape_configs: - job_name: jmx sample_limit: 10000 file_sd_configs: - files: [ "/opt/aws/amazon-cloudwatch-agent/var/prometheus_file_sd.yaml" ] ``` On Windows, enter the following command. ``` PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus.yaml global: scrape_interval: 1m scrape_timeout: 10s scrape_configs: - job_name: jmx sample_limit: 10000 file_sd_configs: - files: [ "C:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\prometheus_file_sd.yaml" ] ``` 1. Set up the scrape targets configuration. On Linux, enter the following command. ``` $ cat /opt/aws/amazon-cloudwatch-agent/var/prometheus_file_sd.yaml - targets: - 127.0.0.1:9404 labels: application: sample_java_app os: linux ``` On Windows, enter the following command. ``` PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus_file_sd.yaml - targets: - 127.0.0.1:9404 labels: application: sample_java_app os: windows ``` 1. Set up the Prometheus scrape configuration by `ec2_sc_config`. Replace *your-ec2-instance-id* with the correct EC2 instance ID. On Linux, enter the following command. ``` $ cat .\prometheus.yaml global: scrape_interval: 1m scrape_timeout: 10s scrape_configs: - job_name: jmx sample_limit: 10000 ec2_sd_configs: - region: us-east-1 port: 9404 filters: - name: instance-id values: - your-ec2-instance-id ``` On Windows, enter the following command. ``` PS C:\ProgramData\Amazon\AmazonCloudWatchAgent> cat prometheus_file_sd.yaml - targets: - 127.0.0.1:9404 labels: application: sample_java_app os: windows ``` 1. Set up the CloudWatch agent configuration. First, navigate to the correct directory. On Linux, it is `/opt/aws/amazon-cloudwatch-agent/var/cwagent-config.json`. On Windows, it is `C:\ProgramData\Amazon\AmazonCloudWatchAgent\cwagent-config.json`. The following is a sample configuration with Java/JHX Prometheus metrics defined. Be sure to replace *path-to-Prometheus-Scrape-Configuration-file* with the correct path. ``` { "agent": { "region": "us-east-1" }, "logs": { "metrics_collected": { "prometheus": { "cluster_name": "my-cluster", "log_group_name": "prometheus-test", "prometheus_config_path": "path-to-Prometheus-Scrape-Configuration-file", "emf_processor": { "metric_declaration_dedup": true, "metric_namespace": "PrometheusTest", "metric_unit":{ "jvm_threads_current": "Count", "jvm_classes_loaded": "Count", "java_lang_operatingsystem_freephysicalmemorysize": "Bytes", "catalina_manager_activesessions": "Count", "jvm_gc_collection_seconds_sum": "Seconds", "catalina_globalrequestprocessor_bytesreceived": "Bytes", "jvm_memory_bytes_used": "Bytes", "jvm_memory_pool_bytes_used": "Bytes" }, "metric_declaration": [ { "source_labels": ["job"], "label_matcher": "^jmx$", "dimensions": [["instance"]], "metric_selectors": [ "^jvm_threads_current$", "^jvm_classes_loaded$", "^java_lang_operatingsystem_freephysicalmemorysize$", "^catalina_manager_activesessions$", "^jvm_gc_collection_seconds_sum$", "^catalina_globalrequestprocessor_bytesreceived$" ] }, { "source_labels": ["job"], "label_matcher": "^jmx$", "dimensions": [["area"]], "metric_selectors": [ "^jvm_memory_bytes_used$" ] }, { "source_labels": ["job"], "label_matcher": "^jmx$", "dimensions": [["pool"]], "metric_selectors": [ "^jvm_memory_pool_bytes_used$" ] } ] } } }, "force_flush_interval": 5 } } ``` 1. Restart the CloudWatch agent by entering one of the following commands. On Linux, enter the following command. ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/var/cwagent-config.json ``` On Windows, enter the following command. ``` & "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:C:\ProgramData\Amazon\AmazonCloudWatchAgent\cwagent-config.json ``` ### Viewing the Prometheus metrics and logs You can now view the Java/JMX metrics being collected. **To view the metrics for your sample Java/JMX workload** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the Region where your cluster is running, choose **Metrics** in the left navigation pane. Find the **PrometheusTest** namespace to see the metrics. 1. To see the CloudWatch Logs events, choose **Log groups** in the navigation pane. The events are in the log group **prometheus-test**. # Configure CloudWatch agent service and environment names for related entities The CloudWatch agent can send metrics and logs with entity data to support the [Explore related pane](ExploreRelated.md) in the CloudWatch console. The service name or environment name can be configured by the [CloudWatch Agent JSON configuration](CloudWatch-Agent-Configuration-File-Details.md). **Note** The agent configuration may be overridden. For details about how the agent decides what data to send for related entities, see [Using the CloudWatch agent with related telemetry](CloudWatch-Agent-RelatedEntities.md). For metrics, it can be configured at the agent, metrics, or plugin level. For logs it can be configured at the agent, logs, or file level. The most specific configuration is always used. For example if the configuration exists at the agent level and metrics level, then metrics will use the metric configuration, and anything else (logs) will use the agent configuration. The following example shows different ways to configure the service name and environment name. ``` { "agent": { "service.name": "agent-level-service", "deployment.environment": "agent-level-environment" }, "metrics": { "service.name": "metric-level-service", "deployment.environment": "metric-level-environment", "metrics_collected": { "statsd": { "service.name": "statsd-level-service", "deployment.environment": "statsd-level-environment", }, "collectd": { "service.name": "collectdd-level-service", "deployment.environment": "collectd-level-environment", } } }, "logs": { "service.name": "log-level-service", "deployment.environment": "log-level-environment", "logs_collected": { "files": { "collect_list": [ { "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log", "log_group_name": "amazon-cloudwatch-agent.log", "log_stream_name": "amazon-cloudwatch-agent.log", "service.name": "file-level-service", "deployment.environment": "file-level-environment" } ] } } } } ``` # Starting the CloudWatch agent You can start the CloudWatch agent using either Systems Manager Run Command or the command line. For information about setting up the agent on a system that has security-enhanced Linux (SELinux) enabled, see [Set up the CloudWatch agent with security-enhanced Linux (SELinux)](CloudWatch-Agent-SELinux.md). ## Start the CloudWatch agent using the command line on Amazon EC2 Follow these steps to use the command line to start the CloudWatch agent on on Amazon EC2. For information about setting up the agent on a system that has security-enhanced Linux (SELinux) enabled, see [Set up the CloudWatch agent with security-enhanced Linux (SELinux)](CloudWatch-Agent-SELinux.md). **To use the command line to start the CloudWatch agent on on Amazon EC2** 1. Copy the agent configuration file that you want to use to the server where you're going to run the agent. Note the pathname where you copy it to. 1. In this command, `-a fetch-config` causes the agent to load the latest version of the CloudWatch agent configuration file, and `-s` starts the agent. Enter one of the following commands. Replace *configuration-file-path* with the path to the agent configuration file. This file is called `config.json` if you created it with the wizard, and might be called `amazon-cloudwatch-agent.json` if you created it manually. On an EC2 instance running Linux, enter the following command. ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path ``` On an on-premises server running Linux, enter the following: ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -s -c file:configuration-file-path ``` On an EC2 instance running Windows Server, enter the following from the PowerShell console: ``` & "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:configuration-file-path ``` On an on-premises server running Windows Server, enter the following from the PowerShell console: ``` & "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m onPremise -s -c file:configuration-file-path ``` ## Start the CloudWatch agent on an on-premises server Follow these steps to start the CloudWatch agent on an on-premises server. **To use SSM Agent to start the CloudWatch agent on an on-premises server** 1. Open the Systems Manager console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/). 1. In the navigation pane, choose **Run Command**. -or- If the AWS Systems Manager home page opens, scroll down and choose **Explore Run Command**. 1. Choose **Run command**. 1. In the **Command document** list, select the button next to **AmazonCloudWatch-ManageAgent**. 1. In the **Targets** area, select the instance where you installed the agent. 1. In the **Action** list, choose **configure**. 1. In the **Mode** list, choose **onPremise**. 1. In the **Optional Configuration Location** box, enter the name of the agent configuration file that you created with the wizard and stored in the Parameter Store. 1. Choose **Run**. The agent starts with the configuration you specified in the configuration file. **To use the command line to start the CloudWatch agent on an on-premises server** + In this command, `-a fetch-config` causes the agent to load the latest version of the CloudWatch agent configuration file, and `-s` starts the agent. Linux: If you saved the configuration file in the Systems Manager Parameter Store, enter the following: ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -s -c ssm:configuration-parameter-store-name ``` Linux: If you saved the configuration file on the local computer, enter the following command. Replace *configuration-file-path* with the path to the agent configuration file. This file is called `config.json` if you created it with the wizard, and might be called `amazon-cloudwatch-agent.json` if you created it manually. ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -s -c file:configuration-file-path ``` Windows Server: If you saved the agent configuration file in Systems Manager Parameter Store, enter the following from the PowerShell console: ``` & "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m onPremise -s -c ssm:configuration-parameter-store-name ``` Windows Server: If you saved the agent configuration file on the local computer, enter the following from the PowerShell console. Replace *configuration-file-path* with the path to the agent configuration file. This file is called `config.json` if you created it with the wizard, and might be called `amazon-cloudwatch-agent.json` if you created it manually. ``` & "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m onPremise -s -c file:configuration-file-path ``` # Metrics collected by the CloudWatch agent You can collect metrics from servers by installing the CloudWatch agent on the server. You can install the agent on both Amazon EC2 instances and on-premises servers. You can also install the agent on computers running Linux, Windows Server, or macOS. If you install the agent on an Amazon EC2 instance, the metrics the agent collects are in addition to the metrics enabled by default on Amazon EC2 instances. For information about installing the CloudWatch agent on an instance, see [Collect metrics, logs, and traces using the CloudWatch agent](Install-CloudWatch-Agent.md). You can use this section to learn about metrics the CloudWatch agent collects. ## Metrics collected by the CloudWatch agent on Windows Server instances On a server running Windows Server, installing the CloudWatch agent enables you to collect the metrics associated with the counters in Windows Performance Monitor. The CloudWatch metric names for these counters are created by putting a space between the object name and the counter name. For example, the `% Interrupt Time` counter of the `Processor` object is given the metric name `Processor % Interrupt Time` in CloudWatch. For more information about Windows Performance Monitor counters, see the Microsoft Windows Server documentation. The default namespace for metrics collected by the CloudWatch agent is `CWAgent`, although you can specify a different namespace when you configure the agent. ## Metrics collected by the CloudWatch agent on Linux and macOS instances The following table lists the metrics that you can collect with the CloudWatch agent on Linux servers and macOS computers. | Metric | Description | | --- | --- | | `cpu_time_active` | The amount of time that the CPU is active in any capacity. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_guest` | The amount of time that the CPU is running a virtual CPU for a guest operating system. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_guest_nice` | The amount of time that the CPU is running a virtual CPU for a guest operating system, which is low-priority and can be interrupted by other processes. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_idle` | The amount of time that the CPU is idle. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_iowait` | The amount of time that the CPU is waiting for I/O operations to complete. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_irq` | The amount of time that the CPU is servicing interrupts. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_nice` | The amount of time that the CPU is in user mode with low-priority processes, which can easily be interrupted by higher-priority processes. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_softirq` | The amount of time that the CPU is servicing software interrupts. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_steal` | The amount of time that the CPU is in *stolen time*, which is time spent in other operating systems in a virtualized environment. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_system` | The amount of time that the CPU is in system mode. This metric is measured in hundredths of a second. Unit: None | | `cpu_time_user` | The amount of time that the CPU is in user mode. This metric is measured in hundredths of a second. Unit: None | | `cpu_usage_active` | The percentage of time that the CPU is active in any capacity. Unit: Percent | | `cpu_usage_guest` | The percentage of time that the CPU is running a virtual CPU for a guest operating system. Unit: Percent | | `cpu_usage_guest_nice` | The percentage of time that the CPU is running a virtual CPU for a guest operating system, which is low-priority and can be interrupted by other processes. Unit: Percent | | `cpu_usage_idle` | The percentage of time that the CPU is idle. Unit: Percent | | `cpu_usage_iowait` | The percentage of time that the CPU is waiting for I/O operations to complete. Unit: Percent | | `cpu_usage_irq` | The percentage of time that the CPU is servicing interrupts. Unit: Percent | | `cpu_usage_nice` | The percentage of time that the CPU is in user mode with low-priority processes, which higher-priority processes can easily interrupt. Unit: Percent | | `cpu_usage_softirq` | The percentage of time that the CPU is servicing software interrupts. Unit: Percent | | `cpu_usage_steal` | The percentage of time that the CPU is in *stolen time*, or time spent in other operating systems in a virtualized environment. Unit: Percent | | `cpu_usage_system` | The percentage of time that the CPU is in system mode. Unit: Percent | | `cpu_usage_user` | The percentage of time that the CPU is in user mode. Unit: Percent | | `disk_free` | Free space on the disks. Unit: Bytes | | `disk_inodes_free` | The number of available index nodes on the disk. Unit: Count | | `disk_inodes_total` | The total number of index nodes reserved on the disk. Unit: Count | | `disk_inodes_used` | The number of used index nodes on the disk. Unit: Count | | `disk_total` | Total space on the disks, including used and free. Unit: Bytes | | `disk_used` | Used space on the disks. Unit: Bytes | | `disk_used_percent` | The percentage of total disk space that is used. Unit: Percent | | `diskio_iops_in_progress` | The number of I/O requests that have been issued to the device driver but have not yet completed. Unit: Count | | `diskio_io_time` | The amount of time that the disk has had I/O requests queued. Unit: Milliseconds The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `diskio_reads` | The number of disk read operations. Unit: Count The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `diskio_read_bytes` | The number of bytes read from the disks. Unit: Bytes The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `diskio_read_time` | The amount of time that read requests have waited on the disks. Multiple read requests waiting at the same time increase the number. For example, if 5 requests all wait for an average of 100 milliseconds, 500 is reported. Unit: Milliseconds The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `diskio_writes` | The number disk write operations. Unit: Count The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `diskio_write_bytes` | The number of bytes written to the disks. Unit: Bytes The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `diskio_write_time` | The amount of time that write requests have waited on the disks. Multiple write requests waiting at the same time increase the number. For example, if 8 requests all wait for an average of 1000 milliseconds, 8000 is reported. Unit: Milliseconds The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `ethtool_bw_in_allowance_exceeded` | The number of packets queued and/or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](CloudWatch-Agent-network-performance.md) Unit: None | | `ethtool_bw_out_allowance_exceeded` | The number of packets queued and/or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](CloudWatch-Agent-network-performance.md) Unit: None | | `ethtool_conntrack_allowance_exceeded` | The number of packets dropped because connection tracking exceeded the maximum for the instance and new connections could not be established. This can result in packet loss for traffic to or from the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](CloudWatch-Agent-network-performance.md) Unit: None | | `ethtool_linklocal_allowance_exceeded` | The number of packets dropped because the PPS of the traffic to local proxy services exceeded the maximum for the network interface. This impacts traffic to the DNS service, the Instance Metadata Service, and the Amazon Time Sync Service. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](CloudWatch-Agent-network-performance.md) Unit: None | | `ethtool_pps_allowance_exceeded` | The number of packets queued and/or dropped because the bidirectional PPS exceeded the maximum for the instance. This metric is collected only if you have listed it in the `ethtool` subsection of the `metrics_collected` section of the CloudWatch agent configuration file. For more information, see [Collect network performance metrics](CloudWatch-Agent-network-performance.md). Unit: None | | `mem_active` | The amount of memory that has been used in some way during the last sample period. Unit: Bytes | | `mem_available` | The amount of memory that is available and can be given instantly to processes. Unit: Bytes | | `mem_available_percent` | The percentage of memory that is available and can be given instantly to processes. Unit: Percent | | `mem_buffered` | The amount of memory that is being used for buffers. Unit: Bytes | | `mem_cached` | The amount of memory that is being used for file caches. Unit: Bytes | | `mem_free` | The amount of memory that isn't being used. Unit: Bytes | | `mem_inactive` | The amount of memory that hasn't been used in some way during the last sample period Unit: Bytes | | `mem_shared` | The amount of memory that is being shared between processes. Unit: Bytes | | `mem_total` | The total amount of memory. Unit: Bytes | | `mem_used` | The amount of memory currently in use. Unit: Bytes | | `mem_used_percent` | The percentage of memory currently in use. Unit: Percent | | `net_bytes_recv` | The number of bytes received by the network interface. Unit: Bytes The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `net_bytes_sent` | The number of bytes sent by the network interface. Unit: Bytes The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `net_drop_in` | The number of packets received by this network interface that were dropped. Unit: Count The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `net_drop_out` | The number of packets transmitted by this network interface that were dropped. Unit: Count The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `net_err_in` | The number of receive errors detected by this network interface. Unit: Count The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `net_err_out` | The number of transmit errors detected by this network interface. Unit: Count The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `net_packets_sent` | The number of packets sent by this network interface. Unit: Count The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `net_packets_recv` | The number of packets received by this network interface. Unit: Count The only statistic that should be used for this metric is `Sum`. Do not use `Average`. | | `netstat_tcp_close` | The number of TCP connections with no state. Unit: Count | | `netstat_tcp_close_wait` | The number of TCP connections waiting for a termination request from the client. Unit: Count | | `netstat_tcp_closing` | The number of TCP connections that are waiting for a termination request with acknowledgment from the client. Unit: Count | | `netstat_tcp_established` | The number of TCP connections established. Unit: Count | | `netstat_tcp_fin_wait1` | The number of TCP connections in the `FIN_WAIT1` state during the process of closing a connection. Unit: Count | | `netstat_tcp_fin_wait2` | The number of TCP connections in the `FIN_WAIT2` state during the process of closing a connection. Unit: Count | | `netstat_tcp_last_ack` | The number of TCP connections waiting for the client to send acknowledgment of the connection termination message. This is the last state right before the connection is closed down. Unit: Count | | `netstat_tcp_listen` | The number of TCP ports currently listening for a connection request. Unit: Count | | `netstat_tcp_none` | The number of TCP connections with inactive clients. Unit: Count | | `netstat_tcp_syn_sent` | The number of TCP connections waiting for a matching connection request after having sent a connection request. Unit: Count | | `netstat_tcp_syn_recv` | The number of TCP connections waiting for connection request acknowledgment after having sent and received a connection request. Unit: Count | | `netstat_tcp_time_wait` | The number of TCP connections currently waiting to ensure that the client received the acknowledgment of its connection termination request. Unit: Count | | `netstat_udp_socket` | The number of current UDP connections. Unit: Count | | `processes_blocked` | The number of processes that are blocked. Unit: Count | | `processes_dead` | The number of processes that are dead, indicated by the `X` state code on Linux. This metric is not collected on macOS computers. Unit: Count | | `processes_idle` | The number of processes that are idle (sleeping for more than 20 seconds). Available only on FreeBSD instances. Unit: Count | | `processes_paging` | The number of processes that are paging, indicated by the `W` state code on Linux. This metric is not collected on macOS computers. Unit: Count | | `processes_running` | The number of processes that are running, indicated by the `R` state code. Unit: Count | | `processes_sleeping` | The number of processes that are sleeping, indicated by the `S` state code. Unit: Count | | `processes_stopped` | The number of processes that are stopped, indicated by the `T` state code. Unit: Count | | `processes_total` | The total number of processes on the instance. Unit: Count | | `processes_total_threads` | The total number of threads making up the processes. This metric is available only on Linux instances. This metric is not collected on macOS computers. Unit: Count | | `processes_wait` | The number of processes that are paging, indicated by the `W` state code on FreeBSD instances. This metric is available only on FreeBSD instances, and is not available on Linux, Windows Server, or macOS instances. Unit: Count | | `processes_zombies` | The number of zombie processes, indicated by the `Z` state code. Unit: Count | | `swap_free` | The amount of swap space that isn't being used. Unit: Bytes | | `swap_used` | The amount of swap space currently in use. Unit: Bytes | | `swap_used_percent` | The percentage of swap space currently in use. Unit: Percent | ## Definitions of memory metrics collected by the CloudWatch agent When the CloudWatch agent collects memory metrics, the source is the host's memory management subsystem. For example, the Linux kernel exposes OS-maintained data in `/proc`. For memory, the data is in `/proc/meminfo`. Each different operating system and architecture has different calculations of the resources that are used by processes. For more information, see the following sections. During each collection interval, the CloudWatch agent on each instance collects the instance resources and calculates the resources being used by all processes which are running in that instance. This information is reported back to CloudWatch metrics. You can configure the length of the collection interval in the CloudWatch agent configuration file. For more information, see [CloudWatch agent configuration file: Agent section](CloudWatch-Agent-Configuration-File-Details.md#CloudWatch-Agent-Configuration-File-Agentsection). The following list explains how the memory metrics that the CloudWatch agent collects are defined. + **Active Memory**– Memory that is being used by a process. In other words, the memory used by current running apps. + **Available Memory**– The memory that can be instantly given to the processes without the system going into swap (also known as virtual memory). + **Buffer Memory**– The data area shared by hardware devices or program processes that operate at different speeds and priorities. + **Cached Memory**– Stores program instructions and data that are used repeatedly in the operation of programs that the CPU is likely to need next. + **Free Memory**– Memory that is not being used at all and is readily available. It is completely free for the system to be used when needed. + **Inactive Memory**– Pages that have not been accessed "recently". + **Total Memory**– The size of the actual physical memory RAM. + **Used Memory**– Memory that is currently in use by programs and processes. **Topics** + [ ### Linux: Metrics collected and calculations used ](#CloudWatch-agent-metrics-definitions-calculations) + [ ### macOS: Metrics collected and calculations used ](#CloudWatch-agent-metrics-definitions-calculations) + [ ### Windows: Metrics collected ](#CloudWatch-agent-metrics-definitions-calculations) + [ ### Example: Calculating memory metrics on Linux ](#CloudWatch-agent-metrics-definitions-LinuxExample) ### Linux: Metrics collected and calculations used Metrics collected and units: + Active (Bytes) + Available (Bytes) + Available Percent (Percent) + Buffered (Bytes) + Cached (Bytes) + Free (Bytes) + Inactive (Bytes) + Total (Bytes) + Used (Bytes) + Used Percent (Percent) **Used memory** = Total Memory - Free Memory - Cached memory - Buffer memory **Total memory** = Used Memory \$1 Free Memory \$1 Cached memory \$1 Buffer memory ### macOS: Metrics collected and calculations used Metrics collected and units: + Active (Bytes) + Available (Bytes) + Available Percent (Percent) + Free (Bytes) + Inactive (Bytes) + Total (Bytes) + Used (Bytes) + Used Percent (Percent) **Available memory** = Free Memory \$1 Inactive memory **Used memory** = Total Memory - Available memory **Total memory** = Available Memory \$1 Used Memory ### Windows: Metrics collected The metrics collected on Windows hosts are listed below. All of these metrics have `None` for `Unit`. + Available bytes + Cache Faults/sec + Page Faults/sec + Pages/sec There are no calculations used for Windows metrics because the CloudWatch agent parses events from performance counters. ### Example: Calculating memory metrics on Linux As an example, suppose that entering the **cat /proc/meminfo** command on a Linux host shows the following results: ``` MemTotal: 3824388 kB MemFree: 462704 kB MemAvailable: 2157328 kB Buffers: 126268 kB Cached: 1560520 kB SReclaimable: 289080 kB> ``` In this example, the CloudWatch agent will collect the following values. All the values that the CloudWatch agent collects and reports are in bytes. + `mem_total`: 3916173312 bytes + `mem_available`: 2209103872 bytes (MemFree \$1 Cached) + `mem_free`: 473808896 bytes + `mem_cached`: 1893990400 bytes (`cached` \$1 `SReclaimable` + `mem_used`: 1419075584 bytes (`MemTotal` – (`MemFree` \$1 `Buffers` \$1 (`Cached` \$1 `SReclaimable`))) + `mem_buffered`: 129667072 bytes + `mem_available_percent`: 56.41% + `mem_used_percent`: 36.24% (`mem_used` / `mem_total`) \$1 100 # Using the CloudWatch agent with related telemetry Metrics and logs that are sent to CloudWatch can include an optional entity to correlate telemetry. Entities are used in the [Explore related](ExploreRelated.md) pane. The CloudWatch agent sends entities with a service name and environment name included. The agent chooses the service name and environment name from the following data. **Service name** The agent chooses the service name from the following options, in priority order: + **Application Signals instrumentation** – The agent sends the service name used by Application Signals. This can be overwritten by changing the `OTEL_SERVICE_NAME` environment variable used by supported OpenTelemetry instrumentation libraries. + **CloudWatch agent configuration** – You can [configure the agent](CloudWatch-Agent-configure-related-telemetry.md) to use a specific service name. This can be configured at the agent, plugin, metrics, logs, or log file level. + **Kubernetes workload name** – For Kubernetes workloads, the agent sends the name of the workload for the corresponding pod, in the following priority order. + Deployment name + ReplicaSet name + StatefulSet name + DaemonSet name + CronJob name + Job name + Pod name + Container name + **Resource tags from instance metadata** – For Amazon EC2 workloads, the agent sends the a name from tags, in the following order. + service + application + app You must [setup instance metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html#allow-access-to-tags-in-IMDS) for the agent to be able to access tags. + **Default** – If no other service name is found, the agent will send the name `Unknown`. **Environment name** The agent chooses the environment name from the following options, in priority order: + **Application Signals instrumentation** – The agent sends the environment name used by Application Signals. This can be overwritten by setting a `deployment.environment` environment variable used by supported OpenTelemetry instrumentation libraries. For example, applications may set the environment variable `OTEL_RESOURCE_ATTRIBUTES=deployment.environment=MyEnvironment`. + **CloudWatch agent configuration** – You can [configure the agent](CloudWatch-Agent-configure-related-telemetry.md) to use a specific environment name. This can be configured at the agent, plugin, metrics, logs, or log file level. + **Cluster name and workspace** – For Amazon EKS, `eks:cluster-name/Namespace`. For native Kubernetes running on Amazon EC2, `k8s:cluster-name/Namespace`. + **Resource tags from instance metadata** – For Amazon EC2 workloads, the agent can will use the `AutoScalingGroup` tag. You must [setup instance metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html#allow-access-to-tags-in-IMDS) for the agent to be able to access tags. + By default, Amazon EC2 instances that aren't running Kubernetes will get the environment name `ec2:default`. # Common scenarios with the CloudWatch agent This section provides you with different scenarios that outline how to complete common configuration and customization tasks for the CloudWatch agent. **Topics** + [ ## Running the CloudWatch agent as a different user ](#CloudWatch-Agent-run-as-user) + [ ## How the CloudWatch agent handles sparse log files ](#CloudWatch-Agent-sparse-log-files) + [ ## Adding custom dimensions to metrics collected by the CloudWatch agent ](#CloudWatch-Agent-adding-custom-dimensions) + [ ## Aggregating or rolling up metrics collected by the CloudWatch agent ](#CloudWatch-Agent-aggregating-metrics) + [ ## Collecting high-resolution metrics with the CloudWatch agent ](#CloudWatch-Agent-collect-high-resolution-metrics) + [ ## Sending metrics, logs, and traces to a different account ](#CloudWatch-Agent-send-to-different-AWS-account) + [ ## Timestamp differences between the CloudWatch agent and the earlier CloudWatch Logs agent ](#CloudWatch-Agent-logs-timestamp-differences) + [ ## Appending OpenTelemetry collector configuration files ](#CloudWatch-Agent-appending-OpenTelemetry-config-files) ## Running the CloudWatch agent as a different user On Linux servers, the CloudWatch runs as the root user by default. To have the agent run as a different user, use the `run_as_user` parameter in the `agent` section in the CloudWatch agent configuration file. This option is available only on Linux servers. If you're already running the agent with the root user and want to change to using a different user, use one of the following procedures. **To run the CloudWatch agent as a different user on an EC2 instance running Linux** 1. Download and install a new CloudWatch agent package. 1. Create a new Linux user or use the default user named `cwagent` that the RPM or DEB file created. 1. Provide credentials for this user in one of these ways: + If the file `.aws/credentials` exists in the home directory of the root user, you must create a credentials file for the user you are going to use to run the CloudWatch agent. This credentials file will be `/home/username/.aws/credentials`. Then set the value of the `shared_credential_file` parameter in `common-config.toml` to the pathname of the credential file. For more information, see [Install the CloudWatch agent using AWS Systems Manager](installing-cloudwatch-agent-ssm.md). + If the file `.aws/credentials` does not exist in the home directory of the root user, you can do one of the following: + Create a credentials file for the user you are going to use to run the CloudWatch agent. This credentials file will be `/home/username/.aws/credentials`. Then set the value of the `shared_credential_file` parameter in `common-config.toml` to the pathname of the credential file. For more information, see [Install the CloudWatch agent using AWS Systems Manager](installing-cloudwatch-agent-ssm.md). + Instead of creating a credentials file, attach an IAM role to the instance. The agent uses this role as the credential provider. 1. In the CloudWatch agent configuration file, add the following line in the `agent` section: ``` "run_as_user": "username" ``` Make other modifications to the configuration file as needed. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md) 1. Give the user the required permissions. The user must have Read (r) permissions for the log files to be collected, and must have Execute (x) permission on every directory in the log files' path. 1. Start the agent with the configuration file that you just modified. ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path ``` **To run the CloudWatch agent as a different user on an on-premises server running Linux** 1. Download and install a new CloudWatch agent package. 1. Create a new Linux user or use the default user named `cwagent` that the RPM or DEB file created. 1. Store the credentials of this user to a path that the user can access, such as `/home/username/.aws/credentials`. 1. Set the value of the `shared_credential_file` parameter in `common-config.toml` to the pathname of the credential file. For more information, see [Install the CloudWatch agent using AWS Systems Manager](installing-cloudwatch-agent-ssm.md). 1. In the CloudWatch agent configuration file, add the following line in the `agent` section: ``` "run_as_user": "username" ``` Make other modifications to the configuration file as needed. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md) 1. Give the user required permissions. The user must have Read (r) permissions for the log files to be collected, and must have Execute (x) permission on every directory in the log files' path. 1. Start the agent with the configuration file that you just modified. ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path ``` ## How the CloudWatch agent handles sparse log files Sparse files are files with both empty blocks and real contents. A sparse file uses disk space more efficiently by writing brief information representing the empty blocks to disk instead of the actual null bytes which makes up the block. This makes the actual size of a sparse file usually much smaller than its apparent size. However, the CloudWatch agent doesn’t treat sparse files differently than it treats normal files. When the agent reads a sparse file, the empty blocks are treated as "real" blocks filled with null bytes. Because of this, the CloudWatch agent publishes as many bytes as the apparent size of a sparse file to CloudWatch. Configuring the CloudWatch agent to publish a sparse file can cause higher than expected CloudWatch costs, so we recommend not to do so. For example, `/var/logs/lastlog` in Linux is usually a very sparse file, and we recommend that you don't publish it to CloudWatch. ## Adding custom dimensions to metrics collected by the CloudWatch agent To add custom dimensions such as tags to metrics collected by the agent, add the `append_dimensions` field to the section of the agent configuration file that lists those metrics. For example, the following example section of the configuration file adds a custom dimension named `stackName` with a value of `Prod` to the `cpu` and `disk` metrics collected by the agent. ``` "cpu":{ "resources":[ "*" ], "measurement":[ "cpu_usage_guest", "cpu_usage_nice", "cpu_usage_idle" ], "totalcpu":false, "append_dimensions":{ "stackName":"Prod" } }, "disk":{ "resources":[ "/", "/tmp" ], "measurement":[ "total", "used" ], "append_dimensions":{ "stackName":"Prod" } } ``` Remember that any time you change the agent configuration file, you must restart the agent to have the changes take effect. ## Aggregating or rolling up metrics collected by the CloudWatch agent To aggregate or roll up metrics collected by the agent, add an `aggregation_dimensions` field to the section for that metric in the agent configuration file. For example, the following configuration file snippet rolls up metrics on the `AutoScalingGroupName` dimension. The metrics from all instances in each Auto Scaling group are aggregated and can be viewed as a whole. ``` "metrics": { "cpu":{...} "disk":{...} "aggregation_dimensions" : [["AutoScalingGroupName"]] } ``` To roll up along the combination of each `InstanceId` and `InstanceType` dimensions in addition to rolling up on the Auto Scaling group name, add the following. ``` "metrics": { "cpu":{...} "disk":{...} "aggregation_dimensions" : [["AutoScalingGroupName"], ["InstanceId", "InstanceType"]] } ``` To roll up metrics into one collection instead, use `[]`. ``` "metrics": { "cpu":{...} "disk":{...} "aggregation_dimensions" : [[]] } ``` Remember that any time you change the agent configuration file, you must restart the agent to have the changes take effect. ## Collecting high-resolution metrics with the CloudWatch agent The `metrics_collection_interval` field specifies the time interval for the metrics collected, in seconds. By specifying a value of less than 60 for this field, the metrics are collected as high-resolution metrics. For example, if your metrics should all be high-resolution and collected every 10 seconds, specify 10 as the value for `metrics_collection_interval` under the `agent` section as a global metrics collection interval. ``` "agent": { "metrics_collection_interval": 10 } ``` Alternatively, the following example sets the `cpu` metrics to be collected every second, and all other metrics are collected every minute. ``` "agent":{ "metrics_collection_interval": 60 }, "metrics":{ "metrics_collected":{ "cpu":{ "resources":[ "*" ], "measurement":[ "cpu_usage_guest" ], "totalcpu":false, "metrics_collection_interval": 1 }, "disk":{ "resources":[ "/", "/tmp" ], "measurement":[ "total", "used" ] } } } ``` Remember that any time you change the agent configuration file, you must restart the agent to have the changes take effect. ## Sending metrics, logs, and traces to a different account To have the CloudWatch agent send the metrics, logs, or traces to a different account, specify a `role_arn` parameter in the agent configuration file on the sending server. The `role_arn` value specifies an IAM role in the target account that the agent uses when sending data to the target account. This role enables the sending account to assume a corresponding role in the target account when delivering the metrics or logs to the target account. You can also specify separate `role_arn` strings in the agent configuration file: one to use when sending metrics, another for sending logs, and another for sending traces. The following example of part of the `agent` section of the configuration file sets the agent to use `CrossAccountAgentRole` when sending data to a different account. ``` { "agent": { "credentials": { "role_arn": "arn:aws:iam::123456789012:role/CrossAccountAgentRole" } }, ..... } ``` Alternatively, the following example sets different roles for the sending account to use for sending metrics, logs, and traces: ``` "metrics": { "credentials": { "role_arn": "RoleToSendMetrics" }, "metrics_collected": {.... ``` ``` "logs": { "credentials": { "role_arn": "RoleToSendLogs" }, .... ``` **Policies needed** When you specify a `role_arn` in the agent configuration file, you must also make sure the IAM roles of the sending and target accounts have certain policies. The roles in both the sending and target accounts should have `CloudWatchAgentServerPolicy`. For more information about assigning this policy to a role, see [Prerequisites](prerequisites.md). The role in the sending account also must include the following policy. You add this policy on the **Permissions** tab in the IAM console when you edit the role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRole" ], "Resource": [ "arn:aws:iam::111122223333:role/agent-role-in-target-account" ] } ] } ``` ------ The role in the target account must include the following policy so that it recognizes the IAM role used by the sending account. You add this policy on the **Trust relationships** tab in the IAM console when you edit the role. This role is the role specified in `agent-role-in-target-account` in the policy used by the sending account. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::111122223333:role/role-in-sender-account" ] }, "Action": "sts:AssumeRole" } ] } ``` ------ ## Timestamp differences between the CloudWatch agent and the earlier CloudWatch Logs agent The CloudWatch agent supports a different set of symbols for timestamp formats, compared to the earlier CloudWatch Logs agent. These differences are shown in the following table. | Symbols supported by both agents | Symbols supported only by CloudWatch agent | Symbols supported only by earlier CloudWatch Logs agent | | --- | --- | --- | | %A, %a, %b, %B, %d, %f, %H, %l, %m, %M, %p, %S, %y, %Y, %Z, %z | %-d, %-l, %-m, %-M, %-S | %c,%j, %U, %W, %w | For more information about the meanings of the symbols supported by the new CloudWatch agent, see [ CloudWatch Agent Configuration File: Logs Section](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Logssection) in the *Amazon CloudWatch User Guide*. For information about symbols supported by the CloudWatch Logs agent, see [Agent Configuration File](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html#agent-configuration-file) in the *Amazon CloudWatch Logs User Guide*. ## Appending OpenTelemetry collector configuration files The CloudWatch agent supports supplemental OpenTelemetry collector configuration files alongside its own configuration files. This feature allows you to use CloudWatch agent features such as CloudWatch Application Signals or Container Insights through the CloudWatch agent configuration and bring in your existing OpenTelemetry collector configuration with a single agent. To prevent merge conflicts with pipelines automatically created by CloudWatch agent, we recommend that you add a custom suffix to each of the components and pipelines in your OpenTelemetry collector configuration. ``` receivers: otlp/custom-suffix: protocols: http: exporters: awscloudwatchlogs/custom-suffix: log_group_name: "test-group" log_stream_name: "test-stream" service: pipelines: logs/custom-suffix: receivers: [otlp/custom-suffix] exporters: [awscloudwatchlogs/custom-suffix] ``` To configure the CloudWatch agent, start the CloudWatch agent using the `fetch-config` option and specify the CloudWatch agent’s configuration file. CloudWatch agent requires at least one CloudWatch agent configuration file. ``` /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -c file:/tmp/agent.json -s ``` Next, use the `append-config` option while specifying the OpenTelemetry collector configuration file. ``` /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -c file:/tmp/otel.yaml -s ``` The agent merges the two configuration files on start up and logs the resolved configuration. # CloudWatch agent credentials preference This section outlines the credentials provider chain the CloudWatch agent uses to obtain credentials when communicating with other AWS services and APIs. The ordering is as follows: **Note** The preferences listed in numbers two through five are the same preference order as defined in the AWS SDK. For more information, see [Specifying Credentials](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials) in the SDK documentation. 1. Shared config and credentials files as defined in the CloudWatch agent’s `common-config.toml` file. For more information, see [Install the CloudWatch agent using AWS Systems Manager](installing-cloudwatch-agent-ssm.md). 1. AWS SDK environment variables **Important** On Linux, if you run the CloudWatch agent using the `amazon-cloudwatch-agent-ctl` script, the script starts the agent as a `systemd` service. In this case, environment variables such as `HOME`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY` are not accessible by the agent. 1. Shared configuration and credentials files found in `$HOME/%USERPROFILE%` **Note** The CloudWatch agent looks for `.aws/credentials` in `$HOME` for Linux and MacOS and looks in `%USERPROFILE%` for Windows. Unlike the AWS SDK, the CloudWatch agent does not have fallback methods to determine the home directory if the environment variables are inaccessible. This difference in behavior is to maintain backwards compatibility with earlier implementations of the AWS SDK. Furthermore, unlike with the shared credentials found in `common-config.toml`, if the AWS SDK-derived shared credentials expire and are rotated, the renewed credentials are not automatically picked up by the CloudWatch agent and require a restart of the agent to do so. 1. An AWS Identity and Access Management role for tasks if an application is present that uses an Amazon Elastic Container Service task definition or a RunTask API operation. 1. An instance profile attached to an Amazon EC2 instance. As a best practice, we recommend that you specify credentials in the following order when you use the CloudWatch agent. 1. Use IAM roles for tasks if your application uses an Amazon Elastic Container Service task definition or a RunTask API operation. 1. Use IAM roles if your application runs on an Amazon EC2 instance. 1. Use the CloudWatch agent `common-config.toml` file to specify the credentials file. This credentials file is the same one used by other AWS SDKs and the AWS CLI. If you’re already using a shared credentials file, you can also use it for this purpose. If you provide it by using the CloudWatch agent’s `common-config.toml` file, you ensure that the agent will consume rotated credentials when they expire and get replaced without requiring you to restart the agent. 1. Use environment variables. Setting environment variables is useful if you’re doing development work on a computer other than an Amazon EC2 instance. **Note** If you send telemetry to a different account as explained in [Sending metrics, logs, and traces to a different account](CloudWatch-Agent-common-scenarios.md#CloudWatch-Agent-send-to-different-AWS-account), the CloudWatch agent uses the credentials provider chain described in this section to obtain the initial set of credentials. It then uses those credentials when assuming the IAM role specified by `role_arn` in the CloudWatch agent configuration file. # Troubleshooting the CloudWatch agent You can use the information in this section to troubleshoot issues you might encounter with the CloudWatch agent. When encountering issues with the CloudWatch agent, you can use the `AWSSupport-TroubleshootCloudWatchAgent` automation runbook. The AWS troubleshooting tool can: + Verify IAM permissions and instance profiles + Check agent status and analyze logs + Test endpoint connectivity + Automatically collect and upload relevant logs to Amazon S3 For detailed information on the AWS troubleshooting tool, see [Support Automation Workflow (SAW) Runbook - Troubleshoot CloudWatch agent](https://repost.aws/articles/ARDFhNRgSMRcahrIbGJaIC4g/support-automation-workflow-saw-runbook-troubleshoot-amazon-cloudwatch-agent). **Topics** + [ ## CloudWatch agent command line parameters ](#CloudWatch-Agent-options-help) + [ ## Install the CloudWatch agent using Run Command fails ](#CloudWatch-Agent-installation-fails) + [ ## The CloudWatch agent won't start ](#CloudWatch-Agent-troubleshooting-cannot-start) + [ ## Verify that the CloudWatch agent is running ](#CloudWatch-Agent-troubleshooting-verify-running) + [ ## The CloudWatch agent won't start, and the error mentions an Amazon EC2 Region ](#CloudWatch-Agent-troubleshooting-EC2-region) + [ ## The CloudWatch agent won't start on Windows Server ](#CloudWatch-Agent-troubleshooting-Windows-start) + [ ## Where are the metrics? ](#CloudWatch-Agent-troubleshooting-no-metrics) + [ ## The CloudWatch agent takes a long time to run in a container or logs a hop limit error ](#CloudWatch-Agent-container-slow) + [ ## I updated my agent configuration but don’t see the new metrics or logs in the CloudWatch console ](#CloudWatch-Agent-troubleshooting-update-no-new-metrics) + [ ## CloudWatch agent files and locations ](#CloudWatch-Agent-files-and-locations) + [ ## Finding information about CloudWatch agent versions ](#CloudWatch-Agent-troubleshooting-agent-version) + [ ## Logs generated by the CloudWatch agent ](#CloudWatch-Agent-troubleshooting-loginfo) + [ ## Stopping and restarting the CloudWatch agent ](#CloudWatch-Agent-troubleshooting-stopping-restarting) ## CloudWatch agent command line parameters To see the full list of parameters supported by the CloudWatch agent, enter the following at the command line at a computer where you have it installed: ``` amazon-cloudwatch-agent-ctl -help ``` ## Install the CloudWatch agent using Run Command fails To install the CloudWatch agent using Systems Manager Run Command, the SSM Agent on the target server must be version 2.2.93.0 or later of the SSM Agent agent. If your SSM Agent isn't the correct version, you might see errors that include the following messages: ``` no latest version found for package AmazonCloudWatchAgent on platform linux ``` ``` failed to download installation package reliably ``` For information about updating your SSM Agent version, see [Installing and Configuring SSM Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html) in the *AWS Systems Manager User Guide*. ## The CloudWatch agent won't start If the CloudWatch agent fails to start, there might be an issue in your configuration. Configuration information is logged in the `configuration-validation.log` file. This file is located in `/opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log` on Linux servers and in `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\configuration-validation.log` on servers running Windows Server. ## Verify that the CloudWatch agent is running You can query the CloudWatch agent to find whether it's running or stopped. You can use AWS Systems Manager to do this remotely. You can also use the command line, but only to check the local server. **To query the status of the CloudWatch agent using Run Command** 1. Open the Systems Manager console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/). 1. In the navigation pane, choose **Run Command**. -or- If the AWS Systems Manager home page opens, scroll down and choose **Explore Run Command**. 1. Choose **Run command**. 1. In the **Command document** list, choose the button next to **AmazonCloudWatch-ManageAgent**. 1. In the **Action** list, choose **status**. 1. For **Optional Configuration Source** choose **default** and keep **Optional Configuration Location** blank. 1. In the **Target** area, choose the instance to check. 1. Choose **Run**. If the agent is running, the output resembles the following. ``` { "status": "running", "starttime": "2017-12-12T18:41:18", "version": "1.73.4" } ``` If the agent is stopped, the `"status"` field displays `"stopped"`. **To query the status of the CloudWatch agent locally using the command line** + On a Linux server, enter the following: ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status ``` On a server running Windows Server, enter the following in PowerShell as an administrator: ``` & $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -m ec2 -a status ``` ## The CloudWatch agent won't start, and the error mentions an Amazon EC2 Region If the agent doesn't start and the error message mentions an Amazon EC2 Region endpoint, you might have configured the agent to need access to the Amazon EC2 endpoint without granting that access. For example, if you specify a value for the `append_dimensions` parameter in the agent configuration file that depends on Amazon EC2 metadata and you use proxies, you must make sure that the server can access the endpoint for Amazon EC2. For more information about these endpoints, see [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region) in the *Amazon Web Services General Reference*. ## The CloudWatch agent won't start on Windows Server On Windows Server, you might see the following error: ``` Start-Service : Service 'Amazon CloudWatch Agent (AmazonCloudWatchAgent)' cannot be started due to the following error: Cannot start service AmazonCloudWatchAgent on computer '.'. At C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1:113 char:12 + $svc | Start-Service + ~~~~~~~~~~~~~ + CategoryInfo : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service], ServiceCommandException + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand ``` To fix this, first make sure that the server service is running. This error can be seen if the agent tries to start when the server service isn't running. If the server service is already running, the following may be the issue. On some Windows Server installations, the CloudWatch agent takes more than 30 seconds to start. Because Windows Server, by default, allows only 30 seconds for services to start, this causes the agent to fail with an error similar to the following: To fix this issue, increase the service timeout value. For more information, see [ A service does not start, and events 7000 and 7011 are logged in the Windows event log](https://support.microsoft.com/en-us/help/922918/a-service-does-not-start-and-events-7000-and-7011-are-logged-in-window). ## Where are the metrics? If the CloudWatch agent has been running but you can't find metrics collected by it in the AWS Management Console or the AWS CLI, confirm that you're using the correct namespace. By default, the namespace for metrics collected by the agent is `CWAgent`. You can customize this namespace using the `namespace` field in the `metrics` section of the agent configuration file. If you don't see the metrics that you expect, check the configuration file to confirm the namespace being used. When you first download the CloudWatch agent package, the agent configuration file is `amazon-cloudwatch-agent.json`. This file is in the directory where you ran the configuration wizard, or you might have moved it to a different directory. If you use the configuration wizard, the agent configuration file output from the wizard is named `config.json`. For more information about the configuration file, including the `namespace` field, see [CloudWatch agent configuration file: Metrics section](CloudWatch-Agent-Configuration-File-Details.md#CloudWatch-Agent-Configuration-File-Metricssection). ## The CloudWatch agent takes a long time to run in a container or logs a hop limit error When you run the CloudWatch agent as a container service and want to add Amazon EC2 metric dimensions to all metrics collected by the agent, you might see the following errors in version v1.247354.0 of the agent: ``` 2022-06-07T03:36:11Z E! [processors.ec2tagger] ec2tagger: Unable to retrieve Instance Metadata Tags. This plugin must only be used on an EC2 instance. 2022-06-07T03:36:11Z E! [processors.ec2tagger] ec2tagger: Please increase hop limit to 2 by following this document https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html#configuring-IMDS-existing-instances. 2022-06-07T03:36:11Z E! [telegraf] Error running agent: could not initialize processor ec2tagger: EC2MetadataRequestError: failed to get EC2 instance identity document caused by: EC2MetadataError: failed to make EC2Metadata request status code: 401, request id: caused by: ``` You might see this error if the agent tries to get metadata from IMDSv2 inside a container without an appropriate hop limit. In versions of the agent earlier than v1.247354.0, you can experience this issue without seeing the log message. To solve this, increase the hop limit to 2 by following the instructions in [ Configure the instance metadata options](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html#configuring-IMDS-existing-instances.). ## I updated my agent configuration but don’t see the new metrics or logs in the CloudWatch console If you update your CloudWatch agent configuration file, the next time that you start the agent, you need to use the **fetch-config** option. For example, if you stored the updated file on the local computer, enter the following command: ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -s -m ec2 -c file:configuration-file-path ``` ## CloudWatch agent files and locations The following table lists the files installed by and used with the CloudWatch agent, along with their locations on servers running Linux or Windows Server. | File | Linux location | Windows Server location | | --- | --- | --- | | The control script that controls starting, stopping, and restarting the agent. | `/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl` or `/usr/bin/amazon-cloudwatch-agent-ctl` | `$Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1` | | The log file the agent writes to. You might need to attach this when contacting AWS Support. | `/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log` or `/var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log` | `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log` | | Agent configuration validation file. | `/opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log` or `/var/log/amazon/amazon-cloudwatch-agent/configuration-validation.log` | `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\configuration-validation.log` | | The JSON file used to configure the agent immediately after the wizard creates it. For more information, see [Create the CloudWatch agent configuration file](create-cloudwatch-agent-configuration-file.md). | `/opt/aws/amazon-cloudwatch-agent/bin/config.json` | `$Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\config.json` | | The JSON file used to configure the agent if this configuration file has been downloaded from Parameter Store. | `/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json` or `/etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json` | `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.json` | | The TOML file used to specify Region and credential information to be used by the agent, overriding system defaults. | `/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml` or `/etc/amazon/amazon-cloudwatch-agent/common-config.toml` | `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml` | | The TOML file that contains the converted contents of the JSON configuration file. The `amazon-cloudwatch-agent-ctl` script generates this file. Users should not directly modify this file. It can be useful for verifying that JSON to TOML translation was successful. | `/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml` or `/etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.toml` | `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.toml` | | The YAML file that contains the converted contents of the JSON configuration file. The `amazon-cloudwatch-agent-ctl` script generates this file. You should not directly modify this file. This file can be useful for verifying that the JSON to YAML translation was successful. | `/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.yaml or /etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.yaml` | `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.yaml` | ## Finding information about CloudWatch agent versions To find the version number of the CloudWatch agent on a Linux server, enter the following command: ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status ``` To find the version number of the CloudWatch agent on Windows Server, enter the following command: ``` & $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -m ec2 -a status ``` **Note** Using this command is the correct way to find the version of the CloudWatch agent. If you use **Programs and Features** in the Control Panel, you will see an incorrect version number. You can also download a README file about the latest changes to the agent, and a file that indicates the version number that is currently available for download. These files are in the following locations: + `https://amazoncloudwatch-agent.s3.amazonaws.com/info/latest/RELEASE_NOTES` or `https://amazoncloudwatch-agent-us-east-1.s3.us-east-1.amazonaws.com/info/latest/RELEASE_NOTES` + `https://amazoncloudwatch-agent.s3.amazonaws.com/info/latest/CWAGENT_VERSION` or `https://amazoncloudwatch-agent-us-east-1.s3.us-east-1.amazonaws.com/info/latest/CWAGENT_VERSION` ## Logs generated by the CloudWatch agent The agent generates a log while it runs. This log includes troubleshooting information. This log is the `amazon-cloudwatch-agent.log` file. This file is located in `/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log` on Linux servers and in `$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log` on servers running Windows Server. You can configure the agent to log additional details in the `amazon-cloudwatch-agent.log` file. In the agent configuration file, in the `agent` section, set the `debug` field to `true`, then reconfigure and restart the CloudWatch agent. To disable the logging of this extra information, set the `debug` field to `false`. Then, reconfigure and restart the agent. For more information, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). In versions 1.247350.0 and later of the CloudWatch agent, you can optionally set the `aws_sdk_log_level` field in the `agent` section of the agent configuration file to one or more of the following options. Separate multiple options with the `|` character. + `LogDebug` + `LogDebugWithSigning` + `LogDebugWithHTTPBody` + `LogDebugRequestRetries` + `LogDebugWithEventStreamBody` For more information about these options, see [LogLevelType](https://docs.aws.amazon.com/sdk-for-go/api/aws/#LogLevelType). ## Stopping and restarting the CloudWatch agent You can manually stop the CloudWatch agent using either AWS Systems Manager or the command line. **To stop the CloudWatch agent using Run Command** 1. Open the Systems Manager console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/). 1. In the navigation pane, choose **Run Command**. -or- If the AWS Systems Manager home page opens, scroll down and choose **Explore Run Command**. 1. Choose **Run command**. 1. In the **Command document** list, choose **AmazonCloudWatch-ManageAgent**. 1. In the **Targets** area, choose the instance where you installed the CloudWatch agent. 1. In the **Action** list, choose **stop**. 1. Keep **Optional Configuration Source** and **Optional Configuration Location** blank. 1. Choose **Run**. **To stop the CloudWatch agent locally using the command line** + On a Linux server, enter the following: ``` sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop ``` On a server running Windows Server, enter the following in PowerShell as an administrator: ``` & $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -m ec2 -a stop ``` To restart the agent, follow the instructions in [(Optional) Modify the common configuration and named profile for CloudWatch agent](installing-cloudwatch-agent-ssm.md#CloudWatch-Agent-profile-instance-fleet).