# Getting started with CloudWatch Logs
To collect logs from your Amazon EC2 instances and on-premises servers into CloudWatch Logs, use the unified CloudWatch agent. It enables you to collect both logs and advanced metrics with one agent. It offers support across operating systems, including servers running Windows Server. This agent also provides better performance.
If you're using the unified CloudWatch agent to collect CloudWatch metrics, it enables the collection of additional system metrics, for in-guest visibility. It also supports collecting custom metrics using `StatsD` or `collectd`.
For more information, see [Installing the CloudWatch Agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance.html) in the *Amazon CloudWatch User Guide*.
The older CloudWatch Logs agent, which supports only the collection of logs from servers running Linux, is deprecated and is no longer supported. For information about migrating from the older CloudWatch Logs agent to the unified agent, see [ Create the CloudWatch agent configuration file with the wizard](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html).
**Topics**
+ [Prerequisites](GettingSetup_cwl.md)
+ [Using the unified CloudWatch agent](UseCloudWatchUnifiedAgent.md)
+ [Using the previous CloudWatch agent](UsePreviousCloudWatchLogsAgent.md)
+ [Quick Start with CloudFormation](QuickStartCloudFormation.md)
# Prerequisites
To use Amazon CloudWatch Logs you need an AWS account. Your AWS account allows you to use services (for example, Amazon EC2) to generate logs that you can view in the CloudWatch console, a web-based interface. In addition, you can install and configure the AWS Command Line Interface (AWS CLI).
## Sign up for an AWS account
If you do not have an AWS account, complete the following steps to create one.
**To sign up for an AWS account**
1. Open [https://portal.aws.amazon.com/billing/signup](https://portal.aws.amazon.com/billing/signup).
1. Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call or text message and entering a verification code on the phone keypad.
When you sign up for an AWS account, an *AWS account root user* is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform [tasks that require root user access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks).
AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to [https://aws.amazon.com/](https://aws.amazon.com/) and choosing **My Account**.
## Create a user with administrative access
After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.
**Secure your AWS account root user**
1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) as the account owner by choosing **Root user** and entering your AWS account email address. On the next page, enter your password.
For help signing in by using root user, see [Signing in as the root user](https://docs.aws.amazon.com/signin/latest/userguide/console-sign-in-tutorials.html#introduction-to-root-user-sign-in-tutorial) in the *AWS Sign-In User Guide*.
1. Turn on multi-factor authentication (MFA) for your root user.
For instructions, see [Enable a virtual MFA device for your AWS account root user (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/enable-virt-mfa-for-root.html) in the *IAM User Guide*.
**Create a user with administrative access**
1. Enable IAM Identity Center.
For instructions, see [Enabling AWS IAM Identity Center](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-set-up-for-idc.html) in the *AWS IAM Identity Center User Guide*.
1. In IAM Identity Center, grant administrative access to a user.
For a tutorial about using the IAM Identity Center directory as your identity source, see [ Configure user access with the default IAM Identity Center directory](https://docs.aws.amazon.com//singlesignon/latest/userguide/quick-start-default-idc.html) in the *AWS IAM Identity Center User Guide*.
**Sign in as the user with administrative access**
+ To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.
For help signing in using an IAM Identity Center user, see [Signing in to the AWS access portal](https://docs.aws.amazon.com/signin/latest/userguide/iam-id-center-sign-in-tutorial.html) in the *AWS Sign-In User Guide*.
**Assign access to additional users**
1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions.
For instructions, see [ Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-started-create-a-permission-set.html) in the *AWS IAM Identity Center User Guide*.
1. Assign users to a group, and then assign single sign-on access to the group.
For instructions, see [ Add groups](https://docs.aws.amazon.com//singlesignon/latest/userguide/addgroups.html) in the *AWS IAM Identity Center User Guide*.
## Set up the Command Line Interface
You can use the AWS CLI to perform CloudWatch Logs operations.
For information about how to install and configure the AWS CLI, see [Getting Set Up with the AWS Command Line Interface](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html) in the *AWS Command Line Interface User Guide*.
# Use the unified CloudWatch agent to get started with CloudWatch Logs
For more information about using the unified CloudWatch agent to get started with CloudWatch Logs, see [Collect Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) in the *Amazon CloudWatch User Guide*. You complete the steps listed in this section to install, configure, and start the agent. If you are not using the agent to also collect CloudWatch metrics, you can ignore any sections that refer to metrics.
If you are currently using the older CloudWatch Logs agent and want to migrate to using the new unified agent, we recommend that you use the wizard included in the new agent package. This wizard can read your current CloudWatch Logs agent configuration file and set up the CloudWatch agent to collect the same logs. For more information about the wizard, see [ Create the CloudWatch Agent Configuration File with the Wizard](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html) in the *Amazon CloudWatch User Guide*.
# Use the previous CloudWatch agent to get started with CloudWatch Logs
**Important**
CloudWatch includes a unified CloudWatch agent that can collect both logs and metrics from EC2 instances and on-premises servers. The older logs-only agent is deprecated and is no longer supported.
For information about migrating from the older logs-only agent to the unified agent, see [ Create the CloudWatch agent configuration file with the wizard](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html).
The rest of this section explains the use of the older CloudWatch Logs agent for customers who are still using it.
Using the CloudWatch Logs agent, you can publish log data from Amazon EC2 instances running Linux or Windows Server, and logged events from AWS CloudTrail. We recommend instead using the CloudWatch unified agent to publish your log data. For more information about the new agent, see [Collect Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) in the *Amazon CloudWatch User Guide*.
**Topics**
+ [CloudWatch Logs agent prerequisites](#CWL_Prerequisites)
+ [Quick Start: Install the agent on a running EC2 Linux instance](QuickStartEC2Instance.md)
+ [Quick Start: Install the agent on an EC2 Linux instance at launch](EC2NewInstanceCWL.md)
+ [Quick Start: Use CloudWatch Logs with Windows Server 2016 instances](QuickStartWindows2016.md)
+ [Quick Start: Use CloudWatch Logs with Windows Server 2012 and Windows Server 2008 instances](QuickStartWindows20082012.md)
+ [Report the CloudWatch Logs agent status](ReportCWLAgentStatus.md)
+ [Start the CloudWatch Logs agent](StartTheCWLAgent.md)
+ [Stop the CloudWatch Logs agent](StopTheCWLAgent.md)
+ [CloudWatch Logs agent reference](AgentReference.md)
## CloudWatch Logs agent prerequisites
The CloudWatch Logs agent requires Python version 2.7, 3.0, or 3.3, and any of the following versions of Linux:
+ Amazon Linux version 2014.03.02 or later. Amazon Linux 2 is not supported
+ Ubuntu Server version 12.04, 14.04, or 16.04
+ CentOS version 6, 6.3, 6.4, 6.5, or 7.0
+ Red Hat Enterprise Linux (RHEL) version 6.5 or 7.0
+ Debian 8.0
# Quick Start: Install and configure the CloudWatch Logs agent on a running EC2 Linux instance
**Important**
The older logs agent is deprecated. CloudWatch includes a unified agent that can collect both logs and metrics from EC2 instances and on-premises servers. For more information, see [Getting started with CloudWatch Logs](CWL_GettingStarted.md).
For information about migrating from the older CloudWatch Logs agent to the unified agent, see [ Create the CloudWatch agent configuration file with the wizard](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html).
The older logs agent supports only versions 2.6 to 3.5 of Python. Additionally, the older CloudWatch Logs agent doesn't support Instance Metadata Service Version 2 (IMDSv2). If your server uses IMDSv2, you must use the newer unified agent instead of the older CloudWatch Logs agent.
The rest of this section explains the use of the older CloudWatch Logs agent for customers who are still using it.
**Tip**
CloudWatch includes a new unified agent that can collect both logs and metrics from EC2 instances and on-premises servers. If you are not already using the older CloudWatch Logs agent, we recommend that you use the newer unified CloudWatch agent. For more information, see [Getting started with CloudWatch Logs](CWL_GettingStarted.md).
Additionally, the older agent doesn't support Instance Metadata Service Version 2 (IMDSv2). If your server uses IMDSv2, you must use the newer unified agent instead of the older CloudWatch Logs agent.
The rest of this section explains the use of the older CloudWatch Logs agent.
## Configure the older CloudWatch Logs agent on a running EC2 Linux instance
You can use the CloudWatch Logs agent installer on an existing EC2 instance to install and configure the CloudWatch Logs agent. After installation is complete, logs automatically flow from the instance to the log stream you create while installing the agent. The agent confirms that it has started and it stays running until you disable it.
In addition to using the agent, you can also publish log data using the AWS CLI, CloudWatch Logs SDK, or the CloudWatch Logs API. The AWS CLI is best suited for publishing data at the command line or through scripts. The CloudWatch Logs SDK is best suited for publishing log data directly from applications or building your own log publishing application.
### Step 1: Configure your IAM role or user for CloudWatch Logs
The CloudWatch Logs agent supports IAM roles and users. If your instance already has an IAM role associated with it, make sure that you include the IAM policy below. If you don't already have an IAM role assigned to your instance, you can use your IAM credentials for the next steps or you can assign an IAM role to that instance. For more information, see [Attaching an IAM Role to an Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#attach-iam-role).
**To configure your IAM role or user for CloudWatch Logs**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the navigation pane, choose **Roles**.
1. Choose the role by selecting the role name (do not select the check box next to the name).
1. Choose **Attach Policies**, **Create Policy**.
A new browser tab or window opens.
1. Choose the **JSON** tab and type the following JSON policy document.
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": [
"*"
]
}
]
}
```
1. When you are finished, choose **Review policy**. The Policy Validator reports any syntax errors.
1. On the **Review Policy** page, type a **Name** and a **Description** (optional) for the policy that you are creating. Review the policy **Summary** to see the permissions that are granted by your policy. Then choose **Create policy** to save your work.
1. Close the browser tab or window, and return to the **Add permissions** page for your role. Choose **Refresh**, and then choose the new policy to attach it to your role.
1. Choose **Attach Policy**.
### Step 2: Install and configure CloudWatch Logs on an existing Amazon EC2 instance
The process for installing the CloudWatch Logs agent differs depending on whether your Amazon EC2 instance is running Amazon Linux, Ubuntu, CentOS, or Red Hat. Use the steps appropriate for the version of Linux on your instance.
**To install and configure CloudWatch Logs on an existing Amazon Linux instance**
Starting with Amazon Linux AMI 2014.09, the CloudWatch Logs agent is available as an RPM installation with the awslogs package. Earlier versions of Amazon Linux can access the awslogs package by updating their instance with the `sudo yum update -y` command. By installing the awslogs package as an RPM instead of the using the CloudWatch Logs installer, your instance receives regular package updates and patches from AWS without having to manually reinstall the CloudWatch Logs agent.
**Warning**
Do not update the CloudWatch Logs agent using the RPM installation method if you previously used the Python script to install the agent. Doing so may cause configuration issues that prevent the CloudWatch Logs agent from sending your logs to CloudWatch.
1. Connect to your Amazon Linux instance. For more information, see [Connect to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-connect-to-instance-linux.html) in the *Amazon EC2 User Guide*.
For more information about connection issues, see [Troubleshooting Connecting to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) in the *Amazon EC2 User Guide*.
1. Update your Amazon Linux instance to pick up the latest changes in the package repositories.
```
sudo yum update -y
```
1. Install the `awslogs` package. This is the recommended method for installing awslogs on Amazon Linux instances.
```
sudo yum install -y awslogs
```
1. Edit the `/etc/awslogs/awslogs.conf` file to configure the logs to track. For more information about editing this file, see [CloudWatch Logs agent reference](AgentReference.md).
1. By default, the `/etc/awslogs/awscli.conf` points to the us-east-1 Region. To push your logs to a different Region, edit the `awscli.conf` file and specify that Region.
1. Start the `awslogs` service.
```
sudo service awslogs start
```
If you are running Amazon Linux 2, start the `awslogs` service with the following command.
```
sudo systemctl start awslogsd
```
1. (Optional) Check the `/var/log/awslogs.log` file for errors logged when starting the service.
1. (Optional) Run the following command to start the `awslogs` service at each system boot.
```
sudo chkconfig awslogs on
```
If you are running Amazon Linux 2, use the following command to start the service at each system boot.
```
sudo systemctl enable awslogsd.service
```
1. You should see the newly created log group and log stream in the CloudWatch console after the agent has been running for a few moments.
For more information, see [View log data sent to CloudWatch Logs](Working-with-log-groups-and-streams.md#ViewingLogData).
**To install and configure CloudWatch Logs on an existing Ubuntu Server, CentOS, or Red Hat instance**
If you're using an AMI running Ubuntu Server, CentOS, or Red Hat, use the following procedure to manually install the CloudWatch Logs agent on your instance.
1. Connect to your EC2 instance. For more information, see [Connect to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-connect-to-instance-linux.html) in the *Amazon EC2 User Guide*.
For more information about connection issues, see [Troubleshooting Connecting to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) in the *Amazon EC2 User Guide*.
1. Run the CloudWatch Logs agent installer using one of two options. You can run it directly from the internet, or download the files and run it standalone.
**Note**
If you are running CentOS 6.x, Red Hat 6.x, or Ubuntu 12.04, use the steps for downloading and running the installer standalone. Installing the CloudWatch Logs agent directly from the internet is not supported on these systems.
**Note**
On Ubuntu, run `apt-get update` before running the commands below.
To run it directly from the internet, use the following commands and follow the prompts:
```
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
```
```
sudo python ./awslogs-agent-setup.py --region us-east-1
```
If the preceding command does not work, try the following:
```
sudo python3 ./awslogs-agent-setup.py --region us-east-1
```
To download and run it standalone, use the following commands and follow the prompts:
```
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
```
```
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/AgentDependencies.tar.gz -O
```
```
tar xvf AgentDependencies.tar.gz -C /tmp/
```
```
sudo python ./awslogs-agent-setup.py --region us-east-1 --dependency-path /tmp/AgentDependencies
```
You can install the CloudWatch Logs agent by specifying the us-east-1, us-west-1, us-west-2, ap-south-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1, eu-central-1, eu-west-1, or sa-east-1 Regions.
**Note**
For more information about the current version and the version history of `awslogs-agent-setup`, see [CHANGELOG.txt](https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/CHANGELOG.txt).
The CloudWatch Logs agent installer requires certain information during setup. Before you start, you need to know which log file to monitor and its time stamp format. You should also have the following information ready.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html)
After you have completed these steps, the installer asks about configuring another log file. You can run the process as many times as you like for each log file. If you have no more log files to monitor, choose **N** when prompted by the installer to set up another log. For more information about the settings in the agent configuration file, see [CloudWatch Logs agent reference](AgentReference.md).
**Note**
Configuring multiple log sources to send data to a single log stream is not supported.
1. You should see the newly created log group and log stream in the CloudWatch console after the agent has been running for a few moments.
For more information, see [View log data sent to CloudWatch Logs](Working-with-log-groups-and-streams.md#ViewingLogData).
# Quick Start: Install and configure the CloudWatch Logs agent on an EC2 Linux instance at launch
**Tip**
The older CloudWatch Logs agent discussed in this section is on the path to deprecation. We strongly recommend that you instead use the new unified CloudWatch agent that can collect both logs and metrics. Additionally, the older CloudWatch Logs agent requires Python 3.3 or earlier, and these versions are not installed on new EC2 instances by default. For more information about the unified CloudWatch agent, see [Installing the CloudWatch Agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance.html).
The rest of this section explains the use of the older CloudWatch Logs agent.
## Installing the older CloudWatch Logs agent on an EC2 Linux instance at launch
You can use Amazon EC2 user data, a feature of Amazon EC2 that allows parametric information to be passed to the instance on launch, to install and configure the CloudWatch Logs agent on that instance. To pass the CloudWatch Logs agent installation and configuration information to Amazon EC2, you can provide the configuration file in a network location such as an Amazon S3 bucket.
Configuring multiple log sources to send data to a single log stream is not supported.
**Prerequisite**
Create an agent configuration file that describes all your log groups and log streams. This is a text file that describes the log files to monitor as well as the log groups and log streams to upload them to. The agent consumes this configuration file and starts monitoring and uploading all the log files described in it. For more information about the settings in the agent configuration file, see [CloudWatch Logs agent reference](AgentReference.md).
The following is a sample agent configuration file for Amazon Linux 2
```
[general]
state_file = /var/lib/awslogs/state/agent-state
[/var/log/messages]
file = /var/log/messages
log_group_name = /var/log/messages
log_stream_name = {instance_id}
datetime_format = %b %d %H:%M:%S
```
The following is a sample agent configuration file for Ubuntu
```
[general]
state_file = /var/awslogs/state/agent-state
[/var/log/syslog]
file = /var/log/syslog
log_group_name = /var/log/syslog
log_stream_name = {instance_id}
datetime_format = %b %d %H:%M:%S
```
**To configure your IAM role**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the navigation pane, choose **Policies**, **Create Policy**.
1. On the **Create Policy** page, for **Create Your Own Policy**, choose **Select**. For more information about creating custom policies, see [IAM Policies for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-policies-for-amazon-ec2.html) in the *Amazon EC2 User Guide*.
1. On the **Review Policy** page, for **Policy Name**, type a name for the policy.
1. For **Policy Document**, paste in the following policy:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:*:*:*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::amzn-s3-demo-bucket/*"
]
}
]
}
```
------
1. Choose **Create Policy**.
1. In the navigation pane, choose **Roles**, **Create New Role**.
1. On the **Set Role Name** page, type a name for the role and then choose **Next Step**.
1. On the **Select Role Type** page, choose **Select** next to **Amazon EC2**.
1. On the **Attach Policy** page, in the table header, choose **Policy Type**, **Customer Managed**.
1. Select the IAM policy that you created and then choose **Next Step**.
1. Choose **Create Role**.
For more information about users and policies, see [IAM Users and Groups](https://docs.aws.amazon.com/IAM/latest/UserGuide/Using_WorkingWithGroupsAndUsers.html) and [Managing IAM Policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingPolicies.html) in the *IAM User Guide*.
**To launch a new instance and enable CloudWatch Logs**
1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/).
1. Choose **Launch Instance**.
For more information, see [Launching an Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launching-instance.html) in *Amazon EC2 User Guide*.
1. On the **Step 1: Choose an Amazon Machine Image (AMI)** page, select the Linux instance type to launch, and then on the **Step 2: Choose an Instance Type** page, choose **Next: Configure Instance Details**.
Make sure that [cloud-init](http://cloudinit.readthedocs.org/en/latest/index.html) is included in your Amazon Machine Image (AMI). Amazon Linux AMIs, and AMIs for Ubuntu and RHEL already include cloud-init, but CentOS and other AMIs in the AWS Marketplace might not.
1. On the **Step 3: Configure Instance Details** page, for **IAM role**, select the IAM role that you created.
1. Under **Advanced Details**, for **User data**, paste the following script into the box. Then update that script by changing the value of the **-c** option to the location of your agent configuration file:
```
#!/bin/bash
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
chmod +x ./awslogs-agent-setup.py
./awslogs-agent-setup.py -n -r us-east-1 -c s3://amzn-s3-demo-bucket/my-config-file
```
1. Make any other changes to the instance, review your launch settings, and then choose **Launch**.
1. You should see the newly created log group and log stream in the CloudWatch console after the agent has been running for a few moments.
For more information, see [View log data sent to CloudWatch Logs](Working-with-log-groups-and-streams.md#ViewingLogData).
# Quick Start: Enable your Amazon EC2 instances running Windows Server 2016 to send logs to CloudWatch Logs using the CloudWatch Logs agent
**Tip**
CloudWatch includes a new unified agent that can collect both logs and metrics from EC2 instances and on-premises servers. We recommend that you use the newer unified CloudWatch agent. For more information, see [Getting started with CloudWatch Logs](CWL_GettingStarted.md).
The rest of this section explains the use of the older CloudWatch Logs agent.
## Enable your Amazon EC2 instances running Windows Server 2016 to send logs to CloudWatch Logs using the older CloudWatch Logs agent
There are multiple methods you can use to enable instances running Windows Server 2016 to send logs to CloudWatch Logs. The steps in this section use Systems Manager Run Command. For information about the other possible methods, see [Sending Logs, Events, and Performance Counters to Amazon CloudWatch](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/send_logs_to_cwl.html).
**Topics**
+ [Download the sample configuration file](#configure_cwl_download)
+ [Configure the JSON file for CloudWatch](#send_logs_to_cwl_json)
+ [Create an IAM role for Systems Manager](#iam_permissions)
+ [Verify Systems Manager prerequisites](#send_logs_cwl_syspre)
+ [Verify internet access](#send_logs_cwl_internet)
+ [Enable CloudWatch Logs using Systems Manager Run Command](#remote-commands-cloudwatch)
### Download the sample configuration file
Download the following sample file to your computer: [https://s3.amazonaws.com/ec2-downloads-windows/CloudWatchConfig/AWS.EC2.Windows.CloudWatch.json](https://s3.amazonaws.com/ec2-downloads-windows/CloudWatchConfig/AWS.EC2.Windows.CloudWatch.json).
### Configure the JSON file for CloudWatch
You determine which logs to send to CloudWatch by specifying your choices in a configuration file. The process of creating this file and specifying your choices can take 30 minutes or more to complete. After you have completed this task once, you can reuse the configuration file on all of your instances.
**Topics**
+ [Step 1: Enable CloudWatch Logs](#enable-CloudWatchLogs-in-JSON-file)
+ [Step 2: Configure settings for CloudWatch](#configure_cwl_credentials)
+ [Step 3: Configure the data to send](#configure_logs)
+ [Step 4: Configure flow control](#configure_log_flow)
+ [Step 5: Save JSON content](#save_json_content)
#### Step 1: Enable CloudWatch Logs
At the top of the JSON file, change "false" to "true" for `IsEnabled`:
```
"IsEnabled": true,
```
#### Step 2: Configure settings for CloudWatch
Specify credentials, Region, a log group name, and a log stream namespace. This enables the instance to send log data to CloudWatch Logs. To send the same log data to different locations, you can add additional sections with unique IDs (for example, "CloudWatchLogs2" and CloudWatchLogs3") and a different Region for each ID.
**To configure settings to send log data to CloudWatch Logs**
1. In the JSON file, locate the `CloudWatchLogs` section.
```
{
"Id": "CloudWatchLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"AccessKey": "",
"SecretKey": "",
"Region": "us-east-1",
"LogGroup": "Default-Log-Group",
"LogStream": "{instance_id}"
}
},
```
1. Leave the `AccessKey` and `SecretKey` field blank. You configure credentials using an IAM role.
1. For `Region`, type the Region to which to send log data (for example, `us-east-2`).
1. For `LogGroup`, type the name for your log group. This name appears on the **Log Groups** screen in the CloudWatch console.
1. For `LogStream`, type the destination log stream. This name appears on the **Log Groups > Streams** screen in the CloudWatch console.
If you use `{instance_id}`, the default, the log stream name is the instance ID of this instance.
If you specify a log stream name that doesn't already exist, CloudWatch Logs automatically creates it for you. You can define a log stream name using a literal string, the predefined variables `{instance_id}`, `{hostname}`, and `{ip_address}`, or a combination of these.
#### Step 3: Configure the data to send
You can send event log data, Event Tracing for Windows (ETW) data, and other log data to CloudWatch Logs.
**To send Windows application event log data to CloudWatch Logs**
1. In the JSON file, locate the `ApplicationEventLog` section.
```
{
"Id": "ApplicationEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Application",
"Levels": "1"
}
},
```
1. For `Levels`, specify the type of messages to upload. You can specify one of the following values:
+ **1** - Upload only error messages.
+ **2** - Upload only warning messages.
+ **4** - Upload only information messages.
You can combine values to include more than one type of message. For example, a value of **3** uploads error messages (**1**) and warning messages (**2**). A value of **7** uploads error messages (**1**), warning messages (**2**), and information messages (**4**).
**To send security log data to CloudWatch Logs**
1. In the JSON file, locate the `SecurityEventLog` section.
```
{
"Id": "SecurityEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Security",
"Levels": "7"
}
},
```
1. For `Levels`, type **7** to upload all messages.
**To send system event log data to CloudWatch Logs**
1. In the JSON file, locate the `SystemEventLog` section.
```
{
"Id": "SystemEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "System",
"Levels": "7"
}
},
```
1. For `Levels`, specify the type of messages to upload. You can specify one of the following values:
+ **1** - Upload only error messages.
+ **2** - Upload only warning messages.
+ **4** - Upload only information messages.
You can combine values to include more than one type of message. For example, a value of **3** uploads error messages (**1**) and warning messages (**2**). A value of **7** uploads error messages (**1**), warning messages (**2**), and information messages (**4**).
**To send other types of event log data to CloudWatch Logs**
1. In the JSON file, add a new section. Each section must have a unique `Id`.
```
{
"Id": "Id-name",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Log-name",
"Levels": "7"
}
},
```
1. For `Id`, type a name for the log to upload (for example, **WindowsBackup**).
1. For `LogName`, type the name of the log to upload. You can find the name of the log as follows.
1. Open Event Viewer.
1. In the navigation pane, choose **Applications and Services Logs**.
1. Navigate to the log, and then choose **Actions**, **Properties**.
1. For `Levels`, specify the type of messages to upload. You can specify one of the following values:
+ **1** - Upload only error messages.
+ **2** - Upload only warning messages.
+ **4** - Upload only information messages.
You can combine values to include more than one type of message. For example, a value of **3** uploads error messages (**1**) and warning messages (**2**). A value of **7** uploads error messages (**1**), warning messages (**2**), and information messages (**4**).
**To send Event Tracing for Windows data to CloudWatch Logs**
ETW (Event Tracing for Windows) provides an efficient and detailed logging mechanism that applications can write logs to. Each ETW is controlled by a session manager that can start and stop the logging session. Each session has a provider and one or more consumers.
1. In the JSON file, locate the `ETW` section.
```
{
"Id": "ETW",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Microsoft-Windows-WinINet/Analytic",
"Levels": "7"
}
},
```
1. For `LogName`, type the name of the log to upload.
1. For `Levels`, specify the type of messages to upload. You can specify one of the following values:
+ **1** - Upload only error messages.
+ **2** - Upload only warning messages.
+ **4** - Upload only information messages.
You can combine values to include more than one type of message. For example, a value of **3** uploads error messages (**1**) and warning messages (**2**). A value of **7** uploads error messages (**1**), warning messages (**2**), and information messages (**4**).
**To send custom logs (any text-based log file) to CloudWatch Logs**
1. In the JSON file, locate the `CustomLogs` section.
```
{
"Id": "CustomLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\CustomLogs\\",
"TimestampFormat": "MM/dd/yyyy HH:mm:ss",
"Encoding": "UTF-8",
"Filter": "",
"CultureName": "en-US",
"TimeZoneKind": "Local",
"LineCount": "5"
}
},
```
1. For `LogDirectoryPath`, type the path where logs are stored on your instance.
1. For `TimestampFormat`, type the time stamp format to use. For more information about supported values, see the [Custom Date and Time Format Strings](https://msdn.microsoft.com/en-us/library/8kb3ddd4(v=vs.110).aspx) topic on MSDN.
**Important**
Your source log file must have the time stamp at the beginning of each log line and there must be a space following the time stamp.
1. For `Encoding`, type the file encoding to use (for example, UTF-8). For a list of supported values, see the [Encoding Class](http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx) topic on MSDN.
**Note**
Use the encoding name, not the display name.
1. (Optional) For `Filter`, type the prefix of log names. Leave this parameter blank to monitor all files. For more information about supported values, see the [FileSystemWatcherFilter Property](https://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.filter.aspx) topic on MSDN.
1. (Optional) For `CultureName`, type the locale where the time stamp is logged. If `CultureName` is blank, it defaults to the same locale currently used by your Windows instance. For more information about, see the `Language tag` column in the table in the [Product Behavior](https://msdn.microsoft.com/en-us/library/cc233982.aspx) topic on MSDN.
**Note**
The `div`, `div-MV`, `hu`, and `hu-HU` values are not supported.
1. (Optional) For `TimeZoneKind`, type `Local` or `UTC`. You can set this to provide time zone information when no time zone information is included in your log's time stamp. If this parameter is left blank and if your time stamp doesn't include time zone information, CloudWatch Logs defaults to the local time zone. This parameter is ignored if your time stamp already contains time zone information.
1. (Optional) For `LineCount`, type the number of lines in the header to identify the log file. For example, IIS log files have virtually identical headers. You could enter **5**, which would read the first three lines of the log file header to identify it. In IIS log files, the third line is the date and time stamp, but the time stamp is not always guaranteed to be different between log files. For this reason, we recommend including at least one line of actual log data to uniquely fingerprint the log file.
**To send IIS log data to CloudWatch Logs**
1. In the JSON file, locate the `IISLog` section.
```
{
"Id": "IISLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\inetpub\\logs\\LogFiles\\W3SVC1",
"TimestampFormat": "yyyy-MM-dd HH:mm:ss",
"Encoding": "UTF-8",
"Filter": "",
"CultureName": "en-US",
"TimeZoneKind": "UTC",
"LineCount": "5"
}
},
```
1. For `LogDirectoryPath`, type the folder where IIS logs are stored for an individual site (for example, `C:\inetpub\logs\LogFiles\W3SVCn`).
**Note**
Only W3C log format is supported. IIS, NCSA, and Custom formats are not supported.
1. For `TimestampFormat`, type the time stamp format to use. For more information about supported values, see the [Custom Date and Time Format Strings](https://msdn.microsoft.com/en-us/library/8kb3ddd4(v=vs.110).aspx) topic on MSDN.
1. For `Encoding`, type the file encoding to use (for example, UTF-8). For more information about supported values, see the [Encoding Class](http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx) topic on MSDN.
**Note**
Use the encoding name, not the display name.
1. (Optional) For `Filter`, type the prefix of log names. Leave this parameter blank to monitor all files. For more information about supported values, see the [FileSystemWatcherFilter Property](https://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.filter.aspx) topic on MSDN.
1. (Optional) For `CultureName`, type the locale where the time stamp is logged. If `CultureName` is blank, it defaults to the same locale currently used by your Windows instance. For more information about supported values, see the `Language tag` column in the table in the [Product Behavior](https://msdn.microsoft.com/en-us/library/cc233982.aspx) topic on MSDN.
**Note**
The `div`, `div-MV`, `hu`, and `hu-HU` values are not supported.
1. (Optional) For `TimeZoneKind`, enter `Local` or `UTC`. You can set this to provide time zone information when no time zone information is included in your log's time stamp. If this parameter is left blank and if your time stamp doesn't include time zone information, CloudWatch Logs defaults to the local time zone. This parameter is ignored if your time stamp already contains time zone information.
1. (Optional) For `LineCount`, type the number of lines in the header to identify the log file. For example, IIS log files have virtually identical headers. You could enter **5**, which would read the first five lines of the log file's header to identify it. In IIS log files, the third line is the date and time stamp, but the time stamp is not always guaranteed to be different between log files. For this reason, we recommend including at least one line of actual log data for uniquely fingerprinting the log file.
#### Step 4: Configure flow control
Each data type must have a corresponding destination in the `Flows` section. For example, to send the custom log, ETW log, and system log to CloudWatch Logs, add `(CustomLogs,ETW,SystemEventLog),CloudWatchLogs` to the `Flows` section.
**Warning**
Adding a step that is not valid blocks the flow. For example, if you add a disk metric step, but your instance doesn't have a disk, all steps in the flow are blocked.
You can send the same log file to more than one destination. For example, to send the application log to two different destinations that you defined in the `CloudWatchLogs` section, add `ApplicationEventLog,(CloudWatchLogs,CloudWatchLogs2)` to the `Flows` section.
**To configure flow control**
1. In the `AWS.EC2.Windows.CloudWatch.json` file, locate the `Flows` section.
```
"Flows": {
"Flows": [
"PerformanceCounter,CloudWatch",
"(PerformanceCounter,PerformanceCounter2), CloudWatch2",
"(CustomLogs, ETW, SystemEventLog),CloudWatchLogs",
"CustomLogs, CloudWatchLogs2",
"ApplicationEventLog,(CloudWatchLogs, CloudWatchLogs2)"
]
}
```
1. For `Flows`, add each data type that is to be uploaded (for example, `ApplicationEventLog`) and its destination (for example, `CloudWatchLogs`).
#### Step 5: Save JSON content
You are now finished editing the JSON file. Save it, and paste the file contents into a text editor in another window. You will need the file contents in a later step of this procedure.
### Create an IAM role for Systems Manager
An IAM role for instance credentials is required when you use Systems Manager Run Command. This role enables Systems Manager to perform actions on the instance. For more information, see [Configuring Security Roles for Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-access.html) in the *AWS Systems Manager User Guide*. For information about how to attach an IAM role to an existing instance, see [Attaching an IAM Role to an Instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/iam-roles-for-amazon-ec2.html#attach-iam-role) in the *Amazon EC2 User Guide*.
### Verify Systems Manager prerequisites
Before you use Systems Manager Run Command to configure integration with CloudWatch Logs, verify that your instances meet the minimum requirements. For more information, see [Systems Manager Prerequisites](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-setting-up.html) in the *AWS Systems Manager User Guide*.
### Verify internet access
Your Amazon EC2 Windows Server instances and managed instances must have outbound internet access in order to send log and event data to CloudWatch. For more information about how to configure internet access, see [Internet Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) in the *Amazon VPC User Guide*.
### Enable CloudWatch Logs using Systems Manager Run Command
Run Command enables you to manage the configuration of your instances on demand. You specify a Systems Manager document, specify parameters, and execute the command on one or more instances. The SSM agent on the instance processes the command and configures the instance as specified.
**To configure integration with CloudWatch Logs using Run Command**
1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/).
1. Open the SSM console at [https://console.aws.amazon.com/systems-manager/](https://console.aws.amazon.com/systems-manager/).
1. In the navigation pane, choose **Run Command**.
1. Choose **Run a command**.
1. For **Command document**, choose **AWS-ConfigureCloudWatch**.
1. For **Target instances**, choose the instances to integrate with CloudWatch Logs. If you do not see an instance in this list, it might not be configured for Run Command. For more information, see [Systems Manager Prerequisites](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/systems-manager-setting-up.html) in the *Amazon EC2 User Guide*.
1. For **Status**, choose **Enabled**.
1. For **Properties**, copy and paste the JSON content you created in the previous tasks.
1. Complete the remaining optional fields and choose **Run**.
Use the following procedure to view the results of command execution in the Amazon EC2 console.
**To view command output in the console**
1. Select a command.
1. Choose the **Output** tab.
1. Choose **View Output**. The command output page shows the results of your command execution.
# Quick Start: Enable your Amazon EC2 instances running Windows Server 2012 and Windows Server 2008 to send logs to CloudWatch Logs
**Tip**
CloudWatch includes a new unified agent that can collect both logs and metrics from EC2 instances and on-premises servers. We recommend that you use the newer unified CloudWatch agent. For more information, see [Getting started with CloudWatch Logs](CWL_GettingStarted.md).
The rest of this section explains the use of the older CloudWatch Logs agent.
## Enable your Amazon EC2 instances running Windows Server 2012 and Windows Server 2008 to send logs to CloudWatch Logs
Use the following steps to enable your instances running Windows Server 2012 and Windows Server 2008 to send logs to CloudWatch Logs.
### Download the sample configuration file
Download the following sample JSON file to your computer: [https://s3.amazonaws.com/ec2-downloads-windows/CloudWatchConfig/AWS.EC2.Windows.CloudWatch.json](https://s3.amazonaws.com/ec2-downloads-windows/CloudWatchConfig/AWS.EC2.Windows.CloudWatch.json). You edit it in the following steps.
### Configure the JSON file for CloudWatch
You determine which logs to send to CloudWatch by specifying your choices in the JSON configuration file. The process of creating this file and specifying your choices can take 30 minutes or more to complete. After you have completed this task once, you can reuse the configuration file on all of your instances.
**Topics**
+ [Step 1: Enable CloudWatch Logs](#enable-CloudWatchLogs-in-JSON-file2012)
+ [Step 2: Configure settings for CloudWatch](#configure_cwl_credentials2012)
+ [Step 3: Configure the data to send](#configure_logs2012)
+ [Step 4: Configure flow control](#configure_log_flow2012)
#### Step 1: Enable CloudWatch Logs
At the top of the JSON file, change "false" to "true" for `IsEnabled`:
```
"IsEnabled": true,
```
#### Step 2: Configure settings for CloudWatch
Specify credentials, Region, a log group name, and a log stream namespace. This enables the instance to send log data to CloudWatch Logs. To send the same log data to different locations, you can add additional sections with unique IDs (for example, "CloudWatchLogs2" and CloudWatchLogs3") and a different Region for each ID.
**To configure settings to send log data to CloudWatch Logs**
1. In the JSON file, locate the `CloudWatchLogs` section.
```
{
"Id": "CloudWatchLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"AccessKey": "",
"SecretKey": "",
"Region": "us-east-1",
"LogGroup": "Default-Log-Group",
"LogStream": "{instance_id}"
}
},
```
1. Leave the `AccessKey` and `SecretKey` field blank. You configure credentials using an IAM role.
1. For `Region`, type the Region to which to send log data (for example, `us-east-2`).
1. For `LogGroup`, type the name for your log group. This name appears on the **Log Groups** screen in the CloudWatch console.
1. For `LogStream`, type the destination log stream. This name appears on the **Log Groups > Streams** screen in the CloudWatch console.
If you use `{instance_id}`, the default, the log stream name is the instance ID of this instance.
If you specify a log stream name that doesn't already exist, CloudWatch Logs automatically creates it for you. You can define a log stream name using a literal string, the predefined variables `{instance_id}`, `{hostname}`, and `{ip_address}`, or a combination of these.
#### Step 3: Configure the data to send
You can send event log data, Event Tracing for Windows (ETW) data, and other log data to CloudWatch Logs.
**To send Windows application event log data to CloudWatch Logs**
1. In the JSON file, locate the `ApplicationEventLog` section.
```
{
"Id": "ApplicationEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Application",
"Levels": "1"
}
},
```
1. For `Levels`, specify the type of messages to upload. You can specify one of the following values:
+ **1** - Upload only error messages.
+ **2** - Upload only warning messages.
+ **4** - Upload only information messages.
You can combine values to include more than one type of message. For example, a value of **3** uploads error messages (**1**) and warning messages (**2**). A value of **7** uploads error messages (**1**), warning messages (**2**), and information messages (**4**).
**To send security log data to CloudWatch Logs**
1. In the JSON file, locate the `SecurityEventLog` section.
```
{
"Id": "SecurityEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Security",
"Levels": "7"
}
},
```
1. For `Levels`, type **7** to upload all messages.
**To send system event log data to CloudWatch Logs**
1. In the JSON file, locate the `SystemEventLog` section.
```
{
"Id": "SystemEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "System",
"Levels": "7"
}
},
```
1. For `Levels`, specify the type of messages to upload. You can specify one of the following values:
+ **1** - Upload only error messages.
+ **2** - Upload only warning messages.
+ **4** - Upload only information messages.
You can combine values to include more than one type of message. For example, a value of **3** uploads error messages (**1**) and warning messages (**2**). A value of **7** uploads error messages (**1**), warning messages (**2**), and information messages (**4**).
**To send other types of event log data to CloudWatch Logs**
1. In the JSON file, add a new section. Each section must have a unique `Id`.
```
{
"Id": "Id-name",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Log-name",
"Levels": "7"
}
},
```
1. For `Id`, type a name for the log to upload (for example, **WindowsBackup**).
1. For `LogName`, type the name of the log to upload. You can find the name of the log as follows.
1. Open Event Viewer.
1. In the navigation pane, choose **Applications and Services Logs**.
1. Navigate to the log, and then choose **Actions**, **Properties**.
1. For `Levels`, specify the type of messages to upload. You can specify one of the following values:
+ **1** - Upload only error messages.
+ **2** - Upload only warning messages.
+ **4** - Upload only information messages.
You can combine values to include more than one type of message. For example, a value of **3** uploads error messages (**1**) and warning messages (**2**). A value of **7** uploads error messages (**1**), warning messages (**2**), and information messages (**4**).
**To send Event Tracing for Windows data to CloudWatch Logs**
ETW (Event Tracing for Windows) provides an efficient and detailed logging mechanism that applications can write logs to. Each ETW is controlled by a session manager that can start and stop the logging session. Each session has a provider and one or more consumers.
1. In the JSON file, locate the `ETW` section.
```
{
"Id": "ETW",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Microsoft-Windows-WinINet/Analytic",
"Levels": "7"
}
},
```
1. For `LogName`, type the name of the log to upload.
1. For `Levels`, specify the type of messages to upload. You can specify one of the following values:
+ **1** - Upload only error messages.
+ **2** - Upload only warning messages.
+ **4** - Upload only information messages.
You can combine values to include more than one type of message. For example, a value of **3** uploads error messages (**1**) and warning messages (**2**). A value of **7** uploads error messages (**1**), warning messages (**2**), and information messages (**4**).
**To send custom logs (any text-based log file) to CloudWatch Logs**
1. In the JSON file, locate the `CustomLogs` section.
```
{
"Id": "CustomLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\CustomLogs\\",
"TimestampFormat": "MM/dd/yyyy HH:mm:ss",
"Encoding": "UTF-8",
"Filter": "",
"CultureName": "en-US",
"TimeZoneKind": "Local",
"LineCount": "5"
}
},
```
1. For `LogDirectoryPath`, type the path where logs are stored on your instance.
1. For `TimestampFormat`, type the time stamp format to use. For more information about supported values, see the [Custom Date and Time Format Strings](https://msdn.microsoft.com/en-us/library/8kb3ddd4(v=vs.110).aspx) topic on MSDN.
**Important**
Your source log file must have the time stamp at the beginning of each log line and there must be a space following the time stamp.
1. For `Encoding`, type the file encoding to use (for example, UTF-8). For more information about supported values, see the [Encoding Class](http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx) topic on MSDN.
**Note**
Use the encoding name, not the display name.
1. (Optional) For `Filter`, type the prefix of log names. Leave this parameter blank to monitor all files. For more information about supported values, see the [FileSystemWatcherFilter Property](https://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.filter.aspx) topic on MSDN.
1. (Optional) For `CultureName`, type the locale where the time stamp is logged. If `CultureName` is blank, it defaults to the same locale currently used by your Windows instance. For more information about supported values, see the `Language tag` column in the table in the [Product Behavior](https://msdn.microsoft.com/en-us/library/cc233982.aspx) topic on MSDN.
**Note**
The `div`, `div-MV`, `hu`, and `hu-HU` values are not supported.
1. (Optional) For `TimeZoneKind`, type `Local` or `UTC`. You can set this to provide time zone information when no time zone information is included in your log's time stamp. If this parameter is left blank and if your time stamp doesn't include time zone information, CloudWatch Logs defaults to the local time zone. This parameter is ignored if your time stamp already contains time zone information.
1. (Optional) For `LineCount`, type the number of lines in the header to identify the log file. For example, IIS log files have virtually identical headers. You could enter **5**, which would read the first three lines of the log file header to identify it. In IIS log files, the third line is the date and time stamp, but the time stamp is not always guaranteed to be different between log files. For this reason, we recommend including at least one line of actual log data to uniquely fingerprint the log file.
**To send IIS log data to CloudWatch Logs**
1. In the JSON file, locate the `IISLog` section.
```
{
"Id": "IISLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\inetpub\\logs\\LogFiles\\W3SVC1",
"TimestampFormat": "yyyy-MM-dd HH:mm:ss",
"Encoding": "UTF-8",
"Filter": "",
"CultureName": "en-US",
"TimeZoneKind": "UTC",
"LineCount": "5"
}
},
```
1. For `LogDirectoryPath`, type the folder where IIS logs are stored for an individual site (for example, `C:\inetpub\logs\LogFiles\W3SVCn`).
**Note**
Only W3C log format is supported. IIS, NCSA, and Custom formats are not supported.
1. For `TimestampFormat`, type the time stamp format to use. For more information about supported values, see the [Custom Date and Time Format Strings](https://msdn.microsoft.com/en-us/library/8kb3ddd4(v=vs.110).aspx) topic on MSDN.
1. For `Encoding`, type the file encoding to use (for example, UTF-8). For more information about supported values, see the [Encoding Class](http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx) topic on MSDN.
**Note**
Use the encoding name, not the display name.
1. (Optional) For `Filter`, type the prefix of log names. Leave this parameter blank to monitor all files. For more information about supported values, see the [FileSystemWatcherFilter Property](https://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.filter.aspx) topic on MSDN.
1. (Optional) For `CultureName`, type the locale where the time stamp is logged. If `CultureName` is blank, it defaults to the same locale currently used by your Windows instance. For more information about supported values, see the `Language tag` column in the table in the [Product Behavior](https://msdn.microsoft.com/en-us/library/cc233982.aspx) topic on MSDN.
**Note**
The `div`, `div-MV`, `hu`, and `hu-HU` values are not supported.
1. (Optional) For `TimeZoneKind`, enter `Local` or `UTC`. You can set this to provide time zone information when no time zone information is included in your log's time stamp. If this parameter is left blank and if your time stamp doesn't include time zone information, CloudWatch Logs defaults to the local time zone. This parameter is ignored if your time stamp already contains time zone information.
1. (Optional) For `LineCount`, type the number of lines in the header to identify the log file. For example, IIS log files have virtually identical headers. You could enter **5**, which would read the first five lines of the log file's header to identify it. In IIS log files, the third line is the date and time stamp, but the time stamp is not always guaranteed to be different between log files. For this reason, we recommend including at least one line of actual log data for uniquely fingerprinting the log file.
#### Step 4: Configure flow control
Each data type must have a corresponding destination in the `Flows` section. For example, to send the custom log, ETW log, and system log to CloudWatch Logs, add `(CustomLogs,ETW,SystemEventLog),CloudWatchLogs` to the `Flows` section.
**Warning**
Adding a step that is not valid blocks the flow. For example, if you add a disk metric step, but your instance doesn't have a disk, all steps in the flow are blocked.
You can send the same log file to more than one destination. For example, to send the application log to two different destinations that you defined in the `CloudWatchLogs` section, add `ApplicationEventLog,(CloudWatchLogs,CloudWatchLogs2)` to the `Flows` section.
**To configure flow control**
1. In the `AWS.EC2.Windows.CloudWatch.json` file, locate the `Flows` section.
```
"Flows": {
"Flows": [
"PerformanceCounter,CloudWatch",
"(PerformanceCounter,PerformanceCounter2), CloudWatch2",
"(CustomLogs, ETW, SystemEventLog),CloudWatchLogs",
"CustomLogs, CloudWatchLogs2",
"ApplicationEventLog,(CloudWatchLogs, CloudWatchLogs2)"
]
}
```
1. For `Flows`, add each data type that is to be uploaded (for example, `ApplicationEventLog`) and its destination (for example, `CloudWatchLogs`).
You are now finished editing the JSON file. You use it in a later step.
### Start the agent
To enable an Amazon EC2 instance running Windows Server 2012 or Windows Server 2008 to send logs to CloudWatch Logs, use the EC2Config service (`EC2Config.exe)`. Your instance should have EC2Config 4.0 or later, and you can use this procedure.
**To configure CloudWatch using EC2Config 4.x**
1. Check the encoding of the `AWS.EC2.Windows.CloudWatch.json` file that you edited earlier in this procedure. Only UTF-8 without BOM encoding is supported. Then save the file in the following folder on your Windows Server 2008 - 2012 R2 instance: `C:\Program Files\Amazon\SSM\Plugins\awsCloudWatch\`.
1. Start or restart the SSM agent (`AmazonSSMAgent.exe`) using the Windows Services control panel or using the following PowerShell command:
```
PS C:\> Restart-Service AmazonSSMAgent
```
After the SSM agent restarts, it detects the configuration file and configures the instance for CloudWatch integration. If you change parameters and settings in the local configuration file, you need to restart the SSM agent to pick up the changes. To disable CloudWatch integration on the instance, change `IsEnabled` to `false` and save your changes in the configuration file.
# Report the CloudWatch Logs agent status
Use the following procedure to report the status of the CloudWatch Logs agent on your EC2 instance.
**To report the agent status**
1. Connect to your EC2 instance. For more information, see [Connect to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-connect-to-instance-linux.html) in the *Amazon EC2 User Guide*.
For more information about connection issues, see [Troubleshooting Connecting to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) in the *Amazon EC2 User Guide*
1. At a command prompt, type the following command:
```
sudo service awslogs status
```
If you are running Amazon Linux 2, type the following command:
```
sudo service awslogsd status
```
1. Check the **/var/log/awslogs.log** file for any errors, warnings, or issues with the CloudWatch Logs agent.
# Start the CloudWatch Logs agent
If the CloudWatch Logs agent on your EC2 instance did not start automatically after installation, or if you stopped the agent, you can use the following procedure to start the agent.
**To start the agent**
1. Connect to your EC2 instance. For more information, see [Connect to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-connect-to-instance-linux.html) in the *Amazon EC2 User Guide*.
For more information about connection issues, see [Troubleshooting Connecting to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) in the *Amazon EC2 User Guide*.
1. At a command prompt, type the following command:
```
sudo service awslogs start
```
If you are running Amazon Linux 2, type the following command:
```
sudo service awslogsd start
```
# Stop the CloudWatch Logs agent
Use the following procedure to stop the CloudWatch Logs agent on your EC2 instance.
**To stop the agent**
1. Connect to your EC2 instance. For more information, see [Connect to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-connect-to-instance-linux.html) in the *Amazon EC2 User Guide*.
For more information about connection issues, see [Troubleshooting Connecting to Your Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) in the *Amazon EC2 User Guide*.
1. At a command prompt, type the following command:
```
sudo service awslogs stop
```
If you are running Amazon Linux 2, type the following command:
```
sudo service awslogsd stop
```
# CloudWatch Logs agent reference
**Important**
This section is a reference for those using the deprecated CloudWatch Logs agent. If you're using Instance Metadata Service Version 2 (IMDSv2), you must use the new unified CloudWatch agent. However, even if you're not using IMDSv2, we strongly recommend using the newer unified CloudWatch agent instead of the deprecated CloudWatch Logs agent. For information about the newer unified CloudWatch agent, see [Collecting metrics and logs from Amazon EC2 instance and on-premises servers with the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html). For information about migrating from the deprecated CloudWatch Logs agent to the unified agent, [Create the CloudWatch agent configuration file with the wizard](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html).
The CloudWatch Logs agent provides an automated way to send log data to CloudWatch Logs from Amazon EC2 instances. The agent includes the following components:
+ A plug-in to the AWS CLI that pushes log data to CloudWatch Logs.
+ A script (daemon) that initiates the process to push data to CloudWatch Logs.
+ A cron job that ensures that the daemon is always running.
## Agent configuration file
The CloudWatch Logs agent configuration file describes information needed by the CloudWatch Logs agent. The agent configuration file's [general] section defines common configurations that apply to all log streams. The [logstream] section defines the information necessary to send a local file to a remote log stream. You can have more than one [logstream] section, but each must have a unique name within the configuration file, e.g., [logstream1], [logstream2], and so on. The [logstream] value along with the first line of data in the log file, define the log file's identity.
```
[general]
state_file = value
logging_config_file = value
use_gzip_http_content_encoding = [true | false]
[logstream1]
log_group_name = value
log_stream_name = value
datetime_format = value
time_zone = [LOCAL|UTC]
file = value
file_fingerprint_lines = integer | integer-integer
multi_line_start_pattern = regex | {datetime_format}
initial_position = [start_of_file | end_of_file]
encoding = [ascii|utf_8|..]
buffer_duration = integer
batch_count = integer
batch_size = integer
[logstream2]
...
```
**state\$1file**
Specifies where the state file is stored.
**logging\$1config\$1file**
(Optional) Specifies the location of the agent logging config file. If you do not specify an agent logging config file here, the default file awslogs.conf is used. The default file location is `/var/awslogs/etc/awslogs.conf` if you installed the agent with a script, and is `/etc/awslogs/awslogs.conf` if you installed the agent with rpm. The file is in Python configuration file format (https://docs.python.org/2/library/logging.config.html\$1logging-config-fileformat). Loggers with the following names can be customized.
```
cwlogs.push
cwlogs.push.reader
cwlogs.push.publisher
cwlogs.push.event
cwlogs.push.batch
cwlogs.push.stream
cwlogs.push.watcher
```
The sample below changes the level of reader and publisher to WARNING while the default value is INFO.
```
[loggers]
keys=root,cwlogs,reader,publisher
[handlers]
keys=consoleHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=INFO
handlers=consoleHandler
[logger_cwlogs]
level=INFO
handlers=consoleHandler
qualname=cwlogs.push
propagate=0
[logger_reader]
level=WARNING
handlers=consoleHandler
qualname=cwlogs.push.reader
propagate=0
[logger_publisher]
level=WARNING
handlers=consoleHandler
qualname=cwlogs.push.publisher
propagate=0
[handler_consoleHandler]
class=logging.StreamHandler
level=INFO
formatter=simpleFormatter
args=(sys.stderr,)
[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(process)d - %(threadName)s - %(message)s
```
**use\$1gzip\$1http\$1content\$1encoding**
When set to true (default), enables gzip http content encoding to send compressed payloads to CloudWatch Logs. This decreases CPU usage, lowers NetworkOut, and decreases put latency. To disable this feature, add **use\$1gzip\$1http\$1content\$1encoding = false** to the **[general]** section of the CloudWatch Logs agent configuration file, and then restart the agent.
This setting is only available in awscli-cwlogs version 1.3.3 and later.
**log\$1group\$1name**
Specifies the destination log group. A log group is created automatically if it doesn't already exist. Log group names can be between 1 and 512 characters long. Allowed characters include a-z, A-Z, 0-9, '\$1' (underscore), '-' (hyphen), '/' (forward slash), and '.' (period).
**log\$1stream\$1name**
Specifies the destination log stream. You can use a literal string or predefined variables (\$1instance\$1id\$1, \$1hostname\$1, \$1ip\$1address\$1), or combination of both to define a log stream name. A log stream is created automatically if it doesn't already exist.
**datetime\$1format**
Specifies how the timestamp is extracted from logs. The timestamp is used for retrieving log events and generating metrics. The current time is used for each log event if the **datetime\$1format** isn't provided. If the provided **datetime\$1format** value is invalid for a given log message, the timestamp from the last log event with a successfully parsed timestamp is used. If no previous log events exist, the current time is used.
The common datetime\$1format codes are listed below. You can also use any datetime\$1format codes supported by Python, datetime.strptime(). The timezone offset (%z) is also supported even though it's not supported until python 3.2, [\$1-]HHMM without colon(:). For more information, see [strftime() and strptime() Behavior](https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior).
**%y**: Year without century as a zero-padded decimal number. 00, 01, ..., 99
**%Y**: Year with century as a decimal number.1970, 1988, 2001, 2013
**%b**: Month as locale's abbreviated name. Jan, Feb, ..., Dec (en\$1US);
**%B**: Month as locale's full name. January, February, ..., December (en\$1US);
**%m**: Month as a zero-padded decimal number. 01, 02, ..., 12
**%d**: Day of the month as a zero-padded decimal number. 01, 02, ..., 31
**%H**: Hour (24-hour clock) as a zero-padded decimal number. 00, 01, ..., 23
**%I**: Hour (12-hour clock) as a zero-padded decimal number. 01, 02, ..., 12
**%p**: Locale's equivalent of either AM or PM.
**%M**: Minute as a zero-padded decimal number. 00, 01, ..., 59
**%S**: Second as a zero-padded decimal number. 00, 01, ..., 59
**%f**: Microsecond as a decimal number, zero-padded on the left. 000000, ..., 999999
**%z**: UTC offset in the form \$1HHMM or -HHMM. \$10000, -0400, \$11030
**Example formats:**
`Syslog: '%b %d %H:%M:%S', e.g. Jan 23 20:59:29`
`Log4j: '%d %b %Y %H:%M:%S', e.g. 24 Jan 2014 05:00:00`
`ISO8601: '%Y-%m-%dT%H:%M:%S%z', e.g. 2014-02-20T05:20:20+0000`
**time\$1zone**
Specifies the time zone of log event timestamp. The two supported values are UTC and LOCAL. The default is LOCAL, which is used if time zone can't be inferred based on **datetime\$1format**.
**file**
Specifies log files that you want to push to CloudWatch Logs. File can point to a specific file or multiple files (using wildcards such as /var/log/system.log\$1). Only the latest file is pushed to CloudWatch Logs based on file modification time. We recommend that you use wildcards to specify a series of files of the same type, such as access\$1log.2014-06-01-01, access\$1log.2014-06-01-02, and so on, but not multiple kinds of files, such as access\$1log\$180 and access\$1log\$1443. To specify multiple kinds of files, add another log stream entry to the configuration file so each kind of log file goes to a different log stream. Zipped files are not supported.
**file\$1fingerprint\$1lines**
Specifies the range of lines for identifying a file. The valid values are one number or two dash delimited numbers, such as '1', '2-5'. The default value is '1' so the first line is used to calculate fingerprint. Fingerprint lines are not sent to CloudWatch Logs unless all the specified lines are available.
**multi\$1line\$1start\$1pattern**
Specifies the pattern for identifying the start of a log message. A log message is made of a line that matches the pattern and any following lines that don't match the pattern. The valid values are regular expression or \$1datetime\$1format\$1. When using \$1datetime\$1format\$1, the datetime\$1format option should be specified. The default value is ‘^[^\$1s]' so any line that begins with non-whitespace character closes the previous log message and starts a new log message.
**initial\$1position**
Specifies where to start to read data (start\$1of\$1file or end\$1of\$1file). The default is start\$1of\$1file. It's only used if there is no state persisted for that log stream.
**encoding**
Specifies the encoding of the log file so that the file can be read correctly. The default is utf\$18. Encodings supported by Python codecs.decode() can be used here.
Specifying an incorrect encoding might cause data loss because characters that cannot be decoded are replaced with some other character.
Below are some common encodings:
`ascii, big5, big5hkscs, cp037, cp424, cp437, cp500, cp720, cp737, cp775, cp850, cp852, cp855, cp856, cp857, cp858, cp860, cp861, cp862, cp863, cp864, cp865, cp866, cp869, cp874, cp875, cp932, cp949, cp950, cp1006, cp1026, cp1140, cp1250, cp1251, cp1252, cp1253, cp1254, cp1255, cp1256, cp1257, cp1258, euc_jp, euc_jis_2004, euc_jisx0213, euc_kr, gb2312, gbk, gb18030, hz, iso2022_jp, iso2022_jp_1, iso2022_jp_2, iso2022_jp_2004, iso2022_jp_3, iso2022_jp_ext, iso2022_kr, latin_1, iso8859_2, iso8859_3, iso8859_4, iso8859_5, iso8859_6, iso8859_7, iso8859_8, iso8859_9, iso8859_10, iso8859_13, iso8859_14, iso8859_15, iso8859_16, johab, koi8_r, koi8_u, mac_cyrillic, mac_greek, mac_iceland, mac_latin2, mac_roman, mac_turkish, ptcp154, shift_jis, shift_jis_2004, shift_jisx0213, utf_32, utf_32_be, utf_32_le, utf_16, utf_16_be, utf_16_le, utf_7, utf_8, utf_8_sig`
**buffer\$1duration**
Specifies the time duration for the batching of log events. The minimum value is 5000ms and default value is 5000ms.
**batch\$1count**
Specifies the max number of log events in a batch, up to 10000. The default value is 10000.
**batch\$1size**
Specifies the max size of log events in a batch, in bytes, up to 1048576 bytes. The default value is 1048576 bytes. This size is calculated as the sum of all event messages in UTF-8, plus 26 bytes for each log event.
## Using the CloudWatch Logs agent with HTTP proxies
You can use the CloudWatch Logs agent with HTTP proxies.
**Note**
HTTP proxies are supported in awslogs-agent-setup.py version 1.3.8 or later.
**To use the CloudWatch Logs agent with HTTP proxies**
1. Do one of the following:
1. For a new installation of the CloudWatch Logs agent, run the following commands:
```
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
```
```
sudo python awslogs-agent-setup.py --region us-east-1 --http-proxy http://your/proxy --https-proxy http://your/proxy --no-proxy 169.254.169.254
```
In order to maintain access to the Amazon EC2 metadata service on EC2 instances, use **--no-proxy 169.254.169.254** (recommended). For more information, see [Instance Metadata and User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) in the *Amazon EC2 User Guide*.
In the values for `http-proxy` and `https-proxy`, you specify the entire URL.
1. For an existing installation of the CloudWatch Logs agent, edit /var/awslogs/etc/proxy.conf, and add your proxies:
```
HTTP_PROXY=
HTTPS_PROXY=
NO_PROXY=
```
1. Restart the agent for the changes to take effect:
```
sudo service awslogs restart
```
If you are using Amazon Linux 2, use the following command to restart the agent:
```
sudo service awslogsd restart
```
## Compartmentalizing CloudWatch Logs agent configuration files
If you're using awslogs-agent-setup.py version 1.3.8 or later with awscli-cwlogs 1.3.3 or later, you can import different stream configurations for various components independently of one another by creating additional configuration files in the **/var/awslogs/etc/config/** directory. When the CloudWatch Logs agent starts, it includes any stream configurations in these additional configuration files. Configuration properties in the [general] section must be defined in the main configuration file (/var/awslogs/etc/awslogs.conf) and are ignored in any additional configuration files found in /var/awslogs/etc/config/.
If you don't have a **/var/awslogs/etc/config/** directory because you installed the agent with rpm, you can use the **/etc/awslogs/config/** directory instead.
Restart the agent for the changes to take effect:
```
sudo service awslogs restart
```
If you are using Amazon Linux 2, use the following command to restart the agent:
```
sudo service awslogsd restart
```
## CloudWatch Logs agent FAQ
**What kinds of file rotations are supported?**
The following file rotation mechanisms are supported:
+ Renaming existing log files with a numerical suffix, then re-creating the original empty log file. For example, /var/log/syslog.log is renamed /var/log/syslog.log.1. If /var/log/syslog.log.1 already exists from a previous rotation, it is renamed /var/log/syslog.log.2.
+ Truncating the original log file in place after creating a copy. For example, /var/log/syslog.log is copied to /var/log/syslog.log.1 and /var/log/syslog.log is truncated. There might be data loss for this case, so be careful about using this file rotation mechanism.
+ Creating a new file with a common pattern as the old one. For example, /var/log/syslog.log.2014-01-01 remains and /var/log/syslog.log.2014-01-02 is created.
The fingerprint (source ID) of the file is calculated by hashing the log stream key and the first line of file content. To override this behavior, the **file\$1fingerprint\$1lines** option can be used. When file rotation happens, the new file is supposed to have new content and the old file is not supposed to have content appended; the agent pushes the new file after it finishes reading the old file.
**How can I determine which version of agent am I using?**
If you used a setup script to install the CloudWatch Logs agent, you can use **/var/awslogs/bin/awslogs-version.sh** to check what version of the agent you are using. It prints out the version of the agent and its major dependencies. If you used yum to install the CloudWatch Logs agent, you can use **"yum info awslogs"** and **"yum info aws-cli-plugin-cloudwatch-logs"** to check the version of the CloudWatch Logs agent and plugin.
**How are log entries converted to log events?**
Log events contain two properties: the timestamp of when the event occurred, and the raw log message. By default, any line that begins with non-whitespace character closes the previous log message if there is one, and starts a new log message. To override this behavior, the **multi\$1line\$1start\$1pattern** can be used and any line that matches the pattern starts a new log message. The pattern could be any regex or '\$1datetime\$1format\$1'. For example, if the first line of every log message contains a timestamp like '2014-01-02T13:13:01Z', then the **multi\$1line\$1start\$1pattern** can be set to '\$1d\$14\$1-\$1d\$12\$1-\$1d\$12\$1T\$1d\$12\$1:\$1d\$12\$1:\$1d\$12\$1Z'. To simplify the configuration, the ‘\$1datetime\$1format\$1' variable can be used if the **datetime\$1format option** is specified. For the same example, if **datetime\$1format** is set to '%Y-%m-%dT%H:%M:%S%z', then multi\$1line\$1start\$1pattern could be simply '\$1datetime\$1format\$1'.
The current time is used for each log event if the **datetime\$1format** isn't provided. If the provided **datetime\$1format** is invalid for a given log message, the timestamp from the last log event with a successfully parsed timestamp is used. If no previous log events exist, the current time is used. A warning message is logged when a log event falls back to the current time or time of previous log event.
Timestamps are used for retrieving log events and generating metrics, so if you specify the wrong format, log events could become non-retrievable and generate wrong metrics.
**How are log events batched?**
A batch becomes full and is published when any of the following conditions are met:
1. The **buffer\$1duration** amount of time has passed since the first log event was added.
1. Less than **batch\$1size** of log events have been accumulated but adding the new log event exceeds the **batch\$1size**.
1. The number of log events has reached **batch\$1count**.
1. Log events from the batch don't span more than 24 hours, but adding the new log event exceeds the 24 hours constraint.
**What would cause log entries, log events, or batches to be skipped or truncated?**
To follow the constraint of the `PutLogEvents` operation, the following issues could cause a log event or batch to be skipped.
The CloudWatch Logs agent writes a warning to its log when data is skipped.
1. If the size of a log event exceeds 256 KB, the log event is skipped completely.
1. If the timestamp of log event is more than 2 hours in future, the log event is skipped.
1. If the timestamp of log event is more than 14 days in past, the log event is skipped.
1. If any log event is older than the retention period of log group, the whole batch is skipped.
1. If the batch of log events in a single `PutLogEvents` request spans more than 24 hours, the `PutLogEvents` operation fails.
**Does stopping the agent cause data loss/duplicates?**
Not as long as the state file is available and no file rotation has happened since the last run. The CloudWatch Logs agent can start from where it stopped and continue pushing the log data.
**Can I point different log files from the same or different hosts to the same log stream?**
Configuring multiple log sources to send data to a single log stream is not supported.
**What API calls does the agent make (or what actions should I add to my IAM policy)?**
The CloudWatch Logs agent requires permission to perform `CreateLogGroup`, `CreateLogStream`, `DescribeLogStreams`, `DescribeLogGrooupd`, `PutLogEvents` and `PutRetentionPolicy` actions. If you're using the latest agent, `DescribeLogStreams` is not needed. See the sample IAM policy below.
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"logs:PutRetentionPolicy"
],
"Resource": [
"arn:aws:logs:*:*:*"
]
}
]
}
```
**I don't want the CloudWatch Logs agent to create either log groups or log streams automatically. How can I prevent the agent from recreating both log groups and log streams?**
In your IAM policy, you can restrict the agent to only the following operations: `DescribeLogStreams`, `PutLogEvents`.
Before you revoke the `CreateLogGroup` and `CreateLogStream` permissions from the agent, be sure to create both the log groups and log streams that you want the agent to use. The logs agent cannot create log streams in a log group that you have created unless it has both the `CreateLogGroup` and `CreateLogStream` permissions.
**What logs should I look at when troubleshooting?**
The agent installation log is at `/var/log/awslogs-agent-setup.log` and the agent log is at `/var/log/awslogs.log`.
# Quick Start: Use CloudFormation to get started with CloudWatch Logs
AWS CloudFormation enables you to describe and provision your AWS resources in JSON format. The advantages of this method include being able to manage a collection of AWS resources as a single unit, and easily replicating your AWS resources across Regions.
When you provision AWS using CloudFormation, you create templates that describe the AWS resources to use. The following example is a template snippet that creates a log group and a metric filter that counts 404 occurrences and sends this count to the log group.
```
"WebServerLogGroup": {
"Type": "AWS::Logs::LogGroup",
"Properties": {
"RetentionInDays": 7
}
},
"404MetricFilter": {
"Type": "AWS::Logs::MetricFilter",
"Properties": {
"LogGroupName": {
"Ref": "WebServerLogGroup"
},
"FilterPattern": "[ip, identity, user_id, timestamp, request, status_code = 404, size, ...]",
"MetricTransformations": [
{
"MetricValue": "1",
"MetricNamespace": "test/404s",
"MetricName": "test404Count"
}
]
}
}
```
This is a basic example. You can set up much richer CloudWatch Logs deployments using CloudFormation. For more information about template examples, see [Amazon CloudWatch Logs Template Snippets](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-cloudwatchlogs.html) in the *AWS CloudFormation User Guide*. For more information about getting started, see [Getting Started with AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/GettingStarted.html) in the *AWS CloudFormation User Guide*.