Amazon ECS container agent
The Amazon ECS agent is a process that runs on every container instance that is registered with your cluster. It facilitates the communication between your container instances and Amazon ECS.
Note
On Linux container instances, the agent container mounts top-level directories such as
/lib
, /lib64
, and /proc
. This is necessary
for ECS features and functionalities such as Amazon EBS volumes, awsvpc
network
mode, Amazon ECS Service Connect, and FireLens for Amazon ECS.
Each Amazon ECS container agent version supports a different feature set and provides bug fixes from previous versions. When possible, we always recommend using the latest version of the Amazon ECS container agent. To update your container agent to the latest version, see Updating the Amazon ECS container agent.
To see which features and enhancements are included with each agent release, see https://github.com/aws/amazon-ecs-agent/releases
Important
The minimum Docker version for reliable metrics is Docker version v20.10.13
and
newer, which is included in Amazon ECS-optimized AMI 20220607
and newer.
Amazon ECS agent versions 1.20.0
and newer have deprecated support for
Docker versions older than 1.9.0
.
Lifecycle
When the Amazon ECS container agent registers an Amazon EC2 instance to your cluster, the Amazon EC2
instance reports its status as ACTIVE
and its agent connection status as
TRUE
. This container instance can accept run task requests.
If you stop (not terminate) a container instance, the status remains
ACTIVE
, but the agent connection status transitions to
FALSE
within a few minutes. Any tasks that were running on the
container instance stop. If you start the container instance again, the container agent
reconnects with the Amazon ECS service, and you are able to run tasks on the instance
again.
Important
If you stop and start a container instance, or reboot that instance, some older
versions of the Amazon ECS container agent register the instance again without
deregistering the original container instance ID. In this case, Amazon ECS lists more
container instances in your cluster than you actually have. (If you have duplicate
container instance IDs for the same Amazon EC2 instance ID, you can safely deregister the
duplicates that are listed as ACTIVE
with an agent connection status of
FALSE
.) This issue is fixed in the current version of the Amazon ECS
container agent. For more information about updating to the current version, see
Updating the Amazon ECS container agent.
If you change the status of a container instance to DRAINING
, new tasks
are not placed on the container instance. Any service tasks running on the container
instance are removed, if possible, so that you can perform system updates. For more
information, see Draining Amazon ECS container instances.
If you deregister or terminate a container instance, the container instance status
changes to INACTIVE
immediately, and the container instance is no longer
reported when you list your container instances. However, you can still describe the
container instance for one hour following termination. After one hour, the instance
description is no longer available.
Important
You can drain the instances manually, or build an Auto Scaling group lifecycle hook to
set the instance status to DRAINING
. See Amazon EC2 Auto Scaling lifecycle
hooks for more information about Auto Scaling lifecycle hooks.
Amazon ECS-optimized AMI
The Linux variants of the Amazon ECS-optimized AMI use the Amazon Linux 2 AMI as their base. The Amazon Linux 2 source AMI name for each variant can be retrieved by querying the Systems Manager Parameter Store API. For more information, see Retrieving Amazon ECS-optimized Linux AMI metadata. When you launch our container instances from the most recent Amazon ECS-optimized Amazon Linux 2 AMI you receive the current container agent version. To launch a container instance with the latest Amazon ECS-optimized Amazon Linux 2 AMI, see Launching an Amazon ECS Linux container instance.
Additional information
The following pages provide additional information about the changes:
-
Amazon ECS Agent changelog
on GitHub -
The source code for the
ecs-init
application and the scripts and configuration for packaging the agent are now part of the agent repository. For older versions ofecs-init
and packaging, see Amazon ecs-init changelogon GitHub -
Docker Engine release notes
in the Docker documentation -
NVIDIA Driver Documentation
in the NVIDIA documentation
Amazon ECS container agent log configuration parameters
The Amazon ECS container agent stores logs on your container instances.
For container agent version 1.36.0 and later, by default the logs are located at
/var/log/ecs/ecs-agent.log
on Linux instances and at
C:\ProgramData\Amazon\ECS\log\ecs-agent.log
on Windows
instances.
For container agent version 1.35.0 and earlier, by default the logs are located at
/var/log/ecs/ecs-agent.log.
on Linux instances and at
timestamp
C:\ProgramData\Amazon\ECS\log\ecs-agent.log.
on Windows instances.timestamp
By default, the agent logs are rotated hourly with a maximum of 24 logs being stored.
The following are the container agent configuration variables that can be used to change the default agent logging behavior. For more information, see Amazon ECS container agent configuration.
ECS_LOGFILE
-
Example values:
/ecs-agent.log
Default value on Linux: Null
Default value on Windows: Null
The location where agent logs should be written. If you are running the agent via
ecs-init
, which is the default method when using the Amazon ECS-optimized AMI, the in-container path is/log
, andecs-init
mounts that out to/var/log/ecs/
on the host. ECS_LOGLEVEL
-
Example values:
crit
,error
,warn
,info
,debug
Default value on Linux:
info
Default value on Windows:
info
The level of detail to log.
ECS_LOGLEVEL_ON_INSTANCE
-
Example values:
none
,crit
,error
,warn
,info
,debug
Default value on Linux:
none
, ifECS_LOG_DRIVER
is explicitly set to a non-empty value; otherwise the same value asECS_LOGLEVEL
Default value on Windows:
none
, ifECS_LOG_DRIVER
is explicitly set to a non-empty value; otherwise the same value asECS_LOGLEVEL
Can be used to override
ECS_LOGLEVEL
and set a level of detail that should be logged in the on-instance log file, separate from the level that is logged in the logging driver. If a logging driver is explicitly set, on-instance logs are turned off by default. They can be turned back on with this variable. ECS_LOG_DRIVER
-
Example values:
awslogs
,fluentd
,gelf
,json-file
,journald
,logentries
syslog
,splunk
Default value on Linux:
json-file
Default value on Windows: Not applicable
Determines the logging driver the agent container uses.
ECS_LOG_ROLLOVER_TYPE
-
Example values:
size
,hourly
Default value on Linux:
hourly
Default value on Windows:
hourly
Determines whether the container agent log file is rotated hourly or based on size. By default, the agent log file is rotated each hour.
ECS_LOG_OUTPUT_FORMAT
-
Example values:
logfmt
,json
Default value on Linux:
logfmt
Default value on Windows:
logfmt
Determines the log output format. When the
json
format is used, each line in the log is a structured JSON map. ECS_LOG_MAX_FILE_SIZE_MB
-
Example values:
10
Default value on Linux:
10
Default value on Windows:
10
When the
ECS_LOG_ROLLOVER_TYPE
variable is set tosize
, this variable determines the maximum size (in MB) of the log file before it's rotated. If the rollover type is set tohourly
, then this variable is ignored. ECS_LOG_MAX_ROLL_COUNT
-
Example values:
24
Default value on Linux:
24
Default value on Windows:
24
Determines the number of rotated log files to keep. Older log files are deleted after this limit is reached.
For container agent version 1.36.0 and later, the following is an example log file
when the logfmt
format is used.
level=info time=2019-12-12T23:43:29Z msg="Loading configuration" module=agent.go
level=info time=2019-12-12T23:43:29Z msg="Image excluded from cleanup: amazon/amazon-ecs-agent:latest" module=parse.go
level=info time=2019-12-12T23:43:29Z msg="Image excluded from cleanup: amazon/amazon-ecs-pause:0.1.0" module=parse.go
level=info time=2019-12-12T23:43:29Z msg="Amazon ECS agent Version: 1.36.0, Commit: ca640387" module=agent.go
level=info time=2019-12-12T23:43:29Z msg="Creating root ecs cgroup: /ecs" module=init_linux.go
level=info time=2019-12-12T23:43:29Z msg="Creating cgroup /ecs" module=cgroup_controller_linux.go
level=info time=2019-12-12T23:43:29Z msg="Loading state!" module=statemanager.go
level=info time=2019-12-12T23:43:29Z msg="Event stream ContainerChange start listening..." module=eventstream.go
level=info time=2019-12-12T23:43:29Z msg="Restored cluster 'auto-robc'" module=agent.go
level=info time=2019-12-12T23:43:29Z msg="Restored from checkpoint file. I am running as 'arn:aws:ecs:us-west-2:0123456789:container-instance/auto-robc/3330a8a91d15464ea30662d5840164cd' in cluster 'auto-robc'" module=agent.go
The following is an example log file when the JSON format is used.
{"time": "2019-11-07T22:52:02Z", "level": "info", "msg": "Starting Amazon Elastic Container Service Agent", "module": "engine.go"}
For container agent versions 1.35.0 and earlier, the following is the format of the log file.
2016-08-15T15:54:41Z [INFO] Starting Agent: Amazon ECS Agent - v1.12.0 (895f3c1)
2016-08-15T15:54:41Z [INFO] Loading configuration
2016-08-15T15:54:41Z [WARN] Invalid value for task cleanup duration, will be overridden to 3h0m0s, parsed value 0, minimum threshold 1m0s
2016-08-15T15:54:41Z [INFO] Checkpointing is enabled. Attempting to load state
2016-08-15T15:54:41Z [INFO] Loading state! module="statemanager"
2016-08-15T15:54:41Z [INFO] Detected Docker versions [1.17 1.18 1.19 1.20 1.21 1.22]
2016-08-15T15:54:41Z [INFO] Registering Instance with ECS
2016-08-15T15:54:41Z [INFO] Registered! module="api client"