# Automate responses to Amazon ECS errors using EventBridge
Using Amazon EventBridge, you can automate your AWS services and respond automatically to system events such as application availability issues or resource changes. Events from AWS services are delivered to EventBridge in near real time. You can write simple rules to indicate which events are of interest to you and what automated actions to take when an event matches a rule. The actions that can be automatically configured to include the following:
+ Adding events to log groups in CloudWatch Logs
+ Invoking an AWS Lambda function
+ Invoking Amazon EC2 Run Command
+ Relaying the event to Amazon Kinesis Data Streams
+ Activating an AWS Step Functions state machine
+ Notifying an Amazon SNS topic or an Amazon Simple Queue Service (Amazon SQS) queue
For more information, see [Getting started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-get-started.html) in the *Amazon EventBridge User Guide*.
You can use Amazon ECS events for EventBridge to receive near real-time notifications regarding the current state of your Amazon ECS clusters. If your tasks are using EC2, you can see the state of both the container instances and the current state of all tasks running on those container instances. If your tasks are using Fargate, you can see the state of the container instances.
Using EventBridge, you can build custom schedulers on top of Amazon ECS that are responsible for orchestrating tasks across clusters and monitoring the state of clusters in near real time. You can eliminate scheduling and monitoring code that continuously polls the Amazon ECS service for status changes and instead handle Amazon ECS state changes asynchronously using any EventBridge target. Targets might include AWS Lambda, Amazon Simple Queue Service, Amazon Simple Notification Service, or Amazon Kinesis Data Streams.
An Amazon ECS event stream ensures that every event is delivered at least one time. If duplicate events are sent, the event provides enough information to identify duplicates. For more information, see [Handling Amazon ECS events](ecs_cwet_handling.md).
Events are relatively ordered, so that you can easily tell when an event occurred in relation to other events.
**Topics**
+ [
# Amazon ECS events
](ecs_cwe_events.md)
+ [
# Handling Amazon ECS events
](ecs_cwet_handling.md)
# Amazon ECS events
Amazon ECS tracks the state of each of your tasks and services. If the state of a task or service changes, an event is generated and is sent to Amazon EventBridge. These events are classified as task state change events and service action events. These events and their possible causes are described in greater detail in the following sections.
Amazon ECS generates and sends the following types of events to EventBridge:
+ Container instance state change
+ Task state change
+ Deployment state change
+ Service action
**Note**
Amazon ECS might add other event types, sources, and details in the future. If you are de-serializing event JSON data in code, make sure that your application is prepared to handle unknown properties to avoid issues if and when these additional properties are added.
In some cases, multiple events are generated for the same activity. For example, when a task is started on a container instance, a task state change event is generated for the new task. A container instance state change event is generated to account for the change in available resources, such as CPU, memory, and available ports, on the container instance. Likewise, if a container instance is terminated, events are generated for the container instance, the container agent connection status, and every task that was running on the container instance.
Container state change and task state change events contain two `version` fields: one in the main body of the event, and one in the `detail` object of the event. The following describes the differences between these two fields:
+ The `version` field in the main body of the event is set to `0` on all events. For more information about EventBridge parameters, see [AWS service event metadata](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events-structure.html) in the *Amazon EventBridge User Guide*.
+ The `version` field in the `detail` object of the event describes the version of the associated resource. Each time a resource changes state, this version is incremented. Because events can be sent multiple times, this field allows you to identify duplicate events. Duplicate events have the same version in the `detail` object. If you are replicating your Amazon ECS container instance and task state with EventBridge, you can compare the version of a resource reported by the Amazon ECS APIs with the version reported in EventBridge for the resource (inside the `detail` object) to verify that the version in your event stream is current.
Service action events only contain the `version` field in the main body.
Service action events specify the service in 2 different fields:
+ For events generated by `create-service`, the service is in the `serviceName` field.
+ For events generated by `update-service`, the service is in the `service` field.
If you use automated tooling for service events, you need to code for both fields.
For information about how to create a rule for service action events, see [Amazon ECS service action events](ecs_service_events.md).
For additional information about how to integrate Amazon ECS and EventBridge, see [Integrating Amazon EventBridge and Amazon ECS](https://aws.amazon.com/blogs/compute/integrating-amazon-eventbridge-and-amazon-ecs/).
# Amazon ECS container instance state change events
The following scenarios cause container instance state change events:
You call the `StartTask`, `RunTask`, or `StopTask` API operations, either directly or with the AWS Management Console or SDKs.
Placing or stopping tasks on a container instance modifies the available resources on the container instance, such as CPU, memory, and available ports.
The Amazon ECS service scheduler starts or stops a task.
Placing or stopping tasks on a container instance modifies the available resources on the container instance, such as CPU, memory, and available ports.
The Amazon ECS container agent calls the `SubmitTaskStateChange` API operation with a `STOPPED` status for a task with a desired status of `RUNNING`.
The Amazon ECS container agent monitors the state of tasks on your container instances, and it reports any state changes. If a task that is supposed to be `RUNNING` is transitioned to `STOPPED`, the agent releases the resources that were allocated to the stopped task, such as CPU, memory, and available ports.
You deregister the container instance with the `DeregisterContainerInstance` API operation, either directly or with the AWS Management Console or SDKs.
Deregistering a container instance changes the status of the container instance and the connection status of the Amazon ECS container agent.
A task was stopped when an EC2 instance was stopped.
When you stop a container instance, the tasks that are running on it are transitioned to the `STOPPED` status.
The Amazon ECS container agent registers a container instance for the first time.
The first time the Amazon ECS container agent registers a container instance (at launch or when first run manually), this creates a state change event for the instance.
The Amazon ECS container agent connects or disconnects from Amazon ECS.
When the Amazon ECS container agent connects or disconnects from the Amazon ECS backend, it changes the `agentConnected` status of the container instance.
The Amazon ECS container agent disconnects and reconnects several times per hour as a part of its normal operation, so agent connection events should be expected. These events are not an indication that there is an issue with the container agent or your container instance.
You upgrade the Amazon ECS container agent on an instance.
The container instance detail contains an object for the container agent version. If you upgrade the agent, this version information changes and generates an event.
**Example Container instance state change event**
Container instance state change events are delivered in the following format. The `detail` section below resembles the [ContainerInstance](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerInstance.html) object that is returned from a [DescribeContainerInstances](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeContainerInstances.html) API operation in the *Amazon Elastic Container Service API Reference*. For more information about EventBridge parameters, see [AWS service event metadata](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events-structure.html) in the *Amazon EventBridge User Guide*.
```
{
"version": "0",
"id": "8952ba83-7be2-4ab5-9c32-6687532d15a2",
"detail-type": "ECS Container Instance State Change",
"source": "aws.ecs",
"account": "111122223333",
"time": "2016-12-06T16:41:06Z",
"region": "us-east-1",
"resources": [
"arn:aws:ecs:us-east-1:111122223333:container-instance/b54a2a04-046f-4331-9d74-3f6d7f6ca315"
],
"detail": {
"agentConnected": true,
"attributes": [
{
"name": "com.amazonaws.ecs.capability.logging-driver.syslog"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role-network-host"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.json-file"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
},
{
"name": "com.amazonaws.ecs.capability.privileged-container"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.20"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.22"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.23"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role"
}
],
"clusterArn": "arn:aws:ecs:us-east-1:111122223333:cluster/default",
"containerInstanceArn": "arn:aws:ecs:us-east-1:111122223333:container-instance/b54a2a04-046f-4331-9d74-3f6d7f6ca315",
"ec2InstanceId": "i-f3a8506b",
"registeredResources": [
{
"name": "CPU",
"type": "INTEGER",
"integerValue": 2048
},
{
"name": "MEMORY",
"type": "INTEGER",
"integerValue": 3767
},
{
"name": "PORTS",
"type": "STRINGSET",
"stringSetValue": [
"22",
"2376",
"2375",
"51678",
"51679"
]
},
{
"name": "PORTS_UDP",
"type": "STRINGSET",
"stringSetValue": []
}
],
"remainingResources": [
{
"name": "CPU",
"type": "INTEGER",
"integerValue": 1988
},
{
"name": "MEMORY",
"type": "INTEGER",
"integerValue": 767
},
{
"name": "PORTS",
"type": "STRINGSET",
"stringSetValue": [
"22",
"2376",
"2375",
"51678",
"51679"
]
},
{
"name": "PORTS_UDP",
"type": "STRINGSET",
"stringSetValue": []
}
],
"status": "ACTIVE",
"version": 14801,
"versionInfo": {
"agentHash": "aebcbca",
"agentVersion": "1.13.0",
"dockerVersion": "DockerVersion: 1.11.2"
},
"updatedAt": "2016-12-06T16:41:06.991Z"
}
}
```
# Amazon ECS task state change events
The following scenarios cause task state change events:
You call the `StartTask`, `RunTask`, or `StopTask` API operations, either directly or with the AWS Management Console, AWS CLI, or SDKs.
Starting or stopping tasks creates new task resources or modifies the state of existing task resources.
The Amazon ECS service scheduler starts or stops a task.
Starting or stopping tasks creates new task resources or modifies the state of existing task resources.
The Amazon ECS container agent calls the `SubmitTaskStateChange` API operation.
For EC2, the Amazon ECS container agent monitors the state of your tasks on your container instances. The Amazon ECS container agent reports any state changes. State changes might include changes from `PENDING` to `RUNNING` or from `RUNNING` to `STOPPED`.
You force deregistration of the underlying container instance with the `DeregisterContainerInstance` API operation and the `force` flag, either directly or with the AWS Management Console or SDKs.
Deregistering a container instance changes the status of the container instance and the connection status of the Amazon ECS container agent. If tasks are running on the container instance, the `force` flag must be set to allow deregistration. This stops all tasks on the instance.
The underlying container instance is stopped or terminated.
When you stop or terminate a container instance, the tasks that are running on it are transitioned to the `STOPPED` status.
A container in the task changes state.
The Amazon ECS container agent monitors the state of containers within tasks. For example, if a container that is running within a task stops, this container state change generates an event.
A task using the Fargate Spot capacity provider receives a termination notice.
When a task is using the `FARGATE_SPOT` capacity provider and is stopped due to a Spot interruption, a task state change event is generated.
**Example Task state change event**
Task state change events are delivered in the following format. Note the following about the fields:
+ The health status of the event is not available in the task state change event. If you need the task health status, you can run [describe-tasks](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeTasks.html).
+ When your containers use an image hosted with Amazon ECR, the `imageDigest` field is returned.
+ The values for the `createdAt`, `connectivityAt`, `pullStartedAt`, `startedAt`, `pullStoppedAt`, and `updatedAt` fields are ISO string timestamps.
+ The `detail-type` value is "ECS Task State Change".
+ When the event is generated for a stopped task, the `stoppedReason` and `stopCode` fields provide additional information about why the task stopped (for example, "User initiated").
For more information about EventBridge parameters, see [AWS service event metadata](https://docs.aws.amazon.com/eventbridge/latest/ref/events-structure.html) in the *Amazon EventBridge Events Reference*.
For information about how to configure an Amazon EventBridge event rule that only captures task events where the task has stopped running because one of its essential containers has terminated, see [Sending Amazon Simple Notification Service alerts for Amazon ECS task stopped events](ecs_cwet2.md)
```
{
"version": "0",
"id": "105f6bb1-4da6-c630-4965-35383018cbca",
"detail-type": "ECS Task State Change",
"source": "aws.ecs",
"account": "123456789012",
"time": "2025-05-06T11:02:34Z",
"region": "us-east-1",
"resources": [
"arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9"
],
"detail": {
"attachments": [
{
"id": "fe3a9a46-6a47-40ee-afd9-7952ae90a75a",
"type": "eni",
"status": "ATTACHED",
"details": [
{
"name": "subnetId",
"value": "subnet-0d0eab1bb38d5ca64"
},
{
"name": "networkInterfaceId",
"value": "eni-0103a2f01bad57d71"
},
{
"name": "macAddress",
"value": "0e:50:d1:c1:77:81"
},
{
"name": "privateDnsName",
"value": "ip-10-0-1-163.ec2.internal"
},
{
"name": "privateIPv4Address",
"value": "10.0.1.163"
}
]
}
],
"attributes": [
{
"name": "ecs.cpu-architecture",
"value": "x86_64"
}
],
"availabilityZone": "us-east-1b",
"capacityProviderName": "FARGATE",
"clusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/example-cluster",
"connectivity": "CONNECTED",
"connectivityAt": "2025-05-06T11:02:17.19Z",
"containers": [
{
"containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example-cluster/a1173316d40a45dea9/a0a99b87-baa8-4bf6-b9f1-a9a95917a635",
"lastStatus": "RUNNING",
"name": "web",
"image": "nginx",
"imageDigest": "sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66",
"runtimeId": "a1173316d40a45dea9-0265927825",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9",
"networkInterfaces": [
{
"attachmentId": "fe3a9a46-6a47-40ee-afd9-7952ae90a75a",
"privateIpv4Address": "10.0.1.163"
}
],
"cpu": "99",
"memory": "100"
},
{
"containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example-cluster/a1173316d40a45dea9/a2010e2d-ba7c-4135-8b79-e0290ff3cd8c",
"lastStatus": "RUNNING",
"name": "aws-guardduty-agent-nm40lC",
"imageDigest": "sha256:bf9197abdf853607e5fa392b4f97ccdd6ca56dd179be3ce8849e552d96582ac8",
"runtimeId": "a1173316d40a45dea9-2098416933",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9",
"networkInterfaces": [
{
"attachmentId": "fe3a9a46-6a47-40ee-afd9-7952ae90a75a",
"privateIpv4Address": "10.0.1.163"
}
],
"cpu": "null"
},
{
"containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example-cluster/a1173316d40a45dea9/dccf0ca2-d929-471f-a5c3-98006fd4379e",
"lastStatus": "RUNNING",
"name": "aws-otel-collector",
"image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.32.0",
"imageDigest": "sha256:7a1b3560655071bcacd66902c20ebe9a69470d5691fe3bd36baace7c2f3c4640",
"runtimeId": "a1173316d40a45dea9-4027662657",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9",
"networkInterfaces": [
{
"attachmentId": "fe3a9a46-6a47-40ee-afd9-7952ae90a75a",
"privateIpv4Address": "10.0.1.163"
}
],
"cpu": "0"
}
],
"cpu": "256",
"createdAt": "2025-05-06T11:02:13.877Z",
"desiredStatus": "RUNNING",
"enableExecuteCommand": false,
"ephemeralStorage": {
"sizeInGiB": 20
},
"group": "family:webserver",
"launchType": "FARGATE",
"lastStatus": "RUNNING",
"memory": "512",
"overrides": {
"containerOverrides": [
{
"name": "web"
},
{
"environment": [
{
"name": "CLUSTER_NAME",
"value": "example-cluster"
},
{
"name": "REGION",
"value": "us-east-1"
},
{
"name": "HOST_PROC",
"value": "/host_proc"
},
{
"name": "AGENT_RUNTIME_ENVIRONMENT",
"value": "ecsfargate"
},
{
"name": "STAGE",
"value": "prod"
}
],
"memory": 128,
"name": "aws-guardduty-agent-nm40lC"
},
{
"name": "aws-otel-collector"
}
]
},
"platformVersion": "1.4.0",
"pullStartedAt": "2025-05-06T11:02:24.162Z",
"pullStoppedAt": "2025-05-06T11:02:33.493Z",
"startedAt": "2025-05-06T11:02:34.325Z",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example-cluster/a1173316d40a45dea9",
"taskDefinitionArn": "arn:aws:ecs:us-east-1:123456789012:task-definition/webserver:5",
"updatedAt": "2025-05-06T11:02:34.325Z",
"version": 3
}
}
```
**Example**
The following is an example of a task state change event for EC2.
```
{
"version": "0",
"id": "a65cf262-f104-0dd5-ceda-4b09ba71a441",
"detail-type": "ECS Task State Change",
"source": "aws.ecs",
"account": "123456789012",
"time": "2025-05-12T13:12:06Z",
"region": "us-east-1",
"resources": [
"arn:aws:ecs:us-east-1:123456789012:task/example/c1ffa94f19a540ed8d9f7e1d2a5d"
],
"detail": {
"attachments": [
{
"id": "52333e3b-b812-41a8-b057-9ed184bbe5e1",
"type": "eni",
"status": "ATTACHED",
"details": [
{
"name": "subnetId",
"value": "subnet-0d0eab1bb38d5ca64"
},
{
"name": "networkInterfaceId",
"value": "eni-0ea90f746500773a4"
},
{
"name": "macAddress",
"value": "0e:d5:9b:ce:49:fb"
},
{
"name": "privateDnsName",
"value": "ip-10-0-1-37.ec2.internal"
},
{
"name": "privateIPv4Address",
"value": "10.0.1.37"
}
]
}
],
"attributes": [
{
"name": "ecs.cpu-architecture",
"value": "x86_64"
}
],
"availabilityZone": "us-east-1b",
"capacityProviderName": "Infra-ECS-Cluster-example-fa84e0cc-AsgCapacityProvider-OseQJU9pizmp",
"clusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/example",
"connectivity": "CONNECTED",
"connectivityAt": "2025-05-12T13:11:44.98Z",
"containerInstanceArn": "arn:aws:ecs:us-east-1:123456789012:container-instance/example/d1d84798400f49f3b21cb61610c1e",
"containers": [
{
"containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example/c1ffa94f19a540ed8d9f7e1d2a5d3626/197d0994-5367-4a6d-9f9a-f075e4a6",
"lastStatus": "RUNNING",
"name": "aws-otel-collector",
"image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.32.0",
"imageDigest": "sha256:7a1b3560655071bcacd66902c20ebe9a69470d5691fe3bd36baace7c2f3c4640",
"runtimeId": "8e926f0ccd8fe2b459926f49584ba6d33a3d9f61398dbabe944ee6a13a8ff3a1",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example/c1ffa94f19a540ed8d9f7e1d2a5d",
"networkInterfaces": [
{
"attachmentId": "52333e3b-b812-41a8-b057-9ed184bbe5e1",
"privateIpv4Address": "10.0.1.37"
}
],
"cpu": "0"
},
{
"containerArn": "arn:aws:ecs:us-east-1:123456789012:container/example/c1ffa94f19a540ed8d9f7e1d2a5d3626/cab39ef0-9c50-459d-844b-b9d51d73d",
"lastStatus": "RUNNING",
"name": "web",
"image": "nginx",
"imageDigest": "sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66",
"runtimeId": "9f1c73f0094f051541d9e5c2ab1e172d83c4eb5171bcc857c4504b02770ff3b8",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example/c1ffa94f19a540ed8d9f7e1d2a5d",
"networkInterfaces": [
{
"attachmentId": "52333e3b-b812-41a8-b057-9ed184bbe5e1",
"privateIpv4Address": "10.0.1.37"
}
],
"cpu": "99",
"memory": "100"
}
],
"cpu": "256",
"createdAt": "2025-05-12T13:11:44.98Z",
"desiredStatus": "RUNNING",
"enableExecuteCommand": false,
"group": "family:webserver",
"launchType": "EC2",
"lastStatus": "RUNNING",
"memory": "512",
"overrides": {
"containerOverrides": [
{
"name": "aws-otel-collector"
},
{
"name": "web"
}
]
},
"pullStartedAt": "2025-05-12T13:11:59.491Z",
"pullStoppedAt": "2025-05-12T13:12:05.896Z",
"startedAt": "2025-05-12T13:12:06.053Z",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/example/c1ffa94f19a540ed8d9f7e1d2a5d",
"taskDefinitionArn": "arn:aws:ecs:us-east-1:123456789012:task-definition/webserver",
"updatedAt": "2025-05-12T13:12:06.053Z",
"version": 4
}
}
```
# Amazon ECS service action events
Amazon ECS sends service action events with the detail type **ECS Service Action**. Unlike the container instance and task state change events, the service action events do not include a version number in the `details` response field. The following is an event pattern that is used to create an EventBridge rule for Amazon ECS service action events. For more information, see [Getting started with EventBridge ](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-get-started.html) in the *Amazon EventBridge User Guide*.
```
{
"source": [
"aws.ecs"
],
"detail-type": [
"ECS Service Action"
]
}
```
Amazon ECS sends events with `INFO`, `WARN`, and `ERROR` event types. The following are the service action events.
## Service action events with `INFO` event type
`SERVICE_STEADY_STATE`
The service is healthy and at the desired number of tasks, thus reaching a steady state. The service scheduler reports the status periodically, so you might receive this message multiple times.
`TASKSET_STEADY_STATE`
The task set is healthy and at the desired number of tasks, thus reaching a steady state.
`CAPACITY_PROVIDER_STEADY_STATE`
A capacity provider associated with a service reaches a steady state.
`SERVICE_DESIRED_COUNT_UPDATED`
When the service scheduler updates the computed desired count for a service or task set. This event is not sent when the desired count is manually updated by a user.
`TASKS_STOPPED`
The service has stopped the running task.
`SERVICE_DEPLOYMENT_IN_PROGRESS`
A service deployment is in progress. The service deployment can be either a rollback, or a new service revision.
`SERVICE_DEPLOYMENT_COMPLETED`
A service deployment is in the steady state and is complete. The service deployment can be either a rollback, or to deploy an updated service revision.
## Service action events with `WARN` event type
`SERVICE_TASK_START_IMPAIRED`
The service is unable to consistently start tasks successfully.
`SERVICE_DISCOVERY_INSTANCE_UNHEALTHY`
A service using service discovery contains an unhealthy task. The service scheduler detects that a task within a service registry is unhealthy.
`VPC_LATTICE_TARGET_UNHEALTHY`
The service using VPC Lattice has detected one of the targets for the VPC Lattice is unhealthy.
## Service action events with `ERROR` event type
`SERVICE_DAEMON_PLACEMENT_CONSTRAINT_VIOLATED`
A task in a service using the `DAEMON` service scheduler strategy no longer meets the placement constraint strategy for the service.
`ECS_OPERATION_THROTTLED`
The service scheduler has been throttled due to the Amazon ECS API throttle limits.
`SERVICE_DISCOVERY_OPERATION_THROTTLED`
The service scheduler has been throttled due to the AWS Cloud Map API throttle limits. This can occur on services configured to use service discovery.
`SERVICE_TASK_PLACEMENT_FAILURE`
The service scheduler is unable to place a task. The cause will be described in the `reason` field.
A common cause for this service event being generated is because of a lack of resources in the cluster to place the task. For example, not enough CPU or memory capacity on the available container instances or no container instances being available. Another common cause is when the Amazon ECS container agent is disconnected on the container instance, causing the scheduler to be unable to place the task.
`SERVICE_TASK_CONFIGURATION_FAILURE`
The service scheduler is unable to place a task due to a configuration error. The cause will be described in the `reason` field.
A common cause of this service event being generated is because tags were being applied to the service but the user or role had not opted in to the new Amazon Resource Name (ARN) format in the Region. For more information, see [Amazon Resource Names (ARNs) and IDs](ecs-account-settings.md#ecs-resource-ids). Another common cause is that Amazon ECS was unable to assume the task IAM role provided.
`SERVICE_HEALTH_UNKNOWN`
The service was unable to describe the health data for tasks.
`SERVICE_DEPLOYMENT_FAILED`
A service deployment did not reach the steady. This happens when a CloudWatch is triggered or the circuit breaker detects a service deployment failure.
**Example Service steady state event**
Service steady state events are delivered in the following format. For more information about EventBridge parameters, see [Events in EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html) in the *Amazon EventBridge User Guide*.
```
{
"version": "0",
"id": "af3c496d-f4a8-65d1-70f4-a69d52e9b584",
"detail-type": "ECS Service Action",
"source": "aws.ecs",
"account": "111122223333",
"time": "2019-11-19T19:27:22Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "INFO",
"eventName": "SERVICE_STEADY_STATE",
"clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default",
"createdAt": "2019-11-19T19:27:22.695Z"
}
}
```
**Example Capacity provider steady state event**
Capacity provider steady state events are delivered in the following format.
```
{
"version": "0",
"id": "b9baa007-2f33-0eb1-5760-0d02a572d81f",
"detail-type": "ECS Service Action",
"source": "aws.ecs",
"account": "111122223333",
"time": "2019-11-19T19:37:00Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "INFO",
"eventName": "CAPACITY_PROVIDER_STEADY_STATE",
"clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default",
"capacityProviderArns": [
"arn:aws:ecs:us-west-2:111122223333:capacity-provider/ASG-tutorial-capacity-provider"
],
"createdAt": "2019-11-19T19:37:00.807Z"
}
}
```
**Example Service task start impaired event**
Service task start impaired events are delivered in the following format.
```
{
"version": "0",
"id": "57c9506e-9d21-294c-d2fe-e8738da7e67d",
"detail-type": "ECS Service Action",
"source": "aws.ecs",
"account": "111122223333",
"time": "2019-11-19T19:55:38Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "WARN",
"eventName": "SERVICE_TASK_START_IMPAIRED",
"clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default",
"createdAt": "2019-11-19T19:55:38.725Z"
}
}
```
**Example Service task placement failure event**
Service task placement failure events are delivered in the following format. For more information, see [Events in EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html) in the *Amazon EventBridge User Guide*.
In the following example, the task was attempting to use the `FARGATE_SPOT` capacity provider but the service scheduler was unable to acquire any Fargate Spot capacity.
```
{
"version": "0",
"id": "ddca6449-b258-46c0-8653-e0e3a6d0468b",
"detail-type": "ECS Service Action",
"source": "aws.ecs",
"account": "111122223333",
"time": "2019-11-19T19:55:38Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "ERROR",
"eventName": "SERVICE_TASK_PLACEMENT_FAILURE",
"clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default",
"capacityProviderArns": [
"arn:aws:ecs:us-west-2:111122223333:capacity-provider/FARGATE_SPOT"
],
"reason": "RESOURCE:FARGATE",
"createdAt": "2019-11-06T19:09:33.087Z"
}
}
```
In the following example for EC2, the task was attempted to launch on the Container Instance `2dd1b186f39845a584488d2ef155c131` but the service scheduler was unable to place the task because of insufficient CPU.
```
{
"version": "0",
"id": "ddca6449-b258-46c0-8653-e0e3a6d0468b",
"detail-type": "ECS Service Action",
"source": "aws.ecs",
"account": "111122223333",
"time": "2019-11-19T19:55:38Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "ERROR",
"eventName": "SERVICE_TASK_PLACEMENT_FAILURE",
"clusterArn": "arn:aws:ecs:us-west-2:111122223333:cluster/default",
"containerInstanceArns": [
"arn:aws:ecs:us-west-2:111122223333:container-instance/default/2dd1b186f39845a584488d2ef155c131"
],
"reason": "RESOURCE:CPU",
"createdAt": "2019-11-06T19:09:33.087Z"
}
}
```
# Amazon ECS service deployment state change events
Amazon ECS sends service deployment change state events with the detail type **ECS Deployment State Change**. The following is an event pattern that is used to create an EventBridge rule for Amazon ECS service deployment state change events. For more information about creating an EventBridge rule, see [Getting started with Amazon EventBridge ](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-get-started.html) in the *Amazon EventBridge User Guide*.
```
{
"source": [
"aws.ecs"
],
"detail-type": [
"ECS Deployment State Change"
]
}
```
Amazon ECS sends events with `INFO` and `ERROR` event types. For more information, see [Amazon ECS service action events](ecs_service_events.md)
The following are the service deployment state change events.
`SERVICE_DEPLOYMENT_IN_PROGRESS`
The service deployment is in progress. This event is sent for both initial deployments and rollback deployments.
`SERVICE_DEPLOYMENT_COMPLETED`
The service deployment has completed. This event is sent once a service reaches a steady state after a deployment.
`SERVICE_DEPLOYMENT_FAILED`
The service deployment has failed. This event is sent for services with deployment circuit breaker logic turned on.
**Example service deployment in progress event**
Service deployment in progress events are delivered when both an initial and a rollback deployment is started. The difference between the two is in the `reason` field. For more information about EventBridge parameters, see [AWS service event metadata;](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events-structure.html) in the *Amazon EventBridge User Guide*.
The following shows an example output for an initial deployment starting.
```
{
"version": "0",
"id": "ddca6449-b258-46c0-8653-e0e3a6EXAMPLE",
"detail-type": "ECS Deployment State Change",
"source": "aws.ecs",
"account": "111122223333",
"time": "2020-05-23T12:31:14Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "INFO",
"eventName": "SERVICE_DEPLOYMENT_IN_PROGRESS",
"deploymentId": "ecs-svc/123",
"updatedAt": "2020-05-23T11:11:11Z",
"reason": "ECS deployment deploymentId in progress."
}
}
```
The following shows an example output for a rollback deployment starting. The `reason` field provides the ID of the deployment the service is rolling back to.
```
{
"version": "0",
"id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE",
"detail-type": "ECS Deployment State Change",
"source": "aws.ecs",
"account": "111122223333",
"time": "2020-05-23T12:31:14Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "INFO",
"eventName": "SERVICE_DEPLOYMENT_IN_PROGRESS",
"deploymentId": "ecs-svc/123",
"updatedAt": "2020-05-23T11:11:11Z",
"reason": "ECS deployment circuit breaker: rolling back to deploymentId deploymentID."
}
}
```
**Example service deployment completed event**
Service deployment completed state events are delivered in the following format. For more information, see [Deploy Amazon ECS services by replacing tasks](deployment-type-ecs.md).
```
{
"version": "0",
"id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE",
"detail-type": "ECS Deployment State Change",
"source": "aws.ecs",
"account": "111122223333",
"time": "2020-05-23T12:31:14Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "INFO",
"eventName": "SERVICE_DEPLOYMENT_COMPLETED",
"deploymentId": "ecs-svc/123",
"updatedAt": "2020-05-23T11:11:11Z",
"reason": "ECS deployment deploymentID completed."
}
}
```
**Example service deployment failed event**
Service deployment failed state events are delivered in the following format. A service deployment failed state event will only be sent for services that have deployment circuit breaker logic turned on. For more information, see [Deploy Amazon ECS services by replacing tasks](deployment-type-ecs.md).
```
{
"version": "0",
"id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE",
"detail-type": "ECS Deployment State Change",
"source": "aws.ecs",
"account": "111122223333",
"time": "2020-05-23T12:31:14Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
],
"detail": {
"eventType": "ERROR",
"eventName": "SERVICE_DEPLOYMENT_FAILED",
"deploymentId": "ecs-svc/123",
"updatedAt": "2020-05-23T11:11:11Z",
"reason": "ECS deployment circuit breaker: task failed to start."
}
}
```
# Handling Amazon ECS events
Amazon ECS sends events on an *at least once* basis. This means you might receive multiple copies of a given event. Additionally, events may not be delivered to your event listeners in the order in which the events occurred.
To order of events properly, the `detail` section of each event contains a `version` property. Each time a resource changes state, this `version` is incremented. Duplicate events have the same `version` in the `detail` object. If you are replicating your Amazon ECS container instance and task state with EventBridge, you can compare the version of a resource reported by the Amazon ECS APIs with the `version` reported in EventBridge for the resource to verify that the version in your event stream is current. Events with a higher version property number should be treated as occurring later than events with lower version numbers.
## Example: Handling events in an AWS Lambda function
The following example shows a Lambda function written in Python 3.9 that captures both task and container instance state change events and saves them to one of two Amazon DynamoDB tables:
+ *ECSCtrInstanceState* – Stores the latest state for a container instance. The table ID is the `containerInstanceArn` value of the container instance.
+ *ECSTaskState* – Stores the latest state for a task. The table ID is the `taskArn` value of the task.
```
import json
import boto3
def lambda_handler(event, context):
id_name = ""
new_record = {}
# For debugging so you can see raw event format.
print('Here is the event:')
print((json.dumps(event)))
if event["source"] != "aws.ecs":
raise ValueError("Function only supports input from events with a source type of: aws.ecs")
# Switch on task/container events.
table_name = ""
if event["detail-type"] == "ECS Task State Change":
table_name = "ECSTaskState"
id_name = "taskArn"
event_id = event["detail"]["taskArn"]
elif event["detail-type"] == "ECS Container Instance State Change":
table_name = "ECSCtrInstanceState"
id_name = "containerInstanceArn"
event_id = event["detail"]["containerInstanceArn"]
else:
raise ValueError("detail-type for event is not a supported type. Exiting without saving event.")
new_record["cw_version"] = event["version"]
new_record.update(event["detail"])
# "status" is a reserved word in DDB, but it appears in containerPort
# state change messages.
if "status" in event:
new_record["current_status"] = event["status"]
new_record.pop("status")
# Look first to see if you have received a newer version of an event ID.
# If the version is OLDER than what you have on file, do not process it.
# Otherwise, update the associated record with this latest information.
print("Looking for recent event with same ID...")
dynamodb = boto3.resource("dynamodb", region_name="us-east-1")
table = dynamodb.Table(table_name)
saved_event = table.get_item(
Key={
id_name : event_id
}
)
if "Item" in saved_event:
# Compare events and reconcile.
print(("EXISTING EVENT DETECTED: Id " + event_id + " - reconciling"))
if saved_event["Item"]["version"] < event["detail"]["version"]:
print("Received event is a more recent version than the stored event - updating")
table.put_item(
Item=new_record
)
else:
print("Received event is an older version than the stored event - ignoring")
else:
print(("Saving new event - ID " + event_id))
table.put_item(
Item=new_record
)
```
The following Fargate example shows a Lambda function written in Python 3.9 that captures task state change events and saves them to the following Amazon DynamoDB table:
```
import json
import boto3
def lambda_handler(event, context):
id_name = ""
new_record = {}
# For debugging so you can see raw event format.
print('Here is the event:')
print((json.dumps(event)))
if event["source"] != "aws.ecs":
raise ValueError("Function only supports input from events with a source type of: aws.ecs")
# Switch on task/container events.
table_name = ""
if event["detail-type"] == "ECS Task State Change":
table_name = "ECSTaskState"
id_name = "taskArn"
event_id = event["detail"]["taskArn"]
else:
raise ValueError("detail-type for event is not a supported type. Exiting without saving event.")
new_record["cw_version"] = event["version"]
new_record.update(event["detail"])
# "status" is a reserved word in DDB, but it appears in containerPort
# state change messages.
if "status" in event:
new_record["current_status"] = event["status"]
new_record.pop("status")
# Look first to see if you have received a newer version of an event ID.
# If the version is OLDER than what you have on file, do not process it.
# Otherwise, update the associated record with this latest information.
print("Looking for recent event with same ID...")
dynamodb = boto3.resource("dynamodb", region_name="us-east-1")
table = dynamodb.Table(table_name)
saved_event = table.get_item(
Key={
id_name : event_id
}
)
if "Item" in saved_event:
# Compare events and reconcile.
print(("EXISTING EVENT DETECTED: Id " + event_id + " - reconciling"))
if saved_event["Item"]["version"] < event["detail"]["version"]:
print("Received event is a more recent version than the stored event - updating")
table.put_item(
Item=new_record
)
else:
print("Received event is an older version than the stored event - ignoring")
else:
print(("Saving new event - ID " + event_id))
table.put_item(
Item=new_record
)
```