# Send Amazon ECS logs to an AWS service or AWS Partner
You can use FireLens for Amazon ECS to use task definition parameters to route logs to an AWS service or AWS Partner Network (APN) destination for log storage and analytics. The AWS Partner Network is a global community of partners that leverages programs, expertise, and resources to build, market, and sell customer offerings. For more information see [AWS Partner](https://aws.amazon.com/partners/work-with-partners/). FireLens works with [Fluentd](https://www.fluentd.org/) and [Fluent Bit](https://fluentbit.io/). We provide the AWS for Fluent Bit image or you can use your own Fluentd or Fluent Bit image.
By default, Amazon ECS configures the container dependency so that the Firelens container starts before any container that uses it. The Firelens container also stops after all containers that use it stop.
To use this feature, you must create an IAM role for your tasks that provides the permissions necessary to use any AWS services that the tasks require. For example, if a container is routing logs to Firehose, the task requires permission to call the `firehose:PutRecordBatch` API. For more information, see [Adding and Removing IAM Identity Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) in the *IAM User Guide*.
Your task might also require the Amazon ECS task execution role under the following conditions. For more information, see [Amazon ECS task execution IAM role](task_execution_IAM_role.md).
+ If your task is hosted on Fargate and you are pulling container images from Amazon ECR or referencing sensitive data from AWS Secrets Manager in your log configuration, then you must include the task execution IAM role.
+ When you use a custom configuration file that's hosted in Amazon S3, your task execution IAM role must include the `s3:GetObject` permission.
Consider the following when using FireLens for Amazon ECS:
+ We recommend that you add `my_service_` to the log container name so that you can easily distinguish container names in the console.
+ Amazon ECS adds a start container order dependency between the application containers and the FireLens container by default. When you specify a container order between the application containers and the FireLens container, then the default start container order is overridden.
+ FireLens for Amazon ECS is supported for tasks that are hosted on both AWS Fargate on Linux and Amazon EC2 on Linux. Windows containers don't support FireLens.
For information about how to configure centralized logging for Windows containers, see [Centralized logging for Windows containers on Amazon ECS using Fluent Bit](https://aws.amazon.com/blogs/containers/centralized-logging-for-windows-containers-on-amazon-ecs-using-fluent-bit/).
+ You can use CloudFormation templates to configure FireLens for Amazon ECS. For more information, see [AWS::ECS::TaskDefinition FirelensConfiguration](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-firelensconfiguration.html) in the *AWS CloudFormation User Guide*
+ FireLens listens on port `24224`, so to ensure that the FireLens log router isn't reachable outside of the task, you must not allow inbound traffic on port `24224` in the security group your task uses. For tasks that use the `awsvpc` network mode, this is the security group associated with the task. For tasks using the `host` network mode, this is the security group that's associated with the Amazon EC2 instance hosting the task. For tasks that use the `bridge` network mode, don't create any port mappings that use port `24224`.
+ For tasks that use the `bridge` network mode, the container with the FireLens configuration must start before any application containers that rely on it start. To control the start order of your containers, use dependency conditions in your task definition. For more information, see [Container dependency](task_definition_parameters.md#container_definition_dependson).
**Note**
If you use dependency condition parameters in container definitions with a FireLens configuration, ensure that each container has a `START` or `HEALTHY` condition requirement.
+ By default, FireLens adds the cluster and task definition name and the Amazon Resource Name (ARN) of the cluster as metadata keys to your stdout/stderr container logs. The following is an example of the metadata format.
```
"ecs_cluster": "cluster-name",
"ecs_task_arn": "arn:aws:ecs:region:111122223333:task/cluster-name/f2ad7dba413f45ddb4EXAMPLE",
"ecs_task_definition": "task-def-name:revision",
```
If you do not want the metadata in your logs, set `enable-ecs-log-metadata` to `false` in the `firelensConfiguration` section of the task definition.
```
"firelensConfiguration":{
"type":"fluentbit",
"options":{
"enable-ecs-log-metadata":"false",
"config-file-type":"file",
"config-file-value":"/extra.conf"
}
```
You can configure the FireLens container to run as a non-root user. Consider the following:
+ To configure the FireLens container to run as a non-root user, you must specify the user in one of the following formats:
+ `uid`
+ `uid:gid`
+ `uid:group`
For more information about specifying a user in a container definition, see [ContainerDefinition](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html) in the *Amazon Elastic Container Service API Reference*.
The FireLens container receives application logs over a UNIX socket. The Amazon ECS agent uses the `uid` to assign ownership of the socket directory to the FireLens container.
+ Configuring the FireLens container to run as a non-root user is supported on Amazon ECS Agent version `1.96.0` and later, and Amazon ECS-optimized AMI version `v20250716` and later.
+ When you specify a user for the FireLens container, the `uid` must be unique and not used for other processes belonging to other containers in the task or the container instance.
For information about how to use multiple configuration files with Amazon ECS, including files that you host or files in Amazon S3, see [Init process for Fluent Bit on ECS, multi-config support](https://github.com/aws/aws-for-fluent-bit/tree/mainline/use_cases/init-process-for-fluent-bit).
For information about example configurations, see [Example Amazon ECS task definition: Route logs to FireLens](firelens-taskdef.md).
For more information about configuring logs for high throughput, see [Configuring Amazon ECS logs for high throughput](firelens-docker-buffer-limit.md).
# Configuring Amazon ECS logs for high throughput
For high log throughput scenarios, we recommend using the `awsfirelens` log driver with FireLens and Fluent Bit. Fluent Bit is a lightweight log processor that's efficient with resources and can handle millions of log records. However, achieving optimal performance at scale requires tuning its configuration.
This section covers advanced Fluent Bit optimization techniques for handling high log throughput while maintaining system stability and ensuring no data loss.
For information about how to use custom configuration files with FireLens, see [Use a custom configuration file](firelens-taskdef.md#firelens-taskdef-customconfig). For additional examples, see [Amazon ECS FireLens examples](https://github.com/aws-samples/amazon-ecs-firelens-examples) on GitHub.
**Note**
Some configuration options in this section, such as `workers` and `threaded`, require AWS for Fluent Bit version 3 or later. For information about available versions, see [AWS for Fluent Bit releases](https://github.com/aws/aws-for-fluent-bit/releases).
## Understanding chunks
Fluent Bit processes data in units called *chunks*. When an INPUT plugin receives data, the engine creates a chunk that gets stored in memory or on the filesystem before being sent to OUTPUT destinations.
Buffering behavior depends on the `storage.type` setting in your INPUT sections. By default, Fluent Bit uses memory buffering. For high-throughput or production scenarios, filesystem buffering provides better resilience.
For more information, see [Chunks](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) in the Fluent Bit documentation and [What is a Chunk?](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/oomkill-prevention#what-is-a-chunk) in the AWS for Fluent Bit examples repository.
## Memory buffering (default)
By default, Fluent Bit uses memory buffering (`storage.type memory`). You can limit memory usage per INPUT plugin using the `Mem_Buf_Limit` parameter.
The following example shows a memory-buffered input configuration:
```
[INPUT]
Name tcp
Tag ApplicationLogs
Port 5170
storage.type memory
Mem_Buf_Limit 5MB
```
**Important**
When `Mem_Buf_Limit` is exceeded for a plugin, Fluent Bit pauses the input and new records are lost. This can cause backpressure and slow down your application. The following warning appears in the Fluent Bit logs:
```
[input] tcp.1 paused (mem buf overlimit)
```
Memory buffering is suitable for simple use cases with low to moderate log throughput. For high-throughput or production scenarios where data loss is a concern, use filesystem buffering instead.
For more information, see [Buffering and Memory](https://docs.fluentbit.io/manual/administration/buffering-and-storage#buffering-and-memory) in the Fluent Bit documentation and [Memory Buffering Only](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/oomkill-prevention#case-1-memory-buffering-only-default-or-storagetype-memory) in the AWS for Fluent Bit examples repository.
## Filesystem buffering
For high-throughput scenarios, we recommend using filesystem buffering. For more information about how Fluent Bit manages buffering and storage, see [Buffering and Storage](https://docs.fluentbit.io/manual/administration/buffering-and-storage) in the Fluent Bit documentation.
Filesystem buffering provides the following advantages:
+ **Larger buffer capacity** – Disk space is typically more abundant than memory.
+ **Persistence** – Buffered data survives Fluent Bit restarts.
+ **Graceful degradation** – During output failures, data accumulates on disk rather than causing memory exhaustion.
To enable filesystem buffering, provide a custom Fluent Bit configuration file. The following example shows the recommended configuration:
```
[SERVICE]
# Flush logs every 1 second
Flush 1
# Wait 120 seconds during shutdown to flush remaining logs
Grace 120
# Directory for filesystem buffering
storage.path /var/log/flb-storage/
# Limit chunks stored 'up' in memory (reduce for memory-constrained environments)
storage.max_chunks_up 32
# Flush backlog chunks to destinations during shutdown (prevents log loss)
storage.backlog.flush_on_shutdown On
[INPUT]
Name forward
unix_path /var/run/fluent.sock
# Run input in separate thread to prevent blocking
threaded true
# Enable filesystem buffering for persistence
storage.type filesystem
[OUTPUT]
Name cloudwatch_logs
Match *
region us-west-2
log_group_name /aws/ecs/my-app
log_stream_name $(ecs_task_id)
# Use multiple workers for parallel processing
workers 2
# Retry failed flushes up to 15 times
retry_limit 15
# Maximum disk space for buffered data for this output
storage.total_limit_size 10G
```
Key configuration parameters:
`storage.path`
The directory where Fluent Bit stores buffered chunks on disk.
`storage.backlog.flush_on_shutdown`
When enabled, Fluent Bit attempts to flush all backlog filesystem chunks to their destinations during shutdown. This helps ensure data delivery before Fluent Bit stops, but may increase shutdown time.
`storage.max_chunks_up`
The number of chunks that remain in memory. The default is 128 chunks, which can consume 500 MB\$1 of memory because each chunk can use up to 4–5 MB. In memory-constrained environments, lower this value. For example, if you have 50 MB available for buffering, set this to 8–10 chunks.
`storage.type filesystem`
Enables filesystem storage for the input plugin. Despite the name, Fluent Bit uses `mmap` to map chunks to both memory and disk, providing persistence without sacrificing performance.
`storage.total_limit_size`
The maximum disk space for buffered data for a specific OUTPUT plugin. When this limit is reached, the oldest records for that output are dropped. For more information about sizing, see [Understanding `storage.total_limit_size`](#firelens-storage-sizing).
`threaded true`
Runs the input in its own thread, separate from Fluent Bit's main event loop. This prevents slow inputs from blocking the entire pipeline.
For more information, see [Filesystem Buffering](https://docs.fluentbit.io/manual/administration/buffering-and-storage#filesystem-buffering) in the Fluent Bit documentation and [Filesystem and Memory Buffering](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/oomkill-prevention#case-2-filesystem-and-memory-buffering-storagetype-filesystem) in the AWS for Fluent Bit examples repository.
## Understanding `storage.total_limit_size`
The `storage.total_limit_size` parameter on each OUTPUT plugin controls the maximum disk space for buffered data for that output. When this limit is reached, the oldest records for that output are dropped to make room for new data. When disk space is completely exhausted, Fluent Bit fails to queue records and they are lost.
Use the following formula to calculate the appropriate `storage.total_limit_size` based on your log rate and desired recovery window:
```
If log rate is in KB/s, convert to MB/s first:
log_rate (MB/s) = log_rate (KB/s) / 1000
storage.total_limit_size (GB) = log_rate (MB/s) × duration (hours) × 3600 (seconds/hour) / 1000 (MB to GB)
```
The following table shows example calculations for common log rates and recovery windows:
| Log Rate | 1 hour | 6 hours | 12 hours | 24 hours |
| --- | --- | --- | --- | --- |
| 0.25 MB/s | 0.9 GB | 5.4 GB | 10.8 GB | 21.6 GB |
| 0.5 MB/s | 1.8 GB | 10.8 GB | 21.6 GB | 43.2 GB |
| 1 MB/s | 3.6 GB | 21.6 GB | 43.2 GB | 86.4 GB |
| 5 MB/s | 18 GB | 108 GB | 216 GB | 432 GB |
| 10 MB/s | 36 GB | 216 GB | 432 GB | 864 GB |
To observe peak throughput and choose appropriate buffer sizes, use the [measure-throughput FireLens sample](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/measure-throughput).
Use the formula, example calculations, and benchmarking to choose a suitable `storage.total_limit_size` that provides runway for best-effort recovery during an outage.
## Amazon ECS task storage requirements
Sum all `storage.total_limit_size` values across OUTPUT sections and add buffer for overhead. This total determines the storage space needed in your Amazon ECS task definition. For example, 3 outputs × 10 GB each = 30 GB \$1 buffer (5–10 GB) = 35–40 GB total required. If the total exceeds available storage, Fluent Bit may fail to queue records and they will be lost.
The following storage options are available:
Bind mounts (ephemeral storage)
+ For AWS Fargate, the default is 20 GB of ephemeral storage (max 200 GB). Configure using `ephemeralStorage` in the task definition. For more information, see [EphemeralStorage](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-ephemeralstorage.html) in the *AWS CloudFormation User Guide*.
+ For EC2, the default is 30 GB when using the Amazon ECS-optimized AMI (shared between the OS and Docker). Increase by changing the root volume size.
Amazon EBS volumes
+ Provides highly available, durable, high-performance block storage.
+ Requires volume configuration and `mountPoint` in the task definition pointing to `storage.path` (default: `/var/log/flb-storage/`).
+ For more information, see [Defer volume configuration to launch time in an Amazon ECS task definition](specify-ebs-config.md).
Amazon EFS volumes
+ Provides simple, scalable file storage.
+ Requires volume configuration and `mountPoint` in the task definition pointing to `storage.path` (default: `/var/log/flb-storage/`).
+ For more information, see [Specify an Amazon EFS file system in an Amazon ECS task definition](specify-efs-config.md).
For more information about data volumes, see [Storage options for Amazon ECS tasks](using_data_volumes.md).
## Optimize output configuration
Network issues, service outages, and destination throttling can prevent logs from being delivered. Proper output configuration ensures resilience without data loss.
When an output flush fails, Fluent Bit can retry the operation. The following parameters control retry behavior:
`retry_limit`
The maximum number of retries after the initial attempt before dropping records. The default is 1. For example, `retry_limit 3` means 4 total attempts (1 initial \$1 3 retries). For production environments, we recommend 15 or higher, which covers several minutes of outage with exponential backoff.
Set to `no_limits` or `False` for infinite retries:
+ With memory buffering, infinite retries cause the input plugin to pause when memory limits are reached.
+ With filesystem buffering, the oldest records are dropped when `storage.total_limit_size` is reached.
After exhausting all retry attempts (1 initial \$1 `retry_limit` retries), records are dropped. AWS plugins with `auto_retry_requests true` (default) provide an additional retry layer before Fluent Bit's retry mechanism. For more information, see [Configure retries](https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configure-retries) in the Fluent Bit documentation.
For example, `retry_limit 3` with default settings (`scheduler.base 5`, `scheduler.cap 2000`, `net.connect_timeout 10s`) provides approximately 70 seconds of scheduler wait time (10s \$1 20s \$1 40s), 40 seconds of network connect timeouts (4 attempts × 10s), plus AWS plugin retries — totaling approximately 2–10 minutes depending on network conditions and OS TCP timeouts.
`scheduler.base`
The base seconds between retries (default: 5). We recommend 10 seconds.
`scheduler.cap`
The maximum seconds between retries (default: 2000). We recommend 60 seconds.
Wait time between retries uses exponential backoff with jitter:
```
wait_time = random(base, min(base × 2^retry_number, cap))
```
For example, with `scheduler.base 10` and `scheduler.cap 60`:
+ First retry: random wait between 10–20 seconds
+ Second retry: random wait between 10–40 seconds
+ Third retry and later: random wait between 10–60 seconds (capped)
For more information, see [Configure wait time for retry](https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configure-wait-time-for-retry) and [Networking](https://docs.fluentbit.io/manual/administration/networking) in the Fluent Bit documentation.
`workers`
The number of threads for parallel output processing. Multiple workers allow concurrent flushes, improving throughput when processing many chunks.
`auto_retry_requests`
An AWS plugin-specific setting that provides an additional retry layer before Fluent Bit's built-in retry mechanism. The default is `true`. When enabled, the AWS output plugin retries failed requests internally before the request is considered a failed flush and subject to the `retry_limit` configuration.
The `Grace` parameter in the `[SERVICE]` section sets the time Fluent Bit waits during shutdown to flush buffered data. The `Grace` period must be coordinated with the container's `stopTimeout`. Ensure that `stopTimeout` exceeds the `Grace` period to allow Fluent Bit to complete flushing before receiving `SIGKILL`. For example, if `Grace` is 120 seconds, set `stopTimeout` to 150 seconds.
The following example shows a complete Fluent Bit configuration with all recommended settings for high-throughput scenarios:
```
[SERVICE]
# Flush logs every 1 second
Flush 1
# Wait 120 seconds during shutdown to flush remaining logs
Grace 120
# Directory for filesystem buffering
storage.path /var/log/flb-storage/
# Limit chunks stored 'up' in memory (reduce for memory-constrained environments)
storage.max_chunks_up 32
# Flush backlog chunks to destinations during shutdown (prevents log loss)
storage.backlog.flush_on_shutdown On
# Minimum seconds between retries
scheduler.base 10
# Maximum seconds between retries (exponential backoff cap)
scheduler.cap 60
[INPUT]
Name forward
unix_path /var/run/fluent.sock
# Run input in separate thread to prevent blocking
threaded true
# Enable filesystem buffering for persistence
storage.type filesystem
[OUTPUT]
Name cloudwatch_logs
Match *
region us-west-2
log_group_name /aws/ecs/my-app
log_stream_name $(ecs_task_id)
# Use multiple workers for parallel processing
workers 2
# Retry failed flushes up to 15 times
retry_limit 15
# Maximum disk space for buffered data for this output
storage.total_limit_size 10G
```
## Understanding data loss scenarios
Records can be lost during extended outages or issues with output destinations. The configuration recommendations in this guide are best-effort approaches to minimize data loss, but cannot guarantee zero loss during prolonged failures. Understanding these scenarios helps you configure Fluent Bit to maximize resilience.
Records can be lost in two ways: oldest records are dropped when storage fills up, or newest records are rejected when the system cannot accept more data.
### Oldest records dropped
The oldest buffered records are dropped when retry attempts are exhausted or when `storage.total_limit_size` fills up and needs to make room for new data.
Retry limit exceeded
Occurs after AWS plugin retries (if `auto_retry_requests true`) plus 1 initial Fluent Bit attempt plus `retry_limit` retries. To mitigate, set `retry_limit no_limits` per OUTPUT plugin for infinite retries:
```
[OUTPUT]
Name cloudwatch_logs
Match ApplicationLogs
retry_limit no_limits
auto_retry_requests true
```
Infinite retries prevent dropping records due to retry exhaustion, but may cause `storage.total_limit_size` to fill up.
Storage limit reached (filesystem buffering)
Occurs when the output destination is unavailable longer than your configured `storage.total_limit_size` can buffer. For example, a 10 GB buffer at 1 MB/s log rate provides approximately 2.7 hours of buffering. To mitigate, increase `storage.total_limit_size` per OUTPUT plugin and provision adequate Amazon ECS task storage:
```
[OUTPUT]
Name cloudwatch_logs
Match ApplicationLogs
storage.total_limit_size 10G
```
### Newest records rejected
The newest records are dropped when disk space is exhausted or when input is paused due to `Mem_Buf_Limit`.
Disk space exhausted (filesystem buffering)
Occurs when disk space is completely exhausted. Fluent Bit fails to queue new records and they are lost. To mitigate, sum all `storage.total_limit_size` values and provision adequate Amazon ECS task storage. For more information, see [Amazon ECS task storage requirements](#firelens-storage-task-requirements).
Memory limit reached (memory buffering)
Occurs when the output destination is unavailable and the memory buffer fills. Paused input plugins stop accepting new records. To mitigate, use `storage.type filesystem` for better resilience, or increase `Mem_Buf_Limit`.
### Best practices to minimize data loss
Consider the following best practices to minimize data loss:
+ **Use filesystem buffering** – Set `storage.type filesystem` for better resilience during outages.
+ **Size storage appropriately** – Calculate `storage.total_limit_size` based on log rate and desired recovery window.
+ **Provision adequate disk** – Ensure the Amazon ECS task has sufficient ephemeral storage, Amazon EBS, or Amazon EFS.
+ **Configure retry behavior** – Balance between `retry_limit` (drops records after exhausting retries) and `no_limits` (retries indefinitely but may fill storage).
## Use multi-destination logging for reliability
Sending logs to multiple destinations eliminates single points of failure. For example, if CloudWatch Logs experiences an outage, logs still reach Amazon S3.
Multi-destination logging provides the following benefits. The Amazon S3 output plugin also supports compression options such as gzip and Parquet format, which can reduce storage costs. For more information, see [S3 compression](https://docs.fluentbit.io/manual/pipeline/outputs/s3#compression) in the Fluent Bit documentation.
Multi-destination logging can provide the following benefits:
+ **Redundancy** – If one destination fails, logs still reach the other.
+ **Recovery** – Reconstruct gaps in one system from the other.
+ **Durability** – Archive logs in Amazon S3 for long-term retention.
+ **Cost optimization** – Keep recent logs in a fast query service like CloudWatch Logs with shorter retention, while archiving all logs to lower-cost Amazon S3 storage for long-term retention.
The following Fluent Bit configuration sends logs to both CloudWatch Logs and Amazon S3:
```
[OUTPUT]
Name cloudwatch_logs
Match *
region us-west-2
log_group_name /aws/ecs/my-app
log_stream_name $(ecs_task_id)
workers 2
retry_limit 15
[OUTPUT]
Name s3
Match *
bucket my-logs-bucket
region us-west-2
total_file_size 100M
s3_key_format /fluent-bit-logs/$(ecs_task_id)/%Y%m%d/%H/%M/$UUID
upload_timeout 10m
# Maximum disk space for buffered data for this output
storage.total_limit_size 5G
```
Both outputs use the same `Match *` pattern, so all records are sent to both destinations independently. During an outage of one destination, logs continue flowing to the other while failed flushes accumulate in the filesystem buffer for later retry.
## Use file-based logging with the tail input plugin
For high-throughput scenarios where log loss is a critical concern, you can use an alternative approach: have your application write logs to files on disk, and configure Fluent Bit to read them using the `tail` input plugin. This approach bypasses the Docker logging driver layer entirely.
File-based logging with the tail plugin provides the following benefits:
+ **Offset tracking** – The tail plugin can store file offsets in a database file (using the `DB` option), providing durability across Fluent Bit restarts. This helps prevent log loss during container restarts.
+ **Input-level buffering** – You can configure memory buffer limits directly on the input plugin using `Mem_Buf_Limit`, providing more granular control over memory usage.
+ **Avoids Docker overhead** – Logs go directly from file to Fluent Bit without passing through Docker's log buffers.
To use this approach, your application must write logs to files instead of `stdout`. Both the application container and the Fluent Bit container mount a shared volume where the log files are stored.
The following example shows a tail input configuration with best practices:
```
[INPUT]
Name tail
# File path or glob pattern to tail
Path /var/log/app.log
# Database file for storing file offsets (enables resuming after restart)
DB /var/log/flb_tail.db
# when true, controls that only fluent-bit will access the database (improves performance)
DB.locking true
# Skip long lines instead of skipping the entire file
Skip_Long_Lines On
# How often (in seconds) to check for new files matching the glob pattern
Refresh_Interval 10
# Extra seconds to monitor a file after rotation to account for pending flush
Rotate_Wait 30
# Maximum size of the buffer for a single line
Buffer_Max_Size 10MB
# Initial allocation size for reading file data
Buffer_Chunk_Size 1MB
# Maximum memory buffer size (tail pauses when full)
Mem_Buf_Limit 75MB
```
When using the tail input plugin, consider the following:
+ Implement log rotation for your application logs to prevent disk exhaustion. Monitor the underlying volume metrics to gauge performance.
+ Consider settings like `Ignore_Older`, `Read_from_Head`, and multiline parsers based on your log format.
For more information, see [Tail](https://docs.fluentbit.io/manual/pipeline/inputs/tail) in the Fluent Bit documentation. For best practices, see [Tail config with best practices](https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#tail-config-with-best-practices) in the AWS for Fluent Bit troubleshooting guide.
## Log directly to FireLens
When the `awsfirelens` log driver is specified in a task definition, the Amazon ECS container agent injects the following environment variables into the container:
`FLUENT_HOST`
The IP address that's assigned to the FireLens container.
If you're using EC2 with the `bridge` network mode, the `FLUENT_HOST` environment variable in your application container can become inaccurate after a restart of the FireLens log router container (the container with the `firelensConfiguration` object in its container definition). This is because `FLUENT_HOST` is a dynamic IP address and can change after a restart. Logging directly from the application container to the `FLUENT_HOST` IP address can start failing after the address changes. For more information about restarting individual containers, see [Restart individual containers in Amazon ECS tasks with container restart policies](container-restart-policy.md).
`FLUENT_PORT`
The port that the Fluent Forward protocol is listening on.
You can use these environment variables to log directly to the Fluent Bit log router from your application code using the Fluent Forward protocol, instead of writing to `stdout`. This approach bypasses the Docker logging driver layer, which provides the following benefits:
+ **Lower latency** – Logs go directly to Fluent Bit without passing through Docker's logging infrastructure.
+ **Structured logging** – Send structured log data natively without JSON encoding overhead.
+ **Better control** – Your application can implement its own buffering and error handling logic.
The following Fluent logger libraries support the Fluent Forward protocol and can be used to send logs directly to Fluent Bit:
+ **Go** – [fluent-logger-golang](https://github.com/fluent/fluent-logger-golang)
+ **Python** – [fluent-logger-python](https://github.com/fluent/fluent-logger-python)
+ **Java** – [fluent-logger-java](https://github.com/fluent/fluent-logger-java)
+ **Node.js** – [fluent-logger-node](https://github.com/fluent/fluent-logger-node)
+ **Ruby** – [fluent-logger-ruby](https://github.com/fluent/fluent-logger-ruby)
## Configure the Docker buffer limit
When you create a task definition, you can specify the number of log lines that are buffered in memory by specifying the value in `log-driver-buffer-limit`. This controls the buffer between Docker and Fluent Bit. For more information, see [Fluentd logging driver](https://docs.docker.com/engine/logging/drivers/fluentd/) in the Docker documentation.
Use this option when there's high throughput, because Docker might run out of buffer memory and discard buffer messages so it can add new messages.
Consider the following when using this option:
+ This option is supported on EC2 and Fargate type with platform version `1.4.0` or later.
+ The option is only valid when `logDriver` is set to `awsfirelens`.
+ The default buffer limit is `1048576` log lines.
+ The buffer limit must be greater than or equal to `0` and less than `536870912` log lines.
+ The maximum amount of memory used for this buffer is the product of the size of each log line and the size of the buffer. For example, if the application's log lines are on average `2` KiB, a buffer limit of 4096 would use at most `8` MiB. The total amount of memory allocated at the task level should be greater than the amount of memory that's allocated for all the containers in addition to the log driver memory buffer.
The following task definition shows how to configure `log-driver-buffer-limit`:
```
{
"containerDefinitions": [
{
"name": "my_service_log_router",
"image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:3",
"cpu": 0,
"memoryReservation": 51,
"essential": true,
"firelensConfiguration": {
"type": "fluentbit"
}
},
{
"essential": true,
"image": "public.ecr.aws/docker/library/httpd:latest",
"name": "app",
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"Name": "firehose",
"region": "us-west-2",
"delivery_stream": "my-stream",
"log-driver-buffer-limit": "52428800"
}
},
"dependsOn": [
{
"containerName": "my_service_log_router",
"condition": "START"
}
],
"memoryReservation": 100
}
]
}
```
# AWS for Fluent Bit image repositories for Amazon ECS
AWS provides a Fluent Bit image with plugins for both CloudWatch Logs and Firehose. We recommend using Fluent Bit as your log router because it has a lower resource utilization rate than Fluentd. For more information, see [CloudWatch Logs for Fluent Bit](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit) and [Amazon Kinesis Firehose for Fluent Bit](https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit).
The **AWS for Fluent Bit** image is available on Amazon ECR on both the Amazon ECR Public Gallery and in an Amazon ECR repository for high availability.
## Amazon ECR Public Gallery
The AWS for Fluent Bit image is available on the Amazon ECR Public Gallery. This is the recommended location to download the AWS for Fluent Bit image because it's a public repository and available to be used from all AWS Regions. For more information, see [aws-for-fluent-bit](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit) on the Amazon ECR Public Gallery.
### Linux
The AWS for Fluent Bit image in the Amazon ECR Public Gallery supports the Amazon Linux operating system with the `ARM64` or `x86-64` architecture.
You can pull the AWS for Fluent Bit image from the Amazon ECR Public Gallery by specifying the repository URL with the desired image tag. The available image tags can be found on the **Image tags** tab on the Amazon ECR Public Gallery.
The following shows the syntax to use for the Docker CLI.
```
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:tag
```
For example, you can pull the latest image in the "3.x" family of AWS for Fluent Bit releases using this Docker CLI command.
```
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:3
```
**Note**
Unauthenticated pulls are allowed, but have a lower rate limit than authenticated pulls. To authenticate using your AWS account before pulling, use the following command.
```
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws
```
#### AWS for Fluent Bit 3.0.0
In addition to the existing AWS for Fluent Bit versions `2.x`, AWS for Fluent Bit supports a new major version `3.x`. The new major version includes upgrading images from Amazon Linux 2 to Amazon Linux 2023 and Fluent Bit version `1.9.10` to `4.1.1`. For more information, see the [AWS for Fluent Bit repository](https://github.com/aws/aws-for-fluent-bit/blob/mainline/VERSIONS.md) on GitHub.
The following examples demonstrate updated tags for AWS for Fluent Bit `3.x` images:
You can use multi-architecture tags for the AWS for Fluent Bit image.
```
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:3
```
### Windows
The AWS for Fluent Bit image in the Amazon ECR Public Gallery supports the `AMD64` architecture with the following operating systems:
+ Windows Server 2022 Full
+ Windows Server 2022 Core
+ Windows Server 2019 Full
+ Windows Server 2019 Core
Windows containers that are on AWS Fargate don't support FireLens.
You can pull the AWS for Fluent Bit image from the Amazon ECR Public Gallery by specifying the repository URL with the desired image tag. The available image tags can be found on the **Image tags** tab on the Amazon ECR Public Gallery.
The following shows the syntax to use for the Docker CLI.
```
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:tag
```
For example, you can pull the newest stable AWS for Fluent Bit image using this Docker CLI command.
```
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:windowsservercore-stable
```
**Note**
Unauthenticated pulls are allowed, but have a lower rate limit than authenticated pulls. To authenticate using your AWS account before pulling, use the following command.
```
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws
```
## Amazon ECR
The AWS for Fluent Bit image is available on Amazon ECR for high availability. The following commands can be used to retreive image URIs and establish image availability in a given AWS Region.
### Linux
The latest stable AWS for Fluent Bit image URI can be retrieved using the following command.
```
aws ssm get-parameters \
--names /aws/service/aws-for-fluent-bit/stable \
--region us-east-1
```
All versions of the AWS for Fluent Bit image can be listed using the following command to query the Systems Manager Parameter Store parameter.
```
aws ssm get-parameters-by-path \
--path /aws/service/aws-for-fluent-bit \
--region us-east-1
```
The newest stable AWS for Fluent Bit image can be referenced in an CloudFormation template by referencing the Systems Manager parameter store name. The following is an example:
```
Parameters:
FireLensImage:
Description: Fluent Bit image for the FireLens Container
Type: AWS::SSM::Parameter::Value
Default: /aws/service/aws-for-fluent-bit/stable
```
**Note**
If the command fails or there is no output, the image isn't available in the AWS Region in which the command is called.
### Windows
The latest stable AWS for Fluent Bit image URI can be retrieved using the following command.
```
aws ssm get-parameters \
--names /aws/service/aws-for-fluent-bit/windowsservercore-stable \
--region us-east-1
```
All versions of the AWS for Fluent Bit image can be listed using the following command to query the Systems Manager Parameter Store parameter.
```
aws ssm get-parameters-by-path \
--path /aws/service/aws-for-fluent-bit/windowsservercore \
--region us-east-1
```
The latest stable AWS for Fluent Bit image can be referenced in an CloudFormation template by referencing the Systems Manager parameter store name. The following is an example:
```
Parameters:
FireLensImage:
Description: Fluent Bit image for the FireLens Container
Type: AWS::SSM::Parameter::Value
Default: /aws/service/aws-for-fluent-bit/windowsservercore-stable
```
# Example Amazon ECS task definition: Route logs to FireLens
To use custom log routing with FireLens, you must specify the following in your task definition:
+ A log router container that contains a FireLens configuration. We recommend that the container be marked as `essential`.
+ One or more application containers that contain a log configuration specifying the `awsfirelens` log driver.
+ A task IAM role Amazon Resource Name (ARN) that contains the permissions needed for the task to route the logs.
When creating a new task definition using the AWS Management Console, there is a FireLens integration section that makes it easy to add a log router container. For more information, see [Creating an Amazon ECS task definition using the console](create-task-definition.md).
Amazon ECS converts the log configuration and generates the Fluentd or Fluent Bit output configuration. The output configuration is mounted in the log routing container at `/fluent-bit/etc/fluent-bit.conf` for Fluent Bit and `/fluentd/etc/fluent.conf` for Fluentd.
**Important**
FireLens listens on port `24224`. Therefore, to ensure that the FireLens log router isn't reachable outside of the task, you must not allow ingress traffic on port `24224` in the security group your task uses. For tasks that use the `awsvpc` network mode, this is the security group that's associated with the task. For tasks that use the `host` network mode, this is the security group that's associated with the Amazon EC2 instance hosting the task. For tasks that use the `bridge` network mode, don't create any port mappings that use port `24224`.
By default, Amazon ECS adds additional fields in your log entries that help identify the source of the logs.
+ `ecs_cluster` – The name of the cluster that the task is part of.
+ `ecs_task_arn` – The full Amazon Resource Name (ARN) of the task that the container is part of.
+ `ecs_task_definition` – The task definition name and revision that the task is using.
+ `ec2_instance_id` – The Amazon EC2 instance ID that the container is hosted on. This field is only valid for tasks using the EC2 launch type.
You can set the `enable-ecs-log-metadata` to `false` if you do not want the metadata.
The following task definition example defines a log router container that uses Fluent Bit to route its logs to CloudWatch Logs. It also defines an application container that uses a log configuration to route logs to Amazon Data Firehose and sets the memory that's used to buffer events to the 2 MiB.
**Note**
For more example task definitions, see [Amazon ECS FireLens examples](https://github.com/aws-samples/amazon-ecs-firelens-examples) on GitHub.
```
{
"family": "firelens-example-firehose",
"taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role",
"containerDefinitions": [
{
"name": "log_router",
"image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:3",
"cpu": 0,
"memoryReservation": 51,
"portMappings": [],
"essential": true,
"environment": [],
"mountPoints": [],
"volumesFrom": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/ecs-aws-firelens-sidecar-container",
"mode": "non-blocking",
"awslogs-create-group": "true",
"max-buffer-size": "25m",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "firelens"
},
"secretOptions": []
},
"systemControls": [],
"firelensConfiguration": {
"type": "fluentbit"
}
},
{
"essential": true,
"image": "public.ecr.aws/docker/library/httpd:latest",
"name": "app",
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"Name": "firehose",
"region": "us-west-2",
"delivery_stream": "my-stream",
"log-driver-buffer-limit": "1048576"
}
},
"memoryReservation": 100
}
]
}
```
The key-value pairs specified as options in the `logConfiguration` object are used to generate the Fluentd or Fluent Bit output configuration. The following is a code example from a Fluent Bit output definition.
```
[OUTPUT]
Name firehose
Match app-firelens*
region us-west-2
delivery_stream my-stream
```
**Note**
FireLens manages the `match` configuration. You do not specify the `match` configuration in your task definition.
## Use a custom configuration file
You can specify a custom configuration file. The configuration file format is the native format for the log router that you're using. For more information, see [Fluentd Config File Syntax](https://docs.fluentd.org/configuration/config-file) and [YAML Configuration](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml).
In your custom configuration file, for tasks using the `bridge` or `awsvpc` network mode, don't set a Fluentd or Fluent Bit forward input over TCP because FireLens adds it to the input configuration.
Your FireLens configuration must contain the following options to specify a custom configuration file:
`config-file-type`
The source location of the custom configuration file. The available options are `s3` or `file`.
Tasks that are hosted on AWS Fargate only support the `file` configuration file type. However, you can use configuration files hosted in Amazon S3 on AWS Fargate by using the AWS for Fluent Bit init container. For more information, see [Init process for Fluent Bit on ECS, multi-config support](https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md) on GitHub.
`config-file-value`
The source for the custom configuration file. If the `s3` config file type is used, the config file value is the full ARN of the Amazon S3 bucket and file. If the `file` config file type is used, the config file value is the full path of the configuration file that exists either in the container image or on a volume that's mounted in the container.
When using a custom configuration file, you must specify a different path than the one FireLens uses. Amazon ECS reserves the `/fluent-bit/etc/fluent-bit.conf` filepath for Fluent Bit and `/fluentd/etc/fluent.conf` for Fluentd.
The following example shows the syntax required when specifying a custom configuration.
**Important**
To specify a custom configuration file that's hosted in Amazon S3, ensure you have created a task execution IAM role with the proper permissions.
The following shows the syntax required when specifying a custom configuration.
```
{
"containerDefinitions": [
{
"essential": true,
"image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:3",
"name": "log_router",
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "s3 | file",
"config-file-value": "arn:aws:s3:::amzn-s3-demo-bucket/fluent.conf | filepath"
}
}
}
]
}
```
**Note**
Tasks hosted on AWS Fargate only support the `file` configuration file type. However, you can use configuration files hosted in Amazon S3 on AWS Fargate by using the AWS for Fluent Bit init container. For more information, see [Init process for Fluent Bit on ECS, multi-config support](https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md) on GitHub.