Amazon S3 logs - Centralized Logging with OpenSearch
Create log ingestion (OpenSearch Engine)

Amazon S3 logs

Amazon S3 server access logging provides detailed records for the requests made to the bucket. S3 Access Logs can be enabled and saved in another S3 bucket.

You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.

Important

  • The S3 Bucket Region must be the same as the Centralized Logging with OpenSearch solution Region.

  • The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.

Create log ingestion (OpenSearch Engine)

Using the Centralized Logging with OpenSearch Console

  1. Sign in to the Centralized Logging with OpenSearch Console.

  2. In the navigation pane, under Log Analytics Pipelines, choose Service Log.

  3. Choose the Create a log ingestion button.

  4. In the AWS Services section, choose Amazon S3.

  5. Choose Next.

  6. Under Specify settings, choose Automatic or Manual for Amazon S3 Access Log enabling. The automatic mode will enable the Amazon S3 Access Log and save the logs to a centralized S3 bucket if logging is not enabled yet.

    • For Automatic mode, choose the S3 bucket from the dropdown list.

    • For Manual mode, enter the Bucket Name and Amazon S3 Access Log location.

    • (Optional) If you are ingesting Amazon S3 logs from another account, select a linked account from the Account dropdown list first.

  7. Choose Next.

  8. In the Specify OpenSearch domain section, select an imported domain for the Amazon OpenSearch Service domain.

  9. Choose Yes for Sample dashboard if you want to ingest an associated built-in Amazon OpenSearch Service dashboard.

  10. You can change the Index Prefix of the target Amazon OpenSearch Service index if needed. The default prefix is your bucket name.

  11. In the Log Lifecycle section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated Index State Management (ISM) policy automatically for this pipeline.

  12. Choose Next.

  13. Add tags if needed.

  14. Choose Create.

Using the CloudFormation Stack

This automated AWS CloudFormation template deploys the Centralized Logging with OpenSearch - Amazon S3 Access Log Ingestion solution in the AWS Cloud.

Launch in AWS Management Console Download Template
AWS Regions

Launch stack button.

Template
AWS China Regions

Launch stack button.

Template

  1. Log in to the AWS Management Console and select the preceding button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.

  2. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.

  3. On the Create stack page, verify that the correct template URL shows in the Amazon S3 URL text box and choose Next.

  4. On the Specify stack details page, assign a name to your solution stack.

  5. Under Parameters, review the parameters for the template and modify them as necessary. This solution uses the following parameters.

    Parameter Default Description
    Log Bucket Name <Requires input> The S3 bucket name which stores the logs.
    Log Bucket Prefix <Requires input> The S3 bucket path prefix which stores the logs.
    Log Source Account ID Optional input The AWS Account ID of the S3 bucket. Required for cross-account log ingestion (add a member account first). By default, the Account ID you logged in at Step 1 will be used.
    Log Source Region Optional input The AWS Region of the S3 bucket. By default, the Region you selected at Step 2 will be used.
    Log Source Account Assume Role Optional input The IAM Role ARN used for cross-account log ingestion. Required for cross-account log ingestion (add a member account first).
    KMS-CMK ARN Optional input The KMS-CMK ARN for encryption. Leave it blank to create a new AWS KMS key.
    Enable OpenSearch Ingestion as processor Optional input Ingestion table ARN. Leave empty if you do not use OSI as Processor.
    Amazon S3 Backup Bucket <Requires input> The Amazon S3 backup bucket name to store the failed ingestion logs.
    Engine Type OpenSearch The engine type of the OpenSearch.
    OpenSearch Domain Name <Requires input> The domain name of the Amazon OpenSearch Service cluster.
    OpenSearch Endpoint <Requires input> The OpenSearch endpoint URL. For example, vpc-your_opensearch_domain_name-xcvgw6uu2o6zafsiefxubwuohe.us-east-1.es.amazonaws.com
    Index Prefix <Requires input> The common prefix of OpenSearch index for the log. The index name will be <Index Prefix>-<Log Type>-<Other Suffix>.
    Create Sample Dashboard Yes Whether to create a sample OpenSearch dashboard.
    VPC ID <Requires input> Select a VPC that has access to the OpenSearch domain. The log processing Lambda will reside in the selected VPC.
    Subnet IDs <Requires input> Select at least two subnets that have access to the OpenSearch domain. The log processing Lambda will reside in the subnets. Make sure that the subnets have access to the Amazon S3 service.
    Security Group ID <Requires input> Select a Security Group that will be associated with the log processing Lambda. Make sure that the Security Group has access to the OpenSearch domain.
    Number Of Shards 5 Number of shards to distribute the index evenly across all data nodes. Keep the size of each shard between 10-50 GB.
    Number of Replicas 1 Number of replicas for OpenSearch Index. Each replica is a full copy of an index. If the OpenSearch option is set to Domain with standby, you need to configure it to 2.
    Age to Warm Storage Optional input The age required to move the index into warm storage (for example, 7d). Index age is the time between its creation and the present. Supported units are d (days) and h (hours). This is only effective when warm storage is enabled in OpenSearch.
    Age to Cold Storage Optional input The age required to move the index into cold storage (for example, 30d). Index age is the time between its creation and the present. Supported units are d (days) and h (hours). This is only effective when cold storage is enabled in OpenSearch.
    Age to Retain Optional input The age to retain the index (for example, 180d). Index age is the time between its creation and the present. Supported units are d (days) and h (hours). If the value is "", the index will not be deleted.
    Rollover Index Size Optional input The minimum size of the shard storage required to roll over the index (for example, 30GB).
    Index Suffix yyyy-MM-dd The common suffix format of OpenSearch index for the log(Example: yyyy-MM-dd, yyyy-MM-dd-HH). The index name will be <Index Prefix>-<Log Type>-<Index Suffix>-000001.
    Compression type best_compression The compression type to use to compress stored data. Available values are best_compression and default.
    Refresh Interval 1s How often the index should refresh, which publishes its most recent changes and makes them available for searching. Can be set to -1 to disable refreshing. Default is 1s.
    EnableS3Notification True An option to enable or disable notifications for Amazon S3 buckets. The default option is recommended for most cases.
    LogProcessorRoleName Optional input Specify a role name for the log processor. The name should NOT duplicate an existing role name. If no name is specified, a random name is generated.
    QueueName Optional input Specify a queue name for an Amazon SQS queue. The name should NOT duplicate an existing queue name. If no name is given, a random name will be generated.

  6. Choose Next.

  7. On the Configure stack options page, choose Next.

  8. On the Review and create page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.

  9. Choose Submit to deploy the stack.

You can view the status of the stack in the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in approximately 10 minutes.

View dashboard

The dashboard includes the following visualizations.

Visualization Name Source Field Description
Total Requests

  • log event

A visualization showing the total number of requests made to the Amazon S3 bucket, including all types of operations (for example, GET, PUT, DELETE).
Unique Visitors

  • log event

This visualization displays the count of unique visitors accessing the Amazon S3 bucket, identified by their IP addresses.
Access History

  • log event

Provides a chronological log of all access events made to the Amazon S3 bucket, including details about the operations and their outcomes.
Request By Operation

  • operation

This visualization categorizes and shows the distribution of requests based on different operations (for example, GET, PUT, DELETE).
Status Code

  • http_status

Displays the count of requests made to the Amazon S3 bucket, grouped by HTTP status codes returned by the server (for example, 200, 404, 403).
Status Code History

  • http_status

Shows the historical trend of HTTP status codes returned by the Amazon S3 server over a specific period of time.
Status Code Pie

  • http_status

Represents the distribution of requests based on different HTTP status codes using a pie chart.
Average Time

  • total_time

This visualization calculates and presents the average time taken for various operations in the Amazon S3 bucket (for example, average time for GET, PUT requests).
Average Turn Around Time

  • turn_around_time

Shows the average turnaround time for different operations, which is the time between receiving a request and sending the response back to the client.
Data Transfer

  • bytes_sent

  • object_size

  • operation

Provides insights into data transfer activities, including the total bytes transferred, object sizes, and different operations involved.
Top Client IPs

  • remote_ip

Displays the top client IP addresses with the highest number of requests made to the Amazon S3 bucket.
Top Request Keys

  • key

  • object_size

Shows the top requested keys in the Amazon S3 bucket along with the corresponding object sizes.
Delete Events

  • operation

  • key

  • version_id

  • object_size

  • remote_ip

  • http_status

  • error_code

Focuses on delete events, including the operation, key, version ID, object size, client IP, HTTP status, and error code associated with the delete requests.
Access Failures

  • operation

  • key

  • version_id

  • object_size

  • remote_ip

  • http_status

  • error_code

Highlights access failures, showing the details of the failed requests, including operation, key, version ID, object size, client IP, HTTP status, and error code.

You can access the built-in dashboard in Amazon OpenSearch Service to view log data. For more information, see the Access Dashboard.

Amazon S3 logs sample dashboard.