Build your own centralized log analytics platform with Amazon OpenSearch Service in 20 minutes
Publication date: March 2023 (last update: August 2024)
The Centralized Logging with OpenSearch solution provides comprehensive log management and analysis functions to help you simplify the build of log analytics pipelines. Built on top of Amazon OpenSearch Service, the solution helps you to streamline log ingestion, log processing, and log visualization. You can use the solution in multiple use cases, such as to abide by security and compliance regulations, achieve refined business operations, and enhance IT troubleshooting and maintenance.
Important
Centralized Logging with OpenSearch supports Amazon OpenSearch Service with OpenSearch 1.3 or later.
Use this navigation table to quickly find responses to these questions:
| If you want to … | Read… |
|---|---|
| Know the cost for running this solution | Cost |
| Understand the security considerations for this solution | Security |
| Know which AWS Regions are supported for this solution | Supported AWS Regions |
| Get started with the solution quickly to import an Amazon OpenSearch Service domain, build a log analytics pipeline, and access the built-in dashboard | Getting started |
| Learn the operations related to Amazon OpenSearch Service domains | Domain management |
| Walk through the processes of building log analytics pipelines | AWS Services logs and Application logs |
| Encountering issues when using the solution | Troubleshooting |
| Go through a hands-on workshop designed for this solution |
Workshop |
This implementation guide describes architectural considerations and configuration steps for deploying the Centralized Logging with OpenSearch solution in the AWS Cloud. It includes links to CloudFormation templates that launch and configure the AWS services required to deploy this solution using AWS best practices for security and availability.
The guide is intended for IT architects, developers, DevOps, and data engineers with practical experience architecting on the AWS Cloud.
Features and benefits
The solution has the following features:
All-in-one log ingestion
Provides a single web console to ingest both application logs and AWS service logs into Log Analytics Engines. For supported AWS service logs, refer to AWS Service Logs. For supported application logs, refer to Application Logs.
Codeless log processor
Supports log processor plugins developed by AWS. You can enrich the raw log data through a few steps on the web console.
Dashboard template
Offers a collection of reference designs of visualization templates, for both commonly used software such as NGINX and Apache HTTP Server, and AWS services such as Amazon S3 and AWS CloudTrail.
Use cases
The solution can be applied to the following use cases:
Security and compliance regulations
Comply with regulatory requirements such as GDPR, PCI DSS, MLPS, and HIPAA. Easily store equipment, network, and application logs in a centralized place for log auditing and threat detection.
Business operations and data analysis
Identify trends and patterns in minutes, and build interactive and intuitive visualization. Derive business insights from logs and inform business decisions with data.
Application and infrastructure troubleshooting
Monitor both application and cloud infrastructure logs with ease, understand and resolve the root cause of issues quickly. Improve the observability of your workloads and achieve better business stability.
Concepts
This section describes key concepts and defines terminology specific to this solution:
Log Analytics Engine
A log analytics engine is a sophisticated tool designed to process, analyze, and visualize vast amounts of log data from diverse systems, applications, and devices. Our solution primarily uses the Amazon OpenSearch Service as the default log analytics engine, complemented by a Light Engine specifically optimized for structured, infrequent logs.
OpenSearch Engine
The OpenSearch Engine in this solution refers to the
Amazon OpenSearch Service
Light Engine
The Light Engine is a serverless log analytics engine that uses AWS services like Athena, Glue, Lambda, and Step Functions. Designed to analyze structured and infrequent logs, it offers up to a 90% cost reduction compared to the OpenSearch Engine.
Log Analytics Pipeline
A Log Analytics Pipeline, or Log Pipeline, represents the entire data flow from the source to the log analytics engines. It typically encompasses the stages of shipping, buffering, processing, filtering, enriching, and storing logs. Centralized Logging with OpenSearch supports two types of Log Analytics Pipelines: the Service Log Pipeline, tailored for ingesting logs generated by AWS Services, and the Application Log Pipeline, designed for ingesting logs from custom applications.
Log Source
A Log Config defines the metadata of your logs, specifying the log type, format, sample logs, filters, and the schema needed to map raw log data into the structured format used by the log analytics engine. A Log Source refers to the location where logs are generated or stored. Centralized Logging with OpenSearch supports ingesting logs from diverse sources, encompassing both application logs and logs from AWS services. For supported AWS service logs, refer to AWS Service Logs. For supported application logs, refer to Application Logs.
Log Agent
A log agent is a program that reads logs from one location and
sends them to another location (for example, OpenSearch).
Currently, Centralized Logging with OpenSearch only supports the
Fluent Bit
1.9
Log Config
A Log Config defines the metadata of your logs, specifying the log type, format, sample logs, filters, and the schema needed to map raw log data into the structured format used by the log analytics engine.
Log Buffer
Log Buffer is a buffer layer between the Log Agent and OpenSearch clusters. The agent uploads logs into the buffer layer before being processed and delivered into the log analytics engine. A buffer layer is a way to protect the log analytics engine from being overwhelmed. For AWS service logs, a log buffer is automatically configured if needed. For Application logs, this solution provides the following types of buffer layers.
-
Amazon S3. Use this option if you can bear minutes-level latency for log ingestion. The log agent periodically uploads logs to an Amazon S3 bucket. The frequency of data delivery to Amazon S3 is determined by Buffer size (default value is 50 MiB) and Buffer interval (default value is 60 seconds) values that you configured when creating the application log analytics pipelines. The condition satisfied first starts data delivery to Amazon S3.
-
Amazon Kinesis Data Streams. Use this option if you need real-time log ingestion. The log agent uploads logs to Amazon Kinesis Data Stream in seconds. The frequency of data delivery to Kinesis Data Streams is determined by Buffer size (10 MiB) and Buffer interval (5 seconds). The condition satisfied first triggers data delivery to Kinesis Data Streams.
Log Buffer is optional when creating an application log analytics pipeline. For all types of application logs, you can use this solution to ingest logs without any buffer layers. However, we only recommend this option when you have small log volume, and you are confident that the logs will not exceed the thresholds at the OpenSearch side.
Instance Group
An Instance Group represents a group of EC2 instances, which
enables the solution to associate a Log Config with multiple
EC2 instances quickly. Centralized Logging with OpenSearch uses
Systems
Manager Agent (SSM Agent) to install/configure Fluent Bit
agent, and sends log data to
Kinesis
Data Streams
Main Account
An AWS account where the Centralized Logging with OpenSearch console is deployed. The Log Analytics Engines must also reside in the same account.
Member Account
Another AWS account from which you want to ingest AWS Service logs or application logs. Logs are sent from Member Accounts to Main Accounts, where they are analyzed using resources in the Main Account.
Access Proxy
An Access Proxy serves as an intermediary for accessing Amazon OpenSearch Service domains from the internet securely. By default, an Amazon OpenSearch Service domain within a VPC is not accessible from the internet. The Centralized Logging with OpenSearch solution implements a Nginx-based proxy stack architecture to enable internet access to OpenSearch Dashboards. This allows users to interact conveniently with the dashboards from anywhere with internet connectivity.
