Use a web console from your AWS account to ingest both application and AWS service logs, then analyze the logs with visualization dashboards.
Overview
Important: This AWS Solution will retire in December 2026. Deployments (via CloudFormation or GitHub) will remain operational, but customers will assume responsibility for maintenance and API-related updates post retirement.
We encourage customers to explore using
Amazon CloudWatch's new unified data management and analytics capabilities
You can find other AWS Solutions in the AWS Solutions Library
Centralized Logging with OpenSearch helps organizations collect, ingest, and visualize log data from various sources using Amazon OpenSearch Service. This AWS Solution provides a web-based console, which you can use to create log ingestion pipelines with a few clicks. Log ingestion pipelines include log collection agent deployment, log enrichment without writing codes, buffer layer creation, and OpenSearch index configuration. After logs are stored in OpenSearch Service, the solution automatically generates ready-to-use dashboards for analyzing AWS service logs and application logs in different formats (for example, Nginx, JSON, and Spring Boot). In combination with other AWS services, this solution provides you with a turnkey environment to begin logging and monitoring your AWS applications.
Benefits
Ease of use
Improved operational efficiency
Serverless technologies with built-in high availability and a pay-for-use billing model reduces the need for infrastructure management, allowing you to focus more on building log analytics for your business.
Open source and customization
The solution is open sourced and free for commercial use. You pay only for the AWS usage. You can take the source code as a reference to make your own implementation that fits your needs.
How it works
You can automatically deploy this architecture using the implementation guide and the AWS CloudFormation templates for AWS Regions or AWS China Regions.
Step 1
Amazon CloudFront distributes the frontend web UI assets hosted in an Amazon S3 bucket.
Amazon Cognito user pool or OpenID Connector (OIDC) can be used for authentication.
AWS AppSync provides the backend GraphQL APIs.
Amazon DynamoDB stores the solution-related information as the backend database.
AWS Lambda interacts with other AWS Services to process the core logic of managing log pipeline, log agents, and obtains information updated in DynamoDB tables.
AWS Step Functions orchestrates the on-demand AWS CloudFormation deployment of a set of predefined stacks for log pipeline management. The log pipeline stacks deploy separate AWS resources and are used to collect and process logs and ingest them into Amazon OpenSearch Service for further analysis and visualization.
Service Log Pipeline or Application Log Pipeline is provisioned on demand via Centralized Logging with the OpenSearch console.
AWS Systems Manager and Amazon EventBridge manage log agents for collecting logs from application servers, such as installing log agents (Fluent Bit) for application servers and monitoring the health status of the agents.
Amazon EC2 or Amazon EKS installs Fluent Bit agents and uploads log data to the application log pipeline.
Application log pipelines read, parse, process application logs, and ingest them into Amazon OpenSearch Service domains or Light Engine.
Service log pipelines read, parse, process AWS service logs and ingest them into Amazon OpenSearch Service domains or Light Engine.
Deploy with confidence
We'll walk you through it
Get started fast. Read the implementation guide for deployment steps, architecture details, cost information, and customization options.Open guide
Let's make it happen
Ready to deploy? Open the CloudFormation template in the AWS Console to begin setting up the infrastructure you need. You'll be prompted to access your AWS account if you haven't yet logged in.Launch in the AWS Console:Launch in a new VPC in AWS RegionsLaunch in an existing VPC in AWS RegionsLaunch in a new VPC in AWS China RegionsLaunch in an existing VPC in China Regions
Deployment Options
Source Code
The source code for this AWS Solution is available in GitHub.
Implementation Guide
Follow the implementation guide for step-by-step instructions to deploy this AWS Solution.
Related content
Solution Web Console
This image shows a preview of the web console for Centralized Logging with OpenSearch.
Blog Amazon CloudWatch Cross-Account Observability
This blog describes an Amazon CloudWatch capability to search, analyze, and correlate cross-account telemetry data stored in CloudWatch such as metrics, logs, and traces.