Using an OpenSearch Ingestion pipeline with Fluent Bit - Amazon OpenSearch Service

Using an OpenSearch Ingestion pipeline with Fluent Bit

This sample Fluent Bit configuration file sends log data from Fluent Bit to an OpenSearch Ingestion pipeline. For more information about ingesting log data, see Log Analytics in the Data Prepper documentation.

Note the following:

  • The host value must be your pipeline endpoint. For example, pipeline-endpoint.us-east-1.osis.amazonaws.com.

  • The aws_service value must be osis.

  • The aws_role_arn value is the ARN of the AWS IAM role for the client to assume and use for Signature Version 4 authentication.

[INPUT] name tail refresh_interval 5 path /var/log/test.log read_from_head true [OUTPUT] Name http Match * Host pipeline-endpoint.us-east-1.osis.amazonaws.com Port 443 URI /log/ingest Format json aws_auth true aws_region us-east-1 aws_service osis aws_role_arn arn:aws:iam::{account-id}:role/ingestion-role Log_Level trace tls On

You can then configure an OpenSearch Ingestion pipeline like the following, which has HTTP as the source:

version: "2" unaggregated-log-pipeline: source: http: path: "/log/ingest" processor: - grok: match: log: - "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:network_node} %{NOTSPACE:network_host} %{IPORHOST:source_ip}:%{NUMBER:source_port:int} -> %{IPORHOST:destination_ip}:%{NUMBER:destination_port:int} %{GREEDYDATA:details}" - grok: match: details: - "'%{NOTSPACE:http_method} %{NOTSPACE:http_uri}' %{NOTSPACE:protocol}" - "TLS%{NOTSPACE:tls_version} %{GREEDYDATA:encryption}" - "%{NUMBER:status_code:int} %{NUMBER:response_size:int}" - delete_entries: with_keys: ["details", "log"] sink: - opensearch: hosts: ["https://search-domain-endpoint.us-east-1.es.amazonaws.com"] index: "index_name" index_type: custom bulk_size: 20 aws: # IAM role that the pipeline assumes to access the domain sink sts_role_arn: "arn:aws:iam::{account-id}:role/pipeline-role" region: "us-east-1"