Overview of Amazon OpenSearch Ingestion
Amazon OpenSearch Ingestion is a fully managed, serverless data collector that streams real-time logs, metrics, and trace data to Amazon OpenSearch Service domains and OpenSearch Serverless collections.
With OpenSearch Ingestion, you no longer need third-party tools like Logstash or Jaeger to ingest data. You configure your data producers to send data to OpenSearch Ingestion, and it automatically delivers it to your specified domain or collection. You can also transform data before delivery.
Because OpenSearch Ingestion is serverless, you don’t have to manage infrastructure, patch software, or scale clusters manually. You can provision ingestion pipelines directly in the AWS Management Console, and OpenSearch Ingestion handles the rest.
As a component of Amazon OpenSearch Service, OpenSearch Ingestion is powered by Data Prepper—an open-source data collector that filters, enriches, transforms, normalizes, and aggregates data for downstream analysis and visualization.
Supported Data Prepper versions
OpenSearch Ingestion currently supports the following major versions of Data Prepper:
-
2.x
When you create a pipeline, use the required version option to specify
the major version of Data Prepper to use. For example, version:
". OpenSearch Ingestion retrieves the latest
supported minor version of that major version and provisions the
pipeline with that version. For more information, see Specifying the pipeline version.2"
OpenSearch Ingestion pipelines are always provisioned with the latest version of Data
Prepper. For information about the features and bug fixes that are in each version of
Data Prepper, see the Releases
When you update a pipeline's YAML configuration file, if there's support for a new
minor version of Data Prepper, OpenSearch Ingestion automatically upgrades the pipeline to
the latest supported minor version of the major version that's specified in the pipeline
configuration. For example, you might have version: "2" in your pipeline
configuration, and OpenSearch Ingestion initially provisioned the pipeline with version
2.6.0. When support for version 2.7.0 is added, and you make a change to the pipeline
configuration, OpenSearch Ingestion upgrades the pipeline to version 2.7.0. This process
keeps your pipeline up to date with the latest bug fixes and performance improvements.
OpenSearch Ingestion can't update the major version of your pipeline unless you manually
change the version option within the pipeline configuration. For more
information, see Updating Amazon OpenSearch Ingestion pipelines.
OpenSearch Ingestion pricing
At any specific time, you only pay for the number of Ingestion OCUs that are allocated to a pipeline, regardless of whether there's data flowing through the pipeline. OpenSearch Ingestion immediately accommodates your workloads by scaling pipeline capacity up or down based on usage.
For full pricing details, see Amazon OpenSearch Service
pricing
Supported AWS Regions
OpenSearch Ingestion is available in a subset of AWS Regions that OpenSearch Service is available in. For a list of supported Regions, see Amazon OpenSearch Service endpoints and quotas in the AWS General Reference.
