Creating Amazon OpenSearch Ingestion pipelines

To create a pipeline

1. Sign in to the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/home.

2. Choose Pipelines in the left navigation pane and choose Create pipeline.

3. Either select a blank pipeline, or choose a configuration blueprint. Blueprints include a preconfigured YAML and JSON configuration file for a variety of common use cases. For more information, see Using blueprints to create a pipeline.

   Choose Select blueprint.

4. Enter a name for the pipeline.

5. (Optional) Choose Enable persistent buffer. A persistent buffer stores your data in a disk-based buffer across multiple AZs. For more information, see Persistent buffering. If you enable persistent buffer, select the AWS Key Management Service key to encrypt the buffer data.

6. Configure the minimum and maximum pipeline capacity in Ingestion OpenSearch Compute Units (OCUs). For more information, see Scaling pipelines.

7. Under Pipeline configuration, provide your pipeline configuration in YAML format. If you're using a blueprint, the configuration is already pre-populated, but you must make some modifications.

   A single pipeline configuration file can contain 1-10 sub-pipelines. Each sub-pipeline is a combination of a single source, zero or more processors, and a single sink. For OpenSearch Ingestion, the sink must always be an OpenSearch Service domain. For a list of supported options, see Supported plugins and options for Amazon OpenSearch Ingestion pipelines.

   Note

   You must include the sts_role_arn option in each sub-pipeline. The pipeline assumes the role defined in sts_role_arn to sign requests to the domain. For more information, see Granting Amazon OpenSearch Ingestion pipelines access to domains.

   The following sample configuration file uses the HTTP source and Grok plugins to process unstructured log data and send it to an OpenSearch Service domain. The sub-pipeline is named log-pipeline.

   version: "2"
   log-pipeline:
     source:
       http:
         path: "/log/ingest"
     processor:
       - grok:
           match:
             log: [ '%{COMMONAPACHELOG}' ]
       - date:
           from_time_received: true
           destination: "@timestamp"
     sink:
       - opensearch:
           hosts: [ "https://search-my-domain.us-east-1.es.amazonaws.com" ]
           index: "apache_logs"
           aws:
             sts_role_arn: "arn:aws:iam::123456789012:role/{pipeline-role}"
             region: "us-east-1"

   You can build your own pipeline configuration, or choose Upload file and import an existing configuration for a self-managed Data Prepper pipeline. Alternatively, you can use a configuration blueprint.

8. After you configure your pipeline, choose Validate pipeline to confirm that your configuration is correct. If the validation fails, fix the errors and re-run the validation.

9. Under Network configuration, choose either VPC access or Public access. If you choose Public access, skip to the next step. If you choose VPC access, configure the following settings:

   Setting	Description
   Endpoint management	Choose whether you want to create your VPC endpoints yourself, or have OpenSearch Ingestion create them for you. Endpoint management defaults to endpoints managed by OpenSearch Ingestion.
   VPC	Choose the ID of the virtual private cloud (VPC) that you want to use. The VPC and pipeline must be in the same AWS Region.
   Subnets	Choose one or more subnets. OpenSearch Service will place a VPC endpoint and elastic network interfaces in the subnets.
   Security groups	Choose one or more VPC security groups that allow your required application to reach the OpenSearch Ingestion pipeline on the ports (80 or 443) and protocols (HTTP or HTTPs) exposed by the pipeline.
   VPC attachment options	If your source is a self-managed endpoint, attach your pipeline to a VPC. Choose one of the default CIDR options provided, or use a custom CIDR.

   For more information, see Configuring VPC access for Amazon OpenSearch Ingestion pipelines.

10. (Optional) Under Tags, add one or more tags (key-value pairs) to your pipeline. For more information, see Tagging Amazon OpenSearch Ingestion pipelines.

11. (Optional) Under Log publishing options, turn on pipeline log publishing to Amazon CloudWatch Logs. We recommend that you enable log publishing so that you can more easily troubleshoot pipeline issues. For more information, see Monitoring pipeline logs.

12. Choose Next.

13. Review your pipeline configuration and choose Create.