Tutorial: Ingesting data into a collection using Amazon OpenSearch Ingestion

To create the pipeline role

1. Open the AWS Identity and Access Management console at https://console.aws.amazon.com/iamv2/.

2. Choose Policies, and then choose Create policy.

3. Select JSON and paste the following policy into the editor. Modify the collection ARN and name accordingly.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Action": [
           "aoss:BatchGetCollection",
           "aoss:APIAccessAll"
         ],
         "Effect": "Allow",
         "Resource": "arn:aws:aoss:us-east-1:your-account-id:collection/collection-id"
       },
       {
         "Action": [
           "aoss:CreateSecurityPolicy",
           "aoss:GetSecurityPolicy",
           "aoss:UpdateSecurityPolicy"
         ],
         "Effect": "Allow",
         "Resource": "*",
         "Condition": {
           "StringEquals": {
             "aoss:collection": "collection-name"
           }
         }
       }
     ]
   }

4. Choose Next, choose Next, and name your policy collection-pipeline-policy.

5. Choose Create policy.

6. Next, create a role and attach the policy to it. Choose Roles, and then choose Create role.

7. Choose Custom trust policy and paste the following policy into the editor:

   {
      "Version":"2012-10-17",
      "Statement":[
         {
            "Effect":"Allow",
            "Principal":{
               "Service":"osis-pipelines.amazonaws.com"
            },
            "Action":"sts:AssumeRole"
         }
      ]
   }

8. Choose Next. Then search for and select collection-pipeline-policy (which you just created).

9. Choose Next and name the role PipelineRole.

10. Choose Create role.

1. Navigate to the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/home.

2. Choose Collections from the left navigation and choose Create collection.

3. Name the collection ingestion-collection.

4. For Security, choose Standard create.

5. Under Network access settings, change the access type to Public.

6. Keep all other settings as their defaults and choose Next.

7. Now, configure a data acces policy for the collection. For Definition method, choose JSON and paste the following policy into the editor. This policy does two things:

   • Allows the pipeline role to write to the collection.

   • Allows you to read from the collection. Later, after you ingest some sample data into the pipeline, you'll query the collection to ensure that the data was successfully ingested and written to the index.

     [
       {
         "Rules": [
           {
             "Resource": [
               "index/ingestion-collection/*"
             ],
             "Permission": [
               "aoss:CreateIndex",
               "aoss:UpdateIndex",
               "aoss:DescribeIndex",
               "aoss:ReadDocument",
               "aoss:WriteDocument"
             ],
             "ResourceType": "index"
           }
         ],
         "Principal": [
           "arn:aws:iam::your-account-id:role/PipelineRole",
           "arn:aws:iam::your-account-id:role/Admin"
         ],
         "Description": "Rule 1"
       }
     ]

8. Replace the Principal elements. The first principal should specify the pipeline role that you created. The second should specify a user or role that you can use to query the collection later.

9. Choose Next. Name the access policy pipeline-domain-access and choose Next again.

10. Review your collection configuration and choose Submit.

To create a pipeline

1. Within the Amazon OpenSearch Service console, choose Pipelines from the left navigation pane.

2. Choose Create pipeline.

3. Select the blank pipeline, then choose Select blueprint.

4. Name the pipeline serverless-ingestion and keep the capacity settings as their defaults.

5. In this tutorial, we'll create a simple sub-pipeline called log-pipeline that uses the HTTP source plugin. The plugin accepts log data in a JSON array format. We'll specify a single OpenSearch Serverless collection as the sink, and ingest all data into the my_logs index.

   Under Pipeline configuration, paste the following YAML configuration into the editor:

   version: "2"
   log-pipeline:
     source:
       http:
         path: "/${pipelineName}/test_ingestion_path"
     processor:
       - date:
           from_time_received: true
           destination: "@timestamp"
     sink:
       - opensearch:
           hosts: [ "https://collection-id.us-east-1.aoss.amazonaws.com" ]
           index: "my_logs"
           aws:
             sts_role_arn: "arn:aws:iam::your-account-id:role/PipelineRole"
             region: "us-east-1"
             serverless: true

6. Replace the hosts URL with the endpoint of the collection that you created in the previous section. Replace the sts_role_arn parameter with the ARN of PipelineRole. Optionally, modify the region.

7. Choose Validate pipeline and make sure that the validation succeeds.

8. For simplicity in this tutorial, we'll configure public access for the pipeline. Under Network, choose Public access.

   For information about configuring VPC access, see Configuring VPC access for Amazon OpenSearch Ingestion pipelines.

9. Keep log publishing enabled in case you encounter any issues while completing this tutorial. For more information, see Monitoring pipeline logs.

   Specify the following log group name: /aws/vendedlogs/OpenSearchIngestion/serverless-ingestion/audit-logs

10. Choose Next. Review your pipeline configuration and choose Create pipeline. The pipeline takes 5–10 minutes to become active.