# Amazon OpenSearch Service tutorials
This chapter includes several start-to-finish tutorials for working with Amazon OpenSearch Service, including how to migrate to the service, build a simple search application, and create a visualization in OpenSearch Dashboards.
**Topics**
+ [Tutorial: Creating and searching for documents in Amazon OpenSearch Service](quick-start.md)
+ [Tutorial: Migrating to Amazon OpenSearch Service](migration.md)
+ [Tutorial: Creating a search application with Amazon OpenSearch Service](search-example.md)
+ [Tutorial: Visualizing customer support calls with OpenSearch Service and OpenSearch Dashboards](walkthrough.md)
# Tutorial: Creating and searching for documents in Amazon OpenSearch Service
In this tutorial, you learn how to create and search for a document in Amazon OpenSearch Service. You add data to an index in the form of a JSON document. OpenSearch Service creates an index around the first document that you add.
This tutorial explains how to make HTTP requests to create documents, automatically generate an ID for a document, and perform basic and advanced searches on your documents.
**Note**
This tutorial uses a domain with open access. For the highest level of security, we recommend that you put your domain inside a virtual private cloud (VPC).
## Prerequisites
This tutorial has the following prerequisites:
+ You must have an AWS account.
+ You must have an active OpenSearch Service domain.
## Adding a document to an index
To add a document to an index, you can use any HTTP tool, such as [Postman](https://www.getpostman.com/), cURL, or the OpenSearch Dashboards console. These examples assume that you’re using the developer console in OpenSearch Dashboards. If you’re using a different tool, adjust accordingly by providing the full URL and credentials, if necessary.
**To add a document to an index**
1. Navigate to the OpenSearch Dashboards URL for your domain. You can find the URL on the domain's dashboard in the OpenSearch Service console. The URL follows this format:
```
domain-endpoint/_dashboards/
```
1. Sign in using your primary username and password.
1. Open the left navigation panel and choose **Dev Tools**.
1. The HTTP verb for creating a new resource is PUT, which is what you use to create a new document and index. Enter the following command in the console:
```
PUT fruit/_doc/1
{
"name":"strawberry",
"color":"red"
}
```
The `PUT` request creates an index named *fruit* and adds a single document to the index with an ID of 1. It produces the following response:
```
{
"_index" : "fruit",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}
```
## Creating automatically generated IDs
OpenSearch Service can automatically generate an ID for your documents. The command to generate IDs uses a POST request instead of a PUT request, and it requires no document ID (in comparison to the previous request).
Enter the following request in the developer console:
```
POST veggies/_doc
{
"name":"beet",
"color":"red",
"classification":"root"
}
```
This request creates an index named *veggies* and adds the document to the index. It produces the following response:
```
{
"_index" : "veggies",
"_type" : "_doc",
"_id" : "3WgyS4IB5DLqbRIvLxtF",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}
```
Note that additional `_id` field in the response, which indicates that an ID was automatically created.
**Note**
You don't provide anything after `_doc` in the URL, where the ID normally goes. Because you’re creating a document with a generated ID, you don’t provide one yet. That’s reserved for updates.
## Updating a document with a POST command
To update a document, you use an HTTP `POST` command with the ID number.
First, create a document with an ID of `42`:
```
POST fruits/_doc/42
{
"name":"banana",
"color":"yellow"
}
```
Then use that ID to update the document:
```
POST fruits/_doc/42
{
"name":"banana",
"color":"yellow",
"classification":"berries"
}
```
This command updates the document with the new field `classification`. It produces the following response:
```
{
"_index" : "fruits",
"_type" : "_doc",
"_id" : "42",
"_version" : 2,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 1
}
```
**Note**
If you try to update a document that does not exist, OpenSearch Service creates the document.
## Performing bulk actions
You can use the `POST _bulk` API operation to perform multiple actions on one or more indexes in one request. Bulk action commands take the following format:
```
POST /_bulk
\n
\n
\n
\n
```
Each action requires two lines of JSON. First, you provide the action description or metadata. On the next line, you provide the data. Each part is separated by a newline (\$1n). An action description for an insert might look like this:
```
{ "create" : { "_index" : "veggies", "_type" : "_doc", "_id" : "7" } }
```
And the next line containing the data might look like this:
```
{ "name":"kale", "color":"green", "classification":"leafy-green" }
```
Taken together, the metadata and the data represent a single action in a bulk operation. You can perform many operations in one request, like this:
```
POST /_bulk
{ "create" : { "_index" : "veggies", "_id" : "35" } }
{ "name":"kale", "color":"green", "classification":"leafy-green" }
{ "create" : { "_index" : "veggies", "_id" : "36" } }
{ "name":"spinach", "color":"green", "classification":"leafy-green" }
{ "create" : { "_index" : "veggies", "_id" : "37" } }
{ "name":"arugula", "color":"green", "classification":"leafy-green" }
{ "create" : { "_index" : "veggies", "_id" : "38" } }
{ "name":"endive", "color":"green", "classification":"leafy-green" }
{ "create" : { "_index" : "veggies", "_id" : "39" } }
{ "name":"lettuce", "color":"green", "classification":"leafy-green" }
{ "delete" : { "_index" : "vegetables", "_id" : "1" } }
```
Notice that the last action is a `delete`. There’s no data following the `delete` action.
## Searching for documents
Now that data exists in your cluster, you can search for it. For example, you might want to search for all root vegetables, or get a count of all leafy greens, or find the number of errors logged per hour.
**Basic searches**
A basic search looks something like this:
```
GET veggies/_search?q=name:l*
```
The request produces a JSON response that contains the lettuce document.
**Advanced searches**
You can perform more advanced searches by providing the query options as JSON in the request body:
```
GET veggies/_search
{
"query": {
"term": {
"name": "lettuce"
}
}
}
```
This example also produces a JSON response with the lettuce document.
**Sorting**
You can perform more of this type of query using sorting. First, you need to recreate the index, because the automatic field mapping chose types that can’t be sorted by default. Send the following requests to delete and recreate the index:
```
DELETE /veggies
PUT /veggies
{
"mappings":{
"properties":{
"name":{
"type":"keyword"
},
"color":{
"type":"keyword"
},
"classification":{
"type":"keyword"
}
}
}
}
```
Then repopulate the index with data:
```
POST /_bulk
{ "create" : { "_index" : "veggies", "_id" : "7" } }
{ "name":"kale", "color":"green", "classification":"leafy-green" }
{ "create" : { "_index" : "veggies", "_id" : "8" } }
{ "name":"spinach", "color":"green", "classification":"leafy-green" }
{ "create" : { "_index" : "veggies", "_id" : "9" } }
{ "name":"arugula", "color":"green", "classification":"leafy-green" }
{ "create" : { "_index" : "veggies", "_id" : "10" } }
{ "name":"endive", "color":"green", "classification":"leafy-green" }
{ "create" : { "_index" : "veggies", "_id" : "11" } }
{ "name":"lettuce", "color":"green", "classification":"leafy-green" }
```
Now you can search with a sort. This request adds an ascending sort by the classification:
```
GET veggies/_search
{
"query" : {
"term": { "color": "green" }
},
"sort" : [
"classification"
]
}
```
## Related resources
For more information, see the following resources:
+ [Getting started with Amazon OpenSearch Service](gsg.md)
+ [Indexing data in Amazon OpenSearch Service](indexing.md)
+ [Searching data in Amazon OpenSearch Service](searching.md)
# Tutorial: Migrating to Amazon OpenSearch Service
Index snapshots are a popular way to migrate from a self-managed OpenSearch or legacy Elasticsearch cluster to Amazon OpenSearch Service. Broadly, the process consists of the following steps:
1. Take a snapshot of the existing cluster, and upload the snapshot to an Amazon S3 bucket.
1. Create an OpenSearch Service domain.
1. Give OpenSearch Service permissions to access the bucket, and ensure you have permissions to work with snapshots.
1. Restore the snapshot on the OpenSearch Service domain.
This walk through provides more detailed steps and alternate options, where applicable.
## Take and upload the snapshot
Although you can use the [repository-s3](https://docs.opensearch.org/latest/opensearch/snapshot-restore/#amazon-s3) plugin to take snapshots directly to S3, you have to install the plugin on every node, tweak `opensearch.yml` (or `elasticsearch.yml` if using an Elasticsearch cluster), restart each node, add your AWS credentials, and finally take the snapshot. The plugin is a great option for ongoing use or for migrating larger clusters.
For smaller clusters, a one-time approach is to take a [shared file system snapshot](https://docs.opensearch.org/latest/opensearch/snapshot-restore/#shared-file-system) and then use the AWS CLI to upload it to S3. If you already have a snapshot, skip to step 4.
****To take a snapshot and upload it to Amazon S3****
1. Add the `path.repo` setting to `opensearch.yml` (or `Elasticsearch.yml`) on all nodes, and then restart each node.
```
path.repo: ["/my/shared/directory/snapshots"]
```
1. Register a [snapshot repository](https://opensearch.org/docs/latest/opensearch/snapshot-restore/#register-repository), which is required before you take a snapshot. A repository is just a storage location: a shared file system, Amazon S3, Hadoop Distributed File System (HDFS), etc. In this case, we'll use a shared file system ("fs"):
```
PUT _snapshot/my-snapshot-repo-name
{
"type": "fs",
"settings": {
"location": "/my/shared/directory/snapshots"
}
}
```
1. Take the snapshot:
```
PUT _snapshot/my-snapshot-repo-name/my-snapshot-name
{
"indices": "migration-index1,migration-index2,other-indices-*",
"include_global_state": false
}
```
1. Install the [AWS CLI](https://aws.amazon.com/cli/), and run `aws configure` to add your credentials.
1. Navigate to the snapshot directory. Then run the following commands to create a new S3 bucket and upload the contents of the snapshot directory to that bucket:
```
aws s3 mb s3://amzn-s3-demo-bucket --region us-west-2
aws s3 sync . s3://amzn-s3-demo-bucket --sse AES256
```
Depending on the size of the snapshot and the speed of your internet connection, this operation can take a while.
## Create a domain
Although the console is the easiest way to create a domain, in this case, you already have the terminal open and the AWS CLI installed. Modify the following command to create a domain that fits your needs:
```
aws opensearch create-domain \
--domain-name migration-domain \
--engine-version OpenSearch_1.0 \
--cluster-config InstanceType=c5.large.search,InstanceCount=2 \
--ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=100 \
--node-to-node-encryption-options Enabled=true \
--encryption-at-rest-options Enabled=true \
--domain-endpoint-options EnforceHTTPS=true,TLSSecurityPolicy=Policy-Min-TLS-1-2-2019-07 \
--advanced-security-options Enabled=true,InternalUserDatabaseEnabled=true,MasterUserOptions='{MasterUserName=master-user,MasterUserPassword=master-user-password}' \
--access-policies '{"Version": "2012-10-17", "Statement":[{"Effect":"Allow","Principal": {"AWS": "arn:aws:iam::aws-region:user/UserName"},"Action":["es:ESHttp*"],"Resource":"arn:aws:es:aws-region:111122223333:domain/migration-domain/*"}]}' \
--region aws-region
```
As is, the command creates an internet-accessible domain with two data nodes, each with 100 GiB of storage. It also enables [fine-grained access control](fgac.md) with HTTP basic authentication and all encryption settings. Use the OpenSearch Service console if you need a more advanced security configuration, such as a VPC.
Before issuing the command, change the domain name, master user credentials, and account number. Specify the same AWS Region that you used for the S3 bucket and an OpenSearch/Elasticsearch version that is compatible with your snapshot.
**Important**
Snapshots are only forward-compatible, and only by one major version. For example, you can't restore a snapshot from an OpenSearch 1.*x* cluster on an Elasticsearch 7.*x* cluster, only an OpenSearch 1.*x* or 2.*x* cluster. Minor version matters, too. You can't restore a snapshot from a self-managed 5.3.3 cluster on a 5.3.2 OpenSearch Service domain. We recommend choosing the most recent version of OpenSearch or Elasticsearch that your snapshot supports. For a table of compatible versions, see [Using a snapshot to migrate data](snapshot-based-migration.md).
## Provide permissions to the S3 bucket
In the AWS Identity and Access Management (IAM) console, [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) with the following permissions and [trust relationship](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-managingrole_edit-trust-policy). When creating the role, choose **S3** as the **AWS Service**. Name the role `OpenSearchSnapshotRole` so it's easy to find.
**Permissions**
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::amzn-s3-demo-bucket"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::amzn-s3-demo-bucket/*"
]
}
]
}
```
------
**Trust relationship**
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": "es.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
------
Then give your personal IAM role permissions to assume `OpenSearchSnapshotRole`. Create the following policy and [attach it](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) to your identity:
**Permissions**
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::123456789012:role/OpenSearchSnapshotRole"
}
]
}
```
------
### Map the snapshot role in OpenSearch Dashboards (if using fine-grained access control)
If you enabled [fine-grained access control](fgac.md#fgac-mapping), even if you use HTTP basic authentication for all other purposes, you need to map the `manage_snapshots` role to your IAM role so you can work with snapshots.
**To give your identity permissions to work with snapshots**
1. Log in to Dashboards using the master user credentials you specified when you created the OpenSearch Service domain. You can find the Dashboards URL in the OpenSearch Service console. It takes the form of `https://domain-endpoint/_dashboards/`.
1. From the main menu choose **Security**, **Roles**, and select the **manage\$1snapshots** role.
1. Choose **Mapped users**, **Manage mapping**.
1. Add the domain ARN of your personal IAM role in the appropriate field. The ARN takes one of the following formats:
```
arn:aws:iam::123456789123:user/user-name
```
```
arn:aws:iam::123456789123:role/role-name
```
1. Select **Map** and confirm role shows up under **Mapped users**.
## Restore the snapshot
At this point, you have two ways to access your OpenSearch Service domain: HTTP basic authentication with your master user credentials or AWS authentication using your IAM credentials. Because snapshots use Amazon S3, which has no concept of the master user, you must use your IAM credentials to register the snapshot repository with your OpenSearch Service domain.
Most programming languages have libraries to assist with signing requests, but the simpler approach is to use a tool like [Postman](https://www.postman.com/downloads/) and put your IAM credentials into the **Authorization** section.
![\[Postman interface showing Authorization settings for AWS API request with Signature type.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/migration2.png)
**To restore the snapshot**
1. Regardless of how you choose to sign your requests, the first step is to register the repository:
```
PUT _snapshot/my-snapshot-repo-name
{
"type": "s3",
"settings": {
"bucket": "amzn-s3-demo-bucket",
"region": "us-west-2",
"role_arn": "arn:aws:iam::123456789012:role/OpenSearchSnapshotRole"
}
}
```
1. Then list the snapshots in the repository, and find the one you want to restore. At this point, you can continue using Postman or switch to a tool like [curl](https://curl.haxx.se/).
**Shorthand**
```
GET _snapshot/my-snapshot-repo-name/_all
```
**curl**
```
curl -XGET -u 'master-user:master-user-password' https://domain-endpoint/_snapshot/my-snapshot-repo-name/_all
```
1. Restore the snapshot.
**Shorthand**
```
POST _snapshot/my-snapshot-repo-name/my-snapshot-name/_restore
{
"indices": "migration-index1,migration-index2,other-indices-*",
"include_global_state": false
}
```
**curl**
```
curl -XPOST -u 'master-user:master-user-password' https://domain-endpoint/_snapshot/my-snapshot-repo-name/my-snapshot-name/_restore \
-H 'Content-Type: application/json' \
-d '{"indices":"migration-index1,migration-index2,other-indices-*","include_global_state":false}'
```
1. Finally, verify that your indexes restored as expected.
**Shorthand**
```
GET _cat/indices?v
```
**curl**
```
curl -XGET -u 'master-user:master-user-password' https://domain-endpoint/_cat/indices?v
```
At this point, the migration is complete. You might configure your clients to use the new OpenSearch Service endpoint, [resize the domain](sizing-domains.md) to suit your workload, check the shard count for your indexes, switch to an [IAM master user](fgac.md#fgac-concepts), or start building visualizations in OpenSearch Dashboards.
# Tutorial: Creating a search application with Amazon OpenSearch Service
A common way to create a search application with Amazon OpenSearch Service is to use web forms to send user queries to a server. Then you can authorize the server to call the OpenSearch APIs directly and have the server send requests to OpenSearch Service. However, if you want to write client-side code that doesn't rely on a server, you should compensate for the security and performance risks. Allowing unsigned, public access to the OpenSearch APIs is inadvisable. Users might access unsecured endpoints or impact cluster performance through overly broad queries (or too many queries).
This chapter presents a solution: use Amazon API Gateway to restrict users to a subset of the OpenSearch APIs and AWS Lambda to sign requests from API Gateway to OpenSearch Service.
![\[Search application flow diagram.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/search-application-diagram.png)
**Note**
Standard API Gateway and Lambda pricing applies, but within the limited usage of this tutorial, costs should be negligible.
## Prerequisites
A prerequisite for this tutorial is an OpenSearch Service domain. If you don't already have one, follow the steps in [Create an OpenSearch Service domain](gsgcreate-domain.md) to create one.
## Step 1: Index sample data
Download [sample-movies.zip](samples/sample-movies.zip), unzip it, and then use the [\$1bulk](https://opensearch.org/docs/latest/api-reference/document-apis/bulk/) API operation to add the 5,000 documents to the `movies` index:
```
POST https://search-my-domain.us-west-1.es.amazonaws.com/_bulk
{ "index": { "_index": "movies", "_id": "tt1979320" } }
{"directors":["Ron Howard"],"release_date":"2013-09-02T00:00:00Z","rating":8.3,"genres":["Action","Biography","Drama","Sport"],"image_url":"http://ia.media-imdb.com/images/M/MV5BMTQyMDE0MTY0OV5BMl5BanBnXkFtZTcwMjI2OTI0OQ@@._V1_SX400_.jpg","plot":"A re-creation of the merciless 1970s rivalry between Formula One rivals James Hunt and Niki Lauda.","title":"Rush","rank":2,"running_time_secs":7380,"actors":["Daniel Brühl","Chris Hemsworth","Olivia Wilde"],"year":2013,"id":"tt1979320","type":"add"}
{ "index": { "_index": "movies", "_id": "tt1951264" } }
{"directors":["Francis Lawrence"],"release_date":"2013-11-11T00:00:00Z","genres":["Action","Adventure","Sci-Fi","Thriller"],"image_url":"http://ia.media-imdb.com/images/M/MV5BMTAyMjQ3OTAxMzNeQTJeQWpwZ15BbWU4MDU0NzA1MzAx._V1_SX400_.jpg","plot":"Katniss Everdeen and Peeta Mellark become targets of the Capitol after their victory in the 74th Hunger Games sparks a rebellion in the Districts of Panem.","title":"The Hunger Games: Catching Fire","rank":4,"running_time_secs":8760,"actors":["Jennifer Lawrence","Josh Hutcherson","Liam Hemsworth"],"year":2013,"id":"tt1951264","type":"add"}
...
```
Note that the above is an example command with a small subset of the available data. To perform the `_bulk` operation, you need to copy and paste the entire contents of the `sample-movies` file. For further instructions, see [Option 2: Upload multiple documents](gsgupload-data.md#gsgmultiple-document).
You can also use the following curl command to achieve the same result:
```
curl -XPOST -u 'master-user:master-user-password' 'domain-endpoint/_bulk' --data-binary @bulk_movies.json -H 'Content-Type: application/json'
```
## Step 2: Create and deploy the Lambda function
Before you create your API in API Gateway, create the Lambda function that it passes requests to.
### Create the Lambda function
In this solution, API Gateway passes requests to a Lambda function, which queries OpenSearch Service and returns results. Because this sample function uses external libraries, you need to create a deployment package and upload it to Lambda.
**To create the deployment package**
1. Open a command prompt and create a `my-opensearch-function` project directory. For example, on macOS:
```
mkdir my-opensearch-function
```
1. Navigate to the `my-sourcecode-function` project directory.
```
cd my-opensearch-function
```
1. Copy the contents of the following sample Python code and save it in a new file named `opensearch-lambda.py`. Add your Region and host endpoint to the file.
```
import boto3
import json
import requests
from requests_aws4auth import AWS4Auth
region = '' # For example, us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
host = '' # The OpenSearch domain endpoint with https:// and without a trailing slash
index = 'movies'
url = host + '/' + index + '/_search'
# Lambda execution starts here
def lambda_handler(event, context):
# Put the user query into the query DSL for more accurate search results.
# Note that certain fields are boosted (^).
query = {
"size": 25,
"query": {
"multi_match": {
"query": event['queryStringParameters']['q'],
"fields": ["title^4", "plot^2", "actors", "directors"]
}
}
}
# Elasticsearch 6.x requires an explicit Content-Type header
headers = { "Content-Type": "application/json" }
# Make the signed HTTP request
r = requests.get(url, auth=awsauth, headers=headers, data=json.dumps(query))
# Create the response and add some extra content to support CORS
response = {
"statusCode": 200,
"headers": {
"Access-Control-Allow-Origin": '*'
},
"isBase64Encoded": False
}
# Add the search results to the response
response['body'] = r.text
return response
```
1. Install the external libraries to a new `package` directory.
```
pip3 install --target ./package boto3
pip3 install --target ./package requests
pip3 install --target ./package requests_aws4auth
```
1. Create a deployment package with the installed library at the root. The following command generates a `my-deployment-package.zip` file in your project directory.
```
cd package
zip -r ../my-deployment-package.zip .
```
1. Add the `opensearch-lambda.py` file to the root of the zip file.
```
cd ..
zip my-deployment-package.zip opensearch-lambda.py
```
For more information about creating Lambda functions and deployment packages, see [Deploy Python Lambda functions with .zip file archives](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html) in the *AWS Lambda Developer Guide* and [Create the Lambda deployment package](integrations-s3-lambda.md#integrations-s3-lambda-deployment-package) in this guide.
To create your function using the Lambda console
1. Navigate to the Lambda console at [https://console.aws.amazon.com/lambda/home](https://console.aws.amazon.com/lambda/home ). On the left navigation pane, choose **Functions**.
1. Select **Create function**.
1. Configure the following fields:
+ Function name: opensearch-function
+ Runtime: Python 3.9
+ Architecture: x86\$164
Keep all other default options and choose **Create function**.
1. In the **Code source** section of the function summary page, choose the **Upload from** dropdown and select **.zip file**. Locate the `my-deployment-package.zip` file that you created and choose **Save**.
1. The *handler* is the method in your function code that processes events. Under **Runtime settings**, choose **Edit** and change the handler name according to the name of the file in your deployment package where the Lambda function is located. Since your file is named `opensearch-lambda.py`, rename the handler to `opensearch-lambda.lambda_handler`. For more information, see [Lambda function handler in Python](https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html).
## Step 3: Create the API in API Gateway
Using API Gateway lets you create a more limited API and simplifies the process of interacting with the OpenSearch `_search` API. API Gateway lets you enable security features like Amazon Cognito authentication and request throttling. Perform the following steps to create and deploy an API:
### Create and configure the API
To create your API using the API Gateway console
1. Navigate to the API Gateway console at [https://console.aws.amazon.com/apigateway/home](https://console.aws.amazon.com/apigateway/home ). On the left navigation pane, choose **APIs**.
1. Locate **REST API** (not private) and choose **Build**.
1. On the following page, locate the **Create new API** section and make sure **New API** is selected.
1. Configure the following fields:
+ API name: **opensearch-api**
+ Description: **Public API for searching an Amazon OpenSearch Service domain**
+ Endpoint Type: **Regional**
1. Choose **Create API**.
1. Choose **Actions** and **Create Method**.
1. Select **GET** in the dropdown and click the checkmark to confirm.
1. Configure the following settings, then choose **Save**:
| Setting | Value |
| --- | --- |
| Integration type | Lambda function |
| Use Lambda proxy integration | Yes |
| Lambda region | us-west-1 |
| Lambda function | opensearch-lambda |
| Use default timeout | Yes |
### Configure the method request
Choose **Method Request** and configure the following settings:
| Setting | Value |
| --- | --- |
| Authorization | NONE |
| Request Validator | Validate query string parameters and headers |
| API Key Required | false |
Under **URL Query String Parameters**, choose **Add query string** and configure the following parameter:
| Setting | Value |
| --- | --- |
| Name | q |
| Required | Yes |
### Deploy the API and configure a stage
The API Gateway console lets you deploy an API by creating a deployment and associating it with a new or existing stage.
1. Choose **Actions** and **Deploy API**.
1. For **Deployment stage** choose **New Stage** and name the stage `opensearch-api-test`.
1. Choose **Deploy.**
1. Configure the following settings in the stage editor, then choose **Save Changes**:
| Setting | Value |
| --- | --- |
| Enable throttling | Yes |
| Rate | 1000 |
| Burst | 500 |
These settings configure an API that has only one method: a `GET` request to the endpoint root (`https://some-id.execute-api.us-west-1.amazonaws.com/search-es-api-test`). The request requires a single parameter (`q`), the query string to search for. When called, the method passes the request to Lambda, which runs the `opensearch-lambda` function. For more information, see [Creating an API in Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-create-api.html) and [Deploying a REST API in Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-deploy-api.html).
## Step 4: (Optional) Modify the domain access policy
Your OpenSearch Service domain must allow the Lambda function to make `GET` requests to the `movies` index. If your domain has an open access policy with fine-grained access control enabled, you can leave it as-is:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-1:123456789012:domain/domain-name/*"
}
]
}
```
------
Alternatively, you can choose to make your domain access policy more granular. For example, the following minimum policy provides `opensearch-lambda-role` (created through Lambda) read access to the `movies` index. To get the exact name of the role that Lambda automatically creates, go to the AWS Identity and Access Management (IAM) console, choose **Roles**, and search for "lambda".
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/service-role/opensearch-lambda-role-1abcdefg"
},
"Action": "es:ESHttpGet",
"Resource": "arn:aws:es:us-west-1:123456789012:domain/domain-name/movies/_search"
}
]
}
```
------
**Important**
If you have fine-grained access control enabled for the domain, you also need to [map the role to a user](fgac.md#fgac-mapping) in OpenSearch Dashboards, otherwise you'll see permissions errors.
### Configure Lambda execution role permissions
In addition to configuring the domain access policy, you must also ensure that the Lambda execution role has the necessary IAM permissions to access your OpenSearch Service domain. The Lambda function requires specific permissions depending on whether you're using a managed domain or OpenSearch Service Serverless collection.
**For managed OpenSearch Service domains:**
Attach the following IAM policy to your Lambda execution role to allow it to make requests to your OpenSearch Service domain:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"es:ESHttpGet",
"es:ESHttpPost"
],
"Resource": "arn:aws:es:us-west-1:123456789012:domain/domain-name/*"
}
]
}
```
------
**For OpenSearch Service Serverless collections:**
If you're using OpenSearch Service Serverless, attach the following IAM policy to your Lambda execution role:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "aoss:*",
"Resource": "arn:aws:aoss:us-west-1:123456789012:collection/collection-id"
}
]
}
```
------
To attach these policies to your Lambda execution role:
1. Navigate to the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. Choose **Roles** and search for your Lambda execution role (typically named `opensearch-lambda-role-xxxxxxxx`).
1. Choose **Add permissions**, then **Create inline policy**.
1. Choose the **JSON** tab and paste the appropriate policy from above, replacing the placeholder values with your actual resource ARNs.
1. Choose **Review policy**, provide a name like `OpenSearchAccess`, and choose **Create policy**.
**Note**
Without these IAM permissions, your Lambda function will receive "Access Denied" errors when attempting to query your OpenSearch Service domain, even if the domain access policy allows the requests.
For more information about access policies, see [Configuring access policies](createupdatedomains.md#createdomain-configure-access-policies).
## Map the Lambda role (if using fine-grained access control)
Fine-grained access control introduces an additional step before you can test the application. Even if you use HTTP basic authentication for all other purposes, you need to map the Lambda role to a user, otherwise you'll see permissions errors.
1. Navigate to the OpenSearch Dashboards URL for the domain.
1. From the main menu, choose **Security**, **Roles**, and select the link to `all_access`, the role you need to map the Lambda role to.
1. Choose **Mapped users**, **Manage mapping**.
1. Under **Backend roles**, add the Amazon Resource Name (ARN) of the Lambda role. The ARN should take the form of `arn:aws:iam::123456789123:role/service-role/opensearch-lambda-role-1abcdefg`.
1. Select **Map** and confirm the user or role shows up under **Mapped users**.
## Step 5: Test the web application
**To test the web application**
1. Download [sample-site.zip](samples/sample-site.zip), unzip it, and open `scripts/search.js` in your favorite text editor.
1. Update the `apigatewayendpoint` variable to point to your API Gateway endpoint and add a backslash to the end of the given path. You can quickly find the endpoint in API Gateway by choosing **Stages** and selecting the name of the API. The `apigatewayendpoint` variable should take the form of `https://some-id.execute-api.us-west-1.amazonaws.com/opensearch-api-test`/.
1. Open `index.html` and try running searches for *thor*, *house*, and a few other terms.
![\[A sample search for thor.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/search-ui.png)
### Troubleshoot CORS errors
Even though the Lambda function includes content in the response to support CORS, you still might see the following error:
```
Access to XMLHttpRequest at '' from origin 'null' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present in the requested resource.
```
If this happens, try the following:
1. [Enable CORS](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-cors-console.html) on the GET resource. Under **Advanced**, set **Access-Control-Allow-Credentials** to `'true'`.
1. Redeploy your API in API Gateway (**Actions**, **Deploy API**).
1. Delete and re-add your Lambda function trigger. Add re-add it, choose **Add trigger** and create the HTTP endpoint that invokes your function. The trigger must have the following configuration:
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/search-example.html)
## Next steps
This chapter is just a starting point to demonstrate a concept. You might consider the following modifications:
+ Add your own data to the OpenSearch Service domain.
+ Add methods to your API.
+ In the Lambda function, modify the search query or boost different fields.
+ Style the results differently or modify `search.js` to display different fields to the user.
# Tutorial: Visualizing customer support calls with OpenSearch Service and OpenSearch Dashboards
This chapter is a full walkthrough of the following situation: a business receives some number of customer support calls and wants to analyze them. What is the subject of each call? How many were positive? How many were negative? How can managers search or review the the transcripts of these calls?
A manual workflow might involve employees listening to recordings, noting the subject of each call, and deciding whether or not the customer interaction was positive.
Such a process would be extremely labor-intensive. Assuming an average time of 10 minutes per call, each employee could listen to only 48 calls per day. Barring human bias, the data they generate would be highly accurate, but the *amount* of data would be minimal: just the subject of the call and a boolean for whether or not the customer was satisfied. Anything more involved, such as a full transcript, would take a huge amount of time.
Using [Amazon S3](https://aws.amazon.com/s3/), [Amazon Transcribe](https://aws.amazon.com/transcribe/), [Amazon Comprehend](https://aws.amazon.com/comprehend/), and Amazon OpenSearch Service, you can automate a similar process with very little code and end up with much more data. For example, you can get a full transcript of the call, keywords from the transcript, and an overall "sentiment" of the call (positive, negative, neutral, or mixed). Then you can use OpenSearch and OpenSearch Dashboards to search and visualize the data.
While you can use this walkthrough as-is, the intent is to spark ideas about how to enrich your JSON documents before you index them in OpenSearch Service.
**Estimated Costs**
In general, performing the steps in this walkthrough should cost less than \$12. The walkthrough uses the following resources:
+ S3 bucket with less than 100 MB transferred and stored
To learn more, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/).
+ OpenSearch Service domain with one `t2.medium` instance and 10 GiB of EBS storage for several hours
To learn more, see [Amazon OpenSearch Service Pricing](https://aws.amazon.com/elasticsearch-service/pricing/).
+ Several calls to Amazon Transcribe
To learn more, see [Amazon Transcribe Pricing](https://aws.amazon.com/transcribe/pricing/).
+ Several natural language processing calls to Amazon Comprehend
To learn more, see [Amazon Comprehend Pricing](https://aws.amazon.com/comprehend/pricing/).
**Topics**
+ [Step 1: Configure prerequisites](#walkthrough-prereq)
+ [Step 2: Copy sample code](#walkthrough-script)
+ [(Optional) Step 3: Index sample data](#walkthrough-sample-data)
+ [Step 4: Analyze and visualize your data](#walkthrough-analysis)
+ [Step 5: Clean up resources and next steps](#walkthrough-next-steps)
## Step 1: Configure prerequisites
Before proceeding, you must have the following resources.
****
| Prerequisite | Description |
| --- | --- |
| Amazon S3 bucket | For more information, see [Creating a Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/CreatingABucket.html) in the Amazon Simple Storage Service User Guide. |
| OpenSearch Service domain | The destination for data. For more information, see [Creating OpenSearch Service domains](createupdatedomains.md#createdomains). |
If you don't already have these resources, you can create them using the following AWS CLI commands:
```
aws s3 mb s3://my-transcribe-test --region us-west-2
```
```
aws opensearch create-domain --domain-name my-transcribe-test --engine-version OpenSearch_1.0 --cluster-config InstanceType=t2.medium.search,InstanceCount=1 --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=10 --access-policies '{"Version": "2012-10-17", "Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::123456789012:root"},"Action":"es:*","Resource":"arn:aws:es:us-west-2:123456789012:domain/my-transcribe-test/*"}]}' --region us-west-2
```
**Note**
These commands use the `us-west-2` Region, but you can use any Region that Amazon Comprehend supports. To learn more, see the [AWS General Reference](https://docs.aws.amazon.com/general/latest/gr/rande.html#comprehend_region).
## Step 2: Copy sample code
1. Copy and paste the following Python 3 sample code into a new file named `call-center.py`:
```
import boto3
import datetime
import json
import requests
from requests_aws4auth import AWS4Auth
import time
import urllib.request
# Variables to update
audio_file_name = '' # For example, 000001.mp3
bucket_name = '' # For example, my-transcribe-test
domain = '' # For example, https://search-my-transcribe-test-12345.us-west-2.es.amazonaws.com
index = 'support-calls'
type = '_doc'
region = 'us-west-2'
# Upload audio file to S3.
s3_client = boto3.client('s3')
audio_file = open(audio_file_name, 'rb')
print('Uploading ' + audio_file_name + '...')
response = s3_client.put_object(
Body=audio_file,
Bucket=bucket_name,
Key=audio_file_name
)
# # Build the URL to the audio file on S3.
# # Only for the us-east-1 region.
# mp3_uri = 'https://' + bucket_name + '.s3.amazonaws.com/' + audio_file_name
# Get the necessary details and build the URL to the audio file on S3.
# For all other regions.
response = s3_client.get_bucket_location(
Bucket=bucket_name
)
bucket_region = response['LocationConstraint']
mp3_uri = 'https://' + bucket_name + '.s3-' + bucket_region + '.amazonaws.com/' + audio_file_name
# Start transcription job.
transcribe_client = boto3.client('transcribe')
print('Starting transcription job...')
response = transcribe_client.start_transcription_job(
TranscriptionJobName=audio_file_name,
LanguageCode='en-US',
MediaFormat='mp3',
Media={
'MediaFileUri': mp3_uri
},
Settings={
'ShowSpeakerLabels': True,
'MaxSpeakerLabels': 2 # assumes two people on a phone call
}
)
# Wait for the transcription job to finish.
print('Waiting for job to complete...')
while True:
response = transcribe_client.get_transcription_job(TranscriptionJobName=audio_file_name)
if response['TranscriptionJob']['TranscriptionJobStatus'] in ['COMPLETED', 'FAILED']:
break
else:
print('Still waiting...')
time.sleep(10)
transcript_uri = response['TranscriptionJob']['Transcript']['TranscriptFileUri']
# Open the JSON file, read it, and get the transcript.
response = urllib.request.urlopen(transcript_uri)
raw_json = response.read()
loaded_json = json.loads(raw_json)
transcript = loaded_json['results']['transcripts'][0]['transcript']
# Send transcript to Comprehend for key phrases and sentiment.
comprehend_client = boto3.client('comprehend')
# If necessary, trim the transcript.
# If the transcript is more than 5 KB, the Comprehend calls fail.
if len(transcript) > 5000:
trimmed_transcript = transcript[:5000]
else:
trimmed_transcript = transcript
print('Detecting key phrases...')
response = comprehend_client.detect_key_phrases(
Text=trimmed_transcript,
LanguageCode='en'
)
keywords = []
for keyword in response['KeyPhrases']:
keywords.append(keyword['Text'])
print('Detecting sentiment...')
response = comprehend_client.detect_sentiment(
Text=trimmed_transcript,
LanguageCode='en'
)
sentiment = response['Sentiment']
# Build the Amazon OpenSearch Service URL.
id = audio_file_name.strip('.mp3')
url = domain + '/' + index + '/' + type + '/' + id
# Create the JSON document.
json_document = {'transcript': transcript, 'keywords': keywords, 'sentiment': sentiment, 'timestamp': datetime.datetime.now().isoformat()}
# Provide all details necessary to sign the indexing request.
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, 'opensearchservice', session_token=credentials.token)
# Index the document.
print('Indexing document...')
response = requests.put(url, auth=awsauth, json=json_document, headers=headers)
print(response)
print(response.json())
```
1. Update the initial six variables.
1. Install the required packages using the following commands:
```
pip install boto3
pip install requests
pip install requests_aws4auth
```
1. Place your MP3 in the same directory as `call-center.py` and run the script. A sample output follows:
```
$ python call-center.py
Uploading 000001.mp3...
Starting transcription job...
Waiting for job to complete...
Still waiting...
Still waiting...
Still waiting...
Still waiting...
Still waiting...
Still waiting...
Still waiting...
Detecting key phrases...
Detecting sentiment...
Indexing document...
{u'_type': u'call', u'_seq_no': 0, u'_shards': {u'successful': 1, u'failed': 0, u'total': 2}, u'_index': u'support-calls4', u'_version': 1, u'_primary_term': 1, u'result': u'created', u'_id': u'000001'}
```
`call-center.py` performs a number of operations:
1. The script uploads an audio file (in this case, an MP3, but Amazon Transcribe supports several formats) to your S3 bucket.
1. It sends the audio file's URL to Amazon Transcribe and waits for the transcription job to finish.
The time to finish the transcription job depends on the length of the audio file. Assume minutes, not seconds.
**Tip**
To improve the quality of the transcription, you can configure a [custom vocabulary](https://docs.aws.amazon.com/transcribe/latest/dg/API_CreateVocabulary.html) for Amazon Transcribe.
1. After the transcription job finishes, the script extracts the transcript, trims it to 5,000 characters, and sends it to Amazon Comprehend for keyword and sentiment analysis.
1. Finally, the script adds the full transcript, keywords, sentiment, and current time stamp to a JSON document and indexes it in OpenSearch Service.
**Tip**
[LibriVox](https://librivox.org/) has public domain audiobooks that you can use for testing.
## (Optional) Step 3: Index sample data
If you don't have a bunch of call recordings handy—and who does?—you can [index](indexing.md) the sample documents in [sample-calls.zip](samples/sample-calls.zip), which are comparable to what `call-center.py` produces.
1. Create a file named `bulk-helper.py`:
```
import boto3
from opensearchpy import OpenSearch, RequestsHttpConnection
import json
from requests_aws4auth import AWS4Auth
host = '' # For example, my-test-domain.us-west-2.es.amazonaws.com
region = '' # For example, us-west-2
service = 'es'
bulk_file = open('sample-calls.bulk', 'r').read()
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
search = OpenSearch(
hosts = [{'host': host, 'port': 443}],
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
response = search.bulk(bulk_file)
print(json.dumps(response, indent=2, sort_keys=True))
```
1. Update the initial two variables for `host` and `region`.
1. Install the required package using the following command:
```
pip install opensearch-py
```
1. Download and unzip [sample-calls.zip](samples/sample-calls.zip).
1. Place `sample-calls.bulk` in the same directory as `bulk-helper.py` and run the helper. A sample output follows:
```
$ python bulk-helper.py
{
"errors": false,
"items": [
{
"index": {
"_id": "1",
"_index": "support-calls",
"_primary_term": 1,
"_seq_no": 42,
"_shards": {
"failed": 0,
"successful": 1,
"total": 2
},
"_type": "_doc",
"_version": 9,
"result": "updated",
"status": 200
}
},
...
],
"took": 27
}
```
## Step 4: Analyze and visualize your data
Now that you have some data in OpenSearch Service, you can visualize it using OpenSearch Dashboards.
1. Navigate to `https://search-domain.region.es.amazonaws.com/_dashboards`.
1. Before you can use OpenSearch Dashboards, you need an index pattern. Dashboards uses index patterns to narrow your analysis to one or more indices. To match the `support-calls` index that `call-center.py` created, go to **Stack Management**, **Index Patterns**, and define an index pattern of `support*`, and then choose **Next step**.
1. For **Time Filter field name**, choose **timestamp**.
1. Now you can start creating visualizations. Choose **Visualize**, and then add a new visualization.
1. Choose the pie chart and the `support*` index pattern.
1. The default visualization is basic, so choose **Split Slices** to create a more interesting visualization.
For **Aggregation**, choose **Terms**. For **Field**, choose **sentiment.keyword**. Then choose **Apply changes** and **Save**.
![\[Sample configuration for a Dashboards pie chart.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/sentiment-pie-chart.png)
1. Return to the **Visualize** page, and add another visualization. This time, choose the horizontal bar chart.
1. Choose **Split Series**.
For **Aggregation**, choose **Terms**. For **Field**, choose **keywords.keyword** and change **Size** to 20. Then choose **Apply Changes** and **Save**.
![\[Sample configuration for a Dashboards horizontal bar chart.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/keyword-bar-chart.png)
1. Return to the **Visualize** page and add one final visualization, a vertical bar chart.
1. Choose **Split Series**. For **Aggregation**, choose **Date Histogram**. For **Field**, choose **timestamp** and change **Interval** to **Daily**.
1. Choose **Metrics & Axes** and change **Mode** to **normal**.
1. Choose **Apply Changes** and **Save**.
![\[Sample configuration for a Dashboards vertical bar chart.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/timestamp-bar-chart-2.png)
1. Now that you have three visualizations, you can add them to a Dashboards visualization. Choose **Dashboard**, create a dashboard, and add your visualizations.
![\[Sample Dashboards visualization.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/dashboard-2.png)
## Step 5: Clean up resources and next steps
To avoid unnecessary charges, delete the S3 bucket and OpenSearch Service domain. To learn more, see [Delete a Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-or-empty-bucket.html#delete-bucket) in the *Amazon Simple Storage Service User Guide* and [Delete an OpenSearch Service domain](gsgdeleting.md) in this guide.
Transcripts require much less disk space than MP3 files. You might be able to shorten your MP3 retention window—for example, from three months of call recordings to one month—retain years of transcripts, and still save on storage costs.
You could also automate the transcription process using AWS Step Functions and Lambda, add additional metadata before indexing, or craft more complex visualizations to fit your exact use case.