Enriching your documents during ingestion
You can alter your content and document metadata fields or attributes during the document ingestion process. With Amazon Kendra's Custom Document Enrichment feature, you can create, modify, or delete document attributes and content when you ingest your documents into Amazon Kendra. This means you can manipulate and ingest your data as you need.
This feature gives you control over how your documents are treated and ingested into Amazon Kendra. For example, you can scrub personally identifiable information in the document metadata while ingesting your documents into Amazon Kendra.
Another way that you can use this feature is to invoke a Lambda function in AWS Lambda to run Optical Character Recognition (OCR) on images, translation on text, and other tasks for preparing the data for search or analysis. For example, you can invoke a function to run OCR on images. The function could interpret text from images and treat each image as a textual document. A company that receives mailed-in customer surveys and stores these surveys as images could ingest these images as textual documents into Amazon Kendra. The company can then search for valuable customer survey information in Amazon Kendra.
You can use basic operations to apply as a first parse of your data, and then use a Lambda function to apply more complex operations on your data. For example, you could use a basic operation to simply remove all values in the document metadata field 'Customer_ID', and then apply a Lambda function to extract text from images of the text in the documents.
How Custom Document Enrichment works
The overall process of Custom Document Enrichment is as follows:
-
You configure Custom Document Enrichment when you create or update your data source, or index your documents directly into Amazon Kendra.
-
Amazon Kendra applies inline configurations or basic logic to alter your data. For more information, see Basic operations to change metadata.
-
If you choose to configure advanced data manipulation, Amazon Kendra can apply this on your original, raw documents or on the structured, parsed documents. For more information, see Lambda functions: extract and change metadata or content.
-
Your altered documents are ingested into Amazon Kendra.
At any point in this process, if your configuration is not valid, Amazon Kendra throws an error.
When you call CreateDataSource, UpdateDataSource, or BatchPutDocument APIs, you
provide your Custom Document Enrichment configuration. If you call
BatchPutDocument, you must configure Custom Document Enrichment with
each request. If you use the console, you select your index and then select
Document enrichments to configure Custom Document
Enrichment.
If you use Document enrichments in the console, you can choose to only configure basic operations or only Lambda functions or both, like you can using the API. You can select Next in the console steps to choose not to configure basic operations and only Lambda functions, including whether to apply to the original (pre-extraction) or structured (post-extraction) data. You can only save your configurations by completing all the steps in the console. Your document configurations are not saved if you don't complete all the steps.
Basic operations to change metadata
You can manipulate your document fields and content using basic logic. This includes removing values in a field, modifying values in a field using a condition, or creating a field. For advanced manipulations that go beyond what you can manipulate using basic logic, invoke a Lambda function. For more information, see Lambda functions: extract and change metadata or content.
To apply basic logic, you specify the target field you want to manipulate using the DocumentAttributeTarget object. You provide the attribute key. For example, the key 'Department' is a field or attribute that holds all the department names associated with the documents. You can also specify a value to use in the target field if a certain condition is met. You set the condition using the DocumentAttributeCondition object. For example, if the 'Source_URI' field contains 'financial' in its URI value, then prefill the target field 'Department' with the target value 'Finance' for the document. You can also delete the values of the target document attribute.
To apply basic logic using the console, select your index and then select Document enrichments in the navigation menu. Go to Configure basic operations to apply basic manipulations to your document fields and content.
The following is an example of using basic logic to remove all customer identification numbers in the document field called 'Customer_ID'.
Example 1: Removing customer identification numbers associated with the documents
Data before basic manipulation applied.
| Document_ID | Body_Text | Customer_ID |
|---|---|---|
| 1 | Lorem Ipsum. | CID1234 |
| 2 | Lorem Ipsum. | CID1235 |
| 3 | Lorem Ipsum. | CID1236 |
Data after basic manipulation applied.
| Document_ID | Body_Text | Customer_ID |
|---|---|---|
| 1 | Lorem Ipsum. | |
| 2 | Lorem Ipsum. | |
| 3 | Lorem Ipsum. |
The following is an example of using basic logic to create a field called 'Department' and prefill this field with the department names based on information from the 'Source_URI' field. This uses the condition that if the 'Source_URI' field contains 'financial' in its URI value, then prefill the target field 'Department' with the target value 'Finance' for the document.
Example 2: Creating 'Department' field and prefilling it with department names associated with the documents using a condition.
Data before basic manipulation applied.
| Document_ID | Body_Text | Source_URI |
|---|---|---|
| 1 | Lorem Ipsum. | financial/1 |
| 2 | Lorem Ipsum. | financial/2 |
| 3 | Lorem Ipsum. | financial/3 |
Data after basic manipulation applied.
| Document_ID | Body_Text | Source_URI | Department |
|---|---|---|---|
| 1 | Lorem Ipsum. | financial/1 | Finance |
| 2 | Lorem Ipsum. | financial/2 | Finance |
| 3 | Lorem Ipsum. | financial/3 | Finance |
Note
Amazon Kendra can't create a target document field if it isn't already
created as an index field. After you create your index field, you can create a
document field using DocumentAttributeTarget. Amazon Kendra then
maps your newly created document metadata field to your index field.
The following code is an example of configuring basic data manipulation to remove customer identification numbers associated with the documents.
Lambda functions: extract and change metadata or content
You can manipulate your document fields and content using Lambda functions. This is useful if you want to go beyond basic logic and apply advanced data manipulations. For example, using Optical Character Recognition (OCR), which interprets text from images, and treats each image as a textual document. Or, retrieving the current date-time in a certain time zone and inserting the date-time where there's an empty value for a date field.
You can apply basic logic first and then use a Lambda function to further manipulate your data, or vice versa. You can also choose to only apply a Lambda function.
Amazon Kendra can invoke a Lambda function to apply advanced data manipulations during the ingestion process as part of your CustomDocumentEnrichmentConfiguration. You specify a role that includes permission to execute the Lambda function and access your Amazon S3 bucket to store the output of your data manipulations—see IAM access roles.
Amazon Kendra can apply a Lambda function on your original, raw documents or on the structured, parsed documents. You can configure a Lambda function that takes your original or raw data and applies your data manipulations using PreExtractionHookConfiguration. You can also configure a Lambda function that takes your structured documents and applies your data manipulations using PostExtractionHookConfiguration. Amazon Kendra extracts the document metadata and text to structure your documents. Your Lambda functions must follow the mandatory request and response structures. For more information, see Data contracts for Lambda functions.
To configure a Lambda function in the console, select your index and then select Document enrichments in the navigation menu. Go to Configure Lambda functions to configure a Lambda function.
You can configure only one Lambda function for
PreExtractionHookConfiguration and and only one Lambda function for
PostExtractionHookConfiguration. However, your Lambda function can
invoke other functions that it requires. You can configure both
PreExtractionHookConfiguration and
PostExtractionHookConfiguration or either one. Your Lambda function for
PreExtractionHookConfiguration must not exceed a run time of 5 minutes
and your Lambda function for PostExtractionHookConfiguration must not
exceed a run time of 1 minute. Configuring Custom Document Enrichment naturally takes
longer to ingest your documents into Amazon Kendra than if you were to not
configure this.
You can configure Amazon Kendra to invoke a Lambda function only if a condition is met. For example, you can specify a condition that if there are empty date-time values, then Amazon Kendra should invoke a function that inserts the current date-time.
The following is an example of using a Lambda function to run OCR to interpret text from images and store this text in a field called 'Document_Image_Text'.
Example 1: Extracting text from images to create textual documents
Data before advanced manipulation applied.
| Document_ID | Document_Image |
|---|---|
| 1 | image_1.png |
| 2 | image_2.png |
| 3 | image_3.png |
Data after advanced manipulation applied.
| Document_ID | Document_Image | Document_Image_Text |
|---|---|---|
| 1 | image_1.png | Mailed survey response |
| 2 | image_2.png | Mailed survey response |
| 3 | image_3.png | Mailed survey response |
The following is an example of using a Lambda function to insert the current date-time for empty date values. This uses the condition that if a date field value is 'null', then replace this with the current date-time.
Example 2: Replacing empty values in the Last_Updated field with the current date-time.
Data before advanced manipulation applied.
| Document_ID | Body_Text | Last_Updated |
|---|---|---|
| 1 | Lorem Ipsum. | January 1, 2020 |
| 2 | Lorem Ipsum. | |
| 3 | Lorem Ipsum. | July 1, 2020 |
Data after advanced manipulation applied.
| Document_ID | Body_Text | Last_Updated |
|---|---|---|
| 1 | Lorem Ipsum. | January 1, 2020 |
| 2 | Lorem Ipsum. | December 1, 2021 |
| 3 | Lorem Ipsum. | July 1, 2020 |
The following code is an example of configuring a Lambda function for advanced data manipulation on the raw, original data.
Data contracts for Lambda functions
Your Lambda functions for advanced data manipulation interact with Amazon Kendra data contracts. The contracts are the mandatory request and response structures of your Lambda functions. If your Lambda functions don't follow these structures, then Amazon Kendra throws an error.
Your Lambda function for PreExtractionHookConfiguration should expect the
following request structure:
{ "version": <str>, "dataBlobStringEncodedInBase64": <str>, //In the case of a data blob "s3Bucket": <str>, //In the case of an S3 bucket "s3ObjectKey": <str>, //In the case of an S3 bucket "metadata": <Metadata> }
The metadata structure, which includes the
CustomDocumentAttribute structure, is as follows:
{ "attributes": [<CustomDocumentAttribute<] } CustomDocumentAttribute { "name": <str>, "value": <CustomDocumentAttributeValue> } CustomDocumentAttributeValue { "stringValue": <str>, "integerValue": <int>, "longValue": <long>, "stringListValue": list<str>, "dateValue": <str> }
Your Lambda function for PreExtractionHookConfiguration must adhere to
the following response structure:
{ "version": <str>, "dataBlobStringEncodedInBase64": <str>, //In the case of a data blob "s3ObjectKey": <str>, //In the case of an S3 bucket "metadataUpdates": [<CustomDocumentAttribute>] }
Your Lambda function for PostExtractionHookConfiguration should expect
the following request structure:
{ "version": <str>, "s3Bucket": <str>, "s3ObjectKey": <str>, "metadata": <Metadata> }
Your Lambda function for PostExtractionHookConfiguration must adhere to
the following response structure:
PostExtractionHookConfiguration Lambda Response { "version": <str>, "s3ObjectKey": <str>, "metadataUpdates": [<CustomDocumentAttribute>] }
Your altered document is uploaded to your Amazon S3 bucket. The altered document must follow the format shown in Structured document format.
Structured document format
Amazon Kendra uploads your structured document to the given Amazon S3 bucket. The structured document follows this format:
Kendra document { "textContent": <TextContent> } TextContent { "documentBodyText": <str> }
Example of a Lambda function that adheres to data contracts
The following Python code is an example of a Lambda function that applies advanced
manipulation of the metadata fields _authors,
_document_title, and the body content on the raw or original
documents.
In the case of the body content residing in an Amazon S3 bucket
import json import boto3 s3 = boto3.client("s3") # Lambda function for advanced data manipulation def lambda_handler(event, context): # Get the value of "S3Bucket" key name or item from the given event input s3_bucket = event.get("s3Bucket") # Get the value of "S3ObjectKey" key name or item from the given event input s3_object_key = event.get("s3ObjectKey") content_object_before_CDE = s3.get_object(Bucket = s3_bucket, Key = s3_object_key) content_before_CDE = content_object_before_CDE["Body"].read().decode("utf-8"); content_after_CDE = "CDEInvolved " + content_before_CDE # Get the value of "metadata" key name or item from the given event input metadata = event.get("metadata") # Get the document "attributes" from the metadata document_attributes = metadata.get("attributes") s3.put_object(Bucket = s3_bucket, Key = "dummy_updated_kendra_document", Body=json.dumps(content_after_CDE)) return { "version": "v0", "s3ObjectKey": "dummy_updated_kendra_document", "metadataUpdates": [ {"name":"_document_title", "value":{"stringValue":"title_from_pre_extraction_lambda"}}, {"name":"_authors", "value":{"stringListValue":["author1", "author2"]}} ] }
In the case of the body content residing in a data blob
import json import boto3 import base64 # Lambda function for advanced data manipulation def lambda_handler(event, context): # Get the value of "dataBlobStringEncodedInBase64" key name or item from the given event input data_blob_string_encoded_in_base64 = event.get("dataBlobStringEncodedInBase64") # Decode the data blob string in UTF-8 data_blob_string = base64.b64decode(data_blob_string_encoded_in_base64).decode("utf-8") # Get the value of "metadata" key name or item from the given event input metadata = event.get("metadata") # Get the document "attributes" from the metadata document_attributes = metadata.get("attributes") new_data_blob = "This should be the modified data in the document by pre processing lambda ".encode("utf-8") return { "version": "v0", "dataBlobStringEncodedInBase64": base64.b64encode(new_data_blob).decode("utf-8"), "metadataUpdates": [ {"name":"_document_title", "value":{"stringValue":"title_from_pre_extraction_lambda"}}, {"name":"_authors", "value":{"stringListValue":["author1", "author2"]}} ] }
The following Python code is an example of a Lambda function that applies advanced
manipulation of the metadata fields _authors,
_document_title, and the body content on the structured or parsed
documents.
import json import boto3 import time s3 = boto3.client("s3") # Lambda function for advanced data manipulation def lambda_handler(event, context): # Get the value of "S3Bucket" key name or item from the given event input s3_bucket = event.get("s3Bucket") # Get the value of "S3ObjectKey" key name or item from the given event input s3_key = event.get("s3ObjectKey") # Get the value of "metadata" key name or item from the given event input metadata = event.get("metadata") # Get the document "attributes" from the metadata document_attributes = metadata.get("attributes") kendra_document_object = s3.get_object(Bucket = s3_bucket, Key = s3_key) kendra_document_string = kendra_document_object['Body'].read().decode('utf-8') kendra_document = json.loads(kendra_document_string) kendra_document["textContent"]["documentBodyText"] = "Changing document body to a short sentence." s3.put_object(Bucket = s3_bucket, Key = "dummy_updated_kendra_document", Body=json.dumps(kendra_document)) return { "version" : "v0", "s3ObjectKey": "dummy_updated_kendra_document", "metadataUpdates": [ {"name": "_document_title", "value":{"stringValue": "title_from_post_extraction_lambda"}}, {"name": "_authors", "value":{"stringListValue":["author1", "author2"]}} ] }
