`, an [Amazon S3 notification](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html) invokes the Lambda function `data-ingestion-processor`.
1. The Lambda function `data-ingestion-processor` is based on a Docker image stored in the Amazon Elastic Container Registry (Amazon ECR) repository `bedrock-rag-template`.
The function uses the [LangChain S3FileLoader](https://python.langchain.com/v0.1/docs/integrations/document_loaders/aws_s3_file/) to read the file as a [LangChain Document](https://api.python.langchain.com/en/v0.0.339/schema/langchain.schema.document.Document.html). Then, the [LangChain RecursiveCharacterTextSplitter](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/recursive_text_splitter/) chunks each document, given a `CHUNK_SIZE` and a `CHUNK_OVERLAP` that depends on the maximum token size of the Amazon Titan Text Embedding V2 embedding model. Next, the Lambda function invokes the embedding model on Amazon Bedrock to embed the chunks into numerical vector representations. Lastly, these vectors are stored in the Aurora PostgreSQL database. To access the database, the Lambda function first retrieves the username and password from AWS Secrets Manager.
1. On the Amazon SageMaker AI [notebook instance](https://docs.aws.amazon.com/sagemaker/latest/dg/nbi.html) `aws-sample-bedrock-rag-template`, the user can write a question prompt. The code invokes Claude 3 on Amazon Bedrock and adds the knowledge base information to the context of the prompt. As a result, Claude 3 provides responses using the information in the documents.
This pattern’s approach to networking and security is as follows:
+ The Lambda function `data-ingestion-processor` is in a private subnet within the virtual private cloud (VPC). The Lambda function isn’t allowed to send traffic to the public internet because of its security group. As a result, the traffic to Amazon S3 and Amazon Bedrock is routed through the VPC endpoints only. Consequently, the traffic doesn’t traverse the public internet, which reduces latency and adds an additional layer of security at the networking level.
+ All the resources and data are encrypted whenever applicable by using the AWS Key Management Service (AWS KMS) key with the alias `aws-sample/bedrock-rag-template`.
**Automation and scale**
This pattern uses Terraform to deploy the infrastructure from the code repository into an AWS account.
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments. In this pattern, Aurora PostgreSQL-Compatible uses the pgvector plugin as the vector database.
+ [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models (FMs) from leading AI startups and Amazon available for your use through a unified API.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command line shell.
+ [Amazon Elastic Container Registry (Amazon ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) is a managed container image registry service that’s secure, scalable, and reliable. In this pattern, Amazon ECR hosts the Docker image for the `data-ingestion-processor` Lambda function.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Key Management Service (AWS KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) helps you create and control cryptographic keys to help protect your data.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use. In this pattern, Lambda ingests data into the vector store.
+ [Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/?id=docs_gateway) is a managed machine learning (ML) service that helps you build and train ML models and then deploy them into a production-ready hosted environment.
+ [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS. The VPC includes subnets and routing tables to control traffic flow.
**Other tools**
+ [Docker](https://docs.docker.com/manuals/) is a set of platform as a service (PaaS) products that use virtualization at the operating-system level to deliver software in containers.
+ [HashiCorp Terraform](https://www.terraform.io/docs) is an infrastructure as code (IaC) tool that helps you use code to provision and manage cloud infrastructure and resources.
+ [Poetry](https://pypi.org/project/poetry/) is a tool for dependency management and packaging in Python.
+ [Python](https://www.python.org/) is a general-purpose computer programming language.
**Code repository**
The code for this pattern is available in the GitHub [terraform-rag-template-using-amazon-bedrock](https://github.com/aws-samples/terraform-rag-template-using-amazon-bedrock) repository.
## Best practices
+ Although this code sample can be deployed into any AWS Region, we recommend that you use US East (N. Virginia) – `us-east-1` or US West (N. California) – `us-west-1`. This recommendation is based on the availability of foundation and embedding models in Amazon Bedrock at the time of this pattern’s publication. For an up-to-date list of Amazon Bedrock foundation model support in AWS Regions, see [Model support by AWS Region](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html) in the Amazon Bedrock documentation. For information about deploying this code sample to other Regions, see [Additional information](#deploy-rag-use-case-on-aws-additional).
+ This pattern provides a proof-of-concept (PoC) or pilot demo only. If you want to take the code to production, be sure to use the following best practices:
+ Enable server access logging for Amazon S3.
+ Set up [monitoring and alerting](https://docs.aws.amazon.com/lambda/latest/dg/lambda-monitoring.html) for the Lambda function.
+ If your use case requires an API, consider adding Amazon API Gateway with a Lambda function that runs retrieval and question-answering tasks.
+ Follow the principle of least privilege and grant the minimum permissions required to perform a task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#grant-least-priv) and [Security best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPracticesAndUseCases.html) in the IAM documentation.
## Epics
### Deploy the solution in an AWS account
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repository. | To clone the GitHub repository provided with this pattern, use the following command:git clone https://github.com/aws-samples/terraform-rag-template-using-amazon-bedrock
| AWS DevOps |
| Configure the variables. | To configure the parameters for this pattern, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-rag-use-case-on-aws.html) | AWS DevOps |
| Deploy the solution. | To deploy the solution, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-rag-use-case-on-aws.html)The infrastructure deployment provisions an SageMaker AI instance inside the VPC and with the permissions to access the Aurora PostgreSQL database. | AWS DevOps |
### Test the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Run the demo. | After the previous infrastructure deployment has succeeded, use the following steps to run the demo in a Jupyter notebook:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-rag-use-case-on-aws.html)The Jupyter notebook guides you through the following process:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-rag-use-case-on-aws.html) | General AWS |
### Clean up infrastucture
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up the infrastructure. | To remove all the resources that you created when they are no longer required, use the following command:terraform destroy -var-file=commons.tfvars
| AWS DevOps |
## Related resources
**AWS resources**
+ [Building Lambda functions with Python](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html)
+ [Inference parameters for foundation models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html)
+ [Access to Amazon Bedrock foundation models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html)
+ [The role of vector databases in generative AI applications](https://aws.amazon.com/blogs/database/the-role-of-vector-datastores-in-generative-ai-applications/) (AWS Database Blog)
+ [Working with Amazon Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html)
**Other resources**
+ [pgvector documentation](https://github.com/pgvector/pgvector)
## Additional information
**Implementing a vector database**
This pattern uses Aurora PostgreSQL-Compatible to implement a vector database for RAG. As alternatives to Aurora PostgreSQL, AWS provides other capabilities and services for RAG, such as Amazon Bedrock Knowledge Bases and Amazon OpenSearch Service. You can choose the solution that best fits your specific requirements:
+ [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html) provides distributed search and analytics engines that you can use to store and query large volumes of data.
+ [Amazon Bedrock Knowledge Bases](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html) is designed for building and deploying knowledge bases as an additional abstraction to simplify the RAG ingestion and retrieval process. Amazon Bedrock Knowledge Bases can work with both Aurora PostgreSQL and Amazon OpenSearch Service.
**Deploying to other AWS Regions**
As described in [Architecture](#deploy-rag-use-case-on-aws-architecture), we recommend that you use either the Region US East (N. Virginia) – `us-east-1` or US West (N. California) – `us-west-1` to deploy this code sample. However, there are two possible ways to deploy this code sample to Regions other than `us-east-1` and `us-west-1`. You can configure the deployment Region in the `commons.tfvars` file. For cross-Region foundation model access, consider the following options:
+ **Traversing the public internet** – If the traffic can traverse the public internet, add internet gateways to the VPC. Then, adjust the security group assigned to the Lambda function `data-ingestion-processor` and the SageMaker AI notebook instance to allow outbound traffic to the public internet.
+ **Not traversing the public internet** – To deploy this sample to any Region other than `us-east-1` or `us-west-1`, do the following:
1. In either the `us-east-1` or `us-west-1` Region, create an additional VPC including a VPC endpoint for `bedrock-runtime`.
1. Create a peering connection by using [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html) or a [transit gateway](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-peering.html) to the application VPC.
1. When configuring the `bedrock-runtime` boto3 client in any Lambda function outside of `us-east-1` or `us-west-1`, pass the private DNS name of the VPC endpoint for `bedrock-runtime` in `us-east-1` or us-west-1 as the `endpoint_url` to the boto3 client.
# Deploy preprocessing logic into an ML model in a single endpoint using an inference pipeline in Amazon SageMaker
*Mohan Gowda Purushothama, Gabriel Rodriguez Garcia, and Mateusz Zaremba, Amazon Web Services*
## Summary
This pattern explains how to deploy multiple pipeline model objects in a single endpoint by using an [inference pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipelines.html) in Amazon SageMaker. The pipeline model object represents different machine learning (ML) workflow stages, such as preprocessing, model inference, and postprocessing. To illustrate the deployment of serially connected pipeline model objects, this pattern shows you how to deploy a preprocessing [Scikit-learn](https://docs.aws.amazon.com/sagemaker/latest/dg/sklearn.html) container and a regression model based on the [linear learner algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/linear-learner.html) built into SageMaker. The deployment is hosted behind a single endpoint in SageMaker.
**Note**
The deployment in this pattern uses the ml.m4.2xlarge instance type. We recommend using an instance type that aligns with your data size requirements and the complexity of your workflow. For more information, see [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing/). This pattern uses [prebuilt Docker images for Scikit-learn](https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-docker-containers-scikit-learn-spark.html), but you can use your own Docker containers and integrate them into your workflow.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ [Python 3.9](https://www.python.org/downloads/release/python-390/)
+ [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/) and [Boto3 library](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)
+ AWS Identity and Access Management (AWS IAM) [role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) with basic SageMaker [permissions](https://docs.aws.amazon.com/sagemaker/latest/dg/api-permissions-reference.html) and Amazon Simple Storage Service (Amazon S3) [permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html)
**Product versions**
+ [Amazon SageMaker Python SDK 2.49.2](https://sagemaker.readthedocs.io/en/v2.49.2/)
## Architecture
**Target technology stack**
+ Amazon Elastic Container Registry (Amazon ECR)
+ Amazon SageMaker
+ Amazon SageMaker Studio
+ Amazon Simple Storage Service (Amazon S3)
+ [Real-time inference](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html) endpoint for Amazon SageMaker
**Target architecture**
The following diagram shows the architecture for the deployment of an Amazon SageMaker pipeline model object.
![\[Architecture for deployment of SageMaker pipeline model object\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/1105d51b-752f-46d7-962c-acef1fb3399f/images/12f06715-b1c2-4de0-b277-99ce87308152.png)
The diagram shows the following workflow:
1. A SageMaker notebook deploys a pipeline model.
1. An S3 bucket stores the model artifacts.
1. Amazon ECR gets the source container images from the S3 bucket.
## Tools
**AWS tools**
+ [Amazon Elastic Container Registry (Amazon ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) is a managed container image registry service that’s secure, scalable, and reliable.
+ [Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html) is a managed ML service that helps you build and train ML models and then deploy them into a production-ready hosted environment.
+ [Amazon SageMaker Studio](https://docs.aws.amazon.com/sagemaker/latest/dg/studio.html) is a web-based, integrated development environment (IDE) for ML that lets you build, train, debug, deploy, and monitor your ML models.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Code**
The code for this pattern is available in the GitHub [Inference Pipeline with Scikit-learn and Linear Learner](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-python-sdk/scikit_learn_inference_pipeline/Inference%20Pipeline%20with%20Scikit-learn%20and%20Linear%20Learner.ipynb) repository.
## Epics
### Prepare the dataset
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the dataset for your regression task. | [Open a notebook](https://docs.aws.amazon.com/sagemaker/latest/dg/notebooks-create-open.html#notebooks-open) in Amazon SageMaker Studio.To import all necessary libraries and initialize your working environment, use the following example code in your notebook:import sagemaker
from sagemaker import get_execution_role
sagemaker_session = sagemaker.Session()
# Get a SageMaker-compatible role used by this Notebook Instance.
role = get_execution_role()
# S3 prefix
bucket = sagemaker_session.default_bucket()
prefix = "Scikit-LinearLearner-pipeline-abalone-example"
To download a sample dataset, add the following code to your notebook:! mkdir abalone_data
! aws s3 cp s3://sagemaker-sample-files/datasets/tabular/uci_abalone/abalone.csv ./abalone_data
** **The example in this pattern uses the [Abalone Data Set](https://archive.ics.uci.edu/ml/datasets/abalone) from the UCI Machine Learning Repository. | Data scientist |
| Upload the dataset to an S3 bucket. | In the notebook where you prepared your dataset earlier, add the following code to upload your sample data to an S3 bucket:WORK_DIRECTORY = "abalone_data"
train_input = sagemaker_session.upload_data(
path="{}/{}".format(WORK_DIRECTORY, "abalone.csv"),
bucket=bucket,
key_prefix="{}/{}".format(prefix, "train"),
)
| Data scientist |
### Create the data preprocessor using SKLearn
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the preprocessor.py script. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-preprocessing-logic-into-an-ml-model-in-a-single-endpoint-using-an-inference-pipeline-in-amazon-sagemaker.html) | Data scientist |
| Create the SKLearn preprocessor object. | To create an SKLearn preprocessor object (called SKLearn Estimator) that you can incorporate into your final inference pipeline, run the following code in your SageMaker notebook:from sagemaker.sklearn.estimator import SKLearn
FRAMEWORK_VERSION = "0.23-1"
script_path = "sklearn_abalone_featurizer.py"
sklearn_preprocessor = SKLearn(
entry_point=script_path,
role=role,
framework_version=FRAMEWORK_VERSION,
instance_type="ml.c4.xlarge",
sagemaker_session=sagemaker_session,
)
sklearn_preprocessor.fit({"train": train_input})
| Data scientist |
| Test the preprocessor's inference. | To confirm that your preprocessor is defined correctly, launch a [batch transform job](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) by entering the following code in your SageMaker notebook:# Define a SKLearn Transformer from the trained SKLearn Estimator
transformer = sklearn_preprocessor.transformer(
instance_count=1, instance_type="ml.m5.xlarge", assemble_with="Line", accept="text/csv"
)
# Preprocess training input
transformer.transform(train_input, content_type="text/csv")
print("Waiting for transform job: " + transformer.latest_transform_job.job_name)
transformer.wait()
preprocessed_train = transformer.output_path
| |
### Create a machine learning model
| Task | Description | Skills required |
| --- | --- | --- |
| Create a model object. | To create a model object based on the linear learner algorithm, enter the following code in your SageMaker notebook:import boto3
from sagemaker.image_uris import retrieve
ll_image = retrieve("linear-learner", boto3.Session().region_name)
s3_ll_output_key_prefix = "ll_training_output"
s3_ll_output_location = "s3://{}/{}/{}/{}".format(
bucket, prefix, s3_ll_output_key_prefix, "ll_model"
)
ll_estimator = sagemaker.estimator.Estimator(
ll_image,
role,
instance_count=1,
instance_type="ml.m4.2xlarge",
volume_size=20,
max_run=3600,
input_mode="File",
output_path=s3_ll_output_location,
sagemaker_session=sagemaker_session,
)
ll_estimator.set_hyperparameters(feature_dim=10, predictor_type="regressor", mini_batch_size=32)
ll_train_data = sagemaker.inputs.TrainingInput(
preprocessed_train,
distribution="FullyReplicated",
content_type="text/csv",
s3_data_type="S3Prefix",
)
data_channels = {"train": ll_train_data}
ll_estimator.fit(inputs=data_channels, logs=True)
The preceding code retrieves the relevant Amazon ECR Docker image from the public Amazon ECR Registry for the model, creates an estimator object, and then uses that object to train the regression model. | Data scientist |
### Deploy the final pipeline
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the pipeline model. | To create a pipeline model object (that is, a preprocessor object) and deploy the object, enter the following code in your SageMaker notebook:from sagemaker.model import Model
from sagemaker.pipeline import PipelineModel
import boto3
from time import gmtime, strftime
timestamp_prefix = strftime("%Y-%m-%d-%H-%M-%S", gmtime())
scikit_learn_inferencee_model = sklearn_preprocessor.create_model()
linear_learner_model = ll_estimator.create_model()
model_name = "inference-pipeline-" + timestamp_prefix
endpoint_name = "inference-pipeline-ep-" + timestamp_prefix
sm_model = PipelineModel(
name=model_name, role=role, models= [scikit_learn_inferencee_model, linear_learner_model]
)
sm_model.deploy(initial_instance_count=1, instance_type="ml.c4.xlarge", endpoint_name=endpoint_name)
You can adjust the instance type used in the model object to meet your needs. | Data scientist |
| Test the inference. | To confirm the endpoint is working correctly, run the following sample inference code in your SageMaker notebook:from sagemaker.predictor import Predictor
from sagemaker.serializers import CSVSerializer
payload = "M, 0.44, 0.365, 0.125, 0.516, 0.2155, 0.114, 0.155"
actual_rings = 10
predictor = Predictor(
endpoint_name=endpoint_name, sagemaker_session=sagemaker_session, serializer=CSVSerializer()
)
print(predictor.predict(payload))
| Data scientist |
## Related resources
+ [Preprocess input data before making predictions using Amazon SageMaker inference pipelines and Scikit-learn](https://aws.amazon.com/blogs/machine-learning/preprocess-input-data-before-making-predictions-using-amazon-sagemaker-inference-pipelines-and-scikit-learn/) (AWS Machine Learning Blog)
+ [End to end Machine Learning with Amazon SageMaker](https://github.com/aws-samples/amazon-sagemaker-build-train-deploy) (GitHub)
# Deploy real-time coding security validation by using an MCP server with Kiro and other coding assistants
*Ivan Girardi and Iker Reina Fuente, Amazon Web Services*
## Summary
This pattern describes how to implement a Model Context Protocol (MCP) server that integrates three industry-standard security scanning tools to provide comprehensive code security analysis. The server enables AI coding assistants (such as Kiro, Amazon Q Developer, and Cline) to automatically scan code snippets and infrastructure as code (IaC) configurations. With these scans, the coding assistants can help identify security vulnerabilities, misconfigurations, and compliance violations.
AI code generators trained on millions of code snippets create a security blind spot—how secure was that training data? This pattern provides real-time security validation during code generation, helping developers identify and understand potential security issues as they code. This approach helps developers address both direct vulnerabilities and inherited risks from dependencies. By bridging the gap between AI efficiency and security compliance, this pattern helps to enable safe adoption of AI-powered development tools.
This pattern helps organizations enhance their development security practices through AI-assisted coding tools, providing continuous security scanning capabilities across multiple programming languages and infrastructure definitions. The solution combines the capabilities of the following tools:
+ Checkov for scanning IaC files, including Terraform, AWS CloudFormation, and Kubernetes manifests
+ Semgrep for analyzing multiple programming languages such as Python, JavaScript, Java, and others
+ Bandit for specialized Python security scanning
Key features of this solution include the following:
+ Delta scanning of new code segments, reducing computational overhead
+ Isolated security tool environments, preventing cross-tool contamination
+ Seamless integration with AI coding assistants (Kiro, Amazon Q Developer, Cline, and others)
+ Real-time security feedback during code generation
+ Customizable scanning rules for organizational compliance
The pattern provides a unified interface for security scanning with standardized response formats, making it easier to integrate security checks into development workflows. The pattern uses Python and the MCP framework to deliver automated security feedback. This approach helps developers identify and address security issues early in the development process while learning about security best practices through detailed findings.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account with access to use Kiro or Amazon Q Developer, if you want to use either of those coding assistants
+ Python version 3.10 or later [installed](https://www.python.org/downloads/)
+ `uv` package manager [installed](https://docs.astral.sh/uv/getting-started/installation/)
+ Familiarity with security scanning tools and concepts
+ Basic understanding of IaC and application security
**Limitations**
+ Bandit scanning is limited to Python files only.
+ Real-time scanning might impact performance for large code bases.
+ Tool-specific limitations are based on supported file formats and languages.
+ Manual review is required to validate security findings.
+ Security scanning results require security expertise for proper interpretation.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
**Product versions**
+ Python version 3.10 or later
+ Checkov version 3.0.0 or later
+ Semgrep version 1.45.0 or later
+ Bandit version 1.7.5 or later
+ MCP[cli] version 1.11.0 or later
+ Pydantic version 1.10.0 or later
+ Loguru version 0.6.0 or later
## Architecture
The following diagram shows the architecture for this solution.
![\[AI assistants send code to MCP security scanner server to route to specialized scanners; scan results sent to developer.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/fa623544-4d54-48af-a4e4-9b6a0624e776/images/9c881f95-76d0-40f6-983e-d987fd2097b8.png)
The diagram shows the following workflow:
1. The developer uses AI assistants (for example, Kiro, Cline, Amazon Q Developer, or Roo Code) to generate or analyze code. The AI assistant sends the code for security scanning.
1. The MCP security scanner server processes the request by routing it to the appropriate specialized scanner: Checkov for IaC files, Semgrep for multiple programming languages analysis, or Bandit for Python-specific security scanning.
1. Scanner results with security findings, severity levels, detailed descriptions, and suggested fixes are sent back to the developer through the AI assistant.
1. A continuous feedback loop is established where the developer receives real-time security validation, enabling automated fixes through AI assistants and promoting security best practices during development.
The architecture mitigates the following common security risks:
+ Command injection
+ Prompt injection
+ Path traversal
+ Dependency attacks
+ Resource exhaustion
The architecture mitigates these common security risks by implementing the following best practices:
+ All user and AI model inputs are written to temporary files.
+ No direct inputs are provided to command line interface (CLI) commands.
+ File system access is restricted to temporary directories and files only.
+ Temporary files are automatically cleaned up.
+ Scanning responses are sanitized.
+ Process isolation that restricts process capabilities is enforced.
+ All scanning activities are logged.
**Automation and scale**
The pattern supports automation through the following capabilities:
+ Integration with AI coding assistants for automatic code scanning
+ Standardized API responses for automated processing
+ Configuration through MCP configuration files
+ Support for batch processing of multiple files
+ Scalable scanning across multiple programming languages and IaC formats
The scanning process can be automated through the provided API endpoints:
+ `scan_with_checkov` for IaC scanning
+ `scan_with_semgrep` for multi-language code scanning
+ `scan_with_bandit` for Python-specific scanning
+ `get_supported_formats` for format validation
When extending the scanning tools, follow the design principles and best practices described earlier in this section. Also see [Best practices](#deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants-best-practices).
## Tools
**AWS services**
+ [Kiro](https://aws.amazon.com/documentation-overview/kiro/) is an agentic coding service that works alongside developers to turn prompts into detailed specs, then into working code, docs, and tests. Kiro agents help developers solve challenging problems and automate tasks like generating documentation and unit tests.
+ [Amazon Q Developer](https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/what-is.html) is a generative AI-powered conversational assistant that can help you understand, build, extend, and operate AWS applications.
**Other tools**
+ [Bandit](https://bandit.readthedocs.io/en/latest/) is a specialized Python security scanner tool. It detects common Python security issues like insecure functions, hardcoded secrets, and injection vulnerabilities. Bandit provides detailed confidence and severity ratings.
+ [Checkov](https://github.com/bridgecrewio/checkov) is a static code-analysis tool that checks IaC for security and compliance misconfigurations. In addition, Checkov detects compliance violations and security best practices.
+ [Cline](https://cline.bot/) is an AI-powered coding assistant that runs in VS Code.
+ [Loguru](https://loguru.readthedocs.io/en/stable/) is a data validation library for Python.
+ [Model Context Protocol (MCP)](https://modelcontextprotocol.io/docs/getting-started/intro) is an open source framework for building AI-assisted development tools.
+ [Pydantic](https://docs.pydantic.dev/latest/) is a data validation library for Python.
+ [Semgrep](https://semgrep.dev/docs/introduction) analyzes source code for security vulnerabilities and bugs. It supports multiple programming languages. Semgrep uses security-focused rulesets for comprehensive analysis. It provides detailed confidence and severity ratings.
**Code repository**
The code for this pattern is available in the GitHub [MCP Security Scanner: Real-Time Protection for AI Code Assistants](https://github.com/aws-samples/sample-mcp-security-scanner) repository. The repository includes the MCP server implementation, details on the MCP configuration for Kiro, Amazon Q Developer, Cline and others, configuration examples, and testing utilities.
The repository structure includes:
+ `security_scanner_mcp_server/` - Main server implementation
+ `docs/` - Documentation and demo materials
+ `tests/` - Test files
+ `mcp-config-example.json` - Example MCP configuration
+ `requirements.txt` - Project dependencies
## Best practices
**Security scanning implementation**
+ Review security findings to validate and prioritize issues.
+ Keep scanning tools (Checkov, Semgrep, and Bandit) updated to latest versions.
+ Use this pattern’s MCP security tool in conjunction with other security measures and tools.
+ Update security rule sets and policies regularly.
**Configuration management**
+ Use the MCP configuration files in the official version control source.
+ Document custom rules and configurations.
**Integration**
+ Integrate security scanning early in the development cycle.
+ Set up automated scanning in pre-commit hooks or continuous integration and continuous deployment (CI/CD) pipelines.
+ Configure appropriate severity thresholds for your environment.
+ Establish clear procedures for handling security findings.
**Operational considerations**
+ Monitor scanning performance and resource usage.
+ Implement proper error handling and logging.
+ Maintain documentation of custom configurations.
+ Establish a process for reviewing and updating security rules.
Also, keep in mind the following best practices:
+ Always validate security findings in your specific context.
+ Keep security tools and dependencies up to date.
+ Use multiple security tools for comprehensive coverage.
+ Follow security best practices in your development process.
## Epics
### (Kiro users) Set up the MCP security scanner server
| Task | Description | Skills required |
| --- | --- | --- |
| Configure MCP settings. | You can edit the configuration files in Kiro either by (Option 1) manually locating the configuration files or (Option 2) by using the Kiro IDE.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html){
"mcpServers": {
"security-scanner": {
"command": "uvx",
"args": [
"--from",
"git+https://github.com/aws-samples/sample-mcp-security-scanner.git@main",
"security_scanner_mcp_server"
],
"env": {
"FASTMCP_LOG_LEVEL": "ERROR"
},
"disabled": false,
"autoApprove": []
}
}
} | App developer |
### (Amazon Q Developer users) Set up the MCP security scanner server
| Task | Description | Skills required |
| --- | --- | --- |
| Configure MCP settings. | To configure the MCP settings manually, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html){
"mcpServers": {
"security-scanner": {
"command": "uvx",
"args": [
"--from",
"git+https://github.com/aws-samples/sample-mcp-security-scanner.git@main",
"security_scanner_mcp_server"
],
"env": {
"FASTMCP_LOG_LEVEL": "ERROR"
}
}
}
} | App developer |
### (Cline users) Set up the MCP security scanner server
| Task | Description | Skills required |
| --- | --- | --- |
| Configure MCP settings. | To configure the MCP settings manually, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html){
"mcpServers": {
"security-scanner": {
"command": "uvx",
"args": [
"--from",
"git+https://github.com/aws-samples/sample-mcp-security-scanner.git@main",
"security_scanner_mcp_server"
],
"env": {
"FASTMCP_LOG_LEVEL": "ERROR"
},
"disabled": false,
"autoApprove": []
}
}
} | App developer |
### Example of code analysis using Python and Bandit
| Task | Description | Skills required |
| --- | --- | --- |
| Perform code analysis. | To perform code analysis by using Python and Bandit, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html) | App developer |
### Example of code analysis using Terraform and Checkov
| Task | Description | Skills required |
| --- | --- | --- |
| Perform code analysis. | To perform code analysis by using Terraform and Checkov, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html) | App developer |
### Example of advanced scanning capabilities
| Task | Description | Skills required |
| --- | --- | --- |
| Perform targeted scanning. | Following are examples of requests that you can use to perform a targeted scan:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html) | App developer |
| Use security scanning with code generation. | To resolve security findings by using code generation loops, use the following steps (this example uses Kiro as the coding assistant):[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html) | App developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Environment setup issues | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html) |
| Scanner issues | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html) |
| Integration problems | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html) |
| Additional support | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-real-time-coding-security-validation-by-using-an-mcp-server-with-kiro-and-other-coding-assistants.html) |
## Related resources
**AWS documentation**
+ [Infrastructure as code](https://docs.aws.amazon.com/whitepapers/latest/introduction-devops-aws/infrastructure-as-code.html) (AWS Whitepaper *Introduction to DevOps on AWS*)
**Other AWS resources**
+ [Best Practices for Security, Identity, & Compliance](https://aws.amazon.com/architecture/security-identity-compliance/)
**Other resources**
+ [Bandit documentation](https://aws.amazon.com/architecture/security-identity-compliance/)
+ [Checkov documentation](https://aws.amazon.com/architecture/security-identity-compliance/)
+ [Model Context Protocol (MCP) documentation](https://aws.amazon.com/architecture/security-identity-compliance/)
+ [OWASP Secure Coding Practices](https://owasp.org/www-project-secure-coding-practices-quick-reference-guide/) (OWASP Foundation website)
+ [Semgrep documentation](https://aws.amazon.com/architecture/security-identity-compliance/)
## Additional information
**Example MCP configuration with auto approved enabled**
Without `autoApprove` configured, the user must grant approval to send the code to the MCP security server for scanning. When `autoApprove` is configured, the code assistant is allowed to invoke the tools without user approval. These tools run locally on the machine, no data is sent out, and only a code scan is performed.
The following configuration enables automatic execution of all security scanning functions:
```
{
"mcpServers": {
"security-scanner": {
"command": "uvx",
"args": [
"--from",
"git+https://github.com/aws-samples/sample-mcp-security-scanner.git@main",
"security_scanner_mcp_server"
],
"env": {
"FASTMCP_LOG_LEVEL": "ERROR"
},
"disabled": false,
"autoApprove": [
"scan_with_checkov",
"scan_with_semgrep",
"scan_with_bandit",
"get_supported_formats"
]
}
}
}
```
To enable debug logging, set `"FASTMCP_LOG_LEVEL"` to `"DEBUG"`.
**File formats supported by security scanning tools**
Each security scanning tool in this solution supports the following file formats:
*Checkov (IaC)*
+ Terraform – .tf, .tfvars, .tfstate
+ CloudFormation – .yaml, .yml, .json, .template
+ Kubernetes – .yaml, .yml
+ Dockerfile – Dockerfile
+ ARM – .json (Azure Resource Manager)
+ Bicep – .bicep
+ Serverless – .yml, .yaml
+ Helm – .yaml, .yml, .tpl
+ GitHub Actions – .yml, .yaml
+ GitLab\$1ci – .yml, .yaml
+ Ansible – .yml, .yaml
*Semgrep (Source code)*
+ Python – .py
+ JavaScript – .js
+ TypeScript – .ts
+ Java – .java
+ Go – .go
+ C – .c
+ C\$1\$1 – .cpp
+ C\$1 – .cs
+ Ruby – .rb
+ PHP – .php
+ Scala – .scala
+ Kotlin – .kt
+ Rust – .rs
*Bandit (Python only)*
+ Python – .py
**Demos**
For code scanning, try the following sample prompts with your AI assistant:
+ "Scan the current script and tell me the results."
+ "Scan lines 20–60 and tell me the results."
+ "Scan this Amazon DynamoDB table resource and tell me the result."
For more information, see this [code scanning demo](https://github.com/aws-samples/sample-mcp-security-scanner/blob/main/docs/demo_code_scan.gif) in this pattern’s GitHub repository.
To generate secure code, try the following sample prompts:
+ "Generate a Terraform configuration to create a DynamoDB table with encryption enabled and scan it for security issues."
+ "Create a Python Lambda function that writes to DynamoDB and scan it for vulnerabilities."
+ "Generate a CloudFormation template for an S3 bucket with proper security settings and verify it passes security checks."
+ "Write a Python script to query DynamoDB with pagination and scan for security best practices."
+ "Create a Kubernetes deployment manifest for a microservice with security hardening and validate it."
For more information, see this [code generation with security scanning](https://github.com/aws-samples/sample-mcp-security-scanner/blob/main/docs/demo_code_generation.gif) demo in this pattern’s GitHub repository.
# Develop advanced generative AI chat-based assistants by using RAG and ReAct prompting
*Praveen Kumar Jeyarajan, Shuai Cao, Noah Hamilton, Kiowa Jackson, Jundong Qiao, and Kara Yang, Amazon Web Services*
## Summary
A typical corporation has 70 percent of its data trapped in siloed systems. You can use generative AI-powered chat-based assistants to unlock insights and relationships between these data silos through natural language interactions. To get the most out of generative AI, the outputs must be trustworthy, accurate, and inclusive of the available corporate data. Successful chat-based assistants depend on the following:
+ Generative AI models (such as Anthropic Claude 2)
+ Data source vectorization
+ Advanced reasoning techniques, such as the [ReAct framework](https://www.promptingguide.ai/techniques/react), for prompting the model
This pattern provides data-retrieval approaches from data sources such as Amazon Simple Storage Service (Amazon S3) buckets, AWS Glue, and Amazon Relational Database Service (Amazon RDS). Value is gained from that data by interleaving [Retrieval Augmented Generation (RAG)](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html) with chain-of-thought methods. The results support complex chat-based assistant conversations that draw on the entirety of your corporation's stored data.
This pattern uses Amazon SageMaker manuals and pricing data tables as an example to explore the capabilities of a generative AI chat-based assistant. You will build a chat-based assistant that helps customers evaluate the SageMaker service by answering questions about pricing and the service's capabilities. The solution uses a Streamlit library for building the frontend application and the LangChain framework for developing the application backend powered by a large language model (LLM).
Inquiries to the chat-based assistant are met with an initial intent classification for routing to one of three possible workflows. The most sophisticated workflow combines general advisory guidance with complex pricing analysis. You can adapt the pattern to suit enterprise, corporate, and industrial use cases.
## Prerequisites and limitations
**Prerequisites**
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed and configured
+ [AWS Cloud Development Kit (AWS CDK) Toolkit 2.114.1 or later](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) installed and configured
+ Basic familiarity with Python and AWS CDK
+ [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed
+ [Docker](https://docs.docker.com/get-docker/) installed
+ [Python 3.11 or later](https://www.python.org/downloads/) installed and configured (for more information, see the [Tools](#develop-advanced-generative-ai-chat-based-assistants-by-using-rag-and-react-prompting-tools) section)
+ An [active AWS account](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-creating.html) bootstrapped by using [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html)
+ Amazon Titan and Anthropic Claude [model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html#add-model-access) enabled in the Amazon Bedrock service
+ [AWS security credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html), including `AWS_ACCESS_KEY_ID`, correctly configured in your terminal environment
**Limitations**
+ LangChain doesn't support every LLM for streaming. The Anthropic Claude models are supported, but models from AI21 Labs are not.
+ This solution is deployed to a single AWS account.
+ This solution can be deployed only in AWS Regions where Amazon Bedrock and Amazon Kendra are available. For information about availability, see the documentation for [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html#bedrock-regions) and [Amazon Kendra](https://docs.aws.amazon.com/general/latest/gr/kendra.html).
**Product versions**
+ Python version 3.11 or later
+ Streamlit version 1.30.0 or later
+ Streamlit-chat version 0.1.1 or later
+ LangChain version 0.1.12 or later
+ AWS CDK version 2.132.1 or later
## Architecture
**Target technology stack**
+ Amazon Athena
+ Amazon Bedrock
+ Amazon Elastic Container Service (Amazon ECS)
+ AWS Glue
+ AWS Lambda
+ Amazon S3
+ Amazon Kendra
+ Elastic Load Balancing
**Target architecture**
The AWS CDK code will deploy all the resources that are required to set up the chat-based assistant application in an AWS account. The chat-based assistant application shown in the following diagram is designed to answer SageMaker related queries from users. Users connect through an Application Load Balancer to a VPC that contains an Amazon ECS cluster hosting the Streamlit application. An orchestration Lambda function connects to the application. S3 bucket data sources provide data to the Lambda function through Amazon Kendra and AWS Glue. The Lambda function connects to Amazon Bedrock for answering queries (questions) from chat-based assistant users.
![\[Architecture diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b4df6405-76ab-4493-a722-15ceca067254/images/4e5856cf-9489-41f8-a411-e3b8d8a50748.png)
1. The orchestration Lambda function sends the LLM prompt request to the Amazon Bedrock model (Claude 2).
1. Amazon Bedrock sends the LLM response back to the orchestration Lambda function.
**Logic flow within the orchestration Lambda function**
When users ask a question through the Streamlit application, it invokes the orchestration Lambda function directly. The following diagram shows the logic flow when the Lambda function is invoked.
![\[Architecture diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b4df6405-76ab-4493-a722-15ceca067254/images/70ae4736-06a6-4d3a-903a-edc5c10d78a0.png)
+ Step 1 – The input `query` (question) is classified into one of the three intents:
+ General SageMaker guidance questions
+ General SageMaker pricing (training/inference) questions
+ Complex questions related to SageMaker and pricing
+ Step 2 – The input `query` initiates one of the three services:
+ `RAG Retrieval service`, which retrieves relevant context from the [Amazon Kendra](https://aws.amazon.com/kendra/) vector database and calls the LLM through [Amazon Bedrock](https://aws.amazon.com/bedrock/) to summarize the retrieved context as the response.
+ `Database Query service`, which uses- the LLM, database metadata, and sample rows from relevant tables to convert the input `query` into a SQL query. Database Query service runs the SQL query against the SageMaker pricing database through [Amazon Athena](https://aws.amazon.com/athena/) and summarizes the query results as the response.
+ `In-context ReACT Agent service`, which breaks down the input `query` into multiple steps before providing a response. The agent uses `RAG Retrieval service` and `Database Query service` as tools to retrieve relevant information during the reasoning process. After the reasoning and actions processes are complete, the agent generates the final answer as the response.
+ Step 3 – The response from the orchestration Lambda function is sent to the Streamlit application as output.
## Tools
**AWS services**
+ [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) is an interactive query service that helps you analyze data directly in Amazon Simple Storage Service (Amazon S3) by using standard SQL.
+ [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models (FMs) from leading AI startups and Amazon available for your use through a unified API.
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/latest/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
+ [Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) is a fast and scalable container management service that helps you run, stop, and manage containers on a cluster.
+ [AWS Glue](https://docs.aws.amazon.com/glue/) is a fully managed extract, transform, and load (ETL) service. It helps you reliably categorize, clean, enrich, and move data between data stores and data streams. This pattern uses an AWS Glue crawler and an AWS Glue Data Catalog table.
+ [Amazon Kendra](https://docs.aws.amazon.com/kendra/latest/dg/what-is-kendra.html) is an intelligent search service that uses natural language processing and advanced machine learning algorithms to return specific answers to search questions from your data.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across Amazon Elastic Compute Cloud (Amazon EC2) instances, containers, and IP addresses in one or more Availability Zones.
**Code repository**
The code for this pattern is available in the GitHub [genai-bedrock-chatbot](https://github.com/awslabs/genai-bedrock-chatbot) repository.
The code repository contains the following files and folders:
+ `assets` folder – The static assets the architecture diagram and the public dataset
+ `code/lambda-container` folder – The Python code that is run in the Lambda function
+ `code/streamlit-app` folder – The Python code that is run as the container image in Amazon ECS
+ `tests` folder – The Python files that are run to unit test the AWS CDK constructs
+ `code/code_stack.py` – The AWS CDK construct Python files used to create AWS resources
+ `app.py` – The AWS CDK stack Python files used to deploy AWS resources in the target AWS account
+ `requirements.txt` – The list of all Python dependencies that must be installed for AWS CDK
+ `requirements-dev.txt` – The list of all Python dependencies that must be installed for AWS CDK to run the unit-test suite
+ `cdk.json` – The input file to provide values required to spin up resources
|
|
| Note: The AWS CDK code uses [L3 (layer 3) constructs](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) and [AWS Identity and Access Management (IAM) policies managed by AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) for deploying the solution. |
| --- |
## Best practices
+ The code example provided here is for a proof-of-concept (PoC) or pilot demo only. If you want to take the code to Production, be sure to use the following best practices:
+ [Amazon S3 access logging is enabled](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html).
+ [VPC Flow Logs is enabled](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html).
+ The [Amazon Kendra Enterprise Edition index](https://docs.aws.amazon.com/whitepapers/latest/how-aws-pricing-works/amazon-kendra.html) is enabled.
+ Set up monitoring and alerting for the Lambda function. For more information, see [Monitoring and troubleshooting Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-monitoring.html). For general best practices when working with Lambda functions, see the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html).
## Epics
### Set up AWS credentials on your local machine
| Task | Description | Skills required |
| --- | --- | --- |
| Export variables for the account and AWS Region where the stack will be deployed. | To provide AWS credentials for AWS CDK by using environment variables, run the following commands.export CDK_DEFAULT_ACCOUNT=<12 Digit AWS Account Number>
export CDK_DEFAULT_REGION=
| DevOps engineer, AWS DevOps |
| Set up the AWS CLI profile. | To set up the AWS CLI profile for the account, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/keys-profiles-credentials.html). | DevOps engineer, AWS DevOps |
### Set up your environment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repo on your local machine. | To clone the repository, run the following command in your terminal.git clone https://github.com/awslabs/genai-bedrock-chatbot.git
| DevOps engineer, AWS DevOps |
| Set up the Python virtual environment and install required dependencies. | To set up the Python virtual environment, run the following commands.cd genai-bedrock-chatbot
python3 -m venv .venv
source .venv/bin/activate
To set up the required dependencies, run the following command.pip3 install -r requirements.txt
| DevOps engineer, AWS DevOps |
| Set up the AWS CDK environment and synthesize the AWS CDK code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/develop-advanced-generative-ai-chat-based-assistants-by-using-rag-and-react-prompting.html) | DevOps engineer, AWS DevOps |
### Configure and deploy the chat-based assistant application
| Task | Description | Skills required |
| --- | --- | --- |
| Provision Claude model access. | To enable Anthropic Claude model access for your AWS account, follow the instructions in the [Amazon Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html#add-model-access). | AWS DevOps |
| Deploy resources in the account. | To deploy resources in the AWS account by using the AWS CDK, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/develop-advanced-generative-ai-chat-based-assistants-by-using-rag-and-react-prompting.html)Upon successful deployment, you can access the chat-based assistant application by using the URL provided in the CloudFormation **Outputs** section. | AWS DevOps, DevOps engineer |
| Run the AWS Glue crawler and create the Data Catalog table. | An [AWS Glue crawler](https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html) is used to keep the data schema dynamic. The solution creates and updates partitions in the [AWS Glue Data Catalog table](https://docs.aws.amazon.com/athena/latest/ug/querying-glue-catalog.html) by running the crawler on demand. After the CSV dataset files are copied into the S3 bucket, run the AWS Glue crawler and create the Data Catalog table schema for testing:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/develop-advanced-generative-ai-chat-based-assistants-by-using-rag-and-react-prompting.html)The AWS CDK code configures the AWS Glue crawler to run on demand, but you can also [schedule](https://docs.aws.amazon.com/glue/latest/dg/schedule-crawler.html) it to run periodically. | DevOps engineer, AWS DevOps |
| Initiate document indexing. | After the files are copied into the S3 bucket, use Amazon Kendra to crawl and index them:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/develop-advanced-generative-ai-chat-based-assistants-by-using-rag-and-react-prompting.html)The AWS CDK code configures the Amazon Kendra index sync to run on demand, but you can also run periodically by using the [Schedule parameter](https://docs.aws.amazon.com/kendra/latest/dg/data-source.html#cron). | AWS DevOps, DevOps engineer |
### Clean up all AWS resources in the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Remove the AWS resources. | After you test the solution, clean up the resources:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/develop-advanced-generative-ai-chat-based-assistants-by-using-rag-and-react-prompting.html) | DevOps engineer, AWS DevOps |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| AWS CDK returns errors. | For help with AWS CDK issues, see [Troubleshooting common AWS CDK issues](https://docs.aws.amazon.com/cdk/v2/guide/troubleshooting.html). |
## Related resources
+ Amazon Bedrock:
+ [Model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html)
+ [Inference parameters for foundation models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html)
+ [Building Lambda functions with Python](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html)
+ [Get started with the AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html)
+ [Working with the AWS CDK in Python](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-python.html)
+ [Generative AI Application Builder on AWS](https://docs.aws.amazon.com/solutions/latest/generative-ai-application-builder-on-aws/solution-overview.html)
+ [LangChain documentation](https://python.langchain.com/docs/get_started/introduction)
+ [Streamlit documentation](https://docs.streamlit.io/)
## Additional information
**AWS CDK commands**
When working with AWS CDK, keep in mind the following useful commands:
+ Lists all stacks in the app
```
cdk ls
```
+ Emits the synthesized AWS CloudFormation template
```
cdk synth
```
+ Deploys the stack to your default AWS account and Region
```
cdk deploy
```
+ Compares the deployed stack with the current state
```
cdk diff
```
+ Opens the AWS CDK documentation
```
cdk docs
```
+ Deletes the CloudFormation stack and removes AWS deployed resources
```
cdk destroy
```
# Develop a fully automated chat-based assistant by using Amazon Bedrock agents and knowledge bases
*Jundong Qiao, Shuai Cao, Noah Hamilton, Kiowa Jackson, Praveen Kumar Jeyarajan, and Kara Yang, Amazon Web Services*
## Summary
Many organizations face challenges when creating a chat-based assistant that is capable of orchestrating diverse data sources to offer comprehensive answers. This pattern presents a solution for developing a chat-based assistant that is capable of answering queries from both documentation and databases, with a straightforward deployment.
Starting with [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html), this fully managed generative artificial intelligence (AI) service provides a wide array of advanced foundation models (FMs). This facilitates the efficient creation of generative AI applications with a strong focus on privacy and security. In the context of documentation retrieval, the [Retrieval Augmented Generation (RAG)](https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-customize-rag.html) is a pivotal feature. It uses [knowledge bases](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html) to augment FM prompts with contextually relevant information from external sources. An [Amazon OpenSearch Serverless](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-overview.html) index serves as the vector database behind the knowledge bases for Amazon Bedrock. This integration is enhanced through careful prompt engineering to minimize inaccuracies and make sure that responses are anchored in factual documentation. For database queries, the FMs of Amazon Bedrock transform textual inquiries into structured SQL queries, incorporating specific parameters. This enables the precise retrieval of data from databases managed by [AWS Glue databases](https://docs.aws.amazon.com/glue/latest/dg/define-database.html). [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) is used for these queries.
For handling more intricate queries, achieving comprehensive answers demands information sourced from both documentation and databases. [Agents for Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html) is a generative AI feature that helps you build autonomous agents that can understand complex tasks and break them down into simpler tasks for orchestration. The combination of insights retrieved from the simplified tasks, facilitated by Amazon Bedrock autonomous agents, enhances the synthesis of information, leading to more thorough and exhaustive answers. This pattern demonstrates how to build a chat-based assistant by using Amazon Bedrock and the related generative AI services and features within an automated solution.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Docker, [installed](https://docs.docker.com/engine/install/)
+ AWS Cloud Development Kit (AWS CDK), [installed](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_tools) and [bootstrapped](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_bootstrap) to the `us-east-1` or `us-west-2` AWS Regions
+ AWS CDK Toolkit version 2.114.1 or later, [installed](https://docs.aws.amazon.com/cdk/v2/guide/cli.html)
+ AWS Command Line Interface (AWS CLI), [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
+ Python version 3.11 or later, [installed](https://www.python.org/downloads/)
+ In Amazon Bedrock, [enable access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) to Claude 2, Claude 2.1, Claude Instant, and Titan Embeddings G1 – Text
**Limitations**
+ This solution is deployed to a single AWS account.
+ This solution can be deployed only in AWS Regions where Amazon Bedrock and Amazon OpenSearch Serverless are supported. For more information, see the documentation for [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-regions.html) and [Amazon OpenSearch Serverless](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-overview.html#serverless-regions).
**Product versions**
+ Llama-index version 0.10.6 or later
+ Sqlalchemy version 2.0.23 or later
+ Opensearch-py version 2.4.2 or later
+ Requests\$1aws4auth version 1.2.3 or later
+ AWS SDK for Python (Boto3) version 1.34.57 or later
## Architecture
**Target technology stack **
The [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/v2/guide/home.html) is an open source software development framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. The AWS CDK stack used in this pattern deploys the following AWS resources:
+ AWS Key Management Service (AWS KMS)
+ Amazon Simple Storage Service (Amazon S3)
+ AWS Glue Data Catalog, for the AWS Glue database component
+ AWS Lambda
+ AWS Identity and Access Management (IAM)
+ Amazon OpenSearch Serverless
+ Amazon Elastic Container Registry (Amazon ECR)
+ Amazon Elastic Container Service (Amazon ECS)
+ AWS Fargate
+ Amazon Virtual Private Cloud (Amazon VPC)
+ [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html)
**Target architecture **
![\[Architecture diagram using an Amazon Bedrock knowledge base and agent\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/15372718-3a5d-4918-9cfa-422c455f288d/images/ff19152e-0bb6-4758-a6dd-4f6140e55113.png)
The diagram shows a comprehensive AWS cloud-native setup within a single AWS Region, using multiple AWS services. The primary interface for the chat-based assistant is a [Streamlit](https://docs.streamlit.io/) application hosted on an Amazon ECS cluster. An [Application Load Balancer](https://aws.amazon.com/elasticloadbalancing/application-load-balancer/) manages accessibility. Queries made through this interface activate the `Invocation` Lambda function, which then interfaces with agents for Amazon Bedrock. This agent responds to user inquiries by either consulting the knowledge bases for Amazon Bedrock or by invoking an `Agent executor` Lambda function. This function triggers a set of actions associated with the agent, following a predefined API schema. The knowledge bases for Amazon Bedrock use an OpenSearch Serverless index as their vector database foundation. Additionally, the `Agent executor` function generates SQL queries that are executed against the AWS Glue database through Amazon Athena.
## Tools
**AWS services**
+ [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) is an interactive query service that helps you analyze data directly in Amazon Simple Storage Service (Amazon S3) by using standard SQL.
+ [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models (FMs) from leading AI startups and Amazon available for your use through a unified API.
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/latest/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command-line shell.
+ [Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) is a fast and scalable container management service that helps you run, stop, and manage containers on a cluster.
+ [Elastic Load Balancing ](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html)distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across Amazon Elastic Compute Cloud (Amazon EC2) instances, containers, and IP addresses in one or more Availability Zones.
+ [AWS Glue](https://docs.aws.amazon.com/glue/) is a fully managed extract, transform, and load (ETL) service. It helps you reliably categorize, clean, enrich, and move data between data stores and data streams. This pattern uses an AWS Glue crawler and an AWS Glue Data Catalog table.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon OpenSearch Serverless](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-overview.html) is an on-demand serverless configuration for Amazon OpenSearch Service. In this pattern, an OpenSearch Serverless index serves as a vector database for the knowledge bases for Amazon Bedrock.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Other tools**
+ [Streamlit](https://docs.streamlit.io/) is an open source Python framework to create data applications.
**Code repository**
The code for this pattern is available in the GitHub [genai-bedrock-agent-chatbot](https://github.com/awslabs/genai-bedrock-agent-chatbot/) repository. The code repository contains the following files and folders:
+ `assets` folder – The static assets, such as the architecture diagram and the public dataset.
+ `code/lambdas/action-lambda` folder – The Python code for the Lambda function that acts as an action for the Amazon Bedrock agent.
+ `code/lambdas/create-index-lambda` folder – The Python code for the Lambda function that creates the OpenSearch Serverless index.
+ `code/lambdas/invoke-lambda` folder – The Python code for the Lambda function that invokes the Amazon Bedrock agent, which is called directly from the Streamlit application.
+ `code/lambdas/update-lambda` folder – The Python code for the Lambda function that updates or deletes resources after the AWS resources are deployed through the AWS CDK.
+ `code/layers/boto3_layer` folder – The AWS CDK stack that creates a Boto3 layer that is shared across all Lambda functions.
+ `code/layers/opensearch_layer` folder – The AWS CDK stack that creates an OpenSearch Serverless layer that installs all dependencies to create the index.
+ `code/streamlit-app` folder – The Python code that is run as the container image in Amazon ECS.
+ `code/code_stack.py` – The AWS CDK construct Python files that create AWS resources.
+ `app.py` – The AWS CDK stack Python files that deploy AWS resources in the target AWS account.
+ `requirements.txt` – The list of all Python dependencies that must be installed for the AWS CDK.
+ `cdk.json` – The input file to provide the values that are required to create resources. Also, in the `context/config` fields, you can customize the solution accordingly. For more information about customization, see the [Additional information](#develop-a-fully-automated-chat-based-assistant-by-using-amazon-bedrock-agents-and-knowledge-bases-additional) section.
## Best practices
+ The code example provided here is for proof-of-concept (PoC) or pilot purposes only. If you want to take the code to production, be sure to use the following best practices:
+ Enable [Amazon S3 access logging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html)
+ Enable [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html)
+ Set up monitoring and alerting for the Lambda functions. For more information, see [Monitoring and troubleshooting Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-monitoring.html). For best practices, see the [Best practices for working with AWS Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html).
## Epics
### Set up AWS credentials on your local workstation
| Task | Description | Skills required |
| --- | --- | --- |
| Export variables for the account and Region. | To provide AWS credentials for the AWS CDK by using environment variables, run the following commands.export CDK_DEFAULT_ACCOUNT=<12-digit AWS account number>
export CDK_DEFAULT_REGION=
| AWS DevOps, DevOps engineer |
| Set up the AWS CLI named profile. | To set up the AWS CLI named profile for the account, follow the instructions in [Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). | AWS DevOps, DevOps engineer |
### Set up your environment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repo to your local workstation. | To clone the repository, run the following command in your terminal.git clone https://github.com/awslabs/genai-bedrock-agent-chatbot.git
| DevOps engineer, AWS DevOps |
| Set up the Python virtual environment. | To set up the Python virtual environment, run the following commands.cd genai-bedrock-agent-chatbot
python3 -m venv .venv
source .venv/bin/activate
To set up the required dependencies, run the following command.pip3 install -r requirements.txt
| DevOps engineer, AWS DevOps |
| Set up the AWS CDK environment. | To convert the code to an AWS CloudFormation template, run the command `cdk synth`. | AWS DevOps, DevOps engineer |
### Configure and deploy the application
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy resources in the account. | To deploy resources in the AWS account by using the AWS CDK, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/develop-a-fully-automated-chat-based-assistant-by-using-amazon-bedrock-agents-and-knowledge-bases.html)After successful deployment, you can access the chat-based assistant application by using the URL provided on the **Outputs** tab in the CloudFormation console. | DevOps engineer, AWS DevOps |
### Clean up all AWS resources in the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Remove the AWS resources. | After you test the solution, to clean up the resources, run the command `cdk destroy`. | AWS DevOps, DevOps engineer |
## Related resources
**AWS documentation**
+ Amazon Bedrock resources:
+ [Model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html)
+ [Inference parameters for foundation models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html)
+ [Agents for Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html)
+ [Knowledge bases for Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html)
+ [Building Lambda functions with Python](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html)
+ AWS CDK resources:
+ [Get started with the AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html)
+ [Troubleshooting common AWS CDK issues](https://docs.aws.amazon.com/cdk/v2/guide/troubleshooting.html)
+ [Working with the AWS CDK in Python](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-python.html)
+ [Generative AI Application Builder on AWS](https://docs.aws.amazon.com/solutions/latest/generative-ai-application-builder-on-aws/solution-overview.html)
**Other AWS resources**
+ [Vector Engine for Amazon OpenSearch Serverless](https://aws.amazon.com/opensearch-service/serverless-vector-engine/)
**Other resources**
+ [LlamaIndex documentation](https://docs.llamaindex.ai/en/stable/)
+ [Streamlit documentation](https://docs.streamlit.io/)
## Additional information
**Customize the chat-based assistant with your own data**
To integrate your custom data for deploying the solution, follow these structured guidelines. These steps are designed to ensure a seamless and efficient integration process, enabling you to deploy the solution effectively with your bespoke data.
*For knowledge base data integration*
**Data preparation**
1. Locate the `assets/knowledgebase_data_source/` directory.
1. Place your dataset within this folder.
**Configuration adjustments**
1. Open the `cdk.json` file.
1. Navigate to the `context/configure/paths/knowledgebase_file_name` field, and then update it accordingly.
1. Navigate to the `bedrock_instructions/knowledgebase_instruction` field, and then update it to accurately reflect the nuances and context of your new dataset.
*For structural data integration*
**Data organization**
1. Within the `assets/data_query_data_source/` directory, create a subdirectory, such as `tabular_data`.
1. Put your structured dataset (acceptable formats include CSV, JSON, ORC, and Parquet) into this newly created subfolder.
1. If you are connecting to an existing database, update the function `create_sql_engine()` in `code/lambda/action-lambda/build_query_engine.py` to connect to your database.
**Configuration and code updates**
1. In the `cdk.json` file, update the `context/configure/paths/athena_table_data_prefix` field to align with the new data path.
1. Revise `code/lambda/action-lambda/dynamic_examples.csv` by incorporating new text-to-SQL examples that correspond with your dataset.
1. Revise `code/lambda/action-lambda/prompt_templates.py` to mirror the attributes of your structured dataset.
1. In the `cdk.json` file, update the `context/configure/bedrock_instructions/action_group_description` field to explain the purpose and functionality of the `Action group` Lambda function.
1. In the `assets/agent_api_schema/artifacts_schema.json` file, explain the new functionalities of your `Action group` Lambda function.
*General update*
In the `cdk.json` file, in the `context/configure/bedrock_instructions/agent_instruction` section, provide a comprehensive description of the Amazon Bedrock agent's intended functionality and design purpose, taking into account the newly integrated data.
# Document institutional knowledge from voice inputs by using Amazon Bedrock and Amazon Transcribe
*Praveen Kumar Jeyarajan, Jundong Qiao, Rajiv Upadhyay, and Megan Wu, Amazon Web Services*
## Summary
Capturing institutional knowledge is paramount for ensuring organizational success and resilience. Institutional knowledge represents the collective wisdom, insights, and experiences accumulated by employees over time, often tacit in nature and passed down informally. This wealth of information encompasses unique approaches, best practices, and solutions to intricate problems that might not be documented elsewhere. By formalizing and documenting this knowledge, companies can preserve institutional memory, foster innovation, enhance decision-making processes, and accelerate learning curves for new employees. Additionally, it promotes collaboration, empowers individuals, and cultivates a culture of continuous improvement. Ultimately, harnessing institutional knowledge helps companies use their most valuable asset—the collective intelligence of their workforce—to navigate challenges, drive growth, and maintain competitive advantage in dynamic business environments.
This pattern explains how to capture institutional knowledge through voice recordings from senior employees. It uses [Amazon Transcribe](https://docs.aws.amazon.com/transcribe/latest/dg/what-is.html) and [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) for systematic documentation and verification. By documenting this informal knowledge, you can preserve it and share it with subsequent cohorts of employees. This endeavor supports operational excellence and improves the effectiveness of training programs through the incorporation of practical knowledge acquired through direct experience.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Docker, [installed](https://docs.docker.com/engine/install/)
+ AWS Cloud Development Kit (AWS CDK) version 2.114.1 or later, [installed](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_tools) and [bootstrapped](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_bootstrap) to the `us-east-1` or `us-west-2` AWS Regions
+ AWS CDK Toolkit version 2.114.1 or later, [installed](https://docs.aws.amazon.com/cdk/v2/guide/cli.html)
+ AWS Command Line Interface (AWS CLI), [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
+ Python version 3.12 or later, [installed](https://www.python.org/downloads/)
+ Permissions to create Amazon Transcribe, Amazon Bedrock, Amazon Simple Storage Service (Amazon S3), and AWS Lambda resources
**Limitations**
+ This solution is deployed to a single AWS account.
+ This solution can be deployed only in AWS Regions where Amazon Bedrock and Amazon Transcribe are available. For information about availability, see the documentation for [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-regions.html) and [Amazon Transcribe](https://docs.aws.amazon.com/transcribe/latest/dg/what-is.html#tsc-regions).
+ The audio files must be in a format that Amazon Transcribe supports. For a list of supported formats, see [Media formats](https://docs.aws.amazon.com/transcribe/latest/dg/how-input.html#how-input-audio) in the Transcribe documentation.
**Product versions**
+ AWS SDK for Python (Boto3) version 1.34.57 or later
+ LangChain version 0.1.12 or later
## Architecture
The architecture represents a serverless workflow on AWS. [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) orchestrates Lambda functions for audio processing, text analysis, and document generation. The following diagram shows the Step Functions workflow, also known as a *state machine*.
![\[Architecture diagram of Step Functions state machine generating a document\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f1e0106d-b046-4adc-9718-c299efb7b436/images/e90298ca-1b7f-4c3e-97bd-311a9d5a4997.png)
Each step in the state machine is handled by a distinct Lambda function. The following are the steps in the document generation process:
1. The `preprocess` Lambda function validates the input passed to Step Functions and lists all of the audio files present in the provided Amazon S3 URI folder path. Downstream Lambda functions in the workflow use the file list to validate, summarize, and generate the document.
1. The `transcribe` Lambda function uses Amazon Transcribe to convert audio files into text transcripts. This Lambda function is responsible for initiating the transcription process and accurately transforming speech into text, which is then stored for subsequent processing.
1. The `validate` Lambda function analyzes the text transcripts, determining the relevance of the responses to the initial questions. By using a large language model (LLM) through Amazon Bedrock, it identifies and separates on-topic answers from off-topic responses.
1. The `summarize` Lambda function uses Amazon Bedrock to generate a coherent and concise summary of the on-topic answers.
1. The `generate` Lambda function assembles the summaries into a well-structured document. It can format the document according to predefined templates and include any additional necessary content or data.
1. If any of the Lambda functions fail, you receive an email notification through Amazon Simple Notification Service (Amazon SNS).
Throughout this process, AWS Step Functions makes sure that each Lambda function is initiated in the correct sequence. This state machine has the capacity for parallel processing to enhance efficiency. An Amazon S3 bucket acts as the central storage repository, supporting the workflow by managing the various media and document formats involved.
## Tools
**AWS services**
+ [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models (FMs) from leading AI startups and Amazon available for your use through a unified API.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Simple Notification Service (Amazon SNS)](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine AWS Lambda functions and other AWS services to build business-critical applications.
+ [Amazon Transcribe](https://docs.aws.amazon.com/transcribe/latest/dg/what-is.html) is an automatic speech recognition service that uses machine learning models to convert audio to text.
**Other tools**
+ [LangChain](https://python.langchain.com/docs/get_started/introduction/) is a framework for developing applications that are powered by large language models (LLMs).
**Code repository**
The code for this pattern is available in the GitHub [genai-knowledge-capture](https://github.com/aws-samples/genai-knowledge-capture) repository.
The code repository contains the following files and folders:
+ `assets` folder – The static assets for the solution, such as the architecture diagram and the public dataset
+ `code/lambdas` folder – The Python code for all Lambda functions
+ `code/lambdas/generate` folder - The Python code that generates a document from the summarized data in the S3 bucket
+ `code/lambdas/preprocess` folder - The Python code that processes the inputs for the Step Functions state machine
+ `code/lambdas/summarize` folder - The Python code that summarizes the transcribed data by using Amazon Bedrock service
+ `code/lambdas/transcribe` folder - The Python code that converts speech data (audio file) into text by using Amazon Transcribe
+ `code/lambdas/validate` folder - The Python code that validates whether all answers pertain to the same topic
+ `code/code_stack.py` – The AWS CDK construct Python file that is used to create AWS resources
+ `app.py` – The AWS CDK app Python file that is used to deploy AWS resources in the target AWS account
+ `requirements.txt` – The list of all Python dependencies that must be installed for the AWS CDK
+ `cdk.json` – The input file to provide values that are required to create resources
## Best practices
The code example provided is for proof-of-concept (PoC) or pilot purposes only. If you want to take the solution to production, use the following best practices:
+ Enable [Amazon S3 access logging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html)
+ Enable [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html)
## Epics
### Set up AWS credentials on your local workstation
| Task | Description | Skills required |
| --- | --- | --- |
| Export variables for the account and AWS Region. | To provide AWS credentials for the AWS CDK by using environment variables, run the following commands.export CDK_DEFAULT_ACCOUNT=<12-digit AWS account number>
export CDK_DEFAULT_REGION=
| AWS DevOps, DevOps engineer |
| Set up the AWS CLI named profile. | To set up the AWS CLI named profile for the account, follow the instructions in [Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). | AWS DevOps, DevOps engineer |
### Set up your environment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repo to your local workstation. | To clone the [genai-knowledge-capture](https://github.com/aws-samples/genai-knowledge-capture) repository, run the following command in your terminal.git clone https://github.com/aws-samples/genai-knowledge-capture
| AWS DevOps, DevOps engineer |
| (Optional) Replace the audio files. | To customize the sample application to incorporate your own data, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/document-institutional-knowledge-from-voice-inputs-by-using-amazon-bedrock-and-amazon-transcribe.html) | AWS DevOps, DevOps engineer |
| Set up the Python virtual environment. | To set up the Python virtual environment, run the following commands.cd genai-knowledge-capture
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
| AWS DevOps, DevOps engineer |
| Synthesize the AWS CDK code. | To convert the code to an AWS CloudFormation stack configuration, run the following command.cdk synth
| AWS DevOps, DevOps engineer |
### Configure and deploy the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Provision foundation model access. | Enable access to the Anthropic Claude 3 Sonnet model for your AWS account. For instructions, see [Add model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html#model-access-add) in the Bedrock documentation. | AWS DevOps |
| Deploy resources in the account. | To deploy resources in the AWS account by using the AWS CDK, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/document-institutional-knowledge-from-voice-inputs-by-using-amazon-bedrock-and-amazon-transcribe.html) | AWS DevOps, DevOps engineer |
| Subscribe to the Amazon SNS topic. | To subscribe to the Amazon SNS topic for notification, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/document-institutional-knowledge-from-voice-inputs-by-using-amazon-bedrock-and-amazon-transcribe.html) | General AWS |
### Test the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Run the state machine. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/document-institutional-knowledge-from-voice-inputs-by-using-amazon-bedrock-and-amazon-transcribe.html) | App developer, General AWS |
### Clean up all AWS resources in the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Remove the AWS resources. | After you test the solution, clean up the resources:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/document-institutional-knowledge-from-voice-inputs-by-using-amazon-bedrock-and-amazon-transcribe.html) | AWS DevOps, DevOps engineer |
## Related resources
**AWS documentation**
+ Amazon Bedrock resources:
+ [Model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html)
+ [Inference parameters for foundation models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html)
+ AWS CDK resources:
+ [Get started with the AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html)
+ [Working with the AWS CDK in Python](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-python.html)
+ [Troubleshooting common AWS CDK issues](https://docs.aws.amazon.com/cdk/v2/guide/troubleshooting.html)
+ [Toolkit commands](https://docs.aws.amazon.com/cdk/v2/guide/cli.html#cli-commands)
+ AWS Step Functions resources:
+ [Getting started with AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/getting-started-with-sfn.html)
+ [Troubleshooting](https://docs.aws.amazon.com/step-functions/latest/dg/troubleshooting.html)
+ [Building Lambda functions with Python](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html)
+ [Generative AI Application Builder on AWS](https://docs.aws.amazon.com/solutions/latest/generative-ai-application-builder-on-aws/solution-overview.html)
**Other resources**
+ [LangChain documentation](https://python.langchain.com/docs/get_started/introduction)
# Generate personalized and re-ranked recommendations using Amazon Personalize
*Mason Cahill, Matthew Chasse, and Tayo Olajide, Amazon Web Services*
## Summary
This pattern shows you how to use Amazon Personalize to generate personalized recommendations—including re-ranked recommendations—for your users based on the ingestion of real-time user-interaction data from those users. The example scenario used in this pattern is based on a pet adoption website that generates recommendations for its users based on their interactions (for example, what pets a user visits). By following the example scenario, you learn to use Amazon Kinesis Data Streams to ingest interaction data, AWS Lambda to generate recommendations and re-rank the recommendations, and Amazon Data Firehose to store the data in an Amazon Simple Storage Service (Amazon S3) bucket. You also learn to use AWS Step Functions to build a state machine that manages the solution version (that is, a trained model) that generates your recommendations.
## Prerequisites and limitations
**Prerequisites**
+ An active [AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) with a [bootstrapped](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html) AWS Cloud Development Kit (AWS CDK)
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) with configured credentials
+ [Python 3.9](https://www.python.org/downloads/release/python-390/)
**Product versions**
+ Python 3.9
+ AWS CDK 2.23.0 or later
+ AWS CLI 2.7.27 or later
## Architecture
**Technology stack**
+ Amazon Data Firehose
+ Amazon Kinesis Data Streams
+ Amazon Personalize
+ Amazon Simple Storage Service (Amazon S3)
+ AWS Cloud Development Kit (AWS CDK)
+ AWS Command Line Interface (AWS CLI)
+ AWS Lambda
+ AWS Step Functions
**Target architecture**
The following diagram illustrates a pipeline for ingesting real-time data into Amazon Personalize. The pipeline then uses that data to generate personalized and re-ranked recommendations for users.
![\[Data ingestion architecture for Amazon Personalize\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/42eb193b-2347-408a-8b25-46beeb3b29ca/images/786dbd56-7d7f-41bb-90f6-d4485d73fe15.png)
The diagram shows the following workflow:
1. Kinesis Data Streams ingests real-time user data (for example, events like visited pets) for processing by Lambda and Firehose.
1. A Lambda function processes the records from Kinesis Data Streams and makes an API call to add the user-interaction in the record to an event tracker in Amazon Personalize.
1. A time-based rule invokes a Step Functions state machine and generates new solution versions for the recommendation and re-ranking models by using the events from the event tracker in Amazon Personalize.
1. Amazon Personalize [campaigns](https://docs.aws.amazon.com/personalize/latest/dg/campaigns.html) are updated by the state machine to use the new [solution version](https://docs.aws.amazon.com/personalize/latest/dg/creating-a-solution-version.html).
1. Lambda re-ranks the list of recommended items by calling the Amazon Personalize re-ranking campaign.
1. Lambda retrieves the list of recommended items by calling the Amazon Personalize recommendations campaign.
1. Firehose saves the events to an S3 bucket where they can be accessed as historical data.
## Tools
**AWS tools**
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/latest/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
+ [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) helps you deliver real-time [streaming data](https://aws.amazon.com/streaming-data/) to other AWS services, custom HTTP endpoints, and HTTP endpoints owned by supported third-party service providers.
+ [Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) helps you collect and process large streams of data records in real time.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Personalize](https://docs.aws.amazon.com/personalize/latest/dg/what-is-personalize.html) is a fully managed machine learning (ML) service that helps you generate item recommendations for your users based on your data.
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine Lambda functions and other AWS services to build business-critical applications.
**Other tools**
+ [pytest ](https://docs.pytest.org/en/7.2.x/index.html)is a Python framework for writing small, readable tests.
+ [Python](https://www.python.org/) is a general-purpose computer programming language.
**Code**
The code for this pattern is available in the GitHub [Animal Recommender](https://github.com/aws-samples/personalize-pet-recommendations) repository. You can use the AWS CloudFormation template from this repository to deploy the resources for the example solution.
**Note**
The Amazon Personalize solution versions, event tracker, and campaigns are backed by [custom resources](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html) (within the infrastructure) that expand on native CloudFormation resources.
## Epics
### Create the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create an isolated Python environment. | **Mac/Linux setup**[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-personalized-and-re-ranked-recommendations-using-amazon-personalize.html)**Windows setup**To manually create a virtual environment, run the `% .venv\Scripts\activate.bat` command from your terminal. | DevOps engineer |
| Synthesize the CloudFormation template. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-personalized-and-re-ranked-recommendations-using-amazon-personalize.html)In step 2, `CDK_ENVIRONMENT` refers to the `config/{env}.yml` file. | DevOps engineer |
| Deploy resources and create infrastructure. | To deploy the solution resources, run the `./deploy.sh` command from your terminal.This command installs the required Python dependencies. A Python script creates an S3 bucket and an AWS Key Management Service (AWS KMS) key, and then adds the seed data for the initial model creations. Finally, the script runs `cdk deploy` to create the remaining infrastructure.The initial model training happens during stack creation. It can take up to two hours for the stack to finish getting created. | DevOps engineer |
## Related resources
+ [Animal Recommender](https://github.com/aws-samples/personalize-pet-recommendations) (GitHub)
+ [AWS CDK Reference Documentation](https://docs.aws.amazon.com/cdk/api/v2/)
+ [Boto3 Documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)
+ [Optimize personalized recommendations for a business metric of your choice with Amazon Personalize](https://aws.amazon.com/blogs/machine-learning/optimize-personalized-recommendations-for-a-business-metric-of-your-choice-with-amazon-personalize/) (AWS Machine Learning Blog)
## Additional information
**Example payloads and responses**
*Recommendation Lambda function*
To retrieve recommendations, submit a request to the recommendation Lambda function with a payload in the following format:
```
{
"userId": "3578196281679609099",
"limit": 6
}
```
The following example response contains a list of animal groups:
```
[{"id": "1-domestic short hair-1-1"},
{"id": "1-domestic short hair-3-3"},
{"id": "1-domestic short hair-3-2"},
{"id": "1-domestic short hair-1-2"},
{"id": "1-domestic short hair-3-1"},
{"id": "2-beagle-3-3"},
```
If you leave out the `userId` field, the function returns general recommendations.
*Re-ranking Lambda function*
To use re-ranking, submit a request to the re-ranking Lambda function. The payload contains the `userId` of all the item IDs to be re-ranked and their metadata. The following example data uses the Oxford Pets classes for `animal_species_id` (1=cat, 2=dog) and integers 1-5 for `animal_age_id` and `animal_size_id`:
```
{
"userId":"12345",
"itemMetadataList":[
{
"itemId":"1",
"animalMetadata":{
"animal_species_id":"2",
"animal_primary_breed_id":"Saint_Bernard",
"animal_size_id":"3",
"animal_age_id":"2"
}
},
{
"itemId":"2",
"animalMetadata":{
"animal_species_id":"1",
"animal_primary_breed_id":"Egyptian_Mau",
"animal_size_id":"1",
"animal_age_id":"1"
}
},
{
"itemId":"3",
"animalMetadata":{
"animal_species_id":"2",
"animal_primary_breed_id":"Saint_Bernard",
"animal_size_id":"3",
"animal_age_id":"2"
}
}
]
}
```
The Lambda function re-ranks these items, and then returns an ordered list that includes the item IDs and the direct response from Amazon Personalize. This is a ranked list of the animal groups that the items are in and their score. Amazon Personalize uses [User-Personalization](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-new-item-USER_PERSONALIZATION.html) and [Personalized-Ranking](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-search.html) recipes to include a score for each item in the recommendations. These scores represent the relative certainty that Amazon Personalize has about which item the user will choose next. Higher scores represent greater certainty.
```
{
"ranking":[
"1",
"3",
"2"
],
"personalizeResponse":{
"ResponseMetadata":{
"RequestId":"a2ec0417-9dcd-4986-8341-a3b3d26cd694",
"HTTPStatusCode":200,
"HTTPHeaders":{
"date":"Thu, 16 Jun 2022 22:23:33 GMT",
"content-type":"application/json",
"content-length":"243",
"connection":"keep-alive",
"x-amzn-requestid":"a2ec0417-9dcd-4986-8341-a3b3d26cd694"
},
"RetryAttempts":0
},
"personalizedRanking":[
{
"itemId":"2-Saint_Bernard-3-2",
"score":0.8947961
},
{
"itemId":"1-Siamese-1-1",
"score":0.105204
}
],
"recommendationId":"RID-d97c7a87-bd4e-47b5-a89b-ac1d19386aec"
}
}
```
*Amazon Kinesis payload*
The payload to send to Amazon Kinesis has the following format:
```
{
"Partitionkey": "randomstring",
"Data": {
"userId": "12345",
"sessionId": "sessionId4545454",
"eventType": "DetailView",
"animalMetadata": {
"animal_species_id": "1",
"animal_primary_breed_id": "Russian_Blue",
"animal_size_id": "1",
"animal_age_id": "2"
},
"animal_id": "98765"
}
}
```
**Note**
The `userId` field is removed for an unauthenticated user.
# Streamline machine learning workflows from local development to scalable experiments by using SageMaker AI and Hydra
*David Sauerwein, Marco Geiger, and Julian Ferdinand Grueber, Amazon Web Services*
## Summary
This pattern provides a unified approach to configuring and running machine learning (ML) algorithms from local testing to production on Amazon SageMaker AI. ML algorithms are the focus of this pattern, but its approach extends to feature engineering, inference, and whole ML pipelines. This pattern demonstrates the transition from local script development to SageMaker AI training jobs through a sample use case.
A typical ML workflow is to develop and test solutions on a local machine, run large scale experiments (for example, with different parameters) in the cloud, and deploy the approved solution in the cloud. Then, the deployed solution must be monitored and maintained. Without a unified approach to this workflow, developers often need to refactor their code at each stage. If the solution depends on a large number of parameters that might change at any stage of this workflow, it can become increasingly difficult to remain organized and consistent.
This pattern addresses these challenges. First, it eliminates the need for code refactoring between environments by providing a unified workflow that remains consistent whether running on local machines, in containers, or on SageMaker AI. Second, it simplifies parameter management through Hydra's configuration system, where parameters are defined in separate configuration files that can be easily modified and combined, with automatic logging of each run's configuration. For more details about how this pattern addresses these challenges, see [Additional information](#streamline-machine-learning-workflows-by-using-amazon-sagemaker-additional).
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An AWS Identity and Access Management (IAM) [user role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) for deploying and starting the SageMaker AI training jobs
+ AWS Command Line Interface (AWS CLI) version 2.0 or later [installed](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
+ [Poetry](https://python-poetry.org/) version 1.8 or later, but earlier than 2.0, installed
+ [Docker](https://www.docker.com/) installed
+ Python [version 3.10.x](https://www.python.org/downloads/release/python-31011/)
**Limitations**
+ The code currently only targets SageMaker AI training jobs. Extending it to processing jobs and whole SageMaker AI pipelines is straightforward.
+ For a fully productionized SageMaker AI setup, additional details need to be in place. Examples could be custom AWS Key Management Service (AWS KMS) keys for compute and storage, or networking configurations. You can also configure these additional options by using Hydra in a dedicated subfolder of the `config` folder.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
## Architecture
The following diagram depicts the architecture of the solution.
![\[Workflow to create and run SageMaker AI training or HPO jobs.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/1db57484-f85c-49a6-b870-471dade02b26/images/d80e7474-a975-4d92-8f66-2d34e33053fd.png)
The diagram shows the following workflow:
1. The data scientist can iterate over the algorithm at small scale in a local environment, adjust parameters, and test the training script rapidly without the need for Docker or SageMaker AI. (For more details, see the "Run locally for quick testing" task in [Epics](#streamline-machine-learning-workflows-by-using-amazon-sagemaker-epics).)
1. Once satisfied with the algorithm, the data scientist builds and pushes the Docker image to the Amazon Elastic Container Registry (Amazon ECR) repository named `hydra-sm-artifact`. (For more details, see "Run workflows on SageMaker AI" in [Epics](#streamline-machine-learning-workflows-by-using-amazon-sagemaker-epics).)
1. The data scientist initiates either SageMaker AI training jobs or hyperparameter optimization (HPO) jobs by using Python scripts. For regular training jobs, the adjusted configuration is written to the Amazon Simple Storage Service (Amazon S3) bucket named `hydra-sample-config`. For HPO jobs, the default configuration set located in the `config` folder is applied.
1. The SageMaker AI training job pulls the Docker image, reads the input data from the Amazon S3 bucket `hydra-sample-data`, and either fetches the configuration from the Amazon S3 bucket `hydra-sample-config` or uses the default configuration. After training, the job saves the output data to the Amazon S3 bucket `hydra-sample-data`.
**Automation and scale**
+ For automated training, retraining, or inference, you can integrate the AWS CLI code with services like [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), [AWS CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html), or [Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html).
+ Scaling can be achieved by changing configurations for instance sizes or by adding configurations for distributed training.
## Tools
**AWS services**
+ [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and AWS Regions.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command-line shell. For this pattern, the AWS CLI is useful for both initial resource configuration and testing.
+ [Amazon Elastic Container Registry (Amazon ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) is a managed container image registry service that’s secure, scalable, and reliable.
+ [Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/?id=docs_gateway) is a managed machine learning (ML) service that helps you build and train ML models and then deploy them into a production-ready hosted environment. SageMaker AI Training is a fully managed ML service within SageMaker AI that enables the training of ML models at scale. The tool can handle the computational demands of training models efficiently, making use of built-in scalability and integration with other AWS services. SageMaker AI Training also supports custom algorithms and containers, making it flexible for a wide range of ML workflows.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Other tools**
+ [Docker](https://www.docker.com/) is a set of platform as a service (PaaS) products that use virtualization at the operating-system level to deliver software in containers. It was used in this pattern to ensure consistent environments across various stages, from development to deployment, and to package dependencies and code reliably. Docker’s containerization allowed for easy scaling and version control across the workflow.
+ [Hydra](https://hydra.cc/) is a configuration management tool that provides flexibility for handling multiple configurations and dynamic resource management. It is instrumental in managing environment configurations, allowing seamless deployment across different environments. For more details about Hydra, see [Additional information](#streamline-machine-learning-workflows-by-using-amazon-sagemaker-additional).
+ [Python](https://www.python.org/) is a general-purpose computer programming language. Python was used to write the ML code and the deployment workflow.
+ [Poetry](https://python-poetry.org/) is a tool for dependency management and packaging in Python.
**Code repository**
The code for this pattern is available in the GitHub [configuring-sagemaker-training-jobs-with-hydra](https://github.com/aws-samples/configuring-sagemaker-training-jobs-with-hydra) repository.
## Best practices
+ Choose an IAM role for deploying and starting the SageMaker AI training jobs that follows the principle of least privilege and grant the minimum permissions required to perform a task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#grant-least-priv) and [Security best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the IAM documentation.
+ Use temporary credentials to access the IAM role in the terminal.
## Epics
### Set up the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Create and activate the virtual environment. | To create and activate the virtual environment, run the following commands in the root of the repository:poetry install
poetry shell
| General AWS |
| Deploy the infrastructure. | To deploy the infrastructure using CloudFormation, run the following command:aws cloudformation deploy --template-file infra/hydra-sagemaker-setup.yaml --stack-name hydra-sagemaker-setup --capabilities CAPABILITY_NAMED_IAM
| General AWS, DevOps engineer |
| Download the sample data. | To download the input data from [openml](https://www.openml.org/) to your local machine, run the following command:python scripts/download_data.py
| General AWS |
| Run locally for quick testing. | To run the training code locally for testing, run the following command:python mypackage/train.py data.train_data_path=data/train.csv evaluation.base_dir_path=data
The logs of all executions are stored by execution time in a folder called `outputs`. For more information, see the "Output" section in the [GitHub repository](https://github.com/aws-samples/configuring-sagemaker-training-jobs-with-hydra).You can also perform multiple trainings in parallel, with different parameters, by using the `--multirun` functionality. For more details, see the [Hydra documentation](https://hydra.cc/docs/tutorials/basic/running_your_app/multi-run/). | Data scientist |
### Run workflows on SageMaker AI
| Task | Description | Skills required |
| --- | --- | --- |
| Set the environment variables. | To run your job on SageMaker AI, set the following environment variables, providing your AWS Region and your AWS account ID:export ECR_REPO_NAME=hydra-sm-artifact
export image_tag=latest
export AWS_REGION="" # for instance, us-east-1
export ACCOUNT_ID=""
export BUCKET_NAME_DATA=hydra-sample-data-$ACCOUNT_ID
export BUCKET_NAME_CONFIG=hydra-sample-config-$ACCOUNT_ID
export AWS_DEFAULT_REGION=$AWS_REGION
export ROLE_ARN=arn:aws:iam::${ACCOUNT_ID}:role/hydra-sample-sagemaker
export INPUT_DATA_S3_PATH=s3://$BUCKET_NAME_DATA/hydra-on-sm/input/
export OUTPUT_DATA_S3_PATH=s3://$BUCKET_NAME_DATA/hydra-on-sm/output/
| General AWS |
| Create and push Docker image. | To create the Docker image and push it to the Amazon ECR repository, run the following command:chmod +x scripts/create_and_push_image.sh
scripts/create_and_push_image.sh $ECR_REPO_NAME $image_tag $AWS_REGION $ACCOUNT_ID
This task assumes that you have valid credentials in your environment. The Docker image is pushed to the Amazon ECR repository specified in the environment variable in the previous task and is used to activate the SageMaker AI container in which the training job will run. | ML engineer, General AWS |
| Copy input data to Amazon S3. | The SageMaker AI training job needs to pick up the input data. To copy the input data to the Amazon S3 bucket for data, run the following command: aws s3 cp data/train.csv "${INPUT_DATA_S3_PATH}train.csv" | Data engineer, General AWS |
| Submit SageMaker AI training jobs. | To simplify the execution of your scripts, specify default configuration parameters in the `default.yaml` file. In addition to ensuring consistency across runs, this approach also offers the flexibility to easily override default settings as needed. See the following example:python scripts/start_sagemaker_training_job.py sagemaker.role_arn=$ROLE_ARN sagemaker.config_s3_bucket=$BUCKET_NAME_CONFIG sagemaker.input_data_s3_path=$INPUT_DATA_S3_PATH sagemaker.output_data_s3_path=$OUTPUT_DATA_S3_PATH
| General AWS, ML engineer, Data scientist |
| Run SageMaker AI hyperparameter tuning. | Running SageMaker AI hyperparameter tuning is similar to submitting a SageMaker AII training job. However, the execution script differs in some important ways as you can see in the [start\$1sagemaker\$1hpo\$1job.py](https://github.com/aws-samples/configuring-sagemaker-training-jobs-with-hydra/blob/main/scripts/start_sagemaker_hpo_job.py) file. The hyperparameters to be tuned must be passed through the boto3 payload, not a channel to the training job.To start the hyperparameter optimization (HPO) job, run the following commands:python scripts/start_sagemaker_hpo_job.py sagemaker.role_arn=$ROLE_ARN sagemaker.config_s3_bucket=$BUCKET_NAME_CONFIG sagemaker.input_data_s3_path=$INPUT_DATA_S3_PATH sagemaker.output_data_s3_path=$OUTPUT_DATA_S3_PATH
| Data scientist |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Expired token | Export fresh AWS credentials. |
| Lack of IAM permissions | Make sure that you export the credentials of an IAM role that has all the required IAM permissions to deploy the CloudFormation template and to start the SageMaker AI training jobs. |
## Related resources
+ [Train a model with Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-training.html) (AWS documentation)
+ [What is Hyperparameter Tuning?](https://aws.amazon.com/what-is/hyperparameter-tuning/#:~:text=Hyperparameter%20tuning%20allows%20data%20scientists,the%20model%20as%20a%20hyperparameter.)
## Additional information
This pattern addresses the following challenges:
**Consistency from local development to at-scale deployment** – With this pattern, developers can use the same workflow, regardless of whether they’re using local Python scripts, running local Docker containers, conducting large experiments on SageMaker AI, or deploying in production on SageMaker AI. This consistency is important for the following reasons:
+ **Faster iteration** – It allows for fast, local experimentation without the need for major adjustments when scaling up.
+ **No refactoring** – Transitioning to larger experiments on SageMaker AI is seamless, requiring no overhaul of the existing setup.
+ **Continuous improvement** – Developing new features and continuously improving the algorithm is straightforward because the code remains the same across environments.
**Configuration management** – This pattern makes use of [Hydra](https://hydra.cc/), a configuration management tool, to provide the following benefits:
+ Parameters are defined in configuration files, separate from the code.
+ Different parameter sets can be swapped or combined easily.
+ Experiment tracking is simplified because each run's configuration is logged automatically.
+ Cloud experiments can use the same configuration structure as local runs, ensuring consistency.
With Hydra, you can manage configuration effectively, enabling the following features:
+ **Divide configurations** – Break your project configurations into smaller, manageable pieces that can be independently modified. This approach makes it easier to handle complex projects.
+ **Adjust defaults easily** – Change your baseline configurations quickly, making it simpler to test new ideas.
+ **Align CLI inputs and config files** – Combine command line inputs with your configuration files smoothly. This approach reduces clutter and confusion, making your project more manageable over time.
# Translate natural language into query DSL for OpenSearch and Elasticsearch queries
*Tabby Ward, Nicholas Switzer, and Breanne Warner, Amazon Web Services*
## Summary
This pattern demonstrates how to use large language models (LLMs) to convert natural language queries into query domain-specific language (query DSL), which makes it easier for users to interact with search services such as OpenSearch and Elasticsearch without extensive knowledge of the query language. This resource is particularly valuable for developers and data scientists who want to enhance search-based applications with natural language querying capabilities, ultimately improving user experience and search functionality.
The pattern illustrates techniques for prompt engineering, iterative refinement, and incorporation of specialized knowledge, all of which are crucial in synthetic data generation. Although this approach focuses primarily on query conversion, it implicitly demonstrates the potential for data augmentation and scalable synthetic data production. This foundation could be extended to more comprehensive synthetic data generation tasks, to highlight the power of LLMs in bridging unstructured natural language inputs with structured, application-specific outputs.
This solution doesn't involve migration or deployment tools in the traditional sense. Instead, it focuses on demonstrating a proof of concept (PoC) for converting natural language queries to query DSL by using LLMs.
+ The pattern uses a Jupyter notebook as a step-by-step guide for setting up the environment and implementing the text-to-query conversion.
+ It uses Amazon Bedrock to access LLMs, which are crucial for interpreting natural language and generating appropriate queries.
+ The solution is designed to work with Amazon OpenSearch Service. You can follow a similar process for Elasticsearch, and the generated queries could potentially be adapted for similar search engines.
[Query DSL](https://opensearch.org/docs/latest/query-dsl/) is a flexible, JSON-based search language that's used to construct complex queries in both Elasticsearch and OpenSearch. It enables you to specify queries in the query parameter of search operations, and supports various query types. A DSL query includes leaf queries and compound queries. Leaf queries search for specific values in certain fields and encompass full-text, term-level, geographic, joining, span, and specialized queries. Compound queries act as wrappers for multiple leaf or compound clauses, and combine their results or modify their behavior. Query DSL supports the creation of sophisticated searches, ranging from simple, match-all queries to complex, multi-clause queries that produce highly specific results. Query DSL is particularly valuable for projects that require advanced search capabilities, flexible query construction, and JSON-based query structures.
This pattern uses techniques such as few-shot prompting, system prompts, structured output, prompt chaining, context provision, and task-specific prompts for text-to-query DSL conversion. For definitions and examples of these techniques, see the [Additional information](#translate-natural-language-query-dsl-opensearch-elasticsearch-additional) section.
## Prerequisites and limitations
**Prerequisites**
To effectively use the Jupyter notebook for converting natural language queries into query DSL queries, you need:
+ **Familiarity with Jupyter notebooks**. Basic understanding of how to navigate and run code in a Jupyter notebook environment.
+ **Python environment**. A working Python environment, preferably Python 3.x, with the necessary libraries installed.
+ **Elasticsearch or OpenSearch knowledge**. Basic knowledge of Elasticsearch or OpenSearch, including its architecture and how to perform queries.
+ **AWS account**. An active AWS account to access Amazon Bedrock and other related services.
+ **Libraries and dependencies**. Installation of specific libraries mentioned in the notebook, such as `boto3` for AWS interaction, and any other dependencies required for LLM integration.
+ **Model access within Amazon Bedrock**. This pattern uses three Claude LLMs from Anthropic. Open the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock/) and choose **Model access**. On the next screen, choose **Enable specific models** and select these three models:
+ Claude 3 Sonnet
+ Claude 3.5 Sonnet
+ Claude 3 Haiku
+ **Proper IAM policies and IAM role**. To run the notebook in an AWS account, your AWS Identity and Access Management (IAM) role requires the `SagemakerFullAccess` policy as well as the policy that’s provided in the [Additional information](#translate-natural-language-query-dsl-opensearch-elasticsearch-additional) section, which you can name `APGtext2querydslpolicy`. This policy includes subscribing to the three Claude models listed.
Having these prerequisites in place ensures a smooth experience when you work with the notebook and implement the text-to-query functionality.
**Limitations**
+ **Proof of concept status**. This project is primarily intended as a proof of concept (PoC). It demonstrates the potential of using LLMs to convert natural language queries into query DSL, but it might not be fully optimized or production-ready.
+ **Model limitations**:
**Context window constraints***.* When using the LLMs that are available on Amazon Bedrock, be aware of the context window limitations:
Claude models (as of September 2024):
+ Claude 3 Opus: 200,000 tokens
+ Claude 3 Sonnet: 200,000 tokens
+ Claude 3 Haiku: 200,000 tokens
Other models on Amazon Bedrock might have different context window sizes. Always check the latest documentation for the latest information.
**Model availability***. *The availability of specific models on Amazon Bedrock can vary. Make sure that you have access to the required models before you implement this solution.
+ **Additional limitations**
+ **Query complexity**. The effectiveness of the natural language to query DSL conversion might vary depending on the complexity of the input query.
+ **Version compatibility**. The generated query DSL might need adjustments based on the specific version of Elasticsearch or OpenSearch that you use.
+ **Performance**. This pattern provides a PoC implementation, so query generation speed and accuracy might not be optimal for large-scale production use.
+ **Cost**. Using LLMs in Amazon Bedrock incurs costs. Be aware of the pricing structure for your chosen model. For more information, see [Amazon Bedrock pricing](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-pricing.html).
+ **Maintenance**. Regular updates to the prompts and model selection might be necessary to keep up with advancements in LLM technology and changes in query DSL syntax.
**Product versions**
This solution was tested in Amazon OpenSearch Service. If you want to use Elasticsearch, you might have to make some changes to replicate the exact functionality of this pattern.
+ **OpenSearch version compatibility**.** **OpenSearch maintains backward compatibility within major versions. For example:
+ OpenSearch 1.x clients are generally compatible with OpenSearch 1.x clusters.
+ OpenSearch 2.x clients are generally compatible with OpenSearch 2.x clusters.
However, it's always best to use the same minor version for both client and cluster when possible.
+ **OpenSearch API compatibility**.** **OpenSearch maintains API compatibility with Elasticsearch OSS 7.10.2 for most operations. However, some differences exist, especially in newer versions.
+ **OpenSearch upgrade considerations**:
+ Direct downgrades are not supported. Use snapshots for rollback if needed.
+ When you upgrade, check the [compatibility matrix and release notes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-operations.html) for any breaking changes.
**Elasticsearch considerations**
+ **Elasticsearch version**. The major version of Elasticsearch you're using is crucial, because query syntax and features can change between major versions. Currently, the latest stable version is Elasticsearch 8.x. Make sure that your queries are compatible with your specific Elasticsearch version.
+ **Elasticsearch query DSL library version**. If you're using the Elasticsearch query DSL Python library, make sure that its version matches your Elasticsearch version. For example:
+ For Elasticsearch 8.x, use an `elasticsearch-dsl` version that's greater or equal to 8.0.0 but smaller than 9.0.0.
+ For Elasticsearch 7.x, use an `elasticsearch-dsl` version that's greater or equal to 7.0.0 but smaller than 8.0.0.
+ **Client library version**. Whether you're using the official Elasticsearch client or a language-specific client, make sure that it's compatible with your Elasticsearch version.
+ **Query DSL version**. Query DSL evolves with Elasticsearch versions. Some query types or parameters might be deprecated or introduced in different versions.
+ **Mapping version**. The way you define mappings for your indexes and change between versions. Always check the mapping documentation for your specific Elasticsearch version.
+ **Analysis tools versions**. If you're using analyzers, tokenizers, or other text analysis tools, their behavior or availability might change between versions.
## Architecture
**Target architecture**
The following diagram illustrates the architecture for this pattern.
![\[Architecture for translating natural language to query DSL in Amazon Bedrock.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/75296405-2893-4328-9551-9bcc6ec7fd3e/images/ffb1b893-d23c-4e1c-b679-8063b4f85a8a.png)
where:
1. User input and system prompt with few-shot prompting examples. The process begins with a user who provides a natural language query or a request for schema generation.
1. Amazon Bedrock. The input is sent to Amazon Bedrock, which serves as the interface to access the Claude LLM.
1. Claude 3 Sonnet LLM. Amazon Bedrock uses Claude 3 Sonnet from the Claude 3 family of LLMs to process the input. It interprets and generates the appropriate Elasticsearch or OpenSearch query DSL. For schema requests, it generates synthetic Elasticsearch or OpenSearch mappings.
1. Query DSL generation. For natural language queries, the application takes the LLM's output and formats it into a valid Elasticsearch or OpenSearch Service query DSL.
1. Synthetic data generation. The application also takes schemas to create synthetic Elasticsearch or OpenSearch data to be loaded into an OpenSearch Serverless collection for testing.
1. OpenSearch or Elasticsearch. The generated Query DSL is queried against an OpenSearch Serverless collection on all indexes. JSON output contains the relevant data and number of *hits* from the data that resides in the OpenSearch Serverless collection.
**Automation and scale**
The code that's provided with this pattern is built strictly for PoC purposes. The following list provides a few suggestions for automating and scaling the solution further and moving the code to production. These enhancements are outside the scope of this pattern.
+ Containerization:
+ Dockerize the application to ensure consistency across different environments.
+ Use container orchestration platforms such as Amazon Elastic Container Service (Amazon ECS) or Kubernetes for scalable deployments.
+ Serverless architecture:
+ Convert the core functionality into AWS Lambda functions.
+ Use Amazon API Gateway to create RESTful endpoints for the natural language query input.
+ Asynchronous processing:
+ Implement Amazon Simple Queue Service (Amazon SQS) to queue incoming queries.
+ Use AWS Step Functions to orchestrate the workflow of processing queries and generating query DSL.
+ Caching:
+ Implement a mechanism to cache the prompts.
+ Monitoring and logging:
+ Use Amazon CloudWatch for monitoring and alerting.
+ Implement centralized logging with Amazon CloudWatch Logs or Amazon OpenSearch Service for log analytics.
+ Security enhancements:
+ Implement IAM roles for fine-grained access control.
+ Use AWS Secrets Manager to securely store and manage API keys and credentials.
+ Multi-Region deployment:
+ Consider deploying the solution across multiple AWS Regions for improved latency and disaster recovery.
+ Use Amazon Route 53 for intelligent request routing.
By implementing these suggestions, you can transform this PoC into a robust, scalable, and production-ready solution. We recommend that you thoroughly test each component and the entire system before full deployment.
## Tools
**Tools**
+ [Amazon SageMaker AI notebooks](https://aws.amazon.com/sagemaker/notebooks/) are fully managed Jupyter notebooks for machine learning development. This pattern uses notebooks as an interactive environment for data exploration, model development, and experimentation in Amazon SageMaker AI. Notebooks provide seamless integration with other SageMaker AI features and AWS services.
+ [Python](https://www.python.org/) is a general-purpose computer programming language. This pattern uses Python as the core language to implement the solution.
+ [Amazon Bedrock](https://aws.amazon.com/bedrock/) is a fully managed service that makes high-performing foundation models (FMs) from leading AI startups and Amazon available for your use through a unified API. Amazon Bedrock provides access to LLMs for natural language processing. This pattern uses Anthropic Claude 3 models.
+ [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) is a software development kit that helps you integrate your Python application, library, or script with AWS services, including Amazon Bedrock.
+ [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html) is a managed service that helps you deploy, operate, and scale OpenSearch Service clusters in the AWS Cloud. This pattern uses OpenSearch Service as the target system for generating query DSL.
**Code repository**
The code for this pattern is available in the GitHub [Prompt Engineering Text-to-QueryDSL Using Claude 3 Models](https://github.com/aws-samples/text-to-queryDSL/blob/main/text2ES_prompting_guide.ipynb) repository. The example uses a health social media app that creates posts for users and user profiles associated with the health application.
## Best practices
When working with this solution, consider the following:
+ The need for proper AWS credentials and permissions to access Amazon Bedrock
+ Potential costs associated with using AWS services and LLMs
+ The importance of understanding query DSL to validate and potentially modify the generated queries
## Epics
### Set up the environment and prepare data
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the development environment. | For detailed instructions and code for this and the other steps in this pattern, see the comprehensive walkthrough in the [GitHub repository](https://github.com/aws-samples/text-to-queryDSL/blob/main/text2ES_prompting_guide.ipynb).[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/translate-natural-language-query-dsl-opensearch-elasticsearch.html) | Python, pip, AWS SDK |
| Set up AWS access. | Set up the Amazon Bedrock client and SageMaker AI session. Retrieve the Amazon Resource Name (ARN) for the SageMaker AI execution role for later use in creating the OpenSearch Serverless collection. | IAM, AWS CLI, Amazon Bedrock, Amazon SageMaker |
| Load health app schemas. | Read and parse JSON schemas for health posts and user profiles from predefined files. Convert schemas to strings for later use in prompts. | DevOps engineer, General AWS, Python, JSON |
### Generate synthetic data
| Task | Description | Skills required |
| --- | --- | --- |
| Create an LLM-based data generator. | Implement the **generate\$1data()** function to call the Amazon Bedrock Converse API with Claude 3 models. Set up model IDs for Sonnet, Sonnet 3.5, and Haiku:model_id_sonnet3_5 = "anthropic.claude-3-5-sonnet-20240620-v1:0"
model_id_sonnet = "anthropic.claude-3-sonnet-20240229-v1:0"
model_id_haiku = "anthropic.claude-3-haiku-20240307-v1:0"
| Python, Amazon Bedrock API, LLM prompting |
| Create synthetic health posts. | Use the **generate\$1data()** function with a specific message prompt to create synthetic health post entries based on the provided schema. The function call looks like this: health_post_data = generate_data(bedrock_rt, model_id_sonnet, system_prompt, message_healthpost, inference_config)
| Python, JSON |
| Create synthetic user profiles. | Use the **generate\$1data()** function with a specific message prompt to create synthetic user profile entries based on the provided schema. This is similar to health posts generation, but uses a different prompt. | Python, JSON |
### Set up OpenSearch and ingest data
| Task | Description | Skills required |
| --- | --- | --- |
| Set up an OpenSearch Serverless collection. | Use Boto3 to create an OpenSearch Serverless collection with appropriate encryption, network, and access policies. The collection creation looks like this: collection = aoss_client.create_collection(name=es_name, type='SEARCH')
For more information about OpenSearch Serverless, see the [AWS documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless.html). | OpenSearch Serverless, IAM |
| Define OpenSearch indexes. | Create indiexes for health posts and user profiles by using the OpenSearch client, based on the predefined schema mappings. The index creation looks like this:response_health = oss_client.indices.create(healthpost_index, body=healthpost_body)
| OpenSearch, JSON |
| Load data into OpenSearch. | Run the **ingest\$1data()** function to bulk insert the synthetic health posts and user profiles into their respective OpenSearch indexes. The function uses the bulk helper from `opensearch-py`:success, failed = bulk(oss_client, actions)
| Python, OpenSearch API, bulk data operations |
### Generate and run queries
| Task | Description | Skills required |
| --- | --- | --- |
| Design few-shot prompt examples. | Generate example queries and corresponding natural language questions by using Claude 3 models to serve as few-shot examples for query generation. The system prompt includes these examples:system_prompt_query_generation = [{"text": f"""You are an expert query dsl generator. ... Examples: {example_prompt} ..."""}] | LLM prompting, query DSL |
| Create a text-to-query DSL converter. | Implement the system prompt, which includes schemas, data, and few-shot examples, for query generation. Use the system prompt to convert natural language queries to query DSL. The function call looks like this:query_response = generate_data(bedrock_rt, model_id, system_prompt_query_generation, query, inference_config)
| Python, Amazon Bedrock API, LLM prompting |
| Test query DSL on OpenSearch. | Run the **query\$1oss()** function to run the generated query DSL against the OpenSearch Serverless collection and return results. The function uses the OpenSearch client's search method:response = oss_client.search(index="_all", body=temp)
| Python, OpenSearch API, query DSL |
### Test and evaluate
| Task | Description | Skills required |
| --- | --- | --- |
| Create a test query set. | Use Claude 3 to generate a diverse set of test questions based on the synthetic data and schemas:test_queries = generate_data(bedrock_rt, model_id_sonnet, query_system_prompt, query_prompt, inference_config)
| LLM prompting |
| Assess the accuracy of the query DSL conversion. | Test the generated query DSL by running queries against OpenSearch and analyzing the returned results for relevance and accuracy. This involves running the query and inspecting the hits:output = query_oss(response1) print("Response after running query against Opensearch") print(output) | Python, data analysis, query DSL |
| Benchmark Claude 3 models. | Compare the performance of different Claude 3 models (Haiku, Sonnet, Sonnet 3.5) for query generation in terms of accuracy and latency. To compare, change the `model_id` when you call **generate\$1data()** and measure execution time. | Python, performance benchmarking |
### Clean up and document
| Task | Description | Skills required |
| --- | --- | --- |
| Develop a cleanup process. | Delete all indexes from the OpenSearch Serverless collection after use. | Python, AWS SDK, OpenSearch API |
## Related resources
+ [Query DSL](https://opensearch.org/docs/latest/query-dsl/) (OpenSearch documentation)
+ [Amazon OpenSearch Service documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html)
+ [OpenSearch Serverless collections](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-manage.html)
+ [Amazon Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html)
+ [Amazon SageMaker AI documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html)
+ [AWS SDK for Python (Boto3) documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)
## Additional information
**IAM policy**
Here’s the `APGtext2querydslpolicy` policy for the IAM role used in this pattern:
```
{
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"Resource": "*"
},
{ "Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::sagemaker-*",
"arn:aws:s3:::sagemaker-*/*"
]
},
{ "Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:log-group:/aws/sagemaker/*"
},
{ "Effect": "Allow",
"Action": [
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Resource": "*"
},
{ "Effect": "Allow",
"Action": [
"aoss:*"
],
"Resource": "*"
},
{ "Effect": "Allow",
"Action": [
"iam:PassRole",
"sagemaker:*"
],
"Resource": [
"arn:aws:iam::*:role/*", "*"
],
"Condition": {
"StringEquals": {
"iam:PassedToService": "sagemaker.amazonaws.com"
}
}
},
{ "Effect": "Allow",
"Action": [
"codecommit:GetBranch",
"codecommit:GetCommit",
"codecommit:GetRepository",
"codecommit:ListBranches",
"codecommit:ListRepositories"
],
"Resource": "*"
},
{ "Effect": "Allow",
"Action": [
"aws-marketplace:Subscribe"
],
"Resource": "*",
"Condition": {
"ForAnyValue:StringEquals": {
"aws-marketplace:ProductId": [
"prod-6dw3qvchef7zy",
"prod-m5ilt4siql27k",
"prod-ozonys2hmmpeu"
]
}
}
},
{ "Effect": "Allow",
"Action": [
"aws-marketplace:Unsubscribe",
"aws-marketplace:ViewSubscriptions"
],
"Resource": "*"
},
{ "Effect": "Allow",
"Action": "iam:*",
"Resource": "*"
}
]
}
```
**Prompt techniques with Anthropic Claude 3 models**
This pattern demonstrates the following prompting techniques for text-to-query DSL conversion using Claude 3 models.
+ **Few-shot prompting:** Few-shot prompting is a powerful technique for improving the performance of Claude 3 models on Amazon Bedrock. This approach involves providing the model with a small number of examples that demonstrate the desired input/output behavior before asking it to perform a similar task. When you use Claude 3 models on Amazon Bedrock, few-shot prompting can be particularly effective for tasks that require specific formatting, reasoning patterns, or domain knowledge. To implement this technique, you typically structure your prompt with two main components: the example section and the actual query. The example section contains one or more input/output pairs that illustrate the task, and the query section presents the new input for which you want a response. This method helps Claude 3 understand the context and expected output format, and often results in a more accurate and consistent response.
Example:
```
"query": {
"bool": {
"must": [
{"match": {"post_type": "recipe"}},
{"range": {"likes_count": {"gte": 100}}},
{"exists": {"field": "media_urls"}}
]
}
}
Question: Find all recipe posts that have at least 100 likes and include media URLs.
```
+ **System prompts: **In addition to few-shot prompting, Claude 3 models on Amazon Bedrock also support the use of system prompts. System prompts are a way to provide overall context, instructions, or guidelines to the model before presenting it with specific user inputs. They are particularly useful for setting the tone, defining the model's role, or establishing constraints for the entire conversation. To use a system prompt with Claude 3 on Amazon Bedrock, you include it in the `system` parameter of your API request. This is separate from the user messages and applies to the entire interaction. Detailed system prompts are used to set context and provide guidelines for the model.
Example:
```
You are an expert query dsl generator. Your task is to take an input question and generate a query dsl to answer the question. Use the schemas and data below to generate the query.
Schemas: [schema details]
Data: [sample data]
Guidelines:
- Ensure the generated query adheres to DSL query syntax
- Do not create new mappings or other items that aren't included in the provided schemas.
```
+ **Structured output**: You can instruct the model to provide output in specific formats, such as JSON or within XML tags.
Example:
```
Put the query in json tags
```
+ **Prompt chaining**: The notebook uses the output of one LLM call as input for another, such as using generated synthetic data to create example questions.
+ **Context provision**: Relevant context, including schemas and sample data, is provided in the prompts.
Example:
```
Schemas: [schema details]
Data: [sample data]
```
+ **Task-specific prompts**: Different prompts are crafted for specific tasks, such as generating synthetic data, creating example questions, and converting natural language queries to query DSL.
Example for generating test questions:
```
Your task is to generate 5 example questions users can ask the health app based on provided schemas and data. Only include the questions generated in the response.
```
# Use Amazon Q Developer as a coding assistant to increase your productivity
*Ram Kandaswamy, Amazon Web Services*
## Summary
This pattern uses a tic-tac-toe game to demonstrate how you can apply Amazon Q Developer across a range of development tasks. It generates code for a tic-tac-toe game as a single-page application (SPA), enhances its UI, and creates scripts to deploy the application on AWS.
Amazon Q Developer functions as a coding assistant to help accelerate software development workflows and enhance productivity for both developers and non-developers. Regardless of your technical expertise, it helps you create architectures and design solutions for business problems, bootstraps your working environment, helps you implement new features, and generates test cases for validation. It uses natural language instructions and AI capabilities to ensure consistent, high-quality code and to mitigate coding challenges regardless of your programming skills.
The key advantage of Amazon Q Developer is its ability to liberate you from repetitive coding tasks. When you use the `@workspace` annotation, Amazon Q Developer ingests and indexes all code files, configurations, and project structure in your integrated development environment (IDE), and provides tailored responses to help you focus on creative problem-solving. You can dedicate more time to designing innovative solutions and enhancing the user experience. If you aren't technical, you can use Amazon Q Developer to streamline workflows and collaborate more effectively with the development team. The Amazon Q Developer **Explain code** feature offers detailed instructions and summaries, so you can navigate complex code bases.
Furthermore, Amazon Q Developer provides a language-agnostic approach that helps junior and mid-level developers expand their skill sets. You can concentrate on core concepts and business logic instead of language-specific syntax. This reduces the learning curve when you switch technologies.
## Prerequisites and limitations
**Prerequisites**
+ IDE (for example, WebStorm or Visual Studio Code) with the Amazon Q Developer plugin installed. For instructions, see [Installing the Amazon Q Developer extension or plugin in your IDE](https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/q-in-IDE-setup.html) in the Amazon Q Developer documentation.
+ An active AWS account set up with Amazon Q Developer. For instructions, see [Getting started](https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/getting-started-q-dev.html) in the Amazon Q Developer documentation.
+ **npm** installed. For instructions, see the [npm documentation](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm). This pattern was tested with npm version 10.8.
+ AWS Command Line Interface (AWS CLI) installed. For instructions, see the [AWS CLI documentation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
**Limitations**
+ Amazon Q Developer can perform only one development task at a time.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service.
## Tools
+ This pattern requires an IDE such as Visual Studio Code or WebStorm. For a list of supported IDEs, see the [Amazon Q Developer documentation](https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/q-in-IDE.html#supported-ides-features).
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command-line shell.
## Best practices
See [Best coding practices with Amazon Q Developer](https://docs.aws.amazon.com/prescriptive-guidance/latest/best-practices-code-generation/best-practices-coding.html) in AWS Prescriptive Guidance. In addition:
+ When you provide prompts to Amazon Q Developer, make sure that your instructions are clear and unambiguous. Add code snippets and annotations such as `@workspace` to the prompt to provide more context for your prompts.
+ Include relevant libraries and import them to avoid conflicts or incorrect guesses by the system.
+ If the code generated isn't accurate or as expected, use the **Provide feedback & regenerate** option. Try breaking the prompts into smaller instructions.
## Epics
### Set up the working environment
| Task | Description | Skills required |
| --- | --- | --- |
| Create a new project. | To create a new project in your working environment, run the following command and accept the default settings for all questions:npx create-next-app@latest
| App developer, Programmer, Software developer |
| Test the base application. | Run the following command and confirm that the base application loads successfully in the browser:npm run dev
| App developer, Programmer, Software developer |
| Clean up the base code. | Navigate to the `page.tsx` file in the `src/app` folder and delete the default content to get a blank page. After deletion, the file should look like this:export default function Home() {
return (
);
} | App developer, Programmer, Software developer |
### Use Amazon Q Developer to design a tic-tac-toe game project
| Task | Description | Skills required |
| --- | --- | --- |
| Get an overview of steps. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/use-q-developer-as-coding-assistant-to-increase-productivity.html) | App developer, Programmer, Software developer |
| Generate code for tic-tac-toe. | In the chat panel, start a development task by using the `/dev` command followed by the description of the task. For example:/dev Create a React-based single-page application written in TypeScript for a tic-tac-toe game with the following specifications:
1. Design an aesthetically pleasing interface with the game grid centered vertically and
horizontally on the page.
2. Include a heading and clear instructions on how to play the game.
3. Implement color-coding for X and O marks to distinguish them easily.
Amazon Q Developer generates code based on your instructions. | App developer, Programmer, Software developer |
| Inspect and accept the generated code. | Visually inspect the code, and choose **Accept code** to automatically replace the `page.tsx` file.If you run into issues, choose **Provide feedback & regenerate** and describe the issue you encountered. | App developer, Programmer, Software developer |
| Fix lint errors. | The example tic-tac-toe game includes a grid. The code that Amazon Q Developer generates might use the default type `any`. You can add type safety by prompting Amazon Q Developer as follows:/dev Ensure proper TypeScript typing for the onSquare Click event handler
to resolve any 'any' type issues.
| App developer, Programmer, Software developer |
| Add visual appeal. | You can break the original requirement into smaller fragments. For example, you can improve the game UI with the following prompts in the dev tasks. This prompt enhances Cascading Style Sheets (CSS) styles and exports the app for deployment./dev Debug and fix any CSS issues to correctly display the game grid and overall layout.
Simplify the code by removing game history functionality and related components.
Implement static file export to an 'out' directory for easy deployment. The solution
should be fully functional, visually appealing, and free of typing errors or layout issues.
| App developer, Programmer, Software developer |
| Test again. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/use-q-developer-as-coding-assistant-to-increase-productivity.html) | App developer, Programmer, Software developer |
### Deploy the application to the AWS Cloud
| Task | Description | Skills required |
| --- | --- | --- |
| Create folders and files for deployment. | In the project in your working environment, create a deployment folder and two files inside it: `pushtos3.sh` and `cloudformation.yml`:mkdir deployment && cd deployment
touch pushtos3.sh && chmod +x pushtos3.sh
touch cloudformation.yml
| App developer, Programmer, Software developer |
| Generate automation code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/use-q-developer-as-coding-assistant-to-increase-productivity.html) | AWS administrator, AWS DevOps, App developer |
| Generate script content. | To create a deployment script, use the following prompt:/dev Modify the pushtos3 shell script so that it can use AWS CLI commands to create a
CloudFormation stack named tictactoe-stack if it does not exist already, and use
cloudformation.yml as the source template. Wait for the stack to complete and sync the
contents from the out folder to the S3 bucket. Perform invalidation of the CloudFront
origin.
| App developer, Programmer, Software developer |
| Deploy the application to the AWS Cloud. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/use-q-developer-as-coding-assistant-to-increase-productivity.html) | AWS administrator, AWS DevOps, Cloud architect, App developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| The build doesn't create a single-page application or export it to the output folder. | Look at the contents of the `next.config.mjs` file.If the code has the following default configuration:const nextConfig = {};modify it as follows:const nextConfig = {
output: 'export',
distDir: 'out',
}; |
## Related resources
+ [Creating a new React project](https://react.dev/learn/start-a-new-react-project) (React documentation)
+ [Amazon Q Developer overview](https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/what-is.html) (AWS documentation)
+ [Amazon Q Developer best practices ](https://docs.aws.amazon.com/prescriptive-guidance/latest/best-practices-code-generation/introduction.html)(AWS Prescriptive Guidance)
+ [Installing, configuring, and using Amazon Q Developer with JetBrains IDEs](https://www.youtube.com/watch?v=-iQfIhTA4J0&pp=ygUSYW1hem9uIHEgZGV2ZWxvcGVy) (YouTube video)
+ [Installing Amazon Q for the command line](https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/command-line-getting-started-installing.html) (AWS documentation)
# Use SageMaker Processing for distributed feature engineering of terabyte-scale ML datasets
*Chris Boomhower, Amazon Web Services*
## Summary
Many terabyte-scale or larger datasets often consist of a hierarchical folder structure, and the files in the dataset sometimes share interdependencies. For this reason, machine learning (ML) engineers and data scientists must make thoughtful design decisions to prepare such data for model training and inference. This pattern demonstrates how you can use manual macrosharding and microsharding techniques in combination with Amazon SageMaker Processing and virtual CPU (vCPU) parallelization to efficiently scale feature engineering processes for complicated big data ML datasets.
This pattern defines *macrosharding* as the splitting of data directories across multiple machines for processing, and *microsharding* as the splitting of data on each machine across multiple processing threads. The pattern demonstrates these techniques by using Amazon SageMaker with sample time-series waveform records from the [PhysioNet MIMIC-III](https://physionet.org/content/mimic3wdb/1.0/) dataset. By implementing the techniques in this pattern, you can minimize the processing time and costs for feature engineering while maximizing resource utilization and throughput efficiency. These optimizations rely on distributed SageMaker Processing on Amazon Elastic Compute Cloud (Amazon EC2) instances and vCPUs for similar, large datasets, regardless of data type.
## Prerequisites and limitations
**Prerequisites**
+ Access to SageMaker notebook instances or SageMaker Studio, if you want to implement this pattern for your own dataset. If you are using Amazon SageMaker for the first time, see [Get started with Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/gs.html) in the AWS documentation.
+ SageMaker Studio, if you want to implement this pattern with the [PhysioNet MIMIC-III](https://physionet.org/content/mimic3wdb/1.0/) sample data.
+ The pattern uses SageMaker Processing, but doesn’t require any experience running SageMaker Processing jobs.
**Limitations**
+ This pattern is well suited to ML datasets that include interdependent files. These interdependencies benefit the most from manual macrosharding and running multiple, single-instance SageMaker Processing jobs in parallel. For datasets where such interdependencies do not exist, the `ShardedByS3Key` feature in SageMaker Processing might be a better alternative to macrosharding, because it sends sharded data to multiple instances that are managed by the same Processing job. However, you can implement this pattern’s microsharding strategy in both scenarios to best utilize instance vCPUs.
**Product versions**
+ Amazon SageMaker Python SDK version 2
## Architecture
**Target technology stack**
+ Amazon Simple Storage Service (Amazon S3)
+ Amazon SageMaker
**Target architecture**
*Macrosharding and distributed EC2 instances*
The 10 parallel processes represented in this architecture reflect the structure of the MIMIC-III dataset. (Processes are represented by ellipses for diagram simplification.) A similar architecture applies to any dataset when you use manual macrosharding. In the case of MIMIC-III, you can use the dataset's raw structure to your advantage by processing each patient group folder separately, with minimal effort. In the following diagram, the record groups block appears on the left (1). Given the distributed nature of the data, it makes sense to shard by patient group.
![\[Architecture for microsharding and distributed EC2 instances\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e7a90b31-de8f-41fd-bb3f-c7c6100fc306/images/c19a8f87-ac59-458e-89cb-50be17ca4a0c.png)
However, manually sharding by patient group means that a separate Processing job is required for each patient group folder, as you can see in the middle section of the diagram (2), instead of a single Processing job with multiple EC2 instances. Because MIMIC-III's data includes both binary waveform files and matching text-based header files, and there is a required dependency on the [wfdb library](https://wfdb.readthedocs.io/en/latest/) for binary data extraction, all the records for a specific patient must be made available on the same instance. The only way to be certain that each binary waveform file's associated header file is also present is to implement manual sharding to run each shard within its own Processing job, and to specify `s3_data_distribution_type='FullyReplicated'` when you define the Processing job input. Alternatively, if all data were available in a single directory and no dependencies existed between files, a more suitable option might be to launch a single Processing job with multiple EC2 instances and `s3_data_distribution_type='ShardedByS3Key'` specified. Specifying `ShardedByS3Key `as the Amazon S3 data distribution type directs SageMaker to manage data sharding automatically across instances.
Launching a Processing job for each folder is a cost-efficient way to preprocess the data, because running multiple instances concurrently saves time. For additional cost and time savings, you can use microsharding within each Processing job.
*Microsharding and parallel vCPUs*
Within each Processing job, the grouped data is further divided to maximize use of all available vCPUs on the SageMaker fully managed EC2 instance. The blocks in the middle section of the diagram (2) depict what happens within each primary Processing job. The contents of the patient record folders are flattened and divided evenly based on the number of available vCPUs on the instance. After the folder contents are divided, the evenly sized set of files are distributed across all vCPUs for processing. When processing is complete, the results from each vCPU are combined into a single data file for each Processing job.
In the attached code, these concepts are represented in the following section of the `src/feature-engineering-pass1/preprocessing.py` file.
```
def chunks(lst, n):
"""
Yield successive n-sized chunks from lst.
:param lst: list of elements to be divided
:param n: number of elements per chunk
:type lst: list
:type n: int
:return: generator comprising evenly sized chunks
:rtype: class 'generator'
"""
for i in range(0, len(lst), n):
yield lst[i:i + n]
# Generate list of data files on machine
data_dir = input_dir
d_subs = next(os.walk(os.path.join(data_dir, '.')))[1]
file_list = []
for ds in d_subs:
file_list.extend(os.listdir(os.path.join(data_dir, ds, '.')))
dat_list = [os.path.join(re.split('_|\.', f)[0].replace('n', ''), f[:-4]) for f in file_list if f[-4:] == '.dat']
# Split list of files into sub-lists
cpu_count = multiprocessing.cpu_count()
splits = int(len(dat_list) / cpu_count)
if splits == 0: splits = 1
dat_chunks = list(chunks(dat_list, splits))
# Parallelize processing of sub-lists across CPUs
ws_df_list = Parallel(n_jobs=-1, verbose=0)(delayed(run_process)(dc) for dc in dat_chunks)
# Compile and pickle patient group dataframe
ws_df_group = pd.concat(ws_df_list)
ws_df_group = ws_df_group.reset_index().rename(columns={'index': 'signal'})
ws_df_group.to_json(os.path.join(output_dir, group_data_out))
```
A function, `chunks`, is first defined to consume a given list by dividing it into evenly sized chunks of length `n `and by returning these results as a generator. Next, the data is flattened across patient folders by compiling a list of all binary waveform files that are present. After this is done, the number of vCPUs available on the EC2 instance is obtained. The list of binary waveform files is evenly divided across these vCPUs by calling `chunks`, and then each waveform sublist is processed on its own vCPU by using [joblib's Parallel class](https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html). Results are automatically combined into a single list of dataframes by the Processing job, which SageMaker then processes further before writing it to Amazon S3 upon job completion. In this example, there are 10 files written to Amazon S3 by the Processing jobs (one for each job).
When all the initial Processing jobs are complete, a secondary Processing job, which is shown in the block to the right of the diagram (3) combines the output files produced by each primary Processing job and writes the combined output to Amazon S3 (4).
## Tools
**Tools**
+ [Python](https://www.python.org/) – The sample code used for this pattern is Python (version 3).
+ [SageMaker Studio](https://docs.aws.amazon.com/sagemaker/latest/dg/studio.html) – Amazon SageMaker Studio is a web-based, integrated development environment (IDE) for machine learning that lets you build, train, debug, deploy, and monitor your machine learning models. You run SageMaker Processing jobs by using Jupyter notebooks inside SageMaker Studio.
+ [SageMaker Processing](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job.html) – Amazon SageMaker Processing provides a simplified way to run your data processing workloads. In this pattern, the feature engineering code is implemented at scale by using SageMaker Processing jobs.
**Code**
The attached .zip file provides the complete code for this pattern. The following section describes the steps to build the architecture for this pattern. Each step is illustrated by sample code from the attachment.
## Epics
### Set up your SageMaker Studio environment
| Task | Description | Skills required |
| --- | --- | --- |
| Access Amazon SageMaker Studio. | Onboard to SageMaker Studio in your AWS account by following the directions provided in the [Amazon SageMaker documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/onboard-quick-start.html). | Data scientist, ML engineer |
| Install the wget utility. | Install *wget* if you onboarded with a new SageMaker Studio configuration or if you've never used these utilities in SageMaker Studio before. To install, open a terminal window in the SageMaker Studio console and run the following command:sudo yum install wget
| Data scientist, ML engineer |
| Download and unzip the sample code. | Download the `attachments.zip` file in the *Attachments* section. In a terminal window, navigate to the folder where you downloaded the file and extract its contents:unzip attachment.zip
Navigate to the folder where you extracted the .zip file, and extract the contents of the `Scaled-Processing.zip` file.unzip Scaled-Processing.zip
| Data scientist, ML engineer |
| Download the sample dataset from physionet.org and upload it to Amazon S3. | Run the `get_data.ipynb` Jupyter notebook within the folder that contains the `Scaled-Processing` files. This notebook downloads a sample MIMIC-III dataset from [physionet.org](https://physionet.org) and uploads it to your SageMaker Studio session bucket in Amazon S3. | Data scientist, ML engineer |
### Configure the first preprocessing script
| Task | Description | Skills required |
| --- | --- | --- |
| Flatten the file hierarchy across all subdirectories. | In large datasets such as MIMIC-III, files are often distributed across multiple subdirectories even within a logical parent group. Your script should be configured to flatten all group files across all subdirectories, as the following code demonstrates.# Generate list of .dat files on machine
data_dir = input_dir
d_subs = next(os.walk(os.path.join(data_dir, '.')))[1]
file_list = []
for ds in d_subs:
file_list.extend(os.listdir(os.path.join(data_dir, ds, '.')))
dat_list = [os.path.join(re.split('_|\.', f)[0].replace('n', ''), f[:-4]) for f in file_list if f[-4:] == '.dat']
The example code snippets in this epic are from the `src/feature-engineering-pass1/preprocessing.py` file, which is provided in the attachment. | Data scientist, ML engineer |
| Divide files into subgroups based on vCPU count. | Files should be divided into evenly sized subgroups, or chunks, depending on the number of vCPUs present on the instance that runs the script. For this step, you can implement code similar to the following.# Split list of files into sub-lists
cpu_count = multiprocessing.cpu_count()
splits = int(len(dat_list) / cpu_count)
if splits == 0: splits = 1
dat_chunks = list(chunks(dat_list, splits))
| Data scientist, ML engineer |
| Parallelize processing of subgroups across vCPUs. | Script logic should be configured to process all subgroups in parallel. To do this, use the Joblib library's `Parallel `class and `delayed `method as follows. # Parallelize processing of sub-lists across CPUs
ws_df_list = Parallel(n_jobs=-1, verbose=0)(delayed(run_process)(dc) for dc in dat_chunks)
| Data scientist, ML engineer |
| Save single file group output to Amazon S3. | When parallel vCPU processing is complete, the results from each vCPU should be combined and uploaded to the file group's S3 bucket path. For this step, you can use code similar to the following.# Compile and pickle patient group dataframe
ws_df_group = pd.concat(ws_df_list)
ws_df_group = ws_df_group.reset_index().rename(columns={'index': 'signal'})
ws_df_group.to_json(os.path.join(output_dir, group_data_out))
| Data scientist, ML engineer |
### Configure the second preprocessing script
| Task | Description | Skills required |
| --- | --- | --- |
| Combine data files produced across all Processing jobs that ran the first script. | The previous script outputs a single file for each SageMaker Processing job that processes a group of files from the dataset. Next, you need to combine these output files into a single object and write a single output dataset to Amazon S3. This is demonstrated in the `src/feature-engineering-pass1p5/preprocessing.py` file, which is provided in the attachment, as follows.def write_parquet(wavs_df, path):
"""
Write waveform summary dataframe to S3 in parquet format.
:param wavs_df: waveform summary dataframe
:param path: S3 directory prefix
:type wavs_df: pandas dataframe
:type path: str
:return: None
"""
extra_args = {"ServerSideEncryption": "aws:kms"}
wr.s3.to_parquet(
df=wavs_df,
path=path,
compression='snappy',
s3_additional_kwargs=extra_args)
def combine_data():
"""
Get combined data and write to parquet.
:return: waveform summary dataframe
:rtype: pandas dataframe
"""
wavs_df = get_data()
wavs_df = normalize_signal_names(wavs_df)
write_parquet(wavs_df, "s3://{}/{}/{}".format(bucket_xform, dataset_prefix, pass1p5out_data))
return wavs_df
wavs_df = combine_data()
| Data scientist, ML engineer |
### Run Processing jobs
| Task | Description | Skills required |
| --- | --- | --- |
| Run the first Processing job. | To perform macrosharding, run a separate Processing job for each file group. Microsharding is performed inside each Processing job, because each job runs your first script. The following code demonstrates how to launch a Processing job for each file group directory in the following snippet (included in `notebooks/FeatExtract_Pass1.ipynb`).pat_groups = list(range(30,40))
ts = str(int(time.time()))
for group in pat_groups:
sklearn_processor = SKLearnProcessor(framework_version='0.20.0',
role=role,
instance_type='ml.m5.4xlarge',
instance_count=1,
volume_size_in_gb=5)
sklearn_processor.run(
code='../src/feature-engineering-pass1/preprocessing.py',
job_name='-'.join(['scaled-processing-p1', str(group), ts]),
arguments=[
"input_path", "/opt/ml/processing/input",
"output_path", "/opt/ml/processing/output",
"group_data_out", "ws_df_group.json"
],
inputs=
[
ProcessingInput(
source=f's3://{sess.default_bucket()}/data_inputs/{group}',
destination='/opt/ml/processing/input',
s3_data_distribution_type='FullyReplicated'
)
],
outputs=
[
ProcessingOutput(
source='/opt/ml/processing/output',
destination=f's3://{sess.default_bucket()}/data_outputs/{group}'
)
],
wait=False
)
| Data scientist, ML engineer |
| Run the second Processing job. | To combine the outputs generated by the first set of processing jobs and perform any additional computations for preprocessing, you run your second script by using a single SageMaker Processing job. The following code demonstrates this (included in `notebooks/FeatExtract_Pass1p5.ipynb`).ts = str(int(time.time()))
bucket = sess.default_bucket()
sklearn_processor = SKLearnProcessor(framework_version='0.20.0',
role=role,
instance_type='ml.t3.2xlarge',
instance_count=1,
volume_size_in_gb=5)
sklearn_processor.run(
code='../src/feature-engineering-pass1p5/preprocessing.py',
job_name='-'.join(['scaled-processing', 'p1p5', ts]),
arguments=['bucket', bucket,
'pass1out_prefix', 'data_outputs',
'pass1out_data', 'ws_df_group.json',
'pass1p5out_data', 'waveform_summary.parquet',
'statsdata_name', 'signal_stats.csv'],
wait=True
)
| Data scientist, ML engineer |
## Related resources
+ [Onboard to Amazon SageMaker Studio Using Quick Start](https://docs.aws.amazon.com/sagemaker/latest/dg/onboard-quick-start.html) (SageMaker documentation)
+ [Process Data](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job.html) (SageMaker documentation)
+ [Data Processing with scikit-learn](https://docs.aws.amazon.com/sagemaker/latest/dg/use-scikit-learn-processing-container.html) (SageMaker documentation)
+ [joblib.Parallel documentation](https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html)
+ Moody, B., Moody, G., Villarroel, M., Clifford, G. D., & Silva, I. (2020). [MIMIC-III Waveform Database](https://doi.org/10.13026/c2607m) (version 1.0). *PhysioNet*.
+ Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., & Mark, R. G. (2016). [MIMIC-III, a freely accessible critical care database](https://dx.doi.org/10.1038/sdata.2016.35). Scientific Data, 3, 160035.
+ [MIMIC-III Waveform Database license](https://physionet.org/content/mimic3wdb/1.0/LICENSE.txt)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/e7a90b31-de8f-41fd-bb3f-c7c6100fc306/attachments/attachment.zip)
# Visualize AI/ML model results using Flask and AWS Elastic Beanstalk
*Chris Caudill and Durga Sury, Amazon Web Services*
## Summary
Visualizing output from artificial intelligence and machine learning (AI/ML) services often requires complex API calls that must be customized by your developers and engineers. This can be a drawback if your analysts want to quickly explore a new dataset.
You can enhance the accessibility of your services and provide a more interactive form of data analysis by using a web-based user interface (UI) that enables users to upload their own data and visualize the model results in a dashboard.
This pattern uses [Flask](https://flask.palletsprojects.com/en/stable/) and [Plotly](https://plotly.com/) to integrate Amazon Comprehend with a custom web application and visualize sentiments and entities from user-provided data. The pattern also provides the steps to deploy an application by using AWS Elastic Beanstalk. You can adapt the application by using [AWS AI services](https://aws.amazon.com/machine-learning/ai-services/) or with a custom trained model hosted on an endpoint (for example, an [Amazon SageMaker endpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html)).
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ AWS Command Line Interface (AWS CLI), installed and configured on your local machine. For more information about this, see [Configuration basics](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) in the AWS CLI documentation. You can also use an AWS Cloud9 integrated development environment (IDE); for more information about this, see [Python tutorial for AWS Cloud9](https://docs.aws.amazon.com/cloud9/latest/user-guide/sample-python.html) and [Previewing running applications in the AWS Cloud9 IDE](https://docs.aws.amazon.com/cloud9/latest/user-guide/app-preview.html) in the AWS Cloud9 documentation.
**Notice**: AWS Cloud9 is no longer available to new customers. Existing customers of AWS Cloud9 can continue to use the service as normal. [Learn more](https://aws.amazon.com/blogs/devops/how-to-migrate-from-aws-cloud9-to-aws-ide-toolkits-or-aws-cloudshell/)
+ An understanding of Flask’s web application framework. For more information about Flask, see the [Quickstart](https://flask.palletsprojects.com/en/stable/quickstart/) in the Flask documentation.
+ Python version 3.6 or later, installed and configured. You can install Python by following the instructions from [Setting up your Python development environment](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/python-development-environment.html) in the AWS Elastic Beanstalk documentation.
+ Elastic Beanstalk Command Line Interface (EB CLI), installed and configured. For more information about this, see [Install the EB CLI](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html) and [Configure the EB CLI](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-configuration.html) from the Elastic Beanstalk documentation.
**Limitations**
+ This pattern’s Flask application is designed to work with .csv files that use a single text column and are restricted to 200 rows. The application code can be adapted to handle other file types and data volumes.
+ The application doesn’t consider data retention and continues to aggregate uploaded user files until they are manually deleted. You can integrate the application with Amazon Simple Storage Service (Amazon S3) for persistent object storage or use a database such as Amazon DynamoDB for serverless key-value storage.
+ The application only considers documents in the English language. However, you can use Amazon Comprehend to detect a document’s primary language. For more information about the supported languages for each action, see [API reference](https://docs.aws.amazon.com/comprehend/latest/dg/API_Reference.html) in the Amazon Comprehend documentation.
## Architecture
**Flask application architecture**
Flask is a lightweight framework for developing web applications in Python. It is designed to combine Python’s powerful data processing with a rich web UI. The pattern’s Flask application shows you how to build a web application that enables users to upload data, sends the data to Amazon Comprehend for inference, and then visualizes the results. The application has the following structure:
+ `static` – Contains all the static files that support the web UI (for example, JavaScript, CSS, and images)
+ `templates` – Contains all of the application's HTML pages
+ `userData` – Stores uploaded user data
+ `application.py` – The Flask application file
+ `comprehend_helper.py` – Functions to make API calls to Amazon Comprehend
+ `config.py` – The application configuration file
+ `requirements.txt` – The Python dependencies required by the application
The `application.py` script contains the web application's core functionality, which consists of four Flask routes. The following diagram shows these Flask routes.
![\[The four Flask routes that make up the web application's core functionality.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/03d80cf1-ec97-43f7-adb5-2746a9ec70e6/images/9ca6bad1-26e2-4262-98d0-d54c172336bf.png)
+ `/` is the application's root and directs users to the `upload.html` page (stored in the `templates` directory).
+ `/saveFile` is a route that is invoked after a user uploads a file. This route receives a `POST` request via an HTML form, which contains the file uploaded by the user. The file is saved in the `userData` directory and the route redirects users to the `/dashboard` route.
+ `/dashboard` sends users to the `dashboard.html` page. Within this page's HTML, it runs the JavaScript code in `static/js/core.js` that reads data from the `/data` route and then builds visualizations for the page.
+ `/data` is a JSON API that presents the data to be visualized in the dashboard. This route reads the user-provided data and uses the functions in `comprehend_helper.py` to send the user data to Amazon Comprehend for sentiment analysis and named entity recognition (NER). The Amazon Comprehend response is formatted and returned as a JSON object.
**Deployment architecture**
![\[Architecture diagram for using Flask and Elastic Beanstalk to visualize AI/ML model results.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/03d80cf1-ec97-43f7-adb5-2746a9ec70e6/images/d691bfd2-e2ec-4830-8bff-ffa1e3a95c4a.png)
[Design considerations](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.concepts.design.html)
For more information about design considerations for applications deployed using Elastic Beanstalk on the AWS Cloud, see in the AWS Elastic Beanstalk documentation.
**Technology stack**
+ Amazon Comprehend
+ Elastic Beanstalk
+ Flask
**Automation and scale**
Elastic Beanstalk deployments are automatically set up with load balancers and auto scaling groups. For more configuration options, see [Configuring Elastic Beanstalk environments](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers.html) in the Elastic Beanstalk documentation.
## Tools
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is a unified tool that provides a consistent interface for interacting with all parts of AWS.
+ [Amazon Comprehend](https://docs.aws.amazon.com/comprehend/latest/dg/comprehend-general.html) uses natural language processing (NLP) to extract insights about the content of documents without requiring special preprocessing.
+ [AWS Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html) helps you quickly deploy and manage applications in the AWS Cloud without having to learn about the infrastructure that runs those applications.
+ [Elastic Beanstalk CLI (EB CLI)](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3.html) is a command line interface for AWS Elastic Beanstalk that provides interactive commands to simplify creating, updating, and monitoring environments from a local repository.
+ The [Flask](https://flask.palletsprojects.com/en/stable/) framework performs data processing and API calls using Python and offers interactive web visualization with Plotly.
**Code repository**
The code for this pattern is available in the GitHub [Visualize AI/ML model results using Flask and AWS Elastic Beanstalk](https://github.com/aws-samples/aws-comprehend-elasticbeanstalk-for-flask) repository.
## Epics
### Set up the Flask application
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the GitHub repository. | Pull the application code from the GitHub [Visualize AI/ML model results using Flask and AWS Elastic Beanstalk](https://github.com/aws-samples/aws-comprehend-elasticbeanstalk-for-flask) repository by running the following command:`git clone git@github.com:aws-samples/aws-comprehend-elasticbeanstalk-for-flask.git`Make sure that you configure your SSH keys with GitHub. | Developer |
| Install the Python modules. | After you clone the repository, a new local `aws-comprehend-elasticbeanstalk-for-flask` directory is created. In that directory, the `requirements.txt` file contains the Python modules and versions that run the application. Use the following commands to install the modules:`cd aws-comprehend-elasticbeanstalk-for-flask``pip install -r requirements.txt` | Python developer |
| Test the application locally. | Start the Flask server by running the following command:`python application.py`This returns information about the running server. You should be able to access the application by opening a browser and visiting http://localhost:5000If you're running the application in an AWS Cloud9 IDE, you need to replace the `application.run()` command in the `application.py` file with the following line:`application.run(host=os.getenv('IP', '0.0.0.0'),port=int(os.getenv('PORT', 8080)))`You must revert this change before deployment. | Python developer |
### Deploy the application to Elastic Beanstalk
| Task | Description | Skills required |
| --- | --- | --- |
| Launch the Elastic Beanstalk application. | To launch your project as an Elastic Beanstalk application, run the following command from your application’s root directory:`eb init -p python-3.6 comprehend_flask --region us-east-1` [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/visualize-ai-ml-model-results-using-flask-and-aws-elastic-beanstalk.html)Run the `eb init -i` command for more deployment configuration options. | Architect, Developer |
| Deploy the Elastic Beanstalk environment. | Run the following command from the application's root directory:`eb create comprehend-flask-env``comprehend-flask-env` is the name of the Elastic Beanstalk environment and can be changed according to your requirements. The name can only contain letters, numbers, and dashes. | Architect, Developer |
| Authorize your deployment to use Amazon Comprehend. | Although your application might be successfully deployed, you should also provide your deployment with access to Amazon Comprehend. `ComprehendFullAccess` is an AWS managed policy that provides the deployed application with permissions to make API calls to Amazon Comprehend.Attach the `ComprehendFullAccess` policy to `aws-elasticbeanstalk-ec2-role` (this role is automatically created for your deployment’s Amazon Elastic Compute Cloud (Amazon EC2) instances) by running the following command:`aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/ComprehendFullAccess --role-name aws-elasticbeanstalk-ec2-role``aws-elasticbeanstalk-ec2-role` is created when your application deploys. You must complete the deployment process before you can attach the AWS Identity and Access Management (IAM) policy. | Developer, Security architect |
| Visit your deployed application. | After your application successfully deploys, you can visit it by running the `eb open` command.You can also run the `eb status` command to receive details about your deployment. The deployment URL is listed under `CNAME`. | Architect, Developer |
### (Optional) Customize the application to your ML model
| Task | Description | Skills required |
| --- | --- | --- |
| Authorize Elastic Beanstalk to access the new model. | Make sure that Elastic Beanstalk has the required access permissions for your new model endpoint. For example, if you use an Amazon SageMaker AI endpoint, your deployment needs to have permission to invoke the endpoint. For more information about this, see [InvokeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html) in the Amazon SageMaker AI documentation. | Developer, Security architect |
| Send the user data to a new model. | To change the underlying ML model in this application, you must change the following files:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/visualize-ai-ml-model-results-using-flask-and-aws-elastic-beanstalk.html) | Data scientist |
| Update the dashboard visualizations. | Typically, incorporating a new ML model means that visualizations must be updated to reflect the new results. These changes are made in the following files:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/visualize-ai-ml-model-results-using-flask-and-aws-elastic-beanstalk.html) | Web developer |
### (Optional) Deploy the updated application
| Task | Description | Skills required |
| --- | --- | --- |
| Update your application's requirements file. | Before sending changes to Elastic Beanstalk, update the `requirements.txt` file to reflect any new Python modules by running the following command in your application's root directory:`pip freeze > requirements.txt` | Python developer |
| Redeploy the Elastic Beanstalk environment. | To ensure that your application changes are reflected in your Elastic Beanstalk deployment, navigate to your application's root directory and run the following command:`eb deploy`This sends the most recent version of the application's code to your existing Elastic Beanstalk deployment. | Systems administrator, Architect |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| `Unable to assume role "arn:aws:iam::xxxxxxxxxx:role/aws-elasticbeanstalk-ec2-role". Verify that the role exists and is configured correctly.` | If this error occurs when you run `eb create`, create a sample application on the Elastic Beanstalk console to create the default instance profile. For more information about this, see [Creating an Elastic Beanstalk environment](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.environments.html) in the AWS Elastic Beanstalk documentation. |
| `Your WSGIPath refers to a file that does not exist.` | This error occurs in deployment logs because Elastic Beanstalk expects the Flask code to be named `application.py`. If you chose a different name, run `eb config` and edit the WSGIPath as shown in the following code sample:aws:elasticbeanstalk:container:python:
NumProcesses: '1'
NumThreads: '15'
StaticFiles: /static/=static/
WSGIPath: application.py
Make sure that you replace `application.py` with your file name.You can also leverage Gunicorn and a Procfile. For more information about this approach, see [Configuring the WSGI server with a Procfile](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/python-configuration-procfile.html) in the AWS Elastic Beanstalk documentation. |
| `Target WSGI script '/opt/python/current/app/application.py' does not contain WSGI application 'application'.` | Elastic Beanstalk expects the variable that represents your Flask application to be named `application`. Make sure that the `application.py` file uses `application` as the variable name:application = Flask(__name__)
|
| `The EB CLI cannot find your SSH key file for keyname` | Use the EB CLI to specify which key pair to use or to create a key pair for your deployment’s Amazon EC2 instances. To resolve the error, run `eb init -i` and one of the options will ask:Do you want to set up SSH for your instances?
Respond with `Y` to either create a key pair or specify an existing key pair. |
| I’ve updated my code and redeployed, but my deployment is not reflecting my changes. | If you’re using a Git repository with your deployment, make sure that you add and commit your changes before redeploying. |
| You are previewing the Flask application from an AWS Cloud9 IDE and run into errors. | For more information about this, see [Previewing running applications in the AWS Cloud9 IDE](https://docs.aws.amazon.com/cloud9/latest/user-guide/app-preview.html) in the AWS Cloud9 documentation. |
## Related resources
+ [Call an Amazon SageMaker AI model endpoint using Amazon API Gateway and AWS Lambda](https://aws.amazon.com/blogs/machine-learning/call-an-amazon-sagemaker-model-endpoint-using-amazon-api-gateway-and-aws-lambda/)
+ [Deploying a Flask application to Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-flask.html)
+ [EB CLI command reference](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-cmd-commands.html)
+ [Setting up your Python development environment](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/python-development-environment.html)
## Additional information
*Natural language processing using Amazon Comprehend*
By choosing to use Amazon Comprehend, you can detect custom entities in individual text documents by running real-time analysis or asynchronous batch jobs. Amazon Comprehend also enables you to train custom entity recognition and text classification models that can be used in real time by creating an endpoint.
This pattern uses asynchronous batch jobs to detect sentiments and entities from an input file that contains multiple documents. The sample application provided by this pattern is designed for users to upload a .csv file containing a single column with one text document per row. The `comprehend_helper.py` file in the GitHub [Visualize AI/ML model results using Flask and AWS Elastic Beanstalk](https://github.com/aws-samples/aws-comprehend-elasticbeanstalk-for-flask) repository reads the input file and sends the input to Amazon Comprehend for processing.
*BatchDetectEntities*
Amazon Comprehend inspects the text of a batch of documents for named entities and returns the detected entity, location, [type of entity](https://docs.aws.amazon.com/comprehend/latest/dg/how-entities.html), and a score that indicates the Amazon Comprehend level of confidence. A maximum of 25 documents can be sent in one API call, with each document smaller than 5,000 bytes in size. You can filter the results to show only certain entities based on the use case. For example, you could skip the `‘quantity’` entity type and set a threshold score for the detected entity (for example, 0.75). We recommend that you explore the results for your specific use case before choosing a threshold value. For more information about this, see [BatchDetectEntities](https://docs.aws.amazon.com/comprehend/latest/dg/API_BatchDetectEntities.html) in the Amazon Comprehend documentation.
*BatchDetectSentiment*
Amazon Comprehend inspects a batch of incoming documents and returns the prevailing sentiment for each document (`POSITIVE`, `NEUTRAL`, `MIXED`, or `NEGATIVE`). A maximum of 25 documents can be sent in one API call, with each document smaller than 5,000 bytes in size. Analyzing the sentiment is straightforward and you choose the sentiment with the highest score to be displayed in the final results. For more information about this, see [BatchDetectSentiment](https://docs.aws.amazon.com/comprehend/latest/dg/API_BatchDetectSentiment.html) in the Amazon Comprehend documentation.
*Flask configuration handling*
Flask servers use a series of [configuration variables](https://flask.palletsprojects.com/en/stable/config/) to control how the server runs. These variables can contain debug output, session tokens, or other application settings. You can also define custom variables that can be accessed while the application is running. There are multiple approaches for setting configuration variables.
In this pattern, the configuration is defined in `config.py` and inherited within `application.py`.
+ `config.py` contains the configuration variables that are set up on the application's startup. In this application, a `DEBUG` variable is defined to tell the application to run the server in [debug mode](https://flask.palletsprojects.com/en/stable/config/#DEBUG).
**Note**
Debug mode should not be used when running an application in a production environment. `UPLOAD_FOLDER` is a custom variable that is defined to be referenced later in the application and inform it where uploaded user data should be stored.
+ `application.py` initiates the Flask application and inherits the configuration settings defined in `config.py`. This is performed by the following code:
```
application = Flask(__name__)
application.config.from_pyfile('config.py')
```
# More patterns
**Topics**
+ [Accelerate MLOps with Backstage and self-service Amazon SageMaker AI templates](accelerate-mlops-with-backstage-and-sagemaker-templates.md)
+ [Automate AWS infrastructure operations by using Amazon Bedrock](automate-aws-infrastructure-operations-by-using-amazon-bedrock.md)
+ [Automate the setup of inter-Region peering with AWS Transit Gateway](automate-the-setup-of-inter-region-peering-with-aws-transit-gateway.md)
+ [Deploy agentic systems on Amazon Bedrock with the CrewAI framework by using Terraform](deploy-agentic-systems-on-amazon-bedrock-with-the-crewai-framework.md)
+ [Deploy a ChatOps solution to manage SAST scan results by using Amazon Q Developer in chat applications custom actions and CloudFormation](deploy-chatops-solution-to-manage-sast-scan-results.md)
+ [Generate data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.md)
+ [Generate Db2 z/OS data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.md)
+ [Give SageMaker notebook instances temporary access to a CodeCommit repository in another AWS account](give-sagemaker-notebook-instances-temporary-access-to-a-codecommit-repository-in-another-aws-account.md)
+ [Manage AWS Organizations policies as code by using AWS CodePipeline and Amazon Bedrock](manage-organizations-policies-as-code.md)
+ [Modernize the CardDemo mainframe application by using AWS Transform](modernize-carddemo-mainframe-app.md)
+ [Modernize and deploy mainframe applications using AWS Transform and Terraform](modernize-mainframe-app-transform-terraform.md)
+ [Perform advanced analytics using Amazon Redshift ML](perform-advanced-analytics-using-amazon-redshift-ml.md)
+ [Streamline Amazon EC2 compliance management with Amazon Bedrock agents and AWS Config](streamline-amazon-ec2-compliance-management-with-amazon-bedrock-agents-and-aws-config.md)
+ [Streamline Amazon Lex bot development and deployment by using an automated workflow](streamline-amazon-lex-bot-development-and-deployment-using-an-automated-workflow.md)
+ [Transform Easytrieve to modern languages by using AWS Transform custom](transform-easytrieve-modern-languages.md)
+ [Troubleshoot states in AWS Step Functions by using Amazon Bedrock](troubleshooting-states-in-aws-step-functions.md)