Coherence: Measures the organization and structure of a generated text. - Thumbs up/down\nAccuracy: Indicates the accuracy of a generated text. - Choice buttons
\nThe evaluation creator should use this space to write detailed descriptions for every rating method so your evaluators know how to properly rate the responses with the buttons on their screen.
\nIf using Likert scale - individual, define the 1 and 5 of the 5 point Likert scale for each metric so your evaluators know if 1 or 5 means favorable/acceptable/preferable.\nIf using Likert scale - comparison, describe what the evaluator is looking for to determine their preference between two responses.\nIf using Choice buttons, describe what is preferred according to your metric and its description.\nIf using Ordinal ranking, define what should receive a #1 ranking according to your metric and its description.\nIf using Thumbs up/down, define what makes an acceptable response according to your metric and its description.
\nDescribe the purpose of your ground truth responses that will be shown on screen next to each model response. Note that the ground truth responses you provide are not rated/scored by the evaluators - they are meant to be a reference standard for comparison against the model responses.
" } } }, "inferenceConfig": { "models": [ { "precomputedInferenceSource": { "inferenceSourceIdentifier": "llama-3-1-80b" } }, { "precomputedInferenceSource": { "inferenceSourceIdentifier": "my_other_model" } } ] }, "outputDataConfig": { "s3Uri": "s3://amzn-s3-demo-bucket/output/" } } ``` ------ #### [ SDK for Python ] The following code example shows you how to create a model evaluation job that uses human workers via the SDK for SDK for Python. ``` import boto3 client = boto3.client('bedrock') job_request = client.create_evaluation_job( jobName="111122223333-job-01", jobDescription="two different task types", roleArn="arn:aws:iam::111122223333:role/example-human-eval-api-role", inferenceConfig={ ## You must specify and array of models "models": [ { "bedrockModel": { "modelIdentifier":"arn:aws:bedrock:us-west-2::foundation-model/amazon.titan-text-lite-v1", "inferenceParams":"{\"inferenceConfig\":{\"maxTokens\": 512,\"temperature\":0.7,\"topP\":0.9}}" } }, { "bedrockModel": { "modelIdentifier": "anthropic.claude-v2", "inferenceParams": "{\"inferenceConfig\":{\"maxTokens\":512,\"temperature\":1,\"topP\":0.999,\"stopSequences\":[\"stop\"]},\"additionalModelRequestFields\":{\"top_k\": 128}}" } } ] }, outputDataConfig={ "s3Uri":"s3://job-bucket/outputs/" }, evaluationConfig={ "human": { "humanWorkflowConfig": { "flowDefinitionArn": "arn:aws:sagemaker:us-west-2:111122223333:flow-definition/example-workflow-arn", "instructions": "some human eval instruction" }, "customMetrics": [ { "name": "IndividualLikertScale", "description": "testing", "ratingMethod": "IndividualLikertScale" } ], "datasetMetricConfigs": [ { "taskType": "Summarization", "dataset": { "name": "Custom_Dataset1", "datasetLocation": { "s3Uri": "s3://job-bucket/custom-datasets/custom-trex.jsonl" } }, "metricNames": [ "IndividualLikertScale" ] } ] } } ) print(job_request) ``` ------ # List model evaluation jobs that use human workers in Amazon Bedrock You can list your current model evaluation jobs that use human workers using the AWS CLI, or a supported AWS SDK. In the Amazon Bedrock console, you can also view a table containing your current model evaluation jobs. The following examples show you how to find your model evaluation jobs using the AWS Management Console, AWS CLI and SDK for Python. ------ #### [ Amazon Bedrock console ] 1. Open the Amazon Bedrock console: [https://console.aws.amazon.com/bedrock/home](https://console.aws.amazon.com/bedrock/home) 1. In the navigation pane, choose **Model evaluation**. 1. In the **Model Evaluation Jobs** card, you can find a table that lists the model evaluation jobs you have already created. ------ #### [ AWS CLI ] In the AWS CLI, you can use the `help` command to view parameters are required, and which parameters are optional when using `list-evaluation-jobs`. ``` aws bedrock list-evaluation-jobs help ``` The follow is an example of using `list-evaluation-jobs` and specifying that maximum of 5 jobs be returned. By default jobs are returned in descending order from the time when they where started. ``` aws bedrock list-evaluation-jobs --max-items 5 ``` ------ #### [ SDK for Python ] The following examples show how to use the AWS SDK for Python to find a model evaluation job you have previously created. ``` import boto3 client = boto3.client('bedrock') job_request = client.list_evaluation_jobs(maxResults=20) print (job_request) ``` ------ # Stop a model evaluation job in Amazon Bedrock You can stop a model evaluation job that is currently processing using the AWS Management Console, AWS CLI, or a supported AWS SDK. The following examples show you how to stop a model evaluation job using the AWS Management Console, AWS CLI, and SDK for Python ------ #### [ Amazon Bedrock console ] The following example shows you how to stop a model evaluation job using the AWS Management Console 1. Open the Amazon Bedrock console: [https://console.aws.amazon.com/bedrock/home](https://console.aws.amazon.com/bedrock/home) 1. In the navigation pane, choose **Model evaluation**. 1. In the **Model Evaluation Jobs** card, you can find a table that lists the model evaluation jobs you have already created. 1. Select the radio button next to your job's name. 1. Then, choose **Stop evaluation**. ------ #### [ SDK for Python ] The following example shows you how to stop a model evaluation job using the SDK for Python ``` import boto3 client = boto3.client('bedrock') response = client.stop_evaluation_job( ## The ARN of the model evaluation job you want to stop. jobIdentifier='arn:aws:bedrock:us-west-2:444455556666:evaluation-job/fxaqujhttcza' ) print(response) ``` ------ #### [ AWS CLI ] In the AWS CLI, you can use the `help` command to see which parameters are required, and which parameters are optional when specifying `add-something` in the AWS CLI. ``` aws bedrock create-evaluation-job help ``` The following example shows you how to stop a model evaluation job using the AWS CLI ``` aws bedrock stop-evaluation-job --job-identifier arn:aws:bedrock:us-west-2:444455556666:evaluation-job/fxaqujhttcza ``` ------ # Delete a model evaluation job in Amazon Bedrock You can delete a model evaluation job by using the Amazon Bedrock console, or by using the [BatchDeleteEvaluationJob](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_BatchDeleteEvaluationJob.html) operation with the AWS CLI, or a supported AWS SDK. Before you can delete a model evaluation job, the status of the job must be `FAILED`, `COMPLETED`, or `STOPPED`. You can get the current status for a job from the Amazon Bedrock console or by calling the [ ListEvaluationJobs](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListEvaluationJobs.html). For more information, see [List automatic model evaluation jobs in Amazon BedrockList model evaluation jobs that use human workers in Amazon Bedrock](model-evaluation-jobs-management-list.md). You can delete up to 25 model evaluation jobs at a time with the console and with the `BatchDeleteEvaluationJob` operation. If you need to delete more jobs, repeat the console procedure or `BatchDeleteEvaluationJob` call. If you delete a model evaluation job with the `BatchDeleteEvaluationJob` operation, you need the Amazon Resource Names (ARNs) of the models that you want to delete. For information about getting the ARN for a model, see [List automatic model evaluation jobs in Amazon BedrockList model evaluation jobs that use human workers in Amazon Bedrock](model-evaluation-jobs-management-list.md). When you delete a model evaluation job all resources in Amazon Bedrock and Amazon SageMaker AI are removed. Any model evaluation job saved in Amazon S3 buckets are left unchanged. Also, for model evaluation jobs that use human workers, deleting a model evaluation job will not delete the workforce or workteam you have configured in Amazon Cognito or SageMaker AI. Use the following sections to see examples of how to delete a model evaluation job. ------ #### [ Amazon Bedrock console ] Use the following procedure to delete model evaluation job using the Amazon Bedrock console. To successfully complete this procedure make sure that your IAM user, group, or role has the sufficient permissions to access the console. To learn more, see [Required console permissions to create a human-based model evaluation job](model-evaluation-type-human.md#more-for-human). **To delete multiple model evaluation jobs.** 1. Open the Amazon Bedrock console: [https://console.aws.amazon.com/bedrock/](https://console.aws.amazon.com/bedrock/) 1. In the navigation pane, choose **Model evaluation**. 1. In the **Model Evaluation Jobs** card, use the table to find the model evaluation jobs that you want to delete, select them using the checkbox next to the job's name. You can select up to 25 jobs. 1. Choose **Delete** to delete the model evaluation jobs. 1. If you need to delete more model evaluation jobs, repeat steps 3 and 4. ------ #### [ AWS CLI ] In the AWS CLI, you can use the `help` command to view parameters are required, and which parameters are optional when using `batch-delete-evaluation-job`. ``` aws bedrock batch-delete-evaluation-job help ``` The follow is an example of using `batch-delete-evaluation-job` and specifying that 2 model evaluation jobs be deleted. You use the `job-identifiers` parameter to specify a list of ARNS for the model evaluation jobs that you want to delete. You can delete up to 25 model evaluation jobs in a single call to `batch-delete-evaluation-job`. If you need to delete more jobs, make further calls to `batch-delete-evaluation-job`. ``` aws bedrock batch-delete-evaluation-job \ --job-identifiers arn:aws:bedrock:us-east-1:111122223333:evaluation-job/rmqp8zg80rvg arn:aws:bedrock:us-east-1:111122223333:evaluation-job/xmfp9zg204fdk ``` After submitting you would get the following response. ``` { "evaluationJobs": [ { "jobIdentifier": "rmqp8zg80rvg", "jobStatus": "Deleting" }, { "jobIdentifier": "xmfp9zg204fdk", "jobStatus": "Deleting" } ], "errors": [] } ``` ------ #### [ SDK for Python ] The following examples show how to use the AWS SDK for Python to delete a model evaluation job. Use the `jobIdentifiers` parameter to specify a list of ARNS for the model evaluation jobs that you want to delete. You can delete up to 25 model evaluation jobs in a single call to `BatchDeleteEvaluationJob`. If you need to delete more jobs, make further calls to `BatchDeleteEvaluationJob`. ``` import boto3 client = boto3.client('bedrock') job_request = client.batch_delete_model_evaluation_job(jobIdentifiers=["arn:aws:bedrock:us-east-1:111122223333:evaluation-job/rmqp8zg80rvg", "arn:aws:bedrock:us-east-1:111122223333:evaluation-job/xmfp9zg204fdk"]) print (job_request) ``` ------ # Manage a work team for human evaluations of models in Amazon Bedrock For evaluation jobs that use human workers you need to have a work team. A work team is a group of workers that *you* choose. These can be employees of your company or a group of subject-matter experts from your industry. **Worker notifications in Amazon Bedrock** When you create an evaluation job in Amazon Bedrock workers are notified of their assigned job *only* when you first add them to a work team If you delete a worker from a work team during evaluation job creation, they will lose access to *all* evaluation jobs they have been assigned too. For any new evaluation jobs that you assign to an existing human worker, you must notify them directly and provide them the URL to the worker portal. Workers must use their previously created login credentials for the worker portal. This worker portal is the same for all evaluation jobs in your AWS account per region You can create a human work team or manage an existing one while setting up an evaluation job workers to a *Private workforce* that is managed by Amazon SageMaker Ground Truth. Amazon SageMaker Ground Truth supports more advanced workforce management features. To learn more about managing your workforce in Amazon SageMaker Ground Truth, see [Create and manage workforces](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management.html). You can delete workers from a work team while setting up a new evaluation job. Otherwise, you must use either the Amazon Cognito console or the Amazon SageMaker Ground Truth console to manage work teams you've created in Amazon Bedrock. If the IAM user, group, or role has the required permissions you will see existing private workforces and work teams you created in Amazon Cognito, Amazon SageMaker Ground Truth, or Amazon Augmented AI visible when you are creating an evaluation job that uses human workers. Amazon Bedrock supports a maximum of 50 workers per work team. In the email addresses field, you can enter up to 50 email addresses at time. To add more workers to your evaluation job use the Amazon Cognito console or the Ground Truth console. The addresses must be separated by a comma. You should include your own email address so that you are part of the workforce and can see the labeling tasks. ## Create instructions for human workers Creating good instructions for your model evaluation jobs improves your worker's accuracy in completing their task. You can modify the default instructions that are provided in the console when creating a model evaluation job. The instructions are shown to the worker on the UI page where they complete their labeling task. To help workers complete their assigned tasks, you can provide instructions in two places. **Provide a good description for each evaluation and rating method** The descriptions should provide a succinct explanation of the metrics selected. The description should expand on the metric, and make clear how you want workers to evaluate the selected rating method. **Provide your workers overall evaluation instructions** These instructions are shown on the same webpage where workers complete a task. You can use this space to provide high level direction for the model evaluation job, and to describe the ground truth responses if you've included them in your prompt dataset. # Evaluate model performance using another LLM as a judge With a model evaluation job that uses a judge model, Amazon Bedrock uses an LLM to score another model's responses and provide an explanation of how it scored each prompt and response pair. Scores and explanations are available in the Amazon Bedrock console through the [Evaluations](https://console.aws.amazon.com/bedrock/home#/eval/evaluation) page. This kind of model evaluation requires two different models, a *generator model* and an *evaluator model*. You define prompts for the generator model in a dataset, and the evaluator model scores the responses to those prompts based on metrics you select. The metrics summary card in the console displays a histogram that shows the number of times a responses received a certain score, and explanations of the score for the first five prompts found in your dataset. The full evaluation job report is available in the Amazon S3 bucket you specify when you create the model evaluation job. When you create the model evaluation job, you can either select an Amazon Bedrock model as the generator model, or you can evaluate a non-Amazon Bedrock model by providing your own inference response data in the prompt dataset. If you provide your own response data, Amazon Bedrock skips the model invoke step and directly evaluates the data you supply. To rate the generator models' responses, Amazon Bedrock provides a set of built-in metrics you can select from. Each metric uses a different prompt for the evaluator model. You can also define your own custom metrics for your particular business case. See [Use metrics to understand model performance](model-evaluation-metrics.md) to learn more. ## Supported models ### Supported evaluator models (built-in metrics) To create an evaluation job that uses an LLM as a judge with Amazon Bedrock's built-in metrics, you need access to at least one of the judge models in the following list. To learn more about gaining access to models and Region availability, see [Access Amazon Bedrock foundation models](model-access.md). + Amazon Nova Pro – `amazon.nova-pro-v1:0` + Anthropic Claude 3.5 Sonnet v1 – `anthropic.claude-3-5-sonnet-20240620-v1:0` + Anthropic Claude 3.5 Sonnet v2 – `anthropic.claude-3-5-sonnet-20241022-v2:0` + Anthropic Claude 3.7 Sonnet – `anthropic.claude-3-7-sonnet-20250219-v1:0` + Anthropic Claude 3 Haiku – `anthropic.claude-3-haiku-20240307-v1:0` + Anthropic Claude 3.5 Haiku – `anthropic.claude-3-5-haiku-20241022-v1:0` + Meta Llama 3.1 70B Instruct – `meta.llama3-1-70b-instruct-v1:0` + Mistral Large – `mistral.mistral-large-2402-v1:0` [Cross Region inference](cross-region-inference.md) profiles are supported for the listed models. To learn more, see [Supported cross-Region inference profiles](inference-profiles-support.md#inference-profiles-support-system). ### Supported evaluator models (custom metrics) To create an evaluation job that uses an LLM as a judge with custom metrics, you need access to at least one of the judge models in the following list. + Mistral Large 24.02 – `mistral.mistral-large-2402-v1:0` + Mistral Large 24.07 – `mistral.mistral-large-2407-v1:0` + Anthropic Claude 3.5 Sonnet v1 – `anthropic.claude-3-5-sonnet-20240620-v1:0` + Anthropic Claude 3.5 Sonnet v2 – `anthropic.claude-3-5-sonnet-20241022-v2:0` + Anthropic Claude 3.7 Sonnet – `anthropic.claude-3-7-sonnet-20250219-v1:0` + Anthropic Claude 3 Haiku 3 – `anthropic.claude-3-haiku-20240307-v1:0` + Anthropic Claude 3 Haiku 3.5 – `anthropic.claude-3-5-haiku-20241022-v1:0` + Meta Llama 3.1 70B Instruct – `meta.llama3-1-70b-instruct-v1:0` + Meta Llama 3.3 70B Instruct – `meta.llama3-3-70b-instruct-v1:0` + Amazon Nova Pro – `amazon.nova-pro-v1:0` [Cross Region inference](cross-region-inference.md) profiles are supported for the listed models. To learn more, see [Supported cross-Region inference profiles](inference-profiles-support.md#inference-profiles-support-system). ### Supported generator models You can use the following model types in Amazon Bedrock as the generator model in an evaluation job. You can also bring your own inference response data from non-Amazon Bedrock models. + Foundation models – [Amazon Bedrock foundation model information](foundation-models-reference.md) + Amazon Bedrock Marketplace models – [Amazon Bedrock Marketplace](amazon-bedrock-marketplace.md) + Customized foundation models – [Customize your model to improve its performance for your use case](custom-models.md) + Imported foundation models – [Use Custom model import to import a customized open-source model into Amazon Bedrock](model-customization-import-model.md) + Prompt routers – [Understanding intelligent prompt routing in Amazon Bedrock](prompt-routing.md) + Models for which you have purchased Provisioned Throughput – [Increase model invocation capacity with Provisioned Throughput in Amazon Bedrock](prov-throughput.md) # Create a prompt dataset for a model evaluation job that uses a model as judge To create a model evaluation job that uses a model as judge you must specify a prompt dataset. This prompt dataset uses the same format as automatic model evaluation jobs and is used during inference with the models you select to evaluate. If you want to evaluate non-Amazon Bedrock models using responses that you've already generated, include them in the prompt dataset as described in [Prepare a dataset for an evaluation job using your own inference response data](#model-evaluation-prompt-datasets-judge-byoir). When you provide your own inference response data, Amazon Bedrock skips the model-invoke step and performs the evaluation job with the data you provide. Custom prompt datasets must be stored in Amazon S3, and use the JSON line format and `.jsonl` file extension. Each line must be a valid JSON object. There can be up to 1000 prompts in your dataset per evaluation job. For jobs created using the console you must update the Cross Origin Resource Sharing (CORS) configuration on the S3 bucket. To learn more about the required CORS permissions, see [Required Cross Origin Resource Sharing (CORS) permissions on S3 buckets](model-evaluation-security-cors.md). ## Prepare a dataset for an evaluation job where Amazon Bedrock invokes models for you To run an evaluation job where Amazon Bedrock invokes the models for you, create a prompt dataset containing the following key-value pairs: + `prompt` – the prompt you want the models to respond to. + `referenceResponse` – (optional) the ground truth response. + `category`– (optional) generates evaluation scores reported for each category. **Note** If you choose to supply a ground truth response (`referenceResponse)`, Amazon Bedrock will use this parameter when calculating the **Completeness** (`Builtin.Completeness`) and **Correctness** (`Builtin.Correctness`) metrics. You can also use these metrics without supplying a ground truth response. To see the judge prompts for both of these scenarios, refer to the section for your chosen judge model in [Built-in metric evaluator prompts for model-as-a-judge evaluation jobs](model-evaluation-type-judge-prompt.md). The following is an example custom dataset that contains 6 inputs and uses the JSON line format. ``` {"prompt":"Provide the prompt you want the model to use during inference","category":"(Optional) Specify an optional category","referenceResponse":"(Optional) Specify a ground truth response."} {"prompt":"Provide the prompt you want the model to use during inference","category":"(Optional) Specify an optional category","referenceResponse":"(Optional) Specify a ground truth response."} {"prompt":"Provide the prompt you want the model to use during inference","category":"(Optional) Specify an optional category","referenceResponse":"(Optional) Specify a ground truth response."} {"prompt":"Provide the prompt you want the model to use during inference","category":"(Optional) Specify an optional category","referenceResponse":"(Optional) Specify a ground truth response."} {"prompt":"Provide the prompt you want the model to use during inference","category":"(Optional) Specify an optional category","referenceResponse":"(Optional) Specify a ground truth response."} {"prompt":"Provide the prompt you want the model to use during inference","category":"(Optional) Specify an optional category","referenceResponse":"(Optional) Specify a ground truth response."} ``` The following example is a single entry expanded for clarity. In your actual prompt dataset each line must be a valid JSON object. ``` { "prompt": "What is high intensity interval training?", "category": "Fitness", "referenceResponse": "High-Intensity Interval Training (HIIT) is a cardiovascular exercise approach that involves short, intense bursts of exercise followed by brief recovery or rest periods." } ``` ## Prepare a dataset for an evaluation job using your own inference response data To run an evaluation job using responses you've already generated, create a prompt dataset containing the following key-value pairs: + `prompt` – the prompt your models used to generate the responses. + `referenceResponse` – (optional) the ground truth response. + `category`– (optional) generates evaluation scores reported for each category. + `modelResponses` – the response from your own inference that you want Amazon Bedrock to evaluate. Evaluation jobs that use a model as a judge support only one model response for each prompt, defined using the following keys: + `response` – a string containing the response from your model inference. + `modelIdentifier` – a string identifying the model that generated the response. You may use only one unique `modelIdentifier` in an evaluation job, and each prompt in your dataset must use this identifier. **Note** If you choose to supply a ground truth response (`referenceResponse)`, Amazon Bedrock will use this parameter when calculating the **Completeness** (`Builtin.Completeness`) and **Correctness** (`Builtin.Correctness`) metrics. You can also use these metrics without supplying a ground truth response. To see the judge prompts for both of these scenarios, refer to the section for your chosen judge model in [Built-in metric evaluator prompts for model-as-a-judge evaluation jobs](model-evaluation-type-judge-prompt.md). The following is a custom example dataset with 6 inputs in JSON line format. ``` {"prompt":"The prompt you used to generate the model response","referenceResponse":"(Optional) a ground truth response","category":"(Optional) a category for the prompt","modelResponses":[{"response":"The response your model generated","modelIdentifier":"A string identifying your model"}]} {"prompt":"The prompt you used to generate the model response","referenceResponse":"(Optional) a ground truth response","category":"(Optional) a category for the prompt","modelResponses":[{"response":"The response your model generated","modelIdentifier":"A string identifying your model"}]} {"prompt":"The prompt you used to generate the model response","referenceResponse":"(Optional) a ground truth response","category":"(Optional) a category for the prompt","modelResponses":[{"response":"The response your model generated","modelIdentifier":"A string identifying your model"}]} {"prompt":"The prompt you used to generate the model response","referenceResponse":"(Optional) a ground truth response","category":"(Optional) a category for the prompt","modelResponses":[{"response":"The response your model generated","modelIdentifier":"A string identifying your model"}]} {"prompt":"The prompt you used to generate the model response","referenceResponse":"(Optional) a ground truth response","category":"(Optional) a category for the prompt","modelResponses":[{"response":"The response your model generated","modelIdentifier":"A string identifying your model"}]} {"prompt":"The prompt you used to generate the model response","referenceResponse":"(Optional) a ground truth response","category":"(Optional) a category for the prompt","modelResponses":[{"response":"The response your model generated","modelIdentifier":"A string identifying your model"}]} ``` The following example shows a single entry in a prompt dataset expanded for clarity. ``` { "prompt": "What is high intensity interval training?", "referenceResponse": "High-Intensity Interval Training (HIIT) is a cardiovascular exercise approach that involves short, intense bursts of exercise followed by brief recovery or rest periods.", "category": "Fitness", "modelResponses": [ { "response": "High intensity interval training (HIIT) is a workout strategy that alternates between short bursts of intense, maximum-effort exercise and brief recovery periods, designed to maximize calorie burn and improve cardiovascular fitness.", "modelIdentifier": "my_model" } ] } ``` # Use metrics to understand model performance When you run a judge-based model evaluation job, the evaluator model you select uses a set of metrics to characterize the performance of the models being evaluated. Amazon Bedrock provides a number of built-in metrics you can choose from, or you can define your own metrics. The following table lists the built-in metrics available in Amazon Bedrock for evaluation jobs that use an LLM as a judge. To learn more about using custom metrics, see [Create a prompt for a custom metric](model-evaluation-custom-metrics-prompt-formats.md) and [Create a model evaluation job using custom metrics](model-evaluation-custom-metrics-create-job.md). | Metric | Description | | --- | --- | | Correctness (Builtin.Correctness) | Measures if the model's response to the prompt is correct. Note that if you supply a reference response (ground truth) as part of your [prompt dataset](model-evaluation-prompt-datasets-judge.md), the evaluator model considers this when scoring the response. | | Completeness (Builtin.Completeness) | Measures how well the model's response answers every question in the prompt. Note that if you supply a reference response (ground truth) as part of your [prompt dataset](model-evaluation-prompt-datasets-judge.md), the evaluator model considers this when scoring the response. | | Faithfulness (Builtin.Faithfulness) | Identifies whether the response contains information not found in the prompt to measure how faithful the response is to the available context. | | Helpfulness (Builtin.Helpfulness) | Measures how helpful the model's response is. The evaluation uses factors including whether the response follows provided instructions, whether the response is sensible and coherent, and whether the response anticipates implicit needs and expectations. | | Logical coherence (Builtin.Coherence) | Measures how coherent the response is by identifying logical gaps, inconsistencies, and contradictions in a model's response to a prompt. | | Relevance (Builtin.Relevance) | Measures how relevant the answer is to the prompt. | | Following instructions (Builtin.FollowingInstructions) | Measures how well the model's response respects the exact directions found in the prompt. | | Professional style and tone (Builtin.ProfessionalStyleAndTone) | Measures how appropriate the response's style, formatting, and tone is for a professional setting. | | Harmfulness (Builtin.Harmfulness) | Evaluates whether the response contains harmful content. | | Stereotyping (Builtin.Stereotyping) | Evaluates whether content in the response contains stereotypes of any kind (either positive or negative). | | Refusal (Builtin.Refusal) | Determines if the response directly declines to answer the prompt or rejects the request by providing reasons. | # Built-in metric evaluator prompts for model-as-a-judge evaluation jobs When you run a judge-based model evaluation job the *evaluator model* scores the *generator model's* responses according to a set of metrics you define. Amazon Bedrock offers a number of built-in metrics, or you can define your own [custom metrics](model-evaluation-custom-metrics-prompt-formats.md). Refer to the following sections to see the prompts used by the available evaluator models for the Amazon Bedrock built-in metrics. **Topics** + [Amazon Nova Pro](model-evaluation-type-judge-prompt-nova.md) + [Anthropic Claude 3.5 Sonnet](model-evaluation-type-judge-prompt-claude-sonnet.md) + [Anthropic Claude 3.5 Sonnet v2](model-evaluation-type-judge-prompt-claude-sonnet35v2.md) + [Anthropic Claude 3.7 Sonnet](model-evaluation-type-judge-prompt-claude-sonnet37.md) + [Anthropic Claude 3 Haiku](model-evaluation-type-judge-prompt-haiku.md) + [Anthropic Claude 3.5 Haiku](model-evaluation-type-judge-prompt-haiku35.md) + [Meta Llama 3.1 70B Instruct](model-evaluation-type-judge-prompt-llama.md) + [Mistral Large](model-evaluation-type-judge-prompt-mistral.md) # Amazon Nova Pro Prompts used with Amazon Nova Pro. ## Logical coherence *Logical coherence* – Looks for logical gaps, inconsistencies, and contradictions in a model's responses to a prompt. Responses are graded on a 5-point Likert scale, and then normalized in the output and the job's report card. The `{{prompt}}` will contain the prompt sent to the generator from your dataset, and the `{{prediction}}` is the generator model's responses. ``` You are a helpful agent that can assess LLM response according to the given rubrics. You are given a question and a response from LLM. Your task is to check if the arguments presented in the response follow logically from one another. When evaluating the logical cohesion of the response, consider the following rubrics: 1. Check for self-contradictions: - Does the response contradict its own previous statements? - If chat history is provided, does the response contradict statements from previous turns without explicitly correcting itself? 2. Identify any logic gaps or errors in reasoning: - Does the response draw false conclusions from the available information? - Does it make "logical leaps" by skipping steps in an argument? - Are there instances where you think, "this does not follow from that" or "these two things cannot be true at the same time"? 3. Evaluate the soundness of the reasoning, not the soundness of the claims: - If the question asks that a question be answered based on a particular set of assumptions, take those assumptions as the basis for argument, even if they are not true. - Evaluate the logical cohesion of the response as if the premises were true. 4. Distinguish between logical cohesion and correctness: - Logical cohesion focuses on how the response arrives at the answer, not whether the answer itself is correct. - A correct answer reached through flawed reasoning should still be penalized for logical cohesion. 5. Relevance of Logical Reasoning: - If the response doesn't require argumentation or inference-making, and simply presents facts without attempting to draw conclusions, it can be considered logically cohesive by default. - In such cases, automatically rate the logical cohesion as 'Yes', as there's no logic gaps. Please rate the logical cohesion of the response based on the following scale: - Not at all: The response contains too many errors of reasoning to be usable, such as contradicting itself, major gaps in reasoning, or failing to present any reasoning where it is required. - Not generally: The response contains a few instances of coherent reasoning, but errors reduce the quality and usability. - Neutral/Mixed: It's unclear whether the reasoning is correct or not, as different users may disagree. The output is neither particularly good nor particularly bad in terms of logical cohesion. - Generally yes: The response contains small issues with reasoning, but the main point is supported and reasonably well-argued. - Yes: There are no issues with logical cohesion at all. The output does not contradict itself, and all reasoning is sound. Here is the actual task: Question: {{prompt}} Response: {{prediction}} The output should be a well-formatted JSON instance that conforms to the JSON schema below. As an example, for the schema {"properties": {"foo": {"title": "Foo", "description": "a list of strings", "type": "array", "items": {"type": "string"}}}, "required": ["foo"]} the object {"foo": ["bar", "baz"]} is a well-formatted instance of the schema. The object {"properties": {"foo": ["bar", "baz"]}} is not well-formatted. Here is the output JSON schema: ``` {"properties": {"reasoning": {"description": "step by step reasoning to derive the final answer", "title": "Reasoning", "type": "string"}, "answer": {"description": "answer should be one of `Not at all`, `Not generally`, `Neutral/Mixed`, `Generally yes`, `Yes`", "enum": ["Not at all", "Not generally", "Neutral/Mixed", "Generally yes", "Yes"], "title": "Answer", "type": "string"}}, "required": ["reasoning", "answer"]} ``` Do not return any preamble or explanations, return only a pure JSON string surrounded by triple backticks (```). ``` ### Score mapping + **Not at all**: `0.0` + **Not generally**: `1.0` + **Neutral/Mixed**: `2.0` + **Generally yes**: `3.0` + **Yes**: `4.0` ## Faithfulness *Faithfulness* – Looks at whether the response contains information not found in the prompt, that cannot be inferred easily from the prompt. Responses are graded a 5-point Likert scale, and then normalized in the output and the job's report card. The `{{prompt}}` will contain the prompt sent to the generator from your dataset, and the `{{prediction}}` is the generator model's responses. ``` You are given a task in some context (Input), and a candidate answer. Is the candidate answer faithful to the task description and context? If the model gives an evasive response without any information, the candidate answer is faithful by default. A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context, like in a summarization task. If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable. Task: {{prompt}} Candidate Response: {{prediction}} Evaluate how much of the information in the answer is faithful to the available context. Label an evasive response as faithful. The output should be formatted as a XML file. 1. Output should conform to the tags below. 2. Remember to always open and close all the tags. 3. Do not invent new tags. As an example, for the tags ["foo", "bar", "baz"]: 1. String ""answerContent": {
"evaluationResults": {
"thumbsUpDown": [{
"metricName": "Relevance",
"modelResponseId": "0",
"result": false
}]
}
} | A list of JSON objects that contain worker responses in `answerContent`. |
| `humanLoopName` | String | system-generated-hash | A system generated 40-character hex string. |
| inputRecord | JSON object | "inputRecord": {
"prompt": "What does vitamin C serum do for skin?",
"category": "Skincare",
"referenceResponse": "Vitamin C serum offers a range of benefits for the skin. Firstly, it acts....
} | A JSON object that contains an entry prompt from the input dataset. |
| modelResponses | List of JSON objects | "modelResponses": [{
"modelIdentifier": "arn:aws:bedrock:us-west-2::foundation-model/model-id",
"response": "the-models-response-to-the-prompt"
}] | The individual responses from the models. |
| inputContent | Object | {
"additionalDataS3Uri":"s3://user-specified-S3-URI-path/datasets/dataset-name/records/record-number/human-loop-additional-data.json",
"evaluationMetrics":[
{
"description":"testing",
"metricName":"IndividualLikertScale",
"ratingMethod":"IndividualLikertScale"
}
],
"instructions":"example instructions"
} | The human loop input content required to start human loop in your S3 bucket. |
| modelResponseIdMap | Object | {
"0":"arn:aws:bedrock:us-west-2::foundation-model/model-id"
} | `humanAnswers.answerContent.evaluationResults` contains `modelResponseId`s. The `modelResponseIdMap` connects the `modelResponseId` to the model name. |
The following is an example of output data from a model evaluation job.
```
{
"humanEvaluationResult": [{
"flowDefinitionArn": "arn:aws:sagemaker:us-west-2:111122223333:flow-definition/flow-definition-name",
"humanAnswers": [{
"acceptanceTime": "2023-11-09T19:17:43.107Z",
"answerContent": {
"evaluationResults": {
"thumbsUpDown": [{
"metricName": "Coherence",
"modelResponseId": "0",
"result": false
}, {
"metricName": "Accuracy",
"modelResponseId": "0",
"result": true
}],
"individualLikertScale": [{
"metricName": "Toxicity",
"modelResponseId": "0",
"result": 1
}]
}
},
"submissionTime": "2023-11-09T19:17:52.101Z",
"timeSpentInSeconds": 8.994,
"workerId": "444455556666",
"workerMetadata": {
"identityData": {
"identityProviderType": "Cognito",
"issuer": "https://cognito-idp.AWS Region.amazonaws.com/AWS Region_111222",
"sub": "c6aa8eb7-9944-42e9-a6b9-"
}
}
}],
...Additional response have been truncated for clarity...
}],
"humanLoopName": "b3b1c64a2166e001e094123456789012",
"inputContent":{
"additionalDataS3Uri":"s3://user-specified-S3-output-path/datasets/dataset-name/records/record-number/human-loop-additional-data.json",
"evaluationMetrics":[
{
"description":"testing",
"metricName":"IndividualLikertScale",
"ratingMethod":"IndividualLikertScale"
}
],
"instructions":"some dummy instructions"
},
"modelResponseIdMap":{
"0":"arn:aws:bedrock:AWS Region::foundation-model/model-id"
}
},
"inputRecord": {
"category": "Fitness",
"prompt": "Human: What is High-Intensity Interval Training ? \\n\\nAssistant:",
"referenceResponse": "High-Intensity Interval Training (HIIT) is a cardiovascular exercise approach that involves short, intense bursts of exercise followed by brief recovery or rest periods. In HIIT, you perform exercises at near-maximum effort during the high-intensity intervals, which can include activities like sprinting, cycling, jumping, or body weight exercises such as burpees or push-ups. These intense intervals are interspersed with short recovery periods to lower your heart rate before the next high-intensity bout. HIIT workouts usually consist of multiple cycles of these intervals, and the number of cycles and the length of intervals can vary. HIIT is known for its time efficiency, making it an excellent choice for those with busy schedules. It offers both aerobic and anaerobic benefits, improving cardiovascular fitness, burning calories, and promoting fat loss, all while boosting metabolism for continued calorie burning post-workout."
},
"modelResponses": [{
"modelIdentifier": "arn:aws:bedrock:AWS Region::foundation-model/model-id",
"response": " High-intensity interval training (HIIT) is a form of exercise that involves short bursts of intense activity alternated with recovery periods. Here are some key features of HIIT:\n\n- Alternates short periods of intense anaerobic exercise with less intense recovery periods. The high-intensity periods are typically done at 80-95% of maximum heart rate.\n\n- The duration of the intense periods can range from 5 seconds to 8 minutes, but typically are in the range of 30 seconds to 4 minutes.\n\n- The total HIIT session typically lasts between 10-30 minutes, including warm-up, recovery periods, and cool down. \n\n- Uses exercises like sprints, cycling, rowing, jumping rope, stair climbing, swimming, etc. Movements that get your heart rate up quickly.\n\n- The short, intense intervals help improve cardiovascular fitness by increasing heart rate and oxygen consumption.\n\n- HIIT burns more calories and fat than steady-state cardio in a shorter time period. It also helps build muscle strength and endurance.\n\n- Considered an efficient and effective form of exercise for fat loss and improving aerobic power. But it requires motivation to push yourself during the intense intervals.\n\n- Not suitable for beginners due to the high-intensity. Start with steady-state cardio and build up endurance before trying HIIT.\n\nIn summary, HIIT intersperses intense bursts of"
}]
}
}
```
The following table explains how the **Rating method** you selected for each metric in the Amazon Bedrock console is returned in your Amazon S3 bucket. The first child-key under `evaluationResults` is how the **Rating method** is returned.
**How rating methods selected in the Amazon Bedrock console are saved in Amazon S3**
| Rating method selected | Saved in Amazon S3 |
| --- | --- |
| Likert scale - Individual | IndividualLikertScale |
| Likert scale - Comparison | ComparisonLikertScale |
| Choice buttons | ComparisonChoice |
| Ordinal rank | ComparisonRank |
| Thumbs up/down | ThumbsUpDown |
# Data management and encryption in Amazon Bedrock evaluation job
During the model evaluation job, Amazon Bedrock makes a temporary copy of your data and stores it in an AWS-owned Amazon S3 bucket. Amazon Bedrock deletes this data after the job finishes. Amazon Bedrock encrypts this data using a AWS KMS key. You can choose to specify your own AWS KMS key or to use an Amazon Bedrock-owned key to encrypt the data.
When you create a model evaluation job using the either the AWS Management Console, AWS CLI, or a supported AWS SDK you can choose to use an Amazon Bedrock owned KMS key or your own customer managed key. If no customer managed key is specified then an Amazon Bedrock owned key is used by default.
To use a customer managed key, you must add the required IAM actions and resources to the IAM service role's policy. You must also add the required AWS KMS key policy elements.
**Topics**
+ [Required Key policy elements to encrypt your model evaluation job using AWS KMS](model-evaluation-security-kms.md)
+ [AWS Key Management Service support in model evaluation jobs](model-evaluation-security-data.md)
+ [Data encryption for knowledge base evaluation jobs](rag-evaluation-security-data.md)
# Required Key policy elements to encrypt your model evaluation job using AWS KMS
Every AWS KMS key must have exactly one key policy. The statements in the key policy determine who has permission to use the AWS KMS key and how they can use it. You can also use IAM policies and grants to control access to the AWS KMS key, but every AWS KMS key must have a key policy.
**Required AWS KMS key policy elements in Amazon Bedrock**
+ `kms:Decrypt` — For files that you've encrypted with your AWS Key Management Service key, provides Amazon Bedrock with permissions to access and decrypt those files.
+ `kms:GenerateDataKey` — Controls permission to use the AWS Key Management Service key to generate data keys. Amazon Bedrock uses `GenerateDataKey` to encrypt the temporary data it stores for the evaluation job.
+ `kms:DescribeKey` — Provides detailed information about a KMS key.
You must add the following statement to your existing AWS KMS key policy. It provides Amazon Bedrock with permissions to temporarily store your data in an Amazon Bedrock service bucket using the AWS KMS that you've specified.
```
{
"Effect": "Allow",
"Principal": {
"Service": "bedrock.amazonaws.com"
},
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"StringLike": {
"kms:EncryptionContext:evaluationJobArn": "arn:aws:bedrock:{{region}}:{{accountId}}:evaluation-job/*",
"aws:SourceArn": "arn:aws:bedrock:{{region}}:{{accountId}}:evaluation-job/*"
}
}
}
```
The following is an example of a complete AWS KMS policy.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Id": "key-consolepolicy-3",
"Statement": [
{
"Sid": "EnableIAMUserPermissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "BedrockDataKeyAndDecrypt",
"Effect": "Allow",
"Principal": {
"Service": "bedrock.amazonaws.com"
},
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": "*",
"Condition": {
"ArnLike": {
"kms:EncryptionContext:evaluationJobArn": "arn:aws:bedrock:us-east-1:123456789012:evaluation-job/*",
"aws:SourceArn": "arn:aws:bedrock:us-east-1:123456789012:evaluation-job/*"
}
}
},
{
"Sid": "BedrockDescribeKey",
"Effect": "Allow",
"Principal": {
"Service": "bedrock.amazonaws.com"
},
"Action": "kms:DescribeKey",
"Resource": "*",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:bedrock:us-east-1:123456789012:evaluation-job/*"
}
}
}
]
}
```
------
## Setting up KMS permissions for roles calling CreateEvaluationJob API
Make sure you have DescribeKey, GenerateDataKey, and Decrypt permissions for your role used to create the evaluation job on the KMS key that you use in your evaluation job.
Example KMS key policy
```
{
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::account-id:role/APICallingRole"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey",
"kmsDescribeKey"
],
"Resource": "*"
}
]
}
```
Example IAM Policy for Role Calling CreateEvaluationJob API
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "CustomKMSKeyProvidedToBedrockEncryption",
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/keyYouUse"
]
}
]
}
```
------
# AWS Key Management Service support in model evaluation jobs
Amazon Bedrock uses the following IAM and AWS KMS permissions to use your AWS KMS key to decrypt your files and access them. It saves those files to an internal Amazon S3 location managed by Amazon Bedrock and uses the following permissions to encrypt them.
## IAM policy requirements
The IAM policy associated with the IAM role that you're using to make requests to Amazon Bedrock must have the following elements. To learn more about managing your AWS KMS keys, see [Using IAM policies with AWS Key Management Service](https://docs.aws.amazon.com/kms/latest/developerguide/iam-policies.html).
Model evaluation jobs in Amazon Bedrock use AWS owned keys. These KMS keys are owned by Amazon Bedrock. To learn more about AWS owned keys, see [AWS owned keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-owned-cmk) in the *AWS Key Management Service Developer Guide*.
**Required IAM policy elements**
+ `kms:Decrypt` — For files that you've encrypted with your AWS Key Management Service key, provides Amazon Bedrock with permissions to access and decrypt those files.
+ `kms:GenerateDataKey` — Controls permission to use the AWS Key Management Service key to generate data keys. Amazon Bedrock uses `GenerateDataKey` to encrypt the temporary data it stores for the evaluation job.
+ `kms:DescribeKey` — Provides detailed information about a KMS key.
+ `kms:ViaService` — The condition key limits use of an KMS key to requests from specified AWS services. You must specify Amazon S3 as a service because Amazon Bedrock stores a temporary copy of your data in an Amazon S3 location that it owns.
The following is an example IAM policy that contains only the required AWS KMS IAM actions and resources.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "CustomKMSKeyProvidedToBedrock",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/[[keyId]]"
]
},
{
"Sid": "CustomKMSDescribeKeyProvidedToBedrock",
"Effect": "Allow",
"Action": [
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/[[keyId]]"
]
}
]
}
```
------
### Setting up KMS permissions for roles calling CreateEvaluationJob API
Make sure you have DescribeKey, GenerateDataKey, and Decrypt permissions for your role used to create the evaluation job on the KMS key that you use in your evaluation job.
Example KMS key policy
```
{
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::account-id:role/APICallingRole"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey",
"kmsDescribeKey"
],
"Resource": "*"
}
]
}
```
Example IAM Policy for Role Calling CreateEvaluationJob API
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "CustomKMSKeyProvidedToBedrockEncryption",
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/keyYouUse"
]
}
]
}
```
------
# Data encryption for knowledge base evaluation jobs
During a knowledge base evaluation job, Amazon Bedrock makes a temporary copy of your data. Amazon Bedrock deletes the data after the job finishes. To encrypt the data, Amazon Bedrock uses a KMS key. It uses either a KMS key that you specify or a key that Amazon Bedrock owns.
Amazon Bedrock requires the IAM and AWS KMS permissions in the following sections so that it can use your KMS key to do the following:
+ Decrypt your data.
+ Encrypt the temporary copy that Amazon Bedrock makes.
When you create a knowledge base evaluation job, you can choose to use a KMS key that Amazon Bedrock owns, or you can choose your own customer-managed key. If you don't specify a customer-managed key, Amazon Bedrock uses its key by default.
Before you can use a customer-managed key, you must do the following:
+ Add the required IAM actions and resources to the IAM service role’s policy.
+ Add the required KMS key policy elements.
+ Create a policy that can interact with your customer-managed key. This is specified in a separate KMS key policy.
## Required policy elements
The IAM and KMS key policies in the following sections include the following required elements:
+ `kms:Decrypt` – For files that you've encrypted with your KMS key, provides Amazon Bedrock with permissions to access and decrypt those files.
+ `kms:GenerateDataKey` – Controls permission to use the KMS key to generate data keys. Amazon Bedrock uses `GenerateDataKey` to encrypt the temporary data that it stores for the evaluation job.
+ `kms:DescribeKey` – Provides detailed information about a KMS key.
+ `kms:ViaService` – The condition key limits use of a KMS key to request from specified AWS services. You must specify the following services:
+ Amazon S3, because Amazon Bedrock stores a temporary copy of your data in an Amazon S3 location that Amazon Bedrock owns.
+ Amazon Bedrock, because the evaluation service calls the Amazon Bedrock Knowledge Bases API to execute the knowledge base workflow.
+ `kms:EncryptionContext:context-key` – This condition key limits access to the AWS KMS operations so that they are specific only to the provided [encryption context](https://docs.aws.amazon.com/kms/latest/developerguide/encrypt_context.html).
## IAM policy requirements
In the IAM role that you use to with Amazon Bedrock, the associated IAM policy must have the following elements. To learn more about managing your AWS KMS keys, see [Using IAM policies with AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/iam-policies.html).
Knowledge base evaluation jobs in Amazon Bedrock use AWS owned keys. For more information about AWS owned keys, see [AWS owned keys](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-owned-cmk) in the *AWS Key Management Service Developer Guide*.
The following is an example IAM policy that contains only the required AWS KMS actions and resources:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "CustomKMSKeyProvidedToBedrockEncryption",
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/*"
],
"Condition": {
"StringEquals": {
"kms:ViaService": [
"s3.us-east-1.amazonaws.com"
]
}
}
},
{
"Sid": "CustomKMSKeyProvidedToBedrockEvalKMS",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/*"
],
"Condition": {
"StringLike": {
"kms:EncryptionContext:evaluationJobArn": "arn:aws:bedrock:us-east-1:123456789012:evaluation-job/*"
}
}
},
{
"Sid": "CustomKMSKeyProvidedToBedrockKBDecryption",
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/*"
],
"Condition": {
"StringLike": {
"kms:EncryptionContext:knowledgeBaseArn": "arn:aws:bedrock:us-east-1:123456789012:knowledge-base/*"
}
}
},
{
"Sid": "CustomKMSKeyProvidedToBedrockKBEncryption",
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/*"
],
"Condition": {
"StringLike": {
"kms:EncryptionContext:knowledgeBaseArn": "arn:aws:bedrock:us-east-1:123456789012:knowledge-base/*"
},
"StringEquals": {
"kms:ViaService": [
"bedrock.us-east-1.amazonaws.com"
]
}
}
},
{
"Sid": "CustomKMSKeyProvidedToBedrockKBGenerateDataKey",
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/*"
],
"Condition": {
"StringLike": {
"kms:EncryptionContext:CustomerAwsAccountId": "123456789012",
"kms:EncryptionContext:SessionId": "*"
},
"StringEquals": {
"kms:ViaService": [
"bedrock.us-east-1.amazonaws.com"
]
}
}
},
{
"Sid": "CustomKMSDescribeKeyProvidedToBedrock",
"Effect": "Allow",
"Action": [
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/*"
]
}
]
}
```
------
## AWS KMS key policy requirements
Every KMS key must have one key policy. The statements in the key policy determine who has permission to use the KMS key and how they can use it. You can also use IAM policies and grants to control access to the KMS keys, but every KMS key must have a key policy.
You must add the following statement to your existing KMS key policy. It provides Amazon Bedrock with permissions to temporarily store your data in an S3 bucket using the KMS key that you've specified.
### Setting up KMS permissions for roles calling CreateEvaluationJob API
Make sure you have DescribeKey, GenerateDataKey, and Decrypt permissions for your role used to create the evaluation job on the KMS key that you use in your evaluation job.
Example KMS key policy
```
{
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::account-id:role/APICallingRole"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey",
"kmsDescribeKey"
],
"Resource": "*"
}
]
}
```
Example IAM Policy for Role Calling CreateEvaluationJob API
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "CustomKMSKeyProvidedToBedrockEncryption",
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/keyYouUse"
]
}
]
}
```
------
# CloudTrail management events in model evaluation jobs
[Management events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-events-with-cloudtrail.html#logging-management-events) provide information about the resource operations performed on or in a resource (for example, reading or writing to an Amazon S3 object). These are also known as data plane operations. Data events are often high-volume activities that CloudTrail doesn’t log by default.
Model evaluation jobs log events for multiple AWS services
**CloudTrail data events by AWS service in model evaluation jobs**
+ **Amazon Bedrock**: Data events for all model inference run during the model evaluation job.
+ **Amazon SageMaker AI**: Data events for all human-based model evaluation jobs.
+ **Amazon S3**: Data events for reading and writing data to the Amazon S3 bucket specified when the model evaluation job was created.
+ **AWS Key Management Service**: Data events related to using customer managed AWS KMS keys.