# Online evaluation
An online evaluation configuration is a resource that defines how your agent is evaluated, including which evaluators to apply, which data sources to monitor, and evaluation parameters.
**Topics**
+ [Prerequisites](evaluations-prerequisites.md)
+ [Create and deploy your agent](create-deploy-agent.md)
+ [Create online evaluation](create-online-evaluations.md)
+ [Get online evaluation](get-online-evaluations.md)
+ [List online evaluations](list-online-evaluations.md)
+ [Update online evaluation](update-online-evaluations.md)
+ [Delete online evaluation](delete-online-evaluations.md)
+ [Results and output](results-and-output.md)
# Prerequisites
Before you begin using Amazon Bedrock AgentCore Evaluations, ensure you have the necessary AWS permissions and service roles configured.
**Topics**
+ [Required permissions](#required-permissions)
+ [IAM user permissions](#iam-user-permissions)
+ [Service execution role](#service-execution-role)
## Required permissions
To use AgentCore Evaluations online evaluation features, you need:
+ ** AWS Account** with appropriate IAM permissions
+ **Amazon Bedrock** access with model invocation permissions (required when using a custom evaluator)
+ **Amazon CloudWatch** access for viewing evaluation results
+ **Transaction Search** enabled in CloudWatch - see Enable Transaction Search
+ ** AWS Distro for OpenTelemetry (ADOT) SDK** instrumenting your agent. Use AgentCore Observability instructions to configure observability for agents hosted on AgentCore Runtime and agents hosted elsewhere.
## IAM user permissions
Your IAM user or role needs the following permissions to create and manage evaluations:
**Topics**
+ [Console and API operations](#console-api-operations)
### Console and API operations
To use Amazon Bedrock AgentCore, you can attach the [BedrockAgentCoreFullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/BedrockAgentCoreFullAccess.html) AWS managed policy to your IAM user or IAM role. This policy grants broad permissions for all AgentCore capabilities. If you only use AgentCore Evaluations, we recommend creating a custom IAM policy that includes only the permissions required for evaluation.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"bedrock-agentcore:CreateEvaluator",
"bedrock-agentcore:GetEvaluator",
"bedrock-agentcore:ListEvaluators",
"bedrock-agentcore:UpdateEvaluator",
"bedrock-agentcore:DeleteEvaluator",
"bedrock-agentcore:CreateOnlineEvaluationConfig",
"bedrock-agentcore:GetOnlineEvaluationConfig",
"bedrock-agentcore:ListOnlineEvaluationConfigs",
"bedrock-agentcore:UpdateOnlineEvaluationConfig",
"bedrock-agentcore:DeleteOnlineEvaluationConfig",
"bedrock-agentcore:Evaluate"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "arn:aws:iam::*:role/AgentCoreEvaluationRole*",
"Condition": {
"StringEquals": {
"iam:PassedToService": "bedrock-agentcore.amazonaws.com"
}
}
},
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:Converse",
"bedrock:InvokeModelWithResponseStream",
"bedrock:ConverseStream"
],
"Resource": [
"arn:aws:bedrock:*::foundation-model/*",
"arn:aws:bedrock:*:*:inference-profile/*"
]
},
{
"Effect": "Allow",
"Action": [
"logs:DescribeIndexPolicies",
"logs:PutIndexPolicy",
"logs:CreateLogGroup"
],
"Resource": "*"
}
]
}
```
## Service execution role
Amazon Bedrock AgentCore Evaluations requires a custom IAM role to access AWS resources on your behalf. This role allows the service to:
+ Invoke Amazon Bedrock models for evaluation (required when using a custom evaluator)
+ Read traces from Amazon CloudWatch
+ Write evaluation results to Amazon CloudWatch
+ Configure log indexing for trace analysis
To create the IAM role you can use the AgentCore Evaluations console, the AWS console, or the AgentCore CLI.
**Topics**
+ [Option 1: Using AgentCore Evaluations Console](#option-console)
+ [Option 2: Using the AgentCore CLI](#option-toolkit)
+ [Option 3: Using the AWS Console](#option-aws-console)
### Option 1: Using AgentCore Evaluations Console
You can create the required IAM role directly through the AgentCore Evaluations console, which provides a streamlined approach with automatic role creation.
**To create an IAM role using the AgentCore Evaluations console**
1. Open the Amazon Bedrock AgentCore console.
1. In the left navigation pane, choose **Evaluation**.
1. Choose **Create evaluation configuration**.
1. In the Permission section, select **Create and use a new service role** and the console will automatically create the IAM role for you.
### Option 2: Using the AgentCore CLI
The AgentCore CLI automatically creates the required IAM role when you deploy your project.
### Option 3: Using the AWS Console
You can manually create the IAM role using the AWS console, which gives you full control over the role configuration and policies.
**To create an IAM role using the AWS console**
1. Open the IAM Console
1. Navigate to Roles and choose **Create role**
1. Select ** AWS service** as the trusted entity type
1. Create an IAM role with the following trust policy to allow Amazon Bedrock AgentCore to assume the role:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "TrustPolicyStatement",
"Effect": "Allow",
"Principal": {
"Service": "bedrock-agentcore.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "{{accountId}}",
"aws:ResourceAccount": "{{accountId}}"
},
"ArnLike": {
"aws:SourceArn": [
"arn:aws:bedrock-agentcore:{{region}}:{{accountId}}:evaluator/*",
"arn:aws:bedrock-agentcore:{{region}}:{{accountId}}:online-evaluation-config/*"
]
}
}
}
]
}
```
1. Attach the following permissions policy to the execution role:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CloudWatchLogReadStatement",
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups",
"logs:GetQueryResults",
"logs:StartQuery"
],
"Resource": "*"
},
{
"Sid": "CloudWatchLogWriteStatement",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:{{region}}:{{accountId}}:log-group:/aws/bedrock-agentcore/evaluations/*"
},
{
"Sid": "CloudWatchIndexPolicyStatement",
"Effect": "Allow",
"Action": [
"logs:DescribeIndexPolicies",
"logs:PutIndexPolicy"
],
"Resource": [
"arn:aws:logs:{{region}}:{{accountId}}:log-group:aws/spans",
"arn:aws:logs:{{region}}:{{accountId}}:log-group:aws/spans:*"
]
},
{
"Sid": "BedrockInvokeStatement",
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"Resource": [
"arn:aws:bedrock:{{region}}::foundation-model/*",
"arn:aws:bedrock:{{region}}:{{accountId}}:inference-profile/*"
]
}
]
}
```
**Note**
Replace \$1 \$1 region\$1\$1 and \$1 \$1 accountId\$1\$1 with your actual AWS region and account ID. If you are using a custom evaluator and have specified a BedrockInvokeStatement, you can also scope the allowed model IDs.
1. Name your role (e.g., AgentCoreEvaluationRole)
1. Review and create the role
# Create and deploy your agent
If you have an agent already up and running in AgentCore Runtime, you can skip the following steps
**Topics**
+ [Pick a supported framework](#supported-frameworks)
+ [Create and deploy your agent](#create-deploy-agent-steps)
## Pick a supported framework
AgentCore Evaluations currently supports the following agentic frameworks and instrumentation libraries
+ Strands Agent
+ LangGraph configured with one of the following instrumentation libraries
+ `opentelemetry-instrumentation-langchain`
+ `openinference-instrumentation-langchain`
## Create and deploy your agent
Create and deploy your agent by following the [Get Started guide for AgentCore Runtime](https://docs.aws.amazon.com/runtime-getting-started.html) . Setup observability using [Get started with AgentCore Observability](https://docs.aws.amazon.com/observability-get-started.html) . You can find additional examples in the [AgentCore Evaluations Samples](https://github.com/awslabs/amazon-bedrock-agentcore-samples/tree/main/01-tutorials/07-AgentCore-evaluations).
# Create online evaluation
The `CreateOnlineEvaluationConfig` API creates a new online evaluation configuration that continuously monitors your agent’s performance using live traffic. This asynchronous operation sets up the service to evaluate agent traces as they are generated during normal operation.
When you create an online evaluation, you specify a unique configuration name, the data source to monitor (either a list of CloudWatch log groups or an agent endpoint), and a list of evaluators to apply (up to 10, combining built-in and custom evaluators). You also provide an IAM service role ARN for execution. The `enableOnCreate` parameter is required and determines whether the evaluation starts running immediately upon creation ( `executionStatus` = true) or remains disabled until explicitly enabled ( `executionStatus` = false).
**Topics**
+ [Execution status control](#execution-status-control)
+ [Evaluator protection](#evaluator-protection)
+ [Code samples for AgentCore CLI, AgentCore SDK, and AWS SDK](#create-online-evaluation-code-samples)
+ [Console](#create-online-evaluation-console)
## Execution status control
The `executionStatus` parameter determines whether the evaluation job actively processes traces:
+ **ENABLED** – The evaluation job runs continuously, processing incoming traces and generating evaluation results.
+ **DISABLED** – The evaluation configuration exists but the job is paused. No traces are processed or evaluated.
You can control execution status using the CLI:
```
# Pause a running online evaluation
agentcore pause online-eval "your_config_name"
# Resume a paused online evaluation
agentcore resume online-eval "your_config_name"
```
## Evaluator protection
When you create an evaluation configuration with `executionStatus` set to `ENABLED` , the system automatically locks any custom evaluators you’ve selected. Once locked:
+ **No modifications allowed** – You cannot update the evaluator’s configuration, prompts, or settings. Clone a new evaluator if you need to make changes.
+ **No deletion allowed** – You cannot delete the evaluator while any evaluation job is using it (running).
## Code samples for AgentCore CLI, AgentCore SDK, and AWS SDK
The following code samples demonstrate how to create online evaluation configurations using different development approaches. Choose the method that best fits your development environment and preferences.
**Example**
1.
```
# Create online evaluation configuration
agentcore add online-eval \
--name "your_config_name" \
--runtime "your_runtime_name" \
--evaluator "Builtin.GoalSuccessRate" "Builtin.Helpfulness" \
--sampling-rate 1.0 \
--enable-on-create
```
This command adds the online evaluation configuration to your local `agentcore.json` . Run `agentcore deploy` to create it in your AWS account.
**Note**
Run this from inside an AgentCore project directory (created with `agentcore create` ).
1. Enter a name for your online evaluation configuration.
![\[Online eval config name input\]](http://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/images/tui/online-eval-add-name.png)
1. Select the evaluators to include. You can choose from built-in evaluators and any custom evaluators you have created.
![\[Evaluator multi-select list\]](http://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/images/tui/online-eval-add-evaluators.png)
1. Set the sampling rate — the percentage of agent requests that will be evaluated.
![\[Sampling rate input\]](http://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/images/tui/online-eval-add-sampling-rate.png)
1. Choose whether to enable evaluation automatically after deployment.
![\[Enable on deploy selection\]](http://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/images/tui/online-eval-add-enable.png)
1. Review the configuration and press Enter to confirm.
![\[Review online eval configuration\]](http://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/images/tui/online-eval-add-confirm.png)
1.
```
from bedrock_agentcore_starter_toolkit import Evaluation
# Initialize the evaluation client
eval_client = Evaluation()
# Replace these with your actual values
config_name = "YOUR_CONFIG_NAME"
agent_id = "YOUR_AGENT_ID" # e.g., agent_myagent-ABC123xyz
# Create online evaluation configuration
config = eval_client.create_online_config(
config_name=config_name, # Must use underscores, not hyphens
agent_id=agent_id, # Agent ID (e.g., agent_myagent-ABC123xyz)
sampling_rate=1.0, # Percentage to evaluate (0-100). 1.0 = evaluate 1% of interactions
evaluator_list=["Builtin.GoalSuccessRate", "Builtin.Helpfulness"], # List of evaluator IDs
config_description="Online Evaluation Config", # Optional description
auto_create_execution_role=True, # Automatically creates IAM role (default: True)
enable_on_create=True # Enable immediately after creation (default: True)
)
print("✅ Online evaluation configuration created!")
print(f"Config ID: {config['onlineEvaluationConfigId']}")
print(f"Status: {config['status']}")
# Save the config ID for later operations
config_id = config['onlineEvaluationConfigId']
print(f"\nSaved config_id: {config_id}")
```
1.
```
import boto3
# Your input log group that contains agent traces
LOG_GROUP_NAME = "/aws/agentcore/test-agent-traces"
# The service.name attribute from Otel
SERVICE_NAME = "strands_healthcare_single_agent.DEFAULT"
# The role created earlier with the required permissions for eval
role_arn = f"arn:aws:iam::{ACCOUNT_ID}:role/AgentCoreEvaluationRole"
client = boto3.client('bedrock-agentcore-control')
create_config_response = client.create_online_evaluation_config(
onlineEvaluationConfigName="strands_healthcare_agent_1",
description="Continuous evaluation of a healthcare agent",
rule={
"samplingConfig": {"samplingPercentage": 80.0}
},
dataSourceConfig={
"cloudWatchLogs": {
"logGroupNames": [LOG_GROUP_NAME],
"serviceNames": [SERVICE_NAME]
}
},
evaluators=[{"evaluatorId":"Builtin.Helpfulness"}],
evaluationExecutionRoleArn=role_arn,
enableOnCreate=True
)
```
1.
```
aws bedrock-agentcore-control create-online-evaluation-config \
--online-evaluation-config-name "strands_healthcare_agent_1" \
--description "Continuous evaluation of a healthcare agent" \
--rule '{"samplingConfig": {"samplingPercentage": 80.0}}' \
--data-source-config '{"cloudWatchLogs": {"logGroupNames": ["/aws/agentcore/test-agent-traces"], "serviceNames": ["strands_healthcare_single_agent.DEFAULT"]}}' \
--evaluators '[{"evaluatorId": "Builtin.Helpfulness"}]' \
--evaluation-execution-role-arn "arn:aws:iam::{YOUR_ACCOUNT_ID}:role/AgentCoreEvaluationRole" \
--enable-on-create
```
## Console
You can create online evaluation configurations using the Amazon Bedrock AgentCore console’s visual interface. This method provides guided forms and validation to help you configure your evaluation settings.
**To create an AgentCore online evaluation**
1. Open the Amazon Bedrock AgentCore console.
1. In the left navigation pane, choose **Evaluation**.
1. Choose **Create evaluation configuration**.
1. (Optional) For **Evaluation name** , enter a name for the online evaluation configuration.
1. (Optional) To enable the evaluation configuration after it’s created, select the checkbox under the evaluation name.
1. (Optional) For **Evaluation configuration description** , enter a description for the AgentCore evaluation configuration.
1. (Optional) For **Session idle timeout** , enter a duration between 1 and 60 minutes. The default is 15 minutes.
1. For **Data source** , choose one of the following:
1. **Define with an agent endpoint** – Choose an agent that you previously created on AgentCore Runtime, or create a new agent by choosing **Agents** . Then, choose an endpoint from the agent.
1. **Select a CloudWatch log group** – Select up to 5 log groups. Enter the service name used by your agent for observability. For agents hosted on AgentCore Runtime, service name follows the format .. For agents running outside AgentCore Runtime, service name is configured in OTEL\$1RESOURCE\$1ATTRIBUTES environment variable.
1. For **Evaluators** , select up to 10 evaluators per evaluation configuration, including built-in and custom evaluators.
1. (Optional) For **Filters** , add up to 5 filters to identify which sessions to evaluate.
1. (Optional) For **Sampling** , choose a percentage between 0.01% and 100% to control the percentage of sessions that are evaluated. The default is 10%.
1. For **Amazon Bedrock IAM role** , choose one of the following:
1. **Use an existing role** – Select an IAM service role that already has the required permissions.
1. **Create a new role** – Create a new IAM service role.
1. Choose **Create evaluation configuration** to create the AgentCore online evaluation configuration.
# Get online evaluation
The `GetOnlineEvaluationConfig` API retrieves the complete details and current status of an existing online evaluation configuration. This synchronous operation returns the full configuration including data sources, evaluators, execution status, and operational metadata.
Use this API to monitor the configuration’s lifecycle status (Creating, Active, Updating, or Deleting), check current execution status (ENABLED or DISABLED), and retrieve all configuration parameters including evaluator lists, data source settings, and output destinations.
## Code samples for AgentCore SDK and AWS SDK
The following code samples demonstrate how to get online evaluation configuration details using different development approaches. Choose the method that best fits your development environment and preferences.
**Example**
1.
```
from bedrock_agentcore_starter_toolkit import Evaluation
# Initialize the evaluation client
eval_client = Evaluation()
config_id = "config-abc123"
print(f"\nUsing config_id: {config_id}")
config_details = eval_client.get_online_config(config_id)
# Display configuration details
print(f"Config Name: {config_details['onlineEvaluationConfigName']}")
print(f"Config ID: {config_details['onlineEvaluationConfigId']}")
print(f"Status: {config_details['status']}")
```
1.
```
import boto3
client = boto3.client('bedrock-agentcore-control')
response = client.get_online_evaluation_config(
onlineEvaluationConfigId='your_config_id'
)
```
1.
```
aws bedrock-agentcore-control get-online-evaluation-config \
--online-evaluation-config-id your_config_id
```
## Console
You can view detailed information about a specific online evaluation configuration through the console interface.
**To get online evaluation configuration details**
1. Open the Amazon Bedrock AgentCore console.
1. In the navigation pane, choose **Evaluation**.
1. In the **Evaluation configurations** card, view the table that lists the evaluation configurations you have created.
1. To view information for a specific evaluation configuration, choose the configuration name to view its details.
# List online evaluations
The `ListOnlineEvaluationConfigs` API retrieves a paginated list of all online evaluation configurations in your account and Region. This synchronous operation uses the POST method and returns summary information for each configuration.
The response includes an array of evaluation configuration summaries containing the configuration ARN, ID, name, description, lifecycle status (Creating, Active, Updating, or Deleting), execution status (ENABLED or DISABLED), creation and update timestamps, and any failure reasons.
## Code samples for AgentCore SDK and AWS SDK
The following code samples demonstrate how to list online evaluation configurations using different development approaches. Choose the method that best fits your development environment and preferences.
**Example**
1.
```
from bedrock_agentcore_starter_toolkit import Evaluation
# Initialize the evaluation client
eval_client = Evaluation()
# List all online evaluation configurations, optionally filtered by agent.
configs= eval_client.list_online_configs()
agent_config_list = configs.get('onlineEvaluationConfigs', [])
print(f"Found {len(agent_config_list)} configuration(s) for this agent")
for cfg in agent_config_list:
print(f" • {cfg['onlineEvaluationConfigName']}")
print(f" ID: {cfg['onlineEvaluationConfigId']}")
print(f" Status: {cfg['status']}")
```
1.
```
import boto3
client = boto3.client('bedrock-agentcore-control')
list_configs_response = client.list_online_evaluation_configs(maxResults=20)
```
1.
```
aws bedrock-agentcore-control list-online-evaluation-configs \
--max-results 20
```
## Console
You can view and manage your online evaluation configurations through a visual interface that displays configuration details in an organized table format.
**To list online evaluation configurations**
1. Open the Amazon Bedrock AgentCore console.
1. In the navigation pane, choose **Evaluation**.
1. In the **Evaluation configurations** card, view the table that lists the evaluation configurations you have created.
# Update online evaluation
The `UpdateOnlineEvaluationConfig` API modifies an existing online evaluation configuration, allowing you to change evaluators, data sources, execution settings, and other parameters. This operation intelligently handles updates with no disruption to running evaluations.
Updates can only be made when your evaluation configuration is in Active or UpdateFailed status. If the configuration is currently being created, updated, or deleted, you’ll receive a conflict error and should retry after the operation completes.
## Execution control
The execution status parameter controls whether your online evaluation configuration actively processes agent traces. Understanding these states helps you manage evaluation costs and performance.
+ **Enabling evaluation** – When changing from DISABLED to ENABLED, the system provisions the service to begin processing traces from your specified data sources.
+ **Disabling evaluation** – When changing from ENABLED to DISABLED, the system stops processing new traces while preserving your configuration for future use.
## Code samples for AgentCore CLI, AgentCore SDK, and AWS SDK
The following code samples demonstrate how to update online evaluation configurations using different development approaches. Choose the method that best fits your development environment and preferences.
**Example**
1. To update an online evaluation configuration with the AgentCore CLI, edit the configuration in your `agentcore.json` file directly, then redeploy:
```
agentcore deploy
```
Open `agentcore.json` , find the configuration in the `onlineEvalConfigs` array, modify its settings, then run `agentcore deploy` . Changes won’t take effect until you deploy.
```
# Pause a running online evaluation
agentcore pause online-eval "your_config_name"
# Resume a paused online evaluation
agentcore resume online-eval "your_config_name"
```
**Note**
The configuration must be in `Active` or `UpdateFailed` lifecycle status before you can update it.
**Note**
To control whether the evaluation job is running ( `ENABLED` / `DISABLED` ), use `agentcore pause online-eval` and `agentcore resume online-eval`.
**Note**
Run this from inside an AgentCore project directory (created with `agentcore create` ).
1.
```
from bedrock_agentcore_starter_toolkit import Evaluation
# Initialize the evaluation client
eval_client = Evaluation()
config_id = "config-abc123"
print(f"\nUsing config_id: {config_id}")
# update description
eval_client.update_online_config(
config_id=config_id,
description="Updated description for online evaluation"
)
print("✅ Description updated")
```
1.
```
import boto3
client = boto3.client('bedrock-agentcore-control')
update_config_response = client.update_online_evaluation_config(
onlineEvaluationConfigId='your_config_id',
description="Updated description for online evaluation"
)
```
1.
```
aws bedrock-agentcore-control update-online-evaluation-config \
--online-evaluation-config-id your_config_id \
--description "Updated description for online evaluation"
```
## Console
Modify your online evaluation configuration settings using the console’s editing interface, which provides form validation and guided configuration options.
**To update an online evaluation configuration**
1. Open the Amazon Bedrock AgentCore console.
1. In the navigation pane, choose **Evaluation**.
1. In the **Evaluation configurations** card, view the table that lists the evaluation configurations you have created.
1. Choose one of the following methods to update the configuration:
+ Choose the evaluation configuration name to view its details, then choose **Edit** in the upper right of the details page.
+ Select the evaluation configuration so that it is highlighted, then choose **Edit** at the top of the **Evaluation configurations** card.
1. Update the fields as needed.
1. Choose **Update evaluation configuration** to save the changes.
# Delete online evaluation
The `DeleteOnlineEvaluationConfig` API permanently removes an online evaluation configuration and stops all associated evaluation processing. This asynchronous operation disables the evaluation service and cleans up all related resources.
An online evaluation can only be deleted when the configuration is in Active, UpdateFailed, or Disabled status. Configurations currently being created or updated must complete their operations before deletion is allowed.
## Code samples for AgentCore CLI, AgentCore SDK, and AWS SDK
The following code samples demonstrate how to delete online evaluation configurations using different development approaches. Choose the method that best fits your development environment and preferences.
**Example**
1.
```
# Delete (with confirmation prompt)
agentcore remove online-eval --name "your_config_name"
agentcore deploy
```
The `remove` command removes the online evaluation configuration from your local project. Run `agentcore deploy` to apply the deletion to your AWS account.
**Note**
Run this from inside an AgentCore project directory (created with `agentcore create` ).
1. \$1 Run `agentcore remove` and select **Online Eval Config** from the resource type menu.
![\[Remove resource type selection\]](http://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/images/tui/eval-remove-select.png)
1.
```
from bedrock_agentcore_starter_toolkit import Evaluation
# Initialize the evaluation client
eval_client = Evaluation()
config_id = "config-abc123"
print(f"\nUsing config_id: {config_id}")
eval_client.delete_online_config(
config_id=config_id,
delete_execution_role=True # Also delete the IAM role
)
```
1.
```
import boto3
client = boto3.client('bedrock-agentcore-control')
delete_config_response = client.delete_online_evaluation_config(
onlineEvaluationConfigId='your_config_id'
)
```
1.
```
aws bedrock-agentcore-control delete-online-evaluation-config \
--online-evaluation-config-id your_config_id
```
## Console
Permanently remove an online evaluation configuration using the console interface, which includes confirmation prompts to prevent accidental deletion.
**To delete an online evaluation configuration**
1. Open the Amazon Bedrock AgentCore console.
1. In the navigation pane, choose **Evaluation**.
1. In the **Evaluation configurations** card, view the table that lists the evaluation configurations you have created.
1. Choose one of the following methods to delete the configuration:
+ Choose the evaluation configuration name to view its details, then choose **Delete** in the upper right of the details page.
+ Select the evaluation configuration so that it is highlighted, then choose **Delete** at the top of the **Evaluation configurations** card.
1. Enter `confirm` to confirm the deletion.
1. Choose **Delete** to delete the configuration.
# Results and output
Online evaluation results are automatically saved to Amazon CloudWatch. When you create an online evaluation configuration, the service creates a dedicated CloudWatch log group to store your evaluation results in JSON format.
**Topics**
+ [Log group structure](#log-group-structure)
+ [Result format](#result-format)
+ [Viewing results in CloudWatch Observability Console](#viewing-results-console)
+ [Viewing evaluation scores in CloudWatch Metrics](#viewing-scores-metrics)
## Log group structure
Evaluation results are stored in a CloudWatch log group with the format `/aws/bedrock-agentcore/evaluations/results/` . The log group can be viewed on evaluation configuration **details page** in Amazon Bedrock AgentCore console.
Each evaluation generates a separate log entry within this log group. Additionally, evaluation scores are emitted as CloudWatch metrics for monitoring and analysis.
## Result format
Evaluations results follow OpenTelemetry semantic conventions for GenAI evaluation result events. The events are parented to the original span ID when possible and contain references the original trace ID and session ID.
You can use CloudWatch Logs Insights to query and analyze your evaluation results, and CloudWatch Metrics to monitor evaluation trends over time.
## Viewing results in CloudWatch Observability Console
You can view and analyze your evaluation results using the CloudWatch Observability Console. The console provides visualizations, metrics, and detailed logs of your agent evaluations.
**To view evaluation results**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/)
1. In the navigation pane, choose **GenAI Observability** > **Bedrock AgentCore**
1. Under the **Agents** section, select the agent and endpoint associated with your evaluation configuration
1. Navigate to the **Evaluations** tab for detailed results
For more details, see [AWS CloudWatch session trace evaluations documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/session-traces-evaluations.html).
## Viewing evaluation scores in CloudWatch Metrics
Evaluation scores are published as CloudWatch metrics. You can view them directly in the CloudWatch Metrics console.
**To view evaluation scores**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/)
1. In the navigation pane, choose **Metrics** > **All Metrics**
1. In the **Browse** tab, select **Bedrock-AgentCore/Evaluations**
1. Select dimension combinations to optionally narrow down results by evaluator type or evaluation label
For more details, see [AWS CloudWatch session trace evaluations documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/session-traces-evaluations.html).