Crear una evaluación en línea

# Create online evaluation configuration
agentcore add online-eval \
  --name "your_config_name" \
  --runtime "your_runtime_name" \
  --evaluator "Builtin.GoalSuccessRate" "Builtin.Helpfulness" \
  --sampling-rate 1.0 \
  --enable-on-create

Este comando agrega la configuración de evaluación en línea a su localagentcore.json. Ejecute agentcore deploy para crearla en su AWS cuenta.

Introduzca un nombre para la configuración de evaluación en línea.

Seleccione los evaluadores que desee incluir. Puede elegir entre los evaluadores integrados y cualquier evaluador personalizado que haya creado.

Establezca la frecuencia de muestreo: el porcentaje de solicitudes de agentes que se evaluarán.

Elija si desea habilitar la evaluación automáticamente después de la implementación.

Revise la configuración y pulse Entrar para confirmar.

from bedrock_agentcore_starter_toolkit import Evaluation

# Initialize the evaluation client
eval_client = Evaluation()

# Replace these with your actual values
config_name = "YOUR_CONFIG_NAME"
agent_id = "YOUR_AGENT_ID"  # e.g., agent_myagent-ABC123xyz

# Create online evaluation configuration
config = eval_client.create_online_config(
    config_name=config_name,                  # Must use underscores, not hyphens
    agent_id=agent_id,                        # Agent ID (e.g., agent_myagent-ABC123xyz)
    sampling_rate=1.0,                        # Percentage to evaluate (0-100). 1.0 = evaluate 1% of interactions
    evaluator_list=["Builtin.GoalSuccessRate", "Builtin.Helpfulness"],  # List of evaluator IDs
    config_description="Online Evaluation Config",  # Optional description
    auto_create_execution_role=True,          # Automatically creates IAM role (default: True)
    enable_on_create=True                     # Enable immediately after creation (default: True)
)

print("✅ Online evaluation configuration created!")
print(f"Config ID: {config['onlineEvaluationConfigId']}")
print(f"Status: {config['status']}")

# Save the config ID for later operations
config_id = config['onlineEvaluationConfigId']
print(f"\nSaved config_id: {config_id}")

import boto3

# Your input log group that contains agent traces
LOG_GROUP_NAME = "/aws/agentcore/test-agent-traces"

# The service.name attribute from Otel
SERVICE_NAME = "strands_healthcare_single_agent.DEFAULT"

# The role created earlier with the required permissions for eval
role_arn = f"arn:aws:iam::{ACCOUNT_ID}:role/AgentCoreEvaluationRole"

client = boto3.client('bedrock-agentcore-control')

create_config_response = client.create_online_evaluation_config(
    onlineEvaluationConfigName="strands_healthcare_agent_1",
    description="Continuous evaluation of a healthcare agent",
    rule={
        "samplingConfig": {"samplingPercentage": 80.0}
    },
    dataSourceConfig={
        "cloudWatchLogs": {
            "logGroupNames": [LOG_GROUP_NAME],
            "serviceNames": [SERVICE_NAME]
        }
    },
    evaluators=[{"evaluatorId":"Builtin.Helpfulness"}],
    evaluationExecutionRoleArn=role_arn,
    enableOnCreate=True
)

aws bedrock-agentcore-control create-online-evaluation-config \
    --online-evaluation-config-name "strands_healthcare_agent_1" \
    --description "Continuous evaluation of a healthcare agent" \
    --rule '{"samplingConfig": {"samplingPercentage": 80.0}}' \
    --data-source-config '{"cloudWatchLogs": {"logGroupNames": ["/aws/agentcore/test-agent-traces"], "serviceNames": ["strands_healthcare_single_agent.DEFAULT"]}}' \
    --evaluators '[{"evaluatorId": "Builtin.Helpfulness"}]' \
    --evaluation-execution-role-arn "arn:aws:iam::{YOUR_ACCOUNT_ID}:role/AgentCoreEvaluationRole" \
    --enable-on-create