온라인 평가 생성

# Create online evaluation configuration
agentcore add online-eval \
  --name "your_config_name" \
  --runtime "your_runtime_name" \
  --evaluator "Builtin.GoalSuccessRate" "Builtin.Helpfulness" \
  --sampling-rate 1.0 \
  --enable-on-create

이 명령은 온라인 평가 구성을 로컬 agentcore.json에 추가합니다. agentcore deploy를 실행하여 AWS 계정에서 생성합니다.

온라인 평가 구성의 이름을 입력합니다.

포함할 평가자를 선택합니다. 기본 제공 평가자와 생성한 사용자 지정 평가자 중에서 선택할 수 있습니다.

샘플링 속도 설정 - 평가할 에이전트 요청의 백분율입니다.

배포 후 평가를 자동으로 활성화할지 여부를 선택합니다.

구성을 검토하고 Enter 키를 눌러 확인합니다.

from bedrock_agentcore_starter_toolkit import Evaluation

# Initialize the evaluation client
eval_client = Evaluation()

# Replace these with your actual values
config_name = "YOUR_CONFIG_NAME"
agent_id = "YOUR_AGENT_ID"  # e.g., agent_myagent-ABC123xyz

# Create online evaluation configuration
config = eval_client.create_online_config(
    config_name=config_name,                  # Must use underscores, not hyphens
    agent_id=agent_id,                        # Agent ID (e.g., agent_myagent-ABC123xyz)
    sampling_rate=1.0,                        # Percentage to evaluate (0-100). 1.0 = evaluate 1% of interactions
    evaluator_list=["Builtin.GoalSuccessRate", "Builtin.Helpfulness"],  # List of evaluator IDs
    config_description="Online Evaluation Config",  # Optional description
    auto_create_execution_role=True,          # Automatically creates IAM role (default: True)
    enable_on_create=True                     # Enable immediately after creation (default: True)
)

print("✅ Online evaluation configuration created!")
print(f"Config ID: {config['onlineEvaluationConfigId']}")
print(f"Status: {config['status']}")

# Save the config ID for later operations
config_id = config['onlineEvaluationConfigId']
print(f"\nSaved config_id: {config_id}")

import boto3

# Your input log group that contains agent traces
LOG_GROUP_NAME = "/aws/agentcore/test-agent-traces"

# The service.name attribute from Otel
SERVICE_NAME = "strands_healthcare_single_agent.DEFAULT"

# The role created earlier with the required permissions for eval
role_arn = f"arn:aws:iam::{ACCOUNT_ID}:role/AgentCoreEvaluationRole"

client = boto3.client('bedrock-agentcore-control')

create_config_response = client.create_online_evaluation_config(
    onlineEvaluationConfigName="strands_healthcare_agent_1",
    description="Continuous evaluation of a healthcare agent",
    rule={
        "samplingConfig": {"samplingPercentage": 80.0}
    },
    dataSourceConfig={
        "cloudWatchLogs": {
            "logGroupNames": [LOG_GROUP_NAME],
            "serviceNames": [SERVICE_NAME]
        }
    },
    evaluators=[{"evaluatorId":"Builtin.Helpfulness"}],
    evaluationExecutionRoleArn=role_arn,
    enableOnCreate=True
)

aws bedrock-agentcore-control create-online-evaluation-config \
    --online-evaluation-config-name "strands_healthcare_agent_1" \
    --description "Continuous evaluation of a healthcare agent" \
    --rule '{"samplingConfig": {"samplingPercentage": 80.0}}' \
    --data-source-config '{"cloudWatchLogs": {"logGroupNames": ["/aws/agentcore/test-agent-traces"], "serviceNames": ["strands_healthcare_single_agent.DEFAULT"]}}' \
    --evaluators '[{"evaluatorId": "Builtin.Helpfulness"}]' \
    --evaluation-execution-role-arn "arn:aws:iam::{YOUR_ACCOUNT_ID}:role/AgentCoreEvaluationRole" \
    --enable-on-create