批量评估入门

agentcore invoke --runtime AcmeSupport --prompt "What's the status of my order ORD-1001?"
agentcore invoke --runtime AcmeSupport --prompt "I need to return order ORD-1001, the shirt doesn't fit."
agentcore invoke --runtime AcmeSupport --prompt "What's the shipping status on ORD-1002?"
agentcore invoke --runtime AcmeSupport --prompt "My order ORD-1003 is delayed, can you help?"
agentcore invoke --runtime AcmeSupport --prompt "I'd like to check on order ORD-1004 please."
agentcore invoke --runtime AcmeSupport --prompt "Can you look up order ORD-1005 for me?"
agentcore invoke --runtime AcmeSupport --prompt "I want to return the coffee maker from order ORD-1005, it's defective."
agentcore invoke --runtime AcmeSupport --prompt "Where is my order ORD-1002? It should have arrived by now."
agentcore invoke --runtime AcmeSupport --prompt "ORD-1003 is really late, I want a discount."
agentcore invoke --runtime AcmeSupport --prompt "Can you check order ORD-1001 and tell me when it was delivered?"

import boto3
import json
import uuid

client = boto3.client("bedrock-agentcore", region_name=REGION)

prompts = [
    "What's the status of my order ORD-1001?",
    "I need to return order ORD-1001, the shirt doesn't fit.",
    "What's the shipping status on ORD-1002?",
    "My order ORD-1003 is delayed, can you help?",
    "I'd like to check on order ORD-1004 please.",
    "Can you look up order ORD-1005 for me?",
    "I want to return the coffee maker from order ORD-1005, it's defective.",
    "Where is my order ORD-1002? It should have arrived by now.",
    "ORD-1003 is really late, I want a discount.",
    "Can you check order ORD-1001 and tell me when it was delivered?",
]

for i, prompt in enumerate(prompts):
    session_id = f"acme-eval-{uuid.uuid4().hex[:12]}"
    print(f"[{i+1}/10] {prompt[:60]}...")

    response = client.invoke_agent_runtime(
        agentRuntimeArn=AGENT_ARN,
        runtimeSessionId=session_id,
        payload=json.dumps({"prompt": prompt}).encode(),
    )
    response_body = response["response"].read()
    print(f"  Done (session: {session_id})")

print("\nAll sessions created.")

agentcore run batch-evaluation \
  --runtime AcmeSupport \
  --evaluator Builtin.GoalSuccessRate Builtin.Helpfulness Builtin.Faithfulness \
  --wait

默认情况下，agentcore run batch-evaluation启动作业并立即返回（不阻塞）。传递--wait到方块，直到作业达到终端状态。使用--wait，CLI 解析项目配置中的 CloudWatch 日志组和服务名称，启动作业，阻塞直到其达到终端状态，然后打印每个评估者的平均分数：

Batch evaluation completed: acme-eval-a1b2c3d4

Sessions: 10 completed, 0 failed, 10 total

Evaluator                           Avg Score
─────────────────────────────────────────────
Builtin.GoalSuccessRate             0.7200
Builtin.Helpfulness                 0.8100
Builtin.Faithfulness                0.8500

Results saved to .cli/jobs/batch-eval-results/

添加--json以生成机器可读的结果（包括batchEvaluationId和每个评估器averageScore），用于编写脚本，并-n <name>标记运行以便您可以比较不同运行的结果。例如：

agentcore run batch-evaluation \
  --runtime AcmeSupport \
  --evaluator Builtin.GoalSuccessRate Builtin.Helpfulness Builtin.Faithfulness \
  -n acme_baseline \
  --wait

import boto3
import uuid
import time
import json

eval_client = boto3.client("bedrock-agentcore", region_name=REGION)

# Start the batch evaluation
response = eval_client.start_batch_evaluation(
    batchEvaluationName=f"acme_baseline_{uuid.uuid4().hex[:8]}",
    evaluators=[
        {"evaluatorId": "Builtin.GoalSuccessRate"},
        {"evaluatorId": "Builtin.Helpfulness"},
        {"evaluatorId": "Builtin.Faithfulness"},
    ],
    dataSourceConfig={
        "cloudWatchLogs": {
            "serviceNames": [SERVICE_NAME],
            "logGroupNames": [LOG_GROUP],
        }
    },
    clientToken=str(uuid.uuid4()),
)

batch_eval_id = response["batchEvaluationId"]
print(f"Started: {batch_eval_id}")

# Poll until complete
while True:
    result = eval_client.get_batch_evaluation(batchEvaluationId=batch_eval_id)
    status = result["status"]
    print(f"Status: {status}")

    if status in ("COMPLETED", "COMPLETED_WITH_ERRORS", "FAILED", "STOPPED"):
        break
    time.sleep(30)

print(json.dumps(result, indent=4, default=str))

CLI 提供一流的命令来查看已完成的批处理评估任务及其结果。按批量评估作业 ID 查看特定作业，或列出过去的作业：

# View a batch evaluation job and its results
agentcore view batch-evaluation acme-eval-a1b2c3d4

# List batch evaluation jobs
agentcore batch-evaluations history

当没有给出任何标志时，这些命令以交互方式运行。例如--json，添加非交互式、机器可读的输出。agentcore view batch-evaluation acme-eval-a1b2c3d4 --json

# Get the output location from the batch evaluation result
output = result["outputConfig"]["cloudWatchConfig"]
log_group = output["logGroupName"]
log_stream = output["logStreamName"]

# Read the events
logs_client = boto3.client("logs", region_name=REGION)
response = logs_client.get_log_events(
    logGroupName=log_group,
    logStreamName=log_stream,
)

for event in response["events"]:
    event_attrs = json.loads(event["message"]).get("attributes", {})
    print(f"Score: {event_attrs.get('gen_ai.evaluation.score.value')}")
    print(f"Label: {event_attrs.get('gen_ai.evaluation.score.label')}")
    print(f"Explanation: {event_attrs.get('gen_ai.evaluation.explanation', '')[:200]}")
    print()