バッチ評価の開始方法

agentcore invoke --runtime AcmeSupport --prompt "What's the status of my order ORD-1001?"
agentcore invoke --runtime AcmeSupport --prompt "I need to return order ORD-1001, the shirt doesn't fit."
agentcore invoke --runtime AcmeSupport --prompt "What's the shipping status on ORD-1002?"
agentcore invoke --runtime AcmeSupport --prompt "My order ORD-1003 is delayed, can you help?"
agentcore invoke --runtime AcmeSupport --prompt "I'd like to check on order ORD-1004 please."
agentcore invoke --runtime AcmeSupport --prompt "Can you look up order ORD-1005 for me?"
agentcore invoke --runtime AcmeSupport --prompt "I want to return the coffee maker from order ORD-1005, it's defective."
agentcore invoke --runtime AcmeSupport --prompt "Where is my order ORD-1002? It should have arrived by now."
agentcore invoke --runtime AcmeSupport --prompt "ORD-1003 is really late, I want a discount."
agentcore invoke --runtime AcmeSupport --prompt "Can you check order ORD-1001 and tell me when it was delivered?"

import boto3
import json
import uuid

client = boto3.client("bedrock-agentcore", region_name=REGION)

prompts = [
    "What's the status of my order ORD-1001?",
    "I need to return order ORD-1001, the shirt doesn't fit.",
    "What's the shipping status on ORD-1002?",
    "My order ORD-1003 is delayed, can you help?",
    "I'd like to check on order ORD-1004 please.",
    "Can you look up order ORD-1005 for me?",
    "I want to return the coffee maker from order ORD-1005, it's defective.",
    "Where is my order ORD-1002? It should have arrived by now.",
    "ORD-1003 is really late, I want a discount.",
    "Can you check order ORD-1001 and tell me when it was delivered?",
]

for i, prompt in enumerate(prompts):
    session_id = f"acme-eval-{uuid.uuid4().hex[:12]}"
    print(f"[{i+1}/10] {prompt[:60]}...")

    response = client.invoke_agent_runtime(
        agentRuntimeArn=AGENT_ARN,
        runtimeSessionId=session_id,
        payload=json.dumps({"prompt": prompt}).encode(),
    )
    response_body = response["response"].read()
    print(f"  Done (session: {session_id})")

print("\nAll sessions created.")

agentcore run batch-evaluation \
  --runtime AcmeSupport \
  --evaluator Builtin.GoalSuccessRate Builtin.Helpfulness Builtin.Faithfulness \
  --wait

デフォルトでは、 はジョブagentcore run batch-evaluationを開始し、すぐに (ブロックせずに) を返します。ジョブが終了状態になるまで をブロック--waitに渡します。では--wait、CLI はプロジェクト設定から CloudWatch ロググループとサービス名を解決し、ジョブを開始し、終了状態になるまでブロックしてから、評価者ごとの平均スコアを出力します。

Batch evaluation completed: acme-eval-a1b2c3d4

Sessions: 10 completed, 0 failed, 10 total

Evaluator                           Avg Score
─────────────────────────────────────────────
Builtin.GoalSuccessRate             0.7200
Builtin.Helpfulness                 0.8100
Builtin.Faithfulness                0.8500

Results saved to .cli/jobs/batch-eval-results/

スクリプト--json用に機械読み取り可能な結果 ( batchEvaluationId および評価者ごとの を含むaverageScore) を出力し、実行間で結果を比較できるように実行-n <name>にラベルを付けるには、 を追加します。例えば、次のようになります。

agentcore run batch-evaluation \
  --runtime AcmeSupport \
  --evaluator Builtin.GoalSuccessRate Builtin.Helpfulness Builtin.Faithfulness \
  -n acme_baseline \
  --wait

import boto3
import uuid
import time
import json

eval_client = boto3.client("bedrock-agentcore", region_name=REGION)

# Start the batch evaluation
response = eval_client.start_batch_evaluation(
    batchEvaluationName=f"acme_baseline_{uuid.uuid4().hex[:8]}",
    evaluators=[
        {"evaluatorId": "Builtin.GoalSuccessRate"},
        {"evaluatorId": "Builtin.Helpfulness"},
        {"evaluatorId": "Builtin.Faithfulness"},
    ],
    dataSourceConfig={
        "cloudWatchLogs": {
            "serviceNames": [SERVICE_NAME],
            "logGroupNames": [LOG_GROUP],
        }
    },
    clientToken=str(uuid.uuid4()),
)

batch_eval_id = response["batchEvaluationId"]
print(f"Started: {batch_eval_id}")

# Poll until complete
while True:
    result = eval_client.get_batch_evaluation(batchEvaluationId=batch_eval_id)
    status = result["status"]
    print(f"Status: {status}")

    if status in ("COMPLETED", "COMPLETED_WITH_ERRORS", "FAILED", "STOPPED"):
        break
    time.sleep(30)

print(json.dumps(result, indent=4, default=str))

CLI には、完了したバッチ評価ジョブとその結果を表示するためのファーストクラスコマンドが用意されています。バッチ評価ジョブ ID で特定のジョブを表示するか、過去のジョブを一覧表示します。

# View a batch evaluation job and its results
agentcore view batch-evaluation acme-eval-a1b2c3d4

# List batch evaluation jobs
agentcore batch-evaluations history

これらのコマンドは、フラグが指定されていない場合にインタラクティブに実行されます。例えば、非インタラクティブで機械読み取り可能な出力--jsonに を追加しますagentcore view batch-evaluation acme-eval-a1b2c3d4 --json。

# Get the output location from the batch evaluation result
output = result["outputConfig"]["cloudWatchConfig"]
log_group = output["logGroupName"]
log_stream = output["logStreamName"]

# Read the events
logs_client = boto3.client("logs", region_name=REGION)
response = logs_client.get_log_events(
    logGroupName=log_group,
    logStreamName=log_stream,
)

for event in response["events"]:
    event_attrs = json.loads(event["message"]).get("attributes", {})
    print(f"Score: {event_attrs.get('gen_ai.evaluation.score.value')}")
    print(f"Label: {event_attrs.get('gen_ai.evaluation.score.label')}")
    print(f"Explanation: {event_attrs.get('gen_ai.evaluation.explanation', '')[:200]}")
    print()