배치 평가 시작하기

agentcore invoke --runtime AcmeSupport --prompt "What's the status of my order ORD-1001?"
agentcore invoke --runtime AcmeSupport --prompt "I need to return order ORD-1001, the shirt doesn't fit."
agentcore invoke --runtime AcmeSupport --prompt "What's the shipping status on ORD-1002?"
agentcore invoke --runtime AcmeSupport --prompt "My order ORD-1003 is delayed, can you help?"
agentcore invoke --runtime AcmeSupport --prompt "I'd like to check on order ORD-1004 please."
agentcore invoke --runtime AcmeSupport --prompt "Can you look up order ORD-1005 for me?"
agentcore invoke --runtime AcmeSupport --prompt "I want to return the coffee maker from order ORD-1005, it's defective."
agentcore invoke --runtime AcmeSupport --prompt "Where is my order ORD-1002? It should have arrived by now."
agentcore invoke --runtime AcmeSupport --prompt "ORD-1003 is really late, I want a discount."
agentcore invoke --runtime AcmeSupport --prompt "Can you check order ORD-1001 and tell me when it was delivered?"

import boto3
import json
import uuid

client = boto3.client("bedrock-agentcore", region_name=REGION)

prompts = [
    "What's the status of my order ORD-1001?",
    "I need to return order ORD-1001, the shirt doesn't fit.",
    "What's the shipping status on ORD-1002?",
    "My order ORD-1003 is delayed, can you help?",
    "I'd like to check on order ORD-1004 please.",
    "Can you look up order ORD-1005 for me?",
    "I want to return the coffee maker from order ORD-1005, it's defective.",
    "Where is my order ORD-1002? It should have arrived by now.",
    "ORD-1003 is really late, I want a discount.",
    "Can you check order ORD-1001 and tell me when it was delivered?",
]

for i, prompt in enumerate(prompts):
    session_id = f"acme-eval-{uuid.uuid4().hex[:12]}"
    print(f"[{i+1}/10] {prompt[:60]}...")

    response = client.invoke_agent_runtime(
        agentRuntimeArn=AGENT_ARN,
        runtimeSessionId=session_id,
        payload=json.dumps({"prompt": prompt}).encode(),
    )
    response_body = response["response"].read()
    print(f"  Done (session: {session_id})")

print("\nAll sessions created.")

agentcore run batch-evaluation \
  --runtime AcmeSupport \
  --evaluator Builtin.GoalSuccessRate Builtin.Helpfulness Builtin.Faithfulness \
  --wait

기본적으로는 작업을 agentcore run batch-evaluation 시작하고 즉시 반환합니다(차단 없음). 작업이 터미널 상태에 도달할 때까지 --wait 블록에 전달합니다. --wait를 사용하면 CLI는 프로젝트 구성에서 CloudWatch 로그 그룹 및 서비스 이름을 확인하고, 작업을 시작하고, 터미널 상태에 도달할 때까지 차단한 다음, 평가자당 평균 점수를 인쇄합니다.

Batch evaluation completed: acme-eval-a1b2c3d4

Sessions: 10 completed, 0 failed, 10 total

Evaluator                           Avg Score
─────────────────────────────────────────────
Builtin.GoalSuccessRate             0.7200
Builtin.Helpfulness                 0.8100
Builtin.Faithfulness                0.8500

Results saved to .cli/jobs/batch-eval-results/

--json를 추가하여 스크립팅을 위해 기계 판독 가능한 결과( batchEvaluationId 및 평가자별 포함averageScore)를 내보내고 -n <name>를 추가하여 실행 간에 결과를 비교할 수 있도록 실행에 레이블을 지정합니다. 예제:

agentcore run batch-evaluation \
  --runtime AcmeSupport \
  --evaluator Builtin.GoalSuccessRate Builtin.Helpfulness Builtin.Faithfulness \
  -n acme_baseline \
  --wait

import boto3
import uuid
import time
import json

eval_client = boto3.client("bedrock-agentcore", region_name=REGION)

# Start the batch evaluation
response = eval_client.start_batch_evaluation(
    batchEvaluationName=f"acme_baseline_{uuid.uuid4().hex[:8]}",
    evaluators=[
        {"evaluatorId": "Builtin.GoalSuccessRate"},
        {"evaluatorId": "Builtin.Helpfulness"},
        {"evaluatorId": "Builtin.Faithfulness"},
    ],
    dataSourceConfig={
        "cloudWatchLogs": {
            "serviceNames": [SERVICE_NAME],
            "logGroupNames": [LOG_GROUP],
        }
    },
    clientToken=str(uuid.uuid4()),
)

batch_eval_id = response["batchEvaluationId"]
print(f"Started: {batch_eval_id}")

# Poll until complete
while True:
    result = eval_client.get_batch_evaluation(batchEvaluationId=batch_eval_id)
    status = result["status"]
    print(f"Status: {status}")

    if status in ("COMPLETED", "COMPLETED_WITH_ERRORS", "FAILED", "STOPPED"):
        break
    time.sleep(30)

print(json.dumps(result, indent=4, default=str))

CLI는 완료된 배치 평가 작업과 그 결과를 볼 수 있는 일류 명령을 제공합니다. 배치 평가 작업 ID별로 특정 작업을 보거나 과거 작업을 나열합니다.

# View a batch evaluation job and its results
agentcore view batch-evaluation acme-eval-a1b2c3d4

# List batch evaluation jobs
agentcore batch-evaluations history

이러한 명령은 플래그가 지정되지 않은 경우 대화형으로 실행됩니다. 예를 들어와 같이 비대화형, 기계 판독 가능 출력--json에를 추가합니다agentcore view batch-evaluation acme-eval-a1b2c3d4 --json.

# Get the output location from the batch evaluation result
output = result["outputConfig"]["cloudWatchConfig"]
log_group = output["logGroupName"]
log_stream = output["logStreamName"]

# Read the events
logs_client = boto3.client("logs", region_name=REGION)
response = logs_client.get_log_events(
    logGroupName=log_group,
    logStreamName=log_stream,
)

for event in response["events"]:
    event_attrs = json.loads(event["message"]).get("attributes", {})
    print(f"Score: {event_attrs.get('gen_ai.evaluation.score.value')}")
    print(f"Label: {event_attrs.get('gen_ai.evaluation.score.label')}")
    print(f"Explanation: {event_attrs.get('gen_ai.evaluation.explanation', '')[:200]}")
    print()