Understanding results and output

Batch evaluation results come in two layers: aggregate summaries in the API response, and per-session detail in CloudWatch Logs.

Aggregate results

When a batch evaluation completes, the GetBatchEvaluation response includes an evaluationResults object with aggregate summaries.

Session counts

Per-evaluator summaries

Each entry in evaluatorSummaries provides aggregate metrics for one evaluator:

Example response

{
    "batchEvaluationId": "12345678-1234-1234-1234-123456789012",
    "status": "COMPLETED",
    "evaluationResults": {
        "numberOfSessionsCompleted": 47,
        "numberOfSessionsFailed": 3,
        "numberOfSessionsIgnored": 0,
        "totalNumberOfSessions": 50,
        "evaluatorSummaries": [
            {
                "evaluatorId": "Builtin.GoalSuccessRate",
                "statistics": {
                    "averageScore": 0.72
                },
                "totalEvaluated": 47,
                "totalFailed": 0
            },
            {
                "evaluatorId": "Builtin.Helpfulness",
                "statistics": {
                    "averageScore": 0.81
                },
                "totalEvaluated": 47,
                "totalFailed": 0
            }
        ]
    }
}

Per-session detail in CloudWatch Logs

The outputConfig field in the GetBatchEvaluation response specifies a CloudWatch Logs location where per-session, per-evaluator results are written as OpenTelemetry events.

{
    "outputConfig": {
        "cloudWatchConfig": {
            "logGroupName": "/aws/bedrock-agentcore/evaluations/batch",
            "logStreamName": "12345678-1234-1234-1234-123456789012"
        }
    }
}

Each event in the log stream contains per-turn, per-evaluator detail:

To read these events, use the CloudWatch Logs API:

import boto3

logs_client = boto3.client("logs", region_name="us-west-2")

response = logs_client.get_log_events(
    logGroupName="/aws/bedrock-agentcore/evaluations/batch",
    logStreamName="12345678-1234-1234-1234-123456789012",
)

for event in response["events"]:
    print(event["message"])

Interpreting scores

Batch evaluation scores follow the same conventions as on-demand evaluation:

Error handling

Job-level errors

If the batch evaluation job fails entirely, the status is FAILED and errorDetails contains one or more error messages describing what went wrong. Common causes:

Session-level errors

Individual sessions can fail while the overall job succeeds. The numberOfSessionsFailed count in evaluationResults indicates how many sessions had errors. Per-session errors are recorded in the CloudWatch Logs output.

Evaluator-level errors

Within a successfully evaluated session, individual evaluators can fail. The totalFailed count on each evaluator summary indicates how many sessions that evaluator could not score. Common causes include malformed spans or missing required attributes.

Comparing results across runs

A common workflow is to run batch evaluation before and after a change (prompt update, model swap, tool modification) and compare the aggregate scores:

# After running two batch evaluations
baseline = client.get_batch_evaluation(batchEvaluationId=baseline_id)
treatment = client.get_batch_evaluation(batchEvaluationId=treatment_id)

baseline_summaries = {
    s["evaluatorId"]: s["statistics"]["averageScore"]
    for s in baseline["evaluationResults"]["evaluatorSummaries"]
}
treatment_summaries = {
    s["evaluatorId"]: s["statistics"]["averageScore"]
    for s in treatment["evaluationResults"]["evaluatorSummaries"]
}

print(f"{'Evaluator':<35} {'Baseline':>10} {'Treatment':>10} {'Delta':>10}")
print("=" * 67)
for eid in baseline_summaries:
    b = baseline_summaries[eid]
    t = treatment_summaries.get(eid, 0)
    delta = t - b
    print(f"{eid:<35} {b:>10.4f} {t:>10.4f} {delta:>+10.4f}")