建立評估者

agentcore add evaluator \
  --name "your_custom_evaluator_name" \
  --config custom_evaluator_config.json \
  --level "TRACE"

此命令會將評估器新增至您的本機agentcore.json組態。執行 agentcore deploy 在您的 AWS 帳戶中建立它。

輸入自訂評估器的名稱。

選取評估層級：工作階段、追蹤或工具呼叫。

選擇 LLM 判斷模型進行評估。

輸入您的評估指示。提示必須至少包含一個預留位置：{context}用於對話歷史記錄或{available_tools}用於工具清單。

選取評分比例預設集或定義自訂比例。

檢閱評估器組態，然後按 Enter 鍵確認。

import json
from bedrock_agentcore_starter_toolkit import Evaluation

eval_client = Evaluation()

# Load the configuration JSON file
with open('custom_evaluator_config.json') as f:
    evaluator_config = json.load(f)

# Create the custom evaluator
custom_evaluator = eval_client.create_evaluator(
    name="your_custom_evaluator_name",
    level="TRACE",
    description="Response quality evaluator",
    config=evaluator_config
)

import boto3
import json

client = boto3.client('bedrock-agentcore-control')

# Load the configuration JSON file
with open('custom_evaluator_config.json') as f:
    evaluator_config = json.load(f)

# Create the custom evaluator
response = client.create_evaluator(
    evaluatorName="your_custom_evaluator_name",
    level="TRACE",
    evaluatorConfig=evaluator_config
)

aws bedrock-agentcore-control create-evaluator \
    --evaluator-name 'your_custom_evaluator_name' \
    --level TRACE \
    --evaluator-config file://custom_evaluator_config.json

此評估器使用 LLM 來比較預期和實際的工具軌跡，允許細微判斷，例如，容忍次要偏差，例如額外的協助工具呼叫。它使用 expected_tool_trajectory和 actual_tool_trajectory 預留位置。

將下列項目儲存為 trajectory_compliance_config.json ：

{
  "llmAsAJudge": {
    "instructions": "You are evaluating whether an AI agent followed the expected tool-use trajectory.\n\nExpected trajectory (ordered list of tool names):\n{expected_tool_trajectory}\n\nActual trajectory (ordered list of tool names the agent used):\n{actual_tool_trajectory}\n\nFull session context:\n{context}\n\nAvailable tools:\n{available_tools}\n\nCompare the expected and actual trajectories. Consider whether the agent called the right tools in the right order. Minor deviations (e.g., an extra logging tool call) are acceptable if the core trajectory is preserved.",
    "ratingScale": {
      "numerical": [
        { "label": "No Match",      "value": 0.0, "definition": "The actual trajectory has no meaningful overlap with the expected trajectory" },
        { "label": "Partial Match", "value": 0.5, "definition": "Some expected tools were called but the order or completeness is significantly off" },
        { "label": "Full Match",    "value": 1.0, "definition": "The actual trajectory matches the expected trajectory in order and completeness" }
      ]
    },
    "modelConfig": {
      "bedrockEvaluatorModelConfig": {
        "modelId": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
        "inferenceConfig": { "maxTokens": 512, "temperature": 0.0 }
      }
    }
  }
}

建立評估者：

aws bedrock-agentcore-control create-evaluator \
  --evaluator-name 'TrajectoryCompliance' \
  --level SESSION \
  --description 'Evaluates whether the agent followed the expected tool trajectory.' \
  --evaluator-config file://trajectory_compliance_config.json

此評估者會檢查客服人員的行為是否符合一組聲明，並傳回分類 PASS/FAIL/INCONCLUSIVE 判斷。它使用assertions預留位置以及 context和 available_tools。

將下列項目儲存為 assertion_checker_config.json ：

{
  "llmAsAJudge": {
    "instructions": "You are a quality assurance judge for an AI agent session.\n\nSession context (full conversation history):\n{context}\n\nAvailable tools:\n{available_tools}\n\nAssertions to verify:\n{assertions}\n\nFor each assertion, determine if the session satisfies it. The overall verdict should be PASS only if ALL assertions are satisfied. If any assertion fails, the verdict is FAIL. If the session data is insufficient to determine, verdict is INCONCLUSIVE.",
    "ratingScale": {
      "categorical": [
        { "label": "PASS",         "definition": "All assertions are satisfied by the session" },
        { "label": "FAIL",         "definition": "One or more assertions are not satisfied" },
        { "label": "INCONCLUSIVE", "definition": "Insufficient information to determine assertion satisfaction" }
      ]
    },
    "modelConfig": {
      "bedrockEvaluatorModelConfig": {
        "modelId": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
        "inferenceConfig": { "maxTokens": 1024, "temperature": 0.0 }
      }
    }
  }
}

建立評估者：

aws bedrock-agentcore-control create-evaluator \
  --evaluator-name 'AssertionChecker' \
  --level SESSION \
  --description 'Checks whether the agent session satisfies a set of assertions.' \
  --evaluator-config file://assertion_checker_config.json

此評估器會將代理程式的實際回應與預期的回應進行比較，以語意相似度評分。它使用expected_response預留位置在評估時接收基本事實。

將下列項目儲存為 response_similarity_config.json ：

{
  "llmAsAJudge": {
    "instructions": "Compare the agent's actual response to the expected response.\n\nConversation context:\n{context}\n\nAgent's actual response:\n{assistant_turn}\n\nExpected response:\n{expected_response}\n\nEvaluate semantic similarity. The agent does not need to match word-for-word, but the meaning, key facts, and intent should align. Penalize missing critical information or contradictions.",
    "ratingScale": {
      "numerical": [
        { "label": "No Match",        "value": 0.0,  "definition": "The response contradicts or is completely unrelated to the expected response" },
        { "label": "Low Similarity",   "value": 0.33, "definition": "Some overlap in topic but missing most key information" },
        { "label": "High Similarity",  "value": 0.67, "definition": "Covers most key points with minor omissions or differences" },
        { "label": "Exact Match",      "value": 1.0,  "definition": "Semantically equivalent to the expected response" }
      ]
    },
    "modelConfig": {
      "bedrockEvaluatorModelConfig": {
        "modelId": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
        "inferenceConfig": { "maxTokens": 512, "temperature": 0.0 }
      }
    }
  }
}

建立評估者：

aws bedrock-agentcore-control create-evaluator \
  --evaluator-name 'ResponseSimilarity' \
  --level TRACE \
  --description 'Evaluates how closely the agent response matches the expected response.' \
  --evaluator-config file://response_similarity_config.json