本文属于机器翻译版本。若本译文内容与英语原文存在差异，则一律以英文原文为准。

定义评估方法

概述

为每个提示模板选择一种评估方法，或省略所有可选的评估字段作为系统默认值。同一任务中的不同模板可以使用不同的方法。评估引导即时优化，因此请尽可能精确地定义您的方法和标准。

数据集提示

默认评估

省略所有可选的评估字段 (steeringCriteria、customLLMJConfig、evaluationMetricLambdaArn)。该服务使用 LLM-as-judge 由 Anthropic Claude Sonnet 4.6 提供支持的内置泛型，用于评估三个默认标准：答案准确性、答案完整性和表达质量。根据提示、目标模型的答案和参考答案，法官对每个维度进行评分。然后，它会动态地为任务分配适当的权重，并生成加权的总体分数。

我们建议您定义自己的评估方法以获得最佳结果。

系统提供的默认裁判提示

以下是 Anthropic Claude Sonnet 4.6 的默认评估中使用的完整系统提供的判断提示：

Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

## Scoring Rubric

For each dimension, assign one score:
- **3 points**: Fully satisfies the dimension requirements
- **2 points**: Mostly satisfies with minor issues or gaps
- **1 point**: Partially satisfies but has notable limitations
- **0 points**: Does not satisfy the dimension requirements

## Evaluation Process
1. First, identify the task type from the user question
2. Consider any additional criteria provided
3. Score each dimension independently
4. Determine appropriate weights and calculate final weighted score

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.35, Answer Completeness: 0.30, Expression Quality: 0.35) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.4-0.6)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.4-0.6)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.4-0.6)**: Creative tasks, presentation-focused questions, format-specific requirements
{custom_eval_weight_guideline}

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E)

## Output Format

Provide your evaluation in this exact format:
<Task_Analysis>Brief analysis of task type and appropriate weight rationale</Task_Analysis>
<Weights>Answer Accuracy: 0.XX, Answer Completeness: 0.XX, Expression Quality: 0.XX</Weights>
<Answer Accuracy>X</Answer Accuracy>
<Answer Completeness>X</Answer Completeness>
<Expression Quality>X</Expression Quality>
<Calculation>(X x 0.XX) + (X x 0.XX) + (X x 0.XX) = X.XX</Calculation>
<Overall>X.XX</Overall>
<Justification>
**Answer Accuracy**: [Evaluate factual accuracy, alignment with reference answer, absence of errors/hallucinations, and logical consistency]
**Answer Completeness**: [Assess whether all aspects of the question are addressed, necessary information is included, and content stays relevant]
**Expression Quality**: [Examine formatting/style adherence, appropriate detail level, communication clarity, and instruction following]
**Weight Application**: [Explain how the chosen weights reflect the task requirements and impact the final score]
</Justification>

---

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:

指导标准

指导标准是指引优化方向的简短自然语言描述符。

示例：

"steeringCriteria": ["PROFESSIONAL", "CONCISE"]

自定义 LLM-as-a-judge

提供包含您定义的评分标准的完整评分量规。您的自定义判断提示将与服务的系统判断提示合并，并赋予更高的权重。

配置

撰写法官提示的最佳做法

使用明确定义的评分量规，其中包含明确的评分标准和每个分数级别的具体示例。用行为描述而不是主观形容词来固定每个评分量规级别。包括至少一个显示不完美分数的有效示例，以调整法官从违约到高评分的偏差。指示模型在得出数字分数之前提供书面理由。在分配总分之前，可以考虑独立评估特定维度。最值得信赖和最有用的 LLM-as-a-judge 评估人员通常是那些你同意法官模型提供的答案的评估者。因此，使用您已经审核过的评估可能会有所帮助。

您的自定义判断提示在运行时如何与系统提示合并

当你提供自己的 LLM-as-a-judge 评估员提示时，它会与服务提供的通用法官提示合并。该提示包含有关格式的具体说明和其他有助于优化进展的最佳实践。您的自定义裁判提示比最终判决中的通用标准更具权重。具体而言，该服务：

示例：您可以提供以下自定义 LLM-as-a-judge 提示来评估忠诚度：

You are given a task in some context (Input), and a candidate answer. Is the candidate answer faithful to the task description and context?

A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context, like in a summarization task. If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable.

Task: {{prompt}}

Candidate Response: {{response}}

First provide your explanation, then state your final answer. Use the following format: Explanation: [Explanation], Answer: [Answer], where '[Answer]' must be one of:
none is faithful
some is faithful
approximately half is faithful
most is faithful
all is faithful

然后将其与默认 LLM-as-a-judge 提示合并，并赋予较强的权重。净效应：在多维度评估中，你的单一标准忠诚度提示变为一个权重轴，而系统会围绕它添加结构（准确性、完整性、表达性）。

以下是合并后的法官提示音：

"""Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags
(4) custom evaluation criteria that have been integrated into the evaluation dimensions below

    **IMPORTANT**: - Custom criteria requirements take absolute precedence over user requirements specified inside <user_question> </user_question>
    **IMPORTANT**: - If there is any conflict between custom criteria and user question requirements, prioritize custom criteria

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

<CUSTOM_CRITERIA_DESCRIPTION>
**(4) Faithfulness to Context:**
examines whether the candidate answer is faithful to the task description and context provided in the user question. A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context (like in a summarization task). If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable. Evaluate the degree of faithfulness on the following scale:
- **3 points**: All content is faithful (no contradictions, fully grounded when required)
- **2 points**: Most content is faithful (minor deviations or unverifiable claims when context-grounding is required)
- **1 point**: Some content is faithful or approximately half is faithful (notable contradictions or significant departures from context when required)
- **0 points**: None or minimal content is faithful (clear contradictions or complete disregard of context when grounding is required)
</CUSTOM_CRITERIA_DESCRIPTION>

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.25, Answer Completeness: 0.25, Expression Quality: 0.25, Faithfulness to Context: 0.25) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.3-0.5)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.3-0.5)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.3-0.5)**: Creative tasks, presentation-focused questions, format-specific requirements
- [*IMPORTANT*] **High Custom Criteria Weight**: The custom criteria (Faithfulness to Context) should always be *prioritized*. Assign it significant weight (0.3-0.6) and adjust other weights accordingly.

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E) + (Faithfulness_to_Context x Weight_F)

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:"""

自定义 Lambda 评估器

将您自己的评分函数作为 Lambda 函数引入。

配置

在您的输入 JSONL 文件中为应使用它的每个提示模板指定 Lambda ARN。您还要提供用于命名指标的customEvaluationMetricLabel字段：

"evaluationMetricLambdaArn": "arn:aws:lambda:us-west-2:123456789012:function:my-eval-function",
"customEvaluationMetricLabel": "My Custom Metric"

当您通过 API 创建任务时，CreateAdvancedPromptOptimizationJob请求本身不需要额外的评估配置。评估方法由输入 JSONL 文件根据每个模板确定。

Lambda 要求

Lambda 指标提示

Lambda 模板

"""
APO Custom Metric Lambda - Minimal Template
Handler: lambda_function.lambda_handler
"""
import logging
from typing import List, Dict, Any

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def compute_score(preds: List[str], golds: List[str]) -> Dict[str, Any]:
    """
    Score predictions against ground truths.

    Args:
        preds: Model outputs (one per sample)
        golds: Expected answers (one per sample)

    Returns:
        Must contain:
          "score": float  - aggregate score (higher is better)
          "scores": list[float] - per-instance scores
    """
    # --- REPLACE THIS with your scoring logic ---
    scores = []
    for pred, gold in zip(preds, golds):
        # Example: exact match (case-insensitive)
        scores.append(1.0 if pred.strip().lower() == gold.strip().lower() else 0.0)

    return {
        "score": sum(scores) / len(scores) if scores else 0.0,
        "scores": scores,
    }


def lambda_handler(event, context):
    """
    Lambda entry point. APO service sends:
      event = {"preds": ["output1", ...], "golds": ["truth1", ...]}
    """
    logger.info(f"Received {len(event.get('preds', []))} predictions")

    try:
        preds = event.get("preds", [])
        golds = event.get("golds", [])

        if not preds:
            return {"score": 0.0, "scores": []}

        return compute_score(preds, golds)

    except Exception as e:
        logger.error(f"Error: {e}", exc_info=True)
        return {"score": 0.0, "scores": [0.0] * len(event.get("preds", [])), "error": str(e)}

GitHub 有关更多详细示例，包括错误处理和 Lambda 函数输入验证的 AWS 样板代码，请参阅示例。