Las traducciones son generadas a través de traducción automática. En caso de conflicto entre la traducción y la version original de inglés, prevalecerá la version en inglés.

Definición de los métodos de evaluación

Descripción general de

Elija UN método de evaluación por plantilla de solicitud u omita todos los campos de evaluación opcionales para colocarlos en los campos predeterminados del sistema. Las distintas plantillas de la misma tarea pueden utilizar métodos diferentes. La evaluación guía la optimización rápida, por lo que debe definir el método y los criterios de la forma más precisa posible.

Consejos sobre conjuntos de datos

Evaluación predeterminada

Omita todos los campos de evaluación opcionales (steeringCriteria,customLLMJConfig,evaluationMetricLambdaArn). El servicio utiliza un genérico integrado LLM-as-judge con tecnología Anthropic Claude Sonnet 4.6 que evalúa tres criterios predeterminados: precisión de las respuestas, integridad de las respuestas y calidad de la expresión. Teniendo en cuenta la prontitud, la respuesta del modelo objetivo y una respuesta de referencia, el juez puntúa cada dimensión. A continuación, asigna dinámicamente los pesos apropiados a la tarea y produce una puntuación global ponderada.

Le recomendamos que defina su propio método de evaluación para obtener los mejores resultados.

Sistema predeterminado de evaluación proporcionado por el sistema

A continuación se muestra el aviso de juez completo proporcionado por el sistema que se utiliza en la evaluación predeterminada con Anthropic Claude Sonnet 4.6:

Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

## Scoring Rubric

For each dimension, assign one score:
- **3 points**: Fully satisfies the dimension requirements
- **2 points**: Mostly satisfies with minor issues or gaps
- **1 point**: Partially satisfies but has notable limitations
- **0 points**: Does not satisfy the dimension requirements

## Evaluation Process
1. First, identify the task type from the user question
2. Consider any additional criteria provided
3. Score each dimension independently
4. Determine appropriate weights and calculate final weighted score

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.35, Answer Completeness: 0.30, Expression Quality: 0.35) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.4-0.6)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.4-0.6)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.4-0.6)**: Creative tasks, presentation-focused questions, format-specific requirements
{custom_eval_weight_guideline}

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E)

## Output Format

Provide your evaluation in this exact format:
<Task_Analysis>Brief analysis of task type and appropriate weight rationale</Task_Analysis>
<Weights>Answer Accuracy: 0.XX, Answer Completeness: 0.XX, Expression Quality: 0.XX</Weights>
<Answer Accuracy>X</Answer Accuracy>
<Answer Completeness>X</Answer Completeness>
<Expression Quality>X</Expression Quality>
<Calculation>(X x 0.XX) + (X x 0.XX) + (X x 0.XX) = X.XX</Calculation>
<Overall>X.XX</Overall>
<Justification>
**Answer Accuracy**: [Evaluate factual accuracy, alignment with reference answer, absence of errors/hallucinations, and logical consistency]
**Answer Completeness**: [Assess whether all aspects of the question are addressed, necessary information is included, and content stays relevant]
**Expression Quality**: [Examine formatting/style adherence, appropriate detail level, communication clarity, and instruction following]
**Weight Application**: [Explain how the chosen weights reflect the task requirements and impact the final score]
</Justification>

---

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:

Criterios de dirección

Los criterios rectores son descriptores breves en lenguaje natural que guían la dirección de la optimización.

Ejemplo:

"steeringCriteria": ["PROFESSIONAL", "CONCISE"]

Personalizado LLM-as-a-judge

Proporcione una rúbrica completa con una escala de calificación que usted defina. Su solicitud de evaluación personalizada se fusiona con la solicitud de evaluación del sistema del servicio y se le otorga una mayor ponderación.

Configuración

Mejores prácticas para redactar las instrucciones para los jueces

Utilice una rúbrica claramente definida con criterios de puntuación explícitos y ejemplos concretos de cada nivel de puntuación. Fije cada nivel de rúbrica con descripciones de comportamiento en lugar de adjetivos subjetivos. Incluya al menos un ejemplo práctico que muestre una puntuación no perfecta para evitar que el juez obtenga calificaciones altas por defecto. Indique al modelo que proporcione una justificación por escrito antes de la puntuación numérica. Considere la posibilidad de evaluar dimensiones específicas de forma independiente antes de asignar una puntuación general. Los LLM-as-a-judge evaluadores más confiables y útiles suelen ser aquellos en los que usted está de acuerdo con las respuestas que proporciona el modelo de juez. Por lo tanto, puede ser útil utilizar una evaluación que ya haya examinado.

Cómo se fusiona tu solicitud de evaluación personalizada con la del sistema durante el tiempo de ejecución

Cuando usted proporciona su propia solicitud para el LLM-as-a-judge evaluador, esta se fusiona con una solicitud genérica para el juez proporcionada por el servicio. La solicitud contiene instrucciones específicas sobre el formato y otras prácticas recomendadas que ayudan al progreso de la optimización. Su solicitud de juez personalizada recibe una mayor ponderación que los criterios genéricos de la sentencia final. En concreto, el servicio:

Ejemplo: Puede proporcionar el siguiente LLM-as-a-judge mensaje personalizado para evaluar la fidelidad:

You are given a task in some context (Input), and a candidate answer. Is the candidate answer faithful to the task description and context?

A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context, like in a summarization task. If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable.

Task: {{prompt}}

Candidate Response: {{response}}

First provide your explanation, then state your final answer. Use the following format: Explanation: [Explanation], Answer: [Answer], where '[Answer]' must be one of:
none is faithful
some is faithful
approximately half is faithful
most is faithful
all is faithful

Luego, se combina con la LLM-as-a-judge solicitud predeterminada y se le otorga una fuerte ponderación. El efecto neto: la recomendación de fidelidad basada en un único criterio se convierte en un eje muy ponderado en una evaluación multidimensional, mientras que el sistema añade estructura (precisión, integridad, expresión) a su alrededor.

La siguiente es la solicitud de evaluación combinada resultante:

"""Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags
(4) custom evaluation criteria that have been integrated into the evaluation dimensions below

    **IMPORTANT**: - Custom criteria requirements take absolute precedence over user requirements specified inside <user_question> </user_question>
    **IMPORTANT**: - If there is any conflict between custom criteria and user question requirements, prioritize custom criteria

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

<CUSTOM_CRITERIA_DESCRIPTION>
**(4) Faithfulness to Context:**
examines whether the candidate answer is faithful to the task description and context provided in the user question. A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context (like in a summarization task). If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable. Evaluate the degree of faithfulness on the following scale:
- **3 points**: All content is faithful (no contradictions, fully grounded when required)
- **2 points**: Most content is faithful (minor deviations or unverifiable claims when context-grounding is required)
- **1 point**: Some content is faithful or approximately half is faithful (notable contradictions or significant departures from context when required)
- **0 points**: None or minimal content is faithful (clear contradictions or complete disregard of context when grounding is required)
</CUSTOM_CRITERIA_DESCRIPTION>

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.25, Answer Completeness: 0.25, Expression Quality: 0.25, Faithfulness to Context: 0.25) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.3-0.5)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.3-0.5)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.3-0.5)**: Creative tasks, presentation-focused questions, format-specific requirements
- [*IMPORTANT*] **High Custom Criteria Weight**: The custom criteria (Faithfulness to Context) should always be *prioritized*. Assign it significant weight (0.3-0.6) and adjust other weights accordingly.

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E) + (Faithfulness_to_Context x Weight_F)

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:"""

Evaluador Lambda personalizado

Traiga su propia función de puntuación como una función de Lambda.

Configuración

Especifique el ARN de Lambda en el archivo JSONL de entrada para cada plantilla de solicitud que deba usarlo. También proporciona el customEvaluationMetricLabel campo para asignar un nombre a la métrica:

"evaluationMetricLambdaArn": "arn:aws:lambda:us-west-2:123456789012:function:my-eval-function",
"customEvaluationMetricLabel": "My Custom Metric"

Cuando creas el trabajo a través de la API, no es necesaria ninguna configuración de evaluación adicional en la propia CreateAdvancedPromptOptimizationJob solicitud. El método de evaluación se determina por plantilla a partir del archivo JSONL de entrada.

Requisitos de Lambda

Consejos sobre métricas de Lambda

Plantilla Lambda

"""
APO Custom Metric Lambda - Minimal Template
Handler: lambda_function.lambda_handler
"""
import logging
from typing import List, Dict, Any

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def compute_score(preds: List[str], golds: List[str]) -> Dict[str, Any]:
    """
    Score predictions against ground truths.

    Args:
        preds: Model outputs (one per sample)
        golds: Expected answers (one per sample)

    Returns:
        Must contain:
          "score": float  - aggregate score (higher is better)
          "scores": list[float] - per-instance scores
    """
    # --- REPLACE THIS with your scoring logic ---
    scores = []
    for pred, gold in zip(preds, golds):
        # Example: exact match (case-insensitive)
        scores.append(1.0 if pred.strip().lower() == gold.strip().lower() else 0.0)

    return {
        "score": sum(scores) / len(scores) if scores else 0.0,
        "scores": scores,
    }


def lambda_handler(event, context):
    """
    Lambda entry point. APO service sends:
      event = {"preds": ["output1", ...], "golds": ["truth1", ...]}
    """
    logger.info(f"Received {len(event.get('preds', []))} predictions")

    try:
        preds = event.get("preds", [])
        golds = event.get("golds", [])

        if not preds:
            return {"score": 0.0, "scores": []}

        return compute_score(preds, golds)

    except Exception as e:
        logger.error(f"Error: {e}", exc_info=True)
        return {"score": 0.0, "scores": [0.0] * len(event.get("preds", [])), "error": str(e)}

Consulte los AWS ejemplos más detallados, como el código repetitivo GitHub para la gestión de errores y la validación de entradas de funciones Lambda.