Terjemahan disediakan oleh mesin penerjemah. Jika konten terjemahan yang diberikan bertentangan dengan versi bahasa Inggris aslinya, utamakan versi bahasa Inggris.

Tentukan metode evaluasi

Gambaran umum

Pilih SATU metode evaluasi per templat prompt, atau hilangkan semua bidang evaluasi opsional untuk default sistem. Template yang berbeda dalam pekerjaan yang sama dapat menggunakan metode yang berbeda. Evaluasi mengarahkan pengoptimalan yang cepat, jadi tentukan metode dan kriteria Anda setepat mungkin.

Kiat kumpulan data

Evaluasi default

Hilangkan semua bidang evaluasi opsional (steeringCriteria,customLLMJConfig,evaluationMetricLambdaArn). Layanan ini menggunakan generik bawaan yang LLM-as-judge didukung oleh Anthropic Claude Sonnet 4.6 yang mengevaluasi tiga kriteria default: Akurasi Jawaban, Kelengkapan Jawaban, dan Kualitas Ekspresi. Mengingat prompt, jawaban model target, dan jawaban referensi, juri menilai setiap dimensi. Kemudian secara dinamis menetapkan bobot yang sesuai untuk tugas, dan menghasilkan skor Keseluruhan tertimbang.

Sebaiknya tentukan metode evaluasi Anda sendiri untuk hasil terbaik.

Prompt hakim yang disediakan sistem default

Berikut ini adalah prompt hakim yang disediakan sistem lengkap yang digunakan oleh evaluasi default dengan Anthropic Claude Sonnet 4.6:

Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

## Scoring Rubric

For each dimension, assign one score:
- **3 points**: Fully satisfies the dimension requirements
- **2 points**: Mostly satisfies with minor issues or gaps
- **1 point**: Partially satisfies but has notable limitations
- **0 points**: Does not satisfy the dimension requirements

## Evaluation Process
1. First, identify the task type from the user question
2. Consider any additional criteria provided
3. Score each dimension independently
4. Determine appropriate weights and calculate final weighted score

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.35, Answer Completeness: 0.30, Expression Quality: 0.35) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.4-0.6)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.4-0.6)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.4-0.6)**: Creative tasks, presentation-focused questions, format-specific requirements
{custom_eval_weight_guideline}

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E)

## Output Format

Provide your evaluation in this exact format:
<Task_Analysis>Brief analysis of task type and appropriate weight rationale</Task_Analysis>
<Weights>Answer Accuracy: 0.XX, Answer Completeness: 0.XX, Expression Quality: 0.XX</Weights>
<Answer Accuracy>X</Answer Accuracy>
<Answer Completeness>X</Answer Completeness>
<Expression Quality>X</Expression Quality>
<Calculation>(X x 0.XX) + (X x 0.XX) + (X x 0.XX) = X.XX</Calculation>
<Overall>X.XX</Overall>
<Justification>
**Answer Accuracy**: [Evaluate factual accuracy, alignment with reference answer, absence of errors/hallucinations, and logical consistency]
**Answer Completeness**: [Assess whether all aspects of the question are addressed, necessary information is included, and content stays relevant]
**Expression Quality**: [Examine formatting/style adherence, appropriate detail level, communication clarity, and instruction following]
**Weight Application**: [Explain how the chosen weights reflect the task requirements and impact the final score]
</Justification>

---

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:

Kriteria kemudi

Kriteria pengarahan adalah deskriptor bahasa alami pendek yang memandu arah pengoptimalan.

Contoh:

"steeringCriteria": ["PROFESSIONAL", "CONCISE"]

Kustom LLM-as-a-judge

Berikan rubrik lengkap dengan skala penilaian yang Anda tentukan. Prompt juri kustom Anda digabungkan dengan prompt juri sistem layanan dan diberi bobot yang lebih kuat.

Konfigurasi

Praktik terbaik untuk menulis petunjuk juri

Gunakan rubrik yang jelas dengan kriteria penilaian eksplisit dan contoh konkret dari setiap tingkat skor. Jangkarkan setiap tingkat rubrik dengan deskripsi perilaku daripada kata sifat subjektif. Sertakan setidaknya satu contoh kerja yang menunjukkan skor yang tidak sempurna untuk mengkalibrasi juri dari default ke peringkat tinggi. Instruksikan model untuk memberikan justifikasi tertulis sebelum skor numerik. Pertimbangkan untuk mengevaluasi dimensi tertentu secara independen sebelum menetapkan skor keseluruhan. LLM-as-a-judge Evaluator yang paling tepercaya dan membantu biasanya adalah orang-orang di mana Anda setuju dengan jawaban yang diberikan model hakim. Oleh karena itu, mungkin membantu untuk menggunakan evaluasi yang telah Anda periksa.

Bagaimana prompt juri kustom Anda digabungkan dengan prompt sistem saat runtime

Ketika Anda memberikan prompt LLM-as-a-judge evaluator Anda sendiri, itu digabungkan dengan prompt hakim yang disediakan layanan generik. Prompt berisi instruksi spesifik tentang pemformatan dan praktik terbaik lainnya yang membantu kemajuan pengoptimalan. Prompt hakim kustom Anda diberi bobot yang lebih kuat daripada kriteria umum dalam penilaian akhir. Secara khusus, layanan:

Contoh: Anda dapat memberikan perintah khusus berikut yang LLM-as-a-judge mengevaluasi kesetiaan:

You are given a task in some context (Input), and a candidate answer. Is the candidate answer faithful to the task description and context?

A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context, like in a summarization task. If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable.

Task: {{prompt}}

Candidate Response: {{response}}

First provide your explanation, then state your final answer. Use the following format: Explanation: [Explanation], Answer: [Answer], where '[Answer]' must be one of:
none is faithful
some is faithful
approximately half is faithful
most is faithful
all is faithful

Ini kemudian digabungkan dengan LLM-as-a-judge prompt default dan diberi bobot yang kuat. Efek bersih: prompt kesetiaan kriteria tunggal Anda menjadi satu sumbu yang sangat tertimbang dalam evaluasi multi-dimensi, sementara sistem menambahkan struktur (Akurasi, Kelengkapan, Ekspresi) di sekitarnya.

Berikut ini adalah prompt hakim gabungan yang dihasilkan:

"""Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer.

You will be given:
(1) a user question, enclosed in <user_question></user_question> tags
(2) an answer, enclosed in <answer></answer> tags
(3) a reference answer, enclosed in <reference_answer></reference_answer> tags
(4) custom evaluation criteria that have been integrated into the evaluation dimensions below

    **IMPORTANT**: - Custom criteria requirements take absolute precedence over user requirements specified inside <user_question> </user_question>
    **IMPORTANT**: - If there is any conflict between custom criteria and user question requirements, prioritize custom criteria

## Universal Evaluation Dimensions

Evaluate the answer across these core dimensions:

**(1) Answer Accuracy:**
examines correctness, consistency, and factuality alignment between the <answer> and the <user_question>;
examines if the <answer> contains irrelevant or wrongful information/hallucination.

**(2) Answer Completeness:**
examines if the <answer> is fully addressing the <user_question>;
examines if the <answer> is good at relevance/informativeness: selection of important/key content from <user_question>

**(3) Expression Quality:**
examines if the <answer> is concise at answering the <user_question>.
NOTE that unless there is special instruction, more concise <answer> is always better,
and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT!
examines the alignment on instruction following, e.g., if the <answer> adheres
to both explicit guidelines and implicit guidelines (like few-shot examples) in the <user_question>;

<CUSTOM_CRITERIA_DESCRIPTION>
**(4) Faithfulness to Context:**
examines whether the candidate answer is faithful to the task description and context provided in the user question. A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context (like in a summarization task). If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable. Evaluate the degree of faithfulness on the following scale:
- **3 points**: All content is faithful (no contradictions, fully grounded when required)
- **2 points**: Most content is faithful (minor deviations or unverifiable claims when context-grounding is required)
- **1 point**: Some content is faithful or approximately half is faithful (notable contradictions or significant departures from context when required)
- **0 points**: None or minimal content is faithful (clear contradictions or complete disregard of context when grounding is required)
</CUSTOM_CRITERIA_DESCRIPTION>

## Dimension Weighting and Final Scoring

**Weight Determination Process:**
Assign weights (must sum to 1.0) based on:
- Explicit weights in evaluation_criteria (if provided)
- Task analysis and question requirements (if no explicit weights)
- Default weights (Answer Accuracy: 0.25, Answer Completeness: 0.25, Expression Quality: 0.25, Faithfulness to Context: 0.25) as fallback

**Weight Guidelines:**
- **High Accuracy Weight (0.3-0.5)**: Factual questions, multiple choice, technical problems
- **High Completeness Weight (0.3-0.5)**: Complex explanatory tasks, multi-part questions
- **High Expression Weight (0.3-0.5)**: Creative tasks, presentation-focused questions, format-specific requirements
- [*IMPORTANT*] **High Custom Criteria Weight**: The custom criteria (Faithfulness to Context) should always be *prioritized*. Assign it significant weight (0.3-0.6) and adjust other weights accordingly.

**Overall Score Calculation:**
Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E) + (Faithfulness_to_Context x Weight_F)

## Current Evaluation Task

<user_question>
{prompt}
</user_question>
<answer>
{prediction}
</answer>
<reference_answer>
{gold}
</reference_answer>

Based on the above guidelines and criteria, provide your evaluation:"""

Evaluator Lambda khusus

Bawa fungsi penilaian Anda sendiri sebagai fungsi Lambda.

Konfigurasi

Tentukan Lambda ARN dalam file JSONL input Anda untuk setiap template prompt yang harus menggunakannya. Anda juga menyediakan customEvaluationMetricLabel bidang untuk memberi nama metrik Anda:

"evaluationMetricLambdaArn": "arn:aws:lambda:us-west-2:123456789012:function:my-eval-function",
"customEvaluationMetricLabel": "My Custom Metric"

Saat Anda membuat pekerjaan melalui API, tidak diperlukan konfigurasi evaluasi tambahan dalam CreateAdvancedPromptOptimizationJob permintaan itu sendiri. Metode evaluasi ditentukan per template dari input file JSONL.

Persyaratan Lambda

Kiat metrik lambda

Templat Lambda

"""
APO Custom Metric Lambda - Minimal Template
Handler: lambda_function.lambda_handler
"""
import logging
from typing import List, Dict, Any

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def compute_score(preds: List[str], golds: List[str]) -> Dict[str, Any]:
    """
    Score predictions against ground truths.

    Args:
        preds: Model outputs (one per sample)
        golds: Expected answers (one per sample)

    Returns:
        Must contain:
          "score": float  - aggregate score (higher is better)
          "scores": list[float] - per-instance scores
    """
    # --- REPLACE THIS with your scoring logic ---
    scores = []
    for pred, gold in zip(preds, golds):
        # Example: exact match (case-insensitive)
        scores.append(1.0 if pred.strip().lower() == gold.strip().lower() else 0.0)

    return {
        "score": sum(scores) / len(scores) if scores else 0.0,
        "scores": scores,
    }


def lambda_handler(event, context):
    """
    Lambda entry point. APO service sends:
      event = {"preds": ["output1", ...], "golds": ["truth1", ...]}
    """
    logger.info(f"Received {len(event.get('preds', []))} predictions")

    try:
        preds = event.get("preds", [])
        golds = event.get("golds", [])

        if not preds:
            return {"score": 0.0, "scores": []}

        return compute_score(preds, golds)

    except Exception as e:
        logger.error(f"Error: {e}", exc_info=True)
        return {"score": 0.0, "scores": [0.0] * len(event.get("preds", [])), "error": str(e)}

Lihat Sam AWS pel GitHub untuk contoh yang lebih rinci termasuk kode boilerplate untuk penanganan kesalahan dan validasi input fungsi Lambda.