Contoh resep evaluasi

Resep-resep ini memungkinkan Anda mengevaluasi kemampuan mendasar model Amazon Nova di seluruh rangkaian tolok ukur teks saja yang komprehensif. Mereka disediakan dalam formatxxx_general_text_benchmark_eval.yaml.

Resep ini memungkinkan Anda membawa kumpulan data Anda sendiri untuk pembandingan dan membandingkan keluaran model untuk referensi jawaban menggunakan berbagai jenis metrik. Mereka disediakan dalam formatxxx_bring_your_own_dataset_eval.yaml.

Berikut ini adalah persyaratan dataset bawalah Anda sendiri:

Berikut adalah entri contoh kumpulan data Anda sendiri

{
   "system":"You are a english major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}
{
   "system":"You are a pattern analysis specialist that provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}
{
   "system":"You have great attention to detail that follows instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}

Untuk menggunakan kumpulan data kustom Anda, ubah resep evaluasi Anda dengan bidang wajib berikut, jangan ubah konten apa pun:

evaluation:
  task: gen_qa
  strategy: gen_qa
  metric: all

Batasan berikut berlaku:

Amazon Nova LLM sebagai Hakim adalah fitur evaluasi model yang memungkinkan pelanggan membandingkan kualitas tanggapan dari satu model ke respons model dasar pada kumpulan data khusus. Ini mengambil dataset dengan prompt, respons dasar, dan respons penantang, dan menggunakan model Nova Judge untuk memberikan metrik tingkat kemenangan berdasarkan Bradley-Terry probabilitas dengan perbandingan berpasangan.

Resepnya disediakan dalam formatxxx_llm_judge_eval.yaml.

Berikut ini adalah persyaratan LLM sebagai Hakim:

Berikut adalah LLM sebagai contoh entri juri

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}

Untuk menggunakan kumpulan data kustom Anda, ubah resep evaluasi Anda dengan bidang wajib berikut, jangan ubah konten apa pun:

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

Batasan berikut berlaku:

Nova LLM Judge untuk multi-modal (gambar), kependekan dari Nova MM_LLM Judge, adalah fitur evaluasi model yang memungkinkan Anda membandingkan kualitas tanggapan dari satu model terhadap respons model dasar menggunakan kumpulan data khusus. Ini menerima kumpulan data yang berisi prompt, respons dasar, dan respons penantang, dan gambar dalam bentuk Base64-encoded string, kemudian menggunakan model Nova Judge untuk memberikan metrik tingkat kemenangan berdasarkan Bradley-Terry probabilitas melalui perbandingan berpasangan. Format resep:xxx_mm_llm_judge _eval.yaml.

Persyaratan dataset Nova LLM

Format file:

Format skema - Setiap baris dalam .jsonl file harus berupa objek JSON dengan bidang berikut.

Entri contoh

Untuk keterbacaan, contoh berikut mencakup baris dan lekukan baru, tetapi dalam kumpulan data aktual, setiap catatan harus berada pada satu baris.

{
  "prompt": "what is in the image?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    }
  ],
  "response_A": "a dog.",
  "response_B": "a cat.",
}
{
  "prompt": "how many animals in echo of the images?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    },
    {
      "data": "data:image/jpeg;Base64,/DKEafe3gihn..."
    }
  ],
  "response_A": "The first image contains one cat and the second image contains one dog",
  "response_B": "The first image has one aminal and the second has one animal",
}           

Untuk menggunakan kumpulan data kustom Anda, ubah resep evaluasi Anda dengan bidang wajib berikut, jangan ubah konten apa pun:

evaluation:
  task: mm_llm_judge
  strategy: judge
  metric: all

Batasan

Rubrik Judge adalah model LLM-as-a-judge evaluasi yang disempurnakan yang dibangun di atas Nova 2.0 Lite. Tidak seperti model hakim asli yang hanya memberikan putusan preferensi (A>B, B>A, atau seri), Hakim Rubrik secara dinamis menghasilkan kriteria evaluasi khusus yang disesuaikan dengan setiap prompt dan memberikan skor granular di berbagai dimensi.

Kemampuan utama:

Contoh kriteria yang dihasilkan oleh model:

price_validation:
  description: "The response includes validation to ensure price is a positive value."
  type: "scale"
  weight: 0.3

Model mengevaluasi kedua tanggapan terhadap semua kriteria yang dihasilkan, kemudian menggunakan skor tingkat kriteria ini untuk menginformasikan keputusan preferensi akhir.

Konfigurasi resep

Resep juri rubrik

Aktifkan Rubrik Judge dengan task: rubric_llm_judge mengatur resep Anda:

run:
  name: nova-eval-job-name                              # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-2-lite-v1:0:256k              # [FIXED] Rubric Judge model type
  model_name_or_path: "nova-lite-2/prod"                # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job

evaluation:
  task: rubric_llm_judge                                # [FIXED] Evaluation task - enables Rubric Judge
  strategy: judge                                       # [FIXED] Evaluation strategy
  metric: all                                           # [FIXED] Metric calculation method

inference:
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

LLM asli sebagai resep Hakim (untuk perbandingan)

Model juri asli menggunakantask: llm_judge:

run:
  name: eval-job-name                                   # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-micro-v1:0:128k               # [FIXED] Model type
  model_name_or_path: "nova-micro/prod"                 # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job

evaluation:
  task: llm_judge                                       # [FIXED] Original judge task
  strategy: judge                                       # [FIXED] Evaluation strategy
  metric: all                                           # [FIXED] Metric calculation method

inference:
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

Format kumpulan data masukan

Format dataset input identik dengan model juri asli:

Bidang wajib:

Contoh dataset (format JSONL):

{"prompt": "What is the most effective way to combat climate change?", "response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.", "response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."}
{"prompt": "Explain how a computer's CPU works", "response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.", "response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."}
{"prompt": "How does photosynthesis work?", "response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.", "response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."}

Persyaratan format:

Keluaran evaluasi

Struktur keluaran

Rubrik Judge menghasilkan metrik evaluasi yang ditingkatkan dibandingkan dengan model juri asli:

{
  "config_general": {
    "lighteval_sha": "string",
    "num_fewshot_seeds": "int",
    "max_samples": "int | null",
    "job_id": "int",
    "start_time": "float",
    "end_time": "float",
    "total_evaluation_time_secondes": "string",
    "model_name": "string",
    "model_sha": "string",
    "model_dtype": "string | null",
    "model_size": "string"
  },
  "results": {
    "custom|rubric_llm_judge_judge|0": {
      "a_scores": "float",
      "a_scores_stderr": "float",
      "b_scores": "float",
      "b_scores_stderr": "float",
      "ties": "float",
      "ties_stderr": "float",
      "inference_error": "float",
      "inference_error_stderr": "float",
      "score": "float",
      "score_stderr": "float",
      "weighted_score_A": "float",
      "weighted_score_A_stderr": "float",
      "weighted_score_B": "float",
      "weighted_score_B_stderr": "float",
      "score_margin": "float",
      "score_margin_stderr": "float",
      "winrate": "float",
      "lower_rate": "float",
      "upper_rate": "float"
    }
  },
  "versions": {
    "custom|rubric_llm_judge_judge|0": "int"
  }
}

Metrik baru di Rubrik Judge

Enam metrik berikut ini unik untuk Rubrik Judge dan memberikan penilaian kualitas terperinci:

Memahami metrik skor tertimbang

Tujuan: Skor tertimbang memberikan pengukuran kualitas berkelanjutan yang melengkapi putusan preferensi biner, memungkinkan wawasan yang lebih dalam tentang kinerja model.

Perbedaan utama dari hakim asli:

Menafsirkan score_margin:

Kasus penggunaan:

Metodologi perhitungan

Skor tertimbang dihitung melalui proses berikut:

Contoh: Jika response_A memiliki jumlah tertimbang 0,65 dan response_b memiliki 0,78, itu score_margin akan menjadi -0,13, menunjukkan response_b memiliki kualitas 13 poin persentase lebih tinggi di semua kriteria tertimbang.

Dukungan model penalaran

Dukungan model penalaran memungkinkan evaluasi dengan model Nova yang mampu bernalar yang melakukan penalaran internal eksplisit sebelum menghasilkan tanggapan akhir. Fitur ini menggunakan API-level kontrol melalui reasoning_effort parameter untuk mengaktifkan atau menonaktifkan fungsionalitas penalaran secara dinamis, yang berpotensi meningkatkan kualitas respons untuk tugas-tugas analitis yang kompleks.

Model yang didukung:

Konfigurasi resep

Aktifkan penalaran dengan menambahkan reasoning_effort parameter ke inference bagian resep Anda:

run:
  name: eval-job-name                                    # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-2-lite-v1:0:256k               # [FIXED] Must be a reasoning-supported model
  model_name_or_path: nova-lite-2/prod                   # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                            # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                       # [MODIFIABLE] Leave empty for SageMaker Training job; optional for SageMaker HyperPod job
  output_s3_path: ""                                     # [MODIFIABLE] Output path for SageMaker HyperPod job (not compatible with SageMaker Training jobs)

evaluation:
  task: mmlu                                             # [MODIFIABLE] Evaluation task
  strategy: generate                                     # [MODIFIABLE] Evaluation strategy
  metric: all                                            # [MODIFIABLE] Metric calculation method

inference:
  reasoning_effort: high                                 # [MODIFIABLE] Enables reasoning mode; options: low/medium/high or null to disable
  max_new_tokens: 200                                    # [MODIFIABLE] Maximum tokens to generate
  top_k: 50                                              # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                             # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                         # [MODIFIABLE] Sampling temperature (0 = deterministic)

Menggunakan parameter resoning_efort

reasoning_effortParameter mengontrol perilaku penalaran untuk model yang mampu bernalar.

Prasyarat:

Pilihan yang tersedia:

Kapan mengaktifkan penalaran

Gunakan mode penalaran (lowmedium,, atauhigh) untuk:

Gunakan mode non-penalaran (menghilangkan parameter) untuk:

Pemecahan masalah

Kesalahan: “Mode penalaran diaktifkan tetapi model tidak mendukung penalaran”

Penyebab: reasoning_effort Parameter disetel ke nilai non-null, tetapi yang ditentukan model_type tidak mendukung penalaran.

Resolusi: