# 在 SageMaker HyperPod 上对 Nova 2.0 进行强化微调（RFT）
本部分介绍示例配方、启动微调作业、超参数指导，以及在 SageMaker HyperPod 上对 Nova 2.0 Lite 进行 RFT 的训练监控。有关数据格式、支持的功能、限制条件以及准备 RFT 训练数据的最佳实践的信息，请参阅 [在 Amazon Nova 2 上为 RFT 准备数据](nova-data-prep-rft-2.md)。
要确定 RFT 是否适合您的应用场景，请参阅 [强化微调（RFT）](nova-hp-rft.md)。
**Topics**
+ [在 SageMaker HyperPod 上启动微调作业](#nova-rft-2-starting-job)
+ [超参数指导](#nova-hp-rft-monitoring-hyperparams)
+ [监控 RFT 训练](nova-hp-rft-monitoring.md)
## RFT 配方示例
```
# Note:
# This recipe can run on p5.48xlarge, p5e.48xlarge, and p5en.48xlarge instance types.
run:
name: "my-rft-run" # Unique run name (appears in logs and artifacts).
model_type: amazon.nova-2-lite-v1:0:256k
model_name_or_path: nova-lite-2/prod
data_s3_path: s3:/// # Training dataset in JSONL format.
replicas: 4 # Number of total training instances.
generation_replicas: 2 # Number of total instances dedicated to response generation.
reward_lambda_arn: arn:aws:lambda:::function:
## MLFlow configs
mlflow_tracking_uri: "" # Required for MLFlow
mlflow_experiment_name: "my-rft-experiment" # Optional for MLFlow. Note: leave this field non-empty
mlflow_run_name: "my-rft-run" # Optional for MLFlow. Note: leave this field non-empty
## SMHP RFT training configs
training_config:
max_length: 8192 # Context window (tokens) for inputs and prompt.
global_batch_size: 32 # Total samples per optimizer step across all replicas (16/32/64/128/256).
reasoning_effort: high # Reasoning mode: high, low, or null for non-reasoning.
data:
shuffle: true # Shuffle training data each epoch.
rollout: # Controls how responses are generated for advantage calculation.
rollout_strategy:
type: off_policy_async # Asynchronous rollout for higher throughput.
age_tolerance: 2 # Maximum policy age before regeneration.
advantage_strategy:
number_generation: 4 # Samples per prompt to estimate advantages (higher = lower variance but higher cost).
generator:
max_new_tokens: 6000 # Cap on tokens generated per sample.
set_random_seed: true # Seed generation for reproducibility across runs.
temperature: 1 # Softmax temperature for sampling.
top_k: 1 # Sample only from top-K logits.
rewards:
preset_reward_function: null # Preset reward functions: exact_match or null for custom.
api_endpoint:
lambda_arn: arn:aws:lambda:::function:
lambda_concurrency_limit: 12 # Max concurrent Lambda invocations (throughput vs. throttling).
lambda_batch_size: 128 # Number of samples per Lambda invocation.
trainer:
max_steps: 2 # Steps to train for. One step = global_batch_size samples.
save_steps: 5 # Save a checkpoint every N steps.
test_steps: 1 # Run validation every N reference model updates.
refit_freq: 4 # Frequency of reference model updates.
clip_ratio_high: 0.2 # PPO clip ratio for policy updates.
loss_scale: 1.0 # Scaling factor for the policy loss.
# RL parameters
ent_coeff: 0.0 # Entropy bonus added to the policy loss (higher = more exploration).
kl_loss_coef: 0.0 # Weight on the KL penalty between the current and reference policy.
optim_config: # Optimizer settings.
lr: 1e-6 # Learning rate.
weight_decay: 0.0 # L2 regularization strength (0.0 to 1.0).
adam_beta1: 0.9
adam_beta2: 0.95
peft: # Parameter-efficient fine-tuning (LoRA).
peft_scheme: "lora" # Enable LoRA for PEFT.
lora_tuning:
alpha: 64 # LoRA scaling factor.
lora_plus_lr_ratio: 64.0 # LoRA+ learning rate scaling factor (0.0 to 100.0).
```
## 在 SageMaker HyperPod 上启动微调作业
### 准备数据
有关数据格式、支持的功能、限制条件以及准备 RFT 训练数据的最佳实践的信息，请参阅 [在 Amazon Nova 2 上为 RFT 准备数据](nova-data-prep-rft-2.md)。
### 上传数据
将您的训练数据集上传到 S3 存储桶。在配方的 `run` 数据块中指定其位置：
```
## Run config
run:
...
data_s3_path: "s3:////.jsonl"
```
**注意**
将 ``、`` 和 `` 替换为实际的 S3 路径。
### 定义您的配置
使用 `run` 数据块中的 `model_type` 和 `model_name_or_path` 字段定义基础模型：
```
## Run config
run:
...
model_type: amazon.nova-2-lite-v1:0:256k
model_name_or_path: nova-lite-2/prod
...
```
## 超参数指导
根据训练方法，建议采用如下超参数：
**常规：**
+ epoch：1
+ 学习率（lr）：1e – 7
+ 生成次数：8
+ 最大新词元数：8192
+ 批处理大小：256
**LoRA（低秩适应）：**
+ LoRA 秩：32
**注意**
请根据数据集规模和验证性能调整这些值。监控训练指标以防止过拟合。