hy3

.gitattributes

@@ -35,4 +35,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 assets/benchmark-appendix.png filter=lfs diff=lfs merge=lfs -text
 assets/benchmark.png filter=lfs diff=lfs merge=lfs -text
-assets/rl-training.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -51,7 +51,6 @@ tags:
   - [vLLM](#vllm)
   - [SGLang](#sglang)
 - [Finetuning](#finetuning)
-- [RL Post-training](#rl-post-training)
 - [Quantization](#quantization)
 - [License](#license)
 - [Contact Us](#contact-us)
@@ -216,10 +215,6 @@ python3 -m sglang.launch_server \
 
 Hy3 provides a complete model finetuning pipeline. For detailed documentation, please refer to: [Finetuning Guide](https://huggingface.co/tencent/Hy3/blob/main/finetune/README.md)
 
-## RL Post-training
-
-Hy3 supports GRPO reinforcement learning training with [verl](https://github.com/volcengine/verl), training on Megatron-LM (model conversion via NVIDIA Megatron-Bridge) with vLLM rollout. For detailed documentation, please refer to: [RL Training Guide](https://huggingface.co/tencent/Hy3/blob/main/rl/README.md)
-
 ## Quantization
 
 We provide [AngelSlim](https://github.com/tencent/AngelSlim), a more accessible, comprehensive, and efficient toolkit for large model compression. AngelSlim supports a comprehensive suite of compression tools for large-scale multimodal models, including common quantization algorithms, low-bit quantization, and speculative sampling.

README_CN.md

@@ -41,7 +41,6 @@
   - [vLLM](#使用-vllm-推理)
   - [SGLang](#使用-sglang-推理)
 - [模型微调](#模型微调)
-- [强化学习训练](#强化学习训练)
 - [量化工具](#量化工具)
 - [许可证](#许可证)
 - [联系我们](#联系我们)
@@ -206,10 +205,6 @@ python3 -m sglang.launch_server \
 
 Hy3 提供了完整的模型微调流程，详细的微调文档请参考：[模型微调指南](https://huggingface.co/tencent/Hy3/blob/main/finetune/README_CN.md)
 
-## 强化学习训练
-
-Hy3 支持基于 [verl](https://github.com/volcengine/verl) 的 GRPO 强化学习训练，训练侧使用 Megatron-LM（通过 NVIDIA Megatron-Bridge 完成模型转换），rollout 侧使用 vLLM。详细文档请参考：[强化学习训练指南](https://huggingface.co/tencent/Hy3/blob/main/rl/README_CN.md)
-
 ## 量化工具
 
 我们提供了 [AngelSlim](https://github.com/tencent/AngelSlim)——一套易用、全面、高效的大模型压缩工具包，涵盖常用量化算法、低比特量化和投机采样等能力。

config.json

@@ -3,7 +3,6 @@
     "HYV3ForCausalLM"
   ],
   "bos_token_id": 120000,
-  "dtype": "bfloat16",
   "enable_attention_fp32_softmax": false,
   "enable_lm_head_fp32": true,
   "enable_moe_fp32_combine": false,

finetune/README.md

@@ -194,7 +194,7 @@ The key parameters in the script are as follows:
 - When resuming from a checkpoint, there may be minor differences in loss due to the randomness of some non-deterministic algorithms. This is normal. See: [HuggingFace Transformers Trainer Randomness](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#randomness)
 - When `--model_name_or_path` is specified, all model-related parameters will be ignored.
 - Samples within a batch are padded to the length of the longest sample in the batch, but the maximum length of each sample is `max_seq_length`. Any excess will be truncated.
-- If you see a warning about **linear layer** bias weights not being loaded, you can ignore it; Hy3's linear layers (q_proj / k_proj / v_proj / o_proj, etc.) do not use bias. Note: the MoE router's `e_score_correction_bias` is a buffer and is auto-loaded by the training script, so please do not ignore its loading failure.
+- If you see a warning about bias weights not being loaded, you can ignore it. Hunyuan-Large does not use bias.
 
 ##### What if GPU Memory is Insufficient?
 

finetune/README_CN.md

@@ -194,7 +194,7 @@ Are you sure you want to continue connecting (yes/no)?
 - 从 ckpt 继续训练时，loss 可能会有微小的偏差，这是由一些非确定性算法带来的随机性，是正常现象。参考：[HuggingFace Transformers Trainer Randomness](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#randomness)
 - 当 `--model_name_or_path` 有效时，所有模型相关的参数都会被忽略
 - 一个 batch 内的样本会通过 padding 对齐 batch 内最长的样本，而每条样本的长度最长为 max_seq_length，超出的部分会被裁剪
-- 如果报出**线性层** bias 权重没有 load 的 warning，忽略即可，Hy3 的线性层（q_proj / k_proj / v_proj / o_proj 等）不使用 bias。注意：MoE 路由的 `e_score_correction_bias` 属于 buffer，已由训练脚本自动加载，如果加载失败请不要忽略。
+- 如果报出 bias 权重没有 load 的 warning，忽略即可，Hunyuan-Large 中不会用到 bias
 
 ##### 显存不足怎么办？
 

rl/README.md

@@ -1,216 +0,0 @@
-# Hy3 Reinforcement Learning Training
-
-English | [简体中文](./README_CN.md)
-
-This document describes how to run reinforcement learning training for Hy3 with [verl](https://github.com/volcengine/verl). Training runs on [Megatron-LM](https://github.com/NVIDIA/Megatron-LM); NVIDIA [Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge) (`HYV3Bridge`) converts the HF checkpoint into a Megatron model on the fly at training startup — no offline conversion needed. Rollout runs on [vLLM](https://github.com/vllm-project/vllm).
-
-Training script: [`examples/grpo_trainer/run_hy_v3_megatron.sh`](https://github.com/verl-project/verl/blob/main/examples/grpo_trainer/run_hy_v3_megatron.sh) in the verl repository.
-
-## Quick Start
-
-### Environment
-
-Recommended image: `verlai/verl:vllm023.dev1`
-
-Verified version combination:
-
-| Component | Version |
-| --- | --- |
-| verl | [220e903](https://github.com/verl-project/verl/commit/220e9039902c6db56860e2afd659803dc34ec005) |
-| Megatron-Bridge | [df0852c](https://github.com/NVIDIA-NeMo/Megatron-Bridge/commit/df0852cf94c674de07f9f7ef933c484de8aca505) |
-| transformers | 5.6.0 |
-| nvidia-modelopt | 0.44.0rc5 |
-
-### Prepare dependencies
-
-Clone verl at the verified version, then clone the dependencies into `third_party/` under the repository root:
-
-```bash
-git clone https://github.com/verl-project/verl.git
-cd verl
-git checkout 220e903
-
-mkdir third_party
-git clone https://github.com/NVIDIA-NeMo/Megatron-Bridge.git third_party/Megatron-Bridge
-git -C third_party/Megatron-Bridge checkout df0852c
-
-git clone https://github.com/Ascend/TransferQueue.git third_party/TransferQueue
-```
-
-Create a `runtime_env.yaml` so the Ray runtime env ships the working directory (including `third_party/`) to every worker node and adds the dependencies to `PYTHONPATH`:
-
-```yaml
-# runtime_env.yaml
-working_dir: ./
-excludes: [
-  "/.git/",
-  "/third_party/Megatron-Bridge/.git/",
-  "/third_party/TransferQueue/.git/",
-  "**/__pycache__/",
-]
-env_vars:
-  PYTHONPATH: third_party/Megatron-Bridge/src:third_party/TransferQueue
-```
-
-### Prepare the dataset
-
-The script defaults to [DAPO-Math-17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k) (train) and [AIME-2024](https://huggingface.co/datasets/BytedTsinghua-SIA/AIME-2024) (validation). Both are already in the format verl expects on HuggingFace — just download and dump to parquet:
-
-```python
-# prepare_data.py
-import datasets
-
-dapo = datasets.load_dataset("BytedTsinghua-SIA/DAPO-Math-17k", "default")["train"]
-dapo.to_parquet("DAPO-Math-17k/dapo-math-17k.parquet")
-
-aime = datasets.load_dataset("BytedTsinghua-SIA/AIME-2024", "default")["train"]
-aime.to_parquet("AIME-2024/aime-2024.parquet")
-```
-
-```bash
-python prepare_data.py   # output layout matches the script's default DATA_DIR
-```
-
-Place the data under the verl repository root (`DATA_DIR` defaults to `$PWD`), or point elsewhere with `DATA_DIR=/path/to/data`.
-
-### Submit the job
-
-Use the HuggingFace checkpoint directly; training data is in parquet format. Point `RAY_ADDRESS` at the cluster head node, then submit with `ray job submit`:
-
-```bash
-export RAY_ADDRESS=http://<head_node_ip>:<port>
-
-ray job submit --no-wait --runtime-env=runtime_env.yaml -- \
-    bash examples/grpo_trainer/run_hy_v3_megatron.sh
-```
-
-> **Note**: on H20, rollout (vLLM inference) requires at least TP=16 (the script defaults to `ROLLOUT_TP=16`) — the weights of a single instance only fit when sharded across 16 GPUs.
-
-### Key parameters
-
-**Hy3 settings**
-
-| Parameter | Value | Reason |
-| --- | --- | --- |
-| `moe_router_enable_expert_bias` | True | Hy3 routes with a per-expert bias (aux-loss-free) |
-| `moe_router_bias_update_rate` | 0 | 0 freezes the bias (it still participates in scoring but is never updated) |
-| `moe_router_load_balancing_type` | none | no auxiliary load-balancing loss |
-
-**Training setup**
-
-| Parameter | What it does |
-| --- | --- |
-| `data.train_batch_size` | prompts sampled per step |
-| `rollout.n` | responses generated per prompt, i.e. the GRPO group size (the in-group advantage baseline is computed over them), typically 8–16 |
-| `actor.ppo_mini_batch_size` | samples per actor parameter update |
-| `actor.optim.lr` | actor learning rate, typically the 1e-6 scale |
-| `data.max_response_length` | maximum generation length, task-dependent |
-
-**Algorithm behavior**
-
-| Parameter | What it does | How to set it |
-| --- | --- | --- |
-| `algorithm.norm_adv_by_std_in_grpo` | whether advantages are divided by the in-group std | True is original GRPO; False is the [Dr.GRPO](https://arxiv.org/abs/2503.20783) fix, preventing too-easy/too-hard prompts with tiny variance from being amplified |
-| `actor.clip_ratio_low` / `clip_ratio_high` | PPO trust-region bounds | a slightly relaxed upper bound (clip-higher, e.g. 0.2/0.28) gives low-probability tokens more room to rise, mitigating entropy collapse |
-| `actor.clip_ratio_c` | [dual-clip](https://arxiv.org/abs/1912.09729) lower-bound constant | caps the penalty on negative-advantage tokens to keep a single step from blowing up the policy |
-| `actor.kl_loss_coef` | KL-regularization strength toward the reference policy | 0 means KL-free (rely on clipping); add a small value (e.g. 1e-3) if training is unstable |
-| `algorithm.rollout_correction.rollout_is` / `rollout_is_threshold` | correction for the rollout/training log-prob mismatch (IcePop) | `token` plus lower/upper bounds (e.g. `0.5_4.0`; token weights outside the range are zeroed); recommended whenever the numeric gap between the two engines is non-negligible |
-| `rollout.temperature` / `top_p` | rollout sampling exploration strength | typically 0.9–1.0; too low a temperature reduces in-group diversity and degrades the GRPO baseline |
-
-**Memory and sequence length**
-
-Parameters that must move together when extending the sequence length (changing `data.max_response_length` alone is not enough):
-
-| Parameter | Coupling |
-| --- | --- |
-| `data.max_response_length` | target response length |
-| `rollout.max_model_len` | vLLM context length, must be ≥ prompt + response |
-| `actor.ppo_max_token_len_per_gpu` | training-side per-GPU token budget; sequences are split across CP ranks, so each GPU actually holds `(prompt+response)/CP` per sequence — the requirement is budget × CP ≥ prompt + response |
-| `actor.megatron.context_parallel_size` | scale up proportionally for much longer sequences (activation memory grows linearly with sequence length and is sharded by CP) |
-
-For training-side OOM (the error occurs during actor update / log_prob), you can try:
-
-1. Lower `actor.ppo_max_token_len_per_gpu` — dynamic batching packs micro batches against it, so it directly bounds the activation peak;
-2. Set `actor.ppo_micro_batch_size_per_gpu` to 1;
-3. Lower `ref.log_prob_max_token_len_per_gpu` / `rollout.log_prob_max_token_len_per_gpu`;
-4. Raise `actor.megatron.context_parallel_size` (activations are sharded by CP) or `pipeline_model_parallel_size` (fewer layers per stage).
-
-Offload and recompute:
-
-| Config | Script default | What it does |
-| --- | --- | --- |
-| `actor.megatron.param_offload` | True | offloads parameters to CPU while training is idle, freeing GPU memory for rollout |
-| `actor.megatron.optimizer_offload` | True | offloads optimizer state (fp32 master weights + momenta — the biggest memory consumer) to CPU |
-| `actor.megatron.grad_offload` | True | offloads gradient buffers to CPU |
-| `override_transformer_config.recompute_granularity` | full | full activation recomputation: forward stores no activations, backward recomputes them — trades ~30% extra compute for most of the activation memory |
-| `override_transformer_config.recompute_method` / `recompute_num_layers` | uniform / 1 | recompute uniformly at 1-layer granularity — the finest granularity, lowest peak |
-
-Both substantially relieve OOM: offload keeps parameters/optimizer state/gradients out of GPU memory at the cost of per-step CPU↔GPU transfers, and recompute keeps activations out of GPU memory at the cost of one extra forward pass during backward.
-
-## Customizing Your Training
-
-### Using your own dataset
-
-verl reads parquet data where each row contains 5 fields (see the verl docs: [Prepare Data](https://verl.readthedocs.io/en/latest/preparation/prepare_data.html)):
-
-```python
-{
-    "data_source": "my_dataset",          # dataset name; the RewardManager uses it to index the scoring function
-    "prompt": [                           # HuggingFace chat template format; the tokenizer renders and tokenizes it
-        {"role": "user", "content": "1+1=?"}
-    ],
-    "ability": "math",                    # task category
-    "reward_model": {
-        "style": "rule",
-        "ground_truth": "2"               # reference answer; the reward function's scoring logic must align with its format
-    },
-    "extra_info": {"split": "train", "index": 0},   # metadata
-}
-```
-
-Write a preprocessing script that converts your data into this format (verl's [`examples/data_preprocess/`](https://github.com/verl-project/verl/tree/main/examples/data_preprocess) ships a dozen ready-to-adapt templates — GSM8K, MATH, etc.), save as parquet, then point the script at your files via environment variables:
-
-```bash
-TRAIN_FILES=/path/to/my_train.parquet \
-VAL_FILES=/path/to/my_val.parquet \
-bash examples/grpo_trainer/run_hy_v3_megatron.sh
-```
-
-**Reward function**: math-style tasks with rule-verifiable answers can reuse the default DAPO reward as-is (it routes to a built-in scoring function by `data_source`); other tasks need a custom reward function, specified via `custom_reward_function.path` (see the verl docs: [Implement Reward Function](https://verl.readthedocs.io/en/latest/preparation/reward_function.html)).
-
-### Switching the RL algorithm
-
-The script defaults to GRPO, but the algorithm layer is decoupled from the model layer, so switching algorithms requires no changes to the Hy3-specific configuration.
-
-**Switch between built-in algorithms (one config key)**: verl ships a dozen advantage estimators, selected via `algorithm.adv_estimator` — options include `gae` (PPO), `grpo`, `rloo`, `remax`, `reinforce_plus_plus`, `opo`, `gpg`, etc. (full list in `AdvantageEstimator` in [`core_algos.py`](https://github.com/verl-project/verl/blob/main/verl/trainer/ppo/core_algos.py)); the policy loss is selected via `actor_rollout_ref.actor.policy_loss.loss_mode` (`vanilla`, `gspo`, `cispo`, `clip_cov`, etc.). See the verl docs for each algorithm's theory and configuration: [PPO](https://verl.readthedocs.io/en/latest/algo/ppo.html) / [GRPO](https://verl.readthedocs.io/en/latest/algo/grpo.html) / [DAPO](https://verl.readthedocs.io/en/latest/algo/dapo.html).
-
-**Training Hy3 with a recipe**: full-pipeline algorithms (e.g. DAPO with dynamic sampling) live as standalone implementations under verl's [`recipe/`](https://github.com/verl-project/verl/tree/main/recipe) directory, each with its own launch script. When pointing a recipe at Hy3, carry over the following required Hy3 settings into the recipe's launch script:
-
-```bash
-# Required Hy3 settings (apply to any recipe)
-actor_rollout_ref.model.path=/path/to/Hy3
-actor_rollout_ref.model.trust_remote_code=True
-data.trust_remote_code=True
-actor_rollout_ref.actor.megatron.use_mbridge=True
-actor_rollout_ref.actor.megatron.vanilla_mbridge=False
-+actor_rollout_ref.actor.megatron.override_transformer_config.moe_router_enable_expert_bias=True
-+actor_rollout_ref.actor.megatron.override_transformer_config.moe_router_bias_update_rate=0
-+actor_rollout_ref.actor.megatron.override_transformer_config.moe_router_load_balancing_type=none
-+actor_rollout_ref.actor.megatron.override_transformer_config.moe_grouped_gemm=True
-```
-
-For implementing entirely new algorithms, see the verl docs: [Extend to other RL algorithms](https://verl.readthedocs.io/en/latest/advance/dpo_extension.html).
-
-## Results
-
-We launched Hy3 GRPO training on 128 H20 GPUs (16 nodes × 8) with [`run_hy_v3_megatron.sh`](https://github.com/verl-project/verl/blob/main/examples/grpo_trainer/run_hy_v3_megatron.sh): math reasoning on the DAPO dataset at 8192 max response length, with PP/CP/EP parallelism plus full offload, and BF16 rollout + BF16 training.
-
-Exact values used in this run: batch 128 prompts × 16 samples (2048 trajectories/step), `ppo_mini_batch_size=128` (one update per step), lr 1e-6, clip 0.2/0.28 (dual-clip c=10.0), KL-free, DAPO overlong buffer (len 4096 / penalty 1.0), IcePop threshold `0.5_4.0`, sampling temperature 0.9 / top_p 1.0.
-
-The training dynamics are stable: the rollout/training log-prob drift (`rollout_probs_diff`) stays below 0.015 throughout, and both reward and the AIME validation score grow steadily over the run.
-
-![Hy3 RL training curves](../assets/rl-training.png)
-
-## Acknowledgements
-
-We thank the Tencent Hunyuan team for their support on model training and engineering infrastructure, as well as the [verl](https://github.com/volcengine/verl), [Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge), [Megatron-LM](https://github.com/NVIDIA/Megatron-LM), and [vLLM](https://github.com/vllm-project/vllm) communities for their help.

rl/README_CN.md

@@ -1,216 +0,0 @@
-# Hy3 强化学习训练
-
-[English](./README.md) | 简体中文
-
-本文档介绍如何使用 [verl](https://github.com/volcengine/verl) 对 Hy3 进行强化学习训练。训练侧使用 [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)，并通过 NVIDIA [Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge)（`HYV3Bridge`）在训练启动时将 HF checkpoint 在线转换为 Megatron 模型，无需离线转换；rollout 侧使用 [vLLM](https://github.com/vllm-project/vllm)。
-
-训练脚本：verl 仓库 [`examples/grpo_trainer/run_hy_v3_megatron.sh`](https://github.com/verl-project/verl/blob/main/examples/grpo_trainer/run_hy_v3_megatron.sh)
-
-## 快速开始
-
-### 环境
-
-推荐镜像：`verlai/verl:vllm023.dev1`
-
-已验证的版本组合：
-
-| 组件 | 版本 |
-| --- | --- |
-| verl | [220e903](https://github.com/verl-project/verl/commit/220e9039902c6db56860e2afd659803dc34ec005) |
-| Megatron-Bridge | [df0852c](https://github.com/NVIDIA-NeMo/Megatron-Bridge/commit/df0852cf94c674de07f9f7ef933c484de8aca505) |
-| transformers | 5.6.0 |
-| nvidia-modelopt | 0.44.0rc5 |
-
-### 准备依赖
-
-clone verl 并切换到已验证版本，然后在仓库根目录下将依赖 clone 到 `third_party/`：
-
-```bash
-git clone https://github.com/verl-project/verl.git
-cd verl
-git checkout 220e903
-
-mkdir third_party
-git clone https://github.com/NVIDIA-NeMo/Megatron-Bridge.git third_party/Megatron-Bridge
-git -C third_party/Megatron-Bridge checkout df0852c
-
-git clone https://github.com/Ascend/TransferQueue.git third_party/TransferQueue
-```
-
-编写 `runtime_env.yaml`，通过 Ray runtime env 将工作目录（含 `third_party/`）分发到各 worker 节点，并加入 `PYTHONPATH`：
-
-```yaml
-# runtime_env.yaml
-working_dir: ./
-excludes: [
-  "/.git/",
-  "/third_party/Megatron-Bridge/.git/",
-  "/third_party/TransferQueue/.git/",
-  "**/__pycache__/",
-]
-env_vars:
-  PYTHONPATH: third_party/Megatron-Bridge/src:third_party/TransferQueue
-```
-
-### 准备数据集
-
-脚本默认使用 [DAPO-Math-17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k)（训练）和 [AIME-2024](https://huggingface.co/datasets/BytedTsinghua-SIA/AIME-2024)（验证），两者在 HuggingFace 上已是 verl 所需格式，下载后转存 parquet 即可：
-
-```python
-# prepare_data.py
-import datasets
-
-dapo = datasets.load_dataset("BytedTsinghua-SIA/DAPO-Math-17k", "default")["train"]
-dapo.to_parquet("DAPO-Math-17k/dapo-math-17k.parquet")
-
-aime = datasets.load_dataset("BytedTsinghua-SIA/AIME-2024", "default")["train"]
-aime.to_parquet("AIME-2024/aime-2024.parquet")
-```
-
-```bash
-python prepare_data.py   # 输出路径与脚本默认的 DATA_DIR 布局一致
-```
-
-数据放在 verl 仓库根目录（`DATA_DIR` 默认为 `$PWD`）下，或通过 `DATA_DIR=/path/to/data` 指定。
-
-### 提交任务
-
-模型直接使用 HuggingFace 格式 checkpoint，训练数据为 parquet 格式。设置 `RAY_ADDRESS` 指向集群 head 节点后，通过 `ray job submit` 提交：
-
-```bash
-export RAY_ADDRESS=http://<head_node_ip>:<port>
-
-ray job submit --no-wait --runtime-env=runtime_env.yaml -- \
-    bash examples/grpo_trainer/run_hy_v3_megatron.sh
-```
-
-> **注**：H20 上 rollout（vLLM 推理）最小需要 TP=16（脚本默认 `ROLLOUT_TP=16`），单实例权重需跨 16 卡切分才放得下。
-
-### 关键参数
-
-**Hy3配置**
-
-| 参数 | 值 | 原因 |
-| --- | --- | --- |
-| `moe_router_enable_expert_bias` | True | Hy3 使用 per-expert bias 路由（aux-loss-free） |
-| `moe_router_bias_update_rate` | 0 | 设 0 冻结 bias（只参与打分、不再更新）。 |
-| `moe_router_load_balancing_type` | none | 不使用辅助负载均衡 loss |
-
-**训练设置**
-
-| 参数 | 作用 | 
-| --- | --- | 
-| `data.train_batch_size` | 每步采样的 prompt 数 | 
-| `rollout.n` | 每个 prompt 生成的 response 数，即 GRPO 组大小（组内计算 advantage 基线），常用 8–16；
-| `actor.ppo_mini_batch_size` | actor 每次参数更新的样本量 | 
-| `actor.optim.lr` | actor 学习率，常用 1e-6 量级 |
-| `data.max_response_length` | 最长生成长度，依任务而定 |
-
-**算法行为**
-
-| 参数 | 作用 | 怎么设 |
-| --- | --- | --- |
-| `algorithm.norm_adv_by_std_in_grpo` | advantage 是否除以组内 std | True 为原版 GRPO；False 为 [Dr.GRPO](https://arxiv.org/abs/2503.20783) 修正，避免过易/过难样本因方差小被放大 |
-| `actor.clip_ratio_low` / `clip_ratio_high` | PPO 信任域上下界 | 上界略放宽（clip-higher，如 0.2/0.28）给低概率 token 更多上升空间，缓解熵坍缩 |
-| `actor.clip_ratio_c` | [dual-clip](https://arxiv.org/abs/1912.09729) 下界常数 | 封顶负 advantage token 的惩罚，防止策略被单步拉爆 |
-| `actor.kl_loss_coef` | 向参考策略的 KL 正则强度 | 0 为 KL-free（靠 clip 约束）；训练不稳时可加小值（如 1e-3） |
-| `algorithm.rollout_correction.rollout_is` / `rollout_is_threshold` | rollout 与训练引擎的 log-prob 偏差校正（IcePop） | `token` + 上下阈值（如 `0.5_4.0`，权重超界的 token 置零）；两侧引擎精度偏差不可忽略时建议开启 |
-| `rollout.temperature` / `top_p` | rollout 采样探索强度 | 常用 0.9–1.0；温度过低组内多样性不足，GRPO 组内基线退化 |
-
-**显存与序列长度**
-
-| 参数 | 联动关系 |
-| --- | --- |
-| `data.max_response_length` | 目标响应长度 |
-| `rollout.max_model_len` | vLLM 上下文长度，需 ≥ prompt + response |
-| `actor.ppo_max_token_len_per_gpu` | 训练侧每卡 token 预算；序列被 CP 切分到多卡，单条序列每卡实际占 `(prompt+response)/CP`，故需满足 预算 × CP ≥ prompt + response |
-| `actor.megatron.context_parallel_size` | 序列显著变长时按比例增大（激活显存随序列长度线性增长，由 CP 分摊） |
-
-训练侧 OOM（报错发生在 actor update / log_prob 阶段），可尝试：
-
-1. 调小 `actor.ppo_max_token_len_per_gpu`——动态 batch 按它打包 micro batch，直接决定激活峰值；
-2. `actor.ppo_micro_batch_size_per_gpu` 调为 1；
-3. 调小 `ref.log_prob_max_token_len_per_gpu` / `rollout.log_prob_max_token_len_per_gpu`；
-4. 增大 `actor.megatron.context_parallel_size`（激活按 CP 分摊）或 `pipeline_model_parallel_size`（每 stage 层数更少）。
-
-offload 与 recompute：
-
-| 配置 | 脚本默认 | 作用 |
-| --- | --- | --- |
-| `actor.megatron.param_offload` | True | 训练空闲期把参数下放 CPU，给 rollout 腾显存 |
-| `actor.megatron.optimizer_offload` | True | 优化器状态（fp32 master 权重 + 动量，显存大头）放 CPU |
-| `actor.megatron.grad_offload` | True | 梯度缓冲放 CPU |
-| `override_transformer_config.recompute_granularity` | full | 全量激活重算：前向不存激活、反向重算，用约 30% 额外计算换掉大部分激活显存 |
-| `override_transformer_config.recompute_method` / `recompute_num_layers` | uniform / 1 | 按每 1 层为单位均匀重算，粒度最细、峰值最低 |
-
-两者都能显著缓解 OOM，offload 用每步 CPU↔GPU 搬运耗时换取参数/优化器/梯度不占显存，recompute 用反向多一次前向计算换取激活不占显存。
-
-## 自定义训练
-
-### 使用自己的数据集
-
-verl 读取 parquet 格式数据，每行需包含 5 个字段（详见 verl 官方文档 [Prepare Data](https://verl.readthedocs.io/en/latest/preparation/prepare_data.html)）：
-
-```python
-{
-    "data_source": "my_dataset",          # 数据集名称，RewardManager 据此索引对应的打分函数
-    "prompt": [                           # HuggingFace chat template 格式，tokenizer 会渲染并分词
-        {"role": "user", "content": "1+1=?"}
-    ],
-    "ability": "math",                    # 任务类别
-    "reward_model": {
-        "style": "rule",
-        "ground_truth": "2"               # 标准答案；reward 函数的判分逻辑须与其格式对齐
-    },
-    "extra_info": {"split": "train", "index": 0},   # 元信息
-}
-```
-
-编写预处理脚本转换为上述格式（verl 的 [`examples/data_preprocess/`](https://github.com/verl-project/verl/tree/main/examples/data_preprocess) 提供了 GSM8K、MATH 等十余个可直接套用的模板），保存为 parquet 后通过环境变量指定：
-
-```bash
-TRAIN_FILES=/path/to/my_train.parquet \
-VAL_FILES=/path/to/my_val.parquet \
-bash examples/grpo_trainer/run_hy_v3_megatron.sh
-```
-
-**reward 函数**：答案可规则判分的数学类任务可直接沿用默认的 DAPO reward（按 `data_source` 自动路由到内置打分函数）；其他任务需自定义 reward 函数，通过 `custom_reward_function.path` 指定（见 verl 文档 [Implement Reward Function](https://verl.readthedocs.io/en/latest/preparation/reward_function.html)）。
-
-
-### 更换 RL 算法
-
-训练脚本默认 GRPO，但算法层与模型层解耦，换算法时 Hy3 相关配置无需改动。
-
-**切换内置算法（改一个配置项）**：verl 内置了十余种 advantage 估计器，通过 `algorithm.adv_estimator` 直接切换，可选值包括 `gae`（PPO）、`grpo`、`rloo`、`remax`、`reinforce_plus_plus`、`opo`、`gpg` 等（完整列表见 [`core_algos.py`](https://github.com/verl-project/verl/blob/main/verl/trainer/ppo/core_algos.py) 的 `AdvantageEstimator`）；policy loss 通过 `actor_rollout_ref.actor.policy_loss.loss_mode` 切换（`vanilla`、`gspo`、`cispo`、`clip_cov` 等）。各算法的原理与配置说明见 verl 文档：[PPO](https://verl.readthedocs.io/en/latest/algo/ppo.html) / [GRPO](https://verl.readthedocs.io/en/latest/algo/grpo.html) / [DAPO](https://verl.readthedocs.io/en/latest/algo/dapo.html)。
-
-
-**使用 recipe 训练 Hy3**：训练流程级的完整算法（如 dynamic sampling 版 DAPO）在 verl [`recipe/`](https://github.com/verl-project/verl/tree/main/recipe) 目录下有独立实现，各自带启动脚本。将其模型指向 Hy3 时，须把以下 Hy3 必需配置带入 recipe 的启动脚本：
-
-```bash
-# Hy3 必需配置（任何 recipe 通用）
-actor_rollout_ref.model.path=/path/to/Hy3
-actor_rollout_ref.model.trust_remote_code=True
-data.trust_remote_code=True
-actor_rollout_ref.actor.megatron.use_mbridge=True
-actor_rollout_ref.actor.megatron.vanilla_mbridge=False
-+actor_rollout_ref.actor.megatron.override_transformer_config.moe_router_enable_expert_bias=True
-+actor_rollout_ref.actor.megatron.override_transformer_config.moe_router_bias_update_rate=0
-+actor_rollout_ref.actor.megatron.override_transformer_config.moe_router_load_balancing_type=none
-+actor_rollout_ref.actor.megatron.override_transformer_config.moe_grouped_gemm=True
-```
-
-自定义全新算法的扩展方式见 verl 文档 [Extend to other RL algorithms](https://verl.readthedocs.io/en/latest/advance/dpo_extension.html)。
-
-## 实验结果
-
-我们在 128 张 H20 GPU（16 机 × 8 卡）上启动了 Hy3 的 GRPO 训练（脚本：[`run_hy_v3_megatron.sh`](https://github.com/verl-project/verl/blob/main/examples/grpo_trainer/run_hy_v3_megatron.sh)）：数学推理任务（DAPO 数据集），max response length 8192，采用 PP/CP/EP 多维并行 + 全量 offload，BF16 rollout + BF16 训练。
-
-本次实验的具体取值：batch 128 prompts × 16 samples（2048 条轨迹/步）、`ppo_mini_batch_size=128`（每步单次更新）、lr 1e-6、clip 0.2/0.28（dual-clip c=10.0）、KL-free、DAPO overlong buffer（len 4096 / penalty 1.0）、IcePop 阈值 `0.5_4.0`、采样 temperature 0.9 / top_p 1.0。
-
-训练动态稳定：rollout 与训练侧的 log-prob 偏差（`rollout_probs_diff`）全程基本小于 0.015，reward 与 AIME 验证集分数在整个训练过程中稳步上升。
-
-![Hy3 RL 训练曲线](../assets/rl-training.png)
-
-## 致谢
-
-感谢腾讯混元（Hunyuan）团队在模型训练与工程落地上的支持，以及 [verl](https://github.com/volcengine/verl)、[Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge)、[Megatron-LM](https://github.com/NVIDIA/Megatron-LM)、[vLLM](https://github.com/vllm-project/vllm) 社区的帮助。