add files 1-400
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- README.md +130 -0
- docs/ENV_NOTES.md +98 -0
- eval/aime_eval.py +223 -0
- eval/aime_launch.py +17 -0
- eval/hmmt_eval.py +165 -0
- eval/merge_eval.py +72 -0
- eval/retrunc_eval.py +105 -0
- eval/rlve_eval.py +212 -0
- frameworks/agentgym_rl/.gitignore +116 -0
- frameworks/agentgym_rl/.readthedocs.yaml +16 -0
- frameworks/agentgym_rl/.style.yapf +5 -0
- frameworks/agentgym_rl/Notice.txt +1 -0
- frameworks/agentgym_rl/VERL_LICENSE +202 -0
- frameworks/agentgym_rl/VERL_README.md +135 -0
- frameworks/agentgym_rl/docs/Makefile +20 -0
- frameworks/agentgym_rl/docs/README.md +19 -0
- frameworks/agentgym_rl/docs/README_vllm0.7.md +65 -0
- frameworks/agentgym_rl/docs/_static/logo.png +0 -0
- frameworks/agentgym_rl/docs/advance/dpo_extension.rst +271 -0
- frameworks/agentgym_rl/docs/advance/fsdp_extension.rst +95 -0
- frameworks/agentgym_rl/docs/advance/megatron_extension.rst +26 -0
- frameworks/agentgym_rl/docs/advance/placement.rst +11 -0
- frameworks/agentgym_rl/docs/conf.py +81 -0
- frameworks/agentgym_rl/docs/examples/config.rst +365 -0
- frameworks/agentgym_rl/docs/examples/gsm8k_example.rst +168 -0
- frameworks/agentgym_rl/docs/examples/ppo_code_architecture.rst +207 -0
- frameworks/agentgym_rl/docs/experiment/ppo.rst +45 -0
- frameworks/agentgym_rl/docs/faq/faq.rst +21 -0
- frameworks/agentgym_rl/docs/hybrid_flow.rst +269 -0
- frameworks/agentgym_rl/docs/index.rst +111 -0
- frameworks/agentgym_rl/docs/perf/perf_tuning.rst +164 -0
- frameworks/agentgym_rl/docs/preparation/prepare_data.rst +126 -0
- frameworks/agentgym_rl/docs/preparation/reward_function.rst +46 -0
- frameworks/agentgym_rl/docs/requirements-docs.txt +9 -0
- frameworks/agentgym_rl/docs/start/install.rst +111 -0
- frameworks/agentgym_rl/docs/start/quickstart.rst +142 -0
- frameworks/agentgym_rl/docs/workers/fsdp_workers.rst +142 -0
- frameworks/agentgym_rl/docs/workers/megatron_workers.rst +200 -0
- frameworks/agentgym_rl/docs/workers/ray_trainer.rst +243 -0
- frameworks/agentgym_rl/pyproject.toml +86 -0
- frameworks/agentgym_rl/requirements.txt +19 -0
- frameworks/agentgym_rl/scripts/format.sh +3 -0
- frameworks/agentgym_rl/scripts/model_merger.py +170 -0
- frameworks/agentgym_rl/scripts/multiple_model_merger.py +178 -0
- frameworks/agentgym_rl/scripts/single_model_merger.py +86 -0
- frameworks/agentgym_rl/setup.py +74 -0
- frameworks/agentgym_rl/verl/__init__.py +27 -0
- frameworks/agentgym_rl/verl/agent_trainer/__init__.py +13 -0
- frameworks/agentgym_rl/verl/agent_trainer/config/evaluation.yaml +3 -0
- frameworks/agentgym_rl/verl/agent_trainer/config/generation.yaml +85 -0
README.md
ADDED
|
@@ -0,0 +1,130 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags: [distillation, on-policy-distillation, rose, opd, verl, trinity, agentgym, rlve, sciworld, dapo]
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# ROSE / OPD 蒸馏实验的完整实现与训练框架
|
| 7 |
+
|
| 8 |
+
三个 domain(RLVE 单轮推理、SciWorld 多轮 agent、DAPO-Math)上对比三种蒸馏方式
|
| 9 |
+
(teacher-SFT / on-policy distillation / online ROSE)的全部代码。
|
| 10 |
+
|
| 11 |
+
配套的 checkpoint 与数据在:
|
| 12 |
+
[OPD-dq](https://huggingface.co/SeanWang0027/OPD-dq) ·
|
| 13 |
+
[ROSE-dq](https://huggingface.co/SeanWang0027/ROSE-dq) ·
|
| 14 |
+
[SciWorld-dq](https://huggingface.co/SeanWang0027/SciWorld-dq) ·
|
| 15 |
+
[data-dq](https://huggingface.co/SeanWang0027/data-dq)
|
| 16 |
+
|
| 17 |
+
## 目录
|
| 18 |
+
|
| 19 |
+
```
|
| 20 |
+
frameworks/
|
| 21 |
+
verl_rose/ ROSE 全部线用的 verl(含 recipe/rose/rose_agent_loop.py)
|
| 22 |
+
verl_opd/ OPD 全部线用的 verl —— 与上游 thunlp/OPD 的计算路径逐字节等价
|
| 23 |
+
verl_sft/ teacher-SFT 用的 verl
|
| 24 |
+
agentgym_rl/ AgentGym-RL —— 本项目里只用于 SciWorld 评测, 未用于训练
|
| 25 |
+
trinity/ Trinity —— SciWorld 的 ROSE/OPD 八条线是它训的
|
| 26 |
+
sciworld/
|
| 27 |
+
impl/ 自写的多轮 ROSE/OPD 实现(不依赖上面任何框架)
|
| 28 |
+
chains/ 编排脚本
|
| 29 |
+
trinity_configs/ Trinity 侧的 workflow 与 yaml
|
| 30 |
+
agentgym_eval/ AgentGym 口径的 200 题评测
|
| 31 |
+
launchers/
|
| 32 |
+
rlve/ dapo/ olmo/
|
| 33 |
+
eval/ AIME25 / HMMT25 / RLVE 评测
|
| 34 |
+
docs/ ENV_NOTES.md —— 跨家族 ROSE 的环境硬约束
|
| 35 |
+
tools/ 归档上传用的脚本
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
## 三种方法
|
| 39 |
+
|
| 40 |
+
**teacher-SFT** 教师采轨迹, 学生离线模仿。
|
| 41 |
+
|
| 42 |
+
**OPD**(on-policy distillation)教师对**学生自己采出来的 token** 打分,
|
| 43 |
+
`advantages = teacher_logprobs - student_logprobs`, `adv_estimator=token_reward_direct`,
|
| 44 |
+
不做归一化。
|
| 45 |
+
|
| 46 |
+
**ROSE** 学生写 K 个 token 的前缀(mask=0), 教师从那里接着写 t 个 token,
|
| 47 |
+
CE 只打在教师 token 上(mask=1)。学生学的是"我走到这个状态之后, 老师会怎么继续"。
|
| 48 |
+
`recipe/rose/rose_agent_loop.py` 里有完整说明。
|
| 49 |
+
|
| 50 |
+
## SciWorld 有两套并行实现
|
| 51 |
+
|
| 52 |
+
| 实现 | 位置 | 产物 |
|
| 53 |
+
|---|---|---|
|
| 54 |
+
| **Trinity** | `frameworks/trinity` + `sciworld/trinity_configs` | `SCIENCEWORLD_{OPD,ROSE}/` 八条线 |
|
| 55 |
+
| **自写多轮** | `sciworld/impl/online_{rose,opd}_sciworld.py` | 老格式 `log.jsonl + step_N/`(checkpoint 已清, 脚本保留) |
|
| 56 |
+
|
| 57 |
+
`sciworld/impl/` 那套不依赖 verl 也不依赖 Trinity, 是从头写的:
|
| 58 |
+
学生走 N 轮 → 按 turn 边界构造 loss mask → 教师接管后续轮次 → 只在教师 turn 上算 CE。
|
| 59 |
+
多轮的 mask 构造见 `online_rose_sciworld.py`; OPD 侧的 `forward_kl_topk` 见 `online_opd_sciworld.py`。
|
| 60 |
+
|
| 61 |
+
**AgentGym-RL 只用于评测**(`sciworld_agentgym_eval.py`, 200 题 / ≤20 轮 / greedy / nothink),
|
| 62 |
+
本项目没有用它训练过。
|
| 63 |
+
|
| 64 |
+
## 结果
|
| 65 |
+
|
| 66 |
+
### RLVE (16k, n=8)
|
| 67 |
+
|
| 68 |
+
| line | avg@8 | pass@8 | 解出 |
|
| 69 |
+
|---|---|---|---|
|
| 70 |
+
| 基线 未训练 | 0.0333 | 0.1111 | 20 |
|
| 71 |
+
| official OPD bf16 (140步) | 0.0458 | 0.1444 | 26 |
|
| 72 |
+
| OPD bf16 mini16 (140步) | 0.0556 | 0.1722 | 31 |
|
| 73 |
+
| **OPD fp32 (105步)** | **0.0694** | 0.1833 | 33 |
|
| 74 |
+
| OPD fp32 (132步) | 0.0667 | 0.1833 | 33 |
|
| 75 |
+
| **official ROSE (140步)** | **0.0701** | 0.1778 | 32 |
|
| 76 |
+
| official ROSE seed1 | 0.0729 | 0.1889 | 34 |
|
| 77 |
+
| teacher-SFT n4 bs256 | 0.0556 | 0.1500 | 27 |
|
| 78 |
+
|
| 79 |
+
fp32 那条与 official bf16 **只差 dtype**(train/mini/n、lr、resp、更新次数全同)。
|
| 80 |
+
上游 `on_policy_distillation.sh` 默认就是 fp32, bf16 是早期的偏离 ——
|
| 81 |
+
lr 1e-6 × advantage 10⁻² 的每步相对更新量低于 bf16 的分辨率, 会被舍入吞掉。
|
| 82 |
+
|
| 83 |
+
### SciWorld (AgentGym 口径, 200 题, ≤20 轮, greedy, nothink)
|
| 84 |
+
|
| 85 |
+
| | Score | Success | Pass(>0) |
|
| 86 |
+
|---|---|---|---|
|
| 87 |
+
| Qwen3-1.7B 基座 | 0.0479 | 0.0000 | 0.1650 |
|
| 88 |
+
| OPD lr1e-6 | 0.0421 | — | — |
|
| 89 |
+
| OPD lr1e-5 | 0.1931 | — | — |
|
| 90 |
+
| teacher-SFT | 0.2077 | 0.0450 | 0.4100 |
|
| 91 |
+
| **ROSE** | **0.3127** | — | — |
|
| 92 |
+
| Qwen3-32B 教师 | 0.4346 | 0.1100 | 0.9350 |
|
| 93 |
+
|
| 94 |
+
ROSE 在 SciWorld 上的优势(+0.105 over SFT, +0.12 over OPD)远大于 RLVE 上的(+0.016 / +0.002)。
|
| 95 |
+
|
| 96 |
+
### DAPO-Math 上的 ROSE 是负结果
|
| 97 |
+
|
| 98 |
+
AIME25 / HMMT25 avg@16, thinking, temp0.7 / top_p0.95 / max_new 31744:
|
| 99 |
+
|
| 100 |
+
| | AIME25 | HMMT25 |
|
| 101 |
+
|---|---|---|
|
| 102 |
+
| Qwen3-32B 教师 | 0.7125 | 0.5042 |
|
| 103 |
+
| Qwen3-4B 未训练 | 0.6521 | — |
|
| 104 |
+
| ROSE←32B 30步 (4B) | 0.6125 | — |
|
| 105 |
+
| Olmo-3-7B 基线 | 0.5917 | 0.4167 |
|
| 106 |
+
| ROSE←32B 30步 (Olmo-7B) | 0.5750 | 0.3854 |
|
| 107 |
+
| **ROSE←32B 220步 (Olmo-7B)** | **0.5583** | **0.3688** |
|
| 108 |
+
|
| 109 |
+
四组独立观测同向, 且 Olmo 从 30 步到 220 步(跑满一个 epoch)继续下降 —— **排除欠训**。
|
| 110 |
+
训练本身也早停滞: ce_loss 从 31 步起斜率 −0.00012, grad_norm 降到 0.18。
|
| 111 |
+
|
| 112 |
+
诊断: `TEACHER_MAX_TOKENS=1024`, 而实测 `ce_target_tokens` 达 **960-1016**
|
| 113 |
+
(占预算 94-99%)—— 教师几乎每条都是写到预算耗尽被截断, 从不自己收尾。
|
| 114 |
+
配合学生 16000-18600 token 的推理链, 监督只覆盖 **token 1024-2048, 约 6%**。
|
| 115 |
+
学生学到的是"在思考中段接着往下写", 从没被教过怎么收敛到答���。
|
| 116 |
+
表现是**输出方差变大而能力边界未动**: pass@16 靠多蒙中一题涨(两个 benchmark 各 +1 题,
|
| 117 |
+
其中一题是 0/16 → 1/16), 而 avg@16 与 pass@1 都下降, 截断率上升。
|
| 118 |
+
|
| 119 |
+
要检验这个诊断, 应该把 K 提到 4096-8192、teacher 预算给到 2048-4096,
|
| 120 |
+
让切点落到推理链中后段。
|
| 121 |
+
|
| 122 |
+
## 跨家族 ROSE(Olmo-3-7B ← Qwen3-32B)
|
| 123 |
+
|
| 124 |
+
学生 vocab 100278 / 教师 151669, 完全不同的词表。`rose_agent_loop` 有现成通路
|
| 125 |
+
(`TEACHER_TOKENIZER` 非 null 即启用): 学生前缀 ids → decode 成文本 →
|
| 126 |
+
套**教师的** chat template → 教师续写 → 取回 text → 用**学生的** tokenizer 重新 encode。
|
| 127 |
+
实测两边对拼接缝文本的 round-trip 都无损, 教师续写语义连贯且不会重复吐 `<think>`。
|
| 128 |
+
|
| 129 |
+
环境上的硬约束(vLLM 版本被夹死在 0.11.0、NCCL 只能关 NVLS、verl 序列打包硬依赖
|
| 130 |
+
flash_attn 等)见 `docs/ENV_NOTES.md` —— 每一条都对应一次失败的启动。
|
docs/ENV_NOTES.md
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 跨家族 ROSE 的环境约束(Olmo-3-7B ← Qwen3-32B)
|
| 2 |
+
|
| 3 |
+
把一个**非 Qwen 家族**的学生接进这套 ROSE 时踩到的全部硬约束。都是实测出来的,
|
| 4 |
+
不是从文档抄的 —— 每一条都对应一次失败的启动。
|
| 5 |
+
|
| 6 |
+
## 1. vLLM 版本被夹死在 0.11.0
|
| 7 |
+
|
| 8 |
+
verl 0.7.0.dev 与 vLLM 之间有三处接口漂移,加上 Olmo3 支持,四条同时满足的只有 **0.11.0**:
|
| 9 |
+
|
| 10 |
+
| 版本 | Olmo3 | `vllm.lora.models` | `vllm.utils.FlexibleArgumentParser` | transformers v4 |
|
| 11 |
+
|---|---|---|---|---|
|
| 12 |
+
| 0.10.2 | 不支持 | 有 | 有 | 要 v4 |
|
| 13 |
+
| **0.11.0** | **支持** | **有** | **有** | **要 v4** |
|
| 14 |
+
| 0.11.1 / 0.11.2 | 支持 | 有 | **没了** | 要 v4 |
|
| 15 |
+
| 0.12.0 | 支持 | 有 | **没了** | 要 v4 |
|
| 16 |
+
| 0.23.0 | 支持 | **没了** | — | 要 v4 |
|
| 17 |
+
| ≥0.24.0 | 支持 | — | — | **强制 v5** |
|
| 18 |
+
|
| 19 |
+
- transformers 必须留在 **4.57.6**:verl 0.7 要 `AutoModelForVision2Seq`,v5 里删了;
|
| 20 |
+
而 4.57.6 本身已经认 `Olmo3Config`,所以不需要为 Olmo 升 transformers。
|
| 21 |
+
- `flashinfer-cubin` 要与 `flashinfer-python` **同版本**(装过别的 vllm 会留下不匹配的
|
| 22 |
+
cubin,报 `flashinfer-cubin version (0.6.12) does not match flashinfer version (0.5.2)`)。
|
| 23 |
+
|
| 24 |
+
## 2. NCCL:只关 NVLS,千万别关 P2P
|
| 25 |
+
|
| 26 |
+
本机(8×H100,驱动 580.65,cu130)默认起 TP>1 会 `NCCL error: unhandled cuda error`。
|
| 27 |
+
逐项扫下来:
|
| 28 |
+
|
| 29 |
+
```
|
| 30 |
+
什么都不设 -> 报错
|
| 31 |
+
NCCL_NVLS_ENABLE=0 -> OK ← 只需要这一个
|
| 32 |
+
NCCL_CUMEM_ENABLE=0 + NVLS=0 -> OK
|
| 33 |
+
NCCL_CUMEM_ENABLE=0 + SHM=1 -> 报错
|
| 34 |
+
NCCL_P2P_LEVEL=NVL -> 报错
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
坏的是 NVLink SHARP 组播(NVLS)的初始化。
|
| 38 |
+
|
| 39 |
+
**代价警告**:一开始我是一组一组加着试的,`NCCL_P2P_DISABLE=1` 跟着
|
| 40 |
+
`NCCL_SHM_DISABLE=1` 一起进去了,试出"能跑"就停了没做减法 —— 结果关掉 NVLink,
|
| 41 |
+
TP 的 all-reduce 退回 PCIe/主机中转,**吞吐从 2000 tok/s 掉到 124 tok/s(16 倍)**。
|
| 42 |
+
凡是靠"加一组变量"试出来的解,必须回头逐个去掉验证哪个是真正必需的。
|
| 43 |
+
|
| 44 |
+
## 3. flashinfer 需要 ninja
|
| 45 |
+
|
| 46 |
+
`FileNotFoundError: 'ninja'` —— flashinfer 要 JIT 编译采样 kernel。
|
| 47 |
+
装 ninja 包不够(pip 装的没有可执行文件在 PATH 上),直接
|
| 48 |
+
`VLLM_USE_FLASHINFER_SAMPLER=0` 绕开最省事。
|
| 49 |
+
|
| 50 |
+
## 4. vLLM 强制 spawn 时,平铺脚本会炸
|
| 51 |
+
|
| 52 |
+
vLLM 0.27 对部分模型强制用 spawn 起 worker,子进程会重新 import 父进程的 `__main__`。
|
| 53 |
+
平铺脚本(逻辑全在模块层)被重新 import 就触发
|
| 54 |
+
`RuntimeError: freeze_support / Safe importing of main module`。
|
| 55 |
+
|
| 56 |
+
`aime_launch.py` 用一个 guard 绕开,**关键是 `runpy.run_path` 的 `run_name` 不能是
|
| 57 |
+
`"__main__"`** —— runpy 在执行期间会把 `sys.modules["__main__"]` 临时换成目标脚本,
|
| 58 |
+
spawn 读到的还是它,等于没绕开(第一版就栽在这)。用别的名字即可。
|
| 59 |
+
|
| 60 |
+
## 5. verl 的序列打包硬依赖 flash_attn
|
| 61 |
+
|
| 62 |
+
`verl/utils/attention_utils.py:28` 在 CUDA 分支直接
|
| 63 |
+
`from flash_attn.bert_padding import ...`,**没有 fallback**,而且与
|
| 64 |
+
`attn_implementation` 无关 —— 就算模型跑 sdpa,只要 `use_remove_padding=True` 就会 import。
|
| 65 |
+
|
| 66 |
+
torch 2.11+cu130 没有 flash-attn 预编译 wheel(源码编译 1-2 小时),所以:
|
| 67 |
+
`use_remove_padding=False` + `attn_implementation=sdpa`。
|
| 68 |
+
代价是 padding 到等长,显存吃满(7B 在 4×H100 上每卡约 71GB)、速度也慢。
|
| 69 |
+
|
| 70 |
+
`run_rose_online.sh` 把 `use_remove_padding=True` 写死在第 250 行且末尾没有 `"$@"` 透传,
|
| 71 |
+
所以复制出 `run_rose_online_sdpa.sh`,末尾加三行可覆盖的开关(默认值与原件一致),
|
| 72 |
+
**原件一字未动**。
|
| 73 |
+
|
| 74 |
+
## 6. 跨分词器本身不用改代码
|
| 75 |
+
|
| 76 |
+
`rose_agent_loop` 早就有这条通路,`TEACHER_TOKENIZER` 非 null 即启用:
|
| 77 |
+
学生前缀 ids → decode 成文本 → 套**教师的** chat template → 教师续写 → 取回 `text`
|
| 78 |
+
→ 用**学生的** tokenizer 重新 encode。
|
| 79 |
+
|
| 80 |
+
Olmo(vocab 100278)× Qwen3-32B(vocab 151669)实测:
|
| 81 |
+
- 拼接缝文本两边 round-trip 都无损
|
| 82 |
+
- 教师续写语义连贯,**没有重复吐 `<think>`**
|
| 83 |
+
- 格式错位是良性的:Olmo 模板在 prompt 尾部就打开 `<think>`,Qwen3 模板停在
|
| 84 |
+
`assistant\n`。教师看到「`assistant\n` + 推理文字」,续写出的 `</think>`+答案
|
| 85 |
+
接在学生的 `<think>` 之后,正好构成 Olmo 期望的完整格式。
|
| 86 |
+
|
| 87 |
+
验证脚本:`xjoin.py`(需要 teacher server 已起)。
|
| 88 |
+
|
| 89 |
+
## 7. `TEACHER_MODEL` 不是路径
|
| 90 |
+
|
| 91 |
+
`bootstrap.sh teacher` 用 `--served-model-name teacher`,所以 API 的 model 字段就是
|
| 92 |
+
字符串 `teacher`(`run_rose_online.sh:45` 的默认值)。传模型路径会 404。
|
| 93 |
+
|
| 94 |
+
## 8. `SAVE_FREQ` 拦不住 final save
|
| 95 |
+
|
| 96 |
+
verl 在训练正常结束时**还会存一次**,与 `save_freq` 无关。3 步的冒烟测试也照存 33G
|
| 97 |
+
(Olmo-7B:model 分片 8.4G + optimizer 15.2G + HF 8.3G),当场把盘撑满。
|
| 98 |
+
冒烟测试前先确认剩余空间,或者跑完立刻清。
|
eval/aime_eval.py
ADDED
|
@@ -0,0 +1,223 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""AIME25 avg@k 评测。
|
| 2 |
+
|
| 3 |
+
针对 non-thinking 模型: prompt 必须用 enable_thinking=False 渲染
|
| 4 |
+
(尾部会带 <think>\n\n</think>\n\n)。见 MEMORY: nothink-flag-for-1p7b-sft。
|
| 5 |
+
|
| 6 |
+
答案抽取按优先级: \boxed{} -> "Answer: X" -> 文末最后一个整数。
|
| 7 |
+
AIME 答案恒为 0-999 的整数, 比较时按 int 比。
|
| 8 |
+
"""
|
| 9 |
+
import argparse
|
| 10 |
+
import json
|
| 11 |
+
import os
|
| 12 |
+
import re
|
| 13 |
+
import sys
|
| 14 |
+
import time
|
| 15 |
+
from collections import Counter
|
| 16 |
+
|
| 17 |
+
p = argparse.ArgumentParser()
|
| 18 |
+
p.add_argument("--model", required=True)
|
| 19 |
+
p.add_argument("--data", default="/home/haojinw2/datasets/aime25/test.jsonl")
|
| 20 |
+
p.add_argument("--out", required=True)
|
| 21 |
+
p.add_argument("--n", type=int, default=16)
|
| 22 |
+
p.add_argument("--max-new", type=int, default=8192)
|
| 23 |
+
p.add_argument("--ctx", type=int, default=10240)
|
| 24 |
+
p.add_argument("--tp", type=int, default=1)
|
| 25 |
+
p.add_argument("--temp", type=float, default=0.7)
|
| 26 |
+
p.add_argument("--top-p", type=float, default=0.8)
|
| 27 |
+
p.add_argument("--top-k", type=int, default=20)
|
| 28 |
+
p.add_argument("--seed", type=int, default=0)
|
| 29 |
+
p.add_argument("--enable-thinking", type=lambda s: s.lower() == "true", default=False,
|
| 30 |
+
help="False = 关闭 think(non-thinking 模型必须用 False)")
|
| 31 |
+
p.add_argument("--gpu-util", type=float, default=0.85)
|
| 32 |
+
a = p.parse_args()
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def log(*x):
|
| 36 |
+
print(f"[{time.strftime('%H:%M:%S')}]", *x, flush=True)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
INSTR = "Please reason step by step, and put your final answer within \\boxed{}."
|
| 40 |
+
|
| 41 |
+
BOXED = re.compile(r"\\boxed\s*\{([^{}]*)\}")
|
| 42 |
+
ANSW = re.compile(r"[Aa]nswer\s*[::]\s*\$?(-?\d+)")
|
| 43 |
+
INTS = re.compile(r"-?\d+")
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def _safe_int(s):
|
| 47 |
+
"""AIME 答案恒为 0-999。超长数字是模型退化的产物(见过 28000 位),
|
| 48 |
+
直接 int() 会撞 Python 3.11 的 int_max_str_digits=4300 抛 ValueError。"""
|
| 49 |
+
if s is None:
|
| 50 |
+
return None
|
| 51 |
+
t = s.lstrip("-")
|
| 52 |
+
if len(t) > 10 or not t:
|
| 53 |
+
return None
|
| 54 |
+
try:
|
| 55 |
+
return int(s)
|
| 56 |
+
except ValueError:
|
| 57 |
+
return None
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def extract(text):
|
| 61 |
+
m = BOXED.findall(text)
|
| 62 |
+
if m:
|
| 63 |
+
digits = INTS.findall(m[-1])
|
| 64 |
+
if digits:
|
| 65 |
+
v = _safe_int(digits[-1])
|
| 66 |
+
if v is not None:
|
| 67 |
+
return v
|
| 68 |
+
m = ANSW.findall(text)
|
| 69 |
+
if m:
|
| 70 |
+
v = _safe_int(m[-1])
|
| 71 |
+
if v is not None:
|
| 72 |
+
return v
|
| 73 |
+
m = INTS.findall(text)
|
| 74 |
+
if m:
|
| 75 |
+
return _safe_int(m[-1])
|
| 76 |
+
return None
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def _kill_descendants():
|
| 80 |
+
"""递归杀掉本进程的所有子孙(vLLM worker 是独立进程, 不杀会变孤儿占满卡)。"""
|
| 81 |
+
import signal
|
| 82 |
+
import subprocess
|
| 83 |
+
me = os.getpid()
|
| 84 |
+
try:
|
| 85 |
+
out = subprocess.run(["ps", "-eo", "pid,ppid", "--no-headers"],
|
| 86 |
+
capture_output=True, text=True, timeout=20).stdout
|
| 87 |
+
except Exception:
|
| 88 |
+
return
|
| 89 |
+
children = {}
|
| 90 |
+
for ln in out.splitlines():
|
| 91 |
+
parts = ln.split()
|
| 92 |
+
if len(parts) != 2:
|
| 93 |
+
continue
|
| 94 |
+
try:
|
| 95 |
+
pid, ppid = int(parts[0]), int(parts[1])
|
| 96 |
+
except ValueError:
|
| 97 |
+
continue
|
| 98 |
+
children.setdefault(ppid, []).append(pid)
|
| 99 |
+
doomed, stack = [], [me]
|
| 100 |
+
while stack:
|
| 101 |
+
for c in children.get(stack.pop(), []):
|
| 102 |
+
doomed.append(c)
|
| 103 |
+
stack.append(c)
|
| 104 |
+
for pid in reversed(doomed): # 先叶子后根
|
| 105 |
+
try:
|
| 106 |
+
os.kill(pid, signal.SIGKILL)
|
| 107 |
+
except OSError:
|
| 108 |
+
pass
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def _hard_exit(code=0):
|
| 112 |
+
sys.stdout.flush()
|
| 113 |
+
sys.stderr.flush()
|
| 114 |
+
_kill_descendants()
|
| 115 |
+
os._exit(code)
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def _die(exc_type, exc, tb):
|
| 119 |
+
import traceback as _tb
|
| 120 |
+
_tb.print_exception(exc_type, exc, tb)
|
| 121 |
+
_hard_exit(1) # 异常路径同样要把 worker 一起带走
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
sys.excepthook = _die
|
| 125 |
+
|
| 126 |
+
rows = [json.loads(l) for l in open(a.data) if l.strip()]
|
| 127 |
+
log(f"{len(rows)} 题 x n={a.n}, enable_thinking={a.enable_thinking}")
|
| 128 |
+
|
| 129 |
+
from transformers import AutoTokenizer
|
| 130 |
+
from vllm import LLM, SamplingParams
|
| 131 |
+
|
| 132 |
+
tok = AutoTokenizer.from_pretrained(a.model, trust_remote_code=True)
|
| 133 |
+
prompts = []
|
| 134 |
+
for r in rows:
|
| 135 |
+
msgs = [{"role": "user", "content": r["problem"].strip() + "\n\n" + INSTR}]
|
| 136 |
+
prompts.append(tok.apply_chat_template(
|
| 137 |
+
msgs, add_generation_prompt=True, tokenize=False,
|
| 138 |
+
enable_thinking=a.enable_thinking))
|
| 139 |
+
log("prompt 尾部抽检: " + repr(prompts[0][-60:]))
|
| 140 |
+
|
| 141 |
+
def _fit_tp(model_path, tp):
|
| 142 |
+
"""attn heads 必须能被 TP 整除, 否则 vLLM 起不来(R1-Distill-1.5B 是 12 heads)。"""
|
| 143 |
+
try:
|
| 144 |
+
cfg = json.load(open(os.path.join(model_path, "config.json")))
|
| 145 |
+
h = cfg.get("num_attention_heads")
|
| 146 |
+
except Exception:
|
| 147 |
+
return tp
|
| 148 |
+
if not h or h % tp == 0:
|
| 149 |
+
return tp
|
| 150 |
+
for cand in range(tp, 0, -1):
|
| 151 |
+
if h % cand == 0:
|
| 152 |
+
log(f"注意: {h} 个 attention head 不能被 TP={tp} 整除, 自动降为 TP={cand}")
|
| 153 |
+
return cand
|
| 154 |
+
return 1
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
a.tp = _fit_tp(a.model, a.tp)
|
| 158 |
+
llm = LLM(model=a.model, tensor_parallel_size=a.tp,
|
| 159 |
+
gpu_memory_utilization=a.gpu_util, max_model_len=a.ctx,
|
| 160 |
+
trust_remote_code=True, seed=a.seed)
|
| 161 |
+
# 不给 SamplingParams 传 seed: per-request seed 会让 n>1 退出批量采样快路径,
|
| 162 |
+
# 480 个 seeded 样本会逐个采样、慢到形同挂死。可复现性由 LLM(seed=) 保证。
|
| 163 |
+
sp = SamplingParams(n=a.n, temperature=a.temp, top_p=a.top_p, top_k=a.top_k,
|
| 164 |
+
max_tokens=a.max_new)
|
| 165 |
+
|
| 166 |
+
t0 = time.time()
|
| 167 |
+
outs = llm.generate(prompts, sp)
|
| 168 |
+
log(f"生成 {time.time()-t0:.0f}s")
|
| 169 |
+
|
| 170 |
+
recs, n_correct, n_tot, n_trunc, lens = [], 0, 0, 0, []
|
| 171 |
+
per_problem = []
|
| 172 |
+
for r, out in zip(rows, outs):
|
| 173 |
+
gt = int(str(r["answer"]).strip())
|
| 174 |
+
hits, samples = 0, []
|
| 175 |
+
for o in out.outputs:
|
| 176 |
+
pred = extract(o.text)
|
| 177 |
+
ok = (pred is not None and pred == gt)
|
| 178 |
+
hits += ok
|
| 179 |
+
n_correct += ok
|
| 180 |
+
n_tot += 1
|
| 181 |
+
n_trunc += (o.finish_reason == "length")
|
| 182 |
+
lens.append(len(o.token_ids))
|
| 183 |
+
samples.append({"pred": pred, "ok": bool(ok),
|
| 184 |
+
"finish": o.finish_reason, "len": len(o.token_ids),
|
| 185 |
+
"text": o.text})
|
| 186 |
+
per_problem.append({"id": r.get("id"), "gt": gt, "hits": hits, "n": len(out.outputs)})
|
| 187 |
+
recs.append({"id": r.get("id"), "problem": r["problem"], "gt": gt,
|
| 188 |
+
"hits": hits, "samples": samples})
|
| 189 |
+
|
| 190 |
+
os.makedirs(os.path.dirname(os.path.abspath(a.out)) or ".", exist_ok=True)
|
| 191 |
+
with open(a.out, "w") as f:
|
| 192 |
+
for rec in recs:
|
| 193 |
+
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
| 194 |
+
|
| 195 |
+
avg = n_correct / n_tot
|
| 196 |
+
solved_any = sum(1 for x in per_problem if x["hits"] > 0)
|
| 197 |
+
solved_all = sum(1 for x in per_problem if x["hits"] == x["n"])
|
| 198 |
+
summary = {
|
| 199 |
+
"model": a.model, "data": a.data, "n": a.n,
|
| 200 |
+
"enable_thinking": a.enable_thinking,
|
| 201 |
+
"temperature": a.temp, "top_p": a.top_p, "top_k": a.top_k,
|
| 202 |
+
"max_new": a.max_new, "seed": a.seed,
|
| 203 |
+
"problems": len(rows), "samples": n_tot,
|
| 204 |
+
f"avg@{a.n}": avg,
|
| 205 |
+
"solved_any": solved_any, "solved_all": solved_all,
|
| 206 |
+
"truncated_rate": n_trunc / n_tot,
|
| 207 |
+
"resp_len_mean": sum(lens) / len(lens),
|
| 208 |
+
"resp_len_max": max(lens),
|
| 209 |
+
"hits_hist": dict(Counter(x["hits"] for x in per_problem)),
|
| 210 |
+
}
|
| 211 |
+
with open(a.out.replace(".jsonl", "_summary.json"), "w") as f:
|
| 212 |
+
json.dump(summary, f, indent=2, ensure_ascii=False)
|
| 213 |
+
|
| 214 |
+
print()
|
| 215 |
+
print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True)
|
| 216 |
+
print()
|
| 217 |
+
print(f"AIME25 avg@{a.n} = {avg:.4f} ({n_correct}/{n_tot})", flush=True)
|
| 218 |
+
print(f" 解出>=1次 {solved_any}/{len(rows)}, 全对 {solved_all}/{len(rows)}", flush=True)
|
| 219 |
+
print(f" 截断率 {n_trunc/n_tot:.3f}, 平均长度 {sum(lens)/len(lens):.0f}", flush=True)
|
| 220 |
+
|
| 221 |
+
# vLLM TP>1 的 worker 常在析构时挂尸、占着整卡不放, 后续任务就起不来。
|
| 222 |
+
# 结果已全部写盘, 直接跳过析构强制退出。
|
| 223 |
+
_hard_exit(0)
|
eval/aime_launch.py
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""spawn 安全的 aime_eval 启动器。
|
| 2 |
+
|
| 3 |
+
vllm 0.27 对 Qwen3.5 这类模型强制用 spawn 起 worker, 子进程会重新 import
|
| 4 |
+
父进程的 __main__。aime_eval.py 是平铺脚本, 被重新 import 就会触发
|
| 5 |
+
multiprocessing 的 "freeze_support / Safe importing of main module" 报错。
|
| 6 |
+
|
| 7 |
+
关键是 run_name 不能用 "__main__": runpy 在执行期间会把 sys.modules["__main__"]
|
| 8 |
+
临时换成目标脚本, spawn 读到的就还是 aime_eval.py, 等于没绕开(先踩过这个)。
|
| 9 |
+
用别的名字, sys.modules["__main__"] 始终是本文件, 子进程 import 到 __mp_main__,
|
| 10 |
+
下面的 guard 不成立, 什么都不执行。aime_eval.py 的逻辑全在模块层, 不依赖 __name__。
|
| 11 |
+
"""
|
| 12 |
+
import runpy
|
| 13 |
+
import sys
|
| 14 |
+
|
| 15 |
+
if __name__ == "__main__":
|
| 16 |
+
sys.argv[0] = "/home/haojinw2/aime_eval.py"
|
| 17 |
+
runpy.run_path("/home/haojinw2/aime_eval.py", run_name="__aime_eval__")
|
eval/hmmt_eval.py
ADDED
|
@@ -0,0 +1,165 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""HMMT25 avg@k 评测。
|
| 2 |
+
|
| 3 |
+
与 aime_eval.py 的差别只在答案比对: HMMT 的标准答案 30 题里只有 14 题是纯整数,
|
| 4 |
+
其余是 LaTeX 表达式(\frac{1}{576}、\frac{9\sqrt{23}}{23}、1-\frac{2}{\pi}、
|
| 5 |
+
甚至 "\frac{-1+\sqrt{17}}{2}, \frac{-1-\sqrt{17}}{2}" 这种多解)。
|
| 6 |
+
按 int() 比会把 16 题全判错, 上限只有 14/30 —— 所以改用 math_verify 做符号等价判定。
|
| 7 |
+
|
| 8 |
+
采样与输出格式跟 aime_eval.py 保持一致, 便于并排比较。
|
| 9 |
+
"""
|
| 10 |
+
import argparse
|
| 11 |
+
import json
|
| 12 |
+
import os
|
| 13 |
+
import sys
|
| 14 |
+
import time
|
| 15 |
+
from collections import Counter
|
| 16 |
+
|
| 17 |
+
p = argparse.ArgumentParser()
|
| 18 |
+
p.add_argument("--model", required=True)
|
| 19 |
+
p.add_argument("--data", default="/home/haojinw2/datasets/hmmt25/test.jsonl")
|
| 20 |
+
p.add_argument("--out", required=True)
|
| 21 |
+
p.add_argument("--n", type=int, default=16)
|
| 22 |
+
p.add_argument("--max-new", type=int, default=31744)
|
| 23 |
+
p.add_argument("--ctx", type=int, default=33792)
|
| 24 |
+
p.add_argument("--tp", type=int, default=8)
|
| 25 |
+
p.add_argument("--temp", type=float, default=0.7)
|
| 26 |
+
p.add_argument("--top-p", type=float, default=0.95)
|
| 27 |
+
p.add_argument("--top-k", type=int, default=-1)
|
| 28 |
+
p.add_argument("--seed", type=int, default=0)
|
| 29 |
+
p.add_argument("--enable-thinking", type=lambda s: s.lower() == "true", default=True)
|
| 30 |
+
p.add_argument("--gpu-util", type=float, default=0.90)
|
| 31 |
+
a = p.parse_args()
|
| 32 |
+
|
| 33 |
+
INSTR = ("\n\nPlease reason step by step, and put your final answer within \\boxed{}.")
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _kill_descendants():
|
| 37 |
+
"""递归杀掉所有子孙 —— vLLM worker 是独立进程, 主进程一退它们就变孤儿占满卡。"""
|
| 38 |
+
import signal
|
| 39 |
+
import subprocess
|
| 40 |
+
me = os.getpid()
|
| 41 |
+
try:
|
| 42 |
+
out = subprocess.run(["ps", "-eo", "pid,ppid"], capture_output=True, text=True, timeout=20).stdout
|
| 43 |
+
except Exception:
|
| 44 |
+
return
|
| 45 |
+
kids = {}
|
| 46 |
+
for line in out.splitlines()[1:]:
|
| 47 |
+
parts = line.split()
|
| 48 |
+
if len(parts) >= 2:
|
| 49 |
+
try:
|
| 50 |
+
kids.setdefault(int(parts[1]), []).append(int(parts[0]))
|
| 51 |
+
except ValueError:
|
| 52 |
+
pass
|
| 53 |
+
order, stack = [], [me]
|
| 54 |
+
while stack:
|
| 55 |
+
cur = stack.pop()
|
| 56 |
+
for k in kids.get(cur, []):
|
| 57 |
+
order.append(k)
|
| 58 |
+
stack.append(k)
|
| 59 |
+
for pid in reversed(order):
|
| 60 |
+
try:
|
| 61 |
+
os.kill(pid, signal.SIGKILL)
|
| 62 |
+
except Exception:
|
| 63 |
+
pass
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def _hard_exit(code=0):
|
| 67 |
+
sys.stdout.flush()
|
| 68 |
+
sys.stderr.flush()
|
| 69 |
+
_kill_descendants()
|
| 70 |
+
os._exit(code)
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
sys.excepthook = lambda *e: (__import__("traceback").print_exception(*e), _hard_exit(1))
|
| 74 |
+
|
| 75 |
+
rows = [json.loads(l) for l in open(a.data) if l.strip()]
|
| 76 |
+
print(f"{len(rows)} 题, n={a.n}, max_new={a.max_new}, thinking={a.enable_thinking}", flush=True)
|
| 77 |
+
|
| 78 |
+
from transformers import AutoTokenizer
|
| 79 |
+
from vllm import LLM, SamplingParams
|
| 80 |
+
|
| 81 |
+
tok = AutoTokenizer.from_pretrained(a.model, trust_remote_code=True)
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def render(problem):
|
| 85 |
+
msgs = [{"role": "user", "content": problem + INSTR}]
|
| 86 |
+
try:
|
| 87 |
+
return tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
|
| 88 |
+
enable_thinking=a.enable_thinking)
|
| 89 |
+
except TypeError:
|
| 90 |
+
return tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
prompts = [render(r["problem"]) for r in rows]
|
| 94 |
+
|
| 95 |
+
llm = LLM(model=a.model, tensor_parallel_size=a.tp, max_model_len=a.ctx,
|
| 96 |
+
gpu_memory_utilization=a.gpu_util, trust_remote_code=True, seed=a.seed)
|
| 97 |
+
sp = SamplingParams(n=a.n, temperature=a.temp, top_p=a.top_p, top_k=a.top_k,
|
| 98 |
+
max_tokens=a.max_new, seed=a.seed)
|
| 99 |
+
t0 = time.time()
|
| 100 |
+
outs = llm.generate(prompts, sp)
|
| 101 |
+
print(f"生成用时 {(time.time() - t0) / 60:.1f} 分钟", flush=True)
|
| 102 |
+
|
| 103 |
+
from math_verify import parse, verify
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def is_correct(text, gt):
|
| 107 |
+
"""math_verify 做符号等价判定; 解析不出来就退回去掉空白的字符串比对。"""
|
| 108 |
+
try:
|
| 109 |
+
pred = parse(text)
|
| 110 |
+
if pred and verify(parse("$" + str(gt) + "$"), pred):
|
| 111 |
+
return True
|
| 112 |
+
except Exception:
|
| 113 |
+
pass
|
| 114 |
+
import re
|
| 115 |
+
m = re.findall(r"\\boxed\{([^{}]*(?:\{[^{}]*\}[^{}]*)*)\}", text)
|
| 116 |
+
if m:
|
| 117 |
+
s = m[-1].replace(" ", "").replace("\\!", "").replace("\\,", "")
|
| 118 |
+
return s == str(gt).replace(" ", "")
|
| 119 |
+
return False
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
recs, per_problem, lens, n_trunc, n_correct, n_tot = [], [], [], 0, 0, 0
|
| 123 |
+
for r, out in zip(rows, outs):
|
| 124 |
+
gt = r["answer"]
|
| 125 |
+
samples, hits = [], 0
|
| 126 |
+
for o in out.outputs:
|
| 127 |
+
ok = is_correct(o.text, gt)
|
| 128 |
+
hits += ok
|
| 129 |
+
n_correct += ok
|
| 130 |
+
n_tot += 1
|
| 131 |
+
lens.append(len(o.token_ids))
|
| 132 |
+
n_trunc += o.finish_reason == "length"
|
| 133 |
+
samples.append({"ok": bool(ok), "finish": o.finish_reason,
|
| 134 |
+
"len": len(o.token_ids), "text": o.text})
|
| 135 |
+
per_problem.append({"id": r["id"], "gt": gt, "hits": hits, "n": len(out.outputs)})
|
| 136 |
+
recs.append({"id": r["id"], "problem": r["problem"], "gt": gt, "hits": hits, "samples": samples})
|
| 137 |
+
|
| 138 |
+
os.makedirs(os.path.dirname(os.path.abspath(a.out)) or ".", exist_ok=True)
|
| 139 |
+
with open(a.out, "w") as f:
|
| 140 |
+
for rec in recs:
|
| 141 |
+
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
| 142 |
+
|
| 143 |
+
avg = n_correct / n_tot
|
| 144 |
+
summary = {
|
| 145 |
+
"model": a.model, "data": a.data, "n": a.n, "enable_thinking": a.enable_thinking,
|
| 146 |
+
"temperature": a.temp, "top_p": a.top_p, "max_new": a.max_new, "seed": a.seed,
|
| 147 |
+
"problems": len(rows), "samples": n_tot,
|
| 148 |
+
f"avg@{a.n}": avg,
|
| 149 |
+
"pass@1_first": sum(1 for x in recs if x["samples"][0]["ok"]) / len(recs),
|
| 150 |
+
"solved_any": sum(1 for x in per_problem if x["hits"] > 0),
|
| 151 |
+
"solved_all": sum(1 for x in per_problem if x["hits"] == x["n"]),
|
| 152 |
+
"truncated_rate": n_trunc / n_tot,
|
| 153 |
+
"resp_len_mean": sum(lens) / len(lens),
|
| 154 |
+
"hits_hist": dict(Counter(x["hits"] for x in per_problem)),
|
| 155 |
+
}
|
| 156 |
+
with open(a.out.replace(".jsonl", "_summary.json"), "w") as f:
|
| 157 |
+
json.dump(summary, f, indent=2, ensure_ascii=False)
|
| 158 |
+
|
| 159 |
+
print()
|
| 160 |
+
print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True)
|
| 161 |
+
print(f"\nHMMT25 avg@{a.n} = {avg:.4f} ({n_correct}/{n_tot})")
|
| 162 |
+
print(f" pass@1(第一条) {summary['pass@1_first']:.4f}")
|
| 163 |
+
print(f" 解出>=1次 {summary['solved_any']}/{len(rows)}, 全对 {summary['solved_all']}/{len(rows)}")
|
| 164 |
+
print(f" 截断率 {summary['truncated_rate']:.3f}, 平均长度 {summary['resp_len_mean']:.0f}", flush=True)
|
| 165 |
+
_hard_exit(0)
|
eval/merge_eval.py
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Merge data-parallel eval shards into one evalout directory.
|
| 3 |
+
|
| 4 |
+
python3 merge_eval.py OUT_DIR SHARD_DIR [SHARD_DIR ...]
|
| 5 |
+
|
| 6 |
+
Each shard is an evalout dir produced with IDX_START/IDX_END. Rows carry their
|
| 7 |
+
true dataframe idx, so merging is a concatenate + sort; the summary is then
|
| 8 |
+
recomputed with the same formulas rlve_eval.py uses, so the output file is
|
| 9 |
+
interchangeable with a single-process run.
|
| 10 |
+
"""
|
| 11 |
+
import json, os, sys
|
| 12 |
+
from collections import Counter
|
| 13 |
+
from math import comb
|
| 14 |
+
|
| 15 |
+
def pass_at_k(N, c, k):
|
| 16 |
+
if N - c < k: return 1.0
|
| 17 |
+
return 1.0 - comb(N - c, k) / comb(N, k)
|
| 18 |
+
|
| 19 |
+
out, shards = sys.argv[1], sys.argv[2:]
|
| 20 |
+
rows, base = [], None
|
| 21 |
+
for sd in shards:
|
| 22 |
+
with open(f"{sd}/rollouts.jsonl") as f:
|
| 23 |
+
rows += [json.loads(l) for l in f]
|
| 24 |
+
if base is None:
|
| 25 |
+
base = json.load(open(f"{sd}/summary.json"))
|
| 26 |
+
seen = set()
|
| 27 |
+
for r in rows:
|
| 28 |
+
if r["idx"] in seen: sys.exit(f"重复 idx {r['idx']} -- 分片区间有重叠, 拒绝合并")
|
| 29 |
+
seen.add(r["idx"])
|
| 30 |
+
rows.sort(key=lambda r: r["idx"])
|
| 31 |
+
|
| 32 |
+
flat = [rec for r in rows for rec in r["recs"]]
|
| 33 |
+
n = len(flat)
|
| 34 |
+
N = base["n"]
|
| 35 |
+
per_env = {}
|
| 36 |
+
for r in rows:
|
| 37 |
+
per_env.setdefault(r["env"], []).extend(rec["accuracy"] for rec in r["recs"])
|
| 38 |
+
passk = {}
|
| 39 |
+
if N > 1:
|
| 40 |
+
for k in (1, 2, 4, 8, 16):
|
| 41 |
+
if k > N: continue
|
| 42 |
+
passk[f"pass@{k}"] = sum(
|
| 43 |
+
pass_at_k(N, sum(rec["accuracy"] for rec in r["recs"]), k) for r in rows) / len(rows)
|
| 44 |
+
passk["solved_any"] = sum(1 for r in rows if any(rec["accuracy"] for rec in r["recs"]))
|
| 45 |
+
passk["solved_all"] = sum(1 for r in rows if all(rec["accuracy"] for rec in r["recs"]))
|
| 46 |
+
|
| 47 |
+
summary = dict(base)
|
| 48 |
+
# base is shard A's summary, whose "tag" carries the shard suffix. Left as-is it
|
| 49 |
+
# labels a 180-problem merge as if it were the 90-problem shard, which silently
|
| 50 |
+
# corrupts any table keyed on tag.
|
| 51 |
+
summary["tag"] = os.path.basename(out.rstrip("/")).replace("rlve_", "", 1)
|
| 52 |
+
summary.update({
|
| 53 |
+
"problems": len(rows),
|
| 54 |
+
"accuracy@1": sum(r["accuracy"] for r in flat) / n,
|
| 55 |
+
"reward_mean@1": sum(r["reward"] for r in flat) / n,
|
| 56 |
+
"format_ok_rate": sum(r["format_score"] for r in flat) / n,
|
| 57 |
+
"truncated_rate": sum(r["finish"] == "length" for r in flat) / n,
|
| 58 |
+
"closed_think_rate": sum(r["closed_think"] for r in flat) / n,
|
| 59 |
+
"resp_len_mean": sum(r["len"] for r in flat) / n,
|
| 60 |
+
"resp_len_max": max(r["len"] for r in flat),
|
| 61 |
+
"reward_hist": dict(Counter(round(r["reward"], 3) for r in flat)),
|
| 62 |
+
"n_envs": len(per_env),
|
| 63 |
+
"merged_from": shards,
|
| 64 |
+
**passk,
|
| 65 |
+
})
|
| 66 |
+
os.makedirs(out, exist_ok=True)
|
| 67 |
+
json.dump(summary, open(f"{out}/summary.json", "w"), indent=2, ensure_ascii=False)
|
| 68 |
+
json.dump({k: sum(v)/len(v) for k, v in sorted(per_env.items())},
|
| 69 |
+
open(f"{out}/per_env.json", "w"), indent=2)
|
| 70 |
+
with open(f"{out}/rollouts.jsonl", "w") as f:
|
| 71 |
+
for r in rows: f.write(json.dumps(r, ensure_ascii=False) + "\n")
|
| 72 |
+
print(json.dumps({k: v for k, v in summary.items() if k != "reward_hist"}, indent=2, ensure_ascii=False))
|
eval/retrunc_eval.py
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Re-score existing rollouts under a smaller generation budget.
|
| 3 |
+
|
| 4 |
+
retrunc_eval.py <BUDGET> <SRC_TAG> <DST_TAG>
|
| 5 |
+
|
| 6 |
+
vLLM's max_tokens does not influence the sampling distribution of earlier
|
| 7 |
+
tokens, so the first BUDGET tokens of a 32k rollout are distributed exactly as a
|
| 8 |
+
BUDGET-token rollout would be. Truncating and re-scoring is an exact
|
| 9 |
+
re-measurement at the smaller budget, not an approximation -- it just costs no
|
| 10 |
+
GPU. finish_reason is fixed up by hand: a sample that ran past BUDGET would have
|
| 11 |
+
been cut off, so it becomes 'length'.
|
| 12 |
+
|
| 13 |
+
Scoring reuses env.verifier() on the decoded text, and the per-environment
|
| 14 |
+
config comes from the source parquet -- rollouts.jsonl does not carry it, and
|
| 15 |
+
verifier() without the right config does not mean the same thing.
|
| 16 |
+
"""
|
| 17 |
+
import json, os, sys
|
| 18 |
+
from collections import Counter
|
| 19 |
+
from math import comb
|
| 20 |
+
|
| 21 |
+
GYM = "/home/haojinw2/rose-bundle-new/RLVE-Eval"
|
| 22 |
+
sys.path.insert(0, GYM)
|
| 23 |
+
import Gym.environments as G
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def score_one(meta, text):
|
| 27 |
+
cls = getattr(G, f"{meta['environment']}_Environment", None)
|
| 28 |
+
if cls is None:
|
| 29 |
+
return None
|
| 30 |
+
env = cls()
|
| 31 |
+
env.set_config(meta.get("config", {}))
|
| 32 |
+
try:
|
| 33 |
+
return env.verifier(text)
|
| 34 |
+
except Exception as e: # noqa: BLE001
|
| 35 |
+
return {"reward": -1.0, "accuracy": 0, "format_score": 0, "err": repr(e)}
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def pass_at_k(N, c, k):
|
| 39 |
+
if N - c < k:
|
| 40 |
+
return 1.0
|
| 41 |
+
return 1.0 - comb(N - c, k) / comb(N, k)
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
BUDGET, SRC, DST = int(sys.argv[1]), sys.argv[2], sys.argv[3]
|
| 45 |
+
D = "/home/haojinw2/evalout"
|
| 46 |
+
src, dst = f"{D}/rlve_{SRC}", f"{D}/rlve_{DST}"
|
| 47 |
+
base = json.load(open(f"{src}/summary.json"))
|
| 48 |
+
|
| 49 |
+
import pandas as pd
|
| 50 |
+
from transformers import AutoTokenizer
|
| 51 |
+
df = pd.read_parquet(base["data"])
|
| 52 |
+
tok = AutoTokenizer.from_pretrained(base["model"])
|
| 53 |
+
|
| 54 |
+
rows, cut, kept = [], 0, 0
|
| 55 |
+
for line in open(f"{src}/rollouts.jsonl"):
|
| 56 |
+
d = json.loads(line)
|
| 57 |
+
meta = json.loads(df.iloc[d["idx"]]["extra_info"]["metadata"])
|
| 58 |
+
assert meta["environment"] == d["env"], (d["idx"], meta["environment"], d["env"])
|
| 59 |
+
recs = []
|
| 60 |
+
for i, ids in enumerate(d["ids"]):
|
| 61 |
+
if len(ids) <= BUDGET:
|
| 62 |
+
recs.append(dict(d["recs"][i])); kept += 1
|
| 63 |
+
continue
|
| 64 |
+
txt = tok.decode(ids[:BUDGET], skip_special_tokens=False)
|
| 65 |
+
v = score_one(meta, txt) or {"reward": 0.0, "accuracy": 0, "format_score": 0}
|
| 66 |
+
r = dict(v); r["len"] = BUDGET; r["finish"] = "length"
|
| 67 |
+
r["closed_think"] = "</think>" in txt
|
| 68 |
+
recs.append(r); cut += 1
|
| 69 |
+
rows.append({"idx": d["idx"], "env": d["env"], "gt": d["gt"], "recs": recs})
|
| 70 |
+
|
| 71 |
+
flat = [x for r in rows for x in r["recs"]]
|
| 72 |
+
n, N = len(flat), base["n"]
|
| 73 |
+
per_env = {}
|
| 74 |
+
for r in rows:
|
| 75 |
+
per_env.setdefault(r["env"], []).extend(x["accuracy"] for x in r["recs"])
|
| 76 |
+
passk = {}
|
| 77 |
+
if N > 1:
|
| 78 |
+
for k in (1, 2, 4, 8, 16):
|
| 79 |
+
if k > N: continue
|
| 80 |
+
passk[f"pass@{k}"] = sum(
|
| 81 |
+
pass_at_k(N, sum(x["accuracy"] for x in r["recs"]), k) for r in rows) / len(rows)
|
| 82 |
+
passk["solved_any"] = sum(1 for r in rows if any(x["accuracy"] for x in r["recs"]))
|
| 83 |
+
passk["solved_all"] = sum(1 for r in rows if all(x["accuracy"] for x in r["recs"]))
|
| 84 |
+
|
| 85 |
+
s = dict(base)
|
| 86 |
+
s.update({"tag": DST, "max_new": BUDGET, "ctx": BUDGET + 2048, "problems": len(rows),
|
| 87 |
+
"accuracy@1": sum(x["accuracy"] for x in flat) / n,
|
| 88 |
+
"reward_mean@1": sum(x["reward"] for x in flat) / n,
|
| 89 |
+
"format_ok_rate": sum(x["format_score"] for x in flat) / n,
|
| 90 |
+
"truncated_rate": sum(x["finish"] == "length" for x in flat) / n,
|
| 91 |
+
"closed_think_rate": sum(x["closed_think"] for x in flat) / n,
|
| 92 |
+
"resp_len_mean": sum(x["len"] for x in flat) / n,
|
| 93 |
+
"resp_len_max": max(x["len"] for x in flat),
|
| 94 |
+
"reward_hist": dict(Counter(round(x["reward"], 3) for x in flat)),
|
| 95 |
+
"n_envs": len(per_env), "retruncated_from": SRC,
|
| 96 |
+
"n_rescored": cut, "n_unchanged": kept, **passk})
|
| 97 |
+
os.makedirs(dst, exist_ok=True)
|
| 98 |
+
json.dump(s, open(f"{dst}/summary.json", "w"), indent=2, ensure_ascii=False)
|
| 99 |
+
json.dump({k: sum(v)/len(v) for k, v in sorted(per_env.items())},
|
| 100 |
+
open(f"{dst}/per_env.json", "w"), indent=2)
|
| 101 |
+
with open(f"{dst}/rollouts.jsonl", "w") as f:
|
| 102 |
+
for r in rows: f.write(json.dumps(r, ensure_ascii=False) + "\n")
|
| 103 |
+
print(" %-20s 重判 %5d / 保留 %5d avg@8 %.4f pass@8 %.4f 解出 %3d fmt %.3f trunc %.3f"
|
| 104 |
+
% (DST, cut, kept, s["accuracy@1"], s.get("pass@8", float("nan")),
|
| 105 |
+
s.get("solved_any", -1), s["format_ok_rate"], s["truncated_rate"]))
|
eval/rlve_eval.py
ADDED
|
@@ -0,0 +1,212 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""mean@1 evaluation of a checkpoint on data/rlve_test.parquet.
|
| 2 |
+
|
| 3 |
+
Separate from training on purpose. The reward logged during ROSE-online scores
|
| 4 |
+
the *spliced* sequence (student prefix + teacher continuation), which never
|
| 5 |
+
reaches an answer -- see recipe/rose/rlve_reward.py. This scores the student.
|
| 6 |
+
|
| 7 |
+
Context is 16384 tokens end to end: max_tokens is computed per prompt as
|
| 8 |
+
CTX - len(prompt_ids) - 8, so nothing gets a larger window than the flag names.
|
| 9 |
+
|
| 10 |
+
Scores come from `env.verifier()` directly rather than recipe/rose/rlve_reward's
|
| 11 |
+
compute_score, which throws away two of the three fields. verifier returns
|
| 12 |
+
reward / accuracy / format_score, and `accuracy` is the metric that survives
|
| 13 |
+
environments disagreeing on reward scale -- they do. The bundle docstring claims
|
| 14 |
+
"-0.5 = answered and wrong"; BannedPointSupersetPathCounting actually uses
|
| 15 |
+
{correct +1.0, wrong_answer 0.0, wrong_range -0.5, wrong_format -1.0}.
|
| 16 |
+
|
| 17 |
+
Everything runs under `if __name__ == "__main__"` because vLLM spawns its engine
|
| 18 |
+
core with the spawn start method, which re-imports this module in the child.
|
| 19 |
+
Module-level work here means the child tries to build a second LLM during import
|
| 20 |
+
and dies in `_check_not_importing_main`.
|
| 21 |
+
|
| 22 |
+
MODEL=<hf dir> TAG=<name> [TP=4] [N=1] [CTX=16384] python3 rlve_eval.py
|
| 23 |
+
"""
|
| 24 |
+
|
| 25 |
+
import json
|
| 26 |
+
import os
|
| 27 |
+
import sys
|
| 28 |
+
import time
|
| 29 |
+
from collections import Counter
|
| 30 |
+
|
| 31 |
+
GYM = "/home/haojinw2/rose-bundle-new/RLVE-Eval"
|
| 32 |
+
sys.path.insert(0, GYM)
|
| 33 |
+
|
| 34 |
+
import Gym.environments as G # noqa: E402
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def score_one(meta, text):
|
| 38 |
+
cls = getattr(G, f"{meta['environment']}_Environment", None)
|
| 39 |
+
if cls is None:
|
| 40 |
+
return None
|
| 41 |
+
env = cls()
|
| 42 |
+
env.set_config(meta.get("config", {}))
|
| 43 |
+
try:
|
| 44 |
+
return env.verifier(text)
|
| 45 |
+
except Exception as e: # noqa: BLE001
|
| 46 |
+
return {"reward": -1.0, "accuracy": 0, "format_score": 0, "err": repr(e)}
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def main():
|
| 50 |
+
CTX = int(os.environ.get("CTX", 16384))
|
| 51 |
+
TP = int(os.environ.get("TP", 4))
|
| 52 |
+
N = int(os.environ.get("N", 1))
|
| 53 |
+
MODEL = os.environ["MODEL"]
|
| 54 |
+
TAG = os.environ.get("TAG", os.path.basename(MODEL.rstrip("/")))
|
| 55 |
+
DATA = os.environ.get("DATA", "/home/haojinw2/rose-bundle-new/data/rlve_test.parquet")
|
| 56 |
+
OUT = os.environ.get("OUT", f"/home/haojinw2/evalout/rlve_{TAG}")
|
| 57 |
+
MAX_NEW = int(os.environ.get("MAX_NEW", 0)) # 0 = use whatever CTX leaves
|
| 58 |
+
os.makedirs(OUT, exist_ok=True)
|
| 59 |
+
|
| 60 |
+
import pandas as pd
|
| 61 |
+
from transformers import AutoTokenizer
|
| 62 |
+
from vllm import LLM, SamplingParams
|
| 63 |
+
|
| 64 |
+
df = pd.read_parquet(DATA)
|
| 65 |
+
tok = AutoTokenizer.from_pretrained(MODEL)
|
| 66 |
+
|
| 67 |
+
prompts, keep = [], []
|
| 68 |
+
# Data-parallel slicing. Two halves on 0-3 and 4-7 beat TP=8 on a 1.7B --
|
| 69 |
+
# no collectives, twice the KV cache. `idx` stays the true dataframe row,
|
| 70 |
+
# so merge_eval.py can concatenate the shards without renumbering.
|
| 71 |
+
IDX_START = int(os.environ.get("IDX_START", 0))
|
| 72 |
+
IDX_END = int(os.environ.get("IDX_END", 0)) or len(df)
|
| 73 |
+
for i in range(IDX_START, IDX_END):
|
| 74 |
+
row = df.iloc[i]
|
| 75 |
+
msgs = [dict(m) for m in row["prompt"]]
|
| 76 |
+
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True)
|
| 77 |
+
budget = CTX - len(ids) - 8
|
| 78 |
+
if MAX_NEW:
|
| 79 |
+
# fixed generation budget (train-set sampling), not "whatever the
|
| 80 |
+
# context leaves over" -- CTX still caps it so nothing overflows
|
| 81 |
+
budget = min(budget, MAX_NEW)
|
| 82 |
+
if budget < 256:
|
| 83 |
+
# no room to answer is a data problem, not a model failure -- drop it
|
| 84 |
+
# loudly instead of scoring it as a format failure
|
| 85 |
+
print(f"!! row {i}: prompt {len(ids)} tok leaves {budget}, skipped")
|
| 86 |
+
continue
|
| 87 |
+
prompts.append(tok.decode(ids))
|
| 88 |
+
keep.append((i, budget))
|
| 89 |
+
|
| 90 |
+
print(f"{len(prompts)}/{len(df)} 题, ctx={CTX}, n={N}, tp={TP}", flush=True)
|
| 91 |
+
|
| 92 |
+
llm = LLM(
|
| 93 |
+
model=MODEL,
|
| 94 |
+
tensor_parallel_size=TP,
|
| 95 |
+
max_model_len=CTX,
|
| 96 |
+
gpu_memory_utilization=float(os.environ.get("GPU_UTIL", 0.85)),
|
| 97 |
+
trust_remote_code=True,
|
| 98 |
+
seed=int(os.environ.get("EVAL_SEED", "0")),
|
| 99 |
+
)
|
| 100 |
+
|
| 101 |
+
# one SamplingParams per prompt so each gets its own exact budget
|
| 102 |
+
# 0.7 / 0.9, no top-k -- the project standard, matching the reference eval
|
| 103 |
+
# script's argparse defaults. Training rollout uses the same values so the
|
| 104 |
+
# states ROSE distils on are the states the model actually visits here.
|
| 105 |
+
TEMP = float(os.environ.get("TEMP", 0.7))
|
| 106 |
+
TOP_P = float(os.environ.get("TOP_P", 0.9))
|
| 107 |
+
TOP_K = int(os.environ.get("TOP_K", -1))
|
| 108 |
+
print(f"sampling: temp={TEMP} top_p={TOP_P} top_k={TOP_K}", flush=True)
|
| 109 |
+
sps = [
|
| 110 |
+
SamplingParams(n=N, temperature=TEMP, top_p=TOP_P, top_k=TOP_K,
|
| 111 |
+
max_tokens=b, seed=int(os.environ.get("EVAL_SEED", "0")))
|
| 112 |
+
for _, b in keep
|
| 113 |
+
]
|
| 114 |
+
|
| 115 |
+
t0 = time.time()
|
| 116 |
+
outs = llm.generate(prompts, sps)
|
| 117 |
+
print(f"生成 {time.time() - t0:.0f}s", flush=True)
|
| 118 |
+
|
| 119 |
+
rows = []
|
| 120 |
+
per_env = {}
|
| 121 |
+
for (idx, _), o in zip(keep, outs):
|
| 122 |
+
row = df.iloc[idx]
|
| 123 |
+
meta = json.loads(row["extra_info"]["metadata"])
|
| 124 |
+
recs = []
|
| 125 |
+
for c in o.outputs:
|
| 126 |
+
v = score_one(meta, c.text) or {"reward": 0.0, "accuracy": 0, "format_score": 0}
|
| 127 |
+
v["len"] = len(c.token_ids)
|
| 128 |
+
v["finish"] = c.finish_reason
|
| 129 |
+
v["closed_think"] = "</think>" in c.text
|
| 130 |
+
recs.append(v)
|
| 131 |
+
rows.append(
|
| 132 |
+
{
|
| 133 |
+
"idx": int(idx),
|
| 134 |
+
"env": meta["environment"],
|
| 135 |
+
"gt": row["reward_model"]["ground_truth"],
|
| 136 |
+
"recs": recs,
|
| 137 |
+
"text": [c.text for c in o.outputs],
|
| 138 |
+
# exact ids, so a downstream teacher-continuation step can join
|
| 139 |
+
# id-to-id instead of re-tokenising the decoded text (which can
|
| 140 |
+
# shift the seam -- the one silent failure in this design)
|
| 141 |
+
"ids": [list(c.token_ids) for c in o.outputs],
|
| 142 |
+
}
|
| 143 |
+
)
|
| 144 |
+
per_env.setdefault(meta["environment"], []).append(
|
| 145 |
+
1.0 if any(r["accuracy"] for r in recs) else 0.0
|
| 146 |
+
)
|
| 147 |
+
|
| 148 |
+
flat = [r for row in rows for r in row["recs"]]
|
| 149 |
+
n = len(flat)
|
| 150 |
+
|
| 151 |
+
# Unbiased pass@k (Chen et al. 2021): with c of N samples correct,
|
| 152 |
+
# pass@k = 1 - C(N-c, k) / C(N, k)
|
| 153 |
+
# Reporting the whole curve costs nothing once the samples exist, and
|
| 154 |
+
# pass@1 from it is the same quantity as accuracy@1 -- a useful self-check.
|
| 155 |
+
from math import comb
|
| 156 |
+
def pass_at_k(N, c, k):
|
| 157 |
+
if N - c < k:
|
| 158 |
+
return 1.0
|
| 159 |
+
return 1.0 - comb(N - c, k) / comb(N, k)
|
| 160 |
+
|
| 161 |
+
passk = {}
|
| 162 |
+
if N > 1:
|
| 163 |
+
for k in [1, 2, 4, 8, 16]:
|
| 164 |
+
if k > N:
|
| 165 |
+
continue
|
| 166 |
+
passk[f"pass@{k}"] = sum(
|
| 167 |
+
pass_at_k(N, sum(r["accuracy"] for r in row["recs"]), k) for row in rows
|
| 168 |
+
) / len(rows)
|
| 169 |
+
# how many problems are solved by at least one sample, ever
|
| 170 |
+
passk["solved_any"] = sum(
|
| 171 |
+
1 for row in rows if any(r["accuracy"] for r in row["recs"])
|
| 172 |
+
)
|
| 173 |
+
passk["solved_all"] = sum(
|
| 174 |
+
1 for row in rows if all(r["accuracy"] for r in row["recs"])
|
| 175 |
+
)
|
| 176 |
+
|
| 177 |
+
summary = {
|
| 178 |
+
"tag": TAG,
|
| 179 |
+
"model": MODEL,
|
| 180 |
+
"ctx": CTX,
|
| 181 |
+
"max_new": MAX_NEW or None,
|
| 182 |
+
"data": DATA,
|
| 183 |
+
"n": N,
|
| 184 |
+
"temperature": TEMP,
|
| 185 |
+
"top_p": TOP_P,
|
| 186 |
+
"top_k": TOP_K,
|
| 187 |
+
"problems": len(rows),
|
| 188 |
+
"accuracy@1": sum(r["accuracy"] for r in flat) / n,
|
| 189 |
+
"reward_mean@1": sum(r["reward"] for r in flat) / n,
|
| 190 |
+
"format_ok_rate": sum(r["format_score"] for r in flat) / n,
|
| 191 |
+
"truncated_rate": sum(r["finish"] == "length" for r in flat) / n,
|
| 192 |
+
"closed_think_rate": sum(r["closed_think"] for r in flat) / n,
|
| 193 |
+
"resp_len_mean": sum(r["len"] for r in flat) / n,
|
| 194 |
+
"resp_len_max": max(r["len"] for r in flat),
|
| 195 |
+
"reward_hist": dict(Counter(round(r["reward"], 3) for r in flat)),
|
| 196 |
+
"n_envs": len(per_env),
|
| 197 |
+
**passk,
|
| 198 |
+
}
|
| 199 |
+
print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True)
|
| 200 |
+
|
| 201 |
+
with open(f"{OUT}/summary.json", "w") as f:
|
| 202 |
+
json.dump(summary, f, indent=2, ensure_ascii=False)
|
| 203 |
+
with open(f"{OUT}/per_env.json", "w") as f:
|
| 204 |
+
json.dump({k: sum(v) / len(v) for k, v in sorted(per_env.items())}, f, indent=2)
|
| 205 |
+
with open(f"{OUT}/rollouts.jsonl", "w") as f:
|
| 206 |
+
for r in rows:
|
| 207 |
+
f.write(json.dumps(r, ensure_ascii=False) + "\n")
|
| 208 |
+
print(f"-> {OUT}", flush=True)
|
| 209 |
+
|
| 210 |
+
|
| 211 |
+
if __name__ == "__main__":
|
| 212 |
+
main()
|
frameworks/agentgym_rl/.gitignore
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
**/*.pt
|
| 2 |
+
**/checkpoints
|
| 3 |
+
**/wget-log
|
| 4 |
+
**/_build/
|
| 5 |
+
**/*.ckpt
|
| 6 |
+
**/outputs
|
| 7 |
+
**/*.tar.gz
|
| 8 |
+
**/playground
|
| 9 |
+
**/wandb
|
| 10 |
+
|
| 11 |
+
# Byte-compiled / optimized / DLL files
|
| 12 |
+
__pycache__/
|
| 13 |
+
*.py[cod]
|
| 14 |
+
*$py.class
|
| 15 |
+
dataset/*
|
| 16 |
+
tensorflow/my_graph/*
|
| 17 |
+
.idea/
|
| 18 |
+
# C extensions
|
| 19 |
+
*.so
|
| 20 |
+
|
| 21 |
+
# Distribution / packaging
|
| 22 |
+
.Python
|
| 23 |
+
env/
|
| 24 |
+
build/
|
| 25 |
+
develop-eggs/
|
| 26 |
+
dist/
|
| 27 |
+
downloads/
|
| 28 |
+
eggs/
|
| 29 |
+
.eggs/
|
| 30 |
+
lib/
|
| 31 |
+
lib64/
|
| 32 |
+
parts/
|
| 33 |
+
sdist/
|
| 34 |
+
var/
|
| 35 |
+
*.egg-info/
|
| 36 |
+
.installed.cfg
|
| 37 |
+
*.egg
|
| 38 |
+
|
| 39 |
+
# PyInstaller
|
| 40 |
+
# Usually these files are written by a python script from a template
|
| 41 |
+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
| 42 |
+
*.manifest
|
| 43 |
+
*.spec
|
| 44 |
+
|
| 45 |
+
# Installer logs
|
| 46 |
+
pip-log.txt
|
| 47 |
+
pip-delete-this-directory.txt
|
| 48 |
+
|
| 49 |
+
# Unit test / coverage reports
|
| 50 |
+
htmlcov/
|
| 51 |
+
.tox/
|
| 52 |
+
.coverage
|
| 53 |
+
.coverage.*
|
| 54 |
+
.cache
|
| 55 |
+
nosetests.xml
|
| 56 |
+
coverage.xml
|
| 57 |
+
*,cover
|
| 58 |
+
.hypothesis/
|
| 59 |
+
|
| 60 |
+
# Translations
|
| 61 |
+
*.mo
|
| 62 |
+
*.pot
|
| 63 |
+
|
| 64 |
+
# Django stuff:
|
| 65 |
+
*.log
|
| 66 |
+
local_settings.py
|
| 67 |
+
|
| 68 |
+
# Flask stuff:
|
| 69 |
+
instance/
|
| 70 |
+
.webassets-cache
|
| 71 |
+
|
| 72 |
+
# Scrapy stuff:
|
| 73 |
+
.scrapy
|
| 74 |
+
|
| 75 |
+
# Sphinx documentation
|
| 76 |
+
docs/_build/
|
| 77 |
+
|
| 78 |
+
# PyBuilder
|
| 79 |
+
target/
|
| 80 |
+
|
| 81 |
+
# IPython Notebook
|
| 82 |
+
.ipynb_checkpoints
|
| 83 |
+
|
| 84 |
+
# pyenv
|
| 85 |
+
.python-version
|
| 86 |
+
|
| 87 |
+
# celery beat schedule file
|
| 88 |
+
celerybeat-schedule
|
| 89 |
+
|
| 90 |
+
# dotenv
|
| 91 |
+
.env
|
| 92 |
+
|
| 93 |
+
# virtualenv
|
| 94 |
+
venv/
|
| 95 |
+
ENV/
|
| 96 |
+
|
| 97 |
+
# Spyder project settings
|
| 98 |
+
.spyderproject
|
| 99 |
+
|
| 100 |
+
# Rope project settings
|
| 101 |
+
.ropeproject
|
| 102 |
+
|
| 103 |
+
# vscode
|
| 104 |
+
.vscode
|
| 105 |
+
|
| 106 |
+
# Mac
|
| 107 |
+
.DS_Store
|
| 108 |
+
|
| 109 |
+
# output logs
|
| 110 |
+
tests/e2e/toy_examples/deepspeed/synchronous/output.txt
|
| 111 |
+
|
| 112 |
+
# vim
|
| 113 |
+
*.swp
|
| 114 |
+
|
| 115 |
+
# ckpt
|
| 116 |
+
*.lock
|
frameworks/agentgym_rl/.readthedocs.yaml
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Read the Docs configuration file
|
| 2 |
+
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
|
| 3 |
+
|
| 4 |
+
version: 2
|
| 5 |
+
|
| 6 |
+
build:
|
| 7 |
+
os: ubuntu-22.04
|
| 8 |
+
tools:
|
| 9 |
+
python: "3.8"
|
| 10 |
+
|
| 11 |
+
sphinx:
|
| 12 |
+
configuration: docs/conf.py
|
| 13 |
+
|
| 14 |
+
python:
|
| 15 |
+
install:
|
| 16 |
+
- requirements: docs/requirements-docs.txt
|
frameworks/agentgym_rl/.style.yapf
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[style]
|
| 2 |
+
based_on_style = google
|
| 3 |
+
column_limit = 120
|
| 4 |
+
indent_width = 4
|
| 5 |
+
split_arguments_when_comma_terminated: true
|
frameworks/agentgym_rl/Notice.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
Copyright 2023-2024 Bytedance Ltd. and/or its affiliates
|
frameworks/agentgym_rl/VERL_LICENSE
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
Apache License
|
| 3 |
+
Version 2.0, January 2004
|
| 4 |
+
http://www.apache.org/licenses/
|
| 5 |
+
|
| 6 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 7 |
+
|
| 8 |
+
1. Definitions.
|
| 9 |
+
|
| 10 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 11 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 12 |
+
|
| 13 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 14 |
+
the copyright owner that is granting the License.
|
| 15 |
+
|
| 16 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 17 |
+
other entities that control, are controlled by, or are under common
|
| 18 |
+
control with that entity. For the purposes of this definition,
|
| 19 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 20 |
+
direction or management of such entity, whether by contract or
|
| 21 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 22 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 23 |
+
|
| 24 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 25 |
+
exercising permissions granted by this License.
|
| 26 |
+
|
| 27 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 28 |
+
including but not limited to software source code, documentation
|
| 29 |
+
source, and configuration files.
|
| 30 |
+
|
| 31 |
+
"Object" form shall mean any form resulting from mechanical
|
| 32 |
+
transformation or translation of a Source form, including but
|
| 33 |
+
not limited to compiled object code, generated documentation,
|
| 34 |
+
and conversions to other media types.
|
| 35 |
+
|
| 36 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 37 |
+
Object form, made available under the License, as indicated by a
|
| 38 |
+
copyright notice that is included in or attached to the work
|
| 39 |
+
(an example is provided in the Appendix below).
|
| 40 |
+
|
| 41 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 42 |
+
form, that is based on (or derived from) the Work and for which the
|
| 43 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 44 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 45 |
+
of this License, Derivative Works shall not include works that remain
|
| 46 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 47 |
+
the Work and Derivative Works thereof.
|
| 48 |
+
|
| 49 |
+
"Contribution" shall mean any work of authorship, including
|
| 50 |
+
the original version of the Work and any modifications or additions
|
| 51 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 52 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 53 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 54 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 55 |
+
means any form of electronic, verbal, or written communication sent
|
| 56 |
+
to the Licensor or its representatives, including but not limited to
|
| 57 |
+
communication on electronic mailing lists, source code control systems,
|
| 58 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 59 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 60 |
+
excluding communication that is conspicuously marked or otherwise
|
| 61 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 62 |
+
|
| 63 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 64 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 65 |
+
subsequently incorporated within the Work.
|
| 66 |
+
|
| 67 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 68 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 69 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 70 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 71 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 72 |
+
Work and such Derivative Works in Source or Object form.
|
| 73 |
+
|
| 74 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 75 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 76 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 77 |
+
(except as stated in this section) patent license to make, have made,
|
| 78 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 79 |
+
where such license applies only to those patent claims licensable
|
| 80 |
+
by such Contributor that are necessarily infringed by their
|
| 81 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 82 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 83 |
+
institute patent litigation against any entity (including a
|
| 84 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 85 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 86 |
+
or contributory patent infringement, then any patent licenses
|
| 87 |
+
granted to You under this License for that Work shall terminate
|
| 88 |
+
as of the date such litigation is filed.
|
| 89 |
+
|
| 90 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 91 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 92 |
+
modifications, and in Source or Object form, provided that You
|
| 93 |
+
meet the following conditions:
|
| 94 |
+
|
| 95 |
+
(a) You must give any other recipients of the Work or
|
| 96 |
+
Derivative Works a copy of this License; and
|
| 97 |
+
|
| 98 |
+
(b) You must cause any modified files to carry prominent notices
|
| 99 |
+
stating that You changed the files; and
|
| 100 |
+
|
| 101 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 102 |
+
that You distribute, all copyright, patent, trademark, and
|
| 103 |
+
attribution notices from the Source form of the Work,
|
| 104 |
+
excluding those notices that do not pertain to any part of
|
| 105 |
+
the Derivative Works; and
|
| 106 |
+
|
| 107 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 108 |
+
distribution, then any Derivative Works that You distribute must
|
| 109 |
+
include a readable copy of the attribution notices contained
|
| 110 |
+
within such NOTICE file, excluding those notices that do not
|
| 111 |
+
pertain to any part of the Derivative Works, in at least one
|
| 112 |
+
of the following places: within a NOTICE text file distributed
|
| 113 |
+
as part of the Derivative Works; within the Source form or
|
| 114 |
+
documentation, if provided along with the Derivative Works; or,
|
| 115 |
+
within a display generated by the Derivative Works, if and
|
| 116 |
+
wherever such third-party notices normally appear. The contents
|
| 117 |
+
of the NOTICE file are for informational purposes only and
|
| 118 |
+
do not modify the License. You may add Your own attribution
|
| 119 |
+
notices within Derivative Works that You distribute, alongside
|
| 120 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 121 |
+
that such additional attribution notices cannot be construed
|
| 122 |
+
as modifying the License.
|
| 123 |
+
|
| 124 |
+
You may add Your own copyright statement to Your modifications and
|
| 125 |
+
may provide additional or different license terms and conditions
|
| 126 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 127 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 128 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 129 |
+
the conditions stated in this License.
|
| 130 |
+
|
| 131 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 132 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 133 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 134 |
+
this License, without any additional terms or conditions.
|
| 135 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 136 |
+
the terms of any separate license agreement you may have executed
|
| 137 |
+
with Licensor regarding such Contributions.
|
| 138 |
+
|
| 139 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 140 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 141 |
+
except as required for reasonable and customary use in describing the
|
| 142 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 143 |
+
|
| 144 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 145 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 146 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 147 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 148 |
+
implied, including, without limitation, any warranties or conditions
|
| 149 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 150 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 151 |
+
appropriateness of using or redistributing the Work and assume any
|
| 152 |
+
risks associated with Your exercise of permissions under this License.
|
| 153 |
+
|
| 154 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 155 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 156 |
+
unless required by applicable law (such as deliberate and grossly
|
| 157 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 158 |
+
liable to You for damages, including any direct, indirect, special,
|
| 159 |
+
incidental, or consequential damages of any character arising as a
|
| 160 |
+
result of this License or out of the use or inability to use the
|
| 161 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 162 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 163 |
+
other commercial damages or losses), even if such Contributor
|
| 164 |
+
has been advised of the possibility of such damages.
|
| 165 |
+
|
| 166 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 167 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 168 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 169 |
+
or other liability obligations and/or rights consistent with this
|
| 170 |
+
License. However, in accepting such obligations, You may act only
|
| 171 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 172 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 173 |
+
defend, and hold each Contributor harmless for any liability
|
| 174 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 175 |
+
of your accepting any such warranty or additional liability.
|
| 176 |
+
|
| 177 |
+
END OF TERMS AND CONDITIONS
|
| 178 |
+
|
| 179 |
+
APPENDIX: How to apply the Apache License to your work.
|
| 180 |
+
|
| 181 |
+
To apply the Apache License to your work, attach the following
|
| 182 |
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
| 183 |
+
replaced with your own identifying information. (Don't include
|
| 184 |
+
the brackets!) The text should be enclosed in the appropriate
|
| 185 |
+
comment syntax for the file format. We also recommend that a
|
| 186 |
+
file or class name and description of purpose be included on the
|
| 187 |
+
same "printed page" as the copyright notice for easier
|
| 188 |
+
identification within third-party archives.
|
| 189 |
+
|
| 190 |
+
Copyright [yyyy] [name of copyright owner]
|
| 191 |
+
|
| 192 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 193 |
+
you may not use this file except in compliance with the License.
|
| 194 |
+
You may obtain a copy of the License at
|
| 195 |
+
|
| 196 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 197 |
+
|
| 198 |
+
Unless required by applicable law or agreed to in writing, software
|
| 199 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 200 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 201 |
+
See the License for the specific language governing permissions and
|
| 202 |
+
limitations under the License.
|
frameworks/agentgym_rl/VERL_README.md
ADDED
|
@@ -0,0 +1,135 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<h1 style="text-align: center;">verl: Volcano Engine Reinforcement Learning for LLM</h1>
|
| 2 |
+
|
| 3 |
+
verl is a flexible, efficient and production-ready RL training library for large language models (LLMs).
|
| 4 |
+
|
| 5 |
+
verl is the open-source version of **[HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2)** paper.
|
| 6 |
+
|
| 7 |
+
verl is flexible and easy to use with:
|
| 8 |
+
|
| 9 |
+
- **Easy extension of diverse RL algorithms**: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code.
|
| 10 |
+
|
| 11 |
+
- **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks.
|
| 12 |
+
|
| 13 |
+
- **Flexible device mapping**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.
|
| 14 |
+
|
| 15 |
+
- Readily integration with popular HuggingFace models
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
verl is fast with:
|
| 19 |
+
|
| 20 |
+
- **State-of-the-art throughput**: By seamlessly integrating existing SOTA LLM training and inference frameworks, verl achieves high generation and training throughput.
|
| 21 |
+
|
| 22 |
+
- **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.
|
| 23 |
+
|
| 24 |
+
<p align="center">
|
| 25 |
+
| <a href="https://verl.readthedocs.io/en/latest/index.html"><b>Documentation</b></a> | <a href="https://arxiv.org/abs/2409.19256v2"><b>Paper</b></a> | <a href="https://join.slack.com/t/verlgroup/shared_invite/zt-2w5p9o4c3-yy0x2Q56s_VlGLsJ93A6vA"><b>Slack</b></a> | <a href="https://raw.githubusercontent.com/eric-haibin-lin/verl-community/refs/heads/main/WeChat.JPG"><b>Wechat</b></a> | <a href="https://x.com/verl_project"><b>Twitter</b></a>
|
| 26 |
+
|
| 27 |
+
<!-- <a href=""><b>Slides</b></a> | -->
|
| 28 |
+
</p>
|
| 29 |
+
|
| 30 |
+
## News
|
| 31 |
+
|
| 32 |
+
- [2025/2] We will present verl in the [Bytedance/NVIDIA/Anyscale Ray Meetup](https://lu.ma/ji7atxux) in bay area on Feb 13th. Come join us in person!
|
| 33 |
+
- [2025/1] [Doubao-1.5-pro](https://team.doubao.com/zh/special/doubao_1_5_pro) is released with SOTA-level performance on LLM & VLM. The RL scaling preview model is trained using verl, reaching OpenAI O1-level performance on math benchmarks (70.0 pass@1 on AIME).
|
| 34 |
+
- [2024/12] The team presented <a href="https://neurips.cc/Expo/Conferences/2024/workshop/100677">Post-training LLMs: From Algorithms to Infrastructure</a> at NeurIPS 2024. [Slides](https://github.com/eric-haibin-lin/verl-data/tree/neurips) and [video](https://neurips.cc/Expo/Conferences/2024/workshop/100677) available.
|
| 35 |
+
- [2024/10] verl is presented at Ray Summit. [Youtube video](https://www.youtube.com/watch?v=MrhMcXkXvJU&list=PLzTswPQNepXntmT8jr9WaNfqQ60QwW7-U&index=37) available.
|
| 36 |
+
- [2024/08] HybridFlow (verl) is accepted to EuroSys 2025.
|
| 37 |
+
|
| 38 |
+
## Key Features
|
| 39 |
+
|
| 40 |
+
- **FSDP** and **Megatron-LM** for training.
|
| 41 |
+
- **vLLM** and **TGI** for rollout generation, **SGLang** support coming soon.
|
| 42 |
+
- huggingface models support
|
| 43 |
+
- Supervised fine-tuning
|
| 44 |
+
- Reinforcement learning from human feedback with [PPO](https://github.com/volcengine/verl/tree/main/examples/ppo_trainer), [GRPO](https://github.com/volcengine/verl/tree/main/examples/grpo_trainer), and [ReMax](https://github.com/volcengine/verl/tree/main/examples/remax_trainer)
|
| 45 |
+
- Support model-based reward and function-based reward (verifiable reward)
|
| 46 |
+
- flash-attention, [sequence packing](examples/ppo_trainer/run_qwen2-7b_seq_balance.sh), [long context](examples/ppo_trainer/run_deepseek7b_llm_sp2.sh) support via DeepSpeed Ulysses, [LoRA](examples/sft/gsm8k/run_qwen_05_peft.sh), [Liger-kernel](examples/sft/gsm8k/run_qwen_05_sp2_liger.sh)
|
| 47 |
+
- scales up to 70B models and hundreds of GPUs
|
| 48 |
+
- experiment tracking with wandb, swanlab and mlflow
|
| 49 |
+
|
| 50 |
+
## Upcoming Features
|
| 51 |
+
- Reward model training
|
| 52 |
+
- DPO training
|
| 53 |
+
- DeepSeek integration with Megatron backend
|
| 54 |
+
- SGLang integration
|
| 55 |
+
|
| 56 |
+
## Getting Started
|
| 57 |
+
|
| 58 |
+
Checkout this [Jupyter Notebook](https://github.com/volcengine/verl/tree/main/examples/ppo_trainer/verl_getting_started.ipynb) to get started with PPO training with a single 24GB L4 GPU (**FREE** GPU quota provided by [Lighting Studio](https://lightning.ai/hlin-verl/studios/verl-getting-started))!
|
| 59 |
+
|
| 60 |
+
**Quickstart:**
|
| 61 |
+
- [Installation](https://verl.readthedocs.io/en/latest/start/install.html)
|
| 62 |
+
- [Quickstart](https://verl.readthedocs.io/en/latest/start/quickstart.html)
|
| 63 |
+
- [Programming Guide](https://verl.readthedocs.io/en/latest/hybrid_flow.html)
|
| 64 |
+
|
| 65 |
+
**Running a PPO example step-by-step:**
|
| 66 |
+
- Data and Reward Preparation
|
| 67 |
+
- [Prepare Data for Post-Training](https://verl.readthedocs.io/en/latest/preparation/prepare_data.html)
|
| 68 |
+
- [Implement Reward Function for Dataset](https://verl.readthedocs.io/en/latest/preparation/reward_function.html)
|
| 69 |
+
- Understanding the PPO Example
|
| 70 |
+
- [PPO Example Architecture](https://verl.readthedocs.io/en/latest/examples/ppo_code_architecture.html)
|
| 71 |
+
- [Config Explanation](https://verl.readthedocs.io/en/latest/examples/config.html)
|
| 72 |
+
- [Run GSM8K Example](https://verl.readthedocs.io/en/latest/examples/gsm8k_example.html)
|
| 73 |
+
|
| 74 |
+
**Reproducible algorithm baselines:**
|
| 75 |
+
- [PPO and GRPO](https://verl.readthedocs.io/en/latest/experiment/ppo.html)
|
| 76 |
+
|
| 77 |
+
**For code explanation and advance usage (extension):**
|
| 78 |
+
- PPO Trainer and Workers
|
| 79 |
+
- [PPO Ray Trainer](https://verl.readthedocs.io/en/latest/workers/ray_trainer.html)
|
| 80 |
+
- [PyTorch FSDP Backend](https://verl.readthedocs.io/en/latest/workers/fsdp_workers.html)
|
| 81 |
+
- [Megatron-LM Backend](https://verl.readthedocs.io/en/latest/index.html)
|
| 82 |
+
- Advance Usage and Extension
|
| 83 |
+
- [Ray API design tutorial](https://verl.readthedocs.io/en/latest/advance/placement.html)
|
| 84 |
+
- [Extend to Other RL(HF) algorithms](https://verl.readthedocs.io/en/latest/advance/dpo_extension.html)
|
| 85 |
+
- [Add Models with the FSDP Backend](https://verl.readthedocs.io/en/latest/advance/fsdp_extension.html)
|
| 86 |
+
- [Add Models with the Megatron-LM Backend](https://verl.readthedocs.io/en/latest/advance/megatron_extension.html)
|
| 87 |
+
- [Deployment using Separate GPU Resources](https://github.com/volcengine/verl/tree/main/examples/split_placement)
|
| 88 |
+
|
| 89 |
+
## Performance Tuning Guide
|
| 90 |
+
The performance is essential for on-policy RL algorithm. We write a detailed performance tuning guide to allow people tune the performance. See [here](https://verl.readthedocs.io/en/latest/perf/perf_tuning.html) for more details.
|
| 91 |
+
|
| 92 |
+
## vLLM v0.7 testing version
|
| 93 |
+
We have released a testing version of veRL that supports vLLM>=0.7.0. Please refer to [this document](https://github.com/volcengine/verl/docs/README_vllm0.7.md) for installation guide and more information.
|
| 94 |
+
|
| 95 |
+
## Contribution Guide
|
| 96 |
+
Contributions from the community are welcome!
|
| 97 |
+
|
| 98 |
+
### Code formatting
|
| 99 |
+
We use yapf (Google style) to enforce strict code formatting when reviewing PRs. To reformat you code locally, make sure you installed **latest** `yapf`
|
| 100 |
+
```bash
|
| 101 |
+
pip3 install yapf --upgrade
|
| 102 |
+
```
|
| 103 |
+
Then, make sure you are at top level of verl repo and run
|
| 104 |
+
```bash
|
| 105 |
+
bash scripts/format.sh
|
| 106 |
+
```
|
| 107 |
+
|
| 108 |
+
## Citation and acknowledgement
|
| 109 |
+
|
| 110 |
+
If you find the project helpful, please cite:
|
| 111 |
+
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2)
|
| 112 |
+
- [A Framework for Training Large Language Models for Code Generation via Proximal Policy Optimization](https://i.cs.hku.hk/~cwu/papers/gmsheng-NL2Code24.pdf)
|
| 113 |
+
|
| 114 |
+
```tex
|
| 115 |
+
@article{sheng2024hybridflow,
|
| 116 |
+
title = {HybridFlow: A Flexible and Efficient RLHF Framework},
|
| 117 |
+
author = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
|
| 118 |
+
year = {2024},
|
| 119 |
+
journal = {arXiv preprint arXiv: 2409.19256}
|
| 120 |
+
}
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
verl is inspired by the design of Nemo-Aligner, Deepspeed-chat and OpenRLHF. The project is adopted and supported by Anyscale, Bytedance, LMSys.org, Shanghai AI Lab, Tsinghua University, UC Berkeley, UCLA, UIUC, and University of Hong Kong.
|
| 124 |
+
|
| 125 |
+
## Awesome work using verl
|
| 126 |
+
- [Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization](https://arxiv.org/abs/2410.09302)
|
| 127 |
+
- [Flaming-hot Initiation with Regular Execution Sampling for Large Language Models](https://arxiv.org/abs/2410.21236)
|
| 128 |
+
- [Process Reinforcement Through Implicit Rewards](https://github.com/PRIME-RL/PRIME/)
|
| 129 |
+
- [TinyZero](https://github.com/Jiayi-Pan/TinyZero): a reproduction of DeepSeek R1 Zero recipe for reasoning tasks
|
| 130 |
+
- [RAGEN](https://github.com/ZihanWang314/ragen): a general-purpose reasoning agent training framework
|
| 131 |
+
- [Logic R1](https://github.com/Unakar/Logic-RL): a reproduced DeepSeek R1 Zero on 2K Tiny Logic Puzzle Dataset.
|
| 132 |
+
- [deepscaler](https://github.com/agentica-project/deepscaler): iterative context scaling with GRPO
|
| 133 |
+
- [critic-rl](https://github.com/HKUNLP/critic-rl): Teaching Language Models to Critique via Reinforcement Learning
|
| 134 |
+
|
| 135 |
+
We are HIRING! Send us an [email](mailto:haibin.lin@bytedance.com) if you are interested in internship/FTE opportunities in MLSys/LLM reasoning/multimodal alignment.
|
frameworks/agentgym_rl/docs/Makefile
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Minimal makefile for Sphinx documentation
|
| 2 |
+
#
|
| 3 |
+
|
| 4 |
+
# You can set these variables from the command line.
|
| 5 |
+
SPHINXOPTS =
|
| 6 |
+
SPHINXBUILD = sphinx-build
|
| 7 |
+
SPHINXPROJ = verl
|
| 8 |
+
SOURCEDIR = .
|
| 9 |
+
BUILDDIR = _build
|
| 10 |
+
|
| 11 |
+
# Put it first so that "make" without argument is like "make help".
|
| 12 |
+
help:
|
| 13 |
+
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
| 14 |
+
|
| 15 |
+
.PHONY: help Makefile
|
| 16 |
+
|
| 17 |
+
# Catch-all target: route all unknown targets to Sphinx using the new
|
| 18 |
+
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
|
| 19 |
+
%: Makefile
|
| 20 |
+
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
frameworks/agentgym_rl/docs/README.md
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# verl documents
|
| 2 |
+
|
| 3 |
+
## Build the docs
|
| 4 |
+
|
| 5 |
+
```bash
|
| 6 |
+
# Install dependencies.
|
| 7 |
+
pip install -r requirements-docs.txt
|
| 8 |
+
|
| 9 |
+
# Build the docs.
|
| 10 |
+
make clean
|
| 11 |
+
make html
|
| 12 |
+
```
|
| 13 |
+
|
| 14 |
+
## Open the docs with your browser
|
| 15 |
+
|
| 16 |
+
```bash
|
| 17 |
+
python -m http.server -d _build/html/
|
| 18 |
+
```
|
| 19 |
+
Launch your browser and open localhost:8000.
|
frameworks/agentgym_rl/docs/README_vllm0.7.md
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Readme for verl(vllm>=0.7) version
|
| 2 |
+
|
| 3 |
+
## Installation
|
| 4 |
+
|
| 5 |
+
Note: This version of veRL supports **FSDP** for training and **vLLM** for rollout. (Megatron-LM is not supported yet.)
|
| 6 |
+
|
| 7 |
+
```
|
| 8 |
+
# Create the conda environment
|
| 9 |
+
conda create -n verl python==3.10
|
| 10 |
+
conda activate verl
|
| 11 |
+
|
| 12 |
+
# Install verl
|
| 13 |
+
git clone https://github.com/volcengine/verl.git
|
| 14 |
+
cd verl
|
| 15 |
+
pip3 install -e .
|
| 16 |
+
|
| 17 |
+
# Install vLLM>=0.7
|
| 18 |
+
# (Option1) pip3 install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
|
| 19 |
+
# (Option2) pip3 install "vllm>=0.7.0"
|
| 20 |
+
|
| 21 |
+
# Install flash-attn
|
| 22 |
+
pip3 install flash-attn --no-build-isolation
|
| 23 |
+
|
| 24 |
+
```
|
| 25 |
+
|
| 26 |
+
Note that if you are installing stable versions of vLLM (Option2), you need to make some tiny patches manually on vllm (/path/to/site-packages/vllm after installation) after the above steps:
|
| 27 |
+
|
| 28 |
+
- vllm/distributed/parallel_state.py: Remove the assertion below:
|
| 29 |
+
|
| 30 |
+
```
|
| 31 |
+
if (world_size
|
| 32 |
+
!= tensor_model_parallel_size * pipeline_model_parallel_size):
|
| 33 |
+
raise RuntimeError(
|
| 34 |
+
f"world_size ({world_size}) is not equal to "
|
| 35 |
+
f"tensor_model_parallel_size ({tensor_model_parallel_size}) x "
|
| 36 |
+
f"pipeline_model_parallel_size ({pipeline_model_parallel_size})")
|
| 37 |
+
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
- vllm/executor/uniproc_executor.py: change `local_rank = rank` to `local_rank = int(os.environ["LOCAL_RANK"])`
|
| 41 |
+
- vllm/model_executor/model_loader/weight_utils.py: remove the `torch.cuda.empty_cache()` in `pt_weights_iterator`
|
| 42 |
+
|
| 43 |
+
These modifications have already been merged into the main branch of vLLM. Thus nightly vLLM or building vLLM from source do not need these patches.
|
| 44 |
+
|
| 45 |
+
## Features
|
| 46 |
+
|
| 47 |
+
### Use cuda graph
|
| 48 |
+
|
| 49 |
+
After installation, examples using FSDP as training backends can be used. By default, the `enforce_eager` is set to True, which disables the cuda graph. To enjoy cuda graphs and the sleep mode of vLLM>=0.7, add the following lines to the bash script:
|
| 50 |
+
|
| 51 |
+
```
|
| 52 |
+
actor_rollout_ref.rollout.enforce_eager=False \
|
| 53 |
+
actor_rollout_ref.rollout.free_cache_engine=False \
|
| 54 |
+
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
For a typical job like examples/ppo_trainer/run_qwen2-7b_seq_balance.sh, the rollout generation time is 115 seconds with vLLM0.6.3, while it is 85 seconds with vLLM0.7.0. By enabling the cudagraph, the generation duration is further reduced to 62 seconds.
|
| 58 |
+
|
| 59 |
+
**Note:** Currently, if the `n` is greater than 1 in `SamplingParams` in vLLM>=0.7, there is a potential performance issue on the stability of rollout generation time (Some iterations would see generation time bursts). We are working with the vLLM team to check this issue.
|
| 60 |
+
|
| 61 |
+
### Other features in vLLM
|
| 62 |
+
|
| 63 |
+
1. **num_scheduler_step>1:** not supported yet (weight loading has not been aligned with `MultiStepModelRunner`)
|
| 64 |
+
2. **Prefix caching:** not supported yet (vLLM sleep mode does not support prefix caching)
|
| 65 |
+
3. **Chunked prefill:** supported
|
frameworks/agentgym_rl/docs/_static/logo.png
ADDED
|
frameworks/agentgym_rl/docs/advance/dpo_extension.rst
ADDED
|
@@ -0,0 +1,271 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Extend to other RL(HF) algorithms
|
| 2 |
+
=================================
|
| 3 |
+
|
| 4 |
+
We already implemented the complete training pipeline of the PPO
|
| 5 |
+
algorithms. To extend to other algorithms, we analyze the high-level
|
| 6 |
+
principle to use verl and provide a tutorial to implement the DPO
|
| 7 |
+
algorithm. Users can follow the similar paradigm to extend to other RL algorithms.
|
| 8 |
+
|
| 9 |
+
.. note:: **Key ideas**: Single process drives multi-process computation and data communication.
|
| 10 |
+
|
| 11 |
+
Overall Approach
|
| 12 |
+
----------------
|
| 13 |
+
|
| 14 |
+
Step 1: Consider what multi-machine multi-GPU computations are needed
|
| 15 |
+
for each model, such as ``generate_sequence`` , ``compute_log_prob`` and
|
| 16 |
+
``update_policy`` in the actor_rollout model. Implement distributed
|
| 17 |
+
single-process-multiple-data (SPMD) computation and encapsulate them
|
| 18 |
+
into APIs
|
| 19 |
+
|
| 20 |
+
Step 2: Based on different distributed scenarios, including FSDP and 3D
|
| 21 |
+
parallelism in Megatron-LM, implement single-process control of data
|
| 22 |
+
interaction among multi-process computations.
|
| 23 |
+
|
| 24 |
+
Step 3: Utilize the encapsulated APIs to implement the control flow
|
| 25 |
+
|
| 26 |
+
Example: Online DPO
|
| 27 |
+
-------------------
|
| 28 |
+
|
| 29 |
+
We use verl to implement a simple online DPO algorithm. The algorithm
|
| 30 |
+
flow of Online DPO is as follows:
|
| 31 |
+
|
| 32 |
+
1. There is a prompt (rollout) generator which has the same weight as
|
| 33 |
+
the actor model. After a batch of prompts are fed into the generator,
|
| 34 |
+
it generates N responses for each prompt.
|
| 35 |
+
2. Send all the prompts + responses to a verifier for scoring, which can
|
| 36 |
+
be reward model or a rule-based function. Then sort them in pairs to
|
| 37 |
+
form a training batch.
|
| 38 |
+
3. Use this training batch to train the actor model using DPO. During
|
| 39 |
+
the process, a reference policy is needed.
|
| 40 |
+
|
| 41 |
+
Step 1: What are the multi-machine multi-GPU computations
|
| 42 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 43 |
+
|
| 44 |
+
**Sample Generator**
|
| 45 |
+
|
| 46 |
+
Implementation details:
|
| 47 |
+
|
| 48 |
+
.. code:: python
|
| 49 |
+
|
| 50 |
+
from verl.single_controller.base import Worker
|
| 51 |
+
from verl.single_controller.ray import RayWorkerGroup, RayClassWithInitArgs, RayResourcePool
|
| 52 |
+
import ray
|
| 53 |
+
|
| 54 |
+
@ray.remote
|
| 55 |
+
class SampleGenerator(Worker):
|
| 56 |
+
def __init__(self, config):
|
| 57 |
+
super().__init__()
|
| 58 |
+
self.config = config
|
| 59 |
+
|
| 60 |
+
def generate_sequences(self, data):
|
| 61 |
+
pass
|
| 62 |
+
|
| 63 |
+
Here, ``SampleGenerator`` can be viewed as a multi-process pulled up by
|
| 64 |
+
``torchrun``, with each process running the same code (SPMD).
|
| 65 |
+
``SampleGenerator`` needs to implement a ``generate_sequences`` API for
|
| 66 |
+
the control flow to call. The implementation details inside can use any
|
| 67 |
+
inference engine including vllm, sglang and huggingface. Users can
|
| 68 |
+
largely reuse the code in
|
| 69 |
+
verl/verl/trainer/ppo/rollout/vllm_rollout/vllm_rollout.py and we won't
|
| 70 |
+
go into details here.
|
| 71 |
+
|
| 72 |
+
**ReferencePolicy inference**
|
| 73 |
+
|
| 74 |
+
API: compute reference log probability
|
| 75 |
+
|
| 76 |
+
.. code:: python
|
| 77 |
+
|
| 78 |
+
from verl.single_controller.base import Worker
|
| 79 |
+
import ray
|
| 80 |
+
|
| 81 |
+
@ray.remote
|
| 82 |
+
class ReferencePolicy(Worker):
|
| 83 |
+
def __init__(self):
|
| 84 |
+
super().__init__()
|
| 85 |
+
self.model = Model()
|
| 86 |
+
|
| 87 |
+
def infer(self, data):
|
| 88 |
+
return self.model(data)
|
| 89 |
+
|
| 90 |
+
**Actor update**
|
| 91 |
+
|
| 92 |
+
API: Update actor model parameters
|
| 93 |
+
|
| 94 |
+
.. code:: python
|
| 95 |
+
|
| 96 |
+
from verl.single_controller.base import Worker
|
| 97 |
+
import ray
|
| 98 |
+
|
| 99 |
+
@ray.remote
|
| 100 |
+
class DPOActor(Worker):
|
| 101 |
+
def __init__(self):
|
| 102 |
+
super().__init__()
|
| 103 |
+
self.model = Model()
|
| 104 |
+
self.model = FSDP(self.model) # or other distributed strategy
|
| 105 |
+
self.optimizer = optim.Adam(self.model.parameters(), lr=1e-3)
|
| 106 |
+
self.loss_fn = xxx
|
| 107 |
+
|
| 108 |
+
def update(self, data):
|
| 109 |
+
self.optimizer.zero_grad()
|
| 110 |
+
logits = self.model(data)
|
| 111 |
+
loss = self.loss_fn(logits)
|
| 112 |
+
loss.backward()
|
| 113 |
+
self.optimizer.step()
|
| 114 |
+
|
| 115 |
+
**Notes: How to distinguish between control processes and distributed computation processes**
|
| 116 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 117 |
+
|
| 118 |
+
- Control processes are generally functions directly decorated with
|
| 119 |
+
``@ray.remote``
|
| 120 |
+
- Computation processes are all wrapped into a ``RayWorkerGroup``.
|
| 121 |
+
|
| 122 |
+
Users can reuse most of the distribtued computation logics implemented
|
| 123 |
+
in PPO algorithm, including FSDP and Megatron-LM backend in
|
| 124 |
+
verl/verl/trainer/ppo.
|
| 125 |
+
|
| 126 |
+
Step 2: Based on different distributed scenarios, implement single-process control of multi-process data interaction
|
| 127 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 128 |
+
|
| 129 |
+
**The core problem to solve here is how a single process sends data to
|
| 130 |
+
multiple processes, drives multi-process computation, and how the
|
| 131 |
+
control process obtains the results of multi-process computation.**
|
| 132 |
+
First, we initialize the multi-process ``WorkerGroup`` in the control
|
| 133 |
+
process.
|
| 134 |
+
|
| 135 |
+
.. code:: python
|
| 136 |
+
|
| 137 |
+
@ray.remote(num_cpus=1)
|
| 138 |
+
def main_task(config):
|
| 139 |
+
# construct SampleGenerator
|
| 140 |
+
resource_pool = RayResourcePool(process_on_nodes=[8] * 2) # 16 GPUs
|
| 141 |
+
ray_cls = RayClassWithInitArgs(SampleGenerator, config=config)
|
| 142 |
+
# put SampleGenerator onto resource pool
|
| 143 |
+
worker_group = RayWorkerGroup(resource_pool, ray_cls)
|
| 144 |
+
|
| 145 |
+
# construct reference policy
|
| 146 |
+
|
| 147 |
+
As we can see, in the control process, multiple processes are wrapped
|
| 148 |
+
into a ``RayWorkerGroup``. Inside this ``WorkerGroup``, there is a
|
| 149 |
+
``self._workers`` member, where each worker is a RayActor
|
| 150 |
+
(https://docs.ray.io/en/latest/ray-core/actors.html) of SampleGenerator.
|
| 151 |
+
ray_trainer.md also provide an implementation of
|
| 152 |
+
``MegatronRayWorkerGroup``.
|
| 153 |
+
|
| 154 |
+
Assuming the model is distributed using FSDP, and there is a batch of
|
| 155 |
+
data on the control process, for data parallelism, the underlying
|
| 156 |
+
calling process is:
|
| 157 |
+
|
| 158 |
+
.. code:: python
|
| 159 |
+
|
| 160 |
+
data = xxx
|
| 161 |
+
data_list = data.chunk(dp_size)
|
| 162 |
+
|
| 163 |
+
output = []
|
| 164 |
+
for d in data_list:
|
| 165 |
+
# worker_group._workers[i] is a SampleGenerator
|
| 166 |
+
output.append(worker_group._workers[i].generate_sequences.remote(d))
|
| 167 |
+
|
| 168 |
+
output = ray.get(output)
|
| 169 |
+
output = torch.cat(output)
|
| 170 |
+
|
| 171 |
+
Single process calling multiple processes involves the following 3
|
| 172 |
+
steps:
|
| 173 |
+
|
| 174 |
+
1. Split the data into DP parts on the control process.
|
| 175 |
+
2. Send the data to remote, call the remote computation through RPC, and
|
| 176 |
+
utilize multi-process computation.
|
| 177 |
+
3. Obtain the computation results of each worker on the control process
|
| 178 |
+
and merge them.
|
| 179 |
+
|
| 180 |
+
Frequently calling these 3 steps on the controller process greatly hurts
|
| 181 |
+
code readability. **In verl, we have abstracted and encapsulated these 3
|
| 182 |
+
steps, so that the worker's method + dispatch + collect can be
|
| 183 |
+
registered into the worker_group**
|
| 184 |
+
|
| 185 |
+
.. code:: python
|
| 186 |
+
|
| 187 |
+
from verl.single_controller.base.decorator import register
|
| 188 |
+
|
| 189 |
+
def dispatch_data(worker_group, data):
|
| 190 |
+
return data.chunk(worker_group.world_size)
|
| 191 |
+
|
| 192 |
+
def collect_data(worker_group, data):
|
| 193 |
+
return torch.cat(data)
|
| 194 |
+
|
| 195 |
+
dispatch_mode = {
|
| 196 |
+
'dispatch_fn': dispatch_data,
|
| 197 |
+
'collect_fn': collect_data
|
| 198 |
+
}
|
| 199 |
+
|
| 200 |
+
@register(dispatch_mode=dispatch_mode)
|
| 201 |
+
def generate_sequences(self, data):
|
| 202 |
+
pass
|
| 203 |
+
|
| 204 |
+
In this way, we can directly call the method inside the worker through
|
| 205 |
+
the ``worker_group`` on the control (driver) process (which is a single
|
| 206 |
+
process):
|
| 207 |
+
|
| 208 |
+
.. code:: python
|
| 209 |
+
|
| 210 |
+
output = worker_group.generate_sequences(data)
|
| 211 |
+
|
| 212 |
+
This single line includes data splitting, data distribution and
|
| 213 |
+
computation, and data collection.
|
| 214 |
+
|
| 215 |
+
Furthermore, the model parallelism size of each model is usually fixed,
|
| 216 |
+
including dp, tp, pp. So for these common distributed scenarios, we have
|
| 217 |
+
pre-implemented specific dispatch and collect methods,in `decorator.py <https://github.com/volcengine/verl/blob/main/verl/single_controller/base/decorator.py>`_, which can be directly used to wrap the computations.
|
| 218 |
+
|
| 219 |
+
.. code:: python
|
| 220 |
+
|
| 221 |
+
from verl.single_controller.base.decorator import register, Dispatch
|
| 222 |
+
|
| 223 |
+
@register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
|
| 224 |
+
def generate_sequences(self, data: DataProto) -> DataProto:
|
| 225 |
+
pass
|
| 226 |
+
|
| 227 |
+
Here it requires the data interface to be ``DataProto``. Definition of
|
| 228 |
+
``DataProto`` is in `protocol.py <https://github.com/volcengine/verl/blob/main/verl/protocol.py>`_.
|
| 229 |
+
|
| 230 |
+
Step 3: Main training loop
|
| 231 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 232 |
+
|
| 233 |
+
With the above training flows, we can implement the algorithm's control
|
| 234 |
+
flow. It is recommended that ``main_task`` is also a ray remote process.
|
| 235 |
+
|
| 236 |
+
.. code:: python
|
| 237 |
+
|
| 238 |
+
@ray.remote(num_cpus=1)
|
| 239 |
+
def main_task(config):
|
| 240 |
+
# construct SampleGenerator
|
| 241 |
+
resource_pool = RayResourcePool(process_on_nodes=[8] * 2) # 16 GPUs
|
| 242 |
+
ray_cls = RayClassWithInitArgs(SampleGenerator, config=config)
|
| 243 |
+
# put SampleGenerator onto resource pool
|
| 244 |
+
sample_gen = RayWorkerGroup(resource_pool, ray_cls)
|
| 245 |
+
|
| 246 |
+
# construct reference policy
|
| 247 |
+
ray_cls = RayClassWithInitArgs(ReferencePolicy)
|
| 248 |
+
ref_policy = RayWorkerGroup(resource_pool, ray_cls)
|
| 249 |
+
|
| 250 |
+
# construct actor
|
| 251 |
+
ray_cls = RayClassWithInitArgs(DPOActor)
|
| 252 |
+
dpo_policy = RayWorkerGroup(resource_pool, ray_cls)
|
| 253 |
+
|
| 254 |
+
dataloader = DataLoader()
|
| 255 |
+
|
| 256 |
+
for data in dataloader:
|
| 257 |
+
# generate data
|
| 258 |
+
data = sample_gen.generate_sequences(data)
|
| 259 |
+
# generate scores for each data
|
| 260 |
+
data = generate_scores(data)
|
| 261 |
+
# generate pairwise data using scores
|
| 262 |
+
data = generate_pairwise_data(data)
|
| 263 |
+
# generate ref_log_prob
|
| 264 |
+
data.batch['ref_log_prob'] = ref_policy.infer(data)
|
| 265 |
+
# update using dpo
|
| 266 |
+
dpo_policy.update(data)
|
| 267 |
+
# logging
|
| 268 |
+
|
| 269 |
+
Here, different ``WorkerGroups`` can be placed in the same resource pool or
|
| 270 |
+
in different resource pools using ``create_colocated_worker_cls``
|
| 271 |
+
similar as in `ray_trainer.py <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py>`_.
|
frameworks/agentgym_rl/docs/advance/fsdp_extension.rst
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
Add models with the FSDP backend
|
| 3 |
+
==================================
|
| 4 |
+
|
| 5 |
+
Model
|
| 6 |
+
--------------------------
|
| 7 |
+
|
| 8 |
+
In principle, our FSDP backend can support any HF model and we can
|
| 9 |
+
sychronoize the actor model weight with vLLM using `hf_weight_loader.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_6_3/hf_weight_loader.py>`_.
|
| 10 |
+
However, ``hf_weight_loader`` is will gather the full state_dict of a
|
| 11 |
+
model during synchronization, which may cause OOM. We suggest using
|
| 12 |
+
``dtensor_weight_loader`` which gather the full model parameter layer by
|
| 13 |
+
layer to reduce the peak memory usage. We already support dtensor weight
|
| 14 |
+
loader for the models below in `dtensor_weight_loader.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_5_4/dtensor_weight_loaders.py>`_.:
|
| 15 |
+
|
| 16 |
+
- ``GPT2LMHeadModel``
|
| 17 |
+
- ``LlamaForCausalLM``
|
| 18 |
+
- ``LLaMAForCausalLM``
|
| 19 |
+
- ``MistralForCausalLM``
|
| 20 |
+
- ``InternLMForCausalLM``
|
| 21 |
+
- ``AquilaModel``
|
| 22 |
+
- ``AquilaForCausalLM``
|
| 23 |
+
- ``Phi3ForCausalLM``
|
| 24 |
+
- ``GemmaForCausalLM``
|
| 25 |
+
- ``Gemma2ForCausalLM``
|
| 26 |
+
- ``GPTBigCodeForCausalLM``
|
| 27 |
+
- ``Starcoder2ForCausalLM``
|
| 28 |
+
- ``Qwen2ForCausalLM``
|
| 29 |
+
- ``DeepseekV2ForCausalLM``
|
| 30 |
+
|
| 31 |
+
To implement ``dtensor_weight_loader`` of a model that's supported in
|
| 32 |
+
vLLM, follow the guide of gemma model below:
|
| 33 |
+
|
| 34 |
+
1. Copy the
|
| 35 |
+
``load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]])`` from the vllm model class
|
| 36 |
+
to ``dtensor_weight_loaders.py``
|
| 37 |
+
2. Modify the arguments to
|
| 38 |
+
``(actor_weights: Dict, vllm_model: nn.Module)``
|
| 39 |
+
3. Replace the ``self`` to ``vllm_model``
|
| 40 |
+
4. Add the
|
| 41 |
+
``local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)``
|
| 42 |
+
before each ``param = params_dict[name]`` and modify the following
|
| 43 |
+
weight loading using ``local_loaded_weight``.
|
| 44 |
+
5. Register the implemented dtensor weight loader to ``__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__``.
|
| 45 |
+
|
| 46 |
+
.. code-block:: diff
|
| 47 |
+
|
| 48 |
+
- def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
|
| 49 |
+
+ def gemma_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module:
|
| 50 |
+
stacked_params_mapping = [
|
| 51 |
+
# (param_name, shard_name, shard_id)
|
| 52 |
+
("qkv_proj", "q_proj", "q"),
|
| 53 |
+
("qkv_proj", "k_proj", "k"),
|
| 54 |
+
("qkv_proj", "v_proj", "v"),
|
| 55 |
+
("gate_up_proj", "gate_proj", 0),
|
| 56 |
+
("gate_up_proj", "up_proj", 1),
|
| 57 |
+
]
|
| 58 |
+
- params_dict = dict(self.named_parameters())
|
| 59 |
+
+ params_dict = dict(vllm_model.named_parameters())
|
| 60 |
+
loaded_params = set()
|
| 61 |
+
- for name, loaded_weight in weights:
|
| 62 |
+
+ for name, loaded_weight in actor_weights.items():
|
| 63 |
+
for (param_name, shard_name, shard_id) in stacked_params_mapping:
|
| 64 |
+
if shard_name not in name:
|
| 65 |
+
continue
|
| 66 |
+
name = name.replace(shard_name, param_name)
|
| 67 |
+
# Skip loading extra bias for GPTQ models.
|
| 68 |
+
if name.endswith(".bias") and name not in params_dict:
|
| 69 |
+
continue
|
| 70 |
+
+ local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)
|
| 71 |
+
param = params_dict[name]
|
| 72 |
+
weight_loader = param.weight_loader
|
| 73 |
+
- weight_loader(param, loaded_weight, shard_id)
|
| 74 |
+
+ weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id)
|
| 75 |
+
break
|
| 76 |
+
else:
|
| 77 |
+
# lm_head is not used in vllm as it is tied with embed_token.
|
| 78 |
+
# To prevent errors, skip loading lm_head.weight.
|
| 79 |
+
if "lm_head.weight" in name:
|
| 80 |
+
continue
|
| 81 |
+
# Skip loading extra bias for GPTQ models.
|
| 82 |
+
if name.endswith(".bias") and name not in params_dict:
|
| 83 |
+
continue
|
| 84 |
+
+ local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)
|
| 85 |
+
param = params_dict[name]
|
| 86 |
+
weight_loader = getattr(param, "weight_loader",
|
| 87 |
+
default_weight_loader)
|
| 88 |
+
- weight_loader(param, loaded_weight)
|
| 89 |
+
+ weight_loader(param, local_loaded_weight.to(dtype=param.dtype))
|
| 90 |
+
loaded_params.add(name)
|
| 91 |
+
unloaded_params = params_dict.keys() - loaded_params
|
| 92 |
+
if unloaded_params:
|
| 93 |
+
raise RuntimeError(
|
| 94 |
+
"Some weights are not initialized from checkpoints: "
|
| 95 |
+
f"{unloaded_params}")
|
frameworks/agentgym_rl/docs/advance/megatron_extension.rst
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Add models with the Megatron-LM backend
|
| 2 |
+
=========================================
|
| 3 |
+
|
| 4 |
+
Model
|
| 5 |
+
-----------
|
| 6 |
+
|
| 7 |
+
The most challenging aspect to use the Megatron-LM backend is implementing
|
| 8 |
+
the models for training. Currently, we implement Llama model that
|
| 9 |
+
support data parallelism, tensor parallelism, pipeline parallelism (also
|
| 10 |
+
vPP) and sequence parallelism. We also implement remove padding (sequence packing) on Llama
|
| 11 |
+
model, which can be found in `modeling_llama_megatron.py <https://github.com/volcengine/verl/blob/main/verl/models/llama/megatron/modeling_llama_megatron.py>`_.
|
| 12 |
+
|
| 13 |
+
To support other model, users are required to implement:
|
| 14 |
+
|
| 15 |
+
1. Implemnt a model similar to ``modeling_llama_megatron.py`` that satisfy the
|
| 16 |
+
parallelism requirements of Megatron-LM. Then register your model in
|
| 17 |
+
the `registry.py <https://github.com/volcengine/verl/blob/main/verl/models/registry.py>`_.
|
| 18 |
+
2. Checkpoint utils that can load full checkpoint (e.g. huggingface
|
| 19 |
+
checkpoint) to partitioned models during the runtime. Then register
|
| 20 |
+
your loader to ``weight_loader_registry`` in `weight_loader_registry.py <https://github.com/volcengine/verl/blob/main/verl/models/weight_loader_registry.py>`_.
|
| 21 |
+
3. Weight loader that synchronize the weight from Megatron to rollout
|
| 22 |
+
(vLLM) model. Note that both the actor model and rollout model are
|
| 23 |
+
partitioned during runtime. So, it's advisable to map the model name
|
| 24 |
+
in actor model implementation. Otherwise, you may need an additional
|
| 25 |
+
name mapping and even weight transformation. The weight loader implementation
|
| 26 |
+
is in `megatron_weight_loaders.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_6_3/megatron_weight_loaders.py>`_.
|
frameworks/agentgym_rl/docs/advance/placement.rst
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Ray API Design Tutorial
|
| 2 |
+
=======================================
|
| 3 |
+
|
| 4 |
+
We provide a tutorial for our Ray API design, including:
|
| 5 |
+
|
| 6 |
+
- Ray basic concepts
|
| 7 |
+
- Resource Pool and RayWorkerGroup
|
| 8 |
+
- Data Dispatch, Execution and Collection
|
| 9 |
+
- Initialize the RayWorkerGroup and execute the distributed computation in the given Resource Pool
|
| 10 |
+
|
| 11 |
+
See details in `tutorial.ipynb <https://github.com/volcengine/verl/blob/main/examples/ray/tutorial.ipynb>`_.
|
frameworks/agentgym_rl/docs/conf.py
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2024 Bytedance Ltd. and/or its affiliates
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
# Configuration file for the Sphinx documentation builder.
|
| 16 |
+
#
|
| 17 |
+
# This file only contains a selection of the most common options. For a full
|
| 18 |
+
# list see the documentation:
|
| 19 |
+
# https://www.sphinx-doc.org/en/master/usage/configuration.html
|
| 20 |
+
|
| 21 |
+
# -- Path setup --------------------------------------------------------------
|
| 22 |
+
|
| 23 |
+
# If extensions (or modules to document with autodoc) are in another directory,
|
| 24 |
+
# add these directories to sys.path here. If the directory is relative to the
|
| 25 |
+
# documentation root, use os.path.abspath to make it absolute, like shown here.
|
| 26 |
+
#
|
| 27 |
+
# import os
|
| 28 |
+
# import sys
|
| 29 |
+
# sys.path.insert(0, os.path.abspath('.'))
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
# -- Project information -----------------------------------------------------
|
| 33 |
+
|
| 34 |
+
project = u'verl'
|
| 35 |
+
# pylint: disable=W0622
|
| 36 |
+
copyright = u'2024 ByteDance Seed Foundation MLSys Team'
|
| 37 |
+
author = u'Guangming Sheng, Chi Zhang, Yanghua Peng, Haibin Lin'
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
# -- General configuration ---------------------------------------------------
|
| 41 |
+
# The master toctree document.
|
| 42 |
+
master_doc = 'index'
|
| 43 |
+
|
| 44 |
+
# Add any Sphinx extension module names here, as strings. They can be
|
| 45 |
+
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
| 46 |
+
# ones.
|
| 47 |
+
extensions = ['recommonmark',
|
| 48 |
+
'sphinx.ext.autosectionlabel',
|
| 49 |
+
]
|
| 50 |
+
|
| 51 |
+
# The suffix(es) of source filenames.
|
| 52 |
+
# You can specify multiple suffix as a list of string:
|
| 53 |
+
source_suffix = ['.rst', 'rest', '.md']
|
| 54 |
+
|
| 55 |
+
# Add any paths that contain templates here, relative to this directory.
|
| 56 |
+
templates_path = ['_templates']
|
| 57 |
+
|
| 58 |
+
# The language for content autogenerated by Sphinx. Refer to documentation
|
| 59 |
+
# for a list of supported languages.
|
| 60 |
+
#
|
| 61 |
+
# This is also used if you do content translation via gettext catalogs.
|
| 62 |
+
# Usually you set "language" from the command line for these cases.
|
| 63 |
+
language = u'en'
|
| 64 |
+
|
| 65 |
+
# List of patterns, relative to source directory, that match files and
|
| 66 |
+
# directories to ignore when looking for source files.
|
| 67 |
+
# This pattern also affects html_static_path and html_extra_path.
|
| 68 |
+
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
# -- Options for HTML output -------------------------------------------------
|
| 72 |
+
|
| 73 |
+
# The theme to use for HTML and HTML Help pages. See the documentation for
|
| 74 |
+
# a list of builtin themes.
|
| 75 |
+
#
|
| 76 |
+
html_theme = 'sphinx_rtd_theme'
|
| 77 |
+
|
| 78 |
+
# Add any paths that contain custom static files (such as style sheets) here,
|
| 79 |
+
# relative to this directory. They are copied after the builtin static files,
|
| 80 |
+
# so a file named "default.css" will overwrite the builtin "default.css".
|
| 81 |
+
html_static_path = ['_static']
|
frameworks/agentgym_rl/docs/examples/config.rst
ADDED
|
@@ -0,0 +1,365 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.. _config-explain-page:
|
| 2 |
+
|
| 3 |
+
Config Explanation
|
| 4 |
+
===================
|
| 5 |
+
|
| 6 |
+
ppo_trainer.yaml for FSDP Backend
|
| 7 |
+
---------------------------------
|
| 8 |
+
|
| 9 |
+
Data
|
| 10 |
+
~~~~
|
| 11 |
+
|
| 12 |
+
.. code:: yaml
|
| 13 |
+
|
| 14 |
+
data:
|
| 15 |
+
tokenizer: null
|
| 16 |
+
train_files: ~/data/rlhf/gsm8k/train.parquet
|
| 17 |
+
val_files: ~/data/rlhf/gsm8k/test.parquet
|
| 18 |
+
prompt_key: prompt
|
| 19 |
+
max_prompt_length: 512
|
| 20 |
+
max_response_length: 512
|
| 21 |
+
train_batch_size: 1024
|
| 22 |
+
val_batch_size: 1312
|
| 23 |
+
return_raw_input_ids: False # This should be set to true when the tokenizer between policy and rm differs
|
| 24 |
+
return_raw_chat: False
|
| 25 |
+
|
| 26 |
+
- ``data.train_files``: Training set parquet. Can be a list or a single
|
| 27 |
+
file. The program will read all files into memory, so it can't be too
|
| 28 |
+
large (< 100GB). The path can be either local path or HDFS path. For
|
| 29 |
+
HDFS path, we provide utils to download it to DRAM and convert the
|
| 30 |
+
HDFS path to local path.
|
| 31 |
+
- ``data.val_files``: Validation parquet. Can be a list or a single
|
| 32 |
+
file.
|
| 33 |
+
- ``data.prompt_key``: The field in the dataset where the prompt is
|
| 34 |
+
located. Default is 'prompt'.
|
| 35 |
+
- ``data.max_prompt_length``: Maximum prompt length. All prompts will be
|
| 36 |
+
left-padded to this length. An error will be reported if the length is
|
| 37 |
+
too long
|
| 38 |
+
- ``data.max_response_length``: Maximum response length. Rollout in RL
|
| 39 |
+
algorithms (e.g. PPO) generates up to this length
|
| 40 |
+
- ``data.train_batch_size``: Batch size sampled for one training
|
| 41 |
+
iteration of different RL algorithms.
|
| 42 |
+
- ``data.val_batch_size``: Batch size sampled for one validation
|
| 43 |
+
iteration.
|
| 44 |
+
- ``data.return_raw_input_ids``: Whether to return the original
|
| 45 |
+
input_ids without adding chat template. This is mainly used to
|
| 46 |
+
accommodate situations where the reward model's chat template differs
|
| 47 |
+
from the policy. It needs to be decoded first, then apply the RM's
|
| 48 |
+
chat template. If using a model-based RM, and the policy and RM
|
| 49 |
+
chat_templates are different, this flag needs to be set
|
| 50 |
+
- ``data.return_raw_chat``:
|
| 51 |
+
- ``data.truncation``: Truncate the input_ids or prompt length if they
|
| 52 |
+
exceed max_prompt_length. Default is 'error', not allow exceed the
|
| 53 |
+
max_prompt_length. The users should increase the max_prompt_length if
|
| 54 |
+
throwing the error.
|
| 55 |
+
|
| 56 |
+
Actor/Rollout/Reference Policy
|
| 57 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 58 |
+
|
| 59 |
+
.. code:: yaml
|
| 60 |
+
|
| 61 |
+
actor_rollout_ref:
|
| 62 |
+
hybrid_engine: True
|
| 63 |
+
model:
|
| 64 |
+
path: ~/models/deepseek-llm-7b-chat
|
| 65 |
+
external_lib: null
|
| 66 |
+
override_config: { }
|
| 67 |
+
enable_gradient_checkpointing: False
|
| 68 |
+
use_remove_padding: False
|
| 69 |
+
actor:
|
| 70 |
+
strategy: fsdp # This is for backward-compatibility
|
| 71 |
+
ppo_mini_batch_size: 256
|
| 72 |
+
ppo_micro_batch_size: null # will be deprecated, use ppo_micro_batch_size_per_gpu
|
| 73 |
+
ppo_micro_batch_size_per_gpu: 8
|
| 74 |
+
use_dynamic_bsz: False
|
| 75 |
+
ppo_max_token_len_per_gpu: 16384 # n * ${data.max_prompt_length} + ${data.max_response_length}
|
| 76 |
+
grad_clip: 1.0
|
| 77 |
+
clip_ratio: 0.2
|
| 78 |
+
entropy_coeff: 0.001
|
| 79 |
+
use_kl_loss: False # True for GRPO
|
| 80 |
+
kl_loss_coef: 0.001 # for grpo
|
| 81 |
+
kl_loss_type: low_var_kl # for grpo
|
| 82 |
+
ppo_epochs: 1
|
| 83 |
+
shuffle: False
|
| 84 |
+
ulysses_sequence_parallel_size: 1 # sp size
|
| 85 |
+
optim:
|
| 86 |
+
lr: 1e-6
|
| 87 |
+
lr_warmup_steps_ratio: 0. # the total steps will be injected during runtime
|
| 88 |
+
min_lr_ratio: null # only useful for warmup with cosine
|
| 89 |
+
warmup_style: constant # select from constant/cosine
|
| 90 |
+
total_training_steps: -1 # must be override by program
|
| 91 |
+
fsdp_config:
|
| 92 |
+
wrap_policy:
|
| 93 |
+
# transformer_layer_cls_to_wrap: None
|
| 94 |
+
min_num_params: 0
|
| 95 |
+
param_offload: False
|
| 96 |
+
grad_offload: False
|
| 97 |
+
optimizer_offload: False
|
| 98 |
+
fsdp_size: -1
|
| 99 |
+
ref:
|
| 100 |
+
fsdp_config:
|
| 101 |
+
param_offload: False
|
| 102 |
+
wrap_policy:
|
| 103 |
+
# transformer_layer_cls_to_wrap: None
|
| 104 |
+
min_num_params: 0
|
| 105 |
+
log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu
|
| 106 |
+
log_prob_micro_batch_size_per_gpu: 16
|
| 107 |
+
log_prob_use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz}
|
| 108 |
+
log_prob_max_token_len_per_gpu: ${actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
|
| 109 |
+
ulysses_sequence_parallel_size: ${actor_rollout_ref.actor.ulysses_sequence_parallel_size} # sp size
|
| 110 |
+
rollout:
|
| 111 |
+
name: vllm
|
| 112 |
+
temperature: 1.0
|
| 113 |
+
top_k: -1 # 0 for hf rollout, -1 for vllm rollout
|
| 114 |
+
top_p: 1
|
| 115 |
+
prompt_length: ${data.max_prompt_length} # not use for opensource
|
| 116 |
+
response_length: ${data.max_response_length}
|
| 117 |
+
# for vllm rollout
|
| 118 |
+
dtype: bfloat16 # should align with FSDP
|
| 119 |
+
gpu_memory_utilization: 0.5
|
| 120 |
+
ignore_eos: False
|
| 121 |
+
enforce_eager: True
|
| 122 |
+
free_cache_engine: True
|
| 123 |
+
load_format: dummy_dtensor
|
| 124 |
+
tensor_model_parallel_size: 2
|
| 125 |
+
max_num_batched_tokens: 8192
|
| 126 |
+
max_num_seqs: 1024
|
| 127 |
+
log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu
|
| 128 |
+
log_prob_micro_batch_size_per_gpu: 16
|
| 129 |
+
log_prob_use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz}
|
| 130 |
+
log_prob_max_token_len_per_gpu: ${actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
|
| 131 |
+
# for hf rollout
|
| 132 |
+
do_sample: True
|
| 133 |
+
# number of responses (i.e. num sample times)
|
| 134 |
+
n: 1 # > 1 for grpo
|
| 135 |
+
|
| 136 |
+
**Common config for actor, rollout and reference model**
|
| 137 |
+
|
| 138 |
+
- ``actor_rollout_ref.hybrid_engine``: Whether it's a hybrid engine,
|
| 139 |
+
currently only supports hybrid engine
|
| 140 |
+
- ``actor_rollout_ref.model.path``: Huggingface model path. This can be
|
| 141 |
+
either local path or HDFS path. For HDFS path, we provide utils to
|
| 142 |
+
download it to DRAM and convert the HDFS path to local path.
|
| 143 |
+
- ``actor_rollout_ref.model.external_libs``: Additional Python packages
|
| 144 |
+
that need to be imported. Used to register models or tokenizers into
|
| 145 |
+
the Huggingface system.
|
| 146 |
+
- ``actor_rollout_ref.model.override_config``: Used to override some of
|
| 147 |
+
the model's original configurations, mainly dropout
|
| 148 |
+
- ``actor_rollout_ref.model.enable_gradient_checkpointing``: Whether to
|
| 149 |
+
enable gradient checkpointing for the actor
|
| 150 |
+
|
| 151 |
+
**Actor model**
|
| 152 |
+
|
| 153 |
+
- ``actor_rollout_ref.actor.strategy``: fsdp or megatron. In this
|
| 154 |
+
example, we use fsdp backend.
|
| 155 |
+
|
| 156 |
+
- ``actor_rollout_ref.actor.ppo_mini_batch_size``: One sample is split
|
| 157 |
+
into multiple sub-batches with batch_size=ppo_mini_batch_size for PPO
|
| 158 |
+
updates. The ppo_mini_batch_size is a global num across all workers/gpus
|
| 159 |
+
|
| 160 |
+
- ``actor_rollout_ref.actor.ppo_micro_batch_size``: [Will be deprecated, use ppo_micro_batch_size_per_gpu]
|
| 161 |
+
Similar to gradient accumulation, the micro_batch_size_per_gpu for one forward pass,
|
| 162 |
+
trading speed for GPU memory. The value represent the global view.
|
| 163 |
+
|
| 164 |
+
- ``actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu``: Similar to gradient
|
| 165 |
+
accumulation, the micro_batch_size_per_gpu for one forward pass, trading speed
|
| 166 |
+
for GPU memory. The value represent the local num per gpu.
|
| 167 |
+
|
| 168 |
+
- ``actor_rollout_ref.actor.grad_clip``: Gradient clipping for actor
|
| 169 |
+
updates
|
| 170 |
+
|
| 171 |
+
- ``actor_rollout_ref.actor.clip_ratio``: PPO clip ratio
|
| 172 |
+
|
| 173 |
+
- ``actor_rollout_ref.actor.entropy_coeff``: The weight of entropy when
|
| 174 |
+
calculating PPO loss
|
| 175 |
+
|
| 176 |
+
- ``actor_rollout_ref.actor.ppo_epochs``: Number of epochs for PPO
|
| 177 |
+
updates on one set of sampled data
|
| 178 |
+
|
| 179 |
+
- ``actor_rollout_ref.actor.shuffle``: Whether to shuffle data when
|
| 180 |
+
there are multiple epochs
|
| 181 |
+
|
| 182 |
+
- ``actor_rollout_ref.actor.optim``: Actor's optimizer parameters
|
| 183 |
+
|
| 184 |
+
- ``actor_rollout_ref.actor.fsdp_config``: FSDP config for actor
|
| 185 |
+
training
|
| 186 |
+
|
| 187 |
+
- ``wrap_policy``: FSDP wrap policy. By default, it uses Huggingface's
|
| 188 |
+
wrap policy, i.e., wrapping by DecoderLayer
|
| 189 |
+
|
| 190 |
+
- No need to set transformer_layer_cls_to_wrap, so we comment it.
|
| 191 |
+
|
| 192 |
+
- ``*_offload``: Whether to enable parameter, gradient and optimizer
|
| 193 |
+
offload
|
| 194 |
+
|
| 195 |
+
- Trading speed for GPU memory.
|
| 196 |
+
|
| 197 |
+
**Reference Model**
|
| 198 |
+
|
| 199 |
+
- ``actor_rollout_ref.ref``: FSDP config same as actor. **For models
|
| 200 |
+
larger than 7B, it's recommended to turn on offload for ref by
|
| 201 |
+
default**
|
| 202 |
+
|
| 203 |
+
- ``actor_rollout_ref.ref.log_prob_micro_batch_size``: [Will be deprecate, use log_prob_micro_batch_size_per_gpu]
|
| 204 |
+
The batch size for one forward pass in the computation of ``ref_log_prob``. The value represent the global num.
|
| 205 |
+
|
| 206 |
+
- ``actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu``: The batch size
|
| 207 |
+
for one forward pass in the computation of ``ref_log_prob``. The value represent the local num per gpu.
|
| 208 |
+
|
| 209 |
+
**Rollout Model**
|
| 210 |
+
|
| 211 |
+
- ``actor_rollout_ref.rollout.name``: hf/vllm. We use vLLM by default
|
| 212 |
+
because it's much efficient and our hybrid engine is implemented with
|
| 213 |
+
vLLM.
|
| 214 |
+
|
| 215 |
+
- Rollout (Auto-regressive) parameters. The key should be equal to the
|
| 216 |
+
property name in vLLM's ``SamplingParams``.
|
| 217 |
+
|
| 218 |
+
- ``temperature``, ``top_k``, ``top_p`` and others: Sampling
|
| 219 |
+
parameters in ``SamplingParams``.
|
| 220 |
+
|
| 221 |
+
- ``dtype``: Rollout model parameters type. This should be align with
|
| 222 |
+
the actor model parameter type in FSDP/Megatron backend.
|
| 223 |
+
|
| 224 |
+
- ``gpu_memory_utilization``: The proportion of the remaining GPU memory
|
| 225 |
+
allocated for kv cache after other models have initialized when using
|
| 226 |
+
vllm.
|
| 227 |
+
|
| 228 |
+
- ``tensor_model_parallel_size``: TP size for rollout. Only effective
|
| 229 |
+
for vllm.
|
| 230 |
+
|
| 231 |
+
- ``actor_rollout_ref.ref.log_prob_micro_batch_size``: [Will be deprecate, use log_prob_micro_batch_size_per_gpu]
|
| 232 |
+
The batch size for one forward pass in the computation of ``log_prob``. The value represent the global num.
|
| 233 |
+
|
| 234 |
+
- ``log_prob_micro_batch_size_per_gpu``: Micro batch size per gpu (The batch size for
|
| 235 |
+
one forward pass) for recalculating ``log_prob``. The value represent the local num per gpu.
|
| 236 |
+
|
| 237 |
+
- ``do_sample``: Whether to sample. If set to False, the rollout model
|
| 238 |
+
will perform greedy sampling. We disable ``do_sample`` during
|
| 239 |
+
validation.
|
| 240 |
+
|
| 241 |
+
- ``actor_rollout_ref.rollout.ignore_eos``: Whether to ignore the EOS
|
| 242 |
+
token and continue generating tokens after the EOS token is generated.
|
| 243 |
+
|
| 244 |
+
- ``actor_rollout_ref.rollout.free_cache_engine``: Offload the KVCache
|
| 245 |
+
after rollout generation stage. Default is True. When set to True, we
|
| 246 |
+
need to disable the usage of CUDAGraph (set ``enforce_eager`` to
|
| 247 |
+
True.)
|
| 248 |
+
|
| 249 |
+
- ``actor_rollout_ref.rollout.enforce_eager``: Whether to use CUDAGraph
|
| 250 |
+
in vLLM generation. Default set to True to disable CUDAGraph.
|
| 251 |
+
|
| 252 |
+
- ``actor_rollout_ref.rollout.load_format``: Which weight loader to use
|
| 253 |
+
to load the actor model weights to the rollout model.
|
| 254 |
+
|
| 255 |
+
- ``auto``: Use Megatron weight loader.
|
| 256 |
+
- ``megatron``: Use Megatron weight loader. Deployed with Megatron
|
| 257 |
+
backend. The input model ``state_dict()`` is already partitioned
|
| 258 |
+
along TP dimension and already gathered along PP dimension. This
|
| 259 |
+
weight loader requires that the Rollout model and Actor model's
|
| 260 |
+
parameters shape and name should be identical.
|
| 261 |
+
- ``dtensor``: Default solution when using Huggingface weight loader.
|
| 262 |
+
Deployed with FSDP backend and the state_dict_type is
|
| 263 |
+
``StateDictType.SHARDED_STATE_DICT``. Recommend to use this weight
|
| 264 |
+
loader
|
| 265 |
+
- ``hf``: Use Huggingface weight loader. Deployed with FSDP backend
|
| 266 |
+
and the state_dict_type is ``StateDictType.FULL_STATE_DICT``. This
|
| 267 |
+
solution doesn't need to rewrite the weight loader for each model
|
| 268 |
+
implemented in vLLM but it results in larger peak memory usage.
|
| 269 |
+
- ``dummy_hf``, ``dummy_megatron``, ``dummy_dtensor``: Random
|
| 270 |
+
initialization.
|
| 271 |
+
|
| 272 |
+
.. note:: **NOTED**: In this config field, users only need to select from ``dummy_megatron``, ``dummy_dtensor``, ``dummy_hf`` for rollout initialization and our hybrid engine will select the corresponding weight loader (i.e., ``megatron``, ``dtensor``, ``hf``) during actor/rollout weight synchronization.
|
| 273 |
+
|
| 274 |
+
Critic Model
|
| 275 |
+
~~~~~~~~~~~~
|
| 276 |
+
|
| 277 |
+
Most parameters for Critic are similar to Actor Model.
|
| 278 |
+
|
| 279 |
+
Reward Model
|
| 280 |
+
~~~~~~~~~~~~
|
| 281 |
+
|
| 282 |
+
.. code:: yaml
|
| 283 |
+
|
| 284 |
+
reward_model:
|
| 285 |
+
enable: False
|
| 286 |
+
model:
|
| 287 |
+
input_tokenizer: ${actor_rollout_ref.model.path} # set this to null if the chat template is identical
|
| 288 |
+
path: ~/models/Anomy-RM-v0.1
|
| 289 |
+
external_lib: ${actor_rollout_ref.model.external_lib}
|
| 290 |
+
fsdp_config:
|
| 291 |
+
min_num_params: 0
|
| 292 |
+
param_offload: False
|
| 293 |
+
micro_batch_size_per_gpu: 16
|
| 294 |
+
max_length: null
|
| 295 |
+
reward_manager: naive
|
| 296 |
+
|
| 297 |
+
- ``reward_model.enable``: Whether to enable reward model. If False, we
|
| 298 |
+
compute the reward only with the user-defined reward functions. In
|
| 299 |
+
GSM8K and Math examples, we disable reward model. For RLHF alignment
|
| 300 |
+
example using full_hh_rlhf, we utilize reward model to assess the
|
| 301 |
+
responses. If False, the following parameters are not effective.
|
| 302 |
+
- ``reward_model.model``
|
| 303 |
+
|
| 304 |
+
- ``input_tokenizer``: Input tokenizer. If the reward model's chat
|
| 305 |
+
template is inconsistent with the policy, we need to first decode to
|
| 306 |
+
plaintext, then apply the rm's chat_template. Then score with RM. If
|
| 307 |
+
chat_templates are consistent, it can be set to null.
|
| 308 |
+
- ``path``: RM's HDFS path or local path. Note that RM only supports
|
| 309 |
+
AutoModelForSequenceClassification. Other model types need to define
|
| 310 |
+
their own RewardModelWorker and pass it from the code.
|
| 311 |
+
- ``reward_model.reward_manager``: Reward Manager. This defines the mechanism
|
| 312 |
+
of computing rule-based reward and handling different reward sources. Default
|
| 313 |
+
if ``naive``. If all verification functions are multiprocessing-safe, the reward
|
| 314 |
+
manager can be set to ``prime`` for parallel verification.
|
| 315 |
+
|
| 316 |
+
Algorithm
|
| 317 |
+
~~~~~~~~~
|
| 318 |
+
|
| 319 |
+
.. code:: yaml
|
| 320 |
+
|
| 321 |
+
algorithm:
|
| 322 |
+
gamma: 1.0
|
| 323 |
+
lam: 1.0
|
| 324 |
+
adv_estimator: gae
|
| 325 |
+
kl_penalty: kl # how to estimate kl divergence
|
| 326 |
+
kl_ctrl:
|
| 327 |
+
type: fixed
|
| 328 |
+
kl_coef: 0.005
|
| 329 |
+
|
| 330 |
+
- ``gemma``: discount factor
|
| 331 |
+
- ``lam``: Trade-off between bias and variance in the GAE estimator
|
| 332 |
+
- ``adv_estimator``: Support ``gae``, ``grpo``, ``reinforce_plus_plus``.
|
| 333 |
+
- ``kl_penalty``: Support ``kl``, ``abs``, ``mse`` and ``full``. How to
|
| 334 |
+
calculate the kl divergence between actor and reference policy. For
|
| 335 |
+
specific options, refer to `core_algos.py <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py#L192>`_ .
|
| 336 |
+
|
| 337 |
+
Trainer
|
| 338 |
+
~~~~~~~
|
| 339 |
+
|
| 340 |
+
.. code:: yaml
|
| 341 |
+
|
| 342 |
+
trainer:
|
| 343 |
+
total_epochs: 30
|
| 344 |
+
project_name: verl_examples
|
| 345 |
+
experiment_name: gsm8k
|
| 346 |
+
logger: ['console', 'wandb']
|
| 347 |
+
nnodes: 1
|
| 348 |
+
n_gpus_per_node: 8
|
| 349 |
+
save_freq: -1
|
| 350 |
+
test_freq: 2
|
| 351 |
+
critic_warmup: 0
|
| 352 |
+
default_hdfs_dir: ~/experiments/gsm8k/ppo/${trainer.experiment_name} # hdfs checkpoint path
|
| 353 |
+
default_local_dir: checkpoints/${trainer.project_name}/${trainer.experiment_name} # local checkpoint path
|
| 354 |
+
|
| 355 |
+
- ``trainer.total_epochs``: Number of epochs in training.
|
| 356 |
+
- ``trainer.project_name``: For wandb
|
| 357 |
+
- ``trainer.experiment_name``: For wandb
|
| 358 |
+
- ``trainer.logger``: Support console and wandb
|
| 359 |
+
- ``trainer.nnodes``: Number of nodes used in the training.
|
| 360 |
+
- ``trainer.n_gpus_per_node``: Number of GPUs per node.
|
| 361 |
+
- ``trainer.save_freq``: The frequency (by iteration) to save checkpoint
|
| 362 |
+
of the actor and critic model.
|
| 363 |
+
- ``trainer.test_freq``: The validation frequency (by iteration).
|
| 364 |
+
- ``trainer.critic_warmup``: The number of iteration to train the critic
|
| 365 |
+
model before actual policy learning.
|
frameworks/agentgym_rl/docs/examples/gsm8k_example.rst
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
GSM8K Example
|
| 2 |
+
=============
|
| 3 |
+
|
| 4 |
+
Introduction
|
| 5 |
+
------------
|
| 6 |
+
|
| 7 |
+
In this example, we train an LLM to tackle the GSM8k task.
|
| 8 |
+
|
| 9 |
+
Paper: https://arxiv.org/pdf/2110.14168
|
| 10 |
+
|
| 11 |
+
Dataset: https://huggingface.co/datasets/gsm8k
|
| 12 |
+
|
| 13 |
+
Note that the original paper mainly focuses on training a verifier (a
|
| 14 |
+
reward model) to solve math problems via Best-of-N sampling. In this
|
| 15 |
+
example, we train an RLHF agent using a rule-based reward model.
|
| 16 |
+
|
| 17 |
+
Dataset Introduction
|
| 18 |
+
--------------------
|
| 19 |
+
|
| 20 |
+
GSM8k is a math problem dataset. The prompt is an elementary school
|
| 21 |
+
problem. The LLM model is required to answer the math problem.
|
| 22 |
+
|
| 23 |
+
The training set contains 7473 samples and the test set contains 1319
|
| 24 |
+
samples.
|
| 25 |
+
|
| 26 |
+
**An example**
|
| 27 |
+
|
| 28 |
+
Prompt
|
| 29 |
+
|
| 30 |
+
Katy makes coffee using teaspoons of sugar and cups of water in the
|
| 31 |
+
ratio of 7:13. If she used a total of 120 teaspoons of sugar and cups
|
| 32 |
+
of water, calculate the number of teaspoonfuls of sugar she used.
|
| 33 |
+
|
| 34 |
+
Solution
|
| 35 |
+
|
| 36 |
+
The total ratio representing the ingredients she used to make the
|
| 37 |
+
coffee is 7+13 = <<7+13=20>>20 Since the fraction representing the
|
| 38 |
+
number of teaspoons she used is 7/20, she used 7/20\ *120 =
|
| 39 |
+
<<7/20*\ 120=42>>42 #### 42
|
| 40 |
+
|
| 41 |
+
Step 1: Prepare dataset
|
| 42 |
+
-----------------------
|
| 43 |
+
|
| 44 |
+
.. code:: bash
|
| 45 |
+
|
| 46 |
+
cd examples/data_preprocess
|
| 47 |
+
python3 gsm8k.py --local_dir ~/data/gsm8k
|
| 48 |
+
|
| 49 |
+
Step 2: Download Model
|
| 50 |
+
----------------------
|
| 51 |
+
|
| 52 |
+
There're three ways to prepare the model checkpoints for post-training:
|
| 53 |
+
|
| 54 |
+
- Download the required models from hugging face
|
| 55 |
+
|
| 56 |
+
.. code:: bash
|
| 57 |
+
|
| 58 |
+
huggingface-cli download deepseek-ai/deepseek-math-7b-instruct --local-dir ~/models/deepseek-math-7b-instruct --local-dir-use-symlinks False
|
| 59 |
+
|
| 60 |
+
- Already store your store model in the local directory or HDFS path.
|
| 61 |
+
- Also, you can directly use the model name in huggingface (e.g.,
|
| 62 |
+
deepseek-ai/deepseek-math-7b-instruct) in
|
| 63 |
+
``actor_rollout_ref.model.path`` and ``critic.model.path`` field in
|
| 64 |
+
the run script.
|
| 65 |
+
|
| 66 |
+
Noted that users should prepare checkpoints for actor, critic and reward
|
| 67 |
+
model.
|
| 68 |
+
|
| 69 |
+
[Optional] Step 3: SFT your Model
|
| 70 |
+
---------------------------------
|
| 71 |
+
|
| 72 |
+
We provide a SFT Trainer using PyTorch FSDP in
|
| 73 |
+
`fsdp_sft_trainer.py <https://github.com/volcengine/verl/blob/main/verl/trainer/fsdp_sft_trainer.py>`_.
|
| 74 |
+
Users can customize their own SFT
|
| 75 |
+
script using our FSDP SFT Trainer.
|
| 76 |
+
|
| 77 |
+
We also provide various training scripts for SFT on GSM8K dataset in `gsm8k sft directory <https://github.com/volcengine/verl/blob/main/examples/sft/gsm8k/>`_.
|
| 78 |
+
|
| 79 |
+
.. code:: shell
|
| 80 |
+
|
| 81 |
+
set -x
|
| 82 |
+
|
| 83 |
+
torchrun -m verl.trainer.fsdp_sft_trainer \
|
| 84 |
+
data.train_files=$HOME/data/gsm8k/train.parquet \
|
| 85 |
+
data.val_files=$HOME/data/gsm8k/test.parquet \
|
| 86 |
+
data.prompt_key=question \
|
| 87 |
+
data.response_key=answer \
|
| 88 |
+
data.micro_batch_size_per_gpu=8 \
|
| 89 |
+
model.partial_pretrain=deepseek-ai/deepseek-coder-6.7b-instruct \
|
| 90 |
+
trainer.default_hdfs_dir=hdfs://user/verl/experiments/gsm8k/deepseek-coder-6.7b-instruct/ \
|
| 91 |
+
trainer.project_name=gsm8k-sft \
|
| 92 |
+
trainer.experiment_name=gsm8k-sft-deepseek-coder-6.7b-instruct \
|
| 93 |
+
trainer.total_epochs=4 \
|
| 94 |
+
trainer.logger=['console','wandb']
|
| 95 |
+
|
| 96 |
+
Step 4: Perform PPO training with your model on GSM8K Dataset
|
| 97 |
+
-------------------------------------------------------------
|
| 98 |
+
|
| 99 |
+
- Prepare your own run.sh script. Here's an example for GSM8k dataset
|
| 100 |
+
and deepseek-llm-7b-chat model.
|
| 101 |
+
- Users could replace the ``data.train_files`` ,\ ``data.val_files``,
|
| 102 |
+
``actor_rollout_ref.model.path`` and ``critic.model.path`` based on
|
| 103 |
+
their environment.
|
| 104 |
+
- See :doc:`config` for detailed explanation of each config field.
|
| 105 |
+
|
| 106 |
+
**Reward Model/Function**
|
| 107 |
+
|
| 108 |
+
We use a rule-based reward model. We force the model to produce a final
|
| 109 |
+
answer following 4 “#” as shown in the solution. We extract the final
|
| 110 |
+
answer from both the solution and model's output using regular
|
| 111 |
+
expression matching. We compare them and assign a reward of 1 to correct
|
| 112 |
+
answer, 0.1 to incorrect answer and 0 to no answer.
|
| 113 |
+
|
| 114 |
+
**Training Script**
|
| 115 |
+
|
| 116 |
+
The training script example for FSDP and Megatron-LM backend are stored in examples/ppo_trainer directory.
|
| 117 |
+
|
| 118 |
+
.. code:: bash
|
| 119 |
+
|
| 120 |
+
cd ../ppo_trainer
|
| 121 |
+
bash run_deepseek7b_llm.sh
|
| 122 |
+
|
| 123 |
+
The script of run_deepseek7b_llm.sh
|
| 124 |
+
|
| 125 |
+
.. code:: bash
|
| 126 |
+
|
| 127 |
+
set -x
|
| 128 |
+
|
| 129 |
+
python3 -m verl.trainer.main_ppo \
|
| 130 |
+
data.train_files=$HOME/data/gsm8k/train.parquet \
|
| 131 |
+
data.val_files=$HOME/data/gsm8k/test.parquet \
|
| 132 |
+
data.train_batch_size=1024 \
|
| 133 |
+
data.val_batch_size=1312 \
|
| 134 |
+
data.max_prompt_length=512 \
|
| 135 |
+
data.max_response_length=512 \
|
| 136 |
+
actor_rollout_ref.model.path=deepseek-ai/deepseek-llm-7b-chat \
|
| 137 |
+
actor_rollout_ref.actor.optim.lr=1e-6 \
|
| 138 |
+
actor_rollout_ref.model.use_remove_padding=True \
|
| 139 |
+
actor_rollout_ref.actor.ppo_mini_batch_size=256 \
|
| 140 |
+
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=16 \
|
| 141 |
+
actor_rollout_ref.actor.fsdp_config.param_offload=False \
|
| 142 |
+
actor_rollout_ref.actor.fsdp_config.grad_offload=False \
|
| 143 |
+
actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
|
| 144 |
+
actor_rollout_ref.model.enable_gradient_checkpointing=True \
|
| 145 |
+
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=32 \
|
| 146 |
+
actor_rollout_ref.rollout.tensor_model_parallel_size=4 \
|
| 147 |
+
actor_rollout_ref.rollout.name=vllm \
|
| 148 |
+
actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \
|
| 149 |
+
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=32 \
|
| 150 |
+
actor_rollout_ref.ref.fsdp_config.param_offload=True \
|
| 151 |
+
critic.optim.lr=1e-5 \
|
| 152 |
+
critic.model.use_remove_padding=True \
|
| 153 |
+
critic.model.path=deepseek-ai/deepseek-llm-7b-chat \
|
| 154 |
+
critic.model.enable_gradient_checkpointing=True \
|
| 155 |
+
critic.ppo_micro_batch_size_per_gpu=32 \
|
| 156 |
+
critic.model.fsdp_config.param_offload=False \
|
| 157 |
+
critic.model.fsdp_config.grad_offload=False \
|
| 158 |
+
critic.model.fsdp_config.optimizer_offload=False \
|
| 159 |
+
algorithm.kl_ctrl.kl_coef=0.001 \
|
| 160 |
+
trainer.critic_warmup=0 \
|
| 161 |
+
trainer.logger=['console','wandb'] \
|
| 162 |
+
trainer.project_name='verl_example_gsm8k' \
|
| 163 |
+
trainer.experiment_name='deepseek_llm_7b_function_rm' \
|
| 164 |
+
trainer.n_gpus_per_node=8 \
|
| 165 |
+
trainer.nnodes=1 \
|
| 166 |
+
trainer.save_freq=-1 \
|
| 167 |
+
trainer.test_freq=1 \
|
| 168 |
+
trainer.total_epochs=15 $@
|
frameworks/agentgym_rl/docs/examples/ppo_code_architecture.rst
ADDED
|
@@ -0,0 +1,207 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
PPO Example Architecture
|
| 2 |
+
========================
|
| 3 |
+
|
| 4 |
+
Let's start with the Proximal Policy Optimization algorithm, which is
|
| 5 |
+
most widely used algorithm in LLM post-training.
|
| 6 |
+
|
| 7 |
+
The main entry point of the PPO algorithm example is:
|
| 8 |
+
`main_ppo.py <https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py>`_.
|
| 9 |
+
In this tutorial, we will go through the code architecture in `main_ppo.py <https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py>`_.
|
| 10 |
+
|
| 11 |
+
Define the data
|
| 12 |
+
---------------
|
| 13 |
+
|
| 14 |
+
Users need to preprocess and store the dataset in parquet files.
|
| 15 |
+
And we implement `RLHFDataset` to load and tokenize the parquet files.
|
| 16 |
+
|
| 17 |
+
For ``RLHFDataset`` (Default), at least 1 fields are required:
|
| 18 |
+
|
| 19 |
+
- ``prompt``: Contains the string prompt
|
| 20 |
+
|
| 21 |
+
We already provide some examples of processing the datasets to parquet
|
| 22 |
+
files in `data_preprocess directory <https://github.com/volcengine/verl/blob/main/examples/data_preprocess>`_. Currently, we support
|
| 23 |
+
preprocess of GSM8k, MATH, Hellasage, Full_hh_rlhf datasets. See :doc:`../preparation/prepare_data` for
|
| 24 |
+
more information.
|
| 25 |
+
|
| 26 |
+
Define the reward functions for different datasets
|
| 27 |
+
--------------------------------------------------
|
| 28 |
+
|
| 29 |
+
In this main entry point, the users only need to define their own reward
|
| 30 |
+
function based on the datasets (or applications) utilized in PPO
|
| 31 |
+
training.
|
| 32 |
+
|
| 33 |
+
For example, we already provide reward functions for `GSM8k <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/gsm8k.py>`_
|
| 34 |
+
and `MATH <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/math.py>`_
|
| 35 |
+
datasets in the ``_select_rm_score_fn``. In the ``RewardManager``, we
|
| 36 |
+
will compute the reward score based on the data_source to select
|
| 37 |
+
corresponding reward functions. For some RLHF datasets (e.g.,
|
| 38 |
+
full_hh_rlhf), the reward model is utilized to assess the responses
|
| 39 |
+
without any reward functions. In this case, the ``RewardManager`` will
|
| 40 |
+
return the ``rm_score`` computed by the reward model directly.
|
| 41 |
+
|
| 42 |
+
See `reward functions <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score>`_ for detailed implementation.
|
| 43 |
+
|
| 44 |
+
Define worker classes
|
| 45 |
+
---------------------
|
| 46 |
+
|
| 47 |
+
.. code:: python
|
| 48 |
+
|
| 49 |
+
if config.actor_rollout_ref.actor.strategy == 'fsdp': # for FSDP backend
|
| 50 |
+
assert config.actor_rollout_ref.actor.strategy == config.critic.strategy
|
| 51 |
+
from verl.workers.fsdp_workers import ActorRolloutRefWorker, CriticWorker
|
| 52 |
+
from verl.single_controller.ray import RayWorkerGroup
|
| 53 |
+
ray_worker_group_cls = RayWorkerGroup
|
| 54 |
+
|
| 55 |
+
elif config.actor_rollout_ref.actor.strategy == 'megatron': # for Megatron backend
|
| 56 |
+
assert config.actor_rollout_ref.actor.strategy == config.critic.strategy
|
| 57 |
+
from verl.workers.megatron_workers import ActorRolloutRefWorker, CriticWorker
|
| 58 |
+
from verl.single_controller.ray.megatron import NVMegatronRayWorkerGroup
|
| 59 |
+
ray_worker_group_cls = NVMegatronRayWorkerGroup # Ray worker class for Megatron-LM
|
| 60 |
+
|
| 61 |
+
else:
|
| 62 |
+
raise NotImplementedError
|
| 63 |
+
|
| 64 |
+
from verl.trainer.ppo.ray_trainer import ResourcePoolManager, Role
|
| 65 |
+
|
| 66 |
+
role_worker_mapping = {
|
| 67 |
+
Role.ActorRollout: ActorRolloutRefWorker,
|
| 68 |
+
Role.Critic: CriticWorker,
|
| 69 |
+
Role.RefPolicy: ActorRolloutRefWorker
|
| 70 |
+
}
|
| 71 |
+
|
| 72 |
+
global_pool_id = 'global_pool'
|
| 73 |
+
resource_pool_spec = {
|
| 74 |
+
global_pool_id: [config.trainer.n_gpus_per_node] * config.trainer.nnodes,
|
| 75 |
+
}
|
| 76 |
+
mapping = {
|
| 77 |
+
Role.ActorRollout: global_pool_id,
|
| 78 |
+
Role.Critic: global_pool_id,
|
| 79 |
+
Role.RefPolicy: global_pool_id,
|
| 80 |
+
}
|
| 81 |
+
|
| 82 |
+
Step 1: Construct the mapping between roles and workers
|
| 83 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 84 |
+
|
| 85 |
+
A role represents a group of workers in the same process. We have
|
| 86 |
+
pre-defined several roles in `ray_trainer.py <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py#L38>`_.
|
| 87 |
+
|
| 88 |
+
.. code:: python
|
| 89 |
+
|
| 90 |
+
class Role(Enum):
|
| 91 |
+
"""
|
| 92 |
+
To create more roles dynamically, you can subclass Role and add new members
|
| 93 |
+
"""
|
| 94 |
+
Actor = 0 # This worker only has Actor
|
| 95 |
+
Rollout = 1 # This worker only has Rollout
|
| 96 |
+
ActorRollout = 2 # This worker has both actor and rollout, it's a HybridEngine
|
| 97 |
+
Critic = 3 # This worker only has critic
|
| 98 |
+
RefPolicy = 4 # This worker only has reference policy
|
| 99 |
+
RewardModel = 5 # This worker only has reward model
|
| 100 |
+
ActorRolloutRef = 6 # This worker contains actor, rollout and reference policy simultaneously
|
| 101 |
+
|
| 102 |
+
Step 2: Define the worker class corresponding to this role
|
| 103 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 104 |
+
|
| 105 |
+
- We have pre-implemented the ``ActorRolloutRefWorker``. Through
|
| 106 |
+
different configs, it can be a standalone actor, a standalone rollout,
|
| 107 |
+
an ActorRollout HybridEngine, or an ActorRolloutRef HybridEngine
|
| 108 |
+
- We also pre-implemented workers for ``Actor``, ``Rollout``,
|
| 109 |
+
``Critic``, ``Reward Model`` and ``Reference model`` on two different
|
| 110 |
+
backend: PyTorch FSDP
|
| 111 |
+
and Megatron-LM.
|
| 112 |
+
See `FSDP Workers <https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py>`_
|
| 113 |
+
and `Megatron-LM Workers <https://github.com/volcengine/verl/blob/main/verl/workers/megatron_workers.py>`_
|
| 114 |
+
for more information.
|
| 115 |
+
|
| 116 |
+
Step 3: Define resource pool id and resource pool spec
|
| 117 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 118 |
+
|
| 119 |
+
- Resource pool is a division of global GPU resources,
|
| 120 |
+
``resource_pool_spec`` is a dict, mapping from id to # of GPUs
|
| 121 |
+
|
| 122 |
+
- In the above example, we defined a global resource pool:
|
| 123 |
+
global_pool_id, and then put all roles on this one resource pool
|
| 124 |
+
with all the GPUs in this post-training task. This refers to
|
| 125 |
+
*co-locate* placement where all the models share the same set of
|
| 126 |
+
GPUs.
|
| 127 |
+
|
| 128 |
+
- See resource pool and placement for advance usage.
|
| 129 |
+
|
| 130 |
+
Defining reward model/function
|
| 131 |
+
------------------------------
|
| 132 |
+
|
| 133 |
+
.. code:: python
|
| 134 |
+
|
| 135 |
+
# we should adopt a multi-source reward function here
|
| 136 |
+
# - for rule-based rm, we directly call a reward score
|
| 137 |
+
# - for model-based rm, we call a model
|
| 138 |
+
# - for code related prompt, we send to a sandbox if there are test cases
|
| 139 |
+
# - finally, we combine all the rewards together
|
| 140 |
+
# - The reward type depends on the tag of the data
|
| 141 |
+
if config.reward_model.enable:
|
| 142 |
+
from verl.workers.fsdp_workers import RewardModelWorker
|
| 143 |
+
role_worker_mapping[Role.RewardModel] = RewardModelWorker
|
| 144 |
+
mapping[Role.RewardModel] = global_pool_id
|
| 145 |
+
|
| 146 |
+
reward_fn = RewardManager(tokenizer=tokenizer, num_examine=0)
|
| 147 |
+
|
| 148 |
+
# Note that we always use function-based RM for validation
|
| 149 |
+
val_reward_fn = RewardManager(tokenizer=tokenizer, num_examine=1)
|
| 150 |
+
|
| 151 |
+
resource_pool_manager = ResourcePoolManager(resource_pool_spec=resource_pool_spec, mapping=mapping)
|
| 152 |
+
|
| 153 |
+
Since not all tasks use model-based RM, users need to define here
|
| 154 |
+
whether it's a model-based RM or a function-based RM
|
| 155 |
+
|
| 156 |
+
- If it's a model-based RM, directly add the ``RewardModel`` role in the
|
| 157 |
+
resource mapping and add it to the resource pool mapping.
|
| 158 |
+
|
| 159 |
+
- Note that the pre-defined ``RewardModelWorker`` only supports models
|
| 160 |
+
with the structure of huggingface
|
| 161 |
+
``AutoModelForSequenceClassification``. If it's not this model, you
|
| 162 |
+
need to define your own RewardModelWorker in `FSDP Workers <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/workers/fsdp_workers.py>`_
|
| 163 |
+
and `Megatron-LM Workers <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/workers/megatron_workers.py>`_.
|
| 164 |
+
|
| 165 |
+
- If it's a function-based RM, the users are required to classified the
|
| 166 |
+
reward function for each datasets.
|
| 167 |
+
|
| 168 |
+
.. code:: python
|
| 169 |
+
|
| 170 |
+
def _select_rm_score_fn(data_source):
|
| 171 |
+
if data_source == 'openai/gsm8k':
|
| 172 |
+
return gsm8k.compute_score
|
| 173 |
+
elif data_source == 'lighteval/MATH':
|
| 174 |
+
return math.compute_score
|
| 175 |
+
else:
|
| 176 |
+
raise NotImplementedError
|
| 177 |
+
|
| 178 |
+
See reward functions implemented in `directory <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/>`_
|
| 179 |
+
for more information.
|
| 180 |
+
|
| 181 |
+
Define, init and run the PPO Trainer
|
| 182 |
+
------------------------------------
|
| 183 |
+
|
| 184 |
+
.. code:: python
|
| 185 |
+
|
| 186 |
+
trainer = RayPPOTrainer(config=config,
|
| 187 |
+
tokenizer=tokenizer,
|
| 188 |
+
role_worker_mapping=role_worker_mapping,
|
| 189 |
+
resource_pool_manager=resource_pool_manager,
|
| 190 |
+
ray_worker_group_cls=ray_worker_group_cls,
|
| 191 |
+
reward_fn=reward_fn,
|
| 192 |
+
val_reward_fn=val_reward_fn)
|
| 193 |
+
trainer.init_workers()
|
| 194 |
+
trainer.fit()
|
| 195 |
+
|
| 196 |
+
- We first initialize the ``RayPPOTrainer`` with user config, tokenizer
|
| 197 |
+
and all the above worker mapping, resource pool, worker group and
|
| 198 |
+
reward functions
|
| 199 |
+
- We first call the ``trainer.init_workers()`` to initialize the models
|
| 200 |
+
on the allocated GPUs (in the resource pool)
|
| 201 |
+
- The actual PPO training will be executed in ``trainer.fit()``
|
| 202 |
+
|
| 203 |
+
verl can be easily extended to other RL algorithms by reusing the Ray
|
| 204 |
+
model workers, resource pool and reward functions. See :doc:`extension<../advance/dpo_extension>` for
|
| 205 |
+
more information.
|
| 206 |
+
|
| 207 |
+
Details of the ``RayPPOTrainer`` is discussed in :doc:`Ray Trainer<../workers/ray_trainer>`.
|
frameworks/agentgym_rl/docs/experiment/ppo.rst
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.. _algo-baseline-page:
|
| 2 |
+
|
| 3 |
+
Algorithm Baselines
|
| 4 |
+
===================
|
| 5 |
+
|
| 6 |
+
GSM8k
|
| 7 |
+
------------------
|
| 8 |
+
|
| 9 |
+
Assuming GSM8k dataset is preprocess via ``python3 examples/data_preprocess/gsm8k.py``
|
| 10 |
+
|
| 11 |
+
Refer to the table below to reproduce PPO training from different pre-trained models.
|
| 12 |
+
|
| 13 |
+
.. _Huggingface: https://huggingface.co/google/gemma-2-2b-it#benchmark-results
|
| 14 |
+
.. _SFT Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/gemma-2-2b-it-sft-0.411.log
|
| 15 |
+
.. _SFT+PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/gemma-2-2b-it-ppo-bsz512_4-prompt1024-resp-512-0.640.log
|
| 16 |
+
.. _wandb: https://api.wandb.ai/links/verl-team/h7ux8602
|
| 17 |
+
.. _Qwen Blog: https://qwenlm.github.io/blog/qwen2.5-llm/
|
| 18 |
+
.. _PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/Qwen2.5-0.5B-bsz256_2-prompt1024-resp512-0.567.log
|
| 19 |
+
.. _Megatron PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/deepseek-llm-7b-chat-megatron-bsz256_4-prompt512-resp512-0.695.log
|
| 20 |
+
.. _Qwen7b GRPO Script: https://github.com/volcengine/verl/blob/a65c9157bc0b85b64cd753de19f94e80a11bd871/examples/grpo_trainer/run_qwen2-7b_seq_balance.sh
|
| 21 |
+
.. _Megatron wandb: https://wandb.ai/verl-team/verl_megatron_gsm8k_examples/runs/10fetyr3
|
| 22 |
+
.. _Qwen7b ReMax Script: https://github.com/eric-haibin-lin/verl/blob/main/examples/remax_trainer/run_qwen2.5-3b_seq_balance.sh
|
| 23 |
+
.. _Qwen7b ReMax Wandb: https://wandb.ai/liziniu1997/verl_remax_example_gsm8k/runs/vxl10pln
|
| 24 |
+
|
| 25 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 26 |
+
| Model | Method | Test score | Details |
|
| 27 |
+
+==================================+========================+============+=====================+=========================================================================+
|
| 28 |
+
| google/gemma-2-2b-it | pretrained checkpoint | 23.9 | `Huggingface`_ |
|
| 29 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 30 |
+
| google/gemma-2-2b-it | SFT | 52.06 | `SFT Command and Logs`_ |
|
| 31 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 32 |
+
| google/gemma-2-2b-it | SFT + PPO | 64.02 | `SFT+PPO Command and Logs`_, `wandb`_ |
|
| 33 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 34 |
+
| Qwen/Qwen2.5-0.5B-Instruct | pretrained checkpoint | 36.4 | `Qwen Blog`_ |
|
| 35 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 36 |
+
| Qwen/Qwen2.5-0.5B-Instruct | PPO | 56.7 | `PPO Command and Logs`_ |
|
| 37 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 38 |
+
| deepseek-ai/deepseek-llm-7b-chat | PPO | 69.5 [1]_ | `Megatron PPO Command and Logs`_, `Megatron wandb`_ |
|
| 39 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 40 |
+
| Qwen/Qwen2-7B-Instruct | GRPO | 89 | `Qwen7b GRPO Script`_ |
|
| 41 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 42 |
+
| Qwen/Qwen2.5-7B-Instruct | ReMax | 97 | `Qwen7b ReMax Script`_, `Qwen7b ReMax Wandb`_ |
|
| 43 |
+
+----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
|
| 44 |
+
|
| 45 |
+
.. [1] During the evaluation, we have only extracted answers following the format "####". A more flexible answer exaction, longer response length and better prompt engineering may lead to higher score.
|
frameworks/agentgym_rl/docs/faq/faq.rst
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Frequently Asked Questions
|
| 2 |
+
====================================
|
| 3 |
+
|
| 4 |
+
Ray related
|
| 5 |
+
------------
|
| 6 |
+
|
| 7 |
+
How to add breakpoint for debugging with distributed Ray?
|
| 8 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 9 |
+
|
| 10 |
+
Please checkout the official debugging guide from Ray: https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
Distributed training
|
| 14 |
+
------------------------
|
| 15 |
+
|
| 16 |
+
How to run multi-node post-training with Ray?
|
| 17 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 18 |
+
|
| 19 |
+
You can start a ray cluster and submit a ray job, following the official guide from Ray: https://docs.ray.io/en/latest/ray-core/starting-ray.html
|
| 20 |
+
|
| 21 |
+
If your cluster is managed by Slurm, please refer to the guide for deploying Ray on Slurm: https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html
|
frameworks/agentgym_rl/docs/hybrid_flow.rst
ADDED
|
@@ -0,0 +1,269 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=========================================================
|
| 2 |
+
HybridFlow Programming Guide
|
| 3 |
+
=========================================================
|
| 4 |
+
|
| 5 |
+
.. _vermouth: https://github.com/vermouth1992
|
| 6 |
+
|
| 7 |
+
Author: `Chi Zhang <https://github.com/vermouth1992>`_
|
| 8 |
+
|
| 9 |
+
verl is an open source implementation of the paper `HybridFlow <https://arxiv.org/abs/2409.19256v2>`_ [1]_. In this section, we will introduce the basic concepts of HybridFlow, the motivation and how to program with verl APIs.
|
| 10 |
+
|
| 11 |
+
Motivation and Design
|
| 12 |
+
------------------------
|
| 13 |
+
We use dataflow to represent RL systems. [4]_.
|
| 14 |
+
|
| 15 |
+
DataFlow
|
| 16 |
+
~~~~~~~~~~~~~~~~~~~~
|
| 17 |
+
|
| 18 |
+
Dataflow is an abstraction of computations. Neural Netowork training is a typical dataflow. It can be represented by computational graph.
|
| 19 |
+
|
| 20 |
+
.. image:: https://github.com/eric-haibin-lin/verl-community/blob/main/docs/dataflow.jpeg?raw=true
|
| 21 |
+
:alt: The dataflow graph from CS231n 2024 lecture 4
|
| 22 |
+
|
| 23 |
+
This figure [2]_ represents the computation graph of a polynomial function followed by a sigmoid function. In the data flow of neural network computation, each node represents an operator, and each edge represents the direction of forward/backward propagation. The computation graph determines the architecture of the neural network.
|
| 24 |
+
|
| 25 |
+
RL as a dataflow problem
|
| 26 |
+
++++++++++++++++++++++++++++++++++++++++++++++
|
| 27 |
+
|
| 28 |
+
Reinforcement learning (RL) training can also be represented as a dataflow. Below is the dataflow graph that represents the PPO algorithm used in RLHF [3]_:
|
| 29 |
+
|
| 30 |
+
.. image:: https://picx.zhimg.com/70/v2-cb8ab5ee946a105aab6a563e92682ffa_1440w.avis?source=172ae18b&biz_tag=Post
|
| 31 |
+
:alt: PPO dataflow graph, credit to Zhihu 低级炼丹师
|
| 32 |
+
|
| 33 |
+
However, the dataflow of RL has fundamental differences compared with dataflow of neural network training as follows:
|
| 34 |
+
|
| 35 |
+
+--------------------------+--------------------------------------------------+---------------------+
|
| 36 |
+
| Workload | Node | Edge |
|
| 37 |
+
+--------------------------+--------------------------------------------------+---------------------+
|
| 38 |
+
| Neural Network Training | Operator (+/-/matmul/softmax) | Tensor movement |
|
| 39 |
+
+--------------------------+--------------------------------------------------+---------------------+
|
| 40 |
+
| Reinforcement Learning | High-level operators (rollout/model forward) | Data Movement |
|
| 41 |
+
+--------------------------+--------------------------------------------------+---------------------+
|
| 42 |
+
|
| 43 |
+
In the case of tabular reinforcement learning, each operator is a simple scalar math operation (e.g., bellman update). In deep reinforcement learning(DRL), each operator is a high-level neural network computation such as model inference/update. This makes RL a two-level dataflow problem:
|
| 44 |
+
|
| 45 |
+
- Control flow: defines how the high-level operators are executed (e.g., In PPO, we first perform rollout. Then, we perform advantage computation. Finally, we perform training). It expresses the **core logics of RL algorithms**.
|
| 46 |
+
- Computation flow: defines the dataflow of **neural network computation** (e.g., model forward/backward/optimizer).
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
Design Choices
|
| 50 |
+
~~~~~~~~~~~~~~~~~~~~
|
| 51 |
+
The model size used in DRL before the LLM era is typically small. Thus, the high-level neural network computation can be done in a single process. This enables embedding the computation flow inside the control flow as a single process.
|
| 52 |
+
|
| 53 |
+
However, in the LLM era, the computation flow (e.g., training neural network) becomes a multi-process program. This naturally leads to two design choices:
|
| 54 |
+
|
| 55 |
+
1. Convert the control flow into a multi-process program as well. Then colocate with computation flow (unified multi-controller)
|
| 56 |
+
|
| 57 |
+
- Advantages:
|
| 58 |
+
|
| 59 |
+
- Achieves the **optimal performance** under fixed computation flow and control flow as the communication overhead in both training and data transfer is minimized.
|
| 60 |
+
|
| 61 |
+
- Disadvantages:
|
| 62 |
+
|
| 63 |
+
- The computation and/or control flow is **hard to reuse** from software perspective as computation code is coupled with specific controller code. For example, the training loop of PPO is generic. Say we have an PPO training flow implemented with a specific computation flow such as FSDP. Neither the control flow or computation flow can be reused if we want to switch the computation flow from FSDP to Megatron, due to the coupling of control and computation flows.
|
| 64 |
+
- Requires more efforts from the user under flexible and dynamic control flows, due to the multi-process nature of the program.
|
| 65 |
+
|
| 66 |
+
2. Separate the flows: single process for the control flow and multi-process for computation flow
|
| 67 |
+
|
| 68 |
+
- Advantages:
|
| 69 |
+
|
| 70 |
+
- The computation flow defined elsewhere can be **easily reused** after the decoupling.
|
| 71 |
+
- The controller runs on a single process. Implementing a new RL algorithm with a **different control flow is simple and easy**.
|
| 72 |
+
|
| 73 |
+
- Disadvantages:
|
| 74 |
+
|
| 75 |
+
- Additional **data communication overhead** each time the controller process and computatation processes interact. The data has to be sent back and forth.
|
| 76 |
+
|
| 77 |
+
In verl, the latter strategy with separate control flow and computation flow is adopted. verl is designed to decouple the control flow of RL algorithms, and the implementation of computation engines.
|
| 78 |
+
|
| 79 |
+
Overall Execution Diagram
|
| 80 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 81 |
+
|
| 82 |
+
Below is a simplified diagram denoting the execution of a reinforcement learning job. In the diagram, the controller runs on a single process, while the generator/actor workers, critic workers run on multiple processes, placed with specific resource groups. For rollout, the controller passes the data to the generator to perform sample generation. When the rollout is done, the data is passed back to controller for the next step of the algorithm. Similar execution is done for other workers. With the hybrid controller design, the data flow and computation is decoupled to provide both efficiency in computation and flexiblity in defining algorithm training loops.
|
| 83 |
+
|
| 84 |
+
.. figure:: https://github.com/eric-haibin-lin/verl-community/blob/main/docs/driver_worker.png?raw=true
|
| 85 |
+
:alt: The execution diagram
|
| 86 |
+
|
| 87 |
+
Codebase walkthrough (PPO)
|
| 88 |
+
------------------------------------------------
|
| 89 |
+
|
| 90 |
+
Entry function
|
| 91 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 92 |
+
Code: https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py
|
| 93 |
+
|
| 94 |
+
In this file, we define a remote function `main_task` that serves as the controller (driver) process as shown in the above figure. We also define a ``RewardManager``, where users can customize their reward function based on the data source in the dataset. Note that `RewardManager` should return the final token-level reward that is optimized by RL algorithms. Note that users can combine model-based rewards and rule-based rewards.
|
| 95 |
+
The ``main_task`` constructs a RayPPOTrainer instance and launch the fit. Note that ``main_task`` **runs as a single process**.
|
| 96 |
+
|
| 97 |
+
We highly recommend that the ``main_task`` is NOT schduled on the head of the ray cluster because ``main_task`` will consume a lot of memory but the head usually contains very few resources.
|
| 98 |
+
|
| 99 |
+
Ray trainer
|
| 100 |
+
~~~~~~~~~~~~~~~~~~~~
|
| 101 |
+
Code: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py
|
| 102 |
+
|
| 103 |
+
The RayPPOTrainer manages
|
| 104 |
+
|
| 105 |
+
- Worker and WorkerGroup construction
|
| 106 |
+
- Runs the main loop of PPO algorithm
|
| 107 |
+
|
| 108 |
+
Note that, the fit function of RayPPOTrainer **runs as a single process**.
|
| 109 |
+
|
| 110 |
+
Worker and WorkerGroup construction
|
| 111 |
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 112 |
+
|
| 113 |
+
Each workerGroup manages a list of workers that runs remotely. Note that the worker group runs in the process of its construtor.
|
| 114 |
+
Each worker inside the WorkerGroup runs on a GPU. The worker group serves as a proxy for the controller process to interact with a list of workers, in order to perform certain computations. **In order to do so, we have to bind the methods of the worker into the method of the WorkerGroup and define the data dispatch and data collection**. This is done via simple decoration that will be introduced in the Worker definition section.
|
| 115 |
+
|
| 116 |
+
For example, in PPO, we define 3 worker groups:
|
| 117 |
+
|
| 118 |
+
- ActorRolloutRef: manages actor, rollout and reference policy. ActorRolloutRefWorker can be instantiated as a single actor, a single rollout, a single reference policy, a combined actor/rollout or a combined actor/rollout/ref. This design is aimed for the maximum code reuse in various scenarios. The reason for colocating actor and rollout is for fast weight transfer using nccl. The reason for coloating actor and reference is to implement an efficient lora PPO as the reference policy is simply the base model of PPO in lora.
|
| 119 |
+
- Critic: manages the critic model
|
| 120 |
+
- Reward: manages the reward model
|
| 121 |
+
|
| 122 |
+
The worker group will be constructed on the resource pool it designates. The resource pool is a set of GPUs in the ray cluster.
|
| 123 |
+
|
| 124 |
+
Worker definition
|
| 125 |
+
~~~~~~~~~~~~~~~~~~~~
|
| 126 |
+
|
| 127 |
+
.. _ActorRolloutRefWorker: https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py
|
| 128 |
+
|
| 129 |
+
We take `ActorRolloutRefWorker <_ActorRolloutRefWorker>`_ for an exmaple.
|
| 130 |
+
The APIs it should expose to the controller process are:
|
| 131 |
+
|
| 132 |
+
- init_model: build the underlying model
|
| 133 |
+
- generate_sequences: given prompts, generate responses
|
| 134 |
+
- compute_log_prob: compute the log-probability of a generated sequence using actor
|
| 135 |
+
- compute_ref_log_prob: compute the log-probability of a generated sequence using reference policy
|
| 136 |
+
- save_checkpoint: save the checkpoint
|
| 137 |
+
|
| 138 |
+
Note that these methods are defined in the worker that can only be invoked via remote calls. For example, if the controller process wants to initialize the model, it has to call
|
| 139 |
+
|
| 140 |
+
.. code-block:: python
|
| 141 |
+
|
| 142 |
+
for worker in actor_rollout_ref_wg:
|
| 143 |
+
worker.init_model.remote()
|
| 144 |
+
|
| 145 |
+
If the controller process wants to generate sequences, it has to call
|
| 146 |
+
|
| 147 |
+
.. code-block:: python
|
| 148 |
+
|
| 149 |
+
data = xxx
|
| 150 |
+
# split the data into dp chunks
|
| 151 |
+
data_dp_lst = data.split(dp_size)
|
| 152 |
+
output_dp_lst = []
|
| 153 |
+
for i, worker in enumerate(actor_rollout_ref_wg):
|
| 154 |
+
output_future = worker.generate_sequences.remote(data_dp_lst[i])
|
| 155 |
+
output_dp_lst.append(output_future)
|
| 156 |
+
output = torch.cat(ray.get(output_dp_lst), dim=0)
|
| 157 |
+
|
| 158 |
+
We observe that controll process calling worker group methods in general can be divided into 3 parts:
|
| 159 |
+
|
| 160 |
+
- Split the data into data parallel sizes
|
| 161 |
+
- Dispatch the corresponding data into each worker
|
| 162 |
+
- Collect and concatenate the data when the computation finishes
|
| 163 |
+
|
| 164 |
+
In verl, we design a syntax sugar to encapsulate the 3 processes into a single call from the controller process.
|
| 165 |
+
|
| 166 |
+
.. code-block:: python
|
| 167 |
+
|
| 168 |
+
@register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
|
| 169 |
+
def generate_sequences(data):
|
| 170 |
+
...
|
| 171 |
+
|
| 172 |
+
# on the driver
|
| 173 |
+
output = actor_rollout_ref_wg.generate_sequences(data)
|
| 174 |
+
|
| 175 |
+
We decorate the method of the worker with a ``register`` that explicitly defines how the input data should be splitted and dispatch to each worker, and how the output data should be collected and concatenated by the controller. For example, ``Dispatch.DP_COMPUTE_PROTO`` splits the input data into dp chunks, dispatch each data to each worker, collect the output and concatenate the results. Note that this function requires the input and output to be a DataProto defined here (https://github.com/volcengine/verl/blob/main/verl/protocol.py).
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
PPO main loop
|
| 179 |
+
~~~~~~~~~~~~~~~~~~~~
|
| 180 |
+
With the aforementioned APIs, we can implement the main loop of PPO as if it is a single process program
|
| 181 |
+
|
| 182 |
+
.. code-block:: python
|
| 183 |
+
|
| 184 |
+
for prompt in dataloader:
|
| 185 |
+
output = actor_rollout_ref_wg.generate_sequences(prompt)
|
| 186 |
+
old_log_prob = actor_rollout_ref_wg.compute_log_prob(output)
|
| 187 |
+
ref_log_prob = actor_rollout_ref_wg.compute_ref_log_prob(output)
|
| 188 |
+
values = critic_wg.compute_values(output)
|
| 189 |
+
rewards = reward_wg.compute_scores(output)
|
| 190 |
+
# compute_advantages is running directly on the control process
|
| 191 |
+
advantages = compute_advantages(values, rewards)
|
| 192 |
+
output = output.union(old_log_prob)
|
| 193 |
+
output = output.union(ref_log_prob)
|
| 194 |
+
output = output.union(values)
|
| 195 |
+
output = output.union(rewards)
|
| 196 |
+
output = output.union(advantages)
|
| 197 |
+
# update actor
|
| 198 |
+
actor_rollout_ref_wg.update_actor(output)
|
| 199 |
+
critic.update_critic(output)
|
| 200 |
+
|
| 201 |
+
Takeaways
|
| 202 |
+
~~~~~~~~~~~~~~~~~~~~
|
| 203 |
+
- This programming paradigm enables users to use different computation backend without modification of the control process.
|
| 204 |
+
- This programming paradigm enables flexible placement (by changing the mapping of WorkerGroup and ResourcePool) without modification of the control process.
|
| 205 |
+
|
| 206 |
+
Repository organization
|
| 207 |
+
------------------------------------------------
|
| 208 |
+
|
| 209 |
+
Important code files in the repository are organized as below:
|
| 210 |
+
|
| 211 |
+
.. code-block:: bash
|
| 212 |
+
|
| 213 |
+
verl # the verl package
|
| 214 |
+
trainer
|
| 215 |
+
main_ppo.py # the entrypoint for RL training
|
| 216 |
+
ppo
|
| 217 |
+
ray_trainer.py # the training loop for RL algorithms such as PPO
|
| 218 |
+
fsdp_sft_trainer.py # the SFT trainer with FSDP backend
|
| 219 |
+
config
|
| 220 |
+
generation.yaml # configuration template for rollout
|
| 221 |
+
ppo_trainer.yaml # configuration template for the RL trainer
|
| 222 |
+
workers
|
| 223 |
+
protocol.py # the interface of DataProto
|
| 224 |
+
fsdp_workers.py # the FSDP worker interfaces: ActorRolloutRefWorker, CriticWorker, RewardModelWorker
|
| 225 |
+
megatron_workers.py # the Megatron worker interfaces: ActorRolloutRefWorker, CriticWorker, RewardModelWorker
|
| 226 |
+
actor
|
| 227 |
+
dp_actor.py # data parallel actor with FSDP backend
|
| 228 |
+
megatron_actor.py # nD parallel actor with Megatron backend
|
| 229 |
+
critic
|
| 230 |
+
dp_critic.py # data parallel critic with FSDP backend
|
| 231 |
+
megatron_critic.py # nD parallel critic with FSDP backend
|
| 232 |
+
reward_model
|
| 233 |
+
megatron
|
| 234 |
+
reward_model.py # reward model with Megatron backend
|
| 235 |
+
rollout
|
| 236 |
+
vllm
|
| 237 |
+
vllm_rollout.py # rollout with vllm backend
|
| 238 |
+
hf_rollout.py # rollout with huggingface TGI backend
|
| 239 |
+
sharding_manager
|
| 240 |
+
fsdp_ulysses.py # data and model resharding when using FSDP + ulysses
|
| 241 |
+
fsdp_vllm.py # data and model resharding when using FSDP + ulysses + vllm
|
| 242 |
+
megatron_vllm.py # data and model resharding when using Megatron + vllm
|
| 243 |
+
utils
|
| 244 |
+
dataset # datasets for SFT/RM/RL
|
| 245 |
+
reward_score # function based reward
|
| 246 |
+
gsm8k.py # reward function for gsm8k dataset
|
| 247 |
+
math.py # reward function for math dataset
|
| 248 |
+
seqlen_balancing.py # the sequence balance optimization
|
| 249 |
+
models
|
| 250 |
+
llama # Megatron implementation for llama, deepseek, mistral, etc
|
| 251 |
+
transformers # ulysses integration with transformer models such as llama, qwen, etc
|
| 252 |
+
weight_loader_registery.py # registry of weight loaders for loading hf ckpt into Megatron
|
| 253 |
+
third_party
|
| 254 |
+
vllm # adaptor for vllm's usage in RL
|
| 255 |
+
vllm_v_0_6_3 # vllm v0.6.3 adaptor
|
| 256 |
+
llm.py # entrypoints for generate, sync_model_weight, offload_model_weights
|
| 257 |
+
parallel_state.py # vllm related device mesh and process groups
|
| 258 |
+
dtensor_weight_loaders.py # weight loader for huggingface models with FSDP
|
| 259 |
+
megatron_weight_loaders.py # weight loader for Megatron models
|
| 260 |
+
vllm_spmd # vllm >= v0.7 adaptor (coming soon)
|
| 261 |
+
examples # example scripts
|
| 262 |
+
tests # integration and unit tests
|
| 263 |
+
.github # the configuration of continuous integration tests
|
| 264 |
+
|
| 265 |
+
|
| 266 |
+
.. [1] HybridFlow: A Flexible and Efficient RLHF Framework: https://arxiv.org/abs/2409.19256v2
|
| 267 |
+
.. [2] Data flow graph credit to CS231n 2024 lecture 4: https://cs231n.stanford.edu/slides/2024/lecture_4.pdf
|
| 268 |
+
.. [3] PPO dataflow graph credit to 低级炼丹师 from Zhihu: https://zhuanlan.zhihu.com/p/635757674
|
| 269 |
+
.. [4] RLFlow
|
frameworks/agentgym_rl/docs/index.rst
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Welcome to verl's documentation!
|
| 2 |
+
================================================
|
| 3 |
+
|
| 4 |
+
.. _hf_arxiv: https://arxiv.org/pdf/2409.19256
|
| 5 |
+
|
| 6 |
+
verl is a flexible, efficient and production-ready RL training framework designed for large language models (LLMs) post-training. It is an open source implementation of the `HybridFlow <hf_arxiv>`_ paper.
|
| 7 |
+
|
| 8 |
+
verl is flexible and easy to use with:
|
| 9 |
+
|
| 10 |
+
- **Easy extension of diverse RL algorithms**: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code.
|
| 11 |
+
|
| 12 |
+
- **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks.
|
| 13 |
+
|
| 14 |
+
- **Flexible device mapping and parallelism**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.
|
| 15 |
+
|
| 16 |
+
- Readily integration with popular HuggingFace models
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
verl is fast with:
|
| 20 |
+
|
| 21 |
+
- **State-of-the-art throughput**: By seamlessly integrating existing SOTA LLM training and inference frameworks, verl achieves high generation and training throughput.
|
| 22 |
+
|
| 23 |
+
- **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.
|
| 24 |
+
|
| 25 |
+
--------------------------------------------
|
| 26 |
+
|
| 27 |
+
.. _Contents:
|
| 28 |
+
|
| 29 |
+
.. toctree::
|
| 30 |
+
:maxdepth: 5
|
| 31 |
+
:caption: Quickstart
|
| 32 |
+
|
| 33 |
+
start/install
|
| 34 |
+
start/quickstart
|
| 35 |
+
|
| 36 |
+
.. toctree::
|
| 37 |
+
:maxdepth: 4
|
| 38 |
+
:caption: Programming guide
|
| 39 |
+
|
| 40 |
+
hybrid_flow
|
| 41 |
+
|
| 42 |
+
.. toctree::
|
| 43 |
+
:maxdepth: 5
|
| 44 |
+
:caption: Data Preparation
|
| 45 |
+
|
| 46 |
+
preparation/prepare_data
|
| 47 |
+
preparation/reward_function
|
| 48 |
+
|
| 49 |
+
.. toctree::
|
| 50 |
+
:maxdepth: 5
|
| 51 |
+
:caption: Configurations
|
| 52 |
+
|
| 53 |
+
examples/config
|
| 54 |
+
|
| 55 |
+
.. toctree::
|
| 56 |
+
:maxdepth: 2
|
| 57 |
+
:caption: PPO Example
|
| 58 |
+
|
| 59 |
+
examples/ppo_code_architecture
|
| 60 |
+
examples/gsm8k_example
|
| 61 |
+
|
| 62 |
+
.. toctree::
|
| 63 |
+
:maxdepth: 1
|
| 64 |
+
:caption: PPO Trainer and Workers
|
| 65 |
+
|
| 66 |
+
workers/ray_trainer
|
| 67 |
+
workers/fsdp_workers
|
| 68 |
+
workers/megatron_workers
|
| 69 |
+
|
| 70 |
+
.. toctree::
|
| 71 |
+
:maxdepth: 1
|
| 72 |
+
:caption: Performance Tuning Guide
|
| 73 |
+
|
| 74 |
+
perf/perf_tuning
|
| 75 |
+
|
| 76 |
+
.. toctree::
|
| 77 |
+
:maxdepth: 1
|
| 78 |
+
:caption: Experimental Results
|
| 79 |
+
|
| 80 |
+
experiment/ppo
|
| 81 |
+
|
| 82 |
+
.. toctree::
|
| 83 |
+
:maxdepth: 1
|
| 84 |
+
:caption: Advance Usage and Extension
|
| 85 |
+
|
| 86 |
+
advance/placement
|
| 87 |
+
advance/dpo_extension
|
| 88 |
+
advance/fsdp_extension
|
| 89 |
+
advance/megatron_extension
|
| 90 |
+
|
| 91 |
+
.. toctree::
|
| 92 |
+
:maxdepth: 1
|
| 93 |
+
:caption: FAQ
|
| 94 |
+
|
| 95 |
+
faq/faq
|
| 96 |
+
|
| 97 |
+
Contribution
|
| 98 |
+
-------------
|
| 99 |
+
|
| 100 |
+
verl is free software; you can redistribute it and/or modify it under the terms
|
| 101 |
+
of the Apache License 2.0. We welcome contributions.
|
| 102 |
+
Join us on `GitHub <https://github.com/volcengine/verl>`_, `Slack <https://join.slack.com/t/verlgroup/shared_invite/zt-2w5p9o4c3-yy0x2Q56s_VlGLsJ93A6vA>`_ and `Wechat <https://raw.githubusercontent.com/eric-haibin-lin/verl-community/refs/heads/main/WeChat.JPG>`_ for discussions.
|
| 103 |
+
|
| 104 |
+
Code formatting
|
| 105 |
+
^^^^^^^^^^^^^^^^^^^^^^^^
|
| 106 |
+
We use yapf (Google style) to enforce strict code formatting when reviewing MRs. Run yapf at the top level of verl repo:
|
| 107 |
+
|
| 108 |
+
.. code-block:: bash
|
| 109 |
+
|
| 110 |
+
pip3 install yapf
|
| 111 |
+
yapf -ir -vv --style ./.style.yapf verl examples tests
|
frameworks/agentgym_rl/docs/perf/perf_tuning.rst
ADDED
|
@@ -0,0 +1,164 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Performance Tuning Guide
|
| 2 |
+
==============================
|
| 3 |
+
|
| 4 |
+
Author: `Guangming Sheng <https://github.com/PeterSH6>`_
|
| 5 |
+
|
| 6 |
+
In this section, we will discuss how to tune the performance of all the stages in verl, including:
|
| 7 |
+
|
| 8 |
+
1. Rollout generation throughput.
|
| 9 |
+
|
| 10 |
+
2. Enable `use_remove_padding=True` for sequence packing (i.e., data packing and remove padding).
|
| 11 |
+
|
| 12 |
+
3. Batch size tuning for forward and backward computation
|
| 13 |
+
|
| 14 |
+
4. Enable ``use_dynamic_bsz=True`` for higher throughput.
|
| 15 |
+
|
| 16 |
+
5. Utilize Ulysses Sequence Parallel for Long Context Training
|
| 17 |
+
|
| 18 |
+
6. LigerKernel for SFT performance optimization
|
| 19 |
+
|
| 20 |
+
Rollout Generation Tuning
|
| 21 |
+
--------------------------
|
| 22 |
+
|
| 23 |
+
verl currently supports two rollout backends: vLLM and TGI (with SGLang support coming soon).
|
| 24 |
+
|
| 25 |
+
Below are key factors for tuning vLLM-based rollout. Before tuning, we recommend setting ``actor_rollout_ref.rollout.disable_log_stats=False`` so that rollout statistics are logged.
|
| 26 |
+
|
| 27 |
+
- Increase ``gpu_memory_utilization``. The vLLM pre-allocates GPU KVCache by using gpu_memory_utilization% of the remaining memory.
|
| 28 |
+
However, if model parameters and optimizer states are not offloaded, using too high a fraction can lead to OOM.
|
| 29 |
+
A value between 0.5 and 0.7 often strikes a good balance between high throughput and avoiding OOM.
|
| 30 |
+
|
| 31 |
+
- Adjust ``max_num_seqs`` or ``max_num_batched_tokens``.
|
| 32 |
+
If the GPU cache utilization is relatively low in the log, increase ``max_num_seqs`` or ``max_num_batched_tokens``
|
| 33 |
+
can enlarge the effective batch size in the decoding stage, allowing more concurrent requests per batch.
|
| 34 |
+
We recommend setting ``max_num_batched_tokens > 2048`` for higher throughput.
|
| 35 |
+
|
| 36 |
+
- Use a smaller ``tensor_parallel_size``.
|
| 37 |
+
When GPU resources allow, a smaller tensor parallel size spawns more vLLM replicas.
|
| 38 |
+
Data parallelism (DP) can yield higher throughput than tensor parallelism (TP), but also increases KVCache consumption.
|
| 39 |
+
Carefully balance the trade-off between more replicas and higher memory usage.
|
| 40 |
+
Our experient in Sec. 8.4 of `HybridFlow paper <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/gsm8k.py>`_ evaluate this trade-off.
|
| 41 |
+
|
| 42 |
+
More tuning details such as dealing with Preemption and Chunked-prefill
|
| 43 |
+
can be found in `vLLM official tuning guide <https://docs.vllm.ai/en/latest/performance/optimization.html>`_
|
| 44 |
+
|
| 45 |
+
The performance of vllm can be further increased if upgrading from v0.6.3 to v0.7. See https://github.com/volcengine/verl/blob/main/docs/README_vllm0.7.md for details on how to upgrade.
|
| 46 |
+
|
| 47 |
+
Enable remove padding (sequence packing)
|
| 48 |
+
-----------------------------------------
|
| 49 |
+
|
| 50 |
+
Currently, for llama, mistral, gemma1 and qwen based models, users can enable `use_remove_padding=True` to utilize the
|
| 51 |
+
sequence packing implementation provided by transformers library.
|
| 52 |
+
|
| 53 |
+
For other models, transformers library may also support it but we haven't tested it yet.
|
| 54 |
+
Users can add the desired model config to the `test_transformer.py <https://github.com/volcengine/verl/blob/main/tests/model/test_transformer.py#L24>`_ file.
|
| 55 |
+
And test its functionaility by running the following command:
|
| 56 |
+
|
| 57 |
+
.. code-block:: bash
|
| 58 |
+
|
| 59 |
+
pytest -s tests/model/test_transformer.py
|
| 60 |
+
|
| 61 |
+
If the test passes, you can add your desired model into the model `registry.py <https://github.com/volcengine/verl/blob/main/verl/models/registry.py#L24>`_ file.
|
| 62 |
+
Then, you can enjoy the performance boost of sequence packing
|
| 63 |
+
and welcome to PR your tested model to verl!
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
Batch Size Tuning
|
| 67 |
+
-----------------
|
| 68 |
+
|
| 69 |
+
To achieve higher throughput in experience preparation (i.e., model fwd) and model update (i.e., actor/critic fwd/bwd),
|
| 70 |
+
users may need to tune the ``*micro_batch_size_per_gpu`` for different computation.
|
| 71 |
+
|
| 72 |
+
In verl, the core principle for setting batch sizes is:
|
| 73 |
+
|
| 74 |
+
- **Algorithmic metrics** (train batch size, PPO mini-batch size) are *global* (from a single-controller perspective),
|
| 75 |
+
normalized in each worker. See the `normalization code <https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py#L120-L122>`_.
|
| 76 |
+
|
| 77 |
+
- **Performance-related parameters** (micro batch size, max token length for dynamic batch size) are *local* parameters that define the per-GPU data allocations.
|
| 78 |
+
See the `normalization code <https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py#L127>`_.
|
| 79 |
+
|
| 80 |
+
.. note:: In your training script, please use ``*micro_batch_size_per_gpu`` instead of ``*micro_batch_size``.
|
| 81 |
+
So that you don't need to consider the normalization of the ``micro_batch_size`` and ``micro_batch_size`` will be deprecated.
|
| 82 |
+
|
| 83 |
+
Batch Size Tuning tips
|
| 84 |
+
""""""""""""""""""""""
|
| 85 |
+
|
| 86 |
+
Therefore, users may need to tune the ``*micro_batch_size_per_gpu`` to accelerate training. Here're some tips:
|
| 87 |
+
|
| 88 |
+
1. **Enable gradient checkpointing**:
|
| 89 |
+
Set ``actor_rollout_ref.model.enable_gradient_checkpointing=True`` and ``critic.model.enable_gradient_checkpointing=True``.
|
| 90 |
+
This often allows for larger micro-batch sizes and will be beneficial for large mini-batch training.
|
| 91 |
+
|
| 92 |
+
2. Increase the ``*micro_batch_size_per_gpu`` as much as possible till equals to normalized ``mini_batch_size``.
|
| 93 |
+
|
| 94 |
+
3. **Use larger forward-only parameters**:
|
| 95 |
+
Forward only parameter, such as ``actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu``,
|
| 96 |
+
``actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu``, ``critic.forward_micro_batch_size_per_gpu`` could be larger (e.g., 2x) than training related micro batch sizes,
|
| 97 |
+
such as ``actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu``, ``critic.ppo_micro_batch_size_per_gpu``.
|
| 98 |
+
|
| 99 |
+
4. **Allow larger micro-batch sizes for Critic and Reward models**:
|
| 100 |
+
micro batch size of Critic and Reward model could be larger than Actor model. This is because the actor model has much larger vocab size in the final layer.
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
Tuning for Dynamic Batch Size
|
| 104 |
+
-----------------------------
|
| 105 |
+
|
| 106 |
+
Dynamic batch size is a technique that allows the model to process similar number of tokens in a single forward pass (with different actual batch sizes).
|
| 107 |
+
This can significantly improve the training efficiency and reduce the memory usage.
|
| 108 |
+
|
| 109 |
+
To utilize this technique, users can set ``use_dynamic_bsz=True`` in actor, ref, critic and reward models.
|
| 110 |
+
With ``use_dynamic_bsz=True``, users don't need to tune ``*micro_batch_size_per_gpu``.
|
| 111 |
+
Instead, users should tune the following parameters:
|
| 112 |
+
|
| 113 |
+
- ``actor_rollout_ref.actor.ppo_max_token_len_per_gpu``, ``critic.ppo_max_token_len_per_gpu``:
|
| 114 |
+
The maximum number of tokens to be processed in fwd and bwd of ``update_policy`` and ``update_critic``.
|
| 115 |
+
|
| 116 |
+
- ``actor_rollout_ref.ref.log_prob_max_token_len_per_gpu`` and ``actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu``:
|
| 117 |
+
The maximum number of tokens to be processed in a the fwd computation of ``compute_log_prob`` and ``comptue_ref_log_prob``.
|
| 118 |
+
|
| 119 |
+
- ``critic.forward_micro_batch_size_per_gpu``, ``reward_model.forward_micro_batch_size_per_gpu``:
|
| 120 |
+
The maximum number of tokens to be processed in a the fwd computation of ``compute_values``, ``compute_rm_score``.
|
| 121 |
+
|
| 122 |
+
Dynamic Batch Size Tuning tips
|
| 123 |
+
""""""""""""""""""""""""""""""
|
| 124 |
+
|
| 125 |
+
Here're some tips to tune the above parameters:
|
| 126 |
+
|
| 127 |
+
1. **Increase** ``actor_rollout_ref.actor.ppo_max_token_len_per_gpu``
|
| 128 |
+
Make it at least 2 x (max_prompt_length + max_response_length). We set it to 3x in `run_qwen2-7b_rm_seq_balance.sh <https://github.com/volcengine/verl/blob/main/examples/ppo_trainer/run_qwen2-7b_rm_seq_balance.sh#L25>`_.
|
| 129 |
+
Try to increase it to get higher throughput.
|
| 130 |
+
|
| 131 |
+
2. **Forward-only parameters can be larger**:
|
| 132 |
+
Similar to the non-dynamic-batch scenario, forward-only token limits can exceed those used in forward/backward operations.
|
| 133 |
+
|
| 134 |
+
3. **Use larger limits for Critic and Reward models**:
|
| 135 |
+
Critic and Reward parameters can be set at least 2× the Actor’s limits. For instance, we set them to 4× here:
|
| 136 |
+
`run_qwen2-7b_rm_seq_balance.sh <https://github.com/volcengine/verl/blob/main/examples/ppo_trainer/run_qwen2-7b_rm_seq_balance.sh#L40>`_
|
| 137 |
+
|
| 138 |
+
.. :math:`\text{critic.ppo_max_token_len_per_gpu} = 2 \times \text{actor.ppo_max_token_len_per_gpu})`.
|
| 139 |
+
|
| 140 |
+
Ulysses Sequence Parallel for Long Context Training
|
| 141 |
+
----------------------------------------------------
|
| 142 |
+
|
| 143 |
+
To utilize this technique, users can set ``ulysses_sequence_parallel_size>1`` in actor, ref, critic and reward models.
|
| 144 |
+
|
| 145 |
+
We support different model utilize different ulysses_sequence_parallel_size sizes.
|
| 146 |
+
|
| 147 |
+
To train log sequence (>32k), users may need to decrease the ``*micro_batch_size_per_gpu`` and ``*max_token_len_per_gpu`` to avoid OOM.
|
| 148 |
+
|
| 149 |
+
LigerKernel for SFT
|
| 150 |
+
----------------------
|
| 151 |
+
|
| 152 |
+
LigerKernel is a high-performance kernel for Supervised Fine-Tuning (SFT) that can improve training efficiency. To enable LigerKernel in your SFT training:
|
| 153 |
+
|
| 154 |
+
1. Install liger-kernel via ``pip3 install liger-kernel``. In your SFT configuration file (e.g., ``verl/trainer/config/sft_trainer.yaml``), set the ``use_liger`` parameter:
|
| 155 |
+
|
| 156 |
+
.. code-block:: yaml
|
| 157 |
+
|
| 158 |
+
model:
|
| 159 |
+
use_liger: True # Enable LigerKernel for SFT
|
| 160 |
+
|
| 161 |
+
2. The default value is ``False``. Enable it only when you want to use LigerKernel's optimizations.
|
| 162 |
+
|
| 163 |
+
3. LigerKernel is particularly useful for improving training performance in SFT scenarios.
|
| 164 |
+
|
frameworks/agentgym_rl/docs/preparation/prepare_data.rst
ADDED
|
@@ -0,0 +1,126 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Prepare Data for Post-Training
|
| 2 |
+
========================================
|
| 3 |
+
|
| 4 |
+
Before starting the post-training job, we need to prepare the data for
|
| 5 |
+
the policy training. The data should be stored in the parquet format.
|
| 6 |
+
|
| 7 |
+
We provide several data preprocess scripts for different datasets,
|
| 8 |
+
including GSM8K, MATH, HelloSwag, Full_hh_rlhf. To prepare other datasets, we need
|
| 9 |
+
to follow the following steps: The data preprocess script can be divided
|
| 10 |
+
into two parts:
|
| 11 |
+
|
| 12 |
+
1. The first part is the common part, which loads the dataset from
|
| 13 |
+
huggingface's ``datasets`` package. Then preprocess the datasets with
|
| 14 |
+
the ``make_map_fn`` and then store in the parquet format.
|
| 15 |
+
|
| 16 |
+
.. code:: python
|
| 17 |
+
|
| 18 |
+
import re
|
| 19 |
+
import os
|
| 20 |
+
import datasets
|
| 21 |
+
|
| 22 |
+
from verl.utils.hdfs_io import copy, makedirs
|
| 23 |
+
import argparse
|
| 24 |
+
|
| 25 |
+
# To extract the solution for each prompts in the dataset
|
| 26 |
+
# def extract_solution(solution_str):
|
| 27 |
+
# ...
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
if __name__ == '__main__':
|
| 31 |
+
parser = argparse.ArgumentParser()
|
| 32 |
+
parser.add_argument('--local_dir', default='/opt/tiger/gsm8k')
|
| 33 |
+
parser.add_argument('--hdfs_dir', default=None)
|
| 34 |
+
|
| 35 |
+
args = parser.parse_args()
|
| 36 |
+
|
| 37 |
+
num_few_shot = 5
|
| 38 |
+
data_source = 'openai/gsm8k'
|
| 39 |
+
|
| 40 |
+
dataset = datasets.load_dataset(data_source, 'main')
|
| 41 |
+
|
| 42 |
+
train_dataset = dataset['train']
|
| 43 |
+
test_dataset = dataset['test']
|
| 44 |
+
|
| 45 |
+
# Construct a `def make_map_fn(split)` for the corresponding datasets.
|
| 46 |
+
# ...
|
| 47 |
+
|
| 48 |
+
train_dataset = train_dataset.map(function=make_map_fn('train'), with_indices=True)
|
| 49 |
+
test_dataset = test_dataset.map(function=make_map_fn('test'), with_indices=True)
|
| 50 |
+
|
| 51 |
+
local_dir = args.local_dir
|
| 52 |
+
hdfs_dir = args.hdfs_dir
|
| 53 |
+
|
| 54 |
+
train_dataset.to_parquet(os.path.join(local_dir, 'train.parquet'))
|
| 55 |
+
test_dataset.to_parquet(os.path.join(local_dir, 'test.parquet'))
|
| 56 |
+
|
| 57 |
+
makedirs(hdfs_dir)
|
| 58 |
+
|
| 59 |
+
copy(src=local_dir, dst=hdfs_dir)
|
| 60 |
+
|
| 61 |
+
2. The users are required to implement the ``make_map_fn()`` function
|
| 62 |
+
(as well as the ``extract_solution``) on their own to support
|
| 63 |
+
different datasets or tasks.
|
| 64 |
+
|
| 65 |
+
We already implemented the data preprocess of GSM8k, MATH, Hellaswag and Full_hh_rlhf
|
| 66 |
+
datasets. And we take the GSM8k dataset as an example:
|
| 67 |
+
|
| 68 |
+
**GSM8K**
|
| 69 |
+
|
| 70 |
+
In the ``make_map_fn``, each data field should consist of the following
|
| 71 |
+
5 fields:
|
| 72 |
+
|
| 73 |
+
1. ``data_source``: The name of the dataset. To index the corresponding
|
| 74 |
+
reward function in the ``RewardModule``
|
| 75 |
+
2. ``prompt``: This field should be constructed in the format of
|
| 76 |
+
huggingface chat_template. The tokenizer in ``RLHFDataset`` will
|
| 77 |
+
apply chat template and tokenize the prompt.
|
| 78 |
+
3. ``ability``: Define the task category.
|
| 79 |
+
4. ``reward_model``: Currently, we only utilize the ``ground_truth``
|
| 80 |
+
field during evaluation. The ``ground_truth`` is computed by the
|
| 81 |
+
``extract_solution`` function. **NOTED** that the implementation of
|
| 82 |
+
the corresponding reward function should align with this extracted
|
| 83 |
+
``ground_truth``.
|
| 84 |
+
5. ``extra_info``: Record some information of the current prompt. Not
|
| 85 |
+
use for now.
|
| 86 |
+
|
| 87 |
+
.. code:: python
|
| 88 |
+
|
| 89 |
+
def extract_solution(solution_str):
|
| 90 |
+
solution = re.search("#### (\\-?[0-9\\.\\,]+)", solution_str) # extract the solution after ####
|
| 91 |
+
assert solution is not None
|
| 92 |
+
final_solution = solution.group(0)
|
| 93 |
+
final_solution = final_solution.split('#### ')[1].replace(',', '')
|
| 94 |
+
return final_solution
|
| 95 |
+
|
| 96 |
+
instruction_following = "Let's think step by step and output the final answer after \"####\"."
|
| 97 |
+
|
| 98 |
+
# add a row to each data item that represents a unique id
|
| 99 |
+
def make_map_fn(split):
|
| 100 |
+
|
| 101 |
+
def process_fn(example, idx):
|
| 102 |
+
question = example.pop('question')
|
| 103 |
+
|
| 104 |
+
question = question + ' ' + instruction_following
|
| 105 |
+
|
| 106 |
+
answer = example.pop('answer')
|
| 107 |
+
solution = extract_solution(answer)
|
| 108 |
+
data = {
|
| 109 |
+
"data_source": data_source,
|
| 110 |
+
"prompt": [{
|
| 111 |
+
"role": "user",
|
| 112 |
+
"content": question
|
| 113 |
+
}],
|
| 114 |
+
"ability": "math",
|
| 115 |
+
"reward_model": {
|
| 116 |
+
"style": "rule",
|
| 117 |
+
"ground_truth": solution
|
| 118 |
+
},
|
| 119 |
+
"extra_info": {
|
| 120 |
+
'split': split,
|
| 121 |
+
'index': idx
|
| 122 |
+
}
|
| 123 |
+
}
|
| 124 |
+
return data
|
| 125 |
+
|
| 126 |
+
return process_fn
|
frameworks/agentgym_rl/docs/preparation/reward_function.rst
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Implement Reward Function for Dataset
|
| 2 |
+
======================================
|
| 3 |
+
|
| 4 |
+
For each dataset, we need to implement a reward function or utilize a reward model to compute the rewards for the generated responses.
|
| 5 |
+
We already pre-implemented some reward functions in `reward_score directory <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score>`_.
|
| 6 |
+
|
| 7 |
+
Currently, we support reward functions for GSM8k and MATH datasets. For RLHF datasets (e.g.,
|
| 8 |
+
full_hh_rlhf) and Code Generation (e.g., APPS), we utilize reward model
|
| 9 |
+
and SandBox (will opensource soon) for evaluation respectively.
|
| 10 |
+
|
| 11 |
+
RewardManager
|
| 12 |
+
-------------
|
| 13 |
+
|
| 14 |
+
In the entrypoint of the PPO Post-Training script `main_ppo.py <https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py#L33>`_,
|
| 15 |
+
we implement a ``RewardManager`` that utilze pre-implemented reward functions to compute the scores for each response.
|
| 16 |
+
|
| 17 |
+
In the ``RewardManager``, we implemented a ``__call__`` function to
|
| 18 |
+
compute the score for each response.
|
| 19 |
+
All the reward functions are executed by ``compute_score_fn``.
|
| 20 |
+
The input is a ``DataProto``, which includes:
|
| 21 |
+
|
| 22 |
+
- ``input_ids``, ``attention_mask``: ``input_ids`` and ``attention_mask`` after applying
|
| 23 |
+
chat_template, including prompt and response
|
| 24 |
+
- ``responses``: response tokens
|
| 25 |
+
- ``ground_truth``: The ground truth string of the current prompt.
|
| 26 |
+
Stored in ``non_tensor_batch`` in the ``DataProto``, which should be
|
| 27 |
+
preprocessed in the parquet files.
|
| 28 |
+
- ``data_source``: The dataset name of the current prompt. Stored in
|
| 29 |
+
``non_tensor_batch`` in the ``DataProto``, which should be
|
| 30 |
+
preprocessed in the parquet files.
|
| 31 |
+
|
| 32 |
+
After detokenize the responses, the responses string and the ground
|
| 33 |
+
truth string will be input to the ``compute_score_fn`` to compute the
|
| 34 |
+
score for each response.
|
| 35 |
+
|
| 36 |
+
Reward Functions
|
| 37 |
+
----------------
|
| 38 |
+
We already pre-implemented some reward functions in `reward_score directory <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score>`_.
|
| 39 |
+
|
| 40 |
+
- In the `GSM8k example <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/gsm8k.py>`_, we
|
| 41 |
+
force the response to output the final answer after four ####, then
|
| 42 |
+
use string matching to compare with the ground truth. If completely
|
| 43 |
+
correct, score 1 point; if the format is correct, score 0.1 points; if
|
| 44 |
+
the format is incorrect, score 0 points.
|
| 45 |
+
- In the `MATH example <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/math.py>`_, we follow
|
| 46 |
+
the implementation in `lm-evaluation-harness repository <https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/hendrycks_math/utils.py>`_.
|
frameworks/agentgym_rl/docs/requirements-docs.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# markdown suport
|
| 2 |
+
recommonmark
|
| 3 |
+
# markdown table suport
|
| 4 |
+
sphinx-markdown-tables
|
| 5 |
+
|
| 6 |
+
# theme default rtd
|
| 7 |
+
|
| 8 |
+
# crate-docs-theme
|
| 9 |
+
sphinx-rtd-theme
|
frameworks/agentgym_rl/docs/start/install.rst
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Installation
|
| 2 |
+
============
|
| 3 |
+
|
| 4 |
+
Requirements
|
| 5 |
+
------------
|
| 6 |
+
|
| 7 |
+
- **Python**: Version >= 3.9
|
| 8 |
+
- **CUDA**: Version >= 12.1
|
| 9 |
+
|
| 10 |
+
verl supports various backends. Currently, the following configurations are available:
|
| 11 |
+
|
| 12 |
+
- **FSDP** and **Megatron-LM** (optional) for training.
|
| 13 |
+
- **vLLM** adn **TGI** for rollout generation, **SGLang** support coming soon.
|
| 14 |
+
|
| 15 |
+
Training backends
|
| 16 |
+
------------------
|
| 17 |
+
|
| 18 |
+
We recommend using **FSDP** backend to investigate, research and prototype different models, datasets and RL algorithms. The guide for using FSDP backend can be found in :doc:`FSDP Workers<../workers/fsdp_workers>`.
|
| 19 |
+
|
| 20 |
+
For users who pursue better scalability, we recommend using **Megatron-LM** backend. Currently, we support Megatron-LM v0.4 [1]_. The guide for using Megatron-LM backend can be found in :doc:`Megatron-LM Workers<../workers/megatron_workers>`.
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
Install from docker image
|
| 24 |
+
-------------------------
|
| 25 |
+
|
| 26 |
+
We provide pre-built Docker images for quick setup.
|
| 27 |
+
|
| 28 |
+
Image and tag: ``verlai/verl:vemlp-th2.4.0-cu124-vllm0.6.3-ray2.10-te1.7-v0.0.3``. See files under ``docker/`` for NGC-based image or if you want to build your own.
|
| 29 |
+
|
| 30 |
+
1. Launch the desired Docker image:
|
| 31 |
+
|
| 32 |
+
.. code:: bash
|
| 33 |
+
|
| 34 |
+
docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN -v <image:tag>
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
2. Inside the container, install verl:
|
| 38 |
+
|
| 39 |
+
.. code:: bash
|
| 40 |
+
|
| 41 |
+
# install the stable version
|
| 42 |
+
pip3 install verl
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
3. Setup Megatron (optional)
|
| 46 |
+
|
| 47 |
+
If you want to enable training with Megatron, Megatron code must be added to PYTHONPATH:
|
| 48 |
+
|
| 49 |
+
.. code:: bash
|
| 50 |
+
|
| 51 |
+
cd ..
|
| 52 |
+
git clone -b core_v0.4.0 https://github.com/NVIDIA/Megatron-LM.git
|
| 53 |
+
cp verl/patches/megatron_v4.patch Megatron-LM/
|
| 54 |
+
cd Megatron-LM && git apply megatron_v4.patch
|
| 55 |
+
pip3 install -e .
|
| 56 |
+
export PYTHONPATH=$PYTHONPATH:$(pwd)
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
You can also get the Megatron code after verl's patch via
|
| 60 |
+
|
| 61 |
+
.. code:: bash
|
| 62 |
+
|
| 63 |
+
git clone -b core_v0.4.0_verl https://github.com/eric-haibin-lin/Megatron-LM
|
| 64 |
+
export PYTHONPATH=$PYTHONPATH:$(pwd)/Megatron-LM
|
| 65 |
+
|
| 66 |
+
Install from custom environment
|
| 67 |
+
---------------------------------
|
| 68 |
+
|
| 69 |
+
To manage environment, we recommend using conda:
|
| 70 |
+
|
| 71 |
+
.. code:: bash
|
| 72 |
+
|
| 73 |
+
conda create -n verl python==3.9
|
| 74 |
+
conda activate verl
|
| 75 |
+
|
| 76 |
+
For installing the latest version of verl, the best way is to clone and
|
| 77 |
+
install it from source. Then you can modify our code to customize your
|
| 78 |
+
own post-training jobs.
|
| 79 |
+
|
| 80 |
+
.. code:: bash
|
| 81 |
+
|
| 82 |
+
# install verl together with some lightweight dependencies in setup.py
|
| 83 |
+
pip3 install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124
|
| 84 |
+
pip3 install flash-attn --no-build-isolation
|
| 85 |
+
pip3 install verl
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
Megatron is optional. It's dependencies can be setup as below:
|
| 89 |
+
|
| 90 |
+
.. code:: bash
|
| 91 |
+
|
| 92 |
+
# apex
|
| 93 |
+
pip3 install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" \
|
| 94 |
+
git+https://github.com/NVIDIA/apex
|
| 95 |
+
|
| 96 |
+
# transformer engine
|
| 97 |
+
pip3 install git+https://github.com/NVIDIA/TransformerEngine.git@v1.7
|
| 98 |
+
|
| 99 |
+
# megatron core v0.4.0: clone and apply the patch
|
| 100 |
+
# You can also get the patched Megatron code patch via
|
| 101 |
+
# git clone -b core_v0.4.0_verl https://github.com/eric-haibin-lin/Megatron-LM
|
| 102 |
+
cd ..
|
| 103 |
+
git clone -b core_v0.4.0 https://github.com/NVIDIA/Megatron-LM.git
|
| 104 |
+
cd Megatron-LM
|
| 105 |
+
cp ../verl/patches/megatron_v4.patch .
|
| 106 |
+
git apply megatron_v4.patch
|
| 107 |
+
pip3 install -e .
|
| 108 |
+
export PYTHONPATH=$PYTHONPATH:$(pwd)
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
.. [1] Megatron v0.4 is supported with verl's patches to fix issues such as virtual pipeline hang. It will be soon updated with latest the version of upstream Megatron-LM without patches.
|
frameworks/agentgym_rl/docs/start/quickstart.rst
ADDED
|
@@ -0,0 +1,142 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.. _quickstart:
|
| 2 |
+
|
| 3 |
+
=========================================================
|
| 4 |
+
Quickstart: PPO training on GSM8K dataset
|
| 5 |
+
=========================================================
|
| 6 |
+
|
| 7 |
+
Post-train a LLM using GSM8K dataset.
|
| 8 |
+
|
| 9 |
+
Introduction
|
| 10 |
+
------------
|
| 11 |
+
|
| 12 |
+
.. _hf_dataset_gsm8k: https://huggingface.co/datasets/gsm8k
|
| 13 |
+
|
| 14 |
+
In this example, we train an LLM to tackle the `GSM8k <hf_dataset_gsm8k>`_ task with function-based rewards. [1]_
|
| 15 |
+
|
| 16 |
+
Prerequisite:
|
| 17 |
+
|
| 18 |
+
- the latest version of ``verl`` and its dependencies installed following the installation guide. Using the docker image is recommended.
|
| 19 |
+
|
| 20 |
+
- an GPU with at least 24 GB HBM
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
Dataset Introduction
|
| 24 |
+
--------------------
|
| 25 |
+
|
| 26 |
+
GSM8k is a math problem dataset. The prompt is an elementary school
|
| 27 |
+
problem. The LLM model is asked to solve the math problem. Below is an example:
|
| 28 |
+
|
| 29 |
+
Prompt
|
| 30 |
+
|
| 31 |
+
Katy makes coffee using teaspoons of sugar and cups of water in the
|
| 32 |
+
ratio of 7:13. If she used a total of 120 teaspoons of sugar and cups
|
| 33 |
+
of water, calculate the number of teaspoonfuls of sugar she used.
|
| 34 |
+
|
| 35 |
+
Solution
|
| 36 |
+
|
| 37 |
+
The total ratio representing the ingredients she used to make the
|
| 38 |
+
coffee is 7+13 = <<7+13=20>>20 Since the fraction representing the
|
| 39 |
+
number of teaspoons she used is 7/20, she used 7/20\ *120 =
|
| 40 |
+
<<7/20*\ 120=42>>42 #### 42
|
| 41 |
+
|
| 42 |
+
Step 1: Prepare the dataset
|
| 43 |
+
----------------------------
|
| 44 |
+
|
| 45 |
+
We preprocess the dataset in parquet format so that (1) it contains necessary fields for computing RL rewards and (2) is faster to read.
|
| 46 |
+
|
| 47 |
+
.. code-block:: bash
|
| 48 |
+
|
| 49 |
+
python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k
|
| 50 |
+
|
| 51 |
+
Step 2: Download a model for post-training
|
| 52 |
+
-------------------------------------------
|
| 53 |
+
|
| 54 |
+
In this example, we start with the ``Qwen2.5-0.5B-Instruct`` model.
|
| 55 |
+
|
| 56 |
+
If you want to perform SFT before RL, refer to the :doc:`Complete GSM8K Example<../examples/gsm8k_example>`, the `sft directory <https://github.com/volcengine/verl/blob/main/examples/sft/gsm8k>`_ and `SFT Trainer <https://github.com/volcengine/verl/blob/main/verl/trainer/fsdp_sft_trainer.py>`_ for further details.
|
| 57 |
+
|
| 58 |
+
.. code-block:: bash
|
| 59 |
+
|
| 60 |
+
python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen2.5-0.5B-Instruct')"
|
| 61 |
+
|
| 62 |
+
Step 3: Perform PPO training with the instruct model
|
| 63 |
+
----------------------------------------------------------------------
|
| 64 |
+
|
| 65 |
+
**Reward Model/Function**
|
| 66 |
+
|
| 67 |
+
We use a pre-defined rule-based reward model. We force the model to produce a final
|
| 68 |
+
answer following 4 “#” as shown in the solution. We extract the final
|
| 69 |
+
answer from both the solution and model's output using regular
|
| 70 |
+
expression matching. We assign a reward of 1 to correct
|
| 71 |
+
answer, 0.1 to incorrect answer and 0 to no answer.
|
| 72 |
+
|
| 73 |
+
For mode details, please refer to `verl/utils/reward_score/gsm8k.py <https://github.com/volcengine/verl/blob/v0.1/verl/utils/reward_score/gsm8k.py>`_.
|
| 74 |
+
|
| 75 |
+
**Training Script**
|
| 76 |
+
|
| 77 |
+
Now let's run PPO training with the dataset and model above. [2]_
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
Set the ``data.train_files`` ,\ ``data.val_files``, ``actor_rollout_ref.model.path`` and ``critic.model.path`` based on your dataset and model names or paths.
|
| 81 |
+
|
| 82 |
+
.. code-block:: bash
|
| 83 |
+
|
| 84 |
+
PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
|
| 85 |
+
data.train_files=$HOME/data/gsm8k/train.parquet \
|
| 86 |
+
data.val_files=$HOME/data/gsm8k/test.parquet \
|
| 87 |
+
data.train_batch_size=256 \
|
| 88 |
+
data.val_batch_size=1312 \
|
| 89 |
+
data.max_prompt_length=512 \
|
| 90 |
+
data.max_response_length=256 \
|
| 91 |
+
actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
|
| 92 |
+
actor_rollout_ref.actor.optim.lr=1e-6 \
|
| 93 |
+
actor_rollout_ref.actor.ppo_mini_batch_size=64 \
|
| 94 |
+
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
|
| 95 |
+
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
|
| 96 |
+
actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
|
| 97 |
+
actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
|
| 98 |
+
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
|
| 99 |
+
critic.optim.lr=1e-5 \
|
| 100 |
+
critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
|
| 101 |
+
critic.ppo_micro_batch_size_per_gpu=4 \
|
| 102 |
+
algorithm.kl_ctrl.kl_coef=0.001 \
|
| 103 |
+
trainer.logger=['console'] \
|
| 104 |
+
+trainer.val_before_train=False \
|
| 105 |
+
trainer.default_hdfs_dir=null \
|
| 106 |
+
trainer.n_gpus_per_node=1 \
|
| 107 |
+
trainer.nnodes=1 \
|
| 108 |
+
trainer.save_freq=10 \
|
| 109 |
+
trainer.test_freq=10 \
|
| 110 |
+
trainer.total_epochs=15 2>&1 | tee verl_demo.log
|
| 111 |
+
|
| 112 |
+
You are expected to see the following logs, indicating training in progress. The key metric ``val/test_score/openai/gsm8k`` is computed every ``trainer.test_freq`` steps:
|
| 113 |
+
|
| 114 |
+
.. code-block:: bash
|
| 115 |
+
|
| 116 |
+
step:0 - timing/gen:21.470 - timing/ref:4.360 - timing/values:5.800 - critic/kl:0.000 - critic/kl_coeff:0.001 - timing/adv:0.109 - timing/update_critic:15.664 - critic/vf_loss:14.947 - critic/vf_clipfrac:0.000 - critic/vpred_mean:-2.056 - critic/grad_norm:1023.278 - critic/lr(1e-4):0.100 - timing/update_actor:20.314 - actor/entropy_loss:0.433 - actor/pg_loss:-0.005 - actor/pg_clipfrac:0.000 - actor/ppo_kl:0.000 - actor/grad_norm:1.992 - actor/lr(1e-4):0.010 - critic/score/mean:0.004 - critic/score/max:1.000 - critic/score/min:0.000 - critic/rewards/mean:0.004 - critic/rewards/max:1.000 - critic/rewards/min:0.000 - critic/advantages/mean:-0.000 - critic/advantages/max:2.360 - critic/advantages/min:-2.280 - critic/returns/mean:0.003 - critic/returns/max:0.000 - critic/returns/min:0.000 - critic/values/mean:-2.045 - critic/values/max:9.500 - critic/values/min:-14.000 - response_length/mean:239.133 - response_length/max:256.000 - response_length/min:77.000 - prompt_length/mean:104.883 - prompt_length/max:175.000 - prompt_length/min:68.000
|
| 117 |
+
step:1 - timing/gen:23.020 - timing/ref:4.322 - timing/values:5.953 - critic/kl:0.000 - critic/kl_coeff:0.001 - timing/adv:0.118 - timing/update_critic:15.646 - critic/vf_loss:18.472 - critic/vf_clipfrac:0.384 - critic/vpred_mean:1.038 - critic/grad_norm:942.924 - critic/lr(1e-4):0.100 - timing/update_actor:20.526 - actor/entropy_loss:0.440 - actor/pg_loss:0.000 - actor/pg_clipfrac:0.002 - actor/ppo_kl:0.000 - actor/grad_norm:2.060 - actor/lr(1e-4):0.010 - critic/score/mean:0.000 - critic/score/max:0.000 - critic/score/min:0.000 - critic/rewards/mean:0.000 - critic/rewards/max:0.000 - critic/rewards/min:0.000 - critic/advantages/mean:0.000 - critic/advantages/max:2.702 - critic/advantages/min:-2.616 - critic/returns/mean:0.000 - critic/returns/max:0.000 - critic/returns/min:0.000 - critic/values/mean:-2.280 - critic/values/max:11.000 - critic/values/min:-16.000 - response_length/mean:232.242 - response_length/max:256.000 - response_length/min:91.000 - prompt_length/mean:102.398 - prompt_length/max:185.000 - prompt_length/min:70.000
|
| 118 |
+
|
| 119 |
+
Checkout :ref:`algo-baseline-page` for full training and validation logs for reference.
|
| 120 |
+
|
| 121 |
+
The checkpoint is saved at the following dir by default: ``checkpoints/${trainer.project_name}/${trainer.experiment_name}``
|
| 122 |
+
|
| 123 |
+
To enable ``wandb`` for experiment tracking, set the following configs:
|
| 124 |
+
|
| 125 |
+
.. code-block:: bash
|
| 126 |
+
|
| 127 |
+
trainer.logger=['console','wandb'] \
|
| 128 |
+
trainer.project_name=$YOUR_PROJECT_NAME \
|
| 129 |
+
trainer.experiment_name=$YOUR_RUN_NAME \
|
| 130 |
+
|
| 131 |
+
If you encounter out of memory issues with HBM less than 32GB, enable the following configs would help:
|
| 132 |
+
|
| 133 |
+
.. code-block:: bash
|
| 134 |
+
|
| 135 |
+
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \
|
| 136 |
+
critic.ppo_micro_batch_size_per_gpu=1 \
|
| 137 |
+
|
| 138 |
+
For the full set of configs, please refer to :ref:`config-explain-page` for detailed explanation and performance tuning.
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
.. [1] The original paper (https://arxiv.org/pdf/2110.14168) mainly focuses on training a verifier (a reward model) to solve math problems via Best-of-N sampling. In this example, we train an RL agent using a rule-based reward model.
|
| 142 |
+
.. [2] More training script examples for FSDP and Megatron-LM backend are stored in `examples/ppo_trainer <https://github.com/volcengine/verl/tree/main/examples/ppo_trainer>`_ directory.
|
frameworks/agentgym_rl/docs/workers/fsdp_workers.rst
ADDED
|
@@ -0,0 +1,142 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
PyTorch FSDP Backend
|
| 2 |
+
======================
|
| 3 |
+
|
| 4 |
+
We support PyTorch FSDP Backend by implementing various workers for
|
| 5 |
+
actor, critic, reference, rollout and reward models. We also implement
|
| 6 |
+
the ``FSDPVLLMShardingManager`` that reshard weight between FSDP and
|
| 7 |
+
vLLM in `fsdp_vllm.py <https://github.com/volcengine/verl/blob/main/verl/workers/sharding_manager/fsdp_vllm.py>`_.
|
| 8 |
+
|
| 9 |
+
**Pros**
|
| 10 |
+
|
| 11 |
+
- Readily support various models.
|
| 12 |
+
|
| 13 |
+
- Users only need to implement the corresponding
|
| 14 |
+
``dtensor_weight_loader`` for weight synchronization between FSDP
|
| 15 |
+
and vLLM. While for ``hf_weight_loader``, users can directly apply
|
| 16 |
+
any models supported both in HF and vLLM without any code change.
|
| 17 |
+
|
| 18 |
+
- Easy to organize the forward and backward computation for each model.
|
| 19 |
+
|
| 20 |
+
**Cons**
|
| 21 |
+
|
| 22 |
+
- Poor scalability when it comes to large-scale models (e.g. Llama 70B
|
| 23 |
+
and 405B)
|
| 24 |
+
- The resharding overhead between actor and rollout could be larger than
|
| 25 |
+
Megatron-LM backend.
|
| 26 |
+
|
| 27 |
+
Due to the simplicity, we recommend using FSDP backend for algorithm
|
| 28 |
+
research and prototyping.
|
| 29 |
+
|
| 30 |
+
FSDP Workers
|
| 31 |
+
--------------
|
| 32 |
+
|
| 33 |
+
ActorRolloutRefWorker
|
| 34 |
+
^^^^^^^^^^^^^^^^^^^^^
|
| 35 |
+
|
| 36 |
+
Actor/Rollout HybridEngine
|
| 37 |
+
''''''''''''''''''''''''''
|
| 38 |
+
|
| 39 |
+
1. HybridEngine, Actor and Rollout initialization API.
|
| 40 |
+
|
| 41 |
+
.. code:: python
|
| 42 |
+
|
| 43 |
+
@register(dispatch_mode=Dispatch.ONE_TO_ALL)
|
| 44 |
+
def init_model(self):
|
| 45 |
+
|
| 46 |
+
``ONE_TO_ALL``: when calling the ``init_model`` function from the driver
|
| 47 |
+
process, each worker (on a GPU) will execute the following model
|
| 48 |
+
initialization process.
|
| 49 |
+
|
| 50 |
+
The initialization details of HybridEngine, Actor and Rollout are
|
| 51 |
+
highlighted below:
|
| 52 |
+
|
| 53 |
+
1. ``DataParallelPPOActor`` implements the simple PPO computation logics
|
| 54 |
+
when the model is built with FSDP, including compute log prob, model
|
| 55 |
+
update.
|
| 56 |
+
2. ``vLLMRollout`` support generation with vLLM. We modify the vLLM
|
| 57 |
+
Engine and make it executed under SPMD to fit into our
|
| 58 |
+
``WorkerGroup`` design.
|
| 59 |
+
3. ``FSDPVLLMShardingManager`` a context manager to perform actual
|
| 60 |
+
resharding between actor and rollout.
|
| 61 |
+
|
| 62 |
+
See `source code <https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py>`_. for more information.
|
| 63 |
+
|
| 64 |
+
1. Generate sequence and recompute log prob
|
| 65 |
+
|
| 66 |
+
.. code:: python
|
| 67 |
+
|
| 68 |
+
@register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
|
| 69 |
+
def generate_sequences(self, prompts: DataProto):
|
| 70 |
+
|
| 71 |
+
- ``Dispatch.DP_COMPUTE_PROTO``: The data will be dispatched and
|
| 72 |
+
collected along the DP dimension
|
| 73 |
+
|
| 74 |
+
- In this function, the rollout model will perform auto-regressive
|
| 75 |
+
generation and the actor model will recompute the old log prob for the
|
| 76 |
+
generated response.
|
| 77 |
+
|
| 78 |
+
3. Update actor model
|
| 79 |
+
|
| 80 |
+
.. code:: python
|
| 81 |
+
|
| 82 |
+
@register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
|
| 83 |
+
def update_actor(self, data: DataProto):
|
| 84 |
+
|
| 85 |
+
- Update the actor model weight using PPO & entropy loss.
|
| 86 |
+
|
| 87 |
+
ReferenceModel
|
| 88 |
+
''''''''''''''
|
| 89 |
+
|
| 90 |
+
1. Reference model initialization
|
| 91 |
+
|
| 92 |
+
The reference model is initialized using the same function as the actor
|
| 93 |
+
model without initializing the HybridEngine and Optimizer. Then the
|
| 94 |
+
actor model is also wrapped by the ``DataParallelPPOActor``.
|
| 95 |
+
|
| 96 |
+
2. Compute reference log prob
|
| 97 |
+
|
| 98 |
+
.. code:: python
|
| 99 |
+
|
| 100 |
+
@register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
|
| 101 |
+
def compute_ref_log_prob(self, data: DataProto):
|
| 102 |
+
|
| 103 |
+
- In this function, the reference model will call the compute log prob
|
| 104 |
+
function in ``DataParallelPPOActor`` to compute the reference log
|
| 105 |
+
prob.
|
| 106 |
+
|
| 107 |
+
CriticWorker and RewardWorker
|
| 108 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 109 |
+
|
| 110 |
+
1. Model initialization
|
| 111 |
+
|
| 112 |
+
Quite similar to reference model. The CriticWorker will perform
|
| 113 |
+
additional initialization for the Optimizer.
|
| 114 |
+
|
| 115 |
+
2. Compute Values for CriticWorker
|
| 116 |
+
|
| 117 |
+
.. code:: python
|
| 118 |
+
|
| 119 |
+
@register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
|
| 120 |
+
def compute_values(self, data: DataProto):
|
| 121 |
+
|
| 122 |
+
3. Update Critic
|
| 123 |
+
|
| 124 |
+
.. code:: python
|
| 125 |
+
|
| 126 |
+
@register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
|
| 127 |
+
def update_critic(self, data: DataProto):
|
| 128 |
+
|
| 129 |
+
4. Compute Reward
|
| 130 |
+
|
| 131 |
+
.. code:: python
|
| 132 |
+
|
| 133 |
+
@register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
|
| 134 |
+
def compute_rm_score(self, data: DataProto):
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
HybridShard
|
| 138 |
+
------------
|
| 139 |
+
|
| 140 |
+
We didn't support FSDP `HybridShard`. To support this, we may need to
|
| 141 |
+
construct a 2D device mesh and test the corresponding
|
| 142 |
+
``dtensor_weight_loader`` and ``hf_weight_loader`` for each model.
|
frameworks/agentgym_rl/docs/workers/megatron_workers.rst
ADDED
|
@@ -0,0 +1,200 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Megatron-LM Backend
|
| 2 |
+
=====================
|
| 3 |
+
|
| 4 |
+
We support Megatron Backend by implementing various workers for actor,
|
| 5 |
+
critic, reference, rollout and reward models. We also implement the
|
| 6 |
+
``3DHybridEngine`` using Megatron-LM and vLLM in `megatron_vllm.py <https://github.com/volcengine/verl/blob/main/verl/workers/sharding_manager/megatron_vllm.py>`_.
|
| 7 |
+
|
| 8 |
+
**Pros**
|
| 9 |
+
|
| 10 |
+
- Support 3D parallelism and sequence parallelism for best scalablility
|
| 11 |
+
and throughput.
|
| 12 |
+
- 3D HybridEngine can significantly reduce peak memory usage and reduce
|
| 13 |
+
weight synchronize overhead between actor and rollout.
|
| 14 |
+
|
| 15 |
+
**Cons**
|
| 16 |
+
|
| 17 |
+
- Users should implement their own models for Megatron-LM
|
| 18 |
+
- Users should implement the corresponding weight_loader to
|
| 19 |
+
|
| 20 |
+
- synchronize the model weight between actor (in Megatron) and rollout
|
| 21 |
+
(in vLLM).
|
| 22 |
+
- load weights from checkpoints to corresponding model in Megatron-LM
|
| 23 |
+
|
| 24 |
+
Megatron Workers
|
| 25 |
+
----------------
|
| 26 |
+
|
| 27 |
+
MegatronWorker
|
| 28 |
+
^^^^^^^^^^^^^^
|
| 29 |
+
|
| 30 |
+
``MegatronWorker`` is the base class of different megatron worker
|
| 31 |
+
classes. In this class, ``get_megatron_global_info`` and
|
| 32 |
+
``get_megatron_rank_info`` function to retrive the 3D parallel world
|
| 33 |
+
size and rank of each ``Worker`` running on specific GPU. These information
|
| 34 |
+
will be used in transfer protocol for Megatron Backend.
|
| 35 |
+
|
| 36 |
+
The following ``Worker`` class for different models will be utilized to
|
| 37 |
+
construct the ``WorkerGroup`` .
|
| 38 |
+
|
| 39 |
+
We implement various of APIs for each ``Worker`` class decorated by the
|
| 40 |
+
``@register(dispatch_mode=)`` . These APIs can be called by the ray
|
| 41 |
+
driver process. The data can be correctly collect and dispatch following
|
| 42 |
+
the ``dispatch_mode`` on each function. The supported dispatch_model
|
| 43 |
+
(i.e., transfer protocols) can be found in `decorator.py <https://github.com/volcengine/verl/blob/main/verl/single_controller/base/decorator.py>`_.
|
| 44 |
+
|
| 45 |
+
ActorRolloutRefWorker
|
| 46 |
+
^^^^^^^^^^^^^^^^^^^^^
|
| 47 |
+
|
| 48 |
+
This class is implemented for Actor/Rollout HybridEngine or for the
|
| 49 |
+
reference model to initialize their model and perform computation.
|
| 50 |
+
|
| 51 |
+
Actor/Rollout HybridEngine
|
| 52 |
+
''''''''''''''''''''''''''
|
| 53 |
+
|
| 54 |
+
1. HybridEngine, Actor and Rollout initialization API.
|
| 55 |
+
|
| 56 |
+
.. code:: python
|
| 57 |
+
|
| 58 |
+
@register(dispatch_mode=Dispatch.ONE_TO_ALL)
|
| 59 |
+
def init_model(self):
|
| 60 |
+
|
| 61 |
+
``ONE_TO_ALL``: when calling the ``init_model`` function from the driver
|
| 62 |
+
process, each worker (on a GPU) will execute the following model
|
| 63 |
+
initialization process.
|
| 64 |
+
|
| 65 |
+
The initialization details of HybridEngine, Actor and Rollout are
|
| 66 |
+
highlighted below:
|
| 67 |
+
|
| 68 |
+
1. ``AllGatherPPModel`` holds memory buffer for both Actor and Rollout
|
| 69 |
+
and support weight resharding between actor and rollout.
|
| 70 |
+
2. ``MegatronPPOActor`` implements the simple PPO computation logics
|
| 71 |
+
when the model is built with Megatron, including compute log prob,
|
| 72 |
+
model update.
|
| 73 |
+
3. ``vLLMRollout`` support generation with vLLM. We modify the vLLM
|
| 74 |
+
Engine and make it executed under SPMD to fit into our
|
| 75 |
+
``WorkerGroup`` design.
|
| 76 |
+
4. ``MegatronVLLMShardingManager`` a context manager to perform actual
|
| 77 |
+
resharding between actor and rollout.
|
| 78 |
+
|
| 79 |
+
See `source code <https://github.com/volcengine/verl/blob/main/verl/workers/megatron_workers.py#L63>`_ for more information.
|
| 80 |
+
|
| 81 |
+
.. code:: python
|
| 82 |
+
|
| 83 |
+
# Initialize the 3D HybridEngine
|
| 84 |
+
hybrid_engine = AllGatherPPModel(model_provider=megatron_actor_model_provider)
|
| 85 |
+
# Fetch the model at current rank
|
| 86 |
+
actor_module = hybrid_engine.this_rank_models
|
| 87 |
+
...
|
| 88 |
+
|
| 89 |
+
# build actor model
|
| 90 |
+
self.actor = MegatronPPOActor(config=self.config.actor,
|
| 91 |
+
model_config=self.actor_model_config,
|
| 92 |
+
megatron_config=megatron_config,
|
| 93 |
+
actor_module=self.actor_module,
|
| 94 |
+
actor_optimizer=self.actor_optimizer,
|
| 95 |
+
actor_optimizer_config=self.actor_optim_config)
|
| 96 |
+
|
| 97 |
+
# build rollout
|
| 98 |
+
# rollout initialization
|
| 99 |
+
rollout = vLLMRollout(actor_module=params,
|
| 100 |
+
config=self.config.rollout,
|
| 101 |
+
tokenizer=self.tokenizer,
|
| 102 |
+
model_hf_config=self.actor_model_config,
|
| 103 |
+
train_tp=mpu.get_tensor_model_parallel_world_size())
|
| 104 |
+
# perform weight resharding between actor and rollout
|
| 105 |
+
sharding_manager = MegatronVLLMShardingManager(module=self.hybrid_engine,
|
| 106 |
+
inference_engine=rollout.inference_engine,
|
| 107 |
+
model_config=self.actor_model_config,
|
| 108 |
+
layer_name_mapping=layer_name_mapping)
|
| 109 |
+
...
|
| 110 |
+
|
| 111 |
+
2. Generate sequence and recompute log prob
|
| 112 |
+
|
| 113 |
+
.. code:: python
|
| 114 |
+
|
| 115 |
+
@register(dispatch_mode=Dispatch.MEGATRON_PP_AS_DP_PROTO)
|
| 116 |
+
def generate_sequences(self, prompts: DataProto):
|
| 117 |
+
|
| 118 |
+
- ``Dispatch.MEGATRON_PP_AS_DP_PROTO``: The PP dimension of the actor
|
| 119 |
+
model will be regarded as DP dimension. Then the driver process will
|
| 120 |
+
dispatch and collect the data according to this reorganization. This
|
| 121 |
+
is because, in HybridEngine, the actor weight, which usually applied
|
| 122 |
+
larger 3D parallel sizes, will be gathered along the PP dimension and
|
| 123 |
+
TP dimension. Therefore, the corresponding data should be dispatched
|
| 124 |
+
and collected through the 3D parallel group of the rollout model,
|
| 125 |
+
rather than the actor model. However, the world_size and rank
|
| 126 |
+
information can only be retrived from ``get_megatron_global_info`` and
|
| 127 |
+
``get_megatron_rank_info``, which records the 3D information for the
|
| 128 |
+
actor model. Moreover, the data resharding inside TP dimension will be
|
| 129 |
+
processed within the HybridEngine.
|
| 130 |
+
|
| 131 |
+
- In this function, the rollout model will perform auto-regressive
|
| 132 |
+
generation and the actor model will recompute the old log prob for the
|
| 133 |
+
generated response.
|
| 134 |
+
|
| 135 |
+
3. Update actor model
|
| 136 |
+
|
| 137 |
+
.. code:: python
|
| 138 |
+
|
| 139 |
+
@register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
|
| 140 |
+
def update_actor(self, data: DataProto):
|
| 141 |
+
|
| 142 |
+
- ``Dispatch.MEGATRON_COMPUTE_PROTO``: User passes the data partitioned
|
| 143 |
+
by DP dimension. The data is dispatched to all tp/pp ranks within the
|
| 144 |
+
same dp group, and ultimately only collects output data from tp=0 and
|
| 145 |
+
the last pp.
|
| 146 |
+
- Update the actor model weight using PPO & entropy loss.
|
| 147 |
+
|
| 148 |
+
ReferenceModel
|
| 149 |
+
''''''''''''''
|
| 150 |
+
|
| 151 |
+
1. Reference model initialization
|
| 152 |
+
|
| 153 |
+
The reference model is initialized using the same function as the actor
|
| 154 |
+
model without initializing the HybridEngine and Optimizer. Then the
|
| 155 |
+
actor model is also wrapped by the ``MegatronPPOActor``.
|
| 156 |
+
|
| 157 |
+
2. Compute reference log prob
|
| 158 |
+
|
| 159 |
+
.. code:: python
|
| 160 |
+
|
| 161 |
+
@register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
|
| 162 |
+
def compute_ref_log_prob(self, data: DataProto):
|
| 163 |
+
|
| 164 |
+
- In this function, the reference model will call the compute log prob
|
| 165 |
+
function in ``MegatronPPOActor`` to compute the reference log prob.
|
| 166 |
+
|
| 167 |
+
CriticWorker and RewardWorker
|
| 168 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 169 |
+
|
| 170 |
+
1. Model initialization
|
| 171 |
+
|
| 172 |
+
Quite similar to reference model. The CriticWorker will perform
|
| 173 |
+
additional initialization for the Optimizer.
|
| 174 |
+
|
| 175 |
+
2. Compute Values for CriticWorker
|
| 176 |
+
|
| 177 |
+
.. code:: python
|
| 178 |
+
|
| 179 |
+
@register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
|
| 180 |
+
def compute_values(self, data: DataProto):
|
| 181 |
+
|
| 182 |
+
3. Update Critic
|
| 183 |
+
|
| 184 |
+
.. code:: python
|
| 185 |
+
|
| 186 |
+
@register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
|
| 187 |
+
def update_critic(self, data: DataProto):
|
| 188 |
+
|
| 189 |
+
4. Compute Reward
|
| 190 |
+
|
| 191 |
+
.. code:: python
|
| 192 |
+
|
| 193 |
+
@register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
|
| 194 |
+
def compute_rm_score(self, data: DataProto):
|
| 195 |
+
|
| 196 |
+
Context Parallel
|
| 197 |
+
----------------
|
| 198 |
+
|
| 199 |
+
This require the developer/contributor to implement the context parallel
|
| 200 |
+
both in Megatron-LM and models.
|
frameworks/agentgym_rl/docs/workers/ray_trainer.rst
ADDED
|
@@ -0,0 +1,243 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
PPO Ray Trainer
|
| 2 |
+
===============
|
| 3 |
+
|
| 4 |
+
We implement the RayPPOTrainer, which is a trainer runs on the driver
|
| 5 |
+
process on a single CPU/GPU node (default is CPU).
|
| 6 |
+
|
| 7 |
+
The PPORayTrainer include 3 core functions for data preparation,
|
| 8 |
+
WorkerGroup initialization and PPO training loop.
|
| 9 |
+
|
| 10 |
+
Data Preparation
|
| 11 |
+
----------------
|
| 12 |
+
|
| 13 |
+
The ``PPORayTrainer``, as a single process, is responsible for loading a
|
| 14 |
+
complete batch of samples (prompts) from the dataset and then dispatch
|
| 15 |
+
to different worker_groups running on different GPUs.
|
| 16 |
+
|
| 17 |
+
To generalize the data loading, we implement the ``RLHFDataset`` class
|
| 18 |
+
to load the preprocessed parquet files, apply chat templates to the
|
| 19 |
+
prompts, add padding, truncate prompts that exceed max prompt length and
|
| 20 |
+
then tokenize.
|
| 21 |
+
|
| 22 |
+
.. code:: python
|
| 23 |
+
|
| 24 |
+
self.train_dataset = RLHFDataset(parquet_files=self.config.data.train_files,
|
| 25 |
+
tokenizer=self.tokenizer,
|
| 26 |
+
prompt_key=self.config.data.prompt_key,
|
| 27 |
+
max_prompt_length=self.config.data.max_prompt_length,
|
| 28 |
+
filter_prompts=True,
|
| 29 |
+
return_raw_chat=self.config.data.get('return_raw_chat', False),
|
| 30 |
+
truncation='error')
|
| 31 |
+
|
| 32 |
+
Then, the dataloader will iterate the dataset under PPO mini batch size.
|
| 33 |
+
|
| 34 |
+
WorkerGroup Initialization
|
| 35 |
+
--------------------------
|
| 36 |
+
|
| 37 |
+
We first introduce a basic implementation of initializing the
|
| 38 |
+
``WorkerGroup`` of the actor model on a given set of GPUs.
|
| 39 |
+
|
| 40 |
+
.. code:: python
|
| 41 |
+
|
| 42 |
+
# max_colocate_count means the number of WorkerGroups (i.e. processes) in each RayResourcePool
|
| 43 |
+
# For FSDP backend, we recommend using max_colocate_count=1 that merge all WorkerGroups into one.
|
| 44 |
+
# For Megatron backend, we recommend using max_colocate_count>1 that can utilize different WorkerGroup for differnt models
|
| 45 |
+
resource_pool = RayResourcePool(process_on_nodes=[config.trainer.n_gpus_per_node] * config.trainer.nnodes,
|
| 46 |
+
use_gpu=True,
|
| 47 |
+
max_colocate_count=1)
|
| 48 |
+
# define actor rollout cls to be init on remote
|
| 49 |
+
actor_rollout_cls = RayClassWithInitArgs(cls=ActorRolloutWorker)
|
| 50 |
+
# define actor_rollout worker group
|
| 51 |
+
actor_rollout_worker_group = MegatronRayWorkerGroup(resource_pool=resource_pool,
|
| 52 |
+
ray_cls_with_init=actor_rollout_cls,
|
| 53 |
+
default_megatron_kwargs=config.actor_rollout.megatron)
|
| 54 |
+
|
| 55 |
+
Different WorkerGroups, like ``actor_rollout_worker_group`` ,
|
| 56 |
+
``critic_worker_group`` and ``ref_worker_group`` lies on a separate
|
| 57 |
+
process in the above implementation.
|
| 58 |
+
|
| 59 |
+
The driver process can then call the distributed compute function within
|
| 60 |
+
the ``actor_rollout_worker_group`` and other roles to construct the RL
|
| 61 |
+
training loop.
|
| 62 |
+
|
| 63 |
+
For models colocated in the same set of GPUs, we further provide a
|
| 64 |
+
fine-grain optimization, which merge the ``worker_group`` of different roles
|
| 65 |
+
in the same process. This optimization can save the redundant
|
| 66 |
+
CUDA/distributed context in different processes.
|
| 67 |
+
|
| 68 |
+
.. code:: python
|
| 69 |
+
|
| 70 |
+
# initialize WorkerGroup
|
| 71 |
+
# NOTE: if you want to use a different resource pool for each role, which can support different parallel size,
|
| 72 |
+
# you should not use `create_colocated_worker_cls`. Instead, directly pass different resource pool to different worker groups.
|
| 73 |
+
# See TODO(url) for more information.
|
| 74 |
+
all_wg = {}
|
| 75 |
+
for resource_pool, class_dict in self.resource_pool_to_cls.items():
|
| 76 |
+
worker_dict_cls = create_colocated_worker_cls(class_dict=class_dict)
|
| 77 |
+
wg_dict = self.ray_worker_group_cls(resource_pool=resource_pool, ray_cls_with_init=worker_dict_cls)
|
| 78 |
+
spawn_wg = wg_dict.spawn(prefix_set=class_dict.keys())
|
| 79 |
+
all_wg.update(spawn_wg)
|
| 80 |
+
|
| 81 |
+
if self.use_critic:
|
| 82 |
+
self.critic_wg = all_wg['critic']
|
| 83 |
+
self.critic_wg.init_model()
|
| 84 |
+
|
| 85 |
+
if self.use_reference_policy:
|
| 86 |
+
self.ref_policy_wg = all_wg['ref']
|
| 87 |
+
self.ref_policy_wg.init_model()
|
| 88 |
+
|
| 89 |
+
if self.use_rm:
|
| 90 |
+
self.rm_wg = all_wg['rm']
|
| 91 |
+
self.rm_wg.init_model()
|
| 92 |
+
|
| 93 |
+
# we should create rollout at the end so that vllm can have a better estimation of kv cache memory
|
| 94 |
+
self.actor_rollout_wg = all_wg['actor_rollout']
|
| 95 |
+
self.actor_rollout_wg.init_model()
|
| 96 |
+
|
| 97 |
+
.. note:: For megatron backend, if we merge the ``worker_groups`` into the same processes, all the roles will utilize the same 3D parallel size. To optimize this, we may need to maintain several 3D process groups for each role in the same distributed context. If you want to use different 3D parallel size for different roles, please follow the similar architecture of the first code block to initialize each role's ``worker_group``
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
PPO Training Loop
|
| 101 |
+
-----------------
|
| 102 |
+
|
| 103 |
+
We implement the PPO training loop by calling the functions in
|
| 104 |
+
worker_group of each role. The input and output data of each function is
|
| 105 |
+
a ``DataProto`` object implemented in `protocol.py <https://github.com/volcengine/verl/blob/main/verl/protocol.py>`_. In the training
|
| 106 |
+
loop, trainer will dispatch/collect the data to/from different GPUs
|
| 107 |
+
following the transfer protocols wrapped in the workers' functions. The
|
| 108 |
+
computation of PPO micro batches is processed in ``update_actor`` and
|
| 109 |
+
``update_critic`` functions.
|
| 110 |
+
|
| 111 |
+
To extend to other RLHF algorithms, such as DPO, GRPO, please refer to
|
| 112 |
+
:doc:`../advance/dpo_extension`.
|
| 113 |
+
|
| 114 |
+
.. code:: python
|
| 115 |
+
|
| 116 |
+
def fit(self):
|
| 117 |
+
"""
|
| 118 |
+
The training loop of PPO.
|
| 119 |
+
The driver process only need to call the compute functions of the worker group through RPC to construct the PPO dataflow.
|
| 120 |
+
The light-weight advantage computation is done on the driver process.
|
| 121 |
+
"""
|
| 122 |
+
from verl.utils.tracking import Tracking
|
| 123 |
+
from omegaconf import OmegaConf
|
| 124 |
+
|
| 125 |
+
logger = Tracking(project_name=self.config.trainer.project_name,
|
| 126 |
+
experiment_name=self.config.trainer.experiment_name,
|
| 127 |
+
default_backend=self.config.trainer.logger,
|
| 128 |
+
config=OmegaConf.to_container(self.config, resolve=True))
|
| 129 |
+
|
| 130 |
+
global_steps = 0
|
| 131 |
+
|
| 132 |
+
# perform validation before training
|
| 133 |
+
# currently, we only support validation using the reward_function.
|
| 134 |
+
if self.val_reward_fn is not None:
|
| 135 |
+
val_metrics = self._validate()
|
| 136 |
+
pprint(f'Initial validation metrics: {val_metrics}')
|
| 137 |
+
|
| 138 |
+
for epoch in range(self.config.trainer.total_epochs):
|
| 139 |
+
for batch_dict in self.train_dataloader:
|
| 140 |
+
metrics = {}
|
| 141 |
+
|
| 142 |
+
batch: DataProto = DataProto.from_single_dict(batch_dict)
|
| 143 |
+
# batch = batch.to('cuda')
|
| 144 |
+
|
| 145 |
+
# pop those keys for generation
|
| 146 |
+
gen_batch = batch.pop(batch_keys=['input_ids', 'attention_mask', 'position_ids'])
|
| 147 |
+
|
| 148 |
+
# generate a batch
|
| 149 |
+
with Timer(name='gen', logger=None) as timer:
|
| 150 |
+
gen_batch_output = self.actor_rollout_wg.generate_sequences(gen_batch)
|
| 151 |
+
metrics['timing/gen'] = timer.last
|
| 152 |
+
|
| 153 |
+
batch = batch.union(gen_batch_output)
|
| 154 |
+
|
| 155 |
+
if self.use_reference_policy:
|
| 156 |
+
# compute reference log_prob
|
| 157 |
+
with Timer(name='ref', logger=None) as timer:
|
| 158 |
+
ref_log_prob = self.ref_policy_wg.compute_ref_log_prob(batch)
|
| 159 |
+
batch = batch.union(ref_log_prob)
|
| 160 |
+
metrics['timing/ref'] = timer.last
|
| 161 |
+
|
| 162 |
+
# compute values
|
| 163 |
+
with Timer(name='values', logger=None) as timer:
|
| 164 |
+
values = self.critic_wg.compute_values(batch)
|
| 165 |
+
batch = batch.union(values)
|
| 166 |
+
metrics['timing/values'] = timer.last
|
| 167 |
+
|
| 168 |
+
with Timer(name='adv', logger=None) as timer:
|
| 169 |
+
# compute scores. Support both model and function-based.
|
| 170 |
+
# We first compute the scores using reward model. Then, we call reward_fn to combine
|
| 171 |
+
# the results from reward model and rule-based results.
|
| 172 |
+
if self.use_rm:
|
| 173 |
+
# we first compute reward model score
|
| 174 |
+
reward_tensor = self.rm_wg.compute_rm_score(batch)
|
| 175 |
+
batch = batch.union(reward_tensor)
|
| 176 |
+
|
| 177 |
+
# we combine with rule-based rm
|
| 178 |
+
reward_tensor = self.reward_fn(batch)
|
| 179 |
+
batch.batch['token_level_scores'] = reward_tensor
|
| 180 |
+
|
| 181 |
+
# compute rewards. apply_kl_penalty if available
|
| 182 |
+
batch, kl_metrics = apply_kl_penalty(batch,
|
| 183 |
+
kl_ctrl=self.kl_ctrl,
|
| 184 |
+
kl_penalty=self.config.algorithm.kl_penalty)
|
| 185 |
+
metrics.update(kl_metrics)
|
| 186 |
+
|
| 187 |
+
# compute advantages, executed on the driver process
|
| 188 |
+
batch = compute_advantage(batch,
|
| 189 |
+
self.config.algorithm.gamma,
|
| 190 |
+
self.config.algorithm.lam,
|
| 191 |
+
adv_estimator=self.config.algorithm.adv_estimator)
|
| 192 |
+
metrics['timing/adv'] = timer.last
|
| 193 |
+
|
| 194 |
+
# update critic
|
| 195 |
+
if self.use_critic:
|
| 196 |
+
with Timer(name='update_critic', logger=None) as timer:
|
| 197 |
+
critic_output = self.critic_wg.update_critic(batch)
|
| 198 |
+
metrics['timing/update_critic'] = timer.last
|
| 199 |
+
critic_output_metrics = reduce_metrics(critic_output.meta_info['metrics'])
|
| 200 |
+
metrics.update(critic_output_metrics)
|
| 201 |
+
|
| 202 |
+
# implement critic warmup
|
| 203 |
+
if self.config.trainer.critic_warmup <= global_steps:
|
| 204 |
+
# update actor
|
| 205 |
+
with Timer(name='update_actor', logger=None) as timer:
|
| 206 |
+
actor_output = self.actor_rollout_wg.update_actor(batch)
|
| 207 |
+
metrics['timing/update_actor'] = timer.last
|
| 208 |
+
actor_output_metrics = reduce_metrics(actor_output.meta_info['metrics'])
|
| 209 |
+
metrics.update(actor_output_metrics)
|
| 210 |
+
|
| 211 |
+
# validate
|
| 212 |
+
if self.val_reward_fn is not None and (global_steps + 1) % self.config.trainer.test_freq == 0:
|
| 213 |
+
with Timer(name='testing', logger=None) as timer:
|
| 214 |
+
val_metrics: dict = self._validate()
|
| 215 |
+
val_metrics = {f'val/{key}': val for key, val in val_metrics.items()}
|
| 216 |
+
metrics['timing/testing'] = timer.last
|
| 217 |
+
metrics.update(val_metrics)
|
| 218 |
+
|
| 219 |
+
# collect metrics
|
| 220 |
+
data_metrics = compute_data_metrics(batch=batch)
|
| 221 |
+
metrics.update(data_metrics)
|
| 222 |
+
|
| 223 |
+
# TODO: make a canonical logger that supports various backend
|
| 224 |
+
logger.log(data=metrics, step=global_steps)
|
| 225 |
+
|
| 226 |
+
if self.config.trainer.save_freq > 0 and (global_steps + 1) % self.config.trainer.save_freq == 0:
|
| 227 |
+
actor_local_path = os.path.join(self.config.trainer.default_local_dir, 'actor',
|
| 228 |
+
f'global_step_{global_steps}')
|
| 229 |
+
actor_remote_path = os.path.join(self.config.trainer.default_hdfs_dir, 'actor')
|
| 230 |
+
self.actor_rollout_wg.save_checkpoint(actor_local_path, actor_remote_path)
|
| 231 |
+
|
| 232 |
+
if self.use_critic:
|
| 233 |
+
critic_local_path = os.path.join(self.config.trainer.default_local_dir, 'critic',
|
| 234 |
+
f'global_step_{global_steps}')
|
| 235 |
+
critic_remote_path = os.path.join(self.config.trainer.default_hdfs_dir, 'critic')
|
| 236 |
+
self.critic_wg.save_checkpoint(critic_local_path, critic_remote_path)
|
| 237 |
+
|
| 238 |
+
global_steps += 1
|
| 239 |
+
|
| 240 |
+
# perform validation after training
|
| 241 |
+
if self.val_reward_fn is not None:
|
| 242 |
+
val_metrics = self._validate()
|
| 243 |
+
pprint(f'Final validation metrics: {val_metrics}')
|
frameworks/agentgym_rl/pyproject.toml
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# -------------------------------
|
| 2 |
+
# build-system
|
| 3 |
+
# -------------------------------
|
| 4 |
+
[build-system]
|
| 5 |
+
requires = [
|
| 6 |
+
"setuptools>=61.0",
|
| 7 |
+
"wheel"
|
| 8 |
+
]
|
| 9 |
+
build-backend = "setuptools.build_meta"
|
| 10 |
+
|
| 11 |
+
# -------------------------------
|
| 12 |
+
# project (PEP 621 metadata)
|
| 13 |
+
# -------------------------------
|
| 14 |
+
[project]
|
| 15 |
+
name = "verl"
|
| 16 |
+
# We'll mark the version as "dynamic" because it's read from the file "verl/version/version"
|
| 17 |
+
# (PEP 621 calls this "dynamic version").
|
| 18 |
+
# The actual version is specified in the [tool.setuptools.dynamic] section below.
|
| 19 |
+
dynamic = ["version"]
|
| 20 |
+
|
| 21 |
+
description = "verl: Volcano Engine Reinforcement Learning for LLM"
|
| 22 |
+
license = {file = "LICENSE"} # or "Apache-2.0", if you prefer an SPDX identifier
|
| 23 |
+
readme = {file = "README.md", content-type = "text/markdown"}
|
| 24 |
+
requires-python = ">=3.8"
|
| 25 |
+
|
| 26 |
+
authors = [
|
| 27 |
+
{ name = "Bytedance - Seed - MLSys", email = "zhangchi.usc1992@bytedance.com" },
|
| 28 |
+
{ name = "Bytedance - Seed - MLSys", email = "gmsheng@connect.hku.hk" },
|
| 29 |
+
]
|
| 30 |
+
|
| 31 |
+
# Dependencies corresponding to install_requires in setup.py
|
| 32 |
+
dependencies = [
|
| 33 |
+
"accelerate",
|
| 34 |
+
"codetiming",
|
| 35 |
+
"datasets",
|
| 36 |
+
"dill",
|
| 37 |
+
"hydra-core",
|
| 38 |
+
"numpy",
|
| 39 |
+
"pandas",
|
| 40 |
+
"peft",
|
| 41 |
+
"pyarrow>=15.0.0",
|
| 42 |
+
"pybind11",
|
| 43 |
+
"pylatexenc",
|
| 44 |
+
"ray>=2.10",
|
| 45 |
+
"tensordict<0.6",
|
| 46 |
+
"transformers",
|
| 47 |
+
"vllm<=0.6.3",
|
| 48 |
+
'wandb',
|
| 49 |
+
'torch==2.4.0',
|
| 50 |
+
]
|
| 51 |
+
|
| 52 |
+
# Optional dependencies (extras_require in setup.py)
|
| 53 |
+
[project.optional-dependencies]
|
| 54 |
+
test = [
|
| 55 |
+
"pytest", "yapf", "py-spy",
|
| 56 |
+
]
|
| 57 |
+
prime = ["pyext"]
|
| 58 |
+
gpu = ["liger-kernel", "flash-attn"]
|
| 59 |
+
|
| 60 |
+
# URLs
|
| 61 |
+
[project.urls]
|
| 62 |
+
Homepage = "https://github.com/volcengine/verl"
|
| 63 |
+
|
| 64 |
+
# -------------------------------
|
| 65 |
+
# tool.setuptools - Additional config
|
| 66 |
+
# -------------------------------
|
| 67 |
+
[tool.setuptools]
|
| 68 |
+
# True means `setuptools` will attempt to include all relevant files in package_data automatically.
|
| 69 |
+
# This corresponds to `include_package_data=True` in setup.py.
|
| 70 |
+
include-package-data = true
|
| 71 |
+
|
| 72 |
+
# We read the version from a file in 'verl/version/version'
|
| 73 |
+
[tool.setuptools.dynamic]
|
| 74 |
+
version = {file = "verl/version/version"}
|
| 75 |
+
|
| 76 |
+
# If you need to mimic `package_dir={'': '.'}`:
|
| 77 |
+
[tool.setuptools.package-dir]
|
| 78 |
+
"" = "."
|
| 79 |
+
|
| 80 |
+
# If you need to include specific non-Python data (like YAML files or version file):
|
| 81 |
+
# This is the rough equivalent of package_data={'': ['version/*'], 'verl': ['trainer/config/*.yaml']}
|
| 82 |
+
[tool.setuptools.package-data]
|
| 83 |
+
verl = [
|
| 84 |
+
"version/*",
|
| 85 |
+
"trainer/config/*.yaml"
|
| 86 |
+
]
|
frameworks/agentgym_rl/requirements.txt
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# requirements.txt records the full set of dependencies for development
|
| 2 |
+
accelerate
|
| 3 |
+
codetiming
|
| 4 |
+
datasets
|
| 5 |
+
dill
|
| 6 |
+
flash-attn
|
| 7 |
+
hydra-core
|
| 8 |
+
liger-kernel
|
| 9 |
+
numpy
|
| 10 |
+
pandas
|
| 11 |
+
peft
|
| 12 |
+
pyarrow>=15.0.0
|
| 13 |
+
pybind11
|
| 14 |
+
pylatexenc
|
| 15 |
+
ray
|
| 16 |
+
tensordict<0.6
|
| 17 |
+
transformers
|
| 18 |
+
vllm<=0.6.3
|
| 19 |
+
wandb
|
frameworks/agentgym_rl/scripts/format.sh
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
pip3 install --upgrade yapf
|
| 3 |
+
python3 -m yapf -ir -vv --style ./.style.yapf verl tests single_controller examples
|
frameworks/agentgym_rl/scripts/model_merger.py
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2024 Bytedance Ltd. and/or its affiliates
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
from typing import List, Tuple, Dict
|
| 16 |
+
import re
|
| 17 |
+
import os
|
| 18 |
+
import torch
|
| 19 |
+
import argparse
|
| 20 |
+
from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification
|
| 21 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 22 |
+
from torch.distributed._tensor import DTensor, Shard, Placement
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def merge_by_placement(tensors: List[torch.Tensor], placement: Placement):
|
| 26 |
+
if placement.is_replicate():
|
| 27 |
+
return tensors[0]
|
| 28 |
+
elif placement.is_partial():
|
| 29 |
+
raise NotImplementedError("Partial placement is not supported yet")
|
| 30 |
+
elif placement.is_shard():
|
| 31 |
+
return torch.cat(tensors, dim=placement.dim).contiguous()
|
| 32 |
+
else:
|
| 33 |
+
raise ValueError(f"Unsupported placement: {placement}")
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
if __name__ == '__main__':
|
| 37 |
+
parser = argparse.ArgumentParser()
|
| 38 |
+
parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model")
|
| 39 |
+
parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload")
|
| 40 |
+
args = parser.parse_args()
|
| 41 |
+
|
| 42 |
+
assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface"
|
| 43 |
+
local_dir = args.local_dir
|
| 44 |
+
|
| 45 |
+
# copy rank zero to find the shape of (dp, fsdp)
|
| 46 |
+
rank = 0
|
| 47 |
+
world_size = 0
|
| 48 |
+
for filename in os.listdir(local_dir):
|
| 49 |
+
match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename)
|
| 50 |
+
if match:
|
| 51 |
+
world_size = match.group(1)
|
| 52 |
+
break
|
| 53 |
+
assert world_size, "No model file with the proper format"
|
| 54 |
+
|
| 55 |
+
state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu')
|
| 56 |
+
pivot_key = sorted(list(state_dict.keys()))[0]
|
| 57 |
+
weight = state_dict[pivot_key]
|
| 58 |
+
assert isinstance(weight, torch.distributed._tensor.DTensor)
|
| 59 |
+
# get sharding info
|
| 60 |
+
device_mesh = weight.device_mesh
|
| 61 |
+
mesh = device_mesh.mesh
|
| 62 |
+
mesh_dim_names = device_mesh.mesh_dim_names
|
| 63 |
+
|
| 64 |
+
print(f'Got device mesh {mesh}, mesh_dim_names {mesh_dim_names}')
|
| 65 |
+
|
| 66 |
+
assert mesh_dim_names in (
|
| 67 |
+
('fsdp',),
|
| 68 |
+
), f'Unsupported mesh_dim_names {mesh_dim_names}'
|
| 69 |
+
|
| 70 |
+
if 'tp' in mesh_dim_names:
|
| 71 |
+
# fsdp * tp
|
| 72 |
+
total_shards = mesh.shape[-1] * mesh.shape[-2]
|
| 73 |
+
mesh_shape = (mesh.shape[-2], mesh.shape[-1])
|
| 74 |
+
else:
|
| 75 |
+
# fsdp
|
| 76 |
+
total_shards = mesh.shape[-1]
|
| 77 |
+
mesh_shape = (mesh.shape[-1],)
|
| 78 |
+
|
| 79 |
+
print(f'Processing model shards with {total_shards} {mesh_shape} in total')
|
| 80 |
+
|
| 81 |
+
model_state_dict_lst = []
|
| 82 |
+
model_state_dict_lst.append(state_dict)
|
| 83 |
+
model_state_dict_lst.extend([""] * (total_shards - 1))
|
| 84 |
+
|
| 85 |
+
def process_one_shard(rank):
|
| 86 |
+
model_path = os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt')
|
| 87 |
+
state_dict = torch.load(model_path, map_location='cpu', weights_only=False)
|
| 88 |
+
model_state_dict_lst[rank] = state_dict
|
| 89 |
+
return state_dict
|
| 90 |
+
|
| 91 |
+
with ThreadPoolExecutor(max_workers=min(32, os.cpu_count())) as executor:
|
| 92 |
+
for rank in range(1, total_shards):
|
| 93 |
+
executor.submit(process_one_shard, rank)
|
| 94 |
+
state_dict = {}
|
| 95 |
+
param_placements: Dict[str, List[Placement]] = {}
|
| 96 |
+
keys = set(model_state_dict_lst[0].keys())
|
| 97 |
+
for key in keys:
|
| 98 |
+
state_dict[key] = []
|
| 99 |
+
for model_state_dict in model_state_dict_lst:
|
| 100 |
+
try:
|
| 101 |
+
tensor = model_state_dict.pop(key)
|
| 102 |
+
except:
|
| 103 |
+
print("-"*30)
|
| 104 |
+
print(model_state_dict)
|
| 105 |
+
if isinstance(tensor, DTensor):
|
| 106 |
+
state_dict[key].append(tensor._local_tensor.bfloat16())
|
| 107 |
+
placements = tuple(tensor.placements)
|
| 108 |
+
# replicated placement at dp dimension can be discarded
|
| 109 |
+
if mesh_dim_names[0] == 'dp':
|
| 110 |
+
placements = placements[1:]
|
| 111 |
+
if key not in param_placements:
|
| 112 |
+
param_placements[key] = placements
|
| 113 |
+
else:
|
| 114 |
+
assert param_placements[key] == placements
|
| 115 |
+
else:
|
| 116 |
+
state_dict[key] = tensor.bfloat16()
|
| 117 |
+
|
| 118 |
+
del model_state_dict_lst
|
| 119 |
+
|
| 120 |
+
for key in sorted(state_dict):
|
| 121 |
+
if not isinstance(state_dict[key], list):
|
| 122 |
+
print(f"No need to merge key {key}")
|
| 123 |
+
continue
|
| 124 |
+
# merge shards
|
| 125 |
+
placements: Tuple[Shard] = param_placements[key]
|
| 126 |
+
if len(mesh_shape) == 1:
|
| 127 |
+
# 1-D list, FSDP without TP
|
| 128 |
+
assert len(placements) == 1
|
| 129 |
+
shards = state_dict[key]
|
| 130 |
+
state_dict[key] = merge_by_placement(shards, placements[0])
|
| 131 |
+
else:
|
| 132 |
+
# 2-D list, FSDP + TP
|
| 133 |
+
raise NotImplementedError("FSDP + TP is not supported yet")
|
| 134 |
+
|
| 135 |
+
print('Writing to local disk')
|
| 136 |
+
hf_path = os.path.join(local_dir, 'huggingface')
|
| 137 |
+
config = AutoConfig.from_pretrained(hf_path)
|
| 138 |
+
|
| 139 |
+
if 'ForTokenClassification' in config.architectures[0]:
|
| 140 |
+
auto_model = AutoModelForTokenClassification
|
| 141 |
+
elif 'ForCausalLM' in config.architectures[0]:
|
| 142 |
+
auto_model = AutoModelForCausalLM
|
| 143 |
+
else:
|
| 144 |
+
raise NotImplementedError(f'Unknown architecture {config["architectures"]}')
|
| 145 |
+
|
| 146 |
+
with torch.device('meta'):
|
| 147 |
+
model = auto_model.from_config(config, torch_dtype=torch.bfloat16)
|
| 148 |
+
model.to_empty(device='cpu')
|
| 149 |
+
|
| 150 |
+
print(f'Saving model to {hf_path}')
|
| 151 |
+
model.save_pretrained(hf_path, state_dict=state_dict)
|
| 152 |
+
del state_dict
|
| 153 |
+
del model
|
| 154 |
+
if args.hf_upload_path:
|
| 155 |
+
# Push to hugging face
|
| 156 |
+
from huggingface_hub import HfApi
|
| 157 |
+
api = HfApi()
|
| 158 |
+
api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True)
|
| 159 |
+
api.upload_folder(
|
| 160 |
+
folder_path=hf_path,
|
| 161 |
+
repo_id=args.hf_upload_path,
|
| 162 |
+
repo_type="model"
|
| 163 |
+
)
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
|
frameworks/agentgym_rl/scripts/multiple_model_merger.py
ADDED
|
@@ -0,0 +1,178 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2024 Bytedance Ltd. and/or its affiliates
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
from typing import List, Tuple, Dict
|
| 16 |
+
import re
|
| 17 |
+
import os
|
| 18 |
+
import torch
|
| 19 |
+
import argparse
|
| 20 |
+
from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification
|
| 21 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 22 |
+
from torch.distributed._tensor import DTensor, Shard, Placement
|
| 23 |
+
|
| 24 |
+
import shutil
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def merge_by_placement(tensors: List[torch.Tensor], placement: Placement):
|
| 28 |
+
if placement.is_replicate():
|
| 29 |
+
return tensors[0]
|
| 30 |
+
elif placement.is_partial():
|
| 31 |
+
raise NotImplementedError("Partial placement is not supported yet")
|
| 32 |
+
elif placement.is_shard():
|
| 33 |
+
return torch.cat(tensors, dim=placement.dim).contiguous()
|
| 34 |
+
else:
|
| 35 |
+
raise ValueError(f"Unsupported placement: {placement}")
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
if __name__ == '__main__':
|
| 39 |
+
parser = argparse.ArgumentParser()
|
| 40 |
+
parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model")
|
| 41 |
+
parser.add_argument('--save_dir', default=None, type = str, help="The path to save your model")
|
| 42 |
+
parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload")
|
| 43 |
+
args = parser.parse_args()
|
| 44 |
+
|
| 45 |
+
assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface"
|
| 46 |
+
local_dir = args.local_dir
|
| 47 |
+
save_dir = args.save_dir
|
| 48 |
+
|
| 49 |
+
# copy rank zero to find the shape of (dp, fsdp)
|
| 50 |
+
rank = 0
|
| 51 |
+
world_size = 0
|
| 52 |
+
for filename in os.listdir(local_dir):
|
| 53 |
+
match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename)
|
| 54 |
+
if match:
|
| 55 |
+
world_size = match.group(1)
|
| 56 |
+
break
|
| 57 |
+
assert world_size, "No model file with the proper format"
|
| 58 |
+
|
| 59 |
+
state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu')
|
| 60 |
+
pivot_key = sorted(list(state_dict.keys()))[0]
|
| 61 |
+
weight = state_dict[pivot_key]
|
| 62 |
+
assert isinstance(weight, torch.distributed._tensor.DTensor)
|
| 63 |
+
# get sharding info
|
| 64 |
+
device_mesh = weight.device_mesh
|
| 65 |
+
mesh = device_mesh.mesh
|
| 66 |
+
mesh_dim_names = device_mesh.mesh_dim_names
|
| 67 |
+
|
| 68 |
+
print(f'Got device mesh {mesh}, mesh_dim_names {mesh_dim_names}')
|
| 69 |
+
|
| 70 |
+
assert mesh_dim_names in (
|
| 71 |
+
('fsdp',),
|
| 72 |
+
), f'Unsupported mesh_dim_names {mesh_dim_names}'
|
| 73 |
+
|
| 74 |
+
if 'tp' in mesh_dim_names:
|
| 75 |
+
# fsdp * tp
|
| 76 |
+
total_shards = mesh.shape[-1] * mesh.shape[-2]
|
| 77 |
+
mesh_shape = (mesh.shape[-2], mesh.shape[-1])
|
| 78 |
+
else:
|
| 79 |
+
# fsdp
|
| 80 |
+
total_shards = mesh.shape[-1]
|
| 81 |
+
mesh_shape = (mesh.shape[-1],)
|
| 82 |
+
|
| 83 |
+
print(f'Processing model shards with {total_shards} {mesh_shape} in total')
|
| 84 |
+
|
| 85 |
+
model_state_dict_lst = []
|
| 86 |
+
model_state_dict_lst.append(state_dict)
|
| 87 |
+
model_state_dict_lst.extend([""] * (total_shards - 1))
|
| 88 |
+
|
| 89 |
+
def process_one_shard(rank):
|
| 90 |
+
model_path = os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt')
|
| 91 |
+
state_dict = torch.load(model_path, map_location='cpu', weights_only=False)
|
| 92 |
+
model_state_dict_lst[rank] = state_dict
|
| 93 |
+
return state_dict
|
| 94 |
+
|
| 95 |
+
with ThreadPoolExecutor(max_workers=min(32, os.cpu_count())) as executor:
|
| 96 |
+
for rank in range(1, total_shards):
|
| 97 |
+
executor.submit(process_one_shard, rank)
|
| 98 |
+
state_dict = {}
|
| 99 |
+
param_placements: Dict[str, List[Placement]] = {}
|
| 100 |
+
keys = set(model_state_dict_lst[0].keys())
|
| 101 |
+
for key in keys:
|
| 102 |
+
state_dict[key] = []
|
| 103 |
+
for model_state_dict in model_state_dict_lst:
|
| 104 |
+
try:
|
| 105 |
+
tensor = model_state_dict.pop(key)
|
| 106 |
+
except:
|
| 107 |
+
print("-"*30)
|
| 108 |
+
print(model_state_dict)
|
| 109 |
+
if isinstance(tensor, DTensor):
|
| 110 |
+
state_dict[key].append(tensor._local_tensor.bfloat16())
|
| 111 |
+
placements = tuple(tensor.placements)
|
| 112 |
+
# replicated placement at dp dimension can be discarded
|
| 113 |
+
if mesh_dim_names[0] == 'dp':
|
| 114 |
+
placements = placements[1:]
|
| 115 |
+
if key not in param_placements:
|
| 116 |
+
param_placements[key] = placements
|
| 117 |
+
else:
|
| 118 |
+
assert param_placements[key] == placements
|
| 119 |
+
else:
|
| 120 |
+
state_dict[key] = tensor.bfloat16()
|
| 121 |
+
|
| 122 |
+
del model_state_dict_lst
|
| 123 |
+
|
| 124 |
+
for key in sorted(state_dict):
|
| 125 |
+
if not isinstance(state_dict[key], list):
|
| 126 |
+
print(f"No need to merge key {key}")
|
| 127 |
+
continue
|
| 128 |
+
# merge shards
|
| 129 |
+
placements: Tuple[Shard] = param_placements[key]
|
| 130 |
+
if len(mesh_shape) == 1:
|
| 131 |
+
# 1-D list, FSDP without TP
|
| 132 |
+
assert len(placements) == 1
|
| 133 |
+
shards = state_dict[key]
|
| 134 |
+
state_dict[key] = merge_by_placement(shards, placements[0])
|
| 135 |
+
else:
|
| 136 |
+
# 2-D list, FSDP + TP
|
| 137 |
+
raise NotImplementedError("FSDP + TP is not supported yet")
|
| 138 |
+
|
| 139 |
+
print('Writing to local disk')
|
| 140 |
+
if save_dir and os.path.abspath(save_dir) != os.path.abspath(local_dir):
|
| 141 |
+
hf_path = os.path.join(save_dir, 'huggingface')
|
| 142 |
+
shutil.copytree(os.path.join(local_dir, 'huggingface'), hf_path)
|
| 143 |
+
else:
|
| 144 |
+
hf_path = os.path.join(local_dir, 'huggingface')
|
| 145 |
+
config = AutoConfig.from_pretrained(hf_path)
|
| 146 |
+
|
| 147 |
+
if 'ForTokenClassification' in config.architectures[0]:
|
| 148 |
+
auto_model = AutoModelForTokenClassification
|
| 149 |
+
elif 'ForCausalLM' in config.architectures[0]:
|
| 150 |
+
auto_model = AutoModelForCausalLM
|
| 151 |
+
else:
|
| 152 |
+
raise NotImplementedError(f'Unknown architecture {config["architectures"]}')
|
| 153 |
+
|
| 154 |
+
with torch.device('meta'):
|
| 155 |
+
model = auto_model.from_config(config, torch_dtype=torch.bfloat16)
|
| 156 |
+
model.to_empty(device='cpu')
|
| 157 |
+
|
| 158 |
+
print(f'Saving model to {hf_path}')
|
| 159 |
+
model.save_pretrained(hf_path, state_dict=state_dict)
|
| 160 |
+
del state_dict
|
| 161 |
+
del model
|
| 162 |
+
if args.hf_upload_path:
|
| 163 |
+
# Push to hugging face
|
| 164 |
+
from huggingface_hub import HfApi
|
| 165 |
+
api = HfApi()
|
| 166 |
+
api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True)
|
| 167 |
+
api.upload_folder(
|
| 168 |
+
folder_path=hf_path,
|
| 169 |
+
repo_id=args.hf_upload_path,
|
| 170 |
+
repo_type="model"
|
| 171 |
+
)
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
|
frameworks/agentgym_rl/scripts/single_model_merger.py
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2024 Bytedance Ltd. and/or its affiliates
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
import re
|
| 16 |
+
import os
|
| 17 |
+
import torch
|
| 18 |
+
import argparse
|
| 19 |
+
from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification
|
| 20 |
+
|
| 21 |
+
import shutil
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
if __name__ == '__main__':
|
| 25 |
+
parser = argparse.ArgumentParser()
|
| 26 |
+
parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model")
|
| 27 |
+
parser.add_argument('--save_dir', default=None, type = str, help="The path to save your model")
|
| 28 |
+
parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload")
|
| 29 |
+
args = parser.parse_args()
|
| 30 |
+
|
| 31 |
+
assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface"
|
| 32 |
+
local_dir = args.local_dir
|
| 33 |
+
save_dir = args.save_dir
|
| 34 |
+
|
| 35 |
+
# copy rank zero to find the shape of (dp, fsdp)
|
| 36 |
+
rank = 0
|
| 37 |
+
world_size = 0
|
| 38 |
+
for filename in os.listdir(local_dir):
|
| 39 |
+
match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename)
|
| 40 |
+
if match:
|
| 41 |
+
world_size = match.group(1)
|
| 42 |
+
break
|
| 43 |
+
assert world_size, "No model file with the proper format"
|
| 44 |
+
|
| 45 |
+
state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu')
|
| 46 |
+
|
| 47 |
+
print('Writing to local disk')
|
| 48 |
+
if save_dir and os.path.abspath(save_dir) != os.path.abspath(local_dir):
|
| 49 |
+
hf_path = os.path.join(save_dir, 'huggingface')
|
| 50 |
+
shutil.copytree(os.path.join(local_dir, 'huggingface'), hf_path)
|
| 51 |
+
else:
|
| 52 |
+
hf_path = os.path.join(local_dir, 'huggingface')
|
| 53 |
+
config = AutoConfig.from_pretrained(hf_path)
|
| 54 |
+
|
| 55 |
+
if 'ForTokenClassification' in config.architectures[0]:
|
| 56 |
+
auto_model = AutoModelForTokenClassification
|
| 57 |
+
elif 'ForCausalLM' in config.architectures[0]:
|
| 58 |
+
auto_model = AutoModelForCausalLM
|
| 59 |
+
else:
|
| 60 |
+
raise NotImplementedError(f'Unknown architecture {config["architectures"]}')
|
| 61 |
+
|
| 62 |
+
with torch.device('meta'):
|
| 63 |
+
model = auto_model.from_config(config, torch_dtype=torch.bfloat16)
|
| 64 |
+
model.to_empty(device='cpu')
|
| 65 |
+
|
| 66 |
+
print(f'Saving model to {hf_path}')
|
| 67 |
+
model.save_pretrained(hf_path, state_dict=state_dict)
|
| 68 |
+
del state_dict
|
| 69 |
+
del model
|
| 70 |
+
if args.hf_upload_path:
|
| 71 |
+
# Push to hugging face
|
| 72 |
+
from huggingface_hub import HfApi
|
| 73 |
+
api = HfApi()
|
| 74 |
+
api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True)
|
| 75 |
+
api.upload_folder(
|
| 76 |
+
folder_path=hf_path,
|
| 77 |
+
repo_id=args.hf_upload_path,
|
| 78 |
+
repo_type="model"
|
| 79 |
+
)
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
|
frameworks/agentgym_rl/setup.py
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2024 Bytedance Ltd. and/or its affiliates
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
# setup.py is the fallback installation script when pyproject.toml does not work
|
| 16 |
+
from setuptools import setup, find_packages
|
| 17 |
+
import os
|
| 18 |
+
|
| 19 |
+
version_folder = os.path.dirname(os.path.join(os.path.abspath(__file__)))
|
| 20 |
+
|
| 21 |
+
with open(os.path.join(version_folder, 'verl/version/version')) as f:
|
| 22 |
+
__version__ = f.read().strip()
|
| 23 |
+
|
| 24 |
+
install_requires = [
|
| 25 |
+
'accelerate',
|
| 26 |
+
'codetiming',
|
| 27 |
+
'datasets',
|
| 28 |
+
'dill',
|
| 29 |
+
'hydra-core',
|
| 30 |
+
'numpy',
|
| 31 |
+
'pandas',
|
| 32 |
+
'peft',
|
| 33 |
+
'pyarrow>=15.0.0',
|
| 34 |
+
'pybind11',
|
| 35 |
+
'pylatexenc',
|
| 36 |
+
'ray>=2.10',
|
| 37 |
+
'tensordict<0.6',
|
| 38 |
+
'transformers',
|
| 39 |
+
'vllm<=0.6.3',
|
| 40 |
+
'wandb',
|
| 41 |
+
]
|
| 42 |
+
|
| 43 |
+
TEST_REQUIRES = ['pytest', 'yapf', 'py-spy']
|
| 44 |
+
PRIME_REQUIRES = ['pyext']
|
| 45 |
+
GPU_REQUIRES = ['liger-kernel', 'flash-attn']
|
| 46 |
+
|
| 47 |
+
extras_require = {
|
| 48 |
+
'test': TEST_REQUIRES,
|
| 49 |
+
'prime': PRIME_REQUIRES,
|
| 50 |
+
'gpu': GPU_REQUIRES,
|
| 51 |
+
}
|
| 52 |
+
|
| 53 |
+
from pathlib import Path
|
| 54 |
+
this_directory = Path(__file__).parent
|
| 55 |
+
long_description = (this_directory / "VERL_README.md").read_text()
|
| 56 |
+
|
| 57 |
+
setup(
|
| 58 |
+
name='verl',
|
| 59 |
+
version=__version__,
|
| 60 |
+
package_dir={'': '.'},
|
| 61 |
+
packages=find_packages(where='.'),
|
| 62 |
+
url='https://github.com/volcengine/verl',
|
| 63 |
+
license='Apache 2.0',
|
| 64 |
+
author='Bytedance - Seed - MLSys',
|
| 65 |
+
author_email='zhangchi.usc1992@bytedance.com, gmsheng@connect.hku.hk',
|
| 66 |
+
description='verl: Volcano Engine Reinforcement Learning for LLM',
|
| 67 |
+
install_requires=install_requires,
|
| 68 |
+
extras_require=extras_require,
|
| 69 |
+
package_data={'': ['version/*'],
|
| 70 |
+
'verl': ['trainer/config/*.yaml'],},
|
| 71 |
+
include_package_data=True,
|
| 72 |
+
long_description=long_description,
|
| 73 |
+
long_description_content_type='text/markdown'
|
| 74 |
+
)
|
frameworks/agentgym_rl/verl/__init__.py
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2024 Bytedance Ltd. and/or its affiliates
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
import os
|
| 16 |
+
|
| 17 |
+
version_folder = os.path.dirname(os.path.join(os.path.abspath(__file__)))
|
| 18 |
+
|
| 19 |
+
with open(os.path.join(version_folder, 'version/version')) as f:
|
| 20 |
+
__version__ = f.read().strip()
|
| 21 |
+
|
| 22 |
+
from .protocol import DataProto
|
| 23 |
+
|
| 24 |
+
from .utils.logging_utils import set_basic_config
|
| 25 |
+
import logging
|
| 26 |
+
|
| 27 |
+
set_basic_config(level=logging.WARNING)
|
frameworks/agentgym_rl/verl/agent_trainer/__init__.py
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2024 Bytedance Ltd. and/or its affiliates
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
frameworks/agentgym_rl/verl/agent_trainer/config/evaluation.yaml
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
data:
|
| 2 |
+
path: null
|
| 3 |
+
reward_model_key: rewards
|
frameworks/agentgym_rl/verl/agent_trainer/config/generation.yaml
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
trainer:
|
| 2 |
+
nnodes: 1
|
| 3 |
+
n_gpus_per_node: 8
|
| 4 |
+
|
| 5 |
+
data:
|
| 6 |
+
path: null
|
| 7 |
+
max_prompt_length: 1024
|
| 8 |
+
max_response_length: 15360
|
| 9 |
+
prompt_key: item_id
|
| 10 |
+
n_samples: 1
|
| 11 |
+
output_path: null
|
| 12 |
+
batch_size: 16
|
| 13 |
+
|
| 14 |
+
agentgym:
|
| 15 |
+
task_name: textcraft
|
| 16 |
+
env_addr: 'http://localhost:5000'
|
| 17 |
+
max_retries: 10
|
| 18 |
+
max_rounds: 10
|
| 19 |
+
timeout: 300
|
| 20 |
+
|
| 21 |
+
model:
|
| 22 |
+
path: null
|
| 23 |
+
external_lib: null
|
| 24 |
+
rollout:
|
| 25 |
+
name: vllm
|
| 26 |
+
temperature: 1.0
|
| 27 |
+
top_k: -1 # 0 for hf rollout, -1 for vllm rollout
|
| 28 |
+
top_p: 1
|
| 29 |
+
prompt_length: ${data.max_prompt_length} # not use for opensource
|
| 30 |
+
response_length: ${data.max_response_length}
|
| 31 |
+
# for vllm rollout
|
| 32 |
+
dtype: bfloat16 # should align with FSDP
|
| 33 |
+
gpu_memory_utilization: 0.5
|
| 34 |
+
ignore_eos: False
|
| 35 |
+
enforce_eager: True
|
| 36 |
+
free_cache_engine: True
|
| 37 |
+
load_format: dummy_dtensor
|
| 38 |
+
tensor_model_parallel_size: 2
|
| 39 |
+
max_num_batched_tokens: 8192
|
| 40 |
+
max_model_len: 32768
|
| 41 |
+
max_num_seqs: 1024
|
| 42 |
+
log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu
|
| 43 |
+
log_prob_micro_batch_size_per_gpu: 1
|
| 44 |
+
log_prob_use_dynamic_bsz: ${actor.use_dynamic_bsz}
|
| 45 |
+
log_prob_max_token_len_per_gpu: ${actor.ppo_max_token_len_per_gpu}
|
| 46 |
+
max_tokens: 1024
|
| 47 |
+
disable_log_stats: True
|
| 48 |
+
enable_chunked_prefill: True # may get higher throughput when set to True. When activated, Please increase max_num_batched_tokens or decrease max_model_len.
|
| 49 |
+
# for hf rollout
|
| 50 |
+
do_sample: True
|
| 51 |
+
# number of responses (i.e. num sample times)
|
| 52 |
+
n: 1 # > 1 for grpo
|
| 53 |
+
send_interval: 1
|
| 54 |
+
rollout_log_dir: null
|
| 55 |
+
|
| 56 |
+
actor:
|
| 57 |
+
strategy: fsdp # This is for backward-compatibility
|
| 58 |
+
ppo_mini_batch_size: 16
|
| 59 |
+
ppo_micro_batch_size: null # will be deprecated, use ppo_micro_batch_size_per_gpu
|
| 60 |
+
ppo_micro_batch_size_per_gpu: 1
|
| 61 |
+
use_dynamic_bsz: False
|
| 62 |
+
ppo_max_token_len_per_gpu: 16384 # n * ${data.max_prompt_length} + ${data.max_response_length}
|
| 63 |
+
grad_clip: 1.0
|
| 64 |
+
clip_ratio: 0.2
|
| 65 |
+
entropy_coeff: 0.001
|
| 66 |
+
use_kl_loss: null # True for GRPO
|
| 67 |
+
kl_loss_coef: 0.001 # for grpo
|
| 68 |
+
kl_loss_type: low_var_kl # for grpo
|
| 69 |
+
ppo_epochs: 1
|
| 70 |
+
shuffle: False
|
| 71 |
+
ulysses_sequence_parallel_size: 1 # sp size
|
| 72 |
+
optim:
|
| 73 |
+
lr: 1e-6
|
| 74 |
+
lr_warmup_steps_ratio: 0. # the total steps will be injected during runtime
|
| 75 |
+
min_lr_ratio: null # only useful for warmup with cosine
|
| 76 |
+
warmup_style: constant # select from constant/cosine
|
| 77 |
+
total_training_steps: -1 # must be override by program
|
| 78 |
+
fsdp_config:
|
| 79 |
+
wrap_policy:
|
| 80 |
+
# transformer_layer_cls_to_wrap: None
|
| 81 |
+
min_num_params: 0
|
| 82 |
+
param_offload: False
|
| 83 |
+
grad_offload: False
|
| 84 |
+
optimizer_offload: False
|
| 85 |
+
fsdp_size: -1
|