SeanWang0027 commited on
Commit
7a71a78
·
verified ·
1 Parent(s): 0817da6

add files 1-400

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +130 -0
  2. docs/ENV_NOTES.md +98 -0
  3. eval/aime_eval.py +223 -0
  4. eval/aime_launch.py +17 -0
  5. eval/hmmt_eval.py +165 -0
  6. eval/merge_eval.py +72 -0
  7. eval/retrunc_eval.py +105 -0
  8. eval/rlve_eval.py +212 -0
  9. frameworks/agentgym_rl/.gitignore +116 -0
  10. frameworks/agentgym_rl/.readthedocs.yaml +16 -0
  11. frameworks/agentgym_rl/.style.yapf +5 -0
  12. frameworks/agentgym_rl/Notice.txt +1 -0
  13. frameworks/agentgym_rl/VERL_LICENSE +202 -0
  14. frameworks/agentgym_rl/VERL_README.md +135 -0
  15. frameworks/agentgym_rl/docs/Makefile +20 -0
  16. frameworks/agentgym_rl/docs/README.md +19 -0
  17. frameworks/agentgym_rl/docs/README_vllm0.7.md +65 -0
  18. frameworks/agentgym_rl/docs/_static/logo.png +0 -0
  19. frameworks/agentgym_rl/docs/advance/dpo_extension.rst +271 -0
  20. frameworks/agentgym_rl/docs/advance/fsdp_extension.rst +95 -0
  21. frameworks/agentgym_rl/docs/advance/megatron_extension.rst +26 -0
  22. frameworks/agentgym_rl/docs/advance/placement.rst +11 -0
  23. frameworks/agentgym_rl/docs/conf.py +81 -0
  24. frameworks/agentgym_rl/docs/examples/config.rst +365 -0
  25. frameworks/agentgym_rl/docs/examples/gsm8k_example.rst +168 -0
  26. frameworks/agentgym_rl/docs/examples/ppo_code_architecture.rst +207 -0
  27. frameworks/agentgym_rl/docs/experiment/ppo.rst +45 -0
  28. frameworks/agentgym_rl/docs/faq/faq.rst +21 -0
  29. frameworks/agentgym_rl/docs/hybrid_flow.rst +269 -0
  30. frameworks/agentgym_rl/docs/index.rst +111 -0
  31. frameworks/agentgym_rl/docs/perf/perf_tuning.rst +164 -0
  32. frameworks/agentgym_rl/docs/preparation/prepare_data.rst +126 -0
  33. frameworks/agentgym_rl/docs/preparation/reward_function.rst +46 -0
  34. frameworks/agentgym_rl/docs/requirements-docs.txt +9 -0
  35. frameworks/agentgym_rl/docs/start/install.rst +111 -0
  36. frameworks/agentgym_rl/docs/start/quickstart.rst +142 -0
  37. frameworks/agentgym_rl/docs/workers/fsdp_workers.rst +142 -0
  38. frameworks/agentgym_rl/docs/workers/megatron_workers.rst +200 -0
  39. frameworks/agentgym_rl/docs/workers/ray_trainer.rst +243 -0
  40. frameworks/agentgym_rl/pyproject.toml +86 -0
  41. frameworks/agentgym_rl/requirements.txt +19 -0
  42. frameworks/agentgym_rl/scripts/format.sh +3 -0
  43. frameworks/agentgym_rl/scripts/model_merger.py +170 -0
  44. frameworks/agentgym_rl/scripts/multiple_model_merger.py +178 -0
  45. frameworks/agentgym_rl/scripts/single_model_merger.py +86 -0
  46. frameworks/agentgym_rl/setup.py +74 -0
  47. frameworks/agentgym_rl/verl/__init__.py +27 -0
  48. frameworks/agentgym_rl/verl/agent_trainer/__init__.py +13 -0
  49. frameworks/agentgym_rl/verl/agent_trainer/config/evaluation.yaml +3 -0
  50. frameworks/agentgym_rl/verl/agent_trainer/config/generation.yaml +85 -0
README.md ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags: [distillation, on-policy-distillation, rose, opd, verl, trinity, agentgym, rlve, sciworld, dapo]
4
+ ---
5
+
6
+ # ROSE / OPD 蒸馏实验的完整实现与训练框架
7
+
8
+ 三个 domain(RLVE 单轮推理、SciWorld 多轮 agent、DAPO-Math)上对比三种蒸馏方式
9
+ (teacher-SFT / on-policy distillation / online ROSE)的全部代码。
10
+
11
+ 配套的 checkpoint 与数据在:
12
+ [OPD-dq](https://huggingface.co/SeanWang0027/OPD-dq) ·
13
+ [ROSE-dq](https://huggingface.co/SeanWang0027/ROSE-dq) ·
14
+ [SciWorld-dq](https://huggingface.co/SeanWang0027/SciWorld-dq) ·
15
+ [data-dq](https://huggingface.co/SeanWang0027/data-dq)
16
+
17
+ ## 目录
18
+
19
+ ```
20
+ frameworks/
21
+ verl_rose/ ROSE 全部线用的 verl(含 recipe/rose/rose_agent_loop.py)
22
+ verl_opd/ OPD 全部线用的 verl —— 与上游 thunlp/OPD 的计算路径逐字节等价
23
+ verl_sft/ teacher-SFT 用的 verl
24
+ agentgym_rl/ AgentGym-RL —— 本项目里只用于 SciWorld 评测, 未用于训练
25
+ trinity/ Trinity —— SciWorld 的 ROSE/OPD 八条线是它训的
26
+ sciworld/
27
+ impl/ 自写的多轮 ROSE/OPD 实现(不依赖上面任何框架)
28
+ chains/ 编排脚本
29
+ trinity_configs/ Trinity 侧的 workflow 与 yaml
30
+ agentgym_eval/ AgentGym 口径的 200 题评测
31
+ launchers/
32
+ rlve/ dapo/ olmo/
33
+ eval/ AIME25 / HMMT25 / RLVE 评测
34
+ docs/ ENV_NOTES.md —— 跨家族 ROSE 的环境硬约束
35
+ tools/ 归档上传用的脚本
36
+ ```
37
+
38
+ ## 三种方法
39
+
40
+ **teacher-SFT** 教师采轨迹, 学生离线模仿。
41
+
42
+ **OPD**(on-policy distillation)教师对**学生自己采出来的 token** 打分,
43
+ `advantages = teacher_logprobs - student_logprobs`, `adv_estimator=token_reward_direct`,
44
+ 不做归一化。
45
+
46
+ **ROSE** 学生写 K 个 token 的前缀(mask=0), 教师从那里接着写 t 个 token,
47
+ CE 只打在教师 token 上(mask=1)。学生学的是"我走到这个状态之后, 老师会怎么继续"。
48
+ `recipe/rose/rose_agent_loop.py` 里有完整说明。
49
+
50
+ ## SciWorld 有两套并行实现
51
+
52
+ | 实现 | 位置 | 产物 |
53
+ |---|---|---|
54
+ | **Trinity** | `frameworks/trinity` + `sciworld/trinity_configs` | `SCIENCEWORLD_{OPD,ROSE}/` 八条线 |
55
+ | **自写多轮** | `sciworld/impl/online_{rose,opd}_sciworld.py` | 老格式 `log.jsonl + step_N/`(checkpoint 已清, 脚本保留) |
56
+
57
+ `sciworld/impl/` 那套不依赖 verl 也不依赖 Trinity, 是从头写的:
58
+ 学生走 N 轮 → 按 turn 边界构造 loss mask → 教师接管后续轮次 → 只在教师 turn 上算 CE。
59
+ 多轮的 mask 构造见 `online_rose_sciworld.py`; OPD 侧的 `forward_kl_topk` 见 `online_opd_sciworld.py`。
60
+
61
+ **AgentGym-RL 只用于评测**(`sciworld_agentgym_eval.py`, 200 题 / ≤20 轮 / greedy / nothink),
62
+ 本项目没有用它训练过。
63
+
64
+ ## 结果
65
+
66
+ ### RLVE (16k, n=8)
67
+
68
+ | line | avg@8 | pass@8 | 解出 |
69
+ |---|---|---|---|
70
+ | 基线 未训练 | 0.0333 | 0.1111 | 20 |
71
+ | official OPD bf16 (140步) | 0.0458 | 0.1444 | 26 |
72
+ | OPD bf16 mini16 (140步) | 0.0556 | 0.1722 | 31 |
73
+ | **OPD fp32 (105步)** | **0.0694** | 0.1833 | 33 |
74
+ | OPD fp32 (132步) | 0.0667 | 0.1833 | 33 |
75
+ | **official ROSE (140步)** | **0.0701** | 0.1778 | 32 |
76
+ | official ROSE seed1 | 0.0729 | 0.1889 | 34 |
77
+ | teacher-SFT n4 bs256 | 0.0556 | 0.1500 | 27 |
78
+
79
+ fp32 那条与 official bf16 **只差 dtype**(train/mini/n、lr、resp、更新次数全同)。
80
+ 上游 `on_policy_distillation.sh` 默认就是 fp32, bf16 是早期的偏离 ——
81
+ lr 1e-6 × advantage 10⁻² 的每步相对更新量低于 bf16 的分辨率, 会被舍入吞掉。
82
+
83
+ ### SciWorld (AgentGym 口径, 200 题, ≤20 轮, greedy, nothink)
84
+
85
+ | | Score | Success | Pass(>0) |
86
+ |---|---|---|---|
87
+ | Qwen3-1.7B 基座 | 0.0479 | 0.0000 | 0.1650 |
88
+ | OPD lr1e-6 | 0.0421 | — | — |
89
+ | OPD lr1e-5 | 0.1931 | — | — |
90
+ | teacher-SFT | 0.2077 | 0.0450 | 0.4100 |
91
+ | **ROSE** | **0.3127** | — | — |
92
+ | Qwen3-32B 教师 | 0.4346 | 0.1100 | 0.9350 |
93
+
94
+ ROSE 在 SciWorld 上的优势(+0.105 over SFT, +0.12 over OPD)远大于 RLVE 上的(+0.016 / +0.002)。
95
+
96
+ ### DAPO-Math 上的 ROSE 是负结果
97
+
98
+ AIME25 / HMMT25 avg@16, thinking, temp0.7 / top_p0.95 / max_new 31744:
99
+
100
+ | | AIME25 | HMMT25 |
101
+ |---|---|---|
102
+ | Qwen3-32B 教师 | 0.7125 | 0.5042 |
103
+ | Qwen3-4B 未训练 | 0.6521 | — |
104
+ | ROSE←32B 30步 (4B) | 0.6125 | — |
105
+ | Olmo-3-7B 基线 | 0.5917 | 0.4167 |
106
+ | ROSE←32B 30步 (Olmo-7B) | 0.5750 | 0.3854 |
107
+ | **ROSE←32B 220步 (Olmo-7B)** | **0.5583** | **0.3688** |
108
+
109
+ 四组独立观测同向, 且 Olmo 从 30 步到 220 步(跑满一个 epoch)继续下降 —— **排除欠训**。
110
+ 训练本身也早停滞: ce_loss 从 31 步起斜率 −0.00012, grad_norm 降到 0.18。
111
+
112
+ 诊断: `TEACHER_MAX_TOKENS=1024`, 而实测 `ce_target_tokens` 达 **960-1016**
113
+ (占预算 94-99%)—— 教师几乎每条都是写到预算耗尽被截断, 从不自己收尾。
114
+ 配合学生 16000-18600 token 的推理链, 监督只覆盖 **token 1024-2048, 约 6%**。
115
+ 学生学到的是"在思考中段接着往下写", 从没被教过怎么收敛到答���。
116
+ 表现是**输出方差变大而能力边界未动**: pass@16 靠多蒙中一题涨(两个 benchmark 各 +1 题,
117
+ 其中一题是 0/16 → 1/16), 而 avg@16 与 pass@1 都下降, 截断率上升。
118
+
119
+ 要检验这个诊断, 应该把 K 提到 4096-8192、teacher 预算给到 2048-4096,
120
+ 让切点落到推理链中后段。
121
+
122
+ ## 跨家族 ROSE(Olmo-3-7B ← Qwen3-32B)
123
+
124
+ 学生 vocab 100278 / 教师 151669, 完全不同的词表。`rose_agent_loop` 有现成通路
125
+ (`TEACHER_TOKENIZER` 非 null 即启用): 学生前缀 ids → decode 成文本 →
126
+ 套**教师的** chat template → 教师续写 → 取回 text → 用**学生的** tokenizer 重新 encode。
127
+ 实测两边对拼接缝文本的 round-trip 都无损, 教师续写语义连贯且不会重复吐 `<think>`。
128
+
129
+ 环境上的硬约束(vLLM 版本被夹死在 0.11.0、NCCL 只能关 NVLS、verl 序列打包硬依赖
130
+ flash_attn 等)见 `docs/ENV_NOTES.md` —— 每一条都对应一次失败的启动。
docs/ENV_NOTES.md ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 跨家族 ROSE 的环境约束(Olmo-3-7B ← Qwen3-32B)
2
+
3
+ 把一个**非 Qwen 家族**的学生接进这套 ROSE 时踩到的全部硬约束。都是实测出来的,
4
+ 不是从文档抄的 —— 每一条都对应一次失败的启动。
5
+
6
+ ## 1. vLLM 版本被夹死在 0.11.0
7
+
8
+ verl 0.7.0.dev 与 vLLM 之间有三处接口漂移,加上 Olmo3 支持,四条同时满足的只有 **0.11.0**:
9
+
10
+ | 版本 | Olmo3 | `vllm.lora.models` | `vllm.utils.FlexibleArgumentParser` | transformers v4 |
11
+ |---|---|---|---|---|
12
+ | 0.10.2 | 不支持 | 有 | 有 | 要 v4 |
13
+ | **0.11.0** | **支持** | **有** | **有** | **要 v4** |
14
+ | 0.11.1 / 0.11.2 | 支持 | 有 | **没了** | 要 v4 |
15
+ | 0.12.0 | 支持 | 有 | **没了** | 要 v4 |
16
+ | 0.23.0 | 支持 | **没了** | — | 要 v4 |
17
+ | ≥0.24.0 | 支持 | — | — | **强制 v5** |
18
+
19
+ - transformers 必须留在 **4.57.6**:verl 0.7 要 `AutoModelForVision2Seq`,v5 里删了;
20
+ 而 4.57.6 本身已经认 `Olmo3Config`,所以不需要为 Olmo 升 transformers。
21
+ - `flashinfer-cubin` 要与 `flashinfer-python` **同版本**(装过别的 vllm 会留下不匹配的
22
+ cubin,报 `flashinfer-cubin version (0.6.12) does not match flashinfer version (0.5.2)`)。
23
+
24
+ ## 2. NCCL:只关 NVLS,千万别关 P2P
25
+
26
+ 本机(8×H100,驱动 580.65,cu130)默认起 TP>1 会 `NCCL error: unhandled cuda error`。
27
+ 逐项扫下来:
28
+
29
+ ```
30
+ 什么都不设 -> 报错
31
+ NCCL_NVLS_ENABLE=0 -> OK ← 只需要这一个
32
+ NCCL_CUMEM_ENABLE=0 + NVLS=0 -> OK
33
+ NCCL_CUMEM_ENABLE=0 + SHM=1 -> 报错
34
+ NCCL_P2P_LEVEL=NVL -> 报错
35
+ ```
36
+
37
+ 坏的是 NVLink SHARP 组播(NVLS)的初始化。
38
+
39
+ **代价警告**:一开始我是一组一组加着试的,`NCCL_P2P_DISABLE=1` 跟着
40
+ `NCCL_SHM_DISABLE=1` 一起进去了,试出"能跑"就停了没做减法 —— 结果关掉 NVLink,
41
+ TP 的 all-reduce 退回 PCIe/主机中转,**吞吐从 2000 tok/s 掉到 124 tok/s(16 倍)**。
42
+ 凡是靠"加一组变量"试出来的解,必须回头逐个去掉验证哪个是真正必需的。
43
+
44
+ ## 3. flashinfer 需要 ninja
45
+
46
+ `FileNotFoundError: 'ninja'` —— flashinfer 要 JIT 编译采样 kernel。
47
+ 装 ninja 包不够(pip 装的没有可执行文件在 PATH 上),直接
48
+ `VLLM_USE_FLASHINFER_SAMPLER=0` 绕开最省事。
49
+
50
+ ## 4. vLLM 强制 spawn 时,平铺脚本会炸
51
+
52
+ vLLM 0.27 对部分模型强制用 spawn 起 worker,子进程会重新 import 父进程的 `__main__`。
53
+ 平铺脚本(逻辑全在模块层)被重新 import 就触发
54
+ `RuntimeError: freeze_support / Safe importing of main module`。
55
+
56
+ `aime_launch.py` 用一个 guard 绕开,**关键是 `runpy.run_path` 的 `run_name` 不能是
57
+ `"__main__"`** —— runpy 在执行期间会把 `sys.modules["__main__"]` 临时换成目标脚本,
58
+ spawn 读到的还是它,等于没绕开(第一版就栽在这)。用别的名字即可。
59
+
60
+ ## 5. verl 的序列打包硬依赖 flash_attn
61
+
62
+ `verl/utils/attention_utils.py:28` 在 CUDA 分支直接
63
+ `from flash_attn.bert_padding import ...`,**没有 fallback**,而且与
64
+ `attn_implementation` 无关 —— 就算模型跑 sdpa,只要 `use_remove_padding=True` 就会 import。
65
+
66
+ torch 2.11+cu130 没有 flash-attn 预编译 wheel(源码编译 1-2 小时),所以:
67
+ `use_remove_padding=False` + `attn_implementation=sdpa`。
68
+ 代价是 padding 到等长,显存吃满(7B 在 4×H100 上每卡约 71GB)、速度也慢。
69
+
70
+ `run_rose_online.sh` 把 `use_remove_padding=True` 写死在第 250 行且末尾没有 `"$@"` 透传,
71
+ 所以复制出 `run_rose_online_sdpa.sh`,末尾加三行可覆盖的开关(默认值与原件一致),
72
+ **原件一字未动**。
73
+
74
+ ## 6. 跨分词器本身不用改代码
75
+
76
+ `rose_agent_loop` 早就有这条通路,`TEACHER_TOKENIZER` 非 null 即启用:
77
+ 学生前缀 ids → decode 成文本 → 套**教师的** chat template → 教师续写 → 取回 `text`
78
+ → 用**学生的** tokenizer 重新 encode。
79
+
80
+ Olmo(vocab 100278)× Qwen3-32B(vocab 151669)实测:
81
+ - 拼接缝文本两边 round-trip 都无损
82
+ - 教师续写语义连贯,**没有重复吐 `<think>`**
83
+ - 格式错位是良性的:Olmo 模板在 prompt 尾部就打开 `<think>`,Qwen3 模板停在
84
+ `assistant\n`。教师看到「`assistant\n` + 推理文字」,续写出的 `</think>`+答案
85
+ 接在学生的 `<think>` 之后,正好构成 Olmo 期望的完整格式。
86
+
87
+ 验证脚本:`xjoin.py`(需要 teacher server 已起)。
88
+
89
+ ## 7. `TEACHER_MODEL` 不是路径
90
+
91
+ `bootstrap.sh teacher` 用 `--served-model-name teacher`,所以 API 的 model 字段就是
92
+ 字符串 `teacher`(`run_rose_online.sh:45` 的默认值)。传模型路径会 404。
93
+
94
+ ## 8. `SAVE_FREQ` 拦不住 final save
95
+
96
+ verl 在训练正常结束时**还会存一次**,与 `save_freq` 无关。3 步的冒烟测试也照存 33G
97
+ (Olmo-7B:model 分片 8.4G + optimizer 15.2G + HF 8.3G),当场把盘撑满。
98
+ 冒烟测试前先确认剩余空间,或者跑完立刻清。
eval/aime_eval.py ADDED
@@ -0,0 +1,223 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """AIME25 avg@k 评测。
2
+
3
+ 针对 non-thinking 模型: prompt 必须用 enable_thinking=False 渲染
4
+ (尾部会带 <think>\n\n</think>\n\n)。见 MEMORY: nothink-flag-for-1p7b-sft。
5
+
6
+ 答案抽取按优先级: \boxed{} -> "Answer: X" -> 文末最后一个整数。
7
+ AIME 答案恒为 0-999 的整数, 比较时按 int 比。
8
+ """
9
+ import argparse
10
+ import json
11
+ import os
12
+ import re
13
+ import sys
14
+ import time
15
+ from collections import Counter
16
+
17
+ p = argparse.ArgumentParser()
18
+ p.add_argument("--model", required=True)
19
+ p.add_argument("--data", default="/home/haojinw2/datasets/aime25/test.jsonl")
20
+ p.add_argument("--out", required=True)
21
+ p.add_argument("--n", type=int, default=16)
22
+ p.add_argument("--max-new", type=int, default=8192)
23
+ p.add_argument("--ctx", type=int, default=10240)
24
+ p.add_argument("--tp", type=int, default=1)
25
+ p.add_argument("--temp", type=float, default=0.7)
26
+ p.add_argument("--top-p", type=float, default=0.8)
27
+ p.add_argument("--top-k", type=int, default=20)
28
+ p.add_argument("--seed", type=int, default=0)
29
+ p.add_argument("--enable-thinking", type=lambda s: s.lower() == "true", default=False,
30
+ help="False = 关闭 think(non-thinking 模型必须用 False)")
31
+ p.add_argument("--gpu-util", type=float, default=0.85)
32
+ a = p.parse_args()
33
+
34
+
35
+ def log(*x):
36
+ print(f"[{time.strftime('%H:%M:%S')}]", *x, flush=True)
37
+
38
+
39
+ INSTR = "Please reason step by step, and put your final answer within \\boxed{}."
40
+
41
+ BOXED = re.compile(r"\\boxed\s*\{([^{}]*)\}")
42
+ ANSW = re.compile(r"[Aa]nswer\s*[::]\s*\$?(-?\d+)")
43
+ INTS = re.compile(r"-?\d+")
44
+
45
+
46
+ def _safe_int(s):
47
+ """AIME 答案恒为 0-999。超长数字是模型退化的产物(见过 28000 位),
48
+ 直接 int() 会撞 Python 3.11 的 int_max_str_digits=4300 抛 ValueError。"""
49
+ if s is None:
50
+ return None
51
+ t = s.lstrip("-")
52
+ if len(t) > 10 or not t:
53
+ return None
54
+ try:
55
+ return int(s)
56
+ except ValueError:
57
+ return None
58
+
59
+
60
+ def extract(text):
61
+ m = BOXED.findall(text)
62
+ if m:
63
+ digits = INTS.findall(m[-1])
64
+ if digits:
65
+ v = _safe_int(digits[-1])
66
+ if v is not None:
67
+ return v
68
+ m = ANSW.findall(text)
69
+ if m:
70
+ v = _safe_int(m[-1])
71
+ if v is not None:
72
+ return v
73
+ m = INTS.findall(text)
74
+ if m:
75
+ return _safe_int(m[-1])
76
+ return None
77
+
78
+
79
+ def _kill_descendants():
80
+ """递归杀掉本进程的所有子孙(vLLM worker 是独立进程, 不杀会变孤儿占满卡)。"""
81
+ import signal
82
+ import subprocess
83
+ me = os.getpid()
84
+ try:
85
+ out = subprocess.run(["ps", "-eo", "pid,ppid", "--no-headers"],
86
+ capture_output=True, text=True, timeout=20).stdout
87
+ except Exception:
88
+ return
89
+ children = {}
90
+ for ln in out.splitlines():
91
+ parts = ln.split()
92
+ if len(parts) != 2:
93
+ continue
94
+ try:
95
+ pid, ppid = int(parts[0]), int(parts[1])
96
+ except ValueError:
97
+ continue
98
+ children.setdefault(ppid, []).append(pid)
99
+ doomed, stack = [], [me]
100
+ while stack:
101
+ for c in children.get(stack.pop(), []):
102
+ doomed.append(c)
103
+ stack.append(c)
104
+ for pid in reversed(doomed): # 先叶子后根
105
+ try:
106
+ os.kill(pid, signal.SIGKILL)
107
+ except OSError:
108
+ pass
109
+
110
+
111
+ def _hard_exit(code=0):
112
+ sys.stdout.flush()
113
+ sys.stderr.flush()
114
+ _kill_descendants()
115
+ os._exit(code)
116
+
117
+
118
+ def _die(exc_type, exc, tb):
119
+ import traceback as _tb
120
+ _tb.print_exception(exc_type, exc, tb)
121
+ _hard_exit(1) # 异常路径同样要把 worker 一起带走
122
+
123
+
124
+ sys.excepthook = _die
125
+
126
+ rows = [json.loads(l) for l in open(a.data) if l.strip()]
127
+ log(f"{len(rows)} 题 x n={a.n}, enable_thinking={a.enable_thinking}")
128
+
129
+ from transformers import AutoTokenizer
130
+ from vllm import LLM, SamplingParams
131
+
132
+ tok = AutoTokenizer.from_pretrained(a.model, trust_remote_code=True)
133
+ prompts = []
134
+ for r in rows:
135
+ msgs = [{"role": "user", "content": r["problem"].strip() + "\n\n" + INSTR}]
136
+ prompts.append(tok.apply_chat_template(
137
+ msgs, add_generation_prompt=True, tokenize=False,
138
+ enable_thinking=a.enable_thinking))
139
+ log("prompt 尾部抽检: " + repr(prompts[0][-60:]))
140
+
141
+ def _fit_tp(model_path, tp):
142
+ """attn heads 必须能被 TP 整除, 否则 vLLM 起不来(R1-Distill-1.5B 是 12 heads)。"""
143
+ try:
144
+ cfg = json.load(open(os.path.join(model_path, "config.json")))
145
+ h = cfg.get("num_attention_heads")
146
+ except Exception:
147
+ return tp
148
+ if not h or h % tp == 0:
149
+ return tp
150
+ for cand in range(tp, 0, -1):
151
+ if h % cand == 0:
152
+ log(f"注意: {h} 个 attention head 不能被 TP={tp} 整除, 自动降为 TP={cand}")
153
+ return cand
154
+ return 1
155
+
156
+
157
+ a.tp = _fit_tp(a.model, a.tp)
158
+ llm = LLM(model=a.model, tensor_parallel_size=a.tp,
159
+ gpu_memory_utilization=a.gpu_util, max_model_len=a.ctx,
160
+ trust_remote_code=True, seed=a.seed)
161
+ # 不给 SamplingParams 传 seed: per-request seed 会让 n>1 退出批量采样快路径,
162
+ # 480 个 seeded 样本会逐个采样、慢到形同挂死。可复现性由 LLM(seed=) 保证。
163
+ sp = SamplingParams(n=a.n, temperature=a.temp, top_p=a.top_p, top_k=a.top_k,
164
+ max_tokens=a.max_new)
165
+
166
+ t0 = time.time()
167
+ outs = llm.generate(prompts, sp)
168
+ log(f"生成 {time.time()-t0:.0f}s")
169
+
170
+ recs, n_correct, n_tot, n_trunc, lens = [], 0, 0, 0, []
171
+ per_problem = []
172
+ for r, out in zip(rows, outs):
173
+ gt = int(str(r["answer"]).strip())
174
+ hits, samples = 0, []
175
+ for o in out.outputs:
176
+ pred = extract(o.text)
177
+ ok = (pred is not None and pred == gt)
178
+ hits += ok
179
+ n_correct += ok
180
+ n_tot += 1
181
+ n_trunc += (o.finish_reason == "length")
182
+ lens.append(len(o.token_ids))
183
+ samples.append({"pred": pred, "ok": bool(ok),
184
+ "finish": o.finish_reason, "len": len(o.token_ids),
185
+ "text": o.text})
186
+ per_problem.append({"id": r.get("id"), "gt": gt, "hits": hits, "n": len(out.outputs)})
187
+ recs.append({"id": r.get("id"), "problem": r["problem"], "gt": gt,
188
+ "hits": hits, "samples": samples})
189
+
190
+ os.makedirs(os.path.dirname(os.path.abspath(a.out)) or ".", exist_ok=True)
191
+ with open(a.out, "w") as f:
192
+ for rec in recs:
193
+ f.write(json.dumps(rec, ensure_ascii=False) + "\n")
194
+
195
+ avg = n_correct / n_tot
196
+ solved_any = sum(1 for x in per_problem if x["hits"] > 0)
197
+ solved_all = sum(1 for x in per_problem if x["hits"] == x["n"])
198
+ summary = {
199
+ "model": a.model, "data": a.data, "n": a.n,
200
+ "enable_thinking": a.enable_thinking,
201
+ "temperature": a.temp, "top_p": a.top_p, "top_k": a.top_k,
202
+ "max_new": a.max_new, "seed": a.seed,
203
+ "problems": len(rows), "samples": n_tot,
204
+ f"avg@{a.n}": avg,
205
+ "solved_any": solved_any, "solved_all": solved_all,
206
+ "truncated_rate": n_trunc / n_tot,
207
+ "resp_len_mean": sum(lens) / len(lens),
208
+ "resp_len_max": max(lens),
209
+ "hits_hist": dict(Counter(x["hits"] for x in per_problem)),
210
+ }
211
+ with open(a.out.replace(".jsonl", "_summary.json"), "w") as f:
212
+ json.dump(summary, f, indent=2, ensure_ascii=False)
213
+
214
+ print()
215
+ print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True)
216
+ print()
217
+ print(f"AIME25 avg@{a.n} = {avg:.4f} ({n_correct}/{n_tot})", flush=True)
218
+ print(f" 解出>=1次 {solved_any}/{len(rows)}, 全对 {solved_all}/{len(rows)}", flush=True)
219
+ print(f" 截断率 {n_trunc/n_tot:.3f}, 平均长度 {sum(lens)/len(lens):.0f}", flush=True)
220
+
221
+ # vLLM TP>1 的 worker 常在析构时挂尸、占着整卡不放, 后续任务就起不来。
222
+ # 结果已全部写盘, 直接跳过析构强制退出。
223
+ _hard_exit(0)
eval/aime_launch.py ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """spawn 安全的 aime_eval 启动器。
2
+
3
+ vllm 0.27 对 Qwen3.5 这类模型强制用 spawn 起 worker, 子进程会重新 import
4
+ 父进程的 __main__。aime_eval.py 是平铺脚本, 被重新 import 就会触发
5
+ multiprocessing 的 "freeze_support / Safe importing of main module" 报错。
6
+
7
+ 关键是 run_name 不能用 "__main__": runpy 在执行期间会把 sys.modules["__main__"]
8
+ 临时换成目标脚本, spawn 读到的就还是 aime_eval.py, 等于没绕开(先踩过这个)。
9
+ 用别的名字, sys.modules["__main__"] 始终是本文件, 子进程 import 到 __mp_main__,
10
+ 下面的 guard 不成立, 什么都不执行。aime_eval.py 的逻辑全在模块层, 不依赖 __name__。
11
+ """
12
+ import runpy
13
+ import sys
14
+
15
+ if __name__ == "__main__":
16
+ sys.argv[0] = "/home/haojinw2/aime_eval.py"
17
+ runpy.run_path("/home/haojinw2/aime_eval.py", run_name="__aime_eval__")
eval/hmmt_eval.py ADDED
@@ -0,0 +1,165 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """HMMT25 avg@k 评测。
2
+
3
+ 与 aime_eval.py 的差别只在答案比对: HMMT 的标准答案 30 题里只有 14 题是纯整数,
4
+ 其余是 LaTeX 表达式(\frac{1}{576}、\frac{9\sqrt{23}}{23}、1-\frac{2}{\pi}、
5
+ 甚至 "\frac{-1+\sqrt{17}}{2}, \frac{-1-\sqrt{17}}{2}" 这种多解)。
6
+ 按 int() 比会把 16 题全判错, 上限只有 14/30 —— 所以改用 math_verify 做符号等价判定。
7
+
8
+ 采样与输出格式跟 aime_eval.py 保持一致, 便于并排比较。
9
+ """
10
+ import argparse
11
+ import json
12
+ import os
13
+ import sys
14
+ import time
15
+ from collections import Counter
16
+
17
+ p = argparse.ArgumentParser()
18
+ p.add_argument("--model", required=True)
19
+ p.add_argument("--data", default="/home/haojinw2/datasets/hmmt25/test.jsonl")
20
+ p.add_argument("--out", required=True)
21
+ p.add_argument("--n", type=int, default=16)
22
+ p.add_argument("--max-new", type=int, default=31744)
23
+ p.add_argument("--ctx", type=int, default=33792)
24
+ p.add_argument("--tp", type=int, default=8)
25
+ p.add_argument("--temp", type=float, default=0.7)
26
+ p.add_argument("--top-p", type=float, default=0.95)
27
+ p.add_argument("--top-k", type=int, default=-1)
28
+ p.add_argument("--seed", type=int, default=0)
29
+ p.add_argument("--enable-thinking", type=lambda s: s.lower() == "true", default=True)
30
+ p.add_argument("--gpu-util", type=float, default=0.90)
31
+ a = p.parse_args()
32
+
33
+ INSTR = ("\n\nPlease reason step by step, and put your final answer within \\boxed{}.")
34
+
35
+
36
+ def _kill_descendants():
37
+ """递归杀掉所有子孙 —— vLLM worker 是独立进程, 主进程一退它们就变孤儿占满卡。"""
38
+ import signal
39
+ import subprocess
40
+ me = os.getpid()
41
+ try:
42
+ out = subprocess.run(["ps", "-eo", "pid,ppid"], capture_output=True, text=True, timeout=20).stdout
43
+ except Exception:
44
+ return
45
+ kids = {}
46
+ for line in out.splitlines()[1:]:
47
+ parts = line.split()
48
+ if len(parts) >= 2:
49
+ try:
50
+ kids.setdefault(int(parts[1]), []).append(int(parts[0]))
51
+ except ValueError:
52
+ pass
53
+ order, stack = [], [me]
54
+ while stack:
55
+ cur = stack.pop()
56
+ for k in kids.get(cur, []):
57
+ order.append(k)
58
+ stack.append(k)
59
+ for pid in reversed(order):
60
+ try:
61
+ os.kill(pid, signal.SIGKILL)
62
+ except Exception:
63
+ pass
64
+
65
+
66
+ def _hard_exit(code=0):
67
+ sys.stdout.flush()
68
+ sys.stderr.flush()
69
+ _kill_descendants()
70
+ os._exit(code)
71
+
72
+
73
+ sys.excepthook = lambda *e: (__import__("traceback").print_exception(*e), _hard_exit(1))
74
+
75
+ rows = [json.loads(l) for l in open(a.data) if l.strip()]
76
+ print(f"{len(rows)} 题, n={a.n}, max_new={a.max_new}, thinking={a.enable_thinking}", flush=True)
77
+
78
+ from transformers import AutoTokenizer
79
+ from vllm import LLM, SamplingParams
80
+
81
+ tok = AutoTokenizer.from_pretrained(a.model, trust_remote_code=True)
82
+
83
+
84
+ def render(problem):
85
+ msgs = [{"role": "user", "content": problem + INSTR}]
86
+ try:
87
+ return tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
88
+ enable_thinking=a.enable_thinking)
89
+ except TypeError:
90
+ return tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
91
+
92
+
93
+ prompts = [render(r["problem"]) for r in rows]
94
+
95
+ llm = LLM(model=a.model, tensor_parallel_size=a.tp, max_model_len=a.ctx,
96
+ gpu_memory_utilization=a.gpu_util, trust_remote_code=True, seed=a.seed)
97
+ sp = SamplingParams(n=a.n, temperature=a.temp, top_p=a.top_p, top_k=a.top_k,
98
+ max_tokens=a.max_new, seed=a.seed)
99
+ t0 = time.time()
100
+ outs = llm.generate(prompts, sp)
101
+ print(f"生成用时 {(time.time() - t0) / 60:.1f} 分钟", flush=True)
102
+
103
+ from math_verify import parse, verify
104
+
105
+
106
+ def is_correct(text, gt):
107
+ """math_verify 做符号等价判定; 解析不出来就退回去掉空白的字符串比对。"""
108
+ try:
109
+ pred = parse(text)
110
+ if pred and verify(parse("$" + str(gt) + "$"), pred):
111
+ return True
112
+ except Exception:
113
+ pass
114
+ import re
115
+ m = re.findall(r"\\boxed\{([^{}]*(?:\{[^{}]*\}[^{}]*)*)\}", text)
116
+ if m:
117
+ s = m[-1].replace(" ", "").replace("\\!", "").replace("\\,", "")
118
+ return s == str(gt).replace(" ", "")
119
+ return False
120
+
121
+
122
+ recs, per_problem, lens, n_trunc, n_correct, n_tot = [], [], [], 0, 0, 0
123
+ for r, out in zip(rows, outs):
124
+ gt = r["answer"]
125
+ samples, hits = [], 0
126
+ for o in out.outputs:
127
+ ok = is_correct(o.text, gt)
128
+ hits += ok
129
+ n_correct += ok
130
+ n_tot += 1
131
+ lens.append(len(o.token_ids))
132
+ n_trunc += o.finish_reason == "length"
133
+ samples.append({"ok": bool(ok), "finish": o.finish_reason,
134
+ "len": len(o.token_ids), "text": o.text})
135
+ per_problem.append({"id": r["id"], "gt": gt, "hits": hits, "n": len(out.outputs)})
136
+ recs.append({"id": r["id"], "problem": r["problem"], "gt": gt, "hits": hits, "samples": samples})
137
+
138
+ os.makedirs(os.path.dirname(os.path.abspath(a.out)) or ".", exist_ok=True)
139
+ with open(a.out, "w") as f:
140
+ for rec in recs:
141
+ f.write(json.dumps(rec, ensure_ascii=False) + "\n")
142
+
143
+ avg = n_correct / n_tot
144
+ summary = {
145
+ "model": a.model, "data": a.data, "n": a.n, "enable_thinking": a.enable_thinking,
146
+ "temperature": a.temp, "top_p": a.top_p, "max_new": a.max_new, "seed": a.seed,
147
+ "problems": len(rows), "samples": n_tot,
148
+ f"avg@{a.n}": avg,
149
+ "pass@1_first": sum(1 for x in recs if x["samples"][0]["ok"]) / len(recs),
150
+ "solved_any": sum(1 for x in per_problem if x["hits"] > 0),
151
+ "solved_all": sum(1 for x in per_problem if x["hits"] == x["n"]),
152
+ "truncated_rate": n_trunc / n_tot,
153
+ "resp_len_mean": sum(lens) / len(lens),
154
+ "hits_hist": dict(Counter(x["hits"] for x in per_problem)),
155
+ }
156
+ with open(a.out.replace(".jsonl", "_summary.json"), "w") as f:
157
+ json.dump(summary, f, indent=2, ensure_ascii=False)
158
+
159
+ print()
160
+ print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True)
161
+ print(f"\nHMMT25 avg@{a.n} = {avg:.4f} ({n_correct}/{n_tot})")
162
+ print(f" pass@1(第一条) {summary['pass@1_first']:.4f}")
163
+ print(f" 解出>=1次 {summary['solved_any']}/{len(rows)}, 全对 {summary['solved_all']}/{len(rows)}")
164
+ print(f" 截断率 {summary['truncated_rate']:.3f}, 平均长度 {summary['resp_len_mean']:.0f}", flush=True)
165
+ _hard_exit(0)
eval/merge_eval.py ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Merge data-parallel eval shards into one evalout directory.
3
+
4
+ python3 merge_eval.py OUT_DIR SHARD_DIR [SHARD_DIR ...]
5
+
6
+ Each shard is an evalout dir produced with IDX_START/IDX_END. Rows carry their
7
+ true dataframe idx, so merging is a concatenate + sort; the summary is then
8
+ recomputed with the same formulas rlve_eval.py uses, so the output file is
9
+ interchangeable with a single-process run.
10
+ """
11
+ import json, os, sys
12
+ from collections import Counter
13
+ from math import comb
14
+
15
+ def pass_at_k(N, c, k):
16
+ if N - c < k: return 1.0
17
+ return 1.0 - comb(N - c, k) / comb(N, k)
18
+
19
+ out, shards = sys.argv[1], sys.argv[2:]
20
+ rows, base = [], None
21
+ for sd in shards:
22
+ with open(f"{sd}/rollouts.jsonl") as f:
23
+ rows += [json.loads(l) for l in f]
24
+ if base is None:
25
+ base = json.load(open(f"{sd}/summary.json"))
26
+ seen = set()
27
+ for r in rows:
28
+ if r["idx"] in seen: sys.exit(f"重复 idx {r['idx']} -- 分片区间有重叠, 拒绝合并")
29
+ seen.add(r["idx"])
30
+ rows.sort(key=lambda r: r["idx"])
31
+
32
+ flat = [rec for r in rows for rec in r["recs"]]
33
+ n = len(flat)
34
+ N = base["n"]
35
+ per_env = {}
36
+ for r in rows:
37
+ per_env.setdefault(r["env"], []).extend(rec["accuracy"] for rec in r["recs"])
38
+ passk = {}
39
+ if N > 1:
40
+ for k in (1, 2, 4, 8, 16):
41
+ if k > N: continue
42
+ passk[f"pass@{k}"] = sum(
43
+ pass_at_k(N, sum(rec["accuracy"] for rec in r["recs"]), k) for r in rows) / len(rows)
44
+ passk["solved_any"] = sum(1 for r in rows if any(rec["accuracy"] for rec in r["recs"]))
45
+ passk["solved_all"] = sum(1 for r in rows if all(rec["accuracy"] for rec in r["recs"]))
46
+
47
+ summary = dict(base)
48
+ # base is shard A's summary, whose "tag" carries the shard suffix. Left as-is it
49
+ # labels a 180-problem merge as if it were the 90-problem shard, which silently
50
+ # corrupts any table keyed on tag.
51
+ summary["tag"] = os.path.basename(out.rstrip("/")).replace("rlve_", "", 1)
52
+ summary.update({
53
+ "problems": len(rows),
54
+ "accuracy@1": sum(r["accuracy"] for r in flat) / n,
55
+ "reward_mean@1": sum(r["reward"] for r in flat) / n,
56
+ "format_ok_rate": sum(r["format_score"] for r in flat) / n,
57
+ "truncated_rate": sum(r["finish"] == "length" for r in flat) / n,
58
+ "closed_think_rate": sum(r["closed_think"] for r in flat) / n,
59
+ "resp_len_mean": sum(r["len"] for r in flat) / n,
60
+ "resp_len_max": max(r["len"] for r in flat),
61
+ "reward_hist": dict(Counter(round(r["reward"], 3) for r in flat)),
62
+ "n_envs": len(per_env),
63
+ "merged_from": shards,
64
+ **passk,
65
+ })
66
+ os.makedirs(out, exist_ok=True)
67
+ json.dump(summary, open(f"{out}/summary.json", "w"), indent=2, ensure_ascii=False)
68
+ json.dump({k: sum(v)/len(v) for k, v in sorted(per_env.items())},
69
+ open(f"{out}/per_env.json", "w"), indent=2)
70
+ with open(f"{out}/rollouts.jsonl", "w") as f:
71
+ for r in rows: f.write(json.dumps(r, ensure_ascii=False) + "\n")
72
+ print(json.dumps({k: v for k, v in summary.items() if k != "reward_hist"}, indent=2, ensure_ascii=False))
eval/retrunc_eval.py ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Re-score existing rollouts under a smaller generation budget.
3
+
4
+ retrunc_eval.py <BUDGET> <SRC_TAG> <DST_TAG>
5
+
6
+ vLLM's max_tokens does not influence the sampling distribution of earlier
7
+ tokens, so the first BUDGET tokens of a 32k rollout are distributed exactly as a
8
+ BUDGET-token rollout would be. Truncating and re-scoring is an exact
9
+ re-measurement at the smaller budget, not an approximation -- it just costs no
10
+ GPU. finish_reason is fixed up by hand: a sample that ran past BUDGET would have
11
+ been cut off, so it becomes 'length'.
12
+
13
+ Scoring reuses env.verifier() on the decoded text, and the per-environment
14
+ config comes from the source parquet -- rollouts.jsonl does not carry it, and
15
+ verifier() without the right config does not mean the same thing.
16
+ """
17
+ import json, os, sys
18
+ from collections import Counter
19
+ from math import comb
20
+
21
+ GYM = "/home/haojinw2/rose-bundle-new/RLVE-Eval"
22
+ sys.path.insert(0, GYM)
23
+ import Gym.environments as G
24
+
25
+
26
+ def score_one(meta, text):
27
+ cls = getattr(G, f"{meta['environment']}_Environment", None)
28
+ if cls is None:
29
+ return None
30
+ env = cls()
31
+ env.set_config(meta.get("config", {}))
32
+ try:
33
+ return env.verifier(text)
34
+ except Exception as e: # noqa: BLE001
35
+ return {"reward": -1.0, "accuracy": 0, "format_score": 0, "err": repr(e)}
36
+
37
+
38
+ def pass_at_k(N, c, k):
39
+ if N - c < k:
40
+ return 1.0
41
+ return 1.0 - comb(N - c, k) / comb(N, k)
42
+
43
+
44
+ BUDGET, SRC, DST = int(sys.argv[1]), sys.argv[2], sys.argv[3]
45
+ D = "/home/haojinw2/evalout"
46
+ src, dst = f"{D}/rlve_{SRC}", f"{D}/rlve_{DST}"
47
+ base = json.load(open(f"{src}/summary.json"))
48
+
49
+ import pandas as pd
50
+ from transformers import AutoTokenizer
51
+ df = pd.read_parquet(base["data"])
52
+ tok = AutoTokenizer.from_pretrained(base["model"])
53
+
54
+ rows, cut, kept = [], 0, 0
55
+ for line in open(f"{src}/rollouts.jsonl"):
56
+ d = json.loads(line)
57
+ meta = json.loads(df.iloc[d["idx"]]["extra_info"]["metadata"])
58
+ assert meta["environment"] == d["env"], (d["idx"], meta["environment"], d["env"])
59
+ recs = []
60
+ for i, ids in enumerate(d["ids"]):
61
+ if len(ids) <= BUDGET:
62
+ recs.append(dict(d["recs"][i])); kept += 1
63
+ continue
64
+ txt = tok.decode(ids[:BUDGET], skip_special_tokens=False)
65
+ v = score_one(meta, txt) or {"reward": 0.0, "accuracy": 0, "format_score": 0}
66
+ r = dict(v); r["len"] = BUDGET; r["finish"] = "length"
67
+ r["closed_think"] = "</think>" in txt
68
+ recs.append(r); cut += 1
69
+ rows.append({"idx": d["idx"], "env": d["env"], "gt": d["gt"], "recs": recs})
70
+
71
+ flat = [x for r in rows for x in r["recs"]]
72
+ n, N = len(flat), base["n"]
73
+ per_env = {}
74
+ for r in rows:
75
+ per_env.setdefault(r["env"], []).extend(x["accuracy"] for x in r["recs"])
76
+ passk = {}
77
+ if N > 1:
78
+ for k in (1, 2, 4, 8, 16):
79
+ if k > N: continue
80
+ passk[f"pass@{k}"] = sum(
81
+ pass_at_k(N, sum(x["accuracy"] for x in r["recs"]), k) for r in rows) / len(rows)
82
+ passk["solved_any"] = sum(1 for r in rows if any(x["accuracy"] for x in r["recs"]))
83
+ passk["solved_all"] = sum(1 for r in rows if all(x["accuracy"] for x in r["recs"]))
84
+
85
+ s = dict(base)
86
+ s.update({"tag": DST, "max_new": BUDGET, "ctx": BUDGET + 2048, "problems": len(rows),
87
+ "accuracy@1": sum(x["accuracy"] for x in flat) / n,
88
+ "reward_mean@1": sum(x["reward"] for x in flat) / n,
89
+ "format_ok_rate": sum(x["format_score"] for x in flat) / n,
90
+ "truncated_rate": sum(x["finish"] == "length" for x in flat) / n,
91
+ "closed_think_rate": sum(x["closed_think"] for x in flat) / n,
92
+ "resp_len_mean": sum(x["len"] for x in flat) / n,
93
+ "resp_len_max": max(x["len"] for x in flat),
94
+ "reward_hist": dict(Counter(round(x["reward"], 3) for x in flat)),
95
+ "n_envs": len(per_env), "retruncated_from": SRC,
96
+ "n_rescored": cut, "n_unchanged": kept, **passk})
97
+ os.makedirs(dst, exist_ok=True)
98
+ json.dump(s, open(f"{dst}/summary.json", "w"), indent=2, ensure_ascii=False)
99
+ json.dump({k: sum(v)/len(v) for k, v in sorted(per_env.items())},
100
+ open(f"{dst}/per_env.json", "w"), indent=2)
101
+ with open(f"{dst}/rollouts.jsonl", "w") as f:
102
+ for r in rows: f.write(json.dumps(r, ensure_ascii=False) + "\n")
103
+ print(" %-20s 重判 %5d / 保留 %5d avg@8 %.4f pass@8 %.4f 解出 %3d fmt %.3f trunc %.3f"
104
+ % (DST, cut, kept, s["accuracy@1"], s.get("pass@8", float("nan")),
105
+ s.get("solved_any", -1), s["format_ok_rate"], s["truncated_rate"]))
eval/rlve_eval.py ADDED
@@ -0,0 +1,212 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """mean@1 evaluation of a checkpoint on data/rlve_test.parquet.
2
+
3
+ Separate from training on purpose. The reward logged during ROSE-online scores
4
+ the *spliced* sequence (student prefix + teacher continuation), which never
5
+ reaches an answer -- see recipe/rose/rlve_reward.py. This scores the student.
6
+
7
+ Context is 16384 tokens end to end: max_tokens is computed per prompt as
8
+ CTX - len(prompt_ids) - 8, so nothing gets a larger window than the flag names.
9
+
10
+ Scores come from `env.verifier()` directly rather than recipe/rose/rlve_reward's
11
+ compute_score, which throws away two of the three fields. verifier returns
12
+ reward / accuracy / format_score, and `accuracy` is the metric that survives
13
+ environments disagreeing on reward scale -- they do. The bundle docstring claims
14
+ "-0.5 = answered and wrong"; BannedPointSupersetPathCounting actually uses
15
+ {correct +1.0, wrong_answer 0.0, wrong_range -0.5, wrong_format -1.0}.
16
+
17
+ Everything runs under `if __name__ == "__main__"` because vLLM spawns its engine
18
+ core with the spawn start method, which re-imports this module in the child.
19
+ Module-level work here means the child tries to build a second LLM during import
20
+ and dies in `_check_not_importing_main`.
21
+
22
+ MODEL=<hf dir> TAG=<name> [TP=4] [N=1] [CTX=16384] python3 rlve_eval.py
23
+ """
24
+
25
+ import json
26
+ import os
27
+ import sys
28
+ import time
29
+ from collections import Counter
30
+
31
+ GYM = "/home/haojinw2/rose-bundle-new/RLVE-Eval"
32
+ sys.path.insert(0, GYM)
33
+
34
+ import Gym.environments as G # noqa: E402
35
+
36
+
37
+ def score_one(meta, text):
38
+ cls = getattr(G, f"{meta['environment']}_Environment", None)
39
+ if cls is None:
40
+ return None
41
+ env = cls()
42
+ env.set_config(meta.get("config", {}))
43
+ try:
44
+ return env.verifier(text)
45
+ except Exception as e: # noqa: BLE001
46
+ return {"reward": -1.0, "accuracy": 0, "format_score": 0, "err": repr(e)}
47
+
48
+
49
+ def main():
50
+ CTX = int(os.environ.get("CTX", 16384))
51
+ TP = int(os.environ.get("TP", 4))
52
+ N = int(os.environ.get("N", 1))
53
+ MODEL = os.environ["MODEL"]
54
+ TAG = os.environ.get("TAG", os.path.basename(MODEL.rstrip("/")))
55
+ DATA = os.environ.get("DATA", "/home/haojinw2/rose-bundle-new/data/rlve_test.parquet")
56
+ OUT = os.environ.get("OUT", f"/home/haojinw2/evalout/rlve_{TAG}")
57
+ MAX_NEW = int(os.environ.get("MAX_NEW", 0)) # 0 = use whatever CTX leaves
58
+ os.makedirs(OUT, exist_ok=True)
59
+
60
+ import pandas as pd
61
+ from transformers import AutoTokenizer
62
+ from vllm import LLM, SamplingParams
63
+
64
+ df = pd.read_parquet(DATA)
65
+ tok = AutoTokenizer.from_pretrained(MODEL)
66
+
67
+ prompts, keep = [], []
68
+ # Data-parallel slicing. Two halves on 0-3 and 4-7 beat TP=8 on a 1.7B --
69
+ # no collectives, twice the KV cache. `idx` stays the true dataframe row,
70
+ # so merge_eval.py can concatenate the shards without renumbering.
71
+ IDX_START = int(os.environ.get("IDX_START", 0))
72
+ IDX_END = int(os.environ.get("IDX_END", 0)) or len(df)
73
+ for i in range(IDX_START, IDX_END):
74
+ row = df.iloc[i]
75
+ msgs = [dict(m) for m in row["prompt"]]
76
+ ids = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True)
77
+ budget = CTX - len(ids) - 8
78
+ if MAX_NEW:
79
+ # fixed generation budget (train-set sampling), not "whatever the
80
+ # context leaves over" -- CTX still caps it so nothing overflows
81
+ budget = min(budget, MAX_NEW)
82
+ if budget < 256:
83
+ # no room to answer is a data problem, not a model failure -- drop it
84
+ # loudly instead of scoring it as a format failure
85
+ print(f"!! row {i}: prompt {len(ids)} tok leaves {budget}, skipped")
86
+ continue
87
+ prompts.append(tok.decode(ids))
88
+ keep.append((i, budget))
89
+
90
+ print(f"{len(prompts)}/{len(df)} 题, ctx={CTX}, n={N}, tp={TP}", flush=True)
91
+
92
+ llm = LLM(
93
+ model=MODEL,
94
+ tensor_parallel_size=TP,
95
+ max_model_len=CTX,
96
+ gpu_memory_utilization=float(os.environ.get("GPU_UTIL", 0.85)),
97
+ trust_remote_code=True,
98
+ seed=int(os.environ.get("EVAL_SEED", "0")),
99
+ )
100
+
101
+ # one SamplingParams per prompt so each gets its own exact budget
102
+ # 0.7 / 0.9, no top-k -- the project standard, matching the reference eval
103
+ # script's argparse defaults. Training rollout uses the same values so the
104
+ # states ROSE distils on are the states the model actually visits here.
105
+ TEMP = float(os.environ.get("TEMP", 0.7))
106
+ TOP_P = float(os.environ.get("TOP_P", 0.9))
107
+ TOP_K = int(os.environ.get("TOP_K", -1))
108
+ print(f"sampling: temp={TEMP} top_p={TOP_P} top_k={TOP_K}", flush=True)
109
+ sps = [
110
+ SamplingParams(n=N, temperature=TEMP, top_p=TOP_P, top_k=TOP_K,
111
+ max_tokens=b, seed=int(os.environ.get("EVAL_SEED", "0")))
112
+ for _, b in keep
113
+ ]
114
+
115
+ t0 = time.time()
116
+ outs = llm.generate(prompts, sps)
117
+ print(f"生成 {time.time() - t0:.0f}s", flush=True)
118
+
119
+ rows = []
120
+ per_env = {}
121
+ for (idx, _), o in zip(keep, outs):
122
+ row = df.iloc[idx]
123
+ meta = json.loads(row["extra_info"]["metadata"])
124
+ recs = []
125
+ for c in o.outputs:
126
+ v = score_one(meta, c.text) or {"reward": 0.0, "accuracy": 0, "format_score": 0}
127
+ v["len"] = len(c.token_ids)
128
+ v["finish"] = c.finish_reason
129
+ v["closed_think"] = "</think>" in c.text
130
+ recs.append(v)
131
+ rows.append(
132
+ {
133
+ "idx": int(idx),
134
+ "env": meta["environment"],
135
+ "gt": row["reward_model"]["ground_truth"],
136
+ "recs": recs,
137
+ "text": [c.text for c in o.outputs],
138
+ # exact ids, so a downstream teacher-continuation step can join
139
+ # id-to-id instead of re-tokenising the decoded text (which can
140
+ # shift the seam -- the one silent failure in this design)
141
+ "ids": [list(c.token_ids) for c in o.outputs],
142
+ }
143
+ )
144
+ per_env.setdefault(meta["environment"], []).append(
145
+ 1.0 if any(r["accuracy"] for r in recs) else 0.0
146
+ )
147
+
148
+ flat = [r for row in rows for r in row["recs"]]
149
+ n = len(flat)
150
+
151
+ # Unbiased pass@k (Chen et al. 2021): with c of N samples correct,
152
+ # pass@k = 1 - C(N-c, k) / C(N, k)
153
+ # Reporting the whole curve costs nothing once the samples exist, and
154
+ # pass@1 from it is the same quantity as accuracy@1 -- a useful self-check.
155
+ from math import comb
156
+ def pass_at_k(N, c, k):
157
+ if N - c < k:
158
+ return 1.0
159
+ return 1.0 - comb(N - c, k) / comb(N, k)
160
+
161
+ passk = {}
162
+ if N > 1:
163
+ for k in [1, 2, 4, 8, 16]:
164
+ if k > N:
165
+ continue
166
+ passk[f"pass@{k}"] = sum(
167
+ pass_at_k(N, sum(r["accuracy"] for r in row["recs"]), k) for row in rows
168
+ ) / len(rows)
169
+ # how many problems are solved by at least one sample, ever
170
+ passk["solved_any"] = sum(
171
+ 1 for row in rows if any(r["accuracy"] for r in row["recs"])
172
+ )
173
+ passk["solved_all"] = sum(
174
+ 1 for row in rows if all(r["accuracy"] for r in row["recs"])
175
+ )
176
+
177
+ summary = {
178
+ "tag": TAG,
179
+ "model": MODEL,
180
+ "ctx": CTX,
181
+ "max_new": MAX_NEW or None,
182
+ "data": DATA,
183
+ "n": N,
184
+ "temperature": TEMP,
185
+ "top_p": TOP_P,
186
+ "top_k": TOP_K,
187
+ "problems": len(rows),
188
+ "accuracy@1": sum(r["accuracy"] for r in flat) / n,
189
+ "reward_mean@1": sum(r["reward"] for r in flat) / n,
190
+ "format_ok_rate": sum(r["format_score"] for r in flat) / n,
191
+ "truncated_rate": sum(r["finish"] == "length" for r in flat) / n,
192
+ "closed_think_rate": sum(r["closed_think"] for r in flat) / n,
193
+ "resp_len_mean": sum(r["len"] for r in flat) / n,
194
+ "resp_len_max": max(r["len"] for r in flat),
195
+ "reward_hist": dict(Counter(round(r["reward"], 3) for r in flat)),
196
+ "n_envs": len(per_env),
197
+ **passk,
198
+ }
199
+ print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True)
200
+
201
+ with open(f"{OUT}/summary.json", "w") as f:
202
+ json.dump(summary, f, indent=2, ensure_ascii=False)
203
+ with open(f"{OUT}/per_env.json", "w") as f:
204
+ json.dump({k: sum(v) / len(v) for k, v in sorted(per_env.items())}, f, indent=2)
205
+ with open(f"{OUT}/rollouts.jsonl", "w") as f:
206
+ for r in rows:
207
+ f.write(json.dumps(r, ensure_ascii=False) + "\n")
208
+ print(f"-> {OUT}", flush=True)
209
+
210
+
211
+ if __name__ == "__main__":
212
+ main()
frameworks/agentgym_rl/.gitignore ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ **/*.pt
2
+ **/checkpoints
3
+ **/wget-log
4
+ **/_build/
5
+ **/*.ckpt
6
+ **/outputs
7
+ **/*.tar.gz
8
+ **/playground
9
+ **/wandb
10
+
11
+ # Byte-compiled / optimized / DLL files
12
+ __pycache__/
13
+ *.py[cod]
14
+ *$py.class
15
+ dataset/*
16
+ tensorflow/my_graph/*
17
+ .idea/
18
+ # C extensions
19
+ *.so
20
+
21
+ # Distribution / packaging
22
+ .Python
23
+ env/
24
+ build/
25
+ develop-eggs/
26
+ dist/
27
+ downloads/
28
+ eggs/
29
+ .eggs/
30
+ lib/
31
+ lib64/
32
+ parts/
33
+ sdist/
34
+ var/
35
+ *.egg-info/
36
+ .installed.cfg
37
+ *.egg
38
+
39
+ # PyInstaller
40
+ # Usually these files are written by a python script from a template
41
+ # before PyInstaller builds the exe, so as to inject date/other infos into it.
42
+ *.manifest
43
+ *.spec
44
+
45
+ # Installer logs
46
+ pip-log.txt
47
+ pip-delete-this-directory.txt
48
+
49
+ # Unit test / coverage reports
50
+ htmlcov/
51
+ .tox/
52
+ .coverage
53
+ .coverage.*
54
+ .cache
55
+ nosetests.xml
56
+ coverage.xml
57
+ *,cover
58
+ .hypothesis/
59
+
60
+ # Translations
61
+ *.mo
62
+ *.pot
63
+
64
+ # Django stuff:
65
+ *.log
66
+ local_settings.py
67
+
68
+ # Flask stuff:
69
+ instance/
70
+ .webassets-cache
71
+
72
+ # Scrapy stuff:
73
+ .scrapy
74
+
75
+ # Sphinx documentation
76
+ docs/_build/
77
+
78
+ # PyBuilder
79
+ target/
80
+
81
+ # IPython Notebook
82
+ .ipynb_checkpoints
83
+
84
+ # pyenv
85
+ .python-version
86
+
87
+ # celery beat schedule file
88
+ celerybeat-schedule
89
+
90
+ # dotenv
91
+ .env
92
+
93
+ # virtualenv
94
+ venv/
95
+ ENV/
96
+
97
+ # Spyder project settings
98
+ .spyderproject
99
+
100
+ # Rope project settings
101
+ .ropeproject
102
+
103
+ # vscode
104
+ .vscode
105
+
106
+ # Mac
107
+ .DS_Store
108
+
109
+ # output logs
110
+ tests/e2e/toy_examples/deepspeed/synchronous/output.txt
111
+
112
+ # vim
113
+ *.swp
114
+
115
+ # ckpt
116
+ *.lock
frameworks/agentgym_rl/.readthedocs.yaml ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Read the Docs configuration file
2
+ # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
3
+
4
+ version: 2
5
+
6
+ build:
7
+ os: ubuntu-22.04
8
+ tools:
9
+ python: "3.8"
10
+
11
+ sphinx:
12
+ configuration: docs/conf.py
13
+
14
+ python:
15
+ install:
16
+ - requirements: docs/requirements-docs.txt
frameworks/agentgym_rl/.style.yapf ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ [style]
2
+ based_on_style = google
3
+ column_limit = 120
4
+ indent_width = 4
5
+ split_arguments_when_comma_terminated: true
frameworks/agentgym_rl/Notice.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ Copyright 2023-2024 Bytedance Ltd. and/or its affiliates
frameworks/agentgym_rl/VERL_LICENSE ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright [yyyy] [name of copyright owner]
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
frameworks/agentgym_rl/VERL_README.md ADDED
@@ -0,0 +1,135 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <h1 style="text-align: center;">verl: Volcano Engine Reinforcement Learning for LLM</h1>
2
+
3
+ verl is a flexible, efficient and production-ready RL training library for large language models (LLMs).
4
+
5
+ verl is the open-source version of **[HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2)** paper.
6
+
7
+ verl is flexible and easy to use with:
8
+
9
+ - **Easy extension of diverse RL algorithms**: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code.
10
+
11
+ - **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks.
12
+
13
+ - **Flexible device mapping**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.
14
+
15
+ - Readily integration with popular HuggingFace models
16
+
17
+
18
+ verl is fast with:
19
+
20
+ - **State-of-the-art throughput**: By seamlessly integrating existing SOTA LLM training and inference frameworks, verl achieves high generation and training throughput.
21
+
22
+ - **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.
23
+
24
+ <p align="center">
25
+ | <a href="https://verl.readthedocs.io/en/latest/index.html"><b>Documentation</b></a> | <a href="https://arxiv.org/abs/2409.19256v2"><b>Paper</b></a> | <a href="https://join.slack.com/t/verlgroup/shared_invite/zt-2w5p9o4c3-yy0x2Q56s_VlGLsJ93A6vA"><b>Slack</b></a> | <a href="https://raw.githubusercontent.com/eric-haibin-lin/verl-community/refs/heads/main/WeChat.JPG"><b>Wechat</b></a> | <a href="https://x.com/verl_project"><b>Twitter</b></a>
26
+
27
+ <!-- <a href=""><b>Slides</b></a> | -->
28
+ </p>
29
+
30
+ ## News
31
+
32
+ - [2025/2] We will present verl in the [Bytedance/NVIDIA/Anyscale Ray Meetup](https://lu.ma/ji7atxux) in bay area on Feb 13th. Come join us in person!
33
+ - [2025/1] [Doubao-1.5-pro](https://team.doubao.com/zh/special/doubao_1_5_pro) is released with SOTA-level performance on LLM & VLM. The RL scaling preview model is trained using verl, reaching OpenAI O1-level performance on math benchmarks (70.0 pass@1 on AIME).
34
+ - [2024/12] The team presented <a href="https://neurips.cc/Expo/Conferences/2024/workshop/100677">Post-training LLMs: From Algorithms to Infrastructure</a> at NeurIPS 2024. [Slides](https://github.com/eric-haibin-lin/verl-data/tree/neurips) and [video](https://neurips.cc/Expo/Conferences/2024/workshop/100677) available.
35
+ - [2024/10] verl is presented at Ray Summit. [Youtube video](https://www.youtube.com/watch?v=MrhMcXkXvJU&list=PLzTswPQNepXntmT8jr9WaNfqQ60QwW7-U&index=37) available.
36
+ - [2024/08] HybridFlow (verl) is accepted to EuroSys 2025.
37
+
38
+ ## Key Features
39
+
40
+ - **FSDP** and **Megatron-LM** for training.
41
+ - **vLLM** and **TGI** for rollout generation, **SGLang** support coming soon.
42
+ - huggingface models support
43
+ - Supervised fine-tuning
44
+ - Reinforcement learning from human feedback with [PPO](https://github.com/volcengine/verl/tree/main/examples/ppo_trainer), [GRPO](https://github.com/volcengine/verl/tree/main/examples/grpo_trainer), and [ReMax](https://github.com/volcengine/verl/tree/main/examples/remax_trainer)
45
+ - Support model-based reward and function-based reward (verifiable reward)
46
+ - flash-attention, [sequence packing](examples/ppo_trainer/run_qwen2-7b_seq_balance.sh), [long context](examples/ppo_trainer/run_deepseek7b_llm_sp2.sh) support via DeepSpeed Ulysses, [LoRA](examples/sft/gsm8k/run_qwen_05_peft.sh), [Liger-kernel](examples/sft/gsm8k/run_qwen_05_sp2_liger.sh)
47
+ - scales up to 70B models and hundreds of GPUs
48
+ - experiment tracking with wandb, swanlab and mlflow
49
+
50
+ ## Upcoming Features
51
+ - Reward model training
52
+ - DPO training
53
+ - DeepSeek integration with Megatron backend
54
+ - SGLang integration
55
+
56
+ ## Getting Started
57
+
58
+ Checkout this [Jupyter Notebook](https://github.com/volcengine/verl/tree/main/examples/ppo_trainer/verl_getting_started.ipynb) to get started with PPO training with a single 24GB L4 GPU (**FREE** GPU quota provided by [Lighting Studio](https://lightning.ai/hlin-verl/studios/verl-getting-started))!
59
+
60
+ **Quickstart:**
61
+ - [Installation](https://verl.readthedocs.io/en/latest/start/install.html)
62
+ - [Quickstart](https://verl.readthedocs.io/en/latest/start/quickstart.html)
63
+ - [Programming Guide](https://verl.readthedocs.io/en/latest/hybrid_flow.html)
64
+
65
+ **Running a PPO example step-by-step:**
66
+ - Data and Reward Preparation
67
+ - [Prepare Data for Post-Training](https://verl.readthedocs.io/en/latest/preparation/prepare_data.html)
68
+ - [Implement Reward Function for Dataset](https://verl.readthedocs.io/en/latest/preparation/reward_function.html)
69
+ - Understanding the PPO Example
70
+ - [PPO Example Architecture](https://verl.readthedocs.io/en/latest/examples/ppo_code_architecture.html)
71
+ - [Config Explanation](https://verl.readthedocs.io/en/latest/examples/config.html)
72
+ - [Run GSM8K Example](https://verl.readthedocs.io/en/latest/examples/gsm8k_example.html)
73
+
74
+ **Reproducible algorithm baselines:**
75
+ - [PPO and GRPO](https://verl.readthedocs.io/en/latest/experiment/ppo.html)
76
+
77
+ **For code explanation and advance usage (extension):**
78
+ - PPO Trainer and Workers
79
+ - [PPO Ray Trainer](https://verl.readthedocs.io/en/latest/workers/ray_trainer.html)
80
+ - [PyTorch FSDP Backend](https://verl.readthedocs.io/en/latest/workers/fsdp_workers.html)
81
+ - [Megatron-LM Backend](https://verl.readthedocs.io/en/latest/index.html)
82
+ - Advance Usage and Extension
83
+ - [Ray API design tutorial](https://verl.readthedocs.io/en/latest/advance/placement.html)
84
+ - [Extend to Other RL(HF) algorithms](https://verl.readthedocs.io/en/latest/advance/dpo_extension.html)
85
+ - [Add Models with the FSDP Backend](https://verl.readthedocs.io/en/latest/advance/fsdp_extension.html)
86
+ - [Add Models with the Megatron-LM Backend](https://verl.readthedocs.io/en/latest/advance/megatron_extension.html)
87
+ - [Deployment using Separate GPU Resources](https://github.com/volcengine/verl/tree/main/examples/split_placement)
88
+
89
+ ## Performance Tuning Guide
90
+ The performance is essential for on-policy RL algorithm. We write a detailed performance tuning guide to allow people tune the performance. See [here](https://verl.readthedocs.io/en/latest/perf/perf_tuning.html) for more details.
91
+
92
+ ## vLLM v0.7 testing version
93
+ We have released a testing version of veRL that supports vLLM>=0.7.0. Please refer to [this document](https://github.com/volcengine/verl/docs/README_vllm0.7.md) for installation guide and more information.
94
+
95
+ ## Contribution Guide
96
+ Contributions from the community are welcome!
97
+
98
+ ### Code formatting
99
+ We use yapf (Google style) to enforce strict code formatting when reviewing PRs. To reformat you code locally, make sure you installed **latest** `yapf`
100
+ ```bash
101
+ pip3 install yapf --upgrade
102
+ ```
103
+ Then, make sure you are at top level of verl repo and run
104
+ ```bash
105
+ bash scripts/format.sh
106
+ ```
107
+
108
+ ## Citation and acknowledgement
109
+
110
+ If you find the project helpful, please cite:
111
+ - [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2)
112
+ - [A Framework for Training Large Language Models for Code Generation via Proximal Policy Optimization](https://i.cs.hku.hk/~cwu/papers/gmsheng-NL2Code24.pdf)
113
+
114
+ ```tex
115
+ @article{sheng2024hybridflow,
116
+ title = {HybridFlow: A Flexible and Efficient RLHF Framework},
117
+ author = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
118
+ year = {2024},
119
+ journal = {arXiv preprint arXiv: 2409.19256}
120
+ }
121
+ ```
122
+
123
+ verl is inspired by the design of Nemo-Aligner, Deepspeed-chat and OpenRLHF. The project is adopted and supported by Anyscale, Bytedance, LMSys.org, Shanghai AI Lab, Tsinghua University, UC Berkeley, UCLA, UIUC, and University of Hong Kong.
124
+
125
+ ## Awesome work using verl
126
+ - [Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization](https://arxiv.org/abs/2410.09302)
127
+ - [Flaming-hot Initiation with Regular Execution Sampling for Large Language Models](https://arxiv.org/abs/2410.21236)
128
+ - [Process Reinforcement Through Implicit Rewards](https://github.com/PRIME-RL/PRIME/)
129
+ - [TinyZero](https://github.com/Jiayi-Pan/TinyZero): a reproduction of DeepSeek R1 Zero recipe for reasoning tasks
130
+ - [RAGEN](https://github.com/ZihanWang314/ragen): a general-purpose reasoning agent training framework
131
+ - [Logic R1](https://github.com/Unakar/Logic-RL): a reproduced DeepSeek R1 Zero on 2K Tiny Logic Puzzle Dataset.
132
+ - [deepscaler](https://github.com/agentica-project/deepscaler): iterative context scaling with GRPO
133
+ - [critic-rl](https://github.com/HKUNLP/critic-rl): Teaching Language Models to Critique via Reinforcement Learning
134
+
135
+ We are HIRING! Send us an [email](mailto:haibin.lin@bytedance.com) if you are interested in internship/FTE opportunities in MLSys/LLM reasoning/multimodal alignment.
frameworks/agentgym_rl/docs/Makefile ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Minimal makefile for Sphinx documentation
2
+ #
3
+
4
+ # You can set these variables from the command line.
5
+ SPHINXOPTS =
6
+ SPHINXBUILD = sphinx-build
7
+ SPHINXPROJ = verl
8
+ SOURCEDIR = .
9
+ BUILDDIR = _build
10
+
11
+ # Put it first so that "make" without argument is like "make help".
12
+ help:
13
+ @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
14
+
15
+ .PHONY: help Makefile
16
+
17
+ # Catch-all target: route all unknown targets to Sphinx using the new
18
+ # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
19
+ %: Makefile
20
+ @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
frameworks/agentgym_rl/docs/README.md ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # verl documents
2
+
3
+ ## Build the docs
4
+
5
+ ```bash
6
+ # Install dependencies.
7
+ pip install -r requirements-docs.txt
8
+
9
+ # Build the docs.
10
+ make clean
11
+ make html
12
+ ```
13
+
14
+ ## Open the docs with your browser
15
+
16
+ ```bash
17
+ python -m http.server -d _build/html/
18
+ ```
19
+ Launch your browser and open localhost:8000.
frameworks/agentgym_rl/docs/README_vllm0.7.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Readme for verl(vllm>=0.7) version
2
+
3
+ ## Installation
4
+
5
+ Note: This version of veRL supports **FSDP** for training and **vLLM** for rollout. (Megatron-LM is not supported yet.)
6
+
7
+ ```
8
+ # Create the conda environment
9
+ conda create -n verl python==3.10
10
+ conda activate verl
11
+
12
+ # Install verl
13
+ git clone https://github.com/volcengine/verl.git
14
+ cd verl
15
+ pip3 install -e .
16
+
17
+ # Install vLLM>=0.7
18
+ # (Option1) pip3 install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
19
+ # (Option2) pip3 install "vllm>=0.7.0"
20
+
21
+ # Install flash-attn
22
+ pip3 install flash-attn --no-build-isolation
23
+
24
+ ```
25
+
26
+ Note that if you are installing stable versions of vLLM (Option2), you need to make some tiny patches manually on vllm (/path/to/site-packages/vllm after installation) after the above steps:
27
+
28
+ - vllm/distributed/parallel_state.py: Remove the assertion below:
29
+
30
+ ```
31
+ if (world_size
32
+ != tensor_model_parallel_size * pipeline_model_parallel_size):
33
+ raise RuntimeError(
34
+ f"world_size ({world_size}) is not equal to "
35
+ f"tensor_model_parallel_size ({tensor_model_parallel_size}) x "
36
+ f"pipeline_model_parallel_size ({pipeline_model_parallel_size})")
37
+
38
+ ```
39
+
40
+ - vllm/executor/uniproc_executor.py: change `local_rank = rank` to `local_rank = int(os.environ["LOCAL_RANK"])`
41
+ - vllm/model_executor/model_loader/weight_utils.py: remove the `torch.cuda.empty_cache()` in `pt_weights_iterator`
42
+
43
+ These modifications have already been merged into the main branch of vLLM. Thus nightly vLLM or building vLLM from source do not need these patches.
44
+
45
+ ## Features
46
+
47
+ ### Use cuda graph
48
+
49
+ After installation, examples using FSDP as training backends can be used. By default, the `enforce_eager` is set to True, which disables the cuda graph. To enjoy cuda graphs and the sleep mode of vLLM>=0.7, add the following lines to the bash script:
50
+
51
+ ```
52
+ actor_rollout_ref.rollout.enforce_eager=False \
53
+ actor_rollout_ref.rollout.free_cache_engine=False \
54
+
55
+ ```
56
+
57
+ For a typical job like examples/ppo_trainer/run_qwen2-7b_seq_balance.sh, the rollout generation time is 115 seconds with vLLM0.6.3, while it is 85 seconds with vLLM0.7.0. By enabling the cudagraph, the generation duration is further reduced to 62 seconds.
58
+
59
+ **Note:** Currently, if the `n` is greater than 1 in `SamplingParams` in vLLM>=0.7, there is a potential performance issue on the stability of rollout generation time (Some iterations would see generation time bursts). We are working with the vLLM team to check this issue.
60
+
61
+ ### Other features in vLLM
62
+
63
+ 1. **num_scheduler_step>1:** not supported yet (weight loading has not been aligned with `MultiStepModelRunner`)
64
+ 2. **Prefix caching:** not supported yet (vLLM sleep mode does not support prefix caching)
65
+ 3. **Chunked prefill:** supported
frameworks/agentgym_rl/docs/_static/logo.png ADDED
frameworks/agentgym_rl/docs/advance/dpo_extension.rst ADDED
@@ -0,0 +1,271 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Extend to other RL(HF) algorithms
2
+ =================================
3
+
4
+ We already implemented the complete training pipeline of the PPO
5
+ algorithms. To extend to other algorithms, we analyze the high-level
6
+ principle to use verl and provide a tutorial to implement the DPO
7
+ algorithm. Users can follow the similar paradigm to extend to other RL algorithms.
8
+
9
+ .. note:: **Key ideas**: Single process drives multi-process computation and data communication.
10
+
11
+ Overall Approach
12
+ ----------------
13
+
14
+ Step 1: Consider what multi-machine multi-GPU computations are needed
15
+ for each model, such as ``generate_sequence`` , ``compute_log_prob`` and
16
+ ``update_policy`` in the actor_rollout model. Implement distributed
17
+ single-process-multiple-data (SPMD) computation and encapsulate them
18
+ into APIs
19
+
20
+ Step 2: Based on different distributed scenarios, including FSDP and 3D
21
+ parallelism in Megatron-LM, implement single-process control of data
22
+ interaction among multi-process computations.
23
+
24
+ Step 3: Utilize the encapsulated APIs to implement the control flow
25
+
26
+ Example: Online DPO
27
+ -------------------
28
+
29
+ We use verl to implement a simple online DPO algorithm. The algorithm
30
+ flow of Online DPO is as follows:
31
+
32
+ 1. There is a prompt (rollout) generator which has the same weight as
33
+ the actor model. After a batch of prompts are fed into the generator,
34
+ it generates N responses for each prompt.
35
+ 2. Send all the prompts + responses to a verifier for scoring, which can
36
+ be reward model or a rule-based function. Then sort them in pairs to
37
+ form a training batch.
38
+ 3. Use this training batch to train the actor model using DPO. During
39
+ the process, a reference policy is needed.
40
+
41
+ Step 1: What are the multi-machine multi-GPU computations
42
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
43
+
44
+ **Sample Generator**
45
+
46
+ Implementation details:
47
+
48
+ .. code:: python
49
+
50
+ from verl.single_controller.base import Worker
51
+ from verl.single_controller.ray import RayWorkerGroup, RayClassWithInitArgs, RayResourcePool
52
+ import ray
53
+
54
+ @ray.remote
55
+ class SampleGenerator(Worker):
56
+ def __init__(self, config):
57
+ super().__init__()
58
+ self.config = config
59
+
60
+ def generate_sequences(self, data):
61
+ pass
62
+
63
+ Here, ``SampleGenerator`` can be viewed as a multi-process pulled up by
64
+ ``torchrun``, with each process running the same code (SPMD).
65
+ ``SampleGenerator`` needs to implement a ``generate_sequences`` API for
66
+ the control flow to call. The implementation details inside can use any
67
+ inference engine including vllm, sglang and huggingface. Users can
68
+ largely reuse the code in
69
+ verl/verl/trainer/ppo/rollout/vllm_rollout/vllm_rollout.py and we won't
70
+ go into details here.
71
+
72
+ **ReferencePolicy inference**
73
+
74
+ API: compute reference log probability
75
+
76
+ .. code:: python
77
+
78
+ from verl.single_controller.base import Worker
79
+ import ray
80
+
81
+ @ray.remote
82
+ class ReferencePolicy(Worker):
83
+ def __init__(self):
84
+ super().__init__()
85
+ self.model = Model()
86
+
87
+ def infer(self, data):
88
+ return self.model(data)
89
+
90
+ **Actor update**
91
+
92
+ API: Update actor model parameters
93
+
94
+ .. code:: python
95
+
96
+ from verl.single_controller.base import Worker
97
+ import ray
98
+
99
+ @ray.remote
100
+ class DPOActor(Worker):
101
+ def __init__(self):
102
+ super().__init__()
103
+ self.model = Model()
104
+ self.model = FSDP(self.model) # or other distributed strategy
105
+ self.optimizer = optim.Adam(self.model.parameters(), lr=1e-3)
106
+ self.loss_fn = xxx
107
+
108
+ def update(self, data):
109
+ self.optimizer.zero_grad()
110
+ logits = self.model(data)
111
+ loss = self.loss_fn(logits)
112
+ loss.backward()
113
+ self.optimizer.step()
114
+
115
+ **Notes: How to distinguish between control processes and distributed computation processes**
116
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
117
+
118
+ - Control processes are generally functions directly decorated with
119
+ ``@ray.remote``
120
+ - Computation processes are all wrapped into a ``RayWorkerGroup``.
121
+
122
+ Users can reuse most of the distribtued computation logics implemented
123
+ in PPO algorithm, including FSDP and Megatron-LM backend in
124
+ verl/verl/trainer/ppo.
125
+
126
+ Step 2: Based on different distributed scenarios, implement single-process control of multi-process data interaction
127
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
128
+
129
+ **The core problem to solve here is how a single process sends data to
130
+ multiple processes, drives multi-process computation, and how the
131
+ control process obtains the results of multi-process computation.**
132
+ First, we initialize the multi-process ``WorkerGroup`` in the control
133
+ process.
134
+
135
+ .. code:: python
136
+
137
+ @ray.remote(num_cpus=1)
138
+ def main_task(config):
139
+ # construct SampleGenerator
140
+ resource_pool = RayResourcePool(process_on_nodes=[8] * 2) # 16 GPUs
141
+ ray_cls = RayClassWithInitArgs(SampleGenerator, config=config)
142
+ # put SampleGenerator onto resource pool
143
+ worker_group = RayWorkerGroup(resource_pool, ray_cls)
144
+
145
+ # construct reference policy
146
+
147
+ As we can see, in the control process, multiple processes are wrapped
148
+ into a ``RayWorkerGroup``. Inside this ``WorkerGroup``, there is a
149
+ ``self._workers`` member, where each worker is a RayActor
150
+ (https://docs.ray.io/en/latest/ray-core/actors.html) of SampleGenerator.
151
+ ray_trainer.md also provide an implementation of
152
+ ``MegatronRayWorkerGroup``.
153
+
154
+ Assuming the model is distributed using FSDP, and there is a batch of
155
+ data on the control process, for data parallelism, the underlying
156
+ calling process is:
157
+
158
+ .. code:: python
159
+
160
+ data = xxx
161
+ data_list = data.chunk(dp_size)
162
+
163
+ output = []
164
+ for d in data_list:
165
+ # worker_group._workers[i] is a SampleGenerator
166
+ output.append(worker_group._workers[i].generate_sequences.remote(d))
167
+
168
+ output = ray.get(output)
169
+ output = torch.cat(output)
170
+
171
+ Single process calling multiple processes involves the following 3
172
+ steps:
173
+
174
+ 1. Split the data into DP parts on the control process.
175
+ 2. Send the data to remote, call the remote computation through RPC, and
176
+ utilize multi-process computation.
177
+ 3. Obtain the computation results of each worker on the control process
178
+ and merge them.
179
+
180
+ Frequently calling these 3 steps on the controller process greatly hurts
181
+ code readability. **In verl, we have abstracted and encapsulated these 3
182
+ steps, so that the worker's method + dispatch + collect can be
183
+ registered into the worker_group**
184
+
185
+ .. code:: python
186
+
187
+ from verl.single_controller.base.decorator import register
188
+
189
+ def dispatch_data(worker_group, data):
190
+ return data.chunk(worker_group.world_size)
191
+
192
+ def collect_data(worker_group, data):
193
+ return torch.cat(data)
194
+
195
+ dispatch_mode = {
196
+ 'dispatch_fn': dispatch_data,
197
+ 'collect_fn': collect_data
198
+ }
199
+
200
+ @register(dispatch_mode=dispatch_mode)
201
+ def generate_sequences(self, data):
202
+ pass
203
+
204
+ In this way, we can directly call the method inside the worker through
205
+ the ``worker_group`` on the control (driver) process (which is a single
206
+ process):
207
+
208
+ .. code:: python
209
+
210
+ output = worker_group.generate_sequences(data)
211
+
212
+ This single line includes data splitting, data distribution and
213
+ computation, and data collection.
214
+
215
+ Furthermore, the model parallelism size of each model is usually fixed,
216
+ including dp, tp, pp. So for these common distributed scenarios, we have
217
+ pre-implemented specific dispatch and collect methods,in `decorator.py <https://github.com/volcengine/verl/blob/main/verl/single_controller/base/decorator.py>`_, which can be directly used to wrap the computations.
218
+
219
+ .. code:: python
220
+
221
+ from verl.single_controller.base.decorator import register, Dispatch
222
+
223
+ @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
224
+ def generate_sequences(self, data: DataProto) -> DataProto:
225
+ pass
226
+
227
+ Here it requires the data interface to be ``DataProto``. Definition of
228
+ ``DataProto`` is in `protocol.py <https://github.com/volcengine/verl/blob/main/verl/protocol.py>`_.
229
+
230
+ Step 3: Main training loop
231
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
232
+
233
+ With the above training flows, we can implement the algorithm's control
234
+ flow. It is recommended that ``main_task`` is also a ray remote process.
235
+
236
+ .. code:: python
237
+
238
+ @ray.remote(num_cpus=1)
239
+ def main_task(config):
240
+ # construct SampleGenerator
241
+ resource_pool = RayResourcePool(process_on_nodes=[8] * 2) # 16 GPUs
242
+ ray_cls = RayClassWithInitArgs(SampleGenerator, config=config)
243
+ # put SampleGenerator onto resource pool
244
+ sample_gen = RayWorkerGroup(resource_pool, ray_cls)
245
+
246
+ # construct reference policy
247
+ ray_cls = RayClassWithInitArgs(ReferencePolicy)
248
+ ref_policy = RayWorkerGroup(resource_pool, ray_cls)
249
+
250
+ # construct actor
251
+ ray_cls = RayClassWithInitArgs(DPOActor)
252
+ dpo_policy = RayWorkerGroup(resource_pool, ray_cls)
253
+
254
+ dataloader = DataLoader()
255
+
256
+ for data in dataloader:
257
+ # generate data
258
+ data = sample_gen.generate_sequences(data)
259
+ # generate scores for each data
260
+ data = generate_scores(data)
261
+ # generate pairwise data using scores
262
+ data = generate_pairwise_data(data)
263
+ # generate ref_log_prob
264
+ data.batch['ref_log_prob'] = ref_policy.infer(data)
265
+ # update using dpo
266
+ dpo_policy.update(data)
267
+ # logging
268
+
269
+ Here, different ``WorkerGroups`` can be placed in the same resource pool or
270
+ in different resource pools using ``create_colocated_worker_cls``
271
+ similar as in `ray_trainer.py <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py>`_.
frameworks/agentgym_rl/docs/advance/fsdp_extension.rst ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Add models with the FSDP backend
3
+ ==================================
4
+
5
+ Model
6
+ --------------------------
7
+
8
+ In principle, our FSDP backend can support any HF model and we can
9
+ sychronoize the actor model weight with vLLM using `hf_weight_loader.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_6_3/hf_weight_loader.py>`_.
10
+ However, ``hf_weight_loader`` is will gather the full state_dict of a
11
+ model during synchronization, which may cause OOM. We suggest using
12
+ ``dtensor_weight_loader`` which gather the full model parameter layer by
13
+ layer to reduce the peak memory usage. We already support dtensor weight
14
+ loader for the models below in `dtensor_weight_loader.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_5_4/dtensor_weight_loaders.py>`_.:
15
+
16
+ - ``GPT2LMHeadModel``
17
+ - ``LlamaForCausalLM``
18
+ - ``LLaMAForCausalLM``
19
+ - ``MistralForCausalLM``
20
+ - ``InternLMForCausalLM``
21
+ - ``AquilaModel``
22
+ - ``AquilaForCausalLM``
23
+ - ``Phi3ForCausalLM``
24
+ - ``GemmaForCausalLM``
25
+ - ``Gemma2ForCausalLM``
26
+ - ``GPTBigCodeForCausalLM``
27
+ - ``Starcoder2ForCausalLM``
28
+ - ``Qwen2ForCausalLM``
29
+ - ``DeepseekV2ForCausalLM``
30
+
31
+ To implement ``dtensor_weight_loader`` of a model that's supported in
32
+ vLLM, follow the guide of gemma model below:
33
+
34
+ 1. Copy the
35
+ ``load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]])`` from the vllm model class
36
+ to ``dtensor_weight_loaders.py``
37
+ 2. Modify the arguments to
38
+ ``(actor_weights: Dict, vllm_model: nn.Module)``
39
+ 3. Replace the ``self`` to ``vllm_model``
40
+ 4. Add the
41
+ ``local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)``
42
+ before each ``param = params_dict[name]`` and modify the following
43
+ weight loading using ``local_loaded_weight``.
44
+ 5. Register the implemented dtensor weight loader to ``__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__``.
45
+
46
+ .. code-block:: diff
47
+
48
+ - def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
49
+ + def gemma_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module:
50
+ stacked_params_mapping = [
51
+ # (param_name, shard_name, shard_id)
52
+ ("qkv_proj", "q_proj", "q"),
53
+ ("qkv_proj", "k_proj", "k"),
54
+ ("qkv_proj", "v_proj", "v"),
55
+ ("gate_up_proj", "gate_proj", 0),
56
+ ("gate_up_proj", "up_proj", 1),
57
+ ]
58
+ - params_dict = dict(self.named_parameters())
59
+ + params_dict = dict(vllm_model.named_parameters())
60
+ loaded_params = set()
61
+ - for name, loaded_weight in weights:
62
+ + for name, loaded_weight in actor_weights.items():
63
+ for (param_name, shard_name, shard_id) in stacked_params_mapping:
64
+ if shard_name not in name:
65
+ continue
66
+ name = name.replace(shard_name, param_name)
67
+ # Skip loading extra bias for GPTQ models.
68
+ if name.endswith(".bias") and name not in params_dict:
69
+ continue
70
+ + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)
71
+ param = params_dict[name]
72
+ weight_loader = param.weight_loader
73
+ - weight_loader(param, loaded_weight, shard_id)
74
+ + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id)
75
+ break
76
+ else:
77
+ # lm_head is not used in vllm as it is tied with embed_token.
78
+ # To prevent errors, skip loading lm_head.weight.
79
+ if "lm_head.weight" in name:
80
+ continue
81
+ # Skip loading extra bias for GPTQ models.
82
+ if name.endswith(".bias") and name not in params_dict:
83
+ continue
84
+ + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)
85
+ param = params_dict[name]
86
+ weight_loader = getattr(param, "weight_loader",
87
+ default_weight_loader)
88
+ - weight_loader(param, loaded_weight)
89
+ + weight_loader(param, local_loaded_weight.to(dtype=param.dtype))
90
+ loaded_params.add(name)
91
+ unloaded_params = params_dict.keys() - loaded_params
92
+ if unloaded_params:
93
+ raise RuntimeError(
94
+ "Some weights are not initialized from checkpoints: "
95
+ f"{unloaded_params}")
frameworks/agentgym_rl/docs/advance/megatron_extension.rst ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Add models with the Megatron-LM backend
2
+ =========================================
3
+
4
+ Model
5
+ -----------
6
+
7
+ The most challenging aspect to use the Megatron-LM backend is implementing
8
+ the models for training. Currently, we implement Llama model that
9
+ support data parallelism, tensor parallelism, pipeline parallelism (also
10
+ vPP) and sequence parallelism. We also implement remove padding (sequence packing) on Llama
11
+ model, which can be found in `modeling_llama_megatron.py <https://github.com/volcengine/verl/blob/main/verl/models/llama/megatron/modeling_llama_megatron.py>`_.
12
+
13
+ To support other model, users are required to implement:
14
+
15
+ 1. Implemnt a model similar to ``modeling_llama_megatron.py`` that satisfy the
16
+ parallelism requirements of Megatron-LM. Then register your model in
17
+ the `registry.py <https://github.com/volcengine/verl/blob/main/verl/models/registry.py>`_.
18
+ 2. Checkpoint utils that can load full checkpoint (e.g. huggingface
19
+ checkpoint) to partitioned models during the runtime. Then register
20
+ your loader to ``weight_loader_registry`` in `weight_loader_registry.py <https://github.com/volcengine/verl/blob/main/verl/models/weight_loader_registry.py>`_.
21
+ 3. Weight loader that synchronize the weight from Megatron to rollout
22
+ (vLLM) model. Note that both the actor model and rollout model are
23
+ partitioned during runtime. So, it's advisable to map the model name
24
+ in actor model implementation. Otherwise, you may need an additional
25
+ name mapping and even weight transformation. The weight loader implementation
26
+ is in `megatron_weight_loaders.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_6_3/megatron_weight_loaders.py>`_.
frameworks/agentgym_rl/docs/advance/placement.rst ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Ray API Design Tutorial
2
+ =======================================
3
+
4
+ We provide a tutorial for our Ray API design, including:
5
+
6
+ - Ray basic concepts
7
+ - Resource Pool and RayWorkerGroup
8
+ - Data Dispatch, Execution and Collection
9
+ - Initialize the RayWorkerGroup and execute the distributed computation in the given Resource Pool
10
+
11
+ See details in `tutorial.ipynb <https://github.com/volcengine/verl/blob/main/examples/ray/tutorial.ipynb>`_.
frameworks/agentgym_rl/docs/conf.py ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 Bytedance Ltd. and/or its affiliates
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ # Configuration file for the Sphinx documentation builder.
16
+ #
17
+ # This file only contains a selection of the most common options. For a full
18
+ # list see the documentation:
19
+ # https://www.sphinx-doc.org/en/master/usage/configuration.html
20
+
21
+ # -- Path setup --------------------------------------------------------------
22
+
23
+ # If extensions (or modules to document with autodoc) are in another directory,
24
+ # add these directories to sys.path here. If the directory is relative to the
25
+ # documentation root, use os.path.abspath to make it absolute, like shown here.
26
+ #
27
+ # import os
28
+ # import sys
29
+ # sys.path.insert(0, os.path.abspath('.'))
30
+
31
+
32
+ # -- Project information -----------------------------------------------------
33
+
34
+ project = u'verl'
35
+ # pylint: disable=W0622
36
+ copyright = u'2024 ByteDance Seed Foundation MLSys Team'
37
+ author = u'Guangming Sheng, Chi Zhang, Yanghua Peng, Haibin Lin'
38
+
39
+
40
+ # -- General configuration ---------------------------------------------------
41
+ # The master toctree document.
42
+ master_doc = 'index'
43
+
44
+ # Add any Sphinx extension module names here, as strings. They can be
45
+ # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
46
+ # ones.
47
+ extensions = ['recommonmark',
48
+ 'sphinx.ext.autosectionlabel',
49
+ ]
50
+
51
+ # The suffix(es) of source filenames.
52
+ # You can specify multiple suffix as a list of string:
53
+ source_suffix = ['.rst', 'rest', '.md']
54
+
55
+ # Add any paths that contain templates here, relative to this directory.
56
+ templates_path = ['_templates']
57
+
58
+ # The language for content autogenerated by Sphinx. Refer to documentation
59
+ # for a list of supported languages.
60
+ #
61
+ # This is also used if you do content translation via gettext catalogs.
62
+ # Usually you set "language" from the command line for these cases.
63
+ language = u'en'
64
+
65
+ # List of patterns, relative to source directory, that match files and
66
+ # directories to ignore when looking for source files.
67
+ # This pattern also affects html_static_path and html_extra_path.
68
+ exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
69
+
70
+
71
+ # -- Options for HTML output -------------------------------------------------
72
+
73
+ # The theme to use for HTML and HTML Help pages. See the documentation for
74
+ # a list of builtin themes.
75
+ #
76
+ html_theme = 'sphinx_rtd_theme'
77
+
78
+ # Add any paths that contain custom static files (such as style sheets) here,
79
+ # relative to this directory. They are copied after the builtin static files,
80
+ # so a file named "default.css" will overwrite the builtin "default.css".
81
+ html_static_path = ['_static']
frameworks/agentgym_rl/docs/examples/config.rst ADDED
@@ -0,0 +1,365 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ .. _config-explain-page:
2
+
3
+ Config Explanation
4
+ ===================
5
+
6
+ ppo_trainer.yaml for FSDP Backend
7
+ ---------------------------------
8
+
9
+ Data
10
+ ~~~~
11
+
12
+ .. code:: yaml
13
+
14
+ data:
15
+ tokenizer: null
16
+ train_files: ~/data/rlhf/gsm8k/train.parquet
17
+ val_files: ~/data/rlhf/gsm8k/test.parquet
18
+ prompt_key: prompt
19
+ max_prompt_length: 512
20
+ max_response_length: 512
21
+ train_batch_size: 1024
22
+ val_batch_size: 1312
23
+ return_raw_input_ids: False # This should be set to true when the tokenizer between policy and rm differs
24
+ return_raw_chat: False
25
+
26
+ - ``data.train_files``: Training set parquet. Can be a list or a single
27
+ file. The program will read all files into memory, so it can't be too
28
+ large (< 100GB). The path can be either local path or HDFS path. For
29
+ HDFS path, we provide utils to download it to DRAM and convert the
30
+ HDFS path to local path.
31
+ - ``data.val_files``: Validation parquet. Can be a list or a single
32
+ file.
33
+ - ``data.prompt_key``: The field in the dataset where the prompt is
34
+ located. Default is 'prompt'.
35
+ - ``data.max_prompt_length``: Maximum prompt length. All prompts will be
36
+ left-padded to this length. An error will be reported if the length is
37
+ too long
38
+ - ``data.max_response_length``: Maximum response length. Rollout in RL
39
+ algorithms (e.g. PPO) generates up to this length
40
+ - ``data.train_batch_size``: Batch size sampled for one training
41
+ iteration of different RL algorithms.
42
+ - ``data.val_batch_size``: Batch size sampled for one validation
43
+ iteration.
44
+ - ``data.return_raw_input_ids``: Whether to return the original
45
+ input_ids without adding chat template. This is mainly used to
46
+ accommodate situations where the reward model's chat template differs
47
+ from the policy. It needs to be decoded first, then apply the RM's
48
+ chat template. If using a model-based RM, and the policy and RM
49
+ chat_templates are different, this flag needs to be set
50
+ - ``data.return_raw_chat``:
51
+ - ``data.truncation``: Truncate the input_ids or prompt length if they
52
+ exceed max_prompt_length. Default is 'error', not allow exceed the
53
+ max_prompt_length. The users should increase the max_prompt_length if
54
+ throwing the error.
55
+
56
+ Actor/Rollout/Reference Policy
57
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
58
+
59
+ .. code:: yaml
60
+
61
+ actor_rollout_ref:
62
+ hybrid_engine: True
63
+ model:
64
+ path: ~/models/deepseek-llm-7b-chat
65
+ external_lib: null
66
+ override_config: { }
67
+ enable_gradient_checkpointing: False
68
+ use_remove_padding: False
69
+ actor:
70
+ strategy: fsdp # This is for backward-compatibility
71
+ ppo_mini_batch_size: 256
72
+ ppo_micro_batch_size: null # will be deprecated, use ppo_micro_batch_size_per_gpu
73
+ ppo_micro_batch_size_per_gpu: 8
74
+ use_dynamic_bsz: False
75
+ ppo_max_token_len_per_gpu: 16384 # n * ${data.max_prompt_length} + ${data.max_response_length}
76
+ grad_clip: 1.0
77
+ clip_ratio: 0.2
78
+ entropy_coeff: 0.001
79
+ use_kl_loss: False # True for GRPO
80
+ kl_loss_coef: 0.001 # for grpo
81
+ kl_loss_type: low_var_kl # for grpo
82
+ ppo_epochs: 1
83
+ shuffle: False
84
+ ulysses_sequence_parallel_size: 1 # sp size
85
+ optim:
86
+ lr: 1e-6
87
+ lr_warmup_steps_ratio: 0. # the total steps will be injected during runtime
88
+ min_lr_ratio: null # only useful for warmup with cosine
89
+ warmup_style: constant # select from constant/cosine
90
+ total_training_steps: -1 # must be override by program
91
+ fsdp_config:
92
+ wrap_policy:
93
+ # transformer_layer_cls_to_wrap: None
94
+ min_num_params: 0
95
+ param_offload: False
96
+ grad_offload: False
97
+ optimizer_offload: False
98
+ fsdp_size: -1
99
+ ref:
100
+ fsdp_config:
101
+ param_offload: False
102
+ wrap_policy:
103
+ # transformer_layer_cls_to_wrap: None
104
+ min_num_params: 0
105
+ log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu
106
+ log_prob_micro_batch_size_per_gpu: 16
107
+ log_prob_use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz}
108
+ log_prob_max_token_len_per_gpu: ${actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
109
+ ulysses_sequence_parallel_size: ${actor_rollout_ref.actor.ulysses_sequence_parallel_size} # sp size
110
+ rollout:
111
+ name: vllm
112
+ temperature: 1.0
113
+ top_k: -1 # 0 for hf rollout, -1 for vllm rollout
114
+ top_p: 1
115
+ prompt_length: ${data.max_prompt_length} # not use for opensource
116
+ response_length: ${data.max_response_length}
117
+ # for vllm rollout
118
+ dtype: bfloat16 # should align with FSDP
119
+ gpu_memory_utilization: 0.5
120
+ ignore_eos: False
121
+ enforce_eager: True
122
+ free_cache_engine: True
123
+ load_format: dummy_dtensor
124
+ tensor_model_parallel_size: 2
125
+ max_num_batched_tokens: 8192
126
+ max_num_seqs: 1024
127
+ log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu
128
+ log_prob_micro_batch_size_per_gpu: 16
129
+ log_prob_use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz}
130
+ log_prob_max_token_len_per_gpu: ${actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
131
+ # for hf rollout
132
+ do_sample: True
133
+ # number of responses (i.e. num sample times)
134
+ n: 1 # > 1 for grpo
135
+
136
+ **Common config for actor, rollout and reference model**
137
+
138
+ - ``actor_rollout_ref.hybrid_engine``: Whether it's a hybrid engine,
139
+ currently only supports hybrid engine
140
+ - ``actor_rollout_ref.model.path``: Huggingface model path. This can be
141
+ either local path or HDFS path. For HDFS path, we provide utils to
142
+ download it to DRAM and convert the HDFS path to local path.
143
+ - ``actor_rollout_ref.model.external_libs``: Additional Python packages
144
+ that need to be imported. Used to register models or tokenizers into
145
+ the Huggingface system.
146
+ - ``actor_rollout_ref.model.override_config``: Used to override some of
147
+ the model's original configurations, mainly dropout
148
+ - ``actor_rollout_ref.model.enable_gradient_checkpointing``: Whether to
149
+ enable gradient checkpointing for the actor
150
+
151
+ **Actor model**
152
+
153
+ - ``actor_rollout_ref.actor.strategy``: fsdp or megatron. In this
154
+ example, we use fsdp backend.
155
+
156
+ - ``actor_rollout_ref.actor.ppo_mini_batch_size``: One sample is split
157
+ into multiple sub-batches with batch_size=ppo_mini_batch_size for PPO
158
+ updates. The ppo_mini_batch_size is a global num across all workers/gpus
159
+
160
+ - ``actor_rollout_ref.actor.ppo_micro_batch_size``: [Will be deprecated, use ppo_micro_batch_size_per_gpu]
161
+ Similar to gradient accumulation, the micro_batch_size_per_gpu for one forward pass,
162
+ trading speed for GPU memory. The value represent the global view.
163
+
164
+ - ``actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu``: Similar to gradient
165
+ accumulation, the micro_batch_size_per_gpu for one forward pass, trading speed
166
+ for GPU memory. The value represent the local num per gpu.
167
+
168
+ - ``actor_rollout_ref.actor.grad_clip``: Gradient clipping for actor
169
+ updates
170
+
171
+ - ``actor_rollout_ref.actor.clip_ratio``: PPO clip ratio
172
+
173
+ - ``actor_rollout_ref.actor.entropy_coeff``: The weight of entropy when
174
+ calculating PPO loss
175
+
176
+ - ``actor_rollout_ref.actor.ppo_epochs``: Number of epochs for PPO
177
+ updates on one set of sampled data
178
+
179
+ - ``actor_rollout_ref.actor.shuffle``: Whether to shuffle data when
180
+ there are multiple epochs
181
+
182
+ - ``actor_rollout_ref.actor.optim``: Actor's optimizer parameters
183
+
184
+ - ``actor_rollout_ref.actor.fsdp_config``: FSDP config for actor
185
+ training
186
+
187
+ - ``wrap_policy``: FSDP wrap policy. By default, it uses Huggingface's
188
+ wrap policy, i.e., wrapping by DecoderLayer
189
+
190
+ - No need to set transformer_layer_cls_to_wrap, so we comment it.
191
+
192
+ - ``*_offload``: Whether to enable parameter, gradient and optimizer
193
+ offload
194
+
195
+ - Trading speed for GPU memory.
196
+
197
+ **Reference Model**
198
+
199
+ - ``actor_rollout_ref.ref``: FSDP config same as actor. **For models
200
+ larger than 7B, it's recommended to turn on offload for ref by
201
+ default**
202
+
203
+ - ``actor_rollout_ref.ref.log_prob_micro_batch_size``: [Will be deprecate, use log_prob_micro_batch_size_per_gpu]
204
+ The batch size for one forward pass in the computation of ``ref_log_prob``. The value represent the global num.
205
+
206
+ - ``actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu``: The batch size
207
+ for one forward pass in the computation of ``ref_log_prob``. The value represent the local num per gpu.
208
+
209
+ **Rollout Model**
210
+
211
+ - ``actor_rollout_ref.rollout.name``: hf/vllm. We use vLLM by default
212
+ because it's much efficient and our hybrid engine is implemented with
213
+ vLLM.
214
+
215
+ - Rollout (Auto-regressive) parameters. The key should be equal to the
216
+ property name in vLLM's ``SamplingParams``.
217
+
218
+ - ``temperature``, ``top_k``, ``top_p`` and others: Sampling
219
+ parameters in ``SamplingParams``.
220
+
221
+ - ``dtype``: Rollout model parameters type. This should be align with
222
+ the actor model parameter type in FSDP/Megatron backend.
223
+
224
+ - ``gpu_memory_utilization``: The proportion of the remaining GPU memory
225
+ allocated for kv cache after other models have initialized when using
226
+ vllm.
227
+
228
+ - ``tensor_model_parallel_size``: TP size for rollout. Only effective
229
+ for vllm.
230
+
231
+ - ``actor_rollout_ref.ref.log_prob_micro_batch_size``: [Will be deprecate, use log_prob_micro_batch_size_per_gpu]
232
+ The batch size for one forward pass in the computation of ``log_prob``. The value represent the global num.
233
+
234
+ - ``log_prob_micro_batch_size_per_gpu``: Micro batch size per gpu (The batch size for
235
+ one forward pass) for recalculating ``log_prob``. The value represent the local num per gpu.
236
+
237
+ - ``do_sample``: Whether to sample. If set to False, the rollout model
238
+ will perform greedy sampling. We disable ``do_sample`` during
239
+ validation.
240
+
241
+ - ``actor_rollout_ref.rollout.ignore_eos``: Whether to ignore the EOS
242
+ token and continue generating tokens after the EOS token is generated.
243
+
244
+ - ``actor_rollout_ref.rollout.free_cache_engine``: Offload the KVCache
245
+ after rollout generation stage. Default is True. When set to True, we
246
+ need to disable the usage of CUDAGraph (set ``enforce_eager`` to
247
+ True.)
248
+
249
+ - ``actor_rollout_ref.rollout.enforce_eager``: Whether to use CUDAGraph
250
+ in vLLM generation. Default set to True to disable CUDAGraph.
251
+
252
+ - ``actor_rollout_ref.rollout.load_format``: Which weight loader to use
253
+ to load the actor model weights to the rollout model.
254
+
255
+ - ``auto``: Use Megatron weight loader.
256
+ - ``megatron``: Use Megatron weight loader. Deployed with Megatron
257
+ backend. The input model ``state_dict()`` is already partitioned
258
+ along TP dimension and already gathered along PP dimension. This
259
+ weight loader requires that the Rollout model and Actor model's
260
+ parameters shape and name should be identical.
261
+ - ``dtensor``: Default solution when using Huggingface weight loader.
262
+ Deployed with FSDP backend and the state_dict_type is
263
+ ``StateDictType.SHARDED_STATE_DICT``. Recommend to use this weight
264
+ loader
265
+ - ``hf``: Use Huggingface weight loader. Deployed with FSDP backend
266
+ and the state_dict_type is ``StateDictType.FULL_STATE_DICT``. This
267
+ solution doesn't need to rewrite the weight loader for each model
268
+ implemented in vLLM but it results in larger peak memory usage.
269
+ - ``dummy_hf``, ``dummy_megatron``, ``dummy_dtensor``: Random
270
+ initialization.
271
+
272
+ .. note:: **NOTED**: In this config field, users only need to select from ``dummy_megatron``, ``dummy_dtensor``, ``dummy_hf`` for rollout initialization and our hybrid engine will select the corresponding weight loader (i.e., ``megatron``, ``dtensor``, ``hf``) during actor/rollout weight synchronization.
273
+
274
+ Critic Model
275
+ ~~~~~~~~~~~~
276
+
277
+ Most parameters for Critic are similar to Actor Model.
278
+
279
+ Reward Model
280
+ ~~~~~~~~~~~~
281
+
282
+ .. code:: yaml
283
+
284
+ reward_model:
285
+ enable: False
286
+ model:
287
+ input_tokenizer: ${actor_rollout_ref.model.path} # set this to null if the chat template is identical
288
+ path: ~/models/Anomy-RM-v0.1
289
+ external_lib: ${actor_rollout_ref.model.external_lib}
290
+ fsdp_config:
291
+ min_num_params: 0
292
+ param_offload: False
293
+ micro_batch_size_per_gpu: 16
294
+ max_length: null
295
+ reward_manager: naive
296
+
297
+ - ``reward_model.enable``: Whether to enable reward model. If False, we
298
+ compute the reward only with the user-defined reward functions. In
299
+ GSM8K and Math examples, we disable reward model. For RLHF alignment
300
+ example using full_hh_rlhf, we utilize reward model to assess the
301
+ responses. If False, the following parameters are not effective.
302
+ - ``reward_model.model``
303
+
304
+ - ``input_tokenizer``: Input tokenizer. If the reward model's chat
305
+ template is inconsistent with the policy, we need to first decode to
306
+ plaintext, then apply the rm's chat_template. Then score with RM. If
307
+ chat_templates are consistent, it can be set to null.
308
+ - ``path``: RM's HDFS path or local path. Note that RM only supports
309
+ AutoModelForSequenceClassification. Other model types need to define
310
+ their own RewardModelWorker and pass it from the code.
311
+ - ``reward_model.reward_manager``: Reward Manager. This defines the mechanism
312
+ of computing rule-based reward and handling different reward sources. Default
313
+ if ``naive``. If all verification functions are multiprocessing-safe, the reward
314
+ manager can be set to ``prime`` for parallel verification.
315
+
316
+ Algorithm
317
+ ~~~~~~~~~
318
+
319
+ .. code:: yaml
320
+
321
+ algorithm:
322
+ gamma: 1.0
323
+ lam: 1.0
324
+ adv_estimator: gae
325
+ kl_penalty: kl # how to estimate kl divergence
326
+ kl_ctrl:
327
+ type: fixed
328
+ kl_coef: 0.005
329
+
330
+ - ``gemma``: discount factor
331
+ - ``lam``: Trade-off between bias and variance in the GAE estimator
332
+ - ``adv_estimator``: Support ``gae``, ``grpo``, ``reinforce_plus_plus``.
333
+ - ``kl_penalty``: Support ``kl``, ``abs``, ``mse`` and ``full``. How to
334
+ calculate the kl divergence between actor and reference policy. For
335
+ specific options, refer to `core_algos.py <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py#L192>`_ .
336
+
337
+ Trainer
338
+ ~~~~~~~
339
+
340
+ .. code:: yaml
341
+
342
+ trainer:
343
+ total_epochs: 30
344
+ project_name: verl_examples
345
+ experiment_name: gsm8k
346
+ logger: ['console', 'wandb']
347
+ nnodes: 1
348
+ n_gpus_per_node: 8
349
+ save_freq: -1
350
+ test_freq: 2
351
+ critic_warmup: 0
352
+ default_hdfs_dir: ~/experiments/gsm8k/ppo/${trainer.experiment_name} # hdfs checkpoint path
353
+ default_local_dir: checkpoints/${trainer.project_name}/${trainer.experiment_name} # local checkpoint path
354
+
355
+ - ``trainer.total_epochs``: Number of epochs in training.
356
+ - ``trainer.project_name``: For wandb
357
+ - ``trainer.experiment_name``: For wandb
358
+ - ``trainer.logger``: Support console and wandb
359
+ - ``trainer.nnodes``: Number of nodes used in the training.
360
+ - ``trainer.n_gpus_per_node``: Number of GPUs per node.
361
+ - ``trainer.save_freq``: The frequency (by iteration) to save checkpoint
362
+ of the actor and critic model.
363
+ - ``trainer.test_freq``: The validation frequency (by iteration).
364
+ - ``trainer.critic_warmup``: The number of iteration to train the critic
365
+ model before actual policy learning.
frameworks/agentgym_rl/docs/examples/gsm8k_example.rst ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ GSM8K Example
2
+ =============
3
+
4
+ Introduction
5
+ ------------
6
+
7
+ In this example, we train an LLM to tackle the GSM8k task.
8
+
9
+ Paper: https://arxiv.org/pdf/2110.14168
10
+
11
+ Dataset: https://huggingface.co/datasets/gsm8k
12
+
13
+ Note that the original paper mainly focuses on training a verifier (a
14
+ reward model) to solve math problems via Best-of-N sampling. In this
15
+ example, we train an RLHF agent using a rule-based reward model.
16
+
17
+ Dataset Introduction
18
+ --------------------
19
+
20
+ GSM8k is a math problem dataset. The prompt is an elementary school
21
+ problem. The LLM model is required to answer the math problem.
22
+
23
+ The training set contains 7473 samples and the test set contains 1319
24
+ samples.
25
+
26
+ **An example**
27
+
28
+ Prompt
29
+
30
+ Katy makes coffee using teaspoons of sugar and cups of water in the
31
+ ratio of 7:13. If she used a total of 120 teaspoons of sugar and cups
32
+ of water, calculate the number of teaspoonfuls of sugar she used.
33
+
34
+ Solution
35
+
36
+ The total ratio representing the ingredients she used to make the
37
+ coffee is 7+13 = <<7+13=20>>20 Since the fraction representing the
38
+ number of teaspoons she used is 7/20, she used 7/20\ *120 =
39
+ <<7/20*\ 120=42>>42 #### 42
40
+
41
+ Step 1: Prepare dataset
42
+ -----------------------
43
+
44
+ .. code:: bash
45
+
46
+ cd examples/data_preprocess
47
+ python3 gsm8k.py --local_dir ~/data/gsm8k
48
+
49
+ Step 2: Download Model
50
+ ----------------------
51
+
52
+ There're three ways to prepare the model checkpoints for post-training:
53
+
54
+ - Download the required models from hugging face
55
+
56
+ .. code:: bash
57
+
58
+ huggingface-cli download deepseek-ai/deepseek-math-7b-instruct --local-dir ~/models/deepseek-math-7b-instruct --local-dir-use-symlinks False
59
+
60
+ - Already store your store model in the local directory or HDFS path.
61
+ - Also, you can directly use the model name in huggingface (e.g.,
62
+ deepseek-ai/deepseek-math-7b-instruct) in
63
+ ``actor_rollout_ref.model.path`` and ``critic.model.path`` field in
64
+ the run script.
65
+
66
+ Noted that users should prepare checkpoints for actor, critic and reward
67
+ model.
68
+
69
+ [Optional] Step 3: SFT your Model
70
+ ---------------------------------
71
+
72
+ We provide a SFT Trainer using PyTorch FSDP in
73
+ `fsdp_sft_trainer.py <https://github.com/volcengine/verl/blob/main/verl/trainer/fsdp_sft_trainer.py>`_.
74
+ Users can customize their own SFT
75
+ script using our FSDP SFT Trainer.
76
+
77
+ We also provide various training scripts for SFT on GSM8K dataset in `gsm8k sft directory <https://github.com/volcengine/verl/blob/main/examples/sft/gsm8k/>`_.
78
+
79
+ .. code:: shell
80
+
81
+ set -x
82
+
83
+ torchrun -m verl.trainer.fsdp_sft_trainer \
84
+ data.train_files=$HOME/data/gsm8k/train.parquet \
85
+ data.val_files=$HOME/data/gsm8k/test.parquet \
86
+ data.prompt_key=question \
87
+ data.response_key=answer \
88
+ data.micro_batch_size_per_gpu=8 \
89
+ model.partial_pretrain=deepseek-ai/deepseek-coder-6.7b-instruct \
90
+ trainer.default_hdfs_dir=hdfs://user/verl/experiments/gsm8k/deepseek-coder-6.7b-instruct/ \
91
+ trainer.project_name=gsm8k-sft \
92
+ trainer.experiment_name=gsm8k-sft-deepseek-coder-6.7b-instruct \
93
+ trainer.total_epochs=4 \
94
+ trainer.logger=['console','wandb']
95
+
96
+ Step 4: Perform PPO training with your model on GSM8K Dataset
97
+ -------------------------------------------------------------
98
+
99
+ - Prepare your own run.sh script. Here's an example for GSM8k dataset
100
+ and deepseek-llm-7b-chat model.
101
+ - Users could replace the ``data.train_files`` ,\ ``data.val_files``,
102
+ ``actor_rollout_ref.model.path`` and ``critic.model.path`` based on
103
+ their environment.
104
+ - See :doc:`config` for detailed explanation of each config field.
105
+
106
+ **Reward Model/Function**
107
+
108
+ We use a rule-based reward model. We force the model to produce a final
109
+ answer following 4 “#” as shown in the solution. We extract the final
110
+ answer from both the solution and model's output using regular
111
+ expression matching. We compare them and assign a reward of 1 to correct
112
+ answer, 0.1 to incorrect answer and 0 to no answer.
113
+
114
+ **Training Script**
115
+
116
+ The training script example for FSDP and Megatron-LM backend are stored in examples/ppo_trainer directory.
117
+
118
+ .. code:: bash
119
+
120
+ cd ../ppo_trainer
121
+ bash run_deepseek7b_llm.sh
122
+
123
+ The script of run_deepseek7b_llm.sh
124
+
125
+ .. code:: bash
126
+
127
+ set -x
128
+
129
+ python3 -m verl.trainer.main_ppo \
130
+ data.train_files=$HOME/data/gsm8k/train.parquet \
131
+ data.val_files=$HOME/data/gsm8k/test.parquet \
132
+ data.train_batch_size=1024 \
133
+ data.val_batch_size=1312 \
134
+ data.max_prompt_length=512 \
135
+ data.max_response_length=512 \
136
+ actor_rollout_ref.model.path=deepseek-ai/deepseek-llm-7b-chat \
137
+ actor_rollout_ref.actor.optim.lr=1e-6 \
138
+ actor_rollout_ref.model.use_remove_padding=True \
139
+ actor_rollout_ref.actor.ppo_mini_batch_size=256 \
140
+ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=16 \
141
+ actor_rollout_ref.actor.fsdp_config.param_offload=False \
142
+ actor_rollout_ref.actor.fsdp_config.grad_offload=False \
143
+ actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
144
+ actor_rollout_ref.model.enable_gradient_checkpointing=True \
145
+ actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=32 \
146
+ actor_rollout_ref.rollout.tensor_model_parallel_size=4 \
147
+ actor_rollout_ref.rollout.name=vllm \
148
+ actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \
149
+ actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=32 \
150
+ actor_rollout_ref.ref.fsdp_config.param_offload=True \
151
+ critic.optim.lr=1e-5 \
152
+ critic.model.use_remove_padding=True \
153
+ critic.model.path=deepseek-ai/deepseek-llm-7b-chat \
154
+ critic.model.enable_gradient_checkpointing=True \
155
+ critic.ppo_micro_batch_size_per_gpu=32 \
156
+ critic.model.fsdp_config.param_offload=False \
157
+ critic.model.fsdp_config.grad_offload=False \
158
+ critic.model.fsdp_config.optimizer_offload=False \
159
+ algorithm.kl_ctrl.kl_coef=0.001 \
160
+ trainer.critic_warmup=0 \
161
+ trainer.logger=['console','wandb'] \
162
+ trainer.project_name='verl_example_gsm8k' \
163
+ trainer.experiment_name='deepseek_llm_7b_function_rm' \
164
+ trainer.n_gpus_per_node=8 \
165
+ trainer.nnodes=1 \
166
+ trainer.save_freq=-1 \
167
+ trainer.test_freq=1 \
168
+ trainer.total_epochs=15 $@
frameworks/agentgym_rl/docs/examples/ppo_code_architecture.rst ADDED
@@ -0,0 +1,207 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ PPO Example Architecture
2
+ ========================
3
+
4
+ Let's start with the Proximal Policy Optimization algorithm, which is
5
+ most widely used algorithm in LLM post-training.
6
+
7
+ The main entry point of the PPO algorithm example is:
8
+ `main_ppo.py <https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py>`_.
9
+ In this tutorial, we will go through the code architecture in `main_ppo.py <https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py>`_.
10
+
11
+ Define the data
12
+ ---------------
13
+
14
+ Users need to preprocess and store the dataset in parquet files.
15
+ And we implement `RLHFDataset` to load and tokenize the parquet files.
16
+
17
+ For ``RLHFDataset`` (Default), at least 1 fields are required:
18
+
19
+ - ``prompt``: Contains the string prompt
20
+
21
+ We already provide some examples of processing the datasets to parquet
22
+ files in `data_preprocess directory <https://github.com/volcengine/verl/blob/main/examples/data_preprocess>`_. Currently, we support
23
+ preprocess of GSM8k, MATH, Hellasage, Full_hh_rlhf datasets. See :doc:`../preparation/prepare_data` for
24
+ more information.
25
+
26
+ Define the reward functions for different datasets
27
+ --------------------------------------------------
28
+
29
+ In this main entry point, the users only need to define their own reward
30
+ function based on the datasets (or applications) utilized in PPO
31
+ training.
32
+
33
+ For example, we already provide reward functions for `GSM8k <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/gsm8k.py>`_
34
+ and `MATH <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/math.py>`_
35
+ datasets in the ``_select_rm_score_fn``. In the ``RewardManager``, we
36
+ will compute the reward score based on the data_source to select
37
+ corresponding reward functions. For some RLHF datasets (e.g.,
38
+ full_hh_rlhf), the reward model is utilized to assess the responses
39
+ without any reward functions. In this case, the ``RewardManager`` will
40
+ return the ``rm_score`` computed by the reward model directly.
41
+
42
+ See `reward functions <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score>`_ for detailed implementation.
43
+
44
+ Define worker classes
45
+ ---------------------
46
+
47
+ .. code:: python
48
+
49
+ if config.actor_rollout_ref.actor.strategy == 'fsdp': # for FSDP backend
50
+ assert config.actor_rollout_ref.actor.strategy == config.critic.strategy
51
+ from verl.workers.fsdp_workers import ActorRolloutRefWorker, CriticWorker
52
+ from verl.single_controller.ray import RayWorkerGroup
53
+ ray_worker_group_cls = RayWorkerGroup
54
+
55
+ elif config.actor_rollout_ref.actor.strategy == 'megatron': # for Megatron backend
56
+ assert config.actor_rollout_ref.actor.strategy == config.critic.strategy
57
+ from verl.workers.megatron_workers import ActorRolloutRefWorker, CriticWorker
58
+ from verl.single_controller.ray.megatron import NVMegatronRayWorkerGroup
59
+ ray_worker_group_cls = NVMegatronRayWorkerGroup # Ray worker class for Megatron-LM
60
+
61
+ else:
62
+ raise NotImplementedError
63
+
64
+ from verl.trainer.ppo.ray_trainer import ResourcePoolManager, Role
65
+
66
+ role_worker_mapping = {
67
+ Role.ActorRollout: ActorRolloutRefWorker,
68
+ Role.Critic: CriticWorker,
69
+ Role.RefPolicy: ActorRolloutRefWorker
70
+ }
71
+
72
+ global_pool_id = 'global_pool'
73
+ resource_pool_spec = {
74
+ global_pool_id: [config.trainer.n_gpus_per_node] * config.trainer.nnodes,
75
+ }
76
+ mapping = {
77
+ Role.ActorRollout: global_pool_id,
78
+ Role.Critic: global_pool_id,
79
+ Role.RefPolicy: global_pool_id,
80
+ }
81
+
82
+ Step 1: Construct the mapping between roles and workers
83
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
84
+
85
+ A role represents a group of workers in the same process. We have
86
+ pre-defined several roles in `ray_trainer.py <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py#L38>`_.
87
+
88
+ .. code:: python
89
+
90
+ class Role(Enum):
91
+ """
92
+ To create more roles dynamically, you can subclass Role and add new members
93
+ """
94
+ Actor = 0 # This worker only has Actor
95
+ Rollout = 1 # This worker only has Rollout
96
+ ActorRollout = 2 # This worker has both actor and rollout, it's a HybridEngine
97
+ Critic = 3 # This worker only has critic
98
+ RefPolicy = 4 # This worker only has reference policy
99
+ RewardModel = 5 # This worker only has reward model
100
+ ActorRolloutRef = 6 # This worker contains actor, rollout and reference policy simultaneously
101
+
102
+ Step 2: Define the worker class corresponding to this role
103
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
104
+
105
+ - We have pre-implemented the ``ActorRolloutRefWorker``. Through
106
+ different configs, it can be a standalone actor, a standalone rollout,
107
+ an ActorRollout HybridEngine, or an ActorRolloutRef HybridEngine
108
+ - We also pre-implemented workers for ``Actor``, ``Rollout``,
109
+ ``Critic``, ``Reward Model`` and ``Reference model`` on two different
110
+ backend: PyTorch FSDP
111
+ and Megatron-LM.
112
+ See `FSDP Workers <https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py>`_
113
+ and `Megatron-LM Workers <https://github.com/volcengine/verl/blob/main/verl/workers/megatron_workers.py>`_
114
+ for more information.
115
+
116
+ Step 3: Define resource pool id and resource pool spec
117
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
118
+
119
+ - Resource pool is a division of global GPU resources,
120
+ ``resource_pool_spec`` is a dict, mapping from id to # of GPUs
121
+
122
+ - In the above example, we defined a global resource pool:
123
+ global_pool_id, and then put all roles on this one resource pool
124
+ with all the GPUs in this post-training task. This refers to
125
+ *co-locate* placement where all the models share the same set of
126
+ GPUs.
127
+
128
+ - See resource pool and placement for advance usage.
129
+
130
+ Defining reward model/function
131
+ ------------------------------
132
+
133
+ .. code:: python
134
+
135
+ # we should adopt a multi-source reward function here
136
+ # - for rule-based rm, we directly call a reward score
137
+ # - for model-based rm, we call a model
138
+ # - for code related prompt, we send to a sandbox if there are test cases
139
+ # - finally, we combine all the rewards together
140
+ # - The reward type depends on the tag of the data
141
+ if config.reward_model.enable:
142
+ from verl.workers.fsdp_workers import RewardModelWorker
143
+ role_worker_mapping[Role.RewardModel] = RewardModelWorker
144
+ mapping[Role.RewardModel] = global_pool_id
145
+
146
+ reward_fn = RewardManager(tokenizer=tokenizer, num_examine=0)
147
+
148
+ # Note that we always use function-based RM for validation
149
+ val_reward_fn = RewardManager(tokenizer=tokenizer, num_examine=1)
150
+
151
+ resource_pool_manager = ResourcePoolManager(resource_pool_spec=resource_pool_spec, mapping=mapping)
152
+
153
+ Since not all tasks use model-based RM, users need to define here
154
+ whether it's a model-based RM or a function-based RM
155
+
156
+ - If it's a model-based RM, directly add the ``RewardModel`` role in the
157
+ resource mapping and add it to the resource pool mapping.
158
+
159
+ - Note that the pre-defined ``RewardModelWorker`` only supports models
160
+ with the structure of huggingface
161
+ ``AutoModelForSequenceClassification``. If it's not this model, you
162
+ need to define your own RewardModelWorker in `FSDP Workers <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/workers/fsdp_workers.py>`_
163
+ and `Megatron-LM Workers <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/workers/megatron_workers.py>`_.
164
+
165
+ - If it's a function-based RM, the users are required to classified the
166
+ reward function for each datasets.
167
+
168
+ .. code:: python
169
+
170
+ def _select_rm_score_fn(data_source):
171
+ if data_source == 'openai/gsm8k':
172
+ return gsm8k.compute_score
173
+ elif data_source == 'lighteval/MATH':
174
+ return math.compute_score
175
+ else:
176
+ raise NotImplementedError
177
+
178
+ See reward functions implemented in `directory <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/>`_
179
+ for more information.
180
+
181
+ Define, init and run the PPO Trainer
182
+ ------------------------------------
183
+
184
+ .. code:: python
185
+
186
+ trainer = RayPPOTrainer(config=config,
187
+ tokenizer=tokenizer,
188
+ role_worker_mapping=role_worker_mapping,
189
+ resource_pool_manager=resource_pool_manager,
190
+ ray_worker_group_cls=ray_worker_group_cls,
191
+ reward_fn=reward_fn,
192
+ val_reward_fn=val_reward_fn)
193
+ trainer.init_workers()
194
+ trainer.fit()
195
+
196
+ - We first initialize the ``RayPPOTrainer`` with user config, tokenizer
197
+ and all the above worker mapping, resource pool, worker group and
198
+ reward functions
199
+ - We first call the ``trainer.init_workers()`` to initialize the models
200
+ on the allocated GPUs (in the resource pool)
201
+ - The actual PPO training will be executed in ``trainer.fit()``
202
+
203
+ verl can be easily extended to other RL algorithms by reusing the Ray
204
+ model workers, resource pool and reward functions. See :doc:`extension<../advance/dpo_extension>` for
205
+ more information.
206
+
207
+ Details of the ``RayPPOTrainer`` is discussed in :doc:`Ray Trainer<../workers/ray_trainer>`.
frameworks/agentgym_rl/docs/experiment/ppo.rst ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ .. _algo-baseline-page:
2
+
3
+ Algorithm Baselines
4
+ ===================
5
+
6
+ GSM8k
7
+ ------------------
8
+
9
+ Assuming GSM8k dataset is preprocess via ``python3 examples/data_preprocess/gsm8k.py``
10
+
11
+ Refer to the table below to reproduce PPO training from different pre-trained models.
12
+
13
+ .. _Huggingface: https://huggingface.co/google/gemma-2-2b-it#benchmark-results
14
+ .. _SFT Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/gemma-2-2b-it-sft-0.411.log
15
+ .. _SFT+PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/gemma-2-2b-it-ppo-bsz512_4-prompt1024-resp-512-0.640.log
16
+ .. _wandb: https://api.wandb.ai/links/verl-team/h7ux8602
17
+ .. _Qwen Blog: https://qwenlm.github.io/blog/qwen2.5-llm/
18
+ .. _PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/Qwen2.5-0.5B-bsz256_2-prompt1024-resp512-0.567.log
19
+ .. _Megatron PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/deepseek-llm-7b-chat-megatron-bsz256_4-prompt512-resp512-0.695.log
20
+ .. _Qwen7b GRPO Script: https://github.com/volcengine/verl/blob/a65c9157bc0b85b64cd753de19f94e80a11bd871/examples/grpo_trainer/run_qwen2-7b_seq_balance.sh
21
+ .. _Megatron wandb: https://wandb.ai/verl-team/verl_megatron_gsm8k_examples/runs/10fetyr3
22
+ .. _Qwen7b ReMax Script: https://github.com/eric-haibin-lin/verl/blob/main/examples/remax_trainer/run_qwen2.5-3b_seq_balance.sh
23
+ .. _Qwen7b ReMax Wandb: https://wandb.ai/liziniu1997/verl_remax_example_gsm8k/runs/vxl10pln
24
+
25
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
26
+ | Model | Method | Test score | Details |
27
+ +==================================+========================+============+=====================+=========================================================================+
28
+ | google/gemma-2-2b-it | pretrained checkpoint | 23.9 | `Huggingface`_ |
29
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
30
+ | google/gemma-2-2b-it | SFT | 52.06 | `SFT Command and Logs`_ |
31
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
32
+ | google/gemma-2-2b-it | SFT + PPO | 64.02 | `SFT+PPO Command and Logs`_, `wandb`_ |
33
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
34
+ | Qwen/Qwen2.5-0.5B-Instruct | pretrained checkpoint | 36.4 | `Qwen Blog`_ |
35
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
36
+ | Qwen/Qwen2.5-0.5B-Instruct | PPO | 56.7 | `PPO Command and Logs`_ |
37
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
38
+ | deepseek-ai/deepseek-llm-7b-chat | PPO | 69.5 [1]_ | `Megatron PPO Command and Logs`_, `Megatron wandb`_ |
39
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
40
+ | Qwen/Qwen2-7B-Instruct | GRPO | 89 | `Qwen7b GRPO Script`_ |
41
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
42
+ | Qwen/Qwen2.5-7B-Instruct | ReMax | 97 | `Qwen7b ReMax Script`_, `Qwen7b ReMax Wandb`_ |
43
+ +----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+
44
+
45
+ .. [1] During the evaluation, we have only extracted answers following the format "####". A more flexible answer exaction, longer response length and better prompt engineering may lead to higher score.
frameworks/agentgym_rl/docs/faq/faq.rst ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Frequently Asked Questions
2
+ ====================================
3
+
4
+ Ray related
5
+ ------------
6
+
7
+ How to add breakpoint for debugging with distributed Ray?
8
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9
+
10
+ Please checkout the official debugging guide from Ray: https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html
11
+
12
+
13
+ Distributed training
14
+ ------------------------
15
+
16
+ How to run multi-node post-training with Ray?
17
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
18
+
19
+ You can start a ray cluster and submit a ray job, following the official guide from Ray: https://docs.ray.io/en/latest/ray-core/starting-ray.html
20
+
21
+ If your cluster is managed by Slurm, please refer to the guide for deploying Ray on Slurm: https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html
frameworks/agentgym_rl/docs/hybrid_flow.rst ADDED
@@ -0,0 +1,269 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ =========================================================
2
+ HybridFlow Programming Guide
3
+ =========================================================
4
+
5
+ .. _vermouth: https://github.com/vermouth1992
6
+
7
+ Author: `Chi Zhang <https://github.com/vermouth1992>`_
8
+
9
+ verl is an open source implementation of the paper `HybridFlow <https://arxiv.org/abs/2409.19256v2>`_ [1]_. In this section, we will introduce the basic concepts of HybridFlow, the motivation and how to program with verl APIs.
10
+
11
+ Motivation and Design
12
+ ------------------------
13
+ We use dataflow to represent RL systems. [4]_.
14
+
15
+ DataFlow
16
+ ~~~~~~~~~~~~~~~~~~~~
17
+
18
+ Dataflow is an abstraction of computations. Neural Netowork training is a typical dataflow. It can be represented by computational graph.
19
+
20
+ .. image:: https://github.com/eric-haibin-lin/verl-community/blob/main/docs/dataflow.jpeg?raw=true
21
+ :alt: The dataflow graph from CS231n 2024 lecture 4
22
+
23
+ This figure [2]_ represents the computation graph of a polynomial function followed by a sigmoid function. In the data flow of neural network computation, each node represents an operator, and each edge represents the direction of forward/backward propagation. The computation graph determines the architecture of the neural network.
24
+
25
+ RL as a dataflow problem
26
+ ++++++++++++++++++++++++++++++++++++++++++++++
27
+
28
+ Reinforcement learning (RL) training can also be represented as a dataflow. Below is the dataflow graph that represents the PPO algorithm used in RLHF [3]_:
29
+
30
+ .. image:: https://picx.zhimg.com/70/v2-cb8ab5ee946a105aab6a563e92682ffa_1440w.avis?source=172ae18b&biz_tag=Post
31
+ :alt: PPO dataflow graph, credit to Zhihu 低级炼丹师
32
+
33
+ However, the dataflow of RL has fundamental differences compared with dataflow of neural network training as follows:
34
+
35
+ +--------------------------+--------------------------------------------------+---------------------+
36
+ | Workload | Node | Edge |
37
+ +--------------------------+--------------------------------------------------+---------------------+
38
+ | Neural Network Training | Operator (+/-/matmul/softmax) | Tensor movement |
39
+ +--------------------------+--------------------------------------------------+---------------------+
40
+ | Reinforcement Learning | High-level operators (rollout/model forward) | Data Movement |
41
+ +--------------------------+--------------------------------------------------+---------------------+
42
+
43
+ In the case of tabular reinforcement learning, each operator is a simple scalar math operation (e.g., bellman update). In deep reinforcement learning(DRL), each operator is a high-level neural network computation such as model inference/update. This makes RL a two-level dataflow problem:
44
+
45
+ - Control flow: defines how the high-level operators are executed (e.g., In PPO, we first perform rollout. Then, we perform advantage computation. Finally, we perform training). It expresses the **core logics of RL algorithms**.
46
+ - Computation flow: defines the dataflow of **neural network computation** (e.g., model forward/backward/optimizer).
47
+
48
+
49
+ Design Choices
50
+ ~~~~~~~~~~~~~~~~~~~~
51
+ The model size used in DRL before the LLM era is typically small. Thus, the high-level neural network computation can be done in a single process. This enables embedding the computation flow inside the control flow as a single process.
52
+
53
+ However, in the LLM era, the computation flow (e.g., training neural network) becomes a multi-process program. This naturally leads to two design choices:
54
+
55
+ 1. Convert the control flow into a multi-process program as well. Then colocate with computation flow (unified multi-controller)
56
+
57
+ - Advantages:
58
+
59
+ - Achieves the **optimal performance** under fixed computation flow and control flow as the communication overhead in both training and data transfer is minimized.
60
+
61
+ - Disadvantages:
62
+
63
+ - The computation and/or control flow is **hard to reuse** from software perspective as computation code is coupled with specific controller code. For example, the training loop of PPO is generic. Say we have an PPO training flow implemented with a specific computation flow such as FSDP. Neither the control flow or computation flow can be reused if we want to switch the computation flow from FSDP to Megatron, due to the coupling of control and computation flows.
64
+ - Requires more efforts from the user under flexible and dynamic control flows, due to the multi-process nature of the program.
65
+
66
+ 2. Separate the flows: single process for the control flow and multi-process for computation flow
67
+
68
+ - Advantages:
69
+
70
+ - The computation flow defined elsewhere can be **easily reused** after the decoupling.
71
+ - The controller runs on a single process. Implementing a new RL algorithm with a **different control flow is simple and easy**.
72
+
73
+ - Disadvantages:
74
+
75
+ - Additional **data communication overhead** each time the controller process and computatation processes interact. The data has to be sent back and forth.
76
+
77
+ In verl, the latter strategy with separate control flow and computation flow is adopted. verl is designed to decouple the control flow of RL algorithms, and the implementation of computation engines.
78
+
79
+ Overall Execution Diagram
80
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
81
+
82
+ Below is a simplified diagram denoting the execution of a reinforcement learning job. In the diagram, the controller runs on a single process, while the generator/actor workers, critic workers run on multiple processes, placed with specific resource groups. For rollout, the controller passes the data to the generator to perform sample generation. When the rollout is done, the data is passed back to controller for the next step of the algorithm. Similar execution is done for other workers. With the hybrid controller design, the data flow and computation is decoupled to provide both efficiency in computation and flexiblity in defining algorithm training loops.
83
+
84
+ .. figure:: https://github.com/eric-haibin-lin/verl-community/blob/main/docs/driver_worker.png?raw=true
85
+ :alt: The execution diagram
86
+
87
+ Codebase walkthrough (PPO)
88
+ ------------------------------------------------
89
+
90
+ Entry function
91
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
92
+ Code: https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py
93
+
94
+ In this file, we define a remote function `main_task` that serves as the controller (driver) process as shown in the above figure. We also define a ``RewardManager``, where users can customize their reward function based on the data source in the dataset. Note that `RewardManager` should return the final token-level reward that is optimized by RL algorithms. Note that users can combine model-based rewards and rule-based rewards.
95
+ The ``main_task`` constructs a RayPPOTrainer instance and launch the fit. Note that ``main_task`` **runs as a single process**.
96
+
97
+ We highly recommend that the ``main_task`` is NOT schduled on the head of the ray cluster because ``main_task`` will consume a lot of memory but the head usually contains very few resources.
98
+
99
+ Ray trainer
100
+ ~~~~~~~~~~~~~~~~~~~~
101
+ Code: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py
102
+
103
+ The RayPPOTrainer manages
104
+
105
+ - Worker and WorkerGroup construction
106
+ - Runs the main loop of PPO algorithm
107
+
108
+ Note that, the fit function of RayPPOTrainer **runs as a single process**.
109
+
110
+ Worker and WorkerGroup construction
111
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
112
+
113
+ Each workerGroup manages a list of workers that runs remotely. Note that the worker group runs in the process of its construtor.
114
+ Each worker inside the WorkerGroup runs on a GPU. The worker group serves as a proxy for the controller process to interact with a list of workers, in order to perform certain computations. **In order to do so, we have to bind the methods of the worker into the method of the WorkerGroup and define the data dispatch and data collection**. This is done via simple decoration that will be introduced in the Worker definition section.
115
+
116
+ For example, in PPO, we define 3 worker groups:
117
+
118
+ - ActorRolloutRef: manages actor, rollout and reference policy. ActorRolloutRefWorker can be instantiated as a single actor, a single rollout, a single reference policy, a combined actor/rollout or a combined actor/rollout/ref. This design is aimed for the maximum code reuse in various scenarios. The reason for colocating actor and rollout is for fast weight transfer using nccl. The reason for coloating actor and reference is to implement an efficient lora PPO as the reference policy is simply the base model of PPO in lora.
119
+ - Critic: manages the critic model
120
+ - Reward: manages the reward model
121
+
122
+ The worker group will be constructed on the resource pool it designates. The resource pool is a set of GPUs in the ray cluster.
123
+
124
+ Worker definition
125
+ ~~~~~~~~~~~~~~~~~~~~
126
+
127
+ .. _ActorRolloutRefWorker: https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py
128
+
129
+ We take `ActorRolloutRefWorker <_ActorRolloutRefWorker>`_ for an exmaple.
130
+ The APIs it should expose to the controller process are:
131
+
132
+ - init_model: build the underlying model
133
+ - generate_sequences: given prompts, generate responses
134
+ - compute_log_prob: compute the log-probability of a generated sequence using actor
135
+ - compute_ref_log_prob: compute the log-probability of a generated sequence using reference policy
136
+ - save_checkpoint: save the checkpoint
137
+
138
+ Note that these methods are defined in the worker that can only be invoked via remote calls. For example, if the controller process wants to initialize the model, it has to call
139
+
140
+ .. code-block:: python
141
+
142
+ for worker in actor_rollout_ref_wg:
143
+ worker.init_model.remote()
144
+
145
+ If the controller process wants to generate sequences, it has to call
146
+
147
+ .. code-block:: python
148
+
149
+ data = xxx
150
+ # split the data into dp chunks
151
+ data_dp_lst = data.split(dp_size)
152
+ output_dp_lst = []
153
+ for i, worker in enumerate(actor_rollout_ref_wg):
154
+ output_future = worker.generate_sequences.remote(data_dp_lst[i])
155
+ output_dp_lst.append(output_future)
156
+ output = torch.cat(ray.get(output_dp_lst), dim=0)
157
+
158
+ We observe that controll process calling worker group methods in general can be divided into 3 parts:
159
+
160
+ - Split the data into data parallel sizes
161
+ - Dispatch the corresponding data into each worker
162
+ - Collect and concatenate the data when the computation finishes
163
+
164
+ In verl, we design a syntax sugar to encapsulate the 3 processes into a single call from the controller process.
165
+
166
+ .. code-block:: python
167
+
168
+ @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
169
+ def generate_sequences(data):
170
+ ...
171
+
172
+ # on the driver
173
+ output = actor_rollout_ref_wg.generate_sequences(data)
174
+
175
+ We decorate the method of the worker with a ``register`` that explicitly defines how the input data should be splitted and dispatch to each worker, and how the output data should be collected and concatenated by the controller. For example, ``Dispatch.DP_COMPUTE_PROTO`` splits the input data into dp chunks, dispatch each data to each worker, collect the output and concatenate the results. Note that this function requires the input and output to be a DataProto defined here (https://github.com/volcengine/verl/blob/main/verl/protocol.py).
176
+
177
+
178
+ PPO main loop
179
+ ~~~~~~~~~~~~~~~~~~~~
180
+ With the aforementioned APIs, we can implement the main loop of PPO as if it is a single process program
181
+
182
+ .. code-block:: python
183
+
184
+ for prompt in dataloader:
185
+ output = actor_rollout_ref_wg.generate_sequences(prompt)
186
+ old_log_prob = actor_rollout_ref_wg.compute_log_prob(output)
187
+ ref_log_prob = actor_rollout_ref_wg.compute_ref_log_prob(output)
188
+ values = critic_wg.compute_values(output)
189
+ rewards = reward_wg.compute_scores(output)
190
+ # compute_advantages is running directly on the control process
191
+ advantages = compute_advantages(values, rewards)
192
+ output = output.union(old_log_prob)
193
+ output = output.union(ref_log_prob)
194
+ output = output.union(values)
195
+ output = output.union(rewards)
196
+ output = output.union(advantages)
197
+ # update actor
198
+ actor_rollout_ref_wg.update_actor(output)
199
+ critic.update_critic(output)
200
+
201
+ Takeaways
202
+ ~~~~~~~~~~~~~~~~~~~~
203
+ - This programming paradigm enables users to use different computation backend without modification of the control process.
204
+ - This programming paradigm enables flexible placement (by changing the mapping of WorkerGroup and ResourcePool) without modification of the control process.
205
+
206
+ Repository organization
207
+ ------------------------------------------------
208
+
209
+ Important code files in the repository are organized as below:
210
+
211
+ .. code-block:: bash
212
+
213
+ verl # the verl package
214
+ trainer
215
+ main_ppo.py # the entrypoint for RL training
216
+ ppo
217
+ ray_trainer.py # the training loop for RL algorithms such as PPO
218
+ fsdp_sft_trainer.py # the SFT trainer with FSDP backend
219
+ config
220
+ generation.yaml # configuration template for rollout
221
+ ppo_trainer.yaml # configuration template for the RL trainer
222
+ workers
223
+ protocol.py # the interface of DataProto
224
+ fsdp_workers.py # the FSDP worker interfaces: ActorRolloutRefWorker, CriticWorker, RewardModelWorker
225
+ megatron_workers.py # the Megatron worker interfaces: ActorRolloutRefWorker, CriticWorker, RewardModelWorker
226
+ actor
227
+ dp_actor.py # data parallel actor with FSDP backend
228
+ megatron_actor.py # nD parallel actor with Megatron backend
229
+ critic
230
+ dp_critic.py # data parallel critic with FSDP backend
231
+ megatron_critic.py # nD parallel critic with FSDP backend
232
+ reward_model
233
+ megatron
234
+ reward_model.py # reward model with Megatron backend
235
+ rollout
236
+ vllm
237
+ vllm_rollout.py # rollout with vllm backend
238
+ hf_rollout.py # rollout with huggingface TGI backend
239
+ sharding_manager
240
+ fsdp_ulysses.py # data and model resharding when using FSDP + ulysses
241
+ fsdp_vllm.py # data and model resharding when using FSDP + ulysses + vllm
242
+ megatron_vllm.py # data and model resharding when using Megatron + vllm
243
+ utils
244
+ dataset # datasets for SFT/RM/RL
245
+ reward_score # function based reward
246
+ gsm8k.py # reward function for gsm8k dataset
247
+ math.py # reward function for math dataset
248
+ seqlen_balancing.py # the sequence balance optimization
249
+ models
250
+ llama # Megatron implementation for llama, deepseek, mistral, etc
251
+ transformers # ulysses integration with transformer models such as llama, qwen, etc
252
+ weight_loader_registery.py # registry of weight loaders for loading hf ckpt into Megatron
253
+ third_party
254
+ vllm # adaptor for vllm's usage in RL
255
+ vllm_v_0_6_3 # vllm v0.6.3 adaptor
256
+ llm.py # entrypoints for generate, sync_model_weight, offload_model_weights
257
+ parallel_state.py # vllm related device mesh and process groups
258
+ dtensor_weight_loaders.py # weight loader for huggingface models with FSDP
259
+ megatron_weight_loaders.py # weight loader for Megatron models
260
+ vllm_spmd # vllm >= v0.7 adaptor (coming soon)
261
+ examples # example scripts
262
+ tests # integration and unit tests
263
+ .github # the configuration of continuous integration tests
264
+
265
+
266
+ .. [1] HybridFlow: A Flexible and Efficient RLHF Framework: https://arxiv.org/abs/2409.19256v2
267
+ .. [2] Data flow graph credit to CS231n 2024 lecture 4: https://cs231n.stanford.edu/slides/2024/lecture_4.pdf
268
+ .. [3] PPO dataflow graph credit to 低级炼丹师 from Zhihu​: https://zhuanlan.zhihu.com/p/635757674
269
+ .. [4] RLFlow
frameworks/agentgym_rl/docs/index.rst ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Welcome to verl's documentation!
2
+ ================================================
3
+
4
+ .. _hf_arxiv: https://arxiv.org/pdf/2409.19256
5
+
6
+ verl is a flexible, efficient and production-ready RL training framework designed for large language models (LLMs) post-training. It is an open source implementation of the `HybridFlow <hf_arxiv>`_ paper.
7
+
8
+ verl is flexible and easy to use with:
9
+
10
+ - **Easy extension of diverse RL algorithms**: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code.
11
+
12
+ - **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks.
13
+
14
+ - **Flexible device mapping and parallelism**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.
15
+
16
+ - Readily integration with popular HuggingFace models
17
+
18
+
19
+ verl is fast with:
20
+
21
+ - **State-of-the-art throughput**: By seamlessly integrating existing SOTA LLM training and inference frameworks, verl achieves high generation and training throughput.
22
+
23
+ - **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.
24
+
25
+ --------------------------------------------
26
+
27
+ .. _Contents:
28
+
29
+ .. toctree::
30
+ :maxdepth: 5
31
+ :caption: Quickstart
32
+
33
+ start/install
34
+ start/quickstart
35
+
36
+ .. toctree::
37
+ :maxdepth: 4
38
+ :caption: Programming guide
39
+
40
+ hybrid_flow
41
+
42
+ .. toctree::
43
+ :maxdepth: 5
44
+ :caption: Data Preparation
45
+
46
+ preparation/prepare_data
47
+ preparation/reward_function
48
+
49
+ .. toctree::
50
+ :maxdepth: 5
51
+ :caption: Configurations
52
+
53
+ examples/config
54
+
55
+ .. toctree::
56
+ :maxdepth: 2
57
+ :caption: PPO Example
58
+
59
+ examples/ppo_code_architecture
60
+ examples/gsm8k_example
61
+
62
+ .. toctree::
63
+ :maxdepth: 1
64
+ :caption: PPO Trainer and Workers
65
+
66
+ workers/ray_trainer
67
+ workers/fsdp_workers
68
+ workers/megatron_workers
69
+
70
+ .. toctree::
71
+ :maxdepth: 1
72
+ :caption: Performance Tuning Guide
73
+
74
+ perf/perf_tuning
75
+
76
+ .. toctree::
77
+ :maxdepth: 1
78
+ :caption: Experimental Results
79
+
80
+ experiment/ppo
81
+
82
+ .. toctree::
83
+ :maxdepth: 1
84
+ :caption: Advance Usage and Extension
85
+
86
+ advance/placement
87
+ advance/dpo_extension
88
+ advance/fsdp_extension
89
+ advance/megatron_extension
90
+
91
+ .. toctree::
92
+ :maxdepth: 1
93
+ :caption: FAQ
94
+
95
+ faq/faq
96
+
97
+ Contribution
98
+ -------------
99
+
100
+ verl is free software; you can redistribute it and/or modify it under the terms
101
+ of the Apache License 2.0. We welcome contributions.
102
+ Join us on `GitHub <https://github.com/volcengine/verl>`_, `Slack <https://join.slack.com/t/verlgroup/shared_invite/zt-2w5p9o4c3-yy0x2Q56s_VlGLsJ93A6vA>`_ and `Wechat <https://raw.githubusercontent.com/eric-haibin-lin/verl-community/refs/heads/main/WeChat.JPG>`_ for discussions.
103
+
104
+ Code formatting
105
+ ^^^^^^^^^^^^^^^^^^^^^^^^
106
+ We use yapf (Google style) to enforce strict code formatting when reviewing MRs. Run yapf at the top level of verl repo:
107
+
108
+ .. code-block:: bash
109
+
110
+ pip3 install yapf
111
+ yapf -ir -vv --style ./.style.yapf verl examples tests
frameworks/agentgym_rl/docs/perf/perf_tuning.rst ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Performance Tuning Guide
2
+ ==============================
3
+
4
+ Author: `Guangming Sheng <https://github.com/PeterSH6>`_
5
+
6
+ In this section, we will discuss how to tune the performance of all the stages in verl, including:
7
+
8
+ 1. Rollout generation throughput.
9
+
10
+ 2. Enable `use_remove_padding=True` for sequence packing (i.e., data packing and remove padding).
11
+
12
+ 3. Batch size tuning for forward and backward computation
13
+
14
+ 4. Enable ``use_dynamic_bsz=True`` for higher throughput.
15
+
16
+ 5. Utilize Ulysses Sequence Parallel for Long Context Training
17
+
18
+ 6. LigerKernel for SFT performance optimization
19
+
20
+ Rollout Generation Tuning
21
+ --------------------------
22
+
23
+ verl currently supports two rollout backends: vLLM and TGI (with SGLang support coming soon).
24
+
25
+ Below are key factors for tuning vLLM-based rollout. Before tuning, we recommend setting ``actor_rollout_ref.rollout.disable_log_stats=False`` so that rollout statistics are logged.
26
+
27
+ - Increase ``gpu_memory_utilization``. The vLLM pre-allocates GPU KVCache by using gpu_memory_utilization% of the remaining memory.
28
+ However, if model parameters and optimizer states are not offloaded, using too high a fraction can lead to OOM.
29
+ A value between 0.5 and 0.7 often strikes a good balance between high throughput and avoiding OOM.
30
+
31
+ - Adjust ``max_num_seqs`` or ``max_num_batched_tokens``.
32
+ If the GPU cache utilization is relatively low in the log, increase ``max_num_seqs`` or ``max_num_batched_tokens``
33
+ can enlarge the effective batch size in the decoding stage, allowing more concurrent requests per batch.
34
+ We recommend setting ``max_num_batched_tokens > 2048`` for higher throughput.
35
+
36
+ - Use a smaller ``tensor_parallel_size``.
37
+ When GPU resources allow, a smaller tensor parallel size spawns more vLLM replicas.
38
+ Data parallelism (DP) can yield higher throughput than tensor parallelism (TP), but also increases KVCache consumption.
39
+ Carefully balance the trade-off between more replicas and higher memory usage.
40
+ Our experient in Sec. 8.4 of `HybridFlow paper <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/gsm8k.py>`_ evaluate this trade-off.
41
+
42
+ More tuning details such as dealing with Preemption and Chunked-prefill
43
+ can be found in `vLLM official tuning guide <https://docs.vllm.ai/en/latest/performance/optimization.html>`_
44
+
45
+ The performance of vllm can be further increased if upgrading from v0.6.3 to v0.7. See https://github.com/volcengine/verl/blob/main/docs/README_vllm0.7.md for details on how to upgrade.
46
+
47
+ Enable remove padding (sequence packing)
48
+ -----------------------------------------
49
+
50
+ Currently, for llama, mistral, gemma1 and qwen based models, users can enable `use_remove_padding=True` to utilize the
51
+ sequence packing implementation provided by transformers library.
52
+
53
+ For other models, transformers library may also support it but we haven't tested it yet.
54
+ Users can add the desired model config to the `test_transformer.py <https://github.com/volcengine/verl/blob/main/tests/model/test_transformer.py#L24>`_ file.
55
+ And test its functionaility by running the following command:
56
+
57
+ .. code-block:: bash
58
+
59
+ pytest -s tests/model/test_transformer.py
60
+
61
+ If the test passes, you can add your desired model into the model `registry.py <https://github.com/volcengine/verl/blob/main/verl/models/registry.py#L24>`_ file.
62
+ Then, you can enjoy the performance boost of sequence packing
63
+ and welcome to PR your tested model to verl!
64
+
65
+
66
+ Batch Size Tuning
67
+ -----------------
68
+
69
+ To achieve higher throughput in experience preparation (i.e., model fwd) and model update (i.e., actor/critic fwd/bwd),
70
+ users may need to tune the ``*micro_batch_size_per_gpu`` for different computation.
71
+
72
+ In verl, the core principle for setting batch sizes is:
73
+
74
+ - **Algorithmic metrics** (train batch size, PPO mini-batch size) are *global* (from a single-controller perspective),
75
+ normalized in each worker. See the `normalization code <https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py#L120-L122>`_.
76
+
77
+ - **Performance-related parameters** (micro batch size, max token length for dynamic batch size) are *local* parameters that define the per-GPU data allocations.
78
+ See the `normalization code <https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py#L127>`_.
79
+
80
+ .. note:: In your training script, please use ``*micro_batch_size_per_gpu`` instead of ``*micro_batch_size``.
81
+ So that you don't need to consider the normalization of the ``micro_batch_size`` and ``micro_batch_size`` will be deprecated.
82
+
83
+ Batch Size Tuning tips
84
+ """"""""""""""""""""""
85
+
86
+ Therefore, users may need to tune the ``*micro_batch_size_per_gpu`` to accelerate training. Here're some tips:
87
+
88
+ 1. **Enable gradient checkpointing**:
89
+ Set ``actor_rollout_ref.model.enable_gradient_checkpointing=True`` and ``critic.model.enable_gradient_checkpointing=True``.
90
+ This often allows for larger micro-batch sizes and will be beneficial for large mini-batch training.
91
+
92
+ 2. Increase the ``*micro_batch_size_per_gpu`` as much as possible till equals to normalized ``mini_batch_size``.
93
+
94
+ 3. **Use larger forward-only parameters**:
95
+ Forward only parameter, such as ``actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu``,
96
+ ``actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu``, ``critic.forward_micro_batch_size_per_gpu`` could be larger (e.g., 2x) than training related micro batch sizes,
97
+ such as ``actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu``, ``critic.ppo_micro_batch_size_per_gpu``.
98
+
99
+ 4. **Allow larger micro-batch sizes for Critic and Reward models**:
100
+ micro batch size of Critic and Reward model could be larger than Actor model. This is because the actor model has much larger vocab size in the final layer.
101
+
102
+
103
+ Tuning for Dynamic Batch Size
104
+ -----------------------------
105
+
106
+ Dynamic batch size is a technique that allows the model to process similar number of tokens in a single forward pass (with different actual batch sizes).
107
+ This can significantly improve the training efficiency and reduce the memory usage.
108
+
109
+ To utilize this technique, users can set ``use_dynamic_bsz=True`` in actor, ref, critic and reward models.
110
+ With ``use_dynamic_bsz=True``, users don't need to tune ``*micro_batch_size_per_gpu``.
111
+ Instead, users should tune the following parameters:
112
+
113
+ - ``actor_rollout_ref.actor.ppo_max_token_len_per_gpu``, ``critic.ppo_max_token_len_per_gpu``:
114
+ The maximum number of tokens to be processed in fwd and bwd of ``update_policy`` and ``update_critic``.
115
+
116
+ - ``actor_rollout_ref.ref.log_prob_max_token_len_per_gpu`` and ``actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu``:
117
+ The maximum number of tokens to be processed in a the fwd computation of ``compute_log_prob`` and ``comptue_ref_log_prob``.
118
+
119
+ - ``critic.forward_micro_batch_size_per_gpu``, ``reward_model.forward_micro_batch_size_per_gpu``:
120
+ The maximum number of tokens to be processed in a the fwd computation of ``compute_values``, ``compute_rm_score``.
121
+
122
+ Dynamic Batch Size Tuning tips
123
+ """"""""""""""""""""""""""""""
124
+
125
+ Here're some tips to tune the above parameters:
126
+
127
+ 1. **Increase** ``actor_rollout_ref.actor.ppo_max_token_len_per_gpu``
128
+ Make it at least 2 x (max_prompt_length + max_response_length). We set it to 3x in `run_qwen2-7b_rm_seq_balance.sh <https://github.com/volcengine/verl/blob/main/examples/ppo_trainer/run_qwen2-7b_rm_seq_balance.sh#L25>`_.
129
+ Try to increase it to get higher throughput.
130
+
131
+ 2. **Forward-only parameters can be larger**:
132
+ Similar to the non-dynamic-batch scenario, forward-only token limits can exceed those used in forward/backward operations.
133
+
134
+ 3. **Use larger limits for Critic and Reward models**:
135
+ Critic and Reward parameters can be set at least 2× the Actor’s limits. For instance, we set them to 4× here:
136
+ `run_qwen2-7b_rm_seq_balance.sh <https://github.com/volcengine/verl/blob/main/examples/ppo_trainer/run_qwen2-7b_rm_seq_balance.sh#L40>`_
137
+
138
+ .. :math:`\text{critic.ppo_max_token_len_per_gpu} = 2 \times \text{actor.ppo_max_token_len_per_gpu})`.
139
+
140
+ Ulysses Sequence Parallel for Long Context Training
141
+ ----------------------------------------------------
142
+
143
+ To utilize this technique, users can set ``ulysses_sequence_parallel_size>1`` in actor, ref, critic and reward models.
144
+
145
+ We support different model utilize different ulysses_sequence_parallel_size sizes.
146
+
147
+ To train log sequence (>32k), users may need to decrease the ``*micro_batch_size_per_gpu`` and ``*max_token_len_per_gpu`` to avoid OOM.
148
+
149
+ LigerKernel for SFT
150
+ ----------------------
151
+
152
+ LigerKernel is a high-performance kernel for Supervised Fine-Tuning (SFT) that can improve training efficiency. To enable LigerKernel in your SFT training:
153
+
154
+ 1. Install liger-kernel via ``pip3 install liger-kernel``. In your SFT configuration file (e.g., ``verl/trainer/config/sft_trainer.yaml``), set the ``use_liger`` parameter:
155
+
156
+ .. code-block:: yaml
157
+
158
+ model:
159
+ use_liger: True # Enable LigerKernel for SFT
160
+
161
+ 2. The default value is ``False``. Enable it only when you want to use LigerKernel's optimizations.
162
+
163
+ 3. LigerKernel is particularly useful for improving training performance in SFT scenarios.
164
+
frameworks/agentgym_rl/docs/preparation/prepare_data.rst ADDED
@@ -0,0 +1,126 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Prepare Data for Post-Training
2
+ ========================================
3
+
4
+ Before starting the post-training job, we need to prepare the data for
5
+ the policy training. The data should be stored in the parquet format.
6
+
7
+ We provide several data preprocess scripts for different datasets,
8
+ including GSM8K, MATH, HelloSwag, Full_hh_rlhf. To prepare other datasets, we need
9
+ to follow the following steps: The data preprocess script can be divided
10
+ into two parts:
11
+
12
+ 1. The first part is the common part, which loads the dataset from
13
+ huggingface's ``datasets`` package. Then preprocess the datasets with
14
+ the ``make_map_fn`` and then store in the parquet format.
15
+
16
+ .. code:: python
17
+
18
+ import re
19
+ import os
20
+ import datasets
21
+
22
+ from verl.utils.hdfs_io import copy, makedirs
23
+ import argparse
24
+
25
+ # To extract the solution for each prompts in the dataset
26
+ # def extract_solution(solution_str):
27
+ # ...
28
+
29
+
30
+ if __name__ == '__main__':
31
+ parser = argparse.ArgumentParser()
32
+ parser.add_argument('--local_dir', default='/opt/tiger/gsm8k')
33
+ parser.add_argument('--hdfs_dir', default=None)
34
+
35
+ args = parser.parse_args()
36
+
37
+ num_few_shot = 5
38
+ data_source = 'openai/gsm8k'
39
+
40
+ dataset = datasets.load_dataset(data_source, 'main')
41
+
42
+ train_dataset = dataset['train']
43
+ test_dataset = dataset['test']
44
+
45
+ # Construct a `def make_map_fn(split)` for the corresponding datasets.
46
+ # ...
47
+
48
+ train_dataset = train_dataset.map(function=make_map_fn('train'), with_indices=True)
49
+ test_dataset = test_dataset.map(function=make_map_fn('test'), with_indices=True)
50
+
51
+ local_dir = args.local_dir
52
+ hdfs_dir = args.hdfs_dir
53
+
54
+ train_dataset.to_parquet(os.path.join(local_dir, 'train.parquet'))
55
+ test_dataset.to_parquet(os.path.join(local_dir, 'test.parquet'))
56
+
57
+ makedirs(hdfs_dir)
58
+
59
+ copy(src=local_dir, dst=hdfs_dir)
60
+
61
+ 2. The users are required to implement the ``make_map_fn()`` function
62
+ (as well as the ``extract_solution``) on their own to support
63
+ different datasets or tasks.
64
+
65
+ We already implemented the data preprocess of GSM8k, MATH, Hellaswag and Full_hh_rlhf
66
+ datasets. And we take the GSM8k dataset as an example:
67
+
68
+ **GSM8K**
69
+
70
+ In the ``make_map_fn``, each data field should consist of the following
71
+ 5 fields:
72
+
73
+ 1. ``data_source``: The name of the dataset. To index the corresponding
74
+ reward function in the ``RewardModule``
75
+ 2. ``prompt``: This field should be constructed in the format of
76
+ huggingface chat_template. The tokenizer in ``RLHFDataset`` will
77
+ apply chat template and tokenize the prompt.
78
+ 3. ``ability``: Define the task category.
79
+ 4. ``reward_model``: Currently, we only utilize the ``ground_truth``
80
+ field during evaluation. The ``ground_truth`` is computed by the
81
+ ``extract_solution`` function. **NOTED** that the implementation of
82
+ the corresponding reward function should align with this extracted
83
+ ``ground_truth``.
84
+ 5. ``extra_info``: Record some information of the current prompt. Not
85
+ use for now.
86
+
87
+ .. code:: python
88
+
89
+ def extract_solution(solution_str):
90
+ solution = re.search("#### (\\-?[0-9\\.\\,]+)", solution_str) # extract the solution after ####
91
+ assert solution is not None
92
+ final_solution = solution.group(0)
93
+ final_solution = final_solution.split('#### ')[1].replace(',', '')
94
+ return final_solution
95
+
96
+ instruction_following = "Let's think step by step and output the final answer after \"####\"."
97
+
98
+ # add a row to each data item that represents a unique id
99
+ def make_map_fn(split):
100
+
101
+ def process_fn(example, idx):
102
+ question = example.pop('question')
103
+
104
+ question = question + ' ' + instruction_following
105
+
106
+ answer = example.pop('answer')
107
+ solution = extract_solution(answer)
108
+ data = {
109
+ "data_source": data_source,
110
+ "prompt": [{
111
+ "role": "user",
112
+ "content": question
113
+ }],
114
+ "ability": "math",
115
+ "reward_model": {
116
+ "style": "rule",
117
+ "ground_truth": solution
118
+ },
119
+ "extra_info": {
120
+ 'split': split,
121
+ 'index': idx
122
+ }
123
+ }
124
+ return data
125
+
126
+ return process_fn
frameworks/agentgym_rl/docs/preparation/reward_function.rst ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Implement Reward Function for Dataset
2
+ ======================================
3
+
4
+ For each dataset, we need to implement a reward function or utilize a reward model to compute the rewards for the generated responses.
5
+ We already pre-implemented some reward functions in `reward_score directory <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score>`_.
6
+
7
+ Currently, we support reward functions for GSM8k and MATH datasets. For RLHF datasets (e.g.,
8
+ full_hh_rlhf) and Code Generation (e.g., APPS), we utilize reward model
9
+ and SandBox (will opensource soon) for evaluation respectively.
10
+
11
+ RewardManager
12
+ -------------
13
+
14
+ In the entrypoint of the PPO Post-Training script `main_ppo.py <https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py#L33>`_,
15
+ we implement a ``RewardManager`` that utilze pre-implemented reward functions to compute the scores for each response.
16
+
17
+ In the ``RewardManager``, we implemented a ``__call__`` function to
18
+ compute the score for each response.
19
+ All the reward functions are executed by ``compute_score_fn``.
20
+ The input is a ``DataProto``, which includes:
21
+
22
+ - ``input_ids``, ``attention_mask``: ``input_ids`` and ``attention_mask`` after applying
23
+ chat_template, including prompt and response
24
+ - ``responses``: response tokens
25
+ - ``ground_truth``: The ground truth string of the current prompt.
26
+ Stored in ``non_tensor_batch`` in the ``DataProto``, which should be
27
+ preprocessed in the parquet files.
28
+ - ``data_source``: The dataset name of the current prompt. Stored in
29
+ ``non_tensor_batch`` in the ``DataProto``, which should be
30
+ preprocessed in the parquet files.
31
+
32
+ After detokenize the responses, the responses string and the ground
33
+ truth string will be input to the ``compute_score_fn`` to compute the
34
+ score for each response.
35
+
36
+ Reward Functions
37
+ ----------------
38
+ We already pre-implemented some reward functions in `reward_score directory <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score>`_.
39
+
40
+ - In the `GSM8k example <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/gsm8k.py>`_, we
41
+ force the response to output the final answer after four ####, then
42
+ use string matching to compare with the ground truth. If completely
43
+ correct, score 1 point; if the format is correct, score 0.1 points; if
44
+ the format is incorrect, score 0 points.
45
+ - In the `MATH example <https://github.com/volcengine/verl/blob/main/verl/utils/reward_score/math.py>`_, we follow
46
+ the implementation in `lm-evaluation-harness repository <https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/hendrycks_math/utils.py>`_.
frameworks/agentgym_rl/docs/requirements-docs.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ # markdown suport
2
+ recommonmark
3
+ # markdown table suport
4
+ sphinx-markdown-tables
5
+
6
+ # theme default rtd
7
+
8
+ # crate-docs-theme
9
+ sphinx-rtd-theme
frameworks/agentgym_rl/docs/start/install.rst ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Installation
2
+ ============
3
+
4
+ Requirements
5
+ ------------
6
+
7
+ - **Python**: Version >= 3.9
8
+ - **CUDA**: Version >= 12.1
9
+
10
+ verl supports various backends. Currently, the following configurations are available:
11
+
12
+ - **FSDP** and **Megatron-LM** (optional) for training.
13
+ - **vLLM** adn **TGI** for rollout generation, **SGLang** support coming soon.
14
+
15
+ Training backends
16
+ ------------------
17
+
18
+ We recommend using **FSDP** backend to investigate, research and prototype different models, datasets and RL algorithms. The guide for using FSDP backend can be found in :doc:`FSDP Workers<../workers/fsdp_workers>`.
19
+
20
+ For users who pursue better scalability, we recommend using **Megatron-LM** backend. Currently, we support Megatron-LM v0.4 [1]_. The guide for using Megatron-LM backend can be found in :doc:`Megatron-LM Workers<../workers/megatron_workers>`.
21
+
22
+
23
+ Install from docker image
24
+ -------------------------
25
+
26
+ We provide pre-built Docker images for quick setup.
27
+
28
+ Image and tag: ``verlai/verl:vemlp-th2.4.0-cu124-vllm0.6.3-ray2.10-te1.7-v0.0.3``. See files under ``docker/`` for NGC-based image or if you want to build your own.
29
+
30
+ 1. Launch the desired Docker image:
31
+
32
+ .. code:: bash
33
+
34
+ docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN -v <image:tag>
35
+
36
+
37
+ 2. Inside the container, install verl:
38
+
39
+ .. code:: bash
40
+
41
+ # install the stable version
42
+ pip3 install verl
43
+
44
+
45
+ 3. Setup Megatron (optional)
46
+
47
+ If you want to enable training with Megatron, Megatron code must be added to PYTHONPATH:
48
+
49
+ .. code:: bash
50
+
51
+ cd ..
52
+ git clone -b core_v0.4.0 https://github.com/NVIDIA/Megatron-LM.git
53
+ cp verl/patches/megatron_v4.patch Megatron-LM/
54
+ cd Megatron-LM && git apply megatron_v4.patch
55
+ pip3 install -e .
56
+ export PYTHONPATH=$PYTHONPATH:$(pwd)
57
+
58
+
59
+ You can also get the Megatron code after verl's patch via
60
+
61
+ .. code:: bash
62
+
63
+ git clone -b core_v0.4.0_verl https://github.com/eric-haibin-lin/Megatron-LM
64
+ export PYTHONPATH=$PYTHONPATH:$(pwd)/Megatron-LM
65
+
66
+ Install from custom environment
67
+ ---------------------------------
68
+
69
+ To manage environment, we recommend using conda:
70
+
71
+ .. code:: bash
72
+
73
+ conda create -n verl python==3.9
74
+ conda activate verl
75
+
76
+ For installing the latest version of verl, the best way is to clone and
77
+ install it from source. Then you can modify our code to customize your
78
+ own post-training jobs.
79
+
80
+ .. code:: bash
81
+
82
+ # install verl together with some lightweight dependencies in setup.py
83
+ pip3 install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124
84
+ pip3 install flash-attn --no-build-isolation
85
+ pip3 install verl
86
+
87
+
88
+ Megatron is optional. It's dependencies can be setup as below:
89
+
90
+ .. code:: bash
91
+
92
+ # apex
93
+ pip3 install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" \
94
+ git+https://github.com/NVIDIA/apex
95
+
96
+ # transformer engine
97
+ pip3 install git+https://github.com/NVIDIA/TransformerEngine.git@v1.7
98
+
99
+ # megatron core v0.4.0: clone and apply the patch
100
+ # You can also get the patched Megatron code patch via
101
+ # git clone -b core_v0.4.0_verl https://github.com/eric-haibin-lin/Megatron-LM
102
+ cd ..
103
+ git clone -b core_v0.4.0 https://github.com/NVIDIA/Megatron-LM.git
104
+ cd Megatron-LM
105
+ cp ../verl/patches/megatron_v4.patch .
106
+ git apply megatron_v4.patch
107
+ pip3 install -e .
108
+ export PYTHONPATH=$PYTHONPATH:$(pwd)
109
+
110
+
111
+ .. [1] Megatron v0.4 is supported with verl's patches to fix issues such as virtual pipeline hang. It will be soon updated with latest the version of upstream Megatron-LM without patches.
frameworks/agentgym_rl/docs/start/quickstart.rst ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ .. _quickstart:
2
+
3
+ =========================================================
4
+ Quickstart: PPO training on GSM8K dataset
5
+ =========================================================
6
+
7
+ Post-train a LLM using GSM8K dataset.
8
+
9
+ Introduction
10
+ ------------
11
+
12
+ .. _hf_dataset_gsm8k: https://huggingface.co/datasets/gsm8k
13
+
14
+ In this example, we train an LLM to tackle the `GSM8k <hf_dataset_gsm8k>`_ task with function-based rewards. [1]_
15
+
16
+ Prerequisite:
17
+
18
+ - the latest version of ``verl`` and its dependencies installed following the installation guide. Using the docker image is recommended.
19
+
20
+ - an GPU with at least 24 GB HBM
21
+
22
+
23
+ Dataset Introduction
24
+ --------------------
25
+
26
+ GSM8k is a math problem dataset. The prompt is an elementary school
27
+ problem. The LLM model is asked to solve the math problem. Below is an example:
28
+
29
+ Prompt
30
+
31
+ Katy makes coffee using teaspoons of sugar and cups of water in the
32
+ ratio of 7:13. If she used a total of 120 teaspoons of sugar and cups
33
+ of water, calculate the number of teaspoonfuls of sugar she used.
34
+
35
+ Solution
36
+
37
+ The total ratio representing the ingredients she used to make the
38
+ coffee is 7+13 = <<7+13=20>>20 Since the fraction representing the
39
+ number of teaspoons she used is 7/20, she used 7/20\ *120 =
40
+ <<7/20*\ 120=42>>42 #### 42
41
+
42
+ Step 1: Prepare the dataset
43
+ ----------------------------
44
+
45
+ We preprocess the dataset in parquet format so that (1) it contains necessary fields for computing RL rewards and (2) is faster to read.
46
+
47
+ .. code-block:: bash
48
+
49
+ python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k
50
+
51
+ Step 2: Download a model for post-training
52
+ -------------------------------------------
53
+
54
+ In this example, we start with the ``Qwen2.5-0.5B-Instruct`` model.
55
+
56
+ If you want to perform SFT before RL, refer to the :doc:`Complete GSM8K Example<../examples/gsm8k_example>`, the `sft directory <https://github.com/volcengine/verl/blob/main/examples/sft/gsm8k>`_ and `SFT Trainer <https://github.com/volcengine/verl/blob/main/verl/trainer/fsdp_sft_trainer.py>`_ for further details.
57
+
58
+ .. code-block:: bash
59
+
60
+ python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen2.5-0.5B-Instruct')"
61
+
62
+ Step 3: Perform PPO training with the instruct model
63
+ ----------------------------------------------------------------------
64
+
65
+ **Reward Model/Function**
66
+
67
+ We use a pre-defined rule-based reward model. We force the model to produce a final
68
+ answer following 4 “#” as shown in the solution. We extract the final
69
+ answer from both the solution and model's output using regular
70
+ expression matching. We assign a reward of 1 to correct
71
+ answer, 0.1 to incorrect answer and 0 to no answer.
72
+
73
+ For mode details, please refer to `verl/utils/reward_score/gsm8k.py <https://github.com/volcengine/verl/blob/v0.1/verl/utils/reward_score/gsm8k.py>`_.
74
+
75
+ **Training Script**
76
+
77
+ Now let's run PPO training with the dataset and model above. [2]_
78
+
79
+
80
+ Set the ``data.train_files`` ,\ ``data.val_files``, ``actor_rollout_ref.model.path`` and ``critic.model.path`` based on your dataset and model names or paths.
81
+
82
+ .. code-block:: bash
83
+
84
+ PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
85
+ data.train_files=$HOME/data/gsm8k/train.parquet \
86
+ data.val_files=$HOME/data/gsm8k/test.parquet \
87
+ data.train_batch_size=256 \
88
+ data.val_batch_size=1312 \
89
+ data.max_prompt_length=512 \
90
+ data.max_response_length=256 \
91
+ actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
92
+ actor_rollout_ref.actor.optim.lr=1e-6 \
93
+ actor_rollout_ref.actor.ppo_mini_batch_size=64 \
94
+ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
95
+ actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
96
+ actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
97
+ actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
98
+ actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
99
+ critic.optim.lr=1e-5 \
100
+ critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
101
+ critic.ppo_micro_batch_size_per_gpu=4 \
102
+ algorithm.kl_ctrl.kl_coef=0.001 \
103
+ trainer.logger=['console'] \
104
+ +trainer.val_before_train=False \
105
+ trainer.default_hdfs_dir=null \
106
+ trainer.n_gpus_per_node=1 \
107
+ trainer.nnodes=1 \
108
+ trainer.save_freq=10 \
109
+ trainer.test_freq=10 \
110
+ trainer.total_epochs=15 2>&1 | tee verl_demo.log
111
+
112
+ You are expected to see the following logs, indicating training in progress. The key metric ``val/test_score/openai/gsm8k`` is computed every ``trainer.test_freq`` steps:
113
+
114
+ .. code-block:: bash
115
+
116
+ step:0 - timing/gen:21.470 - timing/ref:4.360 - timing/values:5.800 - critic/kl:0.000 - critic/kl_coeff:0.001 - timing/adv:0.109 - timing/update_critic:15.664 - critic/vf_loss:14.947 - critic/vf_clipfrac:0.000 - critic/vpred_mean:-2.056 - critic/grad_norm:1023.278 - critic/lr(1e-4):0.100 - timing/update_actor:20.314 - actor/entropy_loss:0.433 - actor/pg_loss:-0.005 - actor/pg_clipfrac:0.000 - actor/ppo_kl:0.000 - actor/grad_norm:1.992 - actor/lr(1e-4):0.010 - critic/score/mean:0.004 - critic/score/max:1.000 - critic/score/min:0.000 - critic/rewards/mean:0.004 - critic/rewards/max:1.000 - critic/rewards/min:0.000 - critic/advantages/mean:-0.000 - critic/advantages/max:2.360 - critic/advantages/min:-2.280 - critic/returns/mean:0.003 - critic/returns/max:0.000 - critic/returns/min:0.000 - critic/values/mean:-2.045 - critic/values/max:9.500 - critic/values/min:-14.000 - response_length/mean:239.133 - response_length/max:256.000 - response_length/min:77.000 - prompt_length/mean:104.883 - prompt_length/max:175.000 - prompt_length/min:68.000
117
+ step:1 - timing/gen:23.020 - timing/ref:4.322 - timing/values:5.953 - critic/kl:0.000 - critic/kl_coeff:0.001 - timing/adv:0.118 - timing/update_critic:15.646 - critic/vf_loss:18.472 - critic/vf_clipfrac:0.384 - critic/vpred_mean:1.038 - critic/grad_norm:942.924 - critic/lr(1e-4):0.100 - timing/update_actor:20.526 - actor/entropy_loss:0.440 - actor/pg_loss:0.000 - actor/pg_clipfrac:0.002 - actor/ppo_kl:0.000 - actor/grad_norm:2.060 - actor/lr(1e-4):0.010 - critic/score/mean:0.000 - critic/score/max:0.000 - critic/score/min:0.000 - critic/rewards/mean:0.000 - critic/rewards/max:0.000 - critic/rewards/min:0.000 - critic/advantages/mean:0.000 - critic/advantages/max:2.702 - critic/advantages/min:-2.616 - critic/returns/mean:0.000 - critic/returns/max:0.000 - critic/returns/min:0.000 - critic/values/mean:-2.280 - critic/values/max:11.000 - critic/values/min:-16.000 - response_length/mean:232.242 - response_length/max:256.000 - response_length/min:91.000 - prompt_length/mean:102.398 - prompt_length/max:185.000 - prompt_length/min:70.000
118
+
119
+ Checkout :ref:`algo-baseline-page` for full training and validation logs for reference.
120
+
121
+ The checkpoint is saved at the following dir by default: ``checkpoints/${trainer.project_name}/${trainer.experiment_name}``
122
+
123
+ To enable ``wandb`` for experiment tracking, set the following configs:
124
+
125
+ .. code-block:: bash
126
+
127
+ trainer.logger=['console','wandb'] \
128
+ trainer.project_name=$YOUR_PROJECT_NAME \
129
+ trainer.experiment_name=$YOUR_RUN_NAME \
130
+
131
+ If you encounter out of memory issues with HBM less than 32GB, enable the following configs would help:
132
+
133
+ .. code-block:: bash
134
+
135
+ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \
136
+ critic.ppo_micro_batch_size_per_gpu=1 \
137
+
138
+ For the full set of configs, please refer to :ref:`config-explain-page` for detailed explanation and performance tuning.
139
+
140
+
141
+ .. [1] The original paper (https://arxiv.org/pdf/2110.14168) mainly focuses on training a verifier (a reward model) to solve math problems via Best-of-N sampling. In this example, we train an RL agent using a rule-based reward model.
142
+ .. [2] More training script examples for FSDP and Megatron-LM backend are stored in `examples/ppo_trainer <https://github.com/volcengine/verl/tree/main/examples/ppo_trainer>`_ directory.
frameworks/agentgym_rl/docs/workers/fsdp_workers.rst ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ PyTorch FSDP Backend
2
+ ======================
3
+
4
+ We support PyTorch FSDP Backend by implementing various workers for
5
+ actor, critic, reference, rollout and reward models. We also implement
6
+ the ``FSDPVLLMShardingManager`` that reshard weight between FSDP and
7
+ vLLM in `fsdp_vllm.py <https://github.com/volcengine/verl/blob/main/verl/workers/sharding_manager/fsdp_vllm.py>`_.
8
+
9
+ **Pros**
10
+
11
+ - Readily support various models.
12
+
13
+ - Users only need to implement the corresponding
14
+ ``dtensor_weight_loader`` for weight synchronization between FSDP
15
+ and vLLM. While for ``hf_weight_loader``, users can directly apply
16
+ any models supported both in HF and vLLM without any code change.
17
+
18
+ - Easy to organize the forward and backward computation for each model.
19
+
20
+ **Cons**
21
+
22
+ - Poor scalability when it comes to large-scale models (e.g. Llama 70B
23
+ and 405B)
24
+ - The resharding overhead between actor and rollout could be larger than
25
+ Megatron-LM backend.
26
+
27
+ Due to the simplicity, we recommend using FSDP backend for algorithm
28
+ research and prototyping.
29
+
30
+ FSDP Workers
31
+ --------------
32
+
33
+ ActorRolloutRefWorker
34
+ ^^^^^^^^^^^^^^^^^^^^^
35
+
36
+ Actor/Rollout HybridEngine
37
+ ''''''''''''''''''''''''''
38
+
39
+ 1. HybridEngine, Actor and Rollout initialization API.
40
+
41
+ .. code:: python
42
+
43
+ @register(dispatch_mode=Dispatch.ONE_TO_ALL)
44
+ def init_model(self):
45
+
46
+ ``ONE_TO_ALL``: when calling the ``init_model`` function from the driver
47
+ process, each worker (on a GPU) will execute the following model
48
+ initialization process.
49
+
50
+ The initialization details of HybridEngine, Actor and Rollout are
51
+ highlighted below:
52
+
53
+ 1. ``DataParallelPPOActor`` implements the simple PPO computation logics
54
+ when the model is built with FSDP, including compute log prob, model
55
+ update.
56
+ 2. ``vLLMRollout`` support generation with vLLM. We modify the vLLM
57
+ Engine and make it executed under SPMD to fit into our
58
+ ``WorkerGroup`` design.
59
+ 3. ``FSDPVLLMShardingManager`` a context manager to perform actual
60
+ resharding between actor and rollout.
61
+
62
+ See `source code <https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py>`_. for more information.
63
+
64
+ 1. Generate sequence and recompute log prob
65
+
66
+ .. code:: python
67
+
68
+ @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
69
+ def generate_sequences(self, prompts: DataProto):
70
+
71
+ - ``Dispatch.DP_COMPUTE_PROTO``: The data will be dispatched and
72
+ collected along the DP dimension
73
+
74
+ - In this function, the rollout model will perform auto-regressive
75
+ generation and the actor model will recompute the old log prob for the
76
+ generated response.
77
+
78
+ 3. Update actor model
79
+
80
+ .. code:: python
81
+
82
+ @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
83
+ def update_actor(self, data: DataProto):
84
+
85
+ - Update the actor model weight using PPO & entropy loss.
86
+
87
+ ReferenceModel
88
+ ''''''''''''''
89
+
90
+ 1. Reference model initialization
91
+
92
+ The reference model is initialized using the same function as the actor
93
+ model without initializing the HybridEngine and Optimizer. Then the
94
+ actor model is also wrapped by the ``DataParallelPPOActor``.
95
+
96
+ 2. Compute reference log prob
97
+
98
+ .. code:: python
99
+
100
+ @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
101
+ def compute_ref_log_prob(self, data: DataProto):
102
+
103
+ - In this function, the reference model will call the compute log prob
104
+ function in ``DataParallelPPOActor`` to compute the reference log
105
+ prob.
106
+
107
+ CriticWorker and RewardWorker
108
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
109
+
110
+ 1. Model initialization
111
+
112
+ Quite similar to reference model. The CriticWorker will perform
113
+ additional initialization for the Optimizer.
114
+
115
+ 2. Compute Values for CriticWorker
116
+
117
+ .. code:: python
118
+
119
+ @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
120
+ def compute_values(self, data: DataProto):
121
+
122
+ 3. Update Critic
123
+
124
+ .. code:: python
125
+
126
+ @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
127
+ def update_critic(self, data: DataProto):
128
+
129
+ 4. Compute Reward
130
+
131
+ .. code:: python
132
+
133
+ @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
134
+ def compute_rm_score(self, data: DataProto):
135
+
136
+
137
+ HybridShard
138
+ ------------
139
+
140
+ We didn't support FSDP `HybridShard`. To support this, we may need to
141
+ construct a 2D device mesh and test the corresponding
142
+ ``dtensor_weight_loader`` and ``hf_weight_loader`` for each model.
frameworks/agentgym_rl/docs/workers/megatron_workers.rst ADDED
@@ -0,0 +1,200 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Megatron-LM Backend
2
+ =====================
3
+
4
+ We support Megatron Backend by implementing various workers for actor,
5
+ critic, reference, rollout and reward models. We also implement the
6
+ ``3DHybridEngine`` using Megatron-LM and vLLM in `megatron_vllm.py <https://github.com/volcengine/verl/blob/main/verl/workers/sharding_manager/megatron_vllm.py>`_.
7
+
8
+ **Pros**
9
+
10
+ - Support 3D parallelism and sequence parallelism for best scalablility
11
+ and throughput.
12
+ - 3D HybridEngine can significantly reduce peak memory usage and reduce
13
+ weight synchronize overhead between actor and rollout.
14
+
15
+ **Cons**
16
+
17
+ - Users should implement their own models for Megatron-LM
18
+ - Users should implement the corresponding weight_loader to
19
+
20
+ - synchronize the model weight between actor (in Megatron) and rollout
21
+ (in vLLM).
22
+ - load weights from checkpoints to corresponding model in Megatron-LM
23
+
24
+ Megatron Workers
25
+ ----------------
26
+
27
+ MegatronWorker
28
+ ^^^^^^^^^^^^^^
29
+
30
+ ``MegatronWorker`` is the base class of different megatron worker
31
+ classes. In this class, ``get_megatron_global_info`` and
32
+ ``get_megatron_rank_info`` function to retrive the 3D parallel world
33
+ size and rank of each ``Worker`` running on specific GPU. These information
34
+ will be used in transfer protocol for Megatron Backend.
35
+
36
+ The following ``Worker`` class for different models will be utilized to
37
+ construct the ``WorkerGroup`` .
38
+
39
+ We implement various of APIs for each ``Worker`` class decorated by the
40
+ ``@register(dispatch_mode=)`` . These APIs can be called by the ray
41
+ driver process. The data can be correctly collect and dispatch following
42
+ the ``dispatch_mode`` on each function. The supported dispatch_model
43
+ (i.e., transfer protocols) can be found in `decorator.py <https://github.com/volcengine/verl/blob/main/verl/single_controller/base/decorator.py>`_.
44
+
45
+ ActorRolloutRefWorker
46
+ ^^^^^^^^^^^^^^^^^^^^^
47
+
48
+ This class is implemented for Actor/Rollout HybridEngine or for the
49
+ reference model to initialize their model and perform computation.
50
+
51
+ Actor/Rollout HybridEngine
52
+ ''''''''''''''''''''''''''
53
+
54
+ 1. HybridEngine, Actor and Rollout initialization API.
55
+
56
+ .. code:: python
57
+
58
+ @register(dispatch_mode=Dispatch.ONE_TO_ALL)
59
+ def init_model(self):
60
+
61
+ ``ONE_TO_ALL``: when calling the ``init_model`` function from the driver
62
+ process, each worker (on a GPU) will execute the following model
63
+ initialization process.
64
+
65
+ The initialization details of HybridEngine, Actor and Rollout are
66
+ highlighted below:
67
+
68
+ 1. ``AllGatherPPModel`` holds memory buffer for both Actor and Rollout
69
+ and support weight resharding between actor and rollout.
70
+ 2. ``MegatronPPOActor`` implements the simple PPO computation logics
71
+ when the model is built with Megatron, including compute log prob,
72
+ model update.
73
+ 3. ``vLLMRollout`` support generation with vLLM. We modify the vLLM
74
+ Engine and make it executed under SPMD to fit into our
75
+ ``WorkerGroup`` design.
76
+ 4. ``MegatronVLLMShardingManager`` a context manager to perform actual
77
+ resharding between actor and rollout.
78
+
79
+ See `source code <https://github.com/volcengine/verl/blob/main/verl/workers/megatron_workers.py#L63>`_ for more information.
80
+
81
+ .. code:: python
82
+
83
+ # Initialize the 3D HybridEngine
84
+ hybrid_engine = AllGatherPPModel(model_provider=megatron_actor_model_provider)
85
+ # Fetch the model at current rank
86
+ actor_module = hybrid_engine.this_rank_models
87
+ ...
88
+
89
+ # build actor model
90
+ self.actor = MegatronPPOActor(config=self.config.actor,
91
+ model_config=self.actor_model_config,
92
+ megatron_config=megatron_config,
93
+ actor_module=self.actor_module,
94
+ actor_optimizer=self.actor_optimizer,
95
+ actor_optimizer_config=self.actor_optim_config)
96
+
97
+ # build rollout
98
+ # rollout initialization
99
+ rollout = vLLMRollout(actor_module=params,
100
+ config=self.config.rollout,
101
+ tokenizer=self.tokenizer,
102
+ model_hf_config=self.actor_model_config,
103
+ train_tp=mpu.get_tensor_model_parallel_world_size())
104
+ # perform weight resharding between actor and rollout
105
+ sharding_manager = MegatronVLLMShardingManager(module=self.hybrid_engine,
106
+ inference_engine=rollout.inference_engine,
107
+ model_config=self.actor_model_config,
108
+ layer_name_mapping=layer_name_mapping)
109
+ ...
110
+
111
+ 2. Generate sequence and recompute log prob
112
+
113
+ .. code:: python
114
+
115
+ @register(dispatch_mode=Dispatch.MEGATRON_PP_AS_DP_PROTO)
116
+ def generate_sequences(self, prompts: DataProto):
117
+
118
+ - ``Dispatch.MEGATRON_PP_AS_DP_PROTO``: The PP dimension of the actor
119
+ model will be regarded as DP dimension. Then the driver process will
120
+ dispatch and collect the data according to this reorganization. This
121
+ is because, in HybridEngine, the actor weight, which usually applied
122
+ larger 3D parallel sizes, will be gathered along the PP dimension and
123
+ TP dimension. Therefore, the corresponding data should be dispatched
124
+ and collected through the 3D parallel group of the rollout model,
125
+ rather than the actor model. However, the world_size and rank
126
+ information can only be retrived from ``get_megatron_global_info`` and
127
+ ``get_megatron_rank_info``, which records the 3D information for the
128
+ actor model. Moreover, the data resharding inside TP dimension will be
129
+ processed within the HybridEngine.
130
+
131
+ - In this function, the rollout model will perform auto-regressive
132
+ generation and the actor model will recompute the old log prob for the
133
+ generated response.
134
+
135
+ 3. Update actor model
136
+
137
+ .. code:: python
138
+
139
+ @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
140
+ def update_actor(self, data: DataProto):
141
+
142
+ - ``Dispatch.MEGATRON_COMPUTE_PROTO``: User passes the data partitioned
143
+ by DP dimension. The data is dispatched to all tp/pp ranks within the
144
+ same dp group, and ultimately only collects output data from tp=0 and
145
+ the last pp.
146
+ - Update the actor model weight using PPO & entropy loss.
147
+
148
+ ReferenceModel
149
+ ''''''''''''''
150
+
151
+ 1. Reference model initialization
152
+
153
+ The reference model is initialized using the same function as the actor
154
+ model without initializing the HybridEngine and Optimizer. Then the
155
+ actor model is also wrapped by the ``MegatronPPOActor``.
156
+
157
+ 2. Compute reference log prob
158
+
159
+ .. code:: python
160
+
161
+ @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
162
+ def compute_ref_log_prob(self, data: DataProto):
163
+
164
+ - In this function, the reference model will call the compute log prob
165
+ function in ``MegatronPPOActor`` to compute the reference log prob.
166
+
167
+ CriticWorker and RewardWorker
168
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
169
+
170
+ 1. Model initialization
171
+
172
+ Quite similar to reference model. The CriticWorker will perform
173
+ additional initialization for the Optimizer.
174
+
175
+ 2. Compute Values for CriticWorker
176
+
177
+ .. code:: python
178
+
179
+ @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
180
+ def compute_values(self, data: DataProto):
181
+
182
+ 3. Update Critic
183
+
184
+ .. code:: python
185
+
186
+ @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
187
+ def update_critic(self, data: DataProto):
188
+
189
+ 4. Compute Reward
190
+
191
+ .. code:: python
192
+
193
+ @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO)
194
+ def compute_rm_score(self, data: DataProto):
195
+
196
+ Context Parallel
197
+ ----------------
198
+
199
+ This require the developer/contributor to implement the context parallel
200
+ both in Megatron-LM and models.
frameworks/agentgym_rl/docs/workers/ray_trainer.rst ADDED
@@ -0,0 +1,243 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ PPO Ray Trainer
2
+ ===============
3
+
4
+ We implement the RayPPOTrainer, which is a trainer runs on the driver
5
+ process on a single CPU/GPU node (default is CPU).
6
+
7
+ The PPORayTrainer include 3 core functions for data preparation,
8
+ WorkerGroup initialization and PPO training loop.
9
+
10
+ Data Preparation
11
+ ----------------
12
+
13
+ The ``PPORayTrainer``, as a single process, is responsible for loading a
14
+ complete batch of samples (prompts) from the dataset and then dispatch
15
+ to different worker_groups running on different GPUs.
16
+
17
+ To generalize the data loading, we implement the ``RLHFDataset`` class
18
+ to load the preprocessed parquet files, apply chat templates to the
19
+ prompts, add padding, truncate prompts that exceed max prompt length and
20
+ then tokenize.
21
+
22
+ .. code:: python
23
+
24
+ self.train_dataset = RLHFDataset(parquet_files=self.config.data.train_files,
25
+ tokenizer=self.tokenizer,
26
+ prompt_key=self.config.data.prompt_key,
27
+ max_prompt_length=self.config.data.max_prompt_length,
28
+ filter_prompts=True,
29
+ return_raw_chat=self.config.data.get('return_raw_chat', False),
30
+ truncation='error')
31
+
32
+ Then, the dataloader will iterate the dataset under PPO mini batch size.
33
+
34
+ WorkerGroup Initialization
35
+ --------------------------
36
+
37
+ We first introduce a basic implementation of initializing the
38
+ ``WorkerGroup`` of the actor model on a given set of GPUs.
39
+
40
+ .. code:: python
41
+
42
+ # max_colocate_count means the number of WorkerGroups (i.e. processes) in each RayResourcePool
43
+ # For FSDP backend, we recommend using max_colocate_count=1 that merge all WorkerGroups into one.
44
+ # For Megatron backend, we recommend using max_colocate_count>1 that can utilize different WorkerGroup for differnt models
45
+ resource_pool = RayResourcePool(process_on_nodes=[config.trainer.n_gpus_per_node] * config.trainer.nnodes,
46
+ use_gpu=True,
47
+ max_colocate_count=1)
48
+ # define actor rollout cls to be init on remote
49
+ actor_rollout_cls = RayClassWithInitArgs(cls=ActorRolloutWorker)
50
+ # define actor_rollout worker group
51
+ actor_rollout_worker_group = MegatronRayWorkerGroup(resource_pool=resource_pool,
52
+ ray_cls_with_init=actor_rollout_cls,
53
+ default_megatron_kwargs=config.actor_rollout.megatron)
54
+
55
+ Different WorkerGroups, like ``actor_rollout_worker_group`` ,
56
+ ``critic_worker_group`` and ``ref_worker_group`` lies on a separate
57
+ process in the above implementation.
58
+
59
+ The driver process can then call the distributed compute function within
60
+ the ``actor_rollout_worker_group`` and other roles to construct the RL
61
+ training loop.
62
+
63
+ For models colocated in the same set of GPUs, we further provide a
64
+ fine-grain optimization, which merge the ``worker_group`` of different roles
65
+ in the same process. This optimization can save the redundant
66
+ CUDA/distributed context in different processes.
67
+
68
+ .. code:: python
69
+
70
+ # initialize WorkerGroup
71
+ # NOTE: if you want to use a different resource pool for each role, which can support different parallel size,
72
+ # you should not use `create_colocated_worker_cls`. Instead, directly pass different resource pool to different worker groups.
73
+ # See TODO(url) for more information.
74
+ all_wg = {}
75
+ for resource_pool, class_dict in self.resource_pool_to_cls.items():
76
+ worker_dict_cls = create_colocated_worker_cls(class_dict=class_dict)
77
+ wg_dict = self.ray_worker_group_cls(resource_pool=resource_pool, ray_cls_with_init=worker_dict_cls)
78
+ spawn_wg = wg_dict.spawn(prefix_set=class_dict.keys())
79
+ all_wg.update(spawn_wg)
80
+
81
+ if self.use_critic:
82
+ self.critic_wg = all_wg['critic']
83
+ self.critic_wg.init_model()
84
+
85
+ if self.use_reference_policy:
86
+ self.ref_policy_wg = all_wg['ref']
87
+ self.ref_policy_wg.init_model()
88
+
89
+ if self.use_rm:
90
+ self.rm_wg = all_wg['rm']
91
+ self.rm_wg.init_model()
92
+
93
+ # we should create rollout at the end so that vllm can have a better estimation of kv cache memory
94
+ self.actor_rollout_wg = all_wg['actor_rollout']
95
+ self.actor_rollout_wg.init_model()
96
+
97
+ .. note:: For megatron backend, if we merge the ``worker_groups`` into the same processes, all the roles will utilize the same 3D parallel size. To optimize this, we may need to maintain several 3D process groups for each role in the same distributed context. If you want to use different 3D parallel size for different roles, please follow the similar architecture of the first code block to initialize each role's ``worker_group``
98
+
99
+
100
+ PPO Training Loop
101
+ -----------------
102
+
103
+ We implement the PPO training loop by calling the functions in
104
+ worker_group of each role. The input and output data of each function is
105
+ a ``DataProto`` object implemented in `protocol.py <https://github.com/volcengine/verl/blob/main/verl/protocol.py>`_. In the training
106
+ loop, trainer will dispatch/collect the data to/from different GPUs
107
+ following the transfer protocols wrapped in the workers' functions. The
108
+ computation of PPO micro batches is processed in ``update_actor`` and
109
+ ``update_critic`` functions.
110
+
111
+ To extend to other RLHF algorithms, such as DPO, GRPO, please refer to
112
+ :doc:`../advance/dpo_extension`.
113
+
114
+ .. code:: python
115
+
116
+ def fit(self):
117
+ """
118
+ The training loop of PPO.
119
+ The driver process only need to call the compute functions of the worker group through RPC to construct the PPO dataflow.
120
+ The light-weight advantage computation is done on the driver process.
121
+ """
122
+ from verl.utils.tracking import Tracking
123
+ from omegaconf import OmegaConf
124
+
125
+ logger = Tracking(project_name=self.config.trainer.project_name,
126
+ experiment_name=self.config.trainer.experiment_name,
127
+ default_backend=self.config.trainer.logger,
128
+ config=OmegaConf.to_container(self.config, resolve=True))
129
+
130
+ global_steps = 0
131
+
132
+ # perform validation before training
133
+ # currently, we only support validation using the reward_function.
134
+ if self.val_reward_fn is not None:
135
+ val_metrics = self._validate()
136
+ pprint(f'Initial validation metrics: {val_metrics}')
137
+
138
+ for epoch in range(self.config.trainer.total_epochs):
139
+ for batch_dict in self.train_dataloader:
140
+ metrics = {}
141
+
142
+ batch: DataProto = DataProto.from_single_dict(batch_dict)
143
+ # batch = batch.to('cuda')
144
+
145
+ # pop those keys for generation
146
+ gen_batch = batch.pop(batch_keys=['input_ids', 'attention_mask', 'position_ids'])
147
+
148
+ # generate a batch
149
+ with Timer(name='gen', logger=None) as timer:
150
+ gen_batch_output = self.actor_rollout_wg.generate_sequences(gen_batch)
151
+ metrics['timing/gen'] = timer.last
152
+
153
+ batch = batch.union(gen_batch_output)
154
+
155
+ if self.use_reference_policy:
156
+ # compute reference log_prob
157
+ with Timer(name='ref', logger=None) as timer:
158
+ ref_log_prob = self.ref_policy_wg.compute_ref_log_prob(batch)
159
+ batch = batch.union(ref_log_prob)
160
+ metrics['timing/ref'] = timer.last
161
+
162
+ # compute values
163
+ with Timer(name='values', logger=None) as timer:
164
+ values = self.critic_wg.compute_values(batch)
165
+ batch = batch.union(values)
166
+ metrics['timing/values'] = timer.last
167
+
168
+ with Timer(name='adv', logger=None) as timer:
169
+ # compute scores. Support both model and function-based.
170
+ # We first compute the scores using reward model. Then, we call reward_fn to combine
171
+ # the results from reward model and rule-based results.
172
+ if self.use_rm:
173
+ # we first compute reward model score
174
+ reward_tensor = self.rm_wg.compute_rm_score(batch)
175
+ batch = batch.union(reward_tensor)
176
+
177
+ # we combine with rule-based rm
178
+ reward_tensor = self.reward_fn(batch)
179
+ batch.batch['token_level_scores'] = reward_tensor
180
+
181
+ # compute rewards. apply_kl_penalty if available
182
+ batch, kl_metrics = apply_kl_penalty(batch,
183
+ kl_ctrl=self.kl_ctrl,
184
+ kl_penalty=self.config.algorithm.kl_penalty)
185
+ metrics.update(kl_metrics)
186
+
187
+ # compute advantages, executed on the driver process
188
+ batch = compute_advantage(batch,
189
+ self.config.algorithm.gamma,
190
+ self.config.algorithm.lam,
191
+ adv_estimator=self.config.algorithm.adv_estimator)
192
+ metrics['timing/adv'] = timer.last
193
+
194
+ # update critic
195
+ if self.use_critic:
196
+ with Timer(name='update_critic', logger=None) as timer:
197
+ critic_output = self.critic_wg.update_critic(batch)
198
+ metrics['timing/update_critic'] = timer.last
199
+ critic_output_metrics = reduce_metrics(critic_output.meta_info['metrics'])
200
+ metrics.update(critic_output_metrics)
201
+
202
+ # implement critic warmup
203
+ if self.config.trainer.critic_warmup <= global_steps:
204
+ # update actor
205
+ with Timer(name='update_actor', logger=None) as timer:
206
+ actor_output = self.actor_rollout_wg.update_actor(batch)
207
+ metrics['timing/update_actor'] = timer.last
208
+ actor_output_metrics = reduce_metrics(actor_output.meta_info['metrics'])
209
+ metrics.update(actor_output_metrics)
210
+
211
+ # validate
212
+ if self.val_reward_fn is not None and (global_steps + 1) % self.config.trainer.test_freq == 0:
213
+ with Timer(name='testing', logger=None) as timer:
214
+ val_metrics: dict = self._validate()
215
+ val_metrics = {f'val/{key}': val for key, val in val_metrics.items()}
216
+ metrics['timing/testing'] = timer.last
217
+ metrics.update(val_metrics)
218
+
219
+ # collect metrics
220
+ data_metrics = compute_data_metrics(batch=batch)
221
+ metrics.update(data_metrics)
222
+
223
+ # TODO: make a canonical logger that supports various backend
224
+ logger.log(data=metrics, step=global_steps)
225
+
226
+ if self.config.trainer.save_freq > 0 and (global_steps + 1) % self.config.trainer.save_freq == 0:
227
+ actor_local_path = os.path.join(self.config.trainer.default_local_dir, 'actor',
228
+ f'global_step_{global_steps}')
229
+ actor_remote_path = os.path.join(self.config.trainer.default_hdfs_dir, 'actor')
230
+ self.actor_rollout_wg.save_checkpoint(actor_local_path, actor_remote_path)
231
+
232
+ if self.use_critic:
233
+ critic_local_path = os.path.join(self.config.trainer.default_local_dir, 'critic',
234
+ f'global_step_{global_steps}')
235
+ critic_remote_path = os.path.join(self.config.trainer.default_hdfs_dir, 'critic')
236
+ self.critic_wg.save_checkpoint(critic_local_path, critic_remote_path)
237
+
238
+ global_steps += 1
239
+
240
+ # perform validation after training
241
+ if self.val_reward_fn is not None:
242
+ val_metrics = self._validate()
243
+ pprint(f'Final validation metrics: {val_metrics}')
frameworks/agentgym_rl/pyproject.toml ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # -------------------------------
2
+ # build-system
3
+ # -------------------------------
4
+ [build-system]
5
+ requires = [
6
+ "setuptools>=61.0",
7
+ "wheel"
8
+ ]
9
+ build-backend = "setuptools.build_meta"
10
+
11
+ # -------------------------------
12
+ # project (PEP 621 metadata)
13
+ # -------------------------------
14
+ [project]
15
+ name = "verl"
16
+ # We'll mark the version as "dynamic" because it's read from the file "verl/version/version"
17
+ # (PEP 621 calls this "dynamic version").
18
+ # The actual version is specified in the [tool.setuptools.dynamic] section below.
19
+ dynamic = ["version"]
20
+
21
+ description = "verl: Volcano Engine Reinforcement Learning for LLM"
22
+ license = {file = "LICENSE"} # or "Apache-2.0", if you prefer an SPDX identifier
23
+ readme = {file = "README.md", content-type = "text/markdown"}
24
+ requires-python = ">=3.8"
25
+
26
+ authors = [
27
+ { name = "Bytedance - Seed - MLSys", email = "zhangchi.usc1992@bytedance.com" },
28
+ { name = "Bytedance - Seed - MLSys", email = "gmsheng@connect.hku.hk" },
29
+ ]
30
+
31
+ # Dependencies corresponding to install_requires in setup.py
32
+ dependencies = [
33
+ "accelerate",
34
+ "codetiming",
35
+ "datasets",
36
+ "dill",
37
+ "hydra-core",
38
+ "numpy",
39
+ "pandas",
40
+ "peft",
41
+ "pyarrow>=15.0.0",
42
+ "pybind11",
43
+ "pylatexenc",
44
+ "ray>=2.10",
45
+ "tensordict<0.6",
46
+ "transformers",
47
+ "vllm<=0.6.3",
48
+ 'wandb',
49
+ 'torch==2.4.0',
50
+ ]
51
+
52
+ # Optional dependencies (extras_require in setup.py)
53
+ [project.optional-dependencies]
54
+ test = [
55
+ "pytest", "yapf", "py-spy",
56
+ ]
57
+ prime = ["pyext"]
58
+ gpu = ["liger-kernel", "flash-attn"]
59
+
60
+ # URLs
61
+ [project.urls]
62
+ Homepage = "https://github.com/volcengine/verl"
63
+
64
+ # -------------------------------
65
+ # tool.setuptools - Additional config
66
+ # -------------------------------
67
+ [tool.setuptools]
68
+ # True means `setuptools` will attempt to include all relevant files in package_data automatically.
69
+ # This corresponds to `include_package_data=True` in setup.py.
70
+ include-package-data = true
71
+
72
+ # We read the version from a file in 'verl/version/version'
73
+ [tool.setuptools.dynamic]
74
+ version = {file = "verl/version/version"}
75
+
76
+ # If you need to mimic `package_dir={'': '.'}`:
77
+ [tool.setuptools.package-dir]
78
+ "" = "."
79
+
80
+ # If you need to include specific non-Python data (like YAML files or version file):
81
+ # This is the rough equivalent of package_data={'': ['version/*'], 'verl': ['trainer/config/*.yaml']}
82
+ [tool.setuptools.package-data]
83
+ verl = [
84
+ "version/*",
85
+ "trainer/config/*.yaml"
86
+ ]
frameworks/agentgym_rl/requirements.txt ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # requirements.txt records the full set of dependencies for development
2
+ accelerate
3
+ codetiming
4
+ datasets
5
+ dill
6
+ flash-attn
7
+ hydra-core
8
+ liger-kernel
9
+ numpy
10
+ pandas
11
+ peft
12
+ pyarrow>=15.0.0
13
+ pybind11
14
+ pylatexenc
15
+ ray
16
+ tensordict<0.6
17
+ transformers
18
+ vllm<=0.6.3
19
+ wandb
frameworks/agentgym_rl/scripts/format.sh ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ #!/bin/bash
2
+ pip3 install --upgrade yapf
3
+ python3 -m yapf -ir -vv --style ./.style.yapf verl tests single_controller examples
frameworks/agentgym_rl/scripts/model_merger.py ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 Bytedance Ltd. and/or its affiliates
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ from typing import List, Tuple, Dict
16
+ import re
17
+ import os
18
+ import torch
19
+ import argparse
20
+ from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification
21
+ from concurrent.futures import ThreadPoolExecutor
22
+ from torch.distributed._tensor import DTensor, Shard, Placement
23
+
24
+
25
+ def merge_by_placement(tensors: List[torch.Tensor], placement: Placement):
26
+ if placement.is_replicate():
27
+ return tensors[0]
28
+ elif placement.is_partial():
29
+ raise NotImplementedError("Partial placement is not supported yet")
30
+ elif placement.is_shard():
31
+ return torch.cat(tensors, dim=placement.dim).contiguous()
32
+ else:
33
+ raise ValueError(f"Unsupported placement: {placement}")
34
+
35
+
36
+ if __name__ == '__main__':
37
+ parser = argparse.ArgumentParser()
38
+ parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model")
39
+ parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload")
40
+ args = parser.parse_args()
41
+
42
+ assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface"
43
+ local_dir = args.local_dir
44
+
45
+ # copy rank zero to find the shape of (dp, fsdp)
46
+ rank = 0
47
+ world_size = 0
48
+ for filename in os.listdir(local_dir):
49
+ match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename)
50
+ if match:
51
+ world_size = match.group(1)
52
+ break
53
+ assert world_size, "No model file with the proper format"
54
+
55
+ state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu')
56
+ pivot_key = sorted(list(state_dict.keys()))[0]
57
+ weight = state_dict[pivot_key]
58
+ assert isinstance(weight, torch.distributed._tensor.DTensor)
59
+ # get sharding info
60
+ device_mesh = weight.device_mesh
61
+ mesh = device_mesh.mesh
62
+ mesh_dim_names = device_mesh.mesh_dim_names
63
+
64
+ print(f'Got device mesh {mesh}, mesh_dim_names {mesh_dim_names}')
65
+
66
+ assert mesh_dim_names in (
67
+ ('fsdp',),
68
+ ), f'Unsupported mesh_dim_names {mesh_dim_names}'
69
+
70
+ if 'tp' in mesh_dim_names:
71
+ # fsdp * tp
72
+ total_shards = mesh.shape[-1] * mesh.shape[-2]
73
+ mesh_shape = (mesh.shape[-2], mesh.shape[-1])
74
+ else:
75
+ # fsdp
76
+ total_shards = mesh.shape[-1]
77
+ mesh_shape = (mesh.shape[-1],)
78
+
79
+ print(f'Processing model shards with {total_shards} {mesh_shape} in total')
80
+
81
+ model_state_dict_lst = []
82
+ model_state_dict_lst.append(state_dict)
83
+ model_state_dict_lst.extend([""] * (total_shards - 1))
84
+
85
+ def process_one_shard(rank):
86
+ model_path = os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt')
87
+ state_dict = torch.load(model_path, map_location='cpu', weights_only=False)
88
+ model_state_dict_lst[rank] = state_dict
89
+ return state_dict
90
+
91
+ with ThreadPoolExecutor(max_workers=min(32, os.cpu_count())) as executor:
92
+ for rank in range(1, total_shards):
93
+ executor.submit(process_one_shard, rank)
94
+ state_dict = {}
95
+ param_placements: Dict[str, List[Placement]] = {}
96
+ keys = set(model_state_dict_lst[0].keys())
97
+ for key in keys:
98
+ state_dict[key] = []
99
+ for model_state_dict in model_state_dict_lst:
100
+ try:
101
+ tensor = model_state_dict.pop(key)
102
+ except:
103
+ print("-"*30)
104
+ print(model_state_dict)
105
+ if isinstance(tensor, DTensor):
106
+ state_dict[key].append(tensor._local_tensor.bfloat16())
107
+ placements = tuple(tensor.placements)
108
+ # replicated placement at dp dimension can be discarded
109
+ if mesh_dim_names[0] == 'dp':
110
+ placements = placements[1:]
111
+ if key not in param_placements:
112
+ param_placements[key] = placements
113
+ else:
114
+ assert param_placements[key] == placements
115
+ else:
116
+ state_dict[key] = tensor.bfloat16()
117
+
118
+ del model_state_dict_lst
119
+
120
+ for key in sorted(state_dict):
121
+ if not isinstance(state_dict[key], list):
122
+ print(f"No need to merge key {key}")
123
+ continue
124
+ # merge shards
125
+ placements: Tuple[Shard] = param_placements[key]
126
+ if len(mesh_shape) == 1:
127
+ # 1-D list, FSDP without TP
128
+ assert len(placements) == 1
129
+ shards = state_dict[key]
130
+ state_dict[key] = merge_by_placement(shards, placements[0])
131
+ else:
132
+ # 2-D list, FSDP + TP
133
+ raise NotImplementedError("FSDP + TP is not supported yet")
134
+
135
+ print('Writing to local disk')
136
+ hf_path = os.path.join(local_dir, 'huggingface')
137
+ config = AutoConfig.from_pretrained(hf_path)
138
+
139
+ if 'ForTokenClassification' in config.architectures[0]:
140
+ auto_model = AutoModelForTokenClassification
141
+ elif 'ForCausalLM' in config.architectures[0]:
142
+ auto_model = AutoModelForCausalLM
143
+ else:
144
+ raise NotImplementedError(f'Unknown architecture {config["architectures"]}')
145
+
146
+ with torch.device('meta'):
147
+ model = auto_model.from_config(config, torch_dtype=torch.bfloat16)
148
+ model.to_empty(device='cpu')
149
+
150
+ print(f'Saving model to {hf_path}')
151
+ model.save_pretrained(hf_path, state_dict=state_dict)
152
+ del state_dict
153
+ del model
154
+ if args.hf_upload_path:
155
+ # Push to hugging face
156
+ from huggingface_hub import HfApi
157
+ api = HfApi()
158
+ api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True)
159
+ api.upload_folder(
160
+ folder_path=hf_path,
161
+ repo_id=args.hf_upload_path,
162
+ repo_type="model"
163
+ )
164
+
165
+
166
+
167
+
168
+
169
+
170
+
frameworks/agentgym_rl/scripts/multiple_model_merger.py ADDED
@@ -0,0 +1,178 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 Bytedance Ltd. and/or its affiliates
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ from typing import List, Tuple, Dict
16
+ import re
17
+ import os
18
+ import torch
19
+ import argparse
20
+ from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification
21
+ from concurrent.futures import ThreadPoolExecutor
22
+ from torch.distributed._tensor import DTensor, Shard, Placement
23
+
24
+ import shutil
25
+
26
+
27
+ def merge_by_placement(tensors: List[torch.Tensor], placement: Placement):
28
+ if placement.is_replicate():
29
+ return tensors[0]
30
+ elif placement.is_partial():
31
+ raise NotImplementedError("Partial placement is not supported yet")
32
+ elif placement.is_shard():
33
+ return torch.cat(tensors, dim=placement.dim).contiguous()
34
+ else:
35
+ raise ValueError(f"Unsupported placement: {placement}")
36
+
37
+
38
+ if __name__ == '__main__':
39
+ parser = argparse.ArgumentParser()
40
+ parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model")
41
+ parser.add_argument('--save_dir', default=None, type = str, help="The path to save your model")
42
+ parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload")
43
+ args = parser.parse_args()
44
+
45
+ assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface"
46
+ local_dir = args.local_dir
47
+ save_dir = args.save_dir
48
+
49
+ # copy rank zero to find the shape of (dp, fsdp)
50
+ rank = 0
51
+ world_size = 0
52
+ for filename in os.listdir(local_dir):
53
+ match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename)
54
+ if match:
55
+ world_size = match.group(1)
56
+ break
57
+ assert world_size, "No model file with the proper format"
58
+
59
+ state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu')
60
+ pivot_key = sorted(list(state_dict.keys()))[0]
61
+ weight = state_dict[pivot_key]
62
+ assert isinstance(weight, torch.distributed._tensor.DTensor)
63
+ # get sharding info
64
+ device_mesh = weight.device_mesh
65
+ mesh = device_mesh.mesh
66
+ mesh_dim_names = device_mesh.mesh_dim_names
67
+
68
+ print(f'Got device mesh {mesh}, mesh_dim_names {mesh_dim_names}')
69
+
70
+ assert mesh_dim_names in (
71
+ ('fsdp',),
72
+ ), f'Unsupported mesh_dim_names {mesh_dim_names}'
73
+
74
+ if 'tp' in mesh_dim_names:
75
+ # fsdp * tp
76
+ total_shards = mesh.shape[-1] * mesh.shape[-2]
77
+ mesh_shape = (mesh.shape[-2], mesh.shape[-1])
78
+ else:
79
+ # fsdp
80
+ total_shards = mesh.shape[-1]
81
+ mesh_shape = (mesh.shape[-1],)
82
+
83
+ print(f'Processing model shards with {total_shards} {mesh_shape} in total')
84
+
85
+ model_state_dict_lst = []
86
+ model_state_dict_lst.append(state_dict)
87
+ model_state_dict_lst.extend([""] * (total_shards - 1))
88
+
89
+ def process_one_shard(rank):
90
+ model_path = os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt')
91
+ state_dict = torch.load(model_path, map_location='cpu', weights_only=False)
92
+ model_state_dict_lst[rank] = state_dict
93
+ return state_dict
94
+
95
+ with ThreadPoolExecutor(max_workers=min(32, os.cpu_count())) as executor:
96
+ for rank in range(1, total_shards):
97
+ executor.submit(process_one_shard, rank)
98
+ state_dict = {}
99
+ param_placements: Dict[str, List[Placement]] = {}
100
+ keys = set(model_state_dict_lst[0].keys())
101
+ for key in keys:
102
+ state_dict[key] = []
103
+ for model_state_dict in model_state_dict_lst:
104
+ try:
105
+ tensor = model_state_dict.pop(key)
106
+ except:
107
+ print("-"*30)
108
+ print(model_state_dict)
109
+ if isinstance(tensor, DTensor):
110
+ state_dict[key].append(tensor._local_tensor.bfloat16())
111
+ placements = tuple(tensor.placements)
112
+ # replicated placement at dp dimension can be discarded
113
+ if mesh_dim_names[0] == 'dp':
114
+ placements = placements[1:]
115
+ if key not in param_placements:
116
+ param_placements[key] = placements
117
+ else:
118
+ assert param_placements[key] == placements
119
+ else:
120
+ state_dict[key] = tensor.bfloat16()
121
+
122
+ del model_state_dict_lst
123
+
124
+ for key in sorted(state_dict):
125
+ if not isinstance(state_dict[key], list):
126
+ print(f"No need to merge key {key}")
127
+ continue
128
+ # merge shards
129
+ placements: Tuple[Shard] = param_placements[key]
130
+ if len(mesh_shape) == 1:
131
+ # 1-D list, FSDP without TP
132
+ assert len(placements) == 1
133
+ shards = state_dict[key]
134
+ state_dict[key] = merge_by_placement(shards, placements[0])
135
+ else:
136
+ # 2-D list, FSDP + TP
137
+ raise NotImplementedError("FSDP + TP is not supported yet")
138
+
139
+ print('Writing to local disk')
140
+ if save_dir and os.path.abspath(save_dir) != os.path.abspath(local_dir):
141
+ hf_path = os.path.join(save_dir, 'huggingface')
142
+ shutil.copytree(os.path.join(local_dir, 'huggingface'), hf_path)
143
+ else:
144
+ hf_path = os.path.join(local_dir, 'huggingface')
145
+ config = AutoConfig.from_pretrained(hf_path)
146
+
147
+ if 'ForTokenClassification' in config.architectures[0]:
148
+ auto_model = AutoModelForTokenClassification
149
+ elif 'ForCausalLM' in config.architectures[0]:
150
+ auto_model = AutoModelForCausalLM
151
+ else:
152
+ raise NotImplementedError(f'Unknown architecture {config["architectures"]}')
153
+
154
+ with torch.device('meta'):
155
+ model = auto_model.from_config(config, torch_dtype=torch.bfloat16)
156
+ model.to_empty(device='cpu')
157
+
158
+ print(f'Saving model to {hf_path}')
159
+ model.save_pretrained(hf_path, state_dict=state_dict)
160
+ del state_dict
161
+ del model
162
+ if args.hf_upload_path:
163
+ # Push to hugging face
164
+ from huggingface_hub import HfApi
165
+ api = HfApi()
166
+ api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True)
167
+ api.upload_folder(
168
+ folder_path=hf_path,
169
+ repo_id=args.hf_upload_path,
170
+ repo_type="model"
171
+ )
172
+
173
+
174
+
175
+
176
+
177
+
178
+
frameworks/agentgym_rl/scripts/single_model_merger.py ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 Bytedance Ltd. and/or its affiliates
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ import re
16
+ import os
17
+ import torch
18
+ import argparse
19
+ from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification
20
+
21
+ import shutil
22
+
23
+
24
+ if __name__ == '__main__':
25
+ parser = argparse.ArgumentParser()
26
+ parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model")
27
+ parser.add_argument('--save_dir', default=None, type = str, help="The path to save your model")
28
+ parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload")
29
+ args = parser.parse_args()
30
+
31
+ assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface"
32
+ local_dir = args.local_dir
33
+ save_dir = args.save_dir
34
+
35
+ # copy rank zero to find the shape of (dp, fsdp)
36
+ rank = 0
37
+ world_size = 0
38
+ for filename in os.listdir(local_dir):
39
+ match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename)
40
+ if match:
41
+ world_size = match.group(1)
42
+ break
43
+ assert world_size, "No model file with the proper format"
44
+
45
+ state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu')
46
+
47
+ print('Writing to local disk')
48
+ if save_dir and os.path.abspath(save_dir) != os.path.abspath(local_dir):
49
+ hf_path = os.path.join(save_dir, 'huggingface')
50
+ shutil.copytree(os.path.join(local_dir, 'huggingface'), hf_path)
51
+ else:
52
+ hf_path = os.path.join(local_dir, 'huggingface')
53
+ config = AutoConfig.from_pretrained(hf_path)
54
+
55
+ if 'ForTokenClassification' in config.architectures[0]:
56
+ auto_model = AutoModelForTokenClassification
57
+ elif 'ForCausalLM' in config.architectures[0]:
58
+ auto_model = AutoModelForCausalLM
59
+ else:
60
+ raise NotImplementedError(f'Unknown architecture {config["architectures"]}')
61
+
62
+ with torch.device('meta'):
63
+ model = auto_model.from_config(config, torch_dtype=torch.bfloat16)
64
+ model.to_empty(device='cpu')
65
+
66
+ print(f'Saving model to {hf_path}')
67
+ model.save_pretrained(hf_path, state_dict=state_dict)
68
+ del state_dict
69
+ del model
70
+ if args.hf_upload_path:
71
+ # Push to hugging face
72
+ from huggingface_hub import HfApi
73
+ api = HfApi()
74
+ api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True)
75
+ api.upload_folder(
76
+ folder_path=hf_path,
77
+ repo_id=args.hf_upload_path,
78
+ repo_type="model"
79
+ )
80
+
81
+
82
+
83
+
84
+
85
+
86
+
frameworks/agentgym_rl/setup.py ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 Bytedance Ltd. and/or its affiliates
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ # setup.py is the fallback installation script when pyproject.toml does not work
16
+ from setuptools import setup, find_packages
17
+ import os
18
+
19
+ version_folder = os.path.dirname(os.path.join(os.path.abspath(__file__)))
20
+
21
+ with open(os.path.join(version_folder, 'verl/version/version')) as f:
22
+ __version__ = f.read().strip()
23
+
24
+ install_requires = [
25
+ 'accelerate',
26
+ 'codetiming',
27
+ 'datasets',
28
+ 'dill',
29
+ 'hydra-core',
30
+ 'numpy',
31
+ 'pandas',
32
+ 'peft',
33
+ 'pyarrow>=15.0.0',
34
+ 'pybind11',
35
+ 'pylatexenc',
36
+ 'ray>=2.10',
37
+ 'tensordict<0.6',
38
+ 'transformers',
39
+ 'vllm<=0.6.3',
40
+ 'wandb',
41
+ ]
42
+
43
+ TEST_REQUIRES = ['pytest', 'yapf', 'py-spy']
44
+ PRIME_REQUIRES = ['pyext']
45
+ GPU_REQUIRES = ['liger-kernel', 'flash-attn']
46
+
47
+ extras_require = {
48
+ 'test': TEST_REQUIRES,
49
+ 'prime': PRIME_REQUIRES,
50
+ 'gpu': GPU_REQUIRES,
51
+ }
52
+
53
+ from pathlib import Path
54
+ this_directory = Path(__file__).parent
55
+ long_description = (this_directory / "VERL_README.md").read_text()
56
+
57
+ setup(
58
+ name='verl',
59
+ version=__version__,
60
+ package_dir={'': '.'},
61
+ packages=find_packages(where='.'),
62
+ url='https://github.com/volcengine/verl',
63
+ license='Apache 2.0',
64
+ author='Bytedance - Seed - MLSys',
65
+ author_email='zhangchi.usc1992@bytedance.com, gmsheng@connect.hku.hk',
66
+ description='verl: Volcano Engine Reinforcement Learning for LLM',
67
+ install_requires=install_requires,
68
+ extras_require=extras_require,
69
+ package_data={'': ['version/*'],
70
+ 'verl': ['trainer/config/*.yaml'],},
71
+ include_package_data=True,
72
+ long_description=long_description,
73
+ long_description_content_type='text/markdown'
74
+ )
frameworks/agentgym_rl/verl/__init__.py ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 Bytedance Ltd. and/or its affiliates
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ import os
16
+
17
+ version_folder = os.path.dirname(os.path.join(os.path.abspath(__file__)))
18
+
19
+ with open(os.path.join(version_folder, 'version/version')) as f:
20
+ __version__ = f.read().strip()
21
+
22
+ from .protocol import DataProto
23
+
24
+ from .utils.logging_utils import set_basic_config
25
+ import logging
26
+
27
+ set_basic_config(level=logging.WARNING)
frameworks/agentgym_rl/verl/agent_trainer/__init__.py ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 Bytedance Ltd. and/or its affiliates
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
frameworks/agentgym_rl/verl/agent_trainer/config/evaluation.yaml ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ data:
2
+ path: null
3
+ reward_model_key: rewards
frameworks/agentgym_rl/verl/agent_trainer/config/generation.yaml ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ trainer:
2
+ nnodes: 1
3
+ n_gpus_per_node: 8
4
+
5
+ data:
6
+ path: null
7
+ max_prompt_length: 1024
8
+ max_response_length: 15360
9
+ prompt_key: item_id
10
+ n_samples: 1
11
+ output_path: null
12
+ batch_size: 16
13
+
14
+ agentgym:
15
+ task_name: textcraft
16
+ env_addr: 'http://localhost:5000'
17
+ max_retries: 10
18
+ max_rounds: 10
19
+ timeout: 300
20
+
21
+ model:
22
+ path: null
23
+ external_lib: null
24
+ rollout:
25
+ name: vllm
26
+ temperature: 1.0
27
+ top_k: -1 # 0 for hf rollout, -1 for vllm rollout
28
+ top_p: 1
29
+ prompt_length: ${data.max_prompt_length} # not use for opensource
30
+ response_length: ${data.max_response_length}
31
+ # for vllm rollout
32
+ dtype: bfloat16 # should align with FSDP
33
+ gpu_memory_utilization: 0.5
34
+ ignore_eos: False
35
+ enforce_eager: True
36
+ free_cache_engine: True
37
+ load_format: dummy_dtensor
38
+ tensor_model_parallel_size: 2
39
+ max_num_batched_tokens: 8192
40
+ max_model_len: 32768
41
+ max_num_seqs: 1024
42
+ log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu
43
+ log_prob_micro_batch_size_per_gpu: 1
44
+ log_prob_use_dynamic_bsz: ${actor.use_dynamic_bsz}
45
+ log_prob_max_token_len_per_gpu: ${actor.ppo_max_token_len_per_gpu}
46
+ max_tokens: 1024
47
+ disable_log_stats: True
48
+ enable_chunked_prefill: True # may get higher throughput when set to True. When activated, Please increase max_num_batched_tokens or decrease max_model_len.
49
+ # for hf rollout
50
+ do_sample: True
51
+ # number of responses (i.e. num sample times)
52
+ n: 1 # > 1 for grpo
53
+ send_interval: 1
54
+ rollout_log_dir: null
55
+
56
+ actor:
57
+ strategy: fsdp # This is for backward-compatibility
58
+ ppo_mini_batch_size: 16
59
+ ppo_micro_batch_size: null # will be deprecated, use ppo_micro_batch_size_per_gpu
60
+ ppo_micro_batch_size_per_gpu: 1
61
+ use_dynamic_bsz: False
62
+ ppo_max_token_len_per_gpu: 16384 # n * ${data.max_prompt_length} + ${data.max_response_length}
63
+ grad_clip: 1.0
64
+ clip_ratio: 0.2
65
+ entropy_coeff: 0.001
66
+ use_kl_loss: null # True for GRPO
67
+ kl_loss_coef: 0.001 # for grpo
68
+ kl_loss_type: low_var_kl # for grpo
69
+ ppo_epochs: 1
70
+ shuffle: False
71
+ ulysses_sequence_parallel_size: 1 # sp size
72
+ optim:
73
+ lr: 1e-6
74
+ lr_warmup_steps_ratio: 0. # the total steps will be injected during runtime
75
+ min_lr_ratio: null # only useful for warmup with cosine
76
+ warmup_style: constant # select from constant/cosine
77
+ total_training_steps: -1 # must be override by program
78
+ fsdp_config:
79
+ wrap_policy:
80
+ # transformer_layer_cls_to_wrap: None
81
+ min_num_params: 0
82
+ param_offload: False
83
+ grad_offload: False
84
+ optimizer_offload: False
85
+ fsdp_size: -1