diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..a8673d825c40ac1f33b895e3396c7e0698548236 --- /dev/null +++ b/README.md @@ -0,0 +1,130 @@ +--- +license: apache-2.0 +tags: [distillation, on-policy-distillation, rose, opd, verl, trinity, agentgym, rlve, sciworld, dapo] +--- + +# ROSE / OPD 蒸馏实验的完整实现与训练框架 + +三个 domain(RLVE 单轮推理、SciWorld 多轮 agent、DAPO-Math)上对比三种蒸馏方式 +(teacher-SFT / on-policy distillation / online ROSE)的全部代码。 + +配套的 checkpoint 与数据在: +[OPD-dq](https://huggingface.co/SeanWang0027/OPD-dq) · +[ROSE-dq](https://huggingface.co/SeanWang0027/ROSE-dq) · +[SciWorld-dq](https://huggingface.co/SeanWang0027/SciWorld-dq) · +[data-dq](https://huggingface.co/SeanWang0027/data-dq) + +## 目录 + +``` +frameworks/ + verl_rose/ ROSE 全部线用的 verl(含 recipe/rose/rose_agent_loop.py) + verl_opd/ OPD 全部线用的 verl —— 与上游 thunlp/OPD 的计算路径逐字节等价 + verl_sft/ teacher-SFT 用的 verl + agentgym_rl/ AgentGym-RL —— 本项目里只用于 SciWorld 评测, 未用于训练 + trinity/ Trinity —— SciWorld 的 ROSE/OPD 八条线是它训的 +sciworld/ + impl/ 自写的多轮 ROSE/OPD 实现(不依赖上面任何框架) + chains/ 编排脚本 + trinity_configs/ Trinity 侧的 workflow 与 yaml + agentgym_eval/ AgentGym 口径的 200 题评测 +launchers/ + rlve/ dapo/ olmo/ +eval/ AIME25 / HMMT25 / RLVE 评测 +docs/ ENV_NOTES.md —— 跨家族 ROSE 的环境硬约束 +tools/ 归档上传用的脚本 +``` + +## 三种方法 + +**teacher-SFT** 教师采轨迹, 学生离线模仿。 + +**OPD**(on-policy distillation)教师对**学生自己采出来的 token** 打分, +`advantages = teacher_logprobs - student_logprobs`, `adv_estimator=token_reward_direct`, +不做归一化。 + +**ROSE** 学生写 K 个 token 的前缀(mask=0), 教师从那里接着写 t 个 token, +CE 只打在教师 token 上(mask=1)。学生学的是"我走到这个状态之后, 老师会怎么继续"。 +`recipe/rose/rose_agent_loop.py` 里有完整说明。 + +## SciWorld 有两套并行实现 + +| 实现 | 位置 | 产物 | +|---|---|---| +| **Trinity** | `frameworks/trinity` + `sciworld/trinity_configs` | `SCIENCEWORLD_{OPD,ROSE}/` 八条线 | +| **自写多轮** | `sciworld/impl/online_{rose,opd}_sciworld.py` | 老格式 `log.jsonl + step_N/`(checkpoint 已清, 脚本保留) | + +`sciworld/impl/` 那套不依赖 verl 也不依赖 Trinity, 是从头写的: +学生走 N 轮 → 按 turn 边界构造 loss mask → 教师接管后续轮次 → 只在教师 turn 上算 CE。 +多轮的 mask 构造见 `online_rose_sciworld.py`; OPD 侧的 `forward_kl_topk` 见 `online_opd_sciworld.py`。 + +**AgentGym-RL 只用于评测**(`sciworld_agentgym_eval.py`, 200 题 / ≤20 轮 / greedy / nothink), +本项目没有用它训练过。 + +## 结果 + +### RLVE (16k, n=8) + +| line | avg@8 | pass@8 | 解出 | +|---|---|---|---| +| 基线 未训练 | 0.0333 | 0.1111 | 20 | +| official OPD bf16 (140步) | 0.0458 | 0.1444 | 26 | +| OPD bf16 mini16 (140步) | 0.0556 | 0.1722 | 31 | +| **OPD fp32 (105步)** | **0.0694** | 0.1833 | 33 | +| OPD fp32 (132步) | 0.0667 | 0.1833 | 33 | +| **official ROSE (140步)** | **0.0701** | 0.1778 | 32 | +| official ROSE seed1 | 0.0729 | 0.1889 | 34 | +| teacher-SFT n4 bs256 | 0.0556 | 0.1500 | 27 | + +fp32 那条与 official bf16 **只差 dtype**(train/mini/n、lr、resp、更新次数全同)。 +上游 `on_policy_distillation.sh` 默认就是 fp32, bf16 是早期的偏离 —— +lr 1e-6 × advantage 10⁻² 的每步相对更新量低于 bf16 的分辨率, 会被舍入吞掉。 + +### SciWorld (AgentGym 口径, 200 题, ≤20 轮, greedy, nothink) + +| | Score | Success | Pass(>0) | +|---|---|---|---| +| Qwen3-1.7B 基座 | 0.0479 | 0.0000 | 0.1650 | +| OPD lr1e-6 | 0.0421 | — | — | +| OPD lr1e-5 | 0.1931 | — | — | +| teacher-SFT | 0.2077 | 0.0450 | 0.4100 | +| **ROSE** | **0.3127** | — | — | +| Qwen3-32B 教师 | 0.4346 | 0.1100 | 0.9350 | + +ROSE 在 SciWorld 上的优势(+0.105 over SFT, +0.12 over OPD)远大于 RLVE 上的(+0.016 / +0.002)。 + +### DAPO-Math 上的 ROSE 是负结果 + +AIME25 / HMMT25 avg@16, thinking, temp0.7 / top_p0.95 / max_new 31744: + +| | AIME25 | HMMT25 | +|---|---|---| +| Qwen3-32B 教师 | 0.7125 | 0.5042 | +| Qwen3-4B 未训练 | 0.6521 | — | +| ROSE←32B 30步 (4B) | 0.6125 | — | +| Olmo-3-7B 基线 | 0.5917 | 0.4167 | +| ROSE←32B 30步 (Olmo-7B) | 0.5750 | 0.3854 | +| **ROSE←32B 220步 (Olmo-7B)** | **0.5583** | **0.3688** | + +四组独立观测同向, 且 Olmo 从 30 步到 220 步(跑满一个 epoch)继续下降 —— **排除欠训**。 +训练本身也早停滞: ce_loss 从 31 步起斜率 −0.00012, grad_norm 降到 0.18。 + +诊断: `TEACHER_MAX_TOKENS=1024`, 而实测 `ce_target_tokens` 达 **960-1016** +(占预算 94-99%)—— 教师几乎每条都是写到预算耗尽被截断, 从不自己收尾。 +配合学生 16000-18600 token 的推理链, 监督只覆盖 **token 1024-2048, 约 6%**。 +学生学到的是"在思考中段接着往下写", 从没被教过怎么收敛到答案。 +表现是**输出方差变大而能力边界未动**: pass@16 靠多蒙中一题涨(两个 benchmark 各 +1 题, +其中一题是 0/16 → 1/16), 而 avg@16 与 pass@1 都下降, 截断率上升。 + +要检验这个诊断, 应该把 K 提到 4096-8192、teacher 预算给到 2048-4096, +让切点落到推理链中后段。 + +## 跨家族 ROSE(Olmo-3-7B ← Qwen3-32B) + +学生 vocab 100278 / 教师 151669, 完全不同的词表。`rose_agent_loop` 有现成通路 +(`TEACHER_TOKENIZER` 非 null 即启用): 学生前缀 ids → decode 成文本 → +套**教师的** chat template → 教师续写 → 取回 text → 用**学生的** tokenizer 重新 encode。 +实测两边对拼接缝文本的 round-trip 都无损, 教师续写语义连贯且不会重复吐 ``。 + +环境上的硬约束(vLLM 版本被夹死在 0.11.0、NCCL 只能关 NVLS、verl 序列打包硬依赖 +flash_attn 等)见 `docs/ENV_NOTES.md` —— 每一条都对应一次失败的启动。 diff --git a/docs/ENV_NOTES.md b/docs/ENV_NOTES.md new file mode 100644 index 0000000000000000000000000000000000000000..5150c4983724eb12c745b186ae8a49d63631255b --- /dev/null +++ b/docs/ENV_NOTES.md @@ -0,0 +1,98 @@ +# 跨家族 ROSE 的环境约束(Olmo-3-7B ← Qwen3-32B) + +把一个**非 Qwen 家族**的学生接进这套 ROSE 时踩到的全部硬约束。都是实测出来的, +不是从文档抄的 —— 每一条都对应一次失败的启动。 + +## 1. vLLM 版本被夹死在 0.11.0 + +verl 0.7.0.dev 与 vLLM 之间有三处接口漂移,加上 Olmo3 支持,四条同时满足的只有 **0.11.0**: + +| 版本 | Olmo3 | `vllm.lora.models` | `vllm.utils.FlexibleArgumentParser` | transformers v4 | +|---|---|---|---|---| +| 0.10.2 | 不支持 | 有 | 有 | 要 v4 | +| **0.11.0** | **支持** | **有** | **有** | **要 v4** | +| 0.11.1 / 0.11.2 | 支持 | 有 | **没了** | 要 v4 | +| 0.12.0 | 支持 | 有 | **没了** | 要 v4 | +| 0.23.0 | 支持 | **没了** | — | 要 v4 | +| ≥0.24.0 | 支持 | — | — | **强制 v5** | + +- transformers 必须留在 **4.57.6**:verl 0.7 要 `AutoModelForVision2Seq`,v5 里删了; + 而 4.57.6 本身已经认 `Olmo3Config`,所以不需要为 Olmo 升 transformers。 +- `flashinfer-cubin` 要与 `flashinfer-python` **同版本**(装过别的 vllm 会留下不匹配的 + cubin,报 `flashinfer-cubin version (0.6.12) does not match flashinfer version (0.5.2)`)。 + +## 2. NCCL:只关 NVLS,千万别关 P2P + +本机(8×H100,驱动 580.65,cu130)默认起 TP>1 会 `NCCL error: unhandled cuda error`。 +逐项扫下来: + +``` +什么都不设 -> 报错 +NCCL_NVLS_ENABLE=0 -> OK ← 只需要这一个 +NCCL_CUMEM_ENABLE=0 + NVLS=0 -> OK +NCCL_CUMEM_ENABLE=0 + SHM=1 -> 报错 +NCCL_P2P_LEVEL=NVL -> 报错 +``` + +坏的是 NVLink SHARP 组播(NVLS)的初始化。 + +**代价警告**:一开始我是一组一组加着试的,`NCCL_P2P_DISABLE=1` 跟着 +`NCCL_SHM_DISABLE=1` 一起进去了,试出"能跑"就停了没做减法 —— 结果关掉 NVLink, +TP 的 all-reduce 退回 PCIe/主机中转,**吞吐从 2000 tok/s 掉到 124 tok/s(16 倍)**。 +凡是靠"加一组变量"试出来的解,必须回头逐个去掉验证哪个是真正必需的。 + +## 3. flashinfer 需要 ninja + +`FileNotFoundError: 'ninja'` —— flashinfer 要 JIT 编译采样 kernel。 +装 ninja 包不够(pip 装的没有可执行文件在 PATH 上),直接 +`VLLM_USE_FLASHINFER_SAMPLER=0` 绕开最省事。 + +## 4. vLLM 强制 spawn 时,平铺脚本会炸 + +vLLM 0.27 对部分模型强制用 spawn 起 worker,子进程会重新 import 父进程的 `__main__`。 +平铺脚本(逻辑全在模块层)被重新 import 就触发 +`RuntimeError: freeze_support / Safe importing of main module`。 + +`aime_launch.py` 用一个 guard 绕开,**关键是 `runpy.run_path` 的 `run_name` 不能是 +`"__main__"`** —— runpy 在执行期间会把 `sys.modules["__main__"]` 临时换成目标脚本, +spawn 读到的还是它,等于没绕开(第一版就栽在这)。用别的名字即可。 + +## 5. verl 的序列打包硬依赖 flash_attn + +`verl/utils/attention_utils.py:28` 在 CUDA 分支直接 +`from flash_attn.bert_padding import ...`,**没有 fallback**,而且与 +`attn_implementation` 无关 —— 就算模型跑 sdpa,只要 `use_remove_padding=True` 就会 import。 + +torch 2.11+cu130 没有 flash-attn 预编译 wheel(源码编译 1-2 小时),所以: +`use_remove_padding=False` + `attn_implementation=sdpa`。 +代价是 padding 到等长,显存吃满(7B 在 4×H100 上每卡约 71GB)、速度也慢。 + +`run_rose_online.sh` 把 `use_remove_padding=True` 写死在第 250 行且末尾没有 `"$@"` 透传, +所以复制出 `run_rose_online_sdpa.sh`,末尾加三行可覆盖的开关(默认值与原件一致), +**原件一字未动**。 + +## 6. 跨分词器本身不用改代码 + +`rose_agent_loop` 早就有这条通路,`TEACHER_TOKENIZER` 非 null 即启用: +学生前缀 ids → decode 成文本 → 套**教师的** chat template → 教师续写 → 取回 `text` +→ 用**学生的** tokenizer 重新 encode。 + +Olmo(vocab 100278)× Qwen3-32B(vocab 151669)实测: +- 拼接缝文本两边 round-trip 都无损 +- 教师续写语义连贯,**没有重复吐 ``** +- 格式错位是良性的:Olmo 模板在 prompt 尾部就打开 ``,Qwen3 模板停在 + `assistant\n`。教师看到「`assistant\n` + 推理文字」,续写出的 ``+答案 + 接在学生的 `` 之后,正好构成 Olmo 期望的完整格式。 + +验证脚本:`xjoin.py`(需要 teacher server 已起)。 + +## 7. `TEACHER_MODEL` 不是路径 + +`bootstrap.sh teacher` 用 `--served-model-name teacher`,所以 API 的 model 字段就是 +字符串 `teacher`(`run_rose_online.sh:45` 的默认值)。传模型路径会 404。 + +## 8. `SAVE_FREQ` 拦不住 final save + +verl 在训练正常结束时**还会存一次**,与 `save_freq` 无关。3 步的冒烟测试也照存 33G +(Olmo-7B:model 分片 8.4G + optimizer 15.2G + HF 8.3G),当场把盘撑满。 +冒烟测试前先确认剩余空间,或者跑完立刻清。 diff --git a/eval/aime_eval.py b/eval/aime_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..8cea424edb00bb416e69b9c7034ee6c136683621 --- /dev/null +++ b/eval/aime_eval.py @@ -0,0 +1,223 @@ +"""AIME25 avg@k 评测。 + +针对 non-thinking 模型: prompt 必须用 enable_thinking=False 渲染 +(尾部会带 \n\n\n\n)。见 MEMORY: nothink-flag-for-1p7b-sft。 + +答案抽取按优先级: \boxed{} -> "Answer: X" -> 文末最后一个整数。 +AIME 答案恒为 0-999 的整数, 比较时按 int 比。 +""" +import argparse +import json +import os +import re +import sys +import time +from collections import Counter + +p = argparse.ArgumentParser() +p.add_argument("--model", required=True) +p.add_argument("--data", default="/home/haojinw2/datasets/aime25/test.jsonl") +p.add_argument("--out", required=True) +p.add_argument("--n", type=int, default=16) +p.add_argument("--max-new", type=int, default=8192) +p.add_argument("--ctx", type=int, default=10240) +p.add_argument("--tp", type=int, default=1) +p.add_argument("--temp", type=float, default=0.7) +p.add_argument("--top-p", type=float, default=0.8) +p.add_argument("--top-k", type=int, default=20) +p.add_argument("--seed", type=int, default=0) +p.add_argument("--enable-thinking", type=lambda s: s.lower() == "true", default=False, + help="False = 关闭 think(non-thinking 模型必须用 False)") +p.add_argument("--gpu-util", type=float, default=0.85) +a = p.parse_args() + + +def log(*x): + print(f"[{time.strftime('%H:%M:%S')}]", *x, flush=True) + + +INSTR = "Please reason step by step, and put your final answer within \\boxed{}." + +BOXED = re.compile(r"\\boxed\s*\{([^{}]*)\}") +ANSW = re.compile(r"[Aa]nswer\s*[::]\s*\$?(-?\d+)") +INTS = re.compile(r"-?\d+") + + +def _safe_int(s): + """AIME 答案恒为 0-999。超长数字是模型退化的产物(见过 28000 位), + 直接 int() 会撞 Python 3.11 的 int_max_str_digits=4300 抛 ValueError。""" + if s is None: + return None + t = s.lstrip("-") + if len(t) > 10 or not t: + return None + try: + return int(s) + except ValueError: + return None + + +def extract(text): + m = BOXED.findall(text) + if m: + digits = INTS.findall(m[-1]) + if digits: + v = _safe_int(digits[-1]) + if v is not None: + return v + m = ANSW.findall(text) + if m: + v = _safe_int(m[-1]) + if v is not None: + return v + m = INTS.findall(text) + if m: + return _safe_int(m[-1]) + return None + + +def _kill_descendants(): + """递归杀掉本进程的所有子孙(vLLM worker 是独立进程, 不杀会变孤儿占满卡)。""" + import signal + import subprocess + me = os.getpid() + try: + out = subprocess.run(["ps", "-eo", "pid,ppid", "--no-headers"], + capture_output=True, text=True, timeout=20).stdout + except Exception: + return + children = {} + for ln in out.splitlines(): + parts = ln.split() + if len(parts) != 2: + continue + try: + pid, ppid = int(parts[0]), int(parts[1]) + except ValueError: + continue + children.setdefault(ppid, []).append(pid) + doomed, stack = [], [me] + while stack: + for c in children.get(stack.pop(), []): + doomed.append(c) + stack.append(c) + for pid in reversed(doomed): # 先叶子后根 + try: + os.kill(pid, signal.SIGKILL) + except OSError: + pass + + +def _hard_exit(code=0): + sys.stdout.flush() + sys.stderr.flush() + _kill_descendants() + os._exit(code) + + +def _die(exc_type, exc, tb): + import traceback as _tb + _tb.print_exception(exc_type, exc, tb) + _hard_exit(1) # 异常路径同样要把 worker 一起带走 + + +sys.excepthook = _die + +rows = [json.loads(l) for l in open(a.data) if l.strip()] +log(f"{len(rows)} 题 x n={a.n}, enable_thinking={a.enable_thinking}") + +from transformers import AutoTokenizer +from vllm import LLM, SamplingParams + +tok = AutoTokenizer.from_pretrained(a.model, trust_remote_code=True) +prompts = [] +for r in rows: + msgs = [{"role": "user", "content": r["problem"].strip() + "\n\n" + INSTR}] + prompts.append(tok.apply_chat_template( + msgs, add_generation_prompt=True, tokenize=False, + enable_thinking=a.enable_thinking)) +log("prompt 尾部抽检: " + repr(prompts[0][-60:])) + +def _fit_tp(model_path, tp): + """attn heads 必须能被 TP 整除, 否则 vLLM 起不来(R1-Distill-1.5B 是 12 heads)。""" + try: + cfg = json.load(open(os.path.join(model_path, "config.json"))) + h = cfg.get("num_attention_heads") + except Exception: + return tp + if not h or h % tp == 0: + return tp + for cand in range(tp, 0, -1): + if h % cand == 0: + log(f"注意: {h} 个 attention head 不能被 TP={tp} 整除, 自动降为 TP={cand}") + return cand + return 1 + + +a.tp = _fit_tp(a.model, a.tp) +llm = LLM(model=a.model, tensor_parallel_size=a.tp, + gpu_memory_utilization=a.gpu_util, max_model_len=a.ctx, + trust_remote_code=True, seed=a.seed) +# 不给 SamplingParams 传 seed: per-request seed 会让 n>1 退出批量采样快路径, +# 480 个 seeded 样本会逐个采样、慢到形同挂死。可复现性由 LLM(seed=) 保证。 +sp = SamplingParams(n=a.n, temperature=a.temp, top_p=a.top_p, top_k=a.top_k, + max_tokens=a.max_new) + +t0 = time.time() +outs = llm.generate(prompts, sp) +log(f"生成 {time.time()-t0:.0f}s") + +recs, n_correct, n_tot, n_trunc, lens = [], 0, 0, 0, [] +per_problem = [] +for r, out in zip(rows, outs): + gt = int(str(r["answer"]).strip()) + hits, samples = 0, [] + for o in out.outputs: + pred = extract(o.text) + ok = (pred is not None and pred == gt) + hits += ok + n_correct += ok + n_tot += 1 + n_trunc += (o.finish_reason == "length") + lens.append(len(o.token_ids)) + samples.append({"pred": pred, "ok": bool(ok), + "finish": o.finish_reason, "len": len(o.token_ids), + "text": o.text}) + per_problem.append({"id": r.get("id"), "gt": gt, "hits": hits, "n": len(out.outputs)}) + recs.append({"id": r.get("id"), "problem": r["problem"], "gt": gt, + "hits": hits, "samples": samples}) + +os.makedirs(os.path.dirname(os.path.abspath(a.out)) or ".", exist_ok=True) +with open(a.out, "w") as f: + for rec in recs: + f.write(json.dumps(rec, ensure_ascii=False) + "\n") + +avg = n_correct / n_tot +solved_any = sum(1 for x in per_problem if x["hits"] > 0) +solved_all = sum(1 for x in per_problem if x["hits"] == x["n"]) +summary = { + "model": a.model, "data": a.data, "n": a.n, + "enable_thinking": a.enable_thinking, + "temperature": a.temp, "top_p": a.top_p, "top_k": a.top_k, + "max_new": a.max_new, "seed": a.seed, + "problems": len(rows), "samples": n_tot, + f"avg@{a.n}": avg, + "solved_any": solved_any, "solved_all": solved_all, + "truncated_rate": n_trunc / n_tot, + "resp_len_mean": sum(lens) / len(lens), + "resp_len_max": max(lens), + "hits_hist": dict(Counter(x["hits"] for x in per_problem)), +} +with open(a.out.replace(".jsonl", "_summary.json"), "w") as f: + json.dump(summary, f, indent=2, ensure_ascii=False) + +print() +print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True) +print() +print(f"AIME25 avg@{a.n} = {avg:.4f} ({n_correct}/{n_tot})", flush=True) +print(f" 解出>=1次 {solved_any}/{len(rows)}, 全对 {solved_all}/{len(rows)}", flush=True) +print(f" 截断率 {n_trunc/n_tot:.3f}, 平均长度 {sum(lens)/len(lens):.0f}", flush=True) + +# vLLM TP>1 的 worker 常在析构时挂尸、占着整卡不放, 后续任务就起不来。 +# 结果已全部写盘, 直接跳过析构强制退出。 +_hard_exit(0) diff --git a/eval/aime_launch.py b/eval/aime_launch.py new file mode 100644 index 0000000000000000000000000000000000000000..b669ef32956af31eeea43fdd19903738acdf10b1 --- /dev/null +++ b/eval/aime_launch.py @@ -0,0 +1,17 @@ +"""spawn 安全的 aime_eval 启动器。 + +vllm 0.27 对 Qwen3.5 这类模型强制用 spawn 起 worker, 子进程会重新 import +父进程的 __main__。aime_eval.py 是平铺脚本, 被重新 import 就会触发 +multiprocessing 的 "freeze_support / Safe importing of main module" 报错。 + +关键是 run_name 不能用 "__main__": runpy 在执行期间会把 sys.modules["__main__"] +临时换成目标脚本, spawn 读到的就还是 aime_eval.py, 等于没绕开(先踩过这个)。 +用别的名字, sys.modules["__main__"] 始终是本文件, 子进程 import 到 __mp_main__, +下面的 guard 不成立, 什么都不执行。aime_eval.py 的逻辑全在模块层, 不依赖 __name__。 +""" +import runpy +import sys + +if __name__ == "__main__": + sys.argv[0] = "/home/haojinw2/aime_eval.py" + runpy.run_path("/home/haojinw2/aime_eval.py", run_name="__aime_eval__") diff --git a/eval/hmmt_eval.py b/eval/hmmt_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..927fd7004a6a4bb6ee64faf987b9ca0420111a79 --- /dev/null +++ b/eval/hmmt_eval.py @@ -0,0 +1,165 @@ +"""HMMT25 avg@k 评测。 + +与 aime_eval.py 的差别只在答案比对: HMMT 的标准答案 30 题里只有 14 题是纯整数, +其余是 LaTeX 表达式(\frac{1}{576}、\frac{9\sqrt{23}}{23}、1-\frac{2}{\pi}、 +甚至 "\frac{-1+\sqrt{17}}{2}, \frac{-1-\sqrt{17}}{2}" 这种多解)。 +按 int() 比会把 16 题全判错, 上限只有 14/30 —— 所以改用 math_verify 做符号等价判定。 + +采样与输出格式跟 aime_eval.py 保持一致, 便于并排比较。 +""" +import argparse +import json +import os +import sys +import time +from collections import Counter + +p = argparse.ArgumentParser() +p.add_argument("--model", required=True) +p.add_argument("--data", default="/home/haojinw2/datasets/hmmt25/test.jsonl") +p.add_argument("--out", required=True) +p.add_argument("--n", type=int, default=16) +p.add_argument("--max-new", type=int, default=31744) +p.add_argument("--ctx", type=int, default=33792) +p.add_argument("--tp", type=int, default=8) +p.add_argument("--temp", type=float, default=0.7) +p.add_argument("--top-p", type=float, default=0.95) +p.add_argument("--top-k", type=int, default=-1) +p.add_argument("--seed", type=int, default=0) +p.add_argument("--enable-thinking", type=lambda s: s.lower() == "true", default=True) +p.add_argument("--gpu-util", type=float, default=0.90) +a = p.parse_args() + +INSTR = ("\n\nPlease reason step by step, and put your final answer within \\boxed{}.") + + +def _kill_descendants(): + """递归杀掉所有子孙 —— vLLM worker 是独立进程, 主进程一退它们就变孤儿占满卡。""" + import signal + import subprocess + me = os.getpid() + try: + out = subprocess.run(["ps", "-eo", "pid,ppid"], capture_output=True, text=True, timeout=20).stdout + except Exception: + return + kids = {} + for line in out.splitlines()[1:]: + parts = line.split() + if len(parts) >= 2: + try: + kids.setdefault(int(parts[1]), []).append(int(parts[0])) + except ValueError: + pass + order, stack = [], [me] + while stack: + cur = stack.pop() + for k in kids.get(cur, []): + order.append(k) + stack.append(k) + for pid in reversed(order): + try: + os.kill(pid, signal.SIGKILL) + except Exception: + pass + + +def _hard_exit(code=0): + sys.stdout.flush() + sys.stderr.flush() + _kill_descendants() + os._exit(code) + + +sys.excepthook = lambda *e: (__import__("traceback").print_exception(*e), _hard_exit(1)) + +rows = [json.loads(l) for l in open(a.data) if l.strip()] +print(f"{len(rows)} 题, n={a.n}, max_new={a.max_new}, thinking={a.enable_thinking}", flush=True) + +from transformers import AutoTokenizer +from vllm import LLM, SamplingParams + +tok = AutoTokenizer.from_pretrained(a.model, trust_remote_code=True) + + +def render(problem): + msgs = [{"role": "user", "content": problem + INSTR}] + try: + return tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, + enable_thinking=a.enable_thinking) + except TypeError: + return tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True) + + +prompts = [render(r["problem"]) for r in rows] + +llm = LLM(model=a.model, tensor_parallel_size=a.tp, max_model_len=a.ctx, + gpu_memory_utilization=a.gpu_util, trust_remote_code=True, seed=a.seed) +sp = SamplingParams(n=a.n, temperature=a.temp, top_p=a.top_p, top_k=a.top_k, + max_tokens=a.max_new, seed=a.seed) +t0 = time.time() +outs = llm.generate(prompts, sp) +print(f"生成用时 {(time.time() - t0) / 60:.1f} 分钟", flush=True) + +from math_verify import parse, verify + + +def is_correct(text, gt): + """math_verify 做符号等价判定; 解析不出来就退回去掉空白的字符串比对。""" + try: + pred = parse(text) + if pred and verify(parse("$" + str(gt) + "$"), pred): + return True + except Exception: + pass + import re + m = re.findall(r"\\boxed\{([^{}]*(?:\{[^{}]*\}[^{}]*)*)\}", text) + if m: + s = m[-1].replace(" ", "").replace("\\!", "").replace("\\,", "") + return s == str(gt).replace(" ", "") + return False + + +recs, per_problem, lens, n_trunc, n_correct, n_tot = [], [], [], 0, 0, 0 +for r, out in zip(rows, outs): + gt = r["answer"] + samples, hits = [], 0 + for o in out.outputs: + ok = is_correct(o.text, gt) + hits += ok + n_correct += ok + n_tot += 1 + lens.append(len(o.token_ids)) + n_trunc += o.finish_reason == "length" + samples.append({"ok": bool(ok), "finish": o.finish_reason, + "len": len(o.token_ids), "text": o.text}) + per_problem.append({"id": r["id"], "gt": gt, "hits": hits, "n": len(out.outputs)}) + recs.append({"id": r["id"], "problem": r["problem"], "gt": gt, "hits": hits, "samples": samples}) + +os.makedirs(os.path.dirname(os.path.abspath(a.out)) or ".", exist_ok=True) +with open(a.out, "w") as f: + for rec in recs: + f.write(json.dumps(rec, ensure_ascii=False) + "\n") + +avg = n_correct / n_tot +summary = { + "model": a.model, "data": a.data, "n": a.n, "enable_thinking": a.enable_thinking, + "temperature": a.temp, "top_p": a.top_p, "max_new": a.max_new, "seed": a.seed, + "problems": len(rows), "samples": n_tot, + f"avg@{a.n}": avg, + "pass@1_first": sum(1 for x in recs if x["samples"][0]["ok"]) / len(recs), + "solved_any": sum(1 for x in per_problem if x["hits"] > 0), + "solved_all": sum(1 for x in per_problem if x["hits"] == x["n"]), + "truncated_rate": n_trunc / n_tot, + "resp_len_mean": sum(lens) / len(lens), + "hits_hist": dict(Counter(x["hits"] for x in per_problem)), +} +with open(a.out.replace(".jsonl", "_summary.json"), "w") as f: + json.dump(summary, f, indent=2, ensure_ascii=False) + +print() +print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True) +print(f"\nHMMT25 avg@{a.n} = {avg:.4f} ({n_correct}/{n_tot})") +print(f" pass@1(第一条) {summary['pass@1_first']:.4f}") +print(f" 解出>=1次 {summary['solved_any']}/{len(rows)}, 全对 {summary['solved_all']}/{len(rows)}") +print(f" 截断率 {summary['truncated_rate']:.3f}, 平均长度 {summary['resp_len_mean']:.0f}", flush=True) +_hard_exit(0) diff --git a/eval/merge_eval.py b/eval/merge_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..1f2bc2680463c09dc8be14b97bd7cbec831e3d1c --- /dev/null +++ b/eval/merge_eval.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +"""Merge data-parallel eval shards into one evalout directory. + + python3 merge_eval.py OUT_DIR SHARD_DIR [SHARD_DIR ...] + +Each shard is an evalout dir produced with IDX_START/IDX_END. Rows carry their +true dataframe idx, so merging is a concatenate + sort; the summary is then +recomputed with the same formulas rlve_eval.py uses, so the output file is +interchangeable with a single-process run. +""" +import json, os, sys +from collections import Counter +from math import comb + +def pass_at_k(N, c, k): + if N - c < k: return 1.0 + return 1.0 - comb(N - c, k) / comb(N, k) + +out, shards = sys.argv[1], sys.argv[2:] +rows, base = [], None +for sd in shards: + with open(f"{sd}/rollouts.jsonl") as f: + rows += [json.loads(l) for l in f] + if base is None: + base = json.load(open(f"{sd}/summary.json")) +seen = set() +for r in rows: + if r["idx"] in seen: sys.exit(f"重复 idx {r['idx']} -- 分片区间有重叠, 拒绝合并") + seen.add(r["idx"]) +rows.sort(key=lambda r: r["idx"]) + +flat = [rec for r in rows for rec in r["recs"]] +n = len(flat) +N = base["n"] +per_env = {} +for r in rows: + per_env.setdefault(r["env"], []).extend(rec["accuracy"] for rec in r["recs"]) +passk = {} +if N > 1: + for k in (1, 2, 4, 8, 16): + if k > N: continue + passk[f"pass@{k}"] = sum( + pass_at_k(N, sum(rec["accuracy"] for rec in r["recs"]), k) for r in rows) / len(rows) + passk["solved_any"] = sum(1 for r in rows if any(rec["accuracy"] for rec in r["recs"])) + passk["solved_all"] = sum(1 for r in rows if all(rec["accuracy"] for rec in r["recs"])) + +summary = dict(base) +# base is shard A's summary, whose "tag" carries the shard suffix. Left as-is it +# labels a 180-problem merge as if it were the 90-problem shard, which silently +# corrupts any table keyed on tag. +summary["tag"] = os.path.basename(out.rstrip("/")).replace("rlve_", "", 1) +summary.update({ + "problems": len(rows), + "accuracy@1": sum(r["accuracy"] for r in flat) / n, + "reward_mean@1": sum(r["reward"] for r in flat) / n, + "format_ok_rate": sum(r["format_score"] for r in flat) / n, + "truncated_rate": sum(r["finish"] == "length" for r in flat) / n, + "closed_think_rate": sum(r["closed_think"] for r in flat) / n, + "resp_len_mean": sum(r["len"] for r in flat) / n, + "resp_len_max": max(r["len"] for r in flat), + "reward_hist": dict(Counter(round(r["reward"], 3) for r in flat)), + "n_envs": len(per_env), + "merged_from": shards, + **passk, +}) +os.makedirs(out, exist_ok=True) +json.dump(summary, open(f"{out}/summary.json", "w"), indent=2, ensure_ascii=False) +json.dump({k: sum(v)/len(v) for k, v in sorted(per_env.items())}, + open(f"{out}/per_env.json", "w"), indent=2) +with open(f"{out}/rollouts.jsonl", "w") as f: + for r in rows: f.write(json.dumps(r, ensure_ascii=False) + "\n") +print(json.dumps({k: v for k, v in summary.items() if k != "reward_hist"}, indent=2, ensure_ascii=False)) diff --git a/eval/retrunc_eval.py b/eval/retrunc_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..481c6c7f03a5175e3d58a603fcf504c469702c44 --- /dev/null +++ b/eval/retrunc_eval.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +"""Re-score existing rollouts under a smaller generation budget. + + retrunc_eval.py + +vLLM's max_tokens does not influence the sampling distribution of earlier +tokens, so the first BUDGET tokens of a 32k rollout are distributed exactly as a +BUDGET-token rollout would be. Truncating and re-scoring is an exact +re-measurement at the smaller budget, not an approximation -- it just costs no +GPU. finish_reason is fixed up by hand: a sample that ran past BUDGET would have +been cut off, so it becomes 'length'. + +Scoring reuses env.verifier() on the decoded text, and the per-environment +config comes from the source parquet -- rollouts.jsonl does not carry it, and +verifier() without the right config does not mean the same thing. +""" +import json, os, sys +from collections import Counter +from math import comb + +GYM = "/home/haojinw2/rose-bundle-new/RLVE-Eval" +sys.path.insert(0, GYM) +import Gym.environments as G + + +def score_one(meta, text): + cls = getattr(G, f"{meta['environment']}_Environment", None) + if cls is None: + return None + env = cls() + env.set_config(meta.get("config", {})) + try: + return env.verifier(text) + except Exception as e: # noqa: BLE001 + return {"reward": -1.0, "accuracy": 0, "format_score": 0, "err": repr(e)} + + +def pass_at_k(N, c, k): + if N - c < k: + return 1.0 + return 1.0 - comb(N - c, k) / comb(N, k) + + +BUDGET, SRC, DST = int(sys.argv[1]), sys.argv[2], sys.argv[3] +D = "/home/haojinw2/evalout" +src, dst = f"{D}/rlve_{SRC}", f"{D}/rlve_{DST}" +base = json.load(open(f"{src}/summary.json")) + +import pandas as pd +from transformers import AutoTokenizer +df = pd.read_parquet(base["data"]) +tok = AutoTokenizer.from_pretrained(base["model"]) + +rows, cut, kept = [], 0, 0 +for line in open(f"{src}/rollouts.jsonl"): + d = json.loads(line) + meta = json.loads(df.iloc[d["idx"]]["extra_info"]["metadata"]) + assert meta["environment"] == d["env"], (d["idx"], meta["environment"], d["env"]) + recs = [] + for i, ids in enumerate(d["ids"]): + if len(ids) <= BUDGET: + recs.append(dict(d["recs"][i])); kept += 1 + continue + txt = tok.decode(ids[:BUDGET], skip_special_tokens=False) + v = score_one(meta, txt) or {"reward": 0.0, "accuracy": 0, "format_score": 0} + r = dict(v); r["len"] = BUDGET; r["finish"] = "length" + r["closed_think"] = "" in txt + recs.append(r); cut += 1 + rows.append({"idx": d["idx"], "env": d["env"], "gt": d["gt"], "recs": recs}) + +flat = [x for r in rows for x in r["recs"]] +n, N = len(flat), base["n"] +per_env = {} +for r in rows: + per_env.setdefault(r["env"], []).extend(x["accuracy"] for x in r["recs"]) +passk = {} +if N > 1: + for k in (1, 2, 4, 8, 16): + if k > N: continue + passk[f"pass@{k}"] = sum( + pass_at_k(N, sum(x["accuracy"] for x in r["recs"]), k) for r in rows) / len(rows) + passk["solved_any"] = sum(1 for r in rows if any(x["accuracy"] for x in r["recs"])) + passk["solved_all"] = sum(1 for r in rows if all(x["accuracy"] for x in r["recs"])) + +s = dict(base) +s.update({"tag": DST, "max_new": BUDGET, "ctx": BUDGET + 2048, "problems": len(rows), + "accuracy@1": sum(x["accuracy"] for x in flat) / n, + "reward_mean@1": sum(x["reward"] for x in flat) / n, + "format_ok_rate": sum(x["format_score"] for x in flat) / n, + "truncated_rate": sum(x["finish"] == "length" for x in flat) / n, + "closed_think_rate": sum(x["closed_think"] for x in flat) / n, + "resp_len_mean": sum(x["len"] for x in flat) / n, + "resp_len_max": max(x["len"] for x in flat), + "reward_hist": dict(Counter(round(x["reward"], 3) for x in flat)), + "n_envs": len(per_env), "retruncated_from": SRC, + "n_rescored": cut, "n_unchanged": kept, **passk}) +os.makedirs(dst, exist_ok=True) +json.dump(s, open(f"{dst}/summary.json", "w"), indent=2, ensure_ascii=False) +json.dump({k: sum(v)/len(v) for k, v in sorted(per_env.items())}, + open(f"{dst}/per_env.json", "w"), indent=2) +with open(f"{dst}/rollouts.jsonl", "w") as f: + for r in rows: f.write(json.dumps(r, ensure_ascii=False) + "\n") +print(" %-20s 重判 %5d / 保留 %5d avg@8 %.4f pass@8 %.4f 解出 %3d fmt %.3f trunc %.3f" + % (DST, cut, kept, s["accuracy@1"], s.get("pass@8", float("nan")), + s.get("solved_any", -1), s["format_ok_rate"], s["truncated_rate"])) diff --git a/eval/rlve_eval.py b/eval/rlve_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..bfe2558d95bc31334c704ab9acc3324ecda7ad60 --- /dev/null +++ b/eval/rlve_eval.py @@ -0,0 +1,212 @@ +"""mean@1 evaluation of a checkpoint on data/rlve_test.parquet. + +Separate from training on purpose. The reward logged during ROSE-online scores +the *spliced* sequence (student prefix + teacher continuation), which never +reaches an answer -- see recipe/rose/rlve_reward.py. This scores the student. + +Context is 16384 tokens end to end: max_tokens is computed per prompt as +CTX - len(prompt_ids) - 8, so nothing gets a larger window than the flag names. + +Scores come from `env.verifier()` directly rather than recipe/rose/rlve_reward's +compute_score, which throws away two of the three fields. verifier returns +reward / accuracy / format_score, and `accuracy` is the metric that survives +environments disagreeing on reward scale -- they do. The bundle docstring claims +"-0.5 = answered and wrong"; BannedPointSupersetPathCounting actually uses +{correct +1.0, wrong_answer 0.0, wrong_range -0.5, wrong_format -1.0}. + +Everything runs under `if __name__ == "__main__"` because vLLM spawns its engine +core with the spawn start method, which re-imports this module in the child. +Module-level work here means the child tries to build a second LLM during import +and dies in `_check_not_importing_main`. + + MODEL= TAG= [TP=4] [N=1] [CTX=16384] python3 rlve_eval.py +""" + +import json +import os +import sys +import time +from collections import Counter + +GYM = "/home/haojinw2/rose-bundle-new/RLVE-Eval" +sys.path.insert(0, GYM) + +import Gym.environments as G # noqa: E402 + + +def score_one(meta, text): + cls = getattr(G, f"{meta['environment']}_Environment", None) + if cls is None: + return None + env = cls() + env.set_config(meta.get("config", {})) + try: + return env.verifier(text) + except Exception as e: # noqa: BLE001 + return {"reward": -1.0, "accuracy": 0, "format_score": 0, "err": repr(e)} + + +def main(): + CTX = int(os.environ.get("CTX", 16384)) + TP = int(os.environ.get("TP", 4)) + N = int(os.environ.get("N", 1)) + MODEL = os.environ["MODEL"] + TAG = os.environ.get("TAG", os.path.basename(MODEL.rstrip("/"))) + DATA = os.environ.get("DATA", "/home/haojinw2/rose-bundle-new/data/rlve_test.parquet") + OUT = os.environ.get("OUT", f"/home/haojinw2/evalout/rlve_{TAG}") + MAX_NEW = int(os.environ.get("MAX_NEW", 0)) # 0 = use whatever CTX leaves + os.makedirs(OUT, exist_ok=True) + + import pandas as pd + from transformers import AutoTokenizer + from vllm import LLM, SamplingParams + + df = pd.read_parquet(DATA) + tok = AutoTokenizer.from_pretrained(MODEL) + + prompts, keep = [], [] + # Data-parallel slicing. Two halves on 0-3 and 4-7 beat TP=8 on a 1.7B -- + # no collectives, twice the KV cache. `idx` stays the true dataframe row, + # so merge_eval.py can concatenate the shards without renumbering. + IDX_START = int(os.environ.get("IDX_START", 0)) + IDX_END = int(os.environ.get("IDX_END", 0)) or len(df) + for i in range(IDX_START, IDX_END): + row = df.iloc[i] + msgs = [dict(m) for m in row["prompt"]] + ids = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True) + budget = CTX - len(ids) - 8 + if MAX_NEW: + # fixed generation budget (train-set sampling), not "whatever the + # context leaves over" -- CTX still caps it so nothing overflows + budget = min(budget, MAX_NEW) + if budget < 256: + # no room to answer is a data problem, not a model failure -- drop it + # loudly instead of scoring it as a format failure + print(f"!! row {i}: prompt {len(ids)} tok leaves {budget}, skipped") + continue + prompts.append(tok.decode(ids)) + keep.append((i, budget)) + + print(f"{len(prompts)}/{len(df)} 题, ctx={CTX}, n={N}, tp={TP}", flush=True) + + llm = LLM( + model=MODEL, + tensor_parallel_size=TP, + max_model_len=CTX, + gpu_memory_utilization=float(os.environ.get("GPU_UTIL", 0.85)), + trust_remote_code=True, + seed=int(os.environ.get("EVAL_SEED", "0")), + ) + + # one SamplingParams per prompt so each gets its own exact budget + # 0.7 / 0.9, no top-k -- the project standard, matching the reference eval + # script's argparse defaults. Training rollout uses the same values so the + # states ROSE distils on are the states the model actually visits here. + TEMP = float(os.environ.get("TEMP", 0.7)) + TOP_P = float(os.environ.get("TOP_P", 0.9)) + TOP_K = int(os.environ.get("TOP_K", -1)) + print(f"sampling: temp={TEMP} top_p={TOP_P} top_k={TOP_K}", flush=True) + sps = [ + SamplingParams(n=N, temperature=TEMP, top_p=TOP_P, top_k=TOP_K, + max_tokens=b, seed=int(os.environ.get("EVAL_SEED", "0"))) + for _, b in keep + ] + + t0 = time.time() + outs = llm.generate(prompts, sps) + print(f"生成 {time.time() - t0:.0f}s", flush=True) + + rows = [] + per_env = {} + for (idx, _), o in zip(keep, outs): + row = df.iloc[idx] + meta = json.loads(row["extra_info"]["metadata"]) + recs = [] + for c in o.outputs: + v = score_one(meta, c.text) or {"reward": 0.0, "accuracy": 0, "format_score": 0} + v["len"] = len(c.token_ids) + v["finish"] = c.finish_reason + v["closed_think"] = "" in c.text + recs.append(v) + rows.append( + { + "idx": int(idx), + "env": meta["environment"], + "gt": row["reward_model"]["ground_truth"], + "recs": recs, + "text": [c.text for c in o.outputs], + # exact ids, so a downstream teacher-continuation step can join + # id-to-id instead of re-tokenising the decoded text (which can + # shift the seam -- the one silent failure in this design) + "ids": [list(c.token_ids) for c in o.outputs], + } + ) + per_env.setdefault(meta["environment"], []).append( + 1.0 if any(r["accuracy"] for r in recs) else 0.0 + ) + + flat = [r for row in rows for r in row["recs"]] + n = len(flat) + + # Unbiased pass@k (Chen et al. 2021): with c of N samples correct, + # pass@k = 1 - C(N-c, k) / C(N, k) + # Reporting the whole curve costs nothing once the samples exist, and + # pass@1 from it is the same quantity as accuracy@1 -- a useful self-check. + from math import comb + def pass_at_k(N, c, k): + if N - c < k: + return 1.0 + return 1.0 - comb(N - c, k) / comb(N, k) + + passk = {} + if N > 1: + for k in [1, 2, 4, 8, 16]: + if k > N: + continue + passk[f"pass@{k}"] = sum( + pass_at_k(N, sum(r["accuracy"] for r in row["recs"]), k) for row in rows + ) / len(rows) + # how many problems are solved by at least one sample, ever + passk["solved_any"] = sum( + 1 for row in rows if any(r["accuracy"] for r in row["recs"]) + ) + passk["solved_all"] = sum( + 1 for row in rows if all(r["accuracy"] for r in row["recs"]) + ) + + summary = { + "tag": TAG, + "model": MODEL, + "ctx": CTX, + "max_new": MAX_NEW or None, + "data": DATA, + "n": N, + "temperature": TEMP, + "top_p": TOP_P, + "top_k": TOP_K, + "problems": len(rows), + "accuracy@1": sum(r["accuracy"] for r in flat) / n, + "reward_mean@1": sum(r["reward"] for r in flat) / n, + "format_ok_rate": sum(r["format_score"] for r in flat) / n, + "truncated_rate": sum(r["finish"] == "length" for r in flat) / n, + "closed_think_rate": sum(r["closed_think"] for r in flat) / n, + "resp_len_mean": sum(r["len"] for r in flat) / n, + "resp_len_max": max(r["len"] for r in flat), + "reward_hist": dict(Counter(round(r["reward"], 3) for r in flat)), + "n_envs": len(per_env), + **passk, + } + print(json.dumps(summary, indent=2, ensure_ascii=False), flush=True) + + with open(f"{OUT}/summary.json", "w") as f: + json.dump(summary, f, indent=2, ensure_ascii=False) + with open(f"{OUT}/per_env.json", "w") as f: + json.dump({k: sum(v) / len(v) for k, v in sorted(per_env.items())}, f, indent=2) + with open(f"{OUT}/rollouts.jsonl", "w") as f: + for r in rows: + f.write(json.dumps(r, ensure_ascii=False) + "\n") + print(f"-> {OUT}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/frameworks/agentgym_rl/.gitignore b/frameworks/agentgym_rl/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..4fbd2436282c071ca0f5d8f0e993fdb0aae39e80 --- /dev/null +++ b/frameworks/agentgym_rl/.gitignore @@ -0,0 +1,116 @@ +**/*.pt +**/checkpoints +**/wget-log +**/_build/ +**/*.ckpt +**/outputs +**/*.tar.gz +**/playground +**/wandb + +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] +*$py.class +dataset/* +tensorflow/my_graph/* +.idea/ +# C extensions +*.so + +# Distribution / packaging +.Python +env/ +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +*.egg-info/ +.installed.cfg +*.egg + +# PyInstaller +# Usually these files are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*,cover +.hypothesis/ + +# Translations +*.mo +*.pot + +# Django stuff: +*.log +local_settings.py + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +target/ + +# IPython Notebook +.ipynb_checkpoints + +# pyenv +.python-version + +# celery beat schedule file +celerybeat-schedule + +# dotenv +.env + +# virtualenv +venv/ +ENV/ + +# Spyder project settings +.spyderproject + +# Rope project settings +.ropeproject + +# vscode +.vscode + +# Mac +.DS_Store + +# output logs +tests/e2e/toy_examples/deepspeed/synchronous/output.txt + +# vim +*.swp + +# ckpt +*.lock \ No newline at end of file diff --git a/frameworks/agentgym_rl/.readthedocs.yaml b/frameworks/agentgym_rl/.readthedocs.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2df37965f7a8176b5ce41bfa5a2755f60658263f --- /dev/null +++ b/frameworks/agentgym_rl/.readthedocs.yaml @@ -0,0 +1,16 @@ +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details + +version: 2 + +build: + os: ubuntu-22.04 + tools: + python: "3.8" + +sphinx: + configuration: docs/conf.py + +python: + install: + - requirements: docs/requirements-docs.txt \ No newline at end of file diff --git a/frameworks/agentgym_rl/.style.yapf b/frameworks/agentgym_rl/.style.yapf new file mode 100644 index 0000000000000000000000000000000000000000..a09a27c312b93a261b060fbe9fb217a22358bb0d --- /dev/null +++ b/frameworks/agentgym_rl/.style.yapf @@ -0,0 +1,5 @@ +[style] +based_on_style = google +column_limit = 120 +indent_width = 4 +split_arguments_when_comma_terminated: true \ No newline at end of file diff --git a/frameworks/agentgym_rl/Notice.txt b/frameworks/agentgym_rl/Notice.txt new file mode 100644 index 0000000000000000000000000000000000000000..ade439da525ac3f82936e131a1ae386f43207fd8 --- /dev/null +++ b/frameworks/agentgym_rl/Notice.txt @@ -0,0 +1 @@ +Copyright 2023-2024 Bytedance Ltd. and/or its affiliates \ No newline at end of file diff --git a/frameworks/agentgym_rl/VERL_LICENSE b/frameworks/agentgym_rl/VERL_LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..d645695673349e3947e8e5ae42332d0ac3164cd7 --- /dev/null +++ b/frameworks/agentgym_rl/VERL_LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/frameworks/agentgym_rl/VERL_README.md b/frameworks/agentgym_rl/VERL_README.md new file mode 100644 index 0000000000000000000000000000000000000000..77740e99acfa8ab746cb5b082512e830efb6e06c --- /dev/null +++ b/frameworks/agentgym_rl/VERL_README.md @@ -0,0 +1,135 @@ +

verl: Volcano Engine Reinforcement Learning for LLM

+ +verl is a flexible, efficient and production-ready RL training library for large language models (LLMs). + +verl is the open-source version of **[HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2)** paper. + +verl is flexible and easy to use with: + +- **Easy extension of diverse RL algorithms**: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code. + +- **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks. + +- **Flexible device mapping**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. + +- Readily integration with popular HuggingFace models + + +verl is fast with: + +- **State-of-the-art throughput**: By seamlessly integrating existing SOTA LLM training and inference frameworks, verl achieves high generation and training throughput. + +- **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. + +

+| Documentation | Paper | Slack | Wechat | Twitter + + +

+ +## News + +- [2025/2] We will present verl in the [Bytedance/NVIDIA/Anyscale Ray Meetup](https://lu.ma/ji7atxux) in bay area on Feb 13th. Come join us in person! +- [2025/1] [Doubao-1.5-pro](https://team.doubao.com/zh/special/doubao_1_5_pro) is released with SOTA-level performance on LLM & VLM. The RL scaling preview model is trained using verl, reaching OpenAI O1-level performance on math benchmarks (70.0 pass@1 on AIME). +- [2024/12] The team presented Post-training LLMs: From Algorithms to Infrastructure at NeurIPS 2024. [Slides](https://github.com/eric-haibin-lin/verl-data/tree/neurips) and [video](https://neurips.cc/Expo/Conferences/2024/workshop/100677) available. +- [2024/10] verl is presented at Ray Summit. [Youtube video](https://www.youtube.com/watch?v=MrhMcXkXvJU&list=PLzTswPQNepXntmT8jr9WaNfqQ60QwW7-U&index=37) available. +- [2024/08] HybridFlow (verl) is accepted to EuroSys 2025. + +## Key Features + +- **FSDP** and **Megatron-LM** for training. +- **vLLM** and **TGI** for rollout generation, **SGLang** support coming soon. +- huggingface models support +- Supervised fine-tuning +- Reinforcement learning from human feedback with [PPO](https://github.com/volcengine/verl/tree/main/examples/ppo_trainer), [GRPO](https://github.com/volcengine/verl/tree/main/examples/grpo_trainer), and [ReMax](https://github.com/volcengine/verl/tree/main/examples/remax_trainer) + - Support model-based reward and function-based reward (verifiable reward) +- flash-attention, [sequence packing](examples/ppo_trainer/run_qwen2-7b_seq_balance.sh), [long context](examples/ppo_trainer/run_deepseek7b_llm_sp2.sh) support via DeepSpeed Ulysses, [LoRA](examples/sft/gsm8k/run_qwen_05_peft.sh), [Liger-kernel](examples/sft/gsm8k/run_qwen_05_sp2_liger.sh) +- scales up to 70B models and hundreds of GPUs +- experiment tracking with wandb, swanlab and mlflow + +## Upcoming Features +- Reward model training +- DPO training +- DeepSeek integration with Megatron backend +- SGLang integration + +## Getting Started + +Checkout this [Jupyter Notebook](https://github.com/volcengine/verl/tree/main/examples/ppo_trainer/verl_getting_started.ipynb) to get started with PPO training with a single 24GB L4 GPU (**FREE** GPU quota provided by [Lighting Studio](https://lightning.ai/hlin-verl/studios/verl-getting-started))! + +**Quickstart:** +- [Installation](https://verl.readthedocs.io/en/latest/start/install.html) +- [Quickstart](https://verl.readthedocs.io/en/latest/start/quickstart.html) +- [Programming Guide](https://verl.readthedocs.io/en/latest/hybrid_flow.html) + +**Running a PPO example step-by-step:** +- Data and Reward Preparation + - [Prepare Data for Post-Training](https://verl.readthedocs.io/en/latest/preparation/prepare_data.html) + - [Implement Reward Function for Dataset](https://verl.readthedocs.io/en/latest/preparation/reward_function.html) +- Understanding the PPO Example + - [PPO Example Architecture](https://verl.readthedocs.io/en/latest/examples/ppo_code_architecture.html) + - [Config Explanation](https://verl.readthedocs.io/en/latest/examples/config.html) + - [Run GSM8K Example](https://verl.readthedocs.io/en/latest/examples/gsm8k_example.html) + +**Reproducible algorithm baselines:** +- [PPO and GRPO](https://verl.readthedocs.io/en/latest/experiment/ppo.html) + +**For code explanation and advance usage (extension):** +- PPO Trainer and Workers + - [PPO Ray Trainer](https://verl.readthedocs.io/en/latest/workers/ray_trainer.html) + - [PyTorch FSDP Backend](https://verl.readthedocs.io/en/latest/workers/fsdp_workers.html) + - [Megatron-LM Backend](https://verl.readthedocs.io/en/latest/index.html) +- Advance Usage and Extension + - [Ray API design tutorial](https://verl.readthedocs.io/en/latest/advance/placement.html) + - [Extend to Other RL(HF) algorithms](https://verl.readthedocs.io/en/latest/advance/dpo_extension.html) + - [Add Models with the FSDP Backend](https://verl.readthedocs.io/en/latest/advance/fsdp_extension.html) + - [Add Models with the Megatron-LM Backend](https://verl.readthedocs.io/en/latest/advance/megatron_extension.html) + - [Deployment using Separate GPU Resources](https://github.com/volcengine/verl/tree/main/examples/split_placement) + +## Performance Tuning Guide +The performance is essential for on-policy RL algorithm. We write a detailed performance tuning guide to allow people tune the performance. See [here](https://verl.readthedocs.io/en/latest/perf/perf_tuning.html) for more details. + +## vLLM v0.7 testing version +We have released a testing version of veRL that supports vLLM>=0.7.0. Please refer to [this document](https://github.com/volcengine/verl/docs/README_vllm0.7.md) for installation guide and more information. + +## Contribution Guide +Contributions from the community are welcome! + +### Code formatting +We use yapf (Google style) to enforce strict code formatting when reviewing PRs. To reformat you code locally, make sure you installed **latest** `yapf` +```bash +pip3 install yapf --upgrade +``` +Then, make sure you are at top level of verl repo and run +```bash +bash scripts/format.sh +``` + +## Citation and acknowledgement + +If you find the project helpful, please cite: +- [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2) +- [A Framework for Training Large Language Models for Code Generation via Proximal Policy Optimization](https://i.cs.hku.hk/~cwu/papers/gmsheng-NL2Code24.pdf) + +```tex +@article{sheng2024hybridflow, + title = {HybridFlow: A Flexible and Efficient RLHF Framework}, + author = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu}, + year = {2024}, + journal = {arXiv preprint arXiv: 2409.19256} +} +``` + +verl is inspired by the design of Nemo-Aligner, Deepspeed-chat and OpenRLHF. The project is adopted and supported by Anyscale, Bytedance, LMSys.org, Shanghai AI Lab, Tsinghua University, UC Berkeley, UCLA, UIUC, and University of Hong Kong. + +## Awesome work using verl +- [Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization](https://arxiv.org/abs/2410.09302) +- [Flaming-hot Initiation with Regular Execution Sampling for Large Language Models](https://arxiv.org/abs/2410.21236) +- [Process Reinforcement Through Implicit Rewards](https://github.com/PRIME-RL/PRIME/) +- [TinyZero](https://github.com/Jiayi-Pan/TinyZero): a reproduction of DeepSeek R1 Zero recipe for reasoning tasks +- [RAGEN](https://github.com/ZihanWang314/ragen): a general-purpose reasoning agent training framework +- [Logic R1](https://github.com/Unakar/Logic-RL): a reproduced DeepSeek R1 Zero on 2K Tiny Logic Puzzle Dataset. +- [deepscaler](https://github.com/agentica-project/deepscaler): iterative context scaling with GRPO +- [critic-rl](https://github.com/HKUNLP/critic-rl): Teaching Language Models to Critique via Reinforcement Learning + +We are HIRING! Send us an [email](mailto:haibin.lin@bytedance.com) if you are interested in internship/FTE opportunities in MLSys/LLM reasoning/multimodal alignment. diff --git a/frameworks/agentgym_rl/docs/Makefile b/frameworks/agentgym_rl/docs/Makefile new file mode 100644 index 0000000000000000000000000000000000000000..8bda904a9b0b29dfcf538cb52b806dd910710a4a --- /dev/null +++ b/frameworks/agentgym_rl/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +SPHINXPROJ = verl +SOURCEDIR = . +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/frameworks/agentgym_rl/docs/README.md b/frameworks/agentgym_rl/docs/README.md new file mode 100644 index 0000000000000000000000000000000000000000..8d411bea0e25c2dce47685341a161178831b62c7 --- /dev/null +++ b/frameworks/agentgym_rl/docs/README.md @@ -0,0 +1,19 @@ +# verl documents + +## Build the docs + +```bash +# Install dependencies. +pip install -r requirements-docs.txt + +# Build the docs. +make clean +make html +``` + +## Open the docs with your browser + +```bash +python -m http.server -d _build/html/ +``` +Launch your browser and open localhost:8000. \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/README_vllm0.7.md b/frameworks/agentgym_rl/docs/README_vllm0.7.md new file mode 100644 index 0000000000000000000000000000000000000000..d2259804a822a51abfe7c1d552f5438493b6c2e2 --- /dev/null +++ b/frameworks/agentgym_rl/docs/README_vllm0.7.md @@ -0,0 +1,65 @@ +# Readme for verl(vllm>=0.7) version + +## Installation + +Note: This version of veRL supports **FSDP** for training and **vLLM** for rollout. (Megatron-LM is not supported yet.) + +``` +# Create the conda environment +conda create -n verl python==3.10 +conda activate verl + +# Install verl +git clone https://github.com/volcengine/verl.git +cd verl +pip3 install -e . + +# Install vLLM>=0.7 +# (Option1) pip3 install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly +# (Option2) pip3 install "vllm>=0.7.0" + +# Install flash-attn +pip3 install flash-attn --no-build-isolation + +``` + +Note that if you are installing stable versions of vLLM (Option2), you need to make some tiny patches manually on vllm (/path/to/site-packages/vllm after installation) after the above steps: + +- vllm/distributed/parallel_state.py: Remove the assertion below: + +``` +if (world_size + != tensor_model_parallel_size * pipeline_model_parallel_size): + raise RuntimeError( + f"world_size ({world_size}) is not equal to " + f"tensor_model_parallel_size ({tensor_model_parallel_size}) x " + f"pipeline_model_parallel_size ({pipeline_model_parallel_size})") + +``` + +- vllm/executor/uniproc_executor.py: change `local_rank = rank` to `local_rank = int(os.environ["LOCAL_RANK"])` +- vllm/model_executor/model_loader/weight_utils.py: remove the `torch.cuda.empty_cache()` in `pt_weights_iterator` + +These modifications have already been merged into the main branch of vLLM. Thus nightly vLLM or building vLLM from source do not need these patches. + +## Features + +### Use cuda graph + +After installation, examples using FSDP as training backends can be used. By default, the `enforce_eager` is set to True, which disables the cuda graph. To enjoy cuda graphs and the sleep mode of vLLM>=0.7, add the following lines to the bash script: + +``` +actor_rollout_ref.rollout.enforce_eager=False \ +actor_rollout_ref.rollout.free_cache_engine=False \ + +``` + +For a typical job like examples/ppo_trainer/run_qwen2-7b_seq_balance.sh, the rollout generation time is 115 seconds with vLLM0.6.3, while it is 85 seconds with vLLM0.7.0. By enabling the cudagraph, the generation duration is further reduced to 62 seconds. + +**Note:** Currently, if the `n` is greater than 1 in `SamplingParams` in vLLM>=0.7, there is a potential performance issue on the stability of rollout generation time (Some iterations would see generation time bursts). We are working with the vLLM team to check this issue. + +### Other features in vLLM + +1. **num_scheduler_step>1:** not supported yet (weight loading has not been aligned with `MultiStepModelRunner`) +2. **Prefix caching:** not supported yet (vLLM sleep mode does not support prefix caching) +3. **Chunked prefill:** supported \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/_static/logo.png b/frameworks/agentgym_rl/docs/_static/logo.png new file mode 100644 index 0000000000000000000000000000000000000000..6a3b61308d73eb72071eaec8c95544ab36cd3970 Binary files /dev/null and b/frameworks/agentgym_rl/docs/_static/logo.png differ diff --git a/frameworks/agentgym_rl/docs/advance/dpo_extension.rst b/frameworks/agentgym_rl/docs/advance/dpo_extension.rst new file mode 100644 index 0000000000000000000000000000000000000000..8ceb4d7f88bbe448ab769a6807f2ca3c6627267c --- /dev/null +++ b/frameworks/agentgym_rl/docs/advance/dpo_extension.rst @@ -0,0 +1,271 @@ +Extend to other RL(HF) algorithms +================================= + +We already implemented the complete training pipeline of the PPO +algorithms. To extend to other algorithms, we analyze the high-level +principle to use verl and provide a tutorial to implement the DPO +algorithm. Users can follow the similar paradigm to extend to other RL algorithms. + +.. note:: **Key ideas**: Single process drives multi-process computation and data communication. + +Overall Approach +---------------- + +Step 1: Consider what multi-machine multi-GPU computations are needed +for each model, such as ``generate_sequence`` , ``compute_log_prob`` and +``update_policy`` in the actor_rollout model. Implement distributed +single-process-multiple-data (SPMD) computation and encapsulate them +into APIs + +Step 2: Based on different distributed scenarios, including FSDP and 3D +parallelism in Megatron-LM, implement single-process control of data +interaction among multi-process computations. + +Step 3: Utilize the encapsulated APIs to implement the control flow + +Example: Online DPO +------------------- + +We use verl to implement a simple online DPO algorithm. The algorithm +flow of Online DPO is as follows: + +1. There is a prompt (rollout) generator which has the same weight as + the actor model. After a batch of prompts are fed into the generator, + it generates N responses for each prompt. +2. Send all the prompts + responses to a verifier for scoring, which can + be reward model or a rule-based function. Then sort them in pairs to + form a training batch. +3. Use this training batch to train the actor model using DPO. During + the process, a reference policy is needed. + +Step 1: What are the multi-machine multi-GPU computations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Sample Generator** + +Implementation details: + +.. code:: python + + from verl.single_controller.base import Worker + from verl.single_controller.ray import RayWorkerGroup, RayClassWithInitArgs, RayResourcePool + import ray + + @ray.remote + class SampleGenerator(Worker): + def __init__(self, config): + super().__init__() + self.config = config + + def generate_sequences(self, data): + pass + +Here, ``SampleGenerator`` can be viewed as a multi-process pulled up by +``torchrun``, with each process running the same code (SPMD). +``SampleGenerator`` needs to implement a ``generate_sequences`` API for +the control flow to call. The implementation details inside can use any +inference engine including vllm, sglang and huggingface. Users can +largely reuse the code in +verl/verl/trainer/ppo/rollout/vllm_rollout/vllm_rollout.py and we won't +go into details here. + +**ReferencePolicy inference** + +API: compute reference log probability + +.. code:: python + + from verl.single_controller.base import Worker + import ray + + @ray.remote + class ReferencePolicy(Worker): + def __init__(self): + super().__init__() + self.model = Model() + + def infer(self, data): + return self.model(data) + +**Actor update** + +API: Update actor model parameters + +.. code:: python + + from verl.single_controller.base import Worker + import ray + + @ray.remote + class DPOActor(Worker): + def __init__(self): + super().__init__() + self.model = Model() + self.model = FSDP(self.model) # or other distributed strategy + self.optimizer = optim.Adam(self.model.parameters(), lr=1e-3) + self.loss_fn = xxx + + def update(self, data): + self.optimizer.zero_grad() + logits = self.model(data) + loss = self.loss_fn(logits) + loss.backward() + self.optimizer.step() + +**Notes: How to distinguish between control processes and distributed computation processes** +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Control processes are generally functions directly decorated with + ``@ray.remote`` +- Computation processes are all wrapped into a ``RayWorkerGroup``. + +Users can reuse most of the distribtued computation logics implemented +in PPO algorithm, including FSDP and Megatron-LM backend in +verl/verl/trainer/ppo. + +Step 2: Based on different distributed scenarios, implement single-process control of multi-process data interaction +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**The core problem to solve here is how a single process sends data to +multiple processes, drives multi-process computation, and how the +control process obtains the results of multi-process computation.** +First, we initialize the multi-process ``WorkerGroup`` in the control +process. + +.. code:: python + + @ray.remote(num_cpus=1) + def main_task(config): + # construct SampleGenerator + resource_pool = RayResourcePool(process_on_nodes=[8] * 2) # 16 GPUs + ray_cls = RayClassWithInitArgs(SampleGenerator, config=config) + # put SampleGenerator onto resource pool + worker_group = RayWorkerGroup(resource_pool, ray_cls) + + # construct reference policy + +As we can see, in the control process, multiple processes are wrapped +into a ``RayWorkerGroup``. Inside this ``WorkerGroup``, there is a +``self._workers`` member, where each worker is a RayActor +(https://docs.ray.io/en/latest/ray-core/actors.html) of SampleGenerator. +ray_trainer.md also provide an implementation of +``MegatronRayWorkerGroup``. + +Assuming the model is distributed using FSDP, and there is a batch of +data on the control process, for data parallelism, the underlying +calling process is: + +.. code:: python + + data = xxx + data_list = data.chunk(dp_size) + + output = [] + for d in data_list: + # worker_group._workers[i] is a SampleGenerator + output.append(worker_group._workers[i].generate_sequences.remote(d)) + + output = ray.get(output) + output = torch.cat(output) + +Single process calling multiple processes involves the following 3 +steps: + +1. Split the data into DP parts on the control process. +2. Send the data to remote, call the remote computation through RPC, and + utilize multi-process computation. +3. Obtain the computation results of each worker on the control process + and merge them. + +Frequently calling these 3 steps on the controller process greatly hurts +code readability. **In verl, we have abstracted and encapsulated these 3 +steps, so that the worker's method + dispatch + collect can be +registered into the worker_group** + +.. code:: python + + from verl.single_controller.base.decorator import register + + def dispatch_data(worker_group, data): + return data.chunk(worker_group.world_size) + + def collect_data(worker_group, data): + return torch.cat(data) + + dispatch_mode = { + 'dispatch_fn': dispatch_data, + 'collect_fn': collect_data + } + + @register(dispatch_mode=dispatch_mode) + def generate_sequences(self, data): + pass + +In this way, we can directly call the method inside the worker through +the ``worker_group`` on the control (driver) process (which is a single +process): + +.. code:: python + + output = worker_group.generate_sequences(data) + +This single line includes data splitting, data distribution and +computation, and data collection. + +Furthermore, the model parallelism size of each model is usually fixed, +including dp, tp, pp. So for these common distributed scenarios, we have +pre-implemented specific dispatch and collect methods,in `decorator.py `_, which can be directly used to wrap the computations. + +.. code:: python + + from verl.single_controller.base.decorator import register, Dispatch + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def generate_sequences(self, data: DataProto) -> DataProto: + pass + +Here it requires the data interface to be ``DataProto``. Definition of +``DataProto`` is in `protocol.py `_. + +Step 3: Main training loop +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +With the above training flows, we can implement the algorithm's control +flow. It is recommended that ``main_task`` is also a ray remote process. + +.. code:: python + + @ray.remote(num_cpus=1) + def main_task(config): + # construct SampleGenerator + resource_pool = RayResourcePool(process_on_nodes=[8] * 2) # 16 GPUs + ray_cls = RayClassWithInitArgs(SampleGenerator, config=config) + # put SampleGenerator onto resource pool + sample_gen = RayWorkerGroup(resource_pool, ray_cls) + + # construct reference policy + ray_cls = RayClassWithInitArgs(ReferencePolicy) + ref_policy = RayWorkerGroup(resource_pool, ray_cls) + + # construct actor + ray_cls = RayClassWithInitArgs(DPOActor) + dpo_policy = RayWorkerGroup(resource_pool, ray_cls) + + dataloader = DataLoader() + + for data in dataloader: + # generate data + data = sample_gen.generate_sequences(data) + # generate scores for each data + data = generate_scores(data) + # generate pairwise data using scores + data = generate_pairwise_data(data) + # generate ref_log_prob + data.batch['ref_log_prob'] = ref_policy.infer(data) + # update using dpo + dpo_policy.update(data) + # logging + +Here, different ``WorkerGroups`` can be placed in the same resource pool or +in different resource pools using ``create_colocated_worker_cls`` +similar as in `ray_trainer.py `_. diff --git a/frameworks/agentgym_rl/docs/advance/fsdp_extension.rst b/frameworks/agentgym_rl/docs/advance/fsdp_extension.rst new file mode 100644 index 0000000000000000000000000000000000000000..c708bf25012525b5865effa826dc835759edc6c6 --- /dev/null +++ b/frameworks/agentgym_rl/docs/advance/fsdp_extension.rst @@ -0,0 +1,95 @@ + +Add models with the FSDP backend +================================== + +Model +-------------------------- + +In principle, our FSDP backend can support any HF model and we can +sychronoize the actor model weight with vLLM using `hf_weight_loader.py `_. +However, ``hf_weight_loader`` is will gather the full state_dict of a +model during synchronization, which may cause OOM. We suggest using +``dtensor_weight_loader`` which gather the full model parameter layer by +layer to reduce the peak memory usage. We already support dtensor weight +loader for the models below in `dtensor_weight_loader.py `_.: + +- ``GPT2LMHeadModel`` +- ``LlamaForCausalLM`` +- ``LLaMAForCausalLM`` +- ``MistralForCausalLM`` +- ``InternLMForCausalLM`` +- ``AquilaModel`` +- ``AquilaForCausalLM`` +- ``Phi3ForCausalLM`` +- ``GemmaForCausalLM`` +- ``Gemma2ForCausalLM`` +- ``GPTBigCodeForCausalLM`` +- ``Starcoder2ForCausalLM`` +- ``Qwen2ForCausalLM`` +- ``DeepseekV2ForCausalLM`` + +To implement ``dtensor_weight_loader`` of a model that's supported in +vLLM, follow the guide of gemma model below: + +1. Copy the + ``load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]])`` from the vllm model class + to ``dtensor_weight_loaders.py`` +2. Modify the arguments to + ``(actor_weights: Dict, vllm_model: nn.Module)`` +3. Replace the ``self`` to ``vllm_model`` +4. Add the + ``local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)`` + before each ``param = params_dict[name]`` and modify the following + weight loading using ``local_loaded_weight``. +5. Register the implemented dtensor weight loader to ``__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__``. + +.. code-block:: diff + + - def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]): + + def gemma_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + - params_dict = dict(self.named_parameters()) + + params_dict = dict(vllm_model.named_parameters()) + loaded_params = set() + - for name, loaded_weight in weights: + + for name, loaded_weight in actor_weights.items(): + for (param_name, shard_name, shard_id) in stacked_params_mapping: + if shard_name not in name: + continue + name = name.replace(shard_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + - weight_loader(param, loaded_weight, shard_id) + + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # lm_head is not used in vllm as it is tied with embed_token. + # To prevent errors, skip loading lm_head.weight. + if "lm_head.weight" in name: + continue + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", + default_weight_loader) + - weight_loader(param, loaded_weight) + + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + loaded_params.add(name) + unloaded_params = params_dict.keys() - loaded_params + if unloaded_params: + raise RuntimeError( + "Some weights are not initialized from checkpoints: " + f"{unloaded_params}") \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/advance/megatron_extension.rst b/frameworks/agentgym_rl/docs/advance/megatron_extension.rst new file mode 100644 index 0000000000000000000000000000000000000000..91d744a8a9d21af126d00014fdbbc1368d04a75c --- /dev/null +++ b/frameworks/agentgym_rl/docs/advance/megatron_extension.rst @@ -0,0 +1,26 @@ +Add models with the Megatron-LM backend +========================================= + +Model +----------- + +The most challenging aspect to use the Megatron-LM backend is implementing +the models for training. Currently, we implement Llama model that +support data parallelism, tensor parallelism, pipeline parallelism (also +vPP) and sequence parallelism. We also implement remove padding (sequence packing) on Llama +model, which can be found in `modeling_llama_megatron.py `_. + +To support other model, users are required to implement: + +1. Implemnt a model similar to ``modeling_llama_megatron.py`` that satisfy the + parallelism requirements of Megatron-LM. Then register your model in + the `registry.py `_. +2. Checkpoint utils that can load full checkpoint (e.g. huggingface + checkpoint) to partitioned models during the runtime. Then register + your loader to ``weight_loader_registry`` in `weight_loader_registry.py `_. +3. Weight loader that synchronize the weight from Megatron to rollout + (vLLM) model. Note that both the actor model and rollout model are + partitioned during runtime. So, it's advisable to map the model name + in actor model implementation. Otherwise, you may need an additional + name mapping and even weight transformation. The weight loader implementation + is in `megatron_weight_loaders.py `_. \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/advance/placement.rst b/frameworks/agentgym_rl/docs/advance/placement.rst new file mode 100644 index 0000000000000000000000000000000000000000..a98caa116eef8e9d5db5f92756d755ce9d1142ce --- /dev/null +++ b/frameworks/agentgym_rl/docs/advance/placement.rst @@ -0,0 +1,11 @@ +Ray API Design Tutorial +======================================= + +We provide a tutorial for our Ray API design, including: + +- Ray basic concepts +- Resource Pool and RayWorkerGroup +- Data Dispatch, Execution and Collection +- Initialize the RayWorkerGroup and execute the distributed computation in the given Resource Pool + +See details in `tutorial.ipynb `_. \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/conf.py b/frameworks/agentgym_rl/docs/conf.py new file mode 100644 index 0000000000000000000000000000000000000000..ed63e6a67b936faf73c3790a485fb97a1f7265a4 --- /dev/null +++ b/frameworks/agentgym_rl/docs/conf.py @@ -0,0 +1,81 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = u'verl' +# pylint: disable=W0622 +copyright = u'2024 ByteDance Seed Foundation MLSys Team' +author = u'Guangming Sheng, Chi Zhang, Yanghua Peng, Haibin Lin' + + +# -- General configuration --------------------------------------------------- +# The master toctree document. +master_doc = 'index' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = ['recommonmark', + 'sphinx.ext.autosectionlabel', +] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +source_suffix = ['.rst', 'rest', '.md'] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = u'en' + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'sphinx_rtd_theme' + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/examples/config.rst b/frameworks/agentgym_rl/docs/examples/config.rst new file mode 100644 index 0000000000000000000000000000000000000000..e8ea05889b27dc8899c372668cf2706e8e5ac1a6 --- /dev/null +++ b/frameworks/agentgym_rl/docs/examples/config.rst @@ -0,0 +1,365 @@ +.. _config-explain-page: + +Config Explanation +=================== + +ppo_trainer.yaml for FSDP Backend +--------------------------------- + +Data +~~~~ + +.. code:: yaml + + data: + tokenizer: null + train_files: ~/data/rlhf/gsm8k/train.parquet + val_files: ~/data/rlhf/gsm8k/test.parquet + prompt_key: prompt + max_prompt_length: 512 + max_response_length: 512 + train_batch_size: 1024 + val_batch_size: 1312 + return_raw_input_ids: False # This should be set to true when the tokenizer between policy and rm differs + return_raw_chat: False + +- ``data.train_files``: Training set parquet. Can be a list or a single + file. The program will read all files into memory, so it can't be too + large (< 100GB). The path can be either local path or HDFS path. For + HDFS path, we provide utils to download it to DRAM and convert the + HDFS path to local path. +- ``data.val_files``: Validation parquet. Can be a list or a single + file. +- ``data.prompt_key``: The field in the dataset where the prompt is + located. Default is 'prompt'. +- ``data.max_prompt_length``: Maximum prompt length. All prompts will be + left-padded to this length. An error will be reported if the length is + too long +- ``data.max_response_length``: Maximum response length. Rollout in RL + algorithms (e.g. PPO) generates up to this length +- ``data.train_batch_size``: Batch size sampled for one training + iteration of different RL algorithms. +- ``data.val_batch_size``: Batch size sampled for one validation + iteration. +- ``data.return_raw_input_ids``: Whether to return the original + input_ids without adding chat template. This is mainly used to + accommodate situations where the reward model's chat template differs + from the policy. It needs to be decoded first, then apply the RM's + chat template. If using a model-based RM, and the policy and RM + chat_templates are different, this flag needs to be set +- ``data.return_raw_chat``: +- ``data.truncation``: Truncate the input_ids or prompt length if they + exceed max_prompt_length. Default is 'error', not allow exceed the + max_prompt_length. The users should increase the max_prompt_length if + throwing the error. + +Actor/Rollout/Reference Policy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: yaml + + actor_rollout_ref: + hybrid_engine: True + model: + path: ~/models/deepseek-llm-7b-chat + external_lib: null + override_config: { } + enable_gradient_checkpointing: False + use_remove_padding: False + actor: + strategy: fsdp # This is for backward-compatibility + ppo_mini_batch_size: 256 + ppo_micro_batch_size: null # will be deprecated, use ppo_micro_batch_size_per_gpu + ppo_micro_batch_size_per_gpu: 8 + use_dynamic_bsz: False + ppo_max_token_len_per_gpu: 16384 # n * ${data.max_prompt_length} + ${data.max_response_length} + grad_clip: 1.0 + clip_ratio: 0.2 + entropy_coeff: 0.001 + use_kl_loss: False # True for GRPO + kl_loss_coef: 0.001 # for grpo + kl_loss_type: low_var_kl # for grpo + ppo_epochs: 1 + shuffle: False + ulysses_sequence_parallel_size: 1 # sp size + optim: + lr: 1e-6 + lr_warmup_steps_ratio: 0. # the total steps will be injected during runtime + min_lr_ratio: null # only useful for warmup with cosine + warmup_style: constant # select from constant/cosine + total_training_steps: -1 # must be override by program + fsdp_config: + wrap_policy: + # transformer_layer_cls_to_wrap: None + min_num_params: 0 + param_offload: False + grad_offload: False + optimizer_offload: False + fsdp_size: -1 + ref: + fsdp_config: + param_offload: False + wrap_policy: + # transformer_layer_cls_to_wrap: None + min_num_params: 0 + log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu + log_prob_micro_batch_size_per_gpu: 16 + log_prob_use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz} + log_prob_max_token_len_per_gpu: ${actor_rollout_ref.actor.ppo_max_token_len_per_gpu} + ulysses_sequence_parallel_size: ${actor_rollout_ref.actor.ulysses_sequence_parallel_size} # sp size + rollout: + name: vllm + temperature: 1.0 + top_k: -1 # 0 for hf rollout, -1 for vllm rollout + top_p: 1 + prompt_length: ${data.max_prompt_length} # not use for opensource + response_length: ${data.max_response_length} + # for vllm rollout + dtype: bfloat16 # should align with FSDP + gpu_memory_utilization: 0.5 + ignore_eos: False + enforce_eager: True + free_cache_engine: True + load_format: dummy_dtensor + tensor_model_parallel_size: 2 + max_num_batched_tokens: 8192 + max_num_seqs: 1024 + log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu + log_prob_micro_batch_size_per_gpu: 16 + log_prob_use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz} + log_prob_max_token_len_per_gpu: ${actor_rollout_ref.actor.ppo_max_token_len_per_gpu} + # for hf rollout + do_sample: True + # number of responses (i.e. num sample times) + n: 1 # > 1 for grpo + +**Common config for actor, rollout and reference model** + +- ``actor_rollout_ref.hybrid_engine``: Whether it's a hybrid engine, + currently only supports hybrid engine +- ``actor_rollout_ref.model.path``: Huggingface model path. This can be + either local path or HDFS path. For HDFS path, we provide utils to + download it to DRAM and convert the HDFS path to local path. +- ``actor_rollout_ref.model.external_libs``: Additional Python packages + that need to be imported. Used to register models or tokenizers into + the Huggingface system. +- ``actor_rollout_ref.model.override_config``: Used to override some of + the model's original configurations, mainly dropout +- ``actor_rollout_ref.model.enable_gradient_checkpointing``: Whether to + enable gradient checkpointing for the actor + +**Actor model** + +- ``actor_rollout_ref.actor.strategy``: fsdp or megatron. In this + example, we use fsdp backend. + +- ``actor_rollout_ref.actor.ppo_mini_batch_size``: One sample is split + into multiple sub-batches with batch_size=ppo_mini_batch_size for PPO + updates. The ppo_mini_batch_size is a global num across all workers/gpus + +- ``actor_rollout_ref.actor.ppo_micro_batch_size``: [Will be deprecated, use ppo_micro_batch_size_per_gpu] + Similar to gradient accumulation, the micro_batch_size_per_gpu for one forward pass, + trading speed for GPU memory. The value represent the global view. + +- ``actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu``: Similar to gradient + accumulation, the micro_batch_size_per_gpu for one forward pass, trading speed + for GPU memory. The value represent the local num per gpu. + +- ``actor_rollout_ref.actor.grad_clip``: Gradient clipping for actor + updates + +- ``actor_rollout_ref.actor.clip_ratio``: PPO clip ratio + +- ``actor_rollout_ref.actor.entropy_coeff``: The weight of entropy when + calculating PPO loss + +- ``actor_rollout_ref.actor.ppo_epochs``: Number of epochs for PPO + updates on one set of sampled data + +- ``actor_rollout_ref.actor.shuffle``: Whether to shuffle data when + there are multiple epochs + +- ``actor_rollout_ref.actor.optim``: Actor's optimizer parameters + +- ``actor_rollout_ref.actor.fsdp_config``: FSDP config for actor + training + + - ``wrap_policy``: FSDP wrap policy. By default, it uses Huggingface's + wrap policy, i.e., wrapping by DecoderLayer + + - No need to set transformer_layer_cls_to_wrap, so we comment it. + + - ``*_offload``: Whether to enable parameter, gradient and optimizer + offload + + - Trading speed for GPU memory. + +**Reference Model** + +- ``actor_rollout_ref.ref``: FSDP config same as actor. **For models + larger than 7B, it's recommended to turn on offload for ref by + default** + +- ``actor_rollout_ref.ref.log_prob_micro_batch_size``: [Will be deprecate, use log_prob_micro_batch_size_per_gpu] + The batch size for one forward pass in the computation of ``ref_log_prob``. The value represent the global num. + +- ``actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu``: The batch size + for one forward pass in the computation of ``ref_log_prob``. The value represent the local num per gpu. + +**Rollout Model** + +- ``actor_rollout_ref.rollout.name``: hf/vllm. We use vLLM by default + because it's much efficient and our hybrid engine is implemented with + vLLM. + +- Rollout (Auto-regressive) parameters. The key should be equal to the + property name in vLLM's ``SamplingParams``. + + - ``temperature``, ``top_k``, ``top_p`` and others: Sampling + parameters in ``SamplingParams``. + +- ``dtype``: Rollout model parameters type. This should be align with + the actor model parameter type in FSDP/Megatron backend. + +- ``gpu_memory_utilization``: The proportion of the remaining GPU memory + allocated for kv cache after other models have initialized when using + vllm. + +- ``tensor_model_parallel_size``: TP size for rollout. Only effective + for vllm. + +- ``actor_rollout_ref.ref.log_prob_micro_batch_size``: [Will be deprecate, use log_prob_micro_batch_size_per_gpu] + The batch size for one forward pass in the computation of ``log_prob``. The value represent the global num. + +- ``log_prob_micro_batch_size_per_gpu``: Micro batch size per gpu (The batch size for + one forward pass) for recalculating ``log_prob``. The value represent the local num per gpu. + +- ``do_sample``: Whether to sample. If set to False, the rollout model + will perform greedy sampling. We disable ``do_sample`` during + validation. + +- ``actor_rollout_ref.rollout.ignore_eos``: Whether to ignore the EOS + token and continue generating tokens after the EOS token is generated. + +- ``actor_rollout_ref.rollout.free_cache_engine``: Offload the KVCache + after rollout generation stage. Default is True. When set to True, we + need to disable the usage of CUDAGraph (set ``enforce_eager`` to + True.) + +- ``actor_rollout_ref.rollout.enforce_eager``: Whether to use CUDAGraph + in vLLM generation. Default set to True to disable CUDAGraph. + +- ``actor_rollout_ref.rollout.load_format``: Which weight loader to use + to load the actor model weights to the rollout model. + + - ``auto``: Use Megatron weight loader. + - ``megatron``: Use Megatron weight loader. Deployed with Megatron + backend. The input model ``state_dict()`` is already partitioned + along TP dimension and already gathered along PP dimension. This + weight loader requires that the Rollout model and Actor model's + parameters shape and name should be identical. + - ``dtensor``: Default solution when using Huggingface weight loader. + Deployed with FSDP backend and the state_dict_type is + ``StateDictType.SHARDED_STATE_DICT``. Recommend to use this weight + loader + - ``hf``: Use Huggingface weight loader. Deployed with FSDP backend + and the state_dict_type is ``StateDictType.FULL_STATE_DICT``. This + solution doesn't need to rewrite the weight loader for each model + implemented in vLLM but it results in larger peak memory usage. + - ``dummy_hf``, ``dummy_megatron``, ``dummy_dtensor``: Random + initialization. + +.. note:: **NOTED**: In this config field, users only need to select from ``dummy_megatron``, ``dummy_dtensor``, ``dummy_hf`` for rollout initialization and our hybrid engine will select the corresponding weight loader (i.e., ``megatron``, ``dtensor``, ``hf``) during actor/rollout weight synchronization. + +Critic Model +~~~~~~~~~~~~ + +Most parameters for Critic are similar to Actor Model. + +Reward Model +~~~~~~~~~~~~ + +.. code:: yaml + + reward_model: + enable: False + model: + input_tokenizer: ${actor_rollout_ref.model.path} # set this to null if the chat template is identical + path: ~/models/Anomy-RM-v0.1 + external_lib: ${actor_rollout_ref.model.external_lib} + fsdp_config: + min_num_params: 0 + param_offload: False + micro_batch_size_per_gpu: 16 + max_length: null + reward_manager: naive + +- ``reward_model.enable``: Whether to enable reward model. If False, we + compute the reward only with the user-defined reward functions. In + GSM8K and Math examples, we disable reward model. For RLHF alignment + example using full_hh_rlhf, we utilize reward model to assess the + responses. If False, the following parameters are not effective. +- ``reward_model.model`` + + - ``input_tokenizer``: Input tokenizer. If the reward model's chat + template is inconsistent with the policy, we need to first decode to + plaintext, then apply the rm's chat_template. Then score with RM. If + chat_templates are consistent, it can be set to null. + - ``path``: RM's HDFS path or local path. Note that RM only supports + AutoModelForSequenceClassification. Other model types need to define + their own RewardModelWorker and pass it from the code. +- ``reward_model.reward_manager``: Reward Manager. This defines the mechanism + of computing rule-based reward and handling different reward sources. Default + if ``naive``. If all verification functions are multiprocessing-safe, the reward + manager can be set to ``prime`` for parallel verification. + +Algorithm +~~~~~~~~~ + +.. code:: yaml + + algorithm: + gamma: 1.0 + lam: 1.0 + adv_estimator: gae + kl_penalty: kl # how to estimate kl divergence + kl_ctrl: + type: fixed + kl_coef: 0.005 + +- ``gemma``: discount factor +- ``lam``: Trade-off between bias and variance in the GAE estimator +- ``adv_estimator``: Support ``gae``, ``grpo``, ``reinforce_plus_plus``. +- ``kl_penalty``: Support ``kl``, ``abs``, ``mse`` and ``full``. How to + calculate the kl divergence between actor and reference policy. For + specific options, refer to `core_algos.py `_ . + +Trainer +~~~~~~~ + +.. code:: yaml + + trainer: + total_epochs: 30 + project_name: verl_examples + experiment_name: gsm8k + logger: ['console', 'wandb'] + nnodes: 1 + n_gpus_per_node: 8 + save_freq: -1 + test_freq: 2 + critic_warmup: 0 + default_hdfs_dir: ~/experiments/gsm8k/ppo/${trainer.experiment_name} # hdfs checkpoint path + default_local_dir: checkpoints/${trainer.project_name}/${trainer.experiment_name} # local checkpoint path + +- ``trainer.total_epochs``: Number of epochs in training. +- ``trainer.project_name``: For wandb +- ``trainer.experiment_name``: For wandb +- ``trainer.logger``: Support console and wandb +- ``trainer.nnodes``: Number of nodes used in the training. +- ``trainer.n_gpus_per_node``: Number of GPUs per node. +- ``trainer.save_freq``: The frequency (by iteration) to save checkpoint + of the actor and critic model. +- ``trainer.test_freq``: The validation frequency (by iteration). +- ``trainer.critic_warmup``: The number of iteration to train the critic + model before actual policy learning. diff --git a/frameworks/agentgym_rl/docs/examples/gsm8k_example.rst b/frameworks/agentgym_rl/docs/examples/gsm8k_example.rst new file mode 100644 index 0000000000000000000000000000000000000000..2804da0cb7105808ed19f04cfebc9e277246781f --- /dev/null +++ b/frameworks/agentgym_rl/docs/examples/gsm8k_example.rst @@ -0,0 +1,168 @@ +GSM8K Example +============= + +Introduction +------------ + +In this example, we train an LLM to tackle the GSM8k task. + +Paper: https://arxiv.org/pdf/2110.14168 + +Dataset: https://huggingface.co/datasets/gsm8k + +Note that the original paper mainly focuses on training a verifier (a +reward model) to solve math problems via Best-of-N sampling. In this +example, we train an RLHF agent using a rule-based reward model. + +Dataset Introduction +-------------------- + +GSM8k is a math problem dataset. The prompt is an elementary school +problem. The LLM model is required to answer the math problem. + +The training set contains 7473 samples and the test set contains 1319 +samples. + +**An example** + +Prompt + + Katy makes coffee using teaspoons of sugar and cups of water in the + ratio of 7:13. If she used a total of 120 teaspoons of sugar and cups + of water, calculate the number of teaspoonfuls of sugar she used. + +Solution + + The total ratio representing the ingredients she used to make the + coffee is 7+13 = <<7+13=20>>20 Since the fraction representing the + number of teaspoons she used is 7/20, she used 7/20\ *120 = + <<7/20*\ 120=42>>42 #### 42 + +Step 1: Prepare dataset +----------------------- + +.. code:: bash + + cd examples/data_preprocess + python3 gsm8k.py --local_dir ~/data/gsm8k + +Step 2: Download Model +---------------------- + +There're three ways to prepare the model checkpoints for post-training: + +- Download the required models from hugging face + +.. code:: bash + + huggingface-cli download deepseek-ai/deepseek-math-7b-instruct --local-dir ~/models/deepseek-math-7b-instruct --local-dir-use-symlinks False + +- Already store your store model in the local directory or HDFS path. +- Also, you can directly use the model name in huggingface (e.g., + deepseek-ai/deepseek-math-7b-instruct) in + ``actor_rollout_ref.model.path`` and ``critic.model.path`` field in + the run script. + +Noted that users should prepare checkpoints for actor, critic and reward +model. + +[Optional] Step 3: SFT your Model +--------------------------------- + +We provide a SFT Trainer using PyTorch FSDP in +`fsdp_sft_trainer.py `_. +Users can customize their own SFT +script using our FSDP SFT Trainer. + +We also provide various training scripts for SFT on GSM8K dataset in `gsm8k sft directory `_. + +.. code:: shell + + set -x + + torchrun -m verl.trainer.fsdp_sft_trainer \ + data.train_files=$HOME/data/gsm8k/train.parquet \ + data.val_files=$HOME/data/gsm8k/test.parquet \ + data.prompt_key=question \ + data.response_key=answer \ + data.micro_batch_size_per_gpu=8 \ + model.partial_pretrain=deepseek-ai/deepseek-coder-6.7b-instruct \ + trainer.default_hdfs_dir=hdfs://user/verl/experiments/gsm8k/deepseek-coder-6.7b-instruct/ \ + trainer.project_name=gsm8k-sft \ + trainer.experiment_name=gsm8k-sft-deepseek-coder-6.7b-instruct \ + trainer.total_epochs=4 \ + trainer.logger=['console','wandb'] + +Step 4: Perform PPO training with your model on GSM8K Dataset +------------------------------------------------------------- + +- Prepare your own run.sh script. Here's an example for GSM8k dataset + and deepseek-llm-7b-chat model. +- Users could replace the ``data.train_files`` ,\ ``data.val_files``, + ``actor_rollout_ref.model.path`` and ``critic.model.path`` based on + their environment. +- See :doc:`config` for detailed explanation of each config field. + +**Reward Model/Function** + +We use a rule-based reward model. We force the model to produce a final +answer following 4 “#” as shown in the solution. We extract the final +answer from both the solution and model's output using regular +expression matching. We compare them and assign a reward of 1 to correct +answer, 0.1 to incorrect answer and 0 to no answer. + +**Training Script** + +The training script example for FSDP and Megatron-LM backend are stored in examples/ppo_trainer directory. + +.. code:: bash + + cd ../ppo_trainer + bash run_deepseek7b_llm.sh + +The script of run_deepseek7b_llm.sh + +.. code:: bash + + set -x + + python3 -m verl.trainer.main_ppo \ + data.train_files=$HOME/data/gsm8k/train.parquet \ + data.val_files=$HOME/data/gsm8k/test.parquet \ + data.train_batch_size=1024 \ + data.val_batch_size=1312 \ + data.max_prompt_length=512 \ + data.max_response_length=512 \ + actor_rollout_ref.model.path=deepseek-ai/deepseek-llm-7b-chat \ + actor_rollout_ref.actor.optim.lr=1e-6 \ + actor_rollout_ref.model.use_remove_padding=True \ + actor_rollout_ref.actor.ppo_mini_batch_size=256 \ + actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=16 \ + actor_rollout_ref.actor.fsdp_config.param_offload=False \ + actor_rollout_ref.actor.fsdp_config.grad_offload=False \ + actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \ + actor_rollout_ref.model.enable_gradient_checkpointing=True \ + actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=32 \ + actor_rollout_ref.rollout.tensor_model_parallel_size=4 \ + actor_rollout_ref.rollout.name=vllm \ + actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \ + actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=32 \ + actor_rollout_ref.ref.fsdp_config.param_offload=True \ + critic.optim.lr=1e-5 \ + critic.model.use_remove_padding=True \ + critic.model.path=deepseek-ai/deepseek-llm-7b-chat \ + critic.model.enable_gradient_checkpointing=True \ + critic.ppo_micro_batch_size_per_gpu=32 \ + critic.model.fsdp_config.param_offload=False \ + critic.model.fsdp_config.grad_offload=False \ + critic.model.fsdp_config.optimizer_offload=False \ + algorithm.kl_ctrl.kl_coef=0.001 \ + trainer.critic_warmup=0 \ + trainer.logger=['console','wandb'] \ + trainer.project_name='verl_example_gsm8k' \ + trainer.experiment_name='deepseek_llm_7b_function_rm' \ + trainer.n_gpus_per_node=8 \ + trainer.nnodes=1 \ + trainer.save_freq=-1 \ + trainer.test_freq=1 \ + trainer.total_epochs=15 $@ diff --git a/frameworks/agentgym_rl/docs/examples/ppo_code_architecture.rst b/frameworks/agentgym_rl/docs/examples/ppo_code_architecture.rst new file mode 100644 index 0000000000000000000000000000000000000000..05fb531563806a9a3156833fe2a7aee8ec219200 --- /dev/null +++ b/frameworks/agentgym_rl/docs/examples/ppo_code_architecture.rst @@ -0,0 +1,207 @@ +PPO Example Architecture +======================== + +Let's start with the Proximal Policy Optimization algorithm, which is +most widely used algorithm in LLM post-training. + +The main entry point of the PPO algorithm example is: +`main_ppo.py `_. +In this tutorial, we will go through the code architecture in `main_ppo.py `_. + +Define the data +--------------- + +Users need to preprocess and store the dataset in parquet files. +And we implement `RLHFDataset` to load and tokenize the parquet files. + +For ``RLHFDataset`` (Default), at least 1 fields are required: + +- ``prompt``: Contains the string prompt + +We already provide some examples of processing the datasets to parquet +files in `data_preprocess directory `_. Currently, we support +preprocess of GSM8k, MATH, Hellasage, Full_hh_rlhf datasets. See :doc:`../preparation/prepare_data` for +more information. + +Define the reward functions for different datasets +-------------------------------------------------- + +In this main entry point, the users only need to define their own reward +function based on the datasets (or applications) utilized in PPO +training. + +For example, we already provide reward functions for `GSM8k `_ +and `MATH `_ +datasets in the ``_select_rm_score_fn``. In the ``RewardManager``, we +will compute the reward score based on the data_source to select +corresponding reward functions. For some RLHF datasets (e.g., +full_hh_rlhf), the reward model is utilized to assess the responses +without any reward functions. In this case, the ``RewardManager`` will +return the ``rm_score`` computed by the reward model directly. + +See `reward functions `_ for detailed implementation. + +Define worker classes +--------------------- + +.. code:: python + + if config.actor_rollout_ref.actor.strategy == 'fsdp': # for FSDP backend + assert config.actor_rollout_ref.actor.strategy == config.critic.strategy + from verl.workers.fsdp_workers import ActorRolloutRefWorker, CriticWorker + from verl.single_controller.ray import RayWorkerGroup + ray_worker_group_cls = RayWorkerGroup + + elif config.actor_rollout_ref.actor.strategy == 'megatron': # for Megatron backend + assert config.actor_rollout_ref.actor.strategy == config.critic.strategy + from verl.workers.megatron_workers import ActorRolloutRefWorker, CriticWorker + from verl.single_controller.ray.megatron import NVMegatronRayWorkerGroup + ray_worker_group_cls = NVMegatronRayWorkerGroup # Ray worker class for Megatron-LM + + else: + raise NotImplementedError + + from verl.trainer.ppo.ray_trainer import ResourcePoolManager, Role + + role_worker_mapping = { + Role.ActorRollout: ActorRolloutRefWorker, + Role.Critic: CriticWorker, + Role.RefPolicy: ActorRolloutRefWorker + } + + global_pool_id = 'global_pool' + resource_pool_spec = { + global_pool_id: [config.trainer.n_gpus_per_node] * config.trainer.nnodes, + } + mapping = { + Role.ActorRollout: global_pool_id, + Role.Critic: global_pool_id, + Role.RefPolicy: global_pool_id, + } + +Step 1: Construct the mapping between roles and workers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A role represents a group of workers in the same process. We have +pre-defined several roles in `ray_trainer.py `_. + +.. code:: python + + class Role(Enum): + """ + To create more roles dynamically, you can subclass Role and add new members + """ + Actor = 0 # This worker only has Actor + Rollout = 1 # This worker only has Rollout + ActorRollout = 2 # This worker has both actor and rollout, it's a HybridEngine + Critic = 3 # This worker only has critic + RefPolicy = 4 # This worker only has reference policy + RewardModel = 5 # This worker only has reward model + ActorRolloutRef = 6 # This worker contains actor, rollout and reference policy simultaneously + +Step 2: Define the worker class corresponding to this role +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- We have pre-implemented the ``ActorRolloutRefWorker``. Through + different configs, it can be a standalone actor, a standalone rollout, + an ActorRollout HybridEngine, or an ActorRolloutRef HybridEngine +- We also pre-implemented workers for ``Actor``, ``Rollout``, + ``Critic``, ``Reward Model`` and ``Reference model`` on two different + backend: PyTorch FSDP + and Megatron-LM. + See `FSDP Workers `_ + and `Megatron-LM Workers `_ + for more information. + +Step 3: Define resource pool id and resource pool spec +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Resource pool is a division of global GPU resources, + ``resource_pool_spec`` is a dict, mapping from id to # of GPUs + + - In the above example, we defined a global resource pool: + global_pool_id, and then put all roles on this one resource pool + with all the GPUs in this post-training task. This refers to + *co-locate* placement where all the models share the same set of + GPUs. + +- See resource pool and placement for advance usage. + +Defining reward model/function +------------------------------ + +.. code:: python + + # we should adopt a multi-source reward function here + # - for rule-based rm, we directly call a reward score + # - for model-based rm, we call a model + # - for code related prompt, we send to a sandbox if there are test cases + # - finally, we combine all the rewards together + # - The reward type depends on the tag of the data + if config.reward_model.enable: + from verl.workers.fsdp_workers import RewardModelWorker + role_worker_mapping[Role.RewardModel] = RewardModelWorker + mapping[Role.RewardModel] = global_pool_id + + reward_fn = RewardManager(tokenizer=tokenizer, num_examine=0) + + # Note that we always use function-based RM for validation + val_reward_fn = RewardManager(tokenizer=tokenizer, num_examine=1) + + resource_pool_manager = ResourcePoolManager(resource_pool_spec=resource_pool_spec, mapping=mapping) + +Since not all tasks use model-based RM, users need to define here +whether it's a model-based RM or a function-based RM + +- If it's a model-based RM, directly add the ``RewardModel`` role in the + resource mapping and add it to the resource pool mapping. + + - Note that the pre-defined ``RewardModelWorker`` only supports models + with the structure of huggingface + ``AutoModelForSequenceClassification``. If it's not this model, you + need to define your own RewardModelWorker in `FSDP Workers `_ + and `Megatron-LM Workers `_. + +- If it's a function-based RM, the users are required to classified the + reward function for each datasets. + +.. code:: python + + def _select_rm_score_fn(data_source): + if data_source == 'openai/gsm8k': + return gsm8k.compute_score + elif data_source == 'lighteval/MATH': + return math.compute_score + else: + raise NotImplementedError + +See reward functions implemented in `directory `_ +for more information. + +Define, init and run the PPO Trainer +------------------------------------ + +.. code:: python + + trainer = RayPPOTrainer(config=config, + tokenizer=tokenizer, + role_worker_mapping=role_worker_mapping, + resource_pool_manager=resource_pool_manager, + ray_worker_group_cls=ray_worker_group_cls, + reward_fn=reward_fn, + val_reward_fn=val_reward_fn) + trainer.init_workers() + trainer.fit() + +- We first initialize the ``RayPPOTrainer`` with user config, tokenizer + and all the above worker mapping, resource pool, worker group and + reward functions +- We first call the ``trainer.init_workers()`` to initialize the models + on the allocated GPUs (in the resource pool) +- The actual PPO training will be executed in ``trainer.fit()`` + +verl can be easily extended to other RL algorithms by reusing the Ray +model workers, resource pool and reward functions. See :doc:`extension<../advance/dpo_extension>` for +more information. + +Details of the ``RayPPOTrainer`` is discussed in :doc:`Ray Trainer<../workers/ray_trainer>`. diff --git a/frameworks/agentgym_rl/docs/experiment/ppo.rst b/frameworks/agentgym_rl/docs/experiment/ppo.rst new file mode 100644 index 0000000000000000000000000000000000000000..c977f3cd7dc535c019f096fb2227a36e62c8ff04 --- /dev/null +++ b/frameworks/agentgym_rl/docs/experiment/ppo.rst @@ -0,0 +1,45 @@ +.. _algo-baseline-page: + +Algorithm Baselines +=================== + +GSM8k +------------------ + +Assuming GSM8k dataset is preprocess via ``python3 examples/data_preprocess/gsm8k.py`` + +Refer to the table below to reproduce PPO training from different pre-trained models. + +.. _Huggingface: https://huggingface.co/google/gemma-2-2b-it#benchmark-results +.. _SFT Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/gemma-2-2b-it-sft-0.411.log +.. _SFT+PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/gemma-2-2b-it-ppo-bsz512_4-prompt1024-resp-512-0.640.log +.. _wandb: https://api.wandb.ai/links/verl-team/h7ux8602 +.. _Qwen Blog: https://qwenlm.github.io/blog/qwen2.5-llm/ +.. _PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/Qwen2.5-0.5B-bsz256_2-prompt1024-resp512-0.567.log +.. _Megatron PPO Command and Logs: https://github.com/eric-haibin-lin/verl-data/blob/experiments/gsm8k/deepseek-llm-7b-chat-megatron-bsz256_4-prompt512-resp512-0.695.log +.. _Qwen7b GRPO Script: https://github.com/volcengine/verl/blob/a65c9157bc0b85b64cd753de19f94e80a11bd871/examples/grpo_trainer/run_qwen2-7b_seq_balance.sh +.. _Megatron wandb: https://wandb.ai/verl-team/verl_megatron_gsm8k_examples/runs/10fetyr3 +.. _Qwen7b ReMax Script: https://github.com/eric-haibin-lin/verl/blob/main/examples/remax_trainer/run_qwen2.5-3b_seq_balance.sh +.. _Qwen7b ReMax Wandb: https://wandb.ai/liziniu1997/verl_remax_example_gsm8k/runs/vxl10pln + ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ +| Model | Method | Test score | Details | ++==================================+========================+============+=====================+=========================================================================+ +| google/gemma-2-2b-it | pretrained checkpoint | 23.9 | `Huggingface`_ | ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ +| google/gemma-2-2b-it | SFT | 52.06 | `SFT Command and Logs`_ | ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ +| google/gemma-2-2b-it | SFT + PPO | 64.02 | `SFT+PPO Command and Logs`_, `wandb`_ | ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ +| Qwen/Qwen2.5-0.5B-Instruct | pretrained checkpoint | 36.4 | `Qwen Blog`_ | ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ +| Qwen/Qwen2.5-0.5B-Instruct | PPO | 56.7 | `PPO Command and Logs`_ | ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ +| deepseek-ai/deepseek-llm-7b-chat | PPO | 69.5 [1]_ | `Megatron PPO Command and Logs`_, `Megatron wandb`_ | ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ +| Qwen/Qwen2-7B-Instruct | GRPO | 89 | `Qwen7b GRPO Script`_ | ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ +| Qwen/Qwen2.5-7B-Instruct | ReMax | 97 | `Qwen7b ReMax Script`_, `Qwen7b ReMax Wandb`_ | ++----------------------------------+------------------------+------------+-----------------------------------------------------------------------------------------------+ + +.. [1] During the evaluation, we have only extracted answers following the format "####". A more flexible answer exaction, longer response length and better prompt engineering may lead to higher score. diff --git a/frameworks/agentgym_rl/docs/faq/faq.rst b/frameworks/agentgym_rl/docs/faq/faq.rst new file mode 100644 index 0000000000000000000000000000000000000000..56725834d25edadf21d93bfa69f7ba1186d5eb9c --- /dev/null +++ b/frameworks/agentgym_rl/docs/faq/faq.rst @@ -0,0 +1,21 @@ +Frequently Asked Questions +==================================== + +Ray related +------------ + +How to add breakpoint for debugging with distributed Ray? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Please checkout the official debugging guide from Ray: https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html + + +Distributed training +------------------------ + +How to run multi-node post-training with Ray? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can start a ray cluster and submit a ray job, following the official guide from Ray: https://docs.ray.io/en/latest/ray-core/starting-ray.html + +If your cluster is managed by Slurm, please refer to the guide for deploying Ray on Slurm: https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html diff --git a/frameworks/agentgym_rl/docs/hybrid_flow.rst b/frameworks/agentgym_rl/docs/hybrid_flow.rst new file mode 100644 index 0000000000000000000000000000000000000000..e2d2d8a7a49bca42b5bc5b5805c4d858a3a7c30d --- /dev/null +++ b/frameworks/agentgym_rl/docs/hybrid_flow.rst @@ -0,0 +1,269 @@ +========================================================= +HybridFlow Programming Guide +========================================================= + +.. _vermouth: https://github.com/vermouth1992 + +Author: `Chi Zhang `_ + +verl is an open source implementation of the paper `HybridFlow `_ [1]_. In this section, we will introduce the basic concepts of HybridFlow, the motivation and how to program with verl APIs. + +Motivation and Design +------------------------ +We use dataflow to represent RL systems. [4]_. + +DataFlow +~~~~~~~~~~~~~~~~~~~~ + +Dataflow is an abstraction of computations. Neural Netowork training is a typical dataflow. It can be represented by computational graph. + +.. image:: https://github.com/eric-haibin-lin/verl-community/blob/main/docs/dataflow.jpeg?raw=true + :alt: The dataflow graph from CS231n 2024 lecture 4 + +This figure [2]_ represents the computation graph of a polynomial function followed by a sigmoid function. In the data flow of neural network computation, each node represents an operator, and each edge represents the direction of forward/backward propagation. The computation graph determines the architecture of the neural network. + +RL as a dataflow problem +++++++++++++++++++++++++++++++++++++++++++++++ + +Reinforcement learning (RL) training can also be represented as a dataflow. Below is the dataflow graph that represents the PPO algorithm used in RLHF [3]_: + +.. image:: https://picx.zhimg.com/70/v2-cb8ab5ee946a105aab6a563e92682ffa_1440w.avis?source=172ae18b&biz_tag=Post + :alt: PPO dataflow graph, credit to Zhihu 低级炼丹师 + +However, the dataflow of RL has fundamental differences compared with dataflow of neural network training as follows: + ++--------------------------+--------------------------------------------------+---------------------+ +| Workload | Node | Edge | ++--------------------------+--------------------------------------------------+---------------------+ +| Neural Network Training | Operator (+/-/matmul/softmax) | Tensor movement | ++--------------------------+--------------------------------------------------+---------------------+ +| Reinforcement Learning | High-level operators (rollout/model forward) | Data Movement | ++--------------------------+--------------------------------------------------+---------------------+ + +In the case of tabular reinforcement learning, each operator is a simple scalar math operation (e.g., bellman update). In deep reinforcement learning(DRL), each operator is a high-level neural network computation such as model inference/update. This makes RL a two-level dataflow problem: + +- Control flow: defines how the high-level operators are executed (e.g., In PPO, we first perform rollout. Then, we perform advantage computation. Finally, we perform training). It expresses the **core logics of RL algorithms**. +- Computation flow: defines the dataflow of **neural network computation** (e.g., model forward/backward/optimizer). + + +Design Choices +~~~~~~~~~~~~~~~~~~~~ +The model size used in DRL before the LLM era is typically small. Thus, the high-level neural network computation can be done in a single process. This enables embedding the computation flow inside the control flow as a single process. + +However, in the LLM era, the computation flow (e.g., training neural network) becomes a multi-process program. This naturally leads to two design choices: + +1. Convert the control flow into a multi-process program as well. Then colocate with computation flow (unified multi-controller) + +- Advantages: + + - Achieves the **optimal performance** under fixed computation flow and control flow as the communication overhead in both training and data transfer is minimized. + +- Disadvantages: + + - The computation and/or control flow is **hard to reuse** from software perspective as computation code is coupled with specific controller code. For example, the training loop of PPO is generic. Say we have an PPO training flow implemented with a specific computation flow such as FSDP. Neither the control flow or computation flow can be reused if we want to switch the computation flow from FSDP to Megatron, due to the coupling of control and computation flows. + - Requires more efforts from the user under flexible and dynamic control flows, due to the multi-process nature of the program. + +2. Separate the flows: single process for the control flow and multi-process for computation flow + +- Advantages: + + - The computation flow defined elsewhere can be **easily reused** after the decoupling. + - The controller runs on a single process. Implementing a new RL algorithm with a **different control flow is simple and easy**. + +- Disadvantages: + + - Additional **data communication overhead** each time the controller process and computatation processes interact. The data has to be sent back and forth. + +In verl, the latter strategy with separate control flow and computation flow is adopted. verl is designed to decouple the control flow of RL algorithms, and the implementation of computation engines. + +Overall Execution Diagram +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Below is a simplified diagram denoting the execution of a reinforcement learning job. In the diagram, the controller runs on a single process, while the generator/actor workers, critic workers run on multiple processes, placed with specific resource groups. For rollout, the controller passes the data to the generator to perform sample generation. When the rollout is done, the data is passed back to controller for the next step of the algorithm. Similar execution is done for other workers. With the hybrid controller design, the data flow and computation is decoupled to provide both efficiency in computation and flexiblity in defining algorithm training loops. + +.. figure:: https://github.com/eric-haibin-lin/verl-community/blob/main/docs/driver_worker.png?raw=true + :alt: The execution diagram + +Codebase walkthrough (PPO) +------------------------------------------------ + +Entry function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Code: https://github.com/volcengine/verl/blob/main/verl/trainer/main_ppo.py + +In this file, we define a remote function `main_task` that serves as the controller (driver) process as shown in the above figure. We also define a ``RewardManager``, where users can customize their reward function based on the data source in the dataset. Note that `RewardManager` should return the final token-level reward that is optimized by RL algorithms. Note that users can combine model-based rewards and rule-based rewards. +The ``main_task`` constructs a RayPPOTrainer instance and launch the fit. Note that ``main_task`` **runs as a single process**. + +We highly recommend that the ``main_task`` is NOT schduled on the head of the ray cluster because ``main_task`` will consume a lot of memory but the head usually contains very few resources. + +Ray trainer +~~~~~~~~~~~~~~~~~~~~ +Code: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py + +The RayPPOTrainer manages + +- Worker and WorkerGroup construction +- Runs the main loop of PPO algorithm + +Note that, the fit function of RayPPOTrainer **runs as a single process**. + +Worker and WorkerGroup construction +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each workerGroup manages a list of workers that runs remotely. Note that the worker group runs in the process of its construtor. +Each worker inside the WorkerGroup runs on a GPU. The worker group serves as a proxy for the controller process to interact with a list of workers, in order to perform certain computations. **In order to do so, we have to bind the methods of the worker into the method of the WorkerGroup and define the data dispatch and data collection**. This is done via simple decoration that will be introduced in the Worker definition section. + +For example, in PPO, we define 3 worker groups: + +- ActorRolloutRef: manages actor, rollout and reference policy. ActorRolloutRefWorker can be instantiated as a single actor, a single rollout, a single reference policy, a combined actor/rollout or a combined actor/rollout/ref. This design is aimed for the maximum code reuse in various scenarios. The reason for colocating actor and rollout is for fast weight transfer using nccl. The reason for coloating actor and reference is to implement an efficient lora PPO as the reference policy is simply the base model of PPO in lora. +- Critic: manages the critic model +- Reward: manages the reward model + +The worker group will be constructed on the resource pool it designates. The resource pool is a set of GPUs in the ray cluster. + +Worker definition +~~~~~~~~~~~~~~~~~~~~ + +.. _ActorRolloutRefWorker: https://github.com/volcengine/verl/blob/main/verl/workers/fsdp_workers.py + +We take `ActorRolloutRefWorker <_ActorRolloutRefWorker>`_ for an exmaple. +The APIs it should expose to the controller process are: + +- init_model: build the underlying model +- generate_sequences: given prompts, generate responses +- compute_log_prob: compute the log-probability of a generated sequence using actor +- compute_ref_log_prob: compute the log-probability of a generated sequence using reference policy +- save_checkpoint: save the checkpoint + +Note that these methods are defined in the worker that can only be invoked via remote calls. For example, if the controller process wants to initialize the model, it has to call + +.. code-block:: python + + for worker in actor_rollout_ref_wg: + worker.init_model.remote() + +If the controller process wants to generate sequences, it has to call + +.. code-block:: python + + data = xxx + # split the data into dp chunks + data_dp_lst = data.split(dp_size) + output_dp_lst = [] + for i, worker in enumerate(actor_rollout_ref_wg): + output_future = worker.generate_sequences.remote(data_dp_lst[i]) + output_dp_lst.append(output_future) + output = torch.cat(ray.get(output_dp_lst), dim=0) + +We observe that controll process calling worker group methods in general can be divided into 3 parts: + +- Split the data into data parallel sizes +- Dispatch the corresponding data into each worker +- Collect and concatenate the data when the computation finishes + +In verl, we design a syntax sugar to encapsulate the 3 processes into a single call from the controller process. + +.. code-block:: python + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def generate_sequences(data): + ... + + # on the driver + output = actor_rollout_ref_wg.generate_sequences(data) + +We decorate the method of the worker with a ``register`` that explicitly defines how the input data should be splitted and dispatch to each worker, and how the output data should be collected and concatenated by the controller. For example, ``Dispatch.DP_COMPUTE_PROTO`` splits the input data into dp chunks, dispatch each data to each worker, collect the output and concatenate the results. Note that this function requires the input and output to be a DataProto defined here (https://github.com/volcengine/verl/blob/main/verl/protocol.py). + + +PPO main loop +~~~~~~~~~~~~~~~~~~~~ +With the aforementioned APIs, we can implement the main loop of PPO as if it is a single process program + +.. code-block:: python + + for prompt in dataloader: + output = actor_rollout_ref_wg.generate_sequences(prompt) + old_log_prob = actor_rollout_ref_wg.compute_log_prob(output) + ref_log_prob = actor_rollout_ref_wg.compute_ref_log_prob(output) + values = critic_wg.compute_values(output) + rewards = reward_wg.compute_scores(output) + # compute_advantages is running directly on the control process + advantages = compute_advantages(values, rewards) + output = output.union(old_log_prob) + output = output.union(ref_log_prob) + output = output.union(values) + output = output.union(rewards) + output = output.union(advantages) + # update actor + actor_rollout_ref_wg.update_actor(output) + critic.update_critic(output) + +Takeaways +~~~~~~~~~~~~~~~~~~~~ +- This programming paradigm enables users to use different computation backend without modification of the control process. +- This programming paradigm enables flexible placement (by changing the mapping of WorkerGroup and ResourcePool) without modification of the control process. + +Repository organization +------------------------------------------------ + +Important code files in the repository are organized as below: + +.. code-block:: bash + + verl # the verl package + trainer + main_ppo.py # the entrypoint for RL training + ppo + ray_trainer.py # the training loop for RL algorithms such as PPO + fsdp_sft_trainer.py # the SFT trainer with FSDP backend + config + generation.yaml # configuration template for rollout + ppo_trainer.yaml # configuration template for the RL trainer + workers + protocol.py # the interface of DataProto + fsdp_workers.py # the FSDP worker interfaces: ActorRolloutRefWorker, CriticWorker, RewardModelWorker + megatron_workers.py # the Megatron worker interfaces: ActorRolloutRefWorker, CriticWorker, RewardModelWorker + actor + dp_actor.py # data parallel actor with FSDP backend + megatron_actor.py # nD parallel actor with Megatron backend + critic + dp_critic.py # data parallel critic with FSDP backend + megatron_critic.py # nD parallel critic with FSDP backend + reward_model + megatron + reward_model.py # reward model with Megatron backend + rollout + vllm + vllm_rollout.py # rollout with vllm backend + hf_rollout.py # rollout with huggingface TGI backend + sharding_manager + fsdp_ulysses.py # data and model resharding when using FSDP + ulysses + fsdp_vllm.py # data and model resharding when using FSDP + ulysses + vllm + megatron_vllm.py # data and model resharding when using Megatron + vllm + utils + dataset # datasets for SFT/RM/RL + reward_score # function based reward + gsm8k.py # reward function for gsm8k dataset + math.py # reward function for math dataset + seqlen_balancing.py # the sequence balance optimization + models + llama # Megatron implementation for llama, deepseek, mistral, etc + transformers # ulysses integration with transformer models such as llama, qwen, etc + weight_loader_registery.py # registry of weight loaders for loading hf ckpt into Megatron + third_party + vllm # adaptor for vllm's usage in RL + vllm_v_0_6_3 # vllm v0.6.3 adaptor + llm.py # entrypoints for generate, sync_model_weight, offload_model_weights + parallel_state.py # vllm related device mesh and process groups + dtensor_weight_loaders.py # weight loader for huggingface models with FSDP + megatron_weight_loaders.py # weight loader for Megatron models + vllm_spmd # vllm >= v0.7 adaptor (coming soon) + examples # example scripts + tests # integration and unit tests + .github # the configuration of continuous integration tests + + +.. [1] HybridFlow: A Flexible and Efficient RLHF Framework: https://arxiv.org/abs/2409.19256v2 +.. [2] Data flow graph credit to CS231n 2024 lecture 4: https://cs231n.stanford.edu/slides/2024/lecture_4.pdf +.. [3] PPO dataflow graph credit to 低级炼丹师 from Zhihu​: https://zhuanlan.zhihu.com/p/635757674 +.. [4] RLFlow \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/index.rst b/frameworks/agentgym_rl/docs/index.rst new file mode 100644 index 0000000000000000000000000000000000000000..b27cebb843de476037ad83890c961a55120bbe52 --- /dev/null +++ b/frameworks/agentgym_rl/docs/index.rst @@ -0,0 +1,111 @@ +Welcome to verl's documentation! +================================================ + +.. _hf_arxiv: https://arxiv.org/pdf/2409.19256 + +verl is a flexible, efficient and production-ready RL training framework designed for large language models (LLMs) post-training. It is an open source implementation of the `HybridFlow `_ paper. + +verl is flexible and easy to use with: + +- **Easy extension of diverse RL algorithms**: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code. + +- **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks. + +- **Flexible device mapping and parallelism**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. + +- Readily integration with popular HuggingFace models + + +verl is fast with: + +- **State-of-the-art throughput**: By seamlessly integrating existing SOTA LLM training and inference frameworks, verl achieves high generation and training throughput. + +- **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. + +-------------------------------------------- + +.. _Contents: + +.. toctree:: + :maxdepth: 5 + :caption: Quickstart + + start/install + start/quickstart + +.. toctree:: + :maxdepth: 4 + :caption: Programming guide + + hybrid_flow + +.. toctree:: + :maxdepth: 5 + :caption: Data Preparation + + preparation/prepare_data + preparation/reward_function + +.. toctree:: + :maxdepth: 5 + :caption: Configurations + + examples/config + +.. toctree:: + :maxdepth: 2 + :caption: PPO Example + + examples/ppo_code_architecture + examples/gsm8k_example + +.. toctree:: + :maxdepth: 1 + :caption: PPO Trainer and Workers + + workers/ray_trainer + workers/fsdp_workers + workers/megatron_workers + +.. toctree:: + :maxdepth: 1 + :caption: Performance Tuning Guide + + perf/perf_tuning + +.. toctree:: + :maxdepth: 1 + :caption: Experimental Results + + experiment/ppo + +.. toctree:: + :maxdepth: 1 + :caption: Advance Usage and Extension + + advance/placement + advance/dpo_extension + advance/fsdp_extension + advance/megatron_extension + +.. toctree:: + :maxdepth: 1 + :caption: FAQ + + faq/faq + +Contribution +------------- + +verl is free software; you can redistribute it and/or modify it under the terms +of the Apache License 2.0. We welcome contributions. +Join us on `GitHub `_, `Slack `_ and `Wechat `_ for discussions. + +Code formatting +^^^^^^^^^^^^^^^^^^^^^^^^ +We use yapf (Google style) to enforce strict code formatting when reviewing MRs. Run yapf at the top level of verl repo: + +.. code-block:: bash + + pip3 install yapf + yapf -ir -vv --style ./.style.yapf verl examples tests diff --git a/frameworks/agentgym_rl/docs/perf/perf_tuning.rst b/frameworks/agentgym_rl/docs/perf/perf_tuning.rst new file mode 100644 index 0000000000000000000000000000000000000000..2028044d47f59926f9de3635b1944cf14d887f8d --- /dev/null +++ b/frameworks/agentgym_rl/docs/perf/perf_tuning.rst @@ -0,0 +1,164 @@ +Performance Tuning Guide +============================== + +Author: `Guangming Sheng `_ + +In this section, we will discuss how to tune the performance of all the stages in verl, including: + +1. Rollout generation throughput. + +2. Enable `use_remove_padding=True` for sequence packing (i.e., data packing and remove padding). + +3. Batch size tuning for forward and backward computation + +4. Enable ``use_dynamic_bsz=True`` for higher throughput. + +5. Utilize Ulysses Sequence Parallel for Long Context Training + +6. LigerKernel for SFT performance optimization + +Rollout Generation Tuning +-------------------------- + +verl currently supports two rollout backends: vLLM and TGI (with SGLang support coming soon). + +Below are key factors for tuning vLLM-based rollout. Before tuning, we recommend setting ``actor_rollout_ref.rollout.disable_log_stats=False`` so that rollout statistics are logged. + +- Increase ``gpu_memory_utilization``. The vLLM pre-allocates GPU KVCache by using gpu_memory_utilization% of the remaining memory. + However, if model parameters and optimizer states are not offloaded, using too high a fraction can lead to OOM. + A value between 0.5 and 0.7 often strikes a good balance between high throughput and avoiding OOM. + +- Adjust ``max_num_seqs`` or ``max_num_batched_tokens``. + If the GPU cache utilization is relatively low in the log, increase ``max_num_seqs`` or ``max_num_batched_tokens`` + can enlarge the effective batch size in the decoding stage, allowing more concurrent requests per batch. + We recommend setting ``max_num_batched_tokens > 2048`` for higher throughput. + +- Use a smaller ``tensor_parallel_size``. + When GPU resources allow, a smaller tensor parallel size spawns more vLLM replicas. + Data parallelism (DP) can yield higher throughput than tensor parallelism (TP), but also increases KVCache consumption. + Carefully balance the trade-off between more replicas and higher memory usage. + Our experient in Sec. 8.4 of `HybridFlow paper `_ evaluate this trade-off. + +More tuning details such as dealing with Preemption and Chunked-prefill +can be found in `vLLM official tuning guide `_ + +The performance of vllm can be further increased if upgrading from v0.6.3 to v0.7. See https://github.com/volcengine/verl/blob/main/docs/README_vllm0.7.md for details on how to upgrade. + +Enable remove padding (sequence packing) +----------------------------------------- + +Currently, for llama, mistral, gemma1 and qwen based models, users can enable `use_remove_padding=True` to utilize the +sequence packing implementation provided by transformers library. + +For other models, transformers library may also support it but we haven't tested it yet. +Users can add the desired model config to the `test_transformer.py `_ file. +And test its functionaility by running the following command: + +.. code-block:: bash + + pytest -s tests/model/test_transformer.py + +If the test passes, you can add your desired model into the model `registry.py `_ file. +Then, you can enjoy the performance boost of sequence packing +and welcome to PR your tested model to verl! + + +Batch Size Tuning +----------------- + +To achieve higher throughput in experience preparation (i.e., model fwd) and model update (i.e., actor/critic fwd/bwd), +users may need to tune the ``*micro_batch_size_per_gpu`` for different computation. + +In verl, the core principle for setting batch sizes is: + +- **Algorithmic metrics** (train batch size, PPO mini-batch size) are *global* (from a single-controller perspective), + normalized in each worker. See the `normalization code `_. + +- **Performance-related parameters** (micro batch size, max token length for dynamic batch size) are *local* parameters that define the per-GPU data allocations. + See the `normalization code `_. + +.. note:: In your training script, please use ``*micro_batch_size_per_gpu`` instead of ``*micro_batch_size``. + So that you don't need to consider the normalization of the ``micro_batch_size`` and ``micro_batch_size`` will be deprecated. + +Batch Size Tuning tips +"""""""""""""""""""""" + +Therefore, users may need to tune the ``*micro_batch_size_per_gpu`` to accelerate training. Here're some tips: + +1. **Enable gradient checkpointing**: + Set ``actor_rollout_ref.model.enable_gradient_checkpointing=True`` and ``critic.model.enable_gradient_checkpointing=True``. + This often allows for larger micro-batch sizes and will be beneficial for large mini-batch training. + +2. Increase the ``*micro_batch_size_per_gpu`` as much as possible till equals to normalized ``mini_batch_size``. + +3. **Use larger forward-only parameters**: + Forward only parameter, such as ``actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu``, + ``actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu``, ``critic.forward_micro_batch_size_per_gpu`` could be larger (e.g., 2x) than training related micro batch sizes, + such as ``actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu``, ``critic.ppo_micro_batch_size_per_gpu``. + +4. **Allow larger micro-batch sizes for Critic and Reward models**: + micro batch size of Critic and Reward model could be larger than Actor model. This is because the actor model has much larger vocab size in the final layer. + + +Tuning for Dynamic Batch Size +----------------------------- + +Dynamic batch size is a technique that allows the model to process similar number of tokens in a single forward pass (with different actual batch sizes). +This can significantly improve the training efficiency and reduce the memory usage. + +To utilize this technique, users can set ``use_dynamic_bsz=True`` in actor, ref, critic and reward models. +With ``use_dynamic_bsz=True``, users don't need to tune ``*micro_batch_size_per_gpu``. +Instead, users should tune the following parameters: + +- ``actor_rollout_ref.actor.ppo_max_token_len_per_gpu``, ``critic.ppo_max_token_len_per_gpu``: + The maximum number of tokens to be processed in fwd and bwd of ``update_policy`` and ``update_critic``. + +- ``actor_rollout_ref.ref.log_prob_max_token_len_per_gpu`` and ``actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu``: + The maximum number of tokens to be processed in a the fwd computation of ``compute_log_prob`` and ``comptue_ref_log_prob``. + +- ``critic.forward_micro_batch_size_per_gpu``, ``reward_model.forward_micro_batch_size_per_gpu``: + The maximum number of tokens to be processed in a the fwd computation of ``compute_values``, ``compute_rm_score``. + +Dynamic Batch Size Tuning tips +"""""""""""""""""""""""""""""" + +Here're some tips to tune the above parameters: + +1. **Increase** ``actor_rollout_ref.actor.ppo_max_token_len_per_gpu`` + Make it at least 2 x (max_prompt_length + max_response_length). We set it to 3x in `run_qwen2-7b_rm_seq_balance.sh `_. + Try to increase it to get higher throughput. + +2. **Forward-only parameters can be larger**: + Similar to the non-dynamic-batch scenario, forward-only token limits can exceed those used in forward/backward operations. + +3. **Use larger limits for Critic and Reward models**: + Critic and Reward parameters can be set at least 2× the Actor’s limits. For instance, we set them to 4× here: + `run_qwen2-7b_rm_seq_balance.sh `_ + +.. :math:`\text{critic.ppo_max_token_len_per_gpu} = 2 \times \text{actor.ppo_max_token_len_per_gpu})`. + +Ulysses Sequence Parallel for Long Context Training +---------------------------------------------------- + +To utilize this technique, users can set ``ulysses_sequence_parallel_size>1`` in actor, ref, critic and reward models. + +We support different model utilize different ulysses_sequence_parallel_size sizes. + +To train log sequence (>32k), users may need to decrease the ``*micro_batch_size_per_gpu`` and ``*max_token_len_per_gpu`` to avoid OOM. + +LigerKernel for SFT +---------------------- + +LigerKernel is a high-performance kernel for Supervised Fine-Tuning (SFT) that can improve training efficiency. To enable LigerKernel in your SFT training: + +1. Install liger-kernel via ``pip3 install liger-kernel``. In your SFT configuration file (e.g., ``verl/trainer/config/sft_trainer.yaml``), set the ``use_liger`` parameter: + + .. code-block:: yaml + + model: + use_liger: True # Enable LigerKernel for SFT + +2. The default value is ``False``. Enable it only when you want to use LigerKernel's optimizations. + +3. LigerKernel is particularly useful for improving training performance in SFT scenarios. + diff --git a/frameworks/agentgym_rl/docs/preparation/prepare_data.rst b/frameworks/agentgym_rl/docs/preparation/prepare_data.rst new file mode 100644 index 0000000000000000000000000000000000000000..de88d8d91a42d6e14982740d9abd6375bd2d6138 --- /dev/null +++ b/frameworks/agentgym_rl/docs/preparation/prepare_data.rst @@ -0,0 +1,126 @@ +Prepare Data for Post-Training +======================================== + +Before starting the post-training job, we need to prepare the data for +the policy training. The data should be stored in the parquet format. + +We provide several data preprocess scripts for different datasets, +including GSM8K, MATH, HelloSwag, Full_hh_rlhf. To prepare other datasets, we need +to follow the following steps: The data preprocess script can be divided +into two parts: + +1. The first part is the common part, which loads the dataset from + huggingface's ``datasets`` package. Then preprocess the datasets with + the ``make_map_fn`` and then store in the parquet format. + +.. code:: python + + import re + import os + import datasets + + from verl.utils.hdfs_io import copy, makedirs + import argparse + + # To extract the solution for each prompts in the dataset + # def extract_solution(solution_str): + # ... + + + if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--local_dir', default='/opt/tiger/gsm8k') + parser.add_argument('--hdfs_dir', default=None) + + args = parser.parse_args() + + num_few_shot = 5 + data_source = 'openai/gsm8k' + + dataset = datasets.load_dataset(data_source, 'main') + + train_dataset = dataset['train'] + test_dataset = dataset['test'] + + # Construct a `def make_map_fn(split)` for the corresponding datasets. + # ... + + train_dataset = train_dataset.map(function=make_map_fn('train'), with_indices=True) + test_dataset = test_dataset.map(function=make_map_fn('test'), with_indices=True) + + local_dir = args.local_dir + hdfs_dir = args.hdfs_dir + + train_dataset.to_parquet(os.path.join(local_dir, 'train.parquet')) + test_dataset.to_parquet(os.path.join(local_dir, 'test.parquet')) + + makedirs(hdfs_dir) + + copy(src=local_dir, dst=hdfs_dir) + +2. The users are required to implement the ``make_map_fn()`` function + (as well as the ``extract_solution``) on their own to support + different datasets or tasks. + +We already implemented the data preprocess of GSM8k, MATH, Hellaswag and Full_hh_rlhf +datasets. And we take the GSM8k dataset as an example: + +**GSM8K** + +In the ``make_map_fn``, each data field should consist of the following +5 fields: + +1. ``data_source``: The name of the dataset. To index the corresponding + reward function in the ``RewardModule`` +2. ``prompt``: This field should be constructed in the format of + huggingface chat_template. The tokenizer in ``RLHFDataset`` will + apply chat template and tokenize the prompt. +3. ``ability``: Define the task category. +4. ``reward_model``: Currently, we only utilize the ``ground_truth`` + field during evaluation. The ``ground_truth`` is computed by the + ``extract_solution`` function. **NOTED** that the implementation of + the corresponding reward function should align with this extracted + ``ground_truth``. +5. ``extra_info``: Record some information of the current prompt. Not + use for now. + +.. code:: python + + def extract_solution(solution_str): + solution = re.search("#### (\\-?[0-9\\.\\,]+)", solution_str) # extract the solution after #### + assert solution is not None + final_solution = solution.group(0) + final_solution = final_solution.split('#### ')[1].replace(',', '') + return final_solution + + instruction_following = "Let's think step by step and output the final answer after \"####\"." + + # add a row to each data item that represents a unique id + def make_map_fn(split): + + def process_fn(example, idx): + question = example.pop('question') + + question = question + ' ' + instruction_following + + answer = example.pop('answer') + solution = extract_solution(answer) + data = { + "data_source": data_source, + "prompt": [{ + "role": "user", + "content": question + }], + "ability": "math", + "reward_model": { + "style": "rule", + "ground_truth": solution + }, + "extra_info": { + 'split': split, + 'index': idx + } + } + return data + + return process_fn diff --git a/frameworks/agentgym_rl/docs/preparation/reward_function.rst b/frameworks/agentgym_rl/docs/preparation/reward_function.rst new file mode 100644 index 0000000000000000000000000000000000000000..3bd2c0632230e806bc53f57b5e4f03896fe8ce3a --- /dev/null +++ b/frameworks/agentgym_rl/docs/preparation/reward_function.rst @@ -0,0 +1,46 @@ +Implement Reward Function for Dataset +====================================== + +For each dataset, we need to implement a reward function or utilize a reward model to compute the rewards for the generated responses. +We already pre-implemented some reward functions in `reward_score directory `_. + +Currently, we support reward functions for GSM8k and MATH datasets. For RLHF datasets (e.g., +full_hh_rlhf) and Code Generation (e.g., APPS), we utilize reward model +and SandBox (will opensource soon) for evaluation respectively. + +RewardManager +------------- + +In the entrypoint of the PPO Post-Training script `main_ppo.py `_, +we implement a ``RewardManager`` that utilze pre-implemented reward functions to compute the scores for each response. + +In the ``RewardManager``, we implemented a ``__call__`` function to +compute the score for each response. +All the reward functions are executed by ``compute_score_fn``. +The input is a ``DataProto``, which includes: + +- ``input_ids``, ``attention_mask``: ``input_ids`` and ``attention_mask`` after applying + chat_template, including prompt and response +- ``responses``: response tokens +- ``ground_truth``: The ground truth string of the current prompt. + Stored in ``non_tensor_batch`` in the ``DataProto``, which should be + preprocessed in the parquet files. +- ``data_source``: The dataset name of the current prompt. Stored in + ``non_tensor_batch`` in the ``DataProto``, which should be + preprocessed in the parquet files. + +After detokenize the responses, the responses string and the ground +truth string will be input to the ``compute_score_fn`` to compute the +score for each response. + +Reward Functions +---------------- +We already pre-implemented some reward functions in `reward_score directory `_. + +- In the `GSM8k example `_, we + force the response to output the final answer after four ####, then + use string matching to compare with the ground truth. If completely + correct, score 1 point; if the format is correct, score 0.1 points; if + the format is incorrect, score 0 points. +- In the `MATH example `_, we follow + the implementation in `lm-evaluation-harness repository `_. diff --git a/frameworks/agentgym_rl/docs/requirements-docs.txt b/frameworks/agentgym_rl/docs/requirements-docs.txt new file mode 100644 index 0000000000000000000000000000000000000000..03cfbf106943cf2204059d7e16fd52f891e231ea --- /dev/null +++ b/frameworks/agentgym_rl/docs/requirements-docs.txt @@ -0,0 +1,9 @@ +# markdown suport +recommonmark +# markdown table suport +sphinx-markdown-tables + +# theme default rtd + +# crate-docs-theme +sphinx-rtd-theme \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/start/install.rst b/frameworks/agentgym_rl/docs/start/install.rst new file mode 100644 index 0000000000000000000000000000000000000000..339f4bd57f4e87c700e1b8976a04aa3bba08fc6a --- /dev/null +++ b/frameworks/agentgym_rl/docs/start/install.rst @@ -0,0 +1,111 @@ +Installation +============ + +Requirements +------------ + +- **Python**: Version >= 3.9 +- **CUDA**: Version >= 12.1 + +verl supports various backends. Currently, the following configurations are available: + +- **FSDP** and **Megatron-LM** (optional) for training. +- **vLLM** adn **TGI** for rollout generation, **SGLang** support coming soon. + +Training backends +------------------ + +We recommend using **FSDP** backend to investigate, research and prototype different models, datasets and RL algorithms. The guide for using FSDP backend can be found in :doc:`FSDP Workers<../workers/fsdp_workers>`. + +For users who pursue better scalability, we recommend using **Megatron-LM** backend. Currently, we support Megatron-LM v0.4 [1]_. The guide for using Megatron-LM backend can be found in :doc:`Megatron-LM Workers<../workers/megatron_workers>`. + + +Install from docker image +------------------------- + +We provide pre-built Docker images for quick setup. + +Image and tag: ``verlai/verl:vemlp-th2.4.0-cu124-vllm0.6.3-ray2.10-te1.7-v0.0.3``. See files under ``docker/`` for NGC-based image or if you want to build your own. + +1. Launch the desired Docker image: + +.. code:: bash + + docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN -v + + +2. Inside the container, install verl: + +.. code:: bash + + # install the stable version + pip3 install verl + + +3. Setup Megatron (optional) + +If you want to enable training with Megatron, Megatron code must be added to PYTHONPATH: + +.. code:: bash + + cd .. + git clone -b core_v0.4.0 https://github.com/NVIDIA/Megatron-LM.git + cp verl/patches/megatron_v4.patch Megatron-LM/ + cd Megatron-LM && git apply megatron_v4.patch + pip3 install -e . + export PYTHONPATH=$PYTHONPATH:$(pwd) + + +You can also get the Megatron code after verl's patch via + +.. code:: bash + + git clone -b core_v0.4.0_verl https://github.com/eric-haibin-lin/Megatron-LM + export PYTHONPATH=$PYTHONPATH:$(pwd)/Megatron-LM + +Install from custom environment +--------------------------------- + +To manage environment, we recommend using conda: + +.. code:: bash + + conda create -n verl python==3.9 + conda activate verl + +For installing the latest version of verl, the best way is to clone and +install it from source. Then you can modify our code to customize your +own post-training jobs. + +.. code:: bash + + # install verl together with some lightweight dependencies in setup.py + pip3 install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124 + pip3 install flash-attn --no-build-isolation + pip3 install verl + + +Megatron is optional. It's dependencies can be setup as below: + +.. code:: bash + + # apex + pip3 install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" \ + git+https://github.com/NVIDIA/apex + + # transformer engine + pip3 install git+https://github.com/NVIDIA/TransformerEngine.git@v1.7 + + # megatron core v0.4.0: clone and apply the patch + # You can also get the patched Megatron code patch via + # git clone -b core_v0.4.0_verl https://github.com/eric-haibin-lin/Megatron-LM + cd .. + git clone -b core_v0.4.0 https://github.com/NVIDIA/Megatron-LM.git + cd Megatron-LM + cp ../verl/patches/megatron_v4.patch . + git apply megatron_v4.patch + pip3 install -e . + export PYTHONPATH=$PYTHONPATH:$(pwd) + + +.. [1] Megatron v0.4 is supported with verl's patches to fix issues such as virtual pipeline hang. It will be soon updated with latest the version of upstream Megatron-LM without patches. \ No newline at end of file diff --git a/frameworks/agentgym_rl/docs/start/quickstart.rst b/frameworks/agentgym_rl/docs/start/quickstart.rst new file mode 100644 index 0000000000000000000000000000000000000000..c3be7d62ece68079a545e71ab329ca86b986f4b2 --- /dev/null +++ b/frameworks/agentgym_rl/docs/start/quickstart.rst @@ -0,0 +1,142 @@ +.. _quickstart: + +========================================================= +Quickstart: PPO training on GSM8K dataset +========================================================= + +Post-train a LLM using GSM8K dataset. + +Introduction +------------ + +.. _hf_dataset_gsm8k: https://huggingface.co/datasets/gsm8k + +In this example, we train an LLM to tackle the `GSM8k `_ task with function-based rewards. [1]_ + +Prerequisite: + +- the latest version of ``verl`` and its dependencies installed following the installation guide. Using the docker image is recommended. + +- an GPU with at least 24 GB HBM + + +Dataset Introduction +-------------------- + +GSM8k is a math problem dataset. The prompt is an elementary school +problem. The LLM model is asked to solve the math problem. Below is an example: + +Prompt + + Katy makes coffee using teaspoons of sugar and cups of water in the + ratio of 7:13. If she used a total of 120 teaspoons of sugar and cups + of water, calculate the number of teaspoonfuls of sugar she used. + +Solution + + The total ratio representing the ingredients she used to make the + coffee is 7+13 = <<7+13=20>>20 Since the fraction representing the + number of teaspoons she used is 7/20, she used 7/20\ *120 = + <<7/20*\ 120=42>>42 #### 42 + +Step 1: Prepare the dataset +---------------------------- + +We preprocess the dataset in parquet format so that (1) it contains necessary fields for computing RL rewards and (2) is faster to read. + +.. code-block:: bash + + python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k + +Step 2: Download a model for post-training +------------------------------------------- + +In this example, we start with the ``Qwen2.5-0.5B-Instruct`` model. + +If you want to perform SFT before RL, refer to the :doc:`Complete GSM8K Example<../examples/gsm8k_example>`, the `sft directory `_ and `SFT Trainer `_ for further details. + +.. code-block:: bash + + python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen2.5-0.5B-Instruct')" + +Step 3: Perform PPO training with the instruct model +---------------------------------------------------------------------- + +**Reward Model/Function** + +We use a pre-defined rule-based reward model. We force the model to produce a final +answer following 4 “#” as shown in the solution. We extract the final +answer from both the solution and model's output using regular +expression matching. We assign a reward of 1 to correct +answer, 0.1 to incorrect answer and 0 to no answer. + +For mode details, please refer to `verl/utils/reward_score/gsm8k.py `_. + +**Training Script** + +Now let's run PPO training with the dataset and model above. [2]_ + + +Set the ``data.train_files`` ,\ ``data.val_files``, ``actor_rollout_ref.model.path`` and ``critic.model.path`` based on your dataset and model names or paths. + +.. code-block:: bash + + PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \ + data.train_files=$HOME/data/gsm8k/train.parquet \ + data.val_files=$HOME/data/gsm8k/test.parquet \ + data.train_batch_size=256 \ + data.val_batch_size=1312 \ + data.max_prompt_length=512 \ + data.max_response_length=256 \ + actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \ + actor_rollout_ref.actor.optim.lr=1e-6 \ + actor_rollout_ref.actor.ppo_mini_batch_size=64 \ + actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \ + actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \ + actor_rollout_ref.rollout.tensor_model_parallel_size=1 \ + actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \ + actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \ + critic.optim.lr=1e-5 \ + critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \ + critic.ppo_micro_batch_size_per_gpu=4 \ + algorithm.kl_ctrl.kl_coef=0.001 \ + trainer.logger=['console'] \ + +trainer.val_before_train=False \ + trainer.default_hdfs_dir=null \ + trainer.n_gpus_per_node=1 \ + trainer.nnodes=1 \ + trainer.save_freq=10 \ + trainer.test_freq=10 \ + trainer.total_epochs=15 2>&1 | tee verl_demo.log + +You are expected to see the following logs, indicating training in progress. The key metric ``val/test_score/openai/gsm8k`` is computed every ``trainer.test_freq`` steps: + +.. code-block:: bash + + step:0 - timing/gen:21.470 - timing/ref:4.360 - timing/values:5.800 - critic/kl:0.000 - critic/kl_coeff:0.001 - timing/adv:0.109 - timing/update_critic:15.664 - critic/vf_loss:14.947 - critic/vf_clipfrac:0.000 - critic/vpred_mean:-2.056 - critic/grad_norm:1023.278 - critic/lr(1e-4):0.100 - timing/update_actor:20.314 - actor/entropy_loss:0.433 - actor/pg_loss:-0.005 - actor/pg_clipfrac:0.000 - actor/ppo_kl:0.000 - actor/grad_norm:1.992 - actor/lr(1e-4):0.010 - critic/score/mean:0.004 - critic/score/max:1.000 - critic/score/min:0.000 - critic/rewards/mean:0.004 - critic/rewards/max:1.000 - critic/rewards/min:0.000 - critic/advantages/mean:-0.000 - critic/advantages/max:2.360 - critic/advantages/min:-2.280 - critic/returns/mean:0.003 - critic/returns/max:0.000 - critic/returns/min:0.000 - critic/values/mean:-2.045 - critic/values/max:9.500 - critic/values/min:-14.000 - response_length/mean:239.133 - response_length/max:256.000 - response_length/min:77.000 - prompt_length/mean:104.883 - prompt_length/max:175.000 - prompt_length/min:68.000 + step:1 - timing/gen:23.020 - timing/ref:4.322 - timing/values:5.953 - critic/kl:0.000 - critic/kl_coeff:0.001 - timing/adv:0.118 - timing/update_critic:15.646 - critic/vf_loss:18.472 - critic/vf_clipfrac:0.384 - critic/vpred_mean:1.038 - critic/grad_norm:942.924 - critic/lr(1e-4):0.100 - timing/update_actor:20.526 - actor/entropy_loss:0.440 - actor/pg_loss:0.000 - actor/pg_clipfrac:0.002 - actor/ppo_kl:0.000 - actor/grad_norm:2.060 - actor/lr(1e-4):0.010 - critic/score/mean:0.000 - critic/score/max:0.000 - critic/score/min:0.000 - critic/rewards/mean:0.000 - critic/rewards/max:0.000 - critic/rewards/min:0.000 - critic/advantages/mean:0.000 - critic/advantages/max:2.702 - critic/advantages/min:-2.616 - critic/returns/mean:0.000 - critic/returns/max:0.000 - critic/returns/min:0.000 - critic/values/mean:-2.280 - critic/values/max:11.000 - critic/values/min:-16.000 - response_length/mean:232.242 - response_length/max:256.000 - response_length/min:91.000 - prompt_length/mean:102.398 - prompt_length/max:185.000 - prompt_length/min:70.000 + +Checkout :ref:`algo-baseline-page` for full training and validation logs for reference. + +The checkpoint is saved at the following dir by default: ``checkpoints/${trainer.project_name}/${trainer.experiment_name}`` + +To enable ``wandb`` for experiment tracking, set the following configs: + +.. code-block:: bash + + trainer.logger=['console','wandb'] \ + trainer.project_name=$YOUR_PROJECT_NAME \ + trainer.experiment_name=$YOUR_RUN_NAME \ + +If you encounter out of memory issues with HBM less than 32GB, enable the following configs would help: + +.. code-block:: bash + + actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \ + critic.ppo_micro_batch_size_per_gpu=1 \ + +For the full set of configs, please refer to :ref:`config-explain-page` for detailed explanation and performance tuning. + + +.. [1] The original paper (https://arxiv.org/pdf/2110.14168) mainly focuses on training a verifier (a reward model) to solve math problems via Best-of-N sampling. In this example, we train an RL agent using a rule-based reward model. +.. [2] More training script examples for FSDP and Megatron-LM backend are stored in `examples/ppo_trainer `_ directory. diff --git a/frameworks/agentgym_rl/docs/workers/fsdp_workers.rst b/frameworks/agentgym_rl/docs/workers/fsdp_workers.rst new file mode 100644 index 0000000000000000000000000000000000000000..b1c78e7c8399e1e543e1c4c91de791a8123df8e6 --- /dev/null +++ b/frameworks/agentgym_rl/docs/workers/fsdp_workers.rst @@ -0,0 +1,142 @@ +PyTorch FSDP Backend +====================== + +We support PyTorch FSDP Backend by implementing various workers for +actor, critic, reference, rollout and reward models. We also implement +the ``FSDPVLLMShardingManager`` that reshard weight between FSDP and +vLLM in `fsdp_vllm.py `_. + +**Pros** + +- Readily support various models. + + - Users only need to implement the corresponding + ``dtensor_weight_loader`` for weight synchronization between FSDP + and vLLM. While for ``hf_weight_loader``, users can directly apply + any models supported both in HF and vLLM without any code change. + +- Easy to organize the forward and backward computation for each model. + +**Cons** + +- Poor scalability when it comes to large-scale models (e.g. Llama 70B + and 405B) +- The resharding overhead between actor and rollout could be larger than + Megatron-LM backend. + +Due to the simplicity, we recommend using FSDP backend for algorithm +research and prototyping. + +FSDP Workers +-------------- + +ActorRolloutRefWorker +^^^^^^^^^^^^^^^^^^^^^ + +Actor/Rollout HybridEngine +'''''''''''''''''''''''''' + +1. HybridEngine, Actor and Rollout initialization API. + +.. code:: python + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def init_model(self): + +``ONE_TO_ALL``: when calling the ``init_model`` function from the driver +process, each worker (on a GPU) will execute the following model +initialization process. + +The initialization details of HybridEngine, Actor and Rollout are +highlighted below: + +1. ``DataParallelPPOActor`` implements the simple PPO computation logics + when the model is built with FSDP, including compute log prob, model + update. +2. ``vLLMRollout`` support generation with vLLM. We modify the vLLM + Engine and make it executed under SPMD to fit into our + ``WorkerGroup`` design. +3. ``FSDPVLLMShardingManager`` a context manager to perform actual + resharding between actor and rollout. + +See `source code `_. for more information. + +1. Generate sequence and recompute log prob + +.. code:: python + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def generate_sequences(self, prompts: DataProto): + +- ``Dispatch.DP_COMPUTE_PROTO``: The data will be dispatched and + collected along the DP dimension + +- In this function, the rollout model will perform auto-regressive + generation and the actor model will recompute the old log prob for the + generated response. + +3. Update actor model + +.. code:: python + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def update_actor(self, data: DataProto): + +- Update the actor model weight using PPO & entropy loss. + +ReferenceModel +'''''''''''''' + +1. Reference model initialization + +The reference model is initialized using the same function as the actor +model without initializing the HybridEngine and Optimizer. Then the +actor model is also wrapped by the ``DataParallelPPOActor``. + +2. Compute reference log prob + +.. code:: python + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def compute_ref_log_prob(self, data: DataProto): + +- In this function, the reference model will call the compute log prob + function in ``DataParallelPPOActor`` to compute the reference log + prob. + +CriticWorker and RewardWorker +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Model initialization + +Quite similar to reference model. The CriticWorker will perform +additional initialization for the Optimizer. + +2. Compute Values for CriticWorker + +.. code:: python + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def compute_values(self, data: DataProto): + +3. Update Critic + +.. code:: python + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def update_critic(self, data: DataProto): + +4. Compute Reward + +.. code:: python + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def compute_rm_score(self, data: DataProto): + + +HybridShard +------------ + +We didn't support FSDP `HybridShard`. To support this, we may need to +construct a 2D device mesh and test the corresponding +``dtensor_weight_loader`` and ``hf_weight_loader`` for each model. diff --git a/frameworks/agentgym_rl/docs/workers/megatron_workers.rst b/frameworks/agentgym_rl/docs/workers/megatron_workers.rst new file mode 100644 index 0000000000000000000000000000000000000000..c9e3b0d2bbe9a970893b5722e8bef863e80a49ed --- /dev/null +++ b/frameworks/agentgym_rl/docs/workers/megatron_workers.rst @@ -0,0 +1,200 @@ +Megatron-LM Backend +===================== + +We support Megatron Backend by implementing various workers for actor, +critic, reference, rollout and reward models. We also implement the +``3DHybridEngine`` using Megatron-LM and vLLM in `megatron_vllm.py `_. + +**Pros** + +- Support 3D parallelism and sequence parallelism for best scalablility + and throughput. +- 3D HybridEngine can significantly reduce peak memory usage and reduce + weight synchronize overhead between actor and rollout. + +**Cons** + +- Users should implement their own models for Megatron-LM +- Users should implement the corresponding weight_loader to + + - synchronize the model weight between actor (in Megatron) and rollout + (in vLLM). + - load weights from checkpoints to corresponding model in Megatron-LM + +Megatron Workers +---------------- + +MegatronWorker +^^^^^^^^^^^^^^ + +``MegatronWorker`` is the base class of different megatron worker +classes. In this class, ``get_megatron_global_info`` and +``get_megatron_rank_info`` function to retrive the 3D parallel world +size and rank of each ``Worker`` running on specific GPU. These information +will be used in transfer protocol for Megatron Backend. + +The following ``Worker`` class for different models will be utilized to +construct the ``WorkerGroup`` . + +We implement various of APIs for each ``Worker`` class decorated by the +``@register(dispatch_mode=)`` . These APIs can be called by the ray +driver process. The data can be correctly collect and dispatch following +the ``dispatch_mode`` on each function. The supported dispatch_model +(i.e., transfer protocols) can be found in `decorator.py `_. + +ActorRolloutRefWorker +^^^^^^^^^^^^^^^^^^^^^ + +This class is implemented for Actor/Rollout HybridEngine or for the +reference model to initialize their model and perform computation. + +Actor/Rollout HybridEngine +'''''''''''''''''''''''''' + +1. HybridEngine, Actor and Rollout initialization API. + +.. code:: python + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def init_model(self): + +``ONE_TO_ALL``: when calling the ``init_model`` function from the driver +process, each worker (on a GPU) will execute the following model +initialization process. + +The initialization details of HybridEngine, Actor and Rollout are +highlighted below: + +1. ``AllGatherPPModel`` holds memory buffer for both Actor and Rollout + and support weight resharding between actor and rollout. +2. ``MegatronPPOActor`` implements the simple PPO computation logics + when the model is built with Megatron, including compute log prob, + model update. +3. ``vLLMRollout`` support generation with vLLM. We modify the vLLM + Engine and make it executed under SPMD to fit into our + ``WorkerGroup`` design. +4. ``MegatronVLLMShardingManager`` a context manager to perform actual + resharding between actor and rollout. + +See `source code `_ for more information. + +.. code:: python + + # Initialize the 3D HybridEngine + hybrid_engine = AllGatherPPModel(model_provider=megatron_actor_model_provider) + # Fetch the model at current rank + actor_module = hybrid_engine.this_rank_models + ... + + # build actor model + self.actor = MegatronPPOActor(config=self.config.actor, + model_config=self.actor_model_config, + megatron_config=megatron_config, + actor_module=self.actor_module, + actor_optimizer=self.actor_optimizer, + actor_optimizer_config=self.actor_optim_config) + + # build rollout + # rollout initialization + rollout = vLLMRollout(actor_module=params, + config=self.config.rollout, + tokenizer=self.tokenizer, + model_hf_config=self.actor_model_config, + train_tp=mpu.get_tensor_model_parallel_world_size()) + # perform weight resharding between actor and rollout + sharding_manager = MegatronVLLMShardingManager(module=self.hybrid_engine, + inference_engine=rollout.inference_engine, + model_config=self.actor_model_config, + layer_name_mapping=layer_name_mapping) + ... + +2. Generate sequence and recompute log prob + +.. code:: python + + @register(dispatch_mode=Dispatch.MEGATRON_PP_AS_DP_PROTO) + def generate_sequences(self, prompts: DataProto): + +- ``Dispatch.MEGATRON_PP_AS_DP_PROTO``: The PP dimension of the actor + model will be regarded as DP dimension. Then the driver process will + dispatch and collect the data according to this reorganization. This + is because, in HybridEngine, the actor weight, which usually applied + larger 3D parallel sizes, will be gathered along the PP dimension and + TP dimension. Therefore, the corresponding data should be dispatched + and collected through the 3D parallel group of the rollout model, + rather than the actor model. However, the world_size and rank + information can only be retrived from ``get_megatron_global_info`` and + ``get_megatron_rank_info``, which records the 3D information for the + actor model. Moreover, the data resharding inside TP dimension will be + processed within the HybridEngine. + +- In this function, the rollout model will perform auto-regressive + generation and the actor model will recompute the old log prob for the + generated response. + +3. Update actor model + +.. code:: python + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def update_actor(self, data: DataProto): + +- ``Dispatch.MEGATRON_COMPUTE_PROTO``: User passes the data partitioned + by DP dimension. The data is dispatched to all tp/pp ranks within the + same dp group, and ultimately only collects output data from tp=0 and + the last pp. +- Update the actor model weight using PPO & entropy loss. + +ReferenceModel +'''''''''''''' + +1. Reference model initialization + +The reference model is initialized using the same function as the actor +model without initializing the HybridEngine and Optimizer. Then the +actor model is also wrapped by the ``MegatronPPOActor``. + +2. Compute reference log prob + +.. code:: python + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def compute_ref_log_prob(self, data: DataProto): + +- In this function, the reference model will call the compute log prob + function in ``MegatronPPOActor`` to compute the reference log prob. + +CriticWorker and RewardWorker +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Model initialization + +Quite similar to reference model. The CriticWorker will perform +additional initialization for the Optimizer. + +2. Compute Values for CriticWorker + +.. code:: python + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def compute_values(self, data: DataProto): + +3. Update Critic + +.. code:: python + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def update_critic(self, data: DataProto): + +4. Compute Reward + +.. code:: python + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def compute_rm_score(self, data: DataProto): + +Context Parallel +---------------- + +This require the developer/contributor to implement the context parallel +both in Megatron-LM and models. diff --git a/frameworks/agentgym_rl/docs/workers/ray_trainer.rst b/frameworks/agentgym_rl/docs/workers/ray_trainer.rst new file mode 100644 index 0000000000000000000000000000000000000000..c679ebf2bd753beb5263e0a87070145f4d6f1899 --- /dev/null +++ b/frameworks/agentgym_rl/docs/workers/ray_trainer.rst @@ -0,0 +1,243 @@ +PPO Ray Trainer +=============== + +We implement the RayPPOTrainer, which is a trainer runs on the driver +process on a single CPU/GPU node (default is CPU). + +The PPORayTrainer include 3 core functions for data preparation, +WorkerGroup initialization and PPO training loop. + +Data Preparation +---------------- + +The ``PPORayTrainer``, as a single process, is responsible for loading a +complete batch of samples (prompts) from the dataset and then dispatch +to different worker_groups running on different GPUs. + +To generalize the data loading, we implement the ``RLHFDataset`` class +to load the preprocessed parquet files, apply chat templates to the +prompts, add padding, truncate prompts that exceed max prompt length and +then tokenize. + +.. code:: python + + self.train_dataset = RLHFDataset(parquet_files=self.config.data.train_files, + tokenizer=self.tokenizer, + prompt_key=self.config.data.prompt_key, + max_prompt_length=self.config.data.max_prompt_length, + filter_prompts=True, + return_raw_chat=self.config.data.get('return_raw_chat', False), + truncation='error') + +Then, the dataloader will iterate the dataset under PPO mini batch size. + +WorkerGroup Initialization +-------------------------- + +We first introduce a basic implementation of initializing the +``WorkerGroup`` of the actor model on a given set of GPUs. + +.. code:: python + + # max_colocate_count means the number of WorkerGroups (i.e. processes) in each RayResourcePool + # For FSDP backend, we recommend using max_colocate_count=1 that merge all WorkerGroups into one. + # For Megatron backend, we recommend using max_colocate_count>1 that can utilize different WorkerGroup for differnt models + resource_pool = RayResourcePool(process_on_nodes=[config.trainer.n_gpus_per_node] * config.trainer.nnodes, + use_gpu=True, + max_colocate_count=1) + # define actor rollout cls to be init on remote + actor_rollout_cls = RayClassWithInitArgs(cls=ActorRolloutWorker) + # define actor_rollout worker group + actor_rollout_worker_group = MegatronRayWorkerGroup(resource_pool=resource_pool, + ray_cls_with_init=actor_rollout_cls, + default_megatron_kwargs=config.actor_rollout.megatron) + +Different WorkerGroups, like ``actor_rollout_worker_group`` , +``critic_worker_group`` and ``ref_worker_group`` lies on a separate +process in the above implementation. + +The driver process can then call the distributed compute function within +the ``actor_rollout_worker_group`` and other roles to construct the RL +training loop. + +For models colocated in the same set of GPUs, we further provide a +fine-grain optimization, which merge the ``worker_group`` of different roles +in the same process. This optimization can save the redundant +CUDA/distributed context in different processes. + +.. code:: python + + # initialize WorkerGroup + # NOTE: if you want to use a different resource pool for each role, which can support different parallel size, + # you should not use `create_colocated_worker_cls`. Instead, directly pass different resource pool to different worker groups. + # See TODO(url) for more information. + all_wg = {} + for resource_pool, class_dict in self.resource_pool_to_cls.items(): + worker_dict_cls = create_colocated_worker_cls(class_dict=class_dict) + wg_dict = self.ray_worker_group_cls(resource_pool=resource_pool, ray_cls_with_init=worker_dict_cls) + spawn_wg = wg_dict.spawn(prefix_set=class_dict.keys()) + all_wg.update(spawn_wg) + + if self.use_critic: + self.critic_wg = all_wg['critic'] + self.critic_wg.init_model() + + if self.use_reference_policy: + self.ref_policy_wg = all_wg['ref'] + self.ref_policy_wg.init_model() + + if self.use_rm: + self.rm_wg = all_wg['rm'] + self.rm_wg.init_model() + + # we should create rollout at the end so that vllm can have a better estimation of kv cache memory + self.actor_rollout_wg = all_wg['actor_rollout'] + self.actor_rollout_wg.init_model() + +.. note:: For megatron backend, if we merge the ``worker_groups`` into the same processes, all the roles will utilize the same 3D parallel size. To optimize this, we may need to maintain several 3D process groups for each role in the same distributed context. If you want to use different 3D parallel size for different roles, please follow the similar architecture of the first code block to initialize each role's ``worker_group`` + + +PPO Training Loop +----------------- + +We implement the PPO training loop by calling the functions in +worker_group of each role. The input and output data of each function is +a ``DataProto`` object implemented in `protocol.py `_. In the training +loop, trainer will dispatch/collect the data to/from different GPUs +following the transfer protocols wrapped in the workers' functions. The +computation of PPO micro batches is processed in ``update_actor`` and +``update_critic`` functions. + +To extend to other RLHF algorithms, such as DPO, GRPO, please refer to +:doc:`../advance/dpo_extension`. + +.. code:: python + + def fit(self): + """ + The training loop of PPO. + The driver process only need to call the compute functions of the worker group through RPC to construct the PPO dataflow. + The light-weight advantage computation is done on the driver process. + """ + from verl.utils.tracking import Tracking + from omegaconf import OmegaConf + + logger = Tracking(project_name=self.config.trainer.project_name, + experiment_name=self.config.trainer.experiment_name, + default_backend=self.config.trainer.logger, + config=OmegaConf.to_container(self.config, resolve=True)) + + global_steps = 0 + + # perform validation before training + # currently, we only support validation using the reward_function. + if self.val_reward_fn is not None: + val_metrics = self._validate() + pprint(f'Initial validation metrics: {val_metrics}') + + for epoch in range(self.config.trainer.total_epochs): + for batch_dict in self.train_dataloader: + metrics = {} + + batch: DataProto = DataProto.from_single_dict(batch_dict) + # batch = batch.to('cuda') + + # pop those keys for generation + gen_batch = batch.pop(batch_keys=['input_ids', 'attention_mask', 'position_ids']) + + # generate a batch + with Timer(name='gen', logger=None) as timer: + gen_batch_output = self.actor_rollout_wg.generate_sequences(gen_batch) + metrics['timing/gen'] = timer.last + + batch = batch.union(gen_batch_output) + + if self.use_reference_policy: + # compute reference log_prob + with Timer(name='ref', logger=None) as timer: + ref_log_prob = self.ref_policy_wg.compute_ref_log_prob(batch) + batch = batch.union(ref_log_prob) + metrics['timing/ref'] = timer.last + + # compute values + with Timer(name='values', logger=None) as timer: + values = self.critic_wg.compute_values(batch) + batch = batch.union(values) + metrics['timing/values'] = timer.last + + with Timer(name='adv', logger=None) as timer: + # compute scores. Support both model and function-based. + # We first compute the scores using reward model. Then, we call reward_fn to combine + # the results from reward model and rule-based results. + if self.use_rm: + # we first compute reward model score + reward_tensor = self.rm_wg.compute_rm_score(batch) + batch = batch.union(reward_tensor) + + # we combine with rule-based rm + reward_tensor = self.reward_fn(batch) + batch.batch['token_level_scores'] = reward_tensor + + # compute rewards. apply_kl_penalty if available + batch, kl_metrics = apply_kl_penalty(batch, + kl_ctrl=self.kl_ctrl, + kl_penalty=self.config.algorithm.kl_penalty) + metrics.update(kl_metrics) + + # compute advantages, executed on the driver process + batch = compute_advantage(batch, + self.config.algorithm.gamma, + self.config.algorithm.lam, + adv_estimator=self.config.algorithm.adv_estimator) + metrics['timing/adv'] = timer.last + + # update critic + if self.use_critic: + with Timer(name='update_critic', logger=None) as timer: + critic_output = self.critic_wg.update_critic(batch) + metrics['timing/update_critic'] = timer.last + critic_output_metrics = reduce_metrics(critic_output.meta_info['metrics']) + metrics.update(critic_output_metrics) + + # implement critic warmup + if self.config.trainer.critic_warmup <= global_steps: + # update actor + with Timer(name='update_actor', logger=None) as timer: + actor_output = self.actor_rollout_wg.update_actor(batch) + metrics['timing/update_actor'] = timer.last + actor_output_metrics = reduce_metrics(actor_output.meta_info['metrics']) + metrics.update(actor_output_metrics) + + # validate + if self.val_reward_fn is not None and (global_steps + 1) % self.config.trainer.test_freq == 0: + with Timer(name='testing', logger=None) as timer: + val_metrics: dict = self._validate() + val_metrics = {f'val/{key}': val for key, val in val_metrics.items()} + metrics['timing/testing'] = timer.last + metrics.update(val_metrics) + + # collect metrics + data_metrics = compute_data_metrics(batch=batch) + metrics.update(data_metrics) + + # TODO: make a canonical logger that supports various backend + logger.log(data=metrics, step=global_steps) + + if self.config.trainer.save_freq > 0 and (global_steps + 1) % self.config.trainer.save_freq == 0: + actor_local_path = os.path.join(self.config.trainer.default_local_dir, 'actor', + f'global_step_{global_steps}') + actor_remote_path = os.path.join(self.config.trainer.default_hdfs_dir, 'actor') + self.actor_rollout_wg.save_checkpoint(actor_local_path, actor_remote_path) + + if self.use_critic: + critic_local_path = os.path.join(self.config.trainer.default_local_dir, 'critic', + f'global_step_{global_steps}') + critic_remote_path = os.path.join(self.config.trainer.default_hdfs_dir, 'critic') + self.critic_wg.save_checkpoint(critic_local_path, critic_remote_path) + + global_steps += 1 + + # perform validation after training + if self.val_reward_fn is not None: + val_metrics = self._validate() + pprint(f'Final validation metrics: {val_metrics}') diff --git a/frameworks/agentgym_rl/pyproject.toml b/frameworks/agentgym_rl/pyproject.toml new file mode 100644 index 0000000000000000000000000000000000000000..fd5692905131116e73a8ff7e0b6e056170d90eb4 --- /dev/null +++ b/frameworks/agentgym_rl/pyproject.toml @@ -0,0 +1,86 @@ +# ------------------------------- +# build-system +# ------------------------------- +[build-system] +requires = [ + "setuptools>=61.0", + "wheel" +] +build-backend = "setuptools.build_meta" + +# ------------------------------- +# project (PEP 621 metadata) +# ------------------------------- +[project] +name = "verl" +# We'll mark the version as "dynamic" because it's read from the file "verl/version/version" +# (PEP 621 calls this "dynamic version"). +# The actual version is specified in the [tool.setuptools.dynamic] section below. +dynamic = ["version"] + +description = "verl: Volcano Engine Reinforcement Learning for LLM" +license = {file = "LICENSE"} # or "Apache-2.0", if you prefer an SPDX identifier +readme = {file = "README.md", content-type = "text/markdown"} +requires-python = ">=3.8" + +authors = [ + { name = "Bytedance - Seed - MLSys", email = "zhangchi.usc1992@bytedance.com" }, + { name = "Bytedance - Seed - MLSys", email = "gmsheng@connect.hku.hk" }, +] + +# Dependencies corresponding to install_requires in setup.py +dependencies = [ + "accelerate", + "codetiming", + "datasets", + "dill", + "hydra-core", + "numpy", + "pandas", + "peft", + "pyarrow>=15.0.0", + "pybind11", + "pylatexenc", + "ray>=2.10", + "tensordict<0.6", + "transformers", + "vllm<=0.6.3", + 'wandb', + 'torch==2.4.0', +] + +# Optional dependencies (extras_require in setup.py) +[project.optional-dependencies] +test = [ + "pytest", "yapf", "py-spy", +] +prime = ["pyext"] +gpu = ["liger-kernel", "flash-attn"] + +# URLs +[project.urls] +Homepage = "https://github.com/volcengine/verl" + +# ------------------------------- +# tool.setuptools - Additional config +# ------------------------------- +[tool.setuptools] +# True means `setuptools` will attempt to include all relevant files in package_data automatically. +# This corresponds to `include_package_data=True` in setup.py. +include-package-data = true + +# We read the version from a file in 'verl/version/version' +[tool.setuptools.dynamic] +version = {file = "verl/version/version"} + +# If you need to mimic `package_dir={'': '.'}`: +[tool.setuptools.package-dir] +"" = "." + +# If you need to include specific non-Python data (like YAML files or version file): +# This is the rough equivalent of package_data={'': ['version/*'], 'verl': ['trainer/config/*.yaml']} +[tool.setuptools.package-data] +verl = [ + "version/*", + "trainer/config/*.yaml" +] diff --git a/frameworks/agentgym_rl/requirements.txt b/frameworks/agentgym_rl/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..d30a8218d60372cc17cf7b0ef7e80d3c445538d6 --- /dev/null +++ b/frameworks/agentgym_rl/requirements.txt @@ -0,0 +1,19 @@ +# requirements.txt records the full set of dependencies for development +accelerate +codetiming +datasets +dill +flash-attn +hydra-core +liger-kernel +numpy +pandas +peft +pyarrow>=15.0.0 +pybind11 +pylatexenc +ray +tensordict<0.6 +transformers +vllm<=0.6.3 +wandb diff --git a/frameworks/agentgym_rl/scripts/format.sh b/frameworks/agentgym_rl/scripts/format.sh new file mode 100644 index 0000000000000000000000000000000000000000..ed49d6f116b0fe5152cbe6c2ab5c36c4688f79b1 --- /dev/null +++ b/frameworks/agentgym_rl/scripts/format.sh @@ -0,0 +1,3 @@ +#!/bin/bash +pip3 install --upgrade yapf +python3 -m yapf -ir -vv --style ./.style.yapf verl tests single_controller examples diff --git a/frameworks/agentgym_rl/scripts/model_merger.py b/frameworks/agentgym_rl/scripts/model_merger.py new file mode 100644 index 0000000000000000000000000000000000000000..1595a9bec0e493ef30df17972e9ec754ab5af136 --- /dev/null +++ b/frameworks/agentgym_rl/scripts/model_merger.py @@ -0,0 +1,170 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import List, Tuple, Dict +import re +import os +import torch +import argparse +from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification +from concurrent.futures import ThreadPoolExecutor +from torch.distributed._tensor import DTensor, Shard, Placement + + +def merge_by_placement(tensors: List[torch.Tensor], placement: Placement): + if placement.is_replicate(): + return tensors[0] + elif placement.is_partial(): + raise NotImplementedError("Partial placement is not supported yet") + elif placement.is_shard(): + return torch.cat(tensors, dim=placement.dim).contiguous() + else: + raise ValueError(f"Unsupported placement: {placement}") + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model") + parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload") + args = parser.parse_args() + + assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface" + local_dir = args.local_dir + + # copy rank zero to find the shape of (dp, fsdp) + rank = 0 + world_size = 0 + for filename in os.listdir(local_dir): + match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename) + if match: + world_size = match.group(1) + break + assert world_size, "No model file with the proper format" + + state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu') + pivot_key = sorted(list(state_dict.keys()))[0] + weight = state_dict[pivot_key] + assert isinstance(weight, torch.distributed._tensor.DTensor) + # get sharding info + device_mesh = weight.device_mesh + mesh = device_mesh.mesh + mesh_dim_names = device_mesh.mesh_dim_names + + print(f'Got device mesh {mesh}, mesh_dim_names {mesh_dim_names}') + + assert mesh_dim_names in ( + ('fsdp',), + ), f'Unsupported mesh_dim_names {mesh_dim_names}' + + if 'tp' in mesh_dim_names: + # fsdp * tp + total_shards = mesh.shape[-1] * mesh.shape[-2] + mesh_shape = (mesh.shape[-2], mesh.shape[-1]) + else: + # fsdp + total_shards = mesh.shape[-1] + mesh_shape = (mesh.shape[-1],) + + print(f'Processing model shards with {total_shards} {mesh_shape} in total') + + model_state_dict_lst = [] + model_state_dict_lst.append(state_dict) + model_state_dict_lst.extend([""] * (total_shards - 1)) + + def process_one_shard(rank): + model_path = os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt') + state_dict = torch.load(model_path, map_location='cpu', weights_only=False) + model_state_dict_lst[rank] = state_dict + return state_dict + + with ThreadPoolExecutor(max_workers=min(32, os.cpu_count())) as executor: + for rank in range(1, total_shards): + executor.submit(process_one_shard, rank) + state_dict = {} + param_placements: Dict[str, List[Placement]] = {} + keys = set(model_state_dict_lst[0].keys()) + for key in keys: + state_dict[key] = [] + for model_state_dict in model_state_dict_lst: + try: + tensor = model_state_dict.pop(key) + except: + print("-"*30) + print(model_state_dict) + if isinstance(tensor, DTensor): + state_dict[key].append(tensor._local_tensor.bfloat16()) + placements = tuple(tensor.placements) + # replicated placement at dp dimension can be discarded + if mesh_dim_names[0] == 'dp': + placements = placements[1:] + if key not in param_placements: + param_placements[key] = placements + else: + assert param_placements[key] == placements + else: + state_dict[key] = tensor.bfloat16() + + del model_state_dict_lst + + for key in sorted(state_dict): + if not isinstance(state_dict[key], list): + print(f"No need to merge key {key}") + continue + # merge shards + placements: Tuple[Shard] = param_placements[key] + if len(mesh_shape) == 1: + # 1-D list, FSDP without TP + assert len(placements) == 1 + shards = state_dict[key] + state_dict[key] = merge_by_placement(shards, placements[0]) + else: + # 2-D list, FSDP + TP + raise NotImplementedError("FSDP + TP is not supported yet") + + print('Writing to local disk') + hf_path = os.path.join(local_dir, 'huggingface') + config = AutoConfig.from_pretrained(hf_path) + + if 'ForTokenClassification' in config.architectures[0]: + auto_model = AutoModelForTokenClassification + elif 'ForCausalLM' in config.architectures[0]: + auto_model = AutoModelForCausalLM + else: + raise NotImplementedError(f'Unknown architecture {config["architectures"]}') + + with torch.device('meta'): + model = auto_model.from_config(config, torch_dtype=torch.bfloat16) + model.to_empty(device='cpu') + + print(f'Saving model to {hf_path}') + model.save_pretrained(hf_path, state_dict=state_dict) + del state_dict + del model + if args.hf_upload_path: + # Push to hugging face + from huggingface_hub import HfApi + api = HfApi() + api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True) + api.upload_folder( + folder_path=hf_path, + repo_id=args.hf_upload_path, + repo_type="model" + ) + + + + + + + diff --git a/frameworks/agentgym_rl/scripts/multiple_model_merger.py b/frameworks/agentgym_rl/scripts/multiple_model_merger.py new file mode 100644 index 0000000000000000000000000000000000000000..c5a0addd2752926b110d0a4829065e76fe37a4e6 --- /dev/null +++ b/frameworks/agentgym_rl/scripts/multiple_model_merger.py @@ -0,0 +1,178 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import List, Tuple, Dict +import re +import os +import torch +import argparse +from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification +from concurrent.futures import ThreadPoolExecutor +from torch.distributed._tensor import DTensor, Shard, Placement + +import shutil + + +def merge_by_placement(tensors: List[torch.Tensor], placement: Placement): + if placement.is_replicate(): + return tensors[0] + elif placement.is_partial(): + raise NotImplementedError("Partial placement is not supported yet") + elif placement.is_shard(): + return torch.cat(tensors, dim=placement.dim).contiguous() + else: + raise ValueError(f"Unsupported placement: {placement}") + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model") + parser.add_argument('--save_dir', default=None, type = str, help="The path to save your model") + parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload") + args = parser.parse_args() + + assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface" + local_dir = args.local_dir + save_dir = args.save_dir + + # copy rank zero to find the shape of (dp, fsdp) + rank = 0 + world_size = 0 + for filename in os.listdir(local_dir): + match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename) + if match: + world_size = match.group(1) + break + assert world_size, "No model file with the proper format" + + state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu') + pivot_key = sorted(list(state_dict.keys()))[0] + weight = state_dict[pivot_key] + assert isinstance(weight, torch.distributed._tensor.DTensor) + # get sharding info + device_mesh = weight.device_mesh + mesh = device_mesh.mesh + mesh_dim_names = device_mesh.mesh_dim_names + + print(f'Got device mesh {mesh}, mesh_dim_names {mesh_dim_names}') + + assert mesh_dim_names in ( + ('fsdp',), + ), f'Unsupported mesh_dim_names {mesh_dim_names}' + + if 'tp' in mesh_dim_names: + # fsdp * tp + total_shards = mesh.shape[-1] * mesh.shape[-2] + mesh_shape = (mesh.shape[-2], mesh.shape[-1]) + else: + # fsdp + total_shards = mesh.shape[-1] + mesh_shape = (mesh.shape[-1],) + + print(f'Processing model shards with {total_shards} {mesh_shape} in total') + + model_state_dict_lst = [] + model_state_dict_lst.append(state_dict) + model_state_dict_lst.extend([""] * (total_shards - 1)) + + def process_one_shard(rank): + model_path = os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt') + state_dict = torch.load(model_path, map_location='cpu', weights_only=False) + model_state_dict_lst[rank] = state_dict + return state_dict + + with ThreadPoolExecutor(max_workers=min(32, os.cpu_count())) as executor: + for rank in range(1, total_shards): + executor.submit(process_one_shard, rank) + state_dict = {} + param_placements: Dict[str, List[Placement]] = {} + keys = set(model_state_dict_lst[0].keys()) + for key in keys: + state_dict[key] = [] + for model_state_dict in model_state_dict_lst: + try: + tensor = model_state_dict.pop(key) + except: + print("-"*30) + print(model_state_dict) + if isinstance(tensor, DTensor): + state_dict[key].append(tensor._local_tensor.bfloat16()) + placements = tuple(tensor.placements) + # replicated placement at dp dimension can be discarded + if mesh_dim_names[0] == 'dp': + placements = placements[1:] + if key not in param_placements: + param_placements[key] = placements + else: + assert param_placements[key] == placements + else: + state_dict[key] = tensor.bfloat16() + + del model_state_dict_lst + + for key in sorted(state_dict): + if not isinstance(state_dict[key], list): + print(f"No need to merge key {key}") + continue + # merge shards + placements: Tuple[Shard] = param_placements[key] + if len(mesh_shape) == 1: + # 1-D list, FSDP without TP + assert len(placements) == 1 + shards = state_dict[key] + state_dict[key] = merge_by_placement(shards, placements[0]) + else: + # 2-D list, FSDP + TP + raise NotImplementedError("FSDP + TP is not supported yet") + + print('Writing to local disk') + if save_dir and os.path.abspath(save_dir) != os.path.abspath(local_dir): + hf_path = os.path.join(save_dir, 'huggingface') + shutil.copytree(os.path.join(local_dir, 'huggingface'), hf_path) + else: + hf_path = os.path.join(local_dir, 'huggingface') + config = AutoConfig.from_pretrained(hf_path) + + if 'ForTokenClassification' in config.architectures[0]: + auto_model = AutoModelForTokenClassification + elif 'ForCausalLM' in config.architectures[0]: + auto_model = AutoModelForCausalLM + else: + raise NotImplementedError(f'Unknown architecture {config["architectures"]}') + + with torch.device('meta'): + model = auto_model.from_config(config, torch_dtype=torch.bfloat16) + model.to_empty(device='cpu') + + print(f'Saving model to {hf_path}') + model.save_pretrained(hf_path, state_dict=state_dict) + del state_dict + del model + if args.hf_upload_path: + # Push to hugging face + from huggingface_hub import HfApi + api = HfApi() + api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True) + api.upload_folder( + folder_path=hf_path, + repo_id=args.hf_upload_path, + repo_type="model" + ) + + + + + + + diff --git a/frameworks/agentgym_rl/scripts/single_model_merger.py b/frameworks/agentgym_rl/scripts/single_model_merger.py new file mode 100644 index 0000000000000000000000000000000000000000..55bfbaa9f274b9b10992e737307747c252ab4424 --- /dev/null +++ b/frameworks/agentgym_rl/scripts/single_model_merger.py @@ -0,0 +1,86 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import re +import os +import torch +import argparse +from transformers import AutoConfig, AutoModelForCausalLM, AutoModelForTokenClassification + +import shutil + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--local_dir', required=True, type = str, help="The path for your saved model") + parser.add_argument('--save_dir', default=None, type = str, help="The path to save your model") + parser.add_argument("--hf_upload_path", default=False, type = str, help="The path of the huggingface repo to upload") + args = parser.parse_args() + + assert not args.local_dir.endswith("huggingface"), "The local_dir should not end with huggingface" + local_dir = args.local_dir + save_dir = args.save_dir + + # copy rank zero to find the shape of (dp, fsdp) + rank = 0 + world_size = 0 + for filename in os.listdir(local_dir): + match = re.match(r"model_world_size_(\d+)_rank_0\.pt", filename) + if match: + world_size = match.group(1) + break + assert world_size, "No model file with the proper format" + + state_dict = torch.load(os.path.join(local_dir, f'model_world_size_{world_size}_rank_{rank}.pt'), map_location='cpu') + + print('Writing to local disk') + if save_dir and os.path.abspath(save_dir) != os.path.abspath(local_dir): + hf_path = os.path.join(save_dir, 'huggingface') + shutil.copytree(os.path.join(local_dir, 'huggingface'), hf_path) + else: + hf_path = os.path.join(local_dir, 'huggingface') + config = AutoConfig.from_pretrained(hf_path) + + if 'ForTokenClassification' in config.architectures[0]: + auto_model = AutoModelForTokenClassification + elif 'ForCausalLM' in config.architectures[0]: + auto_model = AutoModelForCausalLM + else: + raise NotImplementedError(f'Unknown architecture {config["architectures"]}') + + with torch.device('meta'): + model = auto_model.from_config(config, torch_dtype=torch.bfloat16) + model.to_empty(device='cpu') + + print(f'Saving model to {hf_path}') + model.save_pretrained(hf_path, state_dict=state_dict) + del state_dict + del model + if args.hf_upload_path: + # Push to hugging face + from huggingface_hub import HfApi + api = HfApi() + api.create_repo(repo_id=args.hf_upload_path, private=False, exist_ok=True) + api.upload_folder( + folder_path=hf_path, + repo_id=args.hf_upload_path, + repo_type="model" + ) + + + + + + + diff --git a/frameworks/agentgym_rl/setup.py b/frameworks/agentgym_rl/setup.py new file mode 100644 index 0000000000000000000000000000000000000000..9744dcf897e8394d1ebd257096e4262cd721e713 --- /dev/null +++ b/frameworks/agentgym_rl/setup.py @@ -0,0 +1,74 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# setup.py is the fallback installation script when pyproject.toml does not work +from setuptools import setup, find_packages +import os + +version_folder = os.path.dirname(os.path.join(os.path.abspath(__file__))) + +with open(os.path.join(version_folder, 'verl/version/version')) as f: + __version__ = f.read().strip() + +install_requires = [ + 'accelerate', + 'codetiming', + 'datasets', + 'dill', + 'hydra-core', + 'numpy', + 'pandas', + 'peft', + 'pyarrow>=15.0.0', + 'pybind11', + 'pylatexenc', + 'ray>=2.10', + 'tensordict<0.6', + 'transformers', + 'vllm<=0.6.3', + 'wandb', +] + +TEST_REQUIRES = ['pytest', 'yapf', 'py-spy'] +PRIME_REQUIRES = ['pyext'] +GPU_REQUIRES = ['liger-kernel', 'flash-attn'] + +extras_require = { + 'test': TEST_REQUIRES, + 'prime': PRIME_REQUIRES, + 'gpu': GPU_REQUIRES, +} + +from pathlib import Path +this_directory = Path(__file__).parent +long_description = (this_directory / "VERL_README.md").read_text() + +setup( + name='verl', + version=__version__, + package_dir={'': '.'}, + packages=find_packages(where='.'), + url='https://github.com/volcengine/verl', + license='Apache 2.0', + author='Bytedance - Seed - MLSys', + author_email='zhangchi.usc1992@bytedance.com, gmsheng@connect.hku.hk', + description='verl: Volcano Engine Reinforcement Learning for LLM', + install_requires=install_requires, + extras_require=extras_require, + package_data={'': ['version/*'], + 'verl': ['trainer/config/*.yaml'],}, + include_package_data=True, + long_description=long_description, + long_description_content_type='text/markdown' +) \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/__init__.py b/frameworks/agentgym_rl/verl/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..f068717761543cde8dd59ad08b42465160893bb3 --- /dev/null +++ b/frameworks/agentgym_rl/verl/__init__.py @@ -0,0 +1,27 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +version_folder = os.path.dirname(os.path.join(os.path.abspath(__file__))) + +with open(os.path.join(version_folder, 'version/version')) as f: + __version__ = f.read().strip() + +from .protocol import DataProto + +from .utils.logging_utils import set_basic_config +import logging + +set_basic_config(level=logging.WARNING) diff --git a/frameworks/agentgym_rl/verl/agent_trainer/__init__.py b/frameworks/agentgym_rl/verl/agent_trainer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/agent_trainer/config/evaluation.yaml b/frameworks/agentgym_rl/verl/agent_trainer/config/evaluation.yaml new file mode 100644 index 0000000000000000000000000000000000000000..8bf3e5bf6f2278ff90bb48b6764541bc064e2c52 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/config/evaluation.yaml @@ -0,0 +1,3 @@ +data: + path: null + reward_model_key: rewards \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/agent_trainer/config/generation.yaml b/frameworks/agentgym_rl/verl/agent_trainer/config/generation.yaml new file mode 100644 index 0000000000000000000000000000000000000000..dc5ff6ca7020fbe225f04f97b1a77a64ad0a9f45 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/config/generation.yaml @@ -0,0 +1,85 @@ +trainer: + nnodes: 1 + n_gpus_per_node: 8 + +data: + path: null + max_prompt_length: 1024 + max_response_length: 15360 + prompt_key: item_id + n_samples: 1 + output_path: null + batch_size: 16 + +agentgym: + task_name: textcraft + env_addr: 'http://localhost:5000' + max_retries: 10 + max_rounds: 10 + timeout: 300 + +model: + path: null + external_lib: null +rollout: + name: vllm + temperature: 1.0 + top_k: -1 # 0 for hf rollout, -1 for vllm rollout + top_p: 1 + prompt_length: ${data.max_prompt_length} # not use for opensource + response_length: ${data.max_response_length} + # for vllm rollout + dtype: bfloat16 # should align with FSDP + gpu_memory_utilization: 0.5 + ignore_eos: False + enforce_eager: True + free_cache_engine: True + load_format: dummy_dtensor + tensor_model_parallel_size: 2 + max_num_batched_tokens: 8192 + max_model_len: 32768 + max_num_seqs: 1024 + log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu + log_prob_micro_batch_size_per_gpu: 1 + log_prob_use_dynamic_bsz: ${actor.use_dynamic_bsz} + log_prob_max_token_len_per_gpu: ${actor.ppo_max_token_len_per_gpu} + max_tokens: 1024 + disable_log_stats: True + enable_chunked_prefill: True # may get higher throughput when set to True. When activated, Please increase max_num_batched_tokens or decrease max_model_len. + # for hf rollout + do_sample: True + # number of responses (i.e. num sample times) + n: 1 # > 1 for grpo + send_interval: 1 + rollout_log_dir: null + +actor: + strategy: fsdp # This is for backward-compatibility + ppo_mini_batch_size: 16 + ppo_micro_batch_size: null # will be deprecated, use ppo_micro_batch_size_per_gpu + ppo_micro_batch_size_per_gpu: 1 + use_dynamic_bsz: False + ppo_max_token_len_per_gpu: 16384 # n * ${data.max_prompt_length} + ${data.max_response_length} + grad_clip: 1.0 + clip_ratio: 0.2 + entropy_coeff: 0.001 + use_kl_loss: null # True for GRPO + kl_loss_coef: 0.001 # for grpo + kl_loss_type: low_var_kl # for grpo + ppo_epochs: 1 + shuffle: False + ulysses_sequence_parallel_size: 1 # sp size + optim: + lr: 1e-6 + lr_warmup_steps_ratio: 0. # the total steps will be injected during runtime + min_lr_ratio: null # only useful for warmup with cosine + warmup_style: constant # select from constant/cosine + total_training_steps: -1 # must be override by program + fsdp_config: + wrap_policy: + # transformer_layer_cls_to_wrap: None + min_num_params: 0 + param_offload: False + grad_offload: False + optimizer_offload: False + fsdp_size: -1 \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/agent_trainer/config/ppo_trainer.yaml b/frameworks/agentgym_rl/verl/agent_trainer/config/ppo_trainer.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2ba2704ddb4ca62e8c86f8c449e36d2bc1931716 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/config/ppo_trainer.yaml @@ -0,0 +1,198 @@ +data: + tokenizer: null + train_file: ~/data/rlhf/gsm8k/train.parquet + val_files: ~/data/rlhf/gsm8k/test.parquet + prompt_key: item_id + max_prompt_length: 2048 # Maximum length of the general task description prompt in the first turn. + max_response_length: 2048 # Maximum total token length of the interaction trajectory (excluding the task prompt). + train_batch_size: 1024 + val_batch_size: null + return_raw_input_ids: False # This should be set to true when the tokenizer between policy and rm differs + return_raw_chat: True + shuffle: True + +actor_rollout_ref: + hybrid_engine: True + model: + path: ~/models/deepseek-llm-7b-chat + external_lib: null + override_config: { } + enable_gradient_checkpointing: True + use_remove_padding: False + actor: + strategy: fsdp # This is for backward-compatibility + ppo_mini_batch_size: 256 + ppo_micro_batch_size: null # will be deprecated, use ppo_micro_batch_size_per_gpu + ppo_micro_batch_size_per_gpu: null + use_dynamic_bsz: False + ppo_max_token_len_per_gpu: 16384 # n * ${data.max_prompt_length} + ${data.max_response_length} + grad_clip: 1.0 + clip_ratio: 0.2 + entropy_coeff: 0.001 + use_kl_loss: False # True for GRPO + kl_loss_coef: 0.001 # for grpo + kl_loss_type: low_var_kl # for grpo + ppo_epochs: 1 + shuffle: False + ulysses_sequence_parallel_size: 1 # sp size + optim: + lr: 1e-6 + lr_warmup_steps_ratio: 0. # the total steps will be injected during runtime + min_lr_ratio: null # only useful for warmup with cosine + warmup_style: constant # select from constant/cosine + total_training_steps: -1 # must be override by program + fsdp_config: + wrap_policy: + # transformer_layer_cls_to_wrap: None + min_num_params: 0 + param_offload: False + grad_offload: False + optimizer_offload: False + fsdp_size: -1 + ref: + fsdp_config: + param_offload: False + wrap_policy: + # transformer_layer_cls_to_wrap: None + min_num_params: 0 + log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu + log_prob_micro_batch_size_per_gpu: 1 + log_prob_use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz} + log_prob_max_token_len_per_gpu: ${actor_rollout_ref.actor.ppo_max_token_len_per_gpu} + ulysses_sequence_parallel_size: ${actor_rollout_ref.actor.ulysses_sequence_parallel_size} # sp size + agentgym: + task_name: textcraft # Training task name of AgentGym. + env_addr: 'http://localhost:5000' # URL of the AgentGym environment server. + max_retries: 10 # Maximum retry times. + max_rounds: 10 # Maximum token length of a single response per turn. + timeout: 300 # timeout seconds. + rollout: + name: vllm + temperature: 1.0 + top_k: -1 # 0 for hf rollout, -1 for vllm rollout + top_p: 1 + prompt_length: ${data.max_prompt_length} # not use for opensource + response_length: ${data.max_response_length} + # for vllm rollout + dtype: bfloat16 # should align with FSDP + gpu_memory_utilization: 0.5 + ignore_eos: False + enforce_eager: True + free_cache_engine: True + load_format: dummy_dtensor + tensor_model_parallel_size: 2 + max_num_batched_tokens: 8192 + max_model_len: 32768 + max_num_seqs: 1024 + max_tokens: 1024 + log_prob_micro_batch_size: null # will be deprecated, use log_prob_micro_batch_size_per_gpu + log_prob_micro_batch_size_per_gpu: 1 + log_prob_use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz} + log_prob_max_token_len_per_gpu: ${actor_rollout_ref.actor.ppo_max_token_len_per_gpu} + disable_log_stats: True + enable_chunked_prefill: True # may get higher throughput when set to True. When activated, Please increase max_num_batched_tokens or decrease max_model_len. + # for hf rollout + do_sample: True + # number of responses (i.e. num sample times) + n: 1 # > 1 for grpo + send_interval: 1 # # interval when sending request + rollout_log_dir: null # Directory for storing rollout trajectories. + +critic: + rollout_n: ${actor_rollout_ref.rollout.n} + strategy: fsdp # [fsdp, fsdp2] + optim: + lr: 1e-5 + lr_warmup_steps_ratio: 0. # the total steps will be injected during runtime + min_lr_ratio: null # only useful for warmup with cosine + warmup_style: constant # select from constant/cosine + total_training_steps: -1 # must be override by program + model: + path: ~/models/deepseek-llm-7b-chat + tokenizer_path: ${actor_rollout_ref.model.path} + override_config: { } + external_lib: ${actor_rollout_ref.model.external_lib} + enable_gradient_checkpointing: True + use_remove_padding: False + fsdp_config: + param_offload: False + grad_offload: False + optimizer_offload: False + wrap_policy: + # transformer_layer_cls_to_wrap: None + min_num_params: 0 + fsdp_size: -1 + ppo_mini_batch_size: ${actor_rollout_ref.actor.ppo_mini_batch_size} + ppo_micro_batch_size: null # will be deprecated, use ppo_micro_batch_size_per_gpu + ppo_micro_batch_size_per_gpu: null + forward_micro_batch_size: ${critic.ppo_micro_batch_size} + forward_micro_batch_size_per_gpu: ${critic.ppo_micro_batch_size_per_gpu} + use_dynamic_bsz: ${actor_rollout_ref.actor.use_dynamic_bsz} + ppo_max_token_len_per_gpu: 32768 # (${actor_rollout_ref.actor.ppo_max_token_len_per_gpu}) * 2 + forward_max_token_len_per_gpu: ${critic.ppo_max_token_len_per_gpu} + ulysses_sequence_parallel_size: 1 # sp size + ppo_epochs: ${actor_rollout_ref.actor.ppo_epochs} + shuffle: ${actor_rollout_ref.actor.shuffle} + grad_clip: 1.0 + cliprange_value: 0.5 + +reward_model: + enable: False + strategy: fsdp + model: + input_tokenizer: ${actor_rollout_ref.model.path} # set this to null if the chat template is identical + path: ~/models/FsfairX-LLaMA3-RM-v0.1 + external_lib: ${actor_rollout_ref.model.external_lib} + use_remove_padding: False + fsdp_config: + min_num_params: 0 + param_offload: False + fsdp_size: -1 + micro_batch_size: null # will be deprecated, use micro_batch_size_per_gpu + micro_batch_size_per_gpu: null # set a number + max_length: null + ulysses_sequence_parallel_size: 1 # sp size + use_dynamic_bsz: ${critic.use_dynamic_bsz} + forward_max_token_len_per_gpu: ${critic.forward_max_token_len_per_gpu} + reward_manager: null + launch_reward_fn_async: False # custom reward function executed async on CPU, during log_prob + sandbox_fusion: + url: null # faas url to run code in cloud sandbox + max_concurrent: 64 # max concurrent requests to sandbox +custom_reward_function: + path: null + name: null + +algorithm: + gamma: 1.0 + lam: 1.0 + adv_estimator: gae + kl_penalty: kl # how to estimate kl divergence + rounds_ctrl: + type: fixed # Strategy for controlling the maximum number of interaction turns + rounds: 15 # Maximum number of allowed interaction turns. + steps_scaling_inter: 100 # Frequency (in training steps) to increase the maximum number of turns when using `scaling_inter_stepwise`. + kl_ctrl: + type: fixed + kl_coef: 0.001 + +trainer: + total_epochs: 30 + total_training_steps: null + project_name: verl_examples + experiment_name: gsm8k + logger: [ 'console', 'wandb' ] + val_generations_to_log_to_wandb: 0 + nnodes: 1 + n_gpus_per_node: 8 + save_freq: -1 + # auto: find the last ckpt to resume. If can't find, start from scratch + resume_mode: auto # or auto or resume_path if + resume_from_path: False + test_freq: -1 + critic_warmup: 0 + default_hdfs_dir: null + remove_previous_ckpt_in_save: False + del_local_ckpt_after_load: False + default_local_dir: checkpoints/${trainer.project_name}/${trainer.experiment_name} + storage_mode: local diff --git a/frameworks/agentgym_rl/verl/agent_trainer/config/sft_trainer.yaml b/frameworks/agentgym_rl/verl/agent_trainer/config/sft_trainer.yaml new file mode 100644 index 0000000000000000000000000000000000000000..431da9989d7cbd34c51f1edc2a47527e0dfc614c --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/config/sft_trainer.yaml @@ -0,0 +1,43 @@ +data: + train_batch_size: 16 + micro_batch_size: null # will be deprecated, use micro_batch_size_per_gpu + micro_batch_size_per_gpu: 2 # this is also val batch size + train_files: null + prompt_key: conversations + max_length: 4096 + truncation: right + balance_dp_token: False + chat_template: null +model: + partial_pretrain: null + fsdp_config: + wrap_policy: + min_num_params: 0 + cpu_offload: False + offload_params: False + external_lib: null + enable_gradient_checkpointing: False + trust_remote_code: False + lora_rank: 0 # Set to positive value to enable LoRA (e.g., 32) + lora_alpha: 16 # LoRA scaling factor + target_modules: all-linear # Target modules for LoRA adaptation + use_liger: False +optim: + lr: 1e-4 + betas: [0.9, 0.95] + weight_decay: 0.01 + warmup_steps_ratio: 0.1 + clip_grad: 1.0 +ulysses_sequence_parallel_size: 1 +use_remove_padding: False +trainer: + default_local_dir: null + resume_path: null + project_name: verl_agent_sft + experiment_name: verl_agent_sft + total_epochs: 1 + total_training_steps: null + logger: ['console'] + seed: 1 + storage_mode: local + diff --git a/frameworks/agentgym_rl/verl/agent_trainer/fsdp_sft_trainer.py b/frameworks/agentgym_rl/verl/agent_trainer/fsdp_sft_trainer.py new file mode 100644 index 0000000000000000000000000000000000000000..93b7c3e7e77a8048687a56e630590145cc36ce95 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/fsdp_sft_trainer.py @@ -0,0 +1,489 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +A lightweight one-file FSDP SFT Trainer +TODO(zhangchi.usc1992) +- Add calculation of mfu +- Add validation +""" + +import os + +os.environ['NCCL_DEBUG'] = 'WARN' +os.environ['TOKENIZERS_PARALLELISM'] = 'true' + +import logging +import re +from contextlib import nullcontext +import torch +import torch.distributed +from torch import nn, optim +from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, MixedPrecision, ShardingStrategy, CPUOffload +from tqdm import tqdm +from transformers import AutoModelForCausalLM, PreTrainedModel, AutoConfig +from verl.utils.torch_functional import get_cosine_schedule_with_warmup +from tensordict import TensorDict +from torch.utils.data import DataLoader, DistributedSampler +from flash_attn.bert_padding import pad_input, unpad_input, rearrange, index_first_axis + +from verl.utils.fsdp_utils import get_fsdp_wrap_policy, init_fn, get_init_weight_context_manager +from verl.utils.agent_dataset import SFTDataset +from verl.utils.fs import copy_local_path_from_hdfs +from verl.utils.tracking import Tracking +from verl.utils.ulysses import get_ulysses_sequence_parallel_world_size +from torch.distributed.device_mesh import DeviceMesh + +from verl.utils.debug import log_gpu_memory_usage +from peft import LoraConfig, TaskType, get_peft_model + +from verl.workers.sharding_manager import FSDPUlyssesShardingManager +from verl.utils.ulysses import ulysses_pad_and_slice_inputs, gather_outpus_and_unpad + +logger = logging.getLogger(__file__) +logger.setLevel(os.getenv('VERL_SFT_LOGGING_LEVEL', 'WARN')) + + +def extract_step(path): + match = re.search(r'global_step_(\d+)', path) + if match: + return int(match.group(1)) + return None + + +def convert_to_regular_types(obj): + """Convert Hydra configs and other special types to regular Python types.""" + from omegaconf import ListConfig, DictConfig + if isinstance(obj, (ListConfig, DictConfig)): + return {k: convert_to_regular_types(v) for k, v in obj.items()} if isinstance(obj, DictConfig) else list(obj) + elif isinstance(obj, (list, tuple)): + return [convert_to_regular_types(x) for x in obj] + elif isinstance(obj, dict): + return {k: convert_to_regular_types(v) for k, v in obj.items()} + return obj + + +class FSDPSFTTrainer(object): + + def __init__(self, config, device_mesh: DeviceMesh, ulysses_device_mesh: DeviceMesh): + self.config = config + self.device_mesh = device_mesh + self.ulysses_device_mesh = ulysses_device_mesh + self.sharding_manager = FSDPUlyssesShardingManager(self.ulysses_device_mesh) + # build tokenizer first + local_model_path = copy_local_path_from_hdfs(src=self.config.model.partial_pretrain, verbose=True) + from verl.utils import hf_tokenizer + self.tokenizer = hf_tokenizer(local_model_path, trust_remote_code=self.config.model.trust_remote_code) + if self.config.data.chat_template is not None: + raise ValueError('Apply Chat template from config is not supported yet.') + + # normalize dp size + self._normalize_config_bsz() + + # Set sequence parallel size + self.config.ulysses_sequence_parallel_size = getattr(self.config, 'ulysses_sequence_parallel_size', 1) + self.use_remove_padding = getattr(self.config, 'use_remove_padding', False) + if self.device_mesh.get_rank() == 0: + print(f'Using sequence parallel size: {self.config.ulysses_sequence_parallel_size}') + print(f'Using remove padding: {self.use_remove_padding}') + + self._build_dataloader() + # build model + self._build_model_optimizer() + + # TODO: add checkpoint manager + if self.device_mesh.get_rank() == 0: + print(self.config) + + def _normalize_config_bsz(self): + dp_size = self.device_mesh.size(0) if not self.ulysses_device_mesh else self.ulysses_device_mesh.size(0) + if self.device_mesh.get_rank() == 0: + print(f'Normalize batch size by dp {dp_size}') + + assert self.config.data.train_batch_size % dp_size == 0, f"Global batch size {self.config.data.train_batch_size} is not divisible by dp size {dp_size}" + + self.config.data.train_batch_size //= dp_size + + assert self.config.data.train_batch_size % self.config.data.micro_batch_size_per_gpu == 0 + + def _build_dataloader(self): + config = self.config + # build dataset + self.train_dataset = SFTDataset(json_file=config.data.train_files, + tokenizer=self.tokenizer, + prompt_key=config.data.prompt_key) + + # build dataloader + # Use data parallel rank and size instead of global rank and world size + + # If doing SP, we need to use the local rank and size + if self.config.ulysses_sequence_parallel_size > 1: + rank = self.ulysses_device_mesh.get_local_rank('dp') + world_size = self.ulysses_device_mesh.size(0) + if self.ulysses_device_mesh.get_rank() == 0: + print(f'Using SP rank {rank} and size {world_size} for data distribution') + print(f'Each SP rank gets different data, but the same data WITHIN the same rank') + else: + rank = self.device_mesh.get_rank() + world_size = self.device_mesh.size() + if self.device_mesh.get_rank() == 0: + print(f'Using FSDP rank {rank} and size {world_size} for data distribution') + + self.train_sampler = DistributedSampler(self.train_dataset, + shuffle=True, + num_replicas=world_size, + rank=rank, + drop_last=True) + self.train_dataloader = DataLoader(dataset=self.train_dataset, + batch_size=config.data.train_batch_size, + sampler=self.train_sampler, + num_workers=8, + pin_memory=True, + drop_last=True) + + def _build_model_optimizer(self): + # TODO (zhangchi.usc1992): + # 1. support pretrain from random weights + # 2. support init directly from sharded weights + local_model_path = copy_local_path_from_hdfs(src=self.config.model.partial_pretrain, verbose=True) + + if self.config.model.get('external_lib', None) is not None: + # This is used to import external_lib into the huggingface systems + import importlib + importlib.import_module(self.config.model.external_lib) + + log_gpu_memory_usage('Before model allocation', logger=logger) + + trust_remote_code = self.config.model.trust_remote_code + # load config first + config = AutoConfig.from_pretrained(local_model_path, trust_remote_code=trust_remote_code) + if self.config.ulysses_sequence_parallel_size > 1: + assert self.use_remove_padding, "Sequence parallel is only supported when remove_padding is enabled" + from verl.models.registry import check_model_support_rmpad + check_model_support_rmpad(config.model_type) + + if self.use_remove_padding and self.config.ulysses_sequence_parallel_size > 1: + from verl.models.transformers.monkey_patch import apply_monkey_patch + apply_monkey_patch(config, verbose=True) + + # This may be very large + init_context = get_init_weight_context_manager(use_meta_tensor=not config.tie_word_embeddings) + + with init_context(): + self.model: PreTrainedModel = AutoModelForCausalLM.from_pretrained(local_model_path, + config=config, + torch_dtype=torch.float32, + attn_implementation='flash_attention_2', + trust_remote_code=trust_remote_code) + + # Apply Liger kernel if use_liger is enabled + if self.config.model.get('use_liger', False): + from liger_kernel.transformers.monkey_patch import _apply_liger_kernel_to_instance + _apply_liger_kernel_to_instance(model=self.model) + + if self.config.model.get('lora_rank', 0) > 0: + self.model.enable_input_require_grads() + # Convert config to regular Python types before creating PEFT model + lora_config = { + 'task_type': TaskType.CAUSAL_LM, + 'r': self.config.model.lora_rank, + 'lora_alpha': self.config.model.lora_alpha, + 'target_modules': convert_to_regular_types(self.config.model.target_modules), + 'bias': "none" + } + self.model = get_peft_model(self.model, LoraConfig(**lora_config)) + + if self.config.model.enable_gradient_checkpointing: + self.model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={'use_reentrant': False}) + + log_gpu_memory_usage('After model allocation', logger=logger) + + mixed_precision = MixedPrecision(param_dtype=torch.bfloat16, + reduce_dtype=torch.float32, + buffer_dtype=torch.float32) + + auto_wrap_policy = get_fsdp_wrap_policy(self.model, + config=self.config.model.fsdp_config.wrap_policy, + is_lora=self.config.model.get('lora_rank', 0) > 0) + if self.device_mesh.get_rank() == 0: + print(auto_wrap_policy) + + if not self.config.model.fsdp_config.cpu_offload: + cpu_offload = None + else: + cpu_offload = CPUOffload(offload_params=self.config.model.fsdp_config.offload_params) + + self.fsdp_model = FSDP(module=self.model, + auto_wrap_policy=auto_wrap_policy, + param_init_fn=init_fn, + sharding_strategy=ShardingStrategy.FULL_SHARD, + mixed_precision=mixed_precision, + device_mesh=self.device_mesh, + sync_module_states=True, + device_id=torch.cuda.current_device(), + cpu_offload=cpu_offload, + use_orig_params=False) + + log_gpu_memory_usage('After FSDP wrapping', logger=logger) + + self.optimizer = optim.AdamW(self.fsdp_model.parameters(), + lr=self.config.optim.lr, + betas=self.config.optim.betas, + weight_decay=self.config.optim.weight_decay) + + log_gpu_memory_usage('After initialize optimizer', logger=logger) + + self.steps_per_epoch = len(self.train_dataloader) + self.total_steps = self.steps_per_epoch * self.config.trainer.total_epochs + + if self.device_mesh.get_rank() == 0: + print( + f'Number of steps/epoch {self.steps_per_epoch}, number of epochs {self.config.trainer.total_epochs}, total number of steps {self.total_steps}' + ) + + num_warmup_steps = int(self.total_steps * self.config.optim.warmup_steps_ratio) + + self.lr_scheduler = get_cosine_schedule_with_warmup(optimizer=self.optimizer, + num_warmup_steps=num_warmup_steps, + num_training_steps=self.total_steps) + + def _compute_loss_and_backward(self, batch, do_backward=True): + """Compute loss with optional sequence parallelism and remove padding features""" + use_sp = self.use_remove_padding and self.config.ulysses_sequence_parallel_size > 1 + + # Move inputs to GPU and prepare loss mask + input_ids = batch['input_ids'].cuda() + attention_mask = batch['attention_mask'].cuda() + position_ids = batch['position_ids'].cuda() + loss_mask = batch.pop('loss_mask')[:, :-1].reshape(-1).cuda() + loss_fct = nn.CrossEntropyLoss(reduction='none') + + # Context manager for sequence parallel if needed + context = self.sharding_manager if use_sp else nullcontext() + with context: + with torch.autocast(device_type='cuda', dtype=torch.bfloat16): + if not use_sp: + # Standard forward pass without sequence parallel + labels = input_ids[:, 1:].contiguous() + output = self.fsdp_model(input_ids=input_ids, + attention_mask=attention_mask, + position_ids=position_ids, + use_cache=False) + logits = output.logits + + shift_logits = logits[..., :-1, :].contiguous() + shift_labels = labels.contiguous() + # Flatten the tokens + shift_logits = shift_logits.view(-1, self.model.config.vocab_size) + shift_labels = shift_labels.view(-1) + # Enable model parallelism + shift_labels = shift_labels.to(shift_logits.device) + loss = loss_fct(shift_logits, shift_labels) + loss = loss * loss_mask.to(loss.device) + else: + # IMPORTANT: We have a big assumption here, so we can shard the SAME sequence across SP ranks + # i.e., each GPU has <1 sequence, and each SP group has 1 sequence + # 1. All SP ranks will receive the *SAME* batch + # 2. Different SP groups will receive *DIFFERENT* batches + # This is implemented by the DistributedSampler + + batch_size, seqlen = input_ids.shape + # Remove padding + input_ids_rmpad, indices, *_ = unpad_input(input_ids.unsqueeze(-1), + attention_mask) # input_ids_rmpad (total_nnz, ...) + input_ids_rmpad = input_ids_rmpad.transpose(0, 1) # (1, total_nnz) + + # Unpad position_ids to align rotary + position_ids_rmpad = index_first_axis(rearrange(position_ids.unsqueeze(-1), "b s ... -> (b s) ..."), + indices).transpose(0, 1) + + # Pad and slice inputs for sequence parallelism + input_ids_rmpad_sliced, position_ids_rmpad_padded, pad_size = ulysses_pad_and_slice_inputs( + input_ids_rmpad, position_ids_rmpad, sp_size=get_ulysses_sequence_parallel_world_size()) + # For computing loss + input_ids_rmpad_rolled = torch.roll(input_ids_rmpad, shifts=-1, dims=1) # (1, total_nnz) + input_ids_rmpad_rolled, _, _ = ulysses_pad_and_slice_inputs( + input_ids_rmpad_rolled, None, get_ulysses_sequence_parallel_world_size()) + input_ids_rmpad_rolled = input_ids_rmpad_rolled.squeeze(0) # ((total_nnz / sp) + pad) + + # Forward pass + output = self.fsdp_model( + input_ids=input_ids_rmpad_sliced, + attention_mask=None, # Not needed with flash attention varlen + position_ids=position_ids_rmpad_padded, + use_cache=False) + + # Compute loss locally then aggregate + logits_rmpad = output.logits.squeeze(0) + input_ids_rmpad_rolled = input_ids_rmpad_rolled.to(logits_rmpad.device) + loss = loss_fct(logits_rmpad, input_ids_rmpad_rolled) + # Gather and unpad for sequence parallelism + loss = gather_outpus_and_unpad(loss, gather_dim=0, unpad_dim=0, padding_size=pad_size) + + # This is the loss collected from all ulysses ranks + full_loss = pad_input(hidden_states=loss.unsqueeze(-1), + indices=indices, + batch=batch_size, + seqlen=seqlen) + full_loss = full_loss.squeeze(-1)[:, :-1] # Remove last token's loss + full_loss = full_loss.reshape(-1) + loss_mask = loss_mask.to(full_loss.device) + loss = full_loss * loss_mask + + valid_token_this_rank = torch.sum(loss_mask) + + if self.config.data.balance_dp_token: + torch.distributed.all_reduce(valid_token_this_rank) + dp_size = self.ulysses_device_mesh.size('dp') if use_sp else torch.distributed.get_world_size() + else: + dp_size = 1 + + loss = torch.sum(loss) / valid_token_this_rank * dp_size + + if do_backward: + loss.backward() + return loss + + def training_step(self, batch: TensorDict): + self.fsdp_model.train() + + log_gpu_memory_usage('Before optimizer zero_grad', logger=logger) + + self.optimizer.zero_grad() + + log_gpu_memory_usage('After optimizer zero_grad', logger=logger) + + micro_batches = batch.split(self.config.data.micro_batch_size_per_gpu) + n_micro_batches = len(micro_batches) + step_loss = 0 + for micro_batch in micro_batches: + loss = self._compute_loss_and_backward(batch=micro_batch) / n_micro_batches + step_loss += loss.item() + + self.fsdp_model.clip_grad_norm_(max_norm=self.config.optim.clip_grad) + + log_gpu_memory_usage('Before optimizer step', logger=logger) + + self.optimizer.step() + + log_gpu_memory_usage('After optimizer step', logger=logger) + + self.lr_scheduler.step() + + # reduce loss across dp ranks + lr = self.lr_scheduler.get_last_lr()[0] + + log_gpu_memory_usage('After offload weights', logger=logger) + + step_loss = torch.tensor(step_loss).cuda() + torch.distributed.all_reduce(step_loss, op=torch.distributed.ReduceOp.AVG) + return {'train/loss': step_loss.detach().item(), 'train/lr(1e-3)': lr * 1e3} + + def save_checkpoint(self, step): + # save checkpoint + from torch.distributed.fsdp import FullStateDictConfig, StateDictType + cfg = FullStateDictConfig(offload_to_cpu=True, rank0_only=True) + with FSDP.state_dict_type(self.fsdp_model, StateDictType.FULL_STATE_DICT, cfg): + state_dict = self.fsdp_model.state_dict() + + if self.config.trainer.storage_mode == 'aistudio': + from aistudio_checkpoint.aistudio_mnt_checkpointer import AistudioMntCheckpointer + ckpter = AistudioMntCheckpointer() + save_dir = ckpter.get_save_dir(step=step) + path = os.path.join(save_dir, f'global_step_{step}') + elif self.config.trainer.storage_mode == 'local': + path = os.path.join(self.config.trainer.default_local_dir, f'global_step_{step}') + else: + raise NotImplementedError + # save huggingface model + if self.device_mesh.get_rank() == 0: + os.makedirs(path, exist_ok=True) + self.model.save_pretrained(path, state_dict=state_dict) + self.tokenizer.save_pretrained(path) + if self.config.trainer.storage_mode == 'aistudio': + ckpter.commit(memo=self.config.trainer.experiment_name) + torch.distributed.barrier() + + def fit(self): + rank = self.device_mesh.get_rank() + + # TODO: add a unified tracking + if rank == 0: + tracking = Tracking(project_name=self.config.trainer.project_name, + experiment_name=self.config.trainer.experiment_name, + default_backend=self.config.trainer.logger) + + global_step = 0 + # compute the total training steps. + # the total training steps in SFT is mainly for early exit + total_training_steps = len(self.train_dataloader) * self.config.trainer.total_epochs + + if self.config.trainer.total_training_steps is not None: + total_training_steps = self.config.trainer.total_training_steps + + self.total_training_steps = total_training_steps + print(f'Total training steps: {self.total_training_steps}') + + # TODO (zhangchi.usc1992) add back checkpoint manager. Currently, it blocks when uploading to hdfs. So very slow. + + if self.config.trainer.total_epochs > 1 and self.config.trainer.storage_mode == 'aistudio': + self.save_checkpoint(step=global_step) + + for epoch in range(self.config.trainer.total_epochs): + self.train_sampler.set_epoch(epoch=epoch) + for data in tqdm(self.train_dataloader, + total=self.steps_per_epoch, + desc=f"Epoch {epoch+1}/{self.config.trainer.total_epochs}"): + data = TensorDict(data, batch_size=self.config.data.train_batch_size).cuda() + metric = self.training_step(data) + if rank == 0: + tracking.log(data=metric, step=global_step) + global_step += 1 + + # for early exit validation + if global_step >= self.total_training_steps: + torch.distributed.barrier() + + # Save final checkpoint + self.save_checkpoint(step=global_step) + return + + torch.distributed.barrier() + + # save checkpoint + self.save_checkpoint(step=global_step) + + +from verl.agent_trainer.fsdp_sft_trainer import FSDPSFTTrainer +import hydra + +from torch.distributed.device_mesh import init_device_mesh + +from verl.utils.distributed import initialize_global_process_group + + +@hydra.main(config_path='config', config_name='sft_trainer', version_base=None) +def main(config): + local_rank, rank, world_size = initialize_global_process_group() + + device_mesh = init_device_mesh(device_type='cuda', mesh_shape=(world_size,), mesh_dim_names=('fsdp',)) + dp_size = world_size // config.ulysses_sequence_parallel_size + ulysses_device_mesh = init_device_mesh(device_type='cuda', + mesh_shape=(dp_size, config.ulysses_sequence_parallel_size), + mesh_dim_names=('dp', 'sp')) + trainer = FSDPSFTTrainer(config=config, device_mesh=device_mesh, ulysses_device_mesh=ulysses_device_mesh) + trainer.fit() + + +if __name__ == '__main__': + main() diff --git a/frameworks/agentgym_rl/verl/agent_trainer/main_eval.py b/frameworks/agentgym_rl/verl/agent_trainer/main_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..e792c5d0a815e55c595222a82961731bb6a62a6c --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/main_eval.py @@ -0,0 +1,46 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Offline evaluate the performance of a generated file using reward model and ground truth verifier. +The input is a parquet file that contains N generated sequences and (optional) the ground truth. + +""" + +import hydra +from verl.utils.fs import copy_local_path_from_hdfs +import pandas as pd +import numpy as np + + +@hydra.main(config_path='config', config_name='evaluation', version_base=None) +def main(config): + local_path = copy_local_path_from_hdfs(config.data.path) + dataset = pd.read_json(local_path) + reward_model_data = dataset[config.data.reward_model_key] + + passes = 0 + avgs = 0 + + total = len(dataset) + + for i in range(total): + passes += np.max(reward_model_data[i]) + avgs += np.mean(reward_model_data[i]) + + print(f'pass@{len(reward_model_data[0])}: {passes / total}') + print(f'avg@{len(reward_model_data[0])}: {avgs / total}') + + +if __name__ == '__main__': + main() diff --git a/frameworks/agentgym_rl/verl/agent_trainer/main_generation.py b/frameworks/agentgym_rl/verl/agent_trainer/main_generation.py new file mode 100644 index 0000000000000000000000000000000000000000..65ba34753bfed90b9de905562597a9a6bc1ea52a --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/main_generation.py @@ -0,0 +1,154 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Generate responses given a dataset of prompts +""" +from collections import defaultdict +import json +import ray +import numpy as np +import hydra +import verl.utils.torch_functional as verl_F +import os + +os.environ['NCCL_DEBUG'] = 'WARN' +os.environ['TOKENIZERS_PARALLELISM'] = 'true' +# os.environ['TORCH_COMPILE_DISABLE'] = '1' + +from verl.utils.model import compute_position_id_with_mask + +import pandas as pd + +from transformers import AutoTokenizer + +from verl import DataProto +from verl.utils.fs import copy_local_path_from_hdfs +from verl.workers.agent_fsdp_workers import ActorRolloutRefWorker +from verl.utils.hdfs_io import makedirs +from verl.single_controller.ray import RayClassWithInitArgs, RayResourcePool, RayWorkerGroup +from verl.utils.agentgym.client import init_env_client + + +@hydra.main(config_path='config', config_name='generation', version_base=None) +def main(config): + from pprint import pprint + from omegaconf import OmegaConf + pprint(OmegaConf.to_container(config, resolve=True)) # resolve=True will eval symbol values + OmegaConf.resolve(config) + local_path = copy_local_path_from_hdfs(config.model.path) + from verl.utils import hf_tokenizer + tokenizer = hf_tokenizer(local_path) + + if config.rollout.temperature == 0.: + assert config.data.n_samples == 1, 'When temperature=0, n_samples must be 1.' + + # read dataset. Note that the dataset should directly contain chat template format (e.g., a list of dictionary) + dataset = pd.read_json(os.path.join(config.data.path, f"{config.agentgym.task_name}_test.json")) + item_ids = dataset[config.data.prompt_key].tolist() + # load sub category test file + category_files = os.listdir(config.data.path) + category_files = [f for f in category_files if not f.startswith(f"{config.agentgym.task_name}_test")] + category_map = {} + for category_file in category_files: + path = os.path.join(config.data.path, category_file) + with open(path, "r") as f: + datas = json.load(f) + for data in datas: + category_map[data["item_id"]] = category_file.split(".")[0] + + tokenizer.padding_side = 'left' + if tokenizer.pad_token is None: + tokenizer.pad_token = tokenizer.eos_token + + ray_cls_with_init = RayClassWithInitArgs(cls=ray.remote(ActorRolloutRefWorker), config=config, role='rollout') + resource_pool = RayResourcePool(process_on_nodes=[config.trainer.n_gpus_per_node] * config.trainer.nnodes) + wg = RayWorkerGroup(resource_pool=resource_pool, ray_cls_with_init=ray_cls_with_init) + wg.init_model() + + total_samples = len(dataset) + # real_batch_size = data.batch['input_ids'].shape[0] + config_batch_size = config.data.batch_size + dp_size = wg.world_size // config.rollout.tensor_model_parallel_size + num_batch = (total_samples // config_batch_size) + 1 + output_lst = [[] for _ in range(config.data.n_samples)] + env_client = init_env_client(config.agentgym) + + for batch_idx in range(num_batch): + print(f'[{batch_idx+1}/{num_batch}] Start to process.') + start_idx = batch_idx * config_batch_size + end_idx = min(total_samples, start_idx + config_batch_size) + batch_item_ids = item_ids[start_idx: end_idx] + prompt_with_chat_template = ["<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\n" + env_client.conversation_start[0]["value"] + "<|im_end|>\n<|im_start|>assistant\n" + env_client.conversation_start[1]["value"] + "<|im_end|>" for _ in range(len(batch_item_ids))] + messages = [[{"role": "user", "content": env_client.conversation_start[0]["value"]}, + {"role": "assistant", "content": env_client.conversation_start[1]["value"]}] for _ in range(len(batch_item_ids))] + + input_ids, attention_mask = verl_F.tokenize_and_postprocess_data(prompt=prompt_with_chat_template, + tokenizer=tokenizer, + max_length=config.data.max_prompt_length, + pad_token_id=tokenizer.pad_token_id, + left_pad=True) + position_ids = compute_position_id_with_mask(attention_mask) + + batch_dict = {'input_ids': input_ids, 'attention_mask': attention_mask, 'position_ids': position_ids} + + data = DataProto.from_dict(batch_dict) + data.meta_info['global_steps'] = 'test_batch_' + str(batch_idx) + data.meta_info['max_rounds'] = config.agentgym.max_rounds + data.non_tensor_batch["item_id"] = np.array(batch_item_ids, dtype=object) + data.non_tensor_batch["raw_prompt"] = np.array(messages, dtype=object) + real_batch_size = data.batch['input_ids'].shape[0] + if real_batch_size % dp_size != 0: + dummy_data_size = dp_size - real_batch_size % dp_size + dummy_data = data[:dummy_data_size] + data = DataProto.concat([data, dummy_data]) + print( + f'dp_size {dp_size} is not divisible by real_batch_size {real_batch_size}, add {dummy_data_size} dummy data' + ) + + batch_size = data.batch['input_ids'].shape[0] + assert batch_size % dp_size == 0, f'batch_size {batch_size} is not divisible by dp_size {dp_size}' + + print(f'[{batch_idx+1}/{num_batch}] Start to generate.') + + for i in range(config.data.n_samples): + output = wg.generate_sequences(data) + # remove dummy data + output = output[:real_batch_size] + + output_lst[i].extend(output.batch['task_scores'].sum(dim=-1).tolist()) + + # convert output_lst from (n_samples, n_data) to (n_data, n_sampels) + output_np = np.array(output_lst, dtype=object) + output_np = np.transpose(output_np, axes=(1, 0)) + output_lst = output_np.tolist() + + print("============Total Task Evaluation============") + print(f"Avg@{config.data.n_samples}: {np.mean(output_np)}") + print(f"Pass@{config.data.n_samples}: {np.mean(np.max(output_np, axis=-1) > 0)}") + print("============Sub Task Evaluation============") + + category_success_bucket = defaultdict(list) + for item_id, score in zip(item_ids, output_lst): + category = category_map[item_id] + category_success_bucket[category].append(score) + for category_file in category_files: + category = category_file.split(".")[0] + print(f"Category: {category}") + print(f"Avg@{config.data.n_samples}: {np.mean(np.array(category_success_bucket[category]))}") + print(f"Pass@{config.data.n_samples}: {np.mean(np.max(np.array(category_success_bucket[category]), axis=-1) > 0)}") + + + +if __name__ == '__main__': + main() diff --git a/frameworks/agentgym_rl/verl/agent_trainer/main_ppo.py b/frameworks/agentgym_rl/verl/agent_trainer/main_ppo.py new file mode 100644 index 0000000000000000000000000000000000000000..00b43b1c17b59835e849cb6c06c77616bc3ff993 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/main_ppo.py @@ -0,0 +1,96 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Note that we don't combine the main with ray_trainer as ray_trainer is used by other main. +""" +from verl.agent_trainer.ppo.ray_trainer import RayPPOTrainer + +import ray +import hydra + + +@hydra.main(config_path='config', config_name='ppo_trainer', version_base=None) +def main(config): + run_ppo(config) + + +def run_ppo(config): + if not ray.is_initialized(): + # this is for local ray cluster + ray.init(runtime_env={'env_vars': {'TOKENIZERS_PARALLELISM': 'true', 'NCCL_DEBUG': 'WARN'}}) + + ray.get(main_task.remote(config)) + + +@ray.remote(num_cpus=1) # please make sure main_task is not scheduled on head +def main_task(config): + from verl.utils.fs import copy_local_path_from_hdfs + # print initial config + from pprint import pprint + from omegaconf import OmegaConf + pprint(OmegaConf.to_container(config, resolve=True)) # resolve=True will eval symbol values + OmegaConf.resolve(config) + + # download the checkpoint from hdfs + local_path = copy_local_path_from_hdfs(config.actor_rollout_ref.model.path) + + # instantiate tokenizer + from verl.utils import hf_tokenizer + tokenizer = hf_tokenizer(local_path) + + # define worker classes + if config.actor_rollout_ref.actor.strategy == 'fsdp': + assert config.actor_rollout_ref.actor.strategy == config.critic.strategy + from verl.workers.agent_fsdp_workers import ActorRolloutRefWorker, CriticWorker + from verl.single_controller.ray import RayWorkerGroup + ray_worker_group_cls = RayWorkerGroup + + else: + raise NotImplementedError + + from verl.agent_trainer.ppo.ray_trainer import ResourcePoolManager, Role + + role_worker_mapping = { + Role.ActorRollout: ray.remote(ActorRolloutRefWorker), + Role.Critic: ray.remote(CriticWorker), + Role.RefPolicy: ray.remote(ActorRolloutRefWorker) + } + + global_pool_id = 'global_pool' + resource_pool_spec = { + global_pool_id: [config.trainer.n_gpus_per_node] * config.trainer.nnodes, + } + mapping = { + Role.ActorRollout: global_pool_id, + Role.Critic: global_pool_id, + Role.RefPolicy: global_pool_id, + } + + resource_pool_manager = ResourcePoolManager(resource_pool_spec=resource_pool_spec, mapping=mapping) + + trainer = RayPPOTrainer(config=config, + tokenizer=tokenizer, + role_worker_mapping=role_worker_mapping, + resource_pool_manager=resource_pool_manager, + ray_worker_group_cls=ray_worker_group_cls) + trainer.init_workers() + trainer.fit() + + +if __name__ == '__main__': + # import socket + # print(socket.gethostbyname(socket.gethostname())) + # socket.sethostname("localhost") + # print(socket.gethostbyname(socket.gethostname())) + main() diff --git a/frameworks/agentgym_rl/verl/agent_trainer/ppo/__init__.py b/frameworks/agentgym_rl/verl/agent_trainer/ppo/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/ppo/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/agent_trainer/ppo/core_algos.py b/frameworks/agentgym_rl/verl/agent_trainer/ppo/core_algos.py new file mode 100644 index 0000000000000000000000000000000000000000..fe2d1e7d1742c385de807f72ed6d32bab4ee7827 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/ppo/core_algos.py @@ -0,0 +1,387 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2022 The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Core functions to implement PPO algorithms. +The function implemented in this file should be used by trainer with different distributed strategies to +implement PPO +""" + +import numpy as np +import torch +from collections import defaultdict + +import verl.utils.torch_functional as verl_F + + +class AdaptiveKLController: + """ + Adaptive KL controller described in the paper: + https://arxiv.org/pdf/1909.08593.pdf + """ + + def __init__(self, init_kl_coef, target_kl, horizon): + self.value = init_kl_coef + self.target = target_kl + self.horizon = horizon + + def update(self, current_kl, n_steps): + target = self.target + proportional_error = np.clip(current_kl / target - 1, -0.2, 0.2) + mult = 1 + proportional_error * n_steps / self.horizon + self.value *= mult + + +class FixedKLController: + """Fixed KL controller.""" + + def __init__(self, kl_coef): + self.value = kl_coef + + def update(self, current_kl, n_steps): + pass + + +def get_kl_controller(config): + if config.critic.kl_ctrl.type == 'fixed': + kl_ctrl = FixedKLController(kl_coef=config.critic.kl_ctrl.kl_coef) + elif config.critic.kl_ctrl.type == 'adaptive': + assert config.kl_ctrl.horizon > 0, f'horizon must be larger than 0. Got {config.critic.kl_ctrl.horizon}' + kl_ctrl = AdaptiveKLController(init_kl_coef=config.critic.kl_ctrl.kl_coef, + target_kl=config.critic.kl_ctrl.target_kl, + horizon=config.critic.kl_ctrl.horizon) + else: + raise ValueError('Unknown kl_ctrl type') + + return kl_ctrl + + +def compute_gae_advantage_return(token_level_rewards: torch.Tensor, values: torch.Tensor, eos_mask: torch.Tensor, + gamma: torch.Tensor, lam: torch.Tensor): + """Adapted from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py + + Args: + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + values: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length). [EOS] mask. The token after [EOS] have mask zero. + gamma: `(float)` + discounted factor used in RL + lam: `(float)` + lambda value when computing Generalized Advantage Estimation (https://arxiv.org/abs/1506.02438) + + Returns: + advantages: `(torch.Tensor)` + shape: (bs, response_length) + Returns: `(torch.Tensor)` + shape: (bs, response_length) + + """ + with torch.no_grad(): + lastgaelam = 0 + advantages_reversed = [] + gen_len = token_level_rewards.shape[-1] + + for t in reversed(range(gen_len)): + # nextvalues = values[:, t + 1] if t < gen_len - 1 else 0.0 + # delta = token_level_rewards[:, t] + gamma * nextvalues - values[:, t] + nextvalues = (values[:, t + 1] * eos_mask[:, t] + (1 - eos_mask[:, t]) * nextvalues) if t < gen_len - 1 else 0.0 + delta = token_level_rewards[:, t] * eos_mask[:, t] + gamma * nextvalues + (1 - eos_mask[:, t]) * (1 - gamma) * nextvalues - eos_mask[:, t] * values[:, t] + lastgaelam = delta * eos_mask[:, t] + gamma * lam * lastgaelam + (1 - eos_mask[:, t]) * (1 - gamma * lam) * lastgaelam + advantages_reversed.append(lastgaelam) + advantages = torch.stack(advantages_reversed[::-1], dim=1) + + returns = advantages + values + advantages = verl_F.masked_whiten(advantages, eos_mask) + return advantages, returns + + +# NOTE(sgm): this implementation only consider outcome supervision, where the reward is a scalar. +def compute_grpo_outcome_advantage(token_level_rewards: torch.Tensor, + eos_mask: torch.Tensor, + index: torch.Tensor, + epsilon: float = 1e-6): + """ + Compute advantage for GRPO, operating only on Outcome reward + (with only one scalar reward for each response). + Args: + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + + Returns: + advantages: `(torch.Tensor)` + shape: (bs, response_length) + Returns: `(torch.Tensor)` + shape: (bs, response_length) + """ + response_length = token_level_rewards.shape[-1] + scores = token_level_rewards.sum(dim=-1) + + id2score = defaultdict(list) + id2mean = {} + id2std = {} + + with torch.no_grad(): + bsz = scores.shape[0] + for i in range(bsz): + id2score[index[i]].append(scores[i]) + for idx in id2score: + if len(id2score[idx]) == 1: + id2mean[idx] = torch.tensor(0.0) + id2std[idx] = torch.tensor(1.0) + elif len(id2score[idx]) > 1: + id2mean[idx] = torch.mean(torch.tensor(id2score[idx])) + id2std[idx] = torch.std(torch.tensor([id2score[idx]])) + else: + raise ValueError(f"no score in prompt index: {idx}") + for i in range(bsz): + scores[i] = (scores[i] - id2mean[index[i]]) / (id2std[index[i]] + epsilon) + scores = scores.unsqueeze(-1).tile([1, response_length]) * eos_mask + + return scores, scores + + +def compute_reinforce_plus_plus_outcome_advantage(token_level_rewards: torch.Tensor, eos_mask: torch.Tensor, + gamma: torch.Tensor): + """ + Compute advantage for REINFORCE++. + This implementation is based on the paper: https://arxiv.org/abs/2501.03262 + Args: + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + + Returns: + advantages: `(torch.Tensor)` + shape: (bs, response_length) + Returns: `(torch.Tensor)` + shape: (bs, response_length) + """ + + with torch.no_grad(): + returns = torch.zeros_like(token_level_rewards) + running_return = 0 + + for t in reversed(range(token_level_rewards.shape[1])): + # running_return = token_level_rewards[:, t] + gamma * running_return + running_return = token_level_rewards[:, t] * eos_mask[:, t] + gamma * running_return + (1 - eos_mask[:, t]) * (1 - gamma) * running_return + returns[:, t] = running_return * eos_mask[:, t] + # Reset after EOS + # running_return = running_return * eos_mask[:, t] + + advantages = verl_F.masked_whiten(returns, eos_mask) + advantages = advantages * eos_mask + + return advantages, returns + +def compute_rloo_outcome_advantage( + token_level_rewards: torch.Tensor, + eos_mask: torch.Tensor, + index: np.ndarray, + epsilon: float = 1e-6, +) -> tuple[torch.Tensor, torch.Tensor]: + """ + Compute advantage for RLOO based on https://arxiv.org/abs/2402.14740 + + Args: + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + + Returns: + advantages: `(torch.Tensor)` + shape: (bs, response_length) + Returns: `(torch.Tensor)` + shape: (bs, response_length) + """ + scores = token_level_rewards.sum(dim=-1) + + id2score = defaultdict(list) + id2mean = {} + + with torch.no_grad(): + bsz = scores.shape[0] + for i in range(bsz): + id2score[index[i]].append(scores[i]) + for idx in id2score: + if len(id2score[idx]) == 1: + id2mean[idx] = torch.tensor(0.0) + elif len(id2score[idx]) > 1: + id2mean[idx] = torch.mean(torch.stack(id2score[idx])) + else: + raise ValueError(f"no score in prompt index: {idx}") + for i in range(bsz): + response_num = len(id2score[index[i]]) + if response_num > 1: + scores[i] = scores[i] * response_num / (response_num - 1) - id2mean[index[i]] * response_num / ( + response_num - 1 + ) + scores = scores.unsqueeze(-1) * eos_mask + + return scores, scores + +def compute_remax_outcome_advantage(token_level_rewards: torch.Tensor, reward_baselines: torch.Tensor, + eos_mask: torch.Tensor): + """ + Compute advantage for ReMax, operating only on Outcome reward + This implementation is based on the paper: https://arxiv.org/abs/2310.10505 + + (with only one scalar reward for each response). + Args: + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + reward_baselines: `(torch.Tensor)` + shape: (bs,) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + + Returns: + advantages: `(torch.Tensor)` + shape: (bs, response_length) + Returns: `(torch.Tensor)` + shape: (bs, response_length) + """ + response_length = token_level_rewards.shape[-1] + scores = token_level_rewards.sum(dim=-1) + + with torch.no_grad(): + returns = (token_level_rewards * eos_mask).flip(dims=[-1]).cumsum(dim=-1).flip(dims=[-1]) + advantages = returns - reward_baselines.unsqueeze(-1).tile([1, response_length]) * eos_mask + + return advantages, returns + + +def compute_rewards(token_level_scores, old_log_prob, ref_log_prob, kl_ratio): + kl = old_log_prob - ref_log_prob + return token_level_scores - kl * kl_ratio + + +def compute_policy_loss(old_log_prob, log_prob, advantages, eos_mask, cliprange): + """Adapted from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1122 + + Args: + old_log_prob: `(torch.Tensor)` + shape: (bs, response_length) + log_prob: `(torch.Tensor)` + shape: (bs, response_length) + advantages: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + cliprange: (float) + The clip range used in PPO. See https://arxiv.org/abs/1707.06347 + + Returns: + pg_loss: `a scalar torch.Tensor` + policy gradient loss computed via PPO + pg_clipfrac: (float) + a float number indicating the fraction of policy gradient loss being clipped + + """ + negative_approx_kl = log_prob - old_log_prob + ratio = torch.exp(negative_approx_kl) + ppo_kl = verl_F.masked_mean(-negative_approx_kl, eos_mask) + + pg_losses = -advantages * ratio + pg_losses2 = -advantages * torch.clamp(ratio, 1.0 - cliprange, 1.0 + cliprange) + + pg_loss = verl_F.masked_mean(torch.max(pg_losses, pg_losses2), eos_mask) + pg_clipfrac = verl_F.masked_mean(torch.gt(pg_losses2, pg_losses).float(), eos_mask) + return pg_loss, pg_clipfrac, ppo_kl + + +def compute_entropy_loss(logits, eos_mask): + """Compute Categorical entropy loss + + Args: + logits: `(torch.Tensor)` + shape: (bs, response_length, vocab_size) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + + Returns: + entropy: a scalar torch.Tensor + + """ + # compute entropy + entropy = verl_F.entropy_from_logits(logits) # (bs, response_len) + entropy_loss = verl_F.masked_mean(entropy, mask=eos_mask) + return entropy_loss + + +def compute_value_loss(vpreds, returns, values, eos_mask, cliprange_value): + """Compute the value loss. Copied from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1151 + + Args: + vpreds (`torch.FloatTensor`): + Predicted values of the value head, shape (`batch_size`, `response_length`) + values (`torch.FloatTensor`): + Old values of value head, shape (`batch_size`, `response_length`) + returns: (`torch.FloatTensor`): + Ground truth returns, shape (`batch_size`, `response_length`) + + Returns: + vf_loss: a scalar (`torch.FloatTensor`): + value function loss + vf_clipfrac: a float + The ratio of vf being clipped + + """ + vpredclipped = verl_F.clip_by_value(vpreds, values - cliprange_value, values + cliprange_value) + vf_losses1 = (vpreds - returns)**2 + vf_losses2 = (vpredclipped - returns)**2 + vf_loss = 0.5 * verl_F.masked_mean(torch.max(vf_losses1, vf_losses2), eos_mask) + vf_clipfrac = verl_F.masked_mean(torch.gt(vf_losses2, vf_losses1).float(), eos_mask) + return vf_loss, vf_clipfrac + + +def kl_penalty(logprob: torch.FloatTensor, ref_logprob: torch.FloatTensor, kl_penalty) -> torch.FloatTensor: + """Compute KL divergence given logprob and ref_logprob. + Copied from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1104 + + Args: + logprob: + ref_logprob: + + Returns: + + """ + if kl_penalty == "kl": + return logprob - ref_logprob + + if kl_penalty == "abs": + return (logprob - ref_logprob).abs() + + if kl_penalty == "mse": + return 0.5 * (logprob - ref_logprob).square() + + # J. Schulman. Approximating kl divergence, 2020. + # # URL http://joschu.net/blog/kl-approx.html. + if kl_penalty == 'low_var_kl': + kl = ref_logprob - logprob + ratio = torch.exp(kl) + kld = (ratio - kl - 1).contiguous() + return torch.clamp(kld, min=-10, max=10) + + if kl_penalty == "full": + # so, here logprob and ref_logprob should contain the logits for every token in vocabulary + raise NotImplementedError + + raise NotImplementedError diff --git a/frameworks/agentgym_rl/verl/agent_trainer/ppo/ray_trainer.py b/frameworks/agentgym_rl/verl/agent_trainer/ppo/ray_trainer.py new file mode 100644 index 0000000000000000000000000000000000000000..389368e5233b1ffb5b07c986ee20d6087e12d446 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/ppo/ray_trainer.py @@ -0,0 +1,884 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +FSDP PPO Trainer with Ray-based single controller. +This trainer supports model-agonistic model initialization with huggingface +""" + +import os +import uuid +from contextlib import contextmanager +from dataclasses import dataclass, field +from enum import Enum +from typing import List, Type, Dict +from copy import deepcopy + +import numpy as np +from codetiming import Timer +from omegaconf import OmegaConf, open_dict +from verl import DataProto +from verl.single_controller.base import Worker +from verl.single_controller.ray import RayResourcePool, RayWorkerGroup, RayClassWithInitArgs +from verl.single_controller.ray.base import create_colocated_worker_cls +from verl.agent_trainer.ppo import core_algos +from verl.utils.seqlen_balancing import get_seqlen_balanced_partitions, log_seqlen_unbalance +from verl.utils.checkpoint.checkpoint_manager import find_latest_ckpt_path +from verl.utils.agent_dataset.rl_dataset import RLHFDataset, collate_fn +from abc import ABC, abstractmethod + +WorkerType = Type[Worker] + + +class Role(Enum): + """ + To create more roles dynamically, you can subclass Role and add new members + """ + Actor = 0 + Rollout = 1 + ActorRollout = 2 + Critic = 3 + RefPolicy = 4 + RewardModel = 5 + ActorRolloutRef = 6 + + +@dataclass +class ResourcePoolManager: + """ + Define a resource pool specification. Resource pool will be initialized first. + Mapping + """ + resource_pool_spec: dict[str, list[int]] + mapping: dict[Role, str] + resource_pool_dict: dict[str, RayResourcePool] = field(default_factory=dict) + + def create_resource_pool(self): + for resource_pool_name, process_on_nodes in self.resource_pool_spec.items(): + # max_colocate_count means the number of WorkerGroups (i.e. processes) in each RayResourcePool + # For FSDP backend, we recommend using max_colocate_count=1 that merge all WorkerGroups into one. + # For Megatron backend, we recommend using max_colocate_count>1 that can utilize different WorkerGroup for differnt models + resource_pool = RayResourcePool(process_on_nodes=process_on_nodes, + use_gpu=True, + max_colocate_count=1, + name_prefix=resource_pool_name) + self.resource_pool_dict[resource_pool_name] = resource_pool + + def get_resource_pool(self, role: Role) -> RayResourcePool: + """Get the resource pool of the worker_cls""" + return self.resource_pool_dict[self.mapping[role]] + + +import torch +from verl.utils.torch_functional import masked_mean + + +def find_latest_ckpt_path_aistudio(path, directory_format="global_step_{}"): + if path is None: + return None + + from verl.utils.checkpoint.checkpoint_manager import get_checkpoint_tracker_filename + tracker_file = get_checkpoint_tracker_filename(path) + if not os.path.exists(tracker_file): + print("Checkpoint tracker file does not exist: %s", tracker_file) + return None + + from aistudio_checkpoint.aistudio_base_checkpointer import load_checkpoint + with open(tracker_file, "r") as f: + iteration, resuming_path = f.read().split("\n") + ckpt_path = os.path.join(load_checkpoint(resuming_path=resuming_path), directory_format.format(iteration)) + if not os.path.exists(ckpt_path): + print("Checkpoint does not exist: %s", ckpt_path) + return None + + print("Found checkpoint: %s", ckpt_path) + return ckpt_path + + +def apply_kl_penalty(data: DataProto, kl_ctrl: core_algos.AdaptiveKLController, kl_penalty='kl'): + token_level_scores = data.batch['token_level_scores'] + batch_size = data.batch.batch_size[0] + response_mask = data.batch['response_mask'] + + # compute kl between ref_policy and current policy + if 'ref_log_prob' in data.batch.keys(): + kld = core_algos.kl_penalty(data.batch['old_log_probs'], data.batch['ref_log_prob'], + kl_penalty=kl_penalty) # (batch_size, response_length) + kld = kld * response_mask + beta = kl_ctrl.value + else: + beta = 0 + kld = torch.zeros_like(response_mask, dtype=torch.float32) + + token_level_rewards = token_level_scores - beta * kld + + current_kl = masked_mean(kld, mask=response_mask, axis=-1) # average over sequence + current_kl = torch.mean(current_kl, dim=0).item() + + # according to https://github.com/huggingface/trl/blob/951ca1841f29114b969b57b26c7d3e80a39f75a0/trl/trainer/ppo_trainer.py#L837 + kl_ctrl.update(current_kl=current_kl, n_steps=batch_size) + data.batch['token_level_rewards'] = token_level_rewards + + metrics = {'critic/kl': current_kl, 'critic/kl_coeff': beta} + + return data, metrics + + +def compute_advantage(data: DataProto, adv_estimator, gamma=1.0, lam=1.0, num_repeat=1): + # prepare response group + # TODO: add other ways to estimate advantages + if adv_estimator == 'gae': + values = data.batch['values'] + response_mask = data.batch['response_mask'] + token_level_rewards = data.batch['token_level_rewards'] + advantages, returns = core_algos.compute_gae_advantage_return(token_level_rewards=token_level_rewards, + values=values, + eos_mask=response_mask, + gamma=gamma, + lam=lam) + data.batch['advantages'] = advantages + data.batch['returns'] = returns + elif adv_estimator == 'grpo': + token_level_rewards = data.batch['token_level_rewards'] + index = data.non_tensor_batch['uid'] + response_mask = data.batch['response_mask'] + advantages, returns = core_algos.compute_grpo_outcome_advantage(token_level_rewards=token_level_rewards, + eos_mask=response_mask, + index=index) + data.batch['advantages'] = advantages + data.batch['returns'] = returns + elif adv_estimator == 'rloo': + token_level_rewards = data.batch['token_level_rewards'] + index = data.non_tensor_batch['uid'] + response_mask = data.batch['response_mask'] + advantages, returns = core_algos.compute_rloo_outcome_advantage(token_level_rewards=token_level_rewards, + eos_mask=response_mask, + index=index) + data.batch['advantages'] = advantages + data.batch['returns'] = returns + elif adv_estimator == 'reinforce_plus_plus': + token_level_rewards = data.batch['token_level_rewards'] + response_mask = data.batch['response_mask'] + advantages, returns = core_algos.compute_reinforce_plus_plus_outcome_advantage( + token_level_rewards=token_level_rewards, eos_mask=response_mask, gamma=gamma) + data.batch['advantages'] = advantages + data.batch['returns'] = returns + elif adv_estimator == 'remax': + token_level_rewards = data.batch['token_level_rewards'] + index = data.non_tensor_batch['uid'] + response_mask = data.batch['response_mask'] + + reward_baselines = data.batch['reward_baselines'] + + advantages, returns = core_algos.compute_remax_outcome_advantage(token_level_rewards=token_level_rewards, + reward_baselines=reward_baselines, + eos_mask=response_mask) + + data.batch['advantages'] = advantages + data.batch['returns'] = returns + else: + raise NotImplementedError + return data + + +class RoundsScheduler(ABC): + @abstractmethod + def step(self): + raise NotImplementedError + + @abstractmethod + def set_global_steps(self, global_steps: int): + raise NotImplementedError + + @abstractmethod + def get_rounds(self): + raise NotImplementedError + + +class FixedRoundsScheduler(RoundsScheduler): + def __init__(self, rounds: int): + self.max_rounds = rounds + + def step(self): + pass + + def set_global_steps(self, global_steps: int): + pass + + def get_rounds(self): + return self.max_rounds + + +class StepRoundsScheduler(RoundsScheduler): + def __init__(self, steps_scaling_inter: int, rounds_ls: List[int]): + self.rounds_ls = rounds_ls + self.steps_scaling_inter = steps_scaling_inter + self.max_rounds = rounds_ls[0] + self.current_stage = 0 + self.global_steps = 1 # start from 1 + + def set_global_steps(self, global_steps: int): + self.global_steps = global_steps + if (self.global_steps // self.steps_scaling_inter < len(self.rounds_ls)): + self.current_stage = self.global_steps // self.steps_scaling_inter + else: + self.current_stage = len(self.rounds_ls) - 1 + self.max_rounds = self.rounds_ls[self.current_stage] + + def step(self): + if self.current_stage + 1 < len(self.rounds_ls) and self.global_steps % self.steps_scaling_inter == 0: + self.current_stage += 1 + self.max_rounds = self.rounds_ls[self.current_stage] + self.global_steps += 1 + + def get_rounds(self): + return self.max_rounds + + +def reduce_metrics(metrics: dict): + for key, val in metrics.items(): + metrics[key] = np.mean(val) + return metrics + + +def compute_data_metrics(batch, use_critic=True): + # TODO: add response length + sequence_score = batch.batch['token_level_scores'].sum(-1) + sequence_reward = batch.batch['token_level_rewards'].sum(-1) + task_scores = batch.batch["task_scores"].sum(-1) + task_rounds = batch.batch["task_rounds"] + + response_length = batch.batch['response_mask'].sum(-1).float() + prompt_length = batch.batch['attention_mask'].sum(-1).float() - response_length + + advantages = batch.batch['advantages'] + returns = batch.batch['returns'] + + response_mask = batch.batch['response_mask'].bool() + + valid_adv = torch.masked_select(advantages, response_mask) + valid_returns = torch.masked_select(returns, response_mask) + + if use_critic: + values = batch.batch['values'] + valid_values = torch.masked_select(values, response_mask) + return_diff_var = torch.var(valid_returns - valid_values) + return_var = torch.var(valid_returns) + + metrics = { + # score + 'critic/score/mean': + torch.mean(sequence_score).detach().item(), + 'critic/score/max': + torch.max(sequence_score).detach().item(), + 'critic/score/min': + torch.min(sequence_score).detach().item(), + # task score + 'critic/task_score/mean': + torch.mean(task_scores).detach().item(), + 'critic/task_score/max': + torch.max(task_scores).detach().item(), + 'critic/task_score/min': + torch.min(task_scores).detach().item(), + # task round + 'critic/task_round/mean': + torch.mean(task_rounds).detach().item(), + 'critic/task_round/max': + torch.max(task_rounds).detach().item(), + 'critic/task_round/min': + torch.min(task_rounds).detach().item(), + # reward + 'critic/rewards/mean': + torch.mean(sequence_reward).detach().item(), + 'critic/rewards/max': + torch.max(sequence_reward).detach().item(), + 'critic/rewards/min': + torch.min(sequence_reward).detach().item(), + # adv + 'critic/advantages/mean': + torch.mean(valid_adv).detach().item(), + 'critic/advantages/max': + torch.max(valid_adv).detach().item(), + 'critic/advantages/min': + torch.min(valid_adv).detach().item(), + # returns + 'critic/returns/mean': + torch.mean(valid_returns).detach().item(), + 'critic/returns/max': + torch.max(valid_returns).detach().item(), + 'critic/returns/min': + torch.min(valid_returns).detach().item(), + **({ + # values + 'critic/values/mean': torch.mean(valid_values).detach().item(), + 'critic/values/max': torch.max(valid_values).detach().item(), + 'critic/values/min': torch.min(valid_values).detach().item(), + # vf explained var + 'critic/vf_explained_var': (1.0 - return_diff_var / (return_var + 1e-5)).detach().item(), + } if use_critic else {}), + + # response length + 'response_length/mean': + torch.mean(response_length).detach().item(), + 'response_length/max': + torch.max(response_length).detach().item(), + 'response_length/min': + torch.min(response_length).detach().item(), + # prompt length + 'prompt_length/mean': + torch.mean(prompt_length).detach().item(), + 'prompt_length/max': + torch.max(prompt_length).detach().item(), + 'prompt_length/min': + torch.min(prompt_length).detach().item(), + } + return metrics + + +def compute_timing_metrics(batch, timing_raw): + num_overall_tokens = torch.sum(batch.batch['attention_mask']).item() + num_response_tokens = torch.sum(batch.batch['response_mask']).item() + + num_tokens_of_section = { + 'gen': num_response_tokens, + **{ + name: num_overall_tokens for name in ['ref', 'values', 'adv', 'update_critic', 'update_actor'] + }, + } + + return { + **{ + f'timing_s/{name}': value for name, value in timing_raw.items() + }, + **{ + f'timing_per_token_ms/{name}': timing_raw[name] * 1000 / num_tokens_of_section[name] for name in set(num_tokens_of_section.keys( + )) & set(timing_raw.keys()) + }, + } + + +@contextmanager +def _timer(name: str, timing_raw: Dict[str, float]): + with Timer(name=name, logger=None) as timer: + yield + timing_raw[name] = timer.last + + +class RayPPOTrainer(object): + """ + Note that this trainer runs on the driver process on a single CPU/GPU node. + """ + + # TODO: support each role have individual ray_worker_group_cls, + # i.e., support different backend of different role + def __init__(self, + config, + tokenizer, + role_worker_mapping: dict[Role, WorkerType], + resource_pool_manager: ResourcePoolManager, + ray_worker_group_cls: RayWorkerGroup = RayWorkerGroup): + + # assert torch.cuda.is_available(), 'cuda must be available on driver' + + self.tokenizer = tokenizer + self.config = config + + self.hybrid_engine = config.actor_rollout_ref.hybrid_engine + assert self.hybrid_engine, 'Currently, only support hybrid engine' + + if self.hybrid_engine: + assert Role.ActorRollout in role_worker_mapping, f'{role_worker_mapping.keys()=}' + + self.role_worker_mapping = role_worker_mapping + self.resource_pool_manager = resource_pool_manager + self.use_reference_policy = Role.RefPolicy in role_worker_mapping + self.ray_worker_group_cls = ray_worker_group_cls + + # define KL control + if self.use_reference_policy: + if config.algorithm.kl_ctrl.type == 'fixed': + self.kl_ctrl = core_algos.FixedKLController(kl_coef=config.algorithm.kl_ctrl.kl_coef) + elif config.algorithm.kl_ctrl.type == 'adaptive': + assert config.algorithm.kl_ctrl.horizon > 0, f'horizon must be larger than 0. Got {config.critic.kl_ctrl.horizon}' + self.kl_ctrl = core_algos.AdaptiveKLController(init_kl_coef=config.algorithm.kl_ctrl.kl_coef, + target_kl=config.algorithm.kl_ctrl.target_kl, + horizon=config.algorithm.kl_ctrl.horizon) + else: + raise NotImplementedError + else: + self.kl_ctrl = core_algos.FixedKLController(kl_coef=0.) + + if self.config.algorithm.adv_estimator == 'gae': + self.use_critic = True + elif self.config.algorithm.adv_estimator == 'grpo': + self.use_critic = False + elif self.config.algorithm.adv_estimator == 'reinforce_plus_plus': + self.use_critic = False + elif self.config.algorithm.adv_estimator == 'remax': + self.use_critic = False + else: + raise NotImplementedError + + self._validate_config() + self._create_dataloader() + + def _validate_config(self): + config = self.config + # number of GPUs total + n_gpus = config.trainer.n_gpus_per_node * config.trainer.nnodes + + # 1. Check total batch size for data correctness + real_train_batch_size = config.data.train_batch_size * config.actor_rollout_ref.rollout.n + assert real_train_batch_size % n_gpus == 0, \ + f"real_train_batch_size ({real_train_batch_size}) must be divisible by total n_gpus ({n_gpus})." + + # A helper function to check "micro_batch_size" vs "micro_batch_size_per_gpu" + # We throw an error if the user sets both. The new convention is "..._micro_batch_size_per_gpu". + def check_mutually_exclusive(mbs, mbs_per_gpu, name: str): + if mbs is None and mbs_per_gpu is None: + raise ValueError(f"[{name}] Please set at least one of '{name}.micro_batch_size' or " + f"'{name}.micro_batch_size_per_gpu'.") + + if mbs is not None and mbs_per_gpu is not None: + raise ValueError(f"[{name}] You have set both '{name}.micro_batch_size' AND " + f"'{name}.micro_batch_size_per_gpu'. Please remove '{name}.micro_batch_size' " + f"because only '*_micro_batch_size_per_gpu' is supported (the former is deprecated).") + + if not config.actor_rollout_ref.actor.use_dynamic_bsz: + # actor: ppo_micro_batch_size vs. ppo_micro_batch_size_per_gpu + check_mutually_exclusive(config.actor_rollout_ref.actor.ppo_micro_batch_size, + config.actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu, + "actor_rollout_ref.actor") + + # reference: log_prob_micro_batch_size vs. log_prob_micro_batch_size_per_gpu + check_mutually_exclusive(config.actor_rollout_ref.ref.log_prob_micro_batch_size, + config.actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu, + "actor_rollout_ref.ref") + + # The rollout section also has log_prob_micro_batch_size vs. log_prob_micro_batch_size_per_gpu + check_mutually_exclusive(config.actor_rollout_ref.rollout.log_prob_micro_batch_size, + config.actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu, + "actor_rollout_ref.rollout") + + if self.use_critic and not config.critic.use_dynamic_bsz: + # Check for critic micro-batch size conflicts + check_mutually_exclusive(config.critic.ppo_micro_batch_size, config.critic.ppo_micro_batch_size_per_gpu, + "critic") + + # Actor + # if NOT dynamic_bsz, we must ensure: + # ppo_mini_batch_size is divisible by ppo_micro_batch_size + # ppo_micro_batch_size * sequence_parallel_size >= n_gpus + if not config.actor_rollout_ref.actor.use_dynamic_bsz: + sp_size = config.actor_rollout_ref.actor.get('ulysses_sequence_parallel_size', 1) + if config.actor_rollout_ref.actor.ppo_micro_batch_size is not None: + assert config.actor_rollout_ref.actor.ppo_mini_batch_size % config.actor_rollout_ref.actor.ppo_micro_batch_size == 0 + assert config.actor_rollout_ref.actor.ppo_micro_batch_size * sp_size >= n_gpus + + # critic + if self.use_critic and not config.critic.use_dynamic_bsz: + sp_size = config.critic.get('ulysses_sequence_parallel_size', 1) + if config.critic.ppo_micro_batch_size is not None: + assert config.critic.ppo_mini_batch_size % config.critic.ppo_micro_batch_size == 0 + assert config.critic.ppo_micro_batch_size * sp_size >= n_gpus + + # Check if use_remove_padding is enabled when using sequence parallelism for fsdp + if config.actor_rollout_ref.actor.strategy == 'fsdp': + if config.actor_rollout_ref.actor.get('ulysses_sequence_parallel_size', 1) > 1 or \ + config.actor_rollout_ref.ref.get('ulysses_sequence_parallel_size', 1) > 1: + assert config.actor_rollout_ref.model.use_remove_padding, \ + "When using sequence parallelism for actor/ref policy, you must enable `use_remove_padding`." + + if self.use_critic and config.critic.strategy == 'fsdp': + if config.critic.get('ulysses_sequence_parallel_size', 1) > 1: + assert config.critic.model.use_remove_padding, \ + "When using sequence parallelism for critic, you must enable `use_remove_padding`." + + print("[validate_config] All configuration checks passed successfully!") + + def _create_dataloader(self): + from torch.utils.data import DataLoader, RandomSampler, SequentialSampler + # TODO: we have to make sure the batch size is divisible by the dp size + self.train_dataset = RLHFDataset( + data_file=self.config.data.train_file, + tokenizer=self.tokenizer, + data_config=self.config.data, + agentgym_config=self.config.actor_rollout_ref.agentgym, + ) + # use sampler for better ckpt resume + if self.config.data.shuffle: + train_dataloader_generator = torch.Generator() + train_dataloader_generator.manual_seed(self.config.data.get('seed', 1)) + sampler = RandomSampler(data_source=self.train_dataset, generator=train_dataloader_generator) + else: + sampler = SequentialSampler(data_source=self.train_dataset) + + self.train_dataloader = DataLoader(dataset=self.train_dataset, + batch_size=self.config.data.train_batch_size, + drop_last=True, + collate_fn=collate_fn, + sampler=sampler) + + assert len(self.train_dataloader) >= 1 + + print(f'Size of train dataloader: {len(self.train_dataloader)}') + + # inject total_training_steps to actor/critic optim_config. This is hacky. + total_training_steps = len(self.train_dataloader) * self.config.trainer.total_epochs + + if self.config.trainer.total_training_steps is not None: + total_training_steps = self.config.trainer.total_training_steps + + self.total_training_steps = total_training_steps + if self.config.algorithm.rounds_ctrl.type == 'fixed': + self.rounds_scheduler = FixedRoundsScheduler(rounds=self.config.algorithm.rounds_ctrl.rounds) + elif self.config.algorithm.rounds_ctrl.type == 'scaling_inter_stepwise': + self.rounds_scheduler = StepRoundsScheduler(steps_scaling_inter=self.config.algorithm.rounds_ctrl.steps_scaling_inter, + rounds_ls=self.config.algorithm.rounds_ctrl.rounds) + else: + raise NotImplementedError + print(f'Total training steps: {self.total_training_steps}') + + OmegaConf.set_struct(self.config, True) + with open_dict(self.config): + self.config.actor_rollout_ref.actor.optim.total_training_steps = total_training_steps + self.config.critic.optim.total_training_steps = total_training_steps + + def init_workers(self): + """Init resource pool and worker group""" + self.resource_pool_manager.create_resource_pool() + + self.resource_pool_to_cls = {pool: {} for pool in self.resource_pool_manager.resource_pool_dict.values()} + + # create actor and rollout + if self.hybrid_engine: + resource_pool = self.resource_pool_manager.get_resource_pool(Role.ActorRollout) + actor_rollout_cls = RayClassWithInitArgs(cls=self.role_worker_mapping[Role.ActorRollout], + config=self.config.actor_rollout_ref, + role='actor_rollout') + self.resource_pool_to_cls[resource_pool]['actor_rollout'] = actor_rollout_cls + else: + raise NotImplementedError + + # create critic + if self.use_critic: + resource_pool = self.resource_pool_manager.get_resource_pool(Role.Critic) + critic_cls = RayClassWithInitArgs(cls=self.role_worker_mapping[Role.Critic], config=self.config.critic) + self.resource_pool_to_cls[resource_pool]['critic'] = critic_cls + + # create reference policy if needed + if self.use_reference_policy: + resource_pool = self.resource_pool_manager.get_resource_pool(Role.RefPolicy) + ref_policy_cls = RayClassWithInitArgs(self.role_worker_mapping[Role.RefPolicy], + config=self.config.actor_rollout_ref, + role='ref') + self.resource_pool_to_cls[resource_pool]['ref'] = ref_policy_cls + + # initialize WorkerGroup + # NOTE: if you want to use a different resource pool for each role, which can support different parallel size, + # you should not use `create_colocated_worker_cls`. Instead, directly pass different resource pool to different worker groups. + # See https://github.com/volcengine/verl/blob/master/examples/ray/tutorial.ipynb for more information. + all_wg = {} + self.wg_dicts = [] + for resource_pool, class_dict in self.resource_pool_to_cls.items(): + worker_dict_cls = create_colocated_worker_cls(class_dict=class_dict) + wg_dict = self.ray_worker_group_cls(resource_pool=resource_pool, ray_cls_with_init=worker_dict_cls) + spawn_wg = wg_dict.spawn(prefix_set=class_dict.keys()) + all_wg.update(spawn_wg) + # keep the referece of WorkerDict to support ray >= 2.31. Ref: https://github.com/ray-project/ray/pull/45699 + self.wg_dicts.append(wg_dict) + + if self.use_critic: + self.critic_wg = all_wg['critic'] + self.critic_wg.init_model() + + if self.use_reference_policy: + self.ref_policy_wg = all_wg['ref'] + self.ref_policy_wg.init_model() + + # we should create rollout at the end so that vllm can have a better estimation of kv cache memory + self.actor_rollout_wg = all_wg['actor_rollout'] + self.actor_rollout_wg.init_model() + + def _save_checkpoint(self): + if self.config.trainer.storage_mode == 'aistudio': + from aistudio_checkpoint.aistudio_mnt_checkpointer import AistudioMntCheckpointer + ckpter = AistudioMntCheckpointer() + save_dir = ckpter.get_save_dir(step=self.global_steps) + # path: given_path + `/global_step_{global_steps}` + `/actor` + local_global_step_folder = os.path.join(save_dir, + f'global_step_{self.global_steps}') + elif self.config.trainer.storage_mode == 'local': + # path: given_path + `/global_step_{global_steps}` + `/actor` + local_global_step_folder = os.path.join(self.config.trainer.default_local_dir, + f'global_step_{self.global_steps}') + else: + raise NotImplementedError + actor_local_path = os.path.join(local_global_step_folder, 'actor') + + actor_remote_path = None if self.config.trainer.default_hdfs_dir is None else os.path.join( + self.config.trainer.default_hdfs_dir, f'global_step_{self.global_steps}', 'actor') + self.actor_rollout_wg.save_checkpoint(actor_local_path, + actor_remote_path, + self.global_steps, + remove_previous_ckpt=self.config.trainer.remove_previous_ckpt_in_save) + + if self.use_critic: + critic_local_path = os.path.join(local_global_step_folder, 'critic') + critic_remote_path = None if self.config.trainer.default_hdfs_dir is None else os.path.join( + self.config.trainer.default_hdfs_dir, f'global_step_{self.global_steps}', 'critic') + self.critic_wg.save_checkpoint(critic_local_path, + critic_remote_path, + self.global_steps, + remove_previous_ckpt=self.config.trainer.remove_previous_ckpt_in_save) + + # save dataloader + dataloader_local_path = os.path.join(local_global_step_folder, 'data.pt') + import dill + torch.save(self.train_dataloader, dataloader_local_path, pickle_module=dill) + + # latest checkpointed iteration tracker (for atomic usage) + local_latest_checkpointed_iteration = os.path.join(self.config.trainer.default_local_dir, + 'latest_checkpointed_iteration.txt') + with open(local_latest_checkpointed_iteration, 'w') as f: + if self.config.trainer.storage_mode == 'aistudio': + f.write(str(self.global_steps) + "\n" + ckpter.commit(memo=self.config.trainer.experiment_name)) + elif self.config.trainer.storage_mode == 'local': + f.write(str(self.global_steps)) + else: + raise NotImplementedError + + def _load_checkpoint(self): + if self.config.trainer.resume_mode == 'disable': + return 0 + + # load from hdfs + if self.config.trainer.default_hdfs_dir is not None: + NotImplementedError('load from hdfs is not implemented yet') + else: + checkpoint_folder = self.config.trainer.default_local_dir # TODO: check path + if not os.path.isabs(checkpoint_folder): + working_dir = os.getcwd() + checkpoint_folder = os.path.join(working_dir, checkpoint_folder) + if self.config.trainer.storage_mode == 'aistudio': + global_step_folder = find_latest_ckpt_path_aistudio(checkpoint_folder) # None if no latest + elif self.config.trainer.storage_mode == 'local': + global_step_folder = find_latest_ckpt_path(checkpoint_folder) # None if no latest + else: + raise NotImplementedError + + # find global_step_folder + if self.config.trainer.resume_mode == 'auto': + if global_step_folder is None: + print('Training from scratch') + return 0 + else: + if not (self.config.trainer.resume_from_path and global_step_folder is not None): + assert isinstance(self.config.trainer.resume_mode, str), "resume ckpt must be str type" + assert 'global_step_' in self.config.trainer.resume_mode, "resume ckpt must specify the global_steps" + global_step_folder = self.config.trainer.resume_mode + if not os.path.isabs(global_step_folder): + working_dir = os.getcwd() + global_step_folder = os.path.join(working_dir, global_step_folder) + print(f'Load from checkpoint folder: {global_step_folder}') + # set global step + self.global_steps = int(global_step_folder.split('global_step_')[-1]) + self.rounds_scheduler.set_global_steps(self.global_steps) + + print(f'Setting global step to {self.global_steps}') + print(f'Resuming from {global_step_folder}') + + actor_path = os.path.join(global_step_folder, 'actor') + critic_path = os.path.join(global_step_folder, 'critic') + # load actor + self.actor_rollout_wg.load_checkpoint(actor_path, + del_local_after_load=self.config.trainer.del_local_ckpt_after_load) + # load critic + if self.use_critic: + self.critic_wg.load_checkpoint(critic_path, + del_local_after_load=self.config.trainer.del_local_ckpt_after_load) + + # load dataloader, + # TODO: from remote not implemented yet + dataloader_local_path = os.path.join(global_step_folder, 'data.pt') + self.train_dataloader = torch.load(dataloader_local_path) + if isinstance(self.train_dataloader.dataset, RLHFDataset): + self.train_dataloader.dataset.resume_dataset_state() + + def _balance_batch(self, batch: DataProto, metrics, logging_prefix='global_seqlen'): + """Reorder the data on single controller such that each dp rank gets similar total tokens""" + attention_mask = batch.batch['attention_mask'] + batch_size = attention_mask.shape[0] + global_seqlen_lst = batch.batch['attention_mask'].view(batch_size, -1).sum(-1).tolist() # (train_batch_size,) + world_size = self.actor_rollout_wg.world_size + global_partition_lst = get_seqlen_balanced_partitions(global_seqlen_lst, + k_partitions=world_size, + equal_size=True) + # reorder based on index. The data will be automatically equally partitioned by dispatch function + global_idx = torch.tensor([j for partition in global_partition_lst for j in partition]) + batch.reorder(global_idx) + global_balance_stats = log_seqlen_unbalance(seqlen_list=global_seqlen_lst, + partitions=global_partition_lst, + prefix=logging_prefix) + metrics.update(global_balance_stats) + + def fit(self): + """ + The training loop of PPO. + The driver process only need to call the compute functions of the worker group through RPC to construct the PPO dataflow. + The light-weight advantage computation is done on the driver process. + """ + from verl.utils.tracking import Tracking + from omegaconf import OmegaConf + + logger = Tracking(project_name=self.config.trainer.project_name, + experiment_name=self.config.trainer.experiment_name, + default_backend=self.config.trainer.logger, + config=OmegaConf.to_container(self.config, resolve=True)) + + self.global_steps = 0 + + # load checkpoint before doing anything + self._load_checkpoint() + + if self.config.trainer.storage_mode == 'aistudio': + self._save_checkpoint() + + # we start from step 1 + self.global_steps += 1 + + for epoch in range(self.config.trainer.total_epochs): + for batch_dict in self.train_dataloader: + metrics = {} + timing_raw = {} + + batch: DataProto = DataProto.from_single_dict(batch_dict) + + # pop those keys for generation + gen_batch = batch.pop(batch_keys=['input_ids', 'attention_mask', 'position_ids'], non_tensor_batch_keys=['item_id', 'raw_prompt']) + gen_batch.meta_info['global_steps'] = self.global_steps + gen_batch.meta_info['max_rounds'] = self.rounds_scheduler.get_rounds() + metrics.update({ + 'max_rounds': self.rounds_scheduler.get_rounds(), + }) + + with _timer('step', timing_raw): + # generate a batch + with _timer('gen', timing_raw): + gen_batch_output = self.actor_rollout_wg.generate_sequences(gen_batch) + + if self.config.algorithm.adv_estimator == 'remax': + with _timer('gen_max', timing_raw): + gen_baseline_batch = deepcopy(gen_batch) + gen_baseline_batch.meta_info['do_sample'] = False + gen_baseline_output = self.actor_rollout_wg.generate_sequences(gen_baseline_batch) + + batch = batch.union(gen_baseline_output) + reward_baseline_tensor = batch.batch['rewards'] + reward_baseline_tensor = reward_baseline_tensor.sum(dim=-1) + + batch.pop(batch_keys=list(gen_baseline_output.batch.keys())) + + batch.batch['reward_baselines'] = reward_baseline_tensor + + del gen_baseline_batch, gen_baseline_output + + batch.non_tensor_batch['uid'] = np.array([str(uuid.uuid4()) for _ in range(len(batch.batch))], + dtype=object) + # repeat to align with repeated responses in rollout + batch = batch.repeat(repeat_times=self.config.actor_rollout_ref.rollout.n, interleave=True) + batch = batch.union(gen_batch_output) + + # balance the number of valid tokens on each dp rank. + # Note that this breaks the order of data inside the batch. + # Please take care when you implement group based adv computation such as GRPO and rloo + self._balance_batch(batch, metrics=metrics) + + # compute global_valid tokens + batch.meta_info['global_token_num'] = torch.sum(batch.batch['attention_mask'], dim=-1).tolist() + + # recompute old_log_probs + with _timer('old_log_prob', timing_raw): + old_log_prob = self.actor_rollout_wg.compute_log_prob(batch) + batch = batch.union(old_log_prob) + + if self.use_reference_policy: + # compute reference log_prob + with _timer('ref', timing_raw): + ref_log_prob = self.ref_policy_wg.compute_ref_log_prob(batch) + batch = batch.union(ref_log_prob) + + # compute values + if self.use_critic: + with _timer('values', timing_raw): + values = self.critic_wg.compute_values(batch) + batch = batch.union(values) + + with _timer('adv', timing_raw): + # we combine with rule-based rm + reward_tensor = batch.batch['scores'] + batch.batch['token_level_scores'] = reward_tensor + + # compute rewards. apply_kl_penalty if available + if not self.config.actor_rollout_ref.actor.get('use_kl_loss', False): + batch, kl_metrics = apply_kl_penalty(batch, + kl_ctrl=self.kl_ctrl, + kl_penalty=self.config.algorithm.kl_penalty) + metrics.update(kl_metrics) + else: + batch.batch['token_level_rewards'] = batch.batch['token_level_scores'] + + # compute advantages, executed on the driver process + batch = compute_advantage(batch, + adv_estimator=self.config.algorithm.adv_estimator, + gamma=self.config.algorithm.gamma, + lam=self.config.algorithm.lam, + num_repeat=self.config.actor_rollout_ref.rollout.n) + + # update critic + if self.use_critic: + with _timer('update_critic', timing_raw): + critic_output = self.critic_wg.update_critic(batch) + critic_output_metrics = reduce_metrics(critic_output.meta_info['metrics']) + metrics.update(critic_output_metrics) + + # implement critic warmup + if self.config.trainer.critic_warmup <= self.global_steps: + # update actor + with _timer('update_actor', timing_raw): + actor_output = self.actor_rollout_wg.update_actor(batch) + actor_output_metrics = reduce_metrics(actor_output.meta_info['metrics']) + metrics.update(actor_output_metrics) + + if self.config.trainer.save_freq > 0 and \ + self.global_steps % self.config.trainer.save_freq == 0: + with _timer('save_checkpoint', timing_raw): + self._save_checkpoint() + + # collect metrics + metrics.update(compute_data_metrics(batch=batch, use_critic=self.use_critic)) + metrics.update(compute_timing_metrics(batch=batch, timing_raw=timing_raw)) + + # TODO: make a canonical logger that supports various backend + logger.log(data=metrics, step=self.global_steps) + + self.global_steps += 1 + self.rounds_scheduler.step() + + if self.global_steps >= self.total_training_steps: + + if self.config.trainer.save_freq > 0 and \ + (self.global_steps - 1) % self.config.trainer.save_freq != 0: + with _timer('save_checkpoint', timing_raw): + self._save_checkpoint() + return diff --git a/frameworks/agentgym_rl/verl/agent_trainer/runtime_env.yaml b/frameworks/agentgym_rl/verl/agent_trainer/runtime_env.yaml new file mode 100644 index 0000000000000000000000000000000000000000..87bd05a9aabbc5db602626895518bb19add408d1 --- /dev/null +++ b/frameworks/agentgym_rl/verl/agent_trainer/runtime_env.yaml @@ -0,0 +1,5 @@ +working_dir: ./ +excludes: ["/.git/"] +env_vars: + TORCH_NCCL_AVOID_RECORD_STREAMS: "1" + VLLM_ATTENTION_BACKEND: "XFORMERS" \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/models/README.md b/frameworks/agentgym_rl/verl/models/README.md new file mode 100644 index 0000000000000000000000000000000000000000..677b92f3871aa2f76a7f5bd8c07d1050bab14564 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/README.md @@ -0,0 +1,35 @@ +# Models +Common modelzoo such as huggingface/transformers stuggles when using Pytorch native model parallelism. Following the design principle of vLLM, we keep a simple, parallelizable, highly-optimized with packed inputs in verl. +## Adding a New Huggingface Model +### Step 1: Copy the model file from HF to verl +- Add a new file under verl/models/hf +- Copy ONLY the model file from huggingface/transformers/models to verl/models/hf + +### Step 2: Modify the model file to use packed inputs +- Remove all the code related to inference (kv cache) +- Modify the inputs to include only + - input_ids (total_nnz,) + - cu_seqlens (total_nnz + 1,) + - max_seqlen_in_batch: int +- Note that this requires using flash attention with causal mask. + +### Step 2.5: Add tests +- Add a test to compare this version and the huggingface version +- Following the infrastructure and add tests to tests/models/hf + +### Step 3: Add a function to apply tensor parallelism +- Please follow + - https://pytorch.org/docs/stable/distributed.tensor.parallel.html + - https://pytorch.org/tutorials/intermediate/TP_tutorial.html +- General comments + - Tensor Parallelism in native Pytorch is NOT auto-parallelism. The way it works is to specify how model parameters and input/output reshards using configs. These configs are then registered as hooks to perform input/output resharding before/after model forward. + +### Step 4: Add a function to apply data parallelism +- Please use FSDP2 APIs +- See demo here https://github.com/pytorch/torchtitan/blob/main/torchtitan/parallelisms/parallelize_llama.py#L413 + +### Step 5: Add a function to apply pipeline parallelism +- Comes in Pytorch 2.4 +- Currently only in alpha in nightly version +- Check torchtitan for more details + diff --git a/frameworks/agentgym_rl/verl/models/__init__.py b/frameworks/agentgym_rl/verl/models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/models/llama/__init__.py b/frameworks/agentgym_rl/verl/models/llama/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/__init__.py b/frameworks/agentgym_rl/verl/models/llama/megatron/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..b188b3ee62cdfb978fc482984b423ce12e40a962 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/__init__.py @@ -0,0 +1,24 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .modeling_llama_megatron import ( + # original model with megatron + ParallelLlamaModel, + ParallelLlamaForCausalLM, + # rmpad with megatron + ParallelLlamaForCausalLMRmPad, + ParallelLlamaForValueRmPad, + # rmpad with megatron and pipeline parallelism + ParallelLlamaForCausalLMRmPadPP, + ParallelLlamaForValueRmPadPP) diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/__init__.py b/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/llama_loader.py b/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/llama_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..00fb0a9c668be28b4e13abb9a24e42bd7498d088 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/llama_loader.py @@ -0,0 +1,446 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +import time +from typing import Dict, Any, Callable, Optional +import torch.distributed as dist + + +def _megatron_calc_layer_map(config): + """Calculate the mapping of global layer_idx to local layer_idx + Returns: + layer_map (Dict: int -> tuple(int, int, int)): + mapping from the global layer index to + a tuple of (pp_rank, virtual_pp_rank, layer_idx inside model) + """ + import megatron + from megatron.core import mpu + + pp_size = mpu.get_pipeline_model_parallel_world_size() + virtual_pp_size = mpu.get_virtual_pipeline_model_parallel_world_size() or 1 + + layer_map = dict() + num_layers_per_model = config.num_hidden_layers // pp_size // virtual_pp_size + assert num_layers_per_model * pp_size * virtual_pp_size == config.num_hidden_layers + + for pp_rank_idx in range(pp_size): + for virtual_pp_rank_idx in range(virtual_pp_size): + layer_offset = (virtual_pp_rank_idx * (config.num_hidden_layers // virtual_pp_size) + + pp_rank_idx * num_layers_per_model) + for layer_idx in range(num_layers_per_model): + layer_map[layer_offset + layer_idx] = ( + pp_rank_idx, + virtual_pp_rank_idx, + layer_idx, + ) + return layer_map + + +def load_state_dict_to_megatron_llama(state_dict, wrapped_models, config, params_dtype, is_value_model=False): + """Load merged state_dict to sharded Megatron module in training. + """ + import megatron + from megatron.core import mpu + from megatron.utils import print_rank_0, unwrap_model + from megatron.core.transformer.module import Float16Module + from megatron.core import DistributedDataParallel as LocalDDP + from torch.nn.parallel import DistributedDataParallel as torchDDP + + start_time = time.time() + + def _get_gpt_model(model): + return model + + def broadcast_params(module): + for param in module.parameters(): + torch.distributed.broadcast(param.data, + src=mpu.get_data_parallel_src_rank(), + group=mpu.get_data_parallel_group()) + + dp_rank = mpu.get_data_parallel_rank() + pp_rank = mpu.get_pipeline_model_parallel_rank() + pp_size = mpu.get_pipeline_model_parallel_world_size() + virtual_pp_size = mpu.get_virtual_pipeline_model_parallel_world_size() or 1 + mp_group = mpu.get_model_parallel_group() + + if torch.distributed.get_rank() == 0: + assert mp_group.rank() == 0, f"mp_rank:[{mp_group.rank}] != 0 on rank #0" + assert pp_rank == 0, f"pp_rank:[{pp_rank}] != 0 on rank #0" + assert dp_rank == 0, f"dp_rank:[{dp_rank}] != 0 on rank #0" + + if not isinstance(wrapped_models, (list, tuple)): + wrapped_models = list(wrapped_models) + + assert len(wrapped_models) == virtual_pp_size + num_layers_per_model = config.num_hidden_layers // pp_size // virtual_pp_size + assert num_layers_per_model * pp_size * virtual_pp_size == config.num_hidden_layers + + models = [None] * len(wrapped_models) + + for i, wrapped_model in enumerate(wrapped_models): + models[i] = unwrap_model(wrapped_model, (torchDDP, LocalDDP, Float16Module)) + gpt_model_module = _get_gpt_model(models[i]) + assert len(gpt_model_module.model.layers) == num_layers_per_model + + def _broadcast_tensor(tensor, name) -> torch.Tensor: + """broadcast tensor from rank0 across mp_group""" + nonlocal state_dict + nonlocal mp_group + if torch.distributed.get_rank() == 0: + if name in state_dict: + weight = state_dict[name] + tensor_shape = weight.shape + else: + tensor_shape = None + else: + weight = None + tensor_shape = None + + obj_list = [tensor_shape] + dist.broadcast_object_list(obj_list, src=0, group=mp_group) + tensor_shape = obj_list[0] + + if tensor_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tensor:[{name}] not in state_dict, skip load") + return + + if tensor is None: + tensor = torch.empty( + tensor_shape, + dtype=params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + if torch.distributed.get_rank() == 0: + tensor.data.copy_(weight) + dist.broadcast(tensor, src=0, group=mp_group) + + def _broadcast_tp_shard_tensor_vocab(tensor, name, chunk_dim=0, mutate_func=None) -> torch.Tensor: + """broadcast tensor in tp shards across mp_group""" + nonlocal state_dict + nonlocal mp_group + tp_rank = mpu.get_tensor_model_parallel_rank() + tp_size = mpu.get_tensor_model_parallel_world_size() + + if torch.distributed.get_rank() == 0: + if name in state_dict: + full_weight = state_dict[name] + + if mutate_func is not None: + full_weight = mutate_func(full_weight) + tensor_chunk = torch.chunk(full_weight, tp_size, dim=chunk_dim) + chunk_shape = tensor_chunk[0].shape + else: + chunk_shape = None + else: + chunk_shape = None + + obj_list = [chunk_shape] + dist.broadcast_object_list(obj_list, src=0, group=mp_group) + chunk_shape = obj_list[0] + if chunk_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tp_shard tensor:[{name}] not in state_dict, skip loading") + return + + if tensor is None: + sync_tensor = torch.empty( + chunk_shape, + dtype=params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + else: + assert (tensor.shape == chunk_shape + ), f"rank #{torch.distributed.get_rank()} tensor {name} shape {tensor.shape} != {chunk_shape}" + sync_tensor = torch.empty_like(tensor, device=torch.cuda.current_device(), requires_grad=False) + + for i in range(tp_size): + if torch.distributed.get_rank() == 0: + sync_tensor.data.copy_(tensor_chunk[i]) + dist.broadcast(sync_tensor, src=0, group=mp_group) + if (i == tp_rank) and (tensor is not None): + tensor.data.copy_(sync_tensor) + + def _broadcast_tp_shard_tensor(tensor, name, chunk_dim=0, mutate_func=None) -> torch.Tensor: + """broadcast tensor in tp shards across mp_group""" + nonlocal state_dict + nonlocal mp_group + tp_rank = mpu.get_tensor_model_parallel_rank() + tp_size = mpu.get_tensor_model_parallel_world_size() + + if torch.distributed.get_rank() == 0: + if name in state_dict: + full_weight = state_dict[name] + if mutate_func is not None: + full_weight = mutate_func(full_weight) + tensor_chunk = torch.chunk(full_weight, tp_size, dim=chunk_dim) + chunk_shape = tensor_chunk[0].shape + else: + chunk_shape = None + else: + chunk_shape = None + + obj_list = [chunk_shape] + dist.broadcast_object_list(obj_list, src=0, group=mp_group) + chunk_shape = obj_list[0] + if chunk_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tp_shard tensor:[{name}] not in state_dict, skip loading") + return + + if tensor is None: + sync_tensor = torch.empty( + chunk_shape, + dtype=params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + else: + assert (tensor.shape == chunk_shape + ), f"rank #{torch.distributed.get_rank()} tensor {name} shape {tensor.shape} != {chunk_shape}" + sync_tensor = torch.empty_like(tensor, device=torch.cuda.current_device(), requires_grad=False) + + for i in range(tp_size): + if torch.distributed.get_rank() == 0: + sync_tensor.data.copy_(tensor_chunk[i]) + dist.broadcast(sync_tensor, src=0, group=mp_group) + if (i == tp_rank) and (tensor is not None): + tensor.data.copy_(sync_tensor) + + def _broadcast_tp_shard_tensor_gate_up(tensor, gate_name, up_name) -> torch.Tensor: + """broadcast tensor in tp shards across mp_group""" + nonlocal state_dict + nonlocal mp_group + tp_rank = mpu.get_tensor_model_parallel_rank() + tp_size = mpu.get_tensor_model_parallel_world_size() + + if torch.distributed.get_rank() == 0: + gate_weight = state_dict[gate_name] + up_weight = state_dict[up_name] + new_gate_up_weight = torch.empty(config.intermediate_size * 2, + config.hidden_size, + dtype=params_dtype, + device=torch.cuda.current_device()) + for i in range(tp_size): + intermediate_size_tp = config.intermediate_size // tp_size + gate_weight_tp = gate_weight[i * intermediate_size_tp:(i + 1) * intermediate_size_tp] + up_weight_tp = up_weight[i * intermediate_size_tp:(i + 1) * intermediate_size_tp] + new_gate_up_weight[intermediate_size_tp * 2 * i:intermediate_size_tp * 2 * (i + 1)].copy_( + torch.cat([gate_weight_tp, up_weight_tp], dim=0)) + + tensor_chunk = torch.chunk(new_gate_up_weight, tp_size, dim=0) + chunk_shape = tensor_chunk[0].shape + else: + chunk_shape = None + + obj_list = [chunk_shape] + dist.broadcast_object_list(obj_list, src=0, group=mp_group) + chunk_shape = obj_list[0] + if chunk_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tp_shard tensor:[{gate_name, up_name}] not in state_dict, skip loading") + return + + if tensor is None: + sync_tensor = torch.empty( + chunk_shape, + dtype=params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + else: + assert ( + tensor.shape == chunk_shape + ), f"rank #{torch.distributed.get_rank() == 0:} tensor {gate_name, up_name} shape {tensor.shape} != {chunk_shape}" + sync_tensor = torch.empty_like(tensor, device=torch.cuda.current_device(), requires_grad=False) + + for i in range(tp_size): + if torch.distributed.get_rank() == 0: + sync_tensor.data.copy_(tensor_chunk[i]) + dist.broadcast(sync_tensor, src=0, group=mp_group) + if (i == tp_rank) and (tensor is not None): + tensor.data.copy_(sync_tensor) + + def _broadcast_tp_shard_tensor_qkv(tensor, q_name, k_name, v_name) -> torch.Tensor: + """broadcast tensor in tp shards across mp_group""" + nonlocal state_dict + nonlocal mp_group + tp_rank = mpu.get_tensor_model_parallel_rank() + tp_size = mpu.get_tensor_model_parallel_world_size() + + if torch.distributed.get_rank() == 0: + assert (q_name in state_dict and k_name in state_dict and v_name in state_dict) + full_weight_q = state_dict[q_name] + full_weight_k = state_dict[k_name] + full_weight_v = state_dict[v_name] + + hidden_size_per_head = config.hidden_size // config.num_attention_heads + + if config.num_key_value_heads >= tp_size: + q_size_tp = config.hidden_size // tp_size + kv_size_tp = hidden_size_per_head * config.num_key_value_heads // tp_size + total_size = q_size_tp + 2 * kv_size_tp + new_weight_qkv = torch.empty(total_size * tp_size, + config.hidden_size, + dtype=params_dtype, + device=torch.cuda.current_device()) + for i in range(tp_size): + q_part = full_weight_q[i * q_size_tp:(i + 1) * q_size_tp] + k_part = full_weight_k[i * kv_size_tp:(i + 1) * kv_size_tp] + v_part = full_weight_v[i * kv_size_tp:(i + 1) * kv_size_tp] + new_weight_qkv[i * total_size:(i + 1) * total_size].copy_(torch.cat([q_part, k_part, v_part], + dim=0)) + + else: + q_size_tp = config.hidden_size // tp_size + kv_size_tp = hidden_size_per_head + total_size = q_size_tp + 2 * kv_size_tp + new_weight_qkv = torch.empty(total_size * tp_size, + config.hidden_size, + dtype=params_dtype, + device=torch.cuda.current_device()) + for i in range(tp_size): + q_part = full_weight_q[i * q_size_tp:(i + 1) * q_size_tp] + start_idx = i * config.num_key_value_heads // tp_size * hidden_size_per_head + end_idx = (i * config.num_key_value_heads // tp_size + 1) * hidden_size_per_head + k_part = full_weight_k[start_idx:end_idx] + v_part = full_weight_v[start_idx:end_idx] + new_weight_qkv[i * total_size:(i + 1) * total_size].copy_(torch.cat([q_part, k_part, v_part], + dim=0)) + + tensor_chunk = torch.chunk(new_weight_qkv, tp_size, dim=0) + chunk_shape = tensor_chunk[0].shape + else: + chunk_shape = None + + obj_list = [chunk_shape] + dist.broadcast_object_list(obj_list, src=0, group=mp_group) + chunk_shape = obj_list[0] + if chunk_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tp_shard tensor:[{name}] not in state_dict, skip loading") + return + + if tensor is None: + sync_tensor = torch.empty( + chunk_shape, + dtype=params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + else: + assert (tensor.shape == chunk_shape + ), f"rank #{torch.distributed.get_rank()} tensor {q_name} shape {tensor.shape} != {chunk_shape}" + sync_tensor = torch.empty_like(tensor, device=torch.cuda.current_device(), requires_grad=False) + + for i in range(tp_size): + if torch.distributed.get_rank() == 0: + sync_tensor.data.copy_(tensor_chunk[i]) + dist.broadcast(sync_tensor, src=0, group=mp_group) + if (i == tp_rank) and (tensor is not None): + tensor.data.copy_(sync_tensor) + + if dp_rank == 0: + # Embeddings + # ------------------- + print_rank_0("loading embeddings...") + gpt_model_module = _get_gpt_model(models[0]) + embed_tokens_weight = None + if pp_rank == 0: + embed_tokens_weight = gpt_model_module.model.embed_tokens.weight + _broadcast_tp_shard_tensor_vocab(embed_tokens_weight, "model.embed_tokens.weight") + + # Transformer layers + # ------------------- + layer_map = _megatron_calc_layer_map(config) + + for layer in range(config.num_hidden_layers): + print_rank_0(f"loading layer #{layer}...") + layer_name = f"model.layers.{layer}" + dst_pp_rank, dst_virtual_pp_rank, dst_layer_idx = layer_map[layer] + + gpt_model_module = _get_gpt_model(models[dst_virtual_pp_rank]) + sync_layer = gpt_model_module.model.layers[dst_layer_idx] + + _broadcast_tensor( + sync_layer.input_layernorm.weight if dst_pp_rank == pp_rank else None, + f"{layer_name}.input_layernorm.weight", + ) + + _broadcast_tp_shard_tensor_qkv( + sync_layer.self_attn.qkv_proj.weight if dst_pp_rank == pp_rank else None, + f"{layer_name}.self_attn.q_proj.weight", + f"{layer_name}.self_attn.k_proj.weight", + f"{layer_name}.self_attn.v_proj.weight", + ) + + _broadcast_tp_shard_tensor( + sync_layer.self_attn.o_proj.weight if dst_pp_rank == pp_rank else None, + f"{layer_name}.self_attn.o_proj.weight", + chunk_dim=1, + ) + + _broadcast_tensor( + sync_layer.post_attention_layernorm.weight if dst_pp_rank == pp_rank else None, + f"{layer_name}.post_attention_layernorm.weight", + ) + + _broadcast_tp_shard_tensor_gate_up(sync_layer.mlp.gate_up_proj.weight if dst_pp_rank == pp_rank else None, + f"{layer_name}.mlp.gate_proj.weight", f"{layer_name}.mlp.up_proj.weight") + + _broadcast_tp_shard_tensor( + sync_layer.mlp.down_proj.weight if dst_pp_rank == pp_rank else None, + f"{layer_name}.mlp.down_proj.weight", + chunk_dim=1, + ) + # Final Layernorm + # ------------------- + print_rank_0("loading final layernorm...") + gpt_model_module = _get_gpt_model(models[-1]) + _broadcast_tensor( + getattr(gpt_model_module.model.norm, "weight", None), + "model.norm.weight", + ) + + print_rank_0("loading lm_head...") + lm_head_weight = None + if pp_rank + 1 == pp_size: + lm_head_weight = gpt_model_module.lm_head.weight + + if is_value_model: + # if torch.distributed.get_rank() == 0: + if 'lm_head.weight' in state_dict and state_dict['lm_head.weight'].shape[0] == 1: + _broadcast_tensor(lm_head_weight, "lm_head.weight") + elif 'reward_head.weight' in state_dict and state_dict['reward_head.weight'].shape[0] == 1: + _broadcast_tensor(lm_head_weight, "reward_head.weight") + print_rank_0('load lm_head from value_head weight') + else: + _broadcast_tensor(None, "lm_head.weight") + print_rank_0('fail to match lm_head in value_model') + # else: + + # _broadcast_tensor(lm_head_weight, "lm_head.weight") + + else: + _broadcast_tp_shard_tensor(lm_head_weight, "lm_head.weight") + dist.barrier() + # Broadcast weights inside data parallel groups + for wrapped_model in wrapped_models: + broadcast_params(wrapped_model) + + torch.cuda.empty_cache() + print_rank_0(f"loading megatron ckpt done, time elapsed {time.time() - start_time}s") diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/llama_saver.py b/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/llama_saver.py new file mode 100644 index 0000000000000000000000000000000000000000..0764b6fe5020dc8ab3f69d57af9910e267aab52d --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/checkpoint_utils/llama_saver.py @@ -0,0 +1,449 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import megatron +from megatron.core import mpu +from megatron.utils import print_rank_0, unwrap_model +from megatron.model import Float16Module +from megatron.model import DistributedDataParallel as LocalDDP +from torch.nn.parallel import DistributedDataParallel as torchDDP +import torch +import time +from typing import Optional +import torch.distributed as dist +from megatron import get_args + + +def _megatron_calc_global_rank(tp_rank: int = 0, dp_rank: int = 0, pp_rank: int = 0): + """given TP,DP,PP rank to get the global rank.""" + + args = get_args() + tp_size = mpu.get_tensor_model_parallel_world_size() + dp_size = mpu.get_data_parallel_world_size() + pp_size = mpu.get_pipeline_model_parallel_world_size() + assert (tp_size * dp_size * pp_size == torch.distributed.get_world_size() + ), f"{tp_size} x {dp_size} x {pp_size} != {torch.distributed.get_world_size()}" + if args.switch_dp_and_pp_grouping: + # TP-PP-DP grouping + return (dp_rank * pp_size + pp_rank) * tp_size + tp_rank + else: + # TP-DP-PP grouping + return (pp_rank * dp_size + dp_rank) * tp_size + tp_rank + + +def _megatron_calc_layer_map(config): + """Calculate the mapping of global layer_idx to local layer_idx + Returns: + layer_map (Dict: int -> tuple(int, int, int)): + mapping from the global layer index to + a tuple of (pp_rank, virtual_pp_rank, layer_idx inside model) + """ + import megatron + from megatron.core import mpu + + pp_size = mpu.get_pipeline_model_parallel_world_size() + virtual_pp_size = mpu.get_virtual_pipeline_model_parallel_world_size() or 1 + + args = megatron.get_args() + layer_map = dict() + num_layers_per_model = config.num_hidden_layers // pp_size // virtual_pp_size + assert num_layers_per_model * pp_size * virtual_pp_size == config.num_hidden_layers + + for pp_rank_idx in range(pp_size): + for virtual_pp_rank_idx in range(virtual_pp_size): + layer_offset = (virtual_pp_rank_idx * (config.num_hidden_layers // virtual_pp_size) + + pp_rank_idx * num_layers_per_model) + for layer_idx in range(num_layers_per_model): + layer_map[layer_offset + layer_idx] = ( + pp_rank_idx, + virtual_pp_rank_idx, + layer_idx, + ) + return layer_map + + +def merge_megatron_ckpt_llama(wrapped_models, config, is_value_model=False, dtype='bf16'): + """Merge sharded parameters of a Megatron module into a merged checkpoint. + + Args: + wrapped_modelss (list of megatron.model.DistributedDataParallel): + The local DDP wrapped megatron modules. + dtype (str or None): + The data type of state_dict. if None, the data type of the original parameters + is used. + gpt_model_key: key to access model + Returns: + state_dict (dict): + The merged state_dict in rank 0, and an empty dictionary in other ranks. + """ + start_time = time.time() + args = megatron.get_args() + + def _get_gpt_model(model): + return model + + dp_rank = mpu.get_data_parallel_rank() + pp_size = mpu.get_pipeline_model_parallel_world_size() + pp_rank = mpu.get_pipeline_model_parallel_rank() + virtual_pp_size = mpu.get_virtual_pipeline_model_parallel_world_size() or 1 + mp_group = mpu.get_model_parallel_group() + + if dist.get_rank() == 0: + assert mp_group.rank() == 0, f"mp_rank:[{mp_group.rank}] != 0 on rank #0" + assert pp_rank == 0, f"pp_rank:[{pp_rank}] != 0 on rank #0" + assert dp_rank == 0, f"dp_rank:[{dp_rank}] != 0 on rank #0" + + if not isinstance(wrapped_models, (list, tuple)): + wrapped_models = list(wrapped_models) + + assert len(wrapped_models) == virtual_pp_size + num_layers_per_model = config.num_hidden_layers // pp_size // virtual_pp_size + assert num_layers_per_model * pp_size * virtual_pp_size == config.num_hidden_layers + + models = [None] * len(wrapped_models) + + for i, wrapped_model in enumerate(wrapped_models): + models[i] = unwrap_model(wrapped_model, (torchDDP, LocalDDP, Float16Module)) + assert len(models[i].model.layers + ) == num_layers_per_model, 'len model layers {} not equal to num_layers_per_model {}'.format( + len(models[i].model.layers), num_layers_per_model) + + state_dict = dict() + + def _get_cpu_tensor(tensor: torch.Tensor): + if tensor is None: + return None + if tensor.device == torch.device("cpu"): + return tensor.detach().clone() + return tensor.detach().cpu() + + def _broadcast_tensor(tensor, name, src_pp_rank) -> torch.Tensor: + """broadcast tensor across mp_group""" + nonlocal state_dict + nonlocal mp_group + src_rank = _megatron_calc_global_rank(tp_rank=0, dp_rank=0, pp_rank=src_pp_rank) + + if torch.distributed.get_rank() == src_rank: + if tensor is None: + weight = None + tensor_shape = None + else: + weight = tensor + tensor_shape = weight.shape + else: + weight = None + tensor_shape = None + + obj_list = [tensor_shape] + dist.broadcast_object_list(obj_list, src=src_rank, group=mp_group) + tensor_shape = obj_list[0] + + if tensor_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tensor:[{name}] not exist, skip collect") + return + + if weight is None: + weight = torch.empty( + tensor_shape, + dtype=args.params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + + dist.broadcast(weight, src=src_rank, group=mp_group) + + if torch.distributed.get_rank() == 0: + state_dict[name] = _get_cpu_tensor(weight) + + def _broadcast_tp_shard_tensor(tensor, name, src_pp_rank, concat_dim=0, mutate_func=None) -> torch.Tensor: + """broadcast tensor in tp shards across mp_group""" + nonlocal state_dict + nonlocal mp_group + tp_rank = mpu.get_tensor_model_parallel_rank() + tp_size = mpu.get_tensor_model_parallel_world_size() + src_rank = _megatron_calc_global_rank(tp_rank=0, dp_rank=0, pp_rank=src_pp_rank) + + if torch.distributed.get_rank() == src_rank: + chunk_shape = tensor.shape + else: + chunk_shape = None + + obj_list = [chunk_shape] + dist.broadcast_object_list(obj_list, src=src_rank, group=mp_group) + chunk_shape = obj_list[0] + if chunk_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tp_shard tensor:[{name}] not exist, skip collecting") + return + + buffer_tensor = torch.empty( + chunk_shape, + dtype=args.params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + + chunk_tensors = [None] * tp_size + + for i in range(tp_size): + cur_src_rank = _megatron_calc_global_rank(tp_rank=i, dp_rank=0, pp_rank=src_pp_rank) + sync_tensor = tensor if torch.distributed.get_rank() == cur_src_rank else buffer_tensor + dist.broadcast(sync_tensor, src=cur_src_rank, group=mp_group) + + if torch.distributed.get_rank() == 0: + chunk_tensors[i] = _get_cpu_tensor(sync_tensor) + + if torch.distributed.get_rank() == 0: + full_tensor = torch.concat(chunk_tensors, dim=concat_dim) + if mutate_func is not None: + full_tensor = mutate_func(full_tensor) + state_dict[name] = full_tensor + + def _broadcast_tp_shard_tensor_gate_up(tensor, gate_name, up_name, src_pp_rank) -> torch.Tensor: + """broadcast tensor in tp shards across mp_group""" + nonlocal state_dict + nonlocal mp_group + tp_rank = mpu.get_tensor_model_parallel_rank() + tp_size = mpu.get_tensor_model_parallel_world_size() + src_rank = _megatron_calc_global_rank(tp_rank=0, dp_rank=0, pp_rank=src_pp_rank) + + if torch.distributed.get_rank() == src_rank: + chunk_shape = tensor.shape + else: + chunk_shape = None + + obj_list = [chunk_shape] + dist.broadcast_object_list(obj_list, src=src_rank, group=mp_group) + chunk_shape = obj_list[0] + if chunk_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tp_shard tensor:[{gate_name, up_name}] not exist, skip collecting") + return + + buffer_tensor = torch.empty( + chunk_shape, + dtype=args.params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + + chunk_tensors = [None] * tp_size + + for i in range(tp_size): + cur_src_rank = _megatron_calc_global_rank(tp_rank=i, dp_rank=0, pp_rank=src_pp_rank) + sync_tensor = tensor if torch.distributed.get_rank() == cur_src_rank else buffer_tensor + dist.broadcast(sync_tensor, src=cur_src_rank, group=mp_group) + + if torch.distributed.get_rank() == 0: + chunk_tensors[i] = _get_cpu_tensor(sync_tensor) + + if torch.distributed.get_rank() == 0: + full_tensor = torch.concat(chunk_tensors, dim=0) + intermediate_size_tp = config.intermediate_size // tp_size + gate_weight_list = [] + up_weight_list = [] + for i in range(tp_size): + gate_up_weight_tp = full_tensor[intermediate_size_tp * 2 * i:intermediate_size_tp * 2 * (i + 1)] + gate_weight_tp = gate_up_weight_tp[:intermediate_size_tp] + up_weight_tp = gate_up_weight_tp[intermediate_size_tp:] + gate_weight_list.append(gate_weight_tp) + up_weight_list.append(up_weight_tp) + + state_dict[gate_name] = torch.cat(gate_weight_list, dim=0) + state_dict[up_name] = torch.cat(up_weight_list, dim=0) + + def _broadcast_tp_shard_tensor_qkv(tensor, q_name, k_name, v_name, src_pp_rank): + """broadcast tensor in tp shards across mp_group""" + nonlocal state_dict + nonlocal mp_group + tp_rank = mpu.get_tensor_model_parallel_rank() + tp_size = mpu.get_tensor_model_parallel_world_size() + src_rank = _megatron_calc_global_rank(tp_rank=0, dp_rank=0, pp_rank=src_pp_rank) + + if torch.distributed.get_rank() == src_rank: + chunk_shape = tensor.shape + else: + chunk_shape = None + + obj_list = [chunk_shape] + dist.broadcast_object_list(obj_list, src=src_rank, group=mp_group) + chunk_shape = obj_list[0] + if chunk_shape is None: + # all or none ranks in the mp_group should reach here + print_rank_0(f"tp_shard tensor:[{q_name}] not exist, skip collecting") + return + + buffer_tensor = torch.empty( + chunk_shape, + dtype=args.params_dtype, + device=torch.cuda.current_device(), + requires_grad=False, + ) + + chunk_tensors = [None] * tp_size + + for i in range(tp_size): + cur_src_rank = _megatron_calc_global_rank(tp_rank=i, dp_rank=0, pp_rank=src_pp_rank) + sync_tensor = tensor if torch.distributed.get_rank() == cur_src_rank else buffer_tensor + dist.broadcast(sync_tensor, src=cur_src_rank, group=mp_group) + + if torch.distributed.get_rank() == 0: + chunk_tensors[i] = _get_cpu_tensor(sync_tensor) + + if torch.distributed.get_rank() == 0: + full_tensor = torch.concat(chunk_tensors, dim=0) + q_weight_list = [] + k_weight_list = [] + v_weight_list = [] + hidden_size_per_head = config.hidden_size // config.num_attention_heads + + if config.num_key_value_heads >= tp_size: + q_size_tp = config.hidden_size // tp_size + kv_size_tp = hidden_size_per_head * config.num_key_value_heads // tp_size + total_size = q_size_tp + 2 * kv_size_tp + for i in range(tp_size): + qkv_part = full_tensor[i * total_size:(i + 1) * total_size] + q_part = qkv_part[:q_size_tp] + k_part = qkv_part[q_size_tp:q_size_tp + kv_size_tp] + v_part = qkv_part[q_size_tp + kv_size_tp:total_size] + q_weight_list.append(q_part) + k_weight_list.append(k_part) + v_weight_list.append(v_part) + else: + q_size_tp = config.hidden_size // tp_size + kv_size_tp = hidden_size_per_head + total_size = q_size_tp + 2 * kv_size_tp + for i in range(tp_size): + qkv_part = full_tensor[i * total_size:(i + 1) * total_size] + q_part = qkv_part[:q_size_tp] + k_part = qkv_part[q_size_tp:q_size_tp + kv_size_tp] + v_part = qkv_part[q_size_tp + kv_size_tp:total_size] + q_weight_list.append(q_part) + if i * config.num_key_value_heads % tp_size == 0: + k_weight_list.append(k_part) + v_weight_list.append(v_part) + + state_dict[q_name] = torch.cat(q_weight_list, dim=0) + state_dict[k_name] = torch.cat(k_weight_list, dim=0) + state_dict[v_name] = torch.cat(v_weight_list, dim=0) + + # empty cache before collecting weights + torch.cuda.empty_cache() + # Embeddings + # ------------------- + if dp_rank == 0: + # Embeddings + # ------------------- + print_rank_0("collecting embeddings...") + gpt_model_module = _get_gpt_model(models[0]) + _broadcast_tp_shard_tensor( + gpt_model_module.model.embed_tokens.weight if pp_rank == 0 else None, + "model.embed_tokens.weight", + src_pp_rank=0, + ) + + # Transformer layers + # ------------------- + layer_map = _megatron_calc_layer_map(config) + for layer in range(config.num_hidden_layers): + print_rank_0(f"collecting layer #{layer}...") + layer_name = f"model.layers.{layer}" + src_pp_rank, src_virtual_pp_rank, src_layer_idx = layer_map[layer] + + gpt_model_module = _get_gpt_model(models[src_virtual_pp_rank]) + sync_layer = gpt_model_module.model.layers[src_layer_idx] + + _broadcast_tensor( + sync_layer.input_layernorm.weight, + f"{layer_name}.input_layernorm.weight", + src_pp_rank=src_pp_rank, + ) + + _broadcast_tp_shard_tensor_qkv( + sync_layer.self_attn.qkv_proj.weight, + f"{layer_name}.self_attn.q_proj.weight", + f"{layer_name}.self_attn.k_proj.weight", + f"{layer_name}.self_attn.v_proj.weight", + src_pp_rank=src_pp_rank, + ) + + _broadcast_tp_shard_tensor( + sync_layer.self_attn.o_proj.weight, + f"{layer_name}.self_attn.o_proj.weight", + concat_dim=1, + src_pp_rank=src_pp_rank, + ) + + _broadcast_tensor( + sync_layer.post_attention_layernorm.weight, + f"{layer_name}.post_attention_layernorm.weight", + src_pp_rank=src_pp_rank, + ) + + _broadcast_tp_shard_tensor_gate_up(sync_layer.mlp.gate_up_proj.weight, + f"{layer_name}.mlp.gate_proj.weight", + f"{layer_name}.mlp.up_proj.weight", + src_pp_rank=src_pp_rank) + + _broadcast_tp_shard_tensor( + sync_layer.mlp.down_proj.weight, + f"{layer_name}.mlp.down_proj.weight", + concat_dim=1, + src_pp_rank=src_pp_rank, + ) + + # Final Layernorm + # ------------------- + print_rank_0("collecting final layernorm...") + gpt_model_module = _get_gpt_model(models[-1]) + _broadcast_tensor( + getattr(gpt_model_module.model.norm, "weight", None), + "model.norm.weight", + src_pp_rank=pp_size - 1, + ) + + print_rank_0("collecting lm_head...") + + if is_value_model: + _broadcast_tensor(getattr(gpt_model_module.lm_head, "weight", None) if pp_rank == pp_size - 1 else None, + "reward_head.weight", + src_pp_rank=pp_size - 1) + + else: + _broadcast_tp_shard_tensor( + getattr(gpt_model_module.lm_head, "weight", None) if pp_rank == pp_size - 1 else None, + "lm_head.weight", + src_pp_rank=pp_size - 1, + ) + + dist.barrier() + + torch.cuda.empty_cache() + if torch.distributed.get_rank() == 0: + if dtype == "fp16": + dtype = torch.float16 + elif dtype == "bf16": + dtype = torch.bfloat16 + elif dtype is None or dtype == "fp32": + dtype = torch.float32 + else: + print(f'Unknown/unsupported dtype to save: {dtype}"') + exit(1) + for k, v in state_dict.items(): + if dtype != v.dtype: + state_dict[k] = v.to(dtype) + + print_rank_0(f"merge megatron ckpt done, time elapsed {time.time() - start_time}s") + return state_dict diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/layers/__init__.py b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..3761bae7db33c29c66534b9ae4f1d8ec8f63b829 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/__init__.py @@ -0,0 +1,18 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .parallel_attention import ParallelLlamaAttention +from .parallel_decoder import ParallelLlamaDecoderLayer, ParallelLlamaDecoderLayerRmPad +from .parallel_mlp import ParallelLlamaMLP +from .parallel_rmsnorm import ParallelLlamaRMSNorm diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_attention.py b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..f14653fca4ade888f5ee08e32aa57711c1cf5e73 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_attention.py @@ -0,0 +1,418 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. +# +# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX +# and OPT implementations in this library. It has been modified from its +# original forms to accommodate minor architectural differences compared +# to GPT-NeoX and OPT used by the Meta AI team that trained the model. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math +from typing import Optional, Tuple + +import torch +from megatron.core import parallel_state as mpu +from megatron.core import tensor_parallel +from megatron.core import ModelParallelConfig +from torch import nn +from transformers import LlamaConfig +from verl.models.llama.megatron.layers.parallel_linear import QKVParallelLinear + +from verl.utils.megatron import tensor_parallel as tp_utils + + +class LlamaRotaryEmbedding(nn.Module): + + def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None): + super().__init__() + + self.dim = dim + self.max_position_embeddings = max_position_embeddings + self.base = base + inv_freq = 1.0 / (self.base**(torch.arange(0, self.dim, 2).float().to(device) / self.dim)) + self.register_buffer("inv_freq", inv_freq, persistent=False) + + # Build here to make `torch.jit.trace` work. + self._set_cos_sin_cache(seq_len=max_position_embeddings, + device=self.inv_freq.device, + dtype=torch.get_default_dtype()) + + def _set_cos_sin_cache(self, seq_len, device, dtype): + self.max_seq_len_cached = seq_len + t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype) + + freqs = torch.einsum("i,j->ij", t, self.inv_freq) + # Different from paper, but it uses a different permutation in order to obtain the same calculation + emb = torch.cat((freqs, freqs), dim=-1) + self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False) + self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False) + + def forward(self, x, seq_len=None): + # x: [bs, num_attention_heads, seq_len, head_size] + if seq_len > self.max_seq_len_cached: + self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype) + + return ( + self.cos_cached[:seq_len].to(dtype=x.dtype), + self.sin_cached[:seq_len].to(dtype=x.dtype), + ) + + +class LlamaLinearScalingRotaryEmbedding(LlamaRotaryEmbedding): + """LlamaRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev""" + + def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0): + self.scaling_factor = scaling_factor + super().__init__(dim, max_position_embeddings, base, device) + + def _set_cos_sin_cache(self, seq_len, device, dtype): + self.max_seq_len_cached = seq_len + t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype) + t = t / self.scaling_factor + + freqs = torch.einsum("i,j->ij", t, self.inv_freq) + # Different from paper, but it uses a different permutation in order to obtain the same calculation + emb = torch.cat((freqs, freqs), dim=-1) + self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False) + self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False) + + +class LlamaDynamicNTKScalingRotaryEmbedding(LlamaRotaryEmbedding): + """LlamaRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla""" + + def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0): + self.scaling_factor = scaling_factor + super().__init__(dim, max_position_embeddings, base, device) + + def _set_cos_sin_cache(self, seq_len, device, dtype): + self.max_seq_len_cached = seq_len + + if seq_len > self.max_position_embeddings: + base = self.base * ((self.scaling_factor * seq_len / self.max_position_embeddings) - + (self.scaling_factor - 1))**(self.dim / (self.dim - 2)) + inv_freq = 1.0 / (base**(torch.arange(0, self.dim, 2).float().to(device) / self.dim)) + self.register_buffer("inv_freq", inv_freq, persistent=False) + + t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype) + + freqs = torch.einsum("i,j->ij", t, self.inv_freq) + # Different from paper, but it uses a different permutation in order to obtain the same calculation + emb = torch.cat((freqs, freqs), dim=-1) + self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False) + self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False) + + +def rotate_half(x): + """Rotates half the hidden dims of the input.""" + x1 = x[..., :x.shape[-1] // 2] + x2 = x[..., x.shape[-1] // 2:] + return torch.cat((-x2, x1), dim=-1) + + +def apply_rotary_pos_emb(q, k, cos, sin, position_ids): + cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim] + sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim] + q_embed = (q * cos) + (rotate_half(q) * sin) + k_embed = (k * cos) + (rotate_half(k) * sin) + return q_embed, k_embed + + +def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor: + """ + This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, + num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) + """ + batch, num_key_value_heads, slen, head_dim = hidden_states.shape + if n_rep == 1: + return hidden_states + hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim) + return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim) + + +class ParallelLlamaAttention(nn.Module): + """Multi-headed attention from 'Attention Is All You Need' paper""" + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig): + super().__init__() + self.config = config + self.megatron_config = megatron_config + self.hidden_size = config.hidden_size + self.num_heads = config.num_attention_heads + self.head_dim = self.hidden_size // self.num_heads + self.num_key_value_heads = config.num_key_value_heads + self.num_key_value_groups = self.num_heads // self.num_key_value_heads + self.max_position_embeddings = config.max_position_embeddings + self.rope_theta = config.rope_theta + + # assign values after tp + tp_size = mpu.get_tensor_model_parallel_world_size() + assert self.num_heads % tp_size == 0, f'num_head must be divisible by tp_size. Got num_head={self.num_heads}, tp_size={tp_size}' + assert self.num_key_value_heads % tp_size == 0, \ + f'num_key_value_heads must be divisible by tp_size. Got num_key_value_heads={self.num_key_value_heads}, tp_size={tp_size}' + + self.num_heads_per_tp = self.num_heads // tp_size + self.num_key_value_heads_per_tp = self.num_key_value_heads // tp_size + self.hidden_size_per_tp = self.hidden_size // tp_size + + if (self.head_dim * self.num_heads) != self.hidden_size: + raise ValueError(f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}" + f" and `num_heads`: {self.num_heads}).") + + column_kwargs = tp_utils.get_default_kwargs_for_column_parallel_linear() + row_kwargs = tp_utils.get_default_kwargs_for_row_parallel_linear() + + if megatron_config is not None: + assert column_kwargs.get('config', False), 'must have ModelParallelConfig' + assert row_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(column_kwargs, megatron_config) + tp_utils.update_kwargs_with_config(row_kwargs, megatron_config) + + # [self.q_size, self.k_size, self.v_size] + self.qkv_proj = QKVParallelLinear(input_size=self.hidden_size, + num_heads=self.num_heads, + num_key_value_heads=self.num_key_value_heads, + head_dim=self.head_dim, + bias=config.attention_bias, + gather_output=False, + skip_bias_add=False, + **column_kwargs) + + self.q_size = self.num_heads_per_tp * self.head_dim + self.k_size = self.num_key_value_heads_per_tp * self.head_dim + self.v_size = self.num_key_value_heads_per_tp * self.head_dim + + self.o_proj = tensor_parallel.RowParallelLinear(input_size=self.num_heads * self.head_dim, + output_size=self.hidden_size, + bias=config.attention_bias, + input_is_parallel=True, + skip_bias_add=False, + **row_kwargs) + + self._init_rope() + + def _init_rope(self): + if self.config.rope_scaling is None: + self.rotary_emb = LlamaRotaryEmbedding( + self.head_dim, + max_position_embeddings=self.max_position_embeddings, + base=self.rope_theta, + ) + else: + scaling_type = self.config.rope_scaling["type"] + scaling_factor = self.config.rope_scaling["factor"] + if scaling_type == "linear": + self.rotary_emb = LlamaLinearScalingRotaryEmbedding( + self.head_dim, + max_position_embeddings=self.max_position_embeddings, + scaling_factor=scaling_factor, + base=self.rope_theta, + ) + elif scaling_type == "dynamic": + self.rotary_emb = LlamaDynamicNTKScalingRotaryEmbedding( + self.head_dim, + max_position_embeddings=self.max_position_embeddings, + scaling_factor=scaling_factor, + base=self.rope_theta, + ) + else: + raise ValueError(f"Unknown RoPE scaling type {scaling_type}") + + def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int): + return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous() + + def forward( + self, + hidden_states: torch.Tensor, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]: + bsz, q_len, _ = hidden_states.size() + qkv = self.qkv_proj(hidden_states)[0] + query_states, key_states, value_states = qkv.split([self.q_size, self.k_size, self.v_size], dim=-1) + + query_states = query_states.view(bsz, q_len, self.num_heads_per_tp, self.head_dim).transpose(1, 2) + key_states = key_states.view(bsz, q_len, self.num_key_value_heads_per_tp, self.head_dim).transpose(1, 2) + value_states = value_states.view(bsz, q_len, self.num_key_value_heads_per_tp, self.head_dim).transpose(1, 2) + + kv_seq_len = key_states.shape[-2] + cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len) + query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids) + + key_states = repeat_kv(key_states, self.num_key_value_groups) + value_states = repeat_kv(value_states, self.num_key_value_groups) + + attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim) + + if attn_weights.size() != (bsz, self.num_heads_per_tp, q_len, kv_seq_len): + raise ValueError( + f"Attention weights should be of size {(bsz, self.num_heads_per_tp, q_len, kv_seq_len)}, but is" + f" {attn_weights.size()}") + + if attention_mask is not None: + if attention_mask.size() != (bsz, 1, q_len, kv_seq_len): + raise ValueError( + f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}") + attn_weights = attn_weights + attention_mask + + # upcast attention to fp32 + attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype) + attn_output = torch.matmul(attn_weights, value_states) + + if attn_output.size() != (bsz, self.num_heads_per_tp, q_len, self.head_dim): + raise ValueError( + f"`attn_output` should be of size {(bsz, self.num_heads_per_tp, q_len, self.head_dim)}, but is" + f" {attn_output.size()}") + + attn_output = attn_output.transpose(1, 2).contiguous() + attn_output = attn_output.reshape(bsz, q_len, self.hidden_size_per_tp) + attn_output = self.o_proj(attn_output)[0] + return attn_output + + +""" +Remove padding Attention +- Using Flash-attn 2 +- Compatible with sequence parallel +""" + +from transformers.utils import is_flash_attn_2_available +import torch.nn.functional as F + +from einops import rearrange + +if is_flash_attn_2_available(): + from flash_attn import flash_attn_varlen_func + from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input # noqa + + +def apply_rotary_pos_emb_rmpad(q, k, cos, sin, position_ids, indices, sequence_length): + batch_size = position_ids.shape[0] + + q = pad_input(q, indices, batch_size, sequence_length) # (batch_size, seqlen, num_head, head_dim) + k = pad_input(k, indices, batch_size, sequence_length) + cos = cos[position_ids].unsqueeze(2) # [bs, seq_len, 1, dim] + sin = sin[position_ids].unsqueeze(2) # [bs, seq_len, 1, dim] + q_embed = (q * cos) + (rotate_half(q) * sin) + k_embed = (k * cos) + (rotate_half(k) * sin) + + q_embed = index_first_axis(rearrange(q_embed, "b s ... -> (b s) ..."), indices) + k_embed = index_first_axis(rearrange(k_embed, "b s ... -> (b s) ..."), indices) + + return q_embed, k_embed + + +from flash_attn.layers.rotary import apply_rotary_emb + + +# use flash-attn rotary embeddings with rmpad +# cos/sin shoudl be: (seq_length, rotary_dim / 2) +def apply_rotary_pos_emb_rmpad_flash(q, k, cos, sin, cu_seqlens, max_seqlen): + q_embed = apply_rotary_emb(q, + cos, + sin, + interleaved=False, + inplace=False, + cu_seqlens=cu_seqlens, + max_seqlen=max_seqlen) + k_embed = apply_rotary_emb(k, + cos, + sin, + interleaved=False, + inplace=False, + cu_seqlens=cu_seqlens, + max_seqlen=max_seqlen) + return q_embed, k_embed + + +class ParallelLlamaAttentionRmPad(ParallelLlamaAttention): + + def forward(self, + hidden_states: torch.Tensor, + position_ids: Optional[torch.LongTensor] = None, + sequence_length: int = None, + indices: torch.Tensor = None, + cu_seqlens: torch.Tensor = None, + max_seqlen_in_batch: int = None): + total_nnz, _, _ = hidden_states.size() # This is the total_nnz padded after sequence parallel + + if self.megatron_config.sequence_parallel: + total_nnz = total_nnz * mpu.get_tensor_model_parallel_world_size() + + qkv = self.qkv_proj(hidden_states)[0] + query_states, key_states, value_states = qkv.split([self.q_size, self.k_size, self.v_size], + dim=-1) # (total_nnz, 1, hidden_size) + + if self.megatron_config.sequence_parallel: + sequence_parallel_pad = total_nnz - cu_seqlens[-1] + total_nnz = cu_seqlens[-1] # total_nnz before sp padding + query_states = query_states[:total_nnz] + key_states = key_states[:total_nnz] + value_states = value_states[:total_nnz] + + # Flash attention requires the input to have the shape + # batch_size x seq_length x head_dime x hidden_dim + # therefore we just need to keep the original shape + query_states = query_states.view(total_nnz, self.num_heads_per_tp, self.head_dim) + key_states = key_states.view(total_nnz, self.num_key_value_heads_per_tp, self.head_dim) + value_states = value_states.view(total_nnz, self.num_key_value_heads_per_tp, self.head_dim) + + cos, sin = self.rotary_emb(value_states, seq_len=sequence_length) + cos, sin = cos[:, :cos.shape[1] // 2], sin[:, :sin.shape[1] // 2] # flash attn only needs half + query_states, key_states = apply_rotary_pos_emb_rmpad_flash(query_states, + key_states, + cos, + sin, + cu_seqlens=cu_seqlens, + max_seqlen=max_seqlen_in_batch) + # query_states, key_states = apply_rotary_pos_emb_rmpad(query_states, key_states, cos, sin, position_ids, indices, + + # TODO: llama does not have dropout in the config?? + # It is recommended to use dropout with FA according to the docs + # when training. + dropout_rate = 0.0 # if not self.training else self.attn_dropout + + # In PEFT, usually we cast the layer norms in float32 for training stability reasons + # therefore the input hidden states gets silently casted in float32. Hence, we need + # cast them back in float16 just to be sure everything works as expected. + # This might slowdown training & inference so it is recommended to not cast the LayerNorms + # in fp32. (LlamaRMSNorm handles it correctly) + input_dtype = query_states.dtype + if input_dtype == torch.float32: + query_states = query_states.to(torch.float16) + key_states = key_states.to(torch.float16) + value_states = value_states.to(torch.float16) + + attn_output_unpad = flash_attn_varlen_func( + query_states, + key_states, + value_states, + cu_seqlens_q=cu_seqlens, + cu_seqlens_k=cu_seqlens, + max_seqlen_q=max_seqlen_in_batch, + max_seqlen_k=max_seqlen_in_batch, + dropout_p=dropout_rate, + softmax_scale=None, + causal=True, + ) + + attn_output_unpad = attn_output_unpad.to(input_dtype) + attn_output_unpad = attn_output_unpad.reshape(total_nnz, 1, self.hidden_size_per_tp).contiguous() + + # sequence parallel reduce_scatter is performed inside RowColumnParallel if enabled + # Here we need to repad + if self.megatron_config.sequence_parallel: + attn_output_unpad = F.pad(attn_output_unpad, pad=(0, 0, 0, 0, 0, sequence_parallel_pad)) + + attn_output_unpad = self.o_proj(attn_output_unpad)[0] + return attn_output_unpad diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_decoder.py b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_decoder.py new file mode 100644 index 0000000000000000000000000000000000000000..93050a37fefb35d8377a1593f0ea3a4e23938a27 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_decoder.py @@ -0,0 +1,146 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. +# +# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX +# and OPT implementations in this library. It has been modified from its +# original forms to accommodate minor architectural differences compared +# to GPT-NeoX and OPT used by the Meta AI team that trained the model. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Optional, Tuple + +import torch +from torch import nn +from transformers import LlamaConfig +from megatron.core import ModelParallelConfig + +from .parallel_attention import ParallelLlamaAttention, ParallelLlamaAttentionRmPad +from .parallel_mlp import ParallelLlamaMLP +from .parallel_rmsnorm import ParallelLlamaRMSNorm + + +class ParallelLlamaDecoderLayer(nn.Module): + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig): + super().__init__() + self.hidden_size = config.hidden_size + self.self_attn = ParallelLlamaAttention(config=config, megatron_config=megatron_config) + + self.mlp = ParallelLlamaMLP(config, megatron_config=megatron_config) + self.input_layernorm = ParallelLlamaRMSNorm(config, megatron_config) + self.post_attention_layernorm = ParallelLlamaRMSNorm(config, megatron_config) + + def forward( + self, + hidden_states: torch.Tensor, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]: + """ + Args: + hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` + attention_mask (`torch.FloatTensor`, *optional*): attention mask of size + `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. + output_attentions (`bool`, *optional*): + Whether or not to return the attentions tensors of all attention layers. See `attentions` under + returned tensors for more detail. + use_cache (`bool`, *optional*): + If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding + (see `past_key_values`). + past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states + """ + + residual = hidden_states + + hidden_states = self.input_layernorm(hidden_states) + + # Note: sequence parallel is hidden inside ColumnParallelLinear + # reduce scatter is hidden inside RowParallelLinear + + # Self Attention + hidden_states = self.self_attn( + hidden_states=hidden_states, + attention_mask=attention_mask, + position_ids=position_ids, + ) + + # TODO: add sequence parallel operator reduce_scatter here + + hidden_states = residual + hidden_states + + # Fully Connected + residual = hidden_states + hidden_states = self.post_attention_layernorm(hidden_states) + + # TODO: add sequence parallel operator all_gather here + + hidden_states = self.mlp(hidden_states) + + # TODO: add sequence parallel operator reduce_scatter here + + hidden_states = residual + hidden_states + + outputs = hidden_states + + return outputs + + +class ParallelLlamaDecoderLayerRmPad(nn.Module): + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig): + super().__init__() + self.config = config + self.megatron_config = megatron_config + self.hidden_size = config.hidden_size + self.self_attn = ParallelLlamaAttentionRmPad(config=config, megatron_config=megatron_config) + + self.mlp = ParallelLlamaMLP(config, megatron_config=megatron_config) + self.input_layernorm = ParallelLlamaRMSNorm(config, megatron_config) + self.post_attention_layernorm = ParallelLlamaRMSNorm(config, megatron_config) + + def forward( + self, + hidden_states: torch.Tensor, + position_ids: Optional[torch.LongTensor] = None, + sequence_length: int = None, + indices: torch.Tensor = None, + cu_seqlens: int = None, + max_seqlen_in_batch: int = None + ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]: + residual = hidden_states # (total_nnz // sp, 1, hidden_size) + + hidden_states = self.input_layernorm(hidden_states) + + # Self Attention + # (total_nnz // sp, 1, hidden_size) -> all-gather (total_nnz, 1, hidden_size) + # -> col + row -> reduce-scatter -> (total_nnz // sp, 1, hidden_size) + hidden_states = self.self_attn(hidden_states=hidden_states, + position_ids=position_ids, + sequence_length=sequence_length, + indices=indices, + cu_seqlens=cu_seqlens, + max_seqlen_in_batch=max_seqlen_in_batch) + + hidden_states = residual + hidden_states + + # Fully Connected + # shape changes same as attn + residual = hidden_states + hidden_states = self.post_attention_layernorm(hidden_states) + hidden_states = self.mlp(hidden_states) + hidden_states = residual + hidden_states + + outputs = hidden_states + + return outputs diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_linear.py b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_linear.py new file mode 100644 index 0000000000000000000000000000000000000000..bfe5cf4e65e4bdd02ebc64ed8f85943b2f6f3a5f --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_linear.py @@ -0,0 +1,74 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/linear.py + +from typing import Optional, Tuple + +from megatron.core import tensor_parallel + + +class QKVParallelLinear(tensor_parallel.ColumnParallelLinear): + + def __init__(self, + input_size, + num_heads, + num_key_value_heads, + head_dim, + *, + bias=True, + gather_output=True, + skip_bias_add=False, + **kwargs): + # Keep input parameters, and already restrict the head numbers + self.input_size = input_size + self.q_output_size = num_heads * head_dim + self.kv_output_size = num_key_value_heads * head_dim + self.head_dim = head_dim + self.gather_output = gather_output + self.skip_bias_add = skip_bias_add + + input_size = self.input_size + output_size = (num_heads + 2 * num_key_value_heads) * self.head_dim + + super().__init__(input_size=input_size, + output_size=output_size, + bias=bias, + gather_output=gather_output, + skip_bias_add=skip_bias_add, + **kwargs) + + +class MergedColumnParallelLinear(tensor_parallel.ColumnParallelLinear): + + def __init__(self, + input_size, + gate_ouput_size, + up_output_size, + *, + bias=True, + gather_output=True, + skip_bias_add=False, + **kwargs): + # Keep input parameters, and already restrict the head numbers + self.input_size = input_size + self.output_size = gate_ouput_size + up_output_size + self.gather_output = gather_output + self.skip_bias_add = skip_bias_add + + super().__init__(input_size=self.input_size, + output_size=self.output_size, + bias=bias, + gather_output=gather_output, + skip_bias_add=skip_bias_add, + **kwargs) diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_mlp.py b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_mlp.py new file mode 100644 index 0000000000000000000000000000000000000000..21ad9b16a642655dd593ce4d1e5fafb31d81c435 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_mlp.py @@ -0,0 +1,74 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. +# +# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX +# and OPT implementations in this library. It has been modified from its +# original forms to accommodate minor architectural differences compared +# to GPT-NeoX and OPT used by the Meta AI team that trained the model. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from megatron.core import parallel_state as mpu +from megatron.core import tensor_parallel +from megatron.core import ModelParallelConfig +from torch import nn +from transformers.activations import ACT2FN +from verl.models.llama.megatron.layers.parallel_linear import MergedColumnParallelLinear + +from verl.utils.megatron import tensor_parallel as tp_utils + + +class ParallelLlamaMLP(nn.Module): + + def __init__(self, config, megatron_config: ModelParallelConfig = None) -> None: + super().__init__() + self.config = config + self.hidden_size = config.hidden_size + self.intermediate_size = config.intermediate_size + # The weight is only [hidden_size, intermediate_size // model_parallel_world_size] + + column_kwargs = tp_utils.get_default_kwargs_for_column_parallel_linear() + row_kwargs = tp_utils.get_default_kwargs_for_row_parallel_linear() + + if megatron_config is not None: + assert column_kwargs.get('config', False), 'must have ModelParallelConfig' + assert row_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(row_kwargs, megatron_config) + tp_utils.update_kwargs_with_config(column_kwargs, megatron_config) + + tp_size = mpu.get_tensor_model_parallel_world_size() + + self.gate_up_proj = MergedColumnParallelLinear( + input_size=self.hidden_size, + gate_ouput_size=self.intermediate_size, + up_output_size=self.intermediate_size, + bias=False, + gather_output=False, + skip_bias_add=False, + **column_kwargs, + ) + self.gate_size = self.intermediate_size // tp_size + + self.down_proj = tensor_parallel.RowParallelLinear(input_size=self.intermediate_size, + output_size=self.hidden_size, + bias=False, + input_is_parallel=True, + skip_bias_add=False, + **row_kwargs) + + self.act_fn = ACT2FN[config.hidden_act] + + def forward(self, x): + gate_up = self.gate_up_proj(x)[0] + gate, up = gate_up.split(self.gate_size, dim=-1) + return self.down_proj(self.act_fn(gate) * up)[0] diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_rmsnorm.py b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_rmsnorm.py new file mode 100644 index 0000000000000000000000000000000000000000..7027036bf48d47a7f983226e9308336f85ad0461 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/layers/parallel_rmsnorm.py @@ -0,0 +1,46 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import numbers +import torch +from megatron.core import ModelParallelConfig +from torch import nn +from transformers import LlamaConfig + +from apex.normalization.fused_layer_norm import fused_rms_norm_affine +from verl.utils.megatron import sequence_parallel as sp_utils + + +class ParallelLlamaRMSNorm(nn.Module): + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig): + """ + LlamaRMSNorm is equivalent to T5LayerNorm + """ + super().__init__() + if isinstance(config.hidden_size, numbers.Integral): + normalized_shape = (config.hidden_size,) + self.normalized_shape = torch.Size(normalized_shape) + self.weight = nn.Parameter(torch.ones(self.normalized_shape)) + self.variance_epsilon = config.rms_norm_eps + + if megatron_config.sequence_parallel: + sp_utils.mark_parameter_as_sequence_parallel(self.weight) + + def forward(self, hidden_states): + return fused_rms_norm_affine(input=hidden_states, + weight=self.weight, + normalized_shape=self.normalized_shape, + eps=self.variance_epsilon, + memory_efficient=True) \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/models/llama/megatron/modeling_llama_megatron.py b/frameworks/agentgym_rl/verl/models/llama/megatron/modeling_llama_megatron.py new file mode 100644 index 0000000000000000000000000000000000000000..c693f33c5872e341368aad4ee4b0f2b99ed5f5cd --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/llama/megatron/modeling_llama_megatron.py @@ -0,0 +1,656 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. +# +# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX +# and OPT implementations in this library. It has been modified from its +# original forms to accommodate minor architectural differences compared +# to GPT-NeoX and OPT used by the Meta AI team that trained the model. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""PyTorch LLaMA model with Megatron-style acceleration.""" + +from typing import Optional, Tuple, Union + +import torch +import torch.utils.checkpoint +from megatron.core import tensor_parallel +from megatron.core import ModelParallelConfig +from torch import nn +from transformers.modeling_outputs import BaseModelOutputWithPast +from transformers.models.llama.configuration_llama import LlamaConfig +from transformers.models.llama.modeling_llama import CausalLMOutputWithPast + +from verl.utils.megatron import sequence_parallel as sp_utils +from verl.utils.megatron import tensor_parallel as tp_utils +from .layers import ParallelLlamaDecoderLayer, ParallelLlamaRMSNorm, ParallelLlamaDecoderLayerRmPad +""" +TODO: +1. Add weight initialization. Here we need to be careful on TP weight init. +2. Add sequence parallel +3. Load checkpoint from meta LLama pretrained checkpoint +""" + + +# Copied from transformers.models.bart.modeling_bart._make_causal_mask +def _make_causal_mask(input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device): + """ + Make causal mask used for bi-directional self-attention. + """ + bsz, tgt_len = input_ids_shape + mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device) + mask_cond = torch.arange(mask.size(-1), device=device) + mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0) + mask = mask.to(dtype) + return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len) + + +# Copied from transformers.models.bart.modeling_bart._expand_mask +def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None): + """ + Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. + """ + bsz, src_len = mask.size() + tgt_len = tgt_len if tgt_len is not None else src_len + + expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype) + + inverted_mask = 1.0 - expanded_mask + + return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min) + + +class ParallelLlamaModel(nn.Module): + """ + Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] + + Args: + config: LlamaConfig + """ + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig): + super().__init__() + self.padding_idx = config.pad_token_id + self.vocab_size = config.vocab_size + embedding_kwargs = tp_utils.get_default_kwargs_for_parallel_embedding() + if megatron_config is not None: + assert embedding_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(embedding_kwargs, self.megatron_config) + self.embed_tokens = tensor_parallel.VocabParallelEmbedding(num_embeddings=config.vocab_size, + embedding_dim=config.hidden_size, + **embedding_kwargs) + + self.layers = nn.ModuleList( + [ParallelLlamaDecoderLayer(config, megatron_config) for _ in range(config.num_hidden_layers)]) + self.norm = ParallelLlamaRMSNorm(config, megatron_config) + + # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask + def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds): + # create causal mask + # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] + combined_attention_mask = None + if input_shape[-1] > 1: + combined_attention_mask = _make_causal_mask( + input_shape, + inputs_embeds.dtype, + device=inputs_embeds.device, + ) + + if attention_mask is not None: + # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] + expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, + tgt_len=input_shape[-1]).to(inputs_embeds.device) + combined_attention_mask = (expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + + combined_attention_mask) + + return combined_attention_mask + + def forward( + self, + input_ids: torch.LongTensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + ) -> Union[Tuple, BaseModelOutputWithPast]: + """ + + Args: + input_ids: input ids. shape (batch_size, seq_length) + attention_mask: attention_mask. shape (batch_size, seq_length) + position_ids: position ids. shape (batch_size, seq_length) + + Returns: + + """ + batch_size, seq_length = input_ids.shape + inputs_embeds = self.embed_tokens(input_ids) + # embed positions + + attention_mask = self._prepare_decoder_attention_mask(attention_mask, (batch_size, seq_length), inputs_embeds) + + hidden_states = inputs_embeds + + for idx, decoder_layer in enumerate(self.layers): + layer_outputs = decoder_layer( + hidden_states, + attention_mask=attention_mask, + position_ids=position_ids, + ) + + hidden_states = layer_outputs + + hidden_states = self.norm(hidden_states) + + return hidden_states + + +class ParallelLlamaForCausalLM(nn.Module): + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig): + super().__init__() + self.model = ParallelLlamaModel(config, megatron_config=megatron_config) + self.vocab_size = config.vocab_size + + column_kwargs = tp_utils.get_default_kwargs_for_column_parallel_linear() + if megatron_config is not None: + assert column_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(column_kwargs, self.megatron_config) + + self.lm_head = tensor_parallel.ColumnParallelLinear(input_size=config.hidden_size, + output_size=config.vocab_size, + bias=False, + gather_output=False, + skip_bias_add=False, + **column_kwargs) + + def forward( + self, + input_ids: torch.LongTensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + ) -> Union[Tuple, CausalLMOutputWithPast]: + r""" + Args: + labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): + Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., + config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored + (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. + + Returns: + ```""" + + # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn) + outputs = self.model( + input_ids=input_ids, + attention_mask=attention_mask, + position_ids=position_ids, + ) + + hidden_states = outputs + logits = self.lm_head(hidden_states)[0] + + logits = tensor_parallel.gather_from_tensor_model_parallel_region(logits) + + logits = logits.float() + return CausalLMOutputWithPast( + loss=None, + logits=logits, + past_key_values=None, + hidden_states=None, + attentions=None, + ) + + +from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input # noqa + + +class ParallelLlamaModelRmPad(nn.Module): + """ + Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] + + Args: + config: LlamaConfig + """ + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig): + super().__init__() + self.padding_idx = config.pad_token_id + self.vocab_size = config.vocab_size + embedding_kwargs = tp_utils.get_default_kwargs_for_parallel_embedding() + self.megatron_config = megatron_config + if megatron_config is not None: + assert embedding_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(embedding_kwargs, self.megatron_config) + self.embed_tokens = tensor_parallel.VocabParallelEmbedding(num_embeddings=config.vocab_size, + embedding_dim=config.hidden_size, + **embedding_kwargs) + + self.layers = nn.ModuleList( + [ParallelLlamaDecoderLayerRmPad(config, megatron_config) for _ in range(config.num_hidden_layers)]) + self.norm = ParallelLlamaRMSNorm(config, megatron_config) + + def forward(self, + input_ids: torch.Tensor, + position_ids: Optional[torch.LongTensor] = None, + sequence_length: int = None, + indices: torch.Tensor = None, + cu_seqlens: int = None, + max_seqlen_in_batch: int = None) -> Union[Tuple, BaseModelOutputWithPast]: + """ + + Args: + input_ids: input ids. shape (1, totol_nnz) + position_ids: position ids. shape (batch_size, seq_length) + + Returns: + + """ + inputs_embeds = self.embed_tokens(input_ids) # (1, total_nnz) -> (1, total_nnz, hidden_size) + + # (1, total_nnz, hidden_size) -> (total_nnz, 1, hidden_size) -> (total_nnz // sp, 1, hidden_size) + inputs_embeds = inputs_embeds.transpose(0, 1) + if self.megatron_config.sequence_parallel: + inputs_embeds = tensor_parallel.scatter_to_sequence_parallel_region(inputs_embeds) + + hidden_states = inputs_embeds + for idx, decoder_layer in enumerate(self.layers): + layer_outputs = decoder_layer(hidden_states, + position_ids=position_ids, + sequence_length=sequence_length, + indices=indices, + cu_seqlens=cu_seqlens, + max_seqlen_in_batch=max_seqlen_in_batch) + + hidden_states = layer_outputs + + hidden_states = self.norm(hidden_states) + + return hidden_states + + +class ParallelLlamaForCausalLMRmPad(nn.Module): + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig): + super().__init__() + self.config = config + self.megatron_config = megatron_config + self.model = ParallelLlamaModelRmPad(config, megatron_config=megatron_config) + self.vocab_size = config.vocab_size + self._init_head() + + def _init_head(self): + column_kwargs = tp_utils.get_default_kwargs_for_column_parallel_linear() + if self.megatron_config is not None: + assert column_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(column_kwargs, self.megatron_config) + self.lm_head = tensor_parallel.ColumnParallelLinear(input_size=self.config.hidden_size, + output_size=self.config.vocab_size, + bias=False, + gather_output=False, + skip_bias_add=False, + **column_kwargs) + + def _forward_head(self, hidden_states): + # all_gather from sequence parallel region is performed inside lm_head + logits = self.lm_head(hidden_states)[0] + logits = logits.float() # (total_nnz_padded, 1, vocab_size // tp) + logits = tensor_parallel.gather_from_tensor_model_parallel_region(logits) # (total_nnz_padded, 1, vocab_size) + return logits + + def forward( + self, + input_ids: torch.LongTensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + ) -> Union[Tuple, CausalLMOutputWithPast]: + r""" + Args: + labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): + Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., + config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored + (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. + + Returns: + ```""" + batch_size, sequence_length = input_ids.shape + + # remove padding here + input_ids, indices, cu_seqlens, max_seqlen_in_batch, *_ = unpad_input(input_ids.unsqueeze(dim=-1), + attention_mask) # (total_nnz, 1) + + # pad input_ids to multiple of tp for all tp ranks + # TODO: for better performance, the sp padding should be removed at each layer. Not sure the performance gap + if self.megatron_config.sequence_parallel: + input_ids = sp_utils.pad_to_sequence_parallel(input_ids) + + input_ids = input_ids.transpose(0, 1) # (1, total_nnz+pad) + + outputs = self.model(input_ids=input_ids, + position_ids=position_ids, + sequence_length=sequence_length, + indices=indices, + cu_seqlens=cu_seqlens, + max_seqlen_in_batch=max_seqlen_in_batch) + + hidden_states = outputs + + logits = self._forward_head(hidden_states) + + # remove padding from sequence parallel + if self.megatron_config.sequence_parallel: + totol_nnz = cu_seqlens[-1] + logits = logits[:totol_nnz] # (total_nnz_padded) + + logits = torch.squeeze(logits, dim=1) # remove the artificial batch dimension + # add removed padding back + logits = pad_input(logits, indices, batch_size, + seqlen=sequence_length) # (batch_size, sequence_length, vocab_size) + + return CausalLMOutputWithPast( + loss=None, + logits=logits, + past_key_values=None, + hidden_states=None, + attentions=None, + ) + + +class ParallelLlamaForValueRmPad(ParallelLlamaForCausalLMRmPad): + + def _init_head(self): + column_kwargs = tp_utils.get_default_kwargs_for_column_parallel_linear() + if self.megatron_config is not None: + assert column_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(column_kwargs, self.megatron_config) + self.lm_head = nn.Linear(in_features=self.config.hidden_size, out_features=1, bias=False) + # lm_head is effectively the same as sequence parallel + sp_utils.mark_parameter_as_sequence_parallel(self.lm_head.weight) + + def _forward_head(self, hidden_states): + logits = self.lm_head(hidden_states) # (total_nnz_padded // tp, 1, 1) + logits = logits.float() + if self.megatron_config.sequence_parallel: + logits = tensor_parallel.gather_from_sequence_parallel_region(logits, tensor_parallel_output_grad=False) + return logits + + def forward( + self, + input_ids: torch.LongTensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + ) -> Union[Tuple, CausalLMOutputWithPast]: + output = super().forward(input_ids, attention_mask, position_ids) + output.logits = torch.squeeze(output.logits, dim=-1) + return output + + +""" +Support pipeline parallelism +""" + + +class ParallelLlamaModelRmPadPP(nn.Module): + """ + Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] + This model definition supports pipeline parallelism. To support pp and vpp, + - This model only contains layer in this pp stage and vpp chunk + - When calling get_model in Megatron, this rank will instantiate all the vpp chunks in this pp. + Args: + config: LlamaConfig + """ + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig, pre_process, post_process): + super().__init__() + self.padding_idx = config.pad_token_id + self.vocab_size = config.vocab_size + self.pre_process = pre_process + self.post_process = post_process + self.megatron_config = megatron_config + embedding_kwargs = tp_utils.get_default_kwargs_for_parallel_embedding() + if megatron_config is not None: + assert embedding_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(embedding_kwargs, self.megatron_config) + if pre_process: + self.embed_tokens = tensor_parallel.VocabParallelEmbedding(num_embeddings=config.vocab_size, + embedding_dim=config.hidden_size, + **embedding_kwargs) + else: + self.embed_tokens = None + + # pp_rank = megatron_config.pipeline_model_parallel_rank + pp_size = megatron_config.pipeline_model_parallel_size + self.num_layer_per_pp = config.num_hidden_layers // pp_size + vpp_size = megatron_config.virtual_pipeline_model_parallel_size + + if vpp_size is not None: + self.num_layer_vpp_chunk = self.num_layer_per_pp // vpp_size + self.num_layer_this_model = self.num_layer_vpp_chunk + # vpp_rank = megatron_config.virtual_pipeline_model_parallel_rank + # self.offset = vpp_rank * ( + # config.num_hidden_layers // megatron_config.virtual_pipeline_model_parallel_size) + \ + # (megatron_config.pipeline_model_parallel_rank * self.num_layer_vpp_chunk) + else: + self.num_layer_this_model = self.num_layer_per_pp + # self.offset = pp_rank * self.num_layer_per_pp + + layers = [] + for i in range(self.num_layer_this_model): + layer = ParallelLlamaDecoderLayerRmPad(config, megatron_config) + # setattr(layer, 'hidden_layer_index', self.offset + i) + layers.append(layer) + + self.layers = nn.ModuleList(layers) + + if post_process: + self.norm = ParallelLlamaRMSNorm(config, megatron_config) + else: + self.norm = None + + def set_input_tensor(self, input_tensor): + """Set input tensor to be used instead of forward()'s input. + + When doing pipeline parallelism the input from the previous + stage comes from communication, not from the input, so the + model's forward_step_func won't have it. This function is thus + used by internal code to bypass the input provided by the + forward_step_func""" + self.input_tensor = input_tensor + + def forward(self, + input_ids: torch.Tensor, + position_ids: Optional[torch.LongTensor] = None, + sequence_length: int = None, + indices: torch.Tensor = None, + cu_seqlens: int = None, + max_seqlen_in_batch: int = None) -> Union[Tuple, BaseModelOutputWithPast]: + """ + + Args: + input_ids: input ids. shape (1, totol_nnz) + position_ids: position ids. shape (batch_size, seq_length) + + Returns: + + """ + if self.pre_process: + inputs_embeds = self.embed_tokens(input_ids) # (1, total_nnz) -> (1, total_nnz, hidden_size) + + # vocab parallel embedding will not do sequence parallel reduce-scatter in open source megatron + # so need to deal with it by handle here: + # (1, total_nnz, hidden_size) -> (total_nnz, 1, hidden_size) -> (total_nnz // sp, 1, hidden_size) + inputs_embeds = inputs_embeds.transpose(0, 1) + if self.megatron_config.sequence_parallel: + inputs_embeds = tensor_parallel.scatter_to_sequence_parallel_region(inputs_embeds) + + hidden_states = inputs_embeds + else: + # self.hidden_states should be passed by Megatron + hidden_states = self.input_tensor + + for idx, decoder_layer in enumerate(self.layers): + layer_outputs = decoder_layer(hidden_states, + position_ids=position_ids, + sequence_length=sequence_length, + indices=indices, + cu_seqlens=cu_seqlens, + max_seqlen_in_batch=max_seqlen_in_batch) + + hidden_states = layer_outputs + + if self.post_process: + hidden_states = self.norm(hidden_states) + + return hidden_states + + +class ParallelLlamaForCausalLMRmPadPP(nn.Module): + + def __init__(self, config: LlamaConfig, megatron_config: ModelParallelConfig, pre_process, post_process): + super().__init__() + self.config = config + self.megatron_config = megatron_config + self.model = ParallelLlamaModelRmPadPP(config, + megatron_config=megatron_config, + pre_process=pre_process, + post_process=post_process) + self.share_embeddings_and_output_weights = None # workaround, megatron requires this attr + self.vocab_size = config.vocab_size + self.pre_process = pre_process + self.post_process = post_process + if post_process: + self._init_head() + + def set_input_tensor(self, input_tensor): + """Set input tensor to be used instead of forward()'s input. + + When doing pipeline parallelism the input from the previous + stage comes from communication, not from the input, so the + model's forward_step_func won't have it. This function is thus + used by internal code to bypass the input provided by the + forward_step_func""" + assert len(input_tensor) == 1 + self.model.set_input_tensor(input_tensor[0]) + + def _init_head(self): + column_kwargs = tp_utils.get_default_kwargs_for_column_parallel_linear() + if self.megatron_config is not None: + assert column_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(column_kwargs, self.megatron_config) + self.lm_head = tensor_parallel.ColumnParallelLinear(input_size=self.config.hidden_size, + output_size=self.config.vocab_size, + bias=False, + gather_output=False, + skip_bias_add=False, + **column_kwargs) + + def _forward_head(self, hidden_states): + # all_gather from sequence parallel region is performed inside lm_head + # logits shape before forward_head hidden_states.shape: [4, 32, 4096] + logits = self.lm_head(hidden_states)[0] + # logits shape after forward_head logits.shape: [8, 32, 8] + logits = logits.float() # (total_nnz_padded, 1, vocab_size // tp) + return logits + + def forward( + self, + # original input + *, + input_ids: torch.LongTensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + ) -> Union[Tuple, CausalLMOutputWithPast]: + r""" + Args: + labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): + Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., + config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored + (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. + + Returns: + ```""" + + # Note that input_ids, attention_mask and position_ids should be passed to every pp layer. + # In the first pp, input_ids will be used, in other pp layers hidden_states will be used inside self.model + batch_size, sequence_length = input_ids.shape + # remove padding here + input_ids_rmpad, indices, cu_seqlens, max_seqlen_in_batch, *_ = unpad_input(input_ids.unsqueeze(dim=-1), + attention_mask) # (total_nnz, 1) + + # pad input_ids to multiple of tp for all tp ranks + # TODO: for better performance, the sp padding should be removed at each layer. Not sure the performance gap + if self.megatron_config.sequence_parallel: + input_ids_rmpad = sp_utils.pad_to_sequence_parallel(input_ids_rmpad) + + input_ids_rmpad = input_ids_rmpad.transpose(0, 1) # (1, total_nnz+pad) + + outputs = self.model(input_ids=input_ids_rmpad, + position_ids=position_ids, + sequence_length=sequence_length, + indices=indices, + cu_seqlens=cu_seqlens, + max_seqlen_in_batch=max_seqlen_in_batch) + + if self.post_process: + hidden_states = outputs + # print(f'hidden_states.shape = {hidden_states.shape}') # torch.Size([4, 32, 4096]) + logits = self._forward_head(hidden_states) + logits = torch.squeeze(logits, dim=1) # remove the artificial batch dimension # torch.Size([8, 32, 16]) + + # remove padding from sequence parallel + if self.megatron_config.sequence_parallel: + totol_nnz = cu_seqlens[-1] + logits = logits[:totol_nnz] # (total_nnz_padded) + # add removed padding back. If input is already rmpad, we let the caller pad_input + logits = pad_input(logits, indices, batch_size, + seqlen=sequence_length) # (batch_size, sequence_length, vocab_size) + + return CausalLMOutputWithPast( + loss=None, + logits=logits, + past_key_values=None, + hidden_states=None, + attentions=None, + ) + else: + return outputs + + +class ParallelLlamaForValueRmPadPP(ParallelLlamaForCausalLMRmPadPP): + + def _init_head(self): + column_kwargs = tp_utils.get_default_kwargs_for_column_parallel_linear() + if self.megatron_config is not None: + assert column_kwargs.get('config', False), 'must have ModelParallelConfig' + tp_utils.update_kwargs_with_config(column_kwargs, self.megatron_config) + self.lm_head = nn.Linear(in_features=self.config.hidden_size, out_features=1, bias=False) + # lm_head is effectively the same as sequence parallel + sp_utils.mark_parameter_as_sequence_parallel(self.lm_head.weight) + + def _forward_head(self, hidden_states): + logits = self.lm_head(hidden_states) # (total_nnz_padded // tp, 1, 1) + logits = logits.float() + if self.megatron_config.sequence_parallel: + logits = tensor_parallel.gather_from_sequence_parallel_region(logits, tensor_parallel_output_grad=False) + return logits + + def forward( + self, + *, + input_ids: torch.LongTensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + ) -> Union[Tuple, CausalLMOutputWithPast]: + output = super().forward(input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids) + if self.post_process: + output.logits = torch.squeeze(output.logits, dim=-1) + return output + else: + return output diff --git a/frameworks/agentgym_rl/verl/models/registry.py b/frameworks/agentgym_rl/verl/models/registry.py new file mode 100644 index 0000000000000000000000000000000000000000..55ddbd4493d3287511fcaca1c215a22d8930b1a1 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/registry.py @@ -0,0 +1,66 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import importlib +from typing import List, Optional, Type + +import torch.nn as nn + +# Supported models using HF Rmpad +# TODO(sgm): HF may supported more than listed here, we should add more after testing +from transformers import LlamaConfig, MistralConfig, GemmaConfig, Qwen2Config + +_REOVEPAD_MODELS = {'llama': LlamaConfig, 'mistral': MistralConfig, 'gemma': GemmaConfig, 'qwen2': Qwen2Config} + + +def check_model_support_rmpad(model_type: str): + assert isinstance(model_type, str) + if not model_type in _REOVEPAD_MODELS.keys(): + raise ValueError(f"Model architecture {model_type} is not supported for now. " + f"RMPad supported architectures: {_REOVEPAD_MODELS.keys()}." + f"Please set `use_remove_padding=False` in the model config.") + + +# Supported models in Megatron-LM +# Architecture -> (module, class). +_MODELS = { + "LlamaForCausalLM": + ("llama", ("ParallelLlamaForCausalLMRmPadPP", "ParallelLlamaForValueRmPadPP", "ParallelLlamaForCausalLMRmPad")), + "MistralForCausalLM": ("mistral", ("ParallelMistralForCausalLMRmPadPP", "ParallelMistralForValueRmPadPP", + "ParallelMistralForCausalLMRmPad")) +} + + +# return model class +class ModelRegistry: + + @staticmethod + def load_model_cls(model_arch: str, value=False) -> Optional[Type[nn.Module]]: + if model_arch not in _MODELS: + return None + + megatron = "megatron" + + module_name, model_cls_name = _MODELS[model_arch] + if not value: # actor/ref + model_cls_name = model_cls_name[0] + elif value: # critic/rm + model_cls_name = model_cls_name[1] + + module = importlib.import_module(f"verl.models.{module_name}.{megatron}.modeling_{module_name}_megatron") + return getattr(module, model_cls_name, None) + + @staticmethod + def get_supported_archs() -> List[str]: + return list(_MODELS.keys()) diff --git a/frameworks/agentgym_rl/verl/models/transformers/__init__.py b/frameworks/agentgym_rl/verl/models/transformers/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/transformers/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/models/transformers/llama.py b/frameworks/agentgym_rl/verl/models/transformers/llama.py new file mode 100644 index 0000000000000000000000000000000000000000..3527c95829a1b7a989ec60565b157cdfbba75e81 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/transformers/llama.py @@ -0,0 +1,149 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +from typing import Optional, List, Union, Tuple, Callable +import sys +if sys.version_info >= (3, 11): + from typing import Unpack +else: + from typing_extensions import Unpack + +from transformers.models.llama.modeling_llama import apply_rotary_pos_emb, repeat_kv +from transformers.cache_utils import Cache +from transformers.utils import logging +from transformers.modeling_flash_attention_utils import _flash_attention_forward +from verl.utils.ulysses import gather_heads_scatter_seq, gather_seq_scatter_heads, get_ulysses_sequence_parallel_world_size + +logger = logging.get_logger(__name__) + +def llama_flash_attn_forward( + self, + hidden_states: torch.Tensor, + attention_mask: Optional[torch.LongTensor] = None, + position_ids: Optional[torch.LongTensor] = None, + past_key_value: Optional[Cache] = None, + output_attentions: bool = False, + use_cache: bool = False, + cache_position: Optional[torch.LongTensor] = None, + position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None, # will become mandatory in v4.46 + **kwargs, + ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]: + """ + adapt from transformers 4.47.1 + """ + output_attentions = False + + bsz, q_len, _ = hidden_states.size() + + query_states = self.q_proj(hidden_states) + key_states = self.k_proj(hidden_states) + value_states = self.v_proj(hidden_states) + + # Flash attention requires the input to have the shape + # batch_size x seq_length x head_dim x hidden_dim + # therefore we just need to keep the original shape + query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) + key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2) + value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2) + + # trade off: repeat first and then all to all + # key_states = repeat_kv(key_states, self.num_key_value_groups) + # value_states = repeat_kv(value_states, self.num_key_value_groups) + + ########## AlltoAll for Ulysses ########## + ulysses_sp_size = get_ulysses_sequence_parallel_world_size() + + if ulysses_sp_size > 1: + # (bsz, n_head, seq_len/n, head_dim) -> (bsz, n_head/n, seq_len, head_dim) + query_states = gather_seq_scatter_heads(query_states, seq_dim=2, head_dim=1) + key_states = gather_seq_scatter_heads(key_states, seq_dim=2, head_dim=1) + value_states = gather_seq_scatter_heads(value_states, seq_dim=2, head_dim=1) + + full_q_len = query_states.size(2) # full seq length + + if position_embeddings is None: + logger.warning_once( + "The attention layers in this model are transitioning from computing the RoPE embeddings internally " + "through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed " + "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be " + "removed and `position_embeddings` will be mandatory.") + cos, sin = self.rotary_emb(value_states, position_ids) + else: + cos, sin = position_embeddings + query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin) + + if past_key_value is not None: + # sin and cos are specific to RoPE models; cache_position needed for the static cache + cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position} + key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs) + + # TODO: These transpose are quite inefficient but Flash Attention requires the layout [batch_size, sequence_length, num_heads, head_dim]. We would need to refactor the KV cache + # to be able to avoid many of these transpose/reshape/view. + query_states = query_states.transpose(1, 2) + key_states = key_states.transpose(1, 2) + value_states = value_states.transpose(1, 2) + + dropout_rate = self.attention_dropout if self.training else 0.0 + + # In PEFT, usually we cast the layer norms in float32 for training stability reasons + # therefore the input hidden states gets silently casted in float32. Hence, we need + # cast them back in the correct dtype just to be sure everything works as expected. + # This might slowdown training & inference so it is recommended to not cast the LayerNorms + # in fp32. (LlamaRMSNorm handles it correctly) + + input_dtype = query_states.dtype + if input_dtype == torch.float32: + if torch.is_autocast_enabled(): + target_dtype = torch.get_autocast_gpu_dtype() + # Handle the case where the model is quantized + elif hasattr(self.config, "_pre_quantization_dtype"): + target_dtype = self.config._pre_quantization_dtype + else: + target_dtype = self.q_proj.weight.dtype + + logger.warning_once( + f"The input hidden states seems to be silently casted in float32, this might be related to" + f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in" + f" {target_dtype}.") + + query_states = query_states.to(target_dtype) + key_states = key_states.to(target_dtype) + value_states = value_states.to(target_dtype) + + attn_output = _flash_attention_forward( + query_states, + key_states, + value_states, + attention_mask, + full_q_len, + position_ids=position_ids, + dropout=dropout_rate, + sliding_window=getattr(self, "sliding_window", None), + use_top_left_mask=self._flash_attn_uses_top_left_mask, + is_causal=self.is_causal, + **kwargs, + ) + + attn_output = attn_output.reshape(bsz, full_q_len, -1, self.head_dim).contiguous() + ########## AlltoAll for Ulysses ########## + if ulysses_sp_size > 1: + attn_output = gather_heads_scatter_seq(attn_output, seq_dim=1, head_dim=2) + attn_output = attn_output.reshape(bsz, q_len, -1).contiguous() + attn_output = self.o_proj(attn_output) + + if not output_attentions: + attn_weights = None + + return attn_output, attn_weights, past_key_value diff --git a/frameworks/agentgym_rl/verl/models/transformers/monkey_patch.py b/frameworks/agentgym_rl/verl/models/transformers/monkey_patch.py new file mode 100644 index 0000000000000000000000000000000000000000..a11148b4d0ed565d5a9a5b43babe47789a9ce726 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/transformers/monkey_patch.py @@ -0,0 +1,74 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Apply monkey-patch function to models +""" + +#### Open Source Models +#### transformers version < 4.48 + + +def apply_monkey_patch_to_llama(): + from transformers.models.llama.modeling_llama import LlamaFlashAttention2 + from verl.models.transformers.llama import llama_flash_attn_forward + LlamaFlashAttention2.forward = llama_flash_attn_forward + + +def apply_monkey_patch_to_qwen2(): + from transformers.models.qwen2.modeling_qwen2 import Qwen2FlashAttention2 + from verl.models.transformers.qwen2 import qwen2_flash_attn_forward + Qwen2FlashAttention2.forward = qwen2_flash_attn_forward + + +_PATCH_NAME_TO_FUNC = { + 'llama': apply_monkey_patch_to_llama, + 'qwen2': apply_monkey_patch_to_qwen2, +} + +from transformers import PretrainedConfig + + +def apply_monkey_patch(config: PretrainedConfig, verbose=True): + if not is_transformers_version_in_range("4.45.0", "4.47.1"): + raise AssertionError("The installed `transformers` version doesn't support ulysses patch. " + "Please install a version between 4.45.0 and 4.47.1 to use this ulysses feature.") + success_apply_monkey_patch = False + if config.model_type in _PATCH_NAME_TO_FUNC: + _PATCH_NAME_TO_FUNC[config.model_type]() + success_apply_monkey_patch = True + + if success_apply_monkey_patch and verbose: + print(f'Applying monkey patch to model {config.model_type}') + elif not success_apply_monkey_patch: + raise NotImplementedError(f'Ulysses for model {config.model_type} is not implemented, \ + please set `ulysses_sequence_parallel_size=1`') + + return success_apply_monkey_patch + + +from functools import lru_cache +from packaging import version +import importlib.metadata + + +@lru_cache() +def is_transformers_version_in_range(min_version: str, max_version: str) -> bool: + try: + # Get the installed version of the transformers library + transformers_version = importlib.metadata.version("transformers") + except importlib.metadata.PackageNotFoundError: + raise ModuleNotFoundError("The `transformers` package is not installed.") + + # Check if the version is within the specified range + return version.parse(min_version) <= version.parse(transformers_version) <= version.parse(max_version) diff --git a/frameworks/agentgym_rl/verl/models/transformers/qwen2.py b/frameworks/agentgym_rl/verl/models/transformers/qwen2.py new file mode 100644 index 0000000000000000000000000000000000000000..7341bfdbec5ef6b146ab6768a4aa868c28a5960e --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/transformers/qwen2.py @@ -0,0 +1,136 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +from typing import Optional, Tuple + +from transformers.models.llama.modeling_llama import apply_rotary_pos_emb, repeat_kv +from transformers.cache_utils import Cache +from transformers.utils import logging +from transformers.modeling_flash_attention_utils import _flash_attention_forward +from verl.utils.ulysses import gather_heads_scatter_seq, gather_seq_scatter_heads, get_ulysses_sequence_parallel_world_size + +logger = logging.get_logger(__name__) + + +def qwen2_flash_attn_forward( + self, + hidden_states: torch.Tensor, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + past_key_value: Optional[Cache] = None, + output_attentions: bool = False, + use_cache: bool = False, + cache_position: Optional[torch.LongTensor] = None, + position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None, # will become mandatory in v4.46 +): + bsz, q_len, _ = hidden_states.size() + + query_states = self.q_proj(hidden_states) + key_states = self.k_proj(hidden_states) + value_states = self.v_proj(hidden_states) + + query_states = query_states.view(bsz, q_len, -1, self.head_dim).transpose(1, 2) + key_states = key_states.view(bsz, q_len, -1, self.head_dim).transpose(1, 2) + value_states = value_states.view(bsz, q_len, -1, self.head_dim).transpose(1, 2) + + ########## AlltoAll for Ulysses ########## + ulysses_sp_size = get_ulysses_sequence_parallel_world_size() + + if ulysses_sp_size > 1: + # (bsz, n_head, seq_len/n, head_dim) -> (bsz, n_head/n, seq_len, head_dim) + query_states = gather_seq_scatter_heads(query_states, seq_dim=2, head_dim=1) + key_states = gather_seq_scatter_heads(key_states, seq_dim=2, head_dim=1) + value_states = gather_seq_scatter_heads(value_states, seq_dim=2, head_dim=1) + + full_q_len = query_states.size(2) # full seq length + + if position_embeddings is None: + logger.warning_once( + "The attention layers in this model are transitioning from computing the RoPE embeddings internally " + "through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed " + "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be " + "removed and `position_embeddings` will be mandatory.") + cos, sin = self.rotary_emb(value_states, position_ids) + else: + cos, sin = position_embeddings + query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin) + + if past_key_value is not None: + cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position} # Specific to RoPE models + key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs) + + # repeat k/v heads if n_kv_heads < n_heads + key_states = repeat_kv(key_states, self.num_key_value_groups) + value_states = repeat_kv(value_states, self.num_key_value_groups) + dropout_rate = 0.0 if not self.training else self.attention_dropout + + # In PEFT, usually we cast the layer norms in float32 for training stability reasons + # therefore the input hidden states gets silently casted in float32. Hence, we need + # cast them back in float16 just to be sure everything works as expected. + input_dtype = query_states.dtype + if input_dtype == torch.float32: + if torch.is_autocast_enabled(): + target_dtype = torch.get_autocast_gpu_dtype() + # Handle the case where the model is quantized + elif hasattr(self.config, "_pre_quantization_dtype"): + target_dtype = self.config._pre_quantization_dtype + else: + target_dtype = self.q_proj.weight.dtype + + logger.warning_once( + f"The input hidden states seems to be silently casted in float32, this might be related to" + f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in" + f" {target_dtype}.") + + query_states = query_states.to(target_dtype) + key_states = key_states.to(target_dtype) + value_states = value_states.to(target_dtype) + + # Reashape to the expected shape for Flash Attention + query_states = query_states.transpose(1, 2) + key_states = key_states.transpose(1, 2) + value_states = value_states.transpose(1, 2) + + if (self.config.use_sliding_window and getattr(self.config, "sliding_window", None) is not None and + self.layer_idx >= self.config.max_window_layers): + sliding_window = self.config.sliding_window + else: + sliding_window = None + + attn_output = _flash_attention_forward( + query_states, + key_states, + value_states, + attention_mask, + full_q_len, + position_ids=position_ids, + dropout=dropout_rate, + sliding_window=sliding_window, + is_causal=self.is_causal, + use_top_left_mask=self._flash_attn_uses_top_left_mask, + ) + + # use full_q_len to reshape + attn_output = attn_output.reshape(bsz, full_q_len, -1, self.head_dim).contiguous() + ########## AlltoAll for Ulysses ########## + if ulysses_sp_size > 1: + attn_output = gather_heads_scatter_seq(attn_output, seq_dim=1, head_dim=2) + attn_output = attn_output.reshape(bsz, q_len, -1).contiguous() + attn_output = self.o_proj(attn_output) + + if not output_attentions: + attn_weights = None + + return attn_output, attn_weights, past_key_value diff --git a/frameworks/agentgym_rl/verl/models/weight_loader_registry.py b/frameworks/agentgym_rl/verl/models/weight_loader_registry.py new file mode 100644 index 0000000000000000000000000000000000000000..17f0c5cae957d6bd665fd0f9dcdc84c1206adfa8 --- /dev/null +++ b/frameworks/agentgym_rl/verl/models/weight_loader_registry.py @@ -0,0 +1,23 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +def get_weight_loader(arch: str): + from verl.models.llama.megatron.checkpoint_utils.llama_loader import load_state_dict_to_megatron_llama + _MODEL_WEIGHT_MEGATRON_LOADER_REGISTRY = {'LlamaForCausalLM': load_state_dict_to_megatron_llama} + + if arch in _MODEL_WEIGHT_MEGATRON_LOADER_REGISTRY: + return _MODEL_WEIGHT_MEGATRON_LOADER_REGISTRY[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {_MODEL_WEIGHT_MEGATRON_LOADER_REGISTRY.keys()}") diff --git a/frameworks/agentgym_rl/verl/protocol.py b/frameworks/agentgym_rl/verl/protocol.py new file mode 100644 index 0000000000000000000000000000000000000000..7f43446547269ae4a05ac490cb291dfbe996dce0 --- /dev/null +++ b/frameworks/agentgym_rl/verl/protocol.py @@ -0,0 +1,647 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Implement base data transfer protocol between any two functions, modules. +We can subclass Protocol to define more detailed batch info with specific keys +""" + +import pickle +import numpy as np +import pandas as pd +import copy +from dataclasses import dataclass, field +from typing import Callable, Dict, List, Union + +import torch +import tensordict +from tensordict import TensorDict +from torch.utils.data import DataLoader, Dataset + +from verl.utils.py_functional import union_two_dict + +__all__ = ['DataProto', 'union_tensor_dict'] + +try: + tensordict.set_lazy_legacy(False).set() +except: + pass + + +def pad_dataproto_to_divisor(data: 'DataProto', size_divisor: int): + """Pad a DataProto to size divisible by size_divisor + + Args: + size_divisor (int): size divisor + + Returns: + data: (DataProto): the padded DataProto + pad_size (int) + """ + assert isinstance(data, DataProto), 'data must be a DataProto' + if len(data) % size_divisor != 0: + pad_size = size_divisor - len(data) % size_divisor + padding_protos = [] + remaining_pad = pad_size + while remaining_pad > 0: + take_size = min(remaining_pad, len(data)) + padding_protos.append(data[:take_size]) + remaining_pad -= take_size + data_padded = DataProto.concat([data] + padding_protos) + else: + pad_size = 0 + data_padded = data + return data_padded, pad_size + + +def unpad_dataproto(data: 'DataProto', pad_size): + if pad_size != 0: + data = data[:-pad_size] + return data + + +def union_tensor_dict(tensor_dict1: TensorDict, tensor_dict2: TensorDict) -> TensorDict: + """Union two tensordicts.""" + assert tensor_dict1.batch_size == tensor_dict2.batch_size, \ + f'Two tensor dict must have identical batch size. Got {tensor_dict1.batch_size} and {tensor_dict2.batch_size}' + for key in tensor_dict2.keys(): + if key not in tensor_dict1.keys(): + tensor_dict1[key] = tensor_dict2[key] + else: + assert tensor_dict1[key].equal(tensor_dict2[key]), \ + f'{key} in tensor_dict1 and tensor_dict2 are not the same object' + + return tensor_dict1 + + +def union_numpy_dict(tensor_dict1: dict[np.ndarray], tensor_dict2: dict[np.ndarray]) -> dict[np.ndarray]: + for key, val in tensor_dict2.items(): + if key in tensor_dict1: + assert isinstance(tensor_dict2[key], np.ndarray) + assert isinstance(tensor_dict1[key], np.ndarray) + # to properly deal with nan and object type + assert pd.DataFrame(tensor_dict2[key]).equals(pd.DataFrame(tensor_dict1[key])), \ + f'{key} in tensor_dict1 and tensor_dict2 are not the same object' + tensor_dict1[key] = val + + return tensor_dict1 + + +def list_of_dict_to_dict_of_list(list_of_dict: list[dict]): + if len(list_of_dict) == 0: + return {} + keys = list_of_dict[0].keys() + output = {key: [] for key in keys} + for data in list_of_dict: + for key, item in data.items(): + assert key in output + output[key].append(item) + return output + + +def fold_batch_dim(data: 'DataProto', new_batch_size): + """ + Fold a batch dim from [bsz, xxx] into [new_bsz, bsz // new_bsz, xxx] + """ + batch_size = data.batch.batch_size[0] + + assert batch_size % new_batch_size == 0 + + tensor: TensorDict = data.batch + non_tensor = data.non_tensor_batch + + tensor = tensor.view(new_batch_size, -1) + tensor.auto_batch_size_(batch_dims=1) + + for key, val in non_tensor.items(): + non_tensor[key] = np.reshape(val, newshape=(new_batch_size, -1, *val.shape[1:])) + + return DataProto(batch=tensor, non_tensor_batch=non_tensor, meta_info=data.meta_info) + + +def unfold_batch_dim(data: 'DataProto', batch_dims=2): + """ + Unfold the first n dims as new batch dim + """ + tensor: TensorDict = data.batch + non_tensor = data.non_tensor_batch + tensor.auto_batch_size_(batch_dims=batch_dims) + tensor = tensor.view(-1) + + batch_size = tensor.batch_size[0] + + non_tensor_new = {} + + for key, val in non_tensor.items(): + non_tensor_new[key] = np.reshape(val, newshape=(batch_size, *val.shape[batch_dims:])) + + return DataProto(batch=tensor, non_tensor_batch=non_tensor_new, meta_info=data.meta_info) + + +def collate_fn(x: list['DataProtoItem']): + batch = [] + non_tensor_batch = [] + for data in x: + batch.append(data.batch) + non_tensor_batch.append(data.non_tensor_batch) + batch = torch.stack(batch).contiguous() + non_tensor_batch = list_of_dict_to_dict_of_list(non_tensor_batch) + for key, val in non_tensor_batch.items(): + non_tensor_batch[key] = np.array(val, dtype=object) + return DataProto(batch=batch, non_tensor_batch=non_tensor_batch) + + +@dataclass +class DataProtoItem: + # TODO(zhangchi.usc1992) add consistency check + batch: TensorDict = None + non_tensor_batch: Dict = field(default_factory=dict) + meta_info: Dict = field(default_factory=dict) + + +@dataclass +class DataProto: + """ + A DataProto is a data structure that aims to provide a standard protocol for data exchange between functions. + It contains a batch (TensorDict) and a meta_info (Dict). The batch is a TensorDict https://pytorch.org/tensordict/. + TensorDict allows you to manipulate a dictionary of Tensors like a single Tensor. Ideally, the tensors with the + same batch size should be put inside batch. + """ + batch: TensorDict = None + non_tensor_batch: Dict = field(default_factory=dict) + meta_info: Dict = field(default_factory=dict) + + def __post_init__(self): + # perform necessary checking + self.check_consistency() + + def __len__(self): + if self.batch is not None: + return self.batch.batch_size[0] + elif self.non_tensor_batch is not None and len(self.non_tensor_batch) > 0: + random_key = list(self.non_tensor_batch.keys())[0] + return self.non_tensor_batch[random_key].shape[0] + else: + return 0 + + def __getitem__(self, item): + tensor_data = self.batch[item] + non_tensor_data = {key: val[item] for key, val in self.non_tensor_batch.items()} + return DataProtoItem(batch=tensor_data, non_tensor_batch=non_tensor_data, meta_info=self.meta_info) + + def __getstate__(self): + import io + buffer = io.BytesIO() + if tensordict.__version__ >= '0.5.0' and self.batch is not None: + self.batch = self.batch.contiguous() + self.batch = self.batch.consolidate() + torch.save(self.batch, buffer) + buffer_bytes = buffer.getvalue() + return buffer_bytes, self.non_tensor_batch, self.meta_info + + def __setstate__(self, data): + import io + batch_deserialized_bytes, non_tensor_batch, meta_info = data + batch_deserialized = io.BytesIO(initial_bytes=batch_deserialized_bytes) + batch = torch.load(batch_deserialized, + weights_only=False, + map_location='cpu' if not torch.cuda.is_available() else None) + self.batch = batch + self.non_tensor_batch = non_tensor_batch + self.meta_info = meta_info + + def save_to_disk(self, filepath): + with open(filepath, 'wb') as f: + pickle.dump(self, f) + + @staticmethod + def load_from_disk(filepath) -> 'DataProto': + with open(filepath, 'rb') as f: + data = pickle.load(f) + return data + + def print_size(self, prefix=""): + size_of_tensordict = 0 + for key, tensor in self.batch.items(): + size_of_tensordict += tensor.element_size() * tensor.numel() + size_of_numpy_array = 0 + for key, numpy_array in self.non_tensor_batch.items(): + size_of_numpy_array += numpy_array.nbytes + + size_of_numpy_array /= 1024**3 + size_of_tensordict /= 1024**3 + + message = f'Size of tensordict: {size_of_tensordict} GB, size of non_tensor_batch: {size_of_numpy_array} GB' + + if prefix: + message = f'{prefix}, ' + message + print(message) + + def check_consistency(self): + """Check the consistency of the DataProto. Mainly for batch and non_tensor_batch + We expose this function as a public one so that user can call themselves directly + """ + if self.batch is not None: + assert len(self.batch.batch_size) == 1, 'only support num_batch_dims=1' + + if self.non_tensor_batch is not None: + for key, val in self.non_tensor_batch.items(): + assert isinstance(val, np.ndarray) + + if self.batch is not None and len(self.non_tensor_batch) != 0: + # TODO: we can actually lift this restriction if needed + assert len(self.batch.batch_size) == 1, 'only support num_batch_dims=1 when non_tensor_batch is not empty.' + + batch_size = self.batch.batch_size[0] + for key, val in self.non_tensor_batch.items(): + assert isinstance( + val, np.ndarray + ) and val.dtype == object, 'data in the non_tensor_batch must be a numpy.array with dtype=object' + assert val.shape[ + 0] == batch_size, f'key {key} length {len(val)} is not equal to batch size {batch_size}' + + @classmethod + def from_single_dict(cls, data: Dict[str, Union[torch.Tensor, np.ndarray]], meta_info=None): + tensors = {} + non_tensors = {} + + for key, val in data.items(): + if isinstance(val, torch.Tensor): + tensors[key] = val + elif isinstance(val, np.ndarray): + non_tensors[key] = val + else: + raise ValueError(f'Unsupported type in data {type(val)}') + + return DataProto.from_dict(tensors=tensors, non_tensors=non_tensors, meta_info=meta_info) + + @classmethod + def from_dict(cls, tensors: Dict[str, torch.Tensor], non_tensors=None, meta_info=None, num_batch_dims=1): + """Create a DataProto from a dict of tensors. This assumes that + 1. All the tensor in tensors have the same dim0 + 2. Only dim0 is the batch dim + """ + assert len(tensors) > 0, 'tensors must not be empty' + assert num_batch_dims > 0, 'num_batch_dims must be greater than zero' + if non_tensors is not None: + assert num_batch_dims == 1, 'only support num_batch_dims=1 when non_tensors is not None.' + + if meta_info is None: + meta_info = {} + if non_tensors is None: + non_tensors = {} + + assert isinstance(non_tensors, dict) + + # get and check batch size + batch_size = None + pivot_key = None + for key, tensor in tensors.items(): + if batch_size is None: + batch_size = tensor.shape[:num_batch_dims] + pivot_key = key + else: + current_batch = tensor.shape[:num_batch_dims] + assert batch_size == current_batch, \ + f'Not all the tensor in tensors have the same batch size with batch_dims={num_batch_dims}. Got {pivot_key} has {batch_size}, {key} has {current_batch}' + + for key, val in non_tensors.items(): + non_tensors[key] = np.array(val, dtype=object) + + tensor_dict = TensorDict(source=tensors, batch_size=batch_size) + return cls(batch=tensor_dict, non_tensor_batch=non_tensors, meta_info=meta_info) + + def to(self, device) -> 'DataProto': + """move the batch to device + + Args: + device (torch.device, str): torch device + + Returns: + DataProto: the current DataProto + + """ + if self.batch is not None: + self.batch = self.batch.to(device) + return self + + def select(self, batch_keys=None, non_tensor_batch_keys=None, meta_info_keys=None, deepcopy=False) -> 'DataProto': + """Select a subset of the DataProto via batch_keys and meta_info_keys + + Args: + batch_keys (list, optional): a list of strings indicating the keys in batch to select + meta_info_keys (list, optional): a list of keys indicating the meta info to select + + Returns: + DataProto: the DataProto with the selected batch_keys and meta_info_keys + """ + # TODO (zhangchi.usc1992) whether to copy + if batch_keys is not None: + batch_keys = tuple(batch_keys) + sub_batch = self.batch.select(*batch_keys) + else: + sub_batch = self.batch + + if non_tensor_batch_keys is not None: + non_tensor_batch = {key: val for key, val in self.non_tensor_batch.items() if key in non_tensor_batch_keys} + else: + non_tensor_batch = self.non_tensor_batch + + if deepcopy: + non_tensor_batch = copy.deepcopy(non_tensor_batch) + + if meta_info_keys is not None: + sub_meta_info = {key: val for key, val in self.meta_info.items() if key in meta_info_keys} + else: + sub_meta_info = self.meta_info + + if deepcopy: + sub_meta_info = copy.deepcopy(sub_meta_info) + + return DataProto(batch=sub_batch, non_tensor_batch=non_tensor_batch, meta_info=sub_meta_info) + + def pop(self, batch_keys=None, non_tensor_batch_keys=None, meta_info_keys=None) -> 'DataProto': + """Pop a subset of the DataProto via `batch_keys` and `meta_info_keys` + + Args: + batch_keys (list, optional): a list of strings indicating the keys in batch to pop + meta_info_keys (list, optional): a list of keys indicating the meta info to pop + + Returns: + DataProto: the DataProto with the poped batch_keys and meta_info_keys + """ + assert batch_keys is not None + if meta_info_keys is None: + meta_info_keys = [] + if non_tensor_batch_keys is None: + non_tensor_batch_keys = [] + + tensors = {} + # tensor batch + for key in batch_keys: + assert key in self.batch.keys() + tensors[key] = self.batch.pop(key) + non_tensors = {} + # non tensor batch + for key in non_tensor_batch_keys: + assert key in self.non_tensor_batch.keys() + non_tensors[key] = self.non_tensor_batch.pop(key) + meta_info = {} + for key in meta_info_keys: + assert key in self.meta_info.keys() + meta_info[key] = self.meta_info.pop(key) + return DataProto.from_dict(tensors=tensors, non_tensors=non_tensors, meta_info=meta_info) + + def rename(self, old_keys=None, new_keys=None) -> 'DataProto': + """ + Note that this function only rename the key in the batch + """ + + def validate_input(keys): + if keys is not None: + if isinstance(keys, str): + keys = [keys] + elif isinstance(keys, list): + pass + else: + raise TypeError(f'keys must be a list or a string, but got {type(keys)}') + return keys + + old_keys = validate_input(old_keys) + new_keys = validate_input(new_keys) + + if len(new_keys) != len(old_keys): + raise ValueError( + f'new_keys and old_keys must have the same length, but got {len(new_keys)} and {len(old_keys)}') + + self.batch.rename_key_(tuple(old_keys), tuple(new_keys)) + + return self + + def union(self, other: 'DataProto') -> 'DataProto': + """Union with another DataProto. Union batch and meta_info separately. + Throw an error if + - there are conflict keys in batch and they are not equal + - the batch size of two data batch is not the same + - there are conflict keys in meta_info and they are not the same. + + Args: + other (DataProto): another DataProto to union + + Returns: + DataProto: the DataProto after union + """ + self.batch = union_tensor_dict(self.batch, other.batch) + self.non_tensor_batch = union_numpy_dict(self.non_tensor_batch, other.non_tensor_batch) + self.meta_info = union_two_dict(self.meta_info, other.meta_info) + return self + + def make_iterator(self, mini_batch_size, epochs, seed=None, dataloader_kwargs=None): + """Make an iterator from the DataProto. This is built upon that TensorDict can be used as a normal Pytorch + dataset. See https://pytorch.org/tensordict/tutorials/data_fashion for more details. + + Args: + mini_batch_size (int): mini-batch size when iterating the dataset. We require that + ``batch.batch_size[0] % mini_batch_size == 0`` + epochs (int): number of epochs when iterating the dataset. + dataloader_kwargs: internally, it returns a DataLoader over the batch. + The dataloader_kwargs is the kwargs passed to the DataLoader + + Returns: + Iterator: an iterator that yields a mini-batch data at a time. The total number of iteration steps is + ``self.batch.batch_size * epochs // mini_batch_size`` + """ + assert self.batch.batch_size[0] % mini_batch_size == 0, f"{self.batch.batch_size[0]} % {mini_batch_size} != 0" + # we can directly create a dataloader from TensorDict + if dataloader_kwargs is None: + dataloader_kwargs = {} + + if seed is not None: + generator = torch.Generator() + generator.manual_seed(seed) + else: + generator = None + + assert isinstance(dataloader_kwargs, Dict) + train_dataloader = DataLoader(dataset=self, + batch_size=mini_batch_size, + collate_fn=collate_fn, + generator=generator, + **dataloader_kwargs) + + def get_data(): + for _ in range(epochs): + for d in train_dataloader: + d.meta_info = self.meta_info + yield d + + return iter(get_data()) + + def chunk(self, chunks: int) -> List['DataProto']: + """Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. + + Args: + chunks (int): the number of chunks to split on dim=0 + + Returns: + List[DataProto]: a list of DataProto after splitting + """ + assert len( + self) % chunks == 0, f'only support equal chunk. Got size of DataProto {len(self)} and chunk {chunks}.' + + if self.batch is not None: + batch_lst = self.batch.chunk(chunks=chunks, dim=0) + else: + batch_lst = [None for _ in range(chunks)] + + non_tensor_batch_lst = [{} for _ in range(chunks)] + for key, val in self.non_tensor_batch.items(): + assert isinstance(val, np.ndarray) + non_tensor_lst = np.array_split(val, chunks) + assert len(non_tensor_lst) == chunks + for i in range(chunks): + non_tensor_batch_lst[i][key] = non_tensor_lst[i] + + output = [] + for i in range(chunks): + output.append( + DataProto(batch=batch_lst[i], non_tensor_batch=non_tensor_batch_lst[i], meta_info=self.meta_info)) + + return output + + @staticmethod + def concat(data: List['DataProto']) -> 'DataProto': + """Concat a list of DataProto. The batch is concatenated among dim=0. + The meta_info is assumed to be identical and will use the first one. + + Args: + data (List[DataProto]): list of DataProto + + Returns: + DataProto: concatenated DataProto + """ + batch_lst = [] + for batch in data: + batch_lst.append(batch.batch) + if batch_lst[0] is not None: + new_batch = torch.cat(batch_lst, dim=0) + else: + new_batch = None + + non_tensor_batch = list_of_dict_to_dict_of_list(list_of_dict=[d.non_tensor_batch for d in data]) + for key, val in non_tensor_batch.items(): + non_tensor_batch[key] = np.concatenate(val, axis=0) + + return DataProto(batch=new_batch, non_tensor_batch=non_tensor_batch, meta_info=data[0].meta_info) + + def reorder(self, indices): + """ + Note that this operation is in-place + """ + indices_np = indices.detach().numpy() + self.batch = self.batch[indices] + self.non_tensor_batch = {key: val[indices_np] for key, val in self.non_tensor_batch.items()} + + def repeat(self, repeat_times=2, interleave=True): + """ + Repeat the batch data a specified number of times. + + Args: + repeat_times (int): Number of times to repeat the data. + interleave (bool): Whether to interleave the repeated data. + + Returns: + DataProto: A new DataProto with repeated data. + """ + if self.batch is not None: + if interleave: + # Interleave the data + repeated_tensors = { + key: tensor.repeat_interleave(repeat_times, dim=0) for key, tensor in self.batch.items() + } + else: + # Stack the data + repeated_tensors = { + key: tensor.unsqueeze(0).expand(repeat_times, *tensor.shape).reshape(-1, *tensor.shape[1:]) + for key, tensor in self.batch.items() + } + + repeated_batch = TensorDict( + source=repeated_tensors, + batch_size=(self.batch.batch_size[0] * repeat_times,), + ) + else: + repeated_batch = None + + repeated_non_tensor_batch = {} + for key, val in self.non_tensor_batch.items(): + if interleave: + repeated_non_tensor_batch[key] = np.repeat(val, repeat_times, axis=0) + else: + repeated_non_tensor_batch[key] = np.tile(val, (repeat_times,) + (1,) * (val.ndim - 1)) + + return DataProto( + batch=repeated_batch, + non_tensor_batch=repeated_non_tensor_batch, + meta_info=self.meta_info, + ) + + +import ray + + +@dataclass +class DataProtoFuture: + """ + DataProtoFuture aims to eliminate actual data fetching on driver. By doing so, the driver doesn't have to wait + for data so that asynchronous execution becomes possible. + DataProtoFuture contains a list of futures from another WorkerGroup of size world_size. + - collect_fn is a Callable that reduces the list of futures to a DataProto + - dispatch_fn is a Callable that partitions the DataProto into a list of DataProto of size world_size and then select + + Potential issue: we can optimize dispatch_fn(collect_fn) such that only needed data is fetched on destination + - DataProtoFuture only supports directly passing from the output of a method to another input. You can't perform any + operation on the DataProtoFuture in driver. + """ + collect_fn: Callable + futures: List[ray.ObjectRef] + dispatch_fn: Callable = None + + @staticmethod + def concat(data: List[ray.ObjectRef]) -> 'DataProtoFuture': + output = DataProtoFuture(collect_fn=DataProto.concat, futures=data) + return output + + def chunk(self, chunks: int) -> List['DataProtoFuture']: + from functools import partial + + arg_future_lst = [] + for i in range(chunks): + # note that we can't directly pass i and chunks + def dispatch_fn(x, i, chunks): + return x.chunk(chunks=chunks)[i] + + arg_future = DataProtoFuture(collect_fn=self.collect_fn, + dispatch_fn=partial(dispatch_fn, i=i, chunks=chunks), + futures=self.futures) + arg_future_lst.append(arg_future) + return arg_future_lst + + def get(self): + output = ray.get(self.futures) # dp_size. + for o in output: + assert isinstance(o, DataProto) + output = self.collect_fn(output) # select dp, concat + if self.dispatch_fn is not None: + output = self.dispatch_fn(output) # split in batch dim, select using dp + return output diff --git a/frameworks/agentgym_rl/verl/single_controller/__init__.py b/frameworks/agentgym_rl/verl/single_controller/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a169d56751fb7bdbe4a46f7fd21be7752110db04 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/__init__.py @@ -0,0 +1,20 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +version_folder = os.path.dirname(os.path.join(os.path.abspath(__file__))) + +with open(os.path.join(os.path.join(version_folder, os.pardir), 'version/version')) as f: + __version__ = f.read().strip() diff --git a/frameworks/agentgym_rl/verl/single_controller/base/__init__.py b/frameworks/agentgym_rl/verl/single_controller/base/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..75846436cd1285259d2bae6d4a7f190aebed1a80 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/__init__.py @@ -0,0 +1,16 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .worker import Worker +from .worker_group import WorkerGroup, ClassWithInitArgs, ResourcePool diff --git a/frameworks/agentgym_rl/verl/single_controller/base/decorator.py b/frameworks/agentgym_rl/verl/single_controller/base/decorator.py new file mode 100644 index 0000000000000000000000000000000000000000..6fdacb6d97bc5897be837863236f6f057a024739 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/decorator.py @@ -0,0 +1,410 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from enum import Enum +from functools import wraps +from typing import Dict, List, Tuple +from types import FunctionType +from verl.protocol import DataProtoFuture + +# here we add a magic number of avoid user-defined function already have this attribute +MAGIC_ATTR = 'attrs_3141562937' + + +class Dispatch(Enum): + RANK_ZERO = 0 + ONE_TO_ALL = 1 + ALL_TO_ALL = 2 + MEGATRON_COMPUTE = 3 + MEGATRON_PP_AS_DP = 4 + MEGATRON_PP_ONLY = 5 + MEGATRON_COMPUTE_PROTO = 6 + MEGATRON_PP_AS_DP_PROTO = 7 + DP_COMPUTE = 8 + DP_COMPUTE_PROTO = 9 + DP_COMPUTE_PROTO_WITH_FUNC = 10 + DP_COMPUTE_METRIC = 11 + + +class Execute(Enum): + ALL = 0 + RANK_ZERO = 1 + + +def _split_args_kwargs_data_proto(chunks, *args, **kwargs): + from verl.protocol import DataProto, DataProtoFuture + splitted_args = [] + for arg in args: + assert isinstance(arg, (DataProto, DataProtoFuture)) + splitted_args.append(arg.chunk(chunks=chunks)) + + splitted_kwargs = {} + for key, val in kwargs.items(): + assert isinstance(val, (DataProto, DataProtoFuture)) + splitted_kwargs[key] = val.chunk(chunks=chunks) + + return splitted_args, splitted_kwargs + + +def dispatch_one_to_all(worker_group, *args, **kwargs): + args = tuple([arg] * worker_group.world_size for arg in args) + kwargs = {k: [v] * worker_group.world_size for k, v in kwargs.items()} + return args, kwargs + + +def dispatch_all_to_all(worker_group, *args, **kwargs): + return args, kwargs + + +def collect_all_to_all(worker_group, output): + return output + + +def dispatch_megatron_compute(worker_group, *args, **kwargs): + """ + User passes in dp data. The data is dispatched to all tp/pp ranks with the same dp + """ + from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + assert isinstance(worker_group, + MegatronWorkerGroup), f'worker_group must be MegatronWorkerGroup, Got {type(worker_group)}' + + all_args = [] + for arg in args: + assert isinstance(arg, (Tuple, List)) and len(arg) == worker_group.dp_size + transformed_args = [] + for i in range(worker_group.world_size): + local_dp_rank = worker_group.get_megatron_rank_info(rank=i).dp_rank + transformed_args.append(arg[local_dp_rank]) + all_args.append(transformed_args) + all_args = tuple(all_args) + + all_kwargs = {} + for k, v in kwargs.items(): + assert isinstance(v, (Tuple, List)) and len(v) == worker_group.dp_size + transformed_v = [] + for i in range(worker_group.world_size): + local_dp_rank = worker_group.get_megatron_rank_info(rank=i).dp_rank + transformed_v.append(v[local_dp_rank]) + all_kwargs[k] = transformed_v + return all_args, all_kwargs + + +def collect_megatron_compute(worker_group, output): + """ + Only collect the data from the tp=0 and pp=last and every dp ranks + """ + from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + assert isinstance(worker_group, MegatronWorkerGroup) + output_in_dp = [] + pp_size = worker_group.get_megatron_global_info().pp_size + for global_rank in range(worker_group.world_size): + local_rank_info = worker_group.get_megatron_rank_info(rank=global_rank) + if local_rank_info.tp_rank == 0 and local_rank_info.pp_rank == pp_size - 1: + output_in_dp.append(output[global_rank]) + return output_in_dp + + +def dispatch_megatron_compute_data_proto(worker_group, *args, **kwargs): + """ + All the args and kwargs must be DataProto. The batch will be chunked by dp_size and passed to each rank + """ + from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + assert isinstance(worker_group, MegatronWorkerGroup) + + splitted_args, splitted_kwargs = _split_args_kwargs_data_proto(worker_group.dp_size, *args, **kwargs) + return dispatch_megatron_compute(worker_group, *splitted_args, **splitted_kwargs) + + +def _concat_data_proto_or_future(output: List): + from verl.protocol import DataProto, DataProtoFuture + import ray + + # make sure all the elements in output has the same type + for o in output: + assert type(o) == type(output[0]) + + o = output[0] + + if isinstance(o, DataProto): + return DataProto.concat(output) + elif isinstance(o, ray.ObjectRef): + return DataProtoFuture.concat(output) + else: + raise NotImplementedError + + +def collect_megatron_compute_data_proto(worker_group, output): + """ + Each output must be a DataProto. We concat the dim=0 of output + """ + from verl.protocol import DataProto + import ray + + output = collect_megatron_compute(worker_group, output) + for o in output: + assert isinstance(o, (DataProto, ray.ObjectRef)), f"expecting {o} to be DataProto, but got {type(o)}" + + return _concat_data_proto_or_future(output) + + +def dispatch_megatron_pp_as_dp(worker_group, *args, **kwargs): + """ + treat pp as dp. + """ + from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + assert isinstance(worker_group, MegatronWorkerGroup) + + pp_size = worker_group.pp_size + dp_size = worker_group.dp_size + + pp_dp_size = pp_size * dp_size + + all_args = [] + for arg in args: + assert isinstance(arg, (List, Tuple)) and len(arg) == pp_dp_size + transformed_args = [] + for i in range(worker_group.world_size): + local_dp_rank = worker_group.get_megatron_rank_info(rank=i).dp_rank + local_pp_rank = worker_group.get_megatron_rank_info(rank=i).pp_rank + # compute the rank in arg. Note that the order is dp then pp + # Also note that the outputs within a pp group will be firstly allgathered, then only the output of pp0 will be collected. + # For pp=2 dp=4, a batch of data "ABCDEFGH" should be dispatched and collected in below order: + # dispatch: pp_allgther: collect: + # dp 0 1 2 3 dp 0 1 2 3 + # pp +---------+ pp +-------------+ + # 0 | A C E G | 0 | AB CD EF GH | ABCDEFGH + # 1 | B D F H | 1 | AB CD EF GH | + # +---------+ +-------------+ + arg_rank = local_dp_rank * worker_group.pp_size + local_pp_rank + + transformed_args.append(arg[arg_rank]) + all_args.append(transformed_args) + all_args = tuple(all_args) + + all_kwargs = {} + for k, v in kwargs.items(): + assert isinstance(v, (List, Tuple)) and len(v) == pp_dp_size, f'expect len(v)=={pp_dp_size}, got {len(v)}' + transformed_v = [] + for i in range(worker_group.world_size): + local_dp_rank = worker_group.get_megatron_rank_info(rank=i).dp_rank + local_pp_rank = worker_group.get_megatron_rank_info(rank=i).pp_rank + # compute the rank in arg. Note that the order is dp then pp + arg_rank = local_dp_rank * worker_group.pp_size + local_pp_rank + transformed_v.append(v[arg_rank]) + all_kwargs[k] = transformed_v + return all_args, all_kwargs + + +def collect_megatron_pp_as_dp(worker_group, output): + """ + treat pp as dp. Only collect data on tp=0 + """ + from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + assert isinstance(worker_group, MegatronWorkerGroup) + output_in_dp = [] + for global_rank in range(worker_group.world_size): + local_rank_info = worker_group.get_megatron_rank_info(rank=global_rank) + if local_rank_info.tp_rank == 0 and local_rank_info.pp_rank == 0: + output_in_dp.append(output[global_rank]) + return output_in_dp + + +def collect_megatron_pp_only(worker_group, output): + """ + Only collect output of megatron pp. This is useful when examine weight names as they are identical in tp/dp + """ + from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + assert isinstance(worker_group, MegatronWorkerGroup) + output_in_pp = [] + for global_rank in range(worker_group.world_size): + local_rank_info = worker_group.get_megatron_rank_info(rank=global_rank) + if local_rank_info.tp_rank == 0 and local_rank_info.dp_rank == 0: + output_in_pp.append(output[global_rank]) + return output_in_pp + + +def dispatch_megatron_pp_as_dp_data_proto(worker_group, *args, **kwargs): + from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + assert isinstance(worker_group, MegatronWorkerGroup) + + pp_dp_size = worker_group.dp_size * worker_group.pp_size + splitted_args, splitted_kwargs = _split_args_kwargs_data_proto(pp_dp_size, *args, **kwargs) + return dispatch_megatron_pp_as_dp(worker_group, *splitted_args, **splitted_kwargs) + + +def collect_megatron_pp_as_dp_data_proto(worker_group, output): + from verl.protocol import DataProto + from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + assert isinstance(worker_group, MegatronWorkerGroup) + + output = collect_megatron_pp_as_dp(worker_group, output) + return _concat_data_proto_or_future(output) + + +def dispatch_dp_compute(worker_group, *args, **kwargs): + from verl.single_controller.base.worker_group import WorkerGroup + assert isinstance(worker_group, WorkerGroup) + for arg in args: + assert isinstance(arg, (Tuple, List)) and len(arg) == worker_group.world_size + for k, v in kwargs.items(): + assert isinstance(v, (Tuple, List)) and len(v) == worker_group.world_size + return args, kwargs + + +def collect_dp_compute(worker_group, output): + from verl.single_controller.base.worker_group import WorkerGroup + assert isinstance(worker_group, WorkerGroup) + assert len(output) == worker_group.world_size + return output + + +def dispatch_dp_compute_data_proto(worker_group, *args, **kwargs): + from verl.single_controller.base.worker_group import WorkerGroup + assert isinstance(worker_group, WorkerGroup) + splitted_args, splitted_kwargs = _split_args_kwargs_data_proto(worker_group.world_size, *args, **kwargs) + return splitted_args, splitted_kwargs + + +def dispatch_dp_compute_data_proto_with_func(worker_group, *args, **kwargs): + from verl.single_controller.base.worker_group import WorkerGroup + assert isinstance(worker_group, WorkerGroup) + assert type(args[0]) == FunctionType # NOTE: The first one args is a function! + + splitted_args, splitted_kwargs = _split_args_kwargs_data_proto(worker_group.world_size, *args[1:], **kwargs) + splitted_args_with_func = [[args[0]] * worker_group.world_size] + splitted_args + return splitted_args_with_func, splitted_kwargs + + +def collect_dp_compute_data_proto(worker_group, output): + from verl.protocol import DataProto + import ray + + for o in output: + assert isinstance(o, (DataProto, ray.ObjectRef)), f"expecting {o} to be DataProto, but got {type(o)}" + + output = collect_dp_compute(worker_group, output) + return _concat_data_proto_or_future(output) + + +def get_predefined_dispatch_fn(dispatch_mode): + predefined_dispatch_mode_fn = { + Dispatch.ONE_TO_ALL: { + 'dispatch_fn': dispatch_one_to_all, + 'collect_fn': collect_all_to_all, + }, + Dispatch.ALL_TO_ALL: { + 'dispatch_fn': dispatch_all_to_all, + 'collect_fn': collect_all_to_all, + }, + Dispatch.MEGATRON_COMPUTE: { + 'dispatch_fn': dispatch_megatron_compute, + 'collect_fn': collect_megatron_compute, + }, + Dispatch.MEGATRON_PP_AS_DP: { + 'dispatch_fn': dispatch_megatron_pp_as_dp, + 'collect_fn': collect_megatron_pp_as_dp, + }, + Dispatch.MEGATRON_PP_ONLY: { + 'dispatch_fn': dispatch_one_to_all, + 'collect_fn': collect_megatron_pp_only + }, + Dispatch.MEGATRON_COMPUTE_PROTO: { + 'dispatch_fn': dispatch_megatron_compute_data_proto, + 'collect_fn': collect_megatron_compute_data_proto + }, + Dispatch.MEGATRON_PP_AS_DP_PROTO: { + 'dispatch_fn': dispatch_megatron_pp_as_dp_data_proto, + 'collect_fn': collect_megatron_pp_as_dp_data_proto + }, + Dispatch.DP_COMPUTE: { + 'dispatch_fn': dispatch_dp_compute, + 'collect_fn': collect_dp_compute + }, + Dispatch.DP_COMPUTE_PROTO: { + 'dispatch_fn': dispatch_dp_compute_data_proto, + 'collect_fn': collect_dp_compute_data_proto + }, + Dispatch.DP_COMPUTE_PROTO_WITH_FUNC: { + 'dispatch_fn': dispatch_dp_compute_data_proto_with_func, + 'collect_fn': collect_dp_compute_data_proto + }, + Dispatch.DP_COMPUTE_METRIC: { + 'dispatch_fn': dispatch_dp_compute_data_proto, + 'collect_fn': collect_dp_compute + } + } + return predefined_dispatch_mode_fn[dispatch_mode] + + +def get_predefined_execute_fn(execute_mode): + """ + Note that here we only asks execute_all and execute_rank_zero to be implemented + Leave the choice of how these two functions handle argument 'blocking' to users + """ + predefined_execute_mode_fn = { + Execute.ALL: { + 'execute_fn_name': 'execute_all' + }, + Execute.RANK_ZERO: { + 'execute_fn_name': 'execute_rank_zero' + } + } + return predefined_execute_mode_fn[execute_mode] + + +def _check_dispatch_mode(dispatch_mode): + assert isinstance(dispatch_mode, + (Dispatch, Dict)), f'dispatch_mode must be a Dispatch or a Dict. Got {dispatch_mode}' + if isinstance(dispatch_mode, Dict): + necessary_keys = ['dispatch_fn', 'collect_fn'] + for key in necessary_keys: + assert key in dispatch_mode, f'key {key} should be in dispatch_mode if it is a dictionary' + + +def _check_execute_mode(execute_mode): + assert isinstance(execute_mode, Execute), f'execute_mode must be a Execute. Got {execute_mode}' + + +def _materialize_futures(*args, **kwargs): + new_args = [] + for arg in args: + if isinstance(arg, DataProtoFuture): + arg = arg.get() + # add more type to materialize + new_args.append(arg) + for k, v in kwargs.items(): + if isinstance(v, DataProtoFuture): + kwargs[k] = v.get() + + new_args = tuple(new_args) + return new_args, kwargs + + +def register(dispatch_mode=Dispatch.ALL_TO_ALL, execute_mode=Execute.ALL, blocking=True, materialize_futures=True): + _check_dispatch_mode(dispatch_mode=dispatch_mode) + _check_execute_mode(execute_mode=execute_mode) + + def decorator(func): + + @wraps(func) + def inner(*args, **kwargs): + if materialize_futures: + args, kwargs = _materialize_futures(*args, **kwargs) + return func(*args, **kwargs) + + attrs = {'dispatch_mode': dispatch_mode, 'execute_mode': execute_mode, 'blocking': blocking} + setattr(inner, MAGIC_ATTR, attrs) + return inner + + return decorator diff --git a/frameworks/agentgym_rl/verl/single_controller/base/megatron/__init__.py b/frameworks/agentgym_rl/verl/single_controller/base/megatron/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/megatron/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/single_controller/base/megatron/worker.py b/frameworks/agentgym_rl/verl/single_controller/base/megatron/worker.py new file mode 100644 index 0000000000000000000000000000000000000000..2d84d29f16420a5cf976d64f45ecbb599125c43c --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/megatron/worker.py @@ -0,0 +1,39 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +from dataclasses import dataclass +from verl.single_controller.base.worker import Worker, DistRankInfo, DistGlobalInfo + + +class MegatronWorker(Worker): + + def __init__(self, cuda_visible_devices=None) -> None: + super().__init__(cuda_visible_devices) + + def get_megatron_global_info(self): + from megatron.core import parallel_state as mpu + tp_size = mpu.get_tensor_model_parallel_world_size() + dp_size = mpu.get_data_parallel_world_size() + pp_size = mpu.get_pipeline_model_parallel_world_size() + info = DistGlobalInfo(tp_size=tp_size, dp_size=dp_size, pp_size=pp_size) + return info + + def get_megatron_rank_info(self): + from megatron.core import parallel_state as mpu + tp_rank = mpu.get_tensor_model_parallel_rank() + dp_rank = mpu.get_data_parallel_rank() + pp_rank = mpu.get_pipeline_model_parallel_rank() + info = DistRankInfo(tp_rank=tp_rank, dp_rank=dp_rank, pp_rank=pp_rank) + return info \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/single_controller/base/megatron/worker_group.py b/frameworks/agentgym_rl/verl/single_controller/base/megatron/worker_group.py new file mode 100644 index 0000000000000000000000000000000000000000..67c21d309b75f1fc7e76b87c9436efc103570f50 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/megatron/worker_group.py @@ -0,0 +1,51 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Dict + +from .worker import DistRankInfo, DistGlobalInfo +from verl.single_controller.base import ResourcePool, WorkerGroup + + +class MegatronWorkerGroup(WorkerGroup): + + def __init__(self, resource_pool: ResourcePool, **kwargs): + super().__init__(resource_pool=resource_pool, **kwargs) + self._megatron_rank_info = None + self._megatron_global_info: DistGlobalInfo = None + + def init_megatron(self, default_megatron_kwargs: Dict = None): + raise NotImplementedError(f"MegatronWorkerGroup.init_megatron should be overwritten") + + def get_megatron_rank_info(self, rank: int) -> DistRankInfo: + assert 0 <= rank < self.world_size, f'rank must be from [0, world_size), Got {rank}' + return self._megatron_rank_info[rank] + + @property + def tp_size(self): + assert self._megatron_global_info is not None, "MegatronWorkerGroup._megatron_global_info must be initialized" + return self._megatron_global_info.tp_size + + @property + def dp_size(self): + assert self._megatron_global_info is not None, "MegatronWorkerGroup._megatron_global_info must be initialized" + return self._megatron_global_info.dp_size + + @property + def pp_size(self): + assert self._megatron_global_info is not None, "MegatronWorkerGroup._megatron_global_info must be initialized" + return self._megatron_global_info.pp_size + + def get_megatron_global_info(self): + return self._megatron_global_info diff --git a/frameworks/agentgym_rl/verl/single_controller/base/register_center/__init__.py b/frameworks/agentgym_rl/verl/single_controller/base/register_center/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/register_center/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/single_controller/base/register_center/ray.py b/frameworks/agentgym_rl/verl/single_controller/base/register_center/ray.py new file mode 100644 index 0000000000000000000000000000000000000000..430290cf2683d882d35a83256aa363d959265a05 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/register_center/ray.py @@ -0,0 +1,29 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import ray + + +@ray.remote +class WorkerGroupRegisterCenter: + + def __init__(self, rank_zero_info): + self.rank_zero_info = rank_zero_info + + def get_rank_zero_info(self): + return self.rank_zero_info + + +def create_worker_group_register_center(name, info): + return WorkerGroupRegisterCenter.options(name=name).remote(info) diff --git a/frameworks/agentgym_rl/verl/single_controller/base/worker.py b/frameworks/agentgym_rl/verl/single_controller/base/worker.py new file mode 100644 index 0000000000000000000000000000000000000000..de00e1633c8c5e4975a30c606c6082ccfa508120 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/worker.py @@ -0,0 +1,184 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +the class for Worker +""" +import os +import socket +from dataclasses import dataclass +from verl.single_controller.base.decorator import register, Dispatch, Execute + + +@dataclass +class DistRankInfo: + tp_rank: int + dp_rank: int + pp_rank: int + + +@dataclass +class DistGlobalInfo: + tp_size: int + dp_size: int + pp_size: int + + +class WorkerHelper: + + def _get_node_ip(self): + + def get_node_ip_by_sdk(): + if os.getenv("WG_BACKEND", None) == "ray": + import ray + return ray._private.services.get_node_ip_address() + else: + raise NotImplementedError("WG_BACKEND now just support ray mode.") + + host_ipv4 = os.getenv("MY_HOST_IP", None) + host_ipv6 = os.getenv("MY_HOST_IPV6", None) + host_ip_by_env = host_ipv4 or host_ipv6 + host_ip_by_sdk = get_node_ip_by_sdk() + + host_ip = host_ip_by_env or host_ip_by_sdk + return host_ip + + def _get_free_port(self): + with socket.socket() as sock: + sock.bind(('', 0)) + return sock.getsockname()[1] + + def get_availale_master_addr_port(self): + return self._get_node_ip(), str(self._get_free_port()) + + def _get_pid(self): + return + + +class WorkerMeta: + keys = [ + "WORLD_SIZE", "RANK", "LOCAL_WORLD_SIZE", "LOCAL_RANK", "MASTER_ADDR", "MASTER_PORT", "CUDA_VISIBLE_DEVICES" + ] + + def __init__(self, store) -> None: + self._store = store + + def to_dict(self): + return {f"_{key.lower()}": self._store.get(f"_{key.lower()}", None) for key in WorkerMeta.keys} + + +# we assume that in each WorkerGroup, there is a Master Worker +class Worker(WorkerHelper): + + def __new__(cls, *args, **kwargs): + instance = super().__new__(cls) + + # note that here we use int to distinguish + disable_worker_init = int(os.environ.get('DISABLE_WORKER_INIT', 0)) + if disable_worker_init: + return instance + + rank = os.environ.get("RANK", None) + worker_group_prefix = os.environ.get("WG_PREFIX", None) + + # when decorator @ray.remote applies, __new__ will be called while we don't want to apply _configure_before_init + if None not in [rank, worker_group_prefix] and 'ActorClass(' not in cls.__name__: + instance._configure_before_init(f"{worker_group_prefix}_register_center", int(rank)) + + return instance + + def _configure_before_init(self, register_center_name: str, rank: int): + assert isinstance(rank, int), f"rank must be int, instead of {type(rank)}" + + if rank == 0: + master_addr, master_port = self.get_availale_master_addr_port() + rank_zero_info = { + "MASTER_ADDR": master_addr, + "MASTER_PORT": master_port, + } + + if os.getenv("WG_BACKEND", None) == "ray": + from verl.single_controller.base.register_center.ray import create_worker_group_register_center + self.register_center = create_worker_group_register_center(name=register_center_name, + info=rank_zero_info) + + os.environ.update(rank_zero_info) + + def __init__(self, cuda_visible_devices=None) -> None: + # construct a meta from envrionment variable. Note that the import must be inside the class because it is executed remotely + import os + world_size = int(os.environ['WORLD_SIZE']) + rank = int(os.environ['RANK']) + self._rank = rank + self._world_size = world_size + + master_addr = os.environ["MASTER_ADDR"] + master_port = os.environ["MASTER_PORT"] + + local_world_size = int(os.getenv("LOCAL_WORLD_SIZE", "1")) + local_rank = int(os.getenv("LOCAL_RANK", "0")) + + store = { + '_world_size': world_size, + '_rank': rank, + '_local_world_size': local_world_size, + '_local_rank': local_rank, + '_master_addr': master_addr, + '_master_port': master_port + } + if cuda_visible_devices is not None: + store['_cuda_visible_devices'] = cuda_visible_devices + + meta = WorkerMeta(store=store) + self._configure_with_meta(meta=meta) + + def _configure_with_meta(self, meta: WorkerMeta): + """ + This function should only be called inside by WorkerGroup + """ + assert isinstance(meta, WorkerMeta) + self.__dict__.update(meta.to_dict()) # this is hacky + # print(f"__dict__: {self.__dict__}") + for key in WorkerMeta.keys: + val = self.__dict__.get(f"_{key.lower()}", None) + if val is not None: + # print(f"set {key} to {val}") + os.environ[key] = str(val) + os.environ["REDIS_STORE_SERVER_HOST"] = str(self._master_addr).replace("[", "").replace( + "]", "") if self._master_addr else "" + + def get_master_addr_port(self): + return self._master_addr, self._master_port + + def get_cuda_visible_devices(self): + import os + cuda_visible_devices = os.environ.get("CUDA_VISIBLE_DEVICES", "not set") + return cuda_visible_devices + + @property + def world_size(self): + return self._world_size + + @property + def rank(self): + return self._rank + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO_WITH_FUNC) + def execute_with_func_generator(self, func, *args, **kwargs): + ret_proto = func(self, *args, **kwargs) + return ret_proto + + @register(dispatch_mode=Dispatch.ALL_TO_ALL, execute_mode=Execute.RANK_ZERO) + def execute_func_rank_zero(self, func, *args, **kwargs): + result = func(*args, **kwargs) + return result \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/single_controller/base/worker_group.py b/frameworks/agentgym_rl/verl/single_controller/base/worker_group.py new file mode 100644 index 0000000000000000000000000000000000000000..bd584580c5c7223309e41ac39a865bd48c58c7d4 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/base/worker_group.py @@ -0,0 +1,196 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +the class of WorkerGroup +""" +import logging +import threading +import signal +import time +from typing import List, Any, Callable, Dict + +from verl.single_controller.base.decorator import MAGIC_ATTR, Dispatch, get_predefined_dispatch_fn, get_predefined_execute_fn + + +class ResourcePool: + + def __init__(self, process_on_nodes=None, max_collocate_count: int = 10, n_gpus_per_node=8) -> None: + if process_on_nodes is None: + process_on_nodes = [] + self._store = process_on_nodes + self.max_collocate_count = max_collocate_count + self.n_gpus_per_node = n_gpus_per_node # this is left for future huawei GPU that contains 16 GPUs per node + + def add_node(self, process_count): + self._store.append(process_count) + + @property + def world_size(self): + return sum(self._store) + + def __call__(self) -> Any: + return self._store + + @property + def store(self): + return self._store + + def local_world_size_list(self) -> List[int]: + nested_local_world_size_list = [ + [local_world_size for _ in range(local_world_size)] for local_world_size in self._store + ] + return [item for row in nested_local_world_size_list for item in row] + + def local_rank_list(self) -> List[int]: + nested_local_rank_list = [[i for i in range(local_world_size)] for local_world_size in self._store] + return [item for row in nested_local_rank_list for item in row] + + +class ClassWithInitArgs: + """ + This class stores a class constructor and the args/kwargs to construct the class. + It is used to instantiate the remote class. + """ + + def __init__(self, cls, *args, **kwargs) -> None: + self.cls = cls + self.args = args + self.kwargs = kwargs + + # def add_arg(self, arg): + # self.args += (arg,) + + # def add_kwarg(self, key, value): + # self.kwargs[key] = value + + def __call__(self) -> Any: + return self.cls(*self.args, **self.kwargs) + + +def check_workers_alive(workers: List, is_alive: Callable, gap_time: float = 1) -> None: + import time + while True: + for worker in workers: + if not is_alive(worker): + logging.warning(f"worker {worker} is not alive" + " sending signal to main thread") + signal.raise_signal(signal.SIGABRT) + time.sleep(gap_time) + + +class WorkerGroup: + + def __init__(self, resource_pool: ResourcePool, **kwargs) -> None: + self._is_init_with_detached_workers = True if resource_pool is None else False + + if resource_pool is not None: + # handle the case when WorkGroup is attached to an existing one + self._procecss_dispatch_config = resource_pool() + else: + self._procecss_dispatch_config = None + + self._workers = [] + self._worker_names = [] + + self._master_addr = None + self._master_port = None + + self._checker_thread: threading.Thread = None + + def _is_worker_alive(self, worker): + raise NotImplementedError(f"WorkerGroup._is_worker_alive called, should be implemented in derived class.") + + def _block_until_all_workers_alive(self) -> None: + while True: + all_state = [self._is_worker_alive(worker) for worker in self._workers] + if False in all_state: + time.sleep(1) + else: + break + + def start_worker_aliveness_check(self, every_n_seconds=1) -> None: + # before starting checking worker aliveness, make sure all workers are already alive + self._block_until_all_workers_alive() + + self._checker_thread = threading.Thread(target=check_workers_alive, + args=(self._workers, self._is_worker_alive, every_n_seconds)) + self._checker_thread.start() + + @property + def world_size(self): + return len(self._workers) + + # execute_all_async and execute_rank_zero_async should be implemented by RayWorkerGroup, TorchRPCWorkerGroup, + # MegatronWorkerGroup, XperfWorkerGroup should skip + + def _bind_worker_method(self, user_defined_cls, func_generator): + """ + Bind the worker method to the WorkerGroup + """ + + for method_name in dir(user_defined_cls): + + try: + method = getattr(user_defined_cls, method_name) + assert callable(method), f"{method_name} in {user_defined_cls} is not callable" + except Exception as e: + # if it is a property, it will fail because Class doesn't have instance property + continue + + if hasattr(method, MAGIC_ATTR): + # this method is decorated by register + attribute = getattr(method, MAGIC_ATTR) + assert isinstance(attribute, Dict), f'attribute must be a dictionary. Got {type(attribute)}' + assert 'dispatch_mode' in attribute, f'attribute must contain dispatch_mode in its key' + + dispatch_mode = attribute['dispatch_mode'] + execute_mode = attribute['execute_mode'] + blocking = attribute['blocking'] + + # get dispatch fn + if isinstance(dispatch_mode, Dispatch): + # get default dispatch fn + fn = get_predefined_dispatch_fn(dispatch_mode=dispatch_mode) + dispatch_fn = fn['dispatch_fn'] + collect_fn = fn['collect_fn'] + else: + assert isinstance(dispatch_mode, dict) + assert 'dispatch_fn' in dispatch_mode + assert 'collect_fn' in dispatch_mode + dispatch_fn = dispatch_mode['dispatch_fn'] + collect_fn = dispatch_mode['collect_fn'] + + # get execute_fn_name + execute_mode = get_predefined_execute_fn(execute_mode=execute_mode) + wg_execute_fn_name = execute_mode['execute_fn_name'] + + # get execute_fn from string + try: + execute_fn = getattr(self, wg_execute_fn_name) + assert callable(execute_fn), 'execute_fn must be callable' + except Exception as e: + print(f'execute_fn {wg_execute_fn_name} is invalid') + raise + + # bind a new method to the RayWorkerGroup + func = func_generator(self, + method_name, + dispatch_fn=dispatch_fn, + collect_fn=collect_fn, + execute_fn=execute_fn, + blocking=blocking) + + try: + setattr(self, method_name, func) + except Exception as e: + raise ValueError(f'Fail to set method_name {method_name}') diff --git a/frameworks/agentgym_rl/verl/single_controller/ray/__init__.py b/frameworks/agentgym_rl/verl/single_controller/ray/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..4d5783448e68e7207e45303aaec3894e8ea838d1 --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/ray/__init__.py @@ -0,0 +1,16 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .base import RayResourcePool, RayClassWithInitArgs, RayWorkerGroup, create_colocated_worker_cls +from .megatron import (MegatronRayWorkerGroup, DistRankInfo, DistGlobalInfo) \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/single_controller/ray/base.py b/frameworks/agentgym_rl/verl/single_controller/ray/base.py new file mode 100644 index 0000000000000000000000000000000000000000..4763a20d762c77ab510f0e480b3f6f7e0cd11aef --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/ray/base.py @@ -0,0 +1,459 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import time +from typing import Dict, List, Any, Tuple + +import ray +from ray.util import list_named_actors +from ray.util.placement_group import placement_group, PlacementGroup +from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy, NodeAffinitySchedulingStrategy +from ray.experimental.state.api import get_actor + +from verl.single_controller.base import WorkerGroup, ResourcePool, ClassWithInitArgs, Worker + +__all__ = ['Worker'] + + +def get_random_string(length: int) -> str: + import random + import string + letters_digits = string.ascii_letters + string.digits + return ''.join(random.choice(letters_digits) for _ in range(length)) + + +def func_generator(self, method_name, dispatch_fn, collect_fn, execute_fn, blocking): + + def func(*args, **kwargs): + args, kwargs = dispatch_fn(self, *args, **kwargs) + output = execute_fn(method_name, *args, **kwargs) + if blocking: + output = ray.get(output) + output = collect_fn(self, output) + return output + + return func + + +class RayResourcePool(ResourcePool): + + def __init__(self, + process_on_nodes: List[int] = None, + use_gpu: bool = True, + name_prefix: str = "", + max_colocate_count: int = 5, + detached=False) -> None: + super().__init__(process_on_nodes, max_colocate_count) + self.use_gpu = use_gpu + # print(f"in RayProcessDispatchConfiguration: name_prefix = {name_prefix}") + self.name_prefix = name_prefix + self.pgs = None + self.detached = detached + + def get_placement_groups(self, strategy="STRICT_PACK", name=None): + if self.pgs is not None: + return self.pgs + + pg_name_prefix = name if name else \ + f"{self.name_prefix}verl_group_{'_'.join([str(count) for count in self._store])}:" + # print(f"pg_name_prefix = {pg_name_prefix}") + pg_scheme = [[{ + "CPU": self.max_collocate_count, + "GPU": 1 + } if self.use_gpu else { + "CPU": self.max_collocate_count + } for _ in range(process_count)] for process_count in self._store] + + lifetime = 'detached' if self.detached else None + + pgs = [ + placement_group(bundles=bundles, strategy=strategy, name=pg_name_prefix + str(idx), lifetime=lifetime) + for idx, bundles in enumerate(pg_scheme) + ] + + ray.get([pg.ready() for pg in pgs]) + + self.pgs = pgs + return pgs + + +def extract_pg_from_exist(resource_pools: Dict[str, RayResourcePool], src_role_names: List[str], + resource_pool: RayResourcePool) -> List: + + src_pgs = [ + pg for role_name, resource_pool in resource_pools.items() for pg in resource_pool.get_placement_groups() + if role_name in src_role_names + ] + + sorted_src_pgs = sorted(src_pgs, key=lambda pg: pg.bundle_count, reverse=True) + sorted_process_on_nodes = sorted([(val, idx) for idx, val in enumerate(resource_pool.store)], reverse=True) + + unsorted_pgs: List[Tuple[int, PlacementGroup]] = [] + searching_idx = 0 + for request_process, original_idx in sorted_process_on_nodes: + assert searching_idx < len(sorted_src_pgs), f"no enough nodes for request: searching {searching_idx} th node" + assert request_process <= sorted_src_pgs[searching_idx].bundle_count, \ + f"requesting {request_process} processes, bundle count cannot satisfy" + unsorted_pgs.append((original_idx, sorted_src_pgs[searching_idx])) + searching_idx += 1 + + return [pg for _, pg in sorted(unsorted_pgs)] + + +def merge_resource_pool(rp1: RayResourcePool, rp2: RayResourcePool) -> RayResourcePool: + assert rp1.use_gpu == rp2.use_gpu, 'Both RayResourcePool must either use_gpu or not' + assert rp1.max_collocate_count == rp2.max_collocate_count, 'Both RayResourcePool must has the same max_collocate_count' + assert rp1.n_gpus_per_node == rp2.n_gpus_per_node, 'Both RayResourcePool must has the same n_gpus_per_node' + assert rp1.detached == rp2.detached, 'Detached ResourcePool cannot be merged with non-detached ResourcePool' + + new_store = rp1.store + rp2.store + + merged = RayResourcePool(new_store, rp1.use_gpu, f"{rp1.name_prefix}_{rp2.name_prefix}") + merged.pgs = rp1.get_placement_groups() + rp2.get_placement_groups() + + return merged + + +class RayClassWithInitArgs(ClassWithInitArgs): + + def __init__(self, cls, *args, **kwargs) -> None: + # self._options = kwargs.pop('options', dict()) + super().__init__(cls, *args, **kwargs) + self._options = {} + self._additional_resource = {} + + def set_additional_resource(self, additional_resource): + self._additional_resource = additional_resource + + def update_options(self, options: Dict): + self._options.update(options) + + def __call__(self, + placement_group, + placement_group_bundle_idx, + use_gpu: bool = True, + num_gpus=1, + sharing_with=None) -> Any: + if sharing_with is not None: + target_node_id = ray.get(sharing_with.get_node_id.remote()) + cuda_visible_devices = ray.get(sharing_with.get_cuda_visible_devices.remote()) + options = {"scheduling_strategy": NodeAffinitySchedulingStrategy(node_id=target_node_id, soft=False)} + return self.cls.options(**options).remote(*self.args, + cuda_visible_devices=cuda_visible_devices, + **self.kwargs) + + options = { + "scheduling_strategy": + PlacementGroupSchedulingStrategy(placement_group=placement_group, + placement_group_bundle_index=placement_group_bundle_idx) + } + options.update(self._options) + + if use_gpu: + options["num_gpus"] = num_gpus + + if len(self._additional_resource) > 1: + for k, v in self._additional_resource.items(): + options[k] = v + + # print("cls:", self.cls) + # print("args: ", self.args) + # print("kwargs: ", self.kwargs) + return self.cls.options(**options).remote(*self.args, **self.kwargs) + + +class RayWorkerGroup(WorkerGroup): + + def __init__(self, + resource_pool: RayResourcePool = None, + ray_cls_with_init: RayClassWithInitArgs = None, + bin_pack: bool = True, + name_prefix: str = None, + detached=False, + worker_names=None, + **kwargs) -> None: + super().__init__(resource_pool=resource_pool, **kwargs) + self.ray_cls_with_init = ray_cls_with_init + self.name_prefix = get_random_string(length=6) if name_prefix is None else name_prefix + + if worker_names is not None: + assert self._is_init_with_detached_workers + self._worker_names = worker_names + + if self._is_init_with_detached_workers: + self._init_with_detached_workers(worker_names=worker_names) + else: + self._init_with_resource_pool(resource_pool=resource_pool, + ray_cls_with_init=ray_cls_with_init, + bin_pack=bin_pack, + detached=detached) + + if ray_cls_with_init is not None: + self._bind_worker_method(self.ray_cls_with_init.cls, func_generator) + + def _is_worker_alive(self, worker: ray.actor.ActorHandle): + worker_state_dict = get_actor(worker._actor_id.hex()) + return worker_state_dict.get("state", "undefined") == "ALIVE" if worker_state_dict is not None else False + + def _init_with_detached_workers(self, worker_names): + workers = [ray.get_actor(name=name) for name in worker_names] + self._workers = workers + self._world_size = len(worker_names) + + def _init_with_resource_pool(self, resource_pool, ray_cls_with_init, bin_pack, detached): + use_gpu = resource_pool.use_gpu + + strategy = "PACK" + if bin_pack: + strategy = "STRICT_PACK" + pgs = resource_pool.get_placement_groups(strategy=strategy) + world_size = resource_pool.world_size + self._world_size = world_size + # cia.add_kwarg("_world_size", world_size) + num_gpus = 1 / resource_pool.max_collocate_count + + rank = -1 + for pg_idx, local_world_size in enumerate(resource_pool.store): + pg = pgs[pg_idx] + assert local_world_size <= pg.bundle_count, \ + f"when generating for {self.name_prefix}, for the " + for local_rank in range(local_world_size): + rank += 1 + + # we pass in environment variable at option so that Worker can use environment variable to set + env_vars = { + 'WORLD_SIZE': str(world_size), + 'RANK': str(rank), + 'WG_PREFIX': self.name_prefix, + 'WG_BACKEND': 'ray', + 'RAY_LOCAL_WORLD_SIZE': str(local_world_size), + 'RAY_LOCAL_RANK': str(local_rank), + } + if rank != 0: + env_vars['MASTER_ADDR'] = self._master_addr + env_vars['MASTER_PORT'] = self._master_port + + import re + cia_name = type(ray_cls_with_init.cls).__name__ + match = re.search(r"ActorClass\(([^)]+)\)", cia_name) # ray.remote(Obj) -> "ActorClass(Obj)" + cia_name = match.group(1) if match else cia_name # "ActorClass(Obj)" -> "Obj" + name = f"{self.name_prefix}{cia_name}_{pg_idx}:{local_rank}" # e.g. Worker_2:5 + + ray_cls_with_init.update_options({'runtime_env': {'env_vars': env_vars}, 'name': name}) + + if detached: + ray_cls_with_init.update_options({'lifetime': 'detached'}) + + # create a worker + worker = ray_cls_with_init(placement_group=pg, + placement_group_bundle_idx=local_rank, + use_gpu=use_gpu, + num_gpus=num_gpus) + self._workers.append(worker) + self._worker_names.append(name) + + if rank == 0: + register_center_actor = None + for _ in range(120): + if f"{self.name_prefix}_register_center" not in list_named_actors(): + time.sleep(1) + else: + register_center_actor = ray.get_actor(f"{self.name_prefix}_register_center") + break + assert register_center_actor is not None, f"failed to get register_center_actor: {self.name_prefix}_register_center in {list_named_actors(all_namespaces=True)}" + rank_zero_info = ray.get(register_center_actor.get_rank_zero_info.remote()) + self._master_addr, self._master_port = rank_zero_info['MASTER_ADDR'], rank_zero_info['MASTER_PORT'] + # print(f"rank_zero_info: {rank_zero_info}") + # print(f"master_addr: {self._master_addr}, master_port: {self._master_port}") + + @property + def worker_names(self): + return self._worker_names + + @classmethod + def from_detached(cls, worker_names=None, ray_cls_with_init=None): + worker_group = cls(resource_pool=None, + ray_cls_with_init=ray_cls_with_init, + name_prefix=None, + worker_names=worker_names) + return worker_group + + def spawn(self, prefix_set): + """ + spawn to a dictionary of worker groups, each with a subset of method with prefix. + + """ + + def _rebind_actor_methods(worker_group, actor_name): + """ + bind the method with actor_prefix to its original name + """ + prefix: str = actor_name + '_' + for method_name in dir(worker_group): + if method_name.startswith(prefix): + # only valid when Python >= 3.9 + original_method_name = method_name.removeprefix(prefix) + method = getattr(worker_group, method_name) + setattr(worker_group, original_method_name, method) + + new_worker_group_dict = {} + for prefix in prefix_set: + new_worker_group = self.from_detached(worker_names=self._worker_names, + ray_cls_with_init=self.ray_cls_with_init) + + _rebind_actor_methods(new_worker_group, prefix) + new_worker_group_dict[prefix] = new_worker_group + return new_worker_group_dict + + def execute_rank_zero_sync(self, method_name: str, *args, **kwargs): + return ray.get(self.execute_rank_zero_async(method_name, *args, **kwargs)) + + def execute_rank_zero_async(self, method_name: str, *args, **kwargs): + remote_call = getattr(self._workers[0], method_name) + return remote_call.remote(*args, **kwargs) + + def execute_rank_zero(self, method_name: str, *args, **kwargs): + return self.execute_rank_zero_async(method_name, *args, **kwargs) + + def execute_all(self, method_name: str, *args, **kwargs): + return self.execute_all_async(method_name, *args, **kwargs) + + def execute_all_sync(self, method_name: str, *args, **kwargs): + return ray.get(self.execute_all_async(method_name, *args, **kwargs)) + + def execute_all_async(self, method_name: str, *args, **kwargs): + # 这里我们假设,如果 args 和 kwargs 里面所有的参数都是 list,且所有的 list 长度都与 len(self._workers) 一致的话,我们会把 + # list 中的每一个分别发到对应的 worker 上去 + # print(f"execute_all_async: method {method_name}({args}, {kwargs})") + length = len(self._workers) + if all(isinstance(arg, list) for arg in args) and all(isinstance(kwarg, list) for kwarg in kwargs.values()): + if all(len(arg) == length for arg in args) and all(len(kwarg) == length for kwarg in kwargs.values()): + # print(f"splitting args and kwargs into {length} shards") + result = [] + for i in range(length): + sliced_args = tuple(arg[i] for arg in args) + sliced_kwargs = {k: v[i] for k, v in kwargs.items()} + remote_call = getattr(self._workers[i], method_name) + result.append(remote_call.remote(*sliced_args, **sliced_kwargs)) + return result + + return [getattr(worker, method_name).remote(*args, **kwargs) for worker in self._workers] + + @property + def master_address(self): + return self._master_addr + + @property + def master_port(self): + return self._master_port + + @property + def workers(self): + return self._workers + + @property + def world_size(self): + return self._world_size + + +""" +Utilities that enables creating workers inside the same ray.Actor, +with code written in separate ray.Actors. +""" + +from unittest.mock import patch +from verl.single_controller.base.decorator import MAGIC_ATTR +import os + + +def _bind_workers_method_to_parent(cls, key, user_defined_cls): + """ + Binds the methods of each worker to the WorkerDict. + Note that we only bind public methods that are decorated by register + """ + for method_name in dir(user_defined_cls): + try: + method = getattr(user_defined_cls, method_name) + assert callable(method), f"{method_name} in {user_defined_cls} is not callable" + except Exception as e: + # if it is a property, it will fail because Class doesn't have instance property + continue + + if hasattr(method, MAGIC_ATTR): + + def generate_function(name): + + def func(self, *args, **kwargs): + # dispatch to the actual worker + return getattr(self.worker_dict[key], name)(*args, **kwargs) + + return func + + func = generate_function(method_name) + # pass MAGIC_ATTR for outer worker group + setattr(func, MAGIC_ATTR, getattr(method, MAGIC_ATTR)) + try: + method_name_with_prefix = key + '_' + method_name + setattr(cls, method_name_with_prefix, func) + # print(f'Binding {method_name_with_prefix}') + except Exception as e: + raise ValueError(f'Fail to set method_name {method_name}') + + +def _unwrap_ray_remote(cls): + if hasattr(cls, '__ray_actor_class__'): + cls = cls.__ray_actor_class__ + return cls + + +def create_colocated_worker_cls(class_dict: dict[str, RayClassWithInitArgs]): + """ + This function should return a class instance that delegates the calls to every + cls in cls_dict + """ + cls_dict = {} + init_args_dict = {} + worker_cls = None + for key, cls in class_dict.items(): + if worker_cls == None: + worker_cls = cls.cls.__ray_actor_class__.__base__ + else: + assert worker_cls == cls.cls.__ray_actor_class__.__base__, \ + 'the worker class should be the same when share the same process' + cls_dict[key] = cls.cls + init_args_dict[key] = {'args': cls.args, 'kwargs': cls.kwargs} + + assert cls_dict.keys() == init_args_dict.keys() + + # TODO: create a class with customizable name + class WorkerDict(worker_cls): + + def __init__(self): + super().__init__() + self.worker_dict = {} + for key, user_defined_cls in cls_dict.items(): + user_defined_cls = _unwrap_ray_remote(user_defined_cls) + # directly instantiate the class without remote + with patch.dict(os.environ, {'DISABLE_WORKER_INIT': '1'}): + self.worker_dict[key] = user_defined_cls(*init_args_dict[key].get('args', ()), + **init_args_dict[key].get('kwargs', {})) + + # now monkey-patch the methods from inner class to WorkerDict + for key, user_defined_cls in cls_dict.items(): + user_defined_cls = _unwrap_ray_remote(user_defined_cls) + _bind_workers_method_to_parent(WorkerDict, key, user_defined_cls) + + remote_cls = ray.remote(WorkerDict) + remote_cls = RayClassWithInitArgs(cls=remote_cls) + return remote_cls diff --git a/frameworks/agentgym_rl/verl/single_controller/ray/megatron.py b/frameworks/agentgym_rl/verl/single_controller/ray/megatron.py new file mode 100644 index 0000000000000000000000000000000000000000..3ccb23a1559f8fd039b3791a6fa0da808729f6cb --- /dev/null +++ b/frameworks/agentgym_rl/verl/single_controller/ray/megatron.py @@ -0,0 +1,62 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Dict, Optional + +import ray + +from .base import RayWorkerGroup, RayResourcePool, RayClassWithInitArgs +from verl.single_controller.base.megatron.worker import DistRankInfo, DistGlobalInfo +from verl.single_controller.base.megatron.worker_group import MegatronWorkerGroup + + +# NOTE(sgm): for open-source megatron-core +class NVMegatronRayWorkerGroup(RayWorkerGroup, MegatronWorkerGroup): + """ + MegatronWorkerGroup will query each worker of its megatron rank info and store it inside the WorkerGroup + so that the dispatcher can use it to dispatch data. + """ + + def __init__(self, resource_pool: RayResourcePool, ray_cls_with_init: RayClassWithInitArgs, **kwargs): + super().__init__(resource_pool=resource_pool, ray_cls_with_init=ray_cls_with_init, **kwargs) + self._megatron_rank_info: DistRankInfo = self.execute_all_sync(method_name='get_megatron_rank_info') + self._megatron_global_info: DistGlobalInfo = ray.get( + self.execute_rank_zero_async(method_name='get_megatron_global_info')) + + +class MegatronRayWorkerGroup(RayWorkerGroup, MegatronWorkerGroup): + """ + MegatronWorkerGroup will query each worker of its megatron rank info and store it inside the WorkerGroup + so that the dispatcher can use it to dispatch data. + """ + + def __init__(self, + resource_pool: RayResourcePool, + ray_cls_with_init: RayClassWithInitArgs, + default_megatron_kwargs: Dict = None, + **kwargs): + super().__init__(resource_pool=resource_pool, + ray_cls_with_init=ray_cls_with_init, + default_megatron_kwargs=default_megatron_kwargs, + **kwargs) + self.init_megatron(default_megatron_kwargs=default_megatron_kwargs) + self._megatron_rank_info: DistRankInfo = self.execute_all_sync(method_name='get_megatron_rank_info') + self._megatron_global_info: DistGlobalInfo = ray.get( + self.execute_rank_zero_async(method_name='get_megatron_global_info')) + + def init_megatron(self, default_megatron_kwargs: Optional[Dict] = None): + # after super, we will call init of each worker + if not self._is_init_with_detached_workers: + # only init_megatron if the WorkerGroup is created from scratch + self.execute_all_sync(method_name='init_megatron', default_megatron_kwargs=default_megatron_kwargs) diff --git a/frameworks/agentgym_rl/verl/third_party/__init__.py b/frameworks/agentgym_rl/verl/third_party/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/__init__.py b/frameworks/agentgym_rl/verl/third_party/vllm/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..ab925c1a901b6354edb36c94fec641053bb1a244 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/__init__.py @@ -0,0 +1,60 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from importlib.metadata import version, PackageNotFoundError +from packaging import version as vs + + +def get_version(pkg): + try: + return version(pkg) + except PackageNotFoundError: + return None + + +package_name = 'vllm' +package_version = get_version(package_name) +vllm_version = None + +if package_version == '0.3.1': + vllm_version = '0.3.1' + from .vllm_v_0_3_1.llm import LLM + from .vllm_v_0_3_1.llm import LLMEngine + from .vllm_v_0_3_1 import parallel_state +elif package_version == '0.4.2': + vllm_version = '0.4.2' + from .vllm_v_0_4_2.llm import LLM + from .vllm_v_0_4_2.llm import LLMEngine + from .vllm_v_0_4_2 import parallel_state +elif package_version == '0.5.4': + vllm_version = '0.5.4' + from .vllm_v_0_5_4.llm import LLM + from .vllm_v_0_5_4.llm import LLMEngine + from .vllm_v_0_5_4 import parallel_state +elif package_version == '0.6.3': + vllm_version = '0.6.3' + from .vllm_v_0_6_3.llm import LLM + from .vllm_v_0_6_3.llm import LLMEngine + from .vllm_v_0_6_3 import parallel_state +elif vs.parse(package_version) >= vs.parse('0.6.6.post2.dev252+g8027a724'): + # From 0.6.6.post2 on, vllm supports SPMD inference + # See https://github.com/vllm-project/vllm/pull/12071 + + from vllm import LLM + from vllm.distributed import parallel_state + from .vllm_spmd.dtensor_weight_loaders import load_dtensor_weights +else: + raise ValueError( + f'vllm version {package_version} not supported. Currently supported versions are 0.3.1, 0.4.2, 0.5.4, 0.6.3 and 0.7.0+' + ) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_spmd/__init__.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_spmd/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_spmd/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_spmd/dtensor_weight_loaders.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_spmd/dtensor_weight_loaders.py new file mode 100644 index 0000000000000000000000000000000000000000..a3042cabcc4112472b4bbf70a540471eae9e4073 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_spmd/dtensor_weight_loaders.py @@ -0,0 +1,380 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/model_loader + +from typing import Dict + +import torch.nn as nn +from torch.distributed._tensor import DTensor +from vllm.model_executor.model_loader.weight_utils import default_weight_loader +from vllm.model_executor.models.utils import is_pp_missing_parameter + + +def gemma_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + for param_name, shard_name, shard_id in stacked_params_mapping: + if shard_name not in name: + continue + stacked_name = name.replace(shard_name, param_name) + # Skip loading extra bias for GPTQ models. + if stacked_name.endswith(".bias") and stacked_name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[stacked_name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # lm_head is not used in vllm as it is tied with embed_token. + # To prevent errors, skip loading lm_head.weight. + if "lm_head.weight" in name: + continue + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def gptbigcode_dtensor_load_weights(actor_weights: Dict, vllm_model: nn.Module): + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "lm_head.weight" in name: + continue + if ".attn.bias" in name: + # Skip attention mask. + # NOTE: "c_attn.bias" should not be skipped. + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def starcoder2_dtensor_load_weights(actor_weights: Dict, vllm_model: nn.Module): + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ] + + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + + for param_name, weight_name, shard_id in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def llama_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + (".qkv_proj", ".q_proj", "q"), + (".qkv_proj", ".k_proj", "k"), + (".qkv_proj", ".v_proj", "v"), + (".gate_up_proj", ".gate_proj", 0), + (".gate_up_proj", ".up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if "rotary_emb.cos_cached" in name or "rotary_emb.sin_cached" in name: + # Models trained using ColossalAI may include these tensors in + # the checkpoint. Skip them. + continue + # With tie_word_embeddings, we can skip lm_head.weight + # The weight might appear unnecessarily in the files if the model is + # processed with quantization, LoRA, fine-tuning, etc. + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for param_name, weight_name, shard_id in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight) + + +def qwen2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for param_name, weight_name, shard_id in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def qwen2vl_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for param_name, weight_name, shard_id in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +from vllm.model_executor.layers.fused_moe import FusedMoE + + +def deepseekv2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + + # Params for weights, fp8 weight scales, fp8 activation scales + # (param_name, weight_name, expert_id, shard_id) + expert_params_mapping = FusedMoE.make_expert_params_mapping( + ckpt_gate_proj_name="gate_proj", + ckpt_down_proj_name="down_proj", + ckpt_up_proj_name="up_proj", + num_experts=vllm_model.config.n_routed_experts, + ) + + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + for param_name, weight_name, shard_id in stacked_params_mapping: + # Skip non-stacked layers and experts (experts handled below). + if weight_name not in name: + continue + # We have mlp.experts[0].gate_proj in the checkpoint. + # Since we handle the experts below in expert_params_mapping, + # we need to skip here BEFORE we update the name, otherwise + # name will be updated to mlp.experts[0].gate_up_proj, which + # will then be updated below in expert_params_mapping + # for mlp.experts[0].gate_gate_up_proj, which breaks load. + if ("mlp.experts." in name) and name not in params_dict: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + for mapping in expert_params_mapping: + param_name, weight_name, expert_id, shard_id = mapping + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader( + param, + local_loaded_weight.to(dtype=param.dtype), + weight_name, + shard_id=shard_id, + expert_id=expert_id, + ) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def gpt2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + pass + + +def redistribute_dtensor(param_name: str, loaded_weights: DTensor, parallelize_plan: Dict = None): + param_name = _process_parameter_names(name=param_name) + if parallelize_plan is not None: + assert ( + param_name + in parallelize_plan.keys()), f"param name: {param_name} not in parallelize_plan :{parallelize_plan.keys()}" + placement = parallelize_plan[param_name] + local_loaded_weights = loaded_weights.redistribute(device_mesh=loaded_weights.device_mesh, + placements=placement).to_local() + else: + local_loaded_weights = loaded_weights.full_tensor() + return local_loaded_weights + + +def _process_parameter_names(name): + # Remove '.weight' if it exists at the end of the string + if name.endswith(".weight"): + name = name[:-7] + + # Remove 'model.layers.x.' or 'model.' prefix + if "model.layers" in name: + parts = name.split(".") + # Reconstruct the string without 'model.layers.x.' + name = ".".join(parts[3:]) # parts[0] is 'model', parts[1] is 'layers', parts[2] is 'x' + elif name.startswith("model."): + name = name[6:] # Remove 'model.' + + return name + + +__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__ = { + "GPT2LMHeadModel": gpt2_dtensor_weight_loader, + "LlamaForCausalLM": llama_dtensor_weight_loader, + "LLaMAForCausalLM": llama_dtensor_weight_loader, + "MistralForCausalLM": llama_dtensor_weight_loader, # mistral is the same as llama in vLLM + "InternLMForCausalLM": llama_dtensor_weight_loader, + "AquilaModel": llama_dtensor_weight_loader, + "AquilaForCausalLM": llama_dtensor_weight_loader, + "Phi3ForCausalLM": llama_dtensor_weight_loader, + "GemmaForCausalLM": gemma_dtensor_weight_loader, + "Gemma2ForCausalLM": gemma_dtensor_weight_loader, + "GPTBigCodeForCausalLM": gptbigcode_dtensor_load_weights, + "Starcoder2ForCausalLM": starcoder2_dtensor_load_weights, + "Qwen2ForCausalLM": qwen2_dtensor_weight_loader, + "DeepseekV2ForCausalLM": deepseekv2_dtensor_weight_loader, + "Qwen2VLForConditionalGeneration": qwen2vl_dtensor_weight_loader, +} + + +# the actor model is .state_dict() +# Load dtensor weights +def load_dtensor_weights(actor_weights: Dict, vllm_model: nn.Module): + weight_loader = _get_model_weight_loader(vllm_model.__class__.__name__) + weight_loader(actor_weights, vllm_model) + # NOTE(sgm) to reduce peak memory usage, we offload vllm model to cpu + # after init, and we need this after sync model weights for in first iter. + vllm_model = vllm_model.cuda() + + +def _get_model_weight_loader(arch: str): + if arch in __MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__: + return __MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__.keys()}") + + +# NOTE(sgm): we use per-parameter weight loader in each vllm sub +def update_dtensor_weight_loader(): + pass diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/__init__.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/arg_utils.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/arg_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..1ae8f3b8f62fb62a909f4dfe66ede389b64e61b9 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/arg_utils.py @@ -0,0 +1,228 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py +import argparse +import dataclasses +from dataclasses import dataclass +from typing import Dict, Optional, Tuple + +import torch.nn as nn +from vllm.config import (CacheConfig, DeviceConfig, ModelConfig, ParallelConfig, SchedulerConfig, LoRAConfig) +from transformers import PretrainedConfig +from .config import ModelConfig + + +@dataclass +class EngineArgs: + """Arguments for vLLM engine.""" + model_hf_config: PretrainedConfig = None + dtype: str = 'auto' + kv_cache_dtype: str = 'auto' + seed: int = 0 + max_model_len: Optional[int] = None + worker_use_ray: bool = False + pipeline_parallel_size: int = 1 + tensor_parallel_size: int = 1 + max_parallel_loading_workers: Optional[int] = None + block_size: int = 16 + swap_space: int = 4 # GiB + gpu_memory_utilization: float = 0.90 + max_num_batched_tokens: Optional[int] = None + max_num_seqs: int = 256 + max_paddings: int = 256 + disable_log_stats: bool = False + revision: Optional[str] = None + tokenizer_revision: Optional[str] = None + quantization: Optional[str] = None + load_format: str = 'model' + enforce_eager: bool = False + max_context_len_to_capture: int = 8192 + disable_custom_all_reduce: bool = False + enable_lora: bool = False + max_loras: int = 1 + max_lora_rank: int = 16 + lora_extra_vocab_size: int = 256 + lora_dtype = 'auto' + max_cpu_loras: Optional[int] = None + device: str = 'cuda' + + @staticmethod + def add_cli_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser: + """Shared CLI arguments for vLLM engine.""" + # Model arguments + # TODO(shengguangming): delete the unused args + parser.add_argument('--model', + type=str, + default='facebook/opt-125m', + help='name or path of the huggingface model to use') + parser.add_argument('--tokenizer', + type=str, + default=EngineArgs.tokenizer, + help='name or path of the huggingface tokenizer to use') + parser.add_argument('--revision', + type=str, + default=None, + help='the specific model version to use. It can be a branch ' + 'name, a tag name, or a commit id. If unspecified, will use ' + 'the default version.') + parser.add_argument('--tokenizer-revision', + type=str, + default=None, + help='the specific tokenizer version to use. It can be a branch ' + 'name, a tag name, or a commit id. If unspecified, will use ' + 'the default version.') + parser.add_argument('--tokenizer-mode', + type=str, + default=EngineArgs.tokenizer_mode, + choices=['auto', 'slow'], + help='tokenizer mode. "auto" will use the fast ' + 'tokenizer if available, and "slow" will ' + 'always use the slow tokenizer.') + parser.add_argument('--trust-remote-code', action='store_true', help='trust remote code from huggingface') + parser.add_argument('--download-dir', + type=str, + default=EngineArgs.download_dir, + help='directory to download and load the weights, ' + 'default to the default cache dir of ' + 'huggingface') + parser.add_argument('--load-format', + type=str, + default=EngineArgs.load_format, + choices=['auto', 'pt', 'safetensors', 'npcache', 'dummy'], + help='The format of the model weights to load. ' + '"auto" will try to load the weights in the safetensors format ' + 'and fall back to the pytorch bin format if safetensors format ' + 'is not available. ' + '"pt" will load the weights in the pytorch bin format. ' + '"safetensors" will load the weights in the safetensors format. ' + '"npcache" will load the weights in pytorch format and store ' + 'a numpy cache to speed up the loading. ' + '"dummy" will initialize the weights with random values, ' + 'which is mainly for profiling.') + parser.add_argument('--dtype', + type=str, + default=EngineArgs.dtype, + choices=['auto', 'half', 'float16', 'bfloat16', 'float', 'float32'], + help='data type for model weights and activations. ' + 'The "auto" option will use FP16 precision ' + 'for FP32 and FP16 models, and BF16 precision ' + 'for BF16 models.') + parser.add_argument('--max-model-len', + type=int, + default=None, + help='model context length. If unspecified, ' + 'will be automatically derived from the model.') + # Parallel arguments + parser.add_argument('--worker-use-ray', + action='store_true', + help='use Ray for distributed serving, will be ' + 'automatically set when using more than 1 GPU') + parser.add_argument('--pipeline-parallel-size', + '-pp', + type=int, + default=EngineArgs.pipeline_parallel_size, + help='number of pipeline stages') + parser.add_argument('--tensor-parallel-size', + '-tp', + type=int, + default=EngineArgs.tensor_parallel_size, + help='number of tensor parallel replicas') + # KV cache arguments + parser.add_argument('--block-size', + type=int, + default=EngineArgs.block_size, + choices=[8, 16, 32], + help='token block size') + # TODO(woosuk): Support fine-grained seeds (e.g., seed per request). + parser.add_argument('--seed', type=int, default=EngineArgs.seed, help='random seed') + parser.add_argument('--swap-space', + type=int, + default=EngineArgs.swap_space, + help='CPU swap space size (GiB) per GPU') + parser.add_argument('--gpu-memory-utilization', + type=float, + default=EngineArgs.gpu_memory_utilization, + help='the percentage of GPU memory to be used for' + 'the model executor') + parser.add_argument('--max-num-batched-tokens', + type=int, + default=EngineArgs.max_num_batched_tokens, + help='maximum number of batched tokens per ' + 'iteration') + parser.add_argument('--max-num-seqs', + type=int, + default=EngineArgs.max_num_seqs, + help='maximum number of sequences per iteration') + parser.add_argument('--disable-log-stats', action='store_true', help='disable logging statistics') + # Quantization settings. + parser.add_argument('--quantization', + '-q', + type=str, + choices=['awq', None], + default=None, + help='Method used to quantize the weights') + return parser + + @classmethod + def from_cli_args(cls, args: argparse.Namespace) -> 'EngineArgs': + # Get the list of attributes of this dataclass. + attrs = [attr.name for attr in dataclasses.fields(cls)] + # Set the attributes from the parsed arguments. + engine_args = cls(**{attr: getattr(args, attr) for attr in attrs}) + return engine_args + + def create_engine_configs( + self, + ) -> Tuple[ModelConfig, CacheConfig, ParallelConfig, SchedulerConfig]: + device_config = DeviceConfig(self.device) + model_config = ModelConfig(self.model_hf_config, self.dtype, self.seed, self.load_format, self.revision, + self.tokenizer_revision, self.max_model_len, self.quantization, self.enforce_eager, + self.max_context_len_to_capture) + cache_config = CacheConfig(self.block_size, self.gpu_memory_utilization, self.swap_space, self.kv_cache_dtype, + model_config.get_sliding_window()) + parallel_config = ParallelConfig(self.pipeline_parallel_size, self.tensor_parallel_size, self.worker_use_ray, + self.max_parallel_loading_workers, self.disable_custom_all_reduce) + scheduler_config = SchedulerConfig(self.max_num_batched_tokens, self.max_num_seqs, model_config.max_model_len, + self.max_paddings) + lora_config = LoRAConfig(max_lora_rank=self.max_lora_rank, + max_loras=self.max_loras, + lora_extra_vocab_size=self.lora_extra_vocab_size, + lora_dtype=self.lora_dtype, + max_cpu_loras=self.max_cpu_loras if self.max_cpu_loras and self.max_cpu_loras > 0 else + None) if self.enable_lora else None + return (model_config, cache_config, parallel_config, scheduler_config, device_config, lora_config) + + +@dataclass +class AsyncEngineArgs(EngineArgs): + """Arguments for asynchronous vLLM engine.""" + engine_use_ray: bool = False + disable_log_requests: bool = False + max_log_len: Optional[int] = None + + @staticmethod + def add_cli_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser: + parser = EngineArgs.add_cli_args(parser) + parser.add_argument('--engine-use-ray', + action='store_true', + help='use Ray to start the LLM engine in a ' + 'separate process as the server process.') + parser.add_argument('--disable-log-requests', action='store_true', help='disable logging requests') + parser.add_argument('--max-log-len', + type=int, + default=None, + help='max number of prompt characters or prompt ' + 'ID numbers being printed in log. ' + 'Default: unlimited.') + return parser diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/config.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/config.py new file mode 100644 index 0000000000000000000000000000000000000000..1e1fead86283e1c9594b7556555158a6dc72e6f0 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/config.py @@ -0,0 +1,577 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/config.py + +from typing import Optional, Union, ClassVar +from dataclasses import dataclass +import torch +from transformers import PretrainedConfig +from packaging.version import Version + +from vllm.logger import init_logger +from vllm.transformers_utils.config import get_config +from vllm.utils import get_cpu_memory, is_hip, get_nvcc_cuda_version + +logger = init_logger(__name__) + +_GB = 1 << 30 + + +class ModelConfig: + """Configuration for the model. + + Args: + model: Name or path of the huggingface model to use. + tokenizer: Name or path of the huggingface tokenizer to use. + tokenizer_mode: Tokenizer mode. "auto" will use the fast tokenizer if + available, and "slow" will always use the slow tokenizer. + trust_remote_code: Trust remote code (e.g., from HuggingFace) when + downloading the model and tokenizer. + download_dir: Directory to download and load the weights, default to the + default cache directory of huggingface. + load_format: The format of the model weights to load: + "auto" will try to load the weights in the safetensors format and + fall back to the pytorch bin format if safetensors format is + not available. + "pt" will load the weights in the pytorch bin format. + "safetensors" will load the weights in the safetensors format. + "npcache" will load the weights in pytorch format and store + a numpy cache to speed up the loading. + "dummy" will initialize the weights with random values, which is + mainly for profiling. + dtype: Data type for model weights and activations. The "auto" option + will use FP16 precision for FP32 and FP16 models, and BF16 precision + for BF16 models. + seed: Random seed for reproducibility. + revision: The specific model version to use. It can be a branch name, + a tag name, or a commit id. If unspecified, will use the default + version. + tokenizer_revision: The specific tokenizer version to use. It can be a + branch name, a tag name, or a commit id. If unspecified, will use + the default version. + max_model_len: Maximum length of a sequence (including prompt and + output). If None, will be derived from the model. + quantization: Quantization method that was used to quantize the model + weights. If None, we assume the model weights are not quantized. + enforce_eager: Whether to enforce eager execution. If True, we will + disable CUDA graph and always execute the model in eager mode. + If False, we will use CUDA graph and eager execution in hybrid. + max_context_len_to_capture: Maximum context len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode. + """ + + def __init__( + self, + hf_config: PretrainedConfig, + dtype: str, + seed: int, + load_format: str = 'model', + revision: Optional[str] = None, + tokenizer_revision: Optional[str] = None, + max_model_len: Optional[int] = None, + quantization: Optional[str] = None, + trust_remote_code: Optional[bool] = True, + enforce_eager: bool = False, + max_context_len_to_capture: Optional[int] = None, + ) -> None: + self.model = hf_config._name_or_path + self.tokenizer = hf_config._name_or_path + self.load_format = load_format + self.seed = seed + self.revision = revision + self.tokenizer_revision = tokenizer_revision + self.quantization = quantization + self.trust_remote_code = trust_remote_code + self.enforce_eager = enforce_eager + self.max_context_len_to_capture = max_context_len_to_capture + + # self.hf_config = get_config(model, trust_remote_code, revision) + self.hf_config = hf_config + self.dtype = _get_and_verify_dtype(self.hf_config, dtype) + self.max_model_len = _get_and_verify_max_len(self.hf_config, max_model_len) + # self._verify_load_format() + # self._verify_tokenizer_mode() + self._verify_quantization() + self._verify_cuda_graph() + + def _verify_load_format(self) -> None: + load_format = self.load_format.lower() + if load_format not in ["auto", "pt", "safetensors", "npcache", "dummy", "model"]: + raise ValueError(f"Unknown load format: {self.load_format}. Must be one of " + "'auto', 'pt', 'safetensors', 'npcache', 'dummy' or 'model'.") + self.load_format = load_format + + # def _verify_tokenizer_mode(self) -> None: + # tokenizer_mode = self.tokenizer_mode.lower() + # if tokenizer_mode not in ["auto", "slow"]: + # raise ValueError( + # f"Unknown tokenizer mode: {self.tokenizer_mode}. Must be " + # "either 'auto' or 'slow'.") + # self.tokenizer_mode = tokenizer_mode + + def _verify_quantization(self) -> None: + supported_quantization = ["awq", "gptq", "squeezellm"] + rocm_not_supported_quantization = ["awq", "gptq"] + if self.quantization is not None: + self.quantization = self.quantization.lower() + + # Parse quantization method from the HF model config, if available. + hf_quant_config = getattr(self.hf_config, "quantization_config", None) + if hf_quant_config is not None: + hf_quant_method = str(hf_quant_config["quant_method"]).lower() + if self.quantization is None: + self.quantization = hf_quant_method + elif self.quantization != hf_quant_method: + raise ValueError("Quantization method specified in the model config " + f"({hf_quant_method}) does not match the quantization " + f"method specified in the `quantization` argument " + f"({self.quantization}).") + + if self.quantization is not None: + if self.quantization not in supported_quantization: + raise ValueError(f"Unknown quantization method: {self.quantization}. Must " + f"be one of {supported_quantization}.") + if is_hip() and self.quantization in rocm_not_supported_quantization: + raise ValueError(f"{self.quantization} quantization is currently not supported " + f"in ROCm.") + logger.warning(f"{self.quantization} quantization is not fully " + "optimized yet. The speed can be slower than " + "non-quantized models.") + + def _verify_cuda_graph(self) -> None: + if self.max_context_len_to_capture is None: + self.max_context_len_to_capture = self.max_model_len + self.max_context_len_to_capture = min(self.max_context_len_to_capture, self.max_model_len) + if (self.quantization in ["gptq", "squeezellm"] and not self.enforce_eager): + # Related issue: https://github.com/vllm-project/vllm/issues/2147 + logger.warning(f"{self.quantization} does not support CUDA graph " + "yet. Disabling CUDA graph.") + self.enforce_eager = True + + def verify_with_parallel_config( + self, + parallel_config: "ParallelConfig", + ) -> None: + total_num_attention_heads = self.hf_config.num_attention_heads + tensor_parallel_size = parallel_config.tensor_parallel_size + if total_num_attention_heads % tensor_parallel_size != 0: + raise ValueError(f"Total number of attention heads ({total_num_attention_heads})" + " must be divisible by tensor parallel size " + f"({tensor_parallel_size}).") + + total_num_hidden_layers = self.hf_config.num_hidden_layers + pipeline_parallel_size = parallel_config.pipeline_parallel_size + if total_num_hidden_layers % pipeline_parallel_size != 0: + raise ValueError(f"Total number of hidden layers ({total_num_hidden_layers}) " + "must be divisible by pipeline parallel size " + f"({pipeline_parallel_size}).") + + def get_sliding_window(self) -> Optional[int]: + return getattr(self.hf_config, "sliding_window", None) + + def get_vocab_size(self) -> int: + return self.hf_config.vocab_size + + def get_hidden_size(self) -> int: + return self.hf_config.hidden_size + + def get_head_size(self) -> int: + # FIXME(woosuk): This may not be true for all models. + return self.hf_config.hidden_size // self.hf_config.num_attention_heads + + def get_total_num_kv_heads(self) -> int: + """Returns the total number of KV heads.""" + # For GPTBigCode & Falcon: + # NOTE: for falcon, when new_decoder_architecture is True, the + # multi_query flag is ignored and we use n_head_kv for the number of + # KV heads. + falcon_model_types = ["falcon", "RefinedWeb", "RefinedWebModel"] + new_decoder_arch_falcon = (self.hf_config.model_type in falcon_model_types and + getattr(self.hf_config, "new_decoder_architecture", False)) + if not new_decoder_arch_falcon and getattr(self.hf_config, "multi_query", False): + # Multi-query attention, only one KV head. + # Currently, tensor parallelism is not supported in this case. + return 1 + + attributes = [ + # For Falcon: + "n_head_kv", + "num_kv_heads", + # For LLaMA-2: + "num_key_value_heads", + # For ChatGLM: + "multi_query_group_num", + ] + for attr in attributes: + num_kv_heads = getattr(self.hf_config, attr, None) + if num_kv_heads is not None: + return num_kv_heads + + # For non-grouped-query attention models, the number of KV heads is + # equal to the number of attention heads. + return self.hf_config.num_attention_heads + + def get_num_kv_heads(self, parallel_config: "ParallelConfig") -> int: + """Returns the number of KV heads per GPU.""" + total_num_kv_heads = self.get_total_num_kv_heads() + # If tensor parallelism is used, we divide the number of KV heads by + # the tensor parallel size. We will replicate the KV heads in the + # case where the number of KV heads is smaller than the tensor + # parallel size so each GPU has at least one KV head. + return max(1, total_num_kv_heads // parallel_config.tensor_parallel_size) + + def get_num_layers(self, parallel_config: "ParallelConfig") -> int: + total_num_hidden_layers = self.hf_config.num_hidden_layers + return total_num_hidden_layers // parallel_config.pipeline_parallel_size + + +class CacheConfig: + """Configuration for the KV cache. + + Args: + block_size: Size of a cache block in number of tokens. + gpu_memory_utilization: Fraction of GPU memory to use for the + vLLM execution. + swap_space: Size of the CPU swap space per GPU (in GiB). + cache_dtype: Data type for kv cache storage. + """ + + def __init__( + self, + block_size: int, + gpu_memory_utilization: float, + swap_space: int, + cache_dtype: str, + sliding_window: Optional[int] = None, + ) -> None: + self.block_size = block_size + self.gpu_memory_utilization = gpu_memory_utilization + self.swap_space_bytes = swap_space * _GB + self.cache_dtype = cache_dtype + self.sliding_window = sliding_window + self._verify_args() + self._verify_cache_dtype() + + # Will be set after profiling. + self.num_gpu_blocks = None + self.num_cpu_blocks = None + + def _verify_args(self) -> None: + if self.gpu_memory_utilization > 1.0: + raise ValueError("GPU memory utilization must be less than 1.0. Got " + f"{self.gpu_memory_utilization}.") + + def _verify_cache_dtype(self) -> None: + if self.cache_dtype == "auto": + pass + elif self.cache_dtype == "fp8_e5m2": + nvcc_cuda_version = get_nvcc_cuda_version() + if nvcc_cuda_version < Version("11.8"): + raise ValueError("FP8 is not supported when cuda version is lower than 11.8.") + device_name = torch.cuda.get_device_name() + if "AMD" in device_name: + raise NotImplementedError("FP8_E5M2 KV Cache on AMD GPU has not been supported yet.") + logger.info("Using fp8_e5m2 data type to store kv cache. It reduces " + "the GPU memory footprint and boosts the performance. " + "But it may cause slight accuracy drop. " + "Currently we only support fp8 without scaling factors and " + "make e5m2 as a default format.") + else: + raise ValueError(f"Unknown kv cache dtype: {self.cache_dtype}") + + def verify_with_parallel_config( + self, + parallel_config: "ParallelConfig", + ) -> None: + total_cpu_memory = get_cpu_memory() + # FIXME(woosuk): Here, it is assumed that the GPUs in a tensor parallel + # group are in the same node. However, the GPUs may span multiple nodes. + num_gpus_per_node = parallel_config.tensor_parallel_size + cpu_memory_usage = self.swap_space_bytes * num_gpus_per_node + + msg = (f"{cpu_memory_usage / _GB:.2f} GiB out of " + f"the {total_cpu_memory / _GB:.2f} GiB total CPU memory is " + "allocated for the swap space.") + if cpu_memory_usage > 0.7 * total_cpu_memory: + raise ValueError("Too large swap space. " + msg) + elif cpu_memory_usage > 0.4 * total_cpu_memory: + logger.warning("Possibly too large swap space. " + msg) + + +class ParallelConfig: + """Configuration for the distributed execution. + + Args: + pipeline_parallel_size: Number of pipeline parallel groups. + tensor_parallel_size: Number of tensor parallel groups. + worker_use_ray: Whether to use Ray for model workers. Will be set to + True if either pipeline_parallel_size or tensor_parallel_size is + greater than 1. + max_parallel_loading_workers: Maximum number of multiple batches + when load model sequentially. To avoid RAM OOM when using tensor + parallel and large models. + disable_custom_all_reduce: Disable the custom all-reduce kernel and + fall back to NCCL. + """ + + def __init__( + self, + pipeline_parallel_size: int, + tensor_parallel_size: int, + worker_use_ray: bool, + max_parallel_loading_workers: Optional[int] = None, + disable_custom_all_reduce: bool = False, + ) -> None: + self.pipeline_parallel_size = pipeline_parallel_size + self.tensor_parallel_size = tensor_parallel_size + self.worker_use_ray = worker_use_ray + self.max_parallel_loading_workers = max_parallel_loading_workers + self.disable_custom_all_reduce = disable_custom_all_reduce + + self.world_size = pipeline_parallel_size * tensor_parallel_size + if self.world_size > 1: + self.worker_use_ray = True + self._verify_args() + + def _verify_args(self) -> None: + if self.pipeline_parallel_size > 1: + raise NotImplementedError("Pipeline parallelism is not supported yet.") + if not self.disable_custom_all_reduce and self.world_size > 1: + if is_hip(): + self.disable_custom_all_reduce = True + logger.info("Disabled the custom all-reduce kernel because it is not " + "supported on AMD GPUs.") + elif self.pipeline_parallel_size > 1: + self.disable_custom_all_reduce = True + logger.info("Disabled the custom all-reduce kernel because it is not " + "supported with pipeline parallelism.") + + # FIXME(woosuk): Fix the stability issues and re-enable the custom + # all-reduce kernel. + if not self.disable_custom_all_reduce and self.world_size > 1: + self.disable_custom_all_reduce = True + logger.info("Custom all-reduce kernels are temporarily disabled due to " + "stability issues. We will re-enable them once the issues are " + "resolved.") + + +class SchedulerConfig: + """Scheduler configuration. + + Args: + max_num_batched_tokens: Maximum number of tokens to be processed in + a single iteration. + max_num_seqs: Maximum number of sequences to be processed in a single + iteration. + max_model_len: Maximum length of a sequence (including prompt + and generated text). + max_paddings: Maximum number of paddings to be added to a batch. + """ + + def __init__( + self, + max_num_batched_tokens: Optional[int], + max_num_seqs: int, + max_model_len: int, + max_paddings: int, + ) -> None: + if max_num_batched_tokens is not None: + self.max_num_batched_tokens = max_num_batched_tokens + else: + # If max_model_len is too short, use 2048 as the default value for + # higher throughput. + self.max_num_batched_tokens = max(max_model_len, 2048) + self.max_num_seqs = max_num_seqs + self.max_model_len = max_model_len + self.max_paddings = max_paddings + self._verify_args() + + def _verify_args(self) -> None: + if self.max_num_batched_tokens < self.max_model_len: + raise ValueError(f"max_num_batched_tokens ({self.max_num_batched_tokens}) is " + f"smaller than max_model_len ({self.max_model_len}). " + "This effectively limits the maximum sequence length to " + "max_num_batched_tokens and makes vLLM reject longer " + "sequences. Please increase max_num_batched_tokens or " + "decrease max_model_len.") + if self.max_num_batched_tokens < self.max_num_seqs: + raise ValueError(f"max_num_batched_tokens ({self.max_num_batched_tokens}) must " + "be greater than or equal to max_num_seqs " + f"({self.max_num_seqs}).") + + +class DeviceConfig: + + def __init__(self, device: str = "cuda") -> None: + self.device = torch.device(device) + + +@dataclass +class LoRAConfig: + max_lora_rank: int + max_loras: int + max_cpu_loras: Optional[int] = None + lora_dtype: Optional[torch.dtype] = None + lora_extra_vocab_size: int = 256 + # This is a constant. + lora_vocab_padding_size: ClassVar[int] = 256 + + def __post_init__(self): + # Keep this in sync with csrc/punica/bgmv/bgmv_config.h + possible_max_ranks = (8, 16, 32, 64) + possible_lora_extra_vocab_size = (0, 256, 512) + if self.max_lora_rank not in possible_max_ranks: + raise ValueError(f"max_lora_rank ({self.max_lora_rank}) must be one of " + f"{possible_max_ranks}.") + if self.lora_extra_vocab_size not in possible_lora_extra_vocab_size: + raise ValueError(f"lora_extra_vocab_size ({self.lora_extra_vocab_size}) " + f"must be one of {possible_lora_extra_vocab_size}.") + if self.max_loras < 1: + raise ValueError(f"max_loras ({self.max_loras}) must be >= 1.") + if self.max_cpu_loras is None: + self.max_cpu_loras = self.max_loras + elif self.max_cpu_loras < self.max_loras: + raise ValueError(f"max_cpu_loras ({self.max_cpu_loras}) must be >= " + f"max_loras ({self.max_loras})") + + def verify_with_model_config(self, model_config: ModelConfig): + if self.lora_dtype in (None, "auto"): + self.lora_dtype = model_config.dtype + elif isinstance(self.lora_dtype, str): + self.lora_dtype = getattr(torch, self.lora_dtype) + if model_config.quantization is not None: + raise ValueError("LoRA is not supported with quantized models yet.") + + def verify_with_scheduler_config(self, scheduler_config: SchedulerConfig): + if scheduler_config.max_num_batched_tokens > 65528: + raise ValueError("Due to limitations of the custom LoRA CUDA kernel, " + "max_num_batched_tokens must be <= 65528 when " + "LoRA is enabled.") + + +_STR_DTYPE_TO_TORCH_DTYPE = { + "half": torch.float16, + "float16": torch.float16, + "float": torch.float32, + "float32": torch.float32, + "bfloat16": torch.bfloat16, +} + +_ROCM_NOT_SUPPORTED_DTYPE = ["float", "float32"] + + +def _get_and_verify_dtype( + config: PretrainedConfig, + dtype: Union[str, torch.dtype], +) -> torch.dtype: + # NOTE: getattr(config, "torch_dtype", torch.float32) is not correct + # because config.torch_dtype can be None. + config_dtype = getattr(config, "torch_dtype", None) + if config_dtype is None: + config_dtype = torch.float32 + + if isinstance(dtype, str): + dtype = dtype.lower() + if dtype == "auto": + if config_dtype == torch.float32: + # Following the common practice, we use float16 for float32 + # models. + torch_dtype = torch.float16 + else: + torch_dtype = config_dtype + else: + if dtype not in _STR_DTYPE_TO_TORCH_DTYPE: + raise ValueError(f"Unknown dtype: {dtype}") + torch_dtype = _STR_DTYPE_TO_TORCH_DTYPE[dtype] + elif isinstance(dtype, torch.dtype): + torch_dtype = dtype + else: + raise ValueError(f"Unknown dtype: {dtype}") + + if is_hip() and torch_dtype == torch.float32: + rocm_supported_dtypes = [ + k for k, v in _STR_DTYPE_TO_TORCH_DTYPE.items() if (k not in _ROCM_NOT_SUPPORTED_DTYPE) + ] + raise ValueError(f"dtype \'{dtype}\' is not supported in ROCm. " + f"Supported dtypes are {rocm_supported_dtypes}") + + # Verify the dtype. + if torch_dtype != config_dtype: + if torch_dtype == torch.float32: + # Upcasting to float32 is allowed. + pass + elif config_dtype == torch.float32: + # Downcasting from float32 to float16 or bfloat16 is allowed. + pass + else: + # Casting between float16 and bfloat16 is allowed with a warning. + logger.warning(f"Casting {config_dtype} to {torch_dtype}.") + + return torch_dtype + + +def _get_and_verify_max_len( + hf_config: PretrainedConfig, + max_model_len: Optional[int], +) -> int: + """Get and verify the model's maximum length.""" + derived_max_model_len = float("inf") + possible_keys = [ + # OPT + "max_position_embeddings", + # GPT-2 + "n_positions", + # MPT + "max_seq_len", + # ChatGLM2 + "seq_length", + # Others + "max_sequence_length", + "max_seq_length", + "seq_len", + ] + for key in possible_keys: + max_len_key = getattr(hf_config, key, None) + if max_len_key is not None: + derived_max_model_len = min(derived_max_model_len, max_len_key) + if derived_max_model_len == float("inf"): + if max_model_len is not None: + # If max_model_len is specified, we use it. + return max_model_len + + default_max_len = 2048 + logger.warning("The model's config.json does not contain any of the following " + "keys to determine the original maximum length of the model: " + f"{possible_keys}. Assuming the model's maximum length is " + f"{default_max_len}.") + derived_max_model_len = default_max_len + + rope_scaling = getattr(hf_config, "rope_scaling", None) + if rope_scaling is not None: + assert "factor" in rope_scaling + scaling_factor = rope_scaling["factor"] + if rope_scaling["type"] == "yarn": + derived_max_model_len = rope_scaling["original_max_position_embeddings"] + derived_max_model_len *= scaling_factor + + if max_model_len is None: + max_model_len = derived_max_model_len + elif max_model_len > derived_max_model_len: + raise ValueError(f"User-specified max_model_len ({max_model_len}) is greater than " + f"the derived max_model_len ({max_len_key}={derived_max_model_len}" + " in model's config.json). This may lead to incorrect model " + "outputs or CUDA errors. Make sure the value is correct and " + "within the model context size.") + return int(max_model_len) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/llm.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/llm.py new file mode 100644 index 0000000000000000000000000000000000000000..8d2475998ca2658b14d4a572e10cbfc96cfc3d35 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/llm.py @@ -0,0 +1,275 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py + +from typing import Dict, List, Optional, Tuple, Union + +from tqdm import tqdm +from transformers import PreTrainedTokenizer, PreTrainedTokenizerFast +from transformers import PretrainedConfig +import torch.nn as nn +from .arg_utils import EngineArgs +from .llm_engine_sp import LLMEngine +from vllm.lora.request import LoRARequest +from vllm.outputs import RequestOutput +from vllm.sampling_params import SamplingParams +from vllm.utils import Counter +import torch +from torch.nn.utils.rnn import pad_sequence +from verl.workers.rollout.tokenizer import HybridEngineBaseTokenizer + + +class LLM: + """An LLM for generating texts from given prompts and sampling parameters. + + This class includes a tokenizer, a language model (possibly distributed + across multiple GPUs), and GPU memory space allocated for intermediate + states (aka KV cache). Given a batch of prompts and sampling parameters, + this class generates texts from the model, using an intelligent batching + mechanism and efficient memory management. + + NOTE: This class is intended to be used for offline inference. For online + serving, use the `AsyncLLMEngine` class instead. + NOTE: For the comprehensive list of arguments, see `EngineArgs`. + + Args: + model: A HuggingFace Transformers model instance. + tokenizer: A HuggingFace Transformers tokenizer instance. + tokenizer_mode: The tokenizer mode. "auto" will use the fast tokenizer + if available, and "slow" will always use the slow tokenizer. + trust_remote_code: Trust remote code (e.g., from HuggingFace) when + downloading the model and tokenizer. + tensor_parallel_size: The number of GPUs to use for distributed + execution with tensor parallelism. + dtype: The data type for the model weights and activations. Currently, + we support `float32`, `float16`, and `bfloat16`. If `auto`, we use + the `torch_dtype` attribute specified in the model config file. + However, if the `torch_dtype` in the config is `float32`, we will + use `float16` instead. + quantization: The method used to quantize the model weights. Currently, + we support "awq". If None, we assume the model weights are not + quantized and use `dtype` to determine the data type of the weights. + revision: The specific model version to use. It can be a branch name, + a tag name, or a commit id. + tokenizer_revision: The specific tokenizer version to use. It can be a + branch name, a tag name, or a commit id. + seed: The seed to initialize the random number generator for sampling. + gpu_memory_utilization: The ratio (between 0 and 1) of GPU memory to + reserve for the model weights, activations, and KV cache. Higher + values will increase the KV cache size and thus improve the model's + throughput. However, if the value is too high, it may cause out-of- + memory (OOM) errors. + swap_space: The size (GiB) of CPU memory per GPU to use as swap space. + This can be used for temporarily storing the states of the requests + when their `best_of` sampling parameters are larger than 1. If all + requests will have `best_of=1`, you can safely set this to 0. + Otherwise, too small values may cause out-of-memory (OOM) errors. + enforce_eager: Whether to enforce eager execution. If True, we will + disable CUDA graph and always execute the model in eager mode. + If False, we will use CUDA graph and eager execution in hybrid. + max_context_len_to_capture: Maximum context len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode. + disable_custom_all_reduce: See ParallelConfig + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast, HybridEngineBaseTokenizer], + model_hf_config: PretrainedConfig, + tokenizer_mode: str = "auto", + trust_remote_code: bool = False, + tensor_parallel_size: int = 1, + dtype: str = "auto", + quantization: Optional[str] = None, + revision: Optional[str] = None, + tokenizer_revision: Optional[str] = None, + seed: int = 0, + gpu_memory_utilization: float = 0.9, + swap_space: int = 4, + enforce_eager: bool = False, + max_context_len_to_capture: int = 8192, + disable_custom_all_reduce: bool = False, + **kwargs, + ) -> None: + if "disable_log_stats" not in kwargs: + kwargs["disable_log_stats"] = True + engine_args = EngineArgs( + model_hf_config=model_hf_config, + tensor_parallel_size=tensor_parallel_size, + dtype=dtype, + quantization=quantization, + revision=revision, + tokenizer_revision=tokenizer_revision, + seed=seed, + gpu_memory_utilization=gpu_memory_utilization, + swap_space=swap_space, + enforce_eager=enforce_eager, + max_context_len_to_capture=max_context_len_to_capture, + disable_custom_all_reduce=disable_custom_all_reduce, + **kwargs, + ) + tokenizer_cls = (PreTrainedTokenizer, PreTrainedTokenizerFast, HybridEngineBaseTokenizer) + if not isinstance(tokenizer, tokenizer_cls): + raise ValueError( + f"Unexpected tokenizer type: {type(tokenizer)}. Must be" + "one of the following: PreTrainedTokenizer, PreTrainedTokenizerFast, verl.workers.rollout.HybridEngineBaseTokenizer" + ) + self.llm_engine = LLMEngine.from_engine_args(model, tokenizer, engine_args) + self.request_counter = Counter() + + def init_cache_engine(self): + self.llm_engine.init_cache_engine() + + def free_cache_engine(self): + self.llm_engine.free_cache_engine() + + def get_tokenizer(self) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]: + return self.llm_engine.tokenizer + + def set_tokenizer( + self, + tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], + ) -> None: + self.llm_engine.tokenizer = tokenizer + + def generate( + self, + prompts: Optional[Union[str, List[str]]] = None, + sampling_params: Optional[SamplingParams] = None, + prompt_token_ids: Optional[List[List[int]]] = None, + prefix_pos: Optional[Union[int, List[int]]] = None, + use_tqdm: bool = True, + lora_request: Optional[LoRARequest] = None, + ) -> List[RequestOutput]: + """Generates the completions for the input prompts. + + NOTE: This class automatically batches the given prompts, considering + the memory constraint. For the best performance, put all of your prompts + into a single list and pass it to this method. + + Args: + prompts: A list of prompts to generate completions for. + sampling_params: The sampling parameters for text generation. If + None, we use the default sampling parameters. + prompt_token_ids: A list of token IDs for the prompts. If None, we + use the tokenizer to convert the prompts to token IDs. + use_tqdm: Whether to use tqdm to display the progress bar. + + Returns: + A list of `RequestOutput` objects containing the generated + completions in the same order as the input prompts. + """ + if prompts is None and prompt_token_ids is None: + raise ValueError("Either prompts or prompt_token_ids must be " + "provided.") + if isinstance(prompts, str): + # Convert a single prompt to a list. + prompts = [prompts] + if prompts is not None and prompt_token_ids is not None: + if len(prompts) != len(prompt_token_ids): + raise ValueError("The lengths of prompts and prompt_token_ids " + "must be the same.") + if sampling_params is None: + # Use default sampling params. + sampling_params = SamplingParams() + + # Add requests to the engine. + num_requests = len(prompts) if prompts is not None else len(prompt_token_ids) + for i in range(num_requests): + prompt = prompts[i] if prompts is not None else None + prefix_pos_i = prefix_pos[i] if prefix_pos is not None else None + token_ids = None if prompt_token_ids is None else prompt_token_ids[i] + if not isinstance(token_ids, list): + # NOTE(shengguangming): convert the rollout input into List[str] + token_ids = self._pre_process_inputs(token_ids) + self._add_request(prompt, sampling_params, token_ids, lora_request=lora_request, prefix_pos=prefix_pos_i) + return self._run_engine(use_tqdm) + + def _add_request( + self, + prompt: Optional[str], + sampling_params: SamplingParams, + prompt_token_ids: Optional[List[int]], + lora_request: Optional[LoRARequest] = None, + prefix_pos: Optional[int] = None, + ) -> None: + request_id = str(next(self.request_counter)) + self.llm_engine.add_request(request_id, + prompt, + sampling_params, + prompt_token_ids, + lora_request=lora_request, + prefix_pos=prefix_pos) + + def _run_engine(self, use_tqdm: bool) -> List[RequestOutput]: + # Initialize tqdm. + if use_tqdm: + num_requests = self.llm_engine.get_num_unfinished_requests() + pbar = tqdm(total=num_requests, desc="Processed prompts") + # Run the engine. + outputs: List[RequestOutput] = [] + while self.llm_engine.has_unfinished_requests(): + step_outputs = self.llm_engine.step() + for output in step_outputs: + if output.finished: + outputs.append(output) + if use_tqdm: + pbar.update(1) + if use_tqdm: + pbar.close() + # Sort the outputs by request ID. + # This is necessary because some requests may be finished earlier than + # its previous requests. + outputs = sorted(outputs, key=lambda x: int(x.request_id)) + # TODO(shengguangming): maybe we can hack the autoregressive logics without only apply post process for better performance + return self._post_process_outputs(outputs) + + # NOTE(shengguangming): add for verl + # TODO(sgm): we can optimize it by making the dataloader yield List[int] without padding. + def _pre_process_inputs(self, prompt_token_ids: torch.Tensor) -> List[int]: + # remove the left padding in the prompt token_id + pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id + non_pad_index = torch.nonzero(prompt_token_ids != pad_token_id, as_tuple=False)[0][0] + token_ids = prompt_token_ids[non_pad_index:].tolist() + return token_ids + + # NOTE(shengguangming): add for verl + def _post_process_outputs(self, outputs: List[RequestOutput]) -> Tuple[torch.Tensor, torch.Tensor]: + output_token_ids = [] + logprobs = [] + for output in outputs: # List[RequestOutput] + output = output.outputs + for output in output: # List[CompletionOutput], usually len == 1 + output_token_ids.append(torch.tensor(output.token_ids)) + # TODO(shengguangming): can be optimzied by rewrite the Sampler._get_logprobs() logits + logprobs_dicts = output.logprobs + if logprobs_dicts is not None: + logprob = [] + for logprobs_dict, id in zip(logprobs_dicts, output.token_ids): + logprob.append(logprobs_dict[id]) + logprobs.append(torch.tensor(logprob)) + + pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id + output_token_ids = pad_sequence(output_token_ids, batch_first=True, padding_value=pad_token_id) + if len(logprobs) > 0: + logprobs = pad_sequence(logprobs, batch_first=True, padding_value=pad_token_id) + return output_token_ids, logprobs + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor]) -> None: + self.llm_engine.sync_model_weights(actor_weights=actor_weights) + + def offload_model_weights(self) -> None: + self.llm_engine.offload_model_weights() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/llm_engine_sp.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/llm_engine_sp.py new file mode 100644 index 0000000000000000000000000000000000000000..e264a8585bc2bb8c5b64efe339af7c9d02475614 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/llm_engine_sp.py @@ -0,0 +1,765 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/engine/llm_engine.py + +import os +import socket +import time +import torch +from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Tuple, Union + +from vllm.lora.request import LoRARequest +from vllm.config import (CacheConfig, DeviceConfig, ModelConfig, ParallelConfig, SchedulerConfig, LoRAConfig) +from vllm.core.scheduler import Scheduler, SchedulerOutputs +from vllm.logger import init_logger +from vllm.outputs import RequestOutput +from vllm.sampling_params import SamplingParams +from vllm.sequence import (SamplerOutput, Sequence, SequenceGroup, SequenceGroupMetadata, SequenceGroupOutput, + SequenceOutput, SequenceStatus) +from vllm.transformers_utils.tokenizer import detokenize_incrementally +from vllm.engine.metrics import StatLogger, Stats +from vllm.utils import Counter +import torch.nn as nn +from .arg_utils import EngineArgs +from .tokenizer import TokenizerGroup + +logger = init_logger(__name__) +_LOCAL_LOGGING_INTERVAL_SEC = 5 + + +class LLMEngine: + """An LLM engine that receives requests and generates texts. + + This is the main class for the vLLM engine. It receives requests + from clients and generates texts from the LLM. It includes a tokenizer, a + language model (possibly distributed across multiple GPUs), and GPU memory + space allocated for intermediate states (aka KV cache). This class utilizes + iteration-level scheduling and efficient memory management to maximize the + serving throughput. + + The `LLM` class wraps this class for offline batched inference and the + `AsyncLLMEngine` class wraps this class for online serving. + + NOTE: The config arguments are derived from the `EngineArgs` class. For the + comprehensive list of arguments, see `EngineArgs`. + + Args: + model_config: The configuration related to the LLM model. + cache_config: The configuration related to the KV cache memory + management. + parallel_config: The configuration related to distributed execution. + scheduler_config: The configuration related to the request scheduler. + distributed_init_method: The initialization method for distributed + execution. See `torch.distributed.init_process_group` for details. + placement_group: Ray placement group for distributed execution. + Required for distributed execution. + log_stats: Whether to log statistics. + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + tokenizer: nn.Module, + model_config: ModelConfig, + cache_config: CacheConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], + distributed_init_method: str, + placement_group: Optional[None], + log_stats: bool, + ) -> None: + logger.info("Initializing an LLM engine with config: " + f"model={model_config.model!r}, " + f"tokenizer={model_config.tokenizer!r}, " + # f"tokenizer_mode={model_config.tokenizer_mode}, " + f"revision={model_config.revision}, " + f"tokenizer_revision={model_config.tokenizer_revision}, " + # f"trust_remote_code={model_config.trust_remote_code}, " + f"dtype={model_config.dtype}, " + f"max_seq_len={model_config.max_model_len}, " + # f"download_dir={model_config.download_dir!r}, " + # f"load_format={model_config.load_format}, " + f"disable_custom_all_reduce={parallel_config.disable_custom_all_reduce}, " + f"tensor_parallel_size={parallel_config.tensor_parallel_size}, " + f"quantization={model_config.quantization}, " + f"seed={model_config.seed})") + # TODO(woosuk): Print more configs in debug mode. + + self.model_config = model_config # TODO: currently is hfconfig + self.cache_config = cache_config + self.lora_config = lora_config + assert self.cache_config.sliding_window == getattr(self.model_config.hf_config, "sliding_window", None) + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.device_config = device_config + self.log_stats = log_stats + self._verify_args() + + # self.model = model # should not store the model, it should be deleted + # TODO(shengguangming): maybe we can choose init here or from arguments + self._init_tokenizer(tokenizer) + + self.seq_counter = Counter() + + # Create the parallel GPU workers. + self._init_workers_sp(model, distributed_init_method) + + # Profile the memory usage and initialize the cache. + self._init_cache_sp() + + # Create the scheduler. + # NOTE(shengguangming): each process will have independent scheduler + self.scheduler = Scheduler(scheduler_config, cache_config, lora_config) + + # Metric Logging. + if self.log_stats: + self.stat_logger = StatLogger(local_interval=_LOCAL_LOGGING_INTERVAL_SEC) + + # Logging. + self.last_logging_time = 0.0 + # List of (timestamp, num_tokens) + self.num_prompt_tokens: List[Tuple[float, int]] = [] + # List of (timestamp, num_tokens) + self.num_generation_tokens: List[Tuple[float, int]] = [] + + def _init_tokenizer(self, tokenizer, **tokenizer_init_kwargs): + init_kwargs = dict(enable_lora=bool(self.lora_config), + max_num_seqs=self.scheduler_config.max_num_seqs, + max_input_length=None) + init_kwargs.update(tokenizer_init_kwargs) + self.tokenizer: TokenizerGroup = TokenizerGroup(tokenizer, **init_kwargs) + + # TODO: check get_lora_tokenizer func + def get_tokenizer_for_seq(self, sequence: Sequence): + return self.tokenizer.get_lora_tokenizer(sequence.lora_request) + + def _init_workers_sp(self, model, distributed_init_method: str): + # Lazy import the Worker to avoid importing torch.cuda/xformers + # before CUDA_VISIBLE_DEVICES is set in the Worker + from .worker import Worker # pylint: disable=import-outside-toplevel + + rank = int(os.getenv("RANK")) + + self.worker = Worker( + model, + self.model_config, + self.parallel_config, + self.scheduler_config, + self.device_config, + rank, + distributed_init_method, + lora_config=self.lora_config, + kv_cache_dtype=self.cache_config.cache_dtype, + ) + + # NOTE(shengguangming): torch.distributed.init_process_group will be called inside the init_model() + self.worker.init_model() + self.worker.load_model() + + def _verify_args(self) -> None: + self.model_config.verify_with_parallel_config(self.parallel_config) + self.cache_config.verify_with_parallel_config(self.parallel_config) + + def _init_cache_sp(self) -> None: + """Profiles the memory usage and initializes the KV cache.""" + # Get the maximum number of blocks that can be allocated on GPU and CPU. + num_blocks = self.worker.profile_num_available_blocks( + block_size=self.cache_config.block_size, + gpu_memory_utilization=self.cache_config.gpu_memory_utilization, + cpu_swap_space=self.cache_config.swap_space_bytes, + cache_dtype=self.cache_config.cache_dtype, + ) + + # NOTE(shengguangming): Now we don't use a shared centralized controler but each process will + # have its own scheduler + num_gpu_blocks = num_blocks[0] + num_cpu_blocks = num_blocks[1] + + # FIXME(woosuk): Change to debug log. + logger.info(f"# GPU blocks: {num_gpu_blocks}, " + f"# CPU blocks: {num_cpu_blocks}") + + if num_gpu_blocks <= 0: + raise ValueError("No available memory for the cache blocks. " + "Try increasing `gpu_memory_utilization` when " + "initializing the engine.") + + max_seq_len = self.cache_config.block_size * num_gpu_blocks + if self.model_config.max_model_len > max_seq_len: + raise ValueError(f"The model's max seq len ({self.model_config.max_model_len}) " + "is larger than the maximum number of tokens that can be " + f"stored in KV cache ({max_seq_len}). Try increasing " + "`gpu_memory_utilization` or decreasing `max_model_len` when " + "initializing the engine.") + + self.cache_config.num_gpu_blocks = num_gpu_blocks + self.cache_config.num_cpu_blocks = num_cpu_blocks + + # Initialize the cache. + self.worker.init_cache_engine(cache_config=self.cache_config) + self.worker.warm_up_model() + + def init_cache_engine(self): + self.worker.init_cache_engine(cache_config=self.cache_config) + + def free_cache_engine(self): + self.worker.free_cache_engine() + + @classmethod + def from_engine_args(cls, model, tokenizer, engine_args: EngineArgs) -> "LLMEngine": + """Creates an LLM engine from the engine arguments.""" + # Create the engine configs. + engine_configs = engine_args.create_engine_configs() + parallel_config = engine_configs[2] + # Initialize the cluster. + distributed_init_method, placement_group = initialize_cluster(parallel_config) + # Create the LLM engine. + engine = cls(model, + tokenizer, + *engine_configs, + distributed_init_method, + placement_group, + log_stats=not engine_args.disable_log_stats) + return engine + + def add_request( + self, + request_id: str, + prompt: Optional[str], + sampling_params: SamplingParams, + prompt_token_ids: Optional[List[int]] = None, + arrival_time: Optional[float] = None, + lora_request: Optional[LoRARequest] = None, + prefix_pos: Optional[int] = None, + ) -> None: + """Add a request to the engine's request pool. + + The request is added to the request pool and will be processed by the + scheduler as `engine.step()` is called. The exact scheduling policy is + determined by the scheduler. + + Args: + request_id: The unique ID of the request. + prompt: The prompt string. Can be None if prompt_token_ids is + provided. + sampling_params: The sampling parameters for text generation. + prompt_token_ids: The token IDs of the prompt. If None, we + use the tokenizer to convert the prompts to token IDs. + arrival_time: The arrival time of the request. If None, we use + the current monotonic time. + prefix_pos: If not None, we use the given position as the prefix + position for each prompt. We will cache the prefix's KV + cache and reuse it for the next request with the same prefix. + This is an experimental feature, and may be replaced with + automatic prefix caching in the future. + + Details: + - Set arrival_time to the current time if it is None. + - Set prompt_token_ids to the encoded prompt if it is None. + - Create `best_of` number of :class:`~vllm.Sequence` objects. + - Create a :class:`~vllm.SequenceGroup` object + from the list of :class:`~vllm.Sequence`. + - Add the :class:`~vllm.SequenceGroup` object to the scheduler. + + Example: + >>> # initialize engine + >>> engine = LLMEngine.from_engine_args(engine_args) + >>> # set request arguments + >>> example_prompt = "Who is the president of the United States?" + >>> sampling_params = SamplingParams(temperature=0.0) + >>> request_id = 0 + >>> + >>> # add the request to the engine + >>> engine.add_request( + >>> str(request_id), + >>> example_prompt, + >>> SamplingParams(temperature=0.0)) + >>> # continue the request processing + >>> ... + """ + if lora_request is not None and not self.lora_config: + raise ValueError(f"Got lora_request {lora_request} but LoRA is " + "not enabled!") + if arrival_time is None: + arrival_time = time.monotonic() + if prompt_token_ids is None: + assert prompt is not None + prompt_token_ids = self.tokenizer.encode(prompt) + + # Create the sequences. + block_size = self.cache_config.block_size + seq_id = next(self.seq_counter) + seq = Sequence(seq_id, prompt, prompt_token_ids, block_size, lora_request) + + # Check whether the input specifies prefix + prefix = self.scheduler.prefix_pool.add_or_get_prefix(prompt_token_ids[:prefix_pos], lora_request.lora_int_id if + lora_request else 0) if prefix_pos is not None else None + + # Create the sequence group. + seq_group = SequenceGroup(request_id, [seq], sampling_params, arrival_time, lora_request, prefix) + + # Add the sequence group to the scheduler. + self.scheduler.add_seq_group(seq_group) + + def abort_request(self, request_id: Union[str, Iterable[str]]) -> None: + """Aborts a request(s) with the given ID. + + Args: + request_id: The ID(s) of the request to abort. + + Details: + - Refer to the + :meth:`~vllm.core.scheduler.Scheduler.abort_seq_group` + from class :class:`~vllm.core.scheduler.Scheduler`. + + Example: + >>> # initialize engine and add a request with request_id + >>> request_id = str(0) + >>> # abort the request + >>> engine.abort_request(request_id) + """ + self.scheduler.abort_seq_group(request_id) + + def get_model_config(self) -> ModelConfig: + """Gets the model configuration.""" + return self.model_config + + def get_num_unfinished_requests(self) -> int: + """Gets the number of unfinished requests.""" + return self.scheduler.get_num_unfinished_seq_groups() + + def has_unfinished_requests(self) -> bool: + """Returns True if there are unfinished requests.""" + return self.scheduler.has_unfinished_seqs() + + def _check_beam_search_early_stopping( + self, + early_stopping: Union[bool, str], + sampling_params: SamplingParams, + best_running_seq: Sequence, + current_worst_seq: Sequence, + ) -> bool: + assert sampling_params.use_beam_search + length_penalty = sampling_params.length_penalty + if early_stopping is True: + return True + + current_worst_score = (current_worst_seq.get_beam_search_score( + length_penalty=length_penalty, eos_token_id=self.get_tokenizer_for_seq(current_worst_seq).eos_token_id)) + if early_stopping is False: + highest_attainable_score = (best_running_seq.get_beam_search_score( + length_penalty=length_penalty, eos_token_id=self.get_tokenizer_for_seq(best_running_seq).eos_token_id)) + else: + assert early_stopping == "never" + if length_penalty > 0.0: + # If length_penalty > 0.0, beam search will prefer longer + # sequences. The highest attainable score calculation is + # based on the longest possible sequence length in this case. + max_possible_length = max(best_running_seq.get_prompt_len() + sampling_params.max_tokens, + self.scheduler_config.max_model_len) + highest_attainable_score = (best_running_seq.get_beam_search_score( + length_penalty=length_penalty, + eos_token_id=self.get_tokenizer_for_seq(best_running_seq).eos_token_id, + seq_len=max_possible_length)) + else: + # Otherwise, beam search will prefer shorter sequences. The + # highest attainable score calculation is based on the current + # sequence length. + highest_attainable_score = (best_running_seq.get_beam_search_score( + length_penalty=length_penalty, + eos_token_id=self.get_tokenizer_for_seq(best_running_seq).eos_token_id)) + + def _process_sequence_group_outputs(self, seq_group: SequenceGroup, outputs: SequenceGroupOutput) -> None: + + # Process prompt logprobs + prompt_logprobs = outputs.prompt_logprobs + if prompt_logprobs is not None: + seq_group.prompt_logprobs = prompt_logprobs + + # Process samples + samples = outputs.samples + parent_seqs = seq_group.get_seqs(status=SequenceStatus.RUNNING) + existing_finished_seqs = seq_group.get_finished_seqs() + parent_child_dict = {parent_seq.seq_id: [] for parent_seq in parent_seqs} + for sample in samples: + parent_child_dict[sample.parent_seq_id].append(sample) + # List of (child, parent) + child_seqs: List[Tuple[Sequence, Sequence]] = [] + + # Process the child samples for each parent sequence + for parent in parent_seqs: + child_samples: List[SequenceOutput] = parent_child_dict[parent.seq_id] + if len(child_samples) == 0: + # This parent sequence has no children samples. Remove + # the parent sequence from the sequence group since it will + # not be used in the future iterations. + parent.status = SequenceStatus.FINISHED_ABORTED + seq_group.remove(parent.seq_id) + self.scheduler.free_seq(parent) + continue + # Fork the parent sequence if there are multiple child samples. + for child_sample in child_samples[:-1]: + new_child_seq_id = next(self.seq_counter) + child = parent.fork(new_child_seq_id) + child.append_token_id(child_sample.output_token, child_sample.logprobs) + child_seqs.append((child, parent)) + # Continue the parent sequence for the last child sample. + # We reuse the parent sequence here to reduce redundant memory + # copies, especially when using non-beam search sampling methods. + last_child_sample = child_samples[-1] + parent.append_token_id(last_child_sample.output_token, last_child_sample.logprobs) + child_seqs.append((parent, parent)) + + for seq, _ in child_seqs: + # self._decode_sequence(seq, seq_group.sampling_params) + self._check_stop(seq, seq_group.sampling_params) + + # Non-beam search case + if not seq_group.sampling_params.use_beam_search: + # For newly created child sequences, add them to the sequence group + # and fork them in block manager if they are not finished. + for seq, parent in child_seqs: + if seq is not parent: + seq_group.add(seq) + if not seq.is_finished(): + self.scheduler.fork_seq(parent, seq) + + # Free the finished and selected parent sequences' memory in block + # manager. Keep them in the sequence group as candidate output. + # NOTE: we need to fork the new sequences before freeing the + # old sequences. + for seq, parent in child_seqs: + if seq is parent and seq.is_finished(): + self.scheduler.free_seq(seq) + return + + # Beam search case + # Select the child sequences to keep in the sequence group. + selected_child_seqs = [] + unselected_child_seqs = [] + beam_width = seq_group.sampling_params.best_of + length_penalty = seq_group.sampling_params.length_penalty + + # Select the newly finished sequences with the highest scores + # to replace existing finished sequences. + # Tuple of (seq, parent, is_new) + existing_finished_seqs = [(seq, None, False) for seq in existing_finished_seqs] + new_finished_seqs = [(seq, parent, True) for seq, parent in child_seqs if seq.is_finished()] + all_finished_seqs = existing_finished_seqs + new_finished_seqs + # Sort the finished sequences by their scores. + all_finished_seqs.sort(key=lambda x: x[0].get_beam_search_score( + length_penalty=length_penalty, eos_token_id=self.get_tokenizer_for_seq(x[0]).eos_token_id), + reverse=True) + for seq, parent, is_new in all_finished_seqs[:beam_width]: + if is_new: + # A newly generated child sequence finishes and has a high + # score, so we will add it into the sequence group. + selected_child_seqs.append((seq, parent)) + for seq, parent, is_new in all_finished_seqs[beam_width:]: + if is_new: + # A newly generated child sequence finishes but has a low + # score, so we will not add it into the sequence group. + # Additionally, if this sequence is a continuation of a + # parent sequence, we will need remove the parent sequence + # from the sequence group. + unselected_child_seqs.append((seq, parent)) + else: + # An existing finished sequence has a low score, so we will + # remove it from the sequence group. + seq_group.remove(seq.seq_id) + + # select the top beam_width sequences from the running + # sequences for the next iteration to continue the beam + # search. + running_child_seqs = [(seq, parent) for seq, parent in child_seqs if not seq.is_finished()] + # Sort the running sequences by their scores. + running_child_seqs.sort(key=lambda x: x[0].get_beam_search_score( + length_penalty=length_penalty, eos_token_id=self.get_tokenizer_for_seq(x[0]).eos_token_id), + reverse=True) + + # Check if we can stop the beam search. + if len(running_child_seqs) == 0: + # No running sequences, stop the beam search. + stop_beam_search = True + elif len(all_finished_seqs) < beam_width: + # Not enough finished sequences, continue the beam search. + stop_beam_search = False + else: + # Check the early stopping criteria + best_running_seq = running_child_seqs[0][0] + current_worst_seq = all_finished_seqs[beam_width - 1][0] + stop_beam_search = self._check_beam_search_early_stopping(seq_group.sampling_params.early_stopping, + seq_group.sampling_params, best_running_seq, + current_worst_seq) + + if stop_beam_search: + # Stop the beam search and remove all the running sequences from + # the sequence group. + unselected_child_seqs.extend(running_child_seqs) + else: + # Continue the beam search and select the top beam_width sequences + # to continue the beam search. + selected_child_seqs.extend(running_child_seqs[:beam_width]) + # The remaining running sequences will not be used in the next + # iteration. Again, if these sequences are continuations of + # parent sequences, we will need to remove the parent sequences + # from the sequence group. + unselected_child_seqs.extend(running_child_seqs[beam_width:]) + + # For newly created child sequences, add them to the sequence group + # and fork them in block manager if they are not finished. + for seq, parent in selected_child_seqs: + if seq is not parent: + seq_group.add(seq) + if not seq.is_finished(): + self.scheduler.fork_seq(parent, seq) + + # Free the finished and selected parent sequences' memory in block + # manager. Keep them in the sequence group as candidate output. + for seq, parent in selected_child_seqs: + if seq is parent and seq.is_finished(): + self.scheduler.free_seq(seq) + + # Remove the unselected parent sequences from the sequence group and + # free their memory in block manager. + for seq, parent in unselected_child_seqs: + if seq is parent: + # Remove the parent sequence if it is not selected for next + # iteration + seq_group.remove(seq.seq_id) + self.scheduler.free_seq(seq) + + def _process_model_outputs(self, output: SamplerOutput, scheduler_outputs: SchedulerOutputs) -> List[RequestOutput]: + # Update the scheduled sequence groups with the model outputs. + scheduled_seq_groups = scheduler_outputs.scheduled_seq_groups + for seq_group, outputs in zip(scheduled_seq_groups, output): + self._process_sequence_group_outputs(seq_group, outputs) + + # Free the finished sequence groups. + self.scheduler.free_finished_seq_groups() + + # Create the outputs. + request_outputs: List[RequestOutput] = [] + for seq_group in scheduled_seq_groups: + request_output = RequestOutput.from_seq_group(seq_group) + request_outputs.append(request_output) + for seq_group in scheduler_outputs.ignored_seq_groups: + request_output = RequestOutput.from_seq_group(seq_group) + request_outputs.append(request_output) + + # Update prefix state, now all the uncomputed prefixes are computed. + for seq_group in scheduled_seq_groups: + if (seq_group.prefix is not None and seq_group.prefix.allocated and not seq_group.prefix.computed): + seq_group.prefix.computed = True + + # Log stats. + if self.log_stats: + self.stat_logger.log(self._get_stats(scheduler_outputs)) + + return request_outputs + + def step(self) -> List[RequestOutput]: + """Performs one decoding iteration and returns newly generated results. + + This function performs one decoding iteration of the engine. It first + schedules the sequences to be executed in the next iteration and the + token blocks to be swapped in/out/copy. Then, it executes the model + and updates the scheduler with the model outputs. Finally, it decodes + the sequences and returns the newly generated results. + """ + seq_group_metadata_list, scheduler_outputs = self.scheduler.schedule() + if not scheduler_outputs.is_empty(): + output = self.worker.execute_model( + seq_group_metadata_list=seq_group_metadata_list, # TODO: check this input + blocks_to_swap_in=scheduler_outputs.blocks_to_swap_in, + blocks_to_swap_out=scheduler_outputs.blocks_to_swap_out, + blocks_to_copy=scheduler_outputs.blocks_to_copy,) + else: + return [RequestOutput.from_seq_group(seq_group) for seq_group in scheduler_outputs.ignored_seq_groups] + + return self._process_model_outputs(output, scheduler_outputs) + + def do_log_stats(self) -> None: + """Forced log when no requests active.""" + if self.log_stats: + self.stat_logger.log(self._get_stats(scheduler_outputs=None)) + + def _get_stats(self, scheduler_outputs: Optional[SchedulerOutputs]) -> Stats: + """Get Stats to be Logged to Prometheus.""" + now = time.monotonic() + + # KV Cache Usage in %. + num_total_gpu = self.cache_config.num_gpu_blocks + num_free_gpu = self.scheduler.block_manager.get_num_free_gpu_blocks() + gpu_cache_usage = 1.0 - (num_free_gpu / num_total_gpu) + + num_total_cpu = self.cache_config.num_cpu_blocks + cpu_cache_usage = 0. + if num_total_cpu > 0: + num_free_cpu = self.scheduler.block_manager.get_num_free_cpu_blocks() + cpu_cache_usage = 1.0 - (num_free_cpu / num_total_cpu) + + # Scheduler State + num_running = len(self.scheduler.running) + num_swapped = len(self.scheduler.swapped) + num_waiting = len(self.scheduler.waiting) + + # Iteration stats if we have scheduler output. + num_prompt_tokens = 0 + num_generation_tokens = 0 + time_to_first_tokens = [] + time_per_output_tokens = [] + time_e2e_requests = [] + if scheduler_outputs is not None: + prompt_run = scheduler_outputs.prompt_run + + # Number of Tokens. + if prompt_run: + num_prompt_tokens = scheduler_outputs.num_batched_tokens + else: + num_generation_tokens = scheduler_outputs.num_batched_tokens + + # Latency Timings. + time_last_iters = [] + for seq_group in scheduler_outputs.scheduled_seq_groups: + # Time since last token. (n.b. updates seq_group.last_token_time) + time_last_iters.append(seq_group.get_last_latency(now)) + # Time since arrival for all finished requests. + if seq_group.is_finished(): + time_e2e_requests.append(now - seq_group.arrival_time) + + time_to_first_tokens = time_last_iters if prompt_run else [] + time_per_output_tokens = [] if prompt_run else time_last_iters + + return Stats( + now=now, + num_running=num_running, + num_swapped=num_swapped, + num_waiting=num_waiting, + gpu_cache_usage=gpu_cache_usage, + cpu_cache_usage=cpu_cache_usage, + num_prompt_tokens=num_prompt_tokens, + num_generation_tokens=num_generation_tokens, + time_to_first_tokens=time_to_first_tokens, + time_per_output_tokens=time_per_output_tokens, + time_e2e_requests=time_e2e_requests, + ) + + # TODO: we may not need to decode + def _decode_sequence(self, seq: Sequence, prms: SamplingParams) -> None: + """Decodes the new token for a sequence.""" + (new_tokens, new_output_text, prefix_offset, read_offset) = detokenize_incrementally( + self.get_tokenizer_for_seq(seq), + all_input_ids=seq.get_token_ids(), + prev_tokens=seq.tokens, + prefix_offset=seq.prefix_offset, + read_offset=seq.read_offset, + skip_special_tokens=prms.skip_special_tokens, + spaces_between_special_tokens=prms.spaces_between_special_tokens, + ) + if seq.tokens is None: + seq.tokens = new_tokens + else: + seq.tokens.extend(new_tokens) + seq.prefix_offset = prefix_offset + seq.read_offset = read_offset + seq.output_text += new_output_text + + def _check_stop(self, seq: Sequence, sampling_params: SamplingParams) -> None: + """Stop the finished sequences.""" + # for stop_str in sampling_params.stop: + # if seq.output_text.endswith(stop_str): + # self._finalize_sequence(seq, sampling_params, stop_str) + # seq.status = SequenceStatus.FINISHED_STOPPED + # return + # if seq.get_last_token_id() in sampling_params.stop_token_ids: + # stop_str = self.get_tokenizer_for_seq(seq).convert_ids_to_tokens(seq.get_last_token_id()) + # self._finalize_sequence(seq, sampling_params, stop_str) + # seq.status = SequenceStatus.FINISHED_STOPPED + # return + + # Check if the sequence has reached max_model_len. + if seq.get_len() > self.scheduler_config.max_model_len: + seq.status = SequenceStatus.FINISHED_LENGTH_CAPPED + return + + # Check if the sequence has reached max_tokens. + if seq.get_output_len() == sampling_params.max_tokens: + seq.status = SequenceStatus.FINISHED_LENGTH_CAPPED + return + + # Check if the sequence has generated the EOS token. + if ((not sampling_params.ignore_eos) and + seq.get_last_token_id() == self.get_tokenizer_for_seq(seq).eos_token_id): + seq.status = SequenceStatus.FINISHED_STOPPED + return + + def _finalize_sequence(self, seq: Sequence, sampling_params: SamplingParams, stop_string: str) -> None: + if not sampling_params.include_stop_str_in_output and stop_string: + # Truncate the output text so that the stop string is + # not included in the output. + seq.output_text = seq.output_text[:-len(stop_string)] + + def add_lora(self, lora_request: LoRARequest) -> bool: + assert lora_request.lora_int_id > 0, "lora_id must be greater than 0." + return self.worker.add_lora(lora_request) + + def remove_lora(self, lora_id: int) -> bool: + assert lora_id > 0, "lora_id must be greater than 0." + return self.worker.remove_lora(lora_id) + + def list_loras(self) -> List[int]: + return self.worker.list_loras() + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor]) -> None: + self.worker.sync_model_weights(actor_weights=actor_weights) + + def offload_model_weights(self) -> None: + self.worker.offload_model_weights() + + +def initialize_cluster( + parallel_config: ParallelConfig, + engine_use_ray: bool = False, + ray_address: Optional[str] = None, +) -> Tuple[str, Optional[None]]: + """Initialize the distributed cluster probably with Ray. + + Args: + parallel_config: The configurations for parallel execution. + engine_use_ray: Whether to use Ray for async engine. + ray_address: The address of the Ray cluster. If None, uses + the default Ray cluster address. + + Returns: + A tuple of (`distributed_init_method`, `placement_group`). The + `distributed_init_method` is the address for initializing the + distributed backend. `placement_group` includes the specification + of the resources for each distributed worker. + """ + + # Initialize cluster locally. + port = get_open_port() + # We need to setup the distributed init method to make sure + # the distributed megatron code (e.g., get world size) works correctly. + distributed_init_method = f"tcp://localhost:{port}" + return distributed_init_method, None + + +def get_open_port(): + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("", 0)) + return s.getsockname()[1] diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/model_loader.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/model_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..450e2f4b49c22b86b5ce424a303c535fb1596a99 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/model_loader.py @@ -0,0 +1,275 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/model_loader +"""Utilities for selecting and loading models.""" +import contextlib +from typing import Dict, Type, Union + +import torch +import torch.nn as nn +from transformers import PretrainedConfig, PreTrainedModel +from megatron.core.tensor_parallel.utils import VocabUtility + +from vllm.model_executor.models import ModelRegistry +from vllm.model_executor.weight_utils import (get_quant_config, initialize_dummy_weights) + +from .config import ModelConfig +from vllm.config import DeviceConfig, LoRAConfig +from .weight_loaders import * +from vllm.model_executor.sampling_metadata import SamplingMetadata, SamplingTensors +from vllm.sequence import SamplerOutput +from typing import Optional +from vllm.model_executor.layers.sampler import Sampler +from vllm.model_executor.layers.sampler import _prune_hidden_states, _apply_logits_processors, _apply_penalties, _apply_top_k_top_p, _apply_min_p, _apply_penalties, _sample, _get_logprobs, _build_sampler_output + + +@contextlib.contextmanager +def _set_default_torch_dtype(dtype: torch.dtype): + """Sets the default torch dtype to the given dtype.""" + old_dtype = torch.get_default_dtype() + torch.set_default_dtype(dtype) + yield + torch.set_default_dtype(old_dtype) + + +def _get_model_architecture(config: PretrainedConfig) -> Type[nn.Module]: + architectures = getattr(config, "architectures", []) + for arch in architectures: + model_cls = ModelRegistry.load_model_cls(arch) + if model_cls is not None: + return model_cls + raise ValueError(f"Model architectures {architectures} are not supported for now. " + f"Supported architectures: {ModelRegistry.get_supported_archs()}") + + +from vllm.model_executor.layers.linear import * +from vllm.model_executor.layers.vocab_parallel_embedding import VocabParallelEmbedding, ParallelLMHead +from vllm.model_executor.layers.activation import ScaledActivation + +__LAYER_WEIGHT_LOADER_REGISTRY__ = { + ColumnParallelLinear: parallel_weight_loader, + MergedColumnParallelLinear: parallel_weight_loader, + QKVParallelLinear: parallel_weight_loader, + RowParallelLinear: parallel_weight_loader, + VocabParallelEmbedding: parallel_weight_loader, + ParallelLMHead: parallel_weight_loader + # "ScaledActivation.weight_loader": ScaledActivation, # TODO(shengguangming): latest commit in vllm fix awq for this function and add load_weights + # "default_weight_loader": default_weight_loader +} + +# NOTE(gmsheng): change the weight_loader function in runtime +for layer_class, weight_loader in __LAYER_WEIGHT_LOADER_REGISTRY__.items(): + layer_class.weight_loader = weight_loader + +__MODEL_WEIGHT_LOADER_REGISTRY__ = { + 'GPT2LMHeadModel': gpt2_weight_loader, + 'LlamaForCausalLM': llama_weight_loader, + 'LLaMAForCausalLM': llama_weight_loader, + 'MistralForCausalLM': mistral_weight_loader, +} + +# FIXME(shengguangming): the vLLM vocab will pad to 64, which may incur out of bounds +# so we need to rewrite the init function of vocab +DEFAULT_VOCAB_PADDING_SIZE = 64 + + +def vocab_init(self, + num_embeddings: int, + embedding_dim: int, + params_dtype: Optional[torch.dtype] = None, + org_num_embeddings: Optional[int] = None, + padding_size: int = DEFAULT_VOCAB_PADDING_SIZE): + super(VocabParallelEmbedding, self).__init__() + + # Keep the input dimensions. + # TODO (pad to be divided by 4) + self.num_embeddings = num_embeddings + self.org_vocab_size = org_num_embeddings or num_embeddings + + # self.num_embeddings_padded = pad_vocab_size(num_embeddings, + # padding_size) + self.embedding_dim = embedding_dim + if params_dtype is None: + params_dtype = torch.get_default_dtype() + self.tp_size = get_tensor_model_parallel_world_size() + # Divide the weight matrix along the vocaburaly dimension. + + self.vocab_start_index, self.vocab_end_index = (VocabUtility.vocab_range_from_global_vocab_size( + self.num_embeddings, get_tensor_model_parallel_rank(), self.tp_size)) + self.num_embeddings_per_partition = (self.vocab_end_index - self.vocab_start_index) + self.weight = Parameter( + torch.empty( + self.num_embeddings_per_partition, + self.embedding_dim, + # device=torch.cuda.current_device(), + dtype=params_dtype)) + set_weight_attrs(self.weight, {"parallel_dim": 0, "weight_loader": self.weight_loader}) + + +VocabParallelEmbedding.__init__ = vocab_init + + +def _get_model_weight_loader(arch: str): + if arch in __MODEL_WEIGHT_LOADER_REGISTRY__: + return __MODEL_WEIGHT_LOADER_REGISTRY__[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {ModelRegistry.get_supported_archs()}") + + +def get_model(actor_model: Union[PreTrainedModel, Dict], + model_config: ModelConfig, + device_config: DeviceConfig, + lora_config: Optional[LoRAConfig] = None) -> nn.Module: + model_class = _get_model_architecture(model_config.hf_config) + + # Get the quantization config. + linear_method = None + quant_config = None + if model_config.quantization is not None: + quant_config = get_quant_config(model_config.quantization, model_config.model, model_config.hf_config, + model_config.download_dir) + capability = torch.cuda.get_device_capability() + capability = capability[0] * 10 + capability[1] + if capability < quant_config.get_min_capability(): + raise ValueError(f"The quantization method {model_config.quantization} is not " + "supported for the current GPU. " + f"Minimum capability: {quant_config.get_min_capability()}. " + f"Current capability: {capability}.") + supported_dtypes = quant_config.get_supported_act_dtypes() + if model_config.dtype not in supported_dtypes: + raise ValueError(f"{model_config.dtype} is not supported for quantization " + f"method {model_config.quantization}. Supported dtypes: " + f"{supported_dtypes}") + linear_method = quant_config.get_linear_method() + + with _set_default_torch_dtype(model_config.dtype): + # Create a model instance. + # The weights will be initialized as empty tensors. + # with torch.device(device_config.device): + # NOTE(sgm): init the model in cpu + model = model_class(model_config.hf_config, linear_method) + + if model_config.load_format == "dummy": + model = model.cuda() + # NOTE(woosuk): For accurate performance evaluation, we assign + # random values to the weights. + initialize_dummy_weights(model) + elif model_config.load_format == 'model' or model_config.load_format == 'auto': + # NOTE(shengguangming) Load the weights from the actor model + if isinstance(actor_model, nn.Module): + load_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), vllm_model=model) + else: + load_weights(actor_weights=actor_model, vllm_model=model) + + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +# the actor model is .state_dict() +def load_weights(actor_weights: Dict, vllm_model: nn.Module): + weight_loader = _get_model_weight_loader(vllm_model.__class__.__name__) + weight_loader(actor_weights, vllm_model) + # NOTE(sgm) to reduce peak memory usage, we offload vllm model to cpu + # after init, and we need this after sync model weights for in first iter. + vllm_model = vllm_model.cuda() + + +# FIXME(sgm): hack the Sampler function in vllm v0.3.1 +# as they use ray, the sampler result will only need to return to the driver node, +# therefore gather is enough. However, we use SPMD instead of a central scheduler, +# all_gather is required (aligned with v0.2.6) +def _get_logits(self, hidden_states: torch.Tensor, embedding: torch.Tensor, + embedding_bias: Optional[torch.Tensor]) -> torch.Tensor: + # Get the logits for the next tokens. + logits = torch.matmul(hidden_states, embedding.t()) + if embedding_bias is not None: + logits += embedding_bias + logits = tensor_model_parallel_all_gather(logits) + # Remove paddings in vocab (if any). + if logits is not None: + logits = logits[:, :self.org_vocab_size] + return logits + + +def forward( + self, + embedding: torch.Tensor, + hidden_states: torch.Tensor, + sampling_metadata: SamplingMetadata, + embedding_bias: Optional[torch.Tensor] = None, +) -> Optional[SamplerOutput]: + # Get the hidden states that we use for sampling. + hidden_states = _prune_hidden_states(hidden_states, sampling_metadata) + + # Get the logits for the next tokens. + logits = self._get_logits(hidden_states, embedding, embedding_bias) + # save origin logprobs for sampler_output + origin_logprobs = torch.log_softmax(logits, dim=-1, dtype=torch.float) + + # Only perform sampling in the driver worker. + # Note: `_get_logits` is still distributed across TP workers because + # the `embedding` weight is distributed across TP workers. + # TODO(zhuohan): Change the get_logits part to a separate stage. + if not sampling_metadata.perform_sampling: + return None + + assert logits is not None + _, vocab_size = logits.shape + + # Apply logits processors (if any). + logits = _apply_logits_processors(logits, sampling_metadata) + + # Prepare sampling tensors with pinned memory to avoid blocking. + (sampling_tensors, do_penalties, do_top_p_top_k, + do_min_p) = SamplingTensors.from_sampling_metadata(sampling_metadata, vocab_size, logits.device, logits.dtype) + + # Apply presence and frequency penalties. + if do_penalties: + logits = _apply_penalties(logits, sampling_tensors.prompt_tokens, sampling_tensors.output_tokens, + sampling_tensors.presence_penalties, sampling_tensors.frequency_penalties, + sampling_tensors.repetition_penalties) + + # Apply temperature scaling. + # Use in-place division to avoid creating a new tensor. + logits.div_(sampling_tensors.temperatures.unsqueeze_(dim=1)) + + if do_top_p_top_k: + logits = _apply_top_k_top_p(logits, sampling_tensors.top_ps, sampling_tensors.top_ks) + + if do_min_p: + logits = _apply_min_p(logits, sampling_tensors.min_ps) + + # We use float32 for probabilities and log probabilities. + # Compute the probabilities. + probs = torch.softmax(logits, dim=-1, dtype=torch.float) + # Compute the log probabilities. + # Use log_softmax to ensure numerical stability. + logprobs = torch.log_softmax(logits, dim=-1, dtype=torch.float) + + # Sample the next tokens. + sample_results = _sample(probs, logprobs, sampling_metadata) + + # Get the logprobs query results. + # prompt_logprobs, sample_logprobs = _get_logprobs( + # logprobs, sampling_metadata, sample_results) + prompt_logprobs, sample_logprobs = _get_logprobs(origin_logprobs, sampling_metadata, sample_results) + + return _build_sampler_output(sample_results, sampling_metadata, prompt_logprobs, sample_logprobs) + + +from vllm.model_executor.layers.sampler import Sampler + +Sampler._get_logits = _get_logits +Sampler.forward = forward diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/model_runner.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/model_runner.py new file mode 100644 index 0000000000000000000000000000000000000000..4acf3422d43c16091977f598111926d636cc3e29 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/model_runner.py @@ -0,0 +1,285 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/worker/model_runner.py + +from typing import Dict, List, Optional, Tuple, Set, Union +import contextlib +import time +import numpy as np +import torch +import torch.nn as nn + +from vllm.config import (DeviceConfig, ModelConfig, LoRAConfig, ParallelConfig, SchedulerConfig) +from vllm.logger import init_logger +from vllm.model_executor import InputMetadata, SamplingMetadata +from vllm.sampling_params import SamplingParams, SamplingType +from vllm.sequence import SamplerOutput, SequenceData, SequenceGroupMetadata +from vllm.lora.worker_manager import LRUCacheWorkerLoRAManager +from vllm.lora.layers import LoRAMapping +from vllm.lora.request import LoRARequest +from vllm.utils import in_wsl +from vllm.worker.model_runner import ModelRunner, CUDAGraphRunner, _async_h2d + +from .model_loader import get_model + +logger = init_logger(__name__) + +KVCache = Tuple[torch.Tensor, torch.Tensor] +_PAD_SLOT_ID = -1 +LORA_WARMUP_RANK = 8 +# Capture graphs for batch size 1, 2, 4, 8, 16, 24, 32, 40, ..., 256. +# NOTE: _get_graph_batch_size needs to be updated if this list is changed. +_BATCH_SIZES_TO_CAPTURE = [1, 2, 4] + [8 * i for i in range(1, 33)] + + +class ModelRunner(ModelRunner): + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + model_config: ModelConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], + kv_cache_dtype: Optional[str] = "auto", + ): + self.model_config = model_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.lora_config = lora_config + + # model_config can be None in tests/samplers/test_sampler.py. + # FIXME(woosuk): This is a hack to make the tests work. Refactor this. + self.sliding_window = (model_config.get_sliding_window() if model_config is not None else None) + + self.device_config = (device_config if device_config is not None else DeviceConfig()) + self.device = self.device_config.device + + self.model = model # this will be replaced by get_model() + self.block_size = None # Set after initial profiling. + self.lora_manager = None + + self.graph_runners: Dict[int, CUDAGraphRunner] = {} + self.graph_memory_pool = None # Set during graph capture. + + self.max_context_len_to_capture = (self.model_config.max_context_len_to_capture + if self.model_config is not None else 0) + # When using CUDA graph, the input block tables must be padded to + # max_context_len_to_capture. However, creating the block table in + # Python can be expensive. To optimize this, we cache the block table + # in numpy and only copy the actual input content at every iteration. + # The shape of the cached block table will be + # (max batch size to capture, max context len to capture / block size). + self.graph_block_tables = None # Set after initial profiling. + # cache in_wsl result + self.in_wsl = in_wsl() + self.kv_cache_dtype = kv_cache_dtype + + def load_model(self) -> None: + self.model = get_model(actor_model=self.model, + model_config=self.model_config, + device_config=self.device_config, + lora_config=self.lora_config) + vocab_size = self.model.config.vocab_size + + if self.lora_config: + assert hasattr( + self.model, + "supported_lora_modules") and self.model.supported_lora_modules, "Model does not support LoRA" + assert hasattr(self.model, "embedding_modules"), "Model does not have embedding_modules" + assert hasattr(self.model, "embedding_padding_modules"), "Model does not have embedding_padding_modules" + self.lora_manager = LRUCacheWorkerLoRAManager( + self.scheduler_config.max_num_seqs, + self.scheduler_config.max_num_batched_tokens + self.scheduler_config.max_paddings, vocab_size, + self.lora_config, self.device, self.model.embedding_modules, self.model.embedding_padding_modules) + self.model = self.lora_manager.create_lora_manager(self.model) + + def _prepare_sample( + self, + seq_group_metadata_list: List[SequenceGroupMetadata], + prompt_lens: List[int], + subquery_lens: Optional[List[int]], + ) -> SamplingMetadata: + seq_groups: List[Tuple[List[int], SamplingParams]] = [] + selected_token_indices: List[int] = [] + selected_token_start_idx = 0 + categorized_sample_indices = {t: [] for t in SamplingType} + categorized_sample_indices_start_idx = 0 + + max_subquery_len = max(subquery_lens) if subquery_lens else 1 + for i, seq_group_metadata in enumerate(seq_group_metadata_list): + seq_ids = list(seq_group_metadata.seq_data.keys()) + sampling_params = seq_group_metadata.sampling_params + seq_groups.append((seq_ids, sampling_params)) + + if seq_group_metadata.is_prompt: + assert len(seq_ids) == 1 + assert subquery_lens is not None + subquery_len = subquery_lens[i] + if sampling_params.prompt_logprobs is not None: + # NOTE: prompt token positions do not need sample, skip + categorized_sample_indices_start_idx += subquery_len - 1 + + categorized_sample_indices[sampling_params.sampling_type].append(categorized_sample_indices_start_idx) + categorized_sample_indices_start_idx += 1 + + if sampling_params.prompt_logprobs is not None: + selected_token_indices.extend( + range(selected_token_start_idx, selected_token_start_idx + subquery_len - 1)) + selected_token_indices.append(selected_token_start_idx + subquery_len - 1) + selected_token_start_idx += max_subquery_len + else: + num_seqs = len(seq_ids) + selected_token_indices.extend(range(selected_token_start_idx, selected_token_start_idx + num_seqs)) + selected_token_start_idx += num_seqs + + categorized_sample_indices[sampling_params.sampling_type].extend( + range(categorized_sample_indices_start_idx, categorized_sample_indices_start_idx + num_seqs)) + categorized_sample_indices_start_idx += num_seqs + + selected_token_indices = _async_h2d(selected_token_indices, + dtype=torch.long, + target_device=self.device, + pin_memory=not self.in_wsl) + categorized_sample_indices = { + t: _async_h2d(seq_ids, dtype=torch.int, target_device=self.device, pin_memory=not self.in_wsl) + for t, seq_ids in categorized_sample_indices.items() + } + + seq_data: Dict[int, SequenceData] = {} + for seq_group_metadata in seq_group_metadata_list: + seq_data.update(seq_group_metadata.seq_data) + + sampling_metadata = SamplingMetadata( + seq_groups=seq_groups, + seq_data=seq_data, + prompt_lens=prompt_lens, + selected_token_indices=selected_token_indices, + categorized_sample_indices=categorized_sample_indices, + ) + return sampling_metadata + + def prepare_input_tensors( + self, + seq_group_metadata_list: Optional[List[SequenceGroupMetadata]], + ) -> Tuple[torch.Tensor, torch.Tensor, InputMetadata, SamplingMetadata, Set[int], LoRAMapping]: + # NOTE: We assume that all sequences in the group are all prompts or + # all decodes. + is_prompt = seq_group_metadata_list[0].is_prompt + # Prepare input tensors. + if is_prompt: + (input_tokens, input_positions, input_metadata, prompt_lens, subquery_lens, lora_index_mapping, + lora_prompt_mapping, lora_requests) = self._prepare_prompt(seq_group_metadata_list) + else: + (input_tokens, input_positions, input_metadata, lora_index_mapping, lora_prompt_mapping, + lora_requests) = self._prepare_decode(seq_group_metadata_list) + prompt_lens = [] + subquery_lens = None + sampling_metadata = self._prepare_sample(seq_group_metadata_list, prompt_lens, subquery_lens) + if self.lora_config: + flat_lora_index_mapping = [item for sublist in lora_index_mapping for item in sublist] + lora_mapping = LoRAMapping( + flat_lora_index_mapping, + lora_prompt_mapping, + ) + else: + lora_mapping = None + + return (input_tokens, input_positions, input_metadata, sampling_metadata, lora_requests, lora_mapping) + + @torch.inference_mode() + def execute_model( + self, + seq_group_metadata_list: Optional[List[SequenceGroupMetadata]], + kv_caches: List[Tuple[torch.Tensor, torch.Tensor]], + ) -> Optional[SamplerOutput]: + (input_tokens, input_positions, input_metadata, sampling_metadata, lora_requests, + lora_mapping) = self.prepare_input_tensors(seq_group_metadata_list) + + if self.lora_config: + self.set_active_loras(lora_requests, lora_mapping) + + # Execute the model. + if input_metadata.use_cuda_graph: + graph_batch_size = input_tokens.shape[0] + model_executable = self.graph_runners[graph_batch_size] + else: + model_executable = self.model + hidden_states = model_executable( + input_ids=input_tokens, + positions=input_positions, + kv_caches=kv_caches, + input_metadata=input_metadata, + ) + + # Sample the next token. + output = self.model.sample( + hidden_states=hidden_states, + sampling_metadata=sampling_metadata, + ) + return output + + @torch.inference_mode() + def profile_run(self) -> None: + # Enable top-k sampling to reflect the accurate memory usage. + vocab_size = self.model_config.get_vocab_size() + # FIXME(sgm): this sampling params will call cumsum(), causing the + # deterministic cumsum throw error + sampling_params = SamplingParams(top_p=0.99, top_k=vocab_size - 1) + max_num_batched_tokens = self.scheduler_config.max_num_batched_tokens + max_num_seqs = self.scheduler_config.max_num_seqs + + # This represents the maximum number of different requests + # that will have unique loras, an therefore the max amount of memory + # consumption create dummy lora request copies from the lora request + # passed in, which contains a lora from the lora warmup path. + dummy_lora_requests = [] + dummy_lora_requests_per_seq = [] + if self.lora_config: + for idx in range(self.lora_config.max_loras): + lora_id = idx + 1 + dummy_lora_request = LoRARequest( + lora_name=f"warmup_{lora_id}", + lora_int_id=lora_id, + lora_local_path="/not/a/real/path", + ) + self.lora_manager.add_dummy_lora(dummy_lora_request, rank=LORA_WARMUP_RANK) + dummy_lora_requests.append(dummy_lora_request) + dummy_lora_requests_per_seq = [ + dummy_lora_requests[idx % len(dummy_lora_requests)] for idx in range(max_num_seqs) + ] + + # Profile memory usage with max_num_sequences sequences and the total + # number of tokens equal to max_num_batched_tokens. + seqs: List[SequenceGroupMetadata] = [] + for group_id in range(max_num_seqs): + seq_len = (max_num_batched_tokens // max_num_seqs + (group_id < max_num_batched_tokens % max_num_seqs)) + seq_data = SequenceData([0] * seq_len) + seq = SequenceGroupMetadata( + request_id=str(group_id), + is_prompt=True, + seq_data={group_id: seq_data}, + sampling_params=sampling_params, + block_tables=None, + lora_request=dummy_lora_requests_per_seq[group_id] if dummy_lora_requests_per_seq else None, + ) + seqs.append(seq) + + # Run the model with the dummy inputs. + num_layers = self.model_config.get_num_layers(self.parallel_config) + kv_caches = [(None, None)] * num_layers + self.execute_model(seqs, kv_caches) + torch.cuda.synchronize() + return diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/parallel_state.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/parallel_state.py new file mode 100644 index 0000000000000000000000000000000000000000..c3b7a45c8d6b8a62efd5100f27a00c399ec4e9e6 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/parallel_state.py @@ -0,0 +1,147 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Adapted from +# https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/parallel_state.py +# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. +"""Model and data parallel groups.""" + +import torch +import torch.distributed + +import vllm.model_executor.parallel_utils.parallel_state as ps +""" +This version is strongly tied with Megatron to implement HybridEngine and weight sharing between vllm and Megatron. +- We assume the Megatron tp+dp+pp world is already established before calling this function. + +""" + +# Tensor model parallel group that the current rank belongs to. +_TENSOR_MODEL_PARALLEL_GROUP = None + +# Micro Data parallel group. Micro data parallel group is additional dp group that origins from splitting training tp +# into infer_tp and micro_tp. By default, we use order micro_dp - tp +_MICRO_DATA_PARALLEL_GROUP = None + + +def initialize_model_parallel_from_megatron( + tensor_model_parallel_size=None # we set None for backward compatibility to set infer_tp = train_tp +) -> None: + from megatron.core import parallel_state as mpu + from megatron.distributed import new_group + # Get world size and rank. Ensure some consistencies. + assert torch.distributed.is_initialized() + + if tensor_model_parallel_size is None: + tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size() + else: + assert isinstance(tensor_model_parallel_size, int) + + # Build the tensor model-parallel groups. + assert ps._TENSOR_MODEL_PARALLEL_GROUP is None, ("tensor model parallel group is already initialized") + + assert tensor_model_parallel_size <= mpu.get_tensor_model_parallel_world_size( + ), 'Not implemented for infer_tp > train_tp' + + global _TENSOR_MODEL_PARALLEL_GROUP + global _MICRO_DATA_PARALLEL_GROUP + + assert mpu.get_tensor_model_parallel_world_size() % tensor_model_parallel_size == 0 + + micro_dp_size = mpu.get_tensor_model_parallel_world_size() // tensor_model_parallel_size + + world_size: int = torch.distributed.get_world_size() + + num_micro_dp_groups = world_size // micro_dp_size + + rank = torch.distributed.get_rank() + + # Build the micro dp groups. + assert _MICRO_DATA_PARALLEL_GROUP is None, ("micro data parallel group is already initialized") + for i in range(num_micro_dp_groups): + ranks = range(i * micro_dp_size, (i + 1) * micro_dp_size) + group = new_group(rank=rank, ranks=ranks, group_type='micro_dp') + if rank in ranks: + _MICRO_DATA_PARALLEL_GROUP = group + + if tensor_model_parallel_size == mpu.get_tensor_model_parallel_world_size(): + # using the same tp group as Megatron + ps._TENSOR_MODEL_PARALLEL_GROUP = mpu.get_tensor_model_parallel_group() + + _TENSOR_MODEL_PARALLEL_GROUP = mpu.get_tensor_model_parallel_group() + # no _MICRO_DATA_PARALLEL_GROUP + else: + # initialize a micro_dp group and a tp group + # assume training tp=4, infer tp=2, then, weight is partitioned as + # [1], [2], [3], [4] for training and [1,2], [1,2], [3,4], [3,4] for inference + + # Build the inference tp groups + train_tp = mpu.get_tensor_model_parallel_world_size() + num_tensor_model_parallel_groups_per_train_tp = train_tp // tensor_model_parallel_size + num_tensor_model_parallel_groups = world_size // tensor_model_parallel_size + assert _TENSOR_MODEL_PARALLEL_GROUP is None, ("tensor model parallel group is already initialized") + for i in range(num_tensor_model_parallel_groups // num_tensor_model_parallel_groups_per_train_tp): + start = train_tp * i + end = train_tp * (i + 1) + for j in range(num_tensor_model_parallel_groups_per_train_tp): + ranks = list(range(start, end, num_tensor_model_parallel_groups_per_train_tp)) + for i in range(len(ranks)): + ranks[i] += j + # group = torch.distributed.new_group(ranks) + group = new_group(rank=rank, ranks=ranks, group_type='infer_tp') + if rank in ranks: + _TENSOR_MODEL_PARALLEL_GROUP = group + ps._TENSOR_MODEL_PARALLEL_GROUP = _TENSOR_MODEL_PARALLEL_GROUP + # Build the pipeline model-parallel groups. + # global _PIPELINE_MODEL_PARALLEL_GROUP + # global _PIPELINE_GLOBAL_RANKS + # assert ps._PIPELINE_MODEL_PARALLEL_GROUP is None, ("pipeline model parallel group is already initialized") + + # ps._PIPELINE_MODEL_PARALLEL_GROUP = mpu.get_pipeline_model_parallel_group() + # ps._PIPELINE_GLOBAL_RANKS = mpu.get_pipeline_model_parallel_ranks() + + +""" +Tensor model parallel utilities +""" + + +def get_tensor_model_parallel_group(): + """Get the tensor model parallel group the caller rank belongs to.""" + assert _TENSOR_MODEL_PARALLEL_GROUP is not None, ("tensor model parallel group is not initialized") + return _TENSOR_MODEL_PARALLEL_GROUP + + +def get_tensor_model_parallel_world_size(): + """Return world size for the tensor model parallel group.""" + return torch.distributed.get_world_size(group=get_tensor_model_parallel_group()) + + +def get_tensor_model_parallel_rank(): + """Return my rank for the tensor model parallel group.""" + return torch.distributed.get_rank(group=get_tensor_model_parallel_group()) + + +def get_tensor_model_parallel_src_rank(): + """Calculate the global rank corresponding to the first local rank + in the tensor model parallel group.""" + global_rank = torch.distributed.get_rank() + local_world_size = get_tensor_model_parallel_world_size() + return (global_rank // local_world_size) * local_world_size + + +""" +Micro Data parallel group +""" + + +def get_micro_data_parallel_group(): + assert _MICRO_DATA_PARALLEL_GROUP is not None + return _MICRO_DATA_PARALLEL_GROUP + + +def get_micro_data_parallel_world_size(): + return torch.distributed.get_world_size(group=get_micro_data_parallel_group()) + + +def get_micro_data_parallel_rank(): + return torch.distributed.get_rank(group=get_micro_data_parallel_group()) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/tokenizer.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/tokenizer.py new file mode 100644 index 0000000000000000000000000000000000000000..b8de24afb834af4e5c8d60b006e0696206519315 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/tokenizer.py @@ -0,0 +1,72 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/tokenizer_group/tokenizer_group.py + +from typing import List, Optional, Tuple, Union + +from transformers import (AutoTokenizer, PreTrainedTokenizer, PreTrainedTokenizerFast) + +from vllm.lora.request import LoRARequest +from vllm.utils import make_async, LRUCache +from vllm.transformers_utils.tokenizers import * + + +class TokenizerGroup: + """A group of tokenizers that can be used for LoRA adapters.""" + + def __init__(self, tokenizer: PreTrainedTokenizer, enable_lora: bool, max_num_seqs: int, + max_input_length: Optional[int]): + self.enable_lora = enable_lora + self.max_input_length = max_input_length + self.tokenizer = tokenizer + if enable_lora: + self.lora_tokenizers = LRUCache(capacity=max_num_seqs) + else: + self.lora_tokenizers = None + + def encode(self, + prompt: str, + request_id: Optional[str] = None, + lora_request: Optional[LoRARequest] = None) -> List[int]: + tokenizer = self.get_lora_tokenizer(lora_request) + return tokenizer.encode(prompt) + + async def encode_async(self, + prompt: str, + request_id: Optional[str] = None, + lora_request: Optional[LoRARequest] = None) -> List[int]: + tokenizer = await self.get_lora_tokenizer_async(lora_request) + return tokenizer.encode(prompt) + + def get_lora_tokenizer(self, lora_request: Optional[LoRARequest]) -> "PreTrainedTokenizer": + if not lora_request or not self.enable_lora: + return self.tokenizer + if lora_request.lora_int_id not in self.lora_tokenizers: + # TODO(sgm): the lora tokenizer is also passed, but may be different + tokenizer = self.tokenizer + # tokenizer = (get_lora_tokenizer( + # lora_request, **self.tokenizer_config) or self.tokenizer) + self.lora_tokenizers.put(lora_request.lora_int_id, tokenizer) + return tokenizer + else: + return self.lora_tokenizers.get(lora_request.lora_int_id) + + # FIXME(sgm): for simplicity, we assign the special token here + @property + def pad_token_id(self): + return self.tokenizer.pad_token_id + + @property + def eos_token_id(self): + return self.tokenizer.eos_token_id diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/weight_loaders.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/weight_loaders.py new file mode 100644 index 0000000000000000000000000000000000000000..72aa26d06013f5bf29e67bedfcb77fc0af80e1c7 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/weight_loaders.py @@ -0,0 +1,95 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models + +from typing import Dict +import torch +import torch.nn as nn + + +# NOTE(shengguangming): replace the origin weight loader function in the class +def parallel_weight_loader(self, param: torch.Tensor, loaded_weight: torch.Tensor) -> None: + """Parallel Linear weight loader.""" + assert param.size() == loaded_weight.size( + ), 'the parameter size is not align with the loaded weight size, param size: {}, loaded_weight size: {}'.format( + param.size(), loaded_weight.size()) + assert param.data.dtype == loaded_weight.data.dtype, "if we want to shared weights, the data type should also be the same" + + param.data = loaded_weight.data + + +def default_weight_loader(param: torch.Tensor, loaded_weight: torch.Tensor) -> None: + """Default weight loader.""" + assert param.size() == loaded_weight.size() + assert param.data.dtype == loaded_weight.data.dtype, "if we want to shared weights, the data type should also be the same" + + param.data = loaded_weight.data + + +def gpt2_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "lm_head.weight" in name: + # GPT-2 ties the weights of the embedding layer and the final + # linear layer. + continue + if ".attn.bias" in name or ".attn.masked_bias" in name: + # Skip attention mask. + # NOTE: "c_attn.bias" should not be skipped. + continue + if not name.startswith("transformer."): + name = "transformer." + name + param = params_dict[name] + # The HF's GPT-2 implementation uses Conv1D instead of Linear. + # Because of this, we need to transpose the weights. + # Note(zhuohan): the logic below might break quantized models. + for conv1d_weight_name in ["c_attn", "c_proj", "c_fc"]: + if conv1d_weight_name not in name: + continue + if not name.endswith(".weight"): + continue + # TODO: check megatron + loaded_weight = loaded_weight.t() + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + # NOTE(shengguangming): the megatron llama may have this prefix + prefix = '0.module.module.' + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if name[:len(prefix)] == prefix: + name = name[len(prefix):] + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def mistral_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + # TODO: need to implement a general way to deal with prefix + prefix = '0.module.module.' + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if name[:len(prefix)] == prefix: + name = name[len(prefix):] + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/worker.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/worker.py new file mode 100644 index 0000000000000000000000000000000000000000..50eebd70b86c1160896ded81deeb7e6eedd6d605 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_3_1/worker.py @@ -0,0 +1,314 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/worker/worker.py +"""A GPU worker class.""" +import os +import gc +from typing import Dict, List, Tuple, Optional, Union, Set + +import torch +import torch.distributed +import torch.nn as nn + +from vllm.config import (CacheConfig, DeviceConfig, ModelConfig, ParallelConfig, SchedulerConfig, LoRAConfig) +from vllm.model_executor import InputMetadata, set_random_seed +from vllm.model_executor.parallel_utils.parallel_state import (initialize_model_parallel) +from vllm.sampling_params import SamplingParams, SamplingType +from vllm.sequence import SamplerOutput, SequenceData, SequenceGroupMetadata +from vllm.worker.cache_engine import CacheEngine +from vllm.model_executor.parallel_utils.custom_all_reduce import init_custom_ar +from vllm.model_executor.parallel_utils.parallel_state import get_tensor_model_parallel_group + +from .model_runner import ModelRunner +from .model_loader import load_weights +from .parallel_state import initialize_model_parallel_from_megatron +from vllm.lora.request import LoRARequest + + +class Worker: + """A worker class that executes (a partition of) the model on a GPU. + + Each worker is associated with a single GPU. The worker is responsible for + maintaining the KV cache and executing the model on the GPU. In case of + distributed inference, each worker is assigned a partition of the model. + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + model_config: ModelConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + rank: Optional[int] = None, + distributed_init_method: Optional[str] = None, + lora_config: Optional[LoRAConfig] = None, + kv_cache_dtype: Optional[str] = "auto", + ) -> None: + # self.model = model # will be replaced in the init_model + self.model_config = model_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.rank = rank + self.distributed_init_method = distributed_init_method + self.lora_config = lora_config + + self.model_runner = ModelRunner( + model, + model_config, + parallel_config, + scheduler_config, + device_config, + lora_config=self.lora_config, + kv_cache_dtype=kv_cache_dtype, + ) + + # Uninitialized cache engine. Will be initialized by + # self.init_cache_engine(). + self.cache_config = None + self.block_size = None + self.sliding_window = None + self.cache_engine = None + self.cache_events = None + self.gpu_cache = None + + # For offloading inference engine params + self.cpu_model = None + + def init_model(self, cupy_port: Optional[int] = None): + # torch.distributed.all_reduce does not free the input tensor until + # the synchronization point. This causes the memory usage to grow + # as the number of all_reduce calls increases. This env var disables + # this behavior. + # Related issue: + # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573 + os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1" + + # Env vars will be set by TORCHRUN. + self.rank = self.rank if self.rank is not None else int(os.getenv("RANK", "-1")) + local_rank = int(os.getenv("LOCAL_RANK", "0")) + self.device = torch.device(f"cuda:{local_rank}") + if self.rank < 0: + raise ValueError("Invalid or unspecified rank.") + torch.cuda.set_device(self.device) + + _check_if_gpu_supports_dtype(self.model_config.dtype) + + # Initialize the distributed environment. + # TODO: do not use cupy + _init_distributed_environment(self.parallel_config, self.rank, self.distributed_init_method) + if not self.parallel_config.disable_custom_all_reduce: + init_custom_ar() + # Initialize the model. + set_random_seed(self.model_config.seed) + # self.model = get_model(actor_model=self.model, model_config=self.model_config) + + def load_model(self): + self.model_runner.load_model() + + @torch.inference_mode() + def profile_num_available_blocks( + self, + block_size: int, + gpu_memory_utilization: float, + cpu_swap_space: int, + cache_dtype: str, + ) -> Tuple[int, int]: + # Profile the memory usage of the model and get the maximum number of + # cache blocks that can be allocated with the remaining free memory. + torch.cuda.empty_cache() + # torch.cuda.reset_peak_memory_stats() + + # Execute a forward pass with dummy inputs to profile the memory usage + # of the model. + self.model_runner.profile_run() + + # Calculate the number of blocks that can be allocated with the + # profiled peak memory. + torch.cuda.synchronize() + free_gpu_memory, total_gpu_memory = torch.cuda.mem_get_info() + peak_memory = total_gpu_memory - free_gpu_memory + + cache_block_size = CacheEngine.get_cache_block_size(block_size, cache_dtype, self.model_config, + self.parallel_config) + # NOTE(sgm) use the remaining memory + num_gpu_blocks = int((free_gpu_memory * gpu_memory_utilization) // cache_block_size) + # num_gpu_blocks = int((total_gpu_memory * gpu_memory_utilization - peak_memory) // cache_block_size) + num_cpu_blocks = int(cpu_swap_space // cache_block_size) + num_gpu_blocks = max(num_gpu_blocks, 0) + num_cpu_blocks = max(num_cpu_blocks, 0) + if self.model_runner.lora_manager: + self.model_runner.remove_all_loras() + gc.collect() + torch.cuda.empty_cache() + # Synchronize number of blocks with all the rank + num_gpu_blocks = torch.tensor([num_gpu_blocks], device='cuda') + num_cpu_blocks = torch.tensor([num_cpu_blocks], device='cuda') + torch.distributed.all_reduce(num_gpu_blocks, + op=torch.distributed.ReduceOp.MIN, + group=get_tensor_model_parallel_group()) + torch.distributed.all_reduce(num_cpu_blocks, + op=torch.distributed.ReduceOp.MIN, + group=get_tensor_model_parallel_group()) + num_gpu_blocks = num_gpu_blocks.item() + num_cpu_blocks = num_cpu_blocks.item() + return num_gpu_blocks, num_cpu_blocks + + def init_cache_engine(self, cache_config: CacheConfig) -> None: + if self.cache_engine is None and self.gpu_cache is None: + self.cache_config = cache_config + self.cache_engine = CacheEngine(self.cache_config, self.model_config, self.parallel_config) + self.cache_events = self.cache_engine.events + self.gpu_cache = self.cache_engine.gpu_cache + self.model_runner.set_block_size(self.cache_engine.block_size) + + def free_cache_engine(self): + # ensure `enforce_eager=True` + self.cache_engine = None + self.gpu_cache = None + + def warm_up_model(self) -> None: + if not self.model_config.enforce_eager: + self.model_runner.capture_model(self.gpu_cache) + # Reset the seed to ensure that the random state is not affected by + # the model initialization and profiling. + set_random_seed(self.model_config.seed) + + def cache_swap( + self, + blocks_to_swap_in: Dict[int, int], + blocks_to_swap_out: Dict[int, int], + blocks_to_copy: Dict[int, List[int]], + ) -> None: + # Issue cache operations. + issued_cache_op = False + if blocks_to_swap_in: + self.cache_engine.swap_in(blocks_to_swap_in) + issued_cache_op = True + if blocks_to_swap_out: + self.cache_engine.swap_out(blocks_to_swap_out) + issued_cache_op = True + if blocks_to_copy: + self.cache_engine.copy(blocks_to_copy) + issued_cache_op = True + + cache_events = self.cache_events if issued_cache_op else None + + # Wait for cache operations to finish. + # TODO(woosuk): Profile swapping overhead and optimize if needed. + if cache_events is not None: + for event in cache_events: + event.wait() + + @torch.inference_mode() + def execute_model( + self, + seq_group_metadata_list: List[SequenceGroupMetadata], + blocks_to_swap_in: Dict[int, int], + blocks_to_swap_out: Dict[int, int], + blocks_to_copy: Dict[int, List[int]], + ) -> SamplerOutput: + num_seq_groups = len(seq_group_metadata_list) + self.cache_swap(blocks_to_swap_in, blocks_to_swap_out, blocks_to_copy) + + # If there is no input, we don't need to execute the model. + if num_seq_groups == 0: + return {} + output = self.model_runner.execute_model(seq_group_metadata_list, self.gpu_cache) + return output + + # # Prepare input tensors. + # # NOTE(shengguangming): currently we pad in our dataloader and unpad it in pre_process_input, j + # # we can just input un-padded sequence for better performance + # input_tokens, input_positions, input_metadata = self._prepare_inputs(seq_group_metadata_list) + + # # Execute the model. + # output = self.model( + # input_ids=input_tokens, + # positions=input_positions, + # kv_caches=self.gpu_cache, + # input_metadata=input_metadata, + # cache_events=cache_events, + # ) + # return output + + # assume the input is .state_dict() + def sync_model_weights(self, actor_weights: Dict): + load_weights(actor_weights, self.model_runner.model) + + def offload_model_weights(self) -> None: + if self.cpu_model == None: + self.cpu_model = {} + for name, params in self.model_runner.model.named_parameters(): + self.cpu_model[name] = torch.empty_like(params, device='cpu') + params.data = self.cpu_model[name] + else: + for name, params in self.model_runner.model.named_parameters(): + params.data = self.cpu_model[name] + + def add_lora(self, lora_request: LoRARequest) -> bool: + return self.model_runner.add_lora(lora_request) + + def remove_lora(self, lora_id: int) -> bool: + return self.model_runner.remove_lora(lora_id) + + def list_loras(self) -> Set[int]: + return self.model_runner.list_loras() + + +def _init_distributed_environment( + parallel_config: ParallelConfig, + rank: int, + distributed_init_method: Optional[str] = None, +) -> None: + """Initialize the distributed environment.""" + if torch.distributed.is_initialized(): + print('The distributed environment has been initialized before vLLM') + elif not distributed_init_method: + raise ValueError("distributed_init_method must be set if torch.distributed " + "is not already initialized") + else: + torch.distributed.init_process_group( + backend="nccl", + world_size=parallel_config.world_size, + rank=rank, + # init_method=distributed_init_method, + ) + + # A small all_reduce for warmup. + torch.distributed.all_reduce(torch.zeros(1).cuda()) + # TODO (shengguangming): maybe we should also flag the megatron is initialized + if torch.distributed.get_world_size() > 1: + initialize_model_parallel_from_megatron(tensor_model_parallel_size=parallel_config.tensor_parallel_size) + else: + initialize_model_parallel() + + +def _pad_to_alignment(x: List[int], multiple_of: int, pad: int) -> List[int]: + return x + [pad] * ((-len(x)) % multiple_of) + + +def _pad_to_max(x: List[int], max_len: int, pad: int) -> List[int]: + return x + [pad] * (max_len - len(x)) + + +def _check_if_gpu_supports_dtype(torch_dtype: torch.dtype): + # Check if the GPU supports the dtype. + if torch_dtype == torch.bfloat16: + compute_capability = torch.cuda.get_device_capability() + if compute_capability[0] < 8: + gpu_name = torch.cuda.get_device_name() + raise ValueError("Bfloat16 is only supported on GPUs with compute capability " + f"of at least 8.0. Your {gpu_name} GPU has compute capability " + f"{compute_capability[0]}.{compute_capability[1]}.") diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/__init__.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/arg_utils.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/arg_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..089bbd748b202ccceb524f91271e7bf91dc9bdfe --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/arg_utils.py @@ -0,0 +1,320 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py + +import os +import argparse +import dataclasses +from dataclasses import dataclass +from typing import List, Optional, Union + +import torch.nn as nn + +from transformers import PretrainedConfig +from .config import ModelConfig, LoadConfig + +from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig, EngineConfig, LoRAConfig, ParallelConfig, + SchedulerConfig, SpeculativeConfig, TokenizerPoolConfig, VisionLanguageConfig) +from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS +from vllm.utils import str_to_int_tuple + + +def nullable_str(val: str): + if not val or val == "None": + return None + return val + + +@dataclass +class EngineArgs: + """Arguments for vLLM engine.""" + model_hf_config: PretrainedConfig = None + skip_tokenizer_init: bool = False + served_model_name: Optional[Union[str, List[str]]] = None # TODO + download_dir: Optional[str] = None + load_format: str = 'auto' + dtype: str = 'auto' + kv_cache_dtype: str = 'auto' + quantization_param_path: Optional[str] = None + seed: int = 0 + max_model_len: Optional[int] = None + worker_use_ray: bool = False + pipeline_parallel_size: int = 1 + tensor_parallel_size: int = 1 + max_parallel_loading_workers: Optional[int] = None + block_size: int = 16 + enable_prefix_caching: bool = False + use_v2_block_manager: bool = False + swap_space: int = 4 # GiB + gpu_memory_utilization: float = 0.90 + max_num_batched_tokens: Optional[int] = None + max_num_seqs: int = 256 + max_logprobs: int = 5 # OpenAI default value + disable_log_stats: bool = False + revision: Optional[str] = None + code_revision: Optional[str] = None + tokenizer_revision: Optional[str] = None + quantization: Optional[str] = None + enforce_eager: bool = False + max_context_len_to_capture: Optional[int] = None + max_seq_len_to_capture: int = 8192 + disable_custom_all_reduce: bool = False + tokenizer_pool_size: int = 0 + tokenizer_pool_type: str = "ray" + tokenizer_pool_extra_config: Optional[dict] = None + enable_lora: bool = False + max_loras: int = 1 + max_lora_rank: int = 16 + fully_sharded_loras: bool = False + lora_extra_vocab_size: int = 256 + lora_dtype = 'auto' + max_cpu_loras: Optional[int] = None + device: str = 'auto' + ray_workers_use_nsight: bool = False + num_gpu_blocks_override: Optional[int] = None + num_lookahead_slots: int = 0 + model_loader_extra_config: Optional[dict] = None + + # Related to Vision-language models such as llava + image_input_type: Optional[str] = None + image_token_id: Optional[int] = None + image_input_shape: Optional[str] = None + image_feature_size: Optional[int] = None + scheduler_delay_factor: float = 0.0 + enable_chunked_prefill: bool = False + + guided_decoding_backend: str = 'outlines' + # Speculative decoding configuration. + speculative_model: Optional[str] = None + num_speculative_tokens: Optional[int] = None + speculative_max_model_len: Optional[int] = None + ngram_prompt_lookup_max: Optional[int] = None + ngram_prompt_lookup_min: Optional[int] = None + + @staticmethod + def add_cli_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser: + """Shared CLI arguments for vLLM engine.""" + # Model arguments + # TODO(shengguangming): delete the unused args + parser.add_argument('--model', + type=str, + default='facebook/opt-125m', + help='name or path of the huggingface model to use') + parser.add_argument('--tokenizer', + type=str, + default=EngineArgs.tokenizer, + help='name or path of the huggingface tokenizer to use') + parser.add_argument('--revision', + type=str, + default=None, + help='the specific model version to use. It can be a branch ' + 'name, a tag name, or a commit id. If unspecified, will use ' + 'the default version.') + parser.add_argument('--tokenizer-revision', + type=str, + default=None, + help='the specific tokenizer version to use. It can be a branch ' + 'name, a tag name, or a commit id. If unspecified, will use ' + 'the default version.') + parser.add_argument('--tokenizer-mode', + type=str, + default=EngineArgs.tokenizer_mode, + choices=['auto', 'slow'], + help='tokenizer mode. "auto" will use the fast ' + 'tokenizer if available, and "slow" will ' + 'always use the slow tokenizer.') + parser.add_argument('--trust-remote-code', action='store_true', help='trust remote code from huggingface') + parser.add_argument('--download-dir', + type=str, + default=EngineArgs.download_dir, + help='directory to download and load the weights, ' + 'default to the default cache dir of ' + 'huggingface') + parser.add_argument('--load-format', + type=str, + default=EngineArgs.load_format, + choices=['auto', 'pt', 'safetensors', 'npcache', 'dummy'], + help='The format of the model weights to load. ' + '"auto" will try to load the weights in the safetensors format ' + 'and fall back to the pytorch bin format if safetensors format ' + 'is not available. ' + '"pt" will load the weights in the pytorch bin format. ' + '"safetensors" will load the weights in the safetensors format. ' + '"npcache" will load the weights in pytorch format and store ' + 'a numpy cache to speed up the loading. ' + '"dummy" will initialize the weights with random values, ' + 'which is mainly for profiling.') + parser.add_argument('--dtype', + type=str, + default=EngineArgs.dtype, + choices=['auto', 'half', 'float16', 'bfloat16', 'float', 'float32'], + help='data type for model weights and activations. ' + 'The "auto" option will use FP16 precision ' + 'for FP32 and FP16 models, and BF16 precision ' + 'for BF16 models.') + parser.add_argument('--max-model-len', + type=int, + default=None, + help='model context length. If unspecified, ' + 'will be automatically derived from the model.') + # Parallel arguments + parser.add_argument('--worker-use-ray', + action='store_true', + help='use Ray for distributed serving, will be ' + 'automatically set when using more than 1 GPU') + parser.add_argument('--pipeline-parallel-size', + '-pp', + type=int, + default=EngineArgs.pipeline_parallel_size, + help='number of pipeline stages') + parser.add_argument('--tensor-parallel-size', + '-tp', + type=int, + default=EngineArgs.tensor_parallel_size, + help='number of tensor parallel replicas') + # KV cache arguments + parser.add_argument('--block-size', + type=int, + default=EngineArgs.block_size, + choices=[8, 16, 32], + help='token block size') + # TODO(woosuk): Support fine-grained seeds (e.g., seed per request). + parser.add_argument('--seed', type=int, default=EngineArgs.seed, help='random seed') + parser.add_argument('--swap-space', + type=int, + default=EngineArgs.swap_space, + help='CPU swap space size (GiB) per GPU') + parser.add_argument('--gpu-memory-utilization', + type=float, + default=EngineArgs.gpu_memory_utilization, + help='the percentage of GPU memory to be used for' + 'the model executor') + parser.add_argument('--max-num-batched-tokens', + type=int, + default=EngineArgs.max_num_batched_tokens, + help='maximum number of batched tokens per ' + 'iteration') + parser.add_argument('--max-num-seqs', + type=int, + default=EngineArgs.max_num_seqs, + help='maximum number of sequences per iteration') + parser.add_argument('--disable-log-stats', action='store_true', help='disable logging statistics') + # Quantization settings. + parser.add_argument('--quantization', + '-q', + type=str, + choices=['awq', None], + default=None, + help='Method used to quantize the weights') + return parser + + @classmethod + def from_cli_args(cls, args: argparse.Namespace) -> 'EngineArgs': + # Get the list of attributes of this dataclass. + attrs = [attr.name for attr in dataclasses.fields(cls)] + # Set the attributes from the parsed arguments. + engine_args = cls(**{attr: getattr(args, attr) for attr in attrs}) + return engine_args + + def create_engine_config( + self, + ) -> EngineConfig: + device_config = DeviceConfig(self.device) + # NOTE(sgm): we only modify ModelConfig, other configs are import from vllm + model_config = ModelConfig(self.model_hf_config, self.dtype, self.seed, self.revision, self.code_revision, + self.tokenizer_revision, self.max_model_len, self.quantization, + self.quantization_param_path, self.enforce_eager, self.max_context_len_to_capture, + self.max_seq_len_to_capture, self.max_logprobs, self.skip_tokenizer_init, + self.served_model_name) + cache_config = CacheConfig(self.block_size, self.gpu_memory_utilization, + self.swap_space, self.kv_cache_dtype, self.num_gpu_blocks_override, + model_config.get_sliding_window(), self.enable_prefix_caching) + parallel_config = ParallelConfig( + self.pipeline_parallel_size, self.tensor_parallel_size, self.worker_use_ray, + self.max_parallel_loading_workers, self.disable_custom_all_reduce, + TokenizerPoolConfig.create_config( + self.tokenizer_pool_size, + self.tokenizer_pool_type, + self.tokenizer_pool_extra_config, + ), self.ray_workers_use_nsight) + + # Use the world_size set by TORCHRUN + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + parallel_config.world_size = world_size + + # TODO: spec config + speculative_config = SpeculativeConfig.maybe_create_spec_config( + target_model_config=model_config, + target_parallel_config=parallel_config, + target_dtype=self.dtype, + speculative_model=self.speculative_model, + num_speculative_tokens=self.num_speculative_tokens, + speculative_max_model_len=self.speculative_max_model_len, + enable_chunked_prefill=self.enable_chunked_prefill, + use_v2_block_manager=self.use_v2_block_manager, + ngram_prompt_lookup_max=self.ngram_prompt_lookup_max, + ngram_prompt_lookup_min=self.ngram_prompt_lookup_min, + ) + + scheduler_config = SchedulerConfig( + self.max_num_batched_tokens, + self.max_num_seqs, + model_config.max_model_len, + self.use_v2_block_manager, + num_lookahead_slots=(self.num_lookahead_slots + if speculative_config is None else speculative_config.num_lookahead_slots), + delay_factor=self.scheduler_delay_factor, + enable_chunked_prefill=self.enable_chunked_prefill, + ) + + lora_config = LoRAConfig(max_lora_rank=self.max_lora_rank, + max_loras=self.max_loras, + fully_sharded_loras=self.fully_sharded_loras, + lora_extra_vocab_size=self.lora_extra_vocab_size, + lora_dtype=self.lora_dtype, + max_cpu_loras=self.max_cpu_loras if self.max_cpu_loras and self.max_cpu_loras > 0 else + None) if self.enable_lora else None + + load_config = LoadConfig( + load_format=self.load_format, + download_dir=self.download_dir, + model_loader_extra_config=self.model_loader_extra_config, + ) + + if self.image_input_type: + if (not self.image_token_id or not self.image_input_shape or not self.image_feature_size): + raise ValueError('Specify `image_token_id`, `image_input_shape` and ' + '`image_feature_size` together with `image_input_type`.') + vision_language_config = VisionLanguageConfig( + image_input_type=VisionLanguageConfig.get_image_input_enum_type(self.image_input_type), + image_token_id=self.image_token_id, + image_input_shape=str_to_int_tuple(self.image_input_shape), + image_feature_size=self.image_feature_size, + ) + else: + vision_language_config = None + + decoding_config = DecodingConfig(guided_decoding_backend=self.guided_decoding_backend) + + return EngineConfig(model_config=model_config, + cache_config=cache_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + device_config=device_config, + lora_config=lora_config, + vision_language_config=vision_language_config, + speculative_config=speculative_config, + load_config=load_config, + decoding_config=decoding_config) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/config.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/config.py new file mode 100644 index 0000000000000000000000000000000000000000..6af04417b43a2d3672298fcf887b71fc230bb8ae --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/config.py @@ -0,0 +1,200 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/config.py + +import enum +import json +from typing import List, Optional, Union +from dataclasses import dataclass, field, fields + +from transformers import PretrainedConfig + +from vllm.logger import init_logger +from vllm.model_executor.layers.quantization import get_quantization_config +from vllm.transformers_utils.config import get_hf_text_config +from vllm.utils import is_hip +# Add for verl +from vllm.config import ModelConfig, _get_and_verify_dtype, _get_and_verify_max_len + +GPTQMarlinConfig = get_quantization_config("gptq_marlin") + +logger = init_logger(__name__) + +_GB = 1 << 30 + + +class ModelConfig(ModelConfig): + """Configuration for the model. + + Args: + model: Name or path of the huggingface model to use. + tokenizer: Name or path of the huggingface tokenizer to use. + tokenizer_mode: Tokenizer mode. "auto" will use the fast tokenizer if + available, and "slow" will always use the slow tokenizer. + trust_remote_code: Trust remote code (e.g., from HuggingFace) when + downloading the model and tokenizer. + download_dir: Directory to download and load the weights, default to the + default cache directory of huggingface. + load_format: The format of the model weights to load: + "auto" will try to load the weights in the safetensors format and + fall back to the pytorch bin format if safetensors format is + not available. + "pt" will load the weights in the pytorch bin format. + "safetensors" will load the weights in the safetensors format. + "npcache" will load the weights in pytorch format and store + a numpy cache to speed up the loading. + "dummy" will initialize the weights with random values, which is + mainly for profiling. + dtype: Data type for model weights and activations. The "auto" option + will use FP16 precision for FP32 and FP16 models, and BF16 precision + for BF16 models. + seed: Random seed for reproducibility. + revision: The specific model version to use. It can be a branch name, + a tag name, or a commit id. If unspecified, will use the default + version. + code_revision: The specific revision to use for the model code on + Hugging Face Hub. It can be a branch name, a tag name, or a + commit id. If unspecified, will use the default version. + tokenizer_revision: The specific tokenizer version to use. It can be a + branch name, a tag name, or a commit id. If unspecified, will use + the default version. + max_model_len: Maximum length of a sequence (including prompt and + output). If None, will be derived from the model. + quantization: Quantization method that was used to quantize the model + weights. If None, we assume the model weights are not quantized. + quantization_param_path: Path to JSON file containing scaling factors. + Used to load KV cache scaling factors into the model when KV cache + type is FP8_E4M3 on ROCm (AMD GPU). In the future these will also + be used to load activation and weight scaling factors when the + model dtype is FP8_E4M3 on ROCm. + enforce_eager: Whether to enforce eager execution. If True, we will + disable CUDA graph and always execute the model in eager mode. + If False, we will use CUDA graph and eager execution in hybrid. + max_context_len_to_capture: Maximum context len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode (DEPRECATED. Use max_seq_len_to_capture instead). + max_seq_len_to_capture: Maximum sequence len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode + skip_tokenizer_init: If true, skip initialization of tokenizer and + detokenizer. + served_model_name: The model name used in metrics tag `model_name`, + matches the model name exposed via the APIs. If multiple model + names provided, the first name will be used. If not specified, + the model name will be the same as `model`. + """ + + def __init__( + self, + hf_config: PretrainedConfig, + dtype: str, + seed: int, + revision: Optional[str] = None, + code_revision: Optional[str] = None, + tokenizer_revision: Optional[str] = None, + max_model_len: Optional[int] = None, + quantization: Optional[str] = None, + quantization_param_path: Optional[str] = None, + enforce_eager: bool = False, + max_context_len_to_capture: Optional[int] = None, + max_seq_len_to_capture: Optional[int] = None, + max_logprobs: int = 5, + skip_tokenizer_init: bool = False, + served_model_name: Optional[Union[str, List[str]]] = None, + ) -> None: + self.model = hf_config._name_or_path + self.tokenizer = hf_config._name_or_path + self.seed = seed + self.revision = revision + self.code_revision = code_revision + self.tokenizer_revision = tokenizer_revision + self.quantization = quantization + self.quantization_param_path = quantization_param_path + self.enforce_eager = enforce_eager + self.max_context_len_to_capture = max_context_len_to_capture + if self.max_context_len_to_capture is not None: + raise ValueError("`max_context_len_to_capture` is deprecated. " + "Use `max_seq_len_to_capture` instead.") + self.max_seq_len_to_capture = (max_seq_len_to_capture or max_context_len_to_capture) + self.max_logprobs = max_logprobs + self.skip_tokenizer_init = skip_tokenizer_init + + # self.hf_config = get_config(model, trust_remote_code, revision) + self.hf_config = hf_config + self.hf_text_config = get_hf_text_config(hf_config) + # TODO: for multimodal model + self.dtype = _get_and_verify_dtype(self.hf_config, dtype) + self.max_model_len = _get_and_verify_max_len(self.hf_config, max_model_len) + # self.served_model_name = get_served_model_name(model, + # served_model_name) + # self._verify_load_format() + # self._verify_tokenizer_mode() + self._verify_quantization() + self._verify_cuda_graph() + + +class LoadFormat(str, enum.Enum): + AUTO = 'auto' + MEGATRON = "megatron" + HF = "hf" + DTENSOR = 'dtensor' + DUMMY_HF = 'dummy_hf' + DUMMY_MEGATRON = 'dummy_megatron' + DUMMY_DTENSOR = 'dummy_dtensor' + + +@dataclass +class LoadConfig: + """ + download_dir: Directory to download and load the weights, default to the + default cache directory of huggingface. + load_format: The format of the model weights to load: + "auto" will try to load the weights in the safetensors format and + fall back to the pytorch bin format if safetensors format is + not available. + "pt" will load the weights in the pytorch bin format. + "safetensors" will load the weights in the safetensors format. + "npcache" will load the weights in pytorch format and store + a numpy cache to speed up the loading. + "dummy" will initialize the weights with random values, which is + mainly for profiling. + "tensorizer" will use CoreWeave's tensorizer library for + fast weight loading. + """ + + load_format: Union[str, LoadFormat, "BaseModelLoader"] = LoadFormat.AUTO + download_dir: Optional[str] = None + model_loader_extra_config: Optional[Union[str, dict]] = field(default_factory=dict) + + def __post_init__(self): + model_loader_extra_config = self.model_loader_extra_config or {} + if isinstance(model_loader_extra_config, str): + self.model_loader_extra_config = json.loads(model_loader_extra_config) + self._verify_load_format() + + def _verify_load_format(self) -> None: + if not isinstance(self.load_format, str): + return + + load_format = self.load_format.lower() + self.load_format = LoadFormat(load_format) + + rocm_not_supported_load_format: List[str] = [] + if is_hip() and load_format in rocm_not_supported_load_format: + rocm_supported_load_format = [ + f for f in LoadFormat.__members__ if (f not in rocm_not_supported_load_format) + ] + raise ValueError(f"load format '{load_format}' is not supported in ROCm. " + f"Supported load formats are " + f"{rocm_supported_load_format}") diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/dtensor_weight_loaders.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/dtensor_weight_loaders.py new file mode 100644 index 0000000000000000000000000000000000000000..6668b7509161e7d19d4b37f13318a80d59147448 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/dtensor_weight_loaders.py @@ -0,0 +1,269 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models + +from typing import Dict, Iterable, Tuple +import torch +import torch.nn as nn +from torch.distributed._tensor import DTensor, Shard, Replicate + +from vllm.model_executor.layers.linear import * +from vllm.model_executor.models import ModelRegistry +from vllm.model_executor.model_loader.weight_utils import default_weight_loader + + +def gemma_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + for (param_name, shard_name, shard_id) in stacked_params_mapping: + if shard_name not in name: + continue + stacked_name = name.replace(shard_name, param_name) + # Skip loading extra bias for GPTQ models. + if stacked_name.endswith(".bias") and stacked_name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[stacked_name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # lm_head is not used in vllm as it is tied with embed_token. + # To prevent errors, skip loading lm_head.weight. + if "lm_head.weight" in name: + continue + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + # GemmaRMSNorm is different from Llama's in that it multiplies + # (1 + weight) to the output, instead of just weight. + if "norm.weight" in name: + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + + norm_weight = local_loaded_weight + 1.0 + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, norm_weight.to(dtype=param.dtype)) + else: + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def gptbigcode_dtensor_load_weights(actor_weights: Dict, vllm_model: nn.Module): + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "lm_head.weight" in name: + continue + if ".attn.bias" in name: + # Skip attention mask. + # NOTE: "c_attn.bias" should not be skipped. + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def starcoder2_dtensor_load_weights(actor_weights: Dict, vllm_model: nn.Module): + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ] + + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + + for (param_name, weight_name, shard_id) in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def llama_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + (".qkv_proj", ".q_proj", "q"), + (".qkv_proj", ".k_proj", "k"), + (".qkv_proj", ".v_proj", "v"), + (".gate_up_proj", ".gate_proj", 0), + (".gate_up_proj", ".up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if ("rotary_emb.cos_cached" in name or "rotary_emb.sin_cached" in name): + # Models trained using ColossalAI may include these tensors in + # the checkpoint. Skip them. + continue + # With tie_word_embeddings, we can skip lm_head.weight + # The weight might appear unnecessarily in the files if the model is + # processed with quantization, LoRA, fine-tuning, etc. + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for (param_name, weight_name, shard_id) in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight) + + +def qwen2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for (param_name, weight_name, shard_id) in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def gpt2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + pass + + +def redistribute_dtensor(param_name: str, loaded_weights: DTensor, parallelize_plan: Dict = None): + param_name = _process_parameter_names(name=param_name) + if parallelize_plan is not None: + assert param_name in parallelize_plan.keys(), \ + f"param name: {param_name} not in parallelize_plan :{parallelize_plan.keys()}" + placement = parallelize_plan[param_name] + local_loaded_weights = loaded_weights.redistribute(device_mesh=loaded_weights.device_mesh, + placements=placement).to_local() + else: + local_loaded_weights = loaded_weights.full_tensor() + return local_loaded_weights + + +def _process_parameter_names(name): + # Remove '.weight' if it exists at the end of the string + if name.endswith(".weight"): + name = name[:-7] + + # Remove 'model.layers.x.' or 'model.' prefix + if "model.layers" in name: + parts = name.split('.') + # Reconstruct the string without 'model.layers.x.' + name = '.'.join(parts[3:]) # parts[0] is 'model', parts[1] is 'layers', parts[2] is 'x' + elif name.startswith("model."): + name = name[6:] # Remove 'model.' + + return name + + +__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__ = { + 'GPT2LMHeadModel': gpt2_dtensor_weight_loader, + 'LlamaForCausalLM': llama_dtensor_weight_loader, + 'LLaMAForCausalLM': llama_dtensor_weight_loader, + 'MistralForCausalLM': llama_dtensor_weight_loader, # mistral is the same as llama in vLLM + 'InternLMForCausalLM': llama_dtensor_weight_loader, + 'AquilaModel': llama_dtensor_weight_loader, + 'AquilaForCausalLM': llama_dtensor_weight_loader, + 'Phi3ForCausalLM': llama_dtensor_weight_loader, + 'GemmaForCausalLM': gemma_dtensor_weight_loader, + 'GPTBigCodeForCausalLM': gptbigcode_dtensor_load_weights, + 'Starcoder2ForCausalLM': starcoder2_dtensor_load_weights, + 'Qwen2ForCausalLM': qwen2_dtensor_weight_loader +} + + +# the actor model is .state_dict() +# Load dtensor weights +def load_dtensor_weights(actor_weights: Dict, vllm_model: nn.Module): + weight_loader = _get_model_weight_loader(vllm_model.__class__.__name__) + weight_loader(actor_weights, vllm_model) + # NOTE(sgm) to reduce peak memory usage, we offload vllm model to cpu + # after init, and we need this after sync model weights for in first iter. + vllm_model = vllm_model.cuda() + + +def _get_model_weight_loader(arch: str): + if arch in __MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__: + return __MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__.keys()}") + + +# NOTE(sgm): we use per-parameter weight loader in each vllm sub +def update_dtensor_weight_loader(): + pass diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/hf_weight_loader.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/hf_weight_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..0d562e596b8b75ac0a3e81ae651c77cfdc58f3a1 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/hf_weight_loader.py @@ -0,0 +1,91 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models + +from typing import Dict, Union, Optional, Iterable, Tuple + +import torch +import torch.nn as nn + +from vllm.model_executor.model_loader.utils import set_default_torch_dtype +from vllm.model_executor.model_loader.weight_utils import default_weight_loader + + +def update_hf_weight_loader(): + from vllm.model_executor.models.gemma import GemmaForCausalLM + GemmaForCausalLM.load_weights = gemma_load_weights + + +def gemma_load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]): + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(self.named_parameters()) + loaded_params = set() + for name, loaded_weight in weights: + for (param_name, shard_name, shard_id) in stacked_params_mapping: + if shard_name not in name: + continue + name = name.replace(shard_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, loaded_weight, shard_id) + break + else: + # lm_head is not used in vllm as it is tied with embed_token. + # To prevent errors, skip loading lm_head.weight. + if "lm_head.weight" in name: + continue + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + # GemmaRMSNorm is different from Llama's in that it multiplies + # (1 + weight) to the output, instead of just weight. + if "norm.weight" in name: + norm_weight = loaded_weight + 1.0 # prevent inplace modify actor weights + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, norm_weight) + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + loaded_params.add(name) + unloaded_params = params_dict.keys() - loaded_params + if unloaded_params: + raise RuntimeError("Some weights are not initialized from checkpoints: " + f"{unloaded_params}") + + +def load_hf_weights(actor_weights: Dict, vllm_model: nn.Module): + assert isinstance(actor_weights, Dict) + with set_default_torch_dtype(next(vllm_model.parameters()).dtype): # TODO + vllm_model.load_weights(actor_weights.items()) + for _, module in vllm_model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + vllm_model = vllm_model.cuda() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/llm.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/llm.py new file mode 100644 index 0000000000000000000000000000000000000000..94623a41423e841ae8e388a1d508da5b624a559a --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/llm.py @@ -0,0 +1,306 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py + +from typing import Dict, List, Optional, Tuple, Union + +from tqdm import tqdm +from transformers import PreTrainedTokenizer, PreTrainedTokenizerFast +from transformers import PretrainedConfig +import torch.nn as nn +from .arg_utils import EngineArgs +from .llm_engine_sp import LLMEngine +from vllm.lora.request import LoRARequest +from vllm.outputs import RequestOutput +from vllm.sampling_params import SamplingParams +from vllm.sequence import MultiModalData +from vllm.usage.usage_lib import UsageContext +from vllm.utils import Counter +import torch +from torch.nn.utils.rnn import pad_sequence +from verl.workers.rollout.tokenizer import HybridEngineBaseTokenizer + + +class LLM: + """An LLM for generating texts from given prompts and sampling parameters. + + This class includes a tokenizer, a language model (possibly distributed + across multiple GPUs), and GPU memory space allocated for intermediate + states (aka KV cache). Given a batch of prompts and sampling parameters, + this class generates texts from the model, using an intelligent batching + mechanism and efficient memory management. + + NOTE: This class is intended to be used for offline inference. For online + serving, use the `AsyncLLMEngine` class instead. + NOTE: For the comprehensive list of arguments, see `EngineArgs`. + + Args: + model: A HuggingFace Transformers model instance. + tokenizer: A HuggingFace Transformers tokenizer instance. + tokenizer_mode: The tokenizer mode. "auto" will use the fast tokenizer + if available, and "slow" will always use the slow tokenizer. + trust_remote_code: Trust remote code (e.g., from HuggingFace) when + downloading the model and tokenizer. + tensor_parallel_size: The number of GPUs to use for distributed + execution with tensor parallelism. + dtype: The data type for the model weights and activations. Currently, + we support `float32`, `float16`, and `bfloat16`. If `auto`, we use + the `torch_dtype` attribute specified in the model config file. + However, if the `torch_dtype` in the config is `float32`, we will + use `float16` instead. + quantization: The method used to quantize the model weights. Currently, + we support "awq". If None, we assume the model weights are not + quantized and use `dtype` to determine the data type of the weights. + revision: The specific model version to use. It can be a branch name, + a tag name, or a commit id. + tokenizer_revision: The specific tokenizer version to use. It can be a + branch name, a tag name, or a commit id. + seed: The seed to initialize the random number generator for sampling. + gpu_memory_utilization: The ratio (between 0 and 1) of GPU memory to + reserve for the model weights, activations, and KV cache. Higher + values will increase the KV cache size and thus improve the model's + throughput. However, if the value is too high, it may cause out-of- + memory (OOM) errors. + swap_space: The size (GiB) of CPU memory per GPU to use as swap space. + This can be used for temporarily storing the states of the requests + when their `best_of` sampling parameters are larger than 1. If all + requests will have `best_of=1`, you can safely set this to 0. + Otherwise, too small values may cause out-of-memory (OOM) errors. + enforce_eager: Whether to enforce eager execution. If True, we will + disable CUDA graph and always execute the model in eager mode. + If False, we will use CUDA graph and eager execution in hybrid. + max_context_len_to_capture: Maximum context len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode. + disable_custom_all_reduce: See ParallelConfig + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast, HybridEngineBaseTokenizer], + model_hf_config: PretrainedConfig, + tokenizer_mode: str = "auto", + trust_remote_code: bool = False, + tensor_parallel_size: int = 1, + dtype: str = "auto", + quantization: Optional[str] = None, + revision: Optional[str] = None, + tokenizer_revision: Optional[str] = None, + seed: int = 0, + gpu_memory_utilization: float = 0.9, + swap_space: int = 4, + enforce_eager: bool = False, + max_context_len_to_capture: int = None, + disable_custom_all_reduce: bool = False, + load_format = 'auto', + **kwargs, + ) -> None: + if "disable_log_stats" not in kwargs: + kwargs["disable_log_stats"] = True + engine_args = EngineArgs( + model_hf_config=model_hf_config, + tensor_parallel_size=tensor_parallel_size, + dtype=dtype, + quantization=quantization, + revision=revision, + tokenizer_revision=tokenizer_revision, + seed=seed, + gpu_memory_utilization=gpu_memory_utilization, + swap_space=swap_space, + enforce_eager=enforce_eager, + max_context_len_to_capture=max_context_len_to_capture, + disable_custom_all_reduce=disable_custom_all_reduce, + load_format=load_format, + **kwargs, + ) + tokenizer_cls = (PreTrainedTokenizer, PreTrainedTokenizerFast, HybridEngineBaseTokenizer) + if not isinstance(tokenizer, tokenizer_cls): + raise ValueError( + f"Unexpected tokenizer type: {type(tokenizer)}. Must be" + "one of the following: PreTrainedTokenizer, PreTrainedTokenizerFast, verl.workers.rollout.HybridEngineBaseTokenizer" + ) + self.llm_engine = LLMEngine.from_engine_args(model, tokenizer, engine_args) + self.request_counter = Counter() + + def init_cache_engine(self): + self.llm_engine.init_cache_engine() + + def free_cache_engine(self): + self.llm_engine.free_cache_engine() + + def get_tokenizer(self) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]: + return self.llm_engine.tokenizer + + def set_tokenizer( + self, + tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], + ) -> None: + self.llm_engine.tokenizer = tokenizer + + def generate( + self, + prompts: Optional[Union[str, List[str]]] = None, + sampling_params: Optional[Union[SamplingParams, List[SamplingParams]]] = None, + prompt_token_ids: Optional[List[List[int]]] = None, + use_tqdm: bool = True, + lora_request: Optional[LoRARequest] = None, + multi_modal_data: Optional[MultiModalData] = None, + ) -> List[RequestOutput]: + """Generates the completions for the input prompts. + + NOTE: This class automatically batches the given prompts, considering + the memory constraint. For the best performance, put all of your prompts + into a single list and pass it to this method. + + Args: + prompts: A list of prompts to generate completions for. + sampling_params: The sampling parameters for text generation. If + None, we use the default sampling parameters. + When it is a single value, it is applied to every prompt. + When it is a list, the list must have the same length as the + prompts and it is paired one by one with the prompt. + prompt_token_ids: A list of token IDs for the prompts. If None, we + use the tokenizer to convert the prompts to token IDs. + use_tqdm: Whether to use tqdm to display the progress bar. + lora_request: LoRA request to use for generation, if any. + multi_modal_data: Multi modal data. + + Returns: + A list of `RequestOutput` objects containing the generated + completions in the same order as the input prompts. + """ + if prompts is None and prompt_token_ids is None: + raise ValueError("Either prompts or prompt_token_ids must be " + "provided.") + if self.llm_engine.model_config.skip_tokenizer_init \ + and prompts is not None: + raise ValueError("prompts must be None if skip_tokenizer_init " + "is True") + if isinstance(prompts, str): + # Convert a single prompt to a list. + prompts = [prompts] + if (prompts is not None and prompt_token_ids is not None and len(prompts) != len(prompt_token_ids)): + raise ValueError("The lengths of prompts and prompt_token_ids " + "must be the same.") + + if prompts is not None: + num_requests = len(prompts) + else: + assert prompt_token_ids is not None + num_requests = len(prompt_token_ids) + + if sampling_params is None: + # Use default sampling params. + sampling_params = SamplingParams() + + elif isinstance(sampling_params, list) and len(sampling_params) != num_requests: + raise ValueError("The lengths of prompts and sampling_params " + "must be the same.") + if multi_modal_data: + multi_modal_data.data = multi_modal_data.data.to(torch.float16) + + # Add requests to the engine. + for i in range(num_requests): + prompt = prompts[i] if prompts is not None else None + token_ids = None if prompt_token_ids is None else prompt_token_ids[i] + if not isinstance(token_ids, list): + # NOTE(shengguangming): convert the rollout input into List[str] + token_ids = self._pre_process_inputs(token_ids) + self._add_request( + prompt, + sampling_params[i] if isinstance(sampling_params, list) else sampling_params, + token_ids, + lora_request=lora_request, + # Get ith image while maintaining the batch dim. + multi_modal_data=MultiModalData(type=multi_modal_data.type, data=multi_modal_data.data[i].unsqueeze(0)) + if multi_modal_data else None, + ) + return self._run_engine(use_tqdm) + + def _add_request( + self, + prompt: Optional[str], + sampling_params: SamplingParams, + prompt_token_ids: Optional[List[int]], + lora_request: Optional[LoRARequest] = None, + multi_modal_data: Optional[MultiModalData] = None, + ) -> None: + request_id = str(next(self.request_counter)) + self.llm_engine.add_request(request_id, + prompt, + sampling_params, + prompt_token_ids, + lora_request=lora_request, + multi_modal_data=multi_modal_data) + + def _run_engine(self, use_tqdm: bool) -> List[RequestOutput]: + # Initialize tqdm. + if use_tqdm: + num_requests = self.llm_engine.get_num_unfinished_requests() + pbar = tqdm(total=num_requests, desc="Processed prompts", dynamic_ncols=True) + # Run the engine. + outputs: List[RequestOutput] = [] + while self.llm_engine.has_unfinished_requests(): + step_outputs = self.llm_engine.step() + for output in step_outputs: + if output.finished: + outputs.append(output) + if use_tqdm: + pbar.update(1) + if use_tqdm: + pbar.close() + # Sort the outputs by request ID. + # This is necessary because some requests may be finished earlier than + # its previous requests. + outputs = sorted(outputs, key=lambda x: int(x.request_id)) + # TODO(shengguangming): maybe we can hack the autoregressive logics without only apply post process for better performance + return self._post_process_outputs(outputs) + + # NOTE(shengguangming): add for verl + # TODO(sgm): we can optimize it by making the dataloader yield List[int] without padding. + def _pre_process_inputs(self, prompt_token_ids: torch.Tensor) -> List[int]: + # remove the left padding in the prompt token_id + pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id + non_pad_index = torch.nonzero(prompt_token_ids != pad_token_id, as_tuple=False)[0][0] + token_ids = prompt_token_ids[non_pad_index:].tolist() + return token_ids + + # NOTE(shengguangming): add for verl + def _post_process_outputs(self, request_outputs: List[RequestOutput]) -> Tuple[torch.Tensor, torch.Tensor]: + output_token_ids = [] + logprobs = [] + for request_output in request_outputs: # List[RequestOutput] + outputs = request_output.outputs + for output in outputs: # List[CompletionOutput], usually len == 1 + output_token_ids.append(torch.tensor(output.token_ids)) + # TODO(shengguangming): can be optimzied by rewrite the Sampler._get_logprobs() logits + logprobs_dicts = output.logprobs + if logprobs_dicts is not None: + logprob = [] + for logprobs_dict, id in zip(logprobs_dicts, output.token_ids): + logprob.append(logprobs_dict[id].logprob) + logprobs.append(torch.tensor(logprob)) + + pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id + output_token_ids = pad_sequence(output_token_ids, batch_first=True, padding_value=pad_token_id) + if len(logprobs) > 0: + logprobs = pad_sequence(logprobs, batch_first=True, padding_value=pad_token_id) + return output_token_ids, logprobs + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.llm_engine.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + def offload_model_weights(self) -> None: + self.llm_engine.offload_model_weights() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/llm_engine_sp.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/llm_engine_sp.py new file mode 100644 index 0000000000000000000000000000000000000000..75bf11ab319a623daf9d6d57668e17c46c2cc4ec --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/llm_engine_sp.py @@ -0,0 +1,283 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/engine/llm_engine.py + +import torch +from typing import Dict, Optional, Union, Type + +import vllm +from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig, LoRAConfig, ParallelConfig, SchedulerConfig, + SpeculativeConfig, VisionLanguageConfig) +from vllm.core.scheduler import Scheduler +from vllm.engine.output_processor.interfaces import (SequenceGroupOutputProcessor) +from vllm.engine.output_processor.stop_checker import StopChecker +from vllm.executor.executor_base import ExecutorBase +from vllm.logger import init_logger +from vllm.transformers_utils.detokenizer import Detokenizer +from vllm.engine.metrics import StatLogger +from vllm.usage.usage_lib import (UsageContext, is_usage_stats_enabled, usage_message) +from vllm.utils import Counter +from vllm.engine.llm_engine import _load_generation_config_dict +from vllm.engine.llm_engine import LLMEngine + +import torch.nn as nn +from .arg_utils import EngineArgs +from .tokenizer import TokenizerGroup +from .config import ModelConfig, LoadConfig + +logger = init_logger(__name__) +_LOCAL_LOGGING_INTERVAL_SEC = 5 + + +class LLMEngine(LLMEngine): + """An LLM engine that receives requests and generates texts. + + This is the main class for the vLLM engine. It receives requests + from clients and generates texts from the LLM. It includes a tokenizer, a + language model (possibly distributed across multiple GPUs), and GPU memory + space allocated for intermediate states (aka KV cache). This class utilizes + iteration-level scheduling and efficient memory management to maximize the + serving throughput. + + The `LLM` class wraps this class for offline batched inference and the + `AsyncLLMEngine` class wraps this class for online serving. + + NOTE: The config arguments are derived from the `EngineArgs` class. For the + comprehensive list of arguments, see `EngineArgs`. + + Args: + model: the actor model initialize outside vllm (add for verl) + tokenizer: the initialized tokenizer (add for verl) + model_config: The configuration related to the LLM model. + cache_config: The configuration related to the KV cache memory + management. + parallel_config: The configuration related to distributed execution. + scheduler_config: The configuration related to the request scheduler. + distributed_init_method: The initialization method for distributed + execution. See `torch.distributed.init_process_group` for details. + placement_group: Ray placement group for distributed execution. + Required for distributed execution. + log_stats: Whether to log statistics. + """ + + def __init__( + self, + # NOTE(sgm): first two arguments are added for verl + model: Union[nn.Module, Dict], # model itself or its parameter dict + tokenizer: nn.Module, + # NOTE(sgm): vllm original arguments + model_config: ModelConfig, + cache_config: CacheConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + vision_language_config: Optional[VisionLanguageConfig], + speculative_config: Optional[SpeculativeConfig], + decoding_config: Optional[DecodingConfig], + executor_class: Type[ExecutorBase], + log_stats: bool, + usage_context: UsageContext = UsageContext.ENGINE_CONTEXT, + ) -> None: + logger.info( + "Initializing an LLM engine (v%s) with config: " + "model=%r, speculative_config=%r, tokenizer=%r, " + "skip_tokenizer_init=%s, tokenizer_mode=%s, revision=%s, " + "tokenizer_revision=%s, trust_remote_code=%s, dtype=%s, " + "max_seq_len=%d, download_dir=%r, load_format=%s, " + "tensor_parallel_size=%d, disable_custom_all_reduce=%s, " + "quantization=%s, enforce_eager=%s, kv_cache_dtype=%s, " + "quantization_param_path=%s, device_config=%s, " + "decoding_config=%r, seed=%d, served_model_name=%s)", + vllm.__version__, + model_config.model, + speculative_config, + model_config.tokenizer, + model_config.skip_tokenizer_init, + # model_config.tokenizer_mode, + model_config.revision, + model_config.tokenizer_revision, + # model_config.trust_remote_code, + model_config.dtype, + model_config.max_model_len, + load_config.download_dir, + load_config.load_format, + parallel_config.tensor_parallel_size, + parallel_config.disable_custom_all_reduce, + model_config.quantization, + model_config.enforce_eager, + cache_config.cache_dtype, + model_config.quantization_param_path, + device_config.device, + decoding_config, + model_config.seed, + # model_config.served_model_name, + ) + # TODO(woosuk): Print more configs in debug mode. + + self.model_config = model_config # TODO: currently is hfconfig + self.cache_config = cache_config + self.lora_config = lora_config + self.vision_language_config = vision_language_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.device_config = device_config + self.speculative_config = speculative_config + self.load_config = load_config + self.decoding_config = decoding_config or DecodingConfig() + self.log_stats = log_stats + + # self.model = model # should not store the model, it should be deleted + # TODO(shengguangming): maybe we can choose init here or from arguments + if not self.model_config.skip_tokenizer_init: + # TODO: check tokenizer class + self._init_tokenizer(tokenizer) + self.detokenizer = Detokenizer(self.tokenizer) + else: + self.detokenizer = None + self.tokenizer = None + + self.seq_counter = Counter() + # TODO: don't know what's the usage + self.generation_config_fields = _load_generation_config_dict(model_config) + + self.model_executor = executor_class( + model=model, # add for spmd_gpu_executor + model_config=model_config, + cache_config=cache_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + device_config=device_config, + lora_config=lora_config, + vision_language_config=vision_language_config, + speculative_config=speculative_config, + load_config=load_config, + ) + + # Profile the memory usage and initialize the cache. + self._initialize_kv_caches() + + # If usage stat is enabled, collect relevant info. + if is_usage_stats_enabled(): + from vllm.model_executor.model_loader import (get_architecture_class_name) + usage_message.report_usage( + get_architecture_class_name(model_config), + usage_context, + extra_kvs={ + # Common configuration + "dtype": str(model_config.dtype), + "tensor_parallel_size": parallel_config.tensor_parallel_size, + "block_size": cache_config.block_size, + "gpu_memory_utilization": cache_config.gpu_memory_utilization, + + # Quantization + "quantization": model_config.quantization, + "kv_cache_dtype": cache_config.cache_dtype, + + # Feature flags + "enable_lora": bool(lora_config), + "enable_prefix_caching": cache_config.enable_prefix_caching, + "enforce_eager": model_config.enforce_eager, + "disable_custom_all_reduce": parallel_config.disable_custom_all_reduce, + }) + + if self.tokenizer: + # Ping the tokenizer to ensure liveness if it runs in a + # different process. + self.tokenizer.ping() + + # Create the scheduler. + # NOTE: the cache_config here have been updated with the numbers of + # GPU and CPU blocks, which are profiled in the distributed executor. + # NOTE(shengguangming): each process will have independent scheduler + self.scheduler = Scheduler(scheduler_config, cache_config, lora_config) + + # Metric Logging. + if self.log_stats: + self.stat_logger = StatLogger(local_interval=_LOCAL_LOGGING_INTERVAL_SEC, + labels=dict(model_name=model_config.served_model_name), + max_model_len=self.model_config.max_model_len) + self.stat_logger.info("cache_config", self.cache_config) + + # Create sequence output processor, e.g. for beam search or + # speculative decoding. + self.output_processor = (SequenceGroupOutputProcessor.create_output_processor( + self.scheduler_config, + self.detokenizer, + self.scheduler, + self.seq_counter, + self.get_tokenizer_for_seq, + stop_checker=StopChecker( + self.scheduler_config.max_model_len, + self.get_tokenizer_for_seq, + ), + )) + + # TODO(sgm): add for verl but we may not tokenizer in Rollout + def _init_tokenizer(self, tokenizer, **tokenizer_init_kwargs): + init_kwargs = dict(enable_lora=bool(self.lora_config), + max_num_seqs=self.scheduler_config.max_num_seqs, + max_input_length=None) + init_kwargs.update(tokenizer_init_kwargs) + self.tokenizer: TokenizerGroup = TokenizerGroup(tokenizer, **init_kwargs) + + def init_cache_engine(self): + # TODO: check whether we should rebuild the CUDAGraph every iter when offload/load KVCache + # Re-capture CUDAGraph would be time-consuming + self.model_executor.init_cache_engine() + + def free_cache_engine(self): + self.model_executor.free_cache_engine() + + # NOTE(sgm): currently, we only support GPU executor + # The GPUExecutor remove the Ray dependency + @classmethod + def from_engine_args( + cls, + model, + tokenizer, + engine_args: EngineArgs, + usage_context: UsageContext = UsageContext.ENGINE_CONTEXT, + ) -> "LLMEngine": + """Creates an LLM engine from the engine arguments.""" + # Create the engine configs. + engine_config = engine_args.create_engine_config() + + # Initialize the cluster and specify the executor class. + assert engine_config.device_config.device_type == "cuda", \ + "Currently, the vllm in verl only support running on GPU" + + if engine_config.parallel_config.world_size == 1: + engine_config.load_config.load_format = "dummy_hf" + + from .spmd_gpu_executor import SPMDGPUExecutor + executor_class = SPMDGPUExecutor + + # Create the LLM engine. + engine = cls( + model, + tokenizer, + **engine_config.to_dict(), + executor_class=executor_class, + log_stats=not engine_args.disable_log_stats, + usage_context=usage_context, + ) + return engine + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.model_executor.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + def offload_model_weights(self) -> None: + self.model_executor.offload_model_weights() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/megatron_weight_loaders.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/megatron_weight_loaders.py new file mode 100644 index 0000000000000000000000000000000000000000..1a7c2e2cfe57f1bd3a4b49b61d1085a76b0a9b0a --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/megatron_weight_loaders.py @@ -0,0 +1,348 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models + +from typing import Dict +import torch +import torch.nn as nn + +from vllm.model_executor.layers.linear import * +from vllm.model_executor.layers.vocab_parallel_embedding import VocabParallelEmbedding, ParallelLMHead +from vllm.model_executor.layers.activation import ScaledActivation +from vllm.model_executor.models import ModelRegistry + + +# NOTE(shengguangming): replace the origin weight loader function in the class +def parallel_weight_loader(self, param: torch.Tensor, loaded_weight: torch.Tensor) -> None: + """Parallel Linear weight loader.""" + assert param.size() == loaded_weight.size( + ), 'the parameter size is not align with the loaded weight size, param size: {}, loaded_weight size: {}'.format( + param.size(), loaded_weight.size()) + assert param.data.dtype == loaded_weight.data.dtype, "if we want to shared weights, the data type should also be the same" + + param.data = loaded_weight.data + + +def default_weight_loader(param: torch.Tensor, loaded_weight: torch.Tensor) -> None: + """Default weight loader.""" + assert param.size() == loaded_weight.size() + assert param.data.dtype == loaded_weight.data.dtype, "if we want to shared weights, the data type should also be the same" + + param.data = loaded_weight.data + + +def gpt2_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "lm_head.weight" in name: + # GPT-2 ties the weights of the embedding layer and the final + # linear layer. + continue + if ".attn.bias" in name or ".attn.masked_bias" in name: + # Skip attention mask. + # NOTE: "c_attn.bias" should not be skipped. + continue + if not name.startswith("transformer."): + name = "transformer." + name + param = params_dict[name] + # The HF's GPT-2 implementation uses Conv1D instead of Linear. + # Because of this, we need to transpose the weights. + # Note(zhuohan): the logic below might break quantized models. + for conv1d_weight_name in ["c_attn", "c_proj", "c_fc"]: + if conv1d_weight_name not in name: + continue + if not name.endswith(".weight"): + continue + # TODO: check megatron + loaded_weight = loaded_weight.t() + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_te_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv.layer_norm_weight", "input_layernorm.weight"), + ("self_attention.linear_qkv.layer_norm_bias", "input_layernorm.bias"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", 'self_attn.o_proj'), + ('pre_mlp_layernorm', 'post_attention_layernorm'), + ('mlp.linear_fc1.layer_norm_weight', 'post_attention_layernorm.weight'), + ('mlp.linear_fc1.layer_norm_bias', 'post_attention_layernorm.bias'), + ('mlp.linear_fc1', 'mlp.gate_up_proj'), + ('mlp.linear_fc2', 'mlp.down_proj'), + ('decoder.final_layernorm', 'model.norm'), + ('output_layer', 'lm_head'), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith('.bias') and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", 'self_attn.o_proj'), + ( + 'input_layernorm', + 'input_layernorm', + ), + ('pre_mlp_layernorm', 'post_attention_layernorm'), + ('mlp.linear_fc1', 'mlp.gate_up_proj'), + ('mlp.linear_fc2', 'mlp.down_proj'), + ('decoder.final_layernorm', 'model.norm'), + ('output_layer', 'lm_head'), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith('.bias') and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def _replace_name(megatron_name, name_mapping): + for m_name, v_name in name_mapping: + if m_name not in megatron_name: + continue + if 'layers' in megatron_name: # deal with decoder layers + megatron_name = megatron_name.replace('decoder', 'model') + megatron_name_list = megatron_name.split('.') + if 'layer_norm_weight' in megatron_name_list or 'layer_norm_bias' in megatron_name_list: + param_name_list = megatron_name_list[:3] + param_name_list.append(v_name) + param_name = '.'.join(param_name_list) + else: + param_name_list = megatron_name_list[:3] + weight_or_bias = megatron_name_list[-1] + param_name_list.append(v_name) + param_name_list.append(weight_or_bias) + param_name = '.'.join(param_name_list) + return param_name + else: + param_name = megatron_name.replace(m_name, v_name) + return param_name + + +def llama_megatron_core_te_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv.layer_norm_weight", "input_layernorm.weight"), + ("self_attention.linear_qkv.layer_norm_bias", "input_layernorm.bias"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", 'self_attn.o_proj'), + ('pre_mlp_layernorm', 'post_attention_layernorm'), + ('mlp.linear_fc1.layer_norm_weight', 'post_attention_layernorm.weight'), + ('mlp.linear_fc1.layer_norm_bias', 'post_attention_layernorm.bias'), + ('mlp.linear_fc1', 'mlp.gate_up_proj'), + ('mlp.linear_fc2', 'mlp.down_proj'), + ('decoder.final_layernorm', 'model.norm'), + ('output_layer', 'lm_head'), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith('.bias') and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", 'self_attn.o_proj'), + ( + 'input_layernorm', + 'input_layernorm', + ), + ('pre_mlp_layernorm', 'post_attention_layernorm'), + ('mlp.linear_fc1', 'mlp.gate_up_proj'), + ('mlp.linear_fc2', 'mlp.down_proj'), + ('decoder.final_layernorm', 'model.norm'), + ('output_layer', 'lm_head'), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith('.bias') and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def _replace_name(megatron_name, name_mapping): + for m_name, v_name in name_mapping: + if m_name not in megatron_name: + continue + if 'layers' in megatron_name: # deal with decoder layers + megatron_name = megatron_name.replace('decoder', 'model') + megatron_name_list = megatron_name.split('.') + if 'layer_norm_weight' in megatron_name_list or 'layer_norm_bias' in megatron_name_list: + param_name_list = megatron_name_list[:3] + param_name_list.append(v_name) + param_name = '.'.join(param_name_list) + else: + param_name_list = megatron_name_list[:3] + weight_or_bias = megatron_name_list[-1] + param_name_list.append(v_name) + param_name_list.append(weight_or_bias) + param_name = '.'.join(param_name_list) + return param_name + else: + param_name = megatron_name.replace(m_name, v_name) + return param_name + + +def mistral_megatron_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + # TODO: need to implement a general way to deal with prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +__LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__ = { + ColumnParallelLinear: parallel_weight_loader, + MergedColumnParallelLinear: parallel_weight_loader, + QKVParallelLinear: parallel_weight_loader, + RowParallelLinear: parallel_weight_loader, + VocabParallelEmbedding: parallel_weight_loader, + ParallelLMHead: parallel_weight_loader + # "ScaledActivation.weight_loader": ScaledActivation, # TODO(shengguangming): latest commit in vllm fix awq for this function and add load_weights + # "default_weight_loader": default_weight_loader +} + +# for layer_class, weight_loader in __LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__.items(): +# # setattr(layer_class, 'megatron_weight_loader', weight_loader) +# layer_class.weight_loader = weight_loader + +__MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__ = { + 'GPT2LMHeadModel': gpt2_weight_loader, + 'LlamaForCausalLM': llama_megatron_core_te_weight_loader, # use te backend for open-source megatron + 'LLaMAForCausalLM': llama_megatron_core_te_weight_loader, + 'MistralForCausalLM': mistral_megatron_weight_loader, +} + + +# the actor model is .state_dict() +# Load megatron weights +def load_megatron_weights(actor_weights: Dict, vllm_model: nn.Module): + weight_loader = _get_model_weight_loader(vllm_model.__class__.__name__) + weight_loader(actor_weights, vllm_model) + # NOTE(sgm) to reduce peak memory usage, we offload vllm model to cpu + # after init, and we need this after sync model weights for in first iter. + vllm_model = vllm_model.cuda() + + +def _get_model_weight_loader(arch: str): + if arch in __MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__: + return __MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {ModelRegistry.get_supported_archs()}") + + +def update_megatron_weight_loader(): + for layer_class, weight_loader in __LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__.items(): + layer_class.weight_loader = weight_loader + VocabParallelEmbedding.__init__ = vocab_init + + +# FIXME(shengguangming): the vLLM vocab will pad to 64, which may incur out of bounds +# so we need to rewrite the init function of vocab +DEFAULT_VOCAB_PADDING_SIZE = 64 + + +def vocab_init(self, + num_embeddings: int, + embedding_dim: int, + params_dtype: Optional[torch.dtype] = None, + org_num_embeddings: Optional[int] = None, + padding_size: int = DEFAULT_VOCAB_PADDING_SIZE): + super(VocabParallelEmbedding, self).__init__() + + # Keep the input dimensions. + # TODO (pad to be divided by 4) + self.num_embeddings = num_embeddings + self.org_vocab_size = org_num_embeddings or num_embeddings + + # self.num_embeddings_padded = pad_vocab_size(num_embeddings, + # padding_size) + self.embedding_dim = embedding_dim + if params_dtype is None: + params_dtype = torch.get_default_dtype() + self.tp_size = get_tensor_model_parallel_world_size() + # Divide the weight matrix along the vocaburaly dimension. + + # TODO: remove dependencies from megatron + from megatron.core.tensor_parallel.utils import VocabUtility + self.vocab_start_index, self.vocab_end_index = (VocabUtility.vocab_range_from_global_vocab_size( + self.num_embeddings, get_tensor_model_parallel_rank(), self.tp_size)) + self.num_embeddings_per_partition = (self.vocab_end_index - self.vocab_start_index) + self.weight = Parameter( + torch.empty( + self.num_embeddings_per_partition, + self.embedding_dim, + # device=torch.cuda.current_device(), + dtype=params_dtype)) + set_weight_attrs(self.weight, {"parallel_dim": 0, "weight_loader": self.weight_loader}) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/model_loader.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/model_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..5f4013451ff7e4e4612719251f48d5849fdc15d5 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/model_loader.py @@ -0,0 +1,265 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/model_loader +"""Utilities for selecting and loading models.""" +from typing import Dict, Union, Optional, Iterable, Tuple + +import torch +import torch.nn as nn +from transformers import PreTrainedModel + +from vllm.config import (DeviceConfig, LoRAConfig, ParallelConfig, SchedulerConfig, VisionLanguageConfig) +from vllm.model_executor.model_loader import BaseModelLoader +from vllm.model_executor.model_loader.loader import _initialize_model +from vllm.model_executor.model_loader.utils import set_default_torch_dtype +from vllm.distributed.communication_op import tensor_model_parallel_all_gather + +from .config import ModelConfig, LoadFormat, LoadConfig +from .megatron_weight_loaders import load_megatron_weights, update_megatron_weight_loader +from .dtensor_weight_loaders import load_dtensor_weights, update_dtensor_weight_loader +from .hf_weight_loader import update_hf_weight_loader + + +def get_model(actor_model: Union[PreTrainedModel, Dict], model_config: ModelConfig, load_config: LoadConfig, + device_config: DeviceConfig, parallel_config: ParallelConfig, scheduler_config: SchedulerConfig, + lora_config: Optional[LoRAConfig], vision_language_config: Optional[VisionLanguageConfig]) -> nn.Module: + loader = get_model_loader(load_config) + if load_config.load_format.startswith('dummy'): + return loader.load_model(model_config=model_config, + device_config=device_config, + lora_config=lora_config, + vision_language_config=vision_language_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config) + else: + return loader.load_model(actor_model=actor_model, + model_config=model_config, + device_config=device_config, + lora_config=lora_config, + vision_language_config=vision_language_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config) + + +def get_model_loader(load_config: LoadConfig) -> BaseModelLoader: + """Get a model loader based on the load format.""" + + if isinstance(load_config.load_format, type): + return load_config.load_format(load_config) + + if load_config.load_format == LoadFormat.AUTO: + update_megatron_weight_loader() + return MegatronLoader(load_config) + + # NOTE(sgm): change the weight_loader function in runtime + if load_config.load_format == LoadFormat.MEGATRON: + update_megatron_weight_loader() + return MegatronLoader(load_config) + + if load_config.load_format == LoadFormat.HF: + update_hf_weight_loader() + return HFLoader(load_config) + + if load_config.load_format == LoadFormat.DTENSOR: + update_dtensor_weight_loader() + return DTensorLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_HF: + update_hf_weight_loader() + return DummyModelLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_MEGATRON: + update_megatron_weight_loader() + return DummyModelLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_DTENSOR: + update_dtensor_weight_loader() + return DummyModelLoader(load_config) + + raise ValueError('load format not supported in verl: {}, only support {} and {}'.format( + load_config.load_format, LoadFormat.MEGATRON, LoadFormat.HF)) + + +class DummyModelLoader(BaseModelLoader): + """Model loader that will set model weights to random values.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def load_model(self, *, model_config: ModelConfig, device_config: DeviceConfig, lora_config: Optional[LoRAConfig], + vision_language_config: Optional[VisionLanguageConfig], parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, vision_language_config) + # NOTE(woosuk): For accurate performance evaluation, we assign + # random values to the weights. + # initialize_dummy_weights(model) + return model.eval() + + +class MegatronLoader(BaseModelLoader): + """Model loader that can load the model weights from partitioned megatron model.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def _get_weights_iterator(actor_model: Union[PreTrainedModel, Dict]): + # NOTE(shengguangming) Load the weights from the actor model + pass + # if isinstance(actor_model, nn.Module): + # load_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), vllm_model=model) + # else: + # load_weights(actor_weights=actor_model, vllm_model=model) + # return actor_model + + def load_model(self, actor_model: Union[PreTrainedModel, + Dict], model_config: ModelConfig, device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], vision_language_config: Optional[VisionLanguageConfig], + parallel_config: ParallelConfig, scheduler_config: SchedulerConfig) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, vision_language_config) + + # TODO(sgm): This is a hack, we need to register the load_weight() func for each model in vllm + if isinstance(actor_model, nn.Module): + load_megatron_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), + vllm_model=model) + else: + load_megatron_weights(actor_weights=actor_model, vllm_model=model) + + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +class HFLoader(BaseModelLoader): + """Model loader that can load the model weights from model's full params.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def _get_weights_iterator(self, actor_model: Union[PreTrainedModel, Dict]): + if isinstance(actor_model, Dict): + return actor_model.items() + elif isinstance(actor_model, nn.Module): + return dict(actor_model.named_parameters()).items() + else: + raise ValueError(f'actor model should be Dict or nn.Module, but get {type(actor_model)}') + + def load_model(self, actor_model: Union[PreTrainedModel, + Dict], model_config: ModelConfig, device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], vision_language_config: Optional[VisionLanguageConfig], + parallel_config: ParallelConfig, scheduler_config: SchedulerConfig) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + # with torch.device(device_config.device): + # NOTE(sgm): init the model in cpu + model = _initialize_model(model_config, self.load_config, lora_config, vision_language_config) + model.load_weights(self._get_weights_iterator(actor_model)) + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +class DTensorLoader(BaseModelLoader): + """Model loader that can load the model weights from partitioned megatron model.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def _get_weights_iterator(actor_model: Union[PreTrainedModel, Dict]): + # NOTE(shengguangming) Load the weights from the actor model + pass + # if isinstance(actor_model, nn.Module): + # load_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), vllm_model=model) + # else: + # load_weights(actor_weights=actor_model, vllm_model=model) + # return actor_model + + def load_model(self, actor_model: Union[PreTrainedModel, + Dict], model_config: ModelConfig, device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], vision_language_config: Optional[VisionLanguageConfig], + parallel_config: ParallelConfig, scheduler_config: SchedulerConfig) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, vision_language_config) + + # TODO(sgm): This is a hack, we need to register the load_weight() func for each model in vllm + if isinstance(actor_model, nn.Module): + load_dtensor_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), + vllm_model=model) + else: + load_dtensor_weights(actor_weights=actor_model, vllm_model=model) + + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +# FIXME(sgm): hack the _get_logits function in vllm v0.4.2 +# as they use ray, the _get_logits result will only need to return to the driver node, +# therefore gather is enough. However, we use SPMD instead of a central scheduler, +# all_gather is required (aligned with v0.2.6) +def _get_logits(self, hidden_states: torch.Tensor, embedding: torch.Tensor, + embedding_bias: Optional[torch.Tensor]) -> torch.Tensor: + # Get the logits for the next tokens. + logits = torch.matmul(hidden_states, embedding.t()) + if embedding_bias is not None: + logits += embedding_bias + logits = tensor_model_parallel_all_gather(logits) + # Remove paddings in vocab (if any). + if logits is not None: + logits = logits[:, :self.org_vocab_size] + return logits + + +from vllm.model_executor.layers.logits_processor import LogitsProcessor + +LogitsProcessor._get_logits = _get_logits diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/model_runner.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/model_runner.py new file mode 100644 index 0000000000000000000000000000000000000000..1604b03630456a8adc44b31cec767f69f709899e --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/model_runner.py @@ -0,0 +1,281 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/worker/model_runner.py + +import torch +import torch.nn as nn +from enum import IntEnum +from typing import Dict, List, Optional, Set, Tuple, Union + +from vllm.attention import (AttentionMetadata, get_attn_backend) +from vllm.config import (DeviceConfig, LoRAConfig, ParallelConfig, SchedulerConfig, VisionLanguageConfig) +from vllm.logger import init_logger +from vllm.lora.layers import LoRAMapping +from vllm.lora.request import LoRARequest +from vllm.lora.worker_manager import LRUCacheWorkerLoRAManager +from vllm.model_executor import SamplingMetadata +from vllm.sequence import (MultiModalData, SamplerOutput, SequenceData, SequenceGroupMetadata) +from vllm.utils import (CudaMemoryProfiler, is_hip, is_pin_memory_available) +from vllm.worker.model_runner import ModelRunner, CUDAGraphRunner + +from .model_loader import get_model +from .config import ModelConfig, LoadConfig + +logger = init_logger(__name__) + + +# How batches are constructed. +class BatchType(IntEnum): + # Every batch is prefill. + PREFILL = 0 + # Every batch is decode. + DECODE = 1 + # Batch is a mixture of prefill and decode. + MIXED = 2 + + +class ModelRunner(ModelRunner): + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + model_config: ModelConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + kv_cache_dtype: Optional[str] = "auto", + vision_language_config: Optional[VisionLanguageConfig] = None, + ): + self.model_config = model_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.lora_config = lora_config + self.load_config = load_config + + # model_config can be None in tests/samplers/test_sampler.py. + # FIXME(woosuk): This is a hack to make the tests work. Refactor this. + self.sliding_window = (model_config.get_sliding_window() if model_config is not None else None) + self.device_config = (device_config if device_config is not None else DeviceConfig()) + self.device = self.device_config.device + + # NOTE(sgm): add for verl + self.model = model # this will be replaced by get_model() + + # Set after load_model. + self.lora_manager: LRUCacheWorkerLoRAManager = None + + self.graph_runners: Dict[int, CUDAGraphRunner] = {} + self.graph_memory_pool: Optional[Tuple[int, int]] = None # Set during graph capture. + + self.max_seq_len_to_capture = (self.model_config.max_seq_len_to_capture if self.model_config is not None else 0) + + self.pin_memory = is_pin_memory_available() + self.kv_cache_dtype = kv_cache_dtype + self.vision_language_config = vision_language_config + + self.attn_backend = get_attn_backend(self.model_config.dtype if model_config is not None else None) + + # Lazy initialization + self.block_size: int # Set after initial profiling. + # When using CUDA graph, the input block tables must be padded to + # max_seq_len_to_capture. However, creating the block table in + # Python can be expensive. To optimize this, we cache the block table + # in numpy and only copy the actual input content at every iteration. + # The shape of the cached block table will be + # (max batch size to capture, max context len to capture / block size). + self.graph_block_tables: torch.Tensor # Set after initial profiling. + + # Set if the backend is flashinfer. + self.flashinfer_workspace_buffer: torch.Tensor + + # NOTE(sgm): initialize model using the actor model + def load_model(self) -> None: + with CudaMemoryProfiler() as m: + self.model = get_model(actor_model=self.model, + model_config=self.model_config, + device_config=self.device_config, + lora_config=self.lora_config, + load_config=self.load_config, + parallel_config=self.parallel_config, + scheduler_config=self.scheduler_config, + vision_language_config=self.vision_language_config) + self.model_memory_usage = m.consumed_memory + logger.info("Loading model weights took %.4f GB", self.model_memory_usage / float(2**30)) + + if self.lora_config: + assert hasattr(self.model, "supported_lora_modules") and self.model.supported_lora_modules, ( + "Model does not support LoRA") + assert hasattr(self.model, "embedding_modules"), "Model does not have embedding_modules" + assert hasattr(self.model, "embedding_padding_modules"), "Model does not have embedding_padding_modules" + self.lora_manager = LRUCacheWorkerLoRAManager(self.scheduler_config.max_num_seqs, + self.scheduler_config.max_num_batched_tokens, self.vocab_size, + self.lora_config, self.device, self.model.embedding_modules, + self.model.embedding_padding_modules) + self.model = self.lora_manager.create_lora_manager(self.model) + + if self.kv_cache_dtype == "fp8" and is_hip(): + # Currently scaled KV cache is only enabled on ROCm + if self.model_config.quantization_param_path is not None: + if callable(getattr(self.model, "load_kv_cache_scales", None)): + self.model.load_kv_cache_scales(self.model_config.quantization_param_path) + else: + raise RuntimeError( + "Using FP8 KV cache and scaling factors provided but " + "model %s does not support loading scaling factors.", self.model.__class__) + else: + logger.warning("Using FP8 KV cache but no scaling factors " + "provided. Defaulting to scaling factors of 1.0. " + "This may lead to less accurate results!") + elif self.model_config.quantization_param_path is not None: + logger.warning("KV cache scaling factors provided, " + "but the KV cache data type is not FP8. " + "KV cache scaling factors will not be used.") + + def prepare_input_tensors( + self, + seq_group_metadata_list: List[SequenceGroupMetadata], + ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, SamplingMetadata, Set[LoRARequest], LoRAMapping, + torch.Tensor]: + # NOTE(sgm): all workers prepare the input in the same way + prefill_reqs = [] + decode_reqs = [] + for seq_group_meta in seq_group_metadata_list: + if seq_group_meta.is_prompt: + prefill_reqs.append(seq_group_meta) + else: + decode_reqs.append(seq_group_meta) + + # Prepare input tensors. + ( + input_tokens, + input_positions, + prefill_attn_metadata, + seq_lens, + query_lens, + lora_index_mapping, + lora_prompt_mapping, + lora_requests, + multi_modal_input, + slot_mapping, + ) = self._prepare_prompt(prefill_reqs) + ( + decode_input_tokens, + decode_input_positions, + decode_attn_metadata, + decode_lora_index_mapping, + decode_lora_prompt_mapping, + decode_lora_requests, + decode_slot_mapping, + ) = self._prepare_decode(decode_reqs) + sampling_metadata = SamplingMetadata.prepare(seq_group_metadata_list, seq_lens, query_lens, self.device, + self.pin_memory) + + if not self.scheduler_config.chunked_prefill_enabled: + assert (len(prefill_reqs) and len(decode_reqs)) == 0 + + num_prefills = len(seq_lens) + num_prefill_tokens = len(input_tokens) + num_decode_tokens = len(decode_input_tokens) + + # Coalesce tensors. Note that attn_metadata is currently not + # coalesced for simplicity. + input_tokens.extend(decode_input_tokens) + input_positions.extend(decode_input_positions) + slot_mapping.extend(decode_slot_mapping) + lora_index_mapping.extend(decode_lora_index_mapping) + lora_prompt_mapping.extend(decode_lora_prompt_mapping) + lora_requests.update(decode_lora_requests) + + input_tokens = torch.tensor(input_tokens, dtype=torch.long, device=self.device) + input_positions = torch.tensor(input_positions, dtype=torch.long, device=self.device) + slot_mapping = torch.tensor(slot_mapping, dtype=torch.long, device=self.device) + + if self.lora_config: + lora_mapping = LoRAMapping( + lora_index_mapping, + lora_prompt_mapping, + ) + else: + lora_mapping = None + + # Broadcast the metadata. + # If batch contains both prefill and decode, it sends 2 broadcasts. + # If it only contains 1 type, it triggers a single broadcast. + if (prefill_attn_metadata is not None and decode_attn_metadata is not None): + batch_type = BatchType.MIXED + elif prefill_attn_metadata is not None: + batch_type = BatchType.PREFILL + else: + batch_type = BatchType.DECODE + + attn_metadata = AttentionMetadata( + num_prefills=num_prefills, + slot_mapping=slot_mapping, + num_prefill_tokens=num_prefill_tokens, + num_decode_tokens=num_decode_tokens, + prefill_metadata=prefill_attn_metadata, + decode_metadata=decode_attn_metadata, + kv_cache_dtype=self.kv_cache_dtype, + ) + + return (input_tokens, input_positions, attn_metadata, sampling_metadata, lora_requests, lora_mapping, + multi_modal_input) + + @torch.inference_mode() + def execute_model( + self, + seq_group_metadata_list: List[SequenceGroupMetadata], + kv_caches: List[torch.Tensor], + ) -> Optional[SamplerOutput]: + (input_tokens, input_positions, attn_metadata, sampling_metadata, lora_requests, lora_mapping, + multi_modal_input) = self.prepare_input_tensors(seq_group_metadata_list) + + if self.lora_config: + self.set_active_loras(lora_requests, lora_mapping) + + # Currently cuda graph is only supported by the decode phase. + prefill_meta = attn_metadata.prefill_metadata + decode_meta = attn_metadata.decode_metadata + if prefill_meta is None and decode_meta.use_cuda_graph: + graph_batch_size = input_tokens.shape[0] + model_executable = self.graph_runners[graph_batch_size] + else: + model_executable = self.model + execute_model_kwargs = { + "input_ids": input_tokens, + "positions": input_positions, + "kv_caches": kv_caches, + "attn_metadata": attn_metadata, + } + if self.vision_language_config: + execute_model_kwargs.update({"image_input": multi_modal_input}) + hidden_states = model_executable(**execute_model_kwargs) + + # Compute the logits. + logits = self.model.compute_logits(hidden_states, sampling_metadata) + + # Only perform sampling in the driver worker. + # if not self.is_driver_worker: + # return None + + # TODO(sgm): perform sampling on rank 0 + # Sample the next token. + output = self.model.sample( + logits=logits, + sampling_metadata=sampling_metadata, + ) + + return output diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/parallel_state.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/parallel_state.py new file mode 100644 index 0000000000000000000000000000000000000000..9336089ed3dc3fcfb788314a3e9cd0f8161c6084 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/parallel_state.py @@ -0,0 +1,294 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Adapted from +# https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/parallel_state.py +# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. +"""Model and data parallel groups.""" +import os +import torch +import torch.distributed +from typing import Optional + +import vllm.distributed.parallel_state as ps + +import vllm.envs as envs +from vllm.logger import init_logger + +from torch.distributed.device_mesh import init_device_mesh + +logger = init_logger(__name__) +""" +This version is strongly tied with Megatron to implement HybridEngine and weight sharing between vllm and Megatron. +- We assume the Megatron tp+dp+pp world is already established before calling this function. + +""" + +# Device mesh for using DTensor +_DEVICE_MESH = None + +# Tensor model parallel group that the current rank belongs to. +_TP_DEVICE_GROUP = None +_TP_CPU_GROUP = None + + +# This method is for initializing the ParallelGroup when using HybridEngine +def initialize_parallel_state( + distributed_init_method: str = "env://", + backend: str = "nccl", + tensor_model_parallel_size: int = 1, + num_tp_per_train_tp: int = 1, + pipeline_model_parallel_size: int = 1, +): + # torch.distributed.all_reduce does not free the input tensor until + # the synchronization point. This causes the memory usage to grow + # as the number of all_reduce calls increases. This env var disables + # this behavior. + # Related issue: + # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573 + os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1" + + # NOTE(sgm): Modify for verl, Env vars will be set by TORCHRUN. + rank = int(os.getenv("RANK", "-1")) + local_rank = int(os.getenv("LOCAL_RANK", "0")) + + # Use the world_size set by TORCHRUN + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + ps.init_distributed_environment(world_size, rank, distributed_init_method, local_rank, backend) + if torch.distributed.get_world_size() > 1: + # NOTE: build a sepearate inference group with infer tp & micro dp + initialize_model_parallel_for_vllm(tensor_model_parallel_size=tensor_model_parallel_size, + num_tensor_model_parallel_groups_per_train_tp=num_tp_per_train_tp) + else: + initialize_model_parallel(tensor_model_parallel_size, pipeline_model_parallel_size, backend) + + +def ensure_model_parallel_initialized( + tensor_model_parallel_size: int, + pipeline_model_parallel_size: int = 1, + backend: Optional[str] = None, +) -> None: + """Helper to initialize model parallel groups if they are not initialized, + or ensure tensor-parallel and pipeline-parallel sizes are equal to expected + values if the model parallel groups are initialized. + """ + # get the backend of _DEVICE_WORLD_GROUP + backend = backend or torch.distributed.get_backend() + if not model_parallel_is_initialized(): + initialize_model_parallel(tensor_model_parallel_size, pipeline_model_parallel_size, backend) + return + + assert (get_tensor_model_parallel_world_size() == tensor_model_parallel_size), ( + "tensor parallel group already initialized, but of unexpected size: " + f"{get_tensor_model_parallel_world_size()=} vs. " + f"{tensor_model_parallel_size=}") + # assert (get_pipeline_model_parallel_world_size( + # ) == pipeline_model_parallel_size), ( + # "pipeline parallel group already initialized, but of unexpected size: " + # f"{get_pipeline_model_parallel_world_size()=} vs. " + # f"{pipeline_model_parallel_size=}") + + +def model_parallel_is_initialized(): + """Check if tensor and pipeline parallel groups are initialized.""" + return (ps._TP_DEVICE_GROUP is not None) + # and _PIPELINE_MODEL_PARALLEL_GROUP is not None) + + +def initialize_model_parallel_for_vllm(tensor_model_parallel_size: int, + num_tensor_model_parallel_groups_per_train_tp: int = 1) -> None: + from torch.distributed import new_group + # Get world size and rank. Ensure some consistencies. + assert torch.distributed.is_initialized() + + assert isinstance(tensor_model_parallel_size, int) + + # assert num_tensor_model_parallel_groups_per_train_tp == 1 and not different_tp_group + # assert num_tensor_model_parallel_groups_per_train_tp > 1 and different_tp_group + + # Build the tensor model-parallel groups. + assert ps._TP_DEVICE_GROUP is None, ("tensor model parallel group is already initialized") + + global _TP_DEVICE_GROUP + global _TP_CPU_GROUP + global _DEVICE_MESH + + world_size: int = torch.distributed.get_world_size() + + rank = torch.distributed.get_rank() + + backend = torch.distributed.get_backend() + + num_tensor_model_parallel_groups = world_size // tensor_model_parallel_size + + if num_tensor_model_parallel_groups_per_train_tp == 1: + # if tensor_model_parallel_size == train_tensor_parallel_size: + # using the same tp group as Megatron/vllm + for i in range(num_tensor_model_parallel_groups): + ranks = range(i * tensor_model_parallel_size, (i + 1) * tensor_model_parallel_size) + group = torch.distributed.new_group(ranks, backend=backend) + cpu_group = torch.distributed.new_group(ranks, backend="gloo") + if rank in ranks: + _TP_DEVICE_GROUP = group + _TP_CPU_GROUP = cpu_group + ps._TP_DEVICE_GROUP = group + ps._TP_CPU_GROUP = cpu_group + + # no _MICRO_DATA_PARALLEL_GROUP + else: + # initialize a micro_dp group and a tp group + # assume training tp=4, infer tp=2, then, weight is partitioned as + # [1], [2], [3], [4] for training and [1,2], [1,2], [3,4], [3,4] for inference + + # Build the inference tp groups + # train_tp = train_tensor_parallel_size + train_tp = num_tensor_model_parallel_groups_per_train_tp * tensor_model_parallel_size + # num_tensor_model_parallel_groups_per_train_tp = train_tp // tensor_model_parallel_size + assert _TP_DEVICE_GROUP is None, ("tensor model parallel group is already initialized") + for i in range(num_tensor_model_parallel_groups // num_tensor_model_parallel_groups_per_train_tp): + start = train_tp * i + end = train_tp * (i + 1) + for j in range(num_tensor_model_parallel_groups_per_train_tp): + ranks = list(range(start, end, num_tensor_model_parallel_groups_per_train_tp)) + for i in range(len(ranks)): + ranks[i] += j + group = torch.distributed.new_group(ranks) + cpu_group = torch.distributed.new_group(ranks, backend='gloo') + if rank in ranks: + _TP_DEVICE_GROUP = group + _TP_CPU_GROUP = cpu_group + ps._TP_DEVICE_GROUP = _TP_DEVICE_GROUP + ps._TP_CPU_GROUP = cpu_group + + # Build the pipeline model-parallel groups. + # global _PIPELINE_MODEL_PARALLEL_GROUP + # global _PIPELINE_GLOBAL_RANKS + # assert ps._PIPELINE_MODEL_PARALLEL_GROUP is None, ("pipeline model parallel group is already initialized") + + # ps._PIPELINE_MODEL_PARALLEL_GROUP = mpu.get_pipeline_model_parallel_group() + # ps._PIPELINE_GLOBAL_RANKS = mpu.get_pipeline_model_parallel_ranks() + + +def initialize_model_parallel( + tensor_model_parallel_size: int = 1, + pipeline_model_parallel_size: int = 1, + backend: Optional[str] = None, +) -> None: + """ + NOTE: This method is a hack from the open-sourced version without + asertion of world_size = tp * pp + + Initialize model parallel groups. + + Arguments: + tensor_model_parallel_size: number of GPUs used for tensor model + parallelism. + pipeline_model_parallel_size: number of GPUs used for pipeline model + parallelism. + + Let's say we have a total of 8 GPUs denoted by g0 ... g7 and we + use 2 GPUs to parallelize the model tensor, and 4 GPUs to parallelize + the model pipeline. The present function will + create 4 tensor model-parallel groups and 2 pipeline model-parallel groups: + 4 tensor model-parallel groups: + [g0, g1], [g2, g3], [g4, g5], [g6, g7] + 2 pipeline model-parallel groups: + [g0, g2, g4, g6], [g1, g3, g5, g7] + Note that for efficiency, the caller should make sure adjacent ranks + are on the same DGX box. For example if we are using 2 DGX-1 boxes + with a total of 16 GPUs, rank 0 to 7 belong to the first box and + ranks 8 to 15 belong to the second box. + """ + # Get world size and rank. Ensure some consistencies. + assert torch.distributed.is_initialized() + world_size: int = torch.distributed.get_world_size() + # get the backend of _DEVICE_WORLD_GROUP + backend = backend or torch.distributed.get_backend() + + # NOTE(sgm) we don't assert world_size == tp * pp + # DP is not managed by vllm but by the verl WorkerGroup + + num_tensor_model_parallel_groups: int = (world_size // tensor_model_parallel_size) + num_pipeline_model_parallel_groups: int = (world_size // pipeline_model_parallel_size) + rank = torch.distributed.get_rank() + + # Build device mesh for TP + if num_tensor_model_parallel_groups > 1: + device_mesh = init_device_mesh("cuda", (num_tensor_model_parallel_groups, tensor_model_parallel_size), + mesh_dim_names=("replicate", "tp_shard")) + else: + device_mesh = init_device_mesh("cuda", (tensor_model_parallel_size,), mesh_dim_names=["tp_shard"]) + shard_group = device_mesh.get_group(mesh_dim="tp_shard") + + # Build the tensor model-parallel groups. + global _TP_DEVICE_GROUP, _TP_CPU_GROUP + global _DEVICE_MESH + assert _TP_DEVICE_GROUP is None, ("tensor model parallel group is already initialized") + assert _DEVICE_MESH is None, ("device mesh in vllm is already initialized") + + _DEVICE_MESH = device_mesh + # for i in range(num_tensor_model_parallel_groups): + # ranks = range(i * tensor_model_parallel_size, (i + 1) * tensor_model_parallel_size) + # group = torch.distributed.new_group(ranks, backend=backend) + # cpu_group = torch.distributed.new_group(ranks, backend="gloo") + # assert torch.distributed.get_process_group_ranks(shard_group) == torch.distributed.get_process_group_ranks(cpu_group) + # ranks = torch.distributed.get_process_group_ranks(shard_group) + # cpu_group = torch.distributed.new_group(ranks, backend="gloo") # TODO: this will hang + # cpu_group = torch.distributed.new_group(, backend="gloo") + # if rank == 0: + # print(f'rank: {rank}') + # print(f'ranks: {ranks}') + # print(f'torch.distributed.get_process_group_ranks(shard_group): {torch.distributed.get_process_group_ranks(shard_group)}') + # if rank in ranks: + _TP_DEVICE_GROUP = shard_group + ps._TP_DEVICE_GROUP = _TP_DEVICE_GROUP + # ps._TP_CPU_GROUP = cpu_group # TODO: will hang when used with device mesh + + # TODO: init using device mesh + # Build the pipeline model-parallel groups. + assert ps._PIPELINE_MODEL_PARALLEL_GROUP is None, ("pipeline model parallel group is already initialized") + for i in range(num_pipeline_model_parallel_groups): + ranks = range(i, world_size, num_pipeline_model_parallel_groups) + group = torch.distributed.new_group(ranks, backend=backend) + if rank in ranks: + ps._PIPELINE_MODEL_PARALLEL_GROUP = group + ps._PIPELINE_GLOBAL_RANKS = ranks + + +""" +Device mesh utilities +""" + + +def get_device_mesh(): + assert _DEVICE_MESH is not None, ("device mesh is not initialized") + return _DEVICE_MESH + + +""" +Tensor model parallel utilities +""" + + +def get_tensor_model_parallel_group(): + """Get the tensor model parallel group the caller rank belongs to.""" + assert _TP_DEVICE_GROUP is not None, ("tensor model parallel group is not initialized") + return _TP_DEVICE_GROUP + + +def get_tensor_model_parallel_world_size(): + """Return world size for the tensor model parallel group.""" + return torch.distributed.get_world_size(group=get_tensor_model_parallel_group()) + + +def get_tensor_model_parallel_rank(): + """Return my rank for the tensor model parallel group.""" + return torch.distributed.get_rank(group=get_tensor_model_parallel_group()) + + +def get_tensor_model_parallel_src_rank(): + """Calculate the global rank corresponding to the first local rank + in the tensor model parallel group.""" + global_rank = torch.distributed.get_rank() + local_world_size = get_tensor_model_parallel_world_size() + return (global_rank // local_world_size) * local_world_size diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/spmd_gpu_executor.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/spmd_gpu_executor.py new file mode 100644 index 0000000000000000000000000000000000000000..b97bb600ac318cc4b769805a5dcedc24b296f4b3 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/spmd_gpu_executor.py @@ -0,0 +1,218 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/executor/gpu_executor.py +import os +import socket +from typing import Any, Dict, List, Optional, Set, Tuple + +import torch +import vllm.envs as envs +from vllm.executor.executor_base import ExecutorBase, ExecutorAsyncBase +from vllm.logger import init_logger +from vllm.lora.request import LoRARequest +from vllm.sequence import SamplerOutput, ExecuteModelRequest + +from vllm.config import (CacheConfig, DeviceConfig, LoRAConfig, ParallelConfig, SchedulerConfig, SpeculativeConfig, + VisionLanguageConfig) +from .config import ModelConfig, LoadConfig + +logger = init_logger(__name__) + + +class SPMDGPUExecutor(ExecutorBase): + """SPMD-based multi-GPU executor implementations.""" + + def __init__( + self, + model, # pytorch model itself or its parameter dict + model_config: ModelConfig, + cache_config: CacheConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + vision_language_config: Optional[VisionLanguageConfig], + speculative_config: Optional[SpeculativeConfig], + ) -> None: + self.model_config = model_config + self.cache_config = cache_config + self.lora_config = lora_config + self.load_config = load_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.device_config = device_config + self.vision_language_config = vision_language_config + self.speculative_config = speculative_config + + distributed_init_method = initialize_cluster(parallel_config) + self._init_executor(model, distributed_init_method) + + # TODO(sgm): verl not support speculative decode now + def _init_executor(self, model, distributed_init_method) -> None: + assert (not self.speculative_config), "Speculative decoding not yet supported for multi-GPU backend." + + # Create the parallel worker for each GPU. + self._init_workers_sp(model, distributed_init_method) + + def _init_workers_sp(self, model, distributed_init_method: str): + # Lazy import the Worker to avoid importing torch.cuda/xformers + # before CUDA_VISIBLE_DEVICES is set in the Worker + from .worker import Worker # pylint: disable=import-outside-toplevel + + rank = int(os.getenv("RANK")) + local_rank = int(os.getenv("LOCAL_RANK")) + print(f'local rank {local_rank}') + + self.worker = Worker( + model, + self.model_config, + self.parallel_config, + self.scheduler_config, + self.device_config, + self.cache_config, + self.load_config, + local_rank, + rank, + distributed_init_method, + lora_config=self.lora_config, + vision_language_config=self.vision_language_config, + ) + + # NOTE(shengguangming): torch.distributed.init_process_group will be called inside the init_model() + self.worker.init_device() + self.worker.load_model() + + def determine_num_available_blocks(self) -> Tuple[int, int]: + """Determine the number of available KV blocks. + + This invokes `determine_num_available_blocks` on each worker and takes + the min of the results, guaranteeing that the selected cache sizes are + compatible with all workers. + + Returns: + - tuple[num_gpu_blocks, num_cpu_blocks] + """ + # Get the maximum number of blocks that can be allocated on GPU and CPU. + num_blocks = self.worker.determine_num_available_blocks() + + # NOTE(shengguangming): Now we don't use a shared centralized controler but each process will + # have its own scheduler + num_gpu_blocks = num_blocks[0] + num_cpu_blocks = num_blocks[1] + + return num_gpu_blocks, num_cpu_blocks + + def initialize_cache(self, num_gpu_blocks: int, num_cpu_blocks: int) -> None: + """Initialize the KV cache in all workers. + """ + + # NOTE: We log here to avoid multiple logs when number of workers is + # greater than one. We could log in the engine, but not all executors + # have GPUs. + logger.info("# GPU blocks: %d, # CPU blocks: %d", num_gpu_blocks, num_cpu_blocks) + + self.cache_config.num_gpu_blocks = num_gpu_blocks + self.cache_config.num_cpu_blocks = num_cpu_blocks + + if torch.distributed.get_rank() == 0: + print( + f'before init cache memory allocated: {torch.cuda.memory_allocated() / 1e9}GB, reserved: {torch.cuda.memory_reserved() / 1e9}GB' + ) + self.worker.initialize_cache(num_gpu_blocks=num_gpu_blocks, num_cpu_blocks=num_cpu_blocks) + if torch.distributed.get_rank() == 0: + print( + f'after init cache memory allocated: {torch.cuda.memory_allocated() / 1e9}GB, reserved: {torch.cuda.memory_reserved() / 1e9}GB' + ) + + # NOTE(sgm): This will not profile & capture the model(CUDAGraph) when rebuilding KVCache + def init_cache_engine(self) -> None: + self.worker._init_cache_engine() + + def free_cache_engine(self) -> None: + self.worker.free_cache_engine() + + def execute_model(self, execute_model_req) -> List[SamplerOutput]: + all_outputs = self.worker.execute_model(execute_model_req=execute_model_req) + + # NOTE(sgm): + # Each GPU in vllm under verl has its own spmd_gpu_executor, therefore all GPUs should return the outputs + # In vllm with ray, only the driver worker returns the sampling results. + return all_outputs + + def add_lora(self, lora_request: LoRARequest) -> bool: + assert lora_request.lora_int_id > 0, "lora_id must be greater than 0." + return self.worker.add_lora(lora_request=lora_request) + + def remove_lora(self, lora_id: int) -> bool: + assert lora_id > 0, "lora_id must be greater than 0." + return self.worker.remove_lora(lora_id=lora_id) + + def list_loras(self) -> Set[int]: + return self.worker.list_loras() + + def check_health(self) -> None: + # SPMDExecutor will always be healthy as long as + # it's running. + return + + # NOTE(sgm): add for verl + def offload_model_weights(self) -> None: + self.worker.offload_model_weights() + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.worker.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + +def initialize_cluster( + parallel_config: ParallelConfig, + engine_use_ray: bool = False, + ray_address: Optional[str] = None, +) -> Tuple[str, Optional[None]]: + """Initialize the distributed cluster probably with Ray. + + Args: + parallel_config: The configurations for parallel execution. + + Returns: + The `distributed_init_method` is the address for initializing the + distributed backend. + """ + + # Initialize cluster locally. + port = get_open_port() + # We need to setup the distributed init method to make sure + # the distributed megatron code (e.g., get world size) works correctly. + # distributed_init_method = f"tcp://localhost:{port}" + distributed_init_method = 'env://' + return distributed_init_method + + +def get_open_port(): + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("", 0)) + return s.getsockname()[1] + + +# TODO(sgm): not implemented async executor yet +class SPMDGPUExecutorAsync(SPMDGPUExecutor, ExecutorAsyncBase): + + async def execute_model_async(self, execute_model_req: ExecuteModelRequest) -> List[SamplerOutput]: + """Executes one model step on the given sequences.""" + raise NotImplementedError + + async def check_health_async(self) -> None: + """Checks if the executor is healthy. If not, it should raise an + exception.""" + self.check_health() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/tokenizer.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/tokenizer.py new file mode 100644 index 0000000000000000000000000000000000000000..aa625a0338686d61816e838ef802cde327fc95c4 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/tokenizer.py @@ -0,0 +1,77 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/tokenizer_group/tokenizer_group.py + +from typing import List, Optional, Tuple, Union + +from transformers import (AutoTokenizer, PreTrainedTokenizer, PreTrainedTokenizerFast) + +from vllm.lora.request import LoRARequest +from vllm.utils import make_async, LRUCache +from vllm.transformers_utils.tokenizers import * + + +class TokenizerGroup: + """A group of tokenizers that can be used for LoRA adapters.""" + + def __init__(self, tokenizer: PreTrainedTokenizer, enable_lora: bool, max_num_seqs: int, + max_input_length: Optional[int]): + self.enable_lora = enable_lora + self.max_input_length = max_input_length + self.tokenizer = tokenizer + self.lora_tokenizers = LRUCache[PreTrainedTokenizer](capacity=max_num_seqs) if enable_lora else None + + def ping(self) -> bool: + """Check if the tokenizer group is alive.""" + return True + + def get_max_input_len(self, lora_request: Optional[LoRARequest] = None) -> Optional[int]: + """Get the maximum input length for the LoRA request.""" + return self.max_input_length + + def encode(self, + prompt: str, + request_id: Optional[str] = None, + lora_request: Optional[LoRARequest] = None) -> List[int]: + tokenizer = self.get_lora_tokenizer(lora_request) + return tokenizer.encode(prompt) + + async def encode_async(self, + prompt: str, + request_id: Optional[str] = None, + lora_request: Optional[LoRARequest] = None) -> List[int]: + tokenizer = await self.get_lora_tokenizer_async(lora_request) + return tokenizer.encode(prompt) + + def get_lora_tokenizer(self, lora_request: Optional[LoRARequest]) -> "PreTrainedTokenizer": + if not lora_request or not self.enable_lora: + return self.tokenizer + if lora_request.lora_int_id not in self.lora_tokenizers: + # TODO(sgm): the lora tokenizer is also passed, but may be different + tokenizer = self.tokenizer + # tokenizer = (get_lora_tokenizer( + # lora_request, **self.tokenizer_config) or self.tokenizer) + self.lora_tokenizers.put(lora_request.lora_int_id, tokenizer) + return tokenizer + else: + return self.lora_tokenizers.get(lora_request.lora_int_id) + + # FIXME(sgm): for simplicity, we assign the special token here + @property + def pad_token_id(self): + return self.tokenizer.pad_token_id + + @property + def eos_token_id(self): + return self.tokenizer.eos_token_id diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/worker.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/worker.py new file mode 100644 index 0000000000000000000000000000000000000000..1fab3e41fe87ea599f251f5da2ebb67b54f84b81 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_4_2/worker.py @@ -0,0 +1,292 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/worker/worker.py +"""A GPU worker class.""" +import os +import gc +from typing import Dict, List, Tuple, Optional, Union + +import torch +import torch.distributed +import torch.nn as nn + +from vllm.config import (CacheConfig, DeviceConfig, LoRAConfig, ParallelConfig, SchedulerConfig, VisionLanguageConfig) +from vllm.model_executor import set_random_seed +from vllm.sequence import SamplerOutput, ExecuteModelRequest +from vllm.worker.cache_engine import CacheEngine +from vllm.distributed.device_communicators import pynccl_utils +from vllm.distributed.device_communicators.custom_all_reduce import (init_custom_ar) +# TODO(sgm): check why vllm has similar file in vllm.model_executor.parallel_utils.parallel_state +from vllm.distributed import get_tensor_model_parallel_cpu_group, init_distributed_environment, get_tensor_model_parallel_group +from vllm.worker.worker import Worker, _check_if_gpu_supports_dtype + +from .model_runner import ModelRunner +from .megatron_weight_loaders import load_megatron_weights +from .hf_weight_loader import load_hf_weights +from .dtensor_weight_loaders import load_dtensor_weights +from .parallel_state import (ensure_model_parallel_initialized) +from .config import ModelConfig, LoadConfig, LoadFormat + + +class Worker(Worker): + """A worker class that executes (a partition of) the model on a GPU. + + Each worker is associated with a single GPU. The worker is responsible for + maintaining the KV cache and executing the model on the GPU. In case of + distributed inference, each worker is assigned a partition of the model. + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + model_config: ModelConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + cache_config: CacheConfig, + load_config: LoadConfig, + local_rank: int, + rank: int, + distributed_init_method: str, + lora_config: Optional[LoRAConfig] = None, + vision_language_config: Optional[VisionLanguageConfig] = None, + is_driver_worker: bool = False, + ) -> None: + # self.model = model # will be replaced in the init_model + self.model_config = model_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.device_config = device_config + self.cache_config = cache_config + self.local_rank = local_rank + self.rank = rank + self.distributed_init_method = distributed_init_method + self.lora_config = lora_config + self.load_config = load_config + self.is_driver_worker = is_driver_worker + if self.is_driver_worker: + assert self.rank == 0, "The driver worker must have rank 0." + + self.vision_language_config = vision_language_config + if self.vision_language_config: + assert not self.lora_config, ("To be tested: vision language model with LoRA settings.") + + self.model_runner = ModelRunner( + model, + model_config, + parallel_config, + scheduler_config, + device_config, + load_config=load_config, + lora_config=self.lora_config, + kv_cache_dtype=self.cache_config.cache_dtype, + vision_language_config=vision_language_config, + ) + + # Uninitialized cache engine. Will be initialized by + # init_cache_engine. + self.cache_engine: CacheEngine = None + self.gpu_cache: List[torch.Tensor] = None + + # NOTE(sgm): For offloading inference engine params + self.cpu_model = None + + def init_device(self) -> None: + if self.device_config.device.type == "cuda": + # torch.distributed.all_reduce does not free the input tensor until + # the synchronization point. This causes the memory usage to grow + # as the number of all_reduce calls increases. This env var disables + # this behavior. + # Related issue: + # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573 + os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1" + + # NOTE(sgm): Modify for verl, Env vars will be set by TORCHRUN. + self.rank = self.rank if self.rank is not None else int(os.getenv("RANK", "-1")) + local_rank = int(os.getenv("LOCAL_RANK", "0")) + self.device = torch.device(f"cuda:{local_rank}") + if self.rank < 0: + raise ValueError("Invalid or unspecified rank.") + torch.cuda.set_device(self.device) + + # Use the world_size set by TORCHRUN + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + self.parallel_config.world_size = world_size + + _check_if_gpu_supports_dtype(self.model_config.dtype) + torch.cuda.empty_cache() + self.init_gpu_memory = torch.cuda.mem_get_info()[0] + else: + raise RuntimeError(f"Not support device type: {self.device_config.device}") + + # Initialize the distributed environment. + init_worker_distributed_environment(self.parallel_config, self.rank, self.distributed_init_method, + self.local_rank) + # Set random seed. + set_random_seed(self.model_config.seed) + # self.model = get_model(actor_model=self.model, model_config=self.model_config) + + @torch.inference_mode() + def determine_num_available_blocks(self) -> Tuple[int, int]: + """Profiles the peak memory usage of the model to determine how many + KV blocks may be allocated without OOMs. + + The engine will first conduct a profiling of the existing memory usage. + Then, it calculate the maximum possible number of GPU and CPU blocks + that can be allocated with the remaining free memory. + + .. tip:: + You may limit the usage of GPU memory + by adjusting the `gpu_memory_utilization` parameter. + """ + # Profile the memory usage of the model and get the maximum number of + # cache blocks that can be allocated with the remaining free memory. + torch.cuda.empty_cache() + # torch.cuda.reset_peak_memory_stats() + + # Execute a forward pass with dummy inputs to profile the memory usage + # of the model. + self.model_runner.profile_run() + + # Calculate the number of blocks that can be allocated with the + # profiled peak memory. + torch.cuda.synchronize() + free_gpu_memory, total_gpu_memory = torch.cuda.mem_get_info() + peak_memory = total_gpu_memory - free_gpu_memory + + assert peak_memory > 0, ("Error in memory profiling. This happens when the GPU memory was " + "not properly cleaned up before initializing the vLLM instance.") + + cache_block_size = self.get_cache_block_size_bytes() + + # NOTE(sgm) use the remaining memory + num_gpu_blocks = int((free_gpu_memory * self.cache_config.gpu_memory_utilization) // cache_block_size) + # num_gpu_blocks = int((total_gpu_memory * self.cache_config.gpu_memory_utilization - peak_memory) // cache_block_size) + + num_cpu_blocks = int(self.cache_config.swap_space_bytes // cache_block_size) + num_gpu_blocks = max(num_gpu_blocks, 0) + num_cpu_blocks = max(num_cpu_blocks, 0) + if self.model_runner.lora_manager: + self.model_runner.remove_all_loras() + + # NOTE(sgm): Add for verl, synchronize number of blocks with all the rank + num_gpu_blocks = torch.tensor([num_gpu_blocks], device='cuda') + num_cpu_blocks = torch.tensor([num_cpu_blocks], device='cuda') + torch.distributed.all_reduce(num_gpu_blocks, + op=torch.distributed.ReduceOp.MIN, + group=get_tensor_model_parallel_group()) + torch.distributed.all_reduce(num_cpu_blocks, + op=torch.distributed.ReduceOp.MIN, + group=get_tensor_model_parallel_group()) + num_gpu_blocks = num_gpu_blocks.item() + num_cpu_blocks = num_cpu_blocks.item() + gc.collect() + torch.cuda.empty_cache() + return num_gpu_blocks, num_cpu_blocks + + def _init_cache_engine(self): + if self.cache_engine is None and self.gpu_cache is None: + super()._init_cache_engine() + + def free_cache_engine(self): + # ensure `enforce_eager=True` + self.cache_engine = None + self.gpu_cache = None + + @torch.inference_mode() + def execute_model(self, execute_model_req: Optional[ExecuteModelRequest] = None) -> List[SamplerOutput]: + + if execute_model_req is None: + seq_group_metadata_list = None + else: + seq_group_metadata_list = execute_model_req.seq_group_metadata_list + + # NOTE(sgm): each SPMD rank will have identical input + assert seq_group_metadata_list is not None + assert execute_model_req is not None + num_seq_groups = len(seq_group_metadata_list) + blocks_to_swap_in = execute_model_req.blocks_to_swap_in + blocks_to_swap_out = execute_model_req.blocks_to_swap_out + blocks_to_copy = execute_model_req.blocks_to_copy + + self.cache_swap(blocks_to_swap_in, blocks_to_swap_out, blocks_to_copy) + + # If there is no input, we don't need to execute the model. + if num_seq_groups == 0: + return [] + + output = self.model_runner.execute_model(seq_group_metadata_list, self.gpu_cache) + + # Worker only supports single-step execution. Wrap the output in a list + # to conform to interface. + return [output] + + # assume the input is .state_dict() + def sync_model_weights(self, actor_weights: Dict, load_format: str): + if load_format in [LoadFormat.MEGATRON, LoadFormat.AUTO]: + load_megatron_weights(actor_weights, self.model_runner.model) + elif load_format == LoadFormat.HF: + # full model state dict without no sharding + load_hf_weights(actor_weights, self.model_runner.model) + elif load_format == LoadFormat.DTENSOR: + load_dtensor_weights(actor_weights, self.model_runner.model) + + def offload_model_weights(self) -> None: + if self.cpu_model == None: + self.cpu_model = {} + for name, params in self.model_runner.model.named_parameters(): + self.cpu_model[name] = torch.empty_like(params, device='cpu') + params.data = self.cpu_model[name] + else: + for name, params in self.model_runner.model.named_parameters(): + params.data = self.cpu_model[name] + + +def init_worker_distributed_environment( + parallel_config: ParallelConfig, + rank: int, + distributed_init_method: Optional[str] = "env://", + local_rank: int = -1, +) -> None: + """Initialize the distributed environment.""" + # NOTE(sgm) use tcp://localhost:xxxx will hang in HF setting without megatron + init_distributed_environment(parallel_config.world_size, rank, distributed_init_method, local_rank) + + ensure_model_parallel_initialized(tensor_model_parallel_size=parallel_config.tensor_parallel_size, + pipeline_model_parallel_size=parallel_config.pipeline_parallel_size) + + # TODO(sgm): check whether need this + # if pynccl_utils.is_initialized(): + # pynccl_world_size = pynccl_utils.get_world_size() + # if pynccl_world_size != parallel_config.world_size: + # raise RuntimeError( + # "pynccl is already initialized but the pynccl world " + # "size does not match parallel_config.world_size " + # f"({pynccl_world_size} vs. {parallel_config.world_size}).") + # elif parallel_config.world_size > 1: + # # NOTE(woosuk): We don't initialize pynccl process group when world size + # # is 1. + # # NOTE(kaichao): By default, pynccl is initialized for tp group. + # pynccl_utils.init_process_group( + # group=get_tensor_model_parallel_cpu_group()) + + # # Initialize a custom fast all-reduce implementation. + # if not parallel_config.disable_custom_all_reduce: + # init_custom_ar() + + # A small all_reduce for warmup. + torch.distributed.all_reduce(torch.zeros(1).cuda()) + # if pynccl_utils.is_initialized(): + # pynccl_utils.all_reduce(torch.zeros(1).cuda()) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/__init__.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/arg_utils.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/arg_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..6c577277b8621421cb5e1c3dbb713dcb34519215 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/arg_utils.py @@ -0,0 +1,453 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py + +import os +import argparse +import dataclasses +import json +from dataclasses import dataclass +from typing import TYPE_CHECKING, List, Optional, Tuple, Type, Union + +import torch.nn as nn + +from transformers import PretrainedConfig +from .config import ModelConfig, LoadConfig + +from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig, EngineConfig, LoRAConfig, MultiModalConfig, + ObservabilityConfig, ParallelConfig, PromptAdapterConfig, SchedulerConfig, SpeculativeConfig, + TokenizerPoolConfig) +from vllm.executor.executor_base import ExecutorBase +from vllm.logger import init_logger +from vllm.utils import FlexibleArgumentParser +from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS +from vllm.utils import str_to_int_tuple + +if TYPE_CHECKING: + from vllm.transformers_utils.tokenizer_group.base_tokenizer_group import (BaseTokenizerGroup) + +logger = init_logger(__name__) + + +def nullable_str(val: str): + if not val or val == "None": + return None + return val + + +@dataclass +class EngineArgs: + """Arguments for vLLM engine.""" + model_hf_config: PretrainedConfig = None # for verl + served_model_name = None # TODO(sgm): check this + # tokenizer: Optional[str] = None # TODO(sgm): check this + skip_tokenizer_init: bool = False + tokenizer_mode: str = 'auto' + trust_remote_code: bool = False + download_dir: Optional[str] = None + load_format: str = 'auto' + dtype: str = 'auto' + kv_cache_dtype: str = 'auto' + quantization_param_path: Optional[str] = None + seed: int = 0 + max_model_len: Optional[int] = None + worker_use_ray: bool = False + # Note: Specifying a custom executor backend by passing a class + # is intended for expert use only. The API may change without + # notice. + distributed_executor_backend: Optional[Union[str, Type[ExecutorBase]]] = None + pipeline_parallel_size: int = 1 + tensor_parallel_size: int = 1 + max_parallel_loading_workers: Optional[int] = None + block_size: int = 16 + enable_prefix_caching: bool = False + disable_sliding_window: bool = False + use_v2_block_manager: bool = False + swap_space: int = 4 # GiB + cpu_offload_gb: int = 0 # GiB + gpu_memory_utilization: float = 0.90 + max_num_batched_tokens: Optional[int] = None + max_num_seqs: int = 256 + max_logprobs: int = 20 # Default value for OpenAI Chat Completions API + disable_log_stats: bool = False + revision: Optional[str] = None + code_revision: Optional[str] = None + rope_scaling: Optional[dict] = None + rope_theta: Optional[float] = None + tokenizer_revision: Optional[str] = None + quantization: Optional[str] = None + enforce_eager: bool = False + max_context_len_to_capture: Optional[int] = None + max_seq_len_to_capture: int = 8192 + disable_custom_all_reduce: bool = False + tokenizer_pool_size: int = 0 + # Note: Specifying a tokenizer pool by passing a class + # is intended for expert use only. The API may change without + # notice. + tokenizer_pool_type: Union[str, Type["BaseTokenizerGroup"]] = "ray" + tokenizer_pool_extra_config: Optional[dict] = None + enable_lora: bool = False + max_loras: int = 1 + max_lora_rank: int = 16 + enable_prompt_adapter: bool = False + max_prompt_adapters: int = 1 + max_prompt_adapter_token: int = 0 + fully_sharded_loras: bool = False + lora_extra_vocab_size: int = 256 + long_lora_scaling_factors: Optional[Tuple[float]] = None + lora_dtype: str = 'auto' + max_cpu_loras: Optional[int] = None + device: str = 'auto' + ray_workers_use_nsight: bool = False + num_gpu_blocks_override: Optional[int] = None + num_lookahead_slots: int = 0 + model_loader_extra_config: Optional[dict] = None + ignore_patterns: Optional[Union[str, List[str]]] = None + preemption_mode: Optional[str] = None + + scheduler_delay_factor: float = 0.0 + enable_chunked_prefill: Optional[bool] = None + + guided_decoding_backend: str = 'outlines' + # Speculative decoding configuration. + speculative_model: Optional[str] = None + speculative_draft_tensor_parallel_size: Optional[int] = None + num_speculative_tokens: Optional[int] = None + speculative_max_model_len: Optional[int] = None + speculative_disable_by_batch_size: Optional[int] = None + ngram_prompt_lookup_max: Optional[int] = None + ngram_prompt_lookup_min: Optional[int] = None + spec_decoding_acceptance_method: str = 'rejection_sampler' + typical_acceptance_sampler_posterior_threshold: Optional[float] = None + typical_acceptance_sampler_posterior_alpha: Optional[float] = None + qlora_adapter_name_or_path: Optional[str] = None + disable_logprobs_during_spec_decoding: Optional[bool] = None + + otlp_traces_endpoint: Optional[str] = None + + @staticmethod + def add_cli_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser: + """Shared CLI arguments for vLLM engine.""" + # Model arguments + # TODO(shengguangming): delete the unused args + parser.add_argument('--model', + type=str, + default='facebook/opt-125m', + help='name or path of the huggingface model to use') + parser.add_argument('--tokenizer', + type=str, + default=EngineArgs.tokenizer, + help='name or path of the huggingface tokenizer to use') + parser.add_argument('--revision', + type=str, + default=None, + help='the specific model version to use. It can be a branch ' + 'name, a tag name, or a commit id. If unspecified, will use ' + 'the default version.') + parser.add_argument('--tokenizer-revision', + type=str, + default=None, + help='the specific tokenizer version to use. It can be a branch ' + 'name, a tag name, or a commit id. If unspecified, will use ' + 'the default version.') + parser.add_argument('--tokenizer-mode', + type=str, + default=EngineArgs.tokenizer_mode, + choices=['auto', 'slow'], + help='tokenizer mode. "auto" will use the fast ' + 'tokenizer if available, and "slow" will ' + 'always use the slow tokenizer.') + parser.add_argument('--trust-remote-code', action='store_true', help='trust remote code from huggingface') + parser.add_argument('--download-dir', + type=str, + default=EngineArgs.download_dir, + help='directory to download and load the weights, ' + 'default to the default cache dir of ' + 'huggingface') + parser.add_argument('--load-format', + type=str, + default=EngineArgs.load_format, + choices=['auto', 'pt', 'safetensors', 'npcache', 'dummy'], + help='The format of the model weights to load. ' + '"auto" will try to load the weights in the safetensors format ' + 'and fall back to the pytorch bin format if safetensors format ' + 'is not available. ' + '"pt" will load the weights in the pytorch bin format. ' + '"safetensors" will load the weights in the safetensors format. ' + '"npcache" will load the weights in pytorch format and store ' + 'a numpy cache to speed up the loading. ' + '"dummy" will initialize the weights with random values, ' + 'which is mainly for profiling.') + parser.add_argument('--dtype', + type=str, + default=EngineArgs.dtype, + choices=['auto', 'half', 'float16', 'bfloat16', 'float', 'float32'], + help='data type for model weights and activations. ' + 'The "auto" option will use FP16 precision ' + 'for FP32 and FP16 models, and BF16 precision ' + 'for BF16 models.') + parser.add_argument('--max-model-len', + type=int, + default=None, + help='model context length. If unspecified, ' + 'will be automatically derived from the model.') + # Parallel arguments + parser.add_argument('--worker-use-ray', + action='store_true', + help='use Ray for distributed serving, will be ' + 'automatically set when using more than 1 GPU') + parser.add_argument('--pipeline-parallel-size', + '-pp', + type=int, + default=EngineArgs.pipeline_parallel_size, + help='number of pipeline stages') + parser.add_argument('--tensor-parallel-size', + '-tp', + type=int, + default=EngineArgs.tensor_parallel_size, + help='number of tensor parallel replicas') + # KV cache arguments + parser.add_argument('--block-size', + type=int, + default=EngineArgs.block_size, + choices=[8, 16, 32], + help='token block size') + # TODO(woosuk): Support fine-grained seeds (e.g., seed per request). + parser.add_argument('--seed', type=int, default=EngineArgs.seed, help='random seed') + parser.add_argument('--swap-space', + type=int, + default=EngineArgs.swap_space, + help='CPU swap space size (GiB) per GPU') + parser.add_argument('--gpu-memory-utilization', + type=float, + default=EngineArgs.gpu_memory_utilization, + help='the percentage of GPU memory to be used for' + 'the model executor') + parser.add_argument('--max-num-batched-tokens', + type=int, + default=EngineArgs.max_num_batched_tokens, + help='maximum number of batched tokens per ' + 'iteration') + parser.add_argument('--max-num-seqs', + type=int, + default=EngineArgs.max_num_seqs, + help='maximum number of sequences per iteration') + parser.add_argument('--disable-log-stats', action='store_true', help='disable logging statistics') + # Quantization settings. + parser.add_argument('--quantization', + '-q', + type=str, + choices=['awq', None], + default=None, + help='Method used to quantize the weights') + return parser + + @classmethod + def from_cli_args(cls, args: argparse.Namespace) -> 'EngineArgs': + # Get the list of attributes of this dataclass. + attrs = [attr.name for attr in dataclasses.fields(cls)] + # Set the attributes from the parsed arguments. + engine_args = cls(**{attr: getattr(args, attr) for attr in attrs}) + return engine_args + + def create_engine_config( + self, + ) -> EngineConfig: + # bitsandbytes quantization needs a specific model loader + # so we make sure the quant method and the load format are consistent + if (self.quantization == "bitsandbytes" or + self.qlora_adapter_name_or_path is not None) and \ + self.load_format != "bitsandbytes": + raise ValueError("BitsAndBytes quantization and QLoRA adapter only support " + f"'bitsandbytes' load format, but got {self.load_format}") + + if (self.load_format == "bitsandbytes" or + self.qlora_adapter_name_or_path is not None) and \ + self.quantization != "bitsandbytes": + raise ValueError("BitsAndBytes load format and QLoRA adapter only support " + f"'bitsandbytes' quantization, but got {self.quantization}") + + assert self.cpu_offload_gb >= 0, ("CPU offload space must be non-negative" + f", but got {self.cpu_offload_gb}") + + multimodal_config = MultiModalConfig() + device_config = DeviceConfig(self.device) + # NOTE(sgm): we only modify ModelConfig, other configs are import from vllm + model_config = ModelConfig(hf_config=self.model_hf_config, + tokenizer_mode=self.tokenizer_mode, + trust_remote_code=self.trust_remote_code, + dtype=self.dtype, + seed=self.seed, + revision=self.revision, + code_revision=self.code_revision, + rope_scaling=self.rope_scaling, + rope_theta=self.rope_theta, + tokenizer_revision=self.tokenizer_revision, + max_model_len=self.max_model_len, + quantization=self.quantization, + quantization_param_path=self.quantization_param_path, + enforce_eager=self.enforce_eager, + max_context_len_to_capture=self.max_context_len_to_capture, + max_seq_len_to_capture=self.max_seq_len_to_capture, + max_logprobs=self.max_logprobs, + disable_sliding_window=self.disable_sliding_window, + skip_tokenizer_init=self.skip_tokenizer_init, + served_model_name=self.served_model_name, + multimodal_config=multimodal_config) + cache_config = CacheConfig( + block_size=self.block_size, + gpu_memory_utilization=self.gpu_memory_utilization, + swap_space=self.swap_space, + cache_dtype=self.kv_cache_dtype, + num_gpu_blocks_override=self.num_gpu_blocks_override, + sliding_window=model_config.get_sliding_window(), + enable_prefix_caching=self.enable_prefix_caching, + cpu_offload_gb=self.cpu_offload_gb, + ) + parallel_config = ParallelConfig(pipeline_parallel_size=self.pipeline_parallel_size, + tensor_parallel_size=self.tensor_parallel_size, + worker_use_ray=self.worker_use_ray, + max_parallel_loading_workers=self.max_parallel_loading_workers, + disable_custom_all_reduce=self.disable_custom_all_reduce, + tokenizer_pool_config=TokenizerPoolConfig.create_config( + self.tokenizer_pool_size, + self.tokenizer_pool_type, + self.tokenizer_pool_extra_config, + ), + ray_workers_use_nsight=self.ray_workers_use_nsight, + distributed_executor_backend=self.distributed_executor_backend) + + # NOTE[VERL]: Use the world_size set by TORCHRUN + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + parallel_config.world_size = world_size + + max_model_len = model_config.max_model_len + use_long_context = max_model_len > 32768 + if self.enable_chunked_prefill is None: + # If not explicitly set, enable chunked prefill by default for + # long context (> 32K) models. This is to avoid OOM errors in the + # initial memory profiling phase. + if use_long_context: + is_gpu = device_config.device_type == "cuda" + use_sliding_window = (model_config.get_sliding_window() is not None) + use_spec_decode = self.speculative_model is not None + has_seqlen_agnostic_layers = (model_config.contains_seqlen_agnostic_layers(parallel_config)) + if (is_gpu and not use_sliding_window and not use_spec_decode and not self.enable_lora and + not self.enable_prompt_adapter and not self.enable_prefix_caching and + not has_seqlen_agnostic_layers): + self.enable_chunked_prefill = True + logger.warning("Chunked prefill is enabled by default for models with " + "max_model_len > 32K. Currently, chunked prefill might " + "not work with some features or models. If you " + "encounter any issues, please disable chunked prefill " + "by setting --enable-chunked-prefill=False.") + if self.enable_chunked_prefill is None: + self.enable_chunked_prefill = False + + if not self.enable_chunked_prefill and use_long_context: + logger.warning( + "The model has a long context length (%s). This may cause OOM " + "errors during the initial memory profiling phase, or result " + "in low performance due to small KV cache space. Consider " + "setting --max-model-len to a smaller value.", max_model_len) + + # TODO: spec config + speculative_config = SpeculativeConfig.maybe_create_spec_config( + target_model_config=model_config, + target_parallel_config=parallel_config, + target_dtype=self.dtype, + speculative_model=self.speculative_model, + speculative_draft_tensor_parallel_size = \ + self.speculative_draft_tensor_parallel_size, + num_speculative_tokens=self.num_speculative_tokens, + speculative_disable_by_batch_size=self. + speculative_disable_by_batch_size, + speculative_max_model_len=self.speculative_max_model_len, + enable_chunked_prefill=self.enable_chunked_prefill, + use_v2_block_manager=self.use_v2_block_manager, + disable_log_stats=self.disable_log_stats, + ngram_prompt_lookup_max=self.ngram_prompt_lookup_max, + ngram_prompt_lookup_min=self.ngram_prompt_lookup_min, + draft_token_acceptance_method=\ + self.spec_decoding_acceptance_method, + typical_acceptance_sampler_posterior_threshold=self. + typical_acceptance_sampler_posterior_threshold, + typical_acceptance_sampler_posterior_alpha=self. + typical_acceptance_sampler_posterior_alpha, + disable_logprobs=self.disable_logprobs_during_spec_decoding, + ) + + scheduler_config = SchedulerConfig( + max_num_batched_tokens=self.max_num_batched_tokens, + max_num_seqs=self.max_num_seqs, + max_model_len=model_config.max_model_len, + use_v2_block_manager=self.use_v2_block_manager, + num_lookahead_slots=(self.num_lookahead_slots + if speculative_config is None else speculative_config.num_lookahead_slots), + delay_factor=self.scheduler_delay_factor, + enable_chunked_prefill=self.enable_chunked_prefill, + embedding_mode=model_config.embedding_mode, + preemption_mode=self.preemption_mode, + ) + lora_config = LoRAConfig(max_lora_rank=self.max_lora_rank, + max_loras=self.max_loras, + fully_sharded_loras=self.fully_sharded_loras, + lora_extra_vocab_size=self.lora_extra_vocab_size, + long_lora_scaling_factors=self.long_lora_scaling_factors, + lora_dtype=self.lora_dtype, + max_cpu_loras=self.max_cpu_loras if self.max_cpu_loras and self.max_cpu_loras > 0 else + None) if self.enable_lora else None + + if self.qlora_adapter_name_or_path is not None and \ + self.qlora_adapter_name_or_path != "": + if self.model_loader_extra_config is None: + self.model_loader_extra_config = {} + self.model_loader_extra_config["qlora_adapter_name_or_path"] = self.qlora_adapter_name_or_path + + load_config = LoadConfig( + load_format=self.load_format, + download_dir=self.download_dir, + model_loader_extra_config=self.model_loader_extra_config, + ignore_patterns=self.ignore_patterns, + ) + + prompt_adapter_config = PromptAdapterConfig( + max_prompt_adapters=self.max_prompt_adapters, + max_prompt_adapter_token=self.max_prompt_adapter_token) \ + if self.enable_prompt_adapter else None + + decoding_config = DecodingConfig(guided_decoding_backend=self.guided_decoding_backend) + + observability_config = ObservabilityConfig(otlp_traces_endpoint=self.otlp_traces_endpoint) + + if (model_config.get_sliding_window() is not None and scheduler_config.chunked_prefill_enabled and + not scheduler_config.use_v2_block_manager): + raise ValueError("Chunked prefill is not supported with sliding window. " + "Set --disable-sliding-window to disable sliding window.") + + return EngineConfig( + model_config=model_config, + cache_config=cache_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + device_config=device_config, + lora_config=lora_config, + multimodal_config=multimodal_config, + speculative_config=speculative_config, + load_config=load_config, + decoding_config=decoding_config, + observability_config=observability_config, + prompt_adapter_config=prompt_adapter_config, + ) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/config.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/config.py new file mode 100644 index 0000000000000000000000000000000000000000..5fc61e6fe60661d7b5c4bfc77b5c1d3843997e46 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/config.py @@ -0,0 +1,246 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/config.py + +import enum +import json +from typing import List, Optional, Union +from dataclasses import dataclass, field, fields + +import torch +from transformers import PretrainedConfig + +from vllm.logger import init_logger +from vllm.model_executor.layers.quantization import get_quantization_config +from vllm.transformers_utils.config import get_hf_text_config +from vllm.utils import is_hip, print_warning_once +# Add for verl +from vllm.config import ModelConfig, _get_and_verify_dtype, _get_and_verify_max_len, get_served_model_name + +GPTQMarlinConfig = get_quantization_config("gptq_marlin") + +logger = init_logger(__name__) + +_GB = 1 << 30 + + +class ModelConfig(ModelConfig): + """Configuration for the model. + + Args: + model: Name or path of the huggingface model to use. + tokenizer: Name or path of the huggingface tokenizer to use. + tokenizer_mode: Tokenizer mode. "auto" will use the fast tokenizer if + available, and "slow" will always use the slow tokenizer. + trust_remote_code: Trust remote code (e.g., from HuggingFace) when + downloading the model and tokenizer. + download_dir: Directory to download and load the weights, default to the + default cache directory of huggingface. + load_format: The format of the model weights to load: + "auto" will try to load the weights in the safetensors format and + fall back to the pytorch bin format if safetensors format is + not available. + "pt" will load the weights in the pytorch bin format. + "safetensors" will load the weights in the safetensors format. + "npcache" will load the weights in pytorch format and store + a numpy cache to speed up the loading. + "dummy" will initialize the weights with random values, which is + mainly for profiling. + dtype: Data type for model weights and activations. The "auto" option + will use FP16 precision for FP32 and FP16 models, and BF16 precision + for BF16 models. + seed: Random seed for reproducibility. + revision: The specific model version to use. It can be a branch name, + a tag name, or a commit id. If unspecified, will use the default + version. + code_revision: The specific revision to use for the model code on + Hugging Face Hub. It can be a branch name, a tag name, or a + commit id. If unspecified, will use the default version. + tokenizer_revision: The specific tokenizer version to use. It can be a + branch name, a tag name, or a commit id. If unspecified, will use + the default version. + max_model_len: Maximum length of a sequence (including prompt and + output). If None, will be derived from the model. + quantization: Quantization method that was used to quantize the model + weights. If None, we assume the model weights are not quantized. + quantization_param_path: Path to JSON file containing scaling factors. + Used to load KV cache scaling factors into the model when KV cache + type is FP8_E4M3 on ROCm (AMD GPU). In the future these will also + be used to load activation and weight scaling factors when the + model dtype is FP8_E4M3 on ROCm. + enforce_eager: Whether to enforce eager execution. If True, we will + disable CUDA graph and always execute the model in eager mode. + If False, we will use CUDA graph and eager execution in hybrid. + max_context_len_to_capture: Maximum context len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode (DEPRECATED. Use max_seq_len_to_capture instead). + max_seq_len_to_capture: Maximum sequence len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode + skip_tokenizer_init: If true, skip initialization of tokenizer and + detokenizer. + served_model_name: The model name used in metrics tag `model_name`, + matches the model name exposed via the APIs. If multiple model + names provided, the first name will be used. If not specified, + the model name will be the same as `model`. + """ + + def __init__( + self, + hf_config: PretrainedConfig, + tokenizer_mode: str, + trust_remote_code: bool, + dtype: Union[str, torch.dtype], + seed: int, + revision: Optional[str] = None, + code_revision: Optional[str] = None, + rope_scaling: Optional[dict] = None, + rope_theta: Optional[float] = None, + tokenizer_revision: Optional[str] = None, + max_model_len: Optional[int] = None, + quantization: Optional[str] = None, + quantization_param_path: Optional[str] = None, + enforce_eager: bool = False, + max_context_len_to_capture: Optional[int] = None, + max_seq_len_to_capture: Optional[int] = None, + max_logprobs: int = 20, + disable_sliding_window: bool = False, + skip_tokenizer_init: bool = False, + served_model_name: Optional[Union[str, List[str]]] = None, + multimodal_config: Optional["MultiModalConfig"] = None, + ) -> None: + self.model = hf_config._name_or_path + self.tokenizer = hf_config._name_or_path + # NOTE(sgm): same as open-sourced + self.tokenizer_mode = tokenizer_mode + self.trust_remote_code = trust_remote_code + self.seed = seed + self.revision = revision + self.code_revision = code_revision + self.rope_scaling = rope_scaling + self.rope_theta = rope_theta + # The tokenizer version is consistent with the model version by default. + if tokenizer_revision is None: + self.tokenizer_revision = revision + else: + self.tokenizer_revision = tokenizer_revision + self.quantization = quantization + self.quantization_param_path = quantization_param_path + self.enforce_eager = enforce_eager + if max_context_len_to_capture is not None: + raise ValueError("`max_context_len_to_capture` is deprecated. " + "Use `max_seq_len_to_capture` instead.") + self.max_seq_len_to_capture = max_seq_len_to_capture + self.max_logprobs = max_logprobs + self.disable_sliding_window = disable_sliding_window + self.skip_tokenizer_init = skip_tokenizer_init + + # self.hf_config = get_config(model, trust_remote_code, revision) + self.hf_config = hf_config + self.hf_text_config = get_hf_text_config(hf_config) + self.dtype = _get_and_verify_dtype(self.hf_text_config, dtype) + # self.served_model_name = get_served_model_name(model, + # served_model_name) + # self._verify_load_format() + # self._verify_tokenizer_mode() + if (not self.disable_sliding_window and self.hf_text_config.model_type == "gemma2" and + self.hf_text_config.sliding_window is not None): + print_warning_once("Gemma 2 uses sliding window attention for every odd layer, " + "which is currently not supported by vLLM. Disabling sliding " + "window and capping the max length to the sliding window size " + f"({self.hf_text_config.sliding_window}).") + self.disable_sliding_window = True + + self.max_model_len = _get_and_verify_max_len(hf_config=self.hf_text_config, + max_model_len=max_model_len, + disable_sliding_window=self.disable_sliding_window, + sliding_window_len=self.get_hf_config_sliding_window()) + self.served_model_name = get_served_model_name( + self.model, # str + served_model_name) + self.multimodal_config = multimodal_config + + if not self.skip_tokenizer_init: + self._verify_tokenizer_mode() + self._verify_embedding_mode() + self._verify_quantization() + self._verify_cuda_graph() + + +class LoadFormat(str, enum.Enum): + AUTO = 'auto' + MEGATRON = "megatron" + HF = "hf" + DTENSOR = 'dtensor' + DUMMY_HF = 'dummy_hf' + DUMMY_MEGATRON = 'dummy_megatron' + DUMMY_DTENSOR = 'dummy_dtensor' + + +# TODO: check whether this is necessary +@dataclass +class LoadConfig: + """ + download_dir: Directory to download and load the weights, default to the + default cache directory of huggingface. + load_format: The format of the model weights to load: + "auto" will try to load the weights in the safetensors format and + fall back to the pytorch bin format if safetensors format is + not available. + "pt" will load the weights in the pytorch bin format. + "safetensors" will load the weights in the safetensors format. + "npcache" will load the weights in pytorch format and store + a numpy cache to speed up the loading. + "dummy" will initialize the weights with random values, which is + mainly for profiling. + "tensorizer" will use CoreWeave's tensorizer library for + fast weight loading. + "bitsandbytes" will load nf4 type weights. + ignore_patterns: The list of patterns to ignore when loading the model. + Default to "original/**/*" to avoid repeated loading of llama's + checkpoints. + + """ + + load_format: Union[str, LoadFormat, "BaseModelLoader"] = LoadFormat.AUTO + download_dir: Optional[str] = None + model_loader_extra_config: Optional[Union[str, dict]] = field(default_factory=dict) + ignore_patterns: Optional[Union[List[str], str]] = None + + def __post_init__(self): + model_loader_extra_config = self.model_loader_extra_config or {} + if isinstance(model_loader_extra_config, str): + self.model_loader_extra_config = json.loads(model_loader_extra_config) + self._verify_load_format() + + if self.ignore_patterns is not None and len(self.ignore_patterns) > 0: + logger.info("Ignoring the following patterns when downloading weights: %s", self.ignore_patterns) + else: + self.ignore_patterns = ["original/**/*"] + + def _verify_load_format(self) -> None: + if not isinstance(self.load_format, str): + return + + load_format = self.load_format.lower() + self.load_format = LoadFormat(load_format) + + rocm_not_supported_load_format: List[str] = [] + if is_hip() and load_format in rocm_not_supported_load_format: + rocm_supported_load_format = [ + f for f in LoadFormat.__members__ if (f not in rocm_not_supported_load_format) + ] + raise ValueError(f"load format '{load_format}' is not supported in ROCm. " + f"Supported load formats are " + f"{rocm_supported_load_format}") diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/dtensor_weight_loaders.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/dtensor_weight_loaders.py new file mode 100644 index 0000000000000000000000000000000000000000..732b543db6347d2f22db22745a3a7c037636737e --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/dtensor_weight_loaders.py @@ -0,0 +1,340 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models + +from typing import Dict, Iterable, Tuple +import torch +import torch.nn as nn +from torch.distributed._tensor import DTensor, Shard, Replicate + +from vllm.model_executor.layers.linear import * +from vllm.model_executor.models import ModelRegistry +from vllm.model_executor.model_loader.weight_utils import default_weight_loader +from vllm.model_executor.models.utils import is_pp_missing_parameter + + +def gemma_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + for (param_name, shard_name, shard_id) in stacked_params_mapping: + if shard_name not in name: + continue + stacked_name = name.replace(shard_name, param_name) + # Skip loading extra bias for GPTQ models. + if stacked_name.endswith(".bias") and stacked_name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[stacked_name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # lm_head is not used in vllm as it is tied with embed_token. + # To prevent errors, skip loading lm_head.weight. + if "lm_head.weight" in name: + continue + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def gptbigcode_dtensor_load_weights(actor_weights: Dict, vllm_model: nn.Module): + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "lm_head.weight" in name: + continue + if ".attn.bias" in name: + # Skip attention mask. + # NOTE: "c_attn.bias" should not be skipped. + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def starcoder2_dtensor_load_weights(actor_weights: Dict, vllm_model: nn.Module): + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ] + + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + + for (param_name, weight_name, shard_id) in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def llama_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + (".qkv_proj", ".q_proj", "q"), + (".qkv_proj", ".k_proj", "k"), + (".qkv_proj", ".v_proj", "v"), + (".gate_up_proj", ".gate_proj", 0), + (".gate_up_proj", ".up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if ("rotary_emb.cos_cached" in name or "rotary_emb.sin_cached" in name): + # Models trained using ColossalAI may include these tensors in + # the checkpoint. Skip them. + continue + # With tie_word_embeddings, we can skip lm_head.weight + # The weight might appear unnecessarily in the files if the model is + # processed with quantization, LoRA, fine-tuning, etc. + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for (param_name, weight_name, shard_id) in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight) + + +def qwen2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for (param_name, weight_name, shard_id) in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +from vllm.model_executor.layers.fused_moe import FusedMoE + + +def deepseekv2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + + # Params for weights, fp8 weight scales, fp8 activation scales + # (param_name, weight_name, expert_id, shard_id) + expert_params_mapping = FusedMoE.make_expert_params_mapping(ckpt_gate_proj_name="gate_proj", + ckpt_down_proj_name="down_proj", + ckpt_up_proj_name="up_proj", + num_experts=vllm_model.config.n_routed_experts) + + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + for (param_name, weight_name, shard_id) in stacked_params_mapping: + # Skip non-stacked layers and experts (experts handled below). + if weight_name not in name: + continue + # We have mlp.experts[0].gate_proj in the checkpoint. + # Since we handle the experts below in expert_params_mapping, + # we need to skip here BEFORE we update the name, otherwise + # name will be updated to mlp.experts[0].gate_up_proj, which + # will then be updated below in expert_params_mapping + # for mlp.experts[0].gate_gate_up_proj, which breaks load. + if (("mlp.experts." in name) and name not in params_dict): + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + for mapping in expert_params_mapping: + param_name, weight_name, expert_id, shard_id = mapping + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, + local_loaded_weight.to(dtype=param.dtype), + weight_name, + shard_id=shard_id, + expert_id=expert_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def gpt2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + pass + + +def redistribute_dtensor(param_name: str, loaded_weights: DTensor, parallelize_plan: Dict = None): + param_name = _process_parameter_names(name=param_name) + if parallelize_plan is not None: + assert param_name in parallelize_plan.keys(), \ + f"param name: {param_name} not in parallelize_plan :{parallelize_plan.keys()}" + placement = parallelize_plan[param_name] + local_loaded_weights = loaded_weights.redistribute(device_mesh=loaded_weights.device_mesh, + placements=placement).to_local() + else: + local_loaded_weights = loaded_weights.full_tensor() + return local_loaded_weights + + +def _process_parameter_names(name): + # Remove '.weight' if it exists at the end of the string + if name.endswith(".weight"): + name = name[:-7] + + # Remove 'model.layers.x.' or 'model.' prefix + if "model.layers" in name: + parts = name.split('.') + # Reconstruct the string without 'model.layers.x.' + name = '.'.join(parts[3:]) # parts[0] is 'model', parts[1] is 'layers', parts[2] is 'x' + elif name.startswith("model."): + name = name[6:] # Remove 'model.' + + return name + + +__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__ = { + 'GPT2LMHeadModel': gpt2_dtensor_weight_loader, + 'LlamaForCausalLM': llama_dtensor_weight_loader, + 'LLaMAForCausalLM': llama_dtensor_weight_loader, + 'MistralForCausalLM': llama_dtensor_weight_loader, # mistral is the same as llama in vLLM + 'InternLMForCausalLM': llama_dtensor_weight_loader, + 'AquilaModel': llama_dtensor_weight_loader, + 'AquilaForCausalLM': llama_dtensor_weight_loader, + 'Phi3ForCausalLM': llama_dtensor_weight_loader, + 'GemmaForCausalLM': gemma_dtensor_weight_loader, + 'Gemma2ForCausalLM': gemma_dtensor_weight_loader, + 'GPTBigCodeForCausalLM': gptbigcode_dtensor_load_weights, + 'Starcoder2ForCausalLM': starcoder2_dtensor_load_weights, + 'Qwen2ForCausalLM': qwen2_dtensor_weight_loader, + 'DeepseekV2ForCausalLM': deepseekv2_dtensor_weight_loader +} + + +# the actor model is .state_dict() +# Load dtensor weights +def load_dtensor_weights(actor_weights: Dict, vllm_model: nn.Module): + weight_loader = _get_model_weight_loader(vllm_model.__class__.__name__) + weight_loader(actor_weights, vllm_model) + # NOTE(sgm) to reduce peak memory usage, we offload vllm model to cpu + # after init, and we need this after sync model weights for in first iter. + vllm_model = vllm_model.cuda() + + +def _get_model_weight_loader(arch: str): + if arch in __MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__: + return __MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__.keys()}") + + +# NOTE(sgm): we use per-parameter weight loader in each vllm sub +def update_dtensor_weight_loader(): + pass diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/hf_weight_loader.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/hf_weight_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..7af4953f35e7107c9e7e6cd4f597b4a2715d441d --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/hf_weight_loader.py @@ -0,0 +1,44 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models + +from typing import Dict, Union, Optional, Iterable, Tuple + +import torch +import torch.nn as nn + +from vllm.model_executor.model_loader.utils import set_default_torch_dtype +from vllm.model_executor.model_loader.weight_utils import default_weight_loader + + +def update_hf_weight_loader(): + print('no hf weight loader need to be updated') + return + + +def load_hf_weights(actor_weights: Dict, vllm_model: nn.Module): + assert isinstance(actor_weights, Dict) + with set_default_torch_dtype(next(vllm_model.parameters()).dtype): # TODO + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in actor_weights.keys(): + del actor_weights["lm_head.weight"] + vllm_model.load_weights(actor_weights.items()) + for _, module in vllm_model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + vllm_model = vllm_model.cuda() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/llm.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/llm.py new file mode 100644 index 0000000000000000000000000000000000000000..5f56f1e07af01e646e3a096e8a3b931a43dc3747 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/llm.py @@ -0,0 +1,239 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py + +from contextlib import contextmanager +from typing import ClassVar, List, Optional, Sequence, Union, cast, overload, Dict, Tuple + +from tqdm import tqdm +from transformers import PreTrainedTokenizer, PreTrainedTokenizerFast +from transformers import PretrainedConfig +import torch.nn as nn +from .arg_utils import EngineArgs +from .llm_engine_sp import LLMEngine +from vllm import LLM +from vllm.inputs import (PromptInputs, TextPrompt, TokensPrompt, parse_and_batch_prompt) +from vllm.logger import init_logger +from vllm.lora.request import LoRARequest +from vllm.model_executor.guided_decoding import (GuidedDecodingRequest, get_local_guided_decoding_logits_processor) +from vllm.model_executor.guided_decoding.guided_fields import LLMGuidedOptions +from vllm.outputs import EmbeddingRequestOutput, RequestOutput +from vllm.pooling_params import PoolingParams +from vllm.prompt_adapter.request import PromptAdapterRequest +from vllm.sampling_params import SamplingParams +from vllm.transformers_utils.tokenizer import get_cached_tokenizer +from vllm.usage.usage_lib import UsageContext +from vllm.utils import Counter, deprecate_kwargs +import torch +from torch.nn.utils.rnn import pad_sequence +from verl.workers.rollout.tokenizer import HybridEngineBaseTokenizer + + +class LLM(LLM): + """An LLM for generating texts from given prompts and sampling parameters. + + This class includes a tokenizer, a language model (possibly distributed + across multiple GPUs), and GPU memory space allocated for intermediate + states (aka KV cache). Given a batch of prompts and sampling parameters, + this class generates texts from the model, using an intelligent batching + mechanism and efficient memory management. + + NOTE: This class is intended to be used for offline inference. For online + serving, use the `AsyncLLMEngine` class instead. + NOTE: For the comprehensive list of arguments, see `EngineArgs`. + + Args: + model: A HuggingFace Transformers model instance. + tokenizer: A HuggingFace Transformers tokenizer instance. + tokenizer_mode: The tokenizer mode. "auto" will use the fast tokenizer + if available, and "slow" will always use the slow tokenizer. + trust_remote_code: Trust remote code (e.g., from HuggingFace) when + downloading the model and tokenizer. + tensor_parallel_size: The number of GPUs to use for distributed + execution with tensor parallelism. + dtype: The data type for the model weights and activations. Currently, + we support `float32`, `float16`, and `bfloat16`. If `auto`, we use + the `torch_dtype` attribute specified in the model config file. + However, if the `torch_dtype` in the config is `float32`, we will + use `float16` instead. + quantization: The method used to quantize the model weights. Currently, + we support "awq". If None, we assume the model weights are not + quantized and use `dtype` to determine the data type of the weights. + revision: The specific model version to use. It can be a branch name, + a tag name, or a commit id. + tokenizer_revision: The specific tokenizer version to use. It can be a + branch name, a tag name, or a commit id. + seed: The seed to initialize the random number generator for sampling. + gpu_memory_utilization: The ratio (between 0 and 1) of GPU memory to + reserve for the model weights, activations, and KV cache. Higher + values will increase the KV cache size and thus improve the model's + throughput. However, if the value is too high, it may cause out-of- + memory (OOM) errors. + swap_space: The size (GiB) of CPU memory per GPU to use as swap space. + This can be used for temporarily storing the states of the requests + when their `best_of` sampling parameters are larger than 1. If all + requests will have `best_of=1`, you can safely set this to 0. + Otherwise, too small values may cause out-of-memory (OOM) errors. + enforce_eager: Whether to enforce eager execution. If True, we will + disable CUDA graph and always execute the model in eager mode. + If False, we will use CUDA graph and eager execution in hybrid. + max_context_len_to_capture: Maximum context len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode. + disable_custom_all_reduce: See ParallelConfig + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast, HybridEngineBaseTokenizer], + model_hf_config: PretrainedConfig, + tokenizer_mode: str = "auto", + trust_remote_code: bool = False, + skip_tokenizer_init: bool = False, + tensor_parallel_size: int = 1, + dtype: str = "auto", + quantization: Optional[str] = None, + revision: Optional[str] = None, + tokenizer_revision: Optional[str] = None, + seed: int = 0, + gpu_memory_utilization: float = 0.9, + swap_space: int = 4, + cpu_offload_gb: float = 0, + enforce_eager: bool = False, + max_context_len_to_capture: Optional[int] = None, + max_seq_len_to_capture: int = 8192, + disable_custom_all_reduce: bool = False, + load_format = 'auto', + **kwargs, + ) -> None: + if "disable_log_stats" not in kwargs: + kwargs["disable_log_stats"] = True + engine_args = EngineArgs( + model_hf_config=model_hf_config, + tensor_parallel_size=tensor_parallel_size, + dtype=dtype, + quantization=quantization, + revision=revision, + tokenizer_revision=tokenizer_revision, + seed=seed, + gpu_memory_utilization=gpu_memory_utilization, + swap_space=swap_space, + cpu_offload_gb=cpu_offload_gb, + enforce_eager=enforce_eager, + max_context_len_to_capture=max_context_len_to_capture, + max_seq_len_to_capture=max_seq_len_to_capture, + disable_custom_all_reduce=disable_custom_all_reduce, + load_format=load_format, + skip_tokenizer_init=skip_tokenizer_init, + **kwargs, + ) + tokenizer_cls = (PreTrainedTokenizer, PreTrainedTokenizerFast, HybridEngineBaseTokenizer) + if not isinstance(tokenizer, tokenizer_cls): + raise ValueError( + f"Unexpected tokenizer type: {type(tokenizer)}. Must be" + "one of the following: PreTrainedTokenizer, PreTrainedTokenizerFast, verl.workers.rollout.HybridEngineBaseTokenizer" + ) + self.llm_engine = LLMEngine.from_engine_args(model, tokenizer, engine_args) # TODO: check usagecontext + self.request_counter = Counter() + + def init_cache_engine(self): + self.llm_engine.init_cache_engine() + + def free_cache_engine(self): + self.llm_engine.free_cache_engine() + + def get_tokenizer(self) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]: + return self.llm_engine.tokenizer + + def set_tokenizer( + self, + tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], + ) -> None: + self.llm_engine.tokenizer = tokenizer + + def _run_engine(self, *, use_tqdm: bool) -> List[Union[RequestOutput, EmbeddingRequestOutput]]: + # Initialize tqdm. + if use_tqdm: + num_requests = self.llm_engine.get_num_unfinished_requests() + pbar = tqdm( + total=num_requests, + desc="Processed prompts", + dynamic_ncols=True, + postfix=(f"est. speed input: {0:.2f} toks/s, " + f"output: {0:.2f} toks/s"), + ) + # Run the engine. + outputs: List[Union[RequestOutput, EmbeddingRequestOutput]] = [] + total_in_toks = 0 + total_out_toks = 0 + while self.llm_engine.has_unfinished_requests(): + step_outputs = self.llm_engine.step() + for output in step_outputs: + if output.finished: + outputs.append(output) + if use_tqdm: + if isinstance(output, RequestOutput): + # Calculate tokens only for RequestOutput + total_in_toks += len(output.prompt_token_ids) + in_spd = total_in_toks / pbar.format_dict["elapsed"] + total_out_toks += sum(len(stp.token_ids) for stp in output.outputs) + out_spd = total_out_toks / pbar.format_dict["elapsed"] + pbar.postfix = (f"est. speed input: {in_spd:.2f} toks/s, " + f"output: {out_spd:.2f} toks/s") + pbar.update(1) + if use_tqdm: + pbar.close() + # Sort the outputs by request ID. + # This is necessary because some requests may be finished earlier than + # its previous requests. + outputs = sorted(outputs, key=lambda x: int(x.request_id)) + return self._post_process_outputs(outputs) + + # # NOTE(shengguangming): add for verl + # # TODO(sgm): we can optimize it by making the dataloader yield List[int] without padding. + # def _pre_process_inputs(self, prompt_token_ids: torch.Tensor) -> List[int]: + # # remove the left padding in the prompt token_id + # pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id + # non_pad_index = torch.nonzero(prompt_token_ids != pad_token_id, as_tuple=False)[0][0] + # token_ids = prompt_token_ids[non_pad_index:].tolist() + # return token_ids + + # NOTE(shengguangming): add for verl + def _post_process_outputs(self, request_outputs: List[RequestOutput]) -> Tuple[torch.Tensor, torch.Tensor]: + output_token_ids = [] + logprobs = [] + for request_output in request_outputs: # List[RequestOutput] + outputs = request_output.outputs + for output in outputs: # List[CompletionOutput], usually len == 1 + output_token_ids.append(torch.tensor(output.token_ids)) + # TODO(shengguangming): can be optimzied by rewrite the Sampler._get_logprobs() logits + logprobs_dicts = output.logprobs + if logprobs_dicts is not None: + logprob = [] + for logprobs_dict, id in zip(logprobs_dicts, output.token_ids): + logprob.append(logprobs_dict[id].logprob) + logprobs.append(torch.tensor(logprob)) + + pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id + output_token_ids = pad_sequence(output_token_ids, batch_first=True, padding_value=pad_token_id) + if len(logprobs) > 0: + logprobs = pad_sequence(logprobs, batch_first=True, padding_value=pad_token_id) + return output_token_ids, logprobs + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.llm_engine.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + def offload_model_weights(self) -> None: + self.llm_engine.offload_model_weights() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/llm_engine_sp.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/llm_engine_sp.py new file mode 100644 index 0000000000000000000000000000000000000000..8d161e747066ece4b19984dac4aecfa32cecf6e5 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/llm_engine_sp.py @@ -0,0 +1,328 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/engine/llm_engine.py + +import torch +from typing import Dict, Optional, Union, Type + +import vllm.envs as envs +from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig, EngineConfig, LoRAConfig, MultiModalConfig, + ObservabilityConfig, ParallelConfig, PromptAdapterConfig, SchedulerConfig, SpeculativeConfig) +from vllm.core.scheduler import Scheduler +from vllm.engine.output_processor.interfaces import (SequenceGroupOutputProcessor) +from vllm.engine.output_processor.stop_checker import StopChecker +from vllm.executor.executor_base import ExecutorBase +from vllm.inputs import INPUT_REGISTRY, LLMInputs, PromptInputs +from vllm.logger import init_logger +from vllm.transformers_utils.detokenizer import Detokenizer +from vllm.engine.metrics import (LoggingStatLogger, PrometheusStatLogger, StatLoggerBase, Stats) +from vllm.tracing import (SpanAttributes, SpanKind, extract_trace_context, init_tracer) +from vllm.usage.usage_lib import (UsageContext, is_usage_stats_enabled, usage_message) +from vllm.utils import Counter +from vllm.engine.llm_engine import _load_generation_config_dict +from vllm.engine.llm_engine import LLMEngine +from vllm.version import __version__ as VLLM_VERSION + +import torch.nn as nn +from .arg_utils import EngineArgs +from .tokenizer import TokenizerGroup +from .config import ModelConfig, LoadConfig + +logger = init_logger(__name__) +_LOCAL_LOGGING_INTERVAL_SEC = 5 + + +class LLMEngine(LLMEngine): + """An LLM engine that receives requests and generates texts. + + This is the main class for the vLLM engine. It receives requests + from clients and generates texts from the LLM. It includes a tokenizer, a + language model (possibly distributed across multiple GPUs), and GPU memory + space allocated for intermediate states (aka KV cache). This class utilizes + iteration-level scheduling and efficient memory management to maximize the + serving throughput. + + The `LLM` class wraps this class for offline batched inference and the + `AsyncLLMEngine` class wraps this class for online serving. + + NOTE: The config arguments are derived from the `EngineArgs` class. For the + comprehensive list of arguments, see `EngineArgs`. + + Args: + model: the actor model initialize outside vllm (add for verl) + tokenizer: the initialized tokenizer (add for verl) + model_config: The configuration related to the LLM model. + cache_config: The configuration related to the KV cache memory + management. + parallel_config: The configuration related to distributed execution. + scheduler_config: The configuration related to the request scheduler. + distributed_init_method: The initialization method for distributed + execution. See `torch.distributed.init_process_group` for details. + placement_group: Ray placement group for distributed execution. + Required for distributed execution. + log_stats: Whether to log statistics. + """ + + def __init__( + self, + # NOTE(sgm): first two arguments are added for verl + model: Union[nn.Module, Dict], # model itself or its parameter dict + tokenizer: nn.Module, + # NOTE(sgm): vllm original arguments + model_config: ModelConfig, + cache_config: CacheConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + multimodal_config: Optional[MultiModalConfig], + speculative_config: Optional[SpeculativeConfig], + decoding_config: Optional[DecodingConfig], + observability_config: Optional[ObservabilityConfig], + prompt_adapter_config: Optional[PromptAdapterConfig], + executor_class: Type[ExecutorBase], + log_stats: bool, + usage_context: UsageContext = UsageContext.ENGINE_CONTEXT, + stat_loggers: Optional[Dict[str, StatLoggerBase]] = None, + ) -> None: + logger.info( + "Initializing an LLM engine (v%s) with config: " + "model=%r, speculative_config=%r, tokenizer=%r, " + "skip_tokenizer_init=%s, revision=%s, " + "rope_scaling=%r, rope_theta=%r, tokenizer_revision=%s, " + "trust_remote_code=%s, dtype=%s, max_seq_len=%d, " + "download_dir=%r, load_format=%s, tensor_parallel_size=%d, " + "pipeline_parallel_size=%d, " + "disable_custom_all_reduce=%s, quantization=%s, " + "enforce_eager=%s, kv_cache_dtype=%s, " + "quantization_param_path=%s, device_config=%s, " + "decoding_config=%r, observability_config=%r, " + "seed=%d, served_model_name=%s, use_v2_block_manager=%s, " + "enable_prefix_caching=%s)", + VLLM_VERSION, + model_config.model, + speculative_config, + model_config.tokenizer, + model_config.skip_tokenizer_init, + model_config.revision, + model_config.rope_scaling, + model_config.rope_theta, + model_config.tokenizer_revision, + model_config.trust_remote_code, + model_config.dtype, + model_config.max_model_len, + load_config.download_dir, + load_config.load_format, + parallel_config.tensor_parallel_size, + parallel_config.pipeline_parallel_size, + parallel_config.disable_custom_all_reduce, + model_config.quantization, + model_config.enforce_eager, + cache_config.cache_dtype, + model_config.quantization_param_path, + device_config.device, + decoding_config, + observability_config, + model_config.seed, + model_config.served_model_name, + scheduler_config.use_v2_block_manager, + cache_config.enable_prefix_caching, + ) + # TODO(woosuk): Print more configs in debug mode. + + self.model_config = model_config + self.cache_config = cache_config + self.lora_config = lora_config + self.multimodal_config = multimodal_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.device_config = device_config + self.speculative_config = speculative_config + self.load_config = load_config + self.decoding_config = decoding_config or DecodingConfig() + self.prompt_adapter_config = prompt_adapter_config + self.observability_config = observability_config or ObservabilityConfig() + self.log_stats = log_stats + + # self.model = model # should not store the model, it should be deleted + # TODO(shengguangming): maybe we can choose init here or from arguments + if not self.model_config.skip_tokenizer_init: + self.tokenizer = self._init_tokenizer(tokenizer) + self.detokenizer = Detokenizer(self.tokenizer) + else: + self.tokenizer = None + self.detokenizer = None + + self.seq_counter = Counter() + self.generation_config_fields = _load_generation_config_dict(model_config) + + self.input_processor = INPUT_REGISTRY.create_input_processor(self.model_config) + + self.model_executor = executor_class( + model=model, # add for spmd_gpu_executor + model_config=model_config, + cache_config=cache_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + device_config=device_config, + lora_config=lora_config, + multimodal_config=multimodal_config, + speculative_config=speculative_config, + load_config=load_config, + prompt_adapter_config=prompt_adapter_config, + ) + + # Profile the memory usage and initialize the cache. + if not self.model_config.embedding_mode: + self._initialize_kv_caches() + + # If usage stat is enabled, collect relevant info. + if is_usage_stats_enabled(): + from vllm.model_executor.model_loader import (get_architecture_class_name) + usage_message.report_usage( + get_architecture_class_name(model_config), + usage_context, + extra_kvs={ + # Common configuration + "dtype": str(model_config.dtype), + "tensor_parallel_size": parallel_config.tensor_parallel_size, + "block_size": cache_config.block_size, + "gpu_memory_utilization": cache_config.gpu_memory_utilization, + + # Quantization + "quantization": model_config.quantization, + "kv_cache_dtype": str(cache_config.cache_dtype), + + # Feature flags + "enable_lora": bool(lora_config), + "enable_prompt_adapter": bool(prompt_adapter_config), + "enable_prefix_caching": cache_config.enable_prefix_caching, + "enforce_eager": model_config.enforce_eager, + "disable_custom_all_reduce": parallel_config.disable_custom_all_reduce, + }) + + if self.tokenizer: + # Ping the tokenizer to ensure liveness if it runs in a + # different process. + self.tokenizer.ping() + + # Create the scheduler. + # NOTE: the cache_config here have been updated with the numbers of + # GPU and CPU blocks, which are profiled in the distributed executor. + self.scheduler = [ + Scheduler(scheduler_config, cache_config, lora_config, parallel_config.pipeline_parallel_size) + for _ in range(parallel_config.pipeline_parallel_size) + ] + + # Metric Logging. + if self.log_stats: + if stat_loggers is not None: + self.stat_loggers = stat_loggers + else: + self.stat_loggers = { + "logging": + LoggingStatLogger(local_interval=_LOCAL_LOGGING_INTERVAL_SEC), + "prometheus": + PrometheusStatLogger(local_interval=_LOCAL_LOGGING_INTERVAL_SEC, + labels=dict(model_name=model_config.served_model_name), + max_model_len=self.model_config.max_model_len), + } + self.stat_loggers["prometheus"].info("cache_config", self.cache_config) + + self.tracer = None + if self.observability_config.otlp_traces_endpoint: + self.tracer = init_tracer("vllm.llm_engine", self.observability_config.otlp_traces_endpoint) + + # Create sequence output processor, e.g. for beam search or + # speculative decoding. + self.output_processor = (SequenceGroupOutputProcessor.create_output_processor( + self.scheduler_config, + self.detokenizer, + self.scheduler, + self.seq_counter, + self.get_tokenizer_for_seq, + stop_checker=StopChecker( + self.scheduler_config.max_model_len, + self.get_tokenizer_for_seq, + ), + )) + + # TODO(sgm): add for verl but we may not tokenizer in Rollout + def _init_tokenizer(self, tokenizer, **tokenizer_init_kwargs): + init_kwargs = dict(enable_lora=bool(self.lora_config), + max_num_seqs=self.scheduler_config.max_num_seqs, + max_input_length=None) + init_kwargs.update(tokenizer_init_kwargs) + return TokenizerGroup(tokenizer, **init_kwargs) + + def init_cache_engine(self): + # TODO: check whether we should rebuild the CUDAGraph every iter when offload/load KVCache + # Re-capture CUDAGraph would be time-consuming + self.model_executor.init_cache_engine() + + def free_cache_engine(self): + self.model_executor.free_cache_engine() + + # NOTE(sgm): currently, we only support GPU executor + # The GPUExecutor remove the Ray dependency + @classmethod + def _get_executor_cls(cls, engine_config: EngineConfig) -> Type[ExecutorBase]: + assert engine_config.device_config.device_type == "cuda", \ + "Currently, the vllm in verl only support running on GPU" + + if engine_config.parallel_config.world_size == 1: + engine_config.load_config.load_format = "dummy_hf" + + from .spmd_gpu_executor import SPMDGPUExecutor + executor_class = SPMDGPUExecutor + return executor_class + + @classmethod + def from_engine_args( + cls, + model, + tokenizer, + engine_args: EngineArgs, + usage_context: UsageContext = UsageContext.ENGINE_CONTEXT, + stat_loggers: Optional[Dict[str, StatLoggerBase]] = None, + ) -> "LLMEngine": + """Creates an LLM engine from the engine arguments.""" + # Create the engine configs. + engine_config = engine_args.create_engine_config() + executor_class = cls._get_executor_cls(engine_config) + # Initialize the cluster and specify the executor class. + assert engine_config.device_config.device_type == "cuda", \ + "Currently, the vllm in verl only support running on GPU" + + from .spmd_gpu_executor import SPMDGPUExecutor + executor_class = SPMDGPUExecutor + + # Create the LLM engine. + engine = cls( + model, + tokenizer, + **engine_config.to_dict(), + executor_class=executor_class, + log_stats=not engine_args.disable_log_stats, + usage_context=usage_context, + stat_loggers=stat_loggers, + ) + return engine + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.model_executor.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + def offload_model_weights(self) -> None: + self.model_executor.offload_model_weights() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/megatron_weight_loaders.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/megatron_weight_loaders.py new file mode 100644 index 0000000000000000000000000000000000000000..4f2b19a904e77a9c2d10e259d061f797da67ddd8 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/megatron_weight_loaders.py @@ -0,0 +1,307 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models + +from typing import Dict +import torch +import torch.nn as nn + +from vllm.model_executor.layers.linear import * +from vllm.model_executor.layers.vocab_parallel_embedding import VocabParallelEmbedding, ParallelLMHead +from vllm.model_executor.layers.activation import ScaledActivation +from vllm.model_executor.models import ModelRegistry + + +# NOTE(shengguangming): replace the origin weight loader function in the class +def parallel_weight_loader(self, param: torch.Tensor, loaded_weight: torch.Tensor) -> None: + """Parallel Linear weight loader.""" + assert param.size() == loaded_weight.size( + ), 'the parameter size is not align with the loaded weight size, param size: {}, loaded_weight size: {}'.format( + param.size(), loaded_weight.size()) + assert param.data.dtype == loaded_weight.data.dtype, "if we want to shared weights, the data type should also be the same" + + param.data = loaded_weight.data + + +def default_weight_loader(param: torch.Tensor, loaded_weight: torch.Tensor) -> None: + """Default weight loader.""" + assert param.size() == loaded_weight.size() + assert param.data.dtype == loaded_weight.data.dtype, "if we want to shared weights, the data type should also be the same" + + param.data = loaded_weight.data + + +def gpt2_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "lm_head.weight" in name: + # GPT-2 ties the weights of the embedding layer and the final + # linear layer. + continue + if ".attn.bias" in name or ".attn.masked_bias" in name: + # Skip attention mask. + # NOTE: "c_attn.bias" should not be skipped. + continue + if not name.startswith("transformer."): + name = "transformer." + name + param = params_dict[name] + # The HF's GPT-2 implementation uses Conv1D instead of Linear. + # Because of this, we need to transpose the weights. + # Note(zhuohan): the logic below might break quantized models. + for conv1d_weight_name in ["c_attn", "c_proj", "c_fc"]: + if conv1d_weight_name not in name: + continue + if not name.endswith(".weight"): + continue + # TODO: check megatron + loaded_weight = loaded_weight.t() + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_te_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv.layer_norm_weight", "input_layernorm.weight"), + ("self_attention.linear_qkv.layer_norm_bias", "input_layernorm.bias"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", 'self_attn.o_proj'), + ('pre_mlp_layernorm', 'post_attention_layernorm'), + ('mlp.linear_fc1.layer_norm_weight', 'post_attention_layernorm.weight'), + ('mlp.linear_fc1.layer_norm_bias', 'post_attention_layernorm.bias'), + ('mlp.linear_fc1', 'mlp.gate_up_proj'), + ('mlp.linear_fc2', 'mlp.down_proj'), + ('decoder.final_layernorm', 'model.norm'), + ('output_layer', 'lm_head'), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith('.bias') and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", 'self_attn.o_proj'), + ( + 'input_layernorm', + 'input_layernorm', + ), + ('pre_mlp_layernorm', 'post_attention_layernorm'), + ('mlp.linear_fc1', 'mlp.gate_up_proj'), + ('mlp.linear_fc2', 'mlp.down_proj'), + ('decoder.final_layernorm', 'model.norm'), + ('output_layer', 'lm_head'), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith('.bias') and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def _replace_name(megatron_name, name_mapping): + for m_name, v_name in name_mapping: + if m_name not in megatron_name: + continue + if 'layers' in megatron_name: # deal with decoder layers + megatron_name = megatron_name.replace('decoder', 'model') + megatron_name_list = megatron_name.split('.') + if 'layer_norm_weight' in megatron_name_list or 'layer_norm_bias' in megatron_name_list: + param_name_list = megatron_name_list[:3] + param_name_list.append(v_name) + param_name = '.'.join(param_name_list) + else: + param_name_list = megatron_name_list[:3] + weight_or_bias = megatron_name_list[-1] + param_name_list.append(v_name) + param_name_list.append(weight_or_bias) + param_name = '.'.join(param_name_list) + return param_name + else: + param_name = megatron_name.replace(m_name, v_name) + return param_name + + +def llama_megatron_core_te_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv.layer_norm_weight", "input_layernorm.weight"), + ("self_attention.linear_qkv.layer_norm_bias", "input_layernorm.bias"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", 'self_attn.o_proj'), + ('pre_mlp_layernorm', 'post_attention_layernorm'), + ('mlp.linear_fc1.layer_norm_weight', 'post_attention_layernorm.weight'), + ('mlp.linear_fc1.layer_norm_bias', 'post_attention_layernorm.bias'), + ('mlp.linear_fc1', 'mlp.gate_up_proj'), + ('mlp.linear_fc2', 'mlp.down_proj'), + ('decoder.final_layernorm', 'model.norm'), + ('output_layer', 'lm_head'), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith('.bias') and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", 'self_attn.o_proj'), + ( + 'input_layernorm', + 'input_layernorm', + ), + ('pre_mlp_layernorm', 'post_attention_layernorm'), + ('mlp.linear_fc1', 'mlp.gate_up_proj'), + ('mlp.linear_fc2', 'mlp.down_proj'), + ('decoder.final_layernorm', 'model.norm'), + ('output_layer', 'lm_head'), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith('.bias') and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def _replace_name(megatron_name, name_mapping): + for m_name, v_name in name_mapping: + if m_name not in megatron_name: + continue + if 'layers' in megatron_name: # deal with decoder layers + megatron_name = megatron_name.replace('decoder', 'model') + megatron_name_list = megatron_name.split('.') + if 'layer_norm_weight' in megatron_name_list or 'layer_norm_bias' in megatron_name_list: + param_name_list = megatron_name_list[:3] + param_name_list.append(v_name) + param_name = '.'.join(param_name_list) + else: + param_name_list = megatron_name_list[:3] + weight_or_bias = megatron_name_list[-1] + param_name_list.append(v_name) + param_name_list.append(weight_or_bias) + param_name = '.'.join(param_name_list) + return param_name + else: + param_name = megatron_name.replace(m_name, v_name) + return param_name + + +def mistral_megatron_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + # TODO: need to implement a general way to deal with prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +__LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__ = { + ColumnParallelLinear: parallel_weight_loader, + MergedColumnParallelLinear: parallel_weight_loader, + QKVParallelLinear: parallel_weight_loader, + RowParallelLinear: parallel_weight_loader, + VocabParallelEmbedding: parallel_weight_loader, + ParallelLMHead: parallel_weight_loader + # "ScaledActivation.weight_loader": ScaledActivation, # TODO(shengguangming): latest commit in vllm fix awq for this function and add load_weights + # "default_weight_loader": default_weight_loader +} + +# for layer_class, weight_loader in __LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__.items(): +# # setattr(layer_class, 'megatron_weight_loader', weight_loader) +# layer_class.weight_loader = weight_loader + +__MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__ = { + 'GPT2LMHeadModel': gpt2_weight_loader, + 'LlamaForCausalLM': llama_megatron_weight_loader, # use te backend for open-source megatron + 'LLaMAForCausalLM': llama_megatron_weight_loader, + 'MistralForCausalLM': mistral_megatron_weight_loader, +} + + +# the actor model is .state_dict() +# Load megatron weights +def load_megatron_weights(actor_weights: Dict, vllm_model: nn.Module): + weight_loader = _get_model_weight_loader(vllm_model.__class__.__name__) + weight_loader(actor_weights, vllm_model) + # NOTE(sgm) to reduce peak memory usage, we offload vllm model to cpu + # after init, and we need this after sync model weights for in first iter. + vllm_model = vllm_model.cuda() + + +def _get_model_weight_loader(arch: str): + if arch in __MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__: + return __MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {ModelRegistry.get_supported_archs()}") + + +def update_megatron_weight_loader(): + for layer_class, weight_loader in __LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__.items(): + layer_class.weight_loader = weight_loader diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/model_loader.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/model_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..1b675bb79df378b187d856136905104f7aca1146 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/model_loader.py @@ -0,0 +1,302 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/model_loader + +from typing import Dict, Union, Optional, Iterable, Tuple + +import torch +import torch.nn as nn +from transformers import PreTrainedModel + +from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig, ModelConfig, MultiModalConfig, + ParallelConfig, SchedulerConfig) +from vllm.model_executor.model_loader import BaseModelLoader +from vllm.model_executor.model_loader.loader import _initialize_model +from vllm.model_executor.model_loader.utils import set_default_torch_dtype +from vllm.distributed.communication_op import tensor_model_parallel_all_gather + +from .config import ModelConfig, LoadFormat, LoadConfig +from .megatron_weight_loaders import load_megatron_weights, update_megatron_weight_loader +from .dtensor_weight_loaders import load_dtensor_weights, update_dtensor_weight_loader +from .hf_weight_loader import update_hf_weight_loader + + +def get_model(actor_model: Union[PreTrainedModel, Dict], + model_config: ModelConfig, + load_config: LoadConfig, + device_config: DeviceConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + lora_config: Optional[LoRAConfig], + multimodal_config: Optional[MultiModalConfig], + cache_config: CacheConfig = None) -> nn.Module: + loader = get_model_loader(load_config) + if load_config.load_format.startswith('dummy'): + return loader.load_model(model_config=model_config, + device_config=device_config, + lora_config=lora_config, + multimodal_config=multimodal_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + cache_config=cache_config) + else: + return loader.load_model(actor_model=actor_model, + model_config=model_config, + device_config=device_config, + lora_config=lora_config, + multimodal_config=multimodal_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + cache_config=cache_config) + + +def get_model_loader(load_config: LoadConfig) -> BaseModelLoader: + """Get a model loader based on the load format.""" + + if isinstance(load_config.load_format, type): + return load_config.load_format(load_config) + + if load_config.load_format == LoadFormat.AUTO: + update_megatron_weight_loader() + return MegatronLoader(load_config) + + # NOTE(sgm): change the weight_loader function in runtime + if load_config.load_format == LoadFormat.MEGATRON: + update_megatron_weight_loader() + return MegatronLoader(load_config) + + if load_config.load_format == LoadFormat.HF: + update_hf_weight_loader() + return HFLoader(load_config) + + if load_config.load_format == LoadFormat.DTENSOR: + update_dtensor_weight_loader() + return DTensorLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_HF: + update_hf_weight_loader() + return DummyModelLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_MEGATRON: + update_megatron_weight_loader() + return DummyModelLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_DTENSOR: + update_dtensor_weight_loader() + return DummyModelLoader(load_config) + + raise ValueError('load format not supported in verl: {}, only support {} and {}'.format( + load_config.load_format, LoadFormat.MEGATRON, LoadFormat.HF)) + + +class DummyModelLoader(BaseModelLoader): + """Model loader that will set model weights to random values.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def load_model(self, *, model_config: ModelConfig, device_config: DeviceConfig, lora_config: Optional[LoRAConfig], + multimodal_config: Optional[MultiModalConfig], parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, cache_config: CacheConfig) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, multimodal_config, cache_config, + scheduler_config) + # NOTE(woosuk): For accurate performance evaluation, we assign + # random values to the weights. + # initialize_dummy_weights(model) + return model.eval() + + +class MegatronLoader(BaseModelLoader): + """Model loader that can load the model weights from partitioned megatron model.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def _get_weights_iterator(actor_model: Union[PreTrainedModel, Dict]): + # NOTE(shengguangming) Load the weights from the actor model + pass + # if isinstance(actor_model, nn.Module): + # load_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), vllm_model=model) + # else: + # load_weights(actor_weights=actor_model, vllm_model=model) + # return actor_model + + def load_model(self, actor_model: Union[PreTrainedModel, Dict], model_config: ModelConfig, + device_config: DeviceConfig, lora_config: Optional[LoRAConfig], + multimodal_config: Optional[MultiModalConfig], parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, cache_config: CacheConfig) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, multimodal_config, cache_config, + scheduler_config) + + # TODO(sgm): This is a hack, we need to register the load_weight() func for each model in vllm + if isinstance(actor_model, nn.Module): + load_megatron_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), + vllm_model=model) + else: + load_megatron_weights(actor_weights=actor_model, vllm_model=model) + + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +class HFLoader(BaseModelLoader): + """Model loader that can load the model weights from model's full params.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def _get_weights_iterator(self, actor_model: Union[PreTrainedModel, Dict]): + if isinstance(actor_model, Dict): + return actor_model.items() + elif isinstance(actor_model, nn.Module): + return dict(actor_model.named_parameters()).items() + else: + raise ValueError(f'actor model should be Dict or nn.Module, but get {type(actor_model)}') + + def load_model(self, actor_model: Union[PreTrainedModel, Dict], model_config: ModelConfig, + device_config: DeviceConfig, lora_config: Optional[LoRAConfig], + multimodal_config: Optional[MultiModalConfig], parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, cache_config: CacheConfig) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + # with torch.device(device_config.device): + # NOTE(sgm): init the model in cpu + model = _initialize_model(model_config, self.load_config, lora_config, multimodal_config, cache_config, + scheduler_config) + model.load_weights(self._get_weights_iterator(actor_model)) + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +class DTensorLoader(BaseModelLoader): + """Model loader that can load the model weights from partitioned megatron model.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def _get_weights_iterator(actor_model: Union[PreTrainedModel, Dict]): + # NOTE(shengguangming) Load the weights from the actor model + pass + # if isinstance(actor_model, nn.Module): + # load_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), vllm_model=model) + # else: + # load_weights(actor_weights=actor_model, vllm_model=model) + # return actor_model + + def load_model(self, actor_model: Union[PreTrainedModel, Dict], model_config: ModelConfig, + device_config: DeviceConfig, lora_config: Optional[LoRAConfig], + multimodal_config: Optional[MultiModalConfig], parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, cache_config: CacheConfig) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, multimodal_config, cache_config, + scheduler_config) + + # TODO(sgm): This is a hack, we need to register the load_weight() func for each model in vllm + if isinstance(actor_model, nn.Module): + load_dtensor_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), + vllm_model=model) + else: + load_dtensor_weights(actor_weights=actor_model, vllm_model=model) + + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +# FIXME(sgm): hack the _get_logits function in vllm v0.4.2 +# as they use ray, the _get_logits result will only need to return to the driver node, +# therefore gather is enough. However, we use SPMD instead of a central scheduler, +# all_gather is required (aligned with v0.2.6) +def _get_logits(self, hidden_states: torch.Tensor, embedding: torch.Tensor, + embedding_bias: Optional[torch.Tensor]) -> torch.Tensor: + # Get the logits for the next tokens. + logits = torch.matmul(hidden_states, embedding.t()) + if embedding_bias is not None: + logits += embedding_bias + logits = tensor_model_parallel_all_gather(logits) + # Remove paddings in vocab (if any). + if logits is not None: + logits = logits[:, :self.org_vocab_size] + return logits + + +from vllm.model_executor.layers.logits_processor import LogitsProcessor + + +def logitsprocessor_init(self, + vocab_size: int, + org_vocab_size: Optional[int] = None, + scale: float = 1.0, + logits_as_input: bool = False, + soft_cap: Optional[float] = None) -> None: + """ + Args: + scale: A scaling factor to apply to the logits. + """ + super(LogitsProcessor, self).__init__() + self.scale = scale + self.vocab_size = vocab_size + # Whether the input is logits (default is hidden states). + self.logits_as_input = logits_as_input + # original vocabulary size (without LoRA). + self.org_vocab_size = org_vocab_size or vocab_size + # Soft cap the logits. Used in Gemma 2. + self.soft_cap = soft_cap + # Whether to use gather or all-gather to gather the logits. + self.use_gather = False + + +LogitsProcessor.__init__ = logitsprocessor_init # use all_gather diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/model_runner.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/model_runner.py new file mode 100644 index 0000000000000000000000000000000000000000..d6ab232558a4bcdce568cae6e24f658c28628a4e --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/model_runner.py @@ -0,0 +1,150 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/worker/model_runner.py + +import torch +import torch.nn as nn +from enum import IntEnum +from typing import Dict, List, Optional, Set, Tuple, Union +import warnings + +import vllm.envs as envs +from vllm.attention import (AttentionMetadata, get_attn_backend) +from vllm.config import (CacheConfig, DeviceConfig, LoRAConfig, MultiModalConfig, ParallelConfig, PromptAdapterConfig, + SchedulerConfig) +from vllm.logger import init_logger +from vllm.lora.layers import LoRAMapping +from vllm.lora.request import LoRARequest +from vllm.lora.worker_manager import LRUCacheWorkerLoRAManager +from vllm.model_executor import SamplingMetadata +from vllm.model_executor.models.interfaces import (supports_lora, supports_vision) +from vllm.utils import (CudaMemoryProfiler, is_hip, is_pin_memory_available) +from vllm.worker.model_runner import ModelRunner, CUDAGraphRunner +from vllm.prompt_adapter.worker_manager import (LRUCacheWorkerPromptAdapterManager) + +from .model_loader import get_model +from .config import ModelConfig, LoadConfig + +logger = init_logger(__name__) + + +# How batches are constructed. +class BatchType(IntEnum): + # Every batch is prefill. + PREFILL = 0 + # Every batch is decode. + DECODE = 1 + # Batch is a mixture of prefill and decode. + MIXED = 2 + + +class ModelRunner(ModelRunner): + + def __init__( + self, + model: Union[nn.Module, Dict], # [verl] model itself or its parameter dict + model_config: ModelConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + cache_config: CacheConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + kv_cache_dtype: Optional[str] = "auto", + is_driver_worker: bool = False, + prompt_adapter_config: Optional[PromptAdapterConfig] = None, + multimodal_config: Optional[MultiModalConfig] = None, + return_hidden_states: bool = False, + ): + + super().__init__( + model_config, + parallel_config, + scheduler_config, + device_config, + cache_config, + load_config, + lora_config, + kv_cache_dtype, + is_driver_worker=True, # a hack + prompt_adapter_config=prompt_adapter_config, + multimodal_config=multimodal_config, + return_hidden_states=return_hidden_states) + + # NOTE(sgm): add for verl + self.model = model # this will be replaced by get_model() + + # NOTE(sgm): initialize model using the actor model + def load_model(self) -> None: + logger.info("Starting to load model %s...", self.model_config.model) + with CudaMemoryProfiler() as m: + self.model = get_model(actor_model=self.model, + model_config=self.model_config, + device_config=self.device_config, + lora_config=self.lora_config, + load_config=self.load_config, + parallel_config=self.parallel_config, + scheduler_config=self.scheduler_config, + multimodal_config=self.multimodal_config, + cache_config=self.cache_config) + self.model_memory_usage = m.consumed_memory + logger.info("Loading model weights took %.4f GB", self.model_memory_usage / float(2**30)) + + if self.lora_config: + assert supports_lora(self.model), "Model does not support LoRA" + assert not supports_vision(self.model), "To be tested: vision language model with LoRA settings." + + self.lora_manager = LRUCacheWorkerLoRAManager( + self.scheduler_config.max_num_seqs, + self.scheduler_config.max_num_batched_tokens, + self.vocab_size, + self.lora_config, + self.device, + self.model.embedding_modules, + self.model.embedding_padding_modules, + max_position_embeddings=self.model.config.max_position_embeddings, + ) + self.model = self.lora_manager.create_lora_manager(self.model) + + if self.prompt_adapter_config: + self.prompt_adapter_manager = LRUCacheWorkerPromptAdapterManager( + self.scheduler_config.max_num_seqs, self.scheduler_config.max_num_batched_tokens, self.device, + self.prompt_adapter_config) + self.model = (self.prompt_adapter_manager.create_prompt_adapter_manager(self.model)) + + if self.kv_cache_dtype == "fp8" and is_hip(): + # Currently only ROCm accepts kv-cache scaling factors + # via quantization_param_path and this will be deprecated + # in the future. + if self.model_config.quantization_param_path is not None: + if callable(getattr(self.model, "load_kv_cache_scales", None)): + warnings.warn( + "Loading kv cache scaling factor from JSON is " + "deprecated and will be removed. Please include " + "kv cache scaling factors in the model checkpoint.", + FutureWarning, + stacklevel=2) + self.model.load_kv_cache_scales(self.model_config.quantization_param_path) + logger.info("Loaded KV cache scaling factors from %s", self.model_config.quantization_param_path) + else: + raise RuntimeError( + "Using FP8 KV cache and scaling factors provided but " + "model %s does not support loading scaling factors.", self.model.__class__) + else: + logger.warning("Using FP8 KV cache but no scaling factors " + "provided. Defaulting to scaling factors of 1.0. " + "This may lead to less accurate results!") + + if envs.VLLM_TEST_DYNAMO_GRAPH_CAPTURE: + self.model = torch.compile(self.model, fullgraph=True, backend="eager") diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/parallel_state.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/parallel_state.py new file mode 100644 index 0000000000000000000000000000000000000000..4d853ba75af9b80e285e47770f8798f5d9789aac --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/parallel_state.py @@ -0,0 +1,303 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Adapted from +# https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/parallel_state.py +# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. +"""Model and data parallel groups.""" +import os +import torch +import torch.distributed +from typing import Optional + +import vllm.distributed.parallel_state as ps +from vllm.distributed.parallel_state import get_pp_group, get_world_group, init_distributed_environment, init_model_parallel_group + +import vllm.envs as envs +from vllm.logger import init_logger + +from torch.distributed.device_mesh import init_device_mesh + +logger = init_logger(__name__) +""" +This version is strongly tied with Megatron to implement HybridEngine and weight sharing between vllm and Megatron. +- We assume the Megatron tp+dp+pp world is already established before calling this function. + +""" + +# Device mesh for using DTensor +_DEVICE_MESH = None + +# Tensor model parallel group that the current rank belongs to. +_TP = None +# Pipeline model parallel group that the current rank belongs to. +_PP = None + + +# This method is for initializing the ParallelGroup when using HybridEngine +def initialize_parallel_state( + distributed_init_method: str = "env://", + backend: str = "nccl", + tensor_model_parallel_size: int = 1, + num_tp_per_train_tp: int = 1, + pipeline_model_parallel_size: int = 1, +): + # torch.distributed.all_reduce does not free the input tensor until + # the synchronization point. This causes the memory usage to grow + # as the number of all_reduce calls increases. This env var disables + # this behavior. + # Related issue: + # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573 + os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1" + + # NOTE(sgm): Modify for verl, Env vars will be set by TORCHRUN. + rank = int(os.getenv("RANK", "-1")) + local_rank = int(os.getenv("LOCAL_RANK", "0")) + + # Use the world_size set by TORCHRUN + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + init_distributed_environment(world_size, rank, distributed_init_method, local_rank, backend) + if torch.distributed.get_world_size() > 1: + # NOTE: build a sepearate inference group with infer tp & micro dp + initialize_model_parallel_for_vllm(tensor_model_parallel_size=tensor_model_parallel_size, + num_tensor_model_parallel_groups_per_train_tp=num_tp_per_train_tp) + else: + initialize_model_parallel(tensor_model_parallel_size, pipeline_model_parallel_size, backend) + + +def ensure_model_parallel_initialized( + tensor_model_parallel_size: int, + pipeline_model_parallel_size: int = 1, + backend: Optional[str] = None, +) -> None: + """Helper to initialize model parallel groups if they are not initialized, + or ensure tensor-parallel and pipeline-parallel sizes are equal to expected + values if the model parallel groups are initialized. + """ + # get the backend of _DEVICE_WORLD_GROUP + backend = backend or torch.distributed.get_backend(get_world_group().device_group) + if not model_parallel_is_initialized(): + initialize_model_parallel(tensor_model_parallel_size, pipeline_model_parallel_size, backend) + return + + assert (get_tensor_model_parallel_world_size() == tensor_model_parallel_size), ( + "tensor parallel group already initialized, but of unexpected size: " + f"{get_tensor_model_parallel_world_size()=} vs. " + f"{tensor_model_parallel_size=}") + pp_world_size = get_pp_group().world_size + assert (pp_world_size == pipeline_model_parallel_size), ( + "pipeline parallel group already initialized, but of unexpected size: " + f"{pp_world_size=} vs. " + f"{pipeline_model_parallel_size=}") + + +# TODO(sgm): deviate from the v0.5.4, not pp now +def model_parallel_is_initialized(): + """Check if tensor and pipeline parallel groups are initialized.""" + return (ps._TP is not None) + # and _PIPELINE_MODEL_PARALLEL_GROUP is not None) + + +def initialize_model_parallel_for_vllm(tensor_model_parallel_size: int, + num_tensor_model_parallel_groups_per_train_tp: int = 1, + pipeline_model_parallel_size: int = 1) -> None: + from torch.distributed import new_group + # Get world size and rank. Ensure some consistencies. + assert torch.distributed.is_initialized() + + assert isinstance(tensor_model_parallel_size, int) + + # assert num_tensor_model_parallel_groups_per_train_tp == 1 and not different_tp_group + # assert num_tensor_model_parallel_groups_per_train_tp > 1 and different_tp_group + + # Build the tensor model-parallel groups. + assert ps._TP is None, ("tensor model parallel group is already initialized") + + global _TP + + world_size: int = torch.distributed.get_world_size() + + rank = torch.distributed.get_rank() + + backend = torch.distributed.get_backend() + + num_tensor_model_parallel_groups = world_size // tensor_model_parallel_size + + if num_tensor_model_parallel_groups_per_train_tp == 1: + # if tensor_model_parallel_size == train_tensor_parallel_size: + # using the same tp group as Megatron/vllm + assert _TP is None, ("tensor model parallel group is already initialized") + group_ranks = [] + for i in range(num_tensor_model_parallel_groups): + ranks = range(i * tensor_model_parallel_size, (i + 1) * tensor_model_parallel_size) + group_ranks.append(ranks) + _TP = init_model_parallel_group( + group_ranks=group_ranks, + local_rank=get_world_group().local_rank, + backend=backend, + use_custom_allreduce=False, # TODO: check why True is not work in Ray trainer + use_message_queue_broadcaster=True) + ps._TP = _TP + # _MICRO_DATA_PARALLEL_GROUP is move to hybrid engine + else: + # initialize a micro_dp group and a tp group + # assume training tp=4, infer tp=2, then, weight is partitioned as + # [1], [2], [3], [4] for training and [1,2], [1,2], [3,4], [3,4] for inference + + # Build the inference tp groups + # train_tp = train_tensor_parallel_size + train_tp = num_tensor_model_parallel_groups_per_train_tp * tensor_model_parallel_size + # num_tensor_model_parallel_groups_per_train_tp = train_tp // tensor_model_parallel_size + assert _TP is None, ("tensor model parallel group is already initialized") + group_ranks = [] + for i in range(num_tensor_model_parallel_groups // num_tensor_model_parallel_groups_per_train_tp): + start = train_tp * i + end = train_tp * (i + 1) + for j in range(num_tensor_model_parallel_groups_per_train_tp): + ranks = list(range(start, end, num_tensor_model_parallel_groups_per_train_tp)) + for i in range(len(ranks)): + ranks[i] += j + group_ranks.append(ranks) + _TP = init_model_parallel_group( + group_ranks=group_ranks, + local_rank=get_world_group().local_rank, + backend=backend, + use_custom_allreduce=False, # TODO: check why True is not work in Ray trainer + use_message_queue_broadcaster=True) + ps._TP = _TP + + # Build the pipeline model-parallel groups. + # global _PIPELINE_MODEL_PARALLEL_GROUP + # global _PIPELINE_GLOBAL_RANKS + # assert ps._PIPELINE_MODEL_PARALLEL_GROUP is None, ("pipeline model parallel group is already initialized") + + # ps._PIPELINE_MODEL_PARALLEL_GROUP = mpu.get_pipeline_model_parallel_group() + # ps._PIPELINE_GLOBAL_RANKS = mpu.get_pipeline_model_parallel_ranks() + + # TODO: init using device mesh (not support hybrid engine now) + # Build the pipeline model-parallel groups. + num_pipeline_model_parallel_groups: int = (world_size // pipeline_model_parallel_size) + global _PP + assert _PP is None, ("pipeline model parallel group is already initialized") + group_ranks = [] + for i in range(num_pipeline_model_parallel_groups): + ranks = list(range(i, world_size, num_pipeline_model_parallel_groups)) + group_ranks.append(ranks) + # pipeline parallel does not need custom allreduce + _PP = init_model_parallel_group(group_ranks, get_world_group().local_rank, backend, use_custom_allreduce=False) + ps._PP = _PP # for verl + + +def initialize_model_parallel( + tensor_model_parallel_size: int = 1, + pipeline_model_parallel_size: int = 1, + backend: Optional[str] = None, +) -> None: + """ + NOTE: This method is a hack from the open-sourced version without + asertion of world_size = tp * pp + + Initialize model parallel groups. + + Arguments: + tensor_model_parallel_size: number of GPUs used for tensor model + parallelism. + pipeline_model_parallel_size: number of GPUs used for pipeline model + parallelism. + + Let's say we have a total of 8 GPUs denoted by g0 ... g7 and we + use 2 GPUs to parallelize the model tensor, and 4 GPUs to parallelize + the model pipeline. The present function will + create 4 tensor model-parallel groups and 2 pipeline model-parallel groups: + 4 tensor model-parallel groups: + [g0, g1], [g2, g3], [g4, g5], [g6, g7] + 2 pipeline model-parallel groups: + [g0, g2, g4, g6], [g1, g3, g5, g7] + Note that for efficiency, the caller should make sure adjacent ranks + are on the same DGX box. For example if we are using 2 DGX-1 boxes + with a total of 16 GPUs, rank 0 to 7 belong to the first box and + ranks 8 to 15 belong to the second box. + """ + # Get world size and rank. Ensure some consistencies. + assert torch.distributed.is_initialized() + world_size: int = torch.distributed.get_world_size() + backend = backend or torch.distributed.get_backend(ps.get_world_group().device_group) + + # NOTE(sgm) we don't assert world_size == tp * pp + # DP is not managed by vllm but by the verl WorkerGroup + # if (world_size != + # tensor_model_parallel_size * pipeline_model_parallel_size): + # raise RuntimeError( + # f"world_size ({world_size}) is not equal to " + # f"tensor_model_parallel_size ({tensor_model_parallel_size}) x " + # f"pipeline_model_parallel_size ({pipeline_model_parallel_size})") + + num_tensor_model_parallel_groups: int = (world_size // tensor_model_parallel_size) + rank = torch.distributed.get_rank() + global _TP + assert _TP is None, ("tensor model parallel group is already initialized") + group_ranks = [] + for i in range(num_tensor_model_parallel_groups): + ranks = list(range(i * tensor_model_parallel_size, (i + 1) * tensor_model_parallel_size)) + group_ranks.append(ranks) + + # message queue broadcaster is only used in tensor model parallel group + _TP = init_model_parallel_group( + group_ranks, + get_world_group().local_rank, + backend, + use_custom_allreduce=False, # TODO: check why True is not work in Ray trainer + use_message_queue_broadcaster=True) + ps._TP = _TP + + # TODO: init using device mesh (not support hybrid engine now) + # Build the pipeline model-parallel groups. + num_pipeline_model_parallel_groups: int = (world_size // pipeline_model_parallel_size) + global _PP + assert _PP is None, ("pipeline model parallel group is already initialized") + group_ranks = [] + for i in range(num_pipeline_model_parallel_groups): + ranks = list(range(i, world_size, num_pipeline_model_parallel_groups)) + group_ranks.append(ranks) + # pipeline parallel does not need custom allreduce + _PP = init_model_parallel_group(group_ranks, get_world_group().local_rank, backend, use_custom_allreduce=False) + ps._PP = _PP # for verl + + +""" +Device mesh utilities +""" + + +def get_device_mesh(): + assert _DEVICE_MESH is not None, ("device mesh is not initialized") + return _DEVICE_MESH + + +""" +Tensor model parallel utilities +""" + + +def get_tensor_model_parallel_group(): + """Get the tensor model parallel group the caller rank belongs to.""" + assert _TP is not None, ("tensor model parallel group is not initialized") + return _TP.device_group + + +def get_tensor_model_parallel_world_size(): + """Return world size for the tensor model parallel group.""" + return torch.distributed.get_world_size(group=get_tensor_model_parallel_group()) + + +def get_tensor_model_parallel_rank(): + """Return my rank for the tensor model parallel group.""" + return torch.distributed.get_rank(group=get_tensor_model_parallel_group()) + + +def get_tensor_model_parallel_src_rank(): + """Calculate the global rank corresponding to the first local rank + in the tensor model parallel group.""" + global_rank = torch.distributed.get_rank() + local_world_size = get_tensor_model_parallel_world_size() + return (global_rank // local_world_size) * local_world_size diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/spmd_gpu_executor.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/spmd_gpu_executor.py new file mode 100644 index 0000000000000000000000000000000000000000..e9040d52b57558ced35cc37dcbb96014255ccf95 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/spmd_gpu_executor.py @@ -0,0 +1,253 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/executor/gpu_executor.py + +import os +import socket +from typing import Any, Dict, List, Optional, Set, Tuple + +import torch +import vllm.envs as envs +from vllm.executor.executor_base import ExecutorBase, ExecutorAsyncBase +from vllm.logger import init_logger +from vllm.lora.request import LoRARequest +from vllm.sequence import SamplerOutput, ExecuteModelRequest + +from vllm.config import (CacheConfig, DeviceConfig, LoRAConfig, MultiModalConfig, ParallelConfig, PromptAdapterConfig, + SchedulerConfig, SpeculativeConfig) +from .config import ModelConfig, LoadConfig + +logger = init_logger(__name__) + + +class SPMDGPUExecutor(ExecutorBase): + """SPMD-based multi-GPU executor implementations.""" + + def __init__( + self, + model, # pytorch model itself or its parameter dict + model_config: ModelConfig, + cache_config: CacheConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + multimodal_config: Optional[MultiModalConfig], + speculative_config: Optional[SpeculativeConfig], + prompt_adapter_config: Optional[PromptAdapterConfig], + ) -> None: + self.model_config = model_config + self.cache_config = cache_config + self.lora_config = lora_config + self.load_config = load_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.device_config = device_config + self.multimodal_config = multimodal_config + self.speculative_config = speculative_config + self.prompt_adapter_config = prompt_adapter_config + + distributed_init_method = initialize_cluster(parallel_config) + self._init_executor(model, distributed_init_method) + + # TODO(sgm): verl not support speculative decode now + def _init_executor(self, model, distributed_init_method) -> None: + assert (not self.speculative_config), "Speculative decoding not yet supported for multi-GPU backend." + + # Create the parallel worker for each GPU. + self._init_workers_sp(model, distributed_init_method) + + def _init_workers_sp(self, model, distributed_init_method: str): + # Lazy import the Worker to avoid importing torch.cuda/xformers + # before CUDA_VISIBLE_DEVICES is set in the Worker + from .worker import Worker # pylint: disable=import-outside-toplevel + + rank = int(os.getenv("RANK")) + local_rank = int(os.getenv("LOCAL_RANK")) + print(f'local rank {local_rank}') + + # see https://github.com/NVIDIA/nccl/issues/1234 + os.environ['NCCL_CUMEM_ENABLE'] = '0' + + self.worker = Worker( + model, + self.model_config, + self.parallel_config, + self.scheduler_config, + self.device_config, + self.cache_config, + self.load_config, + local_rank, + rank, + distributed_init_method, + lora_config=self.lora_config, + multimodal_config=self.multimodal_config, + speculative_config=None, + prompt_adapter_config=self.speculative_config, + is_driver_worker=True, + model_runner_cls=None, # use the default one + ) + + # NOTE(shengguangming): torch.distributed.init_process_group will be called inside the init_model() + self.worker.init_device() + self.worker.load_model() + + def determine_num_available_blocks(self) -> Tuple[int, int]: + """Determine the number of available KV blocks. + + This invokes `determine_num_available_blocks` on each worker and takes + the min of the results, guaranteeing that the selected cache sizes are + compatible with all workers. + + Returns: + - tuple[num_gpu_blocks, num_cpu_blocks] + """ + # Get the maximum number of blocks that can be allocated on GPU and CPU. + num_blocks = self.worker.determine_num_available_blocks() + + # NOTE(shengguangming): Now we don't use a shared centralized controler but each process will + # have its own scheduler + num_gpu_blocks = num_blocks[0] + num_cpu_blocks = num_blocks[1] + + return num_gpu_blocks, num_cpu_blocks + + def initialize_cache(self, num_gpu_blocks: int, num_cpu_blocks: int) -> None: + """Initialize the KV cache in all workers. + """ + + # NOTE: We log here to avoid multiple logs when number of workers is + # greater than one. We could log in the engine, but not all executors + # have GPUs. + logger.info("# GPU blocks: %d, # CPU blocks: %d", num_gpu_blocks, num_cpu_blocks) + + self.cache_config.num_gpu_blocks = num_gpu_blocks + self.cache_config.num_cpu_blocks = num_cpu_blocks + + if torch.distributed.get_rank() == 0: + print( + f'before init cache memory allocated: {torch.cuda.memory_allocated() / 1e9}GB, reserved: {torch.cuda.memory_reserved() / 1e9}GB' + ) + self.worker.initialize_cache(num_gpu_blocks=num_gpu_blocks, num_cpu_blocks=num_cpu_blocks) + if torch.distributed.get_rank() == 0: + print( + f'after init cache memory allocated: {torch.cuda.memory_allocated() / 1e9}GB, reserved: {torch.cuda.memory_reserved() / 1e9}GB' + ) + + # NOTE(sgm): This will not profile & capture the model(CUDAGraph) when rebuilding KVCache + def init_cache_engine(self) -> None: + self.worker._init_cache_engine() + + def free_cache_engine(self) -> None: + self.worker.free_cache_engine() + + def execute_model(self, execute_model_req) -> List[SamplerOutput]: + all_outputs = self.worker.execute_model(execute_model_req=execute_model_req) + + # NOTE(sgm): + # Each GPU in vllm under verl has its own spmd_gpu_executor, therefore all GPUs should return the outputs + # In vllm with ray, only the driver worker returns the sampling results. + return all_outputs + + def add_lora(self, lora_request: LoRARequest) -> bool: + assert lora_request.lora_int_id > 0, "lora_id must be greater than 0." + return self.worker.add_lora(lora_request=lora_request) + + def remove_lora(self, lora_id: int) -> bool: + assert lora_id > 0, "lora_id must be greater than 0." + return self.worker.remove_lora(lora_id=lora_id) + + def list_loras(self) -> Set[int]: + return self.worker.list_loras() + + def check_health(self) -> None: + # SPMDExecutor will always be healthy as long as + # it's running. + return + + # NOTE(sgm) add for verl to pass the abstract class test, not used + from vllm.prompt_adapter.request import PromptAdapterRequest + + def add_prompt_adapter(self, prompt_adapter_request: PromptAdapterRequest) -> bool: + assert prompt_adapter_request.prompt_adapter_id > 0, \ + "prompt_adapter_id must be greater than 0." + return self.worker.add_prompt_adapter(prompt_adapter_request) + + def list_prompt_adapters(self) -> Set[int]: + return self.worker.list_prompt_adapters() + + def pin_lora(self, lora_id: int) -> bool: + assert lora_id > 0, "lora_id must be greater than 0." + return self.worker.pin_lora(lora_id) + + def pin_prompt_adapter(self, prompt_adapter_id: int) -> bool: + assert prompt_adapter_id > 0, \ + "prompt_adapter_id must be greater than 0." + return self.worker.pin_prompt_adapter(prompt_adapter_id) + + def remove_prompt_adapter(self, prompt_adapter_id: int) -> bool: + assert prompt_adapter_id > 0, \ + "prompt_adapter_id must be greater than 0." + return self.worker.remove_prompt_adapter(prompt_adapter_id) + + # NOTE(sgm): add for verl + def offload_model_weights(self) -> None: + self.worker.offload_model_weights() + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.worker.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + +def initialize_cluster( + parallel_config: ParallelConfig, + engine_use_ray: bool = False, + ray_address: Optional[str] = None, +) -> Tuple[str, Optional[None]]: + """Initialize the distributed cluster probably with Ray. + + Args: + parallel_config: The configurations for parallel execution. + + Returns: + The `distributed_init_method` is the address for initializing the + distributed backend. + """ + + # Initialize cluster locally. + port = get_open_port() + # We need to setup the distributed init method to make sure + # the distributed megatron code (e.g., get world size) works correctly. + # distributed_init_method = f"tcp://localhost:{port}" + distributed_init_method = 'env://' + return distributed_init_method + + +def get_open_port(): + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("", 0)) + return s.getsockname()[1] + + +# TODO(sgm): not implemented async executor yet +class SPMDGPUExecutorAsync(SPMDGPUExecutor, ExecutorAsyncBase): + + async def execute_model_async(self, execute_model_req: ExecuteModelRequest) -> List[SamplerOutput]: + """Executes one model step on the given sequences.""" + raise NotImplementedError + + async def check_health_async(self) -> None: + """Checks if the executor is healthy. If not, it should raise an + exception.""" + self.check_health() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/tokenizer.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/tokenizer.py new file mode 100644 index 0000000000000000000000000000000000000000..aa625a0338686d61816e838ef802cde327fc95c4 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/tokenizer.py @@ -0,0 +1,77 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/tokenizer_group/tokenizer_group.py + +from typing import List, Optional, Tuple, Union + +from transformers import (AutoTokenizer, PreTrainedTokenizer, PreTrainedTokenizerFast) + +from vllm.lora.request import LoRARequest +from vllm.utils import make_async, LRUCache +from vllm.transformers_utils.tokenizers import * + + +class TokenizerGroup: + """A group of tokenizers that can be used for LoRA adapters.""" + + def __init__(self, tokenizer: PreTrainedTokenizer, enable_lora: bool, max_num_seqs: int, + max_input_length: Optional[int]): + self.enable_lora = enable_lora + self.max_input_length = max_input_length + self.tokenizer = tokenizer + self.lora_tokenizers = LRUCache[PreTrainedTokenizer](capacity=max_num_seqs) if enable_lora else None + + def ping(self) -> bool: + """Check if the tokenizer group is alive.""" + return True + + def get_max_input_len(self, lora_request: Optional[LoRARequest] = None) -> Optional[int]: + """Get the maximum input length for the LoRA request.""" + return self.max_input_length + + def encode(self, + prompt: str, + request_id: Optional[str] = None, + lora_request: Optional[LoRARequest] = None) -> List[int]: + tokenizer = self.get_lora_tokenizer(lora_request) + return tokenizer.encode(prompt) + + async def encode_async(self, + prompt: str, + request_id: Optional[str] = None, + lora_request: Optional[LoRARequest] = None) -> List[int]: + tokenizer = await self.get_lora_tokenizer_async(lora_request) + return tokenizer.encode(prompt) + + def get_lora_tokenizer(self, lora_request: Optional[LoRARequest]) -> "PreTrainedTokenizer": + if not lora_request or not self.enable_lora: + return self.tokenizer + if lora_request.lora_int_id not in self.lora_tokenizers: + # TODO(sgm): the lora tokenizer is also passed, but may be different + tokenizer = self.tokenizer + # tokenizer = (get_lora_tokenizer( + # lora_request, **self.tokenizer_config) or self.tokenizer) + self.lora_tokenizers.put(lora_request.lora_int_id, tokenizer) + return tokenizer + else: + return self.lora_tokenizers.get(lora_request.lora_int_id) + + # FIXME(sgm): for simplicity, we assign the special token here + @property + def pad_token_id(self): + return self.tokenizer.pad_token_id + + @property + def eos_token_id(self): + return self.tokenizer.eos_token_id diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/worker.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/worker.py new file mode 100644 index 0000000000000000000000000000000000000000..a5deb675a1180fc9a575ca0898be27f21c173151 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_5_4/worker.py @@ -0,0 +1,323 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/worker/worker.py +"""A GPU worker class.""" +import os +import gc +from typing import Dict, List, Tuple, Optional, Union, Type + +import torch +import torch.distributed +import torch.nn as nn + +from vllm.config import (CacheConfig, DeviceConfig, LoRAConfig, MultiModalConfig, ParallelConfig, PromptAdapterConfig, + SchedulerConfig, SpeculativeConfig) +from vllm.model_executor import set_random_seed +from vllm.sequence import (ExecuteModelRequest, IntermediateTensors, SamplerOutput) +from vllm.worker.cache_engine import CacheEngine +# TODO(sgm): check why vllm has similar file in vllm.model_executor.parallel_utils.parallel_state +from vllm.distributed import (init_distributed_environment, set_custom_all_reduce, get_tensor_model_parallel_group) +from vllm.worker.worker_base import WorkerInput +from vllm.worker.worker import Worker, _check_if_gpu_supports_dtype +from vllm.worker.model_runner_base import ModelRunnerBase, ModelRunnerInputBase +from vllm.worker.embedding_model_runner import EmbeddingModelRunner +from vllm.worker.model_runner import GPUModelRunnerBase +from .model_runner import ModelRunner +from .megatron_weight_loaders import load_megatron_weights +from .hf_weight_loader import load_hf_weights +from .dtensor_weight_loaders import load_dtensor_weights +from .parallel_state import (ensure_model_parallel_initialized) +from .config import ModelConfig, LoadConfig, LoadFormat + + +class Worker(Worker): + """A worker class that executes (a partition of) the model on a GPU. + + Each worker is associated with a single GPU. The worker is responsible for + maintaining the KV cache and executing the model on the GPU. In case of + distributed inference, each worker is assigned a partition of the model. + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + model_config: ModelConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + cache_config: CacheConfig, + load_config: LoadConfig, + local_rank: int, + rank: int, + distributed_init_method: str, + lora_config: Optional[LoRAConfig] = None, + multimodal_config: Optional[MultiModalConfig] = None, + speculative_config: Optional[SpeculativeConfig] = None, + prompt_adapter_config: Optional[PromptAdapterConfig] = None, + is_driver_worker: bool = False, + model_runner_cls: Optional[Type[GPUModelRunnerBase]] = None, + ) -> None: + # self.model = model # will be replaced in the init_model + self.model_config = model_config + self.parallel_config = parallel_config + self.parallel_config.rank = rank + self.scheduler_config = scheduler_config + self.device_config = device_config + self.cache_config = cache_config + self.local_rank = local_rank + self.rank = rank + self.distributed_init_method = distributed_init_method + self.lora_config = lora_config + self.load_config = load_config + self.prompt_adapter_config = prompt_adapter_config + self.is_driver_worker = is_driver_worker # TODO: we don't need driver + # if parallel_config and is_driver_worker: + # assert rank % parallel_config.tensor_parallel_size == 0, \ + # "Driver worker should be rank 0 of tensor parallel group." + if self.model_config.trust_remote_code: + # note: lazy import to avoid importing torch before initializing + from vllm.utils import init_cached_hf_modules + init_cached_hf_modules() + self.multimodal_config = multimodal_config + + # Return hidden states from target model if the draft model is an + # mlp_speculator + speculative_args = {} if speculative_config is None \ + or (speculative_config.draft_model_config.model == + model_config.model) \ + or (speculative_config.draft_model_config.hf_config.model_type + not in ["medusa", "mlp_speculator"]) \ + else {"return_hidden_states": True} + + # TODO(sgm): set correct model runner class + ModelRunnerClass: Type[GPUModelRunnerBase] = ModelRunner + if model_runner_cls is not None: + ModelRunnerClass = model_runner_cls + elif self.model_config.embedding_mode: + ModelRunnerClass = EmbeddingModelRunner + self.model_runner: GPUModelRunnerBase = ModelRunnerClass( + model, # [VERL]: add for verl + model_config, + parallel_config, + scheduler_config, + device_config, + cache_config, + load_config=load_config, + lora_config=self.lora_config, + kv_cache_dtype=self.cache_config.cache_dtype, + is_driver_worker=is_driver_worker, + prompt_adapter_config=prompt_adapter_config, + multimodal_config=multimodal_config, + **speculative_args, + ) + + # Uninitialized cache engine. Will be initialized by + # initialize_cache. + self.cache_engine: List[CacheEngine] = None + # Initialize gpu_cache as embedding models don't initialize kv_caches + self.gpu_cache: Optional[List[List[torch.Tensor]]] = None + + # NOTE(sgm): [VERL] For offloading inference engine params + self.cpu_model = None + + def init_device(self) -> None: + if self.device_config.device.type == "cuda": + # torch.distributed.all_reduce does not free the input tensor until + # the synchronization point. This causes the memory usage to grow + # as the number of all_reduce calls increases. This env var disables + # this behavior. + # Related issue: + # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573 + os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1" + + # NOTE(sgm): Modify for verl, Env vars will be set by TORCHRUN. + self.rank = self.rank if self.rank is not None else int(os.getenv("RANK", "-1")) + local_rank = int(os.getenv("LOCAL_RANK", "0")) + self.device = torch.device(f"cuda:{local_rank}") + if self.rank < 0: + raise ValueError("Invalid or unspecified rank.") + torch.cuda.set_device(self.device) + + # Use the world_size set by TORCHRUN + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + self.parallel_config.world_size = world_size + + _check_if_gpu_supports_dtype(self.model_config.dtype) + torch.cuda.empty_cache() + self.init_gpu_memory = torch.cuda.mem_get_info()[0] + else: + raise RuntimeError(f"Not support device type: {self.device_config.device}") + + # Initialize the distributed environment. + init_worker_distributed_environment(self.parallel_config, self.rank, self.distributed_init_method, + self.local_rank) + # Set random seed. + set_random_seed(self.model_config.seed) + # self.model = get_model(actor_model=self.model, model_config=self.model_config) + + @torch.inference_mode() + def determine_num_available_blocks(self) -> Tuple[int, int]: + """Profiles the peak memory usage of the model to determine how many + KV blocks may be allocated without OOMs. + + The engine will first conduct a profiling of the existing memory usage. + Then, it calculate the maximum possible number of GPU and CPU blocks + that can be allocated with the remaining free memory. + + .. tip:: + You may limit the usage of GPU memory + by adjusting the `gpu_memory_utilization` parameter. + """ + # Profile the memory usage of the model and get the maximum number of + # cache blocks that can be allocated with the remaining free memory. + torch.cuda.empty_cache() + # torch.cuda.reset_peak_memory_stats() + + # Execute a forward pass with dummy inputs to profile the memory usage + # of the model. + self.model_runner.profile_run() + + # Calculate the number of blocks that can be allocated with the + # profiled peak memory. + torch.cuda.synchronize() + free_gpu_memory, total_gpu_memory = torch.cuda.mem_get_info() + peak_memory = total_gpu_memory - free_gpu_memory + + assert peak_memory > 0, ("Error in memory profiling. This happens when the GPU memory was " + "not properly cleaned up before initializing the vLLM instance.") + + cache_block_size = self.get_cache_block_size_bytes() + + # NOTE(sgm) [VERL] use the remaining memory + num_gpu_blocks = int((free_gpu_memory * self.cache_config.gpu_memory_utilization) // cache_block_size) + # num_gpu_blocks = int((total_gpu_memory * self.cache_config.gpu_memory_utilization - peak_memory) // cache_block_size) + + num_cpu_blocks = int(self.cache_config.swap_space_bytes // cache_block_size) + num_gpu_blocks = max(num_gpu_blocks, 0) + num_cpu_blocks = max(num_cpu_blocks, 0) + if self.model_runner.lora_manager: + self.model_runner.remove_all_loras() + + # NOTE(sgm): Add for [VERL], synchronize number of blocks with all the rank + num_gpu_blocks = torch.tensor([num_gpu_blocks], device='cuda') + num_cpu_blocks = torch.tensor([num_cpu_blocks], device='cuda') + + torch.distributed.all_reduce(num_gpu_blocks, + op=torch.distributed.ReduceOp.MIN, + group=get_tensor_model_parallel_group().device_group) + torch.distributed.all_reduce(num_cpu_blocks, + op=torch.distributed.ReduceOp.MIN, + group=get_tensor_model_parallel_group().device_group) + num_gpu_blocks = num_gpu_blocks.item() + num_cpu_blocks = num_cpu_blocks.item() + gc.collect() + torch.cuda.empty_cache() + return num_gpu_blocks, num_cpu_blocks + + def _init_cache_engine(self): + if self.cache_engine is None and self.gpu_cache is None: + super()._init_cache_engine() + + def free_cache_engine(self): + # ensure `enforce_eager=True` + self.cache_engine = None + self.gpu_cache = None + + # NOTE(sgm): [VERL]: adapt from _execute_model_spmd() + def execute_model(self, + execute_model_req: ExecuteModelRequest, + intermediate_tensors: Optional[IntermediateTensors] = None) -> Optional[List[SamplerOutput]]: + """ + Execute model in Single Program Multiple Data (SPMD) fashion. + All workers take the same request, prepare the input and + execute the model. + """ + assert execute_model_req is not None, ("_execute_model_spmd() requires each worker to take in an " + "ExecuteModelRequest") + worker_input: WorkerInput = self.prepare_worker_input(execute_model_req=execute_model_req) + model_input: ModelRunnerInputBase = (self.model_runner.prepare_model_input( + execute_model_req.seq_group_metadata_list)) + + # verl.worker.workerbase.WorkerBase + # swap cache + super().execute_worker(worker_input) + + # If there is no input, we don't need to execute the model. + if worker_input.num_seq_groups == 0: + return [] + + return self.model_runner.execute_model( + model_input, self.kv_cache[worker_input.virtual_engine] if self.kv_cache is not None else None, + intermediate_tensors) + + # assume the input is .state_dict() + def sync_model_weights(self, actor_weights: Dict, load_format: str): + if load_format in [LoadFormat.MEGATRON, LoadFormat.AUTO]: + load_megatron_weights(actor_weights, self.model_runner.model) + elif load_format == LoadFormat.HF: + # full model state dict without no sharding + load_hf_weights(actor_weights, self.model_runner.model) + elif load_format == LoadFormat.DTENSOR: + load_dtensor_weights(actor_weights, self.model_runner.model) + + def offload_model_weights(self) -> None: + if self.cpu_model == None: + self.cpu_model = {} + for name, params in self.model_runner.model.named_parameters(): + self.cpu_model[name] = torch.empty_like(params, device='cpu') + params.data = self.cpu_model[name] + else: + for name, params in self.model_runner.model.named_parameters(): + params.data = self.cpu_model[name] + + +def init_worker_distributed_environment( + parallel_config: ParallelConfig, + rank: int, + distributed_init_method: Optional[str] = "env://", + local_rank: int = -1, +) -> None: + """Initialize the distributed environment.""" + set_custom_all_reduce(not parallel_config.disable_custom_all_reduce) + + # NOTE(sgm) use tcp://localhost:xxxx will hang in HF setting without megatron + init_distributed_environment(parallel_config.world_size, rank, distributed_init_method, local_rank) + + ensure_model_parallel_initialized(tensor_model_parallel_size=parallel_config.tensor_parallel_size, + pipeline_model_parallel_size=parallel_config.pipeline_parallel_size) + + # TODO(sgm): check whether need this + # if pynccl_utils.is_initialized(): + # pynccl_world_size = pynccl_utils.get_world_size() + # if pynccl_world_size != parallel_config.world_size: + # raise RuntimeError( + # "pynccl is already initialized but the pynccl world " + # "size does not match parallel_config.world_size " + # f"({pynccl_world_size} vs. {parallel_config.world_size}).") + # elif parallel_config.world_size > 1: + # # NOTE(woosuk): We don't initialize pynccl process group when world size + # # is 1. + # # NOTE(kaichao): By default, pynccl is initialized for tp group. + # pynccl_utils.init_process_group( + # group=get_tensor_model_parallel_cpu_group()) + + # # Initialize a custom fast all-reduce implementation. + # if not parallel_config.disable_custom_all_reduce: + # init_custom_ar() + + # A small all_reduce for warmup. + torch.distributed.all_reduce(torch.zeros(1).cuda()) + # if pynccl_utils.is_initialized(): + # pynccl_utils.all_reduce(torch.zeros(1).cuda()) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/__init__.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/arg_utils.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/arg_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..bc4685c5f7968e827491f2bc02fdbd59bfac220c --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/arg_utils.py @@ -0,0 +1,78 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py + +import os +from dataclasses import dataclass + +from transformers import PretrainedConfig +from vllm.config import EngineConfig +from vllm.engine.arg_utils import EngineArgs + +from .config import LoadConfig, ModelConfig + + +@dataclass +class EngineArgs(EngineArgs): + model_hf_config: PretrainedConfig = None # for verl + + def __post_init__(self): + pass + + def create_model_config(self) -> ModelConfig: + return ModelConfig( + hf_config=self.model_hf_config, + tokenizer_mode=self.tokenizer_mode, + trust_remote_code=self.trust_remote_code, + dtype=self.dtype, + seed=self.seed, + revision=self.revision, + code_revision=self.code_revision, + rope_scaling=self.rope_scaling, + rope_theta=self.rope_theta, + tokenizer_revision=self.tokenizer_revision, + max_model_len=self.max_model_len, + quantization=self.quantization, + quantization_param_path=self.quantization_param_path, + enforce_eager=self.enforce_eager, + max_context_len_to_capture=self.max_context_len_to_capture, + max_seq_len_to_capture=self.max_seq_len_to_capture, + max_logprobs=self.max_logprobs, + disable_sliding_window=self.disable_sliding_window, + skip_tokenizer_init=self.skip_tokenizer_init, + served_model_name=self.served_model_name, + limit_mm_per_prompt=self.limit_mm_per_prompt, + use_async_output_proc=not self.disable_async_output_proc, + override_neuron_config=self.override_neuron_config, + config_format=self.config_format, + mm_processor_kwargs=self.mm_processor_kwargs, + ) + + def create_load_config(self) -> LoadConfig: + return LoadConfig( + load_format=self.load_format, + download_dir=self.download_dir, + model_loader_extra_config=self.model_loader_extra_config, + ignore_patterns=self.ignore_patterns, + ) + + def create_engine_config(self) -> EngineConfig: + engine_config = super().create_engine_config() + + # NOTE[VERL]: Use the world_size set by torchrun + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + engine_config.parallel_config.world_size = world_size + + return engine_config diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/config.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/config.py new file mode 100644 index 0000000000000000000000000000000000000000..d7cee451416eb1d7d6c9b4b83fc53dc25a336ccf --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/config.py @@ -0,0 +1,105 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/config.py + +import enum +import json +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, List, Optional, Union + +from transformers import PretrainedConfig + +# Add for verl +from vllm.config import ModelConfig +from vllm.logger import init_logger +from vllm.utils import is_hip + +if TYPE_CHECKING: + from vllm.model_executor.model_loader.loader import BaseModelLoader + +logger = init_logger(__name__) + + +class LoadFormat(str, enum.Enum): + AUTO = "auto" + MEGATRON = "megatron" + HF = "hf" + DTENSOR = "dtensor" + DUMMY_HF = "dummy_hf" + DUMMY_MEGATRON = "dummy_megatron" + DUMMY_DTENSOR = "dummy_dtensor" + + +class ModelConfig(ModelConfig): + + def __init__(self, hf_config: PretrainedConfig, *args, **kwargs) -> None: + super().__init__(model=hf_config._name_or_path, tokenizer=hf_config._name_or_path, *args, **kwargs) + self.hf_config = hf_config + + +@dataclass +class LoadConfig: + """ + download_dir: Directory to download and load the weights, default to the + default cache directory of huggingface. + load_format: The format of the model weights to load: + "auto" will try to load the weights in the safetensors format and + fall back to the pytorch bin format if safetensors format is + not available. + "pt" will load the weights in the pytorch bin format. + "safetensors" will load the weights in the safetensors format. + "npcache" will load the weights in pytorch format and store + a numpy cache to speed up the loading. + "dummy" will initialize the weights with random values, which is + mainly for profiling. + "tensorizer" will use CoreWeave's tensorizer library for + fast weight loading. + "bitsandbytes" will load nf4 type weights. + ignore_patterns: The list of patterns to ignore when loading the model. + Default to "original/**/*" to avoid repeated loading of llama's + checkpoints. + + """ + + load_format: Union[str, LoadFormat, "BaseModelLoader"] = LoadFormat.AUTO + download_dir: Optional[str] = None + model_loader_extra_config: Optional[Union[str, dict]] = field(default_factory=dict) + ignore_patterns: Optional[Union[List[str], str]] = None + + def __post_init__(self): + model_loader_extra_config = self.model_loader_extra_config or {} + if isinstance(model_loader_extra_config, str): + self.model_loader_extra_config = json.loads(model_loader_extra_config) + self._verify_load_format() + + if self.ignore_patterns is not None and len(self.ignore_patterns) > 0: + logger.info("Ignoring the following patterns when downloading weights: %s", self.ignore_patterns) + else: + self.ignore_patterns = ["original/**/*"] + + def _verify_load_format(self) -> None: + if not isinstance(self.load_format, str): + return + + load_format = self.load_format.lower() + self.load_format = LoadFormat(load_format) + + rocm_not_supported_load_format: List[str] = [] + if is_hip() and load_format in rocm_not_supported_load_format: + rocm_supported_load_format = [ + f for f in LoadFormat.__members__ if (f not in rocm_not_supported_load_format) + ] + raise ValueError(f"load format '{load_format}' is not supported in ROCm. " + f"Supported load formats are " + f"{rocm_supported_load_format}") diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/dtensor_weight_loaders.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/dtensor_weight_loaders.py new file mode 100644 index 0000000000000000000000000000000000000000..a3042cabcc4112472b4bbf70a540471eae9e4073 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/dtensor_weight_loaders.py @@ -0,0 +1,380 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/model_loader + +from typing import Dict + +import torch.nn as nn +from torch.distributed._tensor import DTensor +from vllm.model_executor.model_loader.weight_utils import default_weight_loader +from vllm.model_executor.models.utils import is_pp_missing_parameter + + +def gemma_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + for param_name, shard_name, shard_id in stacked_params_mapping: + if shard_name not in name: + continue + stacked_name = name.replace(shard_name, param_name) + # Skip loading extra bias for GPTQ models. + if stacked_name.endswith(".bias") and stacked_name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[stacked_name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # lm_head is not used in vllm as it is tied with embed_token. + # To prevent errors, skip loading lm_head.weight. + if "lm_head.weight" in name: + continue + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def gptbigcode_dtensor_load_weights(actor_weights: Dict, vllm_model: nn.Module): + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "lm_head.weight" in name: + continue + if ".attn.bias" in name: + # Skip attention mask. + # NOTE: "c_attn.bias" should not be skipped. + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def starcoder2_dtensor_load_weights(actor_weights: Dict, vllm_model: nn.Module): + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ] + + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + + for param_name, weight_name, shard_id in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def llama_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + (".qkv_proj", ".q_proj", "q"), + (".qkv_proj", ".k_proj", "k"), + (".qkv_proj", ".v_proj", "v"), + (".gate_up_proj", ".gate_proj", 0), + (".gate_up_proj", ".up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if "rotary_emb.cos_cached" in name or "rotary_emb.sin_cached" in name: + # Models trained using ColossalAI may include these tensors in + # the checkpoint. Skip them. + continue + # With tie_word_embeddings, we can skip lm_head.weight + # The weight might appear unnecessarily in the files if the model is + # processed with quantization, LoRA, fine-tuning, etc. + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for param_name, weight_name, shard_id in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight) + + +def qwen2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for param_name, weight_name, shard_id in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def qwen2vl_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("qkv_proj", "q_proj", "q"), + ("qkv_proj", "k_proj", "k"), + ("qkv_proj", "v_proj", "v"), + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in name: + continue + for param_name, weight_name, shard_id in stacked_params_mapping: + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + param = params_dict[name] + weight_loader = param.weight_loader + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +from vllm.model_executor.layers.fused_moe import FusedMoE + + +def deepseekv2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + stacked_params_mapping = [ + # (param_name, shard_name, shard_id) + ("gate_up_proj", "gate_proj", 0), + ("gate_up_proj", "up_proj", 1), + ] + + # Params for weights, fp8 weight scales, fp8 activation scales + # (param_name, weight_name, expert_id, shard_id) + expert_params_mapping = FusedMoE.make_expert_params_mapping( + ckpt_gate_proj_name="gate_proj", + ckpt_down_proj_name="down_proj", + ckpt_up_proj_name="up_proj", + num_experts=vllm_model.config.n_routed_experts, + ) + + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + for param_name, weight_name, shard_id in stacked_params_mapping: + # Skip non-stacked layers and experts (experts handled below). + if weight_name not in name: + continue + # We have mlp.experts[0].gate_proj in the checkpoint. + # Since we handle the experts below in expert_params_mapping, + # we need to skip here BEFORE we update the name, otherwise + # name will be updated to mlp.experts[0].gate_up_proj, which + # will then be updated below in expert_params_mapping + # for mlp.experts[0].gate_gate_up_proj, which breaks load. + if ("mlp.experts." in name) and name not in params_dict: + continue + name = name.replace(weight_name, param_name) + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id) + break + else: + for mapping in expert_params_mapping: + param_name, weight_name, expert_id, shard_id = mapping + if weight_name not in name: + continue + name = name.replace(weight_name, param_name) + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader( + param, + local_loaded_weight.to(dtype=param.dtype), + weight_name, + shard_id=shard_id, + expert_id=expert_id, + ) + break + else: + # Skip loading extra bias for GPTQ models. + if name.endswith(".bias") and name not in params_dict: + continue + + if is_pp_missing_parameter(name, vllm_model): + continue + + param = params_dict[name] + local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight) + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, local_loaded_weight.to(dtype=param.dtype)) + + +def gpt2_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + pass + + +def redistribute_dtensor(param_name: str, loaded_weights: DTensor, parallelize_plan: Dict = None): + param_name = _process_parameter_names(name=param_name) + if parallelize_plan is not None: + assert ( + param_name + in parallelize_plan.keys()), f"param name: {param_name} not in parallelize_plan :{parallelize_plan.keys()}" + placement = parallelize_plan[param_name] + local_loaded_weights = loaded_weights.redistribute(device_mesh=loaded_weights.device_mesh, + placements=placement).to_local() + else: + local_loaded_weights = loaded_weights.full_tensor() + return local_loaded_weights + + +def _process_parameter_names(name): + # Remove '.weight' if it exists at the end of the string + if name.endswith(".weight"): + name = name[:-7] + + # Remove 'model.layers.x.' or 'model.' prefix + if "model.layers" in name: + parts = name.split(".") + # Reconstruct the string without 'model.layers.x.' + name = ".".join(parts[3:]) # parts[0] is 'model', parts[1] is 'layers', parts[2] is 'x' + elif name.startswith("model."): + name = name[6:] # Remove 'model.' + + return name + + +__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__ = { + "GPT2LMHeadModel": gpt2_dtensor_weight_loader, + "LlamaForCausalLM": llama_dtensor_weight_loader, + "LLaMAForCausalLM": llama_dtensor_weight_loader, + "MistralForCausalLM": llama_dtensor_weight_loader, # mistral is the same as llama in vLLM + "InternLMForCausalLM": llama_dtensor_weight_loader, + "AquilaModel": llama_dtensor_weight_loader, + "AquilaForCausalLM": llama_dtensor_weight_loader, + "Phi3ForCausalLM": llama_dtensor_weight_loader, + "GemmaForCausalLM": gemma_dtensor_weight_loader, + "Gemma2ForCausalLM": gemma_dtensor_weight_loader, + "GPTBigCodeForCausalLM": gptbigcode_dtensor_load_weights, + "Starcoder2ForCausalLM": starcoder2_dtensor_load_weights, + "Qwen2ForCausalLM": qwen2_dtensor_weight_loader, + "DeepseekV2ForCausalLM": deepseekv2_dtensor_weight_loader, + "Qwen2VLForConditionalGeneration": qwen2vl_dtensor_weight_loader, +} + + +# the actor model is .state_dict() +# Load dtensor weights +def load_dtensor_weights(actor_weights: Dict, vllm_model: nn.Module): + weight_loader = _get_model_weight_loader(vllm_model.__class__.__name__) + weight_loader(actor_weights, vllm_model) + # NOTE(sgm) to reduce peak memory usage, we offload vllm model to cpu + # after init, and we need this after sync model weights for in first iter. + vllm_model = vllm_model.cuda() + + +def _get_model_weight_loader(arch: str): + if arch in __MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__: + return __MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__.keys()}") + + +# NOTE(sgm): we use per-parameter weight loader in each vllm sub +def update_dtensor_weight_loader(): + pass diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/hf_weight_loader.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/hf_weight_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..a3e5b22b2fed3b17f22f66c7acef8094c1c7871a --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/hf_weight_loader.py @@ -0,0 +1,41 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/model_loader + +from typing import Dict + +import torch.nn as nn +from vllm.model_executor.model_loader.utils import set_default_torch_dtype + + +def update_hf_weight_loader(): + print("no hf weight loader need to be updated") + return + + +def load_hf_weights(actor_weights: Dict, vllm_model: nn.Module): + assert isinstance(actor_weights, Dict) + with set_default_torch_dtype(next(vllm_model.parameters()).dtype): # TODO + if vllm_model.config.tie_word_embeddings and "lm_head.weight" in actor_weights.keys(): + del actor_weights["lm_head.weight"] + vllm_model.load_weights(actor_weights.items()) + for _, module in vllm_model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + vllm_model = vllm_model.cuda() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/llm.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/llm.py new file mode 100644 index 0000000000000000000000000000000000000000..cd3d646db46e0b6085a94a49da695d5a6feb1403 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/llm.py @@ -0,0 +1,200 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py + +from typing import Dict, List, Optional, Tuple, Union + +import torch +import torch.nn as nn +from torch.nn.utils.rnn import pad_sequence +from transformers import PretrainedConfig, PreTrainedTokenizer, PreTrainedTokenizerFast +from verl.workers.rollout.tokenizer import HybridEngineBaseTokenizer +from vllm import LLM +from vllm.outputs import EmbeddingRequestOutput, RequestOutput +from vllm.utils import Counter + +from .arg_utils import EngineArgs +from .llm_engine_sp import LLMEngine + + +class LLM(LLM): + """An LLM for generating texts from given prompts and sampling parameters. + + This class includes a tokenizer, a language model (possibly distributed + across multiple GPUs), and GPU memory space allocated for intermediate + states (aka KV cache). Given a batch of prompts and sampling parameters, + this class generates texts from the model, using an intelligent batching + mechanism and efficient memory management. + + NOTE: This class is intended to be used for offline inference. For online + serving, use the `AsyncLLMEngine` class instead. + NOTE: For the comprehensive list of arguments, see `EngineArgs`. + + Args: + model: A HuggingFace Transformers model instance. + tokenizer: A HuggingFace Transformers tokenizer instance. + tokenizer_mode: The tokenizer mode. "auto" will use the fast tokenizer + if available, and "slow" will always use the slow tokenizer. + trust_remote_code: Trust remote code (e.g., from HuggingFace) when + downloading the model and tokenizer. + tensor_parallel_size: The number of GPUs to use for distributed + execution with tensor parallelism. + dtype: The data type for the model weights and activations. Currently, + we support `float32`, `float16`, and `bfloat16`. If `auto`, we use + the `torch_dtype` attribute specified in the model config file. + However, if the `torch_dtype` in the config is `float32`, we will + use `float16` instead. + quantization: The method used to quantize the model weights. Currently, + we support "awq". If None, we assume the model weights are not + quantized and use `dtype` to determine the data type of the weights. + revision: The specific model version to use. It can be a branch name, + a tag name, or a commit id. + tokenizer_revision: The specific tokenizer version to use. It can be a + branch name, a tag name, or a commit id. + seed: The seed to initialize the random number generator for sampling. + gpu_memory_utilization: The ratio (between 0 and 1) of GPU memory to + reserve for the model weights, activations, and KV cache. Higher + values will increase the KV cache size and thus improve the model's + throughput. However, if the value is too high, it may cause out-of- + memory (OOM) errors. + swap_space: The size (GiB) of CPU memory per GPU to use as swap space. + This can be used for temporarily storing the states of the requests + when their `best_of` sampling parameters are larger than 1. If all + requests will have `best_of=1`, you can safely set this to 0. + Otherwise, too small values may cause out-of-memory (OOM) errors. + enforce_eager: Whether to enforce eager execution. If True, we will + disable CUDA graph and always execute the model in eager mode. + If False, we will use CUDA graph and eager execution in hybrid. + max_context_len_to_capture: Maximum context len covered by CUDA graphs. + When a sequence has context length larger than this, we fall back + to eager mode. + disable_custom_all_reduce: See ParallelConfig + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast, HybridEngineBaseTokenizer], + model_hf_config: PretrainedConfig, + tokenizer_mode: str = "auto", + trust_remote_code: bool = False, + skip_tokenizer_init: bool = False, + tensor_parallel_size: int = 1, + dtype: str = "auto", + quantization: Optional[str] = None, + revision: Optional[str] = None, + tokenizer_revision: Optional[str] = None, + seed: int = 0, + gpu_memory_utilization: float = 0.9, + swap_space: int = 4, + cpu_offload_gb: float = 0, + enforce_eager: bool = False, + max_context_len_to_capture: Optional[int] = None, + max_seq_len_to_capture: int = 8192, + disable_custom_all_reduce: bool = False, + load_format="auto", + **kwargs, + ) -> None: + if "disable_log_stats" not in kwargs: + kwargs["disable_log_stats"] = True + removed_vision_keys = ("image_token_id", "image_feature_size", "image_input_shape", "image_input_type") + if any(k in kwargs for k in removed_vision_keys): + raise TypeError("There is no need to pass vision-related arguments anymore.") + engine_args = EngineArgs( + model_hf_config=model_hf_config, + # tokenizer=tokenizer, + tokenizer_mode=tokenizer_mode, + skip_tokenizer_init=skip_tokenizer_init, + trust_remote_code=trust_remote_code, + tensor_parallel_size=tensor_parallel_size, + dtype=dtype, + quantization=quantization, + revision=revision, + tokenizer_revision=tokenizer_revision, + seed=seed, + gpu_memory_utilization=gpu_memory_utilization, + swap_space=swap_space, + cpu_offload_gb=cpu_offload_gb, + enforce_eager=enforce_eager, + max_context_len_to_capture=max_context_len_to_capture, + max_seq_len_to_capture=max_seq_len_to_capture, + disable_custom_all_reduce=disable_custom_all_reduce, + load_format=load_format, + **kwargs, + ) + tokenizer_cls = (PreTrainedTokenizer, PreTrainedTokenizerFast, HybridEngineBaseTokenizer) + if not isinstance(tokenizer, tokenizer_cls): + raise ValueError( + f"Unexpected tokenizer type: {type(tokenizer)}. Must be" + "one of the following: PreTrainedTokenizer, PreTrainedTokenizerFast, verl.workers.rollout.HybridEngineBaseTokenizer" + ) + self.llm_engine = LLMEngine.from_engine_args(model, tokenizer, engine_args) # TODO: check usagecontext + self.request_counter = Counter() + + def init_cache_engine(self): + self.llm_engine.init_cache_engine() + + def free_cache_engine(self): + self.llm_engine.free_cache_engine() + + def get_tokenizer(self) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]: + return self.llm_engine.tokenizer + + def set_tokenizer( + self, + tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], + ) -> None: + self.llm_engine.tokenizer = tokenizer + + def _run_engine(self, *, use_tqdm: bool) -> List[Union[RequestOutput, EmbeddingRequestOutput]]: + outputs = super()._run_engine(use_tqdm=use_tqdm) + return self._post_process_outputs(outputs) + + # # NOTE(shengguangming): add for verl + # # TODO(sgm): we can optimize it by making the dataloader yield List[int] without padding. + # def _pre_process_inputs(self, prompt_token_ids: torch.Tensor) -> List[int]: + # # remove the left padding in the prompt token_id + # pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id + # non_pad_index = torch.nonzero(prompt_token_ids != pad_token_id, as_tuple=False)[0][0] + # token_ids = prompt_token_ids[non_pad_index:].tolist() + # return token_ids + + # NOTE(shengguangming): add for verl + def _post_process_outputs(self, request_outputs: List[RequestOutput]) -> Tuple[torch.Tensor, torch.Tensor]: + output_token_ids = [] + logprobs = [] + for request_output in request_outputs: # List[RequestOutput] + outputs = request_output.outputs + for output in outputs: # List[CompletionOutput], usually len == 1 + output_token_ids.append(torch.tensor(output.token_ids)) + # TODO(shengguangming): can be optimzied by rewrite the Sampler._get_logprobs() logits + logprobs_dicts = output.logprobs + if logprobs_dicts is not None: + logprob = [] + for logprobs_dict, id in zip(logprobs_dicts, output.token_ids): + logprob.append(logprobs_dict[id].logprob) + logprobs.append(torch.tensor(logprob)) + + pad_token_id = (self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None + else self.llm_engine.tokenizer.eos_token_id) + output_token_ids = pad_sequence(output_token_ids, batch_first=True, padding_value=pad_token_id) + if len(logprobs) > 0: + logprobs = pad_sequence(logprobs, batch_first=True, padding_value=pad_token_id) + return output_token_ids, logprobs + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.llm_engine.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + def offload_model_weights(self) -> None: + self.llm_engine.offload_model_weights() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/llm_engine_sp.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/llm_engine_sp.py new file mode 100644 index 0000000000000000000000000000000000000000..10b112b2595d83514698589bba472efb07dea562 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/llm_engine_sp.py @@ -0,0 +1,408 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/engine/llm_engine.py + +from functools import partial +from typing import Callable, Dict, Optional, Type, Union + +import torch +import torch.nn as nn +from vllm.config import ( + CacheConfig, + DecodingConfig, + DeviceConfig, + EngineConfig, + LoadConfig, + LoRAConfig, + ModelConfig, + ObservabilityConfig, + ParallelConfig, + PromptAdapterConfig, + SchedulerConfig, + SpeculativeConfig, +) +from vllm.core.scheduler import Scheduler +from vllm.engine.arg_utils import EngineArgs +from vllm.engine.llm_engine import LLMEngine, SchedulerContext, SchedulerOutputState, _load_generation_config_dict +from vllm.engine.metrics_types import StatLoggerBase +from vllm.engine.output_processor.interfaces import SequenceGroupOutputProcessor +from vllm.engine.output_processor.stop_checker import StopChecker +from vllm.executor.executor_base import ExecutorBase +from vllm.inputs import INPUT_REGISTRY, InputRegistry +from vllm.inputs.preprocess import InputPreprocessor +from vllm.logger import init_logger +from vllm.sequence import Sequence +from vllm.tracing import init_tracer +from vllm.transformers_utils.detokenizer import Detokenizer +from vllm.transformers_utils.tokenizer import AnyTokenizer +from vllm.usage.usage_lib import UsageContext, is_usage_stats_enabled, usage_message +from vllm.utils import Counter, weak_bind +from vllm.version import __version__ as VLLM_VERSION + +from .arg_utils import EngineArgs +from .config import LoadConfig, ModelConfig +from .tokenizer import TokenizerGroup + +logger = init_logger(__name__) +_LOCAL_LOGGING_INTERVAL_SEC = 5 + + +class LLMEngine(LLMEngine): + """An LLM engine that receives requests and generates texts. + + This is the main class for the vLLM engine. It receives requests + from clients and generates texts from the LLM. It includes a tokenizer, a + language model (possibly distributed across multiple GPUs), and GPU memory + space allocated for intermediate states (aka KV cache). This class utilizes + iteration-level scheduling and efficient memory management to maximize the + serving throughput. + + The :class:`~vllm.LLM` class wraps this class for offline batched inference + and the :class:`AsyncLLMEngine` class wraps this class for online serving. + + The config arguments are derived from :class:`~vllm.EngineArgs`. (See + :ref:`engine_args`) + + Args: + model_config: The configuration related to the LLM model. + cache_config: The configuration related to the KV cache memory + management. + parallel_config: The configuration related to distributed execution. + scheduler_config: The configuration related to the request scheduler. + device_config: The configuration related to the device. + lora_config (Optional): The configuration related to serving multi-LoRA. + speculative_config (Optional): The configuration related to speculative + decoding. + executor_class: The model executor class for managing distributed + execution. + prompt_adapter_config (Optional): The configuration related to serving + prompt adapters. + log_stats: Whether to log statistics. + usage_context: Specified entry point, used for usage info collection. + """ + + def __init__( + self, + # NOTE(sgm): first two arguments are added for verl + model: Union[nn.Module, Dict], # model itself or its parameter dict + tokenizer: nn.Module, + # NOTE(sgm): vllm original arguments + model_config: ModelConfig, + cache_config: CacheConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + speculative_config: Optional[SpeculativeConfig], + decoding_config: Optional[DecodingConfig], + observability_config: Optional[ObservabilityConfig], + prompt_adapter_config: Optional[PromptAdapterConfig], + executor_class: Type[ExecutorBase], + log_stats: bool, + usage_context: UsageContext = UsageContext.ENGINE_CONTEXT, + stat_loggers: Optional[Dict[str, StatLoggerBase]] = None, + input_registry: InputRegistry = INPUT_REGISTRY, + use_cached_outputs: bool = False, + ) -> None: + logger.info( + "Initializing an LLM engine (v%s) with config: " + "model=%r, speculative_config=%r, tokenizer=%r, " + "skip_tokenizer_init=%s, tokenizer_mode=%s, revision=%s, " + "override_neuron_config=%s, " + "rope_scaling=%r, rope_theta=%r, tokenizer_revision=%s, " + "trust_remote_code=%s, dtype=%s, max_seq_len=%d, " + "download_dir=%r, load_format=%s, tensor_parallel_size=%d, " + "pipeline_parallel_size=%d, " + "disable_custom_all_reduce=%s, quantization=%s, " + "enforce_eager=%s, kv_cache_dtype=%s, " + "quantization_param_path=%s, device_config=%s, " + "decoding_config=%r, observability_config=%r, " + "seed=%d, served_model_name=%s, use_v2_block_manager=%s, " + "num_scheduler_steps=%d, chunked_prefill_enabled=%s " + "multi_step_stream_outputs=%s, enable_prefix_caching=%s, " + "use_async_output_proc=%s, use_cached_outputs=%s, " + "mm_processor_kwargs=%s)", + VLLM_VERSION, + model_config.model, + speculative_config, + model_config.tokenizer, + model_config.skip_tokenizer_init, + model_config.tokenizer_mode, + model_config.revision, + model_config.override_neuron_config, + model_config.rope_scaling, + model_config.rope_theta, + model_config.tokenizer_revision, + model_config.trust_remote_code, + model_config.dtype, + model_config.max_model_len, + load_config.download_dir, + load_config.load_format, + parallel_config.tensor_parallel_size, + parallel_config.pipeline_parallel_size, + parallel_config.disable_custom_all_reduce, + model_config.quantization, + model_config.enforce_eager, + cache_config.cache_dtype, + model_config.quantization_param_path, + device_config.device, + decoding_config, + observability_config, + model_config.seed, + model_config.served_model_name, + scheduler_config.use_v2_block_manager, + scheduler_config.num_scheduler_steps, + scheduler_config.chunked_prefill_enabled, + scheduler_config.multi_step_stream_outputs, + cache_config.enable_prefix_caching, + model_config.use_async_output_proc, + use_cached_outputs, + model_config.mm_processor_kwargs, + ) + # TODO(woosuk): Print more configs in debug mode. + self.model_config = model_config + self.cache_config = cache_config + self.lora_config = lora_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.device_config = device_config + self.speculative_config = speculative_config + self.load_config = load_config + self.decoding_config = decoding_config or DecodingConfig() + self.prompt_adapter_config = prompt_adapter_config + self.observability_config = observability_config or ObservabilityConfig() + self.log_stats = log_stats + self.use_cached_outputs = use_cached_outputs + + if not self.model_config.skip_tokenizer_init: + self.tokenizer = self._init_tokenizer(tokenizer) + self.detokenizer = Detokenizer(self.tokenizer) + tokenizer_group = self.get_tokenizer_group() + else: + self.tokenizer = None + self.detokenizer = None + tokenizer_group = None + + # Ensure that the function doesn't contain a reference to self, + # to avoid engine GC issues + def get_tokenizer_for_seq(sequence: Sequence) -> AnyTokenizer: + assert tokenizer_group, "tokenizer_group cannot be None, " "make sure skip_tokenizer_init is False" + return tokenizer_group.get_lora_tokenizer(sequence.lora_request) + + self.seq_counter = Counter() + self.generation_config_fields = _load_generation_config_dict(model_config) + + self.input_preprocessor = InputPreprocessor(model_config, self.tokenizer) + + self.input_registry = input_registry + self.input_processor = input_registry.create_input_processor(model_config) + + self.model_executor = executor_class( + model=model, # add for spmd_gpu_executor + model_config=model_config, + cache_config=cache_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + device_config=device_config, + lora_config=lora_config, + speculative_config=speculative_config, + load_config=load_config, + prompt_adapter_config=prompt_adapter_config, + observability_config=self.observability_config, + ) + + if not self.model_config.embedding_mode: + self._initialize_kv_caches() + + # If usage stat is enabled, collect relevant info. + if is_usage_stats_enabled(): + from vllm.model_executor.model_loader import get_architecture_class_name + + usage_message.report_usage( + get_architecture_class_name(model_config), + usage_context, + extra_kvs={ + # Common configuration + "dtype": str(model_config.dtype), + "tensor_parallel_size": parallel_config.tensor_parallel_size, + "block_size": cache_config.block_size, + "gpu_memory_utilization": cache_config.gpu_memory_utilization, + # Quantization + "quantization": model_config.quantization, + "kv_cache_dtype": str(cache_config.cache_dtype), + # Feature flags + "enable_lora": bool(lora_config), + "enable_prompt_adapter": bool(prompt_adapter_config), + "enable_prefix_caching": cache_config.enable_prefix_caching, + "enforce_eager": model_config.enforce_eager, + "disable_custom_all_reduce": parallel_config.disable_custom_all_reduce, + }, + ) + + if self.tokenizer: + # Ping the tokenizer to ensure liveness if it runs in a + # different process. + self.tokenizer.ping() + + self.cached_scheduler_outputs = [ + SchedulerOutputState() for _ in range(self.parallel_config.pipeline_parallel_size) + ] + + self.scheduler_contexts = [ + SchedulerContext(multi_step_stream_outputs=self.scheduler_config.multi_step_stream_outputs) + for _ in range(self.parallel_config.pipeline_parallel_size) + ] + + if model_config.use_async_output_proc: + process_model_outputs = weak_bind(self._process_model_outputs) + + self.async_callbacks = [ + partial(process_model_outputs, ctx=self.scheduler_contexts[v_id]) + for v_id in range(self.parallel_config.pipeline_parallel_size) + ] + else: + self.async_callbacks = [] + + # Currently used by AsyncLLMEngine to ensure quick append + # of request outputs to asyncio queues + self.process_request_outputs_callback: Optional[Callable] = None + + # Create the scheduler. + # NOTE: the cache_config here have been updated with the numbers of + # GPU and CPU blocks, which are profiled in the distributed executor. + self.scheduler = [ + Scheduler( + scheduler_config, + cache_config, + lora_config, + parallel_config.pipeline_parallel_size, + self.async_callbacks[v_id] if model_config.use_async_output_proc else None, + ) for v_id in range(parallel_config.pipeline_parallel_size) + ] + + # Metric Logging. + if self.log_stats: + if stat_loggers is not None: + self.stat_loggers = stat_loggers + else: + # Lazy import for prometheus multiprocessing. + # We need to set PROMETHEUS_MULTIPROC_DIR environment variable + # before prometheus_client is imported. + # See https://prometheus.github.io/client_python/multiprocess/ + from vllm.engine.metrics import LoggingStatLogger, PrometheusStatLogger + + self.stat_loggers = { + "logging": + LoggingStatLogger(local_interval=_LOCAL_LOGGING_INTERVAL_SEC), + "prometheus": + PrometheusStatLogger( + local_interval=_LOCAL_LOGGING_INTERVAL_SEC, + labels=dict(model_name=model_config.served_model_name), + max_model_len=self.model_config.max_model_len, + ), + } + self.stat_loggers["prometheus"].info("cache_config", self.cache_config) + + self.tracer = None + if self.observability_config.otlp_traces_endpoint: + self.tracer = init_tracer("vllm.llm_engine", self.observability_config.otlp_traces_endpoint) + + # Create sequence output processor, e.g. for beam search or + # speculative decoding. + self.output_processor = SequenceGroupOutputProcessor.create_output_processor( + self.scheduler_config, + self.detokenizer, + self.scheduler, + self.seq_counter, + get_tokenizer_for_seq, + stop_checker=StopChecker( + self.scheduler_config.max_model_len, + get_tokenizer_for_seq, + ), + ) + + # TODO(sgm): add for verl but we may not tokenizer in Rollout + def _init_tokenizer(self, tokenizer, **tokenizer_init_kwargs): + init_kwargs = dict(enable_lora=bool(self.lora_config), + max_num_seqs=self.scheduler_config.max_num_seqs, + max_input_length=None) + init_kwargs.update(tokenizer_init_kwargs) + return TokenizerGroup(tokenizer, **init_kwargs) + + def init_cache_engine(self): + # TODO: check whether we should rebuild the CUDAGraph every iter when offload/load KVCache + # Re-capture CUDAGraph would be time-consuming + self.model_executor.init_cache_engine() + + def free_cache_engine(self): + self.model_executor.free_cache_engine() + + # NOTE(sgm): currently, we only support GPU executor + # The GPUExecutor remove the Ray dependency + @classmethod + def _get_executor_cls(cls, engine_config: EngineConfig) -> Type[ExecutorBase]: + distributed_executor_backend = engine_config.parallel_config.distributed_executor_backend + # Initialize the cluster and specify the executor class.] + assert (engine_config.device_config.device_type == "cuda" + ), "Currently, the vllm in verl only support running on GPU" + + # print('Waiting for debugger'); import os,debugpy; debugpy.listen(('localhost', 5678 + int(os.getenv('RANK', '0')))); debugpy.wait_for_client() + if engine_config.parallel_config.world_size == 1: + engine_config.load_config.load_format = "dummy_hf" + + from .spmd_gpu_executor import SPMDGPUExecutor + + executor_class = SPMDGPUExecutor + + return executor_class + + @classmethod + def from_engine_args( + cls, + model, + tokenizer, + engine_args: EngineArgs, + usage_context: UsageContext = UsageContext.ENGINE_CONTEXT, + stat_loggers: Optional[Dict[str, StatLoggerBase]] = None, + ) -> "LLMEngine": + """Creates an LLM engine from the engine arguments.""" + # Create the engine configs. + engine_config = engine_args.create_engine_config() + executor_class = cls._get_executor_cls(engine_config) + # Initialize the cluster and specify the executor class. + assert (engine_config.device_config.device_type == "cuda" + ), "Currently, the vllm in verl only support running on GPU" + + from .spmd_gpu_executor import SPMDGPUExecutor + + executor_class = SPMDGPUExecutor + + # Create the LLM engine. + engine = cls( + model, + tokenizer, + **engine_config.to_dict(), + executor_class=executor_class, + log_stats=not engine_args.disable_log_stats, + usage_context=usage_context, + stat_loggers=stat_loggers, + ) + return engine + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.model_executor.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + def offload_model_weights(self) -> None: + self.model_executor.offload_model_weights() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/megatron_weight_loaders.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/megatron_weight_loaders.py new file mode 100644 index 0000000000000000000000000000000000000000..7fd6c0e624f7f51f9a16b7d5e8059aa1dbef905b --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/megatron_weight_loaders.py @@ -0,0 +1,308 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/model_loader + +from typing import Dict + +import torch +import torch.nn as nn +from vllm.model_executor.layers.linear import * +from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead, VocabParallelEmbedding +from vllm.model_executor.models import ModelRegistry + + +# NOTE(shengguangming): replace the origin weight loader function in the class +def parallel_weight_loader(self, param: torch.Tensor, loaded_weight: torch.Tensor) -> None: + """Parallel Linear weight loader.""" + assert (param.size() == loaded_weight.size( + )), "the parameter size is not align with the loaded weight size, param size: {}, loaded_weight size: {}".format( + param.size(), loaded_weight.size()) + assert (param.data.dtype == loaded_weight.data.dtype + ), "if we want to shared weights, the data type should also be the same" + + param.data = loaded_weight.data + + +def default_weight_loader(param: torch.Tensor, loaded_weight: torch.Tensor) -> None: + """Default weight loader.""" + assert param.size() == loaded_weight.size() + assert (param.data.dtype == loaded_weight.data.dtype + ), "if we want to shared weights, the data type should also be the same" + + param.data = loaded_weight.data + + +def gpt2_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_dict = dict(vllm_model.named_parameters(remove_duplicate=False)) + for name, loaded_weight in actor_weights.items(): + if "lm_head.weight" in name: + # GPT-2 ties the weights of the embedding layer and the final + # linear layer. + continue + if ".attn.bias" in name or ".attn.masked_bias" in name: + # Skip attention mask. + # NOTE: "c_attn.bias" should not be skipped. + continue + if not name.startswith("transformer."): + name = "transformer." + name + param = params_dict[name] + # The HF's GPT-2 implementation uses Conv1D instead of Linear. + # Because of this, we need to transpose the weights. + # Note(zhuohan): the logic below might break quantized models. + for conv1d_weight_name in ["c_attn", "c_proj", "c_fc"]: + if conv1d_weight_name not in name: + continue + if not name.endswith(".weight"): + continue + # TODO: check megatron + loaded_weight = loaded_weight.t() + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_te_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv.layer_norm_weight", "input_layernorm.weight"), + ("self_attention.linear_qkv.layer_norm_bias", "input_layernorm.bias"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", "self_attn.o_proj"), + ("pre_mlp_layernorm", "post_attention_layernorm"), + ("mlp.linear_fc1.layer_norm_weight", "post_attention_layernorm.weight"), + ("mlp.linear_fc1.layer_norm_bias", "post_attention_layernorm.bias"), + ("mlp.linear_fc1", "mlp.gate_up_proj"), + ("mlp.linear_fc2", "mlp.down_proj"), + ("decoder.final_layernorm", "model.norm"), + ("output_layer", "lm_head"), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith(".bias") and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", "self_attn.o_proj"), + ( + "input_layernorm", + "input_layernorm", + ), + ("pre_mlp_layernorm", "post_attention_layernorm"), + ("mlp.linear_fc1", "mlp.gate_up_proj"), + ("mlp.linear_fc2", "mlp.down_proj"), + ("decoder.final_layernorm", "model.norm"), + ("output_layer", "lm_head"), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith(".bias") and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def _replace_name(megatron_name, name_mapping): + for m_name, v_name in name_mapping: + if m_name not in megatron_name: + continue + if "layers" in megatron_name: # deal with decoder layers + megatron_name = megatron_name.replace("decoder", "model") + megatron_name_list = megatron_name.split(".") + if "layer_norm_weight" in megatron_name_list or "layer_norm_bias" in megatron_name_list: + param_name_list = megatron_name_list[:3] + param_name_list.append(v_name) + param_name = ".".join(param_name_list) + else: + param_name_list = megatron_name_list[:3] + weight_or_bias = megatron_name_list[-1] + param_name_list.append(v_name) + param_name_list.append(weight_or_bias) + param_name = ".".join(param_name_list) + return param_name + else: + param_name = megatron_name.replace(m_name, v_name) + return param_name + + +def llama_megatron_core_te_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv.layer_norm_weight", "input_layernorm.weight"), + ("self_attention.linear_qkv.layer_norm_bias", "input_layernorm.bias"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", "self_attn.o_proj"), + ("pre_mlp_layernorm", "post_attention_layernorm"), + ("mlp.linear_fc1.layer_norm_weight", "post_attention_layernorm.weight"), + ("mlp.linear_fc1.layer_norm_bias", "post_attention_layernorm.bias"), + ("mlp.linear_fc1", "mlp.gate_up_proj"), + ("mlp.linear_fc2", "mlp.down_proj"), + ("decoder.final_layernorm", "model.norm"), + ("output_layer", "lm_head"), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith(".bias") and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def llama_megatron_core_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + params_mapping = [ + # (megatron core gpt model name, vllm model name) + ("embedding.word_embeddings", "model.embed_tokens"), + ("self_attention.linear_qkv", "self_attn.qkv_proj"), + ("self_attention.linear_proj", "self_attn.o_proj"), + ( + "input_layernorm", + "input_layernorm", + ), + ("pre_mlp_layernorm", "post_attention_layernorm"), + ("mlp.linear_fc1", "mlp.gate_up_proj"), + ("mlp.linear_fc2", "mlp.down_proj"), + ("decoder.final_layernorm", "model.norm"), + ("output_layer", "lm_head"), + ] + # NOTE(shengguangming): the megatron llama may have this prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + name = _replace_name(name, params_mapping) + if name.endswith(".bias") and name not in params_dict: + continue + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +def _replace_name(megatron_name, name_mapping): + for m_name, v_name in name_mapping: + if m_name not in megatron_name: + continue + if "layers" in megatron_name: # deal with decoder layers + megatron_name = megatron_name.replace("decoder", "model") + megatron_name_list = megatron_name.split(".") + if "layer_norm_weight" in megatron_name_list or "layer_norm_bias" in megatron_name_list: + param_name_list = megatron_name_list[:3] + param_name_list.append(v_name) + param_name = ".".join(param_name_list) + else: + param_name_list = megatron_name_list[:3] + weight_or_bias = megatron_name_list[-1] + param_name_list.append(v_name) + param_name_list.append(weight_or_bias) + param_name = ".".join(param_name_list) + return param_name + else: + param_name = megatron_name.replace(m_name, v_name) + return param_name + + +def mistral_megatron_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module: + # TODO: need to implement a general way to deal with prefix + params_dict = dict(vllm_model.named_parameters()) + for name, loaded_weight in actor_weights.items(): + if "rotary_emb.inv_freq" in name: + continue + else: + param = params_dict[name] + weight_loader = getattr(param, "weight_loader", default_weight_loader) + weight_loader(param, loaded_weight) + + +__LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__ = { + ColumnParallelLinear: parallel_weight_loader, + MergedColumnParallelLinear: parallel_weight_loader, + QKVParallelLinear: parallel_weight_loader, + RowParallelLinear: parallel_weight_loader, + VocabParallelEmbedding: parallel_weight_loader, + ParallelLMHead: parallel_weight_loader, + # "ScaledActivation.weight_loader": ScaledActivation, # TODO(shengguangming): latest commit in vllm fix awq for this function and add load_weights + # "default_weight_loader": default_weight_loader +} + +# for layer_class, weight_loader in __LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__.items(): +# # setattr(layer_class, 'megatron_weight_loader', weight_loader) +# layer_class.weight_loader = weight_loader + +__MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__ = { + "GPT2LMHeadModel": gpt2_weight_loader, + "LlamaForCausalLM": llama_megatron_weight_loader, # use te backend for open-source megatron + "LLaMAForCausalLM": llama_megatron_weight_loader, + "MistralForCausalLM": mistral_megatron_weight_loader, +} + + +# the actor model is .state_dict() +# Load megatron weights +def load_megatron_weights(actor_weights: Dict, vllm_model: nn.Module): + weight_loader = _get_model_weight_loader(vllm_model.__class__.__name__) + weight_loader(actor_weights, vllm_model) + # NOTE(sgm) to reduce peak memory usage, we offload vllm model to cpu + # after init, and we need this after sync model weights for in first iter. + vllm_model = vllm_model.cuda() + + +def _get_model_weight_loader(arch: str): + if arch in __MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__: + return __MODEL_MEGATRON_WEIGHT_LOADER_REGISTRY__[arch] + raise ValueError(f"Model architectures {arch} are not supported for now. " + f"Supported architectures: {ModelRegistry.get_supported_archs()}") + + +def update_megatron_weight_loader(): + for layer_class, weight_loader in __LAYER_WEIGHT_MEGATRON_LOADER_REGISTRY__.items(): + layer_class.weight_loader = weight_loader diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/model_loader.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/model_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..f146a0eae22563650ec87bc7e5ad3ce2c19e9398 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/model_loader.py @@ -0,0 +1,338 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models +"""Utilities for selecting and loading models.""" +from typing import Dict, Optional, Union + +import torch +import torch.nn as nn +from transformers import PreTrainedModel +from vllm.config import CacheConfig, DeviceConfig, LoadConfig, LoRAConfig, ModelConfig, ParallelConfig, SchedulerConfig +from vllm.distributed.communication_op import tensor_model_parallel_all_gather +from vllm.model_executor.model_loader import BaseModelLoader +from vllm.model_executor.model_loader.loader import _initialize_model +from vllm.model_executor.model_loader.utils import set_default_torch_dtype + +from .config import LoadConfig, LoadFormat, ModelConfig +from .dtensor_weight_loaders import load_dtensor_weights, update_dtensor_weight_loader +from .hf_weight_loader import update_hf_weight_loader +from .megatron_weight_loaders import load_megatron_weights, update_megatron_weight_loader + + +def get_model( + actor_model: Union[PreTrainedModel, Dict], + model_config: ModelConfig, + load_config: LoadConfig, + device_config: DeviceConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + lora_config: Optional[LoRAConfig], + cache_config: CacheConfig = None, +) -> nn.Module: + loader = get_model_loader(load_config) + if load_config.load_format.startswith("dummy"): + return loader.load_model( + model_config=model_config, + device_config=device_config, + lora_config=lora_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + cache_config=cache_config, + ) + else: + return loader.load_model( + actor_model=actor_model, + model_config=model_config, + device_config=device_config, + lora_config=lora_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + cache_config=cache_config, + ) + + +def get_model_loader(load_config: LoadConfig) -> BaseModelLoader: + """Get a model loader based on the load format.""" + + if isinstance(load_config.load_format, type): + return load_config.load_format(load_config) + + if load_config.load_format == LoadFormat.AUTO: + update_megatron_weight_loader() + return MegatronLoader(load_config) + + # NOTE(sgm): change the weight_loader function in runtime + if load_config.load_format == LoadFormat.MEGATRON: + update_megatron_weight_loader() + return MegatronLoader(load_config) + + if load_config.load_format == LoadFormat.HF: + update_hf_weight_loader() + return HFLoader(load_config) + + if load_config.load_format == LoadFormat.DTENSOR: + update_dtensor_weight_loader() + return DTensorLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_HF: + update_hf_weight_loader() + return DummyModelLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_MEGATRON: + update_megatron_weight_loader() + return DummyModelLoader(load_config) + + if load_config.load_format == LoadFormat.DUMMY_DTENSOR: + update_dtensor_weight_loader() + return DummyModelLoader(load_config) + + raise ValueError("load format not supported in verl: {}, only support {} and {}".format( + load_config.load_format, LoadFormat.MEGATRON, LoadFormat.HF)) + + +class DummyModelLoader(BaseModelLoader): + """Model loader that will set model weights to random values.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def download_model(self, model_config: ModelConfig) -> None: + pass + + def load_model( + self, + *, + model_config: ModelConfig, + device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + cache_config: CacheConfig, + ) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, cache_config, scheduler_config) + # NOTE(woosuk): For accurate performance evaluation, we assign + # random values to the weights. + # initialize_dummy_weights(model) + return model.eval() + + +class MegatronLoader(BaseModelLoader): + """Model loader that can load the model weights from partitioned megatron model.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def download_model(self, model_config: ModelConfig) -> None: + pass # Nothing to download + + def _get_weights_iterator(actor_model: Union[PreTrainedModel, Dict]): + # NOTE(shengguangming) Load the weights from the actor model + pass + # if isinstance(actor_model, nn.Module): + # load_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), vllm_model=model) + # else: + # load_weights(actor_weights=actor_model, vllm_model=model) + # return actor_model + + def load_model( + self, + actor_model: Union[PreTrainedModel, Dict], + model_config: ModelConfig, + device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + cache_config: CacheConfig, + ) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, cache_config, scheduler_config) + + # TODO(sgm): This is a hack, we need to register the load_weight() func for each model in vllm + if isinstance(actor_model, nn.Module): + load_megatron_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), + vllm_model=model) + else: + load_megatron_weights(actor_weights=actor_model, vllm_model=model) + + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +class HFLoader(BaseModelLoader): + """Model loader that can load the model weights from model's full params.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def download_model(self, model_config: ModelConfig) -> None: + pass # Nothing to download + + def _get_weights_iterator(self, actor_model: Union[PreTrainedModel, Dict]): + if isinstance(actor_model, Dict): + return actor_model.items() + elif isinstance(actor_model, nn.Module): + return dict(actor_model.named_parameters()).items() + else: + raise ValueError(f"actor model should be Dict or nn.Module, but get {type(actor_model)}") + + def load_model( + self, + actor_model: Union[PreTrainedModel, Dict], + model_config: ModelConfig, + device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + cache_config: CacheConfig, + ) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + # with torch.device(device_config.device): + # NOTE(sgm): init the model in cpu + model = _initialize_model(model_config, self.load_config, lora_config, cache_config, scheduler_config) + model.load_weights(self._get_weights_iterator(actor_model)) + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +class DTensorLoader(BaseModelLoader): + """Model loader that can load the model weights from partitioned megatron model.""" + + def __init__(self, load_config: LoadConfig): + super().__init__(load_config) + if load_config.model_loader_extra_config: + raise ValueError(f"Model loader extra config is not supported for " + f"load format {load_config.load_format}") + + def download_model(self, model_config: ModelConfig) -> None: + pass # Nothing to download + + def _get_weights_iterator(actor_model: Union[PreTrainedModel, Dict]): + # NOTE(shengguangming) Load the weights from the actor model + pass + # if isinstance(actor_model, nn.Module): + # load_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), vllm_model=model) + # else: + # load_weights(actor_weights=actor_model, vllm_model=model) + # return actor_model + + def load_model( + self, + actor_model: Union[PreTrainedModel, Dict], + model_config: ModelConfig, + device_config: DeviceConfig, + lora_config: Optional[LoRAConfig], + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + cache_config: CacheConfig, + ) -> nn.Module: + with set_default_torch_dtype(model_config.dtype): + with torch.device(device_config.device): + model = _initialize_model(model_config, self.load_config, lora_config, cache_config, scheduler_config) + + # TODO(sgm): This is a hack, we need to register the load_weight() func for each model in vllm + if isinstance(actor_model, nn.Module): + load_dtensor_weights(actor_weights=dict(actor_model.named_parameters(remove_duplicate=False)), + vllm_model=model) + else: + load_dtensor_weights(actor_weights=actor_model, vllm_model=model) + + for _, module in model.named_modules(): + quant_method = getattr(module, "quant_method", None) + if quant_method is not None: + quant_method.process_weights_after_loading(module) + # FIXME: Remove this after Mixtral is updated + # to use quant_method. + if hasattr(module, "process_weights_after_loading"): + module.process_weights_after_loading() + # NOTE(sgm) Some weights are point to gpu, but still need this. + model = model.cuda() # NOTE (zhangchi.usc1992) We need this for vllm to profile memory usage + return model.eval() + + +# FIXME(sgm): hack the _get_logits function in vllm v0.4.2 +# as they use ray, the _get_logits result will only need to return to the driver node, +# therefore gather is enough. However, we use SPMD instead of a central scheduler, +# all_gather is required (aligned with v0.2.6) +def _get_logits(self, hidden_states: torch.Tensor, embedding: torch.Tensor, + embedding_bias: Optional[torch.Tensor]) -> torch.Tensor: + # Get the logits for the next tokens. + logits = torch.matmul(hidden_states, embedding.t()) + if embedding_bias is not None: + logits += embedding_bias + logits = tensor_model_parallel_all_gather(logits) + # Remove paddings in vocab (if any). + if logits is not None: + logits = logits[:, :self.org_vocab_size] + return logits + + +from vllm.model_executor.layers.logits_processor import LogitsProcessor + + +def logitsprocessor_init( + self, + vocab_size: int, + org_vocab_size: Optional[int] = None, + scale: float = 1.0, + logits_as_input: bool = False, + soft_cap: Optional[float] = None, +) -> None: + """ + Args: + scale: A scaling factor to apply to the logits. + """ + super(LogitsProcessor, self).__init__() + self.scale = scale + self.vocab_size = vocab_size + # Whether the input is logits (default is hidden states). + self.logits_as_input = logits_as_input + # original vocabulary size (without LoRA). + self.org_vocab_size = org_vocab_size or vocab_size + # Soft cap the logits. Used in Gemma 2. + self.soft_cap = soft_cap + # Whether to use gather or all-gather to gather the logits. + self.use_gather = False + + +LogitsProcessor.__init__ = logitsprocessor_init # use all_gather diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/model_runner.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/model_runner.py new file mode 100644 index 0000000000000000000000000000000000000000..b0cceffb52fd29ae02466b3eec51faaf0bda2bfb --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/model_runner.py @@ -0,0 +1,182 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/worker/model_runner.py + +import warnings +from enum import IntEnum +from typing import Dict, Optional, Union + +import torch +import torch.nn as nn +import vllm.envs as envs +from vllm.compilation.levels import CompilationLevel +from vllm.config import ( + CacheConfig, + DeviceConfig, + LoadConfig, + LoRAConfig, + ModelConfig, + ObservabilityConfig, + ParallelConfig, + PromptAdapterConfig, + SchedulerConfig, +) +from vllm.inputs import INPUT_REGISTRY, InputRegistry +from vllm.logger import init_logger +from vllm.lora.worker_manager import LRUCacheWorkerLoRAManager +from vllm.model_executor.models.interfaces import supports_lora +from vllm.multimodal import MULTIMODAL_REGISTRY, MultiModalRegistry +from vllm.prompt_adapter.worker_manager import LRUCacheWorkerPromptAdapterManager +from vllm.utils import DeviceMemoryProfiler, is_hip, supports_dynamo +from vllm.worker.model_runner import ModelRunner + +from .config import LoadConfig, ModelConfig +from .model_loader import get_model + +logger = init_logger(__name__) + + +# How batches are constructed. +class BatchType(IntEnum): + # Every batch is prefill. + PREFILL = 0 + # Every batch is decode. + DECODE = 1 + # Batch is a mixture of prefill and decode. + MIXED = 2 + + +class ModelRunner(ModelRunner): + + def __init__( + self, + model: Union[nn.Module, Dict], # [verl] model itself or its parameter dict + model_config: ModelConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + cache_config: CacheConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + kv_cache_dtype: Optional[str] = "auto", + is_driver_worker: bool = False, + prompt_adapter_config: Optional[PromptAdapterConfig] = None, + return_hidden_states: bool = False, + observability_config: Optional[ObservabilityConfig] = None, + input_registry: InputRegistry = INPUT_REGISTRY, + mm_registry: MultiModalRegistry = MULTIMODAL_REGISTRY, + ): + + super().__init__( + model_config, + parallel_config, + scheduler_config, + device_config, + cache_config, + load_config, + lora_config, + kv_cache_dtype, + is_driver_worker=True, # a hack + prompt_adapter_config=prompt_adapter_config, + return_hidden_states=return_hidden_states, + observability_config=observability_config, + input_registry=input_registry, + mm_registry=mm_registry, + ) + + # NOTE(sgm): add for verl + self.model = model # this will be replaced by get_model() + + def load_model(self) -> None: + logger.info("Starting to load model %s...", self.model_config.model) + with DeviceMemoryProfiler() as m: + self.model = get_model( + self.model, + model_config=self.model_config, + device_config=self.device_config, + load_config=self.load_config, + lora_config=self.lora_config, + parallel_config=self.parallel_config, + scheduler_config=self.scheduler_config, + cache_config=self.cache_config, + ) + + self.model_memory_usage = m.consumed_memory + logger.info("Loading model weights took %.4f GB", self.model_memory_usage / float(2**30)) + + if self.lora_config: + assert supports_lora(self.model), f"{self.model.__class__.__name__} does not support LoRA yet." + + if supports_multimodal(self.model): + logger.warning("Regarding multimodal models, vLLM currently " + "only supports adding LoRA to language model.") + # It's necessary to distinguish between the max_position_embeddings + # of VLMs and LLMs. + if hasattr(self.model.config, "max_position_embeddings"): + max_pos_embeddings = self.model.config.max_position_embeddings + else: + max_pos_embeddings = self.model.config.text_config.max_position_embeddings + + self.lora_manager = LRUCacheWorkerLoRAManager( + self.scheduler_config.max_num_seqs, + self.scheduler_config.max_num_batched_tokens, + self.vocab_size, + self.lora_config, + self.device, + self.model.embedding_modules, + self.model.embedding_padding_modules, + max_position_embeddings=max_pos_embeddings, + ) + self.model = self.lora_manager.create_lora_manager(self.model) + + if self.prompt_adapter_config: + self.prompt_adapter_manager = LRUCacheWorkerPromptAdapterManager( + self.scheduler_config.max_num_seqs, + self.scheduler_config.max_num_batched_tokens, + self.device, + self.prompt_adapter_config, + ) + self.model = self.prompt_adapter_manager.create_prompt_adapter_manager(self.model) + + if self.kv_cache_dtype == "fp8" and is_hip(): + # Currently only ROCm accepts kv-cache scaling factors + # via quantization_param_path and this will be deprecated + # in the future. + if self.model_config.quantization_param_path is not None: + if callable(getattr(self.model, "load_kv_cache_scales", None)): + warnings.warn( + "Loading kv cache scaling factor from JSON is " + "deprecated and will be removed. Please include " + "kv cache scaling factors in the model checkpoint.", + FutureWarning, + stacklevel=2, + ) + self.model.load_kv_cache_scales(self.model_config.quantization_param_path) + logger.info("Loaded KV cache scaling factors from %s", self.model_config.quantization_param_path) + else: + raise RuntimeError( + "Using FP8 KV cache and scaling factors provided but " + "model %s does not support loading scaling factors.", + self.model.__class__, + ) + else: + logger.warning("Using FP8 KV cache but no scaling factors " + "provided. Defaulting to scaling factors of 1.0. " + "This may lead to less accurate results!") + + if envs.VLLM_TORCH_COMPILE_LEVEL == CompilationLevel.DYNAMO_AS_IS and supports_dynamo(): + from vllm.plugins import get_torch_compile_backend + + backend = get_torch_compile_backend() or "eager" + self.model = torch.compile(self.model, fullgraph=envs.VLLM_TEST_DYNAMO_FULLGRAPH_CAPTURE, backend=backend) diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/parallel_state.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/parallel_state.py new file mode 100644 index 0000000000000000000000000000000000000000..0150c1c678e43dc5a6cb3f4426b5854ab45d8e4a --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/parallel_state.py @@ -0,0 +1,312 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Adapted from +# https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/parallel_state.py +# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. +"""Model and data parallel groups.""" +import os +from typing import Optional + +import torch +import torch.distributed +import vllm.distributed.parallel_state as ps +from vllm.distributed.parallel_state import ( + get_pp_group, + get_world_group, + init_distributed_environment, + init_model_parallel_group, +) +from vllm.logger import init_logger + +logger = init_logger(__name__) +""" +This version is strongly tied with Megatron to implement HybridEngine and weight sharing between vllm and Megatron. +- We assume the Megatron tp+dp+pp world is already established before calling this function. + +""" + +# Device mesh for using DTensor +_DEVICE_MESH = None + +# Tensor model parallel group that the current rank belongs to. +_TP = None +# Pipeline model parallel group that the current rank belongs to. +_PP = None + + +# This method is for initializing the ParallelGroup when using HybridEngine +def initialize_parallel_state( + distributed_init_method: str = "env://", + backend: str = "nccl", + tensor_model_parallel_size: int = 1, + num_tp_per_train_tp: int = 1, + pipeline_model_parallel_size: int = 1, +): + # torch.distributed.all_reduce does not free the input tensor until + # the synchronization point. This causes the memory usage to grow + # as the number of all_reduce calls increases. This env var disables + # this behavior. + # Related issue: + # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573 + os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1" + + # NOTE(sgm): Modify for verl, Env vars will be set by TORCHRUN. + rank = int(os.getenv("RANK", "-1")) + local_rank = int(os.getenv("LOCAL_RANK", "0")) + + # Use the world_size set by TORCHRUN + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + init_distributed_environment(world_size, rank, distributed_init_method, local_rank, backend) + if torch.distributed.get_world_size() > 1: + # NOTE: build a sepearate inference group with infer tp & micro dp + initialize_model_parallel_for_vllm( + tensor_model_parallel_size=tensor_model_parallel_size, + num_tensor_model_parallel_groups_per_train_tp=num_tp_per_train_tp, + ) + else: + initialize_model_parallel(tensor_model_parallel_size, pipeline_model_parallel_size, backend) + + +def ensure_model_parallel_initialized( + tensor_model_parallel_size: int, + pipeline_model_parallel_size: int = 1, + backend: Optional[str] = None, +) -> None: + """Helper to initialize model parallel groups if they are not initialized, + or ensure tensor-parallel and pipeline-parallel sizes are equal to expected + values if the model parallel groups are initialized. + """ + # get the backend of _DEVICE_WORLD_GROUP + backend = backend or torch.distributed.get_backend(get_world_group().device_group) + if not model_parallel_is_initialized(): + initialize_model_parallel(tensor_model_parallel_size, pipeline_model_parallel_size, backend) + return + + assert get_tensor_model_parallel_world_size() == tensor_model_parallel_size, ( + "tensor parallel group already initialized, but of unexpected size: " + f"{get_tensor_model_parallel_world_size()=} vs. " + f"{tensor_model_parallel_size=}") + pp_world_size = get_pp_group().world_size + assert pp_world_size == pipeline_model_parallel_size, ( + "pipeline parallel group already initialized, but of unexpected size: " + f"{pp_world_size=} vs. " + f"{pipeline_model_parallel_size=}") + + +# TODO(sgm): deviate from the v0.5.4, not pp now +def model_parallel_is_initialized(): + """Check if tensor and pipeline parallel groups are initialized.""" + return ps._TP is not None + # and _PIPELINE_MODEL_PARALLEL_GROUP is not None) + + +def initialize_model_parallel_for_vllm( + tensor_model_parallel_size: int, + num_tensor_model_parallel_groups_per_train_tp: int = 1, + pipeline_model_parallel_size: int = 1, +) -> None: + pass + + # Get world size and rank. Ensure some consistencies. + assert torch.distributed.is_initialized() + + assert isinstance(tensor_model_parallel_size, int) + + # assert num_tensor_model_parallel_groups_per_train_tp == 1 and not different_tp_group + # assert num_tensor_model_parallel_groups_per_train_tp > 1 and different_tp_group + + # Build the tensor model-parallel groups. + assert ps._TP is None, "tensor model parallel group is already initialized" + + global _TP + + world_size: int = torch.distributed.get_world_size() + + rank = torch.distributed.get_rank() + + backend = torch.distributed.get_backend() + + num_tensor_model_parallel_groups = world_size // tensor_model_parallel_size + + if num_tensor_model_parallel_groups_per_train_tp == 1: + # if tensor_model_parallel_size == train_tensor_parallel_size: + # using the same tp group as Megatron/vllm + assert _TP is None, "tensor model parallel group is already initialized" + group_ranks = [] + for i in range(num_tensor_model_parallel_groups): + ranks = range(i * tensor_model_parallel_size, (i + 1) * tensor_model_parallel_size) + group_ranks.append(ranks) + _TP = init_model_parallel_group( + group_ranks=group_ranks, + local_rank=get_world_group().local_rank, + backend=backend, + use_custom_allreduce=False, # TODO: check why True is not work in Ray trainer + use_message_queue_broadcaster=True, + ) + ps._TP = _TP + # _MICRO_DATA_PARALLEL_GROUP is move to hybrid engine + else: + # initialize a micro_dp group and a tp group + # assume training tp=4, infer tp=2, then, weight is partitioned as + # [1], [2], [3], [4] for training and [1,2], [1,2], [3,4], [3,4] for inference + + # Build the inference tp groups + # train_tp = train_tensor_parallel_size + train_tp = num_tensor_model_parallel_groups_per_train_tp * tensor_model_parallel_size + # num_tensor_model_parallel_groups_per_train_tp = train_tp // tensor_model_parallel_size + assert _TP is None, "tensor model parallel group is already initialized" + group_ranks = [] + for i in range(num_tensor_model_parallel_groups // num_tensor_model_parallel_groups_per_train_tp): + start = train_tp * i + end = train_tp * (i + 1) + for j in range(num_tensor_model_parallel_groups_per_train_tp): + ranks = list(range(start, end, num_tensor_model_parallel_groups_per_train_tp)) + for i in range(len(ranks)): + ranks[i] += j + group_ranks.append(ranks) + _TP = init_model_parallel_group( + group_ranks=group_ranks, + local_rank=get_world_group().local_rank, + backend=backend, + use_custom_allreduce=False, # TODO: check why True is not work in Ray trainer + use_message_queue_broadcaster=True, + ) + ps._TP = _TP + + # Build the pipeline model-parallel groups. + # global _PIPELINE_MODEL_PARALLEL_GROUP + # global _PIPELINE_GLOBAL_RANKS + # assert ps._PIPELINE_MODEL_PARALLEL_GROUP is None, ("pipeline model parallel group is already initialized") + + # ps._PIPELINE_MODEL_PARALLEL_GROUP = mpu.get_pipeline_model_parallel_group() + # ps._PIPELINE_GLOBAL_RANKS = mpu.get_pipeline_model_parallel_ranks() + + # TODO: init using device mesh (not support hybrid engine now) + # Build the pipeline model-parallel groups. + num_pipeline_model_parallel_groups: int = world_size // pipeline_model_parallel_size + global _PP + assert _PP is None, "pipeline model parallel group is already initialized" + group_ranks = [] + for i in range(num_pipeline_model_parallel_groups): + ranks = list(range(i, world_size, num_pipeline_model_parallel_groups)) + group_ranks.append(ranks) + # pipeline parallel does not need custom allreduce + _PP = init_model_parallel_group(group_ranks, get_world_group().local_rank, backend, use_custom_allreduce=False) + ps._PP = _PP # for verl + + +def initialize_model_parallel( + tensor_model_parallel_size: int = 1, + pipeline_model_parallel_size: int = 1, + backend: Optional[str] = None, +) -> None: + """ + NOTE: This method is a hack from the open-sourced version without + asertion of world_size = tp * pp + + Initialize model parallel groups. + + Arguments: + tensor_model_parallel_size: number of GPUs used for tensor model + parallelism. + pipeline_model_parallel_size: number of GPUs used for pipeline model + parallelism. + + Let's say we have a total of 8 GPUs denoted by g0 ... g7 and we + use 2 GPUs to parallelize the model tensor, and 4 GPUs to parallelize + the model pipeline. The present function will + create 4 tensor model-parallel groups and 2 pipeline model-parallel groups: + 4 tensor model-parallel groups: + [g0, g1], [g2, g3], [g4, g5], [g6, g7] + 2 pipeline model-parallel groups: + [g0, g2, g4, g6], [g1, g3, g5, g7] + Note that for efficiency, the caller should make sure adjacent ranks + are on the same DGX box. For example if we are using 2 DGX-1 boxes + with a total of 16 GPUs, rank 0 to 7 belong to the first box and + ranks 8 to 15 belong to the second box. + """ + # Get world size and rank. Ensure some consistencies. + assert torch.distributed.is_initialized() + world_size: int = torch.distributed.get_world_size() + backend = backend or torch.distributed.get_backend(ps.get_world_group().device_group) + + # NOTE(sgm) we don't assert world_size == tp * pp + # DP is not managed by vllm but by the VeRL WorkerGroup + # if (world_size != + # tensor_model_parallel_size * pipeline_model_parallel_size): + # raise RuntimeError( + # f"world_size ({world_size}) is not equal to " + # f"tensor_model_parallel_size ({tensor_model_parallel_size}) x " + # f"pipeline_model_parallel_size ({pipeline_model_parallel_size})") + + num_tensor_model_parallel_groups: int = world_size // tensor_model_parallel_size + rank = torch.distributed.get_rank() + global _TP + assert _TP is None, "tensor model parallel group is already initialized" + group_ranks = [] + for i in range(num_tensor_model_parallel_groups): + ranks = list(range(i * tensor_model_parallel_size, (i + 1) * tensor_model_parallel_size)) + group_ranks.append(ranks) + + # message queue broadcaster is only used in tensor model parallel group + _TP = init_model_parallel_group( + group_ranks, + get_world_group().local_rank, + backend, + use_custom_allreduce=False, # TODO: check why True is not work in Ray trainer + use_message_queue_broadcaster=True, + ) + ps._TP = _TP + + # TODO: init using device mesh (not support hybrid engine now) + # Build the pipeline model-parallel groups. + num_pipeline_model_parallel_groups: int = world_size // pipeline_model_parallel_size + global _PP + assert _PP is None, "pipeline model parallel group is already initialized" + group_ranks = [] + for i in range(num_pipeline_model_parallel_groups): + ranks = list(range(i, world_size, num_pipeline_model_parallel_groups)) + group_ranks.append(ranks) + # pipeline parallel does not need custom allreduce + _PP = init_model_parallel_group(group_ranks, get_world_group().local_rank, backend, use_custom_allreduce=False) + ps._PP = _PP # for verl + + +""" +Device mesh utilities +""" + + +def get_device_mesh(): + assert _DEVICE_MESH is not None, "device mesh is not initialized" + return _DEVICE_MESH + + +""" +Tensor model parallel utilities +""" + + +def get_tensor_model_parallel_group(): + """Get the tensor model parallel group the caller rank belongs to.""" + assert _TP is not None, "tensor model parallel group is not initialized" + return _TP.device_group + + +def get_tensor_model_parallel_world_size(): + """Return world size for the tensor model parallel group.""" + return torch.distributed.get_world_size(group=get_tensor_model_parallel_group()) + + +def get_tensor_model_parallel_rank(): + """Return my rank for the tensor model parallel group.""" + return torch.distributed.get_rank(group=get_tensor_model_parallel_group()) + + +def get_tensor_model_parallel_src_rank(): + """Calculate the global rank corresponding to the first local rank + in the tensor model parallel group.""" + global_rank = torch.distributed.get_rank() + local_world_size = get_tensor_model_parallel_world_size() + return (global_rank // local_world_size) * local_world_size diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/spmd_gpu_executor.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/spmd_gpu_executor.py new file mode 100644 index 0000000000000000000000000000000000000000..229a424c840226e2f6c148418d7c69a97807afa1 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/spmd_gpu_executor.py @@ -0,0 +1,256 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/executor/gpu_executor.py + +import os +import socket +from typing import Dict, List, Optional, Set, Tuple + +import torch +from vllm.config import ( + CacheConfig, + DeviceConfig, + LoRAConfig, + ObservabilityConfig, + ParallelConfig, + PromptAdapterConfig, + SchedulerConfig, + SpeculativeConfig, +) +from vllm.executor.executor_base import ExecutorAsyncBase, ExecutorBase +from vllm.logger import init_logger +from vllm.lora.request import LoRARequest +from vllm.model_executor.layers.sampler import SamplerOutput +from vllm.sequence import ExecuteModelRequest + +from .config import LoadConfig, ModelConfig + +logger = init_logger(__name__) + + +class SPMDGPUExecutor(ExecutorBase): + """SPMD-based multi-GPU executor implementations.""" + + def __init__( + self, + model, # pytorch model itself or its parameter dict + model_config: ModelConfig, + cache_config: CacheConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + load_config: LoadConfig, + lora_config: Optional[LoRAConfig], + speculative_config: Optional[SpeculativeConfig], + prompt_adapter_config: Optional[PromptAdapterConfig], + observability_config: Optional[ObservabilityConfig], + ) -> None: + self.model_config = model_config + self.cache_config = cache_config + self.lora_config = lora_config + self.load_config = load_config + self.parallel_config = parallel_config + self.scheduler_config = scheduler_config + self.device_config = device_config + self.speculative_config = speculative_config + self.prompt_adapter_config = prompt_adapter_config + self.observability_config = observability_config + + distributed_init_method = initialize_cluster(parallel_config) + self._init_executor(model, distributed_init_method) + + # TODO(sgm): verl not support speculative decode now + def _init_executor(self, model, distributed_init_method) -> None: + assert not self.speculative_config, "Speculative decoding not yet supported for multi-GPU backend." + + # Create the parallel worker for each GPU. + self._init_workers_sp(model, distributed_init_method) + + def _init_workers_sp(self, model, distributed_init_method: str): + # Lazy import the Worker to avoid importing torch.cuda/xformers + # before CUDA_VISIBLE_DEVICES is set in the Worker + from .worker import Worker # pylint: disable=import-outside-toplevel + + rank = int(os.getenv("RANK")) + local_rank = int(os.getenv("LOCAL_RANK")) + print(f"local rank {local_rank}") + + # see https://github.com/NVIDIA/nccl/issues/1234 + os.environ["NCCL_CUMEM_ENABLE"] = "0" + + self.worker = Worker( + model, + self.model_config, + self.parallel_config, + self.scheduler_config, + self.device_config, + self.cache_config, + self.load_config, + local_rank, + rank, + distributed_init_method, + lora_config=self.lora_config, + speculative_config=None, + prompt_adapter_config=self.speculative_config, + is_driver_worker=True, + model_runner_cls=None, # use the default one + ) + + # NOTE(shengguangming): torch.distributed.init_process_group will be called inside the init_model() + self.worker.init_device() + self.worker.load_model() + + def determine_num_available_blocks(self) -> Tuple[int, int]: + """Determine the number of available KV blocks. + + This invokes `determine_num_available_blocks` on each worker and takes + the min of the results, guaranteeing that the selected cache sizes are + compatible with all workers. + + Returns: + - tuple[num_gpu_blocks, num_cpu_blocks] + """ + # Get the maximum number of blocks that can be allocated on GPU and CPU. + num_blocks = self.worker.determine_num_available_blocks() + + # NOTE(shengguangming): Now we don't use a shared centralized controler but each process will + # have its own scheduler + num_gpu_blocks = num_blocks[0] + num_cpu_blocks = num_blocks[1] + + return num_gpu_blocks, num_cpu_blocks + + def initialize_cache(self, num_gpu_blocks: int, num_cpu_blocks: int) -> None: + """Initialize the KV cache in all workers.""" + + # NOTE: We log here to avoid multiple logs when number of workers is + # greater than one. We could log in the engine, but not all executors + # have GPUs. + logger.info("# GPU blocks: %d, # CPU blocks: %d", num_gpu_blocks, num_cpu_blocks) + + self.cache_config.num_gpu_blocks = num_gpu_blocks + self.cache_config.num_cpu_blocks = num_cpu_blocks + + if torch.distributed.get_rank() == 0: + print( + f"before init cache memory allocated: {torch.cuda.memory_allocated() / 1e9}GB, reserved: {torch.cuda.memory_reserved() / 1e9}GB" + ) + self.worker.initialize_cache(num_gpu_blocks=num_gpu_blocks, num_cpu_blocks=num_cpu_blocks) + if torch.distributed.get_rank() == 0: + print( + f"after init cache memory allocated: {torch.cuda.memory_allocated() / 1e9}GB, reserved: {torch.cuda.memory_reserved() / 1e9}GB" + ) + + # NOTE(sgm): This will not profile & capture the model(CUDAGraph) when rebuilding KVCache + def init_cache_engine(self) -> None: + self.worker._init_cache_engine() + + def free_cache_engine(self) -> None: + self.worker.free_cache_engine() + + def execute_model(self, execute_model_req) -> List[SamplerOutput]: + all_outputs = self.worker.execute_model(execute_model_req=execute_model_req) + + # NOTE(sgm): + # Each GPU in vllm under verl has its own spmd_gpu_executor, therefore all GPUs should return the outputs + # In vllm with ray, only the driver worker returns the sampling results. + return all_outputs + + def add_lora(self, lora_request: LoRARequest) -> bool: + assert lora_request.lora_int_id > 0, "lora_id must be greater than 0." + return self.worker.add_lora(lora_request=lora_request) + + def remove_lora(self, lora_id: int) -> bool: + assert lora_id > 0, "lora_id must be greater than 0." + return self.worker.remove_lora(lora_id=lora_id) + + def list_loras(self) -> Set[int]: + return self.worker.list_loras() + + def check_health(self) -> None: + # SPMDExecutor will always be healthy as long as + # it's running. + return + + # NOTE(sgm) add for verl to pass the abstract class test, not used + from vllm.prompt_adapter.request import PromptAdapterRequest + + def add_prompt_adapter(self, prompt_adapter_request: PromptAdapterRequest) -> bool: + assert prompt_adapter_request.prompt_adapter_id > 0, "prompt_adapter_id must be greater than 0." + return self.worker.add_prompt_adapter(prompt_adapter_request) + + def list_prompt_adapters(self) -> Set[int]: + return self.worker.list_prompt_adapters() + + def pin_lora(self, lora_id: int) -> bool: + assert lora_id > 0, "lora_id must be greater than 0." + return self.worker.pin_lora(lora_id) + + def pin_prompt_adapter(self, prompt_adapter_id: int) -> bool: + assert prompt_adapter_id > 0, "prompt_adapter_id must be greater than 0." + return self.worker.pin_prompt_adapter(prompt_adapter_id) + + def remove_prompt_adapter(self, prompt_adapter_id: int) -> bool: + assert prompt_adapter_id > 0, "prompt_adapter_id must be greater than 0." + return self.worker.remove_prompt_adapter(prompt_adapter_id) + + # NOTE(sgm): add for verl + def offload_model_weights(self) -> None: + self.worker.offload_model_weights() + + def sync_model_weights(self, actor_weights: Dict[str, torch.Tensor], load_format: str) -> None: + self.worker.sync_model_weights(actor_weights=actor_weights, load_format=load_format) + + +def initialize_cluster( + parallel_config: ParallelConfig, + engine_use_ray: bool = False, + ray_address: Optional[str] = None, +) -> Tuple[str, Optional[None]]: + """Initialize the distributed cluster probably with Ray. + + Args: + parallel_config: The configurations for parallel execution. + + Returns: + The `distributed_init_method` is the address for initializing the + distributed backend. + """ + + # Initialize cluster locally. + port = get_open_port() + # We need to setup the distributed init method to make sure + # the distributed megatron code (e.g., get world size) works correctly. + # distributed_init_method = f"tcp://localhost:{port}" + distributed_init_method = "env://" + return distributed_init_method + + +def get_open_port(): + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("", 0)) + return s.getsockname()[1] + + +# TODO(sgm): not implemented async executor yet +class SPMDGPUExecutorAsync(SPMDGPUExecutor, ExecutorAsyncBase): + + async def execute_model_async(self, execute_model_req: ExecuteModelRequest) -> List[SamplerOutput]: + """Executes one model step on the given sequences.""" + raise NotImplementedError + + async def check_health_async(self) -> None: + """Checks if the executor is healthy. If not, it should raise an + exception.""" + self.check_health() diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/tokenizer.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/tokenizer.py new file mode 100644 index 0000000000000000000000000000000000000000..b0b4d0e27c84fc0358411d7bf29e0702aac929b9 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/tokenizer.py @@ -0,0 +1,40 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/tokenizer_group/tokenizer_group.py + +from typing import Optional + +from transformers import PreTrainedTokenizer +from vllm.transformers_utils.tokenizer_group import TokenizerGroup +from vllm.utils import LRUCache + + +class TokenizerGroup(TokenizerGroup): + """A group of tokenizers that can be used for LoRA adapters.""" + + def __init__(self, tokenizer: PreTrainedTokenizer, enable_lora: bool, max_num_seqs: int, + max_input_length: Optional[int]): + self.enable_lora = enable_lora + self.max_input_length = max_input_length + self.tokenizer = tokenizer + self.lora_tokenizers = LRUCache[PreTrainedTokenizer](capacity=max_num_seqs) if enable_lora else None + + # FIXME(sgm): for simplicity, we assign the special token here + @property + def pad_token_id(self): + return self.tokenizer.pad_token_id + + @property + def eos_token_id(self): + return self.tokenizer.eos_token_id diff --git a/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/worker.py b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/worker.py new file mode 100644 index 0000000000000000000000000000000000000000..cb1a7ab80c7526177cc0e53963f0e2e85d683334 --- /dev/null +++ b/frameworks/agentgym_rl/verl/third_party/vllm/vllm_v_0_6_3/worker.py @@ -0,0 +1,333 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023 The vLLM team. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/worker/worker.py +"""A GPU worker class.""" +import gc +import os +from typing import Dict, List, Optional, Tuple, Type, Union + +import torch +import torch.distributed +import torch.nn as nn +from vllm.config import ( + CacheConfig, + DeviceConfig, + LoRAConfig, + ParallelConfig, + PromptAdapterConfig, + SchedulerConfig, + SpeculativeConfig, +) + +# TODO(sgm): check why vllm has similar file in vllm.model_executor.parallel_utils.parallel_state +from vllm.distributed import get_tensor_model_parallel_group, init_distributed_environment, set_custom_all_reduce +from vllm.model_executor import set_random_seed +from vllm.model_executor.layers.sampler import SamplerOutput +from vllm.sequence import ExecuteModelRequest, IntermediateTensors +from vllm.worker.cache_engine import CacheEngine +from vllm.worker.embedding_model_runner import EmbeddingModelRunner +from vllm.worker.model_runner import GPUModelRunnerBase +from vllm.worker.model_runner_base import ModelRunnerInputBase +from vllm.worker.worker import Worker, _check_if_gpu_supports_dtype +from vllm.worker.worker_base import WorkerInput + +from .config import LoadConfig, LoadFormat, ModelConfig +from .dtensor_weight_loaders import load_dtensor_weights +from .hf_weight_loader import load_hf_weights +from .megatron_weight_loaders import load_megatron_weights +from .model_runner import ModelRunner +from .parallel_state import ensure_model_parallel_initialized + + +class Worker(Worker): + """A worker class that executes (a partition of) the model on a GPU. + + Each worker is associated with a single GPU. The worker is responsible for + maintaining the KV cache and executing the model on the GPU. In case of + distributed inference, each worker is assigned a partition of the model. + """ + + def __init__( + self, + model: Union[nn.Module, Dict], # model itself or its parameter dict + model_config: ModelConfig, + parallel_config: ParallelConfig, + scheduler_config: SchedulerConfig, + device_config: DeviceConfig, + cache_config: CacheConfig, + load_config: LoadConfig, + local_rank: int, + rank: int, + distributed_init_method: str, + lora_config: Optional[LoRAConfig] = None, + speculative_config: Optional[SpeculativeConfig] = None, + prompt_adapter_config: Optional[PromptAdapterConfig] = None, + is_driver_worker: bool = False, + model_runner_cls: Optional[Type[GPUModelRunnerBase]] = None, + ) -> None: + # self.model = model # will be replaced in the init_model + self.model_config = model_config + self.parallel_config = parallel_config + self.parallel_config.rank = rank + self.scheduler_config = scheduler_config + self.device_config = device_config + self.cache_config = cache_config + self.local_rank = local_rank + self.rank = rank + self.distributed_init_method = distributed_init_method + self.lora_config = lora_config + self.load_config = load_config + self.prompt_adapter_config = prompt_adapter_config + self.is_driver_worker = is_driver_worker # TODO: we don't need driver + # if parallel_config and is_driver_worker: + # assert rank % parallel_config.tensor_parallel_size == 0, \ + # "Driver worker should be rank 0 of tensor parallel group." + if self.model_config.trust_remote_code: + # note: lazy import to avoid importing torch before initializing + from vllm.utils import init_cached_hf_modules + + init_cached_hf_modules() + + # Return hidden states from target model if the draft model is an + # mlp_speculator + speculative_args = ( + {} if speculative_config is None or (speculative_config.draft_model_config.model == model_config.model) or + (speculative_config.draft_model_config.hf_config.model_type not in ["medusa", "mlp_speculator"]) else { + "return_hidden_states": True + }) + + # TODO(sgm): set correct model runner class + ModelRunnerClass: Type[GPUModelRunnerBase] = ModelRunner + if model_runner_cls is not None: + ModelRunnerClass = model_runner_cls + elif self.model_config.embedding_mode: + ModelRunnerClass = EmbeddingModelRunner + self.model_runner: GPUModelRunnerBase = ModelRunnerClass( + model, # [VERL]: add for verl + model_config, + parallel_config, + scheduler_config, + device_config, + cache_config, + load_config=load_config, + lora_config=self.lora_config, + kv_cache_dtype=self.cache_config.cache_dtype, + is_driver_worker=is_driver_worker, + prompt_adapter_config=prompt_adapter_config, + **speculative_args, + ) + + # Uninitialized cache engine. Will be initialized by + # initialize_cache. + self.cache_engine: List[CacheEngine] = None + # Initialize gpu_cache as embedding models don't initialize kv_caches + self.gpu_cache: Optional[List[List[torch.Tensor]]] = None + + # NOTE(sgm): [VERL] For offloading inference engine params + self.cpu_model = None + + def init_device(self) -> None: + if self.device_config.device.type == "cuda": + # torch.distributed.all_reduce does not free the input tensor until + # the synchronization point. This causes the memory usage to grow + # as the number of all_reduce calls increases. This env var disables + # this behavior. + # Related issue: + # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573 + os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1" + + # NOTE(sgm): Modify for verl, Env vars will be set by TORCHRUN. + self.rank = self.rank if self.rank is not None else int(os.getenv("RANK", "-1")) + local_rank = int(os.getenv("LOCAL_RANK", "0")) + self.device = torch.device(f"cuda:{local_rank}") + if self.rank < 0: + raise ValueError("Invalid or unspecified rank.") + torch.cuda.set_device(self.device) + + # Use the world_size set by TORCHRUN + world_size = int(os.getenv("WORLD_SIZE", "-1")) + assert world_size != -1, "The world_size is set to -1, not initialized by TORCHRUN" + self.parallel_config.world_size = world_size + + _check_if_gpu_supports_dtype(self.model_config.dtype) + torch.cuda.empty_cache() + self.init_gpu_memory = torch.cuda.mem_get_info()[0] + else: + raise RuntimeError(f"Not support device type: {self.device_config.device}") + + # Initialize the distributed environment. + init_worker_distributed_environment(self.parallel_config, self.rank, self.distributed_init_method, + self.local_rank) + # Set random seed. + set_random_seed(self.model_config.seed) + # self.model = get_model(actor_model=self.model, model_config=self.model_config) + + @torch.inference_mode() + def determine_num_available_blocks(self) -> Tuple[int, int]: + """Profiles the peak memory usage of the model to determine how many + KV blocks may be allocated without OOMs. + + The engine will first conduct a profiling of the existing memory usage. + Then, it calculate the maximum possible number of GPU and CPU blocks + that can be allocated with the remaining free memory. + + .. tip:: + You may limit the usage of GPU memory + by adjusting the `gpu_memory_utilization` parameter. + """ + # Profile the memory usage of the model and get the maximum number of + # cache blocks that can be allocated with the remaining free memory. + torch.cuda.empty_cache() + # torch.cuda.reset_peak_memory_stats() + + # Execute a forward pass with dummy inputs to profile the memory usage + # of the model. + self.model_runner.profile_run() + + # Calculate the number of blocks that can be allocated with the + # profiled peak memory. + torch.cuda.synchronize() + free_gpu_memory, total_gpu_memory = torch.cuda.mem_get_info() + peak_memory = total_gpu_memory - free_gpu_memory + + assert peak_memory > 0, ("Error in memory profiling. This happens when the GPU memory was " + "not properly cleaned up before initializing the vLLM instance.") + + cache_block_size = self.get_cache_block_size_bytes() + + # NOTE(sgm) [VERL] use the remaining memory + num_gpu_blocks = int((free_gpu_memory * self.cache_config.gpu_memory_utilization) // cache_block_size) + # num_gpu_blocks = int((total_gpu_memory * self.cache_config.gpu_memory_utilization - peak_memory) // cache_block_size) + + num_cpu_blocks = int(self.cache_config.swap_space_bytes // cache_block_size) + num_gpu_blocks = max(num_gpu_blocks, 0) + num_cpu_blocks = max(num_cpu_blocks, 0) + if self.model_runner.lora_manager: + self.model_runner.remove_all_loras() + + # NOTE(sgm): Add for [VERL], synchronize number of blocks with all the rank + num_gpu_blocks = torch.tensor([num_gpu_blocks], device="cuda") + num_cpu_blocks = torch.tensor([num_cpu_blocks], device="cuda") + + torch.distributed.all_reduce(num_gpu_blocks, + op=torch.distributed.ReduceOp.MIN, + group=get_tensor_model_parallel_group().device_group) + torch.distributed.all_reduce(num_cpu_blocks, + op=torch.distributed.ReduceOp.MIN, + group=get_tensor_model_parallel_group().device_group) + num_gpu_blocks = num_gpu_blocks.item() + num_cpu_blocks = num_cpu_blocks.item() + gc.collect() + torch.cuda.empty_cache() + return num_gpu_blocks, num_cpu_blocks + + def _init_cache_engine(self): + if self.cache_engine is None and self.gpu_cache is None: + super()._init_cache_engine() + + def free_cache_engine(self): + # ensure `enforce_eager=True` + self.cache_engine = None + self.gpu_cache = None + + # NOTE(sgm): [VERL]: adapt from _execute_model_spmd() + def execute_model(self, + execute_model_req: ExecuteModelRequest, + intermediate_tensors: Optional[IntermediateTensors] = None) -> Optional[List[SamplerOutput]]: + """ + Execute model in Single Program Multiple Data (SPMD) fashion. + All workers take the same request, prepare the input and + execute the model. + """ + assert execute_model_req is not None, ("_execute_model_spmd() requires each worker to take in an " + "ExecuteModelRequest") + worker_input: WorkerInput = self.prepare_worker_input(execute_model_req=execute_model_req) + model_input: ModelRunnerInputBase = self.model_runner.prepare_model_input( + execute_model_req.seq_group_metadata_list) + + # verl.worker.workerbase.WorkerBase + # swap cache + super().execute_worker(worker_input) + + # If there is no input, we don't need to execute the model. + if worker_input.num_seq_groups == 0: + return [] + + return self.model_runner.execute_model( + model_input, + self.kv_cache[worker_input.virtual_engine] if self.kv_cache is not None else None, + intermediate_tensors, + ) + + # assume the input is .state_dict() + def sync_model_weights(self, actor_weights: Dict, load_format: str): + if load_format in [LoadFormat.MEGATRON, LoadFormat.AUTO]: + load_megatron_weights(actor_weights, self.model_runner.model) + elif load_format == LoadFormat.HF: + # full model state dict without no sharding + load_hf_weights(actor_weights, self.model_runner.model) + elif load_format == LoadFormat.DTENSOR: + load_dtensor_weights(actor_weights, self.model_runner.model) + + def offload_model_weights(self) -> None: + if self.cpu_model == None: + self.cpu_model = {} + for name, params in self.model_runner.model.named_parameters(): + self.cpu_model[name] = torch.empty_like(params, device="cpu") + params.data = self.cpu_model[name] + else: + for name, params in self.model_runner.model.named_parameters(): + params.data = self.cpu_model[name] + + +def init_worker_distributed_environment( + parallel_config: ParallelConfig, + rank: int, + distributed_init_method: Optional[str] = "env://", + local_rank: int = -1, +) -> None: + """Initialize the distributed environment.""" + set_custom_all_reduce(not parallel_config.disable_custom_all_reduce) + + # NOTE(sgm) use tcp://localhost:xxxx will hang in HF setting without megatron + init_distributed_environment(parallel_config.world_size, rank, distributed_init_method, local_rank) + + ensure_model_parallel_initialized( + tensor_model_parallel_size=parallel_config.tensor_parallel_size, + pipeline_model_parallel_size=parallel_config.pipeline_parallel_size, + ) + + # TODO(sgm): check whether need this + # if pynccl_utils.is_initialized(): + # pynccl_world_size = pynccl_utils.get_world_size() + # if pynccl_world_size != parallel_config.world_size: + # raise RuntimeError( + # "pynccl is already initialized but the pynccl world " + # "size does not match parallel_config.world_size " + # f"({pynccl_world_size} vs. {parallel_config.world_size}).") + # elif parallel_config.world_size > 1: + # # NOTE(woosuk): We don't initialize pynccl process group when world size + # # is 1. + # # NOTE(kaichao): By default, pynccl is initialized for tp group. + # pynccl_utils.init_process_group( + # group=get_tensor_model_parallel_cpu_group()) + + # # Initialize a custom fast all-reduce implementation. + # if not parallel_config.disable_custom_all_reduce: + # init_custom_ar() + + # A small all_reduce for warmup. + torch.distributed.all_reduce(torch.zeros(1).cuda()) + # if pynccl_utils.is_initialized(): + # pynccl_utils.all_reduce(torch.zeros(1).cuda()) diff --git a/frameworks/agentgym_rl/verl/utils/__init__.py b/frameworks/agentgym_rl/verl/utils/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e453070a16370cd7006e0a7700c8550a56f19051 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/__init__.py @@ -0,0 +1,18 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from . import tokenizer +from .tokenizer import * + +__all__ = tokenizer.__all__ \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/agent_dataset/README.md b/frameworks/agentgym_rl/verl/utils/agent_dataset/README.md new file mode 100644 index 0000000000000000000000000000000000000000..f886a70aabf443fb167453d667529b62f3311765 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/agent_dataset/README.md @@ -0,0 +1,16 @@ +# Dataset Format +## RLHF dataset +We combine all the data sources into a single parquet files. We directly organize the prompt into the chat format so that multi-turn chats can be easily incorporated. In the prompt, we may add instruction following texts to guide the model output the answers in a particular format so that we can extract the answers. + +Math problems +```json +{ + "data_source": "openai/gsm8k", + "prompt": [{"role": "user", "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Let's think step by step and output the final answer after \"####\""}], + "ability": "math", + "reward_model": { + "style": "rule", + "ground_truth": ["72"] + }, +} +``` diff --git a/frameworks/agentgym_rl/verl/utils/agent_dataset/__init__.py b/frameworks/agentgym_rl/verl/utils/agent_dataset/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..faafe37f608e065b78318dc19bf5f8c7f52585bd --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/agent_dataset/__init__.py @@ -0,0 +1,16 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .rl_dataset import RLHFDataset +from .sft_dataset import SFTDataset \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/agent_dataset/rl_dataset.py b/frameworks/agentgym_rl/verl/utils/agent_dataset/rl_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..3b9082e0ba044e68264ce36066a4a168f7325f53 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/agent_dataset/rl_dataset.py @@ -0,0 +1,159 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2023-2024 SGLang Team +# Copyright 2025 ModelBest Inc. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import copy +import logging +import os +import re +from collections import defaultdict +from typing import List, Optional, Union + +import datasets +import numpy as np +import torch +from omegaconf import DictConfig, ListConfig +from torch.utils.data import Dataset +from transformers import PreTrainedTokenizer, ProcessorMixin + +import verl.utils.torch_functional as verl_F +from verl.utils.model import compute_position_id_with_mask +from verl.utils.agentgym.client import init_env_client +logger = logging.getLogger(__name__) + + +def collate_fn(data_list: list[dict]) -> dict: + """Collate a batch of data.""" + tensors = defaultdict(list) + non_tensors = defaultdict(list) + + for data in data_list: + for key, val in data.items(): + if isinstance(val, torch.Tensor): + tensors[key].append(val) + else: + non_tensors[key].append(val) + + for key, val in tensors.items(): + tensors[key] = torch.stack(val, dim=0) + + for key, val in non_tensors.items(): + non_tensors[key] = np.array(val, dtype=object) + + return {**tensors, **non_tensors} + + +class RLHFDataset(Dataset): + """ + We assume the dataset contains a column that contains prompts and other information + """ + + def __init__( + self, + data_file: str, + tokenizer: PreTrainedTokenizer, + data_config: DictConfig, + agentgym_config: DictConfig, + ): + + self.data_file = copy.deepcopy(data_file) + self.original_data_file = copy.deepcopy(data_file) # use for resume + self.tokenizer = tokenizer + self.data_config = data_config + self.agentgym_config = agentgym_config + + self.cache_dir = os.path.expanduser(data_config.get("cache_dir", "~/.cache/verl/rlhf")) + self.prompt_key = data_config.get("prompt_key", "prompt") + self.image_key = data_config.get("image_key", "images") + self.video_key = data_config.get("video_key", "videos") + self.max_prompt_length = data_config.get("max_prompt_length", 1024) + + self.return_raw_chat = data_config.get("return_raw_chat", False) + self.return_full_prompt = data_config.get("return_full_prompt", False) + self.truncation = data_config.get("truncation", "error") + self.filter_overlong_prompts = data_config.get("filter_overlong_prompts", True) + + self.num_workers = data_config.get("filter_overlong_prompts_workers", max(1, os.cpu_count() // 4)) + self.num_workers = min(self.num_workers, os.cpu_count()) + self.chat_template_func = data_config.get("chat_template_func", None) + self.need_tools_kwargs = data_config.get("need_tools_kwargs", False) + self.filter_prompts = data_config.get("filter_prompts", True) + self.serialize_dataset = False + self._read_files_and_tokenize() + # get agentgym client + self.env_client = init_env_client(self.agentgym_config) + + def _read_files_and_tokenize(self): + self.dataframe = datasets.load_dataset("json", data_files=self.data_file)["train"] + print(f"dataset len: {len(self.dataframe)}") + + def resume_dataset_state(self): + self.serialize_dataset = not hasattr(self, "original_data_file") + # resume dataframe if not it's serialized in data.pt + if not self.serialize_dataset: + self._read_files_and_tokenize() + else: + print(r"old dataloader ckpt file is used, please train from scratch for better ckpt performance") + + def __len__(self): + return len(self.dataframe) + + def _build_messages(self, example: dict): + example["data_source"] = example[self.prompt_key].split("_")[0] + messages = [{"role": "user", "content": self.env_client.conversation_start[0]["value"]}, + {"role": "assistant", "content": self.env_client.conversation_start[1]["value"]}] + prompt_with_chat_template = "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\n" + self.env_client.conversation_start[0]["value"] + "<|im_end|>\n<|im_start|>assistant\n" + self.env_client.conversation_start[1]["value"] + "<|im_end|>" + return messages, prompt_with_chat_template + + def __getitem__(self, item): + """ + Note that we also return the raw_input_ids so that it can be combined with other chat template + """ + row_dict: dict = self.dataframe[item] + + messages, prompt_with_chat_template = self._build_messages(row_dict) + + input_ids, attention_mask = verl_F.tokenize_and_postprocess_data(prompt=prompt_with_chat_template, + tokenizer=self.tokenizer, + max_length=self.max_prompt_length, + pad_token_id=self.tokenizer.pad_token_id, + left_pad=True, + truncation=self.truncation) + + position_ids = compute_position_id_with_mask(attention_mask) + + row_dict['input_ids'] = input_ids[0] + row_dict['attention_mask'] = attention_mask[0] + row_dict['position_ids'] = position_ids[0] + + # encode prompts without chat template + if self.return_raw_chat: + row_dict['raw_prompt'] = messages + + # add index for each prompt + index = row_dict.get("extra_info", {}).get("index", 0) + row_dict["index"] = index + + return row_dict + + def __getstate__(self): + if not self.serialize_dataset: + state = self.__dict__.copy() + + if "dataframe" in state: + del state["dataframe"] + return state + + return self.__dict__.copy() diff --git a/frameworks/agentgym_rl/verl/utils/agent_dataset/sft_dataset.py b/frameworks/agentgym_rl/verl/utils/agent_dataset/sft_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..980f49a4d11be60f7997134d14a2b6ca05c4ecff --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/agent_dataset/sft_dataset.py @@ -0,0 +1,127 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +SFT dataset +- We assume user pass a single parquet file. +- We load all the data into the memory. +Each parquet file contains +""" + +import pandas as pd + +import torch +from torch.utils.data import Dataset +from transformers import PreTrainedTokenizer + +from verl.utils.model import compute_position_id_with_mask +from verl.utils import hf_tokenizer + + +class SFTDataset(Dataset): + """ + This is an in-memory SFTDataset + """ + + def __init__(self, + json_file: str, + tokenizer, + prompt_key='conversations', + max_length=4096, + truncation='right'): + assert truncation in ['error', 'left', 'right'] + self.truncation = truncation + + self.json_file = json_file + if isinstance(tokenizer, str): + tokenizer = hf_tokenizer(tokenizer) + self.tokenizer: PreTrainedTokenizer = tokenizer + + self.prompt_key = prompt_key + + self.max_length = max_length + + self._read_files_and_tokenize() + + def _read_files_and_tokenize(self): + self.dataframe = pd.read_json(self.json_file) + self.prompts = self.dataframe[self.prompt_key].tolist() + + def __len__(self): + return len(self.prompts) + + def __getitem__(self, item): + tokenizer = self.tokenizer + + prompt = self.prompts[item] + + # string + system_chat_dict = {'role': 'system', 'content': ''} + system_chat_str = tokenizer.apply_chat_template([system_chat_dict], tokenize=False) + prompt_ids_output = tokenizer(tokenizer.apply_chat_template([{'role': 'user', 'content': prompt[0]['value']}], tokenize=False), return_tensors='pt', add_special_tokens=False) + input_ids = prompt_ids_output['input_ids'][0] + attention_mask = prompt_ids_output['attention_mask'][0] + loss_mask = torch.zeros_like(input_ids) + for c in prompt[1:]: + if c['from'] == 'system': + prompt_ids_output = tokenizer(tokenizer.apply_chat_template([system_chat_dict, {'role': 'system', 'content': c['value']}], tokenize=False).replace(system_chat_str, ""), return_tensors='pt', add_special_tokens=False) + input_ids = torch.concat([input_ids, prompt_ids_output['input_ids'][0]]) + attention_mask = torch.concat([attention_mask, prompt_ids_output['attention_mask'][0]]) + loss_mask = torch.cat([loss_mask, torch.zeros_like(prompt_ids_output['input_ids'][0])]) + elif c['from'] == 'human': + prompt_ids_output = tokenizer(tokenizer.apply_chat_template([system_chat_dict, {'role': 'user', 'content': c['value']}], tokenize=False).replace(system_chat_str, ""), return_tensors='pt', add_special_tokens=False) + input_ids = torch.concat([input_ids, prompt_ids_output['input_ids'][0]]) + attention_mask = torch.concat([attention_mask, prompt_ids_output['attention_mask'][0]]) + loss_mask = torch.cat([loss_mask, torch.zeros_like(prompt_ids_output['input_ids'][0])]) + elif c['from'] == 'gpt': + prompt_ids_output = tokenizer(tokenizer.apply_chat_template([system_chat_dict, {'role': 'assistant', 'content': c['value']}], tokenize=False).replace(system_chat_str, ""), return_tensors='pt', add_special_tokens=False) + input_ids = torch.concat([input_ids, prompt_ids_output['input_ids'][0]]) + attention_mask = torch.concat([attention_mask, prompt_ids_output['attention_mask'][0]]) + loss_mask = torch.cat([loss_mask, torch.ones_like(prompt_ids_output['input_ids'][0])]) + else: + raise NotImplementedError + + # padding to max length + sequence_length = input_ids.shape[0] + if sequence_length < self.max_length: + padded_input_ids = torch.ones(size=(self.max_length - sequence_length,), + dtype=input_ids.dtype) * self.tokenizer.pad_token_id + padded_attention_mask = torch.zeros(size=(self.max_length - sequence_length,), dtype=attention_mask.dtype) + padded_loss_mask = torch.zeros(size=(self.max_length - sequence_length,), dtype=loss_mask.dtype) + + input_ids = torch.cat((input_ids, padded_input_ids)) + attention_mask = torch.cat((attention_mask, padded_attention_mask)) + loss_mask = torch.cat((loss_mask, padded_loss_mask)) + elif sequence_length > self.max_length: + if self.truncation == 'left': + # actually, left truncation may not be reasonable + input_ids = input_ids[-self.max_length:] + attention_mask = attention_mask[-self.max_length:] + loss_mask = loss_mask[-self.max_length:] + elif self.truncation == 'right': + input_ids = input_ids[:self.max_length] + attention_mask = attention_mask[:self.max_length] + loss_mask = loss_mask[:self.max_length] + elif self.truncation == 'error': + raise NotImplementedError(f'{sequence_length=} is larger than {self.max_length=}') + else: + raise NotImplementedError(f'Unknown truncation method {self.truncation}') + + position_ids = compute_position_id_with_mask(attention_mask) + + return { + 'input_ids': input_ids, + 'attention_mask': attention_mask, + 'position_ids': position_ids, + 'loss_mask': loss_mask + } diff --git a/frameworks/agentgym_rl/verl/utils/agentgym/client.py b/frameworks/agentgym_rl/verl/utils/agentgym/client.py new file mode 100644 index 0000000000000000000000000000000000000000..c0fcc5a02b1be017ec54ebbdb36708a7fd57651d --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/agentgym/client.py @@ -0,0 +1,56 @@ +from contextlib import contextmanager +import os +import time +from agentenv.envs import ( + AcademiaEnvClient, + AlfWorldEnvClient, + BabyAIEnvClient, + MazeEnvClient, + MovieEnvClient, + SciworldEnvClient, + SheetEnvClient, + SqlGymEnvClient, + TextCraftEnvClient, + TodoEnvClient, + WeatherEnvClient, + WebarenaEnvClient, + WebshopEnvClient, + WordleEnvClient, + SearchQAEnvClient, +) + +def init_env_client(args): + # task_name - task dict + envclient_classes = { + "webshop": WebshopEnvClient, + "alfworld": AlfWorldEnvClient, + "babyai": BabyAIEnvClient, + "sciworld": SciworldEnvClient, + "textcraft": TextCraftEnvClient, + "webarena": WebarenaEnvClient, + "sqlgym": SqlGymEnvClient, + "maze": MazeEnvClient, + "wordle": WordleEnvClient, + "weather": WeatherEnvClient, + "todo": TodoEnvClient, + "movie": MovieEnvClient, + "sheet": SheetEnvClient, + "academia": AcademiaEnvClient, + "searchqa": SearchQAEnvClient, + } + # select task according to the name + envclient_class = envclient_classes.get(args.task_name.lower(), None) + if envclient_class is None: + raise ValueError(f"Unsupported task name: {args.task_name}") + retry = 0 + while True: + try: + env_client = envclient_class(env_server_base=args.env_addr, data_len=1, timeout=2400) + break + except Exception as e: + retry += 1 + print(f"Failed to connect to env server, retrying...({retry}/{args.max_retries})") + if retry > args.max_retries: + raise e + time.sleep(5) + return env_client \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/checkpoint/__init__.py b/frameworks/agentgym_rl/verl/utils/checkpoint/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..7a7aadbc9d946451bd151abc3e306357eda8140a --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/checkpoint/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/checkpoint/checkpoint_manager.py b/frameworks/agentgym_rl/verl/utils/checkpoint/checkpoint_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..11a8f55fe6fdc209952123fb6669165901db5084 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/checkpoint/checkpoint_manager.py @@ -0,0 +1,137 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import os +import shutil +from filelock import FileLock +import tempfile + +import torch +import torch.distributed +from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, StateDictType +from transformers import PreTrainedTokenizer +import numpy as np +import random + + +class BaseCheckpointManager: + """ + A checkpoint manager that saves and loads + - model + - optimizer + - lr_scheduler + - extra_states + in a SPMD way. + + We save + - sharded model states and optimizer states + - full lr_scheduler states + - huggingface tokenizer and config for ckpt merge + """ + + def __init__(self, model: FSDP, optimizer: torch.optim.Optimizer, + lr_scheduler: torch.optim.lr_scheduler.LRScheduler, tokenizer: PreTrainedTokenizer): + self.previous_global_step = None + self.previous_save_local_path = None + + self.model = model + self.optimizer = optimizer + self.lr_scheduler = lr_scheduler + self.tokenizer = tokenizer + + assert isinstance(self.model, FSDP) + self.rank = torch.distributed.get_rank() + self.world_size = torch.distributed.get_world_size() + + def load_checkpoint(self, *args, **kwargs): + raise NotImplementedError + + def save_checkpoint(self, *args, **kwargs): + raise NotImplementedError + + def remove_previous_save_local_path(self): + if not self.previous_save_local_path: + return + + abs_path = os.path.abspath(self.previous_save_local_path) + print(f'Checkpoint manager remove previous save local path: {abs_path}') + if not os.path.exists(abs_path): + return + + # remove previous local_path + shutil.rmtree(abs_path, ignore_errors=True) + + @staticmethod + def local_mkdir(path): + if not os.path.isabs(path): + working_dir = os.getcwd() + path = os.path.join(working_dir, path) + + # Using hash value of path as lock file name to avoid long file name + lock_filename = f"ckpt_{hash(path) & 0xFFFFFFFF:08x}.lock" + lock_path = os.path.join(tempfile.gettempdir(), lock_filename) + + try: + with FileLock(lock_path, timeout=60): # Add timeout + # make a new dir + os.makedirs(path, exist_ok=True) + except Exception as e: + print(f"Warning: Failed to acquire lock for {path}: {e}") + # Even if the lock is not acquired, try to create the directory + os.makedirs(path, exist_ok=True) + + return path + + @staticmethod + def get_rng_state(): + rng_state = { + 'cpu': torch.get_rng_state(), + 'cuda': torch.cuda.get_rng_state(), + 'numpy': np.random.get_state(), + 'random': random.getstate(), + } + return rng_state + + @staticmethod + def load_rng_state(rng_state): + torch.set_rng_state(rng_state['cpu']) + torch.cuda.set_rng_state(rng_state['cuda']) + np.random.set_state(rng_state['numpy']) + random.setstate(rng_state['random']) + + +def find_latest_ckpt_path(path, directory_format="global_step_{}"): + if path is None: + return None + + tracker_file = get_checkpoint_tracker_filename(path) + if not os.path.exists(tracker_file): + print("Checkpoint tracker file does not exist: %s", tracker_file) + return None + + with open(tracker_file, "rb") as f: + iteration = int(f.read().decode()) + ckpt_path = os.path.join(path, directory_format.format(iteration)) + if not os.path.exists(ckpt_path): + print("Checkpoint does not exist: %s", ckpt_path) + return None + + print("Found checkpoint: %s", ckpt_path) + return ckpt_path + + +def get_checkpoint_tracker_filename(root_path: str): + """ + Tracker file rescords the latest chckpoint during training to restart from. + """ + return os.path.join(root_path, "latest_checkpointed_iteration.txt") diff --git a/frameworks/agentgym_rl/verl/utils/checkpoint/fsdp_checkpoint_manager.py b/frameworks/agentgym_rl/verl/utils/checkpoint/fsdp_checkpoint_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..e5ec9efd001de7e3f0532b8630a73b72d6309c9b --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/checkpoint/fsdp_checkpoint_manager.py @@ -0,0 +1,149 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import ray +import os + +import warnings + +import torch +import torch.distributed +from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, StateDictType +from torch.distributed.fsdp import ShardedStateDictConfig, ShardedOptimStateDictConfig + +from verl.utils.fs import copy_local_path_from_hdfs, is_non_local + +from transformers import PreTrainedTokenizer + +from .checkpoint_manager import BaseCheckpointManager + + +class FSDPCheckpointManager(BaseCheckpointManager): + """ + A checkpoint manager that saves and loads + - model + - optimizer + - lr_scheduler + - extra_states + in a SPMD way. + + We save + - sharded model states and optimizer states + - full lr_scheduler states + - huggingface tokenizer and config for ckpt merge + """ + + def __init__(self, model: FSDP, optimizer: torch.optim.Optimizer, + lr_scheduler: torch.optim.lr_scheduler.LRScheduler, tokenizer: PreTrainedTokenizer, *args, **kwargs): + super().__init__(model, optimizer, lr_scheduler, tokenizer) + + def load_checkpoint(self, path=None, del_local_after_load=False, *args, **kwargs): + if path is None: + return + + # every rank download its own checkpoint + remote_model_path = os.path.join(path, f'model_world_size_{self.world_size}_rank_{self.rank}.pt') + remote_optim_path = os.path.join(path, f'optim_world_size_{self.world_size}_rank_{self.rank}.pt') + remote_extra_state_path = os.path.join(path, f'extra_state_world_size_{self.world_size}_rank_{self.rank}.pt') + print( + f'[rank-{self.rank}]: Loading from {remote_model_path} and {remote_optim_path} and {remote_extra_state_path}' + ) + local_model_path = copy_local_path_from_hdfs(remote_model_path) + local_optim_path = copy_local_path_from_hdfs(remote_optim_path) + local_extra_state_path = copy_local_path_from_hdfs(remote_extra_state_path) + + model_state_dict = torch.load(local_model_path) + optimizer_state_dict = torch.load(local_optim_path) + extra_state_dict = torch.load(local_extra_state_path) + + if del_local_after_load: + try: + os.remove(local_model_path) if is_non_local(local_model_path) else None + os.remove(local_optim_path) if is_non_local(local_optim_path) else None + os.remove(local_extra_state_path) if is_non_local(local_extra_state_path) else None + except Exception as e: + print( + f'[rank-{self.rank}]: remove local resume ckpt file after loading failed, exception {e} will be ignored' + ) + + lr_scheduler_state_dict = extra_state_dict['lr_scheduler'] + + state_dict_cfg = ShardedStateDictConfig(offload_to_cpu=True) + optim_cfg = ShardedOptimStateDictConfig(offload_to_cpu=True) + with FSDP.state_dict_type(self.model, StateDictType.SHARDED_STATE_DICT, state_dict_cfg, optim_cfg): + self.model.load_state_dict(model_state_dict) + if self.optimizer is not None: + self.optimizer.load_state_dict(optimizer_state_dict) + # recover random state + if 'rng' in extra_state_dict: + # 'rng' may not exist for backward compatibility + self.load_rng_state(extra_state_dict['rng']) + + if self.lr_scheduler is not None: + self.lr_scheduler.load_state_dict(lr_scheduler_state_dict) + + def save_checkpoint(self, local_path: str, global_step: int, remove_previous_ckpt=False, *args, **kwargs): + # record the previous global step + self.previous_global_step = global_step + + # remove previous local_path + # TODO: shall we remove previous ckpt every save? + if remove_previous_ckpt: + self.remove_previous_save_local_path() + local_path = self.local_mkdir(local_path) + torch.distributed.barrier() + + # every rank will save its own model and optim shard + state_dict_cfg = ShardedStateDictConfig(offload_to_cpu=True) + optim_cfg = ShardedOptimStateDictConfig(offload_to_cpu=True) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with FSDP.state_dict_type(self.model, StateDictType.SHARDED_STATE_DICT, state_dict_cfg, optim_cfg): + model_state_dict = self.model.state_dict() + if self.optimizer is not None: + optimizer_state_dict = self.optimizer.state_dict() + else: + optimizer_state_dict = None + if self.lr_scheduler is not None: + lr_scheduler_state_dict = self.lr_scheduler.state_dict() + else: + lr_scheduler_state_dict = None + + extra_state_dict = { + 'lr_scheduler': lr_scheduler_state_dict, + 'rng': self.get_rng_state(), + } + model_path = os.path.join(local_path, f'model_world_size_{self.world_size}_rank_{self.rank}.pt') + optim_path = os.path.join(local_path, f'optim_world_size_{self.world_size}_rank_{self.rank}.pt') + extra_path = os.path.join(local_path, f'extra_state_world_size_{self.world_size}_rank_{self.rank}.pt') + + print(f'[rank-{self.rank}]: Saving model to {os.path.abspath(model_path)}') + print(f'[rank-{self.rank}]: Saving checkpoint to {os.path.abspath(model_path)}') + print(f'[rank-{self.rank}]: Saving extra_state to {os.path.abspath(extra_path)}') + torch.save(model_state_dict, model_path) + torch.save(optimizer_state_dict, optim_path) # TODO: address optimizer is None + torch.save(extra_state_dict, extra_path) + + # wait for everyone to dump to local + torch.distributed.barrier() + + if self.rank == 0: + hf_local_path = os.path.join(local_path, 'huggingface') + os.makedirs(hf_local_path, exist_ok=True) + self.model._fsdp_wrapped_module.config.save_pretrained(hf_local_path) + self.tokenizer.save_pretrained(hf_local_path) + + torch.distributed.barrier() + + self.previous_save_local_path = local_path diff --git a/frameworks/agentgym_rl/verl/utils/config.py b/frameworks/agentgym_rl/verl/utils/config.py new file mode 100644 index 0000000000000000000000000000000000000000..5c9298c42adf89467d047a3d0fdf8919bf772a5a --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/config.py @@ -0,0 +1,23 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Dict + +from omegaconf import DictConfig + + +def update_dict_with_config(dictionary: Dict, config: DictConfig): + for key in dictionary: + if hasattr(config, key): + dictionary[key] = getattr(config, key) diff --git a/frameworks/agentgym_rl/verl/utils/debug/__init__.py b/frameworks/agentgym_rl/verl/utils/debug/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0d0b3432eb4d6200ed84da0f735afa46735ef58e --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/debug/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .performance import log_gpu_memory_usage \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/debug/performance.py b/frameworks/agentgym_rl/verl/utils/debug/performance.py new file mode 100644 index 0000000000000000000000000000000000000000..615475a66a5e45853540df2efd09c25991b43e12 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/debug/performance.py @@ -0,0 +1,30 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +import torch.distributed as dist +import logging + + +def log_gpu_memory_usage(head: str, logger: logging.Logger = None, level=logging.DEBUG, rank: int = 0): + if (not dist.is_initialized()) or (rank is None) or (dist.get_rank() == rank): + memory_allocated = torch.cuda.memory_allocated() / 1024**3 + memory_reserved = torch.cuda.memory_reserved() / 1024**3 + + message = f'{head}, memory allocated (GB): {memory_allocated}, memory reserved (GB): {memory_reserved}' + + if logger is None: + print(message) + else: + logger.log(msg=message, level=level) diff --git a/frameworks/agentgym_rl/verl/utils/debug/trajectory_tracker.py b/frameworks/agentgym_rl/verl/utils/debug/trajectory_tracker.py new file mode 100644 index 0000000000000000000000000000000000000000..33b254685221a86b03f120b57659cd55b29ea0a2 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/debug/trajectory_tracker.py @@ -0,0 +1,108 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Trajectory tracker can be inserted into code to save the intermediate results. +The results will be dump to hdfs for offline comparison. +Each process will have a client that first move all the tensors to CPU +""" + +from verl.utils.hdfs_io import makedirs, copy +import torch +import os +import ray +import io +import tempfile + +from collections import deque + +remote_copy = ray.remote(copy) + + +@ray.remote +def save_to_hdfs(data: io.BytesIO, name, hdfs_dir, verbose): + filename = name + '.pth' + with tempfile.TemporaryDirectory() as tmpdirname: + local_filepath = os.path.join(tmpdirname, filename) + with open(local_filepath, 'wb') as f: + f.write(data.getbuffer()) + # upload to hdfs + + if verbose: + print(f'Saving {local_filepath} to {hdfs_dir}') + try: + copy(local_filepath, hdfs_dir) + except Exception as e: + print(e) + + +@ray.remote +class TrajectoryTracker(): + + def __init__(self, hdfs_dir, verbose) -> None: + self.hdfs_dir = hdfs_dir + makedirs(hdfs_dir) + self.verbose = verbose + + self.handle = deque() + + def dump(self, data: io.BytesIO, name): + # get a temp file and write to it + self.handle.append(save_to_hdfs.remote(data, name, self.hdfs_dir, self.verbose)) + + def wait_for_hdfs(self): + while len(self.handle) != 0: + future = self.handle.popleft() + ray.get(future) + + +def dump_data(data, name): + enable = os.getenv('VERL_ENABLE_TRACKER', '0') == '1' + if not enable: + return + buffer = io.BytesIO() + torch.save(data, buffer) + tracker = get_trajectory_tracker() + ray.get(tracker.dump.remote(buffer, name)) + + +def get_trajectory_tracker(): + hdfs_dir = os.getenv('VERL_TRACKER_HDFS_DIR', default=None) + verbose = os.getenv('VERL_TRACKER_VERBOSE', default='0') == '1' + assert hdfs_dir is not None + tracker = TrajectoryTracker.options(name="global_tracker", get_if_exists=True, + lifetime="detached").remote(hdfs_dir, verbose) + return tracker + + +if __name__ == '__main__': + # testing + os.environ['VERL_ENABLE_TRACKER'] = '1' + os.environ['VERL_TRACKER_HDFS_DIR'] = '~/debug/test' + + @ray.remote + def process(iter): + data = {'obs': torch.randn(10, 20)} + dump_data(data, f'process_{iter}_obs') + + ray.init() + + output_lst = [] + + for i in range(10): + output_lst.append(process.remote(i)) + + out = ray.get(output_lst) + + tracker = get_trajectory_tracker() + ray.get(tracker.wait_for_hdfs.remote()) diff --git a/frameworks/agentgym_rl/verl/utils/distributed.py b/frameworks/agentgym_rl/verl/utils/distributed.py new file mode 100644 index 0000000000000000000000000000000000000000..6fea5a29cd943ef91c8f27f44db2a69e40702cf7 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/distributed.py @@ -0,0 +1,28 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Utilities for distributed training.""" +import os + + +def initialize_global_process_group(timeout_second=36000): + import torch.distributed + from datetime import timedelta + torch.distributed.init_process_group('nccl', timeout=timedelta(seconds=timeout_second)) + local_rank = int(os.environ["LOCAL_RANK"]) + rank = int(os.environ["RANK"]) + world_size = int(os.environ["WORLD_SIZE"]) + + if torch.distributed.is_initialized(): + torch.cuda.set_device(local_rank) + return local_rank, rank, world_size diff --git a/frameworks/agentgym_rl/verl/utils/flops_counter.py b/frameworks/agentgym_rl/verl/utils/flops_counter.py new file mode 100644 index 0000000000000000000000000000000000000000..3c5ac1a91160fc3265589fb6e93e93c8c1efb53e --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/flops_counter.py @@ -0,0 +1,123 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +from transformers import PretrainedConfig, Qwen2Config, LlamaConfig + +VALID_CONFIG_TYPE = (Qwen2Config, LlamaConfig) + + +def get_device_flops(unit="T"): + + def unit_convert(number, level): + units = ["B", "K", "M", "G", "T", "P"] + if number <= 0: + return number + ptr = 0 + while ptr < len(units) and units[ptr] != level: + number /= 1000 + ptr += 1 + return number + + device_name = torch.cuda.get_device_name() + flops = float("inf") # INF flops for unkown gpu type + if "H100" in device_name or "H800" in device_name: + flops = 989e12 + elif "A100" in device_name or "A800" in device_name: + flops = 312e12 + elif "L40" in device_name: + flops = 181.05e12 + elif "L20" in device_name: + flops = 119.5e12 + elif "H20" in device_name: + flops = 148e12 + elif "910B" in device_name: + flops = 354e12 + flops_unit = unit_convert(flops, unit) + return flops_unit + + +class FlopsCounter: + """ + Used to count mfu during training loop + + Example: + flops_counter = FlopsCounter(config) + flops_achieved, flops_promised = flops_counter.estimate_flops(tokens_list, delta_time) + + """ + + def __init__(self, config: PretrainedConfig): + if not isinstance(config, VALID_CONFIG_TYPE): + print(f"Only support config type of {VALID_CONFIG_TYPE}, but got {type(config)}. " + f"MFU will always be zero.") + + self.estimate_func = {"qwen2": self._estimate_qwen2_flops, 'llama': self._estimate_qwen2_flops} + self.config = config + + def _estimate_unknown_flops(self, tokens_sum, batch_seqlens, delta_time): + return 0 + + def _estimate_qwen2_flops(self, tokens_sum, batch_seqlens, delta_time): + assert isinstance(self.config, (Qwen2Config, LlamaConfig)) + hidden_size = self.config.hidden_size + vocab_size = self.config.vocab_size + num_hidden_layers = self.config.num_hidden_layers + num_key_value_heads = self.config.num_key_value_heads + num_attention_heads = self.config.num_attention_heads + intermediate_size = self.config.intermediate_size + + head_dim = hidden_size // num_attention_heads + q_size = num_attention_heads * head_dim + k_size = num_key_value_heads * head_dim + v_size = num_key_value_heads * head_dim + + # non-attn per layer parm + # Qwen2/LLama use SwiGelu, gate, having up and down linear layer in mlp + mlp_N = hidden_size * intermediate_size * 3 + attn_linear_N = hidden_size * (q_size + k_size + v_size + num_attention_heads * head_dim) + emd_and_lm_head_N = vocab_size * hidden_size * 2 + # non-attn all_layer parm + dense_N = (mlp_N + attn_linear_N) * num_hidden_layers + emd_and_lm_head_N + # non-attn all_layer & all_token fwd & bwd flops + dense_N_flops = 6 * dense_N * tokens_sum + + # attn all_layer & all_token fwd & bwd flops + seqlen_square_sum = 0 + for seqlen in batch_seqlens: + seqlen_square_sum += seqlen * seqlen + attn_qkv_flops = 12 * seqlen_square_sum * head_dim * num_attention_heads * num_hidden_layers + + # all_layer & all_token fwd & bwd flops + flops_all_token = dense_N_flops + attn_qkv_flops + flops_achieved = flops_all_token * (1.0 / delta_time) / 1e12 + return flops_achieved + + def estimate_flops(self, batch_seqlens, delta_time): + """ + Estimate the FLOPS based on the number of valid tokens in the current batch and the time taken. + + Args: + batch_seqlens (List[int]): A list where each element represents the number of valid tokens in the current batch. + delta_time (float): The time taken to process the batch, in seconds. + + Returns: + estimated_flops (float): The estimated FLOPS based on the input tokens and time. + promised_flops (float): The expected FLOPS of the current device. + """ + tokens_sum = sum(batch_seqlens) + func = self.estimate_func.get(self.config.model_type, self._estimate_unknown_flops) + estimated_flops = func(tokens_sum, batch_seqlens, delta_time) + promised_flops = get_device_flops() + return estimated_flops, promised_flops diff --git a/frameworks/agentgym_rl/verl/utils/fs.py b/frameworks/agentgym_rl/verl/utils/fs.py new file mode 100644 index 0000000000000000000000000000000000000000..ce11905fb816b4c6a4547ac6d03382efab05bcdb --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/fs.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# -*- coding: utf-8 -*- +"""File-system agnostic IO APIs""" +import os +import tempfile +import hashlib + +try: + from hdfs_io import copy, makedirs, exists # for internal use only +except ImportError: + from .hdfs_io import copy, makedirs, exists + +__all__ = ["copy", "exists", "makedirs"] + +_HDFS_PREFIX = "hdfs://" + + +def is_non_local(path): + return path.startswith(_HDFS_PREFIX) + + +def md5_encode(path: str) -> str: + return hashlib.md5(path.encode()).hexdigest() + + +def get_local_temp_path(hdfs_path: str, cache_dir: str) -> str: + """Return a local temp path that joins cache_dir and basename of hdfs_path + + Args: + hdfs_path: + cache_dir: + + Returns: + + """ + # make a base64 encoding of hdfs_path to avoid directory conflict + encoded_hdfs_path = md5_encode(hdfs_path) + temp_dir = os.path.join(cache_dir, encoded_hdfs_path) + os.makedirs(temp_dir, exist_ok=True) + dst = os.path.join(temp_dir, os.path.basename(hdfs_path)) + return dst + + +def copy_local_path_from_hdfs(src: str, cache_dir=None, filelock='.file.lock', verbose=False) -> str: + """Copy src from hdfs to local if src is on hdfs or directly return src. + If cache_dir is None, we will use the default cache dir of the system. Note that this may cause conflicts if + the src name is the same between calls + + Args: + src (str): a HDFS path of a local path + + Returns: + a local path of the copied file + """ + from filelock import FileLock + + assert src[-1] != '/', f'Make sure the last char in src is not / because it will cause error. Got {src}' + + if is_non_local(src): + # download from hdfs to local + if cache_dir is None: + # get a temp folder + cache_dir = tempfile.gettempdir() + os.makedirs(cache_dir, exist_ok=True) + assert os.path.exists(cache_dir) + local_path = get_local_temp_path(src, cache_dir) + # get a specific lock + filelock = md5_encode(src) + '.lock' + lock_file = os.path.join(cache_dir, filelock) + with FileLock(lock_file=lock_file): + if not os.path.exists(local_path): + if verbose: + print(f'Copy from {src} to {local_path}') + copy(src, local_path) + return local_path + else: + return src diff --git a/frameworks/agentgym_rl/verl/utils/fsdp_utils.py b/frameworks/agentgym_rl/verl/utils/fsdp_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..90c534aac0388059968f222e056816018bf323af --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/fsdp_utils.py @@ -0,0 +1,329 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Dict +import functools +import json +import math +import itertools +import os +from contextlib import contextmanager +from torch.distributed.fsdp.wrap import size_based_auto_wrap_policy, transformer_auto_wrap_policy +from transformers.trainer_pt_utils import get_module_class_from_name +import torch +import torch.nn as nn +import torch.distributed as dist + + +def init_fn(x: torch.nn.Module): + if not torch.distributed.get_rank() == 0: + x = x.to_empty(device=torch.cuda.current_device(), recurse=False) + torch.cuda.empty_cache() + return x + + +def get_init_weight_context_manager(use_meta_tensor=True): + from accelerate import init_empty_weights + cpu_init_weights = lambda: torch.device('cpu') + if use_meta_tensor: + init_context = init_empty_weights if torch.distributed.get_rank() != 0 else cpu_init_weights + else: + init_context = cpu_init_weights + return init_context + + +# Copyright 2020-present the HuggingFace Inc. team. +# Adapted from https://github.com/huggingface/transformers/src/transformers/trainer.py +def get_fsdp_wrap_policy(module, config=None, is_lora=False): + """Get FSDP wrap policy for the module. + + Args: + module: The module to get wrap policy for + config: Configuration for wrap policy + is_lora: Whether to enable lambda policy for LoRA modules + """ + if config is None: + config = {} + + if config.get('disable', False): + return None + + default_transformer_cls_names_to_wrap = getattr(module, "_no_split_modules", None) + fsdp_transformer_layer_cls_to_wrap = config.get("transformer_layer_cls_to_wrap", + default_transformer_cls_names_to_wrap) + min_num_params = config.get('min_num_params', 0) + auto_wrap_policy = None + + policies = [] + + from torch.distributed.fsdp.wrap import _or_policy, lambda_auto_wrap_policy, transformer_auto_wrap_policy + + # Add lambda policy for LoRA modules if is_lora is True + if is_lora: + + def lambda_policy_fn(module): + if (len(list(module.named_children())) == 0 and getattr(module, "weight", None) is not None and + module.weight.requires_grad): + return True + return False + + lambda_policy = functools.partial(lambda_auto_wrap_policy, lambda_fn=lambda_policy_fn) + policies.append(lambda_policy) + + if min_num_params > 0: + size_policy = functools.partial(size_based_auto_wrap_policy, min_num_params=min_num_params) + policies.append(size_policy) + elif fsdp_transformer_layer_cls_to_wrap is not None: + transformer_cls_to_wrap = set() + for layer_class in fsdp_transformer_layer_cls_to_wrap: + transformer_cls = get_module_class_from_name(module, layer_class) + if transformer_cls is None: + raise Exception("Could not find the transformer layer class to wrap in the model.") + else: + transformer_cls_to_wrap.add(transformer_cls) + + transformer_policy = functools.partial( + transformer_auto_wrap_policy, + transformer_layer_cls=transformer_cls_to_wrap, + ) + policies.append(transformer_policy) + + if len(policies) > 0: + auto_wrap_policy = functools.partial(_or_policy, policies=policies) + + return auto_wrap_policy + + +def offload_fsdp_grad(module): + for _, param in module.named_parameters(): + if param.grad is not None: + param.grad = param.grad.to("cpu", non_blocking=True) + torch.cuda.empty_cache() + + +def load_fsdp_grad(module, device_id): + for _, param in module.named_parameters(): + if param.grad is not None: + param.grad = param.grad.to(device_id, non_blocking=True) + torch.cuda.empty_cache() + + +def offload_fsdp_param_and_grad(module, offload_grad=False): + for _, param in module.named_parameters(): + if hasattr(param, "_local_shard"): + param._local_shard = param._local_shard.to("cpu", non_blocking=True) + param.data = param.data.to('cpu', non_blocking=True) + if offload_grad and param.grad is not None: + param.grad = param.grad.to("cpu", non_blocking=True) + torch.cuda.empty_cache() + + +def load_fsdp_param_and_grad(module, device_id, load_grad=False): + for _, param in module.named_parameters(): + if hasattr(param, "_local_shard"): + param._local_shard = param._local_shard.to(device_id, non_blocking=True) + param.data = param.data.to(device_id, non_blocking=True) + if load_grad and param.grad is not None: + param.grad = param.grad.to(device_id, non_blocking=True) + torch.cuda.empty_cache() + + +def offload_fsdp_optimizer(optimizer): + for param_group in optimizer.param_groups: + for param in param_group['params']: + state = optimizer.state[param] + for key, value in state.items(): + if isinstance(value, torch.Tensor): + state[key] = value.to("cpu", non_blocking=True) + torch.cuda.empty_cache() + + +def load_fsdp_optimizer(optimizer, device_id): + for param_group in optimizer.param_groups: + for param in param_group['params']: + state = optimizer.state[param] + for key, value in state.items(): + if isinstance(value, torch.Tensor): + state[key] = value.to(device_id, non_blocking=True) + torch.cuda.empty_cache() + + +@contextmanager +def meta_device_init(): + """ + Create model parameters with meta device. + + Note buffers in model will still be initialized in default device (e.g., CPU), + since the buffers can be non-persistent and filled with expected values that can + NOT be captured in meta device. + """ + device = torch.device("meta") + old_register_parameter = nn.Module.register_parameter + registered = set() + + def register_empty_parameter(module, name, param): + old_register_parameter(module, name, param) + # we will skip register shared parameters as it + # is already registered previously + if param is not None and param not in registered: + param_cls = type(module._parameters[name]) + kwargs = module._parameters[name].__dict__ + kwargs["requires_grad"] = param.requires_grad + module._parameters[name] = param_cls(module._parameters[name].to(device), **kwargs) + registered.add(module._parameters[name]) + + try: + nn.Module.register_parameter = register_empty_parameter + yield + finally: + registered.clear() + nn.Module.register_parameter = old_register_parameter + + +def parallel_load_safetensors(filepath): + """ + Parallel load safetensors from huggingface checkpoint + + Huggingface checkpoint contains: + + - config.json: a json file for model configuration + - model.safetensor.index.json: a json file for safetensors (parameters & buffers) index + - model-000x-of-ooxx.safetensors: a binary file for safetensors (parameters & buffers) chunks + + Or (when model is small), + + - model.safetensors: a binary file for all parameters and buffers + + Each rank will own a part of model chunks and load them directly into GPU memory. + """ + from safetensors.torch import load_file + + safetensors2param = {} + + index_file = os.path.join(filepath, "model.safetensors.index.json") + if os.path.exists(index_file): + index = json.load(open(index_file, "rb")) + for param_name, filename in index["weight_map"].items(): + safetensors2param.setdefault(filename, []).append(param_name) + else: + # in this case, the model is small and we can load it all at once + param_file = os.path.join(filepath, "model.safetensors") + assert os.path.exists(param_file), f"Cannot find {param_file}" + states = load_file(param_file) + for param_name in states: + safetensors2param.setdefault("model.safetensors", []).append(param_name) + del states + + total_files = len(safetensors2param) + ckpt_chunks = sorted(safetensors2param.keys()) + world_size = dist.get_world_size() + size = int(math.ceil(total_files / world_size)) + ckpt_chunks = [ckpt_chunks[rank * size:rank * size + size] for rank in range(world_size)] + + shard_states = {} + device = torch.cuda.current_device() + for rank, files in enumerate(ckpt_chunks): + if rank == dist.get_rank(): + for file in files: + file = os.path.join(filepath, file) + states = load_file(file, device=device) + # print(f"rank {rank} loading {file}...") + shard_states.update(states) + else: + for file in files: + for param_name in safetensors2param[file]: + shard_states[param_name] = rank + return shard_states + + +def parallel_init_module_fn(module: torch.nn.Module, shard_states: Dict[str, torch.nn.Parameter]): + """ + Generate a function to initialize sub-modules in the `module` with `shard_states` + from huggingface checkpoint. + + Args: + module (torch.nn.Module): the global module to be initialized + shard_states (Dict[str, torch.nn.Parameter]): the shard states from huggingface checkpoint + + Returns: + init_fn (Callable): a function to initialize sub-modules in the `module` with `shard_states` + """ + + state2fqn = {} + for name, state in itertools.chain(module.named_parameters(remove_duplicate=False), + module.named_buffers(remove_duplicate=False)): + state2fqn.setdefault(state, []).append(name) + # remove standalone parameters and buffers + shared = {s for s, names in state2fqn.items() if len(names) > 1} + materialized_states = {} + + @torch.no_grad() + def create_and_sync_state(param_name, state, is_param): + assert param_name in shard_states, f"{param_name} not loaded" + device = torch.cuda.current_device() + if is_param: + param = torch.nn.Parameter(torch.empty_like(state.data, device=device), requires_grad=state.requires_grad) + else: # buffer + param = torch.empty_like(state.data, device=device) + loaded = shard_states[param_name] + if isinstance(loaded, (torch.nn.Parameter, torch.Tensor)): + # NOTE: loaded.dtype can be different with param.dtype + param.data.copy_(loaded.data) + dist.broadcast(param.data, src=dist.get_rank()) + else: + assert isinstance(loaded, int) # the rank that holds the state + dist.broadcast(param.data, src=loaded) + shard_states.pop(param_name) + del loaded + return param + + def init_fn(sub_mod: torch.nn.Module, recurse: bool = True): + param_and_buffers = tuple(sub_mod.named_parameters(recurse=False)) + tuple(sub_mod.named_buffers(recurse=False)) + # param_and_buffers = sorted(sub_mod.named_parameters(recurse=False), key=lambda x: x[0]) + for name, state in param_and_buffers: + if not state.is_meta: + continue + is_param = name in sub_mod._parameters + fqn = state2fqn[state].pop(0) + # non-persistent buffers will not be saved in state dict, we can safely skip it + if (not is_param) and fqn not in shard_states: + if state.is_meta: + raise RuntimeError( + f"find a non-persistent buffer ({fqn}) initiated with device meta. " + "Such buffer is not saved in checkpoint and user should guarantee to init in CPU / GPU device.") + continue + # for shared parameter, we get it from the first time it is created + if state in shared: + if state not in materialized_states: + materialized_states[state] = create_and_sync_state(fqn, state, is_param) + else: + if fqn in shard_states: + shard_states.pop(fqn) + materialize_state = materialized_states[state] + # for not shared parameter, we create it directly + else: + materialize_state = create_and_sync_state(fqn, state, is_param) + if is_param: + sub_mod._parameters[name] = materialize_state + else: + sub_mod._buffers[name] = materialize_state + if recurse: + for module in sub_mod.children(): + init_fn(module, recurse=True) + + # for debug + # if len(shard_states) == 0: print("clear") + return sub_mod + + return init_fn diff --git a/frameworks/agentgym_rl/verl/utils/hdfs_io.py b/frameworks/agentgym_rl/verl/utils/hdfs_io.py new file mode 100644 index 0000000000000000000000000000000000000000..08c4ecb9a5956865ce35651d6eaaf6844ba87f41 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/hdfs_io.py @@ -0,0 +1,144 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import shutil +import logging + +logger = logging.getLogger(__file__) +logger.setLevel(os.getenv('VERL_SFT_LOGGING_LEVEL', 'WARN')) + +_HDFS_PREFIX = "hdfs://" + +_HDFS_BIN_PATH = shutil.which('hdfs') + + +def exists(path: str, **kwargs) -> bool: + r"""Works like os.path.exists() but supports hdfs. + + Test whether a path exists. Returns False for broken symbolic links. + + Args: + path (str): path to test + + Returns: + bool: True if the path exists, False otherwise + """ + if _is_non_local(path): + return _exists(path, **kwargs) + return os.path.exists(path) + + +def _exists(file_path: str): + """ hdfs capable to check whether a file_path is exists """ + if file_path.startswith("hdfs"): + return _run_cmd(_hdfs_cmd(f"-test -e {file_path}")) == 0 + return os.path.exists(file_path) + + +def makedirs(name, mode=0o777, exist_ok=False, **kwargs) -> None: + r"""Works like os.makedirs() but supports hdfs. + + Super-mkdir; create a leaf directory and all intermediate ones. Works like + mkdir, except that any intermediate path segment (not just the rightmost) + will be created if it does not exist. If the target directory already + exists, raise an OSError if exist_ok is False. Otherwise no exception is + raised. This is recursive. + + Args: + name (str): directory to create + mode (int): file mode bits + exist_ok (bool): if True, do not raise an exception if the directory already exists + kwargs: keyword arguments for hdfs + + """ + if _is_non_local(name): + # TODO(haibin.lin): + # - handle OSError for hdfs(?) + # - support exist_ok for hdfs(?) + _mkdir(name, **kwargs) + else: + os.makedirs(name, mode=mode, exist_ok=exist_ok) + + +def _mkdir(file_path: str) -> bool: + """hdfs mkdir""" + if file_path.startswith("hdfs"): + _run_cmd(_hdfs_cmd(f"-mkdir -p {file_path}")) + else: + os.makedirs(file_path, exist_ok=True) + return True + + +def copy(src: str, dst: str, **kwargs) -> bool: + r"""Works like shutil.copy() for file, and shutil.copytree for dir, and supports hdfs. + + Copy data and mode bits ("cp src dst"). Return the file's destination. + The destination may be a directory. + If source and destination are the same file, a SameFileError will be + raised. + + Arg: + src (str): source file path + dst (str): destination file path + kwargs: keyword arguments for hdfs copy + + Returns: + str: destination file path + + """ + if _is_non_local(src) or _is_non_local(dst): + # TODO(haibin.lin): + # - handle SameFileError for hdfs files(?) + # - return file destination for hdfs files + return _copy(src, dst) + else: + if os.path.isdir(src): + return shutil.copytree(src, dst, **kwargs) + else: + return shutil.copy(src, dst, **kwargs) + + +def _copy(from_path: str, to_path: str, timeout: int = None) -> bool: + if to_path.startswith("hdfs"): + if from_path.startswith("hdfs"): + returncode = _run_cmd(_hdfs_cmd(f"-cp -f {from_path} {to_path}"), timeout=timeout) + else: + returncode = _run_cmd(_hdfs_cmd(f"-put -f {from_path} {to_path}"), timeout=timeout) + else: + if from_path.startswith("hdfs"): + returncode = _run_cmd(_hdfs_cmd(f"-get \ + {from_path} {to_path}"), timeout=timeout) + else: + try: + shutil.copy(from_path, to_path) + returncode = 0 + except shutil.SameFileError: + returncode = 0 + except Exception as e: + logger.warning(f"copy {from_path} {to_path} failed: {e}") + returncode = -1 + return returncode == 0 + + +def _run_cmd(cmd: str, timeout=None): + return os.system(cmd) + + +def _hdfs_cmd(cmd: str) -> str: + return f"{_HDFS_BIN_PATH} dfs {cmd}" + + +def _is_non_local(path: str): + return path.startswith(_HDFS_PREFIX) diff --git a/frameworks/agentgym_rl/verl/utils/import_utils.py b/frameworks/agentgym_rl/verl/utils/import_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..e5690512d144a30d2a1f0bd128a40eb8876936b7 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/import_utils.py @@ -0,0 +1,48 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Utilities to check if packages are available. +We assume package availability won't change during runtime. +""" + +from functools import cache +from typing import List + + +@cache +def is_megatron_core_available(): + try: + from megatron.core import parallel_state as mpu + return True + except ImportError: + return False + + +@cache +def is_vllm_available(): + try: + import vllm + return True + except ImportError: + return False + + +def import_external_libs(external_libs=None): + if external_libs is None: + return + if not isinstance(external_libs, List): + external_libs = [external_libs] + import importlib + for external_lib in external_libs: + importlib.import_module(external_lib) diff --git a/frameworks/agentgym_rl/verl/utils/logger/__init__.py b/frameworks/agentgym_rl/verl/utils/logger/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/logger/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/utils/logger/aggregate_logger.py b/frameworks/agentgym_rl/verl/utils/logger/aggregate_logger.py new file mode 100644 index 0000000000000000000000000000000000000000..ac57cf58e3de2862b5443189ccec276a7d2fc283 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/logger/aggregate_logger.py @@ -0,0 +1,42 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +A Ray logger will receive logging info from different processes. +""" +import numbers +from typing import Dict + + +def concat_dict_to_str(dict: Dict, step): + output = [f'step:{step}'] + for k, v in dict.items(): + if isinstance(v, numbers.Number): + output.append(f'{k}:{v:.3f}') + output_str = ' - '.join(output) + return output_str + + +class LocalLogger: + + def __init__(self, remote_logger=None, enable_wandb=False, print_to_console=False): + self.print_to_console = print_to_console + if print_to_console: + print('Using LocalLogger is deprecated. The constructor API will change ') + + def flush(self): + pass + + def log(self, data, step): + if self.print_to_console: + print(concat_dict_to_str(data, step=step), flush=True) \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/logging_utils.py b/frameworks/agentgym_rl/verl/utils/logging_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..1bf6e1f0fa70784edb6a7e6efecdba07f0c399b3 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/logging_utils.py @@ -0,0 +1,22 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging + + +def set_basic_config(level): + """ + This function sets the global logging format and level. It will be called when import verl + """ + logging.basicConfig(format='%(levelname)s:%(asctime)s:%(message)s', level=level) diff --git a/frameworks/agentgym_rl/verl/utils/megatron/__init__.py b/frameworks/agentgym_rl/verl/utils/megatron/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/megatron/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/utils/megatron/memory.py b/frameworks/agentgym_rl/verl/utils/megatron/memory.py new file mode 100644 index 0000000000000000000000000000000000000000..5e8570ed495d83e74a1d0c7b1d17181271ce92a6 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/megatron/memory.py @@ -0,0 +1,41 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch + + +class MemoryBuffer: + + def __init__(self, numel, numel_padded, dtype): + self.numel = numel + self.numel_padded = numel_padded + self.dtype = dtype + self.data = torch.zeros(self.numel_padded, + dtype=self.dtype, + device=torch.cuda.current_device(), + requires_grad=False) + + def zero(self): + """Reset the buffer to zero.""" + self.data.zero_() + + def get(self, shape, start_index): + """Return a tensor with the input `shape` as a view into the + 1-D data starting at `start_index`.""" + end_index = start_index + shape.numel() + assert end_index <= self.numel, \ + 'requested tensor is out of the buffer range.' + buffer_tensor = self.data[start_index:end_index] + buffer_tensor = buffer_tensor.view(shape) + return buffer_tensor diff --git a/frameworks/agentgym_rl/verl/utils/megatron/optimizer.py b/frameworks/agentgym_rl/verl/utils/megatron/optimizer.py new file mode 100644 index 0000000000000000000000000000000000000000..9ae70b0876d5255ffd24e132ca4d60faab883582 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/megatron/optimizer.py @@ -0,0 +1,92 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from apex.optimizers import FusedAdam as Adam +from apex.optimizers import FusedSGD as SGD +from megatron.optimizer.distrib_optimizer import DistributedOptimizer +from megatron.optimizer.grad_scaler import ConstantGradScaler, DynamicGradScaler +from megatron.optimizer import Float16OptimizerWithFloat16Params, FP32Optimizer +from megatron.optimizer import get_param_groups + +from verl.utils.megatron.optimizer_config import OptimizerConfig + + +def get_megatron_optimizer( + model, + config: OptimizerConfig, + no_weight_decay_cond=None, + scale_lr_cond=None, + lr_mult=1.0, + check_for_nan_in_loss_and_grad=False, + overlap_param_gather=False # add for verl +): + # Base optimizer. + param_groups = get_param_groups(model, no_weight_decay_cond, scale_lr_cond, lr_mult) + + if config.optimizer == 'adam': + optimizer = Adam(param_groups, + lr=config.lr, + weight_decay=config.weight_decay, + betas=(config.adam_beta1, config.adam_beta2), + eps=config.adam_eps) + elif config.optimizer == 'sgd': + optimizer = SGD(param_groups, lr=config.lr, weight_decay=config.weight_decay, momentum=config.sgd_momentum) + else: + raise Exception('{} optimizer is not supported.'.format(config.optimizer)) + + # Determine whether the params have main-grad field. + params_have_main_grad = True + + # Mixed precision optimizer. + # - Note: both the Float16Optimizer and the DistributedOptimizer inherit + # from the MixedPrecisionOptimizer, which manages any optimizer where + # the model params and main params are distinct. + if config.fp16 or config.bf16 or config.use_distributed_optimizer: + + # Grad scaler: + # if loss-scale is provided, instantiate the constant scaler. + # if we are using fp16 and loss-scale is not present, use a + # dynamic scaler. + # otherwise we are running in bf16 with no loss-scale so + # leave it as None. + grad_scaler = None + + # Constant loss scale. + if config.loss_scale: + grad_scaler = ConstantGradScaler(config.loss_scale) + + # Dynamic loss scale. + else: + if config.fp16: + grad_scaler = DynamicGradScaler(initial_scale=config.initial_loss_scale, + min_scale=config.min_loss_scale, + growth_factor=2.0, + backoff_factor=0.5, + growth_interval=config.loss_scale_window, + hysteresis=config.hysteresis) + + # Megatron optimizer. + if config.use_distributed_optimizer: + return DistributedOptimizer(optimizer, config.clip_grad, config.log_num_zeros_in_grad, + check_for_nan_in_loss_and_grad, params_have_main_grad, config.fp16, config.bf16, + config.params_dtype, grad_scaler, model, overlap_param_gather) + else: + return Float16OptimizerWithFloat16Params(optimizer, config.clip_grad, config.log_num_zeros_in_grad, + check_for_nan_in_loss_and_grad, params_have_main_grad, config.fp16, + config.bf16, config.params_dtype, grad_scaler, model) + + # FP32. + return FP32Optimizer(optimizer, config.clip_grad, config.log_num_zeros_in_grad, check_for_nan_in_loss_and_grad, + params_have_main_grad, model) diff --git a/frameworks/agentgym_rl/verl/utils/megatron/optimizer_config.py b/frameworks/agentgym_rl/verl/utils/megatron/optimizer_config.py new file mode 100644 index 0000000000000000000000000000000000000000..3401de4163aadcad7f3a586587da38989ee19d3d --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/megatron/optimizer_config.py @@ -0,0 +1,129 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from dataclasses import dataclass +from typing import Callable, Optional + +import torch + + +@dataclass +class OptimizerConfig: + """Configuration for optimizer.""" + + ############## + # General + ############## + optimizer: str = 'adam' + """Optimizer to use (one of Adam or SGD).""" + + lr: Optional[float] = None + """Initial learning rate. Depending on decay style and initial warmup, the learning rate at each + iteration would be different. + """ + + min_lr: Optional[float] = None + """Minumum value for learning rate. The scheduler clip values below this threshold.""" + + decoupled_lr: Optional[float] = None + """Separate learning rate for the input and output layer.""" + + decoupled_min_lr: Optional[float] = None + """Minimum value for learning rate for the input and output layer. The scheduler clip values + below this threshold. + """ + + weight_decay: float = 0.01 + """Weight decay coefficient for L2 regularization.""" + + ############## + # Precision + ############## + fp16: bool = False + """If true, train with fp16 mixed precision training. Defaults to False.""" + + bf16: bool = False + """If true, train with bf16 mixed precision training. Defaults to False.""" + + params_dtype: torch.dtype = torch.float32 + """dtype used when intializing the weights. Defaults to torch.float32.""" + + ############### + # Loss scaling + ############### + loss_scale: Optional[float] = None + """Static loss scaling, positive power of 2 values can improve fp16 convergence. If None, + dynamic loss scaling is used. + """ + + initial_loss_scale: float = 2**32 + """Initial loss-scale for dynamic loss scaling.""" + + min_loss_scale: float = 1.0 + """Minimum loss scale for dynamic loss scaling.""" + + loss_scale_window: float = 1000 + """Window over which to raise/lower dynamic scale.""" + + hysteresis: int = 2 + """Hysteresis for dynamic loss scaling.""" + + ############## + # Optimizer + ############## + # Adam + adam_beta1: float = 0.9 + """First coefficient for computing running averages of gradient and its square in Adam + optimizer. + """ + + adam_beta2: float = 0.999 + """Second coefficient for computing running averages of gradient and its square in Adam + optimizer. + """ + + adam_eps: float = 1e-08 + """Term added to the denominator to improve numerical stability in Adam optimizer.""" + + # SGD. + sgd_momentum: float = 0.9 + """Momentum factor for SGD optimizer.""" + + ####################### + # Distributed optimizer + ####################### + use_distributed_optimizer: bool = False + """Distribute optimizer state over data-parallel replicas.""" + + overlap_grad_reduce: bool = False + """If true, overlap grad reduce-scatter with backward compute in distributed optimizer.""" + + overlap_param_gather: bool = False + """If true, overlap param all-gather with forward compute in distributed optimizer.""" + + ################ + # Miscellaneous + ################ + clip_grad: float = 1.0 + """Gradient clipping based on global L2 norm.""" + + log_num_zeros_in_grad: bool = False + """If true, calculate and log the number of zeros in gradient.""" + + barrier_with_L1_time: bool = False + """If true, use barrier with level 1 time measurements.""" + + timers: Callable = None + """Function to get timers.""" diff --git a/frameworks/agentgym_rl/verl/utils/megatron/pipeline_parallel.py b/frameworks/agentgym_rl/verl/utils/megatron/pipeline_parallel.py new file mode 100644 index 0000000000000000000000000000000000000000..3a3790bb1a0fe0340390b7c9083f94d9d56b9383 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/megatron/pipeline_parallel.py @@ -0,0 +1,51 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +from megatron.core import parallel_state as mpu + +from .sequence_parallel import pad_to_sequence_parallel + + +def compute_transformers_input_shapes(batches, meta_info): + from flash_attn.bert_padding import unpad_input # flash 2 is a must for Megatron + # pre-compute input shapes for each micro-batch at each pp stage + input_shapes = [] + for model_inputs in batches: + input_ids = model_inputs['input_ids'] + attention_mask = model_inputs['attention_mask'] + input_ids_rmpad = unpad_input(input_ids.unsqueeze(dim=-1), attention_mask)[0] # (total_nnz, 1) + if meta_info['sequence_parallel']: + input_ids_rmpad = pad_to_sequence_parallel(input_ids_rmpad) + # compute shapes for model_inputs + input_shapes.append( + torch.Size([ + input_ids_rmpad.shape[0] // mpu.get_tensor_model_parallel_world_size(), 1, meta_info['hidden_size'] + ])) + else: + # compute shapes for model_inputs + input_shapes.append(torch.Size([input_ids_rmpad.shape[0], 1, meta_info['hidden_size']])) + return input_shapes + + +def make_batch_generator(batches, vpp_size): + if vpp_size > 1: + # has vpp + batch_generator = [batches] * vpp_size # number of vpp chunks + batch_generator = [iter(b) for b in batch_generator] + else: + # no vpp + batch_generator = iter(batches) + return batch_generator diff --git a/frameworks/agentgym_rl/verl/utils/megatron/sequence_parallel.py b/frameworks/agentgym_rl/verl/utils/megatron/sequence_parallel.py new file mode 100644 index 0000000000000000000000000000000000000000..4b76cb295ef681e30b22d45404d4d5c26493f051 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/megatron/sequence_parallel.py @@ -0,0 +1,54 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +import torch.nn.functional as F +from megatron.core import parallel_state as mpu + + +def mark_parameter_as_sequence_parallel(parameter): + setattr(parameter, 'sequence_parallel', True) + + +def is_sequence_parallel_param(param): + return hasattr(param, 'sequence_parallel') and param.sequence_parallel + + +def pad_to_sequence_parallel(unpad_tokens: torch.Tensor): + """pad the tokens such that the total length is a multiple of sp world size + + Args: + unpad_tokens: (total_nnz, ...). Tokens after removing padding + + Returns: + + """ + total_nnz = unpad_tokens.shape[0] + sp_world_size = mpu.get_tensor_model_parallel_world_size() + + if total_nnz % sp_world_size == 0: + pad_size = 0 + else: + pad_size = sp_world_size - total_nnz % sp_world_size + + if pad_size > 0: + if unpad_tokens.ndim == 1: + unpad_tokens = F.pad(unpad_tokens, (0, pad_size)) + elif unpad_tokens.ndim == 2: + unpad_tokens = F.pad(unpad_tokens, (0, 0, 0, pad_size)) + else: + raise NotImplementedError(f'Padding dim {unpad_tokens.ndim()} is not supported') + + return unpad_tokens diff --git a/frameworks/agentgym_rl/verl/utils/megatron/tensor_parallel.py b/frameworks/agentgym_rl/verl/utils/megatron/tensor_parallel.py new file mode 100644 index 0000000000000000000000000000000000000000..25a8ce422c42498a5e5cbdddc74d6c9f3ae8d06b --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/megatron/tensor_parallel.py @@ -0,0 +1,184 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Utilities for using tensor_parallel in megatron +""" +from typing import Dict +import torch +from torch.nn import init +import torch.distributed as dist +from megatron.core import ModelParallelConfig +from megatron.core import parallel_state as mpu, tensor_parallel +import verl.utils.torch_functional as verl_F + + +def update_kwargs_with_config(dictionary: Dict, config: ModelParallelConfig): + dictionary['config'] = config + return dictionary + + +def get_default_kwargs_for_model_parallel_config(): + model_parallel_config_kwargs = { + 'params_dtype': torch.float32, + 'use_cpu_initialization': False, + 'perform_initialization': True, + 'gradient_accumulation_fusion': False, + 'sequence_parallel': False, + } + return model_parallel_config_kwargs + + +def get_default_model_parallel_config(): + return ModelParallelConfig(**get_default_kwargs_for_model_parallel_config()) + + +def get_common_default_kwargs_for_parallel_linear(): + default_model_parallel_config = get_default_model_parallel_config() + common_default_kwargs = { + 'init_method': init.xavier_normal_, + 'stride': 1, + 'keep_master_weight_for_test': False, + 'config': default_model_parallel_config, + } + return common_default_kwargs + + +def get_default_kwargs_for_column_parallel_linear(): + model_parallel_config_kwargs = get_default_kwargs_for_model_parallel_config() + column_parallel_config_kwargs = { + 'async_tensor_model_parallel_allreduce': False, + } + model_parallel_config_kwargs.update(column_parallel_config_kwargs) + column_default_kwargs = { + 'config': ModelParallelConfig(**model_parallel_config_kwargs), + } + common_default_kwargs = get_common_default_kwargs_for_parallel_linear() + common_default_kwargs.update(column_default_kwargs) + return common_default_kwargs + + +def get_default_kwargs_for_row_parallel_linear(): + common_default_kwargs = get_common_default_kwargs_for_parallel_linear() + return common_default_kwargs + + +def get_default_kwargs_for_parallel_embedding(): + model_parallel_config_kwargs = get_default_kwargs_for_model_parallel_config() + embedding_default_kwargs = { + 'init_method': init.xavier_normal_, + 'config': ModelParallelConfig(**model_parallel_config_kwargs), + } + return embedding_default_kwargs + + +def is_tensor_parallel_param(param): + return (hasattr(param, 'tensor_model_parallel') and param.tensor_model_parallel) + + +def get_tensor_parallel_partition_dim(param): + assert is_tensor_parallel_param(param) + return param.partition_dim + + +def get_tensor_parallel_partition_stride(param): + assert is_tensor_parallel_param(param) + return param.partition_stride + + +class _VocabParallelEntropy(torch.autograd.Function): + + @staticmethod + def forward(ctx, vocab_parallel_logits: torch.Tensor) -> torch.Tensor: + logits_max = vocab_parallel_logits.max(dim=-1, keepdim=True).values + dist.all_reduce(logits_max, op=dist.ReduceOp.MAX, group=mpu.get_tensor_model_parallel_group()) + normalized_vocab_parallel_logits = vocab_parallel_logits - logits_max + normalized_exp_logits = normalized_vocab_parallel_logits.exp() + normalized_sum_exp_logits = normalized_exp_logits.sum(dim=-1, keepdim=True) + dist.all_reduce(normalized_sum_exp_logits, group=mpu.get_tensor_model_parallel_group()) + softmax_logits = normalized_exp_logits / normalized_sum_exp_logits + sum_softmax_times_logits = (softmax_logits * vocab_parallel_logits).sum(dim=-1, keepdim=True) + dist.all_reduce(sum_softmax_times_logits, group=mpu.get_tensor_model_parallel_group()) + entropy = logits_max + normalized_sum_exp_logits.log() - sum_softmax_times_logits + ctx.save_for_backward(vocab_parallel_logits, softmax_logits, sum_softmax_times_logits) + return entropy.squeeze(dim=-1) + + @staticmethod + def backward(ctx, grad_output: torch.Tensor) -> torch.Tensor: + vocab_parallel_logits, softmax_logits, sum_softmax_times_logits = ctx.saved_tensors + grad_input = grad_output.unsqueeze(dim=-1) * softmax_logits * (sum_softmax_times_logits - vocab_parallel_logits) + return grad_input + + +def vocab_parallel_entropy(vocab_parallel_logits: torch.Tensor) -> torch.Tensor: + """Compute entropy when the logits are sharded in tp ranks + + Args: + vocab_parallel_logits: (total_nnz, vocab_size // tp_size) + + Returns: (total_nnz,) + + """ + return _VocabParallelEntropy.apply(vocab_parallel_logits) + + +def vocab_parallel_log_probs_from_logits(logits, labels): + """TODO(zhangchi.usc1992): We may change the implementation later""" + return -tensor_parallel.vocab_parallel_cross_entropy(vocab_parallel_logits=logits, target=labels) + + +def vocab_parallel_log_probs_from_logits_response_rmpad(input_ids, attention_mask, logits_rmpad, response_length): + """Similar to log_probs_from_logits_response_rmpad, but the logits_rmpad is now spliited across tensor parallel region. + This will further reduce the peak memory usage during training + + Args: + input_ids: [batch_size, seqlen] + attention_mask: [batch_size, seqlen] + logits_rmpad: [total_nnz, vocab_size // tp_size] + response_length: int + + """ + from flash_attn.bert_padding import pad_input, unpad_input + + batch_size, seqlen = input_ids.shape + input_ids_rmpad, indices, *_ = unpad_input(input_ids.unsqueeze(-1), attention_mask=attention_mask) + input_ids_rmpad = input_ids_rmpad.squeeze(-1) + input_ids_rmpad_rolled = torch.roll(input_ids_rmpad, shifts=-1, dims=0) + full_log_probs_rmpad = vocab_parallel_log_probs_from_logits(logits=logits_rmpad, + labels=input_ids_rmpad_rolled) # (total_nnz,) + full_output = pad_input(hidden_states=full_log_probs_rmpad.unsqueeze(-1), + indices=indices, + batch=batch_size, + seqlen=seqlen) + output = full_output.squeeze(-1)[:, -response_length - 1:-1] # [batch_size, response_length] + return output + + +def vocab_parallel_compute_entropy_loss(logits, eos_mask): + """Compute Categorical entropy loss + + Args: + logits: `(torch.Tensor)` + shape: (bs, response_length, vocab_size) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + + Returns: + entropy: a scalar torch.Tensor + + """ + # compute entropy + entropy = vocab_parallel_entropy(logits) + entropy_loss = verl_F.masked_mean(entropy, mask=eos_mask) + return entropy_loss diff --git a/frameworks/agentgym_rl/verl/utils/megatron_utils.py b/frameworks/agentgym_rl/verl/utils/megatron_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..fcb6b65a79ea302e3f7eaccd5145e29adbb9edd6 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/megatron_utils.py @@ -0,0 +1,253 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Pretrain utilities.""" +from typing import Any, Dict +import time +from omegaconf import DictConfig +from verl.utils.torch_dtypes import PrecisionType +from verl.utils.memory_buffer import build_memory_reference_from_module +import torch +import torch.nn as nn +import torch.nn.functional as F + +from megatron.core import mpu, tensor_parallel +from megatron.core.utils import get_model_config +from megatron.core.transformer import TransformerConfig +from megatron.core.transformer.module import Float16Module +# from megatron.core.distributed import DistributedDataParallelConfig +from megatron.core.distributed import DistributedDataParallel as DDP +from megatron.core.enums import ModelType + + +def get_model(model_provider_func, model_type=ModelType.encoder_or_decoder, wrap_with_ddp=True): + """Build the model.""" + # Build model. + if mpu.get_pipeline_model_parallel_world_size() > 1 and \ + mpu.get_virtual_pipeline_model_parallel_world_size() is not None: + assert model_type != ModelType.encoder_and_decoder, \ + "Interleaved schedule not supported for model with both encoder and decoder" + model = [] + for i in range(mpu.get_virtual_pipeline_model_parallel_world_size()): + mpu.set_virtual_pipeline_model_parallel_rank(i) + # Set pre_process and post_process only after virtual rank is set. + pre_process = mpu.is_pipeline_first_stage() + post_process = mpu.is_pipeline_last_stage() + this_model = model_provider_func(pre_process=pre_process, post_process=post_process) + this_model.model_type = model_type + model.append(this_model) + else: + pre_process = mpu.is_pipeline_first_stage() + post_process = mpu.is_pipeline_last_stage() + add_encoder = True + add_decoder = True + if model_type == ModelType.encoder_and_decoder: + if mpu.get_pipeline_model_parallel_world_size() > 1: + assert mpu.get_pipeline_model_parallel_split_rank() is not None, \ + "Split rank needs to be specified for model with both encoder and decoder" + rank = mpu.get_pipeline_model_parallel_rank() + split_rank = mpu.get_pipeline_model_parallel_split_rank() + world_size = mpu.get_pipeline_model_parallel_world_size() + pre_process = rank == 0 or rank == split_rank + post_process = (rank == (split_rank - 1)) or (rank == (world_size - 1)) + add_encoder = mpu.is_pipeline_stage_before_split() + add_decoder = mpu.is_pipeline_stage_after_split() + model = model_provider_func(pre_process=pre_process, + post_process=post_process, + add_encoder=add_encoder, + add_decoder=add_decoder) + else: + model = model_provider_func(pre_process=pre_process, post_process=post_process) + model.model_type = model_type + + if not isinstance(model, list): + model = [model] + + # Set tensor model parallel attributes if not set. + # Only parameters that are already tensor model parallel have these + # attributes set for them. We should make sure the default attributes + # are set for all params so the optimizer can use them. + for model_module in model: + for param in model_module.parameters(): + tensor_parallel.set_defaults_if_not_set_tensor_model_parallel_attributes(param) + + # Print number of parameters. + if mpu.get_data_parallel_rank() == 0: + print(' > number of parameters on (tensor, pipeline) ' + 'model parallel rank ({}, {}): {}'.format( + mpu.get_tensor_model_parallel_rank(), mpu.get_pipeline_model_parallel_rank(), + sum([sum([p.nelement() for p in model_module.parameters()]) for model_module in model])), + flush=True) + + # GPU allocation. + for model_module in model: + model_module.cuda(torch.cuda.current_device()) + + # Fp16 conversion. + config = get_model_config(model[0]) + if config.fp16 or config.bf16: # the ModelParallelConfig in GPTModel + model = [Float16Module(config, model_module) for model_module in model] + + if wrap_with_ddp: + model = [ + DDP(config=config, + module=model_chunk, + data_parallel_group=mpu.get_data_parallel_group(with_context_parallel=True), + accumulate_allreduce_grads_in_fp32=True, + overlap_grad_reduce=False, + use_distributed_optimizer=True, + disable_bucketing=(model_chunk_idx > 0)) for (model_chunk_idx, model_chunk) in enumerate(model) + ] + # # Broadcast params from data parallel src rank to other data parallel ranks. + # if args.data_parallel_random_init: + for model_module in model: + model_module.broadcast_params() + return model + + +ALL_MODULE_WRAPPER_CLASSNAMES = (DDP, Float16Module) + + +def unwrap_model(model, module_instances=ALL_MODULE_WRAPPER_CLASSNAMES): + return_list = True + if not isinstance(model, list): + model = [model] + return_list = False + unwrapped_model = [] + for model_module in model: + while isinstance(model_module, module_instances): + model_module = model_module.module + unwrapped_model.append(model_module) + if not return_list: + return unwrapped_model[0] + return unwrapped_model + + +from transformers import PretrainedConfig + + +def convert_config(hf_config: PretrainedConfig, megatron_config) -> TransformerConfig: + print(f'megatron config {megatron_config}') + dt = PrecisionType.to_dtype(megatron_config['param_dtype']) + print(f'pipeline_dtype=megatron_config {dt}') + transformer_config = TransformerConfig( + num_layers=hf_config.num_hidden_layers, + hidden_size=hf_config.hidden_size, + num_attention_heads=hf_config.num_attention_heads, + num_query_groups=hf_config.num_key_value_heads, + ffn_hidden_size=hf_config.intermediate_size, + # max_position_embeddings=hf_config.max_position_embeddings, + activation_func=F.silu, + normalization='RMSNorm', + # rotary_percent=False, # default, + gated_linear_unit=True, # for llama + use_cpu_initialization=True, + apply_residual_connection_post_layernorm=False, # check what's this mean + add_bias_linear=False, + tensor_model_parallel_size=mpu.get_tensor_model_parallel_world_size(), + pipeline_model_parallel_size=mpu.get_pipeline_model_parallel_world_size(), + virtual_pipeline_model_parallel_size=mpu.get_virtual_pipeline_model_parallel_world_size(), + pipeline_dtype=PrecisionType.to_dtype(megatron_config['param_dtype']), + params_dtype=PrecisionType.to_dtype(megatron_config['param_dtype']), + sequence_parallel=megatron_config['sequence_parallel_enabled'], + variable_seq_lengths=True, + masked_softmax_fusion=True, + bf16=PrecisionType.to_dtype(megatron_config['param_dtype']) is torch.bfloat16) + if torch.distributed.get_rank() == 0: + print(f'tensor_parallel_size={transformer_config.tensor_model_parallel_size} \n \ + pipeline_model_parallel_size={transformer_config.pipeline_model_parallel_size} \n \ + virtual_pipeline_model_parallel_size={transformer_config.virtual_pipeline_model_parallel_size} \n \ + pipeline_dtype={transformer_config.pipeline_dtype} \n \ + params_dtype={transformer_config.params_dtype} \n \ + sequence_parallel={transformer_config.sequence_parallel} \n \ + variable_seq_lengths={transformer_config.variable_seq_lengths} \n \ + masked_softmax_fusion={transformer_config.masked_softmax_fusion} \n ') + + return transformer_config + + +# from megatron.core.optimizer import OptimizerConfig + +from verl.utils.megatron.optimizer_config import OptimizerConfig + + +def init_megatron_optim_config(optim_config: Dict) -> OptimizerConfig: + config = OptimizerConfig( + optimizer='adam', + lr=optim_config.get('lr'), + clip_grad=optim_config.get('clip_grad'), + weight_decay=1e-2, + bf16=True, + params_dtype=torch.bfloat16, + use_distributed_optimizer=True, + ) + return config + + +from megatron.core import ModelParallelConfig + + +def init_model_parallel_config(config: DictConfig) -> ModelParallelConfig: + # TODO(sgm): check how to disable megatron timers + timers = FakeTimers() + return ModelParallelConfig(tensor_model_parallel_size=config.get('tensor_model_parallel_size'), + pipeline_model_parallel_size=config.get('pipeline_model_parallel_size'), + virtual_pipeline_model_parallel_size=config.get('virtual_pipeline_model_parallel_size'), + sequence_parallel=config.get('sequence_parallel'), + params_dtype=PrecisionType.to_dtype(config.get('param_dtype')), + pipeline_dtype=PrecisionType.to_dtype(config.get('param_dtype')), + bf16=True, + fp16=False, + timers=timers) + + +class FakeTimers: + """Disable All Megatron Timing with FakeTimers""" + + def __init__(self): + from megatron.timers import DummyTimer + self.dummy_timer = DummyTimer() + + def __call__(self, *args: Any, **kwds: Any) -> Any: + return self.dummy_timer + + +def offload_megatron_param_and_grad(module_list: nn.ModuleList, offload_grad=False, hybrid_engine=None): + if hybrid_engine is not None: + pp_rank = mpu.get_pipeline_model_parallel_rank() + for buffer in hybrid_engine.memory_buffers[pp_rank].values(): + buffer.data = buffer.data.to('cpu', non_blocking=True) + build_memory_reference_from_module(module_list, hybrid_engine.memory_buffers[pp_rank], maintain_weight=True) + else: + for module in module_list: + for _, param in module.named_parameters(): + param.data = param.data.to('cpu', non_blocking=True) + if offload_grad and param.grad is not None: + param.grad = param.grad.to("cpu", non_blocking=True) + torch.cuda.empty_cache() + + +def load_megatron_param_and_grad(module_list: nn.ModuleList, device_id, load_grad=False, hybrid_engine=None): + if hybrid_engine is not None: + pp_rank = mpu.get_pipeline_model_parallel_rank() + for buffer in hybrid_engine.memory_buffers[pp_rank].values(): + buffer.data = buffer.data.to(device_id, non_blocking=True) + build_memory_reference_from_module(module_list, hybrid_engine.memory_buffers[pp_rank], maintain_weight=True) + else: + for module in module_list: + for _, param in module.named_parameters(): + param.data = param.data.to(device_id, non_blocking=True) + if load_grad and param.grad is not None: + param.grad = param.grad.to(device_id, non_blocking=True) + torch.cuda.empty_cache() diff --git a/frameworks/agentgym_rl/verl/utils/memory_buffer.py b/frameworks/agentgym_rl/verl/utils/memory_buffer.py new file mode 100644 index 0000000000000000000000000000000000000000..2e07e42f7bc4648d3376dba404ae122e07ccb0d0 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/memory_buffer.py @@ -0,0 +1,214 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +This file contains utilities to manipulate torch memory buffers +""" + +from typing import Dict, List + +import torch +from torch import nn + + +class MemoryBuffer: + """ + A memory buffer is a contiguous torch tensor that may combine multiple tensors sharing with the underlying + memory. It must have a unique type to support this behavior. + """ + + def __init__(self, numel: int, numel_padded: int, dtype: torch.dtype): + self.numel = numel + self.numel_padded = numel_padded + self.dtype = dtype + self.data = torch.zeros(self.numel_padded, dtype=self.dtype, device='cuda', requires_grad=False) + + def zero(self): + """Reset the buffer to zero.""" + self.data.zero_() + + def get(self, shape, start_index): + """Return a tensor with the input `shape` as a view into the + 1-D data starting at `start_index`.""" + end_index = start_index + shape.numel() + assert end_index <= self.numel, \ + 'requested tensor is out of the buffer range.' + buffer_tensor = self.data[start_index:end_index] + buffer_tensor = buffer_tensor.view(shape) + return buffer_tensor + + +def calc_padded_numel(shape: torch.Size, dtype: torch.dtype): + """for cuda memory alignment, make sure alignment by 128-bits""" + align_numel = 128 // torch.finfo(dtype).bits + numel = shape.numel() + return (numel + align_numel - 1) // align_numel * align_numel + + +def get_weight_buffer_meta_from_module(module: nn.Module) -> Dict[str, Dict]: + """ + Return a dictionary containing name to a shape and dtype. + """ + weight_buffer_meta = {} + for name, param in sorted(module.named_parameters()): + weight_buffer_meta[name] = {'shape': param.shape, 'dtype': param.dtype} + return weight_buffer_meta + + +def build_memory_buffer(weight_buffer_meta: Dict[str, Dict]) -> Dict[torch.dtype, MemoryBuffer]: + """Build the memory buffer given weight_buffer_meta + + Args: + weight_buffer_meta: contains mapping from name to a dictionary containing shape and dtype of the tensors + + Returns: a large memory buffer for each dtype that can hold all the tensors + + """ + memory_buffers = {} + total_numel_map = {} # map from dtype to the total numel + for name, meta_info in sorted(weight_buffer_meta.items()): + shape = meta_info['shape'] + dtype = meta_info['dtype'] + + assert isinstance(shape, torch.Size) + assert isinstance(dtype, torch.dtype) + + if dtype not in total_numel_map: + total_numel_map[dtype] = 0 + + total_numel_map[dtype] += calc_padded_numel(shape, dtype) + + for dtype, total_numel in total_numel_map.items(): + memory_buffers[dtype] = MemoryBuffer(total_numel, total_numel, dtype) + + return memory_buffers + + +def build_memory_reference_from_module(module: torch.nn.Module, + memory_buffers: Dict[torch.dtype, MemoryBuffer], + maintain_weight=True): + start_index = {} + for dtype in memory_buffers.keys(): + start_index[dtype] = 0 + for name, param in sorted(module.named_parameters()): + memory_buffer = memory_buffers[param.dtype] + buffer = memory_buffer.get(shape=param.shape, start_index=start_index[param.dtype]) + # need to increment start_index + start_index[param.dtype] += calc_padded_numel(param.shape, dtype) + if maintain_weight: + buffer.copy_(param.data) + param.data = buffer + + +def build_memory_reference(weight_buffer_meta: Dict[str, Dict], memory_buffers: Dict[torch.dtype, MemoryBuffer]): + """Build the memory references. The memory buffers are built using the build_memory_buffer API. + This API will allocate a weight buffer pointer to the memory buffer according to the weight_buffer_meta. + + Args: + weight_buffer_meta: + memory_buffers: + + Returns: + + """ + start_idx = {} + weight_buffers = {} + for dtype in memory_buffers.keys(): + start_idx[dtype] = 0 + + for name, meta_info in sorted(weight_buffer_meta.items()): + shape = meta_info['shape'] + dtype = meta_info['dtype'] + + buffer = memory_buffers[dtype].get(shape, start_index=start_idx[dtype]) + start_idx[dtype] += calc_padded_numel(shape, dtype) + weight_buffers[name] = buffer + + return weight_buffers + + +class MemoryBufferModuleWrapper: + """ + Note that we do not design MemoryBufferModuleWrapper as an nn.Module due to + - It will change the checkpoint name + """ + + def __init__(self, module: nn.Module): + super().__init__() + self.module = module + self.weight_buffer_meta = get_weight_buffer_meta_from_module(self.module) + self.memory_buffers = build_memory_buffer(self.weight_buffer_meta) + build_memory_reference_from_module(self.module, self.memory_buffers) + + def get_memory_buffers(self): + return self.memory_buffers + + def get_weight_buffer_meta(self): + return self.weight_buffer_meta + + +class MegatronMemoryBufferForRollout(object): + """ + We assume that + - inference engine has tp + dp + - actor has tp + pp + dp + - the tp between inference engine and actor should be the same + - memory_buffers: contains a list of memory_buffers, each is a dict from dtype to MemoryBuffer + - weight_buffers: contains a list of weight_buffers, each is a dict from name to param + - named_parameters: a dict from name to parameter that normalizes the names from pp and vpp. Note that + the named_parameters may not be directly compatible with inference engine. User has to take care of + this part such as the layout mismatches. (e.g. qkv transpose) + - Note that weight_buffer, named_parameters and memory_buffers share the same underlying GPU memory. + - When doing weight sync, the data is transfer via memory buffers + """ + + def __init__(self, transform_memory_param_fn): + self._memory_buffers = [] + self._weight_buffers = [] + self._named_parameters = {} + self.transform_memory_param_fn = transform_memory_param_fn + + def initialize_weight_buffer(self, weight_buffer_meta_pp: List[Dict[str, Dict]]): + """ + Initialize the weight buffer. The weight buffer is obtained according to the actor. We will construct + a large buffer for each dtype in the weight_buffer. + + Args: + weight_buffer_meta: contains pp models, each pp models contains a dictionary of mapping from + + Returns: None + + """ + self.weight_buffer_meta_pp = weight_buffer_meta_pp + + for weight_buffer_meta in self.weight_buffer_meta_pp: + memory_buffer = build_memory_buffer(weight_buffer_meta) + self._memory_buffers.append(memory_buffer) + self._weight_buffers.append(None) + + def build_memory_reference(self): + for i, weight_buffer_meta in enumerate(self.weight_buffer_meta_pp): + self._weight_buffers[i] = build_memory_reference(weight_buffer_meta, self._memory_buffers[i]) + self._named_parameters = self.transform_memory_param_fn(self._weight_buffers) + + @property + def named_parameters(self): + return self._named_parameters + + @property + def weight_buffers(self): + return self._weight_buffers + + @property + def memory_buffers(self): + return self._memory_buffers diff --git a/frameworks/agentgym_rl/verl/utils/model.py b/frameworks/agentgym_rl/verl/utils/model.py new file mode 100644 index 0000000000000000000000000000000000000000..f319e400a67e38f1596f49f7843c84bf01e79123 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/model.py @@ -0,0 +1,349 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Utilities to create common models from huggingface +""" +import os +import warnings +from typing import Dict, Type, Optional + +import numpy as np +import torch +from torch import nn +from transformers import AutoConfig, AutoModelForCausalLM, PretrainedConfig, MistralForSequenceClassification, GenerationConfig +from verl.models.registry import ModelRegistry + + +class LambdaLayer(nn.Module): + + def __init__(self, fn): + super().__init__() + self.fn = fn + + def forward(self, *args, **kwargs): + return self.fn(*args, **kwargs) + + +def squeeze(x): + return torch.squeeze(x, dim=-1) + + +def update_model_config(module_config, override_config_kwargs): + for key, val in override_config_kwargs.items(): + setattr(module_config, key, val) + + +def get_huggingface_actor_config(model_name: str, override_config_kwargs=None, trust_remote_code=False) -> Dict: + if override_config_kwargs is None: + override_config_kwargs = {} + assert isinstance(override_config_kwargs, Dict), \ + f'override_config_kwargs must be a dict, got {type(override_config_kwargs)}' + module_config = AutoConfig.from_pretrained(model_name, trust_remote_code=trust_remote_code) + update_model_config(module_config, override_config_kwargs) + + return module_config + + +def get_generation_config( + model: str, + trust_remote_code: bool = False, +) -> Optional[GenerationConfig]: + try: + return GenerationConfig.from_pretrained(model) + except OSError: # Not found + try: + config = get_huggingface_actor_config( + model, + trust_remote_code=trust_remote_code, + ) + return GenerationConfig.from_model_config(config) + except OSError: # Not found + return None + + +def create_huggingface_actor(model_name: str, override_config_kwargs=None, automodel_kwargs=None) -> nn.Module: + """ + + Args: + model_name: + actor_override_config_kwargs: + + Returns: + + """ + if override_config_kwargs is None: + override_config_kwargs = {} + if automodel_kwargs is None: + automodel_kwargs = {} + assert isinstance(override_config_kwargs, Dict), \ + f'override_config_kwargs must be a dict, got {type(override_config_kwargs)}' + module_config = get_huggingface_actor_config(model_name, + override_config_kwargs, + trust_remote_code=automodel_kwargs.get('trust_remote_code', False)) + module: nn.Module = AutoModelForCausalLM.from_config(module_config, **automodel_kwargs) + return module + + +def create_huggingface_critic(model_name: str, override_config_kwargs=None, automodel_kwargs=None) -> nn.Module: + """ + + Args: + model_name: + override_config_kwargs: + + Returns: + + """ + critic_module: nn.Module = create_huggingface_actor(model_name, + override_config_kwargs=override_config_kwargs, + automodel_kwargs=automodel_kwargs) + if automodel_kwargs is None: + automodel_kwargs = {} + torch_dtype = automodel_kwargs.get('torch_dtype', torch.float32) + critic_module.lm_head = nn.Sequential(nn.Linear(critic_module.config.hidden_size, 1, dtype=torch_dtype), + LambdaLayer(fn=squeeze)) + return critic_module + + +def get_model_size(model: nn.Module, scale='auto'): + n_params = sum(p.numel() for p in model.parameters()) + + if scale == 'auto': + if n_params > 1e9: + scale = 'B' + elif n_params > 1e6: + scale = 'M' + elif n_params > 1e3: + scale = 'K' + else: + scale = '' + + if scale == 'B': + n_params = n_params / 1e9 + elif scale == 'M': + n_params = n_params / 1e6 + elif scale == 'K': + n_params = n_params / 1e3 + elif scale == '': + pass + else: + raise NotImplemented(f'Unknown scale {scale}') + + return n_params, scale + + +def print_model_size(model: nn.Module, name: str = None): + n_params, scale = get_model_size(model, scale='auto') + if name is None: + name = model.__class__.__name__ + print(f'{name} contains {n_params:.2f}{scale} parameters') + + +def create_random_mask(input_ids: torch.Tensor, + max_ratio_of_valid_token: float, + max_ratio_of_left_padding: float, + min_ratio_of_valid_token: float = 0): + """Create a random mask given input_ids. Support left padding and right padding. + Process: + - Sample valid token length + - Sample left_padding length + - Generate padding + + Args: + input_ids: + shape (batch_size, seq_len) + + Returns: + + """ + assert max_ratio_of_valid_token > 0 and max_ratio_of_valid_token <= 1. + assert max_ratio_of_left_padding >= 0 and max_ratio_of_left_padding < 1. + assert min_ratio_of_valid_token <= max_ratio_of_valid_token + + batch_size, sequence_length = input_ids.shape + max_num_valid_tokens = int(sequence_length * max_ratio_of_valid_token) + min_num_valid_tokens = max(1, int(sequence_length * min_ratio_of_valid_token)) + max_left_padding = int(sequence_length * max_ratio_of_left_padding) + assert max_num_valid_tokens + max_left_padding <= sequence_length + assert max_num_valid_tokens > 0 and max_ratio_of_valid_token <= sequence_length + masks = torch.ones_like(input_ids, dtype=torch.int64) + # TODO: we can make this faster + for i in range(batch_size): + num_left_padding = np.random.randint(low=0, high=max_left_padding + 1, dtype=np.int64) + num_valid = np.random.randint(low=min_num_valid_tokens, high=max_num_valid_tokens + 1, dtype=np.int64) + + for index in range(num_left_padding): + masks[i, index] = 0 + + for index in range(num_left_padding + num_valid, sequence_length): + masks[i, index] = 0 + return masks + + +def compute_position_id_with_mask(mask): + return torch.clip(torch.cumsum(mask, dim=-1) - 1, min=0, max=None) + + +def normalize_pp_vpp_params(params, num_hidden_layers, layer_name='layers'): + """ + Normalize the pp vpp params into a complete named parameters. + This is useful when gather parameters from pp ranks and passed to a model without pp + + params: List[List[Dict[str, param]]] + params contains a list of pp, with a list of vpp named_parameters in each vpp chunk. + output: Dict[str, param] + + """ + + def normalize_model_name(name, pp_rank, vpp_rank, pp_size, vpp_size, num_layers): + """ + Transform the model name in each model_chunk in each pp stage into the name in inference engine + """ + if vpp_size > 1: + # print(f'try to bind vpp params to inference engine...') + layers_per_pp = num_layers // pp_size + layers_per_vpp = layers_per_pp // vpp_size + pp_offset = layers_per_vpp * pp_rank + vpp_offset = (layers_per_vpp * pp_size) * vpp_rank + layer_offset = pp_offset + vpp_offset + else: + layers_per_pp = num_layers // pp_size + layer_offset = layers_per_pp * pp_rank + + if layer_name in name: # belong to an intermediate layer + split_name = name.split('.') + # find the num next to split_name + for i, name in enumerate(split_name): + if name == layer_name: + break + layer_num_idx = i + 1 + # check the name + assert len(split_name) >= layer_num_idx + 1, f'split_name = {split_name}' + assert split_name[layer_num_idx].isdigit(), f'split_name = {split_name}' + # increment layer_num_idx by layer_offset + split_name[layer_num_idx] = str(int(split_name[layer_num_idx]) + layer_offset) + name = '.'.join(split_name) # weight name in inference_tp_model + return name + + pp_size = len(params) + normalized_name_to_param = {} + for pp_rank in range(len(params)): + vpp_size = len(params[pp_rank]) + for vpp_rank in range(vpp_size): + for name, param in params[pp_rank][vpp_rank].items(): + normalized_name = normalize_model_name(name, pp_rank, vpp_rank, pp_size, vpp_size, num_hidden_layers) + normalized_name_to_param[normalized_name] = param + + return normalized_name_to_param + + +def get_parallel_model_from_config(config, megatron_config, pre_process=None, post_process=None, value=False): + from megatron.core import ModelParallelConfig + assert isinstance(megatron_config, ModelParallelConfig) + model_class = _get_parallel_model_architecture_from_config(config, value) + + model = model_class(config, megatron_config, pre_process=pre_process, post_process=post_process) + return model + + +def _get_parallel_model_architecture_from_config(config: PretrainedConfig, value=False) -> Type[nn.Module]: + architectures = getattr(config, "architectures", []) + for arch in architectures: + model_cls = ModelRegistry.load_model_cls(arch, value) + if model_cls is not None: + return model_cls + raise ValueError(f"Model architectures {architectures} are not supported for now. " + f"Supported architectures: {ModelRegistry.get_supported_archs()}") + + +def load_megatron_model_weights(config, + model_config, + parallel_model, + params_dtype, + is_value_model=False, + local_cache_path='~/.cache/verl/rlhf'): + assert hasattr(model_config, "architectures"), "architectures cannot be empty when load weight!" + architectures = getattr(model_config, "architectures", []) + local_cache_path = os.path.expanduser(local_cache_path) + + if config.model.path.startswith("hdfs:"): + from verl.utils.fs import copy_local_path_from_hdfs + print(f'start download from {config.model.path}') + local_model_path = copy_local_path_from_hdfs(src=config.model.path, cache_dir=local_cache_path) + print('finish download') + else: + print(f"load from local dir {config.model.path}") + local_model_path = config.model.path + + # TODO: to find a better way to load mistral7b-rm lm_head + if 'mistral7b-rm' in config.model.path: + model = MistralForSequenceClassification.from_pretrained(local_model_path) # use score head instead of lm_head + state_dict = model.state_dict() + state_dict['lm_head.weight'] = state_dict['score.weight'] + state_dict['model.embed_tokens.weight'] = state_dict[ + 'model.embed_tokens.weight'][:32000] # workaround, 32001 -> 32000 + is_value_model = True + else: + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + model = AutoModelForCausalLM.from_pretrained(local_model_path) + state_dict = model.state_dict() + + from verl.models.weight_loader_registry import get_weight_loader + print(f'before weight loader: architectures = {architectures}...') + for arch in architectures: + print(f'call weight loader arch = {arch}, model config = {model.config}') + weight_loader = get_weight_loader(arch) + weight_loader(state_dict=state_dict, + wrapped_models=parallel_model, + config=model.config, + params_dtype=params_dtype, + is_value_model=is_value_model) + + +# pad input_ids_rmpad, cu_seqlens and max_seqlen_in_batch to be divisible by tp +def pad_packed_inputs(unpad_tokens: torch.Tensor, cu_seqlens, max_seqlen_in_batch, size): + """pad the tokens such that the total length is a multiple of size. + This function is useful when applying sequence parallel and context parallel + + Args: + unpad_tokens: (total_nnz, ...). Tokens after removing padding + cu_seqlens: (total_nnz + 1,) + max_seqlen_in_batch: int + + Returns: + + """ + F = nn.functional + + total_nnz = unpad_tokens.shape[0] + + if total_nnz % size == 0: + pad_size = 0 + else: + pad_size = size - total_nnz % size + + # we assume adding a new data in the batch with seqlen pad_size + if pad_size > 0: + if unpad_tokens.ndim == 1: + unpad_tokens = F.pad(unpad_tokens, (0, pad_size)) + elif unpad_tokens.ndim == 2: + unpad_tokens = F.pad(unpad_tokens, (0, 0, 0, pad_size)) + else: + raise NotImplementedError(f'Padding dim {unpad_tokens.ndim()} is not supported') + + cu_seqlens = F.pad(cu_seqlens, (0, 1), value=pad_size + cu_seqlens[-1]) + max_seqlen_in_batch = max(max_seqlen_in_batch, pad_size) + + return unpad_tokens, cu_seqlens, max_seqlen_in_batch diff --git a/frameworks/agentgym_rl/verl/utils/py_functional.py b/frameworks/agentgym_rl/verl/utils/py_functional.py new file mode 100644 index 0000000000000000000000000000000000000000..8f5a0e176779cc19d3035a3af77a1bdf1f39349a --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/py_functional.py @@ -0,0 +1,56 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Contain small python utility functions +""" + +from typing import Dict +from types import SimpleNamespace + + +def union_two_dict(dict1: Dict, dict2: Dict): + """Union two dict. Will throw an error if there is an item not the same object with the same key. + + Args: + dict1: + dict2: + + Returns: + + """ + for key, val in dict2.items(): + if key in dict1: + assert dict2[key] == dict1[key], \ + f'{key} in meta_dict1 and meta_dict2 are not the same object' + dict1[key] = val + + return dict1 + + +def append_to_dict(data: Dict, new_data: Dict): + for key, val in new_data.items(): + if key not in data: + data[key] = [] + data[key].append(val) + + +class NestedNamespace(SimpleNamespace): + + def __init__(self, dictionary, **kwargs): + super().__init__(**kwargs) + for key, value in dictionary.items(): + if isinstance(value, dict): + self.__setattr__(key, NestedNamespace(value)) + else: + self.__setattr__(key, value) diff --git a/frameworks/agentgym_rl/verl/utils/ray_utils.py b/frameworks/agentgym_rl/verl/utils/ray_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..9a75df6c37bc5a295aaa192b2a56cca2423e94b9 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/ray_utils.py @@ -0,0 +1,43 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Contains commonly used utilities for ray +""" + +import ray + +import concurrent.futures + + +def parallel_put(data_list, max_workers=None): + + def put_data(index, data): + return index, ray.put(data) + + if max_workers is None: + max_workers = min(len(data_list), 16) + + with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor: + data_list_f = [executor.submit(put_data, i, data) for i, data in enumerate(data_list)] + res_lst = [] + for future in concurrent.futures.as_completed(data_list_f): + res_lst.append(future.result()) + + # reorder based on index + output = [None for _ in range(len(data_list))] + for res in res_lst: + index, data_ref = res + output[index] = data_ref + + return output diff --git a/frameworks/agentgym_rl/verl/utils/rendezvous/__init__.py b/frameworks/agentgym_rl/verl/utils/rendezvous/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/rendezvous/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/utils/rendezvous/ray_backend.py b/frameworks/agentgym_rl/verl/utils/rendezvous/ray_backend.py new file mode 100644 index 0000000000000000000000000000000000000000..c0d2bd906fe14584896627143dea2d3ec032d912 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/rendezvous/ray_backend.py @@ -0,0 +1,77 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import time + +from cupy.cuda.nccl import NcclCommunicator, get_unique_id + +import ray +from ray.util import list_named_actors + + +@ray.remote +class NCCLIDStore: + + def __init__(self, nccl_id): + self._nccl_id = nccl_id + + def get(self): + return self._nccl_id + + +def get_nccl_id_store_by_name(name): + all_actors = list_named_actors(all_namespaces=True) + matched_actors = [actor for actor in all_actors if actor.get("name", None) == name] + if len(matched_actors) == 1: + actor = matched_actors[0] + return ray.get_actor(**actor) + elif len(matched_actors) > 1: + logging.warning(f"multiple actors with same name found: {matched_actors}") + elif len(matched_actors) == 0: + logging.info(f"failed to get any actor named {name}") + return None + + +def create_nccl_communicator_in_ray(rank: int, + world_size: int, + group_name: str, + max_retries: int = 100, + interval_s: int = 5): + if rank == 0: + nccl_id = get_unique_id() + nccl_id_store = NCCLIDStore.options(name=group_name).remote(nccl_id) + + assert ray.get(nccl_id_store.get.remote()) == nccl_id + communicator = NcclCommunicator( + ndev=world_size, + commId=nccl_id, + rank=0, + ) + return communicator + else: + for i in range(max_retries): + nccl_id_store = get_nccl_id_store_by_name(group_name) + if nccl_id_store is not None: + logging.info(f"nccl_id_store {group_name} got") + nccl_id = ray.get(nccl_id_store.get.remote()) + logging.info(f"nccl id for {group_name} got: {nccl_id}") + communicator = NcclCommunicator( + ndev=world_size, + commId=nccl_id, + rank=rank, + ) + return communicator + logging.info(f"failed to get nccl_id for {i+1} time, sleep for {interval_s} seconds") + time.sleep(interval_s) diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/__init__.py b/frameworks/agentgym_rl/verl/utils/reward_score/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..f53662c10360eda1a6e0744f7be0171731227352 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/__init__.py @@ -0,0 +1,39 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# from . import gsm8k, math, prime_math, prime_code + + +def _default_compute_score(data_source, solution_str, ground_truth): + if data_source == 'openai/gsm8k': + from . import gsm8k + res = gsm8k.compute_score(solution_str, ground_truth) + elif data_source in ['lighteval/MATH', 'DigitalLearningGmbH/MATH-lighteval']: + from . import math + res = math.compute_score(solution_str, ground_truth) + elif data_source in [ + 'numina_aops_forum', 'numina_synthetic_math', 'numina_amc_aime', 'numina_synthetic_amc', 'numina_cn_k12', + 'numina_olympiads' + ]: + from . import prime_math + res = prime_math.compute_score(solution_str, ground_truth) + elif data_source in ['codecontests', 'apps', 'codeforces', 'taco']: + from . import prime_code + res = prime_code.compute_score(solution_str, ground_truth, continuous=True) + else: + raise NotImplementedError + + if isinstance(res, (int, float, bool)): + return float(res) + else: + return float(res[0]) diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/gsm8k.py b/frameworks/agentgym_rl/verl/utils/reward_score/gsm8k.py new file mode 100644 index 0000000000000000000000000000000000000000..7091037643bc656f93c5c1a6acefb643d58421fe --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/gsm8k.py @@ -0,0 +1,63 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import re + + +def extract_solution(solution_str, method='strict'): + assert method in ['strict', 'flexible'] + + if method == 'strict': + # this also tests the formatting of the model + solution = re.search("#### (\\-?[0-9\\.\\,]+)", solution_str) + if solution is None: + final_answer = None + else: + final_answer = solution.group(0) + final_answer = final_answer.split('#### ')[1].replace(',', '').replace('$', '') + elif method == 'flexible': + answer = re.findall("(\\-?[0-9\\.\\,]+)", solution_str) + final_answer = None + if len(answer) == 0: + # no reward is there is no answer + pass + else: + invalid_str = ['', '.'] + # find the last number that is not '.' + for final_answer in reversed(answer): + if final_answer not in invalid_str: + break + return final_answer + + +def compute_score(solution_str, ground_truth, method='strict', format_score=0., score=1.): + """The scoring function for GSM8k. + + Reference: Trung, Luong, et al. "Reft: Reasoning with reinforced fine-tuning." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. + + Args: + solution_str: the solution text + ground_truth: the ground truth + method: the method to extract the solution, choices are 'strict' and 'flexible' + format_score: the score for the format + score: the score for the correct answer + """ + answer = extract_solution(solution_str=solution_str, method=method) + if answer is None: + return 0 + else: + if answer == ground_truth: + return score + else: + return format_score \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/math.py b/frameworks/agentgym_rl/verl/utils/reward_score/math.py new file mode 100644 index 0000000000000000000000000000000000000000..50792aa6edd082091a786f4d4fa29d0a601702cf --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/math.py @@ -0,0 +1,227 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Adapted from https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/hendrycks_math/utils.py + + +def compute_score(solution_str, ground_truth) -> float: + retval = 0. + try: + string_in_last_boxed = last_boxed_only_string(solution_str) + if string_in_last_boxed is not None: + answer = remove_boxed(string_in_last_boxed) + if is_equiv(answer, ground_truth): + retval = 1. + except Exception as e: + print(e) + + return retval + + +# string normalization from https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hendrycks_math.py +def is_equiv(str1, str2, verbose=False): + if str1 is None and str2 is None: + print("WARNING: Both None") + return True + if str1 is None or str2 is None: + return False + + try: + ss1 = strip_string(str1) + ss2 = strip_string(str2) + if verbose: + print(ss1, ss2) + return ss1 == ss2 + except Exception: + return str1 == str2 + + +def remove_boxed(s): + if "\\boxed " in s: + left = "\\boxed " + assert s[:len(left)] == left + return s[len(left):] + + left = "\\boxed{" + + assert s[:len(left)] == left + assert s[-1] == "}" + + return s[len(left):-1] + + +def last_boxed_only_string(string): + idx = string.rfind("\\boxed") + if "\\boxed " in string: + return "\\boxed " + string.split("\\boxed ")[-1].split("$")[0] + if idx < 0: + idx = string.rfind("\\fbox") + if idx < 0: + return None + + i = idx + right_brace_idx = None + num_left_braces_open = 0 + while i < len(string): + if string[i] == "{": + num_left_braces_open += 1 + if string[i] == "}": + num_left_braces_open -= 1 + if num_left_braces_open == 0: + right_brace_idx = i + break + i += 1 + + if right_brace_idx is None: + retval = None + else: + retval = string[idx:right_brace_idx + 1] + + return retval + + +def fix_fracs(string): + substrs = string.split("\\frac") + new_str = substrs[0] + if len(substrs) > 1: + substrs = substrs[1:] + for substr in substrs: + new_str += "\\frac" + if substr[0] == "{": + new_str += substr + else: + try: + assert len(substr) >= 2 + except AssertionError: + return string + a = substr[0] + b = substr[1] + if b != "{": + if len(substr) > 2: + post_substr = substr[2:] + new_str += "{" + a + "}{" + b + "}" + post_substr + else: + new_str += "{" + a + "}{" + b + "}" + else: + if len(substr) > 2: + post_substr = substr[2:] + new_str += "{" + a + "}" + b + post_substr + else: + new_str += "{" + a + "}" + b + string = new_str + return string + + +def fix_a_slash_b(string): + if len(string.split("/")) != 2: + return string + a = string.split("/")[0] + b = string.split("/")[1] + try: + a = int(a) + b = int(b) + assert string == "{}/{}".format(a, b) + new_string = "\\frac{" + str(a) + "}{" + str(b) + "}" + return new_string + except AssertionError: + return string + + +def remove_right_units(string): + # "\\text{ " only ever occurs (at least in the val set) when describing units + if "\\text{ " in string: + splits = string.split("\\text{ ") + assert len(splits) == 2 + return splits[0] + else: + return string + + +def fix_sqrt(string): + if "\\sqrt" not in string: + return string + splits = string.split("\\sqrt") + new_string = splits[0] + for split in splits[1:]: + if split[0] != "{": + a = split[0] + new_substr = "\\sqrt{" + a + "}" + split[1:] + else: + new_substr = "\\sqrt" + split + new_string += new_substr + return new_string + + +def strip_string(string): + # linebreaks + string = string.replace("\n", "") + + # remove inverse spaces + string = string.replace("\\!", "") + + # replace \\ with \ + string = string.replace("\\\\", "\\") + + # replace tfrac and dfrac with frac + string = string.replace("tfrac", "frac") + string = string.replace("dfrac", "frac") + + # remove \left and \right + string = string.replace("\\left", "") + string = string.replace("\\right", "") + + # Remove circ (degrees) + string = string.replace("^{\\circ}", "") + string = string.replace("^\\circ", "") + + # remove dollar signs + string = string.replace("\\$", "") + + # remove units (on the right) + string = remove_right_units(string) + + # remove percentage + string = string.replace("\\%", "") + string = string.replace("\%", "") # noqa: W605 + + # " 0." equivalent to " ." and "{0." equivalent to "{." Alternatively, add "0" if "." is the start of the string + string = string.replace(" .", " 0.") + string = string.replace("{.", "{0.") + # if empty, return empty string + if len(string) == 0: + return string + if string[0] == ".": + string = "0" + string + + # to consider: get rid of e.g. "k = " or "q = " at beginning + if len(string.split("=")) == 2: + if len(string.split("=")[0]) <= 2: + string = string.split("=")[1] + + # fix sqrt3 --> sqrt{3} + string = fix_sqrt(string) + + # remove spaces + string = string.replace(" ", "") + + # \frac1b or \frac12 --> \frac{1}{b} and \frac{1}{2}, etc. Even works with \frac1{72} (but not \frac{72}1). Also does a/b --> \\frac{a}{b} + string = fix_fracs(string) + + # manually change 0.5 --> \frac{1}{2} + if string == "0.5": + string = "\\frac{1}{2}" + + # NOTE: X/Y changed to \frac{X}{Y} in dataset, but in simple cases fix in case the model output is X/Y + string = fix_a_slash_b(string) + + return string diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/__init__.py b/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..49b0d30f57c3a33be08d63af8642575f23661dfe --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/__init__.py @@ -0,0 +1,73 @@ +# Copyright 2024 PRIME team and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .utils import check_correctness as apps_check_correctness +import json +import re +import traceback + + +def compute_score(completion, test_cases, continuous=False): + # try to get code solution from completion. if the completion is pure code, this will not take effect. + solution = completion.split('```python')[-1].split('```')[0] + try: + try: + if not isinstance(test_cases, dict): + test_cases = json.loads(test_cases) + except Exception as e: + print(f"Error:{e}") + + # Complete check on all in-out pairs first. If there is no failure, per-sample test can be skipped. + try: + res, metadata = apps_check_correctness(in_outs=test_cases, generation=solution, timeout=5, debug=False) + metadata = dict(enumerate(metadata))[0] + success = all(map(lambda x: x == True, res)) + if success: + return success, metadata + except Exception as e: + pass + + test_cases_list = [] + inputs = test_cases["inputs"] + outputs = test_cases["outputs"] + for i in range(len(inputs)): + test_cases_list.append({"inputs": [inputs[i]], "outputs": [outputs[i]]}) + + if continuous: + # per sample test: if continuous score is needed, test first 10 samples regardless of failures + # do not test all samples cuz some problems have enormous test cases + metadata_list = [] + res_list = [] + for test_case_id, test_case in enumerate(test_cases_list): + res, metadata = apps_check_correctness(in_outs=test_case, generation=solution, timeout=5, debug=False) + try: + metadata = dict(enumerate(metadata))[0] # metadata can be empty occasionally + except Exception as e: + metadata = {} + metadata["test_case"] = {} + metadata["test_case"]["input"] = str(test_case["inputs"][0]) + metadata["test_case"]["output"] = str(test_case["outputs"][0]) + metadata["test_case"]["res"] = str(res) + metadata_list.append(metadata) + res_list.extend(res) + + if test_case_id >= 9: + break + res_count = len(res_list) if len(res_list) > 0 else 1 + success = sum(map(lambda x: x == True, res_list)) / res_count + except Exception as e: + traceback.print_exc(10) + success = False + metadata_list = None + return success, metadata_list diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/testing_util.py b/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/testing_util.py new file mode 100644 index 0000000000000000000000000000000000000000..34845d07efa9f6e27540614f86a780ba09baedfa --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/testing_util.py @@ -0,0 +1,721 @@ +# Copyright 2024 PRIME team and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import ast +import json +import sys +import faulthandler +import platform + +# used for debugging to time steps +from datetime import datetime + +# to run the solution files we're using a timing based approach +import signal + +import numpy as np + +# for capturing the stdout +from io import StringIO + +# used for testing the code that reads from input +from unittest.mock import patch, mock_open + +from pyext import RuntimeModule + +from enum import Enum + +import traceback + + +def truncatefn(s, length=300): + assert isinstance(s, str) + if len(s) <= length: + return s + + return s[:length // 2] + "...(truncated) ..." + s[-length // 2:] + + +class CODE_TYPE(Enum): + call_based = 0 + standard_input = 1 + + +# stuff for setting up signal timer +class TimeoutException(Exception): + pass + + +def timeout_handler(signum, frame): + print("alarm went off") + return + # raise TimeoutException # this is an unhandled exception. just return None is OK + + +signal.signal(signal.SIGALRM, timeout_handler) + +# timeout = 6 # seconds + + +# used to capture stdout as a list +# from https://stackoverflow.com/a/16571630/6416660 +# alternative use redirect_stdout() from contextlib +class Capturing(list): + + def __enter__(self): + self._stdout = sys.stdout + sys.stdout = self._stringio = StringIO() + # Make closing the StringIO a no-op + self._stringio.close = lambda x: 1 + return self + + def __exit__(self, *args): + self.append(self._stringio.getvalue()) + del self._stringio # free up some memory + sys.stdout = self._stdout + + +def only_int_check(val): + return isinstance(val, int) + + +def string_int_check(val): + return isinstance(val, str) and val.isdigit() + + +def combined_int_check(val): + return only_int_check(val) or string_int_check(val) + + +def clean_traceback(error_traceback): + file_start = error_traceback.find('File \"\"') + # print(file_start) + error_traceback = "Traceback (most recent call last):\n " + error_traceback[file_start:] + return error_traceback + + +def run_test(in_outs, test=None, debug=False, timeout=15): + """ + if test(generated_code) is not None it'll try to run the code. + otherwise it'll just return an input and output pair. + """ + # Disable functionalities that can make destructive changes to the test. + reliability_guard() + + if debug: + print(f"start = {datetime.now().time()}") + + if in_outs: + if in_outs.get("fn_name") is None: + which_type = CODE_TYPE.standard_input # Standard input + method_name = None + else: + which_type = CODE_TYPE.call_based # Call-based + method_name = in_outs["fn_name"] + + if debug: + print(f"loaded input_output = {datetime.now().time()}") + + if test is None: + assert False, "should not happen: test code is none" + return in_outs, {"error": "no test code provided"} + elif test is not None: + results = [] + sol = "from string import *\nfrom re import *\nfrom datetime import *\nfrom collections import *\nfrom heapq import *\nfrom bisect import *\nfrom copy import *\nfrom math import *\nfrom random import *\nfrom statistics import *\nfrom itertools import *\nfrom functools import *\nfrom operator import *\nfrom io import *\nfrom sys import *\nfrom json import *\nfrom builtins import *\nfrom typing import *\nimport string\nimport re\nimport datetime\nimport collections\nimport heapq\nimport bisect\nimport copy\nimport math\nimport random\nimport statistics\nimport itertools\nimport functools\nimport operator\nimport io\nimport sys\nimport json\nsys.setrecursionlimit(6*10**5)\n" + if debug: + print(f"loading test code = {datetime.now().time()}") + + if which_type == CODE_TYPE.call_based: + + sol += test + if debug: + print(f"sol = {sol}") + signal.alarm(timeout) + try: + tmp_sol = RuntimeModule.from_string("tmp_sol", "", sol) + if "class Solution" not in test: + tmp = tmp_sol + else: + tmp = tmp_sol.Solution() + signal.alarm(0) + except Exception as e: + signal.alarm(0) + error_traceback = traceback.format_exc() + if debug: + print(f"type 0 compilation error = {e}") + results.append(-2) + return results, { + "error": repr(e), + # "error_code": -1, + # "error_message": "Compilation Error", + "traceback": clean_traceback(error_traceback), + } + signal.alarm(0) + + elif which_type == CODE_TYPE.standard_input: + # sol + # if code has if __name__ == "__main__": then remove it + try: + astree = ast.parse(test) + last_block = astree.body[-1] + if isinstance(last_block, ast.If): + condition = last_block.test + if ast.unparse(condition).strip() == "__name__ == '__main__'": + test = (ast.unparse(astree.body[:-1]) + "\n" + ast.unparse(last_block.body)) + except: + pass + + tmp_test = test.split("\n") + + new_test = [] + for x in tmp_test: + if (not x.startswith("from ")) and (not x.startswith("import ")): + new_test.append("\t" + x + "\n") + else: + new_test.append(x + "\n") + tmp_test = new_test + + new_test = "" + started = False + for i in tmp_test: + if i.startswith("\t") and not started: + new_test += "stdin = sys.stdin\nstdout = sys.stdout\n" + new_test += "def code():\n" + new_test += i + started = True + elif started and ((i.startswith("from ")) or (i.startswith("import "))): + new_test += "\t" + i + else: + new_test += i + tmp_test = new_test + + sol += tmp_test + if debug: + print(f"sol = {sol}") + method_name = "code" + signal.alarm(timeout) + try: + tmp_sol = RuntimeModule.from_string("tmp_sol", "", sol) + tmp = tmp_sol + signal.alarm(0) + except Exception as e: + signal.alarm(0) + error_traceback = traceback.format_exc() + if debug: + print(f"type 1 compilation error = {e}") + results.append(-2) + return results, { + "error": repr(e), + # "error_code": -1, + # "error_message": "Compilation Error", + "traceback": clean_traceback(error_traceback), + } + signal.alarm(0) + if debug: + print(f"get method = {datetime.now().time()}") + + try: + method = getattr(tmp, method_name) # get_attr second arg must be str + except: + signal.alarm(0) + error_traceback = traceback.format_exc() + e = sys.exc_info() + print(f"unable to get function error = {e}") + results.append(-2) + return results, { + "error": repr(e), + # "error_code": -1, + # "error_message": "Unable to extract code", + "traceback": clean_traceback(error_traceback), + } + + for index, inputs in enumerate(in_outs["inputs"]): + raw_inputs = inputs + raw_outputs = in_outs["outputs"][index] + if which_type == CODE_TYPE.call_based: + inputs = [json.loads(line) for line in inputs.split("\n")] + in_outs["outputs"][index] = json.loads(in_outs["outputs"][index]) + + truncate_line_size = 300 // (raw_inputs.count("\n") + 1) + raw_inputs = "\n".join( + [truncatefn(line, truncate_line_size) for line in raw_inputs.strip().split("\n")]) + raw_outputs = truncatefn(raw_outputs, 200) + else: + raw_inputs = truncatefn(raw_inputs) + raw_outputs = truncatefn(raw_outputs, 200) + # JSON forces dictionaries to have string keys; this undoes this (assuming a singleton list) + try: + if isinstance(inputs[0], dict): + inputs = [{int(k): v for k, v in inputs[0].items()}] + except: + True + try: + if isinstance(in_outs["outputs"][index], dict): + in_outs["outputs"][index] = [{int(k): v for k, v in in_outs["outputs"][index].items()}] + except: + True + try: + if isinstance(in_outs["outputs"][index][0], dict): + in_outs["outputs"][index] = [{int(k): v for k, v in in_outs["outputs"][index][0].items()}] + except: + True + + if debug: + print( + f"time: {datetime.now().time()} testing index = {index} inputs = {inputs}, {type(inputs)}. type = {which_type}" + ) + if which_type == CODE_TYPE.call_based: # Call-based + signal.alarm(timeout) + faulthandler.enable() + try: + output = method(*inputs) + raw_true_output = output + + raw_true_output_copy = json.dumps(output) + raw_true_output_copy = truncatefn(raw_true_output_copy, 200) + + # ground truth sequences are not tuples + if isinstance(output, tuple): + output = list(output) + + tmp_result = output == in_outs["outputs"][index] + if (isinstance(in_outs["outputs"][index], list) and in_outs["outputs"][index]): + tmp_result = tmp_result or (output == in_outs["outputs"][index][0]) + + # ground truth sequences are not tuples + try: + if isinstance(output[0], tuple): + tmp_result = tmp_result or ([list(x) for x in output] == in_outs["outputs"][index][0]) + except: + True + results.append(tmp_result) + if tmp_result != True: + return results, { + "output": raw_true_output_copy, + "expected": raw_outputs, + "inputs": raw_inputs, + # "error_code": -2, + "error_message": "Wrong Answer", + } + # reset the alarm + signal.alarm(0) + except Exception as e: + signal.alarm(0) + error_traceback = traceback.format_exc() + faulthandler.disable() + if debug: + print(f"Standard input runtime error or time limit exceeded error = {e}") + results.append(-1) + if "timeoutexception" in repr(e).lower(): + return results, { + "error": repr(e), + # "error_code": -3, + # "error_message": "Time Limit Exceeded", + # "inputs": raw_inputs, + # "expected": raw_outputs, + "traceback": clean_traceback(error_traceback), + } + else: + return results, { + "error": repr(e), + # "error_code": -4, + # "error_message": "Runtime Error", + # "inputs": raw_inputs, + # "expected": raw_outputs, + "traceback": clean_traceback(error_traceback), + } + faulthandler.disable() + signal.alarm(0) + if debug: + print( + f"outputs = {output}, test outputs = {in_outs['outputs'][index]}, inputs = {inputs}, {type(inputs)}, {output == [in_outs['outputs'][index]]}" + ) + elif which_type == CODE_TYPE.standard_input: # Standard input + faulthandler.enable() + passed = False + + if isinstance(inputs, list): + inputs = "\n".join(inputs) + if isinstance(in_outs["outputs"][index], list): + in_outs["outputs"][index] = "\n".join(in_outs["outputs"][index]) + + signal.alarm(timeout) + with Capturing() as output: + try: + call_method(method, inputs) + # reset the alarm + signal.alarm(0) + passed = True + except Exception as e: + # runtime error or took too long + signal.alarm(0) + error_traceback = traceback.format_exc() + print(f"Call-based runtime error or time limit exceeded error = {repr(e)}{e}") + results.append(-1) + if "timeoutexception" in repr(e).lower(): + return results, { + "error": repr(e), + # "error_code": -3, + # "error_message": "Time Limit Exceeded", + # "inputs": raw_inputs, + # "expected": raw_outputs, + "traceback": clean_traceback(error_traceback), + } + else: + return results, { + "error": repr(e), + # "error_code": -4, + # "error_message": "Runtime Error", + # "inputs": raw_inputs, + # "expected": raw_outputs, + "traceback": clean_traceback(error_traceback), + } + signal.alarm(0) + raw_true_output = output[0] + raw_true_output_copy = truncatefn(raw_true_output, 200) + output = raw_true_output.splitlines() + if not passed: + if debug: + nl = "\n" + if not isinstance(inputs, list): + print( + f"not passed output = {output}, test outputs = {in_outs['outputs'][index]}, inputs = {inputs.replace(nl,' new-line ')}, {type(inputs)}, {output == [in_outs['outputs'][index]]}" + ) + else: + print( + f"not passed output = {output}, test outputs = {in_outs['outputs'][index]}, inputs = {inputs}, {type(inputs)}, {output == [in_outs['outputs'][index]]}" + ) + continue + + if passed and debug: + print(f"==> output = {output}, test outputs = {in_outs['outputs'][index]}") + + if custom_compare_(output, in_outs["outputs"][index]): + tmp_result = True + results.append(tmp_result) + continue + + # ground truth sequences are expressed as lists not tuples + if isinstance(output, tuple): + output = list(output) + + tmp_result = False + try: + tmp_result = output == [in_outs["outputs"][index]] + if isinstance(in_outs["outputs"][index], list): + tmp_result = tmp_result or (output == in_outs["outputs"][index]) + if isinstance(output[0], str): + tmp_result = tmp_result or ([e.strip() for e in output] == in_outs["outputs"][index]) + except Exception as e: + if debug: + print(f"Failed check1 exception = {e}") + pass + + if tmp_result == True: + results.append(tmp_result) + continue + + # try one more time without \n + if isinstance(in_outs["outputs"][index], list): + for tmp_index, i in enumerate(in_outs["outputs"][index]): + in_outs["outputs"][index][tmp_index] = i.split("\n") + in_outs["outputs"][index][tmp_index] = [ + x.strip() for x in in_outs["outputs"][index][tmp_index] if x + ] + else: + in_outs["outputs"][index] = in_outs["outputs"][index].split("\n") + in_outs["outputs"][index] = list(filter(len, in_outs["outputs"][index])) + in_outs["outputs"][index] = list(map(lambda x: x.strip(), in_outs["outputs"][index])) + + try: + tmp_result = output == [in_outs["outputs"][index]] + if isinstance(in_outs["outputs"][index], list): + tmp_result = tmp_result or (output == in_outs["outputs"][index]) + except Exception as e: + if debug: + print(f"Failed check2 exception = {e}") + pass + + if tmp_result == True: + results.append(tmp_result) + continue + + # try by converting the output into a split up list too + if isinstance(output, list): + output = list(filter(len, output)) + + if debug: + nl = "\n" + if not isinstance(inputs, list): + print( + f"@1 output = {output}, test outputs = {in_outs['outputs'][index]}, inputs = {inputs.replace(nl,' new-line ')}, {type(inputs)}, {output == [in_outs['outputs'][index]]} {tmp_result=}" + ) + else: + print( + f"@1 output = {output}, test outputs = {in_outs['outputs'][index]}, inputs = {inputs}, {type(inputs)}, {output == [in_outs['outputs'][index]]} {tmp_result=}" + ) + + if tmp_result == True: + results.append(tmp_result) + continue + + if debug: + print(f"{tmp_result=} @a") + + try: + tmp_result = output == [in_outs["outputs"][index]] + if isinstance(in_outs["outputs"][index], list): + tmp_result = tmp_result or (output == in_outs["outputs"][index]) + except Exception as e: + if debug: + print(f"Failed check3 exception = {e}") + pass + + if debug: + print(f"{tmp_result=} @b") + + try: + all_ints = all( + combined_int_check(e1) and combined_int_check(e2) + for e1, e2 in zip(output, in_outs["outputs"][index])) + if not all_ints: + if debug: + print([ + combined_int_check(e1) and combined_int_check(e2) + for e1, e2 in zip(output, in_outs["outputs"][index]) + ]) + output_float = [float(e) for e in output] + gt_float = [float(e) for e in in_outs["outputs"][index]] + tmp_result = tmp_result or ((len(output_float) == len(gt_float)) and + np.allclose(output_float, gt_float)) + except Exception as e: + pass + + if debug: + print(f"{tmp_result=} @c") + + try: + if isinstance(output[0], list): + all_ints = all( + combined_int_check(e1) and combined_int_check(e2) + for e1, e2 in zip(output[0], in_outs["outputs"][index])) + if not all_ints: + output_float = [float(e) for e in output[0]] + gt_float = [float(e) for e in in_outs["outputs"][index][0]] + tmp_result = tmp_result or ((len(output_float) == len(gt_float)) and + np.allclose(output_float, gt_float)) + except Exception as e: + pass + + if tmp_result == True: + results.append(tmp_result) + continue + + if debug: + print(f"{tmp_result=} @d") + # try by converting the stuff into split up list + if isinstance(in_outs["outputs"][index], list): + for tmp_index, i in enumerate(in_outs["outputs"][index]): + in_outs["outputs"][index][tmp_index] = set(i.split()) + else: + in_outs["outputs"][index] = set(in_outs["outputs"][index].split()) + + if debug: + print(f"{tmp_result=} @e") + + try: + tmp_result = output == in_outs["outputs"][index] + except Exception as e: + if debug: + print(f"Failed check4 exception = {e}") + continue + + if tmp_result == True: + results.append(tmp_result) + continue + + if debug: + print(f"{tmp_result=} @f") + + # try by converting the output into a split up list too + if isinstance(output, list): + for tmp_index, i in enumerate(output): + output[tmp_index] = i.split() + output = list(filter(len, output)) + for tmp_index, i in enumerate(output): + output[tmp_index] = set(i) + else: + output = output.split() + output = list(filter(len, output)) + output = set(output) + + if debug: + print(f"{tmp_result=} @g") + + if tmp_result == True and debug: + print("PASSED") + + results.append(tmp_result) + if tmp_result != True: + return results, { + "output": raw_true_output_copy, + "expected": raw_outputs, + "inputs": raw_inputs, + # "error_code": -2, + "error_message": "Wrong Answer", + } + + if debug: + nl = "\n" + if not isinstance(inputs, list): + print( + f"@2 output = {output}, test outputs = {in_outs['outputs'][index]}, inputs = {inputs.replace(nl,' new-line ')}, {type(inputs)}, {output == [in_outs['outputs'][index]]}" + ) + else: + print( + f"@2 output = {output}, test outputs = {in_outs['outputs'][index]}, inputs = {inputs}, {type(inputs)}, {output == [in_outs['outputs'][index]]}" + ) + + print(f"results = {results}") + + return results, {} + + +def custom_compare_(output, ground_truth): + + if isinstance(output, list): + output_1 = "\n".join(output) + if stripped_string_compare(output_1, ground_truth): + return True + + if isinstance(output, list): + output_2 = [o.lstrip().rstrip() for o in output] + output_2 = "\n".join(output_2) + if stripped_string_compare(output_2, ground_truth): + return True + + return False + + +def stripped_string_compare(s1, s2): + s1 = s1.lstrip().rstrip() + s2 = s2.lstrip().rstrip() + return s1 == s2 + + +def call_method(method, inputs): + + if isinstance(inputs, list): + inputs = "\n".join(inputs) + + inputs_line_iterator = iter(inputs.split("\n")) + + # sys.setrecursionlimit(10000) + + # @patch('builtins.input', side_effect=inputs.split("\n")) + @patch("builtins.open", mock_open(read_data=inputs)) + @patch("sys.stdin", StringIO(inputs)) + @patch("sys.stdin.readline", lambda *args: next(inputs_line_iterator)) + @patch("sys.stdin.readlines", lambda *args: inputs.split("\n")) + @patch("sys.stdin.read", lambda *args: inputs) + # @patch('sys.stdout.write', print) + def _inner_call_method(_method): + try: + return _method() + except SystemExit as e: + pass + finally: + pass + + return _inner_call_method(method) + + +def reliability_guard(maximum_memory_bytes=None): + """ + This disables various destructive functions and prevents the generated code + from interfering with the test (e.g. fork bomb, killing other processes, + removing filesystem files, etc.) + WARNING + This function is NOT a security sandbox. Untrusted code, including, model- + generated code, should not be blindly executed outside of one. See the + Codex paper for more information about OpenAI's code sandbox, and proceed + with caution. + """ + + if maximum_memory_bytes is not None: + import resource + + resource.setrlimit(resource.RLIMIT_AS, (maximum_memory_bytes, maximum_memory_bytes)) + resource.setrlimit(resource.RLIMIT_DATA, (maximum_memory_bytes, maximum_memory_bytes)) + if not platform.uname().system == "Darwin": + resource.setrlimit(resource.RLIMIT_STACK, (maximum_memory_bytes, maximum_memory_bytes)) + + faulthandler.disable() + + import builtins + + builtins.exit = None + builtins.quit = None + + import os + + os.environ["OMP_NUM_THREADS"] = "1" + + os.kill = None + os.system = None # 防止干扰repl评测 + os.putenv = None + os.remove = None + os.removedirs = None + os.rmdir = None + os.fchdir = None + os.setuid = None + os.fork = None + os.forkpty = None + os.killpg = None + os.rename = None + os.renames = None + os.truncate = None + os.replace = None + os.unlink = None + os.fchmod = None + os.fchown = None + os.chmod = None + os.chown = None + os.chroot = None + os.fchdir = None + os.lchflags = None + os.lchmod = None + os.lchown = None + os.getcwd = None + os.chdir = None + + import shutil + + shutil.rmtree = None + shutil.move = None + shutil.chown = None + + import subprocess + + subprocess.Popen = None # type: ignore + + __builtins__["help"] = None + + import sys + + sys.modules["ipdb"] = None + sys.modules["joblib"] = None + sys.modules["resource"] = None + sys.modules["psutil"] = None + sys.modules["tkinter"] = None diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/utils.py b/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..07f3db88e0caae09ab06e7f74654fb788cd86f24 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/prime_code/utils.py @@ -0,0 +1,59 @@ +# Copyright 2024 PRIME team and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Borrowed from: https://huggingface.co/spaces/codeparrot/apps_metric/blob/main/utils.py + +import multiprocessing +from typing import Dict, Optional +from datasets import load_dataset +from .testing_util import run_test +import traceback +import os, sys + + +def _temp_run(sample, generation, debug, result, metadata_list, timeout): + with open(os.devnull, 'w') as devnull: + sys.stdout = devnull + sys.stderr = devnull + try: + res, metadata = run_test(in_outs=sample, test=generation, debug=debug, timeout=timeout) + result.append(res) + metadata_list.append(metadata) + except Exception as e: + # print(e) # some tracebacks are extremely long. + traceback.print_exc(10) + result.append([-1 for i in range(len(sample['inputs']))]) + metadata_list.append({}) + + +def check_correctness(in_outs: Optional[dict], generation, timeout=10, debug=True): + """Check correctness of code generation with a global timeout. + The global timeout is to catch some extreme/rare cases not handled by the timeouts + inside `run_test`""" + + manager = multiprocessing.Manager() + result = manager.list() + metadata_list = manager.list() + p = multiprocessing.Process(target=_temp_run, args=(in_outs, generation, debug, result, metadata_list, timeout)) + p.start() + p.join(timeout=timeout + 1) + if p.is_alive(): + p.kill() + # p.terminate() + if not result: + # consider that all tests failed + result = [[-1 for i in range(len(in_outs["inputs"]))]] + if debug: + print(f"global timeout") + return result[0], metadata_list diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/__init__.py b/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..39b2c831a246f35007c581de4de808ad2ceeddde --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/__init__.py @@ -0,0 +1,402 @@ +# Copyright 2024 PRIME team and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Answer checker API that uses sympy to simplify expressions and check for equality. + +Call grade_answer(given_answer: str, ground_truth: str). + +FROM: https://github.com/openai/prm800k/blob/main/prm800k/grading/grader.py +""" +import re +import sympy +from pylatexenc import latex2text +from sympy.parsing import sympy_parser + +from . import math_normalize +from .grader import math_equal + +# import math_normalize +# from grader import math_equal + +# sympy might hang -- we don't care about trying to be lenient in these cases +BAD_SUBSTRINGS = ["^{", "^("] +BAD_REGEXES = ["\^[0-9]+\^", "\^[0-9][0-9]+"] +TUPLE_CHARS = "()[]" + + +def _sympy_parse(expr: str): + """Parses an expression with sympy.""" + py_expr = expr.replace("^", "**") + return sympy_parser.parse_expr( + py_expr, + transformations=(sympy_parser.standard_transformations + (sympy_parser.implicit_multiplication_application,)), + ) + + +def _parse_latex(expr: str) -> str: + """Attempts to parse latex to an expression sympy can read.""" + expr = expr.replace("\\tfrac", "\\frac") + expr = expr.replace("\\dfrac", "\\frac") + expr = expr.replace("\\frac", " \\frac") # Play nice with mixed numbers. + expr = latex2text.LatexNodes2Text().latex_to_text(expr) + + # Replace the specific characters that this parser uses. + expr = expr.replace("√", "sqrt") + expr = expr.replace("π", "pi") + expr = expr.replace("∞", "inf") + expr = expr.replace("∪", "U") + expr = expr.replace("·", "*") + expr = expr.replace("×", "*") + + return expr.strip() + + +def _is_float(num: str) -> bool: + try: + float(num) + return True + except ValueError: + return False + + +def _is_int(x: float) -> bool: + try: + return abs(x - int(round(x))) <= 1e-7 + except: + return False + + +def _is_frac(expr: str) -> bool: + return bool(re.search(r"^-?[0-9]+.?/0*[1-9][0-9]*.?$", expr)) + + +def _str_is_int(x: str) -> bool: + try: + x = _strip_properly_formatted_commas(x) + x = float(x) + return abs(x - int(round(x))) <= 1e-7 + except: + return False + + +def _str_to_int(x: str) -> bool: + x = x.replace(",", "") + x = float(x) + return int(x) + + +def _inject_implicit_mixed_number(step: str): + """ + Automatically make a mixed number evalable + e.g. 7 3/4 => 7+3/4 + """ + p1 = re.compile("([0-9]) +([0-9])") + step = p1.sub("\\1+\\2", step) ## implicit mults + return step + + +def _strip_properly_formatted_commas(expr: str): + # We want to be careful because we don't want to strip tuple commas + p1 = re.compile("(\d)(,)(\d\d\d)($|\D)") + while True: + next_expr = p1.sub("\\1\\3\\4", expr) + if next_expr == expr: + break + expr = next_expr + return next_expr + + +def _normalize(expr: str) -> str: + """Normalize answer expressions.""" + if expr is None: + return None + + # Remove enclosing `\text{}`. + m = re.search("^\\\\text\{(?P.+?)\}$", expr) + if m is not None: + expr = m.group("text") + + expr = expr.replace("\\%", "%") + expr = expr.replace("\\$", "$") + expr = expr.replace("$", "") + expr = expr.replace("%", "") + expr = expr.replace(" or ", " , ") + expr = expr.replace(" and ", " , ") + + expr = expr.replace("million", "*10^6") + expr = expr.replace("billion", "*10^9") + expr = expr.replace("trillion", "*10^12") + + for unit in [ + "degree", + "cm", + "centimeter", + "meter", + "mile", + "second", + "minute", + "hour", + "day", + "week", + "month", + "year", + "foot", + "feet", + "inch", + "yard", + "liter", + ]: + expr = re.sub(f"{unit}(es)?(s)? *(\^[0-9]+)?", "", expr) + expr = re.sub(f"\^ *\\\\circ", "", expr) + + if len(expr) > 0 and expr[0] == "{" and expr[-1] == "}": + expr = expr[1:-1] + + expr = re.sub(",\\\\! *", "", expr) + if _is_float(expr) and _is_int(float(expr)): + expr = str(int(round(float(expr)))) + if "\\" in expr: + try: + expr = _parse_latex(expr) + except: + pass + + # edge case with mixed numbers and negative signs + expr = re.sub("- *", "-", expr) + + expr = _inject_implicit_mixed_number(expr) + + # don't be case sensitive for text answers + expr = expr.lower() + + if _str_is_int(expr): + expr = str(_str_to_int(expr)) + + return expr + + +def count_unknown_letters_in_expr(expr: str): + expr = expr.replace("sqrt", "") + expr = expr.replace("frac", "") + letters_in_expr = set([x for x in expr if x.isalpha()]) + return len(letters_in_expr) + + +def should_allow_eval(expr: str): + # we don't want to try parsing unknown text or functions of more than two variables + if count_unknown_letters_in_expr(expr) > 2: + return False + + for bad_string in BAD_SUBSTRINGS: + if bad_string in expr: + return False + + for bad_regex in BAD_REGEXES: + if re.search(bad_regex, expr) is not None: + return False + + return True + + +def are_equal_under_sympy(ground_truth_normalized: str, given_normalized: str): + are_equal = False + try: + expr = f"({ground_truth_normalized})-({given_normalized})" + if should_allow_eval(expr): + sympy_diff = _sympy_parse(expr) + simplified = sympy.simplify(sympy_diff) + if simplified == 0: + are_equal = True + except: + pass + return are_equal + + +def split_tuple(expr: str): + """ + Split the elements in a tuple/interval, while handling well-formatted commas in large numbers + """ + expr = _strip_properly_formatted_commas(expr) + if len(expr) == 0: + return [] + if (len(expr) > 2 and expr[0] in TUPLE_CHARS and expr[-1] in TUPLE_CHARS and + all([ch not in expr[1:-1] for ch in TUPLE_CHARS])): + elems = [elem.strip() for elem in expr[1:-1].split(",")] + else: + elems = [expr] + return elems + + +def grade_answer(given_answer: str, ground_truth: str) -> bool: + """ + The answer will be considered correct if: + (a) it normalizes to the same string as the ground truth answer + OR + (b) sympy can simplify the difference between the expressions to 0 + """ + if given_answer is None: + return False + + ground_truth_normalized_mathd = math_normalize.normalize_answer(ground_truth) + given_answer_normalized_mathd = math_normalize.normalize_answer(given_answer) + + # be at least as lenient as mathd + if ground_truth_normalized_mathd == given_answer_normalized_mathd: + return True + + ground_truth_normalized = _normalize(ground_truth) + given_normalized = _normalize(given_answer) + + if ground_truth_normalized is None: + return False + + if ground_truth_normalized == given_normalized: + return True + + if len(given_normalized) == 0: + return False + + ground_truth_elems = split_tuple(ground_truth_normalized) + given_elems = split_tuple(given_normalized) + + if len(ground_truth_elems) > 1 and (ground_truth_normalized[0] != given_normalized[0] or + ground_truth_normalized[-1] != given_normalized[-1]): + is_correct = False + elif len(ground_truth_elems) != len(given_elems): + is_correct = False + else: + for ground_truth_elem, given_elem in zip(ground_truth_elems, given_elems): + if _is_frac(ground_truth_elem) and _is_frac(given_elem): + # if fractions aren't reduced, then shouldn't be marked as correct + # so, we don't want to allow sympy.simplify in this case + is_correct = ground_truth_elem == given_elem + elif _str_is_int(ground_truth_elem) != _str_is_int(given_elem): + # if the ground truth answer is an integer, we require the given answer to be a strict match (no sympy.simplify) + is_correct = False + else: + is_correct = are_equal_under_sympy(ground_truth_elem, given_elem) + if not is_correct: + break + + return is_correct + + +def remove_boxed(s): + left = "\\boxed{" + try: + assert s[:len(left)] == left + assert s[-1] == "}" + return s[len(left):-1] + except: + return None + + +def _last_boxed_only_string(string): + idx = string.rfind("\\boxed") + if idx < 0: + idx = string.rfind("\\fbox") + if idx < 0: + return None + + i = idx + left_brace_idx = None + right_brace_idx = None + num_left_braces_open = 0 + while i < len(string): + if string[i] == "{": + num_left_braces_open += 1 + if left_brace_idx is None: + left_brace_idx = i + elif string[i] == "}": + num_left_braces_open -= 1 + if num_left_braces_open == 0: + right_brace_idx = i + break + + i += 1 + + if left_brace_idx is None or right_brace_idx is None: + return None + + return string[left_brace_idx + 1:right_brace_idx].strip() + + +def match_answer(response): + is_matched = False + for ans_marker in ['answer:', "answer is", "answers are"]: + ans_idx = response.lower().rfind(ans_marker) + if ans_idx != -1: + is_matched = True + response = response[ans_idx + len(ans_marker):].strip() + if response.endswith("\n"): + response = response[:-2] + + for ans_marker in ["is answer", "is the answer", "are answers", "are the answers"]: + ans_idx = response.lower().rfind(ans_marker) + if ans_idx != -1: + is_matched = True + response = response[:ans_idx].strip() + if response.endswith("\n"): + response = response[:-2] + + # Find boxed + ans_boxed = _last_boxed_only_string(response) + if ans_boxed: + is_matched = True + response = ans_boxed + + if ". " in response: + dot_idx = response.lower().rfind(". ") + if dot_idx != -1: + response = response[:dot_idx].strip() + + for ans_marker in ['be ', "is ", "are ", "=", ": ", "get ", 'be\n', "is\n", "are\n", ":\n", "get\n"]: + ans_idx = response.lower().rfind(ans_marker) + if ans_idx != -1: + is_matched = True + response = response[ans_idx + len(ans_marker):].strip() + if response.endswith("\n"): + response = response[:-2] + + is_matched = is_matched if any([c.isdigit() for c in response]) else False # answer must have a digit + # Grade + return is_matched, response + + +import math + + +def compute_score(model_output: str, ground_truth: str) -> bool: + model_output = str(model_output) + ground_truth = str(ground_truth) + + is_matched, extracted_model_output = match_answer(model_output) + format_correctness = "Step 2:" in model_output and "\\box" in model_output + + # grade simple algebra questions. if succeeded, return; otherwise, proceed to more complex grading + if grade_answer(extracted_model_output, ground_truth): + return True, True, extracted_model_output + + try: + if "\pi" in extracted_model_output or "\pi" in ground_truth: + equivs = [] + for pi in [math.pi, 3.14]: + equivs.append(math_equal(extracted_model_output, ground_truth, timeout=True, pi=pi)) + is_correct = any(equivs) + else: + is_correct = math_equal(extracted_model_output, ground_truth, timeout=True) + except: + is_correct = False + + return is_correct, format_correctness, extracted_model_output diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/grader.py b/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/grader.py new file mode 100644 index 0000000000000000000000000000000000000000..e3e1686f156a5c8148874e77405b249d975e8c5b --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/grader.py @@ -0,0 +1,380 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Copyright (c) Microsoft Corporation. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE + +# Copyright (c) 2023 OpenAI +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +# Copyright (c) 2021 Dan Hendrycks +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +# Copyright 2024 PRIME team and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +This logic is largely copied from the Hendrycks' MATH release (math_equivalence), and borrowed from: +- https://github.com/microsoft/ToRA/blob/main/src/eval/grader.py +- https://github.com/microsoft/ProphetNet/tree/master/CRITIC +- https://github.com/openai/prm800k +""" + +import contextlib +import re +import signal +import math +from math import isclose +from typing import Union + +from sympy import N, simplify +from sympy.parsing.latex import parse_latex +from sympy.parsing.sympy_parser import parse_expr + + +def is_digit(s): + try: + if "{,}" in str(s): + num = float(str(s).replace("{,}", "")) + return True, num + + num = float(str(s).replace(",", "")) + return True, num + except ValueError: + return False, None + + +def normalize(answer, pi) -> str: + # checking if answer is $ and removing $ in that case to compare + if isinstance(answer, str) and bool(re.match(r'\$\d+(\.\d+)?', answer)): + return answer[1:] + + # checking if answer is % or \\% and removing % + if isinstance(answer, str) and (bool(re.match(r'^\d+(\.\d+)?%$', answer)) or + bool(re.match(r'^\d+(\.\d+)?\\%$', answer))): + return answer.replace("\\%", "").replace("%", "") + + # handle base + answer = handle_base(answer) + + # handle pi + answer = handle_pi(answer, pi) + + return answer + + +def handle_base(x) -> str: + if isinstance(x, str) and "_" in x: + # Due to base + x = x.split("_")[0] + x = float(x) + return int(x) + return x + + +def handle_pi(string, pi): + if isinstance(string, str) and "\pi" in string: + # Find the first occurrence of "\pi" + idx = string.find("\pi") + + # Iterate over the string and find all occurrences of "\pi" with a valid previous character + while idx != -1: + + if idx > 0 and string[idx - 1].isdigit(): + # Replace "\pi" with "*math.pi" if the previous character is a digit + string = string[:idx] + f"*{pi}" + string[idx + 3:] + else: + # Replace "\pi" with "1*math.pi" if the previous character is not a digit + string = string[:idx] + f"1*{pi}" + string[idx + 3:] + + # Find the next occurrence of "\pi" + idx = string.find("\pi", idx + 1) + + # Evaluate the expression using eval() function + try: + string = eval(string) + except: + pass + + return string + + +def math_equal(prediction: Union[bool, float, str], + reference: Union[float, str], + include_percentage: bool = True, + tolerance: float = 1e-4, + timeout: float = 10.0, + pi: float = math.pi) -> bool: + """ + Exact match of math if and only if: + 1. numerical equal: both can convert to float and are equal + 2. symbolic equal: both can convert to sympy expression and are equal + """ + + prediction = normalize(prediction, pi) + reference = normalize(reference, pi) + + if isinstance(prediction, str) and len(prediction) > 1000: # handling weird corner-cases + prediction = prediction[:1000] + + # 0. string comparison + if isinstance(prediction, str) and isinstance(reference, str): + if prediction.strip().lower() == reference.strip().lower(): + return True + if prediction.replace(" ", "") == reference.replace(" ", ""): + return True + + try: # 1. numerical equal + if is_digit(prediction)[0] and is_digit(reference)[0]: + prediction = is_digit(prediction)[1] + reference = is_digit(reference)[1] + # number questions + if include_percentage: + gt_result = [reference / 100, reference, reference * 100] + else: + gt_result = [reference] + for item in gt_result: + try: + if isclose(item, prediction, rel_tol=tolerance): + return True + except Exception: + continue + return False + except Exception: + pass + + if not prediction and prediction not in [0, False]: + return False + + # 2. symbolic equal + reference = str(reference).strip() + prediction = str(prediction).strip() + + ## deal with [], (), {} + prediction = format_intervals(prediction) + + pred_str, ref_str = prediction, reference + if (prediction.startswith("[") and prediction.endswith("]") and + not reference.startswith("(")) or (prediction.startswith("(") and prediction.endswith(")") and + not reference.startswith("[")): + pred_str = pred_str.strip("[]()") + ref_str = ref_str.strip("[]()") + for s in ["{", "}", "(", ")"]: + ref_str = ref_str.replace(s, "") + pred_str = pred_str.replace(s, "") + if pred_str == ref_str: + return True + + ## [a, b] vs. [c, d], return a==c and b==d + if (prediction and reference and prediction[0] in "([" and prediction[-1] in ")]" and + prediction[0] == reference[0] and prediction[-1] == reference[-1]): + pred_parts = prediction[1:-1].split(",") + ref_parts = reference[1:-1].split(",") + if len(pred_parts) == len(ref_parts): + if all([ + math_equal(pred_pt, ref_pt, include_percentage, tolerance) + for pred_pt, ref_pt in zip(pred_parts, ref_parts) + ]): + return True + + if "," in prediction and "," in reference: + pred_parts = [item.strip() for item in prediction.split(",")] + ref_parts = [item.strip() for item in reference.split(",")] + + if len(pred_parts) == len(ref_parts): + if all([ + math_equal(pred_parts[i], ref_parts[i], include_percentage, tolerance) + for i in range(len(pred_parts)) + ]): + return True + else: + return False + + # if we have point == tuple of values + if prediction.startswith("Point") and reference[0] == "(" and reference[-1] == ")": + pred_parts = prediction[prediction.find("(") + 1:-1].split(",") + ref_parts = reference[1:-1].split(",") + if len(pred_parts) == len(ref_parts): + if all([ + math_equal(pred_pt, ref_pt, include_percentage, tolerance) + for pred_pt, ref_pt in zip(pred_parts, ref_parts) + ]): + return True + + # if reference is a matrix + if "\begin{pmatrix}" in reference and prediction.startswith("Matrix"): + try: + pred_matrix = parse_expr(prediction) + ref_matrix_items = reference.split()[1:-1:2] + if len(pred_matrix) == len(ref_matrix_items): + if all([ + math_equal(pred, ref, include_percentage, tolerance) + for ref, pred in zip(ref_matrix_items, pred_matrix) + ]): + return True + except Exception: + pass + elif "\begin{pmatrix}" in reference and prediction.startswith("[") and prediction.endswith("]"): + if isinstance(eval(prediction), list): + try: + pred_matrix = eval(prediction) + # ref_matrix_items = reference.split()[1:-1:2] + ref_matrix_items = reference.lstrip("\\begin{pmatrix}").lstrip("\begin{pmatrix}").rstrip( + "\\end{pmatrix}").rstrip("\end{pmatrix}") + ref_matrix_items = ref_matrix_items.split("\\") + ref_matrix_items = [row.split("&") if "&" in row else row for row in ref_matrix_items] + if len(pred_matrix) == len(ref_matrix_items): + if all([ + math_equal(pred, ref, include_percentage, tolerance) + for ref, pred in zip(ref_matrix_items, pred_matrix) + ]): + return True + except Exception: + pass + + return symbolic_equal(prediction, reference, tolerance, timeout) + + +def symbolic_equal(a, b, tolerance, timeout=10.0): + + def _parse(s): + for f in [parse_expr, parse_latex]: + try: + with time_limit(timeout): + return f(s) + except Exception: + pass + return s + + a = _parse(a) + b = _parse(b) + + try: + with time_limit(timeout): + if simplify(a - b) == 0: + return True + except Exception: + pass + + try: + with time_limit(timeout): + if isclose(N(a), N(b), rel_tol=tolerance): + return True + except Exception: + pass + return False + + +class TimeoutException(Exception): + pass + + +@contextlib.contextmanager +def time_limit(seconds: float): + + def signal_handler(signum, frame): + raise TimeoutException("Timed out!") + + signal.setitimer(signal.ITIMER_REAL, seconds) + signal.signal(signal.SIGALRM, signal_handler) + try: + yield + finally: + signal.setitimer(signal.ITIMER_REAL, 0) + + +def format_intervals(prediction): + patterns = { + "Interval(": r"^Interval\((.*)\)$", + "Interval.Ropen(": r"^Interval\.Ropen\((.*)\)$", + "Interval.Lopen(": r"^Interval\.Lopen\((.*)\)$", + "Interval.open(": r"^Interval\.open\((.*)\)$", + } + + for key, pattern in patterns.items(): + match = re.match(pattern, prediction) + if match: + inner_content = match.group(1) + + if key == "Interval(": # Intarval(a, b) == [a, b] + return f"[{inner_content}]" + elif key == "Interval.Ropen(": # Intarval.Ropen(a, b) == [a, b) + return f"[{inner_content})" + elif key == "Interval.Lopen(": # Intarval.Lopen(a, b) == (a, b] + return f"({inner_content}]" + elif key == "Interval.open(": # Intarval.open(a, b) == (a, b) + return f"({inner_content})" + + return prediction diff --git a/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/math_normalize.py b/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/math_normalize.py new file mode 100644 index 0000000000000000000000000000000000000000..e9921a5ad2e0742d06f27e1fe1ca1e845e1fc65b --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/reward_score/prime_math/math_normalize.py @@ -0,0 +1,191 @@ +# Copyright 2024 PRIME team and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Copyright (c) 2021 Dan Hendrycks +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. +""" +This logic is largely copied from the Hendrycks' MATH release (math_equivalence). + +From: https://github.com/openai/prm800k/blob/main/prm800k/grading/math_normalize.py +""" +import re +from typing import Optional + + +def normalize_answer(answer: Optional[str]) -> Optional[str]: + if answer is None: + return None + answer = answer.strip() + try: + # Remove enclosing `\text{}`. + m = re.search("^\\\\text\{(?P.+?)\}$", answer) + if m is not None: + answer = m.group("text").strip() + return _strip_string(answer) + except: + return answer + + +def _fix_fracs(string): + substrs = string.split("\\frac") + new_str = substrs[0] + if len(substrs) > 1: + substrs = substrs[1:] + for substr in substrs: + new_str += "\\frac" + if substr[0] == "{": + new_str += substr + else: + try: + assert len(substr) >= 2 + except: + return string + a = substr[0] + b = substr[1] + if b != "{": + if len(substr) > 2: + post_substr = substr[2:] + new_str += "{" + a + "}{" + b + "}" + post_substr + else: + new_str += "{" + a + "}{" + b + "}" + else: + if len(substr) > 2: + post_substr = substr[2:] + new_str += "{" + a + "}" + b + post_substr + else: + new_str += "{" + a + "}" + b + string = new_str + return string + + +def _fix_a_slash_b(string): + if len(string.split("/")) != 2: + return string + a = string.split("/")[0] + b = string.split("/")[1] + try: + a = int(a) + b = int(b) + assert string == "{}/{}".format(a, b) + new_string = "\\frac{" + str(a) + "}{" + str(b) + "}" + return new_string + except: + return string + + +def _remove_right_units(string): + # "\\text{ " only ever occurs (at least in the val set) when describing units + if "\\text{ " in string: + splits = string.split("\\text{ ") + assert len(splits) == 2 + return splits[0] + else: + return string + + +def _fix_sqrt(string): + if "\\sqrt" not in string: + return string + splits = string.split("\\sqrt") + new_string = splits[0] + for split in splits[1:]: + if split[0] != "{": + a = split[0] + new_substr = "\\sqrt{" + a + "}" + split[1:] + else: + new_substr = "\\sqrt" + split + new_string += new_substr + return new_string + + +def _strip_string(string): + # linebreaks + string = string.replace("\n", "") + + # remove inverse spaces + string = string.replace("\\!", "") + + # replace \\ with \ + string = string.replace("\\\\", "\\") + + # replace tfrac and dfrac with frac + string = string.replace("tfrac", "frac") + string = string.replace("dfrac", "frac") + + # remove \left and \right + string = string.replace("\\left", "") + string = string.replace("\\right", "") + + # Remove circ (degrees) + string = string.replace("^{\\circ}", "") + string = string.replace("^\\circ", "") + + # remove dollar signs + string = string.replace("\\$", "") + + # remove units (on the right) + string = _remove_right_units(string) + + # remove percentage + string = string.replace("\\%", "") + string = string.replace("\%", "") + + # " 0." equivalent to " ." and "{0." equivalent to "{." Alternatively, add "0" if "." is the start of the string + string = string.replace(" .", " 0.") + string = string.replace("{.", "{0.") + # if empty, return empty string + if len(string) == 0: + return string + if string[0] == ".": + string = "0" + string + + # to consider: get rid of e.g. "k = " or "q = " at beginning + if len(string.split("=")) == 2: + if len(string.split("=")[0]) <= 2: + string = string.split("=")[1] + + # fix sqrt3 --> sqrt{3} + string = _fix_sqrt(string) + + # remove spaces + string = string.replace(" ", "") + + # \frac1b or \frac12 --> \frac{1}{b} and \frac{1}{2}, etc. Even works with \frac1{72} (but not \frac{72}1). Also does a/b --> \\frac{a}{b} + string = _fix_fracs(string) + + # manually change 0.5 --> \frac{1}{2} + if string == "0.5": + string = "\\frac{1}{2}" + + # NOTE: X/Y changed to \frac{X}{Y} in dataset, but in simple cases fix in case the model output is X/Y + string = _fix_a_slash_b(string) + + return string \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/seqlen_balancing.py b/frameworks/agentgym_rl/verl/utils/seqlen_balancing.py new file mode 100644 index 0000000000000000000000000000000000000000..fee45da0d33264ea40591f95a98bdf35ef0ea4ad --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/seqlen_balancing.py @@ -0,0 +1,265 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import List, Tuple, Callable +import heapq + +import torch +from torch import distributed as dist + +from tensordict import TensorDict +import copy + + +def karmarkar_karp(seqlen_list: List[int], k_partitions: int, equal_size: bool): + # see: https://en.wikipedia.org/wiki/Largest_differencing_method + class Set: + + def __init__(self) -> None: + self.sum = 0 + self.items = [] + + def add(self, idx: int, val: int): + self.items.append((idx, val)) + self.sum += val + + def merge(self, other): + for idx, val in other.items: + self.items.append((idx, val)) + self.sum += val + + def __lt__(self, other): + if self.sum != other.sum: + return self.sum < other.sum + if len(self.items) != len(other.items): + return len(self.items) < len(other.items) + return self.items < other.items + + class State: + + def __init__(self, items: List[Tuple[int, int]], k: int) -> None: + self.k = k + # sets should always be decreasing order + self.sets = [Set() for _ in range(k)] + assert len(items) in [1, k], f"{len(items)} not in [1, {k}]" + for i, (idx, seqlen) in enumerate(items): + self.sets[i].add(idx=idx, val=seqlen) + self.sets = sorted(self.sets, reverse=True) + + def spread(self): + return self.sets[0].sum - self.sets[-1].sum + + def get_partitions(self): + partitions = [] + for i in range(len(self.sets)): + cur_partition = [] + for idx, _ in self.sets[i].items: + cur_partition.append(idx) + partitions.append(cur_partition) + return partitions + + def merge(self, other): + for i in range(self.k): + self.sets[i].merge(other.sets[self.k - 1 - i]) + self.sets = sorted(self.sets, reverse=True) + + @property + def spread(self) -> int: + return self.sets[0].sum - self.sets[-1].sum + + def __lt__(self, other): + # least heap, let the state with largest spread to be popped first, + # if the spread is the same, let the state who has the largest set + # to be popped first. + if self.spread != other.spread: + return self.spread > other.spread + return self.sets[0] > other.sets[0] + + def __repr__(self) -> str: + repr_str = "[" + for i in range(self.k): + if i > 0: + repr_str += "," + repr_str += "{" + for j, (_, seqlen) in enumerate(self.sets[i].items): + if j > 0: + repr_str += "," + repr_str += str(seqlen) + repr_str += "}" + repr_str += "]" + return repr_str + + sorted_seqlen_list = sorted([(seqlen, i) for i, seqlen in enumerate(seqlen_list)]) + states_pq = [] + if equal_size: + assert len(seqlen_list) % k_partitions == 0, f"{len(seqlen_list)} % {k_partitions} != 0" + for offset in range(0, len(sorted_seqlen_list), k_partitions): + items = [] + for i in range(k_partitions): + seqlen, idx = sorted_seqlen_list[offset + i] + items.append((idx, seqlen)) + heapq.heappush(states_pq, State(items=items, k=k_partitions)) + else: + for seqlen, idx in sorted_seqlen_list: + heapq.heappush(states_pq, State(items=[(idx, seqlen)], k=k_partitions)) + + while len(states_pq) > 1: + state0 = heapq.heappop(states_pq) + state1 = heapq.heappop(states_pq) + # merge states + state0.merge(state1) + heapq.heappush(states_pq, state0) + + final_state = states_pq[0] + partitions = final_state.get_partitions() + if equal_size: + for i, partition in enumerate(partitions): + assert len(partition) * \ + k_partitions == len(seqlen_list), f"{len(partition)} * {k_partitions} != {len(seqlen_list)}" + return partitions + + +def greedy_partition(seqlen_list: List[int], k_partitions: int, equal_size: bool): + bias = sum(seqlen_list) + 1 if equal_size else 0 + sorted_seqlen = [(seqlen + bias, i) for i, seqlen in enumerate(seqlen_list)] + partitions = [[] for _ in range(k_partitions)] + partition_sums = [0 for _ in range(k_partitions)] + for seqlen, i in sorted_seqlen: + min_idx = None + for j in range(k_partitions): + if min_idx is None or partition_sums[j] < partition_sums[min_idx]: + min_idx = j + partitions[min_idx].append(i) + partition_sums[min_idx] += seqlen + if equal_size: + for i, partition in enumerate(partitions): + assert len(partition) * \ + k_partitions == len(seqlen_list), f"{len(partition)} * {k_partitions} != {len(seqlen_list)}" + return partitions + + +def get_seqlen_balanced_partitions(seqlen_list: List[int], k_partitions: int, equal_size: bool): + """ get order of seq lengths to make partitions balanced, this is + used in balacing sum of seqlength across dp ranks and microbatches + Parameters: + seqlen_list (List[int]): + seq lengths of each items + k_partitions (int): + resulting number of partitions + equal_size (bool): + if True, number of items in each partitions must be equal. + if False, only consider balancing the sum, each partition can have + variable number of items + Returns: + partitions (List[List[int]]): + return k_partitions list containing the index of items. + """ + assert len(seqlen_list) >= k_partitions, f"number of items:[{len(seqlen_list)}] < k_partitions:[{k_partitions}]" + + def _check_and_sort_partitions(partitions): + assert len(partitions) == k_partitions, f"{len(partitions)} != {k_partitions}" + seen_idx = set() + sorted_partitions = [None] * k_partitions + for i, partition in enumerate(partitions): + assert len(partition) > 0, f"the {i}-th partition is empty" + for idx in partition: + seen_idx.add(idx) + sorted_partitions[i] = sorted(partition) + assert seen_idx == set(range(len(seqlen_list))) + return sorted_partitions + + partitions = karmarkar_karp(seqlen_list=seqlen_list, k_partitions=k_partitions, equal_size=equal_size) + return _check_and_sort_partitions(partitions) + + +def log_seqlen_unbalance(seqlen_list: List[int], partitions: List[List[int]], prefix): + # add some metrics of seqlen sum on dp ranks + k_partition = len(partitions) + # assert len(seqlen_list) % k_partition == 0 + batch_size = len(seqlen_list) // k_partition + min_sum_seqlen = None + max_sum_seqlen = None + total_sum_seqlen = 0 + for offset in range(0, len(seqlen_list), batch_size): + cur_sum_seqlen = sum(seqlen_list[offset:offset + batch_size]) + if min_sum_seqlen is None or cur_sum_seqlen < min_sum_seqlen: + min_sum_seqlen = cur_sum_seqlen + if max_sum_seqlen is None or cur_sum_seqlen > max_sum_seqlen: + max_sum_seqlen = cur_sum_seqlen + total_sum_seqlen += cur_sum_seqlen + + balanced_sum_seqlen_list = [] + for partition in partitions: + cur_sum_seqlen_balanced = sum([seqlen_list[i] for i in partition]) + balanced_sum_seqlen_list.append(cur_sum_seqlen_balanced) + # print("balanced_sum_seqlen_list: ", balanced_sum_seqlen_list) + min_sum_seqlen_balanced = min(balanced_sum_seqlen_list) + max_sum_seqlen_balanced = max(balanced_sum_seqlen_list) + + return { + f'{prefix}/min': min_sum_seqlen, + f'{prefix}/max': max_sum_seqlen, + f'{prefix}/minmax_diff': max_sum_seqlen - min_sum_seqlen, + f'{prefix}/balanced_min': min_sum_seqlen_balanced, + f'{prefix}/balanced_max': max_sum_seqlen_balanced, + f'{prefix}/mean': total_sum_seqlen / len(partitions) + } + + +def ceildiv(a, b): + return -(a // -b) + + +def rearrange_micro_batches(batch: TensorDict, max_token_len, dp_group=None): + """Split the batch into a list of micro_batches, where the max_token_len is smaller than max_token_len + and the number of valid tokens in each micro batch is well balanced. + """ + # this is per local micro_bsz + max_seq_len = batch['attention_mask'].shape[-1] + assert max_token_len >= max_seq_len, \ + f'max_token_len must be greater than the sequence length. Got {max_token_len=} and {max_seq_len=}' + + seq_len_effective: torch.Tensor = batch['attention_mask'].sum(dim=1) + total_seqlen = seq_len_effective.sum().item() + num_micro_batches = ceildiv(total_seqlen, max_token_len) + if dist.is_initialized(): + num_micro_batches = torch.tensor([num_micro_batches], device='cuda') + dist.all_reduce(num_micro_batches, op=dist.ReduceOp.MAX, group=dp_group) + num_micro_batches = num_micro_batches.cpu().item() + + seq_len_effective = seq_len_effective.tolist() + assert num_micro_batches <= len(seq_len_effective) + + micro_bsz_idx = get_seqlen_balanced_partitions(seq_len_effective, num_micro_batches, equal_size=False) + + micro_batches = [] + + for partition in micro_bsz_idx: + curr_micro_batch = [] + for idx in partition: + curr_micro_batch.append(batch[idx:idx + 1]) + curr_micro_batch = torch.cat(curr_micro_batch) + + micro_batches.append(curr_micro_batch) + + return micro_batches, micro_bsz_idx + + +def get_reverse_idx(idx_map): + reverse_idx_map = copy.deepcopy(idx_map) + + for i, idx in enumerate(idx_map): + reverse_idx_map[idx] = i + + return reverse_idx_map diff --git a/frameworks/agentgym_rl/verl/utils/tokenizer.py b/frameworks/agentgym_rl/verl/utils/tokenizer.py new file mode 100644 index 0000000000000000000000000000000000000000..b64b6623ac62b6b3f4288dccf8f5307fc87439c7 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/tokenizer.py @@ -0,0 +1,58 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Utils for tokenization.""" +import warnings + +__all__ = ['hf_tokenizer'] + + +def set_pad_token_id(tokenizer): + """Set pad_token_id to eos_token_id if it is None. + + Args: + tokenizer (transformers.PreTrainedTokenizer): The tokenizer to be set. + + """ + if tokenizer.pad_token_id is None: + tokenizer.pad_token_id = tokenizer.eos_token_id + warnings.warn(f'tokenizer.pad_token_id is None. Now set to {tokenizer.eos_token_id}') + if tokenizer.pad_token is None: + tokenizer.pad_token = tokenizer.eos_token + warnings.warn(f'tokenizer.pad_token is None. Now set to {tokenizer.eos_token}') + + +def hf_tokenizer(name_or_path, correct_pad_token=True, correct_gemma2=True, **kwargs): + """Create a huggingface pretrained tokenizer. + + Args: + name (str): The name of the tokenizer. + correct_pad_token (bool): Whether to correct the pad token id. + correct_gemma2 (bool): Whether to correct the gemma2 tokenizer. + **kwargs: The keyword arguments for the tokenizer. + + Returns: + transformers.PreTrainedTokenizer: The pretrained tokenizer. + + """ + from transformers import AutoTokenizer + if correct_gemma2 and isinstance(name_or_path, str) and 'gemma-2-2b-it' in name_or_path: + # the EOS token in gemma2 is ambiguious, which may worsen RL performance. + # https://huggingface.co/google/gemma-2-2b-it/commit/17a01657f5c87135bcdd0ec7abb4b2dece04408a + warnings.warn('Found gemma-2-2b-it tokenizer. Set eos_token and eos_token_id to and 107.') + kwargs['eos_token'] = '' + kwargs['eos_token_id'] = 107 + tokenizer = AutoTokenizer.from_pretrained(name_or_path, **kwargs) + if correct_pad_token: + set_pad_token_id(tokenizer) + return tokenizer \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/utils/torch_dtypes.py b/frameworks/agentgym_rl/verl/utils/torch_dtypes.py new file mode 100644 index 0000000000000000000000000000000000000000..bb63df13b9c26802dff23c92ae8e36f5c23ae4fd --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/torch_dtypes.py @@ -0,0 +1,82 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Adapted from Cruise. +""" + +import torch + +from typing import Union + +HALF_LIST = [16, "16", "fp16", "float16"] +FLOAT_LIST = [32, "32", "fp32", "float32"] +BFLOAT_LIST = ["bf16", "bfloat16"] + + +class PrecisionType(object): + """Type of precision used. + + >>> PrecisionType.HALF == 16 + True + >>> PrecisionType.HALF in (16, "16") + True + """ + + HALF = "16" + FLOAT = "32" + FULL = "64" + BFLOAT = "bf16" + MIXED = "mixed" + + @staticmethod + def supported_type(precision: Union[str, int]) -> bool: + return any(x == precision for x in PrecisionType) + + @staticmethod + def supported_types() -> list[str]: + return [x.value for x in PrecisionType] + + @staticmethod + def is_fp16(precision): + return precision in HALF_LIST + + @staticmethod + def is_fp32(precision): + return precision in FLOAT_LIST + + @staticmethod + def is_bf16(precision): + return precision in BFLOAT_LIST + + @staticmethod + def to_dtype(precision): + if precision in HALF_LIST: + return torch.float16 + elif precision in FLOAT_LIST: + return torch.float32 + elif precision in BFLOAT_LIST: + return torch.bfloat16 + else: + raise RuntimeError(f"unexpected precision: {precision}") + + @staticmethod + def to_str(precision): + if precision == torch.float16: + return 'fp16' + elif precision == torch.float32: + return 'fp32' + elif precision == torch.bfloat16: + return 'bf16' + else: + raise RuntimeError(f"unexpected precision: {precision}") diff --git a/frameworks/agentgym_rl/verl/utils/torch_functional.py b/frameworks/agentgym_rl/verl/utils/torch_functional.py new file mode 100644 index 0000000000000000000000000000000000000000..32bfda2278f7b686a8c0791f88a5bafeb0f811cb --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/torch_functional.py @@ -0,0 +1,525 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Contain small torch utilities +""" + +from typing import Dict, Union, List, Optional + +import torch +import torch.distributed +import torch.nn.functional as F +from tensordict import TensorDict +from torch import nn + +try: + from flash_attn.ops.triton.cross_entropy import cross_entropy_loss + FLAH_ATTN_CROSS_ENTROPY_LOSS_AVAILABLE = True +except ImportError: + FLAH_ATTN_CROSS_ENTROPY_LOSS_AVAILABLE = False + + +def gather_from_labels(data, label): + """Gather the label from data. The value in label should be [0, vocab_size) + + Args: + data: (..., vocab_size) + label (torch.IntTensor) : (...,) + + Returns: + + """ + + output = torch.gather(data, -1, label.unsqueeze(-1)).squeeze(-1) + return output + + +def logprobs_from_logits(logits, labels): + """ + See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591 + """ + if FLAH_ATTN_CROSS_ENTROPY_LOSS_AVAILABLE: + batch_dim = logits.shape[:-1] + last_dim = logits.shape[-1] + logits = logits.reshape(-1, last_dim) + labels = labels.reshape(-1) + output = logprobs_from_logits_flash_attn(logits, labels) + output = output.view(*batch_dim) + else: + output = logprobs_from_logits_v2(logits, labels) + return output + + +def logprobs_from_logits_flash_attn(logits, labels): + output = cross_entropy_loss(logits, labels) + assert isinstance( + output, tuple), "please make sure flash-attn>=2.4.3 where cross_entropy_loss returns Tuple[losses, z_losses]." + return -output[0] + + +def logprobs_from_logits_naive(logits, labels): + logp = F.log_softmax(logits, dim=-1) + logpy = gather_from_labels(logp, labels) + return logpy + + +def logprobs_from_logits_v2(logits: torch.FloatTensor, labels): + """ + A memory efficient implementation of logprobs_from_logits + """ + if logits.dtype in [torch.float32, torch.float64]: + logits_labels = torch.gather(logits, dim=-1, index=labels.unsqueeze(-1)).squeeze(-1) + # loop to reduce peak mem consumption + logsumexp_values = torch.stack([torch.logsumexp(l, dim=-1) for l in logits]) + logprobs_labels = logits_labels - logsumexp_values # log_softmax(x_i) = x_i - logsumexp(x) + else: + # logsumexp approach is unstable with bfloat16, fall back to slightly less efficent approach + logprobs_labels = [] + for row_logits, row_labels in zip(logits, labels): # loop to reduce peak mem consumption + row_logprobs = F.log_softmax(row_logits, dim=-1) + row_logprobs_labels = row_logprobs.gather(dim=-1, index=row_labels.unsqueeze(-1)).squeeze(-1) + logprobs_labels.append(row_logprobs_labels) + logprobs_labels = torch.stack(logprobs_labels) + return logprobs_labels + + +def clip_by_value(x, tensor_min, tensor_max): + """ + Tensor extenstion to torch.clamp + https://github.com/pytorch/pytorch/issues/2793#issuecomment-428784713 + """ + clipped = torch.max(torch.min(x, tensor_max), tensor_min) + return clipped + + +def entropy_from_logits(logits: torch.Tensor): + """Calculate entropy from logits.""" + pd = torch.nn.functional.softmax(logits, dim=-1) + entropy = torch.logsumexp(logits, dim=-1) - torch.sum(pd * logits, dim=-1) + return entropy + + +def masked_sum(values, mask, axis=None): + """Compute mean of tensor with a masked values.""" + return (values * mask).sum(axis=axis) + + +def masked_mean(values, mask, axis=None): + """Compute mean of tensor with a masked values.""" + return (values * mask).sum(axis=axis) / mask.sum(axis=axis) + + +def masked_var(values, mask, unbiased=True): + """Compute variance of tensor with masked values.""" + mean = masked_mean(values, mask) + centered_values = values - mean + variance = masked_mean(centered_values**2, mask) + if unbiased: + mask_sum = mask.sum() + if mask_sum == 0: + raise ValueError("At least one element in the mask has to be 1.") + # note that if mask_sum == 1, then there is a division by zero issue + # to avoid it you just need to use a larger minibatch_size + if mask_sum == 1: + raise ValueError("The sum of the mask is one, which can cause a division by zero.") + bessel_correction = mask_sum / (mask_sum - 1) + variance = variance * bessel_correction + return variance + + +def masked_whiten(values, mask, shift_mean=True): + """Whiten values with masked values.""" + mean, var = masked_mean(values, mask), masked_var(values, mask) + whitened = (values - mean) * torch.rsqrt(var + 1e-8) + if not shift_mean: + whitened += mean + return whitened + + +def get_eos_mask(response_id: torch.Tensor, eos_token: Union[int, List[int]] = 2, dtype=torch.int64): + ''' + end of sentence token can be int or list: 1 or [1, 2] + e.g. eos_token=1 + response_id: [0, 0, 2, 42, 3, 5, 1, 0, 0] + eos_mask: [1, 1, 1, 1, 1, 1, 1, 0, 0] + ''' + if isinstance(eos_token, int): + eos_token = [eos_token] + + eos_mask = torch.zeros_like(response_id, dtype=torch.bool) + for token in eos_token: + eos_mask |= response_id.eq(token) + + eos_mask = eos_mask.long() + eos_mask = (torch.cumsum(eos_mask, dim=1) - eos_mask).bool() + eos_mask = torch.logical_not(eos_mask).to(dtype) + return eos_mask + + +def compute_grad_norm(model: nn.Module): + total_grad_square = 0 + total_params = 0 + for param in model.parameters(): + if param.grad is not None: + total_grad_square += torch.sum(torch.square(param.grad.detach())).item() + return total_grad_square + + +def broadcast_dict_tensor(tensors: Union[Dict[str, torch.Tensor], TensorDict], src, group): + """ + TODO: optimize this. Technically, we only need one broadcast + """ + + for key in tensors.sorted_keys: + torch.distributed.broadcast(tensors[key], src=src, group=group, async_op=False) + + +def allgather_dict_tensors(tensors: Union[Dict[str, torch.Tensor], TensorDict], size, group, dim=0): + """ + TODO: optimize this. + - We can use async ops + - We can use only one allgather + Args: + tensors: + size: + group: + + Returns: + + """ + if isinstance(tensors, TensorDict): + is_tensor_dict = True + tensors_as_dict = tensors.to_dict() + else: + tensors_as_dict = tensors + is_tensor_dict = False + + output = {} + sorted_keys = sorted(tensors_as_dict.keys()) + for key in sorted_keys: + val = tensors_as_dict[key] + output[key] = [torch.empty_like(val) for _ in range(size)] + torch.distributed.all_gather(output[key], val, group=group, async_op=False) + output[key] = torch.cat(output[key], dim=dim) + + if is_tensor_dict: + output = TensorDict(source=output, batch_size=tensors.batch_size[0] * size) + + return output + + +def split_dict_tensor_into_batches(tensors: TensorDict, batch_size) -> List[TensorDict]: + assert tensors.batch_size[0] % batch_size == 0, \ + f'input data batch size: {tensors.batch_size[0]}, split batch size: {batch_size}' + return tensors.split(batch_size) + + +def pad_2d_list_to_length(response, pad_token_id, max_length=None): + """ + pad a 2D list (e.g. responses, logprobs) to a 2D tensor. + """ + response_length = max(len(sub_list) for sub_list in response) + if max_length is not None and max_length > response_length: + target_length = max_length + else: + target_length = response_length + padded_response = [tuple(sub_list) + (pad_token_id,) * (target_length - len(sub_list)) for sub_list in response] + tensor = torch.tensor(padded_response) + return tensor + + +def pad_sequence_to_length(tensors, max_seq_len, pad_token_id, left_pad=False): + """ + pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. + input shape: [bs, seq_length] + output shape: [bs, max_seq_length] + (0, max_seq_len - tensors.shape[-1]) means right pad to max_seq_length and no left pad + """ + if tensors.shape[-1] >= max_seq_len: + return tensors + pad_tuple = (max_seq_len - tensors.shape[-1], 0) if left_pad else (0, max_seq_len - tensors.shape[-1]) + return F.pad(tensors, pad_tuple, 'constant', pad_token_id) + + +from transformers import PreTrainedTokenizer + + +def tokenize_and_postprocess_data(prompt: str, + tokenizer: PreTrainedTokenizer, + max_length: int, + pad_token_id: int, + left_pad=True, + truncation='error'): + """ + input_data is the output from tokenizer. + """ + assert truncation in ['left', 'right', 'error'] + + input_data = tokenizer(prompt, return_tensors='pt', add_special_tokens=False) + + input_ids = input_data['input_ids'] + attention_mask = input_data['attention_mask'] + + assert input_ids.ndim == 2 + + sequence_length = input_ids.shape[-1] + if sequence_length < max_length: + input_ids = pad_sequence_to_length(input_ids, + max_seq_len=max_length, + pad_token_id=pad_token_id, + left_pad=left_pad) + attention_mask = pad_sequence_to_length(attention_mask, + max_seq_len=max_length, + pad_token_id=0, + left_pad=left_pad) + elif sequence_length > max_length: + if truncation == 'left': + # actually, left truncation may not be reasonable + input_ids = input_ids[:, -max_length:] + attention_mask = attention_mask[:, -max_length:] + elif truncation == 'right': + input_ids = input_ids[:, :max_length] + attention_mask = attention_mask[:, :max_length] + elif truncation == 'error': + raise NotImplementedError(f'{sequence_length=} is larger than {max_length=}') + else: + raise NotImplementedError(f'Unknown truncation method {truncation}') + + return input_ids, attention_mask + + +def remove_pad_token(input_ids: torch.Tensor, attention_mask: torch.Tensor): + """ Remove the pad token. + + Args: + input_ids shape: [bs, seq_length] + attention_mask shape: [bs, seq_length] + Returns: + no_padding_batch(List[List[int]]): contains the rmpad token ids per query. + """ + no_padding_batch = [] + for ids, mask in zip(input_ids, attention_mask): + no_padding_batch.append((ids[len(ids) - mask.sum():]).cpu().numpy().tolist()) + return no_padding_batch + + +def log_probs_from_logits_response(input_ids, logits, response_length): + """Compute the response log_probs from full logits. Note that logits = model(input_ids) + + Args: + input_ids: [batch_size, seqlen] + logits: [batch_size, seqlen, vocab_size] + + Returns: + response_log_prob: + """ + response_logits = logits[:, -response_length - 1:-1] + response = input_ids[:, -response_length:] + response_log_prob = logprobs_from_logits(logits=response_logits, labels=response) + return response_log_prob + + +def log_probs_from_logits_response_rmpad(input_ids, attention_mask, logits_rmpad, response_length): + """Compute the log_probs from logits with rmpad logits and pad input. Note that + logits_rmpad = model(input_ids_rmpad). For each sentences, there is a shift between + logits and input_ids. + The reason for this function to is to compute logprobs_from_logits in rmpad mode because it is memory-intensive + for large vocab_size + + Args: + input_ids: [batch_size, seqlen] + attention_mask: [batch_size, seqlen] + logits_rmpad: [total_nnz, vocab_size] + response_length: int + """ + from flash_attn.bert_padding import pad_input, unpad_input + + batch_size, seqlen = input_ids.shape + input_ids_rmpad, indices, *_ = unpad_input(input_ids.unsqueeze(-1), attention_mask=attention_mask) + input_ids_rmpad = input_ids_rmpad.squeeze(-1) + input_ids_rmpad_rolled = torch.roll(input_ids_rmpad, shifts=-1, dims=0) + full_log_probs_rmpad = logprobs_from_logits(logits=logits_rmpad, labels=input_ids_rmpad_rolled) # (total_nnz,) + full_output = pad_input(hidden_states=full_log_probs_rmpad.unsqueeze(-1), + indices=indices, + batch=batch_size, + seqlen=seqlen) + output = full_output.squeeze(-1)[:, -response_length - 1:-1] # [batch_size, response_length] + return output + + +def log_probs_from_logits_all_rmpad(input_ids_rmpad, logits_rmpad, indices, batch_size, seqlen, response_length): + """Compute the log_probs from logits with rmpad input_ids and logits. Note that + logits_rmpad = model(input_ids_rmpad). For each sentences, there is a shift between + logits and input_ids. + The reason for this function to is to compute logprobs_from_logits in rmpad mode because it is memory-intensive + for large vocab_size + + Args: + input_ids_rmpad: [1, total_nnz] + logits_rmpad: [total_nnz, vocab_size] + indices: [total_nnz] + batch_size: int + seqlen: int + response_length: int + """ + from flash_attn.bert_padding import pad_input + input_ids_rmpad = input_ids_rmpad.transpose(0, 1) # transpose back to [total_nnz, 1] + input_ids_rmpad = input_ids_rmpad.squeeze(-1) + input_ids_rmpad_rolled = torch.roll(input_ids_rmpad, shifts=-1, dims=0) + full_log_probs_rmpad = logprobs_from_logits(logits=logits_rmpad, labels=input_ids_rmpad_rolled) # (total_nnz,) + full_output = pad_input(hidden_states=full_log_probs_rmpad.unsqueeze(-1), + indices=indices, + batch=batch_size, + seqlen=seqlen) + output = full_output.squeeze(-1)[:, -response_length - 1:-1] # [batch_size, response_length] + return output + + +from transformers.generation.logits_process import (TemperatureLogitsWarper, TopKLogitsWarper, TopPLogitsWarper) + + +def post_process_logits(input_ids, logits, temperature, top_k, top_p): + if temperature != 1.: + logits = logits.div_(temperature) # inplace operation to avoid OOM + # TODO: add them back + # if top_k is not None and top_k > 0: + # logits = TopKLogitsWarper(top_k=top_k)(input_ids, logits) + # if top_p is not None and top_p < 1.0 and top_p > 0.0: + # logits = TopPLogitsWarper(top_p=top_p)(input_ids, logits) + return logits + + +""" +Optimizer related +""" + +from torch.optim import Optimizer +from torch.optim.lr_scheduler import LambdaLR +import math + + +def get_cosine_schedule_with_warmup( + optimizer: Optimizer, + num_warmup_steps: int, + num_training_steps: int, + min_lr_ratio: float = 0.0, + num_cycles: float = 0.5, + last_epoch: int = -1, +): + """ + Create a schedule with a learning rate that decreases following the values of the cosine function between the + initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the + initial lr set in the optimizer. + Args: + optimizer (:class:`~torch.optim.Optimizer`): + The optimizer for which to schedule the learning rate. + num_warmup_steps (:obj:`int`): + The number of steps for the warmup phase. + num_training_steps (:obj:`int`): + The total number of training steps. + min_lr_ratio (:obj:`float`, `optional`, defaults to 0.0): + The minimum lr ratio w.r.t the maximum. + num_cycles (:obj:`float`, `optional`, defaults to 0.5): + The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 + following a half-cosine). + last_epoch (:obj:`int`, `optional`, defaults to -1): + The index of the last epoch when resuming training. + Return: + :obj:`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. + """ + assert min_lr_ratio >= 0 and min_lr_ratio <= 1. + coef = (1 - min_lr_ratio) * 0.5 + intercept = (1 + min_lr_ratio) * 0.5 + + def lr_lambda(current_step): + if current_step < num_warmup_steps: + return float(current_step) / float(max(1, num_warmup_steps)) + progress = float(current_step - num_warmup_steps) / float(max(1, num_training_steps - num_warmup_steps)) + x = math.cos(math.pi * float(num_cycles) * 2.0 * progress) + return max(0.0, x * coef + intercept) + + return LambdaLR(optimizer, lr_lambda, last_epoch) + + +def get_constant_schedule_with_warmup( + optimizer: Optimizer, + num_warmup_steps: int, + last_epoch: int = -1, +): + + def lr_lambda(current_step): + return min(1, float(current_step) / float(max(1, num_warmup_steps))) + + return LambdaLR(optimizer, lr_lambda, last_epoch) + + +def prepare_decoder_attention_mask(attention_mask, input_shape, inputs_embeds): + # create causal mask + # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] + combined_attention_mask = None + if input_shape[-1] > 1: + combined_attention_mask = _make_causal_mask( + input_shape, + inputs_embeds.dtype, + device=inputs_embeds.device, + ) + + if attention_mask is not None: + # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] + expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, + tgt_len=input_shape[-1]).to(inputs_embeds.device) + combined_attention_mask = (expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + + combined_attention_mask) + + return combined_attention_mask + + +# Copied from transformers.models.bart.modeling_bart._make_causal_mask +def _make_causal_mask(input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device): + """ + Make causal mask used for bi-directional self-attention. + """ + bsz, tgt_len = input_ids_shape + mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device) + mask_cond = torch.arange(mask.size(-1), device=device) + mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0) + mask = mask.to(dtype) + return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len) + + +# Copied from transformers.models.bart.modeling_bart._expand_mask +def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None): + """ + Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. + """ + bsz, src_len = mask.size() + tgt_len = tgt_len if tgt_len is not None else src_len + + expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype) + + inverted_mask = 1.0 - expanded_mask + + return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min) + + +def get_unpad_data(attention_mask): + seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32) + indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten() + max_seqlen_in_batch = seqlens_in_batch.max().item() + cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.int32), (1, 0)) + return ( + indices, + cu_seqlens, + max_seqlen_in_batch, + ) diff --git a/frameworks/agentgym_rl/verl/utils/tracking.py b/frameworks/agentgym_rl/verl/utils/tracking.py new file mode 100644 index 0000000000000000000000000000000000000000..ff300690cb3ac8c66dac85373df391754291e4a0 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/tracking.py @@ -0,0 +1,121 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +A unified tracking interface that supports logging data to different backend +""" +import dataclasses +from enum import Enum +from functools import partial +from pathlib import Path +from typing import List, Union, Dict, Any + + +class Tracking(object): + supported_backend = ["wandb", "mlflow", "swanlab", "console"] + + def __init__(self, project_name, experiment_name, default_backend: Union[str, List[str]] = 'console', config=None): + if isinstance(default_backend, str): + default_backend = [default_backend] + for backend in default_backend: + if backend == 'tracking': + import warnings + warnings.warn("`tracking` logger is deprecated. use `wandb` instead.", DeprecationWarning) + else: + assert backend in self.supported_backend, f'{backend} is not supported' + + self.logger = {} + + if 'tracking' in default_backend or 'wandb' in default_backend: + import wandb + wandb.init(project=project_name, name=experiment_name, config=config) + self.logger['wandb'] = wandb + + if 'mlflow' in default_backend: + import mlflow + mlflow.start_run(run_name=experiment_name) + mlflow.log_params(_compute_mlflow_params_from_objects(config)) + self.logger['mlflow'] = _MlflowLoggingAdapter() + + if "swanlab" in default_backend: + import swanlab + import os + + SWANLAB_API_KEY = os.environ.get("SWANLAB_API_KEY", None) + SWANLAB_LOG_DIR = os.environ.get("SWANLAB_LOG_DIR", "swanlog") + SWANLAB_MODE = os.environ.get("SWANLAB_MODE", "cloud") + if SWANLAB_API_KEY: + swanlab.login(SWANLAB_API_KEY) # NOTE: previous login information will be overwritten + swanlab.init(project=project_name, + experiment_name=experiment_name, + config=config, + logdir=SWANLAB_LOG_DIR, + mode=SWANLAB_MODE) + self.logger["swanlab"] = swanlab + + if 'console' in default_backend: + from verl.utils.logger.aggregate_logger import LocalLogger + self.console_logger = LocalLogger(print_to_console=True) + self.logger['console'] = self.console_logger + + def log(self, data, step, backend=None): + for default_backend, logger_instance in self.logger.items(): + if backend is None or default_backend in backend: + logger_instance.log(data=data, step=step) + + def __del__(self): + if 'wandb' in self.logger: + self.logger['wandb'].finish(exit_code=0) + if 'swanlab' in self.logger: + self.logger['swanlab'].finish() + + +class _MlflowLoggingAdapter: + + def log(self, data, step): + import mlflow + mlflow.log_metrics(metrics=data, step=step) + + +def _compute_mlflow_params_from_objects(params) -> Dict[str, Any]: + if params is None: + return {} + + return _flatten_dict(_transform_params_to_json_serializable(params, convert_list_to_dict=True), sep='/') + + +def _transform_params_to_json_serializable(x, convert_list_to_dict: bool): + _transform = partial(_transform_params_to_json_serializable, convert_list_to_dict=convert_list_to_dict) + + if dataclasses.is_dataclass(x): + return _transform(dataclasses.asdict(x)) + if isinstance(x, dict): + return {k: _transform(v) for k, v in x.items()} + if isinstance(x, list): + if convert_list_to_dict: + return {'list_len': len(x)} | {f'{i}': _transform(v) for i, v in enumerate(x)} + else: + return [_transform(v) for v in x] + if isinstance(x, Path): + return str(x) + if isinstance(x, Enum): + return x.value + + return x + + +def _flatten_dict(raw: Dict[str, Any], *, sep: str) -> Dict[str, Any]: + import pandas as pd + ans = pd.json_normalize(raw, sep=sep).to_dict(orient='records')[0] + assert isinstance(ans, dict) + return ans diff --git a/frameworks/agentgym_rl/verl/utils/ulysses.py b/frameworks/agentgym_rl/verl/utils/ulysses.py new file mode 100644 index 0000000000000000000000000000000000000000..c085becc591d29a9517966cdee601843bdf24371 --- /dev/null +++ b/frameworks/agentgym_rl/verl/utils/ulysses.py @@ -0,0 +1,288 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Utilities for DeepSpeed Ulysses Sequence Parallelism. +DeepSpeed Ulysses Paper: https://arxiv.org/abs/2309.14509 +Inspired from: https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/sequence/layer.py +""" +from typing import Any, Optional, List, Tuple + +import torch +from torch import Tensor +import torch.distributed as dist +from torch.distributed import ProcessGroup + +_ULYSSES_SEQUENCE_PARALLEL_GROUP = None + + +def set_ulysses_sequence_parallel_group(group: dist.ProcessGroup): + """ + Set ulysses sequence parallel process group. + """ + global _ULYSSES_SEQUENCE_PARALLEL_GROUP + _ULYSSES_SEQUENCE_PARALLEL_GROUP = group + + +def get_ulysses_sequence_parallel_group() -> Optional[dist.ProcessGroup]: + """ + Get ulysses sequence parallel process group. + """ + global _ULYSSES_SEQUENCE_PARALLEL_GROUP + return _ULYSSES_SEQUENCE_PARALLEL_GROUP + + +def get_ulysses_sequence_parallel_world_size(group: ProcessGroup = None) -> int: + """ + Get ulysses sequence parallel world size. + """ + group = get_ulysses_sequence_parallel_group() if group is None else group + return dist.get_world_size(group) if group else 1 + + +def get_ulysses_sequence_parallel_rank(group: ProcessGroup = None) -> int: + """ + Get ulysses sequence parallel rank. + """ + group = get_ulysses_sequence_parallel_group() if group is None else group + return dist.get_rank(group) if group else 0 + + +def gather_seq_scatter_heads( + x: Tensor, + seq_dim: int, + head_dim: int, + unpadded_dim_size: int = 0, + group: ProcessGroup = None, +) -> Tensor: + """ + A func to sync embedding input with alltoall in sequence parallel + gather sequence dimension and scatter head dim: + e.g. seq_dim: 1, head_dim: 2 + [bsz, seq/n, h, ...] -> [bsz, seq, h/n, ...] + """ + group = get_ulysses_sequence_parallel_group() if group is None else group + if not group: + return x + sp_world = get_ulysses_sequence_parallel_world_size(group) + x = SeqAllToAll.apply(group, x, head_dim, seq_dim) + if unpadded_dim_size and unpadded_dim_size % sp_world != 0: + padding_size = x.size(seq_dim) - unpadded_dim_size + x = _unpad_tensor(x, seq_dim, padding_size) + return x + + +def gather_heads_scatter_seq(x: Tensor, head_dim: int, seq_dim: int, group: ProcessGroup = None) -> Tensor: + """ + A func to sync attention result with alltoall in sequence parallel + gather head dimension and scatter seq dim: + e.g. seq_dim: 1, head_dim: 2 + [bsz, seq, h/n, ...] -> [bsz, seq/n, h, ...] + """ + group = get_ulysses_sequence_parallel_group() if group is None else group + if not group: + return x + dim_size = x.size(seq_dim) + sp_world = get_ulysses_sequence_parallel_world_size(group) + if dim_size % sp_world != 0: + padding_size = sp_world - (dim_size % sp_world) + x = _pad_tensor(x, seq_dim, padding_size) + return SeqAllToAll.apply(group, x, seq_dim, head_dim, False) + + +def _pad_tensor(x: Tensor, dim: int, padding_size: int) -> Tensor: + shape = list(x.shape) + shape[dim] = padding_size + pad = torch.zeros(shape, dtype=x.dtype, device=x.device) + return torch.cat([x, pad], dim=dim) + + +def _unpad_tensor(x: Tensor, dim: int, padding_size: int) -> Tensor: + slc = [slice(None)] * len(x.shape) + slc[dim] = slice(0, -padding_size) + return x[slc] + + +def slice_input_tensor(x: Tensor, dim: int, padding: bool = True, group: ProcessGroup = None) -> Tensor: + group = get_ulysses_sequence_parallel_group() if group is None else group + sp_world_size = dist.get_world_size(group) + sp_rank = get_ulysses_sequence_parallel_rank() + dim_size = x.size(dim) + # pad before slice + if padding and dim_size % sp_world_size: + padding_size = sp_world_size - (dim_size % sp_world_size) + x = _pad_tensor(x, dim, padding_size) + # slice the input tensor + parts = x.size(dim) // sp_world_size + slc = [slice(None)] * len(x.shape) + slc[dim] = slice(sp_rank * parts, (sp_rank + 1) * parts) + return x[slc].contiguous() + + +def all_to_all_tensor( + local_input: Tensor, + scatter_dim: int, + gather_dim: int, + group: Optional[dist.ProcessGroup] = None, + async_op: bool = False, +): + group = get_ulysses_sequence_parallel_group() if group is None else group + seq_world_size = dist.get_world_size(group) + input_list = [t.contiguous() for t in torch.tensor_split(local_input, seq_world_size, scatter_dim)] + output_list = [torch.empty_like(input_list[0]) for _ in range(seq_world_size)] + comm = dist.all_to_all(output_list, input_list, group=group, async_op=async_op) + if async_op: + + def wait(): + comm.wait() + return torch.cat(output_list, dim=gather_dim).contiguous() + + return wait + return torch.cat(output_list, dim=gather_dim).contiguous() + + +def all_gather_tensor(local_tensor: Tensor, group: Optional[dist.ProcessGroup] = None, async_op: bool = False): + group = get_ulysses_sequence_parallel_group() if group is None else group + sp_world_size = dist.get_world_size(group=group) + output_shape = list(local_tensor.shape) + output_shape[0] = output_shape[0] * sp_world_size + output = torch.empty(output_shape, dtype=local_tensor.dtype, device=local_tensor.device) + dist.all_gather_into_tensor(output, local_tensor, group=group, async_op=async_op) + return output + + +class SeqAllToAll(torch.autograd.Function): + + @staticmethod + def forward( + ctx: Any, + group: dist.ProcessGroup, + local_input: Tensor, + scatter_dim: int, + gather_dim: int, + async_op: bool = False, + ) -> Tensor: + ctx.group = group + ctx.scatter_dim = scatter_dim + ctx.gather_dim = gather_dim + ctx.async_op = async_op + return all_to_all_tensor(local_input, scatter_dim, gather_dim, group, async_op) + + @staticmethod + def backward(ctx: Any, *grad_output: Tensor) -> Tuple[None, Tensor, None, None]: + if ctx.async_op: + input_t = torch.cat(grad_output[1:], dim=ctx.gather_dim).contiguous() + else: + input_t = grad_output[0] + return ( + None, + all_to_all_tensor(input_t, ctx.gather_dim, ctx.scatter_dim, ctx.group, False), + None, + None, + None, + None, + ) + + +class Gather(torch.autograd.Function): + + @staticmethod + def forward(ctx: Any, + group: dist.ProcessGroup, + local_tensor: Tensor, + gather_dim: int, + grad_scaler: bool = True, + async_op=False) -> Tensor: + ctx.group = group + ctx.gather_dim = gather_dim + ctx.grad_scaler = grad_scaler + ctx.async_op = async_op + + sp_world_size = dist.get_world_size(group=group) + ctx.sp_world_size = sp_world_size + + sp_rank = dist.get_rank(group=group) + ctx.sp_rank = sp_rank + + local_shape = list(local_tensor.size()) + split_size = local_shape[0] + part_size = local_shape[gather_dim] # store original size + ctx.part_size = part_size + + output = all_gather_tensor(local_tensor, group, async_op) + return torch.cat(output.split(split_size, dim=0), dim=gather_dim) + + @staticmethod + def backward(ctx: Any, grad_output: Tensor) -> Any: + if ctx.grad_scaler: + grad_output = grad_output * ctx.sp_world_size + return (None, grad_output.split(ctx.part_size, + dim=ctx.gather_dim)[ctx.sp_rank].contiguous(), None, None, None, None) + + +def gather_outpus_and_unpad(x: Tensor, + gather_dim: int, + unpad_dim: int = None, + padding_size: int = 0, + grad_scaler: bool = True, + group: Optional[dist.ProcessGroup] = None): + group = get_ulysses_sequence_parallel_group() if group is None else group + sp_size = get_ulysses_sequence_parallel_world_size() + if group == None: + return x + x = Gather.apply(group, x, gather_dim, grad_scaler) + if unpad_dim is not None: + assert isinstance(padding_size, int), 'padding size is not given or is not an integer' + if padding_size == 0: + return x + x = _unpad_tensor(x, unpad_dim, padding_size) + return x + + +def ulysses_pad_and_slice_inputs(input_ids_rmpad: torch.Tensor, + position_ids_rmpad: Optional[torch.Tensor] = None, + sp_size: int = 1): + """ + Pad and slice input_ids to be divisible by sp_size + Pad position_ids to be divisible by sp_size. + + Note both input_ids_rmpad and position_ids_rmpad will be padded, + but only input_ids will be sliced. + + The is the utility of pre-forward for ulysses sequence parallelism + + Args: + input_ids_rmpad: shape of [bsz, seqlen] + position_ids_rmpad: shape of [bsz, seqlen], where bsz must be 1 + sp_size (int): ulysses sequence parallelism size + + Returns: + torch.Tensor: padded and sliced input_ids + torch.Tensor: padded and sliced position_ids + int: pad size + """ + if position_ids_rmpad is not None: + assert position_ids_rmpad.size(0) == 1 + assert input_ids_rmpad.size(1) == position_ids_rmpad.size(1) + if sp_size <= 1: + return input_ids_rmpad, position_ids_rmpad, 0 + _, total_seq_len = input_ids_rmpad.shape + pad_size = (sp_size - total_seq_len % sp_size) % sp_size + if pad_size > 0: + input_ids_rmpad = torch.nn.functional.pad(input_ids_rmpad, (0, pad_size), value=0) + if position_ids_rmpad is not None: + pad_pos_ids = torch.arange(pad_size, device=position_ids_rmpad.device).unsqueeze(0) + position_ids_rmpad = torch.cat((position_ids_rmpad, pad_pos_ids), dim=-1) + # we don't need to slice position ids + input_ids_rmpad = slice_input_tensor(input_ids_rmpad, dim=1, padding=False) + return input_ids_rmpad, position_ids_rmpad, pad_size diff --git a/frameworks/agentgym_rl/verl/version/version b/frameworks/agentgym_rl/verl/version/version new file mode 100644 index 0000000000000000000000000000000000000000..c1705ab9902fe7953c64991977540fa93343c7d7 --- /dev/null +++ b/frameworks/agentgym_rl/verl/version/version @@ -0,0 +1 @@ +0.2.0.post2 diff --git a/frameworks/agentgym_rl/verl/workers/__init__.py b/frameworks/agentgym_rl/verl/workers/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce90c5eb352d85c59105c0dc85b5f1dd576f095 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/frameworks/agentgym_rl/verl/workers/agent_actor/__init__.py b/frameworks/agentgym_rl/verl/workers/agent_actor/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..4ef863cea2e9c3ff5309849ca3d92a8f4c1da5a4 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/agent_actor/__init__.py @@ -0,0 +1,18 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .base import BasePPOActor +from .dp_actor import DataParallelPPOActor + +__all__ = ["BasePPOActor", "DataParallelPPOActor"] \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/workers/agent_actor/base.py b/frameworks/agentgym_rl/verl/workers/agent_actor/base.py new file mode 100644 index 0000000000000000000000000000000000000000..144f0b90ef1efa77e5f1d4d26a07291ea89990cf --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/agent_actor/base.py @@ -0,0 +1,66 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +The base class for Actor +""" +from abc import ABC, abstractmethod +from typing import Iterable, Dict + +from verl import DataProto +import torch + +__all__ = ['BasePPOActor'] + + +class BasePPOActor(ABC): + + def __init__(self, config): + """The base class for PPO actor + + Args: + config (DictConfig): a config passed to the PPOActor. We expect the type to be + DictConfig (https://omegaconf.readthedocs.io/), but it can be any namedtuple in general. + """ + super().__init__() + self.config = config + + @abstractmethod + def compute_log_prob(self, data: DataProto) -> torch.Tensor: + """Compute logits given a batch of data. + + Args: + data (DataProto): a batch of data represented by DataProto. It must contain key ```input_ids```, + ```attention_mask``` and ```position_ids```. + + Returns: + DataProto: a DataProto containing the key ```log_probs``` + + + """ + pass + + @abstractmethod + def update_policy(self, data: DataProto) -> Dict: + """Update the policy with an iterator of DataProto + + Args: + data (DataProto): an iterator over the DataProto that returns by + ```make_minibatch_iterator``` + + Returns: + Dict: a dictionary contains anything. Typically, it contains the statistics during updating the model + such as ```loss```, ```grad_norm```, etc,. + + """ + pass diff --git a/frameworks/agentgym_rl/verl/workers/agent_actor/dp_actor.py b/frameworks/agentgym_rl/verl/workers/agent_actor/dp_actor.py new file mode 100644 index 0000000000000000000000000000000000000000..f860cfdd3228d17b220b384afb76826b14c37249 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/agent_actor/dp_actor.py @@ -0,0 +1,286 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Single Process Actor +""" + +import itertools +from typing import Tuple + +import torch +from torch import nn +from torch.distributed.fsdp import FullyShardedDataParallel as FSDP + +from verl import DataProto +from verl.agent_trainer.ppo import core_algos +from verl.workers.agent_actor import BasePPOActor +from verl.utils.py_functional import append_to_dict +from verl.utils.torch_functional import logprobs_from_logits, masked_mean +from verl.utils.ulysses import ulysses_pad_and_slice_inputs, gather_outpus_and_unpad +from verl.utils.seqlen_balancing import rearrange_micro_batches, get_reverse_idx +import verl.utils.torch_functional as verl_F + +from flash_attn.bert_padding import pad_input, unpad_input, rearrange, index_first_axis + +__all__ = ['DataParallelPPOActor'] + + +class DataParallelPPOActor(BasePPOActor): + + def __init__( + self, + config, + actor_module: nn.Module, + actor_optimizer: torch.optim.Optimizer = None, + ): + """When optimizer is None, it is Reference Policy""" + super().__init__(config) + self.actor_module = actor_module + self.actor_optimizer = actor_optimizer + self.use_remove_padding = self.config.get('use_remove_padding', False) + print(f'Actor use_remove_padding={self.use_remove_padding}') + self.ulysses_sequence_parallel_size = self.config.ulysses_sequence_parallel_size + self.use_ulysses_sp = self.ulysses_sequence_parallel_size > 1 + + self.compute_entropy_from_logits = torch.compile(verl_F.entropy_from_logits, dynamic=True) + + def _forward_micro_batch(self, micro_batch, temperature) -> Tuple[torch.Tensor, torch.Tensor]: + """ + Returns: + entropy: # (bs, response_len) + log_probs: # (bs, response_len) + """ + response_length = micro_batch['responses'].size(-1) + with torch.autocast(device_type='cuda', dtype=torch.bfloat16): + input_ids = micro_batch['input_ids'] + batch_size, seqlen = input_ids.shape + attention_mask = micro_batch['attention_mask'] + position_ids = micro_batch['position_ids'] + + if self.use_remove_padding: + input_ids_rmpad, indices, *_ = unpad_input(input_ids.unsqueeze(-1), + attention_mask) # input_ids_rmpad (total_nnz, ...) + input_ids_rmpad = input_ids_rmpad.transpose(0, 1) # (1, total_nnz) + + # unpad the position_ids to align the rotary + position_ids_rmpad = index_first_axis(rearrange(position_ids.unsqueeze(-1), "b s ... -> (b s) ..."), + indices).transpose(0, 1) + + # for compute the log_prob + input_ids_rmpad_rolled = torch.roll(input_ids_rmpad, shifts=-1, dims=1) # (1, total_nnz) + + # pad and slice the inputs if sp > 1 + if self.use_ulysses_sp: + input_ids_rmpad, position_ids_rmpad, pad_size = ulysses_pad_and_slice_inputs(input_ids_rmpad, \ + position_ids_rmpad, \ + sp_size=self.ulysses_sequence_parallel_size) + input_ids_rmpad_rolled, _, _ = ulysses_pad_and_slice_inputs(input_ids_rmpad_rolled, None, + self.ulysses_sequence_parallel_size) + + input_ids_rmpad_rolled = input_ids_rmpad_rolled.squeeze(0) # ((total_nnz / sp) + pad) + + # only pass input_ids and position_ids to enable flash_attn_varlen + output = self.actor_module(input_ids=input_ids_rmpad, + attention_mask=None, + position_ids=position_ids_rmpad, + use_cache=False) # prevent model thinks we are generating + logits_rmpad = output.logits.squeeze(0) # (total_nnz, vocab_size) + + logits_rmpad.div_(temperature) + + # compute entropy + entropy_rmpad = self.compute_entropy_from_logits(logits_rmpad) # ((total_nnz / sp) + pad) + + # if use_sp: ((total_nnz / sp) + pad) ; if not use_sp: (batch, seqlen) + log_probs = logprobs_from_logits(logits=logits_rmpad, labels=input_ids_rmpad_rolled) + + # gather log_prob if sp > 1 + if self.use_ulysses_sp: + # gather and unpad for the ulysses sp + log_probs = gather_outpus_and_unpad(log_probs, gather_dim=0, unpad_dim=0, padding_size=pad_size) + entropy_rmpad = gather_outpus_and_unpad(entropy_rmpad, + gather_dim=0, + unpad_dim=0, + padding_size=pad_size) + # pad back to (bsz, seqlen) + full_entropy = pad_input(hidden_states=entropy_rmpad.unsqueeze(-1), + indices=indices, + batch=batch_size, + seqlen=seqlen) + full_log_probs = pad_input(hidden_states=log_probs.unsqueeze(-1), + indices=indices, + batch=batch_size, + seqlen=seqlen) + + # only return response part: + entropy = full_entropy.squeeze(-1)[:, -response_length - 1:-1] # (bsz, response_length) + log_probs = full_log_probs.squeeze(-1)[:, -response_length - 1:-1] # (bsz, response_length) + + else: # not using rmpad and no ulysses sp + output = self.actor_module(input_ids=input_ids, + attention_mask=attention_mask, + position_ids=position_ids, + use_cache=False) # prevent model thinks we are generating + logits = output.logits + logits.div_(temperature) + logits = logits[:, -response_length - 1:-1, :] # (bsz, response_length, vocab_size) + log_probs = logprobs_from_logits(logits, micro_batch['responses']) + entropy = verl_F.entropy_from_logits(logits) # (bsz, response_length) + + return entropy, log_probs + + def _optimizer_step(self): + assert self.config.grad_clip is not None + + if isinstance(self.actor_module, FSDP): + grad_norm = self.actor_module.clip_grad_norm_(max_norm=self.config.grad_clip) + else: + grad_norm = torch.nn.utils.clip_grad_norm_(self.actor_module.parameters(), max_norm=self.config.grad_clip) + self.actor_optimizer.step() + return grad_norm + + def compute_log_prob(self, data: DataProto) -> torch.Tensor: + """Compute the log probability of the responses given input_ids, attention_mask and position_ids + + Args: + data (DataProto): a DataProto containing keys + + ``input_ids``: tensor of shape [batch_size, sequence_length]. torch.int64. Note that input_ids is the + concatenation of prompt and response. Note that ``sequence_length = prompt_length + response_length``. + + ``attention_mask``: tensor of shape [batch_size, sequence_length]. torch.int64. + + ``position_ids``: tensor of shape [batch_size, sequence_length]. torch.int64. + + ``responses``: tensor of shape [batch_size, response_length]. torch.int64. + + Returns: + torch.Tensor: the log_prob tensor + """ + # set to eval + self.actor_module.eval() + + micro_batch_size = data.meta_info['micro_batch_size'] + temperature = data.meta_info['temperature'] # temperature must be in the data.meta_info to avoid slient error + use_dynamic_bsz = data.meta_info['use_dynamic_bsz'] + + select_keys = ['responses', 'input_ids', 'attention_mask', 'position_ids'] + batch = data.select(batch_keys=select_keys).batch + + if use_dynamic_bsz: + # split using dynamic bsz + max_token_len = data.meta_info['max_token_len'] * self.ulysses_sequence_parallel_size + micro_batches, indices = rearrange_micro_batches(batch=batch, max_token_len=max_token_len) + else: + micro_batches = batch.split(micro_batch_size) + + log_probs_lst = [] + for micro_batch in micro_batches: + with torch.no_grad(): + _, log_probs = self._forward_micro_batch(micro_batch, temperature=temperature) + log_probs_lst.append(log_probs) + log_probs = torch.concat(log_probs_lst, dim=0) + + if use_dynamic_bsz: + indices = list(itertools.chain.from_iterable(indices)) + assert len(indices) == log_probs.size(0), f"{len(indices)} vs. {log_probs.size()}" + revert_indices = torch.tensor(get_reverse_idx(indices), dtype=torch.long) + log_probs = log_probs[revert_indices] + + return log_probs + + def update_policy(self, data: DataProto): + # make sure we are in training mode + self.actor_module.train() + + temperature = data.meta_info['temperature'] # temperature must be in the data.meta_info to avoid slient error + + select_keys = ['input_ids', 'attention_mask', 'position_ids', 'old_log_probs', 'advantages', 'responses', 'response_mask'] + if self.config.use_kl_loss: + select_keys.append('ref_log_prob') + batch = data.select(batch_keys=select_keys).batch + + # Split to make minibatch iterator for updating the actor + # See PPO paper for details. https://arxiv.org/abs/1707.06347 + dataloader = batch.split(self.config.ppo_mini_batch_size) + + metrics = {} + for batch_idx, data in enumerate(dataloader): + # split batch into micro_batches + mini_batch = data + if self.config.use_dynamic_bsz: + max_token_len = self.config.ppo_max_token_len_per_gpu * self.ulysses_sequence_parallel_size + micro_batches, _ = rearrange_micro_batches(batch=mini_batch, max_token_len=max_token_len) + else: + self.gradient_accumulation = self.config.ppo_mini_batch_size // self.config.ppo_micro_batch_size_per_gpu + # split batch into micro_batches + micro_batches = mini_batch.split(self.config.ppo_micro_batch_size_per_gpu) + + self.actor_optimizer.zero_grad() + + for data in micro_batches: + data = data.cuda() # actor device is cpu when using offload + response_mask = data['response_mask'] + old_log_prob = data['old_log_probs'] + advantages = data['advantages'] + + clip_ratio = self.config.clip_ratio + entropy_coeff = self.config.entropy_coeff + + # all return: (bsz, response_length) + entropy, log_prob = self._forward_micro_batch(micro_batch=data, temperature=temperature) + + pg_loss, pg_clipfrac, ppo_kl = core_algos.compute_policy_loss(old_log_prob=old_log_prob, + log_prob=log_prob, + advantages=advantages, + eos_mask=response_mask, + cliprange=clip_ratio) + # compute entropy loss from entropy + entropy_loss = verl_F.masked_mean(entropy, response_mask) + + # compute policy loss + policy_loss = pg_loss - entropy_loss * entropy_coeff + + if self.config.use_kl_loss: + ref_log_prob = data['ref_log_prob'] + # compute kl loss + kld = core_algos.kl_penalty(logprob=log_prob, + ref_logprob=ref_log_prob, + kl_penalty=self.config.kl_loss_type) + kl_loss = masked_mean(kld, response_mask) + + policy_loss = policy_loss + kl_loss * self.config.kl_loss_coef + metrics['actor/kl_loss'] = kl_loss.detach().item() + metrics['actor/kl_coef'] = self.config.kl_loss_coef + + if self.config.use_dynamic_bsz: + # relative to the dynamic bsz + loss = policy_loss * (len(data) / self.config.ppo_mini_batch_size) + else: + loss = policy_loss / self.gradient_accumulation + loss.backward() + + data = { + 'actor/entropy_loss': entropy_loss.detach().item(), + 'actor/pg_loss': pg_loss.detach().item(), + 'actor/pg_clipfrac': pg_clipfrac.detach().item(), + 'actor/ppo_kl': ppo_kl.detach().item(), + } + append_to_dict(metrics, data) + + grad_norm = self._optimizer_step() + data = {'actor/grad_norm': grad_norm.detach().item()} + append_to_dict(metrics, data) + self.actor_optimizer.zero_grad() + return metrics diff --git a/frameworks/agentgym_rl/verl/workers/agent_critic/__init__.py b/frameworks/agentgym_rl/verl/workers/agent_critic/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..42eab51464406619c199c83cbf3377f772b39665 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/agent_critic/__init__.py @@ -0,0 +1,18 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .base import BasePPOCritic +from .dp_critic import DataParallelPPOCritic + +__all__ = ["BasePPOCritic", "DataParallelPPOCritic"] \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/workers/agent_critic/base.py b/frameworks/agentgym_rl/verl/workers/agent_critic/base.py new file mode 100644 index 0000000000000000000000000000000000000000..9d1055df4e04d80624d2ca28afcf6f6df3642b91 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/agent_critic/base.py @@ -0,0 +1,40 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Base class for a critic +""" +from abc import ABC, abstractmethod + +import torch + +from verl import DataProto + +__all__ = ['BasePPOCritic'] + + +class BasePPOCritic(ABC): + + def __init__(self, config): + super().__init__() + self.config = config + + @abstractmethod + def compute_values(self, data: DataProto) -> torch.Tensor: + """Compute values""" + pass + + @abstractmethod + def update_critic(self, data: DataProto): + """Update the critic""" + pass diff --git a/frameworks/agentgym_rl/verl/workers/agent_critic/dp_critic.py b/frameworks/agentgym_rl/verl/workers/agent_critic/dp_critic.py new file mode 100644 index 0000000000000000000000000000000000000000..2087a0493ac5ff8c7b16f04b83cc464e946dea42 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/agent_critic/dp_critic.py @@ -0,0 +1,194 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Implement a multiprocess PPOCritic +""" +import itertools + +import torch +from torch import nn, optim + +from torch.distributed.fsdp import FullyShardedDataParallel as FSDP + +from verl import DataProto +from verl.agent_trainer.ppo import core_algos +from verl.workers.agent_critic import BasePPOCritic +from verl.utils.py_functional import append_to_dict +from verl.utils.torch_functional import masked_mean +from verl.utils.ulysses import ulysses_pad_and_slice_inputs, gather_outpus_and_unpad +from verl.utils.seqlen_balancing import rearrange_micro_batches, get_reverse_idx + +from flash_attn.bert_padding import pad_input, unpad_input, rearrange, index_first_axis + +__all__ = ['DataParallelPPOCritic'] + + +class DataParallelPPOCritic(BasePPOCritic): + + def __init__(self, config, critic_module: nn.Module, critic_optimizer: optim.Optimizer): + super().__init__(config=config) + self.critic_module = critic_module + self.critic_optimizer = critic_optimizer + self.use_remove_padding = self.config.model.get('use_remove_padding', False) + print(f'Critic use_remove_padding={self.use_remove_padding}') + + self.ulysses_sequence_parallel_size = self.config.get('ulysses_sequence_parallel_size', 1) + + def _forward_micro_batch(self, micro_batch): + with torch.autocast(device_type='cuda', dtype=torch.bfloat16): + input_ids = micro_batch['input_ids'] + batch, seqlen = input_ids.shape + attention_mask = micro_batch['attention_mask'] + position_ids = micro_batch['position_ids'] + + if self.use_remove_padding: + input_ids_rmpad, indices, *_ = unpad_input(input_ids.unsqueeze(-1), + attention_mask) # input_ids_rmpad (total_nnz, ...) + input_ids_rmpad = input_ids_rmpad.transpose(0, 1) # (1, total_nnz) + + # unpad the position_ids to align the rotary + position_ids_rmpad = index_first_axis(rearrange(position_ids.unsqueeze(-1), "b s ... -> (b s) ..."), + indices).transpose(0, 1) + + # pad and slice the inputs if sp > 1 + if self.ulysses_sequence_parallel_size > 1: + input_ids_rmpad, position_ids_rmpad, pad_size = ulysses_pad_and_slice_inputs(input_ids_rmpad, \ + position_ids_rmpad, \ + sp_size=self.ulysses_sequence_parallel_size) + + # only pass input_ids and position_ids to enable flash_attn_varlen + output = self.critic_module(input_ids=input_ids_rmpad, + attention_mask=None, + position_ids=position_ids_rmpad, + use_cache=False) # prevent model thinks we are generating + values_rmpad = output.logits.squeeze(0) # (total_nnz) + + # gather output if sp > 1 + if self.ulysses_sequence_parallel_size > 1: + values_rmpad = gather_outpus_and_unpad(values_rmpad, + gather_dim=0, + unpad_dim=0, + padding_size=pad_size) + + # pad it back + values = pad_input(values_rmpad, indices=indices, batch=batch, seqlen=seqlen).squeeze(-1) + else: + output = self.critic_module(input_ids=input_ids, + attention_mask=attention_mask, + position_ids=position_ids, + use_cache=False) # prevent model thinks we are generating + values = output.logits.squeeze(-1) + return values + + def _optimizer_step(self): + assert self.config.grad_clip is not None + + if isinstance(self.critic_module, FSDP): + grad_norm = self.critic_module.clip_grad_norm_(self.config.grad_clip) + else: + grad_norm = torch.nn.utils.clip_grad_norm_(self.critic_module.parameters(), max_norm=self.config.grad_clip) + self.critic_optimizer.step() + return grad_norm + + def compute_values(self, data: DataProto) -> torch.Tensor: + self.critic_module.eval() + micro_batch_size = data.meta_info['micro_batch_size'] + select_keys = ['input_ids', 'attention_mask', 'position_ids'] + batch = data.select(batch_keys=select_keys).batch + use_dynamic_bsz = data.meta_info['use_dynamic_bsz'] + + if use_dynamic_bsz: + # split using dynamic bsz + max_token_len = data.meta_info['max_token_len'] * self.ulysses_sequence_parallel_size + micro_batches, indices = rearrange_micro_batches(batch=batch, max_token_len=max_token_len) + else: + micro_batches = batch.split(micro_batch_size) + + values_lst = [] + for micro_batch in micro_batches: + with torch.no_grad(): + values = self._forward_micro_batch(micro_batch) + values_lst.append(values) + values = torch.concat(values_lst, dim=0) + attention_mask = data.batch['attention_mask'] + values = values * attention_mask + + if use_dynamic_bsz: + indices = list(itertools.chain.from_iterable(indices)) + assert len(indices) == values.size(0), f"{len(indices)} vs. {values.size()}" + revert_indices = torch.tensor(get_reverse_idx(indices), dtype=torch.long) + values = values[revert_indices] + + return values + + def update_critic(self, data: DataProto): + # make sure we are in training mode + self.critic_module.train() + metrics = {} + + select_keys = ['input_ids', 'attention_mask', 'position_ids', 'values', 'returns', 'response_mask'] + batch = data.select(batch_keys=select_keys).batch + # Split to make minibatch iterator for updating the actor + # See PPO paper for details. https://arxiv.org/abs/1707.06347 + dataloader = batch.split(self.config.ppo_mini_batch_size) + + for batch_idx, data in enumerate(dataloader): + # split batch into micro_batches + mini_batch = data + if self.config.use_dynamic_bsz: + max_token_len = self.config.ppo_max_token_len_per_gpu * self.ulysses_sequence_parallel_size + micro_batches, _ = rearrange_micro_batches(batch=mini_batch, max_token_len=max_token_len) + else: + micro_batches = mini_batch.split(self.config.ppo_micro_batch_size_per_gpu) + self.gradient_accumulation = self.config.ppo_mini_batch_size // self.config.ppo_micro_batch_size_per_gpu + + self.critic_optimizer.zero_grad() + + for data in micro_batches: + data = data.cuda() # critic device is cpu when using offload + values = data['values'] + returns = data['returns'] + + eos_mask = data['response_mask'] + + vpreds = self._forward_micro_batch(data) + + # assert not torch.any(torch.isnan(vpreds)).item() + + vf_loss, vf_clipfrac = core_algos.compute_value_loss(vpreds=vpreds, + values=values, + returns=returns, + eos_mask=eos_mask, + cliprange_value=self.config.cliprange_value) + if self.config.use_dynamic_bsz: + # relative to the dynamic bsz + loss = vf_loss * (len(data) / self.config.ppo_mini_batch_size) + else: + loss = vf_loss / self.gradient_accumulation + + loss.backward() + + data = { + 'critic/vf_loss': vf_loss.detach().item(), + 'critic/vf_clipfrac': vf_clipfrac.detach().item(), + 'critic/vpred_mean': masked_mean(vpreds, eos_mask).detach().item(), + } + + append_to_dict(metrics, data) + + grad_norm = self._optimizer_step() + data = {'critic/grad_norm': grad_norm.detach().item()} + append_to_dict(metrics, data) + self.critic_optimizer.zero_grad() + return metrics diff --git a/frameworks/agentgym_rl/verl/workers/agent_fsdp_workers.py b/frameworks/agentgym_rl/verl/workers/agent_fsdp_workers.py new file mode 100644 index 0000000000000000000000000000000000000000..2a1789c90de28b82aa8bb457444abd95cded3cff --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/agent_fsdp_workers.py @@ -0,0 +1,846 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +The main entry point to run the PPO algorithm +""" + +import logging +import os +import warnings + +import torch +import torch.distributed +from torch.distributed.device_mesh import init_device_mesh +from omegaconf import DictConfig, open_dict +from verl import DataProto +from verl.single_controller.base import Worker +from verl.single_controller.base.decorator import register, Dispatch +from verl.utils import hf_tokenizer +from verl.utils.debug import log_gpu_memory_usage +from verl.utils.fs import copy_local_path_from_hdfs +from verl.utils.fsdp_utils import get_fsdp_wrap_policy, offload_fsdp_grad, init_fn, get_init_weight_context_manager +from verl.utils.fsdp_utils import offload_fsdp_optimizer, offload_fsdp_param_and_grad, load_fsdp_optimizer, \ + load_fsdp_param_and_grad +from verl.utils.import_utils import import_external_libs +from verl.utils.flops_counter import FlopsCounter +from verl.utils.checkpoint.fsdp_checkpoint_manager import FSDPCheckpointManager +from verl.workers.sharding_manager.fsdp_ulysses import FSDPUlyssesShardingManager +import datetime + +from codetiming import Timer + +logger = logging.getLogger(__file__) +logger.setLevel(os.getenv('VERL_PPO_LOGGING_LEVEL', 'WARN')) + + +def create_device_mesh(world_size, fsdp_size): + if fsdp_size < 0 or fsdp_size >= world_size: + device_mesh = init_device_mesh('cuda', mesh_shape=(world_size,), mesh_dim_names=['fsdp']) + else: + raise ValueError( + 'HSDP is not supported yet because it produces incorrect results for now. Please set fsdp_size=-1') + return device_mesh + + +def get_sharding_strategy(device_mesh): + from torch.distributed.fsdp import ShardingStrategy + if device_mesh.ndim == 1: + sharding_strategy = ShardingStrategy.FULL_SHARD + elif device_mesh.ndim == 2: + sharding_strategy = ShardingStrategy.HYBRID_SHARD + else: + raise NotImplementedError(f"Get device mesh ndim={device_mesh.ndim}, but only support 1 or 2") + return sharding_strategy + + +class ActorRolloutRefWorker(Worker): + """ + This worker can be instantiated as a standalone actor or a standalone rollout or a standalone reference policy + or a hybrid engine based on the config.rollout + """ + + def __init__(self, config: DictConfig, role: str): + super().__init__() + self.config = config + import torch.distributed + if not torch.distributed.is_initialized(): + torch.distributed.init_process_group(backend="nccl", timeout=datetime.timedelta(seconds=3600)) + + # build device mesh for FSDP + world_size = torch.distributed.get_world_size() + # TODO(sgm): support FSDP hybrid shard for larger model + self.device_mesh = create_device_mesh(world_size=world_size, fsdp_size=self.config.actor.fsdp_config.fsdp_size) + + # build device mesh for Ulysses Sequence Parallel + self.ulysses_device_mesh = None + self.ulysses_sequence_parallel_size = self.config.actor.get('ulysses_sequence_parallel_size', 1) + dp = world_size // self.ulysses_sequence_parallel_size + if self.ulysses_sequence_parallel_size > 1: + self.ulysses_device_mesh = init_device_mesh('cuda', + mesh_shape=(dp, self.ulysses_sequence_parallel_size), + mesh_dim_names=['dp', 'sp']) + + self.ulysses_sharding_manager = FSDPUlyssesShardingManager(self.ulysses_device_mesh) + + self.role = role + assert self.role in ['actor', 'rollout', 'ref', 'actor_rollout', 'actor_rollout_ref'] + + self._is_actor = self.role in ['actor', 'actor_rollout', 'actor_rollout_ref'] + self._is_rollout = self.role in ['rollout', 'actor_rollout', 'actor_rollout_ref'] + self._is_ref = self.role in ['ref', 'actor_rollout_ref'] + + self._is_offload_param = False + self._is_offload_grad = False + self._is_offload_optimizer = False + if self._is_actor: + self._is_offload_param = self.config.actor.fsdp_config.get('param_offload', False) + self._is_offload_grad = self.config.actor.fsdp_config.get('grad_offload', False) + self._is_offload_optimizer = self.config.actor.fsdp_config.get('optimizer_offload', False) + elif self._is_ref: + # TODO: it seems that manual offload is slowly than FSDP offload + self._is_offload_param = self.config.ref.fsdp_config.get('param_offload', False) + + # normalize config + if self._is_actor: + self.config.actor.ppo_mini_batch_size *= self.config.rollout.n + self.config.actor.ppo_mini_batch_size //= (self.device_mesh.shape[0] // self.ulysses_sequence_parallel_size) + # micro bsz + if self.config.actor.ppo_micro_batch_size is not None: + self.config.actor.ppo_micro_batch_size //= (self.device_mesh.shape[0] // + self.ulysses_sequence_parallel_size) + self.config.actor.ppo_micro_batch_size_per_gpu = self.config.actor.ppo_micro_batch_size + assert self.config.actor.ppo_mini_batch_size % self.config.actor.ppo_micro_batch_size_per_gpu == 0 + # normalize rollout config + if self._is_rollout and self.config.rollout.log_prob_micro_batch_size is not None: + self.config.rollout.log_prob_micro_batch_size //= (self.device_mesh.shape[0] // + self.ulysses_sequence_parallel_size) + self.config.rollout.log_prob_micro_batch_size_per_gpu = self.config.rollout.log_prob_micro_batch_size + # normalize ref config + if self._is_ref and self.config.ref.log_prob_micro_batch_size is not None: + self.config.ref.log_prob_micro_batch_size //= (self.device_mesh.shape[0] // + self.ulysses_sequence_parallel_size) + self.config.ref.log_prob_micro_batch_size_per_gpu = self.config.ref.log_prob_micro_batch_size + + def _build_model_optimizer(self, + model_path, + fsdp_config, + optim_config, + override_model_config, + use_remove_padding=False, + enable_gradient_checkpointing=False, + trust_remote_code=False, + use_liger=False, + role='actor'): + from verl.utils.model import print_model_size, update_model_config, get_generation_config + from verl.utils.torch_dtypes import PrecisionType + from transformers import AutoModelForCausalLM, AutoConfig + from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, MixedPrecision, CPUOffload + from torch import optim + + assert role in ['actor', 'ref'] + + log_gpu_memory_usage('Before init from HF AutoModel', logger=logger) + local_path = copy_local_path_from_hdfs(model_path) + + # note that we have to create model in fp32. Otherwise, the optimizer is in bf16, which is incorrect + # TODO(zhangchi.usc1992): 1. support create from random initialized model. 2. Support init with FSDP directly + self.tokenizer = hf_tokenizer(local_path, trust_remote_code=trust_remote_code) + + torch_dtype = fsdp_config.get('model_dtype', None) + if torch_dtype is None: + torch_dtype = torch.float32 if self._is_actor else torch.bfloat16 + else: + torch_dtype = PrecisionType.to_dtype(torch_dtype) + + # override model kwargs + actor_model_config = AutoConfig.from_pretrained(local_path, trust_remote_code=trust_remote_code) + + self.generation_config = get_generation_config(local_path, trust_remote_code=trust_remote_code) + + if use_remove_padding: + from verl.models.registry import check_model_support_rmpad + check_model_support_rmpad(actor_model_config.model_type) + + if use_remove_padding and self.ulysses_sequence_parallel_size > 1: + from verl.models.transformers.monkey_patch import apply_monkey_patch + apply_monkey_patch(actor_model_config, verbose=True) + + override_config_kwargs = { + 'bos_token_id': self.tokenizer.bos_token_id, + 'eos_token_id': self.tokenizer.eos_token_id, + 'pad_token_id': self.tokenizer.pad_token_id, + } + override_config_kwargs.update(override_model_config) + update_model_config(actor_model_config, override_config_kwargs=override_config_kwargs) + if self.rank == 0: + print(f'Model config after override: {actor_model_config}') + + # NOTE(fix me): tie_word_embedding causes meta_tensor init to hang + init_context = get_init_weight_context_manager(use_meta_tensor=not actor_model_config.tie_word_embeddings) + + with init_context(), warnings.catch_warnings(): + warnings.simplefilter("ignore") + actor_module = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path=local_path, + torch_dtype=torch_dtype, + config=actor_model_config, + attn_implementation='flash_attention_2', + trust_remote_code=trust_remote_code) + # Apply Liger kernel to the model if use_liger is set to True + if use_liger: + from liger_kernel.transformers.monkey_patch import _apply_liger_kernel_to_instance + _apply_liger_kernel_to_instance(model=actor_module) + + # some parameters may not in torch_dtype. TODO(zhangchi.usc1992) remove this after we switch to fsdp2 + actor_module.to(torch_dtype) + + if enable_gradient_checkpointing: + actor_module.gradient_checkpointing_enable(gradient_checkpointing_kwargs={'use_reentrant': False}) + torch.distributed.barrier() + + if self.rank == 0: + print_model_size(actor_module) + + log_gpu_memory_usage('After init from HF AutoModel', logger=logger) + + # We wrap FSDP for rollout as well + mixed_precision_config = fsdp_config.get('mixed_precision', None) + if mixed_precision_config is not None: + param_dtype = PrecisionType.to_dtype(mixed_precision_config.get('param_dtype', 'bf16')) + reduce_dtype = PrecisionType.to_dtype(mixed_precision_config.get('reduce_dtype', 'fp32')) + buffer_dtype = PrecisionType.to_dtype(mixed_precision_config.get('buffer_dtype', 'fp32')) + else: + param_dtype = torch.bfloat16 + reduce_dtype = torch.float32 + buffer_dtype = torch.float32 + + mixed_precision = MixedPrecision(param_dtype=param_dtype, reduce_dtype=reduce_dtype, buffer_dtype=buffer_dtype) + + auto_wrap_policy = get_fsdp_wrap_policy(module=actor_module, config=fsdp_config.get('wrap_policy', None)) + + if self._is_rollout and self.config.rollout.name == 'hf': + # TODO(zhangchi.usc1992, shengguangming) fix me. Current, auto_wrap_policy causes HFRollout to hang in Gemma + auto_wrap_policy = None + + print(f'wrap_policy: {auto_wrap_policy}') + + fsdp_mesh = self.device_mesh + sharding_strategy = get_sharding_strategy(fsdp_mesh) + + # TODO: add transformer policy + # We force reference policy to use CPUOffload to save memory. + # We force turn off CPUOffload for actor because it causes incorrect results when using grad accumulation + cpu_offload = None if role == 'actor' else CPUOffload(offload_params=True) + actor_module_fsdp = FSDP( + actor_module, + cpu_offload=cpu_offload, + param_init_fn=init_fn, + use_orig_params=False, + auto_wrap_policy=auto_wrap_policy, + device_id=torch.cuda.current_device(), + sharding_strategy=sharding_strategy, # zero3 + mixed_precision=mixed_precision, + sync_module_states=True, + device_mesh=self.device_mesh, + forward_prefetch=False) + + log_gpu_memory_usage('After Actor FSDP init', logger=logger) + + # TODO: add more optimizer args into config + if role == 'actor': + from verl.utils.torch_functional import get_constant_schedule_with_warmup + actor_optimizer = optim.AdamW(actor_module_fsdp.parameters(), + lr=optim_config.lr, + betas=optim_config.get('betas', (0.9, 0.999)), + weight_decay=optim_config.get('weight_decay', 1e-2)) + + total_steps = optim_config.get('total_training_steps', 0) + num_warmup_steps_ratio = optim_config.get('lr_warmup_steps_ratio', 0.) + num_warmup_steps = int(num_warmup_steps_ratio * total_steps) + + print(f'Total steps: {total_steps}, num_warmup_steps: {num_warmup_steps}') + + actor_lr_scheduler = get_constant_schedule_with_warmup(optimizer=actor_optimizer, + num_warmup_steps=num_warmup_steps) + else: + actor_optimizer = None + actor_lr_scheduler = None + + log_gpu_memory_usage('After actor optimizer init', logger=logger) + + return actor_module_fsdp, actor_optimizer, actor_lr_scheduler, actor_model_config + + def _build_rollout(self): + from torch.distributed.device_mesh import init_device_mesh + # TODO(sgm): support FSDP hybrid shard for larger model + infer_tp = self.config.rollout.tensor_model_parallel_size + dp = self.world_size // infer_tp + assert self.world_size % infer_tp == 0, f'rollout world_size: {self.world_size} is not divisible by infer_tp: {infer_tp}' + rollout_device_mesh = init_device_mesh('cuda', mesh_shape=(dp, infer_tp), mesh_dim_names=['dp', 'infer_tp']) + + from verl.workers.rollout.agent_vllm_rollout import vLLMRollout + from verl.workers.sharding_manager import FSDPVLLMShardingManager + log_gpu_memory_usage('Before building vllm rollout', logger=None) + rollout = vLLMRollout(actor_module=self.actor_module_fsdp, + rollout_config=self.config.rollout, + agentgym_config=self.config.agentgym, + tokenizer=self.tokenizer, + model_hf_config=self.actor_model_config) + log_gpu_memory_usage('After building vllm rollout', logger=None) + if torch.distributed.get_world_size() == 1: + self.config.rollout.load_format = 'dummy_hf' + rollout_sharding_manager = FSDPVLLMShardingManager(module=self.actor_module_fsdp, + inference_engine=rollout.inference_engine, + model_config=self.actor_model_config, + full_params='hf' in self.config.rollout.load_format, + device_mesh=rollout_device_mesh) + log_gpu_memory_usage('After building sharding manager', logger=None) + + return rollout, rollout_sharding_manager + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def init_model(self): + from verl.workers.agent_actor import DataParallelPPOActor + # This is used to import external_lib into the huggingface systems + import_external_libs(self.config.model.get('external_lib', None)) + + from omegaconf import OmegaConf + override_model_config = OmegaConf.to_container(self.config.model.get('override_config', OmegaConf.create())) + + use_remove_padding = self.config.model.get('use_remove_padding', False) + + if self._is_actor or self._is_rollout: + # we need the model for actor and rollout + # if self._is_actor: + # optim_config = self.config.actor.optim + # fsdp_config = self.config.actor.fsdp_config + # else: + # optim_config = None + # fsdp_config = OmegaConf.create() + optim_config = self.config.actor.optim + fsdp_config = self.config.actor.fsdp_config + self.actor_module_fsdp, self.actor_optimizer, self.actor_lr_scheduler, self.actor_model_config = self._build_model_optimizer( + model_path=self.config.model.path, + fsdp_config=fsdp_config, + optim_config=optim_config, + override_model_config=override_model_config, + use_remove_padding=use_remove_padding, + enable_gradient_checkpointing=self.config.model.get('enable_gradient_checkpointing', False), + trust_remote_code=self.config.model.get('trust_remote_code', False), + use_liger=self.config.model.get('use_liger', False), + role='actor') + + # get the original unwrapped module + self.actor_module = self.actor_module_fsdp._fsdp_wrapped_module + + if self._is_offload_param: + # param is require during state_dict in sharding manager + offload_fsdp_grad(module=self.actor_module_fsdp) + log_gpu_memory_usage('After offload actor grad during init', logger=logger) + if self._is_offload_optimizer: + offload_fsdp_optimizer(optimizer=self.actor_optimizer) + log_gpu_memory_usage('After offload actor optimizer during init', logger=logger) + # load from checkpoint + if self._is_actor: + OmegaConf.set_struct(self.config.actor, True) + with open_dict(self.config.actor): + self.config.actor.use_remove_padding = use_remove_padding + self.actor = DataParallelPPOActor(config=self.config.actor, + actor_module=self.actor_module_fsdp, + actor_optimizer=self.actor_optimizer) + + if self._is_rollout: + self.rollout, self.rollout_sharding_manager = self._build_rollout() + + if self._is_ref: + self.ref_module_fsdp = self._build_model_optimizer(model_path=self.config.model.path, + fsdp_config=self.config.ref.fsdp_config, + optim_config=None, + override_model_config=override_model_config, + use_remove_padding=use_remove_padding, + trust_remote_code=self.config.model.get( + 'trust_remote_code', False), + use_liger=self.config.model.get('use_liger', False), + role='ref')[0] + OmegaConf.set_struct(self.config.ref, True) + with open_dict(self.config.ref): + self.config.ref.use_remove_padding = use_remove_padding + self.ref_policy = DataParallelPPOActor(config=self.config.ref, actor_module=self.ref_module_fsdp) + + if self._is_actor: + self.flops_counter = FlopsCounter(self.actor_model_config) + self.checkpoint_manager = FSDPCheckpointManager(model=self.actor_module_fsdp, + optimizer=self.actor.actor_optimizer, + lr_scheduler=self.actor_lr_scheduler, + tokenizer=self.tokenizer) + + torch.cuda.empty_cache() + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def update_actor(self, data: DataProto): + data = data.to('cuda') + + assert self._is_actor + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.actor_module_fsdp, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + if self._is_offload_optimizer: + load_fsdp_optimizer(optimizer=self.actor_optimizer, device_id=torch.cuda.current_device()) + + data.batch = data.batch.cuda() + + log_gpu_memory_usage('Before update policy', logger=logger) + + with self.ulysses_sharding_manager: + data = self.ulysses_sharding_manager.preprocess_data(data=data) + # perform training + with Timer(name='update_policy', logger=None) as timer: + metrics = self.actor.update_policy(data=data) + delta_time = timer.last + global_num_tokens = data.meta_info['global_token_num'] + estimated_flops, promised_flops = self.flops_counter.estimate_flops(global_num_tokens, delta_time) + metrics['mfu/actor'] = estimated_flops * self.config.actor.ppo_epochs / promised_flops / self.world_size + + self.actor_lr_scheduler.step() + lr = self.actor_lr_scheduler.get_last_lr()[0] + metrics['actor/lr'] = lr + + log_gpu_memory_usage('After update policy', logger=logger) + + # TODO: here, we should return all metrics + output = DataProto(meta_info={'metrics': metrics}) + + output = self.ulysses_sharding_manager.postprocess_data(data=output) + output = output.to('cpu') + + if self._is_offload_param: + offload_fsdp_param_and_grad(module=self.actor_module_fsdp, offload_grad=self._is_offload_grad) + if self._is_offload_optimizer: + offload_fsdp_optimizer(optimizer=self.actor_optimizer) + torch.cuda.empty_cache() + return output + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def generate_sequences(self, prompts: DataProto): + prompts = prompts.to('cuda') + + assert self._is_rollout + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.actor_module_fsdp, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + + prompts.batch = prompts.batch.cuda() + meta_info = { + 'eos_token_id': + self.generation_config.eos_token_id + if self.generation_config is not None else self.tokenizer.eos_token_id, + 'pad_token_id': + self.generation_config.pad_token_id + if self.generation_config is not None else self.tokenizer.pad_token_id, + } + prompts.meta_info.update(meta_info) + with self.rollout_sharding_manager: + log_gpu_memory_usage('After entering rollout sharding manager', logger=logger) + + prompts = self.rollout_sharding_manager.preprocess_data(prompts) + output = self.rollout.generate_sequences(prompts=prompts) + + log_gpu_memory_usage('After rollout generation', logger=logger) + + output = self.rollout_sharding_manager.postprocess_data(output) + + output = output.to('cpu') + + if self._is_offload_param: + # NOTE(sgm): the grad is already in CPU, only offload param here + offload_fsdp_param_and_grad(module=self.actor_module_fsdp, offload_grad=self._is_offload_grad) + # clear kv cache + torch.cuda.empty_cache() + log_gpu_memory_usage('After recompute log prob', logger=logger) + return output + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def compute_log_prob(self, data: DataProto): + assert self._is_actor + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.actor_module_fsdp, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + data = data.to('cuda') + # we should always recompute old_log_probs when it is HybridEngine + data.meta_info['micro_batch_size'] = self.config.rollout.log_prob_micro_batch_size_per_gpu + data.meta_info['max_token_len'] = self.config.rollout.log_prob_max_token_len_per_gpu + data.meta_info['use_dynamic_bsz'] = self.config.rollout.log_prob_use_dynamic_bsz + data.meta_info['temperature'] = self.config.rollout.temperature + # perform recompute log_prob + with self.ulysses_sharding_manager: + data = self.ulysses_sharding_manager.preprocess_data(data) + output = self.actor.compute_log_prob(data=data) + output = DataProto.from_dict(tensors={'old_log_probs': output}, + meta_info={'temperature': self.config.rollout.temperature}) + output = self.ulysses_sharding_manager.postprocess_data(output) + + output = output.to('cpu') + + # https://pytorch.org/docs/stable/notes/fsdp.html#fsdp-notes + # unshard the root FSDP module + if self.world_size > 1: + self.actor.actor_module._handle.reshard(True) + + if self._is_offload_param: + # NOTE(sgm): the grad is already in CPU, only offload param here + offload_fsdp_param_and_grad(module=self.actor_module_fsdp, offload_grad=self._is_offload_grad) + + # clear kv cache + torch.cuda.empty_cache() + log_gpu_memory_usage('After compute_log_prob', logger=logger) + return output + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def compute_ref_log_prob(self, data: DataProto): + assert self._is_ref + + data = data.to('cuda') + + micro_batch_size = self.config.ref.log_prob_micro_batch_size_per_gpu + data.meta_info['micro_batch_size'] = micro_batch_size + data.meta_info['temperature'] = self.config.rollout.temperature + data.meta_info['max_token_len'] = self.config.ref.log_prob_max_token_len_per_gpu + data.meta_info['use_dynamic_bsz'] = self.config.ref.log_prob_use_dynamic_bsz + with self.ulysses_sharding_manager: + data = self.ulysses_sharding_manager.preprocess_data(data) + output = self.ref_policy.compute_log_prob(data=data) + output = DataProto.from_dict(tensors={'ref_log_prob': output}) + output = self.ulysses_sharding_manager.postprocess_data(output) + + output = output.to('cpu') + + # https://pytorch.org/docs/stable/notes/fsdp.html#fsdp-notes + # unshard the root FSDP module + if self.world_size > 1: + self.ref_policy.actor_module._handle.reshard(True) + + torch.cuda.empty_cache() + return output + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def save_checkpoint(self, local_path, hdfs_path=None, global_step=0, remove_previous_ckpt=False): + # only support save and load ckpt for actor + assert self._is_actor + import torch + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.actor_module_fsdp, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + + self.checkpoint_manager.save_checkpoint(local_path=local_path, + hdfs_path=hdfs_path, + global_step=global_step, + remove_previous_ckpt=remove_previous_ckpt) + + torch.distributed.barrier() + if self._is_offload_param: + offload_fsdp_param_and_grad(module=self.actor_module_fsdp, offload_grad=self._is_offload_grad) + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def load_checkpoint(self, path, del_local_after_load=False): + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.actor_module_fsdp, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + + self.checkpoint_manager.load_checkpoint(path=path, del_local_after_load=del_local_after_load) + + if self._is_offload_param: + offload_fsdp_param_and_grad(module=self.actor_module_fsdp, offload_grad=self._is_offload_grad) + + +class CriticWorker(Worker): + + def __init__(self, config): + super().__init__() + import torch.distributed + if not torch.distributed.is_initialized(): + torch.distributed.init_process_group(backend="nccl", timeout=datetime.timedelta(seconds=3600)) + self.config = config + + # build device mesh for Ulysses Sequence Parallel + world_size = torch.distributed.get_world_size() + from torch.distributed.device_mesh import init_device_mesh + + fsdp_size = self.config.model.fsdp_config.fsdp_size + self.device_mesh = create_device_mesh(world_size=world_size, fsdp_size=fsdp_size) + + self.ulysses_device_mesh = None + self.ulysses_sequence_parallel_size = self.config.get('ulysses_sequence_parallel_size', 1) + dp = world_size // self.ulysses_sequence_parallel_size + if self.ulysses_sequence_parallel_size > 1: + self.ulysses_device_mesh = init_device_mesh('cuda', + mesh_shape=(dp, self.ulysses_sequence_parallel_size), + mesh_dim_names=['dp', 'sp']) + + self.ulysses_sharding_manager = FSDPUlyssesShardingManager(self.ulysses_device_mesh) + + # set FSDP offload params + self._is_offload_param = self.config.model.fsdp_config.param_offload + self._is_offload_grad = self.config.model.fsdp_config.grad_offload + self._is_offload_optimizer = self.config.model.fsdp_config.optimizer_offload + + # normalize config + self.config.ppo_mini_batch_size //= (torch.distributed.get_world_size() // self.ulysses_sequence_parallel_size) + if self.config.ppo_micro_batch_size is not None: + self.config.ppo_micro_batch_size //= (torch.distributed.get_world_size() // + self.ulysses_sequence_parallel_size) + self.config.forward_micro_batch_size //= (torch.distributed.get_world_size() // + self.ulysses_sequence_parallel_size) + self.config.ppo_micro_batch_size_per_gpu = self.config.ppo_micro_batch_size + self.config.forward_micro_batch_size_per_gpu = self.config.forward_micro_batch_size + assert self.config.ppo_mini_batch_size % self.config.ppo_micro_batch_size_per_gpu == 0 + + def _build_critic_model_optimizer(self, config): + # the following line is necessary + from verl.utils.model import print_model_size + from verl.utils.torch_dtypes import PrecisionType + from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, MixedPrecision + from torch import optim + + local_path = copy_local_path_from_hdfs(config.model.path) + # note that the tokenizer between actor and critic may be different. So override tokenizer info with actor info + # using random initialized model from any architecture. May not be the same as Actor. + + tokenizer_path = copy_local_path_from_hdfs(config.model.tokenizer_path) + self.tokenizer = hf_tokenizer(tokenizer_path, trust_remote_code=config.model.get('trust_remote_code', False)) + + from omegaconf import OmegaConf + override_config = OmegaConf.to_container(self.config.model.get('override_config', OmegaConf.create())) + override_config_kwargs = { + 'bos_token_id': self.tokenizer.bos_token_id, + 'eos_token_id': self.tokenizer.eos_token_id, + 'pad_token_id': self.tokenizer.pad_token_id, + } + override_config_kwargs.update(override_config) + if self.rank == 0: + print(f'Critic overriding config {override_config_kwargs}') + + torch_dtype = self.config.model.fsdp_config.get('model_dtype', 'fp32') + torch_dtype = PrecisionType.to_dtype(torch_dtype) + + from transformers import AutoConfig, AutoModelForTokenClassification + from torch import nn + + trust_remote_code = False + critic_model_config = AutoConfig.from_pretrained(local_path, trust_remote_code=trust_remote_code) + critic_model_config.num_labels = 1 + + use_remove_padding = config.model.get('use_remove_padding', False) + if use_remove_padding: + from verl.models.registry import check_model_support_rmpad + check_model_support_rmpad(critic_model_config.model_type) + + if use_remove_padding and self.ulysses_sequence_parallel_size > 1: + from verl.models.transformers.monkey_patch import apply_monkey_patch + apply_monkey_patch(critic_model_config, verbose=True) + + init_context = get_init_weight_context_manager() + with init_context(), warnings.catch_warnings(): + warnings.simplefilter("ignore") + setattr(critic_model_config, 'classifier_dropout', 0.) + setattr(critic_model_config, 'hidden_dropout', '0') + critic_module = AutoModelForTokenClassification.from_pretrained(pretrained_model_name_or_path=local_path, + torch_dtype=torch_dtype, + config=critic_model_config, + attn_implementation='flash_attention_2', + trust_remote_code=trust_remote_code) + + # some parameters may not in torch_dtype + critic_module.to(torch_dtype) + + if config.model.get('enable_gradient_checkpointing', False): + critic_module.gradient_checkpointing_enable(gradient_checkpointing_kwargs={'use_reentrant': False}) + if self.rank == 0: + print_model_size(critic_module) + + self.critic_model_config = critic_model_config + + fsdp_config = self.config.model.fsdp_config + mixed_precision_config = fsdp_config.get('mixed_precision', None) + if mixed_precision_config is not None: + param_dtype = PrecisionType.to_dtype(mixed_precision_config.get('param_dtype', 'bf16')) + reduce_dtype = PrecisionType.to_dtype(mixed_precision_config.get('reduce_dtype', 'fp32')) + buffer_dtype = PrecisionType.to_dtype(mixed_precision_config.get('buffer_dtype', 'fp32')) + else: + param_dtype = torch.bfloat16 + reduce_dtype = torch.float32 + buffer_dtype = torch.float32 + + mixed_precision = MixedPrecision(param_dtype=param_dtype, reduce_dtype=reduce_dtype, buffer_dtype=buffer_dtype) + + auto_wrap_policy = get_fsdp_wrap_policy(module=critic_module, config=self.config.model.fsdp_config.wrap_policy) + + log_gpu_memory_usage('Before critic FSDP', logger=None) + + fsdp_mesh = self.device_mesh + sharding_strategy = get_sharding_strategy(fsdp_mesh) + + # Note: We force turn off CPUOffload for critic because it causes incorrect results when using grad accumulation + critic_module = FSDP(critic_module, + param_init_fn=init_fn, + use_orig_params=False, + auto_wrap_policy=auto_wrap_policy, + device_id=torch.cuda.current_device(), + sharding_strategy=sharding_strategy, + mixed_precision=mixed_precision, + sync_module_states=True, + forward_prefetch=False, + device_mesh=self.device_mesh, + cpu_offload=None) + + log_gpu_memory_usage('After critic FSDP', logger=None) + + critic_optimizer = optim.AdamW(critic_module.parameters(), + lr=config.optim.lr, + betas=config.optim.get('betas', (0.9, 0.999)), + weight_decay=config.optim.get('weight_decay', 1e-2)) + + total_steps = config.optim.get('total_training_steps', 0) + num_warmup_steps_ratio = config.optim.get('lr_warmup_steps_ratio', 0.) + num_warmup_steps = int(num_warmup_steps_ratio * total_steps) + + print(f'Total steps: {total_steps}, num_warmup_steps: {num_warmup_steps}') + + from verl.utils.torch_functional import get_constant_schedule_with_warmup + critic_lr_scheduler = get_constant_schedule_with_warmup(optimizer=critic_optimizer, + num_warmup_steps=num_warmup_steps) + + return critic_module, critic_optimizer, critic_lr_scheduler + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def init_model(self): + # This is used to import external_lib into the huggingface systems + import_external_libs(self.config.model.get('external_lib', None)) + + from verl.workers.agent_critic import DataParallelPPOCritic + self.critic_module, self.critic_optimizer, self.critic_lr_scheduler = self._build_critic_model_optimizer( + self.config) + + if self._is_offload_param: + offload_fsdp_param_and_grad(module=self.critic_module, offload_grad=self._is_offload_grad) + if self._is_offload_optimizer: + offload_fsdp_optimizer(optimizer=self.critic_optimizer) + + self.critic = DataParallelPPOCritic(config=self.config, + critic_module=self.critic_module, + critic_optimizer=self.critic_optimizer) + + self.flops_counter = FlopsCounter(self.critic_model_config) + self.checkpoint_manager = FSDPCheckpointManager(model=self.critic_module, + optimizer=self.critic_optimizer, + lr_scheduler=self.critic_lr_scheduler, + tokenizer=self.tokenizer) + + torch.cuda.empty_cache() + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def compute_values(self, data: DataProto): + data = data.to('cuda') + + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.critic_module, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + micro_batch_size = self.config.forward_micro_batch_size_per_gpu + data.meta_info['micro_batch_size'] = micro_batch_size + data.meta_info['max_token_len'] = self.config.forward_max_token_len_per_gpu + data.meta_info['use_dynamic_bsz'] = self.config.use_dynamic_bsz + # perform forward computation + with self.ulysses_sharding_manager: + data = self.ulysses_sharding_manager.preprocess_data(data=data) + values = self.critic.compute_values(data=data) + output = DataProto.from_dict(tensors={'values': values}) + output = self.ulysses_sharding_manager.postprocess_data(data=output) + + output = output.to('cpu') + if self._is_offload_param: + offload_fsdp_param_and_grad(module=self.critic_module, offload_grad=self._is_offload_grad) + torch.cuda.empty_cache() + return output + + @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO) + def update_critic(self, data: DataProto): + data = data.to('cuda') + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.critic_module, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + if self._is_offload_optimizer: + load_fsdp_optimizer(optimizer=self.critic_optimizer, device_id=torch.cuda.current_device()) + + # perform forward computation + with self.ulysses_sharding_manager: + data = self.ulysses_sharding_manager.preprocess_data(data=data) + + with Timer(name='update_critic', logger=None) as timer: + metrics = self.critic.update_critic(data=data) + delta_time = timer.last + + global_num_tokens = data.meta_info['global_token_num'] + estimated_flops, promised_flops = self.flops_counter.estimate_flops(global_num_tokens, delta_time) + metrics['mfu/critic'] = estimated_flops * self.config.ppo_epochs / promised_flops / self.world_size + + self.critic_lr_scheduler.step() + lr = self.critic_lr_scheduler.get_last_lr()[0] + metrics['critic/lr'] = lr + + output = DataProto(batch=None, meta_info={'metrics': metrics}) + output = self.ulysses_sharding_manager.postprocess_data(data=output) + + if self._is_offload_param: + offload_fsdp_param_and_grad(module=self.critic_module, offload_grad=self._is_offload_grad) + if self._is_offload_optimizer: + offload_fsdp_optimizer(optimizer=self.critic_optimizer) + torch.cuda.empty_cache() + output = output.to('cpu') + return output + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def save_checkpoint(self, local_path, hdfs_path=None, global_step=0, remove_previous_ckpt=False): + import torch + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.critic_module, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + + self.checkpoint_manager.save_checkpoint(local_path=local_path, + hdfs_path=hdfs_path, + global_step=global_step, + remove_previous_ckpt=remove_previous_ckpt) + + torch.distributed.barrier() + if self._is_offload_param: + offload_fsdp_param_and_grad(module=self.critic_module, offload_grad=self._is_offload_grad) + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def load_checkpoint(self, path, del_local_after_load=True): + import torch + if self._is_offload_param: + load_fsdp_param_and_grad(module=self.critic_module, + device_id=torch.cuda.current_device(), + load_grad=self._is_offload_grad) + + self.checkpoint_manager.load_checkpoint(path=path, del_local_after_load=del_local_after_load) + + torch.distributed.barrier() + if self._is_offload_param: + offload_fsdp_param_and_grad(module=self.critic_module, offload_grad=self._is_offload_grad) diff --git a/frameworks/agentgym_rl/verl/workers/megatron_workers.py b/frameworks/agentgym_rl/verl/workers/megatron_workers.py new file mode 100644 index 0000000000000000000000000000000000000000..894d64cfac86dd3d19a0af6a75d5844ac81231ea --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/megatron_workers.py @@ -0,0 +1,755 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +The main entry point to run the PPO algorithm +""" + +import os +import logging +import ray +import torch +import torch.distributed +import torch.nn as nn +from omegaconf import DictConfig +from verl.single_controller.base.megatron.worker import MegatronWorker +from verl.workers.actor.megatron_actor import MegatronPPOActor +from verl.workers.critic.megatron_critic import MegatronPPOCritic +from verl.workers.sharding_manager import AllGatherPPModel +from verl.workers.reward_model.megatron.reward_model import MegatronRewardModel + +from verl.single_controller.base.decorator import register, Dispatch +from verl import DataProto +from verl.utils.fs import copy_local_path_from_hdfs +from verl.utils.debug import log_gpu_memory_usage +from verl.utils.model import load_megatron_model_weights +from verl.utils.megatron_utils import init_model_parallel_config +from verl.utils.megatron_utils import offload_megatron_param_and_grad, load_megatron_param_and_grad +from verl.utils import hf_tokenizer + +from megatron.core import parallel_state as mpu +from megatron.core import ModelParallelConfig + +logger = logging.getLogger(__file__) +logger.setLevel(os.getenv('VERL_PPO_LOGGING_LEVEL', 'WARN')) + + +def set_random_seed(seed): + import torch + import numpy as np + import random + torch.manual_seed(seed) + np.random.seed(seed) + random.seed(seed) + if torch.cuda.device_count() > 0: + from megatron.core import tensor_parallel + tensor_parallel.model_parallel_cuda_manual_seed(seed) + # FIXME: torch cumsum not support deterministic (used in vllm sampler), + # https://github.com/pytorch/pytorch/issues/89492 + # torch.use_deterministic_algorithms(True, warn_only=True) + # os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8' + + +class ActorRolloutRefWorker(MegatronWorker): + """ + This worker can be instantiated as a standalone actor or a standalone rollout or a standalone reference policy + or a hybrid engine based on the config.rollout + """ + + def __init__(self, config: DictConfig, role: str): + super().__init__() + self.config = config + + # NOTE(sgm): We utilize colocate WorkerGroup by default. + # As a result, Workers for different model share the same process. + # Therefore, we only require one distribute initialization. + # To utilize different parallel startegy in different models: + # 1, users should disable WorkerDict; 2.assign different ResourcePool to different models, + # 3. and apply the following patch in ray==2.10, https://github.com/ray-project/ray/pull/44385 + if not torch.distributed.is_initialized(): + rank = int(os.environ['LOCAL_RANK']) + torch.distributed.init_process_group(backend="nccl") + torch.cuda.set_device(rank) + + if self.config.actor.megatron.sequence_parallel: + os.environ['CUDA_DEVICE_MAX_CONNECTIONS'] = '1' + mpu.initialize_model_parallel( + tensor_model_parallel_size=self.config.actor.megatron.tensor_model_parallel_size, + pipeline_model_parallel_size=self.config.actor.megatron.pipeline_model_parallel_size, + virtual_pipeline_model_parallel_size=None, + pipeline_model_parallel_split_rank=None, + use_sharp=False, + context_parallel_size=1, + expert_model_parallel_size=1, + nccl_communicator_config_path=None, + ) + + set_random_seed(seed=self.config.actor.megatron.seed) + + self.role = role + assert self.role in ['actor', 'rollout', 'ref', 'actor_rollout', 'actor_rollout_ref'] + + self._is_actor = self.role in ['actor', 'actor_rollout', 'actor_rollout_ref'] + self._is_rollout = self.role in ['rollout', 'actor_rollout', 'actor_rollout_ref'] + self._is_ref = self.role in ['ref', 'actor_rollout_ref'] + + # TODO(sgm): Currently, we only support reference model param offload + # will support other offload later + self._is_offload_param = False + self._is_offload_grad = False + self._is_offload_optimizer = False + + # normalize config + if self._is_actor and self._is_rollout: + self.config.actor.ppo_mini_batch_size //= mpu.get_data_parallel_world_size() + if self.config.actor.get('ppo_micro_batch_size', None): + self.config.actor.ppo_micro_batch_size //= mpu.get_data_parallel_world_size() + self.config.rollout.log_prob_micro_batch_size //= mpu.get_data_parallel_world_size() + self.config.actor.ppo_micro_batch_size_per_gpu = self.config.actor.ppo_micro_batch_size + self.config.rollout.log_prob_micro_batch_size_per_gpu = self.config.rollout.log_prob_micro_batch_size + + self._is_offload_param = self.config.actor.get('param_offload', False) + self._is_offload_grad = self.config.actor.get('grad_offload', False) + self._is_offload_optimizer = self.config.actor.get('optimizer_offload', False) + elif self._is_ref: + if self.config.ref.get('ppo_micro_batch_size', None): + self.config.ref.log_prob_micro_batch_size //= mpu.get_data_parallel_world_size() + self.config.ref.ppo_micro_batch_size_per_gpu = self.config.ref.ppo_micro_batch_size + self._is_offload_param = self.config.ref.get('param_offload', False) + + def _build_model_optimizer(self, + model_path, + megatron_config: ModelParallelConfig, + optim_config, + override_model_config, + enable_gradient_checkpointing=False): + from verl.utils.megatron.optimizer import get_megatron_optimizer + from megatron.core.models.gpt.gpt_model import ModelType + from verl.utils.model import print_model_size, update_model_config, get_generation_config + from verl.utils.megatron_utils import get_model, init_megatron_optim_config + from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig, GenerationConfig + + # Step 1: initialize the tokenizer + local_path = copy_local_path_from_hdfs(model_path) + self.tokenizer = hf_tokenizer(local_path) + + # Step 2: get the actor_model_config + actor_model_config = AutoConfig.from_pretrained(local_path) + + self.generation_config = get_generation_config(local_path) + + override_config_kwargs = { + 'bos_token_id': self.tokenizer.bos_token_id, + 'eos_token_id': self.tokenizer.eos_token_id, + 'pad_token_id': self.tokenizer.pad_token_id, + } + override_config_kwargs.update(override_model_config) + update_model_config(actor_model_config, override_config_kwargs=override_config_kwargs) + + if self.rank == 0: + print(f'Model config after override: {actor_model_config}') + + def megatron_actor_model_provider(pre_process, post_process): + from verl.utils.model import get_parallel_model_from_config + # vpp is not supported yet because it will hang for some reason. Need debugging + vpp_rank = mpu.get_virtual_pipeline_model_parallel_rank() # this will be set inside get_model + # this_megatron_config = copy.deepcopy(megatron_config) + # this_megatron_config.virtual_pipeline_model_parallel_rank = vpp_rank + parallel_model = get_parallel_model_from_config(config=actor_model_config, + megatron_config=megatron_config, + pre_process=pre_process, + post_process=post_process, + value=False) + parallel_model.cuda() + return parallel_model + + # Step 3: initialize the megatron model + if self._is_actor and self._is_rollout: + # Initialize the 3D HybridEngine + hybrid_engine = AllGatherPPModel(model_provider=megatron_actor_model_provider) + # Fetch the model at current rank + actor_module = hybrid_engine.this_rank_models + if isinstance(actor_module, nn.ModuleList): + actor_module = [actor_module[0]] + if self.config.actor.load_weight: + load_megatron_model_weights(self.config, + actor_model_config, + actor_module, + params_dtype=megatron_config.params_dtype, + is_value_model=False) + + if self.rank == 0: + print_model_size(actor_module[0]) + log_gpu_memory_usage('After AllGatherPPModel init', logger=logger) + elif self._is_ref: + print(f'self.config.ref.load_weight: {self.config.ref.load_weight}') + ref_module = get_model(model_provider_func=megatron_actor_model_provider, + model_type=ModelType.encoder_or_decoder, + wrap_with_ddp=False) + # ref_module = nn.ModuleList(ref_module) + + if self.config.ref.load_weight: # should align with the actor: + assert self.config.actor.load_weight == self.config.ref.load_weight + print(f'load ref weight start') + load_megatron_model_weights(self.config, + actor_model_config, + ref_module, + params_dtype=megatron_config.params_dtype, + is_value_model=False) + log_gpu_memory_usage('After ref module init', logger=logger) + return ref_module, actor_model_config + + # TODO: add more optimizer args into config + if self._is_actor: + optim_config = init_megatron_optim_config(optim_config) + actor_optimizer = get_megatron_optimizer(model=actor_module, config=optim_config) + else: + optim_config = None + actor_optimizer = None + + log_gpu_memory_usage('After actor optimizer init', logger=logger) + + return actor_module, hybrid_engine, actor_optimizer, actor_model_config, optim_config + + def _build_rollout(self): + if self.config.rollout.name == 'vllm': + from verl.workers.rollout.vllm_rollout import vLLMRollout, vllm_mode + from verl.workers.sharding_manager import MegatronVLLMShardingManager + from verl.utils.model import normalize_pp_vpp_params + + # NOTE(sgm): If the QKV and gate_up projection layer are concate together in actor, + # we will reorganize their weight format when resharding from actor to rollout. + layer_name_mapping = { + "qkv_layer_name": + self.config.rollout.layer_name_map.get("qkv_layer_name", "qkv"), + "gate_proj_layer_name": + self.config.rollout.layer_name_map.get("gate_proj_layer_name", "linear_fc1.weight"), + } + + # reshard the weight partition from actor to rollout to initialize the rollout class + # create a new cuda space for parameters not in this pp rank + self.hybrid_engine.load_params_to_cuda() + # broadcast the parameters from pp rank to other ranks + self.hybrid_engine.allgather_params() + # obtain name to parameters in pp/vpp + params = self.hybrid_engine.get_all_params() + # update the param name for the + params = normalize_pp_vpp_params(params=params, + num_hidden_layers=self.actor_model_config.num_hidden_layers, + layer_name='layers') + assert vllm_mode == 'customized', "Support for vllm>=0.7 for Megatron-LM backend has not been implemented yet." + rollout = vLLMRollout(actor_module=params, + config=self.config.rollout, + tokenizer=self.tokenizer, + model_hf_config=self.actor_model_config, + train_tp=mpu.get_tensor_model_parallel_world_size()) + log_gpu_memory_usage('After building vllm rollout', logger=logger) + + # perform weight resharding between actor and rollout + sharding_manager = MegatronVLLMShardingManager(module=self.hybrid_engine, + inference_engine=rollout.inference_engine, + model_config=self.actor_model_config, + layer_name_mapping=layer_name_mapping) + log_gpu_memory_usage('After building sharding manager', logger=logger) + else: + NotImplementedError('Only vllmRollout is supported with Megatron now') + + return rollout, sharding_manager + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def init_model(self): + if self.config.model.get('external_lib', None) is not None: + # This is used to import external_lib into the huggingface systems + import importlib + importlib.import_module(self.config.model.external_lib) + + from omegaconf import OmegaConf + from verl.utils.torch_dtypes import PrecisionType + override_model_config = OmegaConf.to_container(self.config.model.get('override_config', OmegaConf.create())) + torch_dtype = torch.bfloat16 + + megatron_config = OmegaConf.create({ + 'sequence_parallel': self.config.actor.megatron.get('sequence_parallel', True), + 'param_dtype': PrecisionType.to_str(torch_dtype), + 'tensor_model_parallel_size': mpu.get_tensor_model_parallel_world_size(), + 'pipeline_model_parallel_rank': mpu.get_pipeline_model_parallel_rank(), + 'pipeline_model_parallel_size': mpu.get_pipeline_model_parallel_world_size(), + 'virtual_pipeline_model_parallel_rank': mpu.get_virtual_pipeline_model_parallel_rank(), + 'virtual_pipeline_model_parallel_size': mpu.get_virtual_pipeline_model_parallel_world_size() + }) + + megatron_config = init_model_parallel_config(megatron_config) + + if self._is_actor or self._is_rollout: + # we need the model for actor and rollout + if self._is_actor: + optim_config = self.config.actor.optim + else: + optim_config = None + self.actor_module, self.hybrid_engine, self.actor_optimizer, \ + self.actor_model_config, self.actor_optim_config = self._build_model_optimizer( + model_path=self.config.model.path, + megatron_config=megatron_config, + optim_config=optim_config, + override_model_config=override_model_config, + ) + + if self._is_actor: + self.actor = MegatronPPOActor(config=self.config.actor, + model_config=self.actor_model_config, + megatron_config=megatron_config, + actor_module=self.actor_module, + actor_optimizer=self.actor_optimizer, + actor_optimizer_config=self.actor_optim_config) + + if self._is_rollout: + self.rollout, self.sharding_manager = self._build_rollout() + + if self._is_ref: + self.ref_module, self.ref_model_config = self._build_model_optimizer( + model_path=self.config.model.path, + megatron_config=megatron_config, + optim_config=None, + override_model_config=override_model_config, + ) + self.ref_policy = MegatronPPOActor(config=self.config.ref, + model_config=self.ref_model_config, + megatron_config=megatron_config, + actor_module=self.ref_module, + actor_optimizer=None, + actor_optimizer_config=None) + + torch.cuda.empty_cache() + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def update_actor(self, data: DataProto): + assert self._is_actor + + data.batch = data.batch.cuda() + + log_gpu_memory_usage('Before update policy', logger=logger) + + dataloader = self.actor.make_minibatch_iterator(data=data) + metrics = self.actor.update_policy(dataloader=dataloader) + + log_gpu_memory_usage('After update policy', logger=logger) + + # TODO: here, we should return all metrics + output = DataProto(meta_info={'metrics': metrics}) + output = output.to('cpu') + torch.cuda.empty_cache() + return output + + @register(dispatch_mode=Dispatch.MEGATRON_PP_AS_DP_PROTO) + def generate_sequences(self, prompts: DataProto): + assert self._is_rollout + + prompts.batch = prompts.batch.cuda() + meta_info = { + 'eos_token_id': + self.generation_config.eos_token_id + if self.generation_config is not None else self.tokenizer.eos_token_id, + 'pad_token_id': + self.generation_config.pad_token_id + if self.generation_config is not None else self.tokenizer.pad_token_id, + } + prompts.meta_info.update(meta_info) + with self.sharding_manager: + log_gpu_memory_usage('After entering sharding manager', logger=logger) + + prompts = self.sharding_manager.preprocess_data(prompts) + output = self.rollout.generate_sequences(prompts=prompts) + + log_gpu_memory_usage('After rollout generation', logger=logger) + + output = self.sharding_manager.postprocess_data(output) + + output = output.to('cpu') + # clear kv cache + torch.cuda.empty_cache() + log_gpu_memory_usage('After recompute log prob', logger=logger) + return output + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def compute_ref_log_prob(self, data: DataProto): + data = data.to('cuda') + + assert self._is_ref + if self._is_offload_param: + load_megatron_param_and_grad(self.ref_module, torch.cuda.current_device(), self._is_offload_grad) + + micro_batch_size = self.config.rollout.log_prob_micro_batch_size_per_gpu + data.meta_info['micro_batch_size'] = micro_batch_size + data.meta_info['temperature'] = self.config.rollout.temperature + output = self.ref_policy.compute_log_prob(data=data) + output = DataProto.from_dict(tensors={'ref_log_prob': output}) + output = output.to('cpu') + if self._is_offload_param: + offload_megatron_param_and_grad(self.ref_module, self._is_offload_grad) + torch.cuda.empty_cache() + return output + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def compute_log_prob(self, data: DataProto): + assert self._is_actor + data = data.to('cuda') + output = data + # we should always recompute old_log_probs when it is HybridEngine + output.meta_info['micro_batch_size'] = self.config.rollout.log_prob_micro_batch_size_per_gpu + output.meta_info['temperature'] = self.config.rollout.temperature + old_log_probs = self.actor.compute_log_prob(data=output) + output.batch['old_log_probs'] = old_log_probs + output = output.to('cpu') + # clear kv cache + torch.cuda.empty_cache() + log_gpu_memory_usage('After recompute log prob', logger=logger) + return output + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def load_checkpoint(self, checkpoint_path, **kwargs): + pass + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def load_pretrained_model(self, checkpoint_path, **kwargs): + pass + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def save_checkpoint(self, checkpoint_path, **kwargs): + assert self._is_actor + pass + + +class CriticWorker(MegatronWorker): + + def __init__(self, config): + super().__init__() + self.config = config + + # NOTE(sgm): We utilize colocate WorkerGroup by default. + # As a result, Workers for different model share the same process. + # Therefore, we only require one distribute initialization. + # To utilize different parallel startegy in different models: + # 1, users should disable WorkerDict; 2.assign different ResourcePool to different models, + # 3. and apply the following patch in ray==2.10, https://github.com/ray-project/ray/pull/44385 + if not torch.distributed.is_initialized(): + rank = int(os.environ['LOCAL_RANK']) + torch.distributed.init_process_group(backend="nccl") + torch.cuda.set_device(rank) + + if self.config.megatron.sequence_parallel: + os.environ['CUDA_DEVICE_MAX_CONNECTIONS'] = '1' + mpu.initialize_model_parallel( + tensor_model_parallel_size=self.config.megatron.tensor_model_parallel_size, + pipeline_model_parallel_size=self.config.megatron.pipeline_model_parallel_size, + virtual_pipeline_model_parallel_size=None, + pipeline_model_parallel_split_rank=None, + use_sharp=False, + context_parallel_size=1, + expert_model_parallel_size=1, + nccl_communicator_config_path=None, + ) + + set_random_seed(seed=self.config.megatron.seed) + + # normalize config + self.config.ppo_mini_batch_size //= mpu.get_data_parallel_world_size() + if self.config.get('ppo_micro_batch_size', None): + self.config.ppo_micro_batch_size //= mpu.get_data_parallel_world_size() + self.config.ppo_micro_batch_size_per_gpu = self.config.ppo_micro_batch_size + + # TODO(sgm): support critic model offload + + def _build_critic_model_optimizer(self, + model_path, + megatron_config: ModelParallelConfig, + optim_config, + override_model_config, + enable_gradient_checkpointing=False): + from megatron.core.models.gpt.gpt_model import ModelType + from verl.utils.model import print_model_size, update_model_config + from verl.utils.megatron.optimizer import get_megatron_optimizer + from verl.utils.megatron_utils import get_model, init_megatron_optim_config, init_model_parallel_config + from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig + + # Step 1: initialize the tokenizer + local_path = copy_local_path_from_hdfs(model_path) + self.tokenizer = hf_tokenizer(local_path) + + # Step 2: get the actor_model_config + critic_model_config = AutoConfig.from_pretrained(local_path) + + override_config_kwargs = { + 'bos_token_id': self.tokenizer.bos_token_id, + 'eos_token_id': self.tokenizer.eos_token_id, + 'pad_token_id': self.tokenizer.pad_token_id, + } + override_config_kwargs.update(override_model_config) + update_model_config(critic_model_config, override_config_kwargs=override_config_kwargs) + + if self.rank == 0: + print(f'Model config after override: {critic_model_config}') + + def megatron_critic_model_provider(pre_process, post_process): + from verl.utils.model import get_parallel_model_from_config + # TODO: support vpp here + # vpp_rank = mpu.get_virtual_pipeline_model_parallel_rank() # this will be set inside get_model + # this_megatron_config = copy.deepcopy(megatron_config) + # this_megatron_config.virtual_pipeline_model_parallel_rank = vpp_rank + parallel_model = get_parallel_model_from_config(config=critic_model_config, + megatron_config=megatron_config, + pre_process=pre_process, + post_process=post_process, + value=True) + parallel_model.cuda() + return parallel_model + + # Step 3: initialize the megatron model + critic_module = get_model(model_provider_func=megatron_critic_model_provider, + model_type=ModelType.encoder_or_decoder, + wrap_with_ddp=True) + # note that here critic_module will be a list to be compatible with the construction of interleaved pp (vpp). + # but here, we do not use pp (vpp) yet. For simplicity, we remove the list + # critic_module = nn.ModuleList(critic_module) + + if self.config.load_weight: + load_megatron_model_weights(self.config, + critic_model_config, + critic_module, + params_dtype=megatron_config.params_dtype, + is_value_model=True) + if self.rank == 0: + print_model_size(critic_module[0]) + + # TODO: add more optimizer args into config + optim_config = init_megatron_optim_config(optim_config) + critic_optimizer = get_megatron_optimizer(model=critic_module, config=optim_config) + torch.cuda.empty_cache() + return critic_module, critic_optimizer, critic_model_config, optim_config + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def init_model(self): + # create critic + from omegaconf import OmegaConf + from verl.utils.torch_dtypes import PrecisionType + + if self.config.model.get('external_lib', None) is not None: + # This is used to import external_lib into the huggingface systems + import importlib + importlib.import_module(self.config.model.external_lib) + override_model_config = OmegaConf.to_container(self.config.model.get('override_config', OmegaConf.create())) + torch_dtype = torch.bfloat16 + + megatron_config = OmegaConf.create({ + 'sequence_parallel': self.config.megatron.get('sequence_parallel', True), + 'param_dtype': PrecisionType.to_str(torch_dtype), + 'tensor_model_parallel_size': mpu.get_tensor_model_parallel_world_size(), + 'pipeline_model_parallel_rank': mpu.get_pipeline_model_parallel_rank(), + 'pipeline_model_parallel_size': mpu.get_pipeline_model_parallel_world_size(), + 'virtual_pipeline_model_parallel_rank': mpu.get_virtual_pipeline_model_parallel_rank(), + 'virtual_pipeline_model_parallel_size': mpu.get_virtual_pipeline_model_parallel_world_size() + }) + + megatron_config = init_model_parallel_config(megatron_config) + + critic_module, critic_optimizer, critic_model_config, critic_optimizer_config = self._build_critic_model_optimizer( + model_path=self.config.model.path, + megatron_config=megatron_config, + optim_config=self.config.optim, + override_model_config=override_model_config) + self.critic = MegatronPPOCritic(config=self.config, + model_config=critic_model_config, + megatron_config=megatron_config, + critic_module=critic_module, + critic_optimizer=critic_optimizer, + critic_optimizer_config=critic_optimizer_config) + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def compute_values(self, data: DataProto): + data = data.to('cuda') + values = self.critic.compute_values(data=data) + output = DataProto.from_dict(tensors={'values': values}) + output = output.to('cpu') + return output + + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def update_critic(self, data: DataProto): + data = data.to('cuda') + dataloader = self.critic.make_minibatch_iterator(data) + metrics = self.critic.update_critic(dataloader=dataloader) + output = DataProto(batch=None, meta_info={'metrics': metrics}) + output = output.to('cpu') + return output + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def load_checkpoint(self, checkpoint_path, **kwargs): + pass + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def save_checkpoint(self, checkpoint_path, **kwargs): + pass + + +class RewardModelWorker(MegatronWorker): + """ + Note that we only implement the reward model that is subclass of AutoModelForSequenceClassification. + """ + + def __init__(self, config): + super().__init__() + self.config = config + + # NOTE(sgm): We utilize colocate WorkerGroup by default. + # As a result, Workers for different model share the same process. + # Therefore, we only require one distribute initialization. + # To utilize different parallel startegy in different models: + # 1, users should disable WorkerDict; 2.assign different ResourcePool to different models, + # 3. and apply the following patch in ray==2.10, https://github.com/ray-project/ray/pull/44385 + if not torch.distributed.is_initialized(): + rank = int(os.environ['LOCAL_RANK']) + torch.distributed.init_process_group(backend="nccl") + torch.cuda.set_device(rank) + + if self.config.megatron.sequence_parallel: + os.environ['CUDA_DEVICE_MAX_CONNECTIONS'] = '1' + mpu.initialize_model_parallel( + tensor_model_parallel_size=self.config.megatron.tensor_model_parallel_size, + pipeline_model_parallel_size=self.config.megatron.pipeline_model_parallel_size, + virtual_pipeline_model_parallel_size=None, + pipeline_model_parallel_split_rank=None, + use_sharp=False, + context_parallel_size=1, + expert_model_parallel_size=1, + nccl_communicator_config_path=None, + ) + + set_random_seed(seed=self.config.megatron.seed) + + # normalize config + if self.config.micro_batch_size is not None: + self.config.micro_batch_size //= mpu.get_data_parallel_world_size() + self.config.micro_batch_size_per_gpu = self.config.micro_batch_size + + def _build_rm_model(self, model_path, megatron_config: ModelParallelConfig, override_model_config): + from megatron.core.models.gpt.gpt_model import ModelType + from verl.utils.model import print_model_size, update_model_config + from verl.utils.megatron_utils import get_model + from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig + + # Step 1: initialize the tokenizer + local_path = copy_local_path_from_hdfs(model_path) + self.tokenizer = hf_tokenizer(local_path) + + # Step 2: get the actor_model_config + rm_model_config = AutoConfig.from_pretrained(local_path) + + override_config_kwargs = { + 'bos_token_id': self.tokenizer.bos_token_id, + 'eos_token_id': self.tokenizer.eos_token_id, + 'pad_token_id': self.tokenizer.pad_token_id, + } + override_config_kwargs.update(override_model_config) + update_model_config(rm_model_config, override_config_kwargs=override_config_kwargs) + + if self.rank == 0: + print(f'Model config after override: {rm_model_config}') + + def megatron_rm_model_provider(pre_process, post_process): + from verl.utils.model import get_parallel_model_from_config + # vpp is not supported yet because it will hang for some reason. Need debugging + vpp_rank = mpu.get_virtual_pipeline_model_parallel_rank() # this will be set inside get_model + # this_megatron_config = copy.deepcopy(megatron_config) + # this_megatron_config.virtual_pipeline_model_parallel_rank = vpp_rank + parallel_model = get_parallel_model_from_config(config=rm_model_config, + megatron_config=megatron_config, + pre_process=pre_process, + post_process=post_process, + value=True) + parallel_model.cuda() + return parallel_model + + # Step 3: initialize the megatron model + reward_model = get_model(model_provider_func=megatron_rm_model_provider, + model_type=ModelType.encoder_or_decoder, + wrap_with_ddp=False) + # note that here critic_module will be a list to be compatible with the construction of interleaved pp (vpp). + # but here, we do not use pp (vpp) yet. For simplicity, we remove the list + # reward_model = nn.ModuleList(reward_model) + + if self.config.load_weight: + load_megatron_model_weights(self.config, + rm_model_config, + reward_model, + params_dtype=megatron_config.params_dtype, + is_value_model=True) + + # TODO: add more optimizer args into config + torch.cuda.empty_cache() + return reward_model, rm_model_config + + @register(dispatch_mode=Dispatch.ONE_TO_ALL) + def init_model(self): + # create critic + from omegaconf import OmegaConf + from verl.utils.torch_dtypes import PrecisionType + from transformers import AutoTokenizer + + if self.config.model.get('external_lib', None) is not None: + # This is used to import external_lib into the huggingface systems + import importlib + importlib.import_module(self.config.model.external_lib) + override_model_config = OmegaConf.to_container(self.config.model.get('override_config', OmegaConf.create())) + + sft_tokenizer_local_path = copy_local_path_from_hdfs(self.config.model.input_tokenizer) + sft_tokenizer = hf_tokenizer(sft_tokenizer_local_path) + rm_tokenizer_path = self.config.model.get('rm_tokenizer', None) + rm_tokenizer = None + if rm_tokenizer_path is not None: + rm_tokenizer_local_path = copy_local_path_from_hdfs(rm_tokenizer_path) + rm_tokenizer = hf_tokenizer(rm_tokenizer_local_path) + + torch_dtype = torch.bfloat16 + + megatron_config = OmegaConf.create({ + 'sequence_parallel': self.config.megatron.get('sequence_parallel', True), + 'param_dtype': PrecisionType.to_str(torch_dtype), + 'tensor_model_parallel_size': mpu.get_tensor_model_parallel_world_size(), + 'pipeline_model_parallel_rank': mpu.get_pipeline_model_parallel_rank(), + 'pipeline_model_parallel_size': mpu.get_pipeline_model_parallel_world_size(), + 'virtual_pipeline_model_parallel_rank': mpu.get_virtual_pipeline_model_parallel_rank(), + 'virtual_pipeline_model_parallel_size': mpu.get_virtual_pipeline_model_parallel_world_size() + }) + + megatron_config = init_model_parallel_config(megatron_config) + + reward_model_module, reward_model_config = self._build_rm_model( + model_path=self.config.model.path, + megatron_config=megatron_config, + override_model_config=override_model_config, + ) + # FIXME(sgm): reward model param offload is implemented in MegatronRewardModel + # should be implemented in workers + self.rm = MegatronRewardModel(config=self.config, + reward_model_module=reward_model_module, + model_config=reward_model_config, + megatron_config=megatron_config, + sft_tokenizer=sft_tokenizer, + rm_tokenizer=rm_tokenizer) + + # TODO: reward model use itself tokenizer instead of sft tokenizer + # the input_ids, responses, attention_mask and position_ids may be different! + @register(dispatch_mode=Dispatch.MEGATRON_COMPUTE_PROTO) + def compute_rm_score(self, data: DataProto): + data.batch = data.batch.cuda() + output = self.rm.compute_reward(data) + output = output.to('cpu') + return output diff --git a/frameworks/agentgym_rl/verl/workers/reward_manager/__init__.py b/frameworks/agentgym_rl/verl/workers/reward_manager/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..3de46a7717e5ee4d86db5ecbee56a7344e56be23 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/reward_manager/__init__.py @@ -0,0 +1,16 @@ +# Copyright 2024 PRIME team and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .naive import NaiveRewardManager +from .prime import PrimeRewardManager \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/workers/reward_manager/naive.py b/frameworks/agentgym_rl/verl/workers/reward_manager/naive.py new file mode 100644 index 0000000000000000000000000000000000000000..8cfae4cf6914e281b1ecd8a43b576e60e4ca396d --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/reward_manager/naive.py @@ -0,0 +1,76 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from verl import DataProto +from verl.utils.reward_score import _default_compute_score +import torch + + +class NaiveRewardManager: + """The reward manager. + """ + + def __init__(self, tokenizer, num_examine, compute_score=None) -> None: + self.tokenizer = tokenizer + self.num_examine = num_examine # the number of batches of decoded responses to print to the console + self.compute_score = compute_score or _default_compute_score + + def __call__(self, data: DataProto): + """We will expand this function gradually based on the available datasets""" + + # If there is rm score, we directly return rm score. Otherwise, we compute via rm_score_fn + if 'rm_scores' in data.batch.keys(): + return data.batch['rm_scores'] + + reward_tensor = torch.zeros_like(data.batch['responses'], dtype=torch.float32) + + already_print_data_sources = {} + + for i in range(len(data)): + data_item = data[i] # DataProtoItem + + prompt_ids = data_item.batch['prompts'] + + prompt_length = prompt_ids.shape[-1] + + valid_prompt_length = data_item.batch['attention_mask'][:prompt_length].sum() + valid_prompt_ids = prompt_ids[-valid_prompt_length:] + + response_ids = data_item.batch['responses'] + valid_response_length = data_item.batch['attention_mask'][prompt_length:].sum() + valid_response_ids = response_ids[:valid_response_length] + + # decode + sequences = torch.cat((valid_prompt_ids, valid_response_ids)) + sequences_str = self.tokenizer.decode(sequences) + + ground_truth = data_item.non_tensor_batch['reward_model']['ground_truth'] + + data_source = data_item.non_tensor_batch['data_source'] + + score = self.compute_score( + data_source=data_source, + solution_str=sequences_str, + ground_truth=ground_truth, + ) + reward_tensor[i, valid_response_length - 1] = score + + if data_source not in already_print_data_sources: + already_print_data_sources[data_source] = 0 + + if already_print_data_sources[data_source] < self.num_examine: + already_print_data_sources[data_source] += 1 + print(sequences_str) + + return reward_tensor diff --git a/frameworks/agentgym_rl/verl/workers/reward_manager/prime.py b/frameworks/agentgym_rl/verl/workers/reward_manager/prime.py new file mode 100644 index 0000000000000000000000000000000000000000..f2f8856fe7c07f44d0ad5e7b24cfe0182cb0e934 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/reward_manager/prime.py @@ -0,0 +1,134 @@ +# Copyright 2024 PRIME team and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +from concurrent.futures import ProcessPoolExecutor +from functools import partial + +import torch + +from verl import DataProto +from verl.utils.reward_score import _default_compute_score + + +async def single_compute_score(evaluation_func, completion, reference, task, executor, timeout=300.): + loop = asyncio.get_running_loop() + try: + # Ensure process_completion is called properly + tasks = [ + asyncio.wait_for( + loop.run_in_executor( + executor, + partial(evaluation_func, task, completion, reference) # Ensure synchronous + ), + timeout=timeout) + ] + return await asyncio.gather(*tasks) + except asyncio.TimeoutError: + print(f"Timeout occurred for completion: {completion}") + return None # Default value for timed-out rows + except Exception as e: + print(f"Error processing completion: {completion[:10]}, Error: {e}") + return None # Default value for failed rows + + +async def parallel_compute_score_async(evaluation_func, completions, references, tasks, num_processes=64): + scores = [] + with ProcessPoolExecutor(max_workers=num_processes) as executor: + # Create tasks for all rows + tasks_async = [ + single_compute_score(evaluation_func, completion, reference, task, executor, timeout=300.) + for completion, reference, task in zip(completions, references, tasks) + ] + # to prevent very occasional starvation caused by some anomalous programs ( like infinite loop ), the exceptions in async programs will instantly halt the evaluation, and all summoned processes will be killed. + try: + results = await asyncio.gather(*tasks_async, return_exceptions=False) + except: + for pid, proc in executor._processes.items(): + try: + proc.kill() + except Exception as kill_err: + print('shut down failed: ' + str(kill_err)) + raise + + # Process results + for result, completion, reference, task in zip(results, completions, references, tasks): + if isinstance(result, Exception) or result is None: + # Handle failed or timed-out tasks + scores.append(0.0) + elif isinstance(result[0], (int, float, bool)): + scores.append(float(result[0])) + else: + scores.append(float(result[0][0])) + return scores + + +class PrimeRewardManager: + """ + The Reward Manager used in https://github.com/PRIME-RL/PRIME + """ + + def __init__(self, tokenizer, num_examine, compute_score=None) -> None: + self.tokenizer = tokenizer + self.num_examine = num_examine # the number of batches of decoded responses to print to the console + self.compute_score = compute_score or _default_compute_score + + def __call__(self, data: DataProto): + """We will expand this function gradually based on the available datasets""" + + # If there is rm score, we directly return rm score. Otherwise, we compute via rm_score_fn + if 'rm_scores' in data.batch.keys(): + return data.batch['rm_scores'] + + reward_tensor = torch.zeros_like(data.batch['responses'], dtype=torch.float32) + + already_print_data_sources = {} + + # batched scoring + prompt_ids = data.batch['prompts'] + prompt_length = prompt_ids.shape[-1] + + response_ids = data.batch['responses'] + valid_response_length = data.batch['attention_mask'][:, prompt_length:].sum(dim=-1) + sequences_str = self.tokenizer.batch_decode(response_ids, skip_special_tokens=True) + ground_truth = [data_item.non_tensor_batch['reward_model']['ground_truth'] for data_item in data] + data_sources = data.non_tensor_batch['data_source'] + + assert len(sequences_str) == len(ground_truth) == len(data_sources) + try: + scores = asyncio.run( + parallel_compute_score_async(self.compute_score, + sequences_str, + ground_truth, + data_sources, + num_processes=64)) + except asyncio.TimeoutError as e: + print('Global timeout in reward computing! Setting all as 0.') + scores = [0. for _ in range(len(sequences_str))] + except Exception as e: + print(f"Unexpected error in batched reward computing. Setting all as 0.: {e}") + scores = [0. for _ in range(len(sequences_str))] + + for i in range(len(data)): + data_source = data_sources[i] + reward_tensor[i, valid_response_length[i].item() - 1] = scores[i] + + if data_source not in already_print_data_sources: + already_print_data_sources[data_source] = 0 + + if already_print_data_sources[data_source] < self.num_examine: + already_print_data_sources[data_source] += 1 + print(sequences_str) + + return reward_tensor diff --git a/frameworks/agentgym_rl/verl/workers/reward_model/__init__.py b/frameworks/agentgym_rl/verl/workers/reward_model/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a0b48a750841888b1e220b72422659d8073c22a0 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/reward_model/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .base import BasePPORewardModel diff --git a/frameworks/agentgym_rl/verl/workers/reward_model/base.py b/frameworks/agentgym_rl/verl/workers/reward_model/base.py new file mode 100644 index 0000000000000000000000000000000000000000..c02487db3846d0fcec76c1c216fbbb52d15c64bd --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/reward_model/base.py @@ -0,0 +1,45 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +The base class for reward model +""" + +from abc import ABC, abstractmethod + +from verl import DataProto + + +class BasePPORewardModel(ABC): + + def __init__(self, config): + self.config = config + + @abstractmethod + def compute_reward(self, data: DataProto) -> DataProto: + """Computing reward given input_ids. The transformers should output a tensor with shape + [batch_size, sequence_length], and the value at [EOS] mask should be gathered. + + Args: + data: must contain keys "input_ids", "attention_mask" and "position_ids". + - input_ids: [batch_size, sequence_length] + - attention_mask: [batch_size, sequence_length] + - position_ids: [batch_size, sequence_length] + + Returns: a data pass protocol containing "reward". Only the [EOS] position contains the reward. + Other position should have zero reward. Note that this may change in the future if we use + dense reward. So, we leave the interface for general case. + - reward: [batch_size, sequence_length]. + + """ + pass diff --git a/frameworks/agentgym_rl/verl/workers/reward_model/megatron/__init__.py b/frameworks/agentgym_rl/verl/workers/reward_model/megatron/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..b0956b4cc53b81bf4c675c235968e1fc577a49f9 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/reward_model/megatron/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .reward_model import MegatronRewardModel diff --git a/frameworks/agentgym_rl/verl/workers/reward_model/megatron/reward_model.py b/frameworks/agentgym_rl/verl/workers/reward_model/megatron/reward_model.py new file mode 100644 index 0000000000000000000000000000000000000000..1b58f42c86e84b73c0b73a9df9e343460a9b6927 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/reward_model/megatron/reward_model.py @@ -0,0 +1,275 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Megatron Reward Model. +""" + +from tensordict import TensorDict +from functools import partial +from verl import DataProto +from verl.utils.torch_functional import logprobs_from_logits +import torch +import torch +import torch.distributed + +from verl.utils.torch_functional import get_eos_mask, pad_sequence_to_length +from verl.utils.megatron.pipeline_parallel import (compute_transformers_input_shapes, make_batch_generator) +from verl import DataProto +from verl.utils.torch_functional import logprobs_from_logits, broadcast_dict_tensor, split_dict_tensor_into_batches +from verl.utils.torch_dtypes import PrecisionType +from verl.workers.reward_model.base import BasePPORewardModel +from verl.utils.megatron import sequence_parallel as sp_utils +from megatron.core import parallel_state as mpu +from megatron.core.pipeline_parallel import get_forward_backward_func + + +class MegatronRewardModel(BasePPORewardModel): + + def __init__(self, + config, + model_config, + reward_model_module: torch.nn.ModuleList, + megatron_config, + sft_tokenizer=None, + rm_tokenizer=None): + self.config = config + self.reward_model_module = reward_model_module + self.megatron_config = megatron_config + self.model_config = model_config + self.device = 'cuda' + self.sft_tokenizer = sft_tokenizer + self.rm_tokenizer = rm_tokenizer + self.use_different_tokenizer = rm_tokenizer is not None + + if self.config.param_offload: + self.offload_params_to_cpu() + + def re_encode_by_rm_tokenizer(self, data: DataProto) -> DataProto: + assert self.use_different_tokenizer, 're-encode need rm tokenizer not be None!' + # need to use rm tokenizer to re-generate input_ids, attention_mask and position_ids + # 1. remove pad for each sequence + # 2. decode by sft_tokenizer, remove sft system prompts + # 3. encode by rm_tokenizer with rm system prompts, get rm_input_ids + # 4. generate attention_mask and position_ids + input_ids = data.batch['input_ids'] # (bs, seq_len) + attention_mask = data.batch['attention_mask'] + position_ids = data.batch['position_ids'] + ori_values = {'input_ids': input_ids, 'attention_mask': attention_mask, 'position_ids': position_ids} + ori_bs, ori_seqlen = input_ids.size(0), input_ids.size(1) + input_ids_for_rm = [] + attention_mask_for_rm = [] + position_ids_for_rm = [] + print_decode = True + ori_seqlen = ori_seqlen + 128 + for id, mask in zip(input_ids, attention_mask): + # 1. remove pad for each sequence + non_zero_indices = torch.nonzero(mask).view(-1) + begin_pos, end_pos = non_zero_indices[0].item(), non_zero_indices[-1].item() + valid_id = id[begin_pos:end_pos + 1] + # 2. decode by sft_tokenizer, remove sft system prompts + decode_result = self.sft_tokenizer.decode(valid_id) + # workaround + decode_with_rm_chat = decode_result.replace("<|user|>\n", "[INST] ").replace( + "\n<|assistant|>\n", " [/INST]").replace(" \n<|assistant|>\n", " [/INST]") + "" + if print_decode and torch.distributed.get_rank() == 0: + # only print first decode result + print(f'device {torch.cuda.current_device()}: sft decode result:\n{decode_result}\n \ + \ndevice {torch.cuda.current_device()}: sft decode result with rm chat template:\n{decode_with_rm_chat}\n\n' + ) + print_decode = False + # 3. encode by rm_tokenizer + rm_input_ids = self.rm_tokenizer(decode_with_rm_chat, + return_tensors='pt')['input_ids'][0].to(input_ids.device) + # 4. generate attention_mask and position_ids + rm_attention_mask = torch.ones_like(rm_input_ids, device=input_ids.device) + cur_seqlen = rm_input_ids.shape[-1] + # NOTE(gh): the later reward compute will process the shape (bs, seqlen_pad_128) + if cur_seqlen > ori_seqlen: + print(f'warninig: rm encode seqlen {cur_seqlen} > sft encode seqlen {ori_seqlen}') + rm_input_ids = rm_input_ids[:ori_seqlen] + rm_attention_mask = rm_attention_mask[:ori_seqlen] + else: + # right padding + rm_input_ids = pad_sequence_to_length(rm_input_ids, ori_seqlen, self.rm_tokenizer.pad_token_id) + rm_attention_mask = pad_sequence_to_length(rm_attention_mask, ori_seqlen, 0) + rm_position_ids = torch.arange(0, ori_seqlen, device=input_ids.device) + input_ids_for_rm.append(torch.unsqueeze(rm_input_ids, dim=0)) + attention_mask_for_rm.append(torch.unsqueeze(rm_attention_mask, dim=0)) + position_ids_for_rm.append(torch.unsqueeze(rm_position_ids, dim=0)) + input_ids_for_rm = torch.cat(input_ids_for_rm, dim=0) + attention_mask_for_rm = torch.cat(attention_mask_for_rm, dim=0) + position_ids_for_rm = torch.cat(position_ids_for_rm, dim=0) + + # (bs, seqlen) will not change, but input_ids, attention_mask and position_ids will change + # NOTE(gh): need to replace into origin values after compute reward! + data.batch['input_ids'] = input_ids_for_rm + data.batch['attention_mask'] = attention_mask_for_rm + data.batch['position_ids'] = position_ids_for_rm + + return data, ori_values + + @torch.no_grad() + def compute_reward(self, data: DataProto) -> DataProto: + if self.config.param_offload: + self.load_params_to_cuda() + + if self.use_different_tokenizer: + data, ori_values = self.re_encode_by_rm_tokenizer(data) + + input_ids = data.batch['input_ids'] # (bs, seq_len') + attention_mask = data.batch['attention_mask'] + position_ids = data.batch['position_ids'] + + responses = data.batch['responses'] + batch_size = responses.size(0) + response_length = responses.size(1) + + with torch.no_grad(): + output = self.forward_batch(data) + if mpu.is_pipeline_last_stage(ignore_virtual=True): + logits = torch.cat([o['logits'] for o in output], dim=0) + else: + logits = torch.empty( + (input_ids.shape[0], input_ids.shape[1]), + dtype=torch.bfloat16, # TODO(sgm): check why is bfloat16 + device=input_ids.device) + # broadcast across pp ranks + torch.distributed.broadcast(tensor=logits, + src=mpu.get_pipeline_model_parallel_last_rank(), + group=mpu.get_pipeline_model_parallel_group(), + async_op=False) + + # (bs, seqlen', hidden_size) -> (bs, seqlen', 1) -> (bs, seqlen') + token_level_rewards = logits + # find the last token reward + ends = attention_mask.cumsum(dim=-1).argmax(dim=-1).view(-1, 1) # (bs, 1) + rewards = torch.gather(token_level_rewards, dim=1, index=ends) # (bs, 1) + + if self.use_different_tokenizer: + data.batch.update(ori_values) + input_ids = ori_values['input_ids'] + attention_mask = ori_values['attention_mask'] + position_ids = ori_values['position_ids'] + + token_level_rewards = rewards.expand(attention_mask.shape[0], attention_mask.shape[1]) # (bs, ori_seqlen) + + # assign last valid token reward to ori position + eos_mask_idx = torch.argmax(position_ids * attention_mask, dim=-1) # (bs,) + eos_mask = torch.zeros_like(attention_mask) + eos_mask[torch.arange(batch_size), eos_mask_idx] = 1. + + token_level_rewards = token_level_rewards * eos_mask + token_level_rewards = token_level_rewards[:, -response_length:] + + if self.config.param_offload: + self.offload_params_to_cpu() + else: + # add empty cache after each compute + torch.cuda.empty_cache() + + batch = TensorDict({'rm_scores': token_level_rewards}, batch_size=input_ids.shape[0]) + + return DataProto(batch=batch) + + def forward_batch(self, data: DataProto): + """ + We assume: + - The model takes input: (input_ids, attention_mask, position_ids). No rmpad for the input + - The communication shape is (total_nnz_pad_to_sp // tp_size, 1, hidden_size) if sequence parallel is enabled + """ + # broadcast from last pp rank to all other pp ranks + # TODO: actually, we just need to control the sampling order. + data.batch = data.batch.contiguous() + broadcast_dict_tensor(data.batch, + src=mpu.get_pipeline_model_parallel_last_rank(), + group=mpu.get_pipeline_model_parallel_group()) + + # split into micro-batches + if self.config is not None and 'ppo_micro_batch_size_per_gpu' in self.config: + infer_batch_size = self.config.ppo_micro_batch_size_per_gpu + else: + infer_batch_size = data.batch.batch_size[0] + + data.batch['attention_mask'] = data.batch['attention_mask'].to(bool) + batches = split_dict_tensor_into_batches(data.batch, batch_size=infer_batch_size) + n_micro_batch = len(batches) + seq_len = batches[0]['input_ids'].shape[1] + + # compute input shapes for pp stages + input_shapes = compute_transformers_input_shapes( + batches, + meta_info={ + 'sequence_parallel': self.megatron_config.sequence_parallel, + 'hidden_size': self.model_config.hidden_size + }) + # compute input shapes for pp stages + forward_backward_func = get_forward_backward_func() + + def loss_func(output): + return 1., {'logits': output.logits} + + def forward_step(batch_iter, model): + batch = next(batch_iter) + input_ids = batch['input_ids'] + attention_mask = batch['attention_mask'] + position_ids = batch['position_ids'] + output = model(input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids) + return output, loss_func + + # batch should be a list of batches inside micro-batches + batch_generator = make_batch_generator(batches, vpp_size=len(self.reward_model_module)) + + # TODO: we may use the new schedule instead + # for flash-attn: (seq_len, batch_size, hidden_size) = (mbs*seq_len, 1, hidden_size) + if mpu.get_pipeline_model_parallel_world_size() > 1: + losses_reduced = forward_backward_func( + forward_step_func=forward_step, + data_iterator=batch_generator, + model=self.reward_model_module, + num_microbatches=n_micro_batch, + input_shapes=input_shapes, # must set for flash-attn sequence packing + seq_length=infer_batch_size * seq_len, # no use when input_shapes was set + hidden_size=self.model_config.hidden_size, # no use when input_shapes was set + micro_batch_size=1, # no use when input_shapes was set + forward_only=True, + ) + else: + losses_reduced = forward_backward_func( + forward_step_func=forward_step, + data_iterator=batch_generator, + model=self.reward_model_module, + num_microbatches=n_micro_batch, + seq_length=infer_batch_size * seq_len, # in use for pp = 1 + hidden_size=self.model_config.hidden_size, # in use for pp = 1 + micro_batch_size=1, # in use for pp = 1 + forward_only=True, + ) + # loss_reduces contains the stats returned from loss_func + + return losses_reduced + + def offload_params_to_cpu(self): + if self.device == 'cuda': + for reward_model_module in self.reward_model_module: + for name, param in reward_model_module.named_parameters(): + param.data = param.data.to('cpu', non_blocking=True) + self.device = 'cpu' + torch.cuda.empty_cache() + + def load_params_to_cuda(self): + if self.device == 'cpu': + for reward_model_module in self.reward_model_module: + for name, param in reward_model_module.named_parameters(): + param.data = param.data.to(torch.cuda.current_device(), non_blocking=True) + self.device = 'cuda' diff --git a/frameworks/agentgym_rl/verl/workers/rollout/__init__.py b/frameworks/agentgym_rl/verl/workers/rollout/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..083848c77faafa61d2a449e23707431925fafb40 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/__init__.py @@ -0,0 +1,19 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .base import BaseRollout +from .naive import NaiveRollout +from .hf_rollout import HFRollout + +__all__ = ["BaseRollout", "NaiveRollout", "HFRollout"] diff --git a/frameworks/agentgym_rl/verl/workers/rollout/agent_vllm_rollout/__init__.py b/frameworks/agentgym_rl/verl/workers/rollout/agent_vllm_rollout/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..49fabc132ccdd6afa5dcff8abdff32270af7782a --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/agent_vllm_rollout/__init__.py @@ -0,0 +1,33 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from importlib.metadata import version, PackageNotFoundError + + +def get_version(pkg): + try: + return version(pkg) + except PackageNotFoundError: + return None + + +package_name = 'vllm' +package_version = get_version(package_name) + +if package_version <= '0.6.3': + vllm_mode = 'customized' + from .vllm_rollout import vLLMRollout +# else: +# vllm_mode = 'spmd' +# from .vllm_rollout_spmd import vLLMRollout \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/workers/rollout/agent_vllm_rollout/vllm_rollout.py b/frameworks/agentgym_rl/verl/workers/rollout/agent_vllm_rollout/vllm_rollout.py new file mode 100644 index 0000000000000000000000000000000000000000..a08c6723128288d2700f6eed0c181a9de506d14f --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/agent_vllm_rollout/vllm_rollout.py @@ -0,0 +1,369 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +The vllm_rollout that can be applied in different backend +When working with FSDP: +- Use DTensor weight loader (recommended) or HF weight loader +- Utilize state_dict from the FSDP to synchronize the weights among tp ranks in vLLM +When working with Megatron: +- Use Megatron weight loader +- During training, only the current pp stage holds the parameters +- Before inference, broadcast the parameters of the current pp rank to all other pp ranks (all pp ranks holds all the parameters) +- Bind the parameters to the inference engine +- Do inference in tp. pp is treated as additional dp +- After inference, all the parameters that doesn't belong to this pp rank is freed. +""" +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager +from typing import List +from omegaconf import DictConfig +import torch +import torch.distributed +from torch.nn.utils.rnn import pad_sequence +from tensordict import TensorDict +from torch import nn +from tqdm import tqdm + +from verl import DataProto +from verl.workers.rollout.base import BaseRollout +from verl.third_party.vllm import LLM, vllm_version +from verl.third_party.vllm import parallel_state as vllm_ps +from vllm import SamplingParams + +import os +import json +import time +import requests +from copy import deepcopy +from verl.utils.model import compute_position_id_with_mask +from verl.utils.torch_functional import get_eos_mask, pad_sequence_to_length +from verl.utils.agentgym.client import init_env_client +from verl.workers.rollout.schemas import RolloutHandler, Message, _pre_process_inputs + +# TODO +# 1. support pp in vllm +# 2. passing tokenizer is not necessary? no encoding/decoding is happending here +# 3. simplify init logics + +class vLLMRollout(BaseRollout): + + def __init__(self, actor_module: nn.Module, rollout_config: DictConfig, agentgym_config: DictConfig, tokenizer, model_hf_config, **kwargs): + """A vLLM rollout. It requires the module is supported by the vllm. + + Args: + module: module here follows huggingface APIs + config: DictConfig + tokenizer: the task/model tokenizer + model_hf_config: the huggingface config to initiallize the generating model in vllm + **kwargs: train_tp, for Megatron Backend to initialize hybrid engine (zero redundancy) process group + """ + super().__init__() + self.config = rollout_config + self.agentgym_config = agentgym_config + assert not (not rollout_config.enforce_eager and rollout_config.free_cache_engine), \ + "disable CUDA graph (enforce_eager = False) if free cache engine" + + tensor_parallel_size = self.config.get('tensor_model_parallel_size', 1) + assert tensor_parallel_size <= torch.distributed.get_world_size(), \ + "tensor parallel size should be less than or equal to the world size" + max_num_batched_tokens = self.config.get('max_num_batched_tokens', 8192) + + if kwargs.get('train_tp', None) is not None: + # deployed with megatron + import os + os.environ['CUDA_TIMER_STREAM_KAFKA_ENABLE'] = '0' + os.environ['MEGATRON_IMPORT_TIMERS'] = '0' + train_tp = kwargs.get('train_tp', None) + num_tp_per_train_tp = train_tp // tensor_parallel_size + if vllm_version in ('0.4.2', '0.5.4', '0.6.3'): + vllm_ps.initialize_parallel_state(tensor_model_parallel_size=tensor_parallel_size, + num_tp_per_train_tp=num_tp_per_train_tp) + + self.inference_engine = LLM( + actor_module, + tokenizer=tokenizer, + model_hf_config=model_hf_config, + tensor_parallel_size=tensor_parallel_size, + dtype=rollout_config.dtype, + enforce_eager=rollout_config.enforce_eager, + gpu_memory_utilization=rollout_config.gpu_memory_utilization, + skip_tokenizer_init=False, + load_format=rollout_config.load_format, + disable_log_stats=rollout_config.disable_log_stats, + max_num_batched_tokens=max_num_batched_tokens, + enable_chunked_prefill=rollout_config.enable_chunked_prefill, + ) + + # Offload vllm model to reduce peak memory usage + self.inference_engine.offload_model_weights() + + kwargs = dict( + n=1, + logprobs=1, # can be set to 0 and let actor to recompute + max_tokens=rollout_config.max_tokens, + ) + + # we may detokenize the result all together later + if vllm_version in ('0.4.2', '0.5.4', '0.6.3'): + kwargs['detokenize'] = False + + # supporting adding any sampling params from the config file + for k in rollout_config.keys(): + if hasattr(SamplingParams(), str(k)): + kwargs[k] = rollout_config.get(k) + kwargs["n"] = 1 # because we have repeated task n times + + print(f"kwargs: {kwargs}") + self.sampling_params = SamplingParams(**kwargs) + + self.pad_token_id = tokenizer.pad_token_id + + self.tokenizer = tokenizer + + + @contextmanager + def update_sampling_params(self, **kwargs): + # update sampling params + old_sampling_params_args = {} + if kwargs: + for key, value in kwargs.items(): + if hasattr(self.sampling_params, key): + old_value = getattr(self.sampling_params, key) + old_sampling_params_args[key] = old_value + setattr(self.sampling_params, key, value) + yield + # roll back to previous sampling params + # if len(old_sampling_params_args): + for key, value in old_sampling_params_args.items(): + setattr(self.sampling_params, key, value) + + def preprocess_prompt_to_rollout_handler(self, prompts: DataProto, n: int) -> List[RolloutHandler]: + assert "raw_prompt" in prompts.non_tensor_batch.keys(), "raw_prompt is not in non_tensor_batch, need to set data.return_raw_chat=True" + handler_list = [] + for i, raw_prompt in enumerate(prompts.non_tensor_batch["raw_prompt"]): + for _ in range(n): + # only keep not pad part + input_ids = _pre_process_inputs(self.pad_token_id, prompts.batch['input_ids'][i]) + attention_mask = _pre_process_inputs(0, prompts.batch['attention_mask'][i]) + position_ids = compute_position_id_with_mask(torch.tensor(attention_mask)).tolist() + handler = RolloutHandler( + messages=[ + Message(role=prompt["role"], content=prompt["content"]) for prompt in raw_prompt + ], + task_name=prompts.non_tensor_batch["item_id"][i].split("_")[0], + item_id=int(prompts.non_tensor_batch["item_id"][i].split("_")[-1]), + score=0, + done=False, + input_ids=list(input_ids), + prompt_ids=list(input_ids), + response_ids=[], + attention_mask=list(attention_mask), + prompt_attention_mask=list(attention_mask), + response_attention_mask=[], + position_ids=list(position_ids), + prompt_position_ids=list(position_ids), + response_position_ids=[], + loss_mask=[0] * len(input_ids), + prompt_loss_mask=[0] * len(input_ids), + response_loss_mask=[], + max_response_len=self.config.response_length, + max_model_len=min(self.config.max_model_len, self.config.prompt_length + self.config.response_length) + ) + assert len(handler.input_ids) == len(handler.attention_mask) == len(handler.position_ids) == len(handler.loss_mask), f"RolloutHandler has mismatched length: input_ids={len(handler.input_ids)}, attention_mask={len(handler.attention_mask)}, position_ids={len(handler.position_ids)}, loss_mask={len(handler.loss_mask)}" + handler_list.append(handler) + return handler_list + + + @torch.no_grad() + def generate_sequences(self, prompts: DataProto, **kwargs) -> DataProto: + # rebuild vllm cache engine + if self.config.free_cache_engine: + self.inference_engine.init_cache_engine() + + global_steps = prompts.meta_info.get('global_steps', None) + max_rounds = prompts.meta_info.get('max_rounds', 10) + cur_device = prompts.batch["input_ids"].device + + do_sample = prompts.meta_info.get('do_sample', True) + if not do_sample: + kwargs = { + 'best_of': 1, + 'top_p': 1.0, + 'top_k': -1, + 'min_p': 0.0, + 'temperature': 0, + 'n': 1 # if greedy, only 1 response + } + + # repeat for self.config.n times to rollout + batch_size = prompts.batch['input_ids'].size(0) + batch_size *= self.config.n + rollout_handler_ls = self.preprocess_prompt_to_rollout_handler(prompts, n=self.config.n) + env_clients = [init_env_client(self.agentgym_config) for _ in range(batch_size)] + time.sleep(self.config.send_interval) # take a break before sendng request + all_done_flag = False + for idx, rollout_handler in enumerate(rollout_handler_ls): + try: + env_clients[idx].reset(rollout_handler.item_id) + task = env_clients[idx].observe() + rollout_handler.add_user_message(self.tokenizer, task) + except TimeoutError: + print(f"Reset Timeout: Webarena Env Timeout. item id = {rollout_handler.item_id}") + rollout_handler.done = True + rollout_handler.score = 0 + + rounds = 0 + task_rounds = [0] * batch_size + rollout_bar = tqdm(total = max_rounds, desc="Running rounds", disable=torch.distributed.get_rank() != 0) + def agent_step(i, idx): + content = self.tokenizer.decode(response_ids[i], skip_special_tokens=True) + rollout_handler_ls[idx].add_assistant_message(self.tokenizer, content) + task_rounds[idx] += 1 + try: + step_output = env_clients[idx].step(content) + state, rollout_handler_ls[idx].score, rollout_handler_ls[idx].done = ( + step_output.state, + step_output.reward, + step_output.done, + ) + rollout_handler_ls[idx].add_user_message(self.tokenizer, state) + return step_output.done + except Exception as e: + rollout_handler_ls[idx].score = 0 + rollout_handler_ls[idx].done = True + print(f"Rollou step Error: {e} item id = {rollout_handler_ls[idx].item_id}") + return True + while rounds < max_rounds and not all_done_flag: + # get generation prompt + generation_prompt_idxs = [] + not_done_idxs = [] + for idx, rollout_handler in enumerate(rollout_handler_ls): + if not rollout_handler.done: + generation_prompt_idxs.append(rollout_handler.get_generation_prompt(self.tokenizer)) + not_done_idxs.append(idx) + + rollout_bar.set_description(f"Rounds {rounds + 1}/{max_rounds} | Active agents per gpu: {len(not_done_idxs)}") + # users can customize different sampling_params at different run + with self.update_sampling_params(**kwargs): + output = self.inference_engine.generate( + prompts=None, + prompt_token_ids=generation_prompt_idxs, + sampling_params=self.sampling_params, + use_tqdm=False) + response_ids = output[0].tolist() + all_done_flag = True + time.sleep(self.config.send_interval) # take a break before sendng request + if len(not_done_idxs) > 0: + with ThreadPoolExecutor(max_workers=len(not_done_idxs)) as executor: + step_dones = list(executor.map( + lambda args: agent_step(*args), [(i, idx) for i, idx in enumerate(not_done_idxs)] + )) + all_done_flag = all(step_dones) + rounds += 1 + rollout_bar.update(1) + + # process ids + rollout_bar.close() + response_ids, response_attention_mask, response_position_ids, response_loss_mask = [], [], [], [] + scores, messages = [], [] + + for rollout_handler in rollout_handler_ls: + # check length + rollout_handler.truncate_output_ids() + assert len(rollout_handler.input_ids) == len(rollout_handler.attention_mask) == len(rollout_handler.position_ids) == len(rollout_handler.loss_mask), f"""Rollout Handler has different length of {len(rollout_handler.input_ids)=}, + {len(rollout_handler.attention_mask)=}, {len(rollout_handler.position_ids)=}, {len(rollout_handler.loss_mask)=}""" + assert len(rollout_handler.input_ids) <= self.config.max_model_len, f"Rollout Handler has sequence length {len(rollout_handler.input_ids)} > max_sequence_length {self.config.max_model_len}" + + response_ids.append(torch.tensor(rollout_handler.response_ids, dtype=torch.int, device=cur_device)) + response_attention_mask.append(torch.tensor(rollout_handler.response_attention_mask, dtype=torch.int, device=cur_device)) + response_position_ids.append(torch.tensor(rollout_handler.response_position_ids, dtype=torch.int, device=cur_device)) + response_loss_mask.append(torch.tensor(rollout_handler.response_loss_mask, dtype=torch.int, device=cur_device)) + scores.append(rollout_handler.score) + messages.append(rollout_handler.messages) + + # pad to length + response_ids = pad_sequence(response_ids, batch_first=True, padding_value=self.pad_token_id) + if response_ids.shape[1] < self.config.response_length: + response_ids = pad_sequence_to_length(response_ids, self.config.response_length, self.pad_token_id) + response_attention_mask = pad_sequence(response_attention_mask, batch_first=True, padding_value=0) + if response_attention_mask.shape[1] < self.config.response_length: + response_attention_mask = pad_sequence_to_length(response_attention_mask, self.config.response_length, 0) + response_loss_mask = pad_sequence(response_loss_mask, batch_first=True, padding_value=0) + if response_loss_mask.shape[1] < self.config.response_length: + response_loss_mask = pad_sequence_to_length(response_loss_mask, self.config.response_length, 0) + response_length = response_ids.size(1) + delta_position_ids = torch.arange(1, response_length + 1, device=cur_device) + delta_position_ids = delta_position_ids.unsqueeze(0).repeat(batch_size, 1) + input_ids = prompts.batch['input_ids'] # (bs, prompt_length) + prompt_length = input_ids.size(-1) + # left-padded attention_mask + attention_mask = prompts.batch['attention_mask'] + position_ids = prompts.batch['position_ids'] + input_ids = input_ids.repeat_interleave(self.config.n, dim=0) + attention_mask = attention_mask.repeat_interleave(self.config.n, dim=0) + position_ids = position_ids.repeat_interleave(self.config.n, dim=0) + response_position_ids = position_ids[:, -1:] + delta_position_ids + + seq = torch.cat((input_ids, response_ids), dim=-1) + attention_mask = torch.cat((attention_mask, response_attention_mask), dim=-1) + position_ids = torch.cat((position_ids, response_position_ids), dim=-1) + response_mask = response_loss_mask + + reward_tensor = torch.zeros_like(response_ids, dtype=torch.float32) # (bs, response_length) + valid_response_length = attention_mask[:, prompt_length:].sum(dim=-1) + for i in range(len(scores)): + reward_tensor[i, valid_response_length[i].item() - 1] = scores[i] + + if global_steps: + try: + os.makedirs(os.path.join(self.config.rollout_log_dir, f"step{global_steps}"), exist_ok=True) + with open(os.path.join(self.config.rollout_log_dir, f"step{global_steps}/{torch.distributed.get_rank()}.json"), "w") as f: + json_msg = [] + for idx, msgs in enumerate(messages): + records = { + "item_id": rollout_handler_ls[idx].item_id, + "conversations": [msg.to_dict() for msg in msgs], + "reward": scores[idx] + } + json_msg.append(records) + json.dump(json_msg, f, ensure_ascii=True, indent=4) + except Exception as e: + print(e) + + # close clients + for client in env_clients: + try: + client.close() + except Exception as e: + print(f"Error during closing env: {e}") + + batch = TensorDict( + { + 'prompts': input_ids, + 'responses': response_ids, + 'input_ids': seq, + 'attention_mask': attention_mask, + 'position_ids': position_ids, + 'response_mask': response_mask, + 'scores': reward_tensor, + 'task_rounds': torch.tensor(task_rounds, dtype=torch.float32).to(input_ids.device), + 'task_scores': reward_tensor + }, + batch_size=batch_size) + + # free vllm cache engine + if self.config.free_cache_engine: + self.inference_engine.free_cache_engine() + + return DataProto(batch=batch) diff --git a/frameworks/agentgym_rl/verl/workers/rollout/base.py b/frameworks/agentgym_rl/verl/workers/rollout/base.py new file mode 100644 index 0000000000000000000000000000000000000000..8c2733325bbf7ba4e8c3438a53c4e2b97d60ee83 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/base.py @@ -0,0 +1,37 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from abc import ABC, abstractmethod +from typing import Iterable, Union + +from verl import DataProto + +__all__ = ['BaseRollout'] + + +class BaseRollout(ABC): + + def __init__(self): + """ + + Args: + dataloader: an Iterable of TensorDict that consistently generates prompts. Note that the dataloader + should handle when the training stops. + """ + super().__init__() + + @abstractmethod + def generate_sequences(self, prompts: DataProto) -> DataProto: + """Generate sequences""" + pass diff --git a/frameworks/agentgym_rl/verl/workers/rollout/hf_rollout.py b/frameworks/agentgym_rl/verl/workers/rollout/hf_rollout.py new file mode 100644 index 0000000000000000000000000000000000000000..1d929e5dd439a5c1a3b92b73bd6cb134cbb29f09 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/hf_rollout.py @@ -0,0 +1,140 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Rollout with huggingface models. +TODO: refactor this class. Currently, it will hang when using FSDP HybridShard. We should actually create a single GPU model. +Then, get full state_dict and bind the state_dict to the single GPU model. Then, use the single GPU model to perform generation. +""" +import contextlib +import torch +import torch.distributed +from tensordict import TensorDict +from torch import nn +from torch.distributed.fsdp import FullyShardedDataParallel as FSDP + +from verl import DataProto +from verl.utils.torch_functional import get_eos_mask +from .base import BaseRollout + +from transformers import GenerationConfig + +__all__ = ['HFRollout'] + + +class HFRollout(BaseRollout): + + def __init__(self, module: nn.Module, config): + super().__init__() + self.config = config + self.module = module + + def generate_sequences(self, prompts: DataProto) -> DataProto: + batch_size = prompts.batch.batch_size[0] + num_chunks = max(batch_size // self.config.get('micro_batch_size', batch_size), 1) + batch_prompts = prompts.chunk(chunks=num_chunks) + output = [self._generate_minibatch(p) for p in batch_prompts] + output = DataProto.concat(output) + return output + + @torch.no_grad() + def _generate_minibatch(self, prompts: DataProto) -> DataProto: + idx = prompts.batch['input_ids'] # (bs, prompt_length) + attention_mask = prompts.batch['attention_mask'] # left-padded attention_mask + position_ids = prompts.batch['position_ids'] + + # used to construct attention_mask + eos_token_id = prompts.meta_info['eos_token_id'] + pad_token_id = prompts.meta_info['pad_token_id'] + + batch_size = idx.size(0) + prompt_length = idx.size(1) + + self.module.eval() + param_ctx = contextlib.nullcontext() + + # make sampling args can be overriden by inputs + do_sample = prompts.meta_info.get('do_sample', self.config.do_sample) + response_length = prompts.meta_info.get('response_length', self.config.response_length) + top_p = prompts.meta_info.get('top_p', self.config.get('top_p', 1.0)) + top_k = prompts.meta_info.get('top_k', self.config.get('top_k', 0)) + + if top_k is None: + top_k = 0 + top_k = max(0, top_k) # to be compatible with vllm + + temperature = prompts.meta_info.get('temperature', self.config.temperature) + + generation_config = GenerationConfig(temperature=temperature, top_p=top_p, top_k=top_k) + + if isinstance(self.module, FSDP): + # recurse need to set to False according to https://github.com/pytorch/pytorch/issues/100069 + param_ctx = FSDP.summon_full_params(self.module, writeback=False, recurse=False) + with param_ctx: + with torch.autocast(device_type='cuda', dtype=torch.bfloat16): + output = self.module.generate( + input_ids=idx, + attention_mask=attention_mask, + do_sample=do_sample, + max_new_tokens=response_length, + # max_length=max_length, + eos_token_id=eos_token_id, + pad_token_id=pad_token_id, + generation_config=generation_config, + # renormalize_logits=True, + output_scores=False, # this is potentially very large + return_dict_in_generate=True, + use_cache=True) + # TODO: filter out the seq with no answers like ds-chat + seq = output.sequences + + # huggingface generate will stop generating when all the batch reaches [EOS]. + # We have to pad to response_length + sequence_length = prompt_length + self.config.response_length + delta_length = sequence_length - seq.shape[1] + + if delta_length > 0: + delta_tokens = torch.ones(size=(batch_size, delta_length), device=seq.device, dtype=seq.dtype) + delta_tokens = pad_token_id * delta_tokens + seq = torch.cat((seq, delta_tokens), dim=1) + + assert seq.shape[1] == sequence_length + + prompt = seq[:, :prompt_length] # (bs, prompt_length) + response = seq[:, prompt_length:] # (bs, response_length) + + response_length = response.size(1) + delta_position_id = torch.arange(1, response_length + 1, device=position_ids.device) + delta_position_id = delta_position_id.unsqueeze(0).repeat(batch_size, 1) + + response_position_ids = position_ids[:, -1:] + delta_position_id + position_ids = torch.cat([position_ids, response_position_ids], dim=-1) + + response_attention_mask = get_eos_mask(response_id=response, eos_token=eos_token_id, dtype=attention_mask.dtype) + attention_mask = torch.cat((attention_mask, response_attention_mask), dim=-1) + + batch = TensorDict( + { + 'prompts': prompt, + 'responses': response, + 'input_ids': seq, + 'attention_mask': attention_mask, + 'position_ids': position_ids + }, + batch_size=batch_size) + + # empty cache before compute old_log_prob + torch.cuda.empty_cache() + + self.module.train() + return DataProto(batch=batch) diff --git a/frameworks/agentgym_rl/verl/workers/rollout/naive/__init__.py b/frameworks/agentgym_rl/verl/workers/rollout/naive/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..df81c8603fc41731b2ec2cf007a06f5976e43c06 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/naive/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .naive_rollout import NaiveRollout diff --git a/frameworks/agentgym_rl/verl/workers/rollout/naive/naive_rollout.py b/frameworks/agentgym_rl/verl/workers/rollout/naive/naive_rollout.py new file mode 100644 index 0000000000000000000000000000000000000000..5fbb93da38a4d136ba672cce42076df6b0d5aec8 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/naive/naive_rollout.py @@ -0,0 +1,122 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +In single GPU rollout, the sequences are generated directly by sampling from the model. +The output will contain +1. output_ids +2. attention_masks (left padding) +3. eos_masks +4. log_probs +""" +from typing import Iterable, Union + +import torch +import torch.nn.functional as F +from tensordict import TensorDict +from torch import nn + +from verl import DataProto +from verl.utils.torch_functional import logprobs_from_logits +from ..base import BaseRollout + +__all__ = ['NativeRollout'] + + +class NaiveRollout(BaseRollout): + + def __init__(self, module: nn.Module, config): + """A naive rollout. It requires the module to be compatible with huggingface APIs. That is: + The module should define __call__ to receive input_ids, attention_mask and position_ids. + It outputs a structure that contains logits field. + + Args: + module: module here follows huggingface APIs + config: DictConfig + """ + super().__init__() + self.config = config + self.module = module + + @torch.no_grad() + def generate_sequences(self, prompts: DataProto) -> DataProto: + """Generate sequences""" + idx = prompts.batch['input_ids'] # (bs, prompt_length) + attention_mask = prompts.batch['attention_mask'] # left-padded attention_mask + position_ids = prompts.batch['position_ids'] + + # used to construct attention_mask + eos_token_id = prompts.meta_info['eos_token_id'] + if isinstance(eos_token, int): + eos_token = [eos_token] + + batch_size = idx.size(0) + prompt_length = idx.size(1) + + self.module.eval() + + prev_attention_mask = torch.ones(size=(batch_size, 1), dtype=attention_mask.dtype, device=attention_mask.device) + + logits_lst = [] + for _ in range(self.config.response_length): + # if the sequence context is growing too long we must crop it at block_size + # idx_cond = idx if idx.size(1) <= self.config.block_size else idx[:, -self.config.block_size:] + idx_cond = idx + # forward the model to get the logits for the index in the sequence + # we use huggingface APIs here + output = self.module(input_ids=idx_cond, attention_mask=attention_mask, position_ids=position_ids) + logits = output.logits + # pluck the logits at the final step and scale by desired temperature + logits = logits[:, -1, :] / self.config.temperature # (bs, vocab_size) + # optionally crop the logits to only the top k options + if self.config.top_k is not None: + v, _ = torch.topk(logits, min(self.config.top_k, logits.size(-1))) + logits[logits < v[:, [-1]]] = -float('Inf') + # apply softmax to convert logits to (normalized) probabilities + probs = F.softmax(logits, dim=-1) + # sample from the distribution + if self.config.do_sample: + idx_next = torch.multinomial(probs, num_samples=1) + else: + idx_next = torch.argmax(probs, dim=-1, keepdim=True) + + attention_mask = torch.cat((attention_mask, prev_attention_mask), dim=-1) + + for token_id in eos_token_id: + prev_attention_mask = torch.logical_and(idx_next != token_id, prev_attention_mask.bool()) + prev_attention_mask.to(attention_mask.dtype) + + position_ids = torch.cat((position_ids, position_ids[:, -1:] + 1), dim=-1) + + # append sampled index to the running sequence and continue + idx = torch.cat((idx, idx_next), dim=1) + logits_lst.append(logits) + + logits = torch.stack(logits_lst, dim=1) # (bs, response_length, vocab_size) + prompts = idx[:, :prompt_length] # (bs, prompt_length) + response = idx[:, prompt_length:] # (bs, response_length) + log_probs = logprobs_from_logits(logits=logits, labels=response) + batch = TensorDict( + { + 'input_ids': prompts, + 'responses': response, + 'sequences': idx, + 'old_log_probs': log_probs, + 'attention_mask': attention_mask, + 'position_ids': position_ids, + }, + batch_size=batch_size) + + self.module.train() + + return DataProto(batch=batch) diff --git a/frameworks/agentgym_rl/verl/workers/rollout/schemas.py b/frameworks/agentgym_rl/verl/workers/rollout/schemas.py new file mode 100644 index 0000000000000000000000000000000000000000..059a08cf522251e9b8b7bdc08f825de3425ddc66 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/schemas.py @@ -0,0 +1,171 @@ +from dataclasses import dataclass +from typing import List, Literal +from transformers import PreTrainedTokenizer +import torch + + +def _pre_process_inputs(pad_token_id, prompt_token_ids: torch.Tensor) -> List[int]: + # remove the left padding in the prompt token_id + # pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id + non_pad_index = torch.nonzero(prompt_token_ids != pad_token_id, as_tuple=False)[0][0] + token_ids = prompt_token_ids[non_pad_index:].tolist() + return token_ids + +class Message: + def __init__(self, role: str, content: str): + self.role = role + self.content = content + def to_dict(self): + return {'role': self.role, 'content': self.content} + def __repr__(self): + return str(self.to_dict()) + def __str__(self): + return self.__repr_ + +class RolloutHandler: + def __init__( + self, + messages: List[Message], + task_name: str, + item_id: int, + score: float, + done: bool, + input_ids: List[int], + prompt_ids: List[int], + response_ids: List[int], + attention_mask: List[int], + prompt_attention_mask: List[int], + response_attention_mask: List[int], + position_ids: List[int], + prompt_position_ids: List[int], + response_position_ids: List[int], + loss_mask: List[int], + prompt_loss_mask: List[int], + response_loss_mask: List[int], + max_response_len: int = 8192, + max_model_len: int = 32768 + ): + self.messages = messages + self.task_name = task_name + self.item_id = item_id + self.score = score + self.done = done + self.input_ids = input_ids + self.prompt_ids = prompt_ids + self.response_ids = response_ids + self.attention_mask = attention_mask + self.prompt_attention_mask = prompt_attention_mask + self.response_attention_mask = response_attention_mask + self.position_ids = position_ids + self.prompt_position_ids = prompt_position_ids + self.response_position_ids = response_position_ids + self.loss_mask = loss_mask + self.prompt_loss_mask = prompt_loss_mask + self.response_loss_mask = response_loss_mask + self.max_response_len = max_response_len + self.max_model_len = max_model_len + self.format_config: dict = { + "qwen": { + "assistat_prefix_msg": "\n<|im_start|>assistant\n", + "assistat_suffix_msg": "<|im_end|>", + "user_prefix_msg": "\n<|im_start|>user\n", + "user_suffix_msg": "<|im_end|>", + } + } + + def get_generation_prompt(self, tokenizer: PreTrainedTokenizer) -> List[int]: + conversations = [ + msg.to_dict() for msg in self.messages + ] + return tokenizer.apply_chat_template(conversations, add_generation_prompt=True, tokenize=True) + + + def add_assistant_message( + self, + tokenizer: PreTrainedTokenizer, + content: str, + format: Literal["qwen"] = "qwen", + ) -> None: + msg = Message(role='assistant', content=content) + self.messages.append(msg) + assert format in self.format_config.keys(), f"format {format} not supported" + prefix_msg = self.format_config[format]["assistat_prefix_msg"] + prefix_token_ids = tokenizer.encode(prefix_msg, add_special_tokens=False) + suffix_msg = self.format_config[format]["assistat_suffix_msg"] + suffix_token_ids = tokenizer.encode(suffix_msg, add_special_tokens=False) + response = tokenizer.encode(content, add_special_tokens=False) + if self.input_ids[-len(prefix_token_ids) :] == prefix_token_ids: + append_token_ids = response + _loss_mask = [1] * len(response) + elif self.input_ids[-len(suffix_token_ids) :] == suffix_token_ids: + append_token_ids = prefix_token_ids + response + _loss_mask = [0] * len(prefix_token_ids) + [1] * len(response) + else: + max_len = max(len(prefix_token_ids), len(suffix_token_ids)) + raise ValueError( + f"""Unsupported end of message format: + {tokenizer.decode(self.input_ids[-max_len:])}, {tokenizer.decode(self.input_ids)=}""" + ) + append_token_ids += suffix_token_ids + _loss_mask += [1] * len(suffix_token_ids) + self.input_ids += append_token_ids + _attention_mask = [1] * len(append_token_ids) + self.attention_mask += _attention_mask + _delta_position_ids = [pos_id for pos_id in range(1, len(append_token_ids) + 1)] + last_position_ids = self.position_ids[-1] + _position_ids = [pos_id + last_position_ids for pos_id in _delta_position_ids] + self.loss_mask += _loss_mask + self.position_ids += _position_ids + assert len(self.input_ids) == len(self.attention_mask) == len(self.position_ids) == len(self.loss_mask), f"""Rollout Handler has different length of {len(self.input_ids)=}, + {len(self.attention_mask)=}, {len(self.position_ids)=}, {len(self.loss_mask)=}""" + + def add_user_message( + self, + tokenizer: PreTrainedTokenizer, + content: str, + format: Literal["qwen"] = "qwen", + ) -> None: + msg = Message(role='user', content=content) + self.messages.append(msg) + assert format in self.format_config.keys(), f"format {format} not supported" + prefix_msg = self.format_config[format]["user_prefix_msg"] + prefix_token_ids = tokenizer.encode(prefix_msg, add_special_tokens=False) + suffix_msg = self.format_config[format]["user_suffix_msg"] + suffix_token_ids = tokenizer.encode(suffix_msg, add_special_tokens=False) + content_token_ids = tokenizer.encode(content, add_special_tokens=False) + + if self.input_ids[-len(prefix_token_ids) :] == prefix_token_ids: + append_token_ids = content_token_ids + _loss_mask = [0] * len(content_token_ids) + elif self.input_ids[-len(suffix_token_ids) :] == suffix_token_ids: + append_token_ids = prefix_token_ids + content_token_ids + _loss_mask = [0] * len(prefix_token_ids) + [0] * len(content_token_ids) + else: + max_len = max(len(prefix_token_ids), len(suffix_token_ids)) + raise ValueError( + f"""Unsupported end of message format: + {tokenizer.decode(self.input_ids[-max_len:])}, {tokenizer.decode(self.input_ids)=}""" + ) + + append_token_ids += suffix_token_ids + _loss_mask += [0] * len(suffix_token_ids) + self.input_ids += append_token_ids + _attention_mask = [1] * len(append_token_ids) + self.attention_mask += _attention_mask + _delta_position_ids = [pos_id for pos_id in range(1, len(append_token_ids) + 1)] + last_position_ids = self.position_ids[-1] + _position_ids = [pos_id + last_position_ids for pos_id in _delta_position_ids] + self.loss_mask += _loss_mask + self.position_ids += _position_ids + assert len(self.input_ids) == len(self.attention_mask) == len(self.position_ids) == len(self.loss_mask), f"""Rollout Handler has different length of {len(self.input_ids)=}, + {len(self.attention_mask)=}, {len(self.position_ids)=}, {len(self.loss_mask)=}""" + + def truncate_output_ids(self) -> None: + self.input_ids = self.input_ids[: self.max_model_len] + self.attention_mask = self.attention_mask[: self.max_model_len] + self.position_ids = self.position_ids[: self.max_model_len] + self.loss_mask = self.loss_mask[: self.max_model_len] + self.response_ids = self.input_ids[len(self.prompt_ids) :][: self.max_response_len] + self.response_attention_mask = self.attention_mask[len(self.prompt_attention_mask) :][: self.max_response_len] + self.response_position_ids = self.position_ids[len(self.prompt_position_ids) :][: self.max_response_len] + self.response_loss_mask = self.loss_mask[len(self.prompt_loss_mask) :][: self.max_response_len] diff --git a/frameworks/agentgym_rl/verl/workers/rollout/tokenizer.py b/frameworks/agentgym_rl/verl/workers/rollout/tokenizer.py new file mode 100644 index 0000000000000000000000000000000000000000..c0dfa3a530329605d7af48a2186d304198774e09 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/rollout/tokenizer.py @@ -0,0 +1,162 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +The base tokenizer class, required for any hybrid engine based rollout or inference with vLLM. +""" +from abc import ABC, abstractmethod +from typing import Dict, List, Union + +__all__ = ['HybridEngineBaseTokenizer'] + + +class HybridEngineBaseTokenizer(ABC): + """the tokenizer property and function name should align with HF's to meet vllm requirement""" + + @property + @abstractmethod + def vocab_size(self): + """ + `int`: Size of the base vocabulary (without the added tokens). + """ + pass + + @property + @abstractmethod + def pad_token_id(self): + """ + `Optional[int]`: Id of the padding token in the vocabulary. Returns `None` if the token has not been set. + """ + pass + + @property + @abstractmethod + def eos_token_id(self): + """ + `Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been + set. + """ + pass + + @property + @abstractmethod + def all_special_ids(self) -> List[int]: + """ + `List[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes. + """ + pass + + @property + @abstractmethod + def all_special_tokens(self) -> List[str]: + """ + `List[str]`: A list of the unique special tokens (`''`, `''`, ..., etc.). + + Convert tokens of `tokenizers.AddedToken` type to string. + """ + pass + + @abstractmethod + def encode(self, text): + """ + Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. + + Args: + text (`str`, `List[str]` or `List[int]`): + The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the + `tokenize` method) or a list of integers. + + text_pair (`str`, `List[str]` or `List[int]`, *optional*): + Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using + the `tokenize` method) or a list of integers. + """ + pass + + @abstractmethod + def decode( + self, + token_ids: Union[int, List[int], "np.ndarray", "torch.Tensor", "tf.Tensor"], + skip_special_tokens: bool = False, + clean_up_tokenization_spaces: bool = None, + **kwargs, + ) -> str: + """ + Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special + tokens and clean up tokenization spaces. + + Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. + + Args: + token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): + List of tokenized input ids. Can be obtained using the `__call__` method. + skip_special_tokens (`bool`, *optional*, defaults to `False`): + Whether or not to remove special tokens in the decoding. + clean_up_tokenization_spaces (`bool`, *optional*): + Whether or not to clean up the tokenization spaces. If `None`, will default to + `self.clean_up_tokenization_spaces`. + kwargs (additional keyword arguments, *optional*): + Will be passed to the underlying model specific decode method. + + Returns: + `str`: The decoded sentence. + """ + pass + + @abstractmethod + def convert_ids_to_tokens(self, + ids: Union[int, List[int]], + skip_special_tokens: bool = False) -> Union[str, List[str]]: + """ + Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and + added tokens. + + Args: + ids (`int` or `List[int]`): + The token id (or token ids) to convert to tokens. + skip_special_tokens (`bool`, *optional*, defaults to `False`): + Whether or not to remove special tokens in the decoding. + + Returns: + `str` or `List[str]`: The decoded token(s). + """ + pass + + @abstractmethod + def get_added_vocab(self) -> Dict[str, int]: + """ + Returns the added tokens in the vocabulary as a dictionary of token to index. Results might be different from + the fast call because for now we always add the tokens even if they are already in the vocabulary. This is + something we should change. + + Returns: + `Dict[str, int]`: The added tokens. + """ + pass + + @abstractmethod + def convert_tokens_to_string(self, tokens: List[str]) -> str: + """ + Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we + often want to remove sub-word tokenization artifacts at the same time. + + Args: + tokens (`List[str]`): The token to join in a string. + + Returns: + `str`: The joined tokens. + """ + pass + + @property + def is_fast(self): + return False diff --git a/frameworks/agentgym_rl/verl/workers/sharding_manager/__init__.py b/frameworks/agentgym_rl/verl/workers/sharding_manager/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e72fdf011c2455d920d0857eb3e6eadbaeebc332 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/sharding_manager/__init__.py @@ -0,0 +1,33 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from verl.utils.import_utils import is_vllm_available, is_megatron_core_available + +from .base import BaseShardingManager +from .fsdp_ulysses import FSDPUlyssesShardingManager + +AllGatherPPModel = None + +if is_megatron_core_available() and is_vllm_available(): + from .megatron_vllm import AllGatherPPModel, MegatronVLLMShardingManager +elif AllGatherPPModel is not None: + pass +else: + AllGatherPPModel = None + MegatronVLLMShardingManager = None + +if is_vllm_available(): + from .fsdp_vllm import FSDPVLLMShardingManager +else: + FSDPVLLMShardingManager = None diff --git a/frameworks/agentgym_rl/verl/workers/sharding_manager/base.py b/frameworks/agentgym_rl/verl/workers/sharding_manager/base.py new file mode 100644 index 0000000000000000000000000000000000000000..d8717890f2e2cf4d2c5e7683398e32fa8ebf3765 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/sharding_manager/base.py @@ -0,0 +1,33 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Sharding manager to implement HybridEngine +""" + +from verl import DataProto + + +class BaseShardingManager: + + def __enter__(self): + pass + + def __exit__(self, exc_type, exc_value, traceback): + pass + + def preprocess_data(self, data: DataProto) -> DataProto: + return data + + def postprocess_data(self, data: DataProto) -> DataProto: + return data diff --git a/frameworks/agentgym_rl/verl/workers/sharding_manager/fsdp_ulysses.py b/frameworks/agentgym_rl/verl/workers/sharding_manager/fsdp_ulysses.py new file mode 100644 index 0000000000000000000000000000000000000000..85328cc8d6b47e92ad9d3eff24aaf1390fc2bc7a --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/sharding_manager/fsdp_ulysses.py @@ -0,0 +1,86 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Contains a resharding manager that binds weights from FSDP zero3 to XPerfGPT +""" +from .base import BaseShardingManager + +from torch.distributed.device_mesh import DeviceMesh + +from verl.utils.torch_functional import allgather_dict_tensors +from verl.utils.ulysses import set_ulysses_sequence_parallel_group, get_ulysses_sequence_parallel_group +import numpy as np + +import torch +import torch.distributed + +from verl import DataProto + + +class FSDPUlyssesShardingManager(BaseShardingManager): + """ + Sharding manager to support data resharding when using FSDP + Ulysses + """ + + def __init__(self, device_mesh: DeviceMesh): + super().__init__() + self.device_mesh = device_mesh + self.seed_offset = 12345 + + def __enter__(self): + if self.device_mesh is not None: + # We have a global SP group + # so we have to change to use model-specific sp group + self.prev_sp_group = get_ulysses_sequence_parallel_group() + set_ulysses_sequence_parallel_group(self.device_mesh['sp'].get_group()) + # TODO: check how to set seed for each model + + def __exit__(self, exc_type, exc_value, traceback): + # restore random states + if self.device_mesh is not None: + # revert to previous sp group + set_ulysses_sequence_parallel_group(self.prev_sp_group) + # TODO: check how to set seed for each model + + def preprocess_data(self, data: DataProto) -> DataProto: + """ + AllGather data from sp region + This is because the data is first sharded along the FSDP dimension as we utilize the DP_COMPUTE + In Ulysses, we need to make sure the same data is used across a SP group + """ + if self.device_mesh is not None: + sp_size = self.device_mesh['sp'].size() + group = self.device_mesh['sp'].get_group() + + prev_device = data.batch.device + data.batch = data.batch.cuda(device=torch.cuda.current_device()) + data.batch = allgather_dict_tensors(data.batch.contiguous(), size=sp_size, group=group, dim=0) + data.batch = data.batch.to(prev_device) + # all gather non_tensor_batch + all_non_tensor_batch = [None for _ in range(sp_size)] + torch.distributed.all_gather_object(all_non_tensor_batch, data.non_tensor_batch, group=group) + data.non_tensor_batch = { + k: np.concatenate([d[k] for d in all_non_tensor_batch]) for k in data.non_tensor_batch + } + return data + + def postprocess_data(self, data: DataProto) -> DataProto: + """ + Split the data to follow FSDP partition + """ + if self.device_mesh is not None: + sp_size = self.device_mesh['sp'].size() + sp_rank = self.device_mesh['sp'].get_local_rank() + data = data.chunk(chunks=sp_size)[sp_rank] + return data \ No newline at end of file diff --git a/frameworks/agentgym_rl/verl/workers/sharding_manager/fsdp_vllm.py b/frameworks/agentgym_rl/verl/workers/sharding_manager/fsdp_vllm.py new file mode 100644 index 0000000000000000000000000000000000000000..b90a6c2605570cd7758be2d442384d31b195a71c --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/sharding_manager/fsdp_vllm.py @@ -0,0 +1,163 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import logging +import numpy as np +import torch +from torch.distributed.fsdp.fully_sharded_data_parallel import FullyShardedDataParallel as FSDP +from torch.distributed.fsdp.api import ShardingStrategy, ShardedStateDictConfig, StateDictType, FullStateDictConfig +from torch.distributed.device_mesh import DeviceMesh + +from verl.third_party.vllm import LLM +from verl.third_party.vllm import parallel_state as vllm_ps +from verl import DataProto +from verl.utils.torch_functional import (broadcast_dict_tensor, allgather_dict_tensors) +from verl.utils.debug import log_gpu_memory_usage +from verl.third_party.vllm import vllm_version + +from .base import BaseShardingManager + +logger = logging.getLogger(__file__) +logger.setLevel(os.getenv('VERL_PPO_LOGGING_LEVEL', 'WARN')) + + +class FSDPVLLMShardingManager(BaseShardingManager): + + def __init__(self, + module: FSDP, + inference_engine: LLM, + model_config, + full_params: bool = False, + device_mesh: DeviceMesh = None): + self.module = module + self.inference_engine = inference_engine + self.model_config = model_config + self.device_mesh = device_mesh + + # Full params + self.full_params = full_params + if full_params: + FSDP.set_state_dict_type(self.module, + state_dict_type=StateDictType.FULL_STATE_DICT, + state_dict_config=FullStateDictConfig()) + else: + FSDP.set_state_dict_type(self.module, + state_dict_type=StateDictType.SHARDED_STATE_DICT, + state_dict_config=ShardedStateDictConfig()) + + # Note that torch_random_states may be different on each dp rank + self.torch_random_states = torch.cuda.get_rng_state() + # get a random rng states + if self.device_mesh is not None: + gen_dp_rank = self.device_mesh['dp'].get_local_rank() + torch.cuda.manual_seed(gen_dp_rank + 1000) # make sure all tp ranks have the same random states + self.gen_random_states = torch.cuda.get_rng_state() + torch.cuda.set_rng_state(self.torch_random_states) + else: + self.gen_random_states = None + + def __enter__(self): + log_gpu_memory_usage('Before state_dict() in sharding manager memory', logger=logger) + params = self.module.state_dict() + log_gpu_memory_usage('After state_dict() in sharding manager memory', logger=logger) + # Copy, not share memory + load_format = 'hf' if self.full_params else 'dtensor' + if vllm_version in ('0.4.2', '0.5.4', '0.6.3'): + self.inference_engine.sync_model_weights(params, load_format=load_format) + else: + self.inference_engine.wake_up() + # TODO(ZSL): deal with 'hf' format + if load_format == 'dtensor': + from verl.third_party.vllm import load_dtensor_weights + load_dtensor_weights( + params, self.inference_engine.llm_engine.model_executor.driver_worker.worker.model_runner.model) + else: + raise NotImplementedError(f'load_format {load_format} not implemented') + log_gpu_memory_usage('After sync model weights in sharding manager', logger=logger) + + del params + torch.cuda.empty_cache() + log_gpu_memory_usage('After del state_dict and empty_cache in sharding manager', logger=logger) + + # TODO: offload FSDP model weights + # self.module.cpu() + # torch.cuda.empty_cache() + # if torch.distributed.get_rank() == 0: + # print(f'after model to cpu in sharding manager memory allocated: {torch.cuda.memory_allocated() / 1e9}GB, reserved: {torch.cuda.memory_reserved() / 1e9}GB') + + # important: need to manually set the random states of each tp to be identical. + if self.device_mesh is not None: + self.torch_random_states = torch.cuda.get_rng_state() + torch.cuda.set_rng_state(self.gen_random_states) + + def __exit__(self, exc_type, exc_value, traceback): + log_gpu_memory_usage('Before vllm offload in sharding manager', logger=logger) + # TODO(ZSL): check this + if vllm_version in ('0.4.2', '0.5.4', '0.6.3'): + self.inference_engine.offload_model_weights() + else: + self.inference_engine.sleep(level=1) + log_gpu_memory_usage('After vllm offload in sharding manager', logger=logger) + + # self.module.to('cuda') + # if torch.distributed.get_rank() == 0: + # print(f'after actor module to cuda in sharding manager memory allocated: {torch.cuda.memory_allocated() / 1e9}GB, reserved: {torch.cuda.memory_reserved() / 1e9}GB') + + self.module.train() + + # add empty cache after each compute + torch.cuda.empty_cache() + + # restore random states + if self.device_mesh is not None: + self.gen_random_states = torch.cuda.get_rng_state() + torch.cuda.set_rng_state(self.torch_random_states) + + def preprocess_data(self, data: DataProto) -> DataProto: + # TODO: Current impl doesn't consider FSDP with torch micro-dp + tp_size = vllm_ps.get_tensor_model_parallel_world_size() + if vllm_version in ('0.3.1', '0.4.2', '0.5.4', '0.6.3'): + group = vllm_ps.get_tensor_model_parallel_group() + else: + group = vllm_ps.get_tensor_model_parallel_group().device_group + + prev_device = data.batch.device + data.batch = data.batch.cuda(device=torch.cuda.current_device()) + data.batch = allgather_dict_tensors(data.batch.contiguous(), size=tp_size, group=group, dim=0) + data.batch = data.batch.to(prev_device) + # all gather non_tensor_batch + all_non_tensor_batch = [None for _ in range(tp_size)] + torch.distributed.all_gather_object(all_non_tensor_batch, data.non_tensor_batch, group=group) + data.non_tensor_batch = {k: np.concatenate([d[k] for d in all_non_tensor_batch]) for k in data.non_tensor_batch} + return data + + def postprocess_data(self, data: DataProto) -> DataProto: + # TODO: Current impl doesn't consider FSDP with torch micro-dp + local_world_size = vllm_ps.get_tensor_model_parallel_world_size() + src_rank = (torch.distributed.get_rank() // local_world_size) * local_world_size + if vllm_version in ('0.3.1', '0.4.2', '0.5.4', '0.6.3'): + broadcast_dict_tensor(data.batch, src=src_rank, group=vllm_ps.get_tensor_model_parallel_group()) + else: + broadcast_dict_tensor(data.batch, + src=src_rank, + group=vllm_ps.get_tensor_model_parallel_group().device_group) + dp_rank = torch.distributed.get_rank() + dp_size = torch.distributed.get_world_size() # not consider torch micro-dp + tp_size = vllm_ps.get_tensor_model_parallel_world_size() + if tp_size > 1: + # TODO: shall we build a micro_dp group for vllm when integrating with vLLM? + local_prompts = data.chunk(chunks=tp_size) + data = local_prompts[dp_rank % tp_size] + return data diff --git a/frameworks/agentgym_rl/verl/workers/sharding_manager/megatron_vllm.py b/frameworks/agentgym_rl/verl/workers/sharding_manager/megatron_vllm.py new file mode 100644 index 0000000000000000000000000000000000000000..bc07a5a656445f4ea442440b8634422e1b836ce0 --- /dev/null +++ b/frameworks/agentgym_rl/verl/workers/sharding_manager/megatron_vllm.py @@ -0,0 +1,428 @@ +# Copyright 2024 Bytedance Ltd. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +This file contains a Megatron style Hybrid Engine that shares the weights of the actor with the inference engine. +""" + +import torch +import torch.distributed as dist + +from torch import nn + +from megatron.core import parallel_state as mpu +from megatron.core import DistributedDataParallel as LocalDDP +from megatron.core.transformer.module import Float16Module +from torch.nn.parallel.distributed import DistributedDataParallel as torchDDP +from verl.utils.megatron_utils import get_model, unwrap_model +from verl.utils.memory_buffer import ( + build_memory_buffer, + build_memory_reference_from_module, + get_weight_buffer_meta_from_module, +) + + +class AllGatherPPModel: + + def __init__(self, model_provider) -> None: + + self._pp_group = mpu.get_pipeline_model_parallel_group() + self._pp_rank = mpu.get_pipeline_model_parallel_rank() + self._pp_size = mpu.get_pipeline_model_parallel_world_size() + self._vpp_size = mpu.get_virtual_pipeline_model_parallel_world_size() + self._model_chunk_size = self._vpp_size or 1 + + # each one holds a list of model_chunks in this pp stage + self._pp_models = [None] * self.pp_size + + rank_list = list(range(self.pp_size)) + # make current rank the last one to initialize + rank_list[self.pp_rank], rank_list[-1] = rank_list[-1], rank_list[self.pp_rank] + self._this_rank_models = None + + # store the parameter of each pp stage + self.memory_buffers = [None] * self.pp_size + for cur_pp_rank in rank_list: + print( + f'create pp model', f'torch allocated {torch.cuda.memory_allocated() / 1e9:.4f} GB, ' + f'reserved {torch.cuda.memory_reserved() / 1e9:.4f} GB') + # since the last initialized rank is the current pp rank, after init, the pp rank is still correct + mpu.set_pipeline_model_parallel_rank(cur_pp_rank) + if cur_pp_rank != self.pp_rank: + models = get_model(model_provider, wrap_with_ddp=False) + models = nn.ModuleList(models) + assert len(models) == self._model_chunk_size, f"{len(models)} != {self._model_chunk_size}" + self.pp_models[cur_pp_rank] = models + else: + # for regular model, we wrapped it with DDP + models = get_model(model_provider) + assert len(models) == self._model_chunk_size, f"{len(models)} != {self._model_chunk_size}" + self._this_rank_models = nn.ModuleList(models) + self.pp_models[cur_pp_rank] = nn.ModuleList(unwrap_model(models, (torchDDP, LocalDDP))) + + self._build_param_buffer(cur_pp_rank) + self._build_param_references(cur_pp_rank, maintain_weight=cur_pp_rank == self.pp_rank) + + # TODO: after binding to the memory buffer, we can load the checkpoint here + if cur_pp_rank != self.pp_rank: + for model in self.pp_models[cur_pp_rank]: + model.eval() + self._offload_params_to_cpu(cur_pp_rank) + + def _build_param_buffer(self, pp_rank): + """Build the parameter buffer in each pp rank""" + model = self.pp_models[pp_rank] + weight_buffer_meta = get_weight_buffer_meta_from_module(model) + self.memory_buffers[pp_rank] = build_memory_buffer(weight_buffer_meta) + + def _build_param_references(self, pp_rank, maintain_weight=False): + model = self.pp_models[pp_rank] + build_memory_reference_from_module(model, self.memory_buffers[pp_rank], maintain_weight=maintain_weight) + + def _load_params_to_cuda(self, pp_rank, to_empty=False): + assert pp_rank != self.pp_rank, f"unexpected to load current pp rank [{pp_rank}] back to cuda" + for buffer in self.memory_buffers[pp_rank].values(): + if not to_empty: + buffer.data = buffer.data.to(torch.cuda.current_device(), non_blocking=True) + else: + buffer.data = torch.empty_like(buffer.data, device='cuda') + # rebuild reference after loading to CUDA + self._build_param_references(pp_rank) + + def _offload_params_to_cpu(self, pp_rank, to_empty=False): + assert pp_rank != self.pp_rank, f"unexpected to offload current pp rank [{pp_rank}] to cpu" + for buffer in self.memory_buffers[pp_rank].values(): + if not to_empty: + # offload the whole memory buffer to CPU + buffer.data = buffer.data.to('cpu', non_blocking=True) + else: + buffer.data = torch.empty_like(buffer.data, device='cpu') + self._build_param_references(pp_rank) + + def load_params_to_cuda(self, to_empty=False): + """load all model params to cuda""" + for cur_pp_rank in range(self.pp_size): + if cur_pp_rank != self.pp_rank: + self._load_params_to_cuda(cur_pp_rank, to_empty=to_empty) + + def allgather_params(self): + """allgather params of all pp ranks. Return a list of handles""" + for cur_pp_rank in range(self.pp_size): + global_src = dist.get_global_rank(group=self.pp_group, group_rank=cur_pp_rank) + + # NOTE(sgm): the async op may cause memory leakage of the memory_buffer/pp_models + for memory_buffer in self.memory_buffers[cur_pp_rank].values(): + dist.broadcast(tensor=memory_buffer.data, src=global_src, group=self.pp_group, async_op=False) + + def forward(self, *inputs, **kwargs): + try: + prev_output = None + for cur_chunk_rank in range(self._model_chunk_size): + if self._vpp_size: + mpu.set_virtual_pipeline_model_parallel_rank(cur_chunk_rank) + + for cur_pp_rank in range(self.pp_size): + mpu.set_pipeline_model_parallel_rank(cur_pp_rank) + self.pp_models[cur_pp_rank][cur_chunk_rank].set_input_tensor(prev_output) + ret = self.pp_models[cur_pp_rank][cur_chunk_rank](*inputs, **kwargs) + self.pp_models[cur_pp_rank][cur_chunk_rank].set_input_tensor(None) + prev_output = ret + finally: + if self._vpp_size: + mpu.set_virtual_pipeline_model_parallel_rank(0) + mpu.set_pipeline_model_parallel_rank(self.pp_rank) + return ret + + def __call__(self, *inputs, **kwargs): + return self.forward(*inputs, **kwargs) + + def eval(self): + for model in self.pp_models[self.pp_rank]: + model.eval() + + def train(self): + for model in self.pp_models[self.pp_rank]: + model.train() + + def offload_params_to_cpu(self, to_empty=False): + """offload params of models that are not of current pp rank to cpu""" + for cur_pp_rank in range(self.pp_size): + if cur_pp_rank != self.pp_rank: + self._offload_params_to_cpu(cur_pp_rank, to_empty=to_empty) + + def get_all_params(self): + """Get all the parameters of the models in all pp ranks + + Returns: + params: List[List[Dict[str, Tensor]]]: a list of parameters in all pp, where each is a list of dict + tensors of each model chunk + + """ + params = [] + for pp_rank in range(self.pp_size): + params.append([]) + for model_chunk_idx in range(len(self.pp_models[pp_rank])): + params[pp_rank].append({}) + pp_model = self.pp_models[pp_rank][model_chunk_idx] + pp_model = unwrap_model(pp_model, ((torchDDP, LocalDDP, Float16Module))) # not use Float16Module + for name, param in pp_model.named_parameters(): + # NOTE(gh) workaround: should not get lora params for inference + if 'lora' in name: + continue + params[pp_rank][model_chunk_idx][name] = param + + return params + + def update_this_rank_models(self, new_models): + self._this_rank_models = new_models + self._pp_models[self.pp_rank] = unwrap_model(new_models, (torchDDP, LocalDDP)) + + @property + def this_rank_models(self): + return self._this_rank_models + + @property + def pp_size(self): + return self._pp_size + + @property + def pp_rank(self): + return self._pp_rank + + @property + def pp_group(self): + return self._pp_group + + @property + def pp_models(self): + return self._pp_models + + +""" +Megatron Hybrid Engine: +- During training, only the current pp stage holds the parameters +- Before inference, broadcast the parameters of the current pp rank to all other pp ranks (all pp ranks holds all the parameters) +- Bind the parameters to the inference engine +- Do inference in tp. pp is treated as additional dp +- After inference, all the parameters that doesn't belong to this pp rank is freed. +""" + +from .base import BaseShardingManager + +import torch +from torch import nn +import torch.distributed +from torch.distributed import new_group + +from verl import DataProto +from verl.utils.torch_functional import (broadcast_dict_tensor, allgather_dict_tensors) +import verl.utils.megatron.tensor_parallel as tp_utils +from verl.third_party.vllm import parallel_state as vllm_ps +from verl.third_party.vllm import LLM +from verl.utils.model import normalize_pp_vpp_params +# Micro Data parallel group. Micro data parallel group is additional dp group that origins from splitting training tp +# into infer_tp and micro_tp. By default, we use order micro_dp - tp +_MICRO_DATA_PARALLEL_GROUP = None + + +class MegatronVLLMShardingManager(BaseShardingManager): + + def __init__(self, module: AllGatherPPModel, inference_engine: LLM, model_config, layer_name_mapping): + self.module = module + self.inference_engine = inference_engine + self.model_config = model_config + self.layer_name_mapping = layer_name_mapping + + # initialize micro_dp group for vllm inference + global _MICRO_DATA_PARALLEL_GROUP + world_size = torch.distributed.get_world_size() + rank = torch.distributed.get_rank() + train_tensor_parallel_size = mpu.get_tensor_model_parallel_world_size() + infer_tensor_parallel_size = vllm_ps.get_tensor_model_parallel_world_size() + + # TODO(sgm): this may not be true for FSDP -> vLLM + assert infer_tensor_parallel_size <= train_tensor_parallel_size, \ + 'Not implemented for infer_tp > train_tp' + assert train_tensor_parallel_size % infer_tensor_parallel_size == 0 + + micro_dp_size = train_tensor_parallel_size // infer_tensor_parallel_size + num_micro_dp_groups = world_size // micro_dp_size + assert _MICRO_DATA_PARALLEL_GROUP is None, ("micro data parallel group is already initialized") + for i in range(num_micro_dp_groups): + ranks = range(i * micro_dp_size, (i + 1) * micro_dp_size) + group = new_group(ranks=ranks) + if rank in ranks: + _MICRO_DATA_PARALLEL_GROUP = group + + def default_tp_concat_fn(self, name, param, infer_params, model_config): + """ + name: name of the parameter + param: training parameters + infer_params (List[torch.Tensor]): a list of parameters all-gathered from micro_dp_group + model_config: huggingface model_config + TODO(zhangchi.usc1992): currently, the implementation is adhoc. We can move this function to the model + definition so that it is model-agnostic. If the model doesn't implement this function, + we can throw an error to force user disable TP HybridEngine. + """ + + if self.layer_name_mapping.get("qkv_layer_name") in name: + # if the tensor is qkv, for each param on tp, split into q, k, v + # concat q, k, v separately. + q_lst = [] + k_lst = [] + v_lst = [] + assert model_config.num_attention_heads % model_config.num_key_value_heads == 0 + num_q_per_kv = model_config.num_attention_heads // model_config.num_key_value_heads + assert infer_params[0].shape[0] % (num_q_per_kv + 2) == 0 + kv_size_per_tp = infer_params[0].shape[0] // (num_q_per_kv + 2) + split_size = [kv_size_per_tp * num_q_per_kv, kv_size_per_tp, kv_size_per_tp] + for infer_param in infer_params: + q, k, v = infer_param.split(split_size) + q_lst.append(q) + k_lst.append(k) + v_lst.append(v) + q = torch.cat(q_lst, dim=0) + k = torch.cat(k_lst, dim=0) + v = torch.cat(v_lst, dim=0) + + infer_params = torch.cat((q, k, v), dim=0) + + elif self.layer_name_mapping.get("gate_proj_layer_name") in name: + # if the tensor is gate and proj + gate_lst = [] + up_lst = [] + for infer_param in infer_params: + gate, up = infer_param.chunk(2) + gate_lst.append(gate) + up_lst.append(up) + gate = torch.cat(gate_lst, dim=0) + up = torch.cat(up_lst, dim=0) + infer_params = torch.cat((gate, up), dim=0) + + else: + # concat tensor + infer_params = torch.cat(infer_params, dim=tp_utils.get_tensor_parallel_partition_dim(param)) + + return infer_params + + def _post_process_params(self, params): + """ + For each param, if it is a tp-splited param, we all-gather from micro_dp group. + """ + # here the params are in train tp format. we iterate params and all-gather + # TODO(zhangchi.usc1992) We can consider copy non-tp weight to another infer buffer. + # In this way, all the params in the original memory_buffers and can be offload. + micro_dp_size = get_micro_data_parallel_world_size() + micro_dp_group = get_micro_data_parallel_group() + + if micro_dp_size <= 1: + return + + origin_params = {} + for name in params.keys(): + param = params[name] + if tp_utils.is_tensor_parallel_param(param): + # allocate a new tensor with proper size + infer_params = [torch.empty_like(param) for _ in range(micro_dp_size)] + torch.distributed.all_gather(infer_params, param, group=micro_dp_group) + infer_params = self.default_tp_concat_fn(name, param, infer_params, self.model_config) + # replace with original param + params[name] = infer_params + origin_params[name] = param + + return origin_params + + def __enter__(self): + # create a new cuda space for parameters not in this pp rank + self.module.load_params_to_cuda() + # broadcast the parameters from pp rank to other ranks + self.module.allgather_params() + # obtain name to parameters in pp/vpp + params = self.module.get_all_params() + + # bind the params to inference engine + self.params = normalize_pp_vpp_params(params=params, + num_hidden_layers=self.model_config.num_hidden_layers, + layer_name='layers') + self.origin_params = self._post_process_params(self.params) + self.inference_engine.sync_model_weights(self.params, load_format='megatron') + + def __exit__(self, exc_type, exc_value, traceback): + # offload parameters doesn't belong to this pp rank + self.module.offload_params_to_cpu() + + # FIXME(sgm): the best practice is to delete the cuda tensor + # rebind the model weights, can be any cpu tensor + if get_micro_data_parallel_world_size() > 1: + for name in self.params.keys(): + self.params[name] = self.origin_params[name] + + # self.inference_engine.sync_model_weights(params) + self.inference_engine.offload_model_weights() + + self.module.train() + + # add empty cache after each compute + torch.cuda.empty_cache() + + def preprocess_data(self, data: DataProto) -> DataProto: + # prompts are identical for each training tp. We select for each inference tp + micro_dp_size = get_micro_data_parallel_world_size() + micro_dp_rank = get_micro_data_parallel_rank() + + # broadcast from tp=0 to other tp ranks + broadcast_dict_tensor(data.batch, + src=mpu.get_tensor_model_parallel_src_rank(), + group=mpu.get_tensor_model_parallel_group()) + + if micro_dp_size > 1: + local_prompts = data.chunk(chunks=micro_dp_size) + data = local_prompts[micro_dp_rank] + + return data + + def postprocess_data(self, data: DataProto) -> DataProto: + meta_info = data.meta_info + # all gather batch among micro-dp groups + micro_dp_size = get_micro_data_parallel_world_size() + if micro_dp_size > 1: + data.batch = allgather_dict_tensors(data.batch.contiguous(), + size=get_micro_data_parallel_world_size(), + group=get_micro_data_parallel_group(), + dim=0) + + # all gather batch among pp group + if meta_info.get('allgather_pp_output', True): + data.batch = allgather_dict_tensors(data.batch.contiguous(), + size=mpu.get_pipeline_model_parallel_world_size(), + group=mpu.get_pipeline_model_parallel_group(), + dim=0) + return data + + +""" +Micro Data parallel group +""" + + +def get_micro_data_parallel_group(): + assert _MICRO_DATA_PARALLEL_GROUP is not None + return _MICRO_DATA_PARALLEL_GROUP + + +def get_micro_data_parallel_world_size(): + return torch.distributed.get_world_size(group=get_micro_data_parallel_group()) + + +def get_micro_data_parallel_rank(): + return torch.distributed.get_rank(group=get_micro_data_parallel_group()) diff --git a/frameworks/trinity/__init__.py b/frameworks/trinity/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..b29fbc48919680e94f1c8cfd36d30c3cc238e813 --- /dev/null +++ b/frameworks/trinity/__init__.py @@ -0,0 +1,4 @@ +# -*- coding: utf-8 -*- +"""Trinity-RFT (Reinforcement Fine-Tuning)""" + +__version__ = "0.5.0.dev0" diff --git a/frameworks/trinity/algorithm/__init__.py b/frameworks/trinity/algorithm/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a97652b2530b238ec0108a049da2a04c8fc82fc1 --- /dev/null +++ b/frameworks/trinity/algorithm/__init__.py @@ -0,0 +1,52 @@ +from trinity.algorithm.advantage_fn import ADVANTAGE_FN, AdvantageFn +from trinity.algorithm.algorithm import AlgorithmType +from trinity.algorithm.entropy_loss_fn import ENTROPY_LOSS_FN, EntropyLossFn +from trinity.algorithm.kl_fn import KL_FN, KLFn +from trinity.algorithm.policy_loss_fn.sdpo_policy_loss import compute_self_distillation_loss +from trinity.algorithm.policy_loss_fn import POLICY_LOSS_FN, PolicyLossFn +from trinity.algorithm.sample_strategy import SAMPLE_STRATEGY, SampleStrategy +from trinity.utils.registry import Registry + +ALGORITHM_TYPE: Registry = Registry( + "algorithm", + default_mapping={ + "sft": "trinity.algorithm.algorithm.SFTAlgorithm", + "ppo": "trinity.algorithm.algorithm.PPOAlgorithm", + "grpo": "trinity.algorithm.algorithm.GRPOAlgorithm", + "reinforceplusplus": "trinity.algorithm.algorithm.ReinforcePlusPlusAlgorithm", + "rloo": "trinity.algorithm.algorithm.RLOOAlgorithm", + "opmd": "trinity.algorithm.algorithm.OPMDAlgorithm", + "asymre": "trinity.algorithm.algorithm.AsymREAlgorithm", + "dpo": "trinity.algorithm.algorithm.DPOAlgorithm", + "topr": "trinity.algorithm.algorithm.TOPRAlgorithm", + "cispo": "trinity.algorithm.algorithm.CISPOAlgorithm", + "gspo": "trinity.algorithm.algorithm.GSPOAlgorithm", + "sapo": "trinity.algorithm.algorithm.SAPOAlgorithm", + "mix": "trinity.algorithm.algorithm.MIXAlgorithm", + "mix_chord": "trinity.algorithm.algorithm.MIXCHORDAlgorithm", + "raft": "trinity.algorithm.algorithm.RAFTAlgorithm", + "sppo": "trinity.algorithm.algorithm.sPPOAlgorithm", + "rec": "trinity.algorithm.algorithm.RECAlgorithm", + "multi_step_grpo": "trinity.algorithm.algorithm.MultiStepGRPOAlgorithm", + "on_policy_distill": "trinity.algorithm.algorithm.OnPolicyDistillAlgorithm", + "jsd": "trinity.algorithm.algorithm.JSDAlgorithm", + "sdpo": "trinity.algorithm.algorithm.SDPOAlgorithm", + }, +) + + +__all__ = [ + "ALGORITHM_TYPE", + "AlgorithmType", + "AdvantageFn", + "ADVANTAGE_FN", + "PolicyLossFn", + "POLICY_LOSS_FN", + "KLFn", + "KL_FN", + "EntropyLossFn", + "ENTROPY_LOSS_FN", + "SampleStrategy", + "SAMPLE_STRATEGY", + "compute_self_distillation_loss", +] diff --git a/frameworks/trinity/algorithm/advantage_fn/__init__.py b/frameworks/trinity/algorithm/advantage_fn/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a7dd9e3a3524e871d48643590bb20d580f745367 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/__init__.py @@ -0,0 +1,32 @@ +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn, GroupAdvantage +from trinity.utils.registry import Registry + +ADVANTAGE_FN: Registry = Registry( + "advantage_fn", + default_mapping={ + "ppo": "trinity.algorithm.advantage_fn.ppo_advantage.PPOAdvantageFn", + "grpo": "trinity.algorithm.advantage_fn.grpo_advantage.GRPOGroupedAdvantage", + "grpo_verl": "trinity.algorithm.advantage_fn.grpo_advantage.GRPOAdvantageFn", + "step_wise_grpo": "trinity.algorithm.advantage_fn.multi_step_grpo_advantage.StepWiseGRPOAdvantageFn", + "reinforceplusplus": "trinity.algorithm.advantage_fn.reinforce_plus_plus_advantage.REINFORCEPLUSPLUSAdvantageFn", + "reinforce": "trinity.algorithm.advantage_fn.reinforce_advantage.REINFORCEGroupAdvantage", + "remax": "trinity.algorithm.advantage_fn.remax_advantage.REMAXAdvantageFn", + "rloo": "trinity.algorithm.advantage_fn.rloo_advantage.RLOOAdvantageFn", + "opmd": "trinity.algorithm.advantage_fn.opmd_advantage.OPMDGroupAdvantage", + "opmd_verl": "trinity.algorithm.advantage_fn.opmd_advantage.OPMDAdvantageFn", + "asymre": "trinity.algorithm.advantage_fn.asymre_advantage.ASYMREGroupAdvantage", + "asymre_verl": "trinity.algorithm.advantage_fn.asymre_advantage.ASYMREAdvantageFn", + "rec": "trinity.algorithm.advantage_fn.rec_advantage.RECGroupedAdvantage", + "on_policy_distill": "trinity.algorithm.advantage_fn.on_policy_distill_advantage.OnPolicyDistillAdvantage", + "multi_turn_opd": "trinity.algorithm.advantage_fn.on_policy_distill_advantage.MultiTurnOpdAdvantage", + "guided_opd": "trinity.algorithm.advantage_fn.guided_opd_advantage.GuidedOpdAdvantage", + "jsd": "trinity.algorithm.advantage_fn.jsd_advantage.JSDAdvantage", + "sdpo": "trinity.algorithm.advantage_fn.sdpo_advantage.SDPOAdvantage", + }, +) + +__all__ = [ + "ADVANTAGE_FN", + "AdvantageFn", + "GroupAdvantage", +] diff --git a/frameworks/trinity/algorithm/advantage_fn/advantage_fn.py b/frameworks/trinity/algorithm/advantage_fn/advantage_fn.py new file mode 100644 index 0000000000000000000000000000000000000000..6deee4d257ccfd9ee418d3ba0202cee40ea9f727 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/advantage_fn.py @@ -0,0 +1,83 @@ +from abc import ABC, abstractmethod +from typing import Any, Dict, List, Tuple + +from trinity.buffer.operators import ExperienceOperator +from trinity.common.experience import Experience +from trinity.utils.monitor import gather_metrics + + +class AdvantageFn(ABC): + @abstractmethod + def __call__(self, exps: Any, **kwargs: Dict) -> Tuple[Any, Dict]: + """Calculate advantages from experiences + + Args: + exps (`DataProto`): The input experiences. + kwargs (`Dict`): The step-level parameters for calculating advantages. + + Returns: + `Any`: The experiences with advantages. + `Dict`: The metrics for logging. + """ + + @classmethod + @abstractmethod + def default_args(cls) -> Dict: + """ + Returns: + `Dict`: The default init arguments for the advantage function. + """ + + @classmethod + def compute_in_trainer(cls) -> bool: + """Whether the advantage should be computed in the trainer loop.""" + return True + + +class GroupAdvantage(AdvantageFn, ExperienceOperator): + """For group-based advantages calculation.""" + + @abstractmethod + def group_experiences(self, exps: List[Experience]) -> Dict[str, List[Experience]]: + """Group experiences by a certain criterion. + + Args: + exps (List[Experience]): List of experiences to be grouped. + + Returns: + Dict[str, List[Experience]]: A dictionary where keys are group identifiers and values are lists of experiences. + """ + + def __call__(self, exps, **kwargs): + return self.process(exps) + + @abstractmethod + def calculate_group_advantage( + self, group_id: str, exps: List[Experience] + ) -> Tuple[List[Experience], Dict]: + """Calculate advantages for a group of experiences. + + Args: + group_id (str): The identifier for the group of experiences. + exps (List[Experience]): List of experiences in the group. + + Returns: + List[Experience]: A tuple containing the modified list of experiences and a dictionary of metrics. + """ + + def process(self, exps: List[Experience]) -> Tuple[List[Experience], Dict]: + if len(exps) == 0: + return [], {} + exp_groups = self.group_experiences(exps) + metric_list = [] + for group_id, group_exps in exp_groups.items(): + group_exps, group_metrics = self.calculate_group_advantage(group_id, group_exps) + metric_list.append(group_metrics) + metrics = gather_metrics(metric_list, "group_advantages") + exps = [exp for group in exp_groups.values() for exp in group] # Flatten the list + return exps, metrics + + @classmethod + def compute_in_trainer(cls) -> bool: + """Whether the advantage should be computed in the trainer loop.""" + return False diff --git a/frameworks/trinity/algorithm/advantage_fn/asymre_advantage.py b/frameworks/trinity/algorithm/advantage_fn/asymre_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..5c2ccf99327b5db8cc03d8ea1266b192d255403d --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/asymre_advantage.py @@ -0,0 +1,118 @@ +"""AsymRE advantage computation""" + +from collections import defaultdict +from typing import TYPE_CHECKING, Dict, List, Tuple + +import torch + +if TYPE_CHECKING: + from verl import DataProto + +from trinity.algorithm.advantage_fn import AdvantageFn, GroupAdvantage +from trinity.common.experience import Experience, group_by +from trinity.utils.annotations import Deprecated + + +@Deprecated +class ASYMREAdvantageFn(AdvantageFn): + """AsymRE advantage computation""" + + def __init__( + self, + baseline_shift: float = -0.1, + ) -> None: + self.baseline_shift = baseline_shift + + def __call__( + self, + exps: "DataProto", + **kwargs, + ) -> Tuple["DataProto", Dict]: + """Modified from compute_grpo_outcome_advantage + + Compute advantage for AsymRE, operating only on Outcome reward + (with only one scalar reward for each response). + + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + scores: `(torch.Tensor)` + shape: (bs, response_length) + """ + token_level_rewards = exps.batch["token_level_rewards"] + eos_mask = exps.batch["response_mask"] + + index = exps.non_tensor_batch["uid"] + baseline_shift = self.baseline_shift + + response_length = token_level_rewards.shape[-1] + scores = token_level_rewards.sum(dim=-1) + + id2score = defaultdict(list) + id2baseline = {} + + with torch.no_grad(): + bsz = scores.shape[0] + for i in range(bsz): + id2score[index[i]].append(scores[i]) + for idx in id2score: + if len(id2score[idx]) == 1: + id2baseline[idx] = torch.tensor(0.0) + # TODO: consider id2baseline[idx] = id2score[idx] (so that this sample won't take effect?) + elif len(id2score[idx]) > 1: + id2baseline[idx] = torch.mean(torch.tensor(id2score[idx])) + baseline_shift + else: + raise ValueError(f"no score in prompt index: {idx}") + for i in range(bsz): + scores[i] = scores[i] - id2baseline[index[i]] + scores = scores.unsqueeze(-1).tile([1, response_length]) * eos_mask + + exps.batch["advantages"] = scores + exps.batch["returns"] = scores + + metrics = { + # TODO: add meaningful metrics + } + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "baseline_shift": -0.1, + } + + +class ASYMREGroupAdvantage(GroupAdvantage): + """asymre Group Advantage computation""" + + def __init__(self, baseline_shift: float = -0.1, **kwargs) -> None: + super().__init__(**kwargs) + self.baseline_shift = baseline_shift + + def group_experiences(self, exps): + return group_by(exps, id_type="task") + + def calculate_group_advantage( + self, group_id: str, exps: List[Experience] + ) -> Tuple[List[Experience], Dict]: + with torch.no_grad(): + if len(exps) == 1: + group_baseline = torch.tensor(0.0) + else: + group_rewards = torch.tensor([exp.reward for exp in exps], dtype=torch.float32) + group_baseline = torch.mean(group_rewards) + self.baseline_shift + for exp in exps: + score = exp.reward - group_baseline + exp.advantages = score * exp.action_mask + exp.returns = exp.advantages.clone() + metrics = { + "group_baseline": group_baseline.item(), + "reward_mean": group_baseline.item() - self.baseline_shift, + } + return exps, metrics + + @classmethod + def default_args(cls) -> dict: + return {"baseline_shift": -0.1} diff --git a/frameworks/trinity/algorithm/advantage_fn/grpo_advantage.py b/frameworks/trinity/algorithm/advantage_fn/grpo_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..4562a5a4b981563d093701848686c3c950a44fb4 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/grpo_advantage.py @@ -0,0 +1,232 @@ +"""GRPO advantage computation +""" + +import copy +from collections import defaultdict +from typing import TYPE_CHECKING, Dict, List, Optional, Tuple + +import torch + +if TYPE_CHECKING: + from verl import DataProto + +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn, GroupAdvantage +from trinity.common.experience import Experience, group_by +from trinity.utils.annotations import Deprecated +from trinity.utils.monitor import gather_metrics + + +@Deprecated +class GRPOAdvantageFn(AdvantageFn): + """GRPO advantage computation""" + + def __init__( + self, + epsilon: float = 1e-6, + ) -> None: + self.epsilon = epsilon + + def __call__( + self, + exps: "DataProto", + **kwargs, + ) -> Tuple["DataProto", Dict]: + """ + Compute advantage for GRPO, operating only on Outcome reward + (with only one scalar reward for each response). + Ref: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py + + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + scores: `(torch.Tensor)` + shape: (bs, response_length) + """ + token_level_rewards = exps.batch["token_level_rewards"] + eos_mask = exps.batch["response_mask"] + index = exps.non_tensor_batch["uid"] + epsilon = self.epsilon + + response_length = token_level_rewards.shape[-1] + scores = token_level_rewards.sum(dim=-1) + + id2score = defaultdict(list) + id2mean = {} + id2std = {} + + with torch.no_grad(): + bsz = scores.shape[0] + for i in range(bsz): + id2score[index[i]].append(scores[i]) + for idx in id2score: + if len(id2score[idx]) == 1: + id2mean[idx] = torch.tensor(0.0) + id2std[idx] = torch.tensor(1.0) + elif len(id2score[idx]) > 1: + id2mean[idx] = torch.mean(torch.tensor(id2score[idx], dtype=torch.float32)) + id2std[idx] = torch.std(torch.tensor(id2score[idx], dtype=torch.float32)) + else: + raise ValueError(f"no score in prompt index: {idx}") + for i in range(bsz): + scores[i] = (scores[i] - id2mean[index[i]]) / (id2std[index[i]] + epsilon) + scores = scores.unsqueeze(-1).tile([1, response_length]) * eos_mask + + exps.batch["advantages"] = scores + exps.batch["returns"] = scores + + metrics = { + # TODO: add meaningful metrics + } + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "epsilon": 1e-6, + } + + +class GRPOGroupedAdvantage(GroupAdvantage): + """An advantage class that calculates GRPO advantages.""" + + def __init__( + self, + epsilon: float = 1e-6, + std_threshold: Optional[float] = None, + duplicate_experiences: bool = False, + rank_penalty: Optional[float] = None, + std_cal_level: str = "group", # "group" or "batch" + ) -> None: + """Initialize the GRPO advantage function. + + Args: + epsilon (float): A small value to avoid division by zero. + std_threshold (Optional[float]): If provided, groups with a reward standard deviation equal + or below this threshold will be skipped. + duplicate_experiences (bool): If True, allows duplicate experiences to keep the original experience + count. Only used when `std_threshold` is not None (https://hkunlp.github.io/blog/2025/Polaris). + rank_penalty (Optional[float]): A penalty applied to the rank of rewards to correct for bias + (https://arxiv.org/pdf/2506.02355). + std_cal_level (str): The scope for calculating the reward standard deviation for normalization. + Can be 'group' (default, std is calculated per group) or 'batch' (std is calculated + across the entire batch). The mean is always calculated per group. + Calculating the mean at the local (group) level and the standard deviation at the global (batch) + level enables more robust reward shaping(https://arxiv.org/pdf/2508.08221v1). + """ + self.epsilon = epsilon + self.std_threshold = std_threshold + self.duplicate_experiences = duplicate_experiences + self.rank_penalty = rank_penalty + self.std_cal_level = std_cal_level + if self.std_cal_level not in ["group", "batch"]: + raise ValueError("std_cal_level must be either 'group' or 'batch'") + + def group_experiences(self, exps): + return group_by(exps, id_type="task") + + def calculate_group_advantage( + self, + group_id: str, + exps: List[Experience], + precomputed_std: Optional[torch.Tensor] = None, + ) -> Tuple[List[Experience], Dict]: + metrics = {} + with torch.no_grad(): + if len(exps) == 1: + group_reward_mean = torch.tensor( + 0.0 + ) # check this value (use exps[0].reward may be better) + group_reward_std = torch.tensor(1.0) # set to 1.0 to avoid division by zero + if self.std_threshold is not None: + metrics["skipped_count"] = 1 + exps.clear() # Clear experiences if only one experience + else: + rewards = torch.tensor([exp.reward for exp in exps], dtype=torch.float32) + + if self.rank_penalty is not None: + # Correct bias by adjusting rewards based on their ranks + old_log_probs = torch.tensor( + [torch.mean(exp.logprobs, axis=-1) for exp in exps] + ) + group_ranks = torch.argsort(torch.argsort(old_log_probs)) + group_ranks = group_ranks / len(group_ranks) + rewards = rewards * (1 - group_ranks * self.rank_penalty) + + group_reward_mean = torch.mean(rewards) + group_reward_std = torch.std(rewards) + + # If the reward standard deviation is below a threshold, skip the group + if self.std_threshold is not None and group_reward_std <= self.std_threshold: + metrics["skipped_count"] = len(exps) + exps.clear() + + for exp in exps: + if self.std_cal_level == "batch" and precomputed_std is not None: + score = (exp.reward - group_reward_mean) / (precomputed_std + self.epsilon) + else: + score = (exp.reward - group_reward_mean) / (group_reward_std + self.epsilon) + exp.advantages = score * exp.action_mask + exp.returns = exp.advantages.clone() + + metrics["reward_mean"] = group_reward_mean.item() + metrics["reward_std"] = group_reward_std.item() + + return exps, metrics + + def _duplicate_experiences(self, exp_groups: Dict[str, List[Experience]]) -> List[Experience]: + original_group_num = len(exp_groups) + valid_groups = [group_exps for group_exps in exp_groups.values() if len(group_exps) > 0] + skipped_group_num = original_group_num - len(valid_groups) + + if skipped_group_num == 0: + return [exp for group in valid_groups for exp in group] + elif skipped_group_num == original_group_num: + # All groups are skipped, return an empty list + return [] + else: + idx = torch.randint(0, len(valid_groups), (skipped_group_num,)).tolist() + duplicated_groups = [copy.deepcopy(valid_groups[i]) for i in idx] + duplicated_exps = [exp for group in duplicated_groups for exp in group] + exps = [exp for group in valid_groups for exp in group] + exps.extend(duplicated_exps) + return exps + + def process(self, exps): + exp_groups = self.group_experiences(exps) + metric_list = [] + precomputed_std = None + if self.std_cal_level == "batch": + all_rewards = torch.tensor( + [exp.reward for exp in exps], dtype=torch.float32 + ) # All rewards in the batch + if len(all_rewards) <= 1: + precomputed_std = torch.tensor(1.0) + else: + precomputed_std = torch.std(all_rewards) + for group_id, group_exps in exp_groups.items(): + group_exps, group_metrics = self.calculate_group_advantage( + group_id, group_exps, precomputed_std=precomputed_std + ) + metric_list.append(group_metrics) + + # Update the filtered_count metric + filtered_count = sum(metric.pop("skipped_count", 0) for metric in metric_list) + metrics = gather_metrics(metric_list, "group_advantages") + metrics["filtered_count"] = filtered_count + if self.duplicate_experiences and self.std_threshold is not None: + exps = self._duplicate_experiences(exp_groups) + else: + exps = [exp for group in exp_groups.values() for exp in group] # Flatten the list + return exps, metrics + + @classmethod + def default_args(cls) -> dict: + return { + "epsilon": 1e-6, + "std_threshold": None, + "duplicate_experiences": False, + "rank_penalty": None, + "std_cal_level": "group", + } diff --git a/frameworks/trinity/algorithm/advantage_fn/guided_opd_advantage.py b/frameworks/trinity/algorithm/advantage_fn/guided_opd_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..b647ce50b25944150e7bf2b1086ddd90bf3c282c --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/guided_opd_advantage.py @@ -0,0 +1,163 @@ +# -*- coding: utf-8 -*- +"""Guided-OPD advantage: the divergence is chosen per turn by who generated it. + +Reference: "On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn +Agents" (arXiv:2606.15912), Eqs. (4), (5), (9). + +Vanilla OPD applies one divergence everywhere. Guided-OPD applies a different one +depending on which policy sampled the turn -- and the sampling policy is exactly what +decides which estimator is even valid: + + STUDENT turn -> reverse KL D(pi_theta || pi_phi) + grad D_rev = E_{y~pi_theta}[(log pi_theta - log pi_phi) grad log pi_theta] + because the score-function term E[grad log pi_theta] vanishes. Ascending the + policy gradient with A = log pi_phi - log pi_theta therefore minimises it, which + is precisely what MultiTurnOpdAdvantage already computes. Student turns need no + new code, and this file reuses that formula unchanged. + + TEACHER turn -> forward KL D(pi_phi || pi_theta) + grad D_fwd = -E_{a~pi_phi}[grad log pi_theta(a)] + The log pi_phi(a) term does not depend on theta, so on teacher-SAMPLED tokens the + single-sample estimator of the forward-KL gradient is plain cross-entropy. + Ascending the policy gradient with a positive CONSTANT advantage reproduces that + gradient exactly, up to the scale of the constant. + +So the method needs no new loss function: keep the OPD advantage on student turns, put +a constant on teacher turns. What has to be new is only the plumbing that says which +turn is which -- see `role_field`. + +ESTIMATOR CAVEAT. The above is the sampled estimator of the forward KL. The +distribution-level one sums over the vocabulary (or a top-k truncation of it) and needs +teacher logits at every position, not just the logprob of the realised token. The paper +says "token-level divergence between the teacher and student conditional distributions" +without naming an estimator. This file implements the sampled version, which is the one +that costs nothing extra; if the distribution-level variant is wanted later it needs +teacher top-k logits carried through the workflow, which is deliberately not assumed +here. +""" + +from typing import Dict, Tuple + +import torch +from verl import DataProto + +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn + +# Name of the per-turn role flag. The workflow writes it into `Experience.info` and +# declares a matching `CustomField`; trinity/trainer/verl/utils.py turns that into a +# [num_turns] tensor on the batch. 1.0 == teacher generated this turn. +ROLE_FIELD = "is_teacher_turn" + + +class GuidedOpdAdvantage(AdvantageFn): + """Asymmetric per-turn advantage for Guided-OPD. + + Batch layout is the same as MultiTurnOpdAdvantage: one row per TURN, not per + trajectory, so the role flag is one scalar per row and broadcasts over the sequence. + + Args: + kl_coef: scale on the reverse-KL advantage of student turns. Matches + MultiTurnOpdAdvantage so that beta=0 reproduces vanilla OPD exactly. + forward_kl_coef: the constant advantage placed on teacher turns. This is the + relative weight of the two objectives in Eq. (9); the paper writes + L = L_stu + L_tea with no explicit weight, i.e. 1.0. + role_field: batch key holding the per-turn role flag. + """ + + def __init__( + self, + kl_coef: float = 1.0, + forward_kl_coef: float = 1.0, + role_field: str = ROLE_FIELD, + ) -> None: + self.kl_coef = kl_coef + self.forward_kl_coef = forward_kl_coef + self.role_field = role_field + + def __call__(self, exps: DataProto, **kwargs) -> Tuple[DataProto, Dict]: + old_log_probs = exps.batch["old_log_probs"] # student, recomputed by the actor + teacher_log_probs = exps.batch["teacher_logprobs"] + response_mask = exps.batch["response_mask"] + + if self.role_field not in exps.batch: + raise KeyError( + f"GuidedOpdAdvantage needs a per-turn '{self.role_field}' flag on the batch. " + "The workflow must set Experience.info['is_teacher_turn'] AND declare the " + "matching CustomField on every experience it returns -- " + "trinity/trainer/verl/utils.py only forwards custom fields when they are " + "identical across the whole batch." + ) + + teacher_valid_mask = exps.batch.get("teacher_logprobs_valid_mask") + if teacher_valid_mask is not None: + teacher_valid_mask = teacher_valid_mask.to( + dtype=response_mask.dtype, device=response_mask.device + ) + else: + teacher_valid_mask = response_mask + effective_mask = response_mask & teacher_valid_mask + + # [num_turns] -> [num_turns, 1] so it broadcasts over the response dimension. + is_teacher = ( + exps.batch[self.role_field].to(dtype=old_log_probs.dtype, device=old_log_probs.device) + .view(-1, 1) + ) + + # Reverse KL on student turns; a positive constant (== cross-entropy gradient) + # on teacher turns. Both branches are computed densely and then selected, which + # keeps the shapes static and costs one extra elementwise op. + student_adv = self.kl_coef * (teacher_log_probs - old_log_probs) + teacher_adv = torch.full_like(old_log_probs, float(self.forward_kl_coef)) + advantages = (1.0 - is_teacher) * student_adv + is_teacher * teacher_adv + advantages = advantages * effective_mask + + exps.batch["advantages"] = advantages + exps.batch["returns"] = advantages.clone() + + metrics = self._metrics( + old_log_probs, teacher_log_probs, effective_mask, is_teacher, advantages + ) + return exps, metrics + + def _metrics( + self, + old_log_probs: torch.Tensor, + teacher_log_probs: torch.Tensor, + effective_mask: torch.Tensor, + is_teacher: torch.Tensor, + advantages: torch.Tensor, + ) -> Dict: + # Both divergences are non-negative in expectation, but only under their own + # sampling distribution: (student - teacher) on student-sampled tokens estimates + # the reverse KL, (teacher - student) on teacher-sampled tokens the forward KL. + # Reporting them with a common sign convention would hide exactly that. + stu_tok = effective_mask & (is_teacher < 0.5) + tea_tok = effective_mask & (is_teacher >= 0.5) + + rkl = (old_log_probs - teacher_log_probs) * stu_tok + fkl = (teacher_log_probs - old_log_probs) * tea_tok + + n_stu = stu_tok.sum() + n_tea = tea_tok.sum() + n_turns = is_teacher.numel() + + metrics = { + # Realised beta, not the scheduled one: turns can end early, so the two + # differ and only this one describes the batch that was actually trained on. + "guided/beta_realized": (is_teacher.sum() / max(n_turns, 1)).item(), + "guided/turns_total": float(n_turns), + "guided/tokens_student": n_stu.item(), + "guided/tokens_teacher": n_tea.item(), + "advantages/mean": advantages.sum(dim=-1).mean().item(), + } + # Per-token so the two are comparable to each other and across steps; a sum + # would move with response length instead of with the divergence. + if n_stu > 0: + metrics["guided/reverse_kl_student_turns"] = (rkl.sum() / n_stu).item() + if n_tea > 0: + metrics["guided/forward_kl_teacher_turns"] = (fkl.sum() / n_tea).item() + return metrics + + @classmethod + def default_args(cls) -> Dict: + return {"kl_coef": 1.0, "forward_kl_coef": 1.0} diff --git a/frameworks/trinity/algorithm/advantage_fn/jsd_advantage.py b/frameworks/trinity/algorithm/advantage_fn/jsd_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..bd800ea125410585a1411de749b04dd3ec6d6b6b --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/jsd_advantage.py @@ -0,0 +1,200 @@ +# -*- coding: utf-8 -*- +"""Jensen-Shannon Divergence (JSD) advantage computation. + +JSD(P||Q) = beta * KL(teacher||M) + (1-beta) * KL(student||M), where M = beta*teacher + (1-beta)*student. +When beta=0.5, this gives the standard symmetric JSD. All computations in log-space (no exp). +Aligned with SWIFT: beta=0/1 yield pure KL; temperature and optional chunking supported. +""" + +from typing import Dict, Optional, Tuple + +import torch +from verl import DataProto + +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn + + +class JSDAdvantage(AdvantageFn): + """Advantage function using Jensen-Shannon Divergence (log-space, SWIFT-aligned). + + Computes JSD in log-space only: + - beta=0: JSD = KL(student || teacher) [pure KL] + - beta=1: JSD = KL(teacher || student) [pure KL] + - else: JSD = beta*KL(teacher||M) + (1-beta)*KL(student||M), M = mixture in log-space. + + The teacher_logprobs should be stored in Experience.teacher_logprobs + by the workflow during exploration. + """ + + def __init__( + self, + lambda_coef: float = 0.5, + kl_coef: float = 1.0, + temperature: float = 1.0, + chunk_size: Optional[int] = None, + ) -> None: + """Initialize JSD advantage function. + + Args: + lambda_coef: Weight beta for mixture. JSD = beta*KL(teacher||M) + (1-beta)*KL(student||M). + beta=0 => KL(student||teacher), beta=1 => KL(teacher||student). Range: [0, 1]. + kl_coef: Overall scaling coefficient for advantages. + temperature: Temperature scaling for log-probs (log_probs / temperature). 1.0 = no scaling. + chunk_size: If set, process flattened valid tokens in chunks to reduce peak memory; None = no chunking. + """ + self.lambda_coef = lambda_coef + self.kl_coef = kl_coef + self.temperature = temperature + self.chunk_size = chunk_size + + def _js_divergence_per_token( + self, + student_logprobs: torch.Tensor, + teacher_logprobs: torch.Tensor, + mask: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """ + Compute JSD per token in log-space only (no exp). + - beta=0: KL(student || teacher) = student_logprobs - teacher_logprobs + - beta=1: KL(teacher || student) = teacher_logprobs - student_logprobs + - else: mixture log_m = logsumexp([s+log(1-beta), t+log(beta)]); JSD = beta*(t-log_m) + (1-beta)*(s-log_m). + """ + beta = self.lambda_coef + s = student_logprobs + t = teacher_logprobs + + if beta == 0: + # Pure KL(student || teacher) + jsd = s - t + elif beta == 1: + # Pure KL(teacher || student) + jsd = t - s + else: + # Precompute log coefficients once + dtype, device = s.dtype, s.device + beta_t = torch.tensor(beta, dtype=dtype, device=device) + log_beta = torch.log(beta_t) + log_1_minus_beta = torch.log1p(-beta_t) + + # log(mixture) = log(beta*teacher + (1-beta)*student) + mixture_log = torch.logsumexp( + torch.stack([s + log_1_minus_beta, t + log_beta]), + dim=0, + ) + # KL(teacher || mixture) = t - log_m, KL(student || mixture) = s - log_m (log-space, no exp) + kl_teacher = t - mixture_log + kl_student = s - mixture_log + jsd = beta_t * kl_teacher + (1 - beta_t) * kl_student + + if mask is not None: + jsd = jsd * mask + return jsd + + def _js_divergence_per_token_chunked( + self, + student_logprobs: torch.Tensor, + teacher_logprobs: torch.Tensor, + response_mask: torch.Tensor, + ) -> torch.Tensor: + """Compute JSD per token with optional chunking over valid positions (for memory).""" + flat_s = student_logprobs.reshape(-1) + flat_t = teacher_logprobs.reshape(-1) + flat_mask = response_mask.reshape(-1) + valid = flat_mask > 0 + n_valid = valid.sum().item() + if n_valid == 0: + return (flat_s * 0).reshape_as(response_mask) + + s_valid = flat_s[valid] + t_valid = flat_t[valid] + chunk_size = self.chunk_size or n_valid + beta = self.lambda_coef + + if beta == 0: + jsd_valid = s_valid - t_valid + elif beta == 1: + jsd_valid = t_valid - s_valid + else: + dtype, device = s_valid.dtype, s_valid.device + beta_t = torch.tensor(beta, dtype=dtype, device=device) + log_beta = torch.log(beta_t) + log_1_minus_beta = torch.log1p(-beta_t) + jsd_valid = s_valid.new_zeros(s_valid.shape) + for start in range(0, n_valid, chunk_size): + end = min(start + chunk_size, n_valid) + s_chunk = s_valid[start:end] + t_chunk = t_valid[start:end] + mixture_log = torch.logsumexp( + torch.stack([s_chunk + log_1_minus_beta, t_chunk + log_beta]), + dim=0, + ) + kl_t = t_chunk - mixture_log + kl_s = s_chunk - mixture_log + jsd_valid[start:end] = beta_t * kl_t + (1 - beta_t) * kl_s + + out = flat_s.new_zeros(flat_s.shape) + out[valid] = jsd_valid + return out.reshape_as(response_mask) + + def __call__(self, exps: DataProto, **kwargs) -> Tuple[DataProto, Dict]: + """Compute advantages using JSD. + + advantages = kl_coef * JSD (same sign convention as OnPolicyDistillAdvantage). + When lambda_coef=1: JSD = KL(teacher||student) = (teacher - student), so + advantage = kl_coef * (teacher - student), matching on-policy distillation. + + Args: + exps: DataProto containing: + - old_log_probs: student's sampling logprobs [batch, seq] + - teacher_logprobs: teacher's logprobs [batch, seq] + - response_mask: mask for response tokens [batch, seq] + + Returns: + exps: DataProto with advantages and returns added + metrics: Dict with jsd and advantage statistics + """ + metrics = {} + + old_log_probs = exps.batch["old_log_probs"] # student sampling logprobs + teacher_log_probs = exps.batch["teacher_logprobs"] + response_mask = exps.batch["response_mask"] + + # Temperature scaling (align with SWIFT: logits / T => apply to log-probs here) + if self.temperature != 1.0: + old_log_probs = old_log_probs / self.temperature + teacher_log_probs = teacher_log_probs / self.temperature + + # Compute JSD per token (with optional chunking for memory) + if self.chunk_size is not None: + jsd_per_token = self._js_divergence_per_token_chunked( + old_log_probs, teacher_log_probs, response_mask + ) + else: + jsd_per_token = self._js_divergence_per_token( + old_log_probs, teacher_log_probs, mask=response_mask + ) + + # Same sign as OnPolicyDistill: advantage = kl_coef * (teacher - student) when lambda_coef=1 + advantages = self.kl_coef * jsd_per_token + advantages = advantages * response_mask + + exps.batch["advantages"] = advantages + exps.batch["returns"] = advantages.clone() + + # JSD metrics (over valid tokens) + jsd_sum = (jsd_per_token * response_mask).sum(dim=-1) + metrics["jsd/mean"] = jsd_sum.mean().item() + metrics["jsd/std"] = jsd_sum.std().item() if jsd_sum.numel() > 1 else 0.0 + + metrics["advantages/mean"] = advantages.sum(dim=-1).mean().item() + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "lambda_coef": 0.5, + "kl_coef": 1.0, + "temperature": 1.0, + "chunk_size": None, + } diff --git a/frameworks/trinity/algorithm/advantage_fn/multi_step_grpo_advantage.py b/frameworks/trinity/algorithm/advantage_fn/multi_step_grpo_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..72dea0ac22aab16841bd25027fc95cc52764a876 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/multi_step_grpo_advantage.py @@ -0,0 +1,199 @@ +"""GRPO advantage computation for multi-step scenarios +""" +from typing import Dict, List, Optional, Tuple + +import torch + +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn +from trinity.buffer.operators import ExperienceOperator +from trinity.common.experience import Experience, group_by +from trinity.utils.monitor import gather_metrics + + +class StepWiseGRPOAdvantageFn(AdvantageFn, ExperienceOperator): + """ + An advantage function that broadcasts advantages from the last step to previous steps. + Inspired by rLLM (https://github.com/rllm-org/rllm). + """ + + def __init__( + self, + epsilon: float = 1e-6, + enable_step_norm: bool = False, + std_cal_level: str = "group", # 'group' (task-level) or 'batch' + std_threshold: Optional[float] = None, + **kwargs, + ) -> None: + """Initialize the Step-wise GRPO advantage function. + + Args: + epsilon (float): A small value to avoid division by zero. + enable_step_norm (bool): If True, normalize advantages by trajectory length. + std_cal_level (str): The scope for calculating reward standard deviation. + 'group' (default): Std is calculated per task group. + 'batch': Std is calculated across all last-step rewards in the entire batch. + The mean is always calculated per task group. + std_threshold (Optional[float]): If provided, task groups with a reward standard deviation + equal or below this threshold will be skipped. + """ + self.epsilon = epsilon + self.enable_step_norm = enable_step_norm + self.std_cal_level = std_cal_level + self.std_threshold = std_threshold + if self.std_cal_level not in ["group", "batch"]: + raise ValueError("std_cal_level must be either 'group' or 'batch'") + + def calculate_last_step_advantage( + self, + exps: Dict[str, Experience], + precomputed_std: Optional[torch.Tensor] = None, + ) -> Tuple[Dict[str, float], Dict[str, float], bool]: + """Calculate group advantage for a given group of experiences. + + Args: + exps (Dict[str, Experience]): One experience per run, keyed by run ID. + precomputed_std (Optional[torch.Tensor]): Precomputed standard deviation for batch-level calculation. + + Returns: + Dict[str, float]: Scores for each run. + Dict[str, float]: Metrics for logging. + bool: Whether this group should be skipped. + """ + with torch.no_grad(): + if len(exps) == 1: + group_reward_mean = torch.tensor(0.0) + group_reward_std = torch.tensor(1.0) + else: + rewards = torch.tensor([exp.reward for exp in exps.values()], dtype=torch.float32) + group_reward_mean = torch.mean(rewards) + group_reward_std = torch.std(rewards) + + # Determine if this group should be skipped based on std_threshold + should_skip = False + if self.std_threshold is not None: + if len(exps) == 1 or group_reward_std <= self.std_threshold: + should_skip = True + + scores = {} + for rid, exp in exps.items(): + if self.std_cal_level == "batch" and precomputed_std is not None: + score = (exp.reward - group_reward_mean) / (precomputed_std + self.epsilon) + else: + score = (exp.reward - group_reward_mean) / (group_reward_std + self.epsilon) + scores[rid] = score.item() + metrics = { + "reward_mean": group_reward_mean.item(), + "reward_std": group_reward_std.item(), + } + return scores, metrics, should_skip + + def broadcast_advantages( + self, run_exps: Dict[str, List[Experience]], scores: Dict[str, float] + ) -> Dict[str, List[Experience]]: + """Broadcast the calculated advantages to all previous steps in each run. + + Args: + run_exps (Dict[str, List[Experience]]): Experiences grouped by run ID. + scores (Dict[str, float]): Calculated scores for each run. + + Returns: + Dict[str, List[Experience]]: Updated experiences with advantages broadcasted. + """ + for run_id, exps in run_exps.items(): + score = scores[run_id] + traj_length = len(exps) + for exp in exps: + exp.advantages = exp.action_mask * score # type: ignore [operator] + if self.enable_step_norm: + exp.advantages /= traj_length + exp.returns = exp.advantages.clone() + return run_exps + + def process(self, exps: List[Experience]) -> Tuple[List[Experience], Dict]: + if len(exps) == 0: + return [], {} + cnt = 0 + metric_list = [] + filtered_count = 0 + # Step 1: split the experiences into sub-groups by task + task_exps = group_by(exps, "task") + + # --- Pre-computation step for batch-level standard deviation --- + precomputed_std = None + if self.std_cal_level == "batch": + all_laststep_rewards = [] + for task_exp in task_exps.values(): + # First, group all experiences by run to find the last step of each run + task_run_exps = group_by(task_exp, "run") + # Collect rewards from the last step of every run in the entire batch + last_step_rewards = [ + run_steps[-1].reward for run_steps in task_run_exps.values() if run_steps + ] + all_laststep_rewards.extend(last_step_rewards) + + if len(all_laststep_rewards) <= 1: + precomputed_std = torch.tensor(1.0) + else: + precomputed_std = torch.std(torch.tensor(all_laststep_rewards, dtype=torch.float32)) + # --- End of pre-computation --- + + # Step 2: further split each task's experiences into sub-groups by run + result_exps = [] + total_task_groups = len(task_exps) + skipped_task_groups = 0 + + for task_exp in task_exps.values(): + run_exps = group_by(task_exp, "run") + + # Step3: extract the last experience (last step) from each run and calculate scores + last_step_exps = {run_id: step_exps[-1] for run_id, step_exps in run_exps.items()} + scores, metrics, should_skip = self.calculate_last_step_advantage( + last_step_exps, precomputed_std=precomputed_std + ) + + # Skip this task group if std is below threshold + if should_skip: + # Count all experiences in this task group as filtered + task_exp_count = sum(len(step_exps) for step_exps in run_exps.values()) + filtered_count += task_exp_count + skipped_task_groups += 1 + metric_list.append(metrics) + continue + + metric_list.append(metrics) + + # Step 4: broadcast the advantages to all previous steps + run_exps = self.broadcast_advantages(run_exps, scores) + for exps in run_exps.values(): + cnt += len(exps) + result_exps.extend(exps) + + metrics = gather_metrics(metric_list, "group_advantages") + metrics["experience_count"] = cnt + metrics["filtered_count"] = filtered_count + + # Calculate the ratio of skipped task groups + if total_task_groups > 0: + metrics["skipped_group_ratio"] = skipped_task_groups / total_task_groups + else: + metrics["skipped_group_ratio"] = 0.0 + + return result_exps, metrics + + def __call__(self, exps, **kwargs): + return self.process(exps) + + @classmethod + def compute_in_trainer(cls) -> bool: + """Whether the advantage should be computed in the trainer loop.""" + return False + + @classmethod + def default_args(cls) -> Dict: + """Return the default configuration for this strategy.""" + return { + "epsilon": 1e-6, + "enable_step_norm": False, + "std_threshold": None, + "std_cal_level": "group", + } diff --git a/frameworks/trinity/algorithm/advantage_fn/on_policy_distill_advantage.py b/frameworks/trinity/algorithm/advantage_fn/on_policy_distill_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..da278581e62c6e6753517fca60b5b419a5dcba04 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/on_policy_distill_advantage.py @@ -0,0 +1,196 @@ +# -*- coding: utf-8 -*- +"""On-Policy Distillation advantage computation. + +Reference: Tinker library's on-policy distillation. + +advantages = -(student_logprobs - teacher_logprobs) + = teacher_logprobs - student_logprobs + +For multi-turn workflows (OPD_alfworld_workflow, OPD_scienceworld_workflow, etc.): +- Workflow returns List[Experience] (one per turn), not a single response +- Each turn is one row in the batch; use MultiTurnOpdAdvantage +""" + +from collections import defaultdict +from typing import Dict, List, Tuple + +import torch +from verl import DataProto + +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn + + +def _compute_opd_advantage( + old_log_probs: torch.Tensor, + teacher_log_probs: torch.Tensor, + response_mask: torch.Tensor, + teacher_valid_mask: torch.Tensor, + kl_coef: float, +) -> Tuple[torch.Tensor, Dict]: + """Core OPD advantage computation with optional teacher_valid_mask. + + When teacher_valid_mask is provided (hint workflow), use it to mask positions + where teacher_logprobs was padded (e.g. due to length mismatch). + """ + advantages = kl_coef * (teacher_log_probs - old_log_probs) + + # apply mask + effective_mask = response_mask & teacher_valid_mask + advantages = advantages * effective_mask + + # metrics + kl_per_token = old_log_probs - teacher_log_probs + kl_sum = (kl_per_token * effective_mask).sum(dim=-1) + metrics = { + "kl/mean": kl_sum.mean().item(), + "kl/std": kl_sum.std().item() if kl_sum.numel() > 1 else 0.0, + "advantages/mean": advantages.sum(dim=-1).mean().item(), + } + return advantages, metrics + + +class OnPolicyDistillAdvantage(AdvantageFn): + """Advantage function for on-policy distillation. + + Computes: advantages = kl_coef * (teacher_logprobs - student_logprobs) + + The teacher_logprobs should be stored in Experience.teacher_logprobs + by the workflow during exploration. + """ + + def __init__(self, kl_coef: float = 1.0) -> None: + self.kl_coef = kl_coef + + def __call__(self, exps: DataProto, **kwargs) -> Tuple[DataProto, Dict]: + """Compute advantages from teacher and student logprobs. + + Args: + exps: DataProto containing: + - old_log_probs: student's sampling logprobs [batch, seq] + - teacher_logprobs: teacher's logprobs [batch, seq] + - response_mask: mask for response tokens [batch, seq] + + Returns: + exps: DataProto with advantages and returns added + metrics: Dict with kl and advantage statistics + """ + old_log_probs = exps.batch["old_log_probs"] + teacher_log_probs = exps.batch["teacher_logprobs"] + response_mask = exps.batch["response_mask"] + + # Standard OPD: teacher_valid_mask = response_mask (no length mismatch) + teacher_valid_mask = response_mask + + advantages, metrics = _compute_opd_advantage( + old_log_probs, + teacher_log_probs, + response_mask, + teacher_valid_mask, + self.kl_coef, + ) + + exps.batch["advantages"] = advantages + exps.batch["returns"] = advantages.clone() + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return {"kl_coef": 1.0} + + +def _parse_run_ids(unique_ids) -> List[str]: + """Parse unique_ids (format: batch/task/run/step/suffix) to get run_id (batch/task/run).""" + run_ids = [] + for uid in unique_ids: + parts = str(uid).split("/") + if len(parts) >= 4: + run_ids.append("/".join(parts[:3])) # batch/task/run + else: + run_ids.append(str(uid)) + return run_ids + + +class MultiTurnOpdAdvantage(AdvantageFn): + """Advantage function for multi-turn on-policy distillation. + + Used with OPD_alfworld_workflow, OPD_scienceworld_workflow, OPD_webshop_workflow + and TCOD variants where: + - Workflow returns List[Experience] (turn_responses), one Experience per turn + - Each turn is one row in the batch: batch dim = num_turns across all episodes + - Same per-token formula: advantages = kl_coef * (teacher_logprobs - student_logprobs) + + Unlike single-turn OPD where batch = [num_samples], here batch = [num_turns]. + Adds trajectory-level metrics (kl/trajectory_mean) by grouping turns by run_id. + """ + + def __init__(self, kl_coef: float = 1.0) -> None: + self.kl_coef = kl_coef + + def __call__(self, exps: DataProto, **kwargs) -> Tuple[DataProto, Dict]: + """Compute advantages for multi-turn OPD (list of turn responses). + + Args: + exps: DataProto containing: + - old_log_probs: student's sampling logprobs [num_turns, seq] + - teacher_logprobs: teacher's logprobs [num_turns, seq] + - response_mask: mask for response tokens [num_turns, seq] + - unique_ids: (optional) for trajectory grouping, format batch/task/run/step/suffix + + Returns: + exps: DataProto with advantages and returns added + metrics: Dict with kl, advantage, and trajectory-level statistics + """ + old_log_probs = exps.batch["old_log_probs"] + teacher_log_probs = exps.batch["teacher_logprobs"] + response_mask = exps.batch["response_mask"] + + teacher_valid_mask = exps.batch.get("teacher_logprobs_valid_mask") + if teacher_valid_mask is not None: + teacher_valid_mask = teacher_valid_mask.to( + dtype=response_mask.dtype, device=response_mask.device + ) + else: + teacher_valid_mask = response_mask + + advantages, metrics = _compute_opd_advantage( + old_log_probs, + teacher_log_probs, + response_mask, + teacher_valid_mask, + self.kl_coef, + ) + + exps.batch["advantages"] = advantages + exps.batch["returns"] = advantages.clone() + + # Trajectory-level metrics: group by run_id and sum KL per trajectory + unique_ids = exps.batch.get("unique_ids") + if unique_ids is not None: + run_ids = _parse_run_ids(unique_ids) + kl_per_token = old_log_probs - teacher_log_probs + effective_mask = response_mask & teacher_valid_mask + kl_per_turn = (kl_per_token * effective_mask).sum(dim=-1) + + traj_kl_sums: Dict[str, float] = defaultdict(float) + for run_id, kl in zip(run_ids, kl_per_turn.tolist()): + traj_kl_sums[run_id] += kl + + if traj_kl_sums: + traj_kl_list = list(traj_kl_sums.values()) + n = len(traj_kl_list) + mean_kl = sum(traj_kl_list) / n + metrics["kl/trajectory_mean"] = mean_kl + metrics["kl/trajectory_std"] = ( + (sum((x - mean_kl) ** 2 for x in traj_kl_list) / n) ** 0.5 + if n > 1 + else 0.0 + ) + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return {"kl_coef": 1.0} + + diff --git a/frameworks/trinity/algorithm/advantage_fn/opmd_advantage.py b/frameworks/trinity/algorithm/advantage_fn/opmd_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..8c0a5869861e2001b882ae4a3e08ced3a77b7532 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/opmd_advantage.py @@ -0,0 +1,142 @@ +"""OPMD advantage computation""" + +from collections import defaultdict +from typing import TYPE_CHECKING, Dict, List, Tuple + +import torch + +if TYPE_CHECKING: + from verl import DataProto + +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn, GroupAdvantage +from trinity.common.experience import Experience, group_by +from trinity.utils.annotations import Deprecated + + +@Deprecated +class OPMDAdvantageFn(AdvantageFn): + """OPMD advantage computation""" + + def __init__( + self, + opmd_baseline: str = "mean", + tau: float = 1.0, + ) -> None: + self.opmd_baseline = opmd_baseline + self.tau = tau + + def __call__( + self, + exps: "DataProto", + **kwargs, + ) -> Tuple["DataProto", Dict]: + """Modified from compute_grpo_outcome_advantage + + Compute advantage for OPMD, operating only on Outcome reward + (with only one scalar reward for each response). + + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + scores: `(torch.Tensor)` + shape: (bs, response_length) + """ + token_level_rewards = exps.batch["token_level_rewards"] + eos_mask = exps.batch["response_mask"] + # TODO (yanxi): confirm consistency with exps.batch["attention_mask"][:, -response_length:] in original implementation + index = exps.non_tensor_batch["uid"] + opmd_baseline = self.opmd_baseline + tau = self.tau + + response_length = token_level_rewards.shape[-1] + scores = token_level_rewards.sum(dim=-1) + + id2score = defaultdict(list) + id2baseline = {} + + with torch.no_grad(): + bsz = scores.shape[0] + for i in range(bsz): + id2score[index[i]].append(scores[i]) + for idx in id2score: + if len(id2score[idx]) == 1: + id2baseline[idx] = torch.tensor(0.0) + # TODO: consider id2baseline[idx] = id2score[idx] (so that this sample won't take effect?) + elif len(id2score[idx]) > 1: + if opmd_baseline == "mean": + id2baseline[idx] = torch.mean( + torch.tensor(id2score[idx], dtype=torch.float32) + ) + elif opmd_baseline == "logavgexp": + rewards_tensor = torch.tensor(id2score[idx], dtype=torch.float32) + # here we use the fact that logavgexp(x) = logsumexp(x) - log(len(x)) + id2baseline[idx] = tau * ( + torch.logsumexp(rewards_tensor / tau, dim=-1) + - torch.log(torch.tensor(len(id2score[idx]))) + ) + else: + raise NotImplementedError + else: + raise ValueError(f"no score in prompt index: {idx}") + for i in range(bsz): + scores[i] = scores[i] - id2baseline[index[i]] + scores = scores.unsqueeze(-1).tile([1, response_length]) * eos_mask + + exps.batch["advantages"] = scores + exps.batch["returns"] = scores + + metrics = { + # TODO: add meaningful metrics + } + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "opmd_baseline": "mean", + "tau": 1.0, + } + + +class OPMDGroupAdvantage(GroupAdvantage): + """OPMD Group Advantage computation""" + + def __init__(self, opmd_baseline: str = "mean", tau: float = 1.0, **kwargs) -> None: + super().__init__(**kwargs) + self.opmd_baseline = opmd_baseline + self.tau = tau + + def group_experiences(self, exps): + return group_by(exps, id_type="task") + + def calculate_group_advantage( + self, group_id: str, exps: List[Experience] + ) -> Tuple[List[Experience], Dict]: + with torch.no_grad(): + group_rewards = torch.tensor([exp.reward for exp in exps], dtype=torch.float32) + reward_mean = torch.mean(group_rewards) + if len(exps) == 1: + group_baseline = torch.tensor(0.0) + else: + if self.opmd_baseline == "mean": + group_baseline = reward_mean + else: + group_baseline = self.tau * ( + torch.logsumexp(group_rewards / self.tau, dim=-1) + - torch.log(torch.tensor(len(exps))) + ) + for exp in exps: + score = exp.reward - group_baseline + exp.advantages = score * exp.action_mask + exp.returns = exp.advantages.clone() + metrics = { + "group_baseline": group_baseline.item(), + "reward_mean": reward_mean.item(), + } + return exps, metrics + + @classmethod + def default_args(cls) -> dict: + return {"opmd_baseline": "mean", "tau": 1.0} diff --git a/frameworks/trinity/algorithm/advantage_fn/ppo_advantage.py b/frameworks/trinity/algorithm/advantage_fn/ppo_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..99c787375ca24863a3e2646d0837703292001e01 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/ppo_advantage.py @@ -0,0 +1,85 @@ +"""PPO's GAE advantage computation + +Ref: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py +""" + +from typing import Dict, Tuple + +import torch +from verl import DataProto + +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn +from trinity.algorithm.utils import masked_whiten + + +class PPOAdvantageFn(AdvantageFn): + def __init__( + self, + gamma: float = 1.0, + lam: float = 1.0, + ) -> None: + self.gamma = gamma + self.lam = lam + + def __call__( + self, + exps: DataProto, + **kwargs, + ) -> Tuple[DataProto, Dict]: + """ + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + values: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length). [EOS] mask. The token after [EOS] have mask zero. + gamma: `(float)` + discounted factor used in RL + lam: `(float)` + lambda value when computing Generalized Advantage Estimation (https://arxiv.org/abs/1506.02438) + advantages: `(torch.Tensor)` + shape: (bs, response_length) + returns: `(torch.Tensor)` + shape: (bs, response_length) + """ + token_level_rewards = exps.batch["token_level_rewards"] + values = exps.batch["values"] + eos_mask = exps.batch["response_mask"] + gamma = self.gamma + lam = self.lam + + with torch.no_grad(): + lastgaelam = 0 + advantages_reversed = [] + gen_len = token_level_rewards.shape[-1] + + # values = values * eos_mask TODO: may use in multi-turn + for t in reversed(range(gen_len)): + nextvalues = values[:, t + 1] if t < gen_len - 1 else 0.0 + delta = token_level_rewards[:, t] + gamma * nextvalues - values[:, t] + + lastgaelam = delta + gamma * lam * lastgaelam + # lastgaelam = torch.where( # TODO: may use in multi-turn + # eos_mask[:, t] == 1, delta + gamma * lam * lastgaelam, lastgaelam + # ) + advantages_reversed.append(lastgaelam) + advantages = torch.stack(advantages_reversed[::-1], dim=1) + + returns = advantages + values + advantages = masked_whiten(advantages, eos_mask) + + exps.batch["advantages"] = advantages + exps.batch["returns"] = returns + + metrics = { + # TODO: add meaningful metrics + } + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "gamma": 1.0, + "lam": 1.0, + } diff --git a/frameworks/trinity/algorithm/advantage_fn/rec_advantage.py b/frameworks/trinity/algorithm/advantage_fn/rec_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..0a99137c00a3bcc3ea333451cb1ae32cff689b4c --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/rec_advantage.py @@ -0,0 +1,99 @@ +"""REC advantage computation +""" + +from typing import Dict, List, Optional, Tuple + +import torch + +from trinity.algorithm.advantage_fn.advantage_fn import GroupAdvantage +from trinity.common.experience import Experience, group_by + + +class RECGroupedAdvantage(GroupAdvantage): + """An advantage class that calculates REC advantages.""" + + def __init__( + self, + epsilon: float = 1e-6, + std_normalize: Optional[bool] = False, + drop: Optional[str] = None, + ) -> None: + """Initialize the REC advantage function. + + Args: + epsilon (float): A small value to avoid division by zero. + std_normalize (Optional[bool]): If provided, normalize the advantage with group-level reward standard deviation. + drop (Optional[str]): Strategy to drop experiences. Options are "balance" or None. + """ + self.epsilon = epsilon + self.std_normalize = std_normalize + self.drop = drop + assert self.drop in [None, "balance"], f"Invalid drop: {self.drop}" + + def group_experiences(self, exps): + return group_by(exps, id_type="task") + + def calculate_group_advantage( + self, group_id: str, exps: List[Experience] + ) -> Tuple[List[Experience], Dict]: + # Initialize masks and metrics + N = len(exps) + metrics = {} + with torch.no_grad(): + if len(exps) == 1: + group_reward_mean = torch.tensor(0.0) + group_reward_std = torch.tensor(1.0) # set to 1.0 to avoid division by zero + else: + rewards = torch.tensor([exp.reward for exp in exps], dtype=torch.float32) + group_reward_mean = torch.mean(rewards) + group_reward_std = torch.std(rewards) + + is_pos = rewards >= group_reward_mean + pos_count = is_pos.sum().item() + neg_count = len(exps) - pos_count + + drop_idx = torch.tensor([], dtype=torch.long) + drop_frac = 0.0 + if self.drop == "balance" and neg_count > pos_count: + extra_neg = neg_count - pos_count + neg_idx = (~is_pos).nonzero(as_tuple=True)[0] + perm = torch.randperm(len(neg_idx))[:extra_neg] + drop_idx = neg_idx[perm] + drop_frac = float(extra_neg) / float(max(N, 1)) + metrics["drop_balance"] = drop_frac + keep_mask = torch.ones(N, dtype=torch.bool) + if drop_idx.numel() > 0: + keep_mask[drop_idx] = False + + if keep_mask.sum().item() <= 1: + group_reward_mean = torch.tensor(0.0) + group_reward_std = torch.tensor(1.0) # avoid divide-by-zero + else: + sel_rewards = rewards[keep_mask] + group_reward_mean = sel_rewards.mean() + group_reward_std = sel_rewards.std(unbiased=False) + + for i, exp in enumerate(exps): + if not keep_mask[i]: + adv = torch.tensor(0.0) + else: + if getattr(self, "std_normalize", False): + adv = (rewards[i] - group_reward_mean) / (group_reward_std + self.epsilon) + else: + adv = rewards[i] - group_reward_mean + + exp.advantages = adv * exp.action_mask + exp.returns = exp.advantages.clone() + + metrics["reward_mean"] = group_reward_mean.item() + metrics["reward_std"] = group_reward_std.item() + + return exps, metrics + + @classmethod + def default_args(cls) -> dict: + return { + "epsilon": 1e-6, + "std_normalize": False, + "drop": None, + } diff --git a/frameworks/trinity/algorithm/advantage_fn/reinforce_advantage.py b/frameworks/trinity/algorithm/advantage_fn/reinforce_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..bc8c6546b40c62c456bf9e84b942039ba42ad419 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/reinforce_advantage.py @@ -0,0 +1,35 @@ +"""Reinforce advantage computation""" + +from typing import Dict, List, Tuple + +import torch + +from trinity.algorithm.advantage_fn.advantage_fn import GroupAdvantage +from trinity.common.experience import Experience, group_by + + +class REINFORCEGroupAdvantage(GroupAdvantage): + """Reinforce Group Advantage computation""" + + def group_experiences(self, exps): + return group_by(exps, id_type="task") + + def calculate_group_advantage( + self, group_id: str, exps: List[Experience] + ) -> Tuple[List[Experience], Dict]: + with torch.no_grad(): + rewards = torch.tensor([exp.reward for exp in exps], dtype=torch.float32) + group_reward_mean = torch.mean(rewards) + for exp in exps: + score = torch.tensor(exp.reward, dtype=torch.float32) + exp.advantages = score * exp.action_mask + exp.returns = exp.advantages.clone() + + metrics = { + "reward_mean": group_reward_mean.item(), + } + return exps, metrics + + @classmethod + def default_args(cls) -> dict: + return {} diff --git a/frameworks/trinity/algorithm/advantage_fn/reinforce_plus_plus_advantage.py b/frameworks/trinity/algorithm/advantage_fn/reinforce_plus_plus_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..77e735cafa35275ee3e75442053a53917ddb20a5 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/reinforce_plus_plus_advantage.py @@ -0,0 +1,65 @@ +"""REINFORCE++ advantage computation + +Ref: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py +""" + +from typing import Dict, Tuple + +import torch +from verl import DataProto + +from trinity.algorithm.advantage_fn import AdvantageFn +from trinity.algorithm.utils import masked_whiten + + +class REINFORCEPLUSPLUSAdvantageFn(AdvantageFn): + def __init__(self, gamma: float = 1.0) -> None: + self.gamma = gamma + + def __call__( + self, + exps: DataProto, + **kwargs, + ) -> Tuple[DataProto, Dict]: + """ + Compute advantage for REINFORCE++. + This implementation is based on the paper: https://arxiv.org/abs/2501.03262 + + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + advantages: `(torch.Tensor)` + shape: (bs, response_length) + returns: `(torch.Tensor)` + shape: (bs, response_length) + """ + token_level_rewards = exps.batch["token_level_rewards"] + eos_mask = exps.batch["response_mask"] + gamma = self.gamma + + with torch.no_grad(): + returns = torch.zeros_like(token_level_rewards) + running_return = 0 + + for t in reversed(range(token_level_rewards.shape[1])): + running_return = token_level_rewards[:, t] + gamma * running_return + returns[:, t] = running_return + + advantages = masked_whiten(returns, eos_mask) + advantages = advantages * eos_mask + + exps.batch["advantages"] = advantages + exps.batch["returns"] = returns + + metrics = { + # TODO: add meaningful metrics + } + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "gamma": 1.0, + } diff --git a/frameworks/trinity/algorithm/advantage_fn/remax_advantage.py b/frameworks/trinity/algorithm/advantage_fn/remax_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..d71d7b8310937b0933dddf91dce60620542026a0 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/remax_advantage.py @@ -0,0 +1,65 @@ +"""REMAX advantage computation + +Ref: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py +""" + +from typing import Dict, Tuple + +import torch +from verl import DataProto + +from trinity.algorithm.advantage_fn import AdvantageFn + + +class REMAXAdvantageFn(AdvantageFn): + def __init__(self) -> None: + pass + + def __call__( + self, + exps: DataProto, + **kwargs, + ) -> Tuple[DataProto, Dict]: + """ + Compute advantage for ReMax, operating only on Outcome reward + (with only one scalar reward for each response). + This implementation is based on the paper: https://arxiv.org/abs/2310.10505 + + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + reward_baselines: `(torch.Tensor)` + shape: (bs,) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + advantages: `(torch.Tensor)` + shape: (bs, response_length) + returns: `(torch.Tensor)` + shape: (bs, response_length) + """ + token_level_rewards = exps.batch["token_level_rewards"] + reward_baselines = exps.batch["reward_baselines"] + eos_mask = exps.batch["response_mask"] + + response_length = token_level_rewards.shape[-1] + token_level_rewards.sum(dim=-1) + + with torch.no_grad(): + returns = ( + (token_level_rewards * eos_mask).flip(dims=[-1]).cumsum(dim=-1).flip(dims=[-1]) + ) + advantages = ( + returns - reward_baselines.unsqueeze(-1).tile([1, response_length]) * eos_mask + ) + + exps.batch["advantages"] = advantages + exps.batch["returns"] = returns + + metrics = { + # TODO: add meaningful metrics + } + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return {} diff --git a/frameworks/trinity/algorithm/advantage_fn/rloo_advantage.py b/frameworks/trinity/algorithm/advantage_fn/rloo_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..71914114d10575533304419e450cdbe8c71559b1 --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/rloo_advantage.py @@ -0,0 +1,74 @@ +"""RLOO advantage computation + +Ref: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py +""" + +from collections import defaultdict +from typing import Dict, Tuple + +import torch +from verl import DataProto + +from trinity.algorithm.advantage_fn import AdvantageFn + + +class RLOOAdvantageFn(AdvantageFn): + def __init__(self) -> None: + pass + + def __call__( + self, + exps: DataProto, + **kwargs, + ) -> Tuple[DataProto, Dict]: + """ + Compute advantage for RLOO based on https://arxiv.org/abs/2402.14740 + + token_level_rewards: `(torch.Tensor)` + shape: (bs, response_length) + eos_mask: `(torch.Tensor)` + shape: (bs, response_length) + scores: `(torch.Tensor)` + shape: (bs, response_length) + """ + token_level_rewards = exps.batch["token_level_rewards"] + eos_mask = exps.batch["response_mask"] + index = exps.non_tensor_batch["uid"] + + response_length = token_level_rewards.shape[-1] + scores = token_level_rewards.sum(dim=-1) + + id2score = defaultdict(list) + id2mean = {} + + with torch.no_grad(): + bsz = scores.shape[0] + for i in range(bsz): + id2score[index[i]].append(scores[i]) + for idx in id2score: + if len(id2score[idx]) == 1: + id2mean[idx] = torch.tensor(0.0) + elif len(id2score[idx]) > 1: + id2mean[idx] = torch.mean(torch.tensor(id2score[idx], dtype=torch.float32)) + else: + raise ValueError(f"no score in prompt index: {idx}") + for i in range(bsz): + response_num = len(id2score[index[i]]) + if response_num > 1: + scores[i] = scores[i] * response_num / (response_num - 1) - id2mean[ + index[i] + ] * response_num / (response_num - 1) + scores = scores.unsqueeze(-1).tile([1, response_length]) * eos_mask + + exps.batch["advantages"] = scores + exps.batch["returns"] = scores + + metrics = { + # TODO: add meaningful metrics + } + + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return {} diff --git a/frameworks/trinity/algorithm/advantage_fn/sdpo_advantage.py b/frameworks/trinity/algorithm/advantage_fn/sdpo_advantage.py new file mode 100644 index 0000000000000000000000000000000000000000..b518d02da910e4efffb5ee1addd9a53142b6cbdc --- /dev/null +++ b/frameworks/trinity/algorithm/advantage_fn/sdpo_advantage.py @@ -0,0 +1,200 @@ +# -*- coding: utf-8 -*- +"""SDPO advantage computation. + +Supports three alpha-controlled distillation modes: +- alpha = 0.0: Forward KL (configured as in SDPO policy snippet) +- alpha = 1.0: Reverse KL +- 0 < alpha < 1: Generalized Jensen-Shannon Divergence +""" + +from typing import Dict, Tuple + +import torch +import torch.nn.functional as F +from verl import DataProto + +from trinity.algorithm.advantage_fn.advantage_fn import AdvantageFn + + +def _to_prob_and_log_prob(log_probs: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + probs = torch.exp(log_probs) + probs = probs / (probs.sum(dim=-1, keepdim=True) + 1e-12) + norm_log_probs = torch.log(probs + 1e-12) + return probs, norm_log_probs + + +def _prepare_topk_distribution(topk_log_probs: torch.Tensor, add_tail: bool) -> torch.Tensor: + probs = torch.exp(topk_log_probs) + if add_tail: + tail_prob = 1.0 - probs.sum(dim=-1, keepdim=True) + tail_prob = torch.clamp(tail_prob, min=1e-12) + probs = torch.cat([probs, tail_prob], dim=-1) + return torch.log(probs + 1e-12) + + probs = probs / (probs.sum(dim=-1, keepdim=True) + 1e-12) + return torch.log(probs + 1e-12) + + +def _build_two_class_log_distribution(token_log_probs: torch.Tensor) -> torch.Tensor: + """Build [chosen_token, tail] log distribution from token logprob.""" + chosen_prob = torch.exp(token_log_probs) + chosen_prob = torch.clamp(chosen_prob, min=1e-12, max=1.0 - 1e-12) + tail_prob = torch.clamp(1.0 - chosen_prob, min=1e-12, max=1.0) + probs = torch.stack([chosen_prob, tail_prob], dim=-1) + return torch.log(probs + 1e-12) + + +class SDPOAdvantage(AdvantageFn): + """Advantage function for SDPO. + + Args: + kl_coef: Scaling coefficient for the computed per-token advantages. + alpha: Distillation mode controller. + - 0.0: forward KL branch + - 1.0: reverse KL branch + - (0, 1): generalized JSD branch + full_logit_distillation: If True, use full distribution KL/JSD. + distillation_use_topk: If True (with full_logit_distillation), use top-k distribution. + distillation_add_tail: Whether to add tail probability in top-k mode. + """ + + def __init__( + self, + kl_coef: float = 1.0, + alpha: float = 0.5, + full_logit_distillation: bool = False, + distillation_use_topk: bool = False, + distillation_add_tail: bool = True, + ) -> None: + self.kl_coef = kl_coef + self.alpha = float(alpha) + if not (0.0 <= self.alpha <= 1.0): + raise ValueError(f"Invalid alpha: {self.alpha}. Expected alpha in [0, 1].") + self.full_logit_distillation = full_logit_distillation + self.distillation_use_topk = distillation_use_topk + self.distillation_add_tail = distillation_add_tail + + def _select_student_teacher_distill_log_probs( + self, + old_log_probs: torch.Tensor, + teacher_log_probs: torch.Tensor, + exps: DataProto, + ) -> Tuple[torch.Tensor, torch.Tensor]: + if not self.full_logit_distillation: + # Fallback for token-only pipelines: approximate each token as a 2-class distribution. + return ( + _build_two_class_log_distribution(old_log_probs), + _build_two_class_log_distribution(teacher_log_probs), + ) + + if self.distillation_use_topk: + student_topk_log_probs = exps.batch.get("student_topk_log_probs") + teacher_topk_log_probs = exps.batch.get("teacher_topk_log_probs") + if student_topk_log_probs is None or teacher_topk_log_probs is None: + raise ValueError( + "`distillation_use_topk=True` requires student_topk_log_probs and teacher_topk_log_probs in batch." + ) + return ( + _prepare_topk_distribution(student_topk_log_probs, self.distillation_add_tail), + _prepare_topk_distribution(teacher_topk_log_probs, self.distillation_add_tail), + ) + + student_all_log_probs = exps.batch.get("student_all_log_probs") + teacher_all_log_probs = exps.batch.get("teacher_all_log_probs") + if student_all_log_probs is None or teacher_all_log_probs is None: + raise ValueError( + "`full_logit_distillation=True` requires student_all_log_probs and teacher_all_log_probs in batch." + ) + _, student_distill_log_probs = _to_prob_and_log_prob(student_all_log_probs) + _, teacher_distill_log_probs = _to_prob_and_log_prob(teacher_all_log_probs) + return student_distill_log_probs, teacher_distill_log_probs + + def __call__(self, exps: DataProto, **kwargs) -> Tuple[DataProto, Dict]: + old_log_probs = exps.batch["old_log_probs"] + teacher_log_probs = exps.batch["teacher_logprobs"] + response_mask = exps.batch["response_mask"] + alpha = self.alpha + + student_distill_log_probs, teacher_distill_log_probs = ( + self._select_student_teacher_distill_log_probs(old_log_probs, teacher_log_probs, exps) + ) + + if alpha == 0.0: + # Forward KL branch (follows requested SDPO branch definition) + kl_loss = F.kl_div( + student_distill_log_probs, + teacher_distill_log_probs, + reduction="none", + log_target=True, + ).sum(dim=-1) + kl_mode = "forward_kl" + elif alpha == 1.0: + # Reverse KL branch + kl_loss = F.kl_div( + teacher_distill_log_probs, + student_distill_log_probs, + reduction="none", + log_target=True, + ).sum(dim=-1) + kl_mode = "reverse_kl" + else: + alpha_t = torch.tensor(alpha, dtype=student_distill_log_probs.dtype, device=student_distill_log_probs.device) + mixture_log_probs = torch.logsumexp( + torch.stack( + [ + student_distill_log_probs + torch.log1p(-alpha_t), + teacher_distill_log_probs + torch.log(alpha_t), + ] + ), + dim=0, + ) + kl_teacher = F.kl_div( + mixture_log_probs, + teacher_distill_log_probs, + reduction="none", + log_target=True, + ).sum(dim=-1) + kl_student = F.kl_div( + mixture_log_probs, + student_distill_log_probs, + reduction="none", + log_target=True, + ).sum(dim=-1) + kl_loss = torch.lerp(kl_student, kl_teacher, alpha_t) + kl_mode = "generalized_jsd" + + teacher_valid_mask = exps.batch.get("teacher_logprobs_valid_mask") + if teacher_valid_mask is not None: + teacher_valid_mask = teacher_valid_mask.to( + dtype=response_mask.dtype, device=response_mask.device + ) + effective_mask = response_mask * teacher_valid_mask + else: + effective_mask = response_mask + + # Convert divergence loss to maximize-style signal for policy gradient. + advantages = -self.kl_coef * kl_loss + advantages = advantages * effective_mask + + exps.batch["advantages"] = advantages + exps.batch["returns"] = advantages.clone() + + kl_sum = (kl_loss * effective_mask).sum(dim=-1) + metrics = { + "sdpo/alpha": alpha, + "sdpo/kl_mode": kl_mode, + "sdpo/kl_mean": kl_sum.mean().item(), + "sdpo/kl_std": kl_sum.std().item() if kl_sum.numel() > 1 else 0.0, + "sdpo/advantages_mean": advantages.sum(dim=-1).mean().item(), + } + return exps, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "kl_coef": 1.0, + "alpha": 0.5, + "full_logit_distillation": False, + "distillation_use_topk": False, + "distillation_add_tail": True, + } diff --git a/frameworks/trinity/algorithm/algorithm.py b/frameworks/trinity/algorithm/algorithm.py new file mode 100644 index 0000000000000000000000000000000000000000..f7a4770f3b5dd0fd6229689cc650949a301816a1 --- /dev/null +++ b/frameworks/trinity/algorithm/algorithm.py @@ -0,0 +1,581 @@ +# -*- coding: utf-8 -*- +"""Algorithm classes.""" + +from abc import ABC, ABCMeta, abstractmethod +from typing import Dict + +from trinity.common.config import Config +from trinity.common.constants import SyncMethod +from trinity.utils.log import get_logger + +logger = get_logger(__name__) + + +class ConstantMeta(ABCMeta): + def __setattr__(cls, name, value): + if name in cls.__dict__: + raise AttributeError(f"{name} is already defined in {cls.__name__}") + return super().__setattr__(name, value) + + +class AlgorithmType(ABC, metaclass=ConstantMeta): + use_critic: bool # whether to use critic model + + use_reference: bool # whether to use reference model + + compute_advantage_in_trainer: bool # whether to compute advantage in trainer + # For algorithms that rely on experience grouping, + # we recommend set this value to False + + can_balance_batch: bool # balance batch in trainer + + schema: str # schema of training data + + @classmethod + @abstractmethod + def default_config(cls) -> Dict: + raise NotImplementedError + + @classmethod + def name(cls) -> str: + return cls._name + + @classmethod + def check_config(cls, config: Config) -> None: + pass + + +class SFTAlgorithm(AlgorithmType): + """SFT Algorithm.""" + + use_critic: bool = False + use_reference: bool = False + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "sft" + + @classmethod + def default_config(cls) -> Dict: + return { + "sample_strategy": "default", + "policy_loss_fn": "sft", + "kl_loss_fn": "none", + "entropy_loss_fn": "none", + } + + @classmethod + def check_config(cls, config: Config) -> None: + if config.mode == "train": + if ( + config.buffer.trainer_input.experience_buffer is None + or not config.buffer.trainer_input.experience_buffer.path + ): + raise ValueError( + "`buffer.trainer_input.experience_buffer.path` is required when `algorithm.algorithm_type == sft`" + ) + elif config.mode in ["both", "explore"]: + raise ValueError(f"SFT does not support `{config.mode}` mode") + + if config.synchronizer.sync_method != SyncMethod.CHECKPOINT: + config.synchronizer.sync_method = SyncMethod.CHECKPOINT + logger.warning( + "SFT only supports checkpoint synchronization, set `synchronizer.sync_method` to `checkpoint`." + ) + + config.synchronizer.sync_interval = config.trainer.save_interval + + +class PPOAlgorithm(AlgorithmType): + """PPO Algorithm.""" + + use_critic: bool = True + use_reference: bool = True + compute_advantage_in_trainer: bool = True + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 1, + "sample_strategy": "default", + "policy_loss_fn": "ppo", + "advantage_fn": "ppo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class GRPOAlgorithm(AlgorithmType): + """GRPO algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "advantage_fn": "grpo", + "sample_strategy": "default", + "policy_loss_fn": "ppo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class ReinforcePlusPlusAlgorithm(AlgorithmType): + """Reinforce++ algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = True + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "advantage_fn": "reinforceplusplus", + "sample_strategy": "default", + "policy_loss_fn": "ppo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class RLOOAlgorithm(AlgorithmType): + """RLOO algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = True + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "advantage_fn": "rloo", + "sample_strategy": "default", + "policy_loss_fn": "ppo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class OPMDAlgorithm(AlgorithmType): + """OPMD algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "advantage_fn": "opmd", + "sample_strategy": "default", + "policy_loss_fn": "opmd", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class AsymREAlgorithm(AlgorithmType): + """AsymRE algorithm.""" + + use_critic: bool = False + use_reference: bool = False + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "sample_strategy": "default", + "policy_loss_fn": "opmd", + "advantage_fn": "asymre", + "kl_penalty_fn": "none", + "kl_loss_fn": "none", + "entropy_loss_fn": "none", + } + + +class DPOAlgorithm(AlgorithmType): + """DPO algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = False + schema: str = "dpo" + + @classmethod + def default_config(cls) -> Dict: + return { + "sample_strategy": "default", + "policy_loss_fn": "dpo", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + @classmethod + def check_config(cls, config: Config) -> None: + if config.mode == "train": + if ( + config.buffer.trainer_input.experience_buffer is None + or not config.buffer.trainer_input.experience_buffer.path + ): + raise ValueError( + "`buffer.trainer_input.experience_buffer.path` is required when `algorithm.algorithm_type == dpo`" + ) + elif config.mode in ["both", "explore"]: + raise ValueError(f"DPO does not support `{config.mode}` mode") + + if config.synchronizer.sync_method != SyncMethod.CHECKPOINT: + config.synchronizer.sync_method = SyncMethod.CHECKPOINT + logger.warning( + "DPO only supports checkpoint synchronization, set `synchronizer.sync_method` to `checkpoint`." + ) + config.synchronizer.sync_interval = config.trainer.save_interval + if config.algorithm.repeat_times != 2: + config.algorithm.repeat_times = 2 # Fake repeat times + if config.algorithm.kl_loss_fn in {"none", None}: + config.algorithm.kl_loss_fn = "k2" + logger.warning("DPO must use KL loss. Set `algorithm.kl_loss_fn` to `k2`") + + +class TOPRAlgorithm(AlgorithmType): + """TOPR algorithm. See https://arxiv.org/pdf/2503.14286v1""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "advantage_fn": "reinforce", # or simply use grpo + "sample_strategy": "default", + "policy_loss_fn": "topr", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class CISPOAlgorithm(AlgorithmType): + """CISPO algorithm. See https://arxiv.org/abs/2506.13585""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "advantage_fn": "grpo", + "sample_strategy": "default", + "policy_loss_fn": "cispo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class GSPOAlgorithm(AlgorithmType): + """GSPO algorithm. See https://arxiv.org/pdf/2507.18071""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "advantage_fn": "grpo", + "sample_strategy": "default", + "policy_loss_fn": "gspo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class SAPOAlgorithm(AlgorithmType): + """SAPO (Soft Adaptive Policy Optimization) algorithm. + + SAPO uses a smooth, temperature-controlled soft gate instead of hard clipping + to stabilize training while maintaining effective learning. + """ + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "advantage_fn": "grpo", + "sample_strategy": "default", + "policy_loss_fn": "sapo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class MIXAlgorithm(AlgorithmType): + """MIX algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + use_rollout: bool = True + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 8, + "advantage_fn": "grpo", + "policy_loss_fn": "mix", + "sample_strategy": "mix", + "entropy_loss_fn": "mix", + } + + +class MIXCHORDAlgorithm(AlgorithmType): + """MIX algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + use_rollout: bool = True + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 8, + "policy_loss_fn": "mix_chord", + "advantage_fn": "grpo", + "sample_strategy": "mix", + "entropy_loss_fn": "mix", + } + + +class RAFTAlgorithm(AlgorithmType): + """RAFT Algorithm. + This algorithm is conceptually similar to Supervised Fine-Tuning (SFT) + but is designed to work with `experience` schema from rollouts. + """ + + use_critic: bool = False + use_reference: bool = False + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "sample_strategy": "default", + "policy_loss_fn": "sft", + "kl_loss_fn": "none", + "entropy_loss_fn": "none", + } + + +class sPPOAlgorithm(AlgorithmType): + """sPPO Algorithm.""" + + use_critic: bool = False + use_reference: bool = False + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "sample_strategy": "default", + "policy_loss_fn": "sppo", + "advantage_fn": "opmd", + "kl_penalty_fn": "none", + "kl_loss_fn": "none", + "entropy_loss_fn": "none", + } + + +class RECAlgorithm(AlgorithmType): + """REC Algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 2, + "sample_strategy": "default", + "policy_loss_fn": "rec", + "advantage_fn": "rec", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class MultiStepGRPOAlgorithm(AlgorithmType): + """Multi-Step GRPO Algorithm.""" + + use_critic: bool = False + use_reference: bool = True + compute_advantage_in_trainer: bool = False + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 8, + "advantage_fn": "step_wise_grpo", + "sample_strategy": "default", + "policy_loss_fn": "ppo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + } + + +class OnPolicyDistillAlgorithm(AlgorithmType): + """On-Policy Distillation Algorithm. + + Reference: Tinker library. + + Workflow stores teacher_logprobs in experience.info["teacher_logprobs"]. + Trainer's advantage_fn computes: advantages = teacher_logprobs - student_logprobs + Trainer uses: + importance_sampling loss if no clipping is needed + ppo loss if clipping is needed, for better stability + """ + + use_critic: bool = False + use_reference: bool = False + compute_advantage_in_trainer: bool = True # advantage_fn computes from teacher_logprobs + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 1, + "advantage_fn": "multi_turn_opd", + "advantage_fn_args": {"kl_coef": 1.0}, + "sample_strategy": "default", + "policy_loss_fn": "ppo", + "kl_penalty_fn": "none", + "kl_loss_fn": "none", + "entropy_loss_fn": "none", + } + + +class JSDAlgorithm(AlgorithmType): + """JSD (Jensen-Shannon Divergence) Algorithm. + + Uses JSD between teacher and student for distillation. + Same structure as On-Policy Distill but with JSD advantage function. + """ + + use_critic: bool = False + use_reference: bool = False + compute_advantage_in_trainer: bool = True # advantage_fn computes JSD from teacher_logprobs + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 8, + "advantage_fn": "jsd", + "advantage_fn_args": {"kl_coef": 1.0, "lambda_coef": 0.5}, + "sample_strategy": "default", + "policy_loss_fn": "ppo", + "kl_penalty_fn": "none", + "kl_loss_fn": "none", + "entropy_loss_fn": "none", + } + + +class SDPOAlgorithm(AlgorithmType): + """SDPO (Self-Distilled Policy Optimization) algorithm. + + SDPO computes teacher-student distillation signal in trainer-side advantage_fn, + then optimizes with policy loss (PPO/IS style update). + """ + + use_critic: bool = False + use_reference: bool = False + compute_advantage_in_trainer: bool = True + can_balance_batch: bool = True + schema: str = "experience" + + @classmethod + def default_config(cls) -> Dict: + return { + "repeat_times": 8, + "sample_strategy": "default", + "advantage_fn": "sdpo", + "advantage_fn_args": { + "kl_coef": 1.0, + "alpha": 0.5, + "full_logit_distillation": False, + "distillation_use_topk": False, + "distillation_add_tail": True, + }, + "policy_loss_fn": "sdpo", + "policy_loss_fn_args": { + "is_clip": None, + "loss_agg_mode": "token-mean", + }, + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "none", + } + + diff --git a/frameworks/trinity/algorithm/entropy_loss_fn/__init__.py b/frameworks/trinity/algorithm/entropy_loss_fn/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0d3e1c9735efe22b6548c94f3183b3d723001e15 --- /dev/null +++ b/frameworks/trinity/algorithm/entropy_loss_fn/__init__.py @@ -0,0 +1,16 @@ +from trinity.algorithm.entropy_loss_fn.entropy_loss_fn import EntropyLossFn +from trinity.utils.registry import Registry + +ENTROPY_LOSS_FN: Registry = Registry( + "entropy_loss_fn", + default_mapping={ + "default": "trinity.algorithm.entropy_loss_fn.entropy_loss_fn.DefaultEntropyLossFn", + "mix": "trinity.algorithm.entropy_loss_fn.entropy_loss_fn.MixEntropyLossFn", + "none": "trinity.algorithm.entropy_loss_fn.entropy_loss_fn.DummyEntropyLossFn", + }, +) + +__all__ = [ + "EntropyLossFn", + "ENTROPY_LOSS_FN", +] diff --git a/frameworks/trinity/algorithm/entropy_loss_fn/entropy_loss_fn.py b/frameworks/trinity/algorithm/entropy_loss_fn/entropy_loss_fn.py new file mode 100644 index 0000000000000000000000000000000000000000..75069c3739816e16f15070c6f1bd451be21c2a76 --- /dev/null +++ b/frameworks/trinity/algorithm/entropy_loss_fn/entropy_loss_fn.py @@ -0,0 +1,100 @@ +from abc import ABC, abstractmethod +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.utils import aggregate_loss + + +class EntropyLossFn(ABC): + """ + Entropy loss function. + """ + + @abstractmethod + def __call__( + self, + entropy: torch.Tensor, + action_mask: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + """ + Args: + entropy (`torch.Tensor`): The entropy generated by the policy model. + action_mask (`torch.Tensor`): The action mask. + + Returns: + `torch.Tensor`: The calculated entropy loss. + `Dict`: The metrics for logging + """ + + @classmethod + def default_args(cls) -> Dict: + """ + Returns: + `Dict`: The default arguments for the entropy loss function. + """ + return {"entropy_coef": 0.0} + + +class DefaultEntropyLossFn(EntropyLossFn): + """ + Basic entropy loss function. + """ + + def __init__(self, entropy_coef: float): + self.entropy_coef = entropy_coef + + def __call__( + self, + entropy: torch.Tensor, + action_mask: torch.Tensor, + loss_agg_mode: str = "token-mean", + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + entropy_loss = aggregate_loss(entropy, action_mask, loss_agg_mode=loss_agg_mode) + return entropy_loss * self.entropy_coef, {"entropy_loss": entropy_loss.detach().item()} + + +class MixEntropyLossFn(EntropyLossFn): + """ + Basic entropy loss function for mix algorithm. + """ + + def __init__(self, entropy_coef: float): + self.entropy_coef = entropy_coef + + def __call__( + self, + entropy: torch.Tensor, + action_mask: torch.Tensor, + expert_mask: torch.Tensor = None, + loss_agg_mode: str = "token-mean", + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + if expert_mask is None: + raise ValueError("expert_mask is required for MixEntropyLossFn") + assert ( + len(expert_mask) == entropy.shape[0] + ), f"Error: {len(expert_mask)=} != {entropy.shape[0]=}" + entropy = entropy[~expert_mask] + action_mask = action_mask[~expert_mask] + entropy_loss = aggregate_loss(entropy, action_mask, loss_agg_mode=loss_agg_mode) + return entropy_loss * self.entropy_coef, {"entropy_loss": entropy_loss.detach().item()} + + +class DummyEntropyLossFn(EntropyLossFn): + """ + Dummy entropy loss function. + """ + + def __init__(self, entropy_coef: float): + self.entropy_coef = entropy_coef + + def __call__( + self, + entropy: torch.Tensor, + action_mask: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + return torch.tensor(0.0), {} diff --git a/frameworks/trinity/algorithm/key_mapper.py b/frameworks/trinity/algorithm/key_mapper.py new file mode 100644 index 0000000000000000000000000000000000000000..4813c18fb5e273e639f9e1da67d90f08b489d839 --- /dev/null +++ b/frameworks/trinity/algorithm/key_mapper.py @@ -0,0 +1,30 @@ +# -*- coding: utf-8 -*- +"""Key Mapper""" + +from typing import Dict + + +class KeyMapper: + def __init__(self, to_trinity_map: Dict[str, str]): + self.to_trinity_map = to_trinity_map + self.from_trinity_map = {v: k for k, v in self.to_trinity_map.items()} + + def to_trinity(self, key: str) -> str: + return self.to_trinity_map.get(key, key) + + def from_trinity(self, key: str) -> str: + return self.from_trinity_map.get(key, key) + + +ALL_MAPPERS = { + "verl": KeyMapper( + { + "log_prob": "logprob", + "old_log_probs": "old_logprob", + "ref_log_prob": "ref_logprob", + "response_mask": "action_mask", + "advantages": "advantages", + } + ), + "tinker": KeyMapper({}), +} diff --git a/frameworks/trinity/algorithm/kl_fn/__init__.py b/frameworks/trinity/algorithm/kl_fn/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..2426f07991f8f1502db8f8b3bb88e53753f15b69 --- /dev/null +++ b/frameworks/trinity/algorithm/kl_fn/__init__.py @@ -0,0 +1,17 @@ +from trinity.algorithm.kl_fn.kl_fn import KLFn +from trinity.utils.registry import Registry + +KL_FN: Registry = Registry( + "kl_fn", + default_mapping={ + "none": "trinity.algorithm.kl_fn.kl_fn.DummyKLFn", + "k1": "trinity.algorithm.kl_fn.kl_fn.K1Fn", + "k2": "trinity.algorithm.kl_fn.kl_fn.K2Fn", + "k3": "trinity.algorithm.kl_fn.kl_fn.K3Fn", + "low_var_kl": "trinity.algorithm.kl_fn.kl_fn.LowVarKLFn", + "abs": "trinity.algorithm.kl_fn.kl_fn.AbsFn", + "corrected_k3": "trinity.algorithm.kl_fn.kl_fn.CorrectedK3Fn", + }, +) + +__all__ = ["KLFn", "KL_FN"] diff --git a/frameworks/trinity/algorithm/kl_fn/kl_fn.py b/frameworks/trinity/algorithm/kl_fn/kl_fn.py new file mode 100644 index 0000000000000000000000000000000000000000..312f5303293127d84600a224563a0b07bed5da76 --- /dev/null +++ b/frameworks/trinity/algorithm/kl_fn/kl_fn.py @@ -0,0 +1,278 @@ +"""KL penalty and loss. + +Ref: +https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py +https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py +https://github.com/OpenRLHF/OpenRLHF/blob/main/openrlhf/models/utils.py +""" + +from abc import ABC, abstractmethod +from typing import Any, Dict, Optional, Tuple + +import torch + +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class KLFn(ABC): + """ + KL penalty and loss. + """ + + def __init__( + self, + adaptive: bool = False, + kl_coef: float = 0.001, + target_kl: Optional[float] = None, + horizon: Optional[float] = None, + ) -> None: + self.kl_coef = kl_coef + self.adaptive = adaptive + self.target_kl = target_kl + self.horizon = horizon + if adaptive and (target_kl is None or horizon is None): + raise ValueError("Target KL and horizon must be provided for adaptive KL.") + + def update_kl_coef(self, current_kl: float, batch_size: int) -> None: + """Update kl coefficient.""" + if self.adaptive: + target_kl = self.target_kl + proportional_error = torch.clip(current_kl / target_kl - 1, -0.2, 0.2).item() # type: ignore + multiplier = 1 + proportional_error * batch_size / self.horizon + self.kl_coef *= multiplier + + def apply_kl_penalty_to_reward(self, experiences: Any) -> Tuple[Any, Dict]: + """Apply KL penalty to reward. Only support DataProto input for now.""" + responses = experiences.batch["responses"] + response_length = responses.size(1) + token_level_scores = experiences.batch["token_level_scores"] + batch_size = experiences.batch.batch_size[0] + attention_mask = experiences.batch["attention_mask"] + response_mask = experiences.batch["response_mask"] + assert response_mask.shape == attention_mask[:, -response_length:].shape + logprob = experiences.batch["old_log_probs"] + ref_logprob = experiences.batch["ref_log_prob"] + + if "ref_log_prob" in experiences.batch.keys(): + kl = self.calculate_kl(logprob, ref_logprob) + kl = kl * response_mask + kl_coef = self.kl_coef + experiences.batch["token_level_rewards"] = token_level_scores - kl_coef * kl + else: + kl_coef = 0.0 + kl = torch.zeros_like(response_mask, dtype=torch.float32) + experiences.batch["token_level_rewards"] = token_level_scores + + current_kl = masked_mean(kl, mask=response_mask, axis=-1).mean(dim=0).item() + self.update_kl_coef(current_kl=current_kl, batch_size=batch_size) + + metrics = { + "kl": current_kl, + "kl_coef": kl_coef, + } + + return experiences, metrics + + def calculate_kl_loss( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + response_mask: torch.Tensor, + loss_agg_mode: str, + old_logprob: Optional[torch.Tensor] = None, + ) -> Tuple[torch.Tensor, Dict]: + """Compute KL loss. + + Args: + logprob: Log probabilities from current policy + ref_logprob: Log probabilities from reference policy + response_mask: Mask for valid response tokens + loss_agg_mode: Loss aggregation mode + old_logprob: Log probabilities from old policy (for importance sampling) + """ + kl = self.calculate_kl(logprob, ref_logprob, old_logprob) + kl_loss = aggregate_loss(kl, response_mask, loss_agg_mode=loss_agg_mode) + metrics = { + "kl_loss": kl_loss.detach().item(), + "kl_coef": self.kl_coef, + } + return kl_loss * self.kl_coef, metrics + + @abstractmethod + def calculate_kl( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + old_logprob: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """Compute KL divergence between logprob and ref_logprob. + + Args: + logprob: Log probabilities from current policy + ref_logprob: Log probabilities from reference policy + old_logprob: Log probabilities from old policy (for importance sampling) + """ + + @classmethod + def default_args(cls): + """Get the default initialization arguments.""" + return {"adaptive": False, "kl_coef": 0.001} + + +class DummyKLFn(KLFn): + """ + Dummy KL function. + """ + + def calculate_kl( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + old_logprob: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return torch.zeros_like(logprob) + + def apply_kl_penalty_to_reward(self, experiences: Any) -> Tuple[Any, Dict]: + experiences.batch["token_level_rewards"] = experiences.batch["token_level_scores"] + return experiences, {} + + def calculate_kl_loss( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + response_mask: torch.Tensor, + loss_agg_mode: str, + old_logprob: Optional[torch.Tensor] = None, + ) -> Tuple[torch.Tensor, Dict]: + # return a zero tensor + return torch.tensor(0.0), {} + + +class K1Fn(KLFn): + """ + KL K1 function. + """ + + def calculate_kl( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + old_logprob: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return logprob - ref_logprob + + +class K2Fn(KLFn): + """ + KL K2 function. + """ + + def calculate_kl( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + old_logprob: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return (logprob - ref_logprob).square() * 0.5 + + +class K3Fn(KLFn): + """ + KL K3 function. + """ + + def calculate_kl( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + old_logprob: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + logr = ref_logprob - logprob + return logr.exp() - 1 - logr + + +class LowVarKLFn(KLFn): + """ + Low Variance KL function. + """ + + def calculate_kl( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + old_logprob: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + kl = ref_logprob - logprob + kl = torch.clamp(kl, min=-20, max=20) + ratio = torch.exp(kl) + kld = (ratio - kl - 1).contiguous() + return torch.clamp(kld, min=-10, max=10) + + +class AbsFn(KLFn): + """ + KL Abs function. + """ + + def calculate_kl( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + old_logprob: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return torch.abs(logprob - ref_logprob) + + +class CorrectedK3Fn(KLFn): + """ + Corrected K3 function with importance sampling. + + This method applies importance sampling correction to the standard K3 KL divergence. + The corrected KL is computed as: + + KL_corrected = (π_θ / π_old) * KL_standard(π_ref || π_θ) + + where: + - π_θ: current policy + - π_old: old policy (from rollout) + - π_ref: reference policy + - KL_standard: exp(log(π_ref/π_θ)) - log(π_ref/π_θ) - 1 + + If old_logprob is not provided, it falls back to standard K3. + """ + + def calculate_kl( + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + old_logprob: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """Compute corrected K3 KL divergence with importance sampling. + + Args: + logprob: Log probabilities from current policy (log π_θ) + ref_logprob: Log probabilities from reference policy (log π_ref) + old_logprob: Log probabilities from old policy (log π_old), optional + + Returns: + KL divergence tensor with same shape as input + """ + # Standard K3 KL term: exp(log_ratio) - log_ratio - 1 + # where log_ratio = log(π_ref / π_θ) = ref_logprob - logprob + logr = ref_logprob - logprob + kl_term = logr.exp() - 1 - logr + + if old_logprob is None: + # Fall back to standard K3 if old_logprob is not provided + return kl_term + + # Compute importance sampling ratio: π_θ / π_old + log_ratio_is = logprob - old_logprob + ratio_is = log_ratio_is.exp() + # Clamp ratio for numerical stability, range [0, 2] + ratio_is = torch.clamp(ratio_is, min=0.0, max=2.0) + + # Corrected KL with importance sampling + corrected_kl = ratio_is * kl_term + + return corrected_kl diff --git a/frameworks/trinity/algorithm/policy_loss_fn/__init__.py b/frameworks/trinity/algorithm/policy_loss_fn/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..90fba16b66c58a1d4d81e7d803f771bf04efc9ff --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/__init__.py @@ -0,0 +1,29 @@ +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.utils.registry import Registry + +POLICY_LOSS_FN: Registry = Registry( + "policy_loss_fn", + default_mapping={ + "ppo": "trinity.algorithm.policy_loss_fn.ppo_policy_loss.PPOPolicyLossFn", + "opmd": "trinity.algorithm.policy_loss_fn.opmd_policy_loss.OPMDPolicyLossFn", + "dpo": "trinity.algorithm.policy_loss_fn.dpo_loss.DPOLossFn", + "sft": "trinity.algorithm.policy_loss_fn.sft_loss.SFTLossFn", + "mix": "trinity.algorithm.policy_loss_fn.mix_policy_loss.MIXPolicyLossFn", + "gspo": "trinity.algorithm.policy_loss_fn.gspo_policy_loss.GSPOLossFn", + "topr": "trinity.algorithm.policy_loss_fn.topr_policy_loss.TOPRPolicyLossFn", + "cispo": "trinity.algorithm.policy_loss_fn.cispo_policy_loss.CISPOPolicyLossFn", + "sft_is": "trinity.algorithm.policy_loss_fn.chord_policy_loss.SFTISLossFn", + "sft_phi": "trinity.algorithm.policy_loss_fn.chord_policy_loss.SFTPhiLossFn", + "mix_chord": "trinity.algorithm.policy_loss_fn.chord_policy_loss.MIXCHORDPolicyLossFn", + "sppo": "trinity.algorithm.policy_loss_fn.sppo_loss_fn.sPPOPolicyLossFn", + "rec": "trinity.algorithm.policy_loss_fn.rec_policy_loss.RECPolicyLossFn", + "sapo": "trinity.algorithm.policy_loss_fn.sapo_policy_loss.SAPOPolicyLossFn", + "importance_sampling": "trinity.algorithm.policy_loss_fn.importance_sampling_policy_loss.ImportanceSamplingLossFn", + "sdpo": "trinity.algorithm.policy_loss_fn.sdpo_policy_loss.SDPOPolicyLossFn", + }, +) + +__all__ = [ + "POLICY_LOSS_FN", + "PolicyLossFn", +] diff --git a/frameworks/trinity/algorithm/policy_loss_fn/chord_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/chord_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..6653b4d22588873f3aed8e3cbced6caf3ef08f17 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/chord_policy_loss.py @@ -0,0 +1,257 @@ +"""Implements the CHORD policy loss function.""" + +import math +from typing import Dict, Optional, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.policy_loss_fn.ppo_policy_loss import PPOPolicyLossFn +from trinity.algorithm.policy_loss_fn.sft_loss import SFTLossFn +from trinity.algorithm.utils import aggregate_loss + + +def mu_schedule_function( + global_step: int, mu_warmup_steps: int, mu_decay_steps: int, mu_peak: float, mu_valley: float +) -> float: + """ + Computes a cosine decay schedule with a warmup phase for the mu parameter. + """ + # Warmup + if global_step < mu_warmup_steps: + return (global_step / mu_warmup_steps) * mu_peak + + # Decay + if global_step >= (mu_warmup_steps + mu_decay_steps): + return mu_valley + + adjusted_step = global_step - mu_warmup_steps + cosine_decay = 0.5 * (1 + math.cos(math.pi * adjusted_step / mu_decay_steps)) + decayed_mu = (mu_peak - mu_valley) * cosine_decay + mu_valley + return decayed_mu + + +class SFTISLossFn(PolicyLossFn): + """ + SFT loss with importance sampling + """ + + def __init__(self, backend: str = "verl", loss_agg_mode: str = "token-mean") -> None: + super().__init__(backend=backend) + self.loss_agg_mode = loss_agg_mode + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + action_mask: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + token_prob = torch.exp(logprob) + sft_loss = aggregate_loss( + -logprob * token_prob.detach(), action_mask, loss_agg_mode=self.loss_agg_mode + ) + return sft_loss, {"sft_is_loss": sft_loss.detach().item()} + + @classmethod + def default_args(cls): + return { + "loss_agg_mode": "token-mean", + } + + +def phi_function(token_prob): + """ + The phi function downweights token with extreme probability. + Feel free to modify this function. + """ + return token_prob * (1 - token_prob) + + +class SFTPhiLossFn(PolicyLossFn): + """ + SFT loss with transformed phi function + """ + + def __init__( + self, backend: str = "verl", loss_agg_mode: str = "token-mean", cutoff_prob: float = 1.0 + ) -> None: + super().__init__(backend=backend) + self.loss_agg_mode = loss_agg_mode + self.cutoff_prob = cutoff_prob + assert 0.0 <= self.cutoff_prob <= 1.0 + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + action_mask: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + token_prob = torch.exp(logprob) + if self.cutoff_prob < 1.0: + logprob = torch.clamp(logprob, max=math.log(self.cutoff_prob)) + + weighted_phi = phi_function(token_prob) + + sft_loss = aggregate_loss( + -logprob * weighted_phi.detach(), action_mask, loss_agg_mode=self.loss_agg_mode + ) + return sft_loss, {"sft_phi_loss": sft_loss.detach().item()} + + @classmethod + def default_args(cls): + return { + "loss_agg_mode": "token-mean", + "cutoff_prob": 1.0, + } + + +class MIXCHORDPolicyLossFn(PolicyLossFn): + """Implements a mixed policy loss combining GRPO and SFT losses. + + This loss function applies different loss components to data based on whether + it comes from an expert or not, as indicated by `expert_mask`. It combines: + + - GRPO loss (self.grpo_loss_fn) for non-expert data + - SFT loss (self.sft_loss_fn) for expert data + the weight of SFT loss is globally controled by `mu_schedule` function + the tokenwise weights are calculated using different SFT loss formulas + + The per-sample weights are normalized using either `experience_per_gpu` or + `gradient_accumulation`, depending on whether dynamic batch sizing is enabled, + to ensure consistent weighting across different batches of the same type experiences. + """ + + def __init__( + self, + backend: str = "verl", + mu_warmup_steps: int = 0, + mu_decay_steps: int = 0, + mu_peak: float = 0.1, + mu_valley: float = 0.1, + enable_phi_function: bool = True, + clip_range: Optional[float] = None, + clip_range_low: Optional[float] = None, + clip_range_high: Optional[float] = None, + use_dynamic_bsz: Optional[bool] = None, + ppo_mini_batch_size: int = 1, + ppo_micro_batch_size_per_gpu: int = 1, + ngpus_trainer: int = 1, + train_batch_size_usual: int = 1, + train_batch_size_expert: int = 1, + loss_agg_mode: str = "token-mean", + sft_loss_agg_mode: Optional[str] = None, + grpo_loss_agg_mode: Optional[str] = None, + ) -> None: + super().__init__(backend=backend) + self.mu_warmup_steps = mu_warmup_steps + self.mu_decay_steps = mu_decay_steps + self.mu_peak = mu_peak + self.mu_valley = mu_valley + self.enable_phi_function = enable_phi_function + self.use_dynamic_bsz = use_dynamic_bsz + self.experience_per_gpu = ppo_mini_batch_size // ngpus_trainer + self.gradient_accumulation = ppo_mini_batch_size // ppo_micro_batch_size_per_gpu + self.train_batch_size_usual = train_batch_size_usual // ngpus_trainer + self.train_batch_size_expert = train_batch_size_expert // ngpus_trainer + self.grpo_loss_fn = PPOPolicyLossFn( + clip_range=clip_range, + clip_range_low=clip_range_low, + clip_range_high=clip_range_high, + loss_agg_mode=grpo_loss_agg_mode or loss_agg_mode, + ) + if enable_phi_function: + self.sft_loss_fn = SFTPhiLossFn(loss_agg_mode=sft_loss_agg_mode or loss_agg_mode) + else: + self.sft_loss_fn = SFTLossFn(loss_agg_mode=sft_loss_agg_mode or loss_agg_mode) + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + old_logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, + expert_mask: torch.Tensor, + step: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + assert ( + len(expert_mask) == logprob.shape[0] + ), f"Error: {len(expert_mask)=} != {logprob.shape[0]=}" + + assert len(step) == logprob.shape[0], f"Error: {len(step)=} != {logprob.shape[0]=}" + + assert ( + step.max().item() == step.min().item() + ), f"Error: {step.max().item()} != {step.min().item()}" + current_step = step.max().item() + + n_usual_exp = torch.sum(~expert_mask).item() + n_expert_exp = torch.sum(expert_mask).item() + + if self.use_dynamic_bsz: + per_micro_batch_weight_usual = self.experience_per_gpu / ( + logprob.shape[0] * self.train_batch_size_usual + ) + per_micro_batch_weight_expert = self.experience_per_gpu / ( + logprob.shape[0] * self.train_batch_size_expert + ) + else: + per_micro_batch_weight_usual = self.gradient_accumulation / self.train_batch_size_usual # type: ignore + per_micro_batch_weight_expert = self.gradient_accumulation / self.train_batch_size_expert # type: ignore + + if n_usual_exp > 0: + grpo_loss, grpo_metrics = self.grpo_loss_fn( + logprob[~expert_mask], + old_logprob[~expert_mask], + action_mask[~expert_mask], + advantages[~expert_mask], + **kwargs, + ) + grpo_loss = grpo_loss * n_usual_exp * per_micro_batch_weight_usual + grpo_metrics = { + k: v * n_usual_exp * per_micro_batch_weight_usual for k, v in grpo_metrics.items() + } + else: + grpo_loss = torch.tensor(0.0, device=logprob.device) + grpo_metrics = {} + + # SFT Loss (expert) + if n_expert_exp > 0: + sft_loss, sft_metrics = self.sft_loss_fn( + logprob[expert_mask], + action_mask[expert_mask], + ) + sft_loss = sft_loss * n_expert_exp * per_micro_batch_weight_expert + sft_metrics = { + k: v * n_expert_exp * per_micro_batch_weight_expert for k, v in sft_metrics.items() + } + else: + sft_loss = torch.tensor(0.0, device=logprob.device) + sft_metrics = {} + + mu = mu_schedule_function( + current_step, self.mu_warmup_steps, self.mu_decay_steps, self.mu_peak, self.mu_valley + ) + + loss = (1 - mu) * grpo_loss + mu * sft_loss + + metrics = {f"usual/{k}": v for k, v in grpo_metrics.items()} + metrics.update({f"expert/{k}": v for k, v in sft_metrics.items()}) + metrics.update({"loss": loss.item(), "mu": mu}) + + return loss, metrics + + @classmethod + def default_args(cls) -> Dict: + """ + mu_warmup_steps: int, mu_decay_steps: int, mu_peak: float, mu_valley: float + """ + return { + "mu_warmup_steps": 0, + "mu_decay_steps": 0, + "mu_peak": 0.1, + "mu_valley": 0.1, + "clip_range": 0.2, + "enable_phi_function": True, + "loss_agg_mode": "token-mean", + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/cispo_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/cispo_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..45d4b2277b31b6eadd32f42c32d8ac46d29c21c0 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/cispo_policy_loss.py @@ -0,0 +1,90 @@ +"""CISPO policy loss function. +Refer to https://arxiv.org/abs/2506.13585 for details. +""" + +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class CISPOPolicyLossFn(PolicyLossFn): + def __init__( + self, + backend: str = "verl", + clip_range_low: float = 1.0, + clip_range_high: float = 0.28, + enable_mask_clip: bool = False, + mask_clip_range_low: float = 1.0, + mask_clip_range_high: float = 0.28, + loss_agg_mode: str = "token-mean", + ) -> None: + super().__init__(backend=backend) + self.clip_range_low = clip_range_low + self.clip_range_high = clip_range_high + self.enable_mask_clip = enable_mask_clip + self.mask_clip_range_low = mask_clip_range_low + self.mask_clip_range_high = mask_clip_range_high + self.loss_agg_mode = loss_agg_mode + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + old_logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + negative_approx_kl = logprob - old_logprob + ratio = torch.exp(negative_approx_kl) + ppo_kl = masked_mean(-negative_approx_kl, action_mask) + ratio_clamped = torch.clamp( + ratio, min=1.0 - self.clip_range_low, max=1.0 + self.clip_range_high + ) + + # mask = 0 if ratio > 1.0 + self.clip_range_high and advantages > 0 + # mask = 0 if ratio < 1.0 - self.clip_range_low and advantages < 0 + # else 1 + mask = torch.ones_like(ratio) + if self.enable_mask_clip: + mask = torch.where( + (ratio > 1.0 + self.mask_clip_range_high) & (advantages > 0), + torch.zeros_like(ratio), + mask, + ) + mask = torch.where( + (ratio < 1.0 - self.mask_clip_range_low) & (advantages < 0), + torch.zeros_like(ratio), + mask, + ) + + cispo_loss = -advantages * ratio_clamped.detach() * mask.detach() * logprob + + loss = aggregate_loss(cispo_loss, action_mask, loss_agg_mode=self.loss_agg_mode) + unmasked_frac = masked_mean(mask, action_mask) + + metrics = { + "cispo_loss": loss.detach().item(), + "ppo_kl": ppo_kl.detach().item(), + "unmasked_frac": unmasked_frac.detach().item(), + } + + return loss, metrics + + @classmethod + def default_args(cls) -> Dict: + """ + In the original paper: + we did not impose a lower bound on the IS weight by setting clip_range_low to a high value, instead, we only tuned clip_range_high + + """ + return { + "clip_range_low": 1.0, + "clip_range_high": 0.28, + "enable_mask_clip": False, + "mask_clip_range_low": 1.0, + "mask_clip_range_high": 0.28, + "loss_agg_mode": "token-mean", + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/dpo_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/dpo_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..d1b3a2122d3de6aef44874df3b498e928f1c7be5 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/dpo_loss.py @@ -0,0 +1,66 @@ +"""DPO loss function.""" + +from typing import Dict, Tuple + +import torch +import torch.nn.functional as F + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import masked_sum + + +class DPOLossFn(PolicyLossFn): + def __init__( + self, + backend: str = "verl", + beta: float = 0.1, + label_smoothing: float = 0.0, + ) -> None: + super().__init__(backend=backend) + self.beta = beta + self.label_smoothing = label_smoothing + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + ref_logprob: torch.Tensor, + action_mask: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + chosen_logprob = logprob[::2] + rejected_logprob = logprob[1::2] + chosen_mask = action_mask[::2] + rejected_mask = action_mask[1::2] + chosen_logprob_sum = masked_sum(chosen_logprob, chosen_mask) + rejected_logprob_sum = masked_sum(rejected_logprob, rejected_mask) + + chosen_ref_logprob = ref_logprob[::2] + rejected_ref_logprob = ref_logprob[1::2] + chosen_ref_logprob_sum = masked_sum(chosen_ref_logprob, chosen_mask) + rejected_ref_logprob_sum = masked_sum(rejected_ref_logprob, rejected_mask) + + chosen_ratios = chosen_logprob_sum - chosen_ref_logprob_sum + rejected_ratios = rejected_logprob_sum - rejected_ref_logprob_sum + logits = chosen_ratios - rejected_ratios + # TODO: support other loss functions + losses = ( + -F.logsigmoid(self.beta * logits) * (1 - self.label_smoothing) + - F.logsigmoid(-self.beta * logits) * self.label_smoothing + ) + loss = losses.mean() + chosen_reward = self.beta * chosen_ratios.detach().mean().item() + rejected_reward = self.beta * rejected_ratios.detach().mean().item() + accuracy_mean = (chosen_ratios.detach() > rejected_ratios.detach()).float().mean().item() + return loss, { + "chosen_reward": chosen_reward, + "rejected_reward": rejected_reward, + "accuracy_mean": accuracy_mean, + "dpo_loss": loss.detach().item(), + } + + @classmethod + def default_args(cls) -> Dict: + return { + "beta": 0.1, + "label_smoothing": 0.0, + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/gspo_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/gspo_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..7527603e03b6192df0914877a0a6baca4cba2b46 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/gspo_policy_loss.py @@ -0,0 +1,88 @@ +"""GSPO-token policy loss function. + +Implemented from https://arxiv.org/pdf/2507.18071 +""" + +from typing import Dict, Optional, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class GSPOLossFn(PolicyLossFn): + def __init__( + self, + backend: str = "verl", + clip_range: Optional[float] = None, + clip_range_low: Optional[float] = None, + clip_range_high: Optional[float] = None, + loss_agg_mode: Optional[str] = "seq-mean-token-mean", + ) -> None: + super().__init__(backend=backend) + _clip_range_low = clip_range_low if clip_range_low is not None else clip_range + if _clip_range_low is None: + raise ValueError("Either clip_range or clip_range_low must be specified.") + self.clip_range_low = _clip_range_low + + _clip_range_high = clip_range_high if clip_range_high is not None else clip_range + if _clip_range_high is None: + raise ValueError("Either clip_range or clip_range_high must be specified.") + self.clip_range_high = _clip_range_high + + if loss_agg_mode != "seq-mean-token-mean": + from trinity.utils.log import get_logger + + logger = get_logger(__name__) + logger.warning( + f"The original GSPO paper requires loss_agg_mode to be 'seq-mean-token-mean', but the current setting is '{loss_agg_mode}'." + ) + # loss_agg_mode = "seq-mean-token-mean" + self.loss_agg_mode = loss_agg_mode + + def __call__( # type: ignore + self, + logprob: torch.Tensor, # [batch_size, seq_len] + old_logprob: torch.Tensor, # [batch_size, seq_len] + action_mask: torch.Tensor, # [batch_size, seq_len] + advantages: torch.Tensor, # [batch_size, seq_len] + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + negative_approx_kl = logprob - old_logprob # [batch_size, seq_len] + negative_approx_kl_seq = masked_mean( + negative_approx_kl, action_mask, axis=-1 + ) # [batch_size] + log_seq_importance_ratio = ( + logprob - logprob.detach() + negative_approx_kl_seq.detach().unsqueeze(-1) + ) # [batch_size, seq_len] + ratio = torch.exp(log_seq_importance_ratio) # [batch_size, seq_len] + pg_losses = -advantages * ratio # [batch_size, seq_len] + pg_losses_clipped = -advantages * torch.clamp( + ratio, 1.0 - self.clip_range_low, 1.0 + self.clip_range_high + ) # [batch_size, seq_len] + + pg_loss = aggregate_loss( + values=torch.max(pg_losses, pg_losses_clipped), + mask=action_mask, + loss_agg_mode=self.loss_agg_mode, + ) + pg_clipfrac = masked_mean(torch.gt(pg_losses_clipped, pg_losses).float(), action_mask) + ppo_kl = masked_mean(-negative_approx_kl, action_mask) + ppo_kl_seq = torch.mean(-negative_approx_kl_seq) + metrics = { + "pg_clipfrac": pg_clipfrac.detach().item(), + "ppo_kl": ppo_kl.detach().item(), + "pg_loss": pg_loss.detach().item(), + "ppo_kl_seq": ppo_kl_seq.detach().item(), + } + return pg_loss, metrics + + @classmethod + def default_args(cls) -> Dict: + # See discussion in https://github.com/volcengine/verl/pull/2775#issuecomment-3130065984 + return { + "clip_range_low": 0.0003, + "clip_range_high": 0.0004, + "loss_agg_mode": "seq-mean-token-mean", + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/importance_sampling_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/importance_sampling_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..4f273b3780a1c560b3efa5513d3bc4c62be0a4a3 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/importance_sampling_policy_loss.py @@ -0,0 +1,60 @@ +# -*- coding: utf-8 -*- +"""The most simple Importance Sampling policy loss. + +loss = -(prob_ratio * advantages).sum() +where prob_ratio = exp(current_logprobs - sampling_logprobs) + +Note: This loss is used for on-policy distillation. +""" + +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class ImportanceSamplingLossFn(PolicyLossFn): + """Pure importance sampling loss without clipping. + + loss = -(ratio * advantages) + where ratio = exp(logprob - old_logprob) + """ + + def __init__( + self, + backend: str = "verl", + loss_agg_mode: str = "token-mean", + ) -> None: + super().__init__(backend=backend) + self.loss_agg_mode = loss_agg_mode + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + old_logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + # prob_ratio = exp(current_logprobs - sampling_logprobs) + log_ratio = logprob - old_logprob + log_ratio = torch.clamp(log_ratio, min=-20.0, max=20.0) + ratio = torch.exp(log_ratio) + + # loss = -(prob_ratio * advantages) + pg_losses = -advantages * ratio + pg_loss = aggregate_loss(pg_losses, action_mask, loss_agg_mode=self.loss_agg_mode) + + metrics = { + "pg_loss": pg_loss.detach().item(), + "ratio/mean": masked_mean(ratio, action_mask).detach().item(), + "approx_kl": masked_mean(-log_ratio, action_mask).detach().item(), + } + + return pg_loss, metrics + + @classmethod + def default_args(cls) -> Dict: + return {"loss_agg_mode": "token-mean"} diff --git a/frameworks/trinity/algorithm/policy_loss_fn/mix_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/mix_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..dc5fcc65b5134d5509e7d4f6741601a3017aaed0 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/mix_policy_loss.py @@ -0,0 +1,129 @@ +"""Mix policy loss function.""" + +from typing import Dict, Optional, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.policy_loss_fn.ppo_policy_loss import PPOPolicyLossFn +from trinity.algorithm.policy_loss_fn.sft_loss import SFTLossFn + + +class MIXPolicyLossFn(PolicyLossFn): + """Implements a mixed policy loss combining GRPO and SFT losses. + + This loss function applies different loss components to data based on whether + it comes from an expert or not, as indicated by `expert_mask`. It combines: + - GRPO loss (self.grpo_loss_fn) for non-expert data + - SFT loss (self.sft_loss_fn) for expert data + - Weighting parameter `mu` + + The per-sample weights are normalized using either `experience_per_gpu` or + `gradient_accumulation`, depending on whether dynamic batch sizing is enabled, + to ensure consistent weighting across different batches of the same type experiences. + """ + + def __init__( + self, + backend: str = "verl", + mu: float = 0.1, + clip_range: Optional[float] = None, + clip_range_low: Optional[float] = None, + clip_range_high: Optional[float] = None, + use_dynamic_bsz: Optional[bool] = None, + ppo_mini_batch_size: int = 1, + ppo_micro_batch_size_per_gpu: int = 1, + ngpus_trainer: int = 1, + train_batch_size_usual: int = 1, + train_batch_size_expert: int = 1, + loss_agg_mode: str = "token-mean", + sft_loss_agg_mode: Optional[str] = None, + grpo_loss_agg_mode: Optional[str] = None, + ) -> None: + super().__init__(backend=backend) + self.mu = mu + self.use_dynamic_bsz = use_dynamic_bsz + self.experience_per_gpu = ppo_mini_batch_size // ngpus_trainer + self.gradient_accumulation = ppo_mini_batch_size // ppo_micro_batch_size_per_gpu + self.train_batch_size_usual = train_batch_size_usual // ngpus_trainer + self.train_batch_size_expert = train_batch_size_expert // ngpus_trainer + self.grpo_loss_fn = PPOPolicyLossFn( + clip_range=clip_range, + clip_range_low=clip_range_low, + clip_range_high=clip_range_high, + loss_agg_mode=grpo_loss_agg_mode or loss_agg_mode, + ) + self.sft_loss_fn = SFTLossFn(loss_agg_mode=sft_loss_agg_mode or loss_agg_mode) + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + old_logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, + expert_mask: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + assert ( + len(expert_mask) == logprob.shape[0] + ), f"Error: {len(expert_mask)=} != {logprob.shape[0]=}" + + n_usual_exp = torch.sum(~expert_mask).item() + n_expert_exp = torch.sum(expert_mask).item() + + if self.use_dynamic_bsz: + per_micro_batch_weight_usual = self.experience_per_gpu / ( + logprob.shape[0] * self.train_batch_size_usual + ) + per_micro_batch_weight_expert = self.experience_per_gpu / ( + logprob.shape[0] * self.train_batch_size_expert + ) + else: + per_micro_batch_weight_usual = self.gradient_accumulation / self.train_batch_size_usual # type: ignore + per_micro_batch_weight_expert = self.gradient_accumulation / self.train_batch_size_expert # type: ignore + + if n_usual_exp > 0: + grpo_loss, grpo_metrics = self.grpo_loss_fn( + logprob[~expert_mask], + old_logprob[~expert_mask], + action_mask[~expert_mask], + advantages[~expert_mask], + **kwargs, + ) + grpo_loss = grpo_loss * n_usual_exp * per_micro_batch_weight_usual + grpo_metrics = { + k: v * n_usual_exp * per_micro_batch_weight_usual for k, v in grpo_metrics.items() + } + else: + grpo_loss = torch.tensor(0.0, device=logprob.device) + grpo_metrics = {} + + # SFT Loss (expert) + if n_expert_exp > 0: + sft_loss, sft_metrics = self.sft_loss_fn( + logprob[expert_mask], + action_mask[expert_mask], + ) + sft_loss = sft_loss * n_expert_exp * per_micro_batch_weight_expert + sft_metrics = { + k: v * n_expert_exp * per_micro_batch_weight_expert for k, v in sft_metrics.items() + } + else: + sft_loss = torch.tensor(0.0, device=logprob.device) + sft_metrics = {} + + loss = (1 - self.mu) * grpo_loss + self.mu * sft_loss + + metrics = {f"usual/{k}": v for k, v in grpo_metrics.items()} + metrics.update({f"expert/{k}": v for k, v in sft_metrics.items()}) + metrics["loss"] = loss.item() + + return loss, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "mu": 0.1, + "clip_range": 0.2, + "loss_agg_mode": "token-mean", + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/opmd_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/opmd_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..528e2e4ecfc07eff1babe989cb0a386e7753638a --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/opmd_policy_loss.py @@ -0,0 +1,33 @@ +"""OPMD policy loss function.""" + +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss + + +class OPMDPolicyLossFn(PolicyLossFn): + def __init__( + self, backend: str = "verl", tau: float = 1.0, loss_agg_mode: str = "token-mean" + ) -> None: + super().__init__(backend=backend) + self.tau = tau + self.loss_agg_mode = loss_agg_mode + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + pg_losses = -advantages * logprob + opmd_loss = aggregate_loss(pg_losses, action_mask, loss_agg_mode=self.loss_agg_mode) + opmd_loss = opmd_loss / (1.0 + self.tau) # for regularization (w.r.t. current pi_theta) + return opmd_loss, {"opmd_loss": opmd_loss.detach().item()} + + @classmethod + def default_args(cls) -> Dict: + return {"tau": 1.0, "loss_agg_mode": "token-mean"} diff --git a/frameworks/trinity/algorithm/policy_loss_fn/policy_loss_fn.py b/frameworks/trinity/algorithm/policy_loss_fn/policy_loss_fn.py new file mode 100644 index 0000000000000000000000000000000000000000..8cf73d5dbce0e939c205db372b3eafa404dd33b8 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/policy_loss_fn.py @@ -0,0 +1,118 @@ +import inspect +from abc import ABC, ABCMeta, abstractmethod +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.key_mapper import ALL_MAPPERS + + +class PolicyLossFnMeta(ABCMeta): + """Metaclass for policy loss functions that handles parameter name mapping and filtering.""" + + ignore_keys = {"self", "kwargs", "logprob"} # Keys to exclude from parameter selection + + def __new__(cls, name, bases, dct): + """ + Metaclass constructor that automatically generates parameter handling logic. + + For example with `PPOPolicyLossFn` class: + .. code-block:: python + class PPOPolicyLossFn(PolicyLossFn): + ... + def __call__( + self, + logprob: torch.Tensor, + old_logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + ... + + This metaclass analyzes the __call__ method's parameters to: + 1. Generate _select_keys containing all non-ignored parameters + 2. Create select_keys property that maps parameters to trainer-specific names + 3. Apply decorator to automatically convert input parameter names using the mapper + """ + signature = inspect.signature(dct["__call__"]) + param_names = [ + key for key in signature.parameters.keys() if key not in PolicyLossFnMeta.ignore_keys + ] + dct["_select_keys"] = param_names + + # Property to return trainer-specific parameter names + def select_keys(self): + """Returns parameter keys mapped to the specific training framework's naming convention.""" + keys = [self.mapper.from_trinity(key) for key in self._select_keys] + return keys + + # Decorator to handle parameter name conversion before calling __call__ + def decorator(func): + def wrapper(self, *args, **kwargs): + """Filters and converts parameter names according to the training framework's convention.""" + new_kwargs = {} + for key, value in kwargs.items(): + key = self.mapper.to_trinity(key) + if key == "logprob" or key in self._select_keys: # remove unused keys + new_kwargs[key] = value + return func(self, *args, **new_kwargs) + + return wrapper + + # Add the property and decorated method to the class + dct["select_keys"] = property(select_keys) + dct["__call__"] = decorator(dct["__call__"]) + return super().__new__(cls, name, bases, dct) + + +class PolicyLossFn(ABC, metaclass=PolicyLossFnMeta): + """ + Abstract base class for policy loss functions. + + This class provides the interface for implementing different policy gradient loss functions + while handling parameter name mapping between different training frameworks. + """ + + def __init__(self, backend: str = "verl"): + """ + Initialize the policy loss function. + + Args: + backend: The training framework/backend to use (e.g., "verl") + """ + self.backend = backend + self.mapper = ALL_MAPPERS[self.backend] + + @abstractmethod + def __call__( + self, + logprob: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + """ + Calculate the policy loss. + + Args: + logprob (`torch.Tensor`): The log probability generated by the policy model. + + Kwargs (optional): + old_logprob (`torch.Tensor`): The log probability generated by the reference model. + action_mask (`torch.Tensor`): The action mask. + advantages (`torch.Tensor`): The advantages. + kwargs (`Dict`): The step-level parameters for calculating the policy loss. + + Returns: + `torch.Tensor`: Policy loss + `Dict`: The metrics for logging. + """ + + @classmethod + @abstractmethod + def default_args(cls) -> Dict: + """ + Get default initialization arguments for this loss function. + + Returns: + `Dict`: The default init arguments for the policy loss function. + """ diff --git a/frameworks/trinity/algorithm/policy_loss_fn/ppo_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/ppo_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..7cc708827cd218ee5dc18af51171cf26ead6cc31 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/ppo_policy_loss.py @@ -0,0 +1,127 @@ +"""PPO policy loss function. + +Modified from https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py +""" + +from typing import Dict, Optional, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class PPOPolicyLossFn(PolicyLossFn): + def __init__( + self, + backend: str = "verl", + clip_range: Optional[float] = None, + clip_range_low: Optional[float] = None, + clip_range_high: Optional[float] = None, + clip_ratio_c: float = 3.0, + loss_agg_mode: Optional[str] = "token-mean", + enable_sequence_masking: bool = False, # introduced in DeepseekV3.2 + delta_sequence_masking: float = 0.1, + fallback_to_policy_gradient: bool = False, + ) -> None: + super().__init__(backend=backend) + if clip_range_low is None: + self.clip_range_low = clip_range + else: + self.clip_range_low = clip_range_low + if clip_range_high is None: + self.clip_range_high = clip_range + else: + self.clip_range_high = clip_range_high + self.clip_ratio_c = clip_ratio_c + assert clip_ratio_c > 1.0, "clip_ratio_c must be greater than 1.0." + assert self.clip_range_low is not None, "clip_range_low must be specified." + assert self.clip_range_high is not None, "clip_range_high must be specified." + self.loss_agg_mode = loss_agg_mode + self.enable_sequence_masking = enable_sequence_masking + self.delta_sequence_masking = delta_sequence_masking + self.fallback_to_policy_gradient = fallback_to_policy_gradient + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + old_logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + negative_approx_kl = logprob - old_logprob + if self.fallback_to_policy_gradient: + # ignore vllm logprob difference and use pure policy gradient loss + negative_approx_kl = logprob - logprob.detach() + # Clamp negative_approx_kl for stability + negative_approx_kl = torch.clamp(negative_approx_kl, min=-20.0, max=20.0) + ratio = torch.exp(negative_approx_kl) + ppo_kl = masked_mean(-negative_approx_kl, action_mask) + + pg_losses1 = -advantages * ratio + pg_losses2 = -advantages * torch.clamp( + ratio, 1.0 - self.clip_range_low, 1.0 + self.clip_range_high # type: ignore + ) + + clip_pg_losses1 = torch.maximum(pg_losses1, pg_losses2) + + pg_clip_frac = masked_mean(torch.gt(pg_losses2, pg_losses1).float(), action_mask) + + pg_losses3 = -advantages * self.clip_ratio_c + clip_pg_losses2 = torch.min(pg_losses3, clip_pg_losses1) + pg_clipfrac_lower = masked_mean( + torch.gt(clip_pg_losses1, pg_losses3) * (advantages < 0).float(), action_mask + ) + pg_losses = torch.where(advantages < 0, clip_pg_losses2, clip_pg_losses1) + + # Apply sequence masking if enabled + if self.enable_sequence_masking: + # Compute sequence-level KL divergence: mean KL per sequence + # Shape: (batch_size, seq_len) -> (batch_size,) + kl_per_token = -negative_approx_kl # KL divergence per token + sequence_kl = (kl_per_token * action_mask).sum(dim=-1) / ( + action_mask.sum(dim=-1) + 1e-10 + ) + + # Create mask: mask out tokens with negative advantages when sequence KL is high + # Token-level advantage check: (batch_size, seq_len) + has_negative_advantage = advantages < 0 + # Sequence-level KL check: (batch_size,) -> (batch_size, 1) -> (batch_size, seq_len) + exceeds_kl_threshold = ( + (sequence_kl > self.delta_sequence_masking).unsqueeze(-1).expand_as(advantages) + ) + # Mask tokens that are both negative advantage AND in high-KL sequences + should_mask = has_negative_advantage & exceeds_kl_threshold + sequence_mask = (~should_mask).float() + + # Apply sequence mask to the losses + pg_losses = pg_losses * sequence_mask + + metrics_seq_mask = { + "seq_mask/masked_tokens": should_mask.float().sum().item() + / (action_mask.sum().item() + 1e-10), + "seq_mask/mean_sequence_kl": sequence_kl.mean().detach().item(), + } + + pg_loss = aggregate_loss(pg_losses, action_mask, loss_agg_mode=self.loss_agg_mode) + metrics = { + "pg_clipfrac": pg_clip_frac.detach().item(), + "pg_clipfrac_lower": pg_clipfrac_lower.detach().item(), + "ppo_kl": ppo_kl.detach().item(), + "pg_loss": pg_loss.detach().item(), + } + if self.enable_sequence_masking: + metrics.update(metrics_seq_mask) + return pg_loss, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "clip_range": 0.2, + "clip_ratio_c": 3.0, + "loss_agg_mode": "token-mean", + "enable_sequence_masking": False, + "delta_sequence_masking": 0.1, + "fallback_to_policy_gradient": False, + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/rec_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/rec_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..6f00ac86c17b13de6cd7b0d7dfcac3596af6bb73 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/rec_policy_loss.py @@ -0,0 +1,153 @@ +"""REC-token policy loss function. +""" + +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class RECPolicyLossFn(PolicyLossFn): + def __init__( + self, + backend: str = "verl", + epsilon_low: float = 0.2, + epsilon_high: float = 0.2, + epsilon_low_prime: float = 0.4, + epsilon_high_prime: float = 0.4, + clip_mode: str = "none", + weight: str = "none", + regularizer: str = "none", + regularizer_coef: float = 0.0, + temp: float = 1.0, + ) -> None: + super().__init__(backend=backend) + + self.epsilon_low = epsilon_low + self.epsilon_high = epsilon_high + assert 0.0 < self.epsilon_low <= 1.0, f"Invalid epsilon_low: {self.epsilon_low}" + assert 0.0 < self.epsilon_high, f"Invalid epsilon_high: {self.epsilon_high}" + self.epsilon_low_prime = epsilon_low_prime + self.epsilon_high_prime = epsilon_high_prime + assert ( + 0.0 < self.epsilon_low_prime <= 1.0 + ), f"Invalid epsilon_low_prime: {self.epsilon_low_prime}" + assert ( + 0.0 < self.epsilon_high_prime + ), f"Invalid epsilon_high_prime: {self.epsilon_high_prime}" + self.clip_mode = clip_mode + assert self.clip_mode in [ + "none", + "one-side", + "gspo-one-side", + "two-side", + "ring", + ], f"Invalid clip_mode: {self.clip_mode}" + self.weight = weight + assert self.weight in [ + "none", + "importance_sampling", + "gspo_importance_sampling", + "advantage", + ], f"Invalid weight: {self.weight}" + + self.regularizer = regularizer + assert self.regularizer in [ + "none", + "k2", + "forward-kl", + ], f"Invalid regularizer: {self.regularizer}" + self.regularizer_coef = regularizer_coef + assert self.regularizer_coef >= 0.0, f"Invalid regularizer_coef: {self.regularizer_coef}" + self.temp = temp + assert self.temp > 0.0, f"Invalid temp: {self.temp}" + + def __call__( # type: ignore + self, + logprob: torch.Tensor, # [batch_size, seq_len] + old_logprob: torch.Tensor, # [batch_size, seq_len] + action_mask: torch.Tensor, # [batch_size, seq_len] + advantages: torch.Tensor, # [batch_size, seq_len] + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + """Calculate REC loss.""" + + ratio = torch.exp(logprob - old_logprob).detach() # token-wise prob ratio + + # clipping + if self.clip_mode == "two-side": + is_in_range = (ratio >= (1 - self.epsilon_low)) * (ratio <= (1 + self.epsilon_high)) + elif self.clip_mode == "one-side": + is_in_range = (ratio <= (1 + self.epsilon_high)) * (advantages >= 0) + ( + advantages <= 0 + ) * (ratio >= (1 - self.epsilon_low)) + elif self.clip_mode == "gspo-one-side": + mean_log_prob_diff = masked_mean( + logprob - old_logprob, action_mask, axis=-1 + ).detach() # [batch_size] + normalized_seq_ratio = torch.exp(mean_log_prob_diff).unsqueeze(-1) # [batch_size, 1] + is_in_range = (normalized_seq_ratio <= (1 + self.epsilon_high)) * (advantages >= 0) + ( + normalized_seq_ratio >= (1 - self.epsilon_low) + ) * ( + advantages <= 0 + ) # [batch_size, seq_len] + elif self.clip_mode == "ring": + is_in_range = ( + (ratio >= (1 - self.epsilon_low)) * (ratio <= (1 + self.epsilon_high)) + + (advantages >= 0) * (ratio <= 1 - self.epsilon_low_prime) + + (advantages <= 0) * (ratio >= 1 + self.epsilon_high_prime) + ) + else: # none + is_in_range = torch.ones_like(ratio).bool() + is_clipped_mask = ~is_in_range + + if self.weight == "importance_sampling": + advantages = advantages * ratio # importance sampling + elif self.weight == "gspo_importance_sampling": + advantages = advantages * normalized_seq_ratio + elif self.weight == "advantage": + weight = torch.exp(advantages / self.temp) + advantages = advantages * weight # advantage weighting (unnormalized version) + + pg_losses = -advantages * logprob * is_in_range.float() + + if self.regularizer == "forward-kl": + regularizer_losses = self.regularizer_coef * logprob + pg_losses = pg_losses - regularizer_losses + elif self.regularizer == "k2": + # note that here we absorb the 1/2 in Kimi into \tau + regularizer_losses = self.regularizer_coef * (logprob - old_logprob).square() + pg_losses = pg_losses + regularizer_losses + + if self.clip_mode == "gspo-one-side": + # [EXPERIMENTAL] specialized for gspo-style rec variant for now + pg_loss = aggregate_loss( + values=pg_losses, + mask=action_mask, + loss_agg_mode="seq-mean-token-mean", + ) + else: + pg_loss = masked_mean(pg_losses, action_mask) + + pg_clipfrac = masked_mean(is_clipped_mask.float(), action_mask) + metrics = { + "pg_clipfrac": pg_clipfrac.item(), + "pg_loss": pg_loss.detach().item(), + } + return pg_loss, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "epsilon_low": 0.2, + "epsilon_high": 0.2, + "epsilon_low_prime": 0.6, + "epsilon_high_prime": 2, + "clip_mode": "none", + "weight": "none", + "regularizer": "none", + "regularizer_coef": 0.0, + "temp": 1.0, + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/sapo_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/sapo_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..285a24e3385414cd0089d353299b4632a5a1bd3a --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/sapo_policy_loss.py @@ -0,0 +1,158 @@ +"""SAPO policy loss function. +Soft Adaptive Policy Optimization (SAPO) is a reinforcement learning algorithm +that uses a smooth, temperature-controlled soft gate instead of hard clipping. + +Refer to the SAPO paper for details. https://arxiv.org/abs/2511.20347 +""" + +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class SAPOPolicyLossFn(PolicyLossFn): + def __init__( + self, + backend: str = "verl", + tau_pos: float = 1.0, + tau_neg: float = 1.05, + loss_agg_mode: str = "token-mean", + ) -> None: + """Initialize SAPO policy loss function. + + Args: + backend: The training framework/backend to use (e.g., "verl") + tau_pos: Temperature for positive advantages (τ_pos), default 1.0 + tau_neg: Temperature for negative advantages (τ_neg), default 1.05, should be >= tau_pos + loss_agg_mode: Mode for aggregating loss across tokens + """ + super().__init__(backend=backend) + self.tau_pos = tau_pos + self.tau_neg = tau_neg + self.loss_agg_mode = loss_agg_mode + + # Validate that tau_neg > tau_pos for stability + assert self.tau_neg >= self.tau_pos, ( + f"tau_neg ({self.tau_neg}) should be >= tau_pos ({self.tau_pos}) " + "for better training stability" + ) + + def soft_gate_function(self, ratio: torch.Tensor, advantages: torch.Tensor) -> torch.Tensor: + """Compute the soft gate function f_{i,t}(x). + + The soft gate function is defined as: + f_{i,t}(x) = σ(τ_{i,t} * (x - 1)) * 4 / τ_{i,t} + + where: + - σ is the sigmoid function + - τ_{i,t} is the asymmetric temperature (tau_pos or tau_neg) + - x is the importance sampling ratio r_{i,t}(θ) + + Args: + ratio: Token-level importance sampling ratio r_{i,t}(θ) + advantages: Normalized advantage function Â_i (same for all tokens in a sequence) + + Returns: + The soft gate values for each token + """ + # Select temperature based on advantage sign + # tau_i,t = tau_pos if A_i > 0, else tau_neg + tau = torch.where( + advantages > 0, + torch.tensor(self.tau_pos, device=ratio.device, dtype=ratio.dtype), + torch.tensor(self.tau_neg, device=ratio.device, dtype=ratio.dtype), + ) + + # Compute sigmoid(tau * (ratio - 1)) + sigmoid_input = tau * (ratio - 1) + sigmoid_output = torch.sigmoid(sigmoid_input) + + # Compute the soft gate: sigma(tau * (x - 1)) * 4 / tau + soft_gate = sigmoid_output * (4.0 / tau) + + return soft_gate + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + old_logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + """Compute SAPO policy loss. + + The SAPO objective function is: + J(θ) = E[1/G Σ 1/|y_i| Σ f_{i,t}(r_{i,t}(θ)) * Â_i] + + We minimize the negative of this objective. + + Args: + logprob: Log probabilities from current policy π_θ + old_logprob: Log probabilities from old policy π_{θ_old} + action_mask: Mask indicating valid tokens + advantages: Group-normalized advantage function + + Returns: + loss: The computed policy loss (negative of objective) + metrics: Dictionary of metrics for logging + """ + # Compute token-level importance sampling ratio + # r_{i,t}(θ) = π_θ(y_{i,t}|q, y_{i, 0).float(), action_mask) + + metrics = { + "sapo_loss": loss.detach().item(), + "ppo_kl": ppo_kl.detach().item(), + "avg_soft_gate": avg_soft_gate.detach().item(), + "avg_ratio": avg_ratio.detach().item(), + "pos_adv_frac": pos_adv_frac.detach().item(), + } + + return loss, metrics + + @classmethod + def default_args(cls) -> Dict: + """Get default initialization arguments for SAPO. + + Default configuration (from the SAPO paper): + - tau_pos: 1.0 (temperature for positive advantages) + - tau_neg: 1.05 (temperature for negative advantages) + - loss_agg_mode: "token-mean" (average over tokens) + + The asymmetric temperatures (tau_neg > tau_pos) help stabilize training + by more aggressively suppressing updates from tokens with negative advantages. + + Returns: + Dictionary of default arguments + """ + return { + "tau_pos": 1.0, + "tau_neg": 1.05, + "loss_agg_mode": "token-mean", + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/sdpo_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/sdpo_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..3a3228d2fc0c81384b33d71fb6356ffcb50cd917 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/sdpo_policy_loss.py @@ -0,0 +1,234 @@ +# -*- coding: utf-8 -*- +"""SDPO policy loss implementation.""" + +from typing import Any, Dict, Optional, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +def _to_prob_and_log_prob(log_probs: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + probs = torch.exp(log_probs) + probs = probs / (probs.sum(dim=-1, keepdim=True) + 1e-12) + norm_log_probs = torch.log(probs + 1e-12) + return probs, norm_log_probs + + +def _prepare_topk_distribution( + topk_log_probs: torch.Tensor, + add_tail: bool, +) -> torch.Tensor: + probs = torch.exp(topk_log_probs) + if add_tail: + tail_prob = 1.0 - probs.sum(dim=-1, keepdim=True) + tail_prob = torch.clamp(tail_prob, min=1e-12) + probs = torch.cat([probs, tail_prob], dim=-1) + return torch.log(probs + 1e-12) + + probs = probs / (probs.sum(dim=-1, keepdim=True) + 1e-12) + return torch.log(probs + 1e-12) + + +def compute_self_distillation_loss( + student_log_probs: torch.Tensor, + teacher_log_probs: torch.Tensor, + response_mask: torch.Tensor, + self_distillation_config: Any, + old_log_probs: Optional[torch.Tensor] = None, + student_all_log_probs: Optional[torch.Tensor] = None, + teacher_all_log_probs: Optional[torch.Tensor] = None, + student_topk_log_probs: Optional[torch.Tensor] = None, + teacher_topk_log_probs: Optional[torch.Tensor] = None, + self_distillation_mask: Optional[torch.Tensor] = None, + loss_agg_mode: str = "token-mean", + rollout_is_weights: Optional[torch.Tensor] = None, +) -> Tuple[torch.Tensor, Dict[str, float]]: + """Compute SDPO loss from config-described branches.""" + cfg = self_distillation_config or {} + alpha = float(cfg.get("alpha", 0.5)) + full_logit_distillation = bool(cfg.get("full_logit_distillation", True)) + use_topk = bool(cfg.get("distillation_use_topk", False)) + add_tail = bool(cfg.get("distillation_add_tail", True)) + is_clip = cfg.get("is_clip", None) + + response_mask = response_mask.to(dtype=student_log_probs.dtype) + + if self_distillation_mask is not None: + seq_mask = self_distillation_mask.to(dtype=student_log_probs.dtype).unsqueeze(-1) + loss_mask = response_mask * seq_mask + else: + loss_mask = response_mask + + if full_logit_distillation: + if use_topk: + if student_topk_log_probs is None or teacher_topk_log_probs is None: + raise ValueError( + "`distillation_use_topk=True` requires student_topk_log_probs and teacher_topk_log_probs." + ) + student_distill_log_probs = _prepare_topk_distribution(student_topk_log_probs, add_tail) + teacher_distill_log_probs = _prepare_topk_distribution(teacher_topk_log_probs, add_tail) + else: + if student_all_log_probs is None or teacher_all_log_probs is None: + raise ValueError( + "`full_logit_distillation=True` requires student_all_log_probs and teacher_all_log_probs." + ) + student_distill_log_probs = student_all_log_probs + teacher_distill_log_probs = teacher_all_log_probs + + student_probs, student_distill_log_probs = _to_prob_and_log_prob(student_distill_log_probs) + teacher_probs, teacher_distill_log_probs = _to_prob_and_log_prob(teacher_distill_log_probs) + + if alpha <= 0.0: + # KL(Student || Teacher) + per_token_loss = ( + student_probs * (student_distill_log_probs - teacher_distill_log_probs) + ).sum(dim=-1) + elif alpha >= 1.0: + # KL(Teacher || Student) + per_token_loss = ( + teacher_probs * (teacher_distill_log_probs - student_distill_log_probs) + ).sum(dim=-1) + else: + # Generalized JSD: + # M = (1-alpha) * Student + alpha * Teacher + mixture_probs = (1.0 - alpha) * student_probs + alpha * teacher_probs + mixture_log_probs = torch.log(mixture_probs + 1e-12) + kl_teacher = (mixture_probs * (mixture_log_probs - teacher_distill_log_probs)).sum(dim=-1) + kl_student = (mixture_probs * (mixture_log_probs - student_distill_log_probs)).sum(dim=-1) + per_token_loss = (1.0 - alpha) * kl_student + alpha * kl_teacher + else: + # Token-level distillation + log_ratio = student_log_probs - teacher_log_probs + per_token_loss = -(log_ratio.detach()) * student_log_probs + + if is_clip is not None and old_log_probs is not None: + ratio = torch.exp(student_log_probs - old_log_probs) + ratio = torch.clamp(ratio, min=0.0, max=float(is_clip)) + per_token_loss = per_token_loss * ratio + + if rollout_is_weights is not None: + per_token_loss = per_token_loss * rollout_is_weights + + if loss_mask.sum() <= 0: + loss = student_log_probs.new_tensor(0.0) + metrics = { + "self_distill_loss": 0.0, + "empty_target_batch": 1.0, + } + return loss, metrics + + loss = aggregate_loss(per_token_loss, loss_mask, loss_agg_mode=loss_agg_mode) + metrics = { + "self_distill_loss": loss.detach().item(), + "empty_target_batch": 0.0, + "self_distill_alpha": float(alpha), + "sdpo_alpha": float(alpha), + "self_distill_valid_tokens": float(loss_mask.sum().detach().item()), + "self_distill_per_token_mean": masked_mean(per_token_loss, loss_mask).detach().item(), + } + return loss, metrics + + +class SDPOPolicyLossFn(PolicyLossFn): + """SDPO policy loss. + + Preferred path: consume trainer-computed ``advantages`` (from SDPOAdvantage) + and optimize with an IS-style objective. + + Backward-compatible path: if ``advantages`` is absent, falls back to the + legacy inline self-distillation loss. + """ + + def __init__( + self, + backend: str = "verl", + alpha: float = 0.5, + full_logit_distillation: bool = True, + distillation_use_topk: bool = False, + distillation_add_tail: bool = True, + is_clip: Optional[float] = None, + loss_agg_mode: str = "token-mean", + ) -> None: + super().__init__(backend=backend) + self.loss_agg_mode = loss_agg_mode + self.self_distillation_config = { + "alpha": alpha, + "full_logit_distillation": full_logit_distillation, + "distillation_use_topk": distillation_use_topk, + "distillation_add_tail": distillation_add_tail, + "is_clip": is_clip, + } + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: Optional[torch.Tensor] = None, + teacher_logprobs: Optional[torch.Tensor] = None, + old_logprob: Optional[torch.Tensor] = None, + student_all_log_probs: Optional[torch.Tensor] = None, + teacher_all_log_probs: Optional[torch.Tensor] = None, + student_topk_log_probs: Optional[torch.Tensor] = None, + teacher_topk_log_probs: Optional[torch.Tensor] = None, + self_distillation_mask: Optional[torch.Tensor] = None, + rollout_is_weights: Optional[torch.Tensor] = None, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + # Preferred path: policy-gradient update over trainer-side SDPO advantages. + if advantages is not None and old_logprob is not None: + log_ratio = torch.clamp(logprob - old_logprob, min=-20.0, max=20.0) + ratio = torch.exp(log_ratio) + is_clip = self.self_distillation_config.get("is_clip", None) + if is_clip is not None: + ratio = torch.clamp(ratio, min=0.0, max=float(is_clip)) + if rollout_is_weights is not None: + ratio = ratio * rollout_is_weights + + per_token_loss = -advantages * ratio + loss = aggregate_loss(per_token_loss, action_mask, loss_agg_mode=self.loss_agg_mode) + metrics = { + "sdpo_loss": loss.detach().item(), + "self_distill_loss": loss.detach().item(), + "sdpo_alpha": float(self.self_distillation_config.get("alpha", 0.5)), + "self_distill_alpha": float(self.self_distillation_config.get("alpha", 0.5)), + "ratio/mean": masked_mean(ratio, action_mask).detach().item(), + "approx_kl": masked_mean(-log_ratio, action_mask).detach().item(), + } + return loss, metrics + + # Legacy path: SDPO loss computed directly inside policy loss. + if teacher_logprobs is None: + raise ValueError( + "SDPOPolicyLossFn requires either (advantages + old_logprob) " + "or teacher_logprobs for legacy inline distillation." + ) + + loss, metrics = compute_self_distillation_loss( + student_log_probs=logprob, + teacher_log_probs=teacher_logprobs, + response_mask=action_mask, + self_distillation_config=self.self_distillation_config, + old_log_probs=old_logprob, + student_all_log_probs=student_all_log_probs, + teacher_all_log_probs=teacher_all_log_probs, + student_topk_log_probs=student_topk_log_probs, + teacher_topk_log_probs=teacher_topk_log_probs, + self_distillation_mask=self_distillation_mask, + loss_agg_mode=self.loss_agg_mode, + rollout_is_weights=rollout_is_weights, + ) + metrics["sdpo_loss"] = metrics["self_distill_loss"] + return loss, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "alpha": 0.5, + "full_logit_distillation": True, + "distillation_use_topk": False, + "distillation_add_tail": True, + "is_clip": None, + "loss_agg_mode": "token-mean", + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/sft_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/sft_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..e677e026135311d99a67358a4f8b02ffaab460fb --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/sft_loss.py @@ -0,0 +1,30 @@ +"""SFT loss function.""" + +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss + + +class SFTLossFn(PolicyLossFn): + def __init__(self, backend: str = "verl", loss_agg_mode: str = "token-mean") -> None: + super().__init__(backend=backend) + self.loss_agg_mode = loss_agg_mode + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + action_mask: torch.Tensor, + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + sft_loss = aggregate_loss(-logprob, action_mask, loss_agg_mode=self.loss_agg_mode) + + return sft_loss, {"sft_loss": sft_loss.detach().item()} + + @classmethod + def default_args(cls): + return { + "loss_agg_mode": "token-mean", + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/sppo_loss_fn.py b/frameworks/trinity/algorithm/policy_loss_fn/sppo_loss_fn.py new file mode 100644 index 0000000000000000000000000000000000000000..b9776815adf814db2cea71d924ef32171c0480f4 --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/sppo_loss_fn.py @@ -0,0 +1,56 @@ +"""sPPO-token policy loss function. +Relevant paper: https://arxiv.org/abs/2108.05828. +""" + +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class sPPOPolicyLossFn(PolicyLossFn): + def __init__( + self, + backend: str = "verl", + epsilon: float = 0.3, + loss_agg_mode: str = "token-mean", + ) -> None: + super().__init__(backend=backend) + self.epsilon = epsilon + self.loss_agg_mode = loss_agg_mode + + def __call__( # type: ignore + self, + logprob: torch.Tensor, # [batch_size, seq_len] + old_logprob: torch.Tensor, # [batch_size, seq_len] + action_mask: torch.Tensor, # [batch_size, seq_len] + advantages: torch.Tensor, # [batch_size, seq_len] + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + """Calculate sPPO loss. + The formula is as follows: + advantages*log(clip(ratio, 1/(1+epsilon), 1+epsilon)) + ratio = exp(logprob - old_logprob) + """ + # + # token-wise + ratio = torch.exp(logprob - old_logprob).detach() + is_in_range = (ratio >= (1 / (1 + self.epsilon))) * (ratio <= (1 + self.epsilon)) + is_clipped_mask = ~is_in_range + pg_losses = -advantages * (logprob - old_logprob) * is_in_range.float() + pg_loss = aggregate_loss(pg_losses, action_mask, loss_agg_mode=self.loss_agg_mode) + pg_clipfrac = masked_mean(is_clipped_mask.float(), action_mask) + metrics = { + "pg_clipfrac": pg_clipfrac.item(), + "pg_loss": pg_loss.detach().item(), + } + return pg_loss, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "epsilon": 0.3, + "loss_agg_mode": "token-mean", + } diff --git a/frameworks/trinity/algorithm/policy_loss_fn/topr_policy_loss.py b/frameworks/trinity/algorithm/policy_loss_fn/topr_policy_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..a12a19e77943342943fd24f1a3da33dc3e08036f --- /dev/null +++ b/frameworks/trinity/algorithm/policy_loss_fn/topr_policy_loss.py @@ -0,0 +1,76 @@ +"""TOPR policy loss function. +Refer to https://arxiv.org/pdf/2503.14286v1 +""" +from typing import Dict, Tuple + +import torch + +from trinity.algorithm.policy_loss_fn.policy_loss_fn import PolicyLossFn +from trinity.algorithm.utils import aggregate_loss, masked_mean + + +class TOPRPolicyLossFn(PolicyLossFn): + def __init__( + self, + backend: str = "verl", + advantage_threshold: float = 0.0, + loss_agg_mode: str = "token-mean", + ) -> None: + super().__init__(backend=backend) + self.advantage_threshold = advantage_threshold + self.loss_agg_mode = loss_agg_mode + + def __call__( # type: ignore + self, + logprob: torch.Tensor, + old_logprob: torch.Tensor, + action_mask: torch.Tensor, + advantages: torch.Tensor, # In TOPR, this is actually the rewards R(x,y) + **kwargs, + ) -> Tuple[torch.Tensor, Dict]: + """ + Compute TOPR policy loss. + + In TOPR: + - α = [π(y|x)/μ(y|x)]_0^1 if R(x,y) <= threshold else 1 + - loss = -sg(α) * r(x,y) * log π(y|x) + """ + # in Orginal TOPR paper, advantages are simply rewards + # However, we can use advantages as rewards(Baseline Trick) + rewards = advantages + + # Compute ratio π(y|x) / μ(y|x) in log space for numerical stability + log_ratio = logprob - old_logprob + ratio = torch.exp(log_ratio) + ratio_clamped = torch.clamp(ratio, min=0.0, max=1.0) + + # Apply TOPR's conditional weighting: + # α = ratio clamp min=0 max=1 if R(x,y) <= threshold else 1 + alpha = torch.where( + rewards <= self.advantage_threshold, ratio_clamped, torch.ones_like(ratio) + ) + + # TOPR loss: l = -α * r(x,y) * log π(y|x) + # We want to maximize α * r(x,y) * log π(y|x), so minimize the negative + topr_loss = -alpha.detach() * rewards * logprob # detach alpha as it's used with stop-grad + + # Apply masking and compute mean + loss = aggregate_loss(topr_loss, action_mask, loss_agg_mode=self.loss_agg_mode) + + # Average alpha value for monitoring + avg_alpha = masked_mean(alpha, action_mask) + + metrics = { + "topr_loss": loss.detach().item(), + "avg_alpha": avg_alpha.detach().item(), + "avg_ratio": masked_mean(ratio, action_mask).detach().item(), + } + + return loss, metrics + + @classmethod + def default_args(cls) -> Dict: + return { + "advantage_threshold": 0.0, + "loss_agg_mode": "token-mean", + } diff --git a/frameworks/trinity/algorithm/sample_strategy/__init__.py b/frameworks/trinity/algorithm/sample_strategy/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..067b45a2e2d79f155809e1cf25906876d9f4e20f --- /dev/null +++ b/frameworks/trinity/algorithm/sample_strategy/__init__.py @@ -0,0 +1,17 @@ +from trinity.algorithm.sample_strategy.sample_strategy import SampleStrategy +from trinity.utils.registry import Registry + +SAMPLE_STRATEGY: Registry = Registry( + "sample_strategy", + default_mapping={ + "default": "trinity.algorithm.sample_strategy.sample_strategy.DefaultSampleStrategy", + "warmup": "trinity.algorithm.sample_strategy.sample_strategy.WarmupSampleStrategy", + "staleness_control": "trinity.algorithm.sample_strategy.sample_strategy.StalenessControlSampleStrategy", + "mix": "trinity.algorithm.sample_strategy.mix_sample_strategy.MixSampleStrategy", + }, +) + +__all__ = [ + "SAMPLE_STRATEGY", + "SampleStrategy", +] diff --git a/frameworks/trinity/algorithm/sample_strategy/mix_sample_strategy.py b/frameworks/trinity/algorithm/sample_strategy/mix_sample_strategy.py new file mode 100644 index 0000000000000000000000000000000000000000..8f0f6ed3d532c5f3599b8e108d04774fc843ba9a --- /dev/null +++ b/frameworks/trinity/algorithm/sample_strategy/mix_sample_strategy.py @@ -0,0 +1,118 @@ +import copy +from math import ceil +from typing import Dict, List, Tuple + +import torch + +from trinity.algorithm.sample_strategy.sample_strategy import SampleStrategy +from trinity.algorithm.sample_strategy.utils import representative_sample +from trinity.buffer import get_buffer_reader +from trinity.common.config import BufferConfig +from trinity.common.experience import CustomField, Experience +from trinity.utils.timer import Timer + + +class MixSampleStrategy(SampleStrategy): + """The default sample strategy.""" + + def __init__(self, buffer_config: BufferConfig, **kwargs): + super().__init__(buffer_config) + self.expert_data_ratio = kwargs.get("expert_data_ratio", 0.5) + self.sft_dataset_name = kwargs.get("sft_dataset_name", "sft_dataset") + tot_batch_size = buffer_config.train_batch_size + expert_batch_size = ceil(self.expert_data_ratio * tot_batch_size) + + # experience buffer + usual_buffer_config = copy.deepcopy(buffer_config.trainer_input.experience_buffer) + usual_buffer_config.batch_size = tot_batch_size - expert_batch_size + self.usual_exp_buffer = get_buffer_reader(usual_buffer_config) # type: ignore[arg-type] + + if buffer_config.trainer_input.auxiliary_buffers is None: + raise ValueError( + "`buffer_config.trainer_input.auxiliary_buffers` is required in MIX algorithm" + ) + + if buffer_config.trainer_input.auxiliary_buffers.get(self.sft_dataset_name) is None: + raise ValueError( + f"`{self.sft_dataset_name}` is not found in `buffer_config.trainer_input.auxiliary_buffers`" + ) + expert_storage_config = buffer_config.trainer_input.auxiliary_buffers[self.sft_dataset_name] + + if expert_storage_config.schema_type != "sft": + self.logger.warning( + f"schema_type of {self.sft_dataset_name} is not `sft`, set it to `sft`" + ) + expert_storage_config.schema_type = "sft" + + # expert experience buffer + expert_buffer_config = copy.deepcopy( + buffer_config.trainer_input.auxiliary_buffers[self.sft_dataset_name] + ) + expert_buffer_config.batch_size = expert_batch_size + self.expert_exp_buffer = get_buffer_reader( + expert_buffer_config, + ) + + async def sample(self, step: int) -> Tuple[List[Experience], Dict, List]: + metrics = {} + with Timer(metrics, "time/read_experience"): + usual_exp_list = await self.usual_exp_buffer.read_async() + for exp in usual_exp_list: + if exp.info is None: + exp.info = {} + exp.info["is_expert"] = False + exp.info["step"] = step + + expert_exp_list = await self.expert_exp_buffer.read_async() + for exp in expert_exp_list: + # we add fake rewards and logprobs to make it compatible + exp.reward = 0.0 + exp.logprobs = torch.zeros_like( + exp.tokens[exp.prompt_length :], dtype=torch.float32 + ) + exp.advantages = torch.zeros_like( + exp.tokens[exp.prompt_length :], dtype=torch.float32 + ) + if exp.info is None: + exp.info = {} + exp.info["is_expert"] = True + exp.info["step"] = step + + exp_list = usual_exp_list + expert_exp_list + repr_samples = representative_sample(exp_list) + + self.set_model_version_metric(exp_list, metrics) + custom_fields = [ + CustomField( + source_field="is_expert", + destination_field="expert_mask", + data_type=torch.bool, + ), + CustomField( + source_field="step", + destination_field="step", + data_type=torch.int32, + ), + ] + for exp in exp_list: + exp.custom_fields = custom_fields + return exp_list, metrics, repr_samples + + @classmethod + def default_args(cls) -> Dict: + return { + "expert_data_ratio": 0.5, + "sft_dataset_name": "sft_dataset", + } + + def state_dict(self) -> dict: + return { + "usual_buffer": self.usual_exp_buffer.state_dict(), + "expert_buffer": self.expert_exp_buffer.state_dict(), + } + + def load_state_dict(self, state_dict: dict) -> None: + if state_dict.get("usual_buffer", None): + self.usual_exp_buffer.load_state_dict(state_dict["usual_buffer"]) + if state_dict.get("expert_buffer", None): + self.expert_exp_buffer.load_state_dict(state_dict["expert_buffer"]) diff --git a/frameworks/trinity/algorithm/sample_strategy/sample_strategy.py b/frameworks/trinity/algorithm/sample_strategy/sample_strategy.py new file mode 100644 index 0000000000000000000000000000000000000000..2398961ae69a9af3da9a0fecf7ddffd64cb8a106 --- /dev/null +++ b/frameworks/trinity/algorithm/sample_strategy/sample_strategy.py @@ -0,0 +1,99 @@ +from abc import ABC, abstractmethod +from typing import Dict, List, Tuple + +from trinity.algorithm.sample_strategy.utils import representative_sample +from trinity.buffer import get_buffer_reader +from trinity.common.config import BufferConfig +from trinity.common.experience import Experience +from trinity.utils.annotations import Deprecated +from trinity.utils.monitor import gather_metrics +from trinity.utils.timer import Timer + + +class SampleStrategy(ABC): + def __init__(self, buffer_config: BufferConfig, **kwargs) -> None: + pass + + def set_model_version_metric(self, exp_list: List[Experience], metrics: Dict): + metric_list = [ + {"model_version": exp.info["model_version"]} + for exp in exp_list + if "model_version" in exp.info + ] + metrics.update(gather_metrics(metric_list, "sample")) + + @abstractmethod + async def sample(self, step: int) -> Tuple[List[Experience], Dict, List]: + """Sample data from buffer. + + Args: + step (`int`): The step number of current step. + + Returns: + `List[Experience]`: The sampled List[Experience] data. + `Dict`: Metrics for logging. + `List`: Representative data for logging. + """ + + @classmethod + @abstractmethod + def default_args(cls) -> dict: + """Get the default arguments of the sample strategy.""" + + @abstractmethod + def state_dict(self) -> dict: + """Get the state dict of the sample strategy.""" + + @abstractmethod + def load_state_dict(self, state_dict: dict) -> None: + """Load the state dict of the sample strategy.""" + + +class DefaultSampleStrategy(SampleStrategy): + def __init__(self, buffer_config: BufferConfig, **kwargs): + super().__init__(buffer_config) + self.exp_buffer = get_buffer_reader(buffer_config.trainer_input.experience_buffer) # type: ignore[arg-type] + + async def sample(self, step: int, **kwargs) -> Tuple[List[Experience], Dict, List]: + metrics = {} + with Timer(metrics, "time/read_experience"): + exp_list = await self.exp_buffer.read_async() + repr_samples = representative_sample(exp_list) + self.set_model_version_metric(exp_list, metrics) + return exp_list, metrics, repr_samples + + @classmethod + def default_args(cls) -> dict: + return {} + + def state_dict(self) -> dict: + return self.exp_buffer.state_dict() + + def load_state_dict(self, state_dict: dict) -> None: + if state_dict: + self.exp_buffer.load_state_dict(state_dict) + + +class StalenessControlSampleStrategy(DefaultSampleStrategy): + def __init__(self, buffer_config: BufferConfig, **kwargs): + super().__init__(buffer_config) + self.max_staleness = kwargs.get("max_staleness", float("inf")) + + async def sample(self, step: int, **kwargs) -> Tuple[List[Experience], Dict, List]: + min_model_version = max(step - self.max_staleness, 0) + metrics = {} + with Timer(metrics, "time/read_experience"): + exp_list = await self.exp_buffer.read_async(min_model_version=min_model_version) + repr_samples = representative_sample(exp_list) + self.set_model_version_metric(exp_list, metrics) + return exp_list, metrics, repr_samples + + +@Deprecated +class WarmupSampleStrategy(DefaultSampleStrategy): + """The warmup sample strategy. + Deprecated, keep this class for backward compatibility only. + Please use DefaultSampleStrategy instead.""" + + def __init__(self, buffer_config: BufferConfig, **kwargs): + super().__init__(buffer_config) diff --git a/frameworks/trinity/algorithm/sample_strategy/utils.py b/frameworks/trinity/algorithm/sample_strategy/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..80c76ec016455cc6c5671729c1b43c4322c1f980 --- /dev/null +++ b/frameworks/trinity/algorithm/sample_strategy/utils.py @@ -0,0 +1,42 @@ +import random +from typing import List + +from trinity.common.experience import Experience + + +def representative_sample(experiences: List[Experience]) -> List[dict]: + if experiences[0].reward is None: + sample = random.choice(experiences) + return [ + { + "prompt": sample.prompt_text, + "response": sample.response_text, + } + ] + samples = [] + min_reward_sample = None + max_reward_sample = None + for exp in experiences: + if exp.reward is None: + continue + if min_reward_sample is None or exp.reward < min_reward_sample.reward: + min_reward_sample = exp + if max_reward_sample is None or exp.reward > max_reward_sample.reward: + max_reward_sample = exp + if min_reward_sample is not None: + samples.append( + { + "prompt": min_reward_sample.prompt_text, + "response": min_reward_sample.response_text, + "reward": min_reward_sample.reward, + } + ) + if max_reward_sample is not None: + samples.append( + { + "prompt": max_reward_sample.prompt_text, + "response": max_reward_sample.response_text, + "reward": max_reward_sample.reward, + } + ) + return samples diff --git a/frameworks/trinity/algorithm/utils.py b/frameworks/trinity/algorithm/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..707639f4ccc1d929364e93bf745fca0416526717 --- /dev/null +++ b/frameworks/trinity/algorithm/utils.py @@ -0,0 +1,107 @@ +"""Common utils for algorithm module. + +Modified from https://github.com/volcengine/verl/blob/main/verl/utils/torch_functional.py +""" + +import torch + + +def aggregate_loss(values, mask, loss_agg_mode="token-mean", normalizer=None): + """ + Compute loss from values and mask with various aggregation modes. + Modified from: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py + + Args: + values (torch.Tensor): Arbitrary shape tensor of values to aggregate. + mask (torch.BoolTensor or torch.FloatTensor): Same shape as values, 1/True = include, 0 = ignore. + loss_agg_mode (str): One of the following: + - "token-mean": mean over all unmasked elements. + - "seq-mean-token-sum": average over sequences, where each sequence's loss is sum of unmasked values. + - "seq-mean-token-mean": average over sequences, where each sequence's loss is mean of unmasked values. + - "seq-mean-token-sum-norm": total sum of unmasked values divided by a fixed normalizer (e.g., seq length). + normalizer (float or None): Only used in 'seq-mean-token-sum-norm'. If None, uses mask.shape[-1]. + + Returns: + torch.Tensor: Scalar loss value. + """ + if loss_agg_mode == "token-mean": + return masked_mean(values, mask) + + elif loss_agg_mode == "seq-mean-token-sum": + # Sum over last dimension (token-level), then take mean across batch (sequence-level) + seq_losses = masked_sum(values, mask, axis=-1) # [batch] + return seq_losses.mean() + + elif loss_agg_mode == "seq-mean-token-mean": + # Mean over tokens per sequence, then mean over sequences + seq_losses = masked_mean(values, mask, axis=-1) # [batch] + return seq_losses.mean() + + elif loss_agg_mode == "seq-mean-token-sum-norm": + total_token_sum = masked_sum(values, mask) # scalar + norm = normalizer if normalizer is not None else mask.shape[-1] + return total_token_sum / (norm + 1e-8) + + else: + raise ValueError( + f"Invalid loss_agg_mode: {loss_agg_mode}. " + f"Choose from ['token-mean', 'seq-mean-token-sum', 'seq-mean-token-mean', 'seq-mean-token-sum-norm']" + ) + + +def masked_sum(values, mask, axis=None): + """Compute mean of tensor with a masked values.""" + valid_values = torch.where(mask.bool(), values, 0.0) + return (valid_values * mask).sum(axis=axis) + + +def masked_mean(values, mask, axis=None): + """Compute mean of tensor with a masked values.""" + return masked_sum(values, mask, axis=axis) / (mask.sum(axis=axis) + 1e-8) + + +def masked_var(values, mask, unbiased=True): + """Compute variance of tensor with masked values.""" + mean = masked_mean(values, mask) + centered_values = values - mean + variance = masked_mean(centered_values**2, mask) + if unbiased: + mask_sum = mask.sum() + if mask_sum == 0: + raise ValueError("At least one element in the mask has to be 1.") + # note that if mask_sum == 1, then there is a division by zero issue + # to avoid it you just need to use a larger minibatch_size + if mask_sum == 1: + raise ValueError("The sum of the mask is one, which can cause a division by zero.") + bessel_correction = mask_sum / (mask_sum - 1) + variance = variance * bessel_correction + return variance + + +def masked_whiten(values, mask, shift_mean=True): + """ + Whiten `values` by normalizing with mean and variance computed over `mask`. + + Args: + values (torch.Tensor): Input tensor. + mask (torch.Tensor): Boolean tensor of same shape, selects elements for stats. + shift_mean (bool): If True (default), output is zero-mean; + if False, the original mean is re-added after scaling. + + Returns: + torch.Tensor: Whitened tensor of same shape as `values`. + """ + mean, var = masked_mean(values, mask), masked_var(values, mask) + whitened = (values - mean) * torch.rsqrt(var + 1e-8) + if not shift_mean: + whitened += mean + return whitened + + +def prefix_metrics(src_metrics: dict, prefix: str, dst_metrics: dict = None) -> dict: + if dst_metrics is None: + dst_metrics = {} + for k, v in src_metrics.items(): + dst_metrics[f"{prefix}/{k}"] = v + + return dst_metrics diff --git a/frameworks/trinity/buffer/__init__.py b/frameworks/trinity/buffer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..83c010bcdf365fa701b69e4769fd2f012cba506c --- /dev/null +++ b/frameworks/trinity/buffer/__init__.py @@ -0,0 +1,13 @@ +from trinity.buffer.buffer import ( + BufferReader, + BufferWriter, + get_buffer_reader, + get_buffer_writer, +) + +__all__ = [ + "BufferReader", + "BufferWriter", + "get_buffer_reader", + "get_buffer_writer", +] diff --git a/frameworks/trinity/buffer/buffer.py b/frameworks/trinity/buffer/buffer.py new file mode 100644 index 0000000000000000000000000000000000000000..2eab5791b2a7a2bfd538eed3a8dccc8168df454c --- /dev/null +++ b/frameworks/trinity/buffer/buffer.py @@ -0,0 +1,45 @@ +# -*- coding: utf-8 -*- +"""The buffer module""" +from typing import Union + +from trinity.buffer.buffer_reader import BufferReader +from trinity.buffer.buffer_writer import BufferWriter +from trinity.buffer.reader import READER +from trinity.common.config import ExperienceBufferConfig, StorageConfig, TasksetConfig +from trinity.common.constants import StorageType + +BufferStorageConfig = Union[TasksetConfig, ExperienceBufferConfig, StorageConfig] + + +def get_buffer_reader(config: BufferStorageConfig) -> BufferReader: + """Get a buffer reader for the given dataset name.""" + if not isinstance(config, StorageConfig): + storage_config: StorageConfig = config.to_storage_config() + else: + storage_config = config + reader_cls = READER.get(storage_config.storage_type) + if reader_cls is None: + raise ValueError(f"{storage_config.storage_type} not supported.") + return reader_cls(storage_config) + + +def get_buffer_writer(config: BufferStorageConfig) -> BufferWriter: + """Get a buffer writer for the given dataset name.""" + if not isinstance(config, StorageConfig): + storage_config: StorageConfig = config.to_storage_config() + else: + storage_config = config + if storage_config.storage_type == StorageType.SQL.value: + from trinity.buffer.writer.sql_writer import SQLWriter + + return SQLWriter(storage_config) + elif storage_config.storage_type == StorageType.QUEUE.value: + from trinity.buffer.writer.queue_writer import QueueWriter + + return QueueWriter(storage_config) + elif storage_config.storage_type == StorageType.FILE.value: + from trinity.buffer.writer.file_writer import JSONWriter + + return JSONWriter(storage_config) + else: + raise ValueError(f"{storage_config.storage_type} not supported.") diff --git a/frameworks/trinity/buffer/buffer_reader.py b/frameworks/trinity/buffer/buffer_reader.py new file mode 100644 index 0000000000000000000000000000000000000000..ad4d4145470381e55fd16808559b517b7ff16bb1 --- /dev/null +++ b/frameworks/trinity/buffer/buffer_reader.py @@ -0,0 +1,30 @@ +"""Reader of the buffer.""" +from abc import ABC, abstractmethod +from typing import Dict, List, Optional + + +class BufferReader(ABC): + """Interface of the buffer reader.""" + + @abstractmethod + def read(self, batch_size: Optional[int] = None, **kwargs) -> List: + """Read from buffer.""" + + @abstractmethod + async def read_async(self, batch_size: Optional[int] = None, **kwargs) -> List: + """Read from buffer asynchronously.""" + + def __len__(self) -> int: + """Get the number of samples in buffer.""" + raise NotImplementedError + + def state_dict(self) -> Dict: + """Return the state of the reader as a dict. + Returns: + A dict containing the reader state. At minimum, it should contain + the `current_index` field. + """ + raise NotImplementedError + + def load_state_dict(self, state_dict: Dict) -> None: + raise NotImplementedError diff --git a/frameworks/trinity/buffer/buffer_writer.py b/frameworks/trinity/buffer/buffer_writer.py new file mode 100644 index 0000000000000000000000000000000000000000..3d3e939196115bb03e700616bf71b6994f3e99d7 --- /dev/null +++ b/frameworks/trinity/buffer/buffer_writer.py @@ -0,0 +1,31 @@ +"""Writer of the buffer.""" +from abc import ABC, abstractmethod +from typing import List + + +class BufferWriter(ABC): + """Interface of the buffer writer.""" + + @abstractmethod + def write(self, data: List) -> None: + """Write to buffer.""" + + @abstractmethod + async def write_async(self, data: List) -> None: + """Write to buffer asynchronously.""" + + @abstractmethod + async def acquire(self) -> int: + """Acquire the buffer writer. + + Returns: + `int`: The reference count of the buffer after acquiring. + """ + + @abstractmethod + async def release(self) -> int: + """Release the buffer writer. After release, the buffer writer can not be used again. + + Returns: + `int`: The reference count of the buffer after releasing. + """ diff --git a/frameworks/trinity/buffer/operators/__init__.py b/frameworks/trinity/buffer/operators/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e83b7d05eedd427819a995601ccb71e9e53bc2e9 --- /dev/null +++ b/frameworks/trinity/buffer/operators/__init__.py @@ -0,0 +1,19 @@ +from trinity.buffer.operators.experience_operator import ExperienceOperator +from trinity.utils.registry import Registry + +EXPERIENCE_OPERATORS: Registry = Registry( + "experience_operators", + default_mapping={ + "reward_filter": "trinity.buffer.operators.filters.reward_filter.RewardFilter", + "reward_std_filter": "trinity.buffer.operators.filters.reward_filter.RewardSTDFilter", + "reward_shaping_mapper": "trinity.buffer.operators.mappers.reward_shaping_mapper.RewardShapingMapper", + "pass_rate_calculator": "trinity.buffer.operators.mappers.pass_rate_calculator.PassRateCalculator", + "data_juicer": "trinity.buffer.operators.data_juicer_operator.DataJuicerOperator", + "invalid_reward_filter": "trinity.buffer.operators.filters.reward_filter.InvalidRewardFilter", + }, +) + +__all__ = [ + "ExperienceOperator", + "EXPERIENCE_OPERATORS", +] diff --git a/frameworks/trinity/buffer/operators/data_juicer_operator.py b/frameworks/trinity/buffer/operators/data_juicer_operator.py new file mode 100644 index 0000000000000000000000000000000000000000..4287f79287d1d895d7331bdcf28e1c85afea564d --- /dev/null +++ b/frameworks/trinity/buffer/operators/data_juicer_operator.py @@ -0,0 +1,46 @@ +from typing import Dict, List, Optional, Tuple + +from trinity.buffer.operators.experience_operator import ExperienceOperator +from trinity.common.config import DataJuicerServiceConfig +from trinity.common.experience import Experience +from trinity.service.data_juicer.client import DataJuicerClient + + +class DataJuicerOperator(ExperienceOperator): + def __init__( + self, + service_config: DataJuicerServiceConfig, + operators: Optional[List[Dict]] = None, + config_path: Optional[str] = None, + np: int = 4, + ): + """ + Initialize the DataJuicerOperator. + + Args: + service_config (config): The configuration for the DataJuicer service. + operators(`List[Dict]`): A list of operators with their configurations. + config_path(`str`): Path to the Data-Juicer configuration file. + np (`int`): Number of processes to use for Data-Juicer. Default is 4. + + Note: + - Must include one of the following, and the priority is from high to low: + - `config_path` (`str`) + - `operators` (`List[Dict]`) + """ + self.client = DataJuicerClient(config=service_config) + self.client.initialize( + { + "operators": operators, + "config_path": config_path, + "np": np, + "pipeline_type": "experience", + } + ) + + def process(self, exps: List[Experience]) -> Tuple[List[Experience], Dict]: + return self.client.process_experience(exps) + + def close(self): + """Close the DataJuicer client connection.""" + self.client.close() diff --git a/frameworks/trinity/buffer/operators/experience_operator.py b/frameworks/trinity/buffer/operators/experience_operator.py new file mode 100644 index 0000000000000000000000000000000000000000..7834dcd2be894ba3504f2950af86c64d01e5c224 --- /dev/null +++ b/frameworks/trinity/buffer/operators/experience_operator.py @@ -0,0 +1,50 @@ +from __future__ import annotations + +from abc import ABC, abstractmethod +from typing import Dict, List, Tuple + +from trinity.common.config import OperatorConfig +from trinity.common.experience import Experience + + +class ExperienceOperator(ABC): + """ + Base class for all experience operators in the Trinity framework. + Operators are used to process experiences and perform some transformations based on them. + """ + + @abstractmethod + def process(self, exps: List[Experience]) -> Tuple[List[Experience], Dict]: + """Process a list of experiences and return a transformed list. + + Args: + exps (List[Experience]): List of experiences to process, which contains + all experiences generated by the Explorer in one explore step. + Returns: + Tuple[List[Experience], Dict]: A tuple containing the processed list of experiences and a dictionary of metrics. + """ + + @classmethod + def create_operators(cls, operator_configs: List[OperatorConfig]) -> List[ExperienceOperator]: + """Create a list of ExperienceOperator instances based on the provided operator configurations. + + Args: + operator_configs (List[OperatorConfig]): List of operator configurations. + + Returns: + List[ExperienceOperator]: List of instantiated ExperienceOperator objects. + """ + # Import here to avoid circular import + from trinity.buffer.operators import EXPERIENCE_OPERATORS + + operators = [] + for config in operator_configs: + operator_class = EXPERIENCE_OPERATORS.get(config.name) + if not operator_class: + raise ValueError(f"Unknown operator: {config.name}") + operators.append(operator_class(**config.args)) + return operators + + def close(self): + """Close the operator if it has any resources to release.""" + pass diff --git a/frameworks/trinity/buffer/operators/filters/__init__.py b/frameworks/trinity/buffer/operators/filters/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/frameworks/trinity/buffer/operators/filters/reward_filter.py b/frameworks/trinity/buffer/operators/filters/reward_filter.py new file mode 100644 index 0000000000000000000000000000000000000000..07126db12f226e6f55283021eedd0735932a5021 --- /dev/null +++ b/frameworks/trinity/buffer/operators/filters/reward_filter.py @@ -0,0 +1,68 @@ +from typing import List, Tuple + +import numpy as np + +from trinity.buffer.operators import ExperienceOperator +from trinity.common.experience import Experience, group_by + + +class RewardFilter(ExperienceOperator): + """ + Filter experiences based on the reward value. + + Note: This filter assumes that the reward is already calculated and stored in the Experience object. + """ + + def __init__(self, threshold: float = 0.0): + self.threshold = threshold + + def process(self, exps: List[Experience]) -> Tuple[List[Experience], dict]: + """Filter experiences based on reward value.""" + filtered_exps = [exp for exp in exps if exp.reward >= self.threshold] # type: ignore [operator] + metrics = {"filtered_count": len(exps) - len(filtered_exps)} + return filtered_exps, metrics + + +class RewardSTDFilter(ExperienceOperator): + """ + Filter experiences based on the standard deviation of rewards within each group. + + Note: This filter assumes that the reward is already calculated and stored in the Experience object. + """ + + def __init__(self, threshold: float = 0.0): + self.threshold = threshold + + def process(self, exps: List[Experience]) -> Tuple[List[Experience], dict]: + """Filter experiences based on reward std.""" + metrics = {} + result_exps = [] + original_count = len(exps) + grouped_experiences = group_by(exps, id_type="task") + for _, group_exps in grouped_experiences.items(): + if len(group_exps) < 2: + continue + rewards = [exp.reward for exp in group_exps] + variance = np.std(rewards) + if variance <= self.threshold: + continue + result_exps.extend(group_exps) + final_count = len(result_exps) + metrics["filtered_count"] = original_count - final_count + return result_exps, metrics + + +class InvalidRewardFilter(ExperienceOperator): + """ + Filters out experiences with invalid reward values. + + Note: This operator assumes that rewards are already computed and stored in the + Experience object.Any experience with a missing (`None`) or invalid (`NaN`) + reward is removed to prevent low-quality data from entering the training + pipeline. + """ + + def process(self, exps: List[Experience]) -> Tuple[List[Experience], dict]: + kept = [e for e in exps if e.reward is not None and e.reward == e.reward] + + return kept, {"filtered_count": len(exps) - len(kept)} diff --git a/frameworks/trinity/buffer/operators/mappers/__init__.py b/frameworks/trinity/buffer/operators/mappers/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/frameworks/trinity/buffer/operators/mappers/pass_rate_calculator.py b/frameworks/trinity/buffer/operators/mappers/pass_rate_calculator.py new file mode 100644 index 0000000000000000000000000000000000000000..31dce63d1778fffa7bd4481f2549ed0c04e3767a --- /dev/null +++ b/frameworks/trinity/buffer/operators/mappers/pass_rate_calculator.py @@ -0,0 +1,60 @@ +from collections import defaultdict +from typing import Dict, List, Tuple + +import numpy as np + +from trinity.buffer.operators.experience_operator import ExperienceOperator +from trinity.common.constants import SELECTOR_METRIC +from trinity.common.experience import Experience + + +class PassRateCalculator(ExperienceOperator): + def __init__(self, **kwargs): + pass + + def process(self, exps: List[Experience]) -> Tuple[List[Experience], Dict]: + raw_metric = defaultdict(lambda: defaultdict(list)) + for exp in exps: + task_index = exp.info["task_index"] + assert "taskset_id" in task_index + assert "index" in task_index + raw_metric[task_index["taskset_id"]][task_index["index"]].append(exp.reward) + metric = {} + ref_pass_rates = [] + for taskset_id, taskset_metric in raw_metric.items(): + indices = [] + reward_means = [] + for index, rewards in taskset_metric.items(): + indices.append(index) + reward_means.append(float(np.mean(rewards))) + metric[taskset_id] = { + "indices": indices, + "values": reward_means, + } + ref_pass_rates.extend(reward_means) + ret_metric = {SELECTOR_METRIC: metric} + + valid_ratio = np.mean([1 if 0 < pr < 1 else 0 for pr in ref_pass_rates]) + strict_valid_ratio = np.mean( + [1 if 1 / 16 + 1e-3 < pr < 15 / 16 - 1e-3 else 0 for pr in ref_pass_rates] + ) + less_than_one_ratio = np.mean([1 if pr < 1 else 0 for pr in ref_pass_rates]) + larger_than_zero_ratio = np.mean([1 if pr > 0 else 0 for pr in ref_pass_rates]) + less_than_15_over_16_ratio = np.mean( + [1 if pr < 15 / 16 - 1e-3 else 0 for pr in ref_pass_rates] + ) + larger_than_1_over_16_ratio = np.mean( + [1 if pr > 1 / 16 + 1e-3 else 0 for pr in ref_pass_rates] + ) + ret_metric.update( + { + "selection/valid_ratio": valid_ratio, + "selection/strict_valid_ratio": strict_valid_ratio, + "selection/<1_ratio": less_than_one_ratio, + "selection/>0_ratio": larger_than_zero_ratio, + "selection/<15_16_ratio": less_than_15_over_16_ratio, + "selection/>1_16_ratio": larger_than_1_over_16_ratio, + } + ) + + return exps, ret_metric diff --git a/frameworks/trinity/buffer/operators/mappers/reward_shaping_mapper.py b/frameworks/trinity/buffer/operators/mappers/reward_shaping_mapper.py new file mode 100644 index 0000000000000000000000000000000000000000..a1d56839ff78278e3a86433a4f4de0bef9582c9c --- /dev/null +++ b/frameworks/trinity/buffer/operators/mappers/reward_shaping_mapper.py @@ -0,0 +1,111 @@ +from typing import Dict, List, Optional, Tuple + +from trinity.buffer.operators import ExperienceOperator +from trinity.common.constants import OpType +from trinity.common.experience import Experience + + +class RewardShapingMapper(ExperienceOperator): + """Re-shaping the existing rewards of experiences based on rules or other advanced methods. + + Note: + This mapper assumes that the reward is already calculated and stored in the Experience object, + and the necessary stats are already calculated and stored in the Experience info field. + """ + + def __init__(self, reward_shaping_configs: Optional[List[Dict]] = None): + """Initializes the RewardShapingMapper. + + Args: + reward_shaping_configs (list[dict], optional): A list of dictionaries containing reward shaping + configurations. Each dictionary should include the following keys: + + - stats_key (str): The field key name of target stats used to shape the reward. + - op_type (str): The type of operator to apply between the reward and the target stats. + Should be one of {"ADD", "SUB", "MUL", "DIV"}. + - weight (float): The weight for the target stats. + + Example: + [ + { + "stats_key": "llm_quality_score", + "op_type": "ADD", + "weight": 1.0, + } + ] + """ + if reward_shaping_configs is None: + reward_shaping_configs = [] + self.reward_shaping_configs = reward_shaping_configs + + def process(self, exps: List[Experience]) -> Tuple[List[Experience], Dict]: + res_exps = [] + reward_diff = [] + for exp in exps: + # skip experiences that don't have reward + if exp.reward is None: + res_exps.append(exp) + reward_diff.append(0.0) + continue + res_exp = exp + previous_reward = exp.reward + for reward_shaping_config in self.reward_shaping_configs: + res_exp = self._reward_shaping_single(res_exp, reward_shaping_config) + if res_exp.reward is None: + res_exps.append(exp) + reward_diff.append(0.0) + continue + res_reward = res_exp.reward + reward_diff.append(res_reward - previous_reward) + res_exps.append(res_exp) + if len(reward_diff) == 0: + return res_exps, { + "reward_diff/mean": 0, + "reward_diff/min": 0, + "reward_diff/max": 0, + } + metrics = { + "reward_diff/mean": 1.0 * sum(reward_diff) / len(reward_diff), + "reward_diff/min": min(reward_diff), + "reward_diff/max": max(reward_diff), + } + return res_exps, metrics + + def _reward_shaping_single(self, exp: Experience, reward_shaping_config: Dict): + """Re-shapes the existing reward of one experience based on the given reward_shaping_config. + + Args: + exp (Experience): The experience object whose reward is to be reshaped. + reward_shaping_config (dict): A dictionary containing the reward shaping configuration. + It should include the following keys: + - stats_key (str): The field key name of target stats used to shape the reward. + - op_type (str): The type of operator to apply between the reward and the target stats. + Should be one of {"ADD", "SUB", "MUL", "DIV"}. + - weight (float): The weight for the target stats. + + Returns: + Experience: The experience object with the reshaped reward. + """ + tgt_stats = reward_shaping_config.get("stats_key", None) + op_type = OpType[reward_shaping_config.get("op_type", "ADD")] + weight = reward_shaping_config.get("weight", 1.0) + # if the target stats is not specified, skip the stats and return the original experience + if tgt_stats is None: + return exp + exp_info = exp.info + if exp_info is None or len(exp_info) == 0: + return exp + # if the target stats does not exist in the exp info, skip the stats and return the original experience + if tgt_stats not in exp_info: + return exp + if op_type == OpType.ADD: + exp.reward += weight * exp_info[tgt_stats] + elif op_type == OpType.MUL: + exp.reward *= weight * exp_info[tgt_stats] + elif op_type == OpType.SUB: + exp.reward -= weight * exp_info[tgt_stats] + elif op_type == OpType.DIV: + divisor = weight * exp_info[tgt_stats] + if divisor != 0: + exp.reward /= divisor + return exp diff --git a/frameworks/trinity/buffer/pipelines/__init__.py b/frameworks/trinity/buffer/pipelines/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..c24f9507baac88eb7c019c1fccb3580f199f011c --- /dev/null +++ b/frameworks/trinity/buffer/pipelines/__init__.py @@ -0,0 +1,11 @@ +from trinity.buffer.pipelines.experience_pipeline import ExperiencePipeline +from trinity.buffer.pipelines.task_pipeline import ( + TaskPipeline, + check_and_run_task_pipeline, +) + +__all__ = [ + "ExperiencePipeline", + "TaskPipeline", + "check_and_run_task_pipeline", +] diff --git a/frameworks/trinity/buffer/pipelines/experience_pipeline.py b/frameworks/trinity/buffer/pipelines/experience_pipeline.py new file mode 100644 index 0000000000000000000000000000000000000000..05ef640f943008d50b54b78109c0b319b68a86d5 --- /dev/null +++ b/frameworks/trinity/buffer/pipelines/experience_pipeline.py @@ -0,0 +1,150 @@ +import time +import traceback +from typing import Dict, List, Optional + +from trinity.buffer.buffer import BufferWriter, get_buffer_reader, get_buffer_writer +from trinity.buffer.operators.experience_operator import ExperienceOperator +from trinity.buffer.storage.queue import is_database_url, is_json_file +from trinity.common.config import ( + AlgorithmConfig, + Config, + ExperiencePipelineConfig, + StorageConfig, +) +from trinity.common.constants import SELECTOR_METRIC, StorageType +from trinity.common.experience import Experience +from trinity.utils.log import get_logger +from trinity.utils.plugin_loader import load_plugins +from trinity.utils.timer import Timer + + +def get_input_buffers(pipeline_config: ExperiencePipelineConfig) -> Dict: + """Get input buffers for the experience pipeline.""" + input_buffers = {} + for input_name, input_config in pipeline_config.inputs.items(): + buffer_reader = get_buffer_reader(input_config) + input_buffers[input_name] = buffer_reader + return input_buffers + + +class ExperiencePipeline: + """ + A class to process experiences. + """ + + def __init__(self, config: Config): + self.logger = get_logger(f"{config.explorer.name}_experience_pipeline", in_ray_actor=True) + load_plugins() + pipeline_config = config.data_processor.experience_pipeline + self.input_store = self._init_input_storage(pipeline_config) # type: ignore [arg-type] + try: + self.operators = ExperienceOperator.create_operators(pipeline_config.operators) + except Exception as e: + self.logger.error(f"Failed to create experience operators: {traceback.format_exc()}") + raise e + self._set_algorithm_operators(config.algorithm) + self.output = get_buffer_writer( + config.buffer.trainer_input.experience_buffer, # type: ignore [arg-type] + ) + + def _init_input_storage( + self, + pipeline_config: ExperiencePipelineConfig, + ) -> Optional[BufferWriter]: + """Initialize the input storage if it is not already set.""" + if pipeline_config.save_input: + if pipeline_config.input_save_path is None: + raise ValueError("input_save_path must be set when save_input is True.") + elif is_json_file(pipeline_config.input_save_path): + return get_buffer_writer( + StorageConfig( + storage_type=StorageType.FILE.value, + path=pipeline_config.input_save_path, + schema_type="experience", + wrap_in_ray=False, + ), + ) + elif is_database_url(pipeline_config.input_save_path): + return get_buffer_writer( + StorageConfig( + name="pipeline_input", + storage_type=StorageType.SQL.value, + path=pipeline_config.input_save_path, + schema_type="experience", + wrap_in_ray=False, + ), + ) + else: + raise ValueError( + f"Unsupported save_input format: {pipeline_config.save_input}. " + "Only JSON file path or SQLite URL is supported." + ) + else: + return None + + def _set_algorithm_operators(self, algorithm_config: AlgorithmConfig) -> None: + """Add algorithm-specific operators to the pipeline.""" + from trinity.algorithm import ADVANTAGE_FN, ALGORITHM_TYPE + + algorithm = ALGORITHM_TYPE.get(algorithm_config.algorithm_type) + if not algorithm.compute_advantage_in_trainer and algorithm_config.advantage_fn: + advantage_fn_cls = ADVANTAGE_FN.get(algorithm_config.advantage_fn) + assert ( + advantage_fn_cls is not None + ), f"AdvantageFn {algorithm_config.advantage_fn} not found." + assert ( + not advantage_fn_cls.compute_in_trainer() + ), f"AdvantageFn {algorithm_config.advantage_fn} can only be computed in the trainer, please check your implementation." + self.operators.append(advantage_fn_cls(**algorithm_config.advantage_fn_args)) + + async def prepare(self) -> None: + await self.output.acquire() + + async def process(self, exps: List[Experience]) -> Dict: + """Process a batch of experiences. + + Args: + exps (List[Experience]): List of experiences to process. These experiences are typically generated by an explorer in one step. + + Returns: + Dict: A dictionary containing metrics collected during the processing of experiences. + """ + st = time.time() + if self.input_store is not None: + await self.input_store.write_async(exps) + + metrics = {} + + # Process experiences through operators + for idx, operator in enumerate(self.operators): + with Timer( + metrics, f"time/experience_pipeline/operator/{idx}_{operator.__class__.__name__}" + ): + exps, metric = operator.process(exps) + metrics.update(metric) + metrics["experience_count"] = len(exps) + + # Write processed experiences to output buffer + with Timer(metrics, "time/experience_pipeline/write"): + await self.output.write_async(exps) + metrics["time/experience_pipeline/total"] = time.time() - st + + # prefix metrics keys with 'pipeline/' + result_metrics = {} + for key, value in metrics.items(): + if key.startswith("time/"): + result_metrics[key] = value + elif isinstance(value, (int, float)): + result_metrics[f"experience_pipeline/{key}"] = float(value) + if SELECTOR_METRIC in metrics: + result_metrics[SELECTOR_METRIC] = metrics[SELECTOR_METRIC] + + return result_metrics + + async def close(self) -> None: + try: + await self.output.release() + except Exception as e: + self.logger.error(f"Failed to release output buffer: {e}") + for operator in self.operators: + operator.close() diff --git a/frameworks/trinity/buffer/pipelines/task_pipeline.py b/frameworks/trinity/buffer/pipelines/task_pipeline.py new file mode 100644 index 0000000000000000000000000000000000000000..9107293902a647ba72af7d9acfd9bb65cfbe7ade --- /dev/null +++ b/frameworks/trinity/buffer/pipelines/task_pipeline.py @@ -0,0 +1,74 @@ +from typing import Any, Dict + +from trinity.common.config import Config, OperatorConfig, TaskPipelineConfig +from trinity.utils.log import get_logger + + +def check_and_run_task_pipeline(config: Config) -> Dict: + if config.mode not in {"explore", "train", "both"}: + return {} + if config.data_processor.task_pipeline is None: + return {} + + task_pipeline = TaskPipeline(config) + try: + return task_pipeline.process() + except Exception as e: + raise RuntimeError(f"Task pipeline failed: {e}") + finally: + task_pipeline.close() + + +class TaskPipeline: + """ + A class to process task datasets through DataJuicer. + """ + + def __init__(self, config: Config): + self.logger = get_logger(__name__) + from trinity.service.data_juicer.client import DataJuicerClient + + self.client = DataJuicerClient(config.service.data_juicer) # type: ignore [arg-type] + self.pipeline_config = config.data_processor.task_pipeline + + def convert_pipeline_config(self, pipeline_config: TaskPipelineConfig) -> Dict[str, Any]: + """ + Convert the TaskPipelineConfig to a format suitable for DataJuicer. + """ + + def _convert_operator(operator: OperatorConfig) -> Dict: + return {operator.name: {key: value for key, value in operator.args.items()}} + + if pipeline_config.output.path is None: + raise ValueError("When using task pipeline, taskset.path must be set.") + + converted_config = { + "pipeline_type": "task", + "operators": [_convert_operator(op) for op in pipeline_config.operators], + "np": pipeline_config.num_process, + "config_path": pipeline_config.config_path, + "inputs": [path for path in pipeline_config.inputs], + "target_fields": pipeline_config.target_fields, + "output_dir": pipeline_config.output.path, + "priority_weights": pipeline_config.priority_weights, + "top_k": pipeline_config.top_k, + } + return converted_config + + def process(self) -> Dict[str, Any]: + """ + Process the task datasets using DataJuicer. + + Returns: + Dict[str, Any]: Metrics for logging. + """ + # Convert the pipeline configuration + converted_config = self.convert_pipeline_config(self.pipeline_config) # type: ignore [arg-type] + self.client.initialize(converted_config) + self.logger.info("Starting task processing...") + metrics = self.client.process_task() + self.logger.info("Task processing completed.") + return metrics + + def close(self): + self.client.close() diff --git a/frameworks/trinity/buffer/reader/__init__.py b/frameworks/trinity/buffer/reader/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..b6968a71581c6399e9e71e32e524c58407f266d8 --- /dev/null +++ b/frameworks/trinity/buffer/reader/__init__.py @@ -0,0 +1,12 @@ +from trinity.utils.registry import Registry + +READER = Registry( + "reader", + default_mapping={ + "file": "trinity.buffer.reader.file_reader.FileReader", + "queue": "trinity.buffer.reader.queue_reader.QueueReader", + "sql": "trinity.buffer.reader.sql_reader.SQLReader", + }, +) + +__all__ = ["READER"] diff --git a/frameworks/trinity/buffer/reader/file_reader.py b/frameworks/trinity/buffer/reader/file_reader.py new file mode 100644 index 0000000000000000000000000000000000000000..1c606e450ae725dd9c4e5f6030ea0c808d284b78 --- /dev/null +++ b/frameworks/trinity/buffer/reader/file_reader.py @@ -0,0 +1,242 @@ +"""Filed based buffer reader.""" + +from typing import List, Optional, Tuple + +import datasets +from datasets import Dataset, load_dataset + +from trinity.buffer.buffer_reader import BufferReader +from trinity.buffer.schema import FORMATTER +from trinity.common.config import StorageConfig + + +class DummyProgressBar: + def __init__(self): + pass + + def update(self, num: int): + pass + + def close(self): + pass + + +class _HFBatchReader: + def __init__( + self, + dataset: Dataset, + name: str, + default_batch_size: int, + total_epochs: int = 1, + offset: int = 0, + drop_last: bool = True, + total_steps: Optional[int] = None, + enable_progress_bar: Optional[bool] = True, + ): + self.dataset = dataset + self.dataset_size = len(dataset) + self.name = name + self.current_batch_size = None + self.drop_last = drop_last + + self.current_offset = offset + + # convert epochs/steps to sample number + if total_steps: + self.total_samples = default_batch_size * total_steps + else: + self.total_samples = self.dataset_size * total_epochs + + if enable_progress_bar: + from ray.experimental.tqdm_ray import tqdm + + self.progress_bar = tqdm( + total=self.total_samples, + desc=f"Dataset [{self.name}] Progressing", + ) + else: + self.progress_bar = DummyProgressBar() + + self.progress_bar.update(self.current_offset) + + def read_batch(self, batch_size: int) -> Tuple[List, List]: + batch, indices = [], [] + while len(batch) < batch_size: + if self.current_offset >= self.total_samples: + if not self.drop_last and len(batch) > 0: + break + self.progress_bar.close() + raise StopIteration + index = self.current_offset % self.dataset_size + batch.append(self.dataset[index]) + indices.append(index) + self.current_offset += 1 + + self.progress_bar.update(len(batch)) + return batch, indices + + def select_batch(self, indices: List[int]) -> List: + batch = [] + for i in indices: + assert 0 <= i < self.dataset_size + if self.current_offset >= self.total_samples: + if not self.drop_last and len(batch) > 0: + break + self.progress_bar.close() + raise StopIteration + batch.append(self.dataset[int(i)]) + self.current_offset += 1 + + self.progress_bar.update(len(batch)) # update progress bar + return batch + + +class BaseFileReader(BufferReader): + async def read_async(self, batch_size: Optional[int] = None, **kwargs): + try: + return self.read(batch_size) + except StopIteration as e: + raise StopAsyncIteration from e + + +class FileReader(BaseFileReader): + """Provide a unified interface for Experience and Task file readers.""" + + def __init__(self, config: StorageConfig): + if config.schema_type and config.schema_type != "task": + self.reader = ExperienceFileReader(config) + else: + self.reader = TaskFileReader(config) + + def read(self, batch_size: Optional[int] = None, **kwargs) -> List: + return self.reader.read(batch_size) + + def state_dict(self): + return self.reader.state_dict() + + def load_state_dict(self, state_dict): + return self.reader.load_state_dict(state_dict) + + def feedback(self, **pipeline_metrics): + if self.reader.selector is not None: + self.reader.selector.feedback(**pipeline_metrics) + + def __len__(self): + return self.reader.__len__() + + +class ExperienceFileReader(BaseFileReader): + """Reader for SFT / DPO file data.""" + + def __init__(self, config: StorageConfig): + self.formatter = FORMATTER.get(config.schema_type)( + tokenizer_path=config.tokenizer_path, format_config=config.format + ) + self.read_batch_size = config.batch_size + self.dataset = _HFBatchReader( + load_dataset(config.path, name=config.subset_name, split=config.split), + name=config.name, + default_batch_size=self.read_batch_size, + total_epochs=config.total_epochs, + drop_last=True, + total_steps=config.total_steps, + enable_progress_bar=config.enable_progress_bar, + ) + self.selector = None + + def read(self, batch_size: Optional[int] = None, **kwargs) -> List: + samples, _ = self.dataset.read_batch(batch_size or self.read_batch_size) + exp_list = [] + for sample in samples: + experience = self.formatter.format(sample) + exp_list.append(experience) + return exp_list + + def state_dict(self): + return {"current_index": self.dataset.current_offset} + + def load_state_dict(self, state_dict): + self.dataset.current_offset = state_dict["current_index"] + + def __len__(self): + return self.dataset.dataset_size + + +def _load_task_dataset(config: StorageConfig) -> Dataset: + """Load task dataset from path. Supports both HF dataset and parquet/jsonl file path.""" + path = config.path + split = config.split + if path and str(path).endswith(".parquet"): + return load_dataset( + "parquet", + data_files={split: path}, + split=split, + ) + if path and (str(path).endswith(".jsonl") or str(path).endswith(".json")): + return load_dataset( + "json", + data_files={split: path}, + split=split, + ) + return load_dataset(path, name=config.subset_name, split=split) + + +class TaskFileReader(BaseFileReader): + """A Reader for task file data.""" + + def __init__(self, config: StorageConfig): + self.config = config + self.name = config.name + self.epoch = 0 + datasets.disable_caching() + self.read_batch_size = config.batch_size + self.dataset = _HFBatchReader( + _load_task_dataset(self.config), + name=self.config.name, + default_batch_size=self.read_batch_size, + total_epochs=self.config.total_epochs if not self.config.is_eval else 1, + offset=self.config.index, + drop_last=not self.config.is_eval, + total_steps=self.config.total_steps if not self.config.is_eval else None, + enable_progress_bar=self.config.enable_progress_bar, + ) + self.formatter = FORMATTER.get("task")(config) + if self.config.task_selector is not None: + from trinity.buffer.selector import SELECTORS + from trinity.buffer.selector.selector import BaseSelector + + self.selector: BaseSelector = SELECTORS.get(self.config.task_selector.selector_type)( + self.dataset, self.config.task_selector + ) + else: + self.selector = None + + def _get_tasks(self, samples: List, indices: List) -> List: + tasks = [] + for sample, index in zip(samples, indices): + task = self.formatter.format(sample) + task.index["index"] = int(index) + tasks.append(task) + return tasks + + def read(self, batch_size: Optional[int] = None, **kwargs) -> List: + batch_size = batch_size or self.read_batch_size + if self.selector is not None: + indices = self.selector.get_indices(batch_size) + samples = self.dataset.select_batch(indices) + else: + samples, indices = self.dataset.read_batch(batch_size) + return self._get_tasks(samples, indices) + + def state_dict(self): + if self.selector is not None: + return self.selector.state_dict() + return {"current_index": self.dataset.current_offset} + + def load_state_dict(self, state_dict): + if self.selector is not None: + self.selector.load_state_dict(state_dict) + self.dataset.current_offset = state_dict["current_index"] + + def __len__(self): + return self.dataset.dataset_size diff --git a/frameworks/trinity/buffer/reader/queue_reader.py b/frameworks/trinity/buffer/reader/queue_reader.py new file mode 100644 index 0000000000000000000000000000000000000000..6f8e7c0334334d0071c8e63d0d2743d2947043e2 --- /dev/null +++ b/frameworks/trinity/buffer/reader/queue_reader.py @@ -0,0 +1,49 @@ +"""Reader of the Queue buffer.""" + +from typing import Dict, List, Optional + +import ray + +from trinity.buffer.buffer_reader import BufferReader +from trinity.buffer.storage.queue import QueueStorage +from trinity.common.config import StorageConfig +from trinity.common.constants import StorageType + + +class QueueReader(BufferReader): + """Reader of the Queue buffer.""" + + def __init__(self, config: StorageConfig): + assert config.storage_type == StorageType.QUEUE.value + self.timeout = config.max_read_timeout + self.read_batch_size = config.batch_size + self.queue = QueueStorage.get_wrapper(config) + + def read(self, batch_size: Optional[int] = None, **kwargs) -> List: + try: + batch_size = self.read_batch_size if batch_size is None else batch_size + exps = ray.get(self.queue.get_batch.remote(batch_size, timeout=self.timeout, **kwargs)) + if len(exps) != batch_size: + raise TimeoutError( + f"Read incomplete batch ({len(exps)}/{batch_size}), please check your workflow." + ) + except StopAsyncIteration: + raise StopIteration() + return exps + + async def read_async(self, batch_size: Optional[int] = None, **kwargs) -> List: + batch_size = self.read_batch_size if batch_size is None else batch_size + exps = await self.queue.get_batch.remote(batch_size, timeout=self.timeout, **kwargs) + if len(exps) != batch_size: + raise TimeoutError( + f"Read incomplete batch ({len(exps)}/{batch_size}), please check your workflow." + ) + return exps + + def state_dict(self) -> Dict: + # Queue Not supporting state dict yet + return {"current_index": 0} + + def load_state_dict(self, state_dict): + # Queue Not supporting state dict yet + return None diff --git a/frameworks/trinity/buffer/reader/sql_reader.py b/frameworks/trinity/buffer/reader/sql_reader.py new file mode 100644 index 0000000000000000000000000000000000000000..fd1425bb8f16315910334ab75ca68071cfd0c55c --- /dev/null +++ b/frameworks/trinity/buffer/reader/sql_reader.py @@ -0,0 +1,45 @@ +"""Reader of the SQL buffer.""" + +from typing import Dict, List, Optional + +import ray + +from trinity.buffer.buffer_reader import BufferReader +from trinity.buffer.storage.sql import SQLStorage +from trinity.common.config import StorageConfig +from trinity.common.constants import StorageType + + +class SQLReader(BufferReader): + """Reader of the SQL buffer.""" + + def __init__(self, config: StorageConfig) -> None: + assert config.storage_type == StorageType.SQL.value + self.wrap_in_ray = config.wrap_in_ray + self.read_batch_size = config.batch_size + self.storage = SQLStorage.get_wrapper(config) + + def read(self, batch_size: Optional[int] = None, **kwargs) -> List: + batch_size = self.read_batch_size if batch_size is None else batch_size + if self.wrap_in_ray: + return ray.get(self.storage.read.remote(batch_size, **kwargs)) + else: + return self.storage.read(batch_size, **kwargs) + + async def read_async(self, batch_size: Optional[int] = None, **kwargs) -> List: + batch_size = self.read_batch_size if batch_size is None else batch_size + if self.wrap_in_ray: + try: + return await self.storage.read.remote(batch_size, **kwargs) + except StopIteration: + raise StopAsyncIteration + else: + return self.storage.read(batch_size, **kwargs) + + def state_dict(self) -> Dict: + # SQL Not supporting state dict yet + return {"current_index": 0} + + def load_state_dict(self, state_dict): + # SQL Not supporting state dict yet + return None diff --git a/frameworks/trinity/buffer/schema/__init__.py b/frameworks/trinity/buffer/schema/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..f9f60f3bcd4070af5df5273ee9650a95e6aa0c65 --- /dev/null +++ b/frameworks/trinity/buffer/schema/__init__.py @@ -0,0 +1,23 @@ +from trinity.buffer.schema.sql_schema import init_engine +from trinity.utils.registry import Registry + +FORMATTER: Registry = Registry( + "formatter", + { + "task": "trinity.buffer.schema.formatter.TaskFormatter", + "sft": "trinity.buffer.schema.formatter.SFTFormatter", + "dpo": "trinity.buffer.schema.formatter.DPOFormatter", + }, +) + +SQL_SCHEMA: Registry = Registry( + "sql_schema", + { + "task": "trinity.buffer.schema.sql_schema.TaskModel", + "experience": "trinity.buffer.schema.sql_schema.ExperienceModel", + "sft": "trinity.buffer.schema.sql_schema.SFTDataModel", + "dpo": "trinity.buffer.schema.sql_schema.DPODataModel", + }, +) + +__all__ = ["init_engine", "FORMATTER", "SQL_SCHEMA"] diff --git a/frameworks/trinity/buffer/schema/formatter.py b/frameworks/trinity/buffer/schema/formatter.py new file mode 100644 index 0000000000000000000000000000000000000000..039074521b5a541fe3b9dc9f7995971319fd97d5 --- /dev/null +++ b/frameworks/trinity/buffer/schema/formatter.py @@ -0,0 +1,386 @@ +import json +from abc import ABC, abstractmethod +from typing import Dict, List, Optional + +import transformers + +from trinity.common.config import FormatConfig, StorageConfig +from trinity.common.constants import PromptType +from trinity.common.experience import Experience +from trinity.common.models.utils import get_action_mask_method +from trinity.common.rewards import REWARD_FUNCTIONS +from trinity.common.workflows import WORKFLOWS, Task +from trinity.utils.log import get_logger + + +class ExperienceFormatter(ABC): + @abstractmethod + def format(self, sample: Dict) -> Experience: + """Format a raw sample dict into an experience.""" + + +class TaskFormatter: + """Formatter for task data. + + Example Input: + + .. code-block:: python + + { + "input": "Hello", + "output": "Hi" + } + """ + + def __init__(self, config: StorageConfig): + self.config = config + self.default_workflow_cls = WORKFLOWS.get(config.default_workflow_type) # type: ignore + self.default_reward_fn_cls = REWARD_FUNCTIONS.get(config.default_reward_fn_type) # type: ignore + self.workflow_key = config.format.workflow_key + self.reward_fn_key = config.format.reward_fn_key + + def format(self, sample: Dict) -> Task: + """Format a raw sample dict into a Task.""" + + workflow_name = sample.get(self.workflow_key, None) if self.workflow_key else None + reward_fn_name = sample.get(self.reward_fn_key, None) if self.reward_fn_key else None + + workflow_cls = ( + WORKFLOWS.get(workflow_name) if workflow_name else None + ) or self.default_workflow_cls + reward_fn_cls = ( + REWARD_FUNCTIONS.get(reward_fn_name) if reward_fn_name else None + ) or self.default_reward_fn_cls + assert workflow_cls is not None, "`default_workflow_type` or `workflow_key` is required" + return Task( + workflow=workflow_cls, + reward_fn=reward_fn_cls, + format_args=self.config.format, + repeat_times=self.config.repeat_times, + rollout_args=self.config.rollout_args, + workflow_args=self.config.workflow_args, + reward_fn_args=self.config.reward_fn_args, + is_eval=self.config.is_eval, + raw_task=sample, + ) + + +class SFTFormatter(ExperienceFormatter): + """Formatter for SFT data, supporting both message list and plaintext formats. + + Uses format_config.prompt_type to distinguish between 'messages' and 'plaintext'. + + Example input of MESSAGES: + + .. code-block:: python + + { + "messages": [ + {"role": "user", "content": "Hello, how are you?"}, + {"role": "assistant", "content": "I'm fine, thank you!"} + ] + } + + + Example input of PLAINTEXT: + + .. code-block:: python + + { + "system_prompt_key": "system", + "prompt_key": "prompt", + "response_key": "response", + } + """ + + def __init__(self, tokenizer_path: str, format_config: FormatConfig): + self.logger = get_logger("sft_dataset_formatter", in_ray_actor=True) + self.prompt_type = format_config.prompt_type + self.enable_concatenated_multi_turn = format_config.enable_concatenated_multi_turn + self.tools_key = format_config.tools_key + self.image_key = format_config.image_key + self.video_key = format_config.video_key + if self.image_key is not None or self.video_key is not None: + assert ( + self.enable_concatenated_multi_turn is False + ), "Concatenated multi-turn not supported for multi-modal data yet." + self.processor = transformers.AutoProcessor.from_pretrained(tokenizer_path) + self.tokenizer = self.processor.tokenizer + else: + self.processor = None + self.tokenizer = transformers.AutoTokenizer.from_pretrained(tokenizer_path) + self.chat_template = format_config.chat_template or self.tokenizer.chat_template + # For messages type + if self.prompt_type == PromptType.MESSAGES: + self.messages_key = format_config.messages_key + if format_config.enable_concatenated_multi_turn: + self.action_mask_method = get_action_mask_method(self.chat_template) + # For plaintext type + elif self.prompt_type == PromptType.PLAINTEXT: + self.prompt_key = format_config.prompt_key + self.response_key = format_config.response_key + self.system_prompt_key = format_config.system_prompt_key + self.system_prompt = format_config.system_prompt + self.tools_key = format_config.tools_key + else: + raise ValueError(f"Unsupported prompt_type: {self.prompt_type}") + + def _messages_to_experience( + self, + messages: List[Dict], + tools: Optional[List[Dict] | str] = None, + mm_data: Optional[Dict] = None, + ) -> Experience: + """Convert messages and tools into an Experience object. + + Args: + messages (List[Dict]): The list of message dictionaries. + tools (Optional[List[Dict]|str], optional): The list of tool dictionaries or a JSON string. Defaults to None. + mm_data (Optional[Dict], optional): Multi-modal data such as images or videos. Defaults to None. + + Returns: + Experience: The resulting Experience object. + """ + if tools is not None and isinstance(tools, list): + self.logger.warning( + "[SFT Data Warning] 'tools' is provided as a list of dictionaries. " + "When loading with Huggingface Datasets, schema auto-alignment may set unmatched fields to null, " + "potentially causing undesired behavior. " + "It is recommended to pre-process 'tools' objects with json.dumps before saving/loading, " + "and to restore them with json.loads in this function." + ) + if isinstance(tools, str): + try: + tools = json.loads(tools) + except json.JSONDecodeError: + self.logger.error( + "[SFT Data Error] Failed to decode 'tools' JSON. Please check your data format." + ) + raise ValueError("Invalid JSON format for tools") + if self.enable_concatenated_multi_turn: + token_ids, action_mask, prompt_length = self.action_mask_method( + tokenizer=self.tokenizer, + messages=messages, + tools=tools, + chat_template=self.chat_template, + ) + return Experience( + tokens=token_ids, + action_mask=action_mask[prompt_length:], + prompt_length=prompt_length, + messages=messages, + ) + if mm_data: + return self.convert_mm_data_to_experiences(messages=messages, mm_data=mm_data) + token_ids = self.tokenizer.apply_chat_template( + messages, + tools=tools, + add_generation_prompt=False, + return_tensors="pt", + chat_template=self.chat_template, + )[0] + prompt_tokens_ids = self.tokenizer.apply_chat_template( + messages[:-1], + tools=tools, + add_generation_prompt=True, + return_tensors="pt", + chat_template=self.chat_template, + )[0] + return Experience( + tokens=token_ids, + prompt_length=len(prompt_tokens_ids), + messages=messages, + ) + + def load_mm_data(self, sample: Dict) -> Dict: + """Load multi-modal data such as images or videos. + + NOTE: You can override this method for custom data loading. + + Args: + sample (Dict): The raw sample dictionary containing multi-modal data. + + Returns: + Dict: A dictionary containing multi-modal data. Specifically, it may contain: + - images: A list of `PIL.Image.Image` if `self.image_key` is set + - videos: A list of `numpy.ndarray` if `self.video_key` is set + """ + from verl.utils.dataset.vision_utils import process_image, process_video + + mm_data = {} + if self.image_key: + mm_data["images"] = [process_image(img) for img in sample[self.image_key]] + if self.video_key: + mm_data["videos"] = [process_video(vid).numpy() for vid in sample[self.video_key]] + return mm_data + + def convert_mm_data_to_experiences( + self, + messages: List[Dict], + mm_data: Dict, + ) -> Experience: + from trinity.common.models.mm_utils import ( + build_multi_modal_inputs, + convert_messages_to_mm_format, + ) + + messages = convert_messages_to_mm_format(messages) + sequence: str = self.processor.apply_chat_template( + messages, + add_generation_prompt=False, + chat_template=self.chat_template, + ) + prompt: str = self.processor.apply_chat_template( + messages[:-1], + add_generation_prompt=True, + chat_template=self.chat_template, + ) + sequence_data = build_multi_modal_inputs( + prompt=sequence, + images=mm_data.get("images", None), + videos=mm_data.get("videos", None), + processor=self.processor, + ) + prompt_data = build_multi_modal_inputs( + prompt=prompt, + images=mm_data.get("images", None), + videos=mm_data.get("videos", None), + processor=self.processor, + ) + return Experience( + tokens=sequence_data["prompt_token_ids"], + prompt_length=len(prompt_data["prompt_token_ids"]), + messages=messages, + multi_modal_inputs=sequence_data["multi_modal_inputs"], + ) + + def format(self, sample: Dict) -> Experience: + if self.prompt_type == PromptType.MESSAGES: + messages = sample[self.messages_key] + # load messages from json string if needed + if isinstance(messages, str): + try: + messages = json.loads(messages) + except json.JSONDecodeError: + self.logger.error( + "[SFT Data Error] Failed to decode 'messages' JSON. please check your data format." + ) + raise ValueError("Invalid JSON format for messages") + elif self.prompt_type == PromptType.PLAINTEXT: + messages = [] + if self.system_prompt_key is not None: + system_message = {"role": "system", "content": sample[self.system_prompt_key]} + messages.append(system_message) + elif self.system_prompt is not None: + system_message = {"role": "system", "content": self.system_prompt} + messages.append(system_message) + messages.append({"role": "user", "content": sample[self.prompt_key]}) + messages.append({"role": "assistant", "content": sample[self.response_key]}) + else: + raise ValueError(f"Unsupported prompt_type: {self.prompt_type}") + tools = sample.get(self.tools_key, None) + mm_data = self.load_mm_data(sample) if self.image_key or self.video_key else None + return self._messages_to_experience(messages, tools, mm_data) + + +class DPOFormatter(ExperienceFormatter): + """Formatter for DPO plaintext data. + + Example Input for PLAINTEXT: + + .. code-block:: python + + { + "prompt": "What is your name?", + "chosen": "My name is Assistant.", + "rejected": "I don't have a name." + } + + Example Input for MESSAGES: + + .. code-block:: python + + { + "messages": [ + {"role": "user", "content": "What is your name?"}, + ], + "chosen": [ + {"role": "assistant", "content": "My name is Assistant."}, + ], + "rejected": [ + {"role": "assistant", "content": "I don't have a favorite color."} + ] + } + """ + + def __init__(self, tokenizer_path: str, format_config: FormatConfig): + self.tokenizer = transformers.AutoTokenizer.from_pretrained(tokenizer_path) + self.prompt_type = format_config.prompt_type + self.chat_template = format_config.chat_template + if self.prompt_type == PromptType.PLAINTEXT: + self.prompt_key = format_config.prompt_key + self.chosen_key = format_config.chosen_key + self.rejected_key = format_config.rejected_key + self.system_prompt_key = format_config.system_prompt_key + self.system_prompt = format_config.system_prompt + elif self.prompt_type == PromptType.MESSAGES: + self.messages_key = format_config.messages_key + self.chosen_key = format_config.chosen_key + self.rejected_key = format_config.rejected_key + else: + raise ValueError(f"Unsupported prompt_type: {self.prompt_type}") + # currently DPO not support tools + + def _messages_to_experience( + self, prompt_messages, chosen_messages, rejected_messages + ) -> Experience: + prompt_tokens = self.tokenizer.apply_chat_template( + prompt_messages, + add_generation_prompt=True, + return_tensors="pt", + chat_template=self.chat_template, + )[0] + chosen_tokens = self.tokenizer.apply_chat_template( + prompt_messages + chosen_messages, + add_generation_prompt=False, + return_tensors="pt", + chat_template=self.chat_template, + )[0][len(prompt_tokens) :] + rejected_tokens = self.tokenizer.apply_chat_template( + prompt_messages + rejected_messages, + add_generation_prompt=False, + return_tensors="pt", + chat_template=self.chat_template, + )[0][len(prompt_tokens) :] + return Experience( + tokens=prompt_tokens, + prompt_length=len(prompt_tokens), + chosen=chosen_tokens, + rejected=rejected_tokens, + chosen_messages=prompt_messages + chosen_messages, + rejected_messages=prompt_messages + rejected_messages, + ) + + def format(self, sample: Dict) -> Experience: + if self.prompt_type == PromptType.PLAINTEXT: + messages = [] + if self.system_prompt_key is not None: + system_message = {"role": "system", "content": sample[self.system_prompt_key]} + messages.append(system_message) + elif self.system_prompt is not None: + system_message = {"role": "system", "content": self.system_prompt} + messages.append(system_message) + messages.append({"role": "user", "content": sample[self.prompt_key]}) + chosen = [{"role": "assistant", "content": sample[self.chosen_key]}] + rejected = [{"role": "assistant", "content": sample[self.rejected_key]}] + elif self.prompt_type == PromptType.MESSAGES: + messages = sample[self.messages_key] + chosen = sample[self.chosen_key] + rejected = sample[self.rejected_key] + else: + raise ValueError(f"Unsupported prompt_type: {self.prompt_type}") + return self._messages_to_experience( + prompt_messages=messages, + chosen_messages=chosen, + rejected_messages=rejected, + ) diff --git a/frameworks/trinity/buffer/schema/sql_schema.py b/frameworks/trinity/buffer/schema/sql_schema.py new file mode 100644 index 0000000000000000000000000000000000000000..7f22e5370e2594b998f1cdbbe2d7912b9e2d6716 --- /dev/null +++ b/frameworks/trinity/buffer/schema/sql_schema.py @@ -0,0 +1,149 @@ +"""SQLAlchemy models for different data.""" + +from typing import Dict, Optional, Tuple + +from sqlalchemy import ( + JSON, + Column, + DateTime, + Float, + Integer, + LargeBinary, + String, + create_engine, + func, +) +from sqlalchemy.exc import OperationalError +from sqlalchemy.orm import declarative_base +from sqlalchemy.pool import NullPool + +from trinity.common.experience import Experience +from trinity.utils.log import get_logger + +Base = declarative_base() + + +class TaskModel(Base): # type: ignore + """Model for storing tasks in SQLAlchemy.""" + + __abstract__ = True + + id = Column(Integer, primary_key=True, autoincrement=True) + raw_task = Column(JSON, nullable=False) + + @classmethod + def from_dict(cls, dict: Dict): + return cls(raw_task=dict) + + +class ExperienceModel(Base): # type: ignore + """SQLAlchemy model for Experience.""" + + __abstract__ = True + + id = Column(Integer, primary_key=True, autoincrement=True) + timestamp = Column(DateTime, server_default=func.now()) + task_id = Column(String(64), nullable=True, index=True) # associated task id + run_id = Column(Integer, nullable=True, index=True) # associated run id + msg_id = Column(String(64), nullable=True, index=True) # associated message id + # serialized experience object + model_version = Column(Integer, nullable=True, index=True) + experience_bytes = Column(LargeBinary, nullable=True) + reward = Column(Float, nullable=True) + consumed = Column(Integer, default=0, index=True) + + def to_experience(self) -> Experience: + """Load the experience from the database.""" + exp = Experience.deserialize(self.experience_bytes) + exp.eid.task = self.task_id + exp.eid.run = self.run_id + exp.eid.suffix = self.msg_id + exp.reward = self.reward + exp.info["model_version"] = self.model_version + return exp + + @classmethod + def from_experience(cls, experience: Experience): + """Save the experience to database.""" + return cls( + experience_bytes=experience.serialize(), + reward=experience.reward, + task_id=str(experience.eid.task), + run_id=experience.eid.run, + msg_id=str(experience.eid.suffix), + model_version=experience.info["model_version"], + ) + + +class SFTDataModel(Base): # type: ignore + """SQLAlchemy model for SFT data.""" + + __abstract__ = True + + id = Column(Integer, primary_key=True, autoincrement=True) + message_list = Column(JSON, nullable=True) + experience_bytes = Column(LargeBinary, nullable=True) + + def to_experience(self) -> Experience: + """Load the experience from the database.""" + return Experience.deserialize(self.experience_bytes) + + @classmethod + def from_experience(cls, experience: Experience): + """Save the experience to database.""" + return cls( + experience_bytes=experience.serialize(), + message_list=experience.messages, + ) + + +class DPODataModel(Base): # type: ignore + """SQLAlchemy model for DPO data.""" + + __abstract__ = True + + id = Column(Integer, primary_key=True, autoincrement=True) + chosen_message_list = Column(JSON, nullable=True) + rejected_message_list = Column(JSON, nullable=True) + experience_bytes = Column(LargeBinary, nullable=True) + + def to_experience(self) -> Experience: + """Load the experience from the database.""" + return Experience.deserialize(self.experience_bytes) + + @classmethod + def from_experience(cls, experience: Experience): + """Save the experience to database.""" + return cls( + experience_bytes=experience.serialize(), + chosen_message_list=experience.chosen_messages, + rejected_message_list=experience.rejected_messages, + ) + + +def init_engine(db_url: str, table_name: str, schema_type: Optional[str]) -> Tuple: + """Get the sqlalchemy engine.""" + logger = get_logger(__name__) + engine = create_engine(db_url, poolclass=NullPool) + + if schema_type is None: + schema_type = "task" + + from trinity.buffer.schema import SQL_SCHEMA + + base_class = SQL_SCHEMA.get(schema_type) + + table_attrs = { + "__tablename__": table_name, + "__abstract__": False, + "__table_args__": {"keep_existing": True}, + } + table_cls = type(table_name, (base_class,), table_attrs) + + try: + Base.metadata.create_all(engine, checkfirst=True) + logger.info(f"Created table {table_name} for schema type {schema_type}.") + except OperationalError: + logger.warning(f"Failed to create table {table_name}, assuming it already exists.") + + return engine, table_cls diff --git a/frameworks/trinity/buffer/selector/__init__.py b/frameworks/trinity/buffer/selector/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..3e69c615aab4e6ad37101417339ae7e24751b8d6 --- /dev/null +++ b/frameworks/trinity/buffer/selector/__init__.py @@ -0,0 +1,18 @@ +from trinity.buffer.selector.selector import BaseSelector +from trinity.utils.registry import Registry + +SELECTORS = Registry( + "selectors", + default_mapping={ + "sequential": "trinity.buffer.selector.selector.SequentialSelector", + "shuffle": "trinity.buffer.selector.selector.ShuffleSelector", + "random": "trinity.buffer.selector.selector.RandomSelector", + "offline_easy2hard": "trinity.buffer.selector.selector.OfflineEasy2HardSelector", + "difficulty_based": "trinity.buffer.selector.selector.DifficultyBasedSelector", + }, +) + +__all__ = [ + "BaseSelector", + "SELECTORS", +] diff --git a/frameworks/trinity/buffer/selector/difficulty_estimator.py b/frameworks/trinity/buffer/selector/difficulty_estimator.py new file mode 100644 index 0000000000000000000000000000000000000000..97c4ab61e135849d5357a355c7546ae545a7e590 --- /dev/null +++ b/frameworks/trinity/buffer/selector/difficulty_estimator.py @@ -0,0 +1,119 @@ +from typing import List + +import numpy as np + +from trinity.utils.log import get_logger + + +class BaseBetaPREstimator: + n: int + m: int + lamb: float + rho: float + alphas: np.ndarray + betas: np.ndarray + + def __init__(self, n: int, m: int = 16, lamb: float = 0.2, rho: float = 0.2): + """ + alpha_{t+1} = (1 - lamb) * alpha_t + (1 - rho) * bar{s} + rho * tilde{s} + beta_{t+1} = (1 - lamb) beta_t + (1 - rho) * bar{f} + rho * tilde{f} + + Args: + n (int): number of tasks. + m (int): repeat times per tasks. + timeout (lamb): discount factor of historical estimation. + rho (float): weight of pseudo counts. + """ + self.n = n + self.m = m + self.lamb = lamb + self.rho = rho + self.alphas = np.ones(n, dtype=float) + self.betas = np.ones(n, dtype=float) + self.logger = get_logger("BetaPREstimator") + self.logger.debug( + f"{self.n=}, {self.m=}, {self.lamb=}, {self.rho=}, {self.alphas=}, {self.betas=}" + ) + + def set(self, alphas, betas): + self.alphas = alphas + self.betas = betas + + def _update(self, s_bar, f_bar, p_tilde): + self.alphas = ( + (1 - self.lamb) * self.alphas + + self.lamb + + (1 - self.rho) * s_bar + + self.rho * p_tilde * self.m + ) + self.betas = ( + (1 - self.lamb) * self.betas + + self.lamb + + (1 - self.rho) * f_bar + + self.rho * (1 - p_tilde) * self.m + ) + + def update(self, ref_indices: List[int], ref_pass_rates: List[float]): + raise NotImplementedError + + def predict_pr(self, rng=None, indices=None, do_sample=False): + if rng is None: + rng = np.random.default_rng() + if indices is None: + indices = np.arange(self.n) + if not do_sample: + return self.alphas[indices] / (self.alphas[indices] + self.betas[indices]) + else: + return rng.beta(self.alphas[indices], self.betas[indices]) + + def equivalent_count(self, indices=None): + if indices is None: + indices = np.arange(self.n) + return self.alphas[indices] + self.betas[indices] + + +class InterpolationBetaPREstimator(BaseBetaPREstimator): + def __init__( + self, + features: np.ndarray, + m: int, + lamb, + rho, + cap_coef_update_discount=0.9, + adaptive_rho=False, + ): + super(InterpolationBetaPREstimator, self).__init__(len(features), m, lamb, rho) + self.features = features # [D, 2] + self.cap_coef = None + self.cap_coef_update_discount = cap_coef_update_discount + self.adaptive_rho = adaptive_rho + + def update(self, ref_indices: List[int], ref_pass_rates: List[float]): + ref_pass_rate = np.mean(ref_pass_rates) + ref_anchor_pass_rates = np.mean(self.features[ref_indices], axis=0) + cap_estimate = (ref_pass_rate - ref_anchor_pass_rates[0]) / ( + ref_anchor_pass_rates[1] - ref_anchor_pass_rates[0] + 1e-6 + ) + if self.cap_coef is None: + self.cap_coef = cap_estimate + else: + self.cap_coef = ( + self.cap_coef_update_discount * self.cap_coef + + (1 - self.cap_coef_update_discount) * cap_estimate + ) + s_bar = np.zeros(self.n, dtype=float) + s_bar[ref_indices] = np.array(ref_pass_rates) * self.m + f_bar = np.zeros(self.n, dtype=float) + f_bar[ref_indices] = (1 - np.array(ref_pass_rates)) * self.m + p_tilde = np.clip( + (self.features[:, 1] - self.features[:, 0]) * self.cap_coef + self.features[:, 0], 0, 1 + ) + + predicted_pass_rates = p_tilde[ref_indices] + mean_abs_error = np.mean(np.abs(np.array(predicted_pass_rates) - np.array(ref_pass_rates))) + if self.adaptive_rho and mean_abs_error >= 0.25: + self.rho = self.rho * 0.5 + self.logger.debug(f"{mean_abs_error=}, {self.rho=}") + p_tilde[ref_indices] = np.array(ref_pass_rates) + + self._update(s_bar, f_bar, p_tilde) diff --git a/frameworks/trinity/buffer/selector/selector.py b/frameworks/trinity/buffer/selector/selector.py new file mode 100644 index 0000000000000000000000000000000000000000..a67036dd4538780eaba8ffac0e8d96ab50ad0450 --- /dev/null +++ b/frameworks/trinity/buffer/selector/selector.py @@ -0,0 +1,422 @@ +"""Data selectors.""" +from typing import Dict, List + +import numpy as np +import torch + +from trinity.buffer.reader.file_reader import _HFBatchReader +from trinity.buffer.selector.difficulty_estimator import InterpolationBetaPREstimator +from trinity.common.config import TaskSelectorConfig +from trinity.utils.annotations import Experimental +from trinity.utils.log import get_logger + + +@Experimental +class BaseSelector: + """ + Abstract base class defining the interface for custom data selection strategies. + + A selector determines which samples (by index) are selected from the dataset + during training. It enables flexible sampling beyond simple + sequential or random access, supporting active learning, curriculum learning, + or difficulty-based sampling in the future. + + Subclasses must implement: + - get_indices: returns list of indices for next batch + - update: updates internal state using feedback (e.g., loss values, mean rewards, etc.) + - state_dict / load_state_dict: for saving/loading selector state (checkpointing) + """ + + def __init__(self, data_source: _HFBatchReader, config: TaskSelectorConfig): + self.data_source = data_source + self.config = config + + def get_indices(self, batch_size: int, return_extra_info: bool = False) -> List[int]: + """ + Select a batch of sample indices from the dataset. + + Args: + batch_size (int): Number of indices to return + return_extra_info (bool): If True, may return additional metadata (future use) + + Returns: + List[int]: Selected indices into the dataset + """ + raise NotImplementedError + + def feedback(self, indices: List[int], values: List[float]) -> None: + """ + Update internal state based on feedback (e.g., model loss, accuracy). + + This allows adaptive selectors (like hard example mining) to learn over time. + + Args: + indices (List[int]): Previously selected indices + values (List[float]): Feedback values corresponding to those indices + """ + raise NotImplementedError + + def state_dict(self) -> Dict: + """ + Return serializable state of the selector for checkpointing. + + Returns: + Dict: State information (e.g., current position, etc.) + """ + raise NotImplementedError + + def load_state_dict(self, state_dict: Dict) -> None: + """ + Restore selector state from a saved dictionary. + + Args: + state_dict (Dict): Output from state_dict() + """ + raise NotImplementedError + + +class SequentialSelector(BaseSelector): + """ + Selects data sequentially in fixed order across epochs. + + Example: [0,1,2,...,B-1], then [B,B+1,...,2B-1], etc. + """ + + def __init__(self, data_source: _HFBatchReader, config: TaskSelectorConfig): + super().__init__(data_source, config) + self.dataset_size = data_source.dataset_size + self.current_index = 0 + + def get_indices(self, batch_size: int, return_extra_info: bool = False) -> List[int]: + start = self.current_index % self.dataset_size + end = start + batch_size + self.current_index += batch_size + if end <= self.dataset_size: + return list(range(start, end)) + return list(range(start, self.dataset_size)) + list(range(0, end - self.dataset_size)) + + def feedback(self, indices: List[int], values: List[float]) -> None: + # No-op: sequential selection doesn't adapt based on feedback + pass + + def state_dict(self) -> Dict: + return { + "current_index": self.current_index, + } + + def load_state_dict(self, state_dict): + self.current_index = state_dict.get("current_index", 0) + + +class ShuffleSelector(BaseSelector): + """ + Shuffles dataset once per epoch and iterates through it sequentially. + + Each epoch uses a different permutation of a subset of the full dataset. + When one epoch ends, a new shuffle is triggered. + Mimics standard PyTorch DataLoader with shuffle=True. + """ + + def __init__(self, data_source: _HFBatchReader, config: TaskSelectorConfig): + super().__init__(data_source, config) + self.dataset_size = data_source.dataset_size # Total available samples + self.current_index = 0 # Progress tracker + self.seed = config.seed # For reproducible shuffling + self.orders = self._get_orders() # Current shuffled index order + + def _get_orders(self) -> List[int]: + """ + Generate a new shuffled order for the current epoch. + + Uses NumPy's PCG64 random generator seeded by epoch number for reproducibility. + Ensures different shuffle per epoch while being deterministic if seed is fixed. + """ + rng = np.random.default_rng(self.seed + self.current_index // self.dataset_size) + return rng.permutation(self.dataset_size).tolist() + + def get_indices(self, batch_size: int, return_extra_info: bool = False) -> List[int]: + start = self.current_index % self.dataset_size + end = start + batch_size + if end <= self.dataset_size: + ret = self.orders[start:end] + # At end of epoch, reshuffle for next epoch + if end == self.dataset_size: + self.orders = self._get_orders() + else: + ret = self.orders[start:] + # At end of epoch, reshuffle for next epoch + self.orders = self._get_orders() + ret += self.orders[: (end - self.dataset_size)] + self.current_index += batch_size + return ret + + def feedback(self, indices: List[int], values: List[float]) -> None: + # No-op: static shuffling does not adapt + pass + + def state_dict(self) -> Dict: + return { + "current_index": self.current_index, + } + + def load_state_dict(self, state_dict): + self.current_index = state_dict.get("current_index", 0) + self.orders = self._get_orders() + + +class RandomSelector(BaseSelector): + """ + Uniformly samples batches randomly with replacement *per batch*. + + Unlike ShuffleSelector, there is no concept of an epoch — every batch is independently sampled. + Can result in repeated samples within an epoch. Suitable for online or stochastic training regimes. + """ + + def __init__(self, data_source: _HFBatchReader, config: TaskSelectorConfig): + super().__init__(data_source, config) + self.dataset_size = data_source.dataset_size + self.current_index = 0 + self.seed = config.seed + + def get_indices(self, batch_size, return_extra_info=False): + # Seed varies per batch to ensure repeatability across runs + rng = np.random.default_rng(self.seed + self.current_index) + selected_indices = rng.choice(self.dataset_size, batch_size, replace=False) + self.current_index += batch_size + if return_extra_info: + return selected_indices, {} + else: + return selected_indices + + def feedback(self, indices: List[int], values: List[float]) -> None: + # No-op: basic random selection doesn't adapt + pass + + def state_dict(self) -> Dict: + return { + "current_index": self.current_index, + } + + def load_state_dict(self, state_dict): + self.current_index = state_dict.get("current_index", 0) + + +class OfflineEasy2HardSelector(BaseSelector): + """ + Selects samples in an 'easy-to-hard' curriculum based on pre-defined difficulty features. + + This selector assumes that higher feature values indicate easier examples. + It sorts all data once at initialization by descending feature value(s), then sequentially + serves batches from easy → hard over epochs. The sorting is fixed (offline), so no online + adaptation occurs during training. + + Useful for curriculum learning where sample difficulty is estimated ahead of time + (e.g., via teacher model confidence, length, BLEU score, etc.). + """ + + def __init__(self, data_source, config: TaskSelectorConfig): + super().__init__(data_source, config) + self.logger = get_logger("offline_easy2hard_selector") + + # Extract specified feature columns (e.g., 'loss', 'confidence') used to estimate difficulty + feature_keys = config.feature_keys + self.features = np.concatenate( + [np.array(list(data_source.dataset[k]))[:, None] for k in feature_keys], axis=1 + ) + # Shape: (N, len(feature_keys)) — one row per sample, one column per feature + + # Append index to each feature vector for tracking original positions after sorting + features_with_index = [list(self.features[i]) + [i] for i in range(len(self.features))] + + # Sort by feature values in descending order → highest (easiest) first + features_with_index = sorted(features_with_index)[::-1] + self.logger.debug(f"OfflineEasy2HardSelector, sorted {features_with_index[:20]}") + + # Store the sorted order of indices (from easiest to hardest) + self.sorted_index = np.array([i[-1] for i in features_with_index]) + + # Number of samples per epoch (may be less than full dataset size) + self.dataset_size = data_source.dataset_size + self.current_index = 0 + + def feedback(self, indices: List[int], values: List[float]) -> None: + # No-op: this selector does not adapt based on runtime feedback + pass + + def get_indices(self, batch_size, return_extra_info=False): + """ + Returns next batch of indices in curriculum order (easy → hard). + + Batches are taken sequentially from the pre-sorted list. When epoch ends, + it wraps around to the beginning (i.e., restarts curriculum). + """ + start = self.current_index % self.dataset_size + end = start + batch_size + if end <= self.dataset_size: + selected_indices = self.sorted_index[start:end] + else: + selected_indices = np.concatenate( + [self.sorted_index[start:], self.sorted_index[: (end - self.dataset_size)]] + ) + self.current_index += batch_size + if not return_extra_info: + return selected_indices + else: + extra_info = { + "indices": selected_indices.tolist(), + "feat1": self.features[selected_indices, 0].tolist(), + "feat2": self.features[selected_indices, 1].tolist(), + } + return selected_indices, extra_info + + def state_dict(self) -> Dict: + """ + Save current position in the curriculum for checkpointing. + Allows resuming from same point in the easy→hard progression. + """ + return { + "current_index": self.current_index, + } + + def load_state_dict(self, state_dict): + """ + Restore progress through the curriculum from saved state. + """ + self.current_index = state_dict.get("current_index", 0) + + +class DifficultyBasedSelector(BaseSelector): + """ + Adaptive difficulty-based selector using probabilistic modeling of sample difficulty. + + Uses `InterpolationBetaPREstimator` to model each sample's probability of success (PR), + updated with observed feedback (e.g., loss, accuracy). Then selects samples close to + a target reward (e.g., 1.0 for perfect performance), implementing a form of + *targeted difficulty sampling* — focusing on items near the edge of model capability. + + Supports both greedy selection (`tau=0`) and stochastic sampling (`tau>0`). + """ + + def __init__(self, data_source, config: TaskSelectorConfig) -> None: + super().__init__(data_source, config) + self.logger = get_logger("difficulty_based_selector") + + # Initialize difficulty estimator using two features (assumed: e.g., correctness & uncertainty) + self.diff_estimator = self.build_diff_estimator( + data_source.dataset, config.feature_keys, config.kwargs + ) + self.current_index = 0 + self.seed = config.seed + + self.do_sample = config.kwargs.get( + "do_sample", False + ) # Whether to sample PR during estimation + self.target_reward = config.kwargs.get("target_reward", 1.0) # Desired performance level + self.tau = config.kwargs.get("tau", 1.0) # Temperature for sampling distribution + + def build_diff_estimator(self, dataset, feature_keys: List[str], config: dict): + """ + Constructs a Beta-distribution-based difficulty estimator from features. + + Expects exactly two feature keys (e.g., ['correct', 'uncertainty']), which are concatenated + into a feature matrix and passed to InterpolationBetaPREstimator for modeling P(success). + """ + self.logger.debug(f"{config=}") + if len(feature_keys) != 2: + raise ValueError( + f"DifficultyBasedSelector requires exactly 2 feature keys, but got {len(feature_keys)}." + ) + features = np.concatenate( + [np.array(list(dataset[k]))[:, None] for k in feature_keys], axis=1 + ) + self.logger.debug(f"{features.shape=}") + self.logger.debug(f"{features[:5]=}") + adaptive_rho = config.get("adaptive_rho", False) + return InterpolationBetaPREstimator( + features=features, + m=config.get("m", 16), + lamb=config.get("lamb", 0.2), + rho=config.get("rho", 0.2), + adaptive_rho=adaptive_rho, + ) + + def feedback(self, indices: List[int], values: List[float]) -> None: + """ + Updates the difficulty estimator with observed performance on selected samples. + + Args: + indices (List[int]): Previously selected sample indices + values (List[float]): Observed rewards/scores (e.g., accuracy, BLEU) for those samples + """ + self.diff_estimator.update(indices, values) + + def get_scores(self) -> List[float]: + """ + Computes selection scores: negative distance between predicted PR and target reward. + + Samples whose predicted performance is closest to `target_reward` receive highest scores. + Encourages selection of "just right" difficulty samples (neither too easy nor too hard). + """ + rng = np.random.default_rng(self.seed + self.current_index) + predicted_pr = self.diff_estimator.predict_pr(rng=rng, do_sample=self.do_sample) + scores = -np.abs(self.target_reward - predicted_pr) + return scores + + def get_indices(self, batch_size, return_extra_info=False): + """ + Selects batch of indices based on difficulty proximity to target. + + If tau == 0: take top-k highest scoring samples (greedy). + Else: sample stochastically using softmax(logits / tau). + """ + sampling_scores = self.get_scores() + sampling_scores = torch.from_numpy(sampling_scores) + if self.tau == 0: + selected_indices = torch.topk(sampling_scores, batch_size).indices + else: + sampling_logits = sampling_scores / self.tau + sampling_logits -= sampling_logits.max() + sampling_probabilities = torch.softmax(sampling_logits, dim=0) + rng = torch.Generator() + rng.manual_seed(self.seed + self.current_index) + selected_indices = torch.multinomial( + sampling_probabilities, + batch_size, + replacement=False, + generator=rng, + ) + self.logger.debug(f"{selected_indices=}") + self.logger.debug(f"{sampling_scores=}") + self.logger.debug(f"{sampling_scores[selected_indices]=}") + self.current_index += batch_size + + if return_extra_info: + selected_indices_list = selected_indices.tolist() + alphas = self.diff_estimator.alphas[selected_indices_list] + betas = self.diff_estimator.betas[selected_indices_list] + point_est = alphas / (alphas + betas) + extra_info = { + "indices": selected_indices_list, + "scores": sampling_scores[selected_indices].tolist(), + "alphas": alphas.tolist(), + "betas": betas.tolist(), + "point": point_est.tolist(), + } + return selected_indices, extra_info + else: + return selected_indices + + def state_dict(self) -> Dict: + """ + Save current state for checkpointing. + Only tracks sampling progress; actual difficulty estimates are in diff_estimator. + """ + return { + "current_index": self.current_index, + } + + def load_state_dict(self, state_dict): + """ + Restore selector state from checkpoint. + """ + self.current_index = state_dict.get("current_index", 0) diff --git a/frameworks/trinity/buffer/storage/__init__.py b/frameworks/trinity/buffer/storage/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..d03fbd9f829fed37ba62de09200f5eda89dc614d --- /dev/null +++ b/frameworks/trinity/buffer/storage/__init__.py @@ -0,0 +1,15 @@ +from trinity.buffer.storage.queue import PriorityFunction +from trinity.utils.registry import Registry + +PRIORITY_FUNC = Registry( + "priority_fn", + default_mapping={ + "linear_decay": "trinity.buffer.storage.queue.LinearDecayPriority", + "decay_limit_randomization": "trinity.buffer.storage.queue.LinearDecayUseCountControlPriority", + }, +) + +__all__ = [ + "PriorityFunction", + "PRIORITY_FUNC", +] diff --git a/frameworks/trinity/buffer/storage/file.py b/frameworks/trinity/buffer/storage/file.py new file mode 100644 index 0000000000000000000000000000000000000000..75b3499b3b23fec1d2a3ffa5f3b34a9f0cd36c50 --- /dev/null +++ b/frameworks/trinity/buffer/storage/file.py @@ -0,0 +1,81 @@ +"""File Storage""" +import json +import os +from typing import List + +import ray + +from trinity.common.config import StorageConfig +from trinity.common.experience import EID, Experience +from trinity.common.workflows import Task + + +class _Encoder(json.JSONEncoder): + def default(self, o): + if isinstance(o, Experience): + return o.to_dict() + if isinstance(o, Task): + return o.to_dict() + if isinstance(o, EID): + return o.to_dict() + return super().default(o) + + +class FileStorage: + """ + A wrapper of a local jsonl file. + + If `wrap_in_ray` in `StorageConfig` is `True`, this class will be run as + a Ray Actor, and provide a remote interface to the local file. + + This wrapper is only for writing, if you want to read from the file, use + StorageType.QUEUE instead. + """ + + def __init__(self, config: StorageConfig) -> None: + if not config.path: + raise ValueError("`path` is required for FILE storage type.") + ext = os.path.splitext(config.path)[-1] + if ext != ".jsonl" and ext != ".json": + raise ValueError(f"File path must end with '.json' or '.jsonl', got {config.path}") + path_dir = os.path.dirname(os.path.abspath(config.path)) + os.makedirs(path_dir, exist_ok=True) + self.file = open(config.path, "a", encoding="utf-8") + self.encoder = _Encoder(ensure_ascii=False) + self.ref_count = 0 + + @classmethod + def get_wrapper(cls, config: StorageConfig): + if config.wrap_in_ray: + return ( + ray.remote(cls) + .options( + name=f"json-{config.name}", + namespace=config.ray_namespace or ray.get_runtime_context().namespace, + get_if_exists=True, + ) + .remote(config) + ) + else: + return cls(config) + + def write(self, data: List) -> None: + for item in data: + json_str = self.encoder.encode(item) + self.file.write(json_str + "\n") + self.file.flush() + + def read(self) -> List: + raise NotImplementedError( + "read() is not implemented for FILE Storage, please use QUEUE instead" + ) + + def acquire(self) -> int: + self.ref_count += 1 + return self.ref_count + + def release(self) -> int: + self.ref_count -= 1 + if self.ref_count <= 0: + self.file.close() + return self.ref_count diff --git a/frameworks/trinity/buffer/storage/queue.py b/frameworks/trinity/buffer/storage/queue.py new file mode 100644 index 0000000000000000000000000000000000000000..a0da0438954b073b3b149a56c1ba164ac0fc4492 --- /dev/null +++ b/frameworks/trinity/buffer/storage/queue.py @@ -0,0 +1,420 @@ +"""Ray Queue storage""" + +import asyncio +import time +from abc import ABC, abstractmethod +from collections import deque +from copy import deepcopy +from typing import Dict, List, Optional, Tuple + +import numpy as np +import ray +from sortedcontainers import SortedDict + +from trinity.common.config import StorageConfig +from trinity.common.constants import StorageType +from trinity.common.experience import Experience +from trinity.utils.log import get_logger + + +def is_database_url(path: str) -> bool: + return any(path.startswith(prefix) for prefix in ["sqlite:///", "postgresql://", "mysql://"]) + + +def is_json_file(path: str) -> bool: + return path.endswith(".json") or path.endswith(".jsonl") + + +class PriorityFunction(ABC): + """ + Each priority_fn, + Args: + item: List[Experience], assume that all experiences in it have the same model_version and use_count + priority_fn_args: Dict, the arguments for priority_fn + + Returns: + priority: float + put_into_queue: bool, decide whether to put item into queue + + Note that put_into_queue takes effect both for new item from the explorer and for item sampled from the buffer. + """ + + @abstractmethod + def __call__(self, item: List[Experience]) -> Tuple[float, bool]: + """Calculate the priority of item.""" + + @classmethod + @abstractmethod + def default_config(cls) -> Dict: + """Return the default config.""" + + +class LinearDecayPriority(PriorityFunction): + """Calculate priority by linear decay. + + Priority is calculated as `model_version - decay * use_count. The item is always put back into the queue for reuse (as long as `reuse_cooldown_time` is not None). + """ + + def __init__(self, decay: float = 2.0): + self.decay = decay + + def __call__(self, item: List[Experience]) -> Tuple[float, bool]: + priority = float(item[0].info["model_version"] - self.decay * item[0].info["use_count"]) + put_into_queue = True + return priority, put_into_queue + + @classmethod + def default_config(cls) -> Dict: + return { + "decay": 2.0, + } + + +class LinearDecayUseCountControlPriority(PriorityFunction): + """Calculate priority by linear decay, use count control, and randomization. + + Priority is calculated as `model_version - decay * use_count`; if `sigma` is non-zero, priority is further perturbed by random Gaussian noise with standard deviation `sigma`. The item will be put back into the queue only if use count does not exceed `use_count_limit`. + """ + + def __init__(self, decay: float = 2.0, use_count_limit: int = 3, sigma: float = 0.0): + self.decay = decay + self.use_count_limit = use_count_limit + self.sigma = sigma + + def __call__(self, item: List[Experience]) -> Tuple[float, bool]: + priority = float(item[0].info["model_version"] - self.decay * item[0].info["use_count"]) + if self.sigma > 0.0: + priority += float(np.random.randn() * self.sigma) + put_into_queue = ( + item[0].info["use_count"] < self.use_count_limit if self.use_count_limit > 0 else True + ) + return priority, put_into_queue + + @classmethod + def default_config(cls) -> Dict: + return { + "decay": 2.0, + "use_count_limit": 3, + "sigma": 0.0, + } + + +class QueueBuffer(ABC): + async def set_min_model_version(self, min_model_version: int): + self.min_model_version = max(min_model_version, 0) + + @abstractmethod + async def put(self, exps: List[Experience]) -> None: + """Put a list of experiences into the queue.""" + + @abstractmethod + async def get(self) -> List[Experience]: + """Get a list of experience from the queue.""" + + @abstractmethod + def qsize(self) -> int: + """Get the current size of the queue.""" + + @abstractmethod + async def close(self) -> None: + """Close the queue.""" + + @abstractmethod + def stopped(self) -> bool: + """Check if there is no more data to read.""" + + @classmethod + def get_queue(cls, config: StorageConfig) -> "QueueBuffer": + """Get a queue instance based on the storage configuration.""" + logger = get_logger(__name__) + if config.replay_buffer.enable: + capacity = config.capacity + logger.info( + f"Using AsyncPriorityQueue with capacity {capacity}, reuse_cooldown_time {config.replay_buffer.reuse_cooldown_time}." + ) + return AsyncPriorityQueue( + capacity=capacity, + reuse_cooldown_time=config.replay_buffer.reuse_cooldown_time, + priority_fn=config.replay_buffer.priority_fn, + priority_fn_args=config.replay_buffer.priority_fn_args, + ) + else: + return AsyncQueue(capacity=config.capacity) + + +class AsyncQueue(asyncio.Queue, QueueBuffer): + def __init__(self, capacity: int): + """ + Initialize the async queue with a specified capacity. + + Args: + capacity (`int`): The maximum number of items the queue can hold. + """ + super().__init__(maxsize=capacity) + self._closed = False + self.min_model_version = 0 + + async def put(self, item: List[Experience]): + if len(item) == 0: + return + await super().put(item) + + async def get(self): + while True: + item = await super().get() + if ( + self.min_model_version <= 0 + or item[0].info["model_version"] >= self.min_model_version + ): + return item + + async def close(self) -> None: + """Close the queue.""" + self._closed = True + for getter in self._getters: + if not getter.done(): + getter.set_exception(StopAsyncIteration()) + self._getters.clear() + + def stopped(self) -> bool: + """Check if there is no more data to read.""" + return self._closed and self.empty() + + +class AsyncPriorityQueue(QueueBuffer): + """ + An asynchronous priority queue that manages a fixed-size buffer of experience items. + Items are prioritized using a user-defined function and reinserted after a cooldown period. + + Attributes: + capacity (int): Maximum number of items the queue can hold. This value is automatically + adjusted to be at most twice the read batch size. + reuse_cooldown_time (float): Delay before reusing an item (set to infinity to disable). + priority_fn (callable): Function used to determine the priority of an item. + priority_groups (SortedDict): Maps priorities to deques of items with the same priority. + """ + + def __init__( + self, + capacity: int, + reuse_cooldown_time: Optional[float] = None, + priority_fn: str = "linear_decay", + priority_fn_args: Optional[dict] = None, + ): + """ + Initialize the async priority queue. + + Args: + capacity (`int`): The maximum number of items the queue can store. + reuse_cooldown_time (`float`): Time to wait before reusing an item. Set to None to disable reuse. + priority_fn (`str`): Name of the function to use for determining item priority. + kwargs: Additional keyword arguments for the priority function. + """ + from trinity.buffer.storage import PRIORITY_FUNC + + self.capacity = capacity + self.item_count = 0 + self.priority_groups = SortedDict() # Maps priority -> deque of items + priority_fn_cls = PRIORITY_FUNC.get(priority_fn) + kwargs = priority_fn_cls.default_config() + kwargs.update(priority_fn_args or {}) + self.priority_fn = priority_fn_cls(**kwargs) + self.reuse_cooldown_time = reuse_cooldown_time + self._condition = asyncio.Condition() # For thread-safe operations + self._closed = False + self.min_model_version = 0 + + async def _put(self, item: List[Experience], delay: float = 0) -> None: + """ + Insert an item into the queue, replacing the lowest-priority item if full. + + Args: + item (`List[Experience]`): A list of experiences to add. + delay (`float`): Optional delay before insertion (for simulating timing behavior). + """ + if delay > 0: + await asyncio.sleep(delay) + if len(item) == 0: + return + + priority, put_into_queue = self.priority_fn(item=item) + if not put_into_queue: + return + + async with self._condition: + if self.item_count == self.capacity: + # If full, only insert if new item has higher or equal priority than the lowest + lowest_priority, item_queue = self.priority_groups.peekitem(index=0) + if lowest_priority > priority: + return # Skip insertion if lower priority + # Remove the lowest priority item + item_queue.popleft() + self.item_count -= 1 + if not item_queue: + self.priority_groups.popitem(index=0) + + # Add the new item + if priority not in self.priority_groups: + self.priority_groups[priority] = deque() + self.priority_groups[priority].append(item) + self.item_count += 1 + self._condition.notify() + + async def put(self, item: List[Experience]) -> None: + await self._put(item, delay=0) + + async def get(self) -> List[Experience]: + """ + Retrieve the highest-priority item from the queue. + + Returns: + List[Experience]: The highest-priority item (list of experiences). + + Notes: + - After retrieval, the item is optionally reinserted after a cooldown period. + """ + async with self._condition: + while True: + while len(self.priority_groups) == 0: + if self._closed: + raise StopAsyncIteration() + await self._condition.wait() + + _, item_queue = self.priority_groups.peekitem(index=-1) + item = item_queue.popleft() + self.item_count -= 1 + if not item_queue: + self.priority_groups.popitem(index=-1) + + if ( + self.min_model_version <= 0 + or item[0].info["model_version"] >= self.min_model_version + ): + break + + for exp in item: + exp.info["use_count"] += 1 + # Optionally resubmit the item after a cooldown + if self.reuse_cooldown_time is not None: + asyncio.create_task(self._put(item, delay=self.reuse_cooldown_time)) + + return item + + def qsize(self): + return self.item_count + + async def close(self) -> None: + """ + Close the queue. + """ + async with self._condition: + self._closed = True + # No more items will be added, but existing items can still be processed. + self.reuse_cooldown_time = None + self._condition.notify_all() + + def stopped(self) -> bool: + return self._closed and len(self.priority_groups) == 0 + + +class QueueStorage: + """An wrapper of a async queue.""" + + def __init__(self, config: StorageConfig) -> None: + self.logger = get_logger(f"queue_{config.name}", in_ray_actor=True) + self.config = config + self.capacity = config.capacity + self.queue = QueueBuffer.get_queue(config) + st_config = deepcopy(config) + st_config.wrap_in_ray = False + if st_config.path: + if is_database_url(st_config.path): + from trinity.buffer.writer.sql_writer import SQLWriter + + st_config.storage_type = StorageType.SQL.value + self.writer = SQLWriter(st_config) + elif is_json_file(st_config.path): + from trinity.buffer.writer.file_writer import JSONWriter + + st_config.storage_type = StorageType.FILE.value + self.writer = JSONWriter(st_config) + else: + self.logger.warning("Unknown supported storage path: %s", st_config.path) + self.writer = None + else: + from trinity.buffer.writer.file_writer import JSONWriter + + st_config.storage_type = StorageType.FILE.value + self.writer = JSONWriter(st_config) + self.logger.warning(f"Save experiences in {st_config.path}.") + self.ref_count = 0 + self.exp_pool = deque() # A pool to store experiences + self.closed = False + + async def acquire(self) -> int: + self.ref_count += 1 + return self.ref_count + + async def release(self) -> int: + """Release the queue.""" + self.ref_count -= 1 + if self.ref_count <= 0: + await self.queue.close() + if self.writer is not None: + await self.writer.release() + return self.ref_count + + def length(self) -> int: + """The length of the queue.""" + return self.queue.qsize() + + async def put_batch(self, exp_list: List) -> None: + """Put batch of experience.""" + await self.queue.put(exp_list) + if self.writer is not None: + self.writer.write(exp_list) + + async def get_batch(self, batch_size: int, timeout: float, min_model_version: int = 0) -> List: + """Get batch of experience.""" + await self.queue.set_min_model_version(min_model_version) + start_time = time.time() + result = [] + while len(result) < batch_size: + while len(self.exp_pool) > 0 and len(result) < batch_size: + exp = self.exp_pool.popleft() + if min_model_version > 0 and exp.info["model_version"] < min_model_version: + continue + result.append(exp) + if len(result) >= batch_size: + break + + if self.queue.stopped(): + # If the queue is stopped, ignore the rest of the experiences in the pool + raise StopAsyncIteration("Queue is closed and no more items to get.") + try: + exp_list = await asyncio.wait_for(self.queue.get(), timeout=1.0) + self.exp_pool.extend(exp_list) + except asyncio.TimeoutError: + if time.time() - start_time > timeout: + self.logger.error( + f"Timeout when waiting for experience, only get {len(self.exp_pool)} experiences.\n" + "This phenomenon is usually caused by the workflow not returning enough " + "experiences or running timeout. Please check your workflow implementation." + ) + batch = list(self.exp_pool) + self.exp_pool.clear() + return batch + return result + + @classmethod + def get_wrapper(cls, config: StorageConfig): + """Get the queue actor.""" + return ( + ray.remote(cls) + .options( + name=f"queue-{config.name}", + namespace=config.ray_namespace or ray.get_runtime_context().namespace, + get_if_exists=True, + ) + .remote(config) + ) diff --git a/frameworks/trinity/buffer/storage/sql.py b/frameworks/trinity/buffer/storage/sql.py new file mode 100644 index 0000000000000000000000000000000000000000..eb1097cdc7b1957a5c78b2dfd51901649d431306 --- /dev/null +++ b/frameworks/trinity/buffer/storage/sql.py @@ -0,0 +1,279 @@ +"""SQL database storage""" + +import time +from abc import abstractmethod +from typing import Dict, List, Optional + +import ray +from datasets import Dataset +from sqlalchemy import asc, desc +from sqlalchemy.orm import sessionmaker + +from trinity.buffer.schema import FORMATTER, init_engine +from trinity.buffer.schema.formatter import TaskFormatter +from trinity.buffer.utils import retry_session +from trinity.common.config import StorageConfig +from trinity.common.experience import Experience +from trinity.common.rewards import REWARD_FUNCTIONS +from trinity.common.workflows import WORKFLOWS, Task +from trinity.utils.log import get_logger + + +class SQLStorage: + """ + An Storage based on SQL Database. + + If `wrap_in_ray` in `StorageConfig` is `True`, this class will be run as a Ray Actor, + and provide a remote interface to the local database. + + For databases that do not support multi-processing read/write (e.g. sqlite, duckdb), please + set `wrap_in_ray` to `True`. + """ + + def __init__(self, config: StorageConfig) -> None: + self.logger = get_logger(f"sql_{config.name}", in_ray_actor=True) + if not config.path: + raise ValueError("`path` is required for SQL storage type.") + self.logger.info( + f"Init engine {config.path} with table {config.name} with schema {config.schema_type}" + ) + self.engine, self.table_model_cls = init_engine( + db_url=config.path, + table_name=config.name, + schema_type=config.schema_type, + ) + self.logger.info(f"Init SQL storage at {config.path}") + self.session = sessionmaker(bind=self.engine) + self.max_retry_times = config.max_retry_times + self.max_retry_interval = config.max_retry_interval + self.ref_count = 0 + self.stopped = False + # Assume that the auto-increment ID starts counting from 1, so the default offset should be 0. + self.offset = config.index + + @classmethod + def get_wrapper(cls, config: StorageConfig): + if config.schema_type is None: + storage_cls = SQLTaskStorage + else: + storage_cls = SQLExperienceStorage + if config.wrap_in_ray: + return ( + ray.remote(storage_cls) + .options( + name=f"sql-{config.name}", + namespace=config.ray_namespace or ray.get_runtime_context().namespace, + get_if_exists=True, + max_concurrency=5, + ) + .remote(config) + ) + else: + return storage_cls(config) + + @abstractmethod + def write(self, data: List) -> None: + """Write a batch of data.""" + + @abstractmethod + def read(self, batch_size: Optional[int] = None) -> List: + """Read a batch of data.""" + + def acquire(self) -> int: + self.ref_count += 1 + return self.ref_count + + def release(self) -> int: + self.ref_count -= 1 + if self.ref_count <= 0: + self.stopped = True + return self.ref_count + + +class SQLExperienceStorage(SQLStorage): + """Used as trainer input.""" + + def __init__(self, config: StorageConfig) -> None: + super().__init__(config) + self.max_timeout = config.max_read_timeout + self.batch_size = config.batch_size + self.enable_replay = config.replay_buffer is not None and config.replay_buffer.enable + # TODO: optimize the following logic + if config.schema_type == "experience": + # NOTE: consistent with the old version of experience buffer + self._read_method = self._read_priority + else: + # SFT / DPO uses FIFO style + self._read_method = self._read_fifo + + def write(self, data: List[Experience]) -> None: + with retry_session(self.session, self.max_retry_times, self.max_retry_interval) as session: + experience_models = [self.table_model_cls.from_experience(exp) for exp in data] + session.add_all(experience_models) + self.logger.info(f"Write {len(experience_models)} experiences to SQL storage.") + + def _read_fifo(self, batch_size: int) -> List[Experience]: + """Read experiences in FIFO order.""" + exp_list = [] + start_time = time.time() + while len(exp_list) < batch_size: + if self.stopped: + raise StopIteration() + if time.time() - start_time > self.max_timeout: + self.logger.warning( + f"Max read timeout reached ({self.max_timeout} s), only get {len(exp_list)} experiences, stopping..." + ) + raise StopIteration() + with retry_session( + self.session, self.max_retry_times, self.max_retry_interval + ) as session: + # get a batch of experiences from the database + experiences = ( + session.query(self.table_model_cls) + .filter(self.table_model_cls.id > self.offset) + .order_by(asc(self.table_model_cls.id)) + .limit(batch_size - len(exp_list)) + .all() + ) + if experiences: + self.offset = experiences[-1].id + start_time = time.time() + exp_list.extend([self.table_model_cls.to_experience(exp) for exp in experiences]) + if len(exp_list) < batch_size: + self.logger.info(f"Waiting for {batch_size - len(exp_list)} more experiences...") + time.sleep(1) + return exp_list + + def _read_priority(self, batch_size: int, min_model_version: int = 0) -> List[Experience]: + exp_list = [] + start_time = time.time() + latest_size = 0 + while latest_size < batch_size: + if self.stopped: + raise StopIteration() + if time.time() - start_time > self.max_timeout: + self.logger.warning( + f"Max read timeout reached ({self.max_timeout} s), only get {latest_size} experiences, stopping..." + ) + raise StopIteration() + with retry_session( + self.session, self.max_retry_times, self.max_retry_interval + ) as session: + query = session.query(self.table_model_cls) + if min_model_version > 0: + query = query.filter(self.table_model_cls.model_version >= min_model_version) + if not self.enable_replay: + query = query.filter(self.table_model_cls.consumed == 0) + experiences = ( + query.order_by( + asc(self.table_model_cls.consumed), desc(self.table_model_cls.id) + ) + .limit(batch_size) + .with_for_update() + .all() + ) + if len(experiences) != batch_size: + if latest_size != len(experiences): + latest_size = len(experiences) + start_time = time.time() + else: + ids = [exp.id for exp in experiences] + session.query(self.table_model_cls).filter( + self.table_model_cls.id.in_(ids) + ).update( + {self.table_model_cls.consumed: self.table_model_cls.consumed + 1}, + synchronize_session=False, + ) + exp_list.extend( + [self.table_model_cls.to_experience(exp) for exp in experiences] + ) + break + + self.logger.info(f"Waiting for {batch_size - len(exp_list)} more experiences...") + time.sleep(1) + return exp_list + + def read(self, batch_size: Optional[int] = None, **kwargs) -> List[Experience]: + if self.stopped: + raise StopIteration() + + batch_size = self.batch_size if batch_size is None else batch_size + return self._read_method(batch_size, **kwargs) + + @classmethod + def load_from_dataset(cls, dataset: Dataset, config: StorageConfig) -> "SQLExperienceStorage": + storage = cls(config) + formatter = FORMATTER.get(config.schema_type)( + tokenizer_path=config.tokenizer_path, format_config=config.format + ) + batch_size = storage.batch_size + batch = [] + for item in dataset: + batch.append(formatter.format(item)) + if len(batch) >= batch_size: + storage.write(batch) + batch.clear() + if batch: + storage.write(batch) + return storage + + +class SQLTaskStorage(SQLStorage): + """Used as explorer input.""" + + def __init__(self, config: StorageConfig) -> None: + super().__init__(config) + self.batch_size = config.batch_size + self.is_eval = config.is_eval + self.default_workflow_cls = WORKFLOWS.get(config.default_workflow_type) # type: ignore + self.default_reward_fn_cls = REWARD_FUNCTIONS.get(config.default_reward_fn_type) # type: ignore + self.formatter = TaskFormatter(config) + self.offset = config.index + if config.total_steps: + self.total_samples = self.batch_size * config.total_steps + else: + if config.total_epochs > 1: + self.logger.warning( + f"SQL Storage do not support total_epochs, the value {config.total_epochs} will be ignored" + ) + self.total_samples = float("inf") + + def write(self, data: List[Dict]) -> None: + with retry_session(self.session, self.max_retry_times, self.max_retry_interval) as session: + tasks = [self.table_model_cls.from_dict(item) for item in data] + session.add_all(tasks) + + def read(self, batch_size: Optional[int] = None) -> List[Task]: + if self.stopped: + raise StopIteration() + if self.offset > self.total_samples: + raise StopIteration() + batch_size = self.batch_size if batch_size is None else batch_size + with retry_session(self.session, self.max_retry_times, self.max_retry_interval) as session: + query = ( + session.query(self.table_model_cls) + .filter(self.table_model_cls.id > self.offset) + .order_by(asc(self.table_model_cls.id)) + .limit(batch_size) + ) + results = query.all() + if len(results) == 0: + raise StopIteration() + if not self.is_eval and len(results) < batch_size: + raise StopIteration() + self.offset = results[-1].id + return [self.formatter.format(item.raw_task) for item in results] + + @classmethod + def load_from_dataset(cls, dataset: Dataset, config: StorageConfig) -> "SQLTaskStorage": + storage = cls(config) + batch_size = config.batch_size + batch = [] + for item in dataset: + batch.append(item) + if len(batch) >= batch_size: + storage.write(batch) + batch.clear() + if batch: + storage.write(batch) + return storage diff --git a/frameworks/trinity/buffer/task_scheduler.py b/frameworks/trinity/buffer/task_scheduler.py new file mode 100644 index 0000000000000000000000000000000000000000..c511886145c01ed532ccc83c39f4b2e7cdc6f2c2 --- /dev/null +++ b/frameworks/trinity/buffer/task_scheduler.py @@ -0,0 +1,262 @@ +# -*- coding: utf-8 -*- +"""The taskset scheduler.""" + +from collections import Counter +from copy import deepcopy +from typing import Dict, List + +import numpy as np + +from trinity.buffer.buffer import get_buffer_reader +from trinity.common.config import Config +from trinity.common.constants import SELECTOR_METRIC +from trinity.utils.annotations import Experimental + + +def get_taskset_scheduler(explorer_state: Dict, config: Config) -> "TasksetSchedulerBase": + """Get a taskset scheduler according to the config. + + Args: + explorer_state (Dict): Restoration state from checkpoint (may include progress info) + config (Config): Full system configuration containing buffer and taskset settings + + Returns: + TasksetSchedulerBase: The taskset scheduler instance + """ + taskset_configs = config.buffer.explorer_input.tasksets + if len(taskset_configs) == 1 and taskset_configs[0].task_selector.selector_type == "sequential": + return SimpleTasksetScheduler(explorer_state, config) + else: + return TasksetScheduler(explorer_state, config) + + +class TasksetSchedulerBase: + def __init__(self, explorer_state: Dict, config: Config): + self.config = config + self.explorer_state = explorer_state + + async def read_async(self) -> List: + """Asynchronously reads a batch of tasks according to the current schedule.""" + raise NotImplementedError + + def state_dict(self) -> List[Dict]: + """return persistent state for checkpointing. + + Returns: + List[Dict]: State dicts for all selectors (one per taskset) + """ + raise NotImplementedError + + def feedback(self, pipeline_metrics: Dict) -> None: + """Update selectors using feedback from the training pipeline.""" + raise NotImplementedError + + +class SimpleTasksetScheduler(TasksetSchedulerBase): + """ + A simple taskset scheduler that only reads from one taskset without task selection strategies. + """ + + def __init__(self, explorer_state: Dict, config: Config): + super().__init__(explorer_state, config) + if "latest_task_index" in self.explorer_state: + self.explorer_state["taskset_states"] = [ + { + "current_index": explorer_state["latest_task_index"], + } + ] + index = self.explorer_state.get("taskset_states", [{"current_index": 0}])[0].get( + "current_index", 0 + ) + taskset_config = deepcopy(self.config.buffer.explorer_input.tasksets[0]) + taskset_config.index = index + taskset_config.task_selector = None # disable selection + self.reader = get_buffer_reader(taskset_config) + + async def read_async(self) -> List: + return await self.reader.read_async() + + def state_dict(self) -> List[Dict]: + return [self.reader.state_dict()] + + def feedback(self, pipeline_metrics: Dict) -> None: + # do nothing here + return + + +@Experimental +class TasksetScheduler(TasksetSchedulerBase): + """ + Coordinates multiple datasets (tasksets) with customizable task selection strategies per taskset. + + The scheduler: + - Manages multiple data sources (tasksets) + - Uses a selector per taskset to determine which samples to read + - Shuffles the order of taskset access across epochs + - Supports adaptive selectors via feedback (e.g., difficulty-based sampling) + - Enables curriculum-like or interleaved multi-task training + + It assumes that each call to `read_async()` corresponds to one training step, + and batches are built by aggregating samples from different tasksets based on + a shuffled global schedule. + """ + + def __init__(self, explorer_state: Dict, config: Config): + """ + Initialize the scheduler from configuration and previous state (for resume support). + + Args: + explorer_state (Dict): Restoration state from checkpoint (may include progress info) + config (Config): Full system configuration containing buffer and taskset settings + """ + super().__init__(explorer_state, config) + + # Backward compatibility: old format stored 'latest_task_index' directly + if "latest_task_index" in explorer_state: + assert len(config.buffer.explorer_input.tasksets) == 1 # old format + explorer_state["taskset_states"] = [ + { + "current_index": explorer_state["latest_task_index"], + } + ] + + self.read_batch_size = config.buffer.batch_size + taskset_configs = config.buffer.explorer_input.tasksets + + from trinity.buffer.reader.file_reader import FileReader + + taskset_states = explorer_state.get( + "taskset_states", [{"current_index": 0}] * len(taskset_configs) + ) + self.tasksets = [] + for taskset_config, taskset_state in zip(taskset_configs, taskset_states): + assert not taskset_config.is_eval # assume drop last + taskset = get_buffer_reader(taskset_config) + if not isinstance(taskset, FileReader): + raise TypeError( + f"Taskset '{taskset_config.name}' has an unsupported type '{type(taskset).__name__}'." + f"Currently, only 'FileReader' is supported by TasksetScheduler." + ) + taskset.load_state_dict(taskset_state) # Restore any prior state + self.tasksets.append(taskset) + + # Each explorer step calls read_async once → track step globally + self.step = explorer_state.get("latest_iteration", 0) + + # Build flat list indicating how often each taskset should appear per epoch + self.base_taskset_ids = [] + for i, taskset in enumerate(self.tasksets): + self.base_taskset_ids.extend([i] * len(taskset)) + if len(self.base_taskset_ids) == 0: + raise ValueError("Empty tasksets provided!") + + self.epoch = self.step * self.read_batch_size // len(self.base_taskset_ids) + self.orders = self.build_orders(self.epoch) + + if self.config.buffer.total_steps: + self.max_steps = self.config.buffer.total_steps + else: + self.max_steps = ( + self.config.buffer.total_epochs * len(self.base_taskset_ids) // self.read_batch_size + ) + + def build_orders(self, epoch: int): + """ + Creates a shuffled sequence of taskset IDs to control sampling priority per step. + + At the start of each epoch, all tasksets are shuffled proportionally to their size, + ensuring balanced exposure while introducing randomness in selection order. + + Args: + epoch (int): Epoch ID used as seed for deterministic shuffling + + Returns: + List[int]: Sequence of taskset IDs, length = steps_per_epoch * batch_size + """ + taskset_ids = self.base_taskset_ids.copy() + rng = np.random.default_rng(epoch) + rng.shuffle(taskset_ids) + return taskset_ids + + def _should_stop(self) -> bool: + return self.step >= self.max_steps + + async def read_async(self) -> List: + """ + Asynchronously reads a batch of tasks according to the current schedule. + + For each step: + - Checks if a new epoch has started; rebuilds order if so + - Determines which tasksets contribute to this batch + - Uses each taskset's selector to pick specific samples + - Annotates each task with its source taskset_id + - Returns combined list of tasks + + Raises: + StopAsyncIteration: When total_epochs is reached + + Returns: + List[Task]: A batch of tasks from potentially multiple tasksets + """ + if self._should_stop(): + raise StopAsyncIteration + + batch_size = self.read_batch_size + start = self.step * batch_size % len(self.base_taskset_ids) + end = start + batch_size + if end <= len(self.base_taskset_ids): + taskset_ids = self.orders[start:end] + if end == len(self.base_taskset_ids): + self.epoch += 1 + self.orders = self.build_orders(self.epoch) + else: + taskset_ids = self.orders[start:] + self.epoch += 1 + self.orders = self.build_orders(self.epoch) + taskset_ids += self.orders[: (end - len(self.base_taskset_ids))] + + counter = Counter(taskset_ids) + batch = [] + for taskset_id, count in counter.items(): + tasks = await self.tasksets[taskset_id].read_async(batch_size=count) + # Annotate each task with its origin + for task in tasks: + task.index["taskset_id"] = taskset_id + batch.extend(tasks) + + self.step += 1 + return batch + + def state_dict(self) -> List[Dict]: + """ + Save persistent state for checkpointing. + + Returns: + List[Dict]: State dicts for all tasksets + """ + return [taskset.state_dict() for taskset in self.tasksets] + + def feedback(self, pipeline_metrics: Dict) -> None: + """ + Update selectors in tasksets using feedback from the training pipeline. + + Expected format: + pipeline_metrics = { + SELECTOR_METRIC: { + 0: {"indices": [...], "values": [...]}, + 1: {"indices": [...], "values": [...]} + }, + ... # other metrics + } + + This allows adaptive selectors (like `DifficultyBasedSelector`) to refine difficulty estimates. + + Args: + pipeline_metrics (Dict): Metrics dictionary passed from explorer. + """ + if SELECTOR_METRIC not in pipeline_metrics: + return + selector_metric = pipeline_metrics.pop(SELECTOR_METRIC, {}) + for taskset_id, taskset_kwargs in selector_metric.items(): + taskset = self.tasksets[taskset_id] + taskset.feedback(**taskset_kwargs) diff --git a/frameworks/trinity/buffer/utils.py b/frameworks/trinity/buffer/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..689a8cce3d020d0284c2ff9106f8ac81adcee97f --- /dev/null +++ b/frameworks/trinity/buffer/utils.py @@ -0,0 +1,34 @@ +import time +from contextlib import contextmanager + +from trinity.utils.log import get_logger + + +@contextmanager +def retry_session(session_maker, max_retry_times: int = 2, max_retry_interval: float = 1.0): + """A Context manager for retrying session.""" + logger = get_logger(__name__) + for attempt in range(max_retry_times): + try: + session = session_maker() + yield session + session.commit() + break + except StopIteration as e: + raise e + except Exception as e: + import traceback + + trace_str = traceback.format_exc() + session.rollback() + logger.warning( + f"Attempt {attempt + 1} failed, retrying in {max_retry_interval} seconds..." + ) + logger.warning(f"trace = {trace_str}") + if attempt < max_retry_times - 1: + time.sleep(max_retry_interval) + else: + logger.error("Max retry attempts reached, raising exception.") + raise e + finally: + session.close() diff --git a/frameworks/trinity/buffer/viewer.py b/frameworks/trinity/buffer/viewer.py new file mode 100644 index 0000000000000000000000000000000000000000..86d75240e73f38643919fa1dbb8b994d271e9cad --- /dev/null +++ b/frameworks/trinity/buffer/viewer.py @@ -0,0 +1,336 @@ +import argparse +from typing import List + +import streamlit as st +import streamlit.components.v1 as components +from sqlalchemy.orm import sessionmaker +from transformers import AutoTokenizer + +from trinity.buffer.schema import init_engine +from trinity.common.config import StorageConfig +from trinity.common.experience import Experience +from trinity.utils.log import get_logger + + +class SQLExperienceViewer: + def __init__(self, config: StorageConfig) -> None: + self.logger = get_logger(f"sql_{config.name}", in_ray_actor=True) + if not config.path: + raise ValueError("`path` is required for SQL storage type.") + self.engine, self.table_model_cls = init_engine( + db_url=config.path, + table_name=config.name, + schema_type=config.schema_type, + ) + self.session = sessionmaker(bind=self.engine) + + def get_experiences(self, offset: int, limit: int = 10) -> List[Experience]: + self.logger.info(f"Viewing experiences from offset {offset} with limit {limit}.") + with self.session() as session: + query = session.query(self.table_model_cls).offset(offset).limit(limit) + results = query.all() + exps = [self.table_model_cls.to_experience(row) for row in results] + return exps + + def total_experiences(self) -> int: + with self.session() as session: + count = session.query(self.table_model_cls).count() + return count + + +st.set_page_config(page_title="Trinity-RFT Experience Visualizer", layout="wide") + + +def get_color_for_action_mask(action_mask_value: int) -> str: + """Return color based on action_mask value""" + if action_mask_value == 1: + return "#c8e6c9" + else: + return "#ffcdd2" + + +def render_experience(exp: Experience, tokenizer): + """Render a single experience sequence in Streamlit.""" + token_ids = exp.tokens + logprobs = exp.logprobs + action_mask = exp.action_mask + + prompt_length = exp.prompt_length + + prompt_token_ids = token_ids[:prompt_length] # type: ignore [index] + response_token_ids = token_ids[prompt_length:] # type: ignore [index] + + # Decode tokens + prompt_text = ( + tokenizer.decode(prompt_token_ids) + if hasattr(tokenizer, "decode") + else "".join([str(tid) for tid in prompt_token_ids]) + ) + response_text = ( + tokenizer.decode(response_token_ids) + if hasattr(tokenizer, "decode") + else "".join([str(tid) for tid in response_token_ids]) + ) + + # Get each response token text + response_tokens = [] + for tid in response_token_ids: + if hasattr(tokenizer, "decode"): + token_text = tokenizer.decode([tid]) + else: + token_text = f"[{tid}]" + response_tokens.append(token_text) + + # HTML escape function + def html_escape(text): + return ( + text.replace("&", "&") + .replace("<", "<") + .replace(">", ">") + .replace('"', """) + .replace("'", "'") + ) + + # === Use Streamlit Native Components for Prompt and Response === + st.subheader(f"Experience [{exp.eid}]") + + # Prompt section using st.text_area + st.markdown("**📝 Prompt:**") + st.code(prompt_text, language=None, wrap_lines=True, line_numbers=True) + + # Response section using st.text_area + st.markdown("**💬 Response:**") + st.code(response_text, language=None, wrap_lines=True, line_numbers=True) + + # Reward and other info + st.markdown("**🏆 Reward and Other Info:**") + reward, info, metrics = st.columns(3) + reward.metric("**Reward:**", f"{exp.reward or 0.0:.4f}") + metrics.markdown("**Metrics:**") + metrics.json(exp.metrics or {}, expanded=False) + info.markdown("**Info:**") + info.json(exp.info or {}, expanded=False) + + # Response Tokens Detail section using components.html + st.markdown("**🔍 Response Tokens Detail:**") + + # Build HTML only for Response Tokens Detail + html = """ + + + + + + + +
+ """ + + # Add each response token + for i, (token_text, logprob, mask) in enumerate(zip(response_tokens, logprobs, action_mask)): # type: ignore [arg-type] + bg_color = get_color_for_action_mask(mask) + + # Handle special character display + token_display = token_text.replace(" ", "␣").replace("\n", "↵").replace("\t", "⇥") + token_display = html_escape(token_display) + + html += f""" +
+
{token_display}
+
{logprob:.4f}
+
+ """ + html += """ +
+ + + """ + + # Use components.html for token details only + components.html(html, height=200, scrolling=True) + + +def parse_args(): + parser = argparse.ArgumentParser(description="Experience Visualizer") + parser.add_argument( + "--db-url", + type=str, + help="Path to the experience database.", + ) + parser.add_argument( + "--table", + type=str, + help="Name of the experience table.", + ) + parser.add_argument( + "--tokenizer", + type=str, + help="Path to the tokenizer.", + ) + return parser.parse_args() + + +def main(): + args = parse_args() + + # Initialize SQLExperienceViewer + config = StorageConfig( + name=args.table, + path=args.db_url, + schema_type="experience", + storage_type="sql", + ) + viewer = SQLExperienceViewer(config) + + st.title("🎯 Trinity-RFT Experience Visualizer") + if "page" not in st.session_state: + st.session_state.page = 1 + + # Add instructions + with st.expander("ℹ️ Instructions"): + st.markdown( + """ + - **Green background**: action_mask = 1 + - **Red background**: action_mask = 0 + - **Top**: Token text (special characters: space=␣, newline=↵, tab=⇥) + - **Bottom**: Logprob value of the token + - Hover over token to zoom in + """ + ) + + # Get total sequence number + total_seq_num = viewer.total_experiences() + + # Sidebar configuration + st.sidebar.header("⚙️ Settings") + + # Pagination settings + experiences_per_page = st.sidebar.slider( + "Experiences per page", min_value=1, max_value=20, value=5 + ) + + # Calculate total pages + total_pages = (total_seq_num + experiences_per_page - 1) // experiences_per_page + + # Page selection (sidebar) + current_page = st.sidebar.number_input( + "Select page", + min_value=1, + max_value=max(1, total_pages), + step=1, + value=st.session_state.page, + ) + if current_page != st.session_state.page: + st.session_state.page = current_page + st.rerun() + + # Show statistics + st.sidebar.markdown("---") + st.sidebar.metric("Total experiences", total_seq_num) + st.sidebar.metric("Total pages", total_pages) + st.sidebar.metric("Current page", f"{st.session_state.page}/{total_pages}") + + # Calculate offset + offset = (st.session_state.page - 1) * experiences_per_page + + # Get experiences for current page + experiences = viewer.get_experiences(offset, experiences_per_page) + + tokenizer = AutoTokenizer.from_pretrained(args.tokenizer) + + # Generate catalog in sidebar + exp_catalog = [] # [(eid, subheader_text)] + if experiences: + for exp in experiences: + exp_catalog.append(exp.eid) + + if exp_catalog: + st.sidebar.markdown("---") + st.sidebar.markdown("**Contents**") + catalog_md = "\n".join([f"- [ {eid} ](#exp-{eid})" for eid in exp_catalog]) + st.sidebar.markdown(catalog_md, unsafe_allow_html=True) + + if experiences: + for exp in experiences: + st.markdown(f'', unsafe_allow_html=True) + render_experience(exp, tokenizer) + else: + st.warning("No experience data found") + + # Pagination navigation + st.markdown("---") + col1, col2, col3 = st.columns([1, 2, 1]) + + with col1: + if st.session_state.page > 1: + if st.button("⬅️ Previous Page"): + st.session_state.page = st.session_state.page - 1 + st.rerun() + + with col2: + st.markdown( + f"
Page {st.session_state.page} / {total_pages}
", unsafe_allow_html=True + ) + with col3: + if st.session_state.page < total_pages: + if st.button("Next Page ➡️"): + st.session_state.page = st.session_state.page + 1 + st.rerun() + + +if __name__ == "__main__": + main() diff --git a/frameworks/trinity/buffer/writer/__init__.py b/frameworks/trinity/buffer/writer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/frameworks/trinity/buffer/writer/file_writer.py b/frameworks/trinity/buffer/writer/file_writer.py new file mode 100644 index 0000000000000000000000000000000000000000..cfb5753bb08330e1d2b1b106ce31767e3910391e --- /dev/null +++ b/frameworks/trinity/buffer/writer/file_writer.py @@ -0,0 +1,40 @@ +from typing import List + +import ray + +from trinity.buffer.buffer_writer import BufferWriter +from trinity.buffer.storage.file import FileStorage +from trinity.common.config import StorageConfig +from trinity.common.constants import StorageType + + +class JSONWriter(BufferWriter): + def __init__(self, config: StorageConfig): + assert config.storage_type == StorageType.FILE.value + self.writer = FileStorage.get_wrapper(config) + self.wrap_in_ray = config.wrap_in_ray + + def write(self, data: List) -> None: + if self.wrap_in_ray: + ray.get(self.writer.write.remote(data)) + else: + self.writer.write(data) + + async def write_async(self, data): + if self.wrap_in_ray: + await self.writer.write.remote(data) + else: + self.writer.write(data) + + async def acquire(self) -> int: + if self.wrap_in_ray: + return await self.writer.acquire.remote() + else: + return 0 + + async def release(self) -> int: + if self.wrap_in_ray: + return await self.writer.release.remote() + else: + self.writer.release() + return 0 diff --git a/frameworks/trinity/buffer/writer/queue_writer.py b/frameworks/trinity/buffer/writer/queue_writer.py new file mode 100644 index 0000000000000000000000000000000000000000..951f5445b7db222501ad26a4f7d81028c95cf12e --- /dev/null +++ b/frameworks/trinity/buffer/writer/queue_writer.py @@ -0,0 +1,29 @@ +"""Writer of the Queue buffer.""" +from typing import List + +import ray + +from trinity.buffer.buffer_writer import BufferWriter +from trinity.buffer.storage.queue import QueueStorage +from trinity.common.config import StorageConfig +from trinity.common.constants import StorageType + + +class QueueWriter(BufferWriter): + """Writer of the Queue buffer.""" + + def __init__(self, config: StorageConfig): + assert config.storage_type == StorageType.QUEUE.value + self.queue = QueueStorage.get_wrapper(config) + + def write(self, data: List) -> None: + ray.get(self.queue.put_batch.remote(data)) + + async def write_async(self, data): + return await self.queue.put_batch.remote(data) + + async def acquire(self) -> int: + return await self.queue.acquire.remote() + + async def release(self) -> int: + return await self.queue.release.remote() diff --git a/frameworks/trinity/buffer/writer/sql_writer.py b/frameworks/trinity/buffer/writer/sql_writer.py new file mode 100644 index 0000000000000000000000000000000000000000..4823fa1c7bd1f39e6bd8df12ff419661d5f31239 --- /dev/null +++ b/frameworks/trinity/buffer/writer/sql_writer.py @@ -0,0 +1,43 @@ +"""Writer of the SQL buffer.""" + +import ray + +from trinity.buffer.buffer_writer import BufferWriter +from trinity.buffer.storage.sql import SQLStorage +from trinity.common.config import StorageConfig +from trinity.common.constants import StorageType + + +class SQLWriter(BufferWriter): + """Writer of the SQL buffer.""" + + def __init__(self, config: StorageConfig) -> None: + assert config.storage_type == StorageType.SQL.value + # we only support write RFT algorithm buffer for now + self.wrap_in_ray = config.wrap_in_ray + self.db_wrapper = SQLStorage.get_wrapper(config) + + def write(self, data: list) -> None: + if self.wrap_in_ray: + ray.get(self.db_wrapper.write.remote(data)) + else: + self.db_wrapper.write(data) + + async def write_async(self, data): + if self.wrap_in_ray: + await self.db_wrapper.write.remote(data) + else: + self.db_wrapper.write(data) + + async def acquire(self) -> int: + if self.wrap_in_ray: + return await self.db_wrapper.acquire.remote() + else: + return 0 + + async def release(self) -> int: + if self.wrap_in_ray: + return await self.db_wrapper.release.remote() + else: + self.db_wrapper.release() + return 0 diff --git a/frameworks/trinity/cli/client.py b/frameworks/trinity/cli/client.py new file mode 100644 index 0000000000000000000000000000000000000000..cc3318b57029bad49371c9a1ae9d206816884682 --- /dev/null +++ b/frameworks/trinity/cli/client.py @@ -0,0 +1,44 @@ +import requests + + +def send_get_request(url: str, params: dict): + """ + Send GET request with parameters. + + Args: + url (str): url of server (e.g., http://localhost:8000/api) + params (str): parameters that need to be passed + """ + try: + # send GET request + response = requests.get(url, params=params) + + # check HTTP status code + response.raise_for_status() + + # print status code and return response json + print(f"Status Code: {response.status_code}") + return response.json() + + except requests.exceptions.RequestException as e: + print(f"Failed with: {e}") + return None + + +def request(url, **kwargs): + return send_get_request(url, kwargs) + + +if __name__ == "__main__": + # --- only for local testing + LOCAL_DATA_PROCESSOR_SERVER_URL = "http://127.0.0.1:5005/data_processor" + LOCAL_TRINITY_TRAINING_SERVER_URL = "http://127.0.0.1:5006/trinity_rft" + # --- only for local testing + + res = request( + url=LOCAL_DATA_PROCESSOR_SERVER_URL, + configPath="examples/grpo_gsm8k/gsm8k.yaml", + ) + if res: + print(res) + print(res["message"]) diff --git a/frameworks/trinity/cli/launcher.py b/frameworks/trinity/cli/launcher.py new file mode 100644 index 0000000000000000000000000000000000000000..c077d0bcaafd26132b44b030d43ffc509347cb58 --- /dev/null +++ b/frameworks/trinity/cli/launcher.py @@ -0,0 +1,417 @@ +"""Launch the trainer""" +import argparse +import asyncio +import os +import sys +import traceback +from pathlib import Path +from pprint import pprint +from typing import Optional + +import ray + +from trinity.buffer.pipelines.task_pipeline import check_and_run_task_pipeline +from trinity.common.config import Config, load_config +from trinity.common.constants import DEBUG_NAMESPACE, PLUGIN_DIRS_ENV_VAR +from trinity.explorer.explorer import Explorer +from trinity.manager.checkpoint_converter import Converter +from trinity.manager.state_manager import StateManager +from trinity.trainer.trainer import Trainer +from trinity.utils.dlc_utils import is_running, setup_ray_cluster, stop_ray_cluster +from trinity.utils.log import get_logger +from trinity.utils.plugin_loader import load_plugins + +logger = get_logger(__name__) + + +def bench(config: Config) -> None: + """Evaluate model.""" + config.explorer.name = "benchmark" + try: + explorer = Explorer.get_actor(config) + ray.get(explorer.prepare.remote()) + ray.get(explorer.benchmark.remote()) + logger.info("Benchmark finished.") + ray.get(explorer.shutdown.remote()) + except Exception: + logger.error(f"Benchmark failed:\n{traceback.format_exc()}") + + +def explore(config: Config) -> None: + """Run explorer.""" + try: + explorer = Explorer.get_actor(config) + ray.get(explorer.prepare.remote()) + ray.get(explorer.sync_weight.remote()) + ray.get(explorer.explore.remote()) + ray.get(explorer.shutdown.remote()) + except Exception: + logger.error(f"Explorer failed:\n{traceback.format_exc()}") + + +def train(config: Config) -> None: + """Run trainer.""" + try: + trainer = Trainer.get_actor(config) + ray.get(trainer.prepare.remote()) + ray.get(trainer.sync_weight.remote()) + ray.get(trainer.train.remote()) + ray.get(trainer.shutdown.remote()) + except Exception: + logger.error(f"Trainer failed:\n{traceback.format_exc()}") + + +def serve(config: Config) -> None: + """Run explorer in server mode.""" + try: + explorer = Explorer.get_actor(config) + ray.get(explorer.prepare.remote()) + ray.get(explorer.sync_weight.remote()) + ray.get(explorer.serve.remote()) + ray.get(explorer.shutdown.remote()) + except Exception: + logger.error(f"Explorer failed:\n{traceback.format_exc()}") + + +def both(config: Config) -> None: + """Setup both explorer and trainer. + + For the explorer, a step contains `batch_size * sync_interval` number + of rollout tasks. + + For the trainer, it has to consume all experiences generated by the explorer in + the latest step. The specific number of experiences may vary for different + algorithms and tasks. + """ + try: + explorer = Explorer.get_actor(config) + trainer = Trainer.get_actor(config) + ray.get([explorer.__ray_ready__.remote(), trainer.__ray_ready__.remote()]) + ray.get( + [ + explorer.prepare.remote(), + trainer.prepare.remote(), + ] + ) + ray.get( + [ + explorer.sync_weight.remote(), + trainer.sync_weight.remote(), + ] + ) + ready_ref, wait_ref = ray.wait( + [ + explorer.explore.remote(), + trainer.train.remote(), + ], + num_returns=1, + ) + + ready = ray.get(ready_ref[0]) + if ready == config.trainer.name: + logger.info( + "===========================================================\n" + "> Launcher detected that the `Trainer` process has finished.\n" + "> Stopping the explorer process immediately.\n" + "===========================================================" + ) + ray.wait(wait_ref, timeout=5) + elif ready == config.explorer.name: + logger.info( + "===============================================================\n" + "> Launcher detected that the `Explorer` process has finished.\n" + "> `Trainer` process may need to save the model checkpoint.\n" + f"> Waiting {config.synchronizer.sync_timeout} s for the trainer process...\n" + "> You can force stop the `Trainer` process by pressing Ctrl+C.\n" + "===============================================================" + ) + ray.wait(wait_ref, timeout=config.synchronizer.sync_timeout) + ray.wait( + [explorer.shutdown.remote(), trainer.shutdown.remote()], + timeout=config.synchronizer.sync_timeout, + num_returns=2, + ) + except Exception: + logger.error(f"Explorer or Trainer failed:\n{traceback.format_exc()}") + + +MODE_MAP = { + "explore": explore, + "train": train, + "both": both, + "bench": bench, + "serve": serve, +} + + +def run_stage(config: Config) -> None: + ray.init( + address=config.cluster.ray_address, + ignore_reinit_error=True, + namespace=config.ray_namespace, + runtime_env={"env_vars": config.get_envs()}, + ) + pprint(config) + try: + check_and_run_task_pipeline(config) + MODE_MAP[config.mode](config) + finally: + if config.monitor.enable_ray_timeline: + timeline_file = os.path.join(config.monitor.cache_dir, "timeline.json") + logger.info(f"Exporting Ray timeline to {timeline_file}...") + ray.timeline(filename=timeline_file) + logger.info("Done. You can open the timeline file in `chrome://tracing`") + ray.shutdown() + + +def run(config_path: str, dlc: bool = False, plugin_dir: str = None): + if plugin_dir: + os.environ[PLUGIN_DIRS_ENV_VAR] = plugin_dir + load_plugins() + config = load_config(config_path) + + if dlc: + cluster_namespace = f"{config.project}-{config.name}" + config.cluster.ray_address = setup_ray_cluster(namespace=cluster_namespace) + + if not is_running(): + raise RuntimeError("Ray is not running, please start it by `ray start --head`.") + + try: + if config.stages: + from trinity.trainer.verl.utils import get_latest_hf_checkpoint_path + + state_manager = StateManager( + path=os.path.join(config.checkpoint_root_dir, config.project, config.name) + ) + latest_stage = state_manager.load_stage().get("latest_stage", 0) + prev_stage_checkpoint = None + for i, stage_config in enumerate(config): + if i < latest_stage: + logger.info( + "===========================================================\n" + f"> Skipping completed stage {i + 1}/{len(config.stages)}...\n" + "===========================================================" + ) + stage_config.check_and_update() + else: + logger.info( + "===========================================================\n" + f"> Starting stage {i + 1}/{len(config.stages)}...\n" + "===========================================================" + ) + state_manager.save_stage(i) + if prev_stage_checkpoint is not None: + stage_config.model.model_path = prev_stage_checkpoint + stage_config.check_and_update() + run_stage(stage_config) + logger.info( + "===========================================================\n" + f"> Stage {i + 1}/{len(config.stages)} finished.\n" + "===========================================================" + ) + prev_stage_checkpoint = get_latest_hf_checkpoint_path(stage_config) + else: + config.check_and_update() + run_stage(config) + + finally: + if dlc: + stop_ray_cluster(namespace=cluster_namespace) + + +def studio(port: int = 8501): + from streamlit.web import cli as stcli + + current_dir = Path(__file__).resolve().parent.parent + config_manager_path = os.path.join(current_dir, "manager", "config_manager.py") + + sys.argv = [ + "streamlit", + "run", + config_manager_path, + "--server.port", + str(port), + "--server.fileWatcherType", + "none", + ] + sys.exit(stcli.main()) + + +def debug( + config_path: str, + module: str, + output_dir: str = "debug_output", + disable_overwrite: bool = False, + enable_profiling: bool = False, + port: int = 8502, + plugin_dir: str = None, +): + """Debug a module.""" + if plugin_dir: + os.environ[PLUGIN_DIRS_ENV_VAR] = plugin_dir + load_plugins() + config = load_config(config_path) + config.mode = "explore" + config.check_and_update() + sys.path.insert(0, os.getcwd()) + config.ray_namespace = DEBUG_NAMESPACE + ray.init( + namespace=config.ray_namespace, + runtime_env={"env_vars": config.get_envs()}, + ignore_reinit_error=True, + ) + from trinity.common.models import create_debug_inference_model + + if module == "inference_model": + asyncio.run(create_debug_inference_model(config)) + + elif module == "workflow": + from trinity.explorer.workflow_runner import DebugWorkflowRunner + + runner = DebugWorkflowRunner(config, output_dir, enable_profiling, disable_overwrite) + asyncio.run(runner.debug()) + elif module == "viewer": + from streamlit.web import cli as stcli + + current_dir = Path(__file__).resolve().parent.parent + viewer_path = os.path.join(current_dir, "buffer", "viewer.py") + output_dir_abs = os.path.abspath(output_dir) + if output_dir_abs.endswith("/"): + output_dir_abs = output_dir_abs[:-1] + print(f"sqlite:///{output_dir_abs}/experiences.db") + sys.argv = [ + "streamlit", + "run", + viewer_path, + "--server.port", + str(port), + "--server.fileWatcherType", + "none", + "--", + "--db-url", + f"sqlite:///{output_dir_abs}/experiences.db", + "--table", + "debug_buffer", + "--tokenizer", + config.model.model_path, + ] + sys.exit(stcli.main()) + else: + raise ValueError( + f"Only support 'inference_model' and 'workflow' for debugging, got {module}" + ) + + +def convert(checkpoint_dir: str, base_model_dir: Optional[str] = None) -> None: + if "global_step_" in checkpoint_dir: + while not os.path.basename(checkpoint_dir).startswith("global_step_"): + checkpoint_dir = os.path.dirname(checkpoint_dir) + converter = Converter(base_model_dir) + converter.convert(checkpoint_dir) + + +def main() -> None: + """The main entrypoint.""" + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(dest="command", required=True) + + # run command + run_parser = subparsers.add_parser("run", help="Run RFT process.") + run_parser.add_argument("--config", type=str, required=True, help="Path to the config file.") + run_parser.add_argument( + "--plugin-dir", + type=str, + default=None, + help="Path to the directory containing plugin modules.", + ) + run_parser.add_argument( + "--dlc", action="store_true", help="Specify when running in Aliyun PAI DLC." + ) + + # studio command + studio_parser = subparsers.add_parser("studio", help="Run studio.") + studio_parser.add_argument( + "--port", type=int, default=8501, help="The port for Trinity-Studio." + ) + + # debug command + debug_parser = subparsers.add_parser("debug", help="Debug the code.") + debug_parser.add_argument("--config", type=str, help="Path to the config file.") + debug_parser.add_argument( + "--module", + type=str, + choices=["inference_model", "workflow", "viewer"], + help="The module to start debugging, only support 'inference_model', 'workflow' and 'viewer' for now.", + ) + debug_parser.add_argument( + "--plugin-dir", + type=str, + default=None, + help="Path to the directory containing plugin modules.", + ) + debug_parser.add_argument( + "--output-dir", + type=str, + default="debug_output", + help="The output directory for debug files.", + ) + debug_parser.add_argument( + "--disable-overwrite", action="store_true", help="Disable overwriting the output directory." + ) + debug_parser.add_argument( + "--enable-profiling", + action="store_true", + help="Whether to use viztracer for workflow profiling.", + ) + debug_parser.add_argument( + "--output-file", + type=str, + default=None, + help="[DEPRECATED] Please use --output-dir instead.", + ) + debug_parser.add_argument( + "--port", + type=int, + default=8502, + help="The port for Experience Viewer.", + ) + + convert_parser = subparsers.add_parser( + "convert", help="Convert checkpoint to huggingface format." + ) + convert_parser.add_argument( + "--checkpoint-dir", + type=str, + required=True, + help="The path to the checkpoint directory.", + ) + convert_parser.add_argument( + "--base-model-dir", + type=str, + default=None, + help="The path to the base model.", + ) + + args = parser.parse_args() + if args.command == "run": + # TODO: support parse all args from command line + run(args.config, args.dlc, args.plugin_dir) + elif args.command == "studio": + studio(args.port) + elif args.command == "debug": + debug( + args.config, + args.module, + args.output_dir, + args.disable_overwrite, + args.enable_profiling, + args.port, + args.plugin_dir, + ) + elif args.command == "convert": + convert(args.checkpoint_dir, args.base_model_dir) + + +if __name__ == "__main__": + main() diff --git a/frameworks/trinity/cli/server.py b/frameworks/trinity/cli/server.py new file mode 100644 index 0000000000000000000000000000000000000000..bb792752ca25c457d7f4ede24b3d60d5bbf454b3 --- /dev/null +++ b/frameworks/trinity/cli/server.py @@ -0,0 +1,32 @@ +import traceback + +import fire +from flask import Flask, jsonify, request + +app = Flask(__name__) + +APP_NAME = "trinity_rft" + + +@app.route(f"/{APP_NAME}", methods=["GET"]) +def trinity_training(): + config_path = request.args.get("configPath") + try: + from trinity.cli.launcher import run + + run(config_path) + ret = 0 + msg = "Training Success." + except: # noqa: E722 + traceback.print_exc() + msg = traceback.format_exc() + ret = 1 + return jsonify({"return_code": ret, "message": msg}) + + +def main(port=5006): + app.run(port=port, debug=True) + + +if __name__ == "__main__": + fire.Fire(main) diff --git a/frameworks/trinity/common/__init__.py b/frameworks/trinity/common/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/frameworks/trinity/common/config.py b/frameworks/trinity/common/config.py new file mode 100644 index 0000000000000000000000000000000000000000..916a0514e941bda77cecdd84b9e8dd7510b69089 --- /dev/null +++ b/frameworks/trinity/common/config.py @@ -0,0 +1,952 @@ +# -*- coding: utf-8 -*- +"""Configs for RFT.""" +from __future__ import annotations + +import os +from copy import deepcopy +from dataclasses import dataclass, field +from enum import Enum +from typing import Any, Dict, List, Optional + +from omegaconf import OmegaConf + +from trinity.common.constants import ( + EXPLORER_NAME, + LOG_DIR_ENV_VAR, + LOG_LEVEL_ENV_VAR, + LOG_NODE_IP_ENV_VAR, + PLUGIN_DIRS_ENV_VAR, + TRAINER_NAME, + PromptType, + SaveStrategy, + StorageType, + SyncMethod, + SyncStyle, +) +from trinity.utils.annotations import Experimental +from trinity.utils.log import get_logger + +logger = get_logger(__name__) + + +def set_if_none(obj, attr, val): + if getattr(obj, attr, None) is None: + setattr(obj, attr, val) + + +@dataclass +class FormatConfig: + """Configuration for data formatting""" + + # for sft / dpo + prompt_type: PromptType = PromptType.MESSAGES + + # for plaintext input + prompt_key: str = "prompt" # user prompt + response_key: str = "response" # assistant response + system_prompt_key: Optional[str] = None # If set, use the provided system prompt + system_prompt: Optional[str] = None # has lower priority than system_prompt_key + + # for message list input + messages_key: str = "message" + + # for tools + tools_key: str = "tools" + image_key: Optional[str] = None # used for multi-modal data + video_key: Optional[str] = None # used for multi-modal data + + reply_prefix: Optional[str] = None + + # for sample-level task controlling + workflow_key: str = "" + reward_fn_key: str = "" + + # for dpo dataset + chosen_key: str = "chosen" + rejected_key: str = "rejected" + + # for multi-turn sft + enable_concatenated_multi_turn: bool = False + + # for sft / dpo, if None, use model.custom_chat_template + chat_template: Optional[str] = None + + # for teacher-student distillation: optional key in raw_task for teacher hint (e.g. socratic reasoning) + # when set, format_messages_teacher() can include raw_task[teacher_hint_key] in teacher prompt + teacher_hint_key: Optional[str] = None + + +@dataclass +class GenerationConfig: + temperature: Optional[float] = None # 1.0 + top_p: Optional[float] = None # 1.0 + top_k: int = -1 # -1 means disabled + logprobs: Optional[int] = None # 0 # vLLM return `logprobs + 1` elements + max_tokens: Optional[int] = None # if None, use model.max_response_tokens + # repeat each task for `n` times + # ! DO NOT SET, it will be set by `algorithm.repeat_times` or `buffer.explorer_input.eval_tasksets[i].repeat_times` + n: int = 1 + + +@dataclass +class OptimizerConfig: + lr: float = 1e-6 + lr_warmup_steps: int = -1 + lr_warmup_steps_ratio: float = 0.0 + min_lr_ratio: float = 0.0 + warmup_style: Optional[str] = None # deprecated ! + lr_scheduler_type: str = "constant" + optimizer_type: str = "adam" + betas: List[float] = field(default_factory=lambda: [0.9, 0.999]) + weight_decay: float = 0.01 + clip_grad: float = 1.0 + + +@dataclass +class LoRAConfig: + """LoRA config, only effective for rollout model, not for auxiliary models.""" + + name: Optional[str] = None + path: Optional[str] = None + base_model_name: Optional[str] = None + lora_rank: int = 32 + lora_alpha: int = 32 + lora_dtype: str = "auto" + target_modules: str = "all-linear" + exclude_modules: Optional[str] = None + is_dummy: bool = False # DO NOT SET, automatically set + + +@Experimental +@dataclass +class TaskSelectorConfig: + """Data selector config.""" + + selector_type: Optional[str] = "sequential" + + # For shuffle + seed: int = 42 + + # Estimator Config + feature_keys: List[str] = field(default_factory=lambda: []) + kwargs: dict = field(default_factory=dict) + + +@dataclass +class ReplayBufferConfig: + """Config for replay buffer used in StorageType.QUEUE.""" + + enable: bool = False + priority_fn: str = "linear_decay" + reuse_cooldown_time: Optional[float] = None + priority_fn_args: Dict = field(default_factory=lambda: {"decay": 2.0}) + + +@dataclass +class OverRolloutConfig: + """Config for over-rollout in explorer.""" + + ratio: float = 0.0 # explorer will only wait for (1 - over_rollout.ratio) * batch_size of tasks at each step + wait_after_min: float = 30.0 # wait 30 s after reaching minimum task threshold + # more settings will be added in the future + # e.g., postpone tasks into the next step if not finished in time + + +@dataclass +class DynamicTimeoutConfig: + """Config for dynamic timeout in explorer.""" + + enable: bool = False + ratio: float = 3.0 # the timeout for each step will be min(max_timeout, average_time_per_task * dynamic_timeout.ratio) + + +@dataclass +class StorageConfig: + """Storage config for both taskset and experience buffer. + Not visible to users directly. Please use ExperienceBufferConfig or TasksetConfig instead. + """ + + name: str = "" + storage_type: str = StorageType.FILE.value + path: Optional[str] = None + repeat_times: Optional[int] = None + + # For continuing training + index: int = 0 + + # used for StorageType.FILE + split: str = "train" + subset_name: Optional[str] = None + format: FormatConfig = field(default_factory=FormatConfig) + + # used for StorageType.QUEUE + capacity: int = 10000 + max_read_timeout: float = 1800 + replay_buffer: Optional[ReplayBufferConfig] = field(default_factory=ReplayBufferConfig) + + # used for StorageType.SQL + max_retry_times: int = 3 + max_retry_interval: int = 1 + + # used for rollout tasks + default_workflow_type: Optional[str] = None + default_reward_fn_type: Optional[str] = None + rollout_args: GenerationConfig = field(default_factory=GenerationConfig) + workflow_args: dict = field(default_factory=dict) + reward_fn_args: dict = field(default_factory=dict) + task_selector: TaskSelectorConfig = field(default_factory=TaskSelectorConfig) + + # enable progress bar (tqdm) for _HFBatchReader + enable_progress_bar: Optional[bool] = False + + # get storage from existing experiment + ray_namespace: Optional[str] = None + + # ! DO NOT SET except you know what you are doing + wrap_in_ray: bool = True + + # ! DO NOT SET, automatically set + schema_type: Optional[str] = None + + # ! DO NOT SET, automatically set from buffer.total_epochs + total_epochs: int = 1 # automatically set + + # ! DO NOT SET, automatically set from buffer.total_steps + total_steps: Optional[int] = None # automatically set + + # ! DO NOT SET, automatically set from buffer.batch_size / train_batch_size + batch_size: int = 0 + + # ! DO NOT SET, automatically set from model.model_path + tokenizer_path: Optional[str] = None + + # ! DO NOT SET, automatically set corresponding to train/eval + is_eval: bool = False + + +@dataclass +class TasksetConfig: + name: str = "" + storage_type: str = StorageType.FILE.value + path: Optional[str] = None + + default_workflow_type: Optional[str] = None + default_reward_fn_type: Optional[str] = None + rollout_args: GenerationConfig = field(default_factory=GenerationConfig) + workflow_args: dict = field(default_factory=dict) + reward_fn_args: dict = field(default_factory=dict) + task_selector: TaskSelectorConfig = field(default_factory=TaskSelectorConfig) + + # used for StorageType.FILE + split: str = "train" + subset_name: Optional[str] = None + format: FormatConfig = field(default_factory=FormatConfig) + + # used for StorageType.SQL + max_retry_times: int = 3 + max_retry_interval: int = 1 + + enable_progress_bar: bool = False + + # ! This setting is only valid for `eval_taskset`; for other taskset, it will be overridden by `algorithm.repeat_times`. + repeat_times: int = 1 + # ! DO NOT SET, automatically load from checkpoint + index: int = 0 + # ! DO NOT SET, automatically set based on train/eval + is_eval: bool = False + # ! DO NOT SET, automatically set from buffer.batch_size + batch_size: int = 0 + # ! DO NOT SET, automatically set from buffer.total_epochs + total_epochs: int = 1 # automatically set + # ! DO NOT SET, automatically set from buffer.total_steps + total_steps: Optional[int] = None # automatically set + # ! DO NOT SET, automatically set form ray_namespace + ray_namespace: Optional[str] = None + + def to_storage_config(self) -> StorageConfig: + storage_config = StorageConfig( + name=self.name, + storage_type=self.storage_type, + path=self.path, + task_selector=self.task_selector, + repeat_times=self.repeat_times, + split=self.split, + subset_name=self.subset_name, + format=self.format, + max_retry_times=self.max_retry_times, + max_retry_interval=self.max_retry_interval, + default_workflow_type=self.default_workflow_type, + default_reward_fn_type=self.default_reward_fn_type, + rollout_args=self.rollout_args, + workflow_args=self.workflow_args, + reward_fn_args=self.reward_fn_args, + enable_progress_bar=self.enable_progress_bar, + index=self.index, + is_eval=self.is_eval, + batch_size=self.batch_size, + total_epochs=self.total_epochs, + total_steps=self.total_steps, + ray_namespace=self.ray_namespace, + ) + return storage_config + + +@dataclass +class ExperienceBufferConfig: + """Storage Config for trainer input experience buffer.""" + + name: str = "" + storage_type: str = StorageType.QUEUE.value + path: Optional[str] = None + + # used for StorageType.QUEUE + capacity: int = 10000 + max_read_timeout: float = 1800 + replay_buffer: Optional[ReplayBufferConfig] = field(default_factory=ReplayBufferConfig) + + # used for StorageType.SQL + max_retry_times: int = 3 + max_retry_interval: int = 1 + + # used for StorageType.FILE + split: str = "train" + subset_name: Optional[str] = None + format: FormatConfig = field(default_factory=FormatConfig) + enable_progress_bar: Optional[bool] = False + + # ! DO NOT SET, automatically set + schema_type: Optional[str] = None + # ! DO NOT SET + index: int = 0 + # ! DO NOT SET, automatically set from buffer.batch_size + batch_size: int = 0 + # ! DO NOT SET, automatically set from model.model_path + tokenizer_path: Optional[str] = None + # ! DO NOT SET, automatically set from buffer.total_epochs + total_epochs: int = 1 # automatically set + # ! DO NOT SET, automatically set from buffer.total_steps + total_steps: Optional[int] = None # automatically set + # ! DO NOT SET, automatically set form ray_namespace + ray_namespace: Optional[str] = None + + def to_storage_config(self) -> StorageConfig: + storage_config = StorageConfig( + name=self.name, + storage_type=self.storage_type, + path=self.path, + capacity=self.capacity, + max_read_timeout=self.max_read_timeout, + replay_buffer=self.replay_buffer, + max_retry_times=self.max_retry_times, + max_retry_interval=self.max_retry_interval, + split=self.split, + subset_name=self.subset_name, + format=self.format, + enable_progress_bar=self.enable_progress_bar, + schema_type=self.schema_type, + index=self.index, + batch_size=self.batch_size, + tokenizer_path=self.tokenizer_path, + total_epochs=self.total_epochs, + total_steps=self.total_steps, + ray_namespace=self.ray_namespace, + ) + return storage_config + + +@dataclass +class OperatorConfig: + name: str = "" + args: Dict[str, Any] = field(default_factory=dict) + + +@Experimental +@dataclass +class ExperiencePipelineConfig: + """Config for experience pipeline. + + Experience Pipeline is used to pre-process rollout experiences for better training. + """ + + # The list of experience operators to apply, operators will be applied in the order they are defined + operators: List[OperatorConfig] = field(default_factory=list) + save_input: bool = True # whether to save the input experiences + # the path to save the input experiences, can be a jsonl file or a sqlite database file + input_save_path: Optional[str] = None + + # The following fields are experimental, do not set them unless you know what you are doing + + # A dictionary of input buffers, buffers are indexed by their names. + # users only need to set extra buffers here + inputs: Dict[str, ExperienceBufferConfig] = field(default_factory=dict) + # The output buffer will automatically set to the trainer input buffer, so we do not need to set it here. + output: Optional[ExperienceBufferConfig] = None + + +@Experimental +@dataclass +class TaskPipelineConfig: + """Config for task pipeline. + + Task Pipeline is used to pre-process raw tasks for better exploring. Currently, we only support using + Data-Juicer operators for task pipeline. + """ + + # The list of data-juicer operators to apply, operators will be applied in the order they are defined + operators: List[OperatorConfig] = field(default_factory=list) + # number of process + num_process: int = 4 + # The path to the Data-Juicer config file. If set, operators and num_process will be ignored + config_path: Optional[str] = None + + # Raw input tasksets. Currently, task pipeline only support local file as inputs, + # e.g., /path/to/file.jsonl or /path/to/file.parquet, not a directory or huggingface path + inputs: List[str] = field(default_factory=list) + # Output task buffer, if not set, use `buffer.explorer_input.taskset`. In most cases, users do not need to set this field. + output: Optional[TasksetConfig] = None + + # The list of fields extracted from the input tasksets and processed into the output taskset + target_fields: List[str] = field(default_factory=list) + + # weights for priority computing. Usually including 4 types of weights: + # - difficulty + # - diversity + # - usage_frequency + # - quality + priority_weights: Dict[str, float] = field(default_factory=dict) + + # number of samples to select after task pipeline. -1 means all + top_k: int = -1 + + +@Experimental +@dataclass +class DataProcessorConfig: + """Data Processor config""" + + # support two types of data pipelines for now + # 1. For task. Data preprocessing from raw dataset to the task set + task_pipeline: Optional[TaskPipelineConfig] = None + # 2. For experience. Data processing for rollouts + experience_pipeline: Optional[ExperiencePipelineConfig] = field( + default_factory=ExperiencePipelineConfig + ) + + +@dataclass +class TinkerConfig: + enable: bool = False + rank: int = 32 # lora rank + seed: Optional[int] = None + train_mlp: bool = True + train_attn: bool = True + train_unembed: bool = True + + +@dataclass +class ModelConfig: + # source model path + model_path: str = "" + critic_model_path: str = "" + + custom_chat_template: Optional[str] = None + chat_template_path: Optional[ + str + ] = None # path to the chat template file, e.g., jinja2 type; overrides `custom_chat_template` if set + + # rollout args + temperature: float = 1.0 + top_p: float = 1.0 + top_k: int = -1 + logprobs: int = 0 + + # the total number of tokens the model can handle + max_model_len: Optional[int] = None + + # Note: the following fields are only for the `chat`/`generate` methods in `InferenceModel` + # if you are using openai API, please set them when calling the API. + + # the maximum number of tokens for the prompt + max_prompt_tokens: Optional[int] = None + # the maximum number of tokens for the response + max_response_tokens: Optional[int] = None + # the minimum number of tokens for the response + min_response_tokens: int = 0 + # whether to truncate the prompt; if set to True, the prompt will be truncated to `max_prompt_tokens` tokens; + # not applicable for OpenAI API + enable_prompt_truncation: bool = True + # repetition penalty for response generation + repetition_penalty: float = 1.0 + + # lora config + lora_configs: Optional[List[LoRAConfig]] = None + fully_sharded_loras: bool = False + max_cpu_loras: Optional[int] = None + + # rope config + rope_scaling: Optional[dict] = None + rope_theta: Optional[float] = None + + # tinker config + tinker: TinkerConfig = field(default_factory=TinkerConfig) + + +@dataclass +class InferenceModelConfig: + # ! DO NOT SET in explorer.rollout_model, automatically set from config.model.model_path + model_path: Optional[str] = None + name: Optional[str] = None + + engine_type: str = "vllm" + engine_num: int = 1 + tensor_parallel_size: int = 1 + use_v1: bool = True + enforce_eager: bool = False + enable_prefix_caching: bool = True + enable_chunked_prefill: bool = True + gpu_memory_utilization: float = 0.9 + dtype: str = "bfloat16" + seed: int = 42 + + # rollout args, ! DO NOT SET + temperature: Optional[float] = None + top_p: Optional[float] = None + top_k: Optional[int] = None + logprobs: Optional[int] = None + + # if not set, use `model.max_model_len` + max_model_len: Optional[int] = None + # if not set, use `model.max_prompt_tokens` + max_prompt_tokens: Optional[int] = None + # if not set, use `model.max_response_tokens` + max_response_tokens: Optional[int] = None + # if not set, use `model.min_response_tokens` + min_response_tokens: Optional[int] = None + # if not set, use `model.enable_prompt_truncation` + enable_prompt_truncation: Optional[bool] = None + # If not set, use `model.repetition_penalty` + repetition_penalty: Optional[float] = None + # used for testing very long response generation, do not set it unless you know what you are doing + ignore_eos: bool = False + + # override chat template in model + chat_template: Optional[str] = None + + # For Qwen3 + enable_thinking: bool = False + + # For history recording + enable_history: bool = False + + # For OpenAI API + enable_openai_api: bool = False + enable_log_requests: bool = False # whether to enable request logging in vLLM API server + + # For tool calls in OpenAI API + enable_auto_tool_choice: bool = False + + tool_call_parser: Optional[str] = None + + reasoning_parser: Optional[str] = None + + # ! DO NOT SET + bundle_indices: str = "" + ray_namespace: Optional[str] = None + + # ! DO NOT SET, automatically set from model.lora_configs + enable_lora: bool = False + enable_runtime_lora_updating: bool = False + lora_modules: Optional[List[Dict]] = None + lora_kwargs: Optional[dict] = field(default_factory=dict) + + # ! DO NOT SET, rope config + rope_scaling: Optional[dict] = None + rope_theta: Optional[float] = None + + +@dataclass +class AlgorithmConfig: + """Config for algorithm.""" + + algorithm_type: str = "ppo" + # for GRPO-like algorithms, repeat each task for `repeat_times` times + repeat_times: int = 1 + + optimizer: OptimizerConfig = field(default_factory=OptimizerConfig) + + # the strategy for sampling experiences from the buffer + sample_strategy: Optional[str] = None + sample_strategy_args: Optional[dict] = None + + advantage_fn: Optional[str] = None # "ppo" + # If not set, use AdvantageFn.default_args() + advantage_fn_args: Optional[dict] = None + + kl_penalty_fn: Optional[str] = None # "none" # set to "none" to disable kl penalty in reward + # If not set, use kl_penalty_fn.default_args() + kl_penalty_fn_args: Optional[dict] = None + + policy_loss_fn: Optional[str] = None # "ppo" + # If not set, use PolicyLossFn.default_args() + policy_loss_fn_args: Optional[dict] = None + + kl_loss_fn: Optional[str] = None # "k2" # set to "none" to disable kl loss + # If not set, use kl_loss_fn.default_args() + kl_loss_fn_args: Optional[dict] = None + + entropy_loss_fn: Optional[str] = None # "default" + # If not set, use entropy_loss_fn.default_args() + entropy_loss_fn_args: Optional[dict] = None + + # aggregation mode for losses: 'token-mean' or 'seq-mean-token-sum' or 'seq-mean-token-mean' or 'seq-mean-token-sum-norm' + # If not set, use 'token-mean' + loss_agg_mode: Optional[str] = None + + +@dataclass +class ClusterConfig: + """Config for the cluster.""" + + ray_address: str = "auto" + node_num: int = 0 + gpu_per_node: int = 0 + + # ! DO NOT SET + total_gpu_num: int = 0 + rollout_gpu_num: int = 0 + auxiliary_model_gpu_num: int = 0 + explorer_gpu_num: int = 0 + trainer_gpu_num: int = 0 + trainer_node_num: int = 0 + trainer_gpu_num_per_node: int = 0 + + +@Experimental +@dataclass +class ExplorerInput: + """Config for explorer input.""" + + taskset: Optional[TasksetConfig] = None + tasksets: List[TasksetConfig] = field(default_factory=list) + eval_tasksets: List[TasksetConfig] = field(default_factory=list) + # The following args provide default values for the corresponding args in `taskset` and `eval_tasksets` + default_workflow_type: Optional[str] = None + default_eval_workflow_type: Optional[str] = None + default_reward_fn_type: Optional[str] = None + + +@dataclass +class TrainerInput: + """Config for trainer input.""" + + # The main experience buffer to be used in trainer + # Commonly, it is also the output buffer of the Explorer + experience_buffer: Optional[ExperienceBufferConfig] = None + + # Some auxiliary buffers to facilitate training (e.g., data mixing) + auxiliary_buffers: Dict[str, ExperienceBufferConfig] = field(default_factory=dict) + + +@dataclass +class BufferConfig: + """Config for buffer.""" + + batch_size: int = 1 + train_batch_size: int = 0 # default to `batch_size` * `algorithm.n` + total_epochs: int = 1 + total_steps: Optional[int] = None + + # for explorer + explorer_input: ExplorerInput = field(default_factory=ExplorerInput) + + # for trainer + trainer_input: TrainerInput = field(default_factory=TrainerInput) + + # ! DO NOT SET FOLLOWING FIELDS + explorer_output: Optional[StorageConfig] = None # automatically set + tokenizer_path: Optional[str] = None # automatically set + pad_token_id: Optional[int] = None # automatically set + cache_dir: Optional[str] = None # automatically set + + +@dataclass +class ExplorerConfig: + """Config for explorer.""" + + name: str = EXPLORER_NAME + # for workflow runner + # number of workflow runners. + runner_per_model: int = 8 # number of runners per each rollout model + max_timeout: int = 1800 # wait each task for 30 minutes at most + max_retry_times: int = 2 # retry each task for 2 times if it fails or timeout + env_vars: dict = field(default_factory=dict) # environment variables for workflow runner + + # Workflow Runner Configs for tasks requiring group execution + # how to run a group of tasks in a single workflow runner + # "sequential": run tasks one by one, no requirements on workflow design, but have lower throughput + # "asynchronous": run tasks asynchronously, requires the workflow to be designed with async/await + # syntax, and no sharing of state between tasks + # "multi-threading": run tasks using multi-threading, requires the workflow to be thread-safe, + # and no sharing of state between tasks + concurrent_mode: str = "sequential" + # the number of time to repeat each task in a single workflow runner + # we recommend setting this only when using "sequential" concurrent_mode + max_repeat_times_per_runner: Optional[int] = None + + runner_num: Optional[int] = None # ! Deprecated + + # for inference models + # for rollout model + rollout_model: InferenceModelConfig = field(default_factory=InferenceModelConfig) + # for other models used in the custom workflows + auxiliary_models: List[InferenceModelConfig] = field(default_factory=list) + + # for evaluation + eval_interval: int = 100 + eval_on_startup: bool = True # evalulate at step 0 + + # for benchmark + bench_on_latest_checkpoint: bool = False # only benchmark the latest checkpoint + + # for serve mode proxy + proxy_port: int = 8010 + # listen on all interfaces by default + listen_address: str = "0.0.0.0" + # check the running status of the server every 60 seconds + service_status_check_interval: int = 60 + # keep at least 1 model in running status + min_running_model_num: int = 1 + # db url for proxy history recorder, if not set, use proxy_history.db in buffer cache dir + db_url: Optional[str] = None + + # Experimental feature + over_rollout: OverRolloutConfig = field(default_factory=OverRolloutConfig) + dynamic_timeout: DynamicTimeoutConfig = field(default_factory=DynamicTimeoutConfig) + # report runner state every `runner_state_report_interval` seconds, 0 to disable + runner_state_report_interval: int = 0 + + +@dataclass +class TrainerConfig: + name: str = TRAINER_NAME + trainer_type: str = "verl" + trainer_strategy: str = "fsdp" # "fsdp", "fsdp2" or "megatron" + save_interval: int = 0 + enable_preview: bool = True # enable rollout preview in wandb + total_steps: Optional[ + int + ] = None # total training steps, training stops when reaching this step, None means no limit + + save_hf_checkpoint: str = "last" # whether to save checkpoint in HuggingFace format + # "always": save all checkpoints in HF format + # "never": never save checkpoint in HF format + # "last": only save the last checkpoint in HF format + + # trainer configs + grad_clip: float = 1.0 + use_dynamic_bsz: bool = True + # if None, automatically set to ceil(2 * model.max_model_len / ulysses_sequence_parallel_size) + max_token_len_per_gpu: Optional[int] = None + ulysses_sequence_parallel_size: int = 1 # sp size + fix_actor_microbatch_loss_scale: bool = False # EXPERIMENTAL + # TODO: extract more train-related params from underlying trainer engine + + save_strategy: SaveStrategy = SaveStrategy.UNRESTRICTED + max_checkpoints_to_keep: Optional[int] = None + + trainer_config: Any = field(default_factory=dict) + trainer_config_path: str = "" # deprecated, use `trainer_config` instead + + +@dataclass +class MonitorConfig: + # TODO: support multiple monitors (List[str]) + monitor_type: str = "tensorboard" + # the default args for monitor + monitor_args: Optional[Dict] = None + # whether to return detailed stats (mean, std, max, min) for evaluation metrics + detailed_stats: bool = False + # whether to enable ray timeline profile + # the output file will be saved to `cache_dir/timeline.json` + enable_ray_timeline: bool = False + # ! DO NOT SET, automatically generated as checkpoint_job_dir/monitor + cache_dir: str = "" + + +@dataclass +class SynchronizerConfig: + """Configs for model weight synchronization.""" + + sync_method: SyncMethod = SyncMethod.NCCL + sync_style: SyncStyle = SyncStyle.FIXED + # sync weights every `sync_interval` steps + sync_interval: int = 1 + # allow explorer to run `sync_offset` steps before sync + sync_offset: int = 0 + # waiting for `sync_timeout` seconds before timeout in `nccl` method + sync_timeout: int = 3600 + # wait for the lastest checkpoint to be ready # TODO: to be used + wait_for_checkpoint: bool = False + + # ! DO NOT SET, automatically calculated + explorer_world_size: Optional[int] = None + ray_namespace: str = "" + + +@dataclass +class DataJuicerServiceConfig: + """Config for Data-Juicer. + + Please update `trinity.service.data_juicer.server.server.py` correspondingly if you change the fields here. + """ + + # the url of the Data-Juicer server + server_url: Optional[str] = None + + # whether to start Data-Juicer server automatically + auto_start: bool = False + + # the following fields are only used when `auto_start` is True + # the port of the Data-Juicer server, if not set, a random port will be used + port: Optional[int] = None + # the hostname will be automatically set to "localhost" so we do not need to set it here + + +@dataclass +class ServiceConfig: + """Configs for outside services.""" + + data_juicer: Optional[DataJuicerServiceConfig] = None + + +@dataclass +class LogConfig: + """Configs for logger.""" + + level: str = "INFO" # default log level (DEBUG, INFO, WARNING, ERROR) + group_by_node: bool = False # whether to group logs by node IP in Ray cluster + # ! DO NOT SET, automatically generated as ///log + save_dir: str = "" + + +@dataclass +class StageConfig: + """Configs for a stage.""" + + stage_name: str + mode: Optional[str] = None + algorithm: Optional[AlgorithmConfig] = None + buffer: Optional[BufferConfig] = None + data_processor: Optional[DataProcessorConfig] = None + explorer: Optional[ExplorerConfig] = None + trainer: Optional[TrainerConfig] = None + + +@dataclass +class Config: + """Global Configuration""" + + mode: str = "both" # `explore`, `train`, `both` or `bench` + project: str = "Trinity-RFT" + group: str = "" + name: str = "rft" + # the root dir for checkpoints + checkpoint_root_dir: str = "" + # ! DO NOT SET, automatically generated as `checkpoint_root_dir/project/name` + checkpoint_job_dir: str = "" + # If not set, automatically generated as f"{config.project}-{config.name}" + ray_namespace: str = "" + # whether to continue training from the last checkpoint in checkpoint_job_dir (if any) + continue_from_checkpoint: bool = True + + algorithm: AlgorithmConfig = field(default_factory=AlgorithmConfig) + data_processor: DataProcessorConfig = field(default_factory=DataProcessorConfig) + model: ModelConfig = field(default_factory=ModelConfig) + cluster: ClusterConfig = field(default_factory=ClusterConfig) + buffer: BufferConfig = field(default_factory=BufferConfig) + explorer: ExplorerConfig = field(default_factory=ExplorerConfig) + trainer: TrainerConfig = field(default_factory=TrainerConfig) + monitor: MonitorConfig = field(default_factory=MonitorConfig) + synchronizer: SynchronizerConfig = field(default_factory=SynchronizerConfig) + service: ServiceConfig = field(default_factory=ServiceConfig) + log: LogConfig = field(default_factory=LogConfig) + + # configurations for different training stages + stages: List[StageConfig] = field(default_factory=list) + + def save(self, config_path: str) -> None: + """Save config to file.""" + with open(config_path, "w", encoding="utf-8") as f: + OmegaConf.save(self, f) + + def __iter__(self): + """Iterate over configs with each stage applied in order. + + Yields: + Config: The config after applying each stage. + """ + for stage in self.stages: + new_config = deepcopy(self) + for field_name in stage.__dataclass_fields__: + stage_value = getattr(stage, field_name) + if stage_value is not None and hasattr(new_config, field_name): + setattr(new_config, field_name, stage_value) + if stage.stage_name: + new_config.name = f"{self.name}/{stage.stage_name}" + # set trainer.save_hf_checkpoint to "last" to make sure next stage can load from HF checkpoint + new_config.trainer.save_hf_checkpoint = "last" + new_config.stages = [] + yield new_config + + def check_and_update(self) -> Config: + """Check and update the config.""" + from trinity.common.config_validator import validators + + # validate + for validator in validators: + validator.validate(self) + return self + + def flatten(self) -> Dict[str, Any]: + """Flatten the config into a single-level dict with dot-separated keys for nested fields.""" + + def _flatten(obj, parent_key="", sep="."): + items = {} + if hasattr(obj, "__dataclass_fields__"): + obj = vars(obj) + if isinstance(obj, dict): + for k, v in obj.items(): + new_key = f"{parent_key}{sep}{k}" if parent_key else k + items.update(_flatten(v, new_key, sep=sep)) + elif isinstance(obj, list): + for i, v in enumerate(obj): + new_key = f"{parent_key}{sep}{i}" if parent_key else str(i) + items.update(_flatten(v, new_key, sep=sep)) + elif isinstance(obj, Enum): + items[parent_key] = obj.value + else: + items[parent_key] = obj + return items + + return _flatten(self) + + def get_envs(self) -> Dict[str, str]: + """Get the environment variables from the config.""" + return { + PLUGIN_DIRS_ENV_VAR: os.getenv(PLUGIN_DIRS_ENV_VAR, ""), + LOG_LEVEL_ENV_VAR: self.log.level, + LOG_DIR_ENV_VAR: self.log.save_dir, + LOG_NODE_IP_ENV_VAR: "1" if self.log.group_by_node else "0", + } + + +def load_config(config_path: str) -> Config: + """Load the configuration from the given path.""" + # TODO: add test + schema = OmegaConf.structured(Config) + yaml_config = OmegaConf.load(config_path) + try: + config = OmegaConf.merge(schema, yaml_config) + return OmegaConf.to_object(config) + except Exception as e: + raise ValueError(f"Invalid configuration: {e}") from e diff --git a/frameworks/trinity/common/config_validator.py b/frameworks/trinity/common/config_validator.py new file mode 100644 index 0000000000000000000000000000000000000000..34a082c4e6b91a2a75ffc857e06c29af8928559e --- /dev/null +++ b/frameworks/trinity/common/config_validator.py @@ -0,0 +1,1126 @@ +import math +import os +from abc import ABC, abstractmethod +from datetime import datetime + +import ray +from omegaconf import OmegaConf + +from trinity.common.config import ( + Config, + ExperienceBufferConfig, + TasksetConfig, + set_if_none, +) +from trinity.common.constants import StorageType, SyncMethod, SyncStyle +from trinity.utils.log import get_logger +from trinity.utils.lora_utils import create_dummy_lora + + +class ConfigValidator(ABC): + """Abstract base class for configuration validators. + + Each validator is responsible for checking and potentially modifying specific + aspects of the global configuration to ensure validity, set defaults, or handle + deprecated settings. + """ + + def __init__(self): + self.logger = get_logger(__name__) + + @abstractmethod + def validate(self, config: Config) -> None: + """Validate and potentially modify the given configuration. + + Args: + config: The global configuration object to validate and modify. + """ + pass + + +class DeprecatedConfigValidator(ConfigValidator): + """Validator for handling deprecated configuration options. + + Issues warnings when deprecated configuration parameters are used and suggests + their replacements. + """ + + def validate(self, config: Config) -> None: + """Check for deprecated configuration options and issue warnings. + + Specifically checks for the deprecated `explorer.runner_num` parameter + and recommends using `explorer.runner_per_model` instead. + + Args: + config: The global configuration object to validate. + """ + if config.explorer.runner_num is not None: + self.logger.warning( + "`explorer.runner_num` is deprecated, " + "please use `explorer.runner_per_model` instead." + ) + + +class GlobalConfigValidator(ConfigValidator): + """Validator for global configuration settings. + + Handles validation of the main operating mode, sets up checkpoint directories, + and configures logging paths. Manages experiment naming conflicts by appending + timestamps to avoid overwriting existing experiments. + """ + + def validate(self, config: Config) -> None: + """Validate global configuration settings and set up directory structure. + + - Validates that the mode is one of the supported values + - Creates absolute checkpoint paths and handles experiment naming conflicts + - Sets up the log directory path + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If an invalid mode is specified. + """ + # check mode + if config.mode not in ["explore", "train", "both", "bench", "serve"]: + raise ValueError(f"Invalid mode: {config.mode}") + + # prepare for the checkpoint directory + if not os.path.isabs(config.checkpoint_root_dir): + config.checkpoint_root_dir = os.path.join(os.getcwd(), config.checkpoint_root_dir) + # create a job dir at checkpoint_root_dir/project/name + config.checkpoint_job_dir = os.path.join( + config.checkpoint_root_dir, config.project, config.group, config.name + ) + # rename the experiment when necessary + if not config.continue_from_checkpoint and ( + os.path.exists(config.checkpoint_job_dir) and os.listdir(config.checkpoint_job_dir) + ): + if config.mode == "bench": + self.logger.warning( + "For bench mode, `continue_from_checkpoint` is set as `true` " + "to enable using existing checkpoints." + ) + config.continue_from_checkpoint = True + else: + ori_name = config.name + timestamp = datetime.now().strftime("%Y%m%d%H%M%S") + config.name = f"{ori_name}_{timestamp}" + config.checkpoint_job_dir = f"{config.checkpoint_job_dir}_{timestamp}" + self.logger.warning( + f"Experiment [{ori_name}] already exists, renamed as {config.name}." + ) + os.makedirs(config.checkpoint_job_dir, exist_ok=True) + + # check log + config.log.save_dir = os.path.join(config.checkpoint_job_dir, "log") + + +class RayClusterConfigValidator(ConfigValidator): + """Validator for Ray cluster configuration. + + Handles Ray cluster setup including namespace configuration, automatic detection + of cluster resources (node count and GPUs per node), and GPU allocation validation + based on the current operating mode and model requirements. + """ + + def validate(self, config: Config) -> None: + """Validate and configure Ray cluster settings. + + - Sets the Ray namespace if not provided + - Skips validation if Tinker is enabled + - Automatically detects cluster information if not provided + - Validates GPU allocation based on mode and model requirements + + Args: + config: The global configuration object to validate. + + Raises: + RuntimeError: If no alive nodes are found in the Ray cluster. + ValueError: If GPU allocation requirements cannot be satisfied. + """ + # set namespace + if config.ray_namespace is None or len(config.ray_namespace) == 0: + config.ray_namespace = f"{config.project}/{config.name}" + + if config.model.tinker.enable: + return + + # check cluster infomation + if not config.cluster.node_num or not config.cluster.gpu_per_node: + self._set_cluster_info(config) + self._set_gpu_allocation_info(config) + + def _set_cluster_info(self, config: Config) -> None: + """Automatically detect and set cluster node and GPU information. + + Initializes Ray if not already initialized, queries the cluster for + alive nodes and available GPUs, then sets the configuration accordingly. + + Args: + config: The global configuration object to modify. + + Raises: + RuntimeError: If no alive nodes are found in the Ray cluster. + """ + # init ray cluster to detect node_num and gpu_per_node + was_initialized = ray.is_initialized() + if not was_initialized: + ray.init( + address=config.cluster.ray_address, + ignore_reinit_error=True, + namespace=config.ray_namespace, + ) + + alive_nodes = [n for n in ray.nodes() if n["alive"]] + if not alive_nodes: + raise RuntimeError("Could not find any alive nodes in the Ray cluster.") + + # set node_num + if not config.cluster.node_num: + config.cluster.node_num = len(alive_nodes) + self.logger.info(f"Auto-detected and set node_num: {config.cluster.node_num}") + + # set gpu_per_node + if not config.cluster.gpu_per_node: + gpu_per_node = 0 + for node in alive_nodes: + node_gpus = node.get("Resources", {}).get("GPU") + if node_gpus and node_gpus > 0: + gpu_per_node = int(node_gpus) + break + + config.cluster.gpu_per_node = gpu_per_node + self.logger.info(f"Auto-detected and set gpu_per_node: {config.cluster.gpu_per_node}") + + if not was_initialized: + ray.shutdown() + + def _set_gpu_allocation_info(self, config: Config) -> None: + """Calculate and validate GPU allocation for explorer and trainer components. + + Computes GPU requirements based on model configurations and validates that + the total available GPUs are sufficient for the requested allocation. + + Args: + config: The global configuration object to modify. + + Raises: + ValueError: If GPU allocation requirements cannot be satisfied based on + the current mode and available resources. + """ + cluster = config.cluster + if config.mode != "train": + cluster.rollout_gpu_num = ( + config.explorer.rollout_model.tensor_parallel_size + * config.explorer.rollout_model.engine_num + ) + cluster.auxiliary_model_gpu_num = sum( + model.tensor_parallel_size * model.engine_num + for model in config.explorer.auxiliary_models + ) + cluster.explorer_gpu_num = cluster.rollout_gpu_num + cluster.auxiliary_model_gpu_num + cluster.total_gpu_num = cluster.node_num * cluster.gpu_per_node + if config.mode in ["explore", "bench", "serve"]: + if cluster.explorer_gpu_num > cluster.total_gpu_num: + raise ValueError( + f"Total GPU number ({cluster.total_gpu_num}) is less than " + f"the number of GPUs required for rollout ({cluster.explorer_gpu_num})." + ) + else: + if cluster.explorer_gpu_num >= cluster.total_gpu_num: + raise ValueError( + "Not enough GPUs for trainer. " + f"Explorer requires {cluster.explorer_gpu_num} GPUs, " + f"but total available GPUs are {cluster.total_gpu_num}." + ) + + cluster.trainer_gpu_num = cluster.total_gpu_num - cluster.explorer_gpu_num + if cluster.trainer_gpu_num <= cluster.gpu_per_node: + cluster.trainer_node_num = 1 + cluster.trainer_gpu_num_per_node = cluster.trainer_gpu_num + else: + if cluster.trainer_gpu_num % cluster.gpu_per_node != 0: + raise ValueError( + "Trainer must use an integer number of nodes, " + f"but got trainer_gpu_num ({cluster.trainer_gpu_num}) " + f"with gpu_per_node ({cluster.gpu_per_node}). " + "Please change `engine_num` or `tensor_parallel_size` in explorer config." + ) + cluster.trainer_node_num = cluster.trainer_gpu_num // cluster.gpu_per_node + cluster.trainer_gpu_num_per_node = cluster.gpu_per_node + + +class AlgorithmConfigValidator(ConfigValidator): + """Validator for algorithm-specific configuration. + + Handles algorithm type validation, sets default configuration parameters, + validates function registry entries, and manages deprecated optimizer settings. + """ + + def validate(self, config: Config) -> None: + """Validate and configure algorithm-specific settings. + + - Validates the algorithm type and runs algorithm-specific validation + - Sets default configuration values for various algorithm components + - Validates and configures function registry entries (loss functions, etc.) + - Handles deprecated optimizer configuration parameters + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If invalid algorithm types or function names are specified. + """ + from trinity.algorithm import ( + ADVANTAGE_FN, + ALGORITHM_TYPE, + ENTROPY_LOSS_FN, + KL_FN, + POLICY_LOSS_FN, + SAMPLE_STRATEGY, + ) + + algorithm = ALGORITHM_TYPE.get(config.algorithm.algorithm_type) + algorithm.check_config(config) + default_config = { + "sample_strategy": "warmup", + "policy_loss_fn": "ppo", + "advantage_fn": "ppo", + "kl_penalty_fn": "none", + "kl_loss_fn": "k2", + "entropy_loss_fn": "default", + "loss_agg_mode": "token-mean", + } + default_config.update(algorithm.default_config()) + for key, value in default_config.items(): + set_if_none(config.algorithm, key, value) + + def check_and_set(name, registry, args_attr): + fn_cls = registry.get(getattr(config.algorithm, name)) + if fn_cls is None: + raise ValueError(f"Invalid {name}: {getattr(config.algorithm, name)}") + set_if_none(config.algorithm, args_attr, fn_cls.default_args()) + return fn_cls + + check_and_set("sample_strategy", SAMPLE_STRATEGY, "sample_strategy_args") + check_and_set("policy_loss_fn", POLICY_LOSS_FN, "policy_loss_fn_args") + check_and_set("advantage_fn", ADVANTAGE_FN, "advantage_fn_args") + check_and_set("kl_loss_fn", KL_FN, "kl_loss_fn_args") + check_and_set("kl_penalty_fn", KL_FN, "kl_penalty_fn_args") + check_and_set("entropy_loss_fn", ENTROPY_LOSS_FN, "entropy_loss_fn_args") + if "loss_agg_mode" in config.algorithm.policy_loss_fn_args: # type: ignore [operator] + # override loss_agg_mode in policy_loss_fn_args + config.algorithm.policy_loss_fn_args["loss_agg_mode"] = config.algorithm.loss_agg_mode # type: ignore [index] + + optim_config = config.algorithm.optimizer + if optim_config.warmup_style is not None: + optim_config.lr_scheduler_type = optim_config.warmup_style + self.logger.warning( + "`warmup_style` is deprecated. Please use `lr_scheduler_type` instead. " + f"And `lr_scheduler_type` is set to {optim_config.lr_scheduler_type}." + ) + + +class ModelConfigValidator(ConfigValidator): + """Validator for model configuration settings. + + Handles model path validation, chat template loading, Tinker-specific validation, + and model length parameter validation including prompt/response token limits. + """ + + def validate(self, config: Config) -> None: + """Validate and configure model-specific settings. + + - Sets critic model path to actor model path if not specified + - Loads chat templates from file if path is provided + - Validates Tinker-specific configuration if enabled + - Validates and sets model length parameters (max_model_len, max_prompt_tokens, etc.) + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If chat template file cannot be read, model length constraints + are violated, or Tinker configuration is invalid. + """ + model = config.model + if not model.critic_model_path: + model.critic_model_path = model.model_path + + if model.tinker.enable: + self._check_tinker(config) + + # check template + if model.chat_template_path is not None and model.custom_chat_template is None: + try: + with open(model.chat_template_path, "r") as f: + model.custom_chat_template = f.read() + except Exception as e: + raise ValueError( + f"Failed to read chat template from {model.chat_template_path}: {e}" + ) + + # check max_model_len, max_prompt_tokens, max_response_tokens + self._check_model_len(config) + + def _check_tinker(self, config: Config) -> None: + """Validate Tinker-specific configuration settings. + + - Validates that critic models are not used with Tinker + - Checks that the model is supported by the Tinker service + - Issues warnings about entropy coefficient recommendations + - Forces engine types to 'tinker' for all components + - Disables NCCL synchronization for Tinker + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If critic models are used with Tinker or if the model + is not supported by the Tinker service. + """ + model = config.model + from trinity.algorithm import ALGORITHM_TYPE + + algorithm = ALGORITHM_TYPE.get(config.algorithm.algorithm_type) + if algorithm.use_critic: + raise ValueError("Critic model is not supported when using tinker!") + + import tinker + + service_client = tinker.ServiceClient() + supported_models = { + item.model_name for item in service_client.get_server_capabilities().supported_models + } + if model.model_path not in supported_models: + self.logger.error(f"Supported models: {supported_models}") + raise ValueError(f"{model.model_path} is not supported by tinker!") + + if ( + config.algorithm.entropy_loss_fn != "none" + and config.algorithm.entropy_loss_fn_args.get("entropy_coef", 0.0) != 0.0 + ): + self.logger.warning( + "The entropy in Tinker trainer is an estimated value; " + "it is recommended to set `entropy_coef` to 0." + ) + + if config.explorer.rollout_model.engine_type != "tinker": + config.explorer.rollout_model.engine_type = "tinker" + self.logger.warning("Rollout model engine type is set to `tinker`.") + + for aux_model_config in config.explorer.auxiliary_models: + if aux_model_config.engine_type != "tinker": + aux_model_config.engine_type = "tinker" + self.logger.warning("Auxiliary model engine type is set to `tinker`.") + + if config.trainer.trainer_type != "tinker": + config.trainer.trainer_type = "tinker" + self.logger.warning("Trainer type is set to `tinker`.") + + if config.synchronizer.sync_method == SyncMethod.NCCL: + config.synchronizer.sync_method = SyncMethod.CHECKPOINT + self.logger.warning( + "Tinker do not support NCCL, `synchronizer.sync_method` is set to `checkpoint`." + ) + + def _check_model_len(self, config: Config) -> None: + """Validate and set model length configuration parameters. + + Ensures that max_model_len, max_prompt_tokens, and max_response_tokens + are properly configured and consistent with each other. Sets defaults + when values are missing and validates constraints. + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If model length constraints cannot be satisfied or + if required parameters are missing. + """ + model = config.model + # if all three are set, check if they are valid + if ( + model.max_model_len is not None + and model.max_prompt_tokens is not None + and model.max_response_tokens is not None + ): + if model.max_prompt_tokens + model.max_response_tokens > model.max_model_len: + raise ValueError( + "`max_prompt_tokens` + `max_response_tokens` " + f"({model.max_prompt_tokens} + {model.max_response_tokens}) " + f"exceeds `max_model_len` ({model.max_model_len}). " + "Please adjust them accordingly." + ) + + # check max_model_len first + if model.max_model_len is None: + if model.max_prompt_tokens is not None and model.max_response_tokens is not None: + model.max_model_len = model.max_prompt_tokens + model.max_response_tokens + self.logger.warning( + f"`max_model_len` is set to {model.max_model_len} from " + "`max_prompt_tokens` and `max_response_tokens`." + ) + else: + raise ValueError("Unable to determine `max_model_len`, please set it manually.") + + # both max_prompt_tokens and max_response_tokens are None + if model.max_prompt_tokens is None and model.max_response_tokens is None: + # default to max_model_len / 2 + model.max_prompt_tokens = model.max_model_len // 2 + model.max_response_tokens = model.max_model_len - model.max_prompt_tokens + self.logger.warning( + "`max_prompt_tokens` and `max_response_tokens` are not set, " + f"set to {model.max_prompt_tokens} and {model.max_response_tokens} respectively." + ) + + # only max_prompt_tokens is None + if model.max_prompt_tokens is None and model.max_response_tokens is not None: + model.max_response_tokens = min(model.max_response_tokens, model.max_model_len - 1) + model.max_prompt_tokens = model.max_model_len - model.max_response_tokens + self.logger.warning( + f"`max_prompt_tokens` is set to {model.max_prompt_tokens}, " + f"`max_response_tokens` is set to {model.max_response_tokens}." + ) + + # only max_response_tokens is None + if model.max_response_tokens is None and model.max_prompt_tokens is not None: + model.max_prompt_tokens = min(model.max_prompt_tokens, model.max_model_len - 1) + model.max_response_tokens = model.max_model_len - model.max_prompt_tokens + self.logger.warning( + f"`max_response_tokens` is set to {model.max_response_tokens}, " + f"`max_prompt_tokens` is set to {model.max_prompt_tokens}." + ) + + if model.min_response_tokens >= model.max_response_tokens: # type: ignore [operator] + model.min_response_tokens = max(model.max_response_tokens - 1, 0) # type: ignore [operator] + self.logger.warning(f"`min_response_tokens` is set to {model.min_response_tokens}.") + + if model.enable_prompt_truncation is True: + if model.max_prompt_tokens is None: + raise ValueError( + "When `model.enable_prompt_truncation` is True, " + "`model.max_prompt_tokens` must be set properly. " + "This function does not work with OpenAI API mode." + ) + self.logger.warning( + "`enable_prompt_truncation` is set to True; the prompt will be" + f" truncated to `max_prompt_tokens`={model.max_prompt_tokens} " + "tokens if it is too long." + ) + else: + self.logger.warning( + "`enable_prompt_truncation` is set to False; please make sure " + "the prompt is not too long and `max_model_len` is large enough, " + "otherwise prompt length + response length may exceed `max_model_len`!" + ) + + +class ExplorerConfigValidator(ConfigValidator): + """Validator for explorer configuration settings. + + Handles rollout model configuration inheritance, auxiliary model validation, + over-rollout ratio validation, and LoRA configuration processing. + """ + + def validate(self, config: Config) -> None: + """Validate and configure explorer-specific settings. + + - Inherits model configuration from the global model config to rollout models + - Validates auxiliary model configurations + - Validates over-rollout ratio settings and compatibility with sync style + - Processes LoRA configurations including dummy LoRA creation + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If auxiliary models lack model paths, over-rollout ratio + is invalid, or multiple LoRA adapters are configured. + """ + if config.explorer is None: + return + + rollout_args = ["temperature", "top_p", "top_k", "logprobs", "repetition_penalty"] + length_args = [ + "max_model_len", + "max_prompt_tokens", + "max_response_tokens", + "min_response_tokens", + "enable_prompt_truncation", + ] + rope_args = ["rope_scaling", "rope_theta"] + model_args = rollout_args + length_args + rope_args + + # rollout model + for args in model_args + ["model_path"]: + set_if_none(config.explorer.rollout_model, args, getattr(config.model, args)) + set_if_none( + config.explorer.rollout_model, "chat_template", config.model.custom_chat_template + ) + + # auxiliary models + for aux_model in config.explorer.auxiliary_models: + if not aux_model.model_path: + raise ValueError("auxiliary model's model_path is required.") + for args in model_args: + set_if_none(aux_model, args, getattr(config.model, args)) + + if config.explorer.over_rollout.ratio > 0.0: + if not (0.0 <= config.explorer.over_rollout.ratio < 1.0): + raise ValueError("over_rollout_ratio should be in [0.0, 1.0)") + if config.synchronizer.sync_style == SyncStyle.FIXED: + raise ValueError( + "over_rollout_ratio is not compatible with fixed sync_style, please set " + "`synchronizer.sync_style` to `dynamic_by_explorer` or `dynamic_by_trainer`." + ) + + self._validate_lora(config) + + # check concurrent mode + if config.explorer.concurrent_mode not in ["sequential", "asynchronous", "multi-threading"]: + raise ValueError(f"Invalid explorer.concurrent_mode: {config.explorer.concurrent_mode}") + if config.explorer.concurrent_mode in ["asynchronous", "multi-threading"]: + batch_size = config.buffer.batch_size + max_runner_per_model = math.ceil(batch_size / config.explorer.rollout_model.engine_num) + if config.explorer.runner_per_model > max_runner_per_model: + self.logger.warning( + f"explorer.runner_per_model ({config.explorer.runner_per_model}) is too large " + f"for concurrent_mode '{config.explorer.concurrent_mode}' with batch_size " + f"({batch_size}) and rollout_model.engine_num ({config.explorer.rollout_model.engine_num}). " + f"It is set to {max_runner_per_model}." + ) + config.explorer.runner_per_model = max_runner_per_model + + def _validate_lora(self, config: Config) -> None: + """Process and validate LoRA configuration settings. + + - Enables LoRA for rollout models when LoRA configs are provided + - Validates that only one LoRA adapter is supported + - Creates dummy LoRA adapters when no path is provided + - Configures LoRA modules and kwargs for the rollout model + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If more than one LoRA adapter is configured. + """ + # for lora configs + if not config.model.tinker.enable and config.model.lora_configs is not None: + config.explorer.rollout_model.enable_lora = True + if len(config.model.lora_configs) > 1: + raise ValueError("Only one lora adapter is supported for now.") + lora_config = config.model.lora_configs[0] + if lora_config.path is None: + self.logger.info("Creating dummy lora, since no lora_path is provided.") + lora_path = create_dummy_lora( + model_path=config.model.model_path, + checkpoint_job_dir=config.checkpoint_job_dir, + lora_rank=lora_config.lora_rank, + lora_alpha=lora_config.lora_alpha, + target_modules=lora_config.target_modules, + exclude_modules=lora_config.exclude_modules, + ) + lora_config.path = lora_path + lora_config.is_dummy = True + config.explorer.rollout_model.lora_modules = [ + { + "lora_int_id": i + 1, + "lora_name": cfg.name, + "lora_path": cfg.path, + "base_model_name": cfg.base_model_name, + } + for i, cfg in enumerate(config.model.lora_configs) + ] + config.explorer.rollout_model.lora_kwargs = { + "max_loras": len(config.model.lora_configs), + "max_lora_rank": max( + ( + model_config.lora_rank + for model_config in config.model.lora_configs + if model_config.lora_rank > 0 + ), + default=0, + ), + "default_lora_path": os.path.join( + config.checkpoint_job_dir, "global_step_0", "actor", "lora_adapter" + ), # will be poped later + } + + +class SynchronizerConfigValidator(ConfigValidator): + """Validator for synchronizer configuration settings. + + Handles synchronizer namespace configuration and validates NCCL synchronization + compatibility with different modes and features. + """ + + def validate(self, config: Config) -> None: + """Validate and configure synchronizer settings. + + - Sets the Ray namespace for the synchronizer + - Sets the explorer world size based on rollout GPU count + - Disables NCCL synchronization for incompatible modes and features + + Args: + config: The global configuration object to validate. + """ + config.synchronizer.ray_namespace = config.ray_namespace + config.synchronizer.explorer_world_size = config.cluster.rollout_gpu_num + if config.synchronizer.sync_method == SyncMethod.NCCL: + if config.mode in ["train", "explore", "bench", "serve"]: + config.synchronizer.sync_method = SyncMethod.CHECKPOINT + self.logger.warning( + f"`{config.mode}` mode does not support NCCL synchronization, " + "set `synchronizer.sync_method` to `checkpoint`." + ) + if config.model.lora_configs is not None: + config.synchronizer.sync_method = SyncMethod.CHECKPOINT + self.logger.warning( + "LoRA is not supported with NCCL synchronization, " + "set `synchronizer.sync_method` to `checkpoint`." + ) + + +class IntervalConfigValidator(ConfigValidator): + """Validator for interval configuration settings. + + Validates synchronization and evaluation intervals, ensuring that evaluation + intervals are multiples of synchronization intervals when applicable. + """ + + def validate(self, config: Config) -> None: + """Validate interval configuration settings. + + - Ensures synchronization interval is positive + - Adjusts evaluation interval to be a multiple of sync interval when needed + + Args: + config: The global configuration object to validate. + + Raises: + AssertionError: If synchronization interval is not positive. + """ + assert config.synchronizer.sync_interval > 0, "`sync_interval` must be positive." + + if config.mode != "bench" and config.algorithm.algorithm_type != "dpo": # TODO + # check eval_interval + if config.explorer.eval_interval % config.synchronizer.sync_interval != 0: + config.explorer.eval_interval = ( + max(config.explorer.eval_interval // config.synchronizer.sync_interval, 1) + ) * config.synchronizer.sync_interval + self.logger.warning( + "`eval_interval` is not a multiple of `sync_interval`; " + f"adjusted to the nearest integer={config.explorer.eval_interval}." + ) + + +class MonitorConfigValidator(ConfigValidator): + """Validator for monitor configuration settings. + + Validates monitor type, sets default arguments, and configures monitor cache directory. + """ + + def validate(self, config: Config) -> None: + """Validate and configure monitor settings. + + - Validates that the monitor type is supported + - Sets default monitor arguments if not provided + - Creates the monitor cache directory + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If an invalid monitor type is specified. + """ + from trinity.utils.monitor import MONITOR + + monitor_cls = MONITOR.get(config.monitor.monitor_type) + if monitor_cls is None: + raise ValueError(f"Invalid monitor type: {config.monitor.monitor_type}") + set_if_none(config.monitor, "monitor_args", monitor_cls.default_args()) + # create a job dir in ///monitor + config.monitor.cache_dir = os.path.join(config.checkpoint_job_dir, "monitor") + try: + os.makedirs(config.monitor.cache_dir, exist_ok=True) + except Exception: + self.logger.warning( + f"Failed to create monitor dir {config.monitor.cache_dir}, please check " + f"your checkpoint directory: {config.checkpoint_job_dir}" + ) + + +class BufferConfigValidator(ConfigValidator): + """Validator for buffer configuration settings. + + Handles train batch size validation, buffer directory setup, tokenizer configuration, + and comprehensive validation of explorer/trainer input configurations including + tasksets, experience buffers, and data pipelines. + """ + + def validate(self, config: Config) -> None: + """Validate and configure buffer settings. + + - Sets train batch size based on mode and algorithm configuration + - Creates buffer cache directory + - Configures pad token ID using tokenizer + - Validates explorer input configurations (tasksets, selectors) + - Validates trainer input configurations (experience buffers, auxiliary buffers) + - Validates data processor pipeline configurations + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If required buffer configurations are missing or invalid. + RuntimeError: If buffer directory creation fails. + """ + # check train_batch_size + if not config.buffer.train_batch_size: + if config.mode == "train" or config.algorithm.algorithm_type in ["sft", "dpo"]: + raise ValueError( + "`buffer.train_batch_size` is required when `mode` is 'train' " + "or `algorithm.algorithm_type` is 'sft' or 'dpo'" + ) + self.logger.info( + "`buffer.train_batch_size` is set to `buffer.batch_size` * `algorithm.repeat_times`" + ) + config.buffer.train_batch_size = ( + config.buffer.batch_size * config.algorithm.repeat_times + ) + if ( + config.mode in {"train", "both"} + and config.buffer.train_batch_size % config.cluster.trainer_gpu_num != 0 + ): + raise ValueError( + f"batch_size ({config.buffer.train_batch_size}) must be " + f"divisible by ({config.cluster.trainer_gpu_num})." + ) + + # create buffer.cache_dir at ///buffer + config.buffer.cache_dir = os.path.abspath(os.path.join(config.checkpoint_job_dir, "buffer")) + try: + os.makedirs(config.buffer.cache_dir, exist_ok=True) + except Exception as e: + raise RuntimeError( + f"Failed to create buffer dir {config.buffer.cache_dir}, please check " + f"your checkpoint directory: {config.checkpoint_job_dir}" + ) from e + + # set pad_token_id / tokenizer_path + if config.buffer.pad_token_id is None: + from transformers import AutoTokenizer + + try: + tokenizer = AutoTokenizer.from_pretrained(config.model.model_path) + if tokenizer.pad_token_id is None: + tokenizer.pad_token_id = tokenizer.eos_token_id + self.logger.warning( + f"tokenizer.pad_token_id is None. Now set to {tokenizer.eos_token_id}", + stacklevel=1, + ) + config.buffer.pad_token_id = tokenizer.pad_token_id + + except Exception: + self.logger.warning( + f"Failed to get pad token id from model {config.model.model_path}" + ) + config.buffer.pad_token_id = 0 + + self._check_explorer_input(config) + self._check_trainer_input(config) + self._check_data_processor(config) + + def _check_explorer_input(self, config: Config): + """Validate explorer input configuration including tasksets and selectors. + + - Handles taskset vs tasksets configuration + - Validates that at least one taskset is provided in non-bench modes + - Configures taskset defaults and validates selectors + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If taskset configuration is invalid or selectors are unsupported. + """ + from trinity.buffer.selector import SELECTORS + + if config.mode in {"train", "serve"}: + # no need to check explorer_input in serve mode + return + + explorer_input = config.buffer.explorer_input + + if explorer_input.taskset: + if len(explorer_input.tasksets) > 0: + raise ValueError("Do not support setting `taskset` and `tasksets` simultaneously!") + explorer_input.tasksets = [explorer_input.taskset] + explorer_input.taskset = None + elif config.mode != "bench" and len(explorer_input.tasksets) == 0: + raise ValueError("At least one taskset should be provided in explorer_input!") + + def _fill_taskset_config(taskset: TasksetConfig, index: int, is_eval: bool = False): + if not taskset.path: + prefix = "Eval taskset" if is_eval else "Train taskset" + raise ValueError(f"{prefix} [{taskset}]'s path is not configured.") + + if not taskset.name: + prefix = "eval_" if is_eval else "" + taskset.name = f"{prefix}taskset_{index}" + taskset.is_eval = is_eval + + taskset.batch_size = config.buffer.batch_size + if not is_eval: + taskset.total_epochs = config.buffer.total_epochs + taskset.total_steps = config.buffer.total_steps + if taskset.repeat_times != config.algorithm.repeat_times: + taskset.repeat_times = config.algorithm.repeat_times + self.logger.info( + "`buffer.explorer_input.taskset.repeat_times` is set to " + f"`algorithm.repeat_times` (={config.algorithm.repeat_times})." + ) + + set_if_none(taskset, "default_workflow_type", explorer_input.default_workflow_type) + set_if_none(taskset, "default_reward_fn_type", explorer_input.default_reward_fn_type) + set_if_none(taskset, "ray_namespace", config.ray_namespace) + for attr in ["temperature", "top_p", "top_k", "logprobs"]: + set_if_none(taskset.rollout_args, attr, getattr(config.model, attr)) + set_if_none(taskset.rollout_args, "max_tokens", config.model.max_response_tokens) + set_if_none(taskset.format, "chat_template", config.model.custom_chat_template) + + for i, taskset in enumerate(explorer_input.tasksets): + _fill_taskset_config(taskset, i) + + # check if selector is supported + selector = SELECTORS.get(taskset.task_selector.selector_type) + if selector is None: + raise ValueError( + f"Selector {taskset.task_selector.selector_type} is not supported." + ) + + for idx, taskset in enumerate(explorer_input.eval_tasksets): + # eval_workflow has higher priority than workflow in eval tasksets, so we set it first + set_if_none(taskset, "default_workflow_type", explorer_input.default_eval_workflow_type) + _fill_taskset_config(taskset, idx, is_eval=True) + + def _check_trainer_input(self, config: Config): + """Validate trainer input configuration including experience buffers. + + - Configures experience buffer defaults and storage types + - Validates auxiliary buffer configurations + - Sets buffer schema types based on algorithm + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If required trainer input configurations are missing. + AssertionError: If experience buffer is missing in train mode. + """ + if config.mode == "bench": + # no need to check trainer_input in bench mode + return + + trainer_input = config.buffer.trainer_input + experience_buffer = trainer_input.experience_buffer + + if experience_buffer is None: + experience_buffer = trainer_input.experience_buffer = ExperienceBufferConfig( + name="experience_buffer", + storage_type=StorageType.QUEUE.value, + ) + self.logger.info( + f"Auto set `buffer.trainer_input.experience_buffer` to {experience_buffer}" + ) + elif experience_buffer.storage_type == StorageType.FILE.value and config.mode == "both": + self.logger.warning( + "`FILE` storage is not supported to use as experience_buffer " + "in `both` mode, use `QUEUE` instead." + ) + experience_buffer.storage_type = StorageType.QUEUE.value + + if not experience_buffer.name: + experience_buffer.name = "experience_buffer" + + if not experience_buffer.path: + experience_buffer.path = self._default_storage_path( + config, experience_buffer.storage_type, experience_buffer.name + ) + self.logger.warning( + f"Auto set `buffer.trainer_input.experience_buffer.path` to {experience_buffer.path}" + ) + + from trinity.algorithm import ALGORITHM_TYPE + + experience_buffer.schema_type = ALGORITHM_TYPE.get(config.algorithm.algorithm_type).schema + experience_buffer.batch_size = config.buffer.train_batch_size + experience_buffer.tokenizer_path = config.model.model_path + set_if_none(experience_buffer, "ray_namespace", config.ray_namespace) + set_if_none(experience_buffer.format, "chat_template", config.model.custom_chat_template) + for aux_name, aux_buffer in trainer_input.auxiliary_buffers.items(): + aux_buffer.batch_size = config.buffer.train_batch_size + aux_buffer.tokenizer_path = config.model.model_path + set_if_none(aux_buffer, "ray_namespace", config.ray_namespace) + if aux_buffer.path is None or aux_buffer.path == "": + raise ValueError( + f"`buffer.trainer_input.auxiliary_buffers[{aux_name}].path` is required, " + f"please set it to the path of the auxiliary buffer." + ) + + if config.mode == "train": + assert ( + experience_buffer is not None + ), "`buffer.trainer_input.experience_buffer` is required when `mode` is `train`." + experience_buffer.total_epochs = config.buffer.total_epochs + experience_buffer.total_steps = config.buffer.total_steps + + def _default_storage_path(self, config: Config, storage_type: str, name: str) -> str: + """Generate default storage path based on storage type. + + Args: + config: The global configuration object. + storage_type: The type of storage (SQL, FILE, etc.). + name: The name of the storage component. + + Returns: + The default storage path for the given storage type and name. + """ + if storage_type == StorageType.SQL.value: + return "sqlite:///" + os.path.join(config.buffer.cache_dir, f"{name}.db") # type: ignore[arg-type] + else: + return os.path.join(config.buffer.cache_dir, f"{name}.jsonl") # type: ignore[arg-type] + + def _check_data_processor(self, config: Config): + """Validate data processor pipeline configurations. + + - Configures experience pipeline input save paths + - Integrates Data-Juicer service configuration into operators + - Validates task pipeline output configuration and path conflicts + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If task pipeline output is missing or path already exists. + """ + # check input/output buffers in pipelines + experience_pipeline = config.data_processor.experience_pipeline + if experience_pipeline is not None and config.mode in {"explore", "both", "serve"}: + if experience_pipeline.save_input and experience_pipeline.input_save_path is None: + experience_pipeline.input_save_path = self._default_storage_path( + config, StorageType.SQL.value, "explorer_output" + ) + self.logger.info( + "Auto set `data_processor.experience_pipeline.input_save_path` " + f"to {experience_pipeline.input_save_path}" + ) + + if config.service.data_juicer is not None: + for operator in experience_pipeline.operators: + if operator.name == "data_juicer": + operator.args["service_config"] = config.service.data_juicer + + task_pipeline = config.data_processor.task_pipeline + if task_pipeline is not None and config.mode in {"explore", "train", "both"}: + if task_pipeline.output is None: + if config.mode != "train": + if len(config.buffer.explorer_input.tasksets) > 0: + task_pipeline.output = config.buffer.explorer_input.tasksets[0] + else: + raise ValueError( + "At least one taskset should be provided in explorer_input!" + ) + elif config.mode == "train" and config.algorithm.algorithm_type in {"dpo", "sft"}: + task_pipeline.output = config.buffer.trainer_input.experience_buffer + else: + raise ValueError( + "`data_processor.task_pipeline.output` is missing. " + "Please set it to the desired output storage config." + ) + if task_pipeline.output.path and os.path.exists(task_pipeline.output.path): + raise ValueError( + f"Task pipeline output path {task_pipeline.output.path} already exists.\n" + "Please choose a different output path to avoid overwriting." + ) + + +class TrainerConfigValidator(ConfigValidator): + """Validator for trainer configuration settings. + + Handles trainer type validation, configuration merging, and parameter validation + for different trainer implementations (veRL, Tinker, etc.). + """ + + def validate(self, config: Config) -> None: + """Validate and configure trainer settings. + + - Validates trainer type and handles configuration for different trainer types + - Merges trainer configuration with schema defaults + - Validates save checkpoint strategy options + - Synchronizes trainer configuration with global config + + Args: + config: The global configuration object to validate. + + Raises: + ValueError: If trainer type is invalid, deprecated config path is used, + or save checkpoint strategy is invalid. + """ + if ( + config.mode not in ["train", "both", "bench"] + and config.trainer.trainer_strategy != "megatron" + ): + return + + if config.trainer.trainer_type == "verl": + if config.trainer.trainer_config: + from trinity.common.verl_config import veRLConfig + + trainer_config_schema = OmegaConf.structured(veRLConfig) + trainer_config = OmegaConf.merge( + trainer_config_schema, config.trainer.trainer_config + ) + config.trainer.trainer_config = OmegaConf.to_object(trainer_config) + elif config.trainer.trainer_config_path: + raise ValueError( + "`trainer_config_path` is deprecated; please use `trainer_config` instead." + ) + else: + from trinity.common.verl_config import veRLConfig + + self.logger.info("`trainer_config` is not provided, using default trainer config.") + config.trainer.trainer_config = veRLConfig() + if config.trainer.max_token_len_per_gpu is None: + config.trainer.max_token_len_per_gpu = math.ceil( + 2 * config.model.max_model_len / config.trainer.ulysses_sequence_parallel_size # type: ignore [operator] + ) + if config.trainer.save_hf_checkpoint not in {"last", "always", "never"}: + raise ValueError( + f"Invalid trainer.save_hf_checkpoint: {config.trainer.save_hf_checkpoint}, " + "must be one of 'last', 'always', or 'never'." + ) + config.trainer.trainer_config.synchronize_config(config) + elif config.trainer.trainer_type == "tinker": + config.trainer.trainer_config = None + else: + raise ValueError(f"Invalid trainer type: {config.trainer.trainer_type}") + + +validators = [ + DeprecatedConfigValidator(), + GlobalConfigValidator(), + MonitorConfigValidator(), + AlgorithmConfigValidator(), + ModelConfigValidator(), + RayClusterConfigValidator(), + SynchronizerConfigValidator(), + IntervalConfigValidator(), + ExplorerConfigValidator(), + BufferConfigValidator(), + TrainerConfigValidator(), +] diff --git a/frameworks/trinity/common/constants.py b/frameworks/trinity/common/constants.py new file mode 100644 index 0000000000000000000000000000000000000000..0000d75d4d8a114b42004e103c8e928c5b5092fb --- /dev/null +++ b/frameworks/trinity/common/constants.py @@ -0,0 +1,115 @@ +# -*- coding: utf-8 -*- +"""Constants.""" +from enum import Enum, EnumMeta + +# names + +EXPLORER_NAME = "explorer" +TRAINER_NAME = "trainer" + +SELECTOR_METRIC = "selector_metric" + +ROLLOUT_WEIGHT_SYNC_GROUP_NAME = "rollout_weight_sync" +DEBUG_NAMESPACE = "TRINITY_DEBUG_NAMESPACE" + +# trinity env var names +CHECKPOINT_ROOT_DIR_ENV_VAR = "TRINITY_CHECKPOINT_ROOT_DIR" +PREVIOUS_STAGE_CHECKPOINT_DIR_ENV_VAR = "TRINITY_PREV_STAGE_CKPT_DIR" +MODEL_PATH_ENV_VAR = "TRINITY_MODEL_PATH" +TASKSET_PATH_ENV_VAR = "/nas/wjq/Trinity-RFT/tasksets" +BUFFER_PATH_ENV_VAR = "TRINITY_BUFFER_PATH" +PLUGIN_DIRS_ENV_VAR = "TRINITY_PLUGIN_DIRS" +LOG_DIR_ENV_VAR = "TRINITY_LOG_DIR" # log dir +LOG_LEVEL_ENV_VAR = "TRINITY_LOG_LEVEL" # global log level +LOG_NODE_IP_ENV_VAR = "TRINITY_LOG_NODE_IP" # whether to organize logs by node IP + + +# constants + +MAX_MODEL_LEN = 4096 + + +# enumerate types + + +class CaseInsensitiveEnumMeta(EnumMeta): + def __getitem__(cls, name): + return super().__getitem__(name.upper()) + + def __getattr__(cls, name): + if not name.startswith("_"): + return cls[name.upper()] + return super().__getattr__(name) + + def __call__(cls, value, *args, **kwargs): + return super().__call__(value.lower(), *args, **kwargs) + + +class CaseInsensitiveEnum(Enum, metaclass=CaseInsensitiveEnumMeta): + pass + + +class PromptType(CaseInsensitiveEnum): + """Prompt Type.""" + + MESSAGES = "messages" # a list of message dict + PLAINTEXT = "plaintext" # user prompt text and assistant response text + + +class StorageType(CaseInsensitiveEnum): + """Storage Type.""" + + SQL = "sql" + QUEUE = "queue" + FILE = "file" + + +class SyncMethodEnumMeta(CaseInsensitiveEnumMeta): + def __call__(cls, value, *args, **kwargs): + if value == "online": + value = "nccl" + elif value == "offline": + value = "checkpoint" + try: + return super().__call__(value, *args, **kwargs) + except Exception: + raise ValueError(f"Invalid SyncMethod: {value}") + + +class SyncMethod(CaseInsensitiveEnum, metaclass=SyncMethodEnumMeta): + """Sync Method.""" + + NCCL = "nccl" + CHECKPOINT = "checkpoint" + MEMORY = "memory" + + +class RunningStatus(Enum): + """Running status of explorer and trainer.""" + + RUNNING = "running" + REQUIRE_SYNC = "require_sync" + WAITING_SYNC = "waiting_sync" + STOPPED = "stopped" + + +class OpType(Enum): + """Operator type for reward shaping.""" + + ADD = "add" + SUB = "sub" + MUL = "mul" + DIV = "div" + + +class SyncStyle(CaseInsensitiveEnum): + FIXED = "fixed" + DYNAMIC_BY_TRAINER = "dynamic_by_trainer" + DYNAMIC_BY_EXPLORER = "dynamic_by_explorer" + + +class SaveStrategy(CaseInsensitiveEnum): + SINGLE_THREAD = "single_thread" + SINGLE_PROCESS = "single_process" + SINGLE_NODE = "single_node" + UNRESTRICTED = "unrestricted" diff --git a/frameworks/trinity/common/experience.py b/frameworks/trinity/common/experience.py new file mode 100644 index 0000000000000000000000000000000000000000..997cf3ab4ac86bcd15a81811f6040c4c7f4d8896 --- /dev/null +++ b/frameworks/trinity/common/experience.py @@ -0,0 +1,651 @@ +# -*- coding: utf-8 -*- +"""Experience Class.""" +from __future__ import annotations + +import pickle +import uuid +from dataclasses import asdict, dataclass, field, fields +from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Union + +import torch + +if TYPE_CHECKING: + from datasets import Dataset +from torch import Tensor + + +@dataclass +class EID: + """Experience ID class to uniquely identify an experience. + + To enable the full functionality of the experience grouping, user should manually set the `run` and `step` fields in custom workflows. + """ + + # TODO: do we need to add project/name here to make it unique across different projects? + # Batch number, e.g., the explorer step num + # Automatically set by the workflow runner + batch: Union[int, str] = "" + # Task number, e.g., the task sequence in the batch, the first task in the batch has task=0 + # Automatically set by the workflow runner + task: Union[int, str] = "" + # Run id, e.g., the first run in the task has run=0 + # User should set this field in custom workflows when creating experiences + run: int = 0 + # Step number when running the task, e.g., the first step in the task has step=0 + # User should set this field in custom workflows when creating experiences + step: int = 0 + suffix: str = field( + default_factory=lambda: uuid.uuid4().hex[:6] + ) # Unique identifier suffix, e.g., a UUID + + @property + def uid(self) -> str: + """An unique identifier for the experience.""" + return f"{self.batch}/{self.task}/{self.run}/{self.step}/{self.suffix}" + + @property + def sid(self) -> str: + """Step ID of the experience. + + For example, experiences generated by all runs of a same task at the same step will have the same sid. + """ + return f"{self.batch}/{self.task}/{self.step}" + + @property + def rid(self) -> str: + """Run ID of the experience. + + For example, experiences generated by one run of a task at all steps will have the same run_id. + """ + return f"{self.batch}/{self.task}/{self.run}" + + @property + def tid(self) -> str: + """Task ID for the experience. + + For example, experiences generated by a all run of a same task in GRPO-like algorithms will have the same tid. + """ + return f"{self.batch}/{self.task}" + + def __str__(self): + return self.uid + + def __repr__(self): + return f"EID(batch={self.batch}, task={self.task}, run={self.run}, step={self.step}, uuid={self.suffix})" + + def to_dict(self) -> dict: + """Convert the EID to a dictionary.""" + return { + "batch": self.batch, + "task": self.task, + "run": self.run, + "step": self.step, + "suffix": self.suffix, + } + + +@dataclass(frozen=True) +class CustomField: + """Custom field for Experiences. + + This is used to store additional information into the Experiences class. + """ + + source_field: str # The source field name in the Experience.info + destination_field: str # The destination field name in the Experiences class + data_type: torch.dtype # The data type of the field, e.g., torch.float32, torch.int64, etc. + + +@dataclass +class Experience: + eid: EID = field(default_factory=EID) # Unique identifier for the experience + tokens: Optional[Tensor] = None # [seq_length] + prompt_length: int = 1 # Length of the prompt in tokens, used for generating attention masks + logprobs: Optional[Tensor] = None # [resp_length] + reward: Optional[float] = None + token_level_reward: Optional[Tensor] = None # [resp_length] + advantages: Optional[Tensor] = None # [resp_length] + returns: Optional[Tensor] = None # [resp_length] + truncate_status: Optional[ + str + ] = None # The status of truncation, e.g., "prompt_truncated", "response_truncated"; Not working for openai api + info: dict = field( + default_factory=dict + ) # Additional information about the experience, can also be used to store custom fields + metrics: dict[str, float] = field( + default_factory=dict + ) # Metrics associated with the experience, directly used by the monitor + + # for single-turn experiences + response_text: Optional[str] = None # Text of the response + prompt_text: Optional[str] = None # Text of the prompt + + # for multi-turn experiences + # Action mask indicates which tokens are generated by the model + action_mask: Optional[Tensor] = None # [resp_length] + messages: Optional[List[dict]] = None # List of messages + tools: Optional[List[dict]] = None + + # for dpo experiences + chosen: Optional[Tensor] = None # Token ids of the chosen response [resp_length] + rejected: Optional[Tensor] = None # Token ids of the rejected response [resp_length] + chosen_messages: Optional[List[dict]] = None # Chosen message list (Include prompt message) + rejected_messages: Optional[List[dict]] = None # Rejected message list (Include prompt message) + + # for multi-modal data + multi_modal_inputs: Optional[Dict[str, Tensor]] = None # Multi-modal inputs for verl trainer + + # for on-policy distillation + teacher_logprobs: Optional[Tensor] = None # [resp_length] + + custom_fields: List[CustomField] = field(default_factory=list) + + def __init__( # noqa: C901 + self, + *, + eid=None, + tokens, + logprobs=None, + reward=None, + token_level_reward=None, + advantages=None, + returns=None, + truncate_status=None, + info=None, + metrics=None, + prompt_length=1, + response_text=None, + prompt_text=None, + action_mask=None, + messages=None, + tools=None, + chosen=None, + rejected=None, + chosen_messages=None, + rejected_messages=None, + multi_modal_inputs=None, + teacher_logprobs=None, + custom_fields=None, + ): + if action_mask is not None: + experience_type = "multi_turn" + elif chosen is not None and rejected is not None: + experience_type = "dpo" + else: + experience_type = "single_turn" + + if experience_type == "single_turn": + assert ( + prompt_length > 0 + ), "Prompt length must be greater than 0 for single-turn experiences." + if truncate_status != "prompt_truncated": + assert ( + len(tokens) > prompt_length + ), f"Token ids must be larger than the prompt length. Got len(tokens)={len(tokens)}, prompt_length={prompt_length}." + action_mask = torch.ones(len(tokens) - prompt_length, dtype=torch.bool) + else: + action_mask = torch.zeros(len(logprobs), dtype=torch.bool) + elif experience_type == "dpo": + prompt_length = len(tokens) + if eid is None: + self.eid = EID() + elif isinstance(eid, dict): + self.eid = EID(**eid) + else: + self.eid = eid + if isinstance(tokens, list): + tokens = torch.tensor(tokens, dtype=torch.int32) + self.tokens = tokens + if isinstance(logprobs, list): + logprobs = torch.tensor(logprobs, dtype=torch.float32) + self.logprobs = logprobs + self.reward = reward + if isinstance(token_level_reward, list): + token_level_reward = torch.tensor(token_level_reward, dtype=torch.float32) + self.token_level_reward = token_level_reward + if isinstance(advantages, list): + advantages = torch.tensor(advantages, dtype=torch.float32) + self.advantages = advantages + if isinstance(returns, list): + returns = torch.tensor(returns, dtype=torch.float32) + self.returns = returns + self.experience_type = experience_type + self.info = info or {} + self.metrics = metrics or {} + self.truncate_status = truncate_status + self.prompt_length = prompt_length + self.response_text = response_text + self.prompt_text = prompt_text + if isinstance(action_mask, list): + action_mask = torch.tensor(action_mask, dtype=torch.bool) + self.action_mask = action_mask + self.messages = messages + self.tools = tools + if isinstance(chosen, list): + chosen = torch.tensor(chosen, dtype=torch.int32) + self.chosen = chosen + if isinstance(rejected, list): + rejected = torch.tensor(rejected, dtype=torch.int32) + self.rejected = rejected + self.chosen_messages = chosen_messages + self.rejected_messages = rejected_messages + self.multi_modal_inputs = multi_modal_inputs + if multi_modal_inputs is not None: + self.multi_modal_inputs = {} + for key, value in multi_modal_inputs.items(): + if not isinstance(value, Tensor): + self.multi_modal_inputs[key] = torch.tensor(value) + else: + self.multi_modal_inputs[key] = value + + # Handle teacher_logprobs + if isinstance(teacher_logprobs, list): + teacher_logprobs = torch.tensor(teacher_logprobs, dtype=torch.float32) + self.teacher_logprobs = teacher_logprobs + + if not isinstance(self.tokens, Tensor): + self.tokens = torch.tensor(self.tokens) + if self.logprobs is not None and not isinstance(self.logprobs, Tensor): + self.logprobs = torch.tensor(self.logprobs) + if self.action_mask is not None and not isinstance(self.action_mask, Tensor): + self.action_mask = torch.tensor(self.action_mask) + if self.chosen is not None and not isinstance(self.chosen, Tensor): + self.chosen = torch.tensor(self.chosen) + if self.rejected is not None and not isinstance(self.rejected, Tensor): + self.rejected = torch.tensor(self.rejected) + if self.teacher_logprobs is not None and not isinstance(self.teacher_logprobs, Tensor): + self.teacher_logprobs = torch.tensor(self.teacher_logprobs, dtype=torch.float32) + self.custom_fields = custom_fields or [] + + def serialize(self) -> bytes: + """Serialize the experience to bytes.""" + return pickle.dumps(self) + + @classmethod + def deserialize(cls, data: bytes) -> Experience: + return pickle.loads(data) + + def to_dict(self) -> dict: + """Convert the experience to a dictionary.""" + res = { + "eid": self.eid.to_dict(), + "type": self.experience_type, + "prompt_length": self.prompt_length, + "response_length": len(self.tokens) - self.prompt_length, # type: ignore [arg-type] + "info": self.info, + "metrics": self.metrics, + } + if self.prompt_text is not None: + res["prompt_text"] = self.prompt_text + if self.response_text is not None: + res["response_text"] = self.response_text + if self.messages is not None: + res["messages"] = self.messages + if self.tools is not None: + res["tools"] = self.tools + if self.chosen_messages is not None: + res["chosen_messages"] = self.chosen_messages + if self.rejected_messages is not None: + res["rejected_messages"] = self.rejected_messages + if self.reward is not None: + res["reward"] = float(self.reward) + if self.truncate_status is not None: + res["truncate_status"] = self.truncate_status + return res + + @classmethod + def gather( + cls, + experiences: List[Experience], + pad_token_id: int = 0, + custom_fields: Optional[List[CustomField]] = None, + ) -> Experiences: + if len(experiences) == 0: + return empty_experiences(custom_fields) + exp_type = experiences[0].experience_type + if exp_type == "dpo": + experiences = split_dpo_experience_to_single_turn(experiences) + max_prompt_length = max([exp.prompt_length for exp in experiences]) # type: ignore [type-var] + max_response_length = max([len(exp.tokens) - exp.prompt_length for exp in experiences]) # type: ignore [arg-type] + eids = [exp.eid for exp in experiences] + + # Gather tokens + tokens = gather_token_ids(experiences, max_prompt_length, max_response_length, pad_token_id) + + # Gather rewards + if experiences[0].reward is not None: + rewards = torch.tensor([exp.reward for exp in experiences], dtype=torch.float) + else: + rewards = None + + # Gather token level rewards + if all(exp.token_level_reward is not None for exp in experiences): + token_level_rewards = gather_response_attrs( + experiences, "token_level_reward", max_response_length + ) + else: + token_level_rewards = None + + # gather action_masks + action_masks = gather_action_masks(experiences, max_response_length) + + # gather attention_masks + attention_masks = gather_attention_masks( + experiences, max_prompt_length, max_response_length + ) + + # gather logprobs + if all(exp.logprobs is not None for exp in experiences): + logprobs = gather_response_attrs(experiences, "logprobs", max_response_length) + else: + logprobs = None + + # gather advantages + if all(exp.advantages is not None for exp in experiences): + advantages = gather_response_attrs(experiences, "advantages", max_response_length) + else: + advantages = None + + # gather returns + if all(exp.returns is not None for exp in experiences): + returns = gather_response_attrs(experiences, "returns", max_response_length) + else: + returns = None + + # gather multi_modal_inputs + if all(exp.multi_modal_inputs is not None for exp in experiences): + multi_modal_inputs = gather_multi_modal_inputs(experiences) + else: + multi_modal_inputs = None + + # gather teacher_logprobs + if all(exp.teacher_logprobs is not None for exp in experiences): + teacher_logprobs = gather_response_attrs( + experiences, "teacher_logprobs", max_response_length + ) + else: + teacher_logprobs = None + + exps = Experiences( + eids=eids, + tokens=tokens, + rewards=rewards, + token_level_rewards=token_level_rewards, + advantages=advantages, + returns=returns, + attention_masks=attention_masks, + action_masks=action_masks, + prompt_length=max_prompt_length, + logprobs=logprobs, + multi_modal_inputs=multi_modal_inputs, + teacher_logprobs=teacher_logprobs, + ) + if custom_fields is not None: + for custom_field in custom_fields: + exps.custom_fields.append(custom_field.destination_field) + setattr( + exps, + custom_field.destination_field, + torch.tensor( + [exp.info[custom_field.source_field] for exp in experiences], + dtype=custom_field.data_type, + ), + ) + return exps + + +def split_dpo_experience_to_single_turn(experiences: List[Experience]) -> List[Experience]: + single_turn_experiences = [] + for exp in experiences: + single_turn_experiences.append( + Experience( + eid=EID( + batch=exp.eid.batch, + task=exp.eid.task, + step=exp.eid.step, + run=exp.eid.run, + ), + tokens=torch.cat([exp.tokens, exp.chosen]), + reward=exp.reward, + info=exp.info, + metrics=exp.metrics, + prompt_length=len(exp.tokens), # type: ignore [arg-type] + prompt_text=exp.prompt_text, + messages=exp.chosen_messages, + ) + ) + single_turn_experiences.append( + Experience( + eid=EID( + batch=exp.eid.batch, + task=exp.eid.task, + step=exp.eid.step, + run=exp.eid.run, + ), + tokens=torch.cat([exp.tokens, exp.rejected]), + reward=exp.reward, + info=exp.info, + metrics=exp.metrics, + prompt_length=len(exp.tokens), # type: ignore [arg-type] + prompt_text=exp.prompt_text, + messages=exp.rejected_messages, + ) + ) + return single_turn_experiences + + +@dataclass +class Experiences: + """A container for a batch of experiences, for high performance communication usage. + + Example: + + >>> |<- prompt_length ->| | + >>> tokens: ('P' represents prompt, 'O' represents output) + >>> exp1: |........PPPPPPPPPPP|OOOOOOOOOO.....| + >>> exp2: |......PPPPPPPPPPPPP|OOOOOOO........| + >>> + >>> attention_masks: ('.' represents False and '1' represents True) + >>> exp1: |........11111111111|1111111111.....| + >>> exp2: |......1111111111111|1111111........| + """ + + eids: List[EID] # Experience IDs of each experience in the batch + tokens: Tensor # [batch_size, seq_length] + + # At least one of `rewards` or `token_level_rewards` must be provided (not None). + # If both are provided, `token_level_rewards` will be used and `rewards` will be ignored. + rewards: Tensor # [batch_size] + token_level_rewards: Tensor # [batch_size, response_length] + + advantages: Optional[Tensor] # [batch_size, response_length] + returns: Optional[Tensor] # [batch_size, response_length] + attention_masks: Tensor # [batch_size, sequence_length] + action_masks: Optional[Tensor] # [batch_size, response_length] + prompt_length: int + logprobs: Optional[Tensor] # [batch_size, response_length] + multi_modal_inputs: Optional[Any] + custom_fields: List[str] = field( + default_factory=list + ) # Custom fields to include in the gathered experiences + teacher_logprobs: Optional[Tensor] = None # [batch_size, response_length] + + @property + def batch_size(self) -> int: + """Get the batch size.""" + return self.tokens.size(0) + + @classmethod + def gather_experiences( + cls, + experiences: list[Experience], + pad_token_id: int = 0, + custom_fields: Optional[List[CustomField]] = None, + ) -> Experiences: + """Gather a batch of experiences from a list of experiences. + + This method will automatically pad the `tokens` and `logprobs` of input experiences to the same length. + + Args: + experiences (list[Experience]): A list of experiences to gather. + pad_token_id (int): The token ID to use for padding. Default is 0. + custom_fields (Optional[List[CustomField]]): Custom fields to include in the gathered experiences. + """ + if len(experiences) == 0: + return empty_experiences(custom_fields) + return experiences[0].__class__.gather( + experiences, pad_token_id=pad_token_id, custom_fields=custom_fields + ) + + +def empty_experiences(custom_fields: Optional[List[CustomField]]) -> Experiences: + exps = Experiences( + tokens=torch.empty(0, dtype=torch.int32), + rewards=torch.empty(0, dtype=torch.float32), + token_level_rewards=torch.empty(0, dtype=torch.float32), + advantages=torch.empty(0, dtype=torch.float32), + returns=torch.empty(0, dtype=torch.float32), + attention_masks=torch.empty(0, dtype=torch.bool), + action_masks=torch.empty(0, dtype=torch.bool), + logprobs=torch.empty(0, dtype=torch.float32), + prompt_length=torch.empty(0, dtype=torch.int32), + eids=[], + multi_modal_inputs=torch.empty(0, dtype=torch.float32), + ) + if custom_fields is not None: + for custom_field in custom_fields: + exps.custom_fields.append(custom_field.destination_field) + setattr( + exps, custom_field.destination_field, torch.empty(0, dtype=custom_field.data_type) + ) + return exps + + +def gather_token_ids( + experiences, max_prompt_length: int, max_response_length: int, pad_token_id: int +) -> Tensor: + token_ids_dtype = experiences[0].tokens.dtype + return torch.stack( + [ + torch.cat( + [ + torch.full( + (max_prompt_length - exp.prompt_length,), + pad_token_id, + dtype=token_ids_dtype, + ), + exp.tokens, + torch.full( + (max_response_length + exp.prompt_length - len(exp.tokens),), + pad_token_id, + dtype=token_ids_dtype, + ), + ] + ) + for exp in experiences + ] + ) + + +def gather_action_masks(experiences, max_response_length: int) -> Tensor: + return torch.stack( + [ + torch.cat( + [ + exp.action_mask, + torch.full( + (max_response_length - len(exp.action_mask),), + 0, + dtype=torch.bool, + ), + ] + ) + for exp in experiences + ] + ) + + +def gather_attention_masks(experiences, max_prompt_length: int, max_response_length: int) -> Tensor: + attention_masks = torch.zeros( + (len(experiences), max_prompt_length + max_response_length), dtype=torch.bool + ) + + for i, exp in enumerate(experiences): + start = max_prompt_length - exp.prompt_length + end = start + len(exp.tokens) + attention_masks[i, start:end] = 1 + + return attention_masks + + +def gather_response_attrs( + experiences, attr_name: str, max_response_length: int, pad_value: int = 0 +) -> Tensor: + dtype = getattr(experiences[0], attr_name).dtype + pad_value = torch.tensor(pad_value, dtype=dtype) + return torch.stack( + [ + torch.cat( + [ + getattr(exp, attr_name), + torch.full( + (max_response_length - len(getattr(exp, attr_name)),), + pad_value, + dtype=dtype, + ), + ] + ) + for exp in experiences + ] + ) + + +def gather_multi_modal_inputs(experiences) -> Dict[str, Tensor]: + keys = experiences[0].multi_modal_inputs.keys() + return {key: [exp.multi_modal_inputs[key] for exp in experiences] for key in keys} + + +def group_by( + experiences: List[Experience], id_type: Literal["task", "run", "step"] +) -> Dict[str, List[Experience]]: + """Group experiences by ID.""" + if id_type == "task": + id_type = "tid" + elif id_type == "run": + id_type = "rid" + elif id_type == "step": + id_type = "sid" + else: + raise ValueError(f"Unknown id_type: {id_type}") + grouped = {} + for exp in experiences: + group_id = getattr(exp.eid, id_type) + if group_id not in grouped: + grouped[group_id] = [] + grouped[group_id].append(exp) + return grouped + + +def to_hf_datasets(experiences: list[Experience]) -> "Dataset": + """ + Convert a list of Experience objects to a HuggingFace Dataset, + preserving all fields. + """ + from datasets import Dataset + + return Dataset.from_list([asdict(exp) for exp in experiences]) + + +def from_hf_datasets(dataset: "Dataset") -> List[Experience]: + """ + Convert a HuggingFace Dataset back to a list of Experience objects. + """ + + def dict_to_dataclass(cls, d): + valid_keys = {f.name for f in fields(cls)} + filtered = {k: v for k, v in d.items() if k in valid_keys} + return cls(**filtered) + + experiences = [dict_to_dataclass(Experience, row) for row in dataset.to_list()] + + return experiences diff --git a/frameworks/trinity/common/models/__init__.py b/frameworks/trinity/common/models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..9dd77e6cb46553cc0bb53553c5b064fad5fc0fbb --- /dev/null +++ b/frameworks/trinity/common/models/__init__.py @@ -0,0 +1,221 @@ +import asyncio +from collections import defaultdict +from typing import List, Tuple + +from trinity.common.config import Config +from trinity.common.constants import DEBUG_NAMESPACE +from trinity.common.models.model import InferenceModel +from trinity.utils.log import get_logger + + +class _BundleAllocator: + """An allocator for bundles.""" + + def __init__(self, node_bundle_map: dict[str, list]) -> None: + self.logger = get_logger(__name__, in_ray_actor=True) + self.node_bundle_list = [value for value in node_bundle_map.values()] + self.node_list = [key for key in node_bundle_map.keys()] + self.nid = 0 + self.bid = 0 + + def allocate(self, num: int) -> list: + # allocate num bundles from current node + if self.bid + num > len(self.node_bundle_list[self.nid]): + raise ValueError( + "Bundle allocation error, a tensor parallel group" + " is allocated across multiple nodes." + ) + bundle_list = self.node_bundle_list[self.nid][self.bid : self.bid + num] + self.logger.info(f"Allocate bundles {bundle_list} on node {self.node_list[self.nid]}.") + self.bid += num + if self.bid == len(self.node_bundle_list[self.nid]): + self.bid = 0 + self.nid += 1 + return bundle_list + + +def create_inference_models( + config: Config, +) -> Tuple[List[InferenceModel], List[List[InferenceModel]]]: + """Create `engine_num` rollout models. + + Each model has `tensor_parallel_size` workers. + """ + import ray + from ray.util.placement_group import placement_group, placement_group_table + from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy + + logger = get_logger(__name__) + engine_num = config.explorer.rollout_model.engine_num + tensor_parallel_size = config.explorer.rollout_model.tensor_parallel_size + + rollout_engines = [] + if config.explorer.rollout_model.engine_type.startswith("vllm"): + from trinity.common.models.vllm_model import vLLMRolloutModel + + engine_cls = vLLMRolloutModel + elif config.explorer.rollout_model.engine_type == "tinker": + from trinity.common.models.tinker_model import TinkerModel + + engine_cls = TinkerModel + namespace = config.ray_namespace + rollout_engines = [ + ray.remote(engine_cls) + .options( + name=f"{config.explorer.name}_rollout_model_{i}", + namespace=namespace, + ) + .remote( + config=config.explorer.rollout_model, + ) + for i in range(engine_num) + ] + auxiliary_engines = [ + [ + ray.remote(engine_cls) + .options( + name=f"{config.explorer.name}_auxiliary_model_{i}_{j}", + namespace=namespace, + ) + .remote( + config=config.explorer.auxiliary_models[i], + ) + for j in range(model_config.engine_num) + ] + for i, model_config in enumerate(config.explorer.auxiliary_models) + ] + return rollout_engines, auxiliary_engines + else: + raise ValueError(f"Unknown engine type: {config.explorer.rollout_model.engine_type}") + + main_bundles = [{"GPU": 1} for _ in range(engine_num * tensor_parallel_size)] + auxiliary_bundles = [ + {"GPU": 1} + for _ in range( + sum( + [ + model.engine_num * model.tensor_parallel_size + for model in config.explorer.auxiliary_models + ] + ) + ) + ] + pg = placement_group(main_bundles + auxiliary_bundles, strategy="PACK") + ray.get(pg.ready()) + + rollout_engines = [] + auxiliary_engines = [] + + # to address https://github.com/ray-project/ray/issues/51117 + # aggregate bundles belonging to the same node + bundle_node_map = placement_group_table(pg)["bundles_to_node_id"] + node_bundle_map = defaultdict(list) + for bundle_id, node_id in bundle_node_map.items(): + node_bundle_map[node_id].append(bundle_id) + allocator = _BundleAllocator(node_bundle_map) + namespace = config.ray_namespace + config.explorer.rollout_model.ray_namespace = namespace + # create rollout models + # in 'serve' mode, we always enable openai api for rollout model + if config.mode == "serve": + config.explorer.rollout_model.enable_openai_api = True + for i in range(config.explorer.rollout_model.engine_num): + bundles_for_engine = allocator.allocate(config.explorer.rollout_model.tensor_parallel_size) + config.explorer.rollout_model.bundle_indices = ",".join( + [str(bid) for bid in bundles_for_engine] + ) + rollout_engines.append( + ray.remote(engine_cls) + .options( + name=f"{config.explorer.name}_rollout_model_{i}", + num_cpus=0, + num_gpus=0 if config.explorer.rollout_model.tensor_parallel_size > 1 else 1, + namespace=namespace, + scheduling_strategy=PlacementGroupSchedulingStrategy( + placement_group=pg, + placement_group_capture_child_tasks=True, + placement_group_bundle_index=bundles_for_engine[0], + ), + ) + .remote( + config=config.explorer.rollout_model, + ) + ) + + if config.explorer.rollout_model.enable_history: + logger.info( + "Model History recording is enabled. Please periodically extract " + "history via `extract_experience_from_history` to avoid out-of-memory issues." + ) + # create auxiliary models + for i, model_config in enumerate(config.explorer.auxiliary_models): + engines = [] + model_config.ray_namespace = namespace + for j in range(model_config.engine_num): + bundles_for_engine = allocator.allocate(model_config.tensor_parallel_size) + model_config.enable_openai_api = True + model_config.engine_type = "vllm" + model_config.bundle_indices = ",".join([str(bid) for bid in bundles_for_engine]) + engines.append( + ray.remote(engine_cls) + .options( + name=f"{config.explorer.name}_auxiliary_model_{i}_{j}", + num_cpus=0, + num_gpus=0 if model_config.tensor_parallel_size > 1 else 1, + namespace=namespace, + scheduling_strategy=PlacementGroupSchedulingStrategy( + placement_group=pg, + placement_group_capture_child_tasks=True, + placement_group_bundle_index=bundles_for_engine[0], + ), + ) + .remote(config=model_config) + ) + auxiliary_engines.append(engines) + + return rollout_engines, auxiliary_engines + + +async def create_debug_inference_model(config: Config) -> None: + """Create inference models for debugging.""" + logger = get_logger(__name__) + logger.info("Creating inference models for debugging...") + # only create one engine for each model + config.explorer.rollout_model.engine_num = 1 + for model in config.explorer.auxiliary_models: + model.engine_num = 1 + rollout_models, auxiliary_models = create_inference_models(config) + # make sure models are started + prepare_refs = [m.prepare.remote() for m in rollout_models] + prepare_refs.extend(m.prepare.remote() for models in auxiliary_models for m in models) + await asyncio.gather(*prepare_refs) + logger.info( + "----------------------------------------------------\n" + "Inference models started successfully for debugging.\n" + "Press Ctrl+C to exit.\n" + "----------------------------------------------------" + ) + + try: + while True: + await asyncio.sleep(1) + except KeyboardInterrupt: + logger.info("Exiting...") + + +def get_debug_inference_model(config: Config) -> Tuple[InferenceModel, List[InferenceModel]]: + """Get the inference models for debugging. + The models must be created by `create_debug_inference_model` in another process first. + """ + import ray + + rollout_model = ray.get_actor( + f"{config.explorer.name}_rollout_model_0", namespace=DEBUG_NAMESPACE + ) + auxiliary_models = [] + for i in range(len(config.explorer.auxiliary_models)): + model = ray.get_actor( + f"{config.explorer.name}_auxiliary_model_{i}_0", namespace=DEBUG_NAMESPACE + ) + auxiliary_models.append(model) + return rollout_model, auxiliary_models diff --git a/frameworks/trinity/common/models/mm_utils.py b/frameworks/trinity/common/models/mm_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..e850f190d04c6edce66cb09ed0c30ecd9f074be3 --- /dev/null +++ b/frameworks/trinity/common/models/mm_utils.py @@ -0,0 +1,73 @@ +""""Multi-modal utilities for processing and handling multi-modal data such as images and videos. +Only support Qwen2.5 VL series. + +Modified from: verl/utils/dataset/rl_dataset.py +""" +import re +from typing import Any, Dict, List + +import numpy as np +from PIL import Image + + +def build_multi_modal_inputs( + prompt: str, + images: List[Image.Image], + videos: List[np.ndarray], + processor: Any, +) -> Dict[str, Any]: + """ + Preprocess multi-modal data and build multi-modal inputs + """ + if prompt is None: + raise ValueError("Prompt is required for build multi-modal inputs") + + multi_modal_data = {} + if images: + multi_modal_data["image"] = images + if videos: + multi_modal_data["video"] = videos + + model_inputs = processor( + text=[prompt], + images=multi_modal_data.get("image", None), + videos=multi_modal_data.get("video", None), + return_tensors="pt", + ) + + input_ids = model_inputs.pop("input_ids")[0] + model_inputs.pop("attention_mask") + + if "second_per_grid_ts" in model_inputs: + model_inputs.pop("second_per_grid_ts") + + return { + "prompt": prompt, + "prompt_token_ids": input_ids, + "multi_modal_data": multi_modal_data, + "multi_modal_inputs": dict(model_inputs), + } + + +def convert_messages_to_mm_format(messages: List[Dict]) -> List[Dict]: + for message in messages: + content = message["content"] + content_list = [] + segments = re.split("(|
")[-1] + + # Find boxed + ans_boxed = _last_boxed_only_string(response) + if ans_boxed: + is_matched = True + response = ans_boxed + + return is_matched, response + + +def compute_score(solution_str: str, ground_truth: str) -> float: + """Compute the reward score for a solution. This draws heavily from the LLM-as-judge and PRIME reward functions + + Args: + solution_str: The solution string + ground_truth: The ground truth answer + extra_info: dict with additional info for the score computation + + Returns: + Reward score (1.0 for correct, 0.0 for incorrect) + """ + from verl.utils.reward_score.prime_math.grader import math_equal + + # First assert intended generation and gt type + model_output = str(solution_str) + ground_truth = str(ground_truth) + + # Extract answer from generated output + is_matched, extracted_model_output = match_answer(model_output) + + # TWK NOTE: WE REMOVED THE RESPONSE TRUNCATION FROM math_dapo.compute_score + + # Verify the solution, first check simple comparisons. + correct, pred = grade_answer(extracted_model_output, ground_truth) + + if not correct: + try: + if "\\pi" in extracted_model_output or "\\pi" in ground_truth: + equivs = [] + for pi in [math.pi, 3.14]: + equivs.append( + math_equal(extracted_model_output, ground_truth, tiemout=True, pi=pi) + ) + correct = any(equivs) + else: + correct = math_equal(extracted_model_output, ground_truth, timeout=True) + except Exception: + correct = False + + return 1.0 if correct else 0.0 diff --git a/frameworks/trinity/common/rewards/qwen25_eval.py b/frameworks/trinity/common/rewards/qwen25_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..d4f61e2d3b1e884fac65d2f8b29db20d466d0819 --- /dev/null +++ b/frameworks/trinity/common/rewards/qwen25_eval.py @@ -0,0 +1,580 @@ +# -*- coding: utf-8 -*- +""" +Utility functions for strictly parsing and evaluating mathematical answers. + +This module is a modified and simplified version of the official evaluation code +for Qwen2.5-Math, designed for easier standalone use. + +Original source: https://github.com/QwenLM/Qwen2.5-Math + +Key modifications include: + +1. Retained only the core parsing logic for the common `qwen_boxed` prompt format. +2. Consolidated essential parsing and evaluation functions from multiple files + into this single module. +3. Simplified benchmark handling and conditional logic for broader applicability. +4. Simplified or removed calls to external tools like TIR. +""" + +import re +from math import isclose +from typing import Any, Dict, Optional, Tuple + +import sympy +from sympy import Matrix, N, simplify, sympify +from sympy.parsing.latex import parse_latex +from word2number import w2n + + +def verify_math_answer(response_text: str, ground_truth: str) -> Tuple[float, Dict[str, Any]]: + """Strictly compare the equality of response and groundtruth.""" + try: + # Parse the response + parsed_prediction = extract_answer(response_text) + + # Parse the ground truth + parsed_truth = extract_answer(str(ground_truth)) + + is_correct = math_equal(prediction=parsed_prediction, reference=parsed_truth) + + accuracy = 1.0 if is_correct else 0.0 + + eval_details = { + "parsed_prediction": parsed_prediction, + "ground_truth": parsed_truth, + "is_correct": is_correct, + } + + return accuracy, eval_details + except Exception: + # If any parsing or comparison error occurs, treat as incorrect + return 0.0, { + "parsed_prediction": None, + "ground_truth": str(ground_truth), + "is_correct": False, + } + + +def extract_answer(response_text: str) -> Optional[str]: + """Extract the equation from the string.""" + if not isinstance(response_text, str): + return None + + # Extract '\boxed{...}' + if "boxed" in response_text: + ans_part = response_text.split("boxed")[-1] + if not ans_part: + return None + + if ans_part.startswith("{"): + stack = 1 + extracted_ans = "" + for char in ans_part[1:]: + if char == "{": + stack += 1 + extracted_ans += char + elif char == "}": + stack -= 1 + if stack == 0: + break + extracted_ans += char + else: + extracted_ans += char + + if stack == 0: + return strip_string(extracted_ans) + + match = re.search(r"\{?([^$}]+)\}?", ans_part) + if match: + return strip_string(match.group(1)) + + # Extract 'answer is ...' + search_patterns = [r"(?:final|the)\s+answer\s+is\s*:?\s*(.+)", r"答案是\s*:?\s*(.+)"] + for pattern in search_patterns: + match = re.search(pattern, response_text, re.IGNORECASE) + if match: + pred = strip_string(match.group(1)) + if pred and pred.endswith("."): + pred = pred[:-1] + return pred + + # Extract the last number + text_no_commas = response_text.replace(",", "") + numeric_finds = re.findall(r"[-+]?\d*\.?\d+", text_no_commas) + if numeric_finds: + return numeric_finds[-1] + + return None + + +# units mainly from MathQA +unit_texts = [ + "east", + "degree", + "mph", + "kmph", + "ft", + "m sqaure", + " m east", + "sq m", + "deg", + "mile", + "q .", + "monkey", + "prime", + "ratio", + "profit of rs", + "rd", + "o", + "gm", + "p . m", + "lb", + "tile", + "per", + "dm", + "lt", + "gain", + "ab", + "way", + "west", + "a .", + "b .", + "c .", + "d .", + "e .", + "f .", + "g .", + "h .", + "t", + "a", + "h", + "no change", + "men", + "soldier", + "pie", + "bc", + "excess", + "st", + "inches", + "noon", + "percent", + "by", + "gal", + "kmh", + "c", + "acre", + "rise", + "a . m", + "th", + "π r 2", + "sq", + "mark", + "l", + "toy", + "coin", + "sq . m", + "gallon", + "° f", + "profit", + "minw", + "yr", + "women", + "feet", + "am", + "pm", + "hr", + "cu cm", + "square", + "v â € ™", + "are", + "rupee", + "rounds", + "cubic", + "cc", + "mtr", + "s", + "ohm", + "number", + "kmph", + "day", + "hour", + "minute", + "min", + "second", + "man", + "woman", + "sec", + "cube", + "mt", + "sq inch", + "mp", + "∏ cm ³", + "hectare", + "more", + "sec", + "unit", + "cu . m", + "cm 2", + "rs .", + "rs", + "kg", + "g", + "month", + "km", + "m", + "cm", + "mm", + "apple", + "liter", + "loss", + "yard", + "pure", + "year", + "increase", + "decrease", + "d", + "less", + "Surface", + "litre", + "pi sq m", + "s .", + "metre", + "meter", + "inch", +] + +unit_texts.extend([t + "s" for t in unit_texts if not t.endswith("s")]) + + +def strip_string(input_str: Optional[str]) -> Optional[str]: + """Clean and normalize math answer strings.""" + if input_str is None: + return None + + string = str(input_str).strip() + + # Basic cleaning and formatting + string = string.replace("\n", "") + string = string.rstrip(".") + string = string.replace("\\!", "") + string = string.replace("\\$", "").replace("$", "") + string = string.replace("%", "").replace("\\%", "") + + # Normalization of LaTeX format + string = re.sub(r"\\begin\{array\}\{.*?\}", r"\\begin{pmatrix}", string) + string = re.sub(r"\\end\{array\}", r"\\end{pmatrix}", string) + string = string.replace("bmatrix", "pmatrix") + string = string.replace("tfrac", "frac").replace("dfrac", "frac") + string = string.replace("\\neq", "\\ne").replace("\\leq", "\\le").replace("\\geq", "\\ge") + string = string.replace("\\left", "").replace("\\right", "") + string = string.replace("\\{", "{").replace("\\}", "}") + string = re.sub(r"\\mbox{.*?}", "", string) + string = string.replace("\\mathbf", "") + + string = string.replace("^{\\circ}", "").replace("^\\circ", "") + + # Remove text and units + string = re.sub(r"\\text\{(.*?)\}", r"\1", string) + + for _ in range(2): + for unit_text in unit_texts: + _string = re.sub(r"(^|\W)" + unit_text + r"($|\W)", r"\1\2", string) + if _string != "": + string = _string + + # Clean numerical values + try: + string = str(w2n.word_to_num(string)) + except ValueError: + pass + + string = re.sub(r"^[a-zA-Z]\s*=\s*", "", string) + + string = string.replace("infinity", "\\infty").replace("inf", "\\infty") + + string = re.sub(r"(\d+)\.0+([^\d]|$)", r"\1\2", string) + + string = string.replace(" .", " 0.").replace("{.", "{0.") + if string.startswith("."): + string = "0" + string + + # Fix the structure and final cleanup + string = string.replace(" ", "") + + string = fix_sqrt(string) + string = fix_fracs(string) + string = fix_a_slash_b(string) + + if (string.startswith("{") and string.endswith("}")) or ( + string.startswith("(") and string.endswith(")") + ): + string = string[1:-1] + + if not string: + return None + + return string.strip() + + +def fix_fracs(string): + substrs = string.split("\\frac") + new_str = substrs[0] + if len(substrs) > 1: + substrs = substrs[1:] + for substr in substrs: + new_str += "\\frac" + if len(substr) > 0 and substr[0] == "{": + new_str += substr + else: + try: + assert len(substr) >= 2 + except AssertionError: + return string + a = substr[0] + b = substr[1] + if b != "{": + if len(substr) > 2: + post_substr = substr[2:] + new_str += "{" + a + "}{" + b + "}" + post_substr + else: + new_str += "{" + a + "}{" + b + "}" + else: + if len(substr) > 2: + post_substr = substr[2:] + new_str += "{" + a + "}" + b + post_substr + else: + new_str += "{" + a + "}" + b + string = new_str + return string + + +def fix_a_slash_b(string): + if len(string.split("/")) != 2: + return string + a = string.split("/")[0] + b = string.split("/")[1] + try: + if "sqrt" not in a: + a = int(a) + if "sqrt" not in b: + b = int(b) + assert string == "{}/{}".format(a, b) + new_string = "\\frac{" + str(a) + "}{" + str(b) + "}" + return new_string + except Exception: + return string + + +def fix_sqrt(string): + if "\\sqrt" not in string: + return string + pattern = r"\\sqrt\s*(\w+)" + replacement = r"\\sqrt{\1}" + + return re.sub(pattern, replacement, string) + + +def convert_word_number(text: str) -> str: + try: + text = str(w2n.word_to_num(text)) + except Exception: + pass + return text + + +def _compare_numerical(pred: str, ref: str) -> Optional[bool]: + """Helper for numerical comparison in math_equal.""" + try: + if _is_digit(pred) and _is_digit(ref): + pred_num = _parse_digits(pred) + ref_num = _parse_digits(ref) + # Allow for percentage conversions + possible_ref_values = {ref_num, ref_num / 100, ref_num * 100} + for val in possible_ref_values: + if numeric_equal(pred_num, val): + return True + return False + except (ValueError, TypeError): + pass + return None + + +def _compare_structures(pred: str, ref: str) -> Optional[bool]: + """Helper for structural comparison (intervals, matrices) in math_equal.""" + is_pred_interval = pred.startswith(("(", "[")) and pred.endswith((")", "]")) + is_ref_interval = ref.startswith(("(", "[")) and ref.endswith((")", "]")) + if is_pred_interval and is_ref_interval: + pred_parts = pred[1:-1].split(",") + ref_parts = ref[1:-1].split(",") + if len(pred_parts) == len(ref_parts) and all( + math_equal(p.strip(), r.strip()) for p, r in zip(pred_parts, ref_parts) + ): + return True + + is_pred_matrix = pred.startswith("\\begin{pmatrix}") + is_ref_matrix = ref.startswith("\\begin{pmatrix}") + if is_pred_matrix and is_ref_matrix: + pred_mat_str = pred[len("\\begin{pmatrix}") : -len("\\end{pmatrix}")] + ref_mat_str = ref[len("\\begin{pmatrix}") : -len("\\end{pmatrix}")] + pred_rows = [row.split("&") for row in pred_mat_str.split("\\\\")] + ref_rows = [row.split("&") for row in ref_mat_str.split("\\\\")] + if ( + len(pred_rows) == len(ref_rows) + and len(pred_rows[0]) == len(ref_rows[0]) + and all( + math_equal(p_elem.strip(), r_elem.strip()) + for p_row, r_row in zip(pred_rows, ref_rows) + for p_elem, r_elem in zip(p_row, r_row) + ) + ): + return True + return None + + +def _compare_equations(pred: str, ref: str) -> Optional[bool]: + """Helper for equation comparison in math_equal.""" + if pred.count("=") == 1 and ref.count("=") == 1: + pred_lhs, pred_rhs = (p.strip() for p in pred.split("=")) + ref_lhs, ref_rhs = (r.strip() for r in ref.split("=")) + if symbolic_equal(f"({pred_lhs})-({pred_rhs})", f"({ref_lhs})-({ref_rhs})"): + return True + + if pred.count("=") == 1 and ref.count("=") == 0: + var, val = (p.strip() for p in pred.split("=")) + if len(var) <= 2 and math_equal(val, ref): + return True + + if ref.count("=") == 1 and pred.count("=") == 0: + var, val = (r.strip() for r in ref.split("=")) + if len(var) <= 2 and math_equal(pred, val): + return True + return None + + +def math_equal(prediction: Optional[str], reference: Optional[str]) -> bool: + """Checks the mathematical equality of two strings by trying different methods.""" + if prediction is None or reference is None: + return False + if prediction == reference: + return True + + comparisons = [ + _compare_numerical, + _compare_structures, + _compare_equations, + ] + for func in comparisons: + result = func(prediction, reference) + if result is not None: + return result + + return symbolic_equal(prediction, reference) + + +def numeric_equal(prediction: float, reference: float) -> bool: + return isclose(reference, prediction, rel_tol=1e-4) + + +def _is_digit(s: str) -> bool: + try: + float(s.replace(",", "")) + return True + except (ValueError, TypeError): + return False + + +def _parse_digits(s: str) -> float: + return float(s.replace(",", "")) + + +def _parse_symbolic(s: str) -> Any: + """Parse a string into a sympy expression, trying different methods.""" + if not isinstance(s, str) or not s: + return s + + s_cleaned = s.replace("\\\\", "\\") + try: + # Use a local dict to handle functions like exp + return parse_latex(s_cleaned, locals={"exp": sympy.exp}) + except Exception: # Broad exception is okay here as we have fallbacks + try: + return sympify(s_cleaned, evaluate=True) + except (sympy.SympifyError, TypeError, SyntaxError, AttributeError, ValueError): + return s # Return original string if all parsing fails + + +def _check_direct_or_simplified_equality(a_sym: Any, b_sym: Any) -> Optional[bool]: + """Check for direct or simplified symbolic equality.""" + try: + if a_sym == b_sym: + return True + except (TypeError, ValueError, AttributeError): + pass + try: + if simplify(a_sym - b_sym) == 0: + return True + except (AttributeError, TypeError, ValueError, sympy.SympifyError): + pass + return None + + +def _check_equation_equality(a_sym: Any, b_sym: Any) -> Optional[bool]: + """Check for symbolic equality of two equations.""" + try: + if isinstance(a_sym, sympy.Eq) and isinstance(b_sym, sympy.Eq): + if (a_sym.lhs - a_sym.rhs).equals(b_sym.lhs - b_sym.rhs): + return True + except (AttributeError, TypeError): + pass + return None + + +def _check_numeric_value_equality(a_sym: Any, b_sym: Any) -> Optional[bool]: + """Check for equality of the numerical values of two symbolic expressions.""" + try: + # Check if both are strings (parsing failed), compare directly + if isinstance(a_sym, str) and isinstance(b_sym, str): + return None # Let other checks handle string comparison + if numeric_equal(float(N(a_sym)), float(N(b_sym))): + return True + except (TypeError, ValueError, AttributeError, sympy.SympifyError): + pass + return None + + +def _check_matrix_equality(a_sym: Any, b_sym: Any) -> Optional[bool]: + """Check for symbolic equality of two matrices.""" + try: + if isinstance(a_sym, Matrix) and isinstance(b_sym, Matrix): + if a_sym.shape == b_sym.shape and simplify(a_sym - b_sym).is_zero_matrix: + return True + except (AttributeError, TypeError): + pass + return None + + +def symbolic_equal(a: str, b: str) -> bool: + """Compares two strings for symbolic equivalence using sympy.""" + try: + a_sym = _parse_symbolic(a) + b_sym = _parse_symbolic(b) + + equality_checks = [ + _check_direct_or_simplified_equality, + _check_equation_equality, + _check_numeric_value_equality, + _check_matrix_equality, + ] + + for check_func in equality_checks: + try: + result = check_func(a_sym, b_sym) + if result is True: + return True + except Exception: + continue + + return False + except Exception: + # If any error occurs during symbolic comparison, fall back to string comparison + return str(a) == str(b) diff --git a/frameworks/trinity/common/rewards/reward_fn.py b/frameworks/trinity/common/rewards/reward_fn.py new file mode 100644 index 0000000000000000000000000000000000000000..90acfa340a170ecfec2da31a9a03bd4706b7b4cf --- /dev/null +++ b/frameworks/trinity/common/rewards/reward_fn.py @@ -0,0 +1,101 @@ +# -*- coding: utf-8 -*- +"""Base Reward Function Class.""" +from abc import ABC, abstractmethod +from typing import Any, Dict, List + +from trinity.common.experience import Experience +from trinity.common.rewards.utils import to_rm_gallery_messages + + +class RewardFn(ABC): + """Base Reward Function Class.""" + + @abstractmethod + def __init__(self, **kwargs) -> None: + pass + + @abstractmethod + def __call__(self, **kwargs) -> Dict[str, float]: + pass + + +class RMGalleryFn(RewardFn): + """Reward Function from RMGallery. + https://github.com/modelscope/RM-Gallery + + TODO: Update to OpenJudgeFn + """ + + def __init__( + self, + reward_name, + **kwargs, + ): + from rm_gallery.core.reward.registry import RewardRegistry + + self.reward_model = RewardRegistry.get(reward_name)(**kwargs) + + def __call__(self, experience: Experience, messages: List[Dict[str, Any]], **kwargs) -> Dict[str, float]: # type: ignore + """Call the reward function.""" + + sample = self._build_sample_from_experience(experience, messages, **kwargs) + + sample_with_reward = self.reward_model.evaluate(sample, **kwargs) + + return self._extract_reward(sample_with_reward) + + def _build_sample_from_experience( + self, experience: Experience, messages: List[Dict[str, Any]], **kwargs + ) -> Any: + """Convert experience to sample. + Ref: https://github.com/modelscope/RM-Gallery/blob/main/rm_gallery/core/data/schema.py + """ + from rm_gallery.core.data.schema import DataOutput, DataSample, Step + + output = [ + DataOutput( + answer=Step( + role="assistant", + content=str(experience.response_text), + label={"reference": kwargs.get("ground_truth", "")}, + ), + ) + ] + + sample = DataSample( + unique_id=experience.eid.uid, + input=to_rm_gallery_messages(messages), + output=output, + metadata=experience.info, + ) + return sample + + def _extract_reward(self, sample: Any) -> Dict[str, float]: + """ + Extract reward from DataSample in rm-gallery + """ + reward_dict = {} + + try: + reward_obj = sample.output[0].answer.reward + except Exception as e: + raise ValueError(f"No reward is found in sample: {e}") + + from rm_gallery.core.reward.schema import ( + RewardDimensionWithRank, + RewardDimensionWithScore, + ) + + if reward_obj.details: + for detail in reward_obj.details: + if isinstance(detail, RewardDimensionWithScore): + reward_dict[detail.name] = detail.score + elif isinstance(detail, RewardDimensionWithRank): + # TODO: support multi-ranked dimension + if detail: + top_ranked_item = detail[0] + reward_dict[top_ranked_item.name] = top_ranked_item.score + else: + reward_dict["reward"] = reward_obj.score + + return reward_dict diff --git a/frameworks/trinity/common/rewards/tool_reward.py b/frameworks/trinity/common/rewards/tool_reward.py new file mode 100644 index 0000000000000000000000000000000000000000..2624b4cc9d5c9b1be0e71b6424b3138caa726a3a --- /dev/null +++ b/frameworks/trinity/common/rewards/tool_reward.py @@ -0,0 +1 @@ +# to be implemented diff --git a/frameworks/trinity/common/rewards/utils.py b/frameworks/trinity/common/rewards/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..0b66e6700cbe6f344a03143b4c90f6a8d76f9df6 --- /dev/null +++ b/frameworks/trinity/common/rewards/utils.py @@ -0,0 +1,22 @@ +from typing import Any, Dict, List + + +def to_rm_gallery_messages(messages: List[Dict[str, Any]]) -> Any: + """ + Converts string list to structured ChatMessage list for debugging. + + Args: + messages: List of alternating user/assistant messages + + Returns: + List of structured ChatMessage objects + """ + from rm_gallery.core.model.message import ChatMessage, MessageRole + + role_map = { + "system": MessageRole.SYSTEM, + "user": MessageRole.USER, + "assistant": MessageRole.ASSISTANT, + } + + return [ChatMessage(role=role_map[msg["role"]], content=msg["content"]) for msg in messages] diff --git a/frameworks/trinity/common/verl_config.py b/frameworks/trinity/common/verl_config.py new file mode 100644 index 0000000000000000000000000000000000000000..1b04d8e25ae6f71c0573c6445fa29dc7789cfb61 --- /dev/null +++ b/frameworks/trinity/common/verl_config.py @@ -0,0 +1,604 @@ +import math +import sys +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional + +from omegaconf import OmegaConf +from verl.workers.config import PolicyLossConfig, RouterReplayConfig + +from trinity.algorithm import ALGORITHM_TYPE +from trinity.common.config import Config, SynchronizerConfig, set_if_none +from trinity.common.constants import EXPLORER_NAME +from trinity.utils.log import get_logger + +logger = get_logger(__name__) + + +@dataclass +class Data: + train_batch_size: int = 1024 # kept to pass RayPPOTrainer._validate_config + trust_remote_code: bool = False + + +@dataclass +class FusedKernelOptions: + impl_backend: Optional[str] = None + + +@dataclass +class ActorModel: + path: str = "" + external_lib: Optional[str] = None + override_config: Dict[str, Any] = field(default_factory=dict) + enable_gradient_checkpointing: bool = True + use_remove_padding: bool = True + use_fused_kernels: bool = False + fused_kernel_options: FusedKernelOptions = field(default_factory=FusedKernelOptions) + custom_chat_template: Optional[str] = None + enable_activation_offload: bool = False + use_shm: bool = False + trust_remote_code: bool = False # Whether to enable loading a remote code model + + # lora configs + lora_rank: int = 0 # The rank of the LoRA model, default to 0. If lora_rank > 0, LoRA module is enabled in trainer + lora_alpha: int = 32 + target_modules: Optional[str] = "all-linear" + exclude_modules: Optional[str] = None + lora_adapter_path: Optional[str] = None + + # rope configs + rope_scaling: Optional[dict] = None + rope_theta: Optional[float] = None + + +@dataclass +class Optim: + # For actor, most fields are set in algorithm.optimizer + # For critic, you can set trainer_config.critic.optim + optimizer: str = "adam" + optimizer_impl: str = "torch.optim" + lr: float = 1e-6 + lr_warmup_steps: int = -1 + lr_warmup_steps_ratio: float = 0.0 + min_lr_ratio: Optional[float] = 0.0 + warmup_style: Optional[str] = None # deprecated ! + lr_scheduler_type: str = "constant" + total_training_steps: int = -1 # ! DO NOT SET, use trainer.total_steps + betas: List[float] = field(default_factory=lambda: [0.9, 0.999]) + clip_grad: float = 1.0 + lr_warmup_init: Optional[float] = None # 0.0 + lr_decay_steps: Optional[int] = None + lr_decay_style: Optional[str] = None # "constant" + min_lr: Optional[float] = None # 0.0 + weight_decay: float = 0.01 + weight_decay_incr_style: str = "constant" + lr_wsd_decay_style: str = "exponential" + lr_wsd_decay_steps: Optional[int] = None + use_checkpoint_opt_param_scheduler: bool = False + override_optimizer_config: Optional[dict] = None + + +@dataclass +class WrapPolicy: + min_num_params: int = 0 + + +@dataclass +class FSDPConfig: + _target_: str = "verl.workers.config.FSDPEngineConfig" # DO NOT SET + param_offload: bool = False + optimizer_offload: bool = False + offload_policy: bool = False + reshard_after_forward: bool = True + wrap_policy: WrapPolicy = field(default_factory=WrapPolicy) + fsdp_size: int = -1 + forward_prefetch: bool = False + model_dtype: Optional[str] = None + + +@dataclass +class Checkpoint: + load_contents: List[str] = field(default_factory=lambda: ["model", "optimizer", "extra"]) + save_contents: List[str] = field(default_factory=lambda: ["model", "optimizer", "extra"]) + async_save: bool = False # TODO: testing async save + + +@dataclass +class OverrideTransformerConfig: + recompute_granularity: Optional[str] = "full" + recompute_modules: List[str] = field(default_factory=lambda: ["core_attn"]) + recompute_method: Optional[str] = "uniform" + recompute_num_layers: Optional[int] = 1 + + +@dataclass +class MegatronConfig: + param_offload: bool = False + grad_offload: bool = False + optimizer_offload: bool = False + tensor_model_parallel_size: int = 1 + expert_model_parallel_size: int = 1 + expert_tensor_parallel_size: Optional[int] = None + pipeline_model_parallel_size: int = 1 + virtual_pipeline_model_parallel_size: Optional[int] = None + context_parallel_size: int = 1 + sequence_parallel: bool = True + use_distributed_optimizer: bool = True + use_dist_checkpointing: bool = False + dist_checkpointing_path: Optional[str] = None + seed: int = 42 + override_ddp_config: dict = field(default_factory=dict) + override_transformer_config: OverrideTransformerConfig = field( + default_factory=OverrideTransformerConfig + ) + use_mbridge: bool = False + dtype: str = "bfloat16" + use_remove_padding: bool = True + + +@dataclass +class ProfileConfig: + use_profile: bool = False + profile_ranks: Optional[List[int]] = None + step_start: int = -1 + step_end: int = -1 + save_path: Optional[str] = None + + +@dataclass +class Actor: + strategy: Optional[str] = None + ppo_mini_batch_size: int = 256 + ppo_micro_batch_size: Optional[int] = None + ppo_micro_batch_size_per_gpu: int = 1 + use_dynamic_bsz: Optional[bool] = None + ppo_max_token_len_per_gpu: Optional[int] = None + fix_actor_microbatch_loss_scale: Optional[bool] = None # EXPERIMENTAL + grad_clip: Optional[float] = None + ppo_epochs: int = 1 + shuffle: bool = False + ulysses_sequence_parallel_size: Optional[int] = None + entropy_from_logits_with_chunking: bool = False + entropy_checkpointing: bool = False + checkpoint: Checkpoint = field(default_factory=Checkpoint) + optim: Optim = field(default_factory=Optim) + fsdp_config: FSDPConfig = field(default_factory=FSDPConfig) + megatron: MegatronConfig = field(default_factory=MegatronConfig) + profile: ProfileConfig = field(default_factory=ProfileConfig) + data_loader_seed: Optional[int] = None + load_weight: bool = True + policy_loss: PolicyLossConfig = field(default_factory=PolicyLossConfig) + profiler: dict = field(default_factory=dict) + router_replay: RouterReplayConfig = field(default_factory=RouterReplayConfig) + # do not set + loss_agg_mode: str = "token-mean" + clip_ratio: float = 0.2 + clip_ratio_low: Optional[float] = None + clip_ratio_high: Optional[float] = None + entropy_coeff: float = 0.001 + use_kl_loss: bool = False + + +@dataclass +class Ref: + fsdp_config: FSDPConfig = field(default_factory=FSDPConfig) + log_prob_micro_batch_size: Optional[int] = None + log_prob_micro_batch_size_per_gpu: int = 1 + log_prob_use_dynamic_bsz: Optional[bool] = None + log_prob_max_token_len_per_gpu: Optional[int] = None + ulysses_sequence_parallel_size: Optional[int] = None + entropy_from_logits_with_chunking: bool = False + entropy_checkpointing: bool = False + checkpoint: Checkpoint = field( + default_factory=lambda: Checkpoint(load_contents=["model"], save_contents=["model"]) + ) + megatron: MegatronConfig = field(default_factory=MegatronConfig) + profile: ProfileConfig = field(default_factory=ProfileConfig) + load_weight: bool = True + profiler: dict = field(default_factory=dict) + router_replay: RouterReplayConfig = field(default_factory=RouterReplayConfig) + + +@dataclass +class _ValKwargs: + do_sample: bool = False + + +@dataclass +class _MultiTurn: + enable: bool = False + + +@dataclass +class Rollout: + # do not set + val_kwargs: _ValKwargs = field(default_factory=_ValKwargs) + multi_turn: _MultiTurn = field(default_factory=_MultiTurn) + temperature: float = 1.0 + n: int = 1 # > 1 for grpo + log_prob_use_dynamic_bsz: Optional[bool] = None + log_prob_micro_batch_size: Optional[int] = None + log_prob_micro_batch_size_per_gpu: Optional[int] = None + log_prob_max_token_len_per_gpu: Optional[int] = None + + +@dataclass +class ActorRolloutRef: + hybrid_engine: bool = True + model: ActorModel = field(default_factory=ActorModel) + actor: Actor = field(default_factory=Actor) + ref: Ref = field(default_factory=Ref) + rollout: Rollout = field(default_factory=Rollout) + nccl_timeout: float = 600 # ! DO NOT SET, it will be set by `config.synchronizer.sync_timeout` + synchronizer: Optional[SynchronizerConfig] = None + explorer_name: str = EXPLORER_NAME + + +@dataclass +class CriticModel: + path: str = "" + tokenizer_path: str = "" + override_config: Dict[str, str] = field(default_factory=dict) + external_lib: Optional[str] = None + trust_remote_code: bool = False # Whether to enable loading a remote code model + enable_gradient_checkpointing: bool = True + use_remove_padding: bool = True + fsdp_config: FSDPConfig = field(default_factory=FSDPConfig) + + # rope configs + rope_scaling: Optional[dict] = None + rope_theta: Optional[float] = None + + +@dataclass +class Critic: + enable: bool = False + strategy: Optional[str] = None + optim: Optim = field(default_factory=Optim) + model: CriticModel = field(default_factory=CriticModel) + ppo_mini_batch_size: int = 0 + ppo_micro_batch_size: Optional[int] = None + ppo_micro_batch_size_per_gpu: int = 1 + forward_micro_batch_size: Optional[int] = None + forward_micro_batch_size_per_gpu: Optional[int] = None + use_dynamic_bsz: Optional[bool] = None + ppo_max_token_len_per_gpu: Optional[int] = None + forward_max_token_len_per_gpu: Optional[int] = None + ulysses_sequence_parallel_size: Optional[int] = None + ppo_epochs: int = 1 + shuffle: bool = False + grad_clip: Optional[float] = None + cliprange_value: float = 0.0 + checkpoint: Checkpoint = field(default_factory=Checkpoint) + rollout_n: int = 1 + loss_agg_mode: str = "token-mean" + megatron: MegatronConfig = field(default_factory=MegatronConfig) + profile: ProfileConfig = field(default_factory=ProfileConfig) + data_loader_seed: Optional[int] = None + load_weight: bool = True + nccl_timeout: float = 600 # ! DO NOT SET, it will be set by `config.synchronizer.sync_timeout` + ray_namespace: str = "" # automatically generated + profiler: dict = field(default_factory=dict) + + +@dataclass +class _RewardModel: + input_tokenizer: Optional[str] = None + path: str = "" + external_lib: Optional[str] = None + use_remove_padding: bool = False + fsdp_config: FSDPConfig = field(default_factory=FSDPConfig) + + +@dataclass +class RewardModel: + enable: bool = False + strategy: Optional[str] = None + model: _RewardModel = field(default_factory=_RewardModel) + micro_batch_size_per_gpu: int = 1 + max_length: Optional[int] = None + ulysses_sequence_parallel_size: int = 1 + use_dynamic_bsz: bool = False + forward_max_token_len_per_gpu: int = 0 + reward_manager: str = "naive" + use_reward_loop: bool = True + + +@dataclass +class CustomRewardFunction: + path: Optional[str] = None + name: str = "compute_score" + + +@dataclass +class KL_Ctrl: + type: str = "fixed" + kl_coef: float = 0.001 + horizon: float = 10000 + target_kl: float = 0.1 + + +@dataclass +class RolloutCorrection: + rollout_is: Optional[str] = None + rollout_is_threshold: float = 2.0 + rollout_rs: Optional[str] = None + rollout_rs_threshold: Optional[float] = None + rollout_rs_threshold_lower: Optional[float] = None + rollout_token_veto_threshold: Optional[float] = None + # Because rollout and training in Trinity runs separately, + # rollout_is_batch_normalize is default to True + bypass_mode: bool = True + loss_type: str = "ppo_clip" + rollout_is_batch_normalize: bool = False + + +@dataclass +class Algorithm: + rollout_correction: RolloutCorrection = field(default_factory=RolloutCorrection) + # ! DO NOT SET gamma or lam below; they are kept here merely for compatibility with verl, + # and their values will be overwritten by those in AlgorithmConfig.advantage_fn_args + # if they are really needed (e.g., for GAE advantage/returns computation) + gamma: float = 1.0 + lam: float = 1.0 + adv_estimator: str = "gae" + norm_adv_by_std_in_grpo: bool = True + use_kl_in_reward: bool = False + kl_penalty: str = "kl" + kl_ctrl: KL_Ctrl = field(default_factory=KL_Ctrl) + + +@dataclass +class Trainer: + balance_batch: bool = True + total_epochs: int = 30 + total_training_steps: Optional[ + int + ] = None # ! DO NOT SET, use trainer.total_steps in global_config + project_name: str = "" + group_name: str = "" + experiment_name: str = "" + logger: List[str] = field(default_factory=list) + val_generations_to_log_to_wandb: int = 0 + nnodes: int = 0 + n_gpus_per_node: int = 0 + save_freq: int = 0 + resume_mode: str = "auto" + resume_from_path: str = "" + test_freq: int = 0 + critic_warmup: int = 0 + default_hdfs_dir: Optional[str] = None + remove_previous_ckpt_in_save: bool = False # deprecated + del_local_ckpt_after_load: bool = False + default_local_dir: str = "" + val_before_train: bool = False + training_rollout_mode: str = "parallel" + enable_exp_buffer: bool = True + sync_freq: int = 0 + max_actor_ckpt_to_keep: Optional[int] = None + max_critic_ckpt_to_keep: Optional[int] = None + device: str = "cuda" # default to cuda + + +@dataclass +class veRLConfig: + data: Data = field(default_factory=Data) + actor_rollout_ref: ActorRolloutRef = field(default_factory=ActorRolloutRef) + critic: Critic = field(default_factory=Critic) + reward_model: RewardModel = field(default_factory=RewardModel) + custom_reward_function: CustomRewardFunction = field(default_factory=CustomRewardFunction) + algorithm: Algorithm = field(default_factory=Algorithm) + trainer: Trainer = field(default_factory=Trainer) + global_profiler: dict = field(default_factory=dict) + synchronizer: Optional[SynchronizerConfig] = None + enable_preview: bool = True + + def _adjust_token_len_if_needed( + self, + obj, + config: Config, + component_name: str, + token_len_attr: str = "ppo_max_token_len_per_gpu", + seq_parallel_attr: str = "ulysses_sequence_parallel_size", + ) -> None: + """ + Helper to adjust token length per GPU if current setting is too small. + + Ensures: token_len * seq_parallel >= config.model.max_model_len * 2 + """ + current_token_len = getattr(obj, token_len_attr) + seq_parallel = getattr(obj, seq_parallel_attr) + required_min = config.model.max_model_len * 2 # type: ignore + + if current_token_len * seq_parallel < required_min: + new_token_len = math.ceil(required_min / seq_parallel) + setattr(obj, token_len_attr, new_token_len) + logger.warning( + f"{component_name}.{token_len_attr} is automatically set to {new_token_len} " + f"to match model.max_model_len ({config.model.max_model_len}). If you face OOM issues, " + "please set `model.max_model_len` to a smaller value." + ) + + def synchronize_config(self, config: Config) -> None: # noqa: C901 + """Synchronize config.""" + # Trainer Config + self.trainer.nnodes = config.cluster.trainer_node_num + self.trainer.n_gpus_per_node = config.cluster.trainer_gpu_num_per_node + self.trainer.total_training_steps = config.trainer.total_steps or sys.maxsize + self.trainer.sync_freq = config.synchronizer.sync_interval + self.trainer.save_freq = config.trainer.save_interval + self.trainer.project_name = config.project + self.trainer.group_name = config.group + self.trainer.experiment_name = config.name + self.trainer.default_local_dir = config.checkpoint_job_dir + if config.trainer.max_checkpoints_to_keep is not None: + self.trainer.max_actor_ckpt_to_keep = config.trainer.max_checkpoints_to_keep + self.trainer.max_critic_ckpt_to_keep = config.trainer.max_checkpoints_to_keep + if not config.continue_from_checkpoint: + self.trainer.resume_mode = "disable" + else: + self.trainer.resume_mode = "auto" + + # kept to pass RayPPOTrainer._validate_config + self.data.train_batch_size = config.buffer.train_batch_size + + self.synchronizer = config.synchronizer + self.actor_rollout_ref.nccl_timeout = config.synchronizer.sync_timeout + self.actor_rollout_ref.synchronizer = config.synchronizer + self.actor_rollout_ref.explorer_name = config.explorer.name + algorithm = ALGORITHM_TYPE.get(config.algorithm.algorithm_type) + self.critic.enable = algorithm.use_critic + self.critic.nccl_timeout = config.synchronizer.sync_timeout + self.critic.ray_namespace = config.synchronizer.ray_namespace + + # Actor / Rollout Config + actor_config = self.actor_rollout_ref.actor + rollout_config = self.actor_rollout_ref.rollout + actor_model_config = self.actor_rollout_ref.model + actor_optim = actor_config.optim + actor_model_config.path = config.model.model_path + actor_model_config.custom_chat_template = config.model.custom_chat_template + actor_model_config.rope_scaling = config.model.rope_scaling + actor_model_config.rope_theta = config.model.rope_theta + actor_optim.total_training_steps = self.trainer.total_training_steps + actor_config.ppo_mini_batch_size = config.buffer.train_batch_size + rollout_config.temperature = ( + config.buffer.explorer_input.tasksets[0].rollout_args.temperature + if config.buffer.explorer_input.tasksets + else 1.0 + ) + rollout_config.n = config.algorithm.repeat_times + for actor_attr, trainer_attr in [ + ("grad_clip",) * 2, + ("use_dynamic_bsz",) * 2, + ("fix_actor_microbatch_loss_scale",) * 2, + ("ulysses_sequence_parallel_size",) * 2, + ("ppo_max_token_len_per_gpu", "max_token_len_per_gpu"), + ("strategy", "trainer_strategy"), + ]: + set_if_none(actor_config, actor_attr, getattr(config.trainer, trainer_attr)) + self._adjust_token_len_if_needed( + obj=self.actor_rollout_ref.actor, + config=config, + component_name="actor", + ) + + # Ref Config + ref_config = self.actor_rollout_ref.ref + for ref_attr, trainer_attr in [ + ("log_prob_use_dynamic_bsz", "use_dynamic_bsz"), + ("log_prob_max_token_len_per_gpu", "max_token_len_per_gpu"), + ("ulysses_sequence_parallel_size",) * 2, + ]: + set_if_none(ref_config, ref_attr, getattr(config.trainer, trainer_attr)) + self._adjust_token_len_if_needed( + obj=self.actor_rollout_ref.ref, + config=config, + component_name="ref", + token_len_attr="log_prob_max_token_len_per_gpu", + ) + + # Critic config + critic_optim = self.critic.optim + self.critic.model.path = config.model.critic_model_path + self.critic.model.tokenizer_path = config.model.critic_model_path + self.critic.model.rope_scaling = config.model.rope_scaling + self.critic.model.rope_theta = config.model.rope_theta + self.critic.ppo_mini_batch_size = config.buffer.train_batch_size + self.critic.rollout_n = config.algorithm.repeat_times + critic_optim.total_training_steps = self.trainer.total_training_steps + for critic_attr, trainer_attr in [ + ("grad_clip",) * 2, + ("use_dynamic_bsz",) * 2, + ("ulysses_sequence_parallel_size",) * 2, + ("strategy", "trainer_strategy"), + ("ppo_max_token_len_per_gpu", "max_token_len_per_gpu"), + ]: + set_if_none(self.critic, critic_attr, getattr(config.trainer, trainer_attr)) + + self._adjust_token_len_if_needed( + obj=self.critic, + config=config, + component_name="critic", + ) + set_if_none( + self.critic, "forward_max_token_len_per_gpu", self.critic.ppo_max_token_len_per_gpu + ) + + # LoRA related config + if config.model.lora_configs is not None: + lora_config = config.model.lora_configs[0] + for attr in ["lora_rank", "lora_alpha", "target_modules", "exclude_modules"]: + setattr(actor_model_config, attr, getattr(lora_config, attr)) + if not lora_config.is_dummy: + actor_model_config.lora_adapter_path = lora_config.path + if actor_config.strategy not in ["fsdp", "fsdp2"]: + logger.warning( + f"Lora is only supported for fsdp and fsdp2, but got {actor_config.strategy} instead, changed to fsdp." + ) + actor_config.strategy = "fsdp" + if self.critic.strategy not in ["fsdp", "fsdp2"]: + logger.warning( + f"Lora is only supported for fsdp and fsdp2, but got {self.critic.strategy} instead, changed to fsdp." + ) + self.critic.strategy = "fsdp" + + # Algorithm related config + optim_config = config.algorithm.optimizer + for field_name in optim_config.__dataclass_fields__: + field_value = getattr(optim_config, field_name) + if field_name == "optimizer_type": + setattr(actor_optim, "optimizer", field_value) + elif hasattr(actor_optim, field_name): + setattr(actor_optim, field_name, field_value) + # ensure megatron optimizer config compatibility + set_if_none(actor_optim, "lr_warmup_init", optim_config.min_lr_ratio * optim_config.lr) + set_if_none(actor_optim, "lr_decay_steps", self.trainer.total_training_steps) + set_if_none(actor_optim, "lr_decay_style", optim_config.lr_scheduler_type) + set_if_none(actor_optim, "min_lr", optim_config.min_lr_ratio * optim_config.lr) + set_if_none(critic_optim, "lr_warmup_init", 0.0) + set_if_none(critic_optim, "lr_decay_steps", self.trainer.total_training_steps) + set_if_none(critic_optim, "lr_decay_style", "constant") + set_if_none(critic_optim, "min_lr", 0.0) + # fix optimizer type for fsdp + if config.trainer.trainer_strategy.startswith("fsdp"): + optim_map = { + "adam": "AdamW", + "adamw": "AdamW", + "sgd": "SGD", + } + actor_optim.optimizer = optim_map.get(actor_optim.optimizer, actor_optim.optimizer) + critic_optim.optimizer = optim_map.get(critic_optim.optimizer, critic_optim.optimizer) + actor_config.use_kl_loss = config.algorithm.kl_loss_fn != "none" + self.algorithm.use_kl_in_reward = config.algorithm.kl_penalty_fn != "none" + # TODO (yanxi): it seems that adv_estimator now only affects whether use_critic is set to + # True or False in RayPPOTrainer.__init__() (and hence in VerlPPOTrainerWrapper). + # Need to double check whether this is indeed the case, + # and see if adv_estimator can be removed completely. + + if config.algorithm.algorithm_type == "dpo": # for DPO + logger.warning("DPO micro batch size is doubled for computing loss.") + actor_config.ppo_micro_batch_size_per_gpu *= 2 + ref_config.log_prob_micro_batch_size_per_gpu *= 2 + + # check rollout config (only works for lora) + for rollout_attr, actor_attr in [ + ("log_prob_use_dynamic_bsz", "use_dynamic_bsz"), + ("log_prob_micro_batch_size", "ppo_micro_batch_size"), + ("log_prob_micro_batch_size_per_gpu", "ppo_micro_batch_size_per_gpu"), + ("log_prob_max_token_len_per_gpu", "ppo_max_token_len_per_gpu"), + ]: + set_if_none(rollout_config, rollout_attr, getattr(actor_config, actor_attr)) + + # TODO: check other fields + self.enable_preview = config.trainer.enable_preview + + +def load_config(config_path: str) -> veRLConfig: + schema = OmegaConf.structured(veRLConfig) + yaml_config = OmegaConf.load(config_path) + try: + config = OmegaConf.merge(schema, yaml_config) + return OmegaConf.to_object(config) + except Exception as e: + raise ValueError(f"Invalid configuration: {e}") from e diff --git a/frameworks/trinity/common/workflows/__init__.py b/frameworks/trinity/common/workflows/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..774f85c5bfc606708fa19283f66c68f10e0e2de8 --- /dev/null +++ b/frameworks/trinity/common/workflows/__init__.py @@ -0,0 +1,82 @@ +# -*- coding: utf-8 -*- +"""Workflow module""" +from trinity.common.workflows.workflow import Task, Workflow +from trinity.utils.registry import Registry + +WORKFLOWS: Registry = Registry( + "workflows", + default_mapping={ + # simple/math + "simple_workflow": "trinity.common.workflows.workflow.SimpleWorkflow", + "async_simple_workflow": "trinity.common.workflows.workflow.AsyncSimpleWorkflow", + "math_workflow": "trinity.common.workflows.workflow.MathWorkflow", + "async_math_workflow": "trinity.common.workflows.workflow.AsyncMathWorkflow", + "math_boxed_workflow": "trinity.common.workflows.customized_math_workflows.MathBoxedWorkflow", + "async_math_boxed_workflow": "trinity.common.workflows.customized_math_workflows.AsyncMathBoxedWorkflow", + "math_eval_workflow": "trinity.common.workflows.eval_workflow.MathEvalWorkflow", + "async_math_eval_workflow": "trinity.common.workflows.eval_workflow.AsyncMathEvalWorkflow", + "math_rm_workflow": "trinity.common.workflows.math_rm_workflow.MathRMWorkflow", + "async_math_rm_workflow": "trinity.common.workflows.math_rm_workflow.AsyncMathRMWorkflow", + # tool_call + "tool_call_workflow": "trinity.common.workflows.customized_toolcall_workflows.ToolCallWorkflow", + # agentscope + "agentscope_workflow_adapter": "trinity.common.workflows.agentscope_workflow.AgentScopeWorkflowAdapter", + "agentscope_workflow_adapter_v1": "trinity.common.workflows.agentscope_workflow.AgentScopeWorkflowAdapterV1", + "agentscope_react_workflow": "trinity.common.workflows.agentscope.react.react_workflow.AgentScopeReActWorkflow", + "agentscope_react_math_workflow": "trinity.common.workflows.envs.agentscope.agentscopev1_react_workflow.AgentScopeReactMathWorkflow", + "as_react_workflow": "trinity.common.workflows.agentscope.react.react_workflow.AgentScopeReActWorkflow", + "agentscopev0_react_math_workflow": "trinity.common.workflows.envs.agentscope.agentscopev0_react_workflow.AgentScopeV0ReactMathWorkflow", + "agentscope_v1_react_search_workflow": "trinity.common.workflows.envs.agentscope.agentscopev1_search_workflow.AgentScopeV1ReactSearchWorkflow", + "email_search_workflow": "trinity.common.workflows.envs.email_searcher.workflow.EmailSearchWorkflow", + # concatenated multi-turn + "alfworld_workflow": "trinity.common.workflows.envs.alfworld.alfworld_workflow.AlfworldWorkflow", + "RAFT_alfworld_workflow": "trinity.common.workflows.envs.alfworld.RAFT_alfworld_workflow.RAFTAlfworldWorkflow", + "RAFT_reflect_alfworld_workflow": "trinity.common.workflows.envs.alfworld.RAFT_reflect_alfworld_workflow.RAFTReflectAlfworldWorkflow", + "frozen_lake_workflow": "trinity.common.workflows.envs.frozen_lake.workflow.FrozenLakeWorkflow", + "sciworld_workflow": "trinity.common.workflows.envs.sciworld.sciworld_workflow.SciWorldWorkflow", + "webshop_workflow": "trinity.common.workflows.envs.webshop.webshop_workflow.WebShopWorkflow", + # general multi-turn + "step_wise_alfworld_workflow": "trinity.common.workflows.envs.alfworld.alfworld_workflow.StepWiseAlfworldWorkflow", + "sdop_alfworld_workflow": "trinity.common.workflows.envs.alfworld.sdop_alfworld_workflow.SDOpAlfworldWorkflow", + # ruler/llm_as_a_judge + "async_math_ruler_workflow": "trinity.common.workflows.math_ruler_workflow.AsyncMathRULERWorkflow", + "math_ruler_workflow": "trinity.common.workflows.math_ruler_workflow.MathRULERWorkflow", + "math_trainable_ruler_workflow": "trinity.common.workflows.math_trainable_ruler_workflow.MathTrainableRULERWorkflow", + "rubric_judge_workflow": "trinity.common.workflows.rubric_judge_workflow.RubricJudgeWorkflow", + # others + "simple_mm_workflow": "trinity.common.workflows.simple_mm_workflow.SimpleMMWorkflow", + "async_simple_mm_workflow": "trinity.common.workflows.simple_mm_workflow.AsyncSimpleMMWorkflow", + # on-policy distillation workflows + "on_policy_distill_workflow": "trinity.common.workflows.on_policy_distill_workflow.OnPolicyDistillWorkflow", + "on_policy_distill_math_workflow": "trinity.common.workflows.on_policy_distill_workflow.OnPolicyDistillMathWorkflow", + "on_policy_distill_sciqa_workflow": "trinity.common.workflows.on_policy_distill_workflow.OnPolicyDistillSCIQAWorkflow", + # self-distillation workflows + "sdpo_workflow": "trinity.common.workflows.sdpo.SDPOWorkflow", + "sdpo_scienqa_workflow": "trinity.common.workflows.sdpo.SDPOSCIENQAWorkflow", + # custom workflows + "sudoku_workflow": "trinity.common.workflows.envs.sudoku.sudoku_workflow.SudokuWorkflow", + "OPD_alfworld_workflow": "trinity.common.workflows.envs.TCOD.alfworld.OPD_workflow.OnPolicyDistillVerlAgentAlfworldWorkflow", + "TCOD_f2b_alfworld_workflow": "trinity.common.workflows.envs.TCOD.alfworld.TCOD_f2b_workflow.TCOD_f2b_alfworld_workflow", + "TCOD_b2f_alfworld_workflow": "trinity.common.workflows.envs.TCOD.alfworld.TCOD_b2f_workflow.TCOD_b2f_alfworld_workflow", + "GuidedOPD_alfworld_workflow": "trinity.common.workflows.envs.TCOD.alfworld.GuidedOPD_workflow.GuidedOPD_alfworld_workflow", + "Eval_alfworld_workflow": "trinity.common.workflows.envs.TCOD.alfworld.Eval_workflow.Eval_alfworld_workflow", + "OPD_scienceworld_workflow": "trinity.common.workflows.envs.TCOD.scienceworld.OPD_workflow.OnPolicyDistillVerlAgentScienceworldWorkflow", + "TCOD_f2b_scienceworld_workflow": "trinity.common.workflows.envs.TCOD.scienceworld.TCOD_f2b_workflow.TCOD_f2b_scienceworld_workflow", + "TCOD_b2f_scienceworld_workflow": "trinity.common.workflows.envs.TCOD.scienceworld.TCOD_b2f_workflow.TCOD_b2f_scienceworld_workflow", + "OPD_webshop_workflow": "trinity.common.workflows.envs.TCOD.webshop.OPD_workflow.OnPolicyDistillVerlAgentWebshopWorkflow", + "TCOD_f2b_webshop_workflow": "trinity.common.workflows.envs.TCOD.webshop.TCOD_f2b_workflow.TCOD_f2b_webshop_workflow", + "TCOD_b2f_webshop_workflow": "trinity.common.workflows.envs.TCOD.webshop.TCOD_b2f_workflow.TCOD_b2f_webshop_workflow", + "OPD_textworld_workflow": "trinity.common.workflows.envs.TCOD.textworld.OPD_workflow.OnPolicyDistillVerlAgentTextworldWorkflow", + "TCOD_f2b_textworld_workflow": "trinity.common.workflows.envs.TCOD.textworld.TCOD_f2b_workflow.TCOD_f2b_textworld_workflow", + "TCOD_b2f_textworld_workflow": "trinity.common.workflows.envs.TCOD.textworld.TCOD_b2f_workflow.TCOD_b2f_textworld_workflow", + "OPD_taubench_workflow": "trinity.common.workflows.envs.TCOD.taubench.OPD_workflow.OnPolicyDistillVerlAgentTaubenchWorkflow", + "TCOD_f2b_taubench_workflow": "trinity.common.workflows.envs.TCOD.taubench.TCOD_f2b_workflow.TCOD_f2b_taubench_workflow", + "TCOD_mixed_taubench_workflow": "trinity.common.workflows.envs.TCOD.taubench.TCOD_mixed_workflow.TCOD_mixed_taubench_workflow", + }, +) + +__all__ = [ + "Task", + "Workflow", + "WORKFLOWS", +] diff --git a/frameworks/trinity/common/workflows/agentscope/__init__.py b/frameworks/trinity/common/workflows/agentscope/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..6ce3eac38da337494db0714cc389170babbdf7dd --- /dev/null +++ b/frameworks/trinity/common/workflows/agentscope/__init__.py @@ -0,0 +1 @@ +# This directory contains various agent workflow implementations that utilize the AgentScope framework. diff --git a/frameworks/trinity/common/workflows/agentscope/react/__init__.py b/frameworks/trinity/common/workflows/agentscope/react/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/frameworks/trinity/common/workflows/agentscope/react/react_agent.py b/frameworks/trinity/common/workflows/agentscope/react/react_agent.py new file mode 100644 index 0000000000000000000000000000000000000000..0cf8babe22b9c20e17e42d14003838ca7c4aeaed --- /dev/null +++ b/frameworks/trinity/common/workflows/agentscope/react/react_agent.py @@ -0,0 +1,65 @@ +from typing import Dict, Type + +import openai +from agentscope.agent import ReActAgent +from agentscope.formatter import OpenAIChatFormatter +from agentscope.message import Msg +from agentscope.model import OpenAIChatModel +from agentscope.tool import Toolkit +from pydantic import BaseModel + + +class AgentScopeReActAgent: + def __init__( + self, + openai_client: openai.AsyncOpenAI, + model_name: str, + system_prompt: str, + generate_kwargs: dict, + response_structure: Type[BaseModel], + max_iters: int = 10, + toolkit: Toolkit | None = None, + ): + """Initialize the AgentScope ReAct agent with specified tools and model. + + Args: + openai_client (openai.AsyncOpenAI): An instance of AsyncOpenAI client. + model_name (str): The name of the model to use. + system_prompt (str): The system prompt for the agent. + generate_kwargs (dict): Generation parameters for the model. + response_structure (Type[BaseModel]): A Pydantic model defining the expected response structure. + """ + # patch the OpenAIChatModel to use the openai_client provided by Trinity-RFT + self.agent_model = OpenAIChatModel( + api_key="EMPTY", + model_name=model_name, + generate_kwargs=generate_kwargs, + stream=False, + ) + self.agent_model.client = openai_client + self.agent = ReActAgent( + name="react_agent", + sys_prompt=system_prompt, + model=self.agent_model, + formatter=OpenAIChatFormatter(), + # we enable agentscope's meta tool to allow agent to call tools dynamically without pre-registration + enable_meta_tool=True, + toolkit=toolkit, + max_iters=max_iters, + ) + self.response_structure = response_structure + + async def reply(self, query: str) -> Dict: + """Generate a response from the agent given a query. + + Args: + query (str): The input query for the agent. + + Returns: + Dict: The structured response. + """ + + response = await self.agent.reply( + Msg("user", query, role="user"), structured_model=self.response_structure + ) + return response.metadata or {} diff --git a/frameworks/trinity/common/workflows/agentscope/react/react_workflow.py b/frameworks/trinity/common/workflows/agentscope/react/react_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..203af35623176818cd169585b6cc042b6b782abe --- /dev/null +++ b/frameworks/trinity/common/workflows/agentscope/react/react_workflow.py @@ -0,0 +1,96 @@ +"""An example workflow using AgentScope's ReAct agent to solve tasks. + +This workflow is a demonstration of how to integrate the AgentScope framework within the Trinity-RFT workflow system with minimal modifications. +""" + +from typing import Dict, List, Optional, Union + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.workflow import Task, Workflow + +from .templates import TEMPLATE_MAP + + +class AgentScopeReActWorkflow(Workflow): + is_async: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.model_client = model.get_openai_async_client() + + task_type = task.workflow_args.get("type", "gsm8k") + template = TEMPLATE_MAP.get(task_type, None) + if template is None: + raise ValueError( + f"Unsupported task type {task_type} for AgentScope ReAct Agent, please add a template first." + ) + # extract the query and the answer from the task + self.query = task.raw_task.get(task.format_args.prompt_key) # type: ignore [index] + self.answer = task.raw_task.get(task.format_args.response_key) # type: ignore [index] + self.reward_fn = template.reward_fn_cls(**task.reward_fn_args) + + # import here to avoid the import error if agentscope is not installed and this workflow is not used + try: + from trinity.common.workflows.agentscope.react.react_agent import ( + AgentScopeReActAgent, + ) + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + self.agent = AgentScopeReActAgent( + model_name=self.model_client.model_path, + openai_client=self.model_client, + system_prompt=template.system_prompt, + generate_kwargs={ + "temperature": self.task.rollout_args.temperature, + "max_tokens": self.task.rollout_args.max_tokens or 4096, + }, + response_structure=template.response_structure, + ) + + async def run_async(self): + """Run the workflow asynchronously.""" + # Step 1: call the react agent to solve the task + response = await self.agent.reply(self.query) + # Step 2: calculate the reward based on the response + reward = await self.calculate_reward(response) + # Step 3: construct experiences from the interaction history and return them + return self.construct_experiences(reward) + + async def calculate_reward(self, response) -> Union[float, Dict[str, float]]: + """Calculate the reward for the workflow. + + Returns: + Union[float, Dict[str, float]]: The reward value or a dictionary of reward value. + """ + return self.reward_fn(response=response, truth=self.answer) + + def construct_experiences(self, reward: Union[float, Dict[str, float]]) -> List[Experience]: + """Construct experiences from the agent's interaction history. + + Args: + reward (Union[float, Dict[str, float]]): The reward value to assign to each experience. + + Returns: + List: A list of Experience objects. + """ + exps = self.model.extract_experience_from_history() + for exp in exps: + exp.reward = reward if isinstance(reward, float) else sum(reward.values()) + exp.metrics = {"react_memory_length": len(self.agent.agent.memory.content)} + # record detailed reward if available + if isinstance(reward, dict): + exp.metrics.update(reward) + return exps diff --git a/frameworks/trinity/common/workflows/agentscope/react/templates.py b/frameworks/trinity/common/workflows/agentscope/react/templates.py new file mode 100644 index 0000000000000000000000000000000000000000..0b2ba2fd3f6f17356d7b264c681e53bf53c33a9d --- /dev/null +++ b/frameworks/trinity/common/workflows/agentscope/react/templates.py @@ -0,0 +1,60 @@ +from dataclasses import dataclass +from typing import Dict, Optional, Type + +from pydantic import BaseModel, Field + +from trinity.common.rewards import RewardFn +from trinity.common.rewards.math_reward import MathBoxedRewardFn + +# For GSM8K task +GSM8KSystemPrompt = """You are an agent specialized in solving math problems with tools. Please solve the math problem given to you. You can write and execute Python code to perform calculation or verify your answer. You should return your final answer within \\boxed{{}}.""" + + +class GSM8KResponseStructure(BaseModel): + result: str = Field( + description="Your solution of the given math problem. Put your final answer in boxed format, e.g., \\boxed{42}" + ) + + +class GSM8KRewardFn(MathBoxedRewardFn): + def __call__( # type: ignore [override] + self, + response: dict, + truth: str, + format_score_coef: float = 0.1, + **kwargs, + ) -> Dict[str, float]: + # parse GSM8K truth + if isinstance(truth, str) and "####" in truth: + truth = truth.split("####")[1].strip() + else: + truth = str(truth) + return super().__call__( + response=response.get("result", ""), + truth=truth, + with_think=False, + format_score_coef=format_score_coef, + **kwargs, + ) + + +# Registry for different templates + + +@dataclass +class Template: + """A template for different task types, including system prompt and response structure.""" + + system_prompt: str + response_structure: Type[BaseModel] + reward_fn_cls: Type[RewardFn] + + +TEMPLATE_MAP: Dict[str, Optional[Template]] = { + "gsm8k": Template( + system_prompt=GSM8KSystemPrompt, + response_structure=GSM8KResponseStructure, + reward_fn_cls=GSM8KRewardFn, + ), + # Add more templates for different task types as needed +} diff --git a/frameworks/trinity/common/workflows/agentscope_workflow.py b/frameworks/trinity/common/workflows/agentscope_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..feccfc59328198ad6af5d5de105cf33cc79d2c0b --- /dev/null +++ b/frameworks/trinity/common/workflows/agentscope_workflow.py @@ -0,0 +1,222 @@ +import inspect +from typing import Awaitable, Callable, Dict, List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.workflow import Task, Workflow +from trinity.utils.annotations import Deprecated + + +@Deprecated +class AgentScopeWorkflowAdapter(Workflow): + """Adapter to wrap a agentscope trainable workflow function into a Trinity Workflow. + Only for agentscope versions between 1.0.7 and 1.0.11. + For agentscope >= 1.0.12, please use AgentScopeWorkflowAdapterV1. + """ + + is_async: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + """Initialize the adapter with the task and model.""" + try: + from agentscope.model import TrinityChatModel + except ImportError: + raise ImportError( + "This workflow requires agentscope >= 1.0.7, please install " + "it via `pip install agentscope>=1.0.7`", + ) + + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.workflow_func: Callable[ + [Dict, TrinityChatModel], Awaitable[float] + ] = task.workflow_args.get("workflow_func", None) + + if self.workflow_func is None: + raise ValueError( + "The 'workflow_func' is not provided.", + ) + + self.chat_model: TrinityChatModel = TrinityChatModel( + model.get_openai_async_client(), + generate_kwargs={ + "temperature": self.task.rollout_args.temperature, + "top_p": self.task.rollout_args.top_p, + "max_tokens": self.task.rollout_args.max_tokens or 4096, + "logprobs": True, + "top_logprobs": self.task.rollout_args.logprobs, + }, + ) + + def construct_experiences( + self, + reward: float, + ) -> List[Experience]: + """Construct experiences from the agent's interaction history. + + Args: + reward (float): The reward value to assign to each experience. + + Returns: + List: A list of Experience objects. + """ + exps = self.model.extract_experience_from_history() + for exp in exps: + exp.reward = reward + return exps + + async def run_async(self) -> List[Experience]: + """Run the workflow asynchronously and return experiences.""" + reward = await self.workflow_func(self.task.raw_task, self.chat_model) # type: ignore [arg-type] + return self.construct_experiences(reward) + + +class AgentScopeWorkflowAdapterV1(Workflow): + """A more general adapter to wrap agentscope trainable workflow and judge functions into a Trinity Workflow. + Only for agentscope versions >= 1.0.12. + """ + + is_async: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + """Initialize the adapter with the task and model.""" + try: + from agentscope.model import OpenAIChatModel + except ImportError: + raise ImportError( + "This workflow requires agentscope >= 1.0.12, please install " + "it via `pip install agentscope>=1.0.12`", + ) + + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.workflow_func = task.workflow_args.get("workflow_func", None) + self.judge_func = task.workflow_args.get("judge_func", None) + self._openai_client = self.model.get_openai_async_client() + + if self.workflow_func is None: + raise ValueError( + "The 'workflow_func' is not provided.", + ) + + self.chat_model: OpenAIChatModel = OpenAIChatModel( + api_key="EMPTY", + model_name=self._openai_client.model_path, + stream=False, + generate_kwargs={ + "temperature": self.task.rollout_args.temperature, + "top_p": self.task.rollout_args.top_p, + "max_tokens": self.task.rollout_args.max_tokens or 4096, + "logprobs": True, + "top_logprobs": self.task.rollout_args.logprobs, + }, + ) + self.chat_model.client = self._openai_client + self.auxiliary_chat_models: Dict[str, OpenAIChatModel] = {} + if self.auxiliary_model_wrappers is not None: + for aux_model_wrapper in self.auxiliary_model_wrappers: + aux_model_client = aux_model_wrapper.get_openai_async_client() + aux_chat_model = OpenAIChatModel( + api_key="EMPTY", + model_name=aux_model_client.model_path, + generate_kwargs=aux_model_wrapper.generate_kwargs, + stream=False, + ) + aux_chat_model.client = aux_model_client + assert ( + aux_model_wrapper.model_name is not None + ), "Auxiliary model must have a name. This should not happen." + self.auxiliary_chat_models[aux_model_wrapper.model_name] = aux_chat_model + + def construct_experiences( + self, + reward: float, + metrics: Dict, + ) -> List[Experience]: + """Construct experiences from the agent's interaction history. + + Args: + reward (float): The reward value to assign to each experience. + metrics (Dict): A dictionary of metrics to be attached to the last experience. + + Returns: + List: A list of Experience objects. + """ + exps = self.model.extract_experience_from_history() + for exp in exps: + exp.reward = reward + # only attach metrics to the last experience + if len(exps) > 0: + exps[-1].metrics = metrics + return exps + + async def run_async(self) -> List[Experience]: + """Run the workflow asynchronously and return experiences.""" + try: + from agentscope.tuner import JudgeOutput, WorkflowOutput + except ImportError: + raise ImportError( + "Fail to import agentscope tuner related types. Please ensure agentscope>=1.0.12 is installed." + ) + + metrics = {} + workflow_sig = inspect.signature(self.workflow_func) + if "auxiliary_models" in workflow_sig.parameters: + workflow_output = await self.workflow_func( + task=self.task.raw_task, + model=self.chat_model, + auxiliary_models=self.auxiliary_chat_models, + ) + else: + workflow_output = await self.workflow_func( + task=self.task.raw_task, + model=self.chat_model, + ) + if not isinstance(workflow_output, WorkflowOutput): + raise ValueError( + "The 'workflow_func' must return a WorkflowOutput object.", + ) + metrics.update(workflow_output.metrics or {}) + if self.judge_func is not None: + judge_sig = inspect.signature(self.judge_func) + if "auxiliary_models" in judge_sig.parameters: + judge_output = await self.judge_func( + task=self.task.raw_task, + response=workflow_output.response, + auxiliary_models=self.auxiliary_chat_models, + ) + else: + judge_output = await self.judge_func( + task=self.task.raw_task, + response=workflow_output.response, + ) + if not isinstance(judge_output, JudgeOutput): + raise ValueError( + "The 'judge_func' must return a JudgeOutput object.", + ) + reward = judge_output.reward + metrics.update(judge_output.metrics or {}) + else: + assert ( + workflow_output.reward is not None + ), "Either workflow or judge must provide reward." + reward = workflow_output.reward + return self.construct_experiences(reward, metrics) diff --git a/frameworks/trinity/common/workflows/customized_math_workflows.py b/frameworks/trinity/common/workflows/customized_math_workflows.py new file mode 100644 index 0000000000000000000000000000000000000000..47b33ff3f90ab5216798de9d08b7d666be7658fc --- /dev/null +++ b/frameworks/trinity/common/workflows/customized_math_workflows.py @@ -0,0 +1,134 @@ +# -*- coding: utf-8 -*- +"""We include the customized math workflows in this file.""" + +from typing import List + +from trinity.common.experience import Experience +from trinity.common.rewards.math_reward import MathBoxedRewardFn +from trinity.common.workflows.workflow import SimpleWorkflow, Task + + +class MathBoxedWorkflow(SimpleWorkflow): + """A workflow for math tasks that give answers in boxed format.""" + + def reset(self, task: Task): + self.format_args = task.format_args + self.system_prompt = task.format_args.system_prompt + self.reply_prefix = task.format_args.reply_prefix + + self.raw_task = task.raw_task + self.task_desc = task.task_desc + self.truth = task.truth + + self.workflow_args = task.workflow_args + self.reward_fn_args = task.reward_fn_args + + self.use_base = self.workflow_args.get("use_base", False) + self.with_think = self.workflow_args.get("with_think", False) + self.format_score_coef = self.workflow_args.get("format_score_coef", 0.1) + + default_prompt = ( + """Please reason step by step, and put your final answer within \\boxed{}.""" + ) + + default_prompt_with_think = """You are a helpful assistant that solves MATH problems. You should first thinks about the reasoning process in mind and then provides the user with the answer. You should present your reasoning process using the format: \n ...your reasoning process here... \n first. You should always include your final answer in \\boxed{} as closed-form results.""" + + if self.system_prompt is None: + if self.with_think: + self.system_prompt = default_prompt_with_think + else: + self.system_prompt = default_prompt + + if task.reward_fn is None: + self.reward_fn = MathBoxedRewardFn(**self.reward_fn_args) + else: + self.reward_fn = task.reward_fn(**self.reward_fn_args) + + def format_prompt(self): + prompt_text = "" + if self.system_prompt: + prompt_text += "System:" + self.system_prompt + prompt_text += "\nUser:\n" + self.task_desc + "\nAssistant:\n" + else: + prompt_text += "User:\n" + self.task_desc + "\nAssistant:\n" + return prompt_text + + def run(self) -> List[Experience]: + if not self.use_base: + messages = self.format_messages() + else: + prompt_text = self.format_prompt() + + self.logger.debug("start chat") + if not self.use_base: + responses = self.model.chat(messages, **self.rollout_args) + else: + responses = self.model.generate([prompt_text], **self.rollout_args) + + for i, response in enumerate(responses): + reward_dict = self.reward_fn( # type: ignore [misc] + response=response.response_text, # type: ignore [arg-type] + truth=self.truth, + with_think=self.with_think, + format_score_coef=self.format_score_coef, + response_token=response.tokens[response.prompt_length :], + ) + + if response.metrics is None: + response.metrics = {} + response.metrics.update(reward_dict) + reward = sum(reward_dict.values()) + response.reward = reward + response.eid.run = i + self.run_id_base + + if not self.use_base: + self.logger.debug( + f"self.task_desc: {self.task_desc}, messages: {messages}, response: {response.response_text}, reward: {reward}" + ) + else: + self.logger.debug( + f"self.task_desc: {self.task_desc}, prompt_text: {prompt_text}, response: {response.response_text}, reward: {reward}" + ) + return responses + + +class AsyncMathBoxedWorkflow(MathBoxedWorkflow): + is_async: bool = True + + async def run_async(self) -> List[Experience]: + if not self.use_base: + messages = self.format_messages() + else: + prompt_text = self.format_prompt() + + self.logger.debug("start chat") + if not self.use_base: + responses = await self.model.chat_async(messages, **self.rollout_args) + else: + responses = await self.model.generate_async([prompt_text], **self.rollout_args) + + for i, response in enumerate(responses): + reward_dict = self.reward_fn( # type: ignore [misc] + response=response.response_text, # type: ignore [arg-type] + truth=self.truth, + with_think=self.with_think, + format_score_coef=self.format_score_coef, + response_token=response.tokens[response.prompt_length :], + ) + + if response.metrics is None: + response.metrics = {} + response.metrics.update(reward_dict) + reward = sum(reward_dict.values()) + response.reward = reward + response.eid.run = i + self.run_id_base + + if not self.use_base: + self.logger.debug( + f"self.task_desc: {self.task_desc}, messages: {messages}, response: {response.response_text}, reward: {reward}" + ) + else: + self.logger.debug( + f"self.task_desc: {self.task_desc}, prompt_text: {prompt_text}, response: {response.response_text}, reward: {reward}" + ) + return responses diff --git a/frameworks/trinity/common/workflows/customized_toolcall_workflows.py b/frameworks/trinity/common/workflows/customized_toolcall_workflows.py new file mode 100644 index 0000000000000000000000000000000000000000..938c027b03080ce151664b46534957c7497504b9 --- /dev/null +++ b/frameworks/trinity/common/workflows/customized_toolcall_workflows.py @@ -0,0 +1,264 @@ +# -*- coding: utf-8 -*- +""" +We include the customized toolcall workflows in this file. +Code adapted from https://github.com/NVlabs/Tool-N1 +Reference Paper https://arxiv.org/pdf/2505.00024 for further details. +""" + +import json +import re +from collections import Counter +from typing import List + +from trinity.common.experience import Experience +from trinity.common.workflows.workflow import SimpleWorkflow, Task + +# Adapted from https://github.com/NVlabs/Tool-N1 +qwen_tool_prompts = """# Tool + + +{tools} + + +In each action step, you MUST: +1. Think about the reasoning process in the mind and enclosed your reasoning within XML tags. +2. Then, provide a json object with function names and arguments within XML tags. i.e., [{{"name": , "arguments": }}, {{"name": , "arguments": }}, ...] +3. Make sure both the reasoning and the tool call steps are included together in one single reply. +A complete reply example is: To address the query, I need to send the email to Bob and then buy the banana through walmart. [{{"name": "email", "arguments": {{"receiver": "Bob", "content": "I will bug banana through walmart"}}}}, {{"name": "walmart", "arguments": {{"input": "banana"}}}}]. Please make sure the type of the arguments is correct. +""" + + +# Adapted from https://github.com/NVlabs/Tool-N1 +def construct_prompt(dp): + def format_tools(tools): + tools = json.loads(tools) + string = "" + for tool in tools: + string += json.dumps({"type": "function", "function": tool}) + "\n" + if string[-1] == "\n": + string = string[:-1] + return string + + tools = format_tools(dp["tools"]) + tool_prompt = qwen_tool_prompts.format(tools=tools) + system = dp["raw_system"] + conversations = dp["conversations"] + prompt = [] + prompt.append({"role": "system", "content": system + tool_prompt}) + for tem in conversations: + if tem["from"] == "human" or tem["from"] == "user": + prompt.append({"role": "user", "content": tem["value"]}) + elif tem["from"] == "gpt" or tem["from"] == "assistant": + prompt.append({"role": "assistant", "content": tem["value"]}) + elif tem["from"] == "observation" or tem["from"] == "tool": + prompt.append({"role": "tool", "content": tem["value"]}) + elif tem["from"] == "function_call": + prompt.append({"role": "assistant", "content": json.dumps(tem["value"])}) + return prompt + + +# Adapted from https://github.com/NVlabs/Tool-N1 +def validate_result(result, answer): + if len(result) == 0 or len(answer) == 0: + if len(result) == len(answer): + return 2 + else: + return 0 + else: + try: + counter1_full = Counter( + (item["name"], json.dumps(item["arguments"], sort_keys=True)) for item in result + ) + counter2_full = Counter( + (item["name"], json.dumps(item["arguments"], sort_keys=True)) for item in answer + ) + except TypeError: + return 0 + if counter1_full == counter2_full: + return 2 + + counter1_name = Counter(item["name"] for item in result) + counter2_name = Counter(item["name"] for item in answer) + + if counter1_name == counter2_name: + return 1 + + return 0 + + +# Adapted from https://github.com/NVlabs/Tool-N1 +def validate_format(tool_call_list): + for item in tool_call_list: + if not isinstance(item, dict): + return 0 + for item in tool_call_list: + if "name" not in item.keys() or "arguments" not in item.keys(): + return 0 + return 1 + + +# Adapted from https://github.com/NVlabs/Tool-N1 +def extract_solution_v0_toolN1(tool_call_str): + output_string = tool_call_str + + pattern = r"(.*?)" + matches = list(re.finditer(pattern, tool_call_str, flags=re.DOTALL)) + if not matches: + return None, output_string + last_content = matches[-1].group(1).strip() + try: + return json.loads(last_content), output_string + except json.JSONDecodeError: + return None, output_string + + +# Adapted from https://github.com/NVlabs/Tool-N1 +def compute_score_v0_toolN1( # noqa: C901 + solution_str, + ground_truth, + do_print=False, +): + answer = json.loads(ground_truth) + + result, output_string = extract_solution_v0_toolN1(solution_str) + + if isinstance(result, str): + try: + result = json.loads(result) + except json.JSONDecodeError: + result = None + + if isinstance(result, dict): + tem = [] + tem.append(result) + result = tem + + if isinstance(answer, str): + answer = json.loads(answer) + + if do_print: + print("************solution_str************") + print(solution_str) + print(f"Extracted result: {result}") + print(f"Solution string: {answer}") + + if result is not None: + if "" not in output_string or "" not in output_string: + if do_print: + print("--------" * 5 + "\n\n") + print("not thinking:", -1) + return 0 + + if result is None: + if do_print: + print("--------" * 5 + "\n\n") + print("result is None:", -1) + return 0 + + # added rule1 + if solution_str.count("") != 1 or solution_str.count("") != 1: + if do_print: + print("--------" * 5 + "\n\n") + print( + f"Fail, think tag appear not once: " + f" appear {solution_str.count('')} times, " + f" appear {solution_str.count('')} times", + -1, + ) + return 0 + + # added rule2 + think_end_pos = solution_str.find("") + tool_call_start_pos = solution_str.find("") + + if tool_call_start_pos != -1: + if think_end_pos > tool_call_start_pos: + if do_print: + print("--------" * 5 + "\n\n") + print("Fail: tag must before tag", -1) + return 0 + + if not validate_format(result): + if do_print: + print("--------" * 5 + "\n\n") + print("result wrong formate:", -1) + return 0 + + if validate_result(result, answer) == 2: + if do_print: + print("--------" * 5 + "\n\n") + print("get full core:", 1) + return 1 + else: + if do_print: + print("--------" * 5 + "\n\n") + print("wrong answer", -1) + return 0 + + +def compute_toolcall_reward( + solution_str: str, + ground_truth: str, +) -> float: + res = compute_score_v0_toolN1(solution_str, ground_truth) + if isinstance(res, (int, float, bool)): + return float(res) + else: + return float(res[0]) + + +class ToolCallWorkflow(SimpleWorkflow): + """ + A workflow for toolcall tasks. + Prompt construction and reward function from https://github.com/NVlabs/Tool-N1 + + Only support qwen model for now. You can change the prompt construction and reward calculation by yourself for other models. + """ + + is_async: bool = True + + def reset(self, task: Task): + self.format_args = task.format_args + self.system_prompt = task.format_args.system_prompt + self.reply_prefix = task.format_args.reply_prefix + + self.raw_task = task.raw_task + self.task_desc = task.task_desc + self.truth = task.truth + + self.workflow_args = task.workflow_args + self.reward_fn_args = task.reward_fn_args + + def format_prompt(self): + raw_task = self.raw_task + messages = construct_prompt(raw_task) + return messages + + async def run_async(self) -> List[Experience]: + messages = self.format_prompt() + + self.logger.debug("start chat") + responses = await self.model.chat_async(messages, **self.rollout_args) + + for i, response in enumerate(responses): + reward = 0.0 + + if self.raw_task is not None: + ground_truth = self.raw_task.get("answer") + if ground_truth is not None: + reward = compute_toolcall_reward( + solution_str=response.response_text, + ground_truth=ground_truth, + ) + else: + self.logger.error( + "Key 'answer' not found in self.raw_task. Assigning default reward." + ) + else: + self.logger.error("self.raw_task is None. Assigning default reward.") + self.logger.debug( + f"self.task_desc: {self.task_desc}, messages: {messages}, response: {response.response_text}, reward: {reward}" + ) + response.reward = reward + response.eid.run = i + self.run_id_base + return responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/__init__.py b/frameworks/trinity/common/workflows/envs/TCOD/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/frameworks/trinity/common/workflows/envs/TCOD/alfworld/Eval_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/Eval_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..c701366b78bb2cbb0bd601ff27e2b8730446e72c --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/Eval_workflow.py @@ -0,0 +1,71 @@ +# -*- coding: utf-8 -*- +"""Teacher-free ALFWorld evaluation workflow. + +Evaluation must measure the STUDENT ALONE -- no teacher scoring, no teacher actions -- +but it must walk the environment through exactly the same code the training-time eval +walks, or the numbers are not comparable across the two. + +So this class does not reimplement the episode loop: it subclasses the OPD workflow and +inherits `_run_episode` verbatim. The only thing it replaces is the teacher, with a stub +whose `logprobs_async` returns zeros. That keeps the assertion and the bookkeeping in the +inherited loop satisfied while loading no second model and issuing no second forward +pass. The resulting `teacher_logprobs` are meaningless -- and unused, because bench mode +never computes an advantage. + +The consequence worth stating: `_run_episode` is the single source of truth for how an +ALFWorld episode is rolled out. Prompt template, HISTORY_LENGTH, action parsing, step +budget and the done/reward convention cannot drift between training-time eval and +standalone bench, because there is only one copy of them. + +Use with `mode: bench`, one GPU, tensor_parallel_size 1 -- bench mode starts no trainer +and no synchronizer, so the TP>=2 requirement of the training loop does not apply here. +""" + +from typing import List, Optional + +import torch + +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task + +from trinity.common.workflows.envs.TCOD.alfworld.OPD_workflow import ( + OnPolicyDistillVerlAgentAlfworldWorkflow, +) + + +class _StubTeacher: + """Stands in for the teacher so the inherited loop runs unmodified. + + Returns one entry fewer than the token count, matching what a real + `logprobs_async` yields (next-token predictions), so the length assertion in + `_run_episode` still checks something real about the student's own tensors. + """ + + async def logprobs_async(self, tokens, **kwargs): + return torch.zeros(max(len(tokens) - 1, 0), dtype=torch.float32) + + +@WORKFLOWS.register_module("Eval_alfworld_workflow") +class Eval_alfworld_workflow(OnPolicyDistillVerlAgentAlfworldWorkflow): + """ALFWorld evaluation: student only, no teacher loaded.""" + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + # Deliberately bypass the parent's constructor: it asserts that an auxiliary + # teacher model is configured, which is exactly the cost this class exists to + # avoid. Everything the parent sets up beyond that is reproduced here. + super(OnPolicyDistillVerlAgentAlfworldWorkflow, self).__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + self.teacher_model = _StubTeacher() + self.temperature = task.workflow_args.get("temperature", 0.4) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.is_eval = task.is_eval diff --git a/frameworks/trinity/common/workflows/envs/TCOD/alfworld/GuidedOPD_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/GuidedOPD_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..3b4e6cfc5f7c52fab705f10f8e4d1b29cdb03bf2 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/GuidedOPD_workflow.py @@ -0,0 +1,365 @@ +# -*- coding: utf-8 -*- +"""Guided-OPD workflow for AlfWorld. + +Reference: "On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn +Agents" (arXiv:2606.15912). Eq. (3) for the role draw, Eqs. (6)-(8) for the schedules. + +Difference from OPD_workflow, in one sentence: the role of every turn is drawn from +Bernoulli(beta) and the teacher GENERATES the turns it wins, instead of only scoring +turns the student generated. Everything else -- prompt template, history length, action +parsing, environment construction -- is imported unchanged from utils.py, so a beta of +zero reproduces OPD_alfworld_workflow turn for turn. + +The asymmetric loss that goes with this rollout lives in the advantage function +(`advantage_fn: guided_opd`); this file only decides who speaks and records it. Pairing +it with `multi_turn_opd` instead would silently train reverse KL on teacher-sampled +tokens, which is Table 2(a)'s worst-scoring variant, so the two must be set together. + +REQUIRES `old_log_probs` to be recomputed by the actor, i.e. NO `rollout_corr` config +that sets `bypass_recomputing_logprobs`. On a teacher turn the rollout logprobs belong +to the TEACHER, so bypassing the recompute would feed the teacher's own logprobs in as +the student's and both the reverse-KL advantage and the forward-KL diagnostic would be +wrong. The flag is off unless `rollout_corr` is configured, which these configs do not. + +Teacher and student must share a tokenizer (Qwen3 family, vocab 151936): the trainer +runs the student over the teacher's token ids verbatim. +""" + +import math +import random +import re +import time +from dataclasses import asdict +from typing import List, Optional + +import torch + +from trinity.common.experience import CustomField, Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow + +from trinity.common.workflows.envs.TCOD.alfworld.utils import ( + ALFWORLD_TEMPLATE_NO_HIS, + ALFWORLD_TEMPLATE, + HISTORY_LENGTH, + parse_action, + format_observation, + _extract_task, + _format_history, + _create_alfworld_env, +) + +# Must be declared identically on EVERY experience in the batch: the trainer raises +# "Custom fields are not consistent across experiences" otherwise. +GUIDED_ROLE_CUSTOM_FIELD = CustomField( + source_field="is_teacher_turn", + destination_field="is_teacher_turn", + data_type=torch.float32, +) + + +@WORKFLOWS.register_module("GuidedOPD_alfworld_workflow") +class GuidedOPD_alfworld_workflow(Workflow): + """Guided-OPD workflow for AlfWorld task.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__(task=task, model=model, auxiliary_models=auxiliary_models) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "Guided-OPD requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + # Read workflow_args HERE, not in __init__: with can_reset=True trinity builds + # the workflow once and reuses it across tasks via reset(). Runners serve eval + # and train tasksets alike, so if the first task happens to be an eval one -- + # whose workflow_args carry only max_env_steps -- everything read at __init__ + # is locked to the eval defaults for the rest of the run. Job 20561187 spent + # all six steps at beta~=0.999 exactly this way: total_steps fell back to 250. + wa = task.workflow_args + self.temperature = wa.get("temperature", 1.0) + self.max_env_steps = wa.get("max_env_steps", 30) + # total_steps MUST equal trainer.total_steps or the curriculum finishes at the + # wrong time -- too early and the tail is longer than the paper's 20%, too late + # and beta never reaches zero, so training never converges to pure OPD. + self.total_steps = wa.get("total_steps", 250) + self.beta_start = wa.get("beta_start", 1.0) + self.beta_end = wa.get("beta_end", 0.0) + self.curriculum_ratio = wa.get("curriculum_ratio", 0.8) + self.beta_schedule = wa.get("beta_schedule", "cosine") + # Set False for the black-box variant (advantage_fn kl_coef=0): student turns then + # carry no distillation signal, so scoring them is a teacher forward pass over the + # full context bought for nothing. Skipping it is what makes that variant cheaper + # than vanilla OPD rather than merely equal to it -- the teacher is only ever + # invoked on the turns it generates, and never needs logits at all. + self.score_student_turns = wa.get("score_student_turns", True) + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + def _current_step(self) -> int: + """Read the explorer step out of the task's batch id (same trick as TCOD_f2b).""" + batch_id = getattr(self.task, "batch_id", None) + if isinstance(batch_id, int): + return batch_id + if isinstance(batch_id, str): + match = re.match(r"^(\d+)", batch_id) + if match: + return int(match.group(1)) + return 0 + + def compute_beta(self, step: int) -> float: + """Teacher intervention probability at this training step (Eqs. 6-8).""" + # Evaluation is always intervention-free. A teacher turn at eval would measure + # the pair, not the student, and the student alone is what deploys. + if self.is_eval: + return 0.0 + t_decay = max(1.0, self.curriculum_ratio * self.total_steps) + xi = min(step / t_decay, 1.0) + b0, b1 = float(self.beta_start), float(self.beta_end) + + if self.beta_schedule == "cosine": + return b0 + (b1 - b0) * (1.0 - math.cos(math.pi * xi)) / 2.0 + if self.beta_schedule == "linear": + return b0 + (b1 - b0) * xi + if self.beta_schedule == "exp": + # Eq. (8) is stated only for beta_start, beta_end > 0. With the paper's own + # default beta_end = 0 the ratio is degenerate, so fail loudly rather than + # quietly substituting a different curve. + if b0 <= 0.0 or b1 <= 0.0: + raise ValueError( + "beta_schedule 'exp' requires beta_start > 0 and beta_end > 0 (Eq. 8); " + f"got beta_start={b0}, beta_end={b1}. Use 'cosine' or 'linear' to decay to 0." + ) + return b0 * (b1 / b0) ** xi + raise ValueError( + f"unknown beta_schedule '{self.beta_schedule}', expected cosine|linear|exp" + ) + + async def run_async(self) -> List[Experience]: + step = self._current_step() + # Beta is drawn once and held fixed for the whole trajectory (paper §3.1), so a + # trajectory is a clean sample from the mixed rollout distribution pi^{beta}. + self._beta = self.compute_beta(step) + # Seed from the task identity rather than the clock so a rerun of the same step + # replays the same role assignment; otherwise a resumed run silently changes the + # data it trains on. + eid = getattr(self.task, "batch_id", 0) + self._rng = random.Random(f"{eid}/{self.task_desc}/{getattr(self, 'run_id_base', 0)}") + + env = _create_alfworld_env(self.task_desc) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info = env.reset() + self._env_done = False + self._env_rounds = 0 + + task_description = _extract_task(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + turn_is_teacher: List[bool] = [] + gen_stats = { + role: {"turns": 0, "seconds": 0.0, "out_tokens": 0, "prompt_tokens": 0} + for role in ("student", "teacher") + } + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + # Eq. (3): the role of this turn. Both roles write into the SAME history, so + # the next turn's prompt is structurally identical either way -- the student + # is never told which turns were not its own. + teacher_turn = self._rng.random() < self._beta + actor = self.teacher_model if teacher_turn else self.model + + # Wall-clock around the generation call, split by role. This is the number the + # cost argument actually turns on: whether a teacher turn is meaningfully more + # expensive than a student turn at the concurrency we run at. It is NOT pure + # decode latency -- it includes the Ray round trip and any queueing behind the + # other `runner_per_model` episodes hitting the same engine. That is the right + # quantity for "does the teacher slow training down", and the wrong one for + # "how fast is the model"; do not quote it as the latter. + _t0 = time.perf_counter() + responses = await actor.chat_async(memory, **kwargs) + _dt = time.perf_counter() - _t0 + response = responses[0] + _new_tokens = max(len(response.tokens) - response.prompt_length, 0) + _bucket = gen_stats["teacher" if teacher_turn else "student"] + _bucket["turns"] += 1 + _bucket["seconds"] += _dt + _bucket["out_tokens"] += _new_tokens + _bucket["prompt_tokens"] += response.prompt_length + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "GuidedOPD_alfworld_workflow requires both models to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in the task config." + ) + turn_responses.append(response) + turn_is_teacher.append(teacher_turn) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = 1.0 + break + else: + self._env_rounds = self.max_env_steps + self._final_reward = 0.0 + + per_turn_kl_sums: List[float] = [] + for i, (response, is_teacher) in enumerate(zip(turn_responses, turn_is_teacher)): + scored = True + if is_teacher: + # The teacher produced these tokens, so its logprobs on them already came + # back with the generation. Scoring them again would be a second forward + # pass over the same sequence for the same numbers. + teacher_resp_logprobs = response.logprobs + elif not self.score_student_turns: + # Black-box mode: no teacher logits anywhere. The zeros are placeholders -- + # the validity mask below is what actually keeps them out of the gradient, + # so this stays correct even if someone leaves kl_coef non-zero by mistake. + teacher_resp_logprobs = torch.zeros_like(response.logprobs) + scored = False + else: + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + + assert len(teacher_resp_logprobs) == len(response.logprobs), ( + f"Length mismatch on a student turn: " + f"teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(response.logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + # Always emitted, so the trainer's all-or-nothing gather never drops the field. + # All-True in the default path, so it is a no-op there. + response.teacher_logprobs_valid_mask = torch.full_like( + teacher_resp_logprobs, float(scored), dtype=torch.bool + ) + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + # The role flag the advantage function keys on. Declared on every experience + # -- the trainer drops custom fields that are not identical across the batch. + response.info["is_teacher_turn"] = 1.0 if is_teacher else 0.0 + response.custom_fields = [GUIDED_ROLE_CUSTOM_FIELD] + + # Only student turns carry a reverse KL; a teacher turn's logprob gap is a + # forward KL and averaging the two together would report neither. + if not is_teacher: + per_turn_kl_sums.append( + (response.logprobs - teacher_resp_logprobs).sum().item() + ) + + if turn_responses: + n_teacher = sum(turn_is_teacher) + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = sum(per_turn_kl_sums) + last_response.metrics["beta_scheduled"] = self._beta + last_response.metrics["teacher_turns"] = float(n_teacher) + last_response.metrics["student_turns"] = float(len(turn_responses) - n_teacher) + + # Emit a role's timings only when that role actually spoke this episode -- + # a zero would otherwise be averaged in as if it were a measurement, and at + # the ends of the curriculum one of the two roles never speaks at all. + for role, st in gen_stats.items(): + if st["turns"] == 0: + continue + last_response.metrics[f"gen/{role}_ms_per_turn"] = ( + 1000.0 * st["seconds"] / st["turns"] + ) + last_response.metrics[f"gen/{role}_out_tokens_per_turn"] = ( + st["out_tokens"] / st["turns"] + ) + last_response.metrics[f"gen/{role}_prompt_tokens_per_turn"] = ( + st["prompt_tokens"] / st["turns"] + ) + if st["out_tokens"] > 0: + last_response.metrics[f"gen/{role}_ms_per_out_token"] = ( + 1000.0 * st["seconds"] / st["out_tokens"] + ) + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/alfworld/OPD_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/OPD_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..acb40f1ef66562a03beaef4f49b602b6559de54b --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/OPD_workflow.py @@ -0,0 +1,232 @@ +# -*- coding: utf-8 -*- +"""On-Policy Distillation (OPD) workflow for AlfWorld. + +Reference: OnPolicyDistillWorkflow logic; adapted for multi-turn AlfWorld. + +Algorithm: +1. Student pre-samples trajectory (runs episode turn-by-turn with logprobs) +2. Split by turns: each turn has fixed prefix [system, obs_1, resp_1, ..., obs_t] +3. Teacher computes logprobs on same (prefix + response) per turn +4. Store teacher_logprobs in experience; advantage_fn uses teacher_logprobs - student_logprobs +5. Return one Experience per turn (like OPD returning one per sample) +""" + +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow + +from trinity.common.workflows.envs.TCOD.alfworld.utils import ( + ALFWORLD_TEMPLATE_NO_HIS, + ALFWORLD_TEMPLATE, + HISTORY_LENGTH, + parse_action, + format_observation, + _extract_task, + _format_history, + _create_alfworld_env, +) + +@WORKFLOWS.register_module("OPD_alfworld_workflow") +class OnPolicyDistillVerlAgentAlfworldWorkflow(Workflow): + """On-policy distillation workflow for AlfWorld. + + Computes and stores teacher_logprobs in each turn's experience. + The advantage_fn in trainer will compute: + advantages = teacher_logprobs - student_logprobs + + Use advantage_fn: multi_turn_opd (MultiTurnOpdAdvantage) for this workflow, + since it returns List[Experience] (one per turn), not a single response. + + Logic aligned with OnPolicyDistillWorkflow: + - Student samples (with logprobs); teacher computes logprobs on same sequences. + - Per-turn split: prefix fixed per turn, one Experience per turn. + - compute_reward() can be overridden by subclasses (default: episode final reward). + """ + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.is_eval = task.is_eval + + def reset(self, task: Task): + """Reset the workflow with a new task. + + Unlike BaseSimpleWorkflow, this does NOT require reward_fn. + """ + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def compute_reward(self, response: Experience) -> float: + """Return episode-level reward (same for all turns in the trajectory). + + Set in _run_episode: env reward when done, 0.0 when max steps exhausted. + """ + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + """Format initial messages for the episode. + + Uses ALFWORLD_TEMPLATE_NO_HIS / ALFWORLD_TEMPLATE from utils.py. + No system prompt; each user message is self-contained. + """ + return [] + + async def run_async(self) -> List[Experience]: + game_file_path = self.task_desc + env = _create_alfworld_env(game_file_path) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info = env.reset() + self._env_done = False + self._env_rounds = 0 + + task_description = _extract_task(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join( + history[-HISTORY_LENGTH:] + if len(history) >= HISTORY_LENGTH + else history + ) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + # Step 1: Student samples this turn (same pattern as OnPolicyDistillWorkflow) + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillAlfworldWorkflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = 1.0 + break + else: + self._env_rounds = self.max_env_steps + self._final_reward = 0.0 # failure: exhausted max steps + + # Step 2 & 3: Teacher logprobs and fill experience (mirror OnPolicyDistillWorkflow.run_async) + # response.tokens is the full sequence for this turn: [prefix | response], where + # prefix = system + obs_1 + resp_1 + ... + obs_t (same input as student had). + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), # full input = prefix + student's response + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + # Trajectory-level metrics (computed once for the whole trajectory) + trajectory_kl_divergence = sum(per_turn_kl_sums) + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/alfworld/TCOD_b2f_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/TCOD_b2f_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..08d532d113aa2fb91cb4c138de15dd4478003677 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/TCOD_b2f_workflow.py @@ -0,0 +1,402 @@ +# -*- coding: utf-8 -*- +"""TCOD Backward-to-Forward workflow for AlfWorld.""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.alfworld.utils import ( + ALFWORLD_TEMPLATE_NO_HIS, + ALFWORLD_TEMPLATE, + HISTORY_LENGTH, + parse_action, + format_observation, + _extract_task, + _format_history, + _create_alfworld_env, + _create_alfworld_env_with_checkpoint, +) + + +@WORKFLOWS.register_module("TCOD_b2f_alfworld_workflow") +class TCOD_b2f_alfworld_workflow(Workflow): + """TCOD Backward-to-Forward workflow for AlfWorld task.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.is_eval = task.is_eval + + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = 0 + + def reset(self, task: Task): + """Reset the workflow with a new task.""" + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + """Set current training progress for dynamic checkpoint strategy. + + Args: + current_step: Current training step (e.g., current epoch or iteration) + total_steps: Total training steps (e.g., total epochs) + """ + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _linear_checkpoint_step(self, predefined_actions: Optional[List[str]] = None) -> Optional[int]: + """Compute checkpoint step using linear decay strategy.""" + if predefined_actions is None or len(predefined_actions) == 0: + return 0 + current_step = self._current_training_step + max_expert_actions = len(predefined_actions) - 1 + reduction = current_step // self.checkpoint_steps + checkpoint_step = max(0, min(max_expert_actions - reduction, max_expert_actions)) + return checkpoint_step + + def compute_reward(self, response: Experience) -> float: + """Return episode-level reward.""" + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + """Format initial messages for the episode.""" + return [] + + async def run_async(self) -> List[Experience]: + # If this is an eval task, always use normal _run_episode (no checkpoint) + if self.is_eval: + env = _create_alfworld_env(self.task_desc) + try: + return await self._run_episode(env) + finally: + env.close() + + # Extract current training step from task batch_id + current_step = 0 + if hasattr(self.task, 'batch_id'): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + # Extract the first integer from the string (e.g., "5/eval_task" -> 5) + match = re.match(r'^(\d+)', batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + game_file_path = self.task_desc + predefined_actions = self.raw_task.get("actions", None) + + if self.checkpoint_strategy == "linear": + effective_checkpoint_step = self._linear_checkpoint_step(predefined_actions) + + if effective_checkpoint_step is not None and effective_checkpoint_step > 0 and predefined_actions is not None: + ( + env, + observation, + info, + history, + task_description, + current_step, + replay_done, + ) = ( + _create_alfworld_env_with_checkpoint( + game_file_path, predefined_actions, effective_checkpoint_step + ) + ) + try: + if replay_done: + self._env_done = True + self._env_rounds = current_step + self._final_reward = 1.0 + return [] + return await self._run_episode_from_checkpoint( + env, observation, info, history, task_description, current_step + ) + finally: + env.close() + else: + env = _create_alfworld_env(game_file_path) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info = env.reset() + self._env_done = False + self._env_rounds = 0 + + task_description = _extract_task(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join( + history[-HISTORY_LENGTH:] + if len(history) >= HISTORY_LENGTH + else history + ) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillAlfworldWorkflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = 1.0 + break + else: + self._env_rounds = self.max_env_steps + self._final_reward = 0.0 + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + + if self.is_eval: + total_expert_actions = 0 + else: + total_expert_actions = len(self.raw_task.get("actions", [])) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["student_env_rounds"] = self._env_rounds + last_response.metrics["teacher_env_rounds"] = 0 + last_response.metrics["expected_teacher_env_rounds"] = total_expert_actions + last_response.metrics["if_teacher"] = 0 + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses + + async def _run_episode_from_checkpoint( + self, env, observation, info, history: List[str], task_description: str, start_step: int + ) -> List[Experience]: + """Run episode from a checkpoint with predefined history.""" + self._env_done = False + self._env_rounds = start_step + + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(start_step, self.max_env_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join( + history[-HISTORY_LENGTH:] + if len(history) >= HISTORY_LENGTH + else history + ) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillAlfworldWorkflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = 1.0 + break + else: + self._env_rounds = self.max_env_steps + self._final_reward = 0.0 + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = start_step + i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + total_expert_actions = len(self.raw_task.get("actions", [])) + expert_actions_remaining = total_expert_actions - start_step + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["student_env_rounds"] = self._env_rounds - start_step + last_response.metrics["teacher_env_rounds"] = start_step + last_response.metrics["if_teacher"] = 1 if start_step > 0 else 0 + last_response.metrics["expected_teacher_env_rounds"] = expert_actions_remaining + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/alfworld/TCOD_f2b_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/TCOD_f2b_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..ca6196e277e840f4c8a1c136932fad37957a3cc8 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/TCOD_f2b_workflow.py @@ -0,0 +1,251 @@ +# -*- coding: utf-8 -*- +"""TCOD Forward-to-Backward workflow for AlfWorld.""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.alfworld.utils import ( + ALFWORLD_TEMPLATE_NO_HIS, + ALFWORLD_TEMPLATE, + HISTORY_LENGTH, + parse_action, + format_observation, + _extract_task, + _format_history, + _create_alfworld_env, +) + + +@WORKFLOWS.register_module("TCOD_f2b_alfworld_workflow") +class TCOD_f2b_alfworld_workflow(Workflow): + """TCOD Forward-to-Backward workflow for AlfWorld task.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + self.is_eval = task.is_eval + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = 0 + + def reset(self, task: Task): + """Reset the workflow with a new task.""" + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + """Set current training progress for dynamic distillation window.""" + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _compute_distill_window(self) -> int: + """Compute how many previous steps to distill based on training progress.""" + distill_window = 1 + (self._current_training_step // self.checkpoint_steps) + return distill_window + + def compute_reward(self, response: Experience) -> float: + """Return episode-level reward.""" + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + """Format initial messages for the episode.""" + return [] + + async def run_async(self) -> List[Experience]: + current_step = 0 + if hasattr(self.task, 'batch_id'): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + match = re.match(r'^(\d+)', batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + game_file_path = self.task_desc + env = _create_alfworld_env(game_file_path) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info = env.reset() + self._env_done = False + self._env_rounds = 0 + + task_description = _extract_task(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + distill_window = self._compute_distill_window() + + if self.is_eval: + effective_steps = self.max_env_steps + else: + effective_steps = min(distill_window, self.max_env_steps) + + for r in range(effective_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join( + history[-HISTORY_LENGTH:] + if len(history) >= HISTORY_LENGTH + else history + ) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_f2b_alfworld_workflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = 1.0 + break + else: + self._env_rounds = effective_steps + self._final_reward = 0.0 + + per_turn_kl_sums: List[float] = [] + + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + trajectory_kl_divergence = sum(per_turn_kl_sums) + + if self.is_eval: + total_expert_actions = 0 + else: + total_expert_actions = len(self.raw_task.get("actions", [])) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + expected_teacher_env_rounds = max( + total_expert_actions - self._env_rounds, + 0, + ) + last_response.metrics["student_env_rounds"] = self._env_rounds + last_response.metrics["teacher_env_rounds"] = 0 + last_response.metrics["expected_teacher_env_rounds"] = ( + expected_teacher_env_rounds + ) + last_response.metrics["if_teacher"] = ( + 0 + if self.is_eval + else 1 if expected_teacher_env_rounds > 0 and not self._env_done else 0 + ) + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/alfworld/utils.py b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..4d278a65f5ea6891da5df6b4300213f44240016b --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/alfworld/utils.py @@ -0,0 +1,121 @@ +from typing import List + +# --------------------- ALFWorld --------------------- # +ALFWORLD_TEMPLATE_NO_HIS = """ +You are an expert agent operating in the ALFRED Embodied Environment. +Your current observation is: {current_observation} +Your admissible actions of the current situation are: [{admissible_actions}]. + +Now it's your turn to take an action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" + +ALFWORLD_TEMPLATE = """ +You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description} +Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history} +You are now at step {current_step} and your current observation is: {current_observation} +Your admissible actions of the current situation are: [{admissible_actions}]. + +Now it's your turn to take an action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" + +def parse_action(response): + try: + # parse the action within the tag + action = response.split("")[1].split("")[0].strip() + return action + except Exception as e: + print(f"Error parsing action: {e}, response = {response}") + return "" + +def format_observation(observation: str): + return observation.strip() + + +HISTORY_LENGTH = 2 +MEMORY_FORMAT = "[Observation {step_num}: '{obs}', Action {step_num}: '{act}']" + + +def _extract_task(text_obs: str) -> str: + """Extract the task description from the text observation.""" + task_start = text_obs.find("Your task is to: ") + if task_start != -1: + return text_obs[task_start + len("Your task is to: ") :].strip() + raise ValueError("Task description not found in text observation.") + + +def _format_history(observation: str, step_num: int, act: str) -> str: + """Format observation and action for action_history.""" + return MEMORY_FORMAT.format(step_num=step_num, obs=observation.strip(), act=act) + + +def _create_alfworld_env(game_file_path: str): + """Create AlfWorld textworld environment.""" + try: + import textworld + import textworld.gym + from alfworld.agents.environment.alfred_tw_env import ( + AlfredDemangler, + AlfredExpert, + AlfredExpertType, + ) + + expert = AlfredExpert(expert_type=AlfredExpertType.HANDCODED) + request_infos = textworld.EnvInfos( + description=True, inventory=True, admissible_commands=True + ) + env_id = textworld.gym.register_game( + game_file_path, request_infos, wrappers=[AlfredDemangler(), expert] + ) + return textworld.gym.make(env_id) + except Exception as e: + raise ImportError( + f"Error creating AlfWorld env: {e}. " + "Ensure alfworld is installed: https://github.com/alfworld/alfworld" + ) from e + + +def _create_alfworld_env_with_checkpoint( + game_file_path: str, actions: List[str], checkpoint_step: int +): + """Create AlfWorld environment and execute actions up to checkpoint_step. + + Args: + game_file_path: Path to the game file + actions: List of predefined expert actions + checkpoint_step: Number of actions to execute before letting model take over + + Returns: + tuple: ( + env, + observation, + info, + history, + task_description, + current_step, + done, + ) + """ + env = _create_alfworld_env(game_file_path) + observation, info = env.reset() + + task_description = _extract_task(observation) + history: List[str] = [] + + # Execute predefined actions up to checkpoint_step + done = False + for step in range(min(checkpoint_step, len(actions))): + action = actions[step] + format_obs = format_observation(observation) + history.append(_format_history(format_obs, step + 1, action)) + observation, reward, done, info = env.step(action) + + if done: + # If task completes before checkpoint, return the final state + break + + return env, observation, info, history, task_description, len(history), done + diff --git a/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/OPD_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/OPD_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..6f8cea24632dde740a3b0eabdac81452698a65a0 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/OPD_workflow.py @@ -0,0 +1,195 @@ +# -*- coding: utf-8 -*- +"""On-Policy Distillation (OPD) workflow for ScienceWorld.""" + +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.scienceworld.utils import ( + HISTORY_LENGTH, + SCIWORLD_TEMPLATE, + SCIWORLD_TEMPLATE_NO_HIS, + _create_scienceworld_env, + _format_history, + _get_compact_action_info, + _reset_scienceworld_env, + format_observation, + parse_action, +) + + +@WORKFLOWS.register_module("OPD_scienceworld_workflow") +class OnPolicyDistillVerlAgentScienceworldWorkflow(Workflow): + """On-policy distillation workflow for ScienceWorld.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.is_eval = task.is_eval + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + env = _create_scienceworld_env( + self.task_desc, + max_env_steps=self.max_env_steps, + generate_gold_path=False, + ) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info, task_description = _reset_scienceworld_env(env) + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + best_score = info.get("score", 0) + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + action_templates, objects = _get_compact_action_info(env) + reformatted_actions = ", ".join( + f"'{s}'" for s in action_templates if s != "help" + ) + reformatted_objects = ", ".join(f"'{s}'" for s in objects) + + if len(history) < HISTORY_LENGTH: + user_content = SCIWORLD_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=format_obs, + action_templates=reformatted_actions, + objects=reformatted_objects, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = SCIWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + action_templates=reformatted_actions, + objects=reformatted_objects, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillScienceworldWorkflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + best_score = max(best_score, info.get("score", best_score + reward)) + if done: + self._env_done = True + self._env_rounds = r + 1 + break + else: + self._env_rounds = self.max_env_steps + + self._final_reward = best_score / 100.0 + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/TCOD_b2f_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/TCOD_b2f_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..8c2eb657845ee28305868785e8cda14e28a6f2be --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/TCOD_b2f_workflow.py @@ -0,0 +1,385 @@ +# -*- coding: utf-8 -*- +"""TCOD Backward-to-Forward workflow for ScienceWorld.""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.scienceworld.utils import ( + HISTORY_LENGTH, + SCIWORLD_TEMPLATE, + SCIWORLD_TEMPLATE_NO_HIS, + _create_scienceworld_env, + _create_scienceworld_env_with_checkpoint, + _format_history, + _get_compact_action_info, + _reset_scienceworld_env, + format_observation, + parse_action, +) + + +@WORKFLOWS.register_module("TCOD_b2f_scienceworld_workflow") +class TCOD_b2f_scienceworld_workflow(Workflow): + """TCOD Backward-to-Forward workflow for ScienceWorld.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.is_eval = task.is_eval + + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = 0 + self._expert_actions: List[str] = [] + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _linear_checkpoint_step( + self, predefined_actions: Optional[List[str]] = None + ) -> Optional[int]: + if predefined_actions is None or len(predefined_actions) == 0: + return 0 + current_step = self._current_training_step + max_expert_actions = len(predefined_actions) - 1 + reduction = current_step // self.checkpoint_steps + checkpoint_step = max( + 0, min(max_expert_actions - reduction, max_expert_actions) + ) + return checkpoint_step + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + if self.is_eval: + env = _create_scienceworld_env( + self.task_desc, + max_env_steps=self.max_env_steps, + generate_gold_path=False, + ) + try: + self._expert_actions = [] + return await self._run_episode(env) + finally: + env.close() + + current_step = 0 + if hasattr(self.task, "batch_id"): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + match = re.match(r"^(\d+)", batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + probe_env = _create_scienceworld_env( + self.task_desc, + max_env_steps=self.max_env_steps, + generate_gold_path=True, + ) + try: + self._expert_actions = probe_env.get_gold_action_sequence() + finally: + probe_env.close() + + effective_checkpoint_step = 0 + if self.checkpoint_strategy == "linear": + effective_checkpoint_step = self._linear_checkpoint_step(self._expert_actions) + + if effective_checkpoint_step > 0: + ( + env, + observation, + info, + history, + task_description, + start_step, + replay_done, + checkpoint_reward, + ) = _create_scienceworld_env_with_checkpoint( + self.task_desc, + self._expert_actions, + effective_checkpoint_step, + max_env_steps=self.max_env_steps, + ) + try: + if replay_done: + self._env_done = True + self._env_rounds = start_step + self._final_reward = checkpoint_reward + return [] + return await self._run_episode_from_checkpoint( + env, + observation, + info, + history, + task_description, + start_step, + checkpoint_reward, + ) + finally: + env.close() + + env = _create_scienceworld_env( + self.task_desc, + max_env_steps=self.max_env_steps, + generate_gold_path=False, + ) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info, task_description = _reset_scienceworld_env(env) + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + best_score = info.get("score", 0) + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + action_templates, objects = _get_compact_action_info(env) + reformatted_actions = ", ".join( + f"'{s}'" for s in action_templates if s != "help" + ) + reformatted_objects = ", ".join(f"'{s}'" for s in objects) + + if len(history) < HISTORY_LENGTH: + user_content = SCIWORLD_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=format_obs, + action_templates=reformatted_actions, + objects=reformatted_objects, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = SCIWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + action_templates=reformatted_actions, + objects=reformatted_objects, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_b2f_scienceworld_workflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + best_score = max(best_score, info.get("score", best_score + reward)) + if done: + self._env_done = True + self._env_rounds = r + 1 + break + else: + self._env_rounds = self.max_env_steps + + self._final_reward = best_score / 100.0 + + return await self._finalize_turn_responses(turn_responses, start_step=0) + + async def _run_episode_from_checkpoint( + self, + env, + observation, + info, + history: List[str], + task_description: str, + start_step: int, + checkpoint_reward: float, + ) -> List[Experience]: + self._env_done = False + self._env_rounds = start_step + self._final_reward = 0.0 + + memory = self.format_messages() + turn_responses: List[Experience] = [] + best_score = max(int(round(checkpoint_reward * 100)), info.get("score", 0)) + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(start_step, self.max_env_steps): + format_obs = format_observation(observation) + action_templates, objects = _get_compact_action_info(env) + reformatted_actions = ", ".join( + f"'{s}'" for s in action_templates if s != "help" + ) + reformatted_objects = ", ".join(f"'{s}'" for s in objects) + + if len(history) < HISTORY_LENGTH: + user_content = SCIWORLD_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=format_obs, + action_templates=reformatted_actions, + objects=reformatted_objects, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = SCIWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + action_templates=reformatted_actions, + objects=reformatted_objects, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_b2f_scienceworld_workflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + best_score = max(best_score, info.get("score", best_score + reward)) + if done: + self._env_done = True + self._env_rounds = r + 1 + break + else: + self._env_rounds = self.max_env_steps + + self._final_reward = best_score / 100.0 + + return await self._finalize_turn_responses(turn_responses, start_step=start_step) + + async def _finalize_turn_responses( + self, turn_responses: List[Experience], *, start_step: int + ) -> List[Experience]: + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = start_step + i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + total_expert_actions = 0 if self.is_eval else len(self._expert_actions) + expert_actions_remaining = max(total_expert_actions - start_step, 0) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["student_env_rounds"] = self._env_rounds - start_step + last_response.metrics["teacher_env_rounds"] = start_step + last_response.metrics["if_teacher"] = 1 if start_step > 0 else 0 + last_response.metrics["expected_teacher_env_rounds"] = expert_actions_remaining + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/TCOD_f2b_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/TCOD_f2b_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..7a80e25ffa7ef84d46211ea633df63c3e086a6cf --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/TCOD_f2b_workflow.py @@ -0,0 +1,245 @@ +# -*- coding: utf-8 -*- +"""TCOD Forward-to-Backward workflow for ScienceWorld.""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.scienceworld.utils import ( + HISTORY_LENGTH, + SCIWORLD_TEMPLATE, + SCIWORLD_TEMPLATE_NO_HIS, + _create_scienceworld_env, + _format_history, + _get_compact_action_info, + _reset_scienceworld_env, + format_observation, + parse_action, +) + + +@WORKFLOWS.register_module("TCOD_f2b_scienceworld_workflow") +class TCOD_f2b_scienceworld_workflow(Workflow): + """TCOD Forward-to-Backward workflow for ScienceWorld.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + self.is_eval = task.is_eval + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = 0 + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _compute_distill_window(self) -> int: + return 1 + (self._current_training_step // self.checkpoint_steps) + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + current_step = 0 + if hasattr(self.task, "batch_id"): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + match = re.match(r"^(\d+)", batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + env = _create_scienceworld_env( + self.task_desc, + max_env_steps=self.max_env_steps, + generate_gold_path=False, + ) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info, task_description = _reset_scienceworld_env(env) + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + best_score = info.get("score", 0) + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + distill_window = self._compute_distill_window() + effective_steps = self.max_env_steps if self.is_eval else min( + distill_window, self.max_env_steps + ) + + for r in range(effective_steps): + format_obs = format_observation(observation) + action_templates, objects = _get_compact_action_info(env) + reformatted_actions = ", ".join( + f"'{s}'" for s in action_templates if s != "help" + ) + reformatted_objects = ", ".join(f"'{s}'" for s in objects) + + if len(history) < HISTORY_LENGTH: + user_content = SCIWORLD_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=format_obs, + action_templates=reformatted_actions, + objects=reformatted_objects, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = SCIWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + action_templates=reformatted_actions, + objects=reformatted_objects, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_f2b_scienceworld_workflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + best_score = max(best_score, info.get("score", best_score + reward)) + + if done: + self._env_done = True + self._env_rounds = r + 1 + break + else: + self._env_rounds = effective_steps + + self._final_reward = best_score / 100.0 + + per_turn_kl_sums: List[float] = [] + + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + trajectory_kl_divergence = sum(per_turn_kl_sums) + total_expert_actions = 0 if self.is_eval else len( + self.raw_task.get("actions", []) + ) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + expected_teacher_env_rounds = max( + total_expert_actions - self._env_rounds, + 0, + ) + last_response.metrics["student_env_rounds"] = self._env_rounds + last_response.metrics["teacher_env_rounds"] = 0 + last_response.metrics["expected_teacher_env_rounds"] = ( + expected_teacher_env_rounds + ) + last_response.metrics["if_teacher"] = ( + 0 + if self.is_eval + else 1 if expected_teacher_env_rounds > 0 and not self._env_done else 0 + ) + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/utils.py b/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..abc8d80564f0c2306014db5ab24f7c97bcbd107d --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/scienceworld/utils.py @@ -0,0 +1,174 @@ +import json +from typing import Any, Dict, List, Tuple + +# --------------------- ScienceWorld --------------------- # +SCIWORLD_SYSTEM_PROMPT = """ +You are an expert agent operating in the ScienceWorld text environment. + +At each step, you must first reason step-by-step within tags, +then output exactly one environment action within tags. +Do not talk to the user. Solve the task by interacting with the environment. +""" + +SCIWORLD_TEMPLATE_NO_HIS = """ +Your ScienceWorld task is: {task_description} +Your current observation is: {current_observation} +Available action commands: [{action_templates}] +Available objects you can interact with: [{objects}] + +Now it's your turn to take an action. Combine an action command with appropriate object(s) to form a valid action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose a valid action for the current step and present it within tags. +""" + +SCIWORLD_TEMPLATE = """ +Your ScienceWorld task is: {task_description} +Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history} +You are now at step {current_step} and your current observation is: {current_observation} +Available action commands: [{action_templates}] +Available objects you can interact with: [{objects}] + +Now it's your turn to take an action. Combine an action command with appropriate object(s) to form a valid action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose a valid action for the current step and present it within tags. +""" + + +def parse_action(response: str) -> str: + try: + return response.split("")[1].split("")[0].strip() + except Exception as e: + print(f"Error parsing action: {e}, response = {response}") + return "" + + +def format_observation(observation: str) -> str: + return observation.strip() + + +HISTORY_LENGTH = 2 +MEMORY_FORMAT = "[Observation {step_num}: '{obs}', Action {step_num}: '{act}']" + + +def _extract_task(task_description: str) -> str: + return task_description.strip() + + +def _format_history(observation: str, step_num: int, act: str) -> str: + return MEMORY_FORMAT.format(step_num=step_num, obs=observation.strip(), act=act) + + +def _parse_task_config(task_desc: Any) -> Dict[str, Any]: + if isinstance(task_desc, dict): + return task_desc + if isinstance(task_desc, str): + return json.loads(task_desc) + raise TypeError(f"Unsupported ScienceWorld task description type: {type(task_desc)}") + + +def _get_admissible_commands(info: Dict[str, Any]) -> List[str]: + commands = info.get("valid", []) + if isinstance(commands, str): + return [commands] if commands else [] + if commands and isinstance(commands[0], list): + commands = commands[0] + return [cmd for cmd in commands if cmd] + + +def _get_compact_action_info(env) -> Tuple[List[str], List[str]]: + """Get compact action representation: templates + objects separately. + + ScienceWorld's get_valid_action_object_combinations() returns ~1162 full + combinations per step, which easily exceeds token limits. Instead, we use + get_possible_actions() (~23 templates) and get_possible_objects() (~20 objects) + which together are ~50 items — about 25x smaller. + """ + try: + actions = env.get_possible_actions() + except AttributeError: + actions = env.getPossibleActions() + + try: + objects = env.get_possible_objects() + except AttributeError: + objects = env.getPossibleObjects() + + if isinstance(actions, str): + actions = [actions] if actions else [] + if isinstance(objects, str): + objects = [objects] if objects else [] + + return actions, objects + + +def _create_scienceworld_env( + task_desc: Any, + *, + max_env_steps: int = 30, + generate_gold_path: bool = False, +): + task_config = _parse_task_config(task_desc) + + try: + from scienceworld import ScienceWorldEnv + except Exception as e: + raise ImportError( + f"Error importing ScienceWorldEnv: {e}. " + "Please make sure the scienceworld package is installed successfully: " + "https://github.com/allenai/ScienceWorld" + ) from e + + task_name = task_config["task_name"] + var_num = task_config["var_num"] + jar_path = task_config.get("jar_path", "") + simplification_str = task_config.get("simplification_str", "easy") + env_step_limit = max(task_config.get("env_step_limit", 100), max_env_steps + 5) + + env = ScienceWorldEnv("", jar_path, envStepLimit=env_step_limit) + env.load(task_name, var_num, simplification_str, generateGoldPath=generate_gold_path) + return env + + +def _reset_scienceworld_env(env) -> Tuple[str, Dict[str, Any], str]: + observation, info = env.reset() + task_description = _extract_task(env.get_task_description()) + return observation, info, task_description + + +def _create_scienceworld_env_with_checkpoint( + task_desc: Any, + actions: List[str], + checkpoint_step: int, + *, + max_env_steps: int = 30, +): + env = _create_scienceworld_env( + task_desc, + max_env_steps=max_env_steps, + generate_gold_path=True, + ) + observation, info, task_description = _reset_scienceworld_env(env) + + history: List[str] = [] + best_score = info.get("score", 0) + + done = False + for step in range(min(checkpoint_step, len(actions))): + action = actions[step] + format_obs = format_observation(observation) + history.append(_format_history(format_obs, step + 1, action)) + observation, reward, done, info = env.step(action) + best_score = max(best_score, info.get("score", best_score + reward)) + if done: + break + + return ( + env, + observation, + info, + history, + task_description, + len(history), + done, + best_score / 100.0, + ) diff --git a/frameworks/trinity/common/workflows/envs/TCOD/taubench/OPD_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/taubench/OPD_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..9e5c23cac5d2a69f7d17a7fb79a21970634b500d --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/taubench/OPD_workflow.py @@ -0,0 +1,220 @@ +# -*- coding: utf-8 -*- +"""On-Policy Distillation (OPD) workflow for tau-bench (retail / airline). + +Differences from the gym text-envs (alfworld/sciworld/textworld): +- An action is a tool call OR a `respond` (natural-language turn to the user). +- There is an LLM-simulated USER in the loop. We drive it with a ModelWrapper + (the teacher, reused as user-simulator) and INJECT its reply into env.user + before each env.step(), so the whole loop stays inside vLLM (no litellm calls). +- Reward is outcome-based, computed once by the env at `done` (DB delta + outputs). + +The OPD signal is unchanged: the student emits text actions (its own tokens); the +teacher scores those same tokens via logprobs_async; advantage = teacher - student. +""" + +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.taubench.utils import ( + HISTORY_LENGTH, + RESPOND, + TAUBENCH_SYSTEM_TEMPLATE, + _compact_tools, + _create_taubench_env, + _parse_task_config, + build_user_system_prompt, + format_observation, + parse_action, +) + + +@WORKFLOWS.register_module("OPD_taubench_workflow") +class OnPolicyDistillVerlAgentTaubenchWorkflow(Workflow): + """On-policy distillation workflow for tau-bench.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__(task=task, model=model, auxiliary_models=auxiliary_models) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + # User simulator: a dedicated 2nd auxiliary model if provided, else reuse teacher. + if len(self.auxiliary_model_wrappers) >= 2: + self.user_sim_model = self.auxiliary_model_wrappers[1] + else: + self.user_sim_model = self.teacher_model + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 10) + self.is_eval = task.is_eval + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def _user_say(self, user_messages: List[dict]) -> str: + """Generate the next simulated-user utterance with the user-sim model.""" + responses = await self.user_sim_model.chat_async( + user_messages, n=1, temperature=self.temperature + ) + text = (responses[0].response_text or "").strip() + user_messages.append({"role": "assistant", "content": text}) + return text + + async def run_async(self) -> List[Experience]: + task_config = _parse_task_config(self.task_desc) + env, task_index = _create_taubench_env(task_config) + try: + return await self._run_episode(env, task_index) + finally: + # tau-bench Env has no close(); nothing to release. + pass + + async def _run_episode(self, env, task_index: int) -> List[Experience]: + from tau_bench.types import Action + + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + instruction = env.tasks[task_index].instruction + system_prompt = TAUBENCH_SYSTEM_TEMPLATE.format( + wiki=env.wiki, tools_json=_compact_tools(env.tools_info) + ) + + # User-simulator conversation state (mirrors tau-bench LLMUserSimulationEnv). + user_messages = [ + {"role": "system", "content": build_user_system_prompt(instruction)}, + {"role": "user", "content": "Hi! How can I help you today?"}, + ] + opening = await self._user_say(user_messages) + env.user.set_reply(opening) + reset_resp = env.reset(task_index=task_index) + observation = reset_resp.observation + + agent_messages = [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": f"Customer: {format_observation(observation)}"}, + ] + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + responses = await self.model.chat_async(agent_messages, **kwargs) + response = responses[0] + response_text = response.response_text or "" + agent_messages.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillTaubenchWorkflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + parsed = parse_action(response_text) + name, args = parsed["name"], parsed["arguments"] + + if name == RESPOND: + content = args.get("content", "") + user_messages.append({"role": "user", "content": content}) + reply = await self._user_say(user_messages) + env.user.set_reply(reply) + env_resp = env.step(Action(name=RESPOND, kwargs={"content": content})) + obs_label = "Customer" + else: + env_resp = env.step(Action(name=name, kwargs=args)) + obs_label = "Tool" + + observation = env_resp.observation + self._env_rounds = r + 1 + agent_messages.append( + {"role": "user", "content": f"{obs_label}: {format_observation(observation)}"} + ) + + if env_resp.done: + self._env_done = True + self._final_reward = float(env_resp.reward) + break + + return await self._finalize_turn_responses(turn_responses) + + async def _finalize_turn_responses( + self, turn_responses: List[Experience] + ) -> List[Experience]: + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["task_reward"] = self._final_reward + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/taubench/TCOD_f2b_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/taubench/TCOD_f2b_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..70f8808a746608ad251968f86fbeb6dcf9b81070 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/taubench/TCOD_f2b_workflow.py @@ -0,0 +1,264 @@ +# -*- coding: utf-8 -*- +"""TCOD Forward-to-Backward workflow for tau-bench (retail / airline). + +Same OPD signal as OPD_workflow.py (student acts, teacher scores the student's own +tokens), but the student's rollout is TRUNCATED to a distill window that grows over +training: `1 + current_step // checkpoint_steps`. It first learns the opening move +(typically the initial customer-facing turn or the first lookup tool call), then +progressively longer prefixes of the episode. + +Unlike b2f, f2b needs NO expert gold path — it only truncates the student's own +rollout — which is why tau-bench can support f2b even though it has no replayable +conversational gold path (its expert `actions` are tool calls without the dialogue +turns, so a b2f handoff state cannot be reconstructed faithfully). + +The LLM-simulated user is driven by a ModelWrapper and injected into env.user, exactly +as in OPD_workflow.py: no litellm / external API calls. +""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.taubench.utils import ( + RESPOND, + TAUBENCH_SYSTEM_TEMPLATE, + _compact_tools, + _create_taubench_env, + _parse_task_config, + build_user_system_prompt, + format_observation, + parse_action, +) + + +@WORKFLOWS.register_module("TCOD_f2b_taubench_workflow") +class TCOD_f2b_taubench_workflow(Workflow): + """TCOD Forward-to-Backward workflow for tau-bench. + + The student always starts at step 0 but its rollout is truncated to a distill + window that grows over training (1 + current_step // checkpoint_steps). + """ + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__(task=task, model=model, auxiliary_models=auxiliary_models) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + # User simulator: a dedicated 2nd auxiliary model if provided, else reuse teacher. + if len(self.auxiliary_model_wrappers) >= 2: + self.user_sim_model = self.auxiliary_model_wrappers[1] + else: + self.user_sim_model = self.teacher_model + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 10) + self.is_eval = task.is_eval + + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = self.total_steps + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _compute_distill_window(self) -> int: + return 1 + (self._current_training_step // self.checkpoint_steps) + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def _user_say(self, user_messages: List[dict]) -> str: + """Generate the next simulated-user utterance with the user-sim model.""" + responses = await self.user_sim_model.chat_async( + user_messages, n=1, temperature=self.temperature + ) + text = (responses[0].response_text or "").strip() + user_messages.append({"role": "assistant", "content": text}) + return text + + async def run_async(self) -> List[Experience]: + current_step = 0 + if hasattr(self.task, "batch_id"): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + match = re.match(r"^(\d+)", batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + task_config = _parse_task_config(self.task_desc) + env, task_index = _create_taubench_env(task_config) + try: + return await self._run_episode(env, task_index) + finally: + # tau-bench Env has no close(); nothing to release. + pass + + async def _run_episode(self, env, task_index: int) -> List[Experience]: + from tau_bench.types import Action + + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + instruction = env.tasks[task_index].instruction + system_prompt = TAUBENCH_SYSTEM_TEMPLATE.format( + wiki=env.wiki, tools_json=_compact_tools(env.tools_info) + ) + + # User-simulator conversation state (mirrors tau-bench LLMUserSimulationEnv). + user_messages = [ + {"role": "system", "content": build_user_system_prompt(instruction)}, + {"role": "user", "content": "Hi! How can I help you today?"}, + ] + opening = await self._user_say(user_messages) + env.user.set_reply(opening) + reset_resp = env.reset(task_index=task_index) + observation = reset_resp.observation + + agent_messages = [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": f"Customer: {format_observation(observation)}"}, + ] + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + distill_window = self._compute_distill_window() + effective_steps = ( + self.max_env_steps if self.is_eval else min(distill_window, self.max_env_steps) + ) + + for r in range(effective_steps): + responses = await self.model.chat_async(agent_messages, **kwargs) + response = responses[0] + response_text = response.response_text or "" + agent_messages.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_f2b_taubench_workflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + parsed = parse_action(response_text) + name, args = parsed["name"], parsed["arguments"] + + if name == RESPOND: + content = args.get("content", "") + user_messages.append({"role": "user", "content": content}) + reply = await self._user_say(user_messages) + env.user.set_reply(reply) + env_resp = env.step(Action(name=RESPOND, kwargs={"content": content})) + obs_label = "Customer" + else: + env_resp = env.step(Action(name=name, kwargs=args)) + obs_label = "Tool" + + observation = env_resp.observation + self._env_rounds = r + 1 + agent_messages.append( + {"role": "user", "content": f"{obs_label}: {format_observation(observation)}"} + ) + + if env_resp.done: + self._env_done = True + self._final_reward = float(env_resp.reward) + break + + return await self._finalize_turn_responses(turn_responses) + + async def _finalize_turn_responses( + self, turn_responses: List[Experience] + ) -> List[Experience]: + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["student_env_rounds"] = self._env_rounds + last_response.metrics["teacher_env_rounds"] = 0 + last_response.metrics["if_teacher"] = 0 + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["task_reward"] = self._final_reward + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + last_response.metrics["distill_window"] = self._compute_distill_window() + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/taubench/TCOD_mixed_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/taubench/TCOD_mixed_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..e0815b44605fc8a055537cfc761212f38b938102 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/taubench/TCOD_mixed_workflow.py @@ -0,0 +1,764 @@ +# -*- coding: utf-8 -*- +"""Mixed student/teacher-policy workflows for tau-bench, under one roof. + +Every published method in this family differs ONLY in *which turns the teacher takes +over (or supervises) and how strongly* -- the rollout loop, the environment plumbing +and the loss are otherwise identical. So they are implemented here as pluggable +`InterventionPolicy` objects over a single workflow rather than as five near-duplicate +files. + + policy method teacher's share is... + ------------------ ---------------------------------- -------------------------- + none vanilla OPD (baseline) nothing + prob_cosine Guided-OPD (arXiv:2606.15912) random turns, beta -> 0 (cosine) + prob_linear_floor DAgger for LLM agents (2605.12913) random turns, beta -> floor (linear) + suffix ROSE / AggreVaTe-style prefix a contiguous SUFFIX after handoff + judge SAGE-OPD (arXiv:2606.19659) turns the TEACHER judges worth it + +None of these three papers released code (checked 2026-07), so the policies are +reimplemented from the papers' descriptions; each class docstring records the exact +rule and its source so the reimplementation is auditable. + +ONE LOSS, TWO KLs +----------------- +The advantage is the same everywhere: + + A = kl_coef * (teacher_logprob - student_logprob) + +Applied to tokens the STUDENT sampled this is reverse-KL (standard OPD). Applied to +tokens the TEACHER sampled it is forward-KL, i.e. the SFT/imitation direction. The +asymmetry Guided-OPD describes therefore falls out of *who sampled the tokens* and +needs no special-casing in the trainer: for a teacher turn we simply score the +teacher's text with the STUDENT (`convert_messages_to_experience`) and with the +teacher, and hand the pair to the same advantage function. + +Per-turn weighting (SAGE-OPD's i_t in {0, alpha, 1}) is applied exactly by rewriting + teacher_logprob <- student_logprob + i_t * (teacher_logprob - student_logprob) +which scales A by i_t without touching the advantage function. +""" + +import math +import random +import re +from dataclasses import asdict +from typing import Any, Dict, List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.taubench.utils import ( + RESPOND, + TAUBENCH_SYSTEM_TEMPLATE, + _compact_tools, + _create_taubench_env, + _parse_task_config, + build_user_system_prompt, + format_observation, + parse_action, +) + +# --------------------------------------------------------------------------- # +# Intervention policies +# --------------------------------------------------------------------------- # +class InterventionPolicy: + """Decides, per turn, who acts and how strongly that turn is supervised. + + `teacher_acts(turn_idx)` -> the teacher GENERATES this turn (replacing the student) + `weight(turn_idx)` -> multiplier on the turn's advantage (SAGE-OPD's i_t) + """ + + name = "none" + needs_judge = False + post_hoc = False # True -> decides AFTER seeing the student's proposal + + def __init__(self, args: Dict[str, Any], step: int, total_steps: int, rng: random.Random): + self.args, self.step, self.total_steps, self.rng = args, step, total_steps, rng + + def start(self, planned_len: int) -> None: + """Called once per episode, before the loop.""" + + def teacher_acts(self, turn_idx: int) -> bool: + return False + + def weight(self, turn_idx: int) -> float: + return 1.0 + + @property + def beta(self) -> float: + return 0.0 + + +class NoIntervention(InterventionPolicy): + """Vanilla OPD: the student acts every turn, every turn is supervised.""" + + name = "none" + + +class ProbCosine(InterventionPolicy): + """Guided-OPD (arXiv:2606.15912). + + "the teacher takes over a complete turn with relatively high probability ... As + training progresses, the probability of teacher intervention decays along a + predefined schedule and eventually reaches zero." + + Cosine is the paper's default. Its stated hyperparameters are beta_start=1, + beta_end=0 and curriculum ratio rho=0.8, with + + beta_t = beta_start + (beta_end - beta_start) * (1 - cos(pi * xi_t)) / 2 + xi_t = min(t / T_decay, 1), T_decay = rho * T_total + + so training BEGINS with the teacher taking every turn (pure behaviour cloning) and + decays to pure OPD over the first 80% of steps. Intervention is at turn + granularity: a whole action plus its reasoning trace. + + KNOWN DEVIATION FROM THE PAPER. Guided-OPD constructs the loss asymmetry + EXPLICITLY -- reverse KL D(pi_theta || pi_phi) on student turns (mode-seeking, + "tightens the student on its own turns toward modes preferred by the teacher") and + forward KL D(pi_phi || pi_theta) on teacher turns (mass-covering, so the student's + support covers what the teacher demonstrates). We instead apply the single existing + OPD advantage kl_coef*(teacher_lp - student_lp) to both, which coincides with the + forward direction only in so far as the tokens were teacher-sampled. It is an + approximation of their teacher-turn loss, not that loss, and the gap is most likely + to matter exactly on teacher turns. Reproducing the explicit form would need a + second loss term in the trainer. + """ + + name = "prob_cosine" + + def start(self, planned_len: int) -> None: + a = self.args + b0 = a.get("beta_start", 1.0) + b1 = a.get("beta_end", 0.0) + frac = a.get("curriculum_frac", 0.8) + horizon = max(1.0, self.total_steps * frac) + xi = min(1.0, self.step / horizon) + self._beta = b0 + (b1 - b0) * (1 - math.cos(math.pi * xi)) / 2 + + def teacher_acts(self, turn_idx: int) -> bool: + return self.rng.random() < self._beta + + @property + def beta(self) -> float: + return getattr(self, "_beta", 0.0) + + +class ProbLinearFloor(InterventionPolicy): + """DAgger for LLM agents (arXiv:2605.12913). + + "at iteration i, we define a mixing parameter beta_i in [0,1], which is decayed + towards 0 across iterations", with the reported schedule + + beta_i = max(0.6, 1.0 - 0.2 * (i - 1)) + + Note this floors at 0.6 rather than reaching 0, so the teacher keeps the majority + of turns even at the end of training -- a real difference from Guided-OPD, and one + worth reproducing faithfully rather than "fixing". + + `iteration_steps` maps trainer steps onto the paper's DAgger iterations. + """ + + name = "prob_linear_floor" + + def start(self, planned_len: int) -> None: + a = self.args + it_steps = max(1, a.get("iteration_steps", 1)) + i = self.step // it_steps + 1 + self._beta = max(a.get("beta_floor", 0.6), + a.get("beta_start", 1.0) - a.get("beta_decay", 0.2) * (i - 1)) + + def teacher_acts(self, turn_idx: int) -> bool: + return self.rng.random() < self._beta + + @property + def beta(self) -> float: + return getattr(self, "_beta", 0.0) + + +class SuffixTakeover(InterventionPolicy): + """ROSE / AggreVaTe-style trajectory-prefix mixing. + + The student owns a contiguous PREFIX and the teacher takes over from a handoff + point to the end -- the online counterpart of the offline ROSE sampler in + TCOD_examples/rose/. The handoff fraction rho is sampled per episode by default + (matching `rose_sample.py`), or annealed towards full student control. + + Contrast with the two probabilistic policies above: they scatter teacher turns and + keep returning control to the student, so most states stay on-policy, whereas a + suffix takeover means only the handoff state is a genuine student state and every + later state was produced by the teacher. + """ + + name = "suffix" + + def start(self, planned_len: int) -> None: + a = self.args + mode = a.get("rho_mode", "random") + if mode == "anneal": # student's share grows over training + rho = min(1.0, (self.step + 1) / max(1, self.total_steps)) + else: + rho = self.rng.uniform(a.get("rho_min", 0.0), a.get("rho_max", 1.0)) + self._rho = rho + self._handoff = max(0, int(round(rho * planned_len))) + + def teacher_acts(self, turn_idx: int) -> bool: + return turn_idx >= self._handoff + + @property + def beta(self) -> float: + return getattr(self, "_rho", 0.0) + + +class TeacherJudge(InterventionPolicy): + """SAGE-OPD (arXiv:2606.19659). + + "observes environment feedback and uses teacher judgment to decide whether each + student response should be skipped or intervened on", assigning Skip (no + supervision), Weak (alpha=0.5) or Strong (full) per turn. + + The student still acts every turn; what the judgment changes is how strongly the + turn is supervised. The judging call happens in the workflow (it needs the model), + which stores the verdict here. + """ + + name = "judge" + needs_judge = True + + def start(self, planned_len: int) -> None: + self._weights: Dict[int, float] = {} + + def set_weight(self, turn_idx: int, w: float) -> None: + self._weights[turn_idx] = w + + def weight(self, turn_idx: int) -> float: + return self._weights.get(turn_idx, 1.0) + + + +class GatedPolicy(InterventionPolicy): + """Base for policies that decide AFTER seeing the student's proposed action. + + The schedule-based policies above decide before the turn and therefore cannot look + at the state. These look at the student's actual proposal and intervene only where + it seems to need it -- the online counterpart of the offline placement policy in + TCOD_examples/rose/rose_ask_policy.py. + + Cost is the point. Guided-OPD spends a teacher GENERATION on ~20% of turns and + DAgger on ~72%, chosen blindly. These spend a teacher generation only where the gate + fires, and the gate itself is either free (`self_gated`, the student's own logprobs) + or already paid for (`kl_gated`: OPD computes the teacher's logprobs on the + student's tokens regardless, to form the advantage). So the expensive thing is + gated by the cheap thing -- speculative decoding's accept/reject rule lifted from + tokens to actions. + + `teacher_acts()` is unused here; the workflow calls `gate()` after generation. + """ + + name = "gated" + post_hoc = True + + def start(self, planned_len: int) -> None: + self._fired = 0 + self._seen = 0 + + def teacher_acts(self, turn_idx: int) -> bool: + return False + + def gate(self, signal: Optional[float]) -> bool: + """True -> the teacher should redo this turn.""" + self._seen += 1 + if signal is None: + return False + thr = self.args.get("threshold", -1.0) + budget = self.args.get("max_rate", 1.0) + if self._seen and (self._fired / self._seen) >= budget: + return False # hard cap on the realised intervention rate + if signal < thr: + self._fired += 1 + return True + return False + + @property + def beta(self) -> float: + return (self._fired / self._seen) if getattr(self, "_seen", 0) else 0.0 + + +class SelfGated(GatedPolicy): + """B1 -- the student's own mean logprob on the action it just produced. + + Completely free: the student returns these logprobs anyway. Weakest of the three + signals (LLM confidence is poorly calibrated) but it is the zero-cost baseline any + fancier gate has to beat. + """ + + name = "self_gated" + + +class KLGated(GatedPolicy): + """B2 -- how much the TEACHER dislikes the action the student just proposed. + + signal = mean teacher logprob of the student's own tokens. OPD already computes + exactly this to build its advantage, so the gate costs nothing extra; only the + teacher's *generation* on the turns that fail the test is paid for. + """ + + name = "kl_gated" + + +class LearnedGated(GatedPolicy): + """B3 -- an offline-fit placement policy, frozen, used as the online gate. + + This is the "learn offline when help is needed, then stop guessing online" route: + the weights come from rose_ask_policy.py, fit on ROSE data whose labels are the + counterfactual value of intervention. Falls back to the KL signal if no weights are + configured, so a missing file degrades to B2 rather than to nothing. + """ + + name = "learned_gated" + + def start(self, planned_len: int) -> None: + super().start(planned_len) + self._w = self.args.get("weights") or {} + path = self.args.get("weights_path") + if not self._w and path: + try: + import json as _json + + with open(path) as f: + self._w = _json.load(f).get("weights", {}) + except Exception as e: + print(f"learned_gated: could not load {path}: {e}; falling back to KL") + + def gate(self, signal, features=None): + if self._w and features: + z = sum(self._w.get(k, 0.0) * v for k, v in features.items()) + p = 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, z)))) + self._seen += 1 + budget = self.args.get("max_rate", 1.0) + if self._seen and (self._fired / self._seen) >= budget: + return False + if p > self.args.get("prob_threshold", 0.5): + self._fired += 1 + return True + return False + return super().gate(signal) + + +POLICIES = { + p.name: p + for p in (NoIntervention, ProbCosine, ProbLinearFloor, SuffixTakeover, TeacherJudge, + SelfGated, KLGated, LearnedGated) +} + +JUDGE_PROMPT = """You are supervising a junior customer-service agent. + +Task the customer wants solved: +{instruction} + +The agent's latest step: +{response} + +Environment feedback after that step: +{observation} + +Was this step good enough to learn from as-is, mildly flawed, or clearly wrong? +Answer with exactly one word: SKIP (it was fine and teaches nothing), WEAK (mildly +flawed), or STRONG (clearly wrong and needs correction).""" + + +@WORKFLOWS.register_module("TCOD_mixed_taubench_workflow") +class TCOD_mixed_taubench_workflow(Workflow): + """One workflow, five intervention policies (see module docstring).""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__(task=task, model=model, auxiliary_models=auxiliary_models) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "Mixed-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + self.user_sim_model = ( + self.auxiliary_model_wrappers[1] + if len(self.auxiliary_model_wrappers) >= 2 + else self.teacher_model + ) + + wa = task.workflow_args + self.temperature = wa.get("temperature", 1.0) + self.max_env_steps = wa.get("max_env_steps", 20) + self.total_steps = wa.get("total_steps", 200) + self.policy_name = wa.get("intervention", "none") + self.policy_args = wa.get("intervention_args", {}) or {} + self.alpha_weak = wa.get("alpha_weak", 0.5) + # How the loss on TEACHER-generated turns is formed. See _finalize. + # "forward_kl" -- Guided-OPD's explicit choice, and the paper-faithful one + # "opd" -- reuse the OPD advantage on those turns too (the earlier, + # approximate behaviour; kept so the difference is measurable) + self.teacher_turn_loss = wa.get("teacher_turn_loss", "forward_kl") + self.forward_kl_coef = wa.get("forward_kl_coef", 1.0) + assert self.policy_name in POLICIES, ( + f"unknown intervention '{self.policy_name}', expected one of {sorted(POLICIES)}" + ) + self._current_training_step = 0 + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + self._current_training_step = current_step + self.total_steps = total_steps + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def _user_say(self, user_messages: List[dict]) -> str: + responses = await self.user_sim_model.chat_async( + user_messages, n=1, temperature=self.temperature + ) + text = (responses[0].response_text or "").strip() + user_messages.append({"role": "assistant", "content": text}) + return text + + async def _judge(self, instruction: str, response_text: str, observation: str) -> float: + """SAGE-OPD's teacher judgment -> per-turn supervision weight.""" + prompt = JUDGE_PROMPT.format( + instruction=instruction[:400], + response=response_text[:600], + observation=format_observation(observation)[:400], + ) + try: + out = await self.teacher_model.chat_async( + [{"role": "user", "content": prompt}], n=1, temperature=0.0 + ) + verdict = (out[0].response_text or "").strip().upper() + except Exception: + return 1.0 + if "SKIP" in verdict: + return 0.0 + if "WEAK" in verdict: + return self.alpha_weak + return 1.0 + + async def run_async(self) -> List[Experience]: + # trainer step (drives the schedules); mirrors the b2f/f2b convention + step = 0 + if hasattr(self.task, "batch_id"): + bid = self.task.batch_id + if isinstance(bid, int): + step = bid + elif isinstance(bid, str): + m = re.match(r"^(\d+)", bid) + if m: + step = int(m.group(1)) + self._current_training_step = step + + rng = random.Random(hash((self.task_desc, step)) & 0xFFFFFFFF) + self.policy = POLICIES[self.policy_name]( + self.policy_args, step, self.total_steps, rng + ) + + task_config = _parse_task_config(self.task_desc) + env, task_index = _create_taubench_env(task_config) + return await self._run_episode(env, task_index) + + async def _run_episode(self, env, task_index: int) -> List[Experience]: + from tau_bench.types import Action + + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + instruction = env.tasks[task_index].instruction + system_prompt = TAUBENCH_SYSTEM_TEMPLATE.format( + wiki=env.wiki, tools_json=_compact_tools(env.tools_info) + ) + user_messages = [ + {"role": "system", "content": build_user_system_prompt(instruction)}, + {"role": "user", "content": "Hi! How can I help you today?"}, + ] + opening = await self._user_say(user_messages) + env.user.set_reply(opening) + observation = env.reset(task_index=task_index).observation + + agent_messages = [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": f"Customer: {format_observation(observation)}"}, + ] + + # eval never intervenes: we measure the student ALONE + if self.is_eval: + self.policy = NoIntervention({}, 0, 1, random.Random(0)) + self.policy.start(self.max_env_steps) + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + turns: List[Dict[str, Any]] = [] # {"exp", "by", "weight"} + n_teacher = 0 + # The simulated customer is driven by a model too, on EVERY respond turn and in + # every method including the OPD baseline. That cost belongs to the environment, + # not to the intervention policy, so it is counted separately -- otherwise + # "teacher generation" conflates the method with the user simulator. + n_user_sim = 1 # the opening utterance + n_respond = 0 + n_judge = 0 + n_gate_probes = 0 # student proposals scored by a gate (cheap) + + for r in range(self.max_env_steps): + teacher_turn = (not self.is_eval) and self.policy.teacher_acts(r) + + # Post-hoc gates: let the STUDENT propose first, score that proposal + # cheaply, and only pay for a teacher generation if the proposal is rejected. + if getattr(self.policy, "post_hoc", False) and not self.is_eval: + probe = (await self.model.chat_async(agent_messages, **kwargs))[0] + probe_text = probe.response_text or "" + signal = None + if self.policy.name == "self_gated": + if probe.logprobs is not None and len(probe.logprobs): + signal = float(probe.logprobs.mean().item()) + else: + # kl_gated / learned_gated: the teacher's opinion of the student's + # own tokens. OPD computes this anyway to build the advantage, so + # the gate is free; only the regeneration below costs anything. + try: + tlp = await self.teacher_model.logprobs_async( + tokens=probe.tokens.tolist(), temperature=self.temperature) + seg = tlp[probe.prompt_length - 1:] + if len(seg): + signal = float(seg.mean().item()) + except Exception as e: + print(f"gate scoring failed: {e}") + gate_kwargs = {} + if self.policy.name == "learned_gated": + gate_kwargs["features"] = { + "bias": 1.0, "rho": r / max(1, self.max_env_steps), + "steps_left": 1.0 - r / max(1, self.max_env_steps), + } + teacher_turn = self.policy.gate(signal, **gate_kwargs) + else: + teacher_turn = self.policy.gate(signal) + n_gate_probes += 1 + if not teacher_turn: + # proposal accepted: reuse it, no teacher generation at all + exp = probe + response_text = probe_text + agent_messages.append({"role": "assistant", "content": response_text}) + if exp.logprobs is None: + raise RuntimeError( + "gated policies require the student to return logprobs.") + weight = 1.0 + parsed = parse_action(response_text) + name, args = parsed["name"], parsed["arguments"] + turns.append({"exp": exp, "by": "student", "weight": weight}) + if name == RESPOND: + content = args.get("content", "") + user_messages.append({"role": "user", "content": content}) + reply = await self._user_say(user_messages) + env.user.set_reply(reply) + env_resp = env.step(Action(name=RESPOND, kwargs={"content": content})) + n_user_sim += 1 + n_respond += 1 + label = "Customer" + else: + env_resp = env.step(Action(name=name, kwargs=args)) + label = "Tool" + observation = env_resp.observation + self._env_rounds = r + 1 + agent_messages.append( + {"role": "user", "content": f"{label}: {format_observation(observation)}"}) + if env_resp.done: + self._env_done = True + self._final_reward = float(env_resp.reward) + break + continue + # rejected: fall through and let the teacher redo the turn. Drop the + # student's rejected proposal from the context. + agent_messages = agent_messages[:] + + if teacher_turn: + # Teacher GENERATES this turn; the student scores the teacher's tokens, + # so the same advantage becomes forward-KL (imitation) on them. + out = await self.teacher_model.chat_async( + agent_messages, n=1, temperature=self.temperature + ) + response_text = out[0].response_text or "" + agent_messages.append({"role": "assistant", "content": response_text}) + try: + exp = await self.model.convert_messages_to_experience_async( + agent_messages, temperature=self.temperature + ) + except Exception as e: + print(f"scoring teacher turn failed: {e}") + exp = None + n_teacher += 1 + weight = 1.0 + else: + out = await self.model.chat_async(agent_messages, **kwargs) + exp = out[0] + response_text = exp.response_text or "" + agent_messages.append({"role": "assistant", "content": response_text}) + if exp.logprobs is None: + raise RuntimeError( + "TCOD_mixed_taubench_workflow requires the student to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in the config." + ) + weight = 1.0 + + parsed = parse_action(response_text) + name, args = parsed["name"], parsed["arguments"] + if name == RESPOND: + content = args.get("content", "") + user_messages.append({"role": "user", "content": content}) + reply = await self._user_say(user_messages) + env.user.set_reply(reply) + env_resp = env.step(Action(name=RESPOND, kwargs={"content": content})) + n_user_sim += 1 + n_respond += 1 + label = "Customer" + else: + env_resp = env.step(Action(name=name, kwargs=args)) + label = "Tool" + + observation = env_resp.observation + self._env_rounds = r + 1 + agent_messages.append( + {"role": "user", "content": f"{label}: {format_observation(observation)}"} + ) + + # SAGE-OPD judges the STUDENT's turns, after seeing the env's feedback + if self.policy.needs_judge and not teacher_turn and not self.is_eval: + weight = await self._judge(instruction, response_text, observation) + n_judge += 1 + self.policy.set_weight(r, weight) + + if exp is not None: + turns.append({"exp": exp, "by": "teacher" if teacher_turn else "student", + "weight": weight}) + + if env_resp.done: + self._env_done = True + self._final_reward = float(env_resp.reward) + break + + return await self._finalize( + turns, n_teacher, + {"user_sim_calls": n_user_sim, "respond_turns": n_respond, + "judge_calls": n_judge, "gate_probes": n_gate_probes}, + ) + + async def _finalize(self, turns: List[Dict[str, Any]], n_teacher: int, + gen_counts: Dict[str, int]) -> List[Experience]: + kept: List[Experience] = [] + kl_sums: List[float] = [] + + for i, rec in enumerate(turns): + exp, weight = rec["exp"], rec["weight"] + if weight <= 0.0: # SAGE-OPD "Skip": drop the turn entirely + continue + try: + teacher_lp = await self.teacher_model.logprobs_async( + tokens=exp.tokens.tolist(), temperature=self.temperature + ) + except Exception as e: + print(f"teacher logprobs failed on turn {i}: {e}") + continue + + resp_start = exp.prompt_length - 1 + t_lp = teacher_lp[resp_start:] + s_lp = exp.logprobs + if len(t_lp) != len(s_lp): + print(f"length mismatch on turn {i}: teacher={len(t_lp)} student={len(s_lp)}") + continue + + # ---- how the advantage is formed, per turn ---- + # The trainer computes A = kl_coef * (teacher_logprobs - student_logprobs), + # so writing a chosen value into teacher_logprobs selects the objective: + # + # student turn -> leave t_lp as the teacher's real logprobs. The tokens + # were sampled from the student, so A = kl_coef*(t - s) is exactly the + # REINFORCE estimator of reverse KL D(pi_theta || pi_phi). Already right. + # + # teacher turn -> Guided-OPD specifies FORWARD KL D(pi_phi || pi_theta). + # Its gradient is -E_{x~pi_phi}[grad log pi_theta(x)], and the tokens ARE + # samples from pi_phi, so the empirical gradient is plain cross-entropy + # on them -- i.e. a CONSTANT advantage, not the logprob gap. Setting + # t_lp := s_lp + c makes A = kl_coef*c, which is exactly that. + # Using the gap here instead (the "opd" option) up-weights tokens the + # student already disagrees with, which is a different objective. + if rec["by"] == "teacher" and self.teacher_turn_loss == "forward_kl": + exp.teacher_logprobs = s_lp + weight * self.forward_kl_coef + elif weight != 1.0: + # SAGE-OPD's per-turn weight, applied exactly: scaling (t - s) by w + # scales the advantage by w without touching the advantage function. + exp.teacher_logprobs = s_lp + weight * (t_lp - s_lp) + else: + exp.teacher_logprobs = t_lp + + if exp.metrics is None: + exp.metrics = {} + exp.reward = self.compute_reward(exp) + exp.eid.run = getattr(self, "run_id_base", 0) + exp.eid.step = len(kept) + kl_sums.append((s_lp - t_lp).sum().item()) + kept.append(exp) + + if kept: + last = kept[-1] + if last.metrics is None: + last.metrics = {} + n = len(turns) + last.metrics.update({ + "env_rounds": self._env_rounds, + "env_done": 1.0 if self._env_done else 0.0, + "task_reward": self._final_reward, + "kl_divergence": sum(kl_sums), + "teacher_turns": float(n_teacher), + "student_turns": float(n - n_teacher), + "intervention_rate": (n_teacher / n) if n else 0.0, + "beta": float(self.policy.beta), + "turns_kept": float(len(kept)), + "turns_skipped": float(n - len(kept)), + # teacher GENERATION accounting, split by who pays for it: + # agent_turn_gens -> the intervention policy (the method) + # user_sim_calls -> the environment (identical across methods) + # judge_calls -> SAGE-OPD's extra query per student turn + "agent_turn_gens": float(n_teacher), + "user_sim_calls": float(gen_counts["user_sim_calls"]), + "respond_turns": float(gen_counts["respond_turns"]), + "judge_calls": float(gen_counts["judge_calls"]), + # gate_probes are teacher FORWARD passes (cheap, already paid for by + # OPD); agent_turn_gens are teacher GENERATIONS (expensive). The ratio + # is the whole point of the gated policies. + "gate_probes": float(gen_counts.get("gate_probes", 0)), + "teacher_gens_total": float(n_teacher + gen_counts["user_sim_calls"] + + gen_counts["judge_calls"]), + }) + return kept diff --git a/frameworks/trinity/common/workflows/envs/TCOD/taubench/utils.py b/frameworks/trinity/common/workflows/envs/TCOD/taubench/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..f7a0252e9f8b61ffef07ba9d519f6a5150740f70 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/taubench/utils.py @@ -0,0 +1,261 @@ +import json +import re +from typing import Any, Dict, List, Optional + +# --------------------- tau-bench --------------------- # +# Sierra tau-bench: tool-agent-user interaction (retail / airline). Unlike the +# gym text-envs (alfworld/sciworld/textworld), an episode also involves an +# LLM-simulated USER. tau-bench's own user simulator calls litellm; to keep the +# whole loop inside TCOD's vLLM ModelWrappers (and avoid external API calls) we +# REPLACE env.user with an InjectedUser whose replies we pre-compute with a +# ModelWrapper (the teacher, reused as user-simulator) and inject before each +# env.step(). This works because Env.step() is synchronous but only READS the +# injected value, while the async model call happens in the workflow. + +# Action is emitted by the student as {"name": ..., "arguments": {...}} +# name == "respond" -> talk to the user (kwargs.content); else a tool call. +RESPOND = "respond" + + +TAUBENCH_SYSTEM_TEMPLATE = """You are a customer-service agent. Follow the domain policy exactly. + +# Domain policy +{wiki} + +# Available tools (JSON schemas) +{tools_json} + +# How to act +At each step, first reason step-by-step inside tags. Then output exactly +one action inside tags as a JSON object: + {{"name": "", "arguments": {{...}}}} +- To talk to the customer, use name "respond" with arguments {{"content": ""}}. +- To call a tool, use its name with the required arguments. +The Action JSON must be valid and on the tool/argument schema above. +""" + + +def _extract_json_obj(blob: str) -> Dict[str, Any]: + """Best-effort: parse the first JSON object found in `blob`.""" + blob = blob.strip() + try: + return json.loads(blob) + except Exception: + pass + i = blob.find("{") + if i >= 0: + try: # tolerate trailing prose / extra braces after a valid object + obj, _ = json.JSONDecoder().raw_decode(blob[i:]) + return obj + except Exception: + pass + raise ValueError("no json object") + + +def parse_action(response: str) -> Dict[str, Any]: + """Parse an action from the model response -> {'name':..., 'arguments':{...}}. + + Accepts, in order of preference: + 1. {...} -- the format the system prompt requests + 2. {...} -- Qwen3's native tool-call format + 3. a bare JSON object with a "name" key, anywhere in the text + ... chain-of-thought is ignored. Only when none of these yield + a named action do we fall back to a `respond` carrying the raw text, so a real + tool call is never silently swallowed as customer chit-chat. + """ + text = response or "" + think_stripped = re.sub(r".*?", " ", text, flags=re.DOTALL) + candidates = [] + for open_tag, close_tag in (("", ""), ("", "")): + if open_tag in think_stripped: + after = think_stripped.split(open_tag, 1)[1] + candidates.append(after.split(close_tag, 1)[0] if close_tag in after else after) + candidates.append(think_stripped) # bare JSON anywhere -- last structured resort + for blob in candidates: + try: + parsed = _extract_json_obj(blob) + except Exception: + continue + if not isinstance(parsed, dict): + continue + name = parsed.get("name") + args = parsed.get("arguments", parsed.get("parameters", {})) + if not isinstance(args, dict): + args = {} + if name: + return {"name": name, "arguments": args} + print(f"Error parsing action: no structured action, response = {response[:200]}") + visible = think_stripped + for tag in ("", ""): + if tag in visible: + visible = visible.split(tag)[-1] + return {"name": RESPOND, "arguments": {"content": visible.strip()[:500]}} + + +def format_observation(observation: str) -> str: + return (observation or "").strip() + + +HISTORY_LENGTH = 4 # tool/user exchanges are short lines; keep a few + + +def _parse_task_config(task_desc: Any) -> Dict[str, Any]: + if isinstance(task_desc, dict): + return task_desc + if isinstance(task_desc, str): + return json.loads(task_desc) + raise TypeError(f"Unsupported tau-bench task description type: {type(task_desc)}") + + +def _compact_tools(tools_info: List[Dict[str, Any]], limit: int = 60) -> str: + """Compact tool schema string (name + description + params) to bound tokens.""" + out = [] + for t in tools_info: + fn = t.get("function", t) + name = fn.get("name", "?") + desc = (fn.get("description", "") or "").strip().replace("\n", " ") + params = list((fn.get("parameters", {}) or {}).get("properties", {}).keys()) + out.append(f'- {name}({", ".join(params)}): {desc[:160]}') + if len(out) >= limit: + break + return "\n".join(out) + + +class InjectedUser: + """Drop-in replacement for tau-bench's user simulator. + + Its reset()/step() just return whatever reply the workflow injected via + set_reply(); the actual (async) generation is done by the workflow with a + ModelWrapper. This keeps Env.step() synchronous while the user reply comes + from an in-process vLLM model instead of litellm. + """ + + def __init__(self) -> None: + self._reply = "###STOP###" + self.total_cost = 0.0 + + def set_reply(self, reply: str) -> None: + self._reply = reply + + def reset(self, instruction: Optional[str] = None) -> str: + return self._reply + + def step(self, content: str) -> str: + return self._reply + + def get_total_cost(self) -> float: + return 0.0 + + +def build_user_system_prompt(instruction: Optional[str]) -> str: + """Mirror tau-bench LLMUserSimulationEnv.build_system_prompt (envs/user.py).""" + instruction_display = ( + ("\n\nInstruction: " + instruction + "\n") if instruction else "" + ) + return f"""You are a user interacting with an agent.{instruction_display} +Rules: +- Just generate one line at a time to simulate the user's message. +- Do not give away all the instruction at once. Only provide the information that is necessary for the current step. +- Do not hallucinate information that is not provided in the instruction. For example, if the agent asks for the order id but it is not mentioned in the instruction, do not make up an order id, just say you do not remember or have it. +- If the instruction goal is satisified, generate '###STOP###' as a standalone message without anything else to end the conversation. +- Do not repeat the exact instruction in the conversation. Instead, use your own words to convey the same information. +- Try to make the conversation as natural as possible, and stick to the personalities in the instruction.""" + + +def _create_taubench_env_with_checkpoint( + task_config: Dict[str, Any], + turns: List[Dict[str, Any]], + checkpoint_step: int, + opening: str, +): + """Rebuild a tau-bench env and replay the first `checkpoint_step` recorded turns. + + The ROSE counterpart of the `_create_*_env_with_checkpoint` helpers in the text + envs. Replay here is DETERMINISTIC and costs no model calls: + + - tool calls are pure functions of (env.data, kwargs) -- no randomness -- so + replaying them reproduces the database exactly; + - `respond` actions go through `env.user.step()`, which on an InjectedUser returns + whatever was injected, so we re-inject the customer reply RECORDED during the + original rollout instead of generating a new one. InjectedUser therefore needs + no replay mode of its own: "replay" vs "live" is purely a property of who sets + the reply. + + Replaying through `env.step()` also rebuilds `env.actions`, which matters because + `calculate_reward()` scans it for the task's required outputs. + + `turns` entries are the sampler's turn records: {"action": {"name","arguments"}, + "user_reply": }. + + Returns: + (env, task_index, observation, user_messages, step, done, score) + """ + from tau_bench.types import Action + + env, task_index = _create_taubench_env(task_config) + instruction = env.tasks[task_index].instruction + + user_messages = [ + {"role": "system", "content": build_user_system_prompt(instruction)}, + {"role": "user", "content": "Hi! How can I help you today?"}, + {"role": "assistant", "content": opening}, + ] + env.user.set_reply(opening) + observation = env.reset(task_index=task_index).observation + + done = False + score = 0.0 + executed = 0 + for turn in turns[:checkpoint_step]: + action = turn.get("action") or {} + name = action.get("name") + args = action.get("arguments", {}) or {} + if not name: + break + if name == RESPOND: + content = args.get("content", "") + reply = turn.get("user_reply", "") + user_messages.append({"role": "user", "content": content}) + user_messages.append({"role": "assistant", "content": reply}) + env.user.set_reply(reply) + resp = env.step(Action(name=RESPOND, kwargs={"content": content})) + else: + resp = env.step(Action(name=name, kwargs=args)) + observation = resp.observation + executed += 1 + if resp.done: + done = True + score = float(resp.reward) + break + + return env, task_index, observation, user_messages, executed, done, score + + +def _create_taubench_env(task_config: Dict[str, Any]): + """Construct a tau-bench Env with an InjectedUser (no litellm / no network). + + user_strategy='human' means load_user() builds a HumanUserSimulationEnv that + makes no model call at construction; we then swap in InjectedUser. + """ + try: + from tau_bench.envs import get_env + except Exception as e: + raise ImportError( + f"Error importing tau_bench: {e}. Install it with " + "`pip install -e /path/to/tau-bench` (Sierra tau-bench)." + ) from e + + env_name = task_config.get("env_name", "retail") + task_split = task_config.get("task_split", "test") + task_index = int(task_config.get("task_index", 0)) + + env = get_env( + env_name, + user_strategy="human", # no network at construction; replaced below + user_model="", + task_split=task_split, + user_provider=None, + task_index=task_index, + ) + env.user = InjectedUser() + return env, task_index diff --git a/frameworks/trinity/common/workflows/envs/TCOD/textworld/OPD_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/textworld/OPD_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..19194bf8321fcbd47509a97ec0acb14e5993704b --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/textworld/OPD_workflow.py @@ -0,0 +1,183 @@ +# -*- coding: utf-8 -*- +"""On-Policy Distillation (OPD) workflow for TextWorld.""" + +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.textworld.utils import ( + HISTORY_LENGTH, + TEXTWORLD_TEMPLATE, + TEXTWORLD_TEMPLATE_NO_HIS, + _create_textworld_env, + _format_history, + _get_admissible_commands, + _reset_textworld_env, + _score_fraction, + format_observation, + parse_action, +) + + +@WORKFLOWS.register_module("OPD_textworld_workflow") +class OnPolicyDistillVerlAgentTextworldWorkflow(Workflow): + """On-policy distillation workflow for TextWorld.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__(task=task, model=model, auxiliary_models=auxiliary_models) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 50) + self.is_eval = task.is_eval + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + env = _create_textworld_env(self.task_desc, max_env_steps=self.max_env_steps) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info, task_description = _reset_textworld_env(env) + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + best_score = _score_fraction(info) + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + admissible = _get_admissible_commands(info) + reformatted_admissible = "\n ".join(f"'{s}'" for s in admissible) + + if len(history) < HISTORY_LENGTH: + user_content = TEXTWORLD_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = TEXTWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillTextworldWorkflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + best_score = max(best_score, _score_fraction(info)) + if done: + self._env_done = True + self._env_rounds = r + 1 + break + else: + self._env_rounds = self.max_env_steps + + self._final_reward = best_score + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/textworld/TCOD_b2f_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/textworld/TCOD_b2f_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..d9ec9dcd6465ee5dc82ef74c31e6521d8feee487 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/textworld/TCOD_b2f_workflow.py @@ -0,0 +1,300 @@ +# -*- coding: utf-8 -*- +"""TCOD Backward-to-Forward workflow for TextWorld.""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.textworld.utils import ( + HISTORY_LENGTH, + TEXTWORLD_TEMPLATE, + TEXTWORLD_TEMPLATE_NO_HIS, + _create_textworld_env, + _create_textworld_env_with_checkpoint, + _format_history, + _get_admissible_commands, + _get_walkthrough, + _reset_textworld_env, + _score_fraction, + format_observation, + parse_action, +) + + +@WORKFLOWS.register_module("TCOD_b2f_textworld_workflow") +class TCOD_b2f_textworld_workflow(Workflow): + """TCOD Backward-to-Forward workflow for TextWorld. + + The expert gold path is the env's walkthrough (EnvInfos extras=["walkthrough"]). + At training step 0 the whole walkthrough (minus the last action) is replayed to + fast-forward the env; the student only rolls out the remaining suffix. As training + advances, the handoff point moves earlier (student's share grows). + """ + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__(task=task, model=model, auxiliary_models=auxiliary_models) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 50) + self.is_eval = task.is_eval + + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = 0 + self._expert_actions: List[str] = [] + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _linear_checkpoint_step( + self, predefined_actions: Optional[List[str]] = None + ) -> Optional[int]: + if predefined_actions is None or len(predefined_actions) == 0: + return 0 + current_step = self._current_training_step + max_expert_actions = len(predefined_actions) - 1 + reduction = current_step // self.checkpoint_steps + checkpoint_step = max(0, min(max_expert_actions - reduction, max_expert_actions)) + return checkpoint_step + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + if self.is_eval: + env = _create_textworld_env(self.task_desc, max_env_steps=self.max_env_steps) + try: + self._expert_actions = [] + return await self._run_episode(env) + finally: + env.close() + + current_step = 0 + if hasattr(self.task, "batch_id"): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + match = re.match(r"^(\d+)", batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + self._expert_actions = _get_walkthrough( + self.task_desc, max_env_steps=self.max_env_steps + ) + + effective_checkpoint_step = 0 + if self.checkpoint_strategy == "linear": + effective_checkpoint_step = self._linear_checkpoint_step(self._expert_actions) + + if effective_checkpoint_step > 0: + ( + env, + observation, + info, + history, + task_description, + start_step, + replay_done, + checkpoint_reward, + ) = _create_textworld_env_with_checkpoint( + self.task_desc, + self._expert_actions, + effective_checkpoint_step, + max_env_steps=self.max_env_steps, + ) + try: + if replay_done: + self._env_done = True + self._env_rounds = start_step + self._final_reward = checkpoint_reward + return [] + return await self._run_episode_from_checkpoint( + env, observation, info, history, task_description, start_step, + checkpoint_reward, + ) + finally: + env.close() + + env = _create_textworld_env(self.task_desc, max_env_steps=self.max_env_steps) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _rollout_loop(self, env, observation, info, task_description, history, + turn_responses, start_step, best_score): + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + memory = self.format_messages() + + for r in range(start_step, self.max_env_steps): + format_obs = format_observation(observation) + admissible = _get_admissible_commands(info) + reformatted_admissible = "\n ".join(f"'{s}'" for s in admissible) + + if len(history) < HISTORY_LENGTH: + user_content = TEXTWORLD_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = TEXTWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_b2f_textworld_workflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + best_score = max(best_score, _score_fraction(info)) + if done: + self._env_done = True + self._env_rounds = r + 1 + break + else: + self._env_rounds = self.max_env_steps + + self._final_reward = best_score + + async def _run_episode(self, env) -> List[Experience]: + observation, info, task_description = _reset_textworld_env(env) + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + turn_responses: List[Experience] = [] + await self._rollout_loop( + env, observation, info, task_description, [], turn_responses, 0, + _score_fraction(info), + ) + return await self._finalize_turn_responses(turn_responses, start_step=0) + + async def _run_episode_from_checkpoint( + self, env, observation, info, history, task_description, start_step, + checkpoint_reward, + ) -> List[Experience]: + self._env_done = False + self._env_rounds = start_step + self._final_reward = 0.0 + turn_responses: List[Experience] = [] + await self._rollout_loop( + env, observation, info, task_description, history, turn_responses, + start_step, max(checkpoint_reward, _score_fraction(info)), + ) + return await self._finalize_turn_responses(turn_responses, start_step=start_step) + + async def _finalize_turn_responses( + self, turn_responses: List[Experience], *, start_step: int + ) -> List[Experience]: + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = start_step + i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + total_expert_actions = 0 if self.is_eval else len(self._expert_actions) + expert_actions_remaining = max(total_expert_actions - start_step, 0) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["student_env_rounds"] = self._env_rounds - start_step + last_response.metrics["teacher_env_rounds"] = start_step + last_response.metrics["if_teacher"] = 1 if start_step > 0 else 0 + last_response.metrics["expected_teacher_env_rounds"] = expert_actions_remaining + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/textworld/TCOD_f2b_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/textworld/TCOD_f2b_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..5d0830040b4cb20bba20bb45e02dc91de7328dc4 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/textworld/TCOD_f2b_workflow.py @@ -0,0 +1,225 @@ +# -*- coding: utf-8 -*- +"""TCOD Forward-to-Backward workflow for TextWorld.""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.textworld.utils import ( + HISTORY_LENGTH, + TEXTWORLD_TEMPLATE, + TEXTWORLD_TEMPLATE_NO_HIS, + _create_textworld_env, + _format_history, + _get_admissible_commands, + _reset_textworld_env, + _score_fraction, + format_observation, + parse_action, +) + + +@WORKFLOWS.register_module("TCOD_f2b_textworld_workflow") +class TCOD_f2b_textworld_workflow(Workflow): + """TCOD Forward-to-Backward workflow for TextWorld. + + The student always starts at step 0 but its rollout is truncated to a distill + window that grows over training (1 + current_step // checkpoint_steps): it first + learns the opening move, then progressively longer prefixes of the episode. + """ + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__(task=task, model=model, auxiliary_models=auxiliary_models) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 50) + self.is_eval = task.is_eval + + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = 0 + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _compute_distill_window(self) -> int: + return 1 + (self._current_training_step // self.checkpoint_steps) + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + current_step = 0 + if hasattr(self.task, "batch_id"): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + match = re.match(r"^(\d+)", batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + env = _create_textworld_env(self.task_desc, max_env_steps=self.max_env_steps) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info, task_description = _reset_textworld_env(env) + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + best_score = _score_fraction(info) + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + distill_window = self._compute_distill_window() + effective_steps = ( + self.max_env_steps if self.is_eval else min(distill_window, self.max_env_steps) + ) + + for r in range(effective_steps): + format_obs = format_observation(observation) + admissible = _get_admissible_commands(info) + reformatted_admissible = "\n ".join(f"'{s}'" for s in admissible) + + if len(history) < HISTORY_LENGTH: + user_content = TEXTWORLD_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = TEXTWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_f2b_textworld_workflow requires student model to return " + "logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + best_score = max(best_score, _score_fraction(info)) + if done: + self._env_done = True + self._env_rounds = r + 1 + break + else: + self._env_rounds = effective_steps + + self._final_reward = best_score + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + total_expert_actions = 0 if self.is_eval else 0 + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["student_env_rounds"] = self._env_rounds + last_response.metrics["teacher_env_rounds"] = 0 + last_response.metrics["if_teacher"] = 0 + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/textworld/utils.py b/frameworks/trinity/common/workflows/envs/TCOD/textworld/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..da8f65ced22497fc8f565b95ed50dc737ce0c0df --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/textworld/utils.py @@ -0,0 +1,180 @@ +import json +import tempfile +from typing import Any, Dict, List, Tuple + +# --------------------- TextWorld --------------------- # +# Microsoft TextWorld (the substrate ALFWorld is built on). Games are generated +# deterministically from a seed + size config, so the taskset only stores a small +# JSON per task (no game files to ship). The expert "gold path" for TCOD-b2f comes +# from the env itself: EnvInfos(extras=["walkthrough"]) exposes it as +# info["extra.walkthrough"] at reset -- the same "gold from env" pattern ScienceWorld +# uses via get_gold_action_sequence(). + +TEXTWORLD_TEMPLATE_NO_HIS = """ +You are an expert agent operating in the TextWorld text-adventure environment. Your task is to: {task_description} +Your current observation is: {current_observation} +Your admissible actions of the current situation are: [{admissible_actions}]. + +Now it's your turn to take an action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" + +TEXTWORLD_TEMPLATE = """ +You are an expert agent operating in the TextWorld text-adventure environment. Your task is to: {task_description} +Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history} +You are now at step {current_step} and your current observation is: {current_observation} +Your admissible actions of the current situation are: [{admissible_actions}]. + +Now it's your turn to take an action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" + + +def parse_action(response: str) -> str: + try: + return response.split("")[1].split("")[0].strip() + except Exception as e: + print(f"Error parsing action: {e}, response = {response}") + return "" + + +def format_observation(observation: str) -> str: + return observation.strip() + + +HISTORY_LENGTH = 2 +MEMORY_FORMAT = "[Observation {step_num}: '{obs}', Action {step_num}: '{act}']" + + +def _format_history(observation: str, step_num: int, act: str) -> str: + return MEMORY_FORMAT.format(step_num=step_num, obs=observation.strip(), act=act) + + +def _parse_task_config(task_desc: Any) -> Dict[str, Any]: + if isinstance(task_desc, dict): + return task_desc + if isinstance(task_desc, str): + return json.loads(task_desc) + raise TypeError(f"Unsupported TextWorld task description type: {type(task_desc)}") + + +def _extract_task(info: Dict[str, Any]) -> str: + """Objective string for the current game.""" + objective = (info.get("objective") or "").strip() + return objective or "Explore the world and complete the quest." + + +def _get_admissible_commands(info: Dict[str, Any]) -> List[str]: + commands = info.get("admissible_commands", []) or [] + if commands and isinstance(commands[0], list): + commands = commands[0] + return [c for c in commands if c and c != "help"] + + +def _make_game_options(task_config: Dict[str, Any], max_env_steps: int): + import textworld + + opts = textworld.GameOptions() + opts.seeds = int(task_config.get("seed", 1234)) + opts.nb_rooms = int(task_config.get("nb_rooms", 3)) + opts.nb_objects = int(task_config.get("nb_objects", 4)) + opts.quest_length = int(task_config.get("quest_length", 5)) + opts.path = tempfile.mkdtemp(prefix="tw_game_") + return opts + + +def _create_textworld_env(task_desc: Any, *, max_env_steps: int = 50): + """Generate a TextWorld game from the task config and return a gym env. + + The env's reset() info carries admissible_commands, objective, score/max_score, + won/lost, and (if requested) extra.walkthrough (the gold action sequence). + """ + task_config = _parse_task_config(task_desc) + try: + import textworld + import textworld.gym + from textworld import EnvInfos + except Exception as e: + raise ImportError( + f"Error importing TextWorld: {e}. Please `pip install textworld` " + "(it is also pulled in by alfworld)." + ) from e + + opts = _make_game_options(task_config, max_env_steps) + game_file, _game = textworld.make(opts) + + request_infos = EnvInfos( + description=True, + inventory=True, + admissible_commands=True, + objective=True, + won=True, + lost=True, + score=True, + max_score=True, + extras=["walkthrough"], + ) + env_id = textworld.gym.register_game( + game_file, request_infos, max_episode_steps=max_env_steps + 5 + ) + return textworld.gym.make(env_id) + + +def _reset_textworld_env(env) -> Tuple[str, Dict[str, Any], str]: + observation, info = env.reset() + task_description = _extract_task(info) + return observation, info, task_description + + +def _get_walkthrough(task_desc: Any, *, max_env_steps: int = 50) -> List[str]: + """Generate the game once and read the gold walkthrough from reset() info.""" + env = _create_textworld_env(task_desc, max_env_steps=max_env_steps) + try: + _obs, info = env.reset() + wt = info.get("extra.walkthrough") or [] + return list(wt) + finally: + env.close() + + +def _score_fraction(info: Dict[str, Any]) -> float: + score = info.get("score", 0) or 0 + max_score = info.get("max_score", 0) or 0 + if max_score <= 0: + return 1.0 if info.get("won") else 0.0 + return float(score) / float(max_score) + + +def _create_textworld_env_with_checkpoint( + task_desc: Any, + actions: List[str], + checkpoint_step: int, + *, + max_env_steps: int = 50, +): + """Create a TextWorld env and replay the gold walkthrough up to checkpoint_step.""" + env = _create_textworld_env(task_desc, max_env_steps=max_env_steps) + observation, info, task_description = _reset_textworld_env(env) + + history: List[str] = [] + done = False + for step in range(min(checkpoint_step, len(actions))): + action = actions[step] + format_obs = format_observation(observation) + history.append(_format_history(format_obs, step + 1, action)) + observation, reward, done, info = env.step(action) + if done: + break + + return ( + env, + observation, + info, + history, + task_description, + len(history), + done, + _score_fraction(info), + ) diff --git a/frameworks/trinity/common/workflows/envs/TCOD/webshop/OPD_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/webshop/OPD_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..25cee7408d8fd9aef738bfcacdd845d54314b7b2 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/webshop/OPD_workflow.py @@ -0,0 +1,203 @@ +# -*- coding: utf-8 -*- +"""On-Policy Distillation (OPD) workflow for WebShop.""" + +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow + +from trinity.common.workflows.envs.TCOD.webshop.utils import ( + HISTORY_LENGTH, + WEBSHOP_TEMPLATE, + WEBSHOP_TEMPLATE_NO_HIS, + _create_webshop_env, + _extract_task_description, + _format_available_actions, + _format_history, + format_observation, + parse_action, + validate_action, +) + +@WORKFLOWS.register_module("OPD_webshop_workflow") +class OnPolicyDistillVerlAgentWebshopWorkflow(Workflow): + """On-policy distillation workflow for WebShop.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 15) + self.env = _create_webshop_env() + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + self.repeat_times = task.repeat_times or 1 + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + session_id = int(self.task_desc) + all_turn_responses: List[Experience] = [] + + for rollout_idx in range(self.repeat_times): + rollout_responses = await self._run_episode( + session_id=session_id, + run_id=self.run_id_base + rollout_idx, + ) + all_turn_responses.extend(rollout_responses) + + return all_turn_responses + + async def _run_episode(self, session_id: int, run_id: int) -> List[Experience]: + self.env.reset(session=session_id) + observation = self.env.observation + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + task_description = _extract_task_description(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + available_actions = self.env.get_available_actions() + formatted_observation = format_observation(observation) + formatted_actions = _format_available_actions(available_actions) + + if len(history) < HISTORY_LENGTH: + user_content = WEBSHOP_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=formatted_observation, + available_actions=formatted_actions, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = WEBSHOP_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=formatted_observation, + available_actions=formatted_actions, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillVerlAgentWebshopWorkflow requires student model " + "to return logprobs. Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + action_valid, error_msg = validate_action(action, available_actions) + history.append(_format_history(formatted_observation, r + 1, action)) + + if action_valid: + observation, reward, done, _ = self.env.step(action) + else: + observation = error_msg + reward = 0.0 + done = False + + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = float(reward) + break + else: + self._env_rounds = self.max_env_steps + + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = run_id + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + last_response.metrics["session_id"] = float(session_id) + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/webshop/TCOD_b2f_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/webshop/TCOD_b2f_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..3d4184d0c83859f15423bbe6a63b41085d6d8d0b --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/webshop/TCOD_b2f_workflow.py @@ -0,0 +1,412 @@ +# -*- coding: utf-8 -*- +"""TCOD Backward-to-Forward workflow for WebShop.""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.webshop.utils import ( + HISTORY_LENGTH, + WEBSHOP_TEMPLATE, + WEBSHOP_TEMPLATE_NO_HIS, + _create_webshop_env, + _create_webshop_env_with_checkpoint, + _extract_task_description, + _format_available_actions, + _format_history, + format_observation, + parse_action, + validate_action, +) + + +@WORKFLOWS.register_module("TCOD_b2f_webshop_workflow") +class TCOD_b2f_webshop_workflow(Workflow): + """TCOD Backward-to-Forward workflow for WebShop task.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 15) + self.is_eval = task.is_eval + + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = 0 + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _linear_checkpoint_step( + self, predefined_actions: Optional[List[str]] = None + ) -> Optional[int]: + if predefined_actions is None or len(predefined_actions) == 0: + return 0 + current_step = self._current_training_step + max_expert_actions = len(predefined_actions) - 1 + reduction = current_step // self.checkpoint_steps + checkpoint_step = max(0, min(max_expert_actions - reduction, max_expert_actions)) + return checkpoint_step + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + if self.is_eval: + env = _create_webshop_env() + try: + return await self._run_episode(env, int(self.task_desc)) + finally: + env.close() + + current_step = 0 + if hasattr(self.task, "batch_id"): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + match = re.match(r"^(\d+)", batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + session_id = int(self.task_desc) + predefined_actions = self.raw_task.get("actions", None) + + if self.checkpoint_strategy == "linear": + effective_checkpoint_step = self._linear_checkpoint_step(predefined_actions) + else: + effective_checkpoint_step = 0 + + if ( + effective_checkpoint_step is not None + and effective_checkpoint_step > 0 + and predefined_actions is not None + ): + ( + env, + observation, + history, + task_description, + start_step, + replay_done, + replay_reward, + ) = _create_webshop_env_with_checkpoint( + session_id, predefined_actions, effective_checkpoint_step + ) + try: + if replay_done: + self._env_done = True + self._env_rounds = start_step + self._final_reward = replay_reward + return [] + return await self._run_episode_from_checkpoint( + env, + session_id, + observation, + history, + task_description, + start_step, + ) + finally: + env.close() + + env = _create_webshop_env() + try: + return await self._run_episode(env, session_id) + finally: + env.close() + + async def _run_episode(self, env, session_id: int) -> List[Experience]: + env.reset(session=session_id) + observation = env.observation + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + task_description = _extract_task_description(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + available_actions = env.get_available_actions() + formatted_observation = format_observation(observation) + formatted_actions = _format_available_actions(available_actions) + + if len(history) < HISTORY_LENGTH: + user_content = WEBSHOP_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=formatted_observation, + available_actions=formatted_actions, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = WEBSHOP_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=formatted_observation, + available_actions=formatted_actions, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_b2f_webshop_workflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + action_valid, error_msg = validate_action(action, available_actions) + history.append(_format_history(formatted_observation, r + 1, action)) + + if action_valid: + observation, reward, done, _ = env.step(action) + else: + observation = error_msg + reward = 0.0 + done = False + + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = float(reward) + break + else: + self._env_rounds = self.max_env_steps + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + + if self.is_eval: + total_expert_actions = 0 + else: + total_expert_actions = len(self.raw_task.get("actions", [])) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["student_env_rounds"] = self._env_rounds + last_response.metrics["teacher_env_rounds"] = 0 + last_response.metrics["expected_teacher_env_rounds"] = total_expert_actions + last_response.metrics["if_teacher"] = 0 + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + last_response.metrics["session_id"] = float(session_id) + + return turn_responses + + async def _run_episode_from_checkpoint( + self, + env, + session_id: int, + observation, + history: List[str], + task_description: str, + start_step: int, + ) -> List[Experience]: + self._env_done = False + self._env_rounds = start_step + self._final_reward = 0.0 + + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(start_step, self.max_env_steps): + available_actions = env.get_available_actions() + formatted_observation = format_observation(observation) + formatted_actions = _format_available_actions(available_actions) + + if len(history) < HISTORY_LENGTH: + user_content = WEBSHOP_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=formatted_observation, + available_actions=formatted_actions, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = WEBSHOP_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=formatted_observation, + available_actions=formatted_actions, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_b2f_webshop_workflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + action_valid, error_msg = validate_action(action, available_actions) + history.append(_format_history(formatted_observation, r + 1, action)) + + if action_valid: + observation, reward, done, _ = env.step(action) + else: + observation = error_msg + reward = 0.0 + done = False + + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = float(reward) + break + else: + self._env_rounds = self.max_env_steps + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = start_step + i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + trajectory_kl_divergence = sum(per_turn_kl_sums) + total_expert_actions = len(self.raw_task.get("actions", [])) + expert_actions_remaining = max(total_expert_actions - start_step, 0) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["student_env_rounds"] = self._env_rounds - start_step + last_response.metrics["teacher_env_rounds"] = start_step + last_response.metrics["if_teacher"] = 1 if start_step > 0 else 0 + last_response.metrics["expected_teacher_env_rounds"] = expert_actions_remaining + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + last_response.metrics["session_id"] = float(session_id) + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/webshop/TCOD_f2b_workflow.py b/frameworks/trinity/common/workflows/envs/TCOD/webshop/TCOD_f2b_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..2dbe960fd28ee1baad8e86363da674947e3817a9 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/webshop/TCOD_f2b_workflow.py @@ -0,0 +1,248 @@ +# -*- coding: utf-8 -*- +"""TCOD Forward-to-Backward workflow for WebShop.""" + +import re +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows import WORKFLOWS, Task, Workflow +from trinity.common.workflows.envs.TCOD.webshop.utils import ( + HISTORY_LENGTH, + WEBSHOP_TEMPLATE, + WEBSHOP_TEMPLATE_NO_HIS, + _create_webshop_env, + _extract_task_description, + _format_available_actions, + _format_history, + format_observation, + parse_action, + validate_action, +) + + +@WORKFLOWS.register_module("TCOD_f2b_webshop_workflow") +class TCOD_f2b_webshop_workflow(Workflow): + """TCOD Forward-to-Backward workflow for WebShop task.""" + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = False + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + self.is_eval = task.is_eval + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 15) + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", "linear") + self.checkpoint_steps = task.workflow_args.get("checkpoint_steps", 5) + self.total_steps = task.workflow_args.get("total_steps", 200) + + self._current_training_step = 0 + self._total_training_steps = 0 + + def reset(self, task: Task): + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _compute_distill_window(self) -> int: + distill_window = 1 + (self._current_training_step // self.checkpoint_steps) + return distill_window + + def compute_reward(self, response: Experience) -> float: + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + return [] + + async def run_async(self) -> List[Experience]: + current_step = 0 + if hasattr(self.task, "batch_id"): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + match = re.match(r"^(\d+)", batch_id) + if match: + current_step = int(match.group(1)) + + self.set_training_progress(current_step, self.total_steps) + + session_id = int(self.task_desc) + env = _create_webshop_env() + try: + return await self._run_episode(env, session_id) + finally: + env.close() + + async def _run_episode(self, env, session_id: int) -> List[Experience]: + env.reset(session=session_id) + observation = env.observation + self._env_done = False + self._env_rounds = 0 + self._final_reward = 0.0 + + task_description = _extract_task_description(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + distill_window = self._compute_distill_window() + if self.is_eval: + effective_steps = self.max_env_steps + else: + effective_steps = min(distill_window, self.max_env_steps) + + for r in range(effective_steps): + available_actions = env.get_available_actions() + formatted_observation = format_observation(observation) + formatted_actions = _format_available_actions(available_actions) + + if len(history) < HISTORY_LENGTH: + user_content = WEBSHOP_TEMPLATE_NO_HIS.format( + task_description=task_description, + current_observation=formatted_observation, + available_actions=formatted_actions, + ) + else: + action_history_str = "\n".join(history[-HISTORY_LENGTH:]) + user_content = WEBSHOP_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=formatted_observation, + available_actions=formatted_actions, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "TCOD_f2b_webshop_workflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + action_valid, error_msg = validate_action(action, available_actions) + history.append(_format_history(formatted_observation, r + 1, action)) + + if action_valid: + observation, reward, done, _ = env.step(action) + else: + observation = error_msg + reward = 0.0 + done = False + + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = float(reward) + break + else: + self._env_rounds = effective_steps + + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + if response.metrics is None: + response.metrics = {} + response.reward = self.compute_reward(response) + response.eid.run = getattr(self, "run_id_base", 0) + response.eid.step = i + + trajectory_kl_divergence = sum(per_turn_kl_sums) + + if self.is_eval: + total_expert_actions = 0 + else: + total_expert_actions = len(self.raw_task.get("actions", [])) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + expected_teacher_env_rounds = max( + total_expert_actions - self._env_rounds, + 0, + ) + last_response.metrics["student_env_rounds"] = self._env_rounds + last_response.metrics["teacher_env_rounds"] = 0 + last_response.metrics["expected_teacher_env_rounds"] = ( + expected_teacher_env_rounds + ) + last_response.metrics["if_teacher"] = ( + 0 + if self.is_eval + else 1 if expected_teacher_env_rounds > 0 and not self._env_done else 0 + ) + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + last_response.metrics["session_id"] = float(session_id) + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/TCOD/webshop/utils.py b/frameworks/trinity/common/workflows/envs/TCOD/webshop/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..84cccfa7befd65c6060bdbbb09ca10e689c30bce --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/TCOD/webshop/utils.py @@ -0,0 +1,193 @@ +# Copyright 2025 Nanyang Technological University (NTU), Singapore +# and the verl-agent (GiGPO) team. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import re +from typing import List, Optional + + +# --------------------- WebShop --------------------- # +WEBSHOP_TEMPLATE_NO_HIS = """ +You are an expert autonomous agent operating in the WebShop e-commerce environment. +Your task is to: {task_description}. +Your current observation is: {current_observation}. +Your admissible actions of the current situation are: +[ +{available_actions} +]. + +Now it's your turn to take one action for the current step. +You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" + +WEBSHOP_TEMPLATE = """ +You are an expert autonomous agent operating in the WebShop e-commerce environment. +Your task is to: {task_description}. +Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history} +You are now at step {current_step} and your current observation is: {current_observation}. +Your admissible actions of the current situation are: +[ +{available_actions} +]. + +Now it's your turn to take one action for the current step. +You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" + +HISTORY_LENGTH = 2 +MEMORY_FORMAT = "[Observation {step_num}: '{obs}', Action {step_num}: '{act}']" + + +def parse_action(response: str) -> str: + try: + return response.split("")[1].split("")[0].strip() + except Exception as e: + print(f"Error parsing action: {e}, response = {response}") + return "" + + +def validate_action(action: str, available_actions: dict): + pattern = re.compile(r"(.+)\[(.+)\]") + match = re.match(pattern, action) + if match is None: + action_name = action + action_arg = None + else: + action_name, action_arg = match.groups() + + if action_arg is not None: + action_arg = action_arg.lower() + + if ( + action_name == "search" + and action_arg is not None + and action_arg != "" + and available_actions.get("has_search_bar", False) + ): + return True, "" + if action_name == "click" and action_arg in available_actions.get("clickables", []): + return True, "" + + if action_name == "search": + if action_arg == "" or action_arg is None: + return ( + False, + "Invalid action, please type in the query you want to search in the square brackets here.", + ) + return ( + False, + "Can not perform search action without search bar. Please click the Back to Search button first.", + ) + + if action_name == "click" and action_arg not in available_actions.get("clickables", []): + return ( + False, + "Incorrect action format, make sure you have the correct button name that is " + f"within the current page. The buttons you can click now are: {available_actions.get('clickables', [])}", + ) + + return ( + False, + "Invalid action. You should wrap your action with the tag and follow the action format.", + ) + + +def format_observation(observation: str) -> str: + return observation.strip() + + +def _compact_text(text: str) -> str: + return " ".join(line.strip() for line in text.splitlines() if line.strip()) + + +def _extract_task_description(observation: str) -> str: + """Extract the instruction text from a WebShop observation.""" + lines = [line.strip() for line in observation.splitlines()] + try: + instruction_idx = lines.index("Instruction:") + except ValueError: + return _compact_text(observation) + + task_lines = [] + for line in lines[instruction_idx + 1 :]: + if not line: + continue + if line.startswith("[button]"): + break + task_lines.append(line) + + return " ".join(task_lines).strip() or _compact_text(observation) + + +def _format_history(observation: str, step_num: int, action: str) -> str: + return MEMORY_FORMAT.format( + step_num=step_num, + obs=_compact_text(observation), + act=action, + ) + + +def _format_available_actions(available_actions: dict) -> str: + formatted_actions = [] + if available_actions.get("has_search_bar", False): + formatted_actions.append("search[]") + for clickable in available_actions.get("clickables", []): + formatted_actions.append(f"click[{clickable}]") + return "\n".join(formatted_actions) + + +def _create_webshop_env(): + try: + import gym # type: ignore[import-not-found] + from web_agent_site.envs import WebAgentTextEnv # type: ignore[import-not-found] # noqa: F401 + except Exception as e: + raise ImportError( + f"Error importing WebAgentTextEnv {e}. " + "Please make sure you have installed the web_agent_site package, " + "following the instructions in https://github.com/princeton-nlp/WebShop" + ) from e + + return gym.make( + "WebAgentTextEnv-v0", + observation_mode="text_rich", + num_products=None, + human_goals=True, + ) + + +def _create_webshop_env_with_checkpoint( + session_id: int, + actions: List[str], + checkpoint_step: int, +): + """Create WebShop environment and execute actions up to checkpoint_step.""" + env = _create_webshop_env() + env.reset(session=session_id) + observation = env.observation + history: List[str] = [] + task_description = _extract_task_description(observation) + done = False + reward = -0.1 + + for step in range(min(checkpoint_step, len(actions))): + action = actions[step] + formatted_observation = format_observation(observation) + history.append(_format_history(formatted_observation, step + 1, action)) + observation, reward, done, _ = env.step(action) + if done: + break + + return env, observation, history, task_description, len(history), done, reward \ No newline at end of file diff --git a/frameworks/trinity/common/workflows/envs/agentscope/agentscopev0_react_workflow.py b/frameworks/trinity/common/workflows/envs/agentscope/agentscopev0_react_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..6b55bd588d45f85e44881ce3236eb703a16fcb22 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/agentscope/agentscopev0_react_workflow.py @@ -0,0 +1,150 @@ +# -*- coding: utf-8 -*- +"""We include the agentscope react workflows in this file.""" + +from typing import List, Optional + +from trinity.common.models.model import ModelWrapper +from trinity.common.rewards.math_reward import MathBoxedRewardFn +from trinity.common.workflows.workflow import Task, Workflow +from trinity.utils.annotations import Deprecated + + +@Deprecated +class AgentScopeV0ReactMathWorkflow(Workflow): + """ + This workflow serves as an example of how to use the agentscope framework within the trinity workflow. + We use the AgentScope V0 version here. The code will be deprecated soon. + """ + + can_reset: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + # make sure that we have the correct import + try: + import agentscope + from agentscope.service import ServiceToolkit, execute_python_code + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + + # get openai client from model + self.openai_client = model.get_openai_client() + self.model_name = self.openai_client.model_path + + agentscope.init( + model_configs=[ + { + "model_type": "openai_chat", + "config_name": "my_model", + "model_name": self.model_name, + "api_key": "EMPTY", + "generate_args": { + "temperature": self.task.rollout_args.temperature, + "max_tokens": self.task.rollout_args.max_tokens or 4096, + }, + "use_openai_formatter": True, + } + ], + disable_saving=True, + ) + self.toolkit = ServiceToolkit() + self.toolkit.add( + execute_python_code, + timeout=300, + use_docker=False, + maximum_memory_bytes=None, + ) + self.reset(task) + + def reset(self, task: Task): + self.system_prompt = """ +You are an agent specialized in solving math problems with tools. Please solve the math problem given to you. You can write and execute Python code to perform calculation or verify your answer. You should return your final answer within \\boxed{{}}. +""" + try: + from agentscope.agents import ReActAgentV2 + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + self.agent = ReActAgentV2( + name="math_react_agent", + sys_prompt=self.system_prompt, + model_config_name="my_model", # replace by your model config name + service_toolkit=self.toolkit, + max_iters=5, + verbose=False, + ) + # we set the openai client to the agent's model + self.agent.model.client = self.openai_client + + self.raw_task = task.raw_task + self.task_desc = task.task_desc + self.truth = task.truth + + # we get the answer from gsm8k dataset + try: + if isinstance(self.truth, str) and "####" in self.truth: + # GSM8K dataset + self.answer = self.truth.split("####")[1].strip() + else: + self.answer = str(self.truth) + except Exception as e: + self.logger.debug(f"Error in getting answer from truth: {str(e)}") + self.answer = str(self.truth) + + # we use the boxed format to evaluate the answer + self.reward_fn = MathBoxedRewardFn() + + def run(self): + # make sure that we have the correct import + try: + from agentscope.message import Msg + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + + # provide the task to the react agent + msg = Msg("user", self.task_desc, role="user") + # Note that the main workflow can have arbitrary steps and include different logic + content = self.agent.reply(msg).content + + # unify the response format to text + try: + if isinstance(content, list): + response_text = content[0]["text"] + else: + response_text = content + except Exception as e: + error_message = f"Error in processing the response: {e}" + self.logger.info(error_message) + response_text = str(content) + + reward = self.reward_fn(response_text, self.answer) + reward = sum(reward.values()) + self.logger.debug(f"Reward: {reward}") + experiences = self.model.extract_experience_from_history(clear_history=True) + self.logger.debug(f"Experiences extracted len: {len(experiences)}") + for i, experience in enumerate(experiences): + experience.eid.step = i + experience.reward = reward + agent_metrics = {"react_memory_length": len(self.agent.memory.get_memory())} + if experience.metrics is None: + experience.metrics = {} + experience.metrics.update(agent_metrics) + self.logger.debug( + f"return experience len: {len(experiences)}, run_id: {str(experiences[-1].eid.run)}, final step reward: {experiences[-1].reward}" + ) + return experiences diff --git a/frameworks/trinity/common/workflows/envs/agentscope/agentscopev1_react_workflow.py b/frameworks/trinity/common/workflows/envs/agentscope/agentscopev1_react_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..90ca068e604de94c67c8f94e6559dc8ecbe52193 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/agentscope/agentscopev1_react_workflow.py @@ -0,0 +1,157 @@ +# -*- coding: utf-8 -*- +"""We include the customized math workflows in this file.""" + +from typing import List, Optional + +from trinity.common.models.model import ModelWrapper +from trinity.common.rewards.math_reward import MathBoxedRewardFn +from trinity.common.workflows.workflow import Task, Workflow + + +class AgentScopeReactMathWorkflow(Workflow): + """ + This workflow serves as an example of how to use the agentscope framework within the trinity workflow. + We use the AgentScope V1 version here. + """ + + can_reset: bool = True + is_async: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + # make sure that we have the correct import + try: + from agentscope.formatter import OpenAIChatFormatter + from agentscope.model import OpenAIChatModel + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + + # get openai client from model + self.openai_async_client = model.get_openai_async_client() + self.model_name = self.openai_async_client.model_path + + self.agent_model = OpenAIChatModel( + api_key="EMPTY", + model_name=self.model_name, + stream=False, + generate_kwargs={ + "temperature": self.task.rollout_args.temperature, + "max_tokens": self.task.rollout_args.max_tokens or 4096, + }, + ) + self.agent_model.client = self.openai_async_client + self.agent_model_formatter = OpenAIChatFormatter() + self.reset(task) + + def reset(self, task: Task): + self.system_prompt = """ +You are an agent specialized in solving math problems with tools. Please solve the math problem given to you. You can write and execute Python code to perform calculation or verify your answer. You should return your final answer within \\boxed{{}}. +""" + try: + from agentscope.agent import ReActAgent + from agentscope.memory import InMemoryMemory + from agentscope.tool import Toolkit, execute_python_code + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + self.toolkit = Toolkit() + self.toolkit.register_tool_function(execute_python_code) + self.agent = ReActAgent( + name="math_react_agent", + sys_prompt=self.system_prompt, + model=self.agent_model, + formatter=self.agent_model_formatter, + toolkit=self.toolkit, + memory=InMemoryMemory(), + ) + # we set the openai client to the agent's model + self.agent.model.client = self.openai_async_client + + self.raw_task = task.raw_task + self.task_desc = task.task_desc + self.truth = task.truth + + # we get the answer from gsm8k dataset + try: + if isinstance(self.truth, str) and "####" in self.truth: + # GSM8K dataset + self.answer = self.truth.split("####")[1].strip() + else: + self.answer = str(self.truth) + except Exception as e: + self.logger.debug(f"Error in getting answer from truth: {str(e)}") + self.answer = str(self.truth) + + # we use the boxed format to evaluate the answer + self.reward_fn = MathBoxedRewardFn() + + async def run_async(self): + # make sure that we have the correct import + try: + from agentscope.message import Msg + from pydantic import BaseModel, Field + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + + # provide the task to the react agent + msg = Msg("user", self.task_desc, role="user") + + # Note that the main workflow can have arbitrary steps and include different logic + class FinalResult(BaseModel): + result: str = Field( + description="Your solution of the given math problem. Put your final answer in boxed format, e.g., \\boxed{42}" + ) + + def extract_final_answer(result) -> str: + """Extract the final answer from the agent's response.""" + try: + if ( + hasattr(result, "metadata") + and isinstance(result.metadata, dict) + and "result" in result.metadata + ): + return result.metadata["result"] + if hasattr(result, "content"): + if isinstance(result.content, dict) and "result" in result.content: + return result.content["result"] + return str(result.content) + return str(result) + except Exception as e: + self.logger.warning(f"Extract final answer error: {e}. Raw: {result}") + return str(result) + + result = await self.agent.reply(msg, structured_model=FinalResult) + + final_answer = extract_final_answer(result) + + reward = self.reward_fn(final_answer, self.answer) + reward = sum(reward.values()) + self.logger.debug(f"Reward: {reward}") + experiences = self.model.extract_experience_from_history(clear_history=True) + self.logger.debug(f"Experiences extracted len: {len(experiences)}") + for i, experience in enumerate(experiences): + experience.eid.step = i + experience.reward = reward + agent_metrics = {"react_memory_length": len(self.agent.memory.content)} + if experience.metrics is None: + experience.metrics = {} + experience.metrics.update(agent_metrics) + self.logger.debug( + f"return experience len: {len(experiences)}, run_id: {str(experiences[-1].eid.run)}, final step reward: {experiences[-1].reward}" + ) + return experiences diff --git a/frameworks/trinity/common/workflows/envs/agentscope/agentscopev1_search_workflow.py b/frameworks/trinity/common/workflows/envs/agentscope/agentscopev1_search_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..c24e30d0ce10c809427d1382a0a3c2d236532fea --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/agentscope/agentscopev1_search_workflow.py @@ -0,0 +1,230 @@ +# -*- coding: utf-8 -*- +"""We include simple react deep search workflows in this file. We use AgentScope V1 framework.""" + +import os +import re +from typing import List, Optional + +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.workflow import Task, Workflow + + +class AgentScopeV1ReactSearchWorkflow(Workflow): + """ + This workflow serves as an example of how to use the agentscope framework within the trinity workflow. + """ + + can_reset: bool = True + is_async: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): # get openai client from model + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + # make sure that we have the correct import + try: + from agentscope.formatter import OpenAIChatFormatter + from agentscope.model import OpenAIChatModel + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + + # get openai client from model + self.openai_async_client = model.get_openai_async_client() + self.model_name = self.openai_async_client.model_path + + self.agent_model = OpenAIChatModel( + api_key="EMPTY", + model_name=self.model_name, + stream=False, + generate_kwargs={ + "temperature": self.task.rollout_args.temperature, + "max_tokens": self.task.rollout_args.max_tokens or 4096, + }, + ) + self.agent_model.client = self.openai_async_client + self.agent_model_formatter = OpenAIChatFormatter() + + self.reset(task) + + def reset(self, task: Task): + self.workflow_args = task.workflow_args + self.max_turns = int(self.workflow_args.get("max_turns", 10)) + self.search_client_type = self.workflow_args.get("search_client_type", "searxng") + self.max_model_tokens = int(self.workflow_args.get("max_model_tokens", 24000)) + if self.search_client_type not in ["searxng", "tavily"]: + raise ValueError( + f"search_client_type must be one of ['searxng', 'tavily'], but got {self.search_client_type}" + ) + self.system_prompt = "You are a Web Information Seeking Master. Your task is to thoroughly seek the internet for information and provide accurate answers to questions." + + self.raw_task = task.raw_task + self.task_desc = task.task_desc + self.truth = task.truth + + def judge_result(self, result, question, correct_answer, judge_model=None) -> bool: + """Use LLM to judge whether the answer is correct or not.""" + if result is None: + return False + + def extract_final_answer(result) -> str: + """Extract the final answer from the agent's response.""" + try: + if ( + hasattr(result, "metadata") + and isinstance(result.metadata, dict) + and "result" in result.metadata + ): + return result.metadata["result"] + if hasattr(result, "content"): + if isinstance(result.content, dict) and "result" in result.content: + return result.content["result"] + return str(result.content) + return str(result) + except Exception as e: + self.logger.warning(f"Extract final answer error: {e}. Raw: {result}") + return str(result) + + final_answer = extract_final_answer(result) + + judge_prompt = f"""Judge whether the following [response] to [question] is correct or not based on the [correct_answer] below. + +[question]: {question} + +[response]: {final_answer} + +[correct_answer]: {correct_answer} + +Your judgement must be in the format and criteria specified below: + +1. **Reasoning**: Explain why the [response] is correct or incorrect based on [correct_answer], focusing only on if there are meaningful differences between [correct_answer] and the response. Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than [correct_answer], focus only on whether the answers match. + +2. **Correctness**: Answer exactly "YES" if [response] matches the [correct_answer] given above, or is within a small margin of error for numerical problems and small format issue for text problems (for example, with or without a hyphen should be considered the same). Answer exactly "NO" otherwise, i.e. if there if there is any inconsistency, ambiguity, non-equivalency, or if the [response] is incorrect. +""" + messages = [ + {"role": "system", "content": "You evaluate correctness."}, + {"role": "user", "content": judge_prompt}, + ] + completion = judge_model.chat.completions.create( + model=judge_model.model_path, messages=messages, stream=False + ) + judge_output = completion.choices[0].message.content + + self.logger.info( + f"[judge_result] prompt:\n{judge_prompt}\n\n[judge_result] LLM output:\n{judge_output}" + ) + + # Yes if the response is correct, No otherwise + match = re.search(r"Correctness.*?YES", judge_output, re.IGNORECASE) + return match is not None + + async def run_async(self): + try: + from agentscope.agent import ReActAgent + from agentscope.mcp import StdIOStatefulClient + from agentscope.memory import InMemoryMemory + from agentscope.message import Msg + from pydantic import BaseModel, Field + except ImportError as e: + error_message = f"AgentScope V1 is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + + self.agent = ReActAgent( + name="Friday", + sys_prompt=self.system_prompt, + model=self.agent_model, + formatter=self.agent_model_formatter, + memory=InMemoryMemory(), + max_iters=self.max_turns, + ) + self.agent.model.client = self.openai_async_client + + if self.search_client_type == "tavily": + tavily_api_key = os.getenv("TAVILY_API_KEY", "") + if not tavily_api_key: + raise ValueError( + "TAVILY_API_KEY environment variable is not set. Please set it to use the Tavily search tool." + ) + + self.search_client = StdIOStatefulClient( + name="tavily_mcp", + command="npx", + args=["-y", "tavily-mcp@latest"], + env={"TAVILY_API_KEY": tavily_api_key}, + ) + elif self.search_client_type == "searxng": + searxng_url = os.getenv("SEARXNG_URL", "") + if not searxng_url: + raise ValueError( + "SEARXNG_URL environment variable is not set. Please set it to use the SearXNG search tool." + ) + self.search_client = StdIOStatefulClient( # refer to https://github.com/ihor-sokoliuk/mcp-searxng for more details + name="searxng_mcp", + command="npx", + args=["-y", "mcp-searxng"], + env={"SEARXNG_URL": searxng_url}, + ) + else: + raise ValueError( + f"search_client_type must be one of ['searxng', 'tavily'], but got {self.search_client_type}" + ) + + instruction = Msg("user", content=self.task_desc, role="user") + + class FinalResult(BaseModel): + result: str = Field(description="The final result to the initial user query") + + try: + await self.search_client.connect() + await self.agent.toolkit.register_mcp_client(self.search_client) + result = await self.agent.reply(instruction, structured_model=FinalResult) + except Exception as e: + self.logger.error(f"Error during agent reply: {e}") + result = None + finally: + if self.search_client and self.search_client.is_connected: + await self.search_client.close() + + # Reward calculation (judge_result can stay sync if your judge_model only has sync chat, otherwise you need to make it async) + try: + judge_model = self.auxiliary_models[0] if self.auxiliary_models else None + assert judge_model is not None, "Please provide a judge model for reward calculation." + reward = 1 if self.judge_result(result, self.task_desc, self.truth, judge_model) else 0 + except Exception as e: + self.logger.error(f"Error in judge_model judging: {e}") + reward = 0 + + self.logger.debug(f"Reward: {reward}") + experiences = self.model.extract_experience_from_history(clear_history=True) + return_experiences = [] + self.logger.debug(f"Experiences extracted len: {len(experiences)}") + for i, experience in enumerate(experiences): + experience.eid.step = i + experience.reward = reward + agent_metrics = { + "react_turns": len(self.agent.memory.content) // 2, + "max_turns": self.max_turns, + } + if experience.metrics is None: + experience.metrics = {} + experience.metrics.update(agent_metrics) + if len(experience.tokens) > self.max_model_tokens: + continue + return_experiences.append(experience) + if return_experiences: + self.logger.debug( + f"return experience len: {len(return_experiences)}, run_id: {str(return_experiences[-1].eid.run)}, final step reward: {return_experiences[-1].reward}" + ) + else: + self.logger.info("No valid experiences to return (all filtered out).") + return return_experiences diff --git a/frameworks/trinity/common/workflows/envs/alfworld/RAFT_alfworld_workflow.py b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_alfworld_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..4bc34833cb836f5a483067c5dc956c6ea4e68af9 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_alfworld_workflow.py @@ -0,0 +1,224 @@ +# -*- coding: utf-8 -*- +from datetime import datetime +from typing import Dict, List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.envs.alfworld.RAFT_utils import ( + create_alfworld_environment, + format_observation, + generate_default_empty_experience, + get_jinja_env, + parse_response, + process_messages_to_experience_async, + validate_trajectory_format, +) +from trinity.common.workflows.workflow import Task, Workflow + + +class RAFTAlfworldWorkflow(Workflow): + """ + RAFT workflow for alfworld using trajectory context. + + Process: + 1. First exploration with normal experience generation + 2. Generate SFT data from successful attempt + """ + + can_reset: bool = True + can_repeat: bool = True + is_async: bool = True + + def __init__( + self, + model: ModelWrapper, + task: Task, + auxiliary_models: Optional[List] = None, + ): + super().__init__( + model=model, + task=task, + auxiliary_models=auxiliary_models, + ) + # Initialize workflow parameters + self.temperature = getattr(task.rollout_args, "temperature", 1.0) + self.top_k = getattr(task.rollout_args, "top_k", 20) + self.top_p = getattr(task.rollout_args, "top_p", 0.95) + self.max_env_steps = 50 + self.max_tokens = 4096 + self.task = task + self.is_eval = task.is_eval + + # Setup Jinja2 templates + self.jinja_env = get_jinja_env() + self.alfworld_system_template = self.jinja_env.get_template("alfworld_system.j2") + + print( + f"Initializing RAFTAlfworldWorkflow with RAFT learning, temperature={self.temperature}" + ) + self.reset(task) + + def reset(self, task: Task): + """Reset the workflow with a new task""" + self.game_file_path = task.task_desc or task.raw_task.get("game_file", "") + self.is_eval = task.is_eval + + def create_environment(self, game_file): + """Create alfworld environment""" + return create_alfworld_environment(game_file) + + async def run_single_rollout( + self, env + ) -> tuple[List[Dict[str, str]], float, bool, int, List[Dict[str, str]]]: + """Run a single rollout with RAFT-guided actions""" + observation, info = env.reset() + trajectory = [] + parsed_steps = [] # Store parsed experience, think, action for each step + action_history = [] # Track last 3 actions for repetition detection + + trajectory.append({"role": "system", "content": self.alfworld_system_template.render()}) + + # Track the last reward from environment + last_reward = 0.0 + + for step in range(self.max_env_steps): + trajectory.append({"role": "user", "content": format_observation(observation)}) + + # Get model response with RAFT guidance + responses = await self.model.chat_async( + trajectory, + n=1, + temperature=self.temperature, + top_k=self.top_k, + top_p=self.top_p, + max_tokens=self.max_tokens, + ) + response_text = responses[0].response_text.strip() + trajectory.append({"role": "assistant", "content": response_text}) + + # Parse the three components + parsed = parse_response(response_text) + experience_text, think_text, action_text = ( + parsed["experience"], + parsed["think"], + parsed["action"], + ) + + # Store parsed step for SFT data construction + parsed_steps.append( + { + "observation": observation, + "experience": experience_text, + "think": think_text, + "action": action_text, + "full_response": response_text, + } + ) + + # Check for consecutive action repetition + action_history.append(action_text) + if len(action_history) > 3: + action_history.pop(0) + + # If last 3 actions are the same, terminate with failure + if len(action_history) >= 3 and all( + action == action_history[0] for action in action_history + ): + print(f"Terminating due to 3 consecutive identical actions: {action_text}") + return trajectory, 0.0, False, step + 1, parsed_steps + + # Execute action in environment + observation, reward, done, info = env.step(action_text) + last_reward = reward # Always track the latest reward from environment + + if done: + return trajectory, reward, done, step + 1, parsed_steps + + # If timeout, return the last reward from environment instead of fixed value + return trajectory, last_reward, False, self.max_env_steps, parsed_steps + + async def _execute_first_attempt(self) -> tuple: + """Execute the first attempt and return results""" + env = self.create_environment(self.game_file_path) + + try: + trajectory, reward, done, steps, parsed_steps = await self.run_single_rollout(env) + except Exception as e: + print(f"Single rollout failed: {e}") + env.close() + raise e + + env.close() + success = done and reward >= 1 + traj_format_valid = validate_trajectory_format(parsed_steps) + + return trajectory, reward, done, steps, parsed_steps, success, traj_format_valid + + async def eval_alfworld(self) -> List[Experience]: + """Evaluate a single alfworld trajectory""" + env = self.create_environment(self.game_file_path) + try: + trajectory, reward, done, steps, parsed_steps = await self.run_single_rollout(env) + except Exception as e: + print(f"Single rollout failed during eval: {e}") + env.close() + return [generate_default_empty_experience(f"Eval rollout failed: {str(e)}")] + env.close() + + # Save eval data + task_id = datetime.now().strftime("%Y%m%d_%H%M%S_%f") + success = done and reward >= 1 + + # Convert trajectory to experience + experience = generate_default_empty_experience( + msg="Eval completed successfully", + info={"task_id": task_id, "success": success, "reward": reward, "steps": steps}, + metrics={"success": float(success), "reward": float(reward), "steps": float(steps)}, + ) + + return [experience] + + async def run_async(self) -> List[Experience]: + """Run the RAFT alfworld workflow and return experiences""" + + if self.is_eval: + return await self.eval_alfworld() + + # Execute first attempt + try: + ( + trajectory, + reward, + done, + steps, + parsed_steps, + success, + traj_format_valid, + ) = await self._execute_first_attempt() + except Exception as e: + return [generate_default_empty_experience(f"Training rollout failed: {str(e)}")] + + print(f"Task result: done={done}, reward={reward:.3f}, steps={steps}, success={success}") + + if reward >= 1 and traj_format_valid: + print("✅ Task completed successfully in the first attempt!") + experience = await process_messages_to_experience_async( + self.model, trajectory, info={"success": success, "reward": reward, "steps": steps} + ) + return [experience] + elif not traj_format_valid and reward >= 1: + print( + "❌ Task completed but trajectory format is invalid, skipping SFT data generation." + ) + else: + print("❌ Task failed.") + + experience = generate_default_empty_experience( + "Experience conversion failed: Trajectory format invalid", + metrics={"success": float(success), "reward": float(reward), "steps": float(steps)}, + ) + return [experience] + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.run_id_base = run_id_base diff --git a/frameworks/trinity/common/workflows/envs/alfworld/RAFT_prompt/alfworld_system.j2 b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_prompt/alfworld_system.j2 new file mode 100644 index 0000000000000000000000000000000000000000..d60ba7f62261503d8932fe9530cc72e1cf0ff468 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_prompt/alfworld_system.j2 @@ -0,0 +1,29 @@ +You are an agent interacting with a virtual text-based environment. + +## Response Format: +You MUST use this exact format for every response. All three tags are REQUIRED in sequential order: + +Working principles, strategies, common knowledge, and potential pitfalls relevant to the current task.\n\n +your reasoning process\n\n +exactly one action command + +## Action Commands: + look: look around your current location + inventory: check your current inventory(you can only have 1 item in your inventory) + go to (receptacle): move to a receptacle + open (receptacle): open a receptacle + close (receptacle): close a receptacle + take (object) from (receptacle): take an object from a receptacle + move (object) to (receptacle): place an object in or on a receptacle + examine (something): examine a receptacle or an object + use (object): use an object + heat (object) with (receptacle): heat an object using a receptacle + clean (object) with (receptacle): clean an object using a receptacle + cool (object) with (receptacle): cool an object using a receptacle + slice (object) with (object): slice an object using a sharp object + +For example your output should be like this: +In household tasks, I should start by exploring the environment to understand available objects and receptacles. Common pitfalls include forgetting to check inventory capacity and not examining objects before taking action.\n\n To solve the task, I need first to ... \n\ngo to cabinet 1 + +## Important Note: +You must ensure that the section contains descriptions of working principles, strategies, common sense knowledge, and potential pitfalls that are universally applicable to this type of task, rather than generic statements, placeholder content, or overly specific behavioral guidelines. diff --git a/frameworks/trinity/common/workflows/envs/alfworld/RAFT_prompt/second_attempt_guidance.j2 b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_prompt/second_attempt_guidance.j2 new file mode 100644 index 0000000000000000000000000000000000000000..1945cf314b2a1118c53a6e08edc1482126de6699 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_prompt/second_attempt_guidance.j2 @@ -0,0 +1,3 @@ +{{ reward_feedback }} + +This is your second attempt. You need to perform better this time. Focus on updating a more thoughtful and comprehensive ... section that incorporates strategies, common pitfalls, and domain knowledge to help improve task performance and avoid previous mistakes. Learn from your first attempt and apply that knowledge to make better decisions. IMPORTANT: You must update your experience based on what you learned from the first attempt. Also, DO NOT repeat the same action consecutively as this is inefficient and useless - if you find yourself repeating actions, stop and try a different approach. In your responses, do not explicitly mention this is a second attempt or reference previous attempts. Instead, present your actions as if starting fresh, but incorporate general experience and strategies you can summarize for this type of task. diff --git a/frameworks/trinity/common/workflows/envs/alfworld/RAFT_reflect_alfworld_workflow.py b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_reflect_alfworld_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..be2889a2b5727c2cc36c691f52f7a5df2d6b2bbd --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_reflect_alfworld_workflow.py @@ -0,0 +1,310 @@ +# -*- coding: utf-8 -*- +import os +from datetime import datetime +from typing import Any, Dict, List, Optional, Union + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.envs.alfworld.RAFT_alfworld_workflow import ( + RAFTAlfworldWorkflow, +) +from trinity.common.workflows.envs.alfworld.RAFT_utils import ( + create_alfworld_environment, + format_observation, + generate_default_empty_experience, + generate_reward_feedback, + parse_response, + process_messages_to_experience_async, + save_task_data, + validate_trajectory_format, +) +from trinity.common.workflows.workflow import Task + + +class RAFTReflectAlfworldWorkflow(RAFTAlfworldWorkflow): + """ + RAFT workflow for alfworld using trajectory context. + + Process: + 1. First exploration with normal experience generation + 2. If failed, re-explore with first trajectory as context + 3. Generate SFT data from successful attempt + """ + + def __init__( + self, + model: ModelWrapper, + task: Task, + auxiliary_models: Optional[List] = None, + ): + super().__init__( + model=model, + task=task, + auxiliary_models=auxiliary_models, + ) + + # Create data directories specific to reflect workflow + self.data_dir = "RAFT_reflect_alfworld_data" + self.sft_dir = os.path.join(self.data_dir, "sft_data") + self.non_sft_dir = os.path.join(self.data_dir, "non_sft_data") + + os.makedirs(self.sft_dir, exist_ok=True) + os.makedirs(self.non_sft_dir, exist_ok=True) + + # Setup additional template for second attempt + self.second_attempt_template = self.jinja_env.get_template("second_attempt_guidance.j2") + + print( + f"Initializing RAFTReflectAlfworldWorkflow with RAFT learning, temperature={self.temperature}" + ) + + async def construct_sft_data( + self, + first_trajectory: List[Dict[str, str]], + success: bool, + reward: float, + original_steps: int, + ) -> tuple[List[Dict[str, str]], Dict[str, Any], List[Dict[str, str]]]: + """Generate SFT training data using RAFT learning""" + + # Always perform second attempt with first trajectory as context + ( + new_trajectory, + new_reward, + new_success, + new_steps, + new_parsed_steps, + ) = await self.re_explore_with_context(first_trajectory, reward, success, original_steps) + + # Consider improvement if reward is higher OR same reward with fewer steps + reward_improved = new_reward > reward + efficiency_improved = new_steps < original_steps + + return ( + new_trajectory, + { + "new_reward": new_reward, + "new_steps": new_steps, + "reward_improved": reward_improved, + "efficiency_improved": efficiency_improved, + }, + new_parsed_steps, + ) + + async def re_explore_with_context( + self, + first_trajectory: List[Dict[str, str]], + original_reward: float, + original_success: bool, + original_steps: int, + ) -> tuple[List[Dict[str, str]], float, bool, int, List[Dict[str, str]]]: + """Re-explore with first trajectory as context""" + + env = create_alfworld_environment(self.game_file_path) + + observation, info = env.reset() + + # Use first trajectory as context for generation + context_messages = first_trajectory.copy() + + # Add reward feedback about first attempt + reward_feedback = generate_reward_feedback( + original_reward, original_steps, original_success, self.max_env_steps + ) + context_messages.append( + { + "role": "system", + "content": self.second_attempt_template.render(reward_feedback=reward_feedback), + } + ) + + # Build clean SFT trajectory (like first trajectory format) + sft_trajectory = [{"role": "system", "content": self.alfworld_system_template.render()}] + parsed_steps = [] # Track parsed steps for quality analysis + + for step in range(self.max_env_steps): + # Add to context for generation + context_messages.append({"role": "user", "content": format_observation(observation)}) + + # Add to clean SFT trajectory + sft_trajectory.append({"role": "user", "content": format_observation(observation)}) + + responses = await self.model.chat_async( + context_messages, + n=1, + temperature=self.temperature, + top_k=self.top_k, + top_p=self.top_p, + max_tokens=self.max_tokens, + ) + + response_text = responses[0].response_text.strip() + + # Parse components for quality analysis + parsed = parse_response(response_text) + experience_text, think_text, action_text = ( + parsed["experience"], + parsed["think"], + parsed["action"], + ) + + parsed_steps.append( + { + "observation": observation, + "experience": experience_text, + "think": think_text, + "action": action_text, + "full_response": response_text, + } + ) + + # Add to both trajectories + context_messages.append({"role": "assistant", "content": response_text}) + sft_trajectory.append({"role": "assistant", "content": response_text}) + + observation, reward, done, info = env.step(action_text) + + if done: + env.close() + return sft_trajectory, reward, done and reward > 0, step + 1, parsed_steps + + env.close() + return sft_trajectory, reward, False, self.max_env_steps, parsed_steps + + def _handle_invalid_format_success( + self, success: bool, reward: float, steps: int + ) -> List[Experience]: + """Handle case where task succeeded but format is invalid""" + print("❌ Task completed but trajectory format is invalid, skipping SFT data generation.") + experience = generate_default_empty_experience( + "Experience conversion failed: Trajectory format invalid", + metrics={"success": float(success), "reward": float(reward), "steps": float(steps)}, + ) + return [experience] + + async def _execute_second_attempt( + self, trajectory: list, success: bool, reward: float, steps: int + ) -> Union[tuple[List[Dict[str, str]], Dict[str, Any], List[Dict[str, str]],], Exception]: + """Execute second attempt and return SFT data""" + try: + sft_messages, re_explore_info, new_parsed_steps = await self.construct_sft_data( + trajectory, success, reward, steps + ) + return sft_messages, re_explore_info, new_parsed_steps + except Exception as e: + print(f"SFT data construction failed: {e}") + return e + + def _build_metrics( + self, reward: float, steps: int, new_parsed_steps: list, re_explore_info: dict + ) -> dict: + """Build metrics for tracking""" + return { + "reward": float(reward), + "steps": float(steps), + "trajectory_length": len(new_parsed_steps), + "second_reward": float(re_explore_info["new_reward"]), + "second_steps": float(re_explore_info["new_steps"]), + "improvement": 1.0 if re_explore_info["reward_improved"] else 0.0, + } + + def _should_keep_for_sft(self, second_traj_format_valid: bool, re_explore_info: dict) -> bool: + """Determine if trajectory should be kept for SFT""" + return second_traj_format_valid and ( + re_explore_info["reward_improved"] + or (re_explore_info["efficiency_improved"] and re_explore_info["new_reward"] >= 1.0) + ) + + async def _generate_experience_from_sft(self, sft_messages: list, metrics: dict) -> Experience: + """Generate experience from SFT messages""" + return await process_messages_to_experience_async(self.model, sft_messages, info=metrics) + + async def run_async(self) -> List[Experience]: + """Run the RAFT alfworld workflow and return experiences""" + + if self.is_eval: + return await self.eval_alfworld() + + # Generate unique task ID using timestamp + task_id = datetime.now().strftime("%Y%m%d_%H%M%S_%f") + + # Execute first attempt + try: + ( + trajectory, + reward, + done, + steps, + parsed_steps, + success, + traj_format_valid, + ) = await self._execute_first_attempt() + except Exception as e: + return [generate_default_empty_experience(f"Training rollout failed: {str(e)}")] + + # Handle first attempt success cases + if reward >= 1 and traj_format_valid: + print("✅ Task completed successfully in the first attempt!") + experience = await process_messages_to_experience_async( + self.model, trajectory, info={"success": success, "reward": reward, "steps": steps} + ) + return [experience] + elif not traj_format_valid and reward >= 1: + return self._handle_invalid_format_success(success, reward, steps) + + print(f"Task result: done={done}, reward={reward:.3f}, steps={steps}, success={success}") + + # Execute second attempt + ret = await self._execute_second_attempt(trajectory, success, reward, steps) + if isinstance(ret, Exception): + error = ret + default_experience = generate_default_empty_experience( + f"SFT data construction failed: {str(error)}", + ) + return [default_experience] + else: + sft_messages, re_explore_info, new_parsed_steps = ret + + # Validate second attempt and build metrics + second_success = re_explore_info["new_reward"] >= 1 + second_traj_format_valid = validate_trajectory_format(new_parsed_steps) + metrics = self._build_metrics(reward, steps, new_parsed_steps, re_explore_info) + + # Generate experience if conditions are met + experiences = [] + kept_for_sft = self._should_keep_for_sft(second_traj_format_valid, re_explore_info) + + if kept_for_sft: + experience = await self._generate_experience_from_sft(sft_messages, metrics) + experiences.append(experience) + print( + f"✅ Generated good training data: orig={reward}, steps={steps}, new={re_explore_info['new_reward']}, new_steps={re_explore_info['new_steps']}" + ) + else: + print( + f"❌ Filtered trajectory: orig={reward}, steps={steps}, new={re_explore_info['new_reward']}, new_steps={re_explore_info['new_steps']}, second_traj_format_valid: {second_traj_format_valid}" + ) + + # Save detailed task data + save_task_data( + game_file_path=self.game_file_path, + sft_dir=self.sft_dir, + non_sft_dir=self.non_sft_dir, + task_id=task_id, + first_trajectory=trajectory, + first_reward=reward, + first_steps=steps, + first_success=success, + second_trajectory=sft_messages, + second_reward=re_explore_info["new_reward"], + second_steps=re_explore_info["new_steps"], + second_success=second_success, + kept_for_sft=kept_for_sft, + training_data=sft_messages, + ) + + # Return default experience if no valid experience generated + if not experiences: + experiences.append(generate_default_empty_experience()) + + return experiences diff --git a/frameworks/trinity/common/workflows/envs/alfworld/RAFT_utils.py b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..5e57ba597a061a0ed107df0d6a01c5a7e2bb40a7 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/RAFT_utils.py @@ -0,0 +1,196 @@ +# -*- coding: utf-8 -*- +import json +import os +import re +from datetime import datetime +from typing import Dict, List, Optional + +import torch +from jinja2 import Environment, FileSystemLoader + +from trinity.common.experience import Experience + + +# Setup Jinja2 environment for prompt templates +def get_jinja_env(): + """Get Jinja2 environment for loading templates""" + prompt_dir = os.path.join(os.path.dirname(__file__), "RAFT_prompt") + return Environment(loader=FileSystemLoader(prompt_dir)) + + +def format_observation(observation: str): + """Format observation string with additional guidance""" + if "Nothing happens." in observation: + observation += "Please check if the action you take is valid or you have carefully followed the action format." + return "Observation: " + observation + + +def parse_response(response): + """Parse all three components from response with a single regex""" + try: + # Use single regex to extract all three components at once + pattern = r"\s*(.*?)\s*.*?\s*(.*?)\s*.*?\s*(.*?)\s*" + match = re.search(pattern, response, re.DOTALL) + + if match: + return { + "experience": match.group(1).strip(), + "think": match.group(2).strip(), + "action": match.group(3).strip(), + } + else: + return {"experience": "", "think": "", "action": ""} + except Exception as e: + print(f"Error parsing response: {e}") + return {"experience": "", "think": "", "action": ""} + + +def validate_response_format(parsed: Dict[str, str]) -> bool: + """Validate if parsed response has valid content in all required fields""" + has_think = len(parsed["think"].strip()) > 0 + has_experience = len(parsed["experience"].strip()) > 0 + has_action = len(parsed["action"].strip()) > 0 + + return has_think and has_experience and has_action + + +def validate_trajectory_format(parsed_steps: List[Dict[str, str]]) -> bool: + """Validate if all steps in a trajectory have valid format""" + for step in parsed_steps: + if not validate_response_format(step): + return False + return True + + +def generate_default_empty_experience( + msg: str = "Unknown error", info=None, metrics=None +) -> Experience: + """Generate a default empty experience when errors occur""" + if info is None: + info = {"error_reason": msg, "rollout_failed": True} + if metrics is None: + metrics = {"success": 0.0, "reward": 0.0} + + return Experience( + tokens=torch.tensor([0], dtype=torch.long), + prompt_length=0, + action_mask=torch.tensor([False], dtype=torch.bool), + info=info, + metrics=metrics, + ) + + +def create_alfworld_environment(game_file): + """Create alfworld environment""" + try: + import textworld + import textworld.gym + from alfworld.agents.environment.alfred_tw_env import ( + AlfredDemangler, + AlfredExpert, + AlfredExpertType, + ) + + expert = AlfredExpert(expert_type=AlfredExpertType.HANDCODED) + request_infos = textworld.EnvInfos( + description=True, inventory=True, admissible_commands=True + ) + + env_id = textworld.gym.register_game( + game_file, request_infos, wrappers=[AlfredDemangler(), expert] + ) + env = textworld.gym.make(env_id) + return env + + except Exception as e: + error_message = f"Error importing AlfworldTWEnv {str(e)}. Please make sure you have installed the alfworld package successfully, following the instructions in https://github.com/alfworld/alfworld" + raise ImportError(error_message) + + +async def process_messages_to_experience_async(model, messages, info=None) -> Experience: + """Convert messages to experience for training, with fallback to default empty experience""" + if info is None: + info = {} + + try: + converted_experience = await model.convert_messages_to_experience_async(messages) + + metrics = {} + for k, v in info.items(): + if isinstance(v, float) or isinstance(v, int): + metrics[k] = float(v) + converted_experience.info = info + converted_experience.metrics = metrics + + return converted_experience + except Exception as e: + print(f"Failed to convert messages to experience: {e}") + return generate_default_empty_experience( + f"Experience conversion failed: {str(e)}", + info=info, + metrics={k: float(v) for k, v in info.items() if isinstance(v, (float, int))}, + ) + + +def generate_reward_feedback(reward: float, steps: int, done: bool, max_env_steps: int) -> str: + """Generate natural language feedback about the attempt's performance""" + if done and reward >= 1: + return f"In your attempt, you successfully completed the task in {steps} steps with a reward of {reward:.3f}. Try to maintain this success while being more efficient." + elif done and reward < 1: + return f"In your attempt, you completed the task in {steps} steps but only achieved a reward of {reward:.3f}. You need to improve your performance to achieve full success." + elif not done and steps >= max_env_steps: + return f"In your attempt, you reached the maximum step limit of {max_env_steps} steps without completing the task (reward: {reward:.3f}). You need to be more efficient and focused to complete the task within the step limit." + else: + return f"In your attempt, you stopped after {steps} steps with a reward of {reward:.3f} without completing the task. You need to improve your strategy and persistence to achieve success." + + +def save_task_data( + task_id: str, + game_file_path: str, + first_trajectory: List[Dict[str, str]], + first_reward: float, + first_steps: int, + first_success: bool, + second_trajectory: Optional[List[Dict[str, str]]], + second_reward: Optional[float], + second_steps: Optional[int], + second_success: Optional[bool], + kept_for_sft: bool, + sft_dir: str, + non_sft_dir: str, + training_data: Optional[List[Dict[str, str]]] = None, +): + """Save detailed exploration data to individual task file in appropriate folder""" + task_data = { + "task_id": task_id, + "timestamp": datetime.now().isoformat(), + "game_file": game_file_path, + # First exploration + "first_exploration": { + "trajectory": first_trajectory, + "reward": first_reward, + "steps": first_steps, + "success": first_success, + }, + # Second exploration + "second_exploration": { + "trajectory": second_trajectory, + "reward": second_reward, + "steps": second_steps, + "success": second_success, + }, + # Training data (clean dialogue format for SFT) + "training_data": training_data if training_data is not None else "", + "kept_for_sft": kept_for_sft, + } + + # Determine folder based on SFT data status (following webshop pattern) + if kept_for_sft: + target_dir = sft_dir + else: + target_dir = non_sft_dir + + task_file_path = os.path.join(target_dir, f"{task_id}.json") + + with open(task_file_path, "w", encoding="utf-8") as f: + json.dump(task_data, f, ensure_ascii=False, indent=2) diff --git a/frameworks/trinity/common/workflows/envs/alfworld/alfworld_workflow.py b/frameworks/trinity/common/workflows/envs/alfworld/alfworld_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..a04f3ec821f610df5936ad92184e18aff65600ca --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/alfworld_workflow.py @@ -0,0 +1,289 @@ +# -*- coding: utf-8 -*- +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.step_wise_workflow import RewardPropagationWorkflow +from trinity.common.workflows.workflow import MultiTurnWorkflow, Task + +EXAMPLE_PROMPT = """ +Observation: +-= Welcome to TextWorld, ALFRED! =- + +You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1. + +Your task is to: put some spraybottle on toilet. + +Response: +To solve the task, I need to find and take a spraybottle, then put it on toilet. First I need to find a spraybottle. A spraybottle is more likely to appear in cabinet (1-4), countertop (1), toilet (1), sinkbasin (1-2), garbagecan (1). I can check one by one, starting with cabinet 1.go to cabinet 1 + +Observation: +On the cabinet 1, you see a cloth 1, a soapbar 1, a soapbottle 1. + +Response: +The spraybottle is not here, I should go to cabinet2 to check one by onego to cabinet 2 + +Observation: +The cabinet 2 is closed. + +Response: +I should open the cabinet 2 to see what's in itopen cabinet 2 + +Observation: +You open the cabinet 2. The cabinet 2 is open. In it, you see a candle 1, and a spraybottle 2. + +Response: +Now I find a spraybottle (2). Next, I need to take it.take spraybottle 2 from cabinet 2 + +Observation: +You pick up the spraybottle 2 from the cabinet 2. + +Response: +Now I take a spraybottle (2). Next, I need to put it in/on toilet 1.go to toilet 1 + +Observation: +On the toilet 1, you see a soapbottle 2. + +Response: + Now I should put the taken spraybottle2 on toilet 1 to finish my task. move spraybottle 2 to toilet 1 +""" + +AlfWORLD_SYSTEM_PROMPT = """ +You are an agent interacting with a virtual text-based environment. + +## Notes: +At each step, you should first think then perform action to fulfill the instruction. You should ALWAYS wrap your thinking with the tag and wrap your action with the tag. +You should ALWAYS take one action each step. +DO NOT try to interact with the user at anytime. Finish the task by yourself. + +## Action Format: +Below are the available commands you can use: + look: look around your current location + inventory: check your current inventory(you can only have 1 item in your inventory) + go to (receptacle): move to a receptacle + open (receptacle): open a receptacle + close (receptacle): close a receptacle + take (object) from (receptacle): take an object from a receptacle + move (object) to (receptacle): place an object in or on a receptacle + examine (something): examine a receptacle or an object + use (object): use an object + heat (object) with (receptacle): heat an object using a receptacle + clean (object) with (receptacle): clean an object using a receptacle + cool (object) with (receptacle): cool an object using a receptacle + slice (object) with (object): slice an object using a sharp object + +For example your output should be like this: + To solve the task, I need first to ... go to cabinet 1 +""" + + +def format_observation(observation: str): + if "Nothing happens." in observation: + observation += "Please check if the action you take is valid or you have carefully followed the action format." + return "Observation: " + observation + + +def parse_action(response): + try: + # parse the action within the tag + action = response.split("")[1].split("")[0].strip() + return action + except Exception as e: + print(f"Error parsing action: {e}, response = {response}") + return "" + + +class AlfworldWorkflow(MultiTurnWorkflow): + """A workflow for alfworld task.""" + + is_async: bool = True + can_repeat: bool = False + + def __init__( + self, + model: ModelWrapper, + task: Task, + auxiliary_models: Optional[List] = None, + ): + super().__init__( + model=model, + task=task, + ) + self.task_desc = task.task_desc or "0" + self.repeat_times = task.repeat_times + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + + async def get_model_response(self, messages): + responses = await self.model.chat_async(messages, n=1) + return responses + + async def get_model_response_text(self, messages): + return (await self.get_model_response(messages))[0].response_text + + async def generate_env_inference_samples(self, env) -> List[Experience]: + observation, info = env.reset() + final_reward = -0.1 + memory = [] + memory.append({"role": "system", "content": AlfWORLD_SYSTEM_PROMPT}) + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + memory = memory + [{"role": "user", "content": format_obs}] + response_text = await self.get_model_response_text(memory) + memory.append({"role": "assistant", "content": response_text}) + action = parse_action(response_text) + observation, reward, done, info = env.step(action) + if done: + final_reward = reward + break + experience = await self.process_messages_to_experience_async( + memory, final_reward, {"env_rounds": r, "env_done": 1 if done else 0} + ) + # Close the env to save cpu memory + env.close() + return [experience] + + async def run_async(self) -> List[Experience]: + # assume the task_description is the game_file_path generated. + # see Trinity-RFT/examples/grpo_alfworld/get_alfworld_data.py + game_file_path = self.task_desc + try: + import textworld + import textworld.gym + from alfworld.agents.environment.alfred_tw_env import ( + AlfredDemangler, + AlfredExpert, + AlfredExpertType, + ) + + def create_environment(game_file): + expert = AlfredExpert(expert_type=AlfredExpertType.HANDCODED) + request_infos = textworld.EnvInfos( + description=True, inventory=True, admissible_commands=True + ) + + env_id = textworld.gym.register_game( + game_file, request_infos, wrappers=[AlfredDemangler(), expert] + ) + env = textworld.gym.make(env_id) + return env + + except Exception as e: + print("Please make sure you have installed the alfworld package.") + error_message = f"Error importing AlfworldTWEnv {str(e)}. Please make sure you have installed the alfworld package successfully, following the instructions in https://github.com/alfworld/alfworld" + raise ImportError(error_message) + env = create_environment(game_file_path) + return await self.generate_env_inference_samples(env) + + +class StepWiseAlfworldWorkflow(RewardPropagationWorkflow): + """ + An Alfworld workflow refactored to use the RewardPropagationWorkflow base class. + + This workflow manages an Alfworld environment, interacts with it step-by-step + using a model, and calculates a final reward based on the episode's outcome. + """ + + def __init__( + self, + model: ModelWrapper, + task: Task, + auxiliary_models: Optional[List] = None, + use_openai_client: bool = False, + ): + super().__init__( + model=model, + task=task, + auxiliary_models=auxiliary_models, + use_openai_client=use_openai_client, + ) + self.game_file_path = task.task_desc or "0" + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + + self._setup_environment() + + self.observation: Optional[str] = None + self.done: bool = False + self.final_reward: float = 0.0 + self.memory: List[dict] = [] + + def _setup_environment(self): + """Initializes the Alfworld text-based environment.""" + try: + import textworld + import textworld.gym + from alfworld.agents.environment.alfred_tw_env import ( + AlfredDemangler, + AlfredExpert, + AlfredExpertType, + ) + + def create_environment(game_file): + expert = AlfredExpert(expert_type=AlfredExpertType.HANDCODED) + request_infos = textworld.EnvInfos( + description=True, inventory=True, admissible_commands=True + ) + env_id = textworld.gym.register_game( + game_file, request_infos, wrappers=[AlfredDemangler(), expert] + ) + env = textworld.gym.make(env_id) + return env + + self.env = create_environment(self.game_file_path) + + except ImportError as e: + error_message = ( + f"Error importing Alfworld dependencies: {e}. Please ensure " + "Alfworld is installed correctly by following the instructions at " + "https://github.com/alfworld/alfworld" + ) + raise ImportError(error_message) + + def run(self) -> List[Experience]: + # Reset environment and state for a new episode + self.observation, info = self.env.reset() + self.done = False + self.final_reward = -0.1 + + self.memory.clear() + self.memory.append({"role": "system", "content": AlfWORLD_SYSTEM_PROMPT}) + + return super().run() + + def step(self, step_num: int) -> bool: + if self.done: + return False + + # Format observation for the model + format_obs = format_observation(self.observation) # type: ignore + self.memory.append({"role": "user", "content": format_obs}) + + # Get action from the model + responses = self.model.chat(self.memory) + response_text = responses[0].response_text + self.memory.append({"role": "assistant", "content": response_text}) + action = parse_action(response_text) + + # Execute action in the environment + observation, reward, done, info = self.env.step(action) + + # Update internal state + self.observation = observation + self.done = done + if self.done: + self.final_reward = reward + + # Return False to stop the run if the episode is done + return not self.done + + def reward(self, exps: list[Experience]) -> float: + return self.final_reward + + @property + def max_step_num(self) -> int: + """Return the maximum number of steps allowed in an episode.""" + return self.max_env_steps + + def __del__(self): + """Ensures the environment is closed when the workflow object is destroyed.""" + if hasattr(self, "env"): + self.env.close() diff --git a/frameworks/trinity/common/workflows/envs/alfworld/opd/CHECKPOINT_STRATEGY_README.md b/frameworks/trinity/common/workflows/envs/alfworld/opd/CHECKPOINT_STRATEGY_README.md new file mode 100644 index 0000000000000000000000000000000000000000..a5fa9925998be87ca44c2715e0d48d50834933f1 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/opd/CHECKPOINT_STRATEGY_README.md @@ -0,0 +1,209 @@ +# Checkpoint Strategy for AlfWorld OPD Workflow + +This document explains how to use the checkpoint strategy feature in the `check_opd_verl_agent_alfworld_workflow`. + +## Overview + +The checkpoint strategy allows the model to start from different points in the expert trajectory during training. This implements a curriculum learning approach where the model gradually learns to handle the full task autonomously. + +## Configuration + +### Fixed Checkpoint + +Use a fixed number of expert actions before letting the model take over: + +```yaml +workflow_args: + checkpoint_step: 3 # Always execute first 3 expert actions +``` + +### Linear Decay Strategy (Recommended) + +Gradually reduce the number of expert actions as training progresses: + +```yaml +workflow_args: + checkpoint_strategy: linear_decay + checkpoint_decay_steps: 0.5 # Complete decay in first 50% of training steps +``` + +**Key Points:** +- **Start point (fixed)**: `len(actions) - 1` — Model only executes the last action +- **End point (fixed)**: `0` — Model executes the entire trajectory from beginning +- **Only one parameter**: `checkpoint_decay_steps` controls when decay completes + +**Note**: The parameter name `checkpoint_decay_epochs` is also supported for backward compatibility, but `checkpoint_decay_steps` is recommended when using `total_steps` in your buffer configuration. + +## How It Works + +### Linear Decay Strategy + +The strategy automatically: +1. **Starts** with `checkpoint_step = len(actions) - 1` (easiest: model only does last step) +2. **Decays linearly** to `checkpoint_step = 0` (hardest: model does everything) +3. **Completes decay** at `total_epochs × checkpoint_decay_epochs` +4. **Maintains** `checkpoint_step = 0` for remaining training + +### Formula + +```python +decay_completion_step = total_steps × checkpoint_decay_steps + +if current_step < decay_completion_step: + progress = current_step / decay_completion_step + checkpoint_step = (len(actions) - 1) × (1 - progress) +else: + checkpoint_step = 0 # Full autonomy +``` + +### Examples + +#### Example 1: `checkpoint_decay_steps = 0.5` +With `total_steps = 200` and `num_actions = 7`: + +``` +Step 0: checkpoint=6 → Model executes step 7 only (1 step) +Step 20: checkpoint=5 → Model executes steps 6-7 (2 steps) +Step 40: checkpoint=4 → Model executes steps 5-7 (3 steps) +Step 60: checkpoint=2 → Model executes steps 3-7 (5 steps) +Step 80: checkpoint=1 → Model executes steps 2-7 (6 steps) +Step 100: checkpoint=0 → Model executes steps 1-7 (all 7 steps) ← Decay complete +Step 120-200: checkpoint=0 → Full autonomy continues +``` + +**Meaning**: Decay completes at step 100 (50% of 200 steps), then full autonomy for remaining 50%. + +#### Example 2: `checkpoint_decay_steps = 0.7` +With `total_steps = 200` and `num_actions = 10`: + +``` +Step 0: checkpoint=9 → Model executes step 10 only +Step 20: checkpoint=8 → Model executes steps 9-10 +... +Step 120: checkpoint=1 → Model executes steps 2-10 +Step 140: checkpoint=0 → Model executes all steps ← Decay complete +Step 160-200: checkpoint=0 → Full autonomy continues +``` + +**Meaning**: Decay completes at step 140 (70% of 200 steps), then full autonomy for remaining 30%. + +#### Example 3: `checkpoint_decay_steps = 1.0` +With `total_steps = 200` and `num_actions = 5`: + +``` +Step 0: checkpoint=4 → Model executes step 5 only +Step 40: checkpoint=3 → Model executes steps 4-5 +Step 80: checkpoint=2 → Model executes steps 3-5 +Step 120: checkpoint=2 → Model executes steps 3-5 +Step 160: checkpoint=1 → Model executes steps 2-5 +Step 200: checkpoint=0 → Model executes all steps ← Decay complete +``` + +**Meaning**: Decay completes at step 200 (100% of 200 steps), using the full training duration. + +## Choosing `checkpoint_decay_steps` + +| Value | Behavior | Use Case | +|-------|----------|----------| +| `0.3` | Fast decay (30% of training) | Model learns quickly, want early full autonomy | +| `0.5` | Moderate decay (50% of training) | **Recommended default** — balanced approach | +| `0.7` | Slow decay (70% of training) | Difficult tasks, need more gradual curriculum | +| `1.0` | Full training decay | Use entire training for curriculum | + +**Rule of thumb**: Start with `0.5`. If model struggles, increase to `0.7` or `1.0`. + +## Monitoring + +The workflow records detailed metrics for each experience to help you understand the curriculum progress: + +### Checkpoint Metrics + +| Metric | Description | Example | +|--------|-------------|---------| +| `total_expert_actions` | Total number of actions in the expert trajectory | 7 | +| `expert_actions_executed` | Number of expert actions executed before model takes over | 5 | +| `expert_actions_remaining` | Number of expert actions NOT executed (model had to figure out) | 2 | +| `model_executed_steps` | Number of steps the model executed by itself | 3 | + +### Legacy Metrics (for backward compatibility) + +| Metric | Description | +|--------|-------------| +| `checkpoint_step` | Same as `expert_actions_executed` | +| `used_checkpoint_step` | Same as `expert_actions_executed` | + +### Understanding the Metrics + +**Example 1**: With checkpoint_step=5 and total_expert_actions=7 +```python +total_expert_actions = 7 # Expert trajectory has 7 actions +expert_actions_executed = 5 # First 5 actions executed automatically +expert_actions_remaining = 2 # Last 2 actions not provided to model +model_executed_steps = 3 # Model executed 3 steps (may differ from remaining due to errors/retries) +``` + +**Example 2**: No checkpoint (full autonomy) +```python +total_expert_actions = 7 # Expert trajectory has 7 actions +expert_actions_executed = 0 # No expert guidance +expert_actions_remaining = 7 # Model must figure out all 7 actions +model_executed_steps = 8 # Model took 8 steps (more than expert due to exploration) +``` + +### Monitoring in WandB + +You can track curriculum progress by plotting: +- `expert_actions_executed`: Should decrease over training (from ~total_expert_actions to 0) +- `expert_actions_remaining`: Should increase over training (from 0 to ~total_expert_actions) +- `model_executed_steps`: Shows how many steps model needs (may increase as it gets less guidance) + +Example WandB query: +```python +# Track curriculum decay +wandb.log({ + "curriculum/expert_guidance": expert_actions_executed, + "curriculum/model_autonomy": expert_actions_remaining, + "curriculum/model_steps": model_executed_steps, +}) +``` + +## Complete Configuration Example + +```yaml +buffer: + total_steps: 200 # Or use total_epochs: 10 + batch_size: 16 + explorer_input: + taskset: + path: 'train_expert.jsonl' + split: train + format: + prompt_key: 'game_file' + workflow_args: + max_env_steps: 30 + checkpoint_strategy: linear_decay + checkpoint_decay_steps: 0.5 # Use checkpoint_decay_epochs if using total_epochs + default_workflow_type: 'check_opd_verl_agent_alfworld_workflow' +``` + +This configuration: +- Trains for 200 steps total (or 10 epochs if using `total_epochs`) +- Starts with model executing only the last action +- Linearly decays to full autonomy over first 100 steps (50%) +- Continues with full autonomy for remaining 100 steps + +## Implementation Notes + +1. **Automatic start/end**: No need to specify start and end steps — they're automatically determined from the expert trajectory length +2. **Per-trajectory adaptation**: Each trajectory may have different lengths, so the starting checkpoint adapts automatically +3. **Clamping**: Checkpoint step is always clamped to `[0, len(actions)-1]` +4. **Backward compatible**: Without `checkpoint_strategy`, uses fixed `checkpoint_step` if provided + +## Comparison with Fixed Checkpoint + +| Approach | Configuration | Behavior | +|----------|--------------|----------| +| **Fixed** | `checkpoint_step: 3` | Always execute first 3 actions, never changes | +| **Linear Decay** | `checkpoint_decay_epochs: 0.5` | Start from last action, gradually increase model responsibility | + +**Recommendation**: Use linear decay for curriculum learning. Use fixed checkpoint only for debugging or specific experimental needs. diff --git a/frameworks/trinity/common/workflows/envs/alfworld/opd/bi_opd_alfworld_workflow.py b/frameworks/trinity/common/workflows/envs/alfworld/opd/bi_opd_alfworld_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..9561ce511c2986c1ac2f2f3a2fe17aee0f3f6be0 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/opd/bi_opd_alfworld_workflow.py @@ -0,0 +1,330 @@ +# -*- coding: utf-8 -*- +"""Bidirectional On-Policy Distillation (Bi-OPD) workflow for AlfWorld. + +This workflow implements a gradual distillation strategy where: +- Student executes actions independently (no teacher action intervention) +- Teacher gradually distills more previous steps as training progresses: + * Initially: distills only the first step + * After decay_step: distills first 2 steps + * After 2*decay_step: distills first 3 steps + * Eventually: distills all previous steps + +The distillation happens on the prefix (previous actions), not on the current action. +""" + +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.workflow import Task, Workflow + +from trinity.common.workflows.envs.alfworld.opd.prompts import ( + ALFWORLD_TEMPLATE_NO_HIS, + ALFWORLD_TEMPLATE, +) +from trinity.common.workflows.envs.alfworld.alfworld_workflow import ( + parse_action, +) + +HISTORY_LENGTH = 2 +MEMORY_FORMAT = "[Observation {step_num}: '{obs}', Action {step_num}: '{act}']" + + +def format_observation(observation: str): + return observation.strip() + +def _extract_task(text_obs: str) -> str: + """Extract the task description from the text observation.""" + task_start = text_obs.find("Your task is to: ") + if task_start != -1: + return text_obs[task_start + len("Your task is to: ") :].strip() + raise ValueError("Task description not found in text observation.") + + +def _format_history(observation: str, step_num: int, act: str) -> str: + """Format observation and action for action_history.""" + return MEMORY_FORMAT.format(step_num=step_num, obs=observation.strip(), act=act) + +def _create_alfworld_env(game_file_path: str): + """Create AlfWorld textworld environment.""" + try: + import textworld + import textworld.gym + from alfworld.agents.environment.alfred_tw_env import ( + AlfredDemangler, + AlfredExpert, + AlfredExpertType, + ) + + expert = AlfredExpert(expert_type=AlfredExpertType.HANDCODED) + request_infos = textworld.EnvInfos( + description=True, inventory=True, admissible_commands=True + ) + env_id = textworld.gym.register_game( + game_file_path, request_infos, wrappers=[AlfredDemangler(), expert] + ) + return textworld.gym.make(env_id) + except Exception as e: + raise ImportError( + f"Error creating AlfWorld env: {e}. " + "Ensure alfworld is installed: https://github.com/alfworld/alfworld" + ) from e + + +class BiOpdAlfworldWorkflow(Workflow): + """Bidirectional On-Policy Distillation workflow for AlfWorld. + + Key differences from standard OPD: + - Student executes all actions independently (no teacher intervention) + - Teacher gradually distills more previous steps as training progresses + - Distillation is applied to the prefix (history), not the current action + + The decay_step parameter controls how quickly the distillation window grows: + - Training step 0 to decay_step-1: distill first 1 step + - Training step decay_step to 2*decay_step-1: distill first 2 steps + - Training step 2*decay_step to 3*decay_step-1: distill first 3 steps + - And so on... + + Use advantage_fn: multi_turn_opd (MultiTurnOpdAdvantage) for this workflow. + """ + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + self.is_eval = task.is_eval + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "Bi-OPD requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + + # Decay step: controls how quickly distillation window grows + # e.g., decay_step=5 means every 5 training steps, distill one more previous step + self.decay_step = task.workflow_args.get("decay_step", 5) + + # Track current training step for dynamic distillation window + self._current_training_step = 0 + self._total_training_steps = 0 + + def reset(self, task: Task): + """Reset the workflow with a new task.""" + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + """Set current training progress for dynamic distillation window. + + Args: + current_step: Current training step (e.g., current epoch or iteration) + total_steps: Total training steps (e.g., total epochs) + """ + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _compute_distill_window(self) -> int: + """Compute how many previous steps to distill based on current training progress. + + Returns: + Number of previous steps to distill (starting from step 1) + """ + current_step = self._current_training_step + # Calculate distillation window: starts at 1, increases by 1 every decay_step + distill_window = 1 + (current_step // self.decay_step) + return distill_window + + def compute_reward(self, response: Experience) -> float: + """Return episode-level reward (same for all turns in the trajectory).""" + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + """Format initial messages for the episode.""" + return [] + + async def run_async(self) -> List[Experience]: + # Extract current training step from task batch_id + current_step = 0 + if hasattr(self.task, 'batch_id'): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + import re + match = re.match(r'^(\d+)', batch_id) + if match: + current_step = int(match.group(1)) + + total_steps = self.task.workflow_args.get('total_steps', 200) + self.set_training_progress(current_step, total_steps) + + game_file_path = self.task_desc + env = _create_alfworld_env(game_file_path) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info = env.reset() + self._env_done = False + self._env_rounds = 0 + + task_description = _extract_task(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + # Compute distillation window for this episode + distill_window = self._compute_distill_window() + + # Eval should run normal student rollout up to max_env_steps without distillation. + if self.is_eval: + effective_steps = self.max_env_steps + else: + # Student only executes distill_window steps (the steps teacher will distill) + # But cap it at max_env_steps to avoid exceeding environment limits + effective_steps = min(distill_window, self.max_env_steps) + + for r in range(effective_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join( + history[-HISTORY_LENGTH:] + if len(history) >= HISTORY_LENGTH + else history + ) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + # Student samples this turn independently + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "BiOpdAlfworldWorkflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + + # Track if environment is done during distillation window + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = reward + break + else: + # Completed effective_steps without finishing + self._env_rounds = effective_steps + self._final_reward = 0.0 + + # Teacher distillation: distill all executed steps + per_turn_kl_sums: List[float] = [] + + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + if response.metrics is None: + response.metrics = {} + compute_reward_fn = getattr(self, "compute_reward", None) + response.reward = ( + compute_reward_fn(response) + if callable(compute_reward_fn) + else getattr(self, "_final_reward", 0.0) + ) + response.eid.run = self.run_id_base + response.eid.step = i + + # Trajectory-level metrics + trajectory_kl_divergence = sum(per_turn_kl_sums) + + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + last_response.metrics["distill_window"] = distill_window + last_response.metrics["effective_steps"] = effective_steps + last_response.metrics["total_steps"] = len(turn_responses) + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/alfworld/opd/check_opd_verl_agent_alfworld_workflow.py b/frameworks/trinity/common/workflows/envs/alfworld/opd/check_opd_verl_agent_alfworld_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..aa39667a214903c4fd409e8748fcfb284754a6d3 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/opd/check_opd_verl_agent_alfworld_workflow.py @@ -0,0 +1,599 @@ +# -*- coding: utf-8 -*- +"""On-Policy Distillation (OPD) workflow for AlfWorld. + +Reference: OnPolicyDistillWorkflow logic; adapted for multi-turn AlfWorld. + +Algorithm: +1. Student pre-samples trajectory (runs episode turn-by-turn with logprobs) +2. Split by turns: each turn has fixed prefix [system, obs_1, resp_1, ..., obs_t] +3. Teacher computes logprobs on same (prefix + response) per turn +4. Store teacher_logprobs in experience; advantage_fn uses teacher_logprobs - student_logprobs +5. Return one Experience per turn (like OPD returning one per sample) +""" + +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.workflow import Task, Workflow + +# from trinity.common.workflows.envs.alfworld.opd.prompts import ( +# ALFWORLD_TEMPLATE_NO_HIS, +# ALFWORLD_TEMPLATE, +# ) +from trinity.common.workflows.envs.alfworld.opd.prompts3 import ( + ALFWORLD_TEMPLATE_NO_HIS, + ALFWORLD_TEMPLATE, +) +from trinity.common.workflows.envs.alfworld.alfworld_workflow import ( + # format_observation, + parse_action, +) + +HISTORY_LENGTH = 2 +MEMORY_FORMAT = "[Observation {step_num}: '{obs}', Action {step_num}: '{act}']" + + +def format_observation(observation: str): + return observation.strip() + +def _extract_task(text_obs: str) -> str: + """Extract the task description from the text observation.""" + task_start = text_obs.find("Your task is to: ") + if task_start != -1: + return text_obs[task_start + len("Your task is to: ") :].strip() + raise ValueError("Task description not found in text observation.") + + +def _format_history(observation: str, step_num: int, act: str) -> str: + """Format observation and action for action_history.""" + return MEMORY_FORMAT.format(step_num=step_num, obs=observation.strip(), act=act) + +def _create_alfworld_env(game_file_path: str): + """Create AlfWorld textworld environment.""" + try: + import textworld + import textworld.gym + from alfworld.agents.environment.alfred_tw_env import ( + AlfredDemangler, + AlfredExpert, + AlfredExpertType, + ) + + expert = AlfredExpert(expert_type=AlfredExpertType.HANDCODED) + request_infos = textworld.EnvInfos( + description=True, inventory=True, admissible_commands=True + ) + env_id = textworld.gym.register_game( + game_file_path, request_infos, wrappers=[AlfredDemangler(), expert] + ) + return textworld.gym.make(env_id) + except Exception as e: + raise ImportError( + f"Error creating AlfWorld env: {e}. " + "Ensure alfworld is installed: https://github.com/alfworld/alfworld" + ) from e + + +def _create_alfworld_env_with_checkpoint(game_file_path: str, actions: List[str], checkpoint_step: int): + """Create AlfWorld environment and execute actions up to checkpoint_step. + + Args: + game_file_path: Path to the game file + actions: List of predefined expert actions + checkpoint_step: Number of actions to execute before letting model take over + + Returns: + tuple: (env, observation, info, history, task_description, current_step) + """ + env = _create_alfworld_env(game_file_path) + observation, info = env.reset() + + task_description = _extract_task(observation) + history: List[str] = [] + + # Execute predefined actions up to checkpoint_step + for step in range(min(checkpoint_step, len(actions))): + action = actions[step] + format_obs = format_observation(observation) + history.append(_format_history(format_obs, step + 1, action)) + observation, reward, done, info = env.step(action) + + if done: + # If task completes before checkpoint, return the final state + break + + return env, observation, info, history, task_description, len(history) + + +class OnPolicyDistillVerlAgentAlfworldWorkflow(Workflow): + """On-policy distillation workflow for AlfWorld. + + Computes and stores teacher_logprobs in each turn's experience. + The advantage_fn in trainer will compute: + advantages = teacher_logprobs - student_logprobs + + Use advantage_fn: multi_turn_opd (MultiTurnOpdAdvantage) for this workflow, + since it returns List[Experience] (one per turn), not a single response. + + Logic aligned with OnPolicyDistillWorkflow: + - Student samples (with logprobs); teacher computes logprobs on same sequences. + - Per-turn split: prefix fixed per turn, one Experience per turn. + - compute_reward() can be overridden by subclasses (default: episode final reward). + """ + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.checkpoint_step = task.workflow_args.get("checkpoint_step", None) + self.is_eval = task.is_eval + + # Naive mode: if True, always use _run_episode without any checkpoint (baseline) + self.naive = task.workflow_args.get("naive", False) + + # Checkpoint strategy configuration + self.checkpoint_strategy = task.workflow_args.get("checkpoint_strategy", None) + if self.checkpoint_strategy == "linear_decay": + # checkpoint_decay_steps: number of training steps before reducing expert actions by 1 + # e.g., 5 means every 5 training steps, expert actions decrease by 1 + # Training step 0-4: expert_actions = len(actions) - 1 + # Training step 5-9: expert_actions = len(actions) - 2 + # Training step 10-14: expert_actions = len(actions) - 3 + # ... until expert_actions = 0 + self.checkpoint_decay_steps = task.workflow_args.get( + "checkpoint_decay_steps", 5 + ) + + # Track current training step for dynamic checkpoint + self._current_training_step = 0 + self._total_training_steps = 0 + + def reset(self, task: Task): + """Reset the workflow with a new task. + + Unlike BaseSimpleWorkflow, this does NOT require reward_fn. + """ + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + self.naive = task.workflow_args.get("naive", False) + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def set_training_progress(self, current_step: int, total_steps: int): + """Set current training progress for dynamic checkpoint strategy. + + Args: + current_step: Current training step (e.g., current epoch or iteration) + total_steps: Total training steps (e.g., total epochs) + """ + self._current_training_step = current_step + self._total_training_steps = total_steps + + def _compute_checkpoint_step(self, predefined_actions: Optional[List[str]] = None) -> Optional[int]: + """Compute checkpoint step based on strategy. + + Args: + predefined_actions: List of predefined actions (to get max possible checkpoint) + + Returns: + Checkpoint step to use, or None if no checkpoint should be used + """ + # If no strategy is set, use the fixed checkpoint_step + if self.checkpoint_strategy is None: + return 0 + + elif self.checkpoint_strategy == "linear_decay": + if predefined_actions is None or len(predefined_actions) == 0: + return 0 + + current_step = self._current_training_step + + # Calculate how many expert actions to execute + # Start: len(actions) - 1, decrease by 1 every checkpoint_decay_steps + max_expert_actions = len(predefined_actions) - 1 + reduction = current_step // self.checkpoint_decay_steps + checkpoint_step = max_expert_actions - reduction + + # Clamp to valid range [0, len(actions)-1] + # When checkpoint_step <= 0, return 0 (no expert actions, model starts from beginning) + checkpoint_step = max(0, min(checkpoint_step, len(predefined_actions) - 1)) + + return checkpoint_step + else: + return 0 + + def compute_reward(self, response: Experience) -> float: + """Return episode-level reward (same for all turns in the trajectory). + + Set in _run_episode: env reward when done, 0.0 when max steps exhausted. + """ + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + """Format initial messages for the episode. + + Uses ALFWORLD_TEMPLATE_NO_HIS / ALFWORLD_TEMPLATE from prompts.py. + No system prompt; each user message is self-contained. + """ + return [] + + async def run_async(self) -> List[Experience]: + # If naive mode is enabled, always use normal _run_episode (baseline, no checkpoint) + # If this is an eval task, always use normal _run_episode (no checkpoint) + if self.naive or self.is_eval: + # Set _used_checkpoint_step to 0 for metrics tracking + self._used_checkpoint_step = 0 + env = _create_alfworld_env(self.task_desc) + try: + return await self._run_episode(env) + finally: + env.close() + + # Extract current training step from task batch_id + current_step = 0 + if hasattr(self.task, 'batch_id'): + batch_id = self.task.batch_id + if isinstance(batch_id, int): + current_step = batch_id + elif isinstance(batch_id, str): + # Extract the first integer from the string (e.g., "5/eval_task" -> 5) + import re + match = re.match(r'^(\d+)', batch_id) + if match: + current_step = int(match.group(1)) + + # Get total_steps from workflow_args if provided, otherwise use a reasonable default + # Note: You can add "total_steps: 200" to workflow_args in your config file + # to explicitly set this value + total_steps = self.task.workflow_args.get('total_steps', 200) + + # Update training progress for dynamic checkpoint strategy + # Always update to get the latest training step from batch_id + self.set_training_progress(current_step, total_steps) + + game_file_path = self.task_desc + predefined_actions = self.raw_task.get("actions", None) + + # Compute checkpoint step dynamically based on strategy + effective_checkpoint_step = self._compute_checkpoint_step(predefined_actions) + + # Store for metrics + self._used_checkpoint_step = effective_checkpoint_step if effective_checkpoint_step is not None else 0 + + # Check if we should use checkpoint mode + # Only use checkpoint if effective_checkpoint_step > 0 and predefined_actions exist + if effective_checkpoint_step is not None and effective_checkpoint_step > 0 and predefined_actions is not None: + env, observation, info, history, task_description, current_step = ( + _create_alfworld_env_with_checkpoint( + game_file_path, predefined_actions, effective_checkpoint_step + ) + ) + try: + return await self._run_episode_from_checkpoint( + env, observation, info, history, task_description, current_step + ) + finally: + env.close() + else: + # Original mode: start from beginning (no checkpoint or checkpoint_step == 0) + env = _create_alfworld_env(game_file_path) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info = env.reset() + self._env_done = False + self._env_rounds = 0 + + task_description = _extract_task(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join( + history[-HISTORY_LENGTH:] + if len(history) >= HISTORY_LENGTH + else history + ) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + # Step 1: Student samples this turn (same pattern as OnPolicyDistillWorkflow) + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillAlfworldWorkflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = reward + break + else: + self._env_rounds = self.max_env_steps + self._final_reward = 0.0 # failure: exhausted max steps + + # Step 2 & 3: Teacher logprobs and fill experience (mirror OnPolicyDistillWorkflow.run_async) + # response.tokens is the full sequence for this turn: [prefix | response], where + # prefix = system + obs_1 + resp_1 + ... + obs_t (same input as student had). + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), # full input = prefix + student's response + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + compute_reward_fn = getattr(self, "compute_reward", None) + response.reward = ( + compute_reward_fn(response) + if callable(compute_reward_fn) + else getattr(self, "_final_reward", 0.0) + ) + response.eid.run = self.run_id_base + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + # Trajectory-level metrics (computed once for the whole trajectory) + trajectory_kl_divergence = sum(per_turn_kl_sums) + + # Calculate detailed checkpoint metrics + # In naive/eval mode, we don't use expert actions, so set them to 0 + if self.naive or self.is_eval: + total_expert_actions = 0 + expert_actions_executed = 0 + expert_actions_remaining = 0 + model_executed_steps = self._env_rounds + else: + total_expert_actions = len(self.raw_task.get("actions", [])) + expert_actions_executed = self._used_checkpoint_step # Use the computed checkpoint step + expert_actions_remaining = total_expert_actions - expert_actions_executed + model_executed_steps = self._env_rounds - expert_actions_executed + + # Log trajectory-level metrics once per task to avoid duplicated eval aggregation. + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + # New detailed metrics (more intuitive) + last_response.metrics["total_expert_actions"] = total_expert_actions + last_response.metrics["expert_actions_executed"] = expert_actions_executed + last_response.metrics["expert_actions_remaining"] = expert_actions_remaining + last_response.metrics["model_executed_steps"] = model_executed_steps + + return turn_responses + + async def _run_episode_from_checkpoint( + self, env, observation, info, history: List[str], task_description: str, start_step: int + ) -> List[Experience]: + """Run episode from a checkpoint with predefined history. + + Args: + env: AlfWorld environment already at checkpoint state + observation: Current observation at checkpoint + info: Current info dict at checkpoint + history: List of formatted history strings from predefined actions + task_description: Extracted task description + start_step: Step number to start from (number of predefined actions executed) + + Returns: + List of Experience objects for turns after checkpoint + """ + self._env_done = False + self._env_rounds = start_step + + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(start_step, self.max_env_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join( + history[-HISTORY_LENGTH:] + if len(history) >= HISTORY_LENGTH + else history + ) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + # Student samples this turn + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillAlfworldWorkflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = reward + break + else: + self._env_rounds = self.max_env_steps + self._final_reward = 0.0 # failure: exhausted max steps + + # Teacher logprobs and fill experience + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + compute_reward_fn = getattr(self, "compute_reward", None) + response.reward = ( + compute_reward_fn(response) + if callable(compute_reward_fn) + else getattr(self, "_final_reward", 0.0) + ) + response.eid.run = self.run_id_base + response.eid.step = start_step + i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + # Trajectory-level metrics + trajectory_kl_divergence = sum(per_turn_kl_sums) + + # Calculate detailed checkpoint metrics + total_expert_actions = len(self.raw_task.get("actions", [])) + expert_actions_executed = start_step # Number of expert actions executed before model takes over + expert_actions_remaining = total_expert_actions - expert_actions_executed # Expert actions not executed + model_executed_steps = self._env_rounds - start_step # Steps executed by model + + # Log trajectory-level metrics once per task to avoid duplicated eval aggregation. + if turn_responses: + last_response = turn_responses[-1] + if last_response.metrics is None: + last_response.metrics = {} + last_response.metrics["env_rounds"] = self._env_rounds + last_response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + last_response.metrics["kl_divergence"] = trajectory_kl_divergence + + # New detailed metrics (more intuitive) + last_response.metrics["total_expert_actions"] = total_expert_actions + last_response.metrics["expert_actions_executed"] = expert_actions_executed + last_response.metrics["expert_actions_remaining"] = expert_actions_remaining + last_response.metrics["model_executed_steps"] = model_executed_steps + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/alfworld/opd/opd_verl_agent_alfworld_workflow.py b/frameworks/trinity/common/workflows/envs/alfworld/opd/opd_verl_agent_alfworld_workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..fe6fdcd13c259ca7f2ba78aa3f276036c11d3699 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/opd/opd_verl_agent_alfworld_workflow.py @@ -0,0 +1,280 @@ +# -*- coding: utf-8 -*- +"""On-Policy Distillation (OPD) workflow for AlfWorld. + +Reference: OnPolicyDistillWorkflow logic; adapted for multi-turn AlfWorld. + +Algorithm: +1. Student pre-samples trajectory (runs episode turn-by-turn with logprobs) +2. Split by turns: each turn has fixed prefix [system, obs_1, resp_1, ..., obs_t] +3. Teacher computes logprobs on same (prefix + response) per turn +4. Store teacher_logprobs in experience; advantage_fn uses teacher_logprobs - student_logprobs +5. Return one Experience per turn (like OPD returning one per sample) +""" + +from dataclasses import asdict +from typing import List, Optional + +from trinity.common.experience import Experience +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.workflow import Task, Workflow + +from trinity.common.workflows.envs.alfworld.opd.prompts import ( + ALFWORLD_TEMPLATE_NO_HIS, + ALFWORLD_TEMPLATE, +) +from trinity.common.workflows.envs.alfworld.alfworld_workflow import ( + # format_observation, + parse_action, +) + +HISTORY_LENGTH = 2 +MEMORY_FORMAT = "[Observation {step_num}: '{obs}', Action {step_num}: '{act}']" + + +def format_observation(observation: str): + return observation.strip() + +def _extract_task(text_obs: str) -> str: + """Extract the task description from the text observation.""" + task_start = text_obs.find("Your task is to: ") + if task_start != -1: + return text_obs[task_start + len("Your task is to: ") :].strip() + raise ValueError("Task description not found in text observation.") + + +def _format_history(observation: str, step_num: int, act: str) -> str: + """Format observation and action for action_history.""" + return MEMORY_FORMAT.format(step_num=step_num, obs=observation.strip(), act=act) + +def _create_alfworld_env(game_file_path: str): + """Create AlfWorld textworld environment.""" + try: + import textworld + import textworld.gym + from alfworld.agents.environment.alfred_tw_env import ( + AlfredDemangler, + AlfredExpert, + AlfredExpertType, + ) + + expert = AlfredExpert(expert_type=AlfredExpertType.HANDCODED) + request_infos = textworld.EnvInfos( + description=True, inventory=True, admissible_commands=True + ) + env_id = textworld.gym.register_game( + game_file_path, request_infos, wrappers=[AlfredDemangler(), expert] + ) + return textworld.gym.make(env_id) + except Exception as e: + raise ImportError( + f"Error creating AlfWorld env: {e}. " + "Ensure alfworld is installed: https://github.com/alfworld/alfworld" + ) from e + + +class OnPolicyDistillVerlAgentAlfworldWorkflow(Workflow): + """On-policy distillation workflow for AlfWorld. + + Computes and stores teacher_logprobs in each turn's experience. + The advantage_fn in trainer will compute: + advantages = teacher_logprobs - student_logprobs + + Use advantage_fn: multi_turn_opd (MultiTurnOpdAdvantage) for this workflow, + since it returns List[Experience] (one per turn), not a single response. + + Logic aligned with OnPolicyDistillWorkflow: + - Student samples (with logprobs); teacher computes logprobs on same sequences. + - Per-turn split: prefix fixed per turn, one Experience per turn. + - compute_reward() can be overridden by subclasses (default: episode final reward). + """ + + is_async: bool = True + can_reset: bool = True + can_repeat: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + self.reset(task) + + assert ( + self.auxiliary_model_wrappers is not None + and len(self.auxiliary_model_wrappers) >= 1 + ), "On-policy distillation requires at least one auxiliary model as teacher." + self.teacher_model = self.auxiliary_model_wrappers[0] + + self.temperature = task.workflow_args.get("temperature", 1.0) + self.max_env_steps = task.workflow_args.get("max_env_steps", 30) + self.is_eval = task.is_eval + + def reset(self, task: Task): + """Reset the workflow with a new task. + + Unlike BaseSimpleWorkflow, this does NOT require reward_fn. + """ + self.task = task + self.format_args = task.format_args + self.raw_task = task.raw_task + self.task_desc = task.task_desc or "0" + self.is_eval = task.is_eval + + def set_repeat_times(self, repeat_times, run_id_base): + self.repeat_times = repeat_times + self.task.rollout_args.n = repeat_times + self.run_id_base = run_id_base + + def compute_reward(self, response: Experience) -> float: + """Return episode-level reward (same for all turns in the trajectory). + + Set in _run_episode: env reward when done, 0.0 when max steps exhausted. + """ + return getattr(self, "_final_reward", 0.0) + + @property + def rollout_args(self): + return asdict(self.task.rollout_args) + + def format_messages(self): + """Format initial messages for the episode. + + Uses ALFWORLD_TEMPLATE_NO_HIS / ALFWORLD_TEMPLATE from prompts.py. + No system prompt; each user message is self-contained. + """ + return [] + + async def run_async(self) -> List[Experience]: + game_file_path = self.task_desc + env = _create_alfworld_env(game_file_path) + try: + return await self._run_episode(env) + finally: + env.close() + + async def _run_episode(self, env) -> List[Experience]: + observation, info = env.reset() + self._env_done = False + self._env_rounds = 0 + + task_description = _extract_task(observation) + history: List[str] = [] + memory = self.format_messages() + turn_responses: List[Experience] = [] + + kwargs = {**self.rollout_args, "n": 1} + if kwargs.get("logprobs") is None: + kwargs["logprobs"] = 0 + + for r in range(self.max_env_steps): + format_obs = format_observation(observation) + admissible_commands = info.get("admissible_commands", []) + if admissible_commands and isinstance(admissible_commands[0], list): + admissible_commands = admissible_commands[0] + reformatted_admissible = "\n ".join( + f"'{s}'" for s in admissible_commands if s != "help" + ) + + if len(history) < HISTORY_LENGTH: + user_content = ALFWORLD_TEMPLATE_NO_HIS.format( + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + else: + action_history_str = "\n".join( + history[-HISTORY_LENGTH:] + if len(history) >= HISTORY_LENGTH + else history + ) + user_content = ALFWORLD_TEMPLATE.format( + task_description=task_description, + step_count=r, + history_length=min(HISTORY_LENGTH, len(history)), + action_history=action_history_str, + current_step=r + 1, + current_observation=format_obs, + admissible_actions=reformatted_admissible, + ) + + memory = memory + [{"role": "user", "content": user_content}] + + # Step 1: Student samples this turn (same pattern as OnPolicyDistillWorkflow) + responses = await self.model.chat_async(memory, **kwargs) + response = responses[0] + response_text = response.response_text or "" + memory.append({"role": "assistant", "content": response_text}) + + if response.logprobs is None: + raise RuntimeError( + "OnPolicyDistillAlfworldWorkflow requires student model to return logprobs. " + "Set rollout_args.logprobs (e.g. 0) in task config." + ) + turn_responses.append(response) + + action = parse_action(response_text) + history.append(_format_history(format_obs, r + 1, action)) + observation, reward, done, info = env.step(action) + if done: + self._env_done = True + self._env_rounds = r + 1 + self._final_reward = reward + break + else: + self._env_rounds = self.max_env_steps + self._final_reward = 0.0 # failure: exhausted max steps + + # # Drop failed trajectories that exhaust max_env_steps with zero reward. + # if self._env_rounds >= self.max_env_steps and self._final_reward == 0.0: + # return [] + + # Step 2 & 3: Teacher logprobs and fill experience (mirror OnPolicyDistillWorkflow.run_async) + # response.tokens is the full sequence for this turn: [prefix | response], where + # prefix = system + obs_1 + resp_1 + ... + obs_t (same input as student had). + per_turn_kl_sums: List[float] = [] + for i, response in enumerate(turn_responses): + teacher_logprobs = await self.teacher_model.logprobs_async( + tokens=response.tokens.tolist(), # full input = prefix + student's response + temperature=self.temperature, + ) + + resp_start = response.prompt_length - 1 + teacher_resp_logprobs = teacher_logprobs[resp_start:] + student_resp_logprobs = response.logprobs + + assert len(teacher_resp_logprobs) == len(student_resp_logprobs), ( + f"Length mismatch: teacher_logprobs={len(teacher_resp_logprobs)}, " + f"student_logprobs={len(student_resp_logprobs)}. " + f"tokens={len(response.tokens)}, prompt_length={response.prompt_length}" + ) + + response.teacher_logprobs = teacher_resp_logprobs + + if response.metrics is None: + response.metrics = {} + compute_reward_fn = getattr(self, "compute_reward", None) + response.reward = ( + compute_reward_fn(response) + if callable(compute_reward_fn) + else getattr(self, "_final_reward", 0.0) + ) + response.eid.run = self.run_id_base + response.eid.step = i + + kl_sum = (student_resp_logprobs - teacher_resp_logprobs).sum().item() + per_turn_kl_sums.append(kl_sum) + + # Trajectory-level metrics (computed once for the whole trajectory) + trajectory_kl_divergence = sum(per_turn_kl_sums) + for response in turn_responses: + response.metrics["env_rounds"] = self._env_rounds + response.metrics["env_done"] = 1.0 if self._env_done else 0.0 + response.metrics["kl_divergence"] = trajectory_kl_divergence + + return turn_responses diff --git a/frameworks/trinity/common/workflows/envs/alfworld/opd/prompts.py b/frameworks/trinity/common/workflows/envs/alfworld/opd/prompts.py new file mode 100644 index 0000000000000000000000000000000000000000..6f7244534d1988b3f22988e1f14282cf59d6cd24 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/opd/prompts.py @@ -0,0 +1,36 @@ +# Copyright 2025 Nanyang Technological University (NTU), Singapore +# and the verl-agent (GiGPO) team. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# --------------------- ALFWorld --------------------- # +ALFWORLD_TEMPLATE_NO_HIS = """ +You are an expert agent operating in the ALFRED Embodied Environment. +Your current observation is: {current_observation} +Your admissible actions of the current situation are: [{admissible_actions}]. + +Now it's your turn to take an action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" + +ALFWORLD_TEMPLATE = """ +You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description} +Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history} +You are now at step {current_step} and your current observation is: {current_observation} +Your admissible actions of the current situation are: [{admissible_actions}]. + +Now it's your turn to take an action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" \ No newline at end of file diff --git a/frameworks/trinity/common/workflows/envs/alfworld/opd/prompts3.py b/frameworks/trinity/common/workflows/envs/alfworld/opd/prompts3.py new file mode 100644 index 0000000000000000000000000000000000000000..bde1e290bf6890a93e816db43cf7efbaf80bf899 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/alfworld/opd/prompts3.py @@ -0,0 +1,36 @@ +# Copyright 2025 Nanyang Technological University (NTU), Singapore +# and the verl-agent (GiGPO) team. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# --------------------- ALFWorld --------------------- # +ALFWORLD_TEMPLATE_NO_HIS = """ +You are an expert agent operating in the ALFRED Embodied Environment. +Your current observation is: {current_observation} +Your admissible actions of the current situation are: [{admissible_actions}]. + +Now it's your turn to take an action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" + +ALFWORLD_TEMPLATE = """ +You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task_description} +Prior to this step, you have already taken {step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history} +You are now at step {current_step} and your current observation is: {current_observation} +Your admissible actions of the current situation are: [{admissible_actions}]. + +Now it's your turn to take an action. +You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within tags. +Once you've finished your reasoning, you should choose an admissible action for current step and present it within tags. +""" diff --git a/frameworks/trinity/common/workflows/envs/email_searcher/prepare_data.py b/frameworks/trinity/common/workflows/envs/email_searcher/prepare_data.py new file mode 100644 index 0000000000000000000000000000000000000000..49e12b0bd7d0e70d6c30eefb65a68e670e93fdfe --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/email_searcher/prepare_data.py @@ -0,0 +1,279 @@ +""" +Prepare data for training. +Modified from https://github.com/OpenPipe/ART/blob/art-e/examples/art-e/art_e/data/local_email_db.py +""" + +import logging +import os +import sqlite3 +from datetime import datetime + +from datasets import Dataset, Features, Sequence, Value, load_dataset +from tqdm import tqdm + +# Resolve paths relative to this file so it works regardless of the current working directory +BASE_DIR = os.path.dirname(os.path.abspath(__file__)) +# Database will live in "../data/enron_emails.db" relative to project root +DEFAULT_DB_PATH = os.path.join(BASE_DIR, "..", "..", "data", "enron_emails.db") + +DEFAULT_REPO_ID = "corbt/enron-emails" + +logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s") + + +# --- Database Schema --- +SQL_CREATE_TABLES = """ +DROP TABLE IF EXISTS recipients; +DROP TABLE IF EXISTS emails_fts; +DROP TABLE IF EXISTS emails; + +CREATE TABLE emails ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + message_id TEXT UNIQUE, + subject TEXT, + from_address TEXT, + date TEXT, -- Store as ISO 8601 string 'YYYY-MM-DD HH:MM:SS' + body TEXT, + file_name TEXT +); + +CREATE TABLE recipients ( + email_id INTEGER, + recipient_address TEXT, + recipient_type TEXT, -- 'to', 'cc', 'bcc' + FOREIGN KEY(email_id) REFERENCES emails(id) ON DELETE CASCADE +); +""" + +SQL_CREATE_INDEXES_TRIGGERS = """ +CREATE INDEX idx_emails_from ON emails(from_address); +CREATE INDEX idx_emails_date ON emails(date); +CREATE INDEX idx_emails_message_id ON emails(message_id); +CREATE INDEX idx_recipients_address ON recipients(recipient_address); +CREATE INDEX idx_recipients_type ON recipients(recipient_type); +CREATE INDEX idx_recipients_email_id ON recipients(email_id); +CREATE INDEX idx_recipients_address_email ON recipients(recipient_address, email_id); + +CREATE VIRTUAL TABLE emails_fts USING fts5( + subject, + body, + content='emails', + content_rowid='id' +); + +CREATE TRIGGER emails_ai AFTER INSERT ON emails BEGIN + INSERT INTO emails_fts (rowid, subject, body) + VALUES (new.id, new.subject, new.body); +END; + +CREATE TRIGGER emails_ad AFTER DELETE ON emails BEGIN + DELETE FROM emails_fts WHERE rowid=old.id; +END; + +CREATE TRIGGER emails_au AFTER UPDATE ON emails BEGIN + UPDATE emails_fts SET subject=new.subject, body=new.body WHERE rowid=old.id; +END; + +INSERT INTO emails_fts (rowid, subject, body) SELECT id, subject, body FROM emails; +""" + + +# --- Functions --- + + +def download_dataset(repo_id: str) -> Dataset: + """Downloads the dataset from Hugging Face Hub.""" + logging.info(f"Attempting to download dataset from Hugging Face Hub: {repo_id}") + expected_features = Features( + { + "message_id": Value("string"), + "subject": Value("string"), + "from": Value("string"), + "to": Sequence(Value("string")), + "cc": Sequence(Value("string")), + "bcc": Sequence(Value("string")), + "date": Value("timestamp[us]"), + "body": Value("string"), + "file_name": Value("string"), + } + ) + dataset_obj = load_dataset(repo_id, features=expected_features, split="train") + # Basic type check remains useful + if not isinstance(dataset_obj, Dataset): + raise TypeError(f"Expected Dataset, got {type(dataset_obj)}") + logging.info(f"Successfully loaded dataset '{repo_id}' with {len(dataset_obj)} records.") + return dataset_obj + + +def create_database(db_path: str): + """Creates the SQLite database and tables.""" + logging.info(f"Creating SQLite database and tables at: {db_path}") + conn = sqlite3.connect(db_path) + cursor = conn.cursor() + cursor.executescript(SQL_CREATE_TABLES) + conn.commit() + conn.close() + logging.info("Database tables created successfully.") + + +def populate_database(db_path: str, dataset: Dataset): + """Populates the database with data from the Hugging Face dataset.""" + logging.info(f"Populating database {db_path}...") + conn = sqlite3.connect(db_path) + cursor = conn.cursor() + + # --- Performance Pragmas --- + conn.execute("PRAGMA synchronous = OFF;") + conn.execute("PRAGMA journal_mode = MEMORY;") + + record_count = 0 + skipped_count = 0 # Keep track of skipped emails due to filters + duplicate_count = 0 # Keep track of skipped duplicate emails + processed_emails = set() # Track (subject, body, from) tuples to dedupe + + conn.execute("BEGIN TRANSACTION;") # Single transaction for bulk insert + + for email_data in tqdm(dataset, desc="Inserting emails"): + assert isinstance(email_data, dict) + message_id = email_data["message_id"] + subject = email_data["subject"] + from_address = email_data["from"] + date_obj: datetime = email_data["date"] + body = email_data["body"] + file_name = email_data["file_name"] + to_list_raw = email_data["to"] + cc_list_raw = email_data["cc"] + bcc_list_raw = email_data["bcc"] + + # --- Data Cleaning and Filtering --- + date_str = date_obj.strftime("%Y-%m-%d %H:%M:%S") + to_list = [str(addr) for addr in to_list_raw if addr] + cc_list = [str(addr) for addr in cc_list_raw if addr] + bcc_list = [str(addr) for addr in bcc_list_raw if addr] + + # Check body length + if len(body) > 5000: + logging.debug(f"Skipping email {message_id}: Body length > 5000 characters.") + skipped_count += 1 + continue + + # Check total recipients + total_recipients = len(to_list) + len(cc_list) + len(bcc_list) + if total_recipients > 30: + logging.debug( + f"Skipping email {message_id}: Total recipients ({total_recipients}) > 30." + ) + skipped_count += 1 + continue + # --- End Filtering --- + + # --- Deduplication Check --- + email_key = (subject, body, from_address) + if email_key in processed_emails: + logging.debug( + f"Skipping duplicate email (Subject: {subject[:50]}..., From: {from_address})" + ) + duplicate_count += 1 + continue + else: + processed_emails.add(email_key) + # --- End Deduplication --- + + cursor.execute( + """ + INSERT INTO emails (message_id, subject, from_address, date, body, file_name) + VALUES (?, ?, ?, ?, ?, ?) + """, + (message_id, subject, from_address, date_str, body, file_name), + ) + email_pk_id = cursor.lastrowid + + recipient_data = [] + for addr in to_list: + recipient_data.append((email_pk_id, addr, "to")) + for addr in cc_list: + recipient_data.append((email_pk_id, addr, "cc")) + for addr in bcc_list: + recipient_data.append((email_pk_id, addr, "bcc")) + + if recipient_data: + cursor.executemany( + """ + INSERT INTO recipients (email_id, recipient_address, recipient_type) + VALUES (?, ?, ?) + """, + recipient_data, + ) + record_count += 1 + + conn.commit() + conn.close() + logging.info(f"Successfully inserted {record_count} email records.") + if skipped_count > 0: + logging.info(f"Skipped {skipped_count} email records due to length or recipient limits.") + if duplicate_count > 0: + logging.info( + f"Skipped {duplicate_count} duplicate email records (based on subject, body, from)." + ) + + +def create_indexes_and_triggers(db_path: str): + """Creates indexes and triggers on the populated database.""" + logging.info(f"Creating indexes and triggers for database: {db_path}...") + conn = sqlite3.connect(db_path) + cursor = conn.cursor() + cursor.executescript(SQL_CREATE_INDEXES_TRIGGERS) + conn.commit() + conn.close() + logging.info("Indexes and triggers created successfully.") + + +def generate_database( + repo_id: str = DEFAULT_REPO_ID, db_path: str = DEFAULT_DB_PATH, overwrite: bool = False +): + """ + Generates the SQLite database from the specified Hugging Face dataset. + Simplified version without extensive error handling. + + Args: + repo_id: The Hugging Face repository ID for the dataset. + db_path: The path where the SQLite database file should be created. + overwrite: If True, any existing database file at db_path will be removed. + """ + logging.info(f"Starting database generation for repo '{repo_id}' at '{db_path}'") + logging.info(f"Overwrite existing database: {overwrite}") + + db_dir = os.path.dirname(db_path) + if db_dir and not os.path.exists(db_dir): + logging.info(f"Creating data directory: {db_dir}") + os.makedirs(db_dir) + + if overwrite and os.path.exists(db_path): + logging.warning(f"Removing existing database file: {db_path}") + os.remove(db_path) + elif not overwrite and os.path.exists(db_path): + # If not overwriting and file exists, subsequent steps might fail or behave unexpectedly. + # We are removing the explicit error here as requested. + logging.warning( + f"Database file {db_path} exists and overwrite is False. Assuming file is already generated." + ) + return + + # 1. Download dataset + dataset = download_dataset(repo_id) + + # 2. Create database schema (Tables only) + # Note: This will fail if overwrite=False and the file exists with incompatible schema/data. + create_database(db_path) + + # 3. Populate database + populate_database(db_path, dataset) + + # 4. Create Indexes and Triggers + create_indexes_and_triggers(db_path) + + logging.info(f"Database generation process completed for {db_path}.") + + +if __name__ == "__main__": + generate_database(overwrite=True) diff --git a/frameworks/trinity/common/workflows/envs/email_searcher/react_agent.py b/frameworks/trinity/common/workflows/envs/email_searcher/react_agent.py new file mode 100644 index 0000000000000000000000000000000000000000..5f0c6f8b2d71326995298d08051061441087751f --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/email_searcher/react_agent.py @@ -0,0 +1,127 @@ +import json +import traceback +from dataclasses import asdict +from datetime import datetime, timedelta +from typing import Any + +from agentscope.message import TextBlock +from agentscope.tool import Toolkit, ToolResponse + +from trinity.common.workflows.agentscope.react.react_agent import AgentScopeReActAgent +from trinity.common.workflows.envs.email_searcher.utils import ( + read_email_tool, + search_emails_tool, +) +from trinity.utils.log import get_logger + + +class EmailSearchAgent(AgentScopeReActAgent): + """ + A customized ReAct agent with pre-defined tools for email search and reading. + Ref: https://github.com/OpenPipe/ART/blob/main/dev/art-e/art_e/rollout.py#L260 + """ + + def __init__(self, *args, **kwargs): + self.logger = get_logger(__name__) + self.message_id_list = [] # List to store message IDs found during search + self.ever_read_message_ids = [] # List to store message IDs that have been read + toolkit = Toolkit() + toolkit.register_tool_function(self.search_emails) + toolkit.register_tool_function(self.read_email) + super().__init__(*args, toolkit=toolkit, **kwargs) + + def search_emails( + self, + inbox_address: str, + query_date: str, + keywords: list[str], + **kwargs: Any, + ) -> ToolResponse: + """ + Search the user's email inbox for emails that match the given keywords. + + Args: + inbox_address: The user's email address. + query_date: The date of the query in 'YYYY-MM-DD' format. + keywords (list[str]): A list of keywords to search for in the user's email inbox. + + Returns: + ToolResponse: + A ToolResponse object containing a list of TextBlock objects in the `content` field. + On success, the text field of the TextBlock contains a JSON string representing + a list of email summaries (e.g., message_id, snippet) matching + the search criteria. Each email summary is converted to a dictionary via `asdict`. + On failure, the text indicates an error message. + """ + + try: + next_day = (datetime.strptime(query_date, "%Y-%m-%d") + timedelta(days=1)).strftime( + "%Y-%m-%d" + ) + res = search_emails_tool(inbox=inbox_address, sent_before=next_day, keywords=keywords) + + self.message_id_list.extend([r.message_id for r in res]) + + return ToolResponse( + content=[ + TextBlock( + type="text", + text=json.dumps([asdict(r) for r in res]), + ), + ], + ) + except Exception as e: + self.logger.info(f"Error in search_emails: {e}, traceback: {traceback.format_exc()}") + return ToolResponse( + content=[ + TextBlock( + type="text", + text=f"Error: Failed to search emails.\nError message: {e}", + ), + ], + ) + + def read_email(self, message_id: str, **kwargs: Any) -> ToolResponse: + """ + Read the content of an email from the user's email inbox. Returns the email content. + Args: + message_id (str): The unique identifier of the email to read. + + Returns: + ToolResponse: + A ToolResponse object containing the email content or an error message if the email is not found. + """ + + try: + email_content = read_email_tool(message_id) + + self.ever_read_message_ids.append(message_id) + + if email_content is None: + return ToolResponse( + content=[ + TextBlock( + type="text", + text=f"Error: Email (message_id = {message_id}) not found.", + ), + ], + ) + else: + return ToolResponse( + content=[ + TextBlock( + type="text", + text=json.dumps(email_content.model_dump()), + ), + ], + ) + except Exception as e: + self.logger.info(f"Error in read_email: {e}, traceback: {traceback.format_exc()}") + return ToolResponse( + content=[ + TextBlock( + type="text", + text=f"Error: Failed to read email.\nError message: {e}", + ), + ], + ) diff --git a/frameworks/trinity/common/workflows/envs/email_searcher/utils.py b/frameworks/trinity/common/workflows/envs/email_searcher/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..daf88861db317dd1389e66ff82e35e9f1f18413c --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/email_searcher/utils.py @@ -0,0 +1,333 @@ +""" +This file defines Email Dataclass and three email_search_tools. +Modified from https://github.com/OpenPipe/ART/blob/art-e/examples/art-e/ +""" +import datetime +import os +import sqlite3 +from dataclasses import dataclass +from typing import Any, List, Optional + +from pydantic import BaseModel, Field, field_validator + +from trinity.utils.log import get_logger + +DEFAULT_DB_PATH = os.environ.get("DEFAULT_EMAIL_DB_PATH") +conn = None + + +def get_conn(): + global conn + if conn is None: + conn = sqlite3.connect(f"file:{DEFAULT_DB_PATH}?mode=ro", uri=True, check_same_thread=False) + return conn + + +############ Define dataclass ############ + + +class QueryModel(BaseModel): + id: int + question: str + answer: str + message_ids: List[str] # message_ids (strings) of referenced emails + how_realistic: float + inbox_address: str + query_date: str + + @field_validator("query_date", mode="before") + @classmethod + def format_date(cls, v: Any) -> str: + if isinstance(v, datetime.datetime): + return v.strftime("%Y-%m-%d") + return v + + +class AnswerModel(BaseModel): + answer: str = Field( + description="It should be called with the answer and the sources. If you cannot find the answer, you should return 'I don't know' with an empty list of sources." + ) + sources: List[str] = Field( + description="a list of message ids that are relevant to the query. Usually there will be only one. If you cannot find the answer, you should return an empty list." + ) + + +class Email(BaseModel): + message_id: str + date: str # ISO 8601 string 'YYYY-MM-DD HH:MM:SS' + subject: Optional[str] = None + from_address: Optional[str] = None + to_addresses: List[str] = Field(default_factory=list) + cc_addresses: List[str] = Field(default_factory=list) + bcc_addresses: List[str] = Field(default_factory=list) + body: Optional[str] = None + file_name: Optional[str] = None + + +@dataclass +class SearchResult: + message_id: str + snippet: str + + +class FinalRubric(BaseModel): + answer_correct: bool = False + sources_correct: bool = False + num_turns: int = 0 + attempted_answer: bool = False + ever_found_right_email: bool = False + ever_read_right_email: bool = False + cant_parse_tool_call: bool = False + bad_tool_call_name: bool = False + bad_tool_call_args: bool = False + ran_out_of_turns: bool = False + returned_i_dont_know: bool = False + num_sources: int = 0 + ever_tried_to_read_invalid_email: bool = False + prompt_tokens: int = 0 + completion_tokens: int = 0 + + +############ Define tools for agentscope ############ + + +def search_emails_tool( + inbox: str, + keywords: List[str], + from_addr: Optional[str] = None, + to_addr: Optional[str] = None, + sent_after: Optional[str] = None, + sent_before: Optional[str] = None, + max_results: int = 10, +) -> List[SearchResult]: + """ + Searches the email database based on keywords, inbox, sender, recipient, and date range. + + Args: + inbox: The email address of the user performing the search. + Results include emails sent from or to (inc. cc/bcc) this address. + keywords: A list of keywords that must all appear in the subject or body. + from_addr: Optional email address to filter emails sent *from*. + to_addr: Optional email address to filter emails sent *to* (inc. cc/bcc). + sent_after: Optional date string 'YYYY-MM-DD'. Filters for emails sent on or after this date. + sent_before: Optional date string 'YYYY-MM-DD'. Filters for emails sent before this date. + max_results: The maximum number of results to return. Cannot exceed 10. + + Returns: + A list of SearchResult objects, each containing 'message_id' and 'snippet'. + Returns an empty list if no results are found or an error occurs. + """ + # Initialize sql and params + sql: Optional[str] = None + params: List[str | int] = [] + + cursor = get_conn().cursor() + + # --- Build Query --- + where_clauses: List[str] = [] + + # 1. Keywords (FTS) + if not keywords: + raise ValueError("No keywords provided for search.") + + if max_results > 10: + raise ValueError("max_results must be less than or equal to 10.") + + # FTS5 default is AND, so just join keywords. Escape quotes for safety. + fts_query = " ".join(f""" "{k.replace('"', '""')}" """ for k in keywords) + where_clauses.append("emails_fts MATCH ?") + params.append(fts_query) + + # 2. Inbox filter (must be from OR to/cc/bcc the inbox user) + # Use the composite index idx_recipients_address_email here + where_clauses.append( + """ + (e.from_address = ? OR EXISTS ( + SELECT 1 FROM recipients r_inbox + WHERE r_inbox.recipient_address = ? AND r_inbox.email_id = e.id + )) + """ + ) + params.extend([inbox, inbox]) + + # 3. Optional From filter + if from_addr: + where_clauses.append("e.from_address = ?") + params.append(from_addr) + + # 4. Optional To filter (includes to, cc, bcc) + # Use the composite index idx_recipients_address_email here + if to_addr: + where_clauses.append( + """ + EXISTS ( + SELECT 1 FROM recipients r_to + WHERE r_to.recipient_address = ? AND r_to.email_id = e.id + ) + """ + ) + params.append(to_addr) + + # 5. Optional Sent After filter + if sent_after: + # Assumes date format 'YYYY-MM-DD' + # Compare against the start of the day + where_clauses.append("e.date >= ?") + params.append(f"{sent_after} 00:00:00") + + # 6. Optional Sent Before filter + if sent_before: + # Assumes date format 'YYYY-MM-DD' + # Compare against the start of the day (exclusive) + where_clauses.append("e.date < ?") + params.append(f"{sent_before} 00:00:00") + + # --- Construct Final Query --- + # snippet(, , , , , ) + # -1 means highlight across all columns (subject, body) + sql = f""" + SELECT + e.message_id, + snippet(emails_fts, -1, '', '', ' ... ', 15) as snippet + FROM + emails e JOIN emails_fts fts ON e.id = fts.rowid + WHERE + {" AND ".join(where_clauses)} + ORDER BY + e.date DESC -- Order by date for relevance + LIMIT ?; + """ + logger = get_logger(__name__) + + params.append(max_results) + + # --- Execute and Fetch --- + logger.debug(f"Executing SQL: {sql}") + logger.debug(f"With params: {params}") + cursor.execute(sql, params) + results = cursor.fetchall() + + # Format results + formatted_results = [SearchResult(message_id=row[0], snippet=row[1]) for row in results] + logger.info(f"Search found {len(formatted_results)} results.") + return formatted_results + + +def read_email_tool(message_id: str) -> Optional[Email]: + """ + Retrieves a single email by its message_id from the database. + + Args: + message_id: The unique identifier of the email to retrieve. + + Returns: + An Email object containing the details of the found email, + or None if the email is not found or an error occurs. + """ + logger = get_logger(__name__) + + cursor = get_conn().cursor() + + # --- Query for Email Core Details --- + email_sql = """ + SELECT id, message_id, date, subject, from_address, body, file_name + FROM emails + WHERE message_id = ?; + """ + cursor.execute(email_sql, (message_id,)) + email_row = cursor.fetchone() + + if not email_row: + logger.warning(f"Email with message_id '{message_id}' not found.") + return None + + email_pk_id, msg_id, date, subject, from_addr, body, file_name = email_row + + # DEBUG + logger.info(f"[read_email_tool] input_message_id={message_id}") + logger.info(f"[read_email_tool] db: id={email_pk_id}, message_id={msg_id}") + + # search for recipients by emails.id (rather than message_id) + recipients_sql = """ + SELECT recipient_address, recipient_type + FROM recipients + WHERE email_id = ?; + """ + cursor.execute(recipients_sql, (email_pk_id,)) + recipient_rows = cursor.fetchall() + + to_addresses: List[str] = [] + cc_addresses: List[str] = [] + bcc_addresses: List[str] = [] + + for addr, rtype in recipient_rows: + type_lower = rtype.lower() + if type_lower == "to": + to_addresses.append(addr) + elif type_lower == "cc": + cc_addresses.append(addr) + elif type_lower == "bcc": + bcc_addresses.append(addr) + + # --- Construct Email Object --- + email_obj = Email( + message_id=msg_id, # Convert to string to match Pydantic model + date=date, + subject=subject, + from_address=from_addr, + to_addresses=to_addresses, + cc_addresses=cc_addresses, + bcc_addresses=bcc_addresses, + body=body, + file_name=file_name, + ) + + return email_obj + + +############ LLM-as-a-judge ############ + + +async def judge_correctness( + answer: str, + query: QueryModel, + judger: Any, +) -> bool: + """Use an LLM to decide whether *answer* matches *query.answer*. + + Returns a boolean *accept* flag used for scoring. + """ + + system_prompt = """You are given a question, the reference answer (labelled **Reference answer**), and an answer generated by an AI assistant (labelled **AI answer**). + +Follow these steps to decide whether the AI answer should be accepted: +1. Identify EXACTLY what information the **question** is asking for (e.g. who, what, when, where, why, how, quantity, etc.). +2. From the **Reference answer**, extract ONLY the facts that are required to directly satisfy the information need identified in step 1. Treat all other facts as non-essential context. +3. Verify that every essential fact from step 2 appears in the **AI answer** with the same meaning. Differences in wording, order, or additional non-conflicting details are allowed. +4. If any essential fact is missing or contradicted in the **AI answer**, then *accept* must be **false**. Otherwise *accept* must be **true**. + +Important: Do NOT penalise the **AI answer** for omitting non-essential facts that appear in the **Reference answer**. The answer should only be rejected for errors or omissions in the information explicitly requested by the question. + +Return your judgement **accept** from **true** and **false**. Do not return any other text or formatting. +""" + prompt = ( + f"Question: {query.question}\n" f"Reference answer: {query.answer}\n" f"AI answer: {answer}" + ) + + messages = [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": prompt}, + ] + completion = await judger.chat.completions.create( + model=judger.model_path, messages=messages, stream=False + ) + result = completion.choices[0].message.content + logger = get_logger(__name__) + logger.info(f"LLM judge response: {result}") + + # TODO: more robust judge + accept = False + if "true" in result.lower(): + accept = True + + return accept diff --git a/frameworks/trinity/common/workflows/envs/email_searcher/workflow.py b/frameworks/trinity/common/workflows/envs/email_searcher/workflow.py new file mode 100644 index 0000000000000000000000000000000000000000..a644e32a3c1c9aa86f78083e06fc79c485896a47 --- /dev/null +++ b/frameworks/trinity/common/workflows/envs/email_searcher/workflow.py @@ -0,0 +1,180 @@ +from __future__ import annotations + +from typing import Dict, List, Optional + +from trinity.common.models.model import ModelWrapper +from trinity.common.workflows.envs.email_searcher.utils import ( + AnswerModel, + FinalRubric, + QueryModel, + judge_correctness, +) +from trinity.common.workflows.workflow import Task, Workflow + +SYSTEM_PROMPT = """You are an email search agent. You are given a user query and a list of tools you can use to search the user's email. Use the tools to search the user's emails and find the answer to the user's query. You may take up to {max_turns} turns to find the answer, so if your first seach doesn't find the answer, you can try with different keywords. +Always describe what you see and plan your next steps clearly. When taking actions, explain what you're doing and why. When the answer to the task is found, call `generate_response` to finish the process. Only call `generate_response` when answer is found. You should not respond any next steps in `generate_response`. Complete all steps and then call `generate_response`. + +User's email address is {inbox_address} +Today's date is {query_date} +""" + + +class EmailSearchWorkflow(Workflow): + """ + Multi-turn Email Search workflow (ReAct-style tool use). + """ + + can_reset: bool = True + is_async: bool = True + + def __init__( + self, + *, + task: Task, + model: ModelWrapper, + auxiliary_models: Optional[List[ModelWrapper]] = None, + ): + # get openai client from model + self.openai_client = model.get_openai_async_client() + self.model_name = self.openai_client.model_path + super().__init__( + task=task, + model=model, + auxiliary_models=auxiliary_models, + ) + + self.reset(task) + + def reset(self, task: Task): + self.query = QueryModel.model_validate(task.raw_task) + self.task_desc = task.task_desc # question + self.truth = task.truth # ground truth answer + + self.workflow_args = task.workflow_args + self.max_turns = int(self.workflow_args.get("max_turns", 10)) + self.tool_obs_char_limit = int(self.workflow_args.get("tool_obs_char_limit", 2000)) + self.reward_fn_args = task.reward_fn_args or {} + + self.system_prompt = SYSTEM_PROMPT.format( + max_turns=self.max_turns, + inbox_address=self.query.inbox_address, + query_date=self.query.query_date, + ) + + try: + from trinity.common.workflows.envs.email_searcher.react_agent import ( + EmailSearchAgent, + ) + except ImportError as e: + error_message = f"AgentScope is not installed. Please install the agentscope framework first before running the workflow. Error: {str(e)}" + self.logger.error(error_message) + raise ImportError(error_message) + + self.agent = EmailSearchAgent( + model_name=self.openai_client.model_path, + openai_client=self.openai_client, + system_prompt=self.system_prompt, + generate_kwargs={ + "temperature": self.task.rollout_args.temperature, + "max_tokens": self.task.rollout_args.max_tokens or 4096, + }, + response_structure=AnswerModel, + max_iters=self.max_turns, + ) + + async def run_async(self): + answer_and_sources = await self.agent.reply(self.task_desc) + + experiences = self.model.extract_experience_from_history(clear_history=True) + self.actual_turns = len( + experiences + ) # NOTE: this metrics works only if the agent calls model once in each turn + + reward_dict = await self.calculate_reward(answer_and_sources) + reward = sum(reward_dict.values()) + + for i, experience in enumerate(experiences): + experience.eid.step = i + experience.reward = reward + if experience.metrics is None: + experience.metrics = {} + experience.metrics.update({"actual_turns": self.actual_turns}) + experience.metrics.update(reward_dict) + self.logger.info( + f"return experience len: {len(experiences)}, final step reward: {experiences[-1].reward}" + ) + return experiences + + async def calculate_reward(self, answer_and_sources: Dict) -> Dict[str, float]: + """Ref: calculate_reward in https://github.com/OpenPipe/ART/blob/main/dev/art-e/art_e/rollout.py#L64""" + try: + answer = answer_and_sources.get("answer", None) + sources = answer_and_sources.get("sources", []) + except Exception as e: + self.logger.error(f"Error extracting answer and sources: {e}") + result = {"accuracy": 0.0, "format": -1.0} + return result + + if answer is None: + result = {"accuracy": 0.0, "format": -1.0} + return result + + if not self.reward_fn_args.get("llm_as_a_judge", True): + result = {"accuracy": float(answer.lower() in self.truth.lower()), "format": 0.0} + return result + + rubric = FinalRubric() + rubric.attempted_answer = answer is not None and answer != "" + rubric.returned_i_dont_know = answer == "I don't know" + if len(self.query.message_ids) > 0: + rubric.ever_found_right_email = self.query.message_ids[0] in self.agent.message_id_list + rubric.ever_read_right_email = ( + self.query.message_ids[0] in self.agent.ever_read_message_ids + ) + rubric.sources_correct = self.query.message_ids[0] in sources + rubric.num_sources = len(sources) + rubric.num_turns = self.actual_turns + self.logger.debug(f"Rubric: {rubric.model_dump()}") + + try: + judge_model = self.auxiliary_models[0] if self.auxiliary_models else None + judge_response = await judge_correctness(answer, self.query, judge_model) + rubric.answer_correct = judge_response + + except Exception as e: + self.logger.error(f"Error judging correctness: {e}") + rubric.answer_correct = False + + # Note: make sure all possible partial rewards always sum to less than 0.5. + partial_rewards = 0 + partial_rewards += 0.1 if rubric.ever_found_right_email else 0 + partial_rewards += 0.1 if rubric.ever_read_right_email else 0 + partial_rewards += 0.1 if rubric.sources_correct else 0 + + # No formatting error, but wrong answer: reward will be -1 to 0 + if rubric.attempted_answer and not rubric.answer_correct: + result = {"accuracy": -1.0, "format": partial_rewards} + return result + + # Returned no answer at all: reward will be 0 to 1 + if rubric.returned_i_dont_know or rubric.ran_out_of_turns: + result = {"accuracy": 0.0, "format": partial_rewards} + return result + + # Answer is correct: reward will be 1 to 2 + if rubric.answer_correct: + # Partial credit calculation is different for correct answers. + + reward = 1 + reward += 0.3 if rubric.sources_correct else 0 + + # Extra credit for not including extra sources. + reward += 0.1 / rubric.num_sources if rubric.num_sources > 0 else 0 + + # Extra credit for being faster (taking fewer turns). + reward += 0.1 * (1 - rubric.num_turns / self.max_turns) + result = {"accuracy": 1.0, "format": reward} + return result + + self.logger.error(f"Rubric {rubric} not handled properly") + return {"accuracy": 0.0, "format": 0.0}