DPWriter-Qwen3-4B-K32-ngram-lambda06-v1

First RL run of a reproduction of DPWriter (ACL 2026, arXiv 2601.09609): GRPO with Diverse Planning Branching (K = 32 continuations per plan segment, n-gram diversity metric) and the group-aware diversity reward (lambda = 0.6, tau = 10), trained on Pitt CRC with the vendored verl 0.4.1 code of the DPWriter repository. This is the final checkpoint (global step 430) of run v1, full parameters in bf16.

• Start point: Johnny1024/DPWriter-Qwen3-4B-SFT.
• Prompts: 11,059 DPWriterData prompts kept by the paper's filter (max Skywork reward of 4 SFT samples below the overall mean).
• Reward: Skywork/Skywork-Reward-V2-Llama-3.1-8B score + 0.6 x leave-one-out n-gram diversity bonus when the score exceeds 10; -50 for a missing plan tag.
• Hyper-parameters (identical to the released script): batch 128, mini-batch 32, group size 8, lr 1e-6, KL 0.04 (low_var_kl), entropy 0, temperature 1.0, top-p 1.0, prompt 1024 / response 3072 tokens, 5 epochs (430 steps), fsdp2, dynamic batching with 65,536 tokens per GPU micro-batch, vLLM rollout.
• Compute: 2 x H200 (jobs 4013397 + 4019473, about 60 h) with the reward model on a separate GPU.
• Logs: wandb run https://wandb.ai/jchen-24k-university-of-pittsburgh/DPWriter/runs/4141kfpw

Caveat

Skywork validation scores (128 held-out prompts, one sample at T = 1) peaked around steps 64-128 (WritingPrompts 17.9, CreataSet 20.4, DeepWriting 28.5, COIG-Writer 30.1 at step 128) and drifted down as responses grew longer (over half of the rollouts hit the 3072-token cap in epochs 4-5). The last two steps saw a KL spike (0.05 -> 0.22); at step 430, 37 of the 128 validation outputs miss a plan tag or the closing </think> (per-source means 1.7 / -11.8 / -4.2 / -26.8). Treat this checkpoint as the end-of-schedule model of run v1, not as the best checkpoint; run v2 keeps a checkpoint every 64 steps.