DPWriter-Qwen3-4B-K32-ngram-lambda06-v2

Second RL run of a reproduction of DPWriter (ACL 2026, arXiv 2601.09609): GRPO with Diverse Planning Branching (K = 32 continuations per plan segment, n-gram diversity metric) and the group-aware diversity reward (lambda = 0.6, tau = 10), trained on Pitt CRC with the vendored verl 0.4.1 code of the DPWriter repository. Same recipe as run v1; this run additionally keeps an HF-merged copy of every 64th checkpoint.

• main = the end-of-schedule model (global step 430, 5 epochs).
• Branches step-64, step-128, step-192, step-256, step-320, step-384 = intermediate checkpoints (AutoModelForCausalLM.from_pretrained(repo, revision="step-128")).

All weights are full parameters in bf16.

• Start point: Johnny1024/DPWriter-Qwen3-4B-SFT.
• Prompts: 11,059 DPWriterData prompts kept by the paper's filter (max Skywork reward of 4 SFT samples below the overall mean).
• Reward: Skywork/Skywork-Reward-V2-Llama-3.1-8B score + 0.6 x leave-one-out n-gram diversity bonus when the score exceeds 10; -50 for a missing plan tag.
• Hyper-parameters (identical to the released script): batch 128, mini-batch 32, group size 8, lr 1e-6, KL 0.04 (low_var_kl), entropy 0, temperature 1.0, top-p 1.0, prompt 1024 / response 3072 tokens, 5 epochs (430 steps), fsdp2, dynamic batching with 65,536 tokens per GPU micro-batch, vLLM rollout.
• Compute: 2 x H200 (or 2 x RTX PRO 6000) with the reward model on a separate GPU; wandb project DPWriter.

In run v1 the Skywork validation score peaked around steps 64-128 and the final steps were unstable; pick the revision that works best for your evaluation.