sandeep123/dqo-math-K6a3kl-boxfix-step800

DQO-MATH (K=6, α=3, KL=0.01, boxfix)

RL-fine-tuned checkpoint of Qwen/Qwen2.5-Math-1.5B on the MATH dataset using DQO (Diverse Q-value Optimization) — response-level DPP diversity reward on whole-response embeddings (Chen et al., ICLR 2026).

Trained as the DQO baseline in a fair-matched comparison to STRIDE (step-level Cholesky diversity credit) under identical training infrastructure.

Training config

Setting	Value
Base model	Qwen/Qwen2.5-Math-1.5B
Dataset	DigitalLearningGmbH/MATH-lighteval
Method	DQO (response-level DPP on whole-response embeddings)
K rollouts	6
DQO alpha	3.0
KL coef	0.01
Structure-shaping coef	0.03 (decayed to 0 by step ~300)
Total epochs	15
Best-val step	800
Best-val pass@1	0.4609
Base training script	verl / train_grpo_math.sh

Prompt format

<math problem>

Let's think step by step, break your reasoning into numbered steps.

IMPORTANT rules:
1. You MUST produce at least 2 reasoning steps.
2. The final \boxed{X} must be on its OWN LINE, NOT part of any Step.
3. Do NOT write anything before Step 1 or after \boxed{}.

Respond in this exact format only:
Step 1: <one step of reasoning>
Step 2: <one step of reasoning>
...
Step n: <one step of reasoning>
\boxed{X}

Where X is the final mathematical answer.

Observed reward-hack behavior (research disclosure)

During training, this DQO checkpoint developed a shortcut where ~24% of rollouts emit an empty response (zero-length output). This inflates DPP's embedding-distance bonus (empty is maximally distant from any real answer) at zero cost to the K-1 other rollouts that do answer.

This is an intrinsic vulnerability of response-level DPP on whole-response embeddings and is why we developed STRIDE (step-level Cholesky over parsed step embeddings), which does not exhibit this failure. See the paper for full analysis.

Use this checkpoint with awareness: at generation time, expect ~24% of samples to be empty. Filter accordingly or generate more rollouts than needed.

Companion checkpoint

• STRIDE-MATH counterpart (fair-matched): see sandeep123/stride-math-K6a3kl-boxfix (coming)
• STRIDE-SciQA (Qwen2.5-1.5B-Instruct base): sandeep123/stride-sciqa-K6a3kl-boxfix-step600