math-dqo-a1-step700

DQO (alpha=1.0) baseline on MATH, trained with GRPO on Qwen/Qwen2.5-Math-1.5B.

Selected as best validation pass@1 for this arm (rank 1).

DQO on MATH (Chen et al., ICLR 2026): alpha*logdet(L+I) with a leave-one-out estimator, alpha=1.0 per the paper. Embedding phi = reference-policy hidden states (matching STRIDE) rather than the paper's pretrained sentence encoder, so the comparison isolates the diversity objective.

Training data and format reward

dataset	MATH (math)
epochs / steps	25 / 1450
batch / rollouts	128 prompts, K=6
learning rate	1e-6 constant
KL (in-reward)	0.01
max prompt / response	512 / 1536 tokens
format reward	0.03, constant, no decay
seed	42

SUPERSEDED — read before use. This run used data/math, whose prompt is the plain "Let's think step by step and output the final answer within \boxed{}" instruction with no numbered-step scaffold, while the format reward (0.03) was active. The model was therefore rewarded for a structure its prompt never requested. A corrected suite trained on math_boxfix, where prompt and reward agree, supersedes these checkpoints. Within-suite comparisons among these five arms remain valid because all five shared this identical setting.

Arm-specific: entropy_coeff=0.0, clip=0.2/0.2, rollout temperature=1.0, DQO alpha=1.0 (logdet diversity term, ref-policy embedding).

Do not apply a chat template

Trained on raw prompt text. verl's RLHFDataset has apply_chat_template=False and it was never enabled. Applying Qwen2.5-Math's chat template at inference creates a train/eval mismatch measured at roughly 19 points of pass@1.

from vllm import LLM, SamplingParams
llm = LLM(model="sandeep123/math-dqo-a1-step700", dtype="bfloat16", max_model_len=2048)
params = SamplingParams(n=6, temperature=1.0, top_p=1.0, top_k=-1, max_tokens=1536)
out = llm.generate([prompt_text], sampling_params=params)   # raw string, not llm.chat()

Validation metrics at this checkpoint

metric	value
pass@1	0.7656
pass@6	0.9453
step	700

Answer extraction (pre-registered). An answer is the content of the final \boxed{}, compared by mathematical equivalence. Responses with no extractable answer are scored incorrect, and all K rollouts stay in the denominator.

Validation uses 256 held-out prompts, K=6, temperature 1.0, seed 42 -- pinned in code so every arm, including the temperature-1.2 arm, is scored under identical decoding.

Checkpoint selection

Quality-optimal and diversity-optimal checkpoints differ substantially, so both the best-pass@1 and best-pass@6 checkpoints are published for every arm. Diversity results are therefore never reported from a checkpoint chosen purely for accuracy.