math-grpo-temp12-step200

GRPO + temperature 1.2 baseline on MATH, trained with GRPO on Qwen/Qwen2.5-Math-1.5B.

Selected as best validation pass@1 for this arm (rank 1).

Exploration via the sampling distribution on MATH. Trained at rollout temperature 1.2 but VALIDATED at 1.0 like every other arm, so its curve stays comparable.

Training data and format reward

dataset	MATH (math)
epochs / steps	25 / 1450
batch / rollouts	128 prompts, K=6
learning rate	1e-6 constant
KL (in-reward)	0.01
max prompt / response	512 / 1536 tokens
format reward	0.03, constant, no decay
seed	42

SUPERSEDED — read before use. This run used data/math, whose prompt is the plain "Let's think step by step and output the final answer within \boxed{}" instruction with no numbered-step scaffold, while the format reward (0.03) was active. The model was therefore rewarded for a structure its prompt never requested. A corrected suite trained on math_boxfix, where prompt and reward agree, supersedes these checkpoints. Within-suite comparisons among these five arms remain valid because all five shared this identical setting.

Arm-specific: entropy_coeff=0.0, clip=0.2/0.2, rollout temperature=1.2.

Do not apply a chat template

Trained on raw prompt text. verl's RLHFDataset has apply_chat_template=False and it was never enabled. Applying Qwen2.5-Math's chat template at inference creates a train/eval mismatch measured at roughly 19 points of pass@1.

from vllm import LLM, SamplingParams
llm = LLM(model="sandeep123/math-grpo-temp12-step200", dtype="bfloat16", max_model_len=2048)
params = SamplingParams(n=6, temperature=1.0, top_p=1.0, top_k=-1, max_tokens=1536)
out = llm.generate([prompt_text], sampling_params=params)   # raw string, not llm.chat()

Validation metrics at this checkpoint

metric	value
pass@1	0.7721
pass@6	0.9414
step	200

Answer extraction (pre-registered). An answer is the content of the final \boxed{}, compared by mathematical equivalence. Responses with no extractable answer are scored incorrect, and all K rollouts stay in the denominator.

Validation uses 256 held-out prompts, K=6, temperature 1.0, seed 42 -- pinned in code so every arm, including the temperature-1.2 arm, is scored under identical decoding.

Checkpoint selection

Quality-optimal and diversity-optimal checkpoints differ substantially, so both the best-pass@1 and best-pass@6 checkpoints are published for every arm. Diversity results are therefore never reported from a checkpoint chosen purely for accuracy.