Sinestro38/dsr1-qwen7b-sgd-polaris-step412-final

DS-R1-Distill-Qwen-7B + Polaris RL (PLAIN SGD, step 412, final)

This is the final checkpoint (step 412/412, end of 1 epoch) of a plain-SGD RL fine-tune of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B on POLARIS-Project/Polaris-Dataset-53K. Released as part of an ICML-2026 study on the low-rank structure of SGD vs Adam RL updates for batched-LoRA inference.

Why this checkpoint exists

The motivation is to compare the SVD-compressibility of ΔW = W_ft − W_base between SGD-trained and Adam-trained RL fine-tunes. Existing open RL FTs (POLARIS, Skywork-OR1, DeepCoder, AceReason, ORZ, DAPO) are all Adam-trained; we needed a same-base same-recipe SGD counterpart for the comparison. This is one of those.

Training recipe

field	value
base model	deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
dataset	POLARIS-Project/Polaris-Dataset-53K (52,779 train / 512 val)
algorithm	GRPO (adv_estimator=grpo, use_kl_loss=False, entropy_coeff=0, use_kl_in_reward=False)
optimizer	PLAIN SGD — momentum=0.0, nesterov=false, dampening=0.0, weight_decay=0.0
learning rate	1e-1 (constant)
train batch size	128 (1 grad step per rollout batch)
ppo_micro_batch_size_per_gpu	2
rollout.n	4
rollout.temperature	1.0
max_prompt_length	1024
max_response_length	8192
epochs at this checkpoint	1 (step 412 / 412, full epoch complete)
hardware	8× B200 (179 GB)
total wall time	11h 52m
step time	~115 s/step
trainer	verl (FSDP + vLLM rollout)

The "PLAIN SGD" choice is scientifically load-bearing for the broader study — every claim about SGD update compressibility relies on the update being the pure first-order gradient.

Reward function — important note

DS-R1-Distill emits <think>...</think>\boxed{X} style answers and does NOT produce literal "Answer: X" prose. verl's default math_dapo.compute_score matches Minerva regex r"(?i)Answer\s*:\s*([^\n]+)" and would return -1 for every rollout, killing the gradient.

A custom reward function (box-first, Minerva-fallback) was used: tries \boxed{} extraction first, then falls back to Minerva. Both styles supported.

Results

metric	value
baseline val acc (step 0)	14.3%
val acc at step 100 (peak)	24.85%
val acc at step 200	19.75%
val acc at step 300	19.0%
val acc at this ckpt (step 412)	21.7%

The val acc trajectory is non-monotonic — peaks at step 100, dips to 19% by step 300, partially recovers to 21.7% by step 412. This is the final-step checkpoint (most learning, end of 1 epoch). For the best-val checkpoint see Sinestro38/dsr1-qwen7b-sgd-polaris-step100-best-val.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

m = AutoModelForCausalLM.from_pretrained(
    "Sinestro38/dsr1-qwen7b-sgd-polaris-step412-final",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
tok = AutoTokenizer.from_pretrained("Sinestro38/dsr1-qwen7b-sgd-polaris-step412-final")

prompt = "Find all integer solutions to x^2 + y^2 = 25. Let's think step by step and output the final answer within \\boxed{}."
msgs = [{"role": "user", "content": prompt}]
ids = tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
out = m.generate(ids, max_new_tokens=2048, do_sample=True, temperature=0.6)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Caveats

• Reward signal is sparse — only ~5% of training steps had non-zero gradient norm in early training.
• Val acc dipped after the step-100 peak. If you're picking ONE ckpt for downstream use, prefer step 100. This checkpoint represents the most-trained state, not the best.
• ΔW magnitude is small relative to base weights — feature for compressibility study, but inference behavior is close to base.
• Trained on math only; no code, no instruction-following data.

Citation context

Work in progress — being submitted to ICML 2026 with a paper on plain-SGD RL update compressibility for batched-LoRA serving.

Related models in this study:

• Sinestro38/dsr1-qwen7b-sgd-polaris-step100-best-val — same run, peak-validation checkpoint