Add files using upload-large-folder tool

274951a verified 2 months ago

12.9 kB

---
library_name: peft
license: apache-2.0
tags:
  - qwen3.5
  - reinforcement-learning
  - dpo
  - grpo
  - ppo
  - debugging
base_model:
  - Qwen/Qwen3.5-4B
  - prism-drift/qwen35-4b-m0-v4
  - prism-drift/qwen35-9b-m0-v4
---
# RL Training Debugging Report

Date: 2026-08-26  
Status: diagnostic; no robust performance improvement established

## Executive summary

We ran a sequence of DPO, GRPO, and PPO experiments on Qwen3.5 4B and 9B,
first from project M0 checkpoints and later directly from the original
Qwen/Qwen3.5-4B base model. We varied the learning rate, KL/DPO/GRPO
coefficient, LoRA capacity, batch construction, sequence length, training
duration, and dataset filtering rules. We also rebuilt the 4B DPO data with
fresh Base-model rollouts after determining that the original preference
pairs were tied to M0 and contained substantial truncation and length bias.

The training paths are operational: gradients are non-zero, DPO preference
accuracy and margins move, GRPO has non-zero reward variance, PPO can complete
with non-finite-gradient guards, and merged adapters contain weights that are
different from both the base model and one another. However, these optimization
signals have not translated into a stable improvement on held-out verifier
reward or LiveCodeBench. A few early checkpoints improved small development
subsets, but the gains either disappeared on a larger held-out split or
reversed on an external benchmark. Later checkpoints often performed worse.

The current evidence is therefore consistent with noisy or mismatched
preference/reward supervision, checkpoint overtraining, and generation-length
pathologies rather than a trainer that is simply not updating the model.

## Scope

The experiments covered:

- Models: Qwen3.5 4B and 9B.
- Starting points: project M0-v4 checkpoints and the original Qwen3.5 Base.
- Algorithms: DPO, GRPO, and PPO.
- Main training cohort size: 1,000 prompts.
- Evaluation data: NVIDIA Nemotron-RL coding dev/held-out splits and a small
  LiveCodeBench v6 diagnostic subset.
- Checkpoint cadence: every 4 optimizer steps for the later formal and debug
  runs.
- Checkpoint policy: adapter-only artifacts; optimizer, scheduler, and RNG
  states are intentionally excluded from the shared debug package.

## Experiment progression

### 1. Initial M0-v4 runs

The first formal comparison trained DPO, GRPO, and PPO from the 4B and 9B
M0-v4 checkpoints on aligned 1,000-prompt cohorts.

Representative initial configuration:

| Method | LR | Coefficient | LoRA | Length | Batch construction |
|---|---:|---:|---:|---:|---|
| DPO | 1e-6 | beta=0.05 | r=16, alpha=32 | max length 2,048 | micro-batch 1, grad accumulation 8 |
| GRPO | 1e-6 | beta=0.05 | r=16, alpha=32 | completion 1,024 | 8 prompts x 8 rollouts |
| PPO | 1e-6 | initial KL=0.05 | r=16, alpha=32 | response 1,024 | micro-batch 1, grad accumulation 8 |

The four checkpoint-time verifier estimates at steps 4, 12, 40, and 125 were
non-monotonic. For example:

- 4B DPO: 0.2156, 0.2128, 0.2406, 0.2156.
- 4B GRPO: 0.2208, 0.2401, 0.1993, 0.2186.
- 9B GRPO: 0.2554, 0.2346, 0.2376, 0.2387.
- 9B PPO: 0.2450, 0.2328, 0.2542, 0.2257.

These are noisy checkpoint estimates rather than a fixed, large held-out
evaluation, but they show that training reward did not rise consistently.

PPO additionally encountered non-finite gradients late in the 4B run. The
path was instrumented with finite-loss/gradient diagnostics and guarded update
skipping. The recovered run completed 125 steps after skipping eight
non-finite updates, but this established plumbing robustness, not a reliable
performance gain.

### 2. Hyperparameter probes

The explored ranges included:

- Learning rate: 5e-7, 1e-6, 5e-6, and 1e-5.
- DPO/GRPO beta: 0.01, 0.05, and 0.1.
- PPO initial KL coefficient: 0.05 and 0.1.
- LoRA capacity: r=16/alpha=32 and r=32/alpha=64.
- Micro-batch size: 1, 2, 4, and 8, with gradient accumulation adjusted to
  retain the intended effective batch.
- Schedules: default decay and constant-with-warmup variants.
- GRPO completion length: 1,024, 1,536, and 4,096.
- DPO maximum sequence length: 2,048, 2,560, and 6,144.
- Early stopping/checkpoint selection at steps 4, 8, 12, 16, 28, and 40,
  versus training through step 125.

The 9B tuned formal configuration lowered the learning rate to 5e-7, increased
DPO/GRPO beta or PPO initial KL to 0.1, and increased the DPO/rollout length to
2,560/1,536. Training completed, but it did not establish a robust held-out
gain.

For faster GRPO diagnosis, we then compared three 9B step-16 probes:

- Default probe: LR=1e-6, beta=0.05, r=16/alpha=32.
- Aggressive probe: LR=5e-6, beta=0.01, r=16/alpha=32.
- Higher-capacity probe: LR=1e-6, beta=0.05, r=32/alpha=64.

On one fixed 100-prompt held-out diagnostic, the r=32/alpha=64 checkpoint was
the strongest of these probes:

| Model | Mean test-case pass fraction | Strict full-pass rate |
|---|---:|---:|
| 9B M0 | 22.42% | 16% |
| 9B GRPO r32/alpha64 step 16 | 24.48% | 20% |

The mean improvement was +2.05 percentage points, but its bootstrap interval
for partial reward crossed zero. The strict-rate interval was positive on this
small sample, so this checkpoint is useful for debugging but is not sufficient
evidence of a general improvement.

A 4B high-LR DPO probe (LR=1e-5, beta=0.01, r=16/alpha=32) substantially
changed the training objective: the aggregate train loss was 0.563 and late
mini-batch preference accuracies reached 0.8-0.9. This showed that a stronger
optimizer setting can fit the preference pairs, but no corresponding robust
task-performance gain was established.

Increasing batch size primarily improved throughput. It did not by itself
resolve the performance problem.

### 3. Removing truncated and length-confounded DPO pairs

The original M0-derived DPO artifacts each contained 1,000 pairs. We removed
pairs whose chosen or rejected response was near the generation cap and
controlled response-length mismatch. This left:

- 4B M0: 433 length-clean pairs.
- 9B M0: 245 length-clean pairs.

The 4B M0 length-clean run degraded relative to M0 across most early
checkpoints. The 9B run produced a promising but unstable early result:

| Evaluation | Baseline | Checkpoint | Partial reward | Strict full pass |
|---|---:|---:|---:|---:|
| 9B dev, 250 prompts | M0 | step 8 | 35.13% -> 38.51% | 25.2% -> 29.2% |
| 9B held-out, 500 prompts | M0 | step 12 | 33.49% -> 31.48% | 25.4% -> 22.6% |

Thus, filtering removed a real confounder but did not produce a stable
generalization gain. The apparent early dev improvement reversed on the larger
held-out evaluation.

### 4. Switching from M0 to the original Base model

The original preference data was generated and selected around M0 behavior.
We therefore restarted from Qwen/Qwen3.5-4B and regenerated the DPO pairs.

The first Base rollout reused the old 1,000-prompt cohort:

- 8 samples per prompt.
- Temperature 1.0.
- Maximum completion length 4,096.
- Truncation buffer 24 tokens.
- Maximum chosen/rejected token-count difference 512.
- Minimum reward margin 0.5.
- 709 valid pairs were retained.

After correcting the prompt pipeline, a larger Base-model sweep processed
3,081 candidate prompts from the NVIDIA training pool with the same K=8 and
4,096-token rollout settings. It produced 1,174 eligible pairs, from which a
fixed 1,000-pair DPO cohort was frozen.

This final dataset addressed both the M0-conditioned selection issue and the
known training-pair truncation/length confound.

### 5. Final 4B Base DPO run

Final configuration:

| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen3.5-4B, revision 851bf6e806ef... |
| Training pairs | 1,000 Base-generated, clean-prompt pairs |
| Learning rate | 1e-6 |
| Scheduler | constant with 2 warmup steps |
| DPO beta | 0.1 |
| LoRA | r=16, alpha=32, dropout=0 |
| Effective batch | 8 (micro-batch 2 x accumulation 4) |
| Maximum sequence length | 6,144 |
| Maximum prompt length | 1,024 |
| Training duration | 125 steps |
| Save interval | 4 steps |

The run completed in 2,527 seconds with aggregate train loss 0.6922. At step
125, the logged mini-batch loss was 0.6631 and preference accuracy was 0.625.
This is a weak but non-zero optimization signal.

#### NVIDIA held-out diagnostic: 100 fixed prompts

The same 100 prompts were evaluated for Base, step 28, and step 125 with one
deterministic rollout and an 8,192-token completion cap.

| Model | Mean test-case pass fraction | Strict full-pass rate | Non-zero reward | Truncation |
|---|---:|---:|---:|---:|
| Base | 29.62% | 22% | 43% | 49% |
| Step 28 | 32.44% | 22% | 48% | 43% |
| Step 125 | 27.06% | 17% | 46% | 44% |

Step 28 improved partial reward by +2.82 percentage points relative to Base,
with paired bootstrap 95% interval [+0.05, +6.30]. It did not increase the
number of fully solved prompts. Step 125 was worse than Base by 2.55 points in
partial reward and 5 points in strict full-pass rate.

The 43-49% truncation rates remain a major validity problem even at an 8K
completion cap. These numbers must therefore be treated as provisional.

#### LiveCodeBench v6 diagnostic: 20 fixed problems

The same first 20 sorted release-v6 problems were run with n=1, temperature 0,
a 32,768-token completion budget, and merged adapters.

| Model | pass@1 | Truncation |
|---|---:|---:|
| Base | 11/20 (55%) | 0% |
| Step 28 | 8/20 (40%) | 10% |
| Step 125 | 10/20 (50%) | 0% |

The held-out partial-reward signal at step 28 did not reproduce on this small
external benchmark. The sample is too small for a definitive benchmark claim,
but it provides no support for selecting step 28 as a generally better coding
model.

The evaluation loader was also fixed during this stage. Qwen3.5 PEFT adapters
are merged into the language tower and then placed back into the full Qwen3.5
container before direct vLLM evaluation. Tensor-level checks confirmed that
Base, step 28, and step 125 used distinct weights.

### 6. Base-model GRPO attempt

A separate 4B Base GRPO experiment randomly sampled 1,000 prompts rather than
using the old M0-selected subset. Its main configuration was LR=5e-6,
beta=0.01, K=8, 20 unique prompts per step, 4,096-token completions, and a
larger batch/accumulation arrangement to use the available H200 efficiently.
The run was stopped after step 39 when priority shifted to DPO rollout and DPO
training. It is incomplete and should not be interpreted as a negative GRPO
result.

## Interpretation

The experiments rule out several simple explanations:

- The adapters are not identical to Base; the weights genuinely changed.
- The trainers are not completely signal-free; gradients, DPO margins, GRPO
  reward variance, and preference accuracy move.
- The problem is not solved by a single learning-rate, beta, batch-size, or
  LoRA-rank change.
- Removing obviously truncated pairs is necessary but not sufficient.
- Training longer is not consistently beneficial; early checkpoints are often
  better than step 125.

The remaining leading hypotheses are:

1. Preference-pair noise or mismatch between pair selection and the final
   full-verifier objective.
2. A weak/sparse reward signal that is easy to fit locally but does not improve
   complete solutions.
3. Length and termination behavior that remains pathological at evaluation
   time, even after cleaning the training pairs.
4. Overtraining on a 1,000-prompt cohort, especially after the earliest useful
   updates.
5. Evaluation variance from single-rollout measurements and small benchmark
   subsets.

## Recommended next diagnostics

1. Inspect paired Base/step-28 generations on prompts where partial reward
   changed, separating real algorithmic improvements from formatting or
   termination effects.
2. Evaluate Base and early checkpoints with multiple rollouts per prompt and a
   termination policy that eliminates cap-driven censoring.
3. Audit the verifier-derived DPO pairs for semantic correctness, not only
   reward margin and length validity.
4. Compare gradient/update scale and KL-to-Base across steps 4-32 to determine
   whether useful behavior is overwritten later.
5. Run a larger LiveCodeBench subset only after the termination behavior is
   controlled.

## Shared artifact selection

The accompanying Hugging Face debug repository contains representative
adapter-only checkpoints rather than every saved point:

- 4B Base DPO clean-prompt: steps 28 and 125.
- 9B length-clean DPO: steps 8 and 12, capturing the early-dev improvement and
  subsequent held-out reversal.
- 9B GRPO step-16 probes: default, aggressive LR/beta, and r32/alpha64.
- Run metadata, trainer states, selected reward logs, evaluation summaries,
  configs, and the relevant training/evaluation utility code.

Optimizer, scheduler, RNG, duplicated reference adapters, tokenizer copies,
rollout text dumps, and full merged-model evaluation caches are intentionally
excluded.

	---
	library_name: peft
	license: apache-2.0
	tags:
	- qwen3.5
	- reinforcement-learning
	- dpo
	- grpo
	- ppo
	- debugging
	base_model:
	- Qwen/Qwen3.5-4B
	- prism-drift/qwen35-4b-m0-v4
	- prism-drift/qwen35-9b-m0-v4
	---

	# RL Training Debugging Report

	Date: 2026-08-26
	Status: diagnostic; no robust performance improvement established

	## Executive summary

	We ran a sequence of DPO, GRPO, and PPO experiments on Qwen3.5 4B and 9B,
	first from project M0 checkpoints and later directly from the original
	Qwen/Qwen3.5-4B base model. We varied the learning rate, KL/DPO/GRPO
	coefficient, LoRA capacity, batch construction, sequence length, training
	duration, and dataset filtering rules. We also rebuilt the 4B DPO data with
	fresh Base-model rollouts after determining that the original preference
	pairs were tied to M0 and contained substantial truncation and length bias.

	The training paths are operational: gradients are non-zero, DPO preference
	accuracy and margins move, GRPO has non-zero reward variance, PPO can complete
	with non-finite-gradient guards, and merged adapters contain weights that are
	different from both the base model and one another. However, these optimization
	signals have not translated into a stable improvement on held-out verifier
	reward or LiveCodeBench. A few early checkpoints improved small development
	subsets, but the gains either disappeared on a larger held-out split or
	reversed on an external benchmark. Later checkpoints often performed worse.

	The current evidence is therefore consistent with noisy or mismatched
	preference/reward supervision, checkpoint overtraining, and generation-length
	pathologies rather than a trainer that is simply not updating the model.

	## Scope

	The experiments covered:

	- Models: Qwen3.5 4B and 9B.
	- Starting points: project M0-v4 checkpoints and the original Qwen3.5 Base.
	- Algorithms: DPO, GRPO, and PPO.
	- Main training cohort size: 1,000 prompts.
	- Evaluation data: NVIDIA Nemotron-RL coding dev/held-out splits and a small
	LiveCodeBench v6 diagnostic subset.
	- Checkpoint cadence: every 4 optimizer steps for the later formal and debug
	runs.
	- Checkpoint policy: adapter-only artifacts; optimizer, scheduler, and RNG
	states are intentionally excluded from the shared debug package.

	## Experiment progression

	### 1. Initial M0-v4 runs

	The first formal comparison trained DPO, GRPO, and PPO from the 4B and 9B
	M0-v4 checkpoints on aligned 1,000-prompt cohorts.

	Representative initial configuration:

	\| Method \| LR \| Coefficient \| LoRA \| Length \| Batch construction \|
	\|---\|---:\|---:\|---:\|---:\|---\|
	\| DPO \| 1e-6 \| beta=0.05 \| r=16, alpha=32 \| max length 2,048 \| micro-batch 1, grad accumulation 8 \|
	\| GRPO \| 1e-6 \| beta=0.05 \| r=16, alpha=32 \| completion 1,024 \| 8 prompts x 8 rollouts \|
	\| PPO \| 1e-6 \| initial KL=0.05 \| r=16, alpha=32 \| response 1,024 \| micro-batch 1, grad accumulation 8 \|

	The four checkpoint-time verifier estimates at steps 4, 12, 40, and 125 were
	non-monotonic. For example:

	- 4B DPO: 0.2156, 0.2128, 0.2406, 0.2156.
	- 4B GRPO: 0.2208, 0.2401, 0.1993, 0.2186.
	- 9B GRPO: 0.2554, 0.2346, 0.2376, 0.2387.
	- 9B PPO: 0.2450, 0.2328, 0.2542, 0.2257.

	These are noisy checkpoint estimates rather than a fixed, large held-out
	evaluation, but they show that training reward did not rise consistently.

	PPO additionally encountered non-finite gradients late in the 4B run. The
	path was instrumented with finite-loss/gradient diagnostics and guarded update
	skipping. The recovered run completed 125 steps after skipping eight
	non-finite updates, but this established plumbing robustness, not a reliable
	performance gain.

	### 2. Hyperparameter probes

	The explored ranges included:

	- Learning rate: 5e-7, 1e-6, 5e-6, and 1e-5.
	- DPO/GRPO beta: 0.01, 0.05, and 0.1.
	- PPO initial KL coefficient: 0.05 and 0.1.
	- LoRA capacity: r=16/alpha=32 and r=32/alpha=64.
	- Micro-batch size: 1, 2, 4, and 8, with gradient accumulation adjusted to
	retain the intended effective batch.
	- Schedules: default decay and constant-with-warmup variants.
	- GRPO completion length: 1,024, 1,536, and 4,096.
	- DPO maximum sequence length: 2,048, 2,560, and 6,144.
	- Early stopping/checkpoint selection at steps 4, 8, 12, 16, 28, and 40,
	versus training through step 125.

	The 9B tuned formal configuration lowered the learning rate to 5e-7, increased
	DPO/GRPO beta or PPO initial KL to 0.1, and increased the DPO/rollout length to
	2,560/1,536. Training completed, but it did not establish a robust held-out
	gain.

	For faster GRPO diagnosis, we then compared three 9B step-16 probes:

	- Default probe: LR=1e-6, beta=0.05, r=16/alpha=32.
	- Aggressive probe: LR=5e-6, beta=0.01, r=16/alpha=32.
	- Higher-capacity probe: LR=1e-6, beta=0.05, r=32/alpha=64.

	On one fixed 100-prompt held-out diagnostic, the r=32/alpha=64 checkpoint was
	the strongest of these probes:

	\| Model \| Mean test-case pass fraction \| Strict full-pass rate \|
	\|---\|---:\|---:\|
	\| 9B M0 \| 22.42% \| 16% \|
	\| 9B GRPO r32/alpha64 step 16 \| 24.48% \| 20% \|

	The mean improvement was +2.05 percentage points, but its bootstrap interval
	for partial reward crossed zero. The strict-rate interval was positive on this
	small sample, so this checkpoint is useful for debugging but is not sufficient
	evidence of a general improvement.

	A 4B high-LR DPO probe (LR=1e-5, beta=0.01, r=16/alpha=32) substantially
	changed the training objective: the aggregate train loss was 0.563 and late
	mini-batch preference accuracies reached 0.8-0.9. This showed that a stronger
	optimizer setting can fit the preference pairs, but no corresponding robust
	task-performance gain was established.

	Increasing batch size primarily improved throughput. It did not by itself
	resolve the performance problem.

	### 3. Removing truncated and length-confounded DPO pairs

	The original M0-derived DPO artifacts each contained 1,000 pairs. We removed
	pairs whose chosen or rejected response was near the generation cap and
	controlled response-length mismatch. This left:

	- 4B M0: 433 length-clean pairs.
	- 9B M0: 245 length-clean pairs.

	The 4B M0 length-clean run degraded relative to M0 across most early
	checkpoints. The 9B run produced a promising but unstable early result:

	\| Evaluation \| Baseline \| Checkpoint \| Partial reward \| Strict full pass \|
	\|---\|---:\|---:\|---:\|---:\|
	\| 9B dev, 250 prompts \| M0 \| step 8 \| 35.13% -> 38.51% \| 25.2% -> 29.2% \|
	\| 9B held-out, 500 prompts \| M0 \| step 12 \| 33.49% -> 31.48% \| 25.4% -> 22.6% \|

	Thus, filtering removed a real confounder but did not produce a stable
	generalization gain. The apparent early dev improvement reversed on the larger
	held-out evaluation.

	### 4. Switching from M0 to the original Base model

	The original preference data was generated and selected around M0 behavior.
	We therefore restarted from Qwen/Qwen3.5-4B and regenerated the DPO pairs.

	The first Base rollout reused the old 1,000-prompt cohort:

	- 8 samples per prompt.
	- Temperature 1.0.
	- Maximum completion length 4,096.
	- Truncation buffer 24 tokens.
	- Maximum chosen/rejected token-count difference 512.
	- Minimum reward margin 0.5.
	- 709 valid pairs were retained.

	After correcting the prompt pipeline, a larger Base-model sweep processed
	3,081 candidate prompts from the NVIDIA training pool with the same K=8 and
	4,096-token rollout settings. It produced 1,174 eligible pairs, from which a
	fixed 1,000-pair DPO cohort was frozen.

	This final dataset addressed both the M0-conditioned selection issue and the
	known training-pair truncation/length confound.

	### 5. Final 4B Base DPO run

	Final configuration:

	\| Parameter \| Value \|
	\|---\|---\|
	\| Base model \| Qwen/Qwen3.5-4B, revision 851bf6e806ef... \|
	\| Training pairs \| 1,000 Base-generated, clean-prompt pairs \|
	\| Learning rate \| 1e-6 \|
	\| Scheduler \| constant with 2 warmup steps \|
	\| DPO beta \| 0.1 \|
	\| LoRA \| r=16, alpha=32, dropout=0 \|
	\| Effective batch \| 8 (micro-batch 2 x accumulation 4) \|
	\| Maximum sequence length \| 6,144 \|
	\| Maximum prompt length \| 1,024 \|
	\| Training duration \| 125 steps \|
	\| Save interval \| 4 steps \|

	The run completed in 2,527 seconds with aggregate train loss 0.6922. At step
	125, the logged mini-batch loss was 0.6631 and preference accuracy was 0.625.
	This is a weak but non-zero optimization signal.

	#### NVIDIA held-out diagnostic: 100 fixed prompts

	The same 100 prompts were evaluated for Base, step 28, and step 125 with one
	deterministic rollout and an 8,192-token completion cap.

	\| Model \| Mean test-case pass fraction \| Strict full-pass rate \| Non-zero reward \| Truncation \|
	\|---\|---:\|---:\|---:\|---:\|
	\| Base \| 29.62% \| 22% \| 43% \| 49% \|
	\| Step 28 \| 32.44% \| 22% \| 48% \| 43% \|
	\| Step 125 \| 27.06% \| 17% \| 46% \| 44% \|

	Step 28 improved partial reward by +2.82 percentage points relative to Base,
	with paired bootstrap 95% interval [+0.05, +6.30]. It did not increase the
	number of fully solved prompts. Step 125 was worse than Base by 2.55 points in
	partial reward and 5 points in strict full-pass rate.

	The 43-49% truncation rates remain a major validity problem even at an 8K
	completion cap. These numbers must therefore be treated as provisional.

	#### LiveCodeBench v6 diagnostic: 20 fixed problems

	The same first 20 sorted release-v6 problems were run with n=1, temperature 0,
	a 32,768-token completion budget, and merged adapters.

	\| Model \| pass@1 \| Truncation \|
	\|---\|---:\|---:\|
	\| Base \| 11/20 (55%) \| 0% \|
	\| Step 28 \| 8/20 (40%) \| 10% \|
	\| Step 125 \| 10/20 (50%) \| 0% \|

	The held-out partial-reward signal at step 28 did not reproduce on this small
	external benchmark. The sample is too small for a definitive benchmark claim,
	but it provides no support for selecting step 28 as a generally better coding
	model.

	The evaluation loader was also fixed during this stage. Qwen3.5 PEFT adapters
	are merged into the language tower and then placed back into the full Qwen3.5
	container before direct vLLM evaluation. Tensor-level checks confirmed that
	Base, step 28, and step 125 used distinct weights.

	### 6. Base-model GRPO attempt

	A separate 4B Base GRPO experiment randomly sampled 1,000 prompts rather than
	using the old M0-selected subset. Its main configuration was LR=5e-6,
	beta=0.01, K=8, 20 unique prompts per step, 4,096-token completions, and a
	larger batch/accumulation arrangement to use the available H200 efficiently.
	The run was stopped after step 39 when priority shifted to DPO rollout and DPO
	training. It is incomplete and should not be interpreted as a negative GRPO
	result.

	## Interpretation

	The experiments rule out several simple explanations:

	- The adapters are not identical to Base; the weights genuinely changed.
	- The trainers are not completely signal-free; gradients, DPO margins, GRPO
	reward variance, and preference accuracy move.
	- The problem is not solved by a single learning-rate, beta, batch-size, or
	LoRA-rank change.
	- Removing obviously truncated pairs is necessary but not sufficient.
	- Training longer is not consistently beneficial; early checkpoints are often
	better than step 125.

	The remaining leading hypotheses are:

	1. Preference-pair noise or mismatch between pair selection and the final
	full-verifier objective.
	2. A weak/sparse reward signal that is easy to fit locally but does not improve
	complete solutions.
	3. Length and termination behavior that remains pathological at evaluation
	time, even after cleaning the training pairs.
	4. Overtraining on a 1,000-prompt cohort, especially after the earliest useful
	updates.
	5. Evaluation variance from single-rollout measurements and small benchmark
	subsets.

	## Recommended next diagnostics

	1. Inspect paired Base/step-28 generations on prompts where partial reward
	changed, separating real algorithmic improvements from formatting or
	termination effects.
	2. Evaluate Base and early checkpoints with multiple rollouts per prompt and a
	termination policy that eliminates cap-driven censoring.
	3. Audit the verifier-derived DPO pairs for semantic correctness, not only
	reward margin and length validity.
	4. Compare gradient/update scale and KL-to-Base across steps 4-32 to determine
	whether useful behavior is overwritten later.
	5. Run a larger LiveCodeBench subset only after the termination behavior is
	controlled.

	## Shared artifact selection

	The accompanying Hugging Face debug repository contains representative
	adapter-only checkpoints rather than every saved point:

	- 4B Base DPO clean-prompt: steps 28 and 125.
	- 9B length-clean DPO: steps 8 and 12, capturing the early-dev improvement and
	subsequent held-out reversal.
	- 9B GRPO step-16 probes: default, aggressive LR/beta, and r32/alpha64.
	- Run metadata, trainer states, selected reward logs, evaluation summaries,
	configs, and the relevant training/evaluation utility code.

	Optimizer, scheduler, RNG, duplicated reference adapters, tokenizer copies,
	rollout text dumps, and full merged-model evaluation caches are intentionally
	excluded.