docs(readme): align with v2 GRPO run — correct W&B URL, weights, narrative

- Point W&B link to y82wmj4x (lucky-capybara-5) where reward climbed
0.15 → 0.98 over 70 steps. Was sy1tqun0 (the v1 flat-line run).
- Update reward weights table 0.75/0.20/0.05 → 0.70/0.15/0.15 to match
configs/reward/v1.yaml, and document the soft +0.10 format bonus
(was a hard multiplier in v1).
- Rewrite GRPO Training Evidence section to describe the three v2 fixes
(prompt alignment, soft format bonus, FATHOM_USE_VLLM=0) and the
resulting bimodal cold-start → discovery-phase reward curve.
- Document the FATHOM_USE_VLLM gate in Training Pipeline section.
- Remove the GitHub mirror link (404, never created); link to Colab
notebook via the HF code repo so it actually opens for judges.
- Drop the GitHub mirror checklist item.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

README.md

@@ -15,12 +15,11 @@ Submitted to the Meta × PyTorch × Hugging Face OpenEnv Hackathon Grand Finale
 | **Demo UI (Streamlit Space)** | <https://huggingface.co/spaces/Pratham-math/fathom-demo> |
 | **Demo URL (live)** | <https://Pratham-math-fathom-demo.hf.space> |
 | **Health check** | <https://Pratham-math-fathom-env.hf.space/healthz> |
-| **Code repo (HF)** | <https://huggingface.co/Pratham-math/fathom-code> |
-| **Code repo (GitHub mirror)** | <https://github.com/Pratham-math/fathom> |
+| **Code repo (HF — primary)** | <https://huggingface.co/Pratham-math/fathom-code> |
 | **Trained model + training plots** | <https://huggingface.co/Pratham-math/fathom-1.5b-grpo> |
-| **Colab reproducer notebook** | [`notebooks/fathom_train.ipynb`](notebooks/fathom_train.ipynb) (in-repo, also openable from GitHub mirror once added) |
+| **Colab reproducer notebook** | [`notebooks/fathom_train.ipynb`](https://huggingface.co/Pratham-math/fathom-code/blob/main/notebooks/fathom_train.ipynb) (open via HF code repo) |
 | **Mini-blog** | <https://huggingface.co/spaces/Pratham-math/fathom-blog> |
-| **W&B training run** | <https://wandb.ai/pratham-alwar05-indian-institute-of-information-technolo/huggingface/runs/sy1tqun0> |
+| **W&B training run (v2 — successful learning)** | <https://wandb.ai/pratham-alwar05-indian-institute-of-information-technolo/huggingface/runs/y82wmj4x> |
 
 > The HF Space `/healthz` endpoint cold-starts the first time it's hit; if you get a 503, refresh once and it returns 200.
 
@@ -80,10 +79,10 @@ Deterministic, composable, no LLM-as-judge in the training loop. Every task in o
 
 | Weight | Component | Source | What it scores |
 |--------|-----------|--------|----------------|
-| **gate** | `format_gate.py` | gate (multiplier) | `<answer>…</answer>` tags present and well-formed |
-| 0.75 | `correctness.py` | additive | Normalised exact-match against gold |
-| 0.20 | `token_budget.py` | penalty | Total tool-call tokens (Mercor sub-prize aligned) |
-| 0.05 | `recursion_efficiency.py` | additive | Recursion depth used / depth required |
+| +0.10 bonus | `format_gate.py` | additive (soft) | `<answer>…</answer>` tags present (v2 — was a hard multiplier in v1) |
+| 0.70 | `correctness.py` | additive | Normalised exact-match against gold |
+| 0.15 | `token_budget.py` | penalty | Total tool-call tokens (Mercor sub-prize aligned) |
+| 0.15 | `recursion_efficiency.py` | additive (correctness-gated) | Linear decay on `llm()` call count, only counts when answer is correct |
 
 Composition: `rewards/compose.py` (`make_reward_fn`) wraps each component, logs each scalar separately to W&B (`reward/format_pass_mean`, `reward/correctness_mean`, etc.), and exposes the composite to TRL's `GRPOTrainer.reward_funcs` interface.
 
@@ -116,7 +115,7 @@ python -m train.smoke_test --env-url http://localhost:8001
 
 - [`train/model_load.py`](train/model_load.py) — Unsloth-with-HF-fallback loader
 - [`train/sft.py`](train/sft.py) — TRL `SFTTrainer` warm-start
-- [`train/grpo.py`](train/grpo.py) — TRL `GRPOTrainer` with `vllm_mode='colocate'` (per [TRL #4543](https://github.com/huggingface/trl/issues/4543))
+- [`train/grpo.py`](train/grpo.py) — TRL `GRPOTrainer`. Rollout backend is gated by `FATHOM_USE_VLLM` env var: defaults to `vllm_mode='colocate'` (per [TRL #4543](https://github.com/huggingface/trl/issues/4543)); set `FATHOM_USE_VLLM=0` to fall back to HF `generate()` for QLoRA-stable rollouts (avoids the IS-ratio collapse from merged-4bit weight drift).
 - [`train/smoke_test.py`](train/smoke_test.py) — 6-check pipeline gate
 
 ### 4) Hyperparameters (from [`configs/train/grpo.yaml`](configs/train/grpo.yaml))
@@ -131,10 +130,12 @@ max_prompt_length: 4096
 max_completion_length: 2048
 optim: adamw_8bit
 max_steps: 400                # overridable per-path: 50 sanity / 100 conservative / 400 aggressive
-vllm_mode: colocate
+vllm_mode: colocate           # used when FATHOM_USE_VLLM=1 (default)
 vllm_gpu_memory_utilization: 0.45
 ```
 
+**v2 run note:** the successful `y82wmj4x` W&B run was launched with `FATHOM_USE_VLLM=0` and `train.max_steps=200`. The QLoRA + vLLM colocate path is left in for users on bf16 LoRA who don't hit the IS-ratio drift.
+
 ---
 
 ## Training Evidence
@@ -147,30 +148,30 @@ vllm_gpu_memory_utilization: 0.45
 ![SFT token accuracy](https://huggingface.co/Pratham-math/fathom-1.5b-grpo/resolve/main/plots/sft_token_accuracy.png)
 *Mean per-token accuracy climbs from 0.46 → 0.93 over the SFT epoch — confirms the warm-start adapter generates the correct answer span ~93% of the time on training data.*
 
-### What we learned about reward design (GRPO pipeline)
+### GRPO learning — v2 run after fixing prompt drift, format gating, and IS-ratio collapse
 
-The SFT phase taught the format; GRPO did not converge in our budget, but the env, reward, and pipeline are nonetheless complete and reproducible.
+Our v1 GRPO run produced a flat reward curve. We diagnosed three compounding causes — (a) prompt-shape drift between the SFT user template (`Question: …\n\n[Document excerpt]:\n…`) and the GRPO user template (`Context: …\n\n…`), (b) a multiplicative format gate that zeroed the composite reward whenever the model dropped the `<answer>…</answer>` wrapper, and (c) importance-sampling-ratio collapse from QLoRA + vLLM colocate weight-merge drift — and fixed all three:
 
-The pipeline runs end-to-end on the OpenEnv server, but the reward curve in our initial run was flat:
+1. Aligned the GRPO prompt template byte-for-byte with the SFT chat template.
+2. Replaced the multiplicative format gate with an additive +0.10 soft format bonus, capped at 0.25 when correctness is 0 to block format-only exploits.
+3. Added `FATHOM_USE_VLLM=0` to fall back to HF `generate()` for rollouts so QLoRA's merged-4bit drift no longer triggers TRL's IS-ratio clipping.
 
 ![GRPO reward curve](https://huggingface.co/Pratham-math/fathom-1.5b-grpo/resolve/main/plots/grpo_reward.png)
-*Composite reward curve over 50 GRPO steps for Qwen 2.5 Coder + LoRA on the FATHOM env. β=0.04, lr=5e-6, 8 generations / step. Reward stays at 0.0 — see the diagnosis below.*
-
-![GRPO completion length](https://huggingface.co/Pratham-math/fathom-1.5b-grpo/resolve/main/plots/grpo_completion_length.png)
-*Completion length stays at 2–13 tokens through training: the model is producing bare answer spans like `the man.` instead of the wrapped `<answer>the man</answer>` that `format_gate.py` requires. Because format gate is a multiplier, the entire composite reward is zeroed out.*
+*Composite reward over the v2 GRPO run. After ~45 cold-start steps where the model only earns the soft format bonus (~0.15), the policy discovers the correct-answer mode and reward climbs to 0.6–0.98 with healthy variance (std 0.30–0.50). β=0.0, lr=5e-6, 8 generations/step, HF generate() rollout.*
 
 ![Training summary (8-panel)](https://huggingface.co/Pratham-math/fathom-1.5b-grpo/resolve/main/plots/training_summary.png)
 *All 8 GRPO metrics on one canvas — loss, reward, KL, entropy, grad norm, completion length, learning rate, advantage variance.*
 
 **What this run proves**
 
-1. The OpenEnv environment, sandboxed REPL, GRPO trainer, vLLM colocate rollout, and HF Hub model push all work end-to-end on a real cloud GPU.
-2. The SFT phase achieves a 91% reduction in loss and 2× token-accuracy improvement, demonstrating the warm-start adapter is fit for purpose.
-3. The flat GRPO reward exposes a real reward-design lesson: a multiplicative format gate without a soft-format prior collapses GRPO when the policy strays even slightly from the templated output. Our repo includes the code fixes for the next run (prompt alignment and an additive format bonus), but GRPO did not converge in our hackathon budget. The SFT improvements prove the pipeline is fundamentally sound.
+1. The OpenEnv environment, sandboxed REPL, GRPO trainer, HF generate() rollout, deterministic verifier, and HF Hub model push all work end-to-end on a real cloud GPU.
+2. The SFT warm-start achieves a 91% loss reduction (3.20 → 0.29) and 2× token-accuracy gain (0.46 → 0.93), demonstrating the chat-template + format internalisation works.
+3. The v2 GRPO reward curve exhibits a textbook bimodal cold-start: a flat ~0.15 floor for ~45 steps as the policy practises the format alone, followed by a discovery phase where reward spikes to 0.86–0.98 as some of the 8 group generations land the correct answer and create a non-zero GRPO advantage. From step ~60 onward the high-reward steps are frequent enough to drive the policy toward the correct mode.
+4. The reward design (additive 4-component verifier with correctness-gated efficiency) survives 8 audited adversarial probes (`REWARD_AUDIT.md`). No LLM-as-judge is in the training loop — every gold answer is deterministic.
 
-W&B run (full metric history, 113 steps): <https://wandb.ai/pratham-alwar05-indian-institute-of-information-technolo/huggingface/runs/sy1tqun0>
+W&B v2 run (full metric history, ~80 steps): <https://wandb.ai/pratham-alwar05-indian-institute-of-information-technolo/huggingface/runs/y82wmj4x>
 
-All 10 plot PNGs are also published at <https://huggingface.co/Pratham-math/fathom-1.5b-grpo/tree/main/plots>.
+All plot PNGs are also published at <https://huggingface.co/Pratham-math/fathom-1.5b-grpo/tree/main/plots>.
 
 ---
 
@@ -235,10 +236,9 @@ pytest -q
 - [x] REWARD_AUDIT.md (5 adversarial attacks neutralised)
 - [x] Smoke test green on HF Jobs (`SMOKE_RESULT.md`)
 - [x] Submission preflight passes (`python scripts/submission_preflight.py`)
-- [x] Loss + reward plot PNGs from a real GRPO run (10 PNGs in `outputs/plots/`, mirrored to the model repo)
-- [x] W&B training run linked in Submission Links
-- [x] Mini-blog / video / slide deck link added to Submission Links
-- [x] GitHub mirror URL added to Submission Links
+- [x] Loss + reward plot PNGs from a real GRPO run (12 PNGs on the model repo)
+- [x] W&B training run linked in Submission Links (v2 run `y82wmj4x` showing learning from 0.15 → 0.98)
+- [x] Mini-blog linked in Submission Links
 
 ---