HACKATHON_FINAL_PLAN.md

# FATHOM — End-to-End Hackathon Submission Plan

**For Cursor:** This is a complete execution plan. Read sections 0-1 fully before starting. Then execute Phase A through Phase E in order. Each command has an expected output. If a command fails, follow the inline fallback. **Do not skip the verification steps after each phase.**

---

## 0. PROJECT CONTEXT (read this first)

### What is this project?

**FATHOM — First RL-Trained Recursive Language Model.** Submission for the Meta × PyTorch × Hugging Face OpenEnv Hackathon Grand Finale (Bangalore, April 25–26 2026, Theme 2 — Long-Horizon Planning). We built an OpenEnv environment that teaches a small Qwen 2.5 Coder model (1.5B params, 4-bit + LoRA) to recursively call itself on long documents using a Python REPL + an `llm()` primitive, so it can solve QA tasks on contexts 50× larger than its native window.

**Core narrative:** A 1.5B model trained on our env solves 200K-token QA tasks via recursion — clean reward curve, recursion-tree viz, Pareto frontier of accuracy-vs-tokens.

### Current state (verified facts)

- **HF Space deployed and live:** `https://Pratham-math-fathom-env.hf.space` — `/healthz` returns 200, `/reset` returns valid Observation
- **Code repo on HF:** `https://huggingface.co/Pratham-math/fathom-code` (model-type repo, not Space — used as a code distribution endpoint)
- **Phase 0 done:** env scaffold, REPL sandbox (RestrictedPython + subprocess), llm() primitive
- **Phase 1 done:** training code (`train/sft.py`, `train/grpo.py`, `train/model_load.py`, `train/smoke_test.py`), reward components (`rewards/format_gate.py`, `rewards/correctness.py`, `rewards/token_budget.py`, `rewards/recursion_efficiency.py`, `rewards/compose.py`), 1000+200+500 dataset, REWARD_AUDIT.md with 5 attacks, viz/app.py Streamlit skeleton
- **Smoke test PASSED on HF Jobs (Linux + a10g-large)** — `outputs/smoke/SMOKE_RESULT.md` shows VERDICT: GO with 6/6 checks PASS in 47 seconds, against the live HF Space env
- **Preflight passed:** `python scripts/submission_preflight.py` returns "Submission package looks judge-ready"

### What's untracked locally (Cursor: commit these in Phase A)

```
M README.md
M scripts/deploy_env_space.sh
M scripts/job_smoke.sh
M scripts/job_train.sh
M train/grpo.py
M train/model_load.py
M train/smoke_test.py
?? SMOKE_RESULT.md
?? scripts/deploy_training.py
?? scripts/run_training.py
?? scripts/submission_preflight.py
?? scripts/job_sft_only.sh
?? scripts/make_plots.py
?? notebooks/fathom_train.ipynb
?? space/Dockerfile.train
?? space/README_train.md
```

### Constraints

- **Time:** approximately 5–6 hours from now until submission
- **HF credits:** $30 available (use ~$20 for training, keep $10 as safety net)
- **Hardware:** Local machine is Windows + RTX 4060 8GB (cannot run 1.5B locally) — all training MUST run on HF Jobs
- **Claude credits:** very limited, prefer to make Cursor do the heavy lifting from this plan
- **HF username:** `Pratham-math`
- **HF Space (env):** `Pratham-math/fathom-env`
- **HF code repo:** `Pratham-math/fathom-code`
- **HF model repo (will be created):** `Pratham-math/fathom-1.5b-grpo`

### Hackathon judging weights (target every point)

| Criterion | Weight | What we ship |
|-----------|--------|--------------|
| Environment Innovation | 40% | First publicly-deployed OpenEnv RL env for Recursive LMs (no prior art) |
| Storytelling & Presentation | 30% | README + 60–120s YouTube video + mini-blog on HF |
| Showing Improvement in Rewards | 20% | Reward + loss PNG curves embedded in README + W&B link |
| Reward & Training Pipeline | 10% | REWARD_AUDIT.md (5 attacks), 30+ unit tests, composable reward, smoke green |

### Minimum non-negotiable submission requirements (verify each before submitting)

- [ ] Uses OpenEnv latest (`openenv-core>=0.2.3`)
- [ ] Working training script using Unsloth + TRL — `train/grpo.py` ✓
- [ ] Colab notebook so judges can re-run — `notebooks/fathom_train.ipynb` (already drafted, must be tested)
- [ ] Loss + reward plot PNGs from a real run — to be generated by Phase B + Phase D
- [ ] Mini-blog OR <2min YouTube video OR slide deck — Phase C must produce one of these
- [ ] Env deployed to HF Space ✓ (`Pratham-math/fathom-env`)
- [ ] README with motivation + env explanation + results
- [ ] README links to HF Space + all materials

---

## 1. KEY FILES YOU WILL TOUCH

| File | Purpose | State |
|------|---------|-------|
| `scripts/job_train.sh` | Main HF Job that runs SFT → GRPO → plots → push | Updated, needs commit |
| `scripts/job_sft_only.sh` | Fallback SFT-only path (~30 min) | Created, needs commit |
| `scripts/make_plots.py` | Generates PNGs from `trainer_state.json` | Created, needs commit |
| `scripts/submission_preflight.py` | Validates README + manifest + Dockerfile | Working |
| `notebooks/fathom_train.ipynb` | Colab reproducer for judges | Drafted, needs commit + test |
| `README.md` | Submission landing page | Needs plots + Colab link added in Phase D |
| `viz/app.py` | Streamlit demo | Skeleton only, polish in Phase C |
| `configs/train/grpo.yaml` | GRPO hyperparams | Read-only — already correct |
| `configs/model/qwen_1_5b.yaml` | 1.5B model spec | Read-only — already correct |

---

## PHASE A — COMMIT, PUSH, MIRROR (target: 15 min)

**Goal:** Freeze current good state on HF + GitHub. Nothing in this phase touches training; it's pure version control.

### A.1 Verify current state

```bash
git status --short
ls scripts/make_plots.py scripts/job_sft_only.sh notebooks/fathom_train.ipynb
python scripts/submission_preflight.py
```

**Expected:**
- `git status` shows the modified + untracked list from section 0
- `ls` finds all three files
- preflight prints `Preflight passed. Submission package looks judge-ready.`

**If preflight fails:** read which check failed, fix the README section it points at, re-run.

### A.2 Commit everything

```bash
git add -A
git commit -m "feat: phase 1 complete — smoke green on HF Jobs, training scripts, plot generator, Colab notebook, submission preflight"
```

### A.3 Push to HF (master branch)

```bash
git push hf master
```

**Expected:** `master -> master` push succeeds. If you see "secret detected" — find the offending file, sanitize the token to `os.environ['HF_TOKEN']`, recommit, push.

### A.4 Create GitHub mirror (judges check public repos)

```bash
# Authenticate gh first if needed
gh auth status || gh auth login

# Create + push
gh repo create fathom-openenv --public --source=. --push --description="FATHOM — First RL-trained Recursive Language Model. OpenEnv environment + GRPO training pipeline for Qwen 2.5 Coder 1.5B."
```

**If `gh` CLI not installed:**
```bash
# Manually create repo at https://github.com/new (name: fathom-openenv, public)
git remote add github https://github.com/<your-github-user>/fathom-openenv
git push -u github master
```

**Capture the GitHub URL** — you will paste it into the README + submission form.

### A.5 Verify A is complete

```bash
# Both remotes accessible?
git remote -v
# HF repo browsable?
curl -sI https://huggingface.co/Pratham-math/fathom-code | head -1
# Live env still alive?
curl -sf https://Pratham-math-fathom-env.hf.space/healthz && echo " env OK"
```

All three must succeed before continuing to Phase B.

---

## PHASE B — FIRE THE TRAINING JOB (target: 5 min setup + 5h background)

**Goal:** Get a real reward curve. This is the 20% rubric criterion. We use 1.5B GRPO at reduced step count to fit budget + risk.

### B.1 Strategy (do not skip this decision)

You have two paths. Pick ONE based on time remaining:

**Path 1 — Aggressive (5h, ~$20 of $30 credits):**
1.5B + GRPO 400 steps + SFT warm-start on a100-large. Highest-quality demo if it works.

**Path 2 — Conservative (1.5h, ~$6 of $30 credits):**
1.5B + GRPO 100 steps + SFT warm-start on a100-large. Still produces a reward curve; less convergence but enough for the plot.

**Path 3 — Safe (40 min, ~$0.50 of $30 credits):**
0.5B + GRPO 50 steps on a10g-large. Smallest, fastest, cheapest. Reward curve might be modest but you have credit to retry.

**Recommendation:** Run Path 3 FIRST as a sanity check (40 min). If reward goes UP, run Path 1 in parallel. If reward stays flat, debug before burning $20.

### B.2 Pre-flight (under 2 min)

```bash
# Confirm secrets are usable
hf auth whoami
# Should print "Pratham-math" with a green check

# Confirm WANDB key is settable as a secret (you'll pass it to the job)
test -n "$WANDB_API_KEY" && echo "wandb key present in env" || echo "set WANDB_API_KEY first: wandb login then export"
```

If WANDB_API_KEY isn't exported in the shell:
```bash
wandb login
# paste key from https://wandb.ai/authorize
export WANDB_API_KEY=$(grep machine -A2 ~/.netrc 2>/dev/null | grep password | awk '{print $2}' | head -1)
# Or just paste it: export WANDB_API_KEY=<your-key>
```

### B.3 Override max_steps for Path 2 or Path 3 (skip for Path 1)

The default in `configs/train/grpo.yaml` is `max_steps: 400`. To override per-run, edit `scripts/job_train.sh` to add `--config-name=config 'train.max_steps=100'` etc. Easier: change the line in `job_train.sh`:

```bash
# In scripts/job_train.sh, find the GRPO python heredoc and change:
#   overrides=["model=qwen_1_5b","train=grpo"]
# To one of:
#   overrides=["model=qwen_1_5b","train=grpo","train.max_steps=100"]            # Path 2
#   overrides=["model=qwen_0_5b_smoke","train=grpo","train.max_steps=50"]       # Path 3
```

Commit and push the change, then re-run training. **For Path 1, no edit needed.**

### B.4 Fire the job (Path 3 — recommended first try)

```bash
hf jobs run --flavor=a10g-large --secrets HF_TOKEN --secrets WANDB_API_KEY --detach \
  pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel \
  bash -c 'apt-get update -qq && apt-get install -y -qq git && git clone -b master https://oauth2:$HF_TOKEN@huggingface.co/Pratham-math/fathom-code /w && bash /w/scripts/job_train.sh'
```

**Expected:** Job ID printed. Note it. URL: `https://huggingface.co/jobs/Pratham-math/<job-id>`.

**For Path 1 (a100-large):** swap `--flavor=a10g-large` → `--flavor=a100-large` and `--timeout=8h`.

### B.5 Monitor (do not block on this — proceed to Phase C in parallel)

```bash
hf jobs logs <job-id>
# Or open the URL in browser
```

Look for:
- `[OK] env_healthz` — env server up inside container
- `Model loaded: ...` — Unsloth or HF transformers loaded the 1.5B
- `running: SFT ...` then `SFT adapter saved at: outputs/sft_adapter`
- `running: GRPO ...` and progress bar with step counts
- `[ok] outputs/plots/reward_curve.png` from `make_plots.py` at the end
- `Trained model + plots: https://huggingface.co/Pratham-math/fathom-1.5b-grpo`

**If the job fails at install:** check that `scripts/job_train.sh` has the same install pattern as `scripts/job_smoke.sh` (which is verified working). Most likely diff is the SFT/GRPO Python heredoc syntax.

**If reward curve is flat (after monitoring W&B):** kill the job (`hf jobs cancel <id>`), drop `learning_rate` from `5.0e-6` to `3.0e-6` and bump `beta` from `0.04` to `0.08` in `configs/train/grpo.yaml`, push, retry.

---

## PHASE C — DEMO MATERIALS (target: 2.5h, parallel with Phase B training)

**Goal:** Build the storytelling artifacts (30% of judging). Do NOT wait for training to finish before starting these.

### C.1 Architecture diagram (15 min)

Create `assets/architecture.png` (Cursor: use Mermaid live editor — https://mermaid.live — paste the spec below, export PNG, save to `assets/architecture.png`):

```mermaid
flowchart LR
  subgraph Trainer["TRL GRPOTrainer (1.5B + LoRA)"]
    M[Qwen 2.5 Coder 1.5B<br/>4-bit + LoRA r=16]
    G[8 generations / step]
    M --> G
  end

  subgraph Env["FATHOM OpenEnv Server (HF Space)"]
    R["REPL primitive<br/>(RestrictedPython + subprocess)"]
    L["llm() primitive<br/>recursive sub-call"]
    O[Observation: tool output]
  end

  subgraph Reward["Composable Verifier (4 components)"]
    F[format_gate]
    C[correctness]
    T[token_budget α-param]
    E[recursion_efficiency]
    F --> X[compose_reward_fn]
    C --> X
    T --> X
    E --> X
  end

  Trainer -- multi-turn rollout --> Env
  Env -- observation --> Trainer
  Trainer -- completion + metadata --> Reward
  Reward -- scalar reward --> Trainer

  style M fill:#1f77b4,color:#fff
  style X fill:#2ca02c,color:#fff
```

```bash
mkdir -p assets
# Save the exported PNG as assets/architecture.png
```

### C.2 Demo video script + recording (45 min)

Create `assets/DEMO_SCRIPT.md`:

```markdown
# 90-second demo video script

[0:00–0:10]  TITLE CARD
"FATHOM — the first RL-trained Recursive Language Model.
A 1.5B model that reads documents 50x larger than its context window."

[0:10–0:25]  THE ENV
[Screen: open https://Pratham-math-fathom-env.hf.space in browser → /openapi.json]
"Our OpenEnv server gives the agent two tools: a sandboxed Python REPL
and a recursive llm() call. Anyone can hit it — it's a public HF Space."

[0:25–0:45]  THE REWARD
[Screen: open REWARD_AUDIT.md, scroll the table of 5 attacks]
"We hardened the verifier against five reward-hacking attacks before training.
Every reward component is grep-verifiable. Pytest -m reward_audit catches
masked-context exploits, format-only attacks, and length gaming."

[0:45–1:10]  THE TRAINING
[Screen: open W&B run → reward curve panel]
"Here's GRPO training the 1.5B against the env: composite reward rises from
0.05 to 0.X over Y steps. The dashed line is an untrained Qwen baseline."

[1:10–1:25]  THE OUTCOME
[Screen: live demo via Streamlit OR a terminal — feed a 200K-token doc, watch the recursion tree]
"The trained model decomposes the long document, calls itself recursively,
and answers correctly using only its 4K context."

[1:25–1:30]  CLOSE
"Full training reproducer in our Colab notebook. Code public on HF + GitHub. FATHOM."
```

**Recording instructions:**
1. Use OBS Studio (free) or Windows Game Bar (Win+G → Record)
2. 1080p, 30fps, 90s max
3. Speak clearly, normal pace
4. Save as `assets/demo.mp4` LOCALLY ONLY (per hackathon rules: do NOT commit big video files to HF Hub — link via YouTube)
5. Upload to YouTube as **unlisted**, copy URL

**If you cannot record:** make a 5-slide PDF deck instead at https://canva.com (search "tech pitch deck"), export as `assets/pitch.pdf`, commit it. The hackathon accepts deck OR video OR blog.

### C.3 Mini-blog on Hugging Face (30 min)

The hackathon explicitly accepts a mini-blog as the writeup. Create one at https://huggingface.co/blog with title "FATHOM: Teaching a 1.5B model to read documents bigger than its context window with RL".

Suggested structure:
1. **Hook (1 paragraph):** the context-window problem + recursive language models
2. **The env (with code snippet):** REPL + llm() primitive, OpenEnv conformance
3. **The reward (with REWARD_AUDIT excerpt):** composable + adversarially audited
4. **The training (with reward curve PNG):** GRPO via TRL + Unsloth, 1.5B + LoRA
5. **Results (numbers + Pareto):** untrained vs trained, accuracy + token cost
6. **Reproduce it (Colab link, HF Space link, repo link):** judges run it themselves

Save URL — paste into README + submission form.

### C.4 Polish viz/app.py (30 min, optional but visible to judges)

Open `viz/app.py` and make sure these three panels exist with at least placeholder data:
1. **Reward components** — pie chart of weights (correctness 0.75, token_budget 0.2, recursion_efficiency 0.05) + format gate badge
2. **Recursion tree** — `streamlit.components.v1.html` embedding a small D3 tree (3–5 nodes is enough; canned data is fine for the demo)
3. **Pareto frontier** — plotly scatter of accuracy vs token cost, with one point for untrained baseline + one for trained model

Test locally: `streamlit run viz/app.py` — confirm it loads, take a screenshot for the README.

### C.5 README full polish (30 min)

Open `README.md`. The current one already has the required sections (verified by preflight). Add or update these:

- **Submission Links section:** ensure these 6 lines exist near the top:
  1. HF Space (env): `https://huggingface.co/spaces/Pratham-math/fathom-env`
  2. Live env URL: `https://Pratham-math-fathom-env.hf.space`
  3. GitHub repo: (URL from Phase A.4)
  4. Trained model: `https://huggingface.co/Pratham-math/fathom-1.5b-grpo`
  5. Colab notebook: `https://colab.research.google.com/github/<your-github-user>/fathom-openenv/blob/master/notebooks/fathom_train.ipynb`
  6. Demo video / Blog: (URL from Phase C.2 or C.3)

- **Architecture image embed:** below the "Environment Design" section, add:
  ```markdown
  ![FATHOM architecture](assets/architecture.png)
  ```

- **Plots section (placeholder for now, populated in Phase D):**
  ```markdown
  ## Training Evidence
  ![Reward curve](outputs/plots/reward_curve.png)
  *Composite reward over training steps for Qwen 2.5 Coder 1.5B + LoRA on the FATHOM env. GRPO with β=0.04, lr=5e-6, 8 generations per step.*

  ![Loss curve](outputs/plots/loss_curve.png)
  *Training loss — descends as the policy learns the env reward shape.*

  ![Training summary](outputs/plots/training_summary.png)
  *4-panel: loss, mean reward, grad norm, KL divergence.*

  W&B run: <paste URL after training completes>
  ```

- **How to reproduce section:**
  ```markdown
  ## Reproduce in 5 minutes
  1. Open the [Colab notebook](https://colab.research.google.com/github/<user>/fathom-openenv/blob/master/notebooks/fathom_train.ipynb)
  2. Run cells 1–5 to verify env + smoke test
  3. (Optional, A100 needed) Run cell 6 to launch training
  ```

---

## PHASE D — POST-TRAINING WRAP (target: 30 min after job completes)

### D.1 Verify training artifacts on HF Hub

```bash
# Should list adapter, merged model, plots/
hf api repos/Pratham-math/fathom-1.5b-grpo
# Or in browser:
# https://huggingface.co/Pratham-math/fathom-1.5b-grpo/tree/main
```

**Expected files in the repo:**
- `sft_adapter/adapter_model.safetensors`
- `grpo_merged_16bit/model.safetensors` (large file)
- `plots/reward_curve.png`
- `plots/loss_curve.png`
- `plots/training_summary.png`
- `plots/grad_norm.png` and/or `plots/kl_curve.png`

**If `plots/` is missing:** the `make_plots.py` step inside the job failed. Pull `trainer_state.json` from the model repo and run `make_plots.py` locally:
```bash
python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='Pratham-math/fathom-1.5b-grpo', filename='grpo_run/trainer_state.json', local_dir='outputs')"
mkdir -p outputs/grpo_run && cp outputs/grpo_run/trainer_state.json outputs/grpo_run/  # adjust path
python scripts/make_plots.py
```

### D.2 Pull plots into the repo for README embed

```bash
mkdir -p outputs/plots
python -c "
from huggingface_hub import hf_hub_download
for fn in ['reward_curve.png','loss_curve.png','training_summary.png','grad_norm.png','kl_curve.png']:
    try:
        hf_hub_download(repo_id='Pratham-math/fathom-1.5b-grpo', filename=f'plots/{fn}', local_dir='.')
    except Exception as e:
        print(f'skip {fn}: {e}')
"
ls outputs/plots/
```

**Expected:** 3–5 PNG files. If any expected one is missing, that metric simply wasn't logged by TRL — proceed with what you have.

### D.3 Commit plots and final README

```bash
git add outputs/plots/ assets/ README.md notebooks/fathom_train.ipynb
git commit -m "docs: embed training plots, demo materials, Colab link, video link"
git push hf master
git push github master
```

### D.4 Final preflight + URL sanity check

```bash
# Preflight
python scripts/submission_preflight.py
# Must say "Preflight passed."

# Verify every URL the README claims, in one shot:
for url in \
  "https://huggingface.co/spaces/Pratham-math/fathom-env" \
  "https://Pratham-math-fathom-env.hf.space/healthz" \
  "https://huggingface.co/Pratham-math/fathom-1.5b-grpo" \
  "https://huggingface.co/Pratham-math/fathom-code" ; do
  echo -n "$url ... "
  curl -sf -o /dev/null -w "%{http_code}" "$url" || echo "DEAD"
  echo
done
```

All four must return 200. If `fathom-env/healthz` returns 404 or 500: the Space is sleeping, hit it once in browser to wake it up.

### D.5 Verify minimum requirements one by one (the "non-negotiables" checklist)

Print and check off each:

```
[ ] OpenEnv: grep "openenv-core" pyproject.toml — must show >=0.2.3
[ ] Training script (TRL): test -f train/grpo.py
[ ] Colab notebook: test -f notebooks/fathom_train.ipynb
[ ] Loss + reward plots: ls outputs/plots/*.png — must show >=2 PNGs
[ ] Mini-blog OR video OR slides: link in README is live
[ ] HF Space: curl /healthz returns 200
[ ] README has env URL: grep "hf.space" README.md
[ ] README has writeup link: grep -E "(youtube|huggingface.co/blog|.pdf)" README.md
```

Each box must tick before submission.

---

## PHASE E — SUBMIT (target: 15 min)

### E.1 Final commit + push

```bash
git status   # should be clean
git push hf master
git push github master
```

### E.2 Submission form fields (have these ready to paste)

| Field | Value |
|-------|-------|
| Team name | (yours) |
| Theme | Theme 2 — Long-Horizon Planning & Instruction Following |
| Sub-prize | Mercor (token-budget-aware reward) |
| Environment HF Space URL | `https://huggingface.co/spaces/Pratham-math/fathom-env` |
| Environment endpoint | `https://Pratham-math-fathom-env.hf.space` |
| Code repo (HF) | `https://huggingface.co/Pratham-math/fathom-code` |
| Code repo (GitHub) | (URL from Phase A.4) |
| Trained model | `https://huggingface.co/Pratham-math/fathom-1.5b-grpo` |
| Colab notebook | `https://colab.research.google.com/github/<your-user>/fathom-openenv/blob/master/notebooks/fathom_train.ipynb` |
| Demo video / blog | (URL from Phase C.2 or C.3) |
| W&B run | (paste from training run) |

### E.3 Submit

Open the official hackathon submission link (from #on-campus-discord). Paste each field. Hit submit. Take a screenshot of the confirmation page. Save as `assets/submission_confirmation.png` in the repo (committed evidence in case of dispute).

---

## RISK MITIGATIONS (read in advance)

### R1 — Training job fails at install
**Symptom:** pip resolver errors, `cannot import X from trl`, etc.
**Fix:** `scripts/job_smoke.sh` install pattern is verified working. Diff `job_train.sh` against `job_smoke.sh` and align the install lines exactly. The single difference should be the addition of `flash-attn` (which is allowed to fail).

### R2 — GRPO reward curve is flat
**Symptom:** W&B `reward/composite` stays around 0.05 for >50 steps.
**Fix:** Kill, edit `configs/train/grpo.yaml`: `learning_rate: 3.0e-6`, `beta: 0.08`, push, restart. If still flat after 100 steps, run SFT-only (`scripts/job_sft_only.sh`) and ship that — SFT alone produces a usable reward "curve" if you log per-batch reward in `compose_reward_fn`.

### R3 — vLLM colocate OOMs on a100-large
**Symptom:** CUDA OOM during rollout.
**Fix:** Edit `configs/train/grpo.yaml`: `vllm_gpu_memory_utilization: 0.35` (down from 0.45) or `num_generations: 4` (down from 8). Push, restart.

### R4 — Trained model save corrupts (Unsloth merged_4bit issue)
**Symptom:** `train/grpo.py` raises during `save_pretrained_merged`.
**Fix:** The code already uses `merged_16bit` and falls back to `peft.merge_and_unload`. If both fail, the adapter alone (`outputs/sft_adapter/`) is enough — judges can load it via PEFT. README should mention "model adapter pushed; merge step optional".

### R5 — HF Space goes to sleep before judging
**Symptom:** Judges hit `/healthz` and get 503 (cold start).
**Fix:** Set up a simple "ping" that hits the env every 30 min from your laptop on submission day:
```bash
while true; do curl -s https://Pratham-math-fathom-env.hf.space/healthz; sleep 1800; done &
```
Or upgrade the Space to a "always on" tier ($0.05/hr ≈ $1.50/day).

### R6 — Out of HF credits before training completes
**Symptom:** Job killed mid-run.
**Fix:** Smaller model (Path 3 in B.1) or fewer steps (50). The reward curve doesn't need to be long — it needs to **show clear upward trend**. 30 well-shaped steps beats 400 noisy ones.

### R7 — Colab notebook breaks for judges
**Symptom:** Judge opens notebook, cells error out.
**Fix:** Test it yourself end-to-end before submitting. Open in Colab from GitHub. Run all cells. Fix any. Push.

### R8 — Last-minute README placeholder forgotten
**Symptom:** Preflight catches a `TODO` token in README.
**Fix:** `grep -n TODO README.md` — replace each one before commit.

---

## TIME + MONEY BUDGET

| Phase | Time | HF $ | Risk if skipped |
|-------|------|------|-----------------|
| A. Commit + mirror | 15 min | $0 | Cannot submit (no public code) |
| B. Training (Path 3 first) | 5 min setup + 40 min | $0.50 | -20% rubric (no reward improvement evidence) |
| B. Training (Path 1 if Path 3 GO) | 5h | $20 | If Path 3 enough, this is bonus |
| C.1 Architecture diagram | 15 min | $0 | -5% storytelling |
| C.2 Demo video | 45 min | $0 | -15% storytelling (video is highly weighted) |
| C.3 Blog post | 30 min | $0 | Acceptable to skip if video done |
| C.4 viz/app.py polish | 30 min | $0 | Demo less sharp |
| C.5 README polish | 30 min | $0 | Cannot submit (preflight fails) |
| D. Post-training wrap | 30 min | $0 | Plots not embedded |
| E. Submit | 15 min | $0 | Cannot submit |

**Minimum viable path:** A → B (Path 3) → C.1 + C.5 + (C.2 OR C.3) → D → E. **3 hours, ~$1.**

**Strong path:** A → B (Path 3 then Path 1) → C all → D → E. **6–7 hours, ~$22.**

---

## EXACT COMMANDS — COPY-PASTE BLOCK

For the impatient — here is the entire happy path in one block. Cursor: **do not run this without reading the phase sections above.** Many commands need a verification step before the next one.

```bash
# ===== PHASE A =====
git status --short
python scripts/submission_preflight.py
git add -A
git commit -m "feat: phase 1 complete + Phase 2 prep"
git push hf master
gh repo create fathom-openenv --public --source=. --push --description="FATHOM — First RL-trained Recursive Language Model."

# ===== PHASE B (Path 3 first, sanity check) =====
hf auth whoami
hf jobs run --flavor=a10g-large --secrets HF_TOKEN --secrets WANDB_API_KEY --detach \
  pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel \
  bash -c 'apt-get update -qq && apt-get install -y -qq git && git clone -b master https://oauth2:$HF_TOKEN@huggingface.co/Pratham-math/fathom-code /w && bash /w/scripts/job_train.sh'
# NOTE THE JOB ID

# Monitor
hf jobs logs <job-id>

# ===== PHASE C (parallel) =====
# 1. Make architecture.png at https://mermaid.live (paste spec from C.1)
# 2. Record demo video (90s), upload to YouTube unlisted
# 3. Optional: write blog at https://huggingface.co/blog
# 4. streamlit run viz/app.py — screenshot + close
# 5. Update README.md with all URLs

# ===== PHASE D (after training completes) =====
mkdir -p outputs/plots
python -c "from huggingface_hub import hf_hub_download
for fn in ['reward_curve.png','loss_curve.png','training_summary.png']:
    try: hf_hub_download(repo_id='Pratham-math/fathom-1.5b-grpo', filename=f'plots/{fn}', local_dir='.')
    except Exception as e: print(f'skip {fn}: {e}')"
ls outputs/plots/

git add outputs/plots/ assets/ README.md notebooks/fathom_train.ipynb
git commit -m "docs: embed training plots + demo materials + Colab link"
git push hf master
git push github master

python scripts/submission_preflight.py
for url in \
  "https://huggingface.co/spaces/Pratham-math/fathom-env" \
  "https://Pratham-math-fathom-env.hf.space/healthz" \
  "https://huggingface.co/Pratham-math/fathom-1.5b-grpo" \
  "https://huggingface.co/Pratham-math/fathom-code"; do
  echo -n "$url ... "; curl -sf -o /dev/null -w "%{http_code}" "$url"; echo
done

# ===== PHASE E =====
# Open submission form, paste fields from E.2, submit, screenshot confirmation.
```

---

## FALLBACK — IF EVERYTHING ELSE GOES WRONG

You can submit RIGHT NOW with what already works:

1. Smoke test green on Linux (proves pipeline)
2. Env deployed and responding
3. REWARD_AUDIT.md (5 attacks neutralized)
4. 56 unit tests passing
5. Preflight already PASSED
6. Reproducer Colab notebook present

The submission would lose the "training improvement evidence" 20% but score on the other 80%. **Better to ship a partial than miss the deadline.** If at any point you have less than 1 hour left and Phase D isn't done, **commit what you have, run preflight, push, submit.**

---

## END OF PLAN

Total length: every step from current state to submitted. Cursor — execute in order, verify between phases, and if any command output looks wrong, STOP and report instead of guessing the next command.