## Project **FATHOM — First RL-Trained Recursive Language Model** FATHOM is an OpenEnv environment + GRPO training pipeline that teaches a small open-source language model (Qwen2.5-Coder-1.5B-Instruct, 4-bit + LoRA) to *use* a Recursive Language Model (RLM) scaffold well: slice long contexts with Python, grep for relevant regions, delegate to sub-LM calls only when needed, and answer questions about documents that are 50× larger than its own context window. It is our submission to the Meta × PyTorch × Hugging Face OpenEnv Hackathon Grand Finale (Scaler School of Technology, Bangalore, April 25–26, 2026). **Core Value:** A 1.5B model trained on our environment solves 200K-token QA tasks its own 4K context cannot hold — demonstrated with a clean reward curve, a selective recursion-tree visualization, and a Pareto frontier of accuracy-vs-tokens. If only one thing ships, it is this single demo. ### Constraints - **Timeline**: ~40 usable hours of building across 4 days — every phase must produce a committed, demonstrable artifact before moving on, because there is no slack for context-lost rebuilds. - **Tech stack**: TRL (trainer) + Unsloth (efficiency + inference) + OpenEnv (environment standard) + Hugging Face Spaces (deployment). Dictated by hackathon rubric — deviating costs "judge goodwill" points. - **Algorithm**: GRPO specifically (not PPO/DPO) — hackathon rubric names GRPO as canonical, public reproductions exist to crib from, TRL `GRPOTrainer` is the first-class path. - **Base model**: Qwen2.5-Coder-1.5B-Instruct (4-bit via `unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit`). 3B cut post-research-synthesis — venue A100 flavor unknown, 3B would eat the 40-hour budget for uncertain upside. 1.5B with clean curve beats 3B with partial curve on judging. - **Pre-event hardware**: Laptop RTX 4060, 8 GB VRAM. Can only run CPU/0.5B smoke tests locally — 1.5B+ runs are venue-only. - **Reward verifier must be deterministic**: No LLM-as-judge in the training loop. Every task has an exact-match or short-span gold answer. - **Context window**: Train at 16 K, claim generalization to 200 K at eval. Rollout latency at 200 K is prohibitive for GRPO. - **Recursion depth in training**: ≤ 2. Deeper recursion is demo-only at inference time. - **Judging criteria (hard)**: Submissions are scored on clear env design + objective rewards + demonstrable improvement + anti-reward-hacking + reproducible deployment + sharp demo. Every phase must produce an artifact mapping to at least one criterion. - **Pitch**: 3 minutes, 30% of judging weight goes to storytelling. The Pareto chart and recursion-tree viz must exist before the last 4 hours, not after. ## Technology Stack ## 1. Versions by Role (April 2026 pins) | Role | Package | Version pin | Confidence | Source | |------|---------|-------------|------------|--------| | **OpenEnv core/CLI** | `openenv-core[core]` | `>=0.2.3,<0.3` | HIGH | [meta-pytorch/OpenEnv v0.2.3, Mar 28 2026](https://github.com/meta-pytorch/OpenEnv) | | **TRL (GRPOTrainer + OpenEnv integration)** | `trl` | `==1.2.0` | HIGH | [PyPI trl 1.2.0, Apr 17 2026](https://pypi.org/project/trl/); [TRL OpenEnv guide](https://huggingface.co/docs/trl/main/en/openenv) | | **Unsloth (efficiency + save)** | `unsloth` | `==2026.4.8` (v0.1.36-beta tag, whatever passes Thu smoke test) | MEDIUM | [unslothai/unsloth releases, v0.1.36-beta Apr 8 2026](https://github.com/unslothai/unsloth/releases) | | **Unsloth Zoo** | `unsloth-zoo` | latest-compatible (Unsloth resolves) | MEDIUM | — | | **Torch (CUDA 12.4 for A100)** | `torch` | `==2.6.0` | HIGH | PyTorch 2.6+ is Unsloth-supported; 2.6 ships cu124 wheel | | **Transformers** | `transformers` | `==4.49.0` | HIGH | Pinned by TRL 1.2 requirements | | **Accelerate** | `accelerate` | `==1.5.2` | HIGH | Pinned by TRL 1.2 | | **PEFT** | `peft` | `==0.14.0` | HIGH | Pinned by TRL 1.2 | | **bitsandbytes** | `bitsandbytes` | `==0.45.1` | HIGH | 0.45+ required for cu124 + Ampere+Hopper paths | | **vLLM (for colocate inference)** | `vllm` | `==0.7.3` | HIGH | Matches TRL 1.2 server + colocate contract; Unsloth fast_inference path compatible | | **Datasets** | `datasets` | `==3.3.2` | HIGH | Transformers 4.49 pair | | **HF Hub** | `huggingface_hub` | `>=0.28,<1.0` | HIGH | Space push + snapshot_download | | **FastAPI (env server)** | `fastapi` | `>=0.115,<0.120` | HIGH | OpenEnv scaffold generates FastAPI app | | **Uvicorn** | `uvicorn[standard]` | `>=0.32,<1.0` | HIGH | — | | **Pydantic** | `pydantic` | `>=2.9,<3.0` | HIGH | OpenEnv + TRL both on Pydantic 2 | | **W&B** | `wandb` | `>=0.18,<1.0` | HIGH | Default TRL callback | | **Streamlit (viz + tree)** | `streamlit` | `>=1.39,<2.0` | HIGH | Richer than Gradio for recursion-tree D3 embed (see §1.1) | | **Plotly (Pareto chart)** | `plotly` | `>=5.24,<6.0` | HIGH | — | | **Matplotlib (quick plots, W&B)** | `matplotlib` | `>=3.9,<4.0` | HIGH | — | | **RestrictedPython (REPL AST filter)** | `RestrictedPython` | `>=7.4,<8.0` | HIGH | [zopefoundation/RestrictedPython docs](https://restrictedpython.readthedocs.io/) | | **Claude API (SFT data gen)** | `anthropic` | `>=0.40,<1.0` | HIGH | SFT trace generator only; not a runtime dep | ### 1.1 Streamlit vs Gradio — decided ## 2. Canonical Install — Single Block # Step 0: CUDA 12.4 sanity check # Step 1: create env (uv preferred; pip fallback below) # Step 2: torch first (CUDA 12.4 wheel), own line to avoid resolver confusion # Step 3: core training stack # Step 4: Unsloth (pin to Thu-smoke-test version; DO NOT -U at venue) # Step 5: env server + tooling # Step 6: observability + viz + data gen # Step 7: flash-attention (optional but big speedup for generation) ## 3. Qwen2.5-Coder Specifics ### 3.1 Primary: Qwen2.5-Coder-1.5B-Instruct | Attribute | Value | Source | |-----------|-------|--------| | HF ID (base) | `Qwen/Qwen2.5-Coder-1.5B-Instruct` | [HF model card](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) | | HF ID (Unsloth-preferred for faster load) | `unsloth/Qwen2.5-Coder-1.5B-Instruct` | [unsloth/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/unsloth/Qwen2.5-Coder-1.5B-Instruct) | | HF ID (Unsloth 4-bit bnb pre-quant) | `unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit` | [Unsloth Qwen 2.5 Coder collection](https://huggingface.co/collections/unsloth/qwen-25-coder) | | Parameters | 1.54B total / 1.31B non-embedding | HF card | | BF16 size on disk | ~3.1 GB | HF card | | 4-bit bnb size on disk | ~1.1 GB | Unsloth card | | Context length (native) | 32,768 tokens | HF card config.json | | Context length (with YaRN scaling) | 131,072 tokens (factor 4.0) | HF card long-context section | | Architecture | 28 layers, 12 Q-heads, 2 KV-heads (GQA), SwiGLU, RMSNorm, RoPE | HF card | | Chat template | ChatML-derived with `<|im_start|>` / `<|im_end|>` + system/user/assistant | HF card | ### 3.2 Stretch: Qwen2.5-Coder-3B-Instruct | Attribute | Value | |-----------|-------| | HF ID | `Qwen/Qwen2.5-Coder-3B-Instruct` | | HF ID (Unsloth 4-bit) | `unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit` | | BF16 size on disk | ~6.2 GB | | 4-bit + LoRA (rank 32) VRAM @ 16K context + 8 generations | ~38-45 GB (**fits A100-80GB with headroom; tight/no-go on A100-40GB**) | | Context length (native) | 32,768 tokens | ### 3.3 Chat template — one source of truth (prevents H6) # For training (SFT with assistant response to learn): # For rollout/eval (model to generate): - Use `Qwen2.5-Coder-1.5B-Instruct` (has chat template baked in), never the non-Instruct base. - Do NOT try `return_assistant_tokens_mask=True` — broken on Qwen2.5 per transformers #34172. Full-sequence SFT loss is fine for our scale. - TRL 1.2's OpenEnv integration applies the tokenizer's chat template internally for multi-turn tool calls. Do not hand-concatenate `<|im_start|>` strings anywhere in our code. - Diff-test the smoke test: tokenize one SFT example with our prep code; tokenize the same example through `GRPOTrainer._prepare_inputs` (set a breakpoint); confirm zero token-ID drift. ## 4. GRPOTrainer Config — Minimum Viable ### Passing per-sample reward functions ### `reward_weights` vs compose-in-fn ## 5. Unsloth vs vLLM for Rollouts — Resolved ## 6. LoRA Save Sequence — Verbatim Snippet # === AT END OF GRPO TRAINING (after trainer.train() returns) === # (1) ALWAYS save adapter-only first. Small (~15MB), never hits the broken merge paths. # This is our reproducibility artifact and our safety net. # (2) Push adapter to HF Hub immediately — cheap and protects against local-disk loss. # (3) Save a merged 16-bit model for fast inference at demo time. # save_method="merged_16bit" is the ONE safe choice. Never use: # - "merged_4bit" → corrupt under QLoRA (#1267, #2339) # - "merged_4bit_forced" → silently emits 16bit (#2339) or size-mismatched (#1791) # If the call fails inside Docker with an empty dir (#3882), fall back to # step (4) manual merge via peft. # trainer.model is already an Unsloth-patched PeftModel # (4) Sanity check BEFORE declaring training done: load the merged model, # run 3 eval questions, assert non-empty output. ### Loading back for eval ### RoPE scaling for 200K eval ## 7. Python REPL Sandbox — Choice + Safety Checklist - **Plain subprocess with timeout only** — no AST filter means attribute-access escape paths (`__subclasses__`, `catch_warnings`, traceback walk) still work inside the subprocess. Model can escape to subprocess.Popen within our own subprocess. - **Pyodide-via-subprocess** — spawns a browser-grade WASM runtime per call; 500ms+ startup per REPL action, kills rollout throughput at 8 generations × multiple turns. - **OpenEnv built-in** — OpenEnv ships tool primitives but no Python REPL sandbox; the scaffold expects us to write `env/server/repl.py` (confirmed in ARCHITECTURE.md §1). - RestrictedPython AST-rewrites at compile time: blocks `__class__`, `__subclasses__`, `__globals__`, attribute access into dunders, star imports. Not a full sandbox but stops 90% of trivial Python-sandbox-escape CTF payloads. - Subprocess isolation handles the remaining 10%: if a tricky escape produces an import or system call, it hits the OS-level wall (no network, read-only cwd, ulimit'd CPU/memory, seccomp deny-list). Subprocess crashes don't kill the env server. - `RestrictedPython>=7.4` is actively maintained by zopefoundation and mature. ### Safety checklist (implement as `tests/test_repl_sandbox.py` before shipping) - [ ] `open` — no file I/O - [ ] `exec` — no dynamic code - [ ] `eval` — no dynamic eval - [ ] `compile` — no AST manipulation - [ ] `__import__` — whitelist via custom import hook (see below) - [ ] `__builtins__` — set to a minimal dict; never pass through the raw module - [ ] `input` / `breakpoint` / `help` / `exit` / `quit` — exit paths and interactive prompts - [ ] `re`, `json`, `math`, `statistics`, `collections`, `itertools`, `string`, `functools`, `operator` - [ ] **NO** `os`, `sys`, `subprocess`, `socket`, `urllib`, `http`, `requests`, `pathlib`, `io`, `ctypes`, `threading`, `multiprocessing`, `pickle`, `marshal`, `importlib` - [ ] Separate `subprocess.Popen` with `start_new_session=True` - [ ] `ulimit -v 524288` (512MB RAM) via `preexec_fn` / `resource.setrlimit` - [ ] `ulimit -t 5` (5s CPU), `ulimit -f 0` (no file writes) - [ ] `cwd=/tmp/episode-{uuid}` (ephemeral, wiped on `reset()`) - [ ] Non-root user (venue A100 containers default to uid 1000 — verify at setup) - [ ] Network: `unshare -n` if available, else seccomp deny `socket`, `connect`, `sendto` - [ ] Timeout: hard-kill at 30s wall-clock (`subprocess.wait(timeout=30)`, then `Popen.kill()`) - [ ] `().__class__.__base__.__subclasses__()` — attribute access blocked by RestrictedPython - [ ] `e.__traceback__.tb_frame.f_back.f_builtins` — traceback walk after a raise - [ ] `warnings.catch_warnings()` builtins trick — import blocked by whitelist - [ ] `__import__('os')` — whitelist rejects - [ ] `getattr(__builtins__, 'open')` — `__builtins__` replaced with minimal dict - [ ] 30s-infinite-loop — subprocess killed, env continues - [ ] 1GB memory bomb — rlimit kills, env continues ## 8. `openenv init` Scaffold Workflow ### 8.1 Scaffold # From repo root # or with explicit typed-action template (v0.2.3+): ### 8.2 Edits we make (post-init) - Add to `src/envs/fathom/server/app.py`: - In `environment.py`, declare `SUPPORTS_CONCURRENT_SESSIONS: bool = True` at module scope (required for TRL's 8 parallel generations — confirmed in [TRL OpenEnv guide §Server concurrency](https://huggingface.co/docs/trl/main/en/openenv#server-concurrency)). - Implement REPL + `llm()` primitive per ARCHITECTURE.md §1. ### 8.3 Local run # In another terminal: ### 8.4 Deploy to HF Space # Login once (prerequisite): # Push (creates or updates the Space): # Manual push via huggingface_hub: ## 9. SFT Library Choice - `unsloth.UnslothTrainer` — custom trainer, fewer public examples, harder to swap to GRPO afterward. - Raw training loop — pointless reinvention; SFTTrainer is ~20 LoC of setup. - Same model object then passes to `GRPOTrainer` with zero re-init cost (PITFALLS.md H6 prevention). - TRL 1.2 `SFTTrainer` accepts Unsloth-patched models without special handling. - We keep the chat-template formatting in one place (`formatting_func`) — same template used later in GRPO. ### Snippet — adapting Claude-generated conversations to SFT format # (1) Load base model with LoRA adapters attached (Unsloth patch) # (2) Claude traces on disk look like this (one line of JSONL per trajectory): # {"system": "...", "messages": [{"role":"user","content":"..."}, # {"role":"assistant","content":"..."}, # {"role":"tool","content":"..."}, # {"role":"assistant","content":"..."}]} # (3) formatting_func → apply_chat_template (single source of truth, §3.3 rule) # (4) SFT config — light touch so we don't overfit (PITFALLS.md M7) # Save SFT adapter — GRPO will load from here ## 10. Anti-Patterns — Stack-Specific ### 10.1 `save_pretrained_merged(save_method="merged_4bit")` on a QLoRA model ### 10.2 Mismatched chat template between SFT prep and GRPO rollouts ### 10.3 `max_completion_length > 2048` before the smoke test has run ### 10.4 `vllm_mode="server"` for multi-turn OpenEnv training ### 10.5 `pip install -U unsloth` at the venue ### 10.6 Running REPL subprocess in the env server's cwd ### 10.7 Logging only the composite reward to W&B ### 10.8 Passing raw `<|im_start|>user\n...` strings to GRPOTrainer ### 10.9 Saving merged model without sanity-checking the output ### 10.10 Using `Qwen2.5-Coder-1.5B` (base) instead of `-Instruct` ### 10.11 Trusting `trl vllm-serve` in server mode as a separate process ### 10.12 `gradient_checkpointing=False` "to go fast" ## 11. Prerequisite Setup — One-Time Commands at Venue # ── G1: GPU alive, CUDA visible # expect: CUDA Version 12.4+, A100-SXM4-40GB or A100-80GB, 0 MiB used # ── G2: CUDA toolkit version compatible with torch cu124 wheel # ── G3: Python + uv # ── G4: Install FATHOM env (single pip block from §2 above) # run the §2 commands here # ── G5: Verify torch sees the A100 # expect: True NVIDIA A100-... # ── G6: HF login (write token from hf.co/settings/tokens) # paste token; verify: # ── G7: W&B login (API key from wandb.ai/authorize) # paste key; verify: # ── G8: OpenEnv CLI alive # ── G9: Pre-pull the model into HF cache (avoid mid-training download stalls) # ── G10: Smoke-test GRPOTrainer imports (catches version-skew issues) # ── G11: Sandbox unit tests (these MUST pass before env server goes public) # ── G12: End-to-end smoke — 1 GRPO step ## 12. Confidence & Verification Matrix | Recommendation | Confidence | Basis | If LOW → 10-min venue verification | |---------------|------------|-------|------------------------------------| | TRL == 1.2.0 | HIGH | PyPI dated 2026-04-17 | — | | Unsloth == 2026.4.8 (v0.1.36-beta) | MEDIUM | GitHub releases page; rapid iteration | `python -c "import unsloth; print(unsloth.__version__)"` then run G12 smoke test | | OpenEnv-core >= 0.2.3 | HIGH | GitHub tag + TRL guide references 0.2.1+ | — | | torch 2.6 cu124 | HIGH | PyTorch wheel index stable | `torch.version.cuda` == '12.4' | | vllm 0.7.3 | HIGH | TRL 1.2 OpenEnv examples target it | `vllm --version` | | `vllm_mode="colocate"` for multi-turn | HIGH | TRL OpenEnv guide Wordle example + PITFALLS #4543 | Run G12 smoke with 1 turn, then with 2 turns — advantage should look normal | | `beta=0.04` KL floor | MEDIUM | EDGE-GRPO paper + PITFALLS H1; TRL default is 0.0 | Train 50 steps with β=0.0; if group_std drops <0.05 → switch to 0.04 | | `num_generations=8` | HIGH | TRL default; all OpenEnv examples | — | | `learning_rate=5e-6` with 4-bit | MEDIUM | PITFALLS M1 "4-bit safe band 5e-6 to 1e-5" | First 20 steps — if loss NaN, drop to 3e-6; if flat, raise to 1e-5 | | `max_grad_norm=0.5` | MEDIUM | PITFALLS M1 conservative | Widen to 1.0 only if training too conservative and loss plateaus | | Unsloth Qwen2.5-Coder-1.5B-Instruct-bnb-4bit | HIGH | HF Unsloth collection page | — | | YaRN factor=4.0 for 200K eval | MEDIUM | HF model card long-context section; PITFALLS M8 warns | At eval time, test 50K → 100K → 150K → 200K; note first OOM point, cap claim there | | Merged-16bit save as the only trusted path | HIGH | PITFALLS R4 (5 issue numbers); §6 snippet follows Unsloth docs | — | | RestrictedPython + subprocess sandbox | HIGH | Canonical pattern; dida.do / tianpan.co verified | Run the 7 adversarial unit tests in §7 — if any pass, patch before shipping | | Streamlit (not Gradio) for demo Space | MEDIUM | Our override on FEATURES.md TS-DEM-2 rationale | Build both skeletons Thu; pick the one that renders the D3 tree cleaner in 30 min | | `SFTTrainer` + Unsloth model | HIGH | Unsloth notebook collection; TRL 1.2 supports Unsloth models | — | | `openenv push` for HF Space | MEDIUM | OpenEnv CLI 0.2.3 docs; new CLI | Try `push` Thu; if it fails, use the manual `git push` fallback in §8.4 | | Chat template one-source-of-truth | HIGH | PITFALLS H6 + HF Qwen card | Diff-test token IDs in smoke — zero drift is the gate | - Unsloth exact version pin (re-run G10 smoke after pinning) - vLLM colocate memory fraction — `vllm_gpu_memory_utilization=0.45` may need tuning on A100-40GB vs 80GB; watch `nvidia-smi` during the first 10 GRPO steps - β (KL coefficient) — if group_std stays healthy at 0.0, save the 1% throughput; if collapse appears, bump to 0.04. Cheap to toggle mid-run via config. ## 13. Sources - [TRL 1.2.0 on PyPI (2026-04-17)](https://pypi.org/project/trl/) — HIGH - [TRL OpenEnv Integration guide](https://huggingface.co/docs/trl/main/en/openenv) — HIGH - [TRL GRPOConfig on GitHub main](https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_config.py) — HIGH - [Unsloth releases on GitHub](https://github.com/unslothai/unsloth/releases) — HIGH - [Unsloth RL Guide](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) — HIGH - [Unsloth Qwen 2.5 Coder collection](https://huggingface.co/collections/unsloth/qwen-25-coder) — HIGH - [Qwen2.5-Coder-1.5B-Instruct HF card](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) — HIGH - [Unsloth/Qwen2.5-Coder-1.5B-Instruct HF card](https://huggingface.co/unsloth/Qwen2.5-Coder-1.5B-Instruct) — HIGH - [meta-pytorch/OpenEnv GitHub v0.2.3](https://github.com/meta-pytorch/OpenEnv) — HIGH - [vLLM training/trl docs](https://docs.vllm.ai/en/latest/training/trl/) — HIGH - [RestrictedPython docs](https://restrictedpython.readthedocs.io/) — HIGH - Unsloth LoRA save bug issues: #1791, #2339, #1267, #3882, #1877 (PITFALLS.md R4) — HIGH - TRL multi-turn bug: #4543 (PITFALLS.md H2) — HIGH - Chat template gotcha: transformers #34172 (PITFALLS.md N2) — HIGH - Qwen2.5 + RoPE scaling for 200K: HF card + Unsloth #3083 (PITFALLS.md M8) — HIGH - OpenEnv env server design: ARCHITECTURE.md §1, §8 — HIGH - Reward decomposition pattern: ARCHITECTURE.md §6 Pattern 1 — HIGH - Sandbox pattern choice: dida.do + tianpan.co via PITFALLS.md H3 — HIGH ## Conventions Conventions not yet established. Will populate as patterns emerge during development. ## Architecture Architecture not yet mapped. Follow existing patterns found in the codebase. ## Project Skills No project skills found. Add skills to any of: `.claude/skills/`, `.agents/skills/`, `.cursor/skills/`, or `.github/skills/` with a `SKILL.md` index file. ## GSD Workflow Enforcement Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync. Use these entry points: - `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks - `/gsd-debug` for investigation and bug fixing - `/gsd-execute-phase` for planned phase work Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it. ## Developer Profile > Profile not yet configured. Run `/gsd-profile-user` to generate your developer profile. > This section is managed by `generate-claude-profile` -- do not edit manually.