CLAUDE.md

Project

FATHOM — First RL-Trained Recursive Language Model

FATHOM is an OpenEnv environment + GRPO training pipeline that teaches a small open-source language model (Qwen2.5-Coder-1.5B-Instruct, 4-bit + LoRA) to use a Recursive Language Model (RLM) scaffold well: slice long contexts with Python, grep for relevant regions, delegate to sub-LM calls only when needed, and answer questions about documents that are 50× larger than its own context window. It is our submission to the Meta × PyTorch × Hugging Face OpenEnv Hackathon Grand Finale (Scaler School of Technology, Bangalore, April 25–26, 2026).

Core Value: A 1.5B model trained on our environment solves 200K-token QA tasks its own 4K context cannot hold — demonstrated with a clean reward curve, a selective recursion-tree visualization, and a Pareto frontier of accuracy-vs-tokens. If only one thing ships, it is this single demo.

Constraints

• Timeline: ~40 usable hours of building across 4 days — every phase must produce a committed, demonstrable artifact before moving on, because there is no slack for context-lost rebuilds.
• Tech stack: TRL (trainer) + Unsloth (efficiency + inference) + OpenEnv (environment standard) + Hugging Face Spaces (deployment). Dictated by hackathon rubric — deviating costs "judge goodwill" points.
• Algorithm: GRPO specifically (not PPO/DPO) — hackathon rubric names GRPO as canonical, public reproductions exist to crib from, TRL GRPOTrainer is the first-class path.
• Base model: Qwen2.5-Coder-1.5B-Instruct (4-bit via unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit). 3B cut post-research-synthesis — venue A100 flavor unknown, 3B would eat the 40-hour budget for uncertain upside. 1.5B with clean curve beats 3B with partial curve on judging.
• Pre-event hardware: Laptop RTX 4060, 8 GB VRAM. Can only run CPU/0.5B smoke tests locally — 1.5B+ runs are venue-only.
• Reward verifier must be deterministic: No LLM-as-judge in the training loop. Every task has an exact-match or short-span gold answer.
• Context window: Train at 16 K, claim generalization to 200 K at eval. Rollout latency at 200 K is prohibitive for GRPO.
• Recursion depth in training: ≤ 2. Deeper recursion is demo-only at inference time.
• Judging criteria (hard): Submissions are scored on clear env design + objective rewards + demonstrable improvement + anti-reward-hacking + reproducible deployment + sharp demo. Every phase must produce an artifact mapping to at least one criterion.
• Pitch: 3 minutes, 30% of judging weight goes to storytelling. The Pareto chart and recursion-tree viz must exist before the last 4 hours, not after.

Technology Stack

1. Versions by Role (April 2026 pins)

Role	Package	Version pin	Confidence	Source
OpenEnv core/CLI	openenv-core[core]	>=0.2.3,<0.3	HIGH	meta-pytorch/OpenEnv v0.2.3, Mar 28 2026
TRL (GRPOTrainer + OpenEnv integration)	trl	==1.2.0	HIGH	PyPI trl 1.2.0, Apr 17 2026; TRL OpenEnv guide
Unsloth (efficiency + save)	unsloth	==2026.4.8 (v0.1.36-beta tag, whatever passes Thu smoke test)	MEDIUM	unslothai/unsloth releases, v0.1.36-beta Apr 8 2026
Unsloth Zoo	unsloth-zoo	latest-compatible (Unsloth resolves)	MEDIUM	—
Torch (CUDA 12.4 for A100)	torch	==2.6.0	HIGH	PyTorch 2.6+ is Unsloth-supported; 2.6 ships cu124 wheel
Transformers	transformers	==4.49.0	HIGH	Pinned by TRL 1.2 requirements
Accelerate	accelerate	==1.5.2	HIGH	Pinned by TRL 1.2
PEFT	peft	==0.14.0	HIGH	Pinned by TRL 1.2
bitsandbytes	bitsandbytes	==0.45.1	HIGH	0.45+ required for cu124 + Ampere+Hopper paths
vLLM (for colocate inference)	vllm	==0.7.3	HIGH	Matches TRL 1.2 server + colocate contract; Unsloth fast_inference path compatible
Datasets	datasets	==3.3.2	HIGH	Transformers 4.49 pair
HF Hub	huggingface_hub	>=0.28,<1.0	HIGH	Space push + snapshot_download
FastAPI (env server)	fastapi	>=0.115,<0.120	HIGH	OpenEnv scaffold generates FastAPI app
Uvicorn	uvicorn[standard]	>=0.32,<1.0	HIGH	—
Pydantic	pydantic	>=2.9,<3.0	HIGH	OpenEnv + TRL both on Pydantic 2
W&B	wandb	>=0.18,<1.0	HIGH	Default TRL callback
Streamlit (viz + tree)	streamlit	>=1.39,<2.0	HIGH	Richer than Gradio for recursion-tree D3 embed (see §1.1)
Plotly (Pareto chart)	plotly	>=5.24,<6.0	HIGH	—
Matplotlib (quick plots, W&B)	matplotlib	>=3.9,<4.0	HIGH	—
RestrictedPython (REPL AST filter)	RestrictedPython	>=7.4,<8.0	HIGH	zopefoundation/RestrictedPython docs
Claude API (SFT data gen)	anthropic	>=0.40,<1.0	HIGH	SFT trace generator only; not a runtime dep

1.1 Streamlit vs Gradio — decided

2. Canonical Install — Single Block

Step 0: CUDA 12.4 sanity check

Step 1: create env (uv preferred; pip fallback below)

Step 2: torch first (CUDA 12.4 wheel), own line to avoid resolver confusion

Step 3: core training stack

Step 4: Unsloth (pin to Thu-smoke-test version; DO NOT -U at venue)

Step 5: env server + tooling

Step 6: observability + viz + data gen

Step 7: flash-attention (optional but big speedup for generation)

3. Qwen2.5-Coder Specifics

3.1 Primary: Qwen2.5-Coder-1.5B-Instruct

Attribute	Value	Source
HF ID (base)	Qwen/Qwen2.5-Coder-1.5B-Instruct	HF model card
HF ID (Unsloth-preferred for faster load)	unsloth/Qwen2.5-Coder-1.5B-Instruct	unsloth/Qwen2.5-Coder-1.5B-Instruct
HF ID (Unsloth 4-bit bnb pre-quant)	unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit	Unsloth Qwen 2.5 Coder collection
Parameters	1.54B total / 1.31B non-embedding	HF card
BF16 size on disk	~3.1 GB	HF card
4-bit bnb size on disk	~1.1 GB	Unsloth card
Context length (native)	32,768 tokens	HF card config.json
Context length (with YaRN scaling)	131,072 tokens (factor 4.0)	HF card long-context section
Architecture	28 layers, 12 Q-heads, 2 KV-heads (GQA), SwiGLU, RMSNorm, RoPE	HF card
Chat template	ChatML-derived with `<	im_start

3.2 Stretch: Qwen2.5-Coder-3B-Instruct

Attribute	Value
HF ID	Qwen/Qwen2.5-Coder-3B-Instruct
HF ID (Unsloth 4-bit)	unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit
BF16 size on disk	~6.2 GB
4-bit + LoRA (rank 32) VRAM @ 16K context + 8 generations	~38-45 GB (fits A100-80GB with headroom; tight/no-go on A100-40GB)
Context length (native)	32,768 tokens

3.3 Chat template — one source of truth (prevents H6)

For training (SFT with assistant response to learn):

For rollout/eval (model to generate):

• Use Qwen2.5-Coder-1.5B-Instruct (has chat template baked in), never the non-Instruct base.
• Do NOT try return_assistant_tokens_mask=True — broken on Qwen2.5 per transformers #34172. Full-sequence SFT loss is fine for our scale.
• TRL 1.2's OpenEnv integration applies the tokenizer's chat template internally for multi-turn tool calls. Do not hand-concatenate <|im_start|> strings anywhere in our code.
• Diff-test the smoke test: tokenize one SFT example with our prep code; tokenize the same example through GRPOTrainer._prepare_inputs (set a breakpoint); confirm zero token-ID drift.

4. GRPOTrainer Config — Minimum Viable

Passing per-sample reward functions

reward_weights vs compose-in-fn

5. Unsloth vs vLLM for Rollouts — Resolved

6. LoRA Save Sequence — Verbatim Snippet

=== AT END OF GRPO TRAINING (after trainer.train() returns) ===

(1) ALWAYS save adapter-only first. Small (~15MB), never hits the broken merge paths.

This is our reproducibility artifact and our safety net.

(2) Push adapter to HF Hub immediately — cheap and protects against local-disk loss.

(3) Save a merged 16-bit model for fast inference at demo time.

save_method="merged_16bit" is the ONE safe choice. Never use:

- "merged_4bit" → corrupt under QLoRA (#1267, #2339)

- "merged_4bit_forced" → silently emits 16bit (#2339) or size-mismatched (#1791)

If the call fails inside Docker with an empty dir (#3882), fall back to

step (4) manual merge via peft.

trainer.model is already an Unsloth-patched PeftModel

(4) Sanity check BEFORE declaring training done: load the merged model,

run 3 eval questions, assert non-empty output.

Loading back for eval

RoPE scaling for 200K eval

7. Python REPL Sandbox — Choice + Safety Checklist

• Plain subprocess with timeout only — no AST filter means attribute-access escape paths (__subclasses__, catch_warnings, traceback walk) still work inside the subprocess. Model can escape to subprocess.Popen within our own subprocess.
• Pyodide-via-subprocess — spawns a browser-grade WASM runtime per call; 500ms+ startup per REPL action, kills rollout throughput at 8 generations × multiple turns.
• OpenEnv built-in — OpenEnv ships tool primitives but no Python REPL sandbox; the scaffold expects us to write env/server/repl.py (confirmed in ARCHITECTURE.md §1).
• RestrictedPython AST-rewrites at compile time: blocks __class__, __subclasses__, __globals__, attribute access into dunders, star imports. Not a full sandbox but stops 90% of trivial Python-sandbox-escape CTF payloads.
• Subprocess isolation handles the remaining 10%: if a tricky escape produces an import or system call, it hits the OS-level wall (no network, read-only cwd, ulimit'd CPU/memory, seccomp deny-list). Subprocess crashes don't kill the env server.
• RestrictedPython>=7.4 is actively maintained by zopefoundation and mature.

Safety checklist (implement as tests/test_repl_sandbox.py before shipping)

• open — no file I/O
• exec — no dynamic code
• eval — no dynamic eval
• compile — no AST manipulation
• __import__ — whitelist via custom import hook (see below)
• __builtins__ — set to a minimal dict; never pass through the raw module
• input / breakpoint / help / exit / quit — exit paths and interactive prompts
• re, json, math, statistics, collections, itertools, string, functools, operator
• NO os, sys, subprocess, socket, urllib, http, requests, pathlib, io, ctypes, threading, multiprocessing, pickle, marshal, importlib
• Separate subprocess.Popen with start_new_session=True
• ulimit -v 524288 (512MB RAM) via preexec_fn / resource.setrlimit
• ulimit -t 5 (5s CPU), ulimit -f 0 (no file writes)
• cwd=/tmp/episode-{uuid} (ephemeral, wiped on reset())
• Non-root user (venue A100 containers default to uid 1000 — verify at setup)
• Network: unshare -n if available, else seccomp deny socket, connect, sendto
• Timeout: hard-kill at 30s wall-clock (subprocess.wait(timeout=30), then Popen.kill())
• ().__class__.__base__.__subclasses__() — attribute access blocked by RestrictedPython
• e.__traceback__.tb_frame.f_back.f_builtins — traceback walk after a raise
• warnings.catch_warnings() builtins trick — import blocked by whitelist
• __import__('os') — whitelist rejects
• getattr(__builtins__, 'open') — __builtins__ replaced with minimal dict
• 30s-infinite-loop — subprocess killed, env continues
• 1GB memory bomb — rlimit kills, env continues

8. openenv init Scaffold Workflow

8.1 Scaffold

From repo root

or with explicit typed-action template (v0.2.3+):

8.2 Edits we make (post-init)

• Add to src/envs/fathom/server/app.py:
• In environment.py, declare SUPPORTS_CONCURRENT_SESSIONS: bool = True at module scope (required for TRL's 8 parallel generations — confirmed in TRL OpenEnv guide §Server concurrency).
• Implement REPL + llm() primitive per ARCHITECTURE.md §1.

8.3 Local run

In another terminal:

8.4 Deploy to HF Space

Login once (prerequisite):

Push (creates or updates the Space):

Manual push via huggingface_hub:

9. SFT Library Choice

• unsloth.UnslothTrainer — custom trainer, fewer public examples, harder to swap to GRPO afterward.
• Raw training loop — pointless reinvention; SFTTrainer is ~20 LoC of setup.
• Same model object then passes to GRPOTrainer with zero re-init cost (PITFALLS.md H6 prevention).
• TRL 1.2 SFTTrainer accepts Unsloth-patched models without special handling.
• We keep the chat-template formatting in one place (formatting_func) — same template used later in GRPO.

Snippet — adapting Claude-generated conversations to SFT format

(1) Load base model with LoRA adapters attached (Unsloth patch)

(2) Claude traces on disk look like this (one line of JSONL per trajectory):

{"system": "...", "messages": [{"role":"user","content":"..."},

{"role":"assistant","content":"..."},

{"role":"tool","content":"..."},

{"role":"assistant","content":"..."}]}

(3) formatting_func → apply_chat_template (single source of truth, §3.3 rule)

(4) SFT config — light touch so we don't overfit (PITFALLS.md M7)

Save SFT adapter — GRPO will load from here

10. Anti-Patterns — Stack-Specific

10.1 save_pretrained_merged(save_method="merged_4bit") on a QLoRA model

10.2 Mismatched chat template between SFT prep and GRPO rollouts

10.3 max_completion_length > 2048 before the smoke test has run

10.4 vllm_mode="server" for multi-turn OpenEnv training

10.5 pip install -U unsloth at the venue

10.6 Running REPL subprocess in the env server's cwd

10.7 Logging only the composite reward to W&B

10.8 Passing raw <|im_start|>user\n... strings to GRPOTrainer

10.9 Saving merged model without sanity-checking the output

10.10 Using Qwen2.5-Coder-1.5B (base) instead of -Instruct

10.11 Trusting trl vllm-serve in server mode as a separate process

10.12 gradient_checkpointing=False "to go fast"

11. Prerequisite Setup — One-Time Commands at Venue

── G1: GPU alive, CUDA visible

expect: CUDA Version 12.4+, A100-SXM4-40GB or A100-80GB, 0 MiB used

── G2: CUDA toolkit version compatible with torch cu124 wheel

── G3: Python + uv

── G4: Install FATHOM env (single pip block from §2 above)

run the §2 commands here

── G5: Verify torch sees the A100

expect: True NVIDIA A100-...

── G6: HF login (write token from hf.co/settings/tokens)

paste token; verify:

── G7: W&B login (API key from wandb.ai/authorize)

paste key; verify:

── G8: OpenEnv CLI alive

── G9: Pre-pull the model into HF cache (avoid mid-training download stalls)

── G10: Smoke-test GRPOTrainer imports (catches version-skew issues)

── G11: Sandbox unit tests (these MUST pass before env server goes public)

── G12: End-to-end smoke — 1 GRPO step

12. Confidence & Verification Matrix

Recommendation	Confidence	Basis	If LOW → 10-min venue verification
TRL == 1.2.0	HIGH	PyPI dated 2026-04-17	—
Unsloth == 2026.4.8 (v0.1.36-beta)	MEDIUM	GitHub releases page; rapid iteration	python -c "import unsloth; print(unsloth.__version__)" then run G12 smoke test
OpenEnv-core >= 0.2.3	HIGH	GitHub tag + TRL guide references 0.2.1+	—
torch 2.6 cu124	HIGH	PyTorch wheel index stable	torch.version.cuda == '12.4'
vllm 0.7.3	HIGH	TRL 1.2 OpenEnv examples target it	vllm --version
vllm_mode="colocate" for multi-turn	HIGH	TRL OpenEnv guide Wordle example + PITFALLS #4543	Run G12 smoke with 1 turn, then with 2 turns — advantage should look normal
beta=0.04 KL floor	MEDIUM	EDGE-GRPO paper + PITFALLS H1; TRL default is 0.0	Train 50 steps with β=0.0; if group_std drops <0.05 → switch to 0.04
num_generations=8	HIGH	TRL default; all OpenEnv examples	—
learning_rate=5e-6 with 4-bit	MEDIUM	PITFALLS M1 "4-bit safe band 5e-6 to 1e-5"	First 20 steps — if loss NaN, drop to 3e-6; if flat, raise to 1e-5
max_grad_norm=0.5	MEDIUM	PITFALLS M1 conservative	Widen to 1.0 only if training too conservative and loss plateaus
Unsloth Qwen2.5-Coder-1.5B-Instruct-bnb-4bit	HIGH	HF Unsloth collection page	—
YaRN factor=4.0 for 200K eval	MEDIUM	HF model card long-context section; PITFALLS M8 warns	At eval time, test 50K → 100K → 150K → 200K; note first OOM point, cap claim there
Merged-16bit save as the only trusted path	HIGH	PITFALLS R4 (5 issue numbers); §6 snippet follows Unsloth docs	—
RestrictedPython + subprocess sandbox	HIGH	Canonical pattern; dida.do / tianpan.co verified	Run the 7 adversarial unit tests in §7 — if any pass, patch before shipping
Streamlit (not Gradio) for demo Space	MEDIUM	Our override on FEATURES.md TS-DEM-2 rationale	Build both skeletons Thu; pick the one that renders the D3 tree cleaner in 30 min
SFTTrainer + Unsloth model	HIGH	Unsloth notebook collection; TRL 1.2 supports Unsloth models	—
openenv push for HF Space	MEDIUM	OpenEnv CLI 0.2.3 docs; new CLI	Try push Thu; if it fails, use the manual git push fallback in §8.4
Chat template one-source-of-truth	HIGH	PITFALLS H6 + HF Qwen card	Diff-test token IDs in smoke — zero drift is the gate

• Unsloth exact version pin (re-run G10 smoke after pinning)
• vLLM colocate memory fraction — vllm_gpu_memory_utilization=0.45 may need tuning on A100-40GB vs 80GB; watch nvidia-smi during the first 10 GRPO steps
• β (KL coefficient) — if group_std stays healthy at 0.0, save the 1% throughput; if collapse appears, bump to 0.04. Cheap to toggle mid-run via config.

13. Sources

• TRL 1.2.0 on PyPI (2026-04-17) — HIGH
• TRL OpenEnv Integration guide — HIGH
• TRL GRPOConfig on GitHub main — HIGH
• Unsloth releases on GitHub — HIGH
• Unsloth RL Guide — HIGH
• Unsloth Qwen 2.5 Coder collection — HIGH
• Qwen2.5-Coder-1.5B-Instruct HF card — HIGH
• Unsloth/Qwen2.5-Coder-1.5B-Instruct HF card — HIGH
• meta-pytorch/OpenEnv GitHub v0.2.3 — HIGH
• vLLM training/trl docs — HIGH
• RestrictedPython docs — HIGH
• Unsloth LoRA save bug issues: #1791, #2339, #1267, #3882, #1877 (PITFALLS.md R4) — HIGH
• TRL multi-turn bug: #4543 (PITFALLS.md H2) — HIGH
• Chat template gotcha: transformers #34172 (PITFALLS.md N2) — HIGH
• Qwen2.5 + RoPE scaling for 200K: HF card + Unsloth #3083 (PITFALLS.md M8) — HIGH
• OpenEnv env server design: ARCHITECTURE.md §1, §8 — HIGH
• Reward decomposition pattern: ARCHITECTURE.md §6 Pattern 1 — HIGH
• Sandbox pattern choice: dida.do + tianpan.co via PITFALLS.md H3 — HIGH

Conventions

Conventions not yet established. Will populate as patterns emerge during development.

Architecture

Architecture not yet mapped. Follow existing patterns found in the codebase.

Project Skills

No project skills found. Add skills to any of: .claude/skills/, .agents/skills/, .cursor/skills/, or .github/skills/ with a SKILL.md index file.

GSD Workflow Enforcement

Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.

Use these entry points:

• /gsd-quick for small fixes, doc updates, and ad-hoc tasks
• /gsd-debug for investigation and bug fixing
• /gsd-execute-phase for planned phase work

Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.

Developer Profile

Profile not yet configured. Run /gsd-profile-user to generate your developer profile. This section is managed by generate-claude-profile -- do not edit manually.