> RECOVERED 2026-05-25T13:57:12 from Claude session transcripts (last Write; later Edit-tool revisions not applied). Original file was missing from disk.

# RunPod Guidelines

Source-of-truth for everything we've done on RunPod from aibox. Created 2026-05-25 13:55 UTC per twisted's ask after the heretic-Gembrain Fisher landed on Pro 6000 instead of aibox.

**See also:**
- `EVAL_GUIDELINES.md` — local (aibox) eval pipeline
- `UPLOAD_GUIDELINES.md` — HF upload patterns (most apply on pod too)
- `TRAINING_GUIDELINES.md` — local training conventions
- `.claude/memory/project_b200_runpod_orchestration.md` — original detailed B200 session notes

---

## 0. SSH access

### Keypair (REUSE — same key for every pod)

- Private: `~/.ssh/runpod_b200` (mode 0600, generated 2026-05-16)
- Public: `~/.ssh/runpod_b200.pub`
- **Authorized once at the RunPod account level** ("Settings → SSH Public Keys"). Every pod twisted spins up inherits this key — do NOT generate per-session keys.
- Despite the name, this is the **general aibox→RunPod key**, not B200-specific. Use it for Pro 6000, A100, B200, etc.

### Two SSH paths — always prefer Direct TCP

1. **Direct TCP** via the pod's "Connect → Direct TCP" panel — e.g. `ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST`. **Use this first.** No PTY required, works for stdin piping, SCP/SFTP/rsync all work.
2. **Proxy** via `ssh.runpod.io` — frequently hangs/times out, requires `-tt` PTY flag for command-based interaction, useless for SCP/SFTP/rsync. Only fall back to this if Direct TCP is unavailable (rare).

### Stdin-pipe pattern that works (Direct TCP, no PTY)

```bash
cat /tmp/local_script.py | ssh -i ~/.ssh/runpod_b200 \
    -o StrictHostKeyChecking=no -o ConnectTimeout=20 \
    -p $PORT root@$HOST "cat > /workspace/remote_script.py"
```

### Workaround if BOTH SSH paths fail

Use an HF dataset repo as binary transit. Pattern from 2026-05-16: tarball the runtime + data + scripts, push to `rpDungeon/<name>-transit-<date>` (private), pod pulls via `huggingface_hub.snapshot_download`. Slower than direct upload but works when SSH is fully wedged.

---

## 1. Known hardware tiers + cost calibration

| Tier | VRAM | $/hr | Use-case | Notes |
|---|---|---|---|---|
| B200 secure | 180 GB HBM3 | $5.51 | Long training runs (multi-hour FSDP/QLoRA) | Used 2026-05-16/17 for glimmer-b200-final E4B run, ~7.5h pod lifetime ≈ $42. Billing starts at **spin-up**, not training launch. |
| RTX Pro 6000 | 96 GB | ~$3 | One-off heavy ops that need whole-model-in-VRAM (31B Fisher, large activation studies) | Heretic-Gembrain Fisher launched here 2026-05-25 13:51 UTC. Per `INSTRUCT_MASKING_TECHNIQUES.md` L427: 31B-IT Fisher extraction = 618 s wall (~10 min). |
| A100 80GB | 80 GB | varies | Mid-tier; usually overkill for E4B, undersized for 31B in bf16 | Not yet used by us. |

**Plan billing windows.** Pod is billed from spin-up. Pre-stage everything you can locally (model IDs, scripts, prompt files) so on-pod setup is minutes, not an hour.

---

## 2. Common operations

### 2.1 Fisher extraction on 31B-class models

**Use Pro 6000 (96 GB) — the optimized recipe assumes whole-model-in-VRAM.** On aibox's 2×24 GB 3090s we have to cpu-offload, which destroys the GPU-resident accumulator speedup. Recipe:

```bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python extract_fisher_31b.py \
    --model <HF_ID_or_local_path> \
    --prompts <ifeval_input_data.jsonl> \
    --out <output_dir> \
    --n-prompts 200 \
    --max-length 256 \
    --layers-per-batch 8 \
    --fisher-dtype bf16
```

Output: `fisher.pt` (~35 GB bf16 for 31B) + `layer_importance.json`. Wall: ~10 min on Pro 6000.

**Don't try this on aibox for 31B-class models.** First attempt OOM'd at the per-batch fp32 accumulator allocation (GPU0 hit 23.22 GiB / 24 GiB cap) even with `--cpu-offload --gpu-memory-gib 18,18 --layers-per-batch 8`. The optimization is fundamentally for whole-model-in-VRAM.

### 2.2 Training (long jobs that need beefier GPUs than aibox)

See `project_b200_runpod_orchestration.md` for the full glimmer-b200-final flow. Key gotchas migrated below.

### 2.3 Upload artifacts back to HF

Same `huggingface_hub.HfApi.upload_file` / `upload_folder` patterns as aibox. **Always set `HF_HUB_DISABLE_XET=1`** — see §3 below.

---

## 3. Pod-side upload gotchas

### `gguf` package missing at first use

Pod's stock python doesn't ship the `gguf` package. **Fix:** `pip install gguf` at bootstrap time, not on first import. If the GGUF stage fails and it's later in the chain, **none of the later upload stages run** — so put adapter/safetensor uploads FIRST in your script, before any merge/quant/GGUF stages.

### xet uploads stall silently

Pattern: tqdm bar freezes at some % for an hour, `xet/logs/*.log` last entry is stale, process is alive in `S` state, socket in `CLOSE_WAIT`. **Always upload with `HF_HUB_DISABLE_XET=1`** to use legacy LFS — much more resilient on pod-side networking. Same fix applies to aibox uploads but is critical on pod.

### Verify file sizes byte-exact before declaring "done"

HF's upload "DONE" message can land before xet finishes a final commit. The tree API is ground truth:

```python
import requests
d = requests.get(f"https://huggingface.co/api/models/{REPO}/tree/main/{PATH}",
                 headers={"Authorization": f"Bearer {TOK}"}).json()
for x in d:
    print(x['path'], x.get('size','-'))
```

Compare each against pod-side `ls -la`. Example bar from glimmer-b200-final: adapter 558,181,856 B both sides; merged 15,882,477,500 B both sides.

---

## 4. Watchdog pattern for "stop chain after stage N completes"

Useful when one stage in `pod_train.py` is reliably broken (e.g. GGUF stage when `gguf` not installed) and you want everything before it to upload, then stop:

```bash
while true; do
  sleep 30
  STATE=$(curl -s -H "Authorization: Bearer $TOK" \
    "https://huggingface.co/api/models/$REPO/tree/main/$PATH" | grep -c "$TARGET_FILE")
  if [ "$STATE" -gt 0 ]; then
    ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST 'pkill -9 -f UPLOAD_SCRIPT.py'
    break
  fi
done
```

Used in glimmer-b200-final to short-circuit `pod_train.py` after `merged` uploaded, before it tried the broken GGUF stage.

---

## 5. Bootstrap checklist (pre-stage these locally before pod spins up)

1. Confirm the HF model ID (e.g. `llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic`).
2. Stage your script(s) locally and pre-test for syntax (`bash -n`, `python -m py_compile`).
3. Prepare an environment-setup snippet (`pip install gguf huggingface_hub safetensors transformers torch ...`) — minimize on-pod `pip install` cycles.
4. Know the HF target repo + token env-var name (`HF_TOKEN`).
5. Have the IFEval / calibration prompts JSONL path on aibox; either copy via SSH or download from HF (`google/IFEval`).
6. Pre-write the upload script (don't write it on-pod).

---

## 6. Session log (chronological — what we've actually run)

| Date | Pod | Tier | Job | Duration | Cost | Outcome |
|---|---|---|---|---|---|---|
| 2026-05-16/17 | tnur38utzadupc-64411e3a | B200 secure | E4B LoRA r=64 α=16 + mask r=256 SFT on 7-corpus mix, 16k ctx, 1 ep, 2083 steps | ~6.6h training, ~7.5h pod lifetime | ~$42 | Adapter + merged on `rpDungeon/glimmer-b200-final`. GGUF failed pod-side (missing module), converted locally. |
| 2026-05-25 13:51 UTC | (twisted spinning up) | Pro 6000 96 GB | Heretic-Gembrain (31B) Fisher extraction | ETA ~10-15 min | ~$1 | In progress — uploads to `rpDungeon/gemma-4-masks`. |

---

## 7. Quick command reference

Direct TCP SSH:
```bash
ssh -i ~/.ssh/runpod_b200 -o StrictHostKeyChecking=no -p $PORT root@$HOST
```

Pipe local file to pod:
```bash
cat /local/file | ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST "cat > /workspace/file"
```

SCP a tree:
```bash
scp -i ~/.ssh/runpod_b200 -P $PORT -r local_dir/ root@$HOST:/workspace/
```

Run command and stream stdout:
```bash
ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST "cd /workspace && python script.py" | tee local.log
```