docs/RUNPOD_GUIDELINES.md

RECOVERED 2026-05-25T13:57:12 from Claude session transcripts (last Write; later Edit-tool revisions not applied). Original file was missing from disk.

RunPod Guidelines

Source-of-truth for everything we've done on RunPod from aibox. Created 2026-05-25 13:55 UTC per twisted's ask after the heretic-Gembrain Fisher landed on Pro 6000 instead of aibox.

See also:

• EVAL_GUIDELINES.md — local (aibox) eval pipeline
• UPLOAD_GUIDELINES.md — HF upload patterns (most apply on pod too)
• TRAINING_GUIDELINES.md — local training conventions
• .claude/memory/project_b200_runpod_orchestration.md — original detailed B200 session notes

---

0. SSH access

Keypair (REUSE — same key for every pod)

• Private: ~/.ssh/runpod_b200 (mode 0600, generated 2026-05-16)
• Public: ~/.ssh/runpod_b200.pub
• Authorized once at the RunPod account level ("Settings → SSH Public Keys"). Every pod twisted spins up inherits this key — do NOT generate per-session keys.
• Despite the name, this is the general aibox→RunPod key, not B200-specific. Use it for Pro 6000, A100, B200, etc.

Two SSH paths — always prefer Direct TCP

1. Direct TCP via the pod's "Connect → Direct TCP" panel — e.g. ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST. Use this first. No PTY required, works for stdin piping, SCP/SFTP/rsync all work.
2. Proxy via ssh.runpod.io — frequently hangs/times out, requires -tt PTY flag for command-based interaction, useless for SCP/SFTP/rsync. Only fall back to this if Direct TCP is unavailable (rare).

Stdin-pipe pattern that works (Direct TCP, no PTY)

cat /tmp/local_script.py | ssh -i ~/.ssh/runpod_b200 \
    -o StrictHostKeyChecking=no -o ConnectTimeout=20 \
    -p $PORT root@$HOST "cat > /workspace/remote_script.py"

Workaround if BOTH SSH paths fail

Use an HF dataset repo as binary transit. Pattern from 2026-05-16: tarball the runtime + data + scripts, push to rpDungeon/<name>-transit-<date> (private), pod pulls via huggingface_hub.snapshot_download. Slower than direct upload but works when SSH is fully wedged.

---

1. Known hardware tiers + cost calibration

Tier	VRAM	$/hr	Use-case	Notes
B200 secure	180 GB HBM3	$5.51	Long training runs (multi-hour FSDP/QLoRA)	Used 2026-05-16/17 for glimmer-b200-final E4B run, ~7.5h pod lifetime ≈ $42. Billing starts at spin-up, not training launch.
RTX Pro 6000	96 GB	~$3	One-off heavy ops that need whole-model-in-VRAM (31B Fisher, large activation studies)	Heretic-Gembrain Fisher launched here 2026-05-25 13:51 UTC. Per INSTRUCT_MASKING_TECHNIQUES.md L427: 31B-IT Fisher extraction = 618 s wall (~10 min).
A100 80GB	80 GB	varies	Mid-tier; usually overkill for E4B, undersized for 31B in bf16	Not yet used by us.

Plan billing windows. Pod is billed from spin-up. Pre-stage everything you can locally (model IDs, scripts, prompt files) so on-pod setup is minutes, not an hour.

---

2. Common operations

2.1 Fisher extraction on 31B-class models

Use Pro 6000 (96 GB) — the optimized recipe assumes whole-model-in-VRAM. On aibox's 2×24 GB 3090s we have to cpu-offload, which destroys the GPU-resident accumulator speedup. Recipe:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python extract_fisher_31b.py \
    --model <HF_ID_or_local_path> \
    --prompts <ifeval_input_data.jsonl> \
    --out <output_dir> \
    --n-prompts 200 \
    --max-length 256 \
    --layers-per-batch 8 \
    --fisher-dtype bf16

Output: fisher.pt (~35 GB bf16 for 31B) + layer_importance.json. Wall: ~10 min on Pro 6000.

Don't try this on aibox for 31B-class models. First attempt OOM'd at the per-batch fp32 accumulator allocation (GPU0 hit 23.22 GiB / 24 GiB cap) even with --cpu-offload --gpu-memory-gib 18,18 --layers-per-batch 8. The optimization is fundamentally for whole-model-in-VRAM.

2.2 Training (long jobs that need beefier GPUs than aibox)

See project_b200_runpod_orchestration.md for the full glimmer-b200-final flow. Key gotchas migrated below.

2.3 Upload artifacts back to HF

Same huggingface_hub.HfApi.upload_file / upload_folder patterns as aibox. Always set HF_HUB_DISABLE_XET=1 — see §3 below.

---

3. Pod-side upload gotchas

gguf package missing at first use

Pod's stock python doesn't ship the gguf package. Fix: pip install gguf at bootstrap time, not on first import. If the GGUF stage fails and it's later in the chain, none of the later upload stages run — so put adapter/safetensor uploads FIRST in your script, before any merge/quant/GGUF stages.

xet uploads stall silently

Pattern: tqdm bar freezes at some % for an hour, xet/logs/*.log last entry is stale, process is alive in S state, socket in CLOSE_WAIT. Always upload with HF_HUB_DISABLE_XET=1 to use legacy LFS — much more resilient on pod-side networking. Same fix applies to aibox uploads but is critical on pod.

Verify file sizes byte-exact before declaring "done"

HF's upload "DONE" message can land before xet finishes a final commit. The tree API is ground truth:

import requests
d = requests.get(f"https://huggingface.co/api/models/{REPO}/tree/main/{PATH}",
                 headers={"Authorization": f"Bearer {TOK}"}).json()
for x in d:
    print(x['path'], x.get('size','-'))

Compare each against pod-side ls -la. Example bar from glimmer-b200-final: adapter 558,181,856 B both sides; merged 15,882,477,500 B both sides.

---

4. Watchdog pattern for "stop chain after stage N completes"

Useful when one stage in pod_train.py is reliably broken (e.g. GGUF stage when gguf not installed) and you want everything before it to upload, then stop:

while true; do
  sleep 30
  STATE=$(curl -s -H "Authorization: Bearer $TOK" \
    "https://huggingface.co/api/models/$REPO/tree/main/$PATH" | grep -c "$TARGET_FILE")
  if [ "$STATE" -gt 0 ]; then
    ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST 'pkill -9 -f UPLOAD_SCRIPT.py'
    break
  fi
done

Used in glimmer-b200-final to short-circuit pod_train.py after merged uploaded, before it tried the broken GGUF stage.

---

5. Bootstrap checklist (pre-stage these locally before pod spins up)

1. Confirm the HF model ID (e.g. llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic).
2. Stage your script(s) locally and pre-test for syntax (bash -n, python -m py_compile).
3. Prepare an environment-setup snippet (pip install gguf huggingface_hub safetensors transformers torch ...) — minimize on-pod pip install cycles.
4. Know the HF target repo + token env-var name (HF_TOKEN).
5. Have the IFEval / calibration prompts JSONL path on aibox; either copy via SSH or download from HF (google/IFEval).
6. Pre-write the upload script (don't write it on-pod).

---

6. Session log (chronological — what we've actually run)

Date	Pod	Tier	Job	Duration	Cost	Outcome
2026-05-16/17	tnur38utzadupc-64411e3a	B200 secure	E4B LoRA r=64 α=16 + mask r=256 SFT on 7-corpus mix, 16k ctx, 1 ep, 2083 steps	~6.6h training, ~7.5h pod lifetime	~$42	Adapter + merged on rpDungeon/glimmer-b200-final. GGUF failed pod-side (missing module), converted locally.
2026-05-25 13:51 UTC	(twisted spinning up)	Pro 6000 96 GB	Heretic-Gembrain (31B) Fisher extraction	ETA ~10-15 min	~$1	In progress — uploads to rpDungeon/gemma-4-masks.

---

7. Quick command reference

Direct TCP SSH:

ssh -i ~/.ssh/runpod_b200 -o StrictHostKeyChecking=no -p $PORT root@$HOST

Pipe local file to pod:

cat /local/file | ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST "cat > /workspace/file"

SCP a tree:

scp -i ~/.ssh/runpod_b200 -P $PORT -r local_dir/ root@$HOST:/workspace/

Run command and stream stdout:

ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST "cd /workspace && python script.py" | tee local.log