Initial release: instruct-preserving masked training & merging core tools + guidelines (recovered from dev logs, sanitized)
a4df021 verified | > RECOVERED 2026-05-25T13:57:12 from Claude session transcripts (last Write; later Edit-tool revisions not applied). Original file was missing from disk. | |
| # RunPod Guidelines | |
| Source-of-truth for everything we've done on RunPod from aibox. Created 2026-05-25 13:55 UTC per twisted's ask after the heretic-Gembrain Fisher landed on Pro 6000 instead of aibox. | |
| **See also:** | |
| - `EVAL_GUIDELINES.md` — local (aibox) eval pipeline | |
| - `UPLOAD_GUIDELINES.md` — HF upload patterns (most apply on pod too) | |
| - `TRAINING_GUIDELINES.md` — local training conventions | |
| - `.claude/memory/project_b200_runpod_orchestration.md` — original detailed B200 session notes | |
| --- | |
| ## 0. SSH access | |
| ### Keypair (REUSE — same key for every pod) | |
| - Private: `~/.ssh/runpod_b200` (mode 0600, generated 2026-05-16) | |
| - Public: `~/.ssh/runpod_b200.pub` | |
| - **Authorized once at the RunPod account level** ("Settings → SSH Public Keys"). Every pod twisted spins up inherits this key — do NOT generate per-session keys. | |
| - Despite the name, this is the **general aibox→RunPod key**, not B200-specific. Use it for Pro 6000, A100, B200, etc. | |
| ### Two SSH paths — always prefer Direct TCP | |
| 1. **Direct TCP** via the pod's "Connect → Direct TCP" panel — e.g. `ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST`. **Use this first.** No PTY required, works for stdin piping, SCP/SFTP/rsync all work. | |
| 2. **Proxy** via `ssh.runpod.io` — frequently hangs/times out, requires `-tt` PTY flag for command-based interaction, useless for SCP/SFTP/rsync. Only fall back to this if Direct TCP is unavailable (rare). | |
| ### Stdin-pipe pattern that works (Direct TCP, no PTY) | |
| ```bash | |
| cat /tmp/local_script.py | ssh -i ~/.ssh/runpod_b200 \ | |
| -o StrictHostKeyChecking=no -o ConnectTimeout=20 \ | |
| -p $PORT root@$HOST "cat > /workspace/remote_script.py" | |
| ``` | |
| ### Workaround if BOTH SSH paths fail | |
| Use an HF dataset repo as binary transit. Pattern from 2026-05-16: tarball the runtime + data + scripts, push to `rpDungeon/<name>-transit-<date>` (private), pod pulls via `huggingface_hub.snapshot_download`. Slower than direct upload but works when SSH is fully wedged. | |
| --- | |
| ## 1. Known hardware tiers + cost calibration | |
| | Tier | VRAM | $/hr | Use-case | Notes | | |
| |---|---|---|---|---| | |
| | B200 secure | 180 GB HBM3 | $5.51 | Long training runs (multi-hour FSDP/QLoRA) | Used 2026-05-16/17 for glimmer-b200-final E4B run, ~7.5h pod lifetime ≈ $42. Billing starts at **spin-up**, not training launch. | | |
| | RTX Pro 6000 | 96 GB | ~$3 | One-off heavy ops that need whole-model-in-VRAM (31B Fisher, large activation studies) | Heretic-Gembrain Fisher launched here 2026-05-25 13:51 UTC. Per `INSTRUCT_MASKING_TECHNIQUES.md` L427: 31B-IT Fisher extraction = 618 s wall (~10 min). | | |
| | A100 80GB | 80 GB | varies | Mid-tier; usually overkill for E4B, undersized for 31B in bf16 | Not yet used by us. | | |
| **Plan billing windows.** Pod is billed from spin-up. Pre-stage everything you can locally (model IDs, scripts, prompt files) so on-pod setup is minutes, not an hour. | |
| --- | |
| ## 2. Common operations | |
| ### 2.1 Fisher extraction on 31B-class models | |
| **Use Pro 6000 (96 GB) — the optimized recipe assumes whole-model-in-VRAM.** On aibox's 2×24 GB 3090s we have to cpu-offload, which destroys the GPU-resident accumulator speedup. Recipe: | |
| ```bash | |
| PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ | |
| python extract_fisher_31b.py \ | |
| --model <HF_ID_or_local_path> \ | |
| --prompts <ifeval_input_data.jsonl> \ | |
| --out <output_dir> \ | |
| --n-prompts 200 \ | |
| --max-length 256 \ | |
| --layers-per-batch 8 \ | |
| --fisher-dtype bf16 | |
| ``` | |
| Output: `fisher.pt` (~35 GB bf16 for 31B) + `layer_importance.json`. Wall: ~10 min on Pro 6000. | |
| **Don't try this on aibox for 31B-class models.** First attempt OOM'd at the per-batch fp32 accumulator allocation (GPU0 hit 23.22 GiB / 24 GiB cap) even with `--cpu-offload --gpu-memory-gib 18,18 --layers-per-batch 8`. The optimization is fundamentally for whole-model-in-VRAM. | |
| ### 2.2 Training (long jobs that need beefier GPUs than aibox) | |
| See `project_b200_runpod_orchestration.md` for the full glimmer-b200-final flow. Key gotchas migrated below. | |
| ### 2.3 Upload artifacts back to HF | |
| Same `huggingface_hub.HfApi.upload_file` / `upload_folder` patterns as aibox. **Always set `HF_HUB_DISABLE_XET=1`** — see §3 below. | |
| --- | |
| ## 3. Pod-side upload gotchas | |
| ### `gguf` package missing at first use | |
| Pod's stock python doesn't ship the `gguf` package. **Fix:** `pip install gguf` at bootstrap time, not on first import. If the GGUF stage fails and it's later in the chain, **none of the later upload stages run** — so put adapter/safetensor uploads FIRST in your script, before any merge/quant/GGUF stages. | |
| ### xet uploads stall silently | |
| Pattern: tqdm bar freezes at some % for an hour, `xet/logs/*.log` last entry is stale, process is alive in `S` state, socket in `CLOSE_WAIT`. **Always upload with `HF_HUB_DISABLE_XET=1`** to use legacy LFS — much more resilient on pod-side networking. Same fix applies to aibox uploads but is critical on pod. | |
| ### Verify file sizes byte-exact before declaring "done" | |
| HF's upload "DONE" message can land before xet finishes a final commit. The tree API is ground truth: | |
| ```python | |
| import requests | |
| d = requests.get(f"https://huggingface.co/api/models/{REPO}/tree/main/{PATH}", | |
| headers={"Authorization": f"Bearer {TOK}"}).json() | |
| for x in d: | |
| print(x['path'], x.get('size','-')) | |
| ``` | |
| Compare each against pod-side `ls -la`. Example bar from glimmer-b200-final: adapter 558,181,856 B both sides; merged 15,882,477,500 B both sides. | |
| --- | |
| ## 4. Watchdog pattern for "stop chain after stage N completes" | |
| Useful when one stage in `pod_train.py` is reliably broken (e.g. GGUF stage when `gguf` not installed) and you want everything before it to upload, then stop: | |
| ```bash | |
| while true; do | |
| sleep 30 | |
| STATE=$(curl -s -H "Authorization: Bearer $TOK" \ | |
| "https://huggingface.co/api/models/$REPO/tree/main/$PATH" | grep -c "$TARGET_FILE") | |
| if [ "$STATE" -gt 0 ]; then | |
| ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST 'pkill -9 -f UPLOAD_SCRIPT.py' | |
| break | |
| fi | |
| done | |
| ``` | |
| Used in glimmer-b200-final to short-circuit `pod_train.py` after `merged` uploaded, before it tried the broken GGUF stage. | |
| --- | |
| ## 5. Bootstrap checklist (pre-stage these locally before pod spins up) | |
| 1. Confirm the HF model ID (e.g. `llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic`). | |
| 2. Stage your script(s) locally and pre-test for syntax (`bash -n`, `python -m py_compile`). | |
| 3. Prepare an environment-setup snippet (`pip install gguf huggingface_hub safetensors transformers torch ...`) — minimize on-pod `pip install` cycles. | |
| 4. Know the HF target repo + token env-var name (`HF_TOKEN`). | |
| 5. Have the IFEval / calibration prompts JSONL path on aibox; either copy via SSH or download from HF (`google/IFEval`). | |
| 6. Pre-write the upload script (don't write it on-pod). | |
| --- | |
| ## 6. Session log (chronological — what we've actually run) | |
| | Date | Pod | Tier | Job | Duration | Cost | Outcome | | |
| |---|---|---|---|---|---|---| | |
| | 2026-05-16/17 | tnur38utzadupc-64411e3a | B200 secure | E4B LoRA r=64 α=16 + mask r=256 SFT on 7-corpus mix, 16k ctx, 1 ep, 2083 steps | ~6.6h training, ~7.5h pod lifetime | ~$42 | Adapter + merged on `rpDungeon/glimmer-b200-final`. GGUF failed pod-side (missing module), converted locally. | | |
| | 2026-05-25 13:51 UTC | (twisted spinning up) | Pro 6000 96 GB | Heretic-Gembrain (31B) Fisher extraction | ETA ~10-15 min | ~$1 | In progress — uploads to `rpDungeon/gemma-4-masks`. | | |
| --- | |
| ## 7. Quick command reference | |
| Direct TCP SSH: | |
| ```bash | |
| ssh -i ~/.ssh/runpod_b200 -o StrictHostKeyChecking=no -p $PORT root@$HOST | |
| ``` | |
| Pipe local file to pod: | |
| ```bash | |
| cat /local/file | ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST "cat > /workspace/file" | |
| ``` | |
| SCP a tree: | |
| ```bash | |
| scp -i ~/.ssh/runpod_b200 -P $PORT -r local_dir/ root@$HOST:/workspace/ | |
| ``` | |
| Run command and stream stdout: | |
| ```bash | |
| ssh -i ~/.ssh/runpod_b200 -p $PORT root@$HOST "cd /workspace && python script.py" | tee local.log | |
| ``` | |