Resume bundle: step 100000/154000 checkpoint + code

README_RUN.md

@@ -0,0 +1,94 @@
+# feedforward — resume bundle (step 100000 / 154000)
+
+A training run stopped at optimizer step 100000 of 154000. This bundle has the
+checkpoint and the code. Two things you must supply yourself — see "What you need
+to add" below.
+
+## Contents
+
+| File | Size | Goes to |
+|---|---|---|
+| `checkpoint_last.pt` | 12.9 GiB | `feedforward/outputs/full/checkpoint_last.pt` |
+| `feedforward_code.tar.gz` | 523 KiB | unpacks to `feedforward/` |
+| `SHA256` | — | checksums for the two above |
+
+## What you need to add
+
+**1. The pretrained VGGT-Omega backbone** — `weights/vggt_omega_1b_512.pt`, the
+official 1B fp32 checkpoint. It is **not** in this bundle: the HuggingFace repo is
+gated, so you request access and download it under your own agreement. Once you have
+it, verify you got the identical file:
+
+```
+sha256  c02da418b18bb01d0392598d3f6147366bcde1bb70fd08a5e3bf7925b0667934
+size    4576706117 bytes
+```
+
+Do not substitute `facebook/VGGT-1B` — different architecture, the keys will not load
+(`dense_head` here vs. `depth_head`/`point_head`/`track_head` there). Do not
+substitute the bf16 release either; it shifts the distillation numerics 100k steps
+into a run.
+
+**2. RealEstate10K** in the pixelSplat shard layout (~510 G):
+`<root>/train/*.torch`, `<root>/test/*.torch`, `<root>/test/index.json`.
+
+## Setup
+
+```bash
+sha256sum -c SHA256                       # a 13.8 G transfer earns a check
+
+mkdir -p feedforward && tar -xzf feedforward_code.tar.gz -C feedforward
+cd feedforward
+mkdir -p outputs/full weights
+mv ../checkpoint_last.pt outputs/full/
+cp /wherever/you/downloaded/vggt_omega_1b_512.pt weights/
+
+pip install -r requirements.txt           # install the torch build matching your driver first
+```
+
+## Three things that will bite you
+
+1. **`weights/vggt_omega_1b_512.pt` is not optional, even to resume.**
+   `model.load_pretrained()` at train.py:405 runs unconditionally *before* the resume
+   block at train.py:438, and asserts 0 missing / 0 unexpected. Without the file the
+   job dies within seconds — even though the checkpoint is about to overwrite those
+   weights anyway. The frozen teacher then loads it a second time (train.py:417),
+   because `DISTILL_TEACHER=1` and it is used at every step.
+
+2. **`checkpoint_last.pt` must sit inside `$OUT_DIR`** (`outputs/full` by default).
+   train.py:438 checks that one path and nothing else; anywhere else and the run
+   starts fresh from step 0 without complaining. Confirm `RESUMED from step 100000`
+   appears in the log before walking away.
+
+3. **Throughput drops here.** Step 100000 is exactly where the curriculum's second
+   phase opens (`ANISO_AFTER=100000 SHAPE_AFTER=100000`) and the cosine schedule
+   restarts. Expect ~3 it/s rather than the ~17 it/s of the first 100k steps. All 54k
+   remaining steps are in the expensive phases — budget wall-clock accordingly.
+
+## Running
+
+Smoke test first. It hard-fails on things like gsplat being unable to rasterize, which
+otherwise surfaces as a silent ~15 dB quality drop rather than an exception:
+
+```bash
+DATA_ROOT=/path/to/re10k scripts/smoke.sh
+```
+
+Then confirm the `ARM=` line reads
+`curriculum=[aniso@100000 shapeK3@100000->K5@130000 app@130000]` with
+`restarts=100000,130000`, and launch:
+
+```bash
+GPUS=4 DATA_ROOT=/path/to/re10k scripts/train_full.sh
+```
+
+`docs/TRAINING.md` has the full curriculum, multi-GPU notes, protocols and the traps
+hit along the way. Read it before committing a long allocation.
+
+`logs/` holds the launchers from the originating cluster (SLURM, 4x H100). Their
+partition names, account and module paths are site-specific — treat them as reference,
+not as scripts to run unmodified.
+
+Note: `scripts/train_full.sh` differs from git HEAD by one change (`torchrun` →
+`$PY -m torch.distributed.run`), which fixes a job that died on `torchrun: not found`
+after the recipe check had already passed. That is intentional; keep it.

SHA256

@@ -0,0 +1,2 @@
+793337fabe00fb95c3b76f9fc4446fa8fed55aab3d1da039941e905857a0d58f *checkpoint_last.pt
+5e4754d9eed949ec3e6df771124dd1d76c77bcd4501448b0948eca2aa7d54954 *feedforward_code.tar.gz

checkpoint_last.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:793337fabe00fb95c3b76f9fc4446fa8fed55aab3d1da039941e905857a0d58f
+size 13831021061

feedforward_code.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e4754d9eed949ec3e6df771124dd1d76c77bcd4501448b0948eca2aa7d54954
+size 535391