Handoff: single seed, no warmup, 480p only

HANDOFF.md

@@ -9,7 +9,7 @@ You are an agent on an Apple M5 Mac Studio. Your job is to **install, build the
 - If a fix would need more than a few lines, or touches model math, stop, record the error and logs, and report instead.
 
 ## What is being measured
-FastH3 Omni is a reference-conditioned video+audio model that runs through FastVideo's native MLX runtime. Measure end-to-end time per clip for **5 s clips (124 frames) at 480p (832x480) with the 36-layer decoder**. Run five reference compositions (`image`, `image-voice`, `images3`, `clip`, `video`) with the same prompts, references and seeds as our DGX Spark benchmark. Each composition gets one excluded warmup (seed 42), then measured seeds 0, 1 and 2.
+FastH3 Omni is a reference-conditioned video+audio model that runs through FastVideo's native MLX runtime. Measure end-to-end time per clip for **5 s clips (124 frames) at 480p (832x480) with the 36-layer decoder**. Run five reference compositions (`image`, `image-voice`, `images3`, `clip`, `video`) with the same prompts, references and seeds as our DGX Spark benchmark. **Rough timings only:** one run per composition (seed 0) and no warmup. This is about 20–50 min per clip on an M5 Max, about 2.5–3 h in total.
 
 ## 0. Prerequisites (about 5 min)
 - Plug in power. Close other GPU-heavy apps. Keep the Mac awake for the whole run: prefix long commands with `caffeinate -dims`.
@@ -66,28 +66,32 @@ caffeinate -dims $PY run_matrix.py --smoke --model-dir ~/fasth3-mlx/model --data
 It must produce an MP4 with an audio stream. If it does not, stop and report.
 
 ## 5. The benchmark (the main deliverable)
+Before starting, update to the fixed commit. The weights and the model directory do not change.
+```bash
+cd ~/fasth3-mlx/FastVideo && git fetch && git checkout ef477d68d
+```
+Then run all five rows, seed 0 only, with no warmup. `images3` goes last because it is the slowest:
 ```bash
 caffeinate -dims $PY run_matrix.py \
   --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
   --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY \
   --seconds 5 --canvases 480p --decoders 36l \
-  --compositions image,image-voice,images3,clip,video \
+  --compositions image,image-voice,clip,video,images3 --seeds 0 --no-warmup \
   --out ~/fasth3-mlx/results-480p-36l
 ```
-- This is 5 cells, each with 1 warmup and 3 measured seeds, run strictly one at a time. Do not run anything else on the GPU meanwhile.
-- It is resumable: if it is interrupted, re-run the same command and finished cells are skipped. A failed cell (for example out of memory) is recorded and the matrix continues; do not retry it unless asked.
-- If time is short, add `--seeds 0` and say so in the report.
+- If an earlier run already wrote results to the same `--out`, the finished cells and seeds are skipped automatically.
+- Runs go strictly one at a time. Do not run anything else on the GPU meanwhile.
+- A failed cell (for example out of memory) is recorded and the run continues. Do not retry it.
+- Health check after the first clip: every record in `per-request-records.jsonl` should show `"impl": "simd"` under `vsa`.
 
-## 6. Optional, only after step 5 finishes: fastest-configuration sweep
-This builds INT8 (and BF16 if the Mac has 64 GB or more) and ranks GEMM paths on one cell:
+## 6. Optional (about 10 min): smaller-reference-canvas run
+This is a single extra `image` clip with the opt-in, lossy reference cap, to measure the biggest speed lever:
 ```bash
-cd ~/fasth3-mlx/kit/bench
-caffeinate -dims bash prepare_model_dir.sh --fastvideo ~/fasth3-mlx/FastVideo --python $PY \
-  --model-dir ~/fasth3-mlx/model --sources ~/fasth3-mlx/sources --formats "int8 bf16"
-caffeinate -dims $PY run_matrix.py --m5-sweep --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
-  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --out ~/fasth3-mlx/results-m5-sweep
+caffeinate -dims $PY run_matrix.py --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
+  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --seconds 5 --canvases 480p --decoders 36l \
+  --compositions image --seeds 0 --no-warmup --reference-max-pixels match --out ~/fasth3-mlx/results-480p-36l-match
 ```
-The INT8/BF16 conversion re-downloads the BF16 release if the sources were deleted. That is expected.
+Skip 768p and the `--m5-sweep`.
 
 ## 7. Post the results (needs the owner's GitHub and Hugging Face logins on this Mac)
 The human operator should run `gh auth login` and `hf auth login` beforehand. These are interactive, and you must never handle passwords or tokens yourself. If either is missing, skip that part, and at the end tell the operator to send `SEND_BACK.tar.gz` and your report manually (AirDrop or Drive).
@@ -97,7 +101,7 @@ The human operator should run `gh auth login` and `hf auth login` beforehand. Th
 HFUSER=$(hf auth whoami | head -1 | awk -F= '{print $NF}')   # check the printed username
 hf repo create $HFUSER/fasth3-omni-mlx-m5-results --repo-type dataset
 hf upload $HFUSER/fasth3-omni-mlx-m5-results ~/fasth3-mlx/results-480p-36l results-480p-36l --repo-type dataset
-# also upload results-smoke/ and results-m5-sweep/ the same way if they exist
+# also upload results-smoke/ and results-480p-36l-match/ the same way if they exist
 ```
 Make the dataset public, or add the requester as a collaborator if the operator prefers it private.
 
@@ -118,7 +122,7 @@ gh pr comment 2000 --repo hao-ai-lab/FastVideo --body-file ~/fasth3-mlx/m5_repor
 Write `m5_report.md` with exactly these parts:
 1. **Hardware:** chip, GPU cores, unified memory (`sysctl -n machdep.cpu.brand_string hw.memsize`), macOS (`sw_vers`), mlx version and `mx.device_info()`.
 2. **Code:** commit `ef477d68d`, plus any fixes (link the fix PR or patch).
-3. **Results table:** rows are the compositions; columns are mean ± SD end-to-end seconds over seeds 0/1/2, then mean denoise s, decode s, conditioning s, and peak memory GiB. Use the generated `results-480p-36l/README.md` and `per-request-records.jsonl`.
+3. **Results table:** rows are the compositions; columns are end-to-end seconds (seed 0, single run), denoise s, decode s, conditioning s and peak memory GiB. Add a separate line for the `match` run if you did it. Use the generated `results-480p-36l/README.md` and `per-request-records.jsonl`.
 4. **Link** to the HF dataset with the videos.
 5. **Issues:** anything that failed, was skipped (for example `--seeds 0`) or was changed. Also note the `prepare_model_dir.sh` verification summary, including any hash warnings.
-6. The `--m5-sweep` ranking (`results-m5-sweep/m5-sweep.md`), if you ran it.
+6. Note that these are single-seed, no-warmup rough timings.