# Handoff: benchmark FastH3 Omni (MLX) on an Apple M5 Mac Studio

You are an agent on an Apple M5 Mac Studio. Your job is to **install, build the model, run one fixed benchmark, and post the results, videos included**.

**Rules for code changes:**
- You MAY fix small, clear bugs that block the run. Examples: a crash, a wrong path, a flag typo, an install or dependency problem, an mlx API difference on newer mlx.
- Keep each fix minimal and commit it on a branch. Step 7 covers how. Describe every fix in your report.
- You must NOT change what is measured. That means no performance tuning, no changes to prompts, seeds, resolution, frame count, steps or decoder, and no lossy shortcuts.
- If a fix would need more than a few lines, or touches model math, stop, record the error and logs, and report instead.

## What is being measured
FastH3 Omni is a reference-conditioned video+audio model that runs through FastVideo's native MLX runtime. Measure end-to-end time per clip for **5 s clips (124 frames) at 480p (832x480) with the 36-layer decoder**. Run five reference compositions (`image`, `image-voice`, `images3`, `clip`, `video`) with the same prompts, references and seeds as our DGX Spark benchmark. **Rough timings only:** one run per composition (seed 0) and no warmup. This is about 20–50 min per clip on an M5 Max, about 2.5–3 h in total.

## 0. Prerequisites (about 5 min)
- Plug in power. Close other GPU-heavy apps. Keep the Mac awake for the whole run: prefix long commands with `caffeinate -dims`.
- Free disk: about 130 GB (about 95 GB of downloaded sources, which are deleted after the build, plus about 35 GB of model).
- Install tools:
  ```bash
  brew install ffmpeg uv git
  uv tool install "huggingface_hub[cli]"   # provides `hf`; optional: `hf auth login`
  ```
- Copy the input bundle you received, `omniref-handoff-inputs.tar.gz` (24.6 MB), to the home folder, then:
  ```bash
  cd ~ && shasum -a 256 omniref-handoff-inputs.tar.gz
  # expect 2372442d7350d2c721b9b6ded7546199cf90558c4e135d1f6da9bcc27933b19b
  tar -xzf omniref-handoff-inputs.tar.gz      # creates ~/omniref-handoff
  ```

## 1. Code (pinned commit)
```bash
mkdir -p ~/fasth3-mlx && cd ~/fasth3-mlx
git clone https://github.com/aryan5v/FastVideo.git && cd FastVideo
git checkout ef477d68d        # branch aryan/fasth3-omni-mlx (hao-ai-lab/FastVideo#2000)
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install -e ".[mlx]"
python -c "import mlx.core as mx; print(mx.__version__, mx.device_info())"   # record this output
```
mlx must be **0.32.2 or newer**. If it is older, run `uv pip install -U mlx` and record the version.

## 2. Benchmark kit
```bash
hf download aryan5v/FastH3-Omni-MLX --local-dir ~/fasth3-mlx/kit
```

## 3. Build the model directory (the download dominates; conversion is about 5 min)
```bash
cd ~/fasth3-mlx/kit/bench
caffeinate -dims bash prepare_model_dir.sh \
  --fastvideo ~/fasth3-mlx/FastVideo \
  --python ~/fasth3-mlx/FastVideo/.venv/bin/python \
  --model-dir ~/fasth3-mlx/model --sources ~/fasth3-mlx/sources \
  --formats "int6"
```
- The script downloads the public weights at pinned revisions, converts the INT6 transformer and the text encoder, and verifies SHA-256.
- It is resumable: if it is interrupted, re-run the same command.
- **Required:** it must end with the verification passing. Hash warnings on `dit/*` or the encoder `model.safetensors` are acceptable, but record them. Any other failure means stop and report.

## 4. Smoke test (about 3–5 min)
```bash
cd ~/fasth3-mlx/kit/bench
PY=~/fasth3-mlx/FastVideo/.venv/bin/python
$PY run_matrix.py --self-test --data-root ~/omniref-handoff
caffeinate -dims $PY run_matrix.py --smoke --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --out ~/fasth3-mlx/results-smoke
```
It must produce an MP4 with an audio stream. If it does not, stop and report.

## 5. The benchmark (the main deliverable)
Before starting, update to the fixed commit. The weights and the model directory do not change.
```bash
cd ~/fasth3-mlx/FastVideo && git fetch && git checkout ef477d68d
```
Then run all five rows, seed 0 only, with no warmup. `images3` goes last because it is the slowest:
```bash
caffeinate -dims $PY run_matrix.py \
  --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY \
  --seconds 5 --canvases 480p --decoders 36l \
  --compositions image,image-voice,clip,video,images3 --seeds 0 --no-warmup \
  --out ~/fasth3-mlx/results-480p-36l
```
- If an earlier run already wrote results to the same `--out`, the finished cells and seeds are skipped automatically.
- Runs go strictly one at a time. Do not run anything else on the GPU meanwhile.
- A failed cell (for example out of memory) is recorded and the run continues. Do not retry it.
- Health check after the first clip: every record in `per-request-records.jsonl` should show `"impl": "simd"` under `vsa`.

## 6. Optional (about 10 min): smaller-reference-canvas run
This is a single extra `image` clip with the opt-in, lossy reference cap, to measure the biggest speed lever:
```bash
caffeinate -dims $PY run_matrix.py --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --seconds 5 --canvases 480p --decoders 36l \
  --compositions image --seeds 0 --no-warmup --reference-max-pixels match --out ~/fasth3-mlx/results-480p-36l-match
```
Skip 768p and the `--m5-sweep`.

## 7. Post the results (needs the owner's GitHub and Hugging Face logins on this Mac)
The human operator should run `gh auth login` and `hf auth login` beforehand. These are interactive, and you must never handle passwords or tokens yourself. If either is missing, skip that part, and at the end tell the operator to send `SEND_BACK.tar.gz` and your report manually (AirDrop or Drive).

**a. Videos and full results go to a Hugging Face dataset.** This includes every MP4, the per-request records, latency.csv, prompts, logs and the environment receipt.
```bash
HFUSER=$(hf auth whoami | head -1 | awk -F= '{print $NF}')   # check the printed username
hf repo create $HFUSER/fasth3-omni-mlx-m5-results --repo-type dataset
hf upload $HFUSER/fasth3-omni-mlx-m5-results ~/fasth3-mlx/results-480p-36l results-480p-36l --repo-type dataset
# also upload results-smoke/ and results-480p-36l-match/ the same way if they exist
```
Make the dataset public, or add the requester as a collaborator if the operator prefers it private.

**b. Code fixes, if you made any.**
- If GitHub is available: fork `aryan5v/FastVideo`, push your branch, then open a PR **into branch `aryan/fasth3-omni-mlx` of `aryan5v/FastVideo`**, not into hao-ai-lab:
  ```bash
  gh repo fork aryan5v/FastVideo --remote --remote-name mine
  git push mine HEAD:fasth3-omni-mlx-m5-fixes
  gh pr create --repo aryan5v/FastVideo --base aryan/fasth3-omni-mlx --head <you>:fasth3-omni-mlx-m5-fixes
  ```
  Give it the title `[bugfix]: fixes found running FastH3 Omni MLX on M5` and list each fix in the body.
- If GitHub is not available, run `git format-patch ef477d68d -o ~/fasth3-mlx/results-480p-36l/fixes/` before uploading in step 7a.

**c. Results comment on the upstream draft PR** [hao-ai-lab/FastVideo#2000](https://github.com/hao-ai-lab/FastVideo/pull/2000):
```bash
gh pr comment 2000 --repo hao-ai-lab/FastVideo --body-file ~/fasth3-mlx/m5_report.md
```
Write `m5_report.md` with exactly these parts:
1. **Hardware:** chip, GPU cores, unified memory (`sysctl -n machdep.cpu.brand_string hw.memsize`), macOS (`sw_vers`), mlx version and `mx.device_info()`.
2. **Code:** commit `ef477d68d`, plus any fixes (link the fix PR or patch).
3. **Results table:** rows are the compositions; columns are end-to-end seconds (seed 0, single run), denoise s, decode s, conditioning s and peak memory GiB. Add a separate line for the `match` run if you did it. Use the generated `results-480p-36l/README.md` and `per-request-records.jsonl`.
4. **Link** to the HF dataset with the videos.
5. **Issues:** anything that failed, was skipped (for example `--seeds 0`) or was changed. Also note the `prepare_model_dir.sh` verification summary, including any hash warnings.
6. Note that these are single-seed, no-warmup rough timings.