HANDOFF.md

Handoff: benchmark FastH3 Omni (MLX) on an Apple M5 Mac Studio

You are an agent on an Apple M5 Mac Studio. Your job is to install, build the model, run one fixed benchmark, and post the results, videos included.

Rules for code changes:

• You MAY fix small, clear bugs that block the run. Examples: a crash, a wrong path, a flag typo, an install or dependency problem, an mlx API difference on newer mlx.
• Keep each fix minimal and commit it on a branch. Step 7 covers how. Describe every fix in your report.
• You must NOT change what is measured. That means no performance tuning, no changes to prompts, seeds, resolution, frame count, steps or decoder, and no lossy shortcuts.
• If a fix would need more than a few lines, or touches model math, stop, record the error and logs, and report instead.

What is being measured

FastH3 Omni is a reference-conditioned video+audio model that runs through FastVideo's native MLX runtime. Measure end-to-end time per clip for 5 s clips (124 frames) at 480p (832x480) with the 36-layer decoder. Run five reference compositions (image, image-voice, images3, clip, video) with the same prompts, references and seeds as our DGX Spark benchmark. Rough timings only: one run per composition (seed 0) and no warmup. This is about 20–50 min per clip on an M5 Max, about 2.5–3 h in total.

0. Prerequisites (about 5 min)

• Plug in power. Close other GPU-heavy apps. Keep the Mac awake for the whole run: prefix long commands with caffeinate -dims.
• Free disk: about 130 GB (about 95 GB of downloaded sources, which are deleted after the build, plus about 35 GB of model).
• Install tools:

  brew install ffmpeg uv git
  uv tool install "huggingface_hub[cli]"   # provides `hf`; optional: `hf auth login`
  

• Copy the input bundle you received, omniref-handoff-inputs.tar.gz (24.6 MB), to the home folder, then:

  cd ~ && shasum -a 256 omniref-handoff-inputs.tar.gz
  # expect 2372442d7350d2c721b9b6ded7546199cf90558c4e135d1f6da9bcc27933b19b
  tar -xzf omniref-handoff-inputs.tar.gz      # creates ~/omniref-handoff
  

1. Code (pinned commit)

mkdir -p ~/fasth3-mlx && cd ~/fasth3-mlx
git clone https://github.com/aryan5v/FastVideo.git && cd FastVideo
git checkout ef477d68d        # branch aryan/fasth3-omni-mlx (hao-ai-lab/FastVideo#2000)
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install -e ".[mlx]"
python -c "import mlx.core as mx; print(mx.__version__, mx.device_info())"   # record this output

mlx must be 0.32.2 or newer. If it is older, run uv pip install -U mlx and record the version.

2. Benchmark kit

hf download aryan5v/FastH3-Omni-MLX --local-dir ~/fasth3-mlx/kit

3. Build the model directory (the download dominates; conversion is about 5 min)

cd ~/fasth3-mlx/kit/bench
caffeinate -dims bash prepare_model_dir.sh \
  --fastvideo ~/fasth3-mlx/FastVideo \
  --python ~/fasth3-mlx/FastVideo/.venv/bin/python \
  --model-dir ~/fasth3-mlx/model --sources ~/fasth3-mlx/sources \
  --formats "int6"

• The script downloads the public weights at pinned revisions, converts the INT6 transformer and the text encoder, and verifies SHA-256.
• It is resumable: if it is interrupted, re-run the same command.
• Required: it must end with the verification passing. Hash warnings on dit/* or the encoder model.safetensors are acceptable, but record them. Any other failure means stop and report.

4. Smoke test (about 3–5 min)

cd ~/fasth3-mlx/kit/bench
PY=~/fasth3-mlx/FastVideo/.venv/bin/python
$PY run_matrix.py --self-test --data-root ~/omniref-handoff
caffeinate -dims $PY run_matrix.py --smoke --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --out ~/fasth3-mlx/results-smoke

It must produce an MP4 with an audio stream. If it does not, stop and report.

5. The benchmark (the main deliverable)

Before starting, update to the fixed commit. The weights and the model directory do not change.

cd ~/fasth3-mlx/FastVideo && git fetch && git checkout ef477d68d

Then run all five rows, seed 0 only, with no warmup. images3 goes last because it is the slowest:

caffeinate -dims $PY run_matrix.py \
  --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY \
  --seconds 5 --canvases 480p --decoders 36l \
  --compositions image,image-voice,clip,video,images3 --seeds 0 --no-warmup \
  --out ~/fasth3-mlx/results-480p-36l

• If an earlier run already wrote results to the same --out, the finished cells and seeds are skipped automatically.
• Runs go strictly one at a time. Do not run anything else on the GPU meanwhile.
• A failed cell (for example out of memory) is recorded and the run continues. Do not retry it.
• Health check after the first clip: every record in per-request-records.jsonl should show "impl": "simd" under vsa.

6. Optional (about 10 min): smaller-reference-canvas run

This is a single extra image clip with the opt-in, lossy reference cap, to measure the biggest speed lever:

caffeinate -dims $PY run_matrix.py --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --seconds 5 --canvases 480p --decoders 36l \
  --compositions image --seeds 0 --no-warmup --reference-max-pixels match --out ~/fasth3-mlx/results-480p-36l-match

Skip 768p and the --m5-sweep.

7. Post the results (needs the owner's GitHub and Hugging Face logins on this Mac)

The human operator should run gh auth login and hf auth login beforehand. These are interactive, and you must never handle passwords or tokens yourself. If either is missing, skip that part, and at the end tell the operator to send SEND_BACK.tar.gz and your report manually (AirDrop or Drive).

a. Videos and full results go to a Hugging Face dataset. This includes every MP4, the per-request records, latency.csv, prompts, logs and the environment receipt.

HFUSER=$(hf auth whoami | head -1 | awk -F= '{print $NF}')   # check the printed username
hf repo create $HFUSER/fasth3-omni-mlx-m5-results --repo-type dataset
hf upload $HFUSER/fasth3-omni-mlx-m5-results ~/fasth3-mlx/results-480p-36l results-480p-36l --repo-type dataset
# also upload results-smoke/ and results-480p-36l-match/ the same way if they exist

Make the dataset public, or add the requester as a collaborator if the operator prefers it private.

b. Code fixes, if you made any.

• If GitHub is available: fork aryan5v/FastVideo, push your branch, then open a PR into branch aryan/fasth3-omni-mlx of aryan5v/FastVideo, not into hao-ai-lab:

  gh repo fork aryan5v/FastVideo --remote --remote-name mine
  git push mine HEAD:fasth3-omni-mlx-m5-fixes
  gh pr create --repo aryan5v/FastVideo --base aryan/fasth3-omni-mlx --head <you>:fasth3-omni-mlx-m5-fixes
  

  Give it the title [bugfix]: fixes found running FastH3 Omni MLX on M5 and list each fix in the body.
• If GitHub is not available, run git format-patch ef477d68d -o ~/fasth3-mlx/results-480p-36l/fixes/ before uploading in step 7a.

c. Results comment on the upstream draft PR hao-ai-lab/FastVideo#2000:

gh pr comment 2000 --repo hao-ai-lab/FastVideo --body-file ~/fasth3-mlx/m5_report.md

Write m5_report.md with exactly these parts:

1. Hardware: chip, GPU cores, unified memory (sysctl -n machdep.cpu.brand_string hw.memsize), macOS (sw_vers), mlx version and mx.device_info().
2. Code: commit ef477d68d, plus any fixes (link the fix PR or patch).
3. Results table: rows are the compositions; columns are end-to-end seconds (seed 0, single run), denoise s, decode s, conditioning s and peak memory GiB. Add a separate line for the match run if you did it. Use the generated results-480p-36l/README.md and per-request-records.jsonl.
4. Link to the HF dataset with the videos.
5. Issues: anything that failed, was skipped (for example --seeds 0) or was changed. Also note the prepare_model_dir.sh verification summary, including any hash warnings.
6. Note that these are single-seed, no-warmup rough timings.