FastH3 Omni (Ref2VA) MLX benchmark matrix

Runs the FastH3 OmniRef pipeline on Apple Silicon over the same workload matrix as the DGX Spark benchmark, and writes results in the Spark schema so the two can be compared. You only install and run it; nothing needs editing.

Files here: run_matrix.py (the harness), aggregate.py (summary writer, imported by the harness, also runnable alone), this README.

What it measures

Compositions (all 5 s, 124 frames, 24 fps): image, image-voice, images3, clip, video (also accepted: mixed, clips3, images9).
Decoders: 36l (quality 36-layer) and 8l (fast 8-layer).
Canvases: 480p = 832x480, 768p = 1344x768.
Per cell: one excluded warmup at seed 42, then measured seeds 0, 1, 2, one request at a time. Latency is wall time of pipeline.generate(...) through the finished MP4, with model load excluded. Mean and sample SD over seeds.
Default order: all 480p cells first, 36L before 8L, compositions in the order above (20 cells). Each cell runs in a fresh subprocess so memory never accumulates between cells.

Naming note: columns such as nvfp4_36l_480p_mean_s keep the Spark spelling. On a Mac the NVFP4 VAE decoders run as W4A16 (4-bit packed weights, bf16 activations) and the DiT is the INT6 conversion, not native FP4 hardware math.

Requirements

Apple Silicon Mac, macOS, Python 3.10+ environment that can import mlx.core and import fastvideo (FastVideo branch aryan/omniref-mlx-speed). If FastVideo is a checkout that is not pip installed, pass --fastvideo-root /path/to/FastVideo (or set FASTVIDEO_ROOT).
ffmpeg and ffprobe on PATH (brew install ffmpeg).

The model directory with this layout (this is the layout of the Hugging Face repo):

<model-dir>/root/                      composed model root (root/text_encoder -> ../text_encoder-nvfp4)
<model-dir>/dit/int6/                  mlx_h3_dit.json + mlx_h3_dit.safetensors
<model-dir>/text_encoder-nvfp4/
<model-dir>/vae/fasth3-quality-36L-decoder-nvfp4.safetensors
<model-dir>/vae/fasth3-fast-8L-decoder-nvfp4.safetensors
<model-dir>/vae/lightvae-lite-h3-encoder.safetensors

If bench/ sits inside that repo, --model-dir defaults to the repo root. If the root/text_encoder symlink is missing, the harness automatically uses <model-dir>/text_encoder-nvfp4.

The data bundle omniref-handoff/ (workload manifests, reference media, SHA256SUMS). Pass --data-root or set OMNIREF_DATA (default ~/omniref-handoff). Every file is verified against SHA256SUMS at start.

The harness itself is pure standard library; run it with the Python that has mlx and fastvideo installed (that interpreter is also used for the cell subprocesses; override with --python).

Commands

Check everything without a GPU (manifests parse, CSV schema, paths):

python3 run_matrix.py --self-test --data-root ~/omniref-handoff
python3 run_matrix.py --dry-run --model-dir /path/to/model --data-root ~/omniref-handoff

Smoke test (a few minutes): one image request, seed 42, 36L, 448x256, with reference canvases capped to the smoke canvas (--reference-max-pixels match; pass your own value to override).
```
python3 run_matrix.py --smoke --model-dir /path/to/model --data-root ~/omniref-handoff
```
Success: the last lines say COMPLETE, a SEND_BACK.tar.gz is written, and results-<host>-<date>-smoke/media/*/*-smoke.mp4 plays with audio. Other smoke compositions: --smoke --compositions image-voice,clip.

Full matrix (long; safe to interrupt and re-run):

python3 run_matrix.py --model-dir /path/to/model --data-root ~/omniref-handoff --out ./results-mystudio

Subsets:

python3 run_matrix.py ... --canvases 480p --decoders 36l              # one lane
python3 run_matrix.py ... --compositions image,clip --seeds 0         # fewer cells / seeds
python3 run_matrix.py ... --no-warmup                                 # skip the seed-42 warmup
python3 run_matrix.py ... --dit int8                                  # other DiT conversion under dit/
python3 run_matrix.py ... --reference-max-pixels match                # cap reference canvases (default: trained canvas)

When done, send back SEND_BACK.tar.gz from the --out directory.

Resuming, failures and Ctrl-C

Re-run the exact same command: finished seeds and cells are skipped; an interrupted cell resumes with a fresh process (it re-runs the warmup first so the measured seeds still start warm).
A cell that crashes or runs out of memory is recorded as failed (error tail and signal in per-request-records.jsonl, full output in logs/<cell>.log) and the matrix continues. Failed cells are not retried on re-run unless you pass --retry-failed.
Ctrl-C (or SIGTERM) stops the running cell, releases the GPU lock, rebuilds the summary and bundle, and exits with code 130.
A cell is killed after --cell-timeout-s (default 4 h).

GPU lock

Before any model work the harness takes /tmp/fh3mlx/gpu.lock (atomic mkdir, retry every 30 s, owner PID written to owner) and removes it when done, including on errors and Ctrl-C. If a crash left a lock whose owner PID is dead, pass --steal-stale-lock or rm -rf /tmp/fh3mlx/gpu.lock.

Memory settings

--vae-dtype bf16; --metal-wired-limit-gib auto (0.66 x RAM, capped at 64 GiB; off disables); --mlx-cache-limit-gib 8 (applied through the pipeline's mx.set_cache_limit; negative leaves it unset). A background sampler records MLX active/cache/peak memory and sysctl vm.swapusage every 2 s per request (media/<cell>/*.mem.jsonl, summary in each record). Close other heavy apps for clean numbers.

Output directory

per-request-records.jsonl   every warmup/measured request: wall_s, pipeline timings, peak_memory_gib,
                            memory-sampler summary, ffprobe of the MP4, output/prompt/condition/weight hashes
latency.csv                 Spark column schema (nvfp4_{36l,8l}_{480p,768p}_{mean_s,sample_std_s,status});
                            custom canvases are appended as extra columns
benchmark.json              profiles + performance (Spark benchmark.json layout)
environment_receipt.json    chip, memory, macOS, MLX/Python versions, mx.device_info(), FastVideo git sha,
                            weight-manifest hashes (first 64 MB + size), data-bundle identity, settings
README.md                   generated results table (mean ± sample SD) and peak memory table
media/  logs/  status/      MP4s + memory traces; one log per cell; per-cell state
SEND_BACK.tar.gz            everything above (MP4s trimmed to one seed-0 per cell if the total exceeds ~2 GB)

Rebuild the summary files at any time with python3 aggregate.py --out <results-dir>.

M5 fastest-config sweep

--m5-sweep runs the image composition at 480p with the 36L decoder, seed 0 only (plus the excluded warmup), for DiT int6 and int8 x --quant-gemm {dequant,qmm}, and for bf16 when <model-dir>/dit/bf16 exists (bf16 ignores the GEMM path). Cell ids carry -dit-<fmt>[-gemm-<mode>]; results are ranked in m5-sweep.md / m5-sweep.csv. Resume and the GPU lock behave as in the full matrix. --dit bf16 and --quant-gemm also work on ordinary runs.