bench/README.md

# FastH3 Omni (Ref2VA) MLX benchmark matrix

Runs the FastH3 OmniRef pipeline on Apple Silicon over the same workload
matrix as the DGX Spark benchmark, and writes results in the Spark schema so
the two can be compared. You only install and run it; nothing needs editing.

Files here: `run_matrix.py` (the harness), `aggregate.py` (summary writer,
imported by the harness, also runnable alone), this README.

## What it measures

- Compositions (all 5 s, 124 frames, 24 fps): `image, image-voice, images3, clip, video`
  (also accepted: `mixed, clips3, images9`).
- Decoders: `36l` (quality 36-layer) and `8l` (fast 8-layer).
- Canvases: `480p` = 832x480, `768p` = 1344x768.
- Per cell: one excluded warmup at seed 42, then measured seeds 0, 1, 2, one
  request at a time. Latency is wall time of `pipeline.generate(...)` through
  the finished MP4, with model load excluded. Mean and sample SD over seeds.
- Default order: all 480p cells first, 36L before 8L, compositions in the order
  above (20 cells). Each cell runs in a fresh subprocess so memory never
  accumulates between cells.

Naming note: columns such as `nvfp4_36l_480p_mean_s` keep the Spark spelling.
On a Mac the NVFP4 VAE decoders run as W4A16 (4-bit packed weights, bf16
activations) and the DiT is the INT6 conversion, not native FP4 hardware math.

## Requirements

- Apple Silicon Mac, macOS, Python 3.10+ environment that can
  `import mlx.core` and `import fastvideo` (FastVideo branch
  `aryan/omniref-mlx-speed`). If FastVideo is a checkout that is not pip
  installed, pass `--fastvideo-root /path/to/FastVideo` (or set `FASTVIDEO_ROOT`).
- `ffmpeg` and `ffprobe` on PATH (`brew install ffmpeg`).
- The model directory with this layout (this is the layout of the Hugging Face repo):

  ```
  <model-dir>/root/                      composed model root (root/text_encoder -> ../text_encoder-nvfp4)
  <model-dir>/dit/int6/                  mlx_h3_dit.json + mlx_h3_dit.safetensors
  <model-dir>/text_encoder-nvfp4/
  <model-dir>/vae/fasth3-quality-36L-decoder-nvfp4.safetensors
  <model-dir>/vae/fasth3-fast-8L-decoder-nvfp4.safetensors
  <model-dir>/vae/lightvae-lite-h3-encoder.safetensors
  ```

  If `bench/` sits inside that repo, `--model-dir` defaults to the repo root.
  If the `root/text_encoder` symlink is missing, the harness automatically uses
  `<model-dir>/text_encoder-nvfp4`.
- The data bundle `omniref-handoff/` (workload manifests, reference media,
  `SHA256SUMS`). Pass `--data-root` or set `OMNIREF_DATA` (default
  `~/omniref-handoff`). Every file is verified against `SHA256SUMS` at start.

The harness itself is pure standard library; run it with the Python that has
mlx and fastvideo installed (that interpreter is also used for the cell
subprocesses; override with `--python`).

## Commands

1. Check everything without a GPU (manifests parse, CSV schema, paths):

   ```
   python3 run_matrix.py --self-test --data-root ~/omniref-handoff
   python3 run_matrix.py --dry-run --model-dir /path/to/model --data-root ~/omniref-handoff
   ```

2. Smoke test (a few minutes): one `image` request, seed 42, 36L, 448x256, with
   reference canvases capped to the smoke canvas (`--reference-max-pixels match`;
   pass your own value to override).

   ```
   python3 run_matrix.py --smoke --model-dir /path/to/model --data-root ~/omniref-handoff
   ```

   Success: the last lines say `COMPLETE`, a `SEND_BACK.tar.gz` is written, and
   `results-<host>-<date>-smoke/media/*/*-smoke.mp4` plays with audio. Other
   smoke compositions: `--smoke --compositions image-voice,clip`.

3. Full matrix (long; safe to interrupt and re-run):

   ```
   python3 run_matrix.py --model-dir /path/to/model --data-root ~/omniref-handoff --out ./results-mystudio
   ```

4. Subsets:

   ```
   python3 run_matrix.py ... --canvases 480p --decoders 36l              # one lane
   python3 run_matrix.py ... --compositions image,clip --seeds 0         # fewer cells / seeds
   python3 run_matrix.py ... --no-warmup                                 # skip the seed-42 warmup
   python3 run_matrix.py ... --dit int8                                  # other DiT conversion under dit/
   python3 run_matrix.py ... --reference-max-pixels match                # cap reference canvases (default: trained canvas)
   ```

When done, send back `SEND_BACK.tar.gz` from the `--out` directory.

## Resuming, failures and Ctrl-C

- Re-run the exact same command: finished seeds and cells are skipped; an
  interrupted cell resumes with a fresh process (it re-runs the warmup first so
  the measured seeds still start warm).
- A cell that crashes or runs out of memory is recorded as `failed` (error tail
  and signal in `per-request-records.jsonl`, full output in `logs/<cell>.log`)
  and the matrix continues. Failed cells are not retried on re-run unless you
  pass `--retry-failed`.
- Ctrl-C (or SIGTERM) stops the running cell, releases the GPU lock, rebuilds the
  summary and bundle, and exits with code 130.
- A cell is killed after `--cell-timeout-s` (default 4 h).

## GPU lock

Before any model work the harness takes `/tmp/fh3mlx/gpu.lock` (atomic `mkdir`,
retry every 30 s, owner PID written to `owner`) and removes it when done,
including on errors and Ctrl-C. If a crash left a lock whose owner PID is dead,
pass `--steal-stale-lock` or `rm -rf /tmp/fh3mlx/gpu.lock`.

## Memory settings

`--vae-dtype bf16`; `--metal-wired-limit-gib auto` (0.66 x RAM, capped at 64
GiB; `off` disables); `--mlx-cache-limit-gib 8` (applied through the pipeline's
`mx.set_cache_limit`; negative leaves it unset). A background sampler records MLX
active/cache/peak memory and `sysctl vm.swapusage` every 2 s per request
(`media/<cell>/*.mem.jsonl`, summary in each record). Close other heavy apps for
clean numbers.

## Output directory

```
per-request-records.jsonl   every warmup/measured request: wall_s, pipeline timings, peak_memory_gib,
                            memory-sampler summary, ffprobe of the MP4, output/prompt/condition/weight hashes
latency.csv                 Spark column schema (nvfp4_{36l,8l}_{480p,768p}_{mean_s,sample_std_s,status});
                            custom canvases are appended as extra columns
benchmark.json              profiles + performance (Spark benchmark.json layout)
environment_receipt.json    chip, memory, macOS, MLX/Python versions, mx.device_info(), FastVideo git sha,
                            weight-manifest hashes (first 64 MB + size), data-bundle identity, settings
README.md                   generated results table (mean ± sample SD) and peak memory table
media/  logs/  status/      MP4s + memory traces; one log per cell; per-cell state
SEND_BACK.tar.gz            everything above (MP4s trimmed to one seed-0 per cell if the total exceeds ~2 GB)
```

Rebuild the summary files at any time with
`python3 aggregate.py --out <results-dir>`.

## M5 fastest-config sweep

`--m5-sweep` runs the `image` composition at 480p with the 36L decoder, seed 0 only
(plus the excluded warmup), for DiT `int6` and `int8` x `--quant-gemm {dequant,qmm}`, and
for `bf16` when `<model-dir>/dit/bf16` exists (`bf16` ignores the GEMM path). Cell ids carry
`-dit-<fmt>[-gemm-<mode>]`; results are ranked in `m5-sweep.md` / `m5-sweep.csv`. Resume and
the GPU lock behave as in the full matrix. `--dit bf16` and `--quant-gemm` also work on
ordinary runs.