Handoff: single seed, no warmup, 480p only

e61e73c verified 7 days ago

8.31 kB

# Handoff: benchmark FastH3 Omni (MLX) on an Apple M5 Mac Studio

You are an agent on an Apple M5 Mac Studio. Your job is to **install, build the model, run one fixed benchmark, and post the results, videos included**.

**Rules for code changes:**
- You MAY fix small, clear bugs that block the run. Examples: a crash, a wrong path, a flag typo, an install or dependency problem, an mlx API difference on newer mlx.
- Keep each fix minimal and commit it on a branch. Step 7 covers how. Describe every fix in your report.
- You must NOT change what is measured. That means no performance tuning, no changes to prompts, seeds, resolution, frame count, steps or decoder, and no lossy shortcuts.
- If a fix would need more than a few lines, or touches model math, stop, record the error and logs, and report instead.

## What is being measured
FastH3 Omni is a reference-conditioned video+audio model that runs through FastVideo's native MLX runtime. Measure end-to-end time per clip for **5 s clips (124 frames) at 480p (832x480) with the 36-layer decoder**. Run five reference compositions (`image`, `image-voice`, `images3`, `clip`, `video`) with the same prompts, references and seeds as our DGX Spark benchmark. **Rough timings only:** one run per composition (seed 0) and no warmup. This is about 20–50 min per clip on an M5 Max, about 2.5–3 h in total.

## 0. Prerequisites (about 5 min)
- Plug in power. Close other GPU-heavy apps. Keep the Mac awake for the whole run: prefix long commands with `caffeinate -dims`.
- Free disk: about 130 GB (about 95 GB of downloaded sources, which are deleted after the build, plus about 35 GB of model).
- Install tools:
  ```bash
  brew install ffmpeg uv git
  uv tool install "huggingface_hub[cli]"   # provides `hf`; optional: `hf auth login`
  ```
- Copy the input bundle you received, `omniref-handoff-inputs.tar.gz` (24.6 MB), to the home folder, then:
  ```bash
  cd ~ && shasum -a 256 omniref-handoff-inputs.tar.gz
  # expect 2372442d7350d2c721b9b6ded7546199cf90558c4e135d1f6da9bcc27933b19b
  tar -xzf omniref-handoff-inputs.tar.gz      # creates ~/omniref-handoff
  ```

## 1. Code (pinned commit)
```bash
mkdir -p ~/fasth3-mlx && cd ~/fasth3-mlx
git clone https://github.com/aryan5v/FastVideo.git && cd FastVideo
git checkout ef477d68d        # branch aryan/fasth3-omni-mlx (hao-ai-lab/FastVideo#2000)
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install -e ".[mlx]"
python -c "import mlx.core as mx; print(mx.__version__, mx.device_info())"   # record this output
```
mlx must be **0.32.2 or newer**. If it is older, run `uv pip install -U mlx` and record the version.

## 2. Benchmark kit
```bash
hf download aryan5v/FastH3-Omni-MLX --local-dir ~/fasth3-mlx/kit
```

## 3. Build the model directory (the download dominates; conversion is about 5 min)
```bash
cd ~/fasth3-mlx/kit/bench
caffeinate -dims bash prepare_model_dir.sh \
  --fastvideo ~/fasth3-mlx/FastVideo \
  --python ~/fasth3-mlx/FastVideo/.venv/bin/python \
  --model-dir ~/fasth3-mlx/model --sources ~/fasth3-mlx/sources \
  --formats "int6"
```
- The script downloads the public weights at pinned revisions, converts the INT6 transformer and the text encoder, and verifies SHA-256.
- It is resumable: if it is interrupted, re-run the same command.
- **Required:** it must end with the verification passing. Hash warnings on `dit/*` or the encoder `model.safetensors` are acceptable, but record them. Any other failure means stop and report.

## 4. Smoke test (about 3–5 min)
```bash
cd ~/fasth3-mlx/kit/bench
PY=~/fasth3-mlx/FastVideo/.venv/bin/python
$PY run_matrix.py --self-test --data-root ~/omniref-handoff
caffeinate -dims $PY run_matrix.py --smoke --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --out ~/fasth3-mlx/results-smoke
```
It must produce an MP4 with an audio stream. If it does not, stop and report.

## 5. The benchmark (the main deliverable)
Before starting, update to the fixed commit. The weights and the model directory do not change.
```bash
cd ~/fasth3-mlx/FastVideo && git fetch && git checkout ef477d68d
```
Then run all five rows, seed 0 only, with no warmup. `images3` goes last because it is the slowest:
```bash
caffeinate -dims $PY run_matrix.py \
  --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY \
  --seconds 5 --canvases 480p --decoders 36l \
  --compositions image,image-voice,clip,video,images3 --seeds 0 --no-warmup \
  --out ~/fasth3-mlx/results-480p-36l
```
- If an earlier run already wrote results to the same `--out`, the finished cells and seeds are skipped automatically.
- Runs go strictly one at a time. Do not run anything else on the GPU meanwhile.
- A failed cell (for example out of memory) is recorded and the run continues. Do not retry it.
- Health check after the first clip: every record in `per-request-records.jsonl` should show `"impl": "simd"` under `vsa`.

## 6. Optional (about 10 min): smaller-reference-canvas run
This is a single extra `image` clip with the opt-in, lossy reference cap, to measure the biggest speed lever:
```bash
caffeinate -dims $PY run_matrix.py --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
  --fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --seconds 5 --canvases 480p --decoders 36l \
  --compositions image --seeds 0 --no-warmup --reference-max-pixels match --out ~/fasth3-mlx/results-480p-36l-match
```
Skip 768p and the `--m5-sweep`.

## 7. Post the results (needs the owner's GitHub and Hugging Face logins on this Mac)
The human operator should run `gh auth login` and `hf auth login` beforehand. These are interactive, and you must never handle passwords or tokens yourself. If either is missing, skip that part, and at the end tell the operator to send `SEND_BACK.tar.gz` and your report manually (AirDrop or Drive).

**a. Videos and full results go to a Hugging Face dataset.** This includes every MP4, the per-request records, latency.csv, prompts, logs and the environment receipt.
```bash
HFUSER=$(hf auth whoami | head -1 | awk -F= '{print $NF}')   # check the printed username
hf repo create $HFUSER/fasth3-omni-mlx-m5-results --repo-type dataset
hf upload $HFUSER/fasth3-omni-mlx-m5-results ~/fasth3-mlx/results-480p-36l results-480p-36l --repo-type dataset
# also upload results-smoke/ and results-480p-36l-match/ the same way if they exist
```
Make the dataset public, or add the requester as a collaborator if the operator prefers it private.

**b. Code fixes, if you made any.**
- If GitHub is available: fork `aryan5v/FastVideo`, push your branch, then open a PR **into branch `aryan/fasth3-omni-mlx` of `aryan5v/FastVideo`**, not into hao-ai-lab:
  ```bash
  gh repo fork aryan5v/FastVideo --remote --remote-name mine
  git push mine HEAD:fasth3-omni-mlx-m5-fixes
  gh pr create --repo aryan5v/FastVideo --base aryan/fasth3-omni-mlx --head <you>:fasth3-omni-mlx-m5-fixes
  ```
  Give it the title `[bugfix]: fixes found running FastH3 Omni MLX on M5` and list each fix in the body.
- If GitHub is not available, run `git format-patch ef477d68d -o ~/fasth3-mlx/results-480p-36l/fixes/` before uploading in step 7a.

**c. Results comment on the upstream draft PR** [hao-ai-lab/FastVideo#2000](https://github.com/hao-ai-lab/FastVideo/pull/2000):
```bash
gh pr comment 2000 --repo hao-ai-lab/FastVideo --body-file ~/fasth3-mlx/m5_report.md
```
Write `m5_report.md` with exactly these parts:
1. **Hardware:** chip, GPU cores, unified memory (`sysctl -n machdep.cpu.brand_string hw.memsize`), macOS (`sw_vers`), mlx version and `mx.device_info()`.
2. **Code:** commit `ef477d68d`, plus any fixes (link the fix PR or patch).
3. **Results table:** rows are the compositions; columns are end-to-end seconds (seed 0, single run), denoise s, decode s, conditioning s and peak memory GiB. Add a separate line for the `match` run if you did it. Use the generated `results-480p-36l/README.md` and `per-request-records.jsonl`.
4. **Link** to the HF dataset with the videos.
5. **Issues:** anything that failed, was skipped (for example `--seeds 0`) or was changed. Also note the `prepare_model_dir.sh` verification summary, including any hash warnings.
6. Note that these are single-seed, no-warmup rough timings.

	# Handoff: benchmark FastH3 Omni (MLX) on an Apple M5 Mac Studio

	You are an agent on an Apple M5 Mac Studio. Your job is to install, build the model, run one fixed benchmark, and post the results, videos included.

	Rules for code changes:
	- You MAY fix small, clear bugs that block the run. Examples: a crash, a wrong path, a flag typo, an install or dependency problem, an mlx API difference on newer mlx.
	- Keep each fix minimal and commit it on a branch. Step 7 covers how. Describe every fix in your report.
	- You must NOT change what is measured. That means no performance tuning, no changes to prompts, seeds, resolution, frame count, steps or decoder, and no lossy shortcuts.
	- If a fix would need more than a few lines, or touches model math, stop, record the error and logs, and report instead.

	## What is being measured
	FastH3 Omni is a reference-conditioned video+audio model that runs through FastVideo's native MLX runtime. Measure end-to-end time per clip for 5 s clips (124 frames) at 480p (832x480) with the 36-layer decoder. Run five reference compositions (`image`, `image-voice`, `images3`, `clip`, `video`) with the same prompts, references and seeds as our DGX Spark benchmark. Rough timings only: one run per composition (seed 0) and no warmup. This is about 20–50 min per clip on an M5 Max, about 2.5–3 h in total.

	## 0. Prerequisites (about 5 min)
	- Plug in power. Close other GPU-heavy apps. Keep the Mac awake for the whole run: prefix long commands with `caffeinate -dims`.
	- Free disk: about 130 GB (about 95 GB of downloaded sources, which are deleted after the build, plus about 35 GB of model).
	- Install tools:
	```bash
	brew install ffmpeg uv git
	uv tool install "huggingface_hub[cli]" # provides `hf`; optional: `hf auth login`
	```
	- Copy the input bundle you received, `omniref-handoff-inputs.tar.gz` (24.6 MB), to the home folder, then:
	```bash
	cd ~ && shasum -a 256 omniref-handoff-inputs.tar.gz
	# expect 2372442d7350d2c721b9b6ded7546199cf90558c4e135d1f6da9bcc27933b19b
	tar -xzf omniref-handoff-inputs.tar.gz # creates ~/omniref-handoff
	```

	## 1. Code (pinned commit)
	```bash
	mkdir -p ~/fasth3-mlx && cd ~/fasth3-mlx
	git clone https://github.com/aryan5v/FastVideo.git && cd FastVideo
	git checkout ef477d68d # branch aryan/fasth3-omni-mlx (hao-ai-lab/FastVideo#2000)
	uv venv --python 3.12 --seed && source .venv/bin/activate
	uv pip install -e ".[mlx]"
	python -c "import mlx.core as mx; print(mx.__version__, mx.device_info())" # record this output
	```
	mlx must be 0.32.2 or newer. If it is older, run `uv pip install -U mlx` and record the version.

	## 2. Benchmark kit
	```bash
	hf download aryan5v/FastH3-Omni-MLX --local-dir ~/fasth3-mlx/kit
	```

	## 3. Build the model directory (the download dominates; conversion is about 5 min)
	```bash
	cd ~/fasth3-mlx/kit/bench
	caffeinate -dims bash prepare_model_dir.sh \
	--fastvideo ~/fasth3-mlx/FastVideo \
	--python ~/fasth3-mlx/FastVideo/.venv/bin/python \
	--model-dir ~/fasth3-mlx/model --sources ~/fasth3-mlx/sources \
	--formats "int6"
	```
	- The script downloads the public weights at pinned revisions, converts the INT6 transformer and the text encoder, and verifies SHA-256.
	- It is resumable: if it is interrupted, re-run the same command.
	- Required: it must end with the verification passing. Hash warnings on `dit/*` or the encoder `model.safetensors` are acceptable, but record them. Any other failure means stop and report.

	## 4. Smoke test (about 3–5 min)
	```bash
	cd ~/fasth3-mlx/kit/bench
	PY=~/fasth3-mlx/FastVideo/.venv/bin/python
	$PY run_matrix.py --self-test --data-root ~/omniref-handoff
	caffeinate -dims $PY run_matrix.py --smoke --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
	--fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --out ~/fasth3-mlx/results-smoke
	```
	It must produce an MP4 with an audio stream. If it does not, stop and report.

	## 5. The benchmark (the main deliverable)
	Before starting, update to the fixed commit. The weights and the model directory do not change.
	```bash
	cd ~/fasth3-mlx/FastVideo && git fetch && git checkout ef477d68d
	```
	Then run all five rows, seed 0 only, with no warmup. `images3` goes last because it is the slowest:
	```bash
	caffeinate -dims $PY run_matrix.py \
	--model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
	--fastvideo-root ~/fasth3-mlx/FastVideo --python $PY \
	--seconds 5 --canvases 480p --decoders 36l \
	--compositions image,image-voice,clip,video,images3 --seeds 0 --no-warmup \
	--out ~/fasth3-mlx/results-480p-36l
	```
	- If an earlier run already wrote results to the same `--out`, the finished cells and seeds are skipped automatically.
	- Runs go strictly one at a time. Do not run anything else on the GPU meanwhile.
	- A failed cell (for example out of memory) is recorded and the run continues. Do not retry it.
	- Health check after the first clip: every record in `per-request-records.jsonl` should show `"impl": "simd"` under `vsa`.

	## 6. Optional (about 10 min): smaller-reference-canvas run
	This is a single extra `image` clip with the opt-in, lossy reference cap, to measure the biggest speed lever:
	```bash
	caffeinate -dims $PY run_matrix.py --model-dir ~/fasth3-mlx/model --data-root ~/omniref-handoff \
	--fastvideo-root ~/fasth3-mlx/FastVideo --python $PY --seconds 5 --canvases 480p --decoders 36l \
	--compositions image --seeds 0 --no-warmup --reference-max-pixels match --out ~/fasth3-mlx/results-480p-36l-match
	```
	Skip 768p and the `--m5-sweep`.

	## 7. Post the results (needs the owner's GitHub and Hugging Face logins on this Mac)
	The human operator should run `gh auth login` and `hf auth login` beforehand. These are interactive, and you must never handle passwords or tokens yourself. If either is missing, skip that part, and at the end tell the operator to send `SEND_BACK.tar.gz` and your report manually (AirDrop or Drive).

	a. Videos and full results go to a Hugging Face dataset. This includes every MP4, the per-request records, latency.csv, prompts, logs and the environment receipt.
	```bash
	HFUSER=$(hf auth whoami \| head -1 \| awk -F= '{print $NF}') # check the printed username
	hf repo create $HFUSER/fasth3-omni-mlx-m5-results --repo-type dataset
	hf upload $HFUSER/fasth3-omni-mlx-m5-results ~/fasth3-mlx/results-480p-36l results-480p-36l --repo-type dataset
	# also upload results-smoke/ and results-480p-36l-match/ the same way if they exist
	```
	Make the dataset public, or add the requester as a collaborator if the operator prefers it private.

	b. Code fixes, if you made any.
	- If GitHub is available: fork `aryan5v/FastVideo`, push your branch, then open a PR into branch `aryan/fasth3-omni-mlx` of `aryan5v/FastVideo`, not into hao-ai-lab:
	```bash
	gh repo fork aryan5v/FastVideo --remote --remote-name mine
	git push mine HEAD:fasth3-omni-mlx-m5-fixes
	gh pr create --repo aryan5v/FastVideo --base aryan/fasth3-omni-mlx --head <you>:fasth3-omni-mlx-m5-fixes
	```
	Give it the title `[bugfix]: fixes found running FastH3 Omni MLX on M5` and list each fix in the body.
	- If GitHub is not available, run `git format-patch ef477d68d -o ~/fasth3-mlx/results-480p-36l/fixes/` before uploading in step 7a.

	c. Results comment on the upstream draft PR [hao-ai-lab/FastVideo#2000](https://github.com/hao-ai-lab/FastVideo/pull/2000):
	```bash
	gh pr comment 2000 --repo hao-ai-lab/FastVideo --body-file ~/fasth3-mlx/m5_report.md
	```
	Write `m5_report.md` with exactly these parts:
	1. Hardware: chip, GPU cores, unified memory (`sysctl -n machdep.cpu.brand_string hw.memsize`), macOS (`sw_vers`), mlx version and `mx.device_info()`.
	2. Code: commit `ef477d68d`, plus any fixes (link the fix PR or patch).
	3. Results table: rows are the compositions; columns are end-to-end seconds (seed 0, single run), denoise s, decode s, conditioning s and peak memory GiB. Add a separate line for the `match` run if you did it. Use the generated `results-480p-36l/README.md` and `per-request-records.jsonl`.
	4. Link to the HF dataset with the videos.
	5. Issues: anything that failed, was skipped (for example `--seeds 0`) or was changed. Also note the `prepare_model_dir.sh` verification summary, including any hash warnings.
	6. Note that these are single-seed, no-warmup rough timings.