Buckets:

gijl
/

lv-storage

Files

xet

gijl/lv-storage / llama.cpp /tools /tuning /README.md

gijl

15 days ago

preview code

download

raw

3.89 kB

# ggml-metal-tuning

Offline kernel tuner for the Metal backend.
It sweeps a kernel's config grid on the machine it runs on and prints pasteable table rows for `ggml/src/ggml-metal/ggml-metal-tuning.cpp`.

This is not a test: it never reports pass/fail on performance.
A non-zero exit code means bad arguments or a wrong environment (no Metal device, missing proc bridges), never a perf result.

| tuner | tunes | table |
|---|---|---|
| `fa-vec` | flash-attn vec `(Q, NE)` per `(dtype, head size, KV depth, batch width)` | `fa_vec_tuned_table` |

## Adding a device to the FA-vec table

Build on the target machine:

```bash
cmake -B build -DGGML_METAL=ON
cmake --build build --target ggml-metal-tuning -j
cmake --build build --target test-backend-ops -j
```

Sweep the grid (6 dtypes x 10 head sizes x 4 KV depths x 9 batch widths; a few hours):

```bash
./build/bin/ggml-metal-tuning fa-vec > fa_vec_rows.txt 2> fa_vec_sweep.log
```

`fa_vec_rows.txt` holds nothing but table rows: the min-max-regret target, the aggregate benefit gate, the short-KV drop and the pointwise compression are already applied.
The rows carry the SKU token the runtime reported, but `fa_vec_tuned_table` is keyed by Apple GPU family, so that column has to be retagged before the rows compile.
Your family number is on the `MTLGPUFamilyApple<N>` line the backend logs at init, near the top of `fa_vec_sweep.log`; `N` is the value, and the `MTLGPUFamilyCommon`/`MTLGPUFamilyMetal` lines beside it are not it.
If your log is the only sweep for that family, its rows become the family's segment; where the family already has rows, post the log and let the two be compared before anything is replaced.
A config represents a bucket only if it is no slower than the baseline config at every point that bucket covers, so a config that wins on average but loses at one batch width leaves its bucket at baseline.
`fa_vec_sweep.log` holds the per-cell timings, bucket coverage, noise floor, any cooldown activity, and every config the no-harm rule refused together with the point that refused it.
Post both, always: the rows now speak for every device in the family, so the log is what makes them reviewable.

Long sweeps can be split.
`--dtype f16,q4_0` and `--dk 128,192` restrict the grid, and the emitted rows for one `(dtype, head size)` do not depend on the others.
Concatenating the shard outputs in the order the full grid would visit them gives the same rows a single run prints.

Then validate the numerics, where Metal is compared against the CPU reference:

```bash
./build/bin/test-backend-ops test -o FLASH_ATTN_EXT -b MTL0
```

This forces every legal `(Q, NE)` on `dk=128` and `dk=576`.
The tuner itself does no numerical checks, so the other head sizes have no automated numerical coverage.

If the device is not in `enum ggml_metal_device_id` yet, register it in `ggml/src/ggml-metal/ggml-metal-device.{h,m}` first.
The tuner emits whatever token the runtime reports for the machine, so an unregistered device emits `GGML_METAL_DEVICE_GENERIC` and its rows would apply to every unknown device.

## Thermal throttling

Long sweeps heat the GPU, and a throttled measurement is indistinguishable from a slow kernel.
The tuner re-measures a fixed baseline config every four candidates as an anchor.
When the anchor drifts more than `--cool-drift` (10% by default) from the coolest anchor seen in that cell, the tuner:

1. discards every candidate measured since the last clean anchor,
2. sleeps with exponential backoff until the anchor comes back within `--cool-eps` (3%),
3. re-measures the discarded candidates.

If it cannot cool down within `--cool-max-wait` seconds, or a cell needs more than `--cool-max-retry` rounds, that cell is dropped from the table and reported on stderr.

`--no-cooldown` only warns on drift and keeps the measurement.
Use it to reproduce a sweep taken without cooling.

Xet Storage Details

Size:: 3.89 kB
Xet hash:: 2d97e05611a9d22585e7e5f5abeaebf96ae5750b37f0fb6fe39436e906b3b494

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# ggml-metal-tuning

	Offline kernel tuner for the Metal backend.
	It sweeps a kernel's config grid on the machine it runs on and prints pasteable table rows for `ggml/src/ggml-metal/ggml-metal-tuning.cpp`.

	This is not a test: it never reports pass/fail on performance.
	A non-zero exit code means bad arguments or a wrong environment (no Metal device, missing proc bridges), never a perf result.

	\| tuner \| tunes \| table \|
	\|---\|---\|---\|
	\| `fa-vec` \| flash-attn vec `(Q, NE)` per `(dtype, head size, KV depth, batch width)` \| `fa_vec_tuned_table` \|

	## Adding a device to the FA-vec table

	Build on the target machine:

	```bash
	cmake -B build -DGGML_METAL=ON
	cmake --build build --target ggml-metal-tuning -j
	cmake --build build --target test-backend-ops -j
	```

	Sweep the grid (6 dtypes x 10 head sizes x 4 KV depths x 9 batch widths; a few hours):

	```bash
	./build/bin/ggml-metal-tuning fa-vec > fa_vec_rows.txt 2> fa_vec_sweep.log
	```

	`fa_vec_rows.txt` holds nothing but table rows: the min-max-regret target, the aggregate benefit gate, the short-KV drop and the pointwise compression are already applied.
	The rows carry the SKU token the runtime reported, but `fa_vec_tuned_table` is keyed by Apple GPU family, so that column has to be retagged before the rows compile.
	Your family number is on the `MTLGPUFamilyApple<N>` line the backend logs at init, near the top of `fa_vec_sweep.log`; `N` is the value, and the `MTLGPUFamilyCommon`/`MTLGPUFamilyMetal` lines beside it are not it.
	If your log is the only sweep for that family, its rows become the family's segment; where the family already has rows, post the log and let the two be compared before anything is replaced.
	A config represents a bucket only if it is no slower than the baseline config at every point that bucket covers, so a config that wins on average but loses at one batch width leaves its bucket at baseline.
	`fa_vec_sweep.log` holds the per-cell timings, bucket coverage, noise floor, any cooldown activity, and every config the no-harm rule refused together with the point that refused it.
	Post both, always: the rows now speak for every device in the family, so the log is what makes them reviewable.

	Long sweeps can be split.
	`--dtype f16,q4_0` and `--dk 128,192` restrict the grid, and the emitted rows for one `(dtype, head size)` do not depend on the others.
	Concatenating the shard outputs in the order the full grid would visit them gives the same rows a single run prints.

	Then validate the numerics, where Metal is compared against the CPU reference:

	```bash
	./build/bin/test-backend-ops test -o FLASH_ATTN_EXT -b MTL0
	```

	This forces every legal `(Q, NE)` on `dk=128` and `dk=576`.
	The tuner itself does no numerical checks, so the other head sizes have no automated numerical coverage.

	If the device is not in `enum ggml_metal_device_id` yet, register it in `ggml/src/ggml-metal/ggml-metal-device.{h,m}` first.
	The tuner emits whatever token the runtime reports for the machine, so an unregistered device emits `GGML_METAL_DEVICE_GENERIC` and its rows would apply to every unknown device.

	## Thermal throttling

	Long sweeps heat the GPU, and a throttled measurement is indistinguishable from a slow kernel.
	The tuner re-measures a fixed baseline config every four candidates as an anchor.
	When the anchor drifts more than `--cool-drift` (10% by default) from the coolest anchor seen in that cell, the tuner:

	1. discards every candidate measured since the last clean anchor,
	2. sleeps with exponential backoff until the anchor comes back within `--cool-eps` (3%),
	3. re-measures the discarded candidates.

	If it cannot cool down within `--cool-max-wait` seconds, or a cell needs more than `--cool-max-retry` rounds, that cell is dropped from the table and reported on stderr.

	`--no-cooldown` only warns on drift and keeps the measurement.
	Use it to reproduce a sweep taken without cooling.