File size: 3,300 Bytes
c59fd93 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | # ggml-metal-tuning
Offline kernel tuner for the Metal backend.
It sweeps a kernel's config grid on the machine it runs on and prints pasteable table rows for `ggml/src/ggml-metal/ggml-metal-tuning.cpp`.
This is not a test: it never reports pass/fail on performance.
A non-zero exit code means bad arguments or a wrong environment (no Metal device, missing proc bridges), never a perf result.
| tuner | tunes | table |
|---|---|---|
| `fa-vec` | flash-attn vec `(Q, NE)` per `(dtype, head size, KV depth, batch width)` | `fa_vec_tuned_table` |
## Adding a device to the FA-vec table
Build on the target machine:
```bash
cmake -B build -DGGML_METAL=ON
cmake --build build --target ggml-metal-tuning -j
cmake --build build --target test-backend-ops -j
```
Sweep the grid (6 dtypes x 10 head sizes x 4 KV depths x 9 batch widths; a few hours):
```bash
./build/bin/ggml-metal-tuning fa-vec > fa_vec_rows.txt 2> fa_vec_sweep.log
```
`fa_vec_rows.txt` holds nothing but table rows, ready to paste into `fa_vec_tuned_table`: the min-max-regret target, the aggregate benefit gate, the short-KV drop and the pointwise compression are already applied.
A config represents a bucket only if it is no slower than the baseline config at every point that bucket covers, so a config that wins on average but loses at one batch width leaves its bucket at baseline.
`fa_vec_sweep.log` holds the per-cell timings, bucket coverage, noise floor, any cooldown activity, and every config the no-harm rule refused together with the point that refused it.
Post both: the log is what makes the rows reviewable.
Long sweeps can be split.
`--dtype f16,q4_0` and `--dk 128,192` restrict the grid, and the emitted rows for one `(dtype, head size)` do not depend on the others.
Concatenating the shard outputs in the order the full grid would visit them gives the same rows a single run prints.
Then validate the numerics, where Metal is compared against the CPU reference:
```bash
./build/bin/test-backend-ops test -o FLASH_ATTN_EXT -b MTL0
```
This forces every legal `(Q, NE)` on `dk=128` and `dk=576`.
The tuner itself does no numerical checks, so the other head sizes have no automated numerical coverage.
If the device is not in `enum ggml_metal_device_id` yet, register it in `ggml/src/ggml-metal/ggml-metal-device.{h,m}` first.
The tuner emits whatever token the runtime reports for the machine, so an unregistered device emits `GGML_METAL_DEVICE_GENERIC` and its rows would apply to every unknown device.
## Thermal throttling
Long sweeps heat the GPU, and a throttled measurement is indistinguishable from a slow kernel.
The tuner re-measures a fixed baseline config every four candidates as an anchor.
When the anchor drifts more than `--cool-drift` (10% by default) from the coolest anchor seen in that cell, the tuner:
1. discards every candidate measured since the last clean anchor,
2. sleeps with exponential backoff until the anchor comes back within `--cool-eps` (3%),
3. re-measures the discarded candidates.
If it cannot cool down within `--cool-max-wait` seconds, or a cell needs more than `--cool-max-retry` rounds, that cell is dropped from the table and reported on stderr.
`--no-cooldown` only warns on drift and keeps the measurement.
Use it to reproduce a sweep taken without cooling.
|