| # ggml-metal-tuning |
|
|
| Offline kernel tuner for the Metal backend. |
| It sweeps a kernel's config grid on the machine it runs on and prints pasteable table rows for `ggml/src/ggml-metal/ggml-metal-tuning.cpp`. |
|
|
| This is not a test: it never reports pass/fail on performance. |
| A non-zero exit code means bad arguments or a wrong environment (no Metal device, missing proc bridges), never a perf result. |
|
|
| | tuner | tunes | table | |
| |---|---|---| |
| | `fa-vec` | flash-attn vec `(Q, NE)` per `(dtype, head size, KV depth, batch width)` | `fa_vec_tuned_table` | |
|
|
| ## Adding a device to the FA-vec table |
|
|
| Build on the target machine: |
|
|
| ```bash |
| cmake -B build -DGGML_METAL=ON |
| cmake --build build --target ggml-metal-tuning -j |
| cmake --build build --target test-backend-ops -j |
| ``` |
|
|
| Sweep the grid (6 dtypes x 10 head sizes x 4 KV depths x 9 batch widths; a few hours): |
|
|
| ```bash |
| ./build/bin/ggml-metal-tuning fa-vec > fa_vec_rows.txt 2> fa_vec_sweep.log |
| ``` |
|
|
| `fa_vec_rows.txt` holds nothing but table rows, ready to paste into `fa_vec_tuned_table`: the min-max-regret target, the aggregate benefit gate, the short-KV drop and the pointwise compression are already applied. |
| A config represents a bucket only if it is no slower than the baseline config at every point that bucket covers, so a config that wins on average but loses at one batch width leaves its bucket at baseline. |
| `fa_vec_sweep.log` holds the per-cell timings, bucket coverage, noise floor, any cooldown activity, and every config the no-harm rule refused together with the point that refused it. |
| Post both: the log is what makes the rows reviewable. |
|
|
| Long sweeps can be split. |
| `--dtype f16,q4_0` and `--dk 128,192` restrict the grid, and the emitted rows for one `(dtype, head size)` do not depend on the others. |
| Concatenating the shard outputs in the order the full grid would visit them gives the same rows a single run prints. |
|
|
| Then validate the numerics, where Metal is compared against the CPU reference: |
|
|
| ```bash |
| ./build/bin/test-backend-ops test -o FLASH_ATTN_EXT -b MTL0 |
| ``` |
|
|
| This forces every legal `(Q, NE)` on `dk=128` and `dk=576`. |
| The tuner itself does no numerical checks, so the other head sizes have no automated numerical coverage. |
|
|
| If the device is not in `enum ggml_metal_device_id` yet, register it in `ggml/src/ggml-metal/ggml-metal-device.{h,m}` first. |
| The tuner emits whatever token the runtime reports for the machine, so an unregistered device emits `GGML_METAL_DEVICE_GENERIC` and its rows would apply to every unknown device. |
|
|
| ## Thermal throttling |
|
|
| Long sweeps heat the GPU, and a throttled measurement is indistinguishable from a slow kernel. |
| The tuner re-measures a fixed baseline config every four candidates as an anchor. |
| When the anchor drifts more than `--cool-drift` (10% by default) from the coolest anchor seen in that cell, the tuner: |
|
|
| 1. discards every candidate measured since the last clean anchor, |
| 2. sleeps with exponential backoff until the anchor comes back within `--cool-eps` (3%), |
| 3. re-measures the discarded candidates. |
|
|
| If it cannot cool down within `--cool-max-wait` seconds, or a cell needs more than `--cool-max-retry` rounds, that cell is dropped from the table and reported on stderr. |
|
|
| `--no-cooldown` only warns on drift and keeps the measurement. |
| Use it to reproduce a sweep taken without cooling. |
|
|
