add mul_mat_id, and compile upstream's kernels as they ship
Marc Sun
9534e0b
`mul_mat_id` is ggml's expert-routed matmul: one dispatch for a whole bank, given the router's
choices, where the alternative is a gemv per expert per layer. On Qwen3.5-35B-A3B that moved the
expert banks from 83% of decode time to 18%.
The rest is the vendoring. ggml's Metal backend was one translation unit, so this package cut its
own subset out of it with a `trim_shader.py` pass; upstream splits it per operation now, so
`vendor.py` copies the files this dispatches and `build.toml` compiles them directly -- no
generated shader, no generator.
Pin moves to llama.cpp 50f068ff.
expose ggml's get_rows, and rename the package to match the repo
`get_rows` dequantizes as it gathers, so reading a few rows out of a large table never materializes
the rest -- which is what an embedding lookup wants. `dequantize` becomes the special case: every row
in order. CUDA has no indexed dequantize upstream, so it gathers first and unpacks after, as before.
Metal: cover every quant type upstream's metallib has a gemv for
Marc Sun
0becb0a
The type table was four entries; the metallib always held kernels for 22. The gap was host-side:
CUDA reaches upstream's `mul_mat_vec_q_switch_type`, one function that switches over every type,
while ggml's Metal type->pipeline mapping lives in ggml-metal-device.cpp, which pulls in the whole
backend and is not compiled here. So each type has to be registered by hand.
Transcribed from ggml_metal_library_get_pipeline_mul_mv: the legacy quants, the K quants, the IQ
quants and MXFP4 -- 20 types, parity with the CUDA port apart from NVFP4/Q1_0/Q2_0, which the Metal
metallib has no kernels for. The one field not available as a macro is the grid: ggml-metal-ops.cpp
dispatches the dense types and q8_0 on (ne01 + nr0 - 1)/nr0 and everything else divides by nsg too,
so among quantized types q8_0 stays the only `mv_reduce_across_sgs`.
`GEMV_TYPES` becomes a `gemv_types()` op each backend answers for itself. It was one hardcoded CUDA
list, which Metal inherited and over-claimed by 19 types -- a …
GGUF kernels: dequantize + fused gemv over packed blocks, 9 CUDA variants
