add mul_mat_id, and compile upstream's kernels as they ship
Marc Sun
9534e0b
`mul_mat_id` is ggml's expert-routed matmul: one dispatch for a whole bank, given the router's
choices, where the alternative is a gemv per expert per layer. On Qwen3.5-35B-A3B that moved the
expert banks from 83% of decode time to 18%.
The rest is the vendoring. ggml's Metal backend was one translation unit, so this package cut its
own subset out of it with a `trim_shader.py` pass; upstream splits it per operation now, so
`vendor.py` copies the files this dispatches and `build.toml` compiles them directly -- no
generated shader, no generator.
Pin moves to llama.cpp 50f068ff.
vendor only the files the build reaches
Marc Sun
091adac
`vendor.py` copied whole trees, which carried ~300 files -- most of them ggml's cuda sources -- that
nothing here compiles. The metal build reaches two of upstream's headers, and `trim_shader.py`
regenerates the trimmed shader from a third; those are what it copies now, so a pin bump cannot
quietly re-add the rest. `vendor/include` leaves build.toml with it, having no files left.
Verified: the trimmed shader regenerates byte-identically, the shader compiles against the pruned
includes, and `nix run .#build-and-copy` produces both torch variants with the tests passing.
stop tracking build/, like the other ggml kernels
Marc Sun
a03d80d
`build/` is already gitignored here, and `ggml-attn` and `ggml-gated-delta-net` never tracked it: the
artifacts a client downloads live on the kernel repo's `v1` branch, which `build-and-upload` writes.
Keeping a second copy in git only creates two things to keep in step, and they had already drifted --
what shipped was stamped 09da59d, what was committed here 1f05408.
sync build/ with what was published to the v1 branch
Marc Sun
a5ee429
The uploader rebuilds from the commit it is run on, so what shipped is stamped 09da59d while the
artifacts committed here were stamped 1f05408 -- the same code, but the tree and the branch clients
download would drift apart from the first look. These are the published bytes, fetched back.
rebuild metal for get_rows, and track artifacts by glob
Marc Sun
e6e7898
The published `.so` predates `get_rows`, so a client that asks for it gets an AttributeError and
falls back to unpacking whole tensors. Rebuilt from a clean tree, so the module carries the commit
it was built from rather than a `_dirty` name.
`.gitattributes` listed every artifact by path, one line per rebuild, and the entries outlived the
files. One glob covers whatever hash the next build stamps in.
drop the cuda kernel: metal is the only backend built for now
Marc Sun
77f577c
The cuda variants were shipped from a build nobody was running, and each rebuild had to keep them
alive to avoid committing stale artifacts. `git log -- gguf_cuda` still has the sources, and
`vendor/` keeps ggml's cuda tree, so re-adding the backend is a `build.toml` section again.
expose ggml's get_rows, and rename the package to match the repo
`get_rows` dequantizes as it gathers, so reading a few rows out of a large table never materializes
the rest -- which is what an embedding lookup wants. `dequantize` becomes the special case: every row
in order. CUDA has no indexed dequantize upstream, so it gathers first and unpacks after, as before.
Documents how a partial rebuild turns untouched variants into LFS
pointers-of-pointers, why fsck and .gitattributes both look fine, the size
check to run before committing, and how to repair from history.
Also corrects section 7: reaching an op through an attribute no longer
graph-breaks when the kernel registers fakes, measured at 0 breaks over 225
calls with fullgraph=True. The old advice would have a consumer wrap ops it
does not need to wrap.
Fail when a shipped artifact is not a loadable library
An .so committed while unsmudged stores its own LFS pointer as the object: a
~130-byte file that fails to dlopen. That happened once and was invisible,
because a kernel that will not load looks exactly like a kernel that was never
built for this platform, so every test skipped and the run was green.
test_artifacts.py checks the bytes on disk and that the build for this machine
imports, and it runs even when no kernel loads. Verified by reproducing the
corruption: two named failures instead of a green skip.
Restore the five artifacts that were committed as LFS pointers
ea12071 rebuilt the torch212/213 CUDA variants. The five it did not rebuild --
both Metal variants and the three torch211 CUDA ones -- were unsmudged pointer
text in that working tree, so `git lfs clean` stored the pointer itself as the
object: a 132/133-byte .so that fails to dlopen with 'slice is not valid mach-o
file'. Restored from the last commit where each pointer described a real binary.
Rebuild CUDA variants with the current builder (provenance, digest, no shim)
The test modules import `gguf_kernels`, which only resolved because the CUDA
variants still carry a `gguf_kernels/` shim from a pre-0.17 kernel-builder. The
Metal variants, built with 0.17, do not have it, so the suite could not be
collected against them at all -- and rebuilding CUDA will remove it there too.
Resolving through `kernels.get_local_kernel` instead exercises the same path a
consumer takes and does not depend on the layout. A platform with no build
reports skips rather than a collection error.
Refresh the Metal builds for the 20-type table
Marc Sun
2c9588b
Built from a clean tree at 0becb0a, so the variants carry that revision rather than a dirty stamp.
104 tests pass against torch212-metal on this machine.
Metal: cover every quant type upstream's metallib has a gemv for
Marc Sun
0becb0a
The type table was four entries; the metallib always held kernels for 22. The gap was host-side:
CUDA reaches upstream's `mul_mat_vec_q_switch_type`, one function that switches over every type,
while ggml's Metal type->pipeline mapping lives in ggml-metal-device.cpp, which pulls in the whole
backend and is not compiled here. So each type has to be registered by hand.
Transcribed from ggml_metal_library_get_pipeline_mul_mv: the legacy quants, the K quants, the IQ
quants and MXFP4 -- 20 types, parity with the CUDA port apart from NVFP4/Q1_0/Q2_0, which the Metal
metallib has no kernels for. The one field not available as a macro is the grid: ggml-metal-ops.cpp
dispatches the dense types and q8_0 on (ne01 + nr0 - 1)/nr0 and everything else divides by nsg too,
so among quantized types q8_0 stays the only `mv_reduce_across_sgs`.
`GEMV_TYPES` becomes a `gemv_types()` op each backend answers for itself. It was one hardcoded CUDA
list, which Metal inherited and over-claimed by 19 types -- a …
Minimal README: ops, devices, source and how to refresh it
torch owns the command buffer from its serial dispatch queue. Asking for the
encoder on the calling thread races with torch committing that buffer, and
encoding into a committed buffer aborts with
failed assertion _status < MTLCommandBufferStatusCommitted
which reproduces reliably during model loading, where many copies are committing
concurrently. Isolated calls never hit it, which is why the op-level tests passed.
Add prebuilt Metal variants for torch 2.12 and 2.13
Verified against the pure-torch reference dequantizer: dequantize is bit-exact
for q8_0/q4_K/q5_K/q6_K and mul_mat_vec agrees to ~1e-6 relative on all four.
An untracked result symlink marks the tree dirty, and kernel-builder then stamps
_dirty into the artifact name and provenance, making the build unreproducible.
q8_0's mul_mv splits the K reduction across simdgroups and combines them through
threadgroup memory, where the K-quants give each simdgroup its own rows. It
therefore needs a grid that is not divided by nsg, and it needs smem bound --
upstream branches on exactly this. Without both, q8_0 returned garbage while the
K-quants were correct.
Metal backend (gguf_metal/) driving llama.cpp's ggml-metal.metal, compiled into
the embedded metallib by kernel-builder. The shader includes ggml-common.h from
vendor/src, so the metal kernel section needs an include path like the cuda one.
vendor.py now copies whole trees (include, ggml-cuda, ggml-metal) rather than a
hand-listed file set. The curated list had drifted from build.toml: it no longer
performed the mmvq.cu -> mmvq-impl.cuh rename that ggml_dispatch.cu depends on,
so re-vendoring left build.toml pointing at a file that was never written.
Vendoring everything means a pin bump cannot leave a dangling include again;
only files named in a src list are compiled.
GGUF kernels: dequantize + fused gemv over packed blocks
