Commit History
-
README: the attention implementation binds without naming the function
ce03b52
-
compile upstream's kernels as they ship
Marc Sunggml's Metal backend was one translation unit, so this package cut its own subset out of it with a `trim_shader.py` pass. Upstream splits it per operation now, so `vendor.py` copies the files this dispatches and `build.toml` compiles them directly -- no generated shader, no generator. Pin moves to llama.cpp 50f068ff.062729c -
vendor only the files the build reaches
Marc Sun`vendor.py` copied whole trees, which carried ~300 files -- most of them ggml's cuda sources -- that nothing here compiles. The metal build reaches two of upstream's headers, and `trim_shader.py` regenerates the trimmed shader from a third; those are what it copies now, so a pin bump cannot quietly re-add the rest. `vendor/include` leaves build.toml with it, having no files left. Verified: the trimmed shader regenerates byte-identically, the shader compiles against the pruned includes, and `nix run .#build-and-copy` produces both torch variants with the tests passing.2bc3947 -
drop the cuda scaffolding, and name where this publishes
Marc Sun`ggml_attn_cuda/` was written on a machine with no nvcc and never compiled -- its own README said so, and its `[kernel.*]` section sat commented out beneath 44 lines of build.toml so it could not be built by accident. Carrying an unbuildable backend in a package about to be published is worse than not having one: `git log -- ggml_attn_cuda` has it, and `vendor/` still carries ggml's cuda sources. `[general.hub]` records where `build-and-upload` publishes, so the target is in the file rather than in whoever runs it.92f43ee -
keep instantiations the trim was meant to keep
Marc Sun`_declaration_start` walked back over any line starting with `template`, on the assumption it is the header of the definition being cut. A `template [[host_name(...)]] kernel ... ;` is a whole declaration of its own, so eating it dropped a kernel that was meant to survive the trim -- silently, since the shader still compiles without it and only fails when that kernel is dispatched.fc4deaa -
refuse head dims the build has no kernel for
Marc Sun`flash_attn_forward` fell through to `scaled_dot_product_attention` for a shape upstream has no template for. What ggml instantiates does not depend on the input, so an unsupported model is unsupported for its whole run: falling back means every forward runs torch's attention under a name that promises ggml's, and the speedup that was measured never happens. Raising at the first forward names the problem, the way `flash_attention_2` does, and the message points at `sdpa`. `mask_implementation` goes with it -- it told `transformers` how to build a mask for a kernel that is now either used or refused, never quietly bypassed.7f7cc1b
-
trim
Marc Sunc469057
-
declare the ggml mask implementation
sdpa returns None for a plain-causal mask because sdpa carries causality in its own is_causal flag. ggml has no such flag -- the mask is the causality -- so this package had to rebuild what transformers had dropped, and getting that wrong was silently non-causal rather than merely slow. The new 'ggml' mask never skips at prefill and builds in f16, the dtype the kernels read, so nothing is reconstructed and nothing is cast. Measured: the reconstruction path fires 0 times at prompt 16 and 1024. _causal_mask stays as a defensive fallback, since the attribute is only honoured by a transformers new enough to look for it, and the failure is silent.640b08c -
metal: port the tiled path, so nothing falls back to sdpa
ggml has two implementations of flash_attn_ext and picks by query count: the vector kernel below 20 queries, the tiled one above. Only the vector one was ported, so every prefill went to torch. Both are ported now. The tiled path is three kernels where the vector path is three-with-a-reduce: pad, then blk (a per-block map of which cache blocks the mask rules out), then the attention, which writes dst directly -- 8 queries per threadgroup is parallelism enough without splitting the cache. It uses simdgroup matrix multiplies where the vector one uses reductions, which is why upstream switches at 20: one query row cannot fill an 8x8 matrix unit. Isolated, the tiled kernel beats torch sdpa 1.94x at n_q=64 and 1.37x from 256 to 2048. End to end it first *lost* 8%, because the causal mask this package rebuilds (transformers drops it, sdpa carrying causality in is_causal instead) was f32 and n_q*n_kv -- a 1024x1024 allocation plus an f16 cast per layer per forward. Building it directly in f…606256e -
cuda: scaffolding, gated off
Vendors src/ggml-cuda and adds a CUDA backend that has never been compiled. The build.toml section is commented out and "cuda" is absent from backends, so it cannot be built by accident. Metal and CUDA reach the same op differently, which is why this is not a copy of the metal backend: Metal dispatches shaders directly with no ggml runtime, while CUDA goes through ggml_cuda_flash_attn_ext(ctx, dst) and therefore needs a backend context and ggml_tensors. ggml-cuda.cu, which defines those, is 5441 lines of buffer types, backend registration, a graph executor and CUDA graph capture -- a second allocator and stream regime beside torch's. Flash attention needs three symbols out of that file, so ggml_stubs.cu defines them instead, with the pool backed by torch's caching allocator so kernel scratch shares the model's pool. Seven ggml tensor helpers come along; the set was found by grepping the fattn sources, not by linking. ggml_attn_cuda/README.md lists what is unverified: op_params packi…ab32397 -
rename to ggml-attn
The repo holds ggml's attention op, whichever kernel that op dispatches to internally -- `flash_attn_ext` is one op with several implementations chosen by query count and by hardware, and only the vector dispatch is ported. Naming the repo after one algorithm family would also invite the flash-attention-2 semantics assumption that `mask_implementation = "sdpa"` exists to prevent. The friendly selector name is independent of the repo id -- `flash_attention_3` already maps to `vllm-flash-attn3` -- so a `flash_attention_ggml` registration can still be discoverable without the repo claiming it. The op, the entry point and the predicate keep their ggml names: `flash_attn`, `flash_attn_forward`, `supports_flash_attn`.7921487 -
ggml-flash-attn: ggml's flash attention as a standalone kernels repo
Split out of ggml-attn-kernels so that a repo mapped to a transformers attn_implementation contains only an attention implementation. The gated delta net, l2_norm, swiglu and ssm_conv ops stay behind: they have no library-native extension point yet. Carries a fix the original did not have. ggml_flash_attn_ext takes no is_causal argument -- in ggml the mask is the causality, and a null mask means attend to everything -- while transformers drops the mask whenever sdpa's own is_causal flag would carry it, and that flag never reaches an attention function. So flash_attn_forward received None for a 2..19 token prefill and attended to future positions. It now rebuilds the mask. The failure was silent: on Qwen3.5-4B it moved the last-position logits by 1.44 and left the greedy top-5 unchanged, which is why the original 73 tests passed with the bug present. Six new cases assert both directions -- close to causal and far from bidirectional -- and fail without the fix.8732af0