Commit History
-
vendor only the files the build reaches
Marc Sun`vendor.py` copied whole trees, which carried ~300 files -- most of them ggml's cuda sources -- that nothing here compiles. The metal build reaches two of upstream's headers, and `trim_shader.py` regenerates the trimmed shader from a third; those are what it copies now, so a pin bump cannot quietly re-add the rest. `vendor/include` leaves build.toml with it, having no files left. Verified: the trimmed shader regenerates byte-identically, the shader compiles against the pruned includes, and `nix run .#build-and-copy` produces both torch variants with the tests passing.2bc3947 -
refuse head dims the build has no kernel for
Marc Sun`flash_attn_forward` fell through to `scaled_dot_product_attention` for a shape upstream has no template for. What ggml instantiates does not depend on the input, so an unsupported model is unsupported for its whole run: falling back means every forward runs torch's attention under a name that promises ggml's, and the speedup that was measured never happens. Raising at the first forward names the problem, the way `flash_attention_2` does, and the message points at `sdpa`. `mask_implementation` goes with it -- it told `transformers` how to build a mask for a kernel that is now either used or refused, never quietly bypassed.7f7cc1b
-
declare the ggml mask implementation
sdpa returns None for a plain-causal mask because sdpa carries causality in its own is_causal flag. ggml has no such flag -- the mask is the causality -- so this package had to rebuild what transformers had dropped, and getting that wrong was silently non-causal rather than merely slow. The new 'ggml' mask never skips at prefill and builds in f16, the dtype the kernels read, so nothing is reconstructed and nothing is cast. Measured: the reconstruction path fires 0 times at prompt 16 and 1024. _causal_mask stays as a defensive fallback, since the attribute is only honoured by a transformers new enough to look for it, and the failure is silent.640b08c -
metal: port the tiled path, so nothing falls back to sdpa
ggml has two implementations of flash_attn_ext and picks by query count: the vector kernel below 20 queries, the tiled one above. Only the vector one was ported, so every prefill went to torch. Both are ported now. The tiled path is three kernels where the vector path is three-with-a-reduce: pad, then blk (a per-block map of which cache blocks the mask rules out), then the attention, which writes dst directly -- 8 queries per threadgroup is parallelism enough without splitting the cache. It uses simdgroup matrix multiplies where the vector one uses reductions, which is why upstream switches at 20: one query row cannot fill an 8x8 matrix unit. Isolated, the tiled kernel beats torch sdpa 1.94x at n_q=64 and 1.37x from 256 to 2048. End to end it first *lost* 8%, because the causal mask this package rebuilds (transformers drops it, sdpa carrying causality in is_causal instead) was f32 and n_q*n_kv -- a 1024x1024 allocation plus an f16 cast per layer per forward. Building it directly in f…606256e -
rename to ggml-attn
The repo holds ggml's attention op, whichever kernel that op dispatches to internally -- `flash_attn_ext` is one op with several implementations chosen by query count and by hardware, and only the vector dispatch is ported. Naming the repo after one algorithm family would also invite the flash-attention-2 semantics assumption that `mask_implementation = "sdpa"` exists to prevent. The friendly selector name is independent of the repo id -- `flash_attention_3` already maps to `vllm-flash-attn3` -- so a `flash_attention_ggml` registration can still be discoverable without the repo claiming it. The op, the entry point and the predicate keep their ggml names: `flash_attn`, `flash_attn_forward`, `supports_flash_attn`.7921487 -
ggml-flash-attn: ggml's flash attention as a standalone kernels repo
Split out of ggml-attn-kernels so that a repo mapped to a transformers attn_implementation contains only an attention implementation. The gated delta net, l2_norm, swiglu and ssm_conv ops stay behind: they have no library-native extension point yet. Carries a fix the original did not have. ggml_flash_attn_ext takes no is_causal argument -- in ggml the mask is the causality, and a null mask means attend to everything -- while transformers drops the mask whenever sdpa's own is_causal flag would carry it, and that flag never reaches an attention function. So flash_attn_forward received None for a 2..19 token prefill and attended to future positions. It now rebuilds the mask. The failure was silent: on Qwen3.5-4B it moved the last-position logits by 1.44 and left the greedy top-5 unchanged, which is why the original 73 tests passed with the bug present. Six new cases assert both directions -- close to causal and far from bidirectional -- and fail without the fix.8732af0