Hugging Face's logo

VladHong
/
llama-cpp-K2-FAST2

GGUF
llama.cpp
k2-horizon
sliding-window-attention
long-context
fast-inference
Model card Files
xet
 Community
llama-cpp-K2-FAST2 / FAST2-FORK.md
VladHong's picture
VladHong
llama.cpp k2-fast2: MBZUAI model/K2Horizon + FAST2 sliding-window override
c59fd93 verified
preview code
|
Raw History Blame Contribute Delete
979 Bytes

llama.cpp — K2-FAST2 branch

This repository publishes branch k2-fast2: MBZUAI-IFM's llama.cpp model/K2Horizon support (base commit 42adf01) plus the FAST2 sliding-window attention override for k2-horizon GGUFs (51 lines in src/models/k2-horizon.cpp).

  • Activates k2-horizon.attention.sliding_window GGUF metadata (e.g. the FAST2-SWA8K model) or --override-kv k2-horizon.attention.sliding_window=int:W
  • Measured with K2-Horizon-MoVA-36B-A4B APEX-Mini @61K ctx on RTX 2080 Ti: decode 16.5 -> 50.7 tok/s (3.1x), prefill 692 -> 1482 tok/s (2.1x), ppl inside window identical to full attention
  • Stock behavior is unchanged when no window key is present

Build:

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_CURL=OFF \
      -DCMAKE_CUDA_ARCHITECTURES=<your-sm> -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

The same change as a standalone patch: k2-fast2-swa.patch in the FAST2 model repo.