File size: 979 Bytes
c59fd93 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | # llama.cpp — K2-FAST2 branch
This repository publishes branch **`k2-fast2`**: MBZUAI-IFM's llama.cpp
`model/K2Horizon` support (base commit `42adf01`) plus the FAST2 sliding-window
attention override for `k2-horizon` GGUFs (51 lines in `src/models/k2-horizon.cpp`).
- Activates `k2-horizon.attention.sliding_window` GGUF metadata (e.g. the
FAST2-SWA8K model) or `--override-kv k2-horizon.attention.sliding_window=int:W`
- Measured with K2-Horizon-MoVA-36B-A4B APEX-Mini @61K ctx on RTX 2080 Ti:
decode 16.5 -> 50.7 tok/s (3.1x), prefill 692 -> 1482 tok/s (2.1x),
ppl inside window identical to full attention
- Stock behavior is unchanged when no window key is present
Build:
```bash
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_CURL=OFF \
-DCMAKE_CUDA_ARCHITECTURES=<your-sm> -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j
```
The same change as a standalone patch: `k2-fast2-swa.patch` in the FAST2 model repo.
|