Blame for README.md

---
license: apache-2.0
base_model:
- MBZUAI-IFM/llama.cpp
tags:
- llama.cpp
- k2-horizon
- sliding-window-attention
- long-context
- fast-inference
---

# llama.cpp — K2-FAST2 fork 🚀

**This fork makes K2-Horizon GGUFs up to 3.1× faster at long context.**
It is MBZUAI-IFM's `llama.cpp` `model/K2Horizon` branch (commit `42adf01`) plus one addition:
**sliding-window attention (SWA) support for `k2-horizon`** — 51 lines in `src/models/k2-horizon.cpp`.

> 📦 Companion model (self-contained, window embedded): **K2-Horizon-MoVA-36B-A4B-FAST2**
> — [HuggingFace](https://huggingface.co/VladHong/K2-Horizon-MoVA-36B-A4B-FAST2) ·
> [ModelScope](https://modelscope.cn/models/Dalvlad/K2-Horizon-MoVA-36B-A4B-FAST2)

## Measured speed increase vs vanilla APEX-Mini

Same binary, same frozen 61,337-token prompt, same context window, 5 interleaved rounds,
RTX 2080 Ti 22 GB, all layers in VRAM. Full data: `results-head2head.json` in the model repo.

| @ 61K context (61,354-token prompt) | decode tok/s | prefill tok/s | speedup |
|---|---|---|---|
| vanilla APEX-Mini (q8_0 KV — the only way vanilla fits 22 GB at 61K; default f16 KV OOMs) | 16.48 | 692 | — |
| **FAST2 (this fork + embedded W=8192 window)** | **50.72** | **1,482** | **decode 3.1×, prefill 2.1×** |
| FAST2 with W=16384 | 43.90 | 1,187 | decode 2.7× |

At 32K context (W=8192): decode 52.64 vs 36.78 tok/s (**1.43×**), prefill 1,051 vs 949.
At short context FAST2 equals vanilla — decode there is weight-bandwidth-bound and untouched.

**Quality:** perplexity inside the window is numerically identical to full attention
(2.1814 vs 2.1814 on a 240K-token stratified corpus); needle-in-haystack at 61K passes for
facts inside the window. Recall is limited to the last W tokens by design — pick W to match
your workload (W=16384 is a one-line rebuild, see the model repo's `make_swa_gguf.py`).

## What the patch does

K2-Horizon runs **full attention in all 48 layers** (~192 KiB of KV per token — 6+ GiB read
per generated token at 61K context), which is why vanilla decode collapses from ~57 tok/s at
4K to ~16 tok/s at 61K. The patch lets a GGUF declare a sliding window
(`k2-horizon.attention.sliding_window`), which:

1. `load_arch_hparams`: reads the key (from GGUF metadata or `--override-kv`), sets
   `n_swa`, `swa_type = LLAMA_SWA_TYPE_STANDARD`, and marks the attention layers SWA;
2. `graph()`: routes attention through the iswa input path (`build_attn_inp_kv_iswa`) so each
   layer's KV is capped at the last W tokens — decode cost flattens instead of growing.

Without a window key, behavior is bit-identical to stock `model/K2Horizon`.

## Using it

```bash
# build (same as the MBZUAI fork, plus FA_ALL_QUANTS for mixed KV quant support)
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_CURL=OFF \
      -DCMAKE_CUDA_ARCHITECTURES=<your-sm> -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

# option A: the FAST2 GGUF (window embedded — no flags)
llama-server -m K2-Horizon-MoVA-36B-A4B-FAST2-SWA8K.gguf -c 65536 -ngl 99 --parallel 1 \
    -fa on -b 2048 -ub 2048 -t 8 --port 8099

# option B: any k2-horizon GGUF + window at load time
llama-server -m K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf -c 65536 -ngl 99 \
    --override-kv k2-horizon.attention.sliding_window=int:8192 ...
```

## Where to get what

| artifact | HuggingFace | ModelScope |
|---|---|---|
| FAST2 GGUF (window embedded) + card + measurements | [VladHong/K2-Horizon-MoVA-36B-A4B-FAST2](https://huggingface.co/VladHong/K2-Horizon-MoVA-36B-A4B-FAST2) | [Dalvlad/K2-Horizon-MoVA-36B-A4B-FAST2](https://modelscope.cn/models/Dalvlad/K2-Horizon-MoVA-36B-A4B-FAST2) |
| this fork as a source snapshot | [VladHong/llama-cpp-K2-FAST2](https://huggingface.co/VladHong/llama-cpp-K2-FAST2) | branch `k2-fast2` of [Dalvlad/llama-cpp-K2-FAST2](https://modelscope.cn/models/Dalvlad/llama-cpp-K2-FAST2) |
| standalone patch (apply to stock MBZUAI fork) | `k2-fast2-swa.patch` in either model repo | same |

The upstream llama.cpp README is preserved as `README-upstream.md`. All credit for the
`model/K2Horizon` support belongs to MBZUAI-IFM; the FAST2 change is the 51-line diff above.