VladHong
/

llama-cpp-K2-FAST2

@@ -1,126 +1,82 @@
-# llama.cpp
-![llama](https://raw.githubusercontent.com/ggml-org/llama.brand/refs/heads/master/cover/llama-cpp/cover-llama-cpp-dark.svg)
-<div align="center">
-<b>LLM inference in C/C++</b>
-[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)
-[![Release](https://img.shields.io/github/v/release/ggml-org/llama.cpp?filter=v*&color=brightgreen)](https://github.com/ggml-org/llama.cpp/releases?q=tag:v0)
-[![Nightly](https://img.shields.io/github/v/release/ggml-org/llama.cpp?label=nightly&filter=b*&color=orange)](https://github.com/ggml-org/llama.cpp/releases?q=b)
-[![Server](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/server.yml?label=Server)](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
-[![Docker](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/docker.yml?label=Docker)](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
-[![Winget](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/winget.yml?label=Winget)](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
-[ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
-</div>
-## Quick start
-A few options to get `llama.cpp` installed on your machine:
-- Visit https://llama.app and follow the instructions
-- Run with Docker - see our [Docker documentation](docs/docker.md)
-- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)
-- Build from source by cloning this repository - check out [our build guide](docs/build.md)
-Once installed:
-```sh
-# Download and run a model directly from Hugging Face
-llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
-# Launch OpenAI-compatible API server
-llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
-```
-<table align="center">
-    <tr>
-        <td align="center" width=50%>
-            <img width="1310" height="888" alt="VLM session with `llama cli`" src="https://github.com/user-attachments/assets/88726b48-1713-48aa-a525-95a02e78afc4" />
-            <i>VLM session with <b>llama cli</b></i>
-        </td>
-        <td align="center">
-            <img width="1392" height="958" alt="Built-in web UI against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
-            <i>Built-in web UI against <b>llama serve</b></i>
-        </td>
-    </tr>
-<table>
-## Description
-The main goal of `llama.cpp` is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
-a wide range of hardware - locally and in the cloud.
-- Plain C/C++ implementation without any dependencies
-- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
-- AVX, AVX2, AVX512 and AMX support for x86 architectures
-- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
-- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
-- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
-- Vulkan and SYCL backend support
-- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
-The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-org/ggml) library.
-## Supported backends
-| Backend | Target devices |
-| --- | --- |
-| [BLAS](docs/build.md#blas-build) | All |
-| [BLIS](docs/backend/BLIS.md) | All |
-| [CANN](docs/build.md#cann) | Ascend NPU |
-| [CUDA](docs/build.md#cuda) | Nvidia GPU |
-| [HIP](docs/build.md#hip) | AMD GPU |
-| [Hexagon [In Progress]](docs/backend/snapdragon/README.md) | Snapdragon |
-| [IBM zDNN](docs/backend/zDNN.md) | IBM Z & LinuxONE |
-| [MUSA](docs/build.md#musa) | Moore Threads GPU |
-| [Metal](docs/build.md#metal-build) | Apple Silicon |
-| [OpenCL](docs/backend/OPENCL.md) | Adreno GPU |
-| [OpenVINO [In Progress]](docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs |
-| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All |
-| [SYCL](docs/backend/SYCL.md) | Intel GPU |
-| [VirtGPU](docs/backend/VirtGPU.md) | VirtGPU APIR |
-| [Vulkan](docs/build.md#vulkan) | GPU |
-| [WebGPU](docs/build.md#webgpu) | All |
-| [ZenDNN](docs/build.md#zendnn) | AMD CPU |
-## Documentation
-#### Tools
-- [cli](tools/cli/README.md)
-- [completion](tools/completion/README.md)
-- [server](tools/server/README.md)
-- [GBNF grammars](grammars/README.md)
-#### Development
-- [How to build](docs/build.md)
-- [Running on Docker](docs/docker.md)
-- [Build on Android](docs/android.md)
-- [Multi-GPU usage](docs/multi-gpu.md)
-- [Performance troubleshooting](docs/development/token_generation_performance_tips.md)
-- [GGML tips & tricks](https://github.com/ggml-org/llama.cpp/wiki/GGML-Tips-&-Tricks)
-- [XCFramework](docs/xcframework.md)
-- [Completions](docs/completions.md)
-- [Models](docs/models.md)
-- [Release process](docs/release.md)
-## Contributing
-- Contributors can open PRs
-- Collaborators will be invited based on contributions
-- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the `master` branch
-- Any help with managing issues, PRs and projects is very appreciated!
-- Read the [CONTRIBUTING.md](CONTRIBUTING.md) for more information
-## Acknowledgements
-- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license
-- [nothings/stb](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain
-- [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License
-- [mackron/miniaudio](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain
-- [sheredom/subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain

+---
+license: apache-2.0
+base_model:
+- MBZUAI-IFM/llama.cpp
+tags:
+- llama.cpp
+- k2-horizon
+- sliding-window-attention
+- long-context
+- fast-inference
+---
+# llama.cpp — K2-FAST2 fork 🚀
+**This fork makes K2-Horizon GGUFs up to 3.1× faster at long context.**
+It is MBZUAI-IFM's `llama.cpp` `model/K2Horizon` branch (commit `42adf01`) plus one addition:
+**sliding-window attention (SWA) support for `k2-horizon`** — 51 lines in `src/models/k2-horizon.cpp`.
+> 📦 Companion model (self-contained, window embedded): **K2-Horizon-MoVA-36B-A4B-FAST2**
+> — [HuggingFace](https://huggingface.co/VladHong/K2-Horizon-MoVA-36B-A4B-FAST2) ·
+> [ModelScope](https://modelscope.cn/models/Dalvlad/K2-Horizon-MoVA-36B-A4B-FAST2)
+## Measured speed increase vs vanilla APEX-Mini
+Same binary, same frozen 61,337-token prompt, same context window, 5 interleaved rounds,
+RTX 2080 Ti 22 GB, all layers in VRAM. Full data: `results-head2head.json` in the model repo.
+| @ 61K context (61,354-token prompt) | decode tok/s | prefill tok/s | speedup |
+|---|---|---|---|
+| vanilla APEX-Mini (q8_0 KV — the only way vanilla fits 22 GB at 61K; default f16 KV OOMs) | 16.48 | 692 | — |
+| **FAST2 (this fork + embedded W=8192 window)** | **50.72** | **1,482** | **decode 3.1×, prefill 2.1×** |
+| FAST2 with W=16384 | 43.90 | 1,187 | decode 2.7× |
+At 32K context (W=8192): decode 52.64 vs 36.78 tok/s (**1.43×**), prefill 1,051 vs 949.
+At short context FAST2 equals vanilla — decode there is weight-bandwidth-bound and untouched.
+**Quality:** perplexity inside the window is numerically identical to full attention
+(2.1814 vs 2.1814 on a 240K-token stratified corpus); needle-in-haystack at 61K passes for
+facts inside the window. Recall is limited to the last W tokens by design — pick W to match
+your workload (W=16384 is a one-line rebuild, see the model repo's `make_swa_gguf.py`).
+## What the patch does
+K2-Horizon runs **full attention in all 48 layers** (~192 KiB of KV per token — 6+ GiB read
+per generated token at 61K context), which is why vanilla decode collapses from ~57 tok/s at
+4K to ~16 tok/s at 61K. The patch lets a GGUF declare a sliding window
+(`k2-horizon.attention.sliding_window`), which:
+1. `load_arch_hparams`: reads the key (from GGUF metadata or `--override-kv`), sets
+   `n_swa`, `swa_type = LLAMA_SWA_TYPE_STANDARD`, and marks the attention layers SWA;
+2. `graph()`: routes attention through the iswa input path (`build_attn_inp_kv_iswa`) so each
+   layer's KV is capped at the last W tokens — decode cost flattens instead of growing.
+Without a window key, behavior is bit-identical to stock `model/K2Horizon`.
+## Using it
+```bash
+# build (same as the MBZUAI fork, plus FA_ALL_QUANTS for mixed KV quant support)
+cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_CURL=OFF \
+      -DCMAKE_CUDA_ARCHITECTURES=<your-sm> -DCMAKE_BUILD_TYPE=Release
+cmake --build build --target llama-server -j
+# option A: the FAST2 GGUF (window embedded — no flags)
+llama-server -m K2-Horizon-MoVA-36B-A4B-FAST2-SWA8K.gguf -c 65536 -ngl 99 --parallel 1 \
+    -fa on -b 2048 -ub 2048 -t 8 --port 8099
+# option B: any k2-horizon GGUF + window at load time
+llama-server -m K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf -c 65536 -ngl 99 \
+    --override-kv k2-horizon.attention.sliding_window=int:8192 ...
+```
+## Where to get what
+| artifact | HuggingFace | ModelScope |
+|---|---|---|
+| FAST2 GGUF (window embedded) + card + measurements | [VladHong/K2-Horizon-MoVA-36B-A4B-FAST2](https://huggingface.co/VladHong/K2-Horizon-MoVA-36B-A4B-FAST2) | [Dalvlad/K2-Horizon-MoVA-36B-A4B-FAST2](https://modelscope.cn/models/Dalvlad/K2-Horizon-MoVA-36B-A4B-FAST2) |
+| this fork as a source snapshot | [VladHong/llama-cpp-K2-FAST2](https://huggingface.co/VladHong/llama-cpp-K2-FAST2) | branch `k2-fast2` of [Dalvlad/llama-cpp-K2-FAST2](https://modelscope.cn/models/Dalvlad/llama-cpp-K2-FAST2) |
+| standalone patch (apply to stock MBZUAI fork) | `k2-fast2-swa.patch` in either model repo | same |
+The upstream llama.cpp README is preserved as `README-upstream.md`. All credit for the
+`model/K2Horizon` support belongs to MBZUAI-IFM; the FAST2 change is the 51-line diff above.