Buckets:
| # Multi-head latent attention (MLA) | |
| The original K2-Horizon-MoVA uses grouped-query attention (GQA). For each token and layer, it caches 8 keys and 8 | |
| values. [TransMLA](https://arxiv.org/abs/2502.07864) converts GQA to multi-head latent attention (MLA). The MLA model | |
| caches one latent vector and one position key for each token and layer. The cache is less than half the size. The | |
| cost is some quality and more arithmetic in attention. | |
| ## Key numbers | |
| | | GQA (original) | MLA (rank 768) | | |
| |---|---|---| | |
| | Cache per token, BF16, all layers | 192 KiB | 84 KiB | | |
| | Cache at 200K tokens, BF16 | 39 GB | 17.2 GB | | |
| | KLD against the original BF16, held-out text | 0 (reference) | 0.144 (MLA BF16) | | |
| | Long generations that fell into a loop (oMLX, 48 runs) | 0 | 2 | | |
| Read [The KV cache](kv-cache.md) for the compact cache formats. | |
| ## How it works | |
| ### The conversion | |
| The conversion has two steps. These tools are not in this repository. The | |
| [MLA model card](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-MLA) gives the details. | |
| 1. **TransMLA conversion.** A calibrated conversion with no training. It gave a KLD of 0.154 against the original on | |
| 8 held-out windows (tag `naive-j768`). | |
| 2. **Distillation heal.** Training of only the latent attention projections (142 M parameters) on 49.9 M tokens. It | |
| reduced the KLD on the same windows to 0.091 (branch `heal-pilot`, now `main`). | |
| ### The cache | |
| For layer `l`, with input `x`: | |
| ``` | |
| c = kv_a_x x (+ kv_a_v v, where v comes from the value experts) | |
| k_rope = RoPE(k_rope_proj x) | |
| ``` | |
| `c` is the latent, with rank `r_l`. `k_rope` is one 128-dimension key. The dense layers 0 to 2 fold `v_proj` into | |
| `kv_a_x`. | |
| ### The attention | |
| For query head `a`, with query `q_a`: | |
| ``` | |
| score_a(t) = (q_lat_a q_a) . c_t + RoPE(q_rope_mix_a q_a) . k_rope_t (scale 128^-0.5, as the original) | |
| o_a = v_up_a (sum_t p_t c_t) | |
| ``` | |
| The output then goes through the softplus gate and `o_proj`, as in the original. | |
| There are two ways to compute this: | |
| - **Absorbed.** The query goes into latent space. Attention is then multi-query attention over the latent. This mode | |
| reads the cache once, but it does about 6 times the arithmetic of GQA. | |
| - **Expanded.** Each head computes its keys `q_latᵀ c` and values `v_up c` from the latent. Attention is then normal | |
| multi-head attention. This mode does about a quarter of the absorbed arithmetic, but it must compute keys for all | |
| cached positions. | |
| ### The engines | |
| | Work | Metal | CUDA | | |
| |---|---|---| | |
| | Decode | Absorbed. `k_mla_attn`: flash style, 16 query heads per threadgroup, 64-key tiles, online softmax, split keys | Absorbed. `k_mla_attn_tc`: 16 query heads per block, 16-key tiles double-buffered by `cp.async`, BF16 `mma` | | |
| | Prompts of 256 or more new tokens | Expanded. `k_mla_prefill`, keys expanded 512 positions at a time | Expanded. `k_mla_prefill`, keys and values from tensor-core GEMMs | | |
| | Prompts of fewer than 256 new tokens | Absorbed, so the engine does not expand all cached positions | Absorbed, as on Metal | | |
| On both engines, expansion stops at the last multiple of 256 positions (`mla_expand_min`). Thus a prompt that the cold cache restores in | |
| 256-token blocks gets the same cache, bit for bit. On an M4 Max, 15 new tokens after 12K cached tokens take 0.57 s | |
| absorbed. Expanded, they took 2.8 s. | |
| ### Why MLA decode is slower than GQA decode | |
| At rank 768, the MLA projections are 945 MB in BF16. Decode must read them for each token. This cost is larger than | |
| the time that the smaller cache saves at short contexts. At long contexts, the smaller cache reads fewer bytes, and | |
| the difference becomes smaller. | |
| ## The quality cost | |
| MLA moves the model away from the original. In the R-02 comparison, MLA BF16 had a held-out KLD of 0.144. GQA with P = 4 | |
| seeds had 0.022. Three of 31 held-out windows cause most of this difference. They are ordinary web prose with no | |
| repetition. We do not know the cause yet. Read [Seeds against affine quantization](../results/2026-10-10-seeds-vs-affine.md). | |
| The MLA model also falls into a loop more often in long generations. Set a limit on `max_tokens`, or set a reasoning | |
| budget. | |
| ## Model folder entries | |
| `config.json` has `mla_ranks` (the latent rank of each layer, a multiple of 32, at most 1024) and `mla_rope_dim` | |
| (128). Read [The model folder format](../reference/model-folder.md). | |
| A rank of 0 keeps the original GQA attention in that layer. The folder then keeps `k_proj` (and `v_proj` in the | |
| layers without value experts) for that layer, and the layer has no MLA tensors. This is a hybrid model. Use it when the | |
| latent cannot hold a layer: each GQA layer costs 4 KB of BF16 cache per token, against 1.75 KB for an MLA layer at | |
| rank 768. The MLX loader, the Metal and CUDA engines and the C reference run hybrid models, and the packer and the | |
| seed search accept them. In the engines, a GQA layer keeps a BF16 K and V cache when the MLA layers use FP8 or FP4. | |
| MLA capture (`nslm-mova-mlacapture`) skips the GQA layers. | |
| ## Measurements | |
| - [MLA kernels on Metal and CUDA](../results/2026-10-08-mla-kernels.md) | |
| - [MLA projections in BF16, Q8, Q4 and seeds](../results/2026-10-08-mla-projections.md) | |
| - [Engine KV cache codes](../results/2026-10-08-engine-kv-codes.md) | |
| - [Seeds against affine quantization (R-02)](../results/2026-10-10-seeds-vs-affine.md) | |
| - [A fused MLA prompt kernel in the MLX loader](../results/2026-10-11-mla-fused-prefill.md) | |
| ## See also | |
| - [The KV cache](kv-cache.md) | |
| - [TransMLA](https://arxiv.org/abs/2502.07864), Meng, Yao and Zhang, 2025 | |
Xet Storage Details
- Size:
- 5.54 kB
- Xet hash:
- 90952c4075ddf045b2cb1974d617aa06091e2d5804f117ed976ce9753f768e3f
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.