txgsync's picture
|
download
raw
8.64 kB

The KV cache

While the model reads a conversation, it keeps a note for each token. The note is what the model looks back at when it writes the next word. The set of notes is the KV cache. At long contexts, the KV cache can need more memory than the model weights.

Key numbers

Model and cache Bytes per token (all 48 layers) 200K tokens 512K tokens
GQA (original), BF16 192 KiB 39 GB ≈103 GB
GQA, engine --kv q8 ≈100 KB 20 GB ≈52 GB
MLA, BF16 84 KiB 17.2 GB 45.1 GB
MLA, MLX loader q8 (default) 45 KB ≈9.0 GB 24.0 GB
MLA, MLX loader rq4 27 KB ≈5.4 GB 14.3 GB
MLA, engine --kv fp8 43 KiB 8.9 GB ≈23 GB
MLA, engine --kv fp4 32 KiB 6.6 GB ≈17 GB

The MLA rows are for the rank-768 conversion. Values marked ≈ are calculated from the bytes per token. The other values are measured.

Why MLA keeps a smaller cache

The original model keeps 8 keys and 8 values of 128 numbers for each token in each layer. The MLA model keeps one latent vector of 768 numbers and one position (RoPE) key of 128 numbers. Read Multi-head latent attention.

The MLX loader: q8 and rq4

The MLX loader (tools/nanoseedlm_k2.py, the trust_remote_code model file in the Hugging Face repositories) keeps the MLA cache in one of two compact forms:

  • q8 (the default) rounds each number to 8 bits, in groups of 64 with their own scale. It is half the size of BF16. We did not measure a quality loss.
  • rq4 keeps the latent in 4 bits and the position key in 8 bits. It is a third of the size of BF16.

Plain 4-bit rounding loses too much. A few numbers in each latent are much larger than the others. Before rq4 rounds a latent, it mixes the latent with a fixed Walsh-Hadamard transform. The signs of the transform come from the same 16-bit LFSR that makes the weights. Thus the transform is a seed, and the file stores nothing for it. The mix spreads the large numbers out, and a small, measurable quality cost remains.

The trade-offs

Measured with K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4 on an M4 Max (128 GB), MLX 0.32.3:

BF16 q8 (default) rq4
cache per token 84 KB 45 KB 27 KB
cache at 128K / 256K / 512K tokens 11.3 / 22.5 / 45.1 GB 6.0 / 12.0 / 24.0 GB 3.6 / 7.1 / 14.3 GB
total memory at 512K (P = 4) 69.3 GB 48.4 GB 38.5 GB
quality vs BF16 (held-out KL divergence, lower is better) +0.0004 +0.0039
quality at long context (log-likelihood of text 32K tokens in) +0.005% +0.20%
picks the same next token as BF16 96.4% 94.6%
decode speed at 1K / 4K (tokens/s) 19.5 / 18.8 19.3 / 18.1 17.7 / 16.9
decode speed at 32K / 128K / 512K (tokens/s) 12.5 / 6.2 / 1.3-2.0 12.7 / 5.8 / 1.84 12.5 / 6.3 / 2.13

For scale, the quality cost of rq4 is about half the difference between the P = 4 and P = 3 weights. The cost of q8 is about ten times smaller than the cost of rq4.

Speed:

  • q8 decodes within a few percent of BF16 at short contexts, and as fast or faster from 32K.
  • rq4 is a little slower at short contexts, because it mixes each latent.
  • rq4 is the fastest from 128K. It reads the fewest bytes, and its kernel reads the 4-bit values without a separate unpack step.

Prefix reuse in oMLX

Prefix reuse keeps a long conversation responsive. When oMLX loads the model, the loader registers q8 and rq4 with the oMLX prefix cache, in memory and on SSD. Measured with a 13,000-token prompt on the M4 Max:

Request q8 rq4
First request, time to first token 42 s 42 s
Same prefix again 3.8 s 3.6 s
Same prefix after an oMLX restart (from the SSD cache) 1.2 s 1.1 s

A future oMLX can change the internals that this feature uses. If that occurs, the loader keeps the BF16 cache and writes a message. Prefix reuse is more important than the cache size.

How the loader chooses

The loader uses q8. If q8 cannot hold the context that you ask for, the loader uses rq4. The available memory is the GPU working set that MLX reports, minus the weights and a reserve for prompt processing. When the loader picks rq4, it writes one message on stdout:

NanoSeedLM: KV cache rq4 (4-bit latent) instead of q8: max_context 262,144 tokens needs 12.0 GB as q8, more than the
10.0 GB available after the weights; rq4 needs 7.1 GB at a small quality cost (...)

If rq4 also cannot hold the context, the message gives the number of tokens that fit.

The loader sets the cache format when it creates the cache for a conversation. Thus you must tell the loader the longest context before you start. Use one of these methods:

  • Set NSLM_MAX_CONTEXT (tokens) in the environment of the process that loads the model, for example oMLX.
  • Add "mla_max_context": 262144 to the config.json of the model folder.
  • If you call the model from Python, use model.make_cache(max_context=262144).

If you use none of these methods, the loader uses q8.

How to choose the format yourself

Set the config.json key mla_kv to one of these values:

Value Cache
"auto", or no key The loader chooses, as above.
"bf16" The uncompressed cache. Fastest decode, most memory.
{"bits": 4, "rope_bits": 8, "group_size": 64, "rotate": true} Always rq4.
{"bits": 8, "rope_bits": 8, "group_size": 64, "rotate": false} Always q8.

How much context fits on your Mac

This table gives the longest context for each model and memory size. It uses the default macOS GPU memory limit. The limit is about two thirds of memory up to 36 GB, and three quarters above 36 GB. MLX reports this limit as the recommended working set. The table subtracts the weights (P = 3 20.1 GB, P = 4 22.2 GB, oQ4e 22.9 GB) and 2.5 GB for prompt processing. 512K is the maximum context of the model.

unified memory GPU-usable model BF16 q8 rq4
32 GB ~23 GB P = 3 none 7K 12K
32 GB ~23 GB P = 4, oQ4e none none none
36 GB ~26 GB P = 3 36K 68K 114K
36 GB ~26 GB P = 4 12K 22K 38K
36 GB ~26 GB oQ4e 4K 8K 13K
48 GB ~39 GB P = 3 182K 343K 512K
48 GB ~39 GB P = 4 158K 298K 499K
48 GB ~39 GB oQ4e 150K 283K 475K
64 GB ~52 GB any ~300K 512K 512K
96 GB and up 77 GB+ any 512K 512K 512K

Measured total memory at 512K tokens on the M4 Max:

Model Cache Total
P = 3 rq4 36.4 GB
P = 4 rq4 38.5 GB
P = 4 q8 48.4 GB
P = 4 BF16 69.3 GB

If you raise the GPU memory limit (iogpu.wired_limit_mb), a Mac moves up the table. The loader reads the limit that is in effect.

The engine: q8, fp8 and fp4

The C engine has its own cache options:

  • GQA models: --kv q8 keeps the cache as int8 with a scale for each token and head. It works on Metal and CUDA. It halves the cache. On an M4 Max, the quality cost is within about one standard error, and decode at 4K context is about 10% slower. Read Engine KV cache codes.
  • MLA models: --kv fp8 and --kv fp4 use the codes of DeepSeek-V4.1 (nslm/kvq.h). The engine quantizes each value when it writes it. Each reader decodes the value exactly to BF16, so the attention arithmetic does not change.
    • fp8 keeps the latent and the RoPE key as FP8 E4M3, with a power-of-two scale for each 32 values. It is 52% of BF16.
    • fp4 keeps the RoPE key and the first 256 dimensions of the latent as FP8. It keeps the other latent dimensions as FP4 E2M1, with an E4M3 scale for each 16 values. It is 38% of BF16.
    • TransMLA latents put most of their variance in the first dimensions. An FP4 RoPE key caused most of the quality loss of the all-FP4 layout of DeepSeek. Thus fp4 keeps those parts in FP8.

The engine also has two memory features for the cache:

  • Paged cache (Metal, macOS 26.4 or later). The engine reserves --ctx for each slot but commits memory in 64 KB pages as it writes positions. It gives the pages back when a slot becomes free or holds a shorter conversation. NSLM_KV_DENSE=1 commits all memory at the start.
  • Cold cache on disk. nslm-serve saves long caches to disk in 256-token blocks and reads them back for a later request with the same prefix. Read Run and serve.

See also

Xet Storage Details

Size:
8.64 kB
·
Xet hash:
3da32d7c148f5499c970f46630c4418cc8579d3e08ea67d8733a1f95aeda47b8

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.