LowRAM predictors for Llama-3.1-8B-Instruct

Pretrained predictor checkpoints for LowRAM, a method for compressing the KV cache of long-context LLMs, for use with meta-llama/Llama-3.1-8B-Instruct.

This repository does not contain a language model. The base model's weights stay frozen. These checkpoints are the small networks LowRAM uses to reconstruct the cache, and they are loaded by the code in the GitHub repository.

Paper: LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time Teresa Delgado, Hugo Hazard, Martin Benfeghoul, Jun Wang, Zafeirios Fountas

What is LowRAM?

LowRAM reconstructs values from their keys, stores corrections for the hardest-to-predict values, and compresses keys across layers. Lightweight predictors adapt to each prompt at test time. LowRAM can be combined with quantisation and token eviction.

Folder	File	What it is	Flag in the LowRAM code
`meta_init/`	`meta_mlps.pt` (152 MB), `config.yaml`	Meta-learned initialisation for the per-layer value MLPs (first-order MAML on FineWeb-Edu)	`--meta_weights_path`
`value_mlps/`	`value_mlps.pt` (25 MB)	Pretrained value-MLP weights (plain pretraining, used as a warm start)	`--value_mlp_weights_path`
`attention_predictor/`	`model_ckpt.pt` (66 KB), `config.yaml`	Small shared CNN that predicts attention over key blocks; drives eviction and chooses which value residuals to keep	`--attn_predictor_path` (reads the `config.yaml` next to it)

The config.yaml files record the training configuration of each checkpoint.

Usage

Set up the LowRAM repository first. Then, from the repository root, download the checkpoints into the layout the code expects. This replaces git lfs pull:

import pathlib, shutil
from huggingface_hub import snapshot_download

src = pathlib.Path(snapshot_download("lowram-model/lowram-llama-3.1-8b"))
for part in ["meta_init", "value_mlps", "attention_predictor"]:
    shutil.copytree(src / part, pathlib.Path("checkpoints") / part / "Llama-3.1-8B-Instruct",
                    dirs_exist_ok=True)

Run the demo with the fast LowRAM preset (meta-learned init, two fitting steps, selective reconstruction, 4× target compression):

python -m examples.lowram_demo --method fast

Or evaluate on RULER with xKV keys and meta-learned MLP values at 4×:

python lm_eval_script.py --tasks ruler \
    -kc xkv --comp_ratio 4.0 --xkv_layer_group_size 4 \
    -vc mlp --target_cr 4.0 \
    --num_epochs 2 \
    --meta_weights_path checkpoints/meta_init/Llama-3.1-8B-Instruct/meta_mlps.pt \
    --batch_size 4

To download a single component, use hf_hub_download("lowram-model/lowram-llama-3.1-8b", "meta_mlps.pt", subfolder="meta_init").

Access to meta-llama/Llama-3.1-8B-Instruct must be approved on Hugging Face before the base model can be loaded.

Results with Llama-3.1-8B-Instruct

Result	Retained score / benefit	Setting
4× value-cache compression	99.3% of the uncompressed LongBench score	Values only; Table 5
Up to 5× higher peak decode throughput	Higher aggregate decode throughput across batch sizes	262K-token context; 4× target; selective reconstruction

These are separate experimental settings. Retained score is the method's benchmark score divided by the uncompressed baseline score. Compression ratios refer to the KV cache, not total GPU memory. The throughput result compares peaks across batch sizes, not per-request latency; the uncompressed baseline is faster at small batch sizes. See the paper for full results.

Limitations

These checkpoints only work with Llama-3.1-8B-Instruct. For Mistral, use lowram-model/lowram-mistral-7b.
They are loaded by the LowRAM codebase, not by transformers on their own.
The .pt files are PyTorch pickles. Only load them from this repository or another source you trust.

Citation

@article{lowram2026,
  title   = {{LowRAM}: Low-Rank Keys and Amortised Values for
             Composable {KV} Cache Compression at Test-Time},
  author  = {Delgado, Teresa and Hazard, Hugo and
             Benfeghoul, Martin and Wang, Jun and
             Fountas, Zafeirios},
  journal = {arXiv preprint arXiv:2611.07025},
  year    = {2026},
  doi     = {10.48550/arXiv.2611.07025},
  url     = {https://arxiv.org/abs/2611.07025}
}