LazyWeights
Collection
Decode-on-demand edge LLMs: Huffman-coded llama.cpp formats (arXiv 2611.17108) • 5 items • Updated
Paper (arXiv 2611.17108) · Code (GitHub) · Collection
The operating points of LazyWeights: Decode-on-Demand Edge LLMs are in one repository per model, so that each has its own model tree, GGUF details and hardware estimates:
| Model | Operating points (≤1% / ≤3.5% perplexity budget) | Saving vs llama.cpp |
|---|---|---|
| Phi-3-mini-4k-instruct | ≤1% 4.046 b/w / ≤3.5% 3.649 b/w | 6.2% / 11.2% |
| Mistral-7B-Instruct-v0.2 | ≤1% 3.886 b/w / ≤3.5% 3.422 b/w | 31.4% / 5.7% |
| SmolLM-1.7B-Instruct | ≤1% 5.162 b/w / ≤3.5% 3.934 b/w | 9.8% / 10.3% |
Each repository holds the GGUF files (unmodified llama.cpp) and the Huffman-coded runtime bundles. Code: github.com/arnabsanyal/lazyWeights.
@misc{sanyal2026lazyweightsdecodeondemandedgellms,
title = {{LazyWeights: Decode-on-Demand Edge LLMs}},
author = {Sanyal, Arnab and Datta, Gourav and Chinchali, Sandeep P.},
year = {2026},
eprint = {2611.17108},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2611.17108}
}