LazyWeights: Decode-on-demand edge LLMs

Paper (arXiv 2611.17108) · Code (GitHub) · Collection

LazyWeights

The operating points of LazyWeights: Decode-on-Demand Edge LLMs are in one repository per model, so that each has its own model tree, GGUF details and hardware estimates:

Model Operating points (≤1% / ≤3.5% perplexity budget) Saving vs llama.cpp
Phi-3-mini-4k-instruct ≤1% 4.046 b/w / ≤3.5% 3.649 b/w 6.2% / 11.2%
Mistral-7B-Instruct-v0.2 ≤1% 3.886 b/w / ≤3.5% 3.422 b/w 31.4% / 5.7%
SmolLM-1.7B-Instruct ≤1% 5.162 b/w / ≤3.5% 3.934 b/w 9.8% / 10.3%

Each repository holds the GGUF files (unmodified llama.cpp) and the Huffman-coded runtime bundles. Code: github.com/arnabsanyal/lazyWeights.

Citation

@misc{sanyal2026lazyweightsdecodeondemandedgellms,
  title         = {{LazyWeights: Decode-on-Demand Edge LLMs}},
  author        = {Sanyal, Arnab and Datta, Gourav and Chinchali, Sandeep P.},
  year          = {2026},
  eprint        = {2611.17108},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2611.17108}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including arnabsanyal/lazyWeights

Paper for arnabsanyal/lazyWeights