Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
Abstract
Replacing a learned embedding table with a frozen binary code table reduces model size and parameters while maintaining or improving performance on several benchmarks, and enables faster CPU inference with a ternary body.
In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed. We ask what that table actually buys. We replace it with one frozen table of 512-bit codes, 16 MB, with no trainable parameters, derived once from a pretrained embedding by principal-component projection and sign binarisation, and we train the same Gemma-shaped body from scratch on the same 2.05B tokens with and without it. The learned table buys 0.20 nats of validation loss for 2.6 times the trainable parameters, but on LAMBADA, HellaSwag and PIQA the frozen-table model is better or tied; only ARC-Easy favours the learned table. With a ternary BitNet b1.58 body the whole model is a 46 MB file, still ties the float model on LAMBADA and PIQA, and on a laptop CPU generates 615 to 740 tokens per second in 62 MB of memory, five to six times the speed of the float model's 8-bit GGUF in llama.cpp at a seventh of the memory. The frozen table wins by 1.7 to 2.3 nats on rare words, accepts 10,000 new words after training with a 3% cost where the learned table collapses by 43%, and all of it is released with the weights, the data and every evaluation answer.
Community
In small language models the vocabulary table is the biggest part: 168M of 268M parameters in Gemma 3 270M. This paper replaces it with one frozen 512-bit code per token, derived from Gemma 3's own embedding and never trained. Trained on the same 2.05B tokens as a normal model, it uses 2.6x fewer trainable parameters, ties or beats the learned table on three of four benchmarks, and handles rare and newly added words far better. With a ternary body the whole model is 46 MB and runs at 615 to 740 tok/s on a laptop CPU. Code, data, every checkpoint and the CPU kernel: https://huggingface.co/QLNI/embeddings-are-overpriced
Shipping a 16 MB frozen table on a phone is the part I'd actually test first, and the average loss is the least useful number you could hand me. Break it out per script and per domain — a sign-binarised PCA projection will look fine on Latin and ugly on anything with a long tail of rare codepoints, and the mean hides that completely. The other thing I want is the cost of adding a token after the fact: if the projection is frozen, does a new token mean refitting the projection, or is there a cheap out-of-vocab path, and what does taking it cost in quality? And run the same comparison at a larger token budget, because at small scale the body never gets enough gradient to push back on the embeddings — the interesting regime is where it does and the frozen table becomes the bottleneck. Until then I read this as a solid on-device default, not as evidence that embeddings are overpriced in general.
Get this paper in your agent:
hf papers read 2610.22428 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper