Chorus: Shared N-gram Embeddings with Layer-wise Adaptation
Abstract
Chorus shares one set of n-gram embedding tables across all layers, letting each layer project and rescale the embedding using its hidden state, achieving better quality with much smaller tables and lower optimizer memory.
Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. Sharing a single table, however, gives every layer the same embedding and leaves no room for layer-wise preferences, and it performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own way. Each layer maps the refined embedding through its own projection layer, after rescaling its components by factors computed from its current hidden state. Chorus matches the quality of existing n-gram methods with 87% smaller tables: with 1.2B table parameters it predicts text better than FactorEngram and DeepSeek's Engram do with 9.6B. It outperforms Engram, TN-gram and FactorEngram in every setting where we compared them, across table sizes from 20M to 9.6B parameters. We also show that the recently proposed memory-efficient optimizer Ember works well with Chorus, saving 99.2% of table-optimizer memory compared with AdamW without degrading training performance. These results enable further scaling of n-gram embedding tables along the table-size axis by reducing memory use and allowing shared knowledge to be reused across layers.
Community
N-gram embedding tables improve language models by storing learned representations of short token sequences, but recent designs commonly use separate tables at different layers, consuming substantial memory and preventing sharing. Chorus instead looks up token and n-gram embeddings once, combines and refines them, and shares the resulting representations across the input and intermediate layers. Each layer adapts these shared representations to its own needs.
Experiments on ClimbMix and FineWeb show improvements across model sizes. In our table-size comparison, Chorus with 1.2 billion table parameters outperforms Engram and FactorEngram with 9.6 billion, using roughly 87% smaller tables. Training these tables with the Ember optimizer also reduces optimizer-state memory by 99.2% compared with AdamW while preserving performance. These results allow n-gram tables to grow within a smaller memory budget while reusing knowledge across layers.
Get this paper in your agent:
hf papers read 2611.23386 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper