Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model
Abstract
A dynamic weight precision allocation strategy is proposed, utilizing low-bit quantization and uncertainty-based gating to selectively upgrade weight precision during LLM generation, thereby optimizing computational cost.
Local decoding usually fixes one weight precision for the whole generation. We study a different allocation: store 8-bit weights, decode with their top 4 bits, and redo a step at 8 bits only when the low-precision output distribution looks uncertain (entropy above a threshold or a small top-two margin). Quantization is simulated, the model is Qwen3-0.6B, and the benchmark is GSM8K. Cost is counted in weight bytes read per token. On 36724 replayed steps the entropy of the low pass ranks argmax flips with AUC 0.93. At a 15% escalation budget it catches 78% of flips and the margin gate 94%, against the 15% expected for a random gate of equal rate (a computed expectation, not a run). In 150 live problems all-high scores 64.0% and all-low 55.3% (McNemar p=0.053). An entropy gate with a 16% escalation rate scores 62.7% at 0.87 of the all-high weight bytes. Its paired difference is -1.3 (-8.0 to +5.3) points to all-high and +7.3 (+0.0 to +14.7) points to all-low (p=0.080), so the live data do not separate the gate from either end. Verified speculation reaches zero KL at similar bytes in the same cost model. A live random control, a thinking-mode phase rule, a second model and live KL were not run.
Community
Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model
I want to buy the entropy gate, but I think the baseline is doing a lot of unearned work here. 4 bits always plus 8 bits on ~15% of steps is roughly 5.2 bits per weight read, so the comparison that matters isn't fp16 — it's a plain 5-bit (or 6-bit) decode at the same byte budget on the same GSM9K set, and if that lands within noise then uniform spending wins and the gate is overhead.
Second thing: an argmax flip isn't a wrong answer. Scoring AUC 0.93 for predicting flips tells me the gate is good at detecting disagreement, not that it's buying accuracy — I'd want the y-axis to be final-answer correctness at matched bytes, since plenty of flips land on the same answer once you decode a few more tokens.
Third, and this is the one that would bite me in prod: a per-token branch that re-reads weights is fine single-stream, but under continuous batching one sequence's escalation either stalls the batch behind it or forces both precisions resident in VRAM, which quietly doubles your footprint. Did you measure any of this with concurrent requests, or is it all batch-of-one numbers?
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper