Papers
arxiv:2611.19649

LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers

Published on Nov 14
· Submitted by
siqi zhu
on Nov 17
Authors:
,
,
,

Abstract

LoopFill reduces decoding latency for looped language models by combining multi-token suffix drafts and shallow-loop predictions into a single draft tree with full-depth verification that preserves the autoregressive output distribution.

Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. During autoregressive decoding, however, generating each token requires several sequential passes through these layers, leading to high decoding latency. Pipelined self-speculative methods use shallow-loop predictions to process tokens at different loops in the same batch, but each token must reach the drafting depth before the next can enter the pipeline. We introduce LoopFill, a training-free speculative decoding algorithm for looped LLM that combines multi-token suffix drafts with shallow-loop predictions in a single draft tree. Both drafters contribute to one tree, so a suffix lookup can immediately extend a shallow prediction and several future tokens can enter the pipeline together. The shared block computes the highest-scoring draft candidates at different loop depths in one batch, and full-depth verification preserves the model's autoregressive output distribution. Across seven benchmarks, LoopFill's average speedup over greedy autoregressive decoding is 5.85times on Ouro-2.6B and 5.13times on Huginn-3.5B. The speedups exceed those of LoopAhead, the strongest baseline, by \(40\%\) on Ouro-2.6B and \(25\%\) on Huginn-3.5B.

Community

Paper submitter

Looped Transformers improve parameter effi-
ciency by repeatedly applying the same Trans-
former layers. During autoregressive decoding,
however, generating each token requires several
sequential passes through these layers, leading to
high decoding latency. Pipelined self-speculative
methods use shallow-loop predictions to process
tokens at different loops in the same batch, but
each token must reach the drafting depth before
the next can enter the pipeline. We introduce
LoopFill, a training-free speculative decoding al-
gorithm for looped LLM that combines multi-
token suffix drafts with shallow-loop predictions
in a single draft tree. Both drafters contribute
to one tree, so a suffix lookup can immediately
extend a shallow prediction and several future to-
kens can enter the pipeline together. The shared
block computes the highest-scoring draft candi-
dates at different loop depths in one batch, and
full-depth verification preserves the model’s au-
toregressive output distribution. Across seven
benchmarks, LoopFill’s average speedup over
greedy autoregressive decoding is 5.85×on Ouro-
2.6B and 5.13×on Huginn-3.5B. The speedups
exceed those of LoopAhead, the strongest base-
line, by 40% on Ouro-2.6B and 25% on Huginn-
3.5B.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.19649
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.19649 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.19649 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.19649 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.