Papers
arxiv:2611.07226

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Published on Nov 4
· Submitted by
Xuan Ouyang
on Nov 16
Authors:
,
,
,
,

Abstract

A looped Transformer that separately controls input strength, total update weight, and per-update allocation achieves lower validation loss and higher downstream accuracy than standard and looped residual baselines while using fewer parameters.

Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later reads weight it. A full block output contains both its input and its new update, so weighting that output also reweights the history already carried by the input. Reading individual updates instead, with the shared stack's input kept on a separate path, gives separate controls over input strength, total update weight, and allocation across updates. The Loopback Transformer implements these controls: geometric decay sets the total update weight and supplies a recency prior, and attention learns how to allocate that weight. On ClimbMix, Loopback reaches lower validation loss than standard Transformer baselines with 40-75% of their Transformer parameters (excluding token embeddings and the output head) at reference scales from 191M to 3B, with training tokens and executed depth matched. It also has higher mean accuracy over thirteen downstream tasks at 1.4B and 3B. At 191M, 374M, and 778M, it has the lowest validation loss and the highest thirteen-task mean among the standard Transformer and seven looped residual designs.

Community

Paper author Paper submitter

In this paper, we study how residual connections should retain and reuse information when the same Transformer blocks run repeatedly. Inspired by recurrent memory, we develop a unified view of residual connections as memory operating along network depth. The write–transport–read formulation lets us compare different residual designs through what they store, how they carry information forward, and how they retrieve it for subsequent computation.

Guided by this view, we introduce Loopback Transformer. Loopback stores individual block updates, uses geometric forgetting to progressively discount earlier passes, and learns content-dependent attention over the stored updates. This allows the model to retrieve useful information from earlier computation.

13e96ba0103625b69219978d829be3fd

At matched training tokens and executed depth, Loopback achieves lower validation loss and higher mean accuracy across 13 zero-shot tasks than Standard Transformers at both 1.4B and 3B, retaining only 68% and 75% of their Transformer parameters, respectively. At 3B, validation loss improves from 2.0005 to 1.9849, mean accuracy rises from 48.56% to 49.45%, and Loopback scores higher on 11 of 13 tasks. In our main comparisons at 191M, 374M, and 778M, it also achieves the lowest validation loss and highest thirteen-task mean among Standard and seven looped residual baselines. Parameter counts exclude token embeddings and the output head.

Our results highlight residual design as an important part of parameter-efficient modeling: sharing computation works better when the model also learns how to retain and reuse its outputs. We welcome feedback and discussion!

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.07226 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.07226 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.07226 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.