DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
Abstract
DAGFormer improves Transformer efficiency by using a jointly trained topology predictor to plan cross-layer connections globally, reducing perplexity and boosting downstream task performance across model scales.
We propose DAGFormer, a Transformer language model whose cross-layer connections are planned by a jointly trained topology predictor. Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens. Unlike existing cross-layer routing architectures, which set each layer's connections separately, DAGFormer plans the routing of the whole model at once. On matched OLMo-2 pairs from 75M to 1B parameters, DAGFormer lowers WikiText-2 perplexity by 17% to 25% over the vanilla Transformer, and a 1B DAGFormer trained on 5B tokens outperforms the vanilla model trained on 10B. Against four routing baselines with the same data and budget, it reaches the lowest perplexity at 75M, 300M and 1B. The gain carries over to 13 commonsense, science, reading and math evaluation tasks (2.0 to 2.5 points higher mean accuracy at 300M and 1B) and to instruction finetuning (63% to 70% AlpacaEval win rate against the vanilla Transformer, leading every routing baseline), and under structured pruning DAGFormer degrades the least in five of six settings.
Community
TL;DR: DAGFormer plans a Transformer's cross-layer connections up front as a global topology for each token. A small topology predictor reads the input tokens and, before the backbone runs, outputs a signed weight on every earlier layer for each head's Q, K and V and each residual input.
What's different: in DenseFormer, hyper-connections, mHC and MUDDFormer, each layer decides its own routing, one layer at a time. DAGFormer makes routing global: the whole graph is one object, planned per input, that you can audit and intervene on. It also allows explicit coordination between cross-layer attention heads.
Results (matched OLMo-2 backbones, same data, tokens and seed)
- 17–25% lower WikiText-2 perplexity than vanilla from 75M to 1B
- A 1B DAGFormer trained on 5B tokens beats a vanilla 1B trained on 10B
- vs mHC: 10–18% lower perplexity at every size, +0.7–0.8 pts mean accuracy on 13 tasks
- Lowest perplexity among all four routing baselines at 75M, 300M and 1B
- After instruction tuning: 63–70% AlpacaEval win rate vs vanilla (mHC: 55–62%)
Cost: prefill takes 1.5–1.9× vanilla, in the same range as mHC.
Interactive version: click any token or layer to see the graph DAGFormer plans for it: https://interpretability.web.illinois.edu/wp-content/uploads/2026/10/DAGFormer.html
Get this paper in your agent:
hf papers read 2610.19625 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper