PreNormLoop-0.6B

📚 Paper · 💻 Code

PreNormLoop is a fully-open pretrained looped transformer with PreNorm residual connections, serving as the baseline for ARLoop. This release contains the final checkpoint of the 0.6B, 180B-token training run. It is a base pretrained model, intended for text completion, which can be used for mid-training and post-training.

Load and generate

pip install 'transformers==5.22.0' safetensors
# Install a PyTorch build suitable for your CPU or CUDA environment separately.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "osunlp/PreNormLoop-0.6B"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16,
).to(device).eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to(device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Weights load in BF16 for inference. The stored checkpoint retains the original FP32 weights; normalization statistics use FP32 internally.

Architecture

Property	Value
Physical transformer layers	18, shared across 3 recurrent passes
Effective transformer depth	54
Hidden / MLP intermediate width	1536 / 4608
Attention / KV heads	16 / 4
Context length	4096
RoPE base	50000
Residual connections	Standard PreNorm additions; no attention-residual routing
Value embeddings	Shared across alternating physical layers
Tokenizer vocabulary	49152, SmolLM2 tokenizer used in training
Output head	Tied to token embeddings

License

Model release: Apache License 2.0.

Training checkpoints

main contains the final checkpoint.

Optimizer step	Training tokens	Revision
171,660 (final)	179,998,556,160	main

Citation

@misc{yao2026arloop,
  title={{ARLoop}: Fully-Open Pre-trained Looped Transformers with Attention Residuals},
  author={Yuekun Yao and Yupei Du and Dawei Zhu and Huan Sun},
  year={2026},
  eprint={2611.05234},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2611.05234}
}