---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- arloop
- recurrent
- custom_code
- pretrained
---

# PreNormLoop-0.6B

📚 [Paper](https://arxiv.org/abs/2611.05234) · 💻 [Code](https://github.com/OSU-NLP-Group/ARLoop)

PreNormLoop is a fully-open pretrained looped transformer with PreNorm residual connections, serving as the baseline for ARLoop. This release
contains the final checkpoint of the 0.6B, 180B-token training run.
It is a **base pretrained model**, intended for text completion, which can be used for mid-training and post-training.

![Figure 1: ARLoop benchmark results](assets/figure1.png)

## Load and generate

```bash
pip install 'transformers==5.22.0' safetensors
# Install a PyTorch build suitable for your CPU or CUDA environment separately.
```

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "osunlp/PreNormLoop-0.6B"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16,
).to(device).eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to(device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

Weights load in BF16 for inference. The stored checkpoint retains the original FP32 weights; normalization statistics use FP32 internally.

## Architecture

| Property | Value |
| --- | --- |
| Physical transformer layers | 18, shared across 3 recurrent passes |
| Effective transformer depth | 54 |
| Hidden / MLP intermediate width | 1536 / 4608 |
| Attention / KV heads | 16 / 4 |
| Context length | 4096 |
| RoPE base | 50000 |
| Residual connections | Standard PreNorm additions; no attention-residual routing |
| Value embeddings | Shared across alternating physical layers |
| Tokenizer vocabulary | 49152, SmolLM2 tokenizer used in training |
| Output head | Tied to token embeddings |

## License

Model release: [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).

## Training checkpoints

`main` contains the final checkpoint.

| Optimizer step | Training tokens | Revision |
| ---: | ---: | --- |
| 171,660 (final) | 179,998,556,160 | [main](https://huggingface.co/osunlp/PreNormLoop-0.6B/tree/main) |

## Citation

```bibtex
@misc{yao2026arloop,
  title={{ARLoop}: Fully-Open Pre-trained Looped Transformers with Attention Residuals},
  author={Yuekun Yao and Yupei Du and Dawei Zhu and Huan Sun},
  year={2026},
  eprint={2611.05234},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2611.05234}
}
```