| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - arloop |
| - recurrent |
| - custom_code |
| - pretrained |
| --- |
| |
| # ARLoop-0.6B |
|
|
| 📚 [Paper](https://arxiv.org/abs/2611.05234) · 💻 [Code](https://github.com/OSU-NLP-Group/ARLoop) |
|
|
| ARLoop is a fully-open pretrained looped transformer with attention residuals. This release |
| contains the final checkpoint of the 0.6B, 180B-token training run. |
| It is a **base pretrained model**, intended for text completion, which can be used for mid-training and post-training. |
|
|
|  |
|
|
| ## Load and generate |
|
|
| ```bash |
| pip install 'transformers==5.22.0' safetensors |
| # Install a PyTorch build suitable for your CPU or CUDA environment separately. |
| ``` |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "osunlp/ARLoop-0.6B" |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, trust_remote_code=True, dtype=torch.bfloat16, |
| ).to(device).eval() |
| inputs = tokenizer("The capital of France is", return_tensors="pt").to(device) |
| with torch.inference_mode(): |
| output = model.generate(**inputs, max_new_tokens=64, do_sample=False) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| Weights load in BF16 for inference. The stored checkpoint retains the original FP32 weights; routing and normalization statistics use FP32 internally. |
|
|
| ## Architecture |
|
|
| | Property | Value | |
| | --- | --- | |
| | Physical transformer layers | 18, shared across 3 recurrent passes | |
| | Effective transformer depth | 54 | |
| | Hidden / MLP intermediate width | 1536 / 4608 | |
| | Attention / KV heads | 16 / 4 | |
| | Context length | 4096 | |
| | RoPE base | 50000 | |
| | Attention-residual block size | 18 sublayers | |
| | Recent residual blocks | 3, with input embedding retained | |
| | Value embeddings | Shared across alternating physical layers | |
| | Tokenizer vocabulary | 49152, SmolLM2 tokenizer used in training | |
| | Output head | Tied to token embeddings | |
|
|
| ## License |
|
|
| Model release: [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). |
|
|
| ## Training checkpoints |
|
|
| `main` contains the final checkpoint. |
|
|
| | Optimizer step | Training tokens | Revision | |
| | ---: | ---: | --- | |
| | 171,660 (final) | 179,998,556,160 | [main](https://huggingface.co/osunlp/ARLoop-0.6B/tree/main) | |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{yao2026arloop, |
| title={{ARLoop}: Fully-Open Pre-trained Looped Transformers with Attention Residuals}, |
| author={Yuekun Yao and Yupei Du and Dawei Zhu and Huan Sun}, |
| year={2026}, |
| eprint={2611.05234}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2611.05234} |
| } |
| ``` |
|
|
