File size: 3,458 Bytes
e5419fe
 
 
 
 
 
 
90cfcba
e5419fe
 
 
 
 
90cfcba
e5419fe
 
 
2c509e8
e5419fe
 
 
 
 
 
90cfcba
e5419fe
 
 
 
90cfcba
 
e5419fe
90cfcba
e5419fe
 
 
 
 
 
90cfcba
e5419fe
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
---
language:
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- infuseloop
- looped-transformer
- causal-lm
- research
---

# InfuseLoop-8

A 596,143,587-parameter base language model with 8 parallel recurrent loops and causal cross-trajectory Attention Residuals. Trained on 6,000,082,944 FineWeb-Edu tokens (45,777 updates).

**Project:** [InfuseLoop](https://thuecstasy.github.io/InfuseLoop/) · **Code:** [thuEcstasy/InfuseLoop](https://github.com/thuEcstasy/InfuseLoop).

## Usage

Install the custom reference implementation:

```bash
pip install git+https://github.com/thuEcstasy/InfuseLoop.git
```

```python
from transformers import AutoTokenizer
from infuseloop import InfuseLoop
repo = "thuEcstasy1/InfuseLoop-8"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = InfuseLoop.from_pretrained(repo, device="cuda")
prompt = "The scientist repeated the experiment and found that"
x = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).input_ids.cuda()
y = model.generate(x, max_new_tokens=32, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(y[0, x.shape[1]:], skip_special_tokens=True))
```

This is a base text model, **not an instruction-tuned assistant**. Use natural prefixes without a chat template. The custom `InfuseLoop` class is required; this repository does not register `AutoModelForCausalLM` or promise compatibility with generic Transformers generation pipelines.

The public API is a readable eager PyTorch reference. It does not reproduce the optimized CUDA Graph / FA4 / Triton latency measurements in the paper.

## Model details

- Shared backbone: 28 layers, hidden dimension 1,024, FFN dimension 3,072.
- Attention: 16 query heads, eight KV heads, head dimension 128.
- G-SWA: shared first-loop global KV; 64-token private local windows.
- Routing: shallower-loop outputs from the preceding execution cycle; source RMSNorm and Attention Residuals.
- Training sequence length: 1,024; global batch: 128 sequences.
- Released precision: original FP32, safetensors; no optimizer states or training data.
- Backbone configuration/tokenizer: Qwen3-0.6B, pinned tokenizer revision `c1899de289a04d12100db370d81485cdf75e47ca`.
- Initialization: pretrained from a common random initialization, **not a fine-tune of pretrained Qwen3 weights**.

## Reported evaluation

| WikiText-2 word PPL ↓ | LAMBADA final-word PPL ↓ | Seven-task avg. (%) ↑ |
|---:|---:|---:|
| 35.55 | 42.04 | 43.07 |

The downstream mean includes MMLU, HellaSwag, ARC-E, ARC-C, PIQA, WinoGrande, and OpenBookQA. MMLU uses five-shot evaluation; other tasks use zero-shot evaluation. Exact metric definitions and benchmark scope are documented in the code repository.

## Limitations

Research checkpoint trained on a relatively small 6B-token budget. Text quality can be inconsistent or repetitive; outputs can contain unsupported claims or reflect biases in pretraining data. Evidence currently covers approximately 0.6B dense models and measured recurrent depths of four and eight. Larger dense/MoE architectures and further depth scaling remain open. No downstream chat alignment or safety fine-tuning is claimed.

## Integrity and license

`export-validation.json` records parameter counts, weight SHA256, and exact source-tensor export checks. Code and model weights are released under Apache-2.0. The tokenizer follows its upstream Apache-2.0 license. No pretraining corpus is redistributed.