---
language:
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- foresightloop
- looped-transformer
- causal-lm
- research
---

# ForesightLoop-4

A 596,139,465-parameter base language model with 4 parallel recurrent loops and causal cross-trajectory Attention Residuals. Trained on 6,000,082,944 FineWeb-Edu tokens (45,777 updates).

**Code:** [thuEcstasy/ForesightLoop](https://github.com/thuEcstasy/ForesightLoop).

## Usage

Install the custom reference implementation:

```bash
pip install git+https://github.com/thuEcstasy/ForesightLoop.git
```

```python
from transformers import AutoTokenizer
from foresightloop import ForesightLoop
repo = "thuEcstasy1/ForesightLoop-4"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = ForesightLoop.from_pretrained(repo, device="cuda")
prompt = "The scientist repeated the experiment and found that"
x = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).input_ids.cuda()
y = model.generate(x, max_new_tokens=32, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(y[0, x.shape[1]:], skip_special_tokens=True))
```

This is a base text model, **not an instruction-tuned assistant**. Use natural prefixes without a chat template. The custom `ForesightLoop` class is required; this repository does not register `AutoModelForCausalLM` or promise compatibility with generic Transformers generation pipelines.

The public API is a readable eager PyTorch reference. It does not reproduce the optimized CUDA Graph / FA4 / Triton latency measurements in the paper.

## Model details

- Shared backbone: 28 layers, hidden dimension 1,024, FFN dimension 3,072.
- Attention: 16 query heads, eight KV heads, head dimension 128.
- G-SWA: shared first-loop global KV; 64-token private local windows.
- Routing: shallower-loop outputs from the preceding execution cycle; source RMSNorm and Attention Residuals.
- Training sequence length: 1,024; global batch: 128 sequences.
- Released precision: original FP32, safetensors; no optimizer states or training data.
- Backbone configuration/tokenizer: Qwen3-0.6B, pinned tokenizer revision `c1899de289a04d12100db370d81485cdf75e47ca`.
- Initialization: pretrained from a common random initialization, **not a fine-tune of pretrained Qwen3 weights**.

## Reported evaluation

| WikiText-2 word PPL ↓ | LAMBADA final-word PPL ↓ | Seven-task avg. (%) ↑ |
|---:|---:|---:|
| 35.70 | 45.08 | 43.00 |

The downstream mean includes MMLU, HellaSwag, ARC-E, ARC-C, PIQA, WinoGrande, and OpenBookQA. MMLU uses five-shot evaluation; other tasks use zero-shot evaluation. Exact metric definitions and benchmark scope are documented in the code repository.

## Limitations

Research checkpoint trained on a relatively small 6B-token budget. Text quality can be inconsistent or repetitive; outputs can contain unsupported claims or reflect biases in pretraining data. Evidence currently covers approximately 0.6B dense models and measured recurrent depths of four and eight. Larger dense/MoE architectures and further depth scaling remain open. No downstream chat alignment or safety fine-tuning is claimed.

## Integrity and license

`export-validation.json` records parameter counts, weight SHA256, and exact source-tensor export checks. Code and model weights are released under Apache-2.0. The tokenizer follows its upstream Apache-2.0 license. No pretraining corpus is redistributed.