| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - foresightloop |
| - looped-transformer |
| - causal-lm |
| - research |
| --- |
| |
| # ForesightLoop-4 |
|
|
| A 596,139,465-parameter base language model with 4 parallel recurrent loops and causal cross-trajectory Attention Residuals. Trained on 6,000,082,944 FineWeb-Edu tokens (45,777 updates). |
|
|
| **Code:** [thuEcstasy/ForesightLoop](https://github.com/thuEcstasy/ForesightLoop). |
|
|
| ## Usage |
|
|
| Install the custom reference implementation: |
|
|
| ```bash |
| pip install git+https://github.com/thuEcstasy/ForesightLoop.git |
| ``` |
|
|
| ```python |
| from transformers import AutoTokenizer |
| from foresightloop import ForesightLoop |
| repo = "thuEcstasy1/ForesightLoop-4" |
| tokenizer = AutoTokenizer.from_pretrained(repo) |
| model = ForesightLoop.from_pretrained(repo, device="cuda") |
| prompt = "The scientist repeated the experiment and found that" |
| x = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).input_ids.cuda() |
| y = model.generate(x, max_new_tokens=32, eos_token_id=tokenizer.eos_token_id) |
| print(tokenizer.decode(y[0, x.shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| This is a base text model, **not an instruction-tuned assistant**. Use natural prefixes without a chat template. The custom `ForesightLoop` class is required; this repository does not register `AutoModelForCausalLM` or promise compatibility with generic Transformers generation pipelines. |
|
|
| The public API is a readable eager PyTorch reference. It does not reproduce the optimized CUDA Graph / FA4 / Triton latency measurements in the paper. |
|
|
| ## Model details |
|
|
| - Shared backbone: 28 layers, hidden dimension 1,024, FFN dimension 3,072. |
| - Attention: 16 query heads, eight KV heads, head dimension 128. |
| - G-SWA: shared first-loop global KV; 64-token private local windows. |
| - Routing: shallower-loop outputs from the preceding execution cycle; source RMSNorm and Attention Residuals. |
| - Training sequence length: 1,024; global batch: 128 sequences. |
| - Released precision: original FP32, safetensors; no optimizer states or training data. |
| - Backbone configuration/tokenizer: Qwen3-0.6B, pinned tokenizer revision `c1899de289a04d12100db370d81485cdf75e47ca`. |
| - Initialization: pretrained from a common random initialization, **not a fine-tune of pretrained Qwen3 weights**. |
|
|
| ## Reported evaluation |
|
|
| | WikiText-2 word PPL ↓ | LAMBADA final-word PPL ↓ | Seven-task avg. (%) ↑ | |
| |---:|---:|---:| |
| | 35.70 | 45.08 | 43.00 | |
|
|
| The downstream mean includes MMLU, HellaSwag, ARC-E, ARC-C, PIQA, WinoGrande, and OpenBookQA. MMLU uses five-shot evaluation; other tasks use zero-shot evaluation. Exact metric definitions and benchmark scope are documented in the code repository. |
|
|
| ## Limitations |
|
|
| Research checkpoint trained on a relatively small 6B-token budget. Text quality can be inconsistent or repetitive; outputs can contain unsupported claims or reflect biases in pretraining data. Evidence currently covers approximately 0.6B dense models and measured recurrent depths of four and eight. Larger dense/MoE architectures and further depth scaling remain open. No downstream chat alignment or safety fine-tuning is claimed. |
|
|
| ## Integrity and license |
|
|
| `export-validation.json` records parameter counts, weight SHA256, and exact source-tensor export checks. Code and model weights are released under Apache-2.0. The tokenizer follows its upstream Apache-2.0 license. No pretraining corpus is redistributed. |
|
|
