---
license: apache-2.0
library_name: xllm
pipeline_tag: text-generation
tags:
- xllm
- looped-language-model
- paper-part1
- dense
---

# Dense Split H/L · 336B tokens

Checkpoint of the `dense_split_hl` recipe at the 336B-token endpoint (optimizer step 80,000),
from *Towards Looped Models Done Right. Part I: Topology, Input Injection, Recurrent-State Organization*.

`dense_split_hl` applies each six-layer H/L module once per state update; `dense_split_hl_x2` applies it twice. Both use eight state updates, with input injection and cross-state addition once per update, and share the same parameter count and checkpoint tensor layout.

> [!IMPORTANT]
> **Loading this checkpoint requires the xLLM code.** The weights are stored in xLLM's native
> format (BF16 Safetensors with an xLLM `config.json` and `artifact_manifest.json`). This is not a
> Hugging Face `transformers` checkpoint: `AutoModel.from_pretrained` cannot load it.
> Get the code at [https://github.com/ifm-ai/xllm-loop](https://github.com/ifm-ai/xllm-loop) and load the directory with `xllm.paper_part1.native_inference.load_native_model`.

## Model details

| Field | Value |
|---|---|
| Recipe | `dense_split_hl` |
| Family | Dense H/L state variant |
| Architecture (`model.arch`) | `huginn` |
| Endpoint | `336b`: optimizer step 80,000, 335,544,320,000 training tokens |
| Parameters (resident) | 1,499,030,016 |
| Parameters (activated per token) | 1,499,030,016 |
| Training seed | 1 |
| Optimizer | AdamW; weight decay 0.1 |
| LR schedule | cosine; 200-step warmup; 119,210-step horizon (the 500B endpoint) |
| Global batch | 512 sequences × 8,192 tokens = 4,194,304 tokens/step |
| Weights | BF16 Safetensors, 1 shard of at most 5 GiB |
| Tokenizer | `tokenizer/`, derived from IFM/K2-Horizon-3.7B (see `NOTICE`) |

All Part I endpoints share one 119,210-step learning-rate schedule, so this 336B-token checkpoint
is taken mid-schedule, before the cosine decay completes at the 500B-token endpoint.

## Download

```bash
hf download IFM/LoopedLM-P1-dense-split-hl-336b --local-dir models/dense_split_hl/336b
```

The xLLM loader checks the directory against `artifact_manifest.json`: it rejects symbolic links
and files the manifest does not list, apart from the `.gitattributes` file and the
`.cache/huggingface/` folder that `hf download --local-dir` adds. Download into a directory as
above, not into the Hub cache (`~/.cache/huggingface/hub`), whose files are symbolic links.

## Use

Native inference requires a CUDA GPU and FlashAttention 3; the
[xLLM repository](https://github.com/ifm-ai/xllm-loop) documents installation.

```bash
export ENABLE_FLASH_ATTENTION_3=true
```

```python
from xllm.paper_part1.native_inference import generate_native, load_native_model

model, tokenizer, config = load_native_model("models/dense_split_hl/336b")
tokens = generate_native(model, tokenizer, ["The capital of France is"],
                         max_gen_len=32, use_sampling=False)
print(tokenizer.decode(tokens[0]))
```

The loader verifies the directory against `artifact_manifest.json`, then strictly loads tensor
names, shapes and dtypes. Evaluate with the paper protocol:

```bash
ENABLE_FLASH_ATTENTION_3=true python eval_paper_part1.py --artifact models/dense_split_hl/336b \
    --tasks-root /path/to/composite-eval-root --output /path/to/evaluation-output \
    --protocol paper-part1
```

Retrain from scratch with the same recipe (the base config holds the data, tokenizer, logging and
checkpoint-retention settings):

```bash
ENABLE_FLASH_ATTENTION_3=true python train_paper_part1.py --recipe dense_split_hl --target 336b \
    --base-config /path/to/base.json --dump-dir /path/to/new-run
```

## Files

- `config.json`: the recipe's `model` fields with the training checkpoint's values, and the
  `tokenizer` settings with `tokenizer.path` set to `tokenizer/`.
- `model.safetensors.index.json` and `model-*-of-*.safetensors`: BF16 weights; no tensor is split
  across shards.
- `tokenizer/`: `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`.
- `artifact_manifest.json`: size and SHA-256 of every file.
- `LICENSE`, `NOTICE`.

Optimizer, dataloader and training state are not included; resuming a run mid-training needs the
original training checkpoint.

## Paper and citation

*Towards Looped Models Done Right. Part I: Topology, Input Injection, Recurrent-State Organization*: https://huskydoge.github.io/husky-blog/posts/recursive_models/towards-looped-models-done-right/

```bibtex
@misc{huang2026loopedmodels,
  title  = {Towards Looped Models Done Right. Part I: Topology, Input Injection, Recurrent-State Organization},
  author = {Benhao Huang and Chufan Shi and Junlin Chen and Shicheng Wen and Zhengzhong Liu and Eric Xing and Xuezhe Ma},
  year   = {2026},
  url    = {https://huskydoge.github.io/husky-blog/posts/recursive_models/towards-looped-models-done-right/}
}
```

## License

The weights and tokenizer are released under the Apache License 2.0 (`LICENSE`). `NOTICE`
records the tokenizer's attribution to
[IFM/K2-Horizon-3.7B](https://huggingface.co/IFM/K2-Horizon-3.7B/tree/86a26d7bf092484faa047121d5816f037d4a8ffa)
and its modifications. The xLLM code is distributed under its own license.