| --- |
| license: apache-2.0 |
| library_name: xllm |
| tags: |
| - xllm |
| - looped-language-model |
| - paper-part2 |
| - distillation |
| --- |
| |
| # distilled-s-random-init |
|
|
| Table 4, S, Distilled, random init of *Towards Looped Models Done Right. Part II: Rethinking at Fixed Points*: a distilled prefill path for `huginn-s-learned-entropy0p01`. |
|
|
| > [!IMPORTANT] |
| > **Loading this checkpoint requires the xLLM code.** The weights are stored in xLLM's native |
| > format (BF16 Safetensors with an xLLM `config.json` and `artifact_manifest.json`). This is not a |
| > Hugging Face `transformers` checkpoint: `AutoModel.from_pretrained` cannot load it. |
| > Get the code at [https://github.com/ifm-ai/xllm-loop](https://github.com/ifm-ai/xllm-loop) and load it with its teacher through `xllm.paper_part2.distill.load_student`. |
|
|
| ## Model details |
|
|
| | | | |
| | --- | --- | |
| | Model | a d x d input map and 2 Transformer blocks of the teacher's width | |
| | Parameters | 54,269,952 | |
| | Teacher | [`huginn-s-learned-entropy0p01`](https://huggingface.co/IFM/LoopedLM-P2-huginn-s-learned-entropy0p01) | |
| | Initialization | random | |
| | Target | the teacher's pre-coda state after 5 recurrences, from its prelude output | |
| | Loss | hidden-state MSE over the target energy, plus KL from the teacher's next-token distribution | |
| | Training | recipe `s_distill_random_init`: 2,560 updates of 256 x 8,192 tokens, AdamW (0.9, 0.95), cosine schedule | |
| | Prefill | the student's state, one teacher recurrence and the teacher's coda write the four KV banks; the teacher decodes at R = 5 | |
| | Weights | BF16 Safetensors: the trained FP32 weights rounded to BF16 | |
|
|
| ## Download |
|
|
| ```bash |
| hf download IFM/LoopedLM-P2-huginn-s-learned-entropy0p01 --local-dir huginn-s-learned-entropy0p01 |
| hf download IFM/LoopedLM-P2-distilled-s-random-init --local-dir distilled-s-random-init |
| ``` |
|
|
| The xLLM loader checks the directory against `artifact_manifest.json`: it rejects symbolic links |
| and files the manifest does not list, apart from the `.gitattributes` file and the |
| `.cache/huggingface/` folder that `hf download --local-dir` adds. Download into a directory as |
| above, not into the Hub cache (`~/.cache/huggingface/hub`), whose files are symbolic links. |
|
|
| ## Use |
|
|
| A distilled student is evaluated together with its teacher, which supplies the prelude, one |
| recurrence and the coda. Download both, then run `eval_paper_part2.py` from the xLLM |
| repository: |
|
|
| ```bash |
| ENABLE_FLASH_ATTENTION_3=true python eval_paper_part2.py --artifact huginn-s-learned-entropy0p01 --student distilled-s-random-init \ |
| --data /path/to/eval-data/data.json --out out ppl |
| ``` |
|
|
| `data.json` and the evaluation inputs come from `release/paper-part2/prepare-eval-data.py |
| --output /path/to/eval-data`. |
|
|
| The student's `config.json` pins its teacher's `manifest_sha256`, the digest recorded in the |
| teacher's `artifact_manifest.json` (see Provenance); `xllm.paper_part2.distill.load_student` |
| refuses any other teacher artifact, so use the teacher repository at the matching revision. |
|
|
| ## Provenance |
|
|
| - Teacher manifest_sha256: `9f6da54b5e0748c850624c22af22ff5a01530aa2638f18f1db8d4521f508d307` |
| |
| `artifact_manifest.json` records the size and SHA-256 of every file in this repository. |
|
|
| ## Paper and citation |
|
|
| *Towards Looped Models Done Right. Part II: Rethinking at Fixed Points*: https://arxiv.org/abs/2610.06833 |
|
|
| ```bibtex |
| @article{huang2026fixedpoints, |
| title = {Towards Looped Models Done Right, Part II: Rethinking at Fixed Points}, |
| author = {Benhao Huang and Chufan Shi and Junlin Chen and Shicheng Wen and Zhengzhong Liu and Eric Xing and Xuezhe Ma}, |
| journal = {arXiv preprint arXiv:2610.06833}, |
| year = {2026} |
| } |
| ``` |
|
|
| ## License |
|
|
| The weights are released under the Apache License 2.0 (`LICENSE`); `NOTICE` records the |
| tokenizer's attribution and how the artifact was prepared. The xLLM code is distributed under its |
| own license. |
|
|
