| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: text-generation |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| tags: |
| - proof-of-concept |
| - base-model |
| - masked-diffusion |
| - diffusion-lm |
| - gated-deltanet-2 |
| --- |
| |
| # Rondo-Exp |
|
|
| **A proof of concept, published for comparison only.** Rondo-Exp is a |
| 352M-parameter masked-diffusion language model, trained side by side with its |
| autoregressive twin [Monody-Exp](https://huggingface.co/KerzgrAI/Monody-Exp) |
| to compare how each uses the same data over repeated passes. It is a small base |
| model: it continues text, follows no instructions, and is not meant for real use. |
|
|
| Both models trained on the same 88M FineWeb-Edu tokens, repeated 7 times (617M tokens), with 2,048-token windows and identical optimizer settings. These are the EMA weights after pass 7; held-out loss is on 101M unseen tokens. |
|
|
| | | Rondo-Exp | Monody-Exp | |
| |---|---|---| |
| | Objective | masked diffusion | next token | |
| | Token mixers | 12 bidirectional GDN-2, 4 bidirectional attention | 12 GDN-2, 4 Parallax attention | |
| | Held-out loss | ≤ 3.810 (ELBO bound) | 3.632 (exact NLL) | |
| | Held-out minus training loss | 0.13 | 1.59 | |
| | Zero-shot accuracy, mean of 6 tasks | 44.8% | 45.6% | |
|
|
| The six tasks are LAMBADA, ARC-Easy, PIQA, SciQ, HellaSwag and WinoGrande (1,000 examples each). Rondo's loss is an upper bound on its NLL, so it is not directly comparable to Monody's. |
|
|
| ## Run |
|
|
| Linux with an NVIDIA GPU: |
|
|
| ```bash |
| hf download KerzgrAI/Rondo-Exp --local-dir Rondo-Exp |
| pip install -r Rondo-Exp/requirements.txt |
| python Rondo-Exp/inference.py --prompt "The water cycle is" |
| ``` |
|
|
| The first run installs flash-linear-attention if it is missing. Options: |
| `--max-new-tokens`, `--temperature` (0 is greedy), `--top-p`, `--top-k`, |
| `--seed`, and `--drafts` (tokens drafted and checked per forward; 0 reveals |
| one token per forward). |
|
|
