---
license: apache-2.0
language:
  - en
library_name: pytorch
pipeline_tag: text-generation
datasets:
  - HuggingFaceFW/fineweb-edu
tags:
  - proof-of-concept
  - base-model
  - causal-lm
  - gated-deltanet-2
  - parallax-attention
---

# Monody-Exp

**A proof of concept, published for comparison only.** Monody-Exp is a
353M-parameter autoregressive language model, the baseline twin of the
masked-diffusion model [Rondo-Exp](https://huggingface.co/KerzgrAI/Rondo-Exp),
trained side by side with it to compare how each uses the same data over
repeated passes. It is a small base model: it continues text, follows no
instructions, and is not meant for real use.

Both models trained on the same 88M FineWeb-Edu tokens, repeated 7 times (617M tokens), with 2,048-token windows and identical optimizer settings. These are the EMA weights after pass 7; held-out loss is on 101M unseen tokens.

| | Rondo-Exp | Monody-Exp |
|---|---|---|
| Objective | masked diffusion | next token |
| Token mixers | 12 bidirectional GDN-2, 4 bidirectional attention | 12 GDN-2, 4 Parallax attention |
| Held-out loss | ≤ 3.810 (ELBO bound) | 3.632 (exact NLL) |
| Held-out minus training loss | 0.13 | 1.59 |
| Zero-shot accuracy, mean of 6 tasks | 44.8% | 45.6% |

The six tasks are LAMBADA, ARC-Easy, PIQA, SciQ, HellaSwag and WinoGrande (1,000 examples each). Rondo's loss is an upper bound on its NLL, so it is not directly comparable to Monody's.

## Run

Linux with an NVIDIA GPU:

```bash
hf download KerzgrAI/Monody-Exp --local-dir Monody-Exp
pip install -r Monody-Exp/requirements.txt
python Monody-Exp/inference.py --prompt "The water cycle is"
```

The first run installs flash-linear-attention if it is missing. Options:
`--max-new-tokens`, `--temperature` (0 is greedy), `--top-p`, `--top-k` and
`--seed`.