license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- proof-of-concept
- base-model
- causal-lm
- gated-deltanet-2
- parallax-attention
Monody-Exp
A proof of concept, published for comparison only. Monody-Exp is a 353M-parameter autoregressive language model, the baseline twin of the masked-diffusion model Rondo-Exp, trained side by side with it to compare how each uses the same data over repeated passes. It is a small base model: it continues text, follows no instructions, and is not meant for real use.
Both models trained on the same 88M FineWeb-Edu tokens, repeated 7 times (617M tokens), with 2,048-token windows and identical optimizer settings. These are the EMA weights after pass 7; held-out loss is on 101M unseen tokens.
| Rondo-Exp | Monody-Exp | |
|---|---|---|
| Objective | masked diffusion | next token |
| Token mixers | 12 bidirectional GDN-2, 4 bidirectional attention | 12 GDN-2, 4 Parallax attention |
| Held-out loss | ≤ 3.810 (ELBO bound) | 3.632 (exact NLL) |
| Held-out minus training loss | 0.13 | 1.59 |
| Zero-shot accuracy, mean of 6 tasks | 44.8% | 45.6% |
The six tasks are LAMBADA, ARC-Easy, PIQA, SciQ, HellaSwag and WinoGrande (1,000 examples each). Rondo's loss is an upper bound on its NLL, so it is not directly comparable to Monody's.
Run
Linux with an NVIDIA GPU:
hf download KerzgrAI/Monody-Exp --local-dir Monody-Exp
pip install -r Monody-Exp/requirements.txt
python Monody-Exp/inference.py --prompt "The water cycle is"
The first run installs flash-linear-attention if it is missing. Options:
--max-new-tokens, --temperature (0 is greedy), --top-p, --top-k and
--seed.