File size: 1,804 Bytes
5c63f8f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | ---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- proof-of-concept
- base-model
- causal-lm
- gated-deltanet-2
- parallax-attention
---
# Monody-Exp
**A proof of concept, published for comparison only.** Monody-Exp is a
353M-parameter autoregressive language model, the baseline twin of the
masked-diffusion model [Rondo-Exp](https://huggingface.co/KerzgrAI/Rondo-Exp),
trained side by side with it to compare how each uses the same data over
repeated passes. It is a small base model: it continues text, follows no
instructions, and is not meant for real use.
Both models trained on the same 88M FineWeb-Edu tokens, repeated 7 times (617M tokens), with 2,048-token windows and identical optimizer settings. These are the EMA weights after pass 7; held-out loss is on 101M unseen tokens.
| | Rondo-Exp | Monody-Exp |
|---|---|---|
| Objective | masked diffusion | next token |
| Token mixers | 12 bidirectional GDN-2, 4 bidirectional attention | 12 GDN-2, 4 Parallax attention |
| Held-out loss | ≤ 3.810 (ELBO bound) | 3.632 (exact NLL) |
| Held-out minus training loss | 0.13 | 1.59 |
| Zero-shot accuracy, mean of 6 tasks | 44.8% | 45.6% |
The six tasks are LAMBADA, ARC-Easy, PIQA, SciQ, HellaSwag and WinoGrande (1,000 examples each). Rondo's loss is an upper bound on its NLL, so it is not directly comparable to Monody's.
## Run
Linux with an NVIDIA GPU:
```bash
hf download KerzgrAI/Monody-Exp --local-dir Monody-Exp
pip install -r Monody-Exp/requirements.txt
python Monody-Exp/inference.py --prompt "The water cycle is"
```
The first run installs flash-linear-attention if it is missing. Options:
`--max-new-tokens`, `--temperature` (0 is greedy), `--top-p`, `--top-k` and
`--seed`.
|