Transformer baseline (134M) trained from scratch on BabiStories
A small research model: the official Memory Mosaics v2 architecture (here: its Transformer baseline) trained from scratch on BabiStories, to compare it with a Transformer of the same width, depth and data. It is not derived from any pretrained model (no Qwen, Llama or Mistral weights); only the Mistral tokenizer is reused.
| this model | paired model | |
|---|---|---|
| family | transformer |
mosaics_v2 |
| parameters | 134,105,856 | 148,594,128 |
| held-out loss (BabiStories val, 2M tokens) | 1.7824 | 1.8113 |
Paired model trained the same way: Eureka-COM/MemoryMosaics-v2-149M-BabiStories
Architecture
12 layers, width 768, vocabulary 32,000 (Mistral), context 512. Embedding and output are not tied (49.2M). Each layer: RMSNorm β causal self-attention (6 heads Γ 128, RoPE) β residual β RMSNorm β SwiGLU FFN (2048) β residual. Per layer: attention 2.36M, FFN 4.72M.
Training
- Data: BabiStories (stories generated with Mixtral 8x7B, distributed in facebookresearch/MemoryMosaics, Apache-2.0), tokenized with the Mistral-7B-v0.1 tokenizer; documents kept apart (document-causal masking)
- 15,502 steps Γ 8,192 tokens (16 Γ 512) = 127M tokens, AdamW (Ξ² 0.9/0.95, wd 0.1), lr 3e-4 with 5% warmup and cosine to 1%, bf16, seed 42
- One RTX 5090. The same run on a Colab A100 gave the same losses within 0.003
- Both models were trained with the same data order and the same script
Results (held-out loss through training)
- 10%: 2.5464
- 25%: 2.1532
- 50%: 1.9452
- 75%: 1.8237
- 100%: 1.7824
Memory Mosaics v2 ends 0.029 (1.6%) above the Transformer while having 11% more parameters; the gap narrowed from +0.062 at 25% of training.
Character-name recall (in-distribution)
On 183,315 later mentions of character names in all 21,541 held-out BabiStories stories, the next-token argmax is the name's first token for 64.5% (Memory Mosaics v2) vs 65.0% (Transformer). The Transformer is slightly but consistently ahead (β0.5 points, paired McNemar p = 4e-8); the gap is well under 2 points. Mean log-probability of the name token: β1.39 vs β1.31.
| distance to previous mention | questions | Memory Mosaics v2 | Transformer | p |
|---|---|---|---|---|
| 1β32 tokens | 124,186 | 64.5% | 64.8% | 0.0008 |
| 33β128 tokens | 58,440 | 64.9% | 65.6% | 4e-6 |
| 129β256 tokens | 677 | 35.9% | 37.7% | 0.30 |
| 257+ tokens | 12 | 0% | 25% | (too few) |
Almost all repeats are within 128 tokens, so this mostly measures short-range recall (see Limitations). The same test on 2,000 name mentions in 229 training stories (seen once): 65.75% vs 66.25%; neither model does noticeably better on stories it has seen, so the scores reflect in-context recall rather than memorized stories.
Training cost
| Transformer 134M | Memory Mosaics v2 149M | |
|---|---|---|
| run time on one RTX 5090 (127M tokens + 5 evaluations) | 26 min | 54 min (Γ2.1) |
| of which training only | 21 min | 41 min |
| training speed | 102,883 tokens/s | 51,122 tokens/s |
| peak GPU memory (PyTorch, batch 16 Γ 512) | 7.4 GB | 13.4 GB (Γ1.8) |
| one evaluation (2M held-out tokens + probes) | 1.2 min | 2.7 min |
| Colab A100 80GB, both runs sharing one GPU | 69 min | 152 min (about 24k tokens/s, did not speed up when running alone) |
Memory Mosaics v2 takes about twice as long to train as the Transformer at this size. Both models use the upstream code as released (the leaky-average recurrence runs on the upstream kernel; attention and the memories use the upstream PyTorch code), with no further tuning for speed.
Limitations
- One run per model (seed 42). The gap was the same on a second run on a different GPU (A100 vs RTX 5090, within 0.003), but seed-to-seed variation was not measured.
- The learning rate (3e-4, the upstream default) was not tuned for either model; the best rate may differ between the two architectures.
- Memory Mosaics v2 has 11% more parameters than the Transformer (its two memories per layer are larger than one attention).
- Small scale only: 12 layers, 127M tokens.
- Short texts only; long-range memory is not tested. Training and evaluation use a 512-token context with document-causal masking: every story is its own document and the models never read across stories. BabiStories stories are short (validation set: median 225 tokens, 90th percentile 289, longest 507). In Memory Mosaics v2 the short memory reads the last 256 tokens and the long memory reads tokens more than 64 positions back, so the long memory is active at 72% of positions, but only 3.9% of positions lie more than 256 tokens into a story, the range that only the long memory can reach. These results therefore say little about long-range memory; that needs models trained on long documents (thousands of tokens).
- The
recallentries inevals.jsonlcome from a generic probe (repeat a 5-digit access code) whose format never occurs in BabiStories; both models score 0/20, so it does not compare their abilities. Use the character-name recall below instead. - These are base language models trained on children's stories, not instruction-tuned, and are meant for research comparisons only.
Loading
load_and_generate.py (in this repo) builds the model with the official code of
facebookresearch/MemoryMosaics at commit c2ca0e8, loads model.safetensors, prints the loss of the prompt and continues it.
The tokenizer (tokenizer.json, from Mistral-7B-v0.1, BOS=1, EOS=2) is included.
git clone https://github.com/facebookresearch/MemoryMosaics
git -C MemoryMosaics checkout c2ca0e88c5c89fc18a0a9465d6f7a1c3c8ea8faf
pip install torch safetensors tokenizers pyyaml huggingface_hub
wget https://huggingface.co/Eureka-COM/Transformer-134M-BabiStories/resolve/main/load_and_generate.py
python load_and_generate.py --upstream ./MemoryMosaics --repo Eureka-COM/Transformer-134M-BabiStories --prompt "Once upon a time, a little fox"
A CUDA GPU is needed (the upstream memories use GPU kernels, compiled on first use). Greedy output of both models for the default prompt:
- Memory Mosaics v2: Once upon a time, a little fox named Fifi found a shiny, red apple. She wanted to eat it, but it was too high up. She saw a big, tall tree and had an idea. Fifi ran to the tree and climbed up. ...
- Transformer: Once upon a time, a little fox named Fifi found a shiny, red apple. She ran to show her friend, Amaan. Amaan was a kind boy who loved to play. Fifi showed Amaan the apple. ...
In your own code, the model is the official class with the overrides in config.json β settings:
from safetensors.torch import load_file
from model.mosaics.config import MosaicsConfig # Transformer: model.transformer.config.TransformerConfig
from model.mosaics.model import MemoryMosaics # model.transformer.model.Transformer
model = MemoryMosaics(MosaicsConfig(**cfg["settings"]))
model.load_state_dict(load_file("model.safetensors"))
Citation
The architecture and the training data come from these papers (this repository is an independent small-scale replication, not by their authors):
@inproceedings{zhang2025memorymosaics,
title = {Memory Mosaics},
author = {Zhang, Jianyu and Nolte, Niklas and Sadhukhan, Ranajoy and Chen, Beidi and Bottou, L{\'e}on},
booktitle = {The Thirteenth International Conference on Learning Representations (ICLR)},
year = {2025}
}
@inproceedings{zhang2025memorymosaicsatscale,
title = {Memory Mosaics at scale},
author = {Zhang, Jianyu and Bottou, L{\'e}on},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2025},
eprint = {2507.03285},
archivePrefix = {arXiv}
}
License
Apache-2.0, as the upstream code, BabiStories and the Mistral tokenizer.
- Downloads last month
- 53

