license: apache-2.0
language:
- en
library_name: mlx
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- mlx
- apple-silicon
- pretrained-from-scratch
- gated-deltanet
- linear-attention
- product-key-memory
- long-context
Gala-598M — a Mac-first language model, trained from scratch on one M3 Max
Gala is a non-transformer LM designed around Apple Silicon's actual profile: lots of unified memory, modest FLOPs. Hoard parameters and state (cheap on a Mac), ration FLOPs per token (scarce). Everything here — pretraining, three-seed baselines, a 300M-token scale study, long-context finetunes, memory characterization, and a 10.5M-token context demo — ran on a single MacBook (M3 Max, 40-core GPU, 128 GB) in stock MLX. No CUDA, no custom kernels, no cluster.
Architecture
Four untied cells, each: Gated DeltaNet mixer (fixed-size fp32 fast-weight state) →
hoard MLP (product-key-routed sparse blocks via mx.gather_mm: 1,024 blocks × 64
neurons, top-32 — a token touches 3% of a 537M-parameter pool) → 256-token sliding-window
attention → hoard MLP. 598M total parameters, ~78M active per token. Tied embeddings.
GPT-2 BPE.
Headline results (all measured, logs in results/)
Quality, matched data — 120M identical FineWeb-Edu tokens, same batch order, same optimizer, 3 seeds each:
| seeds | mean ± spread | |
|---|---|---|
| Gala-598M | 3.8145 / 3.8128 / 3.8203 | 3.816 ± 0.008 |
| Transformer-77M (matched active FLOPs) | 3.8887 / 3.8856 / 3.8863 | 3.887 ± 0.003 |
Gap ≈ 10× seed noise. A tied-embeddings Transformer variant (3.860) also loses. At 300M tokens the lead grows: 3.581 vs 3.684.
Context scaling — 10,485,760 real FineWeb tokens streamed through this checkpoint, batch 1, on the laptop: ingest ~6 minutes, decode 382–388 tok/s flat at every depth, loss-on-next-2k flat from 32k to 10.5M, recurrent state 3.07 MB constant, peak RAM 5.3 GB. The matched Transformer decays 643 → 134 tok/s by 32k and would need ~33 GB of KV at 1M.
Memory, characterized honestly — passkey retrieval, 30 trials × 6 depths, chance = 20%: 100% (30/30) when the needle is inside the 256-token window, at both 8k and 32k context; chance at every depth beyond it. The state carries a measurable gist (loss at depth 2–4k into a document is 0.36 better than at depth 0–256) but not verbatim facts. The in-window control shows the boundary is architectural, not model capability.
Files
| file | what |
|---|---|
gala_u_300M.safetensors |
main checkpoint: 300M FineWeb-Edu tokens, val 3.581 |
gala_u_32k_ft.safetensors |
+ curriculum finetune to seq 32k (1k→8k→32k), val 3.564 @32k |
model.py … campaign4.py |
complete training/eval/orchestration code (pure MLX) |
results/ |
raw training logs and campaign verdicts for every number above |
PAPER.md, CHRONICLE.md, NOVELTY.md |
the report, the five-day worklog, the novelty sweep |
Use
pip install "mlx>=0.32" numpy tiktoken
python sample.py --run <dir-with-checkpoint-and-config> --prompt "The history of" --tokens 150
python bench_1m.py --run <dir> --total 10485760 # the 10M demo, ~7 min on an M3 Max
(To use these weights with the scripts: put model.safetensors + config.json in one
directory — e.g. rename gala_u_300M.safetensors and gala_u_300M.config.json.)
Training
Muon (0.02, NorMuon-style row normalization) on hidden matrices + AdamW (3e-3) elsewhere,
fp32 master weights and optimizer state, bf16 compute, fully-fp32 GDN mixer, mx.compile
(+7.5%), seq 1024, batch 32,768 tokens, warmup 100 / 30% linear cooldown, grad clip 1.0.
~11.3k tok/s training throughput on the M3 Max.
One numerics note worth reading if you reimplement chunked delta rules (PAPER.md §6):
computing the chunk's triangular inverse via the exact nilpotent identity
(I−A)(I+A²)(I+A⁴)… detonates in fp32 once training correlates the keys (~500×-per-chunk
state growth into inf, from finite inputs, through a provably bounded recurrence). Use
blocked forward substitution. Instability class independently established by
arXiv:2605.21325; the training-side account and the probe that found it (debug_nan.py)
are in this repo.
Limitations
A research model, not an assistant: 420M training tokens total, FineWeb-Edu only, no instruction tuning, no downstream evals yet. Verbatim recall is window-local by construction (see above). Training wall-clock is ~1.5× slower than the matched Transformer on today's unfused MLX sparse path. Single machine, single dataset; scale conclusions beyond 598M/300M tokens are roadmap, not claims.
Citation
@misc{gala2026,
title = {Gala: a million tokens on a laptop, and what it cost},
author = {junafinity},
year = {2026},
note = {https://huggingface.co/junafinity/Gala-598M-MLX}
}