Meridian-Tiny

A ~12M-parameter language model trained from scratch on a single RTX 3060, built to test whether a stack of recent architecture ideas can be combined, scaled down, and trained on consumer hardware.

Meridian-Tiny is a research toy, not a useful assistant. It produces locally plausible, globally nonsensical text. What makes it interesting is what's inside: hybrid linear/full attention, a latent mixture-of-experts, manifold-constrained hyper-connections, attention residuals across depth, and hashed n-gram memory, all in one model small enough to train in minutes.

Everything here (tokenizer, data pipeline, model code, training loop) was written from scratch and verified component by component against reference implementations.

Architecture

Each layer is built from up to three sublayers, each wrapped in its own hyper-connection:

1. Engram (selected layers only): hashed n-gram memory
2. Mixer: KDA linear attention (3 of every 4 layers) or MLA full attention (every 4th layer)
3. FFN: dense SwiGLU in the first layer, latent MoE everywhere else

Component	What it does	Origin
KDA (Kimi Delta Attention)	Linear attention with a fixed-size memory matrix per head, updated by a gated delta rule with per-channel forgetting. Includes a causal short convolution.	Kimi Linear (Moonshot AI)
MLA (Multi-head Latent Attention)	Full attention with queries and keys/values compressed through low-rank latents, plus a decoupled RoPE key shared across heads and a sigmoid output gate.	DeepSeek-V2/V3; output gate as in Qwen3-Next
Latent MoE	Tokens are projected into a smaller latent space before the routed experts. Sigmoid router, grouped top-k routing, one always-on shared expert, and aux-loss-free load balancing via a per-expert routing bias.	DeepSeek-V3 routing; latent-space experts
mHC (manifold-constrained hyper-connections)	The residual stream is widened to 4 parallel streams. Each sublayer learns dynamic pre/post weights and a stream-mixing matrix constrained to be doubly stochastic via Sinkhorn iterations.	DeepSeek
Block AttnRes (attention residuals)	At block boundaries, the residual state is rebuilt as a softmax-weighted mix of all previous block snapshots, scored by a learned pseudo-query per boundary.	Kimi (Moonshot AI)
Engram	For each token, the preceding 2- and 3-grams are hashed (4 hash functions each) into a large embedding table. Retrieved memory is gated against the current hidden state before being added.	DeepSeek

Implementation notes

This is a from-scratch interpretation, not a reproduction of any official code. Deviations worth knowing:

• mHC + AttnRes together is this project's own integration choice. mHC handles the residual within each block; AttnRes rebuilds the 4-stream state at block boundaries. No published model combines them, as far as I know.
• The KDA forget gate is parameterized as log α = lower_bound · sigmoid(W x) with lower_bound = −5. The fast path uses the chunk_kda kernel from flash-linear-attention, verified to match a hand-written recurrent reference (max abs diff ~2e-4).
• Engram skips the original vocabulary-compression step, uses one shared table per Engram layer with distinct hash multipliers per head, and uses a simplified context gate.
• The MoE runs all experts in batched matmuls using fixed-capacity buffers (capacity factor 2.0). Overflow assignments are dropped for that expert. Verified to match a per-expert loop reference when no tokens overflow.

Model details

	Meridian-Tiny
Parameters (total)	≈11.8M
Token embedding + LM head (untied)	8.4M
Engram table	2.1M
Everything else	≈1.3M (≈1.0M active per token)
Hidden size	128
Layers	4 (KDA, KDA, KDA, MLA)
Heads	2
KDA head dim	64
MLA dims	q rank 64, kv rank 32, nope 32, rope 16, v 64
Experts	8 routed (top-2, 2 groups) + 1 shared; dense FFN in layer 0
Expert latent / intermediate	96 / 64
Hyper-connection streams	4
AttnRes block size	2 layers
Engram	layer 1, 65,521 rows × 32, 2- and 3-grams, 2 hash heads per order
Context length	512
Vocabulary	32,768

Most of Meridian-Tiny's parameters are the embedding and the Engram table. The part of the network doing actual computation is around a million parameters per token.

Tokenizer

Byte-level BPE with a 32,768-token vocabulary, trained on ~1.05B characters sampled from the training mix (Japanese upsampled for the tokenizer only).

• Byte fallback, so no input is ever un-tokenizable
• Japanese script splitting: pre-tokenization splits at kanji / hiragana / katakana boundaries (with ー treated as katakana), so merges follow word stems and inflections
• Individual digits with the leading space attached ( 12 → 1, 2), for more consistent arithmetic
• No Unicode normalization (NFKC would flatten math symbols like x²)
• Special tokens: <|pad|> = 0, <|bos|> = 1, <|eos|> = 2, plus 13 reserved slots

Measured efficiency (characters per token): English 3.38, German 3.07, Russian 2.20, Japanese 1.67, Python 2.38, Rust 2.57.

Training data

A 4B-token corpus was packed; Meridian-Tiny saw 20M tokens of it, sampled according to this mix:

Share	Domain	Source
60%	English web	FineWeb-Edu (sample-10BT)
15%	Math	FineMath (finemath-4plus)
7.5%	Python	StarCoderData
4.5%	C++	StarCoderData
3%	Rust	StarCoderData
4%	German	FineWeb-2 (deu_Latn)
4%	Russian	FineWeb-2 (rus_Cyrl)
2%	Japanese	FineWeb-2 (jpn_Jpan)

Documents are separated by <|eos|>. Training samples are random 513-token windows drawn from per-source token files, so the mix can be changed without re-tokenizing. No additional quality filtering was applied beyond what the source datasets provide.

FineWeb-Edu, FineMath and FineWeb-2 are released under ODC-By 1.0. StarCoderData is derived from The Stack; code in it remains under its original licenses and is subject to the dataset's terms of use.

Training

Tokens	20M
Steps	4,882
Batch	8 × 512 tokens (4,096 tokens per step)
Optimizer	AdamW, β = (0.9, 0.95), weight decay 0.1
Weight decay exclusions	1D params, token embeddings, Engram tables, hyper-connection params
Peak LR	2e-3
Schedule	WSD: 2% linear warmup, stable, 15% linear decay to zero
Gradient clipping	1.0
Precision	bf16 autocast, fp32 master weights
MoE balancing	routing-bias update of ±1e-3 per step, no auxiliary loss
Hardware	1× RTX 3060 12GB
Wall time

Results

Validation loss (nats per token) on held-out data from each source, at the end of training:

mix	web	math	python	cpp	rust	german	russian	japanese
4.949	4.996	4.447	4.224	4.507	3.942	6.335	6.319	7.002

Code and math reached lower loss than English prose, since they're highly structured. The non-English languages lag, as expected given their small share of the data, but each improved substantially (Japanese: 10.85 → 7.00).

Ablations

Each component was removed one at a time and trained under identical conditions: same hyperparameters, same 20M tokens, same data order. The seed1 row is the unmodified baseline with different initial weights, which measures run-to-run noise.

run	mix	web	math	python	cpp	rust	german	russian	japanese
baseline	4.949	4.996	4.447	4.224	4.507	3.942	6.335	6.319	7.002
seed1	4.994	5.003	4.595	4.380	4.691	4.053	6.338	6.246	6.936
no AttnRes	4.990	5.035	4.510	4.285	4.538	4.000	6.346	6.356	6.958
no Engram	5.133	5.162	4.681	4.512	4.822	4.202	6.430	6.372	7.021
no mHC (1 stream)	4.989	5.024	4.525	4.322	4.614	4.026	6.316	6.309	6.939

Noise level. The two baseline seeds land 0.045 apart on the mix loss, so differences smaller than roughly 0.05 can't be distinguished from luck. Per-source numbers are noisier still (math alone differs by 0.15 between seeds), so the mix column is the most reliable signal.

Engram clearly helps. Removing it costs 0.184 nats on the mix, about 4× the seed noise, making it the only component with a measurable effect at this scale. The cost is not evenly spread. Compared with the average of the two baseline seeds:

Domain	Loss increase without Engram
Python / C++ / Rust	+0.21 / +0.22 / +0.20
English web / math	+0.16 / +0.16
German / Russian	+0.09 / +0.09
Japanese	+0.05

This matches what Engram is for: a lookup memory for short repeated phrases. Code is dense with exact recurring n-grams (self., std::, let mut), so it benefits most. The non-English languages, at 2–4% of the data each, likely haven't seen their n-grams often enough for the table to become useful yet.

AttnRes and mHC show no measurable effect here. Removing either changes the mix loss by about 0.04, within seed noise. Both mechanisms route information across depth, and a 4-layer model offers very little depth to route across. This is a "not demonstrated at this scale" result, not evidence that they're useless.

mHC has a real speed cost. Carrying 4 residual streams, plus a Sinkhorn projection for every sublayer, made the baseline noticeably slower than the 1-stream variant, which trained at roughly 38–60K tokens/s. At this scale, dropping mHC gives the same loss for less compute. Whether the extra streams pay for themselves at greater depth is an open question; the full-size model keeps them to find out.

Samples

Temperature 0.8, top-k 50. Prompts in bold.

fn main() { let mut result = 0; ...then collapses into a repetition loop.

def fibonacci(n): produces correctly indented Python with return, elif, except and np. calls, with nonsensical semantics.

猫は その中で、私は、その後には、その方は「その方」を出し、自分の関して、そして、と言わたと思って…

Die Katze sitzt, was unsightly developed in many areas...

The last one shows an interesting failure: German drifts into English at "was" (a word in both languages), while Russian and Japanese prompts stay in their own language. Scripts that don't share tokens with English seem to act as a barrier against drifting into the dominant language.

What's next

Meridian-Tiny was the smoke test. A full-size configuration is training now on the same code and data, keeping every component, including mHC, to see whether depth changes the ablation picture:

	Tiny	Full
Total parameters	≈11.8M	236.3M
Hidden size / layers	128 / 4	512 / 12 (full attention at layers 4, 8, 12)
Experts	8, top-2	32, top-4, in 4 groups
Engram	1 layer	2 layers, 524,287 rows each
Context	512	2,048
Tokens per optimizer step	4K	131K

Usage

Meridian-Tiny uses custom model code. Download model.py, presets.py, tokenizer.json and model.safetensors from this repo, then:

import torch
from safetensors.torch import load_file
from tokenizers import Tokenizer
from model import MiniMeridian
from presets import TINY

tok = Tokenizer.from_file("tokenizer.json")
net = MiniMeridian(TINY["model"])
net.load_state_dict(load_file("model.safetensors"))
net.eval()

ids = torch.tensor([tok.encode("fn main() {").ids])
with torch.no_grad():
    for _ in range(40):
        logits = net(ids)[:, -1, :] / 0.8
        next_id = torch.multinomial(torch.softmax(logits, dim=-1), 1)
        ids = torch.cat([ids, next_id], dim=1)
print(tok.decode(ids[0].tolist()))

Dependencies: torch, tokenizers, safetensors. Optional: flash-linear-attention for the fast KDA kernel on GPU (plus triton-windows on Windows). Without it, the model falls back to a pure-PyTorch recurrent implementation that's slower but numerically equivalent.

Generation recomputes the full sequence for each token (no KV cache), which is fine at this size.

Limitations

• Not useful for any real task. ~1M active compute parameters and 20M training tokens produce fluent-looking fragments, not meaning.
• Not instruction-tuned, not safety-tuned. It continues text; it does not follow instructions.
• Prone to repetition loops, especially in code and math.
• German output tends to drift into English.
• Trained on unfiltered-beyond-source web text, and may reproduce anything found in it.

Acknowledgements

Architecture ideas from Moonshot AI (Kimi Linear, KDA, attention residuals), DeepSeek (MLA, MoE routing, mHC, Engram) and Qwen (gated attention). Fast KDA kernel from the flash-linear-attention project. Data from Hugging Face (FineWeb-Edu, FineMath, FineWeb-2) and BigCode (StarCoderData).