CyanMonkey/Test-Subject-3

Oleander-150M

Training checkpoint (export) — step 6,500 / 28,820 · 22.55% of the planned run.

steps    ██████░░░░░░░░░░░░░░░░░░░░░░  22.55%   6,500 / 28,820
tokens   ██████░░░░░░░░░░░░░░░░░░░░░░  22.93%   3.46B / 15.11B compute
phase    tst   (TST bag training ends at step 8,646)

Trained with the Oleander marimo trainer (molab): hybrid GDN2 (gated delta-rule, gated-delta-net-2) and gated GQA layers with block attention residuals, MuonEq-R + AdamW, WSD schedule.

Training progress

Step	6,500 / 28,820
% complete (steps)	22.55%
Compute tokens seen	3.46B / 15.11B (22.93%)
Raw tokens seen (TST bags unpacked)	20.79B / 37.77B
Cluster tokens (all 1 worker(s))	3.46B / 15.11B compute
Phase	tst
Val loss / ppl	5.5231 / 250.40
Train loss (latest)	5.2200
Throughput	77,100 compute tok/s · 0.145 steps/s
Session elapsed	0d 11h 23m · ETA 1d 18h 53m
Checkpoint written	2026-11-13T07:27:27Z

How to load

The repo is self-contained: config.json carries an auto_map that points at configuration_oleander.py and modeling_oleander.py (shipped here), so remote code is all you need.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "CyanMonkey/Test-Subject-3"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

prompt = "Once upon a time"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=64,
                     do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0], skip_special_tokens=True))

The GDN2 mixers use flash-linear-attention kernels when they are installed and you run in bf16 on a GPU (pip install --no-deps fla-core flash-linear-attention); otherwise a pure-PyTorch recurrent reference is used — the same math, but slower for long prompts. There is no KV cache (use_cache=False).

Architecture

Oleander-150M (28L = 7 x [GDN2 · GDN2 · GDN2 · GQA] · d=576 · SwiGLU(1600) · tied 32,770 vocab)

Parameters	149,976,139 (149.98 M)
Layers	28 = 7 x [gdn2, gdn2, gdn2, gqa]
Hidden size / FFN	576 / 1600 (SwiGLU)
Vocab / context	32,770 (tied embeddings) / 1,024
Features	GDN2 (gated delta net 2) x21: 9h x 64, conv4, gated GQA x7: 9 Q / 3 KV heads, head_dim 64, QK-Norm (offset form), partial RoPE (rotary_dim 32, theta 100000), value residual (v_mix), SwiGLU, RMSNorm, AttnRes (block attention residuals), tied embeddings, TST (token superposition training, bag 6)

Training setup

Optimizer	muoneq_r (Muon) + AdamW — muon_adamw
LR schedule	WSD · warmup 100 · stable until 82.5% · 1-sqrt cooldown
Batch	524,288 compute tokens / step (32 x 16 x 1024)
TST	True · bag 6 · first 30% of steps
Data	CyanMonkey/NextGen-1-32-1024 (train), 14,753,933 rows x 1024 tokens
Precision	bf16 autocast · weights exported as float32

Files

file	what
model.safetensors	weights, trainer parameter names
config.json	auto_map -> AutoConfig / AutoModelForCausalLM remote-code entry points
configuration_oleander.py	OleanderConfig
modeling_oleander.py	OleanderForCausalLM (same math as the trainer)
tokenizer.json, tokenizer_config.json, special_tokens_map.json	tokenizer
training_meta.json	detailed run metadata (this checkpoint)
progress.json	flat step / tokens / % summary
generation_config.json, train_config.json	sampling defaults, full trainer config

Notes

• This is a work-in-progress checkpoint at 22.55% of the planned run, not a finished model: expect repetition and incoherence, especially with few-shot prompts.
• The tokenizer comes from the base model the run started from; the vocabulary is a plain BPE model and is not extended during training.
• training_meta.json is regenerated on every push and is the authoritative record of how far training got.