Cubic Hier 150M

A 157M parameter bilingual research language model built on a custom attention block: a differentiable depth-memory path that lets every layer read a compressed copy of the state at every earlier layer, plus hierarchical group retrieval in the two penultimate attention layers.

Training stage of this checkpoint: complete (dpo done) (step 0).

Read this before comparing the numbers

The table below is not an architecture comparison. This model saw 3 billion pretraining tokens. SmolLM2-135M saw roughly 2 trillion — a 600x larger data budget on a model of the same size. Pythia-160M saw ~300B. Any gap in the table is dominated by that difference, not by the attention design. Read it as "what 3B tokens buys you", nothing more.

All rows were produced by the same script, the same prompt formats and the same log-likelihood protocol in a single run, so the rows are comparable to each other. They are not directly comparable to published numbers from other harnesses, which differ in prompt formatting, few-shot count and normalization.

MMLU at this scale is at chance (25%) for every model in the table, this one included. It is reported because it was requested, not because it is informative below roughly 1B parameters.

Results

Headline metric per task: acc_norm for multiple choice with unequal-length options, acc otherwise. Evaluated on an evenly strided subsample of up to 1500 examples per task.

Read Δ chance, not Avg. The tasks have different random-guess floors — 0% for LAMBADA, 25% for the four-way questions, 50% for the binary ones — so a plain average across them is not a meaningful quantity. Δ chance is the mean margin over the random baseline and is the column that actually compares.

Two further caveats. BoolQ's validation set is about 62% "yes", so any score near 62 means the model is answering yes to everything and carries no signal. ARC-Challenge, OpenBookQA, WinoGrande and MMLU sit at chance for every model of this size and should not be read as differences.

Architecture

• 16 layers, dim 800, 10 heads, SwiGLU 2176, 32k byte-level BPE with digit splitting
• RoPE base 500k, per-head QK-RMSNorm, sequence length 4096
• Depth memory: a shared compressor produces a rank-200 key/value pair from the input of every layer; each later layer attends over that history with a learned per-layer bias, and injects the result through three multiplicative gates (content, agreement, mixing). The path is fully differentiable — it is not detached.
• Hierarchical retrieval in layers 13 and 14: cosine scoring against key-group means over 128-token groups, causal top-group routing, then one fused attention over the selected groups. Layer 15 is full global causal attention.
• Trained with hybrid Muon (matrices, with a boosted depth group) + AdamW (embeddings, norms, gates), warmup-stable-decay, and a one-token-ahead multi-token-prediction auxiliary loss.

Usage

This is not a Transformers-compatible checkpoint. The architecture lives in modeling_cubic.py in this repository.

python inference_example.py

Limitations

Small, experimental, trained on a compute-optimal but tiny token budget. It will confabulate. Do not use it for anything factual, safety-relevant or mathematical without independent verification. Bilingual EN/RU by design, but Russian coverage is 13% of pretraining and correspondingly weaker.

Raw metrics: hellaswag=30.1, arc_easy=39.1, arc_challenge=25.3, piqa=57.8, winogrande=50.8, openbookqa=25.2, sciq=70.4, boolq=61.8, lambada_openai=20.1, mmlu=27.1