dawidkielbasa/qielbas-tiny-9m-m2.5

Qielbas Tiny 9M M2.5

A 9,557,716-parameter English causal language model, trained and evaluated entirely on Labqoat.

Changes

M2.5 extends the original 6.7B-token Muon base to 20B token exposures at peak LR 0.001, preserves its model and optimizer state during that extension, and uses the validation-selected FP32 whole-window average. It then applies the M2 300M-token Wiki15 continuation and anchored ARC fine-tuning. The anchor adds lambda/2 times the summed squared distance to the fixed pre-ARC checkpoint, with lambda 0.001 and all parameters trainable. Adaptation seed 123, epoch 4 was selected by held-out ARC-Easy accuracy.

M2 used the original 6.7B-token base, 300M Wiki15 continuation and unanchored ARC adaptation. M2.5 and M3 retain the same 9,557,716-parameter architecture and 4096-token tokenizer.

Architecture

Four GatedDeltaNet2 blocks and one causal attention block, width 320, SwiGLU hidden size 1048, RMSNorm, RoPE, tied embeddings and context length 256. The native PyTorch loader and frozen benchmark implementation are included.

Training data and recipe

Base training uses the same cached 6,699,999,744-token stream: FineWeb-Edu and Cosmopedia v2, FinePhrase FAQ/Tutorial, DCLM-Edu, FinePDFs-Edu English and FineWiki English. Training to 20B reuses this stream in shuffled passes; 20B describes token exposures, not 20B unique data tokens.

The final continuation adds exactly 300,000,000 tokens: 44,999,936 exposures from audited WikiText-103 raw training text and 255,000,064 from the original mixture. The audited WikiText pool contains 176,056,863 source tokens. The WikiText continuation pool excludes WikiText-2 and WikiText-103 validation/test articles by title and exact text. Total base plus continuation exposure: 20.3B tokens.

Continuation: fresh Muon plus AdamW, seed 7, warmup 100 updates, peak LR 0.0001 with cosine decay to 0.00001, batch 128, context 256 and BF16 on L40S. The final continuation checkpoint is used; its validation measurements are reporting only.

ARC fine-tuning: 3,345 Easy/Challenge training questions, 867 held-out validation questions, five epochs, fresh AdamW, QA batch 16, text replay batch 32 x 256, peak LR 0.00005, warmup 20, cosine to 10% floor, weight decay 0.01. The loss is 0.15 choice CE + 0.85 replay CE plus the lambda 0.001 anchor penalty. Replay includes 15% of the same audited WikiText training pool. Epoch selection uses ARC-Easy held-out accuracy, ties Easy choice CE then earlier epoch. Benchmark tests do not train or select checkpoints/epochs. Both adaptation seeds 7 and 123 were evaluated; this release contains the higher-scoring seed-123 model.

Frozen evaluation

Metric	M2	M2.5	M3
Efficiency	81.416994	82.266717	82.370639
ARC-Easy accuracy	47.432660%	49.200337%	48.947811%
BLiMP accuracy	75.683582%	76.065672%	76.553731%
WikiText-2 byte perplexity	2.336740	2.318994	2.313287

This release's exact Efficiency is 82.26671669168255. Its raw Overall is 73.77442927745844.

Same frozen Glint-1.3 scoring implementation and retained Tiny-ML normalization: context 256, no BOS, unnormalized answer likelihood, 2,376 ARC-Easy questions, 67,000 BLiMP pairs, 423,910 WikiText tokens and 1,292,008 bytes. These are self-reported full evaluations; separate official-harness or Fabryka results are not mixed into this table. Evaluation report and per-example predictions are included.

Both adaptation seeds: 81.519144 / 82.266717 Efficiency for seeds 7 / 123. One base pretraining seed per arm was used. The 0.103922-point best-score difference between M2.5 and M3 is within our working 0.2-point noise heuristic, which is not a statistical significance test.

Retained provenance

Published weights SHA-256: 7c849fd261c5ab9a082505d018aa87b96d2505e31ba67751ba18f3c8834cb65a. Selected adaptation epoch: 4. Base pretraining seed: 7; continuation seed: 7; adaptation seed: 123. Reports for averaging, continuation, fine-tuning and evaluation, plus provenance.json, preserve the exact retained references. Complete training snapshots and every adaptation epoch are retained on Labqoat.

Usage

Requires a BF16-capable CUDA GPU and the pinned dependencies.

pip install -r requirements.txt

Download the repository, then use the native loader:

import torch
from load_model import load_model

model, tokenizer = load_model(".")
inputs = torch.tensor([tokenizer.encode("The Earth revolves around").ids], device="cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    next_token = model(inputs)[0, -1].argmax().item()
print(tokenizer.decode([next_token]))