dawidkielbasa/qielbas-tiny-9m-m3.1

Qielbas Tiny 9M M3.1

A 9,557,716-parameter English causal language model trained and evaluated on Labqoat. This release contains the retained PCGrad + anchor, adaptation seed 123, fixed epoch 5 checkpoint. Its frozen full-evaluation Efficiency is 82.78884212433945.

Changes from M3

The architecture, tokenizer, D scratch-20B base and 300M-token Wiki15 continuation are unchanged from M3. The ARC step uses 4,212 questions, merging the original 3,345 training questions with 867 former-validation questions. Epoch 5 is fixed before evaluation; the former-validation questions are no longer used for epoch selection.

The two weighted task gradients (0.15 ARC choice CE and 0.85 text replay CE) receive symmetric global projections when their dot product is negative, then are summed. The lambda 0.001 anchor gradient is added afterward, preserving a penalty of lambda/2 times the summed squared distance to the fixed pre-ARC source weights. This two-task implementation is based on PCGrad: Gradient Surgery for Multi-Task Learning.

This is an exploratory checkpoint release: seed 123 improves the matched control, while seed 7 does not. It does not establish a consistent recipe improvement across seeds.

Architecture and lineage

Four GatedDeltaNet2 blocks and one causal attention block, width 320, SwiGLU hidden size 1048, RMSNorm, RoPE, tied embeddings, vocabulary 4096 and context length 256. The native CUDA PyTorch loader and unchanged frozen scoring implementation are included.

The D base was trained from scratch with seed 123 to 20,000,000,000 token exposures, using the cached 6,699,999,744-token general stream in shuffled passes. The general sources are FineWeb-Edu/Cosmopedia v2, FinePhrase FAQ/Tutorial, DCLM-Edu, FinePDFs-Edu English and FineWiki English. A whole-window FP32 average was selected using general validation CE.

Continuation adds exactly 300,000,000 exposures: 44,999,936 audited WikiText-103 raw training tokens and 255,000,064 original-mixture tokens. The audited WikiText pool contains 176,056,863 tokens and excludes WikiText-2/103 validation/test articles by title and exact text. Total base-plus-continuation exposure is 20.3B, not 20.3B unique text tokens.

Continuation uses fresh Muon plus AdamW, RNG seed 7, warmup 100 updates, peak LR 1e-4 with cosine decay to 1e-5, batch 128, context 256 and BF16 on L40S. The final continuation checkpoint supplies the anchor reference and fine-tuning source.

ARC fine-tuning

Five passes over the 4,212 ARC Easy/Challenge train-plus-former-validation questions, seed 123, fresh AdamW (betas 0.9/0.95, decay 0.01), QA batch 16, replay batch 32 x 256, peak LR 5e-5, warmup 20, cosine decay to 10% floor, BF16 and clipping at 1. All weights are trainable. The fixed five epochs comprise 1,320 updates, 21,060 question exposures and 10,813,440 replay tokens. Replay uses the original six-source mixture with 15% audited WikiText training text.

No science-question augmentation, sentence-start continuation, EMA or teacher distillation is part of M3.1. Benchmark tests are used only for scoring; they do not train the model or select its epoch. This checkpoint was chosen for publication after an exploratory recipe screen, so the highest observed single score is not an independent confirmation of the recipe-selection process.

Frozen full evaluation

Metric	Published M3	M3.1
Efficiency	82.370639	82.788842
ARC-Easy accuracy	48.947811%	50.000000%
BLiMP accuracy	76.553731%	76.647761%
WikiText-2 byte perplexity	2.313287	2.316022

M3.1 raw Overall is 74.24265637286899. These are self-reported full evaluations under the retained Glint-1.3/Tiny-ML scoring and normalization: no BOS, first token unscored, context 256, raw question-plus-answer likelihood, 2,376 ARC-Easy questions, 67,000 BLiMP pairs, 423,910 WikiText tokens and 1,292,008 bytes. Official leaderboard or alternate-harness results are not implied.

Adaptation seed	PCGrad + anchor	Matched anchored train-plus-validation control	Difference
7	82.199705	82.419416	-0.219710
123 (released)	82.788842	82.588705	+0.200137

One base pretraining seed and two adaptation seeds were used. Differences below about 0.2 Efficiency points are a working noise heuristic, not a significance test. Full evaluation reports, predictions, training reports, paired scores, recipe and provenance are included. All five epoch weights and complete model/optimizer/RNG snapshots remain retained on Labqoat.

Usage

This custom architecture requires the included native loader, a BF16-capable CUDA GPU, and the pinned dependencies. It is a text-completion model, not an instruction-tuned chat assistant.

pip install -r requirements.txt

Download this repository and run:

import torch
from load_model import load_model

model, tokenizer = load_model(".")
tokens = tokenizer.encode("The Earth revolves around").ids
inputs = torch.tensor([tokens], device="cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    next_token = model(inputs)[0, -1].argmax().item()
print(tokenizer.decode([next_token]))

Weights SHA-256: 0cefc9fba70e7774447e6864aecd768eb84c27135b7c145389542b843a963d1e. Base seed 123; continuation RNG seed 7; adaptation seed 123; fixed epoch 5. Immutable asset, checkpoint and run references are recorded in provenance.json.