Qielbas Tiny 6M
Base language model with 6,112,272 parameters, trained from scratch on 3B tokens: 90% FineWeb-Edu and 10% Cosmopedia. Seed 7.
Four GatedDeltaNet2 blocks and one causal attention block; width 256, 4,096-token BPE vocabulary, 256-token context, tied embeddings.
Results
| Benchmark | Score |
|---|---|
| BLiMP | 73.2493% |
| ARC-Easy | 36.2374% |
| WikiText-2 byte perplexity | 3.06862 |
Full evaluation: 67,000 BLiMP pairs, 2,376 ARC questions and 1,292,008 WikiText-2 bytes. BF16 forwards, FP32 log probabilities; unnormalized likelihood scoring, no BOS, 256-token context. WikiText rows are joined with spaces. Scores are self-reported.
Evaluation report · Training report · Predictions
Usage
Requires CUDA. Install requirements.txt and load the native PyTorch checkpoint:
from load_model import load_model
model, tokenizer = load_model(".")
For benchmark reproduction, install requirements-benchmark.txt:
python benchmark.py --model-dir .
The standalone dataset adapter has not yet been rerun.
Release: v1. Future checkpoints use new release tags.