Blame for README.md

---
license: cc-by-sa-4.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- tiny-lm
- arc-easy
- fine-tuned
datasets:
- allenai/qasc
- allenai/cosmos_qa
- allenai/quoref
- allenai/ropes
- tau/commonsense_qa
- rajpurkar/squad
- ucinlp/drop
- google/boolq
- allenai/ai2_arc
---

# Slayer9-DQA-9M

An 8,786,185-parameter decoder-only language model by Fabryka AI (Arkadiusz Słota), trained from scratch, annealed
with a question-answering mix and fine-tuned on ARC train. It is a research artifact for the Glint Tiny-ML leaderboard,
not a general-purpose or instruction-following model.

**Name.** *Slayer* is the SlayerLab model line; *9* is the ~9M-parameter family of this line; *DQA* stands for
decay-phase QA: question-answering data (reading passages with their questions, and short multiple-choice questions)
placed in the decay (annealing) phase of training, which is what distinguishes this recipe from our baseline; this
release uses the licence-clean QA pool listed below. *9M* is the size (8.79M unique parameters, the tied input/output
embedding counted once).

## Evaluation

**Status: test split read once for the selected checkpoint (seed 1373, see below). Weights are not uploaded yet.**

### Selected checkpoint (test, read once)

| metric | Glint-1.3 protocol | lm-evaluation-harness 0.4.13 (second column) |
|---|---|---|
| BLiMP (67 tasks, 67,000 pairs) | 73.83 | not measured |
| ARC-Easy test (2,376 questions) | 44.19 | not measured |
| WikiText-2 test, byte perplexity | 2.3971 | not measured |
| Overall / Efficiency (leaderboard formulas) | 71.16 / 79.60 | — |

For comparison, the leaderboard row of Qielbas Tiny 9M M2 (as of 2026-10-12) is BLiMP 75.68, ARC-Easy 47.43,
WikiText-2 2.3367, Efficiency 81.42; this model does not beat it.

**Validation overestimated this model.** Its validation Efficiency was 81.88; on test it is 79.60. Two causes, both
visible in the numbers: ARC-Easy 46.84 (validation) vs 44.19 (test), expected from choosing the best of fourteen
variants on validation; and BLiMP 77.24 on the validation half (first 34 tasks) vs 73.83 on all 67 tasks. For Qielbas
M2 the same two BLiMP numbers are 77.65 on our validation half A and 75.68 on the leaderboard (all 67 tasks), so the
half-A validation axis overestimates this recipe more than it overestimates M2.

Glint-1.3: zero-shot raw summed log-likelihood of the answer, first 256 tokens, raw accuracy; the protocol adds no
BOS of its own (the tokenizer's template does, see Model). lm-evaluation-harness uses a different prompt
(`Question: …\nAnswer:`) and reports `acc`; the two columns are not comparable as identical measurements. The
leaderboard row uses the Glint-1.3 column.

### Runs so far (validation split, descriptive)

Validation axis: ARC-Easy validation (570 questions), BLiMP half A (first 34 of the 67 BLiMP tasks, 34,000 pairs) and
WikiText-2 validation byte perplexity; the test axis is ARC-Easy test, full BLiMP and WikiText-2 test. Mean ± SD over
seeds. The baseline recipe is identical except that its annealing contains no question-answering pool.

| stage | n seeds | ARC-E val | BLiMP-A | WikiText-2 val byte ppl | Efficiency (val) |
|---|---|---|---|---|---|
| baseline recipe, pretrained | 7 | 40.37 ± 1.21 | 77.22 ± 0.60 | 2.3660 ± 0.0038 | 79.53 ± 0.56 |
| this recipe (QA annealing), pretrained | 7 | 40.40 ± 2.05 | 78.58 ± 0.63 | 2.3662 ± 0.0043 | 80.05 ± 0.82 |
| baseline recipe + ARC fine-tune | 7 | 42.96 ± 1.68 | 76.83 ± 0.57 | 2.3803 ± 0.0041 | 80.31 ± 0.74 |
| this recipe + ARC fine-tune | 7 | 43.58 ± 1.59 | 78.10 ± 0.57 | 2.3811 ± 0.0026 | 81.01 ± 0.67 |
| **Δ (this − baseline) after ARC fine-tune, paired by seed** | 7 | +0.62 ± 2.72 (5/7 > 0) | +1.27 ± 0.99 (7/7) | +0.001 ± 0.004 | **+0.70 ± 1.15 (5/7)** |
| this recipe + ARC fine-tune + 1 extra ARC-Easy epoch | 7 | 44.61 ± 1.64 | 77.95 ± 0.53 | 2.3874 ± 0.0037 | 81.32 ± 0.61 |
| **Δ (extra ARC-Easy epoch − ARC fine-tune), paired by seed** | 7 | | | | **+0.31 ± 0.39 (5/7)** |

Validation Efficiency of the seven QA-annealed models after the ARC fine-tune (seeds 1369–1375): 80.93 / 81.07 /
79.80 / 80.62 / 81.20 / 81.76 / 81.72; after the extra ARC-Easy epoch: 81.69 / 81.56 / 80.26 / 80.69 / 81.88 /
81.75 / 81.44.

**How the checkpoint was chosen.** Seven training seeds (1369–1375) were run with the same recipe. Rule (1) was fixed
before any result was known: take the seed with the highest leaderboard Efficiency on the validation axis (Efficiency
formula on ARC-Easy validation, BLiMP half A and WikiText-2 validation byte perplexity) after the ARC fine-tune
(ties → lower seed). Rule (2) was added at the owner's request after the pre-fine-tune validation results were known,
but before any fine-tuned or extra-epoch result was known: one extra epoch on ARC-Easy train only replaces the
fine-tune for all seeds if it improves validation Efficiency by at least +0.20 on average and for at least 5 of 7
seeds, in which case the highest of the seven extended models is taken. Rule (2) was met (+0.31, 5/7), so the selected
checkpoint is seed 1373 with the extra epoch (validation Efficiency 81.88). The test-axis evaluation was run **once**,
for that checkpoint only; the test splits were never used for training, fine-tuning, epoch choice or seed choice. The
test split was read without a separate confirmation run on fresh seeds (owner's decision). Choosing the best of
fourteen variants on validation inflated its validation number; the test numbers above are the result.

## Model

- 10 layers, d_model 224, 4 heads, RoPE (θ 100,000, interleaved), SwiGLU (×2.667), RMSNorm, QK-norm,
  value residual; output head tied to the token embedding (stored once). Context 1,024 tokens.
- Tokenizer: BPE, 12,288 tokens (`tokenizer.json`, sha256 `235abda6ba9591b7…`). Its template prepends `<|endoftext|>`
  (id 12285) to every encoded text: in training every document was preceded by this separator, so evaluation starts
  the same way. Training data were tokenized without the template (the separator came from the document stream); all
  reported scores use the tokenizer as shipped (default `encode()`).
- Files (after selection): `model.safetensors` (fp32 weights only), `config.json`, `tokenizer.json`, inference code,
  `README.md`. Loading reproduces the logits of the training checkpoint exactly.

## Training

1. **Pretraining + annealing, 2.0B tokens total** (8,720 steps × 229,376 tokens). Pretraining mix: a subset of
   `SlayerLab/minimal-en-corpus-5b` (by Dawid Majewski) restricted to FineWeb-Edu, DCLM-baseline and Wikipedia
   documents (about 47 / 36 / 16 percent of tokens; 2.29B tokens available, ~0.87 epoch used); documents overlapping
   WikiText-2 test/validation or ARC-Easy/Challenge test/validation/train were removed before selection (13-gram scan;
   BLiMP was not scanned).
2. **QA annealing** (final 1,744 steps, 400M tokens): 19.0 percent from a question-answering pool (each passage once,
   followed by all of its questions and answers; 4 passes over 18.99M unique tokens), 15.0 percent WikiText-103 train
   (documents overlapping the union of evaluation test sets removed by a 13-gram filter), 66.0 percent the pretraining
   mix. Pool sources (train splits only): QASC, Cosmos QA, Quoref, ROPES (CC BY 4.0), CommonsenseQA (MIT), SQuAD v1.1,
   DROP (CC BY-SA 4.0), BoolQ (CC BY-SA 3.0). Questions that appear in ARC train were removed; 72 documents overlapping
   WikiText-2 test/validation or ARC-Easy/Challenge test/validation/train were removed by a 13-gram scan (BLiMP was
   not scanned).
3. **ARC fine-tune**: ARC-Easy + ARC-Challenge train (CC BY-SA 4.0; questions overlapping ARC test/validation
   removed), listwise cross-entropy over raw answer likelihoods, 2 epochs (fixed, not selected), AdamW lr 5e-5
   (constant), 15 percent language-model replay from the annealing mix; then **one extra epoch on ARC-Easy train only**
   with the same settings (adopted by rule (2) above).

## Licence and attribution

Weights: CC BY-SA 4.0 (share-alike sources in training: Wikipedia and WikiText-103 (CC BY-SA 3.0 + GFDL), ARC,
SQuAD, DROP (CC BY-SA 4.0), BoolQ (CC BY-SA 3.0)). Other sources: FineWeb-Edu (ODC-By), DCLM-baseline (CC BY 4.0),
QASC, Cosmos QA, Quoref, ROPES (CC BY 4.0), CommonsenseQA (MIT). The pretraining corpus keeps each document's upstream
source and terms.

Sources (repository @ revision, licence): QASC `allenai/qasc`@a34ba204 CC BY 4.0 · Cosmos QA `allenai/cosmos_qa`@ed50fe83
(parquet conversion) CC BY 4.0 · Quoref `allenai/quoref`@41fed5bc CC BY 4.0 · ROPES `allenai/ropes`@d59f1e2e CC BY 4.0 ·
CommonsenseQA `tau/commonsense_qa`@94630fe3 MIT · SQuAD `rajpurkar/squad`@7b6d24c4 CC BY-SA 4.0 · DROP
`ucinlp/drop`@95cda593 CC BY-SA 4.0 · BoolQ `google/boolq`@35b264d0 CC BY-SA 3.0 · ARC `allenai/ai2_arc`@210d026f
CC BY-SA 4.0 · pretraining corpus `SlayerLab/minimal-en-corpus-5b`@bb3aec23, built from FineWeb-Edu
`HuggingFaceFW/fineweb-edu`@87f09149 ODC-By, DCLM-baseline `mlfoundations/dclm-baseline-1.0-parquet`@817d6752 CC BY 4.0
and Wikipedia `wikimedia/wikipedia`@b04c8d1c (20231101.en) CC BY-SA 3.0 + GFDL · WikiText-103
`Salesforce/wikitext`@b08601e0 CC BY-SA 3.0 + GFDL.

## Limitations

Very small model (8.8M parameters); trained on the leaderboard's ARC task (in-domain fine-tune, described above);
one selected seed out of seven.