File size: 3,325 Bytes
e0158d6 4fa87c5 e0158d6 4fa87c5 e1a131b 0e6793e 4fa87c5 e0158d6 0e6793e e0158d6 0e6793e e0158d6 4fa87c5 e0158d6 4fa87c5 e0158d6 4fa87c5 e0158d6 4fa87c5 e0158d6 0e6793e e0158d6 0e6793e e0158d6 4fa87c5 e0158d6 4fa87c5 e0158d6 4fa87c5 e0158d6 4fa87c5 e0158d6 4fa87c5 e0158d6 4fa87c5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
base_model: SlayerLab/Slayer149-balanced
tags:
- tiny-lm
- arc-easy
- fine-tuned
- slayer149
- glint
- fabryka-ai
---
<div align="center">
<img src="fabryka-banner.svg" alt="Fabryka AI" width="480">
<h1>Slayer149-ARC</h1>
<p><strong>Open models. Measured output.</strong></p>
<p>A 149M-parameter language model from <a href="https://fabryka.ai">Fabryka AI</a> — fine-tuned from the <a href="https://huggingface.co/SlayerLab/Slayer149-balanced">Slayer149-balanced</a> base for strong multiple-choice question answering, without losing the general language ability that makes a model actually useful.</p>
</div>
---
<div align="center">
| | |
|:---|:---|
| **ARC-Easy** | `66.58%` — best on the board |
| **BLiMP** | `78.38%` |
| **WikiText-2 byte PPL** | `1.907` |
| **Raw overall** | `81.51` — **#1** |
| **Efficiency** | `81.52` — **#1** |
</div>
> Intelligence, in production.
## Why it matters
- **Highest raw and efficiency score on the Tiny-ML leaderboard.** 81.51 raw overall — every benchmark measured with the board's own pinned harness, on full public test sets, zero shortcuts.
- **Best ARC-Easy result on the board: 66.58%.** The model answers grade-school science questions better than any other submission, at any size.
- **A factory, not a black box.** Base model, tokenizer, and evaluation protocol are all public. Numbers you can reproduce.
## Measured output
Evaluation protocol: lm-evaluation-harness 0.4.13, document-level WikiText, rolling likelihood at 1024-token context, ARC-Easy zero-shot, BLiMP sentence-pair likelihood. No test or validation data was used in training; the fine-tuning set is the clean ARC-Easy train split only.
## Architecture
| Property | Value |
|---|---:|
| Parameters | 149,333,081 |
| Layers | 26 |
| Hidden size | 576 |
| Attention | GQA (9 query / 3 KV heads, 64-dim) |
| Vocabulary | 24,576 BPE tokens |
| Context | 2,048 tokens |
| Embeddings | tied |
Compact decoder-only transformer with grouped-query attention, RMSNorm, SwiGLU MLPs and rotary position embeddings.
## Training
- **Pretraining** (base model): ~31B tokens of filtered educational and general web text — FineWeb-Edu, DCLM-Edu, Cosmopedia, FineMath.
- **Fine-tuning**: the clean ARC-Easy training split (2,235 examples) mixed with a general-text and grammar replay stream, so the model learns the question-answering format while keeping its general language quality.
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("SlayerLab/Slayer149-ARC", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("SlayerLab/Slayer149-ARC")
prompt = "Question: Which property of a mineral can be determined just by looking at it?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=8)
print(tokenizer.decode(out[0]))
```
A research base model, not an instruction-tuned assistant. Trained for English multiple-choice QA and general text.
---
<div align="center">
<sub><b>Fabryka AI</b> · Warsaw · <a href="https://fabryka.ai">fabryka.ai</a></sub><br>
<sub>Open models ✳ Real machines ✳ Measured output ✳ Independent thinking</sub>
</div>
|