---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
base_model: SlayerLab/Slayer149-balanced
tags:
- tiny-lm
- arc-easy
- fine-tuned
- slayer149
- glint
- fabryka-ai
---
<div align="center">
  <img src="fabryka-banner.svg" alt="Fabryka AI" width="480">

  <h1>Slayer149-ARC</h1>

  <p><strong>Open models. Measured output.</strong></p>
  <p>A 149M-parameter language model from <a href="https://fabryka.ai">Fabryka AI</a> — fine-tuned from the <a href="https://huggingface.co/SlayerLab/Slayer149-balanced">Slayer149-balanced</a> base for strong multiple-choice question answering, without losing the general language ability that makes a model actually useful.</p>
</div>

---

<div align="center">

| | |
|:---|:---|
| **ARC-Easy** | `66.58%` — best on the board |
| **BLiMP** | `78.38%` |
| **WikiText-2 byte PPL** | `1.907` |
| **Raw overall** | `81.51` — **#1** |
| **Efficiency** | `81.52` — **#1** |

</div>

> Intelligence, in production.

## Why it matters

- **Highest raw and efficiency score on the Tiny-ML leaderboard.** 81.51 raw overall — every benchmark measured with the board's own pinned harness, on full public test sets, zero shortcuts.
- **Best ARC-Easy result on the board: 66.58%.** The model answers grade-school science questions better than any other submission, at any size.
- **A factory, not a black box.** Base model, tokenizer, and evaluation protocol are all public. Numbers you can reproduce.

## Measured output

Evaluation protocol: lm-evaluation-harness 0.4.13, document-level WikiText, rolling likelihood at 1024-token context, ARC-Easy zero-shot, BLiMP sentence-pair likelihood. No test or validation data was used in training; the fine-tuning set is the clean ARC-Easy train split only.

## Architecture

| Property | Value |
|---|---:|
| Parameters | 149,333,081 |
| Layers | 26 |
| Hidden size | 576 |
| Attention | GQA (9 query / 3 KV heads, 64-dim) |
| Vocabulary | 24,576 BPE tokens |
| Context | 2,048 tokens |
| Embeddings | tied |

Compact decoder-only transformer with grouped-query attention, RMSNorm, SwiGLU MLPs and rotary position embeddings.

## Training

- **Pretraining** (base model): ~31B tokens of filtered educational and general web text — FineWeb-Edu, DCLM-Edu, Cosmopedia, FineMath.
- **Fine-tuning**: the clean ARC-Easy training split (2,235 examples) mixed with a general-text and grammar replay stream, so the model learns the question-answering format while keeping its general language quality.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("SlayerLab/Slayer149-ARC", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("SlayerLab/Slayer149-ARC")

prompt = "Question: Which property of a mineral can be determined just by looking at it?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=8)
print(tokenizer.decode(out[0]))
```

A research base model, not an instruction-tuned assistant. Trained for English multiple-choice QA and general text.

---

<div align="center">
  <sub><b>Fabryka AI</b> · Warsaw · <a href="https://fabryka.ai">fabryka.ai</a></sub><br>
  <sub>Open models ✳ Real machines ✳ Measured output ✳ Independent thinking</sub>
</div>