| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: text-generation |
| base_model: SlayerLab/Slayer149-balanced |
| tags: |
| - tiny-lm |
| - arc-easy |
| - fine-tuned |
| - slayer149 |
| - glint |
| - fabryka-ai |
| --- |
| <div align="center"> |
| <img src="fabryka-banner.svg" alt="Fabryka AI" width="480"> |
|
|
| <h1>Slayer149-ARC</h1> |
|
|
| <p><strong>Open models. Measured output.</strong></p> |
| <p>A 149M-parameter language model from <a href="https://fabryka.ai">Fabryka AI</a> — fine-tuned from the <a href="https://huggingface.co/SlayerLab/Slayer149-balanced">Slayer149-balanced</a> base for strong multiple-choice question answering, without losing the general language ability that makes a model actually useful.</p> |
| </div> |
|
|
| --- |
|
|
| <div align="center"> |
|
|
| | | | |
| |:---|:---| |
| | **ARC-Easy** | `66.58%` — best on the board | |
| | **BLiMP** | `78.38%` | |
| | **WikiText-2 byte PPL** | `1.907` | |
| | **Raw overall** | `81.51` — **#1** | |
| | **Efficiency** | `81.52` — **#1** | |
|
|
| </div> |
|
|
| > Intelligence, in production. |
|
|
| ## Why it matters |
|
|
| - **Highest raw and efficiency score on the Tiny-ML leaderboard.** 81.51 raw overall — every benchmark measured with the board's own pinned harness, on full public test sets, zero shortcuts. |
| - **Best ARC-Easy result on the board: 66.58%.** The model answers grade-school science questions better than any other submission, at any size. |
| - **A factory, not a black box.** Base model, tokenizer, and evaluation protocol are all public. Numbers you can reproduce. |
|
|
| ## Measured output |
|
|
| Evaluation protocol: lm-evaluation-harness 0.4.13, document-level WikiText, rolling likelihood at 1024-token context, ARC-Easy zero-shot, BLiMP sentence-pair likelihood. No test or validation data was used in training; the fine-tuning set is the clean ARC-Easy train split only. |
|
|
| ## Architecture |
|
|
| | Property | Value | |
| |---|---:| |
| | Parameters | 149,333,081 | |
| | Layers | 26 | |
| | Hidden size | 576 | |
| | Attention | GQA (9 query / 3 KV heads, 64-dim) | |
| | Vocabulary | 24,576 BPE tokens | |
| | Context | 2,048 tokens | |
| | Embeddings | tied | |
|
|
| Compact decoder-only transformer with grouped-query attention, RMSNorm, SwiGLU MLPs and rotary position embeddings. |
|
|
| ## Training |
|
|
| - **Pretraining** (base model): ~31B tokens of filtered educational and general web text — FineWeb-Edu, DCLM-Edu, Cosmopedia, FineMath. |
| - **Fine-tuning**: the clean ARC-Easy training split (2,235 examples) mixed with a general-text and grammar replay stream, so the model learns the question-answering format while keeping its general language quality. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("SlayerLab/Slayer149-ARC", trust_remote_code=True) |
| tokenizer = AutoTokenizer.from_pretrained("SlayerLab/Slayer149-ARC") |
| |
| prompt = "Question: Which property of a mineral can be determined just by looking at it?\nAnswer:" |
| inputs = tokenizer(prompt, return_tensors="pt") |
| out = model.generate(**inputs, max_new_tokens=8) |
| print(tokenizer.decode(out[0])) |
| ``` |
|
|
| A research base model, not an instruction-tuned assistant. Trained for English multiple-choice QA and general text. |
|
|
| --- |
|
|
| <div align="center"> |
| <sub><b>Fabryka AI</b> · Warsaw · <a href="https://fabryka.ai">fabryka.ai</a></sub><br> |
| <sub>Open models ✳ Real machines ✳ Measured output ✳ Independent thinking</sub> |
| </div> |
|
|
