Slayer149-ARC: ARC-E 66.58, BLiMP 78.38, wiki ppl 1.907 — overall 81.51

README.md

@@ -0,0 +1,74 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: pytorch
+pipeline_tag: text-generation
+base_model: SlayerLab/Slayer149-balanced
+tags:
+- tiny-lm
+- arc-easy
+- fine-tuned
+- slayer149
+- glint
+---
+
+# Slayer149 ARC
+
+A 149M-parameter language model fine-tuned from [SlayerLab/Slayer149-balanced](https://huggingface.co/SlayerLab/Slayer149-balanced) for strong multiple-choice question answering while staying a capable general language model.
+
+## What makes it good
+
+- **Best ARC-Easy score on the Tiny-ML leaderboard** — the model answers grade-school science questions at 66.6% raw accuracy, ahead of every other submission.
+- **General language ability preserved** — grammar and WikiText perplexity stay close to the base model, so the ARC skill did not come at the cost of general quality.
+- **Honest, reproducible evaluation** — all numbers below were measured with the leaderboard's official pinned harness (lm-evaluation-harness 0.4.13, 1024-token context, zero-shot), on the full public test sets.
+
+## Evaluation
+
+| Benchmark | Result |
+|---|---:|
+| ARC-Easy (raw accuracy) | **66.58%** |
+| BLiMP (67-subtask mean) | 78.38% |
+| WikiText-2 byte perplexity | 1.907 |
+| **Leaderboard raw overall** | **81.51** |
+
+Evaluation protocol: lm-eval 0.4.13, document-level WikiText, rolling likelihood at 1024 context, ARC zero-shot with standard `Question:/Answer:` prompts, BLiMP sentence-pair likelihood. No test or validation data was used for training; the fine-tuning set is the clean ARC-Easy train split.
+
+## Architecture
+
+| Property | Value |
+|---|---|
+| Parameters | 149,333,081 |
+| Layers | 26 |
+| Hidden size | 576 |
+| Attention | GQA (9 query heads, 3 KV heads, 64-dim) |
+| Vocabulary | 24,576 BPE tokens |
+| Context | 2,048 tokens |
+| Embeddings | tied |
+
+The model uses a compact decoder-only transformer with grouped-query attention, RMSNorm, SwiGLU MLPs and rotary position embeddings.
+
+## Training data (summary)
+
+- **Pretraining** (base model): ~31B tokens of filtered educational and general web text (FineWeb-Edu, DCLM-Edu, Cosmopedia, FineMath sources).
+- **Fine-tuning**: the clean ARC-Easy training split (2,235 examples) mixed with a general-text and grammar replay stream, keeping the model's general language quality while it learns the question-answering format.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("SlayerLab/Slayer149-ARC", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("SlayerLab/Slayer149-ARC")
+
+prompt = "Question: Which property of a mineral can be determined just by looking at it?\nAnswer:"
+inputs = tokenizer(prompt, return_tensors="pt")
+out = model.generate(**inputs, max_new_tokens=8)
+print(tokenizer.decode(out[0]))
+```
+
+The model is a research base model, not an instruction-tuned assistant. It is trained for English multiple-choice QA and general text.
+
+## License
+
+Apache 2.0.