README.md

---
language:
- en
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: text-generation
tags:
- text-generation
- causal-lm
- custom-architecture
- slm
- small-language-model
- spab
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/cosmopedia
- agentlans/high-quality-english-sentences
- nampdn-ai/tiny-strange-textbooks
- armanc/ScienceQA
- nvidia/OpenMathInstruct-2
- microsoft/orca-math-word-problems-200k
---
# Tokle-SPAB-3M

## Model Summary

Tokle-SPAB-3M is a decoder-only language model trained on 12B tokens, with 2.91M trainable parameters and 8.39M frozen SPAB parameters, for a total of 11.3M parameters. Its main architectural addition is SPAB (Static Pairwise Attention Bias), a frozen table of token-pair association scores built from Pointwise Mutual Information (PMI) over the training corpus and added to the attention logits.

For every query-key pair, SPAB hashes the two token IDs into the table, pulls
out their PMI value, multiplies it by a learned per-head scale, and adds it to
the attention logits before softmax. The bias ignores position and depends only
on which tokens are involved, so the model starts training already knowing
which tokens tend to co-occur. It only has to learn how much to trust that
prior.

## Model Architecture

| Parameter | Value |
| --- | --- |
| Architecture | Custom decoder-only transformer + SPAB |
| Layers | 9 |
| Hidden size (d_model) | 144 |
| Attention heads | 3 |
| KV heads (GQA) | 1 (multi-query attention) |
| Head dim | 48 |
| FFN intermediate size | 432 |
| Max sequence length | 512 |
| Trainable parameters | 2,908,947 |
| Frozen SPAB table | 8,388,608 (float32 buffer) |
## How to use
This model uses a custom architecture, so it needs `trust_remote_code=True`.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "techdotus/Tokle-SPAB-3M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

ids = tok("The climate change", return_tensors="pt")
with torch.no_grad():
    out = model.generate(**ids, max_new_tokens=32, do_sample=False, repetition_penalty=1.3)  # greedy
print(tok.decode(out[0], skip_special_tokens=True))
```
## Benchmark Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology:
| Hellaswag | ARC-Easy | ARC-Challenge | PIQA | Arithmark-3 |
| --------- | -------- | ------------- | ---- | ----------- |
| 27.22% | 34.68% | 24.49% | 54.95% | 41.70% |
<!-- ## Comparison Results
All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology. 
Scores for the other models are from the Open SLM Leaderboard. **Bold** marks 
the best result in each column.

| Model | Params | Int Index | HellaSwag | ARC-Easy | ARC-Chal | PIQA | ArithMark-3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **Tokle-SPAB-3M** (Tech.us) | 2.9M | **9.16** | 27.22% | **34.68%** | **24.49%** | 54.95% | **41.70%** |
| Ember-2 (SurjoLabs) | 2.96Mx2 | 7.21 | 27.28% | 33.42% | 22.01% | 55.11% | 35.90% |
| BananaMind-2-Micro (BananaMind) | 2.9M | 6.01 | **28.27%** | 33.12% | 21.93% | 53.21% | 34.00% |
| GPT-S-1.4M (Axiomic Labs) | 1.4M | 5.40 | 26.89% | 31.57% | 21.93% | **55.17%** | 30.20% |
 -->
## Training Data Details

We trained on a curated mixture with a strict cleaning pipeline that also 
removed topics not useful for a model of this size.

| Source | Percentage |
| --- | --- |
| FineWeb-Edu | 43.1% |
| Cosmopedia | 24.3% |
| OpenMathInstruct-2 | 13.5% |
| Tiny Strange Textbooks | 9.0% |
| MegaScience (medicine & biology, custom curated) | 5.0% |
| High-Quality English Sentences | 3.0% |
| ScienceQA | 1.2% |
| Orca-Math Word Problems 200k | 0.9% |
| Total | 100% |

- **Tokenizer:** all data was tokenized with the model's 5,048-token BPE
  tokenizer, and 1% was held out for validation.
- **Blending:** sources were blended per dataset using the weights above.

## Limitations

- **Tiny model:** with ~2.9M trainable parameters, ~8.39M frozen SPAB
  parameters and 144-dim hidden states, generations are often repetitive,
  incoherent or factually wrong. The model is a research artifact for studying
  small-scale LMs, not an assistant.
- **Short context:** 512 tokens maximum. RoPE tables are not built beyond that
  length.
- **English only:** trained on English web, educational, synthetic and math text.
- **Not instruction-tuned or safety-aligned:** it may reproduce biases present
  in web data.

## Licenses

**Code**: MIT. The modeling code, tokenizer, and training scripts are released under the MIT license.

**Weights**: CC BY-NC-SA 4.0. The training data includes MegaScience (CC BY-NC-SA 4.0), so the weights are released under the same terms: attribution required, non-commercial use only, and derivatives (including fine-tunes) must be shared under the same license.

## Citation

```bibtex
@misc{tokle2026,
  title        = {{Tokle-SPAB-3M}: Pointwise Mutual Information as an Inductive Bias for Self-Attention},
  author       = {{Tech.us Team}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/techdotus/Tokle-SPAB-3M}}
}
```