README.md

---
language:
- en
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- text-generation
- causal-lm
- custom-architecture
- custom_code
- slm
- small-language-model
datasets:
- agentlans/high-quality-english-sentences
- armanc/ScienceQA
- AxiomicLabs/FactBase-DCLM
- HuggingFaceFW/finephrase
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/cosmopedia
- HuggingFaceTB/finemath
- HuggingFaceTB/smollm-corpus
- IFM/Pretrain-Behaviors
- IFM/TxT360-v2
- microsoft/orca-math-word-problems-200k
- nampdn-ai/tiny-strange-textbooks
- nvidia/OpenMathInstruct-2
- openbmb/UltraData-Math
- sentence-transformers/eli5
- sentence-transformers/gooaq
- sentence-transformers/natural-questions
model-index:
- name: Ethoric-10M
  results:
    - task:
        type: text-generation
      dataset:
        type: allenai/ai2_arc
        name: ARC-Easy
        config: ARC-Easy
        split: test
      metrics:
        - type: acc_norm
          value: 37.58
          name: acc_norm (0-shot)
    - task:
        type: text-generation
      dataset:
        type: allenai/ai2_arc
        name: ARC-Challenge
        config: ARC-Challenge
        split: test
      metrics:
        - type: acc_norm
          value: 24.66
          name: acc_norm (0-shot)
    - task:
        type: text-generation
      dataset:
        type: Rowan/hellaswag
        name: HellaSwag
        split: validation
      metrics:
        - type: acc_norm
          value: 28.75
          name: acc_norm (0-shot)
    - task:
        type: text-generation
      dataset:
        type: ybisk/piqa
        name: PIQA
        split: validation
      metrics:
        - type: acc_norm
          value: 59.85
          name: acc_norm (0-shot)
    - task:
        type: text-generation
      dataset:
        type: AxiomicLabs/Arithmark-3.0
        name: ArithMark-3
        split: test
      metrics:
        - type: acc_norm
          value: 41.60
          name: acc_norm (0-shot)
---
# Ethoric-10M

## Model Summary

Ethoric-10M is a decoder-only small language model with 9.97M parameters. It is built on a modern transformer foundation of RoPE, grouped-query attention and SwiGLU, and adds four changes on top. Peri-LN normalization applies RMSNorm both before and after every sublayer to keep training stable. Canon layers are short causal depthwise convolutions that mix each token with its 3 previous tokens before attention and before the MLP. Per-head QK-norm normalizes the queries and keys in each attention head, and a headwise sigmoid gate lets each head control how much of its output passes through. The model was trained on 15B tokens, followed by a 4B-token cooldown.

## Model Architecture

| Parameter | Value |
|---|---|
| Architecture | Decoder-only transformer (RMSNorm, RoPE, GQA, SwiGLU) |
| Layers | 12 |
| Hidden size (d_model) | 256 |
| Attention heads | 8 |
| KV heads (GQA) | 2 |
| Head dim | 32 |
| FFN intermediate size | 721 |
| Normalization | Peri-LN (RMSNorm, ε = 1e-5) + per-head QK-norm |
| Attention gate | Headwise sigmoid gate |
| Canon layers | Causal depthwise conv, kernel 4 (before attention and MLP) |
| Local attention | 16-token window in layer 0; full causal in layers 1–11 |
| RoPE theta | 10,000 |
| Max sequence length | 512 |
| Vocabulary size | 5,048 |
| Tie word embeddings | Yes |
| Precision | FP32 weights |
| Parameters | 9.97M |

## How to use
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "techdotus/Ethoric-10M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

ids = tok("The climate change", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=32, do_sample=False,
                         eos_token_id=1)  # greedy
print(tok.decode(out[0], skip_special_tokens=True))
```

The model's custom `generate()` supports greedy decoding and top-k sampling (`do_sample=True`, `temperature`, `top_k`). Other options such as `repetition_penalty` are ignored. Prompt plus generated tokens must stay within 512 tokens.

## Benchmark Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology.

| HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-3 |
|---|---|---|---|---|
| 28.75% | 37.58% | 24.66% | 59.85% | 41.60% |

## Comparison Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology. Scores for the other models are from the Open SLM Leaderboard. Bold marks the best result in each column.

| Model | Params | Int Index | HellaSwag | ARC-Easy | ARC-Chal | PIQA | ArithMark-3 |
|---|---|---|---|---|---|---|---|
| Ethoric-10M (Tech.us) | 9.97M | **12.94** | 28.75% | 37.58% | **24.66%** | **59.85%** | **41.60%** |
| ForgePlex-M2.2-10M | 9.99M | 12.12 | **29.19%** | **39.77%** | 22.70% | 59.79% | 37.40% |
| heron1i-flash-lite-v2 | 9.92M | 11.11 | 28.86% | 38.89% | 23.38% | 58.54% | 36.70% |
| min-spark-1.6 | 9.96M×4* | 11.08 | 28.52% | 38.30% | 22.44% | 59.14% | 36.90% |

## Training Details

Ethoric-10M was trained in two stages.

| Stage | Tokens | Focus |
|---|---|---|
| Stable | 15B | Broad mix of educational web text, synthetic textbooks, question-answer data and math |
| Cooldown | 4B | Learning rate decayed; mix shifted toward higher-quality math, QA and tutorial-style text |

Training packs multiple documents into each 512-token sequence. Document masking keeps them separate: attention and Canon convolutions never reach across an EOS token.

## Training Data

The cooldown added new math sources such as FineMath, more question-answer and tutorial-style text such as FinePhrase, ELI5 and GooAQ, and dropped Tiny Strange Textbooks, High-Quality English Sentences and ScienceQA.

| Source | Stage 1 | Stage 2 |
|---|---|---|
| FineWeb-Edu | 35% | 25% |
| TxT360-v2 | 20% | 21% |
| Cosmopedia | 19.5% | 17% |
| Tiny Strange Textbooks | 11% | 0% |
| OpenMathInstruct-2 | 11% | 7% |
| High-Quality English Sentences | 2% | 0% |
| ScienceQA | 1.0% | 0% |
| Orca-Math Word Problems 200k | 0.5% | 5% |
| FineMath | 0% | 11% |
| FinePhrase | 0% | 11% |
| Miscellaneous† | 0% | 3% |

† ELI5, GooAQ, FactBase-DCLM, Cosmopedia-v2, Natural Questions, Pretrain-Behaviors, UltraData-Math

## Limitations

- **Tiny model:** with 9.97M parameters and 256-dim hidden states, generations are often repetitive, incoherent or factually wrong. The model is a research artifact for studying small-scale LMs, not an assistant.
- **Short context:** 512 tokens maximum. RoPE tables are not built beyond that length, and the small 5,048-token vocabulary means 512 tokens covers less text than in models with larger vocabularies.
- **English only:** trained on English web, educational, synthetic, question-answer and math text.
- **Not instruction-tuned or safety-aligned:** it may reproduce biases present in the data.
- **Benchmark scores are near chance on some tasks:** ARC-Challenge (24.66%) and HellaSwag (28.75%) are close to the 25% random baseline, so small differences between models on these tasks should be read with caution.

## License

MIT

## Citation

```bibtex
@misc{ethoric10m2026,
  title        = {{Ethoric-10M}: A Small Language Model with Stabilized,
                  Gated Attention},
  author       = {{Tech.us Team}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/techdotus/Ethoric-10M}}
}
```