| --- |
| language: |
| - en |
| license: mit |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - text-generation |
| - causal-lm |
| - custom-architecture |
| - custom_code |
| - slm |
| - small-language-model |
| datasets: |
| - agentlans/high-quality-english-sentences |
| - armanc/ScienceQA |
| - AxiomicLabs/FactBase-DCLM |
| - HuggingFaceFW/finephrase |
| - HuggingFaceFW/fineweb-edu |
| - HuggingFaceTB/cosmopedia |
| - HuggingFaceTB/finemath |
| - HuggingFaceTB/smollm-corpus |
| - IFM/Pretrain-Behaviors |
| - IFM/TxT360-v2 |
| - microsoft/orca-math-word-problems-200k |
| - nampdn-ai/tiny-strange-textbooks |
| - nvidia/OpenMathInstruct-2 |
| - openbmb/UltraData-Math |
| - sentence-transformers/eli5 |
| - sentence-transformers/gooaq |
| - sentence-transformers/natural-questions |
| model-index: |
| - name: Ethoric-10M |
| results: |
| - task: |
| type: text-generation |
| dataset: |
| type: allenai/ai2_arc |
| name: ARC-Easy |
| config: ARC-Easy |
| split: test |
| metrics: |
| - type: acc_norm |
| value: 37.58 |
| name: acc_norm (0-shot) |
| - task: |
| type: text-generation |
| dataset: |
| type: allenai/ai2_arc |
| name: ARC-Challenge |
| config: ARC-Challenge |
| split: test |
| metrics: |
| - type: acc_norm |
| value: 24.66 |
| name: acc_norm (0-shot) |
| - task: |
| type: text-generation |
| dataset: |
| type: Rowan/hellaswag |
| name: HellaSwag |
| split: validation |
| metrics: |
| - type: acc_norm |
| value: 28.75 |
| name: acc_norm (0-shot) |
| - task: |
| type: text-generation |
| dataset: |
| type: ybisk/piqa |
| name: PIQA |
| split: validation |
| metrics: |
| - type: acc_norm |
| value: 59.85 |
| name: acc_norm (0-shot) |
| - task: |
| type: text-generation |
| dataset: |
| type: AxiomicLabs/Arithmark-3.0 |
| name: ArithMark-3 |
| split: test |
| metrics: |
| - type: acc_norm |
| value: 41.60 |
| name: acc_norm (0-shot) |
| --- |
| # Ethoric-10M |
|
|
| ## Model Summary |
|
|
| Ethoric-10M is a decoder-only small language model with 9.97M parameters. It is built on a modern transformer foundation of RoPE, grouped-query attention and SwiGLU, and adds four changes on top. Peri-LN normalization applies RMSNorm both before and after every sublayer to keep training stable. Canon layers are short causal depthwise convolutions that mix each token with its 3 previous tokens before attention and before the MLP. Per-head QK-norm normalizes the queries and keys in each attention head, and a headwise sigmoid gate lets each head control how much of its output passes through. The model was trained on 15B tokens, followed by a 4B-token cooldown. |
|
|
| ## Model Architecture |
|
|
| | Parameter | Value | |
| |---|---| |
| | Architecture | Decoder-only transformer (RMSNorm, RoPE, GQA, SwiGLU) | |
| | Layers | 12 | |
| | Hidden size (d_model) | 256 | |
| | Attention heads | 8 | |
| | KV heads (GQA) | 2 | |
| | Head dim | 32 | |
| | FFN intermediate size | 721 | |
| | Normalization | Peri-LN (RMSNorm, ε = 1e-5) + per-head QK-norm | |
| | Attention gate | Headwise sigmoid gate | |
| | Canon layers | Causal depthwise conv, kernel 4 (before attention and MLP) | |
| | Local attention | 16-token window in layer 0; full causal in layers 1–11 | |
| | RoPE theta | 10,000 | |
| | Max sequence length | 512 | |
| | Vocabulary size | 5,048 | |
| | Tie word embeddings | Yes | |
| | Precision | FP32 weights | |
| | Parameters | 9.97M | |
| |
| ## How to use |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "techdotus/Ethoric-10M" |
| tok = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval() |
|
|
| ids = tok("The climate change", return_tensors="pt").input_ids |
| with torch.no_grad(): |
| out = model.generate(ids, max_new_tokens=32, do_sample=False, |
| eos_token_id=1) # greedy |
| print(tok.decode(out[0], skip_special_tokens=True)) |
| ``` |
| |
| The model's custom `generate()` supports greedy decoding and top-k sampling (`do_sample=True`, `temperature`, `top_k`). Other options such as `repetition_penalty` are ignored. Prompt plus generated tokens must stay within 512 tokens. |
|
|
| ## Benchmark Results |
|
|
| All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology. |
| |
| | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-3 | |
| |---|---|---|---|---| |
| | 28.75% | 37.58% | 24.66% | 59.85% | 41.60% | |
| |
| ## Comparison Results |
| |
| All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology. Scores for the other models are from the Open SLM Leaderboard. Bold marks the best result in each column. |
|
|
| | Model | Params | Int Index | HellaSwag | ARC-Easy | ARC-Chal | PIQA | ArithMark-3 | |
| |---|---|---|---|---|---|---|---| |
| | Ethoric-10M (Tech.us) | 9.97M | **12.94** | 28.75% | 37.58% | **24.66%** | **59.85%** | **41.60%** | |
| | ForgePlex-M2.2-10M | 9.99M | 12.12 | **29.19%** | **39.77%** | 22.70% | 59.79% | 37.40% | |
| | heron1i-flash-lite-v2 | 9.92M | 11.11 | 28.86% | 38.89% | 23.38% | 58.54% | 36.70% | |
| | min-spark-1.6 | 9.96M×4* | 11.08 | 28.52% | 38.30% | 22.44% | 59.14% | 36.90% | |
|
|
| ## Training Details |
|
|
| Ethoric-10M was trained in two stages. |
|
|
| | Stage | Tokens | Focus | |
| |---|---|---| |
| | Stable | 15B | Broad mix of educational web text, synthetic textbooks, question-answer data and math | |
| | Cooldown | 4B | Learning rate decayed; mix shifted toward higher-quality math, QA and tutorial-style text | |
|
|
| Training packs multiple documents into each 512-token sequence. Document masking keeps them separate: attention and Canon convolutions never reach across an EOS token. |
|
|
| ## Training Data |
|
|
| The cooldown added new math sources such as FineMath, more question-answer and tutorial-style text such as FinePhrase, ELI5 and GooAQ, and dropped Tiny Strange Textbooks, High-Quality English Sentences and ScienceQA. |
|
|
| | Source | Stage 1 | Stage 2 | |
| |---|---|---| |
| | FineWeb-Edu | 35% | 25% | |
| | TxT360-v2 | 20% | 21% | |
| | Cosmopedia | 19.5% | 17% | |
| | Tiny Strange Textbooks | 11% | 0% | |
| | OpenMathInstruct-2 | 11% | 7% | |
| | High-Quality English Sentences | 2% | 0% | |
| | ScienceQA | 1.0% | 0% | |
| | Orca-Math Word Problems 200k | 0.5% | 5% | |
| | FineMath | 0% | 11% | |
| | FinePhrase | 0% | 11% | |
| | Miscellaneous† | 0% | 3% | |
|
|
| † ELI5, GooAQ, FactBase-DCLM, Cosmopedia-v2, Natural Questions, Pretrain-Behaviors, UltraData-Math |
|
|
| ## Limitations |
|
|
| - **Tiny model:** with 9.97M parameters and 256-dim hidden states, generations are often repetitive, incoherent or factually wrong. The model is a research artifact for studying small-scale LMs, not an assistant. |
| - **Short context:** 512 tokens maximum. RoPE tables are not built beyond that length, and the small 5,048-token vocabulary means 512 tokens covers less text than in models with larger vocabularies. |
| - **English only:** trained on English web, educational, synthetic, question-answer and math text. |
| - **Not instruction-tuned or safety-aligned:** it may reproduce biases present in the data. |
| - **Benchmark scores are near chance on some tasks:** ARC-Challenge (24.66%) and HellaSwag (28.75%) are close to the 25% random baseline, so small differences between models on these tasks should be read with caution. |
|
|
| ## License |
|
|
| MIT |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{ethoric10m2026, |
| title = {{Ethoric-10M}: A Small Language Model with Stabilized, |
| Gated Attention}, |
| author = {{Tech.us Team}}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| howpublished = {\url{https://huggingface.co/techdotus/Ethoric-10M}} |
| } |
| ``` |
