Hugging Face's logo

techdotus
/
Ethoric-10M

Text Generation
Transformers
Safetensors
English
ethoric
causal-lm
custom-architecture
custom_code
slm
small-language-model
Eval Results (legacy)
Model card Files
xet
 Community
Ethoric-10M / README.md
anandselvadurai-techus's picture
anandselvadurai-techus
Update README.md
1c35d59 verified
preview code
|
Raw History Blame Contribute Delete
7.45 kB
metadata 
language:
  - en
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
  - text-generation
  - causal-lm
  - custom-architecture
  - custom_code
  - slm
  - small-language-model
datasets:
  - agentlans/high-quality-english-sentences
  - armanc/ScienceQA
  - AxiomicLabs/FactBase-DCLM
  - HuggingFaceFW/finephrase
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceTB/cosmopedia
  - HuggingFaceTB/finemath
  - HuggingFaceTB/smollm-corpus
  - IFM/Pretrain-Behaviors
  - IFM/TxT360-v2
  - microsoft/orca-math-word-problems-200k
  - nampdn-ai/tiny-strange-textbooks
  - nvidia/OpenMathInstruct-2
  - openbmb/UltraData-Math
  - sentence-transformers/eli5
  - sentence-transformers/gooaq
  - sentence-transformers/natural-questions
model-index:
  - name: Ethoric-10M
    results:
      - task:
          type: text-generation
        dataset:
          type: allenai/ai2_arc
          name: ARC-Easy
          config: ARC-Easy
          split: test
        metrics:
          - type: acc_norm
            value: 37.58
            name: acc_norm (0-shot)
      - task:
          type: text-generation
        dataset:
          type: allenai/ai2_arc
          name: ARC-Challenge
          config: ARC-Challenge
          split: test
        metrics:
          - type: acc_norm
            value: 24.66
            name: acc_norm (0-shot)
      - task:
          type: text-generation
        dataset:
          type: Rowan/hellaswag
          name: HellaSwag
          split: validation
        metrics:
          - type: acc_norm
            value: 28.75
            name: acc_norm (0-shot)
      - task:
          type: text-generation
        dataset:
          type: ybisk/piqa
          name: PIQA
          split: validation
        metrics:
          - type: acc_norm
            value: 59.85
            name: acc_norm (0-shot)
      - task:
          type: text-generation
        dataset:
          type: AxiomicLabs/Arithmark-3.0
          name: ArithMark-3
          split: test
        metrics:
          - type: acc_norm
            value: 41.6
            name: acc_norm (0-shot)

Ethoric-10M

Model Summary

Ethoric-10M is a decoder-only small language model with 9.97M parameters. It is built on a modern transformer foundation of RoPE, grouped-query attention and SwiGLU, and adds four changes on top. Peri-LN normalization applies RMSNorm both before and after every sublayer to keep training stable. Canon layers are short causal depthwise convolutions that mix each token with its 3 previous tokens before attention and before the MLP. Per-head QK-norm normalizes the queries and keys in each attention head, and a headwise sigmoid gate lets each head control how much of its output passes through. The model was trained on 15B tokens, followed by a 4B-token cooldown.

Model Architecture

Parameter Value
Architecture Decoder-only transformer (RMSNorm, RoPE, GQA, SwiGLU)
Layers 12
Hidden size (d_model) 256
Attention heads 8
KV heads (GQA) 2
Head dim 32
FFN intermediate size 721
Normalization Peri-LN (RMSNorm, ε = 1e-5) + per-head QK-norm
Attention gate Headwise sigmoid gate
Canon layers Causal depthwise conv, kernel 4 (before attention and MLP)
Local attention 16-token window in layer 0; full causal in layers 1–11
RoPE theta 10,000
Max sequence length 512
Vocabulary size 5,048
Tie word embeddings Yes
Precision FP32 weights
Parameters 9.97M

How to use 

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "techdotus/Ethoric-10M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

ids = tok("The climate change", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=32, do_sample=False,
                         eos_token_id=1)  # greedy
print(tok.decode(out[0], skip_special_tokens=True))

The model's custom generate() supports greedy decoding and top-k sampling (do_sample=True, temperature, top_k). Other options such as repetition_penalty are ignored. Prompt plus generated tokens must stay within 512 tokens.

Benchmark Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology.

HellaSwag ARC-Easy ARC-Challenge PIQA ArithMark-3
28.75% 37.58% 24.66% 59.85% 41.60%

Comparison Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology. Scores for the other models are from the Open SLM Leaderboard. Bold marks the best result in each column.

Model Params Int Index HellaSwag ARC-Easy ARC-Chal PIQA ArithMark-3
Ethoric-10M (Tech.us) 9.97M 12.94 28.75% 37.58% 24.66% 59.85% 41.60%
ForgePlex-M2.2-10M 9.99M 12.12 29.19% 39.77% 22.70% 59.79% 37.40%
heron1i-flash-lite-v2 9.92M 11.11 28.86% 38.89% 23.38% 58.54% 36.70%
min-spark-1.6 9.96M×4* 11.08 28.52% 38.30% 22.44% 59.14% 36.90%

Training Details

Ethoric-10M was trained in two stages.

Stage Tokens Focus
Stable 15B Broad mix of educational web text, synthetic textbooks, question-answer data and math
Cooldown 4B Learning rate decayed; mix shifted toward higher-quality math, QA and tutorial-style text

Training packs multiple documents into each 512-token sequence. Document masking keeps them separate: attention and Canon convolutions never reach across an EOS token.

Training Data

The cooldown added new math sources such as FineMath, more question-answer and tutorial-style text such as FinePhrase, ELI5 and GooAQ, and dropped Tiny Strange Textbooks, High-Quality English Sentences and ScienceQA.

Source Stage 1 Stage 2
FineWeb-Edu 35% 25%
TxT360-v2 20% 21%
Cosmopedia 19.5% 17%
Tiny Strange Textbooks 11% 0%
OpenMathInstruct-2 11% 7%
High-Quality English Sentences 2% 0%
ScienceQA 1.0% 0%
Orca-Math Word Problems 200k 0.5% 5%
FineMath 0% 11%
FinePhrase 0% 11%
Miscellaneous† 0% 3%

† ELI5, GooAQ, FactBase-DCLM, Cosmopedia-v2, Natural Questions, Pretrain-Behaviors, UltraData-Math

Limitations

  • Tiny model: with 9.97M parameters and 256-dim hidden states, generations are often repetitive, incoherent or factually wrong. The model is a research artifact for studying small-scale LMs, not an assistant.
  • Short context: 512 tokens maximum. RoPE tables are not built beyond that length, and the small 5,048-token vocabulary means 512 tokens covers less text than in models with larger vocabularies.
  • English only: trained on English web, educational, synthetic, question-answer and math text.
  • Not instruction-tuned or safety-aligned: it may reproduce biases present in the data.
  • Benchmark scores are near chance on some tasks: ARC-Challenge (24.66%) and HellaSwag (28.75%) are close to the 25% random baseline, so small differences between models on these tasks should be read with caution.

License

MIT

Citation 

@misc{ethoric10m2026,
  title        = {{Ethoric-10M}: A Small Language Model with Stabilized,
                  Gated Attention},
  author       = {{Tech.us Team}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/techdotus/Ethoric-10M}}
}