Ethoric-10M

Model Summary

Ethoric-10M is a decoder-only small language model with 9.97M parameters. It is built on a modern transformer foundation of RoPE, grouped-query attention and SwiGLU, and adds four changes on top. Peri-LN normalization applies RMSNorm both before and after every sublayer to keep training stable. Canon layers are short causal depthwise convolutions that mix each token with its 3 previous tokens before attention and before the MLP. Per-head QK-norm normalizes the queries and keys in each attention head, and a headwise sigmoid gate lets each head control how much of its output passes through. The model was trained on 15B tokens, followed by a 4B-token cooldown.

Model Architecture

Parameter	Value
Architecture	Decoder-only transformer (RMSNorm, RoPE, GQA, SwiGLU)
Layers	12
Hidden size (d_model)	256
Attention heads	8
KV heads (GQA)	2
Head dim	32
FFN intermediate size	721
Normalization	Peri-LN (RMSNorm, ε = 1e-5) + per-head QK-norm
Attention gate	Headwise sigmoid gate
Canon layers	Causal depthwise conv, kernel 4 (before attention and MLP)
Local attention	16-token window in layer 0; full causal in layers 1–11
RoPE theta	10,000
Max sequence length	512
Vocabulary size	5,048
Tie word embeddings	Yes
Precision	FP32 weights
Parameters	9.97M

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "techdotus/Ethoric-10M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

ids = tok("The climate change", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=32, do_sample=False,
                         eos_token_id=1)  # greedy
print(tok.decode(out[0], skip_special_tokens=True))

The model's custom generate() supports greedy decoding and top-k sampling (do_sample=True, temperature, top_k). Other options such as repetition_penalty are ignored. Prompt plus generated tokens must stay within 512 tokens.

Benchmark Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology.

HellaSwag	ARC-Easy	ARC-Challenge	PIQA	ArithMark-3
28.75%	37.58%	24.66%	59.85%	41.60%

Comparison Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology. Scores for the other models are from the Open SLM Leaderboard. Bold marks the best result in each column.

Model	Params	Int Index	HellaSwag	ARC-Easy	ARC-Chal	PIQA	ArithMark-3
Ethoric-10M (Tech.us)	9.97M	12.94	28.75%	37.58%	24.66%	59.85%	41.60%
ForgePlex-M2.2-10M	9.99M	12.12	29.19%	39.77%	22.70%	59.79%	37.40%
heron1i-flash-lite-v2	9.92M	11.11	28.86%	38.89%	23.38%	58.54%	36.70%
min-spark-1.6	9.96M×4*	11.08	28.52%	38.30%	22.44%	59.14%	36.90%

Training Details

Ethoric-10M was trained in two stages.

Stage	Tokens	Focus
Stable	15B	Broad mix of educational web text, synthetic textbooks, question-answer data and math
Cooldown	4B	Learning rate decayed; mix shifted toward higher-quality math, QA and tutorial-style text

Training packs multiple documents into each 512-token sequence. Document masking keeps them separate: attention and Canon convolutions never reach across an EOS token.

Training Data

The cooldown added new math sources such as FineMath, more question-answer and tutorial-style text such as FinePhrase, ELI5 and GooAQ, and dropped Tiny Strange Textbooks, High-Quality English Sentences and ScienceQA.

Source	Stage 1	Stage 2
FineWeb-Edu	35%	25%
TxT360-v2	20%	21%
Cosmopedia	19.5%	17%
Tiny Strange Textbooks	11%	0%
OpenMathInstruct-2	11%	7%
High-Quality English Sentences	2%	0%
ScienceQA	1.0%	0%
Orca-Math Word Problems 200k	0.5%	5%
FineMath	0%	11%
FinePhrase	0%	11%
Miscellaneous†	0%	3%

† ELI5, GooAQ, FactBase-DCLM, Cosmopedia-v2, Natural Questions, Pretrain-Behaviors, UltraData-Math

Limitations

• Tiny model: with 9.97M parameters and 256-dim hidden states, generations are often repetitive, incoherent or factually wrong. The model is a research artifact for studying small-scale LMs, not an assistant.
• Short context: 512 tokens maximum. RoPE tables are not built beyond that length, and the small 5,048-token vocabulary means 512 tokens covers less text than in models with larger vocabularies.
• English only: trained on English web, educational, synthetic, question-answer and math text.
• Not instruction-tuned or safety-aligned: it may reproduce biases present in the data.
• Benchmark scores are near chance on some tasks: ARC-Challenge (24.66%) and HellaSwag (28.75%) are close to the 25% random baseline, so small differences between models on these tasks should be read with caution.

License

MIT

Citation

@misc{ethoric10m2026,
  title        = {{Ethoric-10M}: A Small Language Model with Stabilized,
                  Gated Attention},
  author       = {{Tech.us Team}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/techdotus/Ethoric-10M}}
}