Sane-118M

A 118.5M-parameter English language model trained from scratch — architecture, tokenizer, training code and behaviour data are all original. No existing weights were fine-tuned, distilled or merged.

Same design goal as Sane-47M, 2.5× the size: a small honest chat model whose knowledge is meant to come from a supplied context rather than from its own weights.

Try it in a browser: a Colab notebook — no install, free CPU runtime, about a minute to the first answer. Where it comes from: Sane Labs — four models, each larger than the last, none of them inheriting a single parameter from anywhere else.

Parameters	118.48 M (tied embeddings)
Architecture	RoPE · RMSNorm · SwiGLU · pre-norm, 15 layers, d_model 768, 12 heads, FFN 2048
Context	1024 tokens
Tokenizer	own byte-level BPE, vocabulary 16 000 (shared with Sane-47M)
Training tokens	≈ 3.6 B (≈ 2.9 passes over a 1.25 B-token corpus)
Hardware	Kaggle TPU v5e-8, ~6 hours
Schedule	WSD (warmup → stable 5e-4 → cosine decay to 5e-5 over the last 4 000 steps)
Final step	22 000
Language	English only
Precision	fp16 weights (~249 MiB)

Quickstart

Everything needed is in this repository.

pip install torch tokenizers huggingface_hub
hf download sekund0chka/sane-118m --local-dir sane-118m
cd sane-118m
python sane_chat.py --cpu --ckpt sane_final_fp16.pt --tok tokenizer.json

Runs on CPU. Commands inside the chat: /ctx <text> supplies a retrieval context, /reset clears history, /q quits.

Prompt format

Four control tokens, no system prompt — the persona lives in the weights.

<|user|>
{question}
<|sane|>
{answer}<|end|>

With retrieval, the context goes before the question:

<|context|>
Search results: The Ridgeloom V4 weighs 34 kg and sells for 690 euros.
<|user|>
How much does the Ridgeloom V4 weigh?
<|sane|>

Sampling defaults: temperature 0.7, top_k 50, top_p 0.92, repetition_penalty 1.12. For factual questions over a context, drop to temperature 0.3 or decode greedily.

Evaluation

Zero-shot, log-likelihood scoring, 200 examples per task, measured with the project's own sane_eval.py:

Task	Metric	Sane-118M	Random	GPT-2 small (124M)
HellaSwag	acc_norm	36.0	25.0	~31
ARC-Easy	acc	47.5	25.0	~44
ARC-Easy	acc_norm	28.5	25.0	—
LAMBADA	acc	22.0	~0	~33
FineWeb held-out	perplexity	19.6	—	—

Read this as: roughly GPT-2-small level, on ~40× less training data than GPT-2 saw relative to its size — clearly above chance, clearly not a parrot, and clearly a small model. LAMBADA is the weak spot: it rewards long-range context, which 1024 tokens and 118M parameters do not give much of. PIQA was not measured.

What it does and does not do

Works:

• stable identity — knows it is a model, will not roleplay as a human;
• reads <|context|> and answers from it, including messy search-result text with distractors, and says when the answer is not there;
• short multi-turn conversation inside the window;
• simple code and string tasks (s[::-1] and friends).

Does not work:

• facts without a context are unreliable — a capacity wall, not a tuning problem. Additional factual fine-tuning was tried and rejected: it produced more pattern-matched leakage, not more knowledge;
• arithmetic;
• long free-form generation stays grammatical but loses the thread;
• anything outside English.

Where this model sits

Sane Labs trains a line of models on one thesis that does not change with scale: put the knowledge in the context, not in the weights, and the model's job becomes reading carefully and stopping at the edge of what it was given.

Model	Size	State
Sane-47M	46.7 M	published
Sane-118M	118.5 M	published — this model
Sane-500M	~500 M	trained, kept private
Synth-2	1.23 B (~330 M active)	in training

Synth-2 is where the idea gets carried into the architecture rather than into the training data: a small head on every layer predicts the model's own error for each token, so an answer can underline the span you should check instead of asking to be trusted whole. Its pretraining run log is public and updates itself from the training job, and the working notes record what each measurement changed — including the measurements that killed a favourite explanation.

Bias, risks and limitations

Inherits the biases of web text filtered for educational quality. No safety tuning, no harmful-request refusals, no guardrails — treat output as untrusted text. No memory between sessions, no internet access: anything time-sensitive must be supplied in <|context|>.

Training data

Source	License
HuggingFaceFW/fineweb-edu (sample-10BT)	ODC-By 1.0
HuggingFaceTB/smol-smoltalk	Apache-2.0

The behaviour pool — identity, retrieval-reading, refusal-of-roleplay — is procedurally generated by the author and contains no third-party text.

ODC-By requires attribution; ATTRIBUTION.md in this repository is that attribution and must travel with the model if you redistribute it.

License

Apache-2.0. Use it, change it, ship it, sell it — no revenue ceiling and no permission needed. Earlier copies were released under the Sane Community License 1.0; this repository is now Apache-2.0 and that is the licence that applies.

One obligation that is not mine and cannot be relicensed away: part of the training corpus is FineWeb-Edu under ODC-By 1.0, which requires attribution. ATTRIBUTION.md in this repository is that attribution and must travel with the model if you redistribute it.

Citation

@misc{sane118m,
  title  = {Sane-118M: a small language model trained from scratch},
  author = {S.C.},
  year   = {2026},
  url    = {https://huggingface.co/sekund0chka/sane-118m}
}

Corrections to any number on this card are welcome — every one of them came from a run that can be repeated. ssanelabs@gmail.com