tetu-AI/Tetu-v1

Têtu v1.1

A 96M-active-parameter French language model that thinks in loops and knows when it is unsure.

Têtu is a small research language model trained from scratch on a single consumer GPU (RTX 5070, 12 GB). It is built around three ideas: recurrent depth (a shared core applied 8 times per token), a doubt head that estimates, for every generated token, the probability that it is wrong, and training data that is programmatically generated and verified.

Model summary

Developed by	tetu-AI (independent project)
Model type	Decoder-only transformer, looped mixture-of-experts core
Parameters	181M total · 96M active per iteration
Context length	32,768 tokens
Languages	French (primary), English
Vocabulary	32,768 BPE tokens (FR/EN, single-digit numbers)
License	CC BY-NC 4.0
Precision	bfloat16 weights (model.safetensors, 393 MB)

Architecture

• Prelude → looped core → coda. 2 prelude blocks, a 4-block core repeated 8 times per token (zero initial state, input re-injected at every iteration), 2 coda blocks. d_model 768, 12 heads, 4 KV heads (GQA), FFN 3,072, tied embeddings.
• Mixture of experts in the core: 4 experts per block, one per loop iteration.
• Doubt head: reads how the recurrent state converges across iterations and predicts the error probability of each token. The chat client colors tokens accordingly.
• Pyramid memory (v1.1): the looped core (32 of 36 attention passes) uses a 1,024-token sliding window with a ring KV cache; prelude and coda attend to the full context (RoPE θ = 500,000, log-length query scaling).
• Inference tools: a built-in exact calculator (inserted after "a × b =") and a reflection mode that samples 16 solutions in parallel and votes, with the doubt head as tie-breaker.

Training

Stage	Data
Pre-training	~2B tokens: FineWeb-Edu (EN), FineWeb-2 (FR), FineMath
Instruction tuning	Filtered French conversations; programmatically generated and verified data (school math, word problems, multi-turn follow-ups, math concepts; every equation is recomputed before training); GSM8K train
Long-context tuning (v1.1)	Synthetic long conversations with questions about their beginning, long documents, PoSE position skipping

GSM8K test and MGSM were never used for training.

Evaluation

Benchmark	v1.0	v1.1
Multi-turn follow-ups (1,475 turns)*	93.2%	94.0%
Math concepts, held-out phrasings (350)*	75.4%	83.1%
Templated math (215)*	79.1%	80.9%
MGSM-fr, greedy (250)	4.4%	4.8%
MGSM-fr, 16-way vote (250)	10.4%	11.2%

*In-house generated test sets from the same generators as the training data, with test-only phrasings. They measure skills the model was taught, not general ability. For reference, Qwen2.5-0.5B scores 17.2% on MGSM-fr under our protocol. All comparisons are paired (sign test) with an identical decoder.

Recall of a fact stated at the beginning of a conversation: 98% at ~830 tokens, 88% at ~3,600 and 64% at ~29,600 (v1.0: 54%, 40%, 18%).

Limitations

• Hallucination. The model confidently invents facts on rare topics (see Les Misérables above). The doubt head often flags these tokens, but not always. Do not rely on it for factual, medical, legal or financial use.
• Reasoning. Multi-step reasoning on unseen word problems is weak (≤ 11% on MGSM-fr).
• Coverage. Mostly French conversational data; English chat is limited; no code training.
• Scale. Trained on ~2B tokens; general knowledge is far below models trained on trillions of tokens.

Usage

Custom architecture — the inference code is bundled; it is not compatible with transformers.

git clone https://huggingface.co/tetu-AI/Tetu-v1 && cd Tetu-v1
pip install -r requirements.txt
python speak.py            # interactive chat (CUDA graph decoder on GPU, --cpu otherwise)

In the chat: /pense toggles reflection mode (16 parallel samples + vote), /reset clears the conversation. Word problems trigger reflection automatically on GPU.

import json, torch
from safetensors.torch import load_file
from tetu.model import Tetu, TetuConfig

meta = json.load(open("v1.1/config.json"))
model = Tetu(TetuConfig(**meta["config"])).eval()
model.load_state_dict({k: v.float() for k, v in load_file("v1.1/model.safetensors").items()})

License

CC BY-NC 4.0. The instruction-tuning mix includes non-commercial data (Alpaca) and synthetic conversations.

Citation

@misc{tetu2026,
  title  = {Têtu: a small French looped language model with a doubt head},
  author = {tetu-AI},
  year   = {2026},
  url    = {https://huggingface.co/tetu-AI/Tetu-v1}
}