tetu-AI commited on
Commit
c9f4679
·
verified ·
1 Parent(s): 6852725

Initial release

Browse files
README.md ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ language:
4
+ - fr
5
+ - en
6
+ pipeline_tag: text-generation
7
+ tags:
8
+ - french
9
+ - small-language-model
10
+ - trained-from-scratch
11
+ - custom-architecture
12
+ - calibration
13
+ ---
14
+
15
+ # Têtu-2
16
+
17
+ **A small French language model trained from scratch, with a custom architecture.**
18
+
19
+ Têtu-2 is an independent research model built in October 2026. No weights from any other model went into it. Each
20
+ word it writes gets a color for how sure it is. It also has a built-in calculator for the arithmetic it writes out.
21
+
22
+ It is **not** a reliable assistant: it is small and it is often wrong. Every number below was measured, and its
23
+ limits are given with it.
24
+
25
+ ## Versions in this repository
26
+
27
+ | Folder | Version | What it is |
28
+ |---|---|---|
29
+ | `.` (root, default) | **Têtu-2 chat (v2.1)** | Conversation model: everyday French chat plus step-by-step reasoning |
30
+ | `base/` | Têtu-2 base | Pre-trained model (6.23B tokens), for text completion or your own fine-tuning |
31
+
32
+ ## Model summary
33
+
34
+ | | |
35
+ |---|---|
36
+ | **Developed by** | tetu-AI (independent project) |
37
+ | **Model type** | Decoder-only language model, custom architecture |
38
+ | **Parameters** | 702M in total, of which **115M do the computation** for each word. The rest is a large memory that the model reads at almost no cost. |
39
+ | **Context length** | 32,768 tokens |
40
+ | **Languages** | French first, English |
41
+ | **Vocabulary** | 32,768 tokens (French and English, digits split one by one) |
42
+ | **Precision** | bfloat16 (`model.safetensors`), except the doubt head in float32 |
43
+ | **License** | CC BY-NC 4.0 (non-commercial) |
44
+
45
+ What it does:
46
+ - **Doubt head.** It estimates, for every word, the probability that the word is wrong. The chat script colors the
47
+ words: green when sure, yellow when it hesitates, red when in doubt.
48
+ - **Built-in calculator.** When it writes « 48 + 27 + 66 = », the exact result is inserted.
49
+ - **Fast.** About 275 tokens/s on an RTX 5070 (our previous model, Têtu v1: 135).
50
+
51
+ ## Training
52
+
53
+ | Stage | What |
54
+ |---|---|
55
+ | Pre-training | 6.23B tokens, 18.5 h on one A100. Sources: FineWeb-Edu, FineWeb2-HQ (French), French Wikipedia, FineMath. At the end, some of our own generated data was mixed in; a program checked all of it. |
56
+ | Instruction tuning (v2.0) | Our own generators, checked by program: math, stories, multi-turn conversations that refer back to earlier turns, and concepts. Plus French conversation and code that was run to check it. |
57
+ | Chat tuning (v2.1) | Everyday French conversations written by Qwen2.5-7B-Instruct (Apache 2.0) acting as a teacher. Its reasoning answers were kept only when two samples agreed or the code ran. |
58
+
59
+ ## Evaluation
60
+
61
+ All results come from measurements we had already made. Nothing was re-run for this page.
62
+
63
+ ### MGSM-fr (250 grade-school math problems, translated into French by humans)
64
+
65
+ Same 250 problems, greedy decoding, same answer checker for every model. Têtu-2 never saw these problems during
66
+ training.
67
+
68
+ | Model | Parameters | MGSM-fr |
69
+ |---|---|---|
70
+ | Qwen2.5-1.5B-Instruct | 1,544M | 48.4 % |
71
+ | Qwen2.5-0.5B-Instruct | 494M | 17.2 % |
72
+ | **Têtu-2 chat (v2.1)** | **115M compute (702M total)** | **5.6 %** |
73
+ | Têtu v1.1 (our previous model) | 96M active (181M total) | 4.8 % |
74
+ | Têtu-2 instruct (v2.0) | 115M compute (702M total) | 4.0 % |
75
+ | SmolLM2-135M-Instruct | 135M | 1.6 % |
76
+ | MobileLLM-R1.5-140M | 140M | 0.4 % * |
77
+
78
+ \* With our prompt, MobileLLM-R1.5-140M did not finish its answers within 768 tokens, so this score
79
+ underestimates it.
80
+
81
+ - **The calculator:** the Têtu-2 scores use it. The other models answer without one.
82
+ - **Têtu v1** reaches 11.2 % when it votes over 16 sampled answers. That vote was not run for Têtu-2.
83
+ - **Reading the table:** Têtu-2 beats the other models of its size, but it is far behind Qwen2.5-0.5B, which is
84
+ about 4 times larger in compute and was trained on 3,000 times more tokens.
85
+
86
+ ### Chat quality (pairwise judge)
87
+
88
+ The judge was Qwen2.5-14B-Instruct. It saw the answers in both orders, on 250 held-out requests that no model
89
+ learned from.
90
+
91
+ | | Wins | Losses | Ties | p (sign test) |
92
+ |---|---|---|---|---|
93
+ | Têtu-2 chat v2.1 vs Têtu v1.1 | **189** | 26 | 35 | 10⁻³¹ |
94
+ | Têtu-2 chat v2.1 vs Têtu-2 instruct v2.0 | **156** | 44 | 50 | 7·10⁻¹⁶ |
95
+
96
+ One caveat: the judge comes from the same family as the chat teacher, so it may prefer that teacher's style.
97
+
98
+ ### Our own generated tests
99
+
100
+ These tests use held-out phrasings. The calculator was on.
101
+
102
+ | | Têtu v1.0 | Têtu-2 instruct v2.0 | Têtu-2 chat v2.1 |
103
+ |---|---|---|---|
104
+ | Multi-turn follow-up (1,475 turns) | **93.2 %** | 71.0 % | 61.1 % |
105
+ | Concepts (350) | **75.4 %** | 62.9 % | 52.6 % |
106
+ | Template math (215) | **79.1 %** | 64.7 % | 64.7 % |
107
+
108
+ Têtu v1, a much deeper model per word, stays better at following a conversation over several turns.
109
+
110
+ **Base model:** on the same held-out text windows, the validation loss is 2.680 for Têtu-2 base and 2.838 for our
111
+ previous base model (Têtu v0.2, trained on about 2B tokens).
112
+
113
+ ## Limitations
114
+
115
+ - **It makes things up.** This happens especially on rare topics. Do not use it for medical, legal or financial
116
+ information, or for any important decision.
117
+ - **Its reasoning on new problems is weak.** It can mix up units (60 km/h for 30 min → "1,800 km") or divide by the
118
+ wrong number in an average.
119
+ - **v2.1 sometimes confuses "I" and "you".** Asked « what did I ask you then? », it may answer as if it had been the
120
+ one asking.
121
+ - **It was little trained on English conversation, and it is not a coding model.**
122
+
123
+ ## Usage
124
+
125
+ The architecture is custom, so it does not load with `transformers`. Use the bundled inference code.
126
+
127
+ ```bash
128
+ git clone https://huggingface.co/tetu-AI/Tetu-2 && cd Tetu-2
129
+ pip install -r requirements.txt
130
+ python chat.py # chat with v2.1; --cpu without a GPU; --no-calc to turn the calculator off
131
+ ```
132
+
133
+ Text completion with the base model:
134
+
135
+ ```python
136
+ import torch
137
+ from tetu2.generate import Generator, load
138
+ model, tok = load("base", device="cuda") # or device="cpu", dtype=torch.float32
139
+ gen = Generator(model, tok)
140
+ for piece, doubt in gen.stream("La capitale de la France est", max_new=40, temp=0):
141
+ print(piece, end="")
142
+ ```
143
+
144
+ ## License
145
+
146
+ **CC BY-NC 4.0 (non-commercial).** The fine-tuning data includes material under a non-commercial license and
147
+ synthetic conversations. The pre-training sources (FineWeb-Edu, FineWeb2-HQ, FineMath) are ODC-By; Wikipedia is
148
+ CC BY-SA. The chat teacher, Qwen2.5-7B-Instruct, is Apache 2.0.
149
+
150
+ ## Citation
151
+
152
+ ```bibtex
153
+ @misc{tetu2_2026,
154
+ title = {Têtu-2: a small French language model trained from scratch},
155
+ author = {tetu-AI},
156
+ year = {2026},
157
+ url = {https://huggingface.co/tetu-AI/Tetu-2}
158
+ }
159
+ ```
base/config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "Têtu-2 base",
3
+ "step": 11879,
4
+ "tokens": 6228017152,
5
+ "config": {
6
+ "vocab_size": 32768,
7
+ "d_model": 768,
8
+ "n_layers": 12,
9
+ "n_heads": 12,
10
+ "n_kv_heads": 4,
11
+ "mlp_mult": 4,
12
+ "max_seq_len": 32768,
13
+ "window": 1024,
14
+ "global_every": 4,
15
+ "rope_local": 10000.0,
16
+ "rope_global": 500000.0,
17
+ "log_scale": true,
18
+ "engram_layers": [
19
+ 1,
20
+ 5
21
+ ],
22
+ "engram_orders": [
23
+ 2,
24
+ 3
25
+ ],
26
+ "engram_heads": 2,
27
+ "engram_slots": 524287,
28
+ "engram_dim": 128,
29
+ "n_value_embeds": 3,
30
+ "softcap": 30.0,
31
+ "mtp_weight": 0.3,
32
+ "doubt_weight": 0.1
33
+ }
34
+ }
base/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:943c38e1245ca9ec77eae22894be5245ba214ede4c56d805b4916743b38a3a8f
3
+ size 1405245656
chat.py ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Chat with Têtu-2 in the terminal (it speaks French). Each word is colored by its doubt (green: sure,
2
+ yellow: hesitates, red: in doubt). Commands: /reset, /temp 0.5, /color, /quit.
3
+
4
+ python chat.py (GPU)
5
+ python chat.py --cpu (no GPU, slower)
6
+ python chat.py --no-calc (turn the built-in calculator off)
7
+ """
8
+ import argparse
9
+ import os
10
+ from pathlib import Path
11
+ import sys
12
+ import time
13
+
14
+ ROOT = Path(__file__).resolve().parent
15
+ sys.path.insert(0, str(ROOT))
16
+ import torch
17
+
18
+ from tetu.chat import ChatEncoder
19
+ from tetu2.generate import Generator, load
20
+
21
+ GREEN, YELLOW, RED, BOLD, DIM, RESET = "\033[32m", "\033[33m", "\033[31m", "\033[1m", "\033[2m", "\033[0m"
22
+
23
+
24
+ def color(d):
25
+ return GREEN if d < 0.2 else YELLOW if d < 0.45 else RED
26
+
27
+
28
+ def main():
29
+ p = argparse.ArgumentParser()
30
+ p.add_argument("--ckpt", default=str(ROOT))
31
+ p.add_argument("--cpu", action="store_true")
32
+ p.add_argument("--no-calc", dest="calc", action="store_false")
33
+ p.add_argument("--temp", type=float, default=0.3)
34
+ p.add_argument("--max_new", type=int, default=600)
35
+ p.add_argument("--max_len", type=int, default=8192)
36
+ a = p.parse_args()
37
+ if sys.platform == "win32":
38
+ os.system("")
39
+ dev = "cpu" if a.cpu or not torch.cuda.is_available() else "cuda"
40
+ model, tok = load(a.ckpt, device=dev, dtype=torch.float32 if dev == "cpu" else torch.bfloat16)
41
+ enc = ChatEncoder(tok)
42
+ gen = Generator(model, tok, max_len=a.max_len)
43
+ print(f"{BOLD}Têtu-2{RESET} {DIM}· /reset · /temp x · /color · /quit{RESET}\n")
44
+ turns, temp, use_color = [], a.temp, True
45
+ while True:
46
+ try:
47
+ q = input(f"{BOLD}you>{RESET} ").strip()
48
+ except (EOFError, KeyboardInterrupt):
49
+ break
50
+ if not q:
51
+ continue
52
+ if q in ("/quit", "/q", "/exit"):
53
+ break
54
+ if q == "/reset":
55
+ turns = []
56
+ print(f"{DIM}new conversation{RESET}\n")
57
+ continue
58
+ if q.startswith("/temp"):
59
+ temp = float(q.split()[1])
60
+ continue
61
+ if q == "/color":
62
+ use_color = not use_color
63
+ continue
64
+ turns.append(("user", q))
65
+ ids = enc.encode(turns, generation_prompt=True)[0]
66
+ while len(ids) > a.max_len - a.max_new and len(turns) > 1:
67
+ turns = turns[2:]
68
+ ids = enc.encode(turns, generation_prompt=True)[0]
69
+ print(f"{BOLD}têtu>{RESET} ", end="", flush=True)
70
+ t0, pieces, doubts = time.time(), [], []
71
+ kw = dict(rep=1.0, calc=True, presence=0.5, frequency=0.3) if a.calc else dict(rep=1.05)
72
+ for piece, d in gen.stream(ids=ids, max_new=a.max_new, temp=temp, top_k=40, **kw):
73
+ pieces.append(piece)
74
+ doubts.append(d)
75
+ print((color(d) if use_color else "") + piece + (RESET if use_color else ""), end="", flush=True)
76
+ dt = time.time() - t0
77
+ turns.append(("assistant", "".join(pieces).strip()))
78
+ print(f"\n{DIM}[{len(doubts)} tokens · {len(doubts) / max(dt, 1e-6):.0f} tok/s]{RESET}\n")
79
+
80
+
81
+ if __name__ == "__main__":
82
+ with torch.inference_mode():
83
+ main()
config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "Têtu-2 chat (v2.1)",
3
+ "step": 6772,
4
+ "tokens": 439555328,
5
+ "config": {
6
+ "vocab_size": 32768,
7
+ "d_model": 768,
8
+ "n_layers": 12,
9
+ "n_heads": 12,
10
+ "n_kv_heads": 4,
11
+ "mlp_mult": 4,
12
+ "max_seq_len": 32768,
13
+ "window": 1024,
14
+ "global_every": 4,
15
+ "rope_local": 10000.0,
16
+ "rope_global": 500000.0,
17
+ "log_scale": true,
18
+ "engram_layers": [
19
+ 1,
20
+ 5
21
+ ],
22
+ "engram_orders": [
23
+ 2,
24
+ 3
25
+ ],
26
+ "engram_heads": 2,
27
+ "engram_slots": 524287,
28
+ "engram_dim": 128,
29
+ "n_value_embeds": 3,
30
+ "softcap": 30.0,
31
+ "mtp_weight": 0.3,
32
+ "doubt_weight": 0.1
33
+ }
34
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5d6c5175f6296150cbaa1229d39d915bbe1d4ca92867221a153cac36ad3ace6
3
+ size 1405245664
requirements.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ torch>=2.5
2
+ tokenizers
3
+ safetensors
4
+ numpy
tetu/__init__.py ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ # Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
2
+ import sys
3
+ for _stream in (sys.stdout, sys.stderr):
4
+ if hasattr(_stream, 'reconfigure') and (_stream.encoding or '').lower().replace('-', '') != 'utf8':
5
+ _stream.reconfigure(encoding='utf-8', errors='replace')
tetu/calc.py ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
2
+ from fractions import Fraction
3
+ import re
4
+ ALLOWED = set('0123456789 ,.×*÷/+-−()')
5
+ _OP = re.compile('[\\d)]\\s*[×*÷/+\\-−]\\s*[\\d(]')
6
+ _FR = re.compile('[éèàçêùû«»]|\\b(le|la|les|de|des|est|il|elle|donc|soit|reste|en|sur|pour|fois|moyenne|somme|nombre|vaut|fait)\\b', re.I)
7
+
8
+ def _parse(expr, comma_decimal):
9
+ s = re.sub('(?<=\\d) (?=\\d{3}\\b)', '', expr)
10
+ if comma_decimal:
11
+ s = s.replace(',', '.')
12
+ else:
13
+ s = re.sub('(?<=\\d),(?=\\d{3}\\b)', '', s)
14
+ s = s.replace('×', '*').replace('÷', '/').replace('−', '-')
15
+ toks = re.findall('\\d+(?:\\.\\d+)?|[*/+\\-()]|\\S', s)
16
+ pos = 0
17
+
18
+ def atom():
19
+ nonlocal pos
20
+ if pos >= len(toks):
21
+ raise ValueError
22
+ t = toks[pos]
23
+ pos += 1
24
+ if t == '(':
25
+ v = add()
26
+ if pos >= len(toks) or toks[pos] != ')':
27
+ raise ValueError
28
+ pos += 1
29
+ return v
30
+ if t == '-':
31
+ return -atom()
32
+ if not re.fullmatch('\\d+(?:\\.\\d+)?', t):
33
+ raise ValueError
34
+ return Fraction(t)
35
+
36
+ def mul():
37
+ nonlocal pos
38
+ v = atom()
39
+ while pos < len(toks) and toks[pos] in '*/':
40
+ op = toks[pos]
41
+ pos += 1
42
+ w = atom()
43
+ v = v * w if op == '*' else v / w
44
+ return v
45
+
46
+ def add():
47
+ nonlocal pos
48
+ v = mul()
49
+ while pos < len(toks) and toks[pos] in '+-':
50
+ op = toks[pos]
51
+ pos += 1
52
+ w = mul()
53
+ v = v + w if op == '+' else v - w
54
+ return v
55
+ v = add()
56
+ if pos != len(toks):
57
+ raise ValueError
58
+ return v
59
+
60
+ def _fmt(v, comma):
61
+ if v.denominator == 1:
62
+ return str(v.numerator)
63
+ d = v.denominator
64
+ while d % 2 == 0:
65
+ d //= 2
66
+ while d % 5 == 0:
67
+ d //= 5
68
+ if d != 1:
69
+ return None
70
+ s = f'{float(v):.6f}'.rstrip('0').rstrip('.')
71
+ return s.replace('.', ',') if comma else s
72
+
73
+ def calc_suffix(text):
74
+ stripped = text.rstrip(' ')
75
+ if not stripped.endswith('=') or stripped.endswith(('==', '<=', '>=', '!=')):
76
+ return None
77
+ body = re.sub('(?<=\\d) x (?=\\d)', ' × ', stripped[:-1])
78
+ j = len(body)
79
+ while j > 0 and body[j - 1] in ALLOWED:
80
+ j -= 1
81
+ while j < len(body) and body[j] == ' ':
82
+ j += 1
83
+ expr = body[j:].rstrip()
84
+ if not expr or not (expr[0].isdigit() or expr[0] == '('):
85
+ return None
86
+ if j > 0 and (body[j - 1].isalnum() or body[j - 1] in '_^²³'):
87
+ return None
88
+ if not _OP.search(expr):
89
+ return None
90
+ french = bool(_FR.search(text[-200:]))
91
+ comma_decimal = french or bool(re.search('\\d,(?:\\d{1,2}|\\d{4,})\\b|(?<!\\d)0,\\d', expr))
92
+ try:
93
+ v = _parse(expr, comma_decimal)
94
+ except (ValueError, ZeroDivisionError):
95
+ return None
96
+ if v.denominator != 1 and '/' in expr:
97
+ return None
98
+ out = _fmt(v, comma_decimal)
99
+ if out is None:
100
+ return None
101
+ if text.endswith(' '):
102
+ return out
103
+ spaced = bool(re.search('\\s[×*÷/+\\-−]\\s', expr)) or body.endswith(' ')
104
+ return (' ' if spaced else '') + out
tetu/chat.py ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
2
+ from tokenizers import Tokenizer
3
+ ROLES = ('system', 'user', 'assistant')
4
+
5
+ class ChatEncoder:
6
+
7
+ def __init__(self, tokenizer: Tokenizer):
8
+ self.tok = tokenizer
9
+ self.eos = tokenizer.token_to_id('<|eos|>')
10
+ self.pad = tokenizer.token_to_id('<|pad|>')
11
+ self.heads = {r: self.ids(f'<|{r}|>\n') for r in ROLES}
12
+ self.nl = self.ids('\n')
13
+
14
+ def ids(self, text: str) -> list:
15
+ return self.tok.encode(text, add_special_tokens=False).ids
16
+
17
+ def encode(self, turns, generation_prompt: bool=False):
18
+ ids, labels, span = ([], [], None)
19
+ for role, content in turns:
20
+ head = self.heads[role]
21
+ ids += head
22
+ labels += [-100] * len(head)
23
+ if role == 'assistant':
24
+ pieces = content if isinstance(content, tuple) else (content,)
25
+ for j, piece in enumerate(pieces):
26
+ p = self.ids(piece)
27
+ if j == 0 and isinstance(content, tuple):
28
+ span = (len(ids), len(ids) + len(p))
29
+ ids += p
30
+ labels += p
31
+ ids.append(self.eos)
32
+ labels.append(self.eos)
33
+ else:
34
+ body = self.ids(content) + self.nl
35
+ ids += body
36
+ labels += [-100] * len(body)
37
+ if generation_prompt:
38
+ ids += self.heads['assistant']
39
+ labels += [-100] * len(self.heads['assistant'])
40
+ return (ids, labels, span)
41
+
42
+ def decode(self, ids) -> str:
43
+ return self.tok.decode(ids, skip_special_tokens=True)
44
+
45
+ def encode_turns(enc, turns):
46
+ ids, labels = ([], [])
47
+ for role, text, sup in turns:
48
+ head = enc.heads[role]
49
+ ids += head
50
+ labels += [-100] * len(head)
51
+ if role == 'assistant':
52
+ body = enc.ids(text) + [enc.eos]
53
+ ids += body
54
+ labels += body if sup else [-100] * len(body)
55
+ else:
56
+ body = enc.ids(text) + enc.nl
57
+ ids += body
58
+ labels += [-100] * len(body)
59
+ return (ids, labels)
tetu/fastgen.py ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
2
+ import re
3
+
4
+ import torch
5
+
6
+ _MASKS = {}
7
+
8
+ def word_mask(enc, vocab_size, device):
9
+ key = (id(enc), vocab_size, str(device))
10
+ if key not in _MASKS:
11
+ m = torch.zeros(vocab_size, dtype=torch.bool)
12
+ for i in range(min(vocab_size, enc.tok.get_vocab_size())):
13
+ piece = enc.tok.decode([i])
14
+ letters = sum((c.isalpha() for c in piece))
15
+ m[i] = letters >= 4 and (not any((c.isdigit() for c in piece)))
16
+ _MASKS[key] = m.to(device)
17
+ return _MASKS[key]
18
+
19
+ def _penalize(logits, counts, mask, presence, frequency):
20
+ if presence <= 0 and frequency <= 0:
21
+ return logits
22
+ c = counts * mask
23
+ return logits - presence * (c > 0) - frequency * c
24
+ _EQ_CACHE = {}
25
+
26
+ def _eq_ids(enc):
27
+ key = id(enc)
28
+ if key not in _EQ_CACHE:
29
+ _EQ_CACHE[key] = {i for i in range(enc.tok.get_vocab_size()) if '=' in enc.tok.decode([i])}
30
+ return _EQ_CACHE[key]
31
+ _TAIL_CACHE = {}
32
+
33
+ def tail_ids(enc, vocab_size, device):
34
+ key = (id(enc), vocab_size, str(device))
35
+ if key not in _TAIL_CACHE:
36
+ dig, nz, sep, zero = (torch.zeros(vocab_size, dtype=torch.bool), torch.zeros(vocab_size, dtype=torch.bool), set(), set())
37
+ for i in range(min(vocab_size, enc.tok.get_vocab_size())):
38
+ piece = enc.tok.decode([i])
39
+ dig[i] = bool(re.match('\\d', piece))
40
+ nz[i] = dig[i] and (not re.fullmatch('0+', piece))
41
+ if piece in (',', '.'):
42
+ sep.add(i)
43
+ if re.fullmatch('0+', piece):
44
+ zero.add(i)
45
+ _TAIL_CACHE[key] = (dig.to(device), nz.to(device), sep, zero)
46
+ return _TAIL_CACHE[key]
tetu2/__init__.py ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ # Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
2
+ from .model import Tetu2, Tetu2Config
tetu2/generate.py ADDED
@@ -0,0 +1,322 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
2
+ from pathlib import Path
3
+ import math
4
+ import torch
5
+ import torch.nn.functional as F
6
+ from tokenizers import Tokenizer
7
+ from tetu2.model import Tetu2, Tetu2Config, apply_rope, local_attention, log_scale, rms
8
+ ROOT = Path(__file__).resolve().parents[1]
9
+
10
+ def load(path, device='cuda', dtype=torch.bfloat16):
11
+ if Path(path).is_dir():
12
+ import json
13
+ from safetensors.torch import load_file
14
+ meta = json.loads((Path(path) / 'config.json').read_text(encoding='utf-8'))
15
+ ck = dict(meta, model=load_file(str(Path(path) / 'model.safetensors')))
16
+ else:
17
+ ck = torch.load(path, map_location='cpu', weights_only=False)
18
+ model = Tetu2(Tetu2Config(**ck['config']))
19
+ model.load_state_dict({k: v.float() for k, v in ck['model'].items()})
20
+ for name, p in model.named_parameters():
21
+ if not name.startswith('doubt_head'):
22
+ p.data = p.data.to(dtype)
23
+ model.to(device).eval()
24
+ model.meta = {k: ck.get(k) for k in ('step', 'tokens')}
25
+ return (model, Tokenizer.from_file(str(ROOT / 'tokenizer.json')))
26
+
27
+ class Cache:
28
+
29
+ def __init__(self, model):
30
+ L = model.cfg.n_layers
31
+ self.pos = 0
32
+ self.k, self.v, self.kstart = ([None] * L, [None] * L, [0] * L)
33
+ self.tail_tok = None
34
+ self.tail_s = {}
35
+ self.gate = {}
36
+
37
+ def clone(self):
38
+ c = Cache.__new__(Cache)
39
+ c.pos, c.kstart, c.tail_tok = (self.pos, list(self.kstart), self.tail_tok)
40
+ c.k, c.v, c.tail_s = (list(self.k), list(self.v), dict(self.tail_s))
41
+ return c
42
+
43
+ def _engram(eng, x, idx, cache, i):
44
+ B, T, d = x.shape
45
+ ctx = idx if cache.tail_tok is None else torch.cat([cache.tail_tok, idx], dim=1)
46
+ e = eng.table(eng.slots(ctx)[:, -T:]).reshape(B, T, -1).to(x.dtype)
47
+ k, v = (eng.wk(e), eng.wv(e))
48
+ alpha = torch.sigmoid((rms(x) * rms(k)).sum(-1, keepdim=True) / math.sqrt(d))
49
+ s = alpha * v
50
+ cache.gate[i] = alpha[..., 0].float()
51
+ prev = cache.tail_s.get(i)
52
+ full = s if prev is None else torch.cat([prev, s], dim=1)
53
+ c = eng.conv(F.pad(full.transpose(1, 2), (3, 0))).transpose(1, 2)[:, -T:]
54
+ cache.tail_s[i] = full[:, -3:]
55
+ return s + F.silu(c)
56
+
57
+ def _attention(att, x, cos, sin, qs, ve, lam, cache, i, pos):
58
+ B, T, _ = x.shape
59
+ q = att.wq(x).view(B, T, att.H, att.hd).transpose(1, 2)
60
+ k = att.wk(x).view(B, T, att.Hk, att.hd).transpose(1, 2)
61
+ v = att.wv(x).view(B, T, att.Hk, att.hd).transpose(1, 2)
62
+ if ve is not None:
63
+ v = v + lam * ve.to(v.dtype).view(B, T, att.Hk, att.hd).transpose(1, 2)
64
+ q, k = (apply_rope(rms(q), cos, sin), apply_rope(rms(k), cos, sin))
65
+ if qs is not None:
66
+ q = q * qs.to(q.dtype)[:, None]
67
+ first = cache.k[i] is None
68
+ K = k if first else torch.cat([cache.k[i], k], dim=2)
69
+ V = v if first else torch.cat([cache.v[i], v], dim=2)
70
+ r = att.H // att.Hk
71
+ Kr, Vr = (K.repeat_interleave(r, dim=1), V.repeat_interleave(r, dim=1))
72
+ if first:
73
+ if not att.is_global and T > att.window:
74
+ y = local_attention(q, Kr, Vr, att.window)
75
+ else:
76
+ y = F.scaled_dot_product_attention(q, Kr, Vr, is_causal=True)
77
+ else:
78
+ kpos = torch.arange(cache.kstart[i], cache.kstart[i] + K.size(2), device=x.device)
79
+ ok = kpos[None] <= pos[:, None]
80
+ if not att.is_global:
81
+ ok &= kpos[None] > pos[:, None] - att.window
82
+ y = F.scaled_dot_product_attention(q, Kr, Vr, attn_mask=ok if T > 1 or not att.is_global else None)
83
+ if not att.is_global and K.size(2) > att.window + 16:
84
+ drop = K.size(2) - att.window
85
+ K, V = (K[:, :, drop:], V[:, :, drop:])
86
+ cache.kstart[i] += drop
87
+ cache.k[i], cache.v[i] = (K, V)
88
+ y = y.transpose(1, 2).reshape(B, T, -1) * torch.sigmoid(att.wg(x))
89
+ return att.wo(y)
90
+
91
+ @torch.no_grad()
92
+ def forward_cached(model, idx, cache):
93
+ B, T = idx.shape
94
+ pos = torch.arange(cache.pos, cache.pos + T, device=idx.device)
95
+ loc, glo, qs = model.rope(T, pos)
96
+ x = rms(model.tok_emb(idx))
97
+ L, skips = (model.cfg.n_layers, [])
98
+ for i, blk in enumerate(model.blocks):
99
+ if i >= L - L // 2:
100
+ x = x + model.skip_w[L - 1 - i] * skips.pop()
101
+ j = model.ve_layer.get(i)
102
+ ve = model.value_embeds[j](idx) if j is not None else None
103
+ c, s = glo if blk.attn.is_global else loc
104
+ if blk.engram is not None:
105
+ x = x + _engram(blk.engram, rms(x), idx, cache, i)
106
+ x = x + _attention(blk.attn, rms(x), c, s, qs if blk.attn.is_global else None, ve, model.ve_lambda[i], cache, i, pos)
107
+ x = x + blk.mlp(rms(x))
108
+ if i < L // 2:
109
+ skips.append(x)
110
+ ctx = idx if cache.tail_tok is None else torch.cat([cache.tail_tok, idx], dim=1)
111
+ cache.tail_tok = ctx[:, -2:]
112
+ cache.pos += T
113
+ return rms(x)
114
+
115
+ class GraphStep:
116
+
117
+ def __init__(self, model, max_len=8192):
118
+ self.model, self.max_len = (model, max_len)
119
+ cfg, dev = (model.cfg, next(model.parameters()).device)
120
+ dt = model.tok_emb.weight.dtype
121
+ hd, W = (cfg.d_model // cfg.n_heads, cfg.window)
122
+ self.W = W
123
+ self.S = [max_len if b.attn.is_global else W for b in model.blocks]
124
+ self.K = [torch.zeros(1, cfg.n_kv_heads, s, hd, device=dev, dtype=dt) for s in self.S]
125
+ self.V = [torch.zeros_like(k) for k in self.K]
126
+ self.ring_pos = torch.empty(W, dtype=torch.long, device=dev)
127
+ self.gidx = torch.arange(max_len, device=dev)
128
+ self.tok = torch.zeros(1, 1, dtype=torch.long, device=dev)
129
+ self.pos = torch.zeros(1, dtype=torch.long, device=dev)
130
+ self.pad = cfg.vocab_size
131
+ self.tail_tok = torch.empty(1, 2, dtype=torch.long, device=dev)
132
+ self.tail_s = {i: torch.zeros(1, 3, cfg.d_model, device=dev, dtype=dt) for i, b in enumerate(model.blocks) if b.engram is not None}
133
+ self.reset()
134
+ s = torch.cuda.Stream()
135
+ s.wait_stream(torch.cuda.current_stream())
136
+ with torch.cuda.stream(s):
137
+ for _ in range(3):
138
+ self._step()
139
+ torch.cuda.current_stream().wait_stream(s)
140
+ self.graph = torch.cuda.CUDAGraph()
141
+ with torch.cuda.graph(self.graph):
142
+ self.out = self._step()
143
+ self.reset()
144
+
145
+ def reset(self):
146
+ self.ring_pos.fill_(-(1 << 40))
147
+ self.tail_tok.fill_(self.pad)
148
+ for t in self.tail_s.values():
149
+ t.zero_()
150
+ self.pos.zero_()
151
+
152
+ @torch.no_grad()
153
+ def _step(self):
154
+ m, cfg, idx, pos = (self.model, self.model.cfg, self.tok, self.pos)
155
+ self.ring_pos.index_copy_(0, pos % self.W, pos)
156
+ gmask = (self.gidx <= pos)[None, None, None]
157
+ lmask = ((self.ring_pos <= pos) & (self.ring_pos > pos - self.W))[None, None, None]
158
+ loc, glo = ((m.cos_loc[pos], m.sin_loc[pos]), (m.cos_glo[pos], m.sin_glo[pos]))
159
+ qs = log_scale(pos) if cfg.log_scale else None
160
+ x = rms(m.tok_emb(idx))
161
+ L, skips, gates = (cfg.n_layers, [], [])
162
+ for i, blk in enumerate(m.blocks):
163
+ if i >= L - L // 2:
164
+ x = x + m.skip_w[L - 1 - i] * skips.pop()
165
+ if blk.engram is not None:
166
+ eng, xe = (blk.engram, rms(x))
167
+ e = eng.table(eng.slots(torch.cat([self.tail_tok, idx], dim=1))[:, -1:]).reshape(1, 1, -1).to(x.dtype)
168
+ k, v = (eng.wk(e), eng.wv(e))
169
+ alpha = torch.sigmoid((rms(xe) * rms(k)).sum(-1, keepdim=True) / math.sqrt(cfg.d_model))
170
+ s = alpha * v
171
+ gates.append(alpha.reshape(1).float())
172
+ full = torch.cat([self.tail_s[i], s], dim=1)
173
+ x = x + s + F.silu(eng.conv(full.transpose(1, 2)).transpose(1, 2))
174
+ self.tail_s[i].copy_(full[:, 1:])
175
+ att, xa = (blk.attn, rms(x))
176
+ q = att.wq(xa).view(1, 1, att.H, att.hd).transpose(1, 2)
177
+ k = att.wk(xa).view(1, 1, att.Hk, att.hd).transpose(1, 2)
178
+ v = att.wv(xa).view(1, 1, att.Hk, att.hd).transpose(1, 2)
179
+ j = m.ve_layer.get(i)
180
+ if j is not None:
181
+ v = v + m.ve_lambda[i] * m.value_embeds[j](idx).to(v.dtype).view(1, 1, att.Hk, att.hd).transpose(1, 2)
182
+ c, sn = glo if att.is_global else loc
183
+ q, k = (apply_rope(rms(q), c, sn), apply_rope(rms(k), c, sn))
184
+ if att.is_global and qs is not None:
185
+ q = q * qs.to(q.dtype)[:, None]
186
+ slot = pos if att.is_global else pos % self.W
187
+ self.K[i].index_copy_(2, slot, k)
188
+ self.V[i].index_copy_(2, slot, v)
189
+ y = F.scaled_dot_product_attention(q, self.K[i], self.V[i], attn_mask=gmask if att.is_global else lmask, enable_gqa=True)
190
+ x = x + att.wo(y.transpose(1, 2).reshape(1, 1, -1) * torch.sigmoid(att.wg(xa)))
191
+ x = x + blk.mlp(rms(x))
192
+ if i < L // 2:
193
+ skips.append(x)
194
+ self.tail_tok.copy_(torch.cat([self.tail_tok[:, 1:], idx], dim=1))
195
+ h = rms(x)
196
+ logits = m.logits(h)
197
+ return (logits, torch.sigmoid(m.doubt_head(h, logits)), torch.cat(gates).mean())
198
+
199
+ @torch.no_grad()
200
+ def prefill(self, ids):
201
+ assert len(ids) < self.max_len, f'prompt trop long ({len(ids)} ≥ {self.max_len})'
202
+ self.reset()
203
+ dev = self.tok.device
204
+ cache = Cache(self.model)
205
+ h = forward_cached(self.model, torch.tensor([ids], device=dev), cache)[:, -1:]
206
+ T = len(ids)
207
+ for i, S in enumerate(self.S):
208
+ K, V, k0 = (cache.k[i], cache.v[i], cache.kstart[i])
209
+ if S == self.max_len:
210
+ self.K[i][:, :, :T] = K
211
+ self.V[i][:, :, :T] = V
212
+ else:
213
+ p = torch.arange(max(k0, T - self.W), T, device=dev)
214
+ self.K[i].index_copy_(2, p % self.W, K[:, :, p - k0])
215
+ self.V[i].index_copy_(2, p % self.W, V[:, :, p - k0])
216
+ self.ring_pos.index_copy_(0, p % self.W, p)
217
+ t = cache.tail_tok
218
+ self.tail_tok[:, 2 - t.size(1):] = t
219
+ for i, s in cache.tail_s.items():
220
+ self.tail_s[i][:, 3 - s.size(1):] = s
221
+ self.pos.fill_(T)
222
+ logits = self.model.logits(h)
223
+ gate = torch.stack([g[0, -1] for g in cache.gate.values()]).mean()
224
+ return (logits, torch.sigmoid(self.model.doubt_head(h, logits)), gate)
225
+
226
+ def step(self, t):
227
+ self.tok.fill_(t)
228
+ self.graph.replay()
229
+ self.pos += 1
230
+ return self.out
231
+
232
+ class Generator:
233
+
234
+ def __init__(self, model, tok, graph=True, max_len=8192):
235
+ self.model, self.tok = (model, tok)
236
+ self.eos = tok.token_to_id('<|eos|>')
237
+ self.dev = next(model.parameters()).device
238
+ self.gs = GraphStep(model, max_len) if graph and self.dev.type == 'cuda' else None
239
+
240
+ def head(self, h):
241
+ logits = self.model.logits(h)
242
+ return (logits, torch.sigmoid(self.model.doubt_head(h, logits)))
243
+
244
+ def stream(self, text=None, stop=(), ids=None, **kw):
245
+ ids = ids if ids is not None else self.tok.encode(text, add_special_tokens=False).ids
246
+ out, shown = ([], '')
247
+ for t, d in self.tokens(ids, stop=stop, **kw):
248
+ if t == self.eos or t in stop:
249
+ return
250
+ out.append(t)
251
+ text_now = self.tok.decode(out)
252
+ if not text_now.endswith('�'):
253
+ yield (text_now[len(shown):], d)
254
+ shown = text_now
255
+
256
+ @torch.no_grad()
257
+ def tokens(self, ids, max_new=200, temp=0.7, top_k=40, rep=1.15, seed=0, stop=(), calc=False, presence=0.0, frequency=0.0):
258
+ from tetu.calc import calc_suffix
259
+ from tetu.chat import ChatEncoder
260
+ from tetu.fastgen import _eq_ids, _penalize, tail_ids, word_mask
261
+ g = torch.Generator(device=self.dev).manual_seed(seed)
262
+ if self.gs is not None:
263
+ logits, doubt, gate = self.gs.prefill(ids)
264
+ else:
265
+ cache = Cache(self.model)
266
+ logits, doubt = self.head(forward_cached(self.model, torch.tensor([ids], device=self.dev), cache)[:, -1:])
267
+ gate = torch.stack([g_[0, -1] for g_ in cache.gate.values()]).mean()
268
+ V = logits.size(-1)
269
+ if (calc or presence or frequency) and (not hasattr(self, 'enc')):
270
+ self.enc = ChatEncoder(self.tok)
271
+ if calc:
272
+ eq = _eq_ids(self.enc)
273
+ tdig, tnz, tsep, tzero = tail_ids(self.enc, V, self.dev)
274
+ penal = presence > 0 or frequency > 0
275
+ if penal:
276
+ wmask, counts = (word_mask(self.enc, V, self.dev).float(), torch.zeros(V, device=self.dev))
277
+ recent, out, force, tail = (list(ids[-64:]), [], [], 0)
278
+ for _ in range(max_new):
279
+ logits = logits[0, -1].clone()
280
+ if rep != 1.0:
281
+ r = torch.tensor(sorted(set(recent)), device=self.dev)
282
+ lr = logits[r]
283
+ logits[r] = torch.where(lr > 0, lr / rep, lr * rep)
284
+ if penal:
285
+ logits = _penalize(logits, counts, wmask, presence, frequency)
286
+ guarded, tail = (tail, 0)
287
+ if guarded:
288
+ logits = logits.masked_fill(tdig if guarded == 1 else tnz, -float('inf'))
289
+ if force:
290
+ t = force.pop(0)
291
+ tail = 0 if force else 1
292
+ elif temp <= 0:
293
+ t = int(logits.argmax())
294
+ else:
295
+ v, i = torch.topk(logits / temp, top_k)
296
+ t = int(i[torch.multinomial(torch.softmax(v, -1), 1, generator=g)])
297
+ if guarded:
298
+ tail = 2 if t in tsep or (guarded == 2 and t in tzero) else 0
299
+ self.last_gate = float(gate)
300
+ yield (t, float(doubt[0, -1]))
301
+ if t == self.eos or t in stop:
302
+ return
303
+ out.append(t)
304
+ if calc and (not force) and (t in eq or (len(out) > 1 and out[-2] in eq)):
305
+ add = calc_suffix(self.tok.decode(out[-48:]))
306
+ if add:
307
+ force = self.enc.ids(add)
308
+ if penal:
309
+ counts[t] += 1
310
+ recent = (recent + [t])[-64:]
311
+ if self.gs is not None:
312
+ logits, doubt, gate = self.gs.step(t)
313
+ else:
314
+ logits, doubt = self.head(forward_cached(self.model, torch.tensor([[t]], device=self.dev), cache))
315
+ gate = torch.stack([g_[0, -1] for g_ in cache.gate.values()]).mean()
316
+
317
+ def complete(self, text, **kw):
318
+ pieces, doubts = ([], [])
319
+ for p, d in self.stream(text, **kw):
320
+ pieces.append(p)
321
+ doubts.append(d)
322
+ return (''.join(pieces), sum(doubts) / max(1, len(doubts)))
tetu2/model.py ADDED
@@ -0,0 +1,313 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
2
+ from dataclasses import asdict, dataclass
3
+ import math
4
+ from typing import Optional
5
+ import torch
6
+ import torch.nn as nn
7
+ import torch.nn.functional as F
8
+
9
+ @dataclass
10
+ class Tetu2Config:
11
+ vocab_size: int = 32768
12
+ d_model: int = 768
13
+ n_layers: int = 12
14
+ n_heads: int = 12
15
+ n_kv_heads: int = 4
16
+ mlp_mult: int = 4
17
+ max_seq_len: int = 32768
18
+ window: int = 1024
19
+ global_every: int = 4
20
+ rope_local: float = 10000.0
21
+ rope_global: float = 500000.0
22
+ log_scale: bool = True
23
+ engram_layers: tuple = (1, 5)
24
+ engram_orders: tuple = (2, 3)
25
+ engram_heads: int = 2
26
+ engram_slots: int = 524287
27
+ engram_dim: int = 128
28
+ n_value_embeds: int = 3
29
+ softcap: float = 30.0
30
+ mtp_weight: float = 0.3
31
+ doubt_weight: float = 0.1
32
+
33
+ def __post_init__(self):
34
+ self.engram_layers = tuple(self.engram_layers)
35
+ self.engram_orders = tuple(self.engram_orders)
36
+
37
+ def is_global(self, i):
38
+ return (i + 1) % self.global_every == 0
39
+
40
+ def to_dict(self):
41
+ return asdict(self)
42
+
43
+ def rms(x):
44
+ return F.rms_norm(x, (x.size(-1),))
45
+
46
+ def rope_table(n, hd, theta):
47
+ inv = 1.0 / theta ** (torch.arange(0, hd, 2, dtype=torch.float32) / hd)
48
+ f = torch.outer(torch.arange(n, dtype=torch.float32), inv)
49
+ return (f.cos(), f.sin())
50
+
51
+ def apply_rope(x, cos, sin):
52
+ x1, x2 = x.chunk(2, dim=-1)
53
+ return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1).type_as(x)
54
+
55
+ def log_scale(pos):
56
+ return (torch.log(pos.float() + 1) / math.log(1024)).clamp_min(1.0)
57
+
58
+ def local_attention(q, k, v, W):
59
+ B, H, T, D = q.shape
60
+ n = -(-T // W)
61
+ pad = n * W - T
62
+ q, k, v = (F.pad(t, (0, 0, 0, pad)) for t in (q, k, v))
63
+ a = torch.arange(W, device=q.device)[:, None]
64
+ b = torch.arange(2 * W, device=q.device)[None]
65
+ base = (b > a) & (b <= a + W)
66
+ if not torch.is_grad_enabled():
67
+ out = []
68
+ for i in range(n):
69
+ s0 = max(0, (i - 1) * W)
70
+ out.append(F.scaled_dot_product_attention(q[:, :, i * W:(i + 1) * W], k[:, :, s0:(i + 1) * W], v[:, :, s0:(i + 1) * W], attn_mask=base if i else base[:, W:]))
71
+ return torch.cat(out, dim=2)[:, :, :T]
72
+ qc, kc, vc = (t.view(B, H, n, W, D) for t in (q, k, v))
73
+ kk = torch.cat([F.pad(kc, (0, 0, 0, 0, 1, 0))[:, :, :n], kc], dim=3)
74
+ vv = torch.cat([F.pad(vc, (0, 0, 0, 0, 1, 0))[:, :, :n], vc], dim=3)
75
+ first = torch.arange(n, device=q.device)[:, None, None] == 0
76
+ m = (base[None] & ~(first & (b < W)[None])).unsqueeze(1)
77
+ flat = lambda t, L: t.permute(0, 2, 1, 3, 4).reshape(B * n, H, L, D)
78
+ y = F.scaled_dot_product_attention(flat(qc, W), flat(kk, 2 * W), flat(vv, 2 * W), attn_mask=m.expand(B, n, 1, W, 2 * W).reshape(B * n, 1, W, 2 * W))
79
+ return y.view(B, n, H, W, D).permute(0, 2, 1, 3, 4).reshape(B, H, n * W, D)[:, :, :T]
80
+
81
+ class Linear(nn.Linear):
82
+
83
+ def __init__(self, i, o, zero=False):
84
+ super().__init__(i, o, bias=False)
85
+ with torch.no_grad():
86
+ if zero:
87
+ self.weight.zero_()
88
+ else:
89
+ b = math.sqrt(3) * 0.5 / math.sqrt(i)
90
+ self.weight.uniform_(-b, b)
91
+
92
+ class Engram(nn.Module):
93
+ P = 2147483647
94
+
95
+ def __init__(self, cfg: Tetu2Config):
96
+ super().__init__()
97
+ self.orders, self.K, self.S, self.dm = (cfg.engram_orders, cfg.engram_heads, cfg.engram_slots, cfg.engram_dim)
98
+ self.n_tab = len(self.orders) * self.K
99
+ self.pad_id = cfg.vocab_size
100
+ self.table = nn.Embedding(self.n_tab * self.S, self.dm)
101
+ nn.init.normal_(self.table.weight, std=1.0)
102
+ g = torch.Generator().manual_seed(1234)
103
+ mult = torch.randint(1 << 20, 1 << 30, (self.n_tab, max(self.orders)), generator=g) | 1
104
+ self.register_buffer('mult', mult, persistent=False)
105
+ self.register_buffer('offset', torch.arange(self.n_tab) * self.S, persistent=False)
106
+ d, de = (cfg.d_model, self.n_tab * self.dm)
107
+ self.wk = Linear(de, d)
108
+ self.wv = Linear(de, d, zero=True)
109
+ self.conv = nn.Conv1d(d, d, 4, groups=d, bias=False)
110
+ nn.init.zeros_(self.conv.weight)
111
+
112
+ def slots(self, idx):
113
+ T = idx.size(1)
114
+ grams = [idx] + [F.pad(idx, (j, 0), value=self.pad_id)[:, :T] for j in range(1, max(self.orders))]
115
+ out, t = ([], 0)
116
+ for o in self.orders:
117
+ for _ in range(self.K):
118
+ h = torch.zeros_like(idx)
119
+ for j in range(o):
120
+ h = (h * 1000003 + (grams[j] + 1) * self.mult[t, j]) % self.P
121
+ out.append(h % self.S + self.offset[t])
122
+ t += 1
123
+ return torch.stack(out, dim=-1)
124
+
125
+ def forward(self, x, idx):
126
+ B, T, d = x.shape
127
+ e = self.table(self.slots(idx)).reshape(B, T, -1).to(x.dtype)
128
+ k, v = (self.wk(e), self.wv(e))
129
+ alpha = torch.sigmoid((rms(x) * rms(k)).sum(-1, keepdim=True) / math.sqrt(d))
130
+ s = alpha * v
131
+ c = self.conv(F.pad(s.transpose(1, 2), (3, 0))).transpose(1, 2)
132
+ if not torch.compiler.is_compiling():
133
+ self.last_gate = alpha.detach().float().mean()
134
+ return s + F.silu(c)
135
+
136
+ class Attention(nn.Module):
137
+
138
+ def __init__(self, cfg: Tetu2Config, is_global: bool):
139
+ super().__init__()
140
+ self.H, self.Hk = (cfg.n_heads, cfg.n_kv_heads)
141
+ self.hd = cfg.d_model // cfg.n_heads
142
+ d = cfg.d_model
143
+ self.wq = Linear(d, self.H * self.hd)
144
+ self.wk = Linear(d, self.Hk * self.hd)
145
+ self.wv = Linear(d, self.Hk * self.hd)
146
+ self.wg = Linear(d, self.H * self.hd)
147
+ self.wo = Linear(self.H * self.hd, d, zero=True)
148
+ self.is_global, self.window = (is_global, cfg.window)
149
+
150
+ def forward(self, x, cos, sin, qs=None, ve=None, lam=None):
151
+ B, T, _ = x.shape
152
+ q = self.wq(x).view(B, T, self.H, self.hd).transpose(1, 2)
153
+ k = self.wk(x).view(B, T, self.Hk, self.hd).transpose(1, 2)
154
+ v = self.wv(x).view(B, T, self.Hk, self.hd).transpose(1, 2)
155
+ if ve is not None:
156
+ v = v + lam * ve.to(v.dtype).view(B, T, self.Hk, self.hd).transpose(1, 2)
157
+ q, k = (apply_rope(rms(q), cos, sin), apply_rope(rms(k), cos, sin))
158
+ if qs is not None:
159
+ q = q * qs.to(q.dtype)[:, None]
160
+ r = self.H // self.Hk
161
+ k, v = (k.repeat_interleave(r, dim=1), v.repeat_interleave(r, dim=1))
162
+ if not self.is_global and T > self.window:
163
+ y = local_attention(q, k, v, self.window)
164
+ else:
165
+ y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
166
+ y = y.transpose(1, 2).reshape(B, T, -1) * torch.sigmoid(self.wg(x))
167
+ return self.wo(y)
168
+
169
+ class MLP(nn.Module):
170
+
171
+ def __init__(self, d, h):
172
+ super().__init__()
173
+ self.w1, self.w2 = (Linear(d, h), Linear(h, d, zero=True))
174
+
175
+ def forward(self, x):
176
+ return self.w2(F.relu(self.w1(x)).square())
177
+
178
+ class Block(nn.Module):
179
+
180
+ def __init__(self, cfg: Tetu2Config, i: int):
181
+ super().__init__()
182
+ self.attn = Attention(cfg, cfg.is_global(i))
183
+ self.mlp = MLP(cfg.d_model, cfg.mlp_mult * cfg.d_model)
184
+ self.engram = Engram(cfg) if i in cfg.engram_layers else None
185
+
186
+ def forward(self, x, idx, cos, sin, qs, ve, lam):
187
+ if self.engram is not None:
188
+ x = x + self.engram(rms(x), idx)
189
+ x = x + self.attn(rms(x), cos, sin, qs, ve, lam)
190
+ return x + self.mlp(rms(x))
191
+
192
+ class DoubtHead(nn.Module):
193
+
194
+ def __init__(self, d):
195
+ super().__init__()
196
+ self.net = nn.Sequential(nn.Linear(d + 2, 128), nn.ReLU(), nn.Linear(128, 1))
197
+
198
+ def forward(self, h, logits):
199
+ with torch.no_grad():
200
+ lp = F.log_softmax(logits.float(), dim=-1)
201
+ conf = torch.stack([lp.max(-1).values, -(lp.exp() * lp).sum(-1)], dim=-1) / 4
202
+ return self.net(torch.cat([rms(h.detach()).float(), conf], dim=-1)).squeeze(-1)
203
+
204
+ @dataclass
205
+ class Output:
206
+ logits: torch.Tensor
207
+ loss: Optional[torch.Tensor] = None
208
+ lm_loss: Optional[torch.Tensor] = None
209
+ mtp_loss: Optional[torch.Tensor] = None
210
+ doubt_loss: Optional[torch.Tensor] = None
211
+ doubt: Optional[torch.Tensor] = None
212
+
213
+ class Tetu2(nn.Module):
214
+
215
+ def __init__(self, cfg: Tetu2Config):
216
+ super().__init__()
217
+ self.cfg = cfg
218
+ d, L = (cfg.d_model, cfg.n_layers)
219
+ kv = cfg.n_kv_heads * (d // cfg.n_heads)
220
+ self.tok_emb = nn.Embedding(cfg.vocab_size, d)
221
+ nn.init.normal_(self.tok_emb.weight, std=1.0)
222
+ self.value_embeds = nn.ModuleList((nn.Embedding(cfg.vocab_size, kv) for _ in range(cfg.n_value_embeds)))
223
+ for e in self.value_embeds:
224
+ nn.init.normal_(e.weight, std=1.0)
225
+ n = cfg.n_value_embeds
226
+ self.ve_layer = {i: i for i in range(n)} | {L - n + i: i for i in range(n)}
227
+ self.ve_lambda = nn.Parameter(torch.full((L,), 0.5))
228
+ self.blocks = nn.ModuleList((Block(cfg, i) for i in range(L)))
229
+ self.skip_w = nn.Parameter(torch.ones(L // 2))
230
+ self.lm_head = Linear(d, cfg.vocab_size, zero=True)
231
+ self.mtp_in = Linear(2 * d, d)
232
+ self.mtp_mlp = MLP(d, cfg.mlp_mult * d)
233
+ self.doubt_head = DoubtHead(d)
234
+ hd = d // cfg.n_heads
235
+ for name, theta in (('loc', cfg.rope_local), ('glo', cfg.rope_global)):
236
+ c, s = rope_table(cfg.max_seq_len, hd, theta)
237
+ self.register_buffer(f'cos_{name}', c, persistent=False)
238
+ self.register_buffer(f'sin_{name}', s, persistent=False)
239
+
240
+ def num_params(self):
241
+ return sum((p.numel() for p in self.parameters()))
242
+
243
+ def compute_params(self):
244
+ lookup = self.tok_emb.weight.numel() + sum((e.weight.numel() for e in self.value_embeds))
245
+ lookup += sum((b.engram.table.weight.numel() for b in self.blocks if b.engram is not None))
246
+ return self.num_params() - lookup
247
+
248
+ def param_groups(self):
249
+ tables, head, scal, muon = ([], [], [], [])
250
+ for n, p in self.named_parameters():
251
+ if n.startswith(('tok_emb', 'value_embeds')) or n.endswith('engram.table.weight'):
252
+ tables.append(p)
253
+ elif n.startswith('lm_head'):
254
+ head.append(p)
255
+ elif p.dim() == 2 and (not n.startswith('doubt_head')):
256
+ muon.append(p)
257
+ else:
258
+ scal.append(p)
259
+ return {'tables': tables, 'head': head, 'scalars': scal, 'muon': muon}
260
+
261
+ def rope(self, T, pos=None):
262
+ p = torch.arange(T, device=self.cos_loc.device) if pos is None else pos
263
+ last = T - 1 if pos is None else int(pos[-1])
264
+ qs = log_scale(p) if self.cfg.log_scale and last >= 1024 else None
265
+ return ((self.cos_loc[p], self.sin_loc[p]), (self.cos_glo[p], self.sin_glo[p]), qs)
266
+
267
+ def hidden(self, idx):
268
+ B, T = idx.shape
269
+ loc, glo, qs = self.rope(T)
270
+ x = rms(self.tok_emb(idx))
271
+ L, skips = (self.cfg.n_layers, [])
272
+ for i, blk in enumerate(self.blocks):
273
+ if i >= L - L // 2:
274
+ x = x + self.skip_w[L - 1 - i] * skips.pop()
275
+ j = self.ve_layer.get(i)
276
+ ve = self.value_embeds[j](idx) if j is not None else None
277
+ c, s = glo if blk.attn.is_global else loc
278
+ x = blk(x, idx, c, s, qs if blk.attn.is_global else None, ve, self.ve_lambda[i])
279
+ if i < L // 2:
280
+ skips.append(x)
281
+ return rms(x)
282
+
283
+ def logits(self, h):
284
+ z = self.lm_head(h).float()
285
+ return self.cfg.softcap * torch.tanh(z / self.cfg.softcap)
286
+
287
+ def mtp_logits(self, h, nxt):
288
+ m = self.mtp_in(torch.cat([h, rms(self.tok_emb(nxt))], dim=-1))
289
+ m = m + self.mtp_mlp(rms(m))
290
+ return self.logits(rms(m))
291
+
292
+ def forward(self, idx, targets=None):
293
+ h = self.hidden(idx)
294
+ logits = self.logits(h)
295
+ doubt_logit = self.doubt_head(h, logits)
296
+ out = Output(logits=logits, doubt=torch.sigmoid(doubt_logit))
297
+ if targets is None:
298
+ return out
299
+ mask = targets != -100
300
+ out.lm_loss = F.cross_entropy(logits.flatten(0, 1), targets.flatten(), ignore_index=-100)
301
+ t2 = torch.cat([targets[:, 1:], torch.full_like(targets[:, :1], -100)], dim=1)
302
+ t2 = torch.where(mask, t2, torch.full_like(t2, -100))
303
+ lm2 = self.mtp_logits(h, targets.clamp_min(0))
304
+ out.mtp_loss = F.cross_entropy(lm2.flatten(0, 1), t2.flatten(), ignore_index=-100)
305
+ with torch.no_grad():
306
+ wrong = (logits.argmax(-1) != targets).float()
307
+ bce = F.binary_cross_entropy_with_logits(doubt_logit, wrong, reduction='none')
308
+ out.doubt_loss = (bce * mask).sum() / mask.sum().clamp_min(1)
309
+ out.loss = out.lm_loss + self.cfg.mtp_weight * out.mtp_loss + self.cfg.doubt_weight * out.doubt_loss
310
+ return out
311
+
312
+ def engram_gates(self):
313
+ return [b.engram.last_gate for b in self.blocks if b.engram is not None and hasattr(b.engram, 'last_gate')]
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff