Initial release
Browse files- README.md +159 -0
- base/config.json +34 -0
- base/model.safetensors +3 -0
- chat.py +83 -0
- config.json +34 -0
- model.safetensors +3 -0
- requirements.txt +4 -0
- tetu/__init__.py +5 -0
- tetu/calc.py +104 -0
- tetu/chat.py +59 -0
- tetu/fastgen.py +46 -0
- tetu2/__init__.py +2 -0
- tetu2/generate.py +322 -0
- tetu2/model.py +313 -0
- tokenizer.json +0 -0
README.md
ADDED
|
@@ -0,0 +1,159 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
language:
|
| 4 |
+
- fr
|
| 5 |
+
- en
|
| 6 |
+
pipeline_tag: text-generation
|
| 7 |
+
tags:
|
| 8 |
+
- french
|
| 9 |
+
- small-language-model
|
| 10 |
+
- trained-from-scratch
|
| 11 |
+
- custom-architecture
|
| 12 |
+
- calibration
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# Têtu-2
|
| 16 |
+
|
| 17 |
+
**A small French language model trained from scratch, with a custom architecture.**
|
| 18 |
+
|
| 19 |
+
Têtu-2 is an independent research model built in October 2026. No weights from any other model went into it. Each
|
| 20 |
+
word it writes gets a color for how sure it is. It also has a built-in calculator for the arithmetic it writes out.
|
| 21 |
+
|
| 22 |
+
It is **not** a reliable assistant: it is small and it is often wrong. Every number below was measured, and its
|
| 23 |
+
limits are given with it.
|
| 24 |
+
|
| 25 |
+
## Versions in this repository
|
| 26 |
+
|
| 27 |
+
| Folder | Version | What it is |
|
| 28 |
+
|---|---|---|
|
| 29 |
+
| `.` (root, default) | **Têtu-2 chat (v2.1)** | Conversation model: everyday French chat plus step-by-step reasoning |
|
| 30 |
+
| `base/` | Têtu-2 base | Pre-trained model (6.23B tokens), for text completion or your own fine-tuning |
|
| 31 |
+
|
| 32 |
+
## Model summary
|
| 33 |
+
|
| 34 |
+
| | |
|
| 35 |
+
|---|---|
|
| 36 |
+
| **Developed by** | tetu-AI (independent project) |
|
| 37 |
+
| **Model type** | Decoder-only language model, custom architecture |
|
| 38 |
+
| **Parameters** | 702M in total, of which **115M do the computation** for each word. The rest is a large memory that the model reads at almost no cost. |
|
| 39 |
+
| **Context length** | 32,768 tokens |
|
| 40 |
+
| **Languages** | French first, English |
|
| 41 |
+
| **Vocabulary** | 32,768 tokens (French and English, digits split one by one) |
|
| 42 |
+
| **Precision** | bfloat16 (`model.safetensors`), except the doubt head in float32 |
|
| 43 |
+
| **License** | CC BY-NC 4.0 (non-commercial) |
|
| 44 |
+
|
| 45 |
+
What it does:
|
| 46 |
+
- **Doubt head.** It estimates, for every word, the probability that the word is wrong. The chat script colors the
|
| 47 |
+
words: green when sure, yellow when it hesitates, red when in doubt.
|
| 48 |
+
- **Built-in calculator.** When it writes « 48 + 27 + 66 = », the exact result is inserted.
|
| 49 |
+
- **Fast.** About 275 tokens/s on an RTX 5070 (our previous model, Têtu v1: 135).
|
| 50 |
+
|
| 51 |
+
## Training
|
| 52 |
+
|
| 53 |
+
| Stage | What |
|
| 54 |
+
|---|---|
|
| 55 |
+
| Pre-training | 6.23B tokens, 18.5 h on one A100. Sources: FineWeb-Edu, FineWeb2-HQ (French), French Wikipedia, FineMath. At the end, some of our own generated data was mixed in; a program checked all of it. |
|
| 56 |
+
| Instruction tuning (v2.0) | Our own generators, checked by program: math, stories, multi-turn conversations that refer back to earlier turns, and concepts. Plus French conversation and code that was run to check it. |
|
| 57 |
+
| Chat tuning (v2.1) | Everyday French conversations written by Qwen2.5-7B-Instruct (Apache 2.0) acting as a teacher. Its reasoning answers were kept only when two samples agreed or the code ran. |
|
| 58 |
+
|
| 59 |
+
## Evaluation
|
| 60 |
+
|
| 61 |
+
All results come from measurements we had already made. Nothing was re-run for this page.
|
| 62 |
+
|
| 63 |
+
### MGSM-fr (250 grade-school math problems, translated into French by humans)
|
| 64 |
+
|
| 65 |
+
Same 250 problems, greedy decoding, same answer checker for every model. Têtu-2 never saw these problems during
|
| 66 |
+
training.
|
| 67 |
+
|
| 68 |
+
| Model | Parameters | MGSM-fr |
|
| 69 |
+
|---|---|---|
|
| 70 |
+
| Qwen2.5-1.5B-Instruct | 1,544M | 48.4 % |
|
| 71 |
+
| Qwen2.5-0.5B-Instruct | 494M | 17.2 % |
|
| 72 |
+
| **Têtu-2 chat (v2.1)** | **115M compute (702M total)** | **5.6 %** |
|
| 73 |
+
| Têtu v1.1 (our previous model) | 96M active (181M total) | 4.8 % |
|
| 74 |
+
| Têtu-2 instruct (v2.0) | 115M compute (702M total) | 4.0 % |
|
| 75 |
+
| SmolLM2-135M-Instruct | 135M | 1.6 % |
|
| 76 |
+
| MobileLLM-R1.5-140M | 140M | 0.4 % * |
|
| 77 |
+
|
| 78 |
+
\* With our prompt, MobileLLM-R1.5-140M did not finish its answers within 768 tokens, so this score
|
| 79 |
+
underestimates it.
|
| 80 |
+
|
| 81 |
+
- **The calculator:** the Têtu-2 scores use it. The other models answer without one.
|
| 82 |
+
- **Têtu v1** reaches 11.2 % when it votes over 16 sampled answers. That vote was not run for Têtu-2.
|
| 83 |
+
- **Reading the table:** Têtu-2 beats the other models of its size, but it is far behind Qwen2.5-0.5B, which is
|
| 84 |
+
about 4 times larger in compute and was trained on 3,000 times more tokens.
|
| 85 |
+
|
| 86 |
+
### Chat quality (pairwise judge)
|
| 87 |
+
|
| 88 |
+
The judge was Qwen2.5-14B-Instruct. It saw the answers in both orders, on 250 held-out requests that no model
|
| 89 |
+
learned from.
|
| 90 |
+
|
| 91 |
+
| | Wins | Losses | Ties | p (sign test) |
|
| 92 |
+
|---|---|---|---|---|
|
| 93 |
+
| Têtu-2 chat v2.1 vs Têtu v1.1 | **189** | 26 | 35 | 10⁻³¹ |
|
| 94 |
+
| Têtu-2 chat v2.1 vs Têtu-2 instruct v2.0 | **156** | 44 | 50 | 7·10⁻¹⁶ |
|
| 95 |
+
|
| 96 |
+
One caveat: the judge comes from the same family as the chat teacher, so it may prefer that teacher's style.
|
| 97 |
+
|
| 98 |
+
### Our own generated tests
|
| 99 |
+
|
| 100 |
+
These tests use held-out phrasings. The calculator was on.
|
| 101 |
+
|
| 102 |
+
| | Têtu v1.0 | Têtu-2 instruct v2.0 | Têtu-2 chat v2.1 |
|
| 103 |
+
|---|---|---|---|
|
| 104 |
+
| Multi-turn follow-up (1,475 turns) | **93.2 %** | 71.0 % | 61.1 % |
|
| 105 |
+
| Concepts (350) | **75.4 %** | 62.9 % | 52.6 % |
|
| 106 |
+
| Template math (215) | **79.1 %** | 64.7 % | 64.7 % |
|
| 107 |
+
|
| 108 |
+
Têtu v1, a much deeper model per word, stays better at following a conversation over several turns.
|
| 109 |
+
|
| 110 |
+
**Base model:** on the same held-out text windows, the validation loss is 2.680 for Têtu-2 base and 2.838 for our
|
| 111 |
+
previous base model (Têtu v0.2, trained on about 2B tokens).
|
| 112 |
+
|
| 113 |
+
## Limitations
|
| 114 |
+
|
| 115 |
+
- **It makes things up.** This happens especially on rare topics. Do not use it for medical, legal or financial
|
| 116 |
+
information, or for any important decision.
|
| 117 |
+
- **Its reasoning on new problems is weak.** It can mix up units (60 km/h for 30 min → "1,800 km") or divide by the
|
| 118 |
+
wrong number in an average.
|
| 119 |
+
- **v2.1 sometimes confuses "I" and "you".** Asked « what did I ask you then? », it may answer as if it had been the
|
| 120 |
+
one asking.
|
| 121 |
+
- **It was little trained on English conversation, and it is not a coding model.**
|
| 122 |
+
|
| 123 |
+
## Usage
|
| 124 |
+
|
| 125 |
+
The architecture is custom, so it does not load with `transformers`. Use the bundled inference code.
|
| 126 |
+
|
| 127 |
+
```bash
|
| 128 |
+
git clone https://huggingface.co/tetu-AI/Tetu-2 && cd Tetu-2
|
| 129 |
+
pip install -r requirements.txt
|
| 130 |
+
python chat.py # chat with v2.1; --cpu without a GPU; --no-calc to turn the calculator off
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
Text completion with the base model:
|
| 134 |
+
|
| 135 |
+
```python
|
| 136 |
+
import torch
|
| 137 |
+
from tetu2.generate import Generator, load
|
| 138 |
+
model, tok = load("base", device="cuda") # or device="cpu", dtype=torch.float32
|
| 139 |
+
gen = Generator(model, tok)
|
| 140 |
+
for piece, doubt in gen.stream("La capitale de la France est", max_new=40, temp=0):
|
| 141 |
+
print(piece, end="")
|
| 142 |
+
```
|
| 143 |
+
|
| 144 |
+
## License
|
| 145 |
+
|
| 146 |
+
**CC BY-NC 4.0 (non-commercial).** The fine-tuning data includes material under a non-commercial license and
|
| 147 |
+
synthetic conversations. The pre-training sources (FineWeb-Edu, FineWeb2-HQ, FineMath) are ODC-By; Wikipedia is
|
| 148 |
+
CC BY-SA. The chat teacher, Qwen2.5-7B-Instruct, is Apache 2.0.
|
| 149 |
+
|
| 150 |
+
## Citation
|
| 151 |
+
|
| 152 |
+
```bibtex
|
| 153 |
+
@misc{tetu2_2026,
|
| 154 |
+
title = {Têtu-2: a small French language model trained from scratch},
|
| 155 |
+
author = {tetu-AI},
|
| 156 |
+
year = {2026},
|
| 157 |
+
url = {https://huggingface.co/tetu-AI/Tetu-2}
|
| 158 |
+
}
|
| 159 |
+
```
|
base/config.json
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "Têtu-2 base",
|
| 3 |
+
"step": 11879,
|
| 4 |
+
"tokens": 6228017152,
|
| 5 |
+
"config": {
|
| 6 |
+
"vocab_size": 32768,
|
| 7 |
+
"d_model": 768,
|
| 8 |
+
"n_layers": 12,
|
| 9 |
+
"n_heads": 12,
|
| 10 |
+
"n_kv_heads": 4,
|
| 11 |
+
"mlp_mult": 4,
|
| 12 |
+
"max_seq_len": 32768,
|
| 13 |
+
"window": 1024,
|
| 14 |
+
"global_every": 4,
|
| 15 |
+
"rope_local": 10000.0,
|
| 16 |
+
"rope_global": 500000.0,
|
| 17 |
+
"log_scale": true,
|
| 18 |
+
"engram_layers": [
|
| 19 |
+
1,
|
| 20 |
+
5
|
| 21 |
+
],
|
| 22 |
+
"engram_orders": [
|
| 23 |
+
2,
|
| 24 |
+
3
|
| 25 |
+
],
|
| 26 |
+
"engram_heads": 2,
|
| 27 |
+
"engram_slots": 524287,
|
| 28 |
+
"engram_dim": 128,
|
| 29 |
+
"n_value_embeds": 3,
|
| 30 |
+
"softcap": 30.0,
|
| 31 |
+
"mtp_weight": 0.3,
|
| 32 |
+
"doubt_weight": 0.1
|
| 33 |
+
}
|
| 34 |
+
}
|
base/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:943c38e1245ca9ec77eae22894be5245ba214ede4c56d805b4916743b38a3a8f
|
| 3 |
+
size 1405245656
|
chat.py
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Chat with Têtu-2 in the terminal (it speaks French). Each word is colored by its doubt (green: sure,
|
| 2 |
+
yellow: hesitates, red: in doubt). Commands: /reset, /temp 0.5, /color, /quit.
|
| 3 |
+
|
| 4 |
+
python chat.py (GPU)
|
| 5 |
+
python chat.py --cpu (no GPU, slower)
|
| 6 |
+
python chat.py --no-calc (turn the built-in calculator off)
|
| 7 |
+
"""
|
| 8 |
+
import argparse
|
| 9 |
+
import os
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
import sys
|
| 12 |
+
import time
|
| 13 |
+
|
| 14 |
+
ROOT = Path(__file__).resolve().parent
|
| 15 |
+
sys.path.insert(0, str(ROOT))
|
| 16 |
+
import torch
|
| 17 |
+
|
| 18 |
+
from tetu.chat import ChatEncoder
|
| 19 |
+
from tetu2.generate import Generator, load
|
| 20 |
+
|
| 21 |
+
GREEN, YELLOW, RED, BOLD, DIM, RESET = "\033[32m", "\033[33m", "\033[31m", "\033[1m", "\033[2m", "\033[0m"
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def color(d):
|
| 25 |
+
return GREEN if d < 0.2 else YELLOW if d < 0.45 else RED
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def main():
|
| 29 |
+
p = argparse.ArgumentParser()
|
| 30 |
+
p.add_argument("--ckpt", default=str(ROOT))
|
| 31 |
+
p.add_argument("--cpu", action="store_true")
|
| 32 |
+
p.add_argument("--no-calc", dest="calc", action="store_false")
|
| 33 |
+
p.add_argument("--temp", type=float, default=0.3)
|
| 34 |
+
p.add_argument("--max_new", type=int, default=600)
|
| 35 |
+
p.add_argument("--max_len", type=int, default=8192)
|
| 36 |
+
a = p.parse_args()
|
| 37 |
+
if sys.platform == "win32":
|
| 38 |
+
os.system("")
|
| 39 |
+
dev = "cpu" if a.cpu or not torch.cuda.is_available() else "cuda"
|
| 40 |
+
model, tok = load(a.ckpt, device=dev, dtype=torch.float32 if dev == "cpu" else torch.bfloat16)
|
| 41 |
+
enc = ChatEncoder(tok)
|
| 42 |
+
gen = Generator(model, tok, max_len=a.max_len)
|
| 43 |
+
print(f"{BOLD}Têtu-2{RESET} {DIM}· /reset · /temp x · /color · /quit{RESET}\n")
|
| 44 |
+
turns, temp, use_color = [], a.temp, True
|
| 45 |
+
while True:
|
| 46 |
+
try:
|
| 47 |
+
q = input(f"{BOLD}you>{RESET} ").strip()
|
| 48 |
+
except (EOFError, KeyboardInterrupt):
|
| 49 |
+
break
|
| 50 |
+
if not q:
|
| 51 |
+
continue
|
| 52 |
+
if q in ("/quit", "/q", "/exit"):
|
| 53 |
+
break
|
| 54 |
+
if q == "/reset":
|
| 55 |
+
turns = []
|
| 56 |
+
print(f"{DIM}new conversation{RESET}\n")
|
| 57 |
+
continue
|
| 58 |
+
if q.startswith("/temp"):
|
| 59 |
+
temp = float(q.split()[1])
|
| 60 |
+
continue
|
| 61 |
+
if q == "/color":
|
| 62 |
+
use_color = not use_color
|
| 63 |
+
continue
|
| 64 |
+
turns.append(("user", q))
|
| 65 |
+
ids = enc.encode(turns, generation_prompt=True)[0]
|
| 66 |
+
while len(ids) > a.max_len - a.max_new and len(turns) > 1:
|
| 67 |
+
turns = turns[2:]
|
| 68 |
+
ids = enc.encode(turns, generation_prompt=True)[0]
|
| 69 |
+
print(f"{BOLD}têtu>{RESET} ", end="", flush=True)
|
| 70 |
+
t0, pieces, doubts = time.time(), [], []
|
| 71 |
+
kw = dict(rep=1.0, calc=True, presence=0.5, frequency=0.3) if a.calc else dict(rep=1.05)
|
| 72 |
+
for piece, d in gen.stream(ids=ids, max_new=a.max_new, temp=temp, top_k=40, **kw):
|
| 73 |
+
pieces.append(piece)
|
| 74 |
+
doubts.append(d)
|
| 75 |
+
print((color(d) if use_color else "") + piece + (RESET if use_color else ""), end="", flush=True)
|
| 76 |
+
dt = time.time() - t0
|
| 77 |
+
turns.append(("assistant", "".join(pieces).strip()))
|
| 78 |
+
print(f"\n{DIM}[{len(doubts)} tokens · {len(doubts) / max(dt, 1e-6):.0f} tok/s]{RESET}\n")
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
if __name__ == "__main__":
|
| 82 |
+
with torch.inference_mode():
|
| 83 |
+
main()
|
config.json
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "Têtu-2 chat (v2.1)",
|
| 3 |
+
"step": 6772,
|
| 4 |
+
"tokens": 439555328,
|
| 5 |
+
"config": {
|
| 6 |
+
"vocab_size": 32768,
|
| 7 |
+
"d_model": 768,
|
| 8 |
+
"n_layers": 12,
|
| 9 |
+
"n_heads": 12,
|
| 10 |
+
"n_kv_heads": 4,
|
| 11 |
+
"mlp_mult": 4,
|
| 12 |
+
"max_seq_len": 32768,
|
| 13 |
+
"window": 1024,
|
| 14 |
+
"global_every": 4,
|
| 15 |
+
"rope_local": 10000.0,
|
| 16 |
+
"rope_global": 500000.0,
|
| 17 |
+
"log_scale": true,
|
| 18 |
+
"engram_layers": [
|
| 19 |
+
1,
|
| 20 |
+
5
|
| 21 |
+
],
|
| 22 |
+
"engram_orders": [
|
| 23 |
+
2,
|
| 24 |
+
3
|
| 25 |
+
],
|
| 26 |
+
"engram_heads": 2,
|
| 27 |
+
"engram_slots": 524287,
|
| 28 |
+
"engram_dim": 128,
|
| 29 |
+
"n_value_embeds": 3,
|
| 30 |
+
"softcap": 30.0,
|
| 31 |
+
"mtp_weight": 0.3,
|
| 32 |
+
"doubt_weight": 0.1
|
| 33 |
+
}
|
| 34 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e5d6c5175f6296150cbaa1229d39d915bbe1d4ca92867221a153cac36ad3ace6
|
| 3 |
+
size 1405245664
|
requirements.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
torch>=2.5
|
| 2 |
+
tokenizers
|
| 3 |
+
safetensors
|
| 4 |
+
numpy
|
tetu/__init__.py
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
|
| 2 |
+
import sys
|
| 3 |
+
for _stream in (sys.stdout, sys.stderr):
|
| 4 |
+
if hasattr(_stream, 'reconfigure') and (_stream.encoding or '').lower().replace('-', '') != 'utf8':
|
| 5 |
+
_stream.reconfigure(encoding='utf-8', errors='replace')
|
tetu/calc.py
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
|
| 2 |
+
from fractions import Fraction
|
| 3 |
+
import re
|
| 4 |
+
ALLOWED = set('0123456789 ,.×*÷/+-−()')
|
| 5 |
+
_OP = re.compile('[\\d)]\\s*[×*÷/+\\-−]\\s*[\\d(]')
|
| 6 |
+
_FR = re.compile('[éèàçêùû«»]|\\b(le|la|les|de|des|est|il|elle|donc|soit|reste|en|sur|pour|fois|moyenne|somme|nombre|vaut|fait)\\b', re.I)
|
| 7 |
+
|
| 8 |
+
def _parse(expr, comma_decimal):
|
| 9 |
+
s = re.sub('(?<=\\d) (?=\\d{3}\\b)', '', expr)
|
| 10 |
+
if comma_decimal:
|
| 11 |
+
s = s.replace(',', '.')
|
| 12 |
+
else:
|
| 13 |
+
s = re.sub('(?<=\\d),(?=\\d{3}\\b)', '', s)
|
| 14 |
+
s = s.replace('×', '*').replace('÷', '/').replace('−', '-')
|
| 15 |
+
toks = re.findall('\\d+(?:\\.\\d+)?|[*/+\\-()]|\\S', s)
|
| 16 |
+
pos = 0
|
| 17 |
+
|
| 18 |
+
def atom():
|
| 19 |
+
nonlocal pos
|
| 20 |
+
if pos >= len(toks):
|
| 21 |
+
raise ValueError
|
| 22 |
+
t = toks[pos]
|
| 23 |
+
pos += 1
|
| 24 |
+
if t == '(':
|
| 25 |
+
v = add()
|
| 26 |
+
if pos >= len(toks) or toks[pos] != ')':
|
| 27 |
+
raise ValueError
|
| 28 |
+
pos += 1
|
| 29 |
+
return v
|
| 30 |
+
if t == '-':
|
| 31 |
+
return -atom()
|
| 32 |
+
if not re.fullmatch('\\d+(?:\\.\\d+)?', t):
|
| 33 |
+
raise ValueError
|
| 34 |
+
return Fraction(t)
|
| 35 |
+
|
| 36 |
+
def mul():
|
| 37 |
+
nonlocal pos
|
| 38 |
+
v = atom()
|
| 39 |
+
while pos < len(toks) and toks[pos] in '*/':
|
| 40 |
+
op = toks[pos]
|
| 41 |
+
pos += 1
|
| 42 |
+
w = atom()
|
| 43 |
+
v = v * w if op == '*' else v / w
|
| 44 |
+
return v
|
| 45 |
+
|
| 46 |
+
def add():
|
| 47 |
+
nonlocal pos
|
| 48 |
+
v = mul()
|
| 49 |
+
while pos < len(toks) and toks[pos] in '+-':
|
| 50 |
+
op = toks[pos]
|
| 51 |
+
pos += 1
|
| 52 |
+
w = mul()
|
| 53 |
+
v = v + w if op == '+' else v - w
|
| 54 |
+
return v
|
| 55 |
+
v = add()
|
| 56 |
+
if pos != len(toks):
|
| 57 |
+
raise ValueError
|
| 58 |
+
return v
|
| 59 |
+
|
| 60 |
+
def _fmt(v, comma):
|
| 61 |
+
if v.denominator == 1:
|
| 62 |
+
return str(v.numerator)
|
| 63 |
+
d = v.denominator
|
| 64 |
+
while d % 2 == 0:
|
| 65 |
+
d //= 2
|
| 66 |
+
while d % 5 == 0:
|
| 67 |
+
d //= 5
|
| 68 |
+
if d != 1:
|
| 69 |
+
return None
|
| 70 |
+
s = f'{float(v):.6f}'.rstrip('0').rstrip('.')
|
| 71 |
+
return s.replace('.', ',') if comma else s
|
| 72 |
+
|
| 73 |
+
def calc_suffix(text):
|
| 74 |
+
stripped = text.rstrip(' ')
|
| 75 |
+
if not stripped.endswith('=') or stripped.endswith(('==', '<=', '>=', '!=')):
|
| 76 |
+
return None
|
| 77 |
+
body = re.sub('(?<=\\d) x (?=\\d)', ' × ', stripped[:-1])
|
| 78 |
+
j = len(body)
|
| 79 |
+
while j > 0 and body[j - 1] in ALLOWED:
|
| 80 |
+
j -= 1
|
| 81 |
+
while j < len(body) and body[j] == ' ':
|
| 82 |
+
j += 1
|
| 83 |
+
expr = body[j:].rstrip()
|
| 84 |
+
if not expr or not (expr[0].isdigit() or expr[0] == '('):
|
| 85 |
+
return None
|
| 86 |
+
if j > 0 and (body[j - 1].isalnum() or body[j - 1] in '_^²³'):
|
| 87 |
+
return None
|
| 88 |
+
if not _OP.search(expr):
|
| 89 |
+
return None
|
| 90 |
+
french = bool(_FR.search(text[-200:]))
|
| 91 |
+
comma_decimal = french or bool(re.search('\\d,(?:\\d{1,2}|\\d{4,})\\b|(?<!\\d)0,\\d', expr))
|
| 92 |
+
try:
|
| 93 |
+
v = _parse(expr, comma_decimal)
|
| 94 |
+
except (ValueError, ZeroDivisionError):
|
| 95 |
+
return None
|
| 96 |
+
if v.denominator != 1 and '/' in expr:
|
| 97 |
+
return None
|
| 98 |
+
out = _fmt(v, comma_decimal)
|
| 99 |
+
if out is None:
|
| 100 |
+
return None
|
| 101 |
+
if text.endswith(' '):
|
| 102 |
+
return out
|
| 103 |
+
spaced = bool(re.search('\\s[×*÷/+\\-−]\\s', expr)) or body.endswith(' ')
|
| 104 |
+
return (' ' if spaced else '') + out
|
tetu/chat.py
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
|
| 2 |
+
from tokenizers import Tokenizer
|
| 3 |
+
ROLES = ('system', 'user', 'assistant')
|
| 4 |
+
|
| 5 |
+
class ChatEncoder:
|
| 6 |
+
|
| 7 |
+
def __init__(self, tokenizer: Tokenizer):
|
| 8 |
+
self.tok = tokenizer
|
| 9 |
+
self.eos = tokenizer.token_to_id('<|eos|>')
|
| 10 |
+
self.pad = tokenizer.token_to_id('<|pad|>')
|
| 11 |
+
self.heads = {r: self.ids(f'<|{r}|>\n') for r in ROLES}
|
| 12 |
+
self.nl = self.ids('\n')
|
| 13 |
+
|
| 14 |
+
def ids(self, text: str) -> list:
|
| 15 |
+
return self.tok.encode(text, add_special_tokens=False).ids
|
| 16 |
+
|
| 17 |
+
def encode(self, turns, generation_prompt: bool=False):
|
| 18 |
+
ids, labels, span = ([], [], None)
|
| 19 |
+
for role, content in turns:
|
| 20 |
+
head = self.heads[role]
|
| 21 |
+
ids += head
|
| 22 |
+
labels += [-100] * len(head)
|
| 23 |
+
if role == 'assistant':
|
| 24 |
+
pieces = content if isinstance(content, tuple) else (content,)
|
| 25 |
+
for j, piece in enumerate(pieces):
|
| 26 |
+
p = self.ids(piece)
|
| 27 |
+
if j == 0 and isinstance(content, tuple):
|
| 28 |
+
span = (len(ids), len(ids) + len(p))
|
| 29 |
+
ids += p
|
| 30 |
+
labels += p
|
| 31 |
+
ids.append(self.eos)
|
| 32 |
+
labels.append(self.eos)
|
| 33 |
+
else:
|
| 34 |
+
body = self.ids(content) + self.nl
|
| 35 |
+
ids += body
|
| 36 |
+
labels += [-100] * len(body)
|
| 37 |
+
if generation_prompt:
|
| 38 |
+
ids += self.heads['assistant']
|
| 39 |
+
labels += [-100] * len(self.heads['assistant'])
|
| 40 |
+
return (ids, labels, span)
|
| 41 |
+
|
| 42 |
+
def decode(self, ids) -> str:
|
| 43 |
+
return self.tok.decode(ids, skip_special_tokens=True)
|
| 44 |
+
|
| 45 |
+
def encode_turns(enc, turns):
|
| 46 |
+
ids, labels = ([], [])
|
| 47 |
+
for role, text, sup in turns:
|
| 48 |
+
head = enc.heads[role]
|
| 49 |
+
ids += head
|
| 50 |
+
labels += [-100] * len(head)
|
| 51 |
+
if role == 'assistant':
|
| 52 |
+
body = enc.ids(text) + [enc.eos]
|
| 53 |
+
ids += body
|
| 54 |
+
labels += body if sup else [-100] * len(body)
|
| 55 |
+
else:
|
| 56 |
+
body = enc.ids(text) + enc.nl
|
| 57 |
+
ids += body
|
| 58 |
+
labels += [-100] * len(body)
|
| 59 |
+
return (ids, labels)
|
tetu/fastgen.py
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
|
| 2 |
+
import re
|
| 3 |
+
|
| 4 |
+
import torch
|
| 5 |
+
|
| 6 |
+
_MASKS = {}
|
| 7 |
+
|
| 8 |
+
def word_mask(enc, vocab_size, device):
|
| 9 |
+
key = (id(enc), vocab_size, str(device))
|
| 10 |
+
if key not in _MASKS:
|
| 11 |
+
m = torch.zeros(vocab_size, dtype=torch.bool)
|
| 12 |
+
for i in range(min(vocab_size, enc.tok.get_vocab_size())):
|
| 13 |
+
piece = enc.tok.decode([i])
|
| 14 |
+
letters = sum((c.isalpha() for c in piece))
|
| 15 |
+
m[i] = letters >= 4 and (not any((c.isdigit() for c in piece)))
|
| 16 |
+
_MASKS[key] = m.to(device)
|
| 17 |
+
return _MASKS[key]
|
| 18 |
+
|
| 19 |
+
def _penalize(logits, counts, mask, presence, frequency):
|
| 20 |
+
if presence <= 0 and frequency <= 0:
|
| 21 |
+
return logits
|
| 22 |
+
c = counts * mask
|
| 23 |
+
return logits - presence * (c > 0) - frequency * c
|
| 24 |
+
_EQ_CACHE = {}
|
| 25 |
+
|
| 26 |
+
def _eq_ids(enc):
|
| 27 |
+
key = id(enc)
|
| 28 |
+
if key not in _EQ_CACHE:
|
| 29 |
+
_EQ_CACHE[key] = {i for i in range(enc.tok.get_vocab_size()) if '=' in enc.tok.decode([i])}
|
| 30 |
+
return _EQ_CACHE[key]
|
| 31 |
+
_TAIL_CACHE = {}
|
| 32 |
+
|
| 33 |
+
def tail_ids(enc, vocab_size, device):
|
| 34 |
+
key = (id(enc), vocab_size, str(device))
|
| 35 |
+
if key not in _TAIL_CACHE:
|
| 36 |
+
dig, nz, sep, zero = (torch.zeros(vocab_size, dtype=torch.bool), torch.zeros(vocab_size, dtype=torch.bool), set(), set())
|
| 37 |
+
for i in range(min(vocab_size, enc.tok.get_vocab_size())):
|
| 38 |
+
piece = enc.tok.decode([i])
|
| 39 |
+
dig[i] = bool(re.match('\\d', piece))
|
| 40 |
+
nz[i] = dig[i] and (not re.fullmatch('0+', piece))
|
| 41 |
+
if piece in (',', '.'):
|
| 42 |
+
sep.add(i)
|
| 43 |
+
if re.fullmatch('0+', piece):
|
| 44 |
+
zero.add(i)
|
| 45 |
+
_TAIL_CACHE[key] = (dig.to(device), nz.to(device), sep, zero)
|
| 46 |
+
return _TAIL_CACHE[key]
|
tetu2/__init__.py
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
|
| 2 |
+
from .model import Tetu2, Tetu2Config
|
tetu2/generate.py
ADDED
|
@@ -0,0 +1,322 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
import math
|
| 4 |
+
import torch
|
| 5 |
+
import torch.nn.functional as F
|
| 6 |
+
from tokenizers import Tokenizer
|
| 7 |
+
from tetu2.model import Tetu2, Tetu2Config, apply_rope, local_attention, log_scale, rms
|
| 8 |
+
ROOT = Path(__file__).resolve().parents[1]
|
| 9 |
+
|
| 10 |
+
def load(path, device='cuda', dtype=torch.bfloat16):
|
| 11 |
+
if Path(path).is_dir():
|
| 12 |
+
import json
|
| 13 |
+
from safetensors.torch import load_file
|
| 14 |
+
meta = json.loads((Path(path) / 'config.json').read_text(encoding='utf-8'))
|
| 15 |
+
ck = dict(meta, model=load_file(str(Path(path) / 'model.safetensors')))
|
| 16 |
+
else:
|
| 17 |
+
ck = torch.load(path, map_location='cpu', weights_only=False)
|
| 18 |
+
model = Tetu2(Tetu2Config(**ck['config']))
|
| 19 |
+
model.load_state_dict({k: v.float() for k, v in ck['model'].items()})
|
| 20 |
+
for name, p in model.named_parameters():
|
| 21 |
+
if not name.startswith('doubt_head'):
|
| 22 |
+
p.data = p.data.to(dtype)
|
| 23 |
+
model.to(device).eval()
|
| 24 |
+
model.meta = {k: ck.get(k) for k in ('step', 'tokens')}
|
| 25 |
+
return (model, Tokenizer.from_file(str(ROOT / 'tokenizer.json')))
|
| 26 |
+
|
| 27 |
+
class Cache:
|
| 28 |
+
|
| 29 |
+
def __init__(self, model):
|
| 30 |
+
L = model.cfg.n_layers
|
| 31 |
+
self.pos = 0
|
| 32 |
+
self.k, self.v, self.kstart = ([None] * L, [None] * L, [0] * L)
|
| 33 |
+
self.tail_tok = None
|
| 34 |
+
self.tail_s = {}
|
| 35 |
+
self.gate = {}
|
| 36 |
+
|
| 37 |
+
def clone(self):
|
| 38 |
+
c = Cache.__new__(Cache)
|
| 39 |
+
c.pos, c.kstart, c.tail_tok = (self.pos, list(self.kstart), self.tail_tok)
|
| 40 |
+
c.k, c.v, c.tail_s = (list(self.k), list(self.v), dict(self.tail_s))
|
| 41 |
+
return c
|
| 42 |
+
|
| 43 |
+
def _engram(eng, x, idx, cache, i):
|
| 44 |
+
B, T, d = x.shape
|
| 45 |
+
ctx = idx if cache.tail_tok is None else torch.cat([cache.tail_tok, idx], dim=1)
|
| 46 |
+
e = eng.table(eng.slots(ctx)[:, -T:]).reshape(B, T, -1).to(x.dtype)
|
| 47 |
+
k, v = (eng.wk(e), eng.wv(e))
|
| 48 |
+
alpha = torch.sigmoid((rms(x) * rms(k)).sum(-1, keepdim=True) / math.sqrt(d))
|
| 49 |
+
s = alpha * v
|
| 50 |
+
cache.gate[i] = alpha[..., 0].float()
|
| 51 |
+
prev = cache.tail_s.get(i)
|
| 52 |
+
full = s if prev is None else torch.cat([prev, s], dim=1)
|
| 53 |
+
c = eng.conv(F.pad(full.transpose(1, 2), (3, 0))).transpose(1, 2)[:, -T:]
|
| 54 |
+
cache.tail_s[i] = full[:, -3:]
|
| 55 |
+
return s + F.silu(c)
|
| 56 |
+
|
| 57 |
+
def _attention(att, x, cos, sin, qs, ve, lam, cache, i, pos):
|
| 58 |
+
B, T, _ = x.shape
|
| 59 |
+
q = att.wq(x).view(B, T, att.H, att.hd).transpose(1, 2)
|
| 60 |
+
k = att.wk(x).view(B, T, att.Hk, att.hd).transpose(1, 2)
|
| 61 |
+
v = att.wv(x).view(B, T, att.Hk, att.hd).transpose(1, 2)
|
| 62 |
+
if ve is not None:
|
| 63 |
+
v = v + lam * ve.to(v.dtype).view(B, T, att.Hk, att.hd).transpose(1, 2)
|
| 64 |
+
q, k = (apply_rope(rms(q), cos, sin), apply_rope(rms(k), cos, sin))
|
| 65 |
+
if qs is not None:
|
| 66 |
+
q = q * qs.to(q.dtype)[:, None]
|
| 67 |
+
first = cache.k[i] is None
|
| 68 |
+
K = k if first else torch.cat([cache.k[i], k], dim=2)
|
| 69 |
+
V = v if first else torch.cat([cache.v[i], v], dim=2)
|
| 70 |
+
r = att.H // att.Hk
|
| 71 |
+
Kr, Vr = (K.repeat_interleave(r, dim=1), V.repeat_interleave(r, dim=1))
|
| 72 |
+
if first:
|
| 73 |
+
if not att.is_global and T > att.window:
|
| 74 |
+
y = local_attention(q, Kr, Vr, att.window)
|
| 75 |
+
else:
|
| 76 |
+
y = F.scaled_dot_product_attention(q, Kr, Vr, is_causal=True)
|
| 77 |
+
else:
|
| 78 |
+
kpos = torch.arange(cache.kstart[i], cache.kstart[i] + K.size(2), device=x.device)
|
| 79 |
+
ok = kpos[None] <= pos[:, None]
|
| 80 |
+
if not att.is_global:
|
| 81 |
+
ok &= kpos[None] > pos[:, None] - att.window
|
| 82 |
+
y = F.scaled_dot_product_attention(q, Kr, Vr, attn_mask=ok if T > 1 or not att.is_global else None)
|
| 83 |
+
if not att.is_global and K.size(2) > att.window + 16:
|
| 84 |
+
drop = K.size(2) - att.window
|
| 85 |
+
K, V = (K[:, :, drop:], V[:, :, drop:])
|
| 86 |
+
cache.kstart[i] += drop
|
| 87 |
+
cache.k[i], cache.v[i] = (K, V)
|
| 88 |
+
y = y.transpose(1, 2).reshape(B, T, -1) * torch.sigmoid(att.wg(x))
|
| 89 |
+
return att.wo(y)
|
| 90 |
+
|
| 91 |
+
@torch.no_grad()
|
| 92 |
+
def forward_cached(model, idx, cache):
|
| 93 |
+
B, T = idx.shape
|
| 94 |
+
pos = torch.arange(cache.pos, cache.pos + T, device=idx.device)
|
| 95 |
+
loc, glo, qs = model.rope(T, pos)
|
| 96 |
+
x = rms(model.tok_emb(idx))
|
| 97 |
+
L, skips = (model.cfg.n_layers, [])
|
| 98 |
+
for i, blk in enumerate(model.blocks):
|
| 99 |
+
if i >= L - L // 2:
|
| 100 |
+
x = x + model.skip_w[L - 1 - i] * skips.pop()
|
| 101 |
+
j = model.ve_layer.get(i)
|
| 102 |
+
ve = model.value_embeds[j](idx) if j is not None else None
|
| 103 |
+
c, s = glo if blk.attn.is_global else loc
|
| 104 |
+
if blk.engram is not None:
|
| 105 |
+
x = x + _engram(blk.engram, rms(x), idx, cache, i)
|
| 106 |
+
x = x + _attention(blk.attn, rms(x), c, s, qs if blk.attn.is_global else None, ve, model.ve_lambda[i], cache, i, pos)
|
| 107 |
+
x = x + blk.mlp(rms(x))
|
| 108 |
+
if i < L // 2:
|
| 109 |
+
skips.append(x)
|
| 110 |
+
ctx = idx if cache.tail_tok is None else torch.cat([cache.tail_tok, idx], dim=1)
|
| 111 |
+
cache.tail_tok = ctx[:, -2:]
|
| 112 |
+
cache.pos += T
|
| 113 |
+
return rms(x)
|
| 114 |
+
|
| 115 |
+
class GraphStep:
|
| 116 |
+
|
| 117 |
+
def __init__(self, model, max_len=8192):
|
| 118 |
+
self.model, self.max_len = (model, max_len)
|
| 119 |
+
cfg, dev = (model.cfg, next(model.parameters()).device)
|
| 120 |
+
dt = model.tok_emb.weight.dtype
|
| 121 |
+
hd, W = (cfg.d_model // cfg.n_heads, cfg.window)
|
| 122 |
+
self.W = W
|
| 123 |
+
self.S = [max_len if b.attn.is_global else W for b in model.blocks]
|
| 124 |
+
self.K = [torch.zeros(1, cfg.n_kv_heads, s, hd, device=dev, dtype=dt) for s in self.S]
|
| 125 |
+
self.V = [torch.zeros_like(k) for k in self.K]
|
| 126 |
+
self.ring_pos = torch.empty(W, dtype=torch.long, device=dev)
|
| 127 |
+
self.gidx = torch.arange(max_len, device=dev)
|
| 128 |
+
self.tok = torch.zeros(1, 1, dtype=torch.long, device=dev)
|
| 129 |
+
self.pos = torch.zeros(1, dtype=torch.long, device=dev)
|
| 130 |
+
self.pad = cfg.vocab_size
|
| 131 |
+
self.tail_tok = torch.empty(1, 2, dtype=torch.long, device=dev)
|
| 132 |
+
self.tail_s = {i: torch.zeros(1, 3, cfg.d_model, device=dev, dtype=dt) for i, b in enumerate(model.blocks) if b.engram is not None}
|
| 133 |
+
self.reset()
|
| 134 |
+
s = torch.cuda.Stream()
|
| 135 |
+
s.wait_stream(torch.cuda.current_stream())
|
| 136 |
+
with torch.cuda.stream(s):
|
| 137 |
+
for _ in range(3):
|
| 138 |
+
self._step()
|
| 139 |
+
torch.cuda.current_stream().wait_stream(s)
|
| 140 |
+
self.graph = torch.cuda.CUDAGraph()
|
| 141 |
+
with torch.cuda.graph(self.graph):
|
| 142 |
+
self.out = self._step()
|
| 143 |
+
self.reset()
|
| 144 |
+
|
| 145 |
+
def reset(self):
|
| 146 |
+
self.ring_pos.fill_(-(1 << 40))
|
| 147 |
+
self.tail_tok.fill_(self.pad)
|
| 148 |
+
for t in self.tail_s.values():
|
| 149 |
+
t.zero_()
|
| 150 |
+
self.pos.zero_()
|
| 151 |
+
|
| 152 |
+
@torch.no_grad()
|
| 153 |
+
def _step(self):
|
| 154 |
+
m, cfg, idx, pos = (self.model, self.model.cfg, self.tok, self.pos)
|
| 155 |
+
self.ring_pos.index_copy_(0, pos % self.W, pos)
|
| 156 |
+
gmask = (self.gidx <= pos)[None, None, None]
|
| 157 |
+
lmask = ((self.ring_pos <= pos) & (self.ring_pos > pos - self.W))[None, None, None]
|
| 158 |
+
loc, glo = ((m.cos_loc[pos], m.sin_loc[pos]), (m.cos_glo[pos], m.sin_glo[pos]))
|
| 159 |
+
qs = log_scale(pos) if cfg.log_scale else None
|
| 160 |
+
x = rms(m.tok_emb(idx))
|
| 161 |
+
L, skips, gates = (cfg.n_layers, [], [])
|
| 162 |
+
for i, blk in enumerate(m.blocks):
|
| 163 |
+
if i >= L - L // 2:
|
| 164 |
+
x = x + m.skip_w[L - 1 - i] * skips.pop()
|
| 165 |
+
if blk.engram is not None:
|
| 166 |
+
eng, xe = (blk.engram, rms(x))
|
| 167 |
+
e = eng.table(eng.slots(torch.cat([self.tail_tok, idx], dim=1))[:, -1:]).reshape(1, 1, -1).to(x.dtype)
|
| 168 |
+
k, v = (eng.wk(e), eng.wv(e))
|
| 169 |
+
alpha = torch.sigmoid((rms(xe) * rms(k)).sum(-1, keepdim=True) / math.sqrt(cfg.d_model))
|
| 170 |
+
s = alpha * v
|
| 171 |
+
gates.append(alpha.reshape(1).float())
|
| 172 |
+
full = torch.cat([self.tail_s[i], s], dim=1)
|
| 173 |
+
x = x + s + F.silu(eng.conv(full.transpose(1, 2)).transpose(1, 2))
|
| 174 |
+
self.tail_s[i].copy_(full[:, 1:])
|
| 175 |
+
att, xa = (blk.attn, rms(x))
|
| 176 |
+
q = att.wq(xa).view(1, 1, att.H, att.hd).transpose(1, 2)
|
| 177 |
+
k = att.wk(xa).view(1, 1, att.Hk, att.hd).transpose(1, 2)
|
| 178 |
+
v = att.wv(xa).view(1, 1, att.Hk, att.hd).transpose(1, 2)
|
| 179 |
+
j = m.ve_layer.get(i)
|
| 180 |
+
if j is not None:
|
| 181 |
+
v = v + m.ve_lambda[i] * m.value_embeds[j](idx).to(v.dtype).view(1, 1, att.Hk, att.hd).transpose(1, 2)
|
| 182 |
+
c, sn = glo if att.is_global else loc
|
| 183 |
+
q, k = (apply_rope(rms(q), c, sn), apply_rope(rms(k), c, sn))
|
| 184 |
+
if att.is_global and qs is not None:
|
| 185 |
+
q = q * qs.to(q.dtype)[:, None]
|
| 186 |
+
slot = pos if att.is_global else pos % self.W
|
| 187 |
+
self.K[i].index_copy_(2, slot, k)
|
| 188 |
+
self.V[i].index_copy_(2, slot, v)
|
| 189 |
+
y = F.scaled_dot_product_attention(q, self.K[i], self.V[i], attn_mask=gmask if att.is_global else lmask, enable_gqa=True)
|
| 190 |
+
x = x + att.wo(y.transpose(1, 2).reshape(1, 1, -1) * torch.sigmoid(att.wg(xa)))
|
| 191 |
+
x = x + blk.mlp(rms(x))
|
| 192 |
+
if i < L // 2:
|
| 193 |
+
skips.append(x)
|
| 194 |
+
self.tail_tok.copy_(torch.cat([self.tail_tok[:, 1:], idx], dim=1))
|
| 195 |
+
h = rms(x)
|
| 196 |
+
logits = m.logits(h)
|
| 197 |
+
return (logits, torch.sigmoid(m.doubt_head(h, logits)), torch.cat(gates).mean())
|
| 198 |
+
|
| 199 |
+
@torch.no_grad()
|
| 200 |
+
def prefill(self, ids):
|
| 201 |
+
assert len(ids) < self.max_len, f'prompt trop long ({len(ids)} ≥ {self.max_len})'
|
| 202 |
+
self.reset()
|
| 203 |
+
dev = self.tok.device
|
| 204 |
+
cache = Cache(self.model)
|
| 205 |
+
h = forward_cached(self.model, torch.tensor([ids], device=dev), cache)[:, -1:]
|
| 206 |
+
T = len(ids)
|
| 207 |
+
for i, S in enumerate(self.S):
|
| 208 |
+
K, V, k0 = (cache.k[i], cache.v[i], cache.kstart[i])
|
| 209 |
+
if S == self.max_len:
|
| 210 |
+
self.K[i][:, :, :T] = K
|
| 211 |
+
self.V[i][:, :, :T] = V
|
| 212 |
+
else:
|
| 213 |
+
p = torch.arange(max(k0, T - self.W), T, device=dev)
|
| 214 |
+
self.K[i].index_copy_(2, p % self.W, K[:, :, p - k0])
|
| 215 |
+
self.V[i].index_copy_(2, p % self.W, V[:, :, p - k0])
|
| 216 |
+
self.ring_pos.index_copy_(0, p % self.W, p)
|
| 217 |
+
t = cache.tail_tok
|
| 218 |
+
self.tail_tok[:, 2 - t.size(1):] = t
|
| 219 |
+
for i, s in cache.tail_s.items():
|
| 220 |
+
self.tail_s[i][:, 3 - s.size(1):] = s
|
| 221 |
+
self.pos.fill_(T)
|
| 222 |
+
logits = self.model.logits(h)
|
| 223 |
+
gate = torch.stack([g[0, -1] for g in cache.gate.values()]).mean()
|
| 224 |
+
return (logits, torch.sigmoid(self.model.doubt_head(h, logits)), gate)
|
| 225 |
+
|
| 226 |
+
def step(self, t):
|
| 227 |
+
self.tok.fill_(t)
|
| 228 |
+
self.graph.replay()
|
| 229 |
+
self.pos += 1
|
| 230 |
+
return self.out
|
| 231 |
+
|
| 232 |
+
class Generator:
|
| 233 |
+
|
| 234 |
+
def __init__(self, model, tok, graph=True, max_len=8192):
|
| 235 |
+
self.model, self.tok = (model, tok)
|
| 236 |
+
self.eos = tok.token_to_id('<|eos|>')
|
| 237 |
+
self.dev = next(model.parameters()).device
|
| 238 |
+
self.gs = GraphStep(model, max_len) if graph and self.dev.type == 'cuda' else None
|
| 239 |
+
|
| 240 |
+
def head(self, h):
|
| 241 |
+
logits = self.model.logits(h)
|
| 242 |
+
return (logits, torch.sigmoid(self.model.doubt_head(h, logits)))
|
| 243 |
+
|
| 244 |
+
def stream(self, text=None, stop=(), ids=None, **kw):
|
| 245 |
+
ids = ids if ids is not None else self.tok.encode(text, add_special_tokens=False).ids
|
| 246 |
+
out, shown = ([], '')
|
| 247 |
+
for t, d in self.tokens(ids, stop=stop, **kw):
|
| 248 |
+
if t == self.eos or t in stop:
|
| 249 |
+
return
|
| 250 |
+
out.append(t)
|
| 251 |
+
text_now = self.tok.decode(out)
|
| 252 |
+
if not text_now.endswith('�'):
|
| 253 |
+
yield (text_now[len(shown):], d)
|
| 254 |
+
shown = text_now
|
| 255 |
+
|
| 256 |
+
@torch.no_grad()
|
| 257 |
+
def tokens(self, ids, max_new=200, temp=0.7, top_k=40, rep=1.15, seed=0, stop=(), calc=False, presence=0.0, frequency=0.0):
|
| 258 |
+
from tetu.calc import calc_suffix
|
| 259 |
+
from tetu.chat import ChatEncoder
|
| 260 |
+
from tetu.fastgen import _eq_ids, _penalize, tail_ids, word_mask
|
| 261 |
+
g = torch.Generator(device=self.dev).manual_seed(seed)
|
| 262 |
+
if self.gs is not None:
|
| 263 |
+
logits, doubt, gate = self.gs.prefill(ids)
|
| 264 |
+
else:
|
| 265 |
+
cache = Cache(self.model)
|
| 266 |
+
logits, doubt = self.head(forward_cached(self.model, torch.tensor([ids], device=self.dev), cache)[:, -1:])
|
| 267 |
+
gate = torch.stack([g_[0, -1] for g_ in cache.gate.values()]).mean()
|
| 268 |
+
V = logits.size(-1)
|
| 269 |
+
if (calc or presence or frequency) and (not hasattr(self, 'enc')):
|
| 270 |
+
self.enc = ChatEncoder(self.tok)
|
| 271 |
+
if calc:
|
| 272 |
+
eq = _eq_ids(self.enc)
|
| 273 |
+
tdig, tnz, tsep, tzero = tail_ids(self.enc, V, self.dev)
|
| 274 |
+
penal = presence > 0 or frequency > 0
|
| 275 |
+
if penal:
|
| 276 |
+
wmask, counts = (word_mask(self.enc, V, self.dev).float(), torch.zeros(V, device=self.dev))
|
| 277 |
+
recent, out, force, tail = (list(ids[-64:]), [], [], 0)
|
| 278 |
+
for _ in range(max_new):
|
| 279 |
+
logits = logits[0, -1].clone()
|
| 280 |
+
if rep != 1.0:
|
| 281 |
+
r = torch.tensor(sorted(set(recent)), device=self.dev)
|
| 282 |
+
lr = logits[r]
|
| 283 |
+
logits[r] = torch.where(lr > 0, lr / rep, lr * rep)
|
| 284 |
+
if penal:
|
| 285 |
+
logits = _penalize(logits, counts, wmask, presence, frequency)
|
| 286 |
+
guarded, tail = (tail, 0)
|
| 287 |
+
if guarded:
|
| 288 |
+
logits = logits.masked_fill(tdig if guarded == 1 else tnz, -float('inf'))
|
| 289 |
+
if force:
|
| 290 |
+
t = force.pop(0)
|
| 291 |
+
tail = 0 if force else 1
|
| 292 |
+
elif temp <= 0:
|
| 293 |
+
t = int(logits.argmax())
|
| 294 |
+
else:
|
| 295 |
+
v, i = torch.topk(logits / temp, top_k)
|
| 296 |
+
t = int(i[torch.multinomial(torch.softmax(v, -1), 1, generator=g)])
|
| 297 |
+
if guarded:
|
| 298 |
+
tail = 2 if t in tsep or (guarded == 2 and t in tzero) else 0
|
| 299 |
+
self.last_gate = float(gate)
|
| 300 |
+
yield (t, float(doubt[0, -1]))
|
| 301 |
+
if t == self.eos or t in stop:
|
| 302 |
+
return
|
| 303 |
+
out.append(t)
|
| 304 |
+
if calc and (not force) and (t in eq or (len(out) > 1 and out[-2] in eq)):
|
| 305 |
+
add = calc_suffix(self.tok.decode(out[-48:]))
|
| 306 |
+
if add:
|
| 307 |
+
force = self.enc.ids(add)
|
| 308 |
+
if penal:
|
| 309 |
+
counts[t] += 1
|
| 310 |
+
recent = (recent + [t])[-64:]
|
| 311 |
+
if self.gs is not None:
|
| 312 |
+
logits, doubt, gate = self.gs.step(t)
|
| 313 |
+
else:
|
| 314 |
+
logits, doubt = self.head(forward_cached(self.model, torch.tensor([[t]], device=self.dev), cache))
|
| 315 |
+
gate = torch.stack([g_[0, -1] for g_ in cache.gate.values()]).mean()
|
| 316 |
+
|
| 317 |
+
def complete(self, text, **kw):
|
| 318 |
+
pieces, doubts = ([], [])
|
| 319 |
+
for p, d in self.stream(text, **kw):
|
| 320 |
+
pieces.append(p)
|
| 321 |
+
doubts.append(d)
|
| 322 |
+
return (''.join(pieces), sum(doubts) / max(1, len(doubts)))
|
tetu2/model.py
ADDED
|
@@ -0,0 +1,313 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Têtu-2 — inference code (tetu-AI). License: CC BY-NC 4.0, see README.md.
|
| 2 |
+
from dataclasses import asdict, dataclass
|
| 3 |
+
import math
|
| 4 |
+
from typing import Optional
|
| 5 |
+
import torch
|
| 6 |
+
import torch.nn as nn
|
| 7 |
+
import torch.nn.functional as F
|
| 8 |
+
|
| 9 |
+
@dataclass
|
| 10 |
+
class Tetu2Config:
|
| 11 |
+
vocab_size: int = 32768
|
| 12 |
+
d_model: int = 768
|
| 13 |
+
n_layers: int = 12
|
| 14 |
+
n_heads: int = 12
|
| 15 |
+
n_kv_heads: int = 4
|
| 16 |
+
mlp_mult: int = 4
|
| 17 |
+
max_seq_len: int = 32768
|
| 18 |
+
window: int = 1024
|
| 19 |
+
global_every: int = 4
|
| 20 |
+
rope_local: float = 10000.0
|
| 21 |
+
rope_global: float = 500000.0
|
| 22 |
+
log_scale: bool = True
|
| 23 |
+
engram_layers: tuple = (1, 5)
|
| 24 |
+
engram_orders: tuple = (2, 3)
|
| 25 |
+
engram_heads: int = 2
|
| 26 |
+
engram_slots: int = 524287
|
| 27 |
+
engram_dim: int = 128
|
| 28 |
+
n_value_embeds: int = 3
|
| 29 |
+
softcap: float = 30.0
|
| 30 |
+
mtp_weight: float = 0.3
|
| 31 |
+
doubt_weight: float = 0.1
|
| 32 |
+
|
| 33 |
+
def __post_init__(self):
|
| 34 |
+
self.engram_layers = tuple(self.engram_layers)
|
| 35 |
+
self.engram_orders = tuple(self.engram_orders)
|
| 36 |
+
|
| 37 |
+
def is_global(self, i):
|
| 38 |
+
return (i + 1) % self.global_every == 0
|
| 39 |
+
|
| 40 |
+
def to_dict(self):
|
| 41 |
+
return asdict(self)
|
| 42 |
+
|
| 43 |
+
def rms(x):
|
| 44 |
+
return F.rms_norm(x, (x.size(-1),))
|
| 45 |
+
|
| 46 |
+
def rope_table(n, hd, theta):
|
| 47 |
+
inv = 1.0 / theta ** (torch.arange(0, hd, 2, dtype=torch.float32) / hd)
|
| 48 |
+
f = torch.outer(torch.arange(n, dtype=torch.float32), inv)
|
| 49 |
+
return (f.cos(), f.sin())
|
| 50 |
+
|
| 51 |
+
def apply_rope(x, cos, sin):
|
| 52 |
+
x1, x2 = x.chunk(2, dim=-1)
|
| 53 |
+
return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1).type_as(x)
|
| 54 |
+
|
| 55 |
+
def log_scale(pos):
|
| 56 |
+
return (torch.log(pos.float() + 1) / math.log(1024)).clamp_min(1.0)
|
| 57 |
+
|
| 58 |
+
def local_attention(q, k, v, W):
|
| 59 |
+
B, H, T, D = q.shape
|
| 60 |
+
n = -(-T // W)
|
| 61 |
+
pad = n * W - T
|
| 62 |
+
q, k, v = (F.pad(t, (0, 0, 0, pad)) for t in (q, k, v))
|
| 63 |
+
a = torch.arange(W, device=q.device)[:, None]
|
| 64 |
+
b = torch.arange(2 * W, device=q.device)[None]
|
| 65 |
+
base = (b > a) & (b <= a + W)
|
| 66 |
+
if not torch.is_grad_enabled():
|
| 67 |
+
out = []
|
| 68 |
+
for i in range(n):
|
| 69 |
+
s0 = max(0, (i - 1) * W)
|
| 70 |
+
out.append(F.scaled_dot_product_attention(q[:, :, i * W:(i + 1) * W], k[:, :, s0:(i + 1) * W], v[:, :, s0:(i + 1) * W], attn_mask=base if i else base[:, W:]))
|
| 71 |
+
return torch.cat(out, dim=2)[:, :, :T]
|
| 72 |
+
qc, kc, vc = (t.view(B, H, n, W, D) for t in (q, k, v))
|
| 73 |
+
kk = torch.cat([F.pad(kc, (0, 0, 0, 0, 1, 0))[:, :, :n], kc], dim=3)
|
| 74 |
+
vv = torch.cat([F.pad(vc, (0, 0, 0, 0, 1, 0))[:, :, :n], vc], dim=3)
|
| 75 |
+
first = torch.arange(n, device=q.device)[:, None, None] == 0
|
| 76 |
+
m = (base[None] & ~(first & (b < W)[None])).unsqueeze(1)
|
| 77 |
+
flat = lambda t, L: t.permute(0, 2, 1, 3, 4).reshape(B * n, H, L, D)
|
| 78 |
+
y = F.scaled_dot_product_attention(flat(qc, W), flat(kk, 2 * W), flat(vv, 2 * W), attn_mask=m.expand(B, n, 1, W, 2 * W).reshape(B * n, 1, W, 2 * W))
|
| 79 |
+
return y.view(B, n, H, W, D).permute(0, 2, 1, 3, 4).reshape(B, H, n * W, D)[:, :, :T]
|
| 80 |
+
|
| 81 |
+
class Linear(nn.Linear):
|
| 82 |
+
|
| 83 |
+
def __init__(self, i, o, zero=False):
|
| 84 |
+
super().__init__(i, o, bias=False)
|
| 85 |
+
with torch.no_grad():
|
| 86 |
+
if zero:
|
| 87 |
+
self.weight.zero_()
|
| 88 |
+
else:
|
| 89 |
+
b = math.sqrt(3) * 0.5 / math.sqrt(i)
|
| 90 |
+
self.weight.uniform_(-b, b)
|
| 91 |
+
|
| 92 |
+
class Engram(nn.Module):
|
| 93 |
+
P = 2147483647
|
| 94 |
+
|
| 95 |
+
def __init__(self, cfg: Tetu2Config):
|
| 96 |
+
super().__init__()
|
| 97 |
+
self.orders, self.K, self.S, self.dm = (cfg.engram_orders, cfg.engram_heads, cfg.engram_slots, cfg.engram_dim)
|
| 98 |
+
self.n_tab = len(self.orders) * self.K
|
| 99 |
+
self.pad_id = cfg.vocab_size
|
| 100 |
+
self.table = nn.Embedding(self.n_tab * self.S, self.dm)
|
| 101 |
+
nn.init.normal_(self.table.weight, std=1.0)
|
| 102 |
+
g = torch.Generator().manual_seed(1234)
|
| 103 |
+
mult = torch.randint(1 << 20, 1 << 30, (self.n_tab, max(self.orders)), generator=g) | 1
|
| 104 |
+
self.register_buffer('mult', mult, persistent=False)
|
| 105 |
+
self.register_buffer('offset', torch.arange(self.n_tab) * self.S, persistent=False)
|
| 106 |
+
d, de = (cfg.d_model, self.n_tab * self.dm)
|
| 107 |
+
self.wk = Linear(de, d)
|
| 108 |
+
self.wv = Linear(de, d, zero=True)
|
| 109 |
+
self.conv = nn.Conv1d(d, d, 4, groups=d, bias=False)
|
| 110 |
+
nn.init.zeros_(self.conv.weight)
|
| 111 |
+
|
| 112 |
+
def slots(self, idx):
|
| 113 |
+
T = idx.size(1)
|
| 114 |
+
grams = [idx] + [F.pad(idx, (j, 0), value=self.pad_id)[:, :T] for j in range(1, max(self.orders))]
|
| 115 |
+
out, t = ([], 0)
|
| 116 |
+
for o in self.orders:
|
| 117 |
+
for _ in range(self.K):
|
| 118 |
+
h = torch.zeros_like(idx)
|
| 119 |
+
for j in range(o):
|
| 120 |
+
h = (h * 1000003 + (grams[j] + 1) * self.mult[t, j]) % self.P
|
| 121 |
+
out.append(h % self.S + self.offset[t])
|
| 122 |
+
t += 1
|
| 123 |
+
return torch.stack(out, dim=-1)
|
| 124 |
+
|
| 125 |
+
def forward(self, x, idx):
|
| 126 |
+
B, T, d = x.shape
|
| 127 |
+
e = self.table(self.slots(idx)).reshape(B, T, -1).to(x.dtype)
|
| 128 |
+
k, v = (self.wk(e), self.wv(e))
|
| 129 |
+
alpha = torch.sigmoid((rms(x) * rms(k)).sum(-1, keepdim=True) / math.sqrt(d))
|
| 130 |
+
s = alpha * v
|
| 131 |
+
c = self.conv(F.pad(s.transpose(1, 2), (3, 0))).transpose(1, 2)
|
| 132 |
+
if not torch.compiler.is_compiling():
|
| 133 |
+
self.last_gate = alpha.detach().float().mean()
|
| 134 |
+
return s + F.silu(c)
|
| 135 |
+
|
| 136 |
+
class Attention(nn.Module):
|
| 137 |
+
|
| 138 |
+
def __init__(self, cfg: Tetu2Config, is_global: bool):
|
| 139 |
+
super().__init__()
|
| 140 |
+
self.H, self.Hk = (cfg.n_heads, cfg.n_kv_heads)
|
| 141 |
+
self.hd = cfg.d_model // cfg.n_heads
|
| 142 |
+
d = cfg.d_model
|
| 143 |
+
self.wq = Linear(d, self.H * self.hd)
|
| 144 |
+
self.wk = Linear(d, self.Hk * self.hd)
|
| 145 |
+
self.wv = Linear(d, self.Hk * self.hd)
|
| 146 |
+
self.wg = Linear(d, self.H * self.hd)
|
| 147 |
+
self.wo = Linear(self.H * self.hd, d, zero=True)
|
| 148 |
+
self.is_global, self.window = (is_global, cfg.window)
|
| 149 |
+
|
| 150 |
+
def forward(self, x, cos, sin, qs=None, ve=None, lam=None):
|
| 151 |
+
B, T, _ = x.shape
|
| 152 |
+
q = self.wq(x).view(B, T, self.H, self.hd).transpose(1, 2)
|
| 153 |
+
k = self.wk(x).view(B, T, self.Hk, self.hd).transpose(1, 2)
|
| 154 |
+
v = self.wv(x).view(B, T, self.Hk, self.hd).transpose(1, 2)
|
| 155 |
+
if ve is not None:
|
| 156 |
+
v = v + lam * ve.to(v.dtype).view(B, T, self.Hk, self.hd).transpose(1, 2)
|
| 157 |
+
q, k = (apply_rope(rms(q), cos, sin), apply_rope(rms(k), cos, sin))
|
| 158 |
+
if qs is not None:
|
| 159 |
+
q = q * qs.to(q.dtype)[:, None]
|
| 160 |
+
r = self.H // self.Hk
|
| 161 |
+
k, v = (k.repeat_interleave(r, dim=1), v.repeat_interleave(r, dim=1))
|
| 162 |
+
if not self.is_global and T > self.window:
|
| 163 |
+
y = local_attention(q, k, v, self.window)
|
| 164 |
+
else:
|
| 165 |
+
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
| 166 |
+
y = y.transpose(1, 2).reshape(B, T, -1) * torch.sigmoid(self.wg(x))
|
| 167 |
+
return self.wo(y)
|
| 168 |
+
|
| 169 |
+
class MLP(nn.Module):
|
| 170 |
+
|
| 171 |
+
def __init__(self, d, h):
|
| 172 |
+
super().__init__()
|
| 173 |
+
self.w1, self.w2 = (Linear(d, h), Linear(h, d, zero=True))
|
| 174 |
+
|
| 175 |
+
def forward(self, x):
|
| 176 |
+
return self.w2(F.relu(self.w1(x)).square())
|
| 177 |
+
|
| 178 |
+
class Block(nn.Module):
|
| 179 |
+
|
| 180 |
+
def __init__(self, cfg: Tetu2Config, i: int):
|
| 181 |
+
super().__init__()
|
| 182 |
+
self.attn = Attention(cfg, cfg.is_global(i))
|
| 183 |
+
self.mlp = MLP(cfg.d_model, cfg.mlp_mult * cfg.d_model)
|
| 184 |
+
self.engram = Engram(cfg) if i in cfg.engram_layers else None
|
| 185 |
+
|
| 186 |
+
def forward(self, x, idx, cos, sin, qs, ve, lam):
|
| 187 |
+
if self.engram is not None:
|
| 188 |
+
x = x + self.engram(rms(x), idx)
|
| 189 |
+
x = x + self.attn(rms(x), cos, sin, qs, ve, lam)
|
| 190 |
+
return x + self.mlp(rms(x))
|
| 191 |
+
|
| 192 |
+
class DoubtHead(nn.Module):
|
| 193 |
+
|
| 194 |
+
def __init__(self, d):
|
| 195 |
+
super().__init__()
|
| 196 |
+
self.net = nn.Sequential(nn.Linear(d + 2, 128), nn.ReLU(), nn.Linear(128, 1))
|
| 197 |
+
|
| 198 |
+
def forward(self, h, logits):
|
| 199 |
+
with torch.no_grad():
|
| 200 |
+
lp = F.log_softmax(logits.float(), dim=-1)
|
| 201 |
+
conf = torch.stack([lp.max(-1).values, -(lp.exp() * lp).sum(-1)], dim=-1) / 4
|
| 202 |
+
return self.net(torch.cat([rms(h.detach()).float(), conf], dim=-1)).squeeze(-1)
|
| 203 |
+
|
| 204 |
+
@dataclass
|
| 205 |
+
class Output:
|
| 206 |
+
logits: torch.Tensor
|
| 207 |
+
loss: Optional[torch.Tensor] = None
|
| 208 |
+
lm_loss: Optional[torch.Tensor] = None
|
| 209 |
+
mtp_loss: Optional[torch.Tensor] = None
|
| 210 |
+
doubt_loss: Optional[torch.Tensor] = None
|
| 211 |
+
doubt: Optional[torch.Tensor] = None
|
| 212 |
+
|
| 213 |
+
class Tetu2(nn.Module):
|
| 214 |
+
|
| 215 |
+
def __init__(self, cfg: Tetu2Config):
|
| 216 |
+
super().__init__()
|
| 217 |
+
self.cfg = cfg
|
| 218 |
+
d, L = (cfg.d_model, cfg.n_layers)
|
| 219 |
+
kv = cfg.n_kv_heads * (d // cfg.n_heads)
|
| 220 |
+
self.tok_emb = nn.Embedding(cfg.vocab_size, d)
|
| 221 |
+
nn.init.normal_(self.tok_emb.weight, std=1.0)
|
| 222 |
+
self.value_embeds = nn.ModuleList((nn.Embedding(cfg.vocab_size, kv) for _ in range(cfg.n_value_embeds)))
|
| 223 |
+
for e in self.value_embeds:
|
| 224 |
+
nn.init.normal_(e.weight, std=1.0)
|
| 225 |
+
n = cfg.n_value_embeds
|
| 226 |
+
self.ve_layer = {i: i for i in range(n)} | {L - n + i: i for i in range(n)}
|
| 227 |
+
self.ve_lambda = nn.Parameter(torch.full((L,), 0.5))
|
| 228 |
+
self.blocks = nn.ModuleList((Block(cfg, i) for i in range(L)))
|
| 229 |
+
self.skip_w = nn.Parameter(torch.ones(L // 2))
|
| 230 |
+
self.lm_head = Linear(d, cfg.vocab_size, zero=True)
|
| 231 |
+
self.mtp_in = Linear(2 * d, d)
|
| 232 |
+
self.mtp_mlp = MLP(d, cfg.mlp_mult * d)
|
| 233 |
+
self.doubt_head = DoubtHead(d)
|
| 234 |
+
hd = d // cfg.n_heads
|
| 235 |
+
for name, theta in (('loc', cfg.rope_local), ('glo', cfg.rope_global)):
|
| 236 |
+
c, s = rope_table(cfg.max_seq_len, hd, theta)
|
| 237 |
+
self.register_buffer(f'cos_{name}', c, persistent=False)
|
| 238 |
+
self.register_buffer(f'sin_{name}', s, persistent=False)
|
| 239 |
+
|
| 240 |
+
def num_params(self):
|
| 241 |
+
return sum((p.numel() for p in self.parameters()))
|
| 242 |
+
|
| 243 |
+
def compute_params(self):
|
| 244 |
+
lookup = self.tok_emb.weight.numel() + sum((e.weight.numel() for e in self.value_embeds))
|
| 245 |
+
lookup += sum((b.engram.table.weight.numel() for b in self.blocks if b.engram is not None))
|
| 246 |
+
return self.num_params() - lookup
|
| 247 |
+
|
| 248 |
+
def param_groups(self):
|
| 249 |
+
tables, head, scal, muon = ([], [], [], [])
|
| 250 |
+
for n, p in self.named_parameters():
|
| 251 |
+
if n.startswith(('tok_emb', 'value_embeds')) or n.endswith('engram.table.weight'):
|
| 252 |
+
tables.append(p)
|
| 253 |
+
elif n.startswith('lm_head'):
|
| 254 |
+
head.append(p)
|
| 255 |
+
elif p.dim() == 2 and (not n.startswith('doubt_head')):
|
| 256 |
+
muon.append(p)
|
| 257 |
+
else:
|
| 258 |
+
scal.append(p)
|
| 259 |
+
return {'tables': tables, 'head': head, 'scalars': scal, 'muon': muon}
|
| 260 |
+
|
| 261 |
+
def rope(self, T, pos=None):
|
| 262 |
+
p = torch.arange(T, device=self.cos_loc.device) if pos is None else pos
|
| 263 |
+
last = T - 1 if pos is None else int(pos[-1])
|
| 264 |
+
qs = log_scale(p) if self.cfg.log_scale and last >= 1024 else None
|
| 265 |
+
return ((self.cos_loc[p], self.sin_loc[p]), (self.cos_glo[p], self.sin_glo[p]), qs)
|
| 266 |
+
|
| 267 |
+
def hidden(self, idx):
|
| 268 |
+
B, T = idx.shape
|
| 269 |
+
loc, glo, qs = self.rope(T)
|
| 270 |
+
x = rms(self.tok_emb(idx))
|
| 271 |
+
L, skips = (self.cfg.n_layers, [])
|
| 272 |
+
for i, blk in enumerate(self.blocks):
|
| 273 |
+
if i >= L - L // 2:
|
| 274 |
+
x = x + self.skip_w[L - 1 - i] * skips.pop()
|
| 275 |
+
j = self.ve_layer.get(i)
|
| 276 |
+
ve = self.value_embeds[j](idx) if j is not None else None
|
| 277 |
+
c, s = glo if blk.attn.is_global else loc
|
| 278 |
+
x = blk(x, idx, c, s, qs if blk.attn.is_global else None, ve, self.ve_lambda[i])
|
| 279 |
+
if i < L // 2:
|
| 280 |
+
skips.append(x)
|
| 281 |
+
return rms(x)
|
| 282 |
+
|
| 283 |
+
def logits(self, h):
|
| 284 |
+
z = self.lm_head(h).float()
|
| 285 |
+
return self.cfg.softcap * torch.tanh(z / self.cfg.softcap)
|
| 286 |
+
|
| 287 |
+
def mtp_logits(self, h, nxt):
|
| 288 |
+
m = self.mtp_in(torch.cat([h, rms(self.tok_emb(nxt))], dim=-1))
|
| 289 |
+
m = m + self.mtp_mlp(rms(m))
|
| 290 |
+
return self.logits(rms(m))
|
| 291 |
+
|
| 292 |
+
def forward(self, idx, targets=None):
|
| 293 |
+
h = self.hidden(idx)
|
| 294 |
+
logits = self.logits(h)
|
| 295 |
+
doubt_logit = self.doubt_head(h, logits)
|
| 296 |
+
out = Output(logits=logits, doubt=torch.sigmoid(doubt_logit))
|
| 297 |
+
if targets is None:
|
| 298 |
+
return out
|
| 299 |
+
mask = targets != -100
|
| 300 |
+
out.lm_loss = F.cross_entropy(logits.flatten(0, 1), targets.flatten(), ignore_index=-100)
|
| 301 |
+
t2 = torch.cat([targets[:, 1:], torch.full_like(targets[:, :1], -100)], dim=1)
|
| 302 |
+
t2 = torch.where(mask, t2, torch.full_like(t2, -100))
|
| 303 |
+
lm2 = self.mtp_logits(h, targets.clamp_min(0))
|
| 304 |
+
out.mtp_loss = F.cross_entropy(lm2.flatten(0, 1), t2.flatten(), ignore_index=-100)
|
| 305 |
+
with torch.no_grad():
|
| 306 |
+
wrong = (logits.argmax(-1) != targets).float()
|
| 307 |
+
bce = F.binary_cross_entropy_with_logits(doubt_logit, wrong, reduction='none')
|
| 308 |
+
out.doubt_loss = (bce * mask).sum() / mask.sum().clamp_min(1)
|
| 309 |
+
out.loss = out.lm_loss + self.cfg.mtp_weight * out.mtp_loss + self.cfg.doubt_weight * out.doubt_loss
|
| 310 |
+
return out
|
| 311 |
+
|
| 312 |
+
def engram_gates(self):
|
| 313 |
+
return [b.engram.last_gate for b in self.blocks if b.engram is not None and hasattr(b.engram, 'last_gate')]
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|