levossadtchi commited on
Commit
c8f2f25
·
verified ·
1 Parent(s): 2091236

Replace the research code with the clean code from GitHub; drop an unused network

Browse files
README.md CHANGED
@@ -15,28 +15,35 @@ Turns a text file with LaTeX formulas into handwritten notebook pages. The style
15
 
16
  ![A page written by a fine-tuned version of the models](examples/demo_authors_hand/page_05.png)
17
 
18
- *The page above is in the author's hand, written by personal fine-tuned models that are **not** in this repository. For what these weights give out of the box, see `examples/demo_released_weights/`.*
19
 
20
  The output is pen strokes, not pixels, so the PDF is sharp at any zoom. The models write Russian prose and mathematics.
21
 
 
 
22
  ## Quick start
23
 
24
- Requires Python with `torch`, `numpy` and `pillow`, plus Node.js for the formula layout.
25
 
26
  ```bash
 
 
27
  pip install -r requirements.txt
28
  (cd tools/mathjax && npm ci)
29
- python scripts/write_pages.py examples/physics_oscillator.txt out
 
30
  ```
31
 
32
  This writes to `out/` (the folder must not exist yet):
33
 
34
  - `pages.pdf`: vector, every stroke a smooth curve
35
  - `page_NN.svg` and `page_NN.png` (300 dpi)
36
- - `strokes.json`: the strokes, re-render with `scripts/render_pages.py`
37
- - `report.json`: per-line timing and where each glyph came from
 
 
38
 
39
- Tested only on Python 3.14, torch 2.13, Apple M5 Max (MPS), where 11 pages took about 7 minutes with another job running on the machine. The code selects CUDA or CPU when available, but this package was not run on them.
40
 
41
  ## Writing in your own hand
42
 
@@ -47,10 +54,11 @@ You need one line of your handwriting: 30-50 characters of ordinary prose. There
47
  Crop the photo to **one line** and run it through the [tracer](trace/README.md), which recovers the pen strokes from the image.
48
 
49
  ```bash
 
50
  pip install -r trace/requirements.txt
51
  python trace/scrawler_trace.py my_line.png --out my_line.json
52
- python scripts/make_ref.py my_line.json my_line.npy
53
- REF=$PWD/my_line.npy python scripts/write_pages.py notes.txt out
54
  ```
55
 
56
  For a good result:
@@ -63,18 +71,18 @@ For a good result:
63
 
64
  *Top: a real scan on squared paper (HWR200, open data). Middle: the trace. Bottom: new text and formulas in that style. One example, not a measurement.*
65
 
66
- The tracer weights take 62 MB and have their own licence caveat, see [its training data](trace/TRAINING_DATA.md).
67
 
68
  ### From tablet strokes
69
 
70
  If you have the pen strokes already, skip the tracer. Export them as a JSON list of strokes, each a list of `[x, y]` points with y pointing down, in writing order.
71
 
72
  ```bash
73
- python scripts/make_ref.py my_line.json my_line.npy
74
- REF=$PWD/my_line.npy python scripts/write_pages.py notes.txt out
75
  ```
76
 
77
- The default reference is one traced line of the open HWR200 dataset.
78
 
79
  ## Input format
80
 
@@ -86,7 +94,7 @@ $x(t) = A\cos(\omega_0 t + \varphi)$
86
  An empty line above makes a paragraph gap.
87
  ```
88
 
89
- Long lines are wrapped at about 42 characters, never inside `$...$`. Formulas use the LaTeX that MathJax understands.
90
 
91
  ## How it works
92
 
@@ -96,21 +104,31 @@ Long lines are wrapped at about 42 characters, never inside `$...$`. Formulas us
96
  | formula only | **Formula module**: MathJax places every glyph, a diffusion **glyph model** draws each one, and a classifier picks the most recognisable variant |
97
  | text + `$formula$` | The text model writes the line, then the formula spans are replaced by the formula module |
98
 
99
- The style vector comes from a line-level writer encoder. Fraction bars, radicals, arrows and `|x|` are drawn as pen lines.
100
 
101
- ## Files
 
 
 
 
 
 
 
 
 
 
 
 
 
102
 
103
  | Path | What | Size |
104
  |---|---|---|
105
- | `ckpt_dl/b4styc/fp16/step003000_fp16.pt` | text model | 44 MB |
106
- | `ckpt_dl/lstmal_v6isl3/step100000_fp16.pt` | base network the text model depends on | 18 MB |
107
- | `ckpt_dl/planner7/best.pt` | duration planner | 13 MB |
108
  | `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
109
  | `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
110
  | `ckpt_dl/glyph_cls2.pt` | glyph classifier | 1 MB |
111
- | `trace/weights/` | tracer and ink restorer, optional: only for the photo route | 62 MB |
112
-
113
- `scripts/` also holds training code. It is research code as it was used, not a clean library, and its comments are partly in Russian. The training data is not included.
114
 
115
  ## Evaluation
116
 
@@ -138,6 +156,7 @@ Style similarity by the writer encoder (cosine to the writer's real line, one re
138
  - About 2% of glyphs are the best of the variants the classifier rejected, and may be misshapen.
139
  - Rare symbols without their own class are drawn as a similar one (for example `\simeq` as `\approx`).
140
  - A heading of one short word may come out smaller than the other lines.
 
141
 
142
  ## Training data
143
 
@@ -151,4 +170,4 @@ This is for notes, experiments and research. Do not use it to forge signatures,
151
 
152
  ## License
153
 
154
- Code: Apache-2.0 (`LICENSE`). Weights: CC BY-NC 4.0 (`LICENSE-WEIGHTS`). See `NOTICE` for third-party components. The tracer weights carry further conditions from their training data: [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md).
 
15
 
16
  ![A page written by a fine-tuned version of the models](examples/demo_authors_hand/page_05.png)
17
 
18
+ *The page above is in the author's hand, written by personal fine-tuned models that are **not** released. For what the released weights give out of the box, see [this page](examples/demo_released_weights/page_05.png).*
19
 
20
  The output is pen strokes, not pixels, so the PDF is sharp at any zoom. The models write Russian prose and mathematics.
21
 
22
+ This repository holds the weights and a copy of the code for writing pages. The code lives on [GitHub](https://github.com/Levos06/Scrawler-open), and `download_weights.py` there fetches the weights from here. Training code and data are not included.
23
+
24
  ## Quick start
25
 
26
+ Requires Python and, for formulas, Node.js.
27
 
28
  ```bash
29
+ git clone https://github.com/Levos06/Scrawler-open.git
30
+ cd Scrawler-open
31
  pip install -r requirements.txt
32
  (cd tools/mathjax && npm ci)
33
+ python download_weights.py
34
+ python write_pages.py examples/physics_oscillator.txt out
35
  ```
36
 
37
  This writes to `out/` (the folder must not exist yet):
38
 
39
  - `pages.pdf`: vector, every stroke a smooth curve
40
  - `page_NN.svg` and `page_NN.png` (300 dpi)
41
+ - `strokes.json`: the strokes; `python render_pages.py out/strokes.json thick --pen 1.4` redraws them without the models
42
+ - `report.json`: per-line timing and what wrote each glyph
43
+
44
+ The code selects CUDA, MPS or CPU by itself; set `SCRAWLER_DEVICE` to force one. Results depend on the device: the same seed gives the same pages only on the same device. Tested on Apple Silicon (MPS and CPU); the CUDA path has not been run with this code.
45
 
46
+ The example above is 11 pages and took under 7 minutes on an Apple M5 Max (MPS). The CPU is much slower: a 4-page test with many formulas took 74 minutes.
47
 
48
  ## Writing in your own hand
49
 
 
54
  Crop the photo to **one line** and run it through the [tracer](trace/README.md), which recovers the pen strokes from the image.
55
 
56
  ```bash
57
+ python download_weights.py --tracer
58
  pip install -r trace/requirements.txt
59
  python trace/scrawler_trace.py my_line.png --out my_line.json
60
+ python make_ref.py my_line.json my_line.npy
61
+ python write_pages.py notes.txt out --ref my_line.npy
62
  ```
63
 
64
  For a good result:
 
71
 
72
  *Top: a real scan on squared paper (HWR200, open data). Middle: the trace. Bottom: new text and formulas in that style. One example, not a measurement.*
73
 
74
+ The tracer weights have their own licence caveat, see [its training data](trace/TRAINING_DATA.md).
75
 
76
  ### From tablet strokes
77
 
78
  If you have the pen strokes already, skip the tracer. Export them as a JSON list of strokes, each a list of `[x, y]` points with y pointing down, in writing order.
79
 
80
  ```bash
81
+ python make_ref.py my_line.json my_line.npy
82
+ python write_pages.py notes.txt out --ref my_line.npy
83
  ```
84
 
85
+ The default style is one traced line of the open HWR200 dataset.
86
 
87
  ## Input format
88
 
 
94
  An empty line above makes a paragraph gap.
95
  ```
96
 
97
+ Long lines are wrapped at about 42 characters (`--wrap`), never inside `$...$`. Formulas use the LaTeX that MathJax understands. `python write_pages.py --help` lists the other options.
98
 
99
  ## How it works
100
 
 
104
  | formula only | **Formula module**: MathJax places every glyph, a diffusion **glyph model** draws each one, and a classifier picks the most recognisable variant |
105
  | text + `$formula$` | The text model writes the line, then the formula spans are replaced by the formula module |
106
 
107
+ The style vector comes from a line-level writer encoder. Fraction bars, radicals, arrows and `|x|` are drawn as pen lines by code, not by a model, so they do not take the writer's style.
108
 
109
+ | Path | What |
110
+ |---|---|
111
+ | `write_pages.py` | text file -> pages: wrapping, page layout, the order of the steps |
112
+ | `scrawler/text_model.py` | text model and duration planner |
113
+ | `scrawler/style_encoder.py` | one line -> style vector |
114
+ | `scrawler/glyph_model.py` | glyph model and the classifier that picks variants |
115
+ | `scrawler/formula.py` | formula layout, pen-line marks, inserting formulas into text |
116
+ | `scrawler/render.py`, `render_pages.py` | strokes -> PDF, SVG, PNG |
117
+ | `make_ref.py` | your strokes -> a style line |
118
+ | `trace/` | the tracer, one file |
119
+ | `tools/mathjax/layout.mjs` | LaTeX -> glyph boxes |
120
+ | `data/glyph_stroke_counts.json` | per glyph class, how many real glyphs had 1, 2, ... strokes; counts only |
121
+
122
+ ## Weights
123
 
124
  | Path | What | Size |
125
  |---|---|---|
126
+ | `ckpt_dl/b4styc/fp16/step003000_fp16.pt` | text model | 45 MB |
127
+ | `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
 
128
  | `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
129
  | `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
130
  | `ckpt_dl/glyph_cls2.pt` | glyph classifier | 1 MB |
131
+ | `trace/weights/` | tracer and ink restorer, only for the photo route | 62 MB |
 
 
132
 
133
  ## Evaluation
134
 
 
156
  - About 2% of glyphs are the best of the variants the classifier rejected, and may be misshapen.
157
  - Rare symbols without their own class are drawn as a similar one (for example `\simeq` as `\approx`).
158
  - A heading of one short word may come out smaller than the other lines.
159
+ - A line of several formulas with almost no plain text between them is written by the text model alone instead of the formula module (`fallback_text_model` in `report.json`).
160
 
161
  ## Training data
162
 
 
170
 
171
  ## License
172
 
173
+ Code: Apache-2.0 ([LICENSE](LICENSE)). Weights: CC BY-NC 4.0 ([LICENSE-WEIGHTS](LICENSE-WEIGHTS)); the tracer weights carry further conditions from their training data, see [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md). Third-party components: [NOTICE](NOTICE).
ckpt_dl/lstmal_v6isl3/step100000_fp16.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:eea3551c05cefe2558690550a990ae5f87f32ef7da0ebbe7a24b47ab0b298286
3
- size 18912713
 
 
 
 
data/ink_lines_u2/vocab.json DELETED
@@ -1 +0,0 @@
1
- ["<pad>", "<bos>", "<eos>", "{", "}", "^", "_", "\\frac", "1", "2", "x", "=", ")", "(", "i", "t", "a", "-", "0", "r", "m", "d", "3", "n", "\\sqrt", "4", "+", "5", "6", ",", "k", "9", "7", "\\\\", "\\begin", "\\end", "8", "f", "\\cdot", "\\partial", "c", "|", "s", "p", "y", "\\int", "&", "e", "A", "v", "/", ".", "\\hat", "b", "[", "]", "g", "z", "u", "l", "j", "T", "P", "\\tilde", "R", "X", "C", "\\overline", "E", "S", "N", "o", "\\pi", "V", "q", "B", "\\prime", "L", "F", "h", "M", "\\alpha", "\\mu", "I", "D", "w", "G", "H", "\\sum", "\\theta", "\\prod", "\\lambda", "\\sigma", "\\rho", "Q", "\\}", "\\{", "\\omega", "K", "\\infty", "\\beta", "\\in", "\\epsilon", "U", "Z", "\\phi", "\\gamma", "<", "*", "\\Delta", "\\underline", ":", "\\rightarrow", "W", "Y", "O", "\\nu", "\\le", "\\delta", "\\notin", "\\psi", "\\tau", "!", "J", "\\rangle", "\\vec", "\\times", ">", "\\varphi", "\\mathbb", "\\approx", "\\pm", "\\dot", "\\Omega", "\\eta", "\\ge", "\\langle", "\\xi", "\\Gamma", "\\zeta", "\\Phi", "\\nabla", "\\hbar", "\\equiv", "\\chi", "\\kappa", "\\circ", ";", "\\ne", "\\Sigma", "\\sim", "\\Psi", "\\forall", "\\Lambda", "\\rfloor", "\\lfloor", "\\Theta", "\\Pi", "\\mapsto", "\\aleph", "\\otimes", "\\neg", "\\wedge", "\\vdash", "\\cup", "\\dagger", "\\propto", "\\cap", "\\backslash", "\\subseteq", "\\vee", "\\subset", "\\oplus", "\\simeq", "\\Rightarrow", "\\vartheta", "\\perp", "\\upsilon", "\\emptyset", "\\ll", "\\bigcup", "\\Xi", "\\cong", "\\leftarrow", "\\#", "\\bigcap", "\\varpi", "\\|", "?", "\\oint", "\\%", "\\iint", "\\odot", "\\iota", "\\exists", "\\bigoplus", "\\top", "\\rceil", "\\lceil", "\\iff", "\\gg", "\\subsetneq", "\\Upsilon", "\\bullet", "\\mp", "\\varsigma", "\\models", "\\div", "\\ominus", "\\angle", "\\vdots", "\\longrightarrow", "\\Leftrightarrow", "\\triangle", "\\leftrightarrow", "\\hookrightarrow", "\\bigwedge", "\\Vdash", "\\_", "\\not", "\\rightleftharpoons", "\\supset", "\\ni", "\\triangleq", "\\supseteq", "\\bigvee", "\\bigcirc", "\\&", "\\triangleleft", "\\sqsubseteq", " ", "\"", "#", "$", "%", "'", "@", "\\", "~", "§", "«", "°", "·", "»", "½", "é", "ó", "Δ", "α", "β", "δ", "λ", "μ", "π", "ρ", "τ", "φ", "Ё", "А", "Б", "В", "Г", "Д", "Е", "Ж", "З", "И", "Й", "К", "Л", "М", "Н", "О", "П", "Р", "С", "Т", "У", "Ф", "Х", "Ц", "Ч", "Ш", "Ы", "Э", "Ю", "Я", "а", "б", "в", "г", "д", "е", "ж", "з", "и", "й", "к", "л", "м", "н", "о", "п", "р", "с", "т", "у", "ф", "х", "ц", "ч", "ш", "щ", "ъ", "ы", "ь", "э", "ю", "я", "ё", "і", "—", "“", "„", "•", "₀", "₁", "₂", "ₓ", "ₜ", "№", "←", "↑", "→", "↓", "↔", "↻", "⇒", "⇔", "∀", "∃", "∈", "√", "∞", "∨", "≈", "≠", "≤", "①", "②", "③", "④", "⑤", "⑥", "⑦", "⑧", "⑩", "⑭", "⑱", "○", "♥", "✓"]
 
 
defaults.json DELETED
@@ -1,10 +0,0 @@
1
- {
2
- "B_CKPT": "ckpt_dl/b4styc/fp16/step003000_fp16.pt",
3
- "GDN_CKPT": "ckpt_glyph_dn_u2s/fp16/step050000.pt",
4
- "FW_LIB": "none",
5
- "SEL_RETRY": "16",
6
- "FW_NS": "data/glyph_stroke_counts.json",
7
- "LINE_MARKS": "1",
8
- "KEY": "ref",
9
- "REF": "examples/ref_hwr200.npy"
10
- }
 
 
 
 
 
 
 
 
 
 
 
download_weights.py ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Downloads the model weights from Hugging Face into this folder.
2
+
3
+ python download_weights.py # the models that write pages, about 80 MB
4
+ python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
5
+ """
6
+ import argparse
7
+ import os
8
+
9
+ from huggingface_hub import snapshot_download
10
+
11
+ REPO = "levossadtchi/Scrawler-open"
12
+ WRITER = [
13
+ "ckpt_dl/b4styc/fp16/step003000_fp16.pt",
14
+ "ckpt_dl/planner7/best.pt",
15
+ "ckpt_semb3_all/best.pt",
16
+ "ckpt_glyph_dn_u2s/fp16/step050000.pt",
17
+ "ckpt_dl/glyph_cls2.pt",
18
+ ]
19
+ TRACER = ["trace/weights/*"]
20
+
21
+
22
+ def main():
23
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
24
+ ap.add_argument("--tracer", action="store_true")
25
+ args = ap.parse_args()
26
+ root = os.path.dirname(os.path.abspath(__file__))
27
+ snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []), local_dir=root)
28
+ print("weights are in", root)
29
+
30
+
31
+ if __name__ == "__main__":
32
+ main()
scripts/make_ref.py → make_ref.py RENAMED
@@ -3,8 +3,8 @@
3
  (dx, dy, pen_up; line height = 1, points every 0.05 of it along the pen path, as in training).
4
  input : JSON, a list of strokes, each a list of points [x, y] in any units, y pointing down, in writing order;
5
  or the JSON written by trace/scrawler_trace.py (a photo of your line)
6
- python scripts/make_ref.py my_line.json my_line.npy
7
- then REF=/path/to/my_line.npy python scripts/write_pages.py notes.txt out
8
  Write a line of ordinary prose of 30-50 characters: the style is taken from this one line only."""
9
  import sys, json, numpy as np
10
  STEP = 0.05
 
3
  (dx, dy, pen_up; line height = 1, points every 0.05 of it along the pen path, as in training).
4
  input : JSON, a list of strokes, each a list of points [x, y] in any units, y pointing down, in writing order;
5
  or the JSON written by trace/scrawler_trace.py (a photo of your line)
6
+ python make_ref.py my_line.json my_line.npy
7
+ then python write_pages.py notes.txt out --ref my_line.npy
8
  Write a line of ordinary prose of 30-50 characters: the style is taken from this one line only."""
9
  import sys, json, numpy as np
10
  STEP = 0.05
render_pages.py ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Re-render strokes.json without the models, for example with another pen width.
2
+
3
+ python render_pages.py out/strokes.json out_thick --pen 1.4
4
+ """
5
+ import argparse
6
+ import json
7
+ import os
8
+ import sys
9
+
10
+ from scrawler.render import render
11
+
12
+
13
+ def main():
14
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
15
+ ap.add_argument("strokes", help="strokes.json written by write_pages.py")
16
+ ap.add_argument("out", help="output folder, must not exist")
17
+ ap.add_argument("--pen", type=float, default=1.1, help="pen width in points")
18
+ ap.add_argument("--png-dpi", type=int, default=300)
19
+ ap.add_argument("--no-smooth", action="store_true")
20
+ args = ap.parse_args()
21
+ if os.path.exists(args.out):
22
+ sys.exit(f"{args.out} exists")
23
+ os.makedirs(args.out)
24
+ render(json.load(open(args.strokes)), args.out, pen=args.pen, png_dpi=args.png_dpi, smooth=not args.no_smooth)
25
+ print("rendered ->", args.out)
26
+
27
+
28
+ if __name__ == "__main__":
29
+ main()
requirements.txt CHANGED
@@ -1,3 +1,4 @@
1
  torch
2
  numpy
3
  pillow
 
 
1
  torch
2
  numpy
3
  pillow
4
+ huggingface_hub
scrawler/__init__.py ADDED
@@ -0,0 +1 @@
 
 
1
+ """Scrawler: text with LaTeX formulas -> handwritten pages, in the style of one line of handwriting."""
scrawler/common.py ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Paths and the device, shared by all modules."""
2
+ import os
3
+
4
+ import torch
5
+
6
+ ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
7
+
8
+
9
+ def _pick_device():
10
+ forced = os.environ.get("SCRAWLER_DEVICE")
11
+ if forced:
12
+ return forced
13
+ if torch.cuda.is_available():
14
+ return "cuda"
15
+ if torch.backends.mps.is_available():
16
+ return "mps"
17
+ return "cpu"
18
+
19
+
20
+ DEVICE = _pick_device()
21
+ AMP = DEVICE == "cuda"
22
+ if DEVICE == "cuda":
23
+ torch.backends.cuda.matmul.allow_tf32 = True
24
+ torch.backends.cudnn.allow_tf32 = True
25
+
26
+ CKPT_TEXT = os.path.join(ROOT, "ckpt_dl", "b4styc", "fp16", "step003000_fp16.pt")
27
+ CKPT_PLANNER = os.path.join(ROOT, "ckpt_dl", "planner7", "best.pt")
28
+ CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
29
+ CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
30
+ CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
31
+ STROKE_COUNTS = os.path.join(ROOT, "data", "glyph_stroke_counts.json")
32
+ DEFAULT_REF = os.path.join(ROOT, "examples", "ref_hwr200.npy")
33
+ MATHJAX_DIR = os.path.join(ROOT, "tools", "mathjax")
34
+
35
+
36
+ def load_checkpoint(path):
37
+ if not os.path.exists(path):
38
+ raise SystemExit(f"{path} not found. Run: python download_weights.py")
39
+ return torch.load(path, map_location=DEVICE, weights_only=True)
scrawler/formula.py ADDED
@@ -0,0 +1,637 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Formulas: MathJax says where every symbol stands, the glyph model draws the symbols, and thin marks are drawn as pen lines.
2
+
3
+ Coordinates inside this module: MathJax units x LAYOUT_SCALE, y down.
4
+ """
5
+ import collections
6
+ import json
7
+ import math
8
+ import os
9
+ import subprocess
10
+ import unicodedata
11
+
12
+ import numpy as np
13
+ import torch
14
+
15
+ from .common import MATHJAX_DIR
16
+ from .glyph_model import GlyphModel, GlyphPicker
17
+
18
+ LAYOUT_SCALE = 0.1
19
+ X_HEIGHT = 450 * LAYOUT_SCALE
20
+ X_HEIGHT_EM = 0.442 # x-height of a composed formula in the units compose() returns
21
+ VARIANTS = 4
22
+ RETRIES = 16
23
+ POINT_STEP = 0.05
24
+
25
+ # fractions, radicals and arrows are pen lines over their layout box
26
+ LINE_MARKS = {"BAR", "√", "→", "←", "↑", "↓", "⇒", "⇐", "⇔", "↔", "VEC"}
27
+ # accents, slashes and the like, which the glyph model does not draw well
28
+ OWN_MARKS = {"/", "↔", "↦", "ˆ", "^", "˜", "~", " ̄", "¯", " ̃", " ̂", "‾", "̄", "̂", "̃", "̅"}
29
+ SUBSTITUTE_CODES = {"2A7E": "2265", "2A7D": "2264"} # slanted >= and <= are drawn as the plain ones
30
+ # classes the glyph model has no row for, drawn as a similar shape
31
+ DONORS = {
32
+ "O": "0", "κ": "k", "Σ": "∑",
33
+ "≃": "≈", "∙": "⋅", "*": "∗", "⋯": "…", "⇓": "↓", "⟸": "←", "⇉": "→", "⎧": "{", "⎩": "{", "⎛": "(",
34
+ "∩": "u", "Θ": "θ",
35
+ } # fmt: skip
36
+ KEEP_SHAPE = set("=+−-<>≠±") # a sheared '=' reads as '>'
37
+ VERTICAL_BARS = ("|", "∥", "‖", "∣")
38
+ DOTS = ("⋅", "·", "…", "...", "⋯", ".", ":")
39
+ DOT_ACCENTS = (" ̇", "˙", "̇")
40
+
41
+
42
+ def class_of(code):
43
+ if code == "BAR":
44
+ return "BAR"
45
+ ch = chr(int(code, 16))
46
+ if ch == "⃗":
47
+ return "VEC"
48
+ normal = unicodedata.normalize("NFKC", ch)
49
+ return normal if normal.strip() else ch
50
+
51
+
52
+ def layout(formulas):
53
+ """LaTeX -> [{ok, glyphs: [{c, bbox, node}], ...}] with MathJax under node."""
54
+ lines = "\n".join(json.dumps(dict(id=i, tex=t), ensure_ascii=False) for i, t in enumerate(formulas)) + "\n"
55
+ result = subprocess.run(
56
+ ["node", "layout.mjs"], input=lines, capture_output=True, text=True, cwd=MATHJAX_DIR, env=dict(os.environ, LAYOUT_TEXT="1")
57
+ )
58
+ out = {}
59
+ for line in result.stdout.splitlines():
60
+ try:
61
+ o = json.loads(line)
62
+ out[o["id"]] = o
63
+ except ValueError:
64
+ pass
65
+ return [out.get(i) for i in range(len(formulas))]
66
+
67
+
68
+ def _merge_radicals(glyphs):
69
+ """A radical sign and its overbar -> one box."""
70
+ out = []
71
+ i = 0
72
+ while i < len(glyphs):
73
+ g = glyphs[i]
74
+ if g["c"] == "221A" and i + 1 < len(glyphs) and glyphs[i + 1]["c"] == "BAR" and glyphs[i + 1].get("node") in ("msqrt", "mroot"):
75
+ b = glyphs[i + 1]["bbox"]
76
+ g = dict(c="221A", bbox=[min(g["bbox"][0], b[0]), min(g["bbox"][1], b[1]), max(g["bbox"][2], b[2]), max(g["bbox"][3], b[3])])
77
+ i += 2
78
+ else:
79
+ i += 1
80
+ out.append(g)
81
+ return out
82
+
83
+
84
+ def _fix_limits(glyphs):
85
+ """MathJax puts the lower limit of an integral under its slanted foot; handwriting puts it to the right."""
86
+ for i, g in enumerate(glyphs):
87
+ if g["c"] not in ("222B", "222E"):
88
+ continue
89
+ x0, y0, x1, y1 = g["bbox"]
90
+ middle = (y0 + y1) / 2
91
+ group = [h for h in glyphs[i + 1 : i + 4] if h["bbox"][0] < x1 and h["bbox"][1] > middle]
92
+ if group:
93
+ dx = x1 + 0.05 * (y1 - y0) - min(h["bbox"][0] for h in group)
94
+ for h in group:
95
+ h["bbox"] = [h["bbox"][0] + dx, h["bbox"][1], h["bbox"][2] + dx, h["bbox"][3]]
96
+ return glyphs
97
+
98
+
99
+ def _segment(a, b, n=8):
100
+ return [[a[0] + (b[0] - a[0]) * t, a[1] + (b[1] - a[1]) * t] for t in np.linspace(0, 1, n)]
101
+
102
+
103
+ def _polyline(points, rng, jitter, n=6):
104
+ out = []
105
+ for i in range(len(points) - 1):
106
+ q = _segment(points[i], points[i + 1], n)
107
+ out += q if not out else q[1:]
108
+ return [[x + rng.normal(0, jitter), y + rng.normal(0, jitter), 0.5] for x, y in out]
109
+
110
+
111
+ def _line_mark(c, box, rng):
112
+ """Pen-line strokes for a fraction bar, a radical or an arrow over a layout box."""
113
+ x0, y0, x1, y1 = box
114
+ w = x1 - x0
115
+ h = max(y1 - y0, 1e-6)
116
+ ym = (y0 + y1) / 2
117
+ xm = (x0 + x1) / 2
118
+ j = 0.02 * X_HEIGHT
119
+ tilt = rng.normal(0, 0.01) * w
120
+ if c == "BAR":
121
+ return [_polyline([[x0 - 0.03 * w, ym - tilt], [x1 + 0.03 * w, ym + tilt]], rng, j, 12)]
122
+ if c == "√":
123
+ tw = min(0.5 * h, 0.4 * w)
124
+ return [_polyline([[x0, y0 + 0.6 * h], [x0 + 0.18 * tw, y0 + 0.52 * h], [x0 + 0.55 * tw, y1], [x0 + tw, y0], [x1, y0 + tilt]], rng, j, 6)]
125
+ head = min(0.35 * w, max(0.9 * h, 0.25 * X_HEIGHT)) if c not in ("↑", "↓") else min(0.35 * h, max(0.9 * w, 0.25 * X_HEIGHT))
126
+ if c == "VEC":
127
+ return [_polyline([[x0, ym], [x1, ym]], rng, j, 8), _polyline([[x1 - head, ym - 0.5 * head], [x1, ym], [x1 - head, ym + 0.5 * head]], rng, j, 4)]
128
+ if c in ("↑", "↓"):
129
+ tip, sign = (y1, -1) if c == "↓" else (y0, 1)
130
+ return [
131
+ _polyline([[xm, y0], [xm, y1]], rng, j, 10),
132
+ _polyline([[xm - 0.5 * head, tip + sign * head], [xm, tip], [xm + 0.5 * head, tip + sign * head]], rng, j, 4),
133
+ ]
134
+ double = c in ("⇒", "⇐", "⇔")
135
+ d = 0.22 * h if double else 0.0
136
+ strokes = []
137
+ for yy in [ym - d, ym + d] if double else [ym]:
138
+ strokes.append(_polyline([[x0 + (0.15 * head if c in "⇐⇔" else 0), yy], [x1 - (0.15 * head if c in "⇒⇔" else 0), yy]], rng, j, 10))
139
+ if c in ("→", "⇒", "⇔", "↔"):
140
+ strokes.append(_polyline([[x1 - head, ym - 0.55 * head - d], [x1, ym], [x1 - head, ym + 0.55 * head + d]], rng, j, 4))
141
+ if c in ("←", "⇐", "⇔", "↔"):
142
+ strokes.append(_polyline([[x0 + head, ym - 0.55 * head - d], [x0, ym], [x0 + head, ym + 0.55 * head + d]], rng, j, 4))
143
+ return strokes
144
+
145
+
146
+ def _transform_glyph(strokes, mx, my, size, rng):
147
+ """Small random shear, bend, proportions and pen speed, so that two instances of a symbol are not the same shape."""
148
+ shear_x = rng.normal(0, 0.06)
149
+ shear_y = rng.normal(0, 0.05)
150
+ bend_v = rng.normal(0, 0.10)
151
+ bend_h = rng.normal(0, 0.08)
152
+ aspect = rng.normal(0, 0.04)
153
+ ascender = rng.normal(0, 0.04)
154
+ descender = rng.normal(0, 0.04)
155
+ px, py = rng.normal(0, 0.02, 2)
156
+ speed = 0.5
157
+ pressure = math.exp(rng.normal(0, 0.27))
158
+ all_points = np.concatenate([(p[:, :2] - [mx, my]) / size for p in strokes])
159
+ mean_x2 = float(np.mean(all_points[:, 0] ** 2))
160
+ mean_y2 = float(np.mean(all_points[:, 1] ** 2))
161
+ out = []
162
+ for p in strokes:
163
+ q = p.copy()
164
+ x = (p[:, 0] - mx) / size
165
+ y = (p[:, 1] - my) / size
166
+ if len(p) > 3: # uneven pen speed: the points move along the same path
167
+ d = np.r_[0, np.cumsum(np.linalg.norm(np.diff(np.stack([x, y], 1), axis=0), axis=1))]
168
+ if d[-1] > 0:
169
+ u = np.linspace(0, 1, len(p))
170
+ phase = rng.uniform(0, 6.283)
171
+ w = (
172
+ u
173
+ + speed * np.sin(2 * np.pi * u + phase) / (2 * np.pi)
174
+ - speed * np.sin(phase) / (2 * np.pi) * (1 - u)
175
+ - speed * np.sin(2 * np.pi + phase) / (2 * np.pi) * u
176
+ )
177
+ w = np.clip(np.maximum.accumulate(w), 0, 1) * d[-1]
178
+ x = np.interp(w, d, x)
179
+ y = np.interp(w, d, y)
180
+ if q.shape[1] > 3:
181
+ q[:, 3] = np.interp(w, d, p[:, 3])
182
+ x = x + shear_x * y
183
+ y = y + shear_y * x
184
+ y = y + bend_v * (x**2 - mean_x2)
185
+ x = x + bend_h * (y**2 - mean_y2)
186
+ x = x * math.exp(aspect / 2)
187
+ y = y * math.exp(-aspect / 2)
188
+ y = np.where(y < -0.25, -0.25 + (y + 0.25) * (1 + ascender), np.where(y > 0.25, 0.25 + (y - 0.25) * (1 + descender), y))
189
+ denominator = 1 + px * x + py * y
190
+ x = x / denominator
191
+ y = y / denominator
192
+ q[:, 0] = x * size + mx
193
+ q[:, 1] = y * size + my
194
+ if q.shape[1] > 3:
195
+ q[:, 3] = np.clip(q[:, 3] * pressure, 0, 1)
196
+ out.append(q)
197
+ return out
198
+
199
+
200
+ def _bounds(strokes):
201
+ a = np.array([p[:2] for t in strokes for p in t], np.float32)
202
+ return float(a[:, 0].min()), float(a[:, 1].min()), float(a[:, 0].max()), float(a[:, 1].max())
203
+
204
+
205
+ class FormulaWriter:
206
+ def __init__(self, text_model, reference, semb):
207
+ self.text = text_model
208
+ self.reference = np.asarray(reference, np.float32)
209
+ self.semb = semb
210
+ self.style_row = semb[0].detach().cpu().numpy().astype(np.float32).reshape(-1)
211
+ self.stats = collections.Counter()
212
+ self._glyphs = None
213
+ self._picker = None
214
+ self._base_heights = {}
215
+
216
+ @property
217
+ def glyphs(self):
218
+ if self._glyphs is None:
219
+ self._glyphs = GlyphModel(self.stats)
220
+ return self._glyphs
221
+
222
+ @property
223
+ def picker(self):
224
+ if self._picker is None:
225
+ self._picker = GlyphPicker(self.stats)
226
+ return self._picker
227
+
228
+ def _variants(self, classes, rng, k=1):
229
+ n = len(classes) // max(k, 1)
230
+ out = []
231
+ for j in range(max(k, 1)):
232
+ out += self.glyphs.sample(classes[j * n : (j + 1) * n], self.style_row, rng)
233
+ return out
234
+
235
+ def _write_words(self, words):
236
+ """Words inside \\text{} -> {word: strokes}, written by the text model."""
237
+ words = [w for w in dict.fromkeys(words) if w.strip()]
238
+ if not words:
239
+ return {}
240
+ lines = [list(w) for w in words]
241
+ style = torch.cat([self.text.style(self.semb)] * len(words), 0)
242
+ semb = torch.cat([self.semb] * len(words), 0)
243
+ durations = self.text.durations(lines, semb)
244
+ out = {}
245
+ for word, q in zip(words, self.text.sample(lines, durations, style)):
246
+ q = np.asarray(q, np.float32)
247
+ xy = np.cumsum(q[:, :2], 0) * 100.0
248
+ strokes, current = [], []
249
+ for k in range(len(q)):
250
+ current.append([float(xy[k, 0]), float(xy[k, 1]), 0.5])
251
+ if q[k, 2] > 0.5:
252
+ strokes.append(current)
253
+ current = []
254
+ if current:
255
+ strokes.append(current)
256
+ out[word] = strokes
257
+ return out
258
+
259
+ def _compose_text(self, boxes):
260
+ """Places words of \\text{} in their layout boxes, fitted by x-height."""
261
+ s = LAYOUT_SCALE
262
+ words, current = [], []
263
+ for g in sorted(boxes, key=lambda g: g["bbox"][0]):
264
+ if not g.get("ch", "").strip():
265
+ if current:
266
+ words.append(current)
267
+ current = []
268
+ continue
269
+ if current and g["bbox"][0] - current[-1]["bbox"][2] > 50:
270
+ words.append(current)
271
+ current = []
272
+ current.append(g)
273
+ if current:
274
+ words.append(current)
275
+ written = self._write_words(["".join(g["ch"] for g in w) for w in words])
276
+ out = []
277
+ for word in words:
278
+ strokes = written.get("".join(g["ch"] for g in word))
279
+ if not strokes:
280
+ continue
281
+ a = np.array([p[:2] for t in strokes for p in t], np.float32)
282
+ bx0 = min(g["bbox"][0] for g in word) * s
283
+ bx1 = max(g["bbox"][2] for g in word) * s
284
+ baseline = float(np.median([g["bbox"][3] for g in word])) * s
285
+ box_height = float(np.median([g["bbox"][3] - g["bbox"][1] for g in word])) * s
286
+ word_height = max(np.percentile(a[:, 1], 85) - np.percentile(a[:, 1], 15), 1e-6)
287
+ k = min(1.1 * (bx1 - bx0) / max(np.ptp(a[:, 0]), 1e-6), box_height / word_height)
288
+ bottom = np.percentile(a[:, 1], 85)
289
+ for t in strokes:
290
+ out.append([[float((p[0] - a[:, 0].min()) * k + bx0), float((p[1] - bottom) * k + baseline), float(p[2])] for p in t])
291
+ return out
292
+
293
+ def _own_marks(self, glyphs, rng):
294
+ """Marks the glyph model does not draw -> pen lines. Returns the remaining glyphs and the strokes of the marks."""
295
+ self.glyphs # loaded here so that the order of random draws does not depend on the input
296
+ s = LAYOUT_SCALE
297
+ keep, out = [], []
298
+ for g in glyphs:
299
+ c = class_of(g["c"]) if g["c"] not in ("BAR", "TEXT") else g["c"]
300
+ if g["c"] in SUBSTITUTE_CODES:
301
+ keep.append(dict(g, c=SUBSTITUTE_CODES[g["c"]]))
302
+ continue
303
+ if c not in OWN_MARKS:
304
+ keep.append(g)
305
+ continue
306
+ x0, y0, x1, y1 = [v * s for v in g["bbox"]]
307
+ w = max(x1 - x0, 1e-6)
308
+ h = max(y1 - y0, 1e-6)
309
+ j = 0.015 * X_HEIGHT
310
+ ym = (y0 + y1) / 2
311
+ if c == "↔":
312
+ out += _line_mark("↔", [x0, y0, x1, y1], rng)
313
+ elif c == "↦":
314
+ out += _line_mark("→", [x0, y0, x1, y1], rng) + [_polyline([[x0, ym - 0.4 * h], [x0, ym + 0.4 * h]], rng, j, 5)]
315
+ elif c == "/":
316
+ out.append(_polyline([[x0 + 0.15 * w, y1], [x1 - 0.15 * w, y0]], rng, j, 10))
317
+ elif c in ("ˆ", "^", " ̂", "̂"):
318
+ out.append(_polyline([[x0 + 0.1 * w, y1], [(x0 + x1) / 2, y0], [x1 - 0.1 * w, y1]], rng, j, 5))
319
+ elif c in ("˜", "~", " ̃", "̃"):
320
+ out.append([[float(x0 + t * w), float(ym - 0.35 * h * np.sin(2 * np.pi * t) + rng.normal(0, j)), 0.5] for t in np.linspace(0, 1, 14)])
321
+ else: # overline, macron
322
+ out.append(_polyline([[x0, ym], [x1, ym]], rng, j, 10))
323
+ self.stats["own_mark_" + c] += 1
324
+ return keep, out
325
+
326
+ def _load_base_heights(self, codes):
327
+ new = [c for c in set(codes) if c not in self._base_heights and c not in ("BAR", "TEXT")]
328
+ if new:
329
+ for c, o in zip(new, layout([chr(int(c, 16)) for c in new])):
330
+ self._base_heights[c] = (o["glyphs"][0]["bbox"][3] - o["glyphs"][0]["bbox"][1]) if o and o.get("ok") and o["glyphs"] else None
331
+
332
+ def _script_factor(self, g):
333
+ """Subscripts: MathJax shrinks them by 0.71, handwriting shrinks digits as much but letters less."""
334
+ if g["c"] in ("BAR", "TEXT"):
335
+ return 1.0
336
+ height = self._base_heights.get(g["c"])
337
+ if not height or (g["bbox"][3] - g["bbox"][1]) >= 0.85 * height:
338
+ return 1.0
339
+ ch = unicodedata.normalize("NFKC", chr(int(g["c"], 16)))
340
+ if ch.isdigit():
341
+ return 0.71
342
+ if ch.isascii() and ch.isalpha():
343
+ return 1.0
344
+ return 0.92
345
+
346
+ def _push_right(self, strokes, placed):
347
+ """Moves a glyph right if it collides with a neighbour on the same line."""
348
+ if not strokes or not placed:
349
+ return
350
+ a = np.concatenate([np.asarray(q)[:, :2] for q in strokes])
351
+ bx0, by0, bx1, by1 = a[:, 0].min(), a[:, 1].min(), a[:, 0].max(), a[:, 1].max()
352
+ shift = 0.0
353
+ for px0, py0, px1, py1 in placed:
354
+ overlap = min(by1, py1) - max(by0, py0)
355
+ smaller = min(by1 - by0, py1 - py0)
356
+ same_line = abs((py0 + py1) / 2 - (by0 + by1) / 2) < 0.3 * max(by1 - by0, py1 - py0) # a subscript is not a collision
357
+ if (py1 - py0) > 1.8 * (by1 - by0) and py0 <= by0 and py1 >= by1: # a radical or a tall bracket encloses, it is no neighbour
358
+ continue
359
+ if same_line and smaller > 0.12 * X_HEIGHT and overlap > 0.5 * smaller and (px0 + px1) / 2 < (bx0 + bx1) / 2 and px1 + 0.35 * X_HEIGHT > bx0:
360
+ shift = max(shift, px1 + 0.35 * X_HEIGHT - bx0)
361
+ if 0 < shift < 2.0 * X_HEIGHT:
362
+ for q in strokes:
363
+ q[:, 0] += shift
364
+ self.stats["pushed"] += 1
365
+
366
+ def _compose_layout(self, glyphs, rng):
367
+ s = LAYOUT_SCALE
368
+ xh = X_HEIGHT
369
+ strokes, missing = [], []
370
+ merged = _merge_radicals(glyphs)
371
+ texts = [g for g in merged if g["c"] == "TEXT"]
372
+ glyphs = []
373
+ for g in merged:
374
+ if g["c"] == "TEXT":
375
+ continue
376
+ if g["c"] == "226A": # << = two <
377
+ b = g["bbox"]
378
+ m = (b[0] + b[2]) / 2
379
+ w = b[2] - b[0]
380
+ glyphs += [dict(g, c="3C", bbox=[b[0], b[1], m + 0.08 * w, b[3]]), dict(g, c="3C", bbox=[m - 0.08 * w, b[1], b[2], b[3]])]
381
+ else:
382
+ glyphs.append(g)
383
+ self._load_base_heights([g["c"] for g in glyphs])
384
+ if texts:
385
+ strokes += self._compose_text(texts)
386
+ if not glyphs:
387
+ return strokes, missing
388
+
389
+ class_for_net = lambda c: DONORS.get(c, c)
390
+ x_min = min(g["bbox"][0] for g in glyphs)
391
+ n = len(glyphs)
392
+ top = min(g["bbox"][1] for g in glyphs)
393
+ height = max(max(g["bbox"][3] for g in glyphs) - top, 1e-6)
394
+ variants = self._variants([class_for_net(class_of(g["c"])) for g in glyphs] * VARIANTS, rng, VARIANTS)
395
+ shape_rng = np.random.default_rng(int(rng.integers(1 << 31)))
396
+ gap = base = 0.0
397
+ placed = []
398
+ last = 0
399
+
400
+ for gi, g in enumerate(glyphs):
401
+ if len(strokes) > last:
402
+ placed.append(_bounds(strokes[last:]))
403
+ last = len(strokes)
404
+ upside_down = class_of(g["c"]) == "∩" # ∩ is a u turned over
405
+ c = class_of(g["c"])
406
+ if c in DONORS:
407
+ self.stats["donor_" + c] += 1
408
+ c = DONORS[c]
409
+
410
+ if c in DOTS: # the net draws a dot as a ring
411
+ b = [v * s for v in g["bbox"]]
412
+ r = 0.05 * xh
413
+ if c == ":":
414
+ dots = [((b[0] + b[2]) / 2, b[1] + 0.15 * (b[3] - b[1])), ((b[0] + b[2]) / 2, b[3] - 0.15 * (b[3] - b[1]))]
415
+ else:
416
+ count = 1 if c in ("⋅", "·", ".") else 3
417
+ cy = (b[1] + b[3]) / 2 if c != "." else b[3] - r
418
+ dots = [(b[0] + (k + 0.5) * (b[2] - b[0]) / count, cy) for k in range(count)]
419
+ for cx, cy in dots:
420
+ strokes.append([[float(cx - r), float(cy), 0.5], [float(cx), float(cy + r * 0.6), 0.5], [float(cx + r), float(cy), 0.5]])
421
+ self.stats["dot_mark"] += 1
422
+ continue
423
+ if c in DOT_ACCENTS:
424
+ b = [v * s for v in g["bbox"]]
425
+ cx = (b[0] + b[2]) / 2
426
+ cy = (b[1] + b[3]) / 2
427
+ r = 0.05 * xh
428
+ strokes.append([[float(cx - r), float(cy), 0.5], [float(cx), float(cy + r * 0.6), 0.5], [float(cx + r), float(cy), 0.5]])
429
+ self.stats["dot_mark"] += 1
430
+ continue
431
+ if c in VERTICAL_BARS: # the net draws | as / ( ) ? 7
432
+ b = [v * s for v in g["bbox"]]
433
+ xm = (b[0] + b[2]) / 2
434
+ offsets = [0.0] if c in ("|", "∣") else [-0.22 * (b[2] - b[0]), 0.22 * (b[2] - b[0])]
435
+ bars = [np.asarray(_polyline([[xm + d, b[1]], [xm + d, b[3]]], rng, 0.015 * xh, 10), np.float64) for d in offsets]
436
+ self._push_right(bars, placed)
437
+ strokes += [t.tolist() for t in bars]
438
+ self.stats["vbar_pen"] += 1
439
+ continue
440
+ if c in LINE_MARKS:
441
+ strokes += _line_mark(c, [v * s for v in g["bbox"]], rng)
442
+ self.stats["line_mark"] += 1
443
+ continue
444
+
445
+ relative_height = (g["bbox"][3] - g["bbox"][1]) / height
446
+ q = None
447
+ if c not in ("[", "]"): # square brackets always go through the retry below
448
+ k = self.picker.pick([variants[k * n + gi] for k in range(VARIANTS)], c, relative_height)
449
+ q = variants[k * n + gi] if k is not None else None
450
+ if q is not None and not q["strokes"]:
451
+ q = None
452
+ if q is None:
453
+ if not variants[gi]["strokes"]: # the net does not know this class
454
+ missing.append(c)
455
+ continue
456
+ more = self._variants([class_for_net(class_of(g["c"]))] * RETRIES, rng, 1)
457
+ k = self.picker.pick(more, c, relative_height)
458
+ if k is not None:
459
+ q = more[k]
460
+ self.stats["retry_ok"] += 1
461
+ else:
462
+ everything = more + [variants[j * n + gi] for j in range(VARIANTS)]
463
+ k = self.picker.pick(everything, c, relative_height, force=True)
464
+ q = everything[k if k is not None else 0]
465
+ self.stats["retry_forced"] += 1
466
+
467
+ b = [v * s for v in g["bbox"]]
468
+ bw = b[2] - b[0]
469
+ bh = b[3] - b[1]
470
+ points = [np.asarray(t, np.float32) for t in q["strokes"]]
471
+ all_points = np.concatenate([p[:, :2] for p in points])
472
+ x0, y0 = all_points.min(0)
473
+ x1, y1 = all_points.max(0)
474
+ qw = max(x1 - x0, 1e-3)
475
+ qh = max(y1 - y0, 1e-3)
476
+ if bw > 2.5 * bh:
477
+ kx = bw / qw
478
+ ky = min(kx, bh / qh * 1.5)
479
+ elif bh > 2.2 * bw and c in "()[]{}|∫∮∑∏":
480
+ ky = bh / qh
481
+ kx = min(ky, 1.6 * bw / qw)
482
+ else:
483
+ ky = bh / qh
484
+ kx = ky
485
+ f = 1.0 + rng.normal(0, 0.06)
486
+ kx *= f
487
+ ky *= f
488
+ angle = math.radians(rng.normal(0, 2.5))
489
+ cos_a, sin_a = math.cos(angle), math.sin(angle)
490
+ drift = 0.02 * xh * math.sin((b[0] - x_min * s) / (6 * xh) + rng.uniform(0, 6.28))
491
+ cx = (b[0] + b[2]) / 2 + rng.normal(0, 0.06 * xh)
492
+ cy = (b[1] + b[3]) / 2 + rng.normal(0, 0.06 * xh) + drift
493
+ if c not in ("(", ")", "[", "]", "{", "}", "|") and qw * kx > 1.3 * bw and bw > 0: # much wider than its box: crowds the neighbours
494
+ f2 = 1.3 * bw / (qw * kx)
495
+ kx *= f2
496
+ ky *= f2
497
+ self.stats["narrowed"] += 1
498
+ if class_of(g["c"]) == "Θ": # capital Theta from a lowercase one
499
+ kx *= 1.35
500
+ fs = self._script_factor(g)
501
+ if fs != 1.0:
502
+ kx *= fs
503
+ ky *= fs
504
+ cx = b[0] + (cx - b[0]) * fs
505
+ self.stats["script_scaled"] += 1
506
+ mx = (x0 + x1) / 2
507
+ my = (y0 + y1) / 2
508
+ transformed = _transform_glyph(points, mx, my, max(qw, qh), shape_rng)
509
+ if c not in KEEP_SHAPE: # the random numbers are drawn either way
510
+ points = transformed
511
+ gap = 0.7 * gap + math.sqrt(1 - 0.49) * shape_rng.normal(0, 0.1)
512
+ base = 0.8 * base + 0.6 * shape_rng.normal(0, 0.08)
513
+ upright = int(g["c"], 16) < 128 and chr(int(g["c"], 16)).isalpha() # letters of cos, sin: no gaps inside the word
514
+ cx += (0 if upright else gap) * min(bh, 1.5 * xh)
515
+ cy += base * min(bh, 1.5 * xh)
516
+ if upside_down:
517
+ points = [np.concatenate([2 * mx - p[:, :1], 2 * my - p[:, 1:2], p[:, 2:]], 1) for p in points]
518
+ placed_strokes = []
519
+ for p in points:
520
+ u = (p[:, 0] - mx) * kx
521
+ v = (p[:, 1] - my) * ky
522
+ placed_strokes.append(np.stack([cx + u * cos_a - v * sin_a, cy + u * sin_a + v * cos_a, p[:, 3]], 1))
523
+ self._push_right(placed_strokes, placed)
524
+ strokes += [t.tolist() for t in placed_strokes]
525
+ return strokes, missing
526
+
527
+ def compose(self, tex, rng):
528
+ """LaTeX -> (strokes, things it could not write). Strokes are None if the formula failed."""
529
+ tex = tex.strip().strip("$").strip()
530
+ try:
531
+ o = layout([tex])[0]
532
+ if not o or not o.get("ok"):
533
+ return None, ["LAYOUT"]
534
+ glyphs, extra = self._own_marks(_fix_limits(o["glyphs"]), rng)
535
+ strokes, missing = self._compose_layout(glyphs, rng)
536
+ strokes += extra
537
+ except Exception as e:
538
+ return None, ["ERROR " + repr(e)[:80]]
539
+ if not strokes:
540
+ return None, missing or ["EMPTY"]
541
+ scale = LAYOUT_SCALE * 1000.0
542
+ return [[[p[0] / scale, p[1] / scale] for p in stroke] for stroke in strokes], missing
543
+
544
+ def mixed_line(self, seq, tokens, durations, rng):
545
+ """The text model wrote the whole line, `$...$` included. Its points of each formula are replaced by the composed formula,
546
+ and the text after it moves right. -> (line [n, 3], things it could not write, formulas replaced); line None on failure."""
547
+ q = np.asarray(seq, np.float32)
548
+ char_of_point = np.repeat(np.arange(len(tokens)), [max(1, int(x)) for x in durations])
549
+ if len(char_of_point) != len(q):
550
+ return None, ["PLAN"], 0
551
+ a_ = np.cumsum(q[:, :2], 0)
552
+ pen = q[:, 2].copy()
553
+ spans = _dollar_spans(tokens)
554
+ if not spans:
555
+ return q, [], 0
556
+ in_formula = np.zeros(len(q), bool)
557
+ for a, b in spans:
558
+ in_formula |= (char_of_point >= a) & (char_of_point < b)
559
+ is_text = ~in_formula & np.array([not tokens[c].isspace() for c in char_of_point])
560
+ if is_text.sum() < 10:
561
+ return None, ["NOTEXT"], 0
562
+ y = a_[is_text, 1]
563
+ baseline = float(np.percentile(y, 85))
564
+ xh = max(float(np.percentile(y, 85) - np.percentile(y, 15)), 1e-3)
565
+ gap = 0.6 * xh
566
+ k = xh / X_HEIGHT_EM
567
+ pieces = []
568
+ shift = 0.0
569
+ missing = []
570
+ prev = 0
571
+ replaced = 0
572
+ for a, b in spans:
573
+ ix = np.where((char_of_point >= a) & (char_of_point < b))[0]
574
+ before = np.arange(prev, ix[0])
575
+ pieces.append((a_[before] + [shift, 0], pen[before]))
576
+ x_left = (a_[before, 0].max() + shift) if len(before) else a_[ix, 0].min() + shift
577
+ strokes, ms = self.compose("".join(tokens[a:b]).strip("$"), rng)
578
+ missing += ms
579
+ if not strokes: # keep the text model's own points of the span
580
+ pieces.append((a_[ix] + [shift, 0], pen[ix]))
581
+ prev = ix[-1] + 1
582
+ continue
583
+ f = to_points(strokes, k)
584
+ f[:, 0] += x_left + gap - f[:, 0].min()
585
+ f[:, 1] += baseline
586
+ if len(before):
587
+ pieces[-1][1][-1] = 1.0 # lift the pen before the formula
588
+ pieces.append((f[:, :2], f[:, 2]))
589
+ replaced += 1
590
+ after = np.arange(ix[-1] + 1, len(q))
591
+ x_right = a_[after, 0].min() if len(after) else None
592
+ if x_right is not None:
593
+ shift = f[:, 0].max() + gap - x_right
594
+ prev = ix[-1] + 1
595
+ rest = np.arange(prev, len(q))
596
+ pieces.append((a_[rest] + [shift, 0], pen[rest]))
597
+ points = np.concatenate([p for p, _ in pieces if len(p)])
598
+ pens = np.concatenate([p for _, p in pieces if len(p)])
599
+ pens[-1] = 1.0
600
+ d = np.zeros((len(points), 3), np.float32)
601
+ d[1:, :2] = points[1:] - points[:-1]
602
+ d[0, :2] = 0
603
+ d[:, 2] = pens
604
+ return d, missing, replaced
605
+
606
+
607
+ def _dollar_spans(tokens):
608
+ """Character spans of $...$ and $$...$$."""
609
+ out, start, k = [], None, 0
610
+ while k < len(tokens):
611
+ if tokens[k] == "$":
612
+ n = 2 if k + 1 < len(tokens) and tokens[k + 1] == "$" else 1
613
+ if start is None:
614
+ start = k
615
+ else:
616
+ out.append((start, k + n))
617
+ start = None
618
+ k += n
619
+ continue
620
+ k += 1
621
+ return out
622
+
623
+
624
+ def to_points(strokes, k):
625
+ """Composed strokes, scaled by k -> absolute points [n, 3]: x, y, pen_up, every POINT_STEP along the path."""
626
+ out = []
627
+ for stroke in strokes:
628
+ p = np.asarray(stroke, np.float32)[:, :2] * k
629
+ if len(p) == 1:
630
+ out.append(np.array([[p[0, 0], p[0, 1], 1.0]], np.float32))
631
+ continue
632
+ cum = np.r_[0, np.cumsum(np.hypot(*np.diff(p, axis=0).T))]
633
+ s = np.r_[np.arange(0, cum[-1], POINT_STEP), cum[-1]] if cum[-1] > 1e-6 else np.array([0.0, cum[-1]])
634
+ r = np.stack([np.interp(s, cum, p[:, 0]), np.interp(s, cum, p[:, 1]), np.zeros(len(s))], 1).astype(np.float32)
635
+ r[-1, 2] = 1
636
+ out.append(r)
637
+ return np.concatenate(out)
scrawler/glyph_model.py ADDED
@@ -0,0 +1,269 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The glyph model and its picker.
2
+
3
+ The glyph model is a diffusion denoiser that draws one formula symbol of a given class in the style of a given writer. Several
4
+ variants are drawn for each symbol; a small image classifier keeps the most recognisable one.
5
+ """
6
+ import json
7
+ import math
8
+
9
+ import numpy as np
10
+ import torch
11
+ import torch.nn as nn
12
+ import torch.nn.functional as F
13
+ from PIL import Image, ImageDraw
14
+
15
+ from .common import AMP, CKPT_GLYPH, CKPT_GLYPH_CLASSIFIER, DEVICE, STROKE_COUNTS, load_checkpoint
16
+ from .text_model import ConvBlock, FiLM, alpha_bar, time_embedding
17
+
18
+ STYLE_DIM = 256
19
+ HIDDEN = 256
20
+ GRU_HIDDEN = 256
21
+ COND_DIM = 192
22
+ BLOCKS_BEFORE = 4
23
+ BLOCKS_AFTER = 3
24
+ NOISE_STEPS = 100
25
+ SAMPLE_STEPS = 10
26
+ GUIDANCE = 1.2
27
+ SCALE = 0.15
28
+ PICK_MARGIN = 0.25 # the right class must beat the best other class by this much
29
+ MIN_STROKE_SHARE = 0.4 # a variant's stroke count must be this common among real glyphs of the class
30
+
31
+ ALPHA_BAR = torch.tensor(alpha_bar(NOISE_STEPS), device=DEVICE)
32
+
33
+
34
+ class GlyphDenoiser(nn.Module):
35
+ def __init__(self, classes):
36
+ super().__init__()
37
+ D, G = HIDDEN, COND_DIM
38
+ self.ncls = classes
39
+ self.ec = nn.Embedding(classes + 1, D) # the last row = "dropped" for guidance
40
+ self.gc = nn.Embedding(classes + 1, G)
41
+ self.ew = nn.Linear(STYLE_DIM + 1, D) # style = [embedding, null flag]
42
+ self.gw = nn.Linear(STYLE_DIM + 1, G)
43
+ self.inp = nn.Linear(4 + 4 + 5, D)
44
+ self.tm = nn.Sequential(nn.Linear(128, G), nn.SiLU(), nn.Linear(G, G))
45
+ dilations = [1, 2, 4, 1, 2, 4, 8]
46
+ self.pre = nn.ModuleList([ConvBlock(D, G, dilations[k]) for k in range(BLOCKS_BEFORE)])
47
+ self.gln = nn.LayerNorm(D)
48
+ self.gfilm = FiLM(G, D)
49
+ self.gru = nn.GRU(D, GRU_HIDDEN, num_layers=2, batch_first=True, bidirectional=True)
50
+ self.gout = nn.Linear(2 * GRU_HIDDEN, D)
51
+ self.post = nn.ModuleList([ConvBlock(D, G, dilations[k]) for k in range(BLOCKS_AFTER)])
52
+ self.oln = nn.LayerNorm(D)
53
+ self.head = nn.Linear(D, 4)
54
+
55
+ def forward(self, z, t, classes, style, mask_, self_cond):
56
+ mask = mask_[..., None]
57
+ B, length = mask_.shape
58
+ lengths = mask_.sum(1).clamp(min=1)
59
+ i = torch.arange(length, device=z.device, dtype=torch.float32)[None].expand(B, -1)
60
+ phase = (i / lengths[:, None])[..., None]
61
+ feats = torch.cat(
62
+ [z, self_cond, phase, torch.sin(2 * math.pi * phase), torch.cos(2 * math.pi * phase), torch.log(lengths)[:, None, None].expand(-1, length, -1) / 4.0, i[..., None] / 100.0],
63
+ -1,
64
+ )
65
+ x = (self.inp(feats) + self.ec(classes)[:, None] + self.ew(style)[:, None]) * mask
66
+ g = F.silu(self.tm(time_embedding(t)) + self.gc(classes) + self.gw(style))
67
+ for block in self.pre:
68
+ x = block(x, mask, g)
69
+ h = self.gfilm(self.gln(x), g)
70
+ with torch.autocast(DEVICE if DEVICE != "mps" else "cpu", enabled=False):
71
+ packed = nn.utils.rnn.pack_padded_sequence(h.float(), lengths.long().cpu(), batch_first=True, enforce_sorted=False)
72
+ out, _ = self.gru(packed)
73
+ out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True, total_length=length)
74
+ x = (x + self.gout(out.to(x.dtype))) * mask
75
+ for block in self.post:
76
+ x = block(x, mask, g)
77
+ out = self.head(self.oln(x)).float()
78
+ return out[..., :3] * mask, out[..., 3]
79
+
80
+
81
+ @torch.no_grad()
82
+ def _denoise(net, classes, style, lengths, generator):
83
+ B = len(classes)
84
+ length = int(max(lengths))
85
+ mask_ = torch.zeros(B, length, device=DEVICE)
86
+ for b, n in enumerate(lengths):
87
+ mask_[b, : int(n)] = 1
88
+ mask = mask_[..., None]
89
+ classes = torch.tensor(classes, device=DEVICE, dtype=torch.long)
90
+ style = torch.tensor(np.asarray(style), device=DEVICE, dtype=torch.float32)
91
+ z = torch.randn(B, length, 4, generator=generator).to(DEVICE) * mask
92
+ self_cond = torch.zeros_like(z)
93
+ steps = np.linspace(NOISE_STEPS - 1, 0, SAMPLE_STEPS).round().astype(int)
94
+ classes2 = torch.cat([classes, torch.full_like(classes, net.ncls)])
95
+ style2 = torch.cat([style, style])
96
+ mask2 = torch.cat([mask_, mask_])
97
+ for k, t in enumerate(steps):
98
+ tt = torch.full((B,), int(t), device=DEVICE, dtype=torch.long)
99
+ with torch.autocast("cuda", dtype=torch.bfloat16, enabled=AMP):
100
+ xy, pen = net(torch.cat([z, z]), torch.cat([tt, tt]), classes2, style2, mask2, torch.cat([self_cond, self_cond]))
101
+ xy = xy[B:] + GUIDANCE * (xy[:B] - xy[B:])
102
+ pen = pen[B:] + GUIDANCE * (pen[:B] - pen[B:])
103
+ x0 = torch.cat([xy[..., :2].float(), torch.tanh(pen.float() / 2)[..., None], xy[..., 2:3].float()], -1) * mask
104
+ self_cond = x0
105
+ if k == len(steps) - 1:
106
+ break
107
+ a, a_next = ALPHA_BAR[t], ALPHA_BAR[steps[k + 1]]
108
+ eps = (z - a.sqrt() * x0) / (1 - a).sqrt()
109
+ z = (a_next.sqrt() * x0 + (1 - a_next).sqrt() * eps) * mask
110
+ x0 = x0.cpu().numpy()
111
+ pen_up = (pen.float() > 0.0).cpu().numpy()
112
+ result = []
113
+ for b in range(B):
114
+ n = int(lengths[b])
115
+ q = np.zeros((n, 4), np.float32)
116
+ q[:, 0] = x0[b, :n, 0] * SCALE
117
+ q[:, 1] = x0[b, :n, 1] * SCALE
118
+ q[:, 2] = pen_up[b, :n]
119
+ q[-1, 2] = 1
120
+ q[:, 3] = np.clip((x0[b, :n, 3] + 1) / 2, 0, 1)
121
+ result.append(q)
122
+ return result
123
+
124
+
125
+ def _to_strokes(q):
126
+ strokes, current = [], []
127
+ for k in range(len(q)):
128
+ current.append([float(q[k, 0]), float(q[k, 1]), 0.0, float(q[k, 3])])
129
+ if q[k, 2] > 0.5 or k == len(q) - 1:
130
+ strokes.append(current)
131
+ current = []
132
+ return [s for s in strokes if s]
133
+
134
+
135
+ class GlyphModel:
136
+ def __init__(self, stats):
137
+ state = load_checkpoint(CKPT_GLYPH)
138
+ self.stats = stats
139
+ self.classes = state["classes"]
140
+ self.class_index = {c: i for i, c in enumerate(self.classes)}
141
+ self.net = GlyphDenoiser(len(self.classes)).to(DEVICE)
142
+ self.net.load_state_dict({k: v.float() for k, v in state["e"].items()})
143
+ self.net.eval()
144
+ # how many pen points real glyphs of each class have: a variant's length is drawn from this
145
+ self.lengths = {int(k.split("|")[1]): np.array(v, np.int32) for k, v in state["lens"].items() if k.startswith("-1|")}
146
+
147
+ def _draw_length(self, class_id, rng):
148
+ a = self.lengths.get(class_id)
149
+ if a is None or len(a) == 0:
150
+ return 30
151
+ return int(a[int(rng.integers(len(a)))])
152
+
153
+ def sample(self, classes, style, rng):
154
+ """One batch: class names -> {"strokes": ...}; classes the net does not know come back with no strokes."""
155
+ labels = ["—" if c == "BAR" else "\u20d7" if c == "VEC" else c for c in classes]
156
+ known = [i for i, label in enumerate(labels) if label in self.class_index]
157
+ result = [dict(strokes=[]) for _ in labels]
158
+ self.stats["glyph_net_net"] += len(known)
159
+ self.stats["glyph_net_unknown"] += len(labels) - len(known)
160
+ if not known:
161
+ return result
162
+ ids = [self.class_index[labels[i]] for i in known]
163
+ lengths = [self._draw_length(c, rng) for c in ids]
164
+ rows = np.zeros((len(known), STYLE_DIM + 1), np.float32)
165
+ rows[:, :STYLE_DIM] = style
166
+ generator = torch.Generator(device="cpu").manual_seed(int(rng.integers(1 << 31)))
167
+ out = _denoise(self.net, ids, rows, lengths, generator)
168
+ for j, i in enumerate(known):
169
+ result[i] = dict(strokes=_to_strokes(out[j]))
170
+ return result
171
+
172
+
173
+ class ClassifierNet(nn.Module):
174
+ def __init__(self, n):
175
+ super().__init__()
176
+ self.c = nn.Sequential(
177
+ nn.Conv2d(1, 32, 3, padding=1),
178
+ nn.ReLU(),
179
+ nn.MaxPool2d(2),
180
+ nn.Conv2d(32, 64, 3, padding=1),
181
+ nn.ReLU(),
182
+ nn.MaxPool2d(2),
183
+ nn.Conv2d(64, 128, 3, padding=1),
184
+ nn.ReLU(),
185
+ nn.AdaptiveAvgPool2d(2),
186
+ )
187
+ self.f = nn.Sequential(nn.Linear(128 * 4 + 2, 256), nn.ReLU(), nn.Dropout(0.3), nn.Linear(256, n))
188
+
189
+ def forward(self, image, extra):
190
+ return self.f(torch.cat([self.c(image).flatten(1), extra], 1))
191
+
192
+
193
+ def _raster(strokes, size=32):
194
+ points = np.array([p[:2] for s in strokes for p in s], np.float32)
195
+ x0, y0 = points.min(0)
196
+ x1, y1 = points.max(0)
197
+ scale = (size - 4) / max(x1 - x0, y1 - y0, 1e-6)
198
+ image = Image.new("L", (size, size), 0)
199
+ draw = ImageDraw.Draw(image)
200
+ ox = (size - (x1 - x0) * scale) / 2
201
+ oy = (size - (y1 - y0) * scale) / 2
202
+ for s in strokes:
203
+ q = [((p[0] - x0) * scale + ox, (p[1] - y0) * scale + oy) for p in s]
204
+ if len(q) > 1:
205
+ draw.line(q, fill=255, width=2)
206
+ elif q:
207
+ draw.point(q[0], fill=255)
208
+ return np.asarray(image, np.float32) / 255.0
209
+
210
+
211
+ def _shape_features(strokes, relative_height):
212
+ points = np.array([p[:2] for s in strokes for p in s], np.float32)
213
+ w = np.ptp(points[:, 0])
214
+ h = np.ptp(points[:, 1])
215
+ return np.array([relative_height, np.log((w + 1e-3) / (h + 1e-3))], np.float32)
216
+
217
+
218
+ class GlyphPicker:
219
+ """Chooses the most recognisable of several drawn variants of one symbol."""
220
+
221
+ def __init__(self, stats):
222
+ self.stats = stats
223
+ state = load_checkpoint(CKPT_GLYPH_CLASSIFIER)
224
+ self.net = ClassifierNet(len(state["classes"])).to(DEVICE)
225
+ self.net.load_state_dict(state["state"])
226
+ self.net.eval()
227
+ self.index = {c: i for i, c in enumerate(state["classes"])}
228
+ counts = json.load(open(STROKE_COUNTS, encoding="utf-8"))
229
+ self.stroke_counts = {c: {int(k): v for k, v in d.items()} for c, d in counts.items()}
230
+
231
+ def _stroke_share(self, cls, n):
232
+ counts = self.stroke_counts.get(cls)
233
+ total = sum(counts.values()) if counts else 0
234
+ return counts.get(n, 0) / total if total >= 20 else 1.0
235
+
236
+ def pick(self, variants, cls, relative_height, force=False):
237
+ """Index of the best variant, or None if none is recognised as `cls`. force: the best of what there is."""
238
+ if cls not in self.index:
239
+ self.stats["unscored"] += 1
240
+ return 0
241
+ drawn = lambda q: q["strokes"] and sum(len(t) for t in q["strokes"]) >= 2
242
+ ok = [k for k, q in enumerate(variants) if drawn(q) and self._stroke_share(cls, len(q["strokes"])) >= MIN_STROKE_SHARE]
243
+ if cls in ("0", "O", "o"): # no slashed zeros
244
+ ok = [k for k in ok if len(variants[k]["strokes"]) == 1]
245
+ if not ok and force:
246
+ ok = [k for k, q in enumerate(variants) if drawn(q)]
247
+ if not ok:
248
+ self.stats["fallback_nstrokes"] += 1
249
+ return None
250
+ images = np.stack([_raster(variants[k]["strokes"]) for k in ok])
251
+ extras = np.stack([_shape_features(variants[k]["strokes"], relative_height) for k in ok])
252
+ with torch.no_grad():
253
+ logits = self.net(torch.from_numpy(images)[:, None].to(DEVICE), torch.from_numpy(extras).to(DEVICE))
254
+ probs = torch.softmax(logits, -1).cpu().numpy()
255
+ own = probs[:, self.index[cls]]
256
+ recognised = probs.argmax(1) == self.index[cls]
257
+ others = probs.copy()
258
+ others[:, self.index[cls]] = -1
259
+ own = own - others.max(1)
260
+ recognised = recognised & (own >= PICK_MARGIN)
261
+ if not recognised.any() and force:
262
+ self.stats["forced"] += 1
263
+ return ok[int(np.argmax(own))]
264
+ if not recognised.any():
265
+ self.stats["fallback"] += 1
266
+ return None
267
+ self.stats["selected"] += 1
268
+ self.stats["first_ok"] += int(recognised[0])
269
+ return ok[int(np.argmax(own))]
scrawler/render.py ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Strokes -> vector PDF, SVG and a 300 dpi PNG.
2
+
3
+ Every stroke is a smooth curve: Catmull-Rom through the model's points, written as cubic Beziers. The PDF is written by hand, so
4
+ no extra library is needed. Strokes are lightly smoothed first ([1 2 1]/4 once): the models' points carry a small jitter that a
5
+ vector line would show.
6
+
7
+ A document: {"W": px, "H": px, "dpi": 150, "ruling": [y px, ...], "pages": [[stroke, ...], ...]}, stroke = [[x, y], ...], y down.
8
+ """
9
+ import os
10
+ import zlib
11
+
12
+ import numpy as np
13
+ from PIL import Image, ImageDraw
14
+
15
+
16
+ def _smooth(stroke, enabled):
17
+ s = np.asarray(stroke, np.float64)
18
+ if not enabled or len(s) < 5:
19
+ return s
20
+ t = s.copy()
21
+ t[1:-1] = 0.25 * s[:-2] + 0.5 * s[1:-1] + 0.25 * s[2:]
22
+ return t
23
+
24
+
25
+ def _bezier(s):
26
+ """Points [n, 2] -> cubic segments (p0, c1, c2, p1)."""
27
+ n = len(s)
28
+ if n < 2:
29
+ return []
30
+ P = np.vstack([s[:1], s, s[-1:]])
31
+ out = []
32
+ for i in range(1, n):
33
+ p0, p1 = P[i], P[i + 1]
34
+ out.append((p0, p0 + (P[i + 1] - P[i - 1]) / 6, p1 - (P[i + 2] - P[i]) / 6, p1))
35
+ return out
36
+
37
+
38
+ def _sample(segment, k=8):
39
+ p0, c1, c2, p1 = segment
40
+ t = np.linspace(0, 1, k + 1)[1:, None]
41
+ return (1 - t) ** 3 * p0 + 3 * (1 - t) ** 2 * t * c1 + 3 * (1 - t) * t**2 * c2 + t**3 * p1
42
+
43
+
44
+ def _write_pdf(pages, ruling, W, H, dpi, path, pen, smooth):
45
+ k = 72.0 / dpi
46
+ PW, PH = W * k, H * k
47
+ f = lambda x, y: f"{x * k:.2f} {PH - y * k:.2f}"
48
+ objects = {1: b"<< /Type /Catalog /Pages 2 0 R >>"}
49
+ kids = []
50
+ for i, strokes in enumerate(pages):
51
+ content = ["0.86 G 0.5 w"] + [f"{f(0, y)} m {f(W, y)} l S" for y in ruling]
52
+ content.append(f"0.08 G {pen:.2f} w 1 J 1 j")
53
+ for stroke in strokes:
54
+ s = _smooth(stroke, smooth)
55
+ if len(s) == 1:
56
+ content.append(f"{f(*s[0])} m {f(*s[0])} l S")
57
+ continue
58
+ content.append(f"{f(*s[0])} m " + " ".join(f"{f(*a)} {f(*b)} {f(*d)} c" for _, a, b, d in _bezier(s)) + " S")
59
+ data = zlib.compress("\n".join(content).encode())
60
+ page, stream = 3 + 2 * i, 4 + 2 * i
61
+ kids.append(page)
62
+ objects[page] = f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 {PW:.2f} {PH:.2f}] /Contents {stream} 0 R >>".encode()
63
+ objects[stream] = f"<< /Length {len(data)} /Filter /FlateDecode >>\nstream\n".encode() + data + b"\nendstream"
64
+ objects[2] = f'<< /Type /Pages /Kids [{" ".join(f"{p} 0 R" for p in kids)}] /Count {len(kids)} >>'.encode()
65
+ out = bytearray(b"%PDF-1.4\n")
66
+ offsets = {}
67
+ for n in sorted(objects):
68
+ offsets[n] = len(out)
69
+ out += f"{n} 0 obj\n".encode() + objects[n] + b"\nendobj\n"
70
+ xref = len(out)
71
+ out += f"xref\n0 {len(objects) + 1}\n0000000000 65535 f \n".encode()
72
+ out += b"".join(f"{offsets[n]:010d} 00000 n \n".encode() for n in sorted(objects))
73
+ out += f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n".encode()
74
+ with open(path, "wb") as fh:
75
+ fh.write(out)
76
+
77
+
78
+ def _write_svg(strokes, ruling, W, H, dpi, path, pen, smooth):
79
+ width = pen * dpi / 72.0
80
+ g = lambda p: f"{p[0]:.2f},{p[1]:.2f}"
81
+ parts = []
82
+ for stroke in strokes:
83
+ s = _smooth(stroke, smooth)
84
+ if len(s) == 1:
85
+ parts.append(f"M{g(s[0])} L{g(s[0])}")
86
+ else:
87
+ parts.append(f"M{g(s[0])} " + " ".join(f"C{g(a)} {g(b)} {g(d)}" for _, a, b, d in _bezier(s)))
88
+ lines = "".join(f'<line x1="0" y1="{y:.1f}" x2="{W}" y2="{y:.1f}" stroke="#dcdcdc" stroke-width="1"/>' for y in ruling)
89
+ with open(path, "w") as fh:
90
+ fh.write(
91
+ f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 {W} {H}" width="210mm" height="297mm"><rect width="100%" height="100%" fill="#fff"/>'
92
+ + lines
93
+ + f'<path d="{" ".join(parts)}" fill="none" stroke="#141414" stroke-width="{width:.2f}" stroke-linecap="round" stroke-linejoin="round"/></svg>'
94
+ )
95
+
96
+
97
+ def _write_png(strokes, ruling, W, H, dpi, path, pen, out_dpi, smooth):
98
+ oversample = 4
99
+ k = out_dpi / dpi * oversample
100
+ image = Image.new("L", (int(W * k), int(H * k)), 255)
101
+ draw = ImageDraw.Draw(image)
102
+ width = max(1, int(round(pen / 72.0 * out_dpi * oversample)))
103
+ for y in ruling:
104
+ draw.line([(0, y * k), (W * k, y * k)], fill=222, width=oversample)
105
+ for stroke in strokes:
106
+ s = _smooth(stroke, smooth)
107
+ points = np.vstack([s[:1]] + [_sample(seg) for seg in _bezier(s)]) * k if len(s) > 1 else s * k
108
+ if len(points) > 1:
109
+ draw.line([tuple(p) for p in points], fill=20, width=width, joint="curve")
110
+ for p in (points[0], points[-1]):
111
+ draw.ellipse([p[0] - width / 2, p[1] - width / 2, p[0] + width / 2, p[1] + width / 2], fill=20)
112
+ image.resize((int(W * out_dpi / dpi), int(H * out_dpi / dpi)), Image.LANCZOS).save(path, dpi=(out_dpi, out_dpi))
113
+
114
+
115
+ def render(doc, out, pen=1.1, png_dpi=300, smooth=True):
116
+ W, H, dpi, ruling = doc["W"], doc["H"], doc.get("dpi", 150), doc["ruling"]
117
+ _write_pdf(doc["pages"], ruling, W, H, dpi, os.path.join(out, "pages.pdf"), pen, smooth)
118
+ for i, strokes in enumerate(doc["pages"]):
119
+ _write_svg(strokes, ruling, W, H, dpi, os.path.join(out, f"page_{i + 1:02d}.svg"), pen, smooth)
120
+ _write_png(strokes, ruling, W, H, dpi, os.path.join(out, f"page_{i + 1:02d}.png"), pen, png_dpi, smooth)
scrawler/style_encoder.py ADDED
@@ -0,0 +1,99 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Writer style encoder: one line of handwriting -> a unit style vector [1, 256].
2
+
3
+ A line is an array [n, 3] of (dx, dy, pen_up) with line height 1.
4
+ """
5
+ import numpy as np
6
+ import torch
7
+ import torch.nn as nn
8
+ import torch.nn.functional as F
9
+
10
+ from .common import CKPT_STYLE, DEVICE, load_checkpoint
11
+
12
+ SCALE = 50.0
13
+ STEP = 0.05
14
+ MAX_POINTS = 1200
15
+
16
+
17
+ def resample(seq, step=STEP):
18
+ """Places points every `step` along the pen path, stroke by stroke."""
19
+ seq = np.asarray(seq, np.float32)
20
+ if len(seq) == 0:
21
+ return np.zeros((1, 3), np.float32)
22
+ points = np.cumsum(seq[:, :2], 0)
23
+ ends = np.flatnonzero(seq[:, 2] > 0.5)
24
+ starts = np.concatenate([[0], ends[:-1] + 1]) if len(ends) else np.array([0])
25
+ if len(ends) == 0 or ends[-1] != len(seq) - 1:
26
+ ends = np.append(ends, len(seq) - 1)
27
+ starts = np.concatenate([[0], ends[:-1] + 1])
28
+ out_points, out_pen = [], []
29
+ for a, b in zip(starts, ends):
30
+ if b < a:
31
+ continue
32
+ stroke = points[a : b + 1]
33
+ if len(stroke) > 1:
34
+ cum = np.concatenate([[0], np.cumsum(np.hypot(*np.diff(stroke, axis=0).T))])
35
+ total = cum[-1]
36
+ if total > step * 0.5:
37
+ t = np.arange(0, total, step)
38
+ if total - t[-1] > step * 0.25:
39
+ t = np.append(t, total)
40
+ stroke = np.stack([np.interp(t, cum, stroke[:, 0]), np.interp(t, cum, stroke[:, 1])], 1)
41
+ else:
42
+ stroke = stroke[[0]]
43
+ out_points.append(stroke)
44
+ pen = np.zeros(len(stroke), np.float32)
45
+ pen[-1] = 1
46
+ out_pen.append(pen)
47
+ points = np.concatenate(out_points)
48
+ deltas = np.zeros((len(points), 3), np.float32)
49
+ deltas[1:, :2] = np.diff(points, axis=0)
50
+ deltas[:, 2] = np.concatenate(out_pen)
51
+ return deltas
52
+
53
+
54
+ def features(seq):
55
+ seq = np.asarray(seq, np.float32)[:MAX_POINTS]
56
+ y = np.cumsum(seq[:, 1])
57
+ return np.stack([seq[:, 0] * SCALE, seq[:, 1] * SCALE, seq[:, 2], y - np.median(y)], 1).astype(np.float32)
58
+
59
+
60
+ class StyleNet(nn.Module):
61
+ def __init__(self, out_dim=256, d=192):
62
+ super().__init__()
63
+ self.stem = nn.Sequential(
64
+ nn.Conv1d(4, 128, 5, padding=2),
65
+ nn.GELU(),
66
+ nn.Conv1d(128, 192, 5, stride=2, padding=2),
67
+ nn.GELU(),
68
+ nn.Conv1d(192, 256, 5, stride=2, padding=2),
69
+ nn.GELU(),
70
+ )
71
+ self.gru = nn.GRU(256, d, num_layers=2, batch_first=True, bidirectional=True, dropout=0.1)
72
+ self.head = nn.Sequential(nn.Linear(4 * d, 256), nn.GELU(), nn.Dropout(0.2), nn.Linear(256, out_dim))
73
+
74
+ def forward(self, x, lengths):
75
+ h = self.stem(x.transpose(1, 2)).transpose(1, 2)
76
+ lengths = torch.clamp((lengths + 3) // 4, min=1) # the stem downsamples by 4
77
+ packed = nn.utils.rnn.pack_padded_sequence(h, lengths.cpu(), batch_first=True, enforce_sorted=False)
78
+ out, _ = self.gru(packed)
79
+ out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True)
80
+ mask = (torch.arange(out.size(1), device=out.device)[None] < lengths.to(out.device)[:, None]).float()[..., None]
81
+ mean = (out * mask).sum(1) / mask.sum(1)
82
+ top = (out + (mask - 1) * 1e4).max(1).values
83
+ return self.head(torch.cat([mean, top], -1))
84
+
85
+
86
+ def load_style_net():
87
+ state = load_checkpoint(CKPT_STYLE)
88
+ net = StyleNet(out_dim=256, d=state["D"]).to(DEVICE)
89
+ net.load_state_dict(state["m"])
90
+ net.eval()
91
+ return net
92
+
93
+
94
+ @torch.no_grad()
95
+ def style_vector(net, line):
96
+ x = features(resample(np.asarray(line, np.float32)[:, :3]))
97
+ batch = torch.from_numpy(x[None]).to(DEVICE)
98
+ vector = F.normalize(net(batch, torch.tensor([len(x)])).float(), dim=-1).mean(0, keepdim=True)
99
+ return F.normalize(vector, dim=-1).to(DEVICE)
scrawler/text_model.py ADDED
@@ -0,0 +1,254 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The text model: a diffusion denoiser over the pen points of a whole line, and its duration planner.
2
+
3
+ The planner says how many pen points each character gets. The denoiser then turns noise into points, knowing for every point
4
+ which character it belongs to and how far through that character it is. It is not autoregressive: all points are denoised at once.
5
+ """
6
+ import math
7
+
8
+ import numpy as np
9
+ import torch
10
+ import torch.nn as nn
11
+ import torch.nn.functional as F
12
+
13
+ from .common import AMP, CKPT_PLANNER, CKPT_TEXT, DEVICE, load_checkpoint
14
+
15
+ MAX_POINTS = 1200
16
+ NOISE_STEPS = 100
17
+ SAMPLE_STEPS = 20
18
+ GUIDANCE = 1.5
19
+ PHASES = 8
20
+ HIDDEN = 384
21
+ GRU_HIDDEN = 512
22
+ BLOCKS_BEFORE = 6
23
+ BLOCKS_AFTER = 4
24
+ ADVANCE = 0.017 # mean x advance per point; the net predicts the residual
25
+ SCALE_X = 0.6
26
+ SCALE_Y = 0.25
27
+
28
+
29
+ def alpha_bar(steps=NOISE_STEPS, s=0.008):
30
+ t = np.arange(1, steps + 1) / steps
31
+ f = np.cos((t + s) / (1 + s) * math.pi / 2) ** 2 / math.cos(s / (1 + s) * math.pi / 2) ** 2
32
+ return np.clip(f, 1e-4, 0.9999).astype(np.float32)
33
+
34
+
35
+ ALPHA_BAR = torch.tensor(alpha_bar(), device=DEVICE)
36
+
37
+
38
+ def time_embedding(t, dim=128):
39
+ half = dim // 2
40
+ freqs = torch.exp(-math.log(10000) * torch.arange(half, device=t.device) / half)
41
+ a = t.float()[:, None] * freqs[None] * (1000.0 / NOISE_STEPS)
42
+ return torch.cat([a.sin(), a.cos()], -1)
43
+
44
+
45
+ class FiLM(nn.Module):
46
+ def __init__(self, cond_dim, dim):
47
+ super().__init__()
48
+ self.l = nn.Linear(cond_dim, 2 * dim)
49
+ nn.init.zeros_(self.l.weight)
50
+ nn.init.zeros_(self.l.bias)
51
+
52
+ def forward(self, x, g):
53
+ a, b = self.l(g)[:, None].chunk(2, -1)
54
+ return x * (1 + a) + b
55
+
56
+
57
+ class ConvBlock(nn.Module):
58
+ """A ConvNeXt-style 1D block."""
59
+
60
+ def __init__(self, dim, cond_dim, dilation):
61
+ super().__init__()
62
+ self.dw = nn.Conv1d(dim, dim, 7, padding=3 * dilation, dilation=dilation, groups=dim)
63
+ self.ln = nn.LayerNorm(dim)
64
+ self.film = FiLM(cond_dim, dim)
65
+ self.f1 = nn.Linear(dim, 4 * dim)
66
+ self.f2 = nn.Linear(4 * dim, dim)
67
+ self.gam = nn.Parameter(torch.full((dim,), 0.1))
68
+
69
+ def forward(self, x, mask, g):
70
+ h = self.dw((x * mask).transpose(1, 2)).transpose(1, 2)
71
+ h = self.film(self.ln(h), g)
72
+ h = self.f2(F.gelu(self.f1(h)))
73
+ return (x + self.gam * h) * mask
74
+
75
+
76
+ class Denoiser(nn.Module):
77
+ def __init__(self, vocab_size):
78
+ super().__init__()
79
+ D, G = HIDDEN, 256
80
+ self.ec = nn.Embedding(vocab_size + 1, D) # current, previous and next character; the last row = "dropped" for guidance
81
+ self.ep = nn.Embedding(vocab_size + 1, D)
82
+ self.en = nn.Embedding(vocab_size + 1, D)
83
+ self.eph = nn.Embedding(PHASES, D)
84
+ self.inp = nn.Linear(3 + 3 + 3 + 2, D)
85
+ self.tm = nn.Sequential(nn.Linear(128, G), nn.SiLU(), nn.Linear(G, G))
86
+ self.sp = nn.Linear(256, G)
87
+ self.null_style = nn.Parameter(torch.zeros(256))
88
+ self.semb_proj = nn.Linear(256, 256)
89
+ self.sin = nn.Linear(256, D)
90
+ dilations = [1, 2, 4, 8, 16, 1, 2, 4, 8, 16]
91
+ self.pre = nn.ModuleList([ConvBlock(D, G, dilations[k]) for k in range(BLOCKS_BEFORE)])
92
+ self.gln = nn.LayerNorm(D)
93
+ self.gfilm = FiLM(G, D)
94
+ self.gru = nn.GRU(D, GRU_HIDDEN, num_layers=2, batch_first=True, bidirectional=True)
95
+ self.gout = nn.Linear(2 * GRU_HIDDEN, D)
96
+ self.post = nn.ModuleList([ConvBlock(D, G, dilations[k]) for k in range(BLOCKS_AFTER)])
97
+ self.oln = nn.LayerNorm(D)
98
+ self.head = nn.Linear(D, 3)
99
+
100
+ def forward(self, z, t, cond, style, self_cond):
101
+ mask_ = cond["mask"]
102
+ mask = mask_[..., None]
103
+ length = z.size(1)
104
+ pos = torch.arange(length, device=z.device, dtype=torch.float32)[None, :, None].expand(z.size(0), -1, -1) / 600.0
105
+ phase = cond["progress"][..., None]
106
+ feats = torch.cat(
107
+ [z, self_cond, phase, torch.sin(2 * math.pi * phase), torch.cos(2 * math.pi * phase), torch.log1p(cond["duration"])[..., None] / 3.0, pos],
108
+ -1,
109
+ )
110
+ x = (self.inp(feats) + self.ec(cond["char"]) + self.ep(cond["prev"]) + self.en(cond["next"]) + self.eph(cond["phase"]) + self.sin(style)[:, None]) * mask
111
+ g = F.silu(self.tm(time_embedding(t)) + self.sp(style))
112
+ for block in self.pre:
113
+ x = block(x, mask, g)
114
+ h = self.gfilm(self.gln(x), g)
115
+ lengths = mask_.sum(1).long().clamp(min=1)
116
+ with torch.autocast("cuda", enabled=False):
117
+ packed = nn.utils.rnn.pack_padded_sequence(h.float(), lengths.cpu(), batch_first=True, enforce_sorted=False)
118
+ out, _ = self.gru(packed)
119
+ out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True, total_length=length)
120
+ x = (x + self.gout(out.to(x.dtype))) * mask
121
+ for block in self.post:
122
+ x = block(x, mask, g)
123
+ out = self.head(self.oln(x)).float()
124
+ return out[..., :2] * mask, out[..., 2]
125
+
126
+
127
+ class Planner(nn.Module):
128
+ """Characters + style -> log1p of the number of pen points of each character."""
129
+
130
+ MAX_TOKENS = 200
131
+
132
+ def __init__(self, vocab_size, d=256, layers=4):
133
+ super().__init__()
134
+ self.emb = nn.Embedding(vocab_size, d)
135
+ self.pos = nn.Embedding(self.MAX_TOKENS + 2, d)
136
+ self.sty = nn.Linear(256, d)
137
+ self.null = nn.Parameter(torch.zeros(256))
138
+ self.enc = nn.TransformerEncoder(nn.TransformerEncoderLayer(d, 4, 4 * d, 0.1, batch_first=True, norm_first=True), layers)
139
+ self.out = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 1))
140
+
141
+ def forward(self, tokens, token_mask, style):
142
+ h = self.emb(tokens) + self.pos(torch.arange(tokens.shape[1], device=tokens.device))[None] + self.sty(style)[:, None]
143
+ return self.out(self.enc(h, src_key_padding_mask=token_mask < 0.5))[..., 0]
144
+
145
+
146
+ def _plan(char_ids, durations):
147
+ """Per-character plan -> per-point conditioning."""
148
+ char, prev, next_, phase, progress, duration = [], [], [], [], [], []
149
+ for k, (c, d) in enumerate(zip(char_ids, durations)):
150
+ d = max(1, int(d))
151
+ before = char_ids[k - 1] if k > 0 else 0
152
+ after = char_ids[k + 1] if k + 1 < len(char_ids) else 0
153
+ for q in range(d):
154
+ char.append(c)
155
+ prev.append(before)
156
+ next_.append(after)
157
+ phase.append(min(PHASES - 1, (q * PHASES) // d))
158
+ progress.append(q / d)
159
+ duration.append(d)
160
+ return char, prev, next_, phase, progress, duration
161
+
162
+
163
+ class TextModel:
164
+ def __init__(self):
165
+ state = load_checkpoint(CKPT_TEXT)
166
+ self.itos = state["itos"]
167
+ self.stoi = {s: i for i, s in enumerate(self.itos)}
168
+ self.null_token = len(self.itos)
169
+ self.net = Denoiser(len(self.itos)).to(DEVICE)
170
+ self.net.load_state_dict({k: v.float() for k, v in state["e"].items()})
171
+ self.net.eval()
172
+ state = load_checkpoint(CKPT_PLANNER)
173
+ self.planner = Planner(len(state["itos"])).to(DEVICE)
174
+ self.planner.load_state_dict(state["m"])
175
+ self.planner.eval()
176
+
177
+ @torch.no_grad()
178
+ def style(self, semb):
179
+ """The writer's style vector -> the vector the denoiser takes."""
180
+ return self.net.semb_proj(semb)
181
+
182
+ @torch.no_grad()
183
+ def durations(self, lines, semb):
184
+ """lines: lists of characters; semb [B, 256]. -> points per character for each line."""
185
+ B = len(lines)
186
+ U = min(Planner.MAX_TOKENS + 2, -(-(max(len(t) for t in lines) + 2) // 16) * 16)
187
+ tokens = torch.zeros(B, U, dtype=torch.long)
188
+ token_mask = torch.zeros(B, U)
189
+ for b, line in enumerate(lines):
190
+ ids = [1] + [self.stoi.get(c, 0) for c in line] + [2] # <bos> ... <eos>
191
+ tokens[b, : len(ids)] = torch.tensor(ids)
192
+ token_mask[b, : len(ids)] = 1
193
+ out = self.planner(tokens.to(DEVICE), token_mask.to(DEVICE), semb.to(DEVICE).float())
194
+ d = torch.expm1(out).clamp(min=1).round().long().cpu().numpy()
195
+ return [list(d[b, 1 : 1 + len(t)]) for b, t in enumerate(lines)]
196
+
197
+ @torch.no_grad()
198
+ def sample(self, lines, durations, style):
199
+ """-> for each line an array [n, 3]: dx, dy, pen_up. Uses the global torch random generator."""
200
+ B = len(lines)
201
+ plans = [_plan([self.stoi.get(c, 0) for c in t], d) for t, d in zip(lines, durations)]
202
+ length = min(MAX_POINTS, max(len(p[0]) for p in plans))
203
+ lengths = [min(length, len(p[0])) for p in plans]
204
+ cond = {k: torch.zeros(B, length, dtype=torch.long, device=DEVICE) for k in ("char", "prev", "next", "phase")}
205
+ cond.update(
206
+ progress=torch.zeros(B, length, device=DEVICE),
207
+ duration=torch.zeros(B, length, device=DEVICE),
208
+ mask=torch.zeros(B, length, device=DEVICE),
209
+ )
210
+ for b, plan in enumerate(plans):
211
+ n = lengths[b]
212
+ for key, values in zip(("char", "prev", "next", "phase", "progress", "duration"), plan):
213
+ cond[key][b, :n] = torch.tensor(values[:n], dtype=cond[key].dtype)
214
+ cond["mask"][b, :n] = 1
215
+ style = style.expand(B, -1).clone()
216
+
217
+ # classifier-free guidance: the same batch twice, the second time with the characters dropped
218
+ dropped = {
219
+ k: (torch.where(cond["mask"] > 0, torch.full_like(v, self.null_token), v) if k in ("char", "prev", "next") else v)
220
+ for k, v in cond.items()
221
+ }
222
+ cond2 = {k: torch.cat([cond[k], dropped[k]], 0) for k in cond}
223
+ style2 = torch.cat([style, style], 0)
224
+
225
+ mask = cond["mask"][..., None]
226
+ z = torch.randn(B, length, 3, device=DEVICE) * mask
227
+ self_cond = torch.zeros_like(z)
228
+ steps = np.linspace(NOISE_STEPS - 1, 0, SAMPLE_STEPS).round().astype(int)
229
+ for k, t in enumerate(steps):
230
+ tt = torch.full((B,), int(t), device=DEVICE, dtype=torch.long)
231
+ with torch.autocast("cuda", dtype=torch.bfloat16, enabled=AMP):
232
+ xy, pen = self.net(torch.cat([z, z], 0), torch.cat([tt, tt], 0), cond2, style2, torch.cat([self_cond, self_cond], 0))
233
+ xy = xy[B:] + GUIDANCE * (xy[:B] - xy[B:])
234
+ pen = pen[B:] + GUIDANCE * (pen[:B] - pen[B:])
235
+ x0 = torch.cat([xy.float(), torch.tanh(pen.float() / 2)[..., None]], -1) * mask
236
+ self_cond = x0
237
+ if k == len(steps) - 1:
238
+ break
239
+ a, a_next = ALPHA_BAR[t], ALPHA_BAR[steps[k + 1]]
240
+ eps = (z - a.sqrt() * x0) / (1 - a).sqrt()
241
+ z = (a_next.sqrt() * x0 + (1 - a_next).sqrt() * eps) * mask # DDIM
242
+
243
+ index = torch.arange(length, device=DEVICE, dtype=torch.float32)[None]
244
+ points = torch.stack([x0[..., 0] * SCALE_X + ADVANCE * index, x0[..., 1] * SCALE_Y], -1).cpu().numpy()
245
+ pen_up = (pen.float() > 0.0).float().cpu().numpy()
246
+ result = []
247
+ for b in range(B):
248
+ n = lengths[b]
249
+ d = np.zeros((n, 3), np.float32)
250
+ d[1:, :2] = np.diff(points[b, :n], axis=0)
251
+ d[:, 2] = pen_up[b, :n]
252
+ d[-1, 2] = 1
253
+ result.append(d)
254
+ return result
scripts/formula_line.py DELETED
@@ -1,227 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """04.11 (И164, Лев: «обучить модель формул и собрать устойчивый модуль их написания … генерируем символы и расставляем по латех»):
3
- a formula LINE in the hand of ONE reference line, in the line format of B (dx, dy, pen; line height 1, arc step STEP), so it can be
4
- measured and judged exactly like B's lines (product/eval/run_eval.py FORMULA=1).
5
- layout : MathJax boxes (formula_write.layout / compose_layout with its FIX40 fixes: script sizes, no-collision push, dots, donors)
6
- glyphs : the glyph denoiser with style from a LINE (train_glyph_dn GSTYLE=semb, GDN_CKPT, default ckpt_glyph_dn_u2s/fp16/step050000.pt) —
7
- the reference line's semb3_all embedding goes to glyph_dn.set_style; SEL_K variants per glyph, the glyph classifier picks
8
- the most typical (formula_write SEL_PICK=best); FIX_LIBCLS off (every class the net knows is generated, none from w01's
9
- library); classes the net does not know -> the real library (all writers) -> never silently dropped (counted in `missing`)
10
- marks : fraction bars / radicals / arrows from real hand strokes (formula_write LINE_MARKS=2)
11
- marks : slash, hat, tilde, overline, <-> (unknown to all of the above) -> pen lines over the box (own_marks)
12
- SLANT=1 : shear the line to the reference line's slant
13
- text : \\text{...} words (Cyrillic notes in formulas) are written by B itself with the same reference line (text_words replaced)
14
- python scripts/formula_line.py <out.png> "<tex>" [...] (eyeball: reference = REF_ID line of u2 test, default the first line of WRITER)
15
- As a module: setup(b_ckpt, planner) once, then formula_line(tex, ref_ink, rng) -> (seq [n,3] float32 or None, missing classes)."""
16
- import os, sys, json, numpy as np
17
- P = os.path.dirname(os.path.dirname(os.path.abspath(__file__))); sys.path.insert(0, os.path.join(P, 'scripts')); E = os.environ.get
18
- os.environ.update(PROJ=P); os.environ.setdefault('INK_DS', 'ink_lines_u2')
19
- for _k, _v in (('GEN', 'dn'), ('FIX40', '1'), ('LINE_MARKS', '2'), ('FIX_LIBCLS', ''), ('GDN_CFG', '1.2'), ('SEL_K', '4'), # SEL_K 4 (05.11, fgen_u2s_vbar_k4 vs _vbar: clean 65.4 vs 59.2, style 0.787 vs 0.779, 3x faster); was FIX40's 16
20
- ('GDN_CKPT', os.path.join(P, 'ckpt_glyph_dn_u2s', 'fp16', 'step050000.pt'))): os.environ.setdefault(_k, _v)
21
- STEP = float(E('STEP', '0.05'))
22
- import torch, train_ink_denoise as D, formula_write as FW, glyph_dn as GD
23
- _B = {}
24
-
25
- def setup(b_ckpt=None, planner=None):
26
- if _B: return _B
27
- import train_planner as TP
28
- net, step = D.load_net(b_ckpt or os.path.join(P, 'ckpt_dl', 'b4styc', 'fp16', 'step003000_fp16.pt')); base = D.load_base()
29
- _B.update(net=net, base=base, TP=TP, pnet=TP.load(planner or os.path.join(P, 'ckpt_dl', 'planner7', 'best.pt')), ref=None)
30
- FW.text_words = _text_words_b; FW.compose_text = _compose_text
31
- # 05.11: formula_write switches its generator on only in its own CLI (GEN='VAE' in __main__); imported, FW.GEN stayed None and EVERY glyph
32
- # came from the real library (glyphs_clean3: 94% w01) — the 04.11 runs fdn_u2s50k / fdn_mix2 and pages_v1 were library glyphs, not the net.
33
- # FL_LIB=1 reproduces that (library glyphs in the MathJax layout).
34
- if E('FL_LIB', '0') != '1': FW.GEN = 'VAE'
35
- return _B
36
-
37
- def _text_words_b(words): # formula_write.compose_text hook: words -> {word: [variant strokes [[x,y,p],...]]}, written by B in the reference hand
38
- B = _B; out = {}; words = [w for w in dict.fromkeys(words) if w.strip()]
39
- if not words or B.get('ref') is None: return out
40
- ref = B['ref']; toks = [list(w) for w in words]
41
- sv = torch.cat([D.style_of(B['net'], B['base'], [ref])] * len(words), 0); se = torch.cat([D.semb_vec([ref])] * len(words), 0)
42
- durs = B['TP'].predict(B['pnet'], toks, se); B['net']._tok_base = None
43
- for w, q in zip(words, D.sample(B['net'], toks, durs, sv)):
44
- q = np.asarray(q, np.float32); xy = np.cumsum(q[:, :2], 0) * 100.0; st, cur = [], []
45
- for k in range(len(q)):
46
- cur.append([float(xy[k, 0]), float(xy[k, 1]), 0.5])
47
- if q[k, 2] > 0.5: st.append(cur); cur = []
48
- if cur: st.append(cur)
49
- out[w] = [st]
50
- return out
51
-
52
- def _compose_text(texts, s, rng):
53
- """formula_write.compose_text with the word FITTED BY X-HEIGHT: B's word x-height (15..85% of its y) = the height of the layout's
54
- letter box (MathJax's Cyrillic box is about one x-height); the old 1.3 x box / interquartile rule made a subscript «отн» bigger than v"""
55
- words, cur = [], []
56
- for g in sorted(texts, key=lambda g: g['bbox'][0]):
57
- if not g.get('ch', '').strip():
58
- if cur: words.append(cur); cur = []
59
- continue
60
- if cur and g['bbox'][0] - cur[-1]['bbox'][2] > 50: words.append(cur); cur = []
61
- cur.append(g)
62
- if cur: words.append(cur)
63
- W = FW.text_words([''.join(g['ch'] for g in w) for w in words]); out = []
64
- for w in words:
65
- var = W.get(''.join(g['ch'] for g in w))
66
- if not var: continue
67
- st = var[0]; A = np.array([p[:2] for t in st for p in t], np.float32)
68
- bx0 = min(g['bbox'][0] for g in w) * s; bx1 = max(g['bbox'][2] for g in w) * s; yb = float(np.median([g['bbox'][3] for g in w])) * s
69
- xhb = float(np.median([g['bbox'][3] - g['bbox'][1] for g in w])) * s; xhw = max(np.percentile(A[:, 1], 85) - np.percentile(A[:, 1], 15), 1e-6)
70
- k = min(1.1 * (bx1 - bx0) / max(np.ptp(A[:, 0]), 1e-6), xhb / xhw); ay = np.percentile(A[:, 1], 85)
71
- for t in st: out.append([[float((p[0] - A[:, 0].min()) * k + bx0), float((p[1] - ay) * k + yb), float(p[2] if len(p) > 2 else 0.5)] for p in t])
72
- return out
73
-
74
- def fix_limits(gl):
75
- """an integral's lower limit sits under its foot in the layout (MathJax tucks it in); a hand writes it to the right of the foot:
76
- glyphs right after ∫/∮ whose box lies in the lower half of the sign and starts before its right edge are moved right of it"""
77
- for i, g in enumerate(gl):
78
- if g['c'] not in ('222B', '222E'): continue
79
- x0, y0, x1, y1 = g['bbox']; my = (y0 + y1) / 2; grp = []
80
- for h in gl[i + 1:i + 4]:
81
- if h['bbox'][0] < x1 and h['bbox'][1] > my: grp.append(h)
82
- if grp:
83
- dx = x1 + 0.05 * (y1 - y0) - min(h['bbox'][0] for h in grp)
84
- for h in grp: h['bbox'] = [h['bbox'][0] + dx, h['bbox'][1], h['bbox'][2] + dx, h['bbox'][3]]
85
- return gl
86
-
87
- MINE = {'/', '↔', '↦', 'ˆ', '^', '˜', '~', ' ̄', '¯', ' ̃', ' ̂', '‾', '\u0304', '\u0302', '\u0303', '\u0305'} # always pen marks (the net's '/' was rejected by the picker -> dropped; '↔' went to the hand marks -> library -> dropped)
88
- SUBST = {'2A7E': '2265', '2A7D': '2264'} # ⩾ ⩽ -> ≥ ≤ (the net has no slanted variants)
89
- def own_marks(gl, writer, rng):
90
- """glyphs neither the net, the library nor the hand marks can draw (slash, hat, tilde, overline, <->: they were silently dropped,
91
- 6 of 70 formulas of the measure) -> pen-line strokes over their layout box; returns (the other glyphs, strokes in page units)"""
92
- S = GD._load(); lib, scale = FW.library(); s = scale.get(writer, scale.get('w01', 0.1)); xh = 450 * s; keep, out = [], []
93
- for g in gl:
94
- c = FW.cls_of(g['c']) if g['c'] not in ('BAR', 'TEXT') else g['c']
95
- if g['c'] in SUBST: keep.append(dict(g, c=SUBST[g['c']])); continue
96
- if c not in MINE: keep.append(g); continue
97
- x0, y0, x1, y1 = [v * s for v in g['bbox']]; w = max(x1 - x0, 1e-6); h = max(y1 - y0, 1e-6); j = 0.015 * xh; ym = (y0 + y1) / 2
98
- if c == '↔': out += FW.line_mark('↔', [x0, y0, x1, y1], rng, xh)
99
- elif c == '↦': out += FW.line_mark('→', [x0, y0, x1, y1], rng, xh) + [FW._poly([[x0, ym - 0.4 * h], [x0, ym + 0.4 * h]], rng, j, 5)]
100
- elif c == '/': out.append(FW._poly([[x0 + 0.15 * w, y1], [x1 - 0.15 * w, y0]], rng, j, 10))
101
- elif c in ('ˆ', '^', ' ̂', '\u0302'): out.append(FW._poly([[x0 + 0.1 * w, y1], [(x0 + x1) / 2, y0], [x1 - 0.1 * w, y1]], rng, j, 5))
102
- elif c in ('˜', '~', ' ̃', '\u0303'): out.append([[float(x0 + t * w), float(ym - 0.35 * h * np.sin(2 * np.pi * t) + rng.normal(0, j)), 0.5] for t in np.linspace(0, 1, 14)])
103
- else: out.append(FW._poly([[x0, ym], [x1, ym]], rng, j, 10)) # overline / macron
104
- FW.SEL_STATS['own_mark_' + c] += 1
105
- return keep, out
106
-
107
- def slant_of(q): # as scripts/w03_style_eval.feats: median dx/dy over near-vertical inked steps
108
- q = np.asarray(q, np.float32); st = np.hypot(q[:, 0], q[:, 1]); v = (q[:, 2] < 0.5) & (np.abs(q[:, 1]) > 2 * np.abs(q[:, 0])) & (st > 0)
109
- return float(np.median(q[v, 0] / q[v, 1])) if v.sum() > 5 else None
110
- def shear(q, k): # x += k * (y - mean y) on the absolute points of a [n,3] delta line
111
- a = np.cumsum(q[:, :2], 0); a[:, 0] += k * (a[:, 1] - a[:, 1].mean()); d = q.copy(); d[1:, :2] = a[1:] - a[:-1]; d[0, :2] = 0; return d
112
-
113
- def to_line(strokes):
114
- """[[x,y,p],...] strokes (page units, y down) -> [n,3] dx,dy,pen (line height 1, arc step STEP) as scripts/ink_lines_prep.resample"""
115
- S = [np.asarray(s, np.float32)[:, :2] for s in strokes if len(s)]
116
- if not S: return None
117
- A = np.concatenate(S); x0, y0 = A.min(0); h = max(float(A[:, 1].max() - y0), 1e-3); out = []
118
- for p in S:
119
- p = (p - [x0, y0]) / h
120
- if len(p) == 1: out.append(np.array([[p[0, 0], p[0, 1], 1.0]], np.float32)); continue
121
- cum = np.r_[0, np.cumsum(np.hypot(*np.diff(p, axis=0).T))]
122
- s_ = np.r_[np.arange(0, cum[-1], STEP), cum[-1]] if cum[-1] > 1e-6 else np.array([0.0, cum[-1]])
123
- r = np.stack([np.interp(s_, cum, p[:, 0]), np.interp(s_, cum, p[:, 1]), np.zeros(len(s_))], 1).astype(np.float32); r[-1, 2] = 1; out.append(r)
124
- a = np.concatenate(out); d = a.copy(); d[1:, :2] = a[1:, :2] - a[:-1, :2]; d[0, :2] = 0; return d
125
-
126
- def compose_em(tex, ref_ink, rng, key='ref'):
127
- """tex -> (strokes [[x,y,p],...] in EM units: MathJax baseline y = 0, y down, x-height of 'x' 0.442; or None, missing classes)"""
128
- B = setup(); B['ref'] = np.asarray(ref_ink, np.float32); FW._TW.clear()
129
- GD.set_style(key, D.semb_vec([B['ref']])[0].detach().cpu().numpy())
130
- if E('GLYPH_B') == '1': import glyph_bdn; glyph_bdn.set_ref(key, B['ref']) # 05.11 И165: glyphs by B fine-tuned on glyphs
131
- t = tex.strip().strip('$').strip() # $...$ and $$...$$
132
- try:
133
- o = FW.layout([t])[0]
134
- if not o or not o.get('ok'): return None, ['LAYOUT']
135
- fix_limits(o['glyphs']); o['glyphs'], extra = own_marks(o['glyphs'], key, rng); strokes, missing = FW.compose_layout(o, key, rng); strokes += extra
136
- except Exception as e: return None, ['ERROR ' + repr(e)[:80]]
137
- if not strokes: return None, missing or ['EMPTY']
138
- s = FW.library()[1]; s = s.get(key, s.get('w01', 0.1)) * 1000.0 # compose_layout draws at layout units x s
139
- return [[[p[0] / s, p[1] / s, p[2] if len(p) > 2 else 0.5] for p in st] for st in strokes], missing
140
-
141
- def formula_line(tex, ref_ink, rng, key='ref'):
142
- """tex with or without the $...$; ref_ink [n,3] one line of the writer -> (seq [n,3] or None, missing classes)"""
143
- strokes, missing = compose_em(tex, ref_ink, rng, key)
144
- if not strokes: return None, missing
145
- q = to_line(strokes)
146
- if q is not None and E('SLANT', '0') == '1': # match the reference line's slant (the layout and the picker write upright)
147
- sr, sq = slant_of(_B['ref']), slant_of(q)
148
- if sr is not None and sq is not None: q = shear(q, float(np.clip(sr - sq, -0.5, 0.5)))
149
- return q, missing
150
-
151
- def spans(toks):
152
- """[(a, b)] token index ranges of $...$ (both dollars included)"""
153
- out, a, k = [], None, 0
154
- while k < len(toks):
155
- if toks[k] == '$':
156
- n = 2 if k + 1 < len(toks) and toks[k + 1] == '$' else 1 # $$ ... $$ = one delimiter
157
- if a is None: a = k
158
- else: out.append((a, k + n)); a = None
159
- k += n; continue
160
- k += 1
161
- return out
162
-
163
- def mixed_line(seq, toks, durs, ref_ink, rng, key='ref'):
164
- """MIXED line: B's line `seq` (written with the duration plan `durs` over `toks`) with every $...$ span replaced by the module's
165
- formula: its points (by the plan) are cut out, the formula is scaled so its x-height (0.442 em) = the x-height of B's text on the line
166
- (15..85% of y of the text points), put on B's text baseline (85% of y) one word-space after the text before it, and the text after it
167
- is shifted to follow one word-space later. -> (new seq or None when the plan does not match the line, missing classes, n replaced)"""
168
- q = np.asarray(seq, np.float32); ci = np.repeat(np.arange(len(toks)), [max(1, int(x)) for x in durs])
169
- if len(ci) != len(q): return None, ['PLAN'], 0
170
- A = np.cumsum(q[:, :2], 0); pen = q[:, 2].copy(); sp = spans(toks)
171
- if not sp: return q, [], 0
172
- infm = np.zeros(len(q), bool)
173
- for a, b in sp: infm |= (ci >= a) & (ci < b)
174
- txt = ~infm & np.array([not toks[c].isspace() for c in ci])
175
- if txt.sum() < 10: return None, ['NOTEXT'], 0
176
- y = A[txt, 1]; base = float(np.percentile(y, 85)); xh = max(float(np.percentile(y, 85) - np.percentile(y, 15)), 1e-3); gap = 0.6 * xh
177
- k = xh / 0.442; pieces = []; shift = 0.0; miss = []; prev = 0; nrep = 0
178
- for a, b in sp:
179
- m = (ci >= a) & (ci < b); ix = np.where(m)[0]
180
- before = np.arange(prev, ix[0]); pieces.append((A[before] + [shift, 0], pen[before]))
181
- x_left = (A[before, 0].max() + shift) if len(before) else A[ix, 0].min() + shift
182
- st, ms = compose_em(''.join(toks[a:b]).strip('$'), ref_ink, rng, key); miss += ms
183
- if not st: # the module could not write it -> keep B's own points of the span
184
- pieces.append((A[ix] + [shift, 0], pen[ix])); prev = ix[-1] + 1; continue
185
- f = to_line_abs(st, k); fx0 = f[:, 0].min(); f[:, 0] += x_left + gap - fx0; f[:, 1] += base
186
- if len(before): pieces[-1][1][-1] = 1.0 # lift the pen before the formula
187
- pieces.append((f[:, :2], f[:, 2])); nrep += 1
188
- nxt = np.arange(ix[-1] + 1, len(q)); x_right = A[nxt, 0].min() if len(nxt) else None
189
- if x_right is not None: shift = f[:, 0].max() + gap - x_right
190
- prev = ix[-1] + 1
191
- rest = np.arange(prev, len(q)); pieces.append((A[rest] + [shift, 0], pen[rest]))
192
- P_ = np.concatenate([p for p, _ in pieces if len(p)]); pn = np.concatenate([p for _, p in pieces if len(p)]); pn[-1] = 1.0
193
- d = np.zeros((len(P_), 3), np.float32); d[1:, :2] = P_[1:] - P_[:-1]; d[0, :2] = 0; d[:, 2] = pn; return d, miss, nrep
194
-
195
- def to_line_abs(strokes, k):
196
- """em strokes -> absolute points [n,3] (x, y, pen) scaled by k, resampled at STEP (line units), pen=1 on each stroke's last point"""
197
- out = []
198
- for st in strokes:
199
- p = np.asarray(st, np.float32)[:, :2] * k
200
- if len(p) == 1: out.append(np.array([[p[0, 0], p[0, 1], 1.0]], np.float32)); continue
201
- cum = np.r_[0, np.cumsum(np.hypot(*np.diff(p, axis=0).T))]
202
- s_ = np.r_[np.arange(0, cum[-1], STEP), cum[-1]] if cum[-1] > 1e-6 else np.array([0.0, cum[-1]])
203
- r = np.stack([np.interp(s_, cum, p[:, 0]), np.interp(s_, cum, p[:, 1]), np.zeros(len(s_))], 1).astype(np.float32); r[-1, 2] = 1; out.append(r)
204
- return np.concatenate(out)
205
-
206
- if __name__ == '__main__':
207
- from PIL import Image, ImageDraw, ImageFont
208
- out = sys.argv[1]
209
- if os.path.exists(out): sys.exit(f'{out} exists')
210
- z = np.load(os.path.join(P, 'data', 'ink_lines_u2', 'test_raw.npz'), allow_pickle=True); off = z['pt_off']; ink = lambda i: z['pts'][off[i]:off[i + 1]].astype(np.float32)
211
- meta = [json.loads(l) for l in open(os.path.join(P, 'data', 'ink_lines_u2', 'meta_test.jsonl'))]
212
- rid = int(E('REF_ID')) if E('REF_ID') else next(i for i, m in enumerate(meta) if m.get('writer_id') == E('WRITER', 'w13') and m.get('kind') == 'prose')
213
- rng = np.random.default_rng(int(E('SEED', '0'))); rows = [('образец', ink(rid))]
214
- for tex in sys.argv[2:]:
215
- q, miss = formula_line(tex, ink(rid), rng); print(tex, 'missing', miss, flush=True)
216
- if q is not None: rows.append((tex[:40], q))
217
- font = ImageFont.truetype(os.path.join(P, 'DejaVuSans.ttf'), 14); S = 80; ims = []
218
- for lab, q in rows:
219
- a = np.cumsum(q[:, :2], 0); a -= a.min(0); im = Image.new('L', (int(a[:, 0].max() * S) + 20, int(a[:, 1].max() * S) + 20), 255); d = ImageDraw.Draw(im); prev = None
220
- for (x, y), pen in zip(a, q[:, 2]):
221
- p_ = (10 + x * S, 10 + y * S)
222
- if prev is not None: d.line([prev, p_], fill=0, width=2)
223
- prev = None if pen > 0.5 else p_
224
- ims.append((lab, im))
225
- W = max(im.size[0] for _, im in ims) + 20; sh = Image.new('L', (min(W, 2400), sum(im.size[1] + 20 for _, im in ims)), 255); dr = ImageDraw.Draw(sh); y = 0
226
- for lab, im in ims: dr.text((6, y), lab, fill=120, font=font); sh.paste(im, (10, y + 16)); y += im.size[1] + 20
227
- sh.save(out); print(out, sh.size)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/formula_write.py DELETED
@@ -1,626 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """Write a LaTeX formula in a writer's hand from the library of their REAL glyphs (data/formula_glyphs/glyphs_clean.jsonl):
3
- MathJax layout (tools/mathjax/layout.mjs) gives each glyph's box; every box gets a random real sample of that class (same writer
4
- if available), scaled to the box height (BAR and wide glyphs: to the width), centred on the box, with human jitter: per-glyph
5
- offset (JIT_XY of the writer's x-height), size (JIT_S), rotation (JIT_R degrees), and a slow random walk of the baseline (DRIFT).
6
- Scale em->page units = median over the writer's glyphs of hand box height / layout box height.
7
- As a module: compose(tex, writer, rng, exclude_docs=()) -> (strokes [[x,y,p],...], missing classes).
8
- CLI (judge): N formulas from the valid notebooks of ink_dataset_v2 (their glyphs excluded from the library), each composed and
9
- read by the external VLM as LaTeX; the same is done for the REAL handwritten crops of those formulas (control). Prints CER on
10
- normalised LaTeX and writes figures/formula_write_<tag>.png. Env: PROJ, WRITER (w01), N (40), SEED (5), TAG, JIT_XY (0.06),
11
- JIT_S (0.06), JIT_R (2.5), DRIFT (0.02), OPENROUTER_KEY.
12
- 21.10: SEL_K (1 = old behaviour) -- with GEN_CKPT, draw K samples per glyph from the network and keep the one the glyph
13
- classifier (SEL_CLS, default ckpt_dl/glyph_cls1.pt from scripts/train_glyph_cls.py, trained without the valid notebooks) gives
14
- the highest probability of its own class, among samples whose stroke count has share >= SEL_NMIN (0.4) among the
15
- library's real glyphs of that class (the net draws x as ONE stroke, a half-x that the classifier still calls x); if no sample is recognised as its class (argmax != own for all K) -> a real glyph
16
- of that class from the library. Classes the classifier does not know keep sample 0. LINE_MARKS=1: fraction bars, radicals and
17
- arrows (BAR, sqrt, -> <- up down => <= <=>, <->, vector arrow) are drawn as pen lines over their layout box, not as glyphs.
18
- LINE_MARKS=2 (hand marks): the same marks from the writer's REAL strokes: a real fraction bar of similar length, stretched to
19
- box width x a real hand/layout length ratio (wobble, tilt, pen speed and pressure stay real); a radical = the hook of a real
20
- radical (cut where it reaches the top) scaled to the box height + a real bar continuing it in the same stroke to the box end;
21
- -> => <=> and the vector arrow = a real arrow whose shaft is stretched and head kept; rare arrows (up/down/left) = a real glyph
22
- as in the library mode; if the writer has no samples -> the LINE_MARKS=1 line.
23
- GEN=vae (default: the glyph1 net via GEN_CKPT): glyphs from the glyph VAE (scripts/glyph_vae.py, ckpt_glyph_vae/); with SEL_K the
24
- K variants differ by z (a random real glyph of the writer, shrunk: GVAE_SHRINK, default 0.75) and MDN temperature (GVAE_TAUS,
25
- cycled, default 0.2..0.4); with GEN=vae the pick defaults to SEL_PICK=any: a random variant among those with p(own class) >=
26
- SEL_PMIN (0.8), else the best (SEL_PICK=best = always the most probable = least varied). Classes unknown to the VAE -> library.
27
- XF (I90, 22.10; off by default) = per-glyph deformations on top of the layout: XF=w03 -> shear + parabolas (w03's list), XF=all ->
28
- all below; each amplitude has its own variable (0 switches it off). Glyph coordinates centred and divided by L = max(w, h).
29
- Measured (logs/glyph_transform_stats.py -> figures/glyph_transform_stats.json): top-up of w01's within-class spread over the
30
- VAE's, sqrt(real^2 - vae^2), median of the 30 most frequent classes: shear_x 0.14, shear_y 0.19, parabola_v 1.47, parabola_h 0.37,
31
- log aspect 0.15 -- but this spread is mostly DIFFERENT SHAPES of a class (allographs), not deformation: at 0.8 x these values the
32
- classifier recognised 73% instead of 94% (parabola_v alone -12 pp). Defaults = the largest set that keeps recognition within ~1 pp
33
- (400 held-out glyphs): XF_SHX 0.06 (x += s y), XF_SHY 0.05 (y += s x), XF_PV 0.10 (y += a x^2), XF_PH 0.08 (x += b y^2),
34
- XF_ASP 0.04 (log aspect stretch), XF_ASC 0.04 (separate stretch of the parts above/below the middle band |y|>0.25), XF_PERSP 0.02
35
- (mild projective, not measured), XF_TREM 0 (low-pass tremor; w01's strokes are smoother than the VAE's -> off), XF_SPEED 0.5
36
- (uneven point density along a stroke: her step-length CV 0.64 vs VAE 0.11; shape unchanged), XF_PRES 0.27 (log pressure scale, her
37
- per-class spread 0.34), XF_GAP 0.22 and XF_BASE 0.08 (AR(0.7)/AR(0.8) offsets along the formula in glyph heights: her residual of
38
- hand vs layout centres 0.32 in x, 0.125 in y, minus JIT_XY, x0.7).
39
- GEN=vae2 (M3, 22.10; THE DEFAULT since 22.10 ~11:00 -- GEN=vae = previous VAE mode, GEN=lib = library/glyph1 as before): GEN=vae with the
40
- M3 defaults -- GVAE_CKPT ckpt_glyph_vae_iiC3L/last.pt (22.10 ~12:40, coordinator: best formula judge 35/30/0.07, diversity 0.93; GVAE_CKPT=ckpt_glyph_vae_iiC3/last.pt = the previous one) (VAE on glyphs_clean3 + classifier penalty,
41
- free bits 0.75/dim, KL weight 0.1, decoder input dropout 0.5: z carries ~24 nats -> within-class diversity 0.93 of w01's), GVAE_SHRINK 1.0,
42
- GVAE_TAUS 0.1-0.3,
43
- SEL_CLS ckpt_dl/glyph_cls2.pt (clean classifier), SEL_K 8, library FW_LIB=glyphs_clean3.jsonl (fallback and hand marks); each can be
44
- overridden by its variable. Shape donors (DONORS=1, vae2 only): O <- 0, κ <- k, Σ <- ∑ (the cleaned library has 1, 2 and 12 such
45
- glyphs; the net cannot learn them).
46
- FIX40=1 (22.10, w03: 40/40 by eye; DEFAULT for GEN=vae2, FIX40=0 = without) sets SEL_PICK=best SEL_K=16 XF_GAP=0.1, keeps operators (= + - < > etc.) unsheared under XF, draws
47
- FIX_LIBCLS (λ,∩,d,μ: the VAE draws them badly) and [ ] from the real library, dot accents as a pen dot, and switches on three fixes, each also by its own flag (default off = old results):
48
- FIX_SCRIPT -- sub/superscripts at w01's measured size (logs/script_scale_stats.py: script = layout box < 0.85 x the same char at base
49
- level; her script size relative to MathJax's: digits 0.71, Latin letters 1.08 -> kept 1.0, other 0.92; SCRIPT_DIGIT/SCRIPT_OTHER);
50
- FIX_TEXT -- \text{...} (Cyrillic) written by the LINE model (scripts/text_words.py, dpo1@1500, w01 refs) into the layout's text
51
- boxes (tools/mathjax/layout.mjs LAYOUT_TEXT=1); FIX_DONOR -- << = two '<' glyphs; ->, <-, down/up arrows, vector arrow = a real bar
52
- (shaft) + a real '>' (head), rotated as needed (w01's own => is '=' + '>'; her curled real arrows read as =>).
53
- The 21.10 vae40_sel_hand run = GVAE_SHRINK=0.5 GVAE_TAUS=0.1,0.2,0.3,0.15,0.1,0.25,0.2,0.3 SEL_PICK=best."""
54
- import os, sys, json, math, subprocess, collections, unicodedata, re, numpy as np
55
- from PIL import Image, ImageDraw
56
- PROJ=os.environ.get('PROJ',os.getcwd()); E=os.environ.get
57
- if __name__=='__main__': os.environ.setdefault('GEN','vae2') # only when run as the CLI (importers of compose() keep their own env); M3 (22.10 ~11:00, coordinator): vae2 is the default; GEN=vae = the 21.10/22.10 VAE mode, GEN=lib (any other value) = library / glyph1 via GEN_CKPT as before
58
- if E('GEN') in ('vae2','dn'): os.environ.setdefault('FIX40','1') # 22.10: the 40/40 fixes are the vae2 default (FIX40=0 = M3's vae2 as before)
59
- if E('FIX40')=='1': # 22.10 40/40: the most typical of 16 variants (correct shape over variety), smaller gap jitter (0.22 made glyphs collide)
60
- for _k,_v in (('SEL_PICK','best'),('SEL_K','16'),('XF_GAP','0.1')): os.environ.setdefault(_k,_v)
61
- if E('GEN')=='dn': # И116 (22.10): GEN=dn = the glyph DENOISER (scripts/glyph_dn.py, ckpt GDN_CKPT, CFG GDN_CFG, DDIM GDN_STEPS) with the vae2 picker/library defaults
62
- for _k,_v in (('GDN_CKPT',os.path.join(PROJ,'ckpt_dl','glyphdn','glyphdnB','fp16','step050000.pt')),('GDN_CFG','1.2'),('SEL_CLS',os.path.join(PROJ,'ckpt_dl','glyph_cls2.pt')),('SEL_K','8'),('FW_LIB','glyphs_clean3.jsonl')): os.environ.setdefault(_k,_v)
63
- if E('GEN')=='vae2': # M3 (22.10): the M3 glyph VAE trained on the cleaned library + the clean classifier as the picker; other modes untouched
64
- for _k,_v in (('GVAE_CKPT',os.path.join(PROJ,'ckpt_glyph_vae_iiC3L','last.pt')),('GVAE_SHRINK','1.0'),('GVAE_TAUS','0.1,0.2,0.3,0.15,0.25'),('SEL_CLS',os.path.join(PROJ,'ckpt_dl','glyph_cls2.pt')),('SEL_K','8'),('FW_LIB','glyphs_clean3.jsonl')): os.environ.setdefault(_k,_v)
65
- FIX40=E('FIX40','0')=='1'; FIX_LIBCLS=set(E('FIX_LIBCLS','λ,∩,d,μ,=,M,θ,j,t').split(',')) if E('FIX_LIBCLS','λ,∩,d,μ,=,M,θ,j,t') else set(); XF_KEEP=set('=+−-<>≠±') # XF_KEEP: operators keep their shape under FIX40 (a sheared '=' reads as '>')
66
- FIX_SCRIPT=E('FIX_SCRIPT','1' if FIX40 else '0')=='1'; FIX_TEXT=E('FIX_TEXT','1' if FIX40 else '0')=='1'; FIX_DONOR=E('FIX_DONOR','1' if FIX40 else '0')=='1' # 22.10 40/40 task, see FIX40 in the docstring
67
- SEL_K=int(E('SEL_K','1')); SEL_NMIN=float(E('SEL_NMIN','0.4')); LINE_MARKS=int(E('LINE_MARKS','0')); SEL_STATS=collections.Counter()
68
- JIT_XY=float(E('JIT_XY','0.06')); JIT_S=float(E('JIT_S','0.06')); JIT_R=float(E('JIT_R','2.5')); DRIFT=float(E('DRIFT','0.02'))
69
- D=os.path.join(PROJ,'data','formula_glyphs')
70
- def cls_of(c):
71
- if c=='BAR': return 'BAR'
72
- ch=chr(int(c,16))
73
- if ch=='\u20d7': return 'VEC'
74
- n=unicodedata.normalize('NFKC',ch); return n if n.strip() else ch
75
- _LIB=None
76
- def library():
77
- global _LIB
78
- if _LIB is None and E('FW_LIB')=='none': _LIB=(collections.defaultdict(list),{}) # 06.11 open release: no library of real people's strokes (LINE_MARKS=1, FW_NS)
79
- if _LIB is None:
80
- lib=collections.defaultdict(list); sc=collections.defaultdict(list)
81
- for l in open(os.path.join(D,E('FW_LIB','glyphs_clean.jsonl'))):
82
- g=json.loads(l); lib[(g['src'],g['cls'])].append(g); lib[('*',g['cls'])].append(g)
83
- lh=g['lbox'][3]-g['lbox'][1]
84
- if g['cls'] not in ('BAR',) and lh>200: sc[g['src']].append((g['box'][3]-g['box'][1])/lh)
85
- _LIB=(lib,{k:float(np.median(v)) for k,v in sc.items()})
86
- return _LIB
87
- def layout(texs):
88
- inp='\n'.join(json.dumps(dict(id=i,tex=t),ensure_ascii=False) for i,t in enumerate(texs))+'\n'
89
- res=subprocess.run(['node','layout.mjs'],input=inp,capture_output=True,text=True,cwd=os.path.join(PROJ,'tools','mathjax'),env=dict(os.environ,LAYOUT_TEXT='1' if FIX_TEXT else '0'))
90
- out={}
91
- for l in res.stdout.splitlines():
92
- try: o=json.loads(l); out[o['id']]=o
93
- except ValueError: pass
94
- return [out.get(i) for i in range(len(texs))]
95
- def merge_radicals(gl):
96
- out=[]; i=0
97
- while i<len(gl):
98
- x=gl[i]
99
- if x['c']=='221A' and i+1<len(gl) and gl[i+1]['c']=='BAR' and gl[i+1].get('node') in ('msqrt','mroot'):
100
- b=gl[i+1]['bbox']; x=dict(c='221A',sw=x['bbox'][2]-x['bbox'][0],bbox=[min(x['bbox'][0],b[0]),min(x['bbox'][1],b[1]),max(x['bbox'][2],b[2]),max(x['bbox'][3],b[3])]); i+=2 # sw: width of the sign itself (hand radicals, LINE_MARKS=2)
101
- else: i+=1
102
- out.append(x)
103
- return out
104
- GEN=None
105
- def glyph_char(c):
106
- if c=='BAR': return '\u2014'
107
- if c=='VEC': return '\u20d7'
108
- return c
109
- def load_gen(ckpt):
110
- """glyph generator (train_ink_al2 trained on data/ink_glyphs): returns (module, model)"""
111
- global GEN
112
- os.environ.setdefault('INK_DS','ink_glyphs'); os.environ.setdefault('META_DS','ink_dataset_glyphs'); os.environ.setdefault('ALIGN','align_all.npz'); os.environ.setdefault('HW_RUN','glyph_gen')
113
- os.environ.setdefault('MAXPTS','600'); os.environ.setdefault('MAXTOK','4'); os.environ.setdefault('PRES','1'); os.environ.setdefault('HID','384')
114
- sys.path.insert(0,os.path.join(PROJ,'scripts')); import importlib; TI=importlib.import_module('train_ink_al2'); import torch
115
- st=torch.load(ckpt,map_location=TI.dev); m=TI.Model().to(TI.dev); m.load_state_dict({k:v.float() for k,v in st.get('e',st.get('m',st)).items()}); m.eval(); GEN=(TI,m); return GEN
116
- def gen_samples(chars,writer,rng,exclude_docs=()):
117
- """one generated glyph per char, style from 3 real glyphs of the writer (same page) -> list of strokes lists [[x,y,0,p],...]"""
118
- TI,m=GEN; lib,_=library(); import torch
119
- pool=[q for q in lib.get(('*','x'),[])+lib.get(('*','2'),[])+lib.get(('*','a'),[]) if q['src']==writer and q['doc'] not in exclude_docs]
120
- ref=[]
121
- for q in [pool[int(i)] for i in rng.choice(len(pool),3,replace=False)]:
122
- pts=[np.asarray(st,np.float32) for st in q['strokes']]; allp=np.concatenate([p[:,:2] for p in pts]); h=max(np.ptp(allp[:,1])*2.0,1.0)
123
- seq=[]; prev=allp[0]
124
- for p in pts:
125
- for j,pt in enumerate(p): seq.append([(pt[0]-prev[0])/h,(pt[1]-prev[1])/h,1.0 if j==len(p)-1 else 0.0]); prev=pt[:2]
126
- ref.append(np.asarray(seq,np.float32))
127
- sv=TI.style_from_seqs(m,ref); torch.manual_seed(int(rng.integers(1<<30)))
128
- toks=[[glyph_char(c)] for c in chars]; seqs=TI.sample(m,toks,bias=float(os.environ.get('GBIAS','1.0')),sv=sv) if toks else []
129
- out=[]
130
- for q in seqs:
131
- q=np.asarray(q,np.float32); xy=np.cumsum(q[:,:2],0); pr=q[:,3] if q.shape[1]>3 else np.full(len(q),0.5,np.float32)
132
- st=[]; cur=[]
133
- for k in range(len(q)):
134
- cur.append([float(xy[k,0]),float(xy[k,1]),0.0,float(pr[k])])
135
- if q[k,2]>0.5: st.append(cur); cur=[]
136
- if cur: st.append(cur)
137
- out.append(dict(strokes=st))
138
- return out
139
- LINE_SET={'BAR','√','→','←','↑','↓','⇒','⇐','⇔','↔','VEC'}
140
- def _seg(a,b,n=8): return [[a[0]+(b[0]-a[0])*t,a[1]+(b[1]-a[1])*t] for t in np.linspace(0,1,n)]
141
- def _poly(P,rng,w,n=6):
142
- out=[]
143
- for i in range(len(P)-1): q=_seg(P[i],P[i+1],n); out+=q if not out else q[1:]
144
- return [[x+rng.normal(0,w),y+rng.normal(0,w),0.5] for x,y in out]
145
- def line_mark(c,b,rng,xh):
146
- """pen-line strokes for a fraction bar / radical / arrow over layout box b (page units, y down)"""
147
- x0,y0,x1,y1=b; w=x1-x0; h=max(y1-y0,1e-6); ym=(y0+y1)/2; xm=(x0+x1)/2; j=0.02*xh; tilt=rng.normal(0,0.01)*w
148
- if c=='BAR': return [_poly([[x0-0.03*w,ym-tilt],[x1+0.03*w,ym+tilt]],rng,j,12)]
149
- if c=='√':
150
- tw=min(0.5*h,0.4*w); return [_poly([[x0,y0+0.6*h],[x0+0.18*tw,y0+0.52*h],[x0+0.55*tw,y1],[x0+tw,y0],[x1,y0+tilt]],rng,j,6)]
151
- hl=min(0.35*w,max(0.9*h,0.25*xh)) if c not in ('↑','↓') else min(0.35*h,max(0.9*w,0.25*xh))
152
- if c=='VEC': return [_poly([[x0,ym],[x1,ym]],rng,j,8),_poly([[x1-hl,ym-0.5*hl],[x1,ym],[x1-hl,ym+0.5*hl]],rng,j,4)]
153
- if c in ('↑','↓'):
154
- tip,sg=(y1,-1) if c=='↓' else (y0,1); return [_poly([[xm,y0],[xm,y1]],rng,j,10),_poly([[xm-0.5*hl,tip+sg*hl],[xm,tip],[xm+0.5*hl,tip+sg*hl]],rng,j,4)]
155
- dbl=c in ('⇒','⇐','⇔'); d=0.22*h if dbl else 0.0; st=[]
156
- for yy in ([ym-d,ym+d] if dbl else [ym]): st.append(_poly([[x0+(0.15*hl if c in '⇐⇔' else 0),yy],[x1-(0.15*hl if c in '⇒⇔' else 0),yy]],rng,j,10))
157
- if c in ('→','⇒','⇔','↔'): st.append(_poly([[x1-hl,ym-0.55*hl-d],[x1,ym],[x1-hl,ym+0.55*hl+d]],rng,j,4))
158
- if c in ('←','⇐','⇔','↔'): st.append(_poly([[x0+hl,ym-0.55*hl-d],[x0,ym],[x0+hl,ym+0.55*hl+d]],rng,j,4))
159
- return st
160
- CLS=None
161
- def classifier():
162
- global CLS
163
- if CLS is None:
164
- sys.path.insert(0,os.path.join(PROJ,'scripts')); import train_glyph_cls as GC
165
- net,classes=GC.load(E('SEL_CLS',os.path.join(PROJ,'ckpt_dl','glyph_cls1.pt'))); CLS=(GC,net,{c:i for i,c in enumerate(classes)})
166
- return CLS
167
- _NS=None
168
- def nstroke_p(c,n):
169
- """share of the library's real glyphs of class c with exactly n strokes (structure check: the net draws x as one stroke = half an x)"""
170
- global _NS
171
- if _NS is None and E('FW_NS'): # 06.11: the same statistics precomputed (per class: stroke count -> glyphs), no strokes
172
- _NS={c:(collections.Counter({int(k):v for k,v in d.items()}),sum(d.values())) for c,d in json.load(open(E('FW_NS'))).items()}
173
- if _NS is None:
174
- lib,_=library(); _NS={}
175
- for (src,cc),L in lib.items():
176
- if src=='*': cnt=collections.Counter(len(g['strokes']) for g in L); _NS[cc]=(cnt,len(L))
177
- cnt,tot=_NS.get(c,(None,0))
178
- return cnt[n]/tot if tot>=20 else 1.0
179
- def pick_sample(cands,c,relh,rng=None,force=False):
180
- """index of the best of K generated samples for class c, or None (no sample recognised as c -> library)"""
181
- GC,net,ci=classifier(); import torch
182
- if c not in ci: SEL_STATS['unscored']+=1; return 0
183
- ok=[k for k,q in enumerate(cands) if q['strokes'] and sum(len(t) for t in q['strokes'])>=2 and nstroke_p(c,len(q['strokes']))>=SEL_NMIN]
184
- if FIX40 and c in ('0','O','o'): ok=[k for k in ok if len(cands[k]['strokes'])==1] # no slashed zeros (Ø)
185
- if not ok and force: ok=[k for k,q in enumerate(cands) if q['strokes'] and sum(len(t) for t in q['strokes'])>=2] # force (06.11): the best of what there is, never None while anything was drawn
186
- if not ok: SEL_STATS['fallback_nstrokes']+=1; return None
187
- X=np.stack([GC.render(cands[k]['strokes']) for k in ok]); A=np.stack([GC.feats(cands[k]['strokes'],relh) for k in ok])
188
- with torch.no_grad(): P=torch.softmax(net(torch.from_numpy(X)[:,None].to(GC.dev),torch.from_numpy(A).to(GC.dev)),-1).cpu().numpy()
189
- own=P[:,ci[c]]; rec=P.argmax(1)==ci[c]
190
- if FIX40: # 22.10: a look-alike twin (b/ℓ, t/τ, 0/θ) must lose clearly: rank by the margin over the best other class, need >= SEL_MARGIN
191
- oth=P.copy(); oth[:,ci[c]]=-1; own=own-oth.max(1); rec=rec&(own>=float(E('SEL_MARGIN','0.25')))
192
- if not rec.any() and force: SEL_STATS['forced']+=1; return ok[int(np.argmax(own))]
193
- if not rec.any(): SEL_STATS['fallback']+=1; return None
194
- SEL_STATS['selected']+=1; SEL_STATS['first_ok']+=int(rec[0])
195
- if E('SEL_PICK','any' if GEN=='VAE' else 'best')=='any': # I88: a random variant among the confidently recognised ones (argmax p_own always picks the most typical shape)
196
- top=np.zeros(len(own),bool); top[np.argsort(-own)[:int(E('SEL_TOP','2'))]]=True # only among the SEL_TOP most probable: the classifier gives alpha-shaped lambdas p 0.80-0.97 (lambda labels hold alphas)
197
- r=np.flatnonzero(rec&(own>=float(E('SEL_PMIN','0.8')))&top)
198
- if len(r): return ok[int(r[(rng or np.random.default_rng()).integers(len(r))])]
199
- return ok[int(np.argmax(own))]
200
- _HM=None
201
- def _hm_pools(writer,exclude_docs):
202
- """real strokes for hand marks: bars (one stroke, with hand/layout length ratios), radical hooks, arrows"""
203
- global _HM
204
- key=(writer,tuple(sorted(exclude_docs)))
205
- if _HM is not None and _HM[0]==key: return _HM[1]
206
- lib,scale=library(); sc=scale.get(writer,scale.get('w01',0.1)); P={}
207
- def pool(c): return [g for g in lib.get((writer,c),[]) if g['doc'] not in exclude_docs] or [g for g in lib.get(('*',c),[]) if g['doc'] not in exclude_docs]
208
- bars=[]
209
- for g in pool('BAR'):
210
- if len(g['strokes'])!=1 or len(g['strokes'][0])<4: continue
211
- q=np.asarray(g['strokes'][0],np.float32); L=float(np.ptp(q[:,0]))
212
- if L<1e-3: continue
213
- bars.append((q,L,L/max((g['lbox'][2]-g['lbox'][0])*sc,1e-6)))
214
- P['BAR']=bars; P['BAR_L']=np.array([b[1] for b in bars]); P['BAR_OV']=np.array([b[2] for b in bars])
215
- hooks=[]
216
- for g in pool('√'):
217
- hh=np.ptp(np.concatenate([np.asarray(t,np.float32)[:,1] for t in g['strokes']]))
218
- for t in sorted(g['strokes'],key=lambda t:min(p[0] for p in t)):
219
- q=np.asarray(t,np.float32)
220
- if len(q)<5 or np.ptp(q[:,1])<0.6*hh: continue
221
- lo=int(np.argmax(q[:,1])); top=q[:,1].min()+0.12*np.ptp(q[:,1]); j=next((k for k in range(lo,len(q)) if q[k,1]<=top),None)
222
- if j is None or j-lo<2: break
223
- hk=q[:j+1]; w=np.ptp(hk[:,0]); h=np.ptp(hk[:,1])
224
- if 0.25*h<=w<=1.2*h and hk[0,0]<=q[lo,0]<=hk[-1,0]+0.1*h and hk[0,1]<q[lo,1]-0.2*h: hooks.append(hk) # left start above the bottom, down, then up to the top-right
225
- break
226
- P['√']=hooks
227
- for c in ('=','>','<'): P['G'+c]=[[np.asarray(t,np.float32) for t in g['strokes']] for g in pool(c)] # w01 writes => as '=' + '>' (real column)
228
- def _chev_ok(q,right): # a clean chevron: one point of the extreme x in the middle third of the height, both ends on the other side
229
- A=np.concatenate(q)[:,:2]; w=np.ptp(A[:,0]); h=np.ptp(A[:,1])
230
- if w<1e-6 or not 0.6<=h/w<=3.0: return False
231
- k=int(np.argmax(A[:,0])) if right else int(np.argmin(A[:,0])); yk=(A[k,1]-A[:,1].min())/max(h,1e-6)
232
- ends=[t[0,0] for t in q]+[t[-1,0] for t in q]; xr=[(e-A[:,0].min())/w for e in ends]
233
- return 0.25<=yk<=0.75 and (max(xr)<0.6 if right else min(xr)>0.4)
234
- if FIX_DONOR: P['G>']=[q for q in P['G>'] if _chev_ok(q,True)] or P['G>']; P['G<']=[q for q in P['G<'] if _chev_ok(q,False)] or P['G<']
235
- for c in ('→','⇒','⇔','VEC'):
236
- A=[]
237
- for g in pool(c):
238
- q=[np.asarray(t,np.float32) for t in g['strokes']]; allp=np.concatenate(q); W=float(np.ptp(allp[:,0]))
239
- if W<1e-3 or np.ptp(allp[:,1])<1e-3: continue
240
- r=W/max((g['lbox'][2]-g['lbox'][0])*sc,1e-6)
241
- if W/np.ptp(allp[:,1])<(2.0 if c=="→" else 1.4) or not 0.6<=r<=1.6: continue # whole, straight arrows only (no lone heads / merged neighbours)
242
- A.append((q,r))
243
- P[c]=A
244
- _HM=(key,P); return P
245
- def _real_bar(P,Lt,rng):
246
- L=P['BAR_L']; ok=np.where((L>=0.75*Lt)&(L<=1.33*Lt))[0]
247
- if len(ok)==0: ok=np.argsort(np.abs(np.log(L/Lt)))[:20]
248
- q=P['BAR'][int(ok[rng.integers(len(ok))])][0]; k=Lt/max(float(np.ptp(q[:,0])),1e-6); ky=min(max(k,0.6),1.6)
249
- x=(q[:,0]-q[:,0].min())*k; y=(q[:,1]-q[:,1].mean())*ky; pr=q[:,3] if q.shape[1]>3 else np.full(len(q),0.5,np.float32)
250
- return np.stack([x,y,pr],1)
251
- VBAR=E('VBAR','1')=='1' # 05.11: | and the norm sign from real bars (VBAR=0: the old way, the glyph net / library)
252
- def vbar_mark(c,b,rng,xh,P):
253
- """| (one) or the norm / parallel sign (two) as REAL bars of the hand (fraction bars, _real_bar) stood upright, drawn top-down, the top
254
- leaning right by 0..6 deg; length 0.9..1.0 of the layout box height; the two bars of a norm 0.35..0.45 x-height apart"""
255
- x0,y0,x1,y1=b; bh=max(y1-y0,1e-6); cx=(x0+x1)/2; out=[]
256
- n=1 if c in ('|','∣') else 2; gap=rng.uniform(0.35,0.45)*xh; tl=np.tan(np.radians(rng.uniform(0,6)))
257
- for k in range(n):
258
- Lt=bh*rng.uniform(0.9,1.0); q=_real_bar(P,Lt,rng); dx=(k-(n-1)/2)*gap+rng.normal(0,0.02*xh)
259
- Y=q[:,0]+y0+(bh-Lt)/2+rng.normal(0,0.02*xh); X=cx+dx+q[:,1]+(Lt-q[:,0])*tl-Lt*tl/2
260
- out.append(np.stack([X,Y,q[:,2]],1))
261
- return out
262
- def hand_mark(c,b,rng,xh,writer,exclude_docs,sw=None):
263
- """LINE_MARKS=2: real-stroke mark over layout box b (page units, y down); 'LIB' = use a real library glyph; None = pen line"""
264
- P=_hm_pools(writer,exclude_docs); x0,y0,x1,y1=b; bw=x1-x0; bh=max(y1-y0,1e-6); j=0.03*xh
265
- if c=='BAR' and len(P['BAR'])>=20:
266
- ov=float(P['BAR_OV'][rng.integers(len(P['BAR_OV']))]); ov=min(max(ov,0.85),1.1) # real hand/layout length ratio, clipped so the bar still covers the fraction
267
- q=_real_bar(P,bw*ov,rng); cx=(x0+x1)/2+rng.normal(0,j); q[:,0]+=cx-q[:,0].max()/2; q[:,1]+=(y0+y1)/2+rng.normal(0,j)
268
- return [q.tolist()]
269
- if c=='√' and len(P['√'])>=5 and len(P['BAR'])>=20:
270
- hk=P['√'][int(rng.integers(len(P['√'])))]; k=bh/max(float(np.ptp(hk[:,1])),1e-6)*(1+rng.normal(0,0.04)); kx=k
271
- if sw: kx=min(max(sw*(1+rng.normal(0,0.06))/max(float(np.ptp(hk[:,0])),1e-6),0.35*k),1.2*k) # hook as wide as the layout's sign, not stretched with a tall radicand
272
- hx=(hk[:,0]-hk[:,0].min())*kx+x0; hy=(hk[:,1]-hk[:,1].min())*k+y0; hp=hk[:,3] if hk.shape[1]>3 else np.full(len(hk),0.5,np.float32)
273
- ex,ey=hx[-1],hy[-1]; Lt=max(x1-ex,0.2*xh)*(1+abs(rng.normal(0,0.05)))
274
- q=_real_bar(P,Lt,rng); bx=q[:,0]-q[0,0]+ex; by=q[:,1]-q[0,1]+ey
275
- return [np.stack([np.concatenate([hx,bx[1:]]),np.concatenate([hy,by[1:]]),np.concatenate([hp,q[1:,2]])],1).tolist()]
276
- if c in ('⇒','⇔') and len(P['G='])>=20 and len(P['G>'])>=20 and (c=='⇒' or len(P['G<'])>=20):
277
- def place(q,tb):
278
- allp=np.concatenate(q)[:,:2]; qx0,qy0=allp.min(0); qx1,qy1=allp.max(0); tw=tb[2]-tb[0]; th=tb[3]-tb[1]
279
- ky=th/max(qy1-qy0,1e-6)*(1+rng.normal(0,0.04)); kx=min(max(tw/max(qx1-qx0,1e-6),0.6*ky),1.8*ky); cx=(tb[0]+tb[2])/2+rng.normal(0,j); cy=(tb[1]+tb[3])/2+rng.normal(0,j)
280
- return [np.stack([(t[:,0]-(qx0+qx1)/2)*kx+cx,(t[:,1]-(qy0+qy1)/2)*ky+cy,t[:,3] if t.shape[1]>3 else np.full(len(t),0.5,np.float32)],1).tolist() for t in q]
281
- pick=lambda k: P[k][int(rng.integers(len(P[k])))]; ym=(y0+y1)/2
282
- if c=='⇒': return place(pick('G='),[x0,ym-0.3*bh,x0+0.68*bw,ym+0.3*bh])+place(pick('G>'),[x0+0.55*bw,y0,x1,y1])
283
- return place(pick('G<'),[x0,y0,x0+0.4*bw,y1])+place(pick('G='),[x0+0.3*bw,ym-0.3*bh,x0+0.7*bw,ym+0.3*bh])+place(pick('G>'),[x0+0.6*bw,y0,x1,y1])
284
- if c in ('→','VEC') and len(P[c])>=5:
285
- q,ovs=P[c][int(rng.integers(len(P[c])))]; allp=np.concatenate(q)[:,:2]; qx0,qy0=allp.min(0); qx1,qy1=allp.max(0); qh=max(qy1-qy0,1e-6)
286
- ky=bh/qh*(1+rng.normal(0,0.04)) if c!='VEC' else min(bh/qh,bw/max(qx1-qx0,1e-6)); ov=min(max(float([a[1] for a in P[c]][int(rng.integers(len(P[c])))]),0.85),1.15)
287
- hw=1.1*qh; W0=(qx1-qx0)*ky; Wt=bw*ov; heads=('R','L') if c=='⇔' else ('R',)
288
- S0=W0-hw*ky*len(heads); St=Wt-hw*ky*len(heads); f=St/S0 if S0>0.2*W0 and St>0.2*Wt else Wt/max(W0,1e-6)
289
- def mx(x):
290
- u=(x-qx0)*ky
291
- if f==Wt/max(W0,1e-6) and not (S0>0.2*W0 and St>0.2*Wt): return u*f
292
- a=hw*ky if 'L' in heads else 0.0; e=W0-hw*ky
293
- return np.where(u<a,u,np.where(u<=e,a+(u-a)*f,a+(e-a)*f+(u-e)))
294
- cx=(x0+x1)/2+rng.normal(0,j); cy=(y0+y1)/2+rng.normal(0,j); out=[]
295
- for t in q:
296
- X=mx(t[:,0]); Y=(t[:,1]-(qy0+qy1)/2)*ky+cy; pr=t[:,3] if t.shape[1]>3 else np.full(len(t),0.5,np.float32); out.append(np.stack([X,Y,pr],1))
297
- xs=np.concatenate([o[:,0] for o in out]); sh=cx-(xs.min()+xs.max())/2
298
- for o in out: o[:,0]+=sh
299
- return [o.tolist() for o in out]
300
- if c in ('↑','↓','←','⇐','↔','⇒','⇔'): return 'LIB'
301
- return None
302
- DONOR={'O':'0','κ':'k','Σ':'∑'} if (E('GEN') in ('vae2','dn') and E('DONORS','1')=='1') else {} # M3 (22.10): shape donors for classes the library has (almost) no glyphs of: O 1, κ 2, Σ 12
303
- if E('FW_LIB')=='none': DONOR=dict(DONOR,**{'≃':'≈','∙':'⋅','*':'∗','⋯':'…','⇓':'↓','⟸':'←','⇉':'→','⎧':'{','⎩':'{','⎛':'('}) # 06.11 no library: its rare classes the net has no row for
304
- if FIX_DONOR: DONOR=dict(DONOR,**{'∩':'u','Θ':'θ'}) # 22.10: w01's library has 15 odd ∩; drawn as her u turned upside down
305
- def cls_net(c): return DONOR.get(c,c)
306
- def vae_samples(chars,writer,rng,k=1):
307
- """chars repeated k times (variant-major, as for glyph1); variant j uses temperature GVAE_TAUS[j % len]"""
308
- if E('GEN')=='dn': import glyph_dn as GV # И116: the glyph denoiser, same API (tau/shrink ignored)
309
- else: import glyph_vae as GV
310
- n=len(chars)//max(k,1); taus=[float(t) for t in E('GVAE_TAUS','0.2,0.3,0.4,0.25,0.2,0.35,0.3,0.4').split(',')]; shrink=float(E('GVAE_SHRINK','0.75')); out=[] # I88 defaults (22.10); the 21.10 run used GVAE_SHRINK=0.5 GVAE_TAUS=0.1,0.2,0.3,0.15,0.1,0.25,0.2,0.3 SEL_PICK=best
311
- for j in range(max(k,1)):
312
- r=GV.sample_batch(chars[j*n:(j+1)*n],[writer]*n,rng,tau=taus[j%len(taus)],shrink=shrink)
313
- out+=[q if isinstance(q,dict) else dict(strokes=[]) for q in r]
314
- return out
315
- XF=E('XF','all') # 22.10 00:5x: w03 I90 — transforms on by default (XF= to switch off, XF=w03 for shear+parabolas only)
316
- def _xa(name,dflt):
317
- on={'w03':('SHX','SHY','PV','PH'),'all':None}.get(XF,())
318
- if on is not None and name not in on: return 0.0
319
- return float(E('XF_'+name,str(dflt)))
320
- def xf_glyph(pts,mx,my,L,rng):
321
- """I90 per-glyph deformation (in glyph units: centred on (mx,my), divided by L); returns new point arrays (same columns)"""
322
- shx=rng.normal(0,_xa('SHX',0.06)); shy=rng.normal(0,_xa('SHY',0.05)); a=rng.normal(0,_xa('PV',0.10)); bb=rng.normal(0,_xa('PH',0.08))
323
- asp=rng.normal(0,_xa('ASP',0.04)); eu=rng.normal(0,_xa('ASC',0.04)); ed=rng.normal(0,_xa('ASC',0.04)); px,py=rng.normal(0,_xa('PERSP',0.02),2)
324
- trem=_xa('TREM',0.0); spd=min(_xa('SPEED',0.5),0.9); prs=math.exp(rng.normal(0,_xa('PRES',0.27)))
325
- allp=np.concatenate([(p[:,:2]-[mx,my])/L for p in pts]); mx2=float(np.mean(allp[:,0]**2)); my2=float(np.mean(allp[:,1]**2)); out=[]
326
- for p in pts:
327
- q=p.copy(); x=(p[:,0]-mx)/L; y=(p[:,1]-my)/L
328
- if spd>0 and len(p)>3: # uneven pen speed: re-place the points along the same path (shape unchanged, density not)
329
- d=np.r_[0,np.cumsum(np.linalg.norm(np.diff(np.stack([x,y],1),axis=0),axis=1))]
330
- if d[-1]>0:
331
- u=np.linspace(0,1,len(p)); ph=rng.uniform(0,6.283); w=u+spd*np.sin(2*np.pi*u+ph)/(2*np.pi)-spd*np.sin(ph)/(2*np.pi)*(1-u)-spd*np.sin(2*np.pi+ph)/(2*np.pi)*u
332
- w=np.clip(np.maximum.accumulate(w),0,1)*d[-1]; x=np.interp(w,d,x); y=np.interp(w,d,y)
333
- if q.shape[1]>3: q[:,3]=np.interp(w,d,p[:,3])
334
- x=x+shx*y; y=y+shy*x; y=y+a*(x**2-mx2); x=x+bb*(y**2-my2); x=x*math.exp(asp/2); y=y*math.exp(-asp/2)
335
- y=np.where(y<-0.25,-0.25+(y+0.25)*(1+eu),np.where(y>0.25,0.25+(y-0.25)*(1+ed),y))
336
- den=1+px*x+py*y; x=x/den; y=y/den
337
- if trem>0 and len(p)>3:
338
- k=np.exp(-0.5*(np.arange(-6,7)/3.0)**2); k/=k.sum(); n=rng.normal(0,1,(len(p)+12,2)); n=np.stack([np.convolve(n[:,i],k,'valid') for i in (0,1)],1)[:len(p)]
339
- n/=max(n.std(),1e-6); x=x+trem*n[:,0]; y=y+trem*n[:,1]
340
- q[:,0]=x*L+mx; q[:,1]=y*L+my
341
- if q.shape[1]>3: q[:,3]=np.clip(q[:,3]*prs,0,1)
342
- out.append(q)
343
- return out
344
- _BASEH={}
345
- def base_heights(codes):
346
- """layout box height of each char at base level (cache) -> script detection"""
347
- new=[c for c in set(codes) if c not in _BASEH and c not in ('BAR','TEXT')]
348
- if new:
349
- for c,o in zip(new,layout([chr(int(c,16)) for c in new])):
350
- _BASEH[c]=(o['glyphs'][0]['bbox'][3]-o['glyphs'][0]['bbox'][1]) if o and o.get('ok') and o['glyphs'] else None
351
- return _BASEH
352
- def script_factor(g):
353
- if not FIX_SCRIPT or g['c'] in ('BAR','TEXT'): return 1.0
354
- bh0=_BASEH.get(g['c'])
355
- if not bh0 or (g['bbox'][3]-g['bbox'][1])>=0.85*bh0: return 1.0
356
- ch=unicodedata.normalize('NFKC',chr(int(g['c'],16)))
357
- if ch.isdigit(): return float(E('SCRIPT_DIGIT','0.71'))
358
- if ch.isascii() and ch.isalpha(): return 1.0
359
- return float(E('SCRIPT_OTHER','0.92'))
360
- _TW={}
361
- def text_words(words):
362
- need=[w for w in dict.fromkeys(words) if w not in _TW]
363
- if need:
364
- r=subprocess.run([sys.executable,os.path.join(PROJ,'scripts','text_words.py')],input=json.dumps(need,ensure_ascii=False),capture_output=True,text=True,env=dict(os.environ,PROJ=PROJ))
365
- try: _TW.update(json.loads(r.stdout.strip().splitlines()[-1]))
366
- except Exception: print('text_words failed',r.stderr[-300:],flush=True)
367
- return _TW
368
- def compose_text(texts,s,rng):
369
- """TEXT boxes (one per char) -> words -> line-model ink fitted to each word box (width; x-height from the box)"""
370
- words=[]; cur=[]
371
- for g in sorted(texts,key=lambda g:g['bbox'][0]):
372
- if not g.get('ch','').strip():
373
- if cur: words.append(cur); cur=[]
374
- continue
375
- if cur and g['bbox'][0]-cur[-1]['bbox'][2]>50: words.append(cur); cur=[]
376
- cur.append(g)
377
- if cur: words.append(cur)
378
- W=text_words([''.join(g['ch'] for g in w) for w in words]); out=[]
379
- for w in words:
380
- var=W.get(''.join(g['ch'] for g in w)) or []
381
- if not var: continue
382
- def asp(v):
383
- B=np.array([p[:2] for t in v for p in t],np.float32); return np.ptp(B[:,0])/max(np.percentile(B[:,1],75)-np.percentile(B[:,1],25),1e-6)
384
- a=np.array([asp(v) for v in var]); order=np.argsort(np.abs(np.log(a/np.median(a)))) # a typical variant: some samples are garbage (cut failed, scribbles)
385
- st=var[int(order[int(rng.integers(min(2,len(order))))])]; A=np.array([p[:2] for t in st for p in t],np.float32)
386
- bx0=min(g['bbox'][0] for g in w)*s; bx1=max(g['bbox'][2] for g in w)*s; yb=w[0]['bbox'][3]*s; xhb=(w[0]['bbox'][3]-w[0]['bbox'][1])*s
387
- core=max(np.percentile(A[:,1],75)-np.percentile(A[:,1],25),1e-6); k=min((bx1-bx0)/max(np.ptp(A[:,0]),1e-6),1.3*xhb/core)
388
- ax=(A[:,0].min()+A[:,0].max())/2; ay=np.percentile(A[:,1],75)
389
- for t in st: out.append([[float((p[0]-ax)*k+(bx0+bx1)/2),float((p[1]-ay)*k+yb-0.1*xhb),float(p[2] if len(p)>2 else 0.5)] for p in t])
390
- return out
391
- def arrow_build(c,b,rng,P):
392
- """FIX_DONOR arrows: real bar shaft + real '>' head; c in -> <- down up VEC; b = box (page units)"""
393
- x0,y0,x1,y1=b; bw=x1-x0; bh=max(y1-y0,1e-6); vert=c in ('↓','↑'); L=(bh if vert else bw)*min(max(1+rng.normal(0,0.05),0.9),1.1); T=bw if vert else bh
394
- hh=min(max(0.75*T,0.12*L),0.45*L) if c!='VEC' else min(0.9*T,0.35*L)
395
- q=_real_bar(P,L,rng); sx=q[:,0]-q[:,0].max(); sy=q[:,1]-q[:,1].mean()
396
- g=P['G>'][int(rng.integers(len(P['G>'])))]; A=np.concatenate(g)[:,:2]; ax1=A[:,0].max(); ay=(A[:,1].min()+A[:,1].max())/2; kh=hh/max(np.ptp(A[:,1]),1e-6)
397
- parts=[np.stack([sx,sy,q[:,2]],1)]+[np.stack([(t[:,0]-ax1)*kh,(t[:,1]-ay)*kh,t[:,3] if t.shape[1]>3 else np.full(len(t),0.5,np.float32)],1) for t in g]
398
- cx=(x0+x1)/2; cy=(y0+y1)/2; out=[]
399
- for t in parts:
400
- X,Y=t[:,0]+L/2,t[:,1]
401
- if c=='←': X=-X
402
- if c=='↓': X,Y=-Y,X
403
- if c=='↑': X,Y=Y,-X
404
- out.append(np.stack([X+cx,Y+cy,t[:,2]],1).tolist())
405
- return out
406
- def _push(gs,placed,xh):
407
- """FIX40 no-collision: shift strokes gs (list of arrays [x,y,...]) right past any earlier glyph box on the same line it overlaps"""
408
- if not gs or not placed: return
409
- A=np.concatenate([np.asarray(q)[:,:2] for q in gs]); bx0,by0,bx1,by1=A[:,0].min(),A[:,1].min(),A[:,0].max(),A[:,1].max(); sh=0.0
410
- for px0,py0,px1,py1 in placed:
411
- ov=min(by1,py1)-max(by0,py0)
412
- mh=min(by1-by0,py1-py0)
413
- same_line=abs((py0+py1)/2-(by0+by1)/2)<0.3*max(by1-by0,py1-py0) # a subscript next to its base is not a collision
414
- if (py1-py0)>1.8*(by1-by0) and py0<=by0 and py1>=by1: continue # an encloser (radical, tall bracket) is not a neighbour
415
- if same_line and mh>0.12*xh and ov>0.5*mh and (px0+px1)/2<(bx0+bx1)/2 and px1+0.35*xh>bx0: sh=max(sh,px1+0.35*xh-bx0) # thin things (bars) never push
416
- if 0<sh<2.0*xh:
417
- for q in gs: q[:,0]+=sh
418
- SEL_STATS['pushed']+=1
419
- def compose_layout(o,writer,rng,exclude_docs=()):
420
- lib,scale=library(); s=scale.get(writer,scale.get('w01',0.1)); xh=450*s # ~x-height in page units
421
- strokes=[]; missing=[]; gl=merge_radicals(o['glyphs']); texts=[g for g in gl if g['c']=='TEXT']; gl=[g for g in gl if g['c']!='TEXT']
422
- if FIX_DONOR: # << = two '<'
423
- g2=[]
424
- for g in gl:
425
- if g['c']=='226A': b=g['bbox']; m=(b[0]+b[2])/2; w=b[2]-b[0]; g2+=[dict(g,c='3C',bbox=[b[0],b[1],m+0.08*w,b[3]]),dict(g,c='3C',bbox=[m-0.08*w,b[1],b[2],b[3]])]
426
- else: g2.append(g)
427
- gl=g2
428
- if FIX_SCRIPT: base_heights([g['c'] for g in gl])
429
- if texts: strokes+=compose_text(texts,s,rng)
430
- if not gl: return strokes,missing
431
- x_min=min(g['bbox'][0] for g in gl)
432
- gen=None; n=len(gl); fy0=min(g['bbox'][1] for g in gl); fH=max(max(g['bbox'][3] for g in gl)-fy0,1e-6)
433
- if GEN=='VAE': gen=vae_samples([cls_net(cls_of(g['c'])) for g in gl]*SEL_K,writer,rng,SEL_K)
434
- elif GEN is not None: gen=gen_samples([cls_of(g['c']) for g in gl]*SEL_K,writer,rng,exclude_docs)
435
- xr=np.random.default_rng(int(rng.integers(1<<31))) if XF else None; gap=0.0; base=0.0 # XF draws from its own stream: XF off = old results bit for bit
436
- placed=[]; last_n=0 # text words (added above) count as placed too
437
- def _bb(sts):
438
- A=np.array([p[:2] for t in sts for p in t],np.float32); return (float(A[:,0].min()),float(A[:,1].min()),float(A[:,0].max()),float(A[:,1].max()))
439
- for gi,g in enumerate(gl):
440
- if FIX40 and len(strokes)>last_n: placed.append(_bb(strokes[last_n:])); last_n=len(strokes)
441
- rot180=cls_of(g['c'])=='∩' and FIX_DONOR
442
- c=cls_of(g['c'])
443
- if c in DONOR: SEL_STATS['donor_'+c]+=1; c=DONOR[c] # drawn as the donor shape (O <- 0, kappa <- k, Sigma <- sum), sized by the layout box
444
- force_lib=False
445
- if FIX_DONOR and c in ('⋅','·','…','...','⋯','.',':'): # centred dot / ellipsis / period / colon: pen dots (the net draws a ring)
446
- b=[v*s for v in g['bbox']]; r=0.05*xh
447
- if c==':': pts_d=[((b[0]+b[2])/2,b[1]+0.15*(b[3]-b[1])),((b[0]+b[2])/2,b[3]-0.15*(b[3]-b[1]))]
448
- else:
449
- nd=1 if c in ('⋅','·','.') else 3; cyd=(b[1]+b[3])/2 if c!='.' else b[3]-r
450
- pts_d=[(b[0]+(k+0.5)*(b[2]-b[0])/nd,cyd) for k in range(nd)]
451
- for cxd,cyd in pts_d: strokes.append([[float(cxd-r),float(cyd),0.5],[float(cxd),float(cyd+r*0.6),0.5],[float(cxd+r),float(cyd),0.5]])
452
- SEL_STATS['dot_mark']+=1; continue
453
- if FIX_DONOR and c in (' \u0307','\u02d9','\u0307'): # dot accent: a small pen dot (the net draws it as a ring)
454
- b=[v*s for v in g['bbox']]; cxd=(b[0]+b[2])/2; cyd=(b[1]+b[3])/2; r=0.05*xh
455
- strokes.append([[float(cxd-r),float(cyd),0.5],[float(cxd),float(cyd+r*0.6),0.5],[float(cxd+r),float(cyd),0.5]]); SEL_STATS['dot_mark']+=1; continue
456
- if FIX_DONOR and (c in ('[',']') or c in FIX_LIBCLS): force_lib_d=True # real library glyphs for classes the VAE draws badly (by eye, 22.10)
457
- else: force_lib_d=False
458
- if FIX40 and c=='BAR' and g.get('node')=='menclose' and LINE_MARKS==2: # \boxed{}: a frame of four real bars (w01 boxes results)
459
- P_=_hm_pools(writer,exclude_docs); b=[v*s for v in g['bbox']]; bw_=b[2]-b[0]; bh_=b[3]-b[1]; box=[]
460
- for (xa,ya,L_,vert) in ((b[0],b[1],bw_,False),(b[0],b[3],bw_,False),(b[0],b[1],bh_,True),(b[2],b[1],bh_,True)):
461
- q_=_real_bar(P_,L_*(1+abs(rng.normal(0,0.03))),rng)
462
- X_,Y_=(q_[:,0]+xa-0.02*L_,q_[:,1]+ya) if not vert else (q_[:,1]+xa,q_[:,0]+ya-0.02*L_)
463
- box.append(np.stack([X_,Y_,q_[:,2]],1).tolist())
464
- strokes+=box; SEL_STATS['boxed']+=1; continue
465
- if LINE_MARKS==2 and FIX_DONOR and c in ('→','←','↓','↑','VEC') and len(_hm_pools(writer,exclude_docs).get('G>',[]))>=20:
466
- am=[np.asarray(t,np.float64) for t in arrow_build(c,[v*s for v in g['bbox']],rng,_hm_pools(writer,exclude_docs))]; _push(am,placed,xh); strokes+=[t.tolist() for t in am]; SEL_STATS['arrow_built']+=1; continue
467
- if LINE_MARKS==2 and VBAR and c in ('|','∥','‖','∣') and len(_hm_pools(writer,exclude_docs)['BAR'])>=20: # 05.11: |x| and the norm ||x|| — the glyph nets draw | as / ( ) ? 7
468
- vb=[np.asarray(t,np.float64) for t in vbar_mark(c,[v*s for v in g['bbox']],rng,xh,_hm_pools(writer,exclude_docs))]; _push(vb,placed,xh); strokes+=[t.tolist() for t in vb]; SEL_STATS['vbar']+=1; continue
469
- if LINE_MARKS==1 and VBAR and c in ('|','∥','‖','∣'): # 06.11 no library: |x| and the norm as pen lines (the glyph nets draw | as / ( ) ? 7)
470
- b_=[v*s for v in g['bbox']]; xm_=(b_[0]+b_[2])/2; dx_=[0.0] if c in ('|','∣') else [-0.22*(b_[2]-b_[0]),0.22*(b_[2]-b_[0])]
471
- vb=[np.asarray(_poly([[xm_+d_,b_[1]],[xm_+d_,b_[3]]],rng,0.015*xh,10),np.float64) for d_ in dx_]; _push(vb,placed,xh); strokes+=[t.tolist() for t in vb]; SEL_STATS['vbar_pen']+=1; continue
472
- if LINE_MARKS==2 and c in LINE_SET:
473
- hm=hand_mark(c,[v*s for v in g['bbox']],rng,xh,writer,exclude_docs,sw=g['sw']*s if 'sw' in g else None)
474
- if hm=='LIB': force_lib=True; SEL_STATS['hand_mark_lib']+=1
475
- elif hm is not None:
476
- if FIX40 and c not in ('BAR','√'): hm=[np.asarray(t,np.float64) for t in hm]; _push(hm,placed,xh); hm=[t.tolist() for t in hm] # a radical encloses its radicand: never pushed
477
- strokes+=hm; SEL_STATS['hand_mark']+=1; continue
478
- if LINE_MARKS and c in LINE_SET and not force_lib:
479
- strokes+=line_mark(c,[v*s for v in g['bbox']],rng,xh); SEL_STATS['line_mark']+=1; continue
480
- q=None
481
- if gen is not None and not force_lib and not force_lib_d:
482
- if SEL_K>1:
483
- k=pick_sample([gen[k*n+gi] for k in range(SEL_K)],c,(g['bbox'][3]-g['bbox'][1])/fH,rng)
484
- q=gen[k*n+gi] if k is not None else None
485
- if q is not None and not q['strokes']: q=None # empty variant (class unknown to the generator) -> library
486
- else:
487
- q=gen[gi]
488
- if not q['strokes'] or sum(len(t) for t in q['strokes'])<2: missing.append(c); continue
489
- if q is None:
490
- pool=[q for q in lib.get((writer,c),[]) if q['doc'] not in exclude_docs] or [q for q in lib.get(('*',c),[]) if q['doc'] not in exclude_docs]
491
- if not pool and gen is not None and SEL_K>1 and gen[gi]['strokes'] and int(E('SEL_RETRY','0')): # 06.11 no library (fgen_nolib_k4: -10 p.p., the first rejected variant was used): SEL_RETRY more variants, the first recognised one, else the best
492
- ex_=vae_samples([cls_net(cls_of(g['c']))]*int(E('SEL_RETRY')),writer,rng,1); rh_=(g['bbox'][3]-g['bbox'][1])/fH; kk=pick_sample(ex_,c,rh_,rng)
493
- if kk is not None: q=ex_[kk]; SEL_STATS['retry_ok']+=1
494
- else: al_=ex_+[gen[k_*n+gi] for k_ in range(SEL_K)]; kk=pick_sample(al_,c,rh_,rng,force=True); q=al_[kk if kk is not None else 0]; SEL_STATS['retry_forced']+=1
495
- elif not pool and gen is not None and SEL_K>1 and gen[gi]['strokes']: q=gen[gi]; SEL_STATS['net_unpicked']+=1 # 04.11: the picker rejected every variant and there is no library glyph -> the net's variant, not a dropped glyph
496
- elif not pool: missing.append(c); continue
497
- if gen is not None and SEL_K>1: pool=[q for q in pool if nstroke_p(c,len(q['strokes']))>=SEL_NMIN] or pool
498
- if FIX40 and c in ('0','O','o'): pool=[q for q in pool if len(q['strokes'])==1] or pool # no slashed zeros (Ø) from the library
499
- if FIX40 and c=='=': # a real '=' with two near-parallel bars (some library '=' converge and read as '>')
500
- def _par(q):
501
- if len(q['strokes'])!=2: return False
502
- a=[math.degrees(math.atan2(t[-1][1]-t[0][1],abs(t[-1][0]-t[0][0])+1e-6)) for t in q['strokes']]
503
- L=[abs(t[-1][0]-t[0][0]) for t in q['strokes']]
504
- return abs(a[0]-a[1])<6 and min(L)>0.6*max(L)
505
- pool=[q for q in pool if _par(q)] or pool # selection fallback: library glyph with a typical stroke count (no half-x)
506
- if FIX40 and len(pool)>1: # the classifier picks the most typical of up to 16 real glyphs (the library still holds odd ones: lambda like d)
507
- cands=[pool[int(i)] for i in rng.choice(len(pool),min(16,len(pool)),replace=False)]; kk=pick_sample(cands,c,(g['bbox'][3]-g['bbox'][1])/fH,rng)
508
- q=cands[kk if kk is not None else 0]; SEL_STATS['lib_picked']+=1
509
- elif pool: q=pool[int(rng.integers(len(pool)))] # (empty pool: q is the net's variant, set above)
510
- b=[v*s for v in g['bbox']]; bw=b[2]-b[0]; bh=b[3]-b[1]
511
- pts=[np.asarray(st,np.float32) for st in q['strokes']]; allp=np.concatenate([p[:,:2] for p in pts]); x0,y0=allp.min(0); x1,y1=allp.max(0); qw=max(x1-x0,1e-3); qh=max(y1-y0,1e-3)
512
- if c=='BAR' or bw>2.5*bh: k=bw/qw; kx=k; ky=min(k,bh/qh*1.5) if c!='BAR' else k
513
- elif bh>2.2*bw and c in '()[]{}|∫∮∑∏': ky=bh/qh; kx=min(ky,1.6*bw/qw)
514
- else: ky=bh/qh; kx=ky
515
- f=1.0+rng.normal(0,JIT_S); kx*=f; ky*=f; th=math.radians(rng.normal(0,JIT_R)); ct,sn=math.cos(th),math.sin(th)
516
- drift=DRIFT*xh*math.sin((b[0]-x_min*s)/(6*xh)+rng.uniform(0,6.28))
517
- cx=(b[0]+b[2])/2+rng.normal(0,JIT_XY*xh); cy=(b[1]+b[3])/2+rng.normal(0,JIT_XY*xh)+drift
518
- if FIX40 and c not in ('BAR','(',')','[',']','{','}','|') and qw*kx>1.3*bw and bw>0: f2=1.3*bw/(qw*kx); kx*=f2; ky*=f2; SEL_STATS['narrowed']+=1 # a glyph much wider than its layout box crowds its neighbours
519
- if FIX_DONOR and cls_of(g['c'])=='Θ': kx*=1.35 # capital Theta from her lowercase theta: wider, as a capital
520
- fs=script_factor(g)
521
- if fs!=1.0: kx*=fs; ky*=fs; cx=b[0]+(cx-b[0])*fs; SEL_STATS['script_scaled']+=1 # keep the left edge, shrink to her script size
522
- mx=(x0+x1)/2; my=(y0+y1)/2
523
- if XF:
524
- if not (FIX40 and (c in XF_KEEP or c=='BAR')): pts=xf_glyph(pts,mx,my,max(qw,qh),xr)
525
- else: xf_glyph(pts,mx,my,max(qw,qh),xr) # draw the same random numbers (stream stays aligned), keep the shape
526
- gh=bh if c!='BAR' else xh
527
- gap=0.7*gap+math.sqrt(1-0.49)*xr.normal(0,_xa('GAP',0.22)); base=0.8*base+0.6*xr.normal(0,_xa('BASE',0.08))
528
- upright=FIX40 and int(g['c'],16)<128 and chr(int(g['c'],16)).isalpha() # letters of cos/sin/det/res: no gap jitter inside the word
529
- cx+=(0 if upright else gap)*min(gh,1.5*xh); cy+=base*min(gh,1.5*xh)
530
- if rot180: pts=[np.concatenate([2*mx-p[:,:1],2*my-p[:,1:2],p[:,2:]],1) for p in pts]
531
- gs=[]
532
- for p in pts:
533
- u=(p[:,0]-mx)*kx; v=(p[:,1]-my)*ky; X=cx+u*ct-v*sn; Y=cy+u*sn+v*ct
534
- pr=p[:,3] if p.shape[1]>3 else np.full(len(p),0.5,np.float32)
535
- gs.append(np.stack([X,Y,pr],1))
536
- if FIX40: _push(gs,placed,xh)
537
- strokes+=[q.tolist() for q in gs]
538
- return strokes,missing
539
- def compose(tex,writer='w01',rng=None,exclude_docs=()):
540
- rng=rng or np.random.default_rng(); o=layout([tex])[0]
541
- if not o or not o.get('ok'): return None,['LAYOUT']
542
- return compose_layout(o,writer,rng,exclude_docs)
543
- def render(strokes,H=96,pen=2.4):
544
- if not strokes or not any(len(s) for s in strokes): return Image.new('L',(32,H),255)
545
- pts=np.array([p[:2] for s in strokes for p in s]); x0,y0=pts.min(0); x1,y1=pts.max(0); sc=(H-16)/max(y1-y0,1e-6)
546
- W=int((x1-x0)*sc)+16; im=Image.new('L',(max(W,32),H),255); d=ImageDraw.Draw(im)
547
- for s in strokes:
548
- q=[((p[0]-x0)*sc+8,(p[1]-y0)*sc+8) for p in s]
549
- if len(q)>1: d.line(q,fill=0,width=max(1,int(pen)),joint='curve')
550
- elif q: d.point(q[0],fill=0)
551
- return im
552
- _NSYN=[('\\dfrac','\\frac'),('\\tfrac','\\frac'),('\\to','\\rightarrow'),('\\neq','\\ne'),('\\leq','\\le'),('\\geq','\\ge'),('\\varepsilon','\\epsilon'),
553
- ('\\varphi','\\phi'),('\\vartheta','\\theta'),('\\ldots','\\dots'),('\\cdots','\\dots'),('\\varrho','\\rho'),('\\langle','<'),('\\rangle','>'),
554
- ('\\overrightarrow','\\vec'),('\\neq','\\ne')] # 22.10: handwritten <T> and \\langle T\\rangle look the same; so do \\vec and \\overrightarrow
555
- def norm_tex2(t):
556
- """22.10 (40/40 task): EQUIVALENT notations only -- spacing, \\left/\\right, font wrappers incl. \\mathcal (handwriting has no font),
557
- synonyms (\\to = \\rightarrow, \\neq = \\ne, \\varphi = \\phi ...), prime forms (' = ^{\\prime}), cases = \\{ array{l} \\.,
558
- pmatrix = ( array ), array column specs; then braces and whitespace dropped. FW_NORM=1 -> the old norm_tex."""
559
- t=re.sub(r'\\(left|right)(?![A-Za-z])','',t.strip().strip('$')).replace('\\displaystyle','').replace('\\limits','') # not the \\right of \\rightarrow
560
- t=re.sub(r'(?<!\\)\\(qquad|quad|[,;!: ])','',t).replace('~','') # spacing commands (not the row break \\\\)
561
- for a,b in _NSYN: t=re.sub(re.escape(a)+r'(?![A-Za-z])',b.replace('\\','\\\\'),t)
562
- t=re.sub(r'\s+','',t) # synonyms first: after the spaces go, '\\to x' becomes '\\tox' and no longer matches
563
- for _ in range(3): t=re.sub(r'\\(mathrm|text|mathbf|mathit|operatorname|mathcal|textrm|textit)\{([^{}]*)\}',r'\2',t)
564
- t=re.sub(r'\\(mathcal|mathrm|mathit)([A-Za-z])',r'\2',t)
565
- t=re.sub(r"\^\{?\\prime\}?","'",t).replace('\\prime',"'")
566
- t=re.sub(r'[{}]','',t)
567
- t=re.sub(r'\\beginarray[lcr|]+',r'\\beginarray',t)
568
- for a,b in (('\\\\beginarray','\\begincases'),('\\endarray.','\\endcases'),('(\\beginarray','\\beginpmatrix'),('\\endarray)','\\endpmatrix'),('[\\beginarray','\\beginbmatrix'),('\\endarray]','\\endbmatrix')): t=t.replace(a,b)
569
- return t
570
- def norm_tex(t):
571
- if E('FW_NORM','2')!='1': return norm_tex2(t)
572
- t=t.strip().strip('$').replace('\\left','').replace('\\right','').replace('\\displaystyle','').replace('\\limits','')
573
- for a in ('\\,','\\;','\\!','\\:','\\quad','\\qquad','~'): t=t.replace(a,'')
574
- t=re.sub(r'\\(mathrm|text|mathbf|mathit|operatorname)\{([^{}]*)\}',r'\2',t); t=re.sub(r'[{}\s]','',t); return t
575
- def cer(a,b):
576
- a=norm_tex(a); b=norm_tex(b)
577
- if not a: return 0.0 if not b else 1.0
578
- d=list(range(len(b)+1))
579
- for i in range(1,len(a)+1):
580
- p=d[:]; d[0]=i
581
- for j in range(1,len(b)+1): d[j]=min(p[j]+1,d[j-1]+1,p[j-1]+(a[i-1]!=b[j-1]))
582
- return d[-1]/len(a)
583
- if __name__=='__main__':
584
- import io, base64, urllib.request, time
585
- from concurrent.futures import ThreadPoolExecutor
586
- WRITER=E('WRITER','w01'); N=int(E('N','40')); SEED=int(E('SEED','5')); TAG=E('TAG','lib'); KEY=E('OPENROUTER_KEY'); MODEL=E('MODEL','qwen/qwen3-vl-235b-a22b-instruct')
587
- print('cfg',dict(XF=E('XF'),GVAE_SHRINK=E('GVAE_SHRINK'),GVAE_TAUS=E('GVAE_TAUS'),SEL_PICK=E('SEL_PICK'),SEL_PMIN=E('SEL_PMIN'),GEN=E('GEN','glyph1'),TAG=E('TAG'),N=E('N'),SEED=E('SEED'),GEN_CKPT=E('GEN_CKPT'),SEL_K=SEL_K,SEL_NMIN=SEL_NMIN,LINE_MARKS=int(LINE_MARKS),SEL_CLS=E('SEL_CLS','ckpt_dl/glyph_cls1.pt')),flush=True)
588
- valid_docs={json.loads(l)['doc'] for l in open(os.path.join(PROJ,'data','ink_dataset_v2','valid.jsonl'))}
589
- src={'w01':'data/sdocx_in/w01/lines_v3/all_full.jsonl'}.get(WRITER)
590
- cand=[]
591
- for l in open(os.path.join(PROJ,src)):
592
- r=json.loads(l); lab=r['label']
593
- if r.get('ok') and r['doc'] in valid_docs and lab.startswith('$') and lab.endswith('$') and lab.count('$')==2 and 6<=len(lab)<=70 and r.get('rows',1)<=1.4: cand.append(r)
594
- rng=np.random.default_rng(SEED); pick=[cand[i] for i in rng.choice(len(cand),min(N,len(cand)),replace=False)]
595
- if E('GEN') in ('vae','vae2','dn'): GEN='VAE'; print('glyphs from the glyph VAE',E('GEN'),E('GDN_CKPT') if E('GEN')=='dn' else E('GVAE_CKPT','ckpt_glyph_vae/last.pt'),'picker',E('SEL_CLS','ckpt_dl/glyph_cls1.pt'),'library',E('FW_LIB','glyphs_clean.jsonl'),flush=True)
596
- elif E('GEN_CKPT'): load_gen(E('GEN_CKPT')); print('glyphs from the generator',E('GEN_CKPT'),flush=True)
597
- lays=layout([r['label'][1:-1] for r in pick]); gen=[]; miss=collections.Counter()
598
- for r,o in zip(pick,lays):
599
- if not o or not o.get('ok'): gen.append(None); continue
600
- st,m=compose_layout(o,WRITER,rng,exclude_docs=valid_docs); miss.update(m); gen.append(st)
601
- print('composed',sum(g is not None for g in gen),'/',len(pick),'missing classes',dict(miss.most_common(10)),flush=True)
602
- print('SEL_K',SEL_K,'LINE_MARKS',int(LINE_MARKS),'glyph sourcing',dict(SEL_STATS),flush=True)
603
- P=("This is a handwritten mathematical formula. Transcribe it to LaTeX exactly as written (no $ signs, no explanations). "
604
- "Output only the LaTeX.")
605
- def ask(im):
606
- buf=io.BytesIO(); im.convert('RGB').save(buf,'PNG'); d64=base64.b64encode(buf.getvalue()).decode()
607
- body=json.dumps({"model":MODEL,"temperature":0,"max_tokens":300,"messages":[{"role":"user","content":[{"type":"text","text":P},{"type":"image_url","image_url":{"url":"data:image/png;base64,"+d64}}]}]}).encode()
608
- req=urllib.request.Request("https://openrouter.ai/api/v1/chat/completions",data=body,headers={"Authorization":"Bearer "+KEY,"Content-Type":"application/json"})
609
- for a in range(3):
610
- try: return json.load(urllib.request.urlopen(req,timeout=120))["choices"][0]["message"]["content"].strip().strip('`').replace('latex\n','')
611
- except Exception: time.sleep(3)
612
- return ''
613
- real_ims=[render(r['strokes']) for r in pick]; gen_ims=[render(g) if g else Image.new('L',(32,32),255) for g in gen]
614
- with ThreadPoolExecutor(8) as ex: rr=list(ex.map(ask,real_ims)); gr=list(ex.map(ask,gen_ims))
615
- labs=[r['label'][1:-1] for r in pick]; cr=[cer(l,x) for l,x in zip(labs,rr)]; cg=[cer(l,x) if g else 1.0 for l,x,g in zip(labs,gr,gen)]
616
- for name,c in (('REAL (control)',cr),('NET composed' if (E('GEN_CKPT') or E('GEN') in ('vae','vae2','dn')) else 'LIBRARY composed',cg)):
617
- print(f'=== {name}: <=0.2 {sum(x<=0.2 for x in c)}/{len(c)} <=0.05 {sum(x<=0.05 for x in c)}/{len(c)} mean {np.mean(c):.2f} ===',flush=True)
618
- rows=[]
619
- for i in range(min(16,len(pick))):
620
- a=real_ims[i]; b=gen_ims[i]; w=max(a.size[0],b.size[0]); t=Image.new('L',(w*2+30,max(a.size[1],b.size[1])),255); t.paste(a,(0,0)); t.paste(b,(w+30,0)); rows.append(t)
621
- W=min(1800,max(t.size[0] for t in rows)); sh=Image.new('L',(W,sum(t.size[1]+10 for t in rows)),255); y=0
622
- for t in rows:
623
- if t.size[0]>W: t=t.resize((W,int(t.size[1]*W/t.size[0])))
624
- sh.paste(t,(0,y)); y+=t.size[1]+10
625
- sh.save(os.path.join(PROJ,'figures',f'formula_write_{TAG}.png'))
626
- json.dump(dict(labels=labs,real_reads=rr,gen_reads=gr,cer_real=cr,cer_gen=cg),open(os.path.join(PROJ,'figures',f'formula_write_{TAG}.json'),'w'),ensure_ascii=False)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/glyph_dn.py DELETED
@@ -1,60 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """Glyph denoiser sampler (И116, 22.10): draws ONE formula glyph of a class in a writer's style with the diffusion denoiser of
3
- scripts/train_glyph_dn.py. Same API and output format as scripts/glyph_vae.py (formula_write GEN=vae2 / glyph_m3_eval GEN=vae):
4
- sample(cls, writer, rng, n=1, tau=None, shrink=None) -> list of n dicts {'strokes': [[[x,y,0,p],...], ...]} ([] if the class is
5
- unknown to the net -> the caller falls back to the library)
6
- sample_batch(classes, writers, rng, tau=None, shrink=None) -> one per (class, writer) pair.
7
- _load() -> dict with c2i / cmean (class membership) as glyph_vae._load.
8
- tau / shrink are accepted and ignored (the VAE's MDN temperature / z shrink have no counterpart; variants differ by the noise
9
- seed, drawn from rng). Points: x, y in norm_h units centred on the glyph box, y down, pressure p in 0..1.
10
- Env: PROJ, GDN_CKPT (ckpt_dl/glyphdn/glyphdnB/fp16/step050000.pt), GDN_CFG (1.5; formula_write GEN=dn sets 1.2: classifier-free guidance over the class), GDN_STEPS (10 DDIM
11
- steps), GDN_DEV, GDN_LEN (dist = draw the point count from the (writer, class) train distribution; or an int multiplier x100 of the
12
- median, e.g. 100), GDN_WRITER_NULL (1: an unknown writer -> the null «average» writer; 0 -> w01), GDN_NOISE (1.0: scale of the starting noise, >1 = more diversity), GDN_ETA (0: DDIM noise, 1 = fully stochastic steps -> more within-class diversity), GDN_PEN_THR (0: pen-up logit threshold; the net was trained with pos_weight 3, so 0 = calibrated p > 0.25, 1.1 = p > 0.5).
13
- 04.11 (И164): a GSTYLE=semb checkpoint takes the style from set_style(key, vec) (vec = semb3_all embedding of the reference line(s));
14
- the `writers` given to sample_batch are those keys (unknown key -> the null «average» writer)."""
15
- import os, sys, numpy as np, torch
16
- PROJ=os.environ.get('PROJ',os.path.dirname(os.path.dirname(os.path.abspath(__file__)))); E=os.environ.get
17
- sys.path.insert(0,os.path.join(PROJ,'scripts'))
18
- _S=None; STYLES={}; import collections; N=collections.Counter() # 06.11 N: glyphs drawn by the net ('net') / classes it does not know ('unknown' -> library)
19
- def set_style(key,vec): # 04.11 И164: GSTYLE=semb checkpoints — key -> semb3_all embedding of the reference line(s) [256]
20
- STYLES[key]=np.asarray(vec,np.float32).reshape(-1)
21
- def _load():
22
- global _S
23
- if _S is not None: return _S
24
- if E('GDN_DEV'): os.environ['DEV']=E('GDN_DEV')
25
- import train_glyph_dn as T
26
- ck=E('GDN_CKPT',os.path.join(PROJ,'ckpt_dl','glyphdn','glyphdnB','fp16','step050000.pt')); net,classes,writers,lens,step=T.load_net(ck) # 22.10 22:15: run B (w01-only ink_glyphs3, 50k) — the better of A/B on the M3 instruments
27
- c2i={c:i for i,c in enumerate(classes)}; w2i={w:i for i,w in enumerate(writers)}
28
- _S=dict(T=T,net=net,classes=classes,c2i=c2i,writers=writers,w2i=w2i,lens=lens,step=step,cmean={c:True for c in classes},dev=T.dev)
29
- print(f'glyph denoiser {ck} step {step} classes {len(classes)} writers {len(writers)} cfg {E("GDN_CFG","1.5")} steps {E("GDN_STEPS","10")}',flush=True)
30
- return _S
31
- def label_of(cls):
32
- if cls=='BAR': return '—'
33
- if cls=='VEC': return '⃗'
34
- return cls
35
- def sample_batch(classes,writers,rng,tau=None,shrink=None,maxlen=200):
36
- if E('GLYPH_B')=='1': # 05.11 И165: B fine-tuned on glyphs (scripts/glyph_bdn.py); classes it has no LaTeX for come back to _sample_batch
37
- import glyph_bdn; return glyph_bdn.sample_batch(classes,writers,rng,tau,shrink,maxlen)
38
- return _sample_batch(classes,writers,rng,tau,shrink,maxlen)
39
- @torch.no_grad()
40
- def _sample_batch(classes,writers,rng,tau=None,shrink=None,maxlen=200):
41
- S=_load(); T=S['T']; labs=[label_of(c) for c in classes]; ok=[i for i,l in enumerate(labs) if l in S['c2i']]; res=[[] for _ in labs]
42
- N['net']+=len(ok); N['unknown']+=len(labs)-len(ok)
43
- if not ok: return res
44
- nw=len(S['writers']); wnull=E('GDN_WRITER_NULL','1')=='1'
45
- C=[S['c2i'][labs[i]] for i in ok]; W=[S['w2i'].get(writers[i],nw if wnull else S['w2i'].get('w01',nw)) for i in ok]
46
- L=[T.draw_len(S['lens'],w,c,rng) for w,c in zip(W,C)]
47
- if T.GSTYLE: # style rows [emb, null flag] from set_style; the length plan is then class-wide (W ids above are 'unknown')
48
- SD=T.SDIM; W=np.zeros((len(ok),SD+1),np.float32)
49
- for j,i in enumerate(ok):
50
- if writers[i] in STYLES: W[j,:SD]=STYLES[writers[i]]
51
- else: W[j,-1]=1
52
- if E('GDN_LEN','dist')!='dist':
53
- f=float(E('GDN_LEN'))/100.0; L=[max(4,int(round(float(np.median(S['lens'].get((-1,c),[30])))*f))) for c in C]
54
- g=torch.Generator(device='cpu').manual_seed(int(rng.integers(1<<31)))
55
- out=T.sample(S['net'],C,W,L,cfg=float(E('GDN_CFG','1.5')),steps=int(E('GDN_STEPS','10')),gen=g,pen_thr=float(E('GDN_PEN_THR','0')),eta=float(E('GDN_ETA','0')),noise=float(E('GDN_NOISE','1.0')))
56
- for j,i in enumerate(ok): res[i]=dict(strokes=T.to_strokes(out[j]))
57
- return res
58
- def sample(cls,writer,rng,n=1,tau=None,shrink=None):
59
- r=sample_batch([cls]*n,[writer]*n,rng,tau,shrink)
60
- return [] if not r or r[0]==[] else r
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/render_pages.py DELETED
@@ -1,80 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """06.11 (Лев на увеличенный кусок страницы: «А нахрена такое уродское качество?»): the pages were a 150 dpi raster drawn with
3
- PIL lines without antialiasing, and the PDF re-packed that raster as JPEG (staircase edges + grey JPEG dirt around the strokes).
4
- Now the page is kept as STROKES and drawn three ways from them:
5
- pages.pdf VECTOR: every stroke a smooth curve (Catmull-Rom through the model's points -> cubic Bezier), round caps and joins,
6
- A4; sharp at any zoom, a few hundred KB (Flate-compressed streams). Written by hand, no extra libraries.
7
- page_NN.svg the same curves (for the browser / an editor)
8
- page_NN.png 300 dpi, antialiased (drawn at 4x over the curve sampled 8 points per segment, then downscaled with LANCZOS)
9
- Strokes are lightly smoothed first ([1 2 1]/4 once on strokes of >= 5 points; ends kept): the models' points carry a small jitter
10
- that the old 2-px raster hid and a vector line shows.
11
- Input (strokes.json, written by write_pages.py): {"W": px, "H": px, "dpi": 150, "ruling": [y px, ...], "pages": [[stroke, ...], ...]},
12
- stroke = [[x, y], ...] in page px (y down).
13
- python scripts/render_pages.py <strokes.json> <out_dir> [PEN=1.1 (pt) PNG_DPI=300 SMOOTH=1] (re-render without the models)"""
14
- import os, sys, json, zlib, numpy as np
15
- from PIL import Image, ImageDraw
16
- E = os.environ.get
17
-
18
- def smooth(s):
19
- s = np.asarray(s, np.float64)
20
- if E('SMOOTH', '1') != '1' or len(s) < 5: return s
21
- t = s.copy(); t[1:-1] = 0.25 * s[:-2] + 0.5 * s[1:-1] + 0.25 * s[2:]; return t
22
- def bezier(s):
23
- """points [n,2] -> list of cubic segments (p0, c1, c2, p1), Catmull-Rom through all points"""
24
- n = len(s)
25
- if n < 2: return []
26
- P = np.vstack([s[:1], s, s[-1:]]); out = []
27
- for i in range(1, n):
28
- p0, p1 = P[i], P[i + 1]; out.append((p0, p0 + (P[i + 1] - P[i - 1]) / 6, p1 - (P[i + 2] - P[i]) / 6, p1))
29
- return out
30
- def sample(seg, k=8):
31
- p0, c1, c2, p1 = seg; t = np.linspace(0, 1, k + 1)[1:, None]
32
- return (1 - t) ** 3 * p0 + 3 * (1 - t) ** 2 * t * c1 + 3 * (1 - t) * t ** 2 * c2 + t ** 3 * p1
33
-
34
- def pdf(pages, ruling, W, H, dpi, path, pen):
35
- k = 72.0 / dpi; PW, PH = W * k, H * k; f = lambda x, y: f'{x * k:.2f} {PH - y * k:.2f}'
36
- objs = {1: b'<< /Type /Catalog /Pages 2 0 R >>'}; kids = []
37
- for i, strokes in enumerate(pages):
38
- c = ['0.86 G 0.5 w'] + [f'{f(0, y)} m {f(W, y)} l S' for y in ruling]
39
- c.append(f'0.08 G {pen:.2f} w 1 J 1 j')
40
- for st in strokes:
41
- s = smooth(st)
42
- if len(s) == 1: c.append(f'{f(*s[0])} m {f(*s[0])} l S'); continue
43
- c.append(f'{f(*s[0])} m ' + ' '.join(f'{f(*a)} {f(*b)} {f(*d)} c' for _, a, b, d in bezier(s)) + ' S')
44
- data = zlib.compress('\n'.join(c).encode()); po, co = 3 + 2 * i, 4 + 2 * i; kids.append(po)
45
- objs[po] = f'<< /Type /Page /Parent 2 0 R /MediaBox [0 0 {PW:.2f} {PH:.2f}] /Contents {co} 0 R >>'.encode()
46
- objs[co] = f'<< /Length {len(data)} /Filter /FlateDecode >>\nstream\n'.encode() + data + b'\nendstream'
47
- objs[2] = f'<< /Type /Pages /Kids [{" ".join(f"{p} 0 R" for p in kids)}] /Count {len(kids)} >>'.encode()
48
- b = bytearray(b'%PDF-1.4\n'); off = {}
49
- for n in sorted(objs): off[n] = len(b); b += f'{n} 0 obj\n'.encode() + objs[n] + b'\nendobj\n'
50
- x = len(b); b += f'xref\n0 {len(objs) + 1}\n0000000000 65535 f \n'.encode() + b''.join(f'{off[n]:010d} 00000 n \n'.encode() for n in sorted(objs))
51
- b += f'trailer\n<< /Size {len(objs) + 1} /Root 1 0 R >>\nstartxref\n{x}\n%%EOF\n'.encode(); open(path, 'wb').write(b)
52
-
53
- def svg(strokes, ruling, W, H, dpi, path, pen):
54
- w = pen * dpi / 72.0; g = lambda p: f'{p[0]:.2f},{p[1]:.2f}'; L = []
55
- for st in strokes:
56
- s = smooth(st)
57
- L.append(f'M{g(s[0])} L{g(s[0])}' if len(s) == 1 else f'M{g(s[0])} ' + ' '.join(f'C{g(a)} {g(b)} {g(d)}' for _, a, b, d in bezier(s)))
58
- open(path, 'w').write(f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 {W} {H}" width="210mm" height="297mm"><rect width="100%" height="100%" fill="#fff"/>'
59
- + ''.join(f'<line x1="0" y1="{y:.1f}" x2="{W}" y2="{y:.1f}" stroke="#dcdcdc" stroke-width="1"/>' for y in ruling)
60
- + f'<path d="{" ".join(L)}" fill="none" stroke="#141414" stroke-width="{w:.2f}" stroke-linecap="round" stroke-linejoin="round"/></svg>')
61
-
62
- def png(strokes, ruling, W, H, dpi, path, pen, out_dpi):
63
- SS = 4; k = out_dpi / dpi * SS; im = Image.new('L', (int(W * k), int(H * k)), 255); d = ImageDraw.Draw(im); w = max(1, int(round(pen / 72.0 * out_dpi * SS)))
64
- for y in ruling: d.line([(0, y * k), (W * k, y * k)], fill=222, width=SS)
65
- for st in strokes:
66
- s = smooth(st); pts = np.vstack([s[:1]] + [sample(seg) for seg in bezier(s)]) * k if len(s) > 1 else s * k
67
- if len(pts) > 1: d.line([tuple(p) for p in pts], fill=20, width=w, joint='curve')
68
- for p in (pts[0], pts[-1]): d.ellipse([p[0] - w / 2, p[1] - w / 2, p[0] + w / 2, p[1] + w / 2], fill=20)
69
- im.resize((int(W * out_dpi / dpi), int(H * out_dpi / dpi)), Image.LANCZOS).save(path, dpi=(out_dpi, out_dpi))
70
-
71
- def render(doc, out):
72
- W, H, dpi, R = doc['W'], doc['H'], doc.get('dpi', 150), doc['ruling']; pen = float(E('PEN', '1.1')); od = int(E('PNG_DPI', '300'))
73
- pdf(doc['pages'], R, W, H, dpi, os.path.join(out, 'pages.pdf'), pen)
74
- for i, st in enumerate(doc['pages']):
75
- svg(st, R, W, H, dpi, os.path.join(out, f'page_{i + 1:02d}.svg'), pen); png(st, R, W, H, dpi, os.path.join(out, f'page_{i + 1:02d}.png'), pen, od)
76
-
77
- if __name__ == '__main__':
78
- src, out = sys.argv[1], sys.argv[2]
79
- if os.path.exists(out): sys.exit(f'{out} exists')
80
- os.makedirs(out); render(json.load(open(src)), out); print('rendered ->', out)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/style_classifier.py DELETED
@@ -1,269 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """Writer classifier on REAL lines (data/unified export data/ink_lines_u1) + the numeric STYLE metric for generators (22.10, w03:
3
- «ты это как увидел? померил?» — instead of eyeballing figures/denoise_style_sensitivity*.png). DiffInk's «Style» = accuracy of a
4
- writer classifier on generated lines; here: share of generated lines of writer W (style from K=3 reference lines of W) that the
5
- classifier assigns to W.
6
-
7
- Input of the classifier = the generator's representation of a line: (dx, dy, pen) in line heights, resampled to the dataset's arc
8
- step 0.05 inside every stroke (so point density can not be used as a cue; real lines are already at that step, generated ones are
9
- not exactly), then per point [dx*SCALE, dy*SCALE, pen, y - median(y)] (SCALE = 50 as in train_ink_al2.Inks / the style encoder).
10
- Net: Conv1d stem (x4 downsample) -> BiGRU 2x192 -> masked mean+max pooling -> 15 writers (~1.7M params). Training: balanced
11
- sampling over writers, random 512-point crops, the trainer's affine jitter (AUG_S 0.1, AUG_SH 0.15), label smoothing 0.1.
12
- Splits: the unified train/dev/test by notebooks for the 12 writers with train data; the 3 held-out writers (w17, w06,
13
- w13 — only in test by splits.json) are split here BY NOTEBOOK 70/30 (seed 0) into cls-train / cls-test, so the test is always
14
- on unseen notebooks for all 15 writers (STATS in the eval json say which notebooks).
15
-
16
- MODES (PY=~/.venvs/whisper/bin/python; MPS)
17
- MODE=train [STEPS=8000 BS=64 LR=1e-3 OUT=ckpt_stylecls] -> OUT/best.pt (best dev), OUT/last.pt, OUT/train.log
18
- MODE=eval [CKPT=ckpt_stylecls/best.pt] -> accuracy on test (15 cl.), per writer, prose-only, the 5 sheet writers; OUT/eval.json
19
- MODE=metric DN_CKPT=<denoise ckpt> [TAG=denoise1 CFG=1.5 SSTEPS=20 SCFGS=1,1.5,2,3 N=60 SEED=3 BASE=<lstm fp16>]
20
- -> out/style_metric/<TAG>/: seqs_<writer>_<cond>.npz (cached, generation skipped if present), metric.json, metric.md,
21
- figures/style_metric_<TAG>.png (table + confusion of denoiser SCFG=1), figures/style_metric_<TAG>_sheet.png (eyeball)
22
- Conditions: real (dev+test prose lines of the writer, unseen notebooks — the ceiling), lstm (BASE = lstmal_v6isl3@100k, bias 1.0,
23
- the same refs), dn_s<SCFG> (denoiser, CFG, SSTEPS). Texts = the standard judge's 60 v2-valid prose texts (seed 3), the same for
24
- every writer and condition; refs = exactly those of scripts/denoise_style_sheet.py (w01: v2 valid 0,16,17; others: 3 prose lines
25
- of the largest v6 page); durations from the duration head of BASE; the same torch seed per batch of 16 in every condition."""
26
- import os, sys, json, math, time, collections, numpy as np
27
- PROJ=os.path.dirname(os.path.dirname(os.path.abspath(__file__))); sys.path.insert(0,os.path.join(PROJ,'scripts'))
28
- E=os.environ.get; MODE=E('MODE','train')
29
- import torch, torch.nn as nn, torch.nn.functional as F
30
- dev='cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else 'cpu')
31
- SCALE=50.0; STEP=0.05; MAXPTS=1200; CROP=int(E('CROP','512'))
32
- WRITERS=['w01','w03','w05','w06','w07','w08','w09','w10','w11','w12','w13','w14','w15','w16','w17']
33
- NAMES={'w01':'w01','w03':'Лев','w05':'w05','w06':'w06','w07':'w07','w08':'w08','w09':'w09','w10':'w10','w11':'w11','w12':'w12','w13':'w13','w14':'w14','w15':'w15','w16':'w16','w17':'w17'}
34
- WID={w:i for i,w in enumerate(WRITERS)}; HOLDOUT=['w17','w06','w13']; SHEET5=['w01','w03','w09','w12','w11']
35
- U1=os.path.join(PROJ,'data','ink_lines_u1'); OUT=os.path.join(PROJ,E('OUT','ckpt_stylecls'))
36
-
37
- # ---------------- representation
38
- def resample(seq,step=STEP):
39
- """seq [n,3] (dx,dy,pen: pen-up AFTER this point) raw units -> same format, every stroke resampled to arc step `step`."""
40
- seq=np.asarray(seq,np.float32)
41
- if len(seq)==0: return np.zeros((1,3),np.float32)
42
- P=np.cumsum(seq[:,:2],0); pen=seq[:,2]>0.5; ends=np.flatnonzero(pen); starts=np.concatenate([[0],ends[:-1]+1]) if len(ends) else np.array([0])
43
- if len(ends)==0 or ends[-1]!=len(seq)-1: ends=np.append(ends,len(seq)-1); starts=np.concatenate([[0],ends[:-1]+1])
44
- outP=[];outpen=[]
45
- for a,b in zip(starts,ends):
46
- if b<a: continue
47
- s=P[a:b+1]
48
- if len(s)>1:
49
- L=np.concatenate([[0],np.cumsum(np.hypot(*np.diff(s,axis=0).T))]); tot=L[-1]
50
- if tot>step*0.5:
51
- t=np.arange(0,tot,step)
52
- if tot-t[-1]>step*0.25: t=np.append(t,tot)
53
- s=np.stack([np.interp(t,L,s[:,0]),np.interp(t,L,s[:,1])],1)
54
- else: s=s[[0]]
55
- outP.append(s); p=np.zeros(len(s),np.float32); p[-1]=1; outpen.append(p)
56
- P2=np.concatenate(outP); pen2=np.concatenate(outpen); d=np.zeros((len(P2),3),np.float32); d[1:,:2]=np.diff(P2,axis=0); d[:,2]=pen2; return d
57
- def feats(seq):
58
- """resampled (dx,dy,pen) -> [n,4] features: dx*SCALE, dy*SCALE, pen, y-median(y)"""
59
- seq=np.asarray(seq,np.float32)[:MAXPTS]; y=np.cumsum(seq[:,1]); return np.stack([seq[:,0]*SCALE,seq[:,1]*SCALE,seq[:,2],y-np.median(y)],1).astype(np.float32)
60
-
61
- # ---------------- data
62
- def load_split(sp):
63
- z=np.load(os.path.join(U1,f'{sp}_raw.npz'),allow_pickle=True); meta=[json.loads(l) for l in open(os.path.join(U1,f'meta_{sp}.jsonl'))]
64
- assert len(meta)==len(z['labels']); return z['pts'],z['pt_off'],meta
65
- def cls_splits():
66
- """-> dict split -> list of (pts_array, off_start, off_end, writer, kind, doc, src_split). Held-out writers' notebooks: 70/30 by notebook (seed 0)."""
67
- out={'train':[],'dev':[],'test':[]}; rng=np.random.default_rng(0); hold_docs={}
68
- for sp in ('train','dev','test'):
69
- pts,off,meta=load_split(sp)
70
- for i,r in enumerate(meta):
71
- w=r['writer_id']; rec=(pts,int(off[i]),int(off[i+1]),w,r['kind'],r['doc'],sp)
72
- if w in HOLDOUT:
73
- hold_docs.setdefault(w,{}).setdefault(r['doc'],[]).append(rec)
74
- else: out[sp].append(rec)
75
- assign={}
76
- for w,docs in hold_docs.items():
77
- names=sorted(docs); rng.shuffle(names); n=len(names); ntest=max(1,int(round(0.3*n)))
78
- for k,d in enumerate(names):
79
- s='test' if k<ntest else 'train'; assign[(w,d)]=s; out[s]+=docs[d]
80
- return out,assign
81
- class Lines:
82
- def __init__(s,recs,cache_tag=None):
83
- s.recs=recs; s.y=np.array([WID[r[3]] for r in recs]); s.kind=[r[4] for r in recs]; s.n=len(recs)
84
- s.cache=[None]*s.n
85
- def seq(s,i):
86
- pts,a,b=s.recs[i][:3]; return pts[a:b]
87
- def feat(s,i):
88
- if s.cache[i] is None: s.cache[i]=feats(resample(s.seq(i)))
89
- return s.cache[i]
90
- def by_writer(s):
91
- d=collections.defaultdict(list)
92
- for i,w in enumerate(s.y): d[int(w)].append(i)
93
- return d
94
- AUG_S=float(E('AUG_S','0.1')); AUG_SH=float(E('AUG_SH','0.15'))
95
- def make_batch(fs,train,rng=None):
96
- """fs: list of [n,4] features -> X [B,T,4] padded, lengths. train: random crop CROP + affine jitter (on dx,dy; y recomputed)."""
97
- xs=[]
98
- for f in fs:
99
- f=f.copy()
100
- if train:
101
- if len(f)>CROP: a=rng.integers(0,len(f)-CROP+1); f=f[a:a+CROP]
102
- sx=1+(rng.random()*2-1)*AUG_S; sy=1+(rng.random()*2-1)*AUG_S; sh=(rng.random()*2-1)*AUG_SH
103
- dx=f[:,0]*sx+f[:,1]*sh; dy=f[:,1]*sy; y=np.cumsum(dy)/SCALE; f=np.stack([dx,dy,f[:,2],y-np.median(y)],1)
104
- xs.append(f)
105
- L=np.array([len(x) for x in xs]); T=L.max(); X=np.zeros((len(xs),T,4),np.float32)
106
- for b,x in enumerate(xs): X[b,:len(x)]=x
107
- return torch.from_numpy(X).to(dev),torch.from_numpy(L)
108
-
109
- # ---------------- model
110
- class Net(nn.Module):
111
- def __init__(s,nc=len(WRITERS),d=192):
112
- super().__init__()
113
- s.stem=nn.Sequential(nn.Conv1d(4,128,5,padding=2),nn.GELU(),nn.Conv1d(128,192,5,stride=2,padding=2),nn.GELU(),nn.Conv1d(192,256,5,stride=2,padding=2),nn.GELU())
114
- s.gru=nn.GRU(256,d,num_layers=2,batch_first=True,bidirectional=True,dropout=0.1)
115
- s.head=nn.Sequential(nn.Linear(4*d,256),nn.GELU(),nn.Dropout(0.2),nn.Linear(256,nc))
116
- def forward(s,X,L):
117
- h=s.stem(X.transpose(1,2)).transpose(1,2); L2=torch.clamp((L+3)//4,min=1) # x4 downsample
118
- p=nn.utils.rnn.pack_padded_sequence(h,L2.cpu(),batch_first=True,enforce_sorted=False); o,_=s.gru(p); o,_=nn.utils.rnn.pad_packed_sequence(o,batch_first=True)
119
- m=(torch.arange(o.size(1),device=o.device)[None]<L2.to(o.device)[:,None]).float()[...,None]
120
- mean=(o*m).sum(1)/m.sum(1); mx=(o+(m-1)*1e4).max(1).values; return s.head(torch.cat([mean,mx],-1))
121
- def load_cls(path=None):
122
- path=path or E('CKPT',os.path.join(OUT,'best.pt')); st=torch.load(path,map_location=dev); net=Net().to(dev); net.load_state_dict(st['m']); net.eval(); return net,st
123
- @torch.no_grad()
124
- def predict(net,fs,bs=64):
125
- """fs: list of [n,4] features -> logits [N,15] (numpy). Sorted by length for small padding."""
126
- net.eval(); order=np.argsort([len(f) for f in fs]); out=np.zeros((len(fs),len(WRITERS)),np.float32)
127
- for i in range(0,len(fs),bs):
128
- ids=order[i:i+bs]; X,L=make_batch([fs[j] for j in ids],False); out[ids]=net(X,L).float().cpu().numpy()
129
- return out
130
-
131
- # ---------------- train / eval
132
- def train():
133
- os.makedirs(OUT,exist_ok=True); log=open(os.path.join(OUT,'train.log'),'a'); P=lambda *a: (print(*a,flush=True),log.write(' '.join(str(x) for x in a)+'\n'),log.flush())
134
- sp,assign=cls_splits(); tr=Lines(sp['train']); dv=Lines(sp['dev']); P(f'train {tr.n} dev {dv.n} test {len(sp["test"])} dev={dev} holdout notebooks: {sum(v=="train" for v in assign.values())} train / {sum(v=="test" for v in assign.values())} test')
135
- t0=time.time(); [tr.feat(i) for i in range(tr.n)]; [dv.feat(i) for i in range(dv.n)]; P(f'features (resample+feats) {time.time()-t0:.0f}s')
136
- bw=tr.by_writer(); ws=sorted(bw); P('train per writer',{WRITERS[w]:len(bw[w]) for w in ws})
137
- STEPS=int(E('STEPS','8000')); BS=int(E('BS','64')); LR=float(E('LR','1e-3')); WARM=300
138
- net=Net().to(dev); P(f'params {sum(p.numel() for p in net.parameters())/1e6:.2f}M'); opt=torch.optim.AdamW(net.parameters(),LR,weight_decay=0.01); rng=np.random.default_rng(int(E('SEED','0')))
139
- best=0; step=0; t0=time.time(); lsum=0; asum=0
140
- dvf=[dv.feat(i) for i in range(dv.n)]
141
- while step<STEPS:
142
- lr=LR*min(1,(step+1)/WARM)*0.5*(1+math.cos(math.pi*step/STEPS))
143
- for g in opt.param_groups: g['lr']=lr
144
- wsel=rng.choice(ws,BS); ids=[bw[w][rng.integers(len(bw[w]))] for w in wsel] # balanced over writers
145
- X,L=make_batch([tr.feat(i) for i in ids],True,rng); y=torch.from_numpy(tr.y[ids]).to(dev)
146
- net.train(); o=net(X,L); loss=F.cross_entropy(o,y,label_smoothing=0.1); opt.zero_grad(); loss.backward(); nn.utils.clip_grad_norm_(net.parameters(),5.0); opt.step(); step+=1
147
- lsum+=loss.item(); asum+=(o.argmax(-1)==y).float().mean().item()
148
- if step%100==0: P(f'step {step}/{STEPS} loss {lsum/100:.3f} acc {asum/100:.3f} lr {lr:.2e} {step/(time.time()-t0):.1f}it/s eta {(STEPS-step)/(step/(time.time()-t0))/60:.0f}min'); lsum=asum=0
149
- if step%500==0 or step==STEPS:
150
- pr=predict(net,dvf).argmax(-1); acc=(pr==dv.y).mean(); bal=np.mean([(pr[dv.y==w]==w).mean() for w in np.unique(dv.y)])
151
- P(f' dev acc {acc:.4f} balanced {bal:.4f}'); torch.save(dict(m=net.state_dict(),step=step,dev_acc=float(acc),dev_bal=float(bal)),os.path.join(OUT,'last.pt'))
152
- if bal>best: best=bal; torch.save(dict(m=net.state_dict(),step=step,dev_acc=float(acc),dev_bal=float(bal)),os.path.join(OUT,'best.pt')); P(' best')
153
- P('CLS DONE best dev balanced',best)
154
- def evaluate():
155
- sp,assign=cls_splits(); te=Lines(sp['test']); net,st=load_cls(); print('ckpt step',st.get('step'),'dev acc',st.get('dev_acc'),'dev bal',st.get('dev_bal'))
156
- fs=[te.feat(i) for i in range(te.n)]; pr=predict(net,fs).argmax(-1); y=te.y; kind=np.array(te.kind)
157
- res=dict(step=st.get('step'),n_test=int(te.n),acc=float((pr==y).mean()),balanced=float(np.mean([(pr[y==w]==w).mean() for w in np.unique(y)])),
158
- acc_prose=float((pr==y)[kind=='prose'].mean()),n_prose=int((kind=='prose').sum()),per_writer={},sheet5={},holdout_notebooks={f'{w}:{d}':s for (w,d),s in assign.items()})
159
- for w in WRITERS:
160
- m=y==WID[w]; mp=m&(kind=='prose')
161
- if m.sum(): res['per_writer'][w]=dict(n=int(m.sum()),acc=float((pr[m]==WID[w]).mean()),n_prose=int(mp.sum()),acc_prose=float((pr[mp]==WID[w]).mean()) if mp.sum() else None)
162
- m5=np.isin(y,[WID[w] for w in SHEET5]); res['sheet5']=dict(n=int(m5.sum()),acc=float((pr[m5]==y[m5]).mean()),acc_prose=float((pr==y)[m5&(kind=='prose')].mean()),
163
- acc_within5=float(np.mean([pr[i] in [WID[w] for w in SHEET5] and pr[i]==y[i] for i in np.flatnonzero(m5)])))
164
- cm=np.zeros((len(WRITERS),len(WRITERS)),int)
165
- for a,b in zip(y,pr): cm[a,b]+=1
166
- res['confusion']=cm.tolist(); json.dump(res,open(os.path.join(OUT,'eval.json'),'w'),ensure_ascii=False,indent=1)
167
- print(f"TEST acc {res['acc']:.4f} balanced {res['balanced']:.4f} prose {res['acc_prose']:.4f} (n {res['n_prose']}) | sheet5 acc {res['sheet5']['acc']:.4f} prose {res['sheet5']['acc_prose']:.4f}")
168
- for w,r in res['per_writer'].items(): print(f" {w:10s} n {r['n']:5d} acc {r['acc']:.3f} prose n {r['n_prose']:4d} acc {r['acc_prose'] if r['acc_prose'] is None else round(r['acc_prose'],3)}")
169
-
170
- # ---------------- metric on generated lines
171
- def v6_lines(): # copied from scripts/denoise_style_sheet.py (refs identical to the sheet)
172
- out={}
173
- for sp in ('valid','train'):
174
- z=np.load(os.path.join(PROJ,'data','ink_lines_v6',f'{sp}_raw.npz'),allow_pickle=True); labels=[str(l) for l in z['labels']]; off=z['pt_off']; k=0
175
- for l in open(os.path.join(PROJ,'data','ink_dataset_v6_sorted',f'meta_{sp}.jsonl')):
176
- r=json.loads(l)
177
- if k<len(labels) and r['label']==labels[k]:
178
- if '$' not in labels[k] and 20<=len(labels[k])<=60 and 150<=off[k+1]-off[k]<=1200: out.setdefault(r['src'],{}).setdefault((sp,r['doc'],r['page']),[]).append(z['pts'][off[k]:off[k+1]].astype(np.float32))
179
- k+=1
180
- return out
181
- def refs_for(src,L):
182
- if src=='w01':
183
- z=np.load(os.path.join(PROJ,'data','ink_lines_v2','valid_raw.npz'),allow_pickle=True); off=z['pt_off']; return [z['pts'][off[i]:off[i+1]].astype(np.float32) for i in (0,16,17)],('v2valid',0,16,17)
184
- pages=sorted(L[src].items(),key=lambda kv:-len(kv[1])); return pages[0][1][:3],pages[0][0]
185
- def judge_texts(T2,N,SEED): # exactly the standard judge's pool/choice (train_ink_denoise.judge, KIND=text)
186
- va=T2.Inks('valid'); rng=np.random.RandomState(SEED); pool=[i for i in va.idx if 4<=len(str(va.labels[i]))<=90 and '$' not in str(va.labels[i])]; ids=rng.choice(pool,min(N,len(pool)),replace=False)
187
- return [str(va.labels[i]) for i in ids]
188
- def metric():
189
- os.environ['MODE']='judge'; os.environ.setdefault('PROJ',PROJ) # train_ink_denoise reads MODE before importing the trainer (v2 valid / align_all for the judge's texts)
190
- import train_ink_denoise as DN; T2=DN.T2
191
- TAG=E('TAG','denoise1'); CFG=float(E('CFG','1.5')); SST=int(E('SSTEPS','20')); SCFGS=[float(x) for x in E('SCFGS','1,1.5,2,3').split(',')]; N=int(E('N','60')); SEED=int(E('SEED','3'))
192
- od=os.path.join(PROJ,'out','style_metric',TAG); os.makedirs(od,exist_ok=True); fig=os.path.join(PROJ,'figures')
193
- texts=judge_texts(T2,N,SEED); toks=[list(t) for t in texts]; json.dump(texts,open(os.path.join(od,'texts.json'),'w'),ensure_ascii=False)
194
- base=DN.load_base(); L=v6_lines(); refs={}; refsrc={}
195
- for w in SHEET5: refs[w],refsrc[w]=refs_for(w,L)
196
- svs={w:T2.style_from_seqs(base,refs[w]) for w in SHEET5}; durs=DN.head_durs(base,toks)
197
- conds=['lstm']+[f'dn_s{s:g}' for s in SCFGS]; net=None
198
- def gen(w,cond):
199
- f=os.path.join(od,f'seqs_{w}_{cond}.npz')
200
- if os.path.exists(f): z=np.load(f,allow_pickle=True); return list(z['seqs'])
201
- nonlocal net
202
- if cond!='lstm' and net is None: net,step=DN.load_net(E('DN_CKPT')); print('denoiser step',step,flush=True)
203
- seqs=[]; t0=time.time()
204
- for i in range(0,N,16):
205
- torch.manual_seed(1000+i)
206
- if cond=='lstm': seqs+=T2._sample(base,toks[i:i+16],bias=1.0,sv=svs[w])
207
- else: os.environ['SCFG']=cond[4:]; seqs+=DN.sample(net,toks[i:i+16],durs[i:i+16],svs[w],cfg=CFG,steps=SST)
208
- np.savez(f,seqs=np.array([np.asarray(q,np.float32) for q in seqs],dtype=object),labels=np.array(texts,dtype=object)); print(f' {w} {cond}: {len(seqs)} lines {time.time()-t0:.0f}s',flush=True); return seqs
209
- # real ceiling: dev+test prose lines of the writer (unseen notebooks), up to N, seed-fixed
210
- real={}; rng=np.random.default_rng(SEED)
211
- for sp in ('dev','test'):
212
- pts,off,meta=load_split(sp)
213
- for i,r in enumerate(meta):
214
- if r['writer_id'] in SHEET5 and r['kind']=='prose' and 4<=len(r['label'])<=90: real.setdefault(r['writer_id'],[]).append(pts[off[i]:off[i+1]].astype(np.float32))
215
- for w in SHEET5: ids=rng.permutation(len(real[w]))[:N]; real[w]=[real[w][i] for i in ids]
216
- allseqs={}
217
- for w in SHEET5:
218
- allseqs[(w,'real')]=real[w]
219
- for c in conds: allseqs[(w,c)]=gen(w,c)
220
- cls,st=load_cls(); res=dict(tag=TAG,cfg=CFG,ssteps=SST,n_texts=N,seed=SEED,cls_step=st.get('step'),dn_ckpt=E('DN_CKPT'),refs={w:str(refsrc[w]) for w in SHEET5},acc={},conf={},n={})
221
- for (w,c),seqs in allseqs.items():
222
- fs=[feats(resample(q)) for q in seqs]; pr=predict(cls,fs).argmax(-1); res['acc'].setdefault(c,{})[w]=float((pr==WID[w]).mean()); res['n'].setdefault(c,{})[w]=len(seqs)
223
- res['conf'].setdefault(c,{})[w]=np.bincount(pr,minlength=len(WRITERS)).tolist()
224
- json.dump(res,open(os.path.join(od,'metric.json'),'w'),ensure_ascii=False,indent=1)
225
- # markdown table
226
- allc=['real']+conds; lines=[f'### Метрика стиля ({TAG}, CFG {CFG}, {SST} шагов, {N} текстов seed {SEED}, классификатор шаг {st.get("step")})','',
227
- '| условие | '+' | '.join(NAMES[w] for w in SHEET5)+' | среднее |','|---|'+'---|'*(len(SHEET5)+1)]
228
- for c in allc: a=[res['acc'][c][w] for w in SHEET5]; lines.append(f'| {c} | '+' | '.join(f'{x*100:.0f}%' for x in a)+f' | {np.mean(a)*100:.0f}% |')
229
- lines+=['','Матрицы «заданный писец × узнанный» (строки = заданный, столбцы = узнанный; число строк из %d):'%N]
230
- for c in allc:
231
- lines+=['',f'{c}:','| задан \\ узнан | '+' | '.join(NAMES[w] for w in WRITERS)+' |','|---|'+'---|'*len(WRITERS)]
232
- for w in SHEET5: lines.append(f'| {NAMES[w]} | '+' | '.join(('**%d**' if WRITERS[j]==w else '%d')%v for j,v in enumerate(res['conf'][c][w]))+' |')
233
- open(os.path.join(od,'metric.md'),'w').write('\n'.join(lines)+'\n'); print('\n'.join(lines[:4+len(allc)]),flush=True)
234
- plot(res,allc,os.path.join(fig,f'style_metric_{TAG}.png')); sheet(allseqs,allc,texts,os.path.join(fig,f'style_metric_{TAG}_sheet.png'),DN.render)
235
- print('METRIC DONE',od,flush=True)
236
- def plot(res,allc,path):
237
- import matplotlib; matplotlib.use('Agg'); import matplotlib.pyplot as plt
238
- fig,ax=plt.subplots(1,2,figsize=(15,5.2),gridspec_kw=dict(width_ratios=[1,1.4]))
239
- A=np.array([[res['acc'][c][w] for w in SHEET5] for c in allc])*100; im=ax[0].imshow(A,cmap='Greens',vmin=0,vmax=100)
240
- ax[0].set_xticks(range(len(SHEET5))); ax[0].set_xticklabels([NAMES[w] for w in SHEET5]); ax[0].set_yticks(range(len(allc))); ax[0].set_yticklabels(allc)
241
- for i in range(len(allc)):
242
- for j in range(len(SHEET5)): ax[0].text(j,i,f'{A[i,j]:.0f}',ha='center',va='center',color='black' if A[i,j]<60 else 'white',fontsize=11)
243
- ax[0].set_title('узнан заданный писец, % строк'); ax[0].set_xlabel('заданный писец (опоры K=3)')
244
- c0=[c for c in allc if c.startswith('dn_s1') and c!='dn_s1.5'][0] if any(c=='dn_s1' for c in allc) else allc[-1]
245
- C=np.array([res['conf'][c0][w] for w in SHEET5],float); C=C/C.sum(1,keepdims=True)*100; im2=ax[1].imshow(C,cmap='Blues',vmin=0,vmax=100)
246
- ax[1].set_xticks(range(len(WRITERS))); ax[1].set_xticklabels([NAMES[w] for w in WRITERS],rotation=60,ha='right'); ax[1].set_yticks(range(len(SHEET5))); ax[1].set_yticklabels([NAMES[w] for w in SHEET5])
247
- for i in range(len(SHEET5)):
248
- for j in range(len(WRITERS)):
249
- if C[i,j]>=3: ax[1].text(j,i,f'{C[i,j]:.0f}',ha='center',va='center',color='black' if C[i,j]<60 else 'white',fontsize=9)
250
- ax[1].set_title(f'{c0}: заданный (строки) × узнанный (столбцы), %'); ax[1].set_xlabel('узнанный писец')
251
- fig.suptitle(f"Метрика стиля: классификатор писца (15 кл.) на строках {res['tag']} — {time.strftime('%Y-%m-%d %H:%M')}",fontsize=11); fig.tight_layout(); fig.savefig(path,dpi=130); print('fig',path,flush=True)
252
- def sheet(allseqs,allc,texts,path,render,nt=3):
253
- from PIL import Image, ImageDraw, ImageFont
254
- os.environ.setdefault('RENDER_S','40'); f=ImageFont.truetype(os.path.join(PROJ,'DejaVuSans.ttf'),14); CW=420
255
- def fit(im):
256
- im=im.convert('L'); r=min(1.0,(CW-8)/im.width); return im.resize((max(1,int(im.width*r)),max(1,int(im.height*r))),Image.LANCZOS)
257
- rows=[]
258
- for ti in range(min(nt,len(texts))):
259
- for c in allc: rows.append((f'{c}: {texts[ti][:50]}',[fit(render(allseqs[(w,c)][ti])) for w in SHEET5]))
260
- H=sum(max(im.height for im in ims)+24 for _,ims in rows)+30; sh=Image.new('L',(CW*len(SHEET5)+10,H),255); d=ImageDraw.Draw(sh)
261
- for k,w in enumerate(SHEET5): d.text((10+k*CW,4),NAMES[w],fill=0,font=f)
262
- y=30
263
- for cap,ims in rows:
264
- d.text((10,y),cap,fill=0,font=f); y+=20; hh=max(im.height for im in ims)
265
- for k,im in enumerate(ims): sh.paste(im,(10+k*CW,y))
266
- d.line([(0,y+hh+2),(sh.width,y+hh+2)],fill=200); y+=hh+4
267
- sh.save(path); print('sheet',path,flush=True)
268
- if __name__=='__main__':
269
- {'train':train,'eval':evaluate,'metric':metric}[MODE]()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/train_glyph_cls.py DELETED
@@ -1,53 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """Glyph classifier for SELECTING generated formula glyphs (formula_write.py SEL_K). Same CNN/features as formula_glyph_clean.py
3
- (32x32 render + relative height box_h/fh + log aspect), trained ONCE on data/formula_glyphs/glyphs_clean.jsonl WITHOUT the valid
4
- notebooks of ink_dataset_v2 (the formula judge's 40 formulas come from them), classes with >= MINC samples. Accuracy is reported on
5
- the held-out valid-notebook glyphs. Output: ckpt_dl/glyph_cls1.pt {state, classes}. Env: PROJ, MINC (30), EPOCHS (8)."""
6
- import os, sys, json, collections, numpy as np, torch, torch.nn as nn, torch.nn.functional as F
7
- from PIL import Image, ImageDraw
8
- PROJ=os.environ.get('PROJ',os.getcwd()); E=os.environ.get; MINC=int(E('MINC','30')); EPOCHS=int(E('EPOCHS','8'))
9
- dev='cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else 'cpu')
10
- def render(strokes,S=32):
11
- pts=np.array([p[:2] for s in strokes for p in s],np.float32); x0,y0=pts.min(0); x1,y1=pts.max(0); sc=(S-4)/max(x1-x0,y1-y0,1e-6)
12
- im=Image.new('L',(S,S),0); d=ImageDraw.Draw(im); ox=(S-(x1-x0)*sc)/2; oy=(S-(y1-y0)*sc)/2
13
- for s in strokes:
14
- q=[((p[0]-x0)*sc+ox,(p[1]-y0)*sc+oy) for p in s]
15
- if len(q)>1: d.line(q,fill=255,width=2)
16
- elif q: d.point(q[0],fill=255)
17
- return np.asarray(im,np.float32)/255.0
18
- def feats(strokes,relh):
19
- pts=np.array([p[:2] for s in strokes for p in s],np.float32); w=np.ptp(pts[:,0]); h=np.ptp(pts[:,1])
20
- return np.array([relh,np.log((w+1e-3)/(h+1e-3))],np.float32)
21
- class Net(nn.Module):
22
- def __init__(s,n):
23
- super().__init__(); s.c=nn.Sequential(nn.Conv2d(1,32,3,padding=1),nn.ReLU(),nn.MaxPool2d(2),nn.Conv2d(32,64,3,padding=1),nn.ReLU(),nn.MaxPool2d(2),nn.Conv2d(64,128,3,padding=1),nn.ReLU(),nn.AdaptiveAvgPool2d(2))
24
- s.f=nn.Sequential(nn.Linear(128*4+2,256),nn.ReLU(),nn.Dropout(0.3),nn.Linear(256,n))
25
- def forward(s,x,a): return s.f(torch.cat([s.c(x).flatten(1),a],1))
26
- def load(path):
27
- st=torch.load(path,map_location=dev); net=Net(len(st['classes'])).to(dev); net.load_state_dict(st['state']); net.eval(); return net,st['classes']
28
- if __name__=='__main__':
29
- valid_docs={json.loads(l)['doc'] for l in open(os.path.join(PROJ,'data','ink_dataset_v2','valid.jsonl'))} if os.path.exists(os.path.join(PROJ,'data','ink_dataset_v2','valid.jsonl')) else set(json.load(open(os.path.join(PROJ,'data','m3','valid_docs.json'))))
30
- LIB=E('LIB','') # M3 22.10: LIB=data/m3/clean2.npz (glyph_m3.pack of glyphs_clean2.jsonl), OUT=ckpt_dl/glyph_cls2.pt, EPOCHS 12, MINC 20
31
- if LIB.endswith('.npz'):
32
- sys.path.insert(0,os.path.dirname(os.path.abspath(__file__))); import glyph_m3 as GM
33
- G=[dict(cls=g['cls'],doc=g['doc'],strokes=g['strokes'],relh=g['relh']) for g in GM.load(os.path.join(PROJ,LIB))]
34
- else:
35
- G=[json.loads(l) for l in open(os.path.join(PROJ,'data','formula_glyphs','glyphs_clean.jsonl'))]
36
- for g in G: g['relh']=(g['box'][3]-g['box'][1])/max(g['fh'],1e-6)
37
- tr=[g for g in G if g['doc'] not in valid_docs]; te=[g for g in G if g['doc'] in valid_docs]
38
- cnt=collections.Counter(g['cls'] for g in tr); classes=sorted(c for c,k in cnt.items() if k>=MINC); ci={c:i for i,c in enumerate(classes)}
39
- tr=[g for g in tr if g['cls'] in ci]; te=[g for g in te if g['cls'] in ci]; print('train',len(tr),'held-out (valid notebooks)',len(te),'classes',len(classes),flush=True)
40
- def XY(L): return (np.stack([render(g['strokes']) for g in L]),np.stack([feats(g['strokes'],g['relh']) for g in L]),np.array([ci[g['cls']] for g in L]))
41
- Xa,Aa,Ya=XY(tr); Xt,At,Yt=XY(te); rng=np.random.default_rng(0); torch.manual_seed(0); net=Net(len(classes)).to(dev); opt=torch.optim.AdamW(net.parameters(),2e-3,weight_decay=1e-4)
42
- for ep in range(EPOCHS):
43
- perm=rng.permutation(len(Ya)); net.train()
44
- for b in range(0,len(perm),256):
45
- j=perm[b:b+256]; loss=F.cross_entropy(net(torch.from_numpy(Xa[j])[:,None].to(dev),torch.from_numpy(Aa[j]).to(dev)),torch.from_numpy(Ya[j]).to(dev),label_smoothing=0.05); opt.zero_grad(); loss.backward(); opt.step()
46
- net.eval()
47
- with torch.no_grad(): pr=torch.cat([net(torch.from_numpy(Xt[b:b+512])[:,None].to(dev),torch.from_numpy(At[b:b+512]).to(dev)).argmax(1).cpu() for b in range(0,len(Yt),512)]).numpy()
48
- print(f'epoch {ep+1}: held-out acc {(pr==Yt).mean():.3f}',flush=True)
49
- OUT=E('OUT','ckpt_dl/glyph_cls1.pt'); os.makedirs(os.path.dirname(os.path.join(PROJ,OUT)),exist_ok=True); torch.save(dict(state=net.state_dict(),classes=classes),os.path.join(PROJ,OUT)); print('saved',OUT)
50
- conf=collections.Counter((classes[a],classes[b]) for a,b in zip(Yt,pr) if a!=b); per={c:[0,0] for c in classes}
51
- for a,b in zip(Yt,pr): per[classes[a]][0]+=1; per[classes[a]][1]+=int(a==b)
52
- json.dump(dict(acc=float((pr==Yt).mean()),n=int(len(Yt)),per={c:dict(n=v[0],acc=v[1]/v[0]) for c,v in per.items() if v[0]},confusions=[[a,b,k] for (a,b),k in conf.most_common()]),open(os.path.join(PROJ,OUT.replace('.pt','_confusion.json')),'w'),ensure_ascii=False,indent=1)
53
- print('top confusions (true -> predicted, count):',conf.most_common(25))
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/train_glyph_dn.py DELETED
@@ -1,318 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """И116 (22.10, w03: «перенести рецепт денойзера на формулы — генерировать только символы»): a diffusion DENOISER of ONE formula
3
- glyph, the recipe of scripts/train_ink_denoise.py (И100) transplanted from lines to glyphs.
4
- x0 per point : [ (x - cx)/XS , (y - cy)/YS , pen ±1 , 2*pressure-1 ] — ABSOLUTE coordinates in the glyph's box (cx, cy = box
5
- centre, norm_h units of the VAE data = 2 x median glyph height of its formula), pen = «pen up after this point»
6
- as an analog bit, pressure as an analog channel (XS = YS = 0.15: measured residual sd on 3 000 train glyphs)
7
- condition : class (embedding + FiLM; NULL row for CFG, dropped with p=PDROP per glyph), writer (embedding + FiLM, 17 writers
8
- of v7 + a null row dropped with p=PWDROP = «average writer», also the fallback for an unknown writer), per-point
9
- phase (pos/n, sin, cos), log length, position index; noise level t -> FiLM in every block; self-conditioning
10
- on the previous x0 estimate.
11
- Style: a learned writer vector instead of the frozen v6isl3 line encoder — that encoder reads prose LINES of
12
- ~1 200 points and followed style weakly even there (DIARY 22.10 20:29); a glyph is 27 points, outside its domain.
13
- denoiser : ConvNeXt-1D (dilated depthwise k=7) -> BiGRU -> ConvNeXt-1D, FiLM by (t, class, writer); predicts x0 (xy, pressure)
14
- + a pen logit. D=256, pre 4 / post 3, GRU 2x256 -> ~6M params (glyphs are short: median 27 points, cap MAXLEN 128).
15
- diffusion : cosine schedule, T=100, DDIM (eta 0) with GDN_STEPS (10) at sampling, CFG over the class (GDN_CFG 1.5).
16
- loss : L1 positions (z units) + L1 x20 on point-to-point increments (raw units) + BCE pen (pos_weight 3) + L1 pressure.
17
- plan (length) : at sampling the point count is drawn from the empirical length distribution of the (writer, class) train glyphs
18
- (class over all writers when the writer has < LEN_MIN of it) — saved in the checkpoint (lens).
19
- Data: train_glyph_vae.load_split on INK_DS / META_DS (default ink_glyphs_v7 / ink_dataset_glyphs_v7 = clean3 of all 17 writers,
20
- 110 102 train glyphs, valid = w01's held-out notebooks). Length-bucketed batches, affine augmentation as the VAE (AUG_S, AUG_SH).
21
- GSTYLE=semb (04.11, И164: one reference LINE -> glyphs of that hand, also for writers never seen): the writer condition is the
22
- frozen semb3_all embedding (data/<SEMB_DS>/semb_<SEMB_TAG>_train.npy, the style vector B takes) of a random line of the same writer
23
- from ANOTHER doc (another page if the writer has one doc), drawn afresh every batch; dropped with p=PWDROP to a null flag (the
24
- «average writer»). src in the data becomes "writer<TAB>doc" (wr() strips it). sample() then takes W as float rows [sdim+1].
25
- MODES: MODE=train (06.11: INIT = warm-start checkpoint, INK_DS2 / META_DS2 / REP2 = a second set mixed in; env PROJ, RUN (glyphdn1), STEPS (50000), BS (256), LR (3e-4), DN_D (256), DN_GRU (256), DN_PRE (4), DN_POST (3),
26
- PDROP (0.15), PWDROP (0.1), MAXLEN (128), CK_EVERY (2000), DEV) -> ckpt_glyph_dn_<RUN>/{last.pt, fp16/stepNNNNNN.pt, classes.json,
27
- writers.json, lens.npz}, samples_glyph_dn_<RUN>/stepNNNNNN.png (12 classes x 6 real | 6 generated, w01);
28
- MODE=sheet (CKPT, OUT, CLASSES, WRITER, N) -> an eyeball sheet."""
29
- import os, sys, json, math, time, copy, collections, numpy as np, torch, torch.nn as nn, torch.nn.functional as F
30
- PROJ=os.environ.get('PROJ',os.path.dirname(os.path.dirname(os.path.abspath(__file__)))); E=os.environ.get
31
- sys.path.insert(0,os.path.join(PROJ,'scripts'))
32
- os.environ.setdefault('INK_DS','ink_glyphs_v7'); os.environ.setdefault('META_DS','ink_dataset_glyphs_v7')
33
- dev=E('DEV') or ('cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else 'cpu'))
34
- RUN=E('RUN','glyphdn1'); CKPT=os.path.join(PROJ,'ckpt_glyph_dn_'+RUN); SAMP=os.path.join(PROJ,'samples_glyph_dn_'+RUN)
35
- STEPS=int(E('STEPS','50000')); BS=int(E('BS','256')); LR=float(E('LR','3e-4')); WD=float(E('WD','0.01')); WARM=int(E('WARM','500'))
36
- TD=int(E('TD','100')); PDROP=float(E('PDROP','0.15')); PWDROP=float(E('PWDROP','0.1')); SELFCOND=E('SELFCOND','1')=='1'
37
- GSTYLE=E('GSTYLE',''); SEMB_DS=E('SEMB_DS','ink_lines_u2'); SEMB_TAG=E('SEMB_TAG','3all'); SDIM=256
38
- wr=lambda src: src.split('\t')[0]
39
- DN_D=int(E('DN_D','256')); DN_GRU=int(E('DN_GRU','256')); DN_PRE=int(E('DN_PRE','4')); DN_POST=int(E('DN_POST','3')); G_DIM=192
40
- XS=float(E('XS','0.15')); YS=float(E('YS','0.15')); MAXLEN=int(E('MAXLEN','128'))
41
- DELW=float(E('DELW','1.0')); PENW=float(E('PENW','3.0')); POSW=float(E('POSW','1.0')); PRW=float(E('PRW','0.5'))
42
- AMP=E('AMP','1')=='1' and dev=='cuda'; CK_EVERY=int(E('CK_EVERY','2000')); LOG_EVERY=int(E('LOG_EVERY','100')); LEN_MIN=int(E('LEN_MIN','5'))
43
- if dev=='cuda': torch.backends.cuda.matmul.allow_tf32=True; torch.backends.cudnn.allow_tf32=True
44
-
45
- # ---------------- diffusion schedule (cosine, discrete T=TD) — as train_ink_denoise.alpha_bar
46
- def alpha_bar(Tn=TD,s=0.008):
47
- t=np.arange(1,Tn+1)/Tn; f=np.cos((t+s)/(1+s)*math.pi/2)**2/math.cos(s/(1+s)*math.pi/2)**2; return np.clip(f,1e-4,0.9999).astype(np.float32)
48
- AB=torch.tensor(alpha_bar(),device=dev)
49
-
50
- # ---------------- data
51
- def load_data(split):
52
- import train_glyph_vae as TV
53
- TV.MAXLEN=10**6 # we cap ourselves
54
- out=[]
55
- if GSTYLE: # = TV.load_split, plus the doc of each record (load_split drops 1-point records, so its index is not the meta row)
56
- z=np.load(os.path.join(TV.DS,split+'_raw.npz'),allow_pickle=True); off=z['pt_off']; pts=z['pts']; pres=z['pres']; labs=list(z['labels'])
57
- meta=[json.loads(l) for l in open(os.path.join(TV.META,'meta_'+split+'.jsonl'))]; assert len(meta)==len(labs)
58
- for i in range(len(labs)):
59
- assert meta[i]['label']==labs[i]; d=np.concatenate([pts[off[i]:off[i+1]],pres[off[i]:off[i+1],None]],1).astype(np.float32)
60
- if 2<=len(d)<=MAXLEN: out.append((d,labs[i],meta[i]['src']+'\t'+meta[i].get('doc','')))
61
- return out
62
- for d,lab,src in TV.load_split(split):
63
- if len(d)>MAXLEN or len(d)<2: continue
64
- out.append((d,lab,src))
65
- return out
66
- def load_ds(split,ds,meta): # 06.11: load_data on another INK_DS / META_DS
67
- import train_glyph_vae as TV; o=(TV.DS,TV.META); TV.DS=os.path.join(PROJ,'data',ds); TV.META=os.path.join(PROJ,'data',meta)
68
- try: return load_data(split)
69
- finally: TV.DS,TV.META=o
70
- def to_x0(d):
71
- """d [n,4] dx,dy,pen_up,pressure (norm_h units) -> x0 [n,4] (zx, zy, pen±1, 2p-1) and raw positions P [n,2]"""
72
- P=np.cumsum(d[:,:2],0); c=(P.min(0)+P.max(0))/2; Q=P-c
73
- return np.stack([Q[:,0]/XS,Q[:,1]/YS,d[:,2]*2-1,np.clip(d[:,3],0,1)*2-1],1).astype(np.float32),P.astype(np.float32)
74
- _BANK=None
75
- def style_bank():
76
- """GSTYLE=semb: writer -> (doc array, embedding rows) of its u2 train lines"""
77
- global _BANK
78
- if _BANK is None:
79
- Em=np.load(os.path.join(PROJ,'data',SEMB_DS,f'semb_{SEMB_TAG}_train.npy')).astype(np.float32); M=[json.loads(l) for l in open(os.path.join(PROJ,'data',SEMB_DS,'meta_train.jsonl'))]
80
- by=collections.defaultdict(list)
81
- for i,m in enumerate(M): by[m.get('writer_id')].append(i)
82
- _BANK={w:(np.array([M[i]['doc'] for i in ix]),Em[ix]) for w,ix in by.items()}
83
- return _BANK
84
- def style_rows(srcs,rng):
85
- """GSTYLE=semb: per item a [SDIM+1] row = embedding of a random line of its writer from another doc + null flag 0 (unknown writer -> null)"""
86
- bk=style_bank(); S=np.zeros((len(srcs),SDIM+1),np.float32)
87
- for b,src in enumerate(srcs):
88
- w,_,doc=src.partition('\t')
89
- if w not in bk: S[b,-1]=1; continue
90
- dd,em=bk[w]; ok=np.where(dd!=doc)[0]; ok=ok if len(ok) else np.arange(len(dd)); S[b,:SDIM]=em[ok[rng.integers(len(ok))]]
91
- return S
92
- def make_batch(items,c2i,w2i,aug=False,rng=None):
93
- B=len(items); Tn=max(len(d) for d,_,_ in items); Z=np.zeros((B,Tn,4),np.float32); Mk=np.zeros((B,Tn),np.float32); C=np.zeros(B,np.int64); W=np.zeros(B,np.int64)
94
- if GSTYLE: W=style_rows([src for _,_,src in items],rng if rng is not None else np.random.default_rng(1))
95
- for b,(d,lab,src) in enumerate(items):
96
- q=d.copy()
97
- if aug:
98
- sx,sy=1+rng.uniform(-1,1,2)*float(E('AUG_S','0.08')); sh=rng.uniform(-1,1)*float(E('AUG_SH','0.1'))
99
- dx=q[:,0]*sx+q[:,1]*sh; dy=q[:,1]*sy; q[:,0]=dx; q[:,1]=dy
100
- z,_=to_x0(q); n=len(z); Z[b,:n]=z; Mk[b,:n]=1; C[b]=c2i[lab]
101
- if not GSTYLE: W[b]=w2i.get(src,len(w2i))
102
- t=lambda a: torch.from_numpy(a).to(dev)
103
- return t(Z),t(Mk),t(C),t(W)
104
- def buckets(data,bs,rng):
105
- idx=np.argsort([len(d) for d,_,_ in data]+rng.uniform(0,3,len(data))); bl=[idx[i:i+bs] for i in range(0,len(idx),bs)]; rng.shuffle(bl); return bl
106
-
107
- # ---------------- denoiser
108
- def tembed(t,dim=128):
109
- h=dim//2; fr=torch.exp(-math.log(10000)*torch.arange(h,device=t.device)/h); a=t.float()[:,None]*fr[None]*(1000.0/TD); return torch.cat([a.sin(),a.cos()],-1)
110
- class FiLM(nn.Module):
111
- def __init__(s,G,D): super().__init__(); s.l=nn.Linear(G,2*D); nn.init.zeros_(s.l.weight); nn.init.zeros_(s.l.bias)
112
- def forward(s,x,g): a,b=s.l(g)[:,None].chunk(2,-1); return x*(1+a)+b
113
- class CNX(nn.Module):
114
- def __init__(s,D,G,dil):
115
- super().__init__(); s.dw=nn.Conv1d(D,D,7,padding=3*dil,dilation=dil,groups=D); s.ln=nn.LayerNorm(D); s.film=FiLM(G,D); s.f1=nn.Linear(D,4*D); s.f2=nn.Linear(4*D,D); s.gam=nn.Parameter(torch.full((D,),0.1))
116
- def forward(s,x,m,g):
117
- h=s.dw((x*m).transpose(1,2)).transpose(1,2); h=s.film(s.ln(h),g); h=s.f2(F.gelu(s.f1(h))); return (x+s.gam*h)*m
118
- class Denoiser(nn.Module):
119
- def __init__(s,ncls,nwr):
120
- super().__init__(); D=DN_D; G=G_DIM; s.ncls=ncls; s.nwr=nwr
121
- s.ec=nn.Embedding(ncls+1,D); s.gc=nn.Embedding(ncls+1,G)
122
- if GSTYLE: s.ew=nn.Linear(SDIM+1,D); s.gw=nn.Linear(SDIM+1,G) # W = [embedding, null flag]
123
- else: s.ew=nn.Embedding(nwr+1,D); s.gw=nn.Embedding(nwr+1,G)
124
- s.inp=nn.Linear(4+4+5,D); s.tm=nn.Sequential(nn.Linear(128,G),nn.SiLU(),nn.Linear(G,G))
125
- dil=[1,2,4,1,2,4,8]
126
- s.pre=nn.ModuleList([CNX(D,G,dil[k%len(dil)]) for k in range(DN_PRE)]); s.gln=nn.LayerNorm(D); s.gfilm=FiLM(G,D)
127
- s.gru=nn.GRU(D,DN_GRU,num_layers=2,batch_first=True,bidirectional=True); s.gout=nn.Linear(2*DN_GRU,D)
128
- s.post=nn.ModuleList([CNX(D,G,dil[k%len(dil)]) for k in range(DN_POST)]); s.oln=nn.LayerNorm(D); s.head=nn.Linear(D,4)
129
- def forward(s,zt,t,C,W,Mk,sc):
130
- """zt [B,T,4] noisy x0; t [B]; C [B] class (ncls = NULL); W [B] writer (nwr = null); Mk [B,T]; sc [B,T,4] self-cond (zeros = none)
131
- -> x0 estimate [B,T,3] (zx, zy, pressure) and pen logit [B,T]"""
132
- m=Mk[...,None]; Bn,Tn=Mk.shape; L=Mk.sum(1).clamp(min=1)
133
- i=torch.arange(Tn,device=zt.device,dtype=torch.float32)[None].expand(Bn,-1); pc=(i/L[:,None])[...,None]
134
- feats=torch.cat([zt,sc,pc,torch.sin(2*math.pi*pc),torch.cos(2*math.pi*pc),torch.log(L)[:,None,None].expand(-1,Tn,-1)/4.0,i[...,None]/100.0],-1)
135
- x=(s.inp(feats)+s.ec(C)[:,None]+s.ew(W)[:,None])*m
136
- g=F.silu(s.tm(tembed(t))+s.gc(C)+s.gw(W))
137
- for b in s.pre: x=b(x,m,g)
138
- h=s.gfilm(s.gln(x),g); Ll=L.long()
139
- with torch.autocast(dev if dev!='mps' else 'cpu',enabled=False):
140
- p=nn.utils.rnn.pack_padded_sequence(h.float(),Ll.cpu(),batch_first=True,enforce_sorted=False); o,_=s.gru(p); o,_=nn.utils.rnn.pad_packed_sequence(o,batch_first=True,total_length=Tn)
141
- x=(x+s.gout(o.to(x.dtype)))*m
142
- for b in s.post: x=b(x,m,g)
143
- o=s.head(s.oln(x)).float(); return o[...,:3]*m,o[...,3]
144
-
145
- # ---------------- sampling (DDIM, x0-prediction, CFG over the class)
146
- @torch.no_grad()
147
- def sample(net,C,W,lens,cfg=1.5,steps=10,seed=None,gen=None,pen_thr=0.0,eta=0.0,noise=1.0):
148
- """C, W: lists of class / writer indices; lens: point counts -> list of [n,4] (x, y, pen_up, pressure) in norm_h units, centred on the box."""
149
- net.eval(); Bn=len(C); Tn=int(max(lens)); Mk=torch.zeros(Bn,Tn,device=dev)
150
- for b,n in enumerate(lens): Mk[b,:int(n)]=1
151
- m=Mk[...,None]; C=torch.tensor(C,device=dev,dtype=torch.long); W=torch.tensor(np.asarray(W),device=dev,dtype=torch.float32 if GSTYLE else torch.long) # GSTYLE: rows [SDIM+1]
152
- if gen is None and seed is not None: gen=torch.Generator(device='cpu').manual_seed(int(seed))
153
- z=(torch.randn(Bn,Tn,4,generator=gen).to(dev) if gen is not None else torch.randn(Bn,Tn,4,device=dev))*m*noise # noise>1: a hotter start (diversity lever)
154
- sc=torch.zeros_like(z); ts=np.linspace(TD-1,0,steps).round().astype(int)
155
- if cfg!=1.0: C2=torch.cat([C,torch.full_like(C,net.ncls)]); W2=torch.cat([W,W]); Mk2=torch.cat([Mk,Mk])
156
- for k,t in enumerate(ts):
157
- tt=torch.full((Bn,),int(t),device=dev,dtype=torch.long)
158
- with torch.autocast('cuda',dtype=torch.bfloat16,enabled=AMP):
159
- if cfg!=1.0:
160
- xy,pl=net(torch.cat([z,z]),torch.cat([tt,tt]),C2,W2,Mk2,torch.cat([sc,sc]) if SELFCOND else torch.zeros(2*Bn,Tn,4,device=dev))
161
- xy=xy[Bn:]+cfg*(xy[:Bn]-xy[Bn:]); pl=pl[Bn:]+cfg*(pl[:Bn]-pl[Bn:])
162
- else: xy,pl=net(z,tt,C,W,Mk,sc if SELFCOND else torch.zeros_like(z))
163
- x0=torch.cat([xy[...,:2].float(),torch.tanh(pl.float()/2)[...,None],xy[...,2:3].float()],-1)*m; sc=x0
164
- if k==len(ts)-1: break
165
- a=AB[t]; a2=AB[ts[k+1]]; eps=(z-a.sqrt()*x0)/(1-a).sqrt()
166
- if eta>0: # stochastic DDIM (Song et al. eta): fresh noise at every step -> within-class diversity (eta 0 = deterministic, the x0 of step 1 almost decides the glyph)
167
- sig=eta*((1-a2)/(1-a)).sqrt()*(1-a/a2).sqrt(); nz=torch.randn(z.shape,generator=gen).to(dev) if gen is not None else torch.randn_like(z)
168
- z=(a2.sqrt()*x0+(1-a2-sig**2).clamp(min=0).sqrt()*eps+sig*nz)*m
169
- else: z=(a2.sqrt()*x0+(1-a2).sqrt()*eps)*m
170
- X=x0.cpu().numpy(); pen=(pl.float()>pen_thr).cpu().numpy(); out=[] # pen_thr: logit threshold; trained with pos_weight PENW -> 0 = calibrated p > 1/(1+PENW), log(PENW) = p > 0.5
171
- for b in range(Bn):
172
- n=int(lens[b]); q=np.zeros((n,4),np.float32); q[:,0]=X[b,:n,0]*XS; q[:,1]=X[b,:n,1]*YS; q[:,2]=pen[b,:n]; q[-1,2]=1; q[:,3]=np.clip((X[b,:n,3]+1)/2,0,1); out.append(q)
173
- net.train(); return out
174
- def to_strokes(q):
175
- """[n,4] x,y,pen_up,pressure -> list of strokes [[x,y,0,p],...] (the glyph_vae.sample_batch format)"""
176
- st=[];cur=[]
177
- for k in range(len(q)):
178
- cur.append([float(q[k,0]),float(q[k,1]),0.0,float(q[k,3])])
179
- if q[k,2]>0.5 or k==len(q)-1: st.append(cur); cur=[]
180
- return [s for s in st if s]
181
-
182
- # ---------------- length plan
183
- def len_table(tr,c2i,w2i):
184
- """lens[(w,c)] = sorted array of train point counts; lens[(-1,c)] = over all writers"""
185
- T=collections.defaultdict(list)
186
- for d,lab,src in tr: T[(w2i.get(wr(src),len(w2i)),c2i[lab])].append(len(d)); T[(-1,c2i[lab])].append(len(d))
187
- return {k:np.array(sorted(v),np.int32) for k,v in T.items()}
188
- def draw_len(lens,w,c,rng):
189
- a=lens.get((w,c)); a=a if a is not None and len(a)>=LEN_MIN else lens.get((-1,c))
190
- if a is None or len(a)==0: return 30
191
- return int(a[int(rng.integers(len(a)))])
192
-
193
- # ---------------- eyeball sheet: real | generated, w01
194
- def cell(st,S=56):
195
- from PIL import Image, ImageDraw
196
- im=Image.new('L',(S,S),255)
197
- if not st or not any(len(s) for s in st): return im
198
- d=ImageDraw.Draw(im); pts=np.array([p[:2] for s in st for p in s]); x0,y0=pts.min(0); x1,y1=pts.max(0); sc=(S-10)/max(x1-x0,y1-y0,1e-6); ox=(S-(x1-x0)*sc)/2; oy=(S-(y1-y0)*sc)/2
199
- for s in st:
200
- q=[((p[0]-x0)*sc+ox,(p[1]-y0)*sc+oy) for p in s]
201
- if len(q)>1: d.line(q,fill=0,width=2)
202
- elif q: d.ellipse([q[0][0]-1.5,q[0][1]-1.5,q[0][0]+1.5,q[0][1]+1.5],fill=0)
203
- return im
204
- def real_strokes(d):
205
- P=np.cumsum(d[:,:2],0); q=np.concatenate([P,d[:,2:3],d[:,3:4]],1); return to_strokes(q)
206
- def sheet(net,data,c2i,w2i,lens,path,classes=None,writer='w01',nreal=6,ngen=6,seed=0,cfg=1.5,steps=10,eta=0.0):
207
- from PIL import Image, ImageDraw, ImageFont
208
- classes=classes or [c for c in ('x','2','=','λ','α','∂','ψ','i','±','θ','∫','Q') if c in c2i]; rng=np.random.default_rng(seed); w=w2i.get(writer,len(w2i))
209
- byc=collections.defaultdict(list)
210
- for d,lab,src in data:
211
- if wr(src)==writer and lab in classes: byc[lab].append(d)
212
- S=56; gap=16; f=ImageFont.truetype(os.path.join(PROJ,'DejaVuSans.ttf'),18); fs=ImageFont.truetype(os.path.join(PROJ,'DejaVuSans.ttf'),12)
213
- im=Image.new('L',(40+(nreal+ngen)*S+gap,28+len(classes)*(S+4)),255); dr=ImageDraw.Draw(im); dr.text((40,6),f'real ({writer})',fill=0,font=fs); dr.text((40+nreal*S+gap,6),f'glyph denoiser cfg {cfg} steps {steps} eta {eta}',fill=0,font=fs)
214
- C=[c2i[c] for c in classes for _ in range(ngen)]; W=[w]*len(C); L=[draw_len(lens,w,ci,rng) for ci in C]
215
- if GSTYLE: W=style_rows([writer+'\t']*len(C),rng) # a random line of the writer per glyph
216
- gl=sample(net,C,W,L,cfg=cfg,steps=steps,seed=seed,eta=eta)
217
- for r,c in enumerate(classes):
218
- y=28+r*(S+4); dr.text((8,y+16),c,fill=0,font=f); R=byc.get(c,[])
219
- for j,i in enumerate(rng.choice(len(R),min(nreal,len(R)),replace=False) if R else []): im.paste(cell(real_strokes(R[i])),(40+j*S,y))
220
- for j in range(ngen): im.paste(cell(to_strokes(gl[r*ngen+j])),(40+nreal*S+gap+j*S,y))
221
- im.save(path); return path
222
-
223
- def save_meta(classes,writers,lens):
224
- os.makedirs(os.path.join(CKPT,'fp16'),exist_ok=True); json.dump(classes,open(os.path.join(CKPT,'classes.json'),'w'),ensure_ascii=False); json.dump(writers,open(os.path.join(CKPT,'writers.json'),'w'),ensure_ascii=False)
225
- np.savez(os.path.join(CKPT,'lens.npz'),keys=np.array([f'{w}|{c}' for (w,c) in lens]),**{f'l{k}':v for k,v in enumerate(lens.values())})
226
- def cfg_dict(): return dict(DN_D=DN_D,DN_GRU=DN_GRU,DN_PRE=DN_PRE,DN_POST=DN_POST,G_DIM=G_DIM,XS=XS,YS=YS,TD=TD,MAXLEN=MAXLEN,SELFCOND=SELFCOND,GSTYLE=GSTYLE,SEMB_DS=SEMB_DS,SEMB_TAG=SEMB_TAG)
227
-
228
- # ---------------- training
229
- def train():
230
- tr=load_data('train'); va=load_data('valid')
231
- if E('INK_DS2'): # 06.11 (Лев: «Дообучи на моей руке модель для формул»): a second glyph set mixed in x REP2 (its valid replaces the valid)
232
- tr2=load_ds('train',E('INK_DS2'),E('META_DS2')); va=load_ds('valid',E('INK_DS2'),E('META_DS2')); tr=tr+tr2*int(E('REP2','20'))
233
- print(f'mix: {len(tr2)} x {E("REP2","20")} glyphs of {E("INK_DS2")}, valid = its {len(va)}',flush=True)
234
- ini=torch.load(E('INIT'),map_location='cpu') if E('INIT') else None # 06.11: warm start from a checkpoint (its classes / writers kept)
235
- if ini: classes=list(ini['classes']); writers=list(ini['writers']); tr=[x for x in tr if x[1] in set(classes)]
236
- else: classes=sorted(set(l for _,l,_ in tr)); writers=sorted(set(wr(s) for _,_,s in tr))
237
- c2i={c:i for i,c in enumerate(classes)}; w2i={w:i for i,w in enumerate(writers)}; va=[x for x in va if x[1] in c2i]; lens=len_table(tr,c2i,w2i)
238
- os.makedirs(SAMP,exist_ok=True); save_meta(classes,writers,lens)
239
- net=Denoiser(len(classes),len(writers)).to(dev)
240
- if ini: net.load_state_dict({k:v.float() for k,v in ini['e'].items()}); print('init from',E('INIT'),'step',ini.get('step'),flush=True)
241
- ema=copy.deepcopy(net).eval()
242
- for p in ema.parameters(): p.requires_grad_(False)
243
- opt=torch.optim.AdamW(net.parameters(),LR,weight_decay=WD,betas=(0.9,0.99)); step=0; last=os.path.join(CKPT,'last.pt')
244
- if os.path.exists(last):
245
- st=torch.load(last,map_location=dev); net.load_state_dict(st['m']); ema.load_state_dict(st['e']); opt.load_state_dict(st['o']); step=st['step']; print('resumed',step,flush=True)
246
- npar=sum(p.numel() for p in net.parameters())
247
- print(f'glyph denoiser [{RUN}] dev={dev} train {len(tr)} valid {len(va)} classes {len(classes)} writers {len(writers)} D {DN_D} GRU {DN_GRU} pre/post {DN_PRE}/{DN_POST} params {npar/1e6:.2f}M steps {STEPS} bs {BS} lr {LR} T {TD} pdrop {PDROP} pwdrop {PWDROP} amp {AMP}',flush=True)
248
- SEED=int(E('SEED','0')); torch.manual_seed(SEED); rng=np.random.default_rng(SEED+step); bl=[]; t0=time.time(); s0=step; acc={}; nbad=0
249
- meta=dict(classes=classes,writers=writers,cfg=cfg_dict(),lens={f'{w}|{c}':v.tolist() for (w,c),v in lens.items()})
250
- while step<STEPS:
251
- lr=LR*min(1.0,(step+1)/WARM)*(0.5*(1+math.cos(math.pi*min(1.0,step/STEPS))))
252
- for g in opt.param_groups: g['lr']=lr
253
- if not bl: bl=buckets(tr,BS,rng)
254
- z0,Mk,C,W=make_batch([tr[i] for i in bl.pop()],c2i,w2i,aug=True,rng=rng); Bn,Tn=Mk.shape; m=Mk[...,None]
255
- C=torch.where(torch.rand(Bn,device=dev)<PDROP,torch.full_like(C,len(classes)),C); dw=torch.rand(Bn,device=dev)<PWDROP
256
- if GSTYLE: W=torch.where(dw[:,None],F.one_hot(torch.full_like(C,SDIM),SDIM+1).float(),W)
257
- else: W=torch.where(dw,torch.full_like(W,len(writers)),W)
258
- t=torch.randint(0,TD,(Bn,),device=dev); a=AB[t][:,None,None]; eps=torch.randn_like(z0); zt=(a.sqrt()*z0+(1-a).sqrt()*eps)*m; sc=torch.zeros_like(zt)
259
- with torch.autocast('cuda',dtype=torch.bfloat16,enabled=AMP):
260
- if SELFCOND and np.random.rand()<0.5:
261
- with torch.no_grad(): xy0,pl0=net(zt,t,C,W,Mk,sc); sc=torch.cat([xy0[...,:2].float(),torch.tanh(pl0.float()/2)[...,None],xy0[...,2:3].float()],-1).detach()*m
262
- xy,pl=net(zt,t,C,W,Mk,sc)
263
- xy=xy.float(); pl=pl.float(); nm=Mk.sum()
264
- l_pos=((xy[...,:2]-z0[...,:2]).abs().sum(-1)*Mk).sum()/nm
265
- dm=Mk[:,1:]*Mk[:,:-1]; sc_=torch.tensor([XS,YS],device=dev); dP=(xy[:,1:,:2]-xy[:,:-1,:2])*sc_*20; dT=(z0[:,1:,:2]-z0[:,:-1,:2])*sc_*20
266
- l_del=((dP-dT).abs().sum(-1)*dm).sum()/dm.sum().clamp(min=1)
267
- l_pen=(F.binary_cross_entropy_with_logits(pl,(z0[...,2]>0).float(),reduction='none',pos_weight=torch.tensor(PENW,device=dev))*Mk).sum()/nm
268
- l_pr=((xy[...,2]-z0[...,3]).abs()*Mk).sum()/nm
269
- loss=POSW*l_pos+DELW*l_del+l_pen+PRW*l_pr
270
- if not torch.isfinite(loss):
271
- nbad+=1; print(f' non-finite loss at step {step} (#{nbad})',flush=True)
272
- if nbad>=50: print('GLYPHDN ABORT: persistent NaN',flush=True); raise SystemExit(3)
273
- continue
274
- opt.zero_grad(set_to_none=True); loss.backward(); gn=torch.nn.utils.clip_grad_norm_(net.parameters(),1.0)
275
- if not torch.isfinite(gn): opt.zero_grad(set_to_none=True); continue
276
- opt.step(); step+=1; dec=min(0.999,(1+step)/(10+step))
277
- with torch.no_grad():
278
- for pe,pm in zip(ema.parameters(),net.parameters()): pe.mul_(dec).add_(pm.detach(),alpha=1-dec)
279
- for k_,v_ in (('pos',l_pos),('del',l_del),('pen',l_pen),('pr',l_pr)): acc[k_]=acc.get(k_,0)+float(v_.detach())
280
- if step%LOG_EVERY==0:
281
- sp=(step-s0)/(time.time()-t0); msg=' '.join(f'{k_} {v_/LOG_EVERY:.4f}' for k_,v_ in acc.items()); acc={}
282
- print(f'step {step}/{STEPS} {msg} gn {float(gn):.2f} lr {lr:.2e} {sp:.2f}it/s eta {(STEPS-step)/max(sp,1e-6)/60:.0f}min',flush=True)
283
- open(os.path.join(CKPT,'loss.csv'),'a').write(f'{step},{msg}\n')
284
- if step%CK_EVERY==0 or step==STEPS:
285
- vl=valid_loss(ema,va,c2i,w2i); print(f' valid step {step}: pos {vl[0]:.4f} del {vl[1]:.4f} pen {vl[2]:.4f}',flush=True)
286
- tmp=last+'.tmp'; torch.save(dict(m=net.state_dict(),e=ema.state_dict(),o=opt.state_dict(),step=step,**meta),tmp); os.replace(tmp,last)
287
- torch.save(dict(e={k:v.half() for k,v in ema.state_dict().items()},step=step,**meta),os.path.join(CKPT,'fp16',f'step{step:06d}.pt'))
288
- try: sheet(ema,va+tr[:20000],c2i,w2i,lens,os.path.join(SAMP,f'step{step:06d}.png')); ema.eval()
289
- except Exception as e: print('sheet failed',repr(e),flush=True)
290
- print(' ckpt+sheet',step,flush=True)
291
- print('GLYPHDN DONE',flush=True)
292
- @torch.no_grad()
293
- def valid_loss(net,va,c2i,w2i,n=1024):
294
- net.eval(); rng=np.random.default_rng(1); ids=rng.choice(len(va),min(n,len(va)),replace=False); out=[]
295
- for b in range(0,len(ids),256):
296
- z0,Mk,C,W=make_batch([va[i] for i in ids[b:b+256]],c2i,w2i); Bn=Mk.size(0); m=Mk[...,None]
297
- t=torch.full((Bn,),TD//4,device=dev,dtype=torch.long); a=AB[t][:,None,None]; zt=(a.sqrt()*z0+(1-a).sqrt()*torch.randn_like(z0))*m
298
- with torch.autocast('cuda',dtype=torch.bfloat16,enabled=AMP): xy,pl=net(zt,t,C,W,Mk,torch.zeros_like(zt))
299
- xy=xy.float(); pl=pl.float(); nm=Mk.sum(); sc_=torch.tensor([XS,YS],device=dev); dm=Mk[:,1:]*Mk[:,:-1]
300
- out.append((float(((xy[...,:2]-z0[...,:2]).abs().sum(-1)*Mk).sum()/nm),float((((xy[:,1:,:2]-xy[:,:-1,:2])*sc_*20-(z0[:,1:,:2]-z0[:,:-1,:2])*sc_*20).abs().sum(-1)*dm).sum()/dm.sum()),float((F.binary_cross_entropy_with_logits(pl,(z0[...,2]>0).float(),reduction='none')*Mk).sum()/nm)))
301
- net.train(); return np.mean(out,0)
302
-
303
- def load_net(path):
304
- st=torch.load(path,map_location=dev); cfg=st['cfg']
305
- for k in ('DN_D','DN_GRU','DN_PRE','DN_POST','G_DIM','XS','YS','TD','MAXLEN','SELFCOND'): globals()[k]=cfg[k]
306
- for k in ('GSTYLE','SEMB_DS','SEMB_TAG'): globals()[k]=cfg.get(k,'') # old checkpoints: writer ids
307
- globals()['AB']=torch.tensor(alpha_bar(),device=dev)
308
- net=Denoiser(len(st['classes']),len(st['writers'])).to(dev); net.load_state_dict({k:v.float() for k,v in st['e'].items()}); net.eval()
309
- lens={(int(k.split('|')[0]),int(k.split('|')[1])):np.array(v,np.int32) for k,v in st['lens'].items()}
310
- return net,st['classes'],st['writers'],lens,st.get('step')
311
-
312
- if __name__=='__main__':
313
- MODE=E('MODE','train')
314
- if MODE=='train': train()
315
- elif MODE=='sheet':
316
- net,classes,writers,lens,step=load_net(E('CKPT')); c2i={c:i for i,c in enumerate(classes)}; w2i={w:i for i,w in enumerate(writers)}; data=load_data('valid')+load_data('train')
317
- cl=E('CLASSES','').split(',') if E('CLASSES') else None
318
- print('sheet',sheet(net,data,c2i,w2i,lens,E('OUT',os.path.join(PROJ,'figures',f'glyph_dn_{RUN}_sheet.png')),classes=cl,writer=E('WRITER','w01'),nreal=int(E('N','6')),ngen=int(E('N','6')),seed=int(E('SEED','0')),cfg=float(E('GDN_CFG','1.5')),steps=int(E('GDN_STEPS','10')),eta=float(E('GDN_ETA','0'))),flush=True)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/train_ink.py DELETED
@@ -1,186 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """B2: online handwriting synthesis for formulas (Graves 2013 style, modernized).
3
- Input: LaTeX tokens (from mw_ink_prep). Output: pen trajectory (dx,dy,pen_up) autoregressively.
4
- Model: token encoder (embedding + 2-layer transformer, bidirectional) -> decoder = 3 stacked LSTMs; a Gaussian
5
- MIXTURE ATTENTION WINDOW (K components, monotonic kappa) after LSTM-1 reads the token sequence; output head =
6
- MDN (M bivariate gaussians over (dx,dy)) + Bernoulli pen_up + Bernoulli EOS. Teacher forcing, NLL loss.
7
- Sampling: ancestral with probability bias; renders strokes to PNG with a round pen for eyeballing.
8
- Data: data/mw_ink/{train,valid}_raw.npz + vocab.json. Env: PROJ, STEPS, BS, LR, HID, K, M, MAXPTS, HW_RUN, INIT_FROM."""
9
- import os, sys, json, math, time, glob, copy, numpy as np, torch, torch.nn as nn, torch.nn.functional as F
10
- PROJ=os.environ['PROJ']; D=os.path.join(PROJ,'data',os.environ.get('INK_DS','mw_ink')); RUN=os.environ.get('HW_RUN','ink1'); CKPT=os.path.join(PROJ,'ckpt_'+RUN); SAMP=os.path.join(PROJ,'samples_'+RUN)
11
- os.makedirs(os.path.join(CKPT,'fp16'),exist_ok=True); os.makedirs(SAMP,exist_ok=True)
12
- dev='cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else 'cpu')
13
- STEPS=int(os.environ.get('STEPS','60000')); BS=int(os.environ.get('BS','64')); LR=float(os.environ.get('LR','5e-4'))
14
- WFB=int(os.environ.get('WFB','0')) # 17.10: 1 = exact Graves: LSTM-1 input is [x_t, w_{t-1}] (sequential loop); 0 = old parallel approximation where LSTM-1 never sees the text (cursive text alignment failed: lines1/lines2)
15
- HID=int(os.environ.get('HID','400')); K=int(os.environ.get('K','10')); M=int(os.environ.get('M','20')); MAXPTS=int(os.environ.get('MAXPTS','600')); MAXTOK=int(os.environ.get('MAXTOK','48')); SCALE=float(os.environ.get('SCALE','50.0')); STOP_MARGIN=float(os.environ.get('STOP_MARGIN','1.5')); KSTEP=float(os.environ.get('KSTEP','0.1')); PENW=float(os.environ.get('PENW','5.0')); LAMK=float(os.environ.get('LAMK','0.1')); PTS_PER_TOK=float(os.environ.get('PTS_PER_TOK','60'))
16
- itos=json.load(open(os.path.join(D,'vocab.json'))); stoi={s:i for i,s in enumerate(itos)}; V=len(itos)
17
- PRESW=float(os.environ.get('PRESW','1.0')); PRES=os.environ.get('PRES','0')=='1' # 20.10: pen pressure as an extra target/output (npz 'pres' from ink_lines_prep)
18
- class Inks:
19
- def __init__(s,split):
20
- z=np.load(os.path.join(D,f'{split}_raw.npz'),allow_pickle=True); s.pts=z['pts']; s.off=z['pt_off']; s.toks=[json.loads(t) for t in z['toks']]; s.labels=list(z['labels']); s.pres=z['pres'] if (PRES and 'pres' in z) else None
21
- s.idx=[i for i in range(len(s.toks)) if (s.off[i+1]-s.off[i])<=MAXPTS and len(s.toks[i])<=MAXTOK]
22
- print(f"{split}: {len(s.idx)}/{len(s.toks)} usable",flush=True)
23
- def __len__(s): return len(s.idx)
24
- def get(s,i):
25
- j=s.idx[i]; d=s.pts[s.off[j]:s.off[j+1]]; t=[1]+[stoi.get(x,0) for x in s.toks[j]]+[2]; return d,t
26
- def batch(s,ids):
27
- ds,ts=zip(*[s.get(i) for i in ids]); Tn=max(len(d) for d in ds)+1; Un=max(len(t) for t in ts)
28
- X=np.zeros((len(ids),Tn,3),np.float32); Y=np.zeros((len(ids),Tn,5 if PRES else 4),np.float32); Mk=np.zeros((len(ids),Tn),np.float32); Tk=np.zeros((len(ids),Un),np.int64); Tm=np.zeros((len(ids),Un),np.float32)
29
- for b,(d,t) in enumerate(zip(ds,ts)):
30
- n=len(d); X[b,1:n+1]=d; Y[b,:n,:3]=d; Y[b,n-1,3]=1; Mk[b,:n]=1 # input = shifted (start token = zeros), target = next point; EOS at last point
31
- if PRES and s.pres is not None:
32
- j=s.idx[ids[b]]; pv=s.pres[s.off[j]:s.off[j+1]]
33
- # 20.10 (w03: data without pressure must not break the model): a line whose pressure is constant has NO real
34
- # pressure (GoodNotes/Freenotes: 1.0; old sources: 0.5) -> target -1 = masked out of the pressure loss
35
- Y[b,:n,4]=pv if float(np.ptp(pv[:n]))>1e-3 else -1.0
36
- Tk[b,:len(t)]=t; Tm[b,:len(t)]=1
37
- X[...,:2]*=SCALE; Y[...,:2]*=SCALE # deltas ~0.08 -> ~4: sane MDN gradients (overfit test 14.10)
38
- return [torch.from_numpy(a).to(dev) for a in (X,Y,Mk,Tk,Tm)]
39
- class Model(nn.Module):
40
- def __init__(s):
41
- super().__init__(); s.emb=nn.Embedding(V,256); s.tpos=nn.Parameter(torch.randn(MAXTOK+2,256)*0.02)
42
- s.enc=nn.TransformerEncoder(nn.TransformerEncoderLayer(256,4,512,batch_first=True,norm_first=True),2)
43
- if WFB: s.l1c=nn.LSTMCell(3+256,HID) # sees the previous window (Graves eq. 46-52)
44
- else: s.l1=nn.LSTM(3,HID,batch_first=True)
45
- s.win=nn.Linear(HID,3*K)
46
- s.l2=nn.LSTM(3+256+HID,HID,batch_first=True); s.l3=nn.LSTM(3+256+HID,HID,batch_first=True)
47
- s.out=nn.Linear(3*HID,6*M+(3 if PRES else 2))
48
- def encode(s,tk,tm):
49
- h=s.emb(tk)+s.tpos[:tk.size(1)][None]; return s.enc(h,src_key_padding_mask=(tm<0.5))
50
- def window(s,h1,ctx,tm,kappa_prev):
51
- """h1 [B,T,HID] -> per-step window vectors [B,T,256] with monotone kappa (sequential over T)."""
52
- B,T,_=h1.shape; U=ctx.size(1); u=torch.arange(U,device=h1.device).float()[None,None,:]
53
- p=s.win(h1).view(B,T,3,K); alpha=p[:,:,0].exp(); beta=p[:,:,1].exp(); dk=p[:,:,2].exp() # [B,T,K]
54
- Umax=tm.sum(1)[:,None,None]
55
- kappa=torch.minimum(kappa_prev[:,None,:]+torch.cumsum(dk*KSTEP,1), Umax) # [B,T,K]; window cannot run past the text
56
- phi=(alpha[...,None]*torch.exp(-beta[...,None]*(kappa[...,None]-u[:,:,None,:])**2)).sum(2) # [B,T,U]
57
- phi=phi*tm[:,None,:]; w=phi@ctx # [B,T,256]
58
- return w,kappa[:,-1],phi
59
- def forward(s,X,tk,tm,state=None):
60
- ctx=s.encode(tk,tm); B=X.size(0)
61
- if state is None: st1=st2=st3=None; kap=torch.zeros(B,K,device=X.device); w0=torch.zeros(B,1,256,device=X.device)
62
- else: st1,st2,st3,kap,w0=state
63
- if WFB: # exact Graves: step t of LSTM-1 reads [x_t, w_{t-1}]; window from h1_t; sequential over T
64
- T=X.size(1); U=ctx.size(1); u=torch.arange(U,device=X.device).float()[None,:]; Umax=tm.sum(1)[:,None]
65
- if st1 is None: st1=(torch.zeros(B,HID,device=X.device),torch.zeros(B,HID,device=X.device))
66
- h,c=st1; w=w0[:,0]; hs=[]; ws=[]; phis=[]
67
- for t in range(T):
68
- h,c=s.l1c(torch.cat([X[:,t],w],-1),(h,c))
69
- p=s.win(h).view(B,3,K).clamp(-8,8); alpha=p[:,0].exp(); beta=p[:,1].exp(); dk=p[:,2].exp() # clamp: wfb1 17.10 got inf*0=NaN grads at step 0 (win from a non-WFB ckpt on a fresh LSTMCell)
70
- kap=torch.minimum(kap+dk*KSTEP,Umax)
71
- phi=(alpha[:,:,None]*torch.exp(-beta[:,:,None]*(kap[:,:,None]-u[:,None,:])**2)).sum(1)*tm # [B,U]
72
- w=torch.bmm(phi[:,None,:],ctx)[:,0]; hs.append(h); ws.append(w); phis.append(phi)
73
- h1=torch.stack(hs,1); w=torch.stack(ws,1); phi=torch.stack(phis,1); st1=(h,c)
74
- else:
75
- # old approximation: LSTM-1 sees strokes only; window recomputed from h1 in parallel
76
- h1,st1=s.l1(X,st1)
77
- w,kap,phi=s.window(h1,ctx,tm,kap)
78
- h2,st2=s.l2(torch.cat([X,w,h1],-1),st2); h3,st3=s.l3(torch.cat([X,w,h2],-1),st3)
79
- o=s.out(torch.cat([h1,h2,h3],-1)); return o,(st1,st2,st3,kap,w[:,-1:]),phi
80
- def mdn_nll(o,Y,Mk):
81
- pi=F.log_softmax(o[...,:M],-1); mu=o[...,M:3*M].view(*o.shape[:2],M,2); ls=o[...,3*M:5*M].view(*o.shape[:2],M,2).clamp(-7,5); rho=torch.tanh(o[...,5*M:6*M]).clamp(-0.98,0.98) # |rho|->1 made log1p(-rho^2) -> -inf -> NaN (ink1 run)
82
- pen=o[...,6*M]; eos=o[...,6*M+1]; x=Y[...,None,:2]; z=(x-mu)/ls.exp(); zx,zy=z[...,0],z[...,1]
83
- Z=zx**2+zy**2-2*rho*zx*zy; lp=-Z/(2*(1-rho**2))-ls.sum(-1)-math.log(2*math.pi)-0.5*torch.log1p(-rho**2)
84
- nll=-(torch.logsumexp(pi+lp,-1))+F.binary_cross_entropy_with_logits(pen,Y[...,2],reduction='none',pos_weight=torch.tensor(PENW,device=pen.device))+F.binary_cross_entropy_with_logits(eos,Y[...,3],reduction='none',pos_weight=torch.tensor(PENW,device=pen.device))
85
- if PRES and Y.size(-1)>4: nll=nll+PRESW*F.binary_cross_entropy_with_logits(o[...,6*M+2],Y[...,4].clamp(0,1),reduction='none')*(Y[...,4]>=0).float()
86
- return (nll*Mk).sum()/Mk.sum()
87
- @torch.no_grad()
88
- def sample(model,toks_list,bias=1.0,maxlen=700):
89
- model.eval(); B=len(toks_list); Un=max(len(t) for t in toks_list)+2
90
- Tk=torch.zeros(B,Un,dtype=torch.long,device=dev); Tm=torch.zeros(B,Un,device=dev)
91
- for b,t in enumerate(toks_list): ids=[1]+[stoi.get(x,0) for x in t]+[2]; Tk[b,:len(ids)]=torch.tensor(ids); Tm[b,:len(ids)]=1
92
- x=torch.zeros(B,1,3,device=dev); state=None; out=[]; done=torch.zeros(B,dtype=torch.bool,device=dev); stop_idx=torch.zeros(B,dtype=torch.long,device=dev)
93
- for step in range(maxlen):
94
- o,state,phi=model(x,Tk,Tm,state); o=o[:,-1]
95
- pi=F.softmax(o[:,:M]*(1+bias),-1); mu=o[:,M:3*M].view(B,M,2); ls=(o[:,3*M:5*M].view(B,M,2).clamp(-7,5)-bias); rho=torch.tanh(o[:,5*M:6*M])
96
- k=torch.multinomial(pi,1)[:,0]; m=mu[torch.arange(B),k]; sd=ls[torch.arange(B),k].exp(); r=rho[torch.arange(B),k]
97
- e1=torch.randn(B,device=dev); e2=torch.randn(B,device=dev); dx=m[:,0]+sd[:,0]*e1; dy=m[:,1]+sd[:,1]*(r*e1+torch.sqrt(1-r**2)*e2)
98
- pen=(torch.sigmoid(o[:,6*M])>0.5).float(); eos=torch.sigmoid(o[:,6*M+1])>0.5
99
- U_=Tm.sum(1); kap=state[3].mean(1); eos=eos|((kap>=U_-STOP_MARGIN)&(pen>0.5)) # Graves-style stop: window past the last token and pen up
100
- eos=eos|(torch.full_like(U_,float(step+1))>=(U_-2).clamp(min=1)*PTS_PER_TOK) # runaway guard: hard cap on points per token
101
- x=torch.stack([dx,dy,pen],-1)[:,None]; out.append(x[:,0].clone())
102
- newly=eos&~done; stop_idx[newly]=step+1; done=done|eos
103
- if done.all(): break
104
- model.train(); seq=torch.stack(out,1).cpu().numpy(); seq[:,:,:2]/=SCALE; si=stop_idx.cpu().numpy()
105
- return [seq[b,:int(si[b]) if si[b]>0 else seq.shape[1]] for b in range(B)] # list of per-sample sequences truncated at EOS/stop
106
- def render(seq,S=None,pen_w=None,aa=None):
107
- """Stroke render. Defaults from env RENDER_S (px per unit, 48), RENDER_PEN (2), RENDER_AA (supersampling, 1):
108
- w03 16–17.10 asked for HIGH-RES stroke pictures (they look great) -> for anything shown to him use RENDER_S=160
109
- RENDER_PEN=6 RENDER_AA=2; judges keep the small defaults (CRNN input size)."""
110
- from PIL import Image, ImageDraw
111
- S=S or int(os.environ.get('RENDER_S','48')); pen_w=pen_w or int(os.environ.get('RENDER_PEN','2')); aa=aa or int(os.environ.get('RENDER_AA','1'))
112
- Sa=S*aa; pw=pen_w*aa; m=10*aa
113
- xy=np.cumsum(seq[:,:2],0); pen=seq[:,2]; x0,y0=xy[:,0].min(),xy[:,1].min(); W=int((xy[:,0].max()-x0)*Sa)+2*m; H=int((xy[:,1].max()-y0)*Sa)+2*m
114
- im=Image.new('L',(max(W,40*aa),max(H,40*aa)),255); dr=ImageDraw.Draw(im); prev=None
115
- DOTS=os.environ.get('RENDER_DOTS','0')=='1'
116
- for k,((x,y),p) in enumerate(zip(xy,pen)):
117
- q=(m+(x-x0)*Sa,m+(y-y0)*Sa)
118
- if prev is not None:
119
- dr.line([prev,q],fill=0,width=pw)
120
- if pw>2: dr.ellipse([q[0]-pw/2,q[1]-pw/2,q[0]+pw/2,q[1]+pw/2],fill=0) # round joints at high res
121
- # RENDER_DOTS=1 (21.10): a one-point stroke (a tapped dot, comma or the dot of «!») is drawn as a pen-width dot; off by default so the
122
- # readability judge stays comparable with earlier runs (PDF/sdocx output already pads such strokes, write_page.py)
123
- if DOTS and p>0.5 and prev is None and (k==0 or pen[k-1]>0.5): dr.ellipse([q[0]-pw/2,q[1]-pw/2,q[0]+pw/2,q[1]+pw/2],fill=0)
124
- prev=None if p>0.5 else q
125
- if aa>1: im=im.resize((im.size[0]//aa,im.size[1]//aa),Image.LANCZOS)
126
- return im
127
- def demo(model,va,path,n=8):
128
- from PIL import Image, ImageDraw, ImageFont
129
- ids=list(range(0,len(va),max(1,len(va)//n)))[:n]; toks=[va.get(i)[1] for i in ids]; tl=[[itos[t] for t in tk[1:-1]] for tk in toks]
130
- seqs=sample(model,tl,bias=1.0); ims=[]
131
- f=ImageFont.truetype(os.path.join(PROJ,'DejaVuSans.ttf'),12)
132
- for i,sq in zip(ids,seqs):
133
- real=render(va.get(i)[0]); gen=render(sq)
134
- c=Image.new('L',(max(real.width,gen.width,300),real.height+gen.height+18),255); c.paste(real,(0,0)); c.paste(gen,(0,real.height)); ImageDraw.Draw(c).text((2,real.height+gen.height+2),''.join(va.labels[va.idx[i]])[:70],fill=0,font=f); ims.append(c)
135
- W=max(c.width for c in ims); sheet=Image.new('L',(W,sum(c.height for c in ims)+8*len(ims)),255); y=0
136
- for c in ims: sheet.paste(c,(0,y)); y+=c.height+8
137
- sheet.save(path)
138
- def main():
139
- tr=Inks('train'); va=Inks('valid'); model=Model().to(dev); ema=copy.deepcopy(model).eval()
140
- for p in ema.parameters(): p.requires_grad_(False)
141
- opt=torch.optim.Adam(model.parameters(),LR); step=0; ck=sorted(glob.glob(os.path.join(CKPT,'step*.pt')))
142
- if ck: st=torch.load(ck[-1],map_location=dev); model.load_state_dict(st['m']); ema.load_state_dict(st['e']); opt.load_state_dict(st['o']); step=st['step']; print('resumed',step,flush=True)
143
- elif os.environ.get('INIT_FROM'):
144
- st=torch.load(os.environ['INIT_FROM'],map_location=dev); src=st.get('m',st.get('e',st)); src={k:v.float() for k,v in src.items()}
145
- if WFB and not any(k.startswith('l1c.') for k in src): src={k:v for k,v in src.items() if not k.startswith('win.')} # window head trained on the OLD LSTM-1 + fresh LSTMCell = huge alpha -> exploding recurrent grads (wfb1 NaN, 18.10)
146
- tgt=model.state_dict(); ok={k:v for k,v in src.items() if k in tgt and tgt[k].shape==v.shape}; skip=[k for k in src if k not in ok]
147
- for k in list(skip): # vocab grew (joint set): copy the overlapping embedding rows, keep new rows random
148
- if k in tgt and src[k].dim()==2 and tgt[k].dim()==2 and src[k].shape[1]==tgt[k].shape[1]:
149
- n=min(src[k].shape[0],tgt[k].shape[0]); w=tgt[k].clone(); w[:n]=src[k][:n]; ok[k]=w; skip.remove(k); print(f' partial copy {k}: {n} rows',flush=True)
150
- model.load_state_dict(ok,strict=False); ema.load_state_dict({k:v.clone() for k,v in ok.items()},strict=False)
151
- print(f'warm-start from {os.path.basename(os.environ["INIT_FROM"])}: loaded {len(ok)} tensors, skipped {len(skip)} (shape/vocab mismatch): {skip[:6]}',flush=True)
152
- print(f"ink[{RUN}] dev={dev} V={V} HID={HID} K={K} M={M} BS={BS} params {sum(p.numel() for p in model.parameters())/1e6:.1f}M target {STEPS}",flush=True)
153
- rng=np.random.default_rng(0); t0=time.time()
154
- while step<STEPS:
155
- lr=LR*min(1.0,(step+1)/500)*(0.5*(1+math.cos(math.pi*min(1.0,step/STEPS))))
156
- for g in opt.param_groups: g['lr']=lr
157
- X,Y,Mk,Tk,Tm=tr.batch(rng.integers(0,len(tr),BS))
158
- o,(_,_,_,kap_end,_),phi=model(X,Tk,Tm); loss=mdn_nll(o,Y,Mk)
159
- U_=Tm.sum(1); loss=loss+LAMK*((kap_end.mean(1)/U_.clamp(min=1)-1.0)**2).mean() # alignment: window should END at the last token
160
- if not torch.isfinite(loss):
161
- nbad=globals().get('_nbad',0)+1; globals()['_nbad']=nbad; print(f' non-finite loss at step {step} (#{nbad}) -> skipped',flush=True)
162
- if nbad>=50: print('INK ABORT: persistent NaN',flush=True); raise SystemExit(3)
163
- continue
164
- opt.zero_grad(); loss.backward(); gn=torch.nn.utils.clip_grad_norm_(model.parameters(),5.0)
165
- if not torch.isfinite(gn):
166
- nbad=globals().get('_nbad',0)+1; globals()['_nbad']=nbad; print(f' non-finite grad at step {step} (#{nbad}) -> skipped',flush=True); opt.zero_grad()
167
- if nbad>=50: print('INK ABORT: persistent NaN grad',flush=True); raise SystemExit(3) # wfb1 17.10 looped forever on NaN grads at step 0
168
- continue
169
- opt.step(); step+=1
170
- with torch.no_grad():
171
- for pe,pm in zip(ema.parameters(),model.parameters()): pe.mul_(0.999).add_(pm,alpha=0.001)
172
- if step%100==0:
173
- sp=step/(time.time()-t0)
174
- with torch.no_grad():
175
- U=Tm.sum(1); kr=(kap_end.mean(1)/U.clamp(min=1)).mean().item() # ~1.0 when the window ends at the last token
176
- pen_acc=(((o[...,6*M]>0)==(Y[...,2]>0.5)).float()*Mk).sum()/Mk.sum(); pu=((Y[...,2]>0.5).float()*Mk).sum()/Mk.sum()
177
- print(f"step {step}/{STEPS} nll {loss.item():.3f} kappa_end/U {kr:.2f} pen_acc {pen_acc.item():.3f} (penup {pu.item():.3f}) lr {lr:.2e} {sp:.2f}it/s eta {(STEPS-step)/sp/60:.0f}min",flush=True)
178
- open(os.path.join(CKPT,'loss.csv'),'a').write(f"{step},{loss.item():.4f}\n")
179
- if step%2000==0 or step==STEPS:
180
- torch.save(dict(m=model.state_dict(),e=ema.state_dict(),o=opt.state_dict(),step=step),os.path.join(CKPT,f'step{step:06d}.pt'))
181
- torch.save(dict(e={k:v.half() for k,v in ema.state_dict().items()},step=step,itos=itos),os.path.join(CKPT,'fp16',f'step{step:06d}_fp16.pt'))
182
- try: demo(ema,va,os.path.join(SAMP,f'step{step:06d}.png'))
183
- except Exception as e: print('demo failed',e,flush=True)
184
- print(' ckpt+demo',step,flush=True)
185
- print("INK DONE",flush=True)
186
- if __name__=='__main__': main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/train_ink_al.py DELETED
@@ -1,119 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """ALIGNED LSTM trajectory model: train_ink's 3-layer LSTM + MDN (which fits stroke dynamics well, nll -3) with the
3
- Graves attention window REPLACED by the CTC forced alignment (scripts/ink_ctc_align.py): every input point carries the
4
- embedding of its character and phase (FastSpeech-style), so nothing has to be aligned in free running. A duration head
5
- (log1p points per char) on the text encoder provides the plan at sampling. cuDNN LSTMs -> fast (parallel over T).
6
- Env: as train_ink (PROJ, INK_DS, HW_RUN, STEPS, BS, LR, HID, M, MAXPTS, MAXTOK, SCALE, PENW) + ALIGN (align_text.npz),
7
- NPH (8), DURW (0.1), DUR_SCALE (1.0). MODEL_PY=train_ink_al for judge/demo."""
8
- import os, sys, json, math, time, glob, copy, numpy as np, torch, torch.nn as nn, torch.nn.functional as F
9
- sys.path.insert(0,os.path.dirname(os.path.abspath(__file__)))
10
- os.environ.setdefault('HW_RUN','lstmal1'); os.environ.setdefault('INK_DS','ink_lines5s'); os.environ.setdefault('MAXPTS','1200'); os.environ.setdefault('MAXTOK','200')
11
- import train_ink as B
12
- from train_ink import * # noqa: Inks(base), mdn_nll, render, itos, stoi, V, HID, M, SCALE, PENW, PTS_PER_TOK, dev, D, PROJ, RUN, CKPT, SAMP, STEPS, BS, LR, MAXPTS, MAXTOK
13
- E=os.environ.get; ALIGN=os.path.join(D,E('ALIGN','align_text.npz')); NPH=int(E('NPH','8')); DURW=float(E('DURW','0.1')); DUR_SCALE=float(E('DUR_SCALE','1.0'))
14
- class Inks(B.Inks):
15
- def __init__(s,split):
16
- super().__init__(split); z=np.load(ALIGN,allow_pickle=True); s.al={int(i):a for i,sp,a in zip(z['ids'],z['split'],z['char_idx']) if sp==split}
17
- s.idx=[i for i in s.idx if i in s.al]; print(f"{split}: {len(s.idx)} aligned lines",flush=True)
18
- def batch(s,ids):
19
- X,Y,Mk,Tk,Tm=super().batch(ids); Bn,Tn=Mk.shape; C=np.zeros((Bn,Tn),np.int64); Ph=np.zeros((Bn,Tn),np.int64); Cn=np.zeros((Bn,Tn),np.int64); Du=np.zeros((Bn,Tk.size(1)),np.float32); Dm=np.zeros((Bn,Tk.size(1)),np.float32)
20
- for b,i in enumerate(ids):
21
- j=s.idx[i]; ci=s.al[j].astype(np.int64); n=len(ci); toks=s.toks[j]; tid=np.array([stoi.get(t,0) for t in toks]+[0])
22
- cnt=np.bincount(ci,minlength=len(toks)); start=np.concatenate([[0],np.cumsum(cnt)[:-1]])
23
- pos=np.arange(n)-start[ci]; ph=np.minimum(NPH-1,(pos*NPH)//np.maximum(cnt[ci],1))
24
- C[b,:n]=tid[ci]; Cn[b,:n]=tid[np.minimum(ci+1,len(toks))]; Ph[b,:n]=ph; Du[b,1:1+len(toks)]=np.log1p(cnt); Dm[b,1:1+len(toks)]=1 # next char vectorised (the per-point Python loop took minutes per batch, 18.10)
25
- return X,Y,Mk,Tk,Tm,torch.from_numpy(C).to(dev),torch.from_numpy(Ph).to(dev),torch.from_numpy(Cn).to(dev),torch.from_numpy(Du).to(dev),torch.from_numpy(Dm).to(dev)
26
- class Model(nn.Module):
27
- def __init__(s):
28
- super().__init__(); s.emb=nn.Embedding(V,256); s.tpos=nn.Parameter(torch.randn(MAXTOK+2,256)*0.02)
29
- s.enc=nn.TransformerEncoder(nn.TransformerEncoderLayer(256,4,512,batch_first=True,norm_first=True),2,enable_nested_tensor=False)
30
- s.dur=nn.Sequential(nn.LayerNorm(256),nn.Linear(256,256),nn.GELU(),nn.Linear(256,1))
31
- s.ec=nn.Embedding(V,256); s.eph=nn.Embedding(NPH,256); s.enext=nn.Embedding(V,256) # current char, phase, NEXT char (connections)
32
- s.l1=nn.LSTM(3+256,HID,batch_first=True); s.l2=nn.LSTM(3+256+HID,HID,batch_first=True); s.l3=nn.LSTM(3+256+HID,HID,batch_first=True); s.out=nn.Linear(3*HID,6*M+(3 if PRES else 2))
33
- def encode(s,tk,tm): h=s.emb(tk)+s.tpos[:tk.size(1)][None]; return s.enc(h,src_key_padding_mask=(tm<0.5))
34
- def cond(s,C,Ph,Cn): return s.ec(C)+s.eph(Ph)+s.enext(Cn)
35
- def forward(s,X,C,Ph,Cn,state=None):
36
- w=s.cond(C,Ph,Cn); st1,st2,st3=state if state is not None else (None,None,None)
37
- h1,st1=s.l1(torch.cat([X,w],-1),st1); h2,st2=s.l2(torch.cat([X,w,h1],-1),st2); h3,st3=s.l3(torch.cat([X,w,h2],-1),st3)
38
- return s.out(torch.cat([h1,h2,h3],-1)),(st1,st2,st3)
39
- def next_char(C):
40
- """per-point id of the NEXT character (0 at the end)."""
41
- Cn=torch.zeros_like(C); Bn,Tn=C.shape
42
- for b in range(Bn):
43
- c=C[b].tolist(); nxt=0
44
- for t in range(Tn-1,-1,-1):
45
- if t<Tn-1 and c[t+1]!=c[t]: nxt=c[t+1]
46
- Cn[b,t]=nxt
47
- return Cn
48
- def expand(tok_ids,durs):
49
- C=[];Ph=[];Cn=[]
50
- for k,(t,d) in enumerate(zip(tok_ids,durs)):
51
- d=max(1,int(d)); nx=tok_ids[k+1] if k+1<len(tok_ids) else 0
52
- for q in range(d): C.append(t); Ph.append(min(NPH-1,(q*NPH)//d)); Cn.append(nx)
53
- return C,Ph,Cn
54
- @torch.no_grad()
55
- def sample(model,toks_list,bias=1.0,maxlen=None):
56
- model.eval(); Bn=len(toks_list); Un=max(len(t) for t in toks_list)+2
57
- Tk=torch.zeros(Bn,Un,dtype=torch.long,device=dev); Tm=torch.zeros(Bn,Un,device=dev)
58
- for b,t in enumerate(toks_list): ids=[1]+[stoi.get(x,0) for x in t]+[2]; Tk[b,:len(ids)]=torch.tensor(ids); Tm[b,:len(ids)]=1
59
- ctx=model.encode(Tk,Tm); durs=(torch.expm1(model.dur(ctx)[...,0]).clamp(min=0)*DUR_SCALE).round().long().cpu().numpy()
60
- plans=[expand([stoi.get(x,0) for x in t],durs[b,1:1+len(t)]) for b,t in enumerate(toks_list)]; Tn=min(maxlen or MAXPTS,max(len(p[0]) for p in plans)); L=[min(Tn,len(p[0])) for p in plans]
61
- C=torch.zeros(Bn,Tn,dtype=torch.long,device=dev); Ph=torch.zeros_like(C); Cn=torch.zeros_like(C)
62
- for b,(c,p,nx) in enumerate(plans): n=L[b]; C[b,:n]=torch.tensor(c[:n]); Ph[b,:n]=torch.tensor(p[:n]); Cn[b,:n]=torch.tensor(nx[:n])
63
- x=torch.zeros(Bn,1,3,device=dev); state=None; out=[]
64
- for step in range(Tn):
65
- o,state=model(x,C[:,step:step+1],Ph[:,step:step+1],Cn[:,step:step+1],state); o=o[:,-1]
66
- pi=F.softmax(o[:,:M]*(1+bias),-1); mu=o[:,M:3*M].view(Bn,M,2); ls=(o[:,3*M:5*M].view(Bn,M,2).clamp(-7,5)-bias); rho=torch.tanh(o[:,5*M:6*M])
67
- k=torch.multinomial(pi,1)[:,0]; m=mu[torch.arange(Bn),k]; sd=ls[torch.arange(Bn),k].exp(); r=rho[torch.arange(Bn),k]
68
- e1=torch.randn(Bn,device=dev); e2=torch.randn(Bn,device=dev); dx=m[:,0]+sd[:,0]*e1; dy=m[:,1]+sd[:,1]*(r*e1+torch.sqrt(1-r**2)*e2)
69
- pen=(torch.sigmoid(o[:,6*M])>0.5).float(); x=torch.stack([dx,dy,pen],-1)[:,None]; out.append(x[:,0].clone())
70
- model.train(); seq=torch.stack(out,1).cpu().numpy(); seq[:,:,:2]/=SCALE
71
- res=[]
72
- for b in range(Bn): q=seq[b,:L[b]].copy(); q[-1,2]=1; res.append(q)
73
- return res
74
- def demo(model,va,path,n=8):
75
- from PIL import Image, ImageDraw, ImageFont
76
- ids=list(range(0,len(va),max(1,len(va)//n)))[:n]; toks=[va.get(i)[1] for i in ids]; tl=[[itos[t] for t in tk[1:-1]] for tk in toks]
77
- seqs=sample(model,tl,bias=1.0); ims=[]; f=ImageFont.truetype(os.path.join(PROJ,'DejaVuSans.ttf'),12)
78
- for i,sq in zip(ids,seqs):
79
- real=render(va.get(i)[0]); gen=render(sq)
80
- c=Image.new('L',(max(real.width,gen.width,300),real.height+gen.height+18),255); c.paste(real,(0,0)); c.paste(gen,(0,real.height)); ImageDraw.Draw(c).text((2,real.height+gen.height+2),''.join(va.labels[va.idx[i]])[:70],fill=0,font=f); ims.append(c)
81
- W=max(c.width for c in ims); sheet=Image.new('L',(W,sum(c.height for c in ims)+8*len(ims)),255); y=0
82
- for c in ims: sheet.paste(c,(0,y)); y+=c.height+8
83
- sheet.save(path)
84
- def main():
85
- tr=Inks('train'); va=Inks('valid'); model=Model().to(dev); ema=copy.deepcopy(model).eval()
86
- for p in ema.parameters(): p.requires_grad_(False)
87
- opt=torch.optim.Adam(model.parameters(),LR); step=0; ck=sorted(glob.glob(os.path.join(CKPT,'step*.pt')))
88
- if ck: st=torch.load(ck[-1],map_location=dev); model.load_state_dict(st['m']); ema.load_state_dict(st['e']); opt.load_state_dict(st['o']); step=st['step']; print('resumed',step,flush=True)
89
- elif os.environ.get('INIT_FROM') and os.path.exists(os.environ['INIT_FROM']):
90
- st=torch.load(os.environ['INIT_FROM'],map_location=dev); src={k:v.float() for k,v in st.get('e',st).items()}; tgt=model.state_dict(); ok={k:v for k,v in src.items() if k in tgt and tgt[k].shape==v.shape}; model.load_state_dict(ok,strict=False); ema.load_state_dict({k:v.clone() for k,v in ok.items()},strict=False); print('warm-start',len(ok),'of',len(tgt),flush=True)
91
- print(f"lstmal[{RUN}] dev={dev} V={V} HID={HID} M={M} NPH={NPH} BS={BS} params {sum(p.numel() for p in model.parameters())/1e6:.1f}M target {STEPS}",flush=True)
92
- rng=np.random.default_rng(0); t0=time.time(); nbad=0
93
- while step<STEPS:
94
- lr=LR*min(1.0,(step+1)/500)*(0.5*(1+math.cos(math.pi*min(1.0,step/STEPS))))
95
- for g in opt.param_groups: g['lr']=lr
96
- X,Y,Mk,Tk,Tm,C,Ph,Cn,Du,Dm=tr.batch(rng.integers(0,len(tr),BS))
97
- o,_=model(X,C,Ph,Cn); ld=model.dur(model.encode(Tk,Tm))[...,0]; loss_tok=mdn_nll(o,Y,Mk); loss_dur=((ld-Du)**2*Dm).sum()/Dm.sum(); loss=loss_tok+DURW*loss_dur
98
- if not torch.isfinite(loss):
99
- nbad+=1; print(f' non-finite loss at step {step} (#{nbad})',flush=True)
100
- if nbad>=50: print('INK ABORT: persistent NaN',flush=True); raise SystemExit(3)
101
- continue
102
- opt.zero_grad(); loss.backward(); gn=torch.nn.utils.clip_grad_norm_(model.parameters(),5.0)
103
- if not torch.isfinite(gn): opt.zero_grad(); continue
104
- opt.step(); step+=1
105
- with torch.no_grad():
106
- for pe,pm in zip(ema.parameters(),model.parameters()): pe.mul_(0.999).add_(pm,alpha=0.001)
107
- if step%100==0:
108
- sp=step/(time.time()-t0)
109
- with torch.no_grad(): pen_acc=(((o[...,6*M]>0)==(Y[...,2]>0.5)).float()*Mk).sum()/Mk.sum()
110
- print(f"step {step}/{STEPS} nll {loss_tok.item():.3f} dur {loss_dur.item():.3f} pen_acc {pen_acc.item():.3f} lr {lr:.2e} {sp:.2f}it/s eta {(STEPS-step)/sp/60:.0f}min",flush=True)
111
- open(os.path.join(CKPT,'loss.csv'),'a').write(f"{step},{loss_tok.item():.4f},{loss_dur.item():.4f}\n")
112
- if step%2000==0 or step==STEPS:
113
- torch.save(dict(m=model.state_dict(),e=ema.state_dict(),o=opt.state_dict(),step=step),os.path.join(CKPT,f'step{step:06d}.pt'))
114
- torch.save(dict(e={k:v.half() for k,v in ema.state_dict().items()},step=step,itos=itos),os.path.join(CKPT,'fp16',f'step{step:06d}_fp16.pt'))
115
- try: demo(ema,va,os.path.join(SAMP,f'step{step:06d}.png'))
116
- except Exception as e: print('demo failed',repr(e),flush=True)
117
- print(' ckpt+demo',step,flush=True)
118
- print("INK DONE",flush=True)
119
- if __name__=='__main__': main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/train_ink_al2.py DELETED
@@ -1,231 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """train_ink_al + STYLE conditioning (w03 18.10: a page must come out in ONE hand, and his hand must be set by a few
3
- of his lines): a style encoder (BiGRU over the points of K reference lines from the same page, mean-pooled -> 128-d)
4
- feeds (a) the per-point conditioning and (b) the initial LSTM states (also fixes garbled line starts: the decoder no
5
- longer starts from an all-zero state). Reference lines are OTHER lines of the same page (fallback: same doc / itself);
6
- at sampling, pass reference strokes (real lines of the target writer) via sample(..., ref=[seq,...]).
7
- Env: as train_ink_al + K (3), SDIM (128), ALIGN (align_all.npz by default), PSTYLE_DROP (0.1: style dropped -> model
8
- also works without a reference). MODEL_PY=train_ink_al2."""
9
- import os, sys, json, math, time, glob, copy, numpy as np, torch, torch.nn as nn, torch.nn.functional as F
10
- sys.path.insert(0,os.path.dirname(os.path.abspath(__file__)))
11
- os.environ.setdefault('HW_RUN','lstmal2'); os.environ.setdefault('INK_DS','ink_lines5s'); os.environ.setdefault('MAXPTS','1200'); os.environ.setdefault('MAXTOK','200'); os.environ.setdefault('ALIGN','align_all.npz')
12
- import train_ink_al as A
13
- from train_ink_al import * # noqa
14
- K=int(os.environ.get('K','3')); SDIM=int(os.environ.get('SDIM','128')); PSTYLE_DROP=float(os.environ.get('PSTYLE_DROP','0.1'))
15
- AUG=int(os.environ.get('AUG','0')); AUG_S=float(os.environ.get('AUG_S','0.1')); AUG_SH=float(os.environ.get('AUG_SH','0.15')); AUG_N=float(os.environ.get('AUG_N','0.0')) # 19.10 (Лев: «аугментируешь?»): affine jitter per line, train only
16
- class Inks(A.Inks):
17
- def __init__(s,split):
18
- super().__init__(split)
19
- # page key per line from the ink_dataset jsonl order is not stored in the npz -> use doc/page from the dataset jsonl
20
- meta=os.path.join(PROJ,'data',os.environ.get('META_DS','ink_dataset_all_sorted'),'meta_'+split+'.jsonl'); s.page=[]; k=0
21
- for l in open(meta): # ink_lines_prep drops a few over-long lines: walk the jsonl in order and match labels to the npz order
22
- r=json.loads(l)
23
- if k<len(s.labels) and r['label']==s.labels[k]: s.page.append((r['src'],r['doc'],r['page'])); k+=1
24
- assert len(s.page)==len(s.labels), (len(s.page),len(s.labels))
25
- s.bypage={}
26
- for i in s.idx: s.bypage.setdefault(s.page[i],[]).append(i)
27
- def refs(s,j,rng):
28
- cands=[q for q in s.bypage[s.page[j]] if q!=j] or [j]; out=[]
29
- for _ in range(K):
30
- q=cands[rng.integers(len(cands))]; d=s.pts[s.off[q]:s.off[q+1]][:MAXPTS].copy(); d[:,:2]*=SCALE; out.append(d)
31
- return out
32
- def batch(s,ids,rng=None,aug=None):
33
- rng=rng or np.random.default_rng(); res=super().batch(ids); Rs=[s.refs(s.idx[i],rng) for i in ids]; L=max(len(d) for rs in Rs for d in rs)
34
- R=np.zeros((len(ids),K,L,3),np.float32); RL=np.zeros((len(ids),K),np.int64)
35
- for b,rs in enumerate(Rs):
36
- for k,d in enumerate(rs): R[b,k,:len(d)]=d; RL[b,k]=len(d)
37
- R=torch.from_numpy(R).to(dev); RL=torch.from_numpy(RL).to(dev)
38
- if aug is None: aug=AUG
39
- if aug: # one affine jitter per line, applied to the deltas of the line (inputs X and targets Y) AND its style refs: scale ±AUG_S, shear ±AUG_SH, point noise AUG_N (scaled units)
40
- X,Y=res[0],res[1]; Bn=X.size(0); sx=1+(torch.rand(Bn,device=dev)*2-1)*AUG_S; sy=1+(torch.rand(Bn,device=dev)*2-1)*AUG_S; sh=(torch.rand(Bn,device=dev)*2-1)*AUG_SH
41
- def tf(d,sx,sy,sh): # d [..., 3]: dx' = sx*dx + sh*dy ; dy' = sy*dy
42
- dx=d[...,0]*sx+d[...,1]*sh; dy=d[...,1]*sy; return torch.stack([dx,dy,d[...,2]],-1)
43
- X=tf(X,sx[:,None],sy[:,None],sh[:,None]); Y=torch.cat([tf(Y[...,:3],sx[:,None],sy[:,None],sh[:,None]),Y[...,3:]],-1)
44
- if AUG_N>0: nz=torch.randn_like(X[...,:2])*AUG_N*(X[...,:2].abs()>0); X=torch.cat([X[...,:2]+nz,X[...,2:]],-1)
45
- R=tf(R,sx[:,None,None],sy[:,None,None],sh[:,None,None]); res=(X,Y,*res[2:])
46
- return (*res,R,RL)
47
- class Model(A.Model):
48
- def __init__(s):
49
- super().__init__(); s.sgru=nn.GRU(3,SDIM,batch_first=True,bidirectional=True); s.sproj=nn.Linear(2*SDIM,256); s.h0=nn.Linear(256,6*HID); s.null_style=nn.Parameter(torch.zeros(256))
50
- def style(s,R,RL):
51
- Bn,Kn,L,_=R.shape; x=R.flatten(0,1); p=nn.utils.rnn.pack_padded_sequence(x,RL.flatten().clamp(min=1).cpu(),batch_first=True,enforce_sorted=False); _,h=s.sgru(p) # h [2,B*K,SDIM]
52
- v=s.sproj(torch.cat([h[0],h[1]],-1)).view(Bn,Kn,256).mean(1); return v # [B,256]
53
- def forward(s,X,C,Ph,Cn,state=None,sv=None):
54
- w=s.cond(C,Ph,Cn)+(sv[:,None,:] if sv is not None else 0)
55
- if state is None and sv is not None:
56
- h=s.h0(sv).view(X.size(0),6,HID).permute(1,0,2).contiguous(); state=((h[0:1],h[1:2]),(h[2:3],h[3:4]),(h[4:5],h[5:6]))
57
- st1,st2,st3=state if state is not None else (None,None,None)
58
- h1,st1=s.l1(torch.cat([X,w],-1),st1); h2,st2=s.l2(torch.cat([X,w,h1],-1),st2); h3,st3=s.l3(torch.cat([X,w,h2],-1),st3)
59
- return s.out(torch.cat([h1,h2,h3],-1)),(st1,st2,st3)
60
- @torch.no_grad()
61
- def style_from_seqs(model,seqs):
62
- """seqs: list of [n,3] (dx,dy,pen) in model units (as returned by sample / stored in npz) -> style vector [1,256]."""
63
- L=max(len(q) for q in seqs); R=np.zeros((1,len(seqs),L,3),np.float32); RL=np.zeros((1,len(seqs)),np.int64)
64
- for k,q in enumerate(seqs): d=np.asarray(q,np.float32)[:MAXPTS].copy(); d[:,:2]*=SCALE; R[0,k,:len(d)]=d; RL[0,k]=len(d)
65
- return model.style(torch.from_numpy(R).to(dev),torch.from_numpy(RL).to(dev))
66
- _DUR_STATS=None
67
- def dur_floor(toks_list,durs):
68
- """DUR_MIN=1 (И78, 21.10; default off): clamp each char's planned points to at least the DUR_MIN_Q quantile (0.25) of the REAL
69
- per-letter point counts (data/<INK_DS>/dur_stats.json from scripts/dur_stats.py, or DUR_STATS=<path>), scaled by the line's
70
- planned mean / real mean so the hand's overall speed is kept. durs [B,Un] (index 1..len(t) = chars) -> new array."""
71
- global _DUR_STATS
72
- if _DUR_STATS is None: _DUR_STATS=json.load(open(os.environ.get('DUR_STATS') or os.path.join(A.D,'dur_stats.json')))['chars']
73
- q=f"{float(os.environ.get('DUR_MIN_Q','0.25')):.2f}"; durs=durs.copy()
74
- for b,t in enumerate(toks_list):
75
- ks=[k for k,c in enumerate(t) if c in _DUR_STATS]
76
- if not ks: continue
77
- r=max(1.0,float(np.mean([durs[b,1+k] for k in ks])))/np.mean([_DUR_STATS[t[k]]['mean'] for k in ks])
78
- for k,c in enumerate(t):
79
- if c in _DUR_STATS: durs[b,1+k]=max(int(durs[b,1+k]),int(round(_DUR_STATS[c]['q'][q]*r)))
80
- return durs
81
- @torch.no_grad()
82
- def sample(model,toks_list,bias=1.0,maxlen=None,ref=None,sv=None,durs_list=None,prime=None,return_plan=False):
83
- """prime: (X [1,n,3] model inputs (scaled deltas, as Inks.batch gives), C, Ph, Cn [1,n]) of a REAL aligned line of the
84
- writer -> the decoder is run over it teacher-forced (Graves' priming) before drawing the target text (fixes line starts)."""
85
- if prime is not None: return _sample_primed(model,toks_list,bias,maxlen,ref,sv,durs_list,prime)
86
- return _sample(model,toks_list,bias,maxlen,ref,sv,durs_list,return_plan)
87
- @torch.no_grad()
88
- def _sample_primed(model,toks_list,bias,maxlen,ref,sv,durs_list,prime):
89
- model.eval(); Bn=len(toks_list); Xp,Cp,Pp,Np=[t.to(dev).expand(Bn,*t.shape[1:]).contiguous() for t in prime]
90
- if sv is None and ref is not None: sv=style_from_seqs(model,ref)
91
- if sv is None: sv=model.null_style[None]
92
- sv=sv.expand(Bn,-1); Un=max(len(t) for t in toks_list)+2
93
- Tk=torch.zeros(Bn,Un,dtype=torch.long,device=dev); Tm=torch.zeros(Bn,Un,device=dev)
94
- for b,t in enumerate(toks_list): ids=[1]+[stoi.get(x,0) for x in t]+[2]; Tk[b,:len(ids)]=torch.tensor(ids); Tm[b,:len(ids)]=1
95
- ctx=model.encode(Tk,Tm); durs=(torch.expm1(model.dur(ctx)[...,0]).clamp(min=0)*DUR_SCALE).round().long().cpu().numpy()
96
- if os.environ.get('DUR_MIN')=='1': durs=dur_floor(toks_list,durs)
97
- plans=[expand([stoi.get(x,0) for x in t],(durs_list[b] if durs_list is not None else durs[b,1:1+len(t)])) for b,t in enumerate(toks_list)]; Tn=min(maxlen or MAXPTS,max(len(p[0]) for p in plans)); L=[min(Tn,len(p[0])) for p in plans]
98
- C=torch.zeros(Bn,Tn,dtype=torch.long,device=dev); Ph=torch.zeros_like(C); Cn=torch.zeros_like(C)
99
- for b,(c,p,nx) in enumerate(plans): n=L[b]; C[b,:n]=torch.tensor(c[:n]); Ph[b,:n]=torch.tensor(p[:n]); Cn[b,:n]=torch.tensor(nx[:n])
100
- _,state=model(Xp,Cp,Pp,Np,None,sv) # teacher-forced pass over the primer line
101
- x=Xp[:,-1:].clone(); x[:,:,2]=1.0; out=[] # continue from the primer's last point with pen up
102
- for step in range(Tn):
103
- o,state=model(x,C[:,step:step+1],Ph[:,step:step+1],Cn[:,step:step+1],state,sv); o=o[:,-1]
104
- pi=F.softmax(o[:,:M]*(1+bias),-1); mu=o[:,M:3*M].view(Bn,M,2); ls=(o[:,3*M:5*M].view(Bn,M,2).clamp(-7,5)-bias); rho=torch.tanh(o[:,5*M:6*M])
105
- k=torch.multinomial(pi,1)[:,0]; m=mu[torch.arange(Bn),k]; sd=ls[torch.arange(Bn),k].exp(); r=rho[torch.arange(Bn),k]
106
- e1=torch.randn(Bn,device=dev); e2=torch.randn(Bn,device=dev); dx=m[:,0]+sd[:,0]*e1; dy=m[:,1]+sd[:,1]*(r*e1+torch.sqrt(1-r**2)*e2)
107
- pen=(torch.sigmoid(o[:,6*M])>0.5).float(); x=torch.stack([dx,dy,pen],-1)[:,None]; out.append(x[:,0].clone())
108
- model.train(); seq=torch.stack(out,1).cpu().numpy(); seq[:,:,:2]/=SCALE; res=[]
109
- for b in range(Bn): q=seq[b,:L[b]].copy(); q[0,:2]=0; q[-1,2]=1; res.append(q) # first delta = the jump from the primer: zero it
110
- return res
111
- PEN_SAMPLE=os.environ.get('PEN_SAMPLE','0')=='1'; PEN_THR=float(os.environ.get('PEN_THR','0.5'))
112
- @torch.no_grad()
113
- def _sample(model,toks_list,bias=1.0,maxlen=None,ref=None,sv=None,durs_list=None,return_plan=False):
114
- """ref: list of reference stroke sequences of the target writer (model units) -> one style for all lines; sv: precomputed style [1,256];
115
- durs_list: optional per-line integer durations (points per char) to use instead of the duration head (oracle diagnostic)."""
116
- model.eval(); Bn=len(toks_list); Un=max(len(t) for t in toks_list)+2
117
- if sv is None and ref is None and os.environ.get('REF_DS'): # judge/demo without explicit refs: style from REF_N lines of data/<REF_DS>/<REF_SPLIT>_raw.npz
118
- z=np.load(os.path.join(PROJ,'data',os.environ['REF_DS'],os.environ.get('REF_SPLIT','valid')+'_raw.npz'),allow_pickle=True); off=z['pt_off']
119
- idx=[int(x) for x in os.environ['REF_IDX'].split(',')] if os.environ.get('REF_IDX') else list(range(min(int(os.environ.get('REF_N','3')),len(off)-1)))
120
- ref=[z['pts'][off[i]:off[i+1]].astype(np.float32) for i in idx]
121
- if sv is None and ref is not None: sv=style_from_seqs(model,ref)
122
- if sv is None: sv=model.null_style[None]
123
- sv=sv.expand(Bn,-1)
124
- Tk=torch.zeros(Bn,Un,dtype=torch.long,device=dev); Tm=torch.zeros(Bn,Un,device=dev)
125
- for b,t in enumerate(toks_list): ids=[1]+[stoi.get(x,0) for x in t]+[2]; Tk[b,:len(ids)]=torch.tensor(ids); Tm[b,:len(ids)]=1
126
- ctx=model.encode(Tk,Tm); durs=(torch.expm1(model.dur(ctx)[...,0]).clamp(min=0)*DUR_SCALE).round().long().cpu().numpy()
127
- if os.environ.get('DUR_MIN')=='1': durs=dur_floor(toks_list,durs)
128
- plans=[expand([stoi.get(x,0) for x in t],(durs_list[b] if durs_list is not None else durs[b,1:1+len(t)])) for b,t in enumerate(toks_list)]; Tn=min(maxlen or MAXPTS,max(len(p[0]) for p in plans)); L=[min(Tn,len(p[0])) for p in plans]
129
- C=torch.zeros(Bn,Tn,dtype=torch.long,device=dev); Ph=torch.zeros_like(C); Cn=torch.zeros_like(C)
130
- for b,(c,p,nx) in enumerate(plans): n=L[b]; C[b,:n]=torch.tensor(c[:n]); Ph[b,:n]=torch.tensor(p[:n]); Cn[b,:n]=torch.tensor(nx[:n])
131
- # 22.10 И101 (all off by default = old behaviour): sampling knobs SAMP_PI_BIAS / SAMP_SD_BIAS (separate bias of the mixture weights and of
132
- # the sigmas; default = bias), SAMP_GREEDY=1 (argmax component), SAMP_TOPP (nucleus over components, 1 = off), SAMP_SDMUL (sigma x, 0 = component mean);
133
- # NEG_GUIDE=A|B|R «negative letter»: a 2nd stream of the same LSTM over the same drawn points with the plan's look-alikes swapped in C and Cn
134
- # (NEG_PAIRS, default ы:п,я:л,ю:н,м:л); on those spans (C or Cn differs) with strength NEG_W: A = log pi <- (1+w)log pi_c - w log pi_n,
135
- # B = A + mu <- mu_c + w(mu_c - mu_n) of the chosen component, R = draw NEG_S points from the positive mixture and resample by (p_c/p_n)^w
136
- # (target ~ p_c^(1+w)/p_n^w); the pen logit is mixed the same way unless NEG_PEN=0.
137
- E_=os.environ.get; NG=E_('NEG_GUIDE',''); GW=float(E_('NEG_W','1.0')); NGPEN=E_('NEG_PEN','1')=='1'; NS=int(E_('NEG_S','32'))
138
- pib=float(E_('SAMP_PI_BIAS',bias)); sdb=float(E_('SAMP_SD_BIAS',bias)); GREEDY=E_('SAMP_GREEDY')=='1'; TOPP=float(E_('SAMP_TOPP','1')); SDMUL=float(E_('SAMP_SDMUL','1'))
139
- if NG:
140
- sub=torch.arange(V,device=dev)
141
- for a_,b_ in (q.split(':') for q in E_('NEG_PAIRS','ы:п,я:л,ю:н,м:л').split(',')):
142
- if a_ in stoi and b_ in stoi: sub[stoi[a_]]=stoi[b_]
143
- Cg=sub[C]; Cng=sub[Cn]; SPN=E_('NEG_SPAN','any'); GM=((Cg!=C) if SPN=='c' else (Cng!=Cn) if SPN=='cn' else ((Cg!=C)|(Cng!=Cn))).float()*GW # NEG_SPAN: where to guide (c = the letter's own plan span, cn = previous letter's tail)
144
- C2=torch.cat([C,Cg]); Ph2=torch.cat([Ph,Ph]); Cn2=torch.cat([Cn,Cng]); sv2=torch.cat([sv,sv])
145
- ar=torch.arange(Bn,device=dev)
146
- def mix(o_): # -> log pi (bias, top-p applied), mu, log sd, rho
147
- lp=F.log_softmax(o_[:,:M],-1)*(1+pib); mu_=o_[:,M:3*M].view(Bn,M,2); ls_=(o_[:,3*M:5*M].view(Bn,M,2).clamp(-7,5)-sdb); return lp,mu_,ls_,torch.tanh(o_[:,5*M:6*M])
148
- def mdn_logp(lp,mu_,ls_,rho_,xy): # xy [Bn,S,2] -> log density [Bn,S] under the mixture
149
- d=(xy[:,:,None,:]-mu_[:,None])/ls_.exp()[:,None]; r_=rho_[:,None]; om=(1-r_**2).clamp(min=1e-6)
150
- lg=-(d[...,0]**2+d[...,1]**2-2*r_*d[...,0]*d[...,1])/(2*om)-0.5*torch.log(om)-ls_.sum(-1)[:,None]-math.log(2*math.pi)
151
- return torch.logsumexp(F.log_softmax(lp,-1)[:,None]+lg,-1)
152
- x=torch.zeros(Bn,1,3,device=dev); state=None; out=[]
153
- for step in range(Tn):
154
- if NG:
155
- o2,state=model(torch.cat([x,x]),C2[:,step:step+1],Ph2[:,step:step+1],Cn2[:,step:step+1],state,sv2); o=o2[:Bn,-1]; on=o2[Bn:,-1]; g=GM[:,step]
156
- else: o,state=model(x,C[:,step:step+1],Ph[:,step:step+1],Cn[:,step:step+1],state,sv); o=o[:,-1]
157
- lp,mu,ls,rho=mix(o)
158
- if NG: lpn,mun,lsn,rhon=mix(on)
159
- if NG in ('A','B'): lp=(1+g[:,None])*lp-g[:,None]*lpn
160
- pi=F.softmax(lp,-1)
161
- if TOPP<1:
162
- ps,ix=pi.sort(-1,descending=True); keep=(ps.cumsum(-1)-ps)<TOPP; pi=torch.zeros_like(pi).scatter(-1,ix,ps*keep); pi=pi/pi.sum(-1,keepdim=True)
163
- if NG=='R':
164
- ks=pi.argmax(-1,keepdim=True).expand(Bn,NS) if GREEDY else torch.multinomial(pi,NS,replacement=True); m=mu[ar[:,None],ks]; sd=ls[ar[:,None],ks].exp()*SDMUL; r=rho[ar[:,None],ks]
165
- e1=torch.randn(Bn,NS,device=dev); e2=torch.randn(Bn,NS,device=dev); cand=torch.stack([m[...,0]+sd[...,0]*e1,m[...,1]+sd[...,1]*(r*e1+torch.sqrt(1-r**2)*e2)],-1)
166
- wgt=g[:,None]*(mdn_logp(torch.log(pi+1e-12),mu,ls,rho,cand)-mdn_logp(lpn,mun,lsn,rhon,cand)); j=torch.multinomial(F.softmax(wgt,-1),1)[:,0]; dx,dy=cand[ar,j,0],cand[ar,j,1]
167
- else:
168
- k=pi.argmax(-1) if GREEDY else torch.multinomial(pi,1)[:,0]; m=mu[ar,k]; sd=ls[ar,k].exp()*SDMUL; r=rho[ar,k]
169
- if NG=='B': m=m+g[:,None]*(m-mun[ar,k])
170
- e1=torch.randn(Bn,device=dev); e2=torch.randn(Bn,device=dev); dx=m[:,0]+sd[:,0]*e1; dy=m[:,1]+sd[:,1]*(r*e1+torch.sqrt(1-r**2)*e2)
171
- zp=o[:,6*M]
172
- if NG and NGPEN: zp=zp+g*(zp-on[:,6*M])
173
- pp=torch.sigmoid(zp); pen=(torch.bernoulli(pp) if PEN_SAMPLE else (pp>PEN_THR).float()); x=torch.stack([dx,dy,pen],-1)[:,None]; out.append(torch.cat([x[:,0],torch.sigmoid(o[:,6*M+2:6*M+3])],-1).clone() if PRES else x[:,0].clone()) # 21.10: PEN_SAMPLE=1 / PEN_THR (default 0.5 = old) for the glyph generator
174
- model.train(); seq=torch.stack(out,1).cpu().numpy(); seq[:,:,:2]/=SCALE; res=[]
175
- for b in range(Bn): q=seq[b,:L[b]].copy(); q[-1,2]=1; res.append(q)
176
- # return_plan: also the per-point conditioning (char, phase, next char) of each line, so another model can be run
177
- # teacher-forced over the sampled trajectory (write_page PRES_MODEL: pressure from a PRES model on grpo3's strokes)
178
- if return_plan: return res,[(c[:L[b]],p[:L[b]],nx[:L[b]]) for b,(c,p,nx) in enumerate(plans)]
179
- return res
180
- def demo(model,va,path,n=8):
181
- """rows: real line | generated with the style of OTHER lines of the same page."""
182
- from PIL import Image, ImageDraw, ImageFont
183
- rng=np.random.default_rng(1); ids=list(range(0,len(va),max(1,len(va)//n)))[:n]; f=ImageFont.truetype(os.path.join(PROJ,'DejaVuSans.ttf'),12); ims=[]
184
- for i in ids:
185
- d,t=va.get(i); refs=[q.copy() for q in va.refs(va.idx[i],rng)]
186
- for q in refs: q[:,:2]/=SCALE
187
- sq=sample(model,[[itos[x] for x in t[1:-1]]],bias=1.0,ref=refs)[0]; real=render(d); gen=render(sq)
188
- c=Image.new('L',(max(real.width,gen.width,300),real.height+gen.height+18),255); c.paste(real,(0,0)); c.paste(gen,(0,real.height)); ImageDraw.Draw(c).text((2,real.height+gen.height+2),''.join(va.labels[va.idx[i]])[:70],fill=0,font=f); ims.append(c)
189
- W=max(c.width for c in ims); sheet=Image.new('L',(W,sum(c.height for c in ims)+8*len(ims)),255); y=0
190
- for c in ims: sheet.paste(c,(0,y)); y+=c.height+8
191
- sheet.save(path)
192
- def main():
193
- tr=Inks('train'); va=Inks('valid'); model=Model().to(dev); ema=copy.deepcopy(model).eval()
194
- for p in ema.parameters(): p.requires_grad_(False)
195
- opt=torch.optim.Adam(model.parameters(),LR); step=0; ck=sorted(glob.glob(os.path.join(CKPT,'step*.pt')))
196
- if ck: st=torch.load(ck[-1],map_location=dev); model.load_state_dict(st['m']); ema.load_state_dict(st['e']); opt.load_state_dict(st['o']); step=st['step']; print('resumed',step,flush=True)
197
- elif os.environ.get('INIT_FROM') and os.path.exists(os.environ['INIT_FROM']):
198
- st=torch.load(os.environ['INIT_FROM'],map_location=dev); src={k:v.float() for k,v in st.get('e',st).items()}; tgt=model.state_dict(); ok={k:v for k,v in src.items() if k in tgt and tgt[k].shape==v.shape}
199
- for k_ in ('out.weight','out.bias'):
200
- if k_ in src and k_ in tgt and tgt[k_].shape!=src[k_].shape and tgt[k_].shape[1:]==src[k_].shape[1:]: t_=tgt[k_].clone(); n_=min(t_.shape[0],src[k_].shape[0]); t_[:n_]=src[k_][:n_]; ok[k_]=t_
201
- model.load_state_dict(ok,strict=False); ema.load_state_dict({k:v.clone() for k,v in ok.items()},strict=False); print('warm-start',len(ok),'of',len(tgt),flush=True)
202
- print(f"lstmal2[{RUN}] dev={dev} V={V} HID={HID} K={K} SDIM={SDIM} BS={BS} params {sum(p.numel() for p in model.parameters())/1e6:.1f}M target {STEPS} align {ALIGN}",flush=True)
203
- rng=np.random.default_rng(0); t0=time.time(); nbad=0
204
- while step<STEPS:
205
- lr=LR*min(1.0,(step+1)/500)*(0.5*(1+math.cos(math.pi*min(1.0,step/STEPS))))
206
- for g in opt.param_groups: g['lr']=lr
207
- X,Y,Mk,Tk,Tm,C,Ph,Cn,Du,Dm,R,RL=tr.batch(rng.integers(0,len(tr),BS),rng)
208
- sv=model.style(R,RL); drop=(torch.rand(X.size(0),device=dev)<PSTYLE_DROP)[:,None]; sv=torch.where(drop,model.null_style[None].expand_as(sv),sv)
209
- o,_=model(X,C,Ph,Cn,None,sv); ld=model.dur(model.encode(Tk,Tm))[...,0]; loss_tok=mdn_nll(o,Y,Mk); loss_dur=((ld-Du)**2*Dm).sum()/Dm.sum(); loss=loss_tok+DURW*loss_dur
210
- if not torch.isfinite(loss):
211
- nbad+=1; print(f' non-finite loss at step {step} (#{nbad})',flush=True)
212
- if nbad>=50: print('INK ABORT: persistent NaN',flush=True); raise SystemExit(3)
213
- continue
214
- opt.zero_grad(); loss.backward(); gn=torch.nn.utils.clip_grad_norm_(model.parameters(),5.0)
215
- if not torch.isfinite(gn): opt.zero_grad(); continue
216
- opt.step(); step+=1
217
- with torch.no_grad():
218
- for pe,pm in zip(ema.parameters(),model.parameters()): pe.mul_(0.999).add_(pm,alpha=0.001)
219
- if step%100==0:
220
- sp=step/(time.time()-t0)
221
- with torch.no_grad(): pen_acc=(((o[...,6*M]>0)==(Y[...,2]>0.5)).float()*Mk).sum()/Mk.sum()
222
- print(f"step {step}/{STEPS} nll {loss_tok.item():.3f} dur {loss_dur.item():.3f} pen_acc {pen_acc.item():.3f} lr {lr:.2e} {sp:.2f}it/s eta {(STEPS-step)/sp/60:.0f}min",flush=True)
223
- open(os.path.join(CKPT,'loss.csv'),'a').write(f"{step},{loss_tok.item():.4f},{loss_dur.item():.4f}\n")
224
- if step%2000==0 or step==STEPS:
225
- torch.save(dict(m=model.state_dict(),e=ema.state_dict(),o=opt.state_dict(),step=step),os.path.join(CKPT,f'step{step:06d}.pt'))
226
- torch.save(dict(e={k:v.half() for k,v in ema.state_dict().items()},step=step,itos=itos),os.path.join(CKPT,'fp16',f'step{step:06d}_fp16.pt'))
227
- try: demo(ema,va,os.path.join(SAMP,f'step{step:06d}.png'))
228
- except Exception as e: print('demo failed',repr(e),flush=True)
229
- print(' ckpt+demo',step,flush=True)
230
- print("INK DONE",flush=True)
231
- if __name__=='__main__': main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/train_ink_denoise.py DELETED
The diff for this file is too large to render. See raw diff
 
scripts/train_planner.py DELETED
@@ -1,108 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """И160 (01.11, Лев: «Давай тогда модель длительностей переобучим, а кодировщик оставим»): планировщик длительностей для B — сколько
3
- точек пера на каждую букву строки. Заменяет голову длительностей lstmal_v6isl3 (22.10, старый набор v6, почти одна w01, видит только
4
- текст: ритм у всех одинаковый).
5
- данные : data/ink_lines_u2 train, разметка align_islands3c_clean.npz (та же, на которой учились A и B): длительность буквы = число её точек
6
- вход : текст (тот же словарь) + вектор образца semb3_all (data/ink_lines_u2/semb_3all_train.npy) ДРУГОЙ строки того же писца, по
7
- возможности с другой страницы — как у B; с вероятностью PSTYLE_DROP образец заменяется «никаким» (обучаемый вектор)
8
- модель : эмбеддинг букв + позиция -> 4 слоя Transformer-энкодера d 256 (вектор образца прибавлен к каждой букве) -> log1p(длительность)
9
- лосс : L1 по log1p(длительности) (медиана, устойчиво к выбросам разметки)
10
- отложено: PAGE_HOLD (3%) страниц каждого писца целиком — по ним мера и выбор шага; там же мерится старая голова
11
- Мера (до прогона): ошибка на букву = медиана |log(pred/true)| (старая голова на u2 train ×1.25–1.50), длина строки = pred/true,
12
- разброс 10–90% (старая голова ±25%) и её медиана по писцам.
13
- python scripts/train_planner.py <out_dir> [STEPS=20000 BS=64 LR=1e-3 MAXMIN=40] -> <out_dir>/best.pt, log.txt (перезаписывать отказывается)"""
14
- import os, sys, json, math, time, copy, numpy as np
15
- P = os.path.dirname(os.path.dirname(os.path.abspath(__file__))); sys.path.insert(0, os.path.join(P, 'scripts'))
16
- import torch, torch.nn as nn, torch.nn.functional as F
17
- E = os.environ.get; OUT = sys.argv[1] if __name__ == "__main__" else None
18
- dev = 'cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else 'cpu')
19
- DS = os.path.join(P, 'data', E('INK_DS', 'ink_lines_u2')); itos = json.load(open(os.path.join(DS, 'vocab.json'))); stoi = {s: i for i, s in enumerate(itos)}
20
- MAXTOK = 200; PSTYLE_DROP = float(E('PSTYLE_DROP', '0.2'))
21
-
22
- class Planner(nn.Module):
23
- def __init__(s, V, d=256, L=4):
24
- super().__init__(); s.emb = nn.Embedding(V, d); s.pos = nn.Embedding(MAXTOK + 2, d); s.sty = nn.Linear(256, d); s.null = nn.Parameter(torch.zeros(256))
25
- s.enc = nn.TransformerEncoder(nn.TransformerEncoderLayer(d, 4, 4 * d, 0.1, batch_first=True, norm_first=True), L); s.out = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 1))
26
- def forward(s, Tk, Tm, Se): # Tk [B,U] ids with <bos>/<eos>; Se [B,256] style (rows = null where no reference) -> log1p durations [B,U]
27
- h = s.emb(Tk) + s.pos(torch.arange(Tk.shape[1], device=Tk.device))[None] + s.sty(Se)[:, None]
28
- return s.out(s.enc(h, src_key_padding_mask=Tm < 0.5))[..., 0]
29
-
30
- def batch_of(toks_list, se, d=None):
31
- U = min(MAXTOK + 2, -(-(max(len(t) for t in toks_list) + 2) // 16) * 16) # pad to a multiple of 16: few distinct shapes for MPS (planner1 slowed 20x after step 1000)
32
- Tk = torch.zeros(len(toks_list), U, dtype=torch.long); Tm = torch.zeros(len(toks_list), U)
33
- for b, t in enumerate(toks_list): ids = [1] + [stoi.get(x, 0) for x in t] + [2]; Tk[b, :len(ids)] = torch.tensor(ids); Tm[b, :len(ids)] = 1
34
- d = d or dev; return Tk.to(d), Tm.to(d), se.to(d)
35
-
36
- def load(path):
37
- st = torch.load(path, map_location=dev); n = Planner(len(st['itos'])).to(dev); n.load_state_dict(st['m']); n.eval(); return n
38
-
39
- @torch.no_grad()
40
- def predict(net, toks_list, se):
41
- """toks_list: list of char lists; se [B,256] semb3_all of the reference (None -> no reference) -> per-char point counts (lists of int)"""
42
- d_ = next(net.parameters()).device; se = net.null[None].expand(len(toks_list), -1) if se is None else se.to(d_).float()
43
- Tk, Tm, Se = batch_of(toks_list, se, d_); d = torch.expm1(net(Tk, Tm, Se)).clamp(min=1).round().long().cpu().numpy()
44
- return [list(d[b, 1:1 + len(t)]) for b, t in enumerate(toks_list)]
45
-
46
- if __name__ == '__main__':
47
- if os.path.exists(OUT): sys.exit(f'{OUT} exists')
48
- os.makedirs(OUT); LOG = open(os.path.join(OUT, 'log.txt'), 'w')
49
- def log(*a): s_ = ' '.join(str(x) for x in a); print(s_, flush=True); LOG.write(s_ + '\n'); LOG.flush()
50
- z = np.load(os.path.join(DS, 'train_raw.npz'), allow_pickle=True); TK = [json.loads(t) for t in z['toks']]
51
- meta = [json.loads(l) for l in open(os.path.join(DS, 'meta_train.jsonl'))]; assert len(meta) == len(TK)
52
- SE = np.load(os.path.join(DS, 'semb_3all_train.npy')).astype(np.float32)
53
- al = np.load(os.path.join(DS, 'align_islands3c_clean.npz'), allow_pickle=True)
54
- DU = {int(i): np.bincount(c.astype(np.int64), minlength=len(TK[int(i)])) for i, s, c in zip(al['ids'], al['split'], al['char_idx']) if s == 'train'}
55
- ok = [i for i in DU if 1 <= len(TK[i]) <= MAXTOK and len(DU[i]) == len(TK[i])]
56
- rng = np.random.default_rng(0); pages = {}
57
- for i in ok: pages.setdefault(meta[i]['writer_id'], {}).setdefault((meta[i]['doc'], meta[i]['page']), []).append(i)
58
- hold = set()
59
- for w, pg in pages.items():
60
- ks = sorted(pg); rng.shuffle(ks); n = max(1, int(round(float(E('PAGE_HOLD', '0.03')) * len(ks))))
61
- for k in ks[:n]: hold.update(pg[k])
62
- tr = [i for i in ok if i not in hold]; va = sorted(hold)
63
- byw = {}; byp = {}
64
- for i in tr: w = meta[i]['writer_id']; byw.setdefault(w, []).append(i); byp[i] = (meta[i]['doc'], meta[i]['page'])
65
- def ref_of(i, r): # another TRAIN line of the same writer, preferably from another page (held-out lines take their reference from train too)
66
- c = byw[meta[i]['writer_id']]; q = i
67
- for _ in range(6):
68
- q = c[r.integers(len(c))]
69
- if q != i and byp.get(q) != (meta[i]['doc'], meta[i]['page']): break
70
- return q
71
- log(f'planner: dev {dev}, {len(ok)} aligned lines, train {len(tr)}, held-out {len(va)} lines ({sum(len(v) for v in pages.values())} pages, {PSTYLE_DROP} style drop)')
72
- log('held-out per writer: ' + ', '.join(f'{w} {sum(1 for i in va if meta[i]["writer_id"] == w)}' for w in sorted(pages)))
73
- vr = np.random.default_rng(1); vref = {i: ref_of(i, vr) for i in va}
74
- def evaluate(fn): # fn(toks_list, idx_list) -> durations; returns letter error (x), length ratio stats, per-writer length medians
75
- lr, tot, W = [], [], {}
76
- for k in range(0, len(va), 128):
77
- ids = va[k:k + 128]; pd = fn([TK[i] for i in ids], ids)
78
- for i, p in zip(ids, pd):
79
- p = np.maximum(np.array(p, float), 1); t = np.maximum(DU[i].astype(float), 1); lr.append(np.abs(np.log(p / t))); r = p.sum() / t.sum(); tot.append(r); W.setdefault(meta[i]['writer_id'], []).append(r)
80
- lr = np.concatenate(lr); tot = np.array(tot)
81
- return dict(letter=float(np.exp(np.median(lr))), len_med=float(np.median(tot)), len10=float(np.percentile(tot, 10)), len90=float(np.percentile(tot, 90)),
82
- len_abs=float(np.median(np.abs(np.log(tot)))), per_w={w: round(float(np.median(v)), 3) for w, v in W.items()})
83
- fmt = lambda r: f"letter x{r['letter']:.3f} | length med {r['len_med']:.3f} 10-90% {r['len10']:.2f}-{r['len90']:.2f} |log| {r['len_abs']:.3f} | " + ' '.join(f'{w} {v:.2f}' for w, v in sorted(r['per_w'].items()))
84
- if E('OLD_HEAD', '1') == '1': # the old head on the same held-out lines (text only)
85
- os.environ.update(PROJ=P, INK_DS='ink_lines_u2', META_DS='ink_lines_u2', HW_RUN='planner_cmp'); import train_ink_denoise as D; base = D.load_base()
86
- log('OLD HEAD ' + fmt(evaluate(lambda tl, ids: D.head_durs(base, tl)))); del base
87
- net = Planner(len(itos)).to(dev); STEPS = int(E('STEPS', '20000')); BS = int(E('BS', '64')); LR = float(E('LR', '1e-3')); WARM = 500
88
- opt = torch.optim.AdamW(net.parameters(), LR, weight_decay=0.01); best = 9.0; t0 = time.time(); r = np.random.default_rng(2); hist = []
89
- log(f'params {sum(p.numel() for p in net.parameters()) / 1e6:.2f}M, steps {STEPS}, BS {BS}, LR {LR}')
90
- for step in range(1, STEPS + 1):
91
- lr = LR * min(1.0, step / WARM) * (0.05 + 0.95 * 0.5 * (1 + math.cos(math.pi * step / STEPS)))
92
- for g in opt.param_groups: g['lr'] = lr
93
- ids = r.choice(tr, BS); se = torch.from_numpy(SE[[ref_of(i, r) for i in ids]])
94
- Tk, Tm, Se = batch_of([TK[i] for i in ids], se); drop = (torch.rand(BS, device=dev) < PSTYLE_DROP)[:, None]; Se = torch.where(drop, net.null[None].expand_as(Se), Se)
95
- Y = np.zeros(Tm.shape, np.float32); M = np.zeros(Tm.shape, np.float32) # built on CPU, one copy: per-row slice writes into an MPS tensor compile a graph per (row length, U) and stalled planner1-3 for minutes
96
- for b, i in enumerate(ids): Y[b, 1:1 + len(DU[i])] = np.log1p(DU[i].astype(np.float32)); M[b, 1:1 + len(DU[i])] = 1
97
- Y = torch.from_numpy(Y).to(dev); M = torch.from_numpy(M).to(dev)
98
- net.train(); loss = ((net(Tk, Tm, Se) - Y).abs() * M).sum() / M.sum(); opt.zero_grad(); loss.backward(); nn.utils.clip_grad_norm_(net.parameters(), 1.0); opt.step(); hist.append(loss.item())
99
- stop = (time.time() - t0) / 60 > float(E('MAXMIN', '40'))
100
- if step % 100 == 0: log(f' {step}/{STEPS} loss {np.mean(hist[-100:]):.4f} {time.time() - t0:.0f}s')
101
- if step % 1000 == 0 or step == STEPS or stop:
102
- cn = copy.deepcopy(net).cpu().eval() # held-out check on a CPU copy: planner1-4 stalled for minutes on MPS right after the eval
103
- rs = evaluate(lambda tl, ids_: predict(cn, tl, torch.from_numpy(SE[[vref[i] for i in ids_]])))
104
- rn = evaluate(lambda tl, ids_: predict(cn, tl, None)); del cn
105
- log(f'step {step} loss {np.mean(hist[-1000:]):.4f} {time.time() - t0:.0f}s\n with ref ' + fmt(rs) + '\n no ref ' + fmt(rn))
106
- if rs['letter'] < best: best = rs['letter']; torch.save(dict(m=net.state_dict(), itos=itos, step=step, held=rs, held_noref=rn), os.path.join(OUT, 'best.pt'))
107
- if stop: log(f'MAXMIN reached at step {step}'); break
108
- log(f'PLANNER DONE best letter x{best:.3f} -> {OUT}/best.pt')
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/write_pages.py DELETED
@@ -1,118 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- """06.11 (Лев: «подготовь в отдельной папке полный продукт для генерации страниц с формулами и текстом моей рукой»): a text file ->
3
- notebook pages in the hand of ONE reference line.
4
- input : UTF-8 text, one line of the notes per line; $...$ = a formula (LaTeX); an empty line = a paragraph gap;
5
- a line starting with '# ' = a heading (written the same way, with extra space above). Long lines are wrapped at WRAP
6
- characters (never inside $...$).
7
- prose (no $) -> B (b4styc@3000 + planner7, CFG 1.5, 20 steps), style from the reference line
8
- formula (whole $..$) -> formula_line.compose_em: MathJax layout + glyph denoiser (GDN_CKPT; default the general u2s),
9
- scaled so its x-height = the page's x-height
10
- mixed (text + $..$) -> B's line with every $...$ replaced by the module (formula_line.mixed_line)
11
- the module fails -> B writes the line itself (counted in report.json)
12
- Page: A4 proportions, lines on a baseline grid (pitch >= 3 x-heights, more when a line is taller), x-height from B's text.
13
- python scripts/write_pages.py <in.txt> <out_dir> [REF=examples/ref_hwr200.npy WRAP=42 LINES_PER_PAGE=99 SEED=0 KEY=w03]
14
- -> <out_dir>/pages.pdf (VECTOR), page_NN.svg, page_NN.png (300 dpi, antialiased), strokes.json (re-render with render_pages.py),
15
- report.json (per line: kind, seconds, fallbacks, glyph sources). Refuses to overwrite."""
16
- import os, sys, json, re, time, collections, numpy as np
17
- P = os.path.dirname(os.path.dirname(os.path.abspath(__file__))); sys.path.insert(0, os.path.join(P, 'scripts')); E = os.environ.get
18
- IN, OUT = sys.argv[1], sys.argv[2]
19
- if os.path.exists(OUT): sys.exit(f'{OUT} exists')
20
- if os.path.exists(os.path.join(P, 'defaults.json')): # a packaged release keeps its default models and paths here (the env still wins)
21
- for _k, _v in json.load(open(os.path.join(P, 'defaults.json'))).items(): os.environ.setdefault(_k, _v if os.path.isabs(_v) or not _k.endswith(('CKPT', 'FW_NS')) else os.path.join(P, _v))
22
- os.environ.setdefault('CFG', '1.5'); os.environ.setdefault('SSTEPS', '20')
23
- os.environ.setdefault('GDN_CKPT', os.path.join(P, 'ckpt_glyph_dn_u2s', 'fp16', 'step050000.pt'))
24
- import torch, formula_line as FL, formula_write as FW, glyph_dn as GD, train_ink_denoise as D
25
-
26
- # ---------------- text -> lines
27
- WRAP = int(E('WRAP', '42')); KEY = E('KEY', 'ref')
28
- def wrap(s):
29
- parts = re.split(r'(\$[^$]*\$)', s); out, cur = [], ''
30
- for p in parts:
31
- for w in ([p] if p.startswith('$') else re.split(r'(?<= )', p)):
32
- if not w: continue
33
- if len(cur) + len(w) > WRAP and cur.strip(): out.append(cur.rstrip()); cur = w.lstrip()
34
- else: cur += w
35
- if cur.strip(): out.append(cur.rstrip())
36
- return out
37
- items = [] # (text or None for a gap, heading?)
38
- for raw in open(IN, encoding='utf-8').read().splitlines():
39
- s = raw.strip()
40
- if not s: items.append((None, False)); continue
41
- head = s.startswith('# '); s = s[2:].strip() if head else s
42
- if head: items.append((None, False))
43
- for l in wrap(s):
44
- m = re.fullmatch(r'\s*\$([^$]+)\$\s*([.,:;!?]*)\s*', l) # a wrapped tail «$...$:» -> the formula with its punctuation inside
45
- items.append(('$' + m.group(1) + m.group(2) + '$' if m else l, head)) # (as text the module refused it: no letters for the x-height)
46
- def kind(lab):
47
- if '$' not in lab: return 'prose'
48
- s = lab.strip(); return 'formula' if s.startswith('$') and s.endswith('$') and s.strip('$').count('$') == 0 else 'mixed'
49
- unk = sorted({c for t, _ in items if t for c in t if c not in D.stoi})
50
- if unk: print('characters B does not know (dropped from prose):', ''.join(unk), flush=True)
51
-
52
- # ---------------- models
53
- B = FL.setup(os.path.join(P, E('B_CKPT', 'ckpt_dl/b4styc/fp16/step003000_fp16.pt'))); net, base, TP, pnet = B['net'], B['base'], B['TP'], B['pnet']
54
- ref = np.load(os.path.join(P, E('REF', 'examples/ref_hwr200.npy'))).astype(np.float32)
55
- print(f'reference line {E("REF", "examples/ref_hwr200.npy")} ({len(ref)} points), text model {E("B_CKPT", "ckpt_dl/b4styc/fp16/step003000_fp16.pt")}, glyph model {E("GDN_CKPT")}', flush=True)
56
- def absl(q): a = np.cumsum(np.asarray(q, np.float32)[:, :2], 0); return a, np.asarray(q, np.float32)[:, 2]
57
- def xheight(q):
58
- """B normalises every line by its own height, so the x-height is measured per line: the SHORTEST y-interval holding 60% of the
59
- inked points (the x-band) x 1.29 (= the old 15..85% spread on long lines: 8 B lines of >= 25 chars, 06.11). The old 15..85%
60
- spread blew up on short words with ascenders / descenders («Проба» came out half size). Baseline = the bottom of that band."""
61
- a, pen = absl(q); y = np.sort(a[pen < 0.5, 1] if (pen < 0.5).sum() > 10 else a[:, 1]); n = len(y); k = max(1, int(0.6 * n))
62
- if n - k < 1: return max(float(y[-1] - y[0]), 1e-3), float(y[-1])
63
- w = y[k:] - y[:n - k]; j = int(np.argmin(w)); return max(1.29 * float(w[j]), 1e-3), float(y[j + k])
64
- rng = np.random.default_rng(int(E('SEED', '0'))); torch.manual_seed(int(E('SEED', '0')))
65
- lines = [(k, t) for k, (t, _) in enumerate(items) if t]; T0 = time.time(); rep = []
66
- toks = {k: [c for c in t if c in D.stoi] for k, t in lines}; ks = {k: kind(t) for k, t in lines}
67
- bi = [k for k, _ in lines if ks[k] != 'formula']; seqB = {}; durs = {}
68
- if bi:
69
- sv1 = D.style_of(net, base, [ref]); se1 = D.semb_vec([ref])
70
- for i0 in range(0, len(bi), 16):
71
- kk = bi[i0:i0 + 16]; tl = [toks[k] for k in kk]; dd = TP.predict(pnet, tl, torch.cat([se1] * len(kk), 0)); net._tok_base = None
72
- for k, d, q in zip(kk, dd, D.sample(net, tl, dd, torch.cat([sv1] * len(kk), 0))): seqB[k] = np.asarray(q, np.float32); durs[k] = d
73
- print(f' B: {min(i0 + 16, len(bi))}/{len(bi)} lines', flush=True)
74
- xs = [xheight(seqB[k]) for k in bi if ks[k] == 'prose'] or [xheight(seqB[k]) for k in bi]; t_b = time.time() - T0
75
- out = {}; FW.SEL_STATS.clear(); GD.N.clear() if hasattr(GD, 'N') else None
76
- for n_, (k, t) in enumerate(lines):
77
- t0 = time.time(); r = dict(line=k, kind=ks[k], text=t)
78
- if ks[k] == 'formula':
79
- st, ms = FL.compose_em(t, ref, rng, key=KEY); r['missing'] = ms
80
- if st: a = FL.to_line_abs(st, 1 / 0.442); out[k] = (a[:, :2], a[:, 2])
81
- elif ks[k] == 'mixed':
82
- q2, ms, nr = FL.mixed_line(seqB[k], toks[k], durs[k], ref, rng, key=KEY); r['missing'] = ms
83
- if q2 is not None:
84
- xh, bl = xheight(seqB[k]); a, pen = absl(q2); out[k] = ((a - [0, bl]) / xh, pen)
85
- else:
86
- xh, bl = xheight(seqB[k]); a, pen = absl(seqB[k]); out[k] = ((a - [0, bl]) / xh, pen)
87
- if k not in out: # the module failed -> B writes the whole line
88
- r['fallback_B'] = True; tq = toks[k]; d_ = TP.predict(pnet, [tq], D.semb_vec([ref])); net._tok_base = None
89
- q = np.asarray(D.sample(net, [tq], d_, D.style_of(net, base, [ref]))[0], np.float32); xh, bl = xheight(q); a, pen = absl(q); out[k] = ((a - [0, bl]) / xh, pen)
90
- r['sec'] = round(time.time() - t0, 2); rep.append(r)
91
- if ks[k] != 'prose': print(f' [{n_ + 1}/{len(lines)}] {ks[k]} {r["sec"]:.1f} s{" FALLBACK" if r.get("fallback_B") else ""}: {t[:60]}', flush=True)
92
-
93
- # ---------------- pages
94
- XH = float(E('XH', '15')); W, H = 1240, 1754; ML, MT = 90, 110; LPP = int(E('LINES_PER_PAGE', '99')); PITCH = 3.2 * XH
95
- def on_grid(v): return MT + np.ceil((v - MT) / PITCH - 1e-6) * PITCH # the next ruling line at or below v
96
- RULING = [float(v) for v in np.arange(MT, H - 60, PITCH)] # faint ruling; baselines sit on it
97
- pages = [[]]; y = MT - PITCH; prev_bot = float(MT - 2 * XH); nline = 0 # 06.11: pages are kept as strokes in page px -> render_pages (vector PDF, SVG, 300 dpi PNG)
98
- for k, (t, head) in enumerate(items):
99
- if t is None:
100
- y += PITCH; prev_bot += PITCH; continue # a paragraph gap = one empty ruling line
101
- a, pen = out[k]; top, bot = a[:, 1].min(), a[:, 1].max(); s = XH; wid = (a[:, 0].max() - a[:, 0].min()) * s
102
- if wid > W - ML - 50: s *= (W - ML - 50) / wid
103
- base_y = on_grid(max(y + PITCH, prev_bot + 0.6 * XH - top * s))
104
- if base_y + bot * s > H - 60 or nline >= LPP:
105
- pages.append([]); y = MT - PITCH; prev_bot = float(MT - 2 * XH); nline = 0; base_y = on_grid(max(y + PITCH, prev_bot + 0.6 * XH - top * s))
106
- x0 = a[:, 0].min(); ind = 0 if ks.get(k) != 'formula' else 40; cur = []
107
- for (x, yy), p in zip(a, pen):
108
- cur.append([round(float(ML + ind + (x - x0) * s), 2), round(float(base_y + yy * s), 2)])
109
- if p > 0.5: pages[-1].append(cur); cur = []
110
- if cur: pages[-1].append(cur)
111
- y = base_y; prev_bot = base_y + bot * s; nline += 1
112
- os.makedirs(OUT); doc = dict(W=W, H=H, dpi=150, ruling=RULING, pages=pages)
113
- json.dump(doc, open(os.path.join(OUT, 'strokes.json'), 'w')); import render_pages; render_pages.render(doc, OUT)
114
- src = dict(FW.SEL_STATS); src.update({f'glyph_net_{k}': v for k, v in getattr(GD, 'N', {}).items()})
115
- json.dump(dict(input=os.path.abspath(IN), pages=len(pages), lines=len(lines), kinds=dict(collections.Counter(ks.values())), sec_total=round(time.time() - T0, 1),
116
- sec_B=round(t_b, 1), fallbacks=sum(1 for r in rep if r.get('fallback_B')), glyph_sources=src, glyph_model=E('GDN_CKPT'), text_model=E('B_CKPT', 'ckpt_dl/b4styc/fp16/step003000_fp16.pt'), reference=E('REF', 'examples/ref_hwr200.npy'),
117
- per_line=rep), open(os.path.join(OUT, 'report.json'), 'w'), ensure_ascii=False, indent=1)
118
- print(f'PAGES DONE {len(pages)} pages, {len(lines)} lines in {time.time() - T0:.0f} s -> {OUT} | glyph sources {src}', flush=True)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tools/mathjax/layout.mjs CHANGED
@@ -3,8 +3,7 @@
3
  // "bbox":[x0,y0,x1,y1], "node":<mml node>}], "w", "h", "err"}. Units: 1/1000 em, y grows DOWN (page convention).
4
  // Glyph bbox = extent of the outline's absolute coordinates (MathJax glyph paths use absolute M/L/Q/T commands) under the
5
  // accumulated translate/scale chain; <rect> (fraction bars, radical overbars, table rules) are emitted as c="BAR".
6
- // Used for (1) free labelling of handwritten formula glyphs (align them to the layout of the line's LaTeX label) and
7
- // (2) placing generated glyphs when writing formulas. Usage: node layout.mjs < in.jsonl > out.jsonl
8
  import {mathjax} from 'mathjax-full/js/mathjax.js';
9
  import {TeX} from 'mathjax-full/js/input/tex.js';
10
  import {SVG} from 'mathjax-full/js/output/svg.js';
@@ -45,7 +44,7 @@ function walk(node,M,out,mml){
45
  if(kind==='path'){ const c=adaptor.getAttribute(node,'data-c'); const d=adaptor.getAttribute(node,'d')||'';
46
  const pts=pathPoints(d);
47
  if(c&&pts.length) out.push({c, bbox:boxOf(M2,pts), node:mm}); return; }
48
- if(kind==='text' && process.env.LAYOUT_TEXT==='1'){ // 22.10: unknown-font text (Cyrillic in \\text{}) -> one TEXT box per char:
49
  // advance 600 (MathJax's estimate, see translate(600k) of the siblings), x-height 450 above the baseline, in the accumulated frame
50
  const txt=(adaptor.textContent?adaptor.textContent(node):'')||''; const sc=Math.hypot(M2[0],M2[1])||1; const [bx,by]=app(M2,0,0);
51
  out.push({c:'TEXT', ch:txt, bbox:[bx,by-450*sc,bx+600*sc,by], node:mm}); return; }
 
3
  // "bbox":[x0,y0,x1,y1], "node":<mml node>}], "w", "h", "err"}. Units: 1/1000 em, y grows DOWN (page convention).
4
  // Glyph bbox = extent of the outline's absolute coordinates (MathJax glyph paths use absolute M/L/Q/T commands) under the
5
  // accumulated translate/scale chain; <rect> (fraction bars, radical overbars, table rules) are emitted as c="BAR".
6
+ // Usage: node layout.mjs < in.jsonl > out.jsonl
 
7
  import {mathjax} from 'mathjax-full/js/mathjax.js';
8
  import {TeX} from 'mathjax-full/js/input/tex.js';
9
  import {SVG} from 'mathjax-full/js/output/svg.js';
 
44
  if(kind==='path'){ const c=adaptor.getAttribute(node,'data-c'); const d=adaptor.getAttribute(node,'d')||'';
45
  const pts=pathPoints(d);
46
  if(c&&pts.length) out.push({c, bbox:boxOf(M2,pts), node:mm}); return; }
47
+ if(kind==='text' && process.env.LAYOUT_TEXT==='1'){ // unknown-font text (Cyrillic in \\text{}) -> one TEXT box per char:
48
  // advance 600 (MathJax's estimate, see translate(600k) of the siblings), x-height 450 above the baseline, in the accumulated frame
49
  const txt=(adaptor.textContent?adaptor.textContent(node):'')||''; const sc=Math.hypot(M2[0],M2[1])||1; const [bx,by]=app(M2,0,0);
50
  out.push({c:'TEXT', ch:txt, bbox:[bx,by-450*sc,bx+600*sc,by], node:mm}); return; }
trace/README.md CHANGED
@@ -13,8 +13,8 @@ Image of one handwritten line -> pen trajectory: points, pen lifts, pressure.
13
  | file | what | size |
14
  |---|---|---|
15
  | `scrawler_trace.py` | the whole inference, one file, no training imports | 36 KB |
16
- | `weights/trace8_fp16.pt` | tracer: CNN + 2-layer transformer encoder, autoregressive 6-layer decoder, 23.4M parameters | 46.8 MB |
17
- | `weights/inkfix1.pt` | ink restorer: U-Net, 7.5M parameters | 15.0 MB |
18
  | `requirements.txt` | torch, numpy, pillow, opencv-python | |
19
  | `TRAINING_DATA.md` | training data, its terms and the open ShareAlike question | |
20
  | `example_open_lines.png` | example on open data only | |
@@ -24,6 +24,7 @@ Both weight files load with `torch.load(..., weights_only=True)`. They hold only
24
  ## Run
25
 
26
  ```bash
 
27
  cd trace
28
  pip install -r requirements.txt
29
  python scrawler_trace.py line.png --out trace.json --svg trace.svg --seed 0
@@ -61,8 +62,8 @@ Per source: `BENCH_part_A.md`.
61
 
62
  ```bash
63
  python trace/scrawler_trace.py my_line.png --out my_line.json
64
- python scripts/make_ref.py my_line.json my_line.npy
65
- REF=$PWD/my_line.npy python scripts/write_pages.py notes.txt out
66
  ```
67
 
68
  ## Limits
 
13
  | file | what | size |
14
  |---|---|---|
15
  | `scrawler_trace.py` | the whole inference, one file, no training imports | 36 KB |
16
+ | `weights/trace8_fp16.pt` (on Hugging Face) | tracer: CNN + 2-layer transformer encoder, autoregressive 6-layer decoder, 23.4M parameters | 46.8 MB |
17
+ | `weights/inkfix1.pt` (on Hugging Face) | ink restorer: U-Net, 7.5M parameters | 15.0 MB |
18
  | `requirements.txt` | torch, numpy, pillow, opencv-python | |
19
  | `TRAINING_DATA.md` | training data, its terms and the open ShareAlike question | |
20
  | `example_open_lines.png` | example on open data only | |
 
24
  ## Run
25
 
26
  ```bash
27
+ python download_weights.py --tracer # from the repository root: the weights go to trace/weights/
28
  cd trace
29
  pip install -r requirements.txt
30
  python scrawler_trace.py line.png --out trace.json --svg trace.svg --seed 0
 
62
 
63
  ```bash
64
  python trace/scrawler_trace.py my_line.png --out my_line.json
65
+ python make_ref.py my_line.json my_line.npy
66
+ python write_pages.py notes.txt out --ref my_line.npy
67
  ```
68
 
69
  ## Limits
write_pages.py ADDED
@@ -0,0 +1,237 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Turns a text file with LaTeX formulas into handwritten notebook pages.
2
+
3
+ python write_pages.py notes.txt out
4
+ python write_pages.py notes.txt out --ref my_line.npy
5
+
6
+ Input: UTF-8 text, one line of the notes per line. $...$ is a formula, an empty line is a paragraph gap, and a line starting
7
+ with "# " is a heading. Long lines are wrapped, never inside $...$.
8
+
9
+ Output (the folder must not exist yet): pages.pdf (vector), page_NN.svg, page_NN.png (300 dpi), strokes.json, report.json.
10
+ """
11
+ import argparse
12
+ import collections
13
+ import json
14
+ import os
15
+ import re
16
+ import shutil
17
+ import sys
18
+ import time
19
+
20
+ import numpy as np
21
+ import torch
22
+
23
+ from scrawler.common import DEFAULT_REF, DEVICE, MATHJAX_DIR
24
+ from scrawler.formula import X_HEIGHT_EM, FormulaWriter, to_points
25
+ from scrawler.render import render
26
+ from scrawler.style_encoder import load_style_net, style_vector
27
+ from scrawler.text_model import TextModel
28
+
29
+ PAGE_WIDTH, PAGE_HEIGHT = 1240, 1754 # A4 at 150 dpi
30
+ MARGIN_LEFT, MARGIN_TOP = 90, 110
31
+ BATCH = 16
32
+
33
+
34
+ def wrap(line, width):
35
+ out, current = [], ""
36
+ for part in re.split(r"(\$[^$]*\$)", line):
37
+ for word in [part] if part.startswith("$") else re.split(r"(?<= )", part):
38
+ if not word:
39
+ continue
40
+ if len(current) + len(word) > width and current.strip():
41
+ out.append(current.rstrip())
42
+ current = word.lstrip()
43
+ else:
44
+ current += word
45
+ if current.strip():
46
+ out.append(current.rstrip())
47
+ return out
48
+
49
+
50
+ def read_items(path, width):
51
+ """-> list of lines of the notes; None is a gap."""
52
+ items = []
53
+ with open(path, encoding="utf-8") as fh:
54
+ for raw in fh.read().splitlines():
55
+ s = raw.strip()
56
+ if not s:
57
+ items.append(None)
58
+ continue
59
+ if s.startswith("# "):
60
+ s = s[2:].strip()
61
+ items.append(None)
62
+ for line in wrap(s, width):
63
+ # a wrapped tail like «$...$:» is a formula with its punctuation inside
64
+ m = re.fullmatch(r"\s*\$([^$]+)\$\s*([.,:;!?]*)\s*", line)
65
+ items.append("$" + m.group(1) + m.group(2) + "$" if m else line)
66
+ return items
67
+
68
+
69
+ def kind_of(line):
70
+ if "$" not in line:
71
+ return "prose"
72
+ s = line.strip()
73
+ return "formula" if s.startswith("$") and s.endswith("$") and s.strip("$").count("$") == 0 else "mixed"
74
+
75
+
76
+ def absolute(q):
77
+ q = np.asarray(q, np.float32)
78
+ return np.cumsum(q[:, :2], 0), q[:, 2]
79
+
80
+
81
+ def x_height(q):
82
+ """The shortest y-interval holding 60% of the inked points, x 1.29. -> (x-height, baseline). The text model normalises every
83
+ line by its own height, so this is measured line by line."""
84
+ a, pen = absolute(q)
85
+ y = np.sort(a[pen < 0.5, 1] if (pen < 0.5).sum() > 10 else a[:, 1])
86
+ n = len(y)
87
+ k = max(1, int(0.6 * n))
88
+ if n - k < 1:
89
+ return max(float(y[-1] - y[0]), 1e-3), float(y[-1])
90
+ widths = y[k:] - y[: n - k]
91
+ j = int(np.argmin(widths))
92
+ return max(1.29 * float(widths[j]), 1e-3), float(y[j + k])
93
+
94
+
95
+ def in_x_heights(q):
96
+ """A line [n, 3] -> points with x-height 1 and the baseline at y = 0, and the pen column."""
97
+ xh, baseline = x_height(q)
98
+ a, pen = absolute(q)
99
+ return (a - [0, baseline]) / xh, pen
100
+
101
+
102
+ def lay_out_pages(items, written, kinds, xh, lines_per_page):
103
+ pitch = 3.2 * xh
104
+ on_grid = lambda v: MARGIN_TOP + np.ceil((v - MARGIN_TOP) / pitch - 1e-6) * pitch # the next ruling line at or below v
105
+ ruling = [float(v) for v in np.arange(MARGIN_TOP, PAGE_HEIGHT - 60, pitch)]
106
+ pages = [[]]
107
+ y = MARGIN_TOP - pitch
108
+ previous_bottom = float(MARGIN_TOP - 2 * xh)
109
+ count = 0
110
+ for k, text in enumerate(items):
111
+ if text is None:
112
+ y += pitch
113
+ previous_bottom += pitch
114
+ continue
115
+ a, pen = written[k]
116
+ top, bottom = a[:, 1].min(), a[:, 1].max()
117
+ s = xh
118
+ width = (a[:, 0].max() - a[:, 0].min()) * s
119
+ if width > PAGE_WIDTH - MARGIN_LEFT - 50:
120
+ s *= (PAGE_WIDTH - MARGIN_LEFT - 50) / width
121
+ baseline = on_grid(max(y + pitch, previous_bottom + 0.6 * xh - top * s))
122
+ if baseline + bottom * s > PAGE_HEIGHT - 60 or count >= lines_per_page:
123
+ pages.append([])
124
+ y = MARGIN_TOP - pitch
125
+ previous_bottom = float(MARGIN_TOP - 2 * xh)
126
+ count = 0
127
+ baseline = on_grid(max(y + pitch, previous_bottom + 0.6 * xh - top * s))
128
+ x0 = a[:, 0].min()
129
+ indent = 0 if kinds.get(k) != "formula" else 40
130
+ current = []
131
+ for (px, py), p in zip(a, pen):
132
+ current.append([round(float(MARGIN_LEFT + indent + (px - x0) * s), 2), round(float(baseline + py * s), 2)])
133
+ if p > 0.5:
134
+ pages[-1].append(current)
135
+ current = []
136
+ if current:
137
+ pages[-1].append(current)
138
+ y = baseline
139
+ previous_bottom = baseline + bottom * s
140
+ count += 1
141
+ return dict(W=PAGE_WIDTH, H=PAGE_HEIGHT, dpi=150, ruling=ruling, pages=pages)
142
+
143
+
144
+ def main():
145
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
146
+ ap.add_argument("input", help="the text file")
147
+ ap.add_argument("out", help="output folder, must not exist")
148
+ ap.add_argument("--ref", default=DEFAULT_REF, help="the style: one line of handwriting, [n, 3] .npy from make_ref.py")
149
+ ap.add_argument("--seed", type=int, default=0, help="another seed gives another writing of the same text")
150
+ ap.add_argument("--wrap", type=int, default=42, help="line length in characters")
151
+ ap.add_argument("--x-height", type=float, default=15.0, help="height of small letters in page px; the ruling follows it")
152
+ ap.add_argument("--lines-per-page", type=int, default=99)
153
+ ap.add_argument("--pen", type=float, default=1.1, help="pen width in points")
154
+ args = ap.parse_args()
155
+ if os.path.exists(args.out):
156
+ sys.exit(f"{args.out} exists")
157
+
158
+ items = read_items(args.input, args.wrap)
159
+ lines = [(k, t) for k, t in enumerate(items) if t]
160
+ kinds = {k: kind_of(t) for k, t in lines}
161
+ if any(v != "prose" for v in kinds.values()) and not (shutil.which("node") and os.path.isdir(os.path.join(MATHJAX_DIR, "node_modules"))):
162
+ sys.exit("Formulas need Node.js and MathJax: install Node, then run cd tools/mathjax && npm ci")
163
+
164
+ text = TextModel()
165
+ unknown = sorted({c for _, t in lines for c in t if c not in text.stoi})
166
+ if unknown:
167
+ print("characters the text model does not know (dropped from prose):", "".join(unknown), flush=True)
168
+ reference = np.load(args.ref).astype(np.float32)
169
+ print(f"device {DEVICE}, style from {args.ref} ({len(reference)} points)", flush=True)
170
+ rng = np.random.default_rng(args.seed)
171
+ torch.manual_seed(args.seed)
172
+ started = time.time()
173
+ semb = style_vector(load_style_net(), reference)
174
+ style = text.style(semb)
175
+ formulas = FormulaWriter(text, reference, semb)
176
+
177
+ tokens = {k: [c for c in t if c in text.stoi] for k, t in lines}
178
+ by_text_model = [k for k, _ in lines if kinds[k] != "formula"]
179
+ sampled, durations = {}, {}
180
+ for start in range(0, len(by_text_model), BATCH):
181
+ chunk = by_text_model[start : start + BATCH]
182
+ chars = [tokens[k] for k in chunk]
183
+ planned = text.durations(chars, torch.cat([semb] * len(chunk), 0))
184
+ for k, d, q in zip(chunk, planned, text.sample(chars, planned, torch.cat([style] * len(chunk), 0))):
185
+ sampled[k] = np.asarray(q, np.float32)
186
+ durations[k] = d
187
+ print(f" text: {min(start + BATCH, len(by_text_model))}/{len(by_text_model)} lines", flush=True)
188
+ text_seconds = time.time() - started
189
+
190
+ written, report = {}, []
191
+ for number, (k, t) in enumerate(lines):
192
+ t0 = time.time()
193
+ entry = dict(line=k, kind=kinds[k], text=t)
194
+ if kinds[k] == "formula":
195
+ strokes, entry["missing"] = formulas.compose(t, rng)
196
+ if strokes:
197
+ a = to_points(strokes, 1 / X_HEIGHT_EM)
198
+ written[k] = (a[:, :2], a[:, 2])
199
+ elif kinds[k] == "mixed":
200
+ q, entry["missing"], _ = formulas.mixed_line(sampled[k], tokens[k], durations[k], rng)
201
+ if q is not None:
202
+ xh, baseline = x_height(sampled[k]) # by the line before the formulas went in
203
+ a, pen = absolute(q)
204
+ written[k] = ((a - [0, baseline]) / xh, pen)
205
+ else:
206
+ written[k] = in_x_heights(sampled[k])
207
+ if k not in written: # the formula module failed: the text model writes the line itself
208
+ entry["fallback_text_model"] = True
209
+ planned = text.durations([tokens[k]], semb)
210
+ written[k] = in_x_heights(np.asarray(text.sample([tokens[k]], planned, style)[0], np.float32))
211
+ entry["sec"] = round(time.time() - t0, 2)
212
+ report.append(entry)
213
+ if kinds[k] != "prose":
214
+ print(f" [{number + 1}/{len(lines)}] {kinds[k]} {entry['sec']:.1f} s{' FALLBACK' if entry.get('fallback_text_model') else ''}: {t[:60]}", flush=True)
215
+
216
+ doc = lay_out_pages(items, written, kinds, args.x_height, args.lines_per_page)
217
+ os.makedirs(args.out)
218
+ with open(os.path.join(args.out, "strokes.json"), "w") as fh:
219
+ json.dump(doc, fh)
220
+ render(doc, args.out, pen=args.pen)
221
+ summary = dict(
222
+ pages=len(doc["pages"]),
223
+ lines=len(lines),
224
+ kinds=dict(collections.Counter(kinds.values())),
225
+ sec_total=round(time.time() - started, 1),
226
+ sec_text_model=round(text_seconds, 1),
227
+ fallbacks=sum(1 for r in report if r.get("fallback_text_model")),
228
+ glyph_sources=dict(formulas.stats),
229
+ per_line=report,
230
+ )
231
+ with open(os.path.join(args.out, "report.json"), "w", encoding="utf-8") as fh:
232
+ json.dump(summary, fh, ensure_ascii=False, indent=1)
233
+ print(f"{len(doc['pages'])} pages, {len(lines)} lines in {time.time() - started:.0f} s -> {args.out}", flush=True)
234
+
235
+
236
+ if __name__ == "__main__":
237
+ main()