English and Russian model: write_line.py --model bilingual (non-commercial research only, trained on BRUSH)
Browse files- NOTICE +1 -0
- README.md +36 -0
- ckpt_ctc6/ctc_fp16.pt +3 -0
- ckpt_dl/u4s2/fp16/step020000_fp16.pt +3 -0
- ckpt_semb4/best.pt +3 -0
- download_weights.py +10 -1
- scrawler/common.py +4 -0
- scrawler/site_model.py +37 -10
- scrawler/style_encoder.py +2 -2
- write_line.py +4 -2
NOTICE
CHANGED
|
@@ -9,3 +9,4 @@ examples/from_scan.png shows one line of HWR200.
|
|
| 9 |
trace/example_open_lines.png includes a render of Korean Handwriting Strokes by Jonathan Baudanza (hangul.ink), handwriting by Jo YW, CC BY-SA 4.0, and so is under CC BY-SA 4.0; it also shows OUHD-L (CC BY 4.0) and Digital Peter (MIT).
|
| 10 |
The recogniser (ckpt_ctc4) was also trained on traced HWR200 lines.
|
| 11 |
tablet/samsung/sdocx2ink.mjs uses the npm package @twango/sdocx (GPL-3.0, https://github.com/twangodev/sdocx). npm installs it separately; it is not distributed here.
|
|
|
|
|
|
| 9 |
trace/example_open_lines.png includes a render of Korean Handwriting Strokes by Jonathan Baudanza (hangul.ink), handwriting by Jo YW, CC BY-SA 4.0, and so is under CC BY-SA 4.0; it also shows OUHD-L (CC BY 4.0) and Digital Peter (MIT).
|
| 10 |
The recogniser (ckpt_ctc4) was also trained on traced HWR200 lines.
|
| 11 |
tablet/samsung/sdocx2ink.mjs uses the npm package @twango/sdocx (GPL-3.0, https://github.com/twangodev/sdocx). npm installs it separately; it is not distributed here.
|
| 12 |
+
The English and Russian model (ckpt_dl/u4s2, ckpt_semb4, ckpt_ctc6) was trained on BRUSH (Brown University, https://dsd.cs.brown.edu/), whose terms allow non-commercial research use only; these weights carry that restriction.
|
README.md
CHANGED
|
@@ -2,6 +2,7 @@
|
|
| 2 |
license: cc-by-nc-4.0
|
| 3 |
language:
|
| 4 |
- ru
|
|
|
|
| 5 |
tags:
|
| 6 |
- handwriting-synthesis
|
| 7 |
- online-handwriting
|
|
@@ -133,6 +134,9 @@ The style vector comes from a line-level writer encoder. Fraction bars, radicals
|
|
| 133 |
| `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev), used by `write_line.py`, see below | 51 MB |
|
| 134 |
| `ckpt_ctc4/ctc_fp16.pt` | recogniser, for `write_line.py` | 20 MB |
|
| 135 |
| `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer, for `write_line.py` | 4 MB |
|
|
|
|
|
|
|
|
|
|
| 136 |
| `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
|
| 137 |
| `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
|
| 138 |
| `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
|
|
@@ -180,6 +184,37 @@ Measured on traced lines of 9 writers the model never saw: the cosine of a write
|
|
| 180 |
|
| 181 |
`write_pages.py` does not use the site model: pages and formulas are written by the text model of the first row of the weights table.
|
| 182 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 183 |
## Evaluation
|
| 184 |
|
| 185 |
240 lines of real notes from writers the generator never saw, each written from one reference line by the same writer. An LLM judge (Gemini 4 Argon) lists errors in each line; "clean" means no error of severity 2 or higher. Pairs are re-sampled for the 95% intervals.
|
|
@@ -213,6 +248,7 @@ Style similarity by the writer encoder (cosine to the writer's real line, one re
|
|
| 213 |
|
| 214 |
- **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
|
| 215 |
- **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
|
|
|
|
| 216 |
- **Recogniser:** the tablet lines above, the same lines rendered to images and traced back, and traced lines of HWR200.
|
| 217 |
- **Writer classifier:** tablet lines of 15 writers: 12 of the 13 above and three more.
|
| 218 |
- **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
|
|
|
|
| 2 |
license: cc-by-nc-4.0
|
| 3 |
language:
|
| 4 |
- ru
|
| 5 |
+
- en
|
| 6 |
tags:
|
| 7 |
- handwriting-synthesis
|
| 8 |
- online-handwriting
|
|
|
|
| 134 |
| `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev), used by `write_line.py`, see below | 51 MB |
|
| 135 |
| `ckpt_ctc4/ctc_fp16.pt` | recogniser, for `write_line.py` | 20 MB |
|
| 136 |
| `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer, for `write_line.py` | 4 MB |
|
| 137 |
+
| `ckpt_dl/u4s2/fp16/step020000_fp16.pt` | English and Russian text model, `write_line.py --model bilingual`; **non-commercial research only**, see below | 51 MB |
|
| 138 |
+
| `ckpt_semb4/best.pt` | its style encoder, which also tells English writers apart | 7 MB |
|
| 139 |
+
| `ckpt_ctc6/ctc_fp16.pt` | its recogniser for English lines | 20 MB |
|
| 140 |
| `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
|
| 141 |
| `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
|
| 142 |
| `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
|
|
|
|
| 184 |
|
| 185 |
`write_pages.py` does not use the site model: pages and formulas are written by the text model of the first row of the weights table.
|
| 186 |
|
| 187 |
+
## English and Russian model
|
| 188 |
+
|
| 189 |
+
The model of the site writes English letters legibly but not in the writer's hand: it never saw English pen lines, and its style encoder cannot tell English writers apart (5% in the table below). `--model bilingual` writes both languages from one line of the writer's hand.
|
| 190 |
+
|
| 191 |
+
```bash
|
| 192 |
+
python download_weights.py --bilingual
|
| 193 |
+
python write_line.py "One line, and the rest is written by your hand." out_line --model bilingual --ref my_line.npy --ref-text "what your line says"
|
| 194 |
+
```
|
| 195 |
+
|
| 196 |
+
The reference line can be in either language, and so can the text. The code for this model is in this repository; the GitHub copy does not have it yet.
|
| 197 |
+
|
| 198 |
+
**Licence.** It was trained on [BRUSH](https://dsd.cs.brown.edu/) (Brown University), whose terms allow non-commercial research use only, and the weights inherit that: `ckpt_dl/u4s2/`, `ckpt_semb4/` and `ckpt_ctc6/` are for non-commercial research only, on top of CC BY-NC 4.0.
|
| 199 |
+
|
| 200 |
+
How it was made, from the model of the site, in two stages:
|
| 201 |
+
|
| 202 |
+
1. **Text model, stage 1:** 60 000 more steps on the site model's data, with underlines removed from the tablet lines (they made the first word of a line come out underlined), plus English pen lines of 158 BRUSH writers. 12 more BRUSH writers were kept out of all training to measure on.
|
| 203 |
+
2. **Style encoder:** the release style encoder trained further to tell BRUSH writers apart, while staying close to its old vectors on Russian lines.
|
| 204 |
+
3. **Text model, stage 2:** 20 000 more steps with the new style encoder.
|
| 205 |
+
|
| 206 |
+
The recogniser for English lines is the site's recogniser trained further on the same mix. The recipe of `write_line.py` is the site's, with two changes: an English line is chosen by the English recogniser, and a Russian line that starts with a capital letter gets for that letter the length the planner gives it inside a phrase.
|
| 207 |
+
|
| 208 |
+
Measured on 20 hands per language, one reference line each, best of four attempts. English: 12 BRUSH writers the model never saw and 8 it saw. The writer measure here is a classifier trained on both languages, and "found" means the generated line is closest to its own reference among the 20.
|
| 209 |
+
|
| 210 |
+
| | English, own writer found | English, cosine own minus others | English, legible | Russian, cosine to own line | Russian, legible | Russian, stray stroke on the first word |
|
| 211 |
+
|---|---|---|---|---|---|---|
|
| 212 |
+
| Model of the site | 5% | 0.06 | 95% | 0.64 | 95% | 50% |
|
| 213 |
+
| **English and Russian model** | **65%** | **0.29** | **95%** | **0.67** | **90%** | **5%** |
|
| 214 |
+
| Another real line of the same writer | 85% | 0.49 | | | | |
|
| 215 |
+
|
| 216 |
+
Legible means a recogniser error rate of at most 0.05. The English hands are much closer to their writers than before but still well short of real lines. Without the best of four, English lines often lose spaces between words: 82% legible over all attempts. These numbers were taken on a GPU; the same seed gives other lines on a Mac.
|
| 217 |
+
|
| 218 |
## Evaluation
|
| 219 |
|
| 220 |
240 lines of real notes from writers the generator never saw, each written from one reference line by the same writer. An LLM judge (Gemini 4 Argon) lists errors in each line; "clean" means no error of severity 2 or higher. Pairs are re-sampled for the 95% intervals.
|
|
|
|
| 248 |
|
| 249 |
- **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
|
| 250 |
- **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
|
| 251 |
+
- **English and Russian model:** the same, plus English pen lines of 158 writers of [BRUSH](https://dsd.cs.brown.edu/) (non-commercial research use only). Its style encoder and English recogniser were trained further on the same lines.
|
| 252 |
- **Recogniser:** the tablet lines above, the same lines rendered to images and traced back, and traced lines of HWR200.
|
| 253 |
- **Writer classifier:** tablet lines of 15 writers: 12 of the 13 above and three more.
|
| 254 |
- **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
|
ckpt_ctc6/ctc_fp16.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5c6f4a7609ea9d71811ae4d2d864e9d990610451b174bb707323e8318c79384e
|
| 3 |
+
size 19781805
|
ckpt_dl/u4s2/fp16/step020000_fp16.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bc762ad76f6e887062076e40c291c658280d758a0fb66b13b140ecfd35d065b1
|
| 3 |
+
size 51085797
|
ckpt_semb4/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7b17fc408f3fe89fb0431ba565673725155df2b3ef8b71b4f502a1585b181f8d
|
| 3 |
+
size 7279759
|
download_weights.py
CHANGED
|
@@ -3,6 +3,7 @@
|
|
| 3 |
python download_weights.py # the models that write pages, about 80 MB
|
| 4 |
python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
|
| 5 |
python download_weights.py --site # also the models of synthink.dev for write_line.py, about 75 MB more
|
|
|
|
| 6 |
"""
|
| 7 |
import argparse
|
| 8 |
import os
|
|
@@ -18,6 +19,13 @@ WRITER = [
|
|
| 18 |
"ckpt_dl/glyph_cls2.pt",
|
| 19 |
]
|
| 20 |
TRACER = ["trace/weights/*"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
SITE = [
|
| 22 |
"ckpt_dl/u3ref/fp16/step060000_fp16.pt",
|
| 23 |
"ckpt_ctc4/ctc_fp16.pt",
|
|
@@ -29,9 +37,10 @@ def main():
|
|
| 29 |
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 30 |
ap.add_argument("--tracer", action="store_true")
|
| 31 |
ap.add_argument("--site", action="store_true")
|
|
|
|
| 32 |
args = ap.parse_args()
|
| 33 |
root = os.path.dirname(os.path.abspath(__file__))
|
| 34 |
-
snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []) + (SITE if args.site else []), local_dir=root)
|
| 35 |
print("weights are in", root)
|
| 36 |
|
| 37 |
|
|
|
|
| 3 |
python download_weights.py # the models that write pages, about 80 MB
|
| 4 |
python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
|
| 5 |
python download_weights.py --site # also the models of synthink.dev for write_line.py, about 75 MB more
|
| 6 |
+
python download_weights.py --bilingual # also the English and Russian model for write_line.py --model bilingual, about 150 MB more
|
| 7 |
"""
|
| 8 |
import argparse
|
| 9 |
import os
|
|
|
|
| 19 |
"ckpt_dl/glyph_cls2.pt",
|
| 20 |
]
|
| 21 |
TRACER = ["trace/weights/*"]
|
| 22 |
+
BILINGUAL = [ # trained on BRUSH as well: non-commercial research only, see README
|
| 23 |
+
"ckpt_dl/u4s2/fp16/step020000_fp16.pt",
|
| 24 |
+
"ckpt_semb4/best.pt",
|
| 25 |
+
"ckpt_ctc6/ctc_fp16.pt",
|
| 26 |
+
"ckpt_ctc4/ctc_fp16.pt",
|
| 27 |
+
"ckpt_stylecls/embed_fp16.pt",
|
| 28 |
+
]
|
| 29 |
SITE = [
|
| 30 |
"ckpt_dl/u3ref/fp16/step060000_fp16.pt",
|
| 31 |
"ckpt_ctc4/ctc_fp16.pt",
|
|
|
|
| 37 |
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 38 |
ap.add_argument("--tracer", action="store_true")
|
| 39 |
ap.add_argument("--site", action="store_true")
|
| 40 |
+
ap.add_argument("--bilingual", action="store_true")
|
| 41 |
args = ap.parse_args()
|
| 42 |
root = os.path.dirname(os.path.abspath(__file__))
|
| 43 |
+
snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []) + (SITE if args.site else []) + (BILINGUAL if args.bilingual else []), local_dir=root)
|
| 44 |
print("weights are in", root)
|
| 45 |
|
| 46 |
|
scrawler/common.py
CHANGED
|
@@ -28,6 +28,9 @@ CKPT_PLANNER = os.path.join(ROOT, "ckpt_dl", "planner7", "best.pt")
|
|
| 28 |
CKPT_SITE_TEXT = os.path.join(ROOT, "ckpt_dl", "u3ref", "fp16", "step060000_fp16.pt")
|
| 29 |
CKPT_RECOGNISER = os.path.join(ROOT, "ckpt_ctc4", "ctc_fp16.pt")
|
| 30 |
CKPT_CLASSIFIER = os.path.join(ROOT, "ckpt_stylecls", "embed_fp16.pt")
|
|
|
|
|
|
|
|
|
|
| 31 |
CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
|
| 32 |
CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
|
| 33 |
CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
|
|
@@ -39,5 +42,6 @@ MATHJAX_DIR = os.path.join(ROOT, "tools", "mathjax")
|
|
| 39 |
def load_checkpoint(path):
|
| 40 |
if not os.path.exists(path):
|
| 41 |
hint = " --site" if path in (CKPT_SITE_TEXT, CKPT_RECOGNISER, CKPT_CLASSIFIER) else ""
|
|
|
|
| 42 |
raise SystemExit(f"{path} not found. Run: python download_weights.py{hint}")
|
| 43 |
return torch.load(path, map_location=DEVICE, weights_only=True)
|
|
|
|
| 28 |
CKPT_SITE_TEXT = os.path.join(ROOT, "ckpt_dl", "u3ref", "fp16", "step060000_fp16.pt")
|
| 29 |
CKPT_RECOGNISER = os.path.join(ROOT, "ckpt_ctc4", "ctc_fp16.pt")
|
| 30 |
CKPT_CLASSIFIER = os.path.join(ROOT, "ckpt_stylecls", "embed_fp16.pt")
|
| 31 |
+
CKPT_BILINGUAL_TEXT = os.path.join(ROOT, "ckpt_dl", "u4s2", "fp16", "step020000_fp16.pt")
|
| 32 |
+
CKPT_BILINGUAL_STYLE = os.path.join(ROOT, "ckpt_semb4", "best.pt")
|
| 33 |
+
CKPT_BILINGUAL_RECOGNISER = os.path.join(ROOT, "ckpt_ctc6", "ctc_fp16.pt")
|
| 34 |
CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
|
| 35 |
CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
|
| 36 |
CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
|
|
|
|
| 42 |
def load_checkpoint(path):
|
| 43 |
if not os.path.exists(path):
|
| 44 |
hint = " --site" if path in (CKPT_SITE_TEXT, CKPT_RECOGNISER, CKPT_CLASSIFIER) else ""
|
| 45 |
+
hint = " --bilingual" if path in (CKPT_BILINGUAL_TEXT, CKPT_BILINGUAL_STYLE, CKPT_BILINGUAL_RECOGNISER) else hint
|
| 46 |
raise SystemExit(f"{path} not found. Run: python download_weights.py{hint}")
|
| 47 |
return torch.load(path, map_location=DEVICE, weights_only=True)
|
scrawler/site_model.py
CHANGED
|
@@ -9,6 +9,11 @@ Three things differ from scrawler/text_model.py:
|
|
| 9 |
- Every line is sampled four times. A recogniser keeps the attempts that read best (character error rate within 0.02 of the
|
| 10 |
lowest), and of those a writer classifier picks the one closest to the reference.
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
Lines are arrays [n, 3] of (dx, dy, pen_up) with line height 1, as everywhere in this package.
|
| 13 |
"""
|
| 14 |
import numpy as np
|
|
@@ -17,6 +22,7 @@ import torch.nn as nn
|
|
| 17 |
import torch.nn.functional as F
|
| 18 |
|
| 19 |
from .common import AMP, CKPT_CLASSIFIER, CKPT_PLANNER, CKPT_RECOGNISER, CKPT_SITE_TEXT, DEVICE, load_checkpoint
|
|
|
|
| 20 |
from .style_encoder import SCALE, features, load_style_net, resample, style_vector
|
| 21 |
from .text_model import ADVANCE, ALPHA_BAR, GUIDANCE, MAX_POINTS, NOISE_STEPS, SAMPLE_STEPS, SCALE_X, SCALE_Y
|
| 22 |
from .text_model import Denoiser, Planner, _plan, time_embedding
|
|
@@ -25,6 +31,15 @@ ATTEMPTS = 4
|
|
| 25 |
SPREAD_MIN, SPREAD_MAX = 0.6, 1.4
|
| 26 |
CER_SLACK = 0.02
|
| 27 |
REF_STEP = 0.05
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
|
| 30 |
def to_strokes(line):
|
|
@@ -173,8 +188,12 @@ def _edit_rate(target, hypothesis):
|
|
| 173 |
|
| 174 |
|
| 175 |
class SiteModel:
|
| 176 |
-
def __init__(self):
|
| 177 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 178 |
self.itos = state["itos"]
|
| 179 |
self.stoi = {s: i for i, s in enumerate(self.itos)}
|
| 180 |
self.null_token = len(self.itos)
|
|
@@ -186,11 +205,13 @@ class SiteModel:
|
|
| 186 |
self.planner = Planner(len(state["itos"])).to(DEVICE)
|
| 187 |
self.planner.load_state_dict(state["m"])
|
| 188 |
self.planner.eval()
|
| 189 |
-
self.style_net = load_style_net()
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
|
| 193 |
-
|
|
|
|
|
|
|
| 194 |
state = load_checkpoint(CKPT_CLASSIFIER)
|
| 195 |
self.embedder = WriterEmbedder().to(DEVICE)
|
| 196 |
self.embedder.load_state_dict({k: v.float() for k, v in state["m"].items()})
|
|
@@ -208,11 +229,14 @@ class SiteModel:
|
|
| 208 |
d = torch.expm1(out).clamp(min=1).round().long().cpu().numpy()
|
| 209 |
return [int(v) for v in d[0, 1:-1]]
|
| 210 |
|
|
|
|
|
|
|
|
|
|
| 211 |
@torch.no_grad()
|
| 212 |
-
def _read(self, lines):
|
| 213 |
"""Equal-length lines -> the recognised character ids of each."""
|
| 214 |
x = np.stack([np.stack([q[:, 0] * 20.0, q[:, 1] * 20.0, q[:, 2], np.cumsum(q[:, 1])], 1) for q in lines]).astype(np.float32)
|
| 215 |
-
best = self.
|
| 216 |
out = []
|
| 217 |
for row in best:
|
| 218 |
ids, prev = [], 0
|
|
@@ -259,6 +283,9 @@ class SiteModel:
|
|
| 259 |
raise ValueError("no characters the model knows")
|
| 260 |
ids = [self.stoi[c] for c in text]
|
| 261 |
durations = [max(1, int(round(d * ref["spread"]))) for d in self._durations(text, ref["semb"])]
|
|
|
|
|
|
|
|
|
|
| 262 |
length = sum(durations)
|
| 263 |
if length > MAX_POINTS:
|
| 264 |
raise ValueError(f"the line is too long: {length} pen points, the model takes {MAX_POINTS}. Shorten the text.")
|
|
@@ -304,7 +331,7 @@ class SiteModel:
|
|
| 304 |
d[-1, 2] = 1
|
| 305 |
lines.append(d)
|
| 306 |
|
| 307 |
-
cer = np.array([_edit_rate(ids, h) for h in self._read(lines)], np.float32)
|
| 308 |
cos = np.array([float(self._embed(q) @ ref["embedding"]) for q in lines], np.float32)
|
| 309 |
legible = np.flatnonzero(cer <= cer.min() + CER_SLACK)
|
| 310 |
pick = int(legible[np.argmax(cos[legible])])
|
|
|
|
| 9 |
- Every line is sampled four times. A recogniser keeps the attempts that read best (character error rate within 0.02 of the
|
| 10 |
lowest), and of those a writer classifier picks the one closest to the reference.
|
| 11 |
|
| 12 |
+
SiteModel("bilingual") is the English and Russian model: the site model trained further on English pen lines of BRUSH, with a
|
| 13 |
+
style encoder that also tells English writers apart. The recipe is the same, with two changes measured on it: an English line is
|
| 14 |
+
read by a recogniser that knows English handwriting, and a Russian line that starts with a capital letter gets for that letter
|
| 15 |
+
the length the planner gives it inside a phrase (without this the first word often comes out with a stray stroke).
|
| 16 |
+
|
| 17 |
Lines are arrays [n, 3] of (dx, dy, pen_up) with line height 1, as everywhere in this package.
|
| 18 |
"""
|
| 19 |
import numpy as np
|
|
|
|
| 22 |
import torch.nn.functional as F
|
| 23 |
|
| 24 |
from .common import AMP, CKPT_CLASSIFIER, CKPT_PLANNER, CKPT_RECOGNISER, CKPT_SITE_TEXT, DEVICE, load_checkpoint
|
| 25 |
+
from .common import CKPT_BILINGUAL_RECOGNISER, CKPT_BILINGUAL_STYLE, CKPT_BILINGUAL_TEXT, CKPT_STYLE
|
| 26 |
from .style_encoder import SCALE, features, load_style_net, resample, style_vector
|
| 27 |
from .text_model import ADVANCE, ALPHA_BAR, GUIDANCE, MAX_POINTS, NOISE_STEPS, SAMPLE_STEPS, SCALE_X, SCALE_Y
|
| 28 |
from .text_model import Denoiser, Planner, _plan, time_embedding
|
|
|
|
| 31 |
SPREAD_MIN, SPREAD_MAX = 0.6, 1.4
|
| 32 |
CER_SLACK = 0.02
|
| 33 |
REF_STEP = 0.05
|
| 34 |
+
MODELS = {
|
| 35 |
+
"site": dict(text=CKPT_SITE_TEXT, style=CKPT_STYLE, recognisers=dict(any=CKPT_RECOGNISER)),
|
| 36 |
+
"bilingual": dict(text=CKPT_BILINGUAL_TEXT, style=CKPT_BILINGUAL_STYLE, recognisers=dict(ru=CKPT_RECOGNISER, any=CKPT_BILINGUAL_RECOGNISER)),
|
| 37 |
+
}
|
| 38 |
+
FIRST_LETTER_PREFIX = "и " # bilingual model, Russian: the first letter is planned as if the line went on after this
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def is_russian(text):
|
| 42 |
+
return any("а" <= c.lower() <= "я" or c in "ёЁ" for c in text)
|
| 43 |
|
| 44 |
|
| 45 |
def to_strokes(line):
|
|
|
|
| 188 |
|
| 189 |
|
| 190 |
class SiteModel:
|
| 191 |
+
def __init__(self, kind="site"):
|
| 192 |
+
if kind not in MODELS:
|
| 193 |
+
raise ValueError(f"unknown model {kind!r}: {' or '.join(MODELS)}")
|
| 194 |
+
self.kind = kind
|
| 195 |
+
paths = MODELS[kind]
|
| 196 |
+
state = load_checkpoint(paths["text"])
|
| 197 |
self.itos = state["itos"]
|
| 198 |
self.stoi = {s: i for i, s in enumerate(self.itos)}
|
| 199 |
self.null_token = len(self.itos)
|
|
|
|
| 205 |
self.planner = Planner(len(state["itos"])).to(DEVICE)
|
| 206 |
self.planner.load_state_dict(state["m"])
|
| 207 |
self.planner.eval()
|
| 208 |
+
self.style_net = load_style_net(paths["style"])
|
| 209 |
+
self.recognisers = {}
|
| 210 |
+
for name, path in paths["recognisers"].items():
|
| 211 |
+
state = load_checkpoint(path)
|
| 212 |
+
net = Recogniser(len(state["itos"])).to(DEVICE)
|
| 213 |
+
net.load_state_dict({k: v.float() for k, v in state["m"].items()})
|
| 214 |
+
self.recognisers[name] = net.eval()
|
| 215 |
state = load_checkpoint(CKPT_CLASSIFIER)
|
| 216 |
self.embedder = WriterEmbedder().to(DEVICE)
|
| 217 |
self.embedder.load_state_dict({k: v.float() for k, v in state["m"].items()})
|
|
|
|
| 229 |
d = torch.expm1(out).clamp(min=1).round().long().cpu().numpy()
|
| 230 |
return [int(v) for v in d[0, 1:-1]]
|
| 231 |
|
| 232 |
+
def _recogniser(self, text):
|
| 233 |
+
return self.recognisers["ru" if "ru" in self.recognisers and is_russian(text) else "any"]
|
| 234 |
+
|
| 235 |
@torch.no_grad()
|
| 236 |
+
def _read(self, lines, recogniser=None):
|
| 237 |
"""Equal-length lines -> the recognised character ids of each."""
|
| 238 |
x = np.stack([np.stack([q[:, 0] * 20.0, q[:, 1] * 20.0, q[:, 2], np.cumsum(q[:, 1])], 1) for q in lines]).astype(np.float32)
|
| 239 |
+
best = (recogniser or self.recognisers["any"])(torch.from_numpy(x).to(DEVICE)).argmax(-1).cpu().numpy()
|
| 240 |
out = []
|
| 241 |
for row in best:
|
| 242 |
ids, prev = [], 0
|
|
|
|
| 283 |
raise ValueError("no characters the model knows")
|
| 284 |
ids = [self.stoi[c] for c in text]
|
| 285 |
durations = [max(1, int(round(d * ref["spread"]))) for d in self._durations(text, ref["semb"])]
|
| 286 |
+
if self.kind == "bilingual" and text[0].isupper() and is_russian(text[0]):
|
| 287 |
+
inside = self._durations(FIRST_LETTER_PREFIX + text, ref["semb"])[len(FIRST_LETTER_PREFIX)]
|
| 288 |
+
durations[0] = min(durations[0], max(1, int(round(max(1, inside) * ref["spread"]))))
|
| 289 |
length = sum(durations)
|
| 290 |
if length > MAX_POINTS:
|
| 291 |
raise ValueError(f"the line is too long: {length} pen points, the model takes {MAX_POINTS}. Shorten the text.")
|
|
|
|
| 331 |
d[-1, 2] = 1
|
| 332 |
lines.append(d)
|
| 333 |
|
| 334 |
+
cer = np.array([_edit_rate(ids, h) for h in self._read(lines, self._recogniser(text))], np.float32)
|
| 335 |
cos = np.array([float(self._embed(q) @ ref["embedding"]) for q in lines], np.float32)
|
| 336 |
legible = np.flatnonzero(cer <= cer.min() + CER_SLACK)
|
| 337 |
pick = int(legible[np.argmax(cos[legible])])
|
scrawler/style_encoder.py
CHANGED
|
@@ -83,8 +83,8 @@ class StyleNet(nn.Module):
|
|
| 83 |
return self.head(torch.cat([mean, top], -1))
|
| 84 |
|
| 85 |
|
| 86 |
-
def load_style_net():
|
| 87 |
-
state = load_checkpoint(
|
| 88 |
net = StyleNet(out_dim=256, d=state["D"]).to(DEVICE)
|
| 89 |
net.load_state_dict(state["m"])
|
| 90 |
net.eval()
|
|
|
|
| 83 |
return self.head(torch.cat([mean, top], -1))
|
| 84 |
|
| 85 |
|
| 86 |
+
def load_style_net(path=CKPT_STYLE):
|
| 87 |
+
state = load_checkpoint(path)
|
| 88 |
net = StyleNet(out_dim=256, d=state["D"]).to(DEVICE)
|
| 89 |
net.load_state_dict(state["m"])
|
| 90 |
net.eval()
|
write_line.py
CHANGED
|
@@ -11,7 +11,8 @@ In the folder `out` (it must not exist yet):
|
|
| 11 |
- attempts.png: all attempts, the chosen one marked with a bar on the left
|
| 12 |
- strokes.json: every attempt as strokes, with its character error rate and its closeness to the reference
|
| 13 |
|
| 14 |
-
|
|
|
|
| 15 |
in that line; it sets how wide the writer writes. Without it the recogniser reads the reference, which is right in most
|
| 16 |
letters but not all, so give the text when you know it.
|
| 17 |
"""
|
|
@@ -85,6 +86,7 @@ def main():
|
|
| 85 |
ap.add_argument("out", help="output folder, must not exist")
|
| 86 |
ap.add_argument("--ref", default=DEFAULT_REF, help="the style: one line of handwriting, [n, 3] .npy from make_ref.py")
|
| 87 |
ap.add_argument("--ref-text", help="what is written in the reference line; read by the recogniser if not given")
|
|
|
|
| 88 |
ap.add_argument("--attempts", type=int, default=ATTEMPTS, help="samples per line; the best one is kept")
|
| 89 |
ap.add_argument("--seed", type=int, default=0, help="another seed gives another writing of the same text")
|
| 90 |
ap.add_argument("--pen", type=float, default=1.1, help="pen width in points")
|
|
@@ -97,7 +99,7 @@ def main():
|
|
| 97 |
else:
|
| 98 |
texts = [args.text.strip()]
|
| 99 |
|
| 100 |
-
model = SiteModel()
|
| 101 |
reference = np.load(args.ref).astype(np.float32)
|
| 102 |
ref_text = args.ref_text
|
| 103 |
if ref_text is None and os.path.abspath(args.ref) == os.path.abspath(DEFAULT_REF):
|
|
|
|
| 11 |
- attempts.png: all attempts, the chosen one marked with a bar on the left
|
| 12 |
- strokes.json: every attempt as strokes, with its character error rate and its closeness to the reference
|
| 13 |
|
| 14 |
+
--model bilingual writes English as well as Russian (python download_weights.py --bilingual); its weights are for
|
| 15 |
+
non-commercial research only, see README. The recipe is in scrawler/site_model.py. The style comes from --ref (an .npy from make_ref.py). --ref-text is what is written
|
| 16 |
in that line; it sets how wide the writer writes. Without it the recogniser reads the reference, which is right in most
|
| 17 |
letters but not all, so give the text when you know it.
|
| 18 |
"""
|
|
|
|
| 86 |
ap.add_argument("out", help="output folder, must not exist")
|
| 87 |
ap.add_argument("--ref", default=DEFAULT_REF, help="the style: one line of handwriting, [n, 3] .npy from make_ref.py")
|
| 88 |
ap.add_argument("--ref-text", help="what is written in the reference line; read by the recogniser if not given")
|
| 89 |
+
ap.add_argument("--model", default="site", choices=["site", "bilingual"], help="site: the model of synthink.dev; bilingual: English and Russian")
|
| 90 |
ap.add_argument("--attempts", type=int, default=ATTEMPTS, help="samples per line; the best one is kept")
|
| 91 |
ap.add_argument("--seed", type=int, default=0, help="another seed gives another writing of the same text")
|
| 92 |
ap.add_argument("--pen", type=float, default=1.1, help="pen width in points")
|
|
|
|
| 99 |
else:
|
| 100 |
texts = [args.text.strip()]
|
| 101 |
|
| 102 |
+
model = SiteModel(args.model)
|
| 103 |
reference = np.load(args.ref).astype(np.float32)
|
| 104 |
ref_text = args.ref_text
|
| 105 |
if ref_text is None and os.path.abspath(args.ref) == os.path.abspath(DEFAULT_REF):
|