levossadtchi commited on
Commit
a34a22a
·
verified ·
1 Parent(s): f1fa3dd

English and Russian model: write_line.py --model bilingual (non-commercial research only, trained on BRUSH)

Browse files
NOTICE CHANGED
@@ -9,3 +9,4 @@ examples/from_scan.png shows one line of HWR200.
9
  trace/example_open_lines.png includes a render of Korean Handwriting Strokes by Jonathan Baudanza (hangul.ink), handwriting by Jo YW, CC BY-SA 4.0, and so is under CC BY-SA 4.0; it also shows OUHD-L (CC BY 4.0) and Digital Peter (MIT).
10
  The recogniser (ckpt_ctc4) was also trained on traced HWR200 lines.
11
  tablet/samsung/sdocx2ink.mjs uses the npm package @twango/sdocx (GPL-3.0, https://github.com/twangodev/sdocx). npm installs it separately; it is not distributed here.
 
 
9
  trace/example_open_lines.png includes a render of Korean Handwriting Strokes by Jonathan Baudanza (hangul.ink), handwriting by Jo YW, CC BY-SA 4.0, and so is under CC BY-SA 4.0; it also shows OUHD-L (CC BY 4.0) and Digital Peter (MIT).
10
  The recogniser (ckpt_ctc4) was also trained on traced HWR200 lines.
11
  tablet/samsung/sdocx2ink.mjs uses the npm package @twango/sdocx (GPL-3.0, https://github.com/twangodev/sdocx). npm installs it separately; it is not distributed here.
12
+ The English and Russian model (ckpt_dl/u4s2, ckpt_semb4, ckpt_ctc6) was trained on BRUSH (Brown University, https://dsd.cs.brown.edu/), whose terms allow non-commercial research use only; these weights carry that restriction.
README.md CHANGED
@@ -2,6 +2,7 @@
2
  license: cc-by-nc-4.0
3
  language:
4
  - ru
 
5
  tags:
6
  - handwriting-synthesis
7
  - online-handwriting
@@ -133,6 +134,9 @@ The style vector comes from a line-level writer encoder. Fraction bars, radicals
133
  | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev), used by `write_line.py`, see below | 51 MB |
134
  | `ckpt_ctc4/ctc_fp16.pt` | recogniser, for `write_line.py` | 20 MB |
135
  | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer, for `write_line.py` | 4 MB |
 
 
 
136
  | `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
137
  | `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
138
  | `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
@@ -180,6 +184,37 @@ Measured on traced lines of 9 writers the model never saw: the cosine of a write
180
 
181
  `write_pages.py` does not use the site model: pages and formulas are written by the text model of the first row of the weights table.
182
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
183
  ## Evaluation
184
 
185
  240 lines of real notes from writers the generator never saw, each written from one reference line by the same writer. An LLM judge (Gemini 4 Argon) lists errors in each line; "clean" means no error of severity 2 or higher. Pairs are re-sampled for the 95% intervals.
@@ -213,6 +248,7 @@ Style similarity by the writer encoder (cosine to the writer's real line, one re
213
 
214
  - **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
215
  - **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
 
216
  - **Recogniser:** the tablet lines above, the same lines rendered to images and traced back, and traced lines of HWR200.
217
  - **Writer classifier:** tablet lines of 15 writers: 12 of the 13 above and three more.
218
  - **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
 
2
  license: cc-by-nc-4.0
3
  language:
4
  - ru
5
+ - en
6
  tags:
7
  - handwriting-synthesis
8
  - online-handwriting
 
134
  | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev), used by `write_line.py`, see below | 51 MB |
135
  | `ckpt_ctc4/ctc_fp16.pt` | recogniser, for `write_line.py` | 20 MB |
136
  | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer, for `write_line.py` | 4 MB |
137
+ | `ckpt_dl/u4s2/fp16/step020000_fp16.pt` | English and Russian text model, `write_line.py --model bilingual`; **non-commercial research only**, see below | 51 MB |
138
+ | `ckpt_semb4/best.pt` | its style encoder, which also tells English writers apart | 7 MB |
139
+ | `ckpt_ctc6/ctc_fp16.pt` | its recogniser for English lines | 20 MB |
140
  | `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
141
  | `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
142
  | `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
 
184
 
185
  `write_pages.py` does not use the site model: pages and formulas are written by the text model of the first row of the weights table.
186
 
187
+ ## English and Russian model
188
+
189
+ The model of the site writes English letters legibly but not in the writer's hand: it never saw English pen lines, and its style encoder cannot tell English writers apart (5% in the table below). `--model bilingual` writes both languages from one line of the writer's hand.
190
+
191
+ ```bash
192
+ python download_weights.py --bilingual
193
+ python write_line.py "One line, and the rest is written by your hand." out_line --model bilingual --ref my_line.npy --ref-text "what your line says"
194
+ ```
195
+
196
+ The reference line can be in either language, and so can the text. The code for this model is in this repository; the GitHub copy does not have it yet.
197
+
198
+ **Licence.** It was trained on [BRUSH](https://dsd.cs.brown.edu/) (Brown University), whose terms allow non-commercial research use only, and the weights inherit that: `ckpt_dl/u4s2/`, `ckpt_semb4/` and `ckpt_ctc6/` are for non-commercial research only, on top of CC BY-NC 4.0.
199
+
200
+ How it was made, from the model of the site, in two stages:
201
+
202
+ 1. **Text model, stage 1:** 60 000 more steps on the site model's data, with underlines removed from the tablet lines (they made the first word of a line come out underlined), plus English pen lines of 158 BRUSH writers. 12 more BRUSH writers were kept out of all training to measure on.
203
+ 2. **Style encoder:** the release style encoder trained further to tell BRUSH writers apart, while staying close to its old vectors on Russian lines.
204
+ 3. **Text model, stage 2:** 20 000 more steps with the new style encoder.
205
+
206
+ The recogniser for English lines is the site's recogniser trained further on the same mix. The recipe of `write_line.py` is the site's, with two changes: an English line is chosen by the English recogniser, and a Russian line that starts with a capital letter gets for that letter the length the planner gives it inside a phrase.
207
+
208
+ Measured on 20 hands per language, one reference line each, best of four attempts. English: 12 BRUSH writers the model never saw and 8 it saw. The writer measure here is a classifier trained on both languages, and "found" means the generated line is closest to its own reference among the 20.
209
+
210
+ | | English, own writer found | English, cosine own minus others | English, legible | Russian, cosine to own line | Russian, legible | Russian, stray stroke on the first word |
211
+ |---|---|---|---|---|---|---|
212
+ | Model of the site | 5% | 0.06 | 95% | 0.64 | 95% | 50% |
213
+ | **English and Russian model** | **65%** | **0.29** | **95%** | **0.67** | **90%** | **5%** |
214
+ | Another real line of the same writer | 85% | 0.49 | | | | |
215
+
216
+ Legible means a recogniser error rate of at most 0.05. The English hands are much closer to their writers than before but still well short of real lines. Without the best of four, English lines often lose spaces between words: 82% legible over all attempts. These numbers were taken on a GPU; the same seed gives other lines on a Mac.
217
+
218
  ## Evaluation
219
 
220
  240 lines of real notes from writers the generator never saw, each written from one reference line by the same writer. An LLM judge (Gemini 4 Argon) lists errors in each line; "clean" means no error of severity 2 or higher. Pairs are re-sampled for the 95% intervals.
 
248
 
249
  - **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
250
  - **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
251
+ - **English and Russian model:** the same, plus English pen lines of 158 writers of [BRUSH](https://dsd.cs.brown.edu/) (non-commercial research use only). Its style encoder and English recogniser were trained further on the same lines.
252
  - **Recogniser:** the tablet lines above, the same lines rendered to images and traced back, and traced lines of HWR200.
253
  - **Writer classifier:** tablet lines of 15 writers: 12 of the 13 above and three more.
254
  - **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
ckpt_ctc6/ctc_fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5c6f4a7609ea9d71811ae4d2d864e9d990610451b174bb707323e8318c79384e
3
+ size 19781805
ckpt_dl/u4s2/fp16/step020000_fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bc762ad76f6e887062076e40c291c658280d758a0fb66b13b140ecfd35d065b1
3
+ size 51085797
ckpt_semb4/best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7b17fc408f3fe89fb0431ba565673725155df2b3ef8b71b4f502a1585b181f8d
3
+ size 7279759
download_weights.py CHANGED
@@ -3,6 +3,7 @@
3
  python download_weights.py # the models that write pages, about 80 MB
4
  python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
5
  python download_weights.py --site # also the models of synthink.dev for write_line.py, about 75 MB more
 
6
  """
7
  import argparse
8
  import os
@@ -18,6 +19,13 @@ WRITER = [
18
  "ckpt_dl/glyph_cls2.pt",
19
  ]
20
  TRACER = ["trace/weights/*"]
 
 
 
 
 
 
 
21
  SITE = [
22
  "ckpt_dl/u3ref/fp16/step060000_fp16.pt",
23
  "ckpt_ctc4/ctc_fp16.pt",
@@ -29,9 +37,10 @@ def main():
29
  ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
30
  ap.add_argument("--tracer", action="store_true")
31
  ap.add_argument("--site", action="store_true")
 
32
  args = ap.parse_args()
33
  root = os.path.dirname(os.path.abspath(__file__))
34
- snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []) + (SITE if args.site else []), local_dir=root)
35
  print("weights are in", root)
36
 
37
 
 
3
  python download_weights.py # the models that write pages, about 80 MB
4
  python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
5
  python download_weights.py --site # also the models of synthink.dev for write_line.py, about 75 MB more
6
+ python download_weights.py --bilingual # also the English and Russian model for write_line.py --model bilingual, about 150 MB more
7
  """
8
  import argparse
9
  import os
 
19
  "ckpt_dl/glyph_cls2.pt",
20
  ]
21
  TRACER = ["trace/weights/*"]
22
+ BILINGUAL = [ # trained on BRUSH as well: non-commercial research only, see README
23
+ "ckpt_dl/u4s2/fp16/step020000_fp16.pt",
24
+ "ckpt_semb4/best.pt",
25
+ "ckpt_ctc6/ctc_fp16.pt",
26
+ "ckpt_ctc4/ctc_fp16.pt",
27
+ "ckpt_stylecls/embed_fp16.pt",
28
+ ]
29
  SITE = [
30
  "ckpt_dl/u3ref/fp16/step060000_fp16.pt",
31
  "ckpt_ctc4/ctc_fp16.pt",
 
37
  ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
38
  ap.add_argument("--tracer", action="store_true")
39
  ap.add_argument("--site", action="store_true")
40
+ ap.add_argument("--bilingual", action="store_true")
41
  args = ap.parse_args()
42
  root = os.path.dirname(os.path.abspath(__file__))
43
+ snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []) + (SITE if args.site else []) + (BILINGUAL if args.bilingual else []), local_dir=root)
44
  print("weights are in", root)
45
 
46
 
scrawler/common.py CHANGED
@@ -28,6 +28,9 @@ CKPT_PLANNER = os.path.join(ROOT, "ckpt_dl", "planner7", "best.pt")
28
  CKPT_SITE_TEXT = os.path.join(ROOT, "ckpt_dl", "u3ref", "fp16", "step060000_fp16.pt")
29
  CKPT_RECOGNISER = os.path.join(ROOT, "ckpt_ctc4", "ctc_fp16.pt")
30
  CKPT_CLASSIFIER = os.path.join(ROOT, "ckpt_stylecls", "embed_fp16.pt")
 
 
 
31
  CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
32
  CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
33
  CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
@@ -39,5 +42,6 @@ MATHJAX_DIR = os.path.join(ROOT, "tools", "mathjax")
39
  def load_checkpoint(path):
40
  if not os.path.exists(path):
41
  hint = " --site" if path in (CKPT_SITE_TEXT, CKPT_RECOGNISER, CKPT_CLASSIFIER) else ""
 
42
  raise SystemExit(f"{path} not found. Run: python download_weights.py{hint}")
43
  return torch.load(path, map_location=DEVICE, weights_only=True)
 
28
  CKPT_SITE_TEXT = os.path.join(ROOT, "ckpt_dl", "u3ref", "fp16", "step060000_fp16.pt")
29
  CKPT_RECOGNISER = os.path.join(ROOT, "ckpt_ctc4", "ctc_fp16.pt")
30
  CKPT_CLASSIFIER = os.path.join(ROOT, "ckpt_stylecls", "embed_fp16.pt")
31
+ CKPT_BILINGUAL_TEXT = os.path.join(ROOT, "ckpt_dl", "u4s2", "fp16", "step020000_fp16.pt")
32
+ CKPT_BILINGUAL_STYLE = os.path.join(ROOT, "ckpt_semb4", "best.pt")
33
+ CKPT_BILINGUAL_RECOGNISER = os.path.join(ROOT, "ckpt_ctc6", "ctc_fp16.pt")
34
  CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
35
  CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
36
  CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
 
42
  def load_checkpoint(path):
43
  if not os.path.exists(path):
44
  hint = " --site" if path in (CKPT_SITE_TEXT, CKPT_RECOGNISER, CKPT_CLASSIFIER) else ""
45
+ hint = " --bilingual" if path in (CKPT_BILINGUAL_TEXT, CKPT_BILINGUAL_STYLE, CKPT_BILINGUAL_RECOGNISER) else hint
46
  raise SystemExit(f"{path} not found. Run: python download_weights.py{hint}")
47
  return torch.load(path, map_location=DEVICE, weights_only=True)
scrawler/site_model.py CHANGED
@@ -9,6 +9,11 @@ Three things differ from scrawler/text_model.py:
9
  - Every line is sampled four times. A recogniser keeps the attempts that read best (character error rate within 0.02 of the
10
  lowest), and of those a writer classifier picks the one closest to the reference.
11
 
 
 
 
 
 
12
  Lines are arrays [n, 3] of (dx, dy, pen_up) with line height 1, as everywhere in this package.
13
  """
14
  import numpy as np
@@ -17,6 +22,7 @@ import torch.nn as nn
17
  import torch.nn.functional as F
18
 
19
  from .common import AMP, CKPT_CLASSIFIER, CKPT_PLANNER, CKPT_RECOGNISER, CKPT_SITE_TEXT, DEVICE, load_checkpoint
 
20
  from .style_encoder import SCALE, features, load_style_net, resample, style_vector
21
  from .text_model import ADVANCE, ALPHA_BAR, GUIDANCE, MAX_POINTS, NOISE_STEPS, SAMPLE_STEPS, SCALE_X, SCALE_Y
22
  from .text_model import Denoiser, Planner, _plan, time_embedding
@@ -25,6 +31,15 @@ ATTEMPTS = 4
25
  SPREAD_MIN, SPREAD_MAX = 0.6, 1.4
26
  CER_SLACK = 0.02
27
  REF_STEP = 0.05
 
 
 
 
 
 
 
 
 
28
 
29
 
30
  def to_strokes(line):
@@ -173,8 +188,12 @@ def _edit_rate(target, hypothesis):
173
 
174
 
175
  class SiteModel:
176
- def __init__(self):
177
- state = load_checkpoint(CKPT_SITE_TEXT)
 
 
 
 
178
  self.itos = state["itos"]
179
  self.stoi = {s: i for i, s in enumerate(self.itos)}
180
  self.null_token = len(self.itos)
@@ -186,11 +205,13 @@ class SiteModel:
186
  self.planner = Planner(len(state["itos"])).to(DEVICE)
187
  self.planner.load_state_dict(state["m"])
188
  self.planner.eval()
189
- self.style_net = load_style_net()
190
- state = load_checkpoint(CKPT_RECOGNISER)
191
- self.recogniser = Recogniser(len(state["itos"])).to(DEVICE)
192
- self.recogniser.load_state_dict({k: v.float() for k, v in state["m"].items()})
193
- self.recogniser.eval()
 
 
194
  state = load_checkpoint(CKPT_CLASSIFIER)
195
  self.embedder = WriterEmbedder().to(DEVICE)
196
  self.embedder.load_state_dict({k: v.float() for k, v in state["m"].items()})
@@ -208,11 +229,14 @@ class SiteModel:
208
  d = torch.expm1(out).clamp(min=1).round().long().cpu().numpy()
209
  return [int(v) for v in d[0, 1:-1]]
210
 
 
 
 
211
  @torch.no_grad()
212
- def _read(self, lines):
213
  """Equal-length lines -> the recognised character ids of each."""
214
  x = np.stack([np.stack([q[:, 0] * 20.0, q[:, 1] * 20.0, q[:, 2], np.cumsum(q[:, 1])], 1) for q in lines]).astype(np.float32)
215
- best = self.recogniser(torch.from_numpy(x).to(DEVICE)).argmax(-1).cpu().numpy()
216
  out = []
217
  for row in best:
218
  ids, prev = [], 0
@@ -259,6 +283,9 @@ class SiteModel:
259
  raise ValueError("no characters the model knows")
260
  ids = [self.stoi[c] for c in text]
261
  durations = [max(1, int(round(d * ref["spread"]))) for d in self._durations(text, ref["semb"])]
 
 
 
262
  length = sum(durations)
263
  if length > MAX_POINTS:
264
  raise ValueError(f"the line is too long: {length} pen points, the model takes {MAX_POINTS}. Shorten the text.")
@@ -304,7 +331,7 @@ class SiteModel:
304
  d[-1, 2] = 1
305
  lines.append(d)
306
 
307
- cer = np.array([_edit_rate(ids, h) for h in self._read(lines)], np.float32)
308
  cos = np.array([float(self._embed(q) @ ref["embedding"]) for q in lines], np.float32)
309
  legible = np.flatnonzero(cer <= cer.min() + CER_SLACK)
310
  pick = int(legible[np.argmax(cos[legible])])
 
9
  - Every line is sampled four times. A recogniser keeps the attempts that read best (character error rate within 0.02 of the
10
  lowest), and of those a writer classifier picks the one closest to the reference.
11
 
12
+ SiteModel("bilingual") is the English and Russian model: the site model trained further on English pen lines of BRUSH, with a
13
+ style encoder that also tells English writers apart. The recipe is the same, with two changes measured on it: an English line is
14
+ read by a recogniser that knows English handwriting, and a Russian line that starts with a capital letter gets for that letter
15
+ the length the planner gives it inside a phrase (without this the first word often comes out with a stray stroke).
16
+
17
  Lines are arrays [n, 3] of (dx, dy, pen_up) with line height 1, as everywhere in this package.
18
  """
19
  import numpy as np
 
22
  import torch.nn.functional as F
23
 
24
  from .common import AMP, CKPT_CLASSIFIER, CKPT_PLANNER, CKPT_RECOGNISER, CKPT_SITE_TEXT, DEVICE, load_checkpoint
25
+ from .common import CKPT_BILINGUAL_RECOGNISER, CKPT_BILINGUAL_STYLE, CKPT_BILINGUAL_TEXT, CKPT_STYLE
26
  from .style_encoder import SCALE, features, load_style_net, resample, style_vector
27
  from .text_model import ADVANCE, ALPHA_BAR, GUIDANCE, MAX_POINTS, NOISE_STEPS, SAMPLE_STEPS, SCALE_X, SCALE_Y
28
  from .text_model import Denoiser, Planner, _plan, time_embedding
 
31
  SPREAD_MIN, SPREAD_MAX = 0.6, 1.4
32
  CER_SLACK = 0.02
33
  REF_STEP = 0.05
34
+ MODELS = {
35
+ "site": dict(text=CKPT_SITE_TEXT, style=CKPT_STYLE, recognisers=dict(any=CKPT_RECOGNISER)),
36
+ "bilingual": dict(text=CKPT_BILINGUAL_TEXT, style=CKPT_BILINGUAL_STYLE, recognisers=dict(ru=CKPT_RECOGNISER, any=CKPT_BILINGUAL_RECOGNISER)),
37
+ }
38
+ FIRST_LETTER_PREFIX = "и " # bilingual model, Russian: the first letter is planned as if the line went on after this
39
+
40
+
41
+ def is_russian(text):
42
+ return any("а" <= c.lower() <= "я" or c in "ёЁ" for c in text)
43
 
44
 
45
  def to_strokes(line):
 
188
 
189
 
190
  class SiteModel:
191
+ def __init__(self, kind="site"):
192
+ if kind not in MODELS:
193
+ raise ValueError(f"unknown model {kind!r}: {' or '.join(MODELS)}")
194
+ self.kind = kind
195
+ paths = MODELS[kind]
196
+ state = load_checkpoint(paths["text"])
197
  self.itos = state["itos"]
198
  self.stoi = {s: i for i, s in enumerate(self.itos)}
199
  self.null_token = len(self.itos)
 
205
  self.planner = Planner(len(state["itos"])).to(DEVICE)
206
  self.planner.load_state_dict(state["m"])
207
  self.planner.eval()
208
+ self.style_net = load_style_net(paths["style"])
209
+ self.recognisers = {}
210
+ for name, path in paths["recognisers"].items():
211
+ state = load_checkpoint(path)
212
+ net = Recogniser(len(state["itos"])).to(DEVICE)
213
+ net.load_state_dict({k: v.float() for k, v in state["m"].items()})
214
+ self.recognisers[name] = net.eval()
215
  state = load_checkpoint(CKPT_CLASSIFIER)
216
  self.embedder = WriterEmbedder().to(DEVICE)
217
  self.embedder.load_state_dict({k: v.float() for k, v in state["m"].items()})
 
229
  d = torch.expm1(out).clamp(min=1).round().long().cpu().numpy()
230
  return [int(v) for v in d[0, 1:-1]]
231
 
232
+ def _recogniser(self, text):
233
+ return self.recognisers["ru" if "ru" in self.recognisers and is_russian(text) else "any"]
234
+
235
  @torch.no_grad()
236
+ def _read(self, lines, recogniser=None):
237
  """Equal-length lines -> the recognised character ids of each."""
238
  x = np.stack([np.stack([q[:, 0] * 20.0, q[:, 1] * 20.0, q[:, 2], np.cumsum(q[:, 1])], 1) for q in lines]).astype(np.float32)
239
+ best = (recogniser or self.recognisers["any"])(torch.from_numpy(x).to(DEVICE)).argmax(-1).cpu().numpy()
240
  out = []
241
  for row in best:
242
  ids, prev = [], 0
 
283
  raise ValueError("no characters the model knows")
284
  ids = [self.stoi[c] for c in text]
285
  durations = [max(1, int(round(d * ref["spread"]))) for d in self._durations(text, ref["semb"])]
286
+ if self.kind == "bilingual" and text[0].isupper() and is_russian(text[0]):
287
+ inside = self._durations(FIRST_LETTER_PREFIX + text, ref["semb"])[len(FIRST_LETTER_PREFIX)]
288
+ durations[0] = min(durations[0], max(1, int(round(max(1, inside) * ref["spread"]))))
289
  length = sum(durations)
290
  if length > MAX_POINTS:
291
  raise ValueError(f"the line is too long: {length} pen points, the model takes {MAX_POINTS}. Shorten the text.")
 
331
  d[-1, 2] = 1
332
  lines.append(d)
333
 
334
+ cer = np.array([_edit_rate(ids, h) for h in self._read(lines, self._recogniser(text))], np.float32)
335
  cos = np.array([float(self._embed(q) @ ref["embedding"]) for q in lines], np.float32)
336
  legible = np.flatnonzero(cer <= cer.min() + CER_SLACK)
337
  pick = int(legible[np.argmax(cos[legible])])
scrawler/style_encoder.py CHANGED
@@ -83,8 +83,8 @@ class StyleNet(nn.Module):
83
  return self.head(torch.cat([mean, top], -1))
84
 
85
 
86
- def load_style_net():
87
- state = load_checkpoint(CKPT_STYLE)
88
  net = StyleNet(out_dim=256, d=state["D"]).to(DEVICE)
89
  net.load_state_dict(state["m"])
90
  net.eval()
 
83
  return self.head(torch.cat([mean, top], -1))
84
 
85
 
86
+ def load_style_net(path=CKPT_STYLE):
87
+ state = load_checkpoint(path)
88
  net = StyleNet(out_dim=256, d=state["D"]).to(DEVICE)
89
  net.load_state_dict(state["m"])
90
  net.eval()
write_line.py CHANGED
@@ -11,7 +11,8 @@ In the folder `out` (it must not exist yet):
11
  - attempts.png: all attempts, the chosen one marked with a bar on the left
12
  - strokes.json: every attempt as strokes, with its character error rate and its closeness to the reference
13
 
14
- The recipe is in scrawler/site_model.py. The style comes from --ref (an .npy from make_ref.py). --ref-text is what is written
 
15
  in that line; it sets how wide the writer writes. Without it the recogniser reads the reference, which is right in most
16
  letters but not all, so give the text when you know it.
17
  """
@@ -85,6 +86,7 @@ def main():
85
  ap.add_argument("out", help="output folder, must not exist")
86
  ap.add_argument("--ref", default=DEFAULT_REF, help="the style: one line of handwriting, [n, 3] .npy from make_ref.py")
87
  ap.add_argument("--ref-text", help="what is written in the reference line; read by the recogniser if not given")
 
88
  ap.add_argument("--attempts", type=int, default=ATTEMPTS, help="samples per line; the best one is kept")
89
  ap.add_argument("--seed", type=int, default=0, help="another seed gives another writing of the same text")
90
  ap.add_argument("--pen", type=float, default=1.1, help="pen width in points")
@@ -97,7 +99,7 @@ def main():
97
  else:
98
  texts = [args.text.strip()]
99
 
100
- model = SiteModel()
101
  reference = np.load(args.ref).astype(np.float32)
102
  ref_text = args.ref_text
103
  if ref_text is None and os.path.abspath(args.ref) == os.path.abspath(DEFAULT_REF):
 
11
  - attempts.png: all attempts, the chosen one marked with a bar on the left
12
  - strokes.json: every attempt as strokes, with its character error rate and its closeness to the reference
13
 
14
+ --model bilingual writes English as well as Russian (python download_weights.py --bilingual); its weights are for
15
+ non-commercial research only, see README. The recipe is in scrawler/site_model.py. The style comes from --ref (an .npy from make_ref.py). --ref-text is what is written
16
  in that line; it sets how wide the writer writes. Without it the recogniser reads the reference, which is right in most
17
  letters but not all, so give the text when you know it.
18
  """
 
86
  ap.add_argument("out", help="output folder, must not exist")
87
  ap.add_argument("--ref", default=DEFAULT_REF, help="the style: one line of handwriting, [n, 3] .npy from make_ref.py")
88
  ap.add_argument("--ref-text", help="what is written in the reference line; read by the recogniser if not given")
89
+ ap.add_argument("--model", default="site", choices=["site", "bilingual"], help="site: the model of synthink.dev; bilingual: English and Russian")
90
  ap.add_argument("--attempts", type=int, default=ATTEMPTS, help="samples per line; the best one is kept")
91
  ap.add_argument("--seed", type=int, default=0, help="another seed gives another writing of the same text")
92
  ap.add_argument("--pen", type=float, default=1.1, help="pen width in points")
 
99
  else:
100
  texts = [args.text.strip()]
101
 
102
+ model = SiteModel(args.model)
103
  reference = np.load(args.ref).astype(np.float32)
104
  ref_text = args.ref_text
105
  if ref_text is None and os.path.abspath(args.ref) == os.path.abspath(DEFAULT_REF):