levossadtchi commited on
Commit
877ff59
·
verified ·
1 Parent(s): a34a22a

README rewritten around what to use; same code as GitHub

Browse files
Files changed (3) hide show
  1. README.md +154 -125
  2. download_weights.py +1 -1
  3. write_line.py +3 -1
README.md CHANGED
@@ -12,43 +12,57 @@ tags:
12
 
13
  # Scrawler
14
 
15
- Turns a text file with LaTeX formulas into handwritten notebook pages. The style comes from **one line** of the writer's handwriting, with no fine-tuning.
16
 
17
  ![A page written by a fine-tuned version of the models](examples/demo_authors_hand/page_05.png)
18
 
19
  *The page above is in the author's hand, written by personal fine-tuned models that are **not** released. For what the released weights give out of the box, see [this page](examples/demo_released_weights/page_05.png).*
20
 
21
- The output is pen strokes, not pixels, so the PDF is sharp at any zoom. The models write Russian prose and mathematics.
22
 
23
- This repository holds the weights and a copy of the code for writing pages. The code lives on [GitHub](https://github.com/Levos06/Scrawler-open), and `download_weights.py` there fetches the weights from here. Training code and data are not included.
 
 
 
 
 
 
 
 
 
 
24
 
25
  ## Quick start
26
 
27
- Requires Python and, for formulas, Node.js.
28
 
29
  ```bash
30
  git clone https://github.com/Levos06/Scrawler-open.git
31
  cd Scrawler-open
32
  pip install -r requirements.txt
33
- (cd tools/mathjax && npm ci)
34
- python download_weights.py
35
  python write_pages.py examples/physics_oscillator.txt out
36
  ```
37
 
38
- This writes to `out/` (the folder must not exist yet):
39
 
40
- - `pages.pdf`: vector, every stroke a smooth curve
41
- - `page_NN.svg` and `page_NN.png` (300 dpi)
42
- - `strokes.json`: the strokes; `python render_pages.py out/strokes.json thick --pen 1.4` redraws them without the models
43
- - `report.json`: per-line timing and what wrote each glyph
 
44
 
45
- The code selects CUDA, MPS or CPU by itself; set `SCRAWLER_DEVICE` to force one. Results depend on the device: the same seed gives the same pages only on the same device. Tested on Apple Silicon (MPS and CPU); the CUDA path has not been run with this code.
 
 
 
46
 
47
- The example above is 11 pages and took under 7 minutes on an Apple M5 Max (MPS). The CPU is much slower: a 4-page test with many formulas took 74 minutes.
48
 
49
- ## Writing in your own hand
50
 
51
- You need one line of your handwriting: 30-50 characters of ordinary prose. There are two ways to get it into the models.
52
 
53
  ### From a photo or scan
54
 
@@ -57,16 +71,13 @@ Crop the photo to **one line** and run it through the [tracer](trace/README.md),
57
  ```bash
58
  python download_weights.py --tracer
59
  pip install -r trace/requirements.txt
60
- python trace/scrawler_trace.py my_line.png --out my_line.json
61
  python make_ref.py my_line.json my_line.npy
62
- python write_pages.py notes.txt out --ref my_line.npy
63
  ```
64
 
65
- For a good result:
66
-
67
  - Crop tightly to one line. Parts of the lines above and below can get into the trace.
68
  - Use dark pen on light paper. Colour photos, dark backgrounds and low contrast work poorly.
69
- - Look at the trace before using it: add `--svg my_line.svg` to the tracer command to see the strokes.
70
 
71
  ![A scan, its trace and new text in that style](examples/from_scan.png)
72
 
@@ -76,19 +87,24 @@ The tracer weights have their own licence caveat, see [its training data](trace/
76
 
77
  ### From a tablet
78
 
79
- If the line was written in a note app, its pen strokes are in the app's file and no tracer is needed. `tablet/app_line.py` reads GoodNotes, Notewise, Huawei Notes, Kilonotes and Samsung Notes. The recipe for each app, and for the apps that are not read, is in [tablet/README.md](tablet/README.md).
80
 
81
  ```bash
82
  python tablet/app_line.py notes.goodnotes my_line.json --page 3 --line 2 --svg check.svg
83
  python make_ref.py my_line.json my_line.npy
84
- python write_pages.py notes.txt out --ref my_line.npy
85
  ```
86
 
87
- If you have the strokes from somewhere else, write them as a JSON list of strokes, each a list of `[x, y]` points with y pointing down, in writing order: that is what `make_ref.py` takes.
 
 
 
 
88
 
89
- The default style is one traced line of the open HWR200 dataset.
 
 
90
 
91
- ## Input format
92
 
93
  ```text
94
  # A heading
@@ -100,7 +116,16 @@ An empty line above makes a paragraph gap.
100
 
101
  Long lines are wrapped at about 42 characters (`--wrap`), never inside `$...$`. Formulas use the LaTeX that MathJax understands. `python write_pages.py --help` lists the other options.
102
 
103
- ## How it works
 
 
 
 
 
 
 
 
 
104
 
105
  | On the line | Written by |
106
  |---|---|
@@ -108,150 +133,154 @@ Long lines are wrapped at about 42 characters (`--wrap`), never inside `$...$`.
108
  | formula only | **Formula module**: MathJax places every glyph, a diffusion **glyph model** draws each one, and a classifier picks the most recognisable variant |
109
  | text + `$formula$` | The text model writes the line, then the formula spans are replaced by the formula module |
110
 
111
- The style vector comes from a line-level writer encoder. Fraction bars, radicals, arrows and `|x|` are drawn as pen lines by code, not by a model, so they do not take the writer's style.
112
-
113
- | Path | What |
114
- |---|---|
115
- | `write_pages.py` | text file -> pages: wrapping, page layout, the order of the steps |
116
- | `write_line.py` | one line of text in the hand of a reference line, the way the site does it |
117
- | `scrawler/text_model.py` | text model and duration planner |
118
- | `scrawler/site_model.py` | the site's text model, recogniser, writer classifier and the choice of the best attempt |
119
- | `scrawler/style_encoder.py` | one line -> style vector |
120
- | `scrawler/glyph_model.py` | glyph model and the classifier that picks variants |
121
- | `scrawler/formula.py` | formula layout, pen-line marks, inserting formulas into text |
122
- | `scrawler/render.py`, `render_pages.py` | strokes -> PDF, SVG, PNG |
123
- | `make_ref.py` | your strokes -> a style line |
124
- | `trace/` | the tracer, one file |
125
- | `tablet/` | a line out of a note app: readers of GoodNotes, Notewise, Huawei Notes, Kilonotes and Samsung Notes files |
126
- | `tools/mathjax/layout.mjs` | LaTeX -> glyph boxes |
127
- | `data/glyph_stroke_counts.json` | per glyph class, how many real glyphs had 1, 2, ... strokes; counts only |
128
-
129
- ## Weights
130
 
131
- | Path | What | Size |
132
- |---|---|---|
133
- | `ckpt_dl/b4styc/fp16/step003000_fp16.pt` | text model | 45 MB |
134
- | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev), used by `write_line.py`, see below | 51 MB |
135
- | `ckpt_ctc4/ctc_fp16.pt` | recogniser, for `write_line.py` | 20 MB |
136
- | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer, for `write_line.py` | 4 MB |
137
- | `ckpt_dl/u4s2/fp16/step020000_fp16.pt` | English and Russian text model, `write_line.py --model bilingual`; **non-commercial research only**, see below | 51 MB |
138
- | `ckpt_semb4/best.pt` | its style encoder, which also tells English writers apart | 7 MB |
139
- | `ckpt_ctc6/ctc_fp16.pt` | its recogniser for English lines | 20 MB |
140
- | `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
141
- | `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
142
- | `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
143
- | `ckpt_dl/glyph_cls2.pt` | glyph classifier | 1 MB |
144
- | `trace/weights/` | tracer and ink restorer, only for the photo route | 62 MB |
145
-
146
- ## The model on the site
147
-
148
- [synthink.dev](https://synthink.dev) writes a line in your handwriting right in the browser: [try it there](https://synthink.dev/generate/). Every model the site runs is in this release, and `write_line.py` does the same on your machine: one line of your handwriting from a photo or a tablet goes in, a new line in that hand comes out.
149
 
150
  ```bash
151
- python download_weights.py --site --tracer
152
- pip install -r trace/requirements.txt
153
- python trace/scrawler_trace.py my_line.png --out my_line.json --svg my_line.svg # photo of one line -> pen strokes
154
- python make_ref.py my_line.json my_line.npy # pen strokes -> reference line
155
  python write_line.py "Мороз и солнце; день чудесный!" out_line --ref my_line.npy --ref-text "what your line says"
 
 
156
  ```
157
 
158
- For a line from a tablet, the tracer step is replaced by `tablet/app_line.py`, see [tablet/README.md](tablet/README.md).
159
 
160
- `out_line/` gets `line.svg`, `line.png` and `line.pdf` with the chosen attempt, `attempts.png` with all four, and `strokes.json`. The first argument can also be a `.txt` file with one line of text per row. Formulas are not written here. A line takes about 9 seconds on an Apple M5 Max (MPS).
161
 
162
- | On the site | Here | What it does |
163
- |---|---|---|
164
- | `den.onnx`, `refpts.onnx` | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model that also attends to the pen points of the reference line |
165
- | `planner.onnx` | `ckpt_dl/planner7/best.pt` | pen points per character |
166
- | `semb.onnx` | `ckpt_semb3_all/best.pt` | style vector of the reference line |
167
- | `rec.onnx` | `ckpt_ctc4/ctc_fp16.pt` | recogniser: how legible an attempt is |
168
- | `scls.onnx` | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer: how close an attempt is to the reference |
169
- | `fix.onnx`, `tenc.onnx`, `tstep.onnx` | `trace/weights/` | ink restorer and tracer, for the photo route |
170
 
171
- The weights are in this repository; the site runs the same weights converted to ONNX.
172
 
173
- The recipe, in `scrawler/site_model.py`:
 
 
174
 
175
- 1. The planner gives the number of pen points for every character. These are multiplied by the writer's spread: the length of the reference line in points over the length the planner gives for the text of that line, clipped to [0.6, 1.4]. This is why `--ref-text` is asked for. Without it the recogniser reads the reference, which gets most letters right but not all.
176
- 2. The text model writes the line four times, with guidance 1.5 and 20 steps.
177
- 3. The recogniser keeps the attempts whose character error rate is within 0.02 of the lowest, and of those the writer classifier picks the one closest to the reference.
178
 
179
- The code was checked against the ONNX files the site serves: from the same noise the pen points agree to 0.0003 of the line height and the same attempt is chosen.
180
 
181
- The text model is the text model of this release trained for 60 000 more steps, with two additions: traced lines of 190 HWR200 writers in the training data, and attention from every generated point to the pen points of the reference line. The model reads the reference line itself, not only its style vector.
182
 
183
- Measured on traced lines of 9 writers the model never saw: the cosine of a writer classifier between a generated line and the writer's own line is 0.71, against 0.47 for the release text model; to other writers it is 0.46 for both. Legibility did not change: 47.1% clean lines on the set of the table below, the same as "Text model alone". By eye the gain in likeness is modest.
 
 
 
 
 
 
 
184
 
185
- `write_pages.py` does not use the site model: pages and formulas are written by the text model of the first row of the weights table.
186
 
187
- ## English and Russian model
188
 
189
- The model of the site writes English letters legibly but not in the writer's hand: it never saw English pen lines, and its style encoder cannot tell English writers apart (5% in the table below). `--model bilingual` writes both languages from one line of the writer's hand.
190
 
191
- ```bash
192
- python download_weights.py --bilingual
193
- python write_line.py "One line, and the rest is written by your hand." out_line --model bilingual --ref my_line.npy --ref-text "what your line says"
194
- ```
195
 
196
- The reference line can be in either language, and so can the text. The code for this model is in this repository; the GitHub copy does not have it yet.
 
 
197
 
198
- **Licence.** It was trained on [BRUSH](https://dsd.cs.brown.edu/) (Brown University), whose terms allow non-commercial research use only, and the weights inherit that: `ckpt_dl/u4s2/`, `ckpt_semb4/` and `ckpt_ctc6/` are for non-commercial research only, on top of CC BY-NC 4.0.
199
 
200
- How it was made, from the model of the site, in two stages:
201
 
202
- 1. **Text model, stage 1:** 60 000 more steps on the site model's data, with underlines removed from the tablet lines (they made the first word of a line come out underlined), plus English pen lines of 158 BRUSH writers. 12 more BRUSH writers were kept out of all training to measure on.
203
- 2. **Style encoder:** the release style encoder trained further to tell BRUSH writers apart, while staying close to its old vectors on Russian lines.
204
- 3. **Text model, stage 2:** 20 000 more steps with the new style encoder.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
205
 
206
- The recogniser for English lines is the site's recogniser trained further on the same mix. The recipe of `write_line.py` is the site's, with two changes: an English line is chosen by the English recogniser, and a Russian line that starts with a capital letter gets for that letter the length the planner gives it inside a phrase.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
207
 
208
- Measured on 20 hands per language, one reference line each, best of four attempts. English: 12 BRUSH writers the model never saw and 8 it saw. The writer measure here is a classifier trained on both languages, and "found" means the generated line is closest to its own reference among the 20.
209
 
210
- | | English, own writer found | English, cosine own minus others | English, legible | Russian, cosine to own line | Russian, legible | Russian, stray stroke on the first word |
211
- |---|---|---|---|---|---|---|
212
- | Model of the site | 5% | 0.06 | 95% | 0.64 | 95% | 50% |
213
- | **English and Russian model** | **65%** | **0.29** | **95%** | **0.67** | **90%** | **5%** |
214
- | Another real line of the same writer | 85% | 0.49 | | | | |
215
 
216
- Legible means a recogniser error rate of at most 0.05. The English hands are much closer to their writers than before but still well short of real lines. Without the best of four, English lines often lose spaces between words: 82% legible over all attempts. These numbers were taken on a GPU; the same seed gives other lines on a Mac.
 
 
 
217
 
218
- ## Evaluation
219
 
220
- 240 lines of real notes from writers the generator never saw, each written from one reference line by the same writer. An LLM judge (Gemini 4 Argon) lists errors in each line; "clean" means no error of severity 2 or higher. Pairs are re-sampled for the 95% intervals.
221
 
222
  | Run | Clean, % [95%] | Prose | Mixed | Formula |
223
  |---|---|---|---|---|
224
  | Real handwriting of the same lines | 45.0 [38.8, 51.2] | 47.0 | 35.7 | 51.4 |
225
  | Text model alone | 47.1 [40.8, 53.3] | 63.0 | 41.4 | 30.0 |
226
- | **This release** | **61.3 [55.0, 67.5]** | 63.0 | 54.3 | 65.7 |
227
  | Same, with a library of real glyphs (not released) | 65.4 [59.2, 71.2] | 62.0 | 58.6 | 77.1 |
228
 
229
- How to read this table:
230
-
231
- - The score rewards legibility. Scoring above real notes means the output is tidier than hurried human writing, not that it looks more human.
232
  - Prose is the same set of generated lines in every run but the first, so 62-63 shows judge noise of a few points.
233
  - The private library variant scores 4.2 points higher, 95% interval [-9.6, +1.2] for the difference. That is within the noise, but a small loss from removing it is possible.
234
 
235
- Style similarity by the writer encoder (cosine to the writer's real line, one reference line, 3 unseen writers): 0.59-0.68 generated, against 0.55-0.70 between real lines of the same writers.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
236
 
237
  ## Known weaknesses
238
 
239
- - The text model sometimes swaps, drops or adds a letter. There is no spell check.
240
- - The site model (`write_line.py`) often adds a stray stroke to the first word of a line, an underline or a strike-through: a stroke detector found one in the chosen attempt for 10 of 20 test hands.
241
- - A one-letter formula inside a sentence can collide with the neighbouring word.
242
- - About 2% of glyphs are the best of the variants the classifier rejected, and may be misshapen.
243
- - Rare symbols without their own class are drawn as a similar one (for example `\simeq` as `\approx`).
244
- - A heading of one short word may come out smaller than the other lines.
245
- - A line of several formulas with almost no plain text between them is written by the text model alone instead of the formula module (`fallback_text_model` in `report.json`).
246
 
247
  ## Training data
248
 
249
- - **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
250
- - **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
251
- - **English and Russian model:** the same, plus English pen lines of 158 writers of [BRUSH](https://dsd.cs.brown.edu/) (non-commercial research use only). Its style encoder and English recogniser were trained further on the same lines.
 
252
  - **Recogniser:** the tablet lines above, the same lines rendered to images and traced back, and traced lines of HWR200.
253
  - **Writer classifier:** tablet lines of 15 writers: 12 of the 13 above and three more.
254
- - **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
255
  - **Tracer:** a mix of tablet notes and open sets, some with non-commercial or ShareAlike terms. See [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md), including the open question about ShareAlike.
256
 
257
  ## Responsible use
@@ -260,4 +289,4 @@ This is for notes, experiments and research. Do not use it to forge signatures,
260
 
261
  ## License
262
 
263
- Code: Apache-2.0 ([LICENSE](LICENSE)). Weights: CC BY-NC 4.0 ([LICENSE-WEIGHTS](LICENSE-WEIGHTS)); the tracer weights carry further conditions from their training data, see [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md). Third-party components: [NOTICE](NOTICE).
 
12
 
13
  # Scrawler
14
 
15
+ Handwriting from **one line** of the writer's hand, with no fine-tuning: whole pages of notes with LaTeX formulas, or single lines the way [synthink.dev](https://synthink.dev) writes them. The output is pen strokes, not pixels, so the PDF and SVG are sharp at any zoom.
16
 
17
  ![A page written by a fine-tuned version of the models](examples/demo_authors_hand/page_05.png)
18
 
19
  *The page above is in the author's hand, written by personal fine-tuned models that are **not** released. For what the released weights give out of the box, see [this page](examples/demo_released_weights/page_05.png).*
20
 
21
+ This repository holds the weights and a copy of the code. The code also lives on [GitHub](https://github.com/Levos06/Scrawler-open), and `download_weights.py` fetches the weights from here. Training code and data are not included.
22
 
23
+ ## What to use
24
+
25
+ | You want | Run | Model | Languages | What it cannot do |
26
+ |---|---|---|---|---|
27
+ | Pages of notes with LaTeX formulas | `write_pages.py` | pages model | Russian prose and mathematics | the hand is copied only loosely |
28
+ | A line, or a few lines, in someone's hand, as on [the site](https://synthink.dev/generate/) | `write_line.py` | site model | Russian | no formulas, no page layout |
29
+ | The same in English or Russian | `write_line.py --model bilingual` | bilingual model | English and Russian | no formulas, no page layout; **non-commercial research only** |
30
+
31
+ English pages with formulas are not possible yet: `write_pages.py` runs the pages model only. English prose through the pages model was not measured.
32
+
33
+ Every route needs one line of the writer's hand as the style, the **reference line**. It can come from a photo, from a note app on a tablet, or from your own stroke data: see [Your reference line](#your-reference-line). Without one, a traced line of the open HWR200 dataset is used.
34
 
35
  ## Quick start
36
 
37
+ Requires Python; for formulas also Node.js.
38
 
39
  ```bash
40
  git clone https://github.com/Levos06/Scrawler-open.git
41
  cd Scrawler-open
42
  pip install -r requirements.txt
43
+ python download_weights.py # the pages model, about 80 MB
44
+ (cd tools/mathjax && npm ci) # formula layout
45
  python write_pages.py examples/physics_oscillator.txt out
46
  ```
47
 
48
+ For the other routes, fetch their weights first. The flags can be combined.
49
 
50
+ | Flag | Adds | Size |
51
+ |---|---|---|
52
+ | `--site` | site model, for `write_line.py` | about 75 MB |
53
+ | `--bilingual` | bilingual model, for `write_line.py --model bilingual` | about 100 MB |
54
+ | `--tracer` | the tracer, to take a reference line from a photo | about 60 MB |
55
 
56
+ ```bash
57
+ python download_weights.py --site
58
+ python write_line.py "Мороз и солнце; день чудесный!" out_line
59
+ ```
60
 
61
+ The code selects CUDA, MPS or CPU by itself; set `SCRAWLER_DEVICE` to force one. Results depend on the device: the same seed gives the same output only on the same device. Tested on Apple Silicon (MPS and CPU); the CUDA path has not been run with this code.
62
 
63
+ ## Your reference line
64
 
65
+ One line of ordinary prose, 30-50 characters, without formulas, underlines or corrections. Whatever the source, it ends as an `.npy` file from `make_ref.py`, which every script takes as `--ref`.
66
 
67
  ### From a photo or scan
68
 
 
71
  ```bash
72
  python download_weights.py --tracer
73
  pip install -r trace/requirements.txt
74
+ python trace/scrawler_trace.py my_line.png --out my_line.json --svg my_line.svg
75
  python make_ref.py my_line.json my_line.npy
 
76
  ```
77
 
 
 
78
  - Crop tightly to one line. Parts of the lines above and below can get into the trace.
79
  - Use dark pen on light paper. Colour photos, dark backgrounds and low contrast work poorly.
80
+ - Look at `my_line.svg` before going on: it shows the traced strokes.
81
 
82
  ![A scan, its trace and new text in that style](examples/from_scan.png)
83
 
 
87
 
88
  ### From a tablet
89
 
90
+ If the line was written in a note app, its pen strokes are in the app's file and no tracer is needed. `tablet/app_line.py` reads GoodNotes, Notewise, Huawei Notes, Kilonotes and Samsung Notes. How to export from each app, and what to do with the others, is in [tablet/README.md](tablet/README.md).
91
 
92
  ```bash
93
  python tablet/app_line.py notes.goodnotes my_line.json --page 3 --line 2 --svg check.svg
94
  python make_ref.py my_line.json my_line.npy
 
95
  ```
96
 
97
+ ### Your own stroke data
98
+
99
+ Write the strokes as a JSON list of strokes, each a list of `[x, y]` points with y pointing down, in writing order, in any units. That is what `make_ref.py` takes.
100
+
101
+ ## Pages with formulas: `write_pages.py`
102
 
103
+ ```bash
104
+ python write_pages.py notes.txt out --ref my_line.npy
105
+ ```
106
 
107
+ The input:
108
 
109
  ```text
110
  # A heading
 
116
 
117
  Long lines are wrapped at about 42 characters (`--wrap`), never inside `$...$`. Formulas use the LaTeX that MathJax understands. `python write_pages.py --help` lists the other options.
118
 
119
+ `out/` (the folder must not exist yet) gets:
120
+
121
+ - `pages.pdf`: vector, every stroke a smooth curve
122
+ - `page_NN.svg` and `page_NN.png` (300 dpi)
123
+ - `strokes.json`: the strokes; `python render_pages.py out/strokes.json thick --pen 1.4` redraws them without the models
124
+ - `report.json`: per-line timing and what wrote each glyph
125
+
126
+ The example of the quick start is 11 pages and took under 7 minutes on an Apple M5 Max (MPS). The CPU is much slower: a 4-page test with many formulas took 74 minutes.
127
+
128
+ How a page is written:
129
 
130
  | On the line | Written by |
131
  |---|---|
 
133
  | formula only | **Formula module**: MathJax places every glyph, a diffusion **glyph model** draws each one, and a classifier picks the most recognisable variant |
134
  | text + `$formula$` | The text model writes the line, then the formula spans are replaced by the formula module |
135
 
136
+ The style vector comes from a line-level style encoder. Fraction bars, radicals, arrows and `|x|` are drawn as pen lines by code, not by a model, so they do not take the writer's style.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137
 
138
+ ## Lines in a hand: `write_line.py`
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
139
 
140
  ```bash
 
 
 
 
141
  python write_line.py "Мороз и солнце; день чудесный!" out_line --ref my_line.npy --ref-text "what your line says"
142
+ python write_line.py lines.txt out_line --ref my_line.npy --ref-text "..." # one line of text per row
143
+ python write_line.py "One line, and the rest is written by your hand." out_line --model bilingual --ref my_line.npy --ref-text "..."
144
  ```
145
 
146
+ `out_line/` gets `line.svg`, `line.png` and `line.pdf` with the chosen attempt of every line, `attempts.png` with all attempts, and `strokes.json`. A line takes 5-10 seconds on an Apple M5 Max (MPS).
147
 
148
+ Give `--ref-text`, what is written in the reference line, when you know it. It sets how wide the writer writes; without it the recogniser reads the reference, which gets most letters right but not all.
149
 
150
+ ### The recipe
 
 
 
 
 
 
 
151
 
152
+ The same for both models, in `scrawler/site_model.py`:
153
 
154
+ 1. The planner gives the number of pen points for every character. These are multiplied by the writer's spread: the length of the reference line in points over the length the planner gives for the text of that line, clipped to [0.6, 1.4].
155
+ 2. The text model writes the line four times, with guidance 1.5 and 20 steps. It attends to the pen points of the reference line itself, not only to its style vector.
156
+ 3. A recogniser keeps the attempts whose character error rate is within 0.02 of the lowest, and of those a writer classifier picks the one closest to the reference.
157
 
158
+ The bilingual model changes two things, both measured on it: an English line is chosen by its English recogniser, and a Russian line that starts with a capital letter gets for that letter the length the planner gives it inside a phrase (without this the first word often gets a stray stroke).
 
 
159
 
160
+ ### Site model
161
 
162
+ This is what [synthink.dev](https://synthink.dev/generate/) runs in the browser, as ONNX converted from these weights. The code was checked against the ONNX files the site serves: from the same noise the pen points agree to 0.0003 of the line height and the same attempt is chosen.
163
 
164
+ | On the site | Here | What it does |
165
+ |---|---|---|
166
+ | `den.onnx`, `refpts.onnx` | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model, with attention to the reference points |
167
+ | `planner.onnx` | `ckpt_dl/planner7/best.pt` | pen points per character |
168
+ | `semb.onnx` | `ckpt_semb3_all/best.pt` | style vector of the reference line |
169
+ | `rec.onnx` | `ckpt_ctc4/ctc_fp16.pt` | recogniser: how legible an attempt is |
170
+ | `scls.onnx` | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer: how close an attempt is to the reference |
171
+ | `fix.onnx`, `tenc.onnx`, `tstep.onnx` | `trace/weights/` | ink restorer and tracer, for photos |
172
 
173
+ Its text model is the text model of the pages model trained for 60 000 more steps, with traced lines of 190 HWR200 writers added to the data and attention to the reference points added to the network.
174
 
175
+ ### Bilingual model
176
 
177
+ **Licence.** It was trained on [BRUSH](https://dsd.cs.brown.edu/) (Brown University), whose terms allow non-commercial research use only, and its weights inherit that: `ckpt_dl/u4s2/`, `ckpt_semb4/` and `ckpt_ctc6/` are for non-commercial research only, on top of CC BY-NC 4.0.
178
 
179
+ Made from the site model in three steps:
 
 
 
180
 
181
+ 1. **Text model:** 60 000 more steps on the site model's data, with underlines removed from the tablet lines (they made the first word of a line come out underlined), plus English pen lines of 158 BRUSH writers. 12 more BRUSH writers were kept out of all training to measure on.
182
+ 2. **Style encoder:** trained further to tell BRUSH writers apart, while staying close to its old vectors on Russian lines.
183
+ 3. **Text model:** 20 000 more steps with the new style encoder.
184
 
185
+ Its English recogniser is the site's recogniser trained further on the same mix.
186
 
187
+ ## Weights
188
 
189
+ | Model | Path | What | Size |
190
+ |---|---|---|---|
191
+ | all | `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
192
+ | pages, site | `ckpt_semb3_all/best.pt` | style encoder | 7 MB |
193
+ | pages | `ckpt_dl/b4styc/fp16/step003000_fp16.pt` | text model | 45 MB |
194
+ | pages | `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
195
+ | pages | `ckpt_dl/glyph_cls2.pt` | glyph classifier | 1 MB |
196
+ | site | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model | 51 MB |
197
+ | site, bilingual | `ckpt_ctc4/ctc_fp16.pt` | recogniser (Russian lines) | 20 MB |
198
+ | site, bilingual | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer | 4 MB |
199
+ | bilingual | `ckpt_dl/u4s2/fp16/step020000_fp16.pt` | text model; non-commercial research only | 51 MB |
200
+ | bilingual | `ckpt_semb4/best.pt` | style encoder; non-commercial research only | 7 MB |
201
+ | bilingual | `ckpt_ctc6/ctc_fp16.pt` | recogniser (English lines); non-commercial research only | 20 MB |
202
+ | photos | `trace/weights/` | tracer and ink restorer | 62 MB |
203
+
204
+ The folder names are those of the training runs. `download_weights.py` fetches each group, see [Quick start](#quick-start).
205
+
206
+ ## Code
207
 
208
+ | Path | What |
209
+ |---|---|
210
+ | `write_pages.py` | text file -> pages: wrapping, page layout, the order of the steps |
211
+ | `write_line.py` | lines of text in the hand of a reference line, the way the site does it |
212
+ | `make_ref.py` | your strokes -> a reference line |
213
+ | `scrawler/text_model.py` | text model of the pages model and the duration planner |
214
+ | `scrawler/site_model.py` | text model of the site and bilingual models, recognisers, writer classifier, the recipe |
215
+ | `scrawler/style_encoder.py` | one line -> style vector |
216
+ | `scrawler/glyph_model.py` | glyph model and the classifier that picks variants |
217
+ | `scrawler/formula.py` | formula layout, pen-line marks, inserting formulas into text |
218
+ | `scrawler/render.py`, `render_pages.py` | strokes -> PDF, SVG, PNG |
219
+ | `trace/` | the tracer, one file |
220
+ | `tablet/` | a line out of a note app: readers of GoodNotes, Notewise, Huawei Notes, Kilonotes and Samsung Notes files |
221
+ | `tools/mathjax/layout.mjs` | LaTeX -> glyph boxes |
222
+ | `data/glyph_stroke_counts.json` | per glyph class, how many real glyphs had 1, 2, ... strokes; counts only |
223
 
224
+ ## Evaluation
225
 
226
+ Four measures are used below. They are on different scales, so numbers of different measures cannot be compared.
 
 
 
 
227
 
228
+ - **Judge:** an LLM judge (Gemini 4 Argon) lists errors in each line; "clean" means no error of severity 2 or higher.
229
+ - **Legible:** the recogniser's character error rate is at most 0.05 (`ckpt_ctc4` for Russian, `ckpt_ctc6` for English).
230
+ - **Russian classifier:** cosine between a generated line and the writer's real line, by the writer classifier `ckpt_stylecls` (trained on Russian tablet writers).
231
+ - **Bilingual classifier:** the same kind of classifier trained on both languages (not released). "Own writer found" means the generated line is closer to its own reference than to the other 19.
232
 
233
+ ### Pages model
234
 
235
+ 240 lines of real notes from writers the generator never saw, each written from one reference line by the same writer, scored by the judge. Pairs are re-sampled for the 95% intervals.
236
 
237
  | Run | Clean, % [95%] | Prose | Mixed | Formula |
238
  |---|---|---|---|---|
239
  | Real handwriting of the same lines | 45.0 [38.8, 51.2] | 47.0 | 35.7 | 51.4 |
240
  | Text model alone | 47.1 [40.8, 53.3] | 63.0 | 41.4 | 30.0 |
241
+ | **Pages model: text model + formula module** | **61.3 [55.0, 67.5]** | 63.0 | 54.3 | 65.7 |
242
  | Same, with a library of real glyphs (not released) | 65.4 [59.2, 71.2] | 62.0 | 58.6 | 77.1 |
243
 
244
+ - The judge rewards legibility. Scoring above real notes means the output is tidier than hurried human writing, not that it looks more human.
 
 
245
  - Prose is the same set of generated lines in every run but the first, so 62-63 shows judge noise of a few points.
246
  - The private library variant scores 4.2 points higher, 95% interval [-9.6, +1.2] for the difference. That is within the noise, but a small loss from removing it is possible.
247
 
248
+ Style, by the style encoder `ckpt_semb3_all` (a third scale, used only here): cosine to the writer's real line 0.59-0.68 for generated lines, against 0.55-0.70 between real lines of the same writers, on 3 unseen writers.
249
+
250
+ ### Site model against the pages model
251
+
252
+ Traced lines of 9 writers the models never saw. Russian classifier: 0.71 to the writer's own line for the site model, 0.47 for the text model of the pages model; 0.46 to other writers for both. Judge on the 240 lines above: 47.1% clean, the same as "Text model alone". By eye the gain in likeness is modest.
253
+
254
+ ### Bilingual model against the site model
255
+
256
+ 20 hands per language, one reference line each, best of four attempts. English: 12 BRUSH writers the models never saw and 8 the bilingual model saw. Measured on a GPU; the same seed gives other lines on a Mac.
257
+
258
+ | | English: own writer found (bilingual classifier) | English: cosine own minus others (bilingual classifier) | English: legible | Russian: cosine to own line (Russian classifier) | Russian: legible | Russian: stray stroke on the first word |
259
+ |---|---|---|---|---|---|---|
260
+ | Site model | 5% | 0.06 | 95% | 0.64 | 95% | 50% |
261
+ | **Bilingual model** | **65%** | **0.29** | **95%** | **0.67** | **90%** | **5%** |
262
+ | Another real line of the same writer | 85% | 0.49 | | | | |
263
+
264
+ The English hands are much closer to their writers than with the site model, but still well short of real lines. Without the best of four, English lines often lose spaces between words: 82% legible over all attempts.
265
 
266
  ## Known weaknesses
267
 
268
+ - All text models sometimes swap, drop or add a letter. There is no spell check.
269
+ - The site model often adds a stray stroke to the first word of a line, an underline or a strike-through: in the chosen attempt for 10 of 20 test hands. The bilingual model does it for 1 of 20.
270
+ - Pages: a one-letter formula inside a sentence can collide with the neighbouring word.
271
+ - Pages: about 2% of glyphs are the best of the variants the classifier rejected, and may be misshapen.
272
+ - Pages: rare symbols without their own class are drawn as a similar one (for example `\simeq` as `\approx`).
273
+ - Pages: a heading of one short word may come out smaller than the other lines.
274
+ - Pages: a line of several formulas with almost no plain text between them is written by the text model alone instead of the formula module (`fallback_text_model` in `report.json`).
275
 
276
  ## Training data
277
 
278
+ - **Pages model:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
279
+ - **Site model:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
280
+ - **Bilingual model:** the same, plus English pen lines of 158 writers of [BRUSH](https://dsd.cs.brown.edu/) (non-commercial research use only). Its style encoder and English recogniser were trained further on the same lines.
281
+ - **Style encoder:** the same kind of tablet data, plus traced lines of HWR200.
282
  - **Recogniser:** the tablet lines above, the same lines rendered to images and traced back, and traced lines of HWR200.
283
  - **Writer classifier:** tablet lines of 15 writers: 12 of the 13 above and three more.
 
284
  - **Tracer:** a mix of tablet notes and open sets, some with non-commercial or ShareAlike terms. See [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md), including the open question about ShareAlike.
285
 
286
  ## Responsible use
 
289
 
290
  ## License
291
 
292
+ Code: Apache-2.0 ([LICENSE](LICENSE)). Weights: CC BY-NC 4.0 ([LICENSE-WEIGHTS](LICENSE-WEIGHTS)). The bilingual model is for non-commercial research only (BRUSH). The tracer weights carry further conditions from their training data, see [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md). Third-party components: [NOTICE](NOTICE).
download_weights.py CHANGED
@@ -3,7 +3,7 @@
3
  python download_weights.py # the models that write pages, about 80 MB
4
  python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
5
  python download_weights.py --site # also the models of synthink.dev for write_line.py, about 75 MB more
6
- python download_weights.py --bilingual # also the English and Russian model for write_line.py --model bilingual, about 150 MB more
7
  """
8
  import argparse
9
  import os
 
3
  python download_weights.py # the models that write pages, about 80 MB
4
  python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
5
  python download_weights.py --site # also the models of synthink.dev for write_line.py, about 75 MB more
6
+ python download_weights.py --bilingual # also the English and Russian model for write_line.py --model bilingual, about 100 MB more
7
  """
8
  import argparse
9
  import os
write_line.py CHANGED
@@ -12,7 +12,9 @@ In the folder `out` (it must not exist yet):
12
  - strokes.json: every attempt as strokes, with its character error rate and its closeness to the reference
13
 
14
  --model bilingual writes English as well as Russian (python download_weights.py --bilingual); its weights are for
15
- non-commercial research only, see README. The recipe is in scrawler/site_model.py. The style comes from --ref (an .npy from make_ref.py). --ref-text is what is written
 
 
16
  in that line; it sets how wide the writer writes. Without it the recogniser reads the reference, which is right in most
17
  letters but not all, so give the text when you know it.
18
  """
 
12
  - strokes.json: every attempt as strokes, with its character error rate and its closeness to the reference
13
 
14
  --model bilingual writes English as well as Russian (python download_weights.py --bilingual); its weights are for
15
+ non-commercial research only, see README. The recipe is in scrawler/site_model.py.
16
+
17
+ The style comes from --ref (an .npy from make_ref.py). --ref-text is what is written
18
  in that line; it sets how wide the writer writes. Without it the recogniser reads the reference, which is right in most
19
  letters but not all, so give the text when you know it.
20
  """