levossadtchi commited on
Commit
f1fa3dd
·
verified ·
1 Parent(s): e09074a

The whole site recipe: write_line.py, recogniser and writer classifier; a line from a tablet (tablet/)

Browse files
NOTICE CHANGED
@@ -7,3 +7,5 @@ The style encoder was also trained on traced HWR200 lines.
7
  The tracer (trace/) was trained on data credited in trace/TRAINING_DATA.md, including BRUSH, MathWriting, Korean Handwriting Strokes, OUHD-L, Make Me a Hanzi and Google Fonts.
8
  examples/from_scan.png shows one line of HWR200.
9
  trace/example_open_lines.png includes a render of Korean Handwriting Strokes by Jonathan Baudanza (hangul.ink), handwriting by Jo YW, CC BY-SA 4.0, and so is under CC BY-SA 4.0; it also shows OUHD-L (CC BY 4.0) and Digital Peter (MIT).
 
 
 
7
  The tracer (trace/) was trained on data credited in trace/TRAINING_DATA.md, including BRUSH, MathWriting, Korean Handwriting Strokes, OUHD-L, Make Me a Hanzi and Google Fonts.
8
  examples/from_scan.png shows one line of HWR200.
9
  trace/example_open_lines.png includes a render of Korean Handwriting Strokes by Jonathan Baudanza (hangul.ink), handwriting by Jo YW, CC BY-SA 4.0, and so is under CC BY-SA 4.0; it also shows OUHD-L (CC BY 4.0) and Digital Peter (MIT).
10
+ The recogniser (ckpt_ctc4) was also trained on traced HWR200 lines.
11
+ tablet/samsung/sdocx2ink.mjs uses the npm package @twango/sdocx (GPL-3.0, https://github.com/twangodev/sdocx). npm installs it separately; it is not distributed here.
README.md CHANGED
@@ -73,15 +73,18 @@ For a good result:
73
 
74
  The tracer weights have their own licence caveat, see [its training data](trace/TRAINING_DATA.md).
75
 
76
- ### From tablet strokes
77
 
78
- If you have the pen strokes already, skip the tracer. Export them as a JSON list of strokes, each a list of `[x, y]` points with y pointing down, in writing order.
79
 
80
  ```bash
 
81
  python make_ref.py my_line.json my_line.npy
82
  python write_pages.py notes.txt out --ref my_line.npy
83
  ```
84
 
 
 
85
  The default style is one traced line of the open HWR200 dataset.
86
 
87
  ## Input format
@@ -109,13 +112,16 @@ The style vector comes from a line-level writer encoder. Fraction bars, radicals
109
  | Path | What |
110
  |---|---|
111
  | `write_pages.py` | text file -> pages: wrapping, page layout, the order of the steps |
 
112
  | `scrawler/text_model.py` | text model and duration planner |
 
113
  | `scrawler/style_encoder.py` | one line -> style vector |
114
  | `scrawler/glyph_model.py` | glyph model and the classifier that picks variants |
115
  | `scrawler/formula.py` | formula layout, pen-line marks, inserting formulas into text |
116
  | `scrawler/render.py`, `render_pages.py` | strokes -> PDF, SVG, PNG |
117
  | `make_ref.py` | your strokes -> a style line |
118
  | `trace/` | the tracer, one file |
 
119
  | `tools/mathjax/layout.mjs` | LaTeX -> glyph boxes |
120
  | `data/glyph_stroke_counts.json` | per glyph class, how many real glyphs had 1, 2, ... strokes; counts only |
121
 
@@ -124,7 +130,9 @@ The style vector comes from a line-level writer encoder. Fraction bars, radicals
124
  | Path | What | Size |
125
  |---|---|---|
126
  | `ckpt_dl/b4styc/fp16/step003000_fp16.pt` | text model | 45 MB |
127
- | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev), see below; not used by `write_pages.py` | 51 MB |
 
 
128
  | `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
129
  | `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
130
  | `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
@@ -133,13 +141,44 @@ The style vector comes from a line-level writer encoder. Fraction bars, radicals
133
 
134
  ## The model on the site
135
 
136
- [synthink.dev](https://synthink.dev) writes a line in your handwriting right in the browser: [try it there](https://synthink.dev/generate/). The text model that runs on the site is `ckpt_dl/u3ref/fp16/step060000_fp16.pt` in this repository (a PyTorch checkpoint; the site runs the same weights converted to ONNX).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137
 
138
- It is the text model of this release trained for 60 000 more steps, with two additions: traced lines of 190 HWR200 writers in the training data, and attention from every generated point to the pen points of the reference line. The model reads the reference line itself, not only its style vector.
139
 
140
  Measured on traced lines of 9 writers the model never saw: the cosine of a writer classifier between a generated line and the writer's own line is 0.71, against 0.47 for the release text model; to other writers it is 0.46 for both. Legibility did not change: 47.1% clean lines on the set of the table below, the same as "Text model alone". By eye the gain in likeness is modest.
141
 
142
- `write_pages.py` does not use this checkpoint: `scrawler/text_model.py` has no reference attention yet, and the pages shown here are written by the text model of the first row. On the site every line is sampled four times and the most legible one is kept.
143
 
144
  ## Evaluation
145
 
@@ -163,6 +202,7 @@ Style similarity by the writer encoder (cosine to the writer's real line, one re
163
  ## Known weaknesses
164
 
165
  - The text model sometimes swaps, drops or adds a letter. There is no spell check.
 
166
  - A one-letter formula inside a sentence can collide with the neighbouring word.
167
  - About 2% of glyphs are the best of the variants the classifier rejected, and may be misshapen.
168
  - Rare symbols without their own class are drawn as a similar one (for example `\simeq` as `\approx`).
@@ -173,6 +213,8 @@ Style similarity by the writer encoder (cosine to the writer's real line, one re
173
 
174
  - **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
175
  - **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
 
 
176
  - **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
177
  - **Tracer:** a mix of tablet notes and open sets, some with non-commercial or ShareAlike terms. See [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md), including the open question about ShareAlike.
178
 
 
73
 
74
  The tracer weights have their own licence caveat, see [its training data](trace/TRAINING_DATA.md).
75
 
76
+ ### From a tablet
77
 
78
+ If the line was written in a note app, its pen strokes are in the app's file and no tracer is needed. `tablet/app_line.py` reads GoodNotes, Notewise, Huawei Notes, Kilonotes and Samsung Notes. The recipe for each app, and for the apps that are not read, is in [tablet/README.md](tablet/README.md).
79
 
80
  ```bash
81
+ python tablet/app_line.py notes.goodnotes my_line.json --page 3 --line 2 --svg check.svg
82
  python make_ref.py my_line.json my_line.npy
83
  python write_pages.py notes.txt out --ref my_line.npy
84
  ```
85
 
86
+ If you have the strokes from somewhere else, write them as a JSON list of strokes, each a list of `[x, y]` points with y pointing down, in writing order: that is what `make_ref.py` takes.
87
+
88
  The default style is one traced line of the open HWR200 dataset.
89
 
90
  ## Input format
 
112
  | Path | What |
113
  |---|---|
114
  | `write_pages.py` | text file -> pages: wrapping, page layout, the order of the steps |
115
+ | `write_line.py` | one line of text in the hand of a reference line, the way the site does it |
116
  | `scrawler/text_model.py` | text model and duration planner |
117
+ | `scrawler/site_model.py` | the site's text model, recogniser, writer classifier and the choice of the best attempt |
118
  | `scrawler/style_encoder.py` | one line -> style vector |
119
  | `scrawler/glyph_model.py` | glyph model and the classifier that picks variants |
120
  | `scrawler/formula.py` | formula layout, pen-line marks, inserting formulas into text |
121
  | `scrawler/render.py`, `render_pages.py` | strokes -> PDF, SVG, PNG |
122
  | `make_ref.py` | your strokes -> a style line |
123
  | `trace/` | the tracer, one file |
124
+ | `tablet/` | a line out of a note app: readers of GoodNotes, Notewise, Huawei Notes, Kilonotes and Samsung Notes files |
125
  | `tools/mathjax/layout.mjs` | LaTeX -> glyph boxes |
126
  | `data/glyph_stroke_counts.json` | per glyph class, how many real glyphs had 1, 2, ... strokes; counts only |
127
 
 
130
  | Path | What | Size |
131
  |---|---|---|
132
  | `ckpt_dl/b4styc/fp16/step003000_fp16.pt` | text model | 45 MB |
133
+ | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev), used by `write_line.py`, see below | 51 MB |
134
+ | `ckpt_ctc4/ctc_fp16.pt` | recogniser, for `write_line.py` | 20 MB |
135
+ | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer, for `write_line.py` | 4 MB |
136
  | `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
137
  | `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
138
  | `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
 
141
 
142
  ## The model on the site
143
 
144
+ [synthink.dev](https://synthink.dev) writes a line in your handwriting right in the browser: [try it there](https://synthink.dev/generate/). Every model the site runs is in this release, and `write_line.py` does the same on your machine: one line of your handwriting from a photo or a tablet goes in, a new line in that hand comes out.
145
+
146
+ ```bash
147
+ python download_weights.py --site --tracer
148
+ pip install -r trace/requirements.txt
149
+ python trace/scrawler_trace.py my_line.png --out my_line.json --svg my_line.svg # photo of one line -> pen strokes
150
+ python make_ref.py my_line.json my_line.npy # pen strokes -> reference line
151
+ python write_line.py "Мороз и солнце; день чудесный!" out_line --ref my_line.npy --ref-text "what your line says"
152
+ ```
153
+
154
+ For a line from a tablet, the tracer step is replaced by `tablet/app_line.py`, see [tablet/README.md](tablet/README.md).
155
+
156
+ `out_line/` gets `line.svg`, `line.png` and `line.pdf` with the chosen attempt, `attempts.png` with all four, and `strokes.json`. The first argument can also be a `.txt` file with one line of text per row. Formulas are not written here. A line takes about 9 seconds on an Apple M5 Max (MPS).
157
+
158
+ | On the site | Here | What it does |
159
+ |---|---|---|
160
+ | `den.onnx`, `refpts.onnx` | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model that also attends to the pen points of the reference line |
161
+ | `planner.onnx` | `ckpt_dl/planner7/best.pt` | pen points per character |
162
+ | `semb.onnx` | `ckpt_semb3_all/best.pt` | style vector of the reference line |
163
+ | `rec.onnx` | `ckpt_ctc4/ctc_fp16.pt` | recogniser: how legible an attempt is |
164
+ | `scls.onnx` | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer: how close an attempt is to the reference |
165
+ | `fix.onnx`, `tenc.onnx`, `tstep.onnx` | `trace/weights/` | ink restorer and tracer, for the photo route |
166
+
167
+ The weights are in this repository; the site runs the same weights converted to ONNX.
168
+
169
+ The recipe, in `scrawler/site_model.py`:
170
+
171
+ 1. The planner gives the number of pen points for every character. These are multiplied by the writer's spread: the length of the reference line in points over the length the planner gives for the text of that line, clipped to [0.6, 1.4]. This is why `--ref-text` is asked for. Without it the recogniser reads the reference, which gets most letters right but not all.
172
+ 2. The text model writes the line four times, with guidance 1.5 and 20 steps.
173
+ 3. The recogniser keeps the attempts whose character error rate is within 0.02 of the lowest, and of those the writer classifier picks the one closest to the reference.
174
+
175
+ The code was checked against the ONNX files the site serves: from the same noise the pen points agree to 0.0003 of the line height and the same attempt is chosen.
176
 
177
+ The text model is the text model of this release trained for 60 000 more steps, with two additions: traced lines of 190 HWR200 writers in the training data, and attention from every generated point to the pen points of the reference line. The model reads the reference line itself, not only its style vector.
178
 
179
  Measured on traced lines of 9 writers the model never saw: the cosine of a writer classifier between a generated line and the writer's own line is 0.71, against 0.47 for the release text model; to other writers it is 0.46 for both. Legibility did not change: 47.1% clean lines on the set of the table below, the same as "Text model alone". By eye the gain in likeness is modest.
180
 
181
+ `write_pages.py` does not use the site model: pages and formulas are written by the text model of the first row of the weights table.
182
 
183
  ## Evaluation
184
 
 
202
  ## Known weaknesses
203
 
204
  - The text model sometimes swaps, drops or adds a letter. There is no spell check.
205
+ - The site model (`write_line.py`) often adds a stray stroke to the first word of a line, an underline or a strike-through: a stroke detector found one in the chosen attempt for 10 of 20 test hands.
206
  - A one-letter formula inside a sentence can collide with the neighbouring word.
207
  - About 2% of glyphs are the best of the variants the classifier rejected, and may be misshapen.
208
  - Rare symbols without their own class are drawn as a similar one (for example `\simeq` as `\approx`).
 
213
 
214
  - **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
215
  - **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
216
+ - **Recogniser:** the tablet lines above, the same lines rendered to images and traced back, and traced lines of HWR200.
217
+ - **Writer classifier:** tablet lines of 15 writers: 12 of the 13 above and three more.
218
  - **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
219
  - **Tracer:** a mix of tablet notes and open sets, some with non-commercial or ShareAlike terms. See [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md), including the open question about ShareAlike.
220
 
ckpt_ctc4/ctc_fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:95208859abf7e3890d9dfb2f5a4b7473d74f6038107031660e035e11e9cec3b3
3
+ size 19781805
ckpt_stylecls/embed_fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d601fea48bbbe724477910c368b4f4fd79ef75ea4e3708d4076683ede43c0a38
3
+ size 3513653
download_weights.py CHANGED
@@ -2,6 +2,7 @@
2
 
3
  python download_weights.py # the models that write pages, about 80 MB
4
  python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
 
5
  """
6
  import argparse
7
  import os
@@ -17,14 +18,20 @@ WRITER = [
17
  "ckpt_dl/glyph_cls2.pt",
18
  ]
19
  TRACER = ["trace/weights/*"]
 
 
 
 
 
20
 
21
 
22
  def main():
23
  ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
24
  ap.add_argument("--tracer", action="store_true")
 
25
  args = ap.parse_args()
26
  root = os.path.dirname(os.path.abspath(__file__))
27
- snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []), local_dir=root)
28
  print("weights are in", root)
29
 
30
 
 
2
 
3
  python download_weights.py # the models that write pages, about 80 MB
4
  python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
5
+ python download_weights.py --site # also the models of synthink.dev for write_line.py, about 75 MB more
6
  """
7
  import argparse
8
  import os
 
18
  "ckpt_dl/glyph_cls2.pt",
19
  ]
20
  TRACER = ["trace/weights/*"]
21
+ SITE = [
22
+ "ckpt_dl/u3ref/fp16/step060000_fp16.pt",
23
+ "ckpt_ctc4/ctc_fp16.pt",
24
+ "ckpt_stylecls/embed_fp16.pt",
25
+ ]
26
 
27
 
28
  def main():
29
  ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
30
  ap.add_argument("--tracer", action="store_true")
31
+ ap.add_argument("--site", action="store_true")
32
  args = ap.parse_args()
33
  root = os.path.dirname(os.path.abspath(__file__))
34
+ snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []) + (SITE if args.site else []), local_dir=root)
35
  print("weights are in", root)
36
 
37
 
scrawler/common.py CHANGED
@@ -25,6 +25,9 @@ if DEVICE == "cuda":
25
 
26
  CKPT_TEXT = os.path.join(ROOT, "ckpt_dl", "b4styc", "fp16", "step003000_fp16.pt")
27
  CKPT_PLANNER = os.path.join(ROOT, "ckpt_dl", "planner7", "best.pt")
 
 
 
28
  CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
29
  CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
30
  CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
@@ -35,5 +38,6 @@ MATHJAX_DIR = os.path.join(ROOT, "tools", "mathjax")
35
 
36
  def load_checkpoint(path):
37
  if not os.path.exists(path):
38
- raise SystemExit(f"{path} not found. Run: python download_weights.py")
 
39
  return torch.load(path, map_location=DEVICE, weights_only=True)
 
25
 
26
  CKPT_TEXT = os.path.join(ROOT, "ckpt_dl", "b4styc", "fp16", "step003000_fp16.pt")
27
  CKPT_PLANNER = os.path.join(ROOT, "ckpt_dl", "planner7", "best.pt")
28
+ CKPT_SITE_TEXT = os.path.join(ROOT, "ckpt_dl", "u3ref", "fp16", "step060000_fp16.pt")
29
+ CKPT_RECOGNISER = os.path.join(ROOT, "ckpt_ctc4", "ctc_fp16.pt")
30
+ CKPT_CLASSIFIER = os.path.join(ROOT, "ckpt_stylecls", "embed_fp16.pt")
31
  CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
32
  CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
33
  CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
 
38
 
39
  def load_checkpoint(path):
40
  if not os.path.exists(path):
41
+ hint = " --site" if path in (CKPT_SITE_TEXT, CKPT_RECOGNISER, CKPT_CLASSIFIER) else ""
42
+ raise SystemExit(f"{path} not found. Run: python download_weights.py{hint}")
43
  return torch.load(path, map_location=DEVICE, weights_only=True)
scrawler/site_model.py ADDED
@@ -0,0 +1,311 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The text model of synthink.dev and its recipe: one reference line and a text -> the text as a line in that hand.
2
+
3
+ Three things differ from scrawler/text_model.py:
4
+
5
+ - The denoiser also attends to the pen points of the reference line (two cross-attention layers), so it reads the line itself
6
+ and not only its style vector.
7
+ - The planner's point counts are multiplied by the writer's spread: the length of the reference in points over the length the
8
+ planner gives for the reference text, clipped to [0.6, 1.4]. This needs the text of the reference line.
9
+ - Every line is sampled four times. A recogniser keeps the attempts that read best (character error rate within 0.02 of the
10
+ lowest), and of those a writer classifier picks the one closest to the reference.
11
+
12
+ Lines are arrays [n, 3] of (dx, dy, pen_up) with line height 1, as everywhere in this package.
13
+ """
14
+ import numpy as np
15
+ import torch
16
+ import torch.nn as nn
17
+ import torch.nn.functional as F
18
+
19
+ from .common import AMP, CKPT_CLASSIFIER, CKPT_PLANNER, CKPT_RECOGNISER, CKPT_SITE_TEXT, DEVICE, load_checkpoint
20
+ from .style_encoder import SCALE, features, load_style_net, resample, style_vector
21
+ from .text_model import ADVANCE, ALPHA_BAR, GUIDANCE, MAX_POINTS, NOISE_STEPS, SAMPLE_STEPS, SCALE_X, SCALE_Y
22
+ from .text_model import Denoiser, Planner, _plan, time_embedding
23
+
24
+ ATTEMPTS = 4
25
+ SPREAD_MIN, SPREAD_MAX = 0.6, 1.4
26
+ CER_SLACK = 0.02
27
+ REF_STEP = 0.05
28
+
29
+
30
+ def to_strokes(line):
31
+ """A line [n, 3] -> a list of strokes, each an array [k, 2] of absolute points."""
32
+ line = np.asarray(line, np.float32)
33
+ points = np.cumsum(line[:, :2], 0)
34
+ strokes, start = [], 0
35
+ for i in np.flatnonzero(line[:, 2] > 0.5):
36
+ strokes.append(points[start : i + 1])
37
+ start = i + 1
38
+ if start < len(points):
39
+ strokes.append(points[start:])
40
+ return strokes
41
+
42
+
43
+ def to_reference(strokes):
44
+ """Strokes in any units -> a reference line: height 1, a point every 0.05 along the pen path. The same as make_ref.py."""
45
+ strokes = [np.asarray(s, np.float64)[:, :2] for s in strokes if len(s)]
46
+ everything = np.concatenate(strokes)
47
+ origin = everything.min(0)
48
+ height = float(everything[:, 1].max() - origin[1])
49
+ out = []
50
+ for p in strokes:
51
+ p = (p - origin) / height
52
+ if len(p) == 1:
53
+ out.append(np.array([[p[0, 0], p[0, 1], 1.0]]))
54
+ continue
55
+ cum = np.r_[0, np.cumsum(np.hypot(*np.diff(p, axis=0).T))]
56
+ u = np.r_[np.arange(0, cum[-1], REF_STEP), cum[-1]] if cum[-1] > 1e-6 else np.array([0.0, cum[-1]])
57
+ r = np.stack([np.interp(u, cum, p[:, 0]), np.interp(u, cum, p[:, 1]), np.zeros(len(u))], 1)
58
+ r[-1, 2] = 1
59
+ out.append(r)
60
+ absolute = np.concatenate(out)
61
+ line = absolute.copy()
62
+ line[1:, :2] = absolute[1:, :2] - absolute[:-1, :2]
63
+ line[0, :2] = 0
64
+ return line.astype(np.float32)
65
+
66
+
67
+ class ReferenceEncoder(nn.Module):
68
+ """A reference line -> the keys the denoiser attends to: one learned "sink" key and one key for every 4 pen points."""
69
+
70
+ def __init__(self, d=192):
71
+ super().__init__()
72
+ self.stem = nn.Sequential(
73
+ nn.Conv1d(4, 128, 5, padding=2),
74
+ nn.GELU(),
75
+ nn.Conv1d(128, 192, 5, stride=2, padding=2),
76
+ nn.GELU(),
77
+ nn.Conv1d(192, 256, 5, stride=2, padding=2),
78
+ nn.GELU(),
79
+ )
80
+ self.gru = nn.GRU(256, d, num_layers=2, batch_first=True, bidirectional=True, dropout=0.1)
81
+ self.rkv = nn.Sequential(nn.Linear(2 * d, 256), nn.LayerNorm(256))
82
+ self.sink = nn.Parameter(torch.zeros(1, 256))
83
+
84
+ def forward(self, line):
85
+ """line [n, 3] -> keys [1, 1 + ceil(n / 4), 256]."""
86
+ y = torch.cumsum(line[:, 1], 0)
87
+ middle = torch.sort(y).values[(len(y) - 1) // 2] # the lower median
88
+ x = torch.cat([line[:, :2] * SCALE, line[:, 2:3], (y - middle)[:, None]], -1)[None]
89
+ h = self.stem(x.transpose(1, 2)).transpose(1, 2)
90
+ out, _ = self.gru(h)
91
+ return torch.cat([self.sink[None], self.rkv(out)], 1)
92
+
93
+
94
+ class SiteDenoiser(Denoiser):
95
+ """The denoiser of text_model.py with cross-attention to the reference keys after the GRU and before the head."""
96
+
97
+ def __init__(self, vocab_size):
98
+ super().__init__(vocab_size)
99
+ D = self.head.in_features
100
+ self.enc = ReferenceEncoder()
101
+ self.src_emb = nn.Parameter(torch.zeros(3, 256)) # where the ink came from: 0 tablet, 1 traced scan, 2 unknown
102
+ self.xln = nn.ModuleList([nn.LayerNorm(D) for _ in range(2)])
103
+ self.xatt = nn.ModuleList([nn.MultiheadAttention(D, 4, kdim=256, vdim=256, batch_first=True) for _ in range(2)])
104
+
105
+ def forward(self, z, t, cond, style, self_cond, keys):
106
+ mask_ = cond["mask"]
107
+ mask = mask_[..., None]
108
+ length = z.size(1)
109
+ pos = torch.arange(length, device=z.device, dtype=torch.float32)[None, :, None].expand(z.size(0), -1, -1) / 600.0
110
+ phase = cond["progress"][..., None]
111
+ feats = torch.cat(
112
+ [z, self_cond, phase, torch.sin(2 * np.pi * phase), torch.cos(2 * np.pi * phase), torch.log1p(cond["duration"])[..., None] / 3.0, pos],
113
+ -1,
114
+ )
115
+ x = (self.inp(feats) + self.ec(cond["char"]) + self.ep(cond["prev"]) + self.en(cond["next"]) + self.eph(cond["phase"]) + self.sin(style)[:, None]) * mask
116
+ g = F.silu(self.tm(time_embedding(t)) + self.sp(style))
117
+ for block in self.pre:
118
+ x = block(x, mask, g)
119
+ h = self.gfilm(self.gln(x), g)
120
+ lengths = mask_.sum(1).long().clamp(min=1)
121
+ with torch.autocast("cuda", enabled=False):
122
+ packed = nn.utils.rnn.pack_padded_sequence(h.float(), lengths.cpu(), batch_first=True, enforce_sorted=False)
123
+ out, _ = self.gru(packed)
124
+ out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True, total_length=length)
125
+ x = (x + self.gout(out.to(x.dtype))) * mask
126
+ keys = keys.to(x.dtype)
127
+ x = (x + self.xatt[0](self.xln[0](x), keys, keys, need_weights=False)[0]) * mask
128
+ for block in self.post:
129
+ x = block(x, mask, g)
130
+ x = (x + self.xatt[1](self.xln[1](x), keys, keys, need_weights=False)[0]) * mask
131
+ out = self.head(self.oln(x)).float()
132
+ return out[..., :2] * mask, out[..., 2]
133
+
134
+
135
+ class Recogniser(nn.Module):
136
+ """Pen points -> per-point character scores (CTC). Reads generated lines to tell how legible they are."""
137
+
138
+ def __init__(self, vocab_size, hidden=384):
139
+ super().__init__()
140
+ self.inp = nn.Sequential(nn.Linear(4, hidden), nn.ReLU(), nn.Linear(hidden, hidden))
141
+ self.rnn = nn.LSTM(hidden, hidden, num_layers=3, bidirectional=True, batch_first=True)
142
+ self.out = nn.Linear(2 * hidden, vocab_size)
143
+
144
+ def forward(self, x):
145
+ out, _ = self.rnn(self.inp(x))
146
+ return self.out(out)
147
+
148
+
149
+ class WriterEmbedder(nn.Module):
150
+ """A writer classifier without its class layer: a line -> a vector that is close for lines of the same hand."""
151
+
152
+ def __init__(self, d=192):
153
+ super().__init__()
154
+ self.stem = ReferenceEncoder(d).stem
155
+ self.gru = nn.GRU(256, d, num_layers=2, batch_first=True, bidirectional=True, dropout=0.1)
156
+ self.head = nn.Sequential(nn.Linear(4 * d, 256), nn.GELU())
157
+
158
+ def forward(self, x):
159
+ out, _ = self.gru(self.stem(x.transpose(1, 2)).transpose(1, 2))
160
+ return self.head(torch.cat([out.mean(1), out.max(1).values], -1))
161
+
162
+
163
+ def _edit_rate(target, hypothesis):
164
+ """Edit distance between two id lists over the length of the target."""
165
+ if not target and not hypothesis:
166
+ return 0.0
167
+ row = list(range(len(hypothesis) + 1))
168
+ for i, a in enumerate(target, 1):
169
+ prev, row[0] = row[0], i
170
+ for j, b in enumerate(hypothesis, 1):
171
+ prev, row[j] = row[j], min(row[j] + 1, row[j - 1] + 1, prev + (a != b))
172
+ return row[len(hypothesis)] / max(1, len(target))
173
+
174
+
175
+ class SiteModel:
176
+ def __init__(self):
177
+ state = load_checkpoint(CKPT_SITE_TEXT)
178
+ self.itos = state["itos"]
179
+ self.stoi = {s: i for i, s in enumerate(self.itos)}
180
+ self.null_token = len(self.itos)
181
+ self.net = SiteDenoiser(len(self.itos)).to(DEVICE)
182
+ unused = ("enc.proj.", "enc.cls.") # the classifier head the reference encoder was started from
183
+ self.net.load_state_dict({k: v.float() for k, v in state["e"].items() if not k.startswith(unused)})
184
+ self.net.eval()
185
+ state = load_checkpoint(CKPT_PLANNER)
186
+ self.planner = Planner(len(state["itos"])).to(DEVICE)
187
+ self.planner.load_state_dict(state["m"])
188
+ self.planner.eval()
189
+ self.style_net = load_style_net()
190
+ state = load_checkpoint(CKPT_RECOGNISER)
191
+ self.recogniser = Recogniser(len(state["itos"])).to(DEVICE)
192
+ self.recogniser.load_state_dict({k: v.float() for k, v in state["m"].items()})
193
+ self.recogniser.eval()
194
+ state = load_checkpoint(CKPT_CLASSIFIER)
195
+ self.embedder = WriterEmbedder().to(DEVICE)
196
+ self.embedder.load_state_dict({k: v.float() for k, v in state["m"].items()})
197
+ self.embedder.eval()
198
+
199
+ def clean(self, text):
200
+ """Drops the characters the model does not know."""
201
+ return "".join(c for c in text if c in self.stoi)
202
+
203
+ @torch.no_grad()
204
+ def _durations(self, text, semb):
205
+ ids = [1] + [self.stoi[c] for c in text] + [2] # <bos> ... <eos>
206
+ tokens = torch.tensor([ids], device=DEVICE)
207
+ out = self.planner(tokens, torch.ones(1, len(ids), device=DEVICE), semb.float())
208
+ d = torch.expm1(out).clamp(min=1).round().long().cpu().numpy()
209
+ return [int(v) for v in d[0, 1:-1]]
210
+
211
+ @torch.no_grad()
212
+ def _read(self, lines):
213
+ """Equal-length lines -> the recognised character ids of each."""
214
+ x = np.stack([np.stack([q[:, 0] * 20.0, q[:, 1] * 20.0, q[:, 2], np.cumsum(q[:, 1])], 1) for q in lines]).astype(np.float32)
215
+ best = self.recogniser(torch.from_numpy(x).to(DEVICE)).argmax(-1).cpu().numpy()
216
+ out = []
217
+ for row in best:
218
+ ids, prev = [], 0
219
+ for c in row:
220
+ if c != 0 and c != prev:
221
+ ids.append(int(c))
222
+ prev = c
223
+ out.append(ids)
224
+ return out
225
+
226
+ def read(self, line):
227
+ """What the recogniser reads in a line. Use it to check or fill in the text of your reference."""
228
+ return "".join(s for s in (self.itos[i] for i in self._read([np.asarray(line, np.float32)])[0]) if len(s) == 1)
229
+
230
+ @torch.no_grad()
231
+ def _embed(self, line):
232
+ """The unit writer-classifier vector of a line, after bringing the line to the reference form."""
233
+ x = features(resample(to_reference(to_strokes(line))))
234
+ e = self.embedder(torch.from_numpy(x[None]).to(DEVICE)).float()[0]
235
+ return F.normalize(e, dim=-1)
236
+
237
+ @torch.no_grad()
238
+ def reference(self, line, text):
239
+ """A reference line and what is written in it -> everything the recipe needs from the writer."""
240
+ line = np.asarray(line, np.float32)[:, :3]
241
+ semb = style_vector(self.style_net, line)
242
+ text = self.clean(text)
243
+ spread = 1.0
244
+ if text:
245
+ spread = float(np.clip(len(line) / max(1, sum(self._durations(text, semb))), SPREAD_MIN, SPREAD_MAX))
246
+ return {
247
+ "semb": semb,
248
+ "style": self.net.semb_proj(semb) + self.net.src_emb[0][None], # written as tablet ink
249
+ "keys": self.net.enc(torch.from_numpy(line[:MAX_POINTS]).to(DEVICE)),
250
+ "embedding": self._embed(line),
251
+ "spread": spread,
252
+ }
253
+
254
+ @torch.no_grad()
255
+ def write(self, text, ref, attempts=ATTEMPTS, seed=0):
256
+ """-> {"lines": the attempts, "pick": the index of the chosen one, "cer", "cos", "text": the text after cleaning}."""
257
+ text = self.clean(text)
258
+ if not text:
259
+ raise ValueError("no characters the model knows")
260
+ ids = [self.stoi[c] for c in text]
261
+ durations = [max(1, int(round(d * ref["spread"]))) for d in self._durations(text, ref["semb"])]
262
+ length = sum(durations)
263
+ if length > MAX_POINTS:
264
+ raise ValueError(f"the line is too long: {length} pen points, the model takes {MAX_POINTS}. Shorten the text.")
265
+ B = attempts
266
+ plan = _plan(ids, durations)
267
+ cond = {
268
+ k: torch.tensor(v, dtype=torch.float32 if k in ("progress", "duration") else torch.long, device=DEVICE)[None].expand(B, -1)
269
+ for k, v in zip(("char", "prev", "next", "phase", "progress", "duration"), plan)
270
+ }
271
+ cond["mask"] = torch.ones(B, length, device=DEVICE)
272
+ # classifier-free guidance: the same batch twice, the second time with the characters dropped; style and reference stay
273
+ dropped = {k: (torch.full_like(v, self.null_token) if k in ("char", "prev", "next") else v) for k, v in cond.items()}
274
+ cond2 = {k: torch.cat([cond[k], dropped[k]], 0) for k in cond}
275
+ style2 = ref["style"].expand(2 * B, -1)
276
+ keys2 = ref["keys"].expand(2 * B, -1, -1)
277
+
278
+ generator = torch.Generator().manual_seed(seed)
279
+ z = torch.randn(B, length, 3, generator=generator).to(DEVICE)
280
+ self_cond = torch.zeros_like(z)
281
+ steps = np.linspace(NOISE_STEPS - 1, 0, SAMPLE_STEPS).round().astype(int)
282
+ for k, t in enumerate(steps):
283
+ tt = torch.full((2 * B,), int(t), device=DEVICE, dtype=torch.long)
284
+ with torch.autocast("cuda", dtype=torch.bfloat16, enabled=AMP):
285
+ xy, pen = self.net(torch.cat([z, z], 0), tt, cond2, style2, torch.cat([self_cond, self_cond], 0), keys2)
286
+ xy = xy[B:] + GUIDANCE * (xy[:B] - xy[B:])
287
+ pen = pen[B:] + GUIDANCE * (pen[:B] - pen[B:])
288
+ x0 = torch.cat([xy.float(), torch.tanh(pen.float() / 2)[..., None]], -1)
289
+ self_cond = x0
290
+ if k == len(steps) - 1:
291
+ break
292
+ a, a_next = ALPHA_BAR[t], ALPHA_BAR[steps[k + 1]]
293
+ eps = (z - a.sqrt() * x0) / (1 - a).sqrt()
294
+ z = a_next.sqrt() * x0 + (1 - a_next).sqrt() * eps # DDIM
295
+
296
+ index = torch.arange(length, device=DEVICE, dtype=torch.float32)[None]
297
+ points = torch.stack([x0[..., 0] * SCALE_X + ADVANCE * index, x0[..., 1] * SCALE_Y], -1).cpu().numpy()
298
+ pen_up = (pen.float() > 0.0).float().cpu().numpy()
299
+ lines = []
300
+ for b in range(B):
301
+ d = np.zeros((length, 3), np.float32)
302
+ d[1:, :2] = np.diff(points[b], axis=0)
303
+ d[:, 2] = pen_up[b]
304
+ d[-1, 2] = 1
305
+ lines.append(d)
306
+
307
+ cer = np.array([_edit_rate(ids, h) for h in self._read(lines)], np.float32)
308
+ cos = np.array([float(self._embed(q) @ ref["embedding"]) for q in lines], np.float32)
309
+ legible = np.flatnonzero(cer <= cer.min() + CER_SLACK)
310
+ pick = int(legible[np.argmax(cos[legible])])
311
+ return {"lines": lines, "pick": pick, "cer": cer, "cos": cos, "text": text, "durations": durations}
tablet/README.md ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # A line from a tablet
2
+
3
+ If your handwriting already lives in a note app, the pen strokes are in its files and no tracer is needed: the reference line is taken exactly as you wrote it.
4
+
5
+ The steps are the same for every app:
6
+
7
+ 1. Get the notebook out of the app **in the app's own format**, not as a PDF or a picture (those keep the look of the ink, not the pen path).
8
+ 2. Find the page and the line, and look at the picture to check:
9
+
10
+ ```bash
11
+ python tablet/app_line.py notes.goodnotes
12
+ python tablet/app_line.py notes.goodnotes my_line.json --page 3 --line 2 --svg check.svg
13
+ ```
14
+
15
+ 3. Make the reference and write:
16
+
17
+ ```bash
18
+ python make_ref.py my_line.json my_line.npy
19
+ python write_line.py "Any text you like." out_line --ref my_line.npy --ref-text "what the line says"
20
+ python write_pages.py notes.txt out --ref my_line.npy
21
+ ```
22
+
23
+ Pick a line of ordinary prose, 30-50 characters, without formulas, underlines or corrections. The surest way is to write one such line on a new empty page: then `--page N` alone takes the whole page and nothing has to be split.
24
+
25
+ ## By app
26
+
27
+ | App | File to export | Command |
28
+ |---|---|---|
29
+ | GoodNotes 5 and 6 (iPad) | the notebook as a Goodnotes document, `.goodnotes` | `python tablet/app_line.py notes.goodnotes ...` |
30
+ | Notewise (iPad, Android) | the note as a Notewise file, `.notewise` | `python tablet/app_line.py notes.notewise ...` |
31
+ | Huawei Notes | the note as a note file, `.hinote` | `python tablet/app_line.py notes.hinote ...` |
32
+ | Kilonotes (Android) | the notebook in the app's own export, a `.zip` | `python tablet/app_line.py notes.zip ...` |
33
+ | Samsung Notes | the note as a Samsung Notes file, `.sdocx` | two steps, see below |
34
+ | any other app | a picture of the page | the photo route, see below |
35
+
36
+ The names of the export menus change between versions of the apps, so they are not listed here: look for "export" or "share" and choose the format named in the table.
37
+
38
+ ### Samsung Notes
39
+
40
+ The `.sdocx` format is read by the npm package [@twango/sdocx](https://github.com/twangodev/sdocx) (GPL-3.0), which is installed separately and is not part of this repository. It needs Node.js.
41
+
42
+ ```bash
43
+ (cd tablet/samsung && npm ci)
44
+ node tablet/samsung/sdocx2ink.mjs pages notes.sdocx # writes pages/notes_p00.json, notes_p01.json, ...
45
+ python tablet/app_line.py pages/notes_p00.json my_line.json --line 2 --svg check.svg
46
+ ```
47
+
48
+ ### Any other app
49
+
50
+ Apple Notes, Notability, OneNote, Xournal++ and the rest are not read here. Use the photo route: export the page as a picture or take a screenshot, crop it to one line and run the [tracer](../trace/README.md). Ink from a screen is clean, with no paper, shadow or ruling, which is the easy case for it.
51
+
52
+ ```bash
53
+ python download_weights.py --tracer
54
+ pip install -r trace/requirements.txt
55
+ python trace/scrawler_trace.py my_line.png --out my_line.json --svg check.svg
56
+ python make_ref.py my_line.json my_line.npy
57
+ ```
58
+
59
+ ### Your own strokes
60
+
61
+ If you can get the strokes yourself, write them as a JSON list of strokes, each a list of `[x, y]` points, y pointing down, in writing order, in any units. That is all `make_ref.py` needs.
62
+
63
+ ## What to check
64
+
65
+ - **The picture.** `--svg check.svg` draws the page with the lines numbered and the chosen one in red. The page is split into lines by where the ink is densest from top to bottom. That works on ordinary rows of text and can go wrong on crowded pages, formulas and drawings.
66
+ - **The order of the strokes.** The models read the line in writing order. Strokes come out in the order the app stores them, which is the writing order unless you moved or pasted ink afterwards.
67
+ - **Highlighter and shapes** are skipped: only pen strokes are taken.
68
+
69
+ ## What this was tested on
70
+
71
+ The readers are reverse-engineered: none of these formats has a public description. Each one was run on real notebooks and read all of them: GoodNotes 5 and 6 (56 notebooks), Notewise (27), Huawei Notes (41 exports from two versions of the app, 34 of them with ink), Kilonotes (8 exports), Samsung Notes (31 notes, through @twango/sdocx). A newer version of an app may write something they do not know; if `app_line.py` finds no strokes or draws nonsense, use the photo route.
72
+
73
+ The files are only read as data. Nothing in them is executed and nothing is unpacked to disk.
tablet/app_line.py ADDED
@@ -0,0 +1,184 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """One handwritten line out of a notebook of a tablet app -> the strokes JSON that make_ref.py takes.
2
+
3
+ python tablet/app_line.py notes.goodnotes # what is in the file: pages and the lines found
4
+ python tablet/app_line.py notes.goodnotes my_line.json --page 3 --line 2 --svg check.svg
5
+ python make_ref.py my_line.json my_line.npy
6
+
7
+ Reads GoodNotes (.goodnotes), Notewise (.notewise), Huawei Notes (.hinote), Kilonotes (.zip export) and the page JSON that
8
+ tablet/samsung/sdocx2ink.mjs writes from a Samsung Notes .sdocx. See tablet/README.md for how to get the file out of each app.
9
+
10
+ A page is split into lines by where the ink is densest from top to bottom. That works on ordinary rows of text and can fail on
11
+ crowded pages, formulas and drawings, so look at the --svg picture: the page with the lines numbered and the chosen one in red.
12
+ The surest way is a new page with one line on it; then no --line is needed.
13
+ """
14
+ import argparse
15
+ import json
16
+ import os
17
+ import sys
18
+ import zipfile
19
+
20
+ import numpy as np
21
+
22
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
23
+
24
+
25
+ def read_pages(path):
26
+ """-> (app name, pages); a page = {"width", "height", "strokes": [[[x, y], ...], ...]}, y pointing down."""
27
+ low = path.lower()
28
+ if low.endswith(".json"):
29
+ with open(path, encoding="utf-8") as fh:
30
+ page = json.load(fh)
31
+ if not isinstance(page, dict) or "strokes" not in page:
32
+ raise SystemExit("this JSON is not a page written by tablet/samsung/sdocx2ink.mjs")
33
+ strokes = [[p[:2] for p in s["points"]] for s in page["strokes"] if s.get("points")]
34
+ return {"sdocx": "Samsung Notes"}.get(page.get("source"), "page JSON"), [dict(width=page.get("width"), height=page.get("height"), strokes=strokes)]
35
+ if low.endswith(".sdocx"):
36
+ raise SystemExit("Samsung Notes: first run node tablet/samsung/sdocx2ink.mjs pages notes.sdocx and give this script a page JSON")
37
+ if low.endswith(".goodnotes"):
38
+ import goodnotes
39
+
40
+ return "GoodNotes", goodnotes.read(path)
41
+ if low.endswith(".notewise"):
42
+ import notewise
43
+
44
+ return "Notewise", notewise.read(path)
45
+ if low.endswith(".hinote"):
46
+ import huawei
47
+
48
+ return "Huawei Notes", huawei.read(path)
49
+ if zipfile.is_zipfile(path):
50
+ names = zipfile.ZipFile(path).namelist()
51
+ if any(n.endswith("/info") and "/pages/" in n for n in names):
52
+ import kilonotes
53
+
54
+ return "Kilonotes", kilonotes.read(path)
55
+ if any(n.startswith("notes/") for n in names):
56
+ import goodnotes
57
+
58
+ return "GoodNotes", goodnotes.read(path)
59
+ if any(n.startswith("page/") for n in names):
60
+ import notewise
61
+
62
+ return "Notewise", notewise.read(path)
63
+ if any(n.endswith(".jhinote") for n in names):
64
+ import huawei
65
+
66
+ return "Huawei Notes", huawei.read(path)
67
+ raise SystemExit("unknown file: expected .goodnotes, .notewise, .hinote, a Kilonotes .zip or a page .json from samsung/sdocx2ink.mjs")
68
+
69
+
70
+ def split_lines(strokes):
71
+ """Strokes of a page -> lists of stroke indices, one per text line, top to bottom; writing order is kept inside a line."""
72
+ strokes = [np.asarray(s, np.float64) for s in strokes]
73
+ if not strokes:
74
+ return []
75
+ heights = np.array([np.ptp(s[:, 1]) for s in strokes])
76
+ widths = np.array([np.ptp(s[:, 0]) for s in strokes])
77
+ letters = heights[(heights > 0) & (np.maximum(heights, widths) > 0.2 * np.median(np.maximum(heights, widths)))]
78
+ h = float(np.median(letters)) if len(letters) else float(max(heights.max(), 1.0))
79
+ if h <= 0:
80
+ return [list(range(len(strokes)))]
81
+ # ink along y: every stroke spreads its length evenly over its points
82
+ ys, ws = [], []
83
+ for s in strokes:
84
+ length = float(np.hypot(*np.diff(s, axis=0).T).sum()) if len(s) > 1 else 0.0
85
+ ys.append(s[:, 1])
86
+ ws.append(np.full(len(s), (length + 0.1 * h) / len(s)))
87
+ ys, ws = np.concatenate(ys), np.concatenate(ws)
88
+ step = h / 8
89
+ edges = np.arange(ys.min() - 2 * h, ys.max() + 2 * h + step, step)
90
+ density, _ = np.histogram(ys, edges, weights=ws)
91
+ kernel = np.hanning(2 * 8 + 1) # about one letter height wide
92
+ density = np.convolve(density, kernel / kernel.sum(), mode="same")
93
+ floor = 0.12 * density.max()
94
+ peaks = [i for i in range(1, len(density) - 1) if density[i] > floor and density[i] >= density[i - 1] and density[i] > density[i + 1]]
95
+ merged = []
96
+ for i in peaks: # two maxima closer than a letter height with no real dip between them are one line
97
+ if merged and (i - merged[-1]) * step < 1.1 * h and density[merged[-1] : i + 1].min() > 0.6 * min(density[i], density[merged[-1]]):
98
+ if density[i] > density[merged[-1]]:
99
+ merged[-1] = i
100
+ else:
101
+ merged.append(i)
102
+ centres = np.array([edges[i] + step / 2 for i in merged])
103
+ if len(centres) == 0:
104
+ return [list(range(len(strokes)))]
105
+ lines = [[] for _ in centres]
106
+ for k, s in enumerate(strokes):
107
+ lines[int(np.argmin(np.abs(centres - np.median(s[:, 1]))))].append(k)
108
+ return [line for line in lines if line]
109
+
110
+
111
+ def write_svg(path, page, lines, chosen):
112
+ strokes = [np.asarray(s, np.float64) for s in page["strokes"]]
113
+ points = np.concatenate(strokes)
114
+ x0, y0 = points.min(0)
115
+ x1, y1 = points.max(0)
116
+ pad = 0.04 * max(x1 - x0, y1 - y0, 1.0)
117
+ W, H = x1 - x0 + 2 * pad, y1 - y0 + 2 * pad
118
+ k = 1200.0 / W
119
+ size = max(10.0, 0.012 * 1200)
120
+ out = [f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 {H * k:.0f}" width="1200" height="{H * k:.0f}"><rect width="100%" height="100%" fill="#fff"/>']
121
+ for n, line in enumerate(lines):
122
+ colour = "#c33f2e" if n == chosen else "#141414"
123
+ d = " ".join("M" + " L".join(f"{(x - x0 + pad) * k:.1f},{(y - y0 + pad) * k:.1f}" for x, y in strokes[i]) for i in line)
124
+ out.append(f'<path d="{d}" fill="none" stroke="{colour}" stroke-width="1.3" stroke-linecap="round" stroke-linejoin="round"/>')
125
+ top = min(strokes[i][:, 1].min() for i in line)
126
+ left = min(strokes[i][:, 0].min() for i in line)
127
+ out.append(f'<text x="{max(2.0, (left - x0 + pad) * k - 2.2 * size):.1f}" y="{(top - y0 + pad) * k + size:.1f}" font-family="sans-serif" font-size="{size:.0f}" fill="#2a6fb0">{n + 1}</text>')
128
+ out.append("</svg>")
129
+ with open(path, "w") as fh:
130
+ fh.write("".join(out))
131
+
132
+
133
+ def main():
134
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
135
+ ap.add_argument("notebook", help="the file exported from the app")
136
+ ap.add_argument("out", nargs="?", help="where to write the strokes of the line (JSON); without it the file is only listed")
137
+ ap.add_argument("--page", type=int, help="page number, from 1; default: the only page with ink")
138
+ ap.add_argument("--line", type=int, help="line number on the page, from 1; default: the whole page is the line")
139
+ ap.add_argument("--svg", help="also draw the page with the lines numbered")
140
+ args = ap.parse_args()
141
+
142
+ app, pages = read_pages(args.notebook)
143
+ inked = [n for n, p in enumerate(pages) if p["strokes"]]
144
+ if not inked:
145
+ raise SystemExit(f"{app}: no pen strokes found in this file")
146
+ if args.out is None and args.page is None:
147
+ print(f"{app}: {len(pages)} pages, {len(inked)} with ink")
148
+ for n in inked:
149
+ print(f" page {n + 1}: {len(pages[n]['strokes'])} strokes, {len(split_lines(pages[n]['strokes']))} lines")
150
+ print("pick one: --page N [--line K], and look at it with --svg check.svg")
151
+ return
152
+ if args.page is None:
153
+ if len(inked) > 1:
154
+ raise SystemExit(f"{len(inked)} pages have ink; choose one with --page (run without the output name to list them)")
155
+ args.page = inked[0] + 1
156
+ if not 1 <= args.page <= len(pages) or not pages[args.page - 1]["strokes"]:
157
+ raise SystemExit(f"page {args.page} has no ink")
158
+ page = pages[args.page - 1]
159
+ lines = split_lines(page["strokes"])
160
+ if args.line is None:
161
+ chosen, picked = None, list(range(len(page["strokes"])))
162
+ if len(lines) > 1:
163
+ print(f"the page seems to hold {len(lines)} lines and all of them are taken as one; choose with --line if that is wrong")
164
+ else:
165
+ if not 1 <= args.line <= len(lines):
166
+ raise SystemExit(f"the page has {len(lines)} lines")
167
+ chosen, picked = args.line - 1, lines[args.line - 1]
168
+ for n, line in enumerate(lines):
169
+ xs = np.concatenate([np.asarray(page["strokes"][i], np.float64)[:, 0] for i in line])
170
+ print(f" line {n + 1}: {len(line)} strokes, width {np.ptp(xs):.0f}" + (" <- taken" if n == chosen else ""))
171
+ if args.svg:
172
+ write_svg(args.svg, page, lines if chosen is not None else [picked], chosen if chosen is not None else 0)
173
+ print("picture:", args.svg)
174
+ if args.out:
175
+ if os.path.exists(args.out):
176
+ raise SystemExit(f"{args.out} already exists")
177
+ strokes = [[[round(float(x), 3), round(float(y), 3)] for x, y in page["strokes"][i]] for i in picked]
178
+ with open(args.out, "w") as fh:
179
+ json.dump(strokes, fh)
180
+ print(f"{args.out}: {len(strokes)} strokes from {app}. Next: python make_ref.py {args.out} my_line.npy")
181
+
182
+
183
+ if __name__ == "__main__":
184
+ main()
tablet/goodnotes.py ADDED
@@ -0,0 +1,425 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """GoodNotes 5 and 6 notebooks (.goodnotes) -> pen strokes of every page.
2
+
3
+ The file is only read as data: zipfile, a hand-written protobuf wire decoder and an LZ4 block decoder. Nothing is extracted to
4
+ disk. The format has no public description; these notes are reverse-engineered from real notebooks and may not cover every
5
+ version of the app.
6
+ The notes below come from a fuller converter: they also describe fields, flags and options that this reader does not use.
7
+
8
+ - zip: notes/<uuid> page content, search/<uuid> recognition index, attachments/<uuid> PDF/JPEG/MP4,
9
+ index.events.pb document/page records, index.*.pb file lists.
10
+ - every file is a stream of varint-length-delimited protobuf records.
11
+ - index.events.pb: record field 54 = page {2: page id, 3: {1: template id}, 4: {1: fractional order key}};
12
+ field 2 = page template {2: id, 4: background attachment, 8: {1: w, 2: h} page size in points}.
13
+ notes/ and search/ file uuid = page uuid + 1 (as a 128-bit integer).
14
+ - notes/<uuid>: pairs of records keyed by element uuid: a header {1: uuid, 2: {1: version}, 3: 1 if deleted,
15
+ 9: Lamport clock} and a body. Body field 7 = stroke, field 1 = placed image, field 20 = text box.
16
+ Stroke body: 2 = Apple "bv41" LZ4 frame holding a "tpl" record with signature
17
+ vuA(v)A(S(uu))A(S(uuuu))vA(f) = [2, pen width, segment types, [start point], [(cx,cy,ex,ey) quad segments],
18
+ 1, extra floats]; 4 = RGBA floats 0..1; 5 = 1 for highlighter; 6 = {1: dx, 2: dy} translation (lasso move);
19
+ 7 = {1,2: z-order key}; 9 = recognised shape (line/ellipse/polygon) with empty point list; 14 = 1 deleted.
20
+ - The quad segments are midpoint smoothing of the raw samples: end_i == (c_i + c_{i+1}) / 2 exactly, so the
21
+ raw (already distance-decimated) pen samples are start, c_1..c_n, e_n. Those are what we write.
22
+ - search/<uuid>: one protobuf message, field 6 repeated = recognition entries in reading order:
23
+ {1: bbox {1: origin, 2: size}, 3: top-3 alternatives, 8: line number, 11: locale,
24
+ 7: items {3: stroke uuids of the group, 4: the word's points}}. Entries with one item are words, entries with two
25
+ items are separators (' ' or '\\n') between them. Words are mapped to strokes by exact point match; a cursive
26
+ stroke can be split between two words, so a stroke may be listed under both.
27
+ - GoodNotes 6 (2025+, seen in notebooks of October 2026) writes a second stroke record, signature
28
+ vA(v)A(u)A(u)A(v)A(v)A(u)A(u)A(u)A(u)A(v) = [2, u16 flags per segment, [x, y, w] start sample, [x, y, w]*N raw
29
+ samples, u16 per segment, u16 ops, [x, y]*M, [], [x, y]*K outline, [x, y, w, tx, ty]*N, u16*N]. When the first
30
+ segment flag has bit 4 set (flags 4, 5, ...; seen in 2 of 23 notebooks, mixed with the other kind), the start record is
31
+ [x, y, w, e] and the sample array is per segment [x, y, w, x, y, w, e1, e2, e3] (two samples + an extra triple that
32
+ is skipped; reading it as samples draws fans of lines across the page). Only the samples
33
+ are used: x, y and w = rendered width per point (pressure-driven, 0.6..2.2 pt, no nominal width in the record).
34
+ The width is kept as "widths"; pressure = w / nominal, nominal = 99th percentile of w over the notebook
35
+ (written as width_nominal on the page), page tagged pressure:"width". The old record keeps pressure 1.0.
36
+ - Otherwise there is NO per-point pressure and NO per-point or per-stroke clock time in the file: pressure is written as
37
+ 1.0, t_ms as the point index inside the stroke, t0_ms as null, and the page is tagged time:"ordinal".
38
+ Strokes are ordered by the z-order key (= drawing order unless re-arranged), ties by the Lamport clock.
39
+ """
40
+ import sys
41
+ import os
42
+ import json
43
+ import struct
44
+ import zipfile
45
+ import collections
46
+ import uuid as uuidlib
47
+
48
+
49
+ def varint(b, i):
50
+ r = 0
51
+ s = 0
52
+ while True:
53
+ c = b[i]
54
+ i += 1
55
+ r |= (c & 0x7F) << s
56
+ s += 7
57
+ if c < 0x80:
58
+ return r, i
59
+
60
+
61
+ def fields(b):
62
+ # generic protobuf wire decoder -> list of (field, wiretype, value)
63
+ i = 0
64
+ out = []
65
+ n = len(b)
66
+ while i < n:
67
+ k, i = varint(b, i)
68
+ f, w = k >> 3, k & 7
69
+ if w == 0:
70
+ v, i = varint(b, i)
71
+ elif w == 1:
72
+ v = b[i:i + 8]
73
+ i += 8
74
+ elif w == 2:
75
+ ln, i = varint(b, i)
76
+ v = b[i:i + ln]
77
+ i += ln
78
+ elif w == 5:
79
+ v = b[i:i + 4]
80
+ i += 4
81
+ else:
82
+ raise ValueError('wire type %d' % w)
83
+ if i > n:
84
+ raise ValueError('field overrun')
85
+ out.append((f, w, v))
86
+ return out
87
+
88
+
89
+ def fdict(b):
90
+ # field -> list of values
91
+ d = {}
92
+ for f, w, v in fields(b):
93
+ d.setdefault(f, []).append(v)
94
+ return d
95
+
96
+
97
+ def first(d, k, default=None):
98
+ v = d.get(k)
99
+ return v[0] if v else default
100
+
101
+
102
+ def records(b):
103
+ i = 0
104
+ out = []
105
+ while i < len(b):
106
+ ln, i = varint(b, i)
107
+ out.append(b[i:i + ln])
108
+ i += ln
109
+ return out
110
+
111
+
112
+ def f32(v):
113
+ return struct.unpack('<f', v)[0]
114
+
115
+
116
+ def vec2(b):
117
+ d = fdict(b)
118
+ x = first(d, 1)
119
+ y = first(d, 2)
120
+ return (f32(x) if x is not None else 0.0, f32(y) if y is not None else 0.0)
121
+
122
+
123
+ def lz4_block(src):
124
+ dst = bytearray()
125
+ i = 0
126
+ n = len(src)
127
+ while i < n:
128
+ tok = src[i]
129
+ i += 1
130
+ ll = tok >> 4
131
+ if ll == 15:
132
+ while True:
133
+ c = src[i]
134
+ i += 1
135
+ ll += c
136
+ if c != 255:
137
+ break
138
+ dst += src[i:i + ll]
139
+ i += ll
140
+ if i >= n:
141
+ break
142
+ off = src[i] | (src[i + 1] << 8)
143
+ i += 2
144
+ ml = tok & 15
145
+ if ml == 15:
146
+ while True:
147
+ c = src[i]
148
+ i += 1
149
+ ml += c
150
+ if c != 255:
151
+ break
152
+ ml += 4
153
+ st = len(dst) - off
154
+ if st < 0:
155
+ raise ValueError('lz4 offset')
156
+ for k in range(ml):
157
+ dst.append(dst[st + k])
158
+ return bytes(dst)
159
+
160
+
161
+ def bvx(b):
162
+ # Apple libcompression LZ4 container: blocks "bv41" <raw u32> <packed u32> data | "bv4-" <raw u32> data | "bv4$"
163
+ out = bytearray()
164
+ i = 0
165
+ while i < len(b):
166
+ m = b[i:i + 4]
167
+ if m == b'bv41':
168
+ dl, cl = struct.unpack('<II', b[i + 4:i + 12])
169
+ blk = lz4_block(b[i + 12:i + 12 + cl])
170
+ if len(blk) != dl:
171
+ raise ValueError('lz4 size')
172
+ out += blk
173
+ i += 12 + cl
174
+ elif m == b'bv4-':
175
+ dl, = struct.unpack('<I', b[i + 4:i + 8])
176
+ out += b[i + 8:i + 8 + dl]
177
+ i += 8 + dl
178
+ elif m == b'bv4$':
179
+ break
180
+ else:
181
+ raise ValueError('bvx magic %r' % m)
182
+ return bytes(out)
183
+
184
+
185
+ def tpl_sig(s, i=0):
186
+ out = []
187
+ while i < len(s):
188
+ c = s[i]
189
+ if c in 'vuf':
190
+ out.append(c)
191
+ i += 1
192
+ elif c in 'AS':
193
+ sub, i = tpl_sig(s, i + 2)
194
+ out.append((c, sub))
195
+ elif c == ')':
196
+ return out, i + 1
197
+ else:
198
+ raise ValueError('tpl char %r' % c)
199
+ return out, i
200
+
201
+
202
+ def tpl_node(node, b, o):
203
+ # v = u16, u = 4-byte value (float in practice), f = float, A(..) = u32 count + items, S(..) = struct
204
+ if isinstance(node, str):
205
+ if node == 'v':
206
+ return struct.unpack('<H', b[o:o + 2])[0], o + 2
207
+ if o + 4 > len(b):
208
+ raise ValueError('tpl short')
209
+ return struct.unpack('<f', b[o:o + 4])[0], o + 4
210
+ kind, sub = node
211
+ if kind == 'S':
212
+ vals = []
213
+ for s in sub:
214
+ v, o = tpl_node(s, b, o)
215
+ vals.append(v)
216
+ return tuple(vals), o
217
+ cnt = struct.unpack('<I', b[o:o + 4])[0]
218
+ o += 4
219
+ if cnt * 2 > len(b):
220
+ raise ValueError('tpl count')
221
+ item = sub[0] if len(sub) == 1 else ('S', sub)
222
+ vals = []
223
+ for _ in range(cnt):
224
+ v, o = tpl_node(item, b, o)
225
+ vals.append(v)
226
+ return vals, o
227
+
228
+
229
+ def parse_tpl(b):
230
+ if b[:4] != b'tpl\x00':
231
+ raise ValueError('not tpl')
232
+ e = b.index(b'\x00', 8)
233
+ sig = b[8:e].decode('ascii')
234
+ nodes, _ = tpl_sig(sig)
235
+ o = e + 1
236
+ vals = []
237
+ for n in nodes:
238
+ v, o = tpl_node(n, b, o)
239
+ vals.append(v)
240
+ return sig, vals
241
+
242
+
243
+ STROKE_SIG = 'vuA(v)A(S(uu))A(S(uuuu))vA(f)'
244
+ STROKE_SIG2 = 'vA(v)A(u)A(u)A(v)A(v)A(u)A(u)A(u)A(u)A(v)'
245
+
246
+
247
+ def uuid_plus(u, k):
248
+ return str(uuidlib.UUID(int=(uuidlib.UUID(u).int + k) % (1 << 128))).upper()
249
+
250
+
251
+ def read_pages(z):
252
+ """-> list of dicts {page_id, notes, search, width, height, bg} in notebook order."""
253
+ names = set(z.namelist())
254
+ tmpl = {}
255
+ pages = []
256
+ if 'index.events.pb' in names:
257
+ for r in records(z.read('index.events.pb')):
258
+ fs = fields(r)
259
+ if len(fs) < 2 or fs[1][1] != 2:
260
+ continue
261
+ kind, body = fs[1][0], fs[1][2]
262
+ if kind == 2:
263
+ d = fdict(body)
264
+ tid = first(d, 2, b'').decode('utf-8', 'replace')
265
+ size = vec2(first(d, 8)) if first(d, 8) else None
266
+ tmpl[tid] = dict(size=size, bg=first(d, 4, b'').decode('utf-8', 'replace') or None)
267
+ elif kind == 54:
268
+ d = fdict(body)
269
+ pid = first(d, 2, b'').decode('utf-8', 'replace')
270
+ key = first(fdict(first(d, 4)), 1, b'') if first(d, 4) else b''
271
+ tid = first(fdict(first(d, 3)), 1, b'').decode('utf-8', 'replace') if first(d, 3) else None
272
+ pages.append((key, pid, tid))
273
+ # fractional-index keys sort as plain bytes
274
+ pages.sort()
275
+ out = []
276
+ used = set()
277
+ for key, pid, tid in pages:
278
+ nf = None
279
+ for k in (1, 0):
280
+ try:
281
+ cand = 'notes/' + uuid_plus(pid, k)
282
+ except ValueError:
283
+ continue
284
+ if cand in names:
285
+ nf = cand
286
+ break
287
+ if nf is None:
288
+ continue
289
+ used.add(nf)
290
+ t = tmpl.get(tid, {})
291
+ sz = t.get('size')
292
+ sf = 'search/' + nf[6:]
293
+ out.append(dict(page_id=pid, notes=nf, search=sf if sf in names else None,
294
+ width=sz[0] if sz else None, height=sz[1] if sz else None, bg=t.get('bg')))
295
+ # notes files without a page record go last, in index.notes.pb order
296
+ rest = []
297
+ if 'index.notes.pb' in names:
298
+ for r in records(z.read('index.notes.pb')):
299
+ p = first(fdict(r), 2, b'').decode('utf-8', 'replace')
300
+ if p in names and p not in used:
301
+ rest.append(p)
302
+ rest += sorted(n for n in names if n.startswith('notes/') and n not in used and n not in rest)
303
+ for nf in rest:
304
+ sf = 'search/' + nf[6:]
305
+ out.append(dict(page_id=None, notes=nf, search=sf if sf in names else None, width=None, height=None, bg=None))
306
+ return out
307
+
308
+
309
+ def read_strokes(blob, stats, shapes=False):
310
+ heads = {}
311
+ bodies = {}
312
+ for r in records(blob):
313
+ try:
314
+ fs = fields(r)
315
+ except ValueError:
316
+ stats['bad_record'] += 1
317
+ continue
318
+ if not fs:
319
+ continue
320
+ f0, w0, v0 = fs[0]
321
+ if f0 == 1 and len(fs) > 1 and w0 == 2:
322
+ d = fdict(r)
323
+ u = first(d, 1).decode('utf-8', 'replace')
324
+ ver = first(fdict(first(d, 2, b'')), 1, 0) if first(d, 2) else 0
325
+ if u not in heads or ver >= heads[u]['ver']:
326
+ heads[u] = dict(ver=ver, deleted=first(d, 3, 0) == 1, clock=first(d, 9, 0))
327
+ elif f0 == 7 and w0 == 2:
328
+ sd = fdict(v0)
329
+ u = first(sd, 1, b'').decode('utf-8', 'replace')
330
+ ver = first(fdict(first(sd, 15)), 1, 0) if first(sd, 15) else 0
331
+ if u not in bodies or ver >= bodies[u][0]:
332
+ bodies[u] = (ver, sd)
333
+ elif f0 == 1:
334
+ stats['image'] += 1
335
+ elif f0 == 20:
336
+ stats['textbox'] += 1
337
+ else:
338
+ stats['other_record'] += 1
339
+ out = []
340
+ for u, (ver, sd) in bodies.items():
341
+ h = heads.get(u, {})
342
+ if first(sd, 14, 0) == 1 or h.get('deleted'):
343
+ stats['deleted'] += 1
344
+ continue
345
+ try:
346
+ sig, vals = parse_tpl(bvx(first(sd, 2, b'')))
347
+ except (ValueError, IndexError, struct.error):
348
+ stats['bad_geometry'] += 1
349
+ continue
350
+ rgba = [f32(x) for x in (first(fdict(first(sd, 4)), k) for k in (1, 2, 3, 4)) if x is not None] if first(sd, 4) else []
351
+ color = [int(round(c * 255)) for c in rgba[:3]] if len(rgba) >= 3 else None
352
+ off = vec2(first(sd, 6)) if first(sd, 6) else (0.0, 0.0)
353
+ zk = fdict(first(sd, 7)) if first(sd, 7) else {}
354
+ zk = fdict(first(zk, 1)) if first(zk, 1) else {}
355
+ order = (first(zk, 1, 0), first(zk, 2, 0), first(zk, 3, 0), h.get('clock', 0))
356
+ pen = 'highlighter' if first(sd, 5, 0) == 1 else 'pen'
357
+ if sig == STROKE_SIG2:
358
+ if pen == 'highlighter':
359
+ stats['highlighter'] += 1
360
+ continue
361
+ raw = []
362
+ if len(vals[2]) >= 3:
363
+ raw.append(vals[2][:3])
364
+ v3 = vals[3]
365
+ flags = vals[1]
366
+ if flags and (flags[0] & 4):
367
+ # variant B: start record [x, y, w, e]; each segment = [x, y, w, x, y, w, e1, e2, e3]
368
+ for i in range(0, len(v3) - 8, 9):
369
+ raw += [v3[i:i + 3], v3[i + 3:i + 6]]
370
+ stats['strokes_v2b'] += 1
371
+ else:
372
+ raw += [v3[i:i + 3] for i in range(0, len(v3) - 2, 3)]
373
+ if not raw:
374
+ stats['empty'] += 1
375
+ continue
376
+ pts = [[round(x + off[0], 2), round(y + off[1], 2), i, 1.0] for i, (x, y, w) in enumerate(raw)]
377
+ if off != (0.0, 0.0):
378
+ stats['moved'] += 1
379
+ stats['strokes_v2'] += 1
380
+ out.append((order, dict(t0_ms=None, pen=pen, color=color, width=None, points=pts,
381
+ widths=[round(w, 4) for _, _, w in raw], pressure='width')))
382
+ continue
383
+ if sig != STROKE_SIG:
384
+ stats['other_geometry'] += 1
385
+ continue
386
+ segs = vals[4]
387
+ if not segs or not vals[3]:
388
+ shape = fdict(first(sd, 9)) if first(sd, 9) else {}
389
+ if shape and first(shape, 1) is not None:
390
+ # straight-line shape: {1: {1: {1:x,2:y}, 1: {1:x,2:y}}}
391
+ pts = [vec2(p) for p in fdict(first(shape, 1)).get(1, [])]
392
+ if shapes and len(pts) >= 2:
393
+ stats['shape_line_kept'] += 1
394
+ out.append((order, dict(t0_ms=None, pen='shape-line', color=color, width=vals[1],
395
+ points=[[round(x + off[0], 2), round(y + off[1], 2), i, 1.0] for i, (x, y) in enumerate(pts)])))
396
+ else:
397
+ stats['shape_line'] += 1
398
+ elif shape:
399
+ stats['shape_other'] += 1
400
+ else:
401
+ stats['empty'] += 1
402
+ continue
403
+ if pen == 'highlighter':
404
+ stats['highlighter'] += 1
405
+ continue
406
+ sx, sy = vals[3][0]
407
+ raw = [(sx, sy)] + [(s[0], s[1]) for s in segs] + [(segs[-1][2], segs[-1][3])]
408
+ pts = [[round(x + off[0], 2), round(y + off[1], 2), i, 1.0] for i, (x, y) in enumerate(raw)]
409
+ if off != (0.0, 0.0):
410
+ stats['moved'] += 1
411
+ if vals[6]:
412
+ stats['extra_floats'] += 1
413
+ out.append((order, dict(t0_ms=None, pen=pen, color=color, width=round(vals[1], 4), points=pts)))
414
+ out.sort(key=lambda t: t[0])
415
+ return [s for _, s in out]
416
+
417
+
418
+ def read(path):
419
+ """-> the pages in notebook order: [{"width", "height", "strokes": [[[x, y], ...], ...]}], strokes in writing order."""
420
+ z = zipfile.ZipFile(path)
421
+ out = []
422
+ for pg in read_pages(z):
423
+ strokes = read_strokes(z.read(pg['notes']), collections.Counter())
424
+ out.append(dict(width=pg['width'], height=pg['height'], strokes=[[p[:2] for p in s['points']] for s in strokes if s['pen'] == 'pen']))
425
+ return out
tablet/huawei.py ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Huawei Notes exports (.hinote) -> pen strokes of every page.
2
+
3
+ The file is only read as data. The format has no public description; these notes are reverse-engineered from real notebooks.
4
+ A truncated zip is read through its local file headers.
5
+ The notes below come from a fuller converter: they also describe fields, flags and options that this reader does not use.
6
+
7
+ Layout (zip): <note-uuid>.jhinote (gzip JSON: customNoteContent.noteTitle, pageRatio = width/height), pages/<page-uuid>.jhinote
8
+ (gzip JSON customNotePageContent: pageNumber = order in notebook, pageRatio, attachment[] whose filePath .../hwFile/<name>.bin
9
+ is the page's ink), custom_md.jhinote (hashes), files/<page>_<ts>_<n>.bin (ink) + files/<page>_<ts>_<n>.jpg (rendered preview,
10
+ 763x1080 for A4). Pages without a .bin attachment carry no ink and are skipped.
11
+ .bin "PENCILENGINE", all big-endian: magic[12], u32 header_len=76, u32[4] (0x10001,0,1,0), zero pad to 76. Then nested chunks,
12
+ each = 48-byte header [u32 hdr_len=48, u32 type, u32 version, u32 payload_len, uuid[16], u32 x4] + payload.
13
+ type 2 = layer (one per page): ex = [pre_len=12, n_strokes, strokes_bytes, 1]; payload = u32[3] (0, 1000, 1000) then
14
+ n_strokes stroke chunks, then a trailing type-3 chunk (stroke-uuid index, ignored).
15
+ type 0 = stroke (version 0x10000 or 0x10002, same layout): ex = [68, 48, 116, data_len]; payload = 116-byte header + data.
16
+ header u32 slots: 0-1 i64 t_modified (epoch ms; == t_start unless the stroke was moved/copied later), 3 f32 last point
17
+ index, 4 0x01000000, 5 0xffffffff for free ink (shape id otherwise), 6-7 i64 t_start (pen down, epoch ms), 8-9 i64 t_end,
18
+ 10-16 extra params (non-zero only on tool-11 strokes), 17 u32 tool, 19-21 f32 b,g,r in 0..1 (BGR order!), 22-23 f32 (1.0),
19
+ 24 f32 brush width (page units), rest 0.
20
+ tool: 1 = pen (99.8% of strokes), 11 = a second pen type (same real pressure/time, e.g. rotated red notes),
21
+ 4 = translucent highlighter, 2 = shape tool (lines/rects/circles: every point pressure 1.0 and t 0).
22
+ data: u32 0|2 (2 = shape), u32 n_points, u32 rec_size=36, n_points x [u32 flag (0 down, 4/6 move), f32 index, f32 x,
23
+ f32 y, u32 t_ms since t_start, f32 pressure 0..1, f32 orientation (rad), f32 tilt (rad), f32 0.2], then u32 5 + f32 last
24
+ index (pen-up marker, no coordinates). Point 0 (flag 0) is the pen-down sample at t=0, usually at/next to point 1; kept.
25
+ Coordinates: page units, page width = 1000, height = 1000/pageRatio (1414.5 for A4, 1196.6 for the 0.8357 PDF pages),
26
+ origin top-left, y down; the preview jpg is the same page scaled by jpg_width/1000. Pressure and per-point time are real
27
+ (stylus, ~8-16 ms steps). Strokes that were moved/copied after writing (~1.4%) keep pressure but every point has t=0: for
28
+ them t_ms = point index (header t_start/t_end still real). Erased ink is not stored (no eraser records; renders of the
29
+ stored strokes match the previews), so every stored stroke is visible. Written by default: tools 1 and 11 with slot 5 ==
30
+ 0xffffffff and data flag 0 (pen='pen' / 'pen11'), in file order (= drawing order; t_start decreases only at 0.04% of
31
+ neighbouring pairs, moved strokes), t0_ms = t_start. Shapes (tool 2, or
32
+ slot 5 != 0xffffffff, or data flag 2) only with --shapes as pen='shape'; highlighter (tool 4) only with --highlighter as
33
+ pen='highlighter'. Inserted pictures/PDF backgrounds live in the page JSON (pageElement, bkgAttachmentId), not in .bin,
34
+ and are never written. Colour is kept as stored: 3 pages with dark paper (background base6) are written in white ink.
35
+ """
36
+ import sys, os, json, zipfile, gzip, struct, zlib, io, re
37
+
38
+ TOOLS = {1: 'pen', 11: 'pen11', 4: 'highlighter', 2: 'shape'}
39
+ U = lambda d, o: struct.unpack_from('>I', d, o)[0]
40
+
41
+
42
+ def read_entries(path):
43
+ """name -> bytes. Normal zip via zipfile (in memory); broken/truncated zip by scanning local headers."""
44
+ raw = open(path, 'rb').read()
45
+ try:
46
+ z = zipfile.ZipFile(io.BytesIO(raw)); return {n: z.read(n) for n in z.namelist() if not n.endswith('/')}, None
47
+ except zipfile.BadZipFile:
48
+ pass
49
+ out, o = {}, 0
50
+ while True:
51
+ o = raw.find(b'PK\x03\x04', o)
52
+ if o < 0 or o + 30 > len(raw): break
53
+ ver, flg, meth, _, _, crc, csz, usz, nl, xl = struct.unpack_from('<HHHHHIIIHH', raw, o + 4)
54
+ name = raw[o + 30:o + 30 + nl].decode('utf8', 'replace'); b = o + 30 + nl + xl
55
+ try:
56
+ if meth == 8:
57
+ dec = zlib.decompressobj(-15); data = dec.decompress(raw[b:b + csz] if csz and not flg & 8 else raw[b:])
58
+ if not dec.eof: raise ValueError('truncated')
59
+ end = len(raw) - len(dec.unused_data) if not (csz and not flg & 8) else b + csz
60
+ elif meth == 0 and csz and not flg & 8:
61
+ data = raw[b:b + csz]; end = b + csz
62
+ if len(data) < csz: raise ValueError('truncated')
63
+ else:
64
+ raise ValueError('method')
65
+ if crc and zlib.crc32(data) & 0xFFFFFFFF != crc: raise ValueError('crc')
66
+ if not name.endswith('/'): out[name] = data
67
+ o = max(end, o + 4)
68
+ except Exception:
69
+ o += 4
70
+ return out, 'recovered %d entries from truncated zip' % len(out)
71
+
72
+
73
+ def jload(b):
74
+ try: b = gzip.decompress(b)
75
+ except OSError: pass
76
+ return json.loads(b.decode('utf8'))
77
+
78
+
79
+ def decode_bin(d, shapes=False, highlighter=False):
80
+ """PENCILENGINE bytes -> list of ink-v1 strokes (file = drawing order)."""
81
+ if d[:12] != b'PENCILENGINE': raise ValueError('bad magic')
82
+ o, strokes = U(d, 12), []
83
+ while o + 48 <= len(d):
84
+ hl, typ, ver, L = U(d, o), U(d, o + 4), U(d, o + 8), U(d, o + 12)
85
+ if typ == 2:
86
+ pre, ns = U(d, o + 32), U(d, o + 36); q = o + hl + pre; e = min(o + hl + L, len(d))
87
+ for _ in range(ns):
88
+ if q + 48 > e: break
89
+ shl, styp, SL = U(d, q), U(d, q + 4), U(d, q + 12); h = q + shl; hlen = U(d, q + 40); nq = h + SL
90
+ if styp != 0 or nq > e: q = nq; continue
91
+ q = nq
92
+ tmod, = struct.unpack_from('>q', d, h); tstart, = struct.unpack_from('>q', d, h + 24)
93
+ slot5 = U(d, h + 20); tool = U(d, h + 68)
94
+ b, g, r, _a1, _a2, w = struct.unpack_from('>6f', d, h + 76)
95
+ D = h + hlen; fmt, n, rs = U(d, D), U(d, D + 4), U(d, D + 8)
96
+ pen = 'shape' if (tool == 2 or slot5 != 0xFFFFFFFF or fmt != 0) else TOOLS.get(tool, 'tool%d' % tool)
97
+ if (pen == 'shape' and not shapes) or (pen == 'highlighter' and not highlighter): continue
98
+ if n == 0 or rs < 24 or D + 12 + n * rs > nq: continue
99
+ raw = [struct.unpack_from('>IfffIf', d, D + 12 + rs * i) for i in range(n)]
100
+ notime = n > 1 and all(p[4] == 0 for p in raw)
101
+ P = [[round(x, 3), round(y, 3), i if notime else int(t), round(p, 4)] for i, (_f, _k, x, y, t, p) in enumerate(raw)]
102
+ strokes.append(dict(t0_ms=tstart or tmod, pen=pen, color=[round(255 * c) for c in (r, g, b)], width=round(w, 4), points=P))
103
+ o += hl + L
104
+ return strokes
105
+
106
+
107
+ def read(path):
108
+ """-> the pages in notebook order: [{"width", "height", "strokes": [[[x, y], ...], ...]}], strokes in writing order."""
109
+ ents, _ = read_entries(path)
110
+ head = [n for n in ents if n.endswith('.jhinote') and '/' not in n and n != 'custom_md.jhinote']
111
+ ratio = None
112
+ if head:
113
+ try: ratio = jload(ents[head[0]])['customNoteContent'].get('pageRatio')
114
+ except Exception: pass
115
+ pages = []
116
+ for n, b in ents.items():
117
+ if not (n.startswith('pages/') and n.endswith('.jhinote')): continue
118
+ try: pc = jload(b)['customNotePageContent']
119
+ except Exception: continue
120
+ if pc.get('isDelete'): continue
121
+ bins = [os.path.basename(a.get('filePath', '')) for a in pc.get('attachment', []) if a.get('filePath', '').endswith('.bin') and not a.get('isDelete')]
122
+ pages.append((pc.get('pageNumber', 0), pc.get('createTime', 0), pc['id'], pc.get('pageRatio') or ratio, bins))
123
+ pages.sort()
124
+ out = []
125
+ for _, _, _, pr, bins in pages:
126
+ strokes = []
127
+ for bn in bins:
128
+ if 'files/' + bn in ents: strokes += decode_bin(ents['files/' + bn])
129
+ out.append(dict(width=1000.0, height=round(1000.0 / pr, 3) if pr else 1414.5, strokes=[[p[:2] for p in s['points']] for s in strokes]))
130
+ return out
tablet/kilonotes.py ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Kilonotes zip exports (Android) -> pen strokes of every page.
2
+
3
+ The file is only read as data. The format has no public description; these notes are reverse-engineered from one export.
4
+
5
+ Layout: /<doc-uuid>/info (JSON: title, width and height in points, pages = ordered page uuids) and
6
+ /<doc-uuid>/pages/<page-uuid>/info (JSON: draws[], where '_ct': 'ct_pen' is a handwriting stroke and 'ct_stroke' another drawn
7
+ object; points [{x, y, pressure, eventTime in ms, discontinuity}]; matrix = 3x3 row-major [a b c; d e f; 0 0 1] that maps
8
+ stroke-local points to page coordinates; strokeWidth, strokeColor, visible). Only visible ct_pen strokes are read, ordered by
9
+ the time of their first point.
10
+ """
11
+ import json
12
+ import zipfile
13
+
14
+
15
+ def read(path):
16
+ """-> the pages in notebook order: [{"width", "height", "strokes": [[[x, y], ...], ...]}], strokes in writing order."""
17
+ z = zipfile.ZipFile(path)
18
+ names = set(z.namelist())
19
+ out = []
20
+ for root in sorted(n for n in names if n.endswith('/info') and n.strip('/').count('/') == 1):
21
+ base = root[: -len('info')]
22
+ info = json.loads(z.read(root))
23
+ for page in info.get('pages', []):
24
+ name = f"{base}pages/{page if isinstance(page, str) else page.get('uuid')}/info"
25
+ if name not in names:
26
+ continue
27
+ d = json.loads(z.read(name))
28
+ paper = d.get('paper', {})
29
+ strokes = []
30
+ for q in d.get('draws', []):
31
+ points = q.get('points') or []
32
+ if not q.get('visible', True) or q.get('_ct') != 'ct_pen' or not points:
33
+ continue
34
+ a, b, c, dd, e, f = (q.get('matrix') or [1, 0, 0, 0, 1, 0, 0, 0, 1])[:6]
35
+ t0 = points[0].get('eventTime')
36
+ strokes.append((t0 is None, t0 or 0, len(strokes), [[a * p['x'] + b * p['y'] + c, dd * p['x'] + e * p['y'] + f] for p in points]))
37
+ strokes.sort(key=lambda s: s[:3])
38
+ out.append(dict(width=paper.get('widthInPoint') or info.get('width'), height=paper.get('heightInPoint') or info.get('height'), strokes=[s[3] for s in strokes]))
39
+ return out
tablet/notewise.py ADDED
@@ -0,0 +1,230 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Notewise notebooks (.notewise, iPad) -> pen strokes of every page.
2
+
3
+ The file is only read as data: zipfile in memory and the protobuf wire decoder of goodnotes.py. The format has no public
4
+ description; these notes are reverse-engineered from real notebooks.
5
+ The notes below come from a fuller converter: they also describe fields, flags and options that this reader does not use.
6
+
7
+ - zip with two kinds of entries, each the base64 text of ONE plain protobuf message (no compression inside):
8
+ note = {1: notebook id, 2: title, 4 repeated: page ids in notebook order, 7: ?},
9
+ page/<page id> = {1: page id, 2: page index (absent = 0), 4 repeated: element, 6: paper {1: {1: ?, 3: width,
10
+ 4: height} in pixels (2480 x 3508 = A4 at 300 dpi; the ink coordinates live in that space), 2: {1: '#RRGGBB',
11
+ 2: alpha} background, 7, 8: template}, 7: double sort key (1024 * (index + 1)), 9, 10: ?}.
12
+ - element = {1: id, 2: unix ms, 3: affine transform as a row-major 3x3 float matrix with fields 1..9
13
+ ([1 2 3; 4 5 6; 7 8 9], absent = identity entry; 3/6 = tx/ty of a lasso move, only translations seen),
14
+ 4: stroke | 5: highlighter | 6: shape | 15: other tool, 10: running index inside a pasted group}.
15
+ stroke = {1: pen kind (1 = pressure pen with per-point width, 2 = constant-width pen), 2: nominal width,
16
+ 3: colour {1: '#RRGGBB', 2: alpha}, 4: packed float32 x, 5: packed float32 y, 6: packed float32 rendered width
17
+ per point, 7: 1, 9: ? (12 strokes)}.
18
+ highlighter = {1: width, 2: colour (alpha 0.25), 3/4: packed x/y, 5: 1} (dropped unless --highlighter).
19
+ shape = {1: line colour, 2: width, 4: {4: straight line {1: {1: x, 2: y}, 2: {1: x, 2: y}}}, 5, 6: fill colour}:
20
+ no hand trajectory, dropped unless --shapes (then written as a 2-point pen='shape-line').
21
+ field 15 (one element, kind 12, 13 points): unknown tool, dropped.
22
+ - There is NO per-point clock. Element field 2 is the time the stroke was committed (pen-up), not pen-down:
23
+ consecutive strokes are often closer in time than the earlier one could have been drawn, never than the later
24
+ one. The lower envelope of the gaps grows by about 1.8-2 ms per point of the later stroke, so the samples are
25
+ roughly uniform in time, but that rate is an estimate, so t_ms = point index and the page is tagged
26
+ time:"ordinal"; t0_ms = the commit time (tag t0:"pen_up").
27
+ - There is no raw force either. The per-point rendered width of pen kind 1 tapers at both ends of a stroke and does
28
+ not thin with speed, i.e. it is pressure-driven. pressure = rendered width / nominal width (mostly 0.2..1.0,
29
+ rarely up to about 1.5), the widths are kept as "widths", and the page is tagged pressure:"width".
30
+ Pen kind 2 has a constant width: pressure 1.0 and the stroke is tagged pressure:"none".
31
+ - Copy-paste: the pasted elements get fresh, consecutive commit times (1 ms apart) and a transform; the copied
32
+ raw points are identical to the original's. Elements in such runs (>= 3 elements, steps <= 1 ms) are tagged
33
+ pasted:true (their t0_ms is the paste time); a stroke whose raw points repeat an earlier stroke of the notebook
34
+ is dropped unless --keep-copies (the ink would otherwise be in the data twice).
35
+ - Erasing removes elements: no deletion flag was found, every element in the file is live.
36
+ - Elements are stored in commit order; strokes are ordered by (commit time, file position).
37
+ """
38
+ import sys
39
+ import os
40
+ import json
41
+ import struct
42
+ import base64
43
+ import hashlib
44
+ import zipfile
45
+ import collections
46
+
47
+ from goodnotes import fdict, first
48
+
49
+
50
+ def f32(v, default=0.0):
51
+ return struct.unpack('<f', v)[0] if v is not None and len(v) == 4 else default
52
+
53
+
54
+ def dbl(v, default=0.0):
55
+ return struct.unpack('<d', v)[0] if v is not None and len(v) == 8 else default
56
+
57
+
58
+ def floats(b):
59
+ n = len(b) // 4
60
+ return list(struct.unpack('<%df' % n, b[:4 * n])) if n else []
61
+
62
+
63
+ def s(v):
64
+ return v.decode('utf-8', 'replace') if isinstance(v, (bytes, bytearray)) else v
65
+
66
+
67
+ def colour(b):
68
+ """{1: '#RRGGBB', 2: alpha} -> ([r, g, b], alpha 0..1)."""
69
+ d = fdict(b) if b else {}
70
+ hx = s(first(d, 1, b'#000000')).lstrip('#')
71
+ try:
72
+ rgb = [int(hx[i:i + 2], 16) for i in (0, 2, 4)]
73
+ except ValueError:
74
+ rgb = None
75
+ return rgb, round(f32(first(d, 2), 1.0), 4)
76
+
77
+
78
+ def b64msg(z, name):
79
+ return fdict(base64.b64decode(z.read(name)))
80
+
81
+
82
+ def read_note(z):
83
+ """-> (title, [page entry names] in notebook order)."""
84
+ names = set(z.namelist())
85
+ note = b64msg(z, 'note') if 'note' in names else {}
86
+ order = ['page/' + s(p) for p in note.get(4, [])]
87
+ order = [n for n in order if n in names]
88
+ # pages missing from the list go last, by their own index
89
+ rest = sorted(n for n in names if n.startswith('page/') and n not in order)
90
+ if rest:
91
+ keyed = []
92
+ for n in rest:
93
+ pg = b64msg(z, n)
94
+ keyed.append((dbl(first(pg, 7)), first(pg, 2, 0), n))
95
+ order += [n for _, _, n in sorted(keyed)]
96
+ return s(first(note, 2, b'')), order
97
+
98
+
99
+ IDENTITY = {1: 1.0, 2: 0.0, 3: 0.0, 4: 0.0, 5: 1.0, 6: 0.0}
100
+
101
+
102
+ def transform(b):
103
+ d = fdict(b) if b else {}
104
+ return tuple(f32(first(d, k), IDENTITY[k]) for k in (1, 2, 3, 4, 5, 6))
105
+
106
+
107
+ def apply_tf(tf, x, y):
108
+ a, b, c, d, e, f = tf
109
+ return a * x + b * y + c, d * x + e * y + f
110
+
111
+
112
+ def pasted_ids(els):
113
+ """element positions committed in runs of >= 3 with steps <= 1 ms (copy-paste, not handwriting)."""
114
+ ts = sorted((t, i) for i, t in enumerate(e['t'] for e in els))
115
+ out = set()
116
+ run = []
117
+ for t, i in ts:
118
+ if run and t - run[-1][0] > 1:
119
+ if len(run) >= 3:
120
+ out.update(j for _, j in run)
121
+ run = []
122
+ run.append((t, i))
123
+ if len(run) >= 3:
124
+ out.update(j for _, j in run)
125
+ return out
126
+
127
+
128
+ def read_page(pg, stats, seen, shapes=False, highlighter=False, keep_copies=False):
129
+ els = []
130
+ for k, v in enumerate(pg.get(4, [])):
131
+ e = fdict(v)
132
+ els.append(dict(k=k, e=e, t=first(e, 2, 0)))
133
+ pasted = pasted_ids(els)
134
+ out = []
135
+ for i, el in enumerate(els):
136
+ e = el['e']
137
+ tf = transform(first(e, 3))
138
+ moved = tf != (1.0, 0.0, 0.0, 0.0, 1.0, 0.0)
139
+ if 4 in e:
140
+ st = fdict(first(e, 4))
141
+ kind = first(st, 1, 0)
142
+ nominal = f32(first(st, 2), 0.0) if isinstance(first(st, 2), bytes) else float(first(st, 2, 0))
143
+ rgb, alpha = colour(first(st, 3))
144
+ xb, yb = first(st, 4, b''), first(st, 5, b'')
145
+ xs, ys, ws = floats(xb), floats(yb), floats(first(st, 6, b''))
146
+ if not xs or len(xs) != len(ys):
147
+ stats['bad_points'] += 1
148
+ continue
149
+ if len(ws) != len(xs):
150
+ ws = [nominal] * len(xs)
151
+ h = hashlib.md5(xb + b'|' + yb).hexdigest()
152
+ if h in seen:
153
+ stats['copy_of_earlier_stroke'] += 1
154
+ if not keep_copies:
155
+ continue
156
+ seen.add(h)
157
+ if kind not in (1, 2):
158
+ stats['pen_kind_%d' % kind] += 1
159
+ if 9 in st:
160
+ stats['stroke_field9'] += 1
161
+ const = kind == 2 or nominal <= 0
162
+ pts = []
163
+ for j, (x, y, w) in enumerate(zip(xs, ys, ws)):
164
+ X, Y = apply_tf(tf, x, y)
165
+ p = 1.0 if const else w / nominal
166
+ pts.append([round(X, 2), round(Y, 2), j, round(p, 4)])
167
+ sk = dict(t0_ms=el['t'], pen='pen', color=rgb, width=round(nominal, 3), points=pts,
168
+ widths=[round(w, 3) for w in ws], brush=kind, alpha=alpha)
169
+ if const:
170
+ sk['pressure'] = 'none'
171
+ stats['strokes_const_width'] += 1
172
+ stats['points_const_width'] += len(pts)
173
+ else:
174
+ stats['strokes_width_pressure'] += 1
175
+ stats['points_width_pressure'] += len(pts)
176
+ if i in pasted:
177
+ sk['pasted'] = True
178
+ stats['pasted'] += 1
179
+ if moved:
180
+ stats['moved'] += 1
181
+ out.append((el['t'], el['k'], sk))
182
+ elif 5 in e:
183
+ hl = fdict(first(e, 5))
184
+ xs, ys = floats(first(hl, 3, b'')), floats(first(hl, 4, b''))
185
+ if not highlighter or not xs:
186
+ stats['highlighter'] += 1
187
+ continue
188
+ rgb, alpha = colour(first(hl, 2))
189
+ pts = [[round(X, 2), round(Y, 2), j, 1.0] for j, (X, Y) in enumerate(apply_tf(tf, x, y) for x, y in zip(xs, ys))]
190
+ out.append((el['t'], el['k'], dict(t0_ms=el['t'], pen='highlighter', color=rgb, width=first(hl, 1),
191
+ points=pts, alpha=alpha, pressure='none')))
192
+ stats['highlighter_kept'] += 1
193
+ elif 6 in e:
194
+ sh = fdict(first(e, 6))
195
+ geo = fdict(first(sh, 4, b''))
196
+ if 4 not in geo:
197
+ stats['shape_other'] += 1
198
+ continue
199
+ ln = fdict(first(geo, 4))
200
+ a = fdict(first(ln, 1, b''))
201
+ b = fdict(first(ln, 2, b''))
202
+ if not shapes:
203
+ stats['shape_line'] += 1
204
+ continue
205
+ rgb, alpha = colour(first(sh, 1))
206
+ p1 = apply_tf(tf, f32(first(a, 1)), f32(first(a, 2)))
207
+ p2 = apply_tf(tf, f32(first(b, 1)), f32(first(b, 2)))
208
+ out.append((el['t'], el['k'], dict(t0_ms=el['t'], pen='shape-line', color=rgb, width=first(sh, 2),
209
+ points=[[round(p1[0], 2), round(p1[1], 2), 0, 1.0],
210
+ [round(p2[0], 2), round(p2[1], 2), 1, 1.0]],
211
+ alpha=alpha, pressure='none')))
212
+ stats['shape_line_kept'] += 1
213
+ else:
214
+ stats['other_element_' + '_'.join(str(k) for k in sorted(e) if k not in (1, 2, 3, 10))] += 1
215
+ out.sort(key=lambda r: (r[0], r[1]))
216
+ return [r[2] for r in out]
217
+
218
+
219
+ def read(path):
220
+ """-> the pages in notebook order: [{"width", "height", "strokes": [[[x, y], ...], ...]}], strokes in writing order."""
221
+ z = zipfile.ZipFile(path)
222
+ _, order = read_note(z)
223
+ seen = set()
224
+ out = []
225
+ for name in order:
226
+ pg = b64msg(z, name)
227
+ strokes = read_page(pg, collections.Counter(), seen)
228
+ size = fdict(first(fdict(first(pg, 6, b'')), 1, b''))
229
+ out.append(dict(width=first(size, 3), height=first(size, 4), strokes=[[p[:2] for p in k['points']] for k in strokes if k['pen'] == 'pen']))
230
+ return out
tablet/samsung/package-lock.json ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "scrawler-samsung-notes",
3
+ "lockfileVersion": 3,
4
+ "requires": true,
5
+ "packages": {
6
+ "": {
7
+ "name": "scrawler-samsung-notes",
8
+ "dependencies": {
9
+ "@twango/sdocx": "1.1.0"
10
+ }
11
+ },
12
+ "node_modules/@twango/sdocx": {
13
+ "version": "1.1.0",
14
+ "resolved": "https://registry.npmjs.org/@twango/sdocx/-/sdocx-1.1.0.tgz",
15
+ "integrity": "sha512-tkwkUrUsNR1FHaHqqSYN3qtmXHgaVgO0ey2zpSmzwZ/Bqz6ByQup2qC6t+XwnWRXrU60pbok0qF+j7r2fhsIEg==",
16
+ "license": "GPL-3.0"
17
+ }
18
+ }
19
+ }
tablet/samsung/package.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "scrawler-samsung-notes",
3
+ "private": true,
4
+ "type": "module",
5
+ "description": "Samsung Notes .sdocx -> pen strokes, through @twango/sdocx (GPL-3.0)",
6
+ "dependencies": {
7
+ "@twango/sdocx": "1.1.0"
8
+ }
9
+ }
tablet/samsung/sdocx2ink.mjs ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Samsung Notes .sdocx -> one JSON file per page with the pen strokes, for tablet/app_line.py:
2
+ // {"format": "ink-v1", "source": "sdocx", "page": n, "width", "height", "strokes": [{"points": [[x, y, t_ms, pressure], ...]}]}
3
+ // The parser is the npm package @twango/sdocx (GPL-3.0, https://github.com/twangodev/sdocx); `npm ci` in this folder installs it.
4
+ // node sdocx2ink.mjs <out_dir> <file.sdocx> [...]
5
+ import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
6
+ import { basename, join, dirname } from 'node:path';
7
+ import { fileURLToPath } from 'node:url';
8
+ import init, { parse } from '@twango/sdocx';
9
+ const here = dirname(fileURLToPath(import.meta.url));
10
+ await init(readFileSync(join(here, 'node_modules/@twango/sdocx/sdocx_bg.wasm')));
11
+ const [, , outDir, ...files] = process.argv;
12
+ if (!outDir || !files.length) { console.log('usage: node sdocx2ink.mjs <out_dir> <file.sdocx> [...]'); process.exit(2); }
13
+ mkdirSync(outDir, { recursive: true });
14
+ for (const f of files) {
15
+ let doc;
16
+ try { doc = parse(new Uint8Array(readFileSync(f))); } catch (e) { console.log('FAIL', basename(f), String(e).slice(0, 80)); continue; }
17
+ const stem = basename(f).replace(/\.sdocx$/, '');
18
+ doc.pages.forEach((pg, pi) => {
19
+ const strokes = [];
20
+ for (const ob of pg.objects || []) {
21
+ const s = ob.content?.Stroke;
22
+ if (!s || !s.points?.length) continue;
23
+ strokes.push({ pen: s.rendering?.pen_name?.split('.').pop() ?? null, points: s.points.map((p, i) => [+p.x.toFixed(2), +p.y.toFixed(2), s.timestamps?.[i] ?? i, +(s.pressures?.[i] ?? 1).toFixed(3)]) });
24
+ }
25
+ const name = `${stem}_p${String(pi).padStart(2, '0')}.json`;
26
+ writeFileSync(join(outDir, name), JSON.stringify({ format: 'ink-v1', source: 'sdocx', doc: stem, page: pi, width: pg.width, height: pg.height, strokes }));
27
+ console.log(`${name}: ${strokes.length} strokes`);
28
+ });
29
+ }
write_line.py ADDED
@@ -0,0 +1,157 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Writes a line of text in the hand of one reference line, the way synthink.dev does it.
2
+
3
+ python write_line.py "Мороз и солнце; день чудесный!" out_line --ref my_line.npy --ref-text "what the reference line says"
4
+
5
+ The first argument is the text of one line, or a .txt file with one line of text per row. Formulas are not written here: for
6
+ pages with LaTeX use write_pages.py. Needs the extra weights: python download_weights.py --site
7
+
8
+ In the folder `out` (it must not exist yet):
9
+
10
+ - line.svg, line.png (300 dpi) and line.pdf: the chosen attempt of every line, one under another
11
+ - attempts.png: all attempts, the chosen one marked with a bar on the left
12
+ - strokes.json: every attempt as strokes, with its character error rate and its closeness to the reference
13
+
14
+ The recipe is in scrawler/site_model.py. The style comes from --ref (an .npy from make_ref.py). --ref-text is what is written
15
+ in that line; it sets how wide the writer writes. Without it the recogniser reads the reference, which is right in most
16
+ letters but not all, so give the text when you know it.
17
+ """
18
+ import argparse
19
+ import json
20
+ import os
21
+ import time
22
+
23
+ import numpy as np
24
+ from PIL import Image, ImageDraw
25
+
26
+ from scrawler.common import DEFAULT_REF, DEVICE
27
+ from scrawler.render import _bezier, _sample, _smooth, _write_pdf
28
+ from scrawler.site_model import ATTEMPTS, SiteModel, to_strokes
29
+
30
+ PX = 90.0 # page px per line height at 150 dpi
31
+ GAP = 1.5 # distance between lines, in line heights
32
+ MARGIN = 0.6
33
+
34
+
35
+ def place(line, row):
36
+ """A line [n, 3] -> strokes in page px, the top of the line at `row` line heights."""
37
+ strokes = to_strokes(line)
38
+ top = min(s[:, 1].min() for s in strokes)
39
+ left = min(s[:, 0].min() for s in strokes)
40
+ return [((s - [left, top]) + [MARGIN, MARGIN + row * GAP]) * PX for s in strokes]
41
+
42
+
43
+ def page_size(placed):
44
+ points = np.concatenate([s for strokes in placed for s in strokes])
45
+ return int(points[:, 0].max() + MARGIN * PX) + 1, int(points[:, 1].max() + MARGIN * PX) + 1
46
+
47
+
48
+ def write_svg(strokes, W, H, path, pen):
49
+ width = pen * 150 / 72.0
50
+ g = lambda p: f"{p[0]:.2f},{p[1]:.2f}"
51
+ parts = []
52
+ for stroke in strokes:
53
+ s = _smooth(stroke, True)
54
+ if len(s) == 1:
55
+ parts.append(f"M{g(s[0])} L{g(s[0])}")
56
+ else:
57
+ parts.append(f"M{g(s[0])} " + " ".join(f"C{g(a)} {g(b)} {g(d)}" for _, a, b, d in _bezier(s)))
58
+ with open(path, "w") as fh:
59
+ fh.write(
60
+ f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 {W} {H}" width="{W / 150 * 25.4:.1f}mm" height="{H / 150 * 25.4:.1f}mm">'
61
+ + f'<path d="{" ".join(parts)}" fill="none" stroke="#141414" stroke-width="{width:.2f}" stroke-linecap="round" stroke-linejoin="round"/></svg>'
62
+ )
63
+
64
+
65
+ def write_png(strokes, W, H, path, pen, marks=()):
66
+ over, k = 4, 300 / 150 * 4
67
+ image = Image.new("L", (int(W * k), int(H * k)), 255)
68
+ draw = ImageDraw.Draw(image)
69
+ width = max(1, int(round(pen / 72.0 * 300 * over)))
70
+ for y0, y1 in marks:
71
+ draw.rectangle([0.15 * PX * k, y0 * k, 0.25 * PX * k, y1 * k], fill=120)
72
+ for stroke in strokes:
73
+ s = _smooth(stroke, True)
74
+ points = np.vstack([s[:1]] + [_sample(seg) for seg in _bezier(s)]) * k if len(s) > 1 else s * k
75
+ if len(points) > 1:
76
+ draw.line([tuple(p) for p in points], fill=20, width=width, joint="curve")
77
+ for p in (points[0], points[-1]):
78
+ draw.ellipse([p[0] - width / 2, p[1] - width / 2, p[0] + width / 2, p[1] + width / 2], fill=20)
79
+ image.resize((int(W * 2), int(H * 2)), Image.LANCZOS).save(path, dpi=(300, 300))
80
+
81
+
82
+ def main():
83
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
84
+ ap.add_argument("text", help="one line of text, or a .txt file with a line per row")
85
+ ap.add_argument("out", help="output folder, must not exist")
86
+ ap.add_argument("--ref", default=DEFAULT_REF, help="the style: one line of handwriting, [n, 3] .npy from make_ref.py")
87
+ ap.add_argument("--ref-text", help="what is written in the reference line; read by the recogniser if not given")
88
+ ap.add_argument("--attempts", type=int, default=ATTEMPTS, help="samples per line; the best one is kept")
89
+ ap.add_argument("--seed", type=int, default=0, help="another seed gives another writing of the same text")
90
+ ap.add_argument("--pen", type=float, default=1.1, help="pen width in points")
91
+ args = ap.parse_args()
92
+ if os.path.exists(args.out):
93
+ raise SystemExit(f"{args.out} already exists")
94
+ if os.path.isfile(args.text):
95
+ with open(args.text, encoding="utf-8") as fh:
96
+ texts = [row.strip() for row in fh if row.strip()]
97
+ else:
98
+ texts = [args.text.strip()]
99
+
100
+ model = SiteModel()
101
+ reference = np.load(args.ref).astype(np.float32)
102
+ ref_text = args.ref_text
103
+ if ref_text is None and os.path.abspath(args.ref) == os.path.abspath(DEFAULT_REF):
104
+ with open(os.path.splitext(DEFAULT_REF)[0] + ".json", encoding="utf-8") as fh:
105
+ ref_text = json.load(fh)["text"]
106
+ if ref_text is None:
107
+ ref_text = model.read(reference)
108
+ print(f'the reference reads as "{ref_text}"; pass --ref-text if that is wrong')
109
+ ref = model.reference(reference, ref_text)
110
+ print(f"device {DEVICE}, writer's spread {ref['spread']:.2f}")
111
+
112
+ report, chosen, everything, marks = [], [], [], []
113
+ for n, text in enumerate(texts):
114
+ unknown = sorted(set(text) - set(model.clean(text)))
115
+ if unknown:
116
+ print(f"line {n + 1}: the model does not know {' '.join(unknown)}, dropped")
117
+ started = time.time()
118
+ try:
119
+ result = model.write(text, ref, attempts=args.attempts, seed=args.seed + n)
120
+ except ValueError as error:
121
+ raise SystemExit(f"line {n + 1}: {error}")
122
+ pick = result["pick"]
123
+ print(
124
+ f"line {n + 1}: {time.time() - started:.0f} s, error rate {' '.join(f'{c:.2f}' for c in result['cer'])}, "
125
+ f"closeness {' '.join(f'{c:.2f}' for c in result['cos'])}, chosen {pick + 1}"
126
+ )
127
+ chosen.append(place(result["lines"][pick], n))
128
+ for k, line in enumerate(result["lines"]):
129
+ row = len(everything) + 0.4 * n
130
+ everything.append(place(line, row))
131
+ if k == pick:
132
+ marks.append(((MARGIN + row * GAP) * PX, (MARGIN + row * GAP + 1) * PX))
133
+ report.append(
134
+ {
135
+ "text": result["text"],
136
+ "pick": pick,
137
+ "cer": [round(float(c), 4) for c in result["cer"]],
138
+ "cos": [round(float(c), 4) for c in result["cos"]],
139
+ "attempts": [[[[round(float(x), 4), round(float(y), 4)] for x, y in s] for s in to_strokes(line)] for line in result["lines"]],
140
+ }
141
+ )
142
+
143
+ os.makedirs(args.out)
144
+ W, H = page_size(chosen)
145
+ strokes = [s for line in chosen for s in line]
146
+ write_svg(strokes, W, H, os.path.join(args.out, "line.svg"), args.pen)
147
+ write_png(strokes, W, H, os.path.join(args.out, "line.png"), args.pen)
148
+ _write_pdf([strokes], [], W, H, 150, os.path.join(args.out, "line.pdf"), args.pen, True)
149
+ W, H = page_size(everything)
150
+ write_png([s for line in everything for s in line], W, H, os.path.join(args.out, "attempts.png"), args.pen, marks)
151
+ with open(os.path.join(args.out, "strokes.json"), "w", encoding="utf-8") as fh:
152
+ json.dump({"reference_text": ref_text, "spread": round(ref["spread"], 4), "lines": report}, fh, ensure_ascii=False)
153
+ print("written to", args.out)
154
+
155
+
156
+ if __name__ == "__main__":
157
+ main()