The whole site recipe: write_line.py, recogniser and writer classifier; a line from a tablet (tablet/)
Browse files- NOTICE +2 -0
- README.md +48 -6
- ckpt_ctc4/ctc_fp16.pt +3 -0
- ckpt_stylecls/embed_fp16.pt +3 -0
- download_weights.py +8 -1
- scrawler/common.py +5 -1
- scrawler/site_model.py +311 -0
- tablet/README.md +73 -0
- tablet/app_line.py +184 -0
- tablet/goodnotes.py +425 -0
- tablet/huawei.py +130 -0
- tablet/kilonotes.py +39 -0
- tablet/notewise.py +230 -0
- tablet/samsung/package-lock.json +19 -0
- tablet/samsung/package.json +9 -0
- tablet/samsung/sdocx2ink.mjs +29 -0
- write_line.py +157 -0
NOTICE
CHANGED
|
@@ -7,3 +7,5 @@ The style encoder was also trained on traced HWR200 lines.
|
|
| 7 |
The tracer (trace/) was trained on data credited in trace/TRAINING_DATA.md, including BRUSH, MathWriting, Korean Handwriting Strokes, OUHD-L, Make Me a Hanzi and Google Fonts.
|
| 8 |
examples/from_scan.png shows one line of HWR200.
|
| 9 |
trace/example_open_lines.png includes a render of Korean Handwriting Strokes by Jonathan Baudanza (hangul.ink), handwriting by Jo YW, CC BY-SA 4.0, and so is under CC BY-SA 4.0; it also shows OUHD-L (CC BY 4.0) and Digital Peter (MIT).
|
|
|
|
|
|
|
|
|
| 7 |
The tracer (trace/) was trained on data credited in trace/TRAINING_DATA.md, including BRUSH, MathWriting, Korean Handwriting Strokes, OUHD-L, Make Me a Hanzi and Google Fonts.
|
| 8 |
examples/from_scan.png shows one line of HWR200.
|
| 9 |
trace/example_open_lines.png includes a render of Korean Handwriting Strokes by Jonathan Baudanza (hangul.ink), handwriting by Jo YW, CC BY-SA 4.0, and so is under CC BY-SA 4.0; it also shows OUHD-L (CC BY 4.0) and Digital Peter (MIT).
|
| 10 |
+
The recogniser (ckpt_ctc4) was also trained on traced HWR200 lines.
|
| 11 |
+
tablet/samsung/sdocx2ink.mjs uses the npm package @twango/sdocx (GPL-3.0, https://github.com/twangodev/sdocx). npm installs it separately; it is not distributed here.
|
README.md
CHANGED
|
@@ -73,15 +73,18 @@ For a good result:
|
|
| 73 |
|
| 74 |
The tracer weights have their own licence caveat, see [its training data](trace/TRAINING_DATA.md).
|
| 75 |
|
| 76 |
-
### From
|
| 77 |
|
| 78 |
-
If
|
| 79 |
|
| 80 |
```bash
|
|
|
|
| 81 |
python make_ref.py my_line.json my_line.npy
|
| 82 |
python write_pages.py notes.txt out --ref my_line.npy
|
| 83 |
```
|
| 84 |
|
|
|
|
|
|
|
| 85 |
The default style is one traced line of the open HWR200 dataset.
|
| 86 |
|
| 87 |
## Input format
|
|
@@ -109,13 +112,16 @@ The style vector comes from a line-level writer encoder. Fraction bars, radicals
|
|
| 109 |
| Path | What |
|
| 110 |
|---|---|
|
| 111 |
| `write_pages.py` | text file -> pages: wrapping, page layout, the order of the steps |
|
|
|
|
| 112 |
| `scrawler/text_model.py` | text model and duration planner |
|
|
|
|
| 113 |
| `scrawler/style_encoder.py` | one line -> style vector |
|
| 114 |
| `scrawler/glyph_model.py` | glyph model and the classifier that picks variants |
|
| 115 |
| `scrawler/formula.py` | formula layout, pen-line marks, inserting formulas into text |
|
| 116 |
| `scrawler/render.py`, `render_pages.py` | strokes -> PDF, SVG, PNG |
|
| 117 |
| `make_ref.py` | your strokes -> a style line |
|
| 118 |
| `trace/` | the tracer, one file |
|
|
|
|
| 119 |
| `tools/mathjax/layout.mjs` | LaTeX -> glyph boxes |
|
| 120 |
| `data/glyph_stroke_counts.json` | per glyph class, how many real glyphs had 1, 2, ... strokes; counts only |
|
| 121 |
|
|
@@ -124,7 +130,9 @@ The style vector comes from a line-level writer encoder. Fraction bars, radicals
|
|
| 124 |
| Path | What | Size |
|
| 125 |
|---|---|---|
|
| 126 |
| `ckpt_dl/b4styc/fp16/step003000_fp16.pt` | text model | 45 MB |
|
| 127 |
-
| `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev),
|
|
|
|
|
|
|
| 128 |
| `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
|
| 129 |
| `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
|
| 130 |
| `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
|
|
@@ -133,13 +141,44 @@ The style vector comes from a line-level writer encoder. Fraction bars, radicals
|
|
| 133 |
|
| 134 |
## The model on the site
|
| 135 |
|
| 136 |
-
[synthink.dev](https://synthink.dev) writes a line in your handwriting right in the browser: [try it there](https://synthink.dev/generate/).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
|
| 138 |
-
|
| 139 |
|
| 140 |
Measured on traced lines of 9 writers the model never saw: the cosine of a writer classifier between a generated line and the writer's own line is 0.71, against 0.47 for the release text model; to other writers it is 0.46 for both. Legibility did not change: 47.1% clean lines on the set of the table below, the same as "Text model alone". By eye the gain in likeness is modest.
|
| 141 |
|
| 142 |
-
`write_pages.py` does not use
|
| 143 |
|
| 144 |
## Evaluation
|
| 145 |
|
|
@@ -163,6 +202,7 @@ Style similarity by the writer encoder (cosine to the writer's real line, one re
|
|
| 163 |
## Known weaknesses
|
| 164 |
|
| 165 |
- The text model sometimes swaps, drops or adds a letter. There is no spell check.
|
|
|
|
| 166 |
- A one-letter formula inside a sentence can collide with the neighbouring word.
|
| 167 |
- About 2% of glyphs are the best of the variants the classifier rejected, and may be misshapen.
|
| 168 |
- Rare symbols without their own class are drawn as a similar one (for example `\simeq` as `\approx`).
|
|
@@ -173,6 +213,8 @@ Style similarity by the writer encoder (cosine to the writer's real line, one re
|
|
| 173 |
|
| 174 |
- **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
|
| 175 |
- **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
|
|
|
|
|
|
|
| 176 |
- **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
|
| 177 |
- **Tracer:** a mix of tablet notes and open sets, some with non-commercial or ShareAlike terms. See [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md), including the open question about ShareAlike.
|
| 178 |
|
|
|
|
| 73 |
|
| 74 |
The tracer weights have their own licence caveat, see [its training data](trace/TRAINING_DATA.md).
|
| 75 |
|
| 76 |
+
### From a tablet
|
| 77 |
|
| 78 |
+
If the line was written in a note app, its pen strokes are in the app's file and no tracer is needed. `tablet/app_line.py` reads GoodNotes, Notewise, Huawei Notes, Kilonotes and Samsung Notes. The recipe for each app, and for the apps that are not read, is in [tablet/README.md](tablet/README.md).
|
| 79 |
|
| 80 |
```bash
|
| 81 |
+
python tablet/app_line.py notes.goodnotes my_line.json --page 3 --line 2 --svg check.svg
|
| 82 |
python make_ref.py my_line.json my_line.npy
|
| 83 |
python write_pages.py notes.txt out --ref my_line.npy
|
| 84 |
```
|
| 85 |
|
| 86 |
+
If you have the strokes from somewhere else, write them as a JSON list of strokes, each a list of `[x, y]` points with y pointing down, in writing order: that is what `make_ref.py` takes.
|
| 87 |
+
|
| 88 |
The default style is one traced line of the open HWR200 dataset.
|
| 89 |
|
| 90 |
## Input format
|
|
|
|
| 112 |
| Path | What |
|
| 113 |
|---|---|
|
| 114 |
| `write_pages.py` | text file -> pages: wrapping, page layout, the order of the steps |
|
| 115 |
+
| `write_line.py` | one line of text in the hand of a reference line, the way the site does it |
|
| 116 |
| `scrawler/text_model.py` | text model and duration planner |
|
| 117 |
+
| `scrawler/site_model.py` | the site's text model, recogniser, writer classifier and the choice of the best attempt |
|
| 118 |
| `scrawler/style_encoder.py` | one line -> style vector |
|
| 119 |
| `scrawler/glyph_model.py` | glyph model and the classifier that picks variants |
|
| 120 |
| `scrawler/formula.py` | formula layout, pen-line marks, inserting formulas into text |
|
| 121 |
| `scrawler/render.py`, `render_pages.py` | strokes -> PDF, SVG, PNG |
|
| 122 |
| `make_ref.py` | your strokes -> a style line |
|
| 123 |
| `trace/` | the tracer, one file |
|
| 124 |
+
| `tablet/` | a line out of a note app: readers of GoodNotes, Notewise, Huawei Notes, Kilonotes and Samsung Notes files |
|
| 125 |
| `tools/mathjax/layout.mjs` | LaTeX -> glyph boxes |
|
| 126 |
| `data/glyph_stroke_counts.json` | per glyph class, how many real glyphs had 1, 2, ... strokes; counts only |
|
| 127 |
|
|
|
|
| 130 |
| Path | What | Size |
|
| 131 |
|---|---|---|
|
| 132 |
| `ckpt_dl/b4styc/fp16/step003000_fp16.pt` | text model | 45 MB |
|
| 133 |
+
| `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model of [the site](https://synthink.dev), used by `write_line.py`, see below | 51 MB |
|
| 134 |
+
| `ckpt_ctc4/ctc_fp16.pt` | recogniser, for `write_line.py` | 20 MB |
|
| 135 |
+
| `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer, for `write_line.py` | 4 MB |
|
| 136 |
| `ckpt_dl/planner7/best.pt` | duration planner | 14 MB |
|
| 137 |
| `ckpt_semb3_all/best.pt` | writer style encoder | 7 MB |
|
| 138 |
| `ckpt_glyph_dn_u2s/fp16/step050000.pt` | glyph model | 14 MB |
|
|
|
|
| 141 |
|
| 142 |
## The model on the site
|
| 143 |
|
| 144 |
+
[synthink.dev](https://synthink.dev) writes a line in your handwriting right in the browser: [try it there](https://synthink.dev/generate/). Every model the site runs is in this release, and `write_line.py` does the same on your machine: one line of your handwriting from a photo or a tablet goes in, a new line in that hand comes out.
|
| 145 |
+
|
| 146 |
+
```bash
|
| 147 |
+
python download_weights.py --site --tracer
|
| 148 |
+
pip install -r trace/requirements.txt
|
| 149 |
+
python trace/scrawler_trace.py my_line.png --out my_line.json --svg my_line.svg # photo of one line -> pen strokes
|
| 150 |
+
python make_ref.py my_line.json my_line.npy # pen strokes -> reference line
|
| 151 |
+
python write_line.py "Мороз и солнце; день чудесный!" out_line --ref my_line.npy --ref-text "what your line says"
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
For a line from a tablet, the tracer step is replaced by `tablet/app_line.py`, see [tablet/README.md](tablet/README.md).
|
| 155 |
+
|
| 156 |
+
`out_line/` gets `line.svg`, `line.png` and `line.pdf` with the chosen attempt, `attempts.png` with all four, and `strokes.json`. The first argument can also be a `.txt` file with one line of text per row. Formulas are not written here. A line takes about 9 seconds on an Apple M5 Max (MPS).
|
| 157 |
+
|
| 158 |
+
| On the site | Here | What it does |
|
| 159 |
+
|---|---|---|
|
| 160 |
+
| `den.onnx`, `refpts.onnx` | `ckpt_dl/u3ref/fp16/step060000_fp16.pt` | text model that also attends to the pen points of the reference line |
|
| 161 |
+
| `planner.onnx` | `ckpt_dl/planner7/best.pt` | pen points per character |
|
| 162 |
+
| `semb.onnx` | `ckpt_semb3_all/best.pt` | style vector of the reference line |
|
| 163 |
+
| `rec.onnx` | `ckpt_ctc4/ctc_fp16.pt` | recogniser: how legible an attempt is |
|
| 164 |
+
| `scls.onnx` | `ckpt_stylecls/embed_fp16.pt` | writer classifier without its class layer: how close an attempt is to the reference |
|
| 165 |
+
| `fix.onnx`, `tenc.onnx`, `tstep.onnx` | `trace/weights/` | ink restorer and tracer, for the photo route |
|
| 166 |
+
|
| 167 |
+
The weights are in this repository; the site runs the same weights converted to ONNX.
|
| 168 |
+
|
| 169 |
+
The recipe, in `scrawler/site_model.py`:
|
| 170 |
+
|
| 171 |
+
1. The planner gives the number of pen points for every character. These are multiplied by the writer's spread: the length of the reference line in points over the length the planner gives for the text of that line, clipped to [0.6, 1.4]. This is why `--ref-text` is asked for. Without it the recogniser reads the reference, which gets most letters right but not all.
|
| 172 |
+
2. The text model writes the line four times, with guidance 1.5 and 20 steps.
|
| 173 |
+
3. The recogniser keeps the attempts whose character error rate is within 0.02 of the lowest, and of those the writer classifier picks the one closest to the reference.
|
| 174 |
+
|
| 175 |
+
The code was checked against the ONNX files the site serves: from the same noise the pen points agree to 0.0003 of the line height and the same attempt is chosen.
|
| 176 |
|
| 177 |
+
The text model is the text model of this release trained for 60 000 more steps, with two additions: traced lines of 190 HWR200 writers in the training data, and attention from every generated point to the pen points of the reference line. The model reads the reference line itself, not only its style vector.
|
| 178 |
|
| 179 |
Measured on traced lines of 9 writers the model never saw: the cosine of a writer classifier between a generated line and the writer's own line is 0.71, against 0.47 for the release text model; to other writers it is 0.46 for both. Legibility did not change: 47.1% clean lines on the set of the table below, the same as "Text model alone". By eye the gain in likeness is modest.
|
| 180 |
|
| 181 |
+
`write_pages.py` does not use the site model: pages and formulas are written by the text model of the first row of the weights table.
|
| 182 |
|
| 183 |
## Evaluation
|
| 184 |
|
|
|
|
| 202 |
## Known weaknesses
|
| 203 |
|
| 204 |
- The text model sometimes swaps, drops or adds a letter. There is no spell check.
|
| 205 |
+
- The site model (`write_line.py`) often adds a stray stroke to the first word of a line, an underline or a strike-through: a stroke detector found one in the chosen attempt for 10 of 20 test hands.
|
| 206 |
- A one-letter formula inside a sentence can collide with the neighbouring word.
|
| 207 |
- About 2% of glyphs are the best of the variants the classifier rejected, and may be misshapen.
|
| 208 |
- Rare symbols without their own class are drawn as a similar one (for example `\simeq` as `\approx`).
|
|
|
|
| 213 |
|
| 214 |
- **Generator models:** tablet handwriting of 13 writers who agreed to this use, about 63 000 lines and 86 000 glyphs. Writer names are removed from the weights. The notes themselves are not published.
|
| 215 |
- **Text model of the site:** the same, plus traced lines of 190 writers of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
|
| 216 |
+
- **Recogniser:** the tablet lines above, the same lines rendered to images and traced back, and traced lines of HWR200.
|
| 217 |
+
- **Writer classifier:** tablet lines of 15 writers: 12 of the 13 above and three more.
|
| 218 |
- **Style encoder:** the same kind of tablet data, plus traced lines of [HWR200](https://huggingface.co/datasets/AntiplagiatCompany/HWR200) (Apache-2.0).
|
| 219 |
- **Tracer:** a mix of tablet notes and open sets, some with non-commercial or ShareAlike terms. See [trace/TRAINING_DATA.md](trace/TRAINING_DATA.md), including the open question about ShareAlike.
|
| 220 |
|
ckpt_ctc4/ctc_fp16.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:95208859abf7e3890d9dfb2f5a4b7473d74f6038107031660e035e11e9cec3b3
|
| 3 |
+
size 19781805
|
ckpt_stylecls/embed_fp16.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d601fea48bbbe724477910c368b4f4fd79ef75ea4e3708d4076683ede43c0a38
|
| 3 |
+
size 3513653
|
download_weights.py
CHANGED
|
@@ -2,6 +2,7 @@
|
|
| 2 |
|
| 3 |
python download_weights.py # the models that write pages, about 80 MB
|
| 4 |
python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
|
|
|
|
| 5 |
"""
|
| 6 |
import argparse
|
| 7 |
import os
|
|
@@ -17,14 +18,20 @@ WRITER = [
|
|
| 17 |
"ckpt_dl/glyph_cls2.pt",
|
| 18 |
]
|
| 19 |
TRACER = ["trace/weights/*"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
|
| 22 |
def main():
|
| 23 |
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 24 |
ap.add_argument("--tracer", action="store_true")
|
|
|
|
| 25 |
args = ap.parse_args()
|
| 26 |
root = os.path.dirname(os.path.abspath(__file__))
|
| 27 |
-
snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []), local_dir=root)
|
| 28 |
print("weights are in", root)
|
| 29 |
|
| 30 |
|
|
|
|
| 2 |
|
| 3 |
python download_weights.py # the models that write pages, about 80 MB
|
| 4 |
python download_weights.py --tracer # also the tracer that reads your line from a photo, about 60 MB more
|
| 5 |
+
python download_weights.py --site # also the models of synthink.dev for write_line.py, about 75 MB more
|
| 6 |
"""
|
| 7 |
import argparse
|
| 8 |
import os
|
|
|
|
| 18 |
"ckpt_dl/glyph_cls2.pt",
|
| 19 |
]
|
| 20 |
TRACER = ["trace/weights/*"]
|
| 21 |
+
SITE = [
|
| 22 |
+
"ckpt_dl/u3ref/fp16/step060000_fp16.pt",
|
| 23 |
+
"ckpt_ctc4/ctc_fp16.pt",
|
| 24 |
+
"ckpt_stylecls/embed_fp16.pt",
|
| 25 |
+
]
|
| 26 |
|
| 27 |
|
| 28 |
def main():
|
| 29 |
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 30 |
ap.add_argument("--tracer", action="store_true")
|
| 31 |
+
ap.add_argument("--site", action="store_true")
|
| 32 |
args = ap.parse_args()
|
| 33 |
root = os.path.dirname(os.path.abspath(__file__))
|
| 34 |
+
snapshot_download(REPO, allow_patterns=WRITER + (TRACER if args.tracer else []) + (SITE if args.site else []), local_dir=root)
|
| 35 |
print("weights are in", root)
|
| 36 |
|
| 37 |
|
scrawler/common.py
CHANGED
|
@@ -25,6 +25,9 @@ if DEVICE == "cuda":
|
|
| 25 |
|
| 26 |
CKPT_TEXT = os.path.join(ROOT, "ckpt_dl", "b4styc", "fp16", "step003000_fp16.pt")
|
| 27 |
CKPT_PLANNER = os.path.join(ROOT, "ckpt_dl", "planner7", "best.pt")
|
|
|
|
|
|
|
|
|
|
| 28 |
CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
|
| 29 |
CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
|
| 30 |
CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
|
|
@@ -35,5 +38,6 @@ MATHJAX_DIR = os.path.join(ROOT, "tools", "mathjax")
|
|
| 35 |
|
| 36 |
def load_checkpoint(path):
|
| 37 |
if not os.path.exists(path):
|
| 38 |
-
|
|
|
|
| 39 |
return torch.load(path, map_location=DEVICE, weights_only=True)
|
|
|
|
| 25 |
|
| 26 |
CKPT_TEXT = os.path.join(ROOT, "ckpt_dl", "b4styc", "fp16", "step003000_fp16.pt")
|
| 27 |
CKPT_PLANNER = os.path.join(ROOT, "ckpt_dl", "planner7", "best.pt")
|
| 28 |
+
CKPT_SITE_TEXT = os.path.join(ROOT, "ckpt_dl", "u3ref", "fp16", "step060000_fp16.pt")
|
| 29 |
+
CKPT_RECOGNISER = os.path.join(ROOT, "ckpt_ctc4", "ctc_fp16.pt")
|
| 30 |
+
CKPT_CLASSIFIER = os.path.join(ROOT, "ckpt_stylecls", "embed_fp16.pt")
|
| 31 |
CKPT_STYLE = os.path.join(ROOT, "ckpt_semb3_all", "best.pt")
|
| 32 |
CKPT_GLYPH = os.path.join(ROOT, "ckpt_glyph_dn_u2s", "fp16", "step050000.pt")
|
| 33 |
CKPT_GLYPH_CLASSIFIER = os.path.join(ROOT, "ckpt_dl", "glyph_cls2.pt")
|
|
|
|
| 38 |
|
| 39 |
def load_checkpoint(path):
|
| 40 |
if not os.path.exists(path):
|
| 41 |
+
hint = " --site" if path in (CKPT_SITE_TEXT, CKPT_RECOGNISER, CKPT_CLASSIFIER) else ""
|
| 42 |
+
raise SystemExit(f"{path} not found. Run: python download_weights.py{hint}")
|
| 43 |
return torch.load(path, map_location=DEVICE, weights_only=True)
|
scrawler/site_model.py
ADDED
|
@@ -0,0 +1,311 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The text model of synthink.dev and its recipe: one reference line and a text -> the text as a line in that hand.
|
| 2 |
+
|
| 3 |
+
Three things differ from scrawler/text_model.py:
|
| 4 |
+
|
| 5 |
+
- The denoiser also attends to the pen points of the reference line (two cross-attention layers), so it reads the line itself
|
| 6 |
+
and not only its style vector.
|
| 7 |
+
- The planner's point counts are multiplied by the writer's spread: the length of the reference in points over the length the
|
| 8 |
+
planner gives for the reference text, clipped to [0.6, 1.4]. This needs the text of the reference line.
|
| 9 |
+
- Every line is sampled four times. A recogniser keeps the attempts that read best (character error rate within 0.02 of the
|
| 10 |
+
lowest), and of those a writer classifier picks the one closest to the reference.
|
| 11 |
+
|
| 12 |
+
Lines are arrays [n, 3] of (dx, dy, pen_up) with line height 1, as everywhere in this package.
|
| 13 |
+
"""
|
| 14 |
+
import numpy as np
|
| 15 |
+
import torch
|
| 16 |
+
import torch.nn as nn
|
| 17 |
+
import torch.nn.functional as F
|
| 18 |
+
|
| 19 |
+
from .common import AMP, CKPT_CLASSIFIER, CKPT_PLANNER, CKPT_RECOGNISER, CKPT_SITE_TEXT, DEVICE, load_checkpoint
|
| 20 |
+
from .style_encoder import SCALE, features, load_style_net, resample, style_vector
|
| 21 |
+
from .text_model import ADVANCE, ALPHA_BAR, GUIDANCE, MAX_POINTS, NOISE_STEPS, SAMPLE_STEPS, SCALE_X, SCALE_Y
|
| 22 |
+
from .text_model import Denoiser, Planner, _plan, time_embedding
|
| 23 |
+
|
| 24 |
+
ATTEMPTS = 4
|
| 25 |
+
SPREAD_MIN, SPREAD_MAX = 0.6, 1.4
|
| 26 |
+
CER_SLACK = 0.02
|
| 27 |
+
REF_STEP = 0.05
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def to_strokes(line):
|
| 31 |
+
"""A line [n, 3] -> a list of strokes, each an array [k, 2] of absolute points."""
|
| 32 |
+
line = np.asarray(line, np.float32)
|
| 33 |
+
points = np.cumsum(line[:, :2], 0)
|
| 34 |
+
strokes, start = [], 0
|
| 35 |
+
for i in np.flatnonzero(line[:, 2] > 0.5):
|
| 36 |
+
strokes.append(points[start : i + 1])
|
| 37 |
+
start = i + 1
|
| 38 |
+
if start < len(points):
|
| 39 |
+
strokes.append(points[start:])
|
| 40 |
+
return strokes
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def to_reference(strokes):
|
| 44 |
+
"""Strokes in any units -> a reference line: height 1, a point every 0.05 along the pen path. The same as make_ref.py."""
|
| 45 |
+
strokes = [np.asarray(s, np.float64)[:, :2] for s in strokes if len(s)]
|
| 46 |
+
everything = np.concatenate(strokes)
|
| 47 |
+
origin = everything.min(0)
|
| 48 |
+
height = float(everything[:, 1].max() - origin[1])
|
| 49 |
+
out = []
|
| 50 |
+
for p in strokes:
|
| 51 |
+
p = (p - origin) / height
|
| 52 |
+
if len(p) == 1:
|
| 53 |
+
out.append(np.array([[p[0, 0], p[0, 1], 1.0]]))
|
| 54 |
+
continue
|
| 55 |
+
cum = np.r_[0, np.cumsum(np.hypot(*np.diff(p, axis=0).T))]
|
| 56 |
+
u = np.r_[np.arange(0, cum[-1], REF_STEP), cum[-1]] if cum[-1] > 1e-6 else np.array([0.0, cum[-1]])
|
| 57 |
+
r = np.stack([np.interp(u, cum, p[:, 0]), np.interp(u, cum, p[:, 1]), np.zeros(len(u))], 1)
|
| 58 |
+
r[-1, 2] = 1
|
| 59 |
+
out.append(r)
|
| 60 |
+
absolute = np.concatenate(out)
|
| 61 |
+
line = absolute.copy()
|
| 62 |
+
line[1:, :2] = absolute[1:, :2] - absolute[:-1, :2]
|
| 63 |
+
line[0, :2] = 0
|
| 64 |
+
return line.astype(np.float32)
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
class ReferenceEncoder(nn.Module):
|
| 68 |
+
"""A reference line -> the keys the denoiser attends to: one learned "sink" key and one key for every 4 pen points."""
|
| 69 |
+
|
| 70 |
+
def __init__(self, d=192):
|
| 71 |
+
super().__init__()
|
| 72 |
+
self.stem = nn.Sequential(
|
| 73 |
+
nn.Conv1d(4, 128, 5, padding=2),
|
| 74 |
+
nn.GELU(),
|
| 75 |
+
nn.Conv1d(128, 192, 5, stride=2, padding=2),
|
| 76 |
+
nn.GELU(),
|
| 77 |
+
nn.Conv1d(192, 256, 5, stride=2, padding=2),
|
| 78 |
+
nn.GELU(),
|
| 79 |
+
)
|
| 80 |
+
self.gru = nn.GRU(256, d, num_layers=2, batch_first=True, bidirectional=True, dropout=0.1)
|
| 81 |
+
self.rkv = nn.Sequential(nn.Linear(2 * d, 256), nn.LayerNorm(256))
|
| 82 |
+
self.sink = nn.Parameter(torch.zeros(1, 256))
|
| 83 |
+
|
| 84 |
+
def forward(self, line):
|
| 85 |
+
"""line [n, 3] -> keys [1, 1 + ceil(n / 4), 256]."""
|
| 86 |
+
y = torch.cumsum(line[:, 1], 0)
|
| 87 |
+
middle = torch.sort(y).values[(len(y) - 1) // 2] # the lower median
|
| 88 |
+
x = torch.cat([line[:, :2] * SCALE, line[:, 2:3], (y - middle)[:, None]], -1)[None]
|
| 89 |
+
h = self.stem(x.transpose(1, 2)).transpose(1, 2)
|
| 90 |
+
out, _ = self.gru(h)
|
| 91 |
+
return torch.cat([self.sink[None], self.rkv(out)], 1)
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
class SiteDenoiser(Denoiser):
|
| 95 |
+
"""The denoiser of text_model.py with cross-attention to the reference keys after the GRU and before the head."""
|
| 96 |
+
|
| 97 |
+
def __init__(self, vocab_size):
|
| 98 |
+
super().__init__(vocab_size)
|
| 99 |
+
D = self.head.in_features
|
| 100 |
+
self.enc = ReferenceEncoder()
|
| 101 |
+
self.src_emb = nn.Parameter(torch.zeros(3, 256)) # where the ink came from: 0 tablet, 1 traced scan, 2 unknown
|
| 102 |
+
self.xln = nn.ModuleList([nn.LayerNorm(D) for _ in range(2)])
|
| 103 |
+
self.xatt = nn.ModuleList([nn.MultiheadAttention(D, 4, kdim=256, vdim=256, batch_first=True) for _ in range(2)])
|
| 104 |
+
|
| 105 |
+
def forward(self, z, t, cond, style, self_cond, keys):
|
| 106 |
+
mask_ = cond["mask"]
|
| 107 |
+
mask = mask_[..., None]
|
| 108 |
+
length = z.size(1)
|
| 109 |
+
pos = torch.arange(length, device=z.device, dtype=torch.float32)[None, :, None].expand(z.size(0), -1, -1) / 600.0
|
| 110 |
+
phase = cond["progress"][..., None]
|
| 111 |
+
feats = torch.cat(
|
| 112 |
+
[z, self_cond, phase, torch.sin(2 * np.pi * phase), torch.cos(2 * np.pi * phase), torch.log1p(cond["duration"])[..., None] / 3.0, pos],
|
| 113 |
+
-1,
|
| 114 |
+
)
|
| 115 |
+
x = (self.inp(feats) + self.ec(cond["char"]) + self.ep(cond["prev"]) + self.en(cond["next"]) + self.eph(cond["phase"]) + self.sin(style)[:, None]) * mask
|
| 116 |
+
g = F.silu(self.tm(time_embedding(t)) + self.sp(style))
|
| 117 |
+
for block in self.pre:
|
| 118 |
+
x = block(x, mask, g)
|
| 119 |
+
h = self.gfilm(self.gln(x), g)
|
| 120 |
+
lengths = mask_.sum(1).long().clamp(min=1)
|
| 121 |
+
with torch.autocast("cuda", enabled=False):
|
| 122 |
+
packed = nn.utils.rnn.pack_padded_sequence(h.float(), lengths.cpu(), batch_first=True, enforce_sorted=False)
|
| 123 |
+
out, _ = self.gru(packed)
|
| 124 |
+
out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True, total_length=length)
|
| 125 |
+
x = (x + self.gout(out.to(x.dtype))) * mask
|
| 126 |
+
keys = keys.to(x.dtype)
|
| 127 |
+
x = (x + self.xatt[0](self.xln[0](x), keys, keys, need_weights=False)[0]) * mask
|
| 128 |
+
for block in self.post:
|
| 129 |
+
x = block(x, mask, g)
|
| 130 |
+
x = (x + self.xatt[1](self.xln[1](x), keys, keys, need_weights=False)[0]) * mask
|
| 131 |
+
out = self.head(self.oln(x)).float()
|
| 132 |
+
return out[..., :2] * mask, out[..., 2]
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
class Recogniser(nn.Module):
|
| 136 |
+
"""Pen points -> per-point character scores (CTC). Reads generated lines to tell how legible they are."""
|
| 137 |
+
|
| 138 |
+
def __init__(self, vocab_size, hidden=384):
|
| 139 |
+
super().__init__()
|
| 140 |
+
self.inp = nn.Sequential(nn.Linear(4, hidden), nn.ReLU(), nn.Linear(hidden, hidden))
|
| 141 |
+
self.rnn = nn.LSTM(hidden, hidden, num_layers=3, bidirectional=True, batch_first=True)
|
| 142 |
+
self.out = nn.Linear(2 * hidden, vocab_size)
|
| 143 |
+
|
| 144 |
+
def forward(self, x):
|
| 145 |
+
out, _ = self.rnn(self.inp(x))
|
| 146 |
+
return self.out(out)
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
class WriterEmbedder(nn.Module):
|
| 150 |
+
"""A writer classifier without its class layer: a line -> a vector that is close for lines of the same hand."""
|
| 151 |
+
|
| 152 |
+
def __init__(self, d=192):
|
| 153 |
+
super().__init__()
|
| 154 |
+
self.stem = ReferenceEncoder(d).stem
|
| 155 |
+
self.gru = nn.GRU(256, d, num_layers=2, batch_first=True, bidirectional=True, dropout=0.1)
|
| 156 |
+
self.head = nn.Sequential(nn.Linear(4 * d, 256), nn.GELU())
|
| 157 |
+
|
| 158 |
+
def forward(self, x):
|
| 159 |
+
out, _ = self.gru(self.stem(x.transpose(1, 2)).transpose(1, 2))
|
| 160 |
+
return self.head(torch.cat([out.mean(1), out.max(1).values], -1))
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
def _edit_rate(target, hypothesis):
|
| 164 |
+
"""Edit distance between two id lists over the length of the target."""
|
| 165 |
+
if not target and not hypothesis:
|
| 166 |
+
return 0.0
|
| 167 |
+
row = list(range(len(hypothesis) + 1))
|
| 168 |
+
for i, a in enumerate(target, 1):
|
| 169 |
+
prev, row[0] = row[0], i
|
| 170 |
+
for j, b in enumerate(hypothesis, 1):
|
| 171 |
+
prev, row[j] = row[j], min(row[j] + 1, row[j - 1] + 1, prev + (a != b))
|
| 172 |
+
return row[len(hypothesis)] / max(1, len(target))
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
class SiteModel:
|
| 176 |
+
def __init__(self):
|
| 177 |
+
state = load_checkpoint(CKPT_SITE_TEXT)
|
| 178 |
+
self.itos = state["itos"]
|
| 179 |
+
self.stoi = {s: i for i, s in enumerate(self.itos)}
|
| 180 |
+
self.null_token = len(self.itos)
|
| 181 |
+
self.net = SiteDenoiser(len(self.itos)).to(DEVICE)
|
| 182 |
+
unused = ("enc.proj.", "enc.cls.") # the classifier head the reference encoder was started from
|
| 183 |
+
self.net.load_state_dict({k: v.float() for k, v in state["e"].items() if not k.startswith(unused)})
|
| 184 |
+
self.net.eval()
|
| 185 |
+
state = load_checkpoint(CKPT_PLANNER)
|
| 186 |
+
self.planner = Planner(len(state["itos"])).to(DEVICE)
|
| 187 |
+
self.planner.load_state_dict(state["m"])
|
| 188 |
+
self.planner.eval()
|
| 189 |
+
self.style_net = load_style_net()
|
| 190 |
+
state = load_checkpoint(CKPT_RECOGNISER)
|
| 191 |
+
self.recogniser = Recogniser(len(state["itos"])).to(DEVICE)
|
| 192 |
+
self.recogniser.load_state_dict({k: v.float() for k, v in state["m"].items()})
|
| 193 |
+
self.recogniser.eval()
|
| 194 |
+
state = load_checkpoint(CKPT_CLASSIFIER)
|
| 195 |
+
self.embedder = WriterEmbedder().to(DEVICE)
|
| 196 |
+
self.embedder.load_state_dict({k: v.float() for k, v in state["m"].items()})
|
| 197 |
+
self.embedder.eval()
|
| 198 |
+
|
| 199 |
+
def clean(self, text):
|
| 200 |
+
"""Drops the characters the model does not know."""
|
| 201 |
+
return "".join(c for c in text if c in self.stoi)
|
| 202 |
+
|
| 203 |
+
@torch.no_grad()
|
| 204 |
+
def _durations(self, text, semb):
|
| 205 |
+
ids = [1] + [self.stoi[c] for c in text] + [2] # <bos> ... <eos>
|
| 206 |
+
tokens = torch.tensor([ids], device=DEVICE)
|
| 207 |
+
out = self.planner(tokens, torch.ones(1, len(ids), device=DEVICE), semb.float())
|
| 208 |
+
d = torch.expm1(out).clamp(min=1).round().long().cpu().numpy()
|
| 209 |
+
return [int(v) for v in d[0, 1:-1]]
|
| 210 |
+
|
| 211 |
+
@torch.no_grad()
|
| 212 |
+
def _read(self, lines):
|
| 213 |
+
"""Equal-length lines -> the recognised character ids of each."""
|
| 214 |
+
x = np.stack([np.stack([q[:, 0] * 20.0, q[:, 1] * 20.0, q[:, 2], np.cumsum(q[:, 1])], 1) for q in lines]).astype(np.float32)
|
| 215 |
+
best = self.recogniser(torch.from_numpy(x).to(DEVICE)).argmax(-1).cpu().numpy()
|
| 216 |
+
out = []
|
| 217 |
+
for row in best:
|
| 218 |
+
ids, prev = [], 0
|
| 219 |
+
for c in row:
|
| 220 |
+
if c != 0 and c != prev:
|
| 221 |
+
ids.append(int(c))
|
| 222 |
+
prev = c
|
| 223 |
+
out.append(ids)
|
| 224 |
+
return out
|
| 225 |
+
|
| 226 |
+
def read(self, line):
|
| 227 |
+
"""What the recogniser reads in a line. Use it to check or fill in the text of your reference."""
|
| 228 |
+
return "".join(s for s in (self.itos[i] for i in self._read([np.asarray(line, np.float32)])[0]) if len(s) == 1)
|
| 229 |
+
|
| 230 |
+
@torch.no_grad()
|
| 231 |
+
def _embed(self, line):
|
| 232 |
+
"""The unit writer-classifier vector of a line, after bringing the line to the reference form."""
|
| 233 |
+
x = features(resample(to_reference(to_strokes(line))))
|
| 234 |
+
e = self.embedder(torch.from_numpy(x[None]).to(DEVICE)).float()[0]
|
| 235 |
+
return F.normalize(e, dim=-1)
|
| 236 |
+
|
| 237 |
+
@torch.no_grad()
|
| 238 |
+
def reference(self, line, text):
|
| 239 |
+
"""A reference line and what is written in it -> everything the recipe needs from the writer."""
|
| 240 |
+
line = np.asarray(line, np.float32)[:, :3]
|
| 241 |
+
semb = style_vector(self.style_net, line)
|
| 242 |
+
text = self.clean(text)
|
| 243 |
+
spread = 1.0
|
| 244 |
+
if text:
|
| 245 |
+
spread = float(np.clip(len(line) / max(1, sum(self._durations(text, semb))), SPREAD_MIN, SPREAD_MAX))
|
| 246 |
+
return {
|
| 247 |
+
"semb": semb,
|
| 248 |
+
"style": self.net.semb_proj(semb) + self.net.src_emb[0][None], # written as tablet ink
|
| 249 |
+
"keys": self.net.enc(torch.from_numpy(line[:MAX_POINTS]).to(DEVICE)),
|
| 250 |
+
"embedding": self._embed(line),
|
| 251 |
+
"spread": spread,
|
| 252 |
+
}
|
| 253 |
+
|
| 254 |
+
@torch.no_grad()
|
| 255 |
+
def write(self, text, ref, attempts=ATTEMPTS, seed=0):
|
| 256 |
+
"""-> {"lines": the attempts, "pick": the index of the chosen one, "cer", "cos", "text": the text after cleaning}."""
|
| 257 |
+
text = self.clean(text)
|
| 258 |
+
if not text:
|
| 259 |
+
raise ValueError("no characters the model knows")
|
| 260 |
+
ids = [self.stoi[c] for c in text]
|
| 261 |
+
durations = [max(1, int(round(d * ref["spread"]))) for d in self._durations(text, ref["semb"])]
|
| 262 |
+
length = sum(durations)
|
| 263 |
+
if length > MAX_POINTS:
|
| 264 |
+
raise ValueError(f"the line is too long: {length} pen points, the model takes {MAX_POINTS}. Shorten the text.")
|
| 265 |
+
B = attempts
|
| 266 |
+
plan = _plan(ids, durations)
|
| 267 |
+
cond = {
|
| 268 |
+
k: torch.tensor(v, dtype=torch.float32 if k in ("progress", "duration") else torch.long, device=DEVICE)[None].expand(B, -1)
|
| 269 |
+
for k, v in zip(("char", "prev", "next", "phase", "progress", "duration"), plan)
|
| 270 |
+
}
|
| 271 |
+
cond["mask"] = torch.ones(B, length, device=DEVICE)
|
| 272 |
+
# classifier-free guidance: the same batch twice, the second time with the characters dropped; style and reference stay
|
| 273 |
+
dropped = {k: (torch.full_like(v, self.null_token) if k in ("char", "prev", "next") else v) for k, v in cond.items()}
|
| 274 |
+
cond2 = {k: torch.cat([cond[k], dropped[k]], 0) for k in cond}
|
| 275 |
+
style2 = ref["style"].expand(2 * B, -1)
|
| 276 |
+
keys2 = ref["keys"].expand(2 * B, -1, -1)
|
| 277 |
+
|
| 278 |
+
generator = torch.Generator().manual_seed(seed)
|
| 279 |
+
z = torch.randn(B, length, 3, generator=generator).to(DEVICE)
|
| 280 |
+
self_cond = torch.zeros_like(z)
|
| 281 |
+
steps = np.linspace(NOISE_STEPS - 1, 0, SAMPLE_STEPS).round().astype(int)
|
| 282 |
+
for k, t in enumerate(steps):
|
| 283 |
+
tt = torch.full((2 * B,), int(t), device=DEVICE, dtype=torch.long)
|
| 284 |
+
with torch.autocast("cuda", dtype=torch.bfloat16, enabled=AMP):
|
| 285 |
+
xy, pen = self.net(torch.cat([z, z], 0), tt, cond2, style2, torch.cat([self_cond, self_cond], 0), keys2)
|
| 286 |
+
xy = xy[B:] + GUIDANCE * (xy[:B] - xy[B:])
|
| 287 |
+
pen = pen[B:] + GUIDANCE * (pen[:B] - pen[B:])
|
| 288 |
+
x0 = torch.cat([xy.float(), torch.tanh(pen.float() / 2)[..., None]], -1)
|
| 289 |
+
self_cond = x0
|
| 290 |
+
if k == len(steps) - 1:
|
| 291 |
+
break
|
| 292 |
+
a, a_next = ALPHA_BAR[t], ALPHA_BAR[steps[k + 1]]
|
| 293 |
+
eps = (z - a.sqrt() * x0) / (1 - a).sqrt()
|
| 294 |
+
z = a_next.sqrt() * x0 + (1 - a_next).sqrt() * eps # DDIM
|
| 295 |
+
|
| 296 |
+
index = torch.arange(length, device=DEVICE, dtype=torch.float32)[None]
|
| 297 |
+
points = torch.stack([x0[..., 0] * SCALE_X + ADVANCE * index, x0[..., 1] * SCALE_Y], -1).cpu().numpy()
|
| 298 |
+
pen_up = (pen.float() > 0.0).float().cpu().numpy()
|
| 299 |
+
lines = []
|
| 300 |
+
for b in range(B):
|
| 301 |
+
d = np.zeros((length, 3), np.float32)
|
| 302 |
+
d[1:, :2] = np.diff(points[b], axis=0)
|
| 303 |
+
d[:, 2] = pen_up[b]
|
| 304 |
+
d[-1, 2] = 1
|
| 305 |
+
lines.append(d)
|
| 306 |
+
|
| 307 |
+
cer = np.array([_edit_rate(ids, h) for h in self._read(lines)], np.float32)
|
| 308 |
+
cos = np.array([float(self._embed(q) @ ref["embedding"]) for q in lines], np.float32)
|
| 309 |
+
legible = np.flatnonzero(cer <= cer.min() + CER_SLACK)
|
| 310 |
+
pick = int(legible[np.argmax(cos[legible])])
|
| 311 |
+
return {"lines": lines, "pick": pick, "cer": cer, "cos": cos, "text": text, "durations": durations}
|
tablet/README.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# A line from a tablet
|
| 2 |
+
|
| 3 |
+
If your handwriting already lives in a note app, the pen strokes are in its files and no tracer is needed: the reference line is taken exactly as you wrote it.
|
| 4 |
+
|
| 5 |
+
The steps are the same for every app:
|
| 6 |
+
|
| 7 |
+
1. Get the notebook out of the app **in the app's own format**, not as a PDF or a picture (those keep the look of the ink, not the pen path).
|
| 8 |
+
2. Find the page and the line, and look at the picture to check:
|
| 9 |
+
|
| 10 |
+
```bash
|
| 11 |
+
python tablet/app_line.py notes.goodnotes
|
| 12 |
+
python tablet/app_line.py notes.goodnotes my_line.json --page 3 --line 2 --svg check.svg
|
| 13 |
+
```
|
| 14 |
+
|
| 15 |
+
3. Make the reference and write:
|
| 16 |
+
|
| 17 |
+
```bash
|
| 18 |
+
python make_ref.py my_line.json my_line.npy
|
| 19 |
+
python write_line.py "Any text you like." out_line --ref my_line.npy --ref-text "what the line says"
|
| 20 |
+
python write_pages.py notes.txt out --ref my_line.npy
|
| 21 |
+
```
|
| 22 |
+
|
| 23 |
+
Pick a line of ordinary prose, 30-50 characters, without formulas, underlines or corrections. The surest way is to write one such line on a new empty page: then `--page N` alone takes the whole page and nothing has to be split.
|
| 24 |
+
|
| 25 |
+
## By app
|
| 26 |
+
|
| 27 |
+
| App | File to export | Command |
|
| 28 |
+
|---|---|---|
|
| 29 |
+
| GoodNotes 5 and 6 (iPad) | the notebook as a Goodnotes document, `.goodnotes` | `python tablet/app_line.py notes.goodnotes ...` |
|
| 30 |
+
| Notewise (iPad, Android) | the note as a Notewise file, `.notewise` | `python tablet/app_line.py notes.notewise ...` |
|
| 31 |
+
| Huawei Notes | the note as a note file, `.hinote` | `python tablet/app_line.py notes.hinote ...` |
|
| 32 |
+
| Kilonotes (Android) | the notebook in the app's own export, a `.zip` | `python tablet/app_line.py notes.zip ...` |
|
| 33 |
+
| Samsung Notes | the note as a Samsung Notes file, `.sdocx` | two steps, see below |
|
| 34 |
+
| any other app | a picture of the page | the photo route, see below |
|
| 35 |
+
|
| 36 |
+
The names of the export menus change between versions of the apps, so they are not listed here: look for "export" or "share" and choose the format named in the table.
|
| 37 |
+
|
| 38 |
+
### Samsung Notes
|
| 39 |
+
|
| 40 |
+
The `.sdocx` format is read by the npm package [@twango/sdocx](https://github.com/twangodev/sdocx) (GPL-3.0), which is installed separately and is not part of this repository. It needs Node.js.
|
| 41 |
+
|
| 42 |
+
```bash
|
| 43 |
+
(cd tablet/samsung && npm ci)
|
| 44 |
+
node tablet/samsung/sdocx2ink.mjs pages notes.sdocx # writes pages/notes_p00.json, notes_p01.json, ...
|
| 45 |
+
python tablet/app_line.py pages/notes_p00.json my_line.json --line 2 --svg check.svg
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
### Any other app
|
| 49 |
+
|
| 50 |
+
Apple Notes, Notability, OneNote, Xournal++ and the rest are not read here. Use the photo route: export the page as a picture or take a screenshot, crop it to one line and run the [tracer](../trace/README.md). Ink from a screen is clean, with no paper, shadow or ruling, which is the easy case for it.
|
| 51 |
+
|
| 52 |
+
```bash
|
| 53 |
+
python download_weights.py --tracer
|
| 54 |
+
pip install -r trace/requirements.txt
|
| 55 |
+
python trace/scrawler_trace.py my_line.png --out my_line.json --svg check.svg
|
| 56 |
+
python make_ref.py my_line.json my_line.npy
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
### Your own strokes
|
| 60 |
+
|
| 61 |
+
If you can get the strokes yourself, write them as a JSON list of strokes, each a list of `[x, y]` points, y pointing down, in writing order, in any units. That is all `make_ref.py` needs.
|
| 62 |
+
|
| 63 |
+
## What to check
|
| 64 |
+
|
| 65 |
+
- **The picture.** `--svg check.svg` draws the page with the lines numbered and the chosen one in red. The page is split into lines by where the ink is densest from top to bottom. That works on ordinary rows of text and can go wrong on crowded pages, formulas and drawings.
|
| 66 |
+
- **The order of the strokes.** The models read the line in writing order. Strokes come out in the order the app stores them, which is the writing order unless you moved or pasted ink afterwards.
|
| 67 |
+
- **Highlighter and shapes** are skipped: only pen strokes are taken.
|
| 68 |
+
|
| 69 |
+
## What this was tested on
|
| 70 |
+
|
| 71 |
+
The readers are reverse-engineered: none of these formats has a public description. Each one was run on real notebooks and read all of them: GoodNotes 5 and 6 (56 notebooks), Notewise (27), Huawei Notes (41 exports from two versions of the app, 34 of them with ink), Kilonotes (8 exports), Samsung Notes (31 notes, through @twango/sdocx). A newer version of an app may write something they do not know; if `app_line.py` finds no strokes or draws nonsense, use the photo route.
|
| 72 |
+
|
| 73 |
+
The files are only read as data. Nothing in them is executed and nothing is unpacked to disk.
|
tablet/app_line.py
ADDED
|
@@ -0,0 +1,184 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""One handwritten line out of a notebook of a tablet app -> the strokes JSON that make_ref.py takes.
|
| 2 |
+
|
| 3 |
+
python tablet/app_line.py notes.goodnotes # what is in the file: pages and the lines found
|
| 4 |
+
python tablet/app_line.py notes.goodnotes my_line.json --page 3 --line 2 --svg check.svg
|
| 5 |
+
python make_ref.py my_line.json my_line.npy
|
| 6 |
+
|
| 7 |
+
Reads GoodNotes (.goodnotes), Notewise (.notewise), Huawei Notes (.hinote), Kilonotes (.zip export) and the page JSON that
|
| 8 |
+
tablet/samsung/sdocx2ink.mjs writes from a Samsung Notes .sdocx. See tablet/README.md for how to get the file out of each app.
|
| 9 |
+
|
| 10 |
+
A page is split into lines by where the ink is densest from top to bottom. That works on ordinary rows of text and can fail on
|
| 11 |
+
crowded pages, formulas and drawings, so look at the --svg picture: the page with the lines numbered and the chosen one in red.
|
| 12 |
+
The surest way is a new page with one line on it; then no --line is needed.
|
| 13 |
+
"""
|
| 14 |
+
import argparse
|
| 15 |
+
import json
|
| 16 |
+
import os
|
| 17 |
+
import sys
|
| 18 |
+
import zipfile
|
| 19 |
+
|
| 20 |
+
import numpy as np
|
| 21 |
+
|
| 22 |
+
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def read_pages(path):
|
| 26 |
+
"""-> (app name, pages); a page = {"width", "height", "strokes": [[[x, y], ...], ...]}, y pointing down."""
|
| 27 |
+
low = path.lower()
|
| 28 |
+
if low.endswith(".json"):
|
| 29 |
+
with open(path, encoding="utf-8") as fh:
|
| 30 |
+
page = json.load(fh)
|
| 31 |
+
if not isinstance(page, dict) or "strokes" not in page:
|
| 32 |
+
raise SystemExit("this JSON is not a page written by tablet/samsung/sdocx2ink.mjs")
|
| 33 |
+
strokes = [[p[:2] for p in s["points"]] for s in page["strokes"] if s.get("points")]
|
| 34 |
+
return {"sdocx": "Samsung Notes"}.get(page.get("source"), "page JSON"), [dict(width=page.get("width"), height=page.get("height"), strokes=strokes)]
|
| 35 |
+
if low.endswith(".sdocx"):
|
| 36 |
+
raise SystemExit("Samsung Notes: first run node tablet/samsung/sdocx2ink.mjs pages notes.sdocx and give this script a page JSON")
|
| 37 |
+
if low.endswith(".goodnotes"):
|
| 38 |
+
import goodnotes
|
| 39 |
+
|
| 40 |
+
return "GoodNotes", goodnotes.read(path)
|
| 41 |
+
if low.endswith(".notewise"):
|
| 42 |
+
import notewise
|
| 43 |
+
|
| 44 |
+
return "Notewise", notewise.read(path)
|
| 45 |
+
if low.endswith(".hinote"):
|
| 46 |
+
import huawei
|
| 47 |
+
|
| 48 |
+
return "Huawei Notes", huawei.read(path)
|
| 49 |
+
if zipfile.is_zipfile(path):
|
| 50 |
+
names = zipfile.ZipFile(path).namelist()
|
| 51 |
+
if any(n.endswith("/info") and "/pages/" in n for n in names):
|
| 52 |
+
import kilonotes
|
| 53 |
+
|
| 54 |
+
return "Kilonotes", kilonotes.read(path)
|
| 55 |
+
if any(n.startswith("notes/") for n in names):
|
| 56 |
+
import goodnotes
|
| 57 |
+
|
| 58 |
+
return "GoodNotes", goodnotes.read(path)
|
| 59 |
+
if any(n.startswith("page/") for n in names):
|
| 60 |
+
import notewise
|
| 61 |
+
|
| 62 |
+
return "Notewise", notewise.read(path)
|
| 63 |
+
if any(n.endswith(".jhinote") for n in names):
|
| 64 |
+
import huawei
|
| 65 |
+
|
| 66 |
+
return "Huawei Notes", huawei.read(path)
|
| 67 |
+
raise SystemExit("unknown file: expected .goodnotes, .notewise, .hinote, a Kilonotes .zip or a page .json from samsung/sdocx2ink.mjs")
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def split_lines(strokes):
|
| 71 |
+
"""Strokes of a page -> lists of stroke indices, one per text line, top to bottom; writing order is kept inside a line."""
|
| 72 |
+
strokes = [np.asarray(s, np.float64) for s in strokes]
|
| 73 |
+
if not strokes:
|
| 74 |
+
return []
|
| 75 |
+
heights = np.array([np.ptp(s[:, 1]) for s in strokes])
|
| 76 |
+
widths = np.array([np.ptp(s[:, 0]) for s in strokes])
|
| 77 |
+
letters = heights[(heights > 0) & (np.maximum(heights, widths) > 0.2 * np.median(np.maximum(heights, widths)))]
|
| 78 |
+
h = float(np.median(letters)) if len(letters) else float(max(heights.max(), 1.0))
|
| 79 |
+
if h <= 0:
|
| 80 |
+
return [list(range(len(strokes)))]
|
| 81 |
+
# ink along y: every stroke spreads its length evenly over its points
|
| 82 |
+
ys, ws = [], []
|
| 83 |
+
for s in strokes:
|
| 84 |
+
length = float(np.hypot(*np.diff(s, axis=0).T).sum()) if len(s) > 1 else 0.0
|
| 85 |
+
ys.append(s[:, 1])
|
| 86 |
+
ws.append(np.full(len(s), (length + 0.1 * h) / len(s)))
|
| 87 |
+
ys, ws = np.concatenate(ys), np.concatenate(ws)
|
| 88 |
+
step = h / 8
|
| 89 |
+
edges = np.arange(ys.min() - 2 * h, ys.max() + 2 * h + step, step)
|
| 90 |
+
density, _ = np.histogram(ys, edges, weights=ws)
|
| 91 |
+
kernel = np.hanning(2 * 8 + 1) # about one letter height wide
|
| 92 |
+
density = np.convolve(density, kernel / kernel.sum(), mode="same")
|
| 93 |
+
floor = 0.12 * density.max()
|
| 94 |
+
peaks = [i for i in range(1, len(density) - 1) if density[i] > floor and density[i] >= density[i - 1] and density[i] > density[i + 1]]
|
| 95 |
+
merged = []
|
| 96 |
+
for i in peaks: # two maxima closer than a letter height with no real dip between them are one line
|
| 97 |
+
if merged and (i - merged[-1]) * step < 1.1 * h and density[merged[-1] : i + 1].min() > 0.6 * min(density[i], density[merged[-1]]):
|
| 98 |
+
if density[i] > density[merged[-1]]:
|
| 99 |
+
merged[-1] = i
|
| 100 |
+
else:
|
| 101 |
+
merged.append(i)
|
| 102 |
+
centres = np.array([edges[i] + step / 2 for i in merged])
|
| 103 |
+
if len(centres) == 0:
|
| 104 |
+
return [list(range(len(strokes)))]
|
| 105 |
+
lines = [[] for _ in centres]
|
| 106 |
+
for k, s in enumerate(strokes):
|
| 107 |
+
lines[int(np.argmin(np.abs(centres - np.median(s[:, 1]))))].append(k)
|
| 108 |
+
return [line for line in lines if line]
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def write_svg(path, page, lines, chosen):
|
| 112 |
+
strokes = [np.asarray(s, np.float64) for s in page["strokes"]]
|
| 113 |
+
points = np.concatenate(strokes)
|
| 114 |
+
x0, y0 = points.min(0)
|
| 115 |
+
x1, y1 = points.max(0)
|
| 116 |
+
pad = 0.04 * max(x1 - x0, y1 - y0, 1.0)
|
| 117 |
+
W, H = x1 - x0 + 2 * pad, y1 - y0 + 2 * pad
|
| 118 |
+
k = 1200.0 / W
|
| 119 |
+
size = max(10.0, 0.012 * 1200)
|
| 120 |
+
out = [f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 {H * k:.0f}" width="1200" height="{H * k:.0f}"><rect width="100%" height="100%" fill="#fff"/>']
|
| 121 |
+
for n, line in enumerate(lines):
|
| 122 |
+
colour = "#c33f2e" if n == chosen else "#141414"
|
| 123 |
+
d = " ".join("M" + " L".join(f"{(x - x0 + pad) * k:.1f},{(y - y0 + pad) * k:.1f}" for x, y in strokes[i]) for i in line)
|
| 124 |
+
out.append(f'<path d="{d}" fill="none" stroke="{colour}" stroke-width="1.3" stroke-linecap="round" stroke-linejoin="round"/>')
|
| 125 |
+
top = min(strokes[i][:, 1].min() for i in line)
|
| 126 |
+
left = min(strokes[i][:, 0].min() for i in line)
|
| 127 |
+
out.append(f'<text x="{max(2.0, (left - x0 + pad) * k - 2.2 * size):.1f}" y="{(top - y0 + pad) * k + size:.1f}" font-family="sans-serif" font-size="{size:.0f}" fill="#2a6fb0">{n + 1}</text>')
|
| 128 |
+
out.append("</svg>")
|
| 129 |
+
with open(path, "w") as fh:
|
| 130 |
+
fh.write("".join(out))
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def main():
|
| 134 |
+
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 135 |
+
ap.add_argument("notebook", help="the file exported from the app")
|
| 136 |
+
ap.add_argument("out", nargs="?", help="where to write the strokes of the line (JSON); without it the file is only listed")
|
| 137 |
+
ap.add_argument("--page", type=int, help="page number, from 1; default: the only page with ink")
|
| 138 |
+
ap.add_argument("--line", type=int, help="line number on the page, from 1; default: the whole page is the line")
|
| 139 |
+
ap.add_argument("--svg", help="also draw the page with the lines numbered")
|
| 140 |
+
args = ap.parse_args()
|
| 141 |
+
|
| 142 |
+
app, pages = read_pages(args.notebook)
|
| 143 |
+
inked = [n for n, p in enumerate(pages) if p["strokes"]]
|
| 144 |
+
if not inked:
|
| 145 |
+
raise SystemExit(f"{app}: no pen strokes found in this file")
|
| 146 |
+
if args.out is None and args.page is None:
|
| 147 |
+
print(f"{app}: {len(pages)} pages, {len(inked)} with ink")
|
| 148 |
+
for n in inked:
|
| 149 |
+
print(f" page {n + 1}: {len(pages[n]['strokes'])} strokes, {len(split_lines(pages[n]['strokes']))} lines")
|
| 150 |
+
print("pick one: --page N [--line K], and look at it with --svg check.svg")
|
| 151 |
+
return
|
| 152 |
+
if args.page is None:
|
| 153 |
+
if len(inked) > 1:
|
| 154 |
+
raise SystemExit(f"{len(inked)} pages have ink; choose one with --page (run without the output name to list them)")
|
| 155 |
+
args.page = inked[0] + 1
|
| 156 |
+
if not 1 <= args.page <= len(pages) or not pages[args.page - 1]["strokes"]:
|
| 157 |
+
raise SystemExit(f"page {args.page} has no ink")
|
| 158 |
+
page = pages[args.page - 1]
|
| 159 |
+
lines = split_lines(page["strokes"])
|
| 160 |
+
if args.line is None:
|
| 161 |
+
chosen, picked = None, list(range(len(page["strokes"])))
|
| 162 |
+
if len(lines) > 1:
|
| 163 |
+
print(f"the page seems to hold {len(lines)} lines and all of them are taken as one; choose with --line if that is wrong")
|
| 164 |
+
else:
|
| 165 |
+
if not 1 <= args.line <= len(lines):
|
| 166 |
+
raise SystemExit(f"the page has {len(lines)} lines")
|
| 167 |
+
chosen, picked = args.line - 1, lines[args.line - 1]
|
| 168 |
+
for n, line in enumerate(lines):
|
| 169 |
+
xs = np.concatenate([np.asarray(page["strokes"][i], np.float64)[:, 0] for i in line])
|
| 170 |
+
print(f" line {n + 1}: {len(line)} strokes, width {np.ptp(xs):.0f}" + (" <- taken" if n == chosen else ""))
|
| 171 |
+
if args.svg:
|
| 172 |
+
write_svg(args.svg, page, lines if chosen is not None else [picked], chosen if chosen is not None else 0)
|
| 173 |
+
print("picture:", args.svg)
|
| 174 |
+
if args.out:
|
| 175 |
+
if os.path.exists(args.out):
|
| 176 |
+
raise SystemExit(f"{args.out} already exists")
|
| 177 |
+
strokes = [[[round(float(x), 3), round(float(y), 3)] for x, y in page["strokes"][i]] for i in picked]
|
| 178 |
+
with open(args.out, "w") as fh:
|
| 179 |
+
json.dump(strokes, fh)
|
| 180 |
+
print(f"{args.out}: {len(strokes)} strokes from {app}. Next: python make_ref.py {args.out} my_line.npy")
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
if __name__ == "__main__":
|
| 184 |
+
main()
|
tablet/goodnotes.py
ADDED
|
@@ -0,0 +1,425 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""GoodNotes 5 and 6 notebooks (.goodnotes) -> pen strokes of every page.
|
| 2 |
+
|
| 3 |
+
The file is only read as data: zipfile, a hand-written protobuf wire decoder and an LZ4 block decoder. Nothing is extracted to
|
| 4 |
+
disk. The format has no public description; these notes are reverse-engineered from real notebooks and may not cover every
|
| 5 |
+
version of the app.
|
| 6 |
+
The notes below come from a fuller converter: they also describe fields, flags and options that this reader does not use.
|
| 7 |
+
|
| 8 |
+
- zip: notes/<uuid> page content, search/<uuid> recognition index, attachments/<uuid> PDF/JPEG/MP4,
|
| 9 |
+
index.events.pb document/page records, index.*.pb file lists.
|
| 10 |
+
- every file is a stream of varint-length-delimited protobuf records.
|
| 11 |
+
- index.events.pb: record field 54 = page {2: page id, 3: {1: template id}, 4: {1: fractional order key}};
|
| 12 |
+
field 2 = page template {2: id, 4: background attachment, 8: {1: w, 2: h} page size in points}.
|
| 13 |
+
notes/ and search/ file uuid = page uuid + 1 (as a 128-bit integer).
|
| 14 |
+
- notes/<uuid>: pairs of records keyed by element uuid: a header {1: uuid, 2: {1: version}, 3: 1 if deleted,
|
| 15 |
+
9: Lamport clock} and a body. Body field 7 = stroke, field 1 = placed image, field 20 = text box.
|
| 16 |
+
Stroke body: 2 = Apple "bv41" LZ4 frame holding a "tpl" record with signature
|
| 17 |
+
vuA(v)A(S(uu))A(S(uuuu))vA(f) = [2, pen width, segment types, [start point], [(cx,cy,ex,ey) quad segments],
|
| 18 |
+
1, extra floats]; 4 = RGBA floats 0..1; 5 = 1 for highlighter; 6 = {1: dx, 2: dy} translation (lasso move);
|
| 19 |
+
7 = {1,2: z-order key}; 9 = recognised shape (line/ellipse/polygon) with empty point list; 14 = 1 deleted.
|
| 20 |
+
- The quad segments are midpoint smoothing of the raw samples: end_i == (c_i + c_{i+1}) / 2 exactly, so the
|
| 21 |
+
raw (already distance-decimated) pen samples are start, c_1..c_n, e_n. Those are what we write.
|
| 22 |
+
- search/<uuid>: one protobuf message, field 6 repeated = recognition entries in reading order:
|
| 23 |
+
{1: bbox {1: origin, 2: size}, 3: top-3 alternatives, 8: line number, 11: locale,
|
| 24 |
+
7: items {3: stroke uuids of the group, 4: the word's points}}. Entries with one item are words, entries with two
|
| 25 |
+
items are separators (' ' or '\\n') between them. Words are mapped to strokes by exact point match; a cursive
|
| 26 |
+
stroke can be split between two words, so a stroke may be listed under both.
|
| 27 |
+
- GoodNotes 6 (2025+, seen in notebooks of October 2026) writes a second stroke record, signature
|
| 28 |
+
vA(v)A(u)A(u)A(v)A(v)A(u)A(u)A(u)A(u)A(v) = [2, u16 flags per segment, [x, y, w] start sample, [x, y, w]*N raw
|
| 29 |
+
samples, u16 per segment, u16 ops, [x, y]*M, [], [x, y]*K outline, [x, y, w, tx, ty]*N, u16*N]. When the first
|
| 30 |
+
segment flag has bit 4 set (flags 4, 5, ...; seen in 2 of 23 notebooks, mixed with the other kind), the start record is
|
| 31 |
+
[x, y, w, e] and the sample array is per segment [x, y, w, x, y, w, e1, e2, e3] (two samples + an extra triple that
|
| 32 |
+
is skipped; reading it as samples draws fans of lines across the page). Only the samples
|
| 33 |
+
are used: x, y and w = rendered width per point (pressure-driven, 0.6..2.2 pt, no nominal width in the record).
|
| 34 |
+
The width is kept as "widths"; pressure = w / nominal, nominal = 99th percentile of w over the notebook
|
| 35 |
+
(written as width_nominal on the page), page tagged pressure:"width". The old record keeps pressure 1.0.
|
| 36 |
+
- Otherwise there is NO per-point pressure and NO per-point or per-stroke clock time in the file: pressure is written as
|
| 37 |
+
1.0, t_ms as the point index inside the stroke, t0_ms as null, and the page is tagged time:"ordinal".
|
| 38 |
+
Strokes are ordered by the z-order key (= drawing order unless re-arranged), ties by the Lamport clock.
|
| 39 |
+
"""
|
| 40 |
+
import sys
|
| 41 |
+
import os
|
| 42 |
+
import json
|
| 43 |
+
import struct
|
| 44 |
+
import zipfile
|
| 45 |
+
import collections
|
| 46 |
+
import uuid as uuidlib
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def varint(b, i):
|
| 50 |
+
r = 0
|
| 51 |
+
s = 0
|
| 52 |
+
while True:
|
| 53 |
+
c = b[i]
|
| 54 |
+
i += 1
|
| 55 |
+
r |= (c & 0x7F) << s
|
| 56 |
+
s += 7
|
| 57 |
+
if c < 0x80:
|
| 58 |
+
return r, i
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def fields(b):
|
| 62 |
+
# generic protobuf wire decoder -> list of (field, wiretype, value)
|
| 63 |
+
i = 0
|
| 64 |
+
out = []
|
| 65 |
+
n = len(b)
|
| 66 |
+
while i < n:
|
| 67 |
+
k, i = varint(b, i)
|
| 68 |
+
f, w = k >> 3, k & 7
|
| 69 |
+
if w == 0:
|
| 70 |
+
v, i = varint(b, i)
|
| 71 |
+
elif w == 1:
|
| 72 |
+
v = b[i:i + 8]
|
| 73 |
+
i += 8
|
| 74 |
+
elif w == 2:
|
| 75 |
+
ln, i = varint(b, i)
|
| 76 |
+
v = b[i:i + ln]
|
| 77 |
+
i += ln
|
| 78 |
+
elif w == 5:
|
| 79 |
+
v = b[i:i + 4]
|
| 80 |
+
i += 4
|
| 81 |
+
else:
|
| 82 |
+
raise ValueError('wire type %d' % w)
|
| 83 |
+
if i > n:
|
| 84 |
+
raise ValueError('field overrun')
|
| 85 |
+
out.append((f, w, v))
|
| 86 |
+
return out
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def fdict(b):
|
| 90 |
+
# field -> list of values
|
| 91 |
+
d = {}
|
| 92 |
+
for f, w, v in fields(b):
|
| 93 |
+
d.setdefault(f, []).append(v)
|
| 94 |
+
return d
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def first(d, k, default=None):
|
| 98 |
+
v = d.get(k)
|
| 99 |
+
return v[0] if v else default
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def records(b):
|
| 103 |
+
i = 0
|
| 104 |
+
out = []
|
| 105 |
+
while i < len(b):
|
| 106 |
+
ln, i = varint(b, i)
|
| 107 |
+
out.append(b[i:i + ln])
|
| 108 |
+
i += ln
|
| 109 |
+
return out
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def f32(v):
|
| 113 |
+
return struct.unpack('<f', v)[0]
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def vec2(b):
|
| 117 |
+
d = fdict(b)
|
| 118 |
+
x = first(d, 1)
|
| 119 |
+
y = first(d, 2)
|
| 120 |
+
return (f32(x) if x is not None else 0.0, f32(y) if y is not None else 0.0)
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def lz4_block(src):
|
| 124 |
+
dst = bytearray()
|
| 125 |
+
i = 0
|
| 126 |
+
n = len(src)
|
| 127 |
+
while i < n:
|
| 128 |
+
tok = src[i]
|
| 129 |
+
i += 1
|
| 130 |
+
ll = tok >> 4
|
| 131 |
+
if ll == 15:
|
| 132 |
+
while True:
|
| 133 |
+
c = src[i]
|
| 134 |
+
i += 1
|
| 135 |
+
ll += c
|
| 136 |
+
if c != 255:
|
| 137 |
+
break
|
| 138 |
+
dst += src[i:i + ll]
|
| 139 |
+
i += ll
|
| 140 |
+
if i >= n:
|
| 141 |
+
break
|
| 142 |
+
off = src[i] | (src[i + 1] << 8)
|
| 143 |
+
i += 2
|
| 144 |
+
ml = tok & 15
|
| 145 |
+
if ml == 15:
|
| 146 |
+
while True:
|
| 147 |
+
c = src[i]
|
| 148 |
+
i += 1
|
| 149 |
+
ml += c
|
| 150 |
+
if c != 255:
|
| 151 |
+
break
|
| 152 |
+
ml += 4
|
| 153 |
+
st = len(dst) - off
|
| 154 |
+
if st < 0:
|
| 155 |
+
raise ValueError('lz4 offset')
|
| 156 |
+
for k in range(ml):
|
| 157 |
+
dst.append(dst[st + k])
|
| 158 |
+
return bytes(dst)
|
| 159 |
+
|
| 160 |
+
|
| 161 |
+
def bvx(b):
|
| 162 |
+
# Apple libcompression LZ4 container: blocks "bv41" <raw u32> <packed u32> data | "bv4-" <raw u32> data | "bv4$"
|
| 163 |
+
out = bytearray()
|
| 164 |
+
i = 0
|
| 165 |
+
while i < len(b):
|
| 166 |
+
m = b[i:i + 4]
|
| 167 |
+
if m == b'bv41':
|
| 168 |
+
dl, cl = struct.unpack('<II', b[i + 4:i + 12])
|
| 169 |
+
blk = lz4_block(b[i + 12:i + 12 + cl])
|
| 170 |
+
if len(blk) != dl:
|
| 171 |
+
raise ValueError('lz4 size')
|
| 172 |
+
out += blk
|
| 173 |
+
i += 12 + cl
|
| 174 |
+
elif m == b'bv4-':
|
| 175 |
+
dl, = struct.unpack('<I', b[i + 4:i + 8])
|
| 176 |
+
out += b[i + 8:i + 8 + dl]
|
| 177 |
+
i += 8 + dl
|
| 178 |
+
elif m == b'bv4$':
|
| 179 |
+
break
|
| 180 |
+
else:
|
| 181 |
+
raise ValueError('bvx magic %r' % m)
|
| 182 |
+
return bytes(out)
|
| 183 |
+
|
| 184 |
+
|
| 185 |
+
def tpl_sig(s, i=0):
|
| 186 |
+
out = []
|
| 187 |
+
while i < len(s):
|
| 188 |
+
c = s[i]
|
| 189 |
+
if c in 'vuf':
|
| 190 |
+
out.append(c)
|
| 191 |
+
i += 1
|
| 192 |
+
elif c in 'AS':
|
| 193 |
+
sub, i = tpl_sig(s, i + 2)
|
| 194 |
+
out.append((c, sub))
|
| 195 |
+
elif c == ')':
|
| 196 |
+
return out, i + 1
|
| 197 |
+
else:
|
| 198 |
+
raise ValueError('tpl char %r' % c)
|
| 199 |
+
return out, i
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
def tpl_node(node, b, o):
|
| 203 |
+
# v = u16, u = 4-byte value (float in practice), f = float, A(..) = u32 count + items, S(..) = struct
|
| 204 |
+
if isinstance(node, str):
|
| 205 |
+
if node == 'v':
|
| 206 |
+
return struct.unpack('<H', b[o:o + 2])[0], o + 2
|
| 207 |
+
if o + 4 > len(b):
|
| 208 |
+
raise ValueError('tpl short')
|
| 209 |
+
return struct.unpack('<f', b[o:o + 4])[0], o + 4
|
| 210 |
+
kind, sub = node
|
| 211 |
+
if kind == 'S':
|
| 212 |
+
vals = []
|
| 213 |
+
for s in sub:
|
| 214 |
+
v, o = tpl_node(s, b, o)
|
| 215 |
+
vals.append(v)
|
| 216 |
+
return tuple(vals), o
|
| 217 |
+
cnt = struct.unpack('<I', b[o:o + 4])[0]
|
| 218 |
+
o += 4
|
| 219 |
+
if cnt * 2 > len(b):
|
| 220 |
+
raise ValueError('tpl count')
|
| 221 |
+
item = sub[0] if len(sub) == 1 else ('S', sub)
|
| 222 |
+
vals = []
|
| 223 |
+
for _ in range(cnt):
|
| 224 |
+
v, o = tpl_node(item, b, o)
|
| 225 |
+
vals.append(v)
|
| 226 |
+
return vals, o
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
def parse_tpl(b):
|
| 230 |
+
if b[:4] != b'tpl\x00':
|
| 231 |
+
raise ValueError('not tpl')
|
| 232 |
+
e = b.index(b'\x00', 8)
|
| 233 |
+
sig = b[8:e].decode('ascii')
|
| 234 |
+
nodes, _ = tpl_sig(sig)
|
| 235 |
+
o = e + 1
|
| 236 |
+
vals = []
|
| 237 |
+
for n in nodes:
|
| 238 |
+
v, o = tpl_node(n, b, o)
|
| 239 |
+
vals.append(v)
|
| 240 |
+
return sig, vals
|
| 241 |
+
|
| 242 |
+
|
| 243 |
+
STROKE_SIG = 'vuA(v)A(S(uu))A(S(uuuu))vA(f)'
|
| 244 |
+
STROKE_SIG2 = 'vA(v)A(u)A(u)A(v)A(v)A(u)A(u)A(u)A(u)A(v)'
|
| 245 |
+
|
| 246 |
+
|
| 247 |
+
def uuid_plus(u, k):
|
| 248 |
+
return str(uuidlib.UUID(int=(uuidlib.UUID(u).int + k) % (1 << 128))).upper()
|
| 249 |
+
|
| 250 |
+
|
| 251 |
+
def read_pages(z):
|
| 252 |
+
"""-> list of dicts {page_id, notes, search, width, height, bg} in notebook order."""
|
| 253 |
+
names = set(z.namelist())
|
| 254 |
+
tmpl = {}
|
| 255 |
+
pages = []
|
| 256 |
+
if 'index.events.pb' in names:
|
| 257 |
+
for r in records(z.read('index.events.pb')):
|
| 258 |
+
fs = fields(r)
|
| 259 |
+
if len(fs) < 2 or fs[1][1] != 2:
|
| 260 |
+
continue
|
| 261 |
+
kind, body = fs[1][0], fs[1][2]
|
| 262 |
+
if kind == 2:
|
| 263 |
+
d = fdict(body)
|
| 264 |
+
tid = first(d, 2, b'').decode('utf-8', 'replace')
|
| 265 |
+
size = vec2(first(d, 8)) if first(d, 8) else None
|
| 266 |
+
tmpl[tid] = dict(size=size, bg=first(d, 4, b'').decode('utf-8', 'replace') or None)
|
| 267 |
+
elif kind == 54:
|
| 268 |
+
d = fdict(body)
|
| 269 |
+
pid = first(d, 2, b'').decode('utf-8', 'replace')
|
| 270 |
+
key = first(fdict(first(d, 4)), 1, b'') if first(d, 4) else b''
|
| 271 |
+
tid = first(fdict(first(d, 3)), 1, b'').decode('utf-8', 'replace') if first(d, 3) else None
|
| 272 |
+
pages.append((key, pid, tid))
|
| 273 |
+
# fractional-index keys sort as plain bytes
|
| 274 |
+
pages.sort()
|
| 275 |
+
out = []
|
| 276 |
+
used = set()
|
| 277 |
+
for key, pid, tid in pages:
|
| 278 |
+
nf = None
|
| 279 |
+
for k in (1, 0):
|
| 280 |
+
try:
|
| 281 |
+
cand = 'notes/' + uuid_plus(pid, k)
|
| 282 |
+
except ValueError:
|
| 283 |
+
continue
|
| 284 |
+
if cand in names:
|
| 285 |
+
nf = cand
|
| 286 |
+
break
|
| 287 |
+
if nf is None:
|
| 288 |
+
continue
|
| 289 |
+
used.add(nf)
|
| 290 |
+
t = tmpl.get(tid, {})
|
| 291 |
+
sz = t.get('size')
|
| 292 |
+
sf = 'search/' + nf[6:]
|
| 293 |
+
out.append(dict(page_id=pid, notes=nf, search=sf if sf in names else None,
|
| 294 |
+
width=sz[0] if sz else None, height=sz[1] if sz else None, bg=t.get('bg')))
|
| 295 |
+
# notes files without a page record go last, in index.notes.pb order
|
| 296 |
+
rest = []
|
| 297 |
+
if 'index.notes.pb' in names:
|
| 298 |
+
for r in records(z.read('index.notes.pb')):
|
| 299 |
+
p = first(fdict(r), 2, b'').decode('utf-8', 'replace')
|
| 300 |
+
if p in names and p not in used:
|
| 301 |
+
rest.append(p)
|
| 302 |
+
rest += sorted(n for n in names if n.startswith('notes/') and n not in used and n not in rest)
|
| 303 |
+
for nf in rest:
|
| 304 |
+
sf = 'search/' + nf[6:]
|
| 305 |
+
out.append(dict(page_id=None, notes=nf, search=sf if sf in names else None, width=None, height=None, bg=None))
|
| 306 |
+
return out
|
| 307 |
+
|
| 308 |
+
|
| 309 |
+
def read_strokes(blob, stats, shapes=False):
|
| 310 |
+
heads = {}
|
| 311 |
+
bodies = {}
|
| 312 |
+
for r in records(blob):
|
| 313 |
+
try:
|
| 314 |
+
fs = fields(r)
|
| 315 |
+
except ValueError:
|
| 316 |
+
stats['bad_record'] += 1
|
| 317 |
+
continue
|
| 318 |
+
if not fs:
|
| 319 |
+
continue
|
| 320 |
+
f0, w0, v0 = fs[0]
|
| 321 |
+
if f0 == 1 and len(fs) > 1 and w0 == 2:
|
| 322 |
+
d = fdict(r)
|
| 323 |
+
u = first(d, 1).decode('utf-8', 'replace')
|
| 324 |
+
ver = first(fdict(first(d, 2, b'')), 1, 0) if first(d, 2) else 0
|
| 325 |
+
if u not in heads or ver >= heads[u]['ver']:
|
| 326 |
+
heads[u] = dict(ver=ver, deleted=first(d, 3, 0) == 1, clock=first(d, 9, 0))
|
| 327 |
+
elif f0 == 7 and w0 == 2:
|
| 328 |
+
sd = fdict(v0)
|
| 329 |
+
u = first(sd, 1, b'').decode('utf-8', 'replace')
|
| 330 |
+
ver = first(fdict(first(sd, 15)), 1, 0) if first(sd, 15) else 0
|
| 331 |
+
if u not in bodies or ver >= bodies[u][0]:
|
| 332 |
+
bodies[u] = (ver, sd)
|
| 333 |
+
elif f0 == 1:
|
| 334 |
+
stats['image'] += 1
|
| 335 |
+
elif f0 == 20:
|
| 336 |
+
stats['textbox'] += 1
|
| 337 |
+
else:
|
| 338 |
+
stats['other_record'] += 1
|
| 339 |
+
out = []
|
| 340 |
+
for u, (ver, sd) in bodies.items():
|
| 341 |
+
h = heads.get(u, {})
|
| 342 |
+
if first(sd, 14, 0) == 1 or h.get('deleted'):
|
| 343 |
+
stats['deleted'] += 1
|
| 344 |
+
continue
|
| 345 |
+
try:
|
| 346 |
+
sig, vals = parse_tpl(bvx(first(sd, 2, b'')))
|
| 347 |
+
except (ValueError, IndexError, struct.error):
|
| 348 |
+
stats['bad_geometry'] += 1
|
| 349 |
+
continue
|
| 350 |
+
rgba = [f32(x) for x in (first(fdict(first(sd, 4)), k) for k in (1, 2, 3, 4)) if x is not None] if first(sd, 4) else []
|
| 351 |
+
color = [int(round(c * 255)) for c in rgba[:3]] if len(rgba) >= 3 else None
|
| 352 |
+
off = vec2(first(sd, 6)) if first(sd, 6) else (0.0, 0.0)
|
| 353 |
+
zk = fdict(first(sd, 7)) if first(sd, 7) else {}
|
| 354 |
+
zk = fdict(first(zk, 1)) if first(zk, 1) else {}
|
| 355 |
+
order = (first(zk, 1, 0), first(zk, 2, 0), first(zk, 3, 0), h.get('clock', 0))
|
| 356 |
+
pen = 'highlighter' if first(sd, 5, 0) == 1 else 'pen'
|
| 357 |
+
if sig == STROKE_SIG2:
|
| 358 |
+
if pen == 'highlighter':
|
| 359 |
+
stats['highlighter'] += 1
|
| 360 |
+
continue
|
| 361 |
+
raw = []
|
| 362 |
+
if len(vals[2]) >= 3:
|
| 363 |
+
raw.append(vals[2][:3])
|
| 364 |
+
v3 = vals[3]
|
| 365 |
+
flags = vals[1]
|
| 366 |
+
if flags and (flags[0] & 4):
|
| 367 |
+
# variant B: start record [x, y, w, e]; each segment = [x, y, w, x, y, w, e1, e2, e3]
|
| 368 |
+
for i in range(0, len(v3) - 8, 9):
|
| 369 |
+
raw += [v3[i:i + 3], v3[i + 3:i + 6]]
|
| 370 |
+
stats['strokes_v2b'] += 1
|
| 371 |
+
else:
|
| 372 |
+
raw += [v3[i:i + 3] for i in range(0, len(v3) - 2, 3)]
|
| 373 |
+
if not raw:
|
| 374 |
+
stats['empty'] += 1
|
| 375 |
+
continue
|
| 376 |
+
pts = [[round(x + off[0], 2), round(y + off[1], 2), i, 1.0] for i, (x, y, w) in enumerate(raw)]
|
| 377 |
+
if off != (0.0, 0.0):
|
| 378 |
+
stats['moved'] += 1
|
| 379 |
+
stats['strokes_v2'] += 1
|
| 380 |
+
out.append((order, dict(t0_ms=None, pen=pen, color=color, width=None, points=pts,
|
| 381 |
+
widths=[round(w, 4) for _, _, w in raw], pressure='width')))
|
| 382 |
+
continue
|
| 383 |
+
if sig != STROKE_SIG:
|
| 384 |
+
stats['other_geometry'] += 1
|
| 385 |
+
continue
|
| 386 |
+
segs = vals[4]
|
| 387 |
+
if not segs or not vals[3]:
|
| 388 |
+
shape = fdict(first(sd, 9)) if first(sd, 9) else {}
|
| 389 |
+
if shape and first(shape, 1) is not None:
|
| 390 |
+
# straight-line shape: {1: {1: {1:x,2:y}, 1: {1:x,2:y}}}
|
| 391 |
+
pts = [vec2(p) for p in fdict(first(shape, 1)).get(1, [])]
|
| 392 |
+
if shapes and len(pts) >= 2:
|
| 393 |
+
stats['shape_line_kept'] += 1
|
| 394 |
+
out.append((order, dict(t0_ms=None, pen='shape-line', color=color, width=vals[1],
|
| 395 |
+
points=[[round(x + off[0], 2), round(y + off[1], 2), i, 1.0] for i, (x, y) in enumerate(pts)])))
|
| 396 |
+
else:
|
| 397 |
+
stats['shape_line'] += 1
|
| 398 |
+
elif shape:
|
| 399 |
+
stats['shape_other'] += 1
|
| 400 |
+
else:
|
| 401 |
+
stats['empty'] += 1
|
| 402 |
+
continue
|
| 403 |
+
if pen == 'highlighter':
|
| 404 |
+
stats['highlighter'] += 1
|
| 405 |
+
continue
|
| 406 |
+
sx, sy = vals[3][0]
|
| 407 |
+
raw = [(sx, sy)] + [(s[0], s[1]) for s in segs] + [(segs[-1][2], segs[-1][3])]
|
| 408 |
+
pts = [[round(x + off[0], 2), round(y + off[1], 2), i, 1.0] for i, (x, y) in enumerate(raw)]
|
| 409 |
+
if off != (0.0, 0.0):
|
| 410 |
+
stats['moved'] += 1
|
| 411 |
+
if vals[6]:
|
| 412 |
+
stats['extra_floats'] += 1
|
| 413 |
+
out.append((order, dict(t0_ms=None, pen=pen, color=color, width=round(vals[1], 4), points=pts)))
|
| 414 |
+
out.sort(key=lambda t: t[0])
|
| 415 |
+
return [s for _, s in out]
|
| 416 |
+
|
| 417 |
+
|
| 418 |
+
def read(path):
|
| 419 |
+
"""-> the pages in notebook order: [{"width", "height", "strokes": [[[x, y], ...], ...]}], strokes in writing order."""
|
| 420 |
+
z = zipfile.ZipFile(path)
|
| 421 |
+
out = []
|
| 422 |
+
for pg in read_pages(z):
|
| 423 |
+
strokes = read_strokes(z.read(pg['notes']), collections.Counter())
|
| 424 |
+
out.append(dict(width=pg['width'], height=pg['height'], strokes=[[p[:2] for p in s['points']] for s in strokes if s['pen'] == 'pen']))
|
| 425 |
+
return out
|
tablet/huawei.py
ADDED
|
@@ -0,0 +1,130 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Huawei Notes exports (.hinote) -> pen strokes of every page.
|
| 2 |
+
|
| 3 |
+
The file is only read as data. The format has no public description; these notes are reverse-engineered from real notebooks.
|
| 4 |
+
A truncated zip is read through its local file headers.
|
| 5 |
+
The notes below come from a fuller converter: they also describe fields, flags and options that this reader does not use.
|
| 6 |
+
|
| 7 |
+
Layout (zip): <note-uuid>.jhinote (gzip JSON: customNoteContent.noteTitle, pageRatio = width/height), pages/<page-uuid>.jhinote
|
| 8 |
+
(gzip JSON customNotePageContent: pageNumber = order in notebook, pageRatio, attachment[] whose filePath .../hwFile/<name>.bin
|
| 9 |
+
is the page's ink), custom_md.jhinote (hashes), files/<page>_<ts>_<n>.bin (ink) + files/<page>_<ts>_<n>.jpg (rendered preview,
|
| 10 |
+
763x1080 for A4). Pages without a .bin attachment carry no ink and are skipped.
|
| 11 |
+
.bin "PENCILENGINE", all big-endian: magic[12], u32 header_len=76, u32[4] (0x10001,0,1,0), zero pad to 76. Then nested chunks,
|
| 12 |
+
each = 48-byte header [u32 hdr_len=48, u32 type, u32 version, u32 payload_len, uuid[16], u32 x4] + payload.
|
| 13 |
+
type 2 = layer (one per page): ex = [pre_len=12, n_strokes, strokes_bytes, 1]; payload = u32[3] (0, 1000, 1000) then
|
| 14 |
+
n_strokes stroke chunks, then a trailing type-3 chunk (stroke-uuid index, ignored).
|
| 15 |
+
type 0 = stroke (version 0x10000 or 0x10002, same layout): ex = [68, 48, 116, data_len]; payload = 116-byte header + data.
|
| 16 |
+
header u32 slots: 0-1 i64 t_modified (epoch ms; == t_start unless the stroke was moved/copied later), 3 f32 last point
|
| 17 |
+
index, 4 0x01000000, 5 0xffffffff for free ink (shape id otherwise), 6-7 i64 t_start (pen down, epoch ms), 8-9 i64 t_end,
|
| 18 |
+
10-16 extra params (non-zero only on tool-11 strokes), 17 u32 tool, 19-21 f32 b,g,r in 0..1 (BGR order!), 22-23 f32 (1.0),
|
| 19 |
+
24 f32 brush width (page units), rest 0.
|
| 20 |
+
tool: 1 = pen (99.8% of strokes), 11 = a second pen type (same real pressure/time, e.g. rotated red notes),
|
| 21 |
+
4 = translucent highlighter, 2 = shape tool (lines/rects/circles: every point pressure 1.0 and t 0).
|
| 22 |
+
data: u32 0|2 (2 = shape), u32 n_points, u32 rec_size=36, n_points x [u32 flag (0 down, 4/6 move), f32 index, f32 x,
|
| 23 |
+
f32 y, u32 t_ms since t_start, f32 pressure 0..1, f32 orientation (rad), f32 tilt (rad), f32 0.2], then u32 5 + f32 last
|
| 24 |
+
index (pen-up marker, no coordinates). Point 0 (flag 0) is the pen-down sample at t=0, usually at/next to point 1; kept.
|
| 25 |
+
Coordinates: page units, page width = 1000, height = 1000/pageRatio (1414.5 for A4, 1196.6 for the 0.8357 PDF pages),
|
| 26 |
+
origin top-left, y down; the preview jpg is the same page scaled by jpg_width/1000. Pressure and per-point time are real
|
| 27 |
+
(stylus, ~8-16 ms steps). Strokes that were moved/copied after writing (~1.4%) keep pressure but every point has t=0: for
|
| 28 |
+
them t_ms = point index (header t_start/t_end still real). Erased ink is not stored (no eraser records; renders of the
|
| 29 |
+
stored strokes match the previews), so every stored stroke is visible. Written by default: tools 1 and 11 with slot 5 ==
|
| 30 |
+
0xffffffff and data flag 0 (pen='pen' / 'pen11'), in file order (= drawing order; t_start decreases only at 0.04% of
|
| 31 |
+
neighbouring pairs, moved strokes), t0_ms = t_start. Shapes (tool 2, or
|
| 32 |
+
slot 5 != 0xffffffff, or data flag 2) only with --shapes as pen='shape'; highlighter (tool 4) only with --highlighter as
|
| 33 |
+
pen='highlighter'. Inserted pictures/PDF backgrounds live in the page JSON (pageElement, bkgAttachmentId), not in .bin,
|
| 34 |
+
and are never written. Colour is kept as stored: 3 pages with dark paper (background base6) are written in white ink.
|
| 35 |
+
"""
|
| 36 |
+
import sys, os, json, zipfile, gzip, struct, zlib, io, re
|
| 37 |
+
|
| 38 |
+
TOOLS = {1: 'pen', 11: 'pen11', 4: 'highlighter', 2: 'shape'}
|
| 39 |
+
U = lambda d, o: struct.unpack_from('>I', d, o)[0]
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def read_entries(path):
|
| 43 |
+
"""name -> bytes. Normal zip via zipfile (in memory); broken/truncated zip by scanning local headers."""
|
| 44 |
+
raw = open(path, 'rb').read()
|
| 45 |
+
try:
|
| 46 |
+
z = zipfile.ZipFile(io.BytesIO(raw)); return {n: z.read(n) for n in z.namelist() if not n.endswith('/')}, None
|
| 47 |
+
except zipfile.BadZipFile:
|
| 48 |
+
pass
|
| 49 |
+
out, o = {}, 0
|
| 50 |
+
while True:
|
| 51 |
+
o = raw.find(b'PK\x03\x04', o)
|
| 52 |
+
if o < 0 or o + 30 > len(raw): break
|
| 53 |
+
ver, flg, meth, _, _, crc, csz, usz, nl, xl = struct.unpack_from('<HHHHHIIIHH', raw, o + 4)
|
| 54 |
+
name = raw[o + 30:o + 30 + nl].decode('utf8', 'replace'); b = o + 30 + nl + xl
|
| 55 |
+
try:
|
| 56 |
+
if meth == 8:
|
| 57 |
+
dec = zlib.decompressobj(-15); data = dec.decompress(raw[b:b + csz] if csz and not flg & 8 else raw[b:])
|
| 58 |
+
if not dec.eof: raise ValueError('truncated')
|
| 59 |
+
end = len(raw) - len(dec.unused_data) if not (csz and not flg & 8) else b + csz
|
| 60 |
+
elif meth == 0 and csz and not flg & 8:
|
| 61 |
+
data = raw[b:b + csz]; end = b + csz
|
| 62 |
+
if len(data) < csz: raise ValueError('truncated')
|
| 63 |
+
else:
|
| 64 |
+
raise ValueError('method')
|
| 65 |
+
if crc and zlib.crc32(data) & 0xFFFFFFFF != crc: raise ValueError('crc')
|
| 66 |
+
if not name.endswith('/'): out[name] = data
|
| 67 |
+
o = max(end, o + 4)
|
| 68 |
+
except Exception:
|
| 69 |
+
o += 4
|
| 70 |
+
return out, 'recovered %d entries from truncated zip' % len(out)
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def jload(b):
|
| 74 |
+
try: b = gzip.decompress(b)
|
| 75 |
+
except OSError: pass
|
| 76 |
+
return json.loads(b.decode('utf8'))
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def decode_bin(d, shapes=False, highlighter=False):
|
| 80 |
+
"""PENCILENGINE bytes -> list of ink-v1 strokes (file = drawing order)."""
|
| 81 |
+
if d[:12] != b'PENCILENGINE': raise ValueError('bad magic')
|
| 82 |
+
o, strokes = U(d, 12), []
|
| 83 |
+
while o + 48 <= len(d):
|
| 84 |
+
hl, typ, ver, L = U(d, o), U(d, o + 4), U(d, o + 8), U(d, o + 12)
|
| 85 |
+
if typ == 2:
|
| 86 |
+
pre, ns = U(d, o + 32), U(d, o + 36); q = o + hl + pre; e = min(o + hl + L, len(d))
|
| 87 |
+
for _ in range(ns):
|
| 88 |
+
if q + 48 > e: break
|
| 89 |
+
shl, styp, SL = U(d, q), U(d, q + 4), U(d, q + 12); h = q + shl; hlen = U(d, q + 40); nq = h + SL
|
| 90 |
+
if styp != 0 or nq > e: q = nq; continue
|
| 91 |
+
q = nq
|
| 92 |
+
tmod, = struct.unpack_from('>q', d, h); tstart, = struct.unpack_from('>q', d, h + 24)
|
| 93 |
+
slot5 = U(d, h + 20); tool = U(d, h + 68)
|
| 94 |
+
b, g, r, _a1, _a2, w = struct.unpack_from('>6f', d, h + 76)
|
| 95 |
+
D = h + hlen; fmt, n, rs = U(d, D), U(d, D + 4), U(d, D + 8)
|
| 96 |
+
pen = 'shape' if (tool == 2 or slot5 != 0xFFFFFFFF or fmt != 0) else TOOLS.get(tool, 'tool%d' % tool)
|
| 97 |
+
if (pen == 'shape' and not shapes) or (pen == 'highlighter' and not highlighter): continue
|
| 98 |
+
if n == 0 or rs < 24 or D + 12 + n * rs > nq: continue
|
| 99 |
+
raw = [struct.unpack_from('>IfffIf', d, D + 12 + rs * i) for i in range(n)]
|
| 100 |
+
notime = n > 1 and all(p[4] == 0 for p in raw)
|
| 101 |
+
P = [[round(x, 3), round(y, 3), i if notime else int(t), round(p, 4)] for i, (_f, _k, x, y, t, p) in enumerate(raw)]
|
| 102 |
+
strokes.append(dict(t0_ms=tstart or tmod, pen=pen, color=[round(255 * c) for c in (r, g, b)], width=round(w, 4), points=P))
|
| 103 |
+
o += hl + L
|
| 104 |
+
return strokes
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def read(path):
|
| 108 |
+
"""-> the pages in notebook order: [{"width", "height", "strokes": [[[x, y], ...], ...]}], strokes in writing order."""
|
| 109 |
+
ents, _ = read_entries(path)
|
| 110 |
+
head = [n for n in ents if n.endswith('.jhinote') and '/' not in n and n != 'custom_md.jhinote']
|
| 111 |
+
ratio = None
|
| 112 |
+
if head:
|
| 113 |
+
try: ratio = jload(ents[head[0]])['customNoteContent'].get('pageRatio')
|
| 114 |
+
except Exception: pass
|
| 115 |
+
pages = []
|
| 116 |
+
for n, b in ents.items():
|
| 117 |
+
if not (n.startswith('pages/') and n.endswith('.jhinote')): continue
|
| 118 |
+
try: pc = jload(b)['customNotePageContent']
|
| 119 |
+
except Exception: continue
|
| 120 |
+
if pc.get('isDelete'): continue
|
| 121 |
+
bins = [os.path.basename(a.get('filePath', '')) for a in pc.get('attachment', []) if a.get('filePath', '').endswith('.bin') and not a.get('isDelete')]
|
| 122 |
+
pages.append((pc.get('pageNumber', 0), pc.get('createTime', 0), pc['id'], pc.get('pageRatio') or ratio, bins))
|
| 123 |
+
pages.sort()
|
| 124 |
+
out = []
|
| 125 |
+
for _, _, _, pr, bins in pages:
|
| 126 |
+
strokes = []
|
| 127 |
+
for bn in bins:
|
| 128 |
+
if 'files/' + bn in ents: strokes += decode_bin(ents['files/' + bn])
|
| 129 |
+
out.append(dict(width=1000.0, height=round(1000.0 / pr, 3) if pr else 1414.5, strokes=[[p[:2] for p in s['points']] for s in strokes]))
|
| 130 |
+
return out
|
tablet/kilonotes.py
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Kilonotes zip exports (Android) -> pen strokes of every page.
|
| 2 |
+
|
| 3 |
+
The file is only read as data. The format has no public description; these notes are reverse-engineered from one export.
|
| 4 |
+
|
| 5 |
+
Layout: /<doc-uuid>/info (JSON: title, width and height in points, pages = ordered page uuids) and
|
| 6 |
+
/<doc-uuid>/pages/<page-uuid>/info (JSON: draws[], where '_ct': 'ct_pen' is a handwriting stroke and 'ct_stroke' another drawn
|
| 7 |
+
object; points [{x, y, pressure, eventTime in ms, discontinuity}]; matrix = 3x3 row-major [a b c; d e f; 0 0 1] that maps
|
| 8 |
+
stroke-local points to page coordinates; strokeWidth, strokeColor, visible). Only visible ct_pen strokes are read, ordered by
|
| 9 |
+
the time of their first point.
|
| 10 |
+
"""
|
| 11 |
+
import json
|
| 12 |
+
import zipfile
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def read(path):
|
| 16 |
+
"""-> the pages in notebook order: [{"width", "height", "strokes": [[[x, y], ...], ...]}], strokes in writing order."""
|
| 17 |
+
z = zipfile.ZipFile(path)
|
| 18 |
+
names = set(z.namelist())
|
| 19 |
+
out = []
|
| 20 |
+
for root in sorted(n for n in names if n.endswith('/info') and n.strip('/').count('/') == 1):
|
| 21 |
+
base = root[: -len('info')]
|
| 22 |
+
info = json.loads(z.read(root))
|
| 23 |
+
for page in info.get('pages', []):
|
| 24 |
+
name = f"{base}pages/{page if isinstance(page, str) else page.get('uuid')}/info"
|
| 25 |
+
if name not in names:
|
| 26 |
+
continue
|
| 27 |
+
d = json.loads(z.read(name))
|
| 28 |
+
paper = d.get('paper', {})
|
| 29 |
+
strokes = []
|
| 30 |
+
for q in d.get('draws', []):
|
| 31 |
+
points = q.get('points') or []
|
| 32 |
+
if not q.get('visible', True) or q.get('_ct') != 'ct_pen' or not points:
|
| 33 |
+
continue
|
| 34 |
+
a, b, c, dd, e, f = (q.get('matrix') or [1, 0, 0, 0, 1, 0, 0, 0, 1])[:6]
|
| 35 |
+
t0 = points[0].get('eventTime')
|
| 36 |
+
strokes.append((t0 is None, t0 or 0, len(strokes), [[a * p['x'] + b * p['y'] + c, dd * p['x'] + e * p['y'] + f] for p in points]))
|
| 37 |
+
strokes.sort(key=lambda s: s[:3])
|
| 38 |
+
out.append(dict(width=paper.get('widthInPoint') or info.get('width'), height=paper.get('heightInPoint') or info.get('height'), strokes=[s[3] for s in strokes]))
|
| 39 |
+
return out
|
tablet/notewise.py
ADDED
|
@@ -0,0 +1,230 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Notewise notebooks (.notewise, iPad) -> pen strokes of every page.
|
| 2 |
+
|
| 3 |
+
The file is only read as data: zipfile in memory and the protobuf wire decoder of goodnotes.py. The format has no public
|
| 4 |
+
description; these notes are reverse-engineered from real notebooks.
|
| 5 |
+
The notes below come from a fuller converter: they also describe fields, flags and options that this reader does not use.
|
| 6 |
+
|
| 7 |
+
- zip with two kinds of entries, each the base64 text of ONE plain protobuf message (no compression inside):
|
| 8 |
+
note = {1: notebook id, 2: title, 4 repeated: page ids in notebook order, 7: ?},
|
| 9 |
+
page/<page id> = {1: page id, 2: page index (absent = 0), 4 repeated: element, 6: paper {1: {1: ?, 3: width,
|
| 10 |
+
4: height} in pixels (2480 x 3508 = A4 at 300 dpi; the ink coordinates live in that space), 2: {1: '#RRGGBB',
|
| 11 |
+
2: alpha} background, 7, 8: template}, 7: double sort key (1024 * (index + 1)), 9, 10: ?}.
|
| 12 |
+
- element = {1: id, 2: unix ms, 3: affine transform as a row-major 3x3 float matrix with fields 1..9
|
| 13 |
+
([1 2 3; 4 5 6; 7 8 9], absent = identity entry; 3/6 = tx/ty of a lasso move, only translations seen),
|
| 14 |
+
4: stroke | 5: highlighter | 6: shape | 15: other tool, 10: running index inside a pasted group}.
|
| 15 |
+
stroke = {1: pen kind (1 = pressure pen with per-point width, 2 = constant-width pen), 2: nominal width,
|
| 16 |
+
3: colour {1: '#RRGGBB', 2: alpha}, 4: packed float32 x, 5: packed float32 y, 6: packed float32 rendered width
|
| 17 |
+
per point, 7: 1, 9: ? (12 strokes)}.
|
| 18 |
+
highlighter = {1: width, 2: colour (alpha 0.25), 3/4: packed x/y, 5: 1} (dropped unless --highlighter).
|
| 19 |
+
shape = {1: line colour, 2: width, 4: {4: straight line {1: {1: x, 2: y}, 2: {1: x, 2: y}}}, 5, 6: fill colour}:
|
| 20 |
+
no hand trajectory, dropped unless --shapes (then written as a 2-point pen='shape-line').
|
| 21 |
+
field 15 (one element, kind 12, 13 points): unknown tool, dropped.
|
| 22 |
+
- There is NO per-point clock. Element field 2 is the time the stroke was committed (pen-up), not pen-down:
|
| 23 |
+
consecutive strokes are often closer in time than the earlier one could have been drawn, never than the later
|
| 24 |
+
one. The lower envelope of the gaps grows by about 1.8-2 ms per point of the later stroke, so the samples are
|
| 25 |
+
roughly uniform in time, but that rate is an estimate, so t_ms = point index and the page is tagged
|
| 26 |
+
time:"ordinal"; t0_ms = the commit time (tag t0:"pen_up").
|
| 27 |
+
- There is no raw force either. The per-point rendered width of pen kind 1 tapers at both ends of a stroke and does
|
| 28 |
+
not thin with speed, i.e. it is pressure-driven. pressure = rendered width / nominal width (mostly 0.2..1.0,
|
| 29 |
+
rarely up to about 1.5), the widths are kept as "widths", and the page is tagged pressure:"width".
|
| 30 |
+
Pen kind 2 has a constant width: pressure 1.0 and the stroke is tagged pressure:"none".
|
| 31 |
+
- Copy-paste: the pasted elements get fresh, consecutive commit times (1 ms apart) and a transform; the copied
|
| 32 |
+
raw points are identical to the original's. Elements in such runs (>= 3 elements, steps <= 1 ms) are tagged
|
| 33 |
+
pasted:true (their t0_ms is the paste time); a stroke whose raw points repeat an earlier stroke of the notebook
|
| 34 |
+
is dropped unless --keep-copies (the ink would otherwise be in the data twice).
|
| 35 |
+
- Erasing removes elements: no deletion flag was found, every element in the file is live.
|
| 36 |
+
- Elements are stored in commit order; strokes are ordered by (commit time, file position).
|
| 37 |
+
"""
|
| 38 |
+
import sys
|
| 39 |
+
import os
|
| 40 |
+
import json
|
| 41 |
+
import struct
|
| 42 |
+
import base64
|
| 43 |
+
import hashlib
|
| 44 |
+
import zipfile
|
| 45 |
+
import collections
|
| 46 |
+
|
| 47 |
+
from goodnotes import fdict, first
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def f32(v, default=0.0):
|
| 51 |
+
return struct.unpack('<f', v)[0] if v is not None and len(v) == 4 else default
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def dbl(v, default=0.0):
|
| 55 |
+
return struct.unpack('<d', v)[0] if v is not None and len(v) == 8 else default
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def floats(b):
|
| 59 |
+
n = len(b) // 4
|
| 60 |
+
return list(struct.unpack('<%df' % n, b[:4 * n])) if n else []
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def s(v):
|
| 64 |
+
return v.decode('utf-8', 'replace') if isinstance(v, (bytes, bytearray)) else v
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def colour(b):
|
| 68 |
+
"""{1: '#RRGGBB', 2: alpha} -> ([r, g, b], alpha 0..1)."""
|
| 69 |
+
d = fdict(b) if b else {}
|
| 70 |
+
hx = s(first(d, 1, b'#000000')).lstrip('#')
|
| 71 |
+
try:
|
| 72 |
+
rgb = [int(hx[i:i + 2], 16) for i in (0, 2, 4)]
|
| 73 |
+
except ValueError:
|
| 74 |
+
rgb = None
|
| 75 |
+
return rgb, round(f32(first(d, 2), 1.0), 4)
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def b64msg(z, name):
|
| 79 |
+
return fdict(base64.b64decode(z.read(name)))
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def read_note(z):
|
| 83 |
+
"""-> (title, [page entry names] in notebook order)."""
|
| 84 |
+
names = set(z.namelist())
|
| 85 |
+
note = b64msg(z, 'note') if 'note' in names else {}
|
| 86 |
+
order = ['page/' + s(p) for p in note.get(4, [])]
|
| 87 |
+
order = [n for n in order if n in names]
|
| 88 |
+
# pages missing from the list go last, by their own index
|
| 89 |
+
rest = sorted(n for n in names if n.startswith('page/') and n not in order)
|
| 90 |
+
if rest:
|
| 91 |
+
keyed = []
|
| 92 |
+
for n in rest:
|
| 93 |
+
pg = b64msg(z, n)
|
| 94 |
+
keyed.append((dbl(first(pg, 7)), first(pg, 2, 0), n))
|
| 95 |
+
order += [n for _, _, n in sorted(keyed)]
|
| 96 |
+
return s(first(note, 2, b'')), order
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
IDENTITY = {1: 1.0, 2: 0.0, 3: 0.0, 4: 0.0, 5: 1.0, 6: 0.0}
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def transform(b):
|
| 103 |
+
d = fdict(b) if b else {}
|
| 104 |
+
return tuple(f32(first(d, k), IDENTITY[k]) for k in (1, 2, 3, 4, 5, 6))
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def apply_tf(tf, x, y):
|
| 108 |
+
a, b, c, d, e, f = tf
|
| 109 |
+
return a * x + b * y + c, d * x + e * y + f
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def pasted_ids(els):
|
| 113 |
+
"""element positions committed in runs of >= 3 with steps <= 1 ms (copy-paste, not handwriting)."""
|
| 114 |
+
ts = sorted((t, i) for i, t in enumerate(e['t'] for e in els))
|
| 115 |
+
out = set()
|
| 116 |
+
run = []
|
| 117 |
+
for t, i in ts:
|
| 118 |
+
if run and t - run[-1][0] > 1:
|
| 119 |
+
if len(run) >= 3:
|
| 120 |
+
out.update(j for _, j in run)
|
| 121 |
+
run = []
|
| 122 |
+
run.append((t, i))
|
| 123 |
+
if len(run) >= 3:
|
| 124 |
+
out.update(j for _, j in run)
|
| 125 |
+
return out
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
def read_page(pg, stats, seen, shapes=False, highlighter=False, keep_copies=False):
|
| 129 |
+
els = []
|
| 130 |
+
for k, v in enumerate(pg.get(4, [])):
|
| 131 |
+
e = fdict(v)
|
| 132 |
+
els.append(dict(k=k, e=e, t=first(e, 2, 0)))
|
| 133 |
+
pasted = pasted_ids(els)
|
| 134 |
+
out = []
|
| 135 |
+
for i, el in enumerate(els):
|
| 136 |
+
e = el['e']
|
| 137 |
+
tf = transform(first(e, 3))
|
| 138 |
+
moved = tf != (1.0, 0.0, 0.0, 0.0, 1.0, 0.0)
|
| 139 |
+
if 4 in e:
|
| 140 |
+
st = fdict(first(e, 4))
|
| 141 |
+
kind = first(st, 1, 0)
|
| 142 |
+
nominal = f32(first(st, 2), 0.0) if isinstance(first(st, 2), bytes) else float(first(st, 2, 0))
|
| 143 |
+
rgb, alpha = colour(first(st, 3))
|
| 144 |
+
xb, yb = first(st, 4, b''), first(st, 5, b'')
|
| 145 |
+
xs, ys, ws = floats(xb), floats(yb), floats(first(st, 6, b''))
|
| 146 |
+
if not xs or len(xs) != len(ys):
|
| 147 |
+
stats['bad_points'] += 1
|
| 148 |
+
continue
|
| 149 |
+
if len(ws) != len(xs):
|
| 150 |
+
ws = [nominal] * len(xs)
|
| 151 |
+
h = hashlib.md5(xb + b'|' + yb).hexdigest()
|
| 152 |
+
if h in seen:
|
| 153 |
+
stats['copy_of_earlier_stroke'] += 1
|
| 154 |
+
if not keep_copies:
|
| 155 |
+
continue
|
| 156 |
+
seen.add(h)
|
| 157 |
+
if kind not in (1, 2):
|
| 158 |
+
stats['pen_kind_%d' % kind] += 1
|
| 159 |
+
if 9 in st:
|
| 160 |
+
stats['stroke_field9'] += 1
|
| 161 |
+
const = kind == 2 or nominal <= 0
|
| 162 |
+
pts = []
|
| 163 |
+
for j, (x, y, w) in enumerate(zip(xs, ys, ws)):
|
| 164 |
+
X, Y = apply_tf(tf, x, y)
|
| 165 |
+
p = 1.0 if const else w / nominal
|
| 166 |
+
pts.append([round(X, 2), round(Y, 2), j, round(p, 4)])
|
| 167 |
+
sk = dict(t0_ms=el['t'], pen='pen', color=rgb, width=round(nominal, 3), points=pts,
|
| 168 |
+
widths=[round(w, 3) for w in ws], brush=kind, alpha=alpha)
|
| 169 |
+
if const:
|
| 170 |
+
sk['pressure'] = 'none'
|
| 171 |
+
stats['strokes_const_width'] += 1
|
| 172 |
+
stats['points_const_width'] += len(pts)
|
| 173 |
+
else:
|
| 174 |
+
stats['strokes_width_pressure'] += 1
|
| 175 |
+
stats['points_width_pressure'] += len(pts)
|
| 176 |
+
if i in pasted:
|
| 177 |
+
sk['pasted'] = True
|
| 178 |
+
stats['pasted'] += 1
|
| 179 |
+
if moved:
|
| 180 |
+
stats['moved'] += 1
|
| 181 |
+
out.append((el['t'], el['k'], sk))
|
| 182 |
+
elif 5 in e:
|
| 183 |
+
hl = fdict(first(e, 5))
|
| 184 |
+
xs, ys = floats(first(hl, 3, b'')), floats(first(hl, 4, b''))
|
| 185 |
+
if not highlighter or not xs:
|
| 186 |
+
stats['highlighter'] += 1
|
| 187 |
+
continue
|
| 188 |
+
rgb, alpha = colour(first(hl, 2))
|
| 189 |
+
pts = [[round(X, 2), round(Y, 2), j, 1.0] for j, (X, Y) in enumerate(apply_tf(tf, x, y) for x, y in zip(xs, ys))]
|
| 190 |
+
out.append((el['t'], el['k'], dict(t0_ms=el['t'], pen='highlighter', color=rgb, width=first(hl, 1),
|
| 191 |
+
points=pts, alpha=alpha, pressure='none')))
|
| 192 |
+
stats['highlighter_kept'] += 1
|
| 193 |
+
elif 6 in e:
|
| 194 |
+
sh = fdict(first(e, 6))
|
| 195 |
+
geo = fdict(first(sh, 4, b''))
|
| 196 |
+
if 4 not in geo:
|
| 197 |
+
stats['shape_other'] += 1
|
| 198 |
+
continue
|
| 199 |
+
ln = fdict(first(geo, 4))
|
| 200 |
+
a = fdict(first(ln, 1, b''))
|
| 201 |
+
b = fdict(first(ln, 2, b''))
|
| 202 |
+
if not shapes:
|
| 203 |
+
stats['shape_line'] += 1
|
| 204 |
+
continue
|
| 205 |
+
rgb, alpha = colour(first(sh, 1))
|
| 206 |
+
p1 = apply_tf(tf, f32(first(a, 1)), f32(first(a, 2)))
|
| 207 |
+
p2 = apply_tf(tf, f32(first(b, 1)), f32(first(b, 2)))
|
| 208 |
+
out.append((el['t'], el['k'], dict(t0_ms=el['t'], pen='shape-line', color=rgb, width=first(sh, 2),
|
| 209 |
+
points=[[round(p1[0], 2), round(p1[1], 2), 0, 1.0],
|
| 210 |
+
[round(p2[0], 2), round(p2[1], 2), 1, 1.0]],
|
| 211 |
+
alpha=alpha, pressure='none')))
|
| 212 |
+
stats['shape_line_kept'] += 1
|
| 213 |
+
else:
|
| 214 |
+
stats['other_element_' + '_'.join(str(k) for k in sorted(e) if k not in (1, 2, 3, 10))] += 1
|
| 215 |
+
out.sort(key=lambda r: (r[0], r[1]))
|
| 216 |
+
return [r[2] for r in out]
|
| 217 |
+
|
| 218 |
+
|
| 219 |
+
def read(path):
|
| 220 |
+
"""-> the pages in notebook order: [{"width", "height", "strokes": [[[x, y], ...], ...]}], strokes in writing order."""
|
| 221 |
+
z = zipfile.ZipFile(path)
|
| 222 |
+
_, order = read_note(z)
|
| 223 |
+
seen = set()
|
| 224 |
+
out = []
|
| 225 |
+
for name in order:
|
| 226 |
+
pg = b64msg(z, name)
|
| 227 |
+
strokes = read_page(pg, collections.Counter(), seen)
|
| 228 |
+
size = fdict(first(fdict(first(pg, 6, b'')), 1, b''))
|
| 229 |
+
out.append(dict(width=first(size, 3), height=first(size, 4), strokes=[[p[:2] for p in k['points']] for k in strokes if k['pen'] == 'pen']))
|
| 230 |
+
return out
|
tablet/samsung/package-lock.json
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "scrawler-samsung-notes",
|
| 3 |
+
"lockfileVersion": 3,
|
| 4 |
+
"requires": true,
|
| 5 |
+
"packages": {
|
| 6 |
+
"": {
|
| 7 |
+
"name": "scrawler-samsung-notes",
|
| 8 |
+
"dependencies": {
|
| 9 |
+
"@twango/sdocx": "1.1.0"
|
| 10 |
+
}
|
| 11 |
+
},
|
| 12 |
+
"node_modules/@twango/sdocx": {
|
| 13 |
+
"version": "1.1.0",
|
| 14 |
+
"resolved": "https://registry.npmjs.org/@twango/sdocx/-/sdocx-1.1.0.tgz",
|
| 15 |
+
"integrity": "sha512-tkwkUrUsNR1FHaHqqSYN3qtmXHgaVgO0ey2zpSmzwZ/Bqz6ByQup2qC6t+XwnWRXrU60pbok0qF+j7r2fhsIEg==",
|
| 16 |
+
"license": "GPL-3.0"
|
| 17 |
+
}
|
| 18 |
+
}
|
| 19 |
+
}
|
tablet/samsung/package.json
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "scrawler-samsung-notes",
|
| 3 |
+
"private": true,
|
| 4 |
+
"type": "module",
|
| 5 |
+
"description": "Samsung Notes .sdocx -> pen strokes, through @twango/sdocx (GPL-3.0)",
|
| 6 |
+
"dependencies": {
|
| 7 |
+
"@twango/sdocx": "1.1.0"
|
| 8 |
+
}
|
| 9 |
+
}
|
tablet/samsung/sdocx2ink.mjs
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Samsung Notes .sdocx -> one JSON file per page with the pen strokes, for tablet/app_line.py:
|
| 2 |
+
// {"format": "ink-v1", "source": "sdocx", "page": n, "width", "height", "strokes": [{"points": [[x, y, t_ms, pressure], ...]}]}
|
| 3 |
+
// The parser is the npm package @twango/sdocx (GPL-3.0, https://github.com/twangodev/sdocx); `npm ci` in this folder installs it.
|
| 4 |
+
// node sdocx2ink.mjs <out_dir> <file.sdocx> [...]
|
| 5 |
+
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
|
| 6 |
+
import { basename, join, dirname } from 'node:path';
|
| 7 |
+
import { fileURLToPath } from 'node:url';
|
| 8 |
+
import init, { parse } from '@twango/sdocx';
|
| 9 |
+
const here = dirname(fileURLToPath(import.meta.url));
|
| 10 |
+
await init(readFileSync(join(here, 'node_modules/@twango/sdocx/sdocx_bg.wasm')));
|
| 11 |
+
const [, , outDir, ...files] = process.argv;
|
| 12 |
+
if (!outDir || !files.length) { console.log('usage: node sdocx2ink.mjs <out_dir> <file.sdocx> [...]'); process.exit(2); }
|
| 13 |
+
mkdirSync(outDir, { recursive: true });
|
| 14 |
+
for (const f of files) {
|
| 15 |
+
let doc;
|
| 16 |
+
try { doc = parse(new Uint8Array(readFileSync(f))); } catch (e) { console.log('FAIL', basename(f), String(e).slice(0, 80)); continue; }
|
| 17 |
+
const stem = basename(f).replace(/\.sdocx$/, '');
|
| 18 |
+
doc.pages.forEach((pg, pi) => {
|
| 19 |
+
const strokes = [];
|
| 20 |
+
for (const ob of pg.objects || []) {
|
| 21 |
+
const s = ob.content?.Stroke;
|
| 22 |
+
if (!s || !s.points?.length) continue;
|
| 23 |
+
strokes.push({ pen: s.rendering?.pen_name?.split('.').pop() ?? null, points: s.points.map((p, i) => [+p.x.toFixed(2), +p.y.toFixed(2), s.timestamps?.[i] ?? i, +(s.pressures?.[i] ?? 1).toFixed(3)]) });
|
| 24 |
+
}
|
| 25 |
+
const name = `${stem}_p${String(pi).padStart(2, '0')}.json`;
|
| 26 |
+
writeFileSync(join(outDir, name), JSON.stringify({ format: 'ink-v1', source: 'sdocx', doc: stem, page: pi, width: pg.width, height: pg.height, strokes }));
|
| 27 |
+
console.log(`${name}: ${strokes.length} strokes`);
|
| 28 |
+
});
|
| 29 |
+
}
|
write_line.py
ADDED
|
@@ -0,0 +1,157 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Writes a line of text in the hand of one reference line, the way synthink.dev does it.
|
| 2 |
+
|
| 3 |
+
python write_line.py "Мороз и солнце; день чудесный!" out_line --ref my_line.npy --ref-text "what the reference line says"
|
| 4 |
+
|
| 5 |
+
The first argument is the text of one line, or a .txt file with one line of text per row. Formulas are not written here: for
|
| 6 |
+
pages with LaTeX use write_pages.py. Needs the extra weights: python download_weights.py --site
|
| 7 |
+
|
| 8 |
+
In the folder `out` (it must not exist yet):
|
| 9 |
+
|
| 10 |
+
- line.svg, line.png (300 dpi) and line.pdf: the chosen attempt of every line, one under another
|
| 11 |
+
- attempts.png: all attempts, the chosen one marked with a bar on the left
|
| 12 |
+
- strokes.json: every attempt as strokes, with its character error rate and its closeness to the reference
|
| 13 |
+
|
| 14 |
+
The recipe is in scrawler/site_model.py. The style comes from --ref (an .npy from make_ref.py). --ref-text is what is written
|
| 15 |
+
in that line; it sets how wide the writer writes. Without it the recogniser reads the reference, which is right in most
|
| 16 |
+
letters but not all, so give the text when you know it.
|
| 17 |
+
"""
|
| 18 |
+
import argparse
|
| 19 |
+
import json
|
| 20 |
+
import os
|
| 21 |
+
import time
|
| 22 |
+
|
| 23 |
+
import numpy as np
|
| 24 |
+
from PIL import Image, ImageDraw
|
| 25 |
+
|
| 26 |
+
from scrawler.common import DEFAULT_REF, DEVICE
|
| 27 |
+
from scrawler.render import _bezier, _sample, _smooth, _write_pdf
|
| 28 |
+
from scrawler.site_model import ATTEMPTS, SiteModel, to_strokes
|
| 29 |
+
|
| 30 |
+
PX = 90.0 # page px per line height at 150 dpi
|
| 31 |
+
GAP = 1.5 # distance between lines, in line heights
|
| 32 |
+
MARGIN = 0.6
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def place(line, row):
|
| 36 |
+
"""A line [n, 3] -> strokes in page px, the top of the line at `row` line heights."""
|
| 37 |
+
strokes = to_strokes(line)
|
| 38 |
+
top = min(s[:, 1].min() for s in strokes)
|
| 39 |
+
left = min(s[:, 0].min() for s in strokes)
|
| 40 |
+
return [((s - [left, top]) + [MARGIN, MARGIN + row * GAP]) * PX for s in strokes]
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def page_size(placed):
|
| 44 |
+
points = np.concatenate([s for strokes in placed for s in strokes])
|
| 45 |
+
return int(points[:, 0].max() + MARGIN * PX) + 1, int(points[:, 1].max() + MARGIN * PX) + 1
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def write_svg(strokes, W, H, path, pen):
|
| 49 |
+
width = pen * 150 / 72.0
|
| 50 |
+
g = lambda p: f"{p[0]:.2f},{p[1]:.2f}"
|
| 51 |
+
parts = []
|
| 52 |
+
for stroke in strokes:
|
| 53 |
+
s = _smooth(stroke, True)
|
| 54 |
+
if len(s) == 1:
|
| 55 |
+
parts.append(f"M{g(s[0])} L{g(s[0])}")
|
| 56 |
+
else:
|
| 57 |
+
parts.append(f"M{g(s[0])} " + " ".join(f"C{g(a)} {g(b)} {g(d)}" for _, a, b, d in _bezier(s)))
|
| 58 |
+
with open(path, "w") as fh:
|
| 59 |
+
fh.write(
|
| 60 |
+
f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 {W} {H}" width="{W / 150 * 25.4:.1f}mm" height="{H / 150 * 25.4:.1f}mm">'
|
| 61 |
+
+ f'<path d="{" ".join(parts)}" fill="none" stroke="#141414" stroke-width="{width:.2f}" stroke-linecap="round" stroke-linejoin="round"/></svg>'
|
| 62 |
+
)
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def write_png(strokes, W, H, path, pen, marks=()):
|
| 66 |
+
over, k = 4, 300 / 150 * 4
|
| 67 |
+
image = Image.new("L", (int(W * k), int(H * k)), 255)
|
| 68 |
+
draw = ImageDraw.Draw(image)
|
| 69 |
+
width = max(1, int(round(pen / 72.0 * 300 * over)))
|
| 70 |
+
for y0, y1 in marks:
|
| 71 |
+
draw.rectangle([0.15 * PX * k, y0 * k, 0.25 * PX * k, y1 * k], fill=120)
|
| 72 |
+
for stroke in strokes:
|
| 73 |
+
s = _smooth(stroke, True)
|
| 74 |
+
points = np.vstack([s[:1]] + [_sample(seg) for seg in _bezier(s)]) * k if len(s) > 1 else s * k
|
| 75 |
+
if len(points) > 1:
|
| 76 |
+
draw.line([tuple(p) for p in points], fill=20, width=width, joint="curve")
|
| 77 |
+
for p in (points[0], points[-1]):
|
| 78 |
+
draw.ellipse([p[0] - width / 2, p[1] - width / 2, p[0] + width / 2, p[1] + width / 2], fill=20)
|
| 79 |
+
image.resize((int(W * 2), int(H * 2)), Image.LANCZOS).save(path, dpi=(300, 300))
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def main():
|
| 83 |
+
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 84 |
+
ap.add_argument("text", help="one line of text, or a .txt file with a line per row")
|
| 85 |
+
ap.add_argument("out", help="output folder, must not exist")
|
| 86 |
+
ap.add_argument("--ref", default=DEFAULT_REF, help="the style: one line of handwriting, [n, 3] .npy from make_ref.py")
|
| 87 |
+
ap.add_argument("--ref-text", help="what is written in the reference line; read by the recogniser if not given")
|
| 88 |
+
ap.add_argument("--attempts", type=int, default=ATTEMPTS, help="samples per line; the best one is kept")
|
| 89 |
+
ap.add_argument("--seed", type=int, default=0, help="another seed gives another writing of the same text")
|
| 90 |
+
ap.add_argument("--pen", type=float, default=1.1, help="pen width in points")
|
| 91 |
+
args = ap.parse_args()
|
| 92 |
+
if os.path.exists(args.out):
|
| 93 |
+
raise SystemExit(f"{args.out} already exists")
|
| 94 |
+
if os.path.isfile(args.text):
|
| 95 |
+
with open(args.text, encoding="utf-8") as fh:
|
| 96 |
+
texts = [row.strip() for row in fh if row.strip()]
|
| 97 |
+
else:
|
| 98 |
+
texts = [args.text.strip()]
|
| 99 |
+
|
| 100 |
+
model = SiteModel()
|
| 101 |
+
reference = np.load(args.ref).astype(np.float32)
|
| 102 |
+
ref_text = args.ref_text
|
| 103 |
+
if ref_text is None and os.path.abspath(args.ref) == os.path.abspath(DEFAULT_REF):
|
| 104 |
+
with open(os.path.splitext(DEFAULT_REF)[0] + ".json", encoding="utf-8") as fh:
|
| 105 |
+
ref_text = json.load(fh)["text"]
|
| 106 |
+
if ref_text is None:
|
| 107 |
+
ref_text = model.read(reference)
|
| 108 |
+
print(f'the reference reads as "{ref_text}"; pass --ref-text if that is wrong')
|
| 109 |
+
ref = model.reference(reference, ref_text)
|
| 110 |
+
print(f"device {DEVICE}, writer's spread {ref['spread']:.2f}")
|
| 111 |
+
|
| 112 |
+
report, chosen, everything, marks = [], [], [], []
|
| 113 |
+
for n, text in enumerate(texts):
|
| 114 |
+
unknown = sorted(set(text) - set(model.clean(text)))
|
| 115 |
+
if unknown:
|
| 116 |
+
print(f"line {n + 1}: the model does not know {' '.join(unknown)}, dropped")
|
| 117 |
+
started = time.time()
|
| 118 |
+
try:
|
| 119 |
+
result = model.write(text, ref, attempts=args.attempts, seed=args.seed + n)
|
| 120 |
+
except ValueError as error:
|
| 121 |
+
raise SystemExit(f"line {n + 1}: {error}")
|
| 122 |
+
pick = result["pick"]
|
| 123 |
+
print(
|
| 124 |
+
f"line {n + 1}: {time.time() - started:.0f} s, error rate {' '.join(f'{c:.2f}' for c in result['cer'])}, "
|
| 125 |
+
f"closeness {' '.join(f'{c:.2f}' for c in result['cos'])}, chosen {pick + 1}"
|
| 126 |
+
)
|
| 127 |
+
chosen.append(place(result["lines"][pick], n))
|
| 128 |
+
for k, line in enumerate(result["lines"]):
|
| 129 |
+
row = len(everything) + 0.4 * n
|
| 130 |
+
everything.append(place(line, row))
|
| 131 |
+
if k == pick:
|
| 132 |
+
marks.append(((MARGIN + row * GAP) * PX, (MARGIN + row * GAP + 1) * PX))
|
| 133 |
+
report.append(
|
| 134 |
+
{
|
| 135 |
+
"text": result["text"],
|
| 136 |
+
"pick": pick,
|
| 137 |
+
"cer": [round(float(c), 4) for c in result["cer"]],
|
| 138 |
+
"cos": [round(float(c), 4) for c in result["cos"]],
|
| 139 |
+
"attempts": [[[[round(float(x), 4), round(float(y), 4)] for x, y in s] for s in to_strokes(line)] for line in result["lines"]],
|
| 140 |
+
}
|
| 141 |
+
)
|
| 142 |
+
|
| 143 |
+
os.makedirs(args.out)
|
| 144 |
+
W, H = page_size(chosen)
|
| 145 |
+
strokes = [s for line in chosen for s in line]
|
| 146 |
+
write_svg(strokes, W, H, os.path.join(args.out, "line.svg"), args.pen)
|
| 147 |
+
write_png(strokes, W, H, os.path.join(args.out, "line.png"), args.pen)
|
| 148 |
+
_write_pdf([strokes], [], W, H, 150, os.path.join(args.out, "line.pdf"), args.pen, True)
|
| 149 |
+
W, H = page_size(everything)
|
| 150 |
+
write_png([s for line in everything for s in line], W, H, os.path.join(args.out, "attempts.png"), args.pen, marks)
|
| 151 |
+
with open(os.path.join(args.out, "strokes.json"), "w", encoding="utf-8") as fh:
|
| 152 |
+
json.dump({"reference_text": ref_text, "spread": round(ref["spread"], 4), "lines": report}, fh, ensure_ascii=False)
|
| 153 |
+
print("written to", args.out)
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
if __name__ == "__main__":
|
| 157 |
+
main()
|