| --- |
| license: cc-by-sa-4.0 |
| language: |
| - ja |
| - ko |
| - zh |
| pipeline_tag: image-to-text |
| library_name: onnx |
| base_model: kha-white/manga-ocr-base |
| tags: |
| - ocr |
| - manga |
| - manhwa |
| - manhua |
| - comics |
| - vision-encoder-decoder |
| - onnx |
| --- |
| |
| # Kemolate OCR |
|
|
| Reads the lettering in comic balloons — Japanese, Korean, and simplified and |
| traditional Chinese — from a crop of one block of text. It is the default reader |
| in Kemolate, an app for translating comics, and runs there in the browser or |
| desktop window on the reader's own machine. |
|
|
| It is a retrain of [manga-ocr](https://huggingface.co/kha-white/manga-ocr-base): |
| the same vision encoder-decoder, with its tokenizer extended from 6,144 tokens to |
| 11,331 for hangul and Chinese, trained on lettering from manga, manhwa and manhua. |
|
|
| ## Builds |
|
|
| Each training run is published as a tag on this repository, and `main` holds the |
| newest. **Use a tag, not `main`**: a build is an encoder, a decoder and a |
| tokenizer that only work together, and `main` moves on with the next run. |
|
|
| | Tag | Date | What changed | |
| | ------- | ---------- | ------------ | |
| | `out26` | 2026-10-02 | About 3,800 more real Chinese labels, from some twenty letterers no earlier run had seen | |
|
|
| ## Files |
|
|
| The ONNX export is laid out for raw sessions rather than for |
| `transformers.js`' pipeline: a cacheless decoder, with the encoder-to-decoder |
| projection folded into it. |
|
|
| | File | Precision | Size | |
| | ---- | --------- | ---- | |
| | `encoder_model.onnx`, `decoder_model.onnx` | fp32 | 328 MB, 142 MB | |
| | `encoder_model_fp16.onnx`, `decoder_model_fp16.onnx` | fp16 | 164 MB, 71 MB | |
| | `encoder_model_quantized.onnx`, `decoder_model_quantized.onnx` | int8 | 83 MB, 36 MB | |
| | `tokenizer.json` and the configs | — | — | |
|
|
| - **fp32** is the reference. **fp16** has its inputs and outputs kept fp32, so it |
| is fed and read exactly as fp32 is, and keeps each layer norm's variance in |
| fp32 (the encoder's squared activations would overflow fp16). It read 499 of |
| 500 test crops exactly as fp32 did. **int8** is for CPU and WebAssembly, where |
| integer arithmetic is the cheap kind. |
| - `tokenizer.json` was made from the checkpoint's `BertJapaneseTokenizer` |
| (`vocab.txt` and `added_tokens.json`), which `transformers.js` cannot read; |
| every one of the 11,331 ids decodes as the original does. |
|
|
| ## Reading |
|
|
| - **Input:** one crop, resized to 224 × 224 RGB and normalised with mean and |
| standard deviation 0.5 (`preprocessor_config.json`). Vertical and horizontal |
| lettering both read from the same model. |
| - **Decoding:** greedy, from token 2 until token 3; Kemolate stops at 300 tokens. |
| The numbers below are greedy. `generation_config.json` describes the four-beam |
| search the training pipeline evaluates with, which can differ on a hard |
| balloon. |
|
|
| ## Results |
|
|
| Character error rate (CER, lower is better) and exact matches. |
|
|
| **On real pages no run trained on** — the comparison between builds: |
|
|
| | Benchmark | Crops | `out26` | Previous build | |
| | --------- | ----- | ------- | -------------- | |
| | Chinese chapters, eleven books | 2,025 | 0.049 CER | 0.054 (`out25`) | |
| | Korean books, six | 975 | tied with `out25` | `out25`: 0.065 CER, 74% exact | |
|
|
| On the Chinese set `out26` wins 123 crops to 88, and nine of the eleven books read |
| better, the largest gains on letterers absent from its new labels. Japanese real |
| pages and a held-out Chinese title read the same as `out25`. |
|
|
| **By precision**, 100 crops a language, greedy decode as Kemolate runs it. Every |
| column but `ja-REAL` is this run's own synthetic evaluation slice, so this table |
| compares the files of one build with each other, never one build with another: |
|
|
| | | ja-REAL | ja | ko | zh-Hans | zh-Hant | |
| | - | ------- | -- | -- | ------- | ------- | |
| | int8 | 0.015 / 95% | 0.058 / 77% | 0.079 / 78% | 0.098 / 82% | 0.089 / 84% | |
| | fp32 | 0.020 / 95% | 0.057 / 77% | 0.074 / 79% | 0.098 / 82% | 0.097 / 83% | |
|
|
| ## Limitations |
|
|
| - **Korean comes back without word spacing.** No token is a space, so the |
| tokenizer has no way to write one. |
| - **Long blocks.** Crops of 23 or more characters in a box no taller than it is |
| wide have been left out of training, so a long paragraph — a disclaimer, a |
| credits block — reads worse than a balloon. |
| - **Ellipses** are written as three periods in Korean and as `…` in Japanese and |
| Chinese, as the training labels write them. |
| - One block at a time: it does not find text on a page. Kemolate pairs it with a |
| comic text detector. |
|
|
| ## Training |
|
|
| Initialised from `kha-white/manga-ocr-base`, with its vocabulary extended (every |
| existing id kept) to cover hangul and simplified and traditional Chinese, and |
| later 152 hangul syllables that comic sound effects use and prose almost never |
| does (읏, 흣, 큿 among them). Fine-tuned on synthetic lettering rendered in many |
| typefaces, plus real lettering cropped from comics and labelled for this project: |
| about 2,800 Korean and 6,900 Chinese labels across the runs up to `out26`. |
|
|
| ## Licence and attribution |
|
|
| Released under **CC BY-SA 4.0**, because the decoder descends from |
| [`tohoku-nlp/bert-base-japanese-char-v2`](https://huggingface.co/tohoku-nlp/bert-base-japanese-char-v2), |
| which is licensed CC BY-SA 4.0 and asks that adaptations share alike. |
|
|
| - Encoder: [`facebook/deit-base-patch16-224`](https://huggingface.co/facebook/deit-base-patch16-224), Apache-2.0. |
| - Decoder: [`tohoku-nlp/bert-base-japanese-char-v2`](https://huggingface.co/tohoku-nlp/bert-base-japanese-char-v2), CC BY-SA 4.0. |
| - Starting weights: [`kha-white/manga-ocr-base`](https://huggingface.co/kha-white/manga-ocr-base) |
| by Maciej Budyś, Apache-2.0, itself trained on Manga109-s. |
|
|
