---
license: cc-by-sa-4.0
language:
  - ja
  - ko
  - zh
pipeline_tag: image-to-text
library_name: onnx
base_model: kha-white/manga-ocr-base
tags:
  - ocr
  - manga
  - manhwa
  - manhua
  - comics
  - vision-encoder-decoder
  - onnx
---

# Kemolate OCR

Reads the lettering in comic balloons — Japanese, Korean, and simplified and
traditional Chinese — from a crop of one block of text. It is the default reader
in Kemolate, an app for translating comics, and runs there in the browser or
desktop window on the reader's own machine.

It is a retrain of [manga-ocr](https://huggingface.co/kha-white/manga-ocr-base):
the same vision encoder-decoder, with its tokenizer extended from 6,144 tokens to
11,331 for hangul and Chinese, trained on lettering from manga, manhwa and manhua.

## Builds

Each training run is published as a tag on this repository, and `main` holds the
newest. **Use a tag, not `main`**: a build is an encoder, a decoder and a
tokenizer that only work together, and `main` moves on with the next run.

| Tag     | Date       | What changed |
| ------- | ---------- | ------------ |
| `out26` | 2026-10-02 | About 3,800 more real Chinese labels, from some twenty letterers no earlier run had seen |

## Files

The ONNX export is laid out for raw sessions rather than for
`transformers.js`' pipeline: a cacheless decoder, with the encoder-to-decoder
projection folded into it.

| File | Precision | Size |
| ---- | --------- | ---- |
| `encoder_model.onnx`, `decoder_model.onnx` | fp32 | 328 MB, 142 MB |
| `encoder_model_fp16.onnx`, `decoder_model_fp16.onnx` | fp16 | 164 MB, 71 MB |
| `encoder_model_quantized.onnx`, `decoder_model_quantized.onnx` | int8 | 83 MB, 36 MB |
| `tokenizer.json` and the configs | — | — |

- **fp32** is the reference. **fp16** has its inputs and outputs kept fp32, so it
  is fed and read exactly as fp32 is, and keeps each layer norm's variance in
  fp32 (the encoder's squared activations would overflow fp16). It read 499 of
  500 test crops exactly as fp32 did. **int8** is for CPU and WebAssembly, where
  integer arithmetic is the cheap kind.
- `tokenizer.json` was made from the checkpoint's `BertJapaneseTokenizer`
  (`vocab.txt` and `added_tokens.json`), which `transformers.js` cannot read;
  every one of the 11,331 ids decodes as the original does.

## Reading

- **Input:** one crop, resized to 224 × 224 RGB and normalised with mean and
  standard deviation 0.5 (`preprocessor_config.json`). Vertical and horizontal
  lettering both read from the same model.
- **Decoding:** greedy, from token 2 until token 3; Kemolate stops at 300 tokens.
  The numbers below are greedy. `generation_config.json` describes the four-beam
  search the training pipeline evaluates with, which can differ on a hard
  balloon.

## Results

Character error rate (CER, lower is better) and exact matches.

**On real pages no run trained on** — the comparison between builds:

| Benchmark | Crops | `out26` | Previous build |
| --------- | ----- | ------- | -------------- |
| Chinese chapters, eleven books | 2,025 | 0.049 CER | 0.054 (`out25`) |
| Korean books, six | 975 | tied with `out25` | `out25`: 0.065 CER, 74% exact |

On the Chinese set `out26` wins 123 crops to 88, and nine of the eleven books read
better, the largest gains on letterers absent from its new labels. Japanese real
pages and a held-out Chinese title read the same as `out25`.

**By precision**, 100 crops a language, greedy decode as Kemolate runs it. Every
column but `ja-REAL` is this run's own synthetic evaluation slice, so this table
compares the files of one build with each other, never one build with another:

| | ja-REAL | ja | ko | zh-Hans | zh-Hant |
| - | ------- | -- | -- | ------- | ------- |
| int8 | 0.015 / 95% | 0.058 / 77% | 0.079 / 78% | 0.098 / 82% | 0.089 / 84% |
| fp32 | 0.020 / 95% | 0.057 / 77% | 0.074 / 79% | 0.098 / 82% | 0.097 / 83% |

## Limitations

- **Korean comes back without word spacing.** No token is a space, so the
  tokenizer has no way to write one.
- **Long blocks.** Crops of 23 or more characters in a box no taller than it is
  wide have been left out of training, so a long paragraph — a disclaimer, a
  credits block — reads worse than a balloon.
- **Ellipses** are written as three periods in Korean and as `…` in Japanese and
  Chinese, as the training labels write them.
- One block at a time: it does not find text on a page. Kemolate pairs it with a
  comic text detector.

## Training

Initialised from `kha-white/manga-ocr-base`, with its vocabulary extended (every
existing id kept) to cover hangul and simplified and traditional Chinese, and
later 152 hangul syllables that comic sound effects use and prose almost never
does (읏, 흣, 큿 among them). Fine-tuned on synthetic lettering rendered in many
typefaces, plus real lettering cropped from comics and labelled for this project:
about 2,800 Korean and 6,900 Chinese labels across the runs up to `out26`.

## Licence and attribution

Released under **CC BY-SA 4.0**, because the decoder descends from
[`tohoku-nlp/bert-base-japanese-char-v2`](https://huggingface.co/tohoku-nlp/bert-base-japanese-char-v2),
which is licensed CC BY-SA 4.0 and asks that adaptations share alike.

- Encoder: [`facebook/deit-base-patch16-224`](https://huggingface.co/facebook/deit-base-patch16-224), Apache-2.0.
- Decoder: [`tohoku-nlp/bert-base-japanese-char-v2`](https://huggingface.co/tohoku-nlp/bert-base-japanese-char-v2), CC BY-SA 4.0.
- Starting weights: [`kha-white/manga-ocr-base`](https://huggingface.co/kha-white/manga-ocr-base)
  by Maciej Budyś, Apache-2.0, itself trained on Manga109-s.