GLM-OCR, ONNX, 8-bit vision / 4-bit language model (for handwriting)
This is not a new model. It is zai-org/GLM-OCR (code: https://github.com/zai-org/GLM-OCR), converted to ONNX and quantised after the fact. No training of any kind was done: no retraining, no fine-tuning, no distillation, no calibration data. All credit for the model, its training and its data belongs to its authors (zai-org); for what it is, how it was trained, its intended use, its limitations and how to cite it, see the original model card. Everything under "What was changed" is all that was done to it.
What was changed
- Exported to ONNX in two graphs (vision encoder; language model with output head). For the export the model's code was rewritten
without changing what it computes: the cache objects and Python loops of
transformersreplaced by plain tensor inputs, theConv3d/Conv2dpatch layers written as matrix multiplications, the multi-token-prediction head left out (it is not used for reading). The fp32 export gives the same tokens as the original PyTorch model (checked token for token on test lines). - Quantised after training (post-training, weight-only, round-to-nearest,
onnxruntimeMatMulNBits): the vision encoder's weights to 8 bits (block 128), the language model's and output head's weights to 4 bits (block 128), the word-vector table to int8 with one scale per row. Activations stay in floating point. No data was used to make or tune the quantisation.vision-web.onnxis the same vision encoder with 4-bit weights (block 32), made the same way (see "Browser variant"). - Nothing else. No weights were changed other than by rounding them to fewer bits; the tokenizer vocabulary is the original's.
Because of the rounding the output can differ slightly from the original model's. Questions about the conversion belong here, not with the original authors.
ONNX conversion of zai-org/GLM-OCR (0.9B, MIT) made for reading lines of handwriting
with ONNX Runtime on ordinary devices, together with the scripts that made the files. Tested with ONNX Runtime 1.24 (Python) and 1.28
(Flutter plugin) on a desktop CPU; vision-web.onnx was also run in Chromium (onnxruntime-web 1.30, WebGPU); not tested on phones. It is the handwriting reader of the author's note-taking app (Notation).
| file | what | size |
|---|---|---|
vision.onnx |
vision encoder, weights in 8 bits (MatMulNBits, block 128), activations fp32 | 430 MB |
vision-web.onnx |
the same vision encoder with weights in 4 bits (MatMulNBits, block 32): the variant a browser's graphics card (WebGPU) can run | 261 MB |
text.onnx |
language model + output head, weights in 4 bits (MatMulNBits, block 128) | 297 MB |
embeddings.bin |
the word-vector table: 59392 x 1536 int8 values, then 59392 float32 scales (row = int8 * scale) | 88 MB |
vision-gpu.part1.onnx β¦ part5.onnx (+ .data) |
the vision encoder unquantised (fp16), cut into 5 parts that run one after the other: for QNN on Android phones (see "Phone variant") | 870 MB |
vocab.txt |
the 59246 ordinary tokens, one per line in id order, byte-level BPE (Δ = space, Δ = newline) |
0.6 MB |
SHA256SUMS |
||
scripts/ |
export, quantisation, reference runner (see below) |
Browser variant (vision-web.onnx)
ONNX Runtime's WebGPU provider only runs MatMulNBits with 2 or 4 bits, so it cannot open vision.onnx (8 bits). vision-web.onnx is
the same graph with every weight matrix rounded to 4 bits (126 matrices, block 32, symmetric, no calibration data); use it with the same
text.onnx and embeddings.bin. Measured on a desktop (Ryzen with a Radeon 780M, one vision pass over a line of 10 x 100 patches):
| run | time |
|---|---|
vision.onnx, ONNX Runtime (Python, CPU), 8 threads |
1.8 s |
vision-web.onnx, ONNX Runtime (Python, CPU), 8 threads |
1.5 s |
vision-web.onnx, onnxruntime-web 1.30, WebGPU (Chromium, Radeon 780M) |
3.1 s |
vision.onnx, onnxruntime-web, wasm, one thread |
36.5 s |
So WebGPU on an integrated GPU is not faster than a good native CPU; it is much faster than a browser's own CPU path. The 4-bit weights
cost accuracy. On 68 lines of the author's own handwriting (German and English; strict scoring, the vision encoder changed and the
full-precision language model kept) the character error rate was 13.3 % with this file, 11.5 % with an 8-bit encoder of the same block
size, and 10.9 % for the shipped vision.onnx with the 4-bit text.onnx (that last pair is the combination in the table at the top). The
output of the WebGPU run matches the CPU run to about 1e-3. This file with the 4-bit text.onnx was not scored as a pair.
Phone variant (vision-gpu.part*.onnx)
The vision encoder unquantised (16-bit floats, no MatMulNBits: Qualcomm's QNN runs no quantised operators of that kind), cut into five
graphs by scripts/split_glm_vision.py. Compiling the whole 868 MB encoder for QNN at once took an app past 3.3 GB on a phone with 6 GB
of memory, and Android stopped it; each part is compiled and run on its own.
| part | what | weights |
|---|---|---|
part1 |
the rotary position tables from pos_ids (no weights) |
β |
part2 |
patch embedding and blocks 0β6 | 212 MB |
part3 |
blocks 7β12 | 202 MB |
part4 |
blocks 13β18 | 202 MB |
part5 |
blocks 19β23 and the merger; output features, as the whole encoder's |
253 MB |
Each part takes what it needs of pixel_values, pos_ids and the outputs of the parts before it (by name); the values between parts are
float32. The number of patches is the free dimension 4*m in every part, so fixing that one name fixes the shapes of all of them. Run in a
row on the CPU, the parts give the whole encoder's features to 1.4e-3 (values up to 0.4), the size of the difference ONNX Runtime's own
graph optimisations make in the whole encoder (1.2e-3); cut into one part, they are identical.
Measured only as a proof that the path works, on a Snapdragon 865 (2020, Adreno 650, ONNX Runtime 1.28 with QNN 2.42): the four parts with weights ran on QNN's GPU backend, the final features within 0.007 of the CPU's, at about 12 s a part, four times slower than the CPU of the same phone; the NPU backend did not start on that chip. Newer chips (Hexagon NPU with fp16) were not tested.
Graphs
vision.onnx:pixel_valuesfloat32[N, 1176](the patches as the transformers processor lays them out: 14x14 patches in blocks of 2x2, each patch channel-major with the picture repeated as the model's two frames),pos_idsint64[N, 2](row and column of each patch in that order) ->features[N/4, 1536]. Any size works (N a multiple of 4).text.onnx:inputs_embeds[1, T, 1536],position_idsint64[3, 1, T](the three rotary positions: time, row, column),attention_maskint64[1, P](covers the cache only),past_key_values.{0..15}.{key,value}[1, 8, P, 128]->logitsof the last position[1, 1, 59392],present.{i}.{key,value}[1, 8, P+T, 128]. Start with a one-position zero cache and a mask of[0].- Prompt ids:
[59248, 59250, 59253, 10, 59256]+59280for every 2x2 patches +[59257, 3649, 7404, 49600, 58, 59254, 10]([gMASK]<sop><|user|>\n<|begin_of_image|>...<|end_of_image|>Text Recognition:<|assistant|>\n). Stop at 59246 or 59253. Positions: words before the picture 0, 1, 2...; picture tokens share time = 5 and take row/column from the 2x2-merged grid; the words after continue at 5 + max(rows, columns). scripts/run_glm_onnx.pyis a complete reference reader (it uses the transformers processor for the picture;scripts/patches_check.pyshows the layout the processor makes).scripts/glm_parts.pyhas the two parts written for export;export_glm.pyexports withtorch.onnx.export(dynamo=True);quantize2.pymakes the quantised variants;make_release.pywrites the files above.
What it reads like
Measured on 68 lines of the author's own German/English handwriting (drawn from pen strokes, a few lines with maths; a small, personal sample: treat the numbers as indications). Character error rate, lower is better; LaTeX in the output counts as errors:
| CER | |
|---|---|
| GLM-OCR fp32 (PyTorch) | 11.3 % |
| these files (8-bit vision, 4-bit language model) | 10.8-10.9 % |
| 4-bit vision encoder instead | 13.3-15.5 % |
| TrOCR small handwritten | 42.6 % |
| PaddleOCR-VL 1.5 (4-bit ONNX files of onnx-community, drawn as the app drew them) | 34.2 % |
| Florence-2-base-ft | 37.1 % |
The vision encoder needs 8-bit weights (4 bits cost about 2 points; dynamic int8, which also quantises the activations, breaks it: 94 % CER); the language model, the output head and the embeddings can be 4-bit / int8 at no measurable cost. The vision encoder is about three quarters of the time: about 2 s a line on a desktop CPU with 4 threads at a picture height of 140 px (10 rows of patches), 1.4-1.6 s at 112 px (12.0 % CER).
Known weaknesses (the model's, not the conversion's): it often writes maths as LaTeX ($ \rightarrow $) and sometimes "corrects" a German
word to English (Subtraktion -> Subtraction). Nothing here post-processes the text.
Licence and credits
The original GLM-OCR model is released under the MIT License (see its model card); the
copyright and permission notice is kept in LICENSE here, and applies to these files, which are a modified form of the original's weights.
The original's code repository (https://github.com/zai-org/GLM-OCR) is under the Apache License 2.0; none of its code is in this
repository. The scripts in scripts/ were written for this conversion and are released under the same MIT terms. The full pipeline of
the original also uses PP-DocLayoutV3 (Apache 2.0, a layout detector) for whole pages; it is not part of these files, which read one line
at a time. If you use these files, credit the original GLM-OCR and cite it as its model card asks.
Model tree for nfa04/glm-ocr-onnx
Base model
zai-org/GLM-OCR