GLM-OCR, ONNX, 8-bit vision / 4-bit language model (for handwriting)

This is not a new model. It is zai-org/GLM-OCR (code: https://github.com/zai-org/GLM-OCR), converted to ONNX and quantised after the fact. No training of any kind was done: no retraining, no fine-tuning, no distillation, no calibration data. All credit for the model, its training and its data belongs to its authors (zai-org); for what it is, how it was trained, its intended use, its limitations and how to cite it, see the original model card. Everything under "What was changed" is all that was done to it.

What was changed

  1. Exported to ONNX in two graphs (vision encoder; language model with output head). For the export the model's code was rewritten without changing what it computes: the cache objects and Python loops of transformers replaced by plain tensor inputs, the Conv3d/Conv2d patch layers written as matrix multiplications, the multi-token-prediction head left out (it is not used for reading). The fp32 export gives the same tokens as the original PyTorch model (checked token for token on test lines).
  2. Quantised after training (post-training, weight-only, round-to-nearest, onnxruntime MatMulNBits): the vision encoder's weights to 8 bits (block 128), the language model's and output head's weights to 4 bits (block 128), the word-vector table to int8 with one scale per row. Activations stay in floating point. No data was used to make or tune the quantisation. vision-web.onnx is the same vision encoder with 4-bit weights (block 32), made the same way (see "Browser variant").
  3. Nothing else. No weights were changed other than by rounding them to fewer bits; the tokenizer vocabulary is the original's.

Because of the rounding the output can differ slightly from the original model's. Questions about the conversion belong here, not with the original authors.

ONNX conversion of zai-org/GLM-OCR (0.9B, MIT) made for reading lines of handwriting with ONNX Runtime on ordinary devices, together with the scripts that made the files. Tested with ONNX Runtime 1.24 (Python) and 1.28 (Flutter plugin) on a desktop CPU; vision-web.onnx was also run in Chromium (onnxruntime-web 1.30, WebGPU); not tested on phones. It is the handwriting reader of the author's note-taking app (Notation).

file what size
vision.onnx vision encoder, weights in 8 bits (MatMulNBits, block 128), activations fp32 430 MB
vision-web.onnx the same vision encoder with weights in 4 bits (MatMulNBits, block 32): the variant a browser's graphics card (WebGPU) can run 261 MB
text.onnx language model + output head, weights in 4 bits (MatMulNBits, block 128) 297 MB
embeddings.bin the word-vector table: 59392 x 1536 int8 values, then 59392 float32 scales (row = int8 * scale) 88 MB
vision-gpu.part1.onnx … part5.onnx (+ .data) the vision encoder unquantised (fp16), cut into 5 parts that run one after the other: for QNN on Android phones (see "Phone variant") 870 MB
vocab.txt the 59246 ordinary tokens, one per line in id order, byte-level BPE (Ġ = space, Ċ = newline) 0.6 MB
SHA256SUMS
scripts/ export, quantisation, reference runner (see below)

Browser variant (vision-web.onnx)

ONNX Runtime's WebGPU provider only runs MatMulNBits with 2 or 4 bits, so it cannot open vision.onnx (8 bits). vision-web.onnx is the same graph with every weight matrix rounded to 4 bits (126 matrices, block 32, symmetric, no calibration data); use it with the same text.onnx and embeddings.bin. Measured on a desktop (Ryzen with a Radeon 780M, one vision pass over a line of 10 x 100 patches):

run time
vision.onnx, ONNX Runtime (Python, CPU), 8 threads 1.8 s
vision-web.onnx, ONNX Runtime (Python, CPU), 8 threads 1.5 s
vision-web.onnx, onnxruntime-web 1.30, WebGPU (Chromium, Radeon 780M) 3.1 s
vision.onnx, onnxruntime-web, wasm, one thread 36.5 s

So WebGPU on an integrated GPU is not faster than a good native CPU; it is much faster than a browser's own CPU path. The 4-bit weights cost accuracy. On 68 lines of the author's own handwriting (German and English; strict scoring, the vision encoder changed and the full-precision language model kept) the character error rate was 13.3 % with this file, 11.5 % with an 8-bit encoder of the same block size, and 10.9 % for the shipped vision.onnx with the 4-bit text.onnx (that last pair is the combination in the table at the top). The output of the WebGPU run matches the CPU run to about 1e-3. This file with the 4-bit text.onnx was not scored as a pair.

Phone variant (vision-gpu.part*.onnx)

The vision encoder unquantised (16-bit floats, no MatMulNBits: Qualcomm's QNN runs no quantised operators of that kind), cut into five graphs by scripts/split_glm_vision.py. Compiling the whole 868 MB encoder for QNN at once took an app past 3.3 GB on a phone with 6 GB of memory, and Android stopped it; each part is compiled and run on its own.

part what weights
part1 the rotary position tables from pos_ids (no weights) –
part2 patch embedding and blocks 0–6 212 MB
part3 blocks 7–12 202 MB
part4 blocks 13–18 202 MB
part5 blocks 19–23 and the merger; output features, as the whole encoder's 253 MB

Each part takes what it needs of pixel_values, pos_ids and the outputs of the parts before it (by name); the values between parts are float32. The number of patches is the free dimension 4*m in every part, so fixing that one name fixes the shapes of all of them. Run in a row on the CPU, the parts give the whole encoder's features to 1.4e-3 (values up to 0.4), the size of the difference ONNX Runtime's own graph optimisations make in the whole encoder (1.2e-3); cut into one part, they are identical.

Measured only as a proof that the path works, on a Snapdragon 865 (2020, Adreno 650, ONNX Runtime 1.28 with QNN 2.42): the four parts with weights ran on QNN's GPU backend, the final features within 0.007 of the CPU's, at about 12 s a part, four times slower than the CPU of the same phone; the NPU backend did not start on that chip. Newer chips (Hexagon NPU with fp16) were not tested.

Graphs

  • vision.onnx: pixel_values float32 [N, 1176] (the patches as the transformers processor lays them out: 14x14 patches in blocks of 2x2, each patch channel-major with the picture repeated as the model's two frames), pos_ids int64 [N, 2] (row and column of each patch in that order) -> features [N/4, 1536]. Any size works (N a multiple of 4).
  • text.onnx: inputs_embeds [1, T, 1536], position_ids int64 [3, 1, T] (the three rotary positions: time, row, column), attention_mask int64 [1, P] (covers the cache only), past_key_values.{0..15}.{key,value} [1, 8, P, 128] -> logits of the last position [1, 1, 59392], present.{i}.{key,value} [1, 8, P+T, 128]. Start with a one-position zero cache and a mask of [0].
  • Prompt ids: [59248, 59250, 59253, 10, 59256] + 59280 for every 2x2 patches + [59257, 3649, 7404, 49600, 58, 59254, 10] ([gMASK]<sop><|user|>\n<|begin_of_image|> ... <|end_of_image|>Text Recognition:<|assistant|>\n). Stop at 59246 or 59253. Positions: words before the picture 0, 1, 2...; picture tokens share time = 5 and take row/column from the 2x2-merged grid; the words after continue at 5 + max(rows, columns).
  • scripts/run_glm_onnx.py is a complete reference reader (it uses the transformers processor for the picture; scripts/patches_check.py shows the layout the processor makes). scripts/glm_parts.py has the two parts written for export; export_glm.py exports with torch.onnx.export(dynamo=True); quantize2.py makes the quantised variants; make_release.py writes the files above.

What it reads like

Measured on 68 lines of the author's own German/English handwriting (drawn from pen strokes, a few lines with maths; a small, personal sample: treat the numbers as indications). Character error rate, lower is better; LaTeX in the output counts as errors:

CER
GLM-OCR fp32 (PyTorch) 11.3 %
these files (8-bit vision, 4-bit language model) 10.8-10.9 %
4-bit vision encoder instead 13.3-15.5 %
TrOCR small handwritten 42.6 %
PaddleOCR-VL 1.5 (4-bit ONNX files of onnx-community, drawn as the app drew them) 34.2 %
Florence-2-base-ft 37.1 %

The vision encoder needs 8-bit weights (4 bits cost about 2 points; dynamic int8, which also quantises the activations, breaks it: 94 % CER); the language model, the output head and the embeddings can be 4-bit / int8 at no measurable cost. The vision encoder is about three quarters of the time: about 2 s a line on a desktop CPU with 4 threads at a picture height of 140 px (10 rows of patches), 1.4-1.6 s at 112 px (12.0 % CER).

Known weaknesses (the model's, not the conversion's): it often writes maths as LaTeX ($ \rightarrow $) and sometimes "corrects" a German word to English (Subtraktion -> Subtraction). Nothing here post-processes the text.

Licence and credits

The original GLM-OCR model is released under the MIT License (see its model card); the copyright and permission notice is kept in LICENSE here, and applies to these files, which are a modified form of the original's weights. The original's code repository (https://github.com/zai-org/GLM-OCR) is under the Apache License 2.0; none of its code is in this repository. The scripts in scripts/ were written for this conversion and are released under the same MIT terms. The full pipeline of the original also uses PP-DocLayoutV3 (Apache 2.0, a layout detector) for whole pages; it is not part of these files, which read one line at a time. If you use these files, credit the original GLM-OCR and cite it as its model card asks.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for nfa04/glm-ocr-onnx

Base model

zai-org/GLM-OCR
Quantized
(41)
this model