jefflaporte/chandra-ocr-2-MLX-4bit-mixed_4_8

Chandra OCR 2 — MLX 4-bit (mixed_4_8)

An MLX mixed 4/8-bit quantization of Chandra OCR 2 by Datalab, for Apple Silicon.

These files are modified: the language model's weights are quantized from the original BF16 checkpoint. The vision tower is unchanged (BF16), and so are the tokenizer, chat template and processor configuration. All credit for the model belongs to Datalab; see the original model card for what Chandra does and how it was evaluated.

Chandra OCR 2 is built on Qwen3.5-4B by the Qwen team, Alibaba Cloud, licensed under the Apache License 2.0, a copy of which is included as LICENSE-Qwen.

License and use restrictions

This derivative is distributed under the same license as the original, included as LICENSE (a modified OpenRAIL-M). Its use restrictions apply to these files too, including the commercial restrictions in Attachment A: no use by an organization with more than $2M in annual revenue or in total funding, except personal or research use, and no use by a competitor of Datalab. Commercial licenses are available from Datalab at https://www.datalab.to/.

Quantization

mlx-vlm's mixed_4_8 recipe, which follows llama.cpp's Q4_K_M: projections are affine 4-bit in groups of 64, except these, which are 8-bit:

• the token table, which is also the output head (the model ties them);
• v_proj and mlp.down_proj in the first and last eighth of the layers and in every third layer between them.

Rounded to nearest, with no calibration. The vision tower stays BF16. 3.3 GB.

A plain 4-bit quantization (the token table and head at 4-bit too) misread a date on one of the test pages ("3 September" as "5 September"); this recipe read it correctly.

Made with mlx-vlm 0.7.6 (see quantization.json for the exact source revision and tools):

mlx_vlm convert --hf-path datalab-to/chandra-ocr-2 --mlx-path chandra-ocr-2-MLX-4bit-mixed_4_8 \
  -q --quant-predicate mixed_4_8

mlx-vlm rewrites the processor configuration on conversion; the original preprocessor_config.json, processor_config.json and video_preprocessor_config.json were restored, since vLLM rejects the rewritten ones and the preprocessing is the original model's.

Quality

On nine scanned 1940s typewritten documents (memos, letters, an expense ledger), its text matched the original BF16 model's on six pages. On the other three it differed in layout, not content: line breaks dropped inside table cells, a separator labeled as text rather than a page footer, and a different (and correct) row alignment of a ledger's amounts.

This is a spot check, not a benchmark: run your own evaluation (for example olmOCR-bench) before relying on it.

Usage

With vllm-metal and Chandra's own client (pip install chandra-ocr):

vllm serve jefflaporte/chandra-ocr-2-MLX-4bit-mixed_4_8 --served-model-name chandra \
  --mm-processor-kwargs '{"min_pixels": 3136, "max_pixels": 6291456}'
chandra input.pdf ./output --method vllm

With mlx-vlm:

python -m mlx_vlm.server --model jefflaporte/chandra-ocr-2-MLX-4bit-mixed_4_8

Chandra reads a page at up to 6,291,456 pixels; a server that caps images lower can lose the layout of dense pages.