AtomicChat/Unlimited-OCR-2-GGUF

Unlimited-OCR-2 GGUF

GGUF builds of baidu/Unlimited-OCR-2, Baidu's 3.3B mixture-of-experts model that turns document page images (scans, photos, PDF pages) into text with layout, by AtomicChat.

These files need llama.cpp with Unlimited-OCR-2 support, which mainline llama.cpp does not have yet. v2 uses a page-aware 2D RoPE and SR-SWA (DASH) routed attention. Build llama.cpp b11913 (commit 7aca0d2ea8) with the community patch unlimited-ocr2.patch from RX5950XT/Unlimited-OCR-2-GGUF (sha256 dcdac02ede443987…). Mainline llama.cpp and apps built on it cannot load these files.

• mmproj-Unlimited-OCR-2-F16.gguf is the vision encoder (SAM + CLIP); every text file needs it.
• The DASH router (indexer.* tensors) stays BF16 in every file, so routing follows the original.
• The embedded chat template puts <PAGE> before every image and adds the trained instruction (Multi page parsing.) when a message has no text, so chat clients and llama-server work without a special prompt.
• Atomic Chat support: in progress (AtomicBot-ai/atomic-llama-cpp-turboquant#96).

Files

File	Size on disk	Pages within 1% of the original's text	Character difference vs the original: median / mean	Difference vs BF16 GGUF	Role
Unlimited-OCR-2-BF16.gguf	5.88 GB	24/30	0.05% / 2.94%	0.00%	full precision
Unlimited-OCR-2-Q8_0.gguf	3.13 GB	23/30	0.12% / 2.57%	1.11%	recommended
Unlimited-OCR-2-AD-Q6_K.gguf	2.74 GB	24/30	0.08% / 2.27%	0.72%	smaller, near Q8_0
Unlimited-OCR-2-AD-Q5_K.gguf	2.28 GB	23/30	0.08% / 3.36%	0.52%	smaller
Unlimited-OCR-2-AD-Q4_K.gguf	2.02 GB	24/30	0.08% / 2.94%	0.07%	smallest

Measured on 30 real pages from OmniDocBench (2 per category: textbooks, newspapers, slides, reports, notes, exams, books, magazines, papers and more; seed 0), one page per request. The reference is the original model in BF16 with the vendor's own code (transformers). Character difference = edit distance of the recognized text after removing layout headers (block type and box), divided by the original's length; lower is better. The BF16 row is the gap between the patched llama.cpp and the vendor code; the quantized files add little to it (last column).

That gap comes from near-ties, not from a wrong port. When the BF16 GGUF's output is forced through the original model, the original's own top choice agrees with the llama.cpp token on 98.8% of 34,926 steps on these pages, and almost all differences are bfloat16 near-ties (logit gap above 1 on 33 steps). An early near-tie in the layout, such as a block labelled image or table or a box shifted by a pixel, then changes the rest of the page.

How the AD files are built

• Attention, shared experts, the dense FFN of layer 0, token embeddings and the output head: Q8_0.
• Routed experts (about nine tenths of the weights): gate/up at the file's type (Q6_K, Q5_K, Q4_K).
• ffn_down_exps rows are 896 wide, not a multiple of 256, so K-quants cannot be used there (llama-quantize would silently swap the type). They are set explicitly to Q8_0, Q5_1 and Q5_0.
• No importance matrix yet.

Run

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout b11913
hf download RX5950XT/Unlimited-OCR-2-GGUF llama.cpp-patch/unlimited-ocr2.patch --local-dir .
git apply llama.cpp-patch/unlimited-ocr2.patch
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j --target llama-mtmd-cli llama-server

./build/bin/llama-mtmd-cli -m Unlimited-OCR-2-Q8_0.gguf --mmproj mmproj-Unlimited-OCR-2-F16.gguf \
  --jinja --temp 0 -c 32768 -n 8000 --image page.png -p "<PAGE><__media__>Multi page parsing."

# or as an OpenAI-compatible server: send a page image (text optional), temperature 0
./build/bin/llama-server -m Unlimited-OCR-2-Q8_0.gguf --mmproj mmproj-Unlimited-OCR-2-F16.gguf \
  --jinja -np 1 -c 32768

Drop -DGGML_CUDA=ON without an NVIDIA GPU (Metal is used on Mac automatically).

Limits

• The patch derives page positions from the token history of one sequence: run one request at a time (no parallel server slots).
• Measured on single pages. Multi-page documents are not measured here yet.
• The file format follows the community patch. If mainline llama.cpp adds v2 with different keys, we will convert again.

License

MIT, inherited from baidu/Unlimited-OCR-2. llama.cpp v2 support: patch by RX5950XT. Built by AtomicChat.