AtomicChat/Unlimited-OCR-2-W4A16

Unlimited-OCR-2 W4A16

4-bit weight-only (W4A16, symmetric int4, group size 32, compressed-tensors) build of baidu/Unlimited-OCR-2, by AtomicChat. The feed-forward layers of the MoE text decoder (the routed and shared experts, about nine tenths of the parameters) are 4-bit; attention with its SR-SWA router and indexer, the vision tower, projector, expert routers and lm_head stay in BF16. Round-to-nearest, no calibration data.

Agreement with the original on real documents

Model	Size on disk	Pages within 1% of the original's text	Character difference vs the original: median / mean
W4A16, symmetric int4, group 32 (this repo)	3.06 GB	23/30	0.13% / 1.06%
W4A16, symmetric int4, group 128 (first upload)	2.96 GB	18/30	0.11% / 3.42%

Measured on 30 real pages from OmniDocBench (2 per category: textbooks, newspapers, slides, reports, notes, exams, books, magazines, papers and more; seed 0), one page per request, against the original model in BF16 with the vendor's code (transformers 4.52.1). The original gives the same text on every run (30/30 pages identical across two runs), so these differences come from the 4-bit weights. Character difference = edit distance of the recognized text after removing layout headers (block type and box), divided by the original's length.

Run (transformers)

The repo carries the vendor's remote code and the original config.json (plus the quantization block):

import torch
from compressed_tensors.linear.compressed_linear import CompressedLinear
from transformers import AutoModel, AutoTokenizer

name = "AtomicChat/Unlimited-OCR-2-W4A16"
tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)
model = AutoModel.from_pretrained(name, trust_remote_code=True, torch_dtype=torch.bfloat16).eval().cuda()

# unpack the 4-bit layers into plain BF16 Linear once (~3 s)
for parent in list(model.modules()):
    for child_name, child in list(parent.named_children()):
        if isinstance(child, CompressedLinear):
            lin = torch.nn.Linear(child.in_features, child.out_features, bias=child.bias is not None,
                                  device="cuda", dtype=torch.bfloat16)
            lin.weight.data = child.compressor.decompress_module(child).to(torch.bfloat16)
            setattr(parent, child_name, lin)

text = model.infer(tok, image_file="page.png", output_path="out", save_results=True, max_new_tokens=8192)

• Do not pass CompressedTensorsConfig(run_compressed=False) with transformers 4.52.1. On this MoE (about 2,100 quantized expert layers) the loader's key check becomes quadratic and stalls for hours. The unpack loop above gives the same weights in about 10 s.
• In transformers the gain is the download and disk size, not GPU memory.
• Cap max_new_tokens: the vendor default allows up to 131k new tokens per call.
• Do not re-save the loaded model with save_pretrained. The vendor code rewrites sparse_attention_config at load time (dash_stage0, dash_topk_blocks), and a re-saved config makes the model emit garbage.

Serving engines. The vendor's own SGLang build serves Unlimited-OCR-2 in BF16 only: its MoE path flattens the experts' .weight tensors and cannot load quantized experts. vLLM has no v2 support yet. For local use there are GGUF files: AtomicChat/Unlimited-OCR-2-GGUF.

License

MIT, inherited from baidu/Unlimited-OCR-2. Built by AtomicChat.