Vishva007/dots.mocr-NVFP4

dots.mocr NVFP4 AutoRound Quantized

This repository provides production-grade NVFP4 quantized weights for dots-studio/dots.mocr using Intel AutoRound.

---

Quantization Details

The quantization recipe was tuned for high accuracy and long-context multimodal parsing while preventing degradation of visual features:

• Bits / Weight Type: NVFP4
• Group Size: 16 (Symmetric: True)
• Vision Tower Preservation: quant_nontext_module=False (vision encoder preserved in full precision to retain document grounding accuracy)
• Tuning Recipe: 1,200 iterations, 512 samples, sequence length 4096 with torch.compile acceleration.

Calibration Dataset

The quantization process was calibrated using a curated, mixed dataset of 512 total samples with a sequence length of 4096 tokens. The calibration mixture is designed to balance document grounding accuracy across English, multilingual, and Indic text:

• English Document OCR (allenai/olmOCR-mix-1025): 224 primary samples filtered for English natural_text, supplemented by a 39-sample backfill.
• Multilingual Document OCR Mix (allenai/olmOCR-mix-1025): 144 samples utilizing general multilingual document configurations.
• Indic Languages (CohereForAI/aya_dataset): 12 samples per language spanning 12 Indian languages (Hindi, Tamil, Telugu, Malayalam, Kannada, Marathi, Bengali, Gujarati, Punjabi, Odia, Urdu, and Assamese).

---

Deployment with vLLM

KV Cache Precision Warning: Always keep --kv-cache-dtype set to auto or bfloat16. Do not use low-precision KV cache formats (such as fp8), as doing so will cause the model to fail to generate any output or return empty responses. For high-throughput document parsing, serve the AutoRound variant directly with vLLM:

vllm serve Vishva007/dots.mocr-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --chat-template-content-format string \
  --dtype bfloat16 \
  --kv-cache-dtype auto \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill

Benchmark Highlights (Original Base Model)

• olmOCR-Bench: 83.9 Overall
• OmniDocBench (v1.5): 0.031 TextEdit / 0.029 Read OrderEdit
• Native Structured Output: Direct image-to-Markdown, HTML tables, LaTeX formulas, and SVG generation.

---

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.15

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.15 (CUDA 13.2)	13.2	vishva123/cuda-13.2-pytorch-2.15-runpod	0rsw686010
PyTorch 2.15 (CUDA 13.4)	13.4	vishva123/cuda-13.4-pytorch-2.15-runpod	0s0rhdfaqy

PyTorch 2.14

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.14 (CUDA 12.6)	12.6	vishva123/cuda-12.6-pytorch-2.14-runpod	d7lxsa4w9m
PyTorch 2.14 (CUDA 13.0)	13.0	vishva123/cuda-13.0-pytorch-2.14-runpod	yk0y6j6rpg
PyTorch 2.14 (CUDA 13.2)	13.2	vishva123/cuda-13.2-pytorch-2.14-runpod	gsp4gwx0nw

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.13 (CUDA 12.6)	12.6	vishva123/cuda-12.6-pytorch-2.13-runpod	gmlupxnxfk
PyTorch 2.13 (CUDA 13.0)	13.0	vishva123/cuda-13.0-pytorch-2.13-runpod	y3j8xvk4f4
PyTorch 2.13 (CUDA 13.2)	13.2	vishva123/cuda-13.2-pytorch-2.13-runpod	vigpissn5w

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.12 (CUDA 12.6)	12.6	vishva123/cuda-12.6-pytorch-2.12-runpod	ctmz86zmf0
PyTorch 2.12 (CUDA 13.0)	13.0	vishva123/cuda-13.0-pytorch-2.12-runpod	qjko5yiwzi
PyTorch 2.12 (CUDA 13.2)	13.2	vishva123/cuda-13.2-pytorch-2.12-runpod	ifg6xmye0f

---

Acknowledgements

• Base model developed by dots-studio.
• Quantized using Intel AutoRound.