openbmb
/

MiniCPM-OCR-1.0

@@ -1,133 +0,0 @@
----
-license: apache-2.0
-pipeline_tag: image-text-to-text
-tags:
-- minicpm-ocr
-- ocr
-- document-parsing
-- multimodal
-- vision-language
----
-# MiniCPM-OCR 1.0
-**MiniCPM-OCR 1.0** is a compact end-to-end document OCR model. It reads a page (or a cropped
-document element) and returns its text in reading order, with dedicated output modes for tables
-and display formulas, so a layout model + MiniCPM-OCR 1.0 is enough to rebuild a page as Markdown.
-- **1.3B parameters (778M language + 522M vision), bf16, 2.6 GB** — a single 2.6 GB checkpoint that
-  fits comfortably on one consumer GPU.
-- **4x visual-token compression.** Visual tokens are compressed early inside the vision encoder
-  (insertion at layer 6, in the spirit of [LLaVA-UHD v4](https://github.com/THUMAI-Lab/LLaVA-UHD-v4)),
-  and the projector emits 4x-compressed tokens. This release ships **4x only**, which is also the
-  configuration used for the GGUF build.
-- **Element-level prompts.** The same checkpoint handles plain text, tables and formulas; the task is
-  selected by the prompt (`Text Recognition:` / `Table Recognition:` / `Formula Recognition:`), so it
-  slots directly behind any document layout model.
-- **Slicing for large pages.** `scale_resolution = 448`, `patch_size = 14`, up to 9 slices per page,
-  with the 4x compression keeping the prompt budget low on dense A4/letter pages.
-- **Hybrid text backbone.** 24 layers built from 18 linear-attention and 6 full-attention layers
-  (interval 4), 262144-token context, tied embeddings, no MTP head.
-> **Transformers support:** `model_type` is `minicpm_ocr` and is **not** part of any released
-> `transformers` version yet (checked up to 5.20.2), and this repository intentionally ships **no
-> modeling code / no `auto_map`**. `AutoModelForImageTextToText` / `AutoProcessor` will therefore not
-> load this checkpoint. Use one of the runtimes below.
-## Model architecture
-| Component | Value |
-|---|---|
-| Total parameters | 1.30B (bfloat16) |
-| Language model | Qwen3.5-style hybrid stack, 24 layers (18 linear + 6 full attention), hidden 1024, 8 heads / 2 KV heads, head_dim 256, partial rotary 25% |
-| Context length | 262144 |
-| Vocabulary | 248100 (tied embeddings) |
-| Vision encoder | SigLIP-style ViT, 27 layers, hidden 1152, patch 14, attention 16 heads, early compression at layer 6 |
-| Visual tokens | 4x downsample (`projector scale_factor = 2`) |
-| Preprocessing | `scale_resolution = 448`, `max_slice_nums = 9`, slice mode on, mean/std 0.5 |
-| Special tokens | `image_token_id = 248056`, `video_token_id = 248057`, `bos = 248045`, `eos = 248046` |
-## Supported runtimes
-| Runtime | Status for this release |
-|---|---|
-| **llama.cpp / GGUF** | Supported — see [MiniCPM-OCR-1.0-gguf](https://huggingface.co/openbmb/MiniCPM-OCR-1.0-gguf) (4x mmproj). Requires a llama.cpp build that includes MiniCPM-OCR 1.0 support (`projector_type = minicpm_ocr`). |
-| **SGLang** | Supported — requires SGLang with MiniCPM-OCR 1.0 support. |
-| **vLLM** | Support is being upstreamed; **not** part of this release. |
-| **Transformers** | Not supported yet (see note above). |
-## Quick start
-### SGLang
-```bash
-python -m sglang.launch_server \
-  --model-path openbmb/MiniCPM-OCR-1.0 \
-  --host 127.0.0.1 --port 30000 \
-  --mem-fraction-static 0.45
-```
-```python
-from openai import OpenAI
-client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")
-resp = client.chat.completions.create(
-    model="openbmb/MiniCPM-OCR-1.0",
-    messages=[{
-        "role": "user",
-        "content": [
-            {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_PAGE>"}},
-            {"type": "text", "text": "Text Recognition:"},
-        ],
-    }],
-    max_tokens=8192,
-    temperature=0.0,
-    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
-)
-print(resp.choices[0].message.content)
-```
-### llama.cpp
-```bash
-llama-mtmd-cli \
-  -m ggml-model-f16.gguf \
-  --mmproj mmproj-model-f16-4x.gguf \
-  --image page.png \
-  -p "Text Recognition:" \
-  --jinja -ngl 99 -c 8192 --temp 0
-```
-Note that the GGUF file already hard-disables thinking inside its embedded chat template — no extra
-flag is needed, and `--reasoning on` cannot re-enable it. See the GGUF repository for details.
-## Prompts
-The checkpoint is driven by a task prefix, and the prefix must match the element being cropped:
-| Element | Prompt | Output |
-|---|---|---|
-| Text block / paragraph / title / page | `Text Recognition:` | plain text |
-| Table | `Table Recognition:` | HTML |
-| Display formula | `Formula Recognition:` | LaTeX |
-| Inline formula | `Text Recognition:` | LaTeX |
-A typical page pipeline runs a layout model (e.g. PP-DocLayout) first, crops each element, dispatches
-the prompt above by element type, then concatenates the results in reading order into Markdown.
-## Evaluation
-End-to-end OmniDocBench v1.6 numbers for this release are **being finalized** and will be published
-here. No numbers are reported yet on purpose — please do not cite any third-party reproduction as
-this checkpoint's result.
-## Model weights
-| Repository | Contents |
-|---|---|
-| `openbmb/MiniCPM-OCR-1.0` (this repo) | bf16 weights, tokenizer, chat template, preprocessor configs |
-| [`openbmb/MiniCPM-OCR-1.0-gguf`](https://huggingface.co/openbmb/MiniCPM-OCR-1.0-gguf) | F16 GGUF + 4x mmproj for llama.cpp |
-## License
-Apache-2.0.