Describe the 4-bit model

README.md

@@ -12,9 +12,9 @@ base_model:
   - Helsinki-NLP/opus-mt-es-en
 ---
 
-# pdftool models
+# ocrtool models
 
-ONNX exports of the models used by pdftool, a C++ command-line
+ONNX exports of the models used by ocrtool, a C++ command-line
 tool that converts PDFs to Markdown and translates PDFs in place. They run on ONNX Runtime with no
 Python involved.
 
@@ -23,17 +23,18 @@ weights and stays under the original author's licence.
 
 | Folder | Size | Use | Original weights | Licence |
 | --- | --- | --- | --- | --- |
-| `deepseek-ocr2-fp16/` | 6.6 GB | PDF to Markdown, GPU | [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) | Apache-2.0 |
-| `deepseek-ocr2-fp32/` | 13 GB | PDF to Markdown, CPU | [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) | Apache-2.0 |
+| `deepseek-ocr2-int4/` | 2.9 GB | PDF to Markdown, CPU or GPU (recommended download) | [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) | Apache-2.0 |
+| `deepseek-ocr2-fp16/` | 6.6 GB | PDF to Markdown, GPU, fastest | [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) | Apache-2.0 |
+| `deepseek-ocr2-fp32/` | 13 GB | Full precision; the source for quantizing | [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) | Apache-2.0 |
 | `opus-mt-es-en/` | 0.6 GB | Spanish to English translation | [Helsinki-NLP/opus-mt-es-en](https://huggingface.co/Helsinki-NLP/opus-mt-es-en) | Apache-2.0 |
 
 Each folder is self-contained. Download the whole folder and keep the file names: an `.onnx` file
 loads its weights from the `.onnx.data` file beside it.
 
 ```sh
-hf download baker76/pdftool-models --include "deepseek-ocr2-fp16/*" --local-dir models
-pdftool convert in.pdf -o out.md --model models/deepseek-ocr2-fp16
-pdftool translate in.pdf -o out.pdf --model models/opus-mt-es-en
+hf download baker76/pdftool-models --include "deepseek-ocr2-int4/*" --local-dir models
+ocrtool convert in.pdf -o out.md --model models/deepseek-ocr2-int4
+ocrtool translate in.pdf -o out.pdf --model models/opus-mt-es-en
 ```
 
 ## DeepSeek-OCR-2
@@ -48,9 +49,14 @@ pdftool translate in.pdf -o out.pdf --model models/opus-mt-es-en
 On one test page the fp32 export reproduced the PyTorch model's output token for token; the fp16
 export produced the same text with two bounding-box coordinates differing by 1.
 
+`deepseek-ocr2-int4` is the fp32 export with the decoder's weights stored in 4 bits and the vision
+encoder's in 8 (ONNX Runtime `MatMulNBits`); the two SAM graphs are unchanged. On four test pages its
+text matched the fp16 model's apart from a few formatting differences. It decodes more slowly than
+fp16 on a GPU (about 50 against 80 tokens per second on an RTX 4090).
+
 ## OPUS-MT
 
 `encoder.onnx` and `decoder.onnx` (no cache; the decoder is re-run each step), with the original
 `source.spm`, `target.spm` and `vocab.json`. Logits match PyTorch to within 1e-5.
 
-The scripts that produced these files are in the pdftool repository under `tools/export/`.
+The scripts that produced these files are in the ocrtool repository under `tools/export/`.