README.md

---
library_name: transformers
license: bsd-3-clause
base_model:
- PaddlePaddle/PaddleOCR-VL-1.5
pipeline_tag: image-text-to-text
tags:
- PaddleOCR-VL-1.5
- OCR
- VLM
language:
- zh
- en
---

# PaddleOCR-VL-1.5

This version of PaddleOCR-VL-1.5 has been converted to run on the WaveMatrix NPU using **w4a16** quantization.

Compatible with Pulsar2 version: 5.0

## Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo:

- https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5

- [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html)

- [WaveMatrix NPU HOST LLM Runtime](https://github.com/WaveMatrix/ax-llm)

## Support Platform

- WM9955
  - WM9955 DEMO Board
  - [M.2 Accelerator card](https://docs.m5stack.com/zh_CN/ai_hardware/LLM-8850_Card)

**Image Process**
| Chips | input size | image num | ViT encoder | TTFT (640 tokens) | Decode speed | CMM | Flash |
|--|--|--|--|--|--|--|--|
| WM9955 | 576x768 | 1 | 1685.554 ms | 361.8 ms | 44.6 tokens/sec | TBD | TBD |

## How to use

## Install axllm

```shell
git clone -b axllm https://github.com/WaveMatrix/ax-llm.git
cd ax-llm
./install.sh
```

## Model Download (Hugging Face)
Create directory then enter:

```shell
mkdir -p WaveMatrix/PaddleOCR-VL-1.5
cd WaveMatrix/PaddleOCR-VL-1.5
hf download WaveMatrix/PaddleOCR-VL-1.5 --local-dir .

# Content
tree -L 1
.
|-- README.md
|-- assets
|-- config.json
|-- model.embed_tokens.weight.bfloat16.bin
|-- paddleocr_vl_p128_l0_together.axmodel
...
|-- paddleocr_vl_p128_l17_together.axmodel
|-- paddleocr_vl_post.axmodel
|-- post_config.json
|-- python
|-- tokenizer.model
|-- vision_cache
`-- vit_576x768.axmodel
```

## Inference with WM9955 Host, such as WM9955 DEMO Board or M.2 card

### Run the model via CLI

```shell
axllm run WaveMatrix/PaddleOCR-VL-1.5/
```

After `prompt`, `image >>` will pop up：
- Just press "Enter" will go into "text only" mode
- Or input the path of the image to start the OCR process

Example for image mode:

![TestImage](assets/IMG_0462.JPG)

```shell
root@WM9955 # axllm run WaveMatrix/PaddleOCR-VL-1.5/
20:45:21.515 INF Init:218 | LLM init start
tokenizer_type = 0
 95% | ##############################   |  20 /  21 [1.97s<2.07s, 10.13 count/s] init post axmodel ok,remain_cmm(4330 MB)
20:45:23.490 INF Init:368 | max_token_len : 2047
20:45:23.490 INF Init:371 | kv_cache_size : 256, kv_cache_num: 2047
20:45:23.490 INF Init:374 | prefill_token_num : 128
20:45:23.490 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
20:45:23.490 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
20:45:23.490 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
20:45:23.490 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
20:45:23.490 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
20:45:23.490 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 640
20:45:23.490 INF Init:384 | prefill_max_token_num : 640
20:45:23.490 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  21 /  21 [1.98s<1.98s, 10.62 count/s] embed_selector init ok
20:45:24.028 INF Init:453 | PaddleOCRVL: encoder input nSize=5334336 -> eff_nSize=1333584 (float32 input)
20:45:24.029 WRN Init:469 | Qwen-VL vision size override: cfg=448x448 bytes=602112, model_input_bytes=5334336 -> 756x588 (factor-search).
20:45:24.029 INF Init:661 | PaddleOCR-VL token ids: vision_start=101305 image_pad=100295 video_pad=100295
20:45:24.029 INF Init:686 | VisionModule init ok: type=PaddleOCRVL, tokens_per_block=567, embed_size=1024, out_dtype=fp32
20:45:24.029 WRN Init:695 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
20:45:24.031 INF load_config:282 | load config:
20:45:24.031 INF load_config:282 | {
20:45:24.031 INF load_config:282 |     "enable_repetition_penalty": false,
20:45:24.031 INF load_config:282 |     "enable_temperature": false,
20:45:24.031 INF load_config:282 |     "enable_top_k_sampling": false,
20:45:24.031 INF load_config:282 |     "enable_top_p_sampling": false,
20:45:24.031 INF load_config:282 |     "penalty_window": 20,
20:45:24.031 INF load_config:282 |     "repetition_penalty": 1.0,
20:45:24.031 INF load_config:282 |     "temperature": 0.6,
20:45:24.031 INF load_config:282 |     "top_k": 1,
20:45:24.031 INF load_config:282 |     "top_p": 0.9
20:45:24.031 INF load_config:282 | }
20:45:24.031 INF Init:448 | LLM init ok
Commands:
  /q, /exit  Exit
  /reset     Reset kvcache
  /dd        Delete a round of conversation
  /pp        Print history
Ctrl+C: Stop the current ongoing generation
VLM enabled: after each prompt, input image path (empty = text-only). Use "video:<frames_dir>" for video.
----------------------------------------
prompt >> OCR:
image >> /Path/To/Your/WaveMatrix/PaddleOCR-VL-1.5/assets/IMG_0462.JPG
20:45:30.031 INF EncodeForContent:1058 | PaddleOCRVL pixel_values bytes=1333584 min=0 max=255 (w=756 h=588 ps=14)
20:45:31.726 INF EncodeForContent:1102 | vision cache store: /Path/To/Your/WaveMatrix/PaddleOCR-VL-1.5/assets/IMG_0462.JPG
20:45:31.760 INF SetKVCache:747 | prefill_grpid:6 kv_cache_num:640 precompute_len:0 input_num_token:596
20:45:31.760 INF SetKVCache:749 | current prefill_max_token_num:640
20:45:31.760 INF SetKVCache:752 | first run
20:45:31.761 INF Run:805 | input token num : 596, prefill_split_num : 5
20:45:31.761 INF Run:845 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
20:45:31.761 INF Run:868 | prefill indices shape: p=0 idx_elems=384 idx_rows=3 pos_rows=3
20:45:31.815 INF Run:845 | prefill chunk p=1 history_len=128 grpid=2 kv_cache_num=128 input_tokens=128
20:45:31.816 INF Run:868 | prefill indices shape: p=1 idx_elems=384 idx_rows=3 pos_rows=3
20:45:31.873 INF Run:845 | prefill chunk p=2 history_len=256 grpid=3 kv_cache_num=256 input_tokens=128
20:45:31.873 INF Run:868 | prefill indices shape: p=2 idx_elems=384 idx_rows=3 pos_rows=3
20:45:31.937 INF Run:845 | prefill chunk p=3 history_len=384 grpid=4 kv_cache_num=384 input_tokens=128
20:45:31.937 INF Run:868 | prefill indices shape: p=3 idx_elems=384 idx_rows=3 pos_rows=3
20:45:32.006 INF Run:845 | prefill chunk p=4 history_len=512 grpid=5 kv_cache_num=512 input_tokens=84
20:45:32.006 INF Run:868 | prefill indices shape: p=4 idx_elems=384 idx_rows=3 pos_rows=3
20:45:32.088 INF Run:1010 | ttft: 327.20 ms
James Landay-VR
14175

20:45:32.374 NTC Run:1132 | hit eos,avg 38.47 token/s
20:45:32.374 INF GetKVCache:721 | precompute_len:597, remaining:43
```

### Start the service (OpenAI API compatiable)

```sh
axllm serve WaveMatrix/PaddleOCR-VL-1.5/
```

Example journal:

```shell
root@WM9955 # axllm serve WaveMatrix/PaddleOCR-VL-1.5/
20:47:54.027 INF Init:218 | LLM init start
tokenizer_type = 0
 95% | ##############################   |  20 /  21 [1.96s<2.05s, 10.22 count/s] init post axmodel ok,remain_cmm(4330 MB)
20:47:55.983 INF Init:368 | max_token_len : 2047
20:47:55.983 INF Init:371 | kv_cache_size : 256, kv_cache_num: 2047
20:47:55.983 INF Init:374 | prefill_token_num : 128
20:47:55.983 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
20:47:55.983 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
20:47:55.983 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
20:47:55.983 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
20:47:55.983 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
20:47:55.983 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 640
20:47:55.983 INF Init:384 | prefill_max_token_num : 640
20:47:55.983 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  21 /  21 [1.96s<1.96s, 10.72 count/s] embed_selector init ok
20:47:56.526 INF Init:453 | PaddleOCRVL: encoder input nSize=5334336 -> eff_nSize=1333584 (float32 input)
20:47:56.526 WRN Init:469 | Qwen-VL vision size override: cfg=448x448 bytes=602112, model_input_bytes=5334336 -> 756x588 (factor-search).
20:47:56.526 INF Init:661 | PaddleOCR-VL token ids: vision_start=101305 image_pad=100295 video_pad=100295
20:47:56.526 INF Init:686 | VisionModule init ok: type=PaddleOCRVL, tokens_per_block=567, embed_size=1024, out_dtype=fp32
20:47:56.526 WRN Init:695 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
20:47:56.528 INF load_config:282 | load config:
20:47:56.528 INF load_config:282 | {
20:47:56.528 INF load_config:282 |     "enable_repetition_penalty": false,
20:47:56.528 INF load_config:282 |     "enable_temperature": false,
20:47:56.528 INF load_config:282 |     "enable_top_k_sampling": false,
20:47:56.528 INF load_config:282 |     "enable_top_p_sampling": false,
20:47:56.528 INF load_config:282 |     "penalty_window": 20,
20:47:56.528 INF load_config:282 |     "repetition_penalty": 1.0,
20:47:56.528 INF load_config:282 |     "temperature": 0.6,
20:47:56.528 INF load_config:282 |     "top_k": 1,
20:47:56.528 INF load_config:282 |     "top_p": 0.9
20:47:56.528 INF load_config:282 | }
20:47:56.528 INF Init:448 | LLM init ok
Starting server on port 8000 with model 'WaveMatrix/PaddleOCR-VL-1.5'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
  GET  http://10.168.232.217:8000/health
  GET  http://10.168.232.217:8000/v1/models
  POST http://10.168.232.217:8000/v1/chat/completions
  GET  http://172.17.0.1:8000/health
  GET  http://172.17.0.1:8000/v1/models
  POST http://172.17.0.1:8000/v1/chat/completions
Aliases:
  GET  http://127.0.0.1:8000/models
  POST http://127.0.0.1:8000/chat/completions
  GET  http://10.168.232.217:8000/models
  POST http://10.168.232.217:8000/chat/completions
  GET  http://172.17.0.1:8000/models
  POST http://172.17.0.1:8000/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: WaveMatrix/PaddleOCR-VL-1.5
```

## Python Inference (Optional)

Python inference script is under `python/`

```shell
python3 python/infer_axmodel.py \
  --hf_model ./python/paddleocr_vl_1-5_tokenizer \
  --axmodel_path . \
  --vit_model_path ./vit_576x768.axmodel \
  --image_path ./assets/IMG_0462.JPG \
  --task ocr
```

Task type supported(`--task`):
- `ocr` - General text recognition
- `table` - Table recognition
- `chart` - Chart recognition
- `formula` - Formula cognition
- `spotting` - Text spotting
- `seal` - Seal recognition

## End-to-End Metrics (WM9955N)

| Metric | Value |
| -- | -- |
| Max TTFT (640 tokens) | 361.8 ms |
| Decode speed | 44.6 tokens/s |
| ViT latency (576x768) | 1685.554 ms |

### Subgraph Latency

| Stage | Subgraph | Latency |
| -- | -- | -- |
| Prefill | g1 | 2.551 ms |
| Prefill | g2 | 2.883 ms |
| Prefill | g3 | 3.158 ms |
| Prefill | g4 | 3.413 ms |
| Prefill | g5 | 3.795 ms |
| Prefill | g6 | 4.007 ms |
| Decode | g0 | 0.949 ms |
| Post-process | - | 5.313 ms |
| ViT | - | 1685.554 ms |


## Discussion