| --- |
| license: agpl-3.0 |
| base_model: meta-models/Muse-Glimmer-30B |
| pipeline_tag: image-text-to-text |
| language: |
| - ja |
| library_name: transformers |
| tags: |
| - ocr |
| - document-transcription |
| - japanese |
| - vllm |
| --- |
| |
| # Wasabi-30B |
|
|
| Wasabi-30B reads a Japanese document page image and writes the page as Markdown: |
|
|
| - full text in reading order; |
| - tables as HTML `<table>`; |
| - math as LaTeX, `$...$` inline and `$$...$$` for display; |
| - ruby (furigana) kept inline as `<ruby>base<rt>reading</rt></ruby>`. |
|
|
| ## Model details |
|
|
| - Fine-tune of Muse-Glimmer-30B (https://huggingface.co/meta-models/Muse-Glimmer-30B, revision |
| `97c77dff50b2797bcc558fa2d909761dbc575c59`). |
| - A LoRA adapter trained for Japanese page transcription, merged into BF16 weights. |
| - Two safetensors shards. |
|
|
| ## Files in this repository |
|
|
| | path | what it is | |
| |---|---| |
| | `model-00001-of-00002.safetensors`, `model-00002-of-00002.safetensors`, `model.safetensors.index.json` | merged BF16 weights | |
| | `config.json`, `generation_config.json`, `processor_config.json` | model and processor configuration | |
| | `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` | tokenizer and chat template | |
| | `inference/` | the page reader package (`wasabi_read`, AGPL-3.0) and its two prompts | |
| | `LICENSE`, `NOTICE`, `notices/` | licence texts: this model (AGPL-3.0), the base model (Apache-2.0) and its usage policy | |
| | `SHA256SUMS` | checksums of every file | |
|
|
| ## Inference |
|
|
| The package reads a folder of page images (`.jpg`, `.jpeg`, `.png`) and writes one `.md` file per page. |
|
|
| pip install -e "./inference[dewarp]" |
| |
| Python 3.10+. The reader flattens every page locally with UVDoc on CPU, so torch and OpenCV are required in every |
| mode, including endpoint mode. |
|
|
| Local serving needs vLLM with muse_glimmer support, built from commit `6adad08767583f52eb4d2122111af0bf638ed5e6` |
| (version `0.26.1rc1.dev769`). Released vLLM 0.26.0 has no muse_glimmer model, so it cannot serve this model. A |
| source build needs a CUDA toolchain: |
|
|
| pip install "vllm @ git+https://github.com/vllm-project/vllm.git@6adad08767583f52eb4d2122111af0bf638ed5e6" "transformers==5.15.0" |
| |
| Then, with the Python of that vLLM environment, apply the one-file processor fix. Without it, the muse_glimmer |
| processor of this vLLM commit rejects the `max_image_tokens` setting that every server below is started with, and the |
| server does not start. The script edits one file, checks its sha256 before and after, and can undo the edit |
| (`--check`, `--revert`). A server you start yourself (endpoint mode, direct model use) needs it too. |
| |
| python inference/patch_vllm_max_image_tokens.py --apply |
| |
| An endpoint run needs no local vLLM. |
| |
| Tested with Python 3.12, vLLM `0.26.1rc1.dev769` (commit 6adad087), transformers 5.15.0, torch 2.13.0, |
| opencv-python-headless 5.0.0.93, numpy 2.3.5 and Pillow 13.0.0. |
| |
| Wasabi-30B is the only checkpoint the reader needs: it checks the page orientation and reads the page. |
| |
| Local vLLM (the package starts and stops the servers itself: first the orientation check, then the reader, both from |
| the same weights): |
| |
| python inference/wasabi_read.py pages/ out/ --engine vllm --model-dir /models/Wasabi-30B --tensor-parallel-size 1 --whole-upscale-tokens 8192 |
|
|
| Against OpenAI-compatible servers you already run, both serving Wasabi-30B: the reader started with |
| `--mm-processor-kwargs '{"max_image_tokens": 8192}'`, the orientation check with |
| `--mm-processor-kwargs '{"max_image_tokens": 4096}'`, and both with `--reasoning-parser muse_glimmer`. Name what each |
| server serves with `--endpoint-revision` and `--orient-endpoint-revision`; the addresses and revisions are part of the |
| output identity, so a rerun against another server is refused instead of mixed into the same output directory. |
|
|
| python inference/wasabi_read.py pages/ out/ --engine endpoint --endpoint-url http://gpu-host:8000 --endpoint-revision wasabi-30b-v1.0 --orient-endpoint-url http://gpu-host:8001 --orient-endpoint-revision wasabi-30b-v1.0 |
| |
| `--plan` prints the stages and exits without loading a model. |
|
|
| The API key, the UVDoc cache location and offline UVDoc use are described under |
| [Configuration](inference/README.md#configuration). |
|
|
| ### Direct model use |
|
|
| This sends the page image as it is: no orientation check, no flattening, no enlargement and no retries. It is a basic |
| transcription path, separate from the reader above. It uses the vLLM build and the processor fix from the Inference |
| section. |
|
|
| pip install openai |
| vllm serve /models/Wasabi-30B --served-model-name wasabi --max-model-len 36864 \ |
| --mm-processor-kwargs '{"max_image_tokens": 8192}' --reasoning-parser muse_glimmer |
| |
| Then send one page with the OpenAI-compatible API, using the recommended decoding (greedy, repetition penalty 1.10, |
| seed 42, up to 24,576 new tokens, low reasoning strength): |
|
|
| import base64 |
| from openai import OpenAI |
| |
| client = OpenAI(base_url="http://localhost:8000/v1", api_key="none") |
| png = base64.b64encode(open("page.png", "rb").read()).decode() |
| prompt = open("inference/wasabi_read/prompts/draft.txt", encoding="utf-8").read() |
| r = client.chat.completions.create( |
| model="wasabi", |
| messages=[{"role": "user", "content": [ |
| {"type": "image_url", "image_url": {"url": "data:image/png;base64," + png}}, |
| {"type": "text", "text": prompt}]}], |
| temperature=0.0, |
| seed=42, |
| max_tokens=24576, |
| extra_body={"repetition_penalty": 1.10, |
| "repetition_detection": {"max_pattern_size": 512, "min_pattern_size": 4, "min_count": 16}, |
| "chat_template_kwargs": {"reasoning_strength": "low", "current_date": "2026-10-16"}}, |
| ) |
| print(r.choices[0].message.content) |
| |
| The transcription prompt is the file `inference/wasabi_read/prompts/draft.txt`. The reader pins `current_date` so that the output does |
| not depend on the day you run it, and `repetition_detection` stops a looping answer early. |
|
|
| `generation_config.json` keeps the base model's sampling defaults; pass `temperature=0` (as above) for the |
| recommended greedy transcription. |
|
|
| ## What the reader does |
|
|
| Per page: |
|
|
| 1. Orientation check: Wasabi-30B answers a four-way upright probe at 4,096 image tokens; a turn is applied only when |
| the margin is at least 5.0. |
| 2. Flattening with UVDoc on CPU. The UVDoc code and weights are fetched at first use from a pinned commit and checked |
| by sha256 before loading. |
| 3. A whole-page read at 8,192 image tokens, greedy. |
| 4. If the answer is degenerate, a retry ladder at temperatures 0.1, 0.3, 0.5 and 0.8. |
| 5. Output is written per page as `<stem>.md`, atomically. Rerunning the same command resumes: finished pages are |
| skipped. Pages with no usable answer are listed in `out/.wasabi-read/run-report.json`. |
|
|
| ## Benchmark submission |
|
|
| The MDPBench submission includes additional inference components provided to the organizers. The complete inference |
| code is maintained in a [private repository](https://github.com/sionic-ai/Wasabi-30B). |
| The code included here runs independently of that repository. |
|
|
| ## Limitations |
|
|
| - Japanese-focused. Other languages are not a target. |
| - The `transformers` engine has not been measured and is not guaranteed to follow the vLLM path's image size and |
| decoding; use the vLLM or endpoint engine for the behaviour documented here. |
| - Each orientation decision comes from one greedy probe per rotation, so a different serving stack (attention backend, |
| CUDA build, batching) can change the turn on a few pages. |
| - Ruby appears only as the model writes it inline; there is no separate ruby pass. |
| - Photographed pages and very small print can be misread. |
| - The reader package runs on Linux and macOS only (it uses POSIX file locks). Its local vLLM engine starts the servers |
| with the FlashInfer vision-encoder attention backend; to use another backend, serve the model yourself and use |
| endpoint mode. |
|
|
| ## License |
|
|
| - This model and the reader package: GNU Affero General Public License v3.0 (`LICENSE`). |
| - Base model Muse-Glimmer-30B: Apache License 2.0 (`notices/Muse-Glimmer-30B-LICENSE.txt`), together with its |
| publisher's usage policy (`notices/Muse-Glimmer-30B-USAGE_POLICY.md`). |
| - UVDoc code (commit 4c9b82b5): MIT License, Copyright (c) 2023 Tanguy MAGNE. The reader downloads the UVDoc code |
| and its pretrained weights at first use and does not redistribute them. The weights carry no separate licence |
| statement and remain under their upstream terms. |
|
|
| Copyright 2026 Sionic AI Inc. |
|
|
