---
base_model: zai-org/GLM-OCR
base_model_relation: quantized
library_name: transformers
license: mit
pipeline_tag: image-text-to-text
tags:
  - ocr
  - compressed-tensors
  - vllm
  - quantized
  - image-text-to-text
  - fp8
---
# GLM-OCR · FP8 (dynamic)

8-bit quantization of [`zai-org/GLM-OCR`](https://huggingface.co/zai-org/GLM-OCR) for **vLLM serving**. Quantized by **axteam**.

| | |
|---|---|
| **Base model** | [`zai-org/GLM-OCR`](https://huggingface.co/zai-org/GLM-OCR) |
| **Method** | FP8 (dynamic) |
| **Format** | compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`) |
| **Weights / activations** | W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations |
| **Architecture** | Dense · GLM text backbone + vision encoder · MTP head · document/OCR VLM |
| **Size on disk** | ~2.2 GB |
| **Calibration** | None (data-free) |
| **License** | mit |

## Quantization details
- **Scheme** — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`).
- **Kept in BF16** — the vision tower (`model.visual.*`), the multi-token-prediction layer, token embeddings, the LM head, and all RMSNorms (about 63% of all parameters in this build). Only the language-model backbone's attention & MLP projections are quantized, so the OCR/vision path is untouched.
- Post-training quantization (no retraining). Tokenizer / `chat_template` / processor identical to the original.

## Evaluation (measured)

Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts, text next-token scoring).

| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower = closer) | 1.3726 | 1.3599 (**+0.9%**) |
| Top-1 next-token agreement | 96.5% | 99.4% |

## Requirements

| | |
|---|---|
| **Runtime** | vLLM — the architecture is in vLLM ≥ 0.16.0; this build not run here yet. |
| **vLLM install** | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| **GPU** | NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| **Memory** | Weights: **2.2 GB**. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU. |
| **Transformers (without vLLM)** | `transformers>=5.1.0` + `compressed-tensors>=0.15.0` (text path checked; image input not tested). |
| **trust_remote_code** | not needed |

## Usage (vLLM)

Start an OpenAI-compatible server:

```bash
vllm serve axteam/GLM-OCR-FP8-dynamic --max-model-len 32768
```

Query it:

```bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/GLM-OCR-FP8-dynamic",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "max_tokens": 512
}'
```

Offline inference in Python:

```python
from vllm import LLM, SamplingParams

llm = LLM(model="axteam/GLM-OCR-FP8-dynamic", max_model_len=32768)
params = SamplingParams(max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

# image input (URL or data: URI)
messages = [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "https://example.com/your-image.png"}},
    {"type": "text", "text": "Extract all text from this image."},
]}]
out = llm.chat(messages, params)
print(out[0].outputs[0].text)
```

> Thinking mode is on by default in the chat template; to turn it off pass `chat_template_kwargs={"enable_thinking": False}` (Python `llm.chat(..., chat_template_kwargs=...)`, or `"chat_template_kwargs": {"enable_thinking": false}` in the request body).
> For task-specific OCR prompts (tables, formulas, layout) see the original model card.

## Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a **runtime** choice, independent of the
weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

```bash
vllm serve axteam/GLM-OCR-FP8-dynamic --kv-cache-dtype fp8
```

## License
Quantized derivative of [`zai-org/GLM-OCR`](https://huggingface.co/zai-org/GLM-OCR) (mit). Original weights & processor belong to the upstream authors.

---
## 한국어 요약
[`zai-org/GLM-OCR`](https://huggingface.co/zai-org/GLM-OCR) 를 **FP8 (dynamic)** 로 양자화한 서빙용(8-bit) OCR 모델입니다. 작성: **axteam**.
- **방식** — FP8 E4M3, 가중치 채널별 스케일(정적·보정불필요), 활성화 동적 FP8. 형식 compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`).
- **BF16 유지** — 비전 타워(`model.visual.*`)·MTP 레이어·토큰 임베딩·LM 헤드·모든 norm. 언어모델 백본의 attention·MLP projection 만 양자화(비전/OCR 경로 그대로).
- PTQ(재학습 없음), 토크나이저·chat_template·프로세서 원본 동일.
- **품질(실측)** — BF16 대비 perplexity 1.3726(원본 1.3599, **+0.9%**), top-1 96.5%(원본 99.4%). 내부 영/한 100문항.
- **요구사항** — vLLM ≥0.16.0; GPU Turing(sm75) 이상. FP8 연산은 Ada·Hopper·Blackwell(sm89+), 그 아래는 vLLM 이 가중치 전용 FP8 로 자동 전환; 가중치 2.2 GB + KV 캐시; Transformers 직접 사용: transformers ≥5.1.0(텍스트만 확인).
- **실행** — `vllm serve axteam/GLM-OCR-FP8-dynamic --max-model-len 32768` 로 OpenAI 호환 서버(포트 8000), 또는 Python `LLM(model="axteam/GLM-OCR-FP8-dynamic").chat(...)` (위 예시). thinking 끄기: `chat_template_kwargs={"enable_thinking": False}`. 이미지 입력은 `image_url` 메시지로.
- **KV 캐시 절감(런타임)** — `--kv-cache-dtype fp8`.