DuyTa/Peppercorn_Base

Peppercorn Base (Hạt Tiêu), 0.31B

Vietnamese and English page OCR in 310.8M parameters: a page image in, Markdown out (tables as HTML, formulas as LaTeX), optionally with a labelled bounding box per block. Peppercorn Base is LightOnOCR-3-0.8B pruned to 39% of its size and retrained with distillation; it is the supervised base checkpoint (126,000 steps) of Peppercorn v3, before any reinforcement learning. The name comes from the Vietnamese saying nhỏ nhưng là hạt tiêu, "small but a peppercorn": small but mighty.

What it is good at: Vietnamese born-digital pages and scans (text edit distance on SEA-Vision VI 0.079, level with the 4B teacher and better than its 0.8B parent), tables (TEDS on OmniDocBench-EN above both LightOnOCR-3 models), and block grounding. What it is weaker at: formulas, phone photographs, and long English pages, where it loops more often than its parent (see Limitations).

Architecture

The checkpoint is a plain Qwen3_5ForConditionalGeneration (Qwen3.5-VL layout): transformers>=5 and stock vLLM (>=0.29) load it as is, no plugin.

Part	Peppercorn Base	LightOnOCR-3-0.8B (parent)
Vision	8 of 12 ViT blocks (kept 0, 1, 2, 5, 8, 9, 10, 11), patch 16, 2×2 merge; 72.2M	12 blocks
Decoder	12 of 24 hybrid layers: 8 Gated DeltaNet + 4 gated full attention (kept 0, 1, 2, 6, 7, 10, 11, 14, 15, 16, 22, 23); hidden 1,024; MLP 2,048 of 3,584 neurons	24 layers
Vocabulary	48,033 tokens (kept by frequency on the training labels) + 15 HTML table tokens + 1,001 coordinate tokens <c0>…<c1000>; tied embeddings	248k
Total	310.8M	~0.85B

student_meta.json lists the kept layers, ViT blocks, MLP neurons and the student-to-parent vocabulary map (vocab_map.json).

Training (summary)

1. Pruning of LightOnOCR-3-0.8B. Vocabulary: a merge-closed byte-level BPE subset grown by Vietnamese + English frequency until 99.5% held-out coverage in each language. Decoder layers: greedy removal by KL(full ‖ pruned) on calibration pages, with quotas per layer type. MLP neurons: mean absolute activation (Minitron style). 4 of 12 ViT blocks dropped. Weights are slices of the parent's tensors, nothing else is transformed.
2. ViT heal: 3,000 steps that train only the vision tower (the decoder frozen) so the shortened encoder feeds the decoder what it expects.
3. SFT + KD, 126,000 steps × 16 pages, Muon (LR 3·10⁻⁵, ViT ×0.1), warmup-stable-decay, EMA: 0.5·cross-entropy + 0.5·KL to the labelling teacher's stored top-20 log-probabilities (LightOnOCR-3-4B or 0.8B, mapped to the student vocabulary), on pages that have them. Both modes are trained (45% of samples in grounding mode). Pages at a 1,540 or 2,048 px long edge.

Data: 1.54M page images, 55% Vietnamese and 45% English: Vietnamese web PDFs and scans, government documents, slides, forms, exams, math, charts, receipts, synthetic tables, handwriting lines; English PDFs (PDFA, IDL, DocLayNet, olmOCR-mix), arXiv-style math. Labels come from LightOnOCR-3-4B and 0.8B, from the source datasets, or are exact (rendered synthetic pages). About 1% of the mix is a small set of internal Vietnamese administrative documents, which is not released. Evaluation documents and hosts were removed from training (image hashes, document ids, text shingles).

Results

vLLM, greedy, 2,048 px long edge, max_tokens 4,096, loop guard applied. OmniDocBench-style suite score = mean of 100·(1 − text edit), 100·TEDS and 100·CDM; olmOCR-Bench = test pass rate. MDPBench = the 159 digital + photographed pages of each language; SEA-Vision VI = 1,645 Vietnamese pages.

Suite	Peppercorn Base (0.31B)	LightOnOCR-3-0.8B	LightOnOCR-3-4B
OmniDocBench-EN	76.9	83.2	82.5
olmOCR-Bench	63.4	74.9	74.1
MDPBench EN	55.9	71.3	71.7
MDPBench VI	69.1	84.4	85.6
SEA-Vision VI	89.2	91.0	91.6
Mean of 5	70.9	81.0	81.1

Components:

	Peppercorn Base	LightOnOCR-3-0.8B	LightOnOCR-3-4B
OmniDocBench-EN text edit ↓	0.080	0.053	0.059
OmniDocBench-EN table TEDS ↑	0.637	0.591	0.588
OmniDocBench-EN formula CDM ↑	0.751	0.959	0.946
SEA-Vision VI text edit ↓	0.079	0.089	0.076
SEA-Vision VI table TEDS ↑	0.873	0.892	0.891

Grounding mode against human boxes (Hungarian-matched, class-agnostic F1 at IoU ≥ 0.5; MDPBench: the 40 digital pages per language):

	SEA-Vision VI	MDPBench VI	MDPBench EN
Peppercorn Base	0.788	0.762	0.616
LightOnOCR-3-0.8B	0.746	0.757	0.619
LightOnOCR-3-4B	0.779	0.779	0.670

Serving, one RTX PRO 6000 (96 GB), vLLM 0.29, bf16, greedy, 300 pages at a 1,540 px long edge with 256 in flight: 15.8 pages/s (LightOnOCR-3-0.8B 11.5, 4B 3.7); single-page median latency 1.17 s.

Usage

pip install "vllm>=0.29" openai pillow
bash examples/serve.sh                                   # http://127.0.0.1:8000/v1
python examples/client.py page.png                       # Markdown
python examples/client.py --mode grounding page.png      # Markdown with block boxes

With transformers (>=5):

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

proc = AutoProcessor.from_pretrained("DuyTa/Peppercorn_Base")
model = AutoModelForImageTextToText.from_pretrained("DuyTa/Peppercorn_Base", dtype=torch.bfloat16).cuda().eval()
img = Image.open("page.png").convert("RGB"); img.thumbnail((2048, 2048))
msgs = [{"role": "user", "content": [{"type": "image"}]}]       # + {"type": "text", "text": "grounding"} for boxes
text = proc.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False)
inputs = proc(text=[text], images=[img], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
print(proc.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The prompt is the page image alone (transcription) or the image plus the text grounding. Thinking must be off (serve.sh sets enable_thinking: false). Send pages at a 2,048 px longest edge or less: the processor accepts any size, and a very large scan otherwise becomes tens of thousands of image tokens.

Grounding output: each block starts with ![class]<cX1><cY1><cX2><cY2> (corners on a 0–1000 grid of the page), followed by its Markdown. Classes: text, title, list, table, formula, image (with a short description), chart, caption, header, footer, page_number, footnote, header_image, footer_image, aside_text, code.

![title]<c49><c2><c969><c102> # BƯỚC 3 : ORDER ALACARTE

![title]<c3><c105><c231><c154> ## MÌ - CƠM THỔ

![list]<c134><c168><c503><c477>
- 74 Mì xào thịt heo sốt kiểu Nhật
- 75 Mì xào nấm

![image]<c773><c108><c985><c267> photo of a white and green fish with green and orange vegetables

Limitations

• Loops. With greedy decoding a share of pages repeats a span until max_tokens (MDPBench EN 30%, MDPBench VI 11%, OmniDocBench-EN 5%, SEA-Vision VI 5%; the parent: 15%, 3%, 5%, 2%). Use the loop guard in examples/loop_guard.py (the client applies it), or re-read capped pages with sampling.
• Formulas (CDM 0.75 vs 0.96 for the parent) and phone photographs (MDPBench) are clearly weaker than the parent.
• Only Vietnamese and English were trained; other languages are not supported.
• Labels are largely model-generated (LightOnOCR-3), so the model inherits their conventions and errors.

Licence

CC BY 4.0. The parent model, LightOnOCR-3-0.8B, is released under Apache-2.0 (its notice applies to the weights derived from it); the training datasets keep their own licences.