mlboydaisuke/Unlimited-OCR-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

This model has no row on DeviceMark, the on-device LLM leaderboard.

Unlimited-OCR → Core AI (on-device document OCR)

On-device document → structured-markdown OCR, end-to-end on Apple Core AI. A port of baidu/Unlimited-OCR (3B-A0.5B MoE, MIT): drop a document image, get back markdown — tables as HTML (<table><tr><td>…), formulas as LaTeX, reading order, and <|det|> layout boxes. Japanese + English + multilingual.

Runs on the stock coreai.runtime with no engine patch — the decoder is driven directly on inputs_embeds, so this is a pure-export port (not the static-input-buffer VLM path).

Use it

New to Core AI? Start with CoreAIKit 0.8.0. Follow its requirements and first-run steps for qwen3-0.6b, then open the same release's ChatDemo. The README records the tested OS/SDK and download size; model and device coverage is stated per example.

⚡ One line — run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

let markdown = try await CoreAI.read(documentAt: url, options: .model("unlimited-ocr"))

Every op, one shape — Cookbook.

▶️ Run it (source) — the ReadDoc runner (GUI + CLI, one app for every document-OCR model in the catalog):

git clone --branch 0.8.0 --depth 1 https://github.com/john-rocky/coreai-kit
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-RC.app/Contents/Developer
open -a /Applications/Xcode-27.0.0-RC.app coreai-kit/Examples/ReadDoc/ReadDoc.xcodeproj
# → Run, then pick "Unlimited-OCR" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ReadDoc
swift run -c release readdoc-cli --model unlimited-ocr --image sample.png

Use Xcode build 27A266a from the release's .xcode-pin; adjust the app path if your installation is named differently.

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let reader = try await KitDocReader(catalog: "unlimited-ocr")
let markdown = try await reader.read(imageAt: imageURL)
// markdown: the document as structured text — tables as <table>/<tr>/<td>,
// <|det|> layout boxes, reading order — fully on-device

The take-home is Examples/ReadDoc/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same KitDocReader(catalog:) on the image you pick. One read(imageAt:) call per page; chunk a PDF into page images first. The output keeps the model's structural markup (tables as HTML, formulas as LaTeX, <|det|> boxes) — strip or render it as your app prefers.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit (exact 0.8.0) → product CoreAIKit
• Info.plist: none needed
• Entitlements: none needed
• First run downloads the model — ~4,532 MB (Mac) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

What's exciting (why you'd use it)

• Private OCR: invoices, receipts, contracts, papers, forms never leave the device.
• Structured, not just text: tables → HTML, equations → LaTeX, layout → boxes. RAG-ready ingestion.
• Flat latency: a static-shape decode graph (data-driven KV write + fixed-buffer R-SWA mask) keeps every tensor shape constant, so the runtime compiles once and decode stays flat at 12.7 ms/token (79 tok/s on M4 Max) — no growing-cache recompilation stalls.
• SOTA quality: the source model tops OmniDocBench v1.6 (93.92); this port is byte-faithful to the fp32 reference (decoder 0 flips at the sampled steps; vision encoder cos 1.000000).

Bundles

path	what	dtype	size
vision/unlimited_ocr_vision.aimodel	DeepEncoder (SAM-ViT + CLIP-ViT cascade) → 100 visual tokens	fp16	762 MB
decoder/unlimited_ocr_decoder.aimodel	DeepseekV2 R-SWA MoE decoder, functions prefill + decode sharing one weight set + KV state	sym8	3.2 GB
assets/embed_tokens.f16	token embedding table [129280,1280] (host row-gather)	fp16	316 MB
assets/{image_newline,view_seperator}.f16, assets/prompt_input_ids.i32, assets/recipe.json	arrangement constants + the assembly recipe	—	tiny
tokenizer/	fast tokenizer (tokenizer.json + configs)	—	—

Pipeline (Base mode, 640px)

image → preprocess (pad to 640², normalize mean=std=0.5)
      → vision .aimodel                         → visual tokens [1,100,1280]
      → arrange (10×10 + image_newline per row + view_seperator) → [111,1280]
      → scatter into embed_tokens(prompt_ids)   → prefix [1,115,1280]
      → decoder: prefill(prefix) + greedy decode (no_repeat_ngram=35) → tokens
      → detokenize (keep special tokens)        → markdown

The exact, verified recipe is in assets/recipe.json. Reference implementations (Python end-to-end

• a macOS app, CoreAIOCR, driving the stock runtime) are in the Core AI Model Zoo: conversion/unlimited_ocr/ and apps/CoreAIOCR/.

Notes

• Appropriate input: clean single-page documents (invoice / paper / report / table / formula), roughly square or portrait, with text still legible when fit to 640². Very dense small-text scans (newspaper) want the tiled crop_mode vision export (not included here; Base mode only).
• Prompt is fixed to document parsing (layout + structured extraction).
• Keep the maxTokens of KitDocReader.read(imageAt:maxTokens:) at 1933 or below; the default of 1024, which CoreAI.read uses, fits. The decoder has a fixed KV cache of 2048 positions, shared by the image prompt (115 positions) and the generated tokens (one each), and a page that needs more tokens than maxTokens is cut there. CoreAIKit 0.7.3 and earlier do not stop at the end of the cache, and what they output past it has not been checked. From CoreAIKit 0.8.0, a read whose maxTokens is above 1933 stops at 1933 tokens and returns the text up to there, without an error.
• License: MIT (inherited from baidu/Unlimited-OCR).

Community port — not affiliated with Apple or baidu.

---

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.