clairic-4bit / README.md
lucataco's picture
Clairic, MLX 4-bit, with runtime, server and conversion script
1a48d6b verified
|
Raw History Blame Contribute Delete
13.7 kB
---
license: apache-2.0
base_model: Cloudflare/clairic
base_model_relation: quantized
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
- clairic
- cloudflare
- systemone
- structured-output
- information-extraction
- multimodal
- custom-code
---
# mlx-community/clairic-4bit
An [MLX](https://github.com/ml-explore/mlx) 4-bit conversion of
[`Cloudflare/clairic`](https://huggingface.co/Cloudflare/clairic) for Apple Silicon, with the runtime it
needs and a small HTTP server. The smaller model of the family is at [`mlx-community/clairic-flash-4bit`](https://huggingface.co/mlx-community/clairic-flash-4bit).
**Clairic is a structured-extraction model, not a chat model.** Give it a *state* (text or JSON, plus optional
images or PDFs) and a schema of typed *fields*; it returns one typed value per field with a confidence and a
null probability. It also does numerical estimates and forecasts.
| Field types | Answered by |
|---|---|
| `text`, `number`, `boolean`, `enum`, `object`, and `list` items of those | grammar-constrained decoding, one short prompt per field or list item |
| `date`, `time`, `datetime`, `duration`, `compute`, `estimate`, `forecast`, date lists | a typed head that reads one forward pass |
## Why it needs the included code
`mlx_lm.load` / `mlx_vlm.load` can load the backbone, but they **will not reproduce Clairic**:
- Answers are produced by Cloudflare's request pipeline, not by chatting: it rewrites each field into its own
prompt, constrains decoding with a grammar built from the field's schema (and, for values copied from the
document, from the document itself), scores `null`, retries, and assembles lists and tables.
- Dates, times, durations, computed numbers, estimates and forecasts come from a separate **typed head**
(`clair_head.safetensors`) applied to the backbone's hidden states.
- The upstream runtime is PyTorch with CUDA kernels. This repository keeps Cloudflare's pipeline
(`clairic_runtime/`, Apache-2.0) and replaces its engine with an MLX one (`clairic_runtime/mlx_engine.py`).
Requests and responses have the same format as upstream.
## Quick start
Requires an Apple Silicon Mac and Python 3.11 or later (tested with 3.12). The model uses 16 GB at rest and up to 24 GB on the requests we timed (see [Speed and memory](#speed-and-memory)), so a 32 GB Mac is the practical minimum; 48 GB is comfortable.
```bash
hf download mlx-community/clairic-4bit --local-dir clairic-4bit
cd clairic-4bit
pip install -r requirements.txt
```
`xgrammar` (the grammar engine upstream uses) depends on PyTorch, so `pip` installs `torch`; the model itself
runs entirely in MLX.
```python
from clairic_mlx import load, systemone
model = load(".") # or load("mlx-community/clairic-4bit")
response = systemone(model, {
"state": "Invoice from Northwind Supply. Issued 2026-09-02. Items: Pens; Paper. Total: $1080.00.",
"fields": {
"vendor": {"type": "text", "instructions": "The invoice vendor."},
"total": {"type": "number", "decimals": 2},
"issued": {"type": "date"},
"items": {"type": "list", "items": "text", "instructions": "The purchased item names, in order."},
},
})
print({name: answer["value"] for name, answer in response["answers"].items()})
# {'vendor': 'Northwind Supply', 'total': 1080.0, 'issued': '2026-09-02', 'items': ['Pens', 'Paper']}
```
Images and PDFs (up to four images or pages per request) can be PIL images, file paths or base64 data URIs:
```python
response = systemone(model, {
"state": "Read the attached receipt.",
"images": ["receipt.png"],
"fields": {"merchant": {"type": "text"}, "total": {"type": "number", "decimals": 2}, "date": {"type": "date"}},
})
response = systemone(model, {"documents": ["invoice.pdf"], "fields": {"invoice_number": {"type": "text"}}})
```
Estimates and forecasts return a distribution; `include_quantiles` adds the quantiles:
```python
response = systemone(model, {
"state": "Daily demand for the last seven days: 41, 45, 43, 47, 49, 46, 48.",
"fields": {
"next_day": {"type": "estimate", "min": 0, "max": 100},
"next_three_days": {"type": "forecast", "min": 0, "max": 100, "horizon": 3, "interval": "day"},
},
"options": {"include_quantiles": True},
})
```
Command line and server:
```bash
python clairic_mlx.py --state "Gate B14 closes at 18:20." --fields '{"gate": {"type": "text"}, "closes": {"type": "time"}}'
python clairic_mlx.py --image receipt.png --fields '{"total": {"type": "number", "decimals": 2}}'
python server.py --port 8093
curl localhost:8093/v1/systemone -H 'content-type: application/json' -d '{
"state": "Gate B14 closes at 18:20.",
"fields": {"gate": {"type": "text"}, "closes": {"type": "time"}}}'
```
The server answers one request at a time (a request already fans out into many batched prompts); concurrent
requests wait their turn. Set `CLAIRIC_API_KEY` to require a bearer token. In an async application use
`await model.predict_async(request)`.
The request and response format, the field options (`instructions`, `nullable`, `decimals`, `unit`, `enum`,
`source_only`, `knowledge`, `reference_date`, ...) and the limits (65k-token context, four images or PDF pages,
4 MiB per embedded file) are those of the
[upstream model card](https://huggingface.co/Cloudflare/clairic).
## Fidelity
Tested on 81 requests with known answers: invoices, support tickets, dates and times, computed values, lists and tables, every field option, four languages, documents from 1k to 39k tokens, 13 image requests and 3 PDF requests. They contain 262 fields with exact values, plus estimates and forecasts. The reference is the official Cloudflare runtime on an NVIDIA H100 in its bfloat16 serving configuration, recorded call by call (every prompt, its output tokens and log-probabilities, and every typed-head read).
| Build | Fields with the same value | Prompts decoded to the same tokens | Typed-head reads with the same values | Mean confidence difference | Correct against known answers |
|---|---|---|---|---|---|
| Official runtime, H100 | reference | reference | reference | | 242 / 256 |
| MLX, unquantized bf16 (same code as this repo) | 262 / 262 | 640 / 640 | 55 / 55 | 0.002 | 242 / 256 |
| **This repo, 4-bit** | 257 / 262 | 628 / 639 | 53 / 55 | 0.024 | 245 / 256 |
The unquantized MLX build reproduces the official run on every field and every prompt. At 4-bit, 5 of 262 fields differ (for example `"Room 12"` against `"12"`, or a time's seconds), and accuracy against the known answers is within noise of the official run. All 52 image fields and 9 PDF fields match. Estimate and forecast summaries (mean, median, 10th and 90th percentile) differ from the official run by 1.7% on average.
**The port itself is exact.** With unquantized float32 weights on the CPU, the MLX engine agrees with the official PyTorch runtime to float32 rounding on Clairic Flash (confidences within 4e-06, token log-probabilities within 2e-06, same tokens and values on all 27 prompts and 36 fields checked). On the GPU, where MLX multiplies matrices at reduced precision, the float32 build gives the same value on 246 of 246 fields and the same tokens on 987 of 987 prompts of a 77-request run, images and PDFs included.
Confidence is a product of token probabilities, so it moves more than the answers do: 4-bit confidences differ from the official ones by 0.024 on average and by up to 0.68. Calibrate thresholds on this build rather than reusing ones tuned on the official runtime.
## Speed and memory
MacBook Pro, M5 Max, 128 GB, macOS 27.2, mlx 0.32.4. The model loads in about 2 s and uses 16.4 GB at rest. Times are for a warm model (the first request of a new shape compiles kernels and is slower).
| Request | Fields | Prompt tokens run | Time | Peak memory |
|---|---|---|---|---|
| short text, 1 generated field | 1 | 131 | 0.4 s | 17.0 GB |
| short text, 1 typed field (date) | 1 | 179 | 0.3 s | 16.8 GB |
| short text, 8 fields | 8 | 1,703 | 4.8 s | 19.7 GB |
| 1000-token document, 1 field | 1 | 1,155 | 2.7 s | 18.4 GB |
| 1000-token document, 8 fields | 8 | 11,713 | 20.7 s | 20.2 GB |
| 4000-token document, 1 field | 1 | 4,155 | 7.2 s | 18.7 GB |
| 4000-token document, 8 fields | 8 | 41,713 | 70.7 s | 18.7 GB |
| 8000-token document, 1 field | 1 | 8,155 | 14.3 s | 20.9 GB |
| 8000-token document, 8 fields | 8 | 81,713 | 141.5 s | 20.9 GB |
| 16000-token document, 1 field | 1 | 16,157 | 29.9 s | 21.9 GB |
| 32000-token document, 1 field | 1 | 32,156 | 65.5 s | 24.3 GB |
| receipt image, 3 fields | 3 | 1,526 | 2.5 s | 18.0 GB |
| invoice image, 8 fields incl. line items | 8 | 17,410 | 16.9 s | 21.6 GB |
| two images, 4 fields | 4 | 5,249 | 4.0 s | 18.9 GB |
| 2-page PDF, 3 fields | 3 | 3,629 | 3.4 s | 18.4 GB |
| CSV table, 5 rows x 3 columns | 1 | 2,648 | 7.2 s | 20.7 GB |
| 18-row table, 3 columns | 1 | 145,102 | 255.2 s | 20.9 GB |
| forecast (3 steps) + estimate | 2 | 421 | 0.7 s | 17.1 GB |
"Prompt tokens run" is the total over all of a request's prompts: each generated field or list item is its own prompt and re-reads the document, so a request costs roughly *fields x document length*. Typed fields (dates, times, estimates, forecasts) share one pass. Images are read once per request and reused by every field. Inputs longer than 1,024 tokens are processed in segments, so memory grows slowly with length: the 32k-token document peaks at 24.3 GB.
`load(path, memory_limit_gb=...)` sets a ceiling (default: 75% of the machine's memory) above which a request fails with an error instead of exhausting memory. `batch_size` (8) and `batch_token_budget` (8192) control how many prompts run together; larger values were not faster in our tests.
## How it was converted
`convert.py` (included) reproduces these weights from the upstream checkpoint
(revision `2c161be5cc71bacd6605b99a8c07547e58cfa434`):
```bash
python convert.py <upstream_checkpoint> <output_dir> --bits 4 --keep in_proj_a,in_proj_b
```
- Language model: linear layers, token embedding and vocabulary head quantized to 4 bits (affine, group size 64), 4.5 bits per weight.
- Left in bf16: the vision tower, the Gated DeltaNet gate projections (`in_proj_a`, `in_proj_b`: 0.06% of the weights, and the decay gate is sensitive), convolutions and decay parameters.
- RMSNorm weights are stored as exact float32 `1 + w` and evaluated in float32.
- The typed head is the upstream float32 file, with its weights rounded through bfloat16 at load as upstream's serving path does.
- The unused `mtp.*` draft-head weights are dropped.
## Differences from the upstream runtime
- **Engine.** MLX instead of PyTorch/CUDA. The CUDA-only accelerations (Triton kernels, CUDA graphs,
FlashAttention-3) are not needed and not included.
- **Image position encoding.** Upstream has two code paths that treat image tokens differently: in bfloat16
(its serving configuration, and what it does by default on a GPU) every token of an image gets the same
rotary position, while in float32 it uses the stock Qwen 3-axis positions. This port defaults to the first
(`load(path, rope="temporal")`) and also offers the second (`rope="mrope"`). Text-only requests are
unaffected. Upstream's two paths, which also differ in numeric precision, give the same value on 250 of 252 fields of Clairic Flash on an H100, and this port reproduces each of them with the matching setting.
- **Images are read once per request.** Upstream re-reads the image tokens for every field's prompt. Attention
is causal and the linear layers are recurrent, so the engine runs the shared image prefix once and continues
each prompt from that state. Results are the same up to floating-point rounding; requests with images and
many fields are much faster.
- **Image resizing** uses PIL instead of torchvision; pixel values differ from upstream by at most about 2/255.
- **No PyTorch in the model path.** The typed head runs in MLX and its decoding in NumPy (ported from PyTorch
and checked against it; see `NOTICE`).
## Limitations
- 4-bit quantization changes a small share of answers (see Fidelity) and shifts confidences.
- Speed scales with fields x document length: every generated field re-reads the document, as upstream does.
Long documents with many fields take minutes (see the table above).
- On the M5 Max we tested (mlx 0.32.4), MLX's GPU matrix multiply is accurate to about 1e-3 even for float32
inputs, so results are not bit-identical to PyTorch (see Fidelity). `load(path, device="cpu")` is accurate to float32
rounding and very slow; it exists for verification.
- Scores are model probabilities. Check `p_null`, `confidence`, `valid`, `complete` and `truncated` before
relying on an answer, as the upstream card advises.
- Not tested: requests close to the 65k-token limit.
## Files
| File | Purpose |
|---|---|
| `model-*.safetensors`, `model.safetensors.index.json`, `config.json` | MLX weights (quantized language model, bf16 vision tower) and configuration |
| `clair_head.safetensors`, `clair_head_config.json` | The typed head, unchanged from upstream (fp32) |
| `clairic_mlx.py` | `load`, `systemone`, and the command line |
| `clairic_runtime/` | Cloudflare's request pipeline with the MLX engine (`mlx_engine.py`, `processor.py`) |
| `server.py` | HTTP server: `POST /v1/systemone`, `GET /v1/models`, `/health`, `/stats` |
| `convert.py` | The conversion script |
| tokenizer and processor files, `requirements.txt`, `LICENSE`, `NOTICE` | As named |
## License
Apache-2.0, as upstream. `clairic_runtime/` is Cloudflare's code with the changes listed in `NOTICE`. The
backbone uses [mlx-vlm](https://github.com/Blaizzy/mlx-vlm)'s `qwen3_5` model definition.