How to use from
Hermes Agent
Start the MLX server
# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/clairic-flash-4bit"
Configure Hermes
# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mlx-community/clairic-flash-4bit
Run Hermes
hermes
Quick Links

mlx-community/clairic-flash-4bit

An MLX 4-bit conversion of Cloudflare/clairic-flash for Apple Silicon, with the runtime it needs and a small HTTP server. The larger, more accurate model of the family is at mlx-community/clairic-4bit.

Clairic is a structured-extraction model, not a chat model. Give it a state (text or JSON, plus optional images or PDFs) and a schema of typed fields; it returns one typed value per field with a confidence and a null probability. It also does numerical estimates and forecasts.

Field types Answered by
text, number, boolean, enum, object, and list items of those grammar-constrained decoding, one short prompt per field or list item
date, time, datetime, duration, compute, estimate, forecast, date lists a typed head that reads one forward pass

Why it needs the included code

mlx_lm.load / mlx_vlm.load can load the backbone, but they will not reproduce Clairic:

  • Answers are produced by Cloudflare's request pipeline, not by chatting: it rewrites each field into its own prompt, constrains decoding with a grammar built from the field's schema (and, for values copied from the document, from the document itself), scores null, retries, and assembles lists and tables.
  • Dates, times, durations, computed numbers, estimates and forecasts come from a separate typed head (clair_head.safetensors) applied to the backbone's hidden states.
  • The upstream runtime is PyTorch with CUDA kernels. This repository keeps Cloudflare's pipeline (clairic_runtime/, Apache-2.0) and replaces its engine with an MLX one (clairic_runtime/mlx_engine.py). Requests and responses have the same format as upstream.

Quick start

Requires an Apple Silicon Mac and Python 3.11 or later (tested with 3.12). The model uses 6 GB at rest and up to 11 GB on the requests we timed (see Speed and memory), so a 16 GB Mac is the practical minimum; 24 GB gives more room.

hf download mlx-community/clairic-flash-4bit --local-dir clairic-flash-4bit
cd clairic-flash-4bit
pip install -r requirements.txt

xgrammar (the grammar engine upstream uses) depends on PyTorch, so pip installs torch; the model itself runs entirely in MLX.

from clairic_mlx import load, systemone

model = load(".")          # or load("mlx-community/clairic-flash-4bit")
response = systemone(model, {
    "state": "Invoice from Northwind Supply. Issued 2026-09-02. Items: Pens; Paper. Total: $1080.00.",
    "fields": {
        "vendor": {"type": "text", "instructions": "The invoice vendor."},
        "total": {"type": "number", "decimals": 2},
        "issued": {"type": "date"},
        "items": {"type": "list", "items": "text", "instructions": "The purchased item names, in order."},
    },
})
print({name: answer["value"] for name, answer in response["answers"].items()})
# {'vendor': 'Northwind Supply', 'total': 1080.0, 'issued': '2026-09-02', 'items': ['Pens', 'Paper']}

Images and PDFs (up to four images or pages per request) can be PIL images, file paths or base64 data URIs:

response = systemone(model, {
    "state": "Read the attached receipt.",
    "images": ["receipt.png"],
    "fields": {"merchant": {"type": "text"}, "total": {"type": "number", "decimals": 2}, "date": {"type": "date"}},
})
response = systemone(model, {"documents": ["invoice.pdf"], "fields": {"invoice_number": {"type": "text"}}})

Estimates and forecasts return a distribution; include_quantiles adds the quantiles:

response = systemone(model, {
    "state": "Daily demand for the last seven days: 41, 45, 43, 47, 49, 46, 48.",
    "fields": {
        "next_day": {"type": "estimate", "min": 0, "max": 100},
        "next_three_days": {"type": "forecast", "min": 0, "max": 100, "horizon": 3, "interval": "day"},
    },
    "options": {"include_quantiles": True},
})

Command line and server:

python clairic_mlx.py --state "Gate B14 closes at 18:20." --fields '{"gate": {"type": "text"}, "closes": {"type": "time"}}'
python clairic_mlx.py --image receipt.png --fields '{"total": {"type": "number", "decimals": 2}}'

python server.py --port 8093
curl localhost:8093/v1/systemone -H 'content-type: application/json' -d '{
  "state": "Gate B14 closes at 18:20.",
  "fields": {"gate": {"type": "text"}, "closes": {"type": "time"}}}'

The server answers one request at a time (a request already fans out into many batched prompts); concurrent requests wait their turn. Set CLAIRIC_API_KEY to require a bearer token. In an async application use await model.predict_async(request).

The request and response format, the field options (instructions, nullable, decimals, unit, enum, source_only, knowledge, reference_date, ...) and the limits (65k-token context, four images or PDF pages, 4 MiB per embedded file) are those of the upstream model card.

Fidelity

Tested on 81 requests with known answers: invoices, support tickets, dates and times, computed values, lists and tables, every field option, four languages, documents from 1k to 39k tokens, 13 image requests and 3 PDF requests. They contain 262 fields with exact values, plus estimates and forecasts. The reference is the official Cloudflare runtime on an NVIDIA H100 in its bfloat16 serving configuration, recorded call by call (every prompt, its output tokens and log-probabilities, and every typed-head read).

Build Fields with the same value Prompts decoded to the same tokens Typed-head reads with the same values Mean confidence difference Correct against known answers
Official runtime, H100 reference reference reference 238 / 256
MLX, unquantized bf16 (same code as this repo) 261 / 262 1256 / 1258 54 / 55 0.005 238 / 256
This repo, 4-bit 247 / 262 1243 / 1258 51 / 55 0.031 231 / 256

The unquantized MLX build reproduces the official run almost exactly. 4-bit costs this 9B model some accuracy: 15 of 262 fields differ from the official run, and 7 fewer answers are correct. The typical error is a misread character in a number or identifier, often one read from an image or PDF (2231.5 for 2231.25, KR-2026-187 for KR-2026-1187). Estimate and forecast summaries differ from the official run by 7.8% on average. mlx-community/clairic-4bit, the 27B model, loses almost nothing at 4-bit.

If you have the memory, a higher-precision build of Flash is closer to the original. On the 77 requests of the suite without the longest documents:

Build (size) Fields with the same value Prompts decoded to the same tokens Typed-head reads with the same values Mean confidence difference Correct against known answers
Official runtime, H100 reference reference reference 223 / 240
This repo, 4-bit (6.3 GB) 231 / 246 971 / 986 48 / 52 0.031 216 / 240
Mixed: first 20 layers 6-bit, last 12 layers 4-bit (7.3 GB) 241 / 246 978 / 986 50 / 52 0.016 223 / 240
8-bit (10.8 GB) 246 / 246 985 / 986 51 / 52 0.006 223 / 240

Both can be built from the upstream checkpoint with the included script and run with the same code:

python convert.py <upstream_checkpoint> <output_dir> --bits 4 --high 'layers\.([0-9]|1[0-9])\.' --high-bits 6   # mixed
python convert.py <upstream_checkpoint> <output_dir> --bits 8                                                  # 8-bit

The mixed recipe comes from a sensitivity test: quantizing only the last 12 of the 32 layers to 4 bits changes the output distribution no more than noise, while the first 20 account for the damage. Leaving the vocabulary head or the embeddings unquantized, or raising only the output projections to 8 bits, did not help.

The port itself is exact. With unquantized float32 weights on the CPU, the MLX engine agrees with the official PyTorch runtime to float32 rounding on Clairic Flash (confidences within 4e-06, token log-probabilities within 2e-06, same tokens and values on all 27 prompts and 36 fields checked). On the GPU, where MLX multiplies matrices at reduced precision, the float32 build gives the same value on 246 of 246 fields and the same tokens on 987 of 987 prompts of a 77-request run, images and PDFs included.

Confidence is a product of token probabilities, so it moves more than the answers do: 4-bit confidences differ from the official ones by 0.031 on average and by up to 0.36. Calibrate thresholds on this build rather than reusing ones tuned on the official runtime.

Speed and memory

MacBook Pro, M5 Max, 128 GB, macOS 27.2, mlx 0.32.4. The model loads in about 2 s and uses 6.2 GB at rest. Times are for a warm model (the first request of a new shape compiles kernels and is slower).

Request Fields Prompt tokens run Time Peak memory
short text, 1 generated field 1 131 0.1 s 6.7 GB
short text, 1 typed field (date) 1 207 0.1 s 6.7 GB
short text, 8 fields 8 1,733 1.4 s 8.1 GB
1000-token document, 1 field 1 1,155 0.7 s 7.6 GB
1000-token document, 8 fields 8 11,743 6.0 s 8.4 GB
4000-token document, 1 field 1 4,155 2.1 s 7.8 GB
4000-token document, 8 fields 8 41,743 20.5 s 7.8 GB
8000-token document, 1 field 1 8,155 4.1 s 8.1 GB
8000-token document, 8 fields 8 81,743 40.5 s 8.2 GB
16000-token document, 1 field 1 16,157 8.6 s 9.4 GB
32000-token document, 1 field 1 32,156 19.2 s 10.6 GB
receipt image, 3 fields 3 2,698 2.1 s 7.6 GB
invoice image, 8 fields incl. line items 8 42,541 8.7 s 9.1 GB
two images, 4 fields 4 5,163 1.4 s 7.7 GB
2-page PDF, 3 fields 3 3,629 1.2 s 7.6 GB
CSV table, 5 rows x 3 columns 1 26,955 14.1 s 8.4 GB
18-row table, 3 columns 1 145,102 71.1 s 8.5 GB
forecast (3 steps) + estimate 2 421 0.2 s 7.0 GB

"Prompt tokens run" is the total over all of a request's prompts: each generated field or list item is its own prompt and re-reads the document, so a request costs roughly fields x document length. Typed fields (dates, times, estimates, forecasts) share one pass. Images are read once per request and reused by every field. Inputs longer than 1,024 tokens are processed in segments, so memory grows slowly with length: the 32k-token document peaks at 10.6 GB.

load(path, memory_limit_gb=...) sets a ceiling (default: 75% of the machine's memory) above which a request fails with an error instead of exhausting memory. batch_size (8) and batch_token_budget (8192) control how many prompts run together; larger values were not faster in our tests.

How it was converted

convert.py (included) reproduces these weights from the upstream checkpoint (revision 36ad68b999560f56b57ef16a22180f8cb83c6d77):

python convert.py <upstream_checkpoint> <output_dir> --bits 4 --keep in_proj_a,in_proj_b
  • Language model: linear layers, token embedding and vocabulary head quantized to 4 bits (affine, group size 64), 4.5 bits per weight.
  • Left in bf16: the vision tower, the Gated DeltaNet gate projections (in_proj_a, in_proj_b: 0.07% of the weights), convolutions and decay parameters.
  • RMSNorm weights are stored as exact float32 1 + w and evaluated in float32.
  • The typed head is the upstream float32 file, with its weights rounded through bfloat16 at load as upstream's serving path does.
  • The unused mtp.* draft-head weights are dropped.

Differences from the upstream runtime

  • Engine. MLX instead of PyTorch/CUDA. The CUDA-only accelerations (Triton kernels, CUDA graphs, FlashAttention-3) are not needed and not included.
  • Image position encoding. Upstream has two code paths that treat image tokens differently: in bfloat16 (its serving configuration, and what it does by default on a GPU) every token of an image gets the same rotary position, while in float32 it uses the stock Qwen 3-axis positions. This port defaults to the first (load(path, rope="temporal")) and also offers the second (rope="mrope"). Text-only requests are unaffected. Upstream's two paths, which also differ in numeric precision, give the same value on 250 of 252 fields of Clairic Flash on an H100, and this port reproduces each of them with the matching setting.
  • Images are read once per request. Upstream re-reads the image tokens for every field's prompt. Attention is causal and the linear layers are recurrent, so the engine runs the shared image prefix once and continues each prompt from that state. Results are the same up to floating-point rounding; requests with images and many fields are much faster.
  • Image resizing uses PIL instead of torchvision; pixel values differ from upstream by at most about 2/255.
  • No PyTorch in the model path. The typed head runs in MLX and its decoding in NumPy (ported from PyTorch and checked against it; see NOTICE).

Limitations

  • 4-bit quantization costs this model some accuracy, mainly on numbers and identifiers read from images (see Fidelity), and shifts confidences. Use the 27B model or a higher-precision build when that matters.
  • Speed scales with fields x document length: every generated field re-reads the document, as upstream does. Long documents with many fields take minutes (see the table above).
  • On the M5 Max we tested (mlx 0.32.4), MLX's GPU matrix multiply is accurate to about 1e-3 even for float32 inputs, so results are not bit-identical to PyTorch (see Fidelity). load(path, device="cpu") is accurate to float32 rounding and very slow; it exists for verification.
  • Scores are model probabilities. Check p_null, confidence, valid, complete and truncated before relying on an answer, as the upstream card advises.
  • Not tested: requests close to the 65k-token limit.

Files

File Purpose
model-*.safetensors, model.safetensors.index.json, config.json MLX weights (quantized language model, bf16 vision tower) and configuration
clair_head.safetensors, clair_head_config.json The typed head, unchanged from upstream (fp32)
clairic_mlx.py load, systemone, and the command line
clairic_runtime/ Cloudflare's request pipeline with the MLX engine (mlx_engine.py, processor.py)
server.py HTTP server: POST /v1/systemone, GET /v1/models, /health, /stats
convert.py The conversion script
tokenizer and processor files, requirements.txt, LICENSE, NOTICE As named

License

Apache-2.0, as upstream. clairic_runtime/ is Cloudflare's code with the changes listed in NOTICE. The backbone uses mlx-vlm's qwen3_5 model definition.

Downloads last month
-
Safetensors
Model size
2B params
Tensor type
U32
ยท
F32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mlx-community/clairic-flash-4bit

Finetuned
Qwen/Qwen3.5-9B
Quantized
(2)
this model