FluidInference/gliner2-5-decide-coreml

GLiNER2.5-Decide for Core ML

Fixed-shape Core ML exports of the classification path of fastino/GLiNER2.5-Decide at revision 65624f1a0265b3f612bae66a2685a06b94a68a9d. Fastino authored the Apache-2.0 source model; Fluid Inference converted it. Each package holds the DeBERTa-v3-large encoder and the shared label classifier (436,022,273 exported parameters). The span and count heads are not exported: this release covers classification only.

One call scores up to four decision heads (for example intent, urgency and route) over the same text. The host tokenizes the schema with the upstream GLiNER2 processor and passes the position of every label marker; the package returns per-head logits. Single-label heads use softmax and multi-label heads use sigmoid with cls_threshold, exactly as classify_text does.

Fast Decisions

Scored on Fastino's Fast Decisions development split (revision 1a33070c, 17 domains × 100 rows, 2,900 heads) with the protocol from the dataset card: one call per head with the labels as given, exact set match, mean of the 17 domain accuracies. The 512-token bucket fits every row.

Package (L512)	Size	Average	Pooled	Heads changed vs native
native PyTorch	—	62.93%	61.38%	—
fp16	936 MB	62.93%	61.38%	0
w8	477 MB	63.01%	61.52%	7
lut6	361 MB	63.03%	61.52%	42

The card's published 60.2% for GLiNER2.5-Decide is on the held-out test split (300 rows per domain), which is not public; the card asks that development-split scores not be reported as the benchmark. The table above shows that the Core ML packages reproduce the native checkpoint on the public split, not a new benchmark result.

Rejected variants, kept in reports/ for reference: embedding-only W8 (805 MB, 63.00%) saves little over w8; per-tensor LUT4 (246 MB, 62.26%, 236 heads changed, −4 points on paper_field) loses accuracy. Grouped-channel palettization needs an iOS 18 deployment target and was not tried.

Packages

Bucket	Tokens	Heads × labels	Fast Decisions heads that fit	fp16	w8	lut6
L128_H4_K8	128	4 × 8	10.7%	8.2 ms	10.1 ms	9.9 ms
L256_H4_K32	256	4 × 32	95.0%	14.7 ms	20.6 ms	17.6 ms
L512_H4_K32	512	4 × 32	100%	44.0 ms	42.9 ms	62.9 ms

Latency is the p50 of 100 Python predict calls for a three-head request with compute units ALL on an M5 Pro with macOS 27.0, after 10 warmups (benchmark-latency.py). This is a local measurement for one request shape, not a device-wide benchmark; p95 values are in reports/latency.json. W8 and LUT6 are weight-only compression: they shrink the download and do not make inference faster. CPU_AND_NE is much slower than ALL for this graph (689 ms versus 14.7 ms p50 for fp16 L256), so use ALL.

On the first 20 bucket-fitting rows per domain, the L128 and L256 packages agree with the native predictions: fp16 173/173 and 345/345, w8 172/173 and 345/345, lut6 169/173 and 342/345 (reports/verify-packages-L128-L256.json).

Usage

uv sync
uv run python runtime.py --model-dir . --precision lut6 --length 256 \
  --text "My transfer is still pending and I used the wrong sort code. Can you stop it?" \
  --tasks '{"intent": ["transfer_pending", "transfer_cancel", "beneficiary_add", "card_lost"], "urgency": ["low", "normal", "high"]}'

from runtime import CoreMLDecide

model = CoreMLDecide(".", precision="w8", length=256)
model.classify(
    "Battery dies before lunch, but the keyboard and the screen are the best I have used.",
    {"sentiment": ["positive", "negative", "mixed", "neutral"],
     "aspects": {"labels": ["battery", "keyboard", "screen", "price"], "multi_label": True, "cls_threshold": 0.4}},
)

The runtime loads only the tokenizer files and the Core ML package; it does not load PyTorch weights. Requests that exceed the bucket raise ValueError. When downloading with huggingface_hub.snapshot_download, pass local_dir= so the packages are materialized; Core ML compilation rejects the symlinked weight files in the default Hub cache.

Inputs and outputs

Name	Type	Shape	Meaning
input_ids	int32	[1, L]	Schema and text tokens from the GLiNER2 span collator, padded
attention_mask	int32	[1, L]	1 for real tokens
marker_indices	int32	[1, 4, K]	Token position of each [L] label marker, per head
marker_mask	float32	[1, 4, K]	1 for used label slots
logits	float32	[1, 4, K]	Label logits; unused slots are −10⁴
probabilities	float32	[1, 4, K]	Per-head softmax of logits

Reproduce

uv sync
uv run python convert-coreml.py --precision fp16 --length 512 --max-options 32
uv run python compress-coreml.py --method linear \
  --source build/gliner2_decide_classification_fp16_L512_H4_K32.mlpackage \
  --output build/gliner2_decide_classification_w8_L512_H4_K32.mlpackage
uv run python compress-coreml.py --method lut --bits 6 --granularity per_tensor \
  --source build/gliner2_decide_classification_fp16_L512_H4_K32.mlpackage \
  --output build/gliner2_decide_classification_lut6_L512_H4_K32.mlpackage
uv run python score-fast-decisions.py --backend native --out build/fast-decisions-native.json
uv run python score-fast-decisions.py --backend coreml \
  --package build/gliner2_decide_classification_lut6_L512_H4_K32.mlpackage --out build/fast-decisions-lut6.json

convert-coreml.py checks the export wrapper against the native logits (4.8e-7) and the converted package against native classify_text on three examples. The Core ML deployment target is iOS 17 / macOS 14. Source file hashes are in assets.lock.json; conversion, scoring, agreement and latency reports are in reports/.