A concise card: the files, use, validation and the Galaxy S23 Ultra results

README.md

@@ -14,40 +14,27 @@ tags:
 
 # GLiNER2.5-Decide — LiteRT (Android)
 
-An on-device conversion of GLiNER2.5-Decide for Android's LiteRT with every trained head: classification, and the span
-heads that extract entities, structures and relations, decoded by gliner2's own code. One file for every input length,
-validated against the library on every call type, with phone measurements and Qualcomm NPU files.
+A LiteRT conversion of [fastino/GLiNER2.5-Decide](https://huggingface.co/fastino/GLiNER2.5-Decide) (revision
+`bbe10ff77ebb238777c17d3a8ac9260e30929057`, Apache-2.0, by Fastino), converted by spybyscript. It carries every trained head:
+classification, plus the span heads that extract entities, structures and relations. gliner2's own code decodes the
+outputs. For the model itself (training, labels, benchmarks, intended use), see the
+[original model card](https://huggingface.co/fastino/GLiNER2.5-Decide), copied here as `model-card.md`. Parts of the
+graph code are ported from [litert-community/GLiNER2.5-Decide-LiteRT](https://huggingface.co/litert-community/GLiNER2.5-Decide-LiteRT).
 
-Converted from [fastino/GLiNER2.5-Decide](https://huggingface.co/fastino/GLiNER2.5-Decide) at revision `bbe10ff77ebb238777c17d3a8ac9260e30929057` (Apache-2.0, by Fastino).
-The encoder files have four signatures for the classifier (`s128`, `s160`, `s256`, `s512`: logits at every position) and four
-for the hidden states (`e128` to `e512`), all over one set of weights; the heads file holds the span heads and the
-classifier (the span file and the count file hold the same heads, built for LiteRT's GPU); the token table is a separate float16 file the host maps. `config.json` is the full contract.
+## Files
 
 | File | Size | Contents |
 |---|---|---|
-| `gliner25_decide_wfp16.tflite` | 1028 MB | FULLY_CONNECTED weights stored float16 behind DEQUANTIZE (ai-edge-quantizer FLOAT_CASTING); FP32 activations, relative-position tables, norms and biases; finite -1e4 mask |
-| `gliner25_decide_fp16hw.tflite` | 1028 MB | wfp16's storage over the FP16-hardware graph: LayerNorm power-of-two pre-scale (x*2^-k, eps*4^-k, k from the corpus' FP32 ranges), first feed-forward FULLY_CONNECTED split in two halves, finite -1e4 mask |
-| `gliner25_decide_heads_wfp16.tflite` | 105 MB | `span_rep` (`w64`, `w128`, `w256`, `w512` words), `count` (count prediction), `count_embed` (`c1`/`c4`/`c20` steps × `m8`/`m32` fields), the classifier (`k32`, `k128` rows); float16 weight storage, FP32 compute |
-| `gliner25_decide_span_wfp16.tflite` | 76 MB | the heads file's `span_rep` (`w64` to `w512`), `count` and classifier (`k32`, `k128`), rearranged so LiteRT's GPU compiles them, plus fused span scores (`f{W}_p16`, `f{W}_p128`); float16 weight storage, FP32 compute |
-| `gliner25_decide_count_wfp16.tflite` | 38 MB | the heads file's `count_embed` alone (`c1`/`c4`/`c20` × `m8`/`m32`); with the span file it replaces the heads file |
-| `gliner25_decide_embeddings_f16.bin` | 262 MB | float16 [128011, 1024], little-endian, row-major |
-| `runtime/` | | `litert_span_runtime.py` (the reference runtime), `verify_parity.py`, `parity_corpus.json` |
+| `gliner25_decide_wfp16.tflite` | 1028 MB | encoder for the CPU and the GPU at FP32: `s128` / `s160` / `s256` / `s512` (classifier logits at every position) and `e128` … `e512` (hidden states), one set of float16-stored weights |
+| `gliner25_decide_fp16hw.tflite` | 1028 MB | the same encoder rewritten for FP16 hardware (GPU at FP16, NPU): LayerNorm power-of-two pre-scale, first feed-forward layer split in two |
+| `gliner25_decide_span_wfp16.tflite` | 76 MB | span heads in a form LiteRT's GPU compiles: `w{W}` span representations, `f{W}_p{P}` fused span scores, `count`, classifier `k32` / `k128` |
+| `gliner25_decide_count_wfp16.tflite` | 38 MB | count embedding `c{C}_m{M}` (CPU) |
+| `gliner25_decide_heads_wfp16.tflite` | 105 MB | all span heads in the library's layout (CPU or NPU); the span file and the count file replace it |
+| `gliner25_decide_embeddings_f16.bin` | 262 MB | token table, float16 [128011, 1024], little-endian, row-major |
+| `runtime/` | | `litert_span_runtime.py` (reference runtime), `verify_parity.py`, `parity_corpus.json` |
 | `npu/` | | Qualcomm NPU files (below) |
 
-## What it runs
-
-`runtime/litert_span_runtime.py` is gliner2's span extractor with its five trained modules (encoder, span
-representation, count prediction, count embedding, classifier) on LiteRT; the schema handling, scoring and decoding are
-the library's own code, and no PyTorch weights are loaded. Every public call works: `extract`, `classify_text`,
-`extract_entities`, `extract_json`, `extract_relations`, and their `batch_*` and `*_long` forms.
-
-- Classification, single- and multi-label, with label descriptions and several tasks in one call.
-- Entities, structures (`extract_json`, with choice fields) and relations, through the span heads.
-- **Sizes:** a request holds at most 512 tokens (schema and text); spans up to 8 words and up to 20 instances per
-  structure (the checkpoint's own limits); up to 32 fields or entity types per task. A larger request is refused with a
-  ValueError, never truncated: use the `*_long` calls with a chunk that fits, e.g. `extract_entities_long(text, labels,
-  chunk_size=300)`. The original runs with dynamic shapes.
-- **Weights:** only their storage and compute precision change; nothing is retrained or pruned.
+`config.json` lists every graph's signatures, shapes and host steps.
 
 ## Use
 
@@ -56,257 +43,124 @@ the library's own code, and no PyTorch weights are loaded. Every public call wor
 import sys; sys.path.insert(0, "runtime")
 from litert_span_runtime import LiteRTSpanExtractor
 
-model = LiteRTSpanExtractor(".")          # this repository's folder
+model = LiteRTSpanExtractor(".", span="gliner25_decide_span_wfp16.tflite", count="gliner25_decide_count_wfp16.tflite")
 model.classify_text("Please cancel my order.", {"intent": ["cancel", "refund", "track"]})
 model.extract_entities("Alice founded Acme in Toronto in 2020.", ["person", "organization", "city"])
 model.extract_relations("Alice founded Acme in Toronto.", ["founded", "located_in"])
-
-# The span file and the count file in place of the heads file: the same results; the span heads take about half the CPU time
-model = LiteRTSpanExtractor(".", span="gliner25_decide_span_wfp16.tflite", count="gliner25_decide_count_wfp16.tflite")
 ```
 
-On a phone, run the same graphs with LiteRT's `CompiledModel` and port the library's decoding steps the runtime uses.
-
-### The classification graphs on their own
-
-Tokenize with `tokenizer.json` and pack the prompt exactly as gliner2 2.0.0 does. Pick the smallest signature that holds the ids
-(refuse more than 512), zero-pad the ids, look each one up in the table (row `id` at byte `id * 2048`) and upcast to float32 into
-`inputs_embeds` [1, N, 1024]; `attention_mask` [1, N] is 1.0 for tokens and 0.0 for padding. `logits` [1, N] holds the classifier
-at every position: read each task's `[L]` marker positions and take a softmax over them (sigmoid for multi-label tasks).
+Every public gliner2 call works: `extract`, `classify_text`, `extract_entities`, `extract_json` and `extract_relations`,
+plus their `batch_*` and `*_long` forms. The runtime runs gliner2's span extractor with the five trained modules on
+LiteRT. Schema handling, scoring and decoding are the library's own code.
 
-LiteRT `CompiledModel`: on the GPU set the precision explicitly (FP32 for `wfp16`) and constant tensor sharing; on the CPU set
-XNNPACK's weight cache path, or XNNPACK packs a copy of the weights for every signature.
+**On a phone,** run the same graphs with LiteRT's `CompiledModel`:
+1. Tokenize with `tokenizer.json` and pack the prompt as gliner2 2.0.0 does.
+2. Look the ids up in the table (row `id` at byte `id * 2048`) into `inputs_embeds`.
+3. Call the smallest signature that holds the request.
+4. For classification, read each task's `[L]` marker positions from `logits`.
 
-### The span file and the count file
+GPU settings: set the precision explicitly (FP32 for `wfp16`, FP16 for `fp16hw`), turn on constant tensor sharing, and use
+NORMAL priority (see the S23 results). On the CPU, set XNNPACK's weight cache path. On a Snapdragon NPU, load the chip's
+precompiled file.
 
-`gliner25_decide_span_wfp16.tflite` computes what the heads file's span signatures compute, rearranged so that LiteRT's GPU delegate
-compiles it and every accelerator does less work. Each change is exact in real arithmetic; only the order of float sums
-differs.
+**Fixed sizes.** The original runs with dynamic shapes; this conversion has fixed limits:
+- a request holds at most 512 tokens (schema and text);
+- spans up to 8 words, and up to 20 instances per structure (the checkpoint's own limits);
+- up to 32 fields or entity types per task.
 
-- **`w{W}`** (64, 128, 256 or 512 words): `words` [1, W, 1024] -> `span_rep` [1, W, 8, 1024], the library's layout. The
-  span head's first layer acts on a start/end pair, and a linear layer over a concatenation is the sum of its two halves,
-  so both halves are computed once per word instead of once per span: a span's hidden layer is relu(A_i + B_(i+k)). Spans
-  that end past the bucket read zero padding; they also end past the text, which the library masks.
-- **`f{W}_p{P}`** (P = 16 or 128 queries): `words` [1, W, 1024] and `queries` [P, 1024] (struct_proj flattened over
-  count and fields) -> `scores` [P, W, 8], the raw logits the library computes as einsum("lkd,bpd->bplk", span_rep,
-  struct_proj). The span head's last layer folds into the queries: q . (W2 h + b2) = (q W2) . h + q . b2, so a host that
-  needs only the scores never builds the [W, 8, 1024] span tensor.
-- **`count`** and the classifier **`k32`**, **`k128`**, as in the heads file.
-- Every tensor is [1, rows, features]. The GPU delegate rejected the heads file's FULLY_CONNECTED on 4-D inputs and its
-  BROADCAST_TO.
+A larger request is refused with a ValueError, never truncated. For long text, use the `*_long` calls with a chunk that
+fits. The weights change only in storage and compute precision.
 
-`gliner25_decide_count_wfp16.tflite` holds `count_embed` alone. Its GRU does not compile on LiteRT's GPU; on the CPU one step over up to
-8 fields takes about 2 ms.
+**The span file** computes the heads file's span representations with two exact rearrangements (only the order of float
+sums differs):
+- the first layer of the span head is computed once per word instead of once per span;
+- `f{W}_p{P}` folds the last layer into the queries, so a host that needs only the scores skips the [W, 8, 1024] span tensor.
 
-Against the heads file, `w{W}` is within 4e-5 on every span inside the bucket, and the fused scores are within 0.003 of the
-einsum (scores reach about 7,000). Through the reference runtime with both files in place of the heads file:
-296 of 296 corpus requests identical (confidences within 0.0013), and 11 of 11 of FluidInference's fixtures.
+Every tensor is [1, rows, features], the shapes LiteRT's GPU delegate accepts.
 
 ## Validation
 
-### Every call type against the library
-
 `runtime/verify_parity.py` runs each request through gliner2 with the original weights and through the reference runtime
-(LiteRT 2.2.0 on the CPU, `wfp16` encoder and the heads file), and compares the structured results. The corpus
-(`runtime/parity_corpus.json`) is 48 short synthetic texts with schema templates for every call type, 296 requests.
-
-| Call type | Requests | Identical results | Span F1 (lowest) | Largest confidence difference |
-|---|---|---|---|---|
-| `entities` | 40 | 40 | 1.0 | 0.000214 |
-| `entities_single` | 8 | 8 | 1.0 | 0.000236 |
-| `relations_work` | 24 | 24 | 1.0 | 0.000351 |
-| `record_natural` | 16 | 16 | 1.0 | 0.000303 |
-| `record_latent` | 24 | 24 | 1.0 | 0.000489 |
-| `attributes` | 16 | 16 | 1.0 | 0.00045 |
-| `classification_topic` | 40 | 40 | 1.0 | 0.000204 |
-| `mixed` | 16 | 16 | 1.0 | 0.000207 |
-| `relations_family` | 8 | 8 | 1.0 | 0.000234 |
-| `classification_described` | 24 | 24 | 1.0 | 9.3e-05 |
-| `record_anchorless` | 8 | 8 | 1.0 | 0.000269 |
-| `structure_event` | 24 | 24 | 1.0 | 0.001315 |
-| `classification_multi` | 16 | 16 | 1.0 | 0.000334 |
-| `entities_described` | 16 | 16 | 1.0 | 0.000182 |
-| `structure_order` | 8 | 8 | 1.0 | 0.000579 |
-| `relations_event` | 8 | 8 | 1.0 | 0.000118 |
-
-All 296 of 296 requests give identical results; FluidInference's eleven schema-path fixtures (from its
-gliner2.5-base-v1 port) also match, 11 of 11. With the span file and the count file in place of the heads file: 296 of 296, and 11 of 11
-fixtures. `validation-full.json` holds the numbers.
-
-### The recorded classification calls (Mac, not a phone)
-
-616 Decide calls (2,338 decisions) recorded from the gliner2 2.0.0 library on a synthetic English corpus of short
-conversational statements, packed by the library's collator, padded to the smallest
-bucket and decoded by a Python mirror of an on-device host. ai-edge-litert 2.2.0 on arm64 Mac (Apple M5 Max, macOS 27.0.1):
-the CPU, and the Mac's GPU through the same ML Drift accelerator the phone uses (Metal here, OpenCL on Android) as a check of
-the precision modes. These timings are Mac measurements, not phone numbers.
-
-| Variant | Runtime | Labels as the library | Max probability drift | Mac ms per call p50 by bucket | Gate |
-|---|---|---|---|---|---|
-| `wfp16` | CPU (XNNPACK, 4 threads) | 2338/2338 (100.00%) | 0.00032 | 128: 58.3, 160: 68.1, 512: 248.6 | pass |
-| `wfp16` | Mac GPU (Metal, ML Drift), FP32 precision | 2338/2338 (100.00%) | 0.00032 | 128: 21.0, 160: 23.8, 512: 76.1 | pass |
-| `wfp16` | Mac GPU (Metal, ML Drift), default precision (FP16) | 2226/2338 (95.21%) | 0.30555 | 128: 16.3, 160: 20.5, 512: 49.6 | fail |
-| `fp16hw` | CPU (XNNPACK, 4 threads) | 2338/2338 (100.00%) | 0.00032 | 128: 60.2, 160: 70.5, 512: 255.1 | pass |
-| `fp16hw` | Mac GPU (Metal, ML Drift), FP32 precision | 2338/2338 (100.00%) | 0.00032 | 128: 22.1, 160: 25.7, 512: 79.8 | pass |
-| `fp16hw` | Mac GPU (Metal, ML Drift), default precision (FP16) | 2332/2338 (99.74%) | 0.01158 | 128: 16.6, 160: 22.1, 512: 50.7 | pass |
-
-The acceptance gate: label agreement ≥ 95 %, largest probability drift ≤ 0.05. `validation.json` holds every number.
-
-## On phones
-
-### Galaxy Z Fold 8 (SM8850, 2026-10-28/29): classification on the GPU and the CPU
-
-Snapdragon 8 Elite Gen 5 (SM8850), Adreno 840, Android 17, LiteRT 2.2.0. One request per signature from the
-test set, warm, on a rested phone (it throttles within minutes of continuous runs):
-
-| Setting | s128 | s160 | s512 | RAM taken | Labels as the library |
-|---|---|---|---|---|---|
-| `fp16hw`, GPU FP16 | 62 ms | 81 ms | 439 ms | about 1.0 GB | 9 / 9 |
-| `fp16hw`, GPU FP16 with FP32 accumulation | 80 ms | 99 ms | 496 ms | about 1.05 GB | 9 / 9 |
-| `wfp16`, GPU FP32 | 136 ms | 173 ms | 758 ms | about 1.9 GB | 9 / 9 |
-| `wfp16`, CPU (4 threads, XNNPACK weight cache) | 120 ms | 155 ms | 972 ms | about 0.4–0.5 GB, plus droppable file pages | 9 / 9 |
+(LiteRT CPU), then compares the structured results:
+- **The corpus** (`runtime/parity_corpus.json`): 296 requests over 16 call types (entities, relations, structures, the
+  three record modes, attributes, single- and multi-label classification, mixed schemas). All 296 are identical, with
+  span F1 1.0 and confidences within 0.0013.
+- **The same check with the span file and the count file:** also 296 of 296.
+- **FluidInference's eleven schema-path fixtures:** 11 of 11.
+- **2,338 recorded classification decisions:** `wfp16` gives the library's label on all of them, and `fp16hw` at FP16
+  precision on 2,332.
 
-- **Set constant tensor sharing on the GPU.** Without it each signature holds its own copy of the weights: 5.3 GB of graphics memory against 1.9 GB. The "graphics" figure in PSS counts the weights twice; the RAM the system loses is about 1.0 GB.
-- **On LiteRT 2.2.0, LOW GPU priority kept a rendering app's frames.** At the default, NORMAL or HIGH priority, each 512-token call held the app's renderer off the GPU for about 400 ms. LOW cost no frames but added about 95–130 ms to every call. On LiteRT 2.3.2 and a Galaxy S23 Ultra, LOW no longer keeps the frames (below).
-- **The first load is the slowest.** It takes 6–8 s on the GPU. The program cache shortens later loads very little.
-- **These are LiteRT 2.2.0's numbers.** On LiteRT 2.3.2 the GPU at LOW priority is slower on the same phone: 472 ms a call at s160 (2026-10-30; about 176 ms on 2.2.0) and 2.75 s at s512 (2026-11-01).
+`validation-full.json` and `validation.json` hold the numbers.
 
-### Galaxy S23 Ultra (SM8550, 2026-11-02): extraction
+## Galaxy S23 Ultra (SM8550)
 
-Android 16, LiteRT 2.3.2 with `qnn-runtime` 2.50.0. Each signature ran on a request recorded from the corpus, 10 calls
-after a 30 s rest, with thermal status 0 throughout. The CPU used 4 threads (XNNPACK, weight cache). GPU times are whole
-calls, because the GPU runs asynchronously. CPU and NPU times are the model alone. NORMAL and LOW are LiteRT's GPU
-priority. The FP16 columns use `fp16hw` for the encoder.
+Snapdragon 8 Gen 2, Android 16, LiteRT 2.3.2 with `qnn-runtime` 2.50.0. Each graph ran on a request recorded from the
+corpus: 10 calls after a 30 s rest, at thermal status 0. The CPU ran 4 threads; the GPU ran at NORMAL priority, timed as
+whole calls. CPU and NPU times are the model alone.
 
-| Graph | CPU | GPU FP32, NORMAL | GPU FP16, NORMAL | GPU FP16, LOW | NPU |
-|---|---|---|---|---|---|
-| encoder `e128` / `e160` / `e256` / `e512` | 628 / 866 / 1,467 / 3,486 ms | 202 / 268 / 471 / 1,549 ms | 103 / 124 / 227 / 773 ms | 289 / 353 / 617 / 2,082 ms | 34 / 55 / 168 ms (precompiled; `e512` not compiled) |
-| span file `w64` / `w128` / `w256` / `w512` | 43 / 105 / 221 / 450 ms | 19 / 26 / 42 / 79 ms | 10 / 15 / 25 / 47 ms | 27 / 35 / 58 / 102 ms | 7.3 / 6.0 / 14 / 42 ms |
-| span file `f64` / `f128` / `f256` / `f512`, 16 queries | 26 / 55 / 111 / 220 ms | 15 / 18 / 28 / 46 ms | 8 / 10 / 16 / 27 ms | 23 / 25 / 38 / 62 ms | 3.8 / 5.6 / 14 / 44 ms |
-| heads file `w64` / `w128` / `w256` / `w512` | 90 / 200 / 440 / 935 ms | does not compile | does not compile | does not compile | 7.4 / 13.7 / 24 / 56 ms |
-| `count_embed`, 1 step × 8 fields / 20 steps × 32 fields | 2.1 / 169 ms (count file) | does not compile | does not compile | does not compile | 1.4 / 5.4 ms (heads file) |
-| classifier, 32 / 128 rows | 0.9 / 3.8 ms | 1.6 / 2.4 ms | 1.2 / 1.5 ms | 2.8 / 2.8 ms | 0.8 / 1.3 ms |
-| dirty memory, encoder / span file | 1.9 / 1.7 GB | 4.6 / 1.3 GB | 2.7 / 0.8 GB | 2.7 / 0.9 GB | 0.12 / 0.21 GB |
-
-- **An extraction request of up to 128 tokens and 64 words** takes these times:
-  - about 115 ms on the GPU at FP16 and NORMAL priority;
-  - about 220 ms at FP32 and NORMAL;
-  - about 320 ms at FP16 and LOW;
-  - about 43 ms on the NPU;
-  - about 660 ms on the CPU with the span file, and 720 ms with the heads file.
-
-  `count_embed` runs on the CPU in every GPU case.
-- **Answers on the phone's accelerators.** 32 recorded requests, two per call type, ran with the phone's outputs through the
-  rest of the pipeline:
-  - the encoder: unchanged 32 of 32 on the NPU, at GPU FP32 and on the CPU; at GPU FP16, 31 of 32 (the same relation
-    request with or without FP32 accumulation);
-  - the span file: unchanged 32 of 32 at GPU FP16 (confidences within 0.005) and on the NPU (within 0.0002);
-  - the heads file on the NPU: unchanged 32 of 32 (within 0.0006).
-- **GPU priority beside a rendering app** (same phone and LiteRT version). An app rendered every frame on the same GPU
-  (Godot 4.7, Vulkan, 120 Hz) while the model ran three classification calls each round (`s160` and two `s256`, `fp16hw`
-  at GPU FP16, with the host's tokenizing and table lookup). Each setting ran 6 rounds; the two GPU settings ran twice:
-
-  | The model on | The app's longest frame per round (median) | The three calls |
-  |---|---|---|
-  | GPU, LOW priority | 596–602 ms | about 1.6 s (350 + 620 + 630 ms) |
-  | GPU, NORMAL priority | 228–229 ms | about 0.65 s (150 + 245 + 250 ms) |
-  | NPU (precompiled on the phone) | 81 ms | about 0.5 s (92 + 210 + 205 ms) |
-  | CPU, 4 threads | 48 ms | 4–5 s |
-
-  On this phone and LiteRT version, LOW priority does not let the app's rendering in. Either way the app stops for one
-  whole call, and LOW makes each call 2–3 times longer. On the GPU, use NORMAL and keep each call short (the smallest
-  signature that fits). The NPU keeps the frames.
-
-## Qualcomm NPU (`npu/`)
-
-Files for the Hexagon NPU of Snapdragon phones, through LiteRT's Qualcomm accelerator: LiteRT 2.3.2 with
-`litert-npu-runtime-qualcomm` 2.3.2 and Qualcomm's `qnn-runtime` 2.50.0 (Qualcomm's AI Stack License). They are not part
-of the GPU and CPU files above. Every one comes from `gliner25_decide_fp16hw.tflite`, because the NPU computes at FP16.
-Inputs, outputs and host steps are the same as above; `npu/manifest.json` lists each file with its chip, signatures,
-size, SHA-256 and the versions it was built with.
-
-| Files | What they are |
-|---|---|
-| `npu/gliner25_decide_fp16hw_s128.tflite`, `_s160`, `_s256` (660 MB / 673 MB / 710 MB) | One signature each, cut from `fp16hw` (on LiteRT's CPU their logits equal the full file's). The phone compiles them for its own NPU at first use, so they serve any Snapdragon LiteRT supports: about 20 / 45 / 130 s on an SM8850, once per OS build. s512 is left out: compiling it on a phone ran out of memory on a Galaxy S26 (litert-community). |
-| `npu/qualcomm/gliner25_decide_fp16hw_<chip>.tflite` | Precompiled on x86-64 Linux (ai-edge-litert 2.3.0, QAIRT 2.50.0.260828): all four signatures over one copy of the weights (`qualcomm_enable_weight_sharing=true`, the compiler's default O3). Each signature runs all 1,846 operations in one NPU partition, so nothing in the file falls back to the GPU or the CPU. The phone loads it without compiling. It runs only on its chip (`Build.SOC_MODEL`) and only with the runtime versions above. Building one takes 13–18 minutes and up to 24 GB of memory on the Linux host. |
-
-| Chip | Snapdragon | Hexagon | Precompiled file | Size | Run on |
-|---|---|---|---|---|---|
-| SM8850 | 8 Elite Gen 5 | V81 | `npu/qualcomm/gliner25_decide_fp16hw_SM8850.tflite` | 877 MB | Galaxy Z Fold 8, 2026-11-01 |
-| SM8750 | 8 Elite | V79 | `npu/qualcomm/gliner25_decide_fp16hw_SM8750.tflite` | 875 MB | compiled, not run on a phone yet |
-| SM8650 | 8 Gen 3 | V75 | `npu/qualcomm/gliner25_decide_fp16hw_SM8650.tflite` | 860 MB | compiled, not run on a phone yet |
-| SM8550 | 8 Gen 2 | V73 | `npu/qualcomm/gliner25_decide_fp16hw_SM8550.tflite` | 862 MB | Galaxy S23 Ultra, 2026-11-02 |
-| SM8450 | 8 Gen 1 | V69 | `npu/qualcomm/gliner25_decide_fp16hw_SM8450.tflite` | 856 MB | compiled, not run on a phone yet |
-
-### On a Galaxy Z Fold 8 (SM8850, 2026-11-01)
-
-LiteRT 2.3.2 with `qnn-runtime` 2.50.0, Android 17. Each file ran the test requests that fit its signature, 10 calls
-each, after a 60 s rest (thermal status 0 before and after). "Run" is the model alone, without the host's table lookup.
-
-| File | Load | Run p50 | Largest logit difference from PyTorch FP32 | Labels as the library |
+| Graph | CPU | GPU FP16 (`fp16hw`) | GPU FP32 (`wfp16`) | NPU (precompiled) |
 |---|---|---|---|---|
-| `npu/qualcomm/gliner25_decide_fp16hw_SM8850.tflite` at s160 | 1.5 s | 31.4 ms | 0.0032 | 12 / 12 |
-| `npu/qualcomm/gliner25_decide_fp16hw_SM8850.tflite` at s512 | 1.8 s | 644 ms | 0.015 | 16 / 16 |
-| the cuts, compiled on the phone: s128 / s160 / s256 | 0.45–0.72 s once compiled | 21.3 / 30.0 / 87.6 ms | 0.0032 / 0.0032 / 0.0029 | 4 / 4, 12 / 12, 12 / 12 |
-| the cuts, each precompiled on Linux for SM8850 | 0.65–0.77 s | 22.4 / 29.9 / 86.8 ms | 0.0032 / 0.0032 / 0.0029 | 4 / 4, 12 / 12, 12 / 12 |
-
-- **Precompiled files run like the phone's own compile,** within about 1 ms, with the same logits, and skip the compile.
-- **s512 on the NPU:** the four-signature file holds 52 MB of dirty memory and 859 MB of file pages there. On the same
-  phone and LiteRT version the GPU path (`fp16hw`, FP16, LOW priority) took 2.75 s a call at s512, 24 s to load and
-  2.0 GB of dirty memory.
-- s128 and s256 from the four-signature file have not been run yet. From a three-signature file built the same way
-  (s128 to s256, 755 MB) they ran at 21.8 and 88.3 ms.
-
-### On a Galaxy S23 Ultra (SM8550, 2026-11-02)
-
-LiteRT 2.3.2 with `qnn-runtime` 2.50.0, Android 16, 8 GB of memory. The same method as on the Fold. For the GPU rows the
-"Run" column is the whole call, because the GPU runs asynchronously.
-
-| File | Load | Run p50 | Largest logit difference from PyTorch FP32 | Labels as the library |
-|---|---|---|---|---|
-| `npu/qualcomm/gliner25_decide_fp16hw_SM8550.tflite` at s128 / s160 / s256 | 1.5–1.6 s | 32.2 / 53.9 / 170 ms | 0.0039 / 0.0042 / 0.0042 | 4 / 4, 12 / 12, 12 / 12 |
-| `npu/qualcomm/gliner25_decide_fp16hw_SM8550.tflite` at s512 | 1.7 s | 1,149 ms | 0.014 | 16 / 16 |
-| the cuts, compiled on the phone: s128 / s160 / s256 | 21 / 41 / 117 s to compile | 31.3 / 53.2 / 164 ms | 0.0039 / 0.0042 / 0.0042 | 4 / 4, 12 / 12, 12 / 12 |
-| the cuts, each precompiled on Linux for SM8550 | 0.5–0.65 s | 32.0 / 53.9 / 162 ms | 0.0039 / 0.0042 / 0.0042 | 4 / 4, 12 / 12, 12 / 12 |
-| `fp16hw` on the GPU (FP16, LOW priority), s160 / s512 | 6.3 / 7.7 s | 369 / 2,099 ms | 0.019 / 0.053 | 12 / 12, 16 / 16 |
-
-- **Precompiled files run like the phone's own compile** here too, with the same logits.
-- **Memory:** the four-signature file holds 50–69 MB of dirty memory and 853 MB of file pages. The GPU held 1.4 GB (s160)
-  and 2.0 GB (s512) of dirty memory. A compile on the phone holds about 1.3 GB of dirty memory in the process that
-  compiles; reloaded from LiteRT's compiler cache, the model holds about 50 MB, like a precompiled file.
-- **s512:** 1.17 s a call on the NPU against 2.10 s on the GPU.
+| encoder, 128 / 160 / 256 / 512 tokens | 628 / 866 / 1,467 / 3,486 ms | 103 / 124 / 227 / 773 ms | 202 / 268 / 471 / 1,549 ms | 34 / 55 / 168 / 1,149 ms |
+| span file `w64` / `w128` / `w256` / `w512` | 43 / 105 / 221 / 450 ms | 10 / 15 / 25 / 47 ms | 19 / 26 / 42 / 79 ms | 7.3 / 6.0 / 14 / 42 ms |
+| fused scores `f64` … `f512`, 16 queries | 26 / 55 / 111 / 220 ms | 8 / 10 / 16 / 27 ms | 15 / 18 / 28 / 46 ms | 3.8 / 5.6 / 14 / 44 ms |
+| count embedding, 1 step × 8 fields / 20 × 32 | 2.1 / 169 ms | not supported by LiteRT's GPU | not supported | 1.4 / 5.4 ms |
+| classifier, 32 / 128 rows | 0.9 / 3.8 ms | 1.2 / 1.5 ms | 1.6 / 2.4 ms | 0.8 / 1.3 ms |
+| dirty memory, encoder / span file | 1.9 / 1.7 GB | 2.7 / 0.8 GB | 4.6 / 1.3 GB | 0.12 / 0.21 GB |
+
+- **The NPU's 512-token time** comes from the classification file; the hidden-state file stops at 256 tokens.
+- **An extraction request** (up to 128 tokens and 64 words) takes about 43 ms on the NPU. On the GPU it takes about
+  115 ms at FP16 and 220 ms at FP32; on the CPU, about 660 ms.
+- **Loading:** a precompiled NPU file loads in 0.5–1.7 s. The encoder's first GPU load takes 6–10 s.
+- **Answers with the phone's outputs.** 32 recorded requests, two per call type, ran through the rest of the pipeline:
+  - the encoder: unchanged on the NPU, at GPU FP32 and on the CPU; at GPU FP16, 31 of 32 (one relation request changed);
+  - the span file: unchanged at GPU FP16 and on the NPU;
+  - the heads file: unchanged on the NPU.
+- **GPU priority beside a rendering app.** An app rendered every frame (Godot 4.7, Vulkan, 120 Hz) while the model ran
+  three classification calls each round (`s160` and two `s256`, `fp16hw` at FP16):
+
+  | The model on | The app's longest frame per round | The three calls |
+  |---|---|---|
+  | GPU, NORMAL priority | 228 ms | about 0.65 s |
+  | GPU, LOW priority | 600 ms | about 1.6 s |
+  | NPU | 81 ms | about 0.5 s |
+  | CPU | 48 ms | 4–5 s |
 
-### Extraction on the NPU
+  On this phone and LiteRT version, LOW does not let the app's rendering in; it only makes each call longer. On the GPU,
+  use NORMAL and the smallest signature that fits. The NPU keeps the frames.
 
-The extraction path needs the encoder's `e{N}` signatures and the heads file on the NPU. For each chip, `npu/qualcomm/`
-holds three more precompiled files: `gliner25_decide_fp16hw_se128-256_<chip>.tflite` (the classifier's `s128`/`s160`/`s256`
-and the hidden states' `e128`/`e160`/`e256` over one copy of the weights, so one file serves both kinds of call) and
-`gliner25_decide_heads_<chip>.tflite` (every signature of the heads file), and `gliner25_decide_span_<chip>.tflite`
-(every signature of the span file, each in one NPU partition; 7–16 s to build). On an SM8550 the precompiled heads file loads in
-0.5 s and holds 0.11 GB of dirty memory; compiled on the phone instead, the same file takes 63 s and 1.1 GB, with the same
-outputs and speed.
+## Qualcomm NPU files (`npu/`)
 
-| Chip | Snapdragon | Hexagon | Classification + hidden states, 128-256 tokens | Heads | Span file | Run on |
-|---|---|---|---|---|---|---|
-| SM8850 | 8 Elite Gen 5 | V81 | `npu/qualcomm/gliner25_decide_fp16hw_se128-256_SM8850.tflite` (792 MB) | `npu/qualcomm/gliner25_decide_heads_SM8850.tflite` (115 MB) | `npu/qualcomm/gliner25_decide_span_SM8850.tflite` (81 MB) | compiled, not run on a phone yet |
-| SM8750 | 8 Elite | V79 | `npu/qualcomm/gliner25_decide_fp16hw_se128-256_SM8750.tflite` (781 MB) | `npu/qualcomm/gliner25_decide_heads_SM8750.tflite` (114 MB) | `npu/qualcomm/gliner25_decide_span_SM8750.tflite` (81 MB) | compiled, not run on a phone yet |
-| SM8650 | 8 Gen 3 | V75 | `npu/qualcomm/gliner25_decide_fp16hw_se128-256_SM8650.tflite` (777 MB) | `npu/qualcomm/gliner25_decide_heads_SM8650.tflite` (114 MB) | `npu/qualcomm/gliner25_decide_span_SM8650.tflite` (81 MB) | compiled, not run on a phone yet |
-| SM8550 | 8 Gen 2 | V73 | `npu/qualcomm/gliner25_decide_fp16hw_se128-256_SM8550.tflite` (778 MB) | `npu/qualcomm/gliner25_decide_heads_SM8550.tflite` (114 MB) | `npu/qualcomm/gliner25_decide_span_SM8550.tflite` (81 MB) | Galaxy S23 Ultra, 2026-11-02 |
-| SM8450 | 8 Gen 1 | V69 | `npu/qualcomm/gliner25_decide_fp16hw_se128-256_SM8450.tflite` (777 MB) | `npu/qualcomm/gliner25_decide_heads_SM8450.tflite` (108 MB) | `npu/qualcomm/gliner25_decide_span_SM8450.tflite` (80 MB) | compiled, not run on a phone yet |
+The NPU path runs through LiteRT's Qualcomm accelerator:
+- LiteRT 2.3.2 with `litert-npu-runtime-qualcomm` 2.3.2 and Qualcomm's `qnn-runtime` 2.50.0 (Qualcomm's AI Stack License);
+- files precompiled on x86-64 Linux with QAIRT 2.50.0;
+- each signature in one NPU partition;
+- each file runs only on its chip (`Build.SOC_MODEL`) and with those runtime versions.
 
-## Compared with litert-community/GLiNER2.5-Decide-LiteRT
+`npu/manifest.json` lists every file with its chip, signatures, size and SHA-256.
 
-| | this conversion | litert-community (rev c687e899) |
+| File | Contents | Size |
 |---|---|---|
-| Files | one `.tflite` with signatures s128 / s160 / s256 / s512 over one set of weights (1.03 GB), plus the 262 MB table | one file per length: s128 / s256 / s512, 2.18 GB together, plus the table |
-| Lengths | adds s160: requests of 129 to 160 tokens no longer pad to 256 | 128 / 256 / 512 |
-| Output | `logits` at every position (the host reads the `[L]` markers, as smdesai's Core ML port does) | `label_routing` input, 32 label logits out |
-| Extraction | entities, structures, relations (the span heads and the `e{N}` signatures) | classification only |
-| Validation | every recorded library call (616 calls, 2,338 decisions): wfp16 2,338 / 2,338; fp16hw at GPU FP16 2,332 / 2,338 | 42 decisions |
-| GPU speed on SM8850 | s128 62 ms at FP16 (Fold 8) | s128 60 ms with FP32 accumulation (S26) |
+| `npu/qualcomm/gliner25_decide_fp16hw_<chip>.tflite` | classification, `s128` … `s512` | 856–877 MB |
+| `npu/qualcomm/gliner25_decide_fp16hw_se128-256_<chip>.tflite` | classification and hidden states, 128–256 tokens | 777–792 MB |
+| `npu/qualcomm/gliner25_decide_heads_<chip>.tflite` | the heads file | 108–115 MB |
+| `npu/qualcomm/gliner25_decide_span_<chip>.tflite` | the span file | 80–81 MB |
+| `npu/gliner25_decide_fp16hw_s128.tflite`, `_s160`, `_s256` | single-signature cuts that the phone compiles itself, once (on the S23: 21 / 41 / 117 s) | 660–710 MB |
+
+| Chip | Snapdragon | Hexagon | Run on a phone |
+|---|---|---|---|
+| SM8850 | 8 Elite Gen 5 | V81 | classification file (Galaxy Z Fold 8) |
+| SM8750 | 8 Elite | V79 | not yet |
+| SM8650 | 8 Gen 3 | V75 | not yet |
+| SM8550 | 8 Gen 2 | V73 | all four files (Galaxy S23 Ultra, the results above) |
+| SM8450 | 8 Gen 1 | V69 | not yet |
 
 ## Files
 
-`LICENSE` (Apache-2.0), `NOTICE.txt` (attribution and changes), `model-card.md` (the original card), `config.json`
-(contract), `validation.json` (the recorded classification calls), `validation-full.json` (every call type),
-`SHA256SUMS`, tokenizer files, `source/` (the checkpoint's configs), `runtime/`, and `npu/` (the Qualcomm NPU files above).
+- `LICENSE` (Apache-2.0) and `NOTICE.txt` (attribution and changes)
+- `model-card.md`, the original card
+- `config.json`, the contract
+- `validation.json` and `validation-full.json`
+- `SHA256SUMS`
+- the tokenizer files
+- `source/` (the checkpoint's configs)
+- `runtime/` and `npu/`

SHA256SUMS

@@ -1,6 +1,6 @@
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
 c810d9c1789eef9890a1844136e77a9fb475c950df02a19ffdf3ecbbc50ec7be  NOTICE.txt
-3c45bb9d7b7b72738de6778ad1ec05f8fe8b7940e83ad02d0a74b1b33b86ae1b  README.md
+58a06d79fa17abe067a350bfb6e2b11f198ab5463d019e61570381e22eec0753  README.md
 98e61f6cec59c2238b2a80c4d3e829de88ba570ebd7ab22187672caa6737d9e7  config.json
 43a132d3571d4a028013990f4866e266d01e35394819d89cbb1cd24d558608bc  gliner25_decide_count_wfp16.tflite
 f90f5d837721b3d4df7194a89ea18bbea5c449e6944fb6078664f1a8d20484fd  gliner25_decide_embeddings_f16.bin