---
license: apache-2.0
language:
- en
base_model: fastino/GLiNER2.5-Decide
library_name: litert
tags:
- gliner2
- litert
- tflite
- android
- on-device
---

# GLiNER2.5-Decide — LiteRT (Android)

A LiteRT conversion of [fastino/GLiNER2.5-Decide](https://huggingface.co/fastino/GLiNER2.5-Decide) (revision
`bbe10ff77ebb238777c17d3a8ac9260e30929057`, Apache-2.0, by Fastino), converted by spybyscript. It carries every trained head:
classification, plus the span heads that extract entities, structures and relations. gliner2's own code decodes the
outputs. For the model itself (training, labels, benchmarks, intended use), see the
[original model card](https://huggingface.co/fastino/GLiNER2.5-Decide), copied here as `model-card.md`. Parts of the
graph code are ported from [litert-community/GLiNER2.5-Decide-LiteRT](https://huggingface.co/litert-community/GLiNER2.5-Decide-LiteRT).

## Files

| File | Size | Contents |
|---|---|---|
| `gliner25_decide_wfp16.tflite` | 1028 MB | encoder for the CPU and the GPU at FP32: `s128` / `s160` / `s256` / `s512` (classifier logits at every position) and `e128` … `e512` (hidden states), one set of float16-stored weights |
| `gliner25_decide_fp16hw.tflite` | 1028 MB | the same encoder rewritten for FP16 hardware (GPU at FP16, NPU): LayerNorm power-of-two pre-scale, first feed-forward layer split in two |
| `gliner25_decide_span_wfp16.tflite` | 76 MB | span heads in a form LiteRT's GPU compiles: `w{W}` span representations, `f{W}_p{P}` fused span scores, `count`, classifier `k32` / `k128` |
| `gliner25_decide_count_wfp16.tflite` | 38 MB | count embedding `c{C}_m{M}` (CPU) |
| `gliner25_decide_heads_wfp16.tflite` | 105 MB | all span heads in the library's layout (CPU or NPU); the span file and the count file replace it |
| `gliner25_decide_embeddings_f16.bin` | 262 MB | token table, float16 [128011, 1024], little-endian, row-major |
| `runtime/` | | `litert_span_runtime.py` (reference runtime), `verify_parity.py`, `parity_corpus.json` |
| `npu/` | | Qualcomm NPU files (below) |

`config.json` lists every graph's signatures, shapes and host steps.

## Use

```python
# pip install gliner2==2.0.0 torch numpy ai-edge-litert
import sys; sys.path.insert(0, "runtime")
from litert_span_runtime import LiteRTSpanExtractor

model = LiteRTSpanExtractor(".", span="gliner25_decide_span_wfp16.tflite", count="gliner25_decide_count_wfp16.tflite")
model.classify_text("Please cancel my order.", {"intent": ["cancel", "refund", "track"]})
model.extract_entities("Alice founded Acme in Toronto in 2020.", ["person", "organization", "city"])
model.extract_relations("Alice founded Acme in Toronto.", ["founded", "located_in"])
```

Every public gliner2 call works: `extract`, `classify_text`, `extract_entities`, `extract_json` and `extract_relations`,
plus their `batch_*` and `*_long` forms. The runtime runs gliner2's span extractor with the five trained modules on
LiteRT. Schema handling, scoring and decoding are the library's own code.

**On a phone,** run the same graphs with LiteRT's `CompiledModel`:
1. Tokenize with `tokenizer.json` and pack the prompt as gliner2 2.0.0 does.
2. Look the ids up in the table (row `id` at byte `id * 2048`) into `inputs_embeds`.
3. Call the smallest signature that holds the request.
4. For classification, read each task's `[L]` marker positions from `logits`.

GPU settings: set the precision explicitly (FP32 for `wfp16`, FP16 for `fp16hw`) and turn on constant tensor sharing.
The right priority depends on the GPU; see "GPU priority" below. On the CPU, set XNNPACK's weight cache path. On a
Snapdragon NPU, load the chip's precompiled file.

**Fixed sizes.** The original runs with dynamic shapes; this conversion has fixed limits:
- a request holds at most 512 tokens (schema and text);
- spans up to 8 words, and up to 20 instances per structure (the checkpoint's own limits);
- up to 32 fields or entity types per task.

A larger request is refused with a ValueError, never truncated. For long text, use the `*_long` calls with a chunk that
fits. The weights change only in storage and compute precision.

**The span file** computes the heads file's span representations with two exact rearrangements (only the order of float
sums differs):
- the first layer of the span head is computed once per word instead of once per span;
- `f{W}_p{P}` folds the last layer into the queries, so a host that needs only the scores skips the [W, 8, 1024] span tensor.

Every tensor is [1, rows, features], the shapes LiteRT's GPU delegate accepts.

## Validation

`runtime/verify_parity.py` runs each request through gliner2 with the original weights and through the reference runtime
(LiteRT CPU), then compares the structured results:
- **The corpus** (`runtime/parity_corpus.json`): 296 requests over 16 call types (entities, relations, structures, the
  three record modes, attributes, single- and multi-label classification, mixed schemas). All 296 are identical, with
  span F1 1.0 and confidences within 0.0013.
- **The same check with the span file and the count file:** also 296 of 296.
- **FluidInference's eleven schema-path fixtures:** 11 of 11.
- **2,338 recorded classification decisions:** `wfp16` gives the library's label on all of them, and `fp16hw` at FP16
  precision on 2,332.

`validation-full.json` and `validation.json` hold the numbers.

## Phones

LiteRT 2.3.2 with `qnn-runtime` 2.50.0. Each graph ran on a request recorded from the corpus: 10 calls after a rest, at
thermal status 0. The CPU ran 4 threads; the GPU ran at NORMAL priority, timed as whole calls. CPU and NPU times are the
model alone. The NPU rows use the chip's precompiled files; at 512 tokens they use the classification file, because the
hidden-state file stops at 256.

### Galaxy Z Fold 8 (SM8850, Snapdragon 8 Elite Gen 5, Android 17)

| Graph | CPU | GPU FP16 (`fp16hw`) | GPU FP32 (`wfp16`) | NPU |
|---|---|---|---|---|
| encoder, 128 / 160 / 256 / 512 tokens | 115 / 145 / 262 / 722 ms | 60 / 72 / 121 / 404 ms | 105 / 137 / 214 / 610 ms | 23 / 30 / 95 / 644 ms |
| span file `w64` / `w128` / `w256` / `w512` | 10 / 20 / 38 / 77 ms | 5.3 / 7.1 / 12 / 24 ms | 9.5 / 14 / 35 / 76 ms | 7.0 / 4.4 / 8.7 / 31 ms |
| fused scores `f64` … `f512`, 16 queries | 6.4 / 14 / 30 / 55 ms | 4.4 / 5.4 / 7.9 / 15 ms | 7.6 / 10 / 16 / 27 ms | 3.1 / 3.9 / 8.2 / 33 ms |
| count embedding, 1 step × 8 fields / 20 × 32 | 1.5 / 37 ms | not supported by LiteRT's GPU | not supported | 0.9 / 2.6 ms |
| classifier, 32 / 128 rows | 0.2 / 0.8 ms | 0.5 / 0.6 ms | 1.0 / 1.0 ms | 0.5 / 0.8 ms |
| dirty memory, encoder / span file | 2.3 / 1.4 GB | 2.8 / 0.8 GB | 4.8 / 1.4 GB | 0.14 / 0.19 GB |

- **An extraction request** (up to 128 tokens and 64 words) takes about 35 ms on the NPU. On the GPU it takes about
  70 ms at FP16 and 125 ms at FP32; on the CPU, about 135 ms.
- **Loading:** the precompiled NPU files load in 0.3–1.0 s; the encoder's first GPU load takes 11–13 s.
- **Answers with the phone's outputs:**
  - the encoder: unchanged on the CPU and at GPU FP32 and FP16; on the NPU, 31 of 32 (one family-relation request changed);
  - the span file and the heads file: unchanged on the GPU and the NPU.

### Galaxy S23 Ultra (SM8550, Snapdragon 8 Gen 2, Android 16)

| Graph | CPU | GPU FP16 (`fp16hw`) | GPU FP32 (`wfp16`) | NPU |
|---|---|---|---|---|
| encoder, 128 / 160 / 256 / 512 tokens | 628 / 866 / 1,467 / 3,486 ms | 103 / 124 / 227 / 773 ms | 202 / 268 / 471 / 1,549 ms | 34 / 55 / 168 / 1,149 ms |
| span file `w64` / `w128` / `w256` / `w512` | 43 / 105 / 221 / 450 ms | 10 / 15 / 25 / 47 ms | 19 / 26 / 42 / 79 ms | 7.3 / 6.0 / 14 / 42 ms |
| fused scores `f64` … `f512`, 16 queries | 26 / 55 / 111 / 220 ms | 8 / 10 / 16 / 27 ms | 15 / 18 / 28 / 46 ms | 3.8 / 5.6 / 14 / 44 ms |
| count embedding, 1 step × 8 fields / 20 × 32 | 2.1 / 169 ms | not supported by LiteRT's GPU | not supported | 1.4 / 5.4 ms |
| classifier, 32 / 128 rows | 0.9 / 3.8 ms | 1.2 / 1.5 ms | 1.6 / 2.4 ms | 0.8 / 1.3 ms |
| dirty memory, encoder / span file | 1.9 / 1.7 GB | 2.7 / 0.8 GB | 4.6 / 1.3 GB | 0.12 / 0.21 GB |

- **An extraction request** (up to 128 tokens and 64 words) takes about 43 ms on the NPU. On the GPU it takes about
  115 ms at FP16 and 220 ms at FP32; on the CPU, about 660 ms.
- **Loading:** the precompiled NPU files load in 0.5–1.7 s; the encoder's first GPU load takes 6–10 s.
- **Answers with the phone's outputs:**
  - the encoder: unchanged on the NPU, at GPU FP32 and on the CPU; at GPU FP16, 31 of 32 (one relation request changed);
  - the span file and the heads file: unchanged on the GPU and the NPU.

Each answer check ran 32 recorded requests (two per call type) with the phone's outputs through the rest of the pipeline.

### GPU priority beside a rendering app

An app rendered every frame (Godot 4.7, Vulkan) while the model ran three classification calls each round (`s160` and two
`s256`, `fp16hw` at GPU FP16). Each cell gives the app's longest frame per round (median) and the three calls' total time.

| The model on | Galaxy Z Fold 8 (Adreno 840, 60 Hz) | Galaxy S23 Ultra (Adreno 740, 120 Hz) |
|---|---|---|
| GPU, LOW priority | **37 ms**, 0.66 s | 600 ms, 1.6 s |
| GPU, NORMAL priority | 110 ms, 0.37 s | **228 ms**, 0.65 s |
| NPU | 39 ms, 0.24 s | 81 ms, 0.5 s |
| CPU | 41 ms, 0.81 s | 48 ms, 4–5 s |

- **LOW priority depends on the GPU.** On the Fold's Adreno 840, LOW keeps the frames and costs about 0.3 s per round. On
  the S23's Adreno 740, LOW does not let the app's rendering in, and only makes each call longer.
- **Measure on the target GPU before choosing.** Use the smallest signature that fits.
- **The NPU keeps the frames on both phones** and is the fastest option.

## Qualcomm NPU files (`npu/`)

The NPU path runs through LiteRT's Qualcomm accelerator:
- LiteRT 2.3.2 with `litert-npu-runtime-qualcomm` 2.3.2 and Qualcomm's `qnn-runtime` 2.50.0 (Qualcomm's AI Stack License);
- files precompiled on x86-64 Linux with QAIRT 2.50.0;
- each signature in one NPU partition;
- each file runs only on its chip (`Build.SOC_MODEL`) and with those runtime versions.

`npu/manifest.json` lists every file with its chip, signatures, size and SHA-256.

| File | Contents | Size |
|---|---|---|
| `npu/qualcomm/gliner25_decide_fp16hw_<chip>.tflite` | classification, `s128` … `s512` | 856–877 MB |
| `npu/qualcomm/gliner25_decide_fp16hw_se128-256_<chip>.tflite` | classification and hidden states, 128–256 tokens | 777–792 MB |
| `npu/qualcomm/gliner25_decide_heads_<chip>.tflite` | the heads file | 108–115 MB |
| `npu/qualcomm/gliner25_decide_span_<chip>.tflite` | the span file | 80–81 MB |
| `npu/gliner25_decide_fp16hw_s128.tflite`, `_s160`, `_s256` | single-signature cuts that the phone compiles itself, once (on the S23: 21 / 41 / 117 s) | 660–710 MB |

| Chip | Snapdragon | Hexagon | Run on a phone |
|---|---|---|---|
| SM8850 | 8 Elite Gen 5 | V81 | all four files (Galaxy Z Fold 8, the results above) |
| SM8750 | 8 Elite | V79 | not yet |
| SM8650 | 8 Gen 3 | V75 | not yet |
| SM8550 | 8 Gen 2 | V73 | all four files (Galaxy S23 Ultra, the results above) |
| SM8450 | 8 Gen 1 | V69 | not yet |

## Files

- `LICENSE` (Apache-2.0) and `NOTICE.txt` (attribution and changes)
- `model-card.md`, the original card
- `config.json`, the contract
- `validation.json` and `validation-full.json`
- `SHA256SUMS`
- the tokenizer files
- `source/` (the checkpoint's configs)
- `runtime/` and `npu/`