| --- |
| license: apache-2.0 |
| language: |
| - en |
| base_model: fastino/gliner2.5-base-v1 |
| library_name: litert |
| tags: |
| - gliner2 |
| - litert |
| - tflite |
| - android |
| - on-device |
| --- |
| |
| # GLiNER2.5 base (gliner2.5-base-v1) — LiteRT (Android) |
|
|
| A LiteRT conversion of [fastino/gliner2.5-base-v1](https://huggingface.co/fastino/gliner2.5-base-v1) (revision |
| `1a8bc24e00dc7300b9017c81d63e3dcdabb26596`, Apache-2.0, by Fastino), converted by spybyscript. It carries every trained head: |
| entities, classification, structures, relations and records. gliner2's own code decodes the outputs. For the model itself |
| (training, labels, benchmarks, intended use), see the [original model card](https://huggingface.co/fastino/gliner2.5-base-v1), |
| copied here as `model-card.md`. The split into graphs follows FluidInference's Core ML port |
| ([gliner2-5-base-coreml](https://huggingface.co/FluidInference/gliner2-5-base-coreml), Apache-2.0). |
|
|
| ## Files |
|
|
| | File | Size | Contents | |
| |---|---|---| |
| | `gliner2_base_encoder_wfp16.tflite` | 189 MB | encoder for the CPU and the GPU at FP32 (`inputs_embeds`, `attention_mask` → `hidden_states`), float16-stored weights | |
| | `gliner2_base_encoder_fp16hw.tflite` | 189 MB | the same encoder rewritten for FP16 hardware (NPU): LayerNorm power-of-two pre-scale, first feed-forward layer split in two | |
| | `npu/qualcomm/gliner2_base_encoder_fp16hw_<chip>.tflite` | 182 MB each | `fp16hw` precompiled for one Snapdragon NPU (below) | |
| | `gliner2_base_head_fp32.tflite` | 8.5 MB | boundary head: routed states, boundary marginals, pool projections, null and count heads, classification logits | |
| | `gliner2_base_scorer_fp32.tflite` | 2.1 MB | candidate scorer and record candidate states | |
| | `gliner2_base_explicit_fp32.tflite` | 2.1 MB | explicit-span scorer (entity attributes, choice fields) | |
| | `gliner2_base_relation_fp32.tflite` | 23.7 MB | relation scorer over head/tail pairs | |
| | `gliner2_base_record_assignment_fp32.tflite` | 1.2 MB | record decoder: instance-to-field assignment, object and latent-seed scores | |
| | `gliner2_base_record_anchorless_fp32.tflite` | 2.9 MB | record decoder for anchorless records | |
| | `gliner2_base_embeddings_f16.bin` | 197 MB | token table, float16 [128011, 768], little-endian, row-major | |
| | `runtime/` | | `litert_boundary_runtime.py` (reference runtime), `verify_parity.py`, `parity_corpus.json` | |
|
|
| `config.json` lists every graph's signatures and shapes. |
|
|
| ## Use |
|
|
| ```python |
| # pip install gliner2==2.0.0 torch numpy ai-edge-litert |
| import sys; sys.path.insert(0, "runtime") |
| from litert_boundary_runtime import LiteRTBoundaryExtractor |
| from gliner2 import Schema |
| |
| model = LiteRTBoundaryExtractor(".") # this repository's folder |
| model.extract_entities("Alice founded Acme in Toronto in 2020.", ["person", "organization", "city"]) |
| model.extract_relations("Alice founded Acme in Toronto.", ["founded", "located_in"]) |
| schema = Schema().entities(["person", "organization"]).relations(["works_for"]) |
| model.extract("Ravi Shah is an engineer at Tidewater Systems.", schema, include_confidence=True) |
| ``` |
|
|
| Every public gliner2 call works: `extract`, `extract_entities`, `classify_text`, `extract_json` and `extract_relations`, |
| plus their `batch_*` and `*_long` forms. The runtime runs gliner2's BoundaryExtractor with every trained module on |
| LiteRT. Schema handling, the candidate pool and the decoders are the library's own code. |
|
|
| **On a phone,** run the same graphs with LiteRT's `CompiledModel`; the runtime shows the host steps. |
| - **The encoder** is the only graph worth an accelerator. Use the NPU with your chip's precompiled file from `npu/` |
| (or `fp16hw`, which the phone compiles in about 6 s), the GPU with `wfp16` at FP32 (or `fp16hw` at FP16 where it |
| stays finite), or the CPU. |
| - **The six head graphs** run on the CPU in 1–16 ms each. |
|
|
| **Fixed sizes.** The original runs with dynamic shapes; these are FluidInference's bucket sizes. One request holds at most: |
| - 128 subword tokens (schema and text) and 64 text words; |
| - 8 queries (entity labels, structure fields, relation roles) and 8 classification choices; |
| - a candidate pool of 192 (the checkpoint's own); |
| - 4 relation types with 256 head/tail pairs; |
| - 64 explicit spans per query; |
| - records of up to 8 fields with 192 candidates each. |
|
|
| A larger request is refused with a ValueError, never truncated. Run long text through the `*_long` calls, for example |
| `extract_entities_long(text, labels, chunk_size=40, chunk_overlap=8)`. The weights change only in storage and compute |
| precision. |
|
|
| ## Validation |
|
|
| `runtime/verify_parity.py` runs each request through gliner2 with the original weights and through the reference runtime |
| (LiteRT CPU), then compares the structured results: |
| - **The corpus** (`runtime/parity_corpus.json`): 296 requests over 16 call types (entities, relations, structures, the |
| three record modes, attributes, single- and multi-label classification, mixed schemas). All 296 are identical, with |
| span F1 1.0 and confidences within 0.0014. |
| - **FluidInference's eleven schema-path fixtures:** 11 of 11. |
|
|
| `validation-full.json` and `validation.json` hold the numbers. |
|
|
| ## Phones |
|
|
| LiteRT 2.3.2 with `qnn-runtime` 2.50.0. Each graph ran on a request recorded from the corpus: 10 calls after a rest, at |
| thermal status 0. The CPU ran 4 threads; the GPU ran at NORMAL priority, timed as whole calls. The NPU compiled the graphs |
| on the phone; for the encoder that takes about 6 s, once. |
|
|
| | Graph | Galaxy Z Fold 8 (SM8850): CPU / GPU / NPU | Galaxy S23 Ultra (SM8550): CPU / GPU / NPU | |
| |---|---|---| |
| | encoder, 128 tokens | 38 / 37 (FP32), 20 (FP16) / 9.9 ms | 143 / 67 (FP32) / 13 ms | |
| | head | 0.7 / — / 0.6 ms | 1.2 / — / 1.2 ms | |
| | scorer | 2.6 / — / does not run | 2.9 / — / 3.4 ms | |
| | explicit spans | 5.4 / — / does not run | 5.6 / — / does not run | |
| | relations | 4.9 / — / 1.3 ms | 16 / — / 2.8 ms | |
| | record assignment | 6.0 / not measured / 4.3 ms | 8.9 / 21.5 ms (LOW priority) / 4.0 ms | |
| | anchorless records | 2.9 / — / 0.6 ms | 10.8 / — / 1.3 ms | |
|
|
| — means LiteRT's GPU does not support the graph; run it on the CPU. |
|
|
| - **The precompiled encoder** (`npu/qualcomm/gliner2_base_encoder_fp16hw_SM8850.tflite`) on the Fold: 8.1 ms a call, loaded |
| in 0.2 s, with 31 of 32 recorded requests unchanged (below), as with the encoder the phone compiled itself. |
| - **A request** with the encoder on the NPU and the heads on the CPU takes about 15 ms on the Fold and 20 ms on the S23. |
| On the CPU alone it takes about 45 ms on the Fold and 150 ms on the S23. |
| - **Answers with the phones' encoder outputs.** 32 recorded requests, two per call type, ran through the rest of the |
| pipeline: |
| - GPU FP32: unchanged on both phones; |
| - NPU: 31 of 32 on the Fold and 30 of 32 on the S23 (only latent-record requests changed); |
| - `fp16hw` at GPU FP16: 31 of 32 on the Fold. On the S23's Adreno 740 it gives non-finite outputs, with or without FP32 |
| accumulation, so use FP32 there. |
| - **Dirty memory, encoder:** about 0.45 GB on the CPU, 0.6 GB on the GPU at FP16 and 0.85 GB at FP32. The NPU holds about |
| 0.5 GB after compiling on the phone. |
| - **GPU priority depends on the GPU.** LOW priority is meant to let other GPU work go first. On the Fold's Adreno 840 it |
| keeps an app's rendering smooth; on the S23's Adreno 740 it does not, and only slows each call. The measurements are |
| with GLiNER2.5-Decide's encoder: see |
| [spybyscript/gliner25-decide-litert](https://huggingface.co/spybyscript/gliner25-decide-litert). |
| |
| ## Qualcomm NPU (`npu/`) |
|
|
| **Runtime:** |
| - LiteRT 2.3.2 with `litert-npu-runtime-qualcomm` 2.3.2 and Qualcomm's `qnn-runtime` 2.50.0 (Qualcomm's AI Stack License); |
| - files precompiled on x86-64 Linux with QAIRT 2.50.0, each in one NPU partition (930 of 930 operators); |
| - a file runs only on its chip (`Build.SOC_MODEL`) and with these runtime versions. |
|
|
| Each file is `gliner2_base_encoder_fp16hw.tflite` (signature `t128`) compiled ahead of time, so the phone loads it without |
| compiling. `npu/manifest.json` lists every file's chip, size and SHA-256. Keep the head graphs on the CPU. |
|
|
| | Chip | Snapdragon | Hexagon | File | Run on a phone | |
| |---|---|---|---|---| |
| | SM8850 | 8 Elite Gen 5 | V81 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8850.tflite` | Galaxy Z Fold 8 | |
| | SM8750 | 8 Elite | V79 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8750.tflite` | not yet | |
| | SM8650 | 8 Gen 3 | V75 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8650.tflite` | not yet | |
| | SM8550 | 8 Gen 2 | V73 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8550.tflite` | not yet | |
| | SM8450 | 8 Gen 1 | V69 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8450.tflite` | not yet | |
|
|
| ## Files |
|
|
| - `LICENSE` (Apache-2.0) and `NOTICE.txt` |
| - `model-card.md`, the original card |
| - `config.json`, the contract |
| - `validation.json` and `validation-full.json` |
| - `SHA256SUMS` |
| - `tokenizer/` |
| - `source/` (the checkpoint's configs) |
| - `runtime/` |
| - `npu/` (the precompiled encoder per chip and `npu/manifest.json`) |
|
|
