--- license: apache-2.0 language: - en base_model: fastino/gliner2.5-base-v1 library_name: litert tags: - gliner2 - litert - tflite - android - on-device --- # GLiNER2.5 base (gliner2.5-base-v1) — LiteRT (Android) A LiteRT conversion of [fastino/gliner2.5-base-v1](https://huggingface.co/fastino/gliner2.5-base-v1) (revision `1a8bc24e00dc7300b9017c81d63e3dcdabb26596`, Apache-2.0, by Fastino), converted by spybyscript. It carries every trained head: entities, classification, structures, relations and records. gliner2's own code decodes the outputs. For the model itself (training, labels, benchmarks, intended use), see the [original model card](https://huggingface.co/fastino/gliner2.5-base-v1), copied here as `model-card.md`. The split into graphs follows FluidInference's Core ML port ([gliner2-5-base-coreml](https://huggingface.co/FluidInference/gliner2-5-base-coreml), Apache-2.0). ## Files | File | Size | Contents | |---|---|---| | `gliner2_base_encoder_wfp16.tflite` | 189 MB | encoder for the CPU and the GPU at FP32 (`inputs_embeds`, `attention_mask` → `hidden_states`), float16-stored weights | | `gliner2_base_encoder_fp16hw.tflite` | 189 MB | the same encoder rewritten for FP16 hardware (NPU): LayerNorm power-of-two pre-scale, first feed-forward layer split in two | | `npu/qualcomm/gliner2_base_encoder_fp16hw_.tflite` | 182 MB each | `fp16hw` precompiled for one Snapdragon NPU (below) | | `gliner2_base_head_fp32.tflite` | 8.5 MB | boundary head: routed states, boundary marginals, pool projections, null and count heads, classification logits | | `gliner2_base_scorer_fp32.tflite` | 2.1 MB | candidate scorer and record candidate states | | `gliner2_base_explicit_fp32.tflite` | 2.1 MB | explicit-span scorer (entity attributes, choice fields) | | `gliner2_base_relation_fp32.tflite` | 23.7 MB | relation scorer over head/tail pairs | | `gliner2_base_record_assignment_fp32.tflite` | 1.2 MB | record decoder: instance-to-field assignment, object and latent-seed scores | | `gliner2_base_record_anchorless_fp32.tflite` | 2.9 MB | record decoder for anchorless records | | `gliner2_base_embeddings_f16.bin` | 197 MB | token table, float16 [128011, 768], little-endian, row-major | | `runtime/` | | `litert_boundary_runtime.py` (reference runtime), `verify_parity.py`, `parity_corpus.json` | `config.json` lists every graph's signatures and shapes. ## Use ```python # pip install gliner2==2.0.0 torch numpy ai-edge-litert import sys; sys.path.insert(0, "runtime") from litert_boundary_runtime import LiteRTBoundaryExtractor from gliner2 import Schema model = LiteRTBoundaryExtractor(".") # this repository's folder model.extract_entities("Alice founded Acme in Toronto in 2020.", ["person", "organization", "city"]) model.extract_relations("Alice founded Acme in Toronto.", ["founded", "located_in"]) schema = Schema().entities(["person", "organization"]).relations(["works_for"]) model.extract("Ravi Shah is an engineer at Tidewater Systems.", schema, include_confidence=True) ``` Every public gliner2 call works: `extract`, `extract_entities`, `classify_text`, `extract_json` and `extract_relations`, plus their `batch_*` and `*_long` forms. The runtime runs gliner2's BoundaryExtractor with every trained module on LiteRT. Schema handling, the candidate pool and the decoders are the library's own code. **On a phone,** run the same graphs with LiteRT's `CompiledModel`; the runtime shows the host steps. - **The encoder** is the only graph worth an accelerator. Use the NPU with your chip's precompiled file from `npu/` (or `fp16hw`, which the phone compiles in about 6 s), the GPU with `wfp16` at FP32 (or `fp16hw` at FP16 where it stays finite), or the CPU. - **The six head graphs** run on the CPU in 1–16 ms each. **Fixed sizes.** The original runs with dynamic shapes; these are FluidInference's bucket sizes. One request holds at most: - 128 subword tokens (schema and text) and 64 text words; - 8 queries (entity labels, structure fields, relation roles) and 8 classification choices; - a candidate pool of 192 (the checkpoint's own); - 4 relation types with 256 head/tail pairs; - 64 explicit spans per query; - records of up to 8 fields with 192 candidates each. A larger request is refused with a ValueError, never truncated. Run long text through the `*_long` calls, for example `extract_entities_long(text, labels, chunk_size=40, chunk_overlap=8)`. The weights change only in storage and compute precision. ## Validation `runtime/verify_parity.py` runs each request through gliner2 with the original weights and through the reference runtime (LiteRT CPU), then compares the structured results: - **The corpus** (`runtime/parity_corpus.json`): 296 requests over 16 call types (entities, relations, structures, the three record modes, attributes, single- and multi-label classification, mixed schemas). All 296 are identical, with span F1 1.0 and confidences within 0.0014. - **FluidInference's eleven schema-path fixtures:** 11 of 11. `validation-full.json` and `validation.json` hold the numbers. ## Phones LiteRT 2.3.2 with `qnn-runtime` 2.50.0. Each graph ran on a request recorded from the corpus: 10 calls after a rest, at thermal status 0. The CPU ran 4 threads; the GPU ran at NORMAL priority, timed as whole calls. The NPU compiled the graphs on the phone; for the encoder that takes about 6 s, once. | Graph | Galaxy Z Fold 8 (SM8850): CPU / GPU / NPU | Galaxy S23 Ultra (SM8550): CPU / GPU / NPU | |---|---|---| | encoder, 128 tokens | 38 / 37 (FP32), 20 (FP16) / 9.9 ms | 143 / 67 (FP32) / 13 ms | | head | 0.7 / — / 0.6 ms | 1.2 / — / 1.2 ms | | scorer | 2.6 / — / does not run | 2.9 / — / 3.4 ms | | explicit spans | 5.4 / — / does not run | 5.6 / — / does not run | | relations | 4.9 / — / 1.3 ms | 16 / — / 2.8 ms | | record assignment | 6.0 / not measured / 4.3 ms | 8.9 / 21.5 ms (LOW priority) / 4.0 ms | | anchorless records | 2.9 / — / 0.6 ms | 10.8 / — / 1.3 ms | — means LiteRT's GPU does not support the graph; run it on the CPU. - **The precompiled encoder** (`npu/qualcomm/gliner2_base_encoder_fp16hw_SM8850.tflite`) on the Fold: 8.1 ms a call, loaded in 0.2 s, with 31 of 32 recorded requests unchanged (below), as with the encoder the phone compiled itself. - **A request** with the encoder on the NPU and the heads on the CPU takes about 15 ms on the Fold and 20 ms on the S23. On the CPU alone it takes about 45 ms on the Fold and 150 ms on the S23. - **Answers with the phones' encoder outputs.** 32 recorded requests, two per call type, ran through the rest of the pipeline: - GPU FP32: unchanged on both phones; - NPU: 31 of 32 on the Fold and 30 of 32 on the S23 (only latent-record requests changed); - `fp16hw` at GPU FP16: 31 of 32 on the Fold. On the S23's Adreno 740 it gives non-finite outputs, with or without FP32 accumulation, so use FP32 there. - **Dirty memory, encoder:** about 0.45 GB on the CPU, 0.6 GB on the GPU at FP16 and 0.85 GB at FP32. The NPU holds about 0.5 GB after compiling on the phone. - **GPU priority depends on the GPU.** LOW priority is meant to let other GPU work go first. On the Fold's Adreno 840 it keeps an app's rendering smooth; on the S23's Adreno 740 it does not, and only slows each call. The measurements are with GLiNER2.5-Decide's encoder: see [spybyscript/gliner25-decide-litert](https://huggingface.co/spybyscript/gliner25-decide-litert). ## Qualcomm NPU (`npu/`) **Runtime:** - LiteRT 2.3.2 with `litert-npu-runtime-qualcomm` 2.3.2 and Qualcomm's `qnn-runtime` 2.50.0 (Qualcomm's AI Stack License); - files precompiled on x86-64 Linux with QAIRT 2.50.0, each in one NPU partition (930 of 930 operators); - a file runs only on its chip (`Build.SOC_MODEL`) and with these runtime versions. Each file is `gliner2_base_encoder_fp16hw.tflite` (signature `t128`) compiled ahead of time, so the phone loads it without compiling. `npu/manifest.json` lists every file's chip, size and SHA-256. Keep the head graphs on the CPU. | Chip | Snapdragon | Hexagon | File | Run on a phone | |---|---|---|---|---| | SM8850 | 8 Elite Gen 5 | V81 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8850.tflite` | Galaxy Z Fold 8 | | SM8750 | 8 Elite | V79 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8750.tflite` | not yet | | SM8650 | 8 Gen 3 | V75 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8650.tflite` | not yet | | SM8550 | 8 Gen 2 | V73 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8550.tflite` | not yet | | SM8450 | 8 Gen 1 | V69 | `npu/qualcomm/gliner2_base_encoder_fp16hw_SM8450.tflite` | not yet | ## Files - `LICENSE` (Apache-2.0) and `NOTICE.txt` - `model-card.md`, the original card - `config.json`, the contract - `validation.json` and `validation-full.json` - `SHA256SUMS` - `tokenizer/` - `source/` (the checkpoint's configs) - `runtime/` - `npu/` (the precompiled encoder per chip and `npu/manifest.json`)