Add onnx files

LICENSE

@@ -0,0 +1,5 @@
+LICENSE
+
+This model is licensed under the nvidia-open-model-license.
+
+License text: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/

README.md

@@ -1,193 +1,53 @@
 ---
-library_name: gliner
-license: other
-license_name: nvidia-open-model-license
-license_link: >-
-  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
-
-pipeline_tag: token-classification
-language:
-  - en
+library_name: glitext
+license: apache-2.0
 tags:
-  - nvidia
-  - pytorch
-  - PII
-  - PHI
-  - GLiNER
-  - information extraction
-  - entity recognition
-  - privacy
+- glitext
+glitext:
+  name: pii-large
+  label: GliText PII Recognition (Accurate)
+  description: An efficient zero-shot named entity recognition model tuned for PII extraction with high accuracy.
+  recognition: true
+  classification: false
+  association: false
+  span_mode: true
+  size_gb: 1.7
+  hf_repo: rpeel/glitext-pii-large
+  source_url: nvidia/gliner-PII
 ---
 
-# GLiNER-PII Model Overview
-
-### Description:
-GLiNER-PII is inspired by the Gretel GLiNER PII/PHI models. Built on the GLiNER large-v2.1 base, it detects and classifies a broad range of Personally Identifiable Information (PII) and Protected Health Information (PHI) in structured and unstructured text. It is non-generative and produces span-level entity annotations with confidence scores across 55+ categories. This model was developed by NVIDIA.
-
-This model is ready for commercial/non-commercial use. <br>
-
-### License/Terms of Use
-Use of this model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). <br>
-
-### Deployment Geography:
-Global <br>
-
-### Use Case: <br>
-GLiNER-PII supports detection and redaction of sensitive information across regulated and enterprise scenarios. 
-
-- **Healthcare**: Redact PHI in clinical notes, reports, and medical documents.
-- **Finance**: Identify account numbers, SSNs, and transaction details in banking and insurance documents.
-- **Legal**: Protect client information in contracts, filings, and discovery materials.
-- **Enterprise Data Governance**: Scan documents, emails, and data stores for sensitive information.
-- **Data Privacy Compliance**: Support GDPR, HIPAA, and CCPA workflows across varied document types.
-- **Cybersecurity**: Detect sensitive data in logs, security reports, and incident records.
-- **Content Moderation**: Flag personal information in user-generated content.
-
-Note: performance varies by domain, format, and threshold, so validation and human review are recommended for high‑stakes deployments. <br>
-
-### Release Date:  <br>
-Hugging Face 10/28/2025 via https://huggingface.co/nvidia/gliner-pii <br> 
-
-## References:
-- GLiNER base (Hugging Face): https://huggingface.co/urchade/gliner_large-v2.1
-- Gretel GLiNER PII/PHI models: https://huggingface.co/gretelai/gretel-gliner-bi-large-v1.0
-- Training dataset: https://huggingface.co/datasets/nvidia/nemotron-pii
-- GLiNER library: https://pypi.org/project/gliner/
-
-## Model Architecture:
-**Architecture Type:** Transformer <br>
-
-**Network Architecture:** GLiNER <br>
-
-**This model was developed based on urchade/gliner_large-v2.1** <br> 
-**Number of model parameters: 5.7 × 10^8** <br>
-
-## Input: <br>
-**Input Type(s):** Text <br>
-**Input Format:** UTF-8 string(s) <br>
-**Input Parameters:** One-Dimensional (1D) <br>
-**Other Properties Related to Input:** supports structured and unstructured text <br>
-
-## Output: <br>
-**Output Type(s):** Text <br>
-**Output Format:** String <br>
-**Output Parameters:** One-Dimensional (1D) <br>
-**Other Properties Related to Output:** List of dictionaries with keys {text, label, start, end, score} <br>
-
-## Software Integration:
-**Runtime Engine(s):** 
-* PyTorch, GLiNER Python library <br> 
-
-**Supported Hardware Microarchitecture Compatibility:** <br>
-* NVIDIA Ampere <br>
-* NVIDIA Blackwell <br>
-* NVIDIA Hopper <br>
-* NVIDIA Lovelace <br>
-* NVIDIA Pascal <br>
-* NVIDIA Turing <br>
-* NVIDIA Volta <br>
-* CPU (x86_64) <br>
+# rpeel/glitext-pii-large
 
-**Preferred/Supported Operating System(s):**
-* Linux <br>
+An efficient zero-shot named entity recognition model tuned for PII extraction with high accuracy.
 
-The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. <br>
+## Requirements
 
-## Model Version(s):
-- nvidia/gliner-pii
-- Version: v1.0
+To download this model to the SAS GLiText server:
 
-## Training and Evaluation Datasets:
-
-### Training Dataset
-
-**Link:** [nvidia/nemotron-pii](https://huggingface.co/datasets/nvidia/nemotron-pii) <br>
-**Data Modality:** Text <br>
-**Text Training Data Size:** \~100k records (\~10^5, <1B tokens) <br>
-**Data Collection Method:** Synthetic <br>
-**Labeling Method:** Synthetic <br>
-
-**Properties:**
-Synthetic persona-grounded dataset generated with NVIDIA NeMo Data Designer, spanning 50+ industries and 55+ entity types (U.S. and international formats). Includes both structured and unstructured records. Labels automatically injected during generation.
-
-## Evaluation Datasets
-
-* [Argilla PII](https://huggingface.co/argilla)
-* [AI4Privacy](https://huggingface.co/ai4privacy)
-* [Gretel PII Dataset V1/V2](https://huggingface.co/datasets/gretelai/gretel-pii-masking-en-v1)
-
-**Data Collection Method:** Hybrid: Automated, Human <br>
-**Labeling Method:** Hybrid: Automated, Human <br>
-
-**Evaluation Results** <br>
-From the combined evaluation across Argilla, AI4Privacy, and Gretel PII datasets:
-
-| Benchmark           |   Strict F1  |
-| --------------------| -----------: |
-| Argilla PII         |         0.70 |
-| AI4Privacy          |         0.64 |
-| nvidia/Nemotron-PII |         0.87 |
----
-We evaluated the model using `threshold=0.3`. <br>
-
-# Inference:
-**Acceleration Engine:** PyTorch (via Hugging Face Transformers) <br>
-**Test Hardware:** NVIDIA A100 (Ampere, PCIe/SXM) <br>
-
-# Usage Recommendation
+```
+POST /v1/models/download?name=pii-large
+```
 
-First, make sure you have the gliner library installed:
+To download and load into memory in one step:
 
 ```
-pip install gliner
+PUT /v1/models?name=pii-large
 ```
-Now, let's try to find an email, SSN, and phone number in a messy block of text.
 
-```
-from gliner import GLiNER
-# 1. Define our new text
-text = "Hi support, I can't log in! My account username is 'johndoe88'. Every time I try, it says 'invalid credentials'. Please reset my password. You can reach me at (555) 123-4567 or johnd@example.com"
+## Source Model
 
-# 2. Define the labels we're hunting for.
-labels = ["email", "phone_number", "user_name"]
+Exported from [nvidia/gliner-PII](https://huggingface.co/nvidia/gliner-PII).
+See the [original model card](https://huggingface.co/nvidia/gliner-PII) for full architecture and training details.
 
-# 3. Load the PII model
-model = GLiNER.from_pretrained("nvidia/gliner-pii")
+## Security Scan
 
-# 4. Run the prediction at given threshold
-entities = model.predict_entities(text, labels, threshold=0.5)
-```
+Scanned with [modelaudit](https://github.com/promptfoo/modelaudit) v0.2.40 on 2026-04-27. 29/29 checks passed. [Full results](modelaudit.json).
 
-Sample output:
-```
-[
-  {
-    "start": 52, 
-    "end": 61, 
-    "text": "johndoe88", 
-    "label": "user_name",
-    "score": 0.99
-  },
-  {
-    "start": 159, 
-    "end": 173, 
-    "text": "(555) 123-4567", 
-    "label": "phone_number", 
-    "score": 0.99
-  },
-  {
-    "start": 177, 
-    "end": 194, 
-    "text": "johnd@example.com", 
-    "label": "email", 
-    "score": 0.99
-  }
-]
-```
 
-## Ethical Considerations:
-NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br> 
+| File | Size | SHA-256 |
+|------|------|---------|
+| `model.onnx` | 1784.6 MB | `e6f5db22bf3a2c06…` |
 
-For more detailed information on ethical considerations for this model, please see the Bias, Explainability, Safety & Security, and Privacy Subcards. <br> 
+## License
 
-Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).  <br>
+[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). Derived from [nvidia/gliner-PII](https://huggingface.co/nvidia/gliner-PII) by [nvidia](https://huggingface.co/nvidia).

gliner_config.json

@@ -7,21 +7,13 @@
     "add_cross_attention": false,
     "architectures": null,
     "attention_probs_dropout_prob": 0.1,
-    "bad_words_ids": null,
-    "begin_suppress_tokens": null,
     "bos_token_id": null,
     "chunk_size_feed_forward": 0,
     "cross_attention_hidden_size": null,
     "decoder_start_token_id": null,
-    "diversity_penalty": 0.0,
-    "do_sample": false,
-    "early_stopping": false,
-    "encoder_no_repeat_ngram_size": 0,
+    "dtype": null,
     "eos_token_id": null,
-    "exponential_decay_length_penalty": null,
     "finetuning_task": null,
-    "forced_bos_token_id": null,
-    "forced_eos_token_id": null,
     "hidden_act": "gelu",
     "hidden_dropout_prob": 0.1,
     "hidden_size": 1024,
@@ -38,22 +30,15 @@
       "LABEL_1": 1
     },
     "layer_norm_eps": 1e-07,
-    "length_penalty": 1.0,
-    "max_length": 20,
+    "legacy": true,
     "max_position_embeddings": 512,
     "max_relative_positions": -1,
-    "min_length": 0,
     "model_type": "deberta-v2",
-    "no_repeat_ngram_size": 0,
     "norm_rel_ebd": "layer_norm",
     "num_attention_heads": 16,
-    "num_beam_groups": 1,
-    "num_beams": 1,
     "num_hidden_layers": 24,
-    "num_return_sequences": 1,
     "output_attentions": false,
     "output_hidden_states": false,
-    "output_scores": false,
     "pad_token_id": 0,
     "pooler_dropout": 0,
     "pooler_hidden_act": "gelu",
@@ -68,25 +53,16 @@
     "problem_type": null,
     "pruned_heads": {},
     "relative_attention": true,
-    "remove_invalid_values": false,
-    "repetition_penalty": 1.0,
     "return_dict": true,
-    "return_dict_in_generate": false,
     "sep_token_id": null,
     "share_att_key": true,
-    "suppress_tokens": null,
     "task_specific_params": null,
-    "temperature": 1.0,
     "tf_legacy_loss": false,
     "tie_encoder_decoder": false,
     "tie_word_embeddings": true,
     "tokenizer_class": null,
-    "top_k": 50,
-    "top_p": 1.0,
-    "torch_dtype": null,
     "torchscript": false,
     "type_vocab_size": 0,
-    "typical_p": 1.0,
     "use_bfloat16": false,
     "vocab_size": 128004
   },
@@ -105,19 +81,24 @@
   "max_types": 25,
   "max_width": 12,
   "model_name": "microsoft/deberta-v3-large",
-  "model_type": "gliner",
+  "model_type": null,
   "name": "correct",
+  "neg_spans_ratio": 1.0,
   "num_post_fusion_layers": 1,
+  "num_rnn_layers": 1,
   "num_steps": 30000,
   "post_fusion_schema": "",
   "random_drop": true,
+  "represent_spans": false,
   "sep_token": "<<SEP>>",
   "shuffle_types": true,
   "size_sup": -1,
+  "span_loss_coef": 1.0,
   "span_mode": "markerV0",
   "subtoken_pooling": "first",
+  "token_loss_coef": 1.0,
   "train_batch_size": 8,
-  "transformers_version": "4.45.2",
+  "transformers_version": "5.1.0",
   "vocab_size": 128004,
   "warmup_ratio": 3000,
   "words_splitter_type": "whitespace"

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6f5db22bf3a2c0608bae20d80b02eb18e8e6b278ddabc43c176f3c408aafb5f
+size 1784572593

modelaudit.json

@@ -0,0 +1,695 @@
+{
+  "tool": "modelaudit",
+  "tool_version": "0.2.40",
+  "scanned_at": "2026-04-27T01:48:47Z",
+  "files": {
+    "model.onnx": {
+      "size_mb": 1784.6,
+      "sha256": "e6f5db22bf3a2c0608bae20d80b02eb18e8e6b278ddabc43c176f3c408aafb5f"
+    }
+  },
+  "audit": {
+    "bytes_scanned": 1795692820,
+    "issues": [
+      {
+        "message": "Weight distribution analysis skipped one or more eligible ONNX initializers",
+        "severity": "info",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {
+          "scan_outcome_reason": "onnx_weight_distribution_analysis_incomplete",
+          "coverage_gap": "partial_initializer_coverage",
+          "eligible_initializers": 158,
+          "analyzed_initializers": 157,
+          "external_initializers_skipped": 0,
+          "oversized_initializers_skipped": 1,
+          "extraction_failures": 0,
+          "max_array_size": 104857600
+        },
+        "timestamp": 1777254517.375655,
+        "type": "onnx_check",
+        "rule_code": "S902"
+      },
+      {
+        "message": "File does not contain expected XGBoost binary model markers",
+        "severity": "info",
+        "location": "/opt/sas/model-gli-text/models/pii-large/spm.model",
+        "details": {
+          "expected_patterns": [
+            "binf",
+            "gbtree",
+            "gblinear",
+            "dart",
+            "reg:",
+            "binary:",
+            "multi:"
+          ]
+        },
+        "why": "Missing XGBoost markers may indicate a truncated, corrupted, or mislabeled model file",
+        "timestamp": 1777254523.9662325,
+        "type": "xgboost_check"
+      }
+    ],
+    "checks": [
+      {
+        "name": "Path Exists",
+        "status": "passed",
+        "message": "Path exists",
+        "location": "/opt/sas/model-gli-text/models/pii-large/README.md",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/README.md"
+        },
+        "timestamp": 1777253507.7469912
+      },
+      {
+        "name": "Path Readable",
+        "status": "passed",
+        "message": "Path is readable",
+        "location": "/opt/sas/model-gli-text/models/pii-large/README.md",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/README.md"
+        },
+        "timestamp": 1777253507.7470188
+      },
+      {
+        "name": "File Type Validation",
+        "status": "passed",
+        "message": "File type validation passed",
+        "location": "/opt/sas/model-gli-text/models/pii-large/README.md",
+        "details": {},
+        "timestamp": 1777253507.7470477
+      },
+      {
+        "name": "Path Exists",
+        "status": "passed",
+        "message": "Path exists",
+        "location": "/opt/sas/model-gli-text/models/pii-large/gliner_config.json",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/gliner_config.json"
+        },
+        "timestamp": 1777253507.8376338
+      },
+      {
+        "name": "Path Readable",
+        "status": "passed",
+        "message": "Path is readable",
+        "location": "/opt/sas/model-gli-text/models/pii-large/gliner_config.json",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/gliner_config.json"
+        },
+        "timestamp": 1777253507.8376603
+      },
+      {
+        "name": "File Type Validation",
+        "status": "passed",
+        "message": "File type validation passed",
+        "location": "/opt/sas/model-gli-text/models/pii-large/gliner_config.json",
+        "details": {},
+        "timestamp": 1777253507.837688
+      },
+      {
+        "name": "Model Name Policy Check",
+        "status": "passed",
+        "message": "Model Name Policy Check completed successfully",
+        "location": "/opt/sas/model-gli-text/models/pii-large/gliner_config.json",
+        "details": {
+          "component_count": 2
+        },
+        "timestamp": 1777253507.839352
+      },
+      {
+        "name": "Path Exists",
+        "status": "passed",
+        "message": "Path exists",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/model.onnx"
+        },
+        "timestamp": 1777253507.869744
+      },
+      {
+        "name": "Path Readable",
+        "status": "passed",
+        "message": "Path is readable",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/model.onnx"
+        },
+        "timestamp": 1777253507.869775
+      },
+      {
+        "name": "File Type Validation",
+        "status": "passed",
+        "message": "File type validation passed",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {},
+        "timestamp": 1777253507.8698015
+      },
+      {
+        "name": "File Integrity Hash",
+        "status": "passed",
+        "message": "File integrity hashes calculated",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {
+          "md5": "715d740b8e15bd272bca06334ec5827f",
+          "sha256": "e6f5db22bf3a2c0608bae20d80b02eb18e8e6b278ddabc43c176f3c408aafb5f",
+          "sha512": "150c51adb5a0806140ccd55812fafcb4d564de332248b7963a697db6d9f429b18f9fd538d26e3f179c1982d0861a0134cba6411b2b768ee0597579fe399a8873",
+          "file_size": 1784572593
+        },
+        "timestamp": 1777253516.9992
+      },
+      {
+        "name": "JIT/Script Code Execution Detection",
+        "status": "passed",
+        "message": "No JIT/Script code execution risks detected",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {},
+        "timestamp": 1777254512.1935756
+      },
+      {
+        "name": "Network Communication Detection",
+        "status": "passed",
+        "message": "No network communication patterns detected",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {},
+        "timestamp": 1777254512.1937573
+      },
+      {
+        "name": "Custom Operator Domain Check",
+        "status": "passed",
+        "message": "All operators use standard ONNX domains",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {
+          "safe_nodes": 9913
+        },
+        "timestamp": 1777254512.239958
+      },
+      {
+        "name": "Python Operator Detection",
+        "status": "passed",
+        "message": "No Python operators detected",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {
+          "nodes_checked": 9913
+        },
+        "timestamp": 1777254512.2400036
+      },
+      {
+        "name": "Tensor Size Validation",
+        "status": "passed",
+        "message": "Tensor Size Validation completed successfully",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx (tensor: core.token_rep_layer.bert_layer.model.embeddings.word_embeddings.weight)",
+        "details": {
+          "component_count": 411
+        },
+        "timestamp": 1777254515.519565
+      },
+      {
+        "name": "Weight Distribution Analysis Coverage",
+        "status": "failed",
+        "message": "Weight distribution analysis skipped one or more eligible ONNX initializers",
+        "severity": "info",
+        "location": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "details": {
+          "scan_outcome_reason": "onnx_weight_distribution_analysis_incomplete",
+          "coverage_gap": "partial_initializer_coverage",
+          "eligible_initializers": 158,
+          "analyzed_initializers": 157,
+          "external_initializers_skipped": 0,
+          "oversized_initializers_skipped": 1,
+          "extraction_failures": 0,
+          "max_array_size": 104857600
+        },
+        "rule_code": "S902",
+        "timestamp": 1777254517.3755178
+      },
+      {
+        "name": "Path Exists",
+        "status": "passed",
+        "message": "Path exists",
+        "location": "/opt/sas/model-gli-text/models/pii-large/spm.model",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/spm.model"
+        },
+        "timestamp": 1777254523.9484646
+      },
+      {
+        "name": "Path Readable",
+        "status": "passed",
+        "message": "Path is readable",
+        "location": "/opt/sas/model-gli-text/models/pii-large/spm.model",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/spm.model"
+        },
+        "timestamp": 1777254523.9484947
+      },
+      {
+        "name": "File Type Validation",
+        "status": "passed",
+        "message": "File type validation passed",
+        "location": "/opt/sas/model-gli-text/models/pii-large/spm.model",
+        "details": {},
+        "timestamp": 1777254523.9485226
+      },
+      {
+        "name": "File Integrity Hash",
+        "status": "passed",
+        "message": "File integrity hashes calculated",
+        "location": "/opt/sas/model-gli-text/models/pii-large/spm.model",
+        "details": {
+          "md5": "1613fcbf3b82999c187b09c9db79b568",
+          "sha256": "c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd",
+          "sha512": "950cf1b397c25342053efaea576419e104f11e085796d741e5e78d6765ffe5474f4f096ee578099b04a611bde03e12808c51e2b6e89d2962a8665a35da6eb081",
+          "file_size": 2464616
+        },
+        "timestamp": 1777254523.9640443
+      },
+      {
+        "name": "Binary Structure Validation",
+        "status": "failed",
+        "message": "File does not contain expected XGBoost binary model markers",
+        "severity": "info",
+        "location": "/opt/sas/model-gli-text/models/pii-large/spm.model",
+        "details": {
+          "expected_patterns": [
+            "binf",
+            "gbtree",
+            "gblinear",
+            "dart",
+            "reg:",
+            "binary:",
+            "multi:"
+          ]
+        },
+        "why": "Missing XGBoost markers may indicate a truncated, corrupted, or mislabeled model file",
+        "timestamp": 1777254523.9662135
+      },
+      {
+        "name": "XGBoost Loading",
+        "status": "passed",
+        "message": "XGBoost loading disabled (safe mode)",
+        "location": "/opt/sas/model-gli-text/models/pii-large/spm.model",
+        "details": {
+          "safe_mode": true
+        },
+        "timestamp": 1777254523.9664836
+      },
+      {
+        "name": "Path Exists",
+        "status": "passed",
+        "message": "Path exists",
+        "location": "/opt/sas/model-gli-text/models/pii-large/tokenizer.json",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/tokenizer.json"
+        },
+        "timestamp": 1777254526.045291
+      },
+      {
+        "name": "Path Readable",
+        "status": "passed",
+        "message": "Path is readable",
+        "location": "/opt/sas/model-gli-text/models/pii-large/tokenizer.json",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/tokenizer.json"
+        },
+        "timestamp": 1777254526.045329
+      },
+      {
+        "name": "File Type Validation",
+        "status": "passed",
+        "message": "File type validation passed",
+        "location": "/opt/sas/model-gli-text/models/pii-large/tokenizer.json",
+        "details": {},
+        "timestamp": 1777254526.0453677
+      },
+      {
+        "name": "Template Extraction",
+        "status": "passed",
+        "message": "No Jinja2 templates found in file",
+        "location": "/opt/sas/model-gli-text/models/pii-large/tokenizer.json",
+        "details": {
+          "file_type": "tokenizer_config"
+        },
+        "timestamp": 1777254526.4614794
+      },
+      {
+        "name": "Path Exists",
+        "status": "passed",
+        "message": "Path exists",
+        "location": "/opt/sas/model-gli-text/models/pii-large/tokenizer_config.json",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/tokenizer_config.json"
+        },
+        "timestamp": 1777254526.4704058
+      },
+      {
+        "name": "Path Readable",
+        "status": "passed",
+        "message": "Path is readable",
+        "location": "/opt/sas/model-gli-text/models/pii-large/tokenizer_config.json",
+        "details": {
+          "path": "/opt/sas/model-gli-text/models/pii-large/tokenizer_config.json"
+        },
+        "timestamp": 1777254526.470432
+      },
+      {
+        "name": "File Type Validation",
+        "status": "passed",
+        "message": "File type validation passed",
+        "location": "/opt/sas/model-gli-text/models/pii-large/tokenizer_config.json",
+        "details": {},
+        "timestamp": 1777254526.4704616
+      },
+      {
+        "name": "Template Extraction",
+        "status": "passed",
+        "message": "No Jinja2 templates found in file",
+        "location": "/opt/sas/model-gli-text/models/pii-large/tokenizer_config.json",
+        "details": {
+          "file_type": "tokenizer_config"
+        },
+        "timestamp": 1777254526.470916
+      }
+    ],
+    "files_scanned": 9,
+    "assets": [
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/LICENSE",
+        "type": "unknown"
+      },
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/README.md",
+        "type": "metadata",
+        "size": 1438
+      },
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/added_tokens.json",
+        "type": "unknown"
+      },
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/gliner_config.json",
+        "type": "manifest",
+        "size": 2710,
+        "keys": [
+          "class_token_index",
+          "dropout",
+          "embed_ent_token",
+          "encoder_config",
+          "ent_token",
+          "eval_every",
+          "fine_tune",
+          "fuse_layers",
+          "has_rnn",
+          "hidden_size",
+          "labels_encoder",
+          "labels_encoder_config",
+          "lr_encoder",
+          "lr_others",
+          "max_len",
+          "max_neg_type_ratio",
+          "max_types",
+          "max_width",
+          "model_name",
+          "model_type",
+          "name",
+          "neg_spans_ratio",
+          "num_post_fusion_layers",
+          "num_rnn_layers",
+          "num_steps",
+          "post_fusion_schema",
+          "random_drop",
+          "represent_spans",
+          "sep_token",
+          "shuffle_types",
+          "size_sup",
+          "span_loss_coef",
+          "span_mode",
+          "subtoken_pooling",
+          "token_loss_coef",
+          "train_batch_size",
+          "transformers_version",
+          "vocab_size",
+          "warmup_ratio",
+          "words_splitter_type"
+        ]
+      },
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/model.onnx",
+        "type": "onnx",
+        "size": 1784572593
+      },
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/special_tokens_map.json",
+        "type": "unknown"
+      },
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/spm.model",
+        "type": "xgboost",
+        "size": 2464616
+      },
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/tokenizer.json",
+        "type": "jinja2_template",
+        "size": 8649672
+      },
+      {
+        "path": "/opt/sas/model-gli-text/models/pii-large/tokenizer_config.json",
+        "type": "jinja2_template",
+        "size": 564
+      }
+    ],
+    "has_errors": false,
+    "scanner_names": [
+      "metadata",
+      "manifest",
+      "onnx",
+      "xgboost",
+      "jinja2_template"
+    ],
+    "file_metadata": {
+      "/opt/sas/model-gli-text/models/pii-large/LICENSE": {
+        "license_info": [],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": false,
+        "is_model": false,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777253507.7414448,
+        "content_hash": "a9d02548b7b0ac7400816a1a89ad61d1bbc0c5ced16e11f2fe05765f4d81435e"
+      },
+      "/opt/sas/model-gli-text/models/pii-large/README.md": {
+        "file_size": 1438,
+        "license_info": [
+          {
+            "spdx_id": "Apache-2.0",
+            "name": "Apache License 2.0",
+            "confidence": 0.8,
+            "source": "file_header",
+            "commercial_allowed": true
+          }
+        ],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": false,
+        "is_model": false,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777253507.7526639,
+        "content_hash": "2139d5d4e9ba7eb619144420d8bfe443efa0016322b8f433ed68009a5f34c79f"
+      },
+      "/opt/sas/model-gli-text/models/pii-large/added_tokens.json": {
+        "license_info": [],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": true,
+        "is_model": false,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777253507.8301353,
+        "content_hash": "0a4df7f7953a90443fa51b6ca723242f6cb29e7fc34b1f27fc320c8e60de45b7"
+      },
+      "/opt/sas/model-gli-text/models/pii-large/gliner_config.json": {
+        "file_size": 2710,
+        "license_info": [],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": true,
+        "is_model": false,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777253507.8408313,
+        "root_type": "dict",
+        "keys": [
+          "class_token_index",
+          "dropout",
+          "embed_ent_token",
+          "encoder_config",
+          "ent_token",
+          "eval_every",
+          "fine_tune",
+          "fuse_layers",
+          "has_rnn",
+          "hidden_size",
+          "labels_encoder",
+          "labels_encoder_config",
+          "lr_encoder",
+          "lr_others",
+          "max_len",
+          "max_neg_type_ratio",
+          "max_types",
+          "max_width",
+          "model_name",
+          "model_type",
+          "name",
+          "neg_spans_ratio",
+          "num_post_fusion_layers",
+          "num_rnn_layers",
+          "num_steps",
+          "post_fusion_schema",
+          "random_drop",
+          "represent_spans",
+          "sep_token",
+          "shuffle_types",
+          "size_sup",
+          "span_loss_coef",
+          "span_mode",
+          "subtoken_pooling",
+          "token_loss_coef",
+          "train_batch_size",
+          "transformers_version",
+          "vocab_size",
+          "warmup_ratio",
+          "words_splitter_type"
+        ],
+        "content_hash": "831f96d6d04033ead8842f77d6688117fbe5a700de2a29cf4ad525f253bdaca9"
+      },
+      "/opt/sas/model-gli-text/models/pii-large/model.onnx": {
+        "file_size": 1784572593,
+        "file_hashes": {
+          "md5": "715d740b8e15bd272bca06334ec5827f",
+          "sha256": "e6f5db22bf3a2c0608bae20d80b02eb18e8e6b278ddabc43c176f3c408aafb5f",
+          "sha512": "150c51adb5a0806140ccd55812fafcb4d564de332248b7963a697db6d9f429b18f9fd538d26e3f179c1982d0861a0134cba6411b2b768ee0597579fe399a8873"
+        },
+        "license_info": [],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": false,
+        "is_model": true,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777254523.932818,
+        "ir_version": 7,
+        "producer_name": "pytorch",
+        "node_count": 9913,
+        "scan_outcome": "inconclusive",
+        "scan_outcome_reasons": [
+          "onnx_weight_distribution_analysis_incomplete"
+        ],
+        "layers_analyzed": 157,
+        "anomalies_found": 0,
+        "content_hash": "e6f5db22bf3a2c0608bae20d80b02eb18e8e6b278ddabc43c176f3c408aafb5f"
+      },
+      "/opt/sas/model-gli-text/models/pii-large/special_tokens_map.json": {
+        "license_info": [],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": true,
+        "is_model": false,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777254523.9398346,
+        "content_hash": "b2f1b2f15f29a6b6d9d6ea4eca1675d2c231a71477f151d48f79cc83a625ba21"
+      },
+      "/opt/sas/model-gli-text/models/pii-large/spm.model": {
+        "file_size": 2464616,
+        "file_hashes": {
+          "md5": "1613fcbf3b82999c187b09c9db79b568",
+          "sha256": "c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd",
+          "sha512": "950cf1b397c25342053efaea576419e104f11e085796d741e5e78d6765ffe5474f4f096ee578099b04a611bde03e12808c51e2b6e89d2962a8665a35da6eb081"
+        },
+        "license_info": [
+          {
+            "spdx_id": "RAIL",
+            "name": "Responsible AI License",
+            "confidence": 0.8,
+            "source": "file_header",
+            "commercial_allowed": true
+          }
+        ],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": false,
+        "is_model": false,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777254524.0049138,
+        "scan_outcome": "inconclusive",
+        "scan_outcome_reasons": [
+          "xgboost_binary_structure_unrecognized"
+        ],
+        "analysis_incomplete": true,
+        "content_hash": "c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd"
+      },
+      "/opt/sas/model-gli-text/models/pii-large/tokenizer.json": {
+        "file_size": 8649672,
+        "ml_context": {
+          "frameworks": {},
+          "overall_confidence": 0.0,
+          "is_ml_content": false,
+          "detected_patterns": [],
+          "optimization_hints": [],
+          "file_type": "tokenizer_config",
+          "is_tokenizer": true,
+          "confidence": 2
+        },
+        "license_info": [],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": true,
+        "is_model": false,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777254526.4654963,
+        "content_hash": "d448b305cb7be09e3bf95e3852b5c8f4726fafe2f7b56d5cfcf26c1d3ebabe8b"
+      },
+      "/opt/sas/model-gli-text/models/pii-large/tokenizer_config.json": {
+        "file_size": 564,
+        "ml_context": {
+          "frameworks": {},
+          "overall_confidence": 0.0,
+          "is_ml_content": false,
+          "detected_patterns": [],
+          "optimization_hints": [],
+          "file_type": "tokenizer_config",
+          "is_tokenizer": true,
+          "confidence": 2
+        },
+        "license_info": [],
+        "copyright_notices": [],
+        "license_files_nearby": [
+          "/opt/sas/model-gli-text/models/pii-large/LICENSE"
+        ],
+        "is_dataset": true,
+        "is_model": false,
+        "risk_score": 0.0,
+        "scan_timestamp": 1777254526.474051,
+        "content_hash": "1b2caa2d31ebbbf90e61aa8127fadf4f8b930d8b19530e6e88cd32c32418c5cc"
+      }
+    },
+    "content_hash": "78113bd1672c2748e21158ba339dd629fd5fec043042aa9d5030da4fbfdb458b",
+    "start_time": 1777253505.3059766,
+    "duration": 1021.1788341999054,
+    "total_checks": 29,
+    "passed_checks": 29,
+    "failed_checks": 0,
+    "success": false
+  }
+}

tokenizer.json

@@ -156,6 +156,12 @@
           "type_id": 0
         }
       },
+      {
+        "SpecialToken": {
+          "id": "[CLS]",
+          "type_id": 1
+        }
+      },
       {
         "Sequence": {
           "id": "B",

tokenizer_config.json

@@ -1,75 +1,11 @@
 {
-  "added_tokens_decoder": {
-    "0": {
-      "content": "[PAD]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "1": {
-      "content": "[CLS]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "2": {
-      "content": "[SEP]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "3": {
-      "content": "[UNK]",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "128000": {
-      "content": "[MASK]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "128001": {
-      "content": "[FLERT]",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "128002": {
-      "content": "<<ENT>>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "128003": {
-      "content": "<<SEP>>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    }
-  },
+  "backend": "tokenizers",
   "bos_token": "[CLS]",
   "clean_up_tokenization_spaces": false,
   "cls_token": "[CLS]",
   "do_lower_case": false,
   "eos_token": "[SEP]",
+  "is_local": true,
   "mask_token": "[MASK]",
   "max_length": null,
   "model_max_length": 1000000000000000019884624838656,
@@ -80,7 +16,7 @@
   "sep_token": "[SEP]",
   "sp_model_kwargs": {},
   "split_by_punct": false,
-  "tokenizer_class": "DebertaV2Tokenizer",
+  "tokenizer_class": "TokenizersBackend",
   "unk_token": "[UNK]",
   "vocab_type": "spm"
 }