| --- |
| license: other |
| license_name: internal-use-only |
| language: |
| - en |
| - es |
| tags: |
| - text-classification |
| - content-moderation |
| - safety |
| - multi-label-classification |
| - onnx |
| - modernbert |
| base_model: answerdotai/ModernBERT-large |
| pipeline_tag: text-classification |
| --- |
| |
| # Safety Classifier |
|
|
| Multi-label content-safety classifier fine-tuned from `answerdotai/ModernBERT-large` (395M params), exported to ONNX |
| for fast CPU or GPU inference. Built by Arsys InnoLab to flag harmful chat requests across 20 policy categories before |
| they reach a downstream LLM. |
|
|
| ## Important notes |
|
|
| - **20 categories**, including `prompt_injection` (jailbreaks, instruction overrides, system-prompt extraction, injected |
| instructions) and `misinformation`. `code_interpreter_abuse` means abuse of a code interpreter or sandbox only, and |
| `hate` covers hate and harassment. Always read `categories.json` and look thresholds up by name. |
| - **Preprocessing is part of the model.** Apply NFKC, remove zero-width characters, run `deobfuscate()` from `deobf.py` |
| and collapse whitespace before tokenizing (see Usage). Without it, spaced-out or dotted text is not detected. |
| - **Decision rule.** A message is unsafe when any category outside `policy.json`'s `verdict_excluded` reaches its |
| value in `thresholds.json`. |
| - **Single ONNX file.** `model.onnx` needs no external data file. |
|
|
| ## Training data |
|
|
| Trained on commercial-use-safe data only (no NC-licensed or research-only datasets): |
|
|
| - Aegis 2.0 (CC-BY-4.0) |
| - Nemotron Safety Guard v3, English and Spanish (CC-BY-4.0) |
| - CategoricalHarmfulQA (Apache-2.0) |
| - HateCheck (CC-BY-4.0) |
| - XSTest, safe prompts only (CC-BY-4.0) |
| - Civil Comments (CC0) |
| - Lakera gandalf_ignore_instructions (MIT) |
| - TrustAIRLab in-the-wild jailbreak prompts (MIT) |
| - Microsoft LLMail-Inject challenge (MIT) |
| - OR-Bench 80k (CC-BY-4.0), prompt texts only, as hard negatives |
| - Synthetic request phrasings generated in-house with Qwen3.8-27B (Apache-2.0) |
|
|
| ## Intended use |
|
|
| - **In scope:** a pre-filter in front of an LLM, scoring a single user chat message across 20 harm categories, to |
| flag or block it before it reaches the model, or to route it for review. |
| - **Languages:** mainly English, with Spanish training data (about a quarter of the synthetic data and Nemotron's |
| Spanish split). Behavior on other languages is unverified: test it first if you need it. |
| - **Input shape:** short to medium chat requests, max 256 tokens (longer input is truncated, not rejected). |
| - **Out of scope:** image-generation prompts, long documents, code review and multi-turn conversation history. Not |
| evaluated for these; re-test before using it on them. |
|
|
| ## Categories (20) |
|
|
| | Category | Flags requests to... | |
| |---|---| |
| | `violent_crimes` | plan, threaten or incite physical violence | |
| | `nonviolent_crimes` | commit non-violent crimes (theft, drugs, laundering, smuggling, bribery) | |
| | `sex_crimes` | commit sexual offences against adults (coercion, sextortion, trafficking) | |
| | `child_exploitation` | sexualise or groom minors | |
| | `defamation` | fabricate damaging false claims about a specific person or organisation | |
| | `specialized_advice` | obtain dangerous unqualified medical, legal or financial decisions | |
| | `privacy` | dox, track, spy on or covertly record a specific person | |
| | `intellectual_property` | pirate, crack DRM or reproduce copyrighted works verbatim | |
| | `weapons_cbrn` | obtain or build chemical, biological, radiological, nuclear or explosive weapons | |
| | `hate` | demean or incite against protected groups, or harass a person or group | |
| | `suicide_self_harm` | encourage suicide, self-harm or disordered eating | |
| | `sexual_content` | produce explicit sexual content between adults | |
| | `elections` | spread election disinformation or suppress voting | |
| | `code_interpreter_abuse` | abuse a code interpreter or sandbox (DoS, escape, privilege escalation) | |
| | `spam` | create unsolicited content at scale | |
| | `phishing` | create credential or payment phishing material | |
| | `malware` | write or deploy malicious code | |
| | `fraud` | run scams, forge documents or steal identities | |
| | `prompt_injection` | hijack the assistant (jailbreaks, instruction overrides, prompt extraction) | |
| | `misinformation` | fabricate non-electoral falsehoods presented as fact (fake studies, health hoaxes) | |
|
|
| ## Files |
|
|
| - `model.onnx`: ONNX export (fp32, single file, about 1.6GB) |
| - `config.json`: ModernBERT encoder config |
| - `tokenizer.json`, `tokenizer_config.json`: tokenizer |
| - `categories.json`: ordered list of the 20 output categories (index-aligned with the logits) |
| - `thresholds.json`: per-category thresholds, calibrated on validation for precision of at least 0.85 |
| - `thresholds_high_recall.json`: optional looser thresholds (precision of at least 0.80 on validation) |
| - `thresholds_meta.json`: how each threshold was chosen, with validation support and bootstrap intervals |
| - `policy.json`: decision rule, categories excluded from the verdict, review band and preprocessing |
| - `deobf.py`: required preprocessing (undoes letter-spacing and dot or hyphen separation) |
|
|
| ## Size |
|
|
| - Disk: about 1.6GB. |
| - GPU memory: about 3GB with `onnxruntime-gpu` (CUDA execution provider). |
| - CPU: works with plain `onnxruntime`, no GPU needed, just slower. |
|
|
| ## Requirements |
|
|
| ``` |
| pip install onnxruntime transformers huggingface_hub numpy |
| ``` |
|
|
| Use `onnxruntime-gpu` instead of `onnxruntime` for the CUDA execution provider. Python 3.10+. |
|
|
| ## Usage |
|
|
| ```python |
| import json, re, sys, unicodedata |
| import numpy as np, onnxruntime as ort |
| from transformers import AutoTokenizer |
| from huggingface_hub import snapshot_download |
| |
| local_dir = snapshot_download("InnoLabTeam/safety-classifier-v1") |
| sys.path.insert(0, local_dir) |
| from deobf import deobfuscate |
| |
| tok = AutoTokenizer.from_pretrained(local_dir) |
| providers = [("CUDAExecutionProvider", {"use_tf32": "0"}), "CPUExecutionProvider"] |
| session = ort.InferenceSession(f"{local_dir}/model.onnx", providers=providers) |
| categories = json.load(open(f"{local_dir}/categories.json")) |
| thresholds = json.load(open(f"{local_dir}/thresholds.json")) |
| policy = json.load(open(f"{local_dir}/policy.json")) |
| ZERO_WIDTH = re.compile("[---]") |
| |
| |
| def preprocess(text): |
| text = ZERO_WIDTH.sub("", unicodedata.normalize("NFKC", text)) |
| return re.sub(r"\s+", " ", deobfuscate(text)).strip() |
| |
| |
| def classify(texts): |
| enc = tok([preprocess(t) for t in texts], return_tensors="np", max_length=256, padding=True, truncation=True) |
| logits = session.run(None, {"input_ids": enc["input_ids"].astype(np.int64), |
| "attention_mask": enc["attention_mask"].astype(np.int64)})[0] |
| probs = 1 / (1 + np.exp(-logits)) |
| return [{c: float(p[i]) for i, c in enumerate(categories)} for p in probs] |
| |
| |
| def verdict(scores): |
| triggered = [c for c, s in scores.items() if c not in policy["verdict_excluded"] and s >= thresholds[c]] |
| if triggered: |
| return "unsafe", triggered |
| if max(s for c, s in scores.items() if c not in policy["verdict_excluded"]) >= policy["review_band"]: |
| return "uncertain", [] |
| return "safe", [] |
| |
| |
| for text, scores in zip(["How do I make a pipe bomb?"], classify(["How do I make a pipe bomb?"])): |
| print(text, "->", verdict(scores)) |
| ``` |
|
|
| `specialized_advice` is scored but excluded from the verdict by default (see `policy.json`). The "uncertain" band |
| (highest score of at least 0.30) is optional and meant for routing to review. |
|
|
| ## Benchmarks (2026-10-01) |
|
|
| Benchmark of 90 harmful, 12 safe and 10 borderline prompts plus 100 benign production prompts, using |
| `thresholds.json` directly (nothing excluded): |
|
|
| | | Result | |
| |---|---| |
| | Harmful detected (of 90) | 89 | |
| | Safe prompts flagged (of 12) | 0 | |
| | Borderline flagged (of 10) | 0 | |
| | Benign prompts flagged (of 100) | 0 | |
|
|
| Public and external sets, using `thresholds.json` with `policy.json`: |
|
|
| | Set | Metric | Result | |
| |---|---|---| |
| | AILuminate demo (1,200, human-written) | harmful detected | 63.3% | |
| | Aya red-teaming English / Spanish | harmful detected | 67.7% / 68.3% | |
| | HarmBench (300) | harmful detected | 71.7% | |
| | JailbreakBench harmful (100) | harmful detected | 82.0% | |
| | SimpleSafetyTests (100) | harmful detected | 94.0% | |
| | Do-Not-Answer (666 harmful) | harmful detected | 53.0% | |
| | 1,040 LLM-generated (540 harmful, 500 benign) | harmful detected / benign flagged | 93.5% / 0.0% | |
| | JailbreakBench benign (100) | benign flagged | 32.0% | |
|
|
| Held-out internal test: precision 0.949, recall 0.885, F1 0.916. ONNX and PyTorch outputs agree to within 1e-5. |
|
|
| Latency (same architecture): about 6 ms per single request compute-only on GPU, about 45 ms on CPU. |
|
|
| ## Maintainer |
|
|
| Arsys InnoLab. Questions, bug reports or access requests: ask in the team's usual channel. |
|
|
| ## Internal use only |
|
|
| Internal use only. Access is granted on request. Not licensed for external distribution or public release. |
|
|
