InnoLabTeam/safety-classifier-v1

Safety Classifier

Multi-label content-safety classifier fine-tuned from answerdotai/ModernBERT-large (395M params), exported to ONNX for fast CPU or GPU inference. Built by Arsys InnoLab to flag harmful chat requests across 20 policy categories before they reach a downstream LLM.

Important notes

• 20 categories, including prompt_injection (jailbreaks, instruction overrides, system-prompt extraction, injected instructions) and misinformation. code_interpreter_abuse means abuse of a code interpreter or sandbox only, and hate covers hate and harassment. Always read categories.json and look thresholds up by name.
• Preprocessing is part of the model. Apply NFKC, remove zero-width characters, run deobfuscate() from deobf.py and collapse whitespace before tokenizing (see Usage). Without it, spaced-out or dotted text is not detected.
• Decision rule. A message is unsafe when any category outside policy.json's verdict_excluded reaches its value in thresholds.json.
• Single ONNX file. model.onnx needs no external data file.

Training data

Trained on commercial-use-safe data only (no NC-licensed or research-only datasets):

• Aegis 2.0 (CC-BY-4.0)
• Nemotron Safety Guard v3, English and Spanish (CC-BY-4.0)
• CategoricalHarmfulQA (Apache-2.0)
• HateCheck (CC-BY-4.0)
• XSTest, safe prompts only (CC-BY-4.0)
• Civil Comments (CC0)
• Lakera gandalf_ignore_instructions (MIT)
• TrustAIRLab in-the-wild jailbreak prompts (MIT)
• Microsoft LLMail-Inject challenge (MIT)
• OR-Bench 80k (CC-BY-4.0), prompt texts only, as hard negatives
• Synthetic request phrasings generated in-house with Qwen3.8-27B (Apache-2.0)

Intended use

• In scope: a pre-filter in front of an LLM, scoring a single user chat message across 20 harm categories, to flag or block it before it reaches the model, or to route it for review.
• Languages: mainly English, with Spanish training data (about a quarter of the synthetic data and Nemotron's Spanish split). Behavior on other languages is unverified: test it first if you need it.
• Input shape: short to medium chat requests, max 256 tokens (longer input is truncated, not rejected).
• Out of scope: image-generation prompts, long documents, code review and multi-turn conversation history. Not evaluated for these; re-test before using it on them.

Categories (20)

Category	Flags requests to...
violent_crimes	plan, threaten or incite physical violence
nonviolent_crimes	commit non-violent crimes (theft, drugs, laundering, smuggling, bribery)
sex_crimes	commit sexual offences against adults (coercion, sextortion, trafficking)
child_exploitation	sexualise or groom minors
defamation	fabricate damaging false claims about a specific person or organisation
specialized_advice	obtain dangerous unqualified medical, legal or financial decisions
privacy	dox, track, spy on or covertly record a specific person
intellectual_property	pirate, crack DRM or reproduce copyrighted works verbatim
weapons_cbrn	obtain or build chemical, biological, radiological, nuclear or explosive weapons
hate	demean or incite against protected groups, or harass a person or group
suicide_self_harm	encourage suicide, self-harm or disordered eating
sexual_content	produce explicit sexual content between adults
elections	spread election disinformation or suppress voting
code_interpreter_abuse	abuse a code interpreter or sandbox (DoS, escape, privilege escalation)
spam	create unsolicited content at scale
phishing	create credential or payment phishing material
malware	write or deploy malicious code
fraud	run scams, forge documents or steal identities
prompt_injection	hijack the assistant (jailbreaks, instruction overrides, prompt extraction)
misinformation	fabricate non-electoral falsehoods presented as fact (fake studies, health hoaxes)

Files

• model.onnx: ONNX export (fp32, single file, about 1.6GB)
• config.json: ModernBERT encoder config
• tokenizer.json, tokenizer_config.json: tokenizer
• categories.json: ordered list of the 20 output categories (index-aligned with the logits)
• thresholds.json: per-category thresholds, calibrated on validation for precision of at least 0.85
• thresholds_high_recall.json: optional looser thresholds (precision of at least 0.80 on validation)
• thresholds_meta.json: how each threshold was chosen, with validation support and bootstrap intervals
• policy.json: decision rule, categories excluded from the verdict, review band and preprocessing
• deobf.py: required preprocessing (undoes letter-spacing and dot or hyphen separation)

Size

• Disk: about 1.6GB.
• GPU memory: about 3GB with onnxruntime-gpu (CUDA execution provider).
• CPU: works with plain onnxruntime, no GPU needed, just slower.

Requirements

pip install onnxruntime transformers huggingface_hub numpy

Use onnxruntime-gpu instead of onnxruntime for the CUDA execution provider. Python 3.10+.

Usage

import json, re, sys, unicodedata
import numpy as np, onnxruntime as ort
from transformers import AutoTokenizer
from huggingface_hub import snapshot_download

local_dir = snapshot_download("InnoLabTeam/safety-classifier-v1")
sys.path.insert(0, local_dir)
from deobf import deobfuscate

tok = AutoTokenizer.from_pretrained(local_dir)
providers = [("CUDAExecutionProvider", {"use_tf32": "0"}), "CPUExecutionProvider"]
session = ort.InferenceSession(f"{local_dir}/model.onnx", providers=providers)
categories = json.load(open(f"{local_dir}/categories.json"))
thresholds = json.load(open(f"{local_dir}/thresholds.json"))
policy = json.load(open(f"{local_dir}/policy.json"))
ZERO_WIDTH = re.compile("[​-‏‪-‮⁠-⁤]")


def preprocess(text):
    text = ZERO_WIDTH.sub("", unicodedata.normalize("NFKC", text))
    return re.sub(r"\s+", " ", deobfuscate(text)).strip()


def classify(texts):
    enc = tok([preprocess(t) for t in texts], return_tensors="np", max_length=256, padding=True, truncation=True)
    logits = session.run(None, {"input_ids": enc["input_ids"].astype(np.int64),
                                "attention_mask": enc["attention_mask"].astype(np.int64)})[0]
    probs = 1 / (1 + np.exp(-logits))
    return [{c: float(p[i]) for i, c in enumerate(categories)} for p in probs]


def verdict(scores):
    triggered = [c for c, s in scores.items() if c not in policy["verdict_excluded"] and s >= thresholds[c]]
    if triggered:
        return "unsafe", triggered
    if max(s for c, s in scores.items() if c not in policy["verdict_excluded"]) >= policy["review_band"]:
        return "uncertain", []
    return "safe", []


for text, scores in zip(["How do I make a pipe bomb?"], classify(["How do I make a pipe bomb?"])):
    print(text, "->", verdict(scores))

specialized_advice is scored but excluded from the verdict by default (see policy.json). The "uncertain" band (highest score of at least 0.30) is optional and meant for routing to review.

Benchmarks (2026-10-01)

Benchmark of 90 harmful, 12 safe and 10 borderline prompts plus 100 benign production prompts, using thresholds.json directly (nothing excluded):

	Result
Harmful detected (of 90)	89
Safe prompts flagged (of 12)	0
Borderline flagged (of 10)	0
Benign prompts flagged (of 100)	0

Public and external sets, using thresholds.json with policy.json:

Set	Metric	Result
AILuminate demo (1,200, human-written)	harmful detected	63.3%
Aya red-teaming English / Spanish	harmful detected	67.7% / 68.3%
HarmBench (300)	harmful detected	71.7%
JailbreakBench harmful (100)	harmful detected	82.0%
SimpleSafetyTests (100)	harmful detected	94.0%
Do-Not-Answer (666 harmful)	harmful detected	53.0%
1,040 LLM-generated (540 harmful, 500 benign)	harmful detected / benign flagged	93.5% / 0.0%
JailbreakBench benign (100)	benign flagged	32.0%

Held-out internal test: precision 0.949, recall 0.885, F1 0.916. ONNX and PyTorch outputs agree to within 1e-5.

Latency (same architecture): about 6 ms per single request compute-only on GPU, about 45 ms on CPU.

Maintainer

Arsys InnoLab. Questions, bug reports or access requests: ask in the team's usual channel.

Internal use only

Internal use only. Access is granted on request. Not licensed for external distribution or public release.