Text Classification
PEFT
Safetensors
English
safety
guard
agent-safety
agent-trajectories
lora
alert-only
fast
Instructions to use LyzrAI/IntuitionGuard-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use LyzrAI/IntuitionGuard-3B with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("mistralai/Ministral-3-3B-Instruct-2512-BF16") model = PeftModel.from_pretrained(base_model, "LyzrAI/IntuitionGuard-3B") - Notebooks
- Google Colab
- Kaggle
IntuitionGuard-3B: three w25 seed adapters (H54), model card, scorer, evaluation reports
ab3b173 verified |
Download README.md from LyzrAI/IntuitionGuard-3B: direct link, hf CLI and curl.
- Browser
- Download file 9.71 kB
-
https://huggingface.co/LyzrAI/IntuitionGuard-3B/resolve/main/README.md
- Command line
-
hf download hf://LyzrAI/IntuitionGuard-3B/README.md
-
curl -L -o README.md https://huggingface.co/LyzrAI/IntuitionGuard-3B/resolve/main/README.md
9.71 kB
| license: apache-2.0 | |
| base_model: mistralai/Ministral-3-3B-Instruct-2512-BF16 | |
| library_name: peft | |
| pipeline_tag: text-classification | |
| language: | |
| - en | |
| tags: | |
| - safety | |
| - guard | |
| - agent-safety | |
| - agent-trajectories | |
| - lora | |
| - alert-only | |
| - fast | |
| datasets: | |
| - ninty-seven/stepguard_data | |
| - karanxa/agent-action-safety-dataset | |
| # IntuitionGuard-3B | |
| **The fast tier of the IntuitionGuard family.** An alert-only guard for AI-agent trajectories: give it a whole | |
| trajectory (the user's task, the agent's thoughts and tool calls, the tool outputs) and it returns one number, | |
| `p_unsafe`. One forward pass, no generation. It never blocks anything β it is built to raise alerts for a human or | |
| a policy layer to act on. | |
| It is a LoRA adapter (rank 16) on **Ministral-3-3B-Instruct-2512**, trained with the same recipe as | |
| [IntuitionGuard-8B](https://huggingface.co/LyzrAI/IntuitionGuard-8B) β distilling two released open agent guards, **StepGuard-4B** and **AgentDoG1.5-4B** (both | |
| Apache-2.0), mixed with gold labels, on 6,000 permissively licensed trajectories. **No frontier-model outputs were | |
| used.** Three training seeds are released: seed 0 at the repository root, seeds 1 and 2 under `seeds/`. | |
| ## At a glance | |
| | | | | |
| |---|---| | |
| | **Size** | 3B base β **smaller than the 4B open guards** it competes with | | |
| | **Speed** | **~2Γ faster than an 8B guard**; ~53 ms median on an L40S for R-Judge-scale trajectories (single forward pass, no generation) | | |
| | **R-Judge** | AUROC **0.972** (3-seed ensemble), accuracy 88.3β91.1% across seeds | | |
| | **ATBench-Codex** | AUROC **0.909** (ensemble) β above AgentDoG-4B and StepGuard-4B on every seed | | |
| | **ATBench** | AUROC **0.814** (ensemble) β above AgentDoG-4B (+0.22) and AgentDoG1.5-4B (+0.08) | | |
| | **Taught by** | open guards only (StepGuard-4B, AgentDoG1.5-4B) β no frontier models | | |
| **When to use the 3B vs the 8B.** Reach for the 3B when throughput, cost or latency matter: it runs at about half | |
| the 8B's latency and three-quarters of the 4B guards' size, and out-ranks two of the three released 4B guards on the | |
| fresh tool-use benchmarks. Reach for [IntuitionGuard-8B](https://huggingface.co/LyzrAI/IntuitionGuard-8B) when you want the last point of accuracy β the 8B ties | |
| the strongest released open guard on ATBench/Codex ranking where the 3B trails it (see Limitations). | |
| ## What the evidence supports | |
| All numbers are from our own harness, where every released guard was scored on the same items with the prompt from | |
| its own model card. Paired bootstrap, 2,000 resamples, 95% percentile intervals, three training seeds. "Above" means | |
| the interval of the AUROC difference excludes zero. | |
| - **Above AgentDoG-4B and AgentDoG1.5-4B on ATBench ranking, and above AgentDoG-4B and StepGuard-4B on | |
| ATBench-Codex ranking, on every seed** β at three-quarters of their parameter count. | |
| - On ATBench the 3B leads AgentDoG-4B by **+0.217** AUROC and AgentDoG1.5-4B by **+0.076**; on Codex it leads | |
| AgentDoG-4B by **+0.053** and StepGuard-4B by **+0.043**. | |
| - **Ties** StepGuard-4B on ATBench ranking and AgentDoG1.5-4B on Codex ranking (intervals cover zero). | |
| - **R-Judge** AUROC 0.957 / 0.970 / 0.978 across seeds (0.972 ensemble); accuracy 88.3 / 89.7 / 91.1%. | |
| - We **do not** claim the 3B beats every released guard: it trails the strongest (AgentDoG1.5-Unified-4B) on ATBench | |
| (see Limitations). The honest one-line summary is below the table. | |
| ## Results | |
| AUROC / accuracy at threshold 0.5, full sets. Released guards scored in our harness with their model-card prompts. | |
| | Set (n) | AgentDoG-4B | StepGuard-4B | AgentDoG1.5-4B | Unified-4B | **3B seed 0** | seed 1 | seed 2 | **3B ensemble** | | |
| |---|---|---|---|---|---|---|---|---| | |
| | R-Judge (571) | .967 / 88.4 | .932 / 89.5 | .970 / 90.4 | .957 / 90.0 | .957 / 88.3 | .970 / 89.7 | .978 / 91.1 | **.972 / 91.1** | | |
| | ATBench (1,000) | .603 / 60.2 | .787 / 73.0 | .741 / 67.4 | .844 / 78.3 | .812 / 69.0 | .811 / 68.7 | .797 / 66.6 | **.814 / 68.6** | | |
| | ATBench-Codex (500) | .833 / 80.2 | .843 / 78.2 | .874 / 77.4 | .893 / 83.2 | .910 / 80.2 | .889 / 77.8 | .900 / 80.0 | **.909 / 79.8** | | |
| | ASSEBench strict (1,476) | .875 / 79.7 | .732 / 68.0 | .871 / 75.9 | .831 / 79.5 | .869 / 66.3 | .859 / 67.5 | .854 / 72.4 | **.866 / 68.8** | | |
| | ATBench-Claw, native (500) | .915 / 85.8 | .886 / 81.2 | .866 / 77.1 | .891 / 88.8 | .904 / 75.4 | .939 / 77.8 | .932 / 77.2 | **.931 / 76.8** | | |
| | TraceSafe-Bench (1,170), AUROC | .460 | .523 | .556 | .580 | .559 | .563 | .607 | **.579** | | |
| **Honest one-line summary.** Above AgentDoG-4B and AgentDoG1.5-4B on ATBench ranking and above AgentDoG-4B and | |
| StepGuard-4B on Codex ranking on every seed; ties StepGuard-4B on ATBench and the Unified 4B on Codex; behind the | |
| Unified 4B on ATBench ranking (by 0.03β0.05 AUROC, 9β12 points at 0.5); R-Judge 88.3β91.1%; **about half the 8B's | |
| latency**. We do not claim "best open guard". | |
| TraceSafe-Bench (web-agent traces) is near chance for every guard we scored, ours included. Treat it as out of scope. | |
| ## Speed | |
| Latency is a single forward pass per trajectory, no generation. Median per trajectory (p50) on one L40S, bf16, with | |
| the 8B on the same box shown in parentheses: | |
| | Set | 3B | 8B | speed-up | | |
| |---|---|---|---| | |
| | R-Judge | **53 ms** | 111 ms | 2.1Γ | | |
| | ASSEBench | **49 ms** | 99 ms | 2.0Γ | | |
| | ATBench | **89 ms** | 181 ms | 2.0Γ | | |
| | ATBench-Codex | **77 ms** | 158 ms | 2.1Γ | | |
| The ensemble is three passes. Needs one GPU with about 8 GB free in bf16. | |
| ## How it was trained | |
| Identical recipe to IntuitionGuard-8B, on the 3B base: | |
| - **Base:** `mistralai/Ministral-3-3B-Instruct-2512-BF16` in bf16. LoRA r = 16, Ξ± = 32, dropout 0.05, on the language | |
| model's q, k, v, o, gate, up and down projections. | |
| - **Pool:** 6,000 trajectories: 4,000 from StepGuard-Data (`ninty-seven/stepguard_data`, Apache-2.0), 1,500 from | |
| `karanxa/agent-action-safety-dataset` (Apache-2.0), 500 from AgentDrift (CC BY 4.0). Decontaminated against every | |
| evaluation set by exact task match, 13-gram overlap and embedding similarity of the task description. | |
| - **Targets:** 0.25 Γ gold label + 0.75 Γ the mean `p_unsafe` of StepGuard-4B and AgentDoG1.5-4B on that trajectory. | |
| Loss: soft cross-entropy on the two-way answer mass (safe vs unsafe) at the verdict token. | |
| - **Schedule:** one epoch, learning rate 1e-4, seeds 0, 1, 2. | |
| - **Evaluation sets** (never used for training or for choosing anything): R-Judge, ASSEBench (strict), ATBench, | |
| ATBench-Codex, ATBench-Claw, TraceSafe-Bench. R-Judge has no licence and was used for evaluation only. | |
| - No frontier-model (OpenAI, Google, Anthropic) outputs were used to train these weights. | |
| ## Usage | |
| `score.py` in this repository is the complete scorer (shared with the 8B; it takes the base as an argument). It reads | |
| trajectories in the AgentDoG / ATBench "contents" format (a user turn, then agent turns with `thought` and `action`, | |
| and environment turns with the tool output): | |
| ```bash | |
| pip install "transformers>=5.0" "peft>=0.17" torch | |
| BASE=mistralai/Ministral-3-3B-Instruct-2512-BF16 | |
| python score.py --input trajectories.jsonl --output scores.jsonl --base $BASE # seed 0 | |
| python score.py --input trajectories.jsonl --adapters . seeds/s1 seeds/s2 --base $BASE # 3-seed ensemble | |
| ``` | |
| Each output row is `{"id", "p_unsafe", "truncated"}`. The evaluated operating point is `p_unsafe >= 0.5`. A threshold | |
| fitted on StepGuard's validation split (β0.75) raises ATBench and Codex accuracy and lowers R-Judge and ASSEBench | |
| accuracy; pick the threshold on your own labelled traffic, never on a benchmark you report. | |
| In Python: | |
| ```python | |
| import torch | |
| from transformers import AutoModelForImageTextToText, AutoTokenizer | |
| from peft import PeftModel | |
| base = "mistralai/Ministral-3-3B-Instruct-2512-BF16" | |
| tok = AutoTokenizer.from_pretrained(base, fix_mistral_regex=True) | |
| model = PeftModel.from_pretrained(AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map="cuda"), | |
| "LyzrAI/IntuitionGuard-3B") # subfolder="seeds/s1" for another seed | |
| ``` | |
| and build the prompt with the functions in `score.py`. | |
| ## Limitations | |
| - **Alert-only by design.** It scores trajectories; it does not localise the unsafe step and it does not explain. | |
| - **Behind the strongest released guard on ATBench.** The Unified 4B leads the 3B on ATBench ranking by 0.03β0.05 | |
| AUROC and by 9β12 points at the 0.5 threshold. If your traffic looks like ATBench and you need the top of the | |
| leaderboard, use IntuitionGuard-8B (which ties the Unified there) β that is the whole point of the size ladder. | |
| - **Size costs a little accuracy.** The 3B is 0.01β0.04 AUROC behind the 8B on the public sets; you trade it for ~2Γ | |
| throughput. Below 8B this cost is real (unlike 8Bβ14B, which bought nothing). | |
| - **Fixed-threshold accuracy trails** the strongest released guard on ATBench and ASSEBench; measure before choosing. | |
| - **Web-agent traces (TraceSafe) are out of scope**: near chance, like every other released guard we scored. | |
| - **English, text-only.** Trained on agent trajectories with tool calls. | |
| - Three seeds vary by up to 0.02 AUROC on a set; the ensemble removes that variance. | |
| ## Disclosure | |
| Base model Ministral-3-3B-Instruct-2512 (Mistral AI, Apache-2.0). Teachers StepGuard-4B and AgentDoG1.5-4B | |
| (Apache-2.0). Prompt format from StepGuard. Rival numbers were measured by us in a single harness; published figures | |
| are quoted where they exist. All evaluation reports, including the paired tests and the failed pre-registered claims, | |
| are in `evaluation/`. | |
| ## Licence | |
| Apache-2.0 for these adapter weights and `score.py`. The base model and the teachers are Apache-2.0; the training data | |
| licences are listed above. | |