Instructions to use LyzrAI/IntuitionGuard-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use LyzrAI/IntuitionGuard-3B with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("mistralai/Ministral-3-3B-Instruct-2512-BF16") model = PeftModel.from_pretrained(base_model, "LyzrAI/IntuitionGuard-3B") - Notebooks
- Google Colab
- Kaggle
IntuitionGuard-3B
The fast tier of the IntuitionGuard family. An alert-only guard for AI-agent trajectories: give it a whole
trajectory (the user's task, the agent's thoughts and tool calls, the tool outputs) and it returns one number,
p_unsafe. One forward pass, no generation. It never blocks anything β it is built to raise alerts for a human or
a policy layer to act on.
It is a LoRA adapter (rank 16) on Ministral-3-3B-Instruct-2512, trained with the same recipe as
IntuitionGuard-8B β distilling two released open agent guards, StepGuard-4B and AgentDoG1.5-4B (both
Apache-2.0), mixed with gold labels, on 6,000 permissively licensed trajectories. No frontier-model outputs were
used. Three training seeds are released: seed 0 at the repository root, seeds 1 and 2 under seeds/.
At a glance
| Size | 3B base β smaller than the 4B open guards it competes with |
| Speed | ~2Γ faster than an 8B guard; ~53 ms median on an L40S for R-Judge-scale trajectories (single forward pass, no generation) |
| R-Judge | AUROC 0.972 (3-seed ensemble), accuracy 88.3β91.1% across seeds |
| ATBench-Codex | AUROC 0.909 (ensemble) β above AgentDoG-4B and StepGuard-4B on every seed |
| ATBench | AUROC 0.814 (ensemble) β above AgentDoG-4B (+0.22) and AgentDoG1.5-4B (+0.08) |
| Taught by | open guards only (StepGuard-4B, AgentDoG1.5-4B) β no frontier models |
When to use the 3B vs the 8B. Reach for the 3B when throughput, cost or latency matter: it runs at about half the 8B's latency and three-quarters of the 4B guards' size, and out-ranks two of the three released 4B guards on the fresh tool-use benchmarks. Reach for IntuitionGuard-8B when you want the last point of accuracy β the 8B ties the strongest released open guard on ATBench/Codex ranking where the 3B trails it (see Limitations).
What the evidence supports
All numbers are from our own harness, where every released guard was scored on the same items with the prompt from its own model card. Paired bootstrap, 2,000 resamples, 95% percentile intervals, three training seeds. "Above" means the interval of the AUROC difference excludes zero.
- Above AgentDoG-4B and AgentDoG1.5-4B on ATBench ranking, and above AgentDoG-4B and StepGuard-4B on ATBench-Codex ranking, on every seed β at three-quarters of their parameter count.
- On ATBench the 3B leads AgentDoG-4B by +0.217 AUROC and AgentDoG1.5-4B by +0.076; on Codex it leads AgentDoG-4B by +0.053 and StepGuard-4B by +0.043.
- Ties StepGuard-4B on ATBench ranking and AgentDoG1.5-4B on Codex ranking (intervals cover zero).
- R-Judge AUROC 0.957 / 0.970 / 0.978 across seeds (0.972 ensemble); accuracy 88.3 / 89.7 / 91.1%.
- We do not claim the 3B beats every released guard: it trails the strongest (AgentDoG1.5-Unified-4B) on ATBench (see Limitations). The honest one-line summary is below the table.
Results
AUROC / accuracy at threshold 0.5, full sets. Released guards scored in our harness with their model-card prompts.
| Set (n) | AgentDoG-4B | StepGuard-4B | AgentDoG1.5-4B | Unified-4B | 3B seed 0 | seed 1 | seed 2 | 3B ensemble |
|---|---|---|---|---|---|---|---|---|
| R-Judge (571) | .967 / 88.4 | .932 / 89.5 | .970 / 90.4 | .957 / 90.0 | .957 / 88.3 | .970 / 89.7 | .978 / 91.1 | .972 / 91.1 |
| ATBench (1,000) | .603 / 60.2 | .787 / 73.0 | .741 / 67.4 | .844 / 78.3 | .812 / 69.0 | .811 / 68.7 | .797 / 66.6 | .814 / 68.6 |
| ATBench-Codex (500) | .833 / 80.2 | .843 / 78.2 | .874 / 77.4 | .893 / 83.2 | .910 / 80.2 | .889 / 77.8 | .900 / 80.0 | .909 / 79.8 |
| ASSEBench strict (1,476) | .875 / 79.7 | .732 / 68.0 | .871 / 75.9 | .831 / 79.5 | .869 / 66.3 | .859 / 67.5 | .854 / 72.4 | .866 / 68.8 |
| ATBench-Claw, native (500) | .915 / 85.8 | .886 / 81.2 | .866 / 77.1 | .891 / 88.8 | .904 / 75.4 | .939 / 77.8 | .932 / 77.2 | .931 / 76.8 |
| TraceSafe-Bench (1,170), AUROC | .460 | .523 | .556 | .580 | .559 | .563 | .607 | .579 |
Honest one-line summary. Above AgentDoG-4B and AgentDoG1.5-4B on ATBench ranking and above AgentDoG-4B and StepGuard-4B on Codex ranking on every seed; ties StepGuard-4B on ATBench and the Unified 4B on Codex; behind the Unified 4B on ATBench ranking (by 0.03β0.05 AUROC, 9β12 points at 0.5); R-Judge 88.3β91.1%; about half the 8B's latency. We do not claim "best open guard".
TraceSafe-Bench (web-agent traces) is near chance for every guard we scored, ours included. Treat it as out of scope.
Speed
Latency is a single forward pass per trajectory, no generation. Median per trajectory (p50) on one L40S, bf16, with the 8B on the same box shown in parentheses:
| Set | 3B | 8B | speed-up |
|---|---|---|---|
| R-Judge | 53 ms | 111 ms | 2.1Γ |
| ASSEBench | 49 ms | 99 ms | 2.0Γ |
| ATBench | 89 ms | 181 ms | 2.0Γ |
| ATBench-Codex | 77 ms | 158 ms | 2.1Γ |
The ensemble is three passes. Needs one GPU with about 8 GB free in bf16.
How it was trained
Identical recipe to IntuitionGuard-8B, on the 3B base:
- Base:
mistralai/Ministral-3-3B-Instruct-2512-BF16in bf16. LoRA r = 16, Ξ± = 32, dropout 0.05, on the language model's q, k, v, o, gate, up and down projections. - Pool: 6,000 trajectories: 4,000 from StepGuard-Data (
ninty-seven/stepguard_data, Apache-2.0), 1,500 fromkaranxa/agent-action-safety-dataset(Apache-2.0), 500 from AgentDrift (CC BY 4.0). Decontaminated against every evaluation set by exact task match, 13-gram overlap and embedding similarity of the task description. - Targets: 0.25 Γ gold label + 0.75 Γ the mean
p_unsafeof StepGuard-4B and AgentDoG1.5-4B on that trajectory. Loss: soft cross-entropy on the two-way answer mass (safe vs unsafe) at the verdict token. - Schedule: one epoch, learning rate 1e-4, seeds 0, 1, 2.
- Evaluation sets (never used for training or for choosing anything): R-Judge, ASSEBench (strict), ATBench, ATBench-Codex, ATBench-Claw, TraceSafe-Bench. R-Judge has no licence and was used for evaluation only.
- No frontier-model (OpenAI, Google, Anthropic) outputs were used to train these weights.
Usage
score.py in this repository is the complete scorer (shared with the 8B; it takes the base as an argument). It reads
trajectories in the AgentDoG / ATBench "contents" format (a user turn, then agent turns with thought and action,
and environment turns with the tool output):
pip install "transformers>=5.0" "peft>=0.17" torch
BASE=mistralai/Ministral-3-3B-Instruct-2512-BF16
python score.py --input trajectories.jsonl --output scores.jsonl --base $BASE # seed 0
python score.py --input trajectories.jsonl --adapters . seeds/s1 seeds/s2 --base $BASE # 3-seed ensemble
Each output row is {"id", "p_unsafe", "truncated"}. The evaluated operating point is p_unsafe >= 0.5. A threshold
fitted on StepGuard's validation split (β0.75) raises ATBench and Codex accuracy and lowers R-Judge and ASSEBench
accuracy; pick the threshold on your own labelled traffic, never on a benchmark you report.
In Python:
import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel
base = "mistralai/Ministral-3-3B-Instruct-2512-BF16"
tok = AutoTokenizer.from_pretrained(base, fix_mistral_regex=True)
model = PeftModel.from_pretrained(AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map="cuda"),
"LyzrAI/IntuitionGuard-3B") # subfolder="seeds/s1" for another seed
and build the prompt with the functions in score.py.
Limitations
- Alert-only by design. It scores trajectories; it does not localise the unsafe step and it does not explain.
- Behind the strongest released guard on ATBench. The Unified 4B leads the 3B on ATBench ranking by 0.03β0.05 AUROC and by 9β12 points at the 0.5 threshold. If your traffic looks like ATBench and you need the top of the leaderboard, use IntuitionGuard-8B (which ties the Unified there) β that is the whole point of the size ladder.
- Size costs a little accuracy. The 3B is 0.01β0.04 AUROC behind the 8B on the public sets; you trade it for ~2Γ throughput. Below 8B this cost is real (unlike 8Bβ14B, which bought nothing).
- Fixed-threshold accuracy trails the strongest released guard on ATBench and ASSEBench; measure before choosing.
- Web-agent traces (TraceSafe) are out of scope: near chance, like every other released guard we scored.
- English, text-only. Trained on agent trajectories with tool calls.
- Three seeds vary by up to 0.02 AUROC on a set; the ensemble removes that variance.
Disclosure
Base model Ministral-3-3B-Instruct-2512 (Mistral AI, Apache-2.0). Teachers StepGuard-4B and AgentDoG1.5-4B
(Apache-2.0). Prompt format from StepGuard. Rival numbers were measured by us in a single harness; published figures
are quoted where they exist. All evaluation reports, including the paired tests and the failed pre-registered claims,
are in evaluation/.
Licence
Apache-2.0 for these adapter weights and score.py. The base model and the teachers are Apache-2.0; the training data
licences are listed above.
- Downloads last month
- 15
Model tree for LyzrAI/IntuitionGuard-3B
Base model
mistralai/Ministral-3-3B-Base-2512
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("mistralai/Ministral-3-3B-Instruct-2512-BF16") model = PeftModel.from_pretrained(base_model, "LyzrAI/IntuitionGuard-3B")