IntuitionGuard-3B

The fast tier of the IntuitionGuard family. An alert-only guard for AI-agent trajectories: give it a whole trajectory (the user's task, the agent's thoughts and tool calls, the tool outputs) and it returns one number, p_unsafe. One forward pass, no generation. It never blocks anything β€” it is built to raise alerts for a human or a policy layer to act on.

It is a LoRA adapter (rank 16) on Ministral-3-3B-Instruct-2512, trained with the same recipe as IntuitionGuard-8B β€” distilling two released open agent guards, StepGuard-4B and AgentDoG1.5-4B (both Apache-2.0), mixed with gold labels, on 6,000 permissively licensed trajectories. No frontier-model outputs were used. Three training seeds are released: seed 0 at the repository root, seeds 1 and 2 under seeds/.

At a glance

Size 3B base β€” smaller than the 4B open guards it competes with
Speed ~2Γ— faster than an 8B guard; ~53 ms median on an L40S for R-Judge-scale trajectories (single forward pass, no generation)
R-Judge AUROC 0.972 (3-seed ensemble), accuracy 88.3–91.1% across seeds
ATBench-Codex AUROC 0.909 (ensemble) β€” above AgentDoG-4B and StepGuard-4B on every seed
ATBench AUROC 0.814 (ensemble) β€” above AgentDoG-4B (+0.22) and AgentDoG1.5-4B (+0.08)
Taught by open guards only (StepGuard-4B, AgentDoG1.5-4B) β€” no frontier models

When to use the 3B vs the 8B. Reach for the 3B when throughput, cost or latency matter: it runs at about half the 8B's latency and three-quarters of the 4B guards' size, and out-ranks two of the three released 4B guards on the fresh tool-use benchmarks. Reach for IntuitionGuard-8B when you want the last point of accuracy β€” the 8B ties the strongest released open guard on ATBench/Codex ranking where the 3B trails it (see Limitations).

What the evidence supports

All numbers are from our own harness, where every released guard was scored on the same items with the prompt from its own model card. Paired bootstrap, 2,000 resamples, 95% percentile intervals, three training seeds. "Above" means the interval of the AUROC difference excludes zero.

  • Above AgentDoG-4B and AgentDoG1.5-4B on ATBench ranking, and above AgentDoG-4B and StepGuard-4B on ATBench-Codex ranking, on every seed β€” at three-quarters of their parameter count.
  • On ATBench the 3B leads AgentDoG-4B by +0.217 AUROC and AgentDoG1.5-4B by +0.076; on Codex it leads AgentDoG-4B by +0.053 and StepGuard-4B by +0.043.
  • Ties StepGuard-4B on ATBench ranking and AgentDoG1.5-4B on Codex ranking (intervals cover zero).
  • R-Judge AUROC 0.957 / 0.970 / 0.978 across seeds (0.972 ensemble); accuracy 88.3 / 89.7 / 91.1%.
  • We do not claim the 3B beats every released guard: it trails the strongest (AgentDoG1.5-Unified-4B) on ATBench (see Limitations). The honest one-line summary is below the table.

Results

AUROC / accuracy at threshold 0.5, full sets. Released guards scored in our harness with their model-card prompts.

Set (n) AgentDoG-4B StepGuard-4B AgentDoG1.5-4B Unified-4B 3B seed 0 seed 1 seed 2 3B ensemble
R-Judge (571) .967 / 88.4 .932 / 89.5 .970 / 90.4 .957 / 90.0 .957 / 88.3 .970 / 89.7 .978 / 91.1 .972 / 91.1
ATBench (1,000) .603 / 60.2 .787 / 73.0 .741 / 67.4 .844 / 78.3 .812 / 69.0 .811 / 68.7 .797 / 66.6 .814 / 68.6
ATBench-Codex (500) .833 / 80.2 .843 / 78.2 .874 / 77.4 .893 / 83.2 .910 / 80.2 .889 / 77.8 .900 / 80.0 .909 / 79.8
ASSEBench strict (1,476) .875 / 79.7 .732 / 68.0 .871 / 75.9 .831 / 79.5 .869 / 66.3 .859 / 67.5 .854 / 72.4 .866 / 68.8
ATBench-Claw, native (500) .915 / 85.8 .886 / 81.2 .866 / 77.1 .891 / 88.8 .904 / 75.4 .939 / 77.8 .932 / 77.2 .931 / 76.8
TraceSafe-Bench (1,170), AUROC .460 .523 .556 .580 .559 .563 .607 .579

Honest one-line summary. Above AgentDoG-4B and AgentDoG1.5-4B on ATBench ranking and above AgentDoG-4B and StepGuard-4B on Codex ranking on every seed; ties StepGuard-4B on ATBench and the Unified 4B on Codex; behind the Unified 4B on ATBench ranking (by 0.03–0.05 AUROC, 9–12 points at 0.5); R-Judge 88.3–91.1%; about half the 8B's latency. We do not claim "best open guard".

TraceSafe-Bench (web-agent traces) is near chance for every guard we scored, ours included. Treat it as out of scope.

Speed

Latency is a single forward pass per trajectory, no generation. Median per trajectory (p50) on one L40S, bf16, with the 8B on the same box shown in parentheses:

Set 3B 8B speed-up
R-Judge 53 ms 111 ms 2.1Γ—
ASSEBench 49 ms 99 ms 2.0Γ—
ATBench 89 ms 181 ms 2.0Γ—
ATBench-Codex 77 ms 158 ms 2.1Γ—

The ensemble is three passes. Needs one GPU with about 8 GB free in bf16.

How it was trained

Identical recipe to IntuitionGuard-8B, on the 3B base:

  • Base: mistralai/Ministral-3-3B-Instruct-2512-BF16 in bf16. LoRA r = 16, Ξ± = 32, dropout 0.05, on the language model's q, k, v, o, gate, up and down projections.
  • Pool: 6,000 trajectories: 4,000 from StepGuard-Data (ninty-seven/stepguard_data, Apache-2.0), 1,500 from karanxa/agent-action-safety-dataset (Apache-2.0), 500 from AgentDrift (CC BY 4.0). Decontaminated against every evaluation set by exact task match, 13-gram overlap and embedding similarity of the task description.
  • Targets: 0.25 Γ— gold label + 0.75 Γ— the mean p_unsafe of StepGuard-4B and AgentDoG1.5-4B on that trajectory. Loss: soft cross-entropy on the two-way answer mass (safe vs unsafe) at the verdict token.
  • Schedule: one epoch, learning rate 1e-4, seeds 0, 1, 2.
  • Evaluation sets (never used for training or for choosing anything): R-Judge, ASSEBench (strict), ATBench, ATBench-Codex, ATBench-Claw, TraceSafe-Bench. R-Judge has no licence and was used for evaluation only.
  • No frontier-model (OpenAI, Google, Anthropic) outputs were used to train these weights.

Usage

score.py in this repository is the complete scorer (shared with the 8B; it takes the base as an argument). It reads trajectories in the AgentDoG / ATBench "contents" format (a user turn, then agent turns with thought and action, and environment turns with the tool output):

pip install "transformers>=5.0" "peft>=0.17" torch
BASE=mistralai/Ministral-3-3B-Instruct-2512-BF16
python score.py --input trajectories.jsonl --output scores.jsonl --base $BASE            # seed 0
python score.py --input trajectories.jsonl --adapters . seeds/s1 seeds/s2 --base $BASE    # 3-seed ensemble

Each output row is {"id", "p_unsafe", "truncated"}. The evaluated operating point is p_unsafe >= 0.5. A threshold fitted on StepGuard's validation split (β‰ˆ0.75) raises ATBench and Codex accuracy and lowers R-Judge and ASSEBench accuracy; pick the threshold on your own labelled traffic, never on a benchmark you report.

In Python:

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel
base = "mistralai/Ministral-3-3B-Instruct-2512-BF16"
tok = AutoTokenizer.from_pretrained(base, fix_mistral_regex=True)
model = PeftModel.from_pretrained(AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map="cuda"),
                                  "LyzrAI/IntuitionGuard-3B")           # subfolder="seeds/s1" for another seed

and build the prompt with the functions in score.py.

Limitations

  • Alert-only by design. It scores trajectories; it does not localise the unsafe step and it does not explain.
  • Behind the strongest released guard on ATBench. The Unified 4B leads the 3B on ATBench ranking by 0.03–0.05 AUROC and by 9–12 points at the 0.5 threshold. If your traffic looks like ATBench and you need the top of the leaderboard, use IntuitionGuard-8B (which ties the Unified there) β€” that is the whole point of the size ladder.
  • Size costs a little accuracy. The 3B is 0.01–0.04 AUROC behind the 8B on the public sets; you trade it for ~2Γ— throughput. Below 8B this cost is real (unlike 8Bβ†’14B, which bought nothing).
  • Fixed-threshold accuracy trails the strongest released guard on ATBench and ASSEBench; measure before choosing.
  • Web-agent traces (TraceSafe) are out of scope: near chance, like every other released guard we scored.
  • English, text-only. Trained on agent trajectories with tool calls.
  • Three seeds vary by up to 0.02 AUROC on a set; the ensemble removes that variance.

Disclosure

Base model Ministral-3-3B-Instruct-2512 (Mistral AI, Apache-2.0). Teachers StepGuard-4B and AgentDoG1.5-4B (Apache-2.0). Prompt format from StepGuard. Rival numbers were measured by us in a single harness; published figures are quoted where they exist. All evaluation reports, including the paired tests and the failed pre-registered claims, are in evaluation/.

Licence

Apache-2.0 for these adapter weights and score.py. The base model and the teachers are Apache-2.0; the training data licences are listed above.

Downloads last month
15
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for LyzrAI/IntuitionGuard-3B

Datasets used to train LyzrAI/IntuitionGuard-3B