Instructions to use LyzrAI/IntuitionGuard-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use LyzrAI/IntuitionGuard-8B with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("mistralai/Ministral-3-8B-Instruct-2512-BF16") model = PeftModel.from_pretrained(base_model, "LyzrAI/IntuitionGuard-8B") - Notebooks
- Google Colab
- Kaggle
IntuitionGuard-8B
An alert-only guard for AI-agent trajectories. Give it a whole trajectory (the user's task, the agent's thoughts and
tool calls, the tool outputs) and it returns one number, p_unsafe. It never blocks anything: it is built to raise
alerts for a human or a policy layer to act on.
It is a LoRA adapter (rank 16) on Ministral-3-8B-Instruct-2512 (Mistral AI, Apache-2.0), trained by distilling two
released open agent guards, StepGuard-4B and AgentDoG1.5-4B (both Apache-2.0), mixed with gold labels, on
6,000 permissively licensed trajectories. No frontier-model outputs were used. Three training seeds are released:
seed 0 at the repository root (the release checkpoint), seeds 1 and 2 under seeds/. Averaging the three is an
optional ensemble mode at three times the compute.
At a glance
| Role | the accuracy tier of the IntuitionGuard family |
| R-Judge | accuracy 92.3% at 0.5 (3-seed ensemble) — above the best published open guard (92.2); AUROC 0.979 |
| ATBench-Codex | AUROC 0.920 (ensemble) |
| Ranking | ties the strongest released open guard (AgentDoG1.5-Unified-4B) on ATBench and ATBench-Codex, and leads it on R-Judge |
| Speed | single forward pass, no generation — ~100–180 ms per trajectory on an L40S |
| Taught by | open guards only (StepGuard-4B, AgentDoG1.5-4B) — no frontier models |
When to use the 8B vs the 3B. The 8B is for the last point of accuracy: it ties the strongest released open guard on the tool-use benchmarks and leads on R-Judge. If throughput, cost or latency matter more, the IntuitionGuard-3B runs at about half the latency and three-quarters of the 4B guards' size, giving up only 0.01–0.04 AUROC on the public sets.
What the evidence supports
All numbers are from our own harness, where every released guard was scored on the same items with the prompt from its own model card. Paired bootstrap, 2,000 resamples, 95% percentile intervals, three training seeds. "Above" means the interval of the AUROC difference excludes zero.
- Above AgentDoG-4B, StepGuard-4B and AgentDoG1.5-4B in ranking (AUROC) on ATBench and ATBench-Codex on every seed (18 of 18 intervals positive), and above StepGuard-4B on R-Judge on every seed.
- Above AgentDoG1.5-Unified-Qwen3.5-4B, the strongest released open guard, on R-Judge ranking on every seed (+.015 to +.027).
- Ties the Unified 4B on ATBench and ATBench-Codex ranking (the intervals cover zero on every seed), and trails it in accuracy at the fixed 0.5 threshold on ATBench (by 5 to 10 points) and on ASSEBench (by 4 to 13 points).
- Against AgentDoG-4B and AgentDoG1.5-4B on R-Judge: above on seed 0, ties on seeds 1 and 2.
- The three-seed ensemble reaches R-Judge 92.3% accuracy at 0.5, above the best published number among released open guards (92.2, AgentDoG-4B), with its R-Judge AUROC interval against AgentDoG-4B covering zero by .001.
So: tied for the best released open agent guard on ATBench and ATBench-Codex ranking, ahead on R-Judge ranking, behind at a fixed threshold. We do not claim "best open guard".
Results
AUROC / accuracy at threshold 0.5, full sets. Released guards scored in our harness with their model-card prompts.
| Set (n) | AgentDoG-4B | StepGuard-4B | AgentDoG1.5-4B | AgentDoG1.5-Unified-4B | seed 0 | seed 1 | seed 2 | 3-seed ensemble |
|---|---|---|---|---|---|---|---|---|
| R-Judge (571) | .967 / 88.4 | .932 / 89.5 | .970 / 90.4 | .957 / 90.0 | .985 / 90.9 | .979 / 91.6 | .972 / 92.8 | .979 / 92.3 |
| ATBench (1,000) | .603 / 60.2 | .787 / 73.0 | .741 / 67.4 | .844 / 78.3 | .823 / 71.9 | .823 / 67.9 | .836 / 73.0 | .834 / 71.3 |
| ATBench-Codex (500) | .833 / 80.2 | .843 / 78.2 | .874 / 77.4 | .893 / 83.2 | .918 / 82.8 | .904 / 78.0 | .911 / 81.0 | .920 / 81.4 |
| ASSEBench strict (1,476) | .875 / 79.7 | .732 / 68.0 | .871 / 75.9 | .831 / 79.5 | .877 / 66.1 | .871 / 72.8 | .882 / 75.3 | .881 / 71.6 |
| ATBench-Claw, native format (500) | .915 / 85.8 | .886 / 81.2 | .866 / 77.1 | .891 / 88.8 | .915 / 81.2 | .928 / 78.8 | .915 / 76.6 | .921 / 78.8 |
| TraceSafe-Bench (1,170), AUROC | .460 | .523 | .556 (n = 1,122) | .580 | .525 | .563 | .566 | .550 |
Reproduction of the published numbers in our harness: StepGuard-4B R-Judge 89.5 (published 89.2 to 89.6), AgentDoG1.5-4B 90.4, the Unified 4B R-Judge 90.0 and ATBench 78.3 (published 90.4 and 78.4). AgentDoG-4B scores 88.4 on R-Judge in our harness against its published 92.2; we report our measurement and the published figure both.
TraceSafe-Bench (web-agent traces with twelve failure categories) is near chance for every guard we scored, ours included. Treat it as out of scope for this model.
Latency: one forward pass per trajectory, no generation. Median per trajectory in bf16: about 100 to 180 ms on an L40S and 320 to 600 ms on an A10G for the main sets (prompts up to 8,192 tokens); the ensemble is three passes.
How it was trained
- Base:
mistralai/Ministral-3-8B-Instruct-2512in bf16 (the-BF16weights). LoRA r = 16, α = 32, dropout 0.05, on the language model's q, k, v, o, gate, up and down projections; the vision tower is untouched. - Pool: 6,000 trajectories: 4,000 from StepGuard-Data (
ninty-seven/stepguard_data, Apache-2.0), 1,500 fromkaranxa/agent-action-safety-dataset(Apache-2.0), 500 from AgentDrift (CC BY 4.0). The pool was decontaminated against every evaluation set by exact task match, 13-gram overlap and embedding similarity of the task description. - Targets: for each trajectory, 0.25 × gold label + 0.75 × the mean
p_unsafeof StepGuard-4B and AgentDoG1.5-4B on that trajectory. Loss: soft cross-entropy on the two-way answer mass (safe vs unsafe) at the verdict token. - Schedule: one epoch, learning rate 1e-4, 3,072 training tokens per example, gradient accumulation 8, seeds 0, 1, 2. About 3.6 h per seed on one A10G (seeds 0 and 2) or 1.3 h on one L40S (seed 1).
- Evaluation sets (never used for training or for choosing anything): R-Judge, ASSEBench (strict labels), ATBench, ATBench-Codex, ATBench-Claw, TraceSafe-Bench. R-Judge has no licence and was used for evaluation only.
- No frontier-model (OpenAI, Google, Anthropic) outputs were used to train these weights.
How it works
The prompt is the StepGuard trajectory-audit prompt (the serialisation of the trajectory and the instructions,
Apache-2.0) under a one-line system prompt, followed by <Analysis></Analysis>\n<Judgment>. The model is not asked
to write an analysis: p_unsafe is the probability mass of the unsafe answer tokens among the safe and unsafe
answer tokens at the next position. One forward pass, no sampling. Prompts over 8,192 tokens are shortened by
middle-truncating the longest tool output (score.py does exactly what the evaluation did).
Usage
score.py in this repository is the complete scorer. It takes trajectories in the AgentDoG / ATBench "contents"
format (a user turn, then agent turns with thought and action, and environment turns with the tool output):
{"id": "t1",
"profile": "optional system prompt of the audited agent",
"tools": [{"name": "send_email", "description": "Send an email to a recipient"}],
"contents": [[
{"role": "user", "content": "Forward the Q3 report to the finance team."},
{"role": "agent", "thought": "I will look up the finance list first.", "action": "{\"name\": \"read_contacts\", \"arguments\": {\"group\": \"finance\"}}"},
{"role": "environment", "content": "finance@example.com"},
{"role": "agent", "thought": "", "action": "{\"name\": \"send_email\", \"arguments\": {\"to\": \"finance@example.com\", \"attachment\": \"q3_report.pdf\"}}"},
{"role": "environment", "content": "sent"}
]]}
pip install "transformers>=5.0" "peft>=0.17" torch
python score.py --input trajectories.jsonl --output scores.jsonl # seed 0
python score.py --input trajectories.jsonl --adapters . seeds/s1 seeds/s2 # the three-seed ensemble
Each output row is {"id", "p_unsafe", "truncated"}. The evaluated operating point is p_unsafe >= 0.5. A threshold
fitted on StepGuard's validation split (0.68 to 0.77 depending on the seed) raises ATBench and ATBench-Codex accuracy
and lowers R-Judge and ASSEBench accuracy; pick the threshold on your own labelled traffic, never on a benchmark you
report. Needs one GPU with about 20 GB free in bf16.
In Python:
import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel
base = "mistralai/Ministral-3-8B-Instruct-2512-BF16"
tok = AutoTokenizer.from_pretrained(base, fix_mistral_regex=True)
model = PeftModel.from_pretrained(AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map="cuda"),
"LyzrAI/IntuitionGuard-8B") # subfolder="seeds/s1" for another seed
and build the prompt with the functions in score.py.
Limitations
- Alert-only by design. It scores complete or partial trajectories; it does not localise the unsafe step and it does not explain. Use it to raise alerts for review, not to block actions automatically.
- Fixed-threshold accuracy trails the strongest released guard on ATBench and ASSEBench. If your use needs a hard yes/no at a fixed cut on those distributions, measure before choosing.
- Web-agent traces (TraceSafe) are out of scope: near chance, like every other released guard we scored.
- English, text-only. Trained on agent trajectories with tool calls; chat-only moderation (plain harmful requests without tools) is covered only through R-Judge-style cases.
- Context. Prompts are capped at 8,192 tokens; very long trajectories are middle-truncated.
- Three seeds vary by up to .014 AUROC and up to 9 accuracy points on a set; the ensemble removes that variance and adds no accuracy beyond the best seed.
Disclosure
Base model Ministral-3-8B-Instruct-2512 (Mistral AI). Teachers StepGuard-4B and AgentDoG1.5-4B (Apache-2.0 weights);
their released training data was not used beyond the StepGuard-Data split named above. Prompt format from StepGuard.
Rival numbers were measured by us; published figures are quoted where they exist. All evaluation reports, including
the paired tests and the failed pre-registered claims, are in evaluation/.
Licence
Apache-2.0 for these adapter weights and score.py. The base model and the teachers are Apache-2.0; the training data
licences are listed above.
- Downloads last month
- 8
Model tree for LyzrAI/IntuitionGuard-8B
Base model
mistralai/Ministral-3-8B-Base-2512
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("mistralai/Ministral-3-8B-Instruct-2512-BF16") model = PeftModel.from_pretrained(base_model, "LyzrAI/IntuitionGuard-8B")