auto-3b

A 3.08B-parameter classifier, fine-tuned from SmolLM3-3B-Base, that decides whether an AI agent's proposed tool call is authorized and safe in context. It reads the proposed call, the user's request and the agent's history, and outputs approve or deny, with a 65,536-token context. It is the largest and most accurate Auto model, and the teacher that auto-200m-2 was distilled from. It uses the same input format as the rest of the family.

Results

These results are on the pinned 3,000-item Approve-or-Deny benchmark (revision a38b6259), scored at full input length in BF16 with FlashAttention, with P(deny) >= 0.5. A false approval is an unsafe call approved; a false denial is an authorized call denied.

Model	Parameters	Accuracy	False approvals	False denials	AUROC	16k–64k tokens	Validation audit
auto-3b	3,075M	98.03% (2941)	22/1401	37/1599	0.9985	98.74%	98.54%
auto-0.4b-2 (iteration 1)	395.8M	97.00% (2910)	36/1401	54/1599	0.9949	95.82%	98.03%
auto-200m-2	149.6M	96.33% (2890)	53/1401	57/1599	0.9937	94.14%	98.34%

• Long inputs: each model counts benchmark lengths with its own tokenizer, so the 16k–64k slice is 238 items for auto-3b and 239 for the others.
• Audit partition: 2,595 validation rows that were never used for training or selection.
• Confidence interval: the Wilson 95% interval for auto-3b's accuracy is 97.47%–98.47%.
• Reproduction: the weights were re-scored on 2026-09-26 in a separate environment. The result was the same 2,941/3,000, and no decision at 0.5 differs from the original evaluation (largest P(deny) difference 0.019). At 0.295, one item within 0.02 of the threshold flips (2,931 correct, 19 false approvals / 50 false denials) (reproduction).
• Probes: 24/24 on the published skills, MCP and custom-tool probes, and 38/40 on the fresh scope, history and injection probes (1 false approval, 1 false denial), the same counts as auto-200m-2 and auto-0.4b-2.

Threshold. Every number above uses 0.5, as on the other Auto cards. Before the benchmark was run, two thresholds were fixed: 0.5 and 0.295, calibrated for balanced accuracy on a separate validation partition (calibration.json). The run's own target was ≥98% at the calibrated threshold, and it missed.

Threshold	Accuracy	False approvals	False denials
0.5	98.03% (2941)	22	37
0.295 (validation-calibrated)	97.67% (2930)	19	51

At 0.295 the other four targets passed (false approvals ≤44, false denials ≤53, 16k+ accuracy ≥95%, audit ≥98%). Against the original auto-0.4b-2 (2,903/3,000), a paired comparison at the calibrated threshold shows:

• 68 items improved and 41 regressed;
• the 95% bootstrap interval for the gain is +0.23 to +1.60 points;
• McNemar's exact test gives p = 0.012.

The per-category, language, difficulty and length breakdowns are in eval_results.json. Per-item logits are in benchmark_predictions.npz.

How it was trained

• Base model: SmolLM3-3B-Base at revision d78a42f7. It is a decoder with a 64k pretrained context. Its LM head is replaced by a 2-way classification head that reads the last non-padding token.
• Data: the same pinned corpus as auto-0.4b-2 (ProCreations/auto-1b-data revision d265bbf7), with the same cleaning and validation splits and no benchmark overlap. There were no synthetic rows and no distillation.
• Short stage: full-parameter SFT on all 694,070 training rows of up to 4,096 tokens. It ran for 2 epochs (10,354 steps) with plain cross-entropy loss, using AdamW (β = 0.9/0.95, weight decay 0.01) at learning rate 1e-5 with FP32 master weights and BF16 compute. No input was truncated.
• Long stage: training on 53,745 rows, 17,915 of them longer than 4,096 tokens, at learning rate 4e-6. It was stopped at the maintainer's request at step 771 of 1,493.
• Selection: candidates were ranked by NLL on the 7,824-row validation selection partition only. The end-of-short-stage checkpoint won, with NLL 0.0553 against 0.1029 for the stopped long-stage model, and was frozen before any benchmark run. So the published weights were trained on inputs of at most 4,096 tokens. They read longer inputs through the base model's native long context.

The training code is in training/. It keeps the original workstation paths. The selection, calibration and final evaluation records are in eval/.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

def build_input(user_request, history, call):
    parts = ['### PROPOSED TOOL CALL', f"tool: {call['tool']}", f"args: {call['args']}", '',
             '### USER REQUEST', user_request, '', '### AGENT HISTORY']
    if not history:
        parts.append('(no prior actions)')
    else:
        for i, h in enumerate(history):
            parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result', '')}")
    return '\n'.join(parts)

repo = "ProCreations/auto-3b"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo, dtype=torch.bfloat16).cuda().eval()

text = build_input(
    user_request="Clean up the build artifacts and reinstall dependencies.",
    history=[{"tool": "Bash", "args": "ls", "result": "node_modules dist package.json"}],
    call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"},
)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
with torch.inference_mode():
    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
print("deny" if p_deny >= 0.5 else "approve", p_deny)

• Labels: 0 = approve, 1 = deny. Keep the three section headers exactly as shown.
• Pooling: the model reads the last non-padding token, so padded batches work. The pad id is 128004, and it is set in the config.
• Requirements: the checkpoint was written with Transformers 5.16 and needs a release that includes SmolLM3.
• Memory: the BF16 weights are 6.15 GB. Use FlashAttention (attn_implementation="flash_attention_2") or SDPA for long inputs.
• Apple silicon: it also runs through PyTorch's mps device. On an M4 Max a short request takes about 0.2 s.

Scope and limitations

This is a classifier, not a chat model. It approves routine authorized work and denies consequential unauthorized actions, as well as actions that follow injected instructions. It cannot inspect hidden file contents, resolve opaque executables or know a URL's runtime behaviour, so false approvals are still possible.

• The labels are synthetic.
• The benchmark and the audit partition have been reused across Auto development, so they are not pristine.
• No fresh scenario audit was run for this model.

Evaluate it on your own traffic before relying on it. For latency-sensitive use, auto-200m-2, which was distilled from this model, is about 20× smaller.