qm-shell-judge-int8

A shell-command risk judge for AI coding agents. Given a command an agent wants to run, it scores 12 yes/no risk questions; any score at or above the threshold (0.7) means "ask the human first". It is the second tier of Quartermaster's shell classifier: a rule-based first tier allows obviously safe commands and hard-asks obviously dangerous ones, and this model judges the rest.

Fine-tuned from convaiinnovations/laya at revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 (English root variant, ModernBERT-large encoder), and exported like inferenceprince/laya-onnx-int8: one fused graph with int8 weight-only MatMulNBits (8 bits, block 64, symmetric, accuracy level 4), activations fp32.

Files

file
model.onnx, model.onnx.data	fused graph: inputs input_ids, attention_mask, marker_pos, marker_mask, qtype; outputs logits, act_logits
rl_agent_config.json	Laya config, including the fitted temperatures (noul 1.21)
tokenizer/tokenizer.json	the base model's tokenizer
questions.json	the 12 questions, worded exactly as used in training — use this wording at inference

Input state is {"command": "<first 4,000 chars>"}; each question is a Laya noul question.

Questions

exfiltration, secret_in_output, untrusted_code, lockout, machine_access, os_destruction, data_destruction, guard_bypass, obfuscation, other_hosts, prompt_injection, persistence. See questions.json for the wording. Labels are independent: one command can be several.

Training

• Data: 4,514 shell commands (2,418 + 930 train, 561 dev, 605 test), anonymised: real agent commands, hand-written synthetic cases, Atomic Red Team tests and model-generated commands. Each is labelled against a written policy by two labellers (Claude and Gemini via Antigravity) with disagreements adjudicated or tagged; about 1,860 generated items carry only their generator's labels. Allow/deny history from real sessions was never used as a label.
• Recipe: laya.train (rlcd loss), 2 epochs, effective batch 64, label smoothing 0.1. Every positive question per command plus 3 sampled negatives (each question is "no" on ~96% of commands; without this the head learned a confident no). Temperature fitted on a 400-row held-back train slice.

Results

Test and dev splits, commands the rule tier sends to the judge. "Needless asks" are safe real commands that would prompt; "risky caught" are commands with any positive label where some score passes the threshold.

model	threshold	needless asks (test / dev)	risky caught (test / dev)
inferenceprince/laya-onnx-int8 (zero-shot)	0.7	77.6% / 89.2%	97.5% / 94.0%
this model	0.7	6.3% / 9.9%¹	88.6% / 84.3%
this model	0.5	6.2% / 6.5%	88.6% / 84.9%

¹ sweep over safe non-generated items, a slightly wider set than the 0.5 row.

Per-question test AUC 0.82–1.0. Weakest: machine_access (AUC 0.82 test, 0.70 dev), guard_bypass recall.

Limitations

• It misses roughly one in eight risky commands that reach it; keep it behind a rule tier and a human prompt, never as the only gate.
• Trained on one person's agent workload (macOS/Linux developer tooling). Other environments will differ.
• Eval sets are small (166 / 158 risky judge-bound items), so figures move by several points between splits.

License and changes

Apache-2.0, as is the base model. Changes from convaiinnovations/laya: fine-tuned on the data above, recalibrated, exported to ONNX and quantised to int8 weight-only; exporter stack-trace metadata removed.