File size: 2,650 Bytes
c852c8d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | ---
license: apache-2.0
base_model: Qwen/Qwen3-4B
pipeline_tag: text-generation
tags: [agent-safety, ai-monitoring, foresight, qwen3]
---
# sdqa — step-level agent-safety auditors
Two Qwen3-4B full-finetunes that judge **one proposed agent action, before it
executes**, against the observed prefix and the task policy, and emit a single
verdict box. They differ only in SFT learning rate.
| subfolder | learning rate | pick it for |
| --- | --- | --- |
| `lr5e-6` | 5e-6 | the better operating point — reward +0.165, miss 0.175, false-ask 0.034 |
| `lr1e-5` | 1e-5 | the better ranking — AUROC 0.780 |
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("Caaaarr1e/sdqa", subfolder="lr5e-6")
t = AutoTokenizer.from_pretrained("Caaaarr1e/sdqa", subfolder="lr5e-6")
```
## Output contract
```
\boxed{act:STOP, binary:UNSAFE, risk:90, chan:NONE, pert:NONE, obj:NONE,
fail:data_exposure, harm:privacy, src:compromised_tool, q:NONE, cite:3}
```
`act` is CONTINUE / ASK / STOP. The binary projection is CONTINUE=SAFE, STOP=UNSAFE,
and ASK=UNSAFE at `risk >= 50`. `cite` points at an earlier visible step, or NONE.
`fail` / `harm` / `src` come from a 16 failure-mode, 11 harm-type, 10 risk-source
taxonomy; `NONE` is an absence sentinel, not an additional class.
## Training
Full SFT from `Qwen/Qwen3-4B`, epoch 2, thinking enabled, on 2,464 step-level records.
Prompts were rendered through the evaluator's own pipeline, so the training and
inference formats match. Weights are the original fp32.
## Limitations — please read
- **The training data has a known 15.1% defect rate.** In 372 of 2,464 records the
judged action was truncated out of its own prompt by a history-character cap. The
damage is biased, not random: it concentrates at deep steps, which is exactly where
the decision boundary and the clarification window live. Those targets are not
learnable, and the models were trained with them in place.
- **Risk scores are not calibrated probabilities.** Do not threshold them as such
without your own calibration.
- **Parse rate is not an improvement over the base model.** Base Qwen3-4B parses this
contract at 99.5%; these sit at 97.4–99.0%, i.e. at or slightly below.
- Evaluated only on an internal foresight suite. No claim is made about general
agent-safety benchmarks or production monitoring.
- The box is emitted reliably only under the matching auditor prompt; a different
prompt format degrades output sharply.
- These are research artifacts, not a safety guarantee. Do not deploy as the only
control on a consequential action.
|