| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-4B |
| pipeline_tag: text-generation |
| tags: [agent-safety, ai-monitoring, foresight, qwen3] |
| --- |
| |
| # sdqa — step-level agent-safety auditors |
|
|
| Two Qwen3-4B full-finetunes that judge **one proposed agent action, before it |
| executes**, against the observed prefix and the task policy, and emit a single |
| verdict box. They differ only in SFT learning rate. |
|
|
| | subfolder | learning rate | pick it for | |
| | --- | --- | --- | |
| | `lr5e-6` | 5e-6 | the better operating point — reward +0.165, miss 0.175, false-ask 0.034 | |
| | `lr1e-5` | 1e-5 | the better ranking — AUROC 0.780 | |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| m = AutoModelForCausalLM.from_pretrained("Caaaarr1e/sdqa", subfolder="lr5e-6") |
| t = AutoTokenizer.from_pretrained("Caaaarr1e/sdqa", subfolder="lr5e-6") |
| ``` |
|
|
| ## Output contract |
|
|
| ``` |
| \boxed{act:STOP, binary:UNSAFE, risk:90, chan:NONE, pert:NONE, obj:NONE, |
| fail:data_exposure, harm:privacy, src:compromised_tool, q:NONE, cite:3} |
| ``` |
|
|
| `act` is CONTINUE / ASK / STOP. The binary projection is CONTINUE=SAFE, STOP=UNSAFE, |
| and ASK=UNSAFE at `risk >= 50`. `cite` points at an earlier visible step, or NONE. |
| `fail` / `harm` / `src` come from a 16 failure-mode, 11 harm-type, 10 risk-source |
| taxonomy; `NONE` is an absence sentinel, not an additional class. |
|
|
| ## Training |
|
|
| Full SFT from `Qwen/Qwen3-4B`, epoch 2, thinking enabled, on 2,464 step-level records. |
| Prompts were rendered through the evaluator's own pipeline, so the training and |
| inference formats match. Weights are the original fp32. |
|
|
| ## Limitations — please read |
|
|
| - **The training data has a known 15.1% defect rate.** In 372 of 2,464 records the |
| judged action was truncated out of its own prompt by a history-character cap. The |
| damage is biased, not random: it concentrates at deep steps, which is exactly where |
| the decision boundary and the clarification window live. Those targets are not |
| learnable, and the models were trained with them in place. |
| - **Risk scores are not calibrated probabilities.** Do not threshold them as such |
| without your own calibration. |
| - **Parse rate is not an improvement over the base model.** Base Qwen3-4B parses this |
| contract at 99.5%; these sit at 97.4–99.0%, i.e. at or slightly below. |
| - Evaluated only on an internal foresight suite. No claim is made about general |
| agent-safety benchmarks or production monitoring. |
| - The box is emitted reliably only under the matching auditor prompt; a different |
| prompt format degrades output sharply. |
| - These are research artifacts, not a safety guarantee. Do not deploy as the only |
| control on a consequential action. |
|
|
