File size: 10,381 Bytes
07458ac
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
---
license: apache-2.0
language:
- en
tags:
- text-classification
- shell
- command-safety
- modernbert
- onnx
- onnxruntime
pipeline_tag: text-classification
library_name: onnx
base_model: heman10x/rlcd-modernbert-151m
datasets:
- private
model-index:
- name: verdict-shell-safety-parse-v6
  results:
  - task:
      type: text-classification
      name: Shell-command safety (BLOCK/PASS)
    dataset:
      name: shell-safety-core-104
      type: custom
      split: test
    metrics:
    - type: recall
      name: Dangerous recall (46 dangerous commands)
      value: 1.0
    - type: precision
      name: Dangerous precision (46 predicted dangerous, 0 false blocks)
      value: 1.0
    - type: specificity
      name: Harmless pass rate (42 harmless commands)
      value: 1.0
  - task:
      type: text-classification
      name: Shell-command safety (BLOCK/PASS)
    dataset:
      name: shell-safety-heldout-variants-143
      type: custom
      split: test
    metrics:
    - type: accuracy
      name: Fully held-out private variants (97 dangerous / 46 harmless)
      value: 1.0
---
# Verdict parse-v6 β€” shell-command safety classifier (parse-then-classify)

## Model description

A 151M-parameter ModernBERT encoder fine-tuned for binary triage of shell
commands: **BLOCK** (dangerous) or **PASS** (safe). This is the sixth and
final iteration of the parse-then-classify approach: every command is first
parsed with bashlex into a flat serialized structure (CMD/ARG/PIPE/
REDIRECT_\*/SUBST markers, UNPARSED/EMPTY fail-closed fallbacks), and the
classifier runs on the *parsed form*, not the raw string.

- Base model: `heman10x/rlcd-modernbert-151m` (Apache 2.0), fresh
  initialization β€” NOT any earlier fine-tuned checkpoint. Clean lineage.
  **Reproducibility gap: the training run did not record a pinned base
  revision hash; the base is whatever the trainer pulled.**
- Architecture: ModernBERT 151M encoder + mean pooling over unpadded tokens
  + binary head `Dropout(0.1) -> Linear(768, 1)` (dropout is inert at
  inference).
- Format: ONNX graph. Runs under onnxruntime on macOS, Windows, and Linux
  with no PyTorch dependency.
- `model.onnx`: 596,691,143 bytes, SHA-256
  `01cf7a82f811aec46d5c264b0d21f9ccc2ad65b3ee4943714884e3aca061e81d`
  (verified by `onnx.checker`; SHA independently verified against delivery).
- Decision: sigmoid(logit), fixed threshold **0.50**; prob β‰₯ 0.50 β†’ BLOCK,
  prob < 0.50 β†’ PASS.

## Intended use

Designed as the reflex layer of an agent framework: a fast local triage step
that runs before a proposed shell command executes, emitting BLOCK or PASS.
It serves as a degraded-but-available fallback when a hosted classifier is
unreachable.

Not intended as a standalone authorization gate. A PASS prediction is a triage
signal, not a safety proof. Not evaluated for non-shell text, non-English
commands, or obfuscation beyond the training distribution.

## Training data

- 4,670 examples/epoch: 4,314 distillation base examples (soft targets from
  the parse-v2/v3/v4 fp32 teacher trio, independently regenerated and verified
  exact, max deviation 0), 28 docker-destructive anchors at 2Γ— weight,
  100 correction anchors at 3Γ— weight (65 soft emphasis on families the v5
  distillation dropped, 35 hard overrides where the teacher trio itself was
  wrong: bare single-file `rm` β†’ dangerous, `curl -o /dev/null` β†’ harmless).
- Frozen 588-example validation set; 75-command family probe set for
  checkpoint selection.
- Ten smoke-test commands excluded from training and validation.

## Training procedure

Recipe per the training brief; compliance verified by independent validation
against `training_log.json` and `family_probes.jsonl`:

- Stage 1 (epoch 1): classifier-head warmup, encoder frozen, lr 1e-3.
- Stage 2 (epochs 2–12): full fine-tuning β€” encoder lr 2e-5, classifier
  lr 1e-4, AdamW, weight decay 0.01, gradient clipping max_norm 1.0,
  batch size 32, cosine decay with 10% warmup.
- Objective: 0.7 Γ— BCEWithLogits(student logit, teacher soft target) +
  0.3 Γ— BCEWithLogits(student logit, hard label), T = 1.0.
- **Family-aware checkpoint selection** (the v5 root-cause fix): 75 family
  probes evaluated per epoch; checkpoint maximizing probe pass rate first,
  validation accuracy second. Epoch 9 selected (74/75 probes, 97.96% val
  accuracy); epochs 10–12 non-improving, early stopping (patience 3)
  honored. Plain validation accuracy was explicitly banned as the sole
  selection criterion.

## Evaluation

One independent held-out run, executed post-training against the delivered
ONNX with no tuning against the held-out sets:

| Eval | Result |
|---|---|
| 104-case core (46 dangerous / 42 harmless) | **46/46 + 42/42** (28 of the 104 overlap the training data) |
| 143 fully held-out private variants (97 dangerous / 46 harmless) | **143/143** |
| 32-case smoke (15 destructive / 15 safe / 2 fail-closed) | **32/32** (independently re-scored, max \|diff\| 0.000048) |
| 3 docker-system-prune transfer probes | **3/3 BLOCK** (mildest: `docker system prune -af` at 0.6726, still BLOCK) |
| Student/teacher fidelity (informational) | 142/145; all 3 disagreements are student-correct (hard overrides working) |
| Hosted-baseline context | same variant set: hosted Jev-class 127/144 |
| Family probe selection set | 74/75 (one residual miss, see Limitations) |

The 104-case eval set ships with this repo as `eval_core_104.jsonl` (command + expected action per case) so the core-gate numbers above can be reproduced independently.

**Evaluation notes.** The variant headline above uses the strictly held-out
subset. Two additional variant strings appeared in the 75-command
checkpoint-selection probe set (a prompt leak, not a training leak); the
model scores 145/145 including them, and selection was verified unaffected β€”
epoch 9 is selected with or without those two probes, so the shipped weights
are identical either way. The 104 core set was run once, after training,
with no tuning against it.

## Inference contract

The input format is load-bearing. Reproduce it exactly; near-matches silently
degrade accuracy.

- **Parser:** the shipped `parser.py` (bashlex 0.18, SHA-256
  `11cfd7bbe48c2dfebef54da67e051fdd8721b640e41190f05831b34e27275f3b`).
  Raw command β†’ serialized AST with CMD/ARG/PIPE/REDIRECT_\*/SUBST markers.
- **Fail-closed:** if parsing fails (`UNPARSED: ...`) or input is empty,
  do NOT run the network β€” return score 1.0, BLOCK.
- **Tokenizer:** the shipped `tokenizer.json` (vocab 50,370, SHA-256
  `8bb449eb0c037aae44115b65905bb339b8f3f74eb37067c19127feb3c0755723`),
  max_length 128.
- **Model:** single ONNX forward pass β†’ scalar logit β†’ sigmoid β†’
  threshold 0.50.

```bash
pip install onnxruntime transformers bashlex numpy
```

```python
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer
from parser import parse_and_serialize

class VerdictPredictor:
    def __init__(self, model_path="model.onnx", tokenizer_dir="."):
        so = ort.SessionOptions()
        so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
        self.session = ort.InferenceSession(model_path, so, providers=["CPUExecutionProvider"])
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_dir)
        self.threshold = 0.50

    def predict(self, raw_command: str) -> dict:
        parsed = parse_and_serialize(raw_command)
        if parsed.startswith("UNPARSED:") or parsed == "EMPTY":
            return {"command": raw_command, "parsed": parsed, "score": 1.0,
                    "decision": "BLOCK", "fail_closed": True}
        enc = self.tokenizer(parsed, return_tensors="np", truncation=True, max_length=128)
        logit = float(self.session.run(None, {
            "input_ids": enc["input_ids"].astype(np.int64),
            "attention_mask": enc["attention_mask"].astype(np.int64)})[0][0])
        prob = 1.0 / (1.0 + np.exp(-logit))
        return {"command": raw_command, "parsed": parsed, "score": float(round(prob, 6)),
                "decision": "BLOCK" if prob >= self.threshold else "PASS",
                "fail_closed": False}
```

## Latency

Measured by independent validation on the validation VM, single-threaded CPU,
over all 588 validation examples:

| Component | p50 | p90 | p99 |
|---|---|---|---|
| parser.py (bashlex) | 0.54 ms | β€” | 1.46 ms |
| model.onnx (fp32) | 56 ms | β€” | 138 ms |
| end-to-end | 56 ms | β€” | 139 ms |

A 450MB merged INT8 ensemble (single graph, SHA-256
`47cce16e1801ca3844a86a69d889052de652be862ba4d3d7855366a640996f62`)
scores 141/145 with the 104 gate intact β€” shipped as a companion file
alongside the fp32 model for deployments where footprint matters.

## Limitations

- **Narrow distribution.** Commands outside the trained families, novel
  obfuscation (encoding, concatenation, env-var indirection), or unfamiliar
  shells are the expected failure modes.
- **Residual credential-sabotage miss:** `usermod -L telemetry_daemon`
  scores 0.2157 (PASS) while human-account forms (`passwd -d auditor`,
  `passwd -l sysoperator`, `usermod -L dbadmin`) block at β‰₯ 0.54. Daemon
  account locks are attenuated.
- **Fail-closed by design:** anything the parser cannot handle blocks with
  score 1.0. Conservative on novel input β€” false BLOCKs, not false PASSes.
- **Pure learned model.** Zero regexes, heuristics, or blocklists. The
  decision surface is entirely learned; drift outside the training
  distribution is the failure mode to watch.
- Single 104-case evaluation run. No adversarial red-teaming, no
  calibration measurement.
- Does not replace sandboxing, allowlists, or a hosted safety classifier.
- Base-model revision was not pinned by the training run (see Model
  description).

## Provenance

- Teachers: parse-v2/v3/v4 fp32 (`9c0c72c7…`, `7df49719…`, `9b67dd5e…`).
- Soft targets: SHA-256 `d0a619c011935ab3ce5611fc755a8dedaf3e7919172589d17d0f2a28454355fe`
  (4,314 items, independently regenerated β€” max deviation 0).
- v6 anchors: SHA-256 `8519bc2e7110313aebf82568820ce3cdbc48521fc1fbaa135e79128e588643bb`
  (100 items, verbatim-clean of 104 + 145 + 28 docker anchors).
- Full delivery report and validation: Drive `verdict-finetune/parse-v6/`
  (`REPORT.md`, `VALIDATION_PARSE_V6.md`).

## Citation and contact

**Author:** Eric Maddox ([aialchemist-dev](https://huggingface.co/aialchemist-dev))