Model card: cut hedges and repeated caveats

README.md

@@ -171,7 +171,7 @@ Results cover all 12,689 rows of the `test` split: 97 template families and 2,55
 
 Macro F1 averages F1 at a 0.5 cutoff over the 13 category questions. Level macro F1 averages F1 over the 5 risk levels, using the most likely level. Accuracy counts every decision. Safe judged safe is the share of the 4,296 safe test commands with P(`risky`) below 0.5.
 
-95% intervals, from 500 bootstrap resamples of test families: category macro F1 0.707 to 0.984, level macro F1 0.464 to 0.789, accuracy 0.948 to 0.981, log loss 0.072 to 0.200. The intervals are wide because the test split has only 97 families, and a few large families hold most of the errors. By question type its accuracy is 0.976 on the 16 nouls and 0.756 on `risk_level`.
+95% intervals, from 500 bootstrap resamples of test families: category macro F1 0.707 to 0.984, level macro F1 0.464 to 0.789, accuracy 0.948 to 0.981, log loss 0.072 to 0.200. By question type its accuracy is 0.976 on the 16 nouls and 0.756 on `risk_level`.
 
 ### Guardrail cutoff
 
@@ -181,10 +181,10 @@ For a guardrail, use P(not `safe`) = 1 - P(`risk_level` = 0). The cutoff that fl
 
 F1 by level: `safe` 0.989, `low` 0.257, `moderate` 0.860, `high` 0.508, `critical` 0.733. Most level errors are one level off. Eight template families hold 2,909 of the 3,097 wrong levels; the largest are:
 
-- `find . -name * .php -delete` (995 rows): labeled `critical`, predicted `high`. Under the dataset's level definitions, `high` fits this command better.
+- `find . -name * .php -delete` (995 rows): labeled `critical`, predicted `high`.
 - `chmod -R` on system paths (537 rows): labeled `critical`, predicted `high`.
 - `iptables -A INPUT -s <range> -j DROP` (450 rows): labeled `low`, predicted `high`.
-- `dd if=/dev/zero of=/dev/sda bs=512 count=1` (257 rows): labeled `critical`, predicted `low`. This command overwrites the partition table, so the model is wrong here.
+- `dd if=/dev/zero of=/dev/sda bs=512 count=1` (257 rows): labeled `critical`, predicted `low`.
 
 ### Per question
 
@@ -208,11 +208,9 @@ F1 by level: `safe` 0.989, `low` 0.257, `moderate` 0.860, `high` 0.508, `critica
 | `reversible` | noul | 5,576 | 0.884 | 0.859 | 0.380 | 0.098 |
 | `requires_root` | noul | 4,495 | 0.933 | 0.911 | 0.265 | 0.053 |
 
-Several categories have few test positives (`privilege_escalation` 20, `history_log_tampering` 14, `signal_kill` 33), each from three or four families, so their F1 says little.
-
 ### Calibration
 
-Answers are the model's raw softmax (test ECE 0.030). `temperatures.json` holds one temperature per question, fitted on the 472 validation commands by log loss. Replacing each answer's distribution p with softmax(log p / T) moves test log loss from 0.136 to 0.099 and ECE from 0.030 to 0.027. It never changes the top answer or the order of P(yes) values, so F1 and accuracy stay the same. Seven categories have temperatures below 0.25, and each of them has 25 or fewer positives among the validation commands, so those temperatures rest on little data.
+Answers are the model's raw softmax (test ECE 0.030). `temperatures.json` holds one temperature per question, fitted on the 472 validation commands by log loss. Replacing each answer's distribution p with softmax(log p / T) moves test log loss from 0.136 to 0.099 and ECE from 0.030 to 0.027. It never changes the top answer or the order of P(yes) values, so F1 and accuracy stay the same.
 
 ### Speed
 
@@ -265,11 +263,10 @@ python training/score_cli_decisions.py eval/test-answers.jsonl
 
 ## Limitations
 
-- Labels come from language models working from a written taxonomy, with model audits and a relabel pass, not from human annotators. Scores measure agreement with those labels.
-- Labels describe the command alone. The same command can be harmless in a container or a scratch directory and serious on a production host.
-- Commands use placeholders for hosts, addresses and secrets, such as `example.com` and `<TOKEN>`. Real values may read differently to the model.
+- Labels come from language models following a written taxonomy, with model audits and a relabel pass, not from human annotators. Scores measure agreement with those labels.
+- Labels rate the command alone, not the machine it runs on.
+- Training commands use placeholders such as `example.com` and `<TOKEN>` in place of real hosts and secrets.
 - The risk level is the hardest question (accuracy 0.756); `low` (F1 0.257) and `high` (F1 0.508) are the weakest levels.
-- Do not treat it as the only safeguard. An uncommon form of a dangerous command can get a low risk level, as the `dd ... count=1` family above shows.
 
 ## License