bebrws commited on
Commit
fef61ed
·
verified ·
1 Parent(s): 644833d

Model card: add Foundation-Sec benchmark comparison incl. v7cti

Browse files
Files changed (1) hide show
  1. README.md +40 -0
README.md CHANGED
@@ -54,6 +54,46 @@ meaningful for this suite. Known v4 gap: `atk-sqli` — the model prefers to sta
54
  staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts
55
  (trace-verified behavioral prior, targeted in round 5).
56
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57
  ## Usage
58
 
59
  vLLM, short-task/eval serving (disable static YaRN):
 
54
  staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts
55
  (trace-verified behavioral prior, targeted in round 5).
56
 
57
+ ## External benchmark: vs Cisco Foundation-Sec-8B (MCQ knowledge eval, 2026-08-01)
58
+
59
+ Independent comparison on 1,210 multiple-choice cybersecurity questions
60
+ (CyberMetric-500, CTI-Bench-MCQ 500/2500 sample, SecQA v1+v2 test; seed 42,
61
+ temperature 0). Each model was evaluated in its **native prompting mode**:
62
+ Foundation-Sec-8B is a *base* model and was evaluated with 3-shot raw completion
63
+ (its intended usage per Cisco's model card); k3-sec-8b was evaluated zero-shot
64
+ chat (Qwen3 template, non-thinking). Identical question sets for every model.
65
+
66
+ | Benchmark (n) | Foundation-Sec-8B | k3 v4 | v5 | v6 | v7cti (chat) | v7cti (completion) |
67
+ |---|---:|---:|---:|---:|---:|---:|
68
+ | CyberMetric-500 | 84.20% | **86.80%** | **86.80%** | **86.80%** | 86.20% | **87.80%** |
69
+ | CTI-Bench-MCQ (500) | **66.00%** | 59.20% | 59.80% | 58.80% | 60.40% | 62.60% |
70
+ | SecQA v1+v2 (210) | 98.10% | 98.10% | 98.10% | 98.10% | 97.62% | **98.57%** |
71
+ | **Overall (1,210)** | **79.09%** | 77.36% | 77.60% | 77.19% | 77.52% | **79.26%** |
72
+
73
+ Parse failures: 0 for all models in native modes.
74
+
75
+ **Prompting-mode sensitivity.** Foundation-Sec-8B collapses under zero-shot chat
76
+ prompting (~34% accuracy, ~40% whitespace/echo responses) and only performs via
77
+ few-shot completion — do not deploy it behind a naive chat template. k3-sec-8b is
78
+ robust across both modes with 100% answer-format compliance.
79
+
80
+ **CTI rounds (v6cti / v7cti branches).** Two rounds of ATT&CK-STIX-derived CTI data
81
+ (~2.5k decontaminated MCQ+free-form traces, GLM-5.2/Foundation-Sec teachers,
82
+ deepseek-v4-flash judge) were trained on top of the r6 mix. Round 6 (justification
83
+ style, upsampled) produced churn without net gain and a chat-mode regression —
84
+ diagnosed as format conflict + template overfitting + step-share dilution. Round 7
85
+ fixed all three (bare-letter format-matched training, GLM-5.2 paraphrases,
86
+ multi-turn packing to 38% of gradient steps): chat CTI-Bench **+1.6** to 60.4%,
87
+ CyberMetric completion **87.8%** and SecQA completion **98.6%** (k3 records), and
88
+ **v7cti-completion ties Foundation-Sec overall (79.26% vs 79.09%)** while beating it
89
+ on CyberMetric by 3.6 pts. Foundation-Sec still leads CTI-Bench (66.0%) — the
90
+ remaining target for the next round (failure-targeted DPO or light domain CPT).
91
+
92
+ **Qualitative** (6 open-ended security prompts): Foundation-Sec gives terse,
93
+ mostly-correct completion-style answers; k3 gives structured markdown answers with
94
+ finer-grained precision (e.g. LSASS dumping -> T1003.001 sub-technique + detections).
95
+ Eval harness + raw per-question results: `sec-eval/` (drive_eval.py, results_*.json).
96
+
97
  ## Usage
98
 
99
  vLLM, short-task/eval serving (disable static YaRN):