bebrws commited on
Commit
644f802
·
verified ·
1 Parent(s): 734d08e

Model card: main=v7cti note + Foundation-Sec benchmark (incl 1.1-Instruct) + GGUF/LM Studio usage

Browse files
Files changed (1) hide show
  1. README.md +54 -1
README.md CHANGED
@@ -4,7 +4,12 @@ base_model: Qwen/Qwen3-8B
4
  tags: [cybersecurity, agentic, red-team, blue-team, distillation, glm-5.2, kimi-k3]
5
  ---
6
 
7
- # k3-sec-8b (v7)
 
 
 
 
 
8
 
9
  An 8B cybersecurity agent (offense + defense) fine-tuned from `Qwen/Qwen3-8B` on
10
  **2061 verified agentic + Q&A traces**, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6
@@ -56,6 +61,53 @@ meaningful for this suite. Known v4 gap: `atk-sqli` — the model prefers to sta
56
  staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts
57
  (trace-verified behavioral prior, targeted in round 5).
58
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
  ## Usage
60
 
61
  vLLM, short-task/eval serving (disable static YaRN):
@@ -89,6 +141,7 @@ competence is deliberately scoped to CTF/lab-grade tasks.
89
  | v5 | 891 | 6.67/10 (3 runs: 4,8,8) | sqli breakthrough, ioc fixed; harden regressed (newline stripping) |
90
  | v6 | 1076 | 7.0/10 (3 runs: 7,8,6) | harden fixed, webshell solid |
91
  | **v7** | **2061** | **8.33/10 (3 runs: 8,7,10)** | GATE PASSED; sqli 3/3; DNS pair last flip-flop |
 
92
 
93
  Weights are Apache-2.0 per the Qwen3 base; training traces were generated by GLM-5.2
94
  and Kimi teachers and filtered by automated judging.
 
4
  tags: [cybersecurity, agentic, red-team, blue-team, distillation, glm-5.2, kimi-k3]
5
  ---
6
 
7
+ # k3-sec-8b (main = v7cti)
8
+
9
+ > **`main` currently carries the `v7cti` weights** (v6 agentic lineage + round-7 CTI
10
+ > knowledge SFT) plus GGUF builds for LM Studio. The newest agentic-lineage model is
11
+ > `v7` (branch), which passed the internal 10-task gate at 8.33/10 — its card content
12
+ > follows below. All versions live on branches: `v7`, `v7cti`, `v6`, `v5`, `v4`, `v3`, `v2`.
13
 
14
  An 8B cybersecurity agent (offense + defense) fine-tuned from `Qwen/Qwen3-8B` on
15
  **2061 verified agentic + Q&A traces**, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6
 
61
  staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts
62
  (trace-verified behavioral prior, targeted in round 5).
63
 
64
+ ## External benchmark: vs Cisco Foundation-Sec (MCQ knowledge eval, 2026-08-01)
65
+
66
+ Independent comparison on 1,210 multiple-choice cybersecurity questions
67
+ (CyberMetric-500, CTI-Bench-MCQ 500/2500 sample, SecQA v1+v2 test; seed 42,
68
+ temperature 0). Each model evaluated in its **native prompting mode**:
69
+ `Foundation-Sec-8B` is a *base* model (few-shot completion is its only usable mode;
70
+ zero-shot chat collapses to ~34%), `Foundation-Sec-1.1-8B-Instruct` and k3-sec-8b
71
+ were evaluated zero-shot chat (Qwen3 template, non-thinking). Identical question
72
+ sets for every model.
73
+
74
+ | Benchmark (n) | F-Sec-8B (base) | F-Sec-1.1-8B-Instruct | k3 v6 | v7cti (chat) | v7cti (completion) |
75
+ |---|---:|---:|---:|---:|---:|
76
+ | CyberMetric-500 | 84.20% | 85.80% | **86.80%** | 86.20% | **87.80%** |
77
+ | CTI-Bench-MCQ (500) | 66.00% | **67.40%** | 58.80% | 60.40% | 62.60% |
78
+ | SecQA v1+v2 (210) | 98.10% | **99.05%** | 98.10% | 97.62% | 98.57% |
79
+ | **Overall (1,210)** | 79.09% | **80.50%** | 77.19% | 77.52% | 79.26% |
80
+
81
+ Parse failures: 0 for all models in native modes.
82
+
83
+ **CTI rounds (v6cti / v7cti branches).** Two rounds of ATT&CK-STIX-derived CTI data
84
+ (~2.5k decontaminated MCQ+free-form traces, GLM-5.2/Foundation-Sec teachers,
85
+ deepseek-v4-flash judge) were trained on top of the r6 mix. Round 6 (justification
86
+ style, upsampled) produced churn without net gain and a chat-mode regression —
87
+ diagnosed as format conflict + template overfitting + step-share dilution. Round 7
88
+ fixed all three (bare-letter format-matched training, GLM-5.2 paraphrases,
89
+ multi-turn packing to 38% of gradient steps): chat CTI-Bench **+1.6** to 60.4%,
90
+ CyberMetric completion **87.8%** and SecQA completion **98.6%** (k3 records).
91
+ Against the base Foundation-Sec, v7cti-completion ties overall (79.26% vs 79.09%);
92
+ against the stronger **Foundation-Sec-1.1-8B-Instruct (80.50% overall)** k3 leads
93
+ only on CyberMetric (+2.0) and trails on CTI-Bench (62.6% vs 67.4%) — CTI/ATT&CK
94
+ knowledge remains a pretraining-data advantage and the target of the next round
95
+ (light domain CPT, then failure-targeted DPO).
96
+
97
+ **Qualitative** (6 open-ended security prompts): Foundation-Sec gives terse,
98
+ mostly-correct completion-style answers; k3 gives structured markdown answers with
99
+ finer-grained precision (e.g. LSASS dumping -> T1003.001 sub-technique + detections).
100
+ Eval harness + raw per-question results: `sec-eval/` (drive_eval.py, results_*.json).
101
+
102
+ ## GGUF / LM Studio
103
+
104
+ `main` ships GGUF builds of v7cti (llama.cpp b10219): `k3-sec-8b-v7cti-Q4_K_M.gguf`
105
+ and `k3-sec-8b-v7cti-Q8_0.gguf`.
106
+
107
+ ```bash
108
+ lms get bebrws/k3-sec-8b
109
+ ```
110
+
111
  ## Usage
112
 
113
  vLLM, short-task/eval serving (disable static YaRN):
 
141
  | v5 | 891 | 6.67/10 (3 runs: 4,8,8) | sqli breakthrough, ioc fixed; harden regressed (newline stripping) |
142
  | v6 | 1076 | 7.0/10 (3 runs: 7,8,6) | harden fixed, webshell solid |
143
  | **v7** | **2061** | **8.33/10 (3 runs: 8,7,10)** | GATE PASSED; sqli 3/3; DNS pair last flip-flop |
144
+ | **v7cti** (on `main`) | 1076 + 3k CTI | (agentic lab not rerun) | CTI knowledge round; MCQ records (CyberMetric 87.8 / SecQA 98.6 / chat-CTI 60.4) |
145
 
146
  Weights are Apache-2.0 per the Qwen3 base; training traces were generated by GLM-5.2
147
  and Kimi teachers and filtered by automated judging.