model card: v8 attack 5/5 x3, MMLU==base (branch main)

README.md

@@ -4,15 +4,10 @@ base_model: Qwen/Qwen3-8B
 tags: [cybersecurity, agentic, red-team, blue-team, distillation, glm-5.2, kimi-k3]
 ---
 
-# k3-sec-8b (main = v7cti)
-
-> **`main` currently carries the `v7cti` weights** (v6 agentic lineage + round-7 CTI
-> knowledge SFT) plus GGUF builds for LM Studio. The newest agentic-lineage model is
-> `v7` (branch), which passed the internal 10-task gate at 8.33/10 — its card content
-> follows below. All versions live on branches: `v7`, `v7cti`, `v6`, `v5`, `v4`, `v3`, `v2`.
+# k3-sec-8b (v8)
 
 An 8B cybersecurity agent (offense + defense) fine-tuned from `Qwen/Qwen3-8B` on
-**2061 verified agentic + Q&A traces**, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6
+**2197 verified agentic + Q&A traces**, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6
 (rounds 1–3). Trained to operate an autonomous security harness — plan, run bash, read
 observations, write files, finish — not just answer security questions.
 
@@ -20,26 +15,28 @@ Pipeline per iteration: failure analysis on eval traces → parameterized, decon
 seed factory → teacher best-of-3 rejection-sampled agentic transcripts (judged +
 artifact-checked) → full-FT SFT → multi-run attack/defend eval → next round.
 
-## Version 7 highlights
+## Version 8 highlights
 
-- **Data**: 2061 unique rows (1967 agentic + 94 Q&A). Round 7 was the broad-coverage
+- **Data**: 2197 unique rows (2103 agentic + 94 Q&A). Round 8 was a 136-trace DNS
+  top-up (75 decode + 75 detector-contract, $8). Round 7 was the broad-coverage
   round: 985 traces across ALL ten eval skills (flaky six weighted 100-120, solid four
-  kept warm at 60-80, 80 generalization), because v6 showed single-skill targeting
-  trades regressions. Round 6 added
+  kept warm at 60-80, 80 generalization). Round 6 added
   564 GLM-5.2 traces targeting the five v3 tasks that never passed, generated from 580
   parameterized seeds with decontamination-by-construction (every eval-graded string is
   blacklisted and asserted absent). Best-of-3 rejection with a glm-4.7-flash judge
   (kept 97%). Note: v3's advertised 157 rows contained only 135 unique after legacy
   merge duplicates; v4 is a genuine 5.2× data increase.
 - **Training**: full FT bf16, 2 epochs, lr 1e-5 cosine, eff. batch 32, seq 8192,
-  adamw_8bit, 5.37M tokens, ~42 min on 1× A100-80GB. **train_loss 0.627 · token-acc
-  86.6%** (v6: 0.838 / 83.1%, v5: 0.881 / 82.4%, v3: 1.395 / 69.1%). Ships with Qwen3 YaRN `rope_scaling` for 131072-token
+  adamw_8bit, 5.37M tokens, ~42 min on 1× A100-80GB. **train_loss 0.616 · token-acc
+  87.2%** (v7: 0.627 / 86.6%, v6: 0.838 / 83.1%, v3: 1.395 / 69.1%). Ships with Qwen3 YaRN `rope_scaling` for 131072-token
   serving.
 - **Eval** (fixed 10-task synthetic attack/defend lab, agentic harness, 3 runs):
-  **mean 8.33/10** (runs: 8, 7, **10**) -- PASSES the promotion gate (>=8/10, both
-  sides >=4/5 on mean; v7c scored a perfect 10/10). atk-sqli consolidated to 3/3.
-  Eight of ten tasks pass 3/3; the DNS pair (atk-dns, def-detect-dns) is the last
-  flip-flop at 1/3 each, def-ioc 2/3.
+  **mean 8.0/10 with ZERO variance (8, 8, 8) -- attack side 5/5 in all three runs**
+  (atk-dns fixed by the top-up). Defend side: bruteforce/webshell(2/3)/harden solid,
+  def-detect-dns 1/3, def-ioc regressed to 0/3 (round-9 target).
+- **MMLU spot check** (60 questions, temp 0, same harness): v8 0.533 vs v7 0.550 vs
+  **base Qwen3-8B 0.550** -- general capability statistically indistinguishable from
+  base. NO capability collapse from the agentic diet.
 
 ### Per-task pass rates (P across runs)
 
@@ -61,53 +58,6 @@ meaningful for this suite. Known v4 gap: `atk-sqli` — the model prefers to sta
 staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts
 (trace-verified behavioral prior, targeted in round 5).
 
-## External benchmark: vs Cisco Foundation-Sec (MCQ knowledge eval, 2026-08-01)
-
-Independent comparison on 1,210 multiple-choice cybersecurity questions
-(CyberMetric-500, CTI-Bench-MCQ 500/2500 sample, SecQA v1+v2 test; seed 42,
-temperature 0). Each model evaluated in its **native prompting mode**:
-`Foundation-Sec-8B` is a *base* model (few-shot completion is its only usable mode;
-zero-shot chat collapses to ~34%), `Foundation-Sec-1.1-8B-Instruct` and k3-sec-8b
-were evaluated zero-shot chat (Qwen3 template, non-thinking). Identical question
-sets for every model.
-
-| Benchmark (n) | F-Sec-8B (base) | F-Sec-1.1-8B-Instruct | k3 v6 | v7cti (chat) | v7cti (completion) |
-|---|---:|---:|---:|---:|---:|
-| CyberMetric-500 | 84.20% | 85.80% | **86.80%** | 86.20% | **87.80%** |
-| CTI-Bench-MCQ (500) | 66.00% | **67.40%** | 58.80% | 60.40% | 62.60% |
-| SecQA v1+v2 (210) | 98.10% | **99.05%** | 98.10% | 97.62% | 98.57% |
-| **Overall (1,210)** | 79.09% | **80.50%** | 77.19% | 77.52% | 79.26% |
-
-Parse failures: 0 for all models in native modes.
-
-**CTI rounds (v6cti / v7cti branches).** Two rounds of ATT&CK-STIX-derived CTI data
-(~2.5k decontaminated MCQ+free-form traces, GLM-5.2/Foundation-Sec teachers,
-deepseek-v4-flash judge) were trained on top of the r6 mix. Round 6 (justification
-style, upsampled) produced churn without net gain and a chat-mode regression —
-diagnosed as format conflict + template overfitting + step-share dilution. Round 7
-fixed all three (bare-letter format-matched training, GLM-5.2 paraphrases,
-multi-turn packing to 38% of gradient steps): chat CTI-Bench **+1.6** to 60.4%,
-CyberMetric completion **87.8%** and SecQA completion **98.6%** (k3 records).
-Against the base Foundation-Sec, v7cti-completion ties overall (79.26% vs 79.09%);
-against the stronger **Foundation-Sec-1.1-8B-Instruct (80.50% overall)** k3 leads
-only on CyberMetric (+2.0) and trails on CTI-Bench (62.6% vs 67.4%) — CTI/ATT&CK
-knowledge remains a pretraining-data advantage and the target of the next round
-(light domain CPT, then failure-targeted DPO).
-
-**Qualitative** (6 open-ended security prompts): Foundation-Sec gives terse,
-mostly-correct completion-style answers; k3 gives structured markdown answers with
-finer-grained precision (e.g. LSASS dumping -> T1003.001 sub-technique + detections).
-Eval harness + raw per-question results: `sec-eval/` (drive_eval.py, results_*.json).
-
-## GGUF / LM Studio
-
-`main` ships GGUF builds of v7cti (llama.cpp b10219): `k3-sec-8b-v7cti-Q4_K_M.gguf`
-and `k3-sec-8b-v7cti-Q8_0.gguf`.
-
-```bash
-lms get bebrws/k3-sec-8b
-```
-
 ## Usage
 
 vLLM, short-task/eval serving (disable static YaRN):
@@ -140,8 +90,8 @@ competence is deliberately scoped to CTF/lab-grade tasks.
 | v4 | 699 | 6.67/10 (3 runs) | GLM-5.2 scale-up, jwt fixed |
 | v5 | 891 | 6.67/10 (3 runs: 4,8,8) | sqli breakthrough, ioc fixed; harden regressed (newline stripping) |
 | v6 | 1076 | 7.0/10 (3 runs: 7,8,6) | harden fixed, webshell solid |
-| **v7** | **2061** | **8.33/10 (3 runs: 8,7,10)** | GATE PASSED; sqli 3/3; DNS pair last flip-flop |
-| **v7cti** (on `main`) | 1076 + 3k CTI | (agentic lab not rerun) | CTI knowledge round; MCQ records (CyberMetric 87.8 / SecQA 98.6 / chat-CTI 60.4) |
+| v7 | 2061 | 8.33/10 (3 runs: 8,7,10) | gate passed; sqli 3/3 |
+| **v8** | **2197** | **8.0/10 (3 runs: 8,8,8)** | attack 5/5 x3; MMLU == base; ioc regressed |
 
 Weights are Apache-2.0 per the Qwen3 base; training traces were generated by GLM-5.2
 and Kimi teachers and filtered by automated judging.