model card: v6 eval results (mean 7.0/10) (branch main)

README.md

@@ -4,10 +4,10 @@ base_model: Qwen/Qwen3-8B
 tags: [cybersecurity, agentic, red-team, blue-team, distillation, glm-5.2, kimi-k3]
 ---
 
-# k3-sec-8b (v5)
+# k3-sec-8b (v6)
 
 An 8B cybersecurity agent (offense + defense) fine-tuned from `Qwen/Qwen3-8B` on
-**891 verified agentic + Q&A traces**, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6
+**1076 verified agentic + Q&A traces**, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6
 (rounds 1–3). Trained to operate an autonomous security harness — plan, run bash, read
 observations, write files, finish — not just answer security questions.
 
@@ -15,23 +15,24 @@ Pipeline per iteration: failure analysis on eval traces → parameterized, decon
 seed factory → teacher best-of-3 rejection-sampled agentic transcripts (judged +
 artifact-checked) → full-FT SFT → multi-run attack/defend eval → next round.
 
-## Version 5 highlights
+## Version 6 highlights
 
-- **Data**: 891 unique rows (797 agentic + 94 Q&A). Round 5 added 192 traces (80 targeting
-  sqli live-fuzzing temptation-resistance). Round 4 added
+- **Data**: 1076 unique rows (982 agentic + 94 Q&A). Round 6 added 185 traces:
+  60 verify-after-edit config hardening (v5's newline-stripping regression), 60 sqli
+  temptation consolidation, 65 flaky-skill reinforcement. Round 5 added
   564 GLM-5.2 traces targeting the five v3 tasks that never passed, generated from 580
   parameterized seeds with decontamination-by-construction (every eval-graded string is
   blacklisted and asserted absent). Best-of-3 rejection with a glm-4.7-flash judge
   (kept 97%). Note: v3's advertised 157 rows contained only 135 unique after legacy
   merge duplicates; v4 is a genuine 5.2× data increase.
 - **Training**: full FT bf16, 2 epochs, lr 1e-5 cosine, eff. batch 32, seq 8192,
-  adamw_8bit, 5.37M tokens, ~42 min on 1× A100-80GB. **train_loss 0.881 · token-acc
-  82.4%** (v4: 0.977 / 80.0%, v3: 1.395 / 69.1%). Ships with Qwen3 YaRN `rope_scaling` for 131072-token
+  adamw_8bit, 5.37M tokens, ~42 min on 1× A100-80GB. **train_loss 0.838 · token-acc
+  83.1%** (v5: 0.881 / 82.4%, v4: 0.977 / 80.0%, v3: 1.395 / 69.1%). Ships with Qwen3 YaRN `rope_scaling` for 131072-token
   serving.
 - **Eval** (fixed 10-task synthetic attack/defend lab, agentic harness, 3 runs):
-  **mean 6.67/10** (runs: 4, 8, 8) vs v3 mean 4.0/10 (4 runs). Two of three runs hit
-  the gate score of 8/10; `atk-sqli` passed for the first time ever (v5b) after
-  temptation-resistance training.
+  **mean 7.0/10** (runs: 7, 8, 6) -- best checkpoint. def-harden recovered to 3/3 after
+  verify-after-edit training; atk-sqli passes 1/3; atk-jwt/hash/bruteforce/webshell/
+  harden all solid at 3/3. Remaining flip-flop tasks: def-detect-dns, def-ioc, atk-re.
 
 ### Per-task pass rates (P across runs)
 
@@ -53,59 +54,6 @@ meaningful for this suite. Known v4 gap: `atk-sqli` — the model prefers to sta
 staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts
 (trace-verified behavioral prior, targeted in round 5).
 
-## External benchmark: vs Cisco Foundation-Sec-8B (MCQ knowledge eval, 2026-08-01)
-
-Independent comparison on 1,210 multiple-choice cybersecurity questions
-(CyberMetric-500, CTI-Bench-MCQ 500/2500 sample, SecQA v1+v2 test; seed 42,
-temperature 0). Each model was evaluated in its **native prompting mode**:
-Foundation-Sec-8B is a *base* model and was evaluated with 3-shot raw completion
-(its intended usage per Cisco's model card); k3-sec-8b was evaluated zero-shot
-chat (Qwen3 template, non-thinking). Identical question sets for every model.
-
-| Benchmark (n) | Foundation-Sec-8B | k3-sec-8b v4 | k3-sec-8b v5 | k3-sec-8b v6 |
-|---|---:|---:|---:|---:|
-| CyberMetric-500 | 84.20% | **86.80%** | **86.80%** | **86.80%** |
-| CTI-Bench-MCQ (500) | **66.00%** | 59.20% | 59.80% | 58.80% |
-| SecQA v1+v2 (210) | 98.10% | 98.10% | 98.10% | 98.10% |
-| **Overall (1,210)** | **79.09%** | 77.36% | 77.60% | 77.44% |
-
-*v5 → v6 is knowledge-neutral on MCQ (r6 targeted agentic robustness). Few-shot
-completion mode (base-model style) lifts all k3 versions: v6 scores 87.6/62.8/98.1.*
-
-Parse failures: 0 for all models in native modes.
-
-**Question-level overlap, v5 vs Foundation-Sec** (exclusive wins): CyberMetric —
-k3 +43 / F-Sec +30 · CTI-Bench — F-Sec +81 / k3 +50 · SecQA — 4 / 4 (both miss
-the same 8 questions). **v4 → v5 flips:** +8 gained / −5 lost across all
-benchmarks — knowledge MCQ is flat by design, since the v5 round targeted
-agentic SQLi behavior, not knowledge.
-
-**Prompting-mode sensitivity.** Foundation-Sec-8B collapses under zero-shot chat
-prompting (~34% accuracy, ~40% whitespace/echo responses) and only performs via
-few-shot completion — do not deploy it behind a naive chat template. k3-sec-8b
-is robust across both modes (86.8% chat vs 87.4% completion on CyberMetric, both
-versions) with 100% answer-format compliance.
-
-**Qualitative** (6 open-ended security prompts): Foundation-Sec gives terse,
-mostly-correct completion-style answers (~152 words avg); k3 v4/v5 give
-structured markdown answers (~264/~307 words avg) with finer-grained precision
-(e.g. LSASS dumping → T1003.001 sub-technique + detection methods, vs T1003 only).
-
-**CTI round experiment (v6cti branch, 2026-08-01):** a 2,521-trace ATT&CK-STIX-derived
-MCQ seed factory (decontaminated vs all three benchmarks, GLM-5.2 + Foundation-Sec
-teachers, deepseek-v4-flash judge) was trained on top of the r6 mix. Result: large
-CTI-Bench churn (+34/−37 flips) but no net gain and ~1.5 pt chat-mode regression —
-format conflict between justification-style traces and letter-only prompting was the
-main diagnosed cause. Negative result documented in the eval report; v2 plan is
-format-matched MCQ training + failure-targeted DPO. The `v6cti` branch remains
-available for inspection.
-
-**Takeaways.** k3-sec-8b (v4/v5) is stronger on broad general security knowledge
-(CyberMetric) and far more usable interactively; Foundation-Sec-8B is stronger on
-MITRE ATT&CK / threat-intel mapping (CTI-Bench) — the clearest data target for
-the next SFT round. Serving stack: vLLM 0.23, bf16, `max_model_len=8192`.
-Eval harness + raw per-question results: `sec-eval/` (drive_eval.py, results_*.json).
-
 ## Usage
 
 vLLM, short-task/eval serving (disable static YaRN):
@@ -136,7 +84,8 @@ competence is deliberately scoped to CTF/lab-grade tasks.
 | v2 | 149 traces | 6/10 single | failure-targeted r2 |
 | v3 | 157 (135 unique) | 4.0/10 (4 runs) | parser-fixed harness baseline |
 | v4 | 699 | 6.67/10 (3 runs) | GLM-5.2 scale-up, jwt fixed |
-| **v5** | **891** | **6.67/10 (3 runs: 4,8,8)** | sqli breakthrough, ioc fixed; harden regressed (newline stripping) |
+| v5 | 891 | 6.67/10 (3 runs: 4,8,8) | sqli breakthrough, ioc fixed; harden regressed (newline stripping) |
+| **v6** | **1076** | **7.0/10 (3 runs: 7,8,6)** | harden fixed, webshell solid; detect-dns/ioc still flip-flop |
 
 Weights are Apache-2.0 per the Qwen3 base; training traces were generated by GLM-5.2
 and Kimi teachers and filtered by automated judging.