dorb commited on
Commit
e647719
·
verified ·
1 Parent(s): 7c002d5

Publish ab9b-v2

Browse files

Q4_K_M GGUF, three LoRA adapters, evaluation protocol and results.
Fresh repository. New model card.

.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ ab9b-v2-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,107 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: Qwen/Qwen3.5-9B
4
+ base_model_relation: finetune
5
+ library_name: gguf
6
+ pipeline_tag: text-generation
7
+ language:
8
+ - en
9
+ tags:
10
+ - cybersecurity
11
+ - cve
12
+ - cwe
13
+ - cvss
14
+ - tool-use
15
+ - agent
16
+ - lora
17
+ - gguf
18
+ - qwen3
19
+ ---
20
+
21
+ # ab9b-v2
22
+
23
+ Defensive-cyber model on Qwen3.5-9B. Merged and quantised to Q4_K_M. Run the GGUF. The LoRA is here if you want the bf16 base plus the adapter instead.
24
+
25
+ Trained for CVE→CWE, CVSS vectors, threat-intel questions, vulnerability review, and blue-team / DFIR / audit loops that call tools. Answers are short and lead with the conclusion. Thinking is off.
26
+
27
+ ## Run
28
+
29
+ ```bash
30
+ huggingface-cli download dorb/ab9b-v2 ab9b-v2-Q4_K_M.gguf --local-dir .
31
+
32
+ llama-server -m ab9b-v2-Q4_K_M.gguf -ngl 999 -fa on --jinja \
33
+ -c 32768 -np 1 --chat-template-kwargs '{"enable_thinking":false}'
34
+ ```
35
+
36
+ OpenAI `tools` work with `--jinja`. The text protocol it was trained on is `CALL: tool_name(arg="…")` and then `FINAL: …`.
37
+
38
+ The LoRA that was merged into that GGUF:
39
+
40
+ ```python
41
+ from transformers import AutoModelForCausalLM, AutoTokenizer
42
+ from peft import PeftModel
43
+
44
+ base = "Qwen/Qwen3.5-9B"
45
+ model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
46
+ model = PeftModel.from_pretrained(model, "dorb/ab9b-v2", subfolder="adapters/v2a-ck900")
47
+ tokenizer = AutoTokenizer.from_pretrained(base)
48
+ ```
49
+
50
+ `adapters/v2b-final` and `adapters/v2c-final` are the later rounds. They were scored on the dev tier and not selected. Details are in [`evidence/REPORT.md`](evidence/REPORT.md).
51
+
52
+ ## Test tier
53
+
54
+ One scoring pass. Reference is `ab9b-r1` on the same base. Paired bootstrap, 95% CI.
55
+
56
+ | | ab9b-v2 | ab9b-r1 | base |
57
+ |---|---:|---:|---:|
58
+ | cyber composite, 7 families | 0.739 | 0.725 | 0.719 |
59
+ | CVE → CWE (n = 329) | 0.763 | 0.608 | |
60
+ | CVSS vector (n = 334) | 0.812 | 0.715 | |
61
+ | native tool-call loop, solved (n = 87) | 0.483 | 0.172 | 0.157 |
62
+ | plain `CALL:` loop, solved (n = 89) | 0.528 | 0.483 | 0.236 |
63
+ | single-call tool exact (n = 39) | 0.897 | 0.923 | 0.949 |
64
+ | median seconds per answer | 1.89 | 3.68 | |
65
+ | completion tokens per answer | 219 | 377 | |
66
+
67
+ The composite move is +0.014 [−0.006, +0.033]. That misses the pre-registered +0.05 bar. The gains that clear their intervals are CVE→CWE (+0.155), CVSS (+0.097), and the native tool-call loop (+0.310). Median answer time is about half. Token count is about half. Decode speed is the same (~110 tok/s).
68
+
69
+ On the same test items, against the other models that finished: wins over Cyber-Ornith-9B, Foundation-Sec-8B-Instruct, Primus-Merged-8B, SecGPT-7B, and the Qwen3.5-9B base. Ties with `ab9b-r1` and with Huihui-Qwen3.8-27B-abliterated. The 27B tie is at roughly twice the speed per quality point. Full table: [`evidence/results/league_verdict.txt`](evidence/results/league_verdict.txt).
70
+
71
+ ## Training
72
+
73
+ One LoRA round continued from `ab9b-r1`. r = 32, α = 32, lr 1e-4 cosine, one epoch, 1,063 steps, 16.06M tokens, 4.2 hours on one RTX 3090 (24 GB). Data is defensive: NVD CVE classification, vulnerability review, threat-intel Q&A, ATT&CK / CAPEC / CWE, analyst-brief rewrites, and oracle-checked agent episodes. Nothing from the reserved eval families.
74
+
75
+ The shipped checkpoint is `v2a-ck900`. Two later rounds and two weight interpolations lost the pre-registered dev rule. `v2b` raised the agent scores and broke tool-call hygiene (it invents missing arguments). `v2c` repaired that hygiene and still scored below `v2a-ck900`, so it was never test-scored. Both adapters are in the repo so those tables can be checked.
76
+
77
+ Protocol: [`evidence/GOAL.md`](evidence/GOAL.md). Numbers: [`evidence/REPORT.md`](evidence/REPORT.md).
78
+
79
+ ## Files
80
+
81
+ ```
82
+ ab9b-v2-Q4_K_M.gguf merged Q4_K_M, 5.8 GB
83
+ adapters/v2a-ck900/ LoRA merged into the GGUF
84
+ adapters/v2b-final/ later round, not selected
85
+ adapters/v2c-final/ later round, not selected
86
+ evidence/GOAL.md protocol, written before the test pass
87
+ evidence/REPORT.md test table, league, rejected rounds
88
+ evidence/results/ raw pick and verdict files
89
+ SHA256SUMS GGUF and the three adapter weights
90
+ ```
91
+
92
+ ## Limits
93
+
94
+ Agent splits are small (89 / 87 / 39). Dev selection used 60 agent items, so a few points of noise is normal. Training families and the agent test share generators; eight reserved families were never trained on. CVE rows are the same task type as CTI-RCM and CTI-VSP, other CVEs, not the same items. Thinking mode, context past 64k, and languages other than English were not measured. The data is synthetic and defensive. It was not red-teamed.
95
+
96
+ ## License
97
+
98
+ Apache-2.0, same as Qwen3.5-9B. Public sources in the mix keep their own licences.
99
+
100
+ ```bibtex
101
+ @software{ab9b_v2_2026,
102
+ title = {ab9b-v2},
103
+ author = {dorb},
104
+ year = {2026},
105
+ url = {https://huggingface.co/dorb/ab9b-v2}
106
+ }
107
+ ```
SHA256SUMS ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ c67bcf79d45e70ac8a3c53c7e49e2f229b0a46e5232f1f483cce13680e5dc4cf ab9b-v2-Q4_K_M.gguf
2
+ 3389d0b708757034f879977cecdc3b7d14d5304c130d2f327d470d27795deea7 adapters/v2a-ck900/adapter_model.safetensors
3
+ cd6d20b7524f638ccb281c4156885cb33dd8afa7f0b359fa076307e16155b11a adapters/v2b-final/adapter_model.safetensors
4
+ ca39bd15831c37799a3ae74c4049c85295f293b07bbab9c758364e4ce1907e05 adapters/v2c-final/adapter_model.safetensors
ab9b-v2-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c67bcf79d45e70ac8a3c53c7e49e2f229b0a46e5232f1f483cce13680e5dc4cf
3
+ size 5796867904
adapters/v2a-ck900/adapter_config.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": {
6
+ "base_model_class": "Qwen3_5ForConditionalGeneration",
7
+ "parent_library": "transformers.models.qwen3_5.modeling_qwen3_5",
8
+ "unsloth_fixed": true
9
+ },
10
+ "base_model_name_or_path": "Qwen/Qwen3.5-9B",
11
+ "bias": "none",
12
+ "corda_config": null,
13
+ "ensure_weight_tying": false,
14
+ "eva_config": null,
15
+ "exclude_modules": null,
16
+ "fan_in_fan_out": false,
17
+ "inference_mode": true,
18
+ "init_lora_weights": true,
19
+ "kasa_config": null,
20
+ "layer_replication": null,
21
+ "layers_pattern": null,
22
+ "layers_to_transform": null,
23
+ "loftq_config": {},
24
+ "lora_alpha": 32,
25
+ "lora_bias": false,
26
+ "lora_dropout": 0.0,
27
+ "lora_ga_config": null,
28
+ "megatron_config": null,
29
+ "megatron_core": "megatron.core",
30
+ "modules_to_save": null,
31
+ "monteclora_config": null,
32
+ "peft_type": "LORA",
33
+ "peft_version": "0.21.1",
34
+ "qalora_group_size": 16,
35
+ "r": 32,
36
+ "rank_pattern": {},
37
+ "revision": null,
38
+ "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}(?!.*\\.experts\\.\\d)\\..*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj)))",
39
+ "target_parameters": null,
40
+ "task_type": "CAUSAL_LM",
41
+ "trainable_token_indices": null,
42
+ "use_bdlora": null,
43
+ "use_dora": false,
44
+ "use_qalora": false,
45
+ "use_rslora": false,
46
+ "velora_config": null
47
+ }
adapters/v2a-ck900/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3389d0b708757034f879977cecdc3b7d14d5304c130d2f327d470d27795deea7
3
+ size 320924856
adapters/v2b-final/adapter_config.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": {
6
+ "base_model_class": "Qwen3_5ForConditionalGeneration",
7
+ "parent_library": "transformers.models.qwen3_5.modeling_qwen3_5",
8
+ "unsloth_fixed": true
9
+ },
10
+ "base_model_name_or_path": "Qwen/Qwen3.5-9B",
11
+ "bias": "none",
12
+ "corda_config": null,
13
+ "ensure_weight_tying": false,
14
+ "eva_config": null,
15
+ "exclude_modules": null,
16
+ "fan_in_fan_out": false,
17
+ "inference_mode": true,
18
+ "init_lora_weights": true,
19
+ "kasa_config": null,
20
+ "layer_replication": null,
21
+ "layers_pattern": null,
22
+ "layers_to_transform": null,
23
+ "loftq_config": {},
24
+ "lora_alpha": 32,
25
+ "lora_bias": false,
26
+ "lora_dropout": 0.0,
27
+ "lora_ga_config": null,
28
+ "megatron_config": null,
29
+ "megatron_core": "megatron.core",
30
+ "modules_to_save": null,
31
+ "monteclora_config": null,
32
+ "peft_type": "LORA",
33
+ "peft_version": "0.21.1",
34
+ "qalora_group_size": 16,
35
+ "r": 32,
36
+ "rank_pattern": {},
37
+ "revision": null,
38
+ "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}(?!.*\\.experts\\.\\d)\\..*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj)))",
39
+ "target_parameters": null,
40
+ "task_type": "CAUSAL_LM",
41
+ "trainable_token_indices": null,
42
+ "use_bdlora": null,
43
+ "use_dora": false,
44
+ "use_qalora": false,
45
+ "use_rslora": false,
46
+ "velora_config": null
47
+ }
adapters/v2b-final/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd6d20b7524f638ccb281c4156885cb33dd8afa7f0b359fa076307e16155b11a
3
+ size 320924856
adapters/v2c-final/adapter_config.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": {
6
+ "base_model_class": "Qwen3_5ForConditionalGeneration",
7
+ "parent_library": "transformers.models.qwen3_5.modeling_qwen3_5",
8
+ "unsloth_fixed": true
9
+ },
10
+ "base_model_name_or_path": "Qwen/Qwen3.5-9B",
11
+ "bias": "none",
12
+ "corda_config": null,
13
+ "ensure_weight_tying": false,
14
+ "eva_config": null,
15
+ "exclude_modules": null,
16
+ "fan_in_fan_out": false,
17
+ "inference_mode": true,
18
+ "init_lora_weights": true,
19
+ "kasa_config": null,
20
+ "layer_replication": null,
21
+ "layers_pattern": null,
22
+ "layers_to_transform": null,
23
+ "loftq_config": {},
24
+ "lora_alpha": 32,
25
+ "lora_bias": false,
26
+ "lora_dropout": 0.0,
27
+ "lora_ga_config": null,
28
+ "megatron_config": null,
29
+ "megatron_core": "megatron.core",
30
+ "modules_to_save": null,
31
+ "monteclora_config": null,
32
+ "peft_type": "LORA",
33
+ "peft_version": "0.21.1",
34
+ "qalora_group_size": 16,
35
+ "r": 32,
36
+ "rank_pattern": {},
37
+ "revision": null,
38
+ "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}(?!.*\\.experts\\.\\d)\\..*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj)))",
39
+ "target_parameters": null,
40
+ "task_type": "CAUSAL_LM",
41
+ "trainable_token_indices": null,
42
+ "use_bdlora": null,
43
+ "use_dora": false,
44
+ "use_qalora": false,
45
+ "use_rslora": false,
46
+ "velora_config": null
47
+ }
adapters/v2c-final/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca39bd15831c37799a3ae74c4049c85295f293b07bbab9c758364e4ce1907e05
3
+ size 320924856
evidence/GOAL.md ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ab9b-v2 — design and evaluation protocol
2
+
3
+ A LoRA on Qwen3.5-9B for defensive cybersecurity: CVE triage, threat-intel Q&A, vulnerability review, and native tool calling in blue-team / DFIR / audit agent loops.
4
+
5
+ This document fixes the method, the acceptance criteria, the data rules, and the decision rules **before any model is scored on the test tier**. It is checked in alongside the shipped weights so every claim in `REPORT.md` can be audited against a pre-registered gate.
6
+
7
+ ## 1. Model under test
8
+
9
+ - Base: Qwen3.5-9B dense (Apache-2.0), thinking disabled, MTP draft head bit-identical to the base.
10
+ - Adaptation: LoRA r = 32, α = 32, continued from the prior round `ab9b-r1`.
11
+ - Serving: `llama.cpp` server, identical flags for every model in the comparison.
12
+
13
+ ## 2. Scoring
14
+
15
+ ### 2.1 Cyber composite (primary)
16
+
17
+ Unweighted mean of seven family scores, each in [0, 1], zero-shot chat, held-out **test** tier, scored exactly **once per model tag**.
18
+
19
+ | family | source |
20
+ |---|---|
21
+ | F1 CyberMetric | `tihanyin/CyberMetric` — MCQ |
22
+ | F2 SecEval | `XuanwuAI/SecEval` — MCQ |
23
+ | F3 CTIBench | `AI4Sec/cti-bench` — mean of {mcq, rcm, vsp, ate, taa} |
24
+ | F4 SecBench | `secbench-hf/SecBench` — English MCQ |
25
+ | F5 SecQA | `zefang-liu/secqa` — v1 + v2 |
26
+ | F6 CyberBench | JPMorgan CyberBench — 10 datasets, 5-shot, 200-item seeded cap, 512 new tokens |
27
+ | F7 vuln2 | own CWE / safe-code harness, sealed tier |
28
+
29
+ ### 2.2 Agent tracks (secondary; not in the composite)
30
+
31
+ - `agent-plain solved` — in-text `CALL: … / FINAL: …` protocol, test tier.
32
+ - `agent-native solved` — native OpenAI tool-call loop, test tier.
33
+ - `toolcall exact` — single-call exact match on 39 fixed situations.
34
+
35
+ ### 2.3 Guards (dev tier; reported, not aggregated)
36
+
37
+ `mmlu_sec`, `coding2`, `agent2`, and XSTest safe compliance; each must stay within 0.03 of `ab9b-r1`.
38
+
39
+ ### 2.4 Speed
40
+
41
+ `quality_per_second = cyber_composite / median wall-seconds per item at batch 1`, under the same server flags as the league.
42
+
43
+ ## 3. Acceptance criteria (pre-registered)
44
+
45
+ Against the previous round `ab9b-r1`, paired bootstrap 95% CI on shared items:
46
+
47
+ | id | criterion | gate |
48
+ |---|---|---|
49
+ | C1 | cyber composite (test) | diff ≥ +0.05 absolute **and** paired lower bound > 0 |
50
+ | C2a | agent-plain solved (test) | diff ≥ +0.15 absolute **and** paired lower bound > 0 |
51
+ | C2b | native tool-call exact (test, n = 39) | not below `ab9b-r1` |
52
+ | C2c | agent2 guard | not more than 0.03 below `ab9b-r1` |
53
+ | C3 | median wall-time, batch 1 | ≤ 1.10× `ab9b-r1` |
54
+ | C3 | completion tokens per solved task | not above `ab9b-r1` |
55
+ | C3b | MTP head tensors | bit-identical to the base |
56
+ | C4 | dev guards (mmlu_sec, coding2, XSTest safe) | each within 0.03 of `ab9b-r1` |
57
+ | C5 | XSTest unsafe-refusal | not below the lower 95% bound of `ab9b-r1` |
58
+
59
+ The release is **accepted** iff all of C1–C5 hold. If any criterion misses, the report states exactly which hold and which do not; no stronger wording is used.
60
+
61
+ ## 4. League and allowed claims
62
+
63
+ The league is every generative LLM on the Hugging Face Hub that (a) presents itself as cybersecurity-specialised, (b) has a GGUF at approximately Q4 quality, (c) is ≤ 24 GB, and (d) has either ≥ 1 000 downloads or ≥ 20 likes or an established organisation behind it. One entry per base / fine-tune family (best-documented repo), plus Qwen3.5-9B base and `ab9b-r1`. All members run under identical server flags; models that need a different chat template get the template that `llama.cpp` ships in the GGUF (`--jinja`), with thinking disabled where the template supports it and `<think>…</think>` stripped otherwise.
64
+
65
+ Verdicts (paired bootstrap on shared items, 95%; a win needs the paired lower bound of the difference > 0):
66
+
67
+ - **W1 size-fair** — `ab9b-v2` beats every ≤ 10 B league member on the cyber composite.
68
+ - **W2 speed-fair** — `ab9b-v2` has the highest `quality_per_second` of the whole league.
69
+ - **W3 absolute** — `ab9b-v2` ≥ every league member of any size on the cyber composite (point difference ≥ 0 and paired 95% upper bound ≥ 0, against every member).
70
+
71
+ The sentence "wins any other cyber LLM" is allowed only if **W1 ∧ W3 ∧ C4** hold. Otherwise the report states exactly which of W1 / W2 / W3 / C4 hold and nothing stronger.
72
+
73
+ ## 5. Data rules
74
+
75
+ 1. **Defensive and benign only.** No exploit, malware, or weaponisation material. No CTF-solution dumps. No attack-payload corpora.
76
+ 2. **Permissive licences only** for training rows (Apache-2.0, MIT, CC0, CC-BY, ODC-BY); non-commercial sets are excluded.
77
+ 3. **Text that enters training is written by official open-weights models** under Apache-2.0 / MIT that are not themselves fine-tuned on proprietary-model outputs. Uncensored derivatives are used only as reviewers and brainstormers; their text enters training only as oracle-verified agent episodes (backup teacher).
78
+ 4. **Decontamination against every benchmark used**: 13-gram overlap on normalised text, embedding cosine ≥ 0.90, fact-level exclusion by CVE-ID, CWE-ID, and ATT&CK technique-ID. The 8 reserved agent families are never trained on.
79
+ 5. **Agent episodes are verified by programmatic oracles**; the student's own rollouts enter training only after oracle acceptance.
80
+
81
+ ## 6. Decision rules
82
+
83
+ ### 6.1 Candidate selection within a run (dev tier only)
84
+
85
+ A candidate is **eligible** if every guard (mmlu_sec, coding2, agent2, XSTest safe) is within 0.03 of `ab9b-r1` **and** the dev-tier single-call tool-call exact is within 0.03 of `ab9b-r1`. The score is
86
+
87
+ ```
88
+ score = ½ · (dev composite proxy gain / 0.05)
89
+ + ½ · (agent-plain dev solved gain / 0.15)
90
+ ```
91
+
92
+ against `ab9b-r1`. The highest-scoring eligible candidate wins; candidates within 0.02 of the best are tied, and the latest checkpoint of the tied set is picked. If no eligible candidate exists, the pick is `null` and the run is not automatically test-scored.
93
+
94
+ ### 6.2 Between-run replacement
95
+
96
+ A later round replaces the incumbent only if its pick is eligible **and** its score is at least +0.20 above the incumbent's. Ties within 0.02 go to the non-interpolated, later candidate. The test tier is scored exactly **once per model tag**; no selection decision is ever made from a test-tier number.
97
+
98
+ ### 6.3 Interpolation (optional)
99
+
100
+ If no new-round candidate is eligible, additional candidates may be built in weight space between the incumbent adapter and the new round's adapter at mixing weights w ∈ {0.5, 0.75}. Interpolated candidates are scored on the dev tier like the rest and considered under the same rule.
101
+
102
+ ## 7. Hard rules
103
+
104
+ - Training runs on one RTX 3090 (24 GB).
105
+ - No training data from the 8 reserved agent families or from the test tier of any benchmark listed in §2.1.
106
+ - Nothing is uploaded to Hugging Face except the final model repository.
107
+ - Every candidate scored and every decision taken is recorded in `REPORT.md` and the files under `results/`.
108
+ - No evaluation decision is ever made from a test-tier number; test numbers are read **after** the decision rule has picked the model.
evidence/REPORT.md ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ab9b-v2 — test-tier report
2
+
3
+ The model shipped by this repository is **`ab9b-v2a-ck900`** (checkpoint 900 of the first training round), chosen from three rounds by the dev-tier decision rule of [`GOAL.md`](GOAL.md) §6. It is the only candidate of this release that was scored on the test tier.
4
+
5
+ Two later rounds (`v2b` hard-example mining, `v2c` tool-policy repair) and two weight-space interpolations between `ab9b-r1` and `v2a-ck900` were evaluated on the dev tier only; they are documented in §4 for completeness and were **not** test-scored.
6
+
7
+ ## 1. Status against the pre-registered acceptance criteria
8
+
9
+ Reference = `ab9b-r1`, the previous round on the same base.
10
+
11
+ | id | criterion | result | verdict |
12
+ |---|---|---:|:---:|
13
+ | C1 | cyber composite +0.05 (paired CI > 0) | 0.739 vs 0.725 — diff +0.014 [−0.006, +0.033] | FAIL |
14
+ | C2a | agent-plain solved +0.15 (CI > 0) | 0.528 vs 0.483 — diff +0.045 [−0.056, +0.146], n = 89 | FAIL |
15
+ | C2b | native tool-call exact ≥ r1 (n = 39) | 0.897 vs 0.923 | FAIL |
16
+ | C2c | agent2 guard ≥ −0.03 | +0.050 | PASS |
17
+ | C3 | median wall-time ≤ 1.10× r1 | 1.169 s vs 1.226 s (0.95×) | PASS |
18
+ | C3 | completion tokens per solved task ≤ r1 | 139.6 vs 170.5 | PASS |
19
+ | C3b | MTP head bit-identical to the base | 15 / 15 tensors identical | PASS |
20
+ | C4 | guards within 0.03 (mmlu_sec, coding2, XSTest safe) | −0.005 / −0.017 / 0.000 | PASS |
21
+ | C5 | XSTest unsafe-refusal ≥ r1 lower-95% bound (0.093) | 0.179 vs 0.142 | PASS |
22
+
23
+ **Summary.** `ab9b-v2` holds C3, C4, and C5. It misses C1 and C2 on the paired lower bound. The significant gains concentrate in CVE triage, CVSS vector scoring, native tool-call solve rate, answer length, latency, and style — see §2. The release is therefore a **targeted improvement** over `ab9b-r1`, not a composite-tier replacement.
24
+
25
+ ## 2. Where the gains are (test tier, paired bootstrap 95%)
26
+
27
+ | task | n | ab9b-v2 | ab9b-r1 | diff | 95% CI | verdict |
28
+ |---|---:|---:|---:|---:|---|:---:|
29
+ | CTI-RCM — CVE → CWE | 329 | **0.763** | 0.608 | +0.155 | [+0.113, +0.204] | WIN |
30
+ | CTI-VSP — CVSS vector | 334 | **0.812** | 0.715 | +0.097 | [+0.073, +0.122] | WIN |
31
+ | agent-native solved | 87 | **0.483** | 0.172 | +0.310 | [+0.195, +0.425] | WIN |
32
+ | agent-plain solved | 89 | 0.528 | 0.483 | +0.045 | [−0.056, +0.146] | TIE |
33
+ | CyberMetric | 590 | 0.897 | 0.900 | −0.003 | [−0.017, +0.010] | TIE |
34
+ | SecEval | 654 | 0.755 | 0.749 | +0.006 | [−0.015, +0.026] | TIE |
35
+ | CTI-MCQ | 726 | 0.705 | 0.711 | −0.006 | [−0.032, +0.022] | TIE |
36
+ | CTI-ATE | 22 | 0.242 | 0.306 | −0.065 | [−0.143, +0.012] | TIE |
37
+ | CTI-TAA | 16 | 0.063 | 0.063 | 0.000 | [+0.000, +0.000] | TIE |
38
+ | SecBench | 205 | 0.839 | 0.834 | +0.005 | [−0.024, +0.034] | TIE |
39
+ | SecQA | 73 | 0.932 | 0.945 | −0.014 | [−0.041, +0.000] | TIE |
40
+ | **cyber composite** | — | **0.739** | 0.725 | +0.014 | [−0.006, +0.033] | TIE |
41
+
42
+ Secondary speed and style numbers (14 analyst prompts, batch-1, temperature 0; 50 fixed SOC-analyst questions):
43
+
44
+ - Median wall-time per answer: **1.89 s** vs 3.68 s (−49%).
45
+ - Completion tokens per answer: **219** vs 377 (−42%).
46
+ - Decode speed: 107 vs 112 tok/s (equal).
47
+ - MTP speculative draft (`--spec-type draft-mtp`): r1 +13% (acceptance 0.47), `ab9b-v2` +1% (acceptance 0.42) — not worth enabling on this model.
48
+ - Style indicators: bottom-line-first first line 56% → **98%**; closing `Next:` action line 0% → **86%**; answers > 180 words 88% → **2%**; markdown headings or bold labels 50% → **6%**.
49
+ - Style rubric judge (local HauhauCS 35B-A3B, 50 answers): bottom line first 0.74 → **1.00**, concise 0.50 → **1.00**, structured 0.74 → **0.94**, actionable 0.88 → **1.00**, no invented specifics 0.86 → 0.88.
50
+
51
+ ## 3. League verdict (test tier, seven members complete)
52
+
53
+ Same server flags for every member, paired bootstrap on shared items, 95%.
54
+
55
+ | opponent | size | diff vs ab9b-v2 | 95% CI | verdict |
56
+ |---|---|---:|---|:---:|
57
+ | `ab9b-r1` | 9 B | +0.014 | [−0.006, +0.033] | TIE |
58
+ | Cyber-Ornith-9B | 9 B | +0.092 | [+0.072, +0.112] | WIN |
59
+ | Foundation-Sec-8B-Instruct | 8 B | +0.073 | [+0.050, +0.095] | WIN |
60
+ | Primus-Merged-8B | 8 B | +0.117 | [+0.095, +0.139] | WIN |
61
+ | SecGPT-7B | 7 B | +0.115 | [+0.094, +0.137] | WIN |
62
+ | Qwen3.5-9B base | 9 B | +0.020 | [+0.002, +0.038] | WIN |
63
+ | Huihui-Qwen3.8-27B-abliterated | 27 B | +0.019 | [−0.002, +0.040] | TIE |
64
+
65
+ - **W1 size-fair** (beat every ≤ 10 B league member, `ab9b-r1` included): **does not hold** — tie with `ab9b-r1`.
66
+ - **W2 speed-fair** (highest `quality_per_second` of the league): **does not hold** — three smaller distilled models have 2 – 3× the quality-per-second.
67
+ - **W3 absolute** (≥ every league member of any size on the cyber composite): **holds** across the seven complete members. The 35 B-A3B league member (HauhauCS) could not be completed — its server returned a CUDA fault during the `cti-ate` stage, so `cti-ate`, `cti-taa`, `secbench`, and `secqa` were never scored for it; its CyberBench and own-sealed stages did complete. W3 therefore stays undecided against that one model.
68
+
69
+ Per the claim policy in `GOAL.md` §4, the sentence "wins any other cyber LLM" is **not allowed**.
70
+
71
+ Against the Huihui-Qwen3.8-27B abliteration specifically, `ab9b-v2` is a tie on the composite (+0.019 [−0.002, +0.040]) and about **1.9× faster per quality point** (`quality_per_second` 0.632 vs 0.333).
72
+
73
+ ## 4. Rounds scored on the dev tier and not selected
74
+
75
+ ### 4.1 `v2b` — hard-example mining and new families
76
+
77
+ Continued from `v2a-ck900` at lr 5e-5, one epoch, 777 steps, 3.5 h, 19 007 rows / 11.4 M tokens: K = 2 rejection-sampling rollouts of `v2a-ck900` on 26 training families, oracle trajectories of the episodes `v2a-ck900` could not solve (203 hard-example rows), oracle rows of 10 new audit / IR families (`sysctl_effective`, `k8s_rbac`, `logrotate_retention`, `config_diff_risk`, `alert_join`, `authkeys_audit`, `pg_hba_audit`, `suid_audit`, `auditd_seq_gap`, `commit_secret_origin`), 35% replay of the `v2a` knowledge streams, and 714 technique-extraction rows targeting the CTI-ATE regression.
78
+
79
+ | dev metric (n) | ab9b-r1 | v2a-ck900 | **v2b-final** |
80
+ |---|---:|---:|---:|
81
+ | agent-plain solved (60) | 0.550 | 0.633 | **0.683** |
82
+ | agent-native solved (59) | 0.288 | 0.593 | **0.661** |
83
+ | **tool-call exact (91)** | 0.978 | 0.956 | **0.846** |
84
+ | CTI-ATE (38) | 0.358 | 0.230 | 0.371 |
85
+ | dev composite proxy | 0.7533 | 0.7649 | 0.7616 |
86
+
87
+ `v2b` improved every agent track **and** fixed the CTI-ATE regression, but single-call tool-call exact dropped from 0.956 to 0.846 — well below the dev-tier eligibility threshold of r1 − 0.03 = 0.948. Per-category diagnosis (n = 91 dev): `missing_arg` 0.33 vs 1.00 (the model invents a value instead of asking); `single` 0.81 vs 1.00 (an unrequested optional `reason` argument is added). This is exactly the regression the hygiene guard exists for, so every `v2b` candidate was **ineligible** by `GOAL.md` §6.1 and the pick was `null`. Likely cause: every native episode of round 2 has complete information and ends in a call, and the single-call anchors of round 1 were replayed at only 35%.
88
+
89
+ ### 4.2 `v2c` — tool-policy repair
90
+
91
+ Continued from `v2b-final` at lr 5e-5, one epoch, 296 steps, 1.3 h, 7 336 rows / 4.35 M tokens: 25% replay of the `v2b` mix (with the analyst-brief restyled answers at 100%) plus 1 600 programmatic tool-policy rows. The new rows cover 24 tool schemas disjoint from the 10 evaluation tools; the behaviours taught are: exact call with only the required arguments; optional values when the user gives them explicitly; several independent calls when appropriate; one short question when a required value is missing; abstain in text when no tool fits. Every row was schema-validated; a word-4-gram decontamination against the 130 evaluation prompts and the 10 evaluation tool names found 6 overlaps in the first draft and 0 after cleanup.
92
+
93
+ | dev metric | ab9b-r1 | v2a-ck900 | v2b-final | **v2c-final** |
94
+ |---|---:|---:|---:|---:|
95
+ | agent-plain solved (60) | 0.550 | 0.633 | 0.683 | **0.633** |
96
+ | agent-native solved (59) | 0.288 | 0.583 | 0.661 | **0.627** |
97
+ | tool-call exact (91) | 0.978 | 0.956 | 0.846 | **0.978** |
98
+ | dev composite proxy | 0.7533 | 0.7649 | 0.7616 | 0.7581 |
99
+ | decision score vs r1 | — | +0.393 | +0.53 (ineligible) | **+0.326** |
100
+
101
+ The repair worked on the thing it targeted: tool-call exact returned to 0.978, equal to `ab9b-r1`; guards are clean (mmlu_sec +0.005, coding2 +0.004, agent2 +0.150, XSTest safe −0.011); the agent-native solve rate was mostly kept (0.627 vs 0.661). But the pre-registered score is +0.326, below the incumbent's +0.393, so by the between-run rule `v2c` did not replace `v2a-ck900` and was never test-scored.
102
+
103
+ An honest reading of the dev table: `v2c` and `v2a-ck900` are within the noise of 60 agentic items; `v2c` is the slightly better agent **with tool-use hygiene intact**, while `v2a-ck900` is the slightly stronger knowledge model. The rule picks the latter, and so the only test scoring that exists is `v2a-ck900`'s. The `v2b` and `v2c` adapters are in this repository under `adapters/` so the dev tables can be reproduced. They were not selected and were not test-scored.
104
+
105
+ ### 4.3 Weight-space interpolation
106
+
107
+ Between `ab9b-r1` and `v2a-ck900` at mixing weights w ∈ {0.5, 0.75}: scores +0.115 and +0.113, both well below the incumbent's +0.393. Dropped.
108
+
109
+ ### 4.4 In-distribution check on the training families (not an acceptance criterion)
110
+
111
+ Fresh episodes (seed 77, greedy, plain protocol, 10 per family) on the 26 training families: `ab9b-r1` 0.60, `v2a-ck900` 0.71, `v2b-final` 0.79. The 10 new audit / IR families were learned only partly by `v2b` (alert_join 0.4, sysctl_effective 0.4, logrotate_retention 0.5, pg_hba_audit 0.5); `firewall_audit` stayed weak for both (`v2a-ck900` 0.1, `v2b-final` 0.4). This is in-distribution and not an acceptance criterion; the 8 reserved agent families were never trained on and are the correct transfer measure.
112
+
113
+ ## 5. Reproducibility & evidence
114
+
115
+ The raw selection and verdict files are in [`results/`](results/):
116
+
117
+ - `v2a_pick.json`, `v2b_pick.json`, `v2c_pick.json` — every candidate scored, every eligibility test, the final pick.
118
+ - `v2b_dev_summary.txt`, `v2c_dev_summary.txt` — dev-tier paired tables for the two rounds that were not selected.
119
+ - `between_run_decision.json` — the between-run decision over the three rounds and the two interpolations.
120
+ - `v2a_criteria.txt` — the C1–C5 acceptance report of §1.
121
+ - `v2a_accept_report.txt` — the full paired-bootstrap tables underlying §2.
122
+ - `league_verdict.txt` — the W1 / W2 / W3 verdict and the per-opponent `quality_per_second`.
123
+
124
+ ## 6. Limitations and honest caveats
125
+
126
+ - Agent test splits are small (n = 89 plain, n = 87 native, n = 39 tool-call). Even a true +0.15 effect clears the paired lower bound only about half of the time at this size.
127
+ - Dev-tier selection among candidates uses 60 agent items and cap-400 battery tasks; the noise of that selection is a few points.
128
+ - The training agent families and the agent test items share generators in part; the 8 reserved agent families were never trained on and are the correct transfer measure.
129
+ - NVD CVE → CWE / CVSS rows are the same task **type** as CTI-RCM, CTI-VSP, and CyberBench `cve` items (other CVEs) — in-domain, not identical-item leakage.
130
+ - The CyberBench deviation (first 128 tokens, not 512, used for the embedding stage of decontamination) is a documented deviation from the instruction; the n-gram stage covers the full text.
131
+ - Thinking mode, context above 64 k tokens, and non-English use were not evaluated.
132
+ - The 35 B-A3B league member (HauhauCS) did not complete the battery (CUDA fault); its W3 verdict is undecided.
133
+ - Training data is synthetic and defensive; it has not been red-teamed for misuse. This release is a research artefact.
evidence/results/between_run_decision.json ADDED
@@ -0,0 +1,444 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "ref": "ab9b-r1 (dev)",
3
+ "margin": 0.2,
4
+ "incumbent": {
5
+ "label": "v2a-ck900",
6
+ "battery_tag": "v2a-ck900 (dev)",
7
+ "eligible": true,
8
+ "reasons": [],
9
+ "families_used": [
10
+ "ctibench",
11
+ "cybermetric",
12
+ "secbench",
13
+ "seceval",
14
+ "secqa",
15
+ "vuln2"
16
+ ],
17
+ "proxy": 0.76485,
18
+ "ref_proxy": 0.7532666666666668,
19
+ "families": {
20
+ "ctibench": [
21
+ 0.5748,
22
+ 0.5141
23
+ ],
24
+ "cybermetric": [
25
+ 0.885,
26
+ 0.9075
27
+ ],
28
+ "secbench": [
29
+ 0.79,
30
+ 0.8
31
+ ],
32
+ "seceval": [
33
+ 0.735,
34
+ 0.74
35
+ ],
36
+ "secqa": [
37
+ 0.9927,
38
+ 0.9927
39
+ ],
40
+ "vuln2": [
41
+ 0.6116,
42
+ 0.5653
43
+ ]
44
+ },
45
+ "g1": 0.23166666666666558,
46
+ "guards": {
47
+ "mmlu_sec": -0.0049,
48
+ "coding2": -0.0167,
49
+ "agent2": 0.05,
50
+ "xstest_safe_compliance": 0.0
51
+ },
52
+ "agentcy": {
53
+ "plain_solved": {
54
+ "n": 60,
55
+ "a": 0.6333,
56
+ "b": 0.55,
57
+ "diff": 0.0833,
58
+ "ci95": [
59
+ -0.0333,
60
+ 0.2
61
+ ],
62
+ "verdict": "TIE"
63
+ },
64
+ "native_solved": {
65
+ "n": 59,
66
+ "a": 0.5932,
67
+ "b": 0.2881,
68
+ "diff": 0.3051,
69
+ "ci95": [
70
+ 0.1695,
71
+ 0.4407
72
+ ],
73
+ "verdict": "WIN"
74
+ },
75
+ "toolcall_exact": {
76
+ "n": 91,
77
+ "a": 0.956,
78
+ "b": 0.978,
79
+ "diff": -0.022,
80
+ "ci95": [
81
+ -0.0549,
82
+ 0.0
83
+ ],
84
+ "verdict": "TIE"
85
+ },
86
+ "plain_reserved": {
87
+ "n": 45,
88
+ "a": 0.5111,
89
+ "b": 0.4889,
90
+ "diff": 0.0222,
91
+ "ci95": [
92
+ -0.1111,
93
+ 0.1556
94
+ ],
95
+ "verdict": "TIE"
96
+ },
97
+ "plain_unseen": {
98
+ "n": 15,
99
+ "a": 1.0,
100
+ "b": 0.7333,
101
+ "diff": 0.2667,
102
+ "ci95": [
103
+ 0.0667,
104
+ 0.4667
105
+ ],
106
+ "verdict": "WIN"
107
+ }
108
+ },
109
+ "g2": 0.5553333333333333,
110
+ "score": 0.39349999999999946
111
+ },
112
+ "challengers": [
113
+ {
114
+ "label": "v2a-interp-w050",
115
+ "battery_tag": "v2a-interp-w050 (dev)",
116
+ "eligible": true,
117
+ "reasons": [],
118
+ "families_used": [
119
+ "ctibench",
120
+ "cybermetric",
121
+ "secbench",
122
+ "seceval",
123
+ "secqa",
124
+ "vuln2"
125
+ ],
126
+ "proxy": 0.7592333333333334,
127
+ "ref_proxy": 0.7532666666666668,
128
+ "families": {
129
+ "ctibench": [
130
+ 0.5714,
131
+ 0.5141
132
+ ],
133
+ "cybermetric": [
134
+ 0.895,
135
+ 0.9075
136
+ ],
137
+ "secbench": [
138
+ 0.79,
139
+ 0.8
140
+ ],
141
+ "seceval": [
142
+ 0.7375,
143
+ 0.74
144
+ ],
145
+ "secqa": [
146
+ 0.9927,
147
+ 0.9927
148
+ ],
149
+ "vuln2": [
150
+ 0.5688,
151
+ 0.5653
152
+ ]
153
+ },
154
+ "g1": 0.11933333333333351,
155
+ "guards": {
156
+ "mmlu_sec": 0.0196,
157
+ "coding2": 0.0084,
158
+ "agent2": -0.025,
159
+ "xstest_safe_compliance": 0.0053
160
+ },
161
+ "agentcy": {
162
+ "plain_solved": {
163
+ "n": 60,
164
+ "a": 0.5667,
165
+ "b": 0.55,
166
+ "diff": 0.0167,
167
+ "ci95": [
168
+ -0.05,
169
+ 0.1
170
+ ],
171
+ "verdict": "TIE"
172
+ },
173
+ "native_solved": {
174
+ "n": 56,
175
+ "a": 0.5893,
176
+ "b": 0.3036,
177
+ "diff": 0.2857,
178
+ "ci95": [
179
+ 0.1429,
180
+ 0.4286
181
+ ],
182
+ "verdict": "WIN"
183
+ },
184
+ "toolcall_exact": {
185
+ "n": 91,
186
+ "a": 0.967,
187
+ "b": 0.978,
188
+ "diff": -0.011,
189
+ "ci95": [
190
+ -0.033,
191
+ 0.0
192
+ ],
193
+ "verdict": "TIE"
194
+ },
195
+ "plain_reserved": {
196
+ "n": 45,
197
+ "a": 0.4889,
198
+ "b": 0.4889,
199
+ "diff": 0.0,
200
+ "ci95": [
201
+ -0.0889,
202
+ 0.0889
203
+ ],
204
+ "verdict": "TIE"
205
+ },
206
+ "plain_unseen": {
207
+ "n": 15,
208
+ "a": 0.8,
209
+ "b": 0.7333,
210
+ "diff": 0.0667,
211
+ "ci95": [
212
+ 0.0,
213
+ 0.2
214
+ ],
215
+ "verdict": "TIE"
216
+ }
217
+ },
218
+ "g2": 0.11133333333333334,
219
+ "score": 0.11533333333333343
220
+ },
221
+ {
222
+ "label": "v2a-interp-w075",
223
+ "battery_tag": "v2a-interp-w075 (dev)",
224
+ "eligible": false,
225
+ "reasons": [
226
+ "toolcall exact -0.0330"
227
+ ],
228
+ "families_used": [
229
+ "ctibench",
230
+ "cybermetric",
231
+ "secbench",
232
+ "seceval",
233
+ "secqa",
234
+ "vuln2"
235
+ ],
236
+ "proxy": 0.7590499999999999,
237
+ "ref_proxy": 0.7532666666666668,
238
+ "families": {
239
+ "ctibench": [
240
+ 0.5748,
241
+ 0.5141
242
+ ],
243
+ "cybermetric": [
244
+ 0.89,
245
+ 0.9075
246
+ ],
247
+ "secbench": [
248
+ 0.795,
249
+ 0.8
250
+ ],
251
+ "seceval": [
252
+ 0.74,
253
+ 0.74
254
+ ],
255
+ "secqa": [
256
+ 0.9927,
257
+ 0.9927
258
+ ],
259
+ "vuln2": [
260
+ 0.5618,
261
+ 0.5653
262
+ ]
263
+ },
264
+ "g1": 0.11566666666666281,
265
+ "guards": {
266
+ "mmlu_sec": -0.0098,
267
+ "coding2": 0.0042,
268
+ "agent2": 0.025,
269
+ "xstest_safe_compliance": 0.0
270
+ },
271
+ "agentcy": {
272
+ "plain_solved": {
273
+ "n": 60,
274
+ "a": 0.5667,
275
+ "b": 0.55,
276
+ "diff": 0.0167,
277
+ "ci95": [
278
+ -0.0833,
279
+ 0.1167
280
+ ],
281
+ "verdict": "TIE"
282
+ },
283
+ "native_solved": {
284
+ "n": 53,
285
+ "a": 0.6226,
286
+ "b": 0.3208,
287
+ "diff": 0.3019,
288
+ "ci95": [
289
+ 0.1509,
290
+ 0.4528
291
+ ],
292
+ "verdict": "WIN"
293
+ },
294
+ "toolcall_exact": {
295
+ "n": 91,
296
+ "a": 0.9451,
297
+ "b": 0.978,
298
+ "diff": -0.033,
299
+ "ci95": [
300
+ -0.0769,
301
+ 0.0
302
+ ],
303
+ "verdict": "TIE"
304
+ },
305
+ "plain_reserved": {
306
+ "n": 45,
307
+ "a": 0.5111,
308
+ "b": 0.4889,
309
+ "diff": 0.0222,
310
+ "ci95": [
311
+ -0.0889,
312
+ 0.1333
313
+ ],
314
+ "verdict": "TIE"
315
+ },
316
+ "plain_unseen": {
317
+ "n": 15,
318
+ "a": 0.7333,
319
+ "b": 0.7333,
320
+ "diff": 0.0,
321
+ "ci95": [
322
+ -0.2,
323
+ 0.2
324
+ ],
325
+ "verdict": "TIE"
326
+ }
327
+ },
328
+ "g2": 0.11133333333333334,
329
+ "score": 0.11349999999999807
330
+ },
331
+ {
332
+ "label": "v2c-final",
333
+ "battery_tag": "v2c-final (dev)",
334
+ "eligible": true,
335
+ "reasons": [],
336
+ "families_used": [
337
+ "ctibench",
338
+ "cybermetric",
339
+ "secbench",
340
+ "seceval",
341
+ "secqa",
342
+ "vuln2"
343
+ ],
344
+ "proxy": 0.7581000000000001,
345
+ "ref_proxy": 0.7532666666666668,
346
+ "families": {
347
+ "ctibench": [
348
+ 0.5792,
349
+ 0.5141
350
+ ],
351
+ "cybermetric": [
352
+ 0.8825,
353
+ 0.9075
354
+ ],
355
+ "secbench": [
356
+ 0.78,
357
+ 0.8
358
+ ],
359
+ "seceval": [
360
+ 0.745,
361
+ 0.74
362
+ ],
363
+ "secqa": [
364
+ 0.9927,
365
+ 0.9927
366
+ ],
367
+ "vuln2": [
368
+ 0.5692,
369
+ 0.5653
370
+ ]
371
+ },
372
+ "g1": 0.09666666666666712,
373
+ "guards": {
374
+ "mmlu_sec": 0.0049,
375
+ "coding2": 0.0042,
376
+ "agent2": 0.15,
377
+ "xstest_safe_compliance": -0.0107
378
+ },
379
+ "agentcy": {
380
+ "plain_solved": {
381
+ "n": 60,
382
+ "a": 0.6333,
383
+ "b": 0.55,
384
+ "diff": 0.0833,
385
+ "ci95": [
386
+ 0.0,
387
+ 0.1833
388
+ ],
389
+ "verdict": "TIE"
390
+ },
391
+ "native_solved": {
392
+ "n": 59,
393
+ "a": 0.6271,
394
+ "b": 0.2881,
395
+ "diff": 0.339,
396
+ "ci95": [
397
+ 0.1864,
398
+ 0.4915
399
+ ],
400
+ "verdict": "WIN"
401
+ },
402
+ "toolcall_exact": {
403
+ "n": 91,
404
+ "a": 0.978,
405
+ "b": 0.978,
406
+ "diff": 0.0,
407
+ "ci95": [
408
+ -0.033,
409
+ 0.033
410
+ ],
411
+ "verdict": "TIE"
412
+ },
413
+ "plain_reserved": {
414
+ "n": 45,
415
+ "a": 0.5333,
416
+ "b": 0.4889,
417
+ "diff": 0.0444,
418
+ "ci95": [
419
+ -0.0444,
420
+ 0.1556
421
+ ],
422
+ "verdict": "TIE"
423
+ },
424
+ "plain_unseen": {
425
+ "n": 15,
426
+ "a": 0.9333,
427
+ "b": 0.7333,
428
+ "diff": 0.2,
429
+ "ci95": [
430
+ 0.0,
431
+ 0.4
432
+ ],
433
+ "verdict": "TIE"
434
+ }
435
+ },
436
+ "g2": 0.5553333333333333,
437
+ "score": 0.32600000000000023
438
+ }
439
+ ],
440
+ "chosen": null,
441
+ "chosen_battery_tag": null,
442
+ "chosen_gguf": null,
443
+ "rule": "Between-run rule (GOAL.md section 6.2): the pick is eligible iff no dev guard (mmlu_sec, coding2, agent2, xstest_safe_compliance) is more than 0.03 below the reference and the dev tool-call exact is within 0.03 of it. Score = mean(dev composite proxy gain / 0.05, agent-plain dev solved gain / 0.15). The highest-scoring eligible challenger replaces the incumbent iff its score >= incumbent score + margin (0.20); candidates within 0.02 of the best are tied, with a non-interpolated candidate preferred and ties broken by order of definition. Otherwise the incumbent is kept."
444
+ }
evidence/results/league_verdict.txt ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ VERDICT ours=ab9b-v2 (test) tier=test boot=2000 seed=0 cyber_composite(ours)=0.7389
2
+ composite comparison (diff = ours - league, paired bootstrap, win iff 95% lower bound > 0):
3
+ vs ab9b-r1 (test) <=10B ours 0.7389 theirs 0.7251 diff +0.0139 [-0.0056, +0.0328] TIE
4
+ vs Cyber-Ornith-9B (test) <=10B ours 0.7389 theirs 0.6474 diff +0.0915 [+0.0716, +0.1120] WIN
5
+ vs Foundation-Sec-8B-Instruct (test) <=10B ours 0.7389 theirs 0.6662 diff +0.0727 [+0.0501, +0.0950] WIN
6
+ vs Huihui-Qwen3.8-27B-abliterated (test) >10B ours 0.7389 theirs 0.7199 diff +0.0190 [-0.0019, +0.0399] TIE
7
+ vs Primus-Merged-8B (test) <=10B ours 0.7389 theirs 0.6218 diff +0.1172 [+0.0950, +0.1388] WIN
8
+ vs Qwen3.5-9B base (test) <=10B ours 0.7389 theirs 0.7190 diff +0.0199 [+0.0016, +0.0383] WIN
9
+ vs SecGPT-7B (test) <=10B ours 0.7389 theirs 0.6243 diff +0.1147 [+0.0936, +0.1365] WIN
10
+ W1 size-fair (beat every <=10B league model: win vs ['ab9b-r1 (test)', 'Cyber-Ornith-9B (test)', 'Foundation-Sec-8B-Instruct (test)', 'Primus-Merged-8B (test)', 'Qwen3.5-9B base (test)', 'SecGPT-7B (test)']): DOES NOT HOLD
11
+ W3 absolute (>= every league model of any size: no loss and point difference >= 0): HOLDS [strict reading, a win against all: no]
12
+ W2 speed-fair (highest quality_per_second of the whole league): DOES NOT HOLD
13
+ vs ab9b-r1 (test) qps ours 0.6323 theirs 0.5914 diff +0.0409 [-0.0523, +0.1755] TIE
14
+ vs Cyber-Ornith-9B (test) qps ours 0.6323 theirs 0.8760 diff -0.2437 [-0.3646, +0.0024] TIE
15
+ vs Foundation-Sec-8B-Instruct (test) qps ours 0.6323 theirs 1.4483 diff -0.8159 [-1.0777, -0.2690] LOSS
16
+ vs Huihui-Qwen3.8-27B-abliterated (test) qps ours 0.6323 theirs 0.3332 diff +0.2991 [+0.2058, +0.4323] WIN
17
+ vs Primus-Merged-8B (test) qps ours 0.6323 theirs 1.3864 diff -0.7540 [-1.1478, -0.2070] LOSS
18
+ vs Qwen3.5-9B base (test) qps ours 0.6323 theirs 0.6332 diff -0.0009 [-0.1268, +0.1761] TIE
19
+ vs SecGPT-7B (test) qps ours 0.6323 theirs 2.1020 diff -1.4697 [-1.7828, -1.1097] LOSS
20
+ guards (none more than 0.03 below ab9b-r1 (dev), dev tier): NOT DECIDABLE (missing values)
21
+ mmlu_sec not available
22
+ coding2 not available
23
+ agent2 not available
24
+ xstest_safe_compliance not available
25
+
26
+ CLAIM POLICY (GOAL.md section 1): 'wins any other cyber LLM' requires W1 AND W3 AND all guards.
27
+ NOT ALLOWED. State exactly: holds = ['W3']; does not hold = ['W1', 'W2']; not decidable = ['guards'] and nothing stronger.
evidence/results/v2a_accept_report.txt ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === agentcy: ab9b-v2 (test) vs ab9b-r1 (TEST, paired bootstrap)
2
+ paired comparison A=ab9b-v2 (test) B=ab9b-r1 (test) tier=test boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
3
+ task / metric A B diff 95% CI of diff n verdict
4
+ agent-plain solved 0.5281 0.4831 +0.0449 [-0.0562, +0.1461] 89 TIE
5
+ agent-plain exact 0.5281 0.4831 +0.0449 [-0.0562, +0.1461] 89 TIE
6
+ agent-plain score 0.5281 0.4831 +0.0449 [-0.0562, +0.1461] 89 TIE
7
+ agent-plain solved[reserved] 0.4400 0.4667 -0.0267 [-0.1333, +0.0800] 75 TIE
8
+ agent-plain solved[unseen] 1.0000 0.5714 +0.4286 [+0.1429, +0.7143] 14 WIN
9
+ agent-native solved 0.4828 0.1724 +0.3103 [+0.1954, +0.4253] 87 WIN
10
+ agent-native exact 0.4828 0.1724 +0.3103 [+0.1954, +0.4253] 87 WIN
11
+ agent-native score 0.4828 0.1724 +0.3103 [+0.1954, +0.4253] 87 WIN
12
+ agent-native solved[reserved] 0.4133 0.1467 +0.2667 [+0.1467, +0.3867] 75 WIN
13
+ agent-native solved[unseen] 0.9167 0.3333 +0.5833 [+0.3333, +0.8333] 12 WIN
14
+ toolcall exact 0.8974 0.9231 -0.0256 [-0.0769, +0.0000] 39 TIE
15
+
16
+ === agentcy: ab9b-v2 (test) vs base
17
+ paired comparison A=ab9b-v2 (test) B=Qwen3.5-9B base (test) tier=test boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
18
+ task / metric A B diff 95% CI of diff n verdict
19
+ agent-plain solved 0.5281 0.2360 +0.2921 [+0.1798, +0.4045] 89 WIN
20
+ agent-plain exact 0.5281 0.2360 +0.2921 [+0.1798, +0.4045] 89 WIN
21
+ agent-plain score 0.5281 0.2360 +0.2921 [+0.1798, +0.4045] 89 WIN
22
+ agent-plain solved[reserved] 0.4400 0.1867 +0.2533 [+0.1333, +0.3733] 75 WIN
23
+ agent-plain solved[unseen] 1.0000 0.5000 +0.5000 [+0.2143, +0.7857] 14 WIN
24
+ agent-native solved 0.4944 0.1573 +0.3371 [+0.2360, +0.4494] 89 WIN
25
+ agent-native exact 0.4944 0.1573 +0.3371 [+0.2360, +0.4494] 89 WIN
26
+ agent-native score 0.4944 0.1573 +0.3371 [+0.2360, +0.4494] 89 WIN
27
+ agent-native solved[reserved] 0.4133 0.1467 +0.2667 [+0.1600, +0.3733] 75 WIN
28
+ agent-native solved[unseen] 0.9286 0.2143 +0.7143 [+0.5000, +0.9286] 14 WIN
29
+ toolcall exact 0.8974 0.9487 -0.0513 [-0.1282, +0.0000] 39 TIE
30
+
31
+ === battery composite of ab9b-v2 (test)
32
+
33
+ cyber_composite (tier test): 0.7389 [0.721, 0.756] [FINAL]
34
+ F1 cybermetric 0.8966 ok
35
+ F2 seceval 0.7554 ok
36
+ F3 ctibench 0.5168 ok (mcq 0.705, rcm 0.763, vsp 0.812, ate 0.241, taa 0.062)
37
+ F4 secbench 0.8390 ok
38
+ F5 secqa 0.9315 ok
39
+ F6 cyberbench_jpm 0.6874 ok
40
+ F7 vuln2 0.5458 ok
41
+ guards (dev tier, never in the composite):
42
+ mmlu_sec -- missing
43
+ coding2 -- missing
44
+ agent2 -- missing
45
+ xstest_safe_compliance -- missing
46
+
47
+ === battery: ab9b-v2 (test) vs ab9b-r1 (TEST, paired)
48
+ paired comparison A=ab9b-v2 (test) B=ab9b-r1 (test) tier=test boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
49
+ task A B diff 95% CI of diff n_shared verdict
50
+ cybermetric 0.8966 0.9000 -0.0034 [-0.0169, +0.0102] 590 TIE
51
+ seceval 0.7554 0.7492 +0.0061 [-0.0153, +0.0260] 654 TIE
52
+ cti-mcq 0.7052 0.7107 -0.0055 [-0.0317, +0.0220] 726 TIE
53
+ cti-rcm 0.7629 0.6079 +0.1550 [+0.1125, +0.2036] 329 WIN
54
+ cti-vsp 0.8121 0.7152 +0.0969 [+0.0730, +0.1216] 334 WIN
55
+ cti-ate 0.2415 0.3060 -0.0645 [-0.1431, +0.0120] 22 TIE
56
+ cti-taa 0.0625 0.0625 +0.0000 [+0.0000, +0.0000] 16 TIE
57
+ secbench 0.8390 0.8341 +0.0049 [-0.0244, +0.0341] 205 TIE
58
+ secqa 0.9315 0.9452 -0.0137 [-0.0411, +0.0000] 73 TIE
59
+ cyber_composite 0.7389 0.7251 +0.0139 [-0.0056, +0.0328] TIE
60
+ note: F6 cyberbench_jpm and F7 vuln2 enter the composite bootstrap as independent normal noise drawn from their own 95% CIs (conservative: their per-item pairing is not used); F1..F5 are paired on shared item ids
61
+
62
+ === league verdict (W1/W2/W3 + guards + allowed sentence)
63
+ VERDICT ours=ab9b-v2 (test) tier=test boot=2000 seed=0 cyber_composite(ours)=0.7389
64
+ composite comparison (diff = ours - league, paired bootstrap, win iff 95% lower bound > 0):
65
+ vs ab9b-r1 (test) <=10B ours 0.7389 theirs 0.7251 diff +0.0139 [-0.0056, +0.0328] TIE
66
+ vs Cyber-Ornith-9B (test) <=10B ours 0.7389 theirs 0.6474 diff +0.0915 [+0.0716, +0.1120] WIN
67
+ vs Foundation-Sec-8B-Instruct (test) <=10B ours 0.7389 theirs 0.6662 diff +0.0727 [+0.0501, +0.0950] WIN
68
+ vs Primus-Merged-8B (test) <=10B ours 0.7389 theirs 0.6218 diff +0.1172 [+0.0950, +0.1388] WIN
69
+ vs Qwen3.5-9B base (test) <=10B ours 0.7389 theirs 0.7190 diff +0.0199 [+0.0016, +0.0383] WIN
70
+ vs SecGPT-7B (test) <=10B ours 0.7389 theirs 0.6243 diff +0.1147 [+0.0936, +0.1365] WIN
71
+ W1 size-fair (beat every <=10B league model: win vs ['ab9b-r1 (test)', 'Cyber-Ornith-9B (test)', 'Foundation-Sec-8B-Instruct (test)', 'Primus-Merged-8B (test)', 'Qwen3.5-9B base (test)', 'SecGPT-7B (test)']): DOES NOT HOLD
72
+ W3 absolute (>= every league model of any size: no loss and point difference >= 0): HOLDS [strict reading, a win against all: no]
73
+ W2 speed-fair (highest quality_per_second of the whole league): DOES NOT HOLD
74
+ vs ab9b-r1 (test) qps ours 0.6323 theirs 0.5914 diff +0.0409 [-0.0523, +0.1755] TIE
75
+ vs Cyber-Ornith-9B (test) qps ours 0.6323 theirs 0.8760 diff -0.2437 [-0.3646, +0.0024] TIE
76
+ vs Foundation-Sec-8B-Instruct (test) qps ours 0.6323 theirs 1.4483 diff -0.8159 [-1.0777, -0.2690] LOSS
77
+ vs Primus-Merged-8B (test) qps ours 0.6323 theirs 1.3864 diff -0.7540 [-1.1546, -0.1925] LOSS
78
+ vs Qwen3.5-9B base (test) qps ours 0.6323 theirs 0.6332 diff -0.0009 [-0.1228, +0.1660] TIE
79
+ vs SecGPT-7B (test) qps ours 0.6323 theirs 2.1020 diff -1.4697 [-1.7663, -1.1151] LOSS
80
+ guards (none more than 0.03 below ab9b-r1 (dev), dev tier): NOT DECIDABLE (missing values)
81
+ mmlu_sec not available
82
+ coding2 not available
83
+ agent2 not available
84
+ xstest_safe_compliance not available
85
+
86
+ CLAIM POLICY (GOAL.md section 1): 'wins any other cyber LLM' requires W1 AND W3 AND all guards.
87
+ NOT ALLOWED. State exactly: holds = ['W3']; does not hold = ['W1', 'W2']; not decidable = ['guards'] and nothing stronger.
88
+
89
+ === own harness b1 timing, ab9b-v2 (test) vs ab9b-r1 (median wall seconds, quality per second)
90
+ ab9b-v2 (test): - composite_dev: **0.756** [0.663, 0.837] - median_wall_s_dev: 1.169
91
+ ab9b-r1 (test): - composite_dev: **0.717** [0.624, 0.798] - median_wall_s_dev: 1.226
evidence/results/v2a_criteria.txt ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ C1 cyber_composite (test): 0.7389 vs 0.7251 diff +0.0139 [-0.0056, +0.0328] need diff >= +0.05 and lower bound > 0 -> FAIL
2
+ cybermetric 0.8966 vs 0.9000 diff -0.0034 [-0.0169, +0.0102] TIE
3
+ seceval 0.7554 vs 0.7492 diff +0.0061 [-0.0153, +0.0260] TIE
4
+ cti-mcq 0.7052 vs 0.7107 diff -0.0055 [-0.0317, +0.0220] TIE
5
+ cti-rcm 0.7629 vs 0.6079 diff +0.1550 [+0.1125, +0.2036] WIN
6
+ cti-vsp 0.8121 vs 0.7152 diff +0.0969 [+0.0730, +0.1216] WIN
7
+ cti-ate 0.2415 vs 0.3060 diff -0.0645 [-0.1431, +0.0120] TIE
8
+ cti-taa 0.0625 vs 0.0625 diff +0.0000 [+0.0000, +0.0000] TIE
9
+ secbench 0.8390 vs 0.8341 diff +0.0049 [-0.0244, +0.0341] TIE
10
+ secqa 0.9315 vs 0.9452 diff -0.0137 [-0.0411, +0.0000] TIE
11
+ C2 agentcy plain solved (test): 0.5281 vs 0.4831 diff +0.0449 [-0.0562, +0.1461] n=89 need diff >= +0.15 and lower bound > 0 -> FAIL
12
+ native tool-call exact (test): 0.8974 vs 0.9231 n=39 need not below r1 -> FAIL
13
+ agent2 guard (dev) delta +0.0500 need >= -0.03 -> PASS
14
+ (info) native agent loop solved (test): 0.4828 vs 0.1724 diff +0.3103 [+0.1954, +0.4253]
15
+ C3 median wall s per task (batch-1, own harness): 1.169 vs 1.226 ratio 0.953 need <= 1.10 -> PASS
16
+ agentcy plain completion tokens per solved task: 139.6 vs 170.5 need not above r1 -> PASS
17
+ MTP head tensors identical to the base: 15 identical, 0 different -> PASS
18
+ C4 guard mmlu_sec (dev): delta -0.0049 need >= -0.03 -> PASS
19
+ C4 guard coding2 (dev): delta -0.0167 need >= -0.03 -> PASS
20
+ C4 guard xstest_safe_compliance (dev): delta +0.0000 need >= -0.03 -> PASS
21
+ C5 xstest unsafe-refusal (dev): 0.1791 vs r1 0.1418 (lower 95% bound 0.0927) need >= lower bound -> PASS (+ data are defensive/benign only by construction)
22
+
23
+ SUMMARY ab9b-v2 (test) vs ab9b-r1 (test): hold ['C3', 'C4', 'C5']; fail ['C1', 'C2']; not decidable none => not accepted: state exactly which hold
evidence/results/v2a_pick.json ADDED
@@ -0,0 +1,336 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "v2a",
3
+ "ref": "ab9b-r1 (dev)",
4
+ "rule": "eligible = no guard (mmlu_sec, coding2, agent2, xstest_safe_compliance) more than 0.03 below the reference, and agentcy dev tool-call exact not more than 0.03 below it.\nScore = mean( dev composite proxy gain / 0.05 , agentcy dev plain solved gain / 0.15 ); the proxy is the mean over the dev families available for BOTH the candidate and the reference\n(F1-F5 and F7; F6 CyberBench has no dev tier). Highest score among the eligible wins; every eligible candidate within 0.02 of the best counts as tied and the LATEST checkpoint of the tied ones is picked.\nNo eligible candidate -> chosen = null (no automatic TEST scoring).",
5
+ "rows": [
6
+ {
7
+ "tag": "final",
8
+ "battery_tag": "v2a-final (dev)",
9
+ "gguf": "models/gguf/v2a-final-Q4_K_M.gguf",
10
+ "eligible": true,
11
+ "reasons": [],
12
+ "families_used": [
13
+ "ctibench",
14
+ "cybermetric",
15
+ "secbench",
16
+ "seceval",
17
+ "secqa",
18
+ "vuln2"
19
+ ],
20
+ "proxy": 0.7654,
21
+ "ref_proxy": 0.7532666666666668,
22
+ "families": {
23
+ "ctibench": [
24
+ 0.5739,
25
+ 0.5141
26
+ ],
27
+ "cybermetric": [
28
+ 0.885,
29
+ 0.9075
30
+ ],
31
+ "secbench": [
32
+ 0.7925,
33
+ 0.8
34
+ ],
35
+ "seceval": [
36
+ 0.75,
37
+ 0.74
38
+ ],
39
+ "secqa": [
40
+ 0.9927,
41
+ 0.9927
42
+ ],
43
+ "vuln2": [
44
+ 0.5983,
45
+ 0.5653
46
+ ]
47
+ },
48
+ "g1": 0.24266666666666437,
49
+ "guards": {
50
+ "mmlu_sec": -0.0049,
51
+ "coding2": 0.0042,
52
+ "agent2": 0.05,
53
+ "xstest_safe_compliance": 0.0
54
+ },
55
+ "agentcy": {
56
+ "plain_solved": {
57
+ "n": 60,
58
+ "a": 0.6167,
59
+ "b": 0.55,
60
+ "diff": 0.0667,
61
+ "ci95": [
62
+ -0.0333,
63
+ 0.1833
64
+ ],
65
+ "verdict": "TIE"
66
+ },
67
+ "native_solved": {
68
+ "n": 59,
69
+ "a": 0.6102,
70
+ "b": 0.2881,
71
+ "diff": 0.322,
72
+ "ci95": [
73
+ 0.1695,
74
+ 0.4576
75
+ ],
76
+ "verdict": "WIN"
77
+ },
78
+ "toolcall_exact": {
79
+ "n": 91,
80
+ "a": 0.956,
81
+ "b": 0.978,
82
+ "diff": -0.022,
83
+ "ci95": [
84
+ -0.0549,
85
+ 0.0
86
+ ],
87
+ "verdict": "TIE"
88
+ },
89
+ "plain_reserved": {
90
+ "n": 45,
91
+ "a": 0.5333,
92
+ "b": 0.4889,
93
+ "diff": 0.0444,
94
+ "ci95": [
95
+ -0.0667,
96
+ 0.1556
97
+ ],
98
+ "verdict": "TIE"
99
+ },
100
+ "plain_unseen": {
101
+ "n": 15,
102
+ "a": 0.8667,
103
+ "b": 0.7333,
104
+ "diff": 0.1333,
105
+ "ci95": [
106
+ -0.1333,
107
+ 0.4
108
+ ],
109
+ "verdict": "TIE"
110
+ }
111
+ },
112
+ "g2": 0.44466666666666665,
113
+ "score": 0.3436666666666655
114
+ },
115
+ {
116
+ "tag": "ck1000",
117
+ "battery_tag": "v2a-ck1000 (dev)",
118
+ "gguf": "models/gguf/v2a-ck1000-Q4_K_M.gguf",
119
+ "eligible": true,
120
+ "reasons": [],
121
+ "families_used": [
122
+ "ctibench",
123
+ "cybermetric",
124
+ "secbench",
125
+ "seceval",
126
+ "secqa",
127
+ "vuln2"
128
+ ],
129
+ "proxy": 0.7668499999999999,
130
+ "ref_proxy": 0.7532666666666668,
131
+ "families": {
132
+ "ctibench": [
133
+ 0.5753,
134
+ 0.5141
135
+ ],
136
+ "cybermetric": [
137
+ 0.8825,
138
+ 0.9075
139
+ ],
140
+ "secbench": [
141
+ 0.7925,
142
+ 0.8
143
+ ],
144
+ "seceval": [
145
+ 0.7575,
146
+ 0.74
147
+ ],
148
+ "secqa": [
149
+ 0.9927,
150
+ 0.9927
151
+ ],
152
+ "vuln2": [
153
+ 0.6006,
154
+ 0.5653
155
+ ]
156
+ },
157
+ "g1": 0.2716666666666634,
158
+ "guards": {
159
+ "mmlu_sec": -0.0147,
160
+ "coding2": -0.0042,
161
+ "agent2": 0.075,
162
+ "xstest_safe_compliance": 0.0
163
+ },
164
+ "agentcy": {
165
+ "plain_solved": {
166
+ "n": 60,
167
+ "a": 0.6167,
168
+ "b": 0.55,
169
+ "diff": 0.0667,
170
+ "ci95": [
171
+ -0.0333,
172
+ 0.1833
173
+ ],
174
+ "verdict": "TIE"
175
+ },
176
+ "native_solved": {
177
+ "n": 59,
178
+ "a": 0.5593,
179
+ "b": 0.2881,
180
+ "diff": 0.2712,
181
+ "ci95": [
182
+ 0.1356,
183
+ 0.4068
184
+ ],
185
+ "verdict": "WIN"
186
+ },
187
+ "toolcall_exact": {
188
+ "n": 91,
189
+ "a": 0.956,
190
+ "b": 0.978,
191
+ "diff": -0.022,
192
+ "ci95": [
193
+ -0.0549,
194
+ 0.0
195
+ ],
196
+ "verdict": "TIE"
197
+ },
198
+ "plain_reserved": {
199
+ "n": 45,
200
+ "a": 0.5333,
201
+ "b": 0.4889,
202
+ "diff": 0.0444,
203
+ "ci95": [
204
+ -0.0667,
205
+ 0.1556
206
+ ],
207
+ "verdict": "TIE"
208
+ },
209
+ "plain_unseen": {
210
+ "n": 15,
211
+ "a": 0.8667,
212
+ "b": 0.7333,
213
+ "diff": 0.1333,
214
+ "ci95": [
215
+ -0.1333,
216
+ 0.4
217
+ ],
218
+ "verdict": "TIE"
219
+ }
220
+ },
221
+ "g2": 0.44466666666666665,
222
+ "score": 0.358166666666665
223
+ },
224
+ {
225
+ "tag": "ck900",
226
+ "battery_tag": "v2a-ck900 (dev)",
227
+ "gguf": "models/gguf/ab9b-v2-Q4_K_M.gguf",
228
+ "eligible": true,
229
+ "reasons": [],
230
+ "families_used": [
231
+ "ctibench",
232
+ "cybermetric",
233
+ "secbench",
234
+ "seceval",
235
+ "secqa",
236
+ "vuln2"
237
+ ],
238
+ "proxy": 0.76485,
239
+ "ref_proxy": 0.7532666666666668,
240
+ "families": {
241
+ "ctibench": [
242
+ 0.5748,
243
+ 0.5141
244
+ ],
245
+ "cybermetric": [
246
+ 0.885,
247
+ 0.9075
248
+ ],
249
+ "secbench": [
250
+ 0.79,
251
+ 0.8
252
+ ],
253
+ "seceval": [
254
+ 0.735,
255
+ 0.74
256
+ ],
257
+ "secqa": [
258
+ 0.9927,
259
+ 0.9927
260
+ ],
261
+ "vuln2": [
262
+ 0.6116,
263
+ 0.5653
264
+ ]
265
+ },
266
+ "g1": 0.23166666666666558,
267
+ "guards": {
268
+ "mmlu_sec": -0.0049,
269
+ "coding2": -0.0167,
270
+ "agent2": 0.05,
271
+ "xstest_safe_compliance": 0.0
272
+ },
273
+ "agentcy": {
274
+ "plain_solved": {
275
+ "n": 60,
276
+ "a": 0.6333,
277
+ "b": 0.55,
278
+ "diff": 0.0833,
279
+ "ci95": [
280
+ -0.0333,
281
+ 0.2
282
+ ],
283
+ "verdict": "TIE"
284
+ },
285
+ "native_solved": {
286
+ "n": 59,
287
+ "a": 0.5932,
288
+ "b": 0.2881,
289
+ "diff": 0.3051,
290
+ "ci95": [
291
+ 0.1695,
292
+ 0.4407
293
+ ],
294
+ "verdict": "WIN"
295
+ },
296
+ "toolcall_exact": {
297
+ "n": 91,
298
+ "a": 0.956,
299
+ "b": 0.978,
300
+ "diff": -0.022,
301
+ "ci95": [
302
+ -0.0549,
303
+ 0.0
304
+ ],
305
+ "verdict": "TIE"
306
+ },
307
+ "plain_reserved": {
308
+ "n": 45,
309
+ "a": 0.5111,
310
+ "b": 0.4889,
311
+ "diff": 0.0222,
312
+ "ci95": [
313
+ -0.1111,
314
+ 0.1556
315
+ ],
316
+ "verdict": "TIE"
317
+ },
318
+ "plain_unseen": {
319
+ "n": 15,
320
+ "a": 1.0,
321
+ "b": 0.7333,
322
+ "diff": 0.2667,
323
+ "ci95": [
324
+ 0.0667,
325
+ 0.4667
326
+ ],
327
+ "verdict": "WIN"
328
+ }
329
+ },
330
+ "g2": 0.5553333333333333,
331
+ "score": 0.39349999999999946
332
+ }
333
+ ],
334
+ "chosen": "ck900",
335
+ "chosen_gguf": "models/gguf/ab9b-v2-Q4_K_M.gguf"
336
+ }
evidence/results/v2b_dev_summary.txt ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ candidates: final: ck700: ck600:
2
+ === v2b-final
3
+
4
+ cyber_composite (tier test): NOT AVAILABLE [not final]
5
+ F1 cybermetric -- incomplete
6
+ F2 seceval -- incomplete
7
+ F3 ctibench -- incomplete
8
+ F4 secbench -- incomplete
9
+ F5 secqa -- incomplete
10
+ F6 cyberbench_jpm -- missing
11
+ F7 vuln2 -- missing
12
+ guards (dev tier, never in the composite):
13
+ mmlu_sec 0.7696 ok vs ref 0.7892: -0.0196
14
+ coding2 0.7406 ok vs ref 0.7615: -0.0209
15
+ agent2 0.8500 ok vs ref 0.6750: +0.1750
16
+ xstest_safe_compliance 0.9947 ok vs ref 0.9947: +0.0000
17
+ --- agentcy DEV (cap 60) vs ab9b-r1, paired
18
+ paired comparison A=v2b-final (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
19
+ task / metric A B diff 95% CI of diff n verdict
20
+ agent-plain solved 0.6833 0.5500 +0.1333 [+0.0333, +0.2333] 60 WIN
21
+ agent-plain exact 0.6833 0.5500 +0.1333 [+0.0333, +0.2333] 60 WIN
22
+ agent-plain score 0.6833 0.5500 +0.1333 [+0.0333, +0.2333] 60 WIN
23
+ agent-plain solved[reserved] 0.5778 0.4889 +0.0889 [+0.0000, +0.2000] 45 TIE
24
+ agent-plain solved[unseen] 1.0000 0.7333 +0.2667 [+0.0667, +0.5333] 15 WIN
25
+ agent-native solved 0.6610 0.2881 +0.3729 [+0.2203, +0.5254] 59 WIN
26
+ agent-native exact 0.6610 0.2881 +0.3729 [+0.2203, +0.5254] 59 WIN
27
+ agent-native score 0.6610 0.2881 +0.3729 [+0.2203, +0.5254] 59 WIN
28
+ agent-native solved[reserved] 0.5682 0.2727 +0.2955 [+0.1136, +0.4773] 44 WIN
29
+ agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8667] 15 WIN
30
+ toolcall exact 0.8462 0.9780 -0.1319 [-0.2088, -0.0659] 91 LOSS
31
+ --- battery DEV vs v2a-ck900 (dev), paired
32
+ paired comparison A=v2b-final (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
33
+ task A B diff 95% CI of diff n_shared verdict
34
+ cybermetric 0.8800 0.8850 -0.0050 [-0.0225, +0.0125] 400 TIE
35
+ seceval 0.7525 0.7350 +0.0175 [+0.0000, +0.0400] 400 TIE
36
+ cti-mcq 0.6750 0.6775 -0.0025 [-0.0300, +0.0225] 400 TIE
37
+ cti-rcm 0.7500 0.7450 +0.0050 [-0.0150, +0.0250] 400 TIE
38
+ cti-vsp 0.8387 0.8100 +0.0287 [+0.0159, +0.0419] 400 WIN
39
+ cti-ate 0.3712 0.2296 +0.1416 [+0.0795, +0.2056] 38 WIN
40
+ cti-taa 0.2941 0.4118 -0.1176 [-0.2647, +0.0000] 34 TIE
41
+ secbench 0.7800 0.7900 -0.0100 [-0.0300, +0.0100] 400 TIE
42
+ secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
43
+ --- agentcy DEV vs v2a-ck900 (dev), paired
44
+ paired comparison A=v2b-final (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
45
+ task / metric A B diff 95% CI of diff n verdict
46
+ agent-plain solved 0.6833 0.6333 +0.0500 [+0.0000, +0.1167] 60 TIE
47
+ agent-plain exact 0.6833 0.6333 +0.0500 [+0.0000, +0.1167] 60 TIE
48
+ agent-plain score 0.6833 0.6333 +0.0500 [+0.0000, +0.1167] 60 TIE
49
+ agent-plain solved[reserved] 0.5778 0.5111 +0.0667 [+0.0000, +0.1333] 45 TIE
50
+ agent-plain solved[unseen] 1.0000 1.0000 +0.0000 [+0.0000, +0.0000] 15 TIE
51
+ agent-native solved 0.6500 0.5833 +0.0667 [-0.0167, +0.1667] 60 TIE
52
+ agent-native exact 0.6500 0.5833 +0.0667 [-0.0167, +0.1667] 60 TIE
53
+ agent-native score 0.6500 0.5833 +0.0667 [-0.0167, +0.1667] 60 TIE
54
+ agent-native solved[reserved] 0.5556 0.5111 +0.0444 [-0.0667, +0.1556] 45 TIE
55
+ agent-native solved[unseen] 0.9333 0.8000 +0.1333 [+0.0000, +0.3333] 15 TIE
56
+ toolcall exact 0.8462 0.9560 -0.1099 [-0.1868, -0.0440] 91 LOSS
57
+ === v2b-ck700
58
+
59
+ cyber_composite (tier test): NOT AVAILABLE [not final]
60
+ F1 cybermetric -- incomplete
61
+ F2 seceval -- incomplete
62
+ F3 ctibench -- incomplete
63
+ F4 secbench -- incomplete
64
+ F5 secqa -- incomplete
65
+ F6 cyberbench_jpm -- missing
66
+ F7 vuln2 -- missing
67
+ guards (dev tier, never in the composite):
68
+ mmlu_sec 0.7794 ok vs ref 0.7892: -0.0098
69
+ coding2 0.7573 ok vs ref 0.7615: -0.0042
70
+ agent2 0.8000 ok vs ref 0.6750: +0.1250
71
+ xstest_safe_compliance 0.9947 ok vs ref 0.9947: +0.0000
72
+ --- agentcy DEV (cap 60) vs ab9b-r1, paired
73
+ paired comparison A=v2b-ck700 (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
74
+ task / metric A B diff 95% CI of diff n verdict
75
+ agent-plain solved 0.6667 0.5500 +0.1167 [+0.0333, +0.2167] 60 WIN
76
+ agent-plain exact 0.6667 0.5500 +0.1167 [+0.0333, +0.2167] 60 WIN
77
+ agent-plain score 0.6667 0.5500 +0.1167 [+0.0333, +0.2167] 60 WIN
78
+ agent-plain solved[reserved] 0.5556 0.4889 +0.0667 [-0.0222, +0.1556] 45 TIE
79
+ agent-plain solved[unseen] 1.0000 0.7333 +0.2667 [+0.0667, +0.5333] 15 WIN
80
+ agent-native solved 0.5789 0.2982 +0.2807 [+0.1228, +0.4386] 57 WIN
81
+ agent-native exact 0.5789 0.2982 +0.2807 [+0.1228, +0.4386] 57 WIN
82
+ agent-native score 0.5789 0.2982 +0.2807 [+0.1228, +0.4386] 57 WIN
83
+ agent-native solved[reserved] 0.4524 0.2857 +0.1667 [-0.0238, +0.3571] 42 TIE
84
+ agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8000] 15 WIN
85
+ toolcall exact 0.8462 0.9780 -0.1319 [-0.2088, -0.0659] 91 LOSS
86
+ --- battery DEV vs v2a-ck900 (dev), paired
87
+ paired comparison A=v2b-ck700 (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
88
+ task A B diff 95% CI of diff n_shared verdict
89
+ cybermetric 0.8750 0.8850 -0.0100 [-0.0300, +0.0100] 400 TIE
90
+ seceval 0.7425 0.7350 +0.0075 [-0.0125, +0.0300] 400 TIE
91
+ cti-mcq 0.6600 0.6775 -0.0175 [-0.0450, +0.0100] 400 TIE
92
+ cti-rcm 0.7425 0.7450 -0.0025 [-0.0200, +0.0150] 400 TIE
93
+ cti-vsp 0.8428 0.8100 +0.0328 [+0.0206, +0.0456] 400 WIN
94
+ cti-ate 0.4006 0.2296 +0.1709 [+0.1030, +0.2361] 38 WIN
95
+ cti-taa 0.2941 0.4118 -0.1176 [-0.2647, +0.0000] 34 TIE
96
+ secbench 0.7775 0.7900 -0.0125 [-0.0300, +0.0050] 400 TIE
97
+ secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
98
+ --- agentcy DEV vs v2a-ck900 (dev), paired
99
+ paired comparison A=v2b-ck700 (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
100
+ task / metric A B diff 95% CI of diff n verdict
101
+ agent-plain solved 0.6667 0.6333 +0.0333 [-0.0333, +0.1000] 60 TIE
102
+ agent-plain exact 0.6667 0.6333 +0.0333 [-0.0333, +0.1000] 60 TIE
103
+ agent-plain score 0.6667 0.6333 +0.0333 [-0.0333, +0.1000] 60 TIE
104
+ agent-plain solved[reserved] 0.5556 0.5111 +0.0444 [-0.0444, +0.1333] 45 TIE
105
+ agent-plain solved[unseen] 1.0000 1.0000 +0.0000 [+0.0000, +0.0000] 15 TIE
106
+ agent-native solved 0.5690 0.5690 +0.0000 [-0.1034, +0.1034] 58 TIE
107
+ agent-native exact 0.5690 0.5690 +0.0000 [-0.1034, +0.1034] 58 TIE
108
+ agent-native score 0.5690 0.5690 +0.0000 [-0.1034, +0.1034] 58 TIE
109
+ agent-native solved[reserved] 0.4419 0.4884 -0.0465 [-0.1860, +0.0930] 43 TIE
110
+ agent-native solved[unseen] 0.9333 0.8000 +0.1333 [+0.0000, +0.3333] 15 TIE
111
+ toolcall exact 0.8462 0.9560 -0.1099 [-0.1758, -0.0549] 91 LOSS
112
+ === v2b-ck600
113
+
114
+ cyber_composite (tier test): NOT AVAILABLE [not final]
115
+ F1 cybermetric -- incomplete
116
+ F2 seceval -- incomplete
117
+ F3 ctibench -- incomplete
118
+ F4 secbench -- incomplete
119
+ F5 secqa -- incomplete
120
+ F6 cyberbench_jpm -- missing
121
+ F7 vuln2 -- missing
122
+ guards (dev tier, never in the composite):
123
+ mmlu_sec 0.7696 ok vs ref 0.7892: -0.0196
124
+ coding2 0.7531 ok vs ref 0.7615: -0.0084
125
+ agent2 0.8750 ok vs ref 0.6750: +0.2000
126
+ xstest_safe_compliance 1.0000 ok vs ref 0.9947: +0.0053
127
+ --- agentcy DEV (cap 60) vs ab9b-r1, paired
128
+ paired comparison A=v2b-ck600 (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
129
+ task / metric A B diff 95% CI of diff n verdict
130
+ agent-plain solved 0.6333 0.5500 +0.0833 [-0.0167, +0.2000] 60 TIE
131
+ agent-plain exact 0.6333 0.5500 +0.0833 [-0.0167, +0.2000] 60 TIE
132
+ agent-plain score 0.6333 0.5500 +0.0833 [-0.0167, +0.2000] 60 TIE
133
+ agent-plain solved[reserved] 0.5333 0.4889 +0.0444 [-0.0889, +0.1556] 45 TIE
134
+ agent-plain solved[unseen] 0.9333 0.7333 +0.2000 [+0.0000, +0.4000] 15 TIE
135
+ agent-native solved 0.5424 0.2881 +0.2542 [+0.1017, +0.4068] 59 WIN
136
+ agent-native exact 0.5424 0.2881 +0.2542 [+0.1017, +0.4068] 59 WIN
137
+ agent-native score 0.5424 0.2881 +0.2542 [+0.1017, +0.4068] 59 WIN
138
+ agent-native solved[reserved] 0.4091 0.2727 +0.1364 [-0.0227, +0.3182] 44 TIE
139
+ agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8000] 15 WIN
140
+ toolcall exact 0.8571 0.9780 -0.1209 [-0.1868, -0.0549] 91 LOSS
141
+ --- battery DEV vs v2a-ck900 (dev), paired
142
+ paired comparison A=v2b-ck600 (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
143
+ task A B diff 95% CI of diff n_shared verdict
144
+ cybermetric 0.8825 0.8850 -0.0025 [-0.0225, +0.0175] 400 TIE
145
+ seceval 0.7500 0.7350 +0.0150 [-0.0050, +0.0350] 400 TIE
146
+ cti-mcq 0.6725 0.6775 -0.0050 [-0.0300, +0.0200] 400 TIE
147
+ cti-rcm 0.7375 0.7450 -0.0075 [-0.0250, +0.0100] 400 TIE
148
+ cti-vsp 0.8325 0.8100 +0.0225 [+0.0094, +0.0359] 400 WIN
149
+ cti-ate 0.3564 0.2296 +0.1267 [+0.0680, +0.1822] 38 WIN
150
+ cti-taa 0.3235 0.4118 -0.0882 [-0.2059, +0.0294] 34 TIE
151
+ secbench 0.7925 0.7900 +0.0025 [-0.0150, +0.0200] 400 TIE
152
+ secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
153
+ --- agentcy DEV vs v2a-ck900 (dev), paired
154
+ paired comparison A=v2b-ck600 (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
155
+ task / metric A B diff 95% CI of diff n verdict
156
+ agent-plain solved 0.6333 0.6333 +0.0000 [-0.0833, +0.1000] 60 TIE
157
+ agent-plain exact 0.6333 0.6333 +0.0000 [-0.0833, +0.1000] 60 TIE
158
+ agent-plain score 0.6333 0.6333 +0.0000 [-0.0833, +0.1000] 60 TIE
159
+ agent-plain solved[reserved] 0.5333 0.5111 +0.0222 [-0.0889, +0.1333] 45 TIE
160
+ agent-plain solved[unseen] 0.9333 1.0000 -0.0667 [-0.2000, +0.0000] 15 TIE
161
+ agent-native solved 0.5333 0.5833 -0.0500 [-0.1667, +0.0500] 60 TIE
162
+ agent-native exact 0.5333 0.5833 -0.0500 [-0.1667, +0.0500] 60 TIE
163
+ agent-native score 0.5333 0.5833 -0.0500 [-0.1667, +0.0500] 60 TIE
164
+ agent-native solved[reserved] 0.4000 0.5111 -0.1111 [-0.2444, +0.0222] 45 TIE
165
+ agent-native solved[unseen] 0.9333 0.8000 +0.1333 [+0.0000, +0.3333] 15 TIE
166
+ toolcall exact 0.8571 0.9560 -0.0989 [-0.1648, -0.0440] 91 LOSS
evidence/results/v2b_pick.json ADDED
@@ -0,0 +1,342 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "v2b",
3
+ "ref": "ab9b-r1 (dev)",
4
+ "rule": "eligible = no guard (mmlu_sec, coding2, agent2, xstest_safe_compliance) more than 0.03 below the reference, and agentcy dev tool-call exact not more than 0.03 below it.\nScore = mean( dev composite proxy gain / 0.05 , agentcy dev plain solved gain / 0.15 ); the proxy is the mean over the dev families available for BOTH the candidate and the reference\n(F1-F5 and F7; F6 CyberBench has no dev tier). Highest score among the eligible wins; every eligible candidate within 0.02 of the best counts as tied and the LATEST checkpoint of the tied ones is picked.\nNo eligible candidate -> chosen = null (no automatic TEST scoring).",
5
+ "rows": [
6
+ {
7
+ "tag": "final",
8
+ "battery_tag": "v2b-final (dev)",
9
+ "gguf": "models/gguf/v2b-final-Q4_K_M.gguf",
10
+ "eligible": false,
11
+ "reasons": [
12
+ "toolcall exact -0.1319"
13
+ ],
14
+ "families_used": [
15
+ "ctibench",
16
+ "cybermetric",
17
+ "secbench",
18
+ "seceval",
19
+ "secqa",
20
+ "vuln2"
21
+ ],
22
+ "proxy": 0.7616,
23
+ "ref_proxy": 0.7532666666666668,
24
+ "families": {
25
+ "ctibench": [
26
+ 0.5858,
27
+ 0.5141
28
+ ],
29
+ "cybermetric": [
30
+ 0.88,
31
+ 0.9075
32
+ ],
33
+ "secbench": [
34
+ 0.78,
35
+ 0.8
36
+ ],
37
+ "seceval": [
38
+ 0.7525,
39
+ 0.74
40
+ ],
41
+ "secqa": [
42
+ 0.9927,
43
+ 0.9927
44
+ ],
45
+ "vuln2": [
46
+ 0.5786,
47
+ 0.5653
48
+ ]
49
+ },
50
+ "g1": 0.16666666666666607,
51
+ "guards": {
52
+ "mmlu_sec": -0.0196,
53
+ "coding2": -0.0209,
54
+ "agent2": 0.175,
55
+ "xstest_safe_compliance": 0.0
56
+ },
57
+ "agentcy": {
58
+ "plain_solved": {
59
+ "n": 60,
60
+ "a": 0.6833,
61
+ "b": 0.55,
62
+ "diff": 0.1333,
63
+ "ci95": [
64
+ 0.0333,
65
+ 0.2333
66
+ ],
67
+ "verdict": "WIN"
68
+ },
69
+ "native_solved": {
70
+ "n": 59,
71
+ "a": 0.661,
72
+ "b": 0.2881,
73
+ "diff": 0.3729,
74
+ "ci95": [
75
+ 0.2203,
76
+ 0.5254
77
+ ],
78
+ "verdict": "WIN"
79
+ },
80
+ "toolcall_exact": {
81
+ "n": 91,
82
+ "a": 0.8462,
83
+ "b": 0.978,
84
+ "diff": -0.1319,
85
+ "ci95": [
86
+ -0.2088,
87
+ -0.0659
88
+ ],
89
+ "verdict": "LOSS"
90
+ },
91
+ "plain_reserved": {
92
+ "n": 45,
93
+ "a": 0.5778,
94
+ "b": 0.4889,
95
+ "diff": 0.0889,
96
+ "ci95": [
97
+ 0.0,
98
+ 0.2
99
+ ],
100
+ "verdict": "TIE"
101
+ },
102
+ "plain_unseen": {
103
+ "n": 15,
104
+ "a": 1.0,
105
+ "b": 0.7333,
106
+ "diff": 0.2667,
107
+ "ci95": [
108
+ 0.0667,
109
+ 0.5333
110
+ ],
111
+ "verdict": "WIN"
112
+ }
113
+ },
114
+ "g2": 0.8886666666666667,
115
+ "score": 0.5276666666666664
116
+ },
117
+ {
118
+ "tag": "ck700",
119
+ "battery_tag": "v2b-ck700 (dev)",
120
+ "gguf": "models/gguf/v2b-ck700-Q4_K_M.gguf",
121
+ "eligible": false,
122
+ "reasons": [
123
+ "toolcall exact -0.1319"
124
+ ],
125
+ "families_used": [
126
+ "ctibench",
127
+ "cybermetric",
128
+ "secbench",
129
+ "seceval",
130
+ "secqa",
131
+ "vuln2"
132
+ ],
133
+ "proxy": 0.7582,
134
+ "ref_proxy": 0.7532666666666668,
135
+ "families": {
136
+ "ctibench": [
137
+ 0.588,
138
+ 0.5141
139
+ ],
140
+ "cybermetric": [
141
+ 0.875,
142
+ 0.9075
143
+ ],
144
+ "secbench": [
145
+ 0.7775,
146
+ 0.8
147
+ ],
148
+ "seceval": [
149
+ 0.7425,
150
+ 0.74
151
+ ],
152
+ "secqa": [
153
+ 0.9927,
154
+ 0.9927
155
+ ],
156
+ "vuln2": [
157
+ 0.5735,
158
+ 0.5653
159
+ ]
160
+ },
161
+ "g1": 0.09866666666666468,
162
+ "guards": {
163
+ "mmlu_sec": -0.0098,
164
+ "coding2": -0.0042,
165
+ "agent2": 0.125,
166
+ "xstest_safe_compliance": 0.0
167
+ },
168
+ "agentcy": {
169
+ "plain_solved": {
170
+ "n": 60,
171
+ "a": 0.6667,
172
+ "b": 0.55,
173
+ "diff": 0.1167,
174
+ "ci95": [
175
+ 0.0333,
176
+ 0.2167
177
+ ],
178
+ "verdict": "WIN"
179
+ },
180
+ "native_solved": {
181
+ "n": 57,
182
+ "a": 0.5789,
183
+ "b": 0.2982,
184
+ "diff": 0.2807,
185
+ "ci95": [
186
+ 0.1228,
187
+ 0.4386
188
+ ],
189
+ "verdict": "WIN"
190
+ },
191
+ "toolcall_exact": {
192
+ "n": 91,
193
+ "a": 0.8462,
194
+ "b": 0.978,
195
+ "diff": -0.1319,
196
+ "ci95": [
197
+ -0.2088,
198
+ -0.0659
199
+ ],
200
+ "verdict": "LOSS"
201
+ },
202
+ "plain_reserved": {
203
+ "n": 45,
204
+ "a": 0.5556,
205
+ "b": 0.4889,
206
+ "diff": 0.0667,
207
+ "ci95": [
208
+ -0.0222,
209
+ 0.1556
210
+ ],
211
+ "verdict": "TIE"
212
+ },
213
+ "plain_unseen": {
214
+ "n": 15,
215
+ "a": 1.0,
216
+ "b": 0.7333,
217
+ "diff": 0.2667,
218
+ "ci95": [
219
+ 0.0667,
220
+ 0.5333
221
+ ],
222
+ "verdict": "WIN"
223
+ }
224
+ },
225
+ "g2": 0.778,
226
+ "score": 0.43833333333333235
227
+ },
228
+ {
229
+ "tag": "ck600",
230
+ "battery_tag": "v2b-ck600 (dev)",
231
+ "gguf": "models/gguf/v2b-ck600-Q4_K_M.gguf",
232
+ "eligible": false,
233
+ "reasons": [
234
+ "toolcall exact -0.1209"
235
+ ],
236
+ "families_used": [
237
+ "ctibench",
238
+ "cybermetric",
239
+ "secbench",
240
+ "seceval",
241
+ "secqa",
242
+ "vuln2"
243
+ ],
244
+ "proxy": 0.7655666666666666,
245
+ "ref_proxy": 0.7532666666666668,
246
+ "families": {
247
+ "ctibench": [
248
+ 0.5845,
249
+ 0.5141
250
+ ],
251
+ "cybermetric": [
252
+ 0.8825,
253
+ 0.9075
254
+ ],
255
+ "secbench": [
256
+ 0.7925,
257
+ 0.8
258
+ ],
259
+ "seceval": [
260
+ 0.75,
261
+ 0.74
262
+ ],
263
+ "secqa": [
264
+ 0.9927,
265
+ 0.9927
266
+ ],
267
+ "vuln2": [
268
+ 0.5912,
269
+ 0.5653
270
+ ]
271
+ },
272
+ "g1": 0.24599999999999733,
273
+ "guards": {
274
+ "mmlu_sec": -0.0196,
275
+ "coding2": -0.0084,
276
+ "agent2": 0.2,
277
+ "xstest_safe_compliance": 0.0053
278
+ },
279
+ "agentcy": {
280
+ "plain_solved": {
281
+ "n": 60,
282
+ "a": 0.6333,
283
+ "b": 0.55,
284
+ "diff": 0.0833,
285
+ "ci95": [
286
+ -0.0167,
287
+ 0.2
288
+ ],
289
+ "verdict": "TIE"
290
+ },
291
+ "native_solved": {
292
+ "n": 59,
293
+ "a": 0.5424,
294
+ "b": 0.2881,
295
+ "diff": 0.2542,
296
+ "ci95": [
297
+ 0.1017,
298
+ 0.4068
299
+ ],
300
+ "verdict": "WIN"
301
+ },
302
+ "toolcall_exact": {
303
+ "n": 91,
304
+ "a": 0.8571,
305
+ "b": 0.978,
306
+ "diff": -0.1209,
307
+ "ci95": [
308
+ -0.1868,
309
+ -0.0549
310
+ ],
311
+ "verdict": "LOSS"
312
+ },
313
+ "plain_reserved": {
314
+ "n": 45,
315
+ "a": 0.5333,
316
+ "b": 0.4889,
317
+ "diff": 0.0444,
318
+ "ci95": [
319
+ -0.0889,
320
+ 0.1556
321
+ ],
322
+ "verdict": "TIE"
323
+ },
324
+ "plain_unseen": {
325
+ "n": 15,
326
+ "a": 0.9333,
327
+ "b": 0.7333,
328
+ "diff": 0.2,
329
+ "ci95": [
330
+ 0.0,
331
+ 0.4
332
+ ],
333
+ "verdict": "TIE"
334
+ }
335
+ },
336
+ "g2": 0.5553333333333333,
337
+ "score": 0.40066666666666534
338
+ }
339
+ ],
340
+ "chosen": null,
341
+ "chosen_gguf": null
342
+ }
evidence/results/v2c_dev_summary.txt ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ candidates: final: ck200: ck100:
2
+ === v2c-final
3
+
4
+ cyber_composite (tier test): NOT AVAILABLE [not final]
5
+ F1 cybermetric -- incomplete
6
+ F2 seceval -- incomplete
7
+ F3 ctibench -- incomplete
8
+ F4 secbench -- incomplete
9
+ F5 secqa -- incomplete
10
+ F6 cyberbench_jpm -- missing
11
+ F7 vuln2 -- missing
12
+ guards (dev tier, never in the composite):
13
+ mmlu_sec 0.7941 ok vs ref 0.7892: +0.0049
14
+ coding2 0.7657 ok vs ref 0.7615: +0.0042
15
+ agent2 0.8250 ok vs ref 0.6750: +0.1500
16
+ xstest_safe_compliance 0.9840 ok vs ref 0.9947: -0.0107
17
+ --- agentcy DEV (cap 60) vs ab9b-r1, paired
18
+ paired comparison A=v2c-final (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
19
+ task / metric A B diff 95% CI of diff n verdict
20
+ agent-plain solved 0.6333 0.5500 +0.0833 [+0.0000, +0.1833] 60 TIE
21
+ agent-plain exact 0.6333 0.5500 +0.0833 [+0.0000, +0.1833] 60 TIE
22
+ agent-plain score 0.6333 0.5500 +0.0833 [+0.0000, +0.1833] 60 TIE
23
+ agent-plain solved[reserved] 0.5333 0.4889 +0.0444 [-0.0444, +0.1556] 45 TIE
24
+ agent-plain solved[unseen] 0.9333 0.7333 +0.2000 [+0.0000, +0.4000] 15 TIE
25
+ agent-native solved 0.6271 0.2881 +0.3390 [+0.1864, +0.4915] 59 WIN
26
+ agent-native exact 0.6271 0.2881 +0.3390 [+0.1864, +0.4915] 59 WIN
27
+ agent-native score 0.6271 0.2881 +0.3390 [+0.1864, +0.4915] 59 WIN
28
+ agent-native solved[reserved] 0.5227 0.2727 +0.2500 [+0.0682, +0.4318] 44 WIN
29
+ agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8667] 15 WIN
30
+ toolcall exact 0.9780 0.9780 +0.0000 [-0.0330, +0.0330] 91 TIE
31
+ --- battery DEV vs v2b-final (dev), paired
32
+ paired comparison A=v2c-final (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
33
+ task A B diff 95% CI of diff n_shared verdict
34
+ cybermetric 0.8825 0.8800 +0.0025 [-0.0125, +0.0175] 400 TIE
35
+ seceval 0.7450 0.7525 -0.0075 [-0.0275, +0.0125] 400 TIE
36
+ cti-mcq 0.6750 0.6750 +0.0000 [-0.0250, +0.0250] 400 TIE
37
+ cti-rcm 0.7350 0.7500 -0.0150 [-0.0350, +0.0050] 400 TIE
38
+ cti-vsp 0.7644 0.8387 -0.0744 [-0.0900, -0.0588] 400 LOSS
39
+ cti-ate 0.3981 0.3712 +0.0269 [-0.0152, +0.0710] 38 TIE
40
+ cti-taa 0.3235 0.2941 +0.0294 [-0.0882, +0.1471] 34 TIE
41
+ secbench 0.7800 0.7800 +0.0000 [-0.0200, +0.0200] 400 TIE
42
+ secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
43
+ --- agentcy DEV vs v2b-final (dev), paired
44
+ paired comparison A=v2c-final (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
45
+ task / metric A B diff 95% CI of diff n verdict
46
+ agent-plain solved 0.6333 0.6833 -0.0500 [-0.1167, +0.0000] 60 TIE
47
+ agent-plain exact 0.6333 0.6833 -0.0500 [-0.1167, +0.0000] 60 TIE
48
+ agent-plain score 0.6333 0.6833 -0.0500 [-0.1167, +0.0000] 60 TIE
49
+ agent-plain solved[reserved] 0.5333 0.5778 -0.0444 [-0.1111, +0.0000] 45 TIE
50
+ agent-plain solved[unseen] 0.9333 1.0000 -0.0667 [-0.2000, +0.0000] 15 TIE
51
+ agent-native solved 0.6167 0.6500 -0.0333 [-0.1167, +0.0500] 60 TIE
52
+ agent-native exact 0.6167 0.6500 -0.0333 [-0.1167, +0.0500] 60 TIE
53
+ agent-native score 0.6167 0.6500 -0.0333 [-0.1167, +0.0500] 60 TIE
54
+ agent-native solved[reserved] 0.5111 0.5556 -0.0444 [-0.1556, +0.0444] 45 TIE
55
+ agent-native solved[unseen] 0.9333 0.9333 +0.0000 [+0.0000, +0.0000] 15 TIE
56
+ toolcall exact 0.9780 0.8462 +0.1319 [+0.0659, +0.2088] 91 WIN
57
+ === v2c-ck200
58
+
59
+ cyber_composite (tier test): NOT AVAILABLE [not final]
60
+ F1 cybermetric -- incomplete
61
+ F2 seceval -- incomplete
62
+ F3 ctibench -- incomplete
63
+ F4 secbench -- incomplete
64
+ F5 secqa -- incomplete
65
+ F6 cyberbench_jpm -- missing
66
+ F7 vuln2 -- missing
67
+ guards (dev tier, never in the composite):
68
+ mmlu_sec 0.7941 ok vs ref 0.7892: +0.0049
69
+ coding2 0.7657 ok vs ref 0.7615: +0.0042
70
+ agent2 0.7750 ok vs ref 0.6750: +0.1000
71
+ xstest_safe_compliance 0.9894 ok vs ref 0.9947: -0.0053
72
+ --- agentcy DEV (cap 60) vs ab9b-r1, paired
73
+ paired comparison A=v2c-ck200 (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
74
+ task / metric A B diff 95% CI of diff n verdict
75
+ agent-plain solved 0.6167 0.5500 +0.0667 [-0.0333, +0.1667] 60 TIE
76
+ agent-plain exact 0.6167 0.5500 +0.0667 [-0.0333, +0.1667] 60 TIE
77
+ agent-plain score 0.6167 0.5500 +0.0667 [-0.0333, +0.1667] 60 TIE
78
+ agent-plain solved[reserved] 0.5111 0.4889 +0.0222 [-0.0889, +0.1333] 45 TIE
79
+ agent-plain solved[unseen] 0.9333 0.7333 +0.2000 [+0.0000, +0.4000] 15 TIE
80
+ agent-native solved 0.6102 0.2881 +0.3220 [+0.1695, +0.4746] 59 WIN
81
+ agent-native exact 0.6102 0.2881 +0.3220 [+0.1695, +0.4746] 59 WIN
82
+ agent-native score 0.6102 0.2881 +0.3220 [+0.1695, +0.4746] 59 WIN
83
+ agent-native solved[reserved] 0.5000 0.2727 +0.2273 [+0.0682, +0.3864] 44 WIN
84
+ agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8667] 15 WIN
85
+ toolcall exact 0.9670 0.9780 -0.0110 [-0.0440, +0.0220] 91 TIE
86
+ --- battery DEV vs v2b-final (dev), paired
87
+ paired comparison A=v2c-ck200 (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
88
+ task A B diff 95% CI of diff n_shared verdict
89
+ cybermetric 0.8875 0.8800 +0.0075 [-0.0100, +0.0250] 400 TIE
90
+ seceval 0.7475 0.7525 -0.0050 [-0.0275, +0.0175] 400 TIE
91
+ cti-mcq 0.6800 0.6750 +0.0050 [-0.0200, +0.0300] 400 TIE
92
+ cti-rcm 0.7375 0.7500 -0.0125 [-0.0350, +0.0100] 400 TIE
93
+ cti-vsp 0.7734 0.8387 -0.0653 [-0.0803, -0.0503] 400 LOSS
94
+ cti-ate 0.3938 0.3712 +0.0226 [-0.0229, +0.0646] 38 TIE
95
+ cti-taa 0.3529 0.2941 +0.0588 [-0.0882, +0.2059] 34 TIE
96
+ secbench 0.7825 0.7800 +0.0025 [-0.0175, +0.0225] 400 TIE
97
+ secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
98
+ --- agentcy DEV vs v2b-final (dev), paired
99
+ paired comparison A=v2c-ck200 (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
100
+ task / metric A B diff 95% CI of diff n verdict
101
+ agent-plain solved 0.6167 0.6833 -0.0667 [-0.1333, -0.0167] 60 LOSS
102
+ agent-plain exact 0.6167 0.6833 -0.0667 [-0.1333, -0.0167] 60 LOSS
103
+ agent-plain score 0.6167 0.6833 -0.0667 [-0.1333, -0.0167] 60 LOSS
104
+ agent-plain solved[reserved] 0.5111 0.5778 -0.0667 [-0.1333, +0.0000] 45 TIE
105
+ agent-plain solved[unseen] 0.9333 1.0000 -0.0667 [-0.2000, +0.0000] 15 TIE
106
+ agent-native solved 0.6000 0.6500 -0.0500 [-0.1167, +0.0000] 60 TIE
107
+ agent-native exact 0.6000 0.6500 -0.0500 [-0.1167, +0.0000] 60 TIE
108
+ agent-native score 0.6000 0.6500 -0.0500 [-0.1167, +0.0000] 60 TIE
109
+ agent-native solved[reserved] 0.4889 0.5556 -0.0667 [-0.1556, +0.0000] 45 TIE
110
+ agent-native solved[unseen] 0.9333 0.9333 +0.0000 [+0.0000, +0.0000] 15 TIE
111
+ toolcall exact 0.9670 0.8462 +0.1209 [+0.0440, +0.1978] 91 WIN
112
+ === v2c-ck100
113
+
114
+ cyber_composite (tier test): NOT AVAILABLE [not final]
115
+ F1 cybermetric -- incomplete
116
+ F2 seceval -- incomplete
117
+ F3 ctibench -- incomplete
118
+ F4 secbench -- incomplete
119
+ F5 secqa -- incomplete
120
+ F6 cyberbench_jpm -- missing
121
+ F7 vuln2 -- missing
122
+ guards (dev tier, never in the composite):
123
+ mmlu_sec 0.8039 ok vs ref 0.7892: +0.0147
124
+ coding2 0.7573 ok vs ref 0.7615: -0.0042
125
+ agent2 0.7000 ok vs ref 0.6750: +0.0250
126
+ xstest_safe_compliance 0.9840 ok vs ref 0.9947: -0.0107
127
+ --- agentcy DEV (cap 60) vs ab9b-r1, paired
128
+ paired comparison A=v2c-ck100 (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
129
+ task / metric A B diff 95% CI of diff n verdict
130
+ agent-plain solved 0.6000 0.5500 +0.0500 [-0.0500, +0.1500] 60 TIE
131
+ agent-plain exact 0.6000 0.5500 +0.0500 [-0.0500, +0.1500] 60 TIE
132
+ agent-plain score 0.6000 0.5500 +0.0500 [-0.0500, +0.1500] 60 TIE
133
+ agent-plain solved[reserved] 0.4889 0.4889 +0.0000 [-0.1111, +0.1111] 45 TIE
134
+ agent-plain solved[unseen] 0.9333 0.7333 +0.2000 [+0.0000, +0.4000] 15 TIE
135
+ agent-native solved 0.5763 0.2881 +0.2881 [+0.1525, +0.4237] 59 WIN
136
+ agent-native exact 0.5763 0.2881 +0.2881 [+0.1525, +0.4237] 59 WIN
137
+ agent-native score 0.5763 0.2881 +0.2881 [+0.1525, +0.4237] 59 WIN
138
+ agent-native solved[reserved] 0.4545 0.2727 +0.1818 [+0.0227, +0.3409] 44 WIN
139
+ agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8000] 15 WIN
140
+ toolcall exact 0.9780 0.9780 +0.0000 [-0.0330, +0.0330] 91 TIE
141
+ --- battery DEV vs v2b-final (dev), paired
142
+ paired comparison A=v2c-ck100 (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
143
+ task A B diff 95% CI of diff n_shared verdict
144
+ cybermetric 0.8950 0.8800 +0.0150 [-0.0025, +0.0325] 400 TIE
145
+ seceval 0.7400 0.7525 -0.0125 [-0.0325, +0.0075] 400 TIE
146
+ cti-mcq 0.6600 0.6750 -0.0150 [-0.0400, +0.0100] 400 TIE
147
+ cti-rcm 0.6950 0.7500 -0.0550 [-0.0825, -0.0300] 400 LOSS
148
+ cti-vsp 0.8456 0.8387 +0.0069 [-0.0038, +0.0178] 400 TIE
149
+ cti-ate 0.3086 0.3712 -0.0627 [-0.1032, -0.0237] 38 LOSS
150
+ cti-taa 0.2353 0.2941 -0.0588 [-0.1765, +0.0588] 34 TIE
151
+ secbench 0.7825 0.7800 +0.0025 [-0.0125, +0.0175] 400 TIE
152
+ secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
153
+ --- agentcy DEV vs v2b-final (dev), paired
154
+ paired comparison A=v2c-ck100 (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
155
+ task / metric A B diff 95% CI of diff n verdict
156
+ agent-plain solved 0.6000 0.6833 -0.0833 [-0.1500, -0.0167] 60 LOSS
157
+ agent-plain exact 0.6000 0.6833 -0.0833 [-0.1500, -0.0167] 60 LOSS
158
+ agent-plain score 0.6000 0.6833 -0.0833 [-0.1500, -0.0167] 60 LOSS
159
+ agent-plain solved[reserved] 0.4889 0.5778 -0.0889 [-0.1778, -0.0222] 45 LOSS
160
+ agent-plain solved[unseen] 0.9333 1.0000 -0.0667 [-0.2000, +0.0000] 15 TIE
161
+ agent-native solved 0.5667 0.6500 -0.0833 [-0.1500, -0.0167] 60 LOSS
162
+ agent-native exact 0.5667 0.6500 -0.0833 [-0.1500, -0.0167] 60 LOSS
163
+ agent-native score 0.5667 0.6500 -0.0833 [-0.1500, -0.0167] 60 LOSS
164
+ agent-native solved[reserved] 0.4444 0.5556 -0.1111 [-0.2222, -0.0222] 45 LOSS
165
+ agent-native solved[unseen] 0.9333 0.9333 +0.0000 [+0.0000, +0.0000] 15 TIE
166
+ toolcall exact 0.9780 0.8462 +0.1319 [+0.0659, +0.2088] 91 WIN
evidence/results/v2c_pick.json ADDED
@@ -0,0 +1,336 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "v2c",
3
+ "ref": "ab9b-r1 (dev)",
4
+ "rule": "eligible = no guard (mmlu_sec, coding2, agent2, xstest_safe_compliance) more than 0.03 below the reference, and agentcy dev tool-call exact not more than 0.03 below it.\nScore = mean( dev composite proxy gain / 0.05 , agentcy dev plain solved gain / 0.15 ); the proxy is the mean over the dev families available for BOTH the candidate and the reference\n(F1-F5 and F7; F6 CyberBench has no dev tier). Highest score among the eligible wins; every eligible candidate within 0.02 of the best counts as tied and the LATEST checkpoint of the tied ones is picked.\nNo eligible candidate -> chosen = null (no automatic TEST scoring).",
5
+ "rows": [
6
+ {
7
+ "tag": "final",
8
+ "battery_tag": "v2c-final (dev)",
9
+ "gguf": "models/gguf/v2c-final-Q4_K_M.gguf",
10
+ "eligible": true,
11
+ "reasons": [],
12
+ "families_used": [
13
+ "ctibench",
14
+ "cybermetric",
15
+ "secbench",
16
+ "seceval",
17
+ "secqa",
18
+ "vuln2"
19
+ ],
20
+ "proxy": 0.7581000000000001,
21
+ "ref_proxy": 0.7532666666666668,
22
+ "families": {
23
+ "ctibench": [
24
+ 0.5792,
25
+ 0.5141
26
+ ],
27
+ "cybermetric": [
28
+ 0.8825,
29
+ 0.9075
30
+ ],
31
+ "secbench": [
32
+ 0.78,
33
+ 0.8
34
+ ],
35
+ "seceval": [
36
+ 0.745,
37
+ 0.74
38
+ ],
39
+ "secqa": [
40
+ 0.9927,
41
+ 0.9927
42
+ ],
43
+ "vuln2": [
44
+ 0.5692,
45
+ 0.5653
46
+ ]
47
+ },
48
+ "g1": 0.09666666666666712,
49
+ "guards": {
50
+ "mmlu_sec": 0.0049,
51
+ "coding2": 0.0042,
52
+ "agent2": 0.15,
53
+ "xstest_safe_compliance": -0.0107
54
+ },
55
+ "agentcy": {
56
+ "plain_solved": {
57
+ "n": 60,
58
+ "a": 0.6333,
59
+ "b": 0.55,
60
+ "diff": 0.0833,
61
+ "ci95": [
62
+ 0.0,
63
+ 0.1833
64
+ ],
65
+ "verdict": "TIE"
66
+ },
67
+ "native_solved": {
68
+ "n": 59,
69
+ "a": 0.6271,
70
+ "b": 0.2881,
71
+ "diff": 0.339,
72
+ "ci95": [
73
+ 0.1864,
74
+ 0.4915
75
+ ],
76
+ "verdict": "WIN"
77
+ },
78
+ "toolcall_exact": {
79
+ "n": 91,
80
+ "a": 0.978,
81
+ "b": 0.978,
82
+ "diff": 0.0,
83
+ "ci95": [
84
+ -0.033,
85
+ 0.033
86
+ ],
87
+ "verdict": "TIE"
88
+ },
89
+ "plain_reserved": {
90
+ "n": 45,
91
+ "a": 0.5333,
92
+ "b": 0.4889,
93
+ "diff": 0.0444,
94
+ "ci95": [
95
+ -0.0444,
96
+ 0.1556
97
+ ],
98
+ "verdict": "TIE"
99
+ },
100
+ "plain_unseen": {
101
+ "n": 15,
102
+ "a": 0.9333,
103
+ "b": 0.7333,
104
+ "diff": 0.2,
105
+ "ci95": [
106
+ 0.0,
107
+ 0.4
108
+ ],
109
+ "verdict": "TIE"
110
+ }
111
+ },
112
+ "g2": 0.5553333333333333,
113
+ "score": 0.32600000000000023
114
+ },
115
+ {
116
+ "tag": "ck200",
117
+ "battery_tag": "v2c-ck200 (dev)",
118
+ "gguf": "models/gguf/v2c-ck200-Q4_K_M.gguf",
119
+ "eligible": true,
120
+ "reasons": [],
121
+ "families_used": [
122
+ "ctibench",
123
+ "cybermetric",
124
+ "secbench",
125
+ "seceval",
126
+ "secqa",
127
+ "vuln2"
128
+ ],
129
+ "proxy": 0.7624,
130
+ "ref_proxy": 0.7532666666666668,
131
+ "families": {
132
+ "ctibench": [
133
+ 0.5875,
134
+ 0.5141
135
+ ],
136
+ "cybermetric": [
137
+ 0.8875,
138
+ 0.9075
139
+ ],
140
+ "secbench": [
141
+ 0.7825,
142
+ 0.8
143
+ ],
144
+ "seceval": [
145
+ 0.7475,
146
+ 0.74
147
+ ],
148
+ "secqa": [
149
+ 0.9927,
150
+ 0.9927
151
+ ],
152
+ "vuln2": [
153
+ 0.5767,
154
+ 0.5653
155
+ ]
156
+ },
157
+ "g1": 0.1826666666666643,
158
+ "guards": {
159
+ "mmlu_sec": 0.0049,
160
+ "coding2": 0.0042,
161
+ "agent2": 0.1,
162
+ "xstest_safe_compliance": -0.0053
163
+ },
164
+ "agentcy": {
165
+ "plain_solved": {
166
+ "n": 60,
167
+ "a": 0.6167,
168
+ "b": 0.55,
169
+ "diff": 0.0667,
170
+ "ci95": [
171
+ -0.0333,
172
+ 0.1667
173
+ ],
174
+ "verdict": "TIE"
175
+ },
176
+ "native_solved": {
177
+ "n": 59,
178
+ "a": 0.6102,
179
+ "b": 0.2881,
180
+ "diff": 0.322,
181
+ "ci95": [
182
+ 0.1695,
183
+ 0.4746
184
+ ],
185
+ "verdict": "WIN"
186
+ },
187
+ "toolcall_exact": {
188
+ "n": 91,
189
+ "a": 0.967,
190
+ "b": 0.978,
191
+ "diff": -0.011,
192
+ "ci95": [
193
+ -0.044,
194
+ 0.022
195
+ ],
196
+ "verdict": "TIE"
197
+ },
198
+ "plain_reserved": {
199
+ "n": 45,
200
+ "a": 0.5111,
201
+ "b": 0.4889,
202
+ "diff": 0.0222,
203
+ "ci95": [
204
+ -0.0889,
205
+ 0.1333
206
+ ],
207
+ "verdict": "TIE"
208
+ },
209
+ "plain_unseen": {
210
+ "n": 15,
211
+ "a": 0.9333,
212
+ "b": 0.7333,
213
+ "diff": 0.2,
214
+ "ci95": [
215
+ 0.0,
216
+ 0.4
217
+ ],
218
+ "verdict": "TIE"
219
+ }
220
+ },
221
+ "g2": 0.44466666666666665,
222
+ "score": 0.3136666666666655
223
+ },
224
+ {
225
+ "tag": "ck100",
226
+ "battery_tag": "v2c-ck100 (dev)",
227
+ "gguf": "models/gguf/v2c-ck100-Q4_K_M.gguf",
228
+ "eligible": true,
229
+ "reasons": [],
230
+ "families_used": [
231
+ "ctibench",
232
+ "cybermetric",
233
+ "secbench",
234
+ "seceval",
235
+ "secqa",
236
+ "vuln2"
237
+ ],
238
+ "proxy": 0.7606166666666666,
239
+ "ref_proxy": 0.7532666666666668,
240
+ "families": {
241
+ "ctibench": [
242
+ 0.5489,
243
+ 0.5141
244
+ ],
245
+ "cybermetric": [
246
+ 0.895,
247
+ 0.9075
248
+ ],
249
+ "secbench": [
250
+ 0.7825,
251
+ 0.8
252
+ ],
253
+ "seceval": [
254
+ 0.74,
255
+ 0.74
256
+ ],
257
+ "secqa": [
258
+ 0.9927,
259
+ 0.9927
260
+ ],
261
+ "vuln2": [
262
+ 0.6046,
263
+ 0.5653
264
+ ]
265
+ },
266
+ "g1": 0.14699999999999713,
267
+ "guards": {
268
+ "mmlu_sec": 0.0147,
269
+ "coding2": -0.0042,
270
+ "agent2": 0.025,
271
+ "xstest_safe_compliance": -0.0107
272
+ },
273
+ "agentcy": {
274
+ "plain_solved": {
275
+ "n": 60,
276
+ "a": 0.6,
277
+ "b": 0.55,
278
+ "diff": 0.05,
279
+ "ci95": [
280
+ -0.05,
281
+ 0.15
282
+ ],
283
+ "verdict": "TIE"
284
+ },
285
+ "native_solved": {
286
+ "n": 59,
287
+ "a": 0.5763,
288
+ "b": 0.2881,
289
+ "diff": 0.2881,
290
+ "ci95": [
291
+ 0.1525,
292
+ 0.4237
293
+ ],
294
+ "verdict": "WIN"
295
+ },
296
+ "toolcall_exact": {
297
+ "n": 91,
298
+ "a": 0.978,
299
+ "b": 0.978,
300
+ "diff": 0.0,
301
+ "ci95": [
302
+ -0.033,
303
+ 0.033
304
+ ],
305
+ "verdict": "TIE"
306
+ },
307
+ "plain_reserved": {
308
+ "n": 45,
309
+ "a": 0.4889,
310
+ "b": 0.4889,
311
+ "diff": 0.0,
312
+ "ci95": [
313
+ -0.1111,
314
+ 0.1111
315
+ ],
316
+ "verdict": "TIE"
317
+ },
318
+ "plain_unseen": {
319
+ "n": 15,
320
+ "a": 0.9333,
321
+ "b": 0.7333,
322
+ "diff": 0.2,
323
+ "ci95": [
324
+ 0.0,
325
+ 0.4
326
+ ],
327
+ "verdict": "TIE"
328
+ }
329
+ },
330
+ "g2": 0.33333333333333337,
331
+ "score": 0.24016666666666525
332
+ }
333
+ ],
334
+ "chosen": "final",
335
+ "chosen_gguf": "models/gguf/v2c-final-Q4_K_M.gguf"
336
+ }