Text Generation
GGUF
Safetensors
English
cybersecurity
cve
cwe
cvss
tool-use
agent
lora
qwen3
conversational
Instructions to use dorb/ab9b-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use dorb/ab9b-v2 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf dorb/ab9b-v2:Q4_K_M # Run inference directly in the terminal: llama cli -hf dorb/ab9b-v2:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf dorb/ab9b-v2:Q4_K_M # Run inference directly in the terminal: llama cli -hf dorb/ab9b-v2:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf dorb/ab9b-v2:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf dorb/ab9b-v2:Q4_K_M
Use Docker
docker model run hf.co/dorb/ab9b-v2:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use dorb/ab9b-v2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "dorb/ab9b-v2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dorb/ab9b-v2", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/dorb/ab9b-v2:Q4_K_M
- Ollama
How to use dorb/ab9b-v2 with Ollama:
ollama run hf.co/dorb/ab9b-v2:Q4_K_M
- Unsloth Desktop
- Pi
How to use dorb/ab9b-v2 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf dorb/ab9b-v2:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "dorb/ab9b-v2:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use dorb/ab9b-v2 with Docker Model Runner:
docker model run hf.co/dorb/ab9b-v2:Q4_K_M
- Lemonade
How to use dorb/ab9b-v2 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull dorb/ab9b-v2:Q4_K_M
Run and chat with the model
lemonade run user.ab9b-v2-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use dorb/ab9b-v2 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf dorb/ab9b-v2:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default dorb/ab9b-v2:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use dorb/ab9b-v2 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf dorb/ab9b-v2:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "dorb/ab9b-v2:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Publish ab9b-v2
Browse filesQ4_K_M GGUF, three LoRA adapters, evaluation protocol and results.
Fresh repository. New model card.
- .gitattributes +1 -0
- README.md +107 -0
- SHA256SUMS +4 -0
- ab9b-v2-Q4_K_M.gguf +3 -0
- adapters/v2a-ck900/adapter_config.json +47 -0
- adapters/v2a-ck900/adapter_model.safetensors +3 -0
- adapters/v2b-final/adapter_config.json +47 -0
- adapters/v2b-final/adapter_model.safetensors +3 -0
- adapters/v2c-final/adapter_config.json +47 -0
- adapters/v2c-final/adapter_model.safetensors +3 -0
- evidence/GOAL.md +108 -0
- evidence/REPORT.md +133 -0
- evidence/results/between_run_decision.json +444 -0
- evidence/results/league_verdict.txt +27 -0
- evidence/results/v2a_accept_report.txt +91 -0
- evidence/results/v2a_criteria.txt +23 -0
- evidence/results/v2a_pick.json +336 -0
- evidence/results/v2b_dev_summary.txt +166 -0
- evidence/results/v2b_pick.json +342 -0
- evidence/results/v2c_dev_summary.txt +166 -0
- evidence/results/v2c_pick.json +336 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
ab9b-v2-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,107 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: Qwen/Qwen3.5-9B
|
| 4 |
+
base_model_relation: finetune
|
| 5 |
+
library_name: gguf
|
| 6 |
+
pipeline_tag: text-generation
|
| 7 |
+
language:
|
| 8 |
+
- en
|
| 9 |
+
tags:
|
| 10 |
+
- cybersecurity
|
| 11 |
+
- cve
|
| 12 |
+
- cwe
|
| 13 |
+
- cvss
|
| 14 |
+
- tool-use
|
| 15 |
+
- agent
|
| 16 |
+
- lora
|
| 17 |
+
- gguf
|
| 18 |
+
- qwen3
|
| 19 |
+
---
|
| 20 |
+
|
| 21 |
+
# ab9b-v2
|
| 22 |
+
|
| 23 |
+
Defensive-cyber model on Qwen3.5-9B. Merged and quantised to Q4_K_M. Run the GGUF. The LoRA is here if you want the bf16 base plus the adapter instead.
|
| 24 |
+
|
| 25 |
+
Trained for CVE→CWE, CVSS vectors, threat-intel questions, vulnerability review, and blue-team / DFIR / audit loops that call tools. Answers are short and lead with the conclusion. Thinking is off.
|
| 26 |
+
|
| 27 |
+
## Run
|
| 28 |
+
|
| 29 |
+
```bash
|
| 30 |
+
huggingface-cli download dorb/ab9b-v2 ab9b-v2-Q4_K_M.gguf --local-dir .
|
| 31 |
+
|
| 32 |
+
llama-server -m ab9b-v2-Q4_K_M.gguf -ngl 999 -fa on --jinja \
|
| 33 |
+
-c 32768 -np 1 --chat-template-kwargs '{"enable_thinking":false}'
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
OpenAI `tools` work with `--jinja`. The text protocol it was trained on is `CALL: tool_name(arg="…")` and then `FINAL: …`.
|
| 37 |
+
|
| 38 |
+
The LoRA that was merged into that GGUF:
|
| 39 |
+
|
| 40 |
+
```python
|
| 41 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 42 |
+
from peft import PeftModel
|
| 43 |
+
|
| 44 |
+
base = "Qwen/Qwen3.5-9B"
|
| 45 |
+
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
|
| 46 |
+
model = PeftModel.from_pretrained(model, "dorb/ab9b-v2", subfolder="adapters/v2a-ck900")
|
| 47 |
+
tokenizer = AutoTokenizer.from_pretrained(base)
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
+
`adapters/v2b-final` and `adapters/v2c-final` are the later rounds. They were scored on the dev tier and not selected. Details are in [`evidence/REPORT.md`](evidence/REPORT.md).
|
| 51 |
+
|
| 52 |
+
## Test tier
|
| 53 |
+
|
| 54 |
+
One scoring pass. Reference is `ab9b-r1` on the same base. Paired bootstrap, 95% CI.
|
| 55 |
+
|
| 56 |
+
| | ab9b-v2 | ab9b-r1 | base |
|
| 57 |
+
|---|---:|---:|---:|
|
| 58 |
+
| cyber composite, 7 families | 0.739 | 0.725 | 0.719 |
|
| 59 |
+
| CVE → CWE (n = 329) | 0.763 | 0.608 | |
|
| 60 |
+
| CVSS vector (n = 334) | 0.812 | 0.715 | |
|
| 61 |
+
| native tool-call loop, solved (n = 87) | 0.483 | 0.172 | 0.157 |
|
| 62 |
+
| plain `CALL:` loop, solved (n = 89) | 0.528 | 0.483 | 0.236 |
|
| 63 |
+
| single-call tool exact (n = 39) | 0.897 | 0.923 | 0.949 |
|
| 64 |
+
| median seconds per answer | 1.89 | 3.68 | |
|
| 65 |
+
| completion tokens per answer | 219 | 377 | |
|
| 66 |
+
|
| 67 |
+
The composite move is +0.014 [−0.006, +0.033]. That misses the pre-registered +0.05 bar. The gains that clear their intervals are CVE→CWE (+0.155), CVSS (+0.097), and the native tool-call loop (+0.310). Median answer time is about half. Token count is about half. Decode speed is the same (~110 tok/s).
|
| 68 |
+
|
| 69 |
+
On the same test items, against the other models that finished: wins over Cyber-Ornith-9B, Foundation-Sec-8B-Instruct, Primus-Merged-8B, SecGPT-7B, and the Qwen3.5-9B base. Ties with `ab9b-r1` and with Huihui-Qwen3.8-27B-abliterated. The 27B tie is at roughly twice the speed per quality point. Full table: [`evidence/results/league_verdict.txt`](evidence/results/league_verdict.txt).
|
| 70 |
+
|
| 71 |
+
## Training
|
| 72 |
+
|
| 73 |
+
One LoRA round continued from `ab9b-r1`. r = 32, α = 32, lr 1e-4 cosine, one epoch, 1,063 steps, 16.06M tokens, 4.2 hours on one RTX 3090 (24 GB). Data is defensive: NVD CVE classification, vulnerability review, threat-intel Q&A, ATT&CK / CAPEC / CWE, analyst-brief rewrites, and oracle-checked agent episodes. Nothing from the reserved eval families.
|
| 74 |
+
|
| 75 |
+
The shipped checkpoint is `v2a-ck900`. Two later rounds and two weight interpolations lost the pre-registered dev rule. `v2b` raised the agent scores and broke tool-call hygiene (it invents missing arguments). `v2c` repaired that hygiene and still scored below `v2a-ck900`, so it was never test-scored. Both adapters are in the repo so those tables can be checked.
|
| 76 |
+
|
| 77 |
+
Protocol: [`evidence/GOAL.md`](evidence/GOAL.md). Numbers: [`evidence/REPORT.md`](evidence/REPORT.md).
|
| 78 |
+
|
| 79 |
+
## Files
|
| 80 |
+
|
| 81 |
+
```
|
| 82 |
+
ab9b-v2-Q4_K_M.gguf merged Q4_K_M, 5.8 GB
|
| 83 |
+
adapters/v2a-ck900/ LoRA merged into the GGUF
|
| 84 |
+
adapters/v2b-final/ later round, not selected
|
| 85 |
+
adapters/v2c-final/ later round, not selected
|
| 86 |
+
evidence/GOAL.md protocol, written before the test pass
|
| 87 |
+
evidence/REPORT.md test table, league, rejected rounds
|
| 88 |
+
evidence/results/ raw pick and verdict files
|
| 89 |
+
SHA256SUMS GGUF and the three adapter weights
|
| 90 |
+
```
|
| 91 |
+
|
| 92 |
+
## Limits
|
| 93 |
+
|
| 94 |
+
Agent splits are small (89 / 87 / 39). Dev selection used 60 agent items, so a few points of noise is normal. Training families and the agent test share generators; eight reserved families were never trained on. CVE rows are the same task type as CTI-RCM and CTI-VSP, other CVEs, not the same items. Thinking mode, context past 64k, and languages other than English were not measured. The data is synthetic and defensive. It was not red-teamed.
|
| 95 |
+
|
| 96 |
+
## License
|
| 97 |
+
|
| 98 |
+
Apache-2.0, same as Qwen3.5-9B. Public sources in the mix keep their own licences.
|
| 99 |
+
|
| 100 |
+
```bibtex
|
| 101 |
+
@software{ab9b_v2_2026,
|
| 102 |
+
title = {ab9b-v2},
|
| 103 |
+
author = {dorb},
|
| 104 |
+
year = {2026},
|
| 105 |
+
url = {https://huggingface.co/dorb/ab9b-v2}
|
| 106 |
+
}
|
| 107 |
+
```
|
SHA256SUMS
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
c67bcf79d45e70ac8a3c53c7e49e2f229b0a46e5232f1f483cce13680e5dc4cf ab9b-v2-Q4_K_M.gguf
|
| 2 |
+
3389d0b708757034f879977cecdc3b7d14d5304c130d2f327d470d27795deea7 adapters/v2a-ck900/adapter_model.safetensors
|
| 3 |
+
cd6d20b7524f638ccb281c4156885cb33dd8afa7f0b359fa076307e16155b11a adapters/v2b-final/adapter_model.safetensors
|
| 4 |
+
ca39bd15831c37799a3ae74c4049c85295f293b07bbab9c758364e4ce1907e05 adapters/v2c-final/adapter_model.safetensors
|
ab9b-v2-Q4_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c67bcf79d45e70ac8a3c53c7e49e2f229b0a46e5232f1f483cce13680e5dc4cf
|
| 3 |
+
size 5796867904
|
adapters/v2a-ck900/adapter_config.json
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"alora_invocation_tokens": null,
|
| 3 |
+
"alpha_pattern": {},
|
| 4 |
+
"arrow_config": null,
|
| 5 |
+
"auto_mapping": {
|
| 6 |
+
"base_model_class": "Qwen3_5ForConditionalGeneration",
|
| 7 |
+
"parent_library": "transformers.models.qwen3_5.modeling_qwen3_5",
|
| 8 |
+
"unsloth_fixed": true
|
| 9 |
+
},
|
| 10 |
+
"base_model_name_or_path": "Qwen/Qwen3.5-9B",
|
| 11 |
+
"bias": "none",
|
| 12 |
+
"corda_config": null,
|
| 13 |
+
"ensure_weight_tying": false,
|
| 14 |
+
"eva_config": null,
|
| 15 |
+
"exclude_modules": null,
|
| 16 |
+
"fan_in_fan_out": false,
|
| 17 |
+
"inference_mode": true,
|
| 18 |
+
"init_lora_weights": true,
|
| 19 |
+
"kasa_config": null,
|
| 20 |
+
"layer_replication": null,
|
| 21 |
+
"layers_pattern": null,
|
| 22 |
+
"layers_to_transform": null,
|
| 23 |
+
"loftq_config": {},
|
| 24 |
+
"lora_alpha": 32,
|
| 25 |
+
"lora_bias": false,
|
| 26 |
+
"lora_dropout": 0.0,
|
| 27 |
+
"lora_ga_config": null,
|
| 28 |
+
"megatron_config": null,
|
| 29 |
+
"megatron_core": "megatron.core",
|
| 30 |
+
"modules_to_save": null,
|
| 31 |
+
"monteclora_config": null,
|
| 32 |
+
"peft_type": "LORA",
|
| 33 |
+
"peft_version": "0.21.1",
|
| 34 |
+
"qalora_group_size": 16,
|
| 35 |
+
"r": 32,
|
| 36 |
+
"rank_pattern": {},
|
| 37 |
+
"revision": null,
|
| 38 |
+
"target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}(?!.*\\.experts\\.\\d)\\..*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj)))",
|
| 39 |
+
"target_parameters": null,
|
| 40 |
+
"task_type": "CAUSAL_LM",
|
| 41 |
+
"trainable_token_indices": null,
|
| 42 |
+
"use_bdlora": null,
|
| 43 |
+
"use_dora": false,
|
| 44 |
+
"use_qalora": false,
|
| 45 |
+
"use_rslora": false,
|
| 46 |
+
"velora_config": null
|
| 47 |
+
}
|
adapters/v2a-ck900/adapter_model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3389d0b708757034f879977cecdc3b7d14d5304c130d2f327d470d27795deea7
|
| 3 |
+
size 320924856
|
adapters/v2b-final/adapter_config.json
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"alora_invocation_tokens": null,
|
| 3 |
+
"alpha_pattern": {},
|
| 4 |
+
"arrow_config": null,
|
| 5 |
+
"auto_mapping": {
|
| 6 |
+
"base_model_class": "Qwen3_5ForConditionalGeneration",
|
| 7 |
+
"parent_library": "transformers.models.qwen3_5.modeling_qwen3_5",
|
| 8 |
+
"unsloth_fixed": true
|
| 9 |
+
},
|
| 10 |
+
"base_model_name_or_path": "Qwen/Qwen3.5-9B",
|
| 11 |
+
"bias": "none",
|
| 12 |
+
"corda_config": null,
|
| 13 |
+
"ensure_weight_tying": false,
|
| 14 |
+
"eva_config": null,
|
| 15 |
+
"exclude_modules": null,
|
| 16 |
+
"fan_in_fan_out": false,
|
| 17 |
+
"inference_mode": true,
|
| 18 |
+
"init_lora_weights": true,
|
| 19 |
+
"kasa_config": null,
|
| 20 |
+
"layer_replication": null,
|
| 21 |
+
"layers_pattern": null,
|
| 22 |
+
"layers_to_transform": null,
|
| 23 |
+
"loftq_config": {},
|
| 24 |
+
"lora_alpha": 32,
|
| 25 |
+
"lora_bias": false,
|
| 26 |
+
"lora_dropout": 0.0,
|
| 27 |
+
"lora_ga_config": null,
|
| 28 |
+
"megatron_config": null,
|
| 29 |
+
"megatron_core": "megatron.core",
|
| 30 |
+
"modules_to_save": null,
|
| 31 |
+
"monteclora_config": null,
|
| 32 |
+
"peft_type": "LORA",
|
| 33 |
+
"peft_version": "0.21.1",
|
| 34 |
+
"qalora_group_size": 16,
|
| 35 |
+
"r": 32,
|
| 36 |
+
"rank_pattern": {},
|
| 37 |
+
"revision": null,
|
| 38 |
+
"target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}(?!.*\\.experts\\.\\d)\\..*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj)))",
|
| 39 |
+
"target_parameters": null,
|
| 40 |
+
"task_type": "CAUSAL_LM",
|
| 41 |
+
"trainable_token_indices": null,
|
| 42 |
+
"use_bdlora": null,
|
| 43 |
+
"use_dora": false,
|
| 44 |
+
"use_qalora": false,
|
| 45 |
+
"use_rslora": false,
|
| 46 |
+
"velora_config": null
|
| 47 |
+
}
|
adapters/v2b-final/adapter_model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cd6d20b7524f638ccb281c4156885cb33dd8afa7f0b359fa076307e16155b11a
|
| 3 |
+
size 320924856
|
adapters/v2c-final/adapter_config.json
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"alora_invocation_tokens": null,
|
| 3 |
+
"alpha_pattern": {},
|
| 4 |
+
"arrow_config": null,
|
| 5 |
+
"auto_mapping": {
|
| 6 |
+
"base_model_class": "Qwen3_5ForConditionalGeneration",
|
| 7 |
+
"parent_library": "transformers.models.qwen3_5.modeling_qwen3_5",
|
| 8 |
+
"unsloth_fixed": true
|
| 9 |
+
},
|
| 10 |
+
"base_model_name_or_path": "Qwen/Qwen3.5-9B",
|
| 11 |
+
"bias": "none",
|
| 12 |
+
"corda_config": null,
|
| 13 |
+
"ensure_weight_tying": false,
|
| 14 |
+
"eva_config": null,
|
| 15 |
+
"exclude_modules": null,
|
| 16 |
+
"fan_in_fan_out": false,
|
| 17 |
+
"inference_mode": true,
|
| 18 |
+
"init_lora_weights": true,
|
| 19 |
+
"kasa_config": null,
|
| 20 |
+
"layer_replication": null,
|
| 21 |
+
"layers_pattern": null,
|
| 22 |
+
"layers_to_transform": null,
|
| 23 |
+
"loftq_config": {},
|
| 24 |
+
"lora_alpha": 32,
|
| 25 |
+
"lora_bias": false,
|
| 26 |
+
"lora_dropout": 0.0,
|
| 27 |
+
"lora_ga_config": null,
|
| 28 |
+
"megatron_config": null,
|
| 29 |
+
"megatron_core": "megatron.core",
|
| 30 |
+
"modules_to_save": null,
|
| 31 |
+
"monteclora_config": null,
|
| 32 |
+
"peft_type": "LORA",
|
| 33 |
+
"peft_version": "0.21.1",
|
| 34 |
+
"qalora_group_size": 16,
|
| 35 |
+
"r": 32,
|
| 36 |
+
"rank_pattern": {},
|
| 37 |
+
"revision": null,
|
| 38 |
+
"target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}(?!.*\\.experts\\.\\d)\\..*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?\\.(?:(?:down_proj|gate_proj|in_proj_qkv|in_proj_z|k_proj|o_proj|out_proj|q_proj|up_proj|v_proj)))",
|
| 39 |
+
"target_parameters": null,
|
| 40 |
+
"task_type": "CAUSAL_LM",
|
| 41 |
+
"trainable_token_indices": null,
|
| 42 |
+
"use_bdlora": null,
|
| 43 |
+
"use_dora": false,
|
| 44 |
+
"use_qalora": false,
|
| 45 |
+
"use_rslora": false,
|
| 46 |
+
"velora_config": null
|
| 47 |
+
}
|
adapters/v2c-final/adapter_model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ca39bd15831c37799a3ae74c4049c85295f293b07bbab9c758364e4ce1907e05
|
| 3 |
+
size 320924856
|
evidence/GOAL.md
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ab9b-v2 — design and evaluation protocol
|
| 2 |
+
|
| 3 |
+
A LoRA on Qwen3.5-9B for defensive cybersecurity: CVE triage, threat-intel Q&A, vulnerability review, and native tool calling in blue-team / DFIR / audit agent loops.
|
| 4 |
+
|
| 5 |
+
This document fixes the method, the acceptance criteria, the data rules, and the decision rules **before any model is scored on the test tier**. It is checked in alongside the shipped weights so every claim in `REPORT.md` can be audited against a pre-registered gate.
|
| 6 |
+
|
| 7 |
+
## 1. Model under test
|
| 8 |
+
|
| 9 |
+
- Base: Qwen3.5-9B dense (Apache-2.0), thinking disabled, MTP draft head bit-identical to the base.
|
| 10 |
+
- Adaptation: LoRA r = 32, α = 32, continued from the prior round `ab9b-r1`.
|
| 11 |
+
- Serving: `llama.cpp` server, identical flags for every model in the comparison.
|
| 12 |
+
|
| 13 |
+
## 2. Scoring
|
| 14 |
+
|
| 15 |
+
### 2.1 Cyber composite (primary)
|
| 16 |
+
|
| 17 |
+
Unweighted mean of seven family scores, each in [0, 1], zero-shot chat, held-out **test** tier, scored exactly **once per model tag**.
|
| 18 |
+
|
| 19 |
+
| family | source |
|
| 20 |
+
|---|---|
|
| 21 |
+
| F1 CyberMetric | `tihanyin/CyberMetric` — MCQ |
|
| 22 |
+
| F2 SecEval | `XuanwuAI/SecEval` — MCQ |
|
| 23 |
+
| F3 CTIBench | `AI4Sec/cti-bench` — mean of {mcq, rcm, vsp, ate, taa} |
|
| 24 |
+
| F4 SecBench | `secbench-hf/SecBench` — English MCQ |
|
| 25 |
+
| F5 SecQA | `zefang-liu/secqa` — v1 + v2 |
|
| 26 |
+
| F6 CyberBench | JPMorgan CyberBench — 10 datasets, 5-shot, 200-item seeded cap, 512 new tokens |
|
| 27 |
+
| F7 vuln2 | own CWE / safe-code harness, sealed tier |
|
| 28 |
+
|
| 29 |
+
### 2.2 Agent tracks (secondary; not in the composite)
|
| 30 |
+
|
| 31 |
+
- `agent-plain solved` — in-text `CALL: … / FINAL: …` protocol, test tier.
|
| 32 |
+
- `agent-native solved` — native OpenAI tool-call loop, test tier.
|
| 33 |
+
- `toolcall exact` — single-call exact match on 39 fixed situations.
|
| 34 |
+
|
| 35 |
+
### 2.3 Guards (dev tier; reported, not aggregated)
|
| 36 |
+
|
| 37 |
+
`mmlu_sec`, `coding2`, `agent2`, and XSTest safe compliance; each must stay within 0.03 of `ab9b-r1`.
|
| 38 |
+
|
| 39 |
+
### 2.4 Speed
|
| 40 |
+
|
| 41 |
+
`quality_per_second = cyber_composite / median wall-seconds per item at batch 1`, under the same server flags as the league.
|
| 42 |
+
|
| 43 |
+
## 3. Acceptance criteria (pre-registered)
|
| 44 |
+
|
| 45 |
+
Against the previous round `ab9b-r1`, paired bootstrap 95% CI on shared items:
|
| 46 |
+
|
| 47 |
+
| id | criterion | gate |
|
| 48 |
+
|---|---|---|
|
| 49 |
+
| C1 | cyber composite (test) | diff ≥ +0.05 absolute **and** paired lower bound > 0 |
|
| 50 |
+
| C2a | agent-plain solved (test) | diff ≥ +0.15 absolute **and** paired lower bound > 0 |
|
| 51 |
+
| C2b | native tool-call exact (test, n = 39) | not below `ab9b-r1` |
|
| 52 |
+
| C2c | agent2 guard | not more than 0.03 below `ab9b-r1` |
|
| 53 |
+
| C3 | median wall-time, batch 1 | ≤ 1.10× `ab9b-r1` |
|
| 54 |
+
| C3 | completion tokens per solved task | not above `ab9b-r1` |
|
| 55 |
+
| C3b | MTP head tensors | bit-identical to the base |
|
| 56 |
+
| C4 | dev guards (mmlu_sec, coding2, XSTest safe) | each within 0.03 of `ab9b-r1` |
|
| 57 |
+
| C5 | XSTest unsafe-refusal | not below the lower 95% bound of `ab9b-r1` |
|
| 58 |
+
|
| 59 |
+
The release is **accepted** iff all of C1–C5 hold. If any criterion misses, the report states exactly which hold and which do not; no stronger wording is used.
|
| 60 |
+
|
| 61 |
+
## 4. League and allowed claims
|
| 62 |
+
|
| 63 |
+
The league is every generative LLM on the Hugging Face Hub that (a) presents itself as cybersecurity-specialised, (b) has a GGUF at approximately Q4 quality, (c) is ≤ 24 GB, and (d) has either ≥ 1 000 downloads or ≥ 20 likes or an established organisation behind it. One entry per base / fine-tune family (best-documented repo), plus Qwen3.5-9B base and `ab9b-r1`. All members run under identical server flags; models that need a different chat template get the template that `llama.cpp` ships in the GGUF (`--jinja`), with thinking disabled where the template supports it and `<think>…</think>` stripped otherwise.
|
| 64 |
+
|
| 65 |
+
Verdicts (paired bootstrap on shared items, 95%; a win needs the paired lower bound of the difference > 0):
|
| 66 |
+
|
| 67 |
+
- **W1 size-fair** — `ab9b-v2` beats every ≤ 10 B league member on the cyber composite.
|
| 68 |
+
- **W2 speed-fair** — `ab9b-v2` has the highest `quality_per_second` of the whole league.
|
| 69 |
+
- **W3 absolute** — `ab9b-v2` ≥ every league member of any size on the cyber composite (point difference ≥ 0 and paired 95% upper bound ≥ 0, against every member).
|
| 70 |
+
|
| 71 |
+
The sentence "wins any other cyber LLM" is allowed only if **W1 ∧ W3 ∧ C4** hold. Otherwise the report states exactly which of W1 / W2 / W3 / C4 hold and nothing stronger.
|
| 72 |
+
|
| 73 |
+
## 5. Data rules
|
| 74 |
+
|
| 75 |
+
1. **Defensive and benign only.** No exploit, malware, or weaponisation material. No CTF-solution dumps. No attack-payload corpora.
|
| 76 |
+
2. **Permissive licences only** for training rows (Apache-2.0, MIT, CC0, CC-BY, ODC-BY); non-commercial sets are excluded.
|
| 77 |
+
3. **Text that enters training is written by official open-weights models** under Apache-2.0 / MIT that are not themselves fine-tuned on proprietary-model outputs. Uncensored derivatives are used only as reviewers and brainstormers; their text enters training only as oracle-verified agent episodes (backup teacher).
|
| 78 |
+
4. **Decontamination against every benchmark used**: 13-gram overlap on normalised text, embedding cosine ≥ 0.90, fact-level exclusion by CVE-ID, CWE-ID, and ATT&CK technique-ID. The 8 reserved agent families are never trained on.
|
| 79 |
+
5. **Agent episodes are verified by programmatic oracles**; the student's own rollouts enter training only after oracle acceptance.
|
| 80 |
+
|
| 81 |
+
## 6. Decision rules
|
| 82 |
+
|
| 83 |
+
### 6.1 Candidate selection within a run (dev tier only)
|
| 84 |
+
|
| 85 |
+
A candidate is **eligible** if every guard (mmlu_sec, coding2, agent2, XSTest safe) is within 0.03 of `ab9b-r1` **and** the dev-tier single-call tool-call exact is within 0.03 of `ab9b-r1`. The score is
|
| 86 |
+
|
| 87 |
+
```
|
| 88 |
+
score = ½ · (dev composite proxy gain / 0.05)
|
| 89 |
+
+ ½ · (agent-plain dev solved gain / 0.15)
|
| 90 |
+
```
|
| 91 |
+
|
| 92 |
+
against `ab9b-r1`. The highest-scoring eligible candidate wins; candidates within 0.02 of the best are tied, and the latest checkpoint of the tied set is picked. If no eligible candidate exists, the pick is `null` and the run is not automatically test-scored.
|
| 93 |
+
|
| 94 |
+
### 6.2 Between-run replacement
|
| 95 |
+
|
| 96 |
+
A later round replaces the incumbent only if its pick is eligible **and** its score is at least +0.20 above the incumbent's. Ties within 0.02 go to the non-interpolated, later candidate. The test tier is scored exactly **once per model tag**; no selection decision is ever made from a test-tier number.
|
| 97 |
+
|
| 98 |
+
### 6.3 Interpolation (optional)
|
| 99 |
+
|
| 100 |
+
If no new-round candidate is eligible, additional candidates may be built in weight space between the incumbent adapter and the new round's adapter at mixing weights w ∈ {0.5, 0.75}. Interpolated candidates are scored on the dev tier like the rest and considered under the same rule.
|
| 101 |
+
|
| 102 |
+
## 7. Hard rules
|
| 103 |
+
|
| 104 |
+
- Training runs on one RTX 3090 (24 GB).
|
| 105 |
+
- No training data from the 8 reserved agent families or from the test tier of any benchmark listed in §2.1.
|
| 106 |
+
- Nothing is uploaded to Hugging Face except the final model repository.
|
| 107 |
+
- Every candidate scored and every decision taken is recorded in `REPORT.md` and the files under `results/`.
|
| 108 |
+
- No evaluation decision is ever made from a test-tier number; test numbers are read **after** the decision rule has picked the model.
|
evidence/REPORT.md
ADDED
|
@@ -0,0 +1,133 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ab9b-v2 — test-tier report
|
| 2 |
+
|
| 3 |
+
The model shipped by this repository is **`ab9b-v2a-ck900`** (checkpoint 900 of the first training round), chosen from three rounds by the dev-tier decision rule of [`GOAL.md`](GOAL.md) §6. It is the only candidate of this release that was scored on the test tier.
|
| 4 |
+
|
| 5 |
+
Two later rounds (`v2b` hard-example mining, `v2c` tool-policy repair) and two weight-space interpolations between `ab9b-r1` and `v2a-ck900` were evaluated on the dev tier only; they are documented in §4 for completeness and were **not** test-scored.
|
| 6 |
+
|
| 7 |
+
## 1. Status against the pre-registered acceptance criteria
|
| 8 |
+
|
| 9 |
+
Reference = `ab9b-r1`, the previous round on the same base.
|
| 10 |
+
|
| 11 |
+
| id | criterion | result | verdict |
|
| 12 |
+
|---|---|---:|:---:|
|
| 13 |
+
| C1 | cyber composite +0.05 (paired CI > 0) | 0.739 vs 0.725 — diff +0.014 [−0.006, +0.033] | FAIL |
|
| 14 |
+
| C2a | agent-plain solved +0.15 (CI > 0) | 0.528 vs 0.483 — diff +0.045 [−0.056, +0.146], n = 89 | FAIL |
|
| 15 |
+
| C2b | native tool-call exact ≥ r1 (n = 39) | 0.897 vs 0.923 | FAIL |
|
| 16 |
+
| C2c | agent2 guard ≥ −0.03 | +0.050 | PASS |
|
| 17 |
+
| C3 | median wall-time ≤ 1.10× r1 | 1.169 s vs 1.226 s (0.95×) | PASS |
|
| 18 |
+
| C3 | completion tokens per solved task ≤ r1 | 139.6 vs 170.5 | PASS |
|
| 19 |
+
| C3b | MTP head bit-identical to the base | 15 / 15 tensors identical | PASS |
|
| 20 |
+
| C4 | guards within 0.03 (mmlu_sec, coding2, XSTest safe) | −0.005 / −0.017 / 0.000 | PASS |
|
| 21 |
+
| C5 | XSTest unsafe-refusal ≥ r1 lower-95% bound (0.093) | 0.179 vs 0.142 | PASS |
|
| 22 |
+
|
| 23 |
+
**Summary.** `ab9b-v2` holds C3, C4, and C5. It misses C1 and C2 on the paired lower bound. The significant gains concentrate in CVE triage, CVSS vector scoring, native tool-call solve rate, answer length, latency, and style — see §2. The release is therefore a **targeted improvement** over `ab9b-r1`, not a composite-tier replacement.
|
| 24 |
+
|
| 25 |
+
## 2. Where the gains are (test tier, paired bootstrap 95%)
|
| 26 |
+
|
| 27 |
+
| task | n | ab9b-v2 | ab9b-r1 | diff | 95% CI | verdict |
|
| 28 |
+
|---|---:|---:|---:|---:|---|:---:|
|
| 29 |
+
| CTI-RCM — CVE → CWE | 329 | **0.763** | 0.608 | +0.155 | [+0.113, +0.204] | WIN |
|
| 30 |
+
| CTI-VSP — CVSS vector | 334 | **0.812** | 0.715 | +0.097 | [+0.073, +0.122] | WIN |
|
| 31 |
+
| agent-native solved | 87 | **0.483** | 0.172 | +0.310 | [+0.195, +0.425] | WIN |
|
| 32 |
+
| agent-plain solved | 89 | 0.528 | 0.483 | +0.045 | [−0.056, +0.146] | TIE |
|
| 33 |
+
| CyberMetric | 590 | 0.897 | 0.900 | −0.003 | [−0.017, +0.010] | TIE |
|
| 34 |
+
| SecEval | 654 | 0.755 | 0.749 | +0.006 | [−0.015, +0.026] | TIE |
|
| 35 |
+
| CTI-MCQ | 726 | 0.705 | 0.711 | −0.006 | [−0.032, +0.022] | TIE |
|
| 36 |
+
| CTI-ATE | 22 | 0.242 | 0.306 | −0.065 | [−0.143, +0.012] | TIE |
|
| 37 |
+
| CTI-TAA | 16 | 0.063 | 0.063 | 0.000 | [+0.000, +0.000] | TIE |
|
| 38 |
+
| SecBench | 205 | 0.839 | 0.834 | +0.005 | [−0.024, +0.034] | TIE |
|
| 39 |
+
| SecQA | 73 | 0.932 | 0.945 | −0.014 | [−0.041, +0.000] | TIE |
|
| 40 |
+
| **cyber composite** | — | **0.739** | 0.725 | +0.014 | [−0.006, +0.033] | TIE |
|
| 41 |
+
|
| 42 |
+
Secondary speed and style numbers (14 analyst prompts, batch-1, temperature 0; 50 fixed SOC-analyst questions):
|
| 43 |
+
|
| 44 |
+
- Median wall-time per answer: **1.89 s** vs 3.68 s (−49%).
|
| 45 |
+
- Completion tokens per answer: **219** vs 377 (−42%).
|
| 46 |
+
- Decode speed: 107 vs 112 tok/s (equal).
|
| 47 |
+
- MTP speculative draft (`--spec-type draft-mtp`): r1 +13% (acceptance 0.47), `ab9b-v2` +1% (acceptance 0.42) — not worth enabling on this model.
|
| 48 |
+
- Style indicators: bottom-line-first first line 56% → **98%**; closing `Next:` action line 0% → **86%**; answers > 180 words 88% → **2%**; markdown headings or bold labels 50% → **6%**.
|
| 49 |
+
- Style rubric judge (local HauhauCS 35B-A3B, 50 answers): bottom line first 0.74 → **1.00**, concise 0.50 → **1.00**, structured 0.74 → **0.94**, actionable 0.88 → **1.00**, no invented specifics 0.86 → 0.88.
|
| 50 |
+
|
| 51 |
+
## 3. League verdict (test tier, seven members complete)
|
| 52 |
+
|
| 53 |
+
Same server flags for every member, paired bootstrap on shared items, 95%.
|
| 54 |
+
|
| 55 |
+
| opponent | size | diff vs ab9b-v2 | 95% CI | verdict |
|
| 56 |
+
|---|---|---:|---|:---:|
|
| 57 |
+
| `ab9b-r1` | 9 B | +0.014 | [−0.006, +0.033] | TIE |
|
| 58 |
+
| Cyber-Ornith-9B | 9 B | +0.092 | [+0.072, +0.112] | WIN |
|
| 59 |
+
| Foundation-Sec-8B-Instruct | 8 B | +0.073 | [+0.050, +0.095] | WIN |
|
| 60 |
+
| Primus-Merged-8B | 8 B | +0.117 | [+0.095, +0.139] | WIN |
|
| 61 |
+
| SecGPT-7B | 7 B | +0.115 | [+0.094, +0.137] | WIN |
|
| 62 |
+
| Qwen3.5-9B base | 9 B | +0.020 | [+0.002, +0.038] | WIN |
|
| 63 |
+
| Huihui-Qwen3.8-27B-abliterated | 27 B | +0.019 | [−0.002, +0.040] | TIE |
|
| 64 |
+
|
| 65 |
+
- **W1 size-fair** (beat every ≤ 10 B league member, `ab9b-r1` included): **does not hold** — tie with `ab9b-r1`.
|
| 66 |
+
- **W2 speed-fair** (highest `quality_per_second` of the league): **does not hold** — three smaller distilled models have 2 – 3× the quality-per-second.
|
| 67 |
+
- **W3 absolute** (≥ every league member of any size on the cyber composite): **holds** across the seven complete members. The 35 B-A3B league member (HauhauCS) could not be completed — its server returned a CUDA fault during the `cti-ate` stage, so `cti-ate`, `cti-taa`, `secbench`, and `secqa` were never scored for it; its CyberBench and own-sealed stages did complete. W3 therefore stays undecided against that one model.
|
| 68 |
+
|
| 69 |
+
Per the claim policy in `GOAL.md` §4, the sentence "wins any other cyber LLM" is **not allowed**.
|
| 70 |
+
|
| 71 |
+
Against the Huihui-Qwen3.8-27B abliteration specifically, `ab9b-v2` is a tie on the composite (+0.019 [−0.002, +0.040]) and about **1.9× faster per quality point** (`quality_per_second` 0.632 vs 0.333).
|
| 72 |
+
|
| 73 |
+
## 4. Rounds scored on the dev tier and not selected
|
| 74 |
+
|
| 75 |
+
### 4.1 `v2b` — hard-example mining and new families
|
| 76 |
+
|
| 77 |
+
Continued from `v2a-ck900` at lr 5e-5, one epoch, 777 steps, 3.5 h, 19 007 rows / 11.4 M tokens: K = 2 rejection-sampling rollouts of `v2a-ck900` on 26 training families, oracle trajectories of the episodes `v2a-ck900` could not solve (203 hard-example rows), oracle rows of 10 new audit / IR families (`sysctl_effective`, `k8s_rbac`, `logrotate_retention`, `config_diff_risk`, `alert_join`, `authkeys_audit`, `pg_hba_audit`, `suid_audit`, `auditd_seq_gap`, `commit_secret_origin`), 35% replay of the `v2a` knowledge streams, and 714 technique-extraction rows targeting the CTI-ATE regression.
|
| 78 |
+
|
| 79 |
+
| dev metric (n) | ab9b-r1 | v2a-ck900 | **v2b-final** |
|
| 80 |
+
|---|---:|---:|---:|
|
| 81 |
+
| agent-plain solved (60) | 0.550 | 0.633 | **0.683** |
|
| 82 |
+
| agent-native solved (59) | 0.288 | 0.593 | **0.661** |
|
| 83 |
+
| **tool-call exact (91)** | 0.978 | 0.956 | **0.846** |
|
| 84 |
+
| CTI-ATE (38) | 0.358 | 0.230 | 0.371 |
|
| 85 |
+
| dev composite proxy | 0.7533 | 0.7649 | 0.7616 |
|
| 86 |
+
|
| 87 |
+
`v2b` improved every agent track **and** fixed the CTI-ATE regression, but single-call tool-call exact dropped from 0.956 to 0.846 — well below the dev-tier eligibility threshold of r1 − 0.03 = 0.948. Per-category diagnosis (n = 91 dev): `missing_arg` 0.33 vs 1.00 (the model invents a value instead of asking); `single` 0.81 vs 1.00 (an unrequested optional `reason` argument is added). This is exactly the regression the hygiene guard exists for, so every `v2b` candidate was **ineligible** by `GOAL.md` §6.1 and the pick was `null`. Likely cause: every native episode of round 2 has complete information and ends in a call, and the single-call anchors of round 1 were replayed at only 35%.
|
| 88 |
+
|
| 89 |
+
### 4.2 `v2c` — tool-policy repair
|
| 90 |
+
|
| 91 |
+
Continued from `v2b-final` at lr 5e-5, one epoch, 296 steps, 1.3 h, 7 336 rows / 4.35 M tokens: 25% replay of the `v2b` mix (with the analyst-brief restyled answers at 100%) plus 1 600 programmatic tool-policy rows. The new rows cover 24 tool schemas disjoint from the 10 evaluation tools; the behaviours taught are: exact call with only the required arguments; optional values when the user gives them explicitly; several independent calls when appropriate; one short question when a required value is missing; abstain in text when no tool fits. Every row was schema-validated; a word-4-gram decontamination against the 130 evaluation prompts and the 10 evaluation tool names found 6 overlaps in the first draft and 0 after cleanup.
|
| 92 |
+
|
| 93 |
+
| dev metric | ab9b-r1 | v2a-ck900 | v2b-final | **v2c-final** |
|
| 94 |
+
|---|---:|---:|---:|---:|
|
| 95 |
+
| agent-plain solved (60) | 0.550 | 0.633 | 0.683 | **0.633** |
|
| 96 |
+
| agent-native solved (59) | 0.288 | 0.583 | 0.661 | **0.627** |
|
| 97 |
+
| tool-call exact (91) | 0.978 | 0.956 | 0.846 | **0.978** |
|
| 98 |
+
| dev composite proxy | 0.7533 | 0.7649 | 0.7616 | 0.7581 |
|
| 99 |
+
| decision score vs r1 | — | +0.393 | +0.53 (ineligible) | **+0.326** |
|
| 100 |
+
|
| 101 |
+
The repair worked on the thing it targeted: tool-call exact returned to 0.978, equal to `ab9b-r1`; guards are clean (mmlu_sec +0.005, coding2 +0.004, agent2 +0.150, XSTest safe −0.011); the agent-native solve rate was mostly kept (0.627 vs 0.661). But the pre-registered score is +0.326, below the incumbent's +0.393, so by the between-run rule `v2c` did not replace `v2a-ck900` and was never test-scored.
|
| 102 |
+
|
| 103 |
+
An honest reading of the dev table: `v2c` and `v2a-ck900` are within the noise of 60 agentic items; `v2c` is the slightly better agent **with tool-use hygiene intact**, while `v2a-ck900` is the slightly stronger knowledge model. The rule picks the latter, and so the only test scoring that exists is `v2a-ck900`'s. The `v2b` and `v2c` adapters are in this repository under `adapters/` so the dev tables can be reproduced. They were not selected and were not test-scored.
|
| 104 |
+
|
| 105 |
+
### 4.3 Weight-space interpolation
|
| 106 |
+
|
| 107 |
+
Between `ab9b-r1` and `v2a-ck900` at mixing weights w ∈ {0.5, 0.75}: scores +0.115 and +0.113, both well below the incumbent's +0.393. Dropped.
|
| 108 |
+
|
| 109 |
+
### 4.4 In-distribution check on the training families (not an acceptance criterion)
|
| 110 |
+
|
| 111 |
+
Fresh episodes (seed 77, greedy, plain protocol, 10 per family) on the 26 training families: `ab9b-r1` 0.60, `v2a-ck900` 0.71, `v2b-final` 0.79. The 10 new audit / IR families were learned only partly by `v2b` (alert_join 0.4, sysctl_effective 0.4, logrotate_retention 0.5, pg_hba_audit 0.5); `firewall_audit` stayed weak for both (`v2a-ck900` 0.1, `v2b-final` 0.4). This is in-distribution and not an acceptance criterion; the 8 reserved agent families were never trained on and are the correct transfer measure.
|
| 112 |
+
|
| 113 |
+
## 5. Reproducibility & evidence
|
| 114 |
+
|
| 115 |
+
The raw selection and verdict files are in [`results/`](results/):
|
| 116 |
+
|
| 117 |
+
- `v2a_pick.json`, `v2b_pick.json`, `v2c_pick.json` — every candidate scored, every eligibility test, the final pick.
|
| 118 |
+
- `v2b_dev_summary.txt`, `v2c_dev_summary.txt` — dev-tier paired tables for the two rounds that were not selected.
|
| 119 |
+
- `between_run_decision.json` — the between-run decision over the three rounds and the two interpolations.
|
| 120 |
+
- `v2a_criteria.txt` — the C1–C5 acceptance report of §1.
|
| 121 |
+
- `v2a_accept_report.txt` — the full paired-bootstrap tables underlying §2.
|
| 122 |
+
- `league_verdict.txt` — the W1 / W2 / W3 verdict and the per-opponent `quality_per_second`.
|
| 123 |
+
|
| 124 |
+
## 6. Limitations and honest caveats
|
| 125 |
+
|
| 126 |
+
- Agent test splits are small (n = 89 plain, n = 87 native, n = 39 tool-call). Even a true +0.15 effect clears the paired lower bound only about half of the time at this size.
|
| 127 |
+
- Dev-tier selection among candidates uses 60 agent items and cap-400 battery tasks; the noise of that selection is a few points.
|
| 128 |
+
- The training agent families and the agent test items share generators in part; the 8 reserved agent families were never trained on and are the correct transfer measure.
|
| 129 |
+
- NVD CVE → CWE / CVSS rows are the same task **type** as CTI-RCM, CTI-VSP, and CyberBench `cve` items (other CVEs) — in-domain, not identical-item leakage.
|
| 130 |
+
- The CyberBench deviation (first 128 tokens, not 512, used for the embedding stage of decontamination) is a documented deviation from the instruction; the n-gram stage covers the full text.
|
| 131 |
+
- Thinking mode, context above 64 k tokens, and non-English use were not evaluated.
|
| 132 |
+
- The 35 B-A3B league member (HauhauCS) did not complete the battery (CUDA fault); its W3 verdict is undecided.
|
| 133 |
+
- Training data is synthetic and defensive; it has not been red-teamed for misuse. This release is a research artefact.
|
evidence/results/between_run_decision.json
ADDED
|
@@ -0,0 +1,444 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ref": "ab9b-r1 (dev)",
|
| 3 |
+
"margin": 0.2,
|
| 4 |
+
"incumbent": {
|
| 5 |
+
"label": "v2a-ck900",
|
| 6 |
+
"battery_tag": "v2a-ck900 (dev)",
|
| 7 |
+
"eligible": true,
|
| 8 |
+
"reasons": [],
|
| 9 |
+
"families_used": [
|
| 10 |
+
"ctibench",
|
| 11 |
+
"cybermetric",
|
| 12 |
+
"secbench",
|
| 13 |
+
"seceval",
|
| 14 |
+
"secqa",
|
| 15 |
+
"vuln2"
|
| 16 |
+
],
|
| 17 |
+
"proxy": 0.76485,
|
| 18 |
+
"ref_proxy": 0.7532666666666668,
|
| 19 |
+
"families": {
|
| 20 |
+
"ctibench": [
|
| 21 |
+
0.5748,
|
| 22 |
+
0.5141
|
| 23 |
+
],
|
| 24 |
+
"cybermetric": [
|
| 25 |
+
0.885,
|
| 26 |
+
0.9075
|
| 27 |
+
],
|
| 28 |
+
"secbench": [
|
| 29 |
+
0.79,
|
| 30 |
+
0.8
|
| 31 |
+
],
|
| 32 |
+
"seceval": [
|
| 33 |
+
0.735,
|
| 34 |
+
0.74
|
| 35 |
+
],
|
| 36 |
+
"secqa": [
|
| 37 |
+
0.9927,
|
| 38 |
+
0.9927
|
| 39 |
+
],
|
| 40 |
+
"vuln2": [
|
| 41 |
+
0.6116,
|
| 42 |
+
0.5653
|
| 43 |
+
]
|
| 44 |
+
},
|
| 45 |
+
"g1": 0.23166666666666558,
|
| 46 |
+
"guards": {
|
| 47 |
+
"mmlu_sec": -0.0049,
|
| 48 |
+
"coding2": -0.0167,
|
| 49 |
+
"agent2": 0.05,
|
| 50 |
+
"xstest_safe_compliance": 0.0
|
| 51 |
+
},
|
| 52 |
+
"agentcy": {
|
| 53 |
+
"plain_solved": {
|
| 54 |
+
"n": 60,
|
| 55 |
+
"a": 0.6333,
|
| 56 |
+
"b": 0.55,
|
| 57 |
+
"diff": 0.0833,
|
| 58 |
+
"ci95": [
|
| 59 |
+
-0.0333,
|
| 60 |
+
0.2
|
| 61 |
+
],
|
| 62 |
+
"verdict": "TIE"
|
| 63 |
+
},
|
| 64 |
+
"native_solved": {
|
| 65 |
+
"n": 59,
|
| 66 |
+
"a": 0.5932,
|
| 67 |
+
"b": 0.2881,
|
| 68 |
+
"diff": 0.3051,
|
| 69 |
+
"ci95": [
|
| 70 |
+
0.1695,
|
| 71 |
+
0.4407
|
| 72 |
+
],
|
| 73 |
+
"verdict": "WIN"
|
| 74 |
+
},
|
| 75 |
+
"toolcall_exact": {
|
| 76 |
+
"n": 91,
|
| 77 |
+
"a": 0.956,
|
| 78 |
+
"b": 0.978,
|
| 79 |
+
"diff": -0.022,
|
| 80 |
+
"ci95": [
|
| 81 |
+
-0.0549,
|
| 82 |
+
0.0
|
| 83 |
+
],
|
| 84 |
+
"verdict": "TIE"
|
| 85 |
+
},
|
| 86 |
+
"plain_reserved": {
|
| 87 |
+
"n": 45,
|
| 88 |
+
"a": 0.5111,
|
| 89 |
+
"b": 0.4889,
|
| 90 |
+
"diff": 0.0222,
|
| 91 |
+
"ci95": [
|
| 92 |
+
-0.1111,
|
| 93 |
+
0.1556
|
| 94 |
+
],
|
| 95 |
+
"verdict": "TIE"
|
| 96 |
+
},
|
| 97 |
+
"plain_unseen": {
|
| 98 |
+
"n": 15,
|
| 99 |
+
"a": 1.0,
|
| 100 |
+
"b": 0.7333,
|
| 101 |
+
"diff": 0.2667,
|
| 102 |
+
"ci95": [
|
| 103 |
+
0.0667,
|
| 104 |
+
0.4667
|
| 105 |
+
],
|
| 106 |
+
"verdict": "WIN"
|
| 107 |
+
}
|
| 108 |
+
},
|
| 109 |
+
"g2": 0.5553333333333333,
|
| 110 |
+
"score": 0.39349999999999946
|
| 111 |
+
},
|
| 112 |
+
"challengers": [
|
| 113 |
+
{
|
| 114 |
+
"label": "v2a-interp-w050",
|
| 115 |
+
"battery_tag": "v2a-interp-w050 (dev)",
|
| 116 |
+
"eligible": true,
|
| 117 |
+
"reasons": [],
|
| 118 |
+
"families_used": [
|
| 119 |
+
"ctibench",
|
| 120 |
+
"cybermetric",
|
| 121 |
+
"secbench",
|
| 122 |
+
"seceval",
|
| 123 |
+
"secqa",
|
| 124 |
+
"vuln2"
|
| 125 |
+
],
|
| 126 |
+
"proxy": 0.7592333333333334,
|
| 127 |
+
"ref_proxy": 0.7532666666666668,
|
| 128 |
+
"families": {
|
| 129 |
+
"ctibench": [
|
| 130 |
+
0.5714,
|
| 131 |
+
0.5141
|
| 132 |
+
],
|
| 133 |
+
"cybermetric": [
|
| 134 |
+
0.895,
|
| 135 |
+
0.9075
|
| 136 |
+
],
|
| 137 |
+
"secbench": [
|
| 138 |
+
0.79,
|
| 139 |
+
0.8
|
| 140 |
+
],
|
| 141 |
+
"seceval": [
|
| 142 |
+
0.7375,
|
| 143 |
+
0.74
|
| 144 |
+
],
|
| 145 |
+
"secqa": [
|
| 146 |
+
0.9927,
|
| 147 |
+
0.9927
|
| 148 |
+
],
|
| 149 |
+
"vuln2": [
|
| 150 |
+
0.5688,
|
| 151 |
+
0.5653
|
| 152 |
+
]
|
| 153 |
+
},
|
| 154 |
+
"g1": 0.11933333333333351,
|
| 155 |
+
"guards": {
|
| 156 |
+
"mmlu_sec": 0.0196,
|
| 157 |
+
"coding2": 0.0084,
|
| 158 |
+
"agent2": -0.025,
|
| 159 |
+
"xstest_safe_compliance": 0.0053
|
| 160 |
+
},
|
| 161 |
+
"agentcy": {
|
| 162 |
+
"plain_solved": {
|
| 163 |
+
"n": 60,
|
| 164 |
+
"a": 0.5667,
|
| 165 |
+
"b": 0.55,
|
| 166 |
+
"diff": 0.0167,
|
| 167 |
+
"ci95": [
|
| 168 |
+
-0.05,
|
| 169 |
+
0.1
|
| 170 |
+
],
|
| 171 |
+
"verdict": "TIE"
|
| 172 |
+
},
|
| 173 |
+
"native_solved": {
|
| 174 |
+
"n": 56,
|
| 175 |
+
"a": 0.5893,
|
| 176 |
+
"b": 0.3036,
|
| 177 |
+
"diff": 0.2857,
|
| 178 |
+
"ci95": [
|
| 179 |
+
0.1429,
|
| 180 |
+
0.4286
|
| 181 |
+
],
|
| 182 |
+
"verdict": "WIN"
|
| 183 |
+
},
|
| 184 |
+
"toolcall_exact": {
|
| 185 |
+
"n": 91,
|
| 186 |
+
"a": 0.967,
|
| 187 |
+
"b": 0.978,
|
| 188 |
+
"diff": -0.011,
|
| 189 |
+
"ci95": [
|
| 190 |
+
-0.033,
|
| 191 |
+
0.0
|
| 192 |
+
],
|
| 193 |
+
"verdict": "TIE"
|
| 194 |
+
},
|
| 195 |
+
"plain_reserved": {
|
| 196 |
+
"n": 45,
|
| 197 |
+
"a": 0.4889,
|
| 198 |
+
"b": 0.4889,
|
| 199 |
+
"diff": 0.0,
|
| 200 |
+
"ci95": [
|
| 201 |
+
-0.0889,
|
| 202 |
+
0.0889
|
| 203 |
+
],
|
| 204 |
+
"verdict": "TIE"
|
| 205 |
+
},
|
| 206 |
+
"plain_unseen": {
|
| 207 |
+
"n": 15,
|
| 208 |
+
"a": 0.8,
|
| 209 |
+
"b": 0.7333,
|
| 210 |
+
"diff": 0.0667,
|
| 211 |
+
"ci95": [
|
| 212 |
+
0.0,
|
| 213 |
+
0.2
|
| 214 |
+
],
|
| 215 |
+
"verdict": "TIE"
|
| 216 |
+
}
|
| 217 |
+
},
|
| 218 |
+
"g2": 0.11133333333333334,
|
| 219 |
+
"score": 0.11533333333333343
|
| 220 |
+
},
|
| 221 |
+
{
|
| 222 |
+
"label": "v2a-interp-w075",
|
| 223 |
+
"battery_tag": "v2a-interp-w075 (dev)",
|
| 224 |
+
"eligible": false,
|
| 225 |
+
"reasons": [
|
| 226 |
+
"toolcall exact -0.0330"
|
| 227 |
+
],
|
| 228 |
+
"families_used": [
|
| 229 |
+
"ctibench",
|
| 230 |
+
"cybermetric",
|
| 231 |
+
"secbench",
|
| 232 |
+
"seceval",
|
| 233 |
+
"secqa",
|
| 234 |
+
"vuln2"
|
| 235 |
+
],
|
| 236 |
+
"proxy": 0.7590499999999999,
|
| 237 |
+
"ref_proxy": 0.7532666666666668,
|
| 238 |
+
"families": {
|
| 239 |
+
"ctibench": [
|
| 240 |
+
0.5748,
|
| 241 |
+
0.5141
|
| 242 |
+
],
|
| 243 |
+
"cybermetric": [
|
| 244 |
+
0.89,
|
| 245 |
+
0.9075
|
| 246 |
+
],
|
| 247 |
+
"secbench": [
|
| 248 |
+
0.795,
|
| 249 |
+
0.8
|
| 250 |
+
],
|
| 251 |
+
"seceval": [
|
| 252 |
+
0.74,
|
| 253 |
+
0.74
|
| 254 |
+
],
|
| 255 |
+
"secqa": [
|
| 256 |
+
0.9927,
|
| 257 |
+
0.9927
|
| 258 |
+
],
|
| 259 |
+
"vuln2": [
|
| 260 |
+
0.5618,
|
| 261 |
+
0.5653
|
| 262 |
+
]
|
| 263 |
+
},
|
| 264 |
+
"g1": 0.11566666666666281,
|
| 265 |
+
"guards": {
|
| 266 |
+
"mmlu_sec": -0.0098,
|
| 267 |
+
"coding2": 0.0042,
|
| 268 |
+
"agent2": 0.025,
|
| 269 |
+
"xstest_safe_compliance": 0.0
|
| 270 |
+
},
|
| 271 |
+
"agentcy": {
|
| 272 |
+
"plain_solved": {
|
| 273 |
+
"n": 60,
|
| 274 |
+
"a": 0.5667,
|
| 275 |
+
"b": 0.55,
|
| 276 |
+
"diff": 0.0167,
|
| 277 |
+
"ci95": [
|
| 278 |
+
-0.0833,
|
| 279 |
+
0.1167
|
| 280 |
+
],
|
| 281 |
+
"verdict": "TIE"
|
| 282 |
+
},
|
| 283 |
+
"native_solved": {
|
| 284 |
+
"n": 53,
|
| 285 |
+
"a": 0.6226,
|
| 286 |
+
"b": 0.3208,
|
| 287 |
+
"diff": 0.3019,
|
| 288 |
+
"ci95": [
|
| 289 |
+
0.1509,
|
| 290 |
+
0.4528
|
| 291 |
+
],
|
| 292 |
+
"verdict": "WIN"
|
| 293 |
+
},
|
| 294 |
+
"toolcall_exact": {
|
| 295 |
+
"n": 91,
|
| 296 |
+
"a": 0.9451,
|
| 297 |
+
"b": 0.978,
|
| 298 |
+
"diff": -0.033,
|
| 299 |
+
"ci95": [
|
| 300 |
+
-0.0769,
|
| 301 |
+
0.0
|
| 302 |
+
],
|
| 303 |
+
"verdict": "TIE"
|
| 304 |
+
},
|
| 305 |
+
"plain_reserved": {
|
| 306 |
+
"n": 45,
|
| 307 |
+
"a": 0.5111,
|
| 308 |
+
"b": 0.4889,
|
| 309 |
+
"diff": 0.0222,
|
| 310 |
+
"ci95": [
|
| 311 |
+
-0.0889,
|
| 312 |
+
0.1333
|
| 313 |
+
],
|
| 314 |
+
"verdict": "TIE"
|
| 315 |
+
},
|
| 316 |
+
"plain_unseen": {
|
| 317 |
+
"n": 15,
|
| 318 |
+
"a": 0.7333,
|
| 319 |
+
"b": 0.7333,
|
| 320 |
+
"diff": 0.0,
|
| 321 |
+
"ci95": [
|
| 322 |
+
-0.2,
|
| 323 |
+
0.2
|
| 324 |
+
],
|
| 325 |
+
"verdict": "TIE"
|
| 326 |
+
}
|
| 327 |
+
},
|
| 328 |
+
"g2": 0.11133333333333334,
|
| 329 |
+
"score": 0.11349999999999807
|
| 330 |
+
},
|
| 331 |
+
{
|
| 332 |
+
"label": "v2c-final",
|
| 333 |
+
"battery_tag": "v2c-final (dev)",
|
| 334 |
+
"eligible": true,
|
| 335 |
+
"reasons": [],
|
| 336 |
+
"families_used": [
|
| 337 |
+
"ctibench",
|
| 338 |
+
"cybermetric",
|
| 339 |
+
"secbench",
|
| 340 |
+
"seceval",
|
| 341 |
+
"secqa",
|
| 342 |
+
"vuln2"
|
| 343 |
+
],
|
| 344 |
+
"proxy": 0.7581000000000001,
|
| 345 |
+
"ref_proxy": 0.7532666666666668,
|
| 346 |
+
"families": {
|
| 347 |
+
"ctibench": [
|
| 348 |
+
0.5792,
|
| 349 |
+
0.5141
|
| 350 |
+
],
|
| 351 |
+
"cybermetric": [
|
| 352 |
+
0.8825,
|
| 353 |
+
0.9075
|
| 354 |
+
],
|
| 355 |
+
"secbench": [
|
| 356 |
+
0.78,
|
| 357 |
+
0.8
|
| 358 |
+
],
|
| 359 |
+
"seceval": [
|
| 360 |
+
0.745,
|
| 361 |
+
0.74
|
| 362 |
+
],
|
| 363 |
+
"secqa": [
|
| 364 |
+
0.9927,
|
| 365 |
+
0.9927
|
| 366 |
+
],
|
| 367 |
+
"vuln2": [
|
| 368 |
+
0.5692,
|
| 369 |
+
0.5653
|
| 370 |
+
]
|
| 371 |
+
},
|
| 372 |
+
"g1": 0.09666666666666712,
|
| 373 |
+
"guards": {
|
| 374 |
+
"mmlu_sec": 0.0049,
|
| 375 |
+
"coding2": 0.0042,
|
| 376 |
+
"agent2": 0.15,
|
| 377 |
+
"xstest_safe_compliance": -0.0107
|
| 378 |
+
},
|
| 379 |
+
"agentcy": {
|
| 380 |
+
"plain_solved": {
|
| 381 |
+
"n": 60,
|
| 382 |
+
"a": 0.6333,
|
| 383 |
+
"b": 0.55,
|
| 384 |
+
"diff": 0.0833,
|
| 385 |
+
"ci95": [
|
| 386 |
+
0.0,
|
| 387 |
+
0.1833
|
| 388 |
+
],
|
| 389 |
+
"verdict": "TIE"
|
| 390 |
+
},
|
| 391 |
+
"native_solved": {
|
| 392 |
+
"n": 59,
|
| 393 |
+
"a": 0.6271,
|
| 394 |
+
"b": 0.2881,
|
| 395 |
+
"diff": 0.339,
|
| 396 |
+
"ci95": [
|
| 397 |
+
0.1864,
|
| 398 |
+
0.4915
|
| 399 |
+
],
|
| 400 |
+
"verdict": "WIN"
|
| 401 |
+
},
|
| 402 |
+
"toolcall_exact": {
|
| 403 |
+
"n": 91,
|
| 404 |
+
"a": 0.978,
|
| 405 |
+
"b": 0.978,
|
| 406 |
+
"diff": 0.0,
|
| 407 |
+
"ci95": [
|
| 408 |
+
-0.033,
|
| 409 |
+
0.033
|
| 410 |
+
],
|
| 411 |
+
"verdict": "TIE"
|
| 412 |
+
},
|
| 413 |
+
"plain_reserved": {
|
| 414 |
+
"n": 45,
|
| 415 |
+
"a": 0.5333,
|
| 416 |
+
"b": 0.4889,
|
| 417 |
+
"diff": 0.0444,
|
| 418 |
+
"ci95": [
|
| 419 |
+
-0.0444,
|
| 420 |
+
0.1556
|
| 421 |
+
],
|
| 422 |
+
"verdict": "TIE"
|
| 423 |
+
},
|
| 424 |
+
"plain_unseen": {
|
| 425 |
+
"n": 15,
|
| 426 |
+
"a": 0.9333,
|
| 427 |
+
"b": 0.7333,
|
| 428 |
+
"diff": 0.2,
|
| 429 |
+
"ci95": [
|
| 430 |
+
0.0,
|
| 431 |
+
0.4
|
| 432 |
+
],
|
| 433 |
+
"verdict": "TIE"
|
| 434 |
+
}
|
| 435 |
+
},
|
| 436 |
+
"g2": 0.5553333333333333,
|
| 437 |
+
"score": 0.32600000000000023
|
| 438 |
+
}
|
| 439 |
+
],
|
| 440 |
+
"chosen": null,
|
| 441 |
+
"chosen_battery_tag": null,
|
| 442 |
+
"chosen_gguf": null,
|
| 443 |
+
"rule": "Between-run rule (GOAL.md section 6.2): the pick is eligible iff no dev guard (mmlu_sec, coding2, agent2, xstest_safe_compliance) is more than 0.03 below the reference and the dev tool-call exact is within 0.03 of it. Score = mean(dev composite proxy gain / 0.05, agent-plain dev solved gain / 0.15). The highest-scoring eligible challenger replaces the incumbent iff its score >= incumbent score + margin (0.20); candidates within 0.02 of the best are tied, with a non-interpolated candidate preferred and ties broken by order of definition. Otherwise the incumbent is kept."
|
| 444 |
+
}
|
evidence/results/league_verdict.txt
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
VERDICT ours=ab9b-v2 (test) tier=test boot=2000 seed=0 cyber_composite(ours)=0.7389
|
| 2 |
+
composite comparison (diff = ours - league, paired bootstrap, win iff 95% lower bound > 0):
|
| 3 |
+
vs ab9b-r1 (test) <=10B ours 0.7389 theirs 0.7251 diff +0.0139 [-0.0056, +0.0328] TIE
|
| 4 |
+
vs Cyber-Ornith-9B (test) <=10B ours 0.7389 theirs 0.6474 diff +0.0915 [+0.0716, +0.1120] WIN
|
| 5 |
+
vs Foundation-Sec-8B-Instruct (test) <=10B ours 0.7389 theirs 0.6662 diff +0.0727 [+0.0501, +0.0950] WIN
|
| 6 |
+
vs Huihui-Qwen3.8-27B-abliterated (test) >10B ours 0.7389 theirs 0.7199 diff +0.0190 [-0.0019, +0.0399] TIE
|
| 7 |
+
vs Primus-Merged-8B (test) <=10B ours 0.7389 theirs 0.6218 diff +0.1172 [+0.0950, +0.1388] WIN
|
| 8 |
+
vs Qwen3.5-9B base (test) <=10B ours 0.7389 theirs 0.7190 diff +0.0199 [+0.0016, +0.0383] WIN
|
| 9 |
+
vs SecGPT-7B (test) <=10B ours 0.7389 theirs 0.6243 diff +0.1147 [+0.0936, +0.1365] WIN
|
| 10 |
+
W1 size-fair (beat every <=10B league model: win vs ['ab9b-r1 (test)', 'Cyber-Ornith-9B (test)', 'Foundation-Sec-8B-Instruct (test)', 'Primus-Merged-8B (test)', 'Qwen3.5-9B base (test)', 'SecGPT-7B (test)']): DOES NOT HOLD
|
| 11 |
+
W3 absolute (>= every league model of any size: no loss and point difference >= 0): HOLDS [strict reading, a win against all: no]
|
| 12 |
+
W2 speed-fair (highest quality_per_second of the whole league): DOES NOT HOLD
|
| 13 |
+
vs ab9b-r1 (test) qps ours 0.6323 theirs 0.5914 diff +0.0409 [-0.0523, +0.1755] TIE
|
| 14 |
+
vs Cyber-Ornith-9B (test) qps ours 0.6323 theirs 0.8760 diff -0.2437 [-0.3646, +0.0024] TIE
|
| 15 |
+
vs Foundation-Sec-8B-Instruct (test) qps ours 0.6323 theirs 1.4483 diff -0.8159 [-1.0777, -0.2690] LOSS
|
| 16 |
+
vs Huihui-Qwen3.8-27B-abliterated (test) qps ours 0.6323 theirs 0.3332 diff +0.2991 [+0.2058, +0.4323] WIN
|
| 17 |
+
vs Primus-Merged-8B (test) qps ours 0.6323 theirs 1.3864 diff -0.7540 [-1.1478, -0.2070] LOSS
|
| 18 |
+
vs Qwen3.5-9B base (test) qps ours 0.6323 theirs 0.6332 diff -0.0009 [-0.1268, +0.1761] TIE
|
| 19 |
+
vs SecGPT-7B (test) qps ours 0.6323 theirs 2.1020 diff -1.4697 [-1.7828, -1.1097] LOSS
|
| 20 |
+
guards (none more than 0.03 below ab9b-r1 (dev), dev tier): NOT DECIDABLE (missing values)
|
| 21 |
+
mmlu_sec not available
|
| 22 |
+
coding2 not available
|
| 23 |
+
agent2 not available
|
| 24 |
+
xstest_safe_compliance not available
|
| 25 |
+
|
| 26 |
+
CLAIM POLICY (GOAL.md section 1): 'wins any other cyber LLM' requires W1 AND W3 AND all guards.
|
| 27 |
+
NOT ALLOWED. State exactly: holds = ['W3']; does not hold = ['W1', 'W2']; not decidable = ['guards'] and nothing stronger.
|
evidence/results/v2a_accept_report.txt
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== agentcy: ab9b-v2 (test) vs ab9b-r1 (TEST, paired bootstrap)
|
| 2 |
+
paired comparison A=ab9b-v2 (test) B=ab9b-r1 (test) tier=test boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 3 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 4 |
+
agent-plain solved 0.5281 0.4831 +0.0449 [-0.0562, +0.1461] 89 TIE
|
| 5 |
+
agent-plain exact 0.5281 0.4831 +0.0449 [-0.0562, +0.1461] 89 TIE
|
| 6 |
+
agent-plain score 0.5281 0.4831 +0.0449 [-0.0562, +0.1461] 89 TIE
|
| 7 |
+
agent-plain solved[reserved] 0.4400 0.4667 -0.0267 [-0.1333, +0.0800] 75 TIE
|
| 8 |
+
agent-plain solved[unseen] 1.0000 0.5714 +0.4286 [+0.1429, +0.7143] 14 WIN
|
| 9 |
+
agent-native solved 0.4828 0.1724 +0.3103 [+0.1954, +0.4253] 87 WIN
|
| 10 |
+
agent-native exact 0.4828 0.1724 +0.3103 [+0.1954, +0.4253] 87 WIN
|
| 11 |
+
agent-native score 0.4828 0.1724 +0.3103 [+0.1954, +0.4253] 87 WIN
|
| 12 |
+
agent-native solved[reserved] 0.4133 0.1467 +0.2667 [+0.1467, +0.3867] 75 WIN
|
| 13 |
+
agent-native solved[unseen] 0.9167 0.3333 +0.5833 [+0.3333, +0.8333] 12 WIN
|
| 14 |
+
toolcall exact 0.8974 0.9231 -0.0256 [-0.0769, +0.0000] 39 TIE
|
| 15 |
+
|
| 16 |
+
=== agentcy: ab9b-v2 (test) vs base
|
| 17 |
+
paired comparison A=ab9b-v2 (test) B=Qwen3.5-9B base (test) tier=test boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 18 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 19 |
+
agent-plain solved 0.5281 0.2360 +0.2921 [+0.1798, +0.4045] 89 WIN
|
| 20 |
+
agent-plain exact 0.5281 0.2360 +0.2921 [+0.1798, +0.4045] 89 WIN
|
| 21 |
+
agent-plain score 0.5281 0.2360 +0.2921 [+0.1798, +0.4045] 89 WIN
|
| 22 |
+
agent-plain solved[reserved] 0.4400 0.1867 +0.2533 [+0.1333, +0.3733] 75 WIN
|
| 23 |
+
agent-plain solved[unseen] 1.0000 0.5000 +0.5000 [+0.2143, +0.7857] 14 WIN
|
| 24 |
+
agent-native solved 0.4944 0.1573 +0.3371 [+0.2360, +0.4494] 89 WIN
|
| 25 |
+
agent-native exact 0.4944 0.1573 +0.3371 [+0.2360, +0.4494] 89 WIN
|
| 26 |
+
agent-native score 0.4944 0.1573 +0.3371 [+0.2360, +0.4494] 89 WIN
|
| 27 |
+
agent-native solved[reserved] 0.4133 0.1467 +0.2667 [+0.1600, +0.3733] 75 WIN
|
| 28 |
+
agent-native solved[unseen] 0.9286 0.2143 +0.7143 [+0.5000, +0.9286] 14 WIN
|
| 29 |
+
toolcall exact 0.8974 0.9487 -0.0513 [-0.1282, +0.0000] 39 TIE
|
| 30 |
+
|
| 31 |
+
=== battery composite of ab9b-v2 (test)
|
| 32 |
+
|
| 33 |
+
cyber_composite (tier test): 0.7389 [0.721, 0.756] [FINAL]
|
| 34 |
+
F1 cybermetric 0.8966 ok
|
| 35 |
+
F2 seceval 0.7554 ok
|
| 36 |
+
F3 ctibench 0.5168 ok (mcq 0.705, rcm 0.763, vsp 0.812, ate 0.241, taa 0.062)
|
| 37 |
+
F4 secbench 0.8390 ok
|
| 38 |
+
F5 secqa 0.9315 ok
|
| 39 |
+
F6 cyberbench_jpm 0.6874 ok
|
| 40 |
+
F7 vuln2 0.5458 ok
|
| 41 |
+
guards (dev tier, never in the composite):
|
| 42 |
+
mmlu_sec -- missing
|
| 43 |
+
coding2 -- missing
|
| 44 |
+
agent2 -- missing
|
| 45 |
+
xstest_safe_compliance -- missing
|
| 46 |
+
|
| 47 |
+
=== battery: ab9b-v2 (test) vs ab9b-r1 (TEST, paired)
|
| 48 |
+
paired comparison A=ab9b-v2 (test) B=ab9b-r1 (test) tier=test boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 49 |
+
task A B diff 95% CI of diff n_shared verdict
|
| 50 |
+
cybermetric 0.8966 0.9000 -0.0034 [-0.0169, +0.0102] 590 TIE
|
| 51 |
+
seceval 0.7554 0.7492 +0.0061 [-0.0153, +0.0260] 654 TIE
|
| 52 |
+
cti-mcq 0.7052 0.7107 -0.0055 [-0.0317, +0.0220] 726 TIE
|
| 53 |
+
cti-rcm 0.7629 0.6079 +0.1550 [+0.1125, +0.2036] 329 WIN
|
| 54 |
+
cti-vsp 0.8121 0.7152 +0.0969 [+0.0730, +0.1216] 334 WIN
|
| 55 |
+
cti-ate 0.2415 0.3060 -0.0645 [-0.1431, +0.0120] 22 TIE
|
| 56 |
+
cti-taa 0.0625 0.0625 +0.0000 [+0.0000, +0.0000] 16 TIE
|
| 57 |
+
secbench 0.8390 0.8341 +0.0049 [-0.0244, +0.0341] 205 TIE
|
| 58 |
+
secqa 0.9315 0.9452 -0.0137 [-0.0411, +0.0000] 73 TIE
|
| 59 |
+
cyber_composite 0.7389 0.7251 +0.0139 [-0.0056, +0.0328] TIE
|
| 60 |
+
note: F6 cyberbench_jpm and F7 vuln2 enter the composite bootstrap as independent normal noise drawn from their own 95% CIs (conservative: their per-item pairing is not used); F1..F5 are paired on shared item ids
|
| 61 |
+
|
| 62 |
+
=== league verdict (W1/W2/W3 + guards + allowed sentence)
|
| 63 |
+
VERDICT ours=ab9b-v2 (test) tier=test boot=2000 seed=0 cyber_composite(ours)=0.7389
|
| 64 |
+
composite comparison (diff = ours - league, paired bootstrap, win iff 95% lower bound > 0):
|
| 65 |
+
vs ab9b-r1 (test) <=10B ours 0.7389 theirs 0.7251 diff +0.0139 [-0.0056, +0.0328] TIE
|
| 66 |
+
vs Cyber-Ornith-9B (test) <=10B ours 0.7389 theirs 0.6474 diff +0.0915 [+0.0716, +0.1120] WIN
|
| 67 |
+
vs Foundation-Sec-8B-Instruct (test) <=10B ours 0.7389 theirs 0.6662 diff +0.0727 [+0.0501, +0.0950] WIN
|
| 68 |
+
vs Primus-Merged-8B (test) <=10B ours 0.7389 theirs 0.6218 diff +0.1172 [+0.0950, +0.1388] WIN
|
| 69 |
+
vs Qwen3.5-9B base (test) <=10B ours 0.7389 theirs 0.7190 diff +0.0199 [+0.0016, +0.0383] WIN
|
| 70 |
+
vs SecGPT-7B (test) <=10B ours 0.7389 theirs 0.6243 diff +0.1147 [+0.0936, +0.1365] WIN
|
| 71 |
+
W1 size-fair (beat every <=10B league model: win vs ['ab9b-r1 (test)', 'Cyber-Ornith-9B (test)', 'Foundation-Sec-8B-Instruct (test)', 'Primus-Merged-8B (test)', 'Qwen3.5-9B base (test)', 'SecGPT-7B (test)']): DOES NOT HOLD
|
| 72 |
+
W3 absolute (>= every league model of any size: no loss and point difference >= 0): HOLDS [strict reading, a win against all: no]
|
| 73 |
+
W2 speed-fair (highest quality_per_second of the whole league): DOES NOT HOLD
|
| 74 |
+
vs ab9b-r1 (test) qps ours 0.6323 theirs 0.5914 diff +0.0409 [-0.0523, +0.1755] TIE
|
| 75 |
+
vs Cyber-Ornith-9B (test) qps ours 0.6323 theirs 0.8760 diff -0.2437 [-0.3646, +0.0024] TIE
|
| 76 |
+
vs Foundation-Sec-8B-Instruct (test) qps ours 0.6323 theirs 1.4483 diff -0.8159 [-1.0777, -0.2690] LOSS
|
| 77 |
+
vs Primus-Merged-8B (test) qps ours 0.6323 theirs 1.3864 diff -0.7540 [-1.1546, -0.1925] LOSS
|
| 78 |
+
vs Qwen3.5-9B base (test) qps ours 0.6323 theirs 0.6332 diff -0.0009 [-0.1228, +0.1660] TIE
|
| 79 |
+
vs SecGPT-7B (test) qps ours 0.6323 theirs 2.1020 diff -1.4697 [-1.7663, -1.1151] LOSS
|
| 80 |
+
guards (none more than 0.03 below ab9b-r1 (dev), dev tier): NOT DECIDABLE (missing values)
|
| 81 |
+
mmlu_sec not available
|
| 82 |
+
coding2 not available
|
| 83 |
+
agent2 not available
|
| 84 |
+
xstest_safe_compliance not available
|
| 85 |
+
|
| 86 |
+
CLAIM POLICY (GOAL.md section 1): 'wins any other cyber LLM' requires W1 AND W3 AND all guards.
|
| 87 |
+
NOT ALLOWED. State exactly: holds = ['W3']; does not hold = ['W1', 'W2']; not decidable = ['guards'] and nothing stronger.
|
| 88 |
+
|
| 89 |
+
=== own harness b1 timing, ab9b-v2 (test) vs ab9b-r1 (median wall seconds, quality per second)
|
| 90 |
+
ab9b-v2 (test): - composite_dev: **0.756** [0.663, 0.837] - median_wall_s_dev: 1.169
|
| 91 |
+
ab9b-r1 (test): - composite_dev: **0.717** [0.624, 0.798] - median_wall_s_dev: 1.226
|
evidence/results/v2a_criteria.txt
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
C1 cyber_composite (test): 0.7389 vs 0.7251 diff +0.0139 [-0.0056, +0.0328] need diff >= +0.05 and lower bound > 0 -> FAIL
|
| 2 |
+
cybermetric 0.8966 vs 0.9000 diff -0.0034 [-0.0169, +0.0102] TIE
|
| 3 |
+
seceval 0.7554 vs 0.7492 diff +0.0061 [-0.0153, +0.0260] TIE
|
| 4 |
+
cti-mcq 0.7052 vs 0.7107 diff -0.0055 [-0.0317, +0.0220] TIE
|
| 5 |
+
cti-rcm 0.7629 vs 0.6079 diff +0.1550 [+0.1125, +0.2036] WIN
|
| 6 |
+
cti-vsp 0.8121 vs 0.7152 diff +0.0969 [+0.0730, +0.1216] WIN
|
| 7 |
+
cti-ate 0.2415 vs 0.3060 diff -0.0645 [-0.1431, +0.0120] TIE
|
| 8 |
+
cti-taa 0.0625 vs 0.0625 diff +0.0000 [+0.0000, +0.0000] TIE
|
| 9 |
+
secbench 0.8390 vs 0.8341 diff +0.0049 [-0.0244, +0.0341] TIE
|
| 10 |
+
secqa 0.9315 vs 0.9452 diff -0.0137 [-0.0411, +0.0000] TIE
|
| 11 |
+
C2 agentcy plain solved (test): 0.5281 vs 0.4831 diff +0.0449 [-0.0562, +0.1461] n=89 need diff >= +0.15 and lower bound > 0 -> FAIL
|
| 12 |
+
native tool-call exact (test): 0.8974 vs 0.9231 n=39 need not below r1 -> FAIL
|
| 13 |
+
agent2 guard (dev) delta +0.0500 need >= -0.03 -> PASS
|
| 14 |
+
(info) native agent loop solved (test): 0.4828 vs 0.1724 diff +0.3103 [+0.1954, +0.4253]
|
| 15 |
+
C3 median wall s per task (batch-1, own harness): 1.169 vs 1.226 ratio 0.953 need <= 1.10 -> PASS
|
| 16 |
+
agentcy plain completion tokens per solved task: 139.6 vs 170.5 need not above r1 -> PASS
|
| 17 |
+
MTP head tensors identical to the base: 15 identical, 0 different -> PASS
|
| 18 |
+
C4 guard mmlu_sec (dev): delta -0.0049 need >= -0.03 -> PASS
|
| 19 |
+
C4 guard coding2 (dev): delta -0.0167 need >= -0.03 -> PASS
|
| 20 |
+
C4 guard xstest_safe_compliance (dev): delta +0.0000 need >= -0.03 -> PASS
|
| 21 |
+
C5 xstest unsafe-refusal (dev): 0.1791 vs r1 0.1418 (lower 95% bound 0.0927) need >= lower bound -> PASS (+ data are defensive/benign only by construction)
|
| 22 |
+
|
| 23 |
+
SUMMARY ab9b-v2 (test) vs ab9b-r1 (test): hold ['C3', 'C4', 'C5']; fail ['C1', 'C2']; not decidable none => not accepted: state exactly which hold
|
evidence/results/v2a_pick.json
ADDED
|
@@ -0,0 +1,336 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "v2a",
|
| 3 |
+
"ref": "ab9b-r1 (dev)",
|
| 4 |
+
"rule": "eligible = no guard (mmlu_sec, coding2, agent2, xstest_safe_compliance) more than 0.03 below the reference, and agentcy dev tool-call exact not more than 0.03 below it.\nScore = mean( dev composite proxy gain / 0.05 , agentcy dev plain solved gain / 0.15 ); the proxy is the mean over the dev families available for BOTH the candidate and the reference\n(F1-F5 and F7; F6 CyberBench has no dev tier). Highest score among the eligible wins; every eligible candidate within 0.02 of the best counts as tied and the LATEST checkpoint of the tied ones is picked.\nNo eligible candidate -> chosen = null (no automatic TEST scoring).",
|
| 5 |
+
"rows": [
|
| 6 |
+
{
|
| 7 |
+
"tag": "final",
|
| 8 |
+
"battery_tag": "v2a-final (dev)",
|
| 9 |
+
"gguf": "models/gguf/v2a-final-Q4_K_M.gguf",
|
| 10 |
+
"eligible": true,
|
| 11 |
+
"reasons": [],
|
| 12 |
+
"families_used": [
|
| 13 |
+
"ctibench",
|
| 14 |
+
"cybermetric",
|
| 15 |
+
"secbench",
|
| 16 |
+
"seceval",
|
| 17 |
+
"secqa",
|
| 18 |
+
"vuln2"
|
| 19 |
+
],
|
| 20 |
+
"proxy": 0.7654,
|
| 21 |
+
"ref_proxy": 0.7532666666666668,
|
| 22 |
+
"families": {
|
| 23 |
+
"ctibench": [
|
| 24 |
+
0.5739,
|
| 25 |
+
0.5141
|
| 26 |
+
],
|
| 27 |
+
"cybermetric": [
|
| 28 |
+
0.885,
|
| 29 |
+
0.9075
|
| 30 |
+
],
|
| 31 |
+
"secbench": [
|
| 32 |
+
0.7925,
|
| 33 |
+
0.8
|
| 34 |
+
],
|
| 35 |
+
"seceval": [
|
| 36 |
+
0.75,
|
| 37 |
+
0.74
|
| 38 |
+
],
|
| 39 |
+
"secqa": [
|
| 40 |
+
0.9927,
|
| 41 |
+
0.9927
|
| 42 |
+
],
|
| 43 |
+
"vuln2": [
|
| 44 |
+
0.5983,
|
| 45 |
+
0.5653
|
| 46 |
+
]
|
| 47 |
+
},
|
| 48 |
+
"g1": 0.24266666666666437,
|
| 49 |
+
"guards": {
|
| 50 |
+
"mmlu_sec": -0.0049,
|
| 51 |
+
"coding2": 0.0042,
|
| 52 |
+
"agent2": 0.05,
|
| 53 |
+
"xstest_safe_compliance": 0.0
|
| 54 |
+
},
|
| 55 |
+
"agentcy": {
|
| 56 |
+
"plain_solved": {
|
| 57 |
+
"n": 60,
|
| 58 |
+
"a": 0.6167,
|
| 59 |
+
"b": 0.55,
|
| 60 |
+
"diff": 0.0667,
|
| 61 |
+
"ci95": [
|
| 62 |
+
-0.0333,
|
| 63 |
+
0.1833
|
| 64 |
+
],
|
| 65 |
+
"verdict": "TIE"
|
| 66 |
+
},
|
| 67 |
+
"native_solved": {
|
| 68 |
+
"n": 59,
|
| 69 |
+
"a": 0.6102,
|
| 70 |
+
"b": 0.2881,
|
| 71 |
+
"diff": 0.322,
|
| 72 |
+
"ci95": [
|
| 73 |
+
0.1695,
|
| 74 |
+
0.4576
|
| 75 |
+
],
|
| 76 |
+
"verdict": "WIN"
|
| 77 |
+
},
|
| 78 |
+
"toolcall_exact": {
|
| 79 |
+
"n": 91,
|
| 80 |
+
"a": 0.956,
|
| 81 |
+
"b": 0.978,
|
| 82 |
+
"diff": -0.022,
|
| 83 |
+
"ci95": [
|
| 84 |
+
-0.0549,
|
| 85 |
+
0.0
|
| 86 |
+
],
|
| 87 |
+
"verdict": "TIE"
|
| 88 |
+
},
|
| 89 |
+
"plain_reserved": {
|
| 90 |
+
"n": 45,
|
| 91 |
+
"a": 0.5333,
|
| 92 |
+
"b": 0.4889,
|
| 93 |
+
"diff": 0.0444,
|
| 94 |
+
"ci95": [
|
| 95 |
+
-0.0667,
|
| 96 |
+
0.1556
|
| 97 |
+
],
|
| 98 |
+
"verdict": "TIE"
|
| 99 |
+
},
|
| 100 |
+
"plain_unseen": {
|
| 101 |
+
"n": 15,
|
| 102 |
+
"a": 0.8667,
|
| 103 |
+
"b": 0.7333,
|
| 104 |
+
"diff": 0.1333,
|
| 105 |
+
"ci95": [
|
| 106 |
+
-0.1333,
|
| 107 |
+
0.4
|
| 108 |
+
],
|
| 109 |
+
"verdict": "TIE"
|
| 110 |
+
}
|
| 111 |
+
},
|
| 112 |
+
"g2": 0.44466666666666665,
|
| 113 |
+
"score": 0.3436666666666655
|
| 114 |
+
},
|
| 115 |
+
{
|
| 116 |
+
"tag": "ck1000",
|
| 117 |
+
"battery_tag": "v2a-ck1000 (dev)",
|
| 118 |
+
"gguf": "models/gguf/v2a-ck1000-Q4_K_M.gguf",
|
| 119 |
+
"eligible": true,
|
| 120 |
+
"reasons": [],
|
| 121 |
+
"families_used": [
|
| 122 |
+
"ctibench",
|
| 123 |
+
"cybermetric",
|
| 124 |
+
"secbench",
|
| 125 |
+
"seceval",
|
| 126 |
+
"secqa",
|
| 127 |
+
"vuln2"
|
| 128 |
+
],
|
| 129 |
+
"proxy": 0.7668499999999999,
|
| 130 |
+
"ref_proxy": 0.7532666666666668,
|
| 131 |
+
"families": {
|
| 132 |
+
"ctibench": [
|
| 133 |
+
0.5753,
|
| 134 |
+
0.5141
|
| 135 |
+
],
|
| 136 |
+
"cybermetric": [
|
| 137 |
+
0.8825,
|
| 138 |
+
0.9075
|
| 139 |
+
],
|
| 140 |
+
"secbench": [
|
| 141 |
+
0.7925,
|
| 142 |
+
0.8
|
| 143 |
+
],
|
| 144 |
+
"seceval": [
|
| 145 |
+
0.7575,
|
| 146 |
+
0.74
|
| 147 |
+
],
|
| 148 |
+
"secqa": [
|
| 149 |
+
0.9927,
|
| 150 |
+
0.9927
|
| 151 |
+
],
|
| 152 |
+
"vuln2": [
|
| 153 |
+
0.6006,
|
| 154 |
+
0.5653
|
| 155 |
+
]
|
| 156 |
+
},
|
| 157 |
+
"g1": 0.2716666666666634,
|
| 158 |
+
"guards": {
|
| 159 |
+
"mmlu_sec": -0.0147,
|
| 160 |
+
"coding2": -0.0042,
|
| 161 |
+
"agent2": 0.075,
|
| 162 |
+
"xstest_safe_compliance": 0.0
|
| 163 |
+
},
|
| 164 |
+
"agentcy": {
|
| 165 |
+
"plain_solved": {
|
| 166 |
+
"n": 60,
|
| 167 |
+
"a": 0.6167,
|
| 168 |
+
"b": 0.55,
|
| 169 |
+
"diff": 0.0667,
|
| 170 |
+
"ci95": [
|
| 171 |
+
-0.0333,
|
| 172 |
+
0.1833
|
| 173 |
+
],
|
| 174 |
+
"verdict": "TIE"
|
| 175 |
+
},
|
| 176 |
+
"native_solved": {
|
| 177 |
+
"n": 59,
|
| 178 |
+
"a": 0.5593,
|
| 179 |
+
"b": 0.2881,
|
| 180 |
+
"diff": 0.2712,
|
| 181 |
+
"ci95": [
|
| 182 |
+
0.1356,
|
| 183 |
+
0.4068
|
| 184 |
+
],
|
| 185 |
+
"verdict": "WIN"
|
| 186 |
+
},
|
| 187 |
+
"toolcall_exact": {
|
| 188 |
+
"n": 91,
|
| 189 |
+
"a": 0.956,
|
| 190 |
+
"b": 0.978,
|
| 191 |
+
"diff": -0.022,
|
| 192 |
+
"ci95": [
|
| 193 |
+
-0.0549,
|
| 194 |
+
0.0
|
| 195 |
+
],
|
| 196 |
+
"verdict": "TIE"
|
| 197 |
+
},
|
| 198 |
+
"plain_reserved": {
|
| 199 |
+
"n": 45,
|
| 200 |
+
"a": 0.5333,
|
| 201 |
+
"b": 0.4889,
|
| 202 |
+
"diff": 0.0444,
|
| 203 |
+
"ci95": [
|
| 204 |
+
-0.0667,
|
| 205 |
+
0.1556
|
| 206 |
+
],
|
| 207 |
+
"verdict": "TIE"
|
| 208 |
+
},
|
| 209 |
+
"plain_unseen": {
|
| 210 |
+
"n": 15,
|
| 211 |
+
"a": 0.8667,
|
| 212 |
+
"b": 0.7333,
|
| 213 |
+
"diff": 0.1333,
|
| 214 |
+
"ci95": [
|
| 215 |
+
-0.1333,
|
| 216 |
+
0.4
|
| 217 |
+
],
|
| 218 |
+
"verdict": "TIE"
|
| 219 |
+
}
|
| 220 |
+
},
|
| 221 |
+
"g2": 0.44466666666666665,
|
| 222 |
+
"score": 0.358166666666665
|
| 223 |
+
},
|
| 224 |
+
{
|
| 225 |
+
"tag": "ck900",
|
| 226 |
+
"battery_tag": "v2a-ck900 (dev)",
|
| 227 |
+
"gguf": "models/gguf/ab9b-v2-Q4_K_M.gguf",
|
| 228 |
+
"eligible": true,
|
| 229 |
+
"reasons": [],
|
| 230 |
+
"families_used": [
|
| 231 |
+
"ctibench",
|
| 232 |
+
"cybermetric",
|
| 233 |
+
"secbench",
|
| 234 |
+
"seceval",
|
| 235 |
+
"secqa",
|
| 236 |
+
"vuln2"
|
| 237 |
+
],
|
| 238 |
+
"proxy": 0.76485,
|
| 239 |
+
"ref_proxy": 0.7532666666666668,
|
| 240 |
+
"families": {
|
| 241 |
+
"ctibench": [
|
| 242 |
+
0.5748,
|
| 243 |
+
0.5141
|
| 244 |
+
],
|
| 245 |
+
"cybermetric": [
|
| 246 |
+
0.885,
|
| 247 |
+
0.9075
|
| 248 |
+
],
|
| 249 |
+
"secbench": [
|
| 250 |
+
0.79,
|
| 251 |
+
0.8
|
| 252 |
+
],
|
| 253 |
+
"seceval": [
|
| 254 |
+
0.735,
|
| 255 |
+
0.74
|
| 256 |
+
],
|
| 257 |
+
"secqa": [
|
| 258 |
+
0.9927,
|
| 259 |
+
0.9927
|
| 260 |
+
],
|
| 261 |
+
"vuln2": [
|
| 262 |
+
0.6116,
|
| 263 |
+
0.5653
|
| 264 |
+
]
|
| 265 |
+
},
|
| 266 |
+
"g1": 0.23166666666666558,
|
| 267 |
+
"guards": {
|
| 268 |
+
"mmlu_sec": -0.0049,
|
| 269 |
+
"coding2": -0.0167,
|
| 270 |
+
"agent2": 0.05,
|
| 271 |
+
"xstest_safe_compliance": 0.0
|
| 272 |
+
},
|
| 273 |
+
"agentcy": {
|
| 274 |
+
"plain_solved": {
|
| 275 |
+
"n": 60,
|
| 276 |
+
"a": 0.6333,
|
| 277 |
+
"b": 0.55,
|
| 278 |
+
"diff": 0.0833,
|
| 279 |
+
"ci95": [
|
| 280 |
+
-0.0333,
|
| 281 |
+
0.2
|
| 282 |
+
],
|
| 283 |
+
"verdict": "TIE"
|
| 284 |
+
},
|
| 285 |
+
"native_solved": {
|
| 286 |
+
"n": 59,
|
| 287 |
+
"a": 0.5932,
|
| 288 |
+
"b": 0.2881,
|
| 289 |
+
"diff": 0.3051,
|
| 290 |
+
"ci95": [
|
| 291 |
+
0.1695,
|
| 292 |
+
0.4407
|
| 293 |
+
],
|
| 294 |
+
"verdict": "WIN"
|
| 295 |
+
},
|
| 296 |
+
"toolcall_exact": {
|
| 297 |
+
"n": 91,
|
| 298 |
+
"a": 0.956,
|
| 299 |
+
"b": 0.978,
|
| 300 |
+
"diff": -0.022,
|
| 301 |
+
"ci95": [
|
| 302 |
+
-0.0549,
|
| 303 |
+
0.0
|
| 304 |
+
],
|
| 305 |
+
"verdict": "TIE"
|
| 306 |
+
},
|
| 307 |
+
"plain_reserved": {
|
| 308 |
+
"n": 45,
|
| 309 |
+
"a": 0.5111,
|
| 310 |
+
"b": 0.4889,
|
| 311 |
+
"diff": 0.0222,
|
| 312 |
+
"ci95": [
|
| 313 |
+
-0.1111,
|
| 314 |
+
0.1556
|
| 315 |
+
],
|
| 316 |
+
"verdict": "TIE"
|
| 317 |
+
},
|
| 318 |
+
"plain_unseen": {
|
| 319 |
+
"n": 15,
|
| 320 |
+
"a": 1.0,
|
| 321 |
+
"b": 0.7333,
|
| 322 |
+
"diff": 0.2667,
|
| 323 |
+
"ci95": [
|
| 324 |
+
0.0667,
|
| 325 |
+
0.4667
|
| 326 |
+
],
|
| 327 |
+
"verdict": "WIN"
|
| 328 |
+
}
|
| 329 |
+
},
|
| 330 |
+
"g2": 0.5553333333333333,
|
| 331 |
+
"score": 0.39349999999999946
|
| 332 |
+
}
|
| 333 |
+
],
|
| 334 |
+
"chosen": "ck900",
|
| 335 |
+
"chosen_gguf": "models/gguf/ab9b-v2-Q4_K_M.gguf"
|
| 336 |
+
}
|
evidence/results/v2b_dev_summary.txt
ADDED
|
@@ -0,0 +1,166 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
candidates: final: ck700: ck600:
|
| 2 |
+
=== v2b-final
|
| 3 |
+
|
| 4 |
+
cyber_composite (tier test): NOT AVAILABLE [not final]
|
| 5 |
+
F1 cybermetric -- incomplete
|
| 6 |
+
F2 seceval -- incomplete
|
| 7 |
+
F3 ctibench -- incomplete
|
| 8 |
+
F4 secbench -- incomplete
|
| 9 |
+
F5 secqa -- incomplete
|
| 10 |
+
F6 cyberbench_jpm -- missing
|
| 11 |
+
F7 vuln2 -- missing
|
| 12 |
+
guards (dev tier, never in the composite):
|
| 13 |
+
mmlu_sec 0.7696 ok vs ref 0.7892: -0.0196
|
| 14 |
+
coding2 0.7406 ok vs ref 0.7615: -0.0209
|
| 15 |
+
agent2 0.8500 ok vs ref 0.6750: +0.1750
|
| 16 |
+
xstest_safe_compliance 0.9947 ok vs ref 0.9947: +0.0000
|
| 17 |
+
--- agentcy DEV (cap 60) vs ab9b-r1, paired
|
| 18 |
+
paired comparison A=v2b-final (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 19 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 20 |
+
agent-plain solved 0.6833 0.5500 +0.1333 [+0.0333, +0.2333] 60 WIN
|
| 21 |
+
agent-plain exact 0.6833 0.5500 +0.1333 [+0.0333, +0.2333] 60 WIN
|
| 22 |
+
agent-plain score 0.6833 0.5500 +0.1333 [+0.0333, +0.2333] 60 WIN
|
| 23 |
+
agent-plain solved[reserved] 0.5778 0.4889 +0.0889 [+0.0000, +0.2000] 45 TIE
|
| 24 |
+
agent-plain solved[unseen] 1.0000 0.7333 +0.2667 [+0.0667, +0.5333] 15 WIN
|
| 25 |
+
agent-native solved 0.6610 0.2881 +0.3729 [+0.2203, +0.5254] 59 WIN
|
| 26 |
+
agent-native exact 0.6610 0.2881 +0.3729 [+0.2203, +0.5254] 59 WIN
|
| 27 |
+
agent-native score 0.6610 0.2881 +0.3729 [+0.2203, +0.5254] 59 WIN
|
| 28 |
+
agent-native solved[reserved] 0.5682 0.2727 +0.2955 [+0.1136, +0.4773] 44 WIN
|
| 29 |
+
agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8667] 15 WIN
|
| 30 |
+
toolcall exact 0.8462 0.9780 -0.1319 [-0.2088, -0.0659] 91 LOSS
|
| 31 |
+
--- battery DEV vs v2a-ck900 (dev), paired
|
| 32 |
+
paired comparison A=v2b-final (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 33 |
+
task A B diff 95% CI of diff n_shared verdict
|
| 34 |
+
cybermetric 0.8800 0.8850 -0.0050 [-0.0225, +0.0125] 400 TIE
|
| 35 |
+
seceval 0.7525 0.7350 +0.0175 [+0.0000, +0.0400] 400 TIE
|
| 36 |
+
cti-mcq 0.6750 0.6775 -0.0025 [-0.0300, +0.0225] 400 TIE
|
| 37 |
+
cti-rcm 0.7500 0.7450 +0.0050 [-0.0150, +0.0250] 400 TIE
|
| 38 |
+
cti-vsp 0.8387 0.8100 +0.0287 [+0.0159, +0.0419] 400 WIN
|
| 39 |
+
cti-ate 0.3712 0.2296 +0.1416 [+0.0795, +0.2056] 38 WIN
|
| 40 |
+
cti-taa 0.2941 0.4118 -0.1176 [-0.2647, +0.0000] 34 TIE
|
| 41 |
+
secbench 0.7800 0.7900 -0.0100 [-0.0300, +0.0100] 400 TIE
|
| 42 |
+
secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
|
| 43 |
+
--- agentcy DEV vs v2a-ck900 (dev), paired
|
| 44 |
+
paired comparison A=v2b-final (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 45 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 46 |
+
agent-plain solved 0.6833 0.6333 +0.0500 [+0.0000, +0.1167] 60 TIE
|
| 47 |
+
agent-plain exact 0.6833 0.6333 +0.0500 [+0.0000, +0.1167] 60 TIE
|
| 48 |
+
agent-plain score 0.6833 0.6333 +0.0500 [+0.0000, +0.1167] 60 TIE
|
| 49 |
+
agent-plain solved[reserved] 0.5778 0.5111 +0.0667 [+0.0000, +0.1333] 45 TIE
|
| 50 |
+
agent-plain solved[unseen] 1.0000 1.0000 +0.0000 [+0.0000, +0.0000] 15 TIE
|
| 51 |
+
agent-native solved 0.6500 0.5833 +0.0667 [-0.0167, +0.1667] 60 TIE
|
| 52 |
+
agent-native exact 0.6500 0.5833 +0.0667 [-0.0167, +0.1667] 60 TIE
|
| 53 |
+
agent-native score 0.6500 0.5833 +0.0667 [-0.0167, +0.1667] 60 TIE
|
| 54 |
+
agent-native solved[reserved] 0.5556 0.5111 +0.0444 [-0.0667, +0.1556] 45 TIE
|
| 55 |
+
agent-native solved[unseen] 0.9333 0.8000 +0.1333 [+0.0000, +0.3333] 15 TIE
|
| 56 |
+
toolcall exact 0.8462 0.9560 -0.1099 [-0.1868, -0.0440] 91 LOSS
|
| 57 |
+
=== v2b-ck700
|
| 58 |
+
|
| 59 |
+
cyber_composite (tier test): NOT AVAILABLE [not final]
|
| 60 |
+
F1 cybermetric -- incomplete
|
| 61 |
+
F2 seceval -- incomplete
|
| 62 |
+
F3 ctibench -- incomplete
|
| 63 |
+
F4 secbench -- incomplete
|
| 64 |
+
F5 secqa -- incomplete
|
| 65 |
+
F6 cyberbench_jpm -- missing
|
| 66 |
+
F7 vuln2 -- missing
|
| 67 |
+
guards (dev tier, never in the composite):
|
| 68 |
+
mmlu_sec 0.7794 ok vs ref 0.7892: -0.0098
|
| 69 |
+
coding2 0.7573 ok vs ref 0.7615: -0.0042
|
| 70 |
+
agent2 0.8000 ok vs ref 0.6750: +0.1250
|
| 71 |
+
xstest_safe_compliance 0.9947 ok vs ref 0.9947: +0.0000
|
| 72 |
+
--- agentcy DEV (cap 60) vs ab9b-r1, paired
|
| 73 |
+
paired comparison A=v2b-ck700 (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 74 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 75 |
+
agent-plain solved 0.6667 0.5500 +0.1167 [+0.0333, +0.2167] 60 WIN
|
| 76 |
+
agent-plain exact 0.6667 0.5500 +0.1167 [+0.0333, +0.2167] 60 WIN
|
| 77 |
+
agent-plain score 0.6667 0.5500 +0.1167 [+0.0333, +0.2167] 60 WIN
|
| 78 |
+
agent-plain solved[reserved] 0.5556 0.4889 +0.0667 [-0.0222, +0.1556] 45 TIE
|
| 79 |
+
agent-plain solved[unseen] 1.0000 0.7333 +0.2667 [+0.0667, +0.5333] 15 WIN
|
| 80 |
+
agent-native solved 0.5789 0.2982 +0.2807 [+0.1228, +0.4386] 57 WIN
|
| 81 |
+
agent-native exact 0.5789 0.2982 +0.2807 [+0.1228, +0.4386] 57 WIN
|
| 82 |
+
agent-native score 0.5789 0.2982 +0.2807 [+0.1228, +0.4386] 57 WIN
|
| 83 |
+
agent-native solved[reserved] 0.4524 0.2857 +0.1667 [-0.0238, +0.3571] 42 TIE
|
| 84 |
+
agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8000] 15 WIN
|
| 85 |
+
toolcall exact 0.8462 0.9780 -0.1319 [-0.2088, -0.0659] 91 LOSS
|
| 86 |
+
--- battery DEV vs v2a-ck900 (dev), paired
|
| 87 |
+
paired comparison A=v2b-ck700 (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 88 |
+
task A B diff 95% CI of diff n_shared verdict
|
| 89 |
+
cybermetric 0.8750 0.8850 -0.0100 [-0.0300, +0.0100] 400 TIE
|
| 90 |
+
seceval 0.7425 0.7350 +0.0075 [-0.0125, +0.0300] 400 TIE
|
| 91 |
+
cti-mcq 0.6600 0.6775 -0.0175 [-0.0450, +0.0100] 400 TIE
|
| 92 |
+
cti-rcm 0.7425 0.7450 -0.0025 [-0.0200, +0.0150] 400 TIE
|
| 93 |
+
cti-vsp 0.8428 0.8100 +0.0328 [+0.0206, +0.0456] 400 WIN
|
| 94 |
+
cti-ate 0.4006 0.2296 +0.1709 [+0.1030, +0.2361] 38 WIN
|
| 95 |
+
cti-taa 0.2941 0.4118 -0.1176 [-0.2647, +0.0000] 34 TIE
|
| 96 |
+
secbench 0.7775 0.7900 -0.0125 [-0.0300, +0.0050] 400 TIE
|
| 97 |
+
secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
|
| 98 |
+
--- agentcy DEV vs v2a-ck900 (dev), paired
|
| 99 |
+
paired comparison A=v2b-ck700 (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 100 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 101 |
+
agent-plain solved 0.6667 0.6333 +0.0333 [-0.0333, +0.1000] 60 TIE
|
| 102 |
+
agent-plain exact 0.6667 0.6333 +0.0333 [-0.0333, +0.1000] 60 TIE
|
| 103 |
+
agent-plain score 0.6667 0.6333 +0.0333 [-0.0333, +0.1000] 60 TIE
|
| 104 |
+
agent-plain solved[reserved] 0.5556 0.5111 +0.0444 [-0.0444, +0.1333] 45 TIE
|
| 105 |
+
agent-plain solved[unseen] 1.0000 1.0000 +0.0000 [+0.0000, +0.0000] 15 TIE
|
| 106 |
+
agent-native solved 0.5690 0.5690 +0.0000 [-0.1034, +0.1034] 58 TIE
|
| 107 |
+
agent-native exact 0.5690 0.5690 +0.0000 [-0.1034, +0.1034] 58 TIE
|
| 108 |
+
agent-native score 0.5690 0.5690 +0.0000 [-0.1034, +0.1034] 58 TIE
|
| 109 |
+
agent-native solved[reserved] 0.4419 0.4884 -0.0465 [-0.1860, +0.0930] 43 TIE
|
| 110 |
+
agent-native solved[unseen] 0.9333 0.8000 +0.1333 [+0.0000, +0.3333] 15 TIE
|
| 111 |
+
toolcall exact 0.8462 0.9560 -0.1099 [-0.1758, -0.0549] 91 LOSS
|
| 112 |
+
=== v2b-ck600
|
| 113 |
+
|
| 114 |
+
cyber_composite (tier test): NOT AVAILABLE [not final]
|
| 115 |
+
F1 cybermetric -- incomplete
|
| 116 |
+
F2 seceval -- incomplete
|
| 117 |
+
F3 ctibench -- incomplete
|
| 118 |
+
F4 secbench -- incomplete
|
| 119 |
+
F5 secqa -- incomplete
|
| 120 |
+
F6 cyberbench_jpm -- missing
|
| 121 |
+
F7 vuln2 -- missing
|
| 122 |
+
guards (dev tier, never in the composite):
|
| 123 |
+
mmlu_sec 0.7696 ok vs ref 0.7892: -0.0196
|
| 124 |
+
coding2 0.7531 ok vs ref 0.7615: -0.0084
|
| 125 |
+
agent2 0.8750 ok vs ref 0.6750: +0.2000
|
| 126 |
+
xstest_safe_compliance 1.0000 ok vs ref 0.9947: +0.0053
|
| 127 |
+
--- agentcy DEV (cap 60) vs ab9b-r1, paired
|
| 128 |
+
paired comparison A=v2b-ck600 (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 129 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 130 |
+
agent-plain solved 0.6333 0.5500 +0.0833 [-0.0167, +0.2000] 60 TIE
|
| 131 |
+
agent-plain exact 0.6333 0.5500 +0.0833 [-0.0167, +0.2000] 60 TIE
|
| 132 |
+
agent-plain score 0.6333 0.5500 +0.0833 [-0.0167, +0.2000] 60 TIE
|
| 133 |
+
agent-plain solved[reserved] 0.5333 0.4889 +0.0444 [-0.0889, +0.1556] 45 TIE
|
| 134 |
+
agent-plain solved[unseen] 0.9333 0.7333 +0.2000 [+0.0000, +0.4000] 15 TIE
|
| 135 |
+
agent-native solved 0.5424 0.2881 +0.2542 [+0.1017, +0.4068] 59 WIN
|
| 136 |
+
agent-native exact 0.5424 0.2881 +0.2542 [+0.1017, +0.4068] 59 WIN
|
| 137 |
+
agent-native score 0.5424 0.2881 +0.2542 [+0.1017, +0.4068] 59 WIN
|
| 138 |
+
agent-native solved[reserved] 0.4091 0.2727 +0.1364 [-0.0227, +0.3182] 44 TIE
|
| 139 |
+
agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8000] 15 WIN
|
| 140 |
+
toolcall exact 0.8571 0.9780 -0.1209 [-0.1868, -0.0549] 91 LOSS
|
| 141 |
+
--- battery DEV vs v2a-ck900 (dev), paired
|
| 142 |
+
paired comparison A=v2b-ck600 (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 143 |
+
task A B diff 95% CI of diff n_shared verdict
|
| 144 |
+
cybermetric 0.8825 0.8850 -0.0025 [-0.0225, +0.0175] 400 TIE
|
| 145 |
+
seceval 0.7500 0.7350 +0.0150 [-0.0050, +0.0350] 400 TIE
|
| 146 |
+
cti-mcq 0.6725 0.6775 -0.0050 [-0.0300, +0.0200] 400 TIE
|
| 147 |
+
cti-rcm 0.7375 0.7450 -0.0075 [-0.0250, +0.0100] 400 TIE
|
| 148 |
+
cti-vsp 0.8325 0.8100 +0.0225 [+0.0094, +0.0359] 400 WIN
|
| 149 |
+
cti-ate 0.3564 0.2296 +0.1267 [+0.0680, +0.1822] 38 WIN
|
| 150 |
+
cti-taa 0.3235 0.4118 -0.0882 [-0.2059, +0.0294] 34 TIE
|
| 151 |
+
secbench 0.7925 0.7900 +0.0025 [-0.0150, +0.0200] 400 TIE
|
| 152 |
+
secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
|
| 153 |
+
--- agentcy DEV vs v2a-ck900 (dev), paired
|
| 154 |
+
paired comparison A=v2b-ck600 (dev) B=v2a-ck900 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 155 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 156 |
+
agent-plain solved 0.6333 0.6333 +0.0000 [-0.0833, +0.1000] 60 TIE
|
| 157 |
+
agent-plain exact 0.6333 0.6333 +0.0000 [-0.0833, +0.1000] 60 TIE
|
| 158 |
+
agent-plain score 0.6333 0.6333 +0.0000 [-0.0833, +0.1000] 60 TIE
|
| 159 |
+
agent-plain solved[reserved] 0.5333 0.5111 +0.0222 [-0.0889, +0.1333] 45 TIE
|
| 160 |
+
agent-plain solved[unseen] 0.9333 1.0000 -0.0667 [-0.2000, +0.0000] 15 TIE
|
| 161 |
+
agent-native solved 0.5333 0.5833 -0.0500 [-0.1667, +0.0500] 60 TIE
|
| 162 |
+
agent-native exact 0.5333 0.5833 -0.0500 [-0.1667, +0.0500] 60 TIE
|
| 163 |
+
agent-native score 0.5333 0.5833 -0.0500 [-0.1667, +0.0500] 60 TIE
|
| 164 |
+
agent-native solved[reserved] 0.4000 0.5111 -0.1111 [-0.2444, +0.0222] 45 TIE
|
| 165 |
+
agent-native solved[unseen] 0.9333 0.8000 +0.1333 [+0.0000, +0.3333] 15 TIE
|
| 166 |
+
toolcall exact 0.8571 0.9560 -0.0989 [-0.1648, -0.0440] 91 LOSS
|
evidence/results/v2b_pick.json
ADDED
|
@@ -0,0 +1,342 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "v2b",
|
| 3 |
+
"ref": "ab9b-r1 (dev)",
|
| 4 |
+
"rule": "eligible = no guard (mmlu_sec, coding2, agent2, xstest_safe_compliance) more than 0.03 below the reference, and agentcy dev tool-call exact not more than 0.03 below it.\nScore = mean( dev composite proxy gain / 0.05 , agentcy dev plain solved gain / 0.15 ); the proxy is the mean over the dev families available for BOTH the candidate and the reference\n(F1-F5 and F7; F6 CyberBench has no dev tier). Highest score among the eligible wins; every eligible candidate within 0.02 of the best counts as tied and the LATEST checkpoint of the tied ones is picked.\nNo eligible candidate -> chosen = null (no automatic TEST scoring).",
|
| 5 |
+
"rows": [
|
| 6 |
+
{
|
| 7 |
+
"tag": "final",
|
| 8 |
+
"battery_tag": "v2b-final (dev)",
|
| 9 |
+
"gguf": "models/gguf/v2b-final-Q4_K_M.gguf",
|
| 10 |
+
"eligible": false,
|
| 11 |
+
"reasons": [
|
| 12 |
+
"toolcall exact -0.1319"
|
| 13 |
+
],
|
| 14 |
+
"families_used": [
|
| 15 |
+
"ctibench",
|
| 16 |
+
"cybermetric",
|
| 17 |
+
"secbench",
|
| 18 |
+
"seceval",
|
| 19 |
+
"secqa",
|
| 20 |
+
"vuln2"
|
| 21 |
+
],
|
| 22 |
+
"proxy": 0.7616,
|
| 23 |
+
"ref_proxy": 0.7532666666666668,
|
| 24 |
+
"families": {
|
| 25 |
+
"ctibench": [
|
| 26 |
+
0.5858,
|
| 27 |
+
0.5141
|
| 28 |
+
],
|
| 29 |
+
"cybermetric": [
|
| 30 |
+
0.88,
|
| 31 |
+
0.9075
|
| 32 |
+
],
|
| 33 |
+
"secbench": [
|
| 34 |
+
0.78,
|
| 35 |
+
0.8
|
| 36 |
+
],
|
| 37 |
+
"seceval": [
|
| 38 |
+
0.7525,
|
| 39 |
+
0.74
|
| 40 |
+
],
|
| 41 |
+
"secqa": [
|
| 42 |
+
0.9927,
|
| 43 |
+
0.9927
|
| 44 |
+
],
|
| 45 |
+
"vuln2": [
|
| 46 |
+
0.5786,
|
| 47 |
+
0.5653
|
| 48 |
+
]
|
| 49 |
+
},
|
| 50 |
+
"g1": 0.16666666666666607,
|
| 51 |
+
"guards": {
|
| 52 |
+
"mmlu_sec": -0.0196,
|
| 53 |
+
"coding2": -0.0209,
|
| 54 |
+
"agent2": 0.175,
|
| 55 |
+
"xstest_safe_compliance": 0.0
|
| 56 |
+
},
|
| 57 |
+
"agentcy": {
|
| 58 |
+
"plain_solved": {
|
| 59 |
+
"n": 60,
|
| 60 |
+
"a": 0.6833,
|
| 61 |
+
"b": 0.55,
|
| 62 |
+
"diff": 0.1333,
|
| 63 |
+
"ci95": [
|
| 64 |
+
0.0333,
|
| 65 |
+
0.2333
|
| 66 |
+
],
|
| 67 |
+
"verdict": "WIN"
|
| 68 |
+
},
|
| 69 |
+
"native_solved": {
|
| 70 |
+
"n": 59,
|
| 71 |
+
"a": 0.661,
|
| 72 |
+
"b": 0.2881,
|
| 73 |
+
"diff": 0.3729,
|
| 74 |
+
"ci95": [
|
| 75 |
+
0.2203,
|
| 76 |
+
0.5254
|
| 77 |
+
],
|
| 78 |
+
"verdict": "WIN"
|
| 79 |
+
},
|
| 80 |
+
"toolcall_exact": {
|
| 81 |
+
"n": 91,
|
| 82 |
+
"a": 0.8462,
|
| 83 |
+
"b": 0.978,
|
| 84 |
+
"diff": -0.1319,
|
| 85 |
+
"ci95": [
|
| 86 |
+
-0.2088,
|
| 87 |
+
-0.0659
|
| 88 |
+
],
|
| 89 |
+
"verdict": "LOSS"
|
| 90 |
+
},
|
| 91 |
+
"plain_reserved": {
|
| 92 |
+
"n": 45,
|
| 93 |
+
"a": 0.5778,
|
| 94 |
+
"b": 0.4889,
|
| 95 |
+
"diff": 0.0889,
|
| 96 |
+
"ci95": [
|
| 97 |
+
0.0,
|
| 98 |
+
0.2
|
| 99 |
+
],
|
| 100 |
+
"verdict": "TIE"
|
| 101 |
+
},
|
| 102 |
+
"plain_unseen": {
|
| 103 |
+
"n": 15,
|
| 104 |
+
"a": 1.0,
|
| 105 |
+
"b": 0.7333,
|
| 106 |
+
"diff": 0.2667,
|
| 107 |
+
"ci95": [
|
| 108 |
+
0.0667,
|
| 109 |
+
0.5333
|
| 110 |
+
],
|
| 111 |
+
"verdict": "WIN"
|
| 112 |
+
}
|
| 113 |
+
},
|
| 114 |
+
"g2": 0.8886666666666667,
|
| 115 |
+
"score": 0.5276666666666664
|
| 116 |
+
},
|
| 117 |
+
{
|
| 118 |
+
"tag": "ck700",
|
| 119 |
+
"battery_tag": "v2b-ck700 (dev)",
|
| 120 |
+
"gguf": "models/gguf/v2b-ck700-Q4_K_M.gguf",
|
| 121 |
+
"eligible": false,
|
| 122 |
+
"reasons": [
|
| 123 |
+
"toolcall exact -0.1319"
|
| 124 |
+
],
|
| 125 |
+
"families_used": [
|
| 126 |
+
"ctibench",
|
| 127 |
+
"cybermetric",
|
| 128 |
+
"secbench",
|
| 129 |
+
"seceval",
|
| 130 |
+
"secqa",
|
| 131 |
+
"vuln2"
|
| 132 |
+
],
|
| 133 |
+
"proxy": 0.7582,
|
| 134 |
+
"ref_proxy": 0.7532666666666668,
|
| 135 |
+
"families": {
|
| 136 |
+
"ctibench": [
|
| 137 |
+
0.588,
|
| 138 |
+
0.5141
|
| 139 |
+
],
|
| 140 |
+
"cybermetric": [
|
| 141 |
+
0.875,
|
| 142 |
+
0.9075
|
| 143 |
+
],
|
| 144 |
+
"secbench": [
|
| 145 |
+
0.7775,
|
| 146 |
+
0.8
|
| 147 |
+
],
|
| 148 |
+
"seceval": [
|
| 149 |
+
0.7425,
|
| 150 |
+
0.74
|
| 151 |
+
],
|
| 152 |
+
"secqa": [
|
| 153 |
+
0.9927,
|
| 154 |
+
0.9927
|
| 155 |
+
],
|
| 156 |
+
"vuln2": [
|
| 157 |
+
0.5735,
|
| 158 |
+
0.5653
|
| 159 |
+
]
|
| 160 |
+
},
|
| 161 |
+
"g1": 0.09866666666666468,
|
| 162 |
+
"guards": {
|
| 163 |
+
"mmlu_sec": -0.0098,
|
| 164 |
+
"coding2": -0.0042,
|
| 165 |
+
"agent2": 0.125,
|
| 166 |
+
"xstest_safe_compliance": 0.0
|
| 167 |
+
},
|
| 168 |
+
"agentcy": {
|
| 169 |
+
"plain_solved": {
|
| 170 |
+
"n": 60,
|
| 171 |
+
"a": 0.6667,
|
| 172 |
+
"b": 0.55,
|
| 173 |
+
"diff": 0.1167,
|
| 174 |
+
"ci95": [
|
| 175 |
+
0.0333,
|
| 176 |
+
0.2167
|
| 177 |
+
],
|
| 178 |
+
"verdict": "WIN"
|
| 179 |
+
},
|
| 180 |
+
"native_solved": {
|
| 181 |
+
"n": 57,
|
| 182 |
+
"a": 0.5789,
|
| 183 |
+
"b": 0.2982,
|
| 184 |
+
"diff": 0.2807,
|
| 185 |
+
"ci95": [
|
| 186 |
+
0.1228,
|
| 187 |
+
0.4386
|
| 188 |
+
],
|
| 189 |
+
"verdict": "WIN"
|
| 190 |
+
},
|
| 191 |
+
"toolcall_exact": {
|
| 192 |
+
"n": 91,
|
| 193 |
+
"a": 0.8462,
|
| 194 |
+
"b": 0.978,
|
| 195 |
+
"diff": -0.1319,
|
| 196 |
+
"ci95": [
|
| 197 |
+
-0.2088,
|
| 198 |
+
-0.0659
|
| 199 |
+
],
|
| 200 |
+
"verdict": "LOSS"
|
| 201 |
+
},
|
| 202 |
+
"plain_reserved": {
|
| 203 |
+
"n": 45,
|
| 204 |
+
"a": 0.5556,
|
| 205 |
+
"b": 0.4889,
|
| 206 |
+
"diff": 0.0667,
|
| 207 |
+
"ci95": [
|
| 208 |
+
-0.0222,
|
| 209 |
+
0.1556
|
| 210 |
+
],
|
| 211 |
+
"verdict": "TIE"
|
| 212 |
+
},
|
| 213 |
+
"plain_unseen": {
|
| 214 |
+
"n": 15,
|
| 215 |
+
"a": 1.0,
|
| 216 |
+
"b": 0.7333,
|
| 217 |
+
"diff": 0.2667,
|
| 218 |
+
"ci95": [
|
| 219 |
+
0.0667,
|
| 220 |
+
0.5333
|
| 221 |
+
],
|
| 222 |
+
"verdict": "WIN"
|
| 223 |
+
}
|
| 224 |
+
},
|
| 225 |
+
"g2": 0.778,
|
| 226 |
+
"score": 0.43833333333333235
|
| 227 |
+
},
|
| 228 |
+
{
|
| 229 |
+
"tag": "ck600",
|
| 230 |
+
"battery_tag": "v2b-ck600 (dev)",
|
| 231 |
+
"gguf": "models/gguf/v2b-ck600-Q4_K_M.gguf",
|
| 232 |
+
"eligible": false,
|
| 233 |
+
"reasons": [
|
| 234 |
+
"toolcall exact -0.1209"
|
| 235 |
+
],
|
| 236 |
+
"families_used": [
|
| 237 |
+
"ctibench",
|
| 238 |
+
"cybermetric",
|
| 239 |
+
"secbench",
|
| 240 |
+
"seceval",
|
| 241 |
+
"secqa",
|
| 242 |
+
"vuln2"
|
| 243 |
+
],
|
| 244 |
+
"proxy": 0.7655666666666666,
|
| 245 |
+
"ref_proxy": 0.7532666666666668,
|
| 246 |
+
"families": {
|
| 247 |
+
"ctibench": [
|
| 248 |
+
0.5845,
|
| 249 |
+
0.5141
|
| 250 |
+
],
|
| 251 |
+
"cybermetric": [
|
| 252 |
+
0.8825,
|
| 253 |
+
0.9075
|
| 254 |
+
],
|
| 255 |
+
"secbench": [
|
| 256 |
+
0.7925,
|
| 257 |
+
0.8
|
| 258 |
+
],
|
| 259 |
+
"seceval": [
|
| 260 |
+
0.75,
|
| 261 |
+
0.74
|
| 262 |
+
],
|
| 263 |
+
"secqa": [
|
| 264 |
+
0.9927,
|
| 265 |
+
0.9927
|
| 266 |
+
],
|
| 267 |
+
"vuln2": [
|
| 268 |
+
0.5912,
|
| 269 |
+
0.5653
|
| 270 |
+
]
|
| 271 |
+
},
|
| 272 |
+
"g1": 0.24599999999999733,
|
| 273 |
+
"guards": {
|
| 274 |
+
"mmlu_sec": -0.0196,
|
| 275 |
+
"coding2": -0.0084,
|
| 276 |
+
"agent2": 0.2,
|
| 277 |
+
"xstest_safe_compliance": 0.0053
|
| 278 |
+
},
|
| 279 |
+
"agentcy": {
|
| 280 |
+
"plain_solved": {
|
| 281 |
+
"n": 60,
|
| 282 |
+
"a": 0.6333,
|
| 283 |
+
"b": 0.55,
|
| 284 |
+
"diff": 0.0833,
|
| 285 |
+
"ci95": [
|
| 286 |
+
-0.0167,
|
| 287 |
+
0.2
|
| 288 |
+
],
|
| 289 |
+
"verdict": "TIE"
|
| 290 |
+
},
|
| 291 |
+
"native_solved": {
|
| 292 |
+
"n": 59,
|
| 293 |
+
"a": 0.5424,
|
| 294 |
+
"b": 0.2881,
|
| 295 |
+
"diff": 0.2542,
|
| 296 |
+
"ci95": [
|
| 297 |
+
0.1017,
|
| 298 |
+
0.4068
|
| 299 |
+
],
|
| 300 |
+
"verdict": "WIN"
|
| 301 |
+
},
|
| 302 |
+
"toolcall_exact": {
|
| 303 |
+
"n": 91,
|
| 304 |
+
"a": 0.8571,
|
| 305 |
+
"b": 0.978,
|
| 306 |
+
"diff": -0.1209,
|
| 307 |
+
"ci95": [
|
| 308 |
+
-0.1868,
|
| 309 |
+
-0.0549
|
| 310 |
+
],
|
| 311 |
+
"verdict": "LOSS"
|
| 312 |
+
},
|
| 313 |
+
"plain_reserved": {
|
| 314 |
+
"n": 45,
|
| 315 |
+
"a": 0.5333,
|
| 316 |
+
"b": 0.4889,
|
| 317 |
+
"diff": 0.0444,
|
| 318 |
+
"ci95": [
|
| 319 |
+
-0.0889,
|
| 320 |
+
0.1556
|
| 321 |
+
],
|
| 322 |
+
"verdict": "TIE"
|
| 323 |
+
},
|
| 324 |
+
"plain_unseen": {
|
| 325 |
+
"n": 15,
|
| 326 |
+
"a": 0.9333,
|
| 327 |
+
"b": 0.7333,
|
| 328 |
+
"diff": 0.2,
|
| 329 |
+
"ci95": [
|
| 330 |
+
0.0,
|
| 331 |
+
0.4
|
| 332 |
+
],
|
| 333 |
+
"verdict": "TIE"
|
| 334 |
+
}
|
| 335 |
+
},
|
| 336 |
+
"g2": 0.5553333333333333,
|
| 337 |
+
"score": 0.40066666666666534
|
| 338 |
+
}
|
| 339 |
+
],
|
| 340 |
+
"chosen": null,
|
| 341 |
+
"chosen_gguf": null
|
| 342 |
+
}
|
evidence/results/v2c_dev_summary.txt
ADDED
|
@@ -0,0 +1,166 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
candidates: final: ck200: ck100:
|
| 2 |
+
=== v2c-final
|
| 3 |
+
|
| 4 |
+
cyber_composite (tier test): NOT AVAILABLE [not final]
|
| 5 |
+
F1 cybermetric -- incomplete
|
| 6 |
+
F2 seceval -- incomplete
|
| 7 |
+
F3 ctibench -- incomplete
|
| 8 |
+
F4 secbench -- incomplete
|
| 9 |
+
F5 secqa -- incomplete
|
| 10 |
+
F6 cyberbench_jpm -- missing
|
| 11 |
+
F7 vuln2 -- missing
|
| 12 |
+
guards (dev tier, never in the composite):
|
| 13 |
+
mmlu_sec 0.7941 ok vs ref 0.7892: +0.0049
|
| 14 |
+
coding2 0.7657 ok vs ref 0.7615: +0.0042
|
| 15 |
+
agent2 0.8250 ok vs ref 0.6750: +0.1500
|
| 16 |
+
xstest_safe_compliance 0.9840 ok vs ref 0.9947: -0.0107
|
| 17 |
+
--- agentcy DEV (cap 60) vs ab9b-r1, paired
|
| 18 |
+
paired comparison A=v2c-final (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 19 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 20 |
+
agent-plain solved 0.6333 0.5500 +0.0833 [+0.0000, +0.1833] 60 TIE
|
| 21 |
+
agent-plain exact 0.6333 0.5500 +0.0833 [+0.0000, +0.1833] 60 TIE
|
| 22 |
+
agent-plain score 0.6333 0.5500 +0.0833 [+0.0000, +0.1833] 60 TIE
|
| 23 |
+
agent-plain solved[reserved] 0.5333 0.4889 +0.0444 [-0.0444, +0.1556] 45 TIE
|
| 24 |
+
agent-plain solved[unseen] 0.9333 0.7333 +0.2000 [+0.0000, +0.4000] 15 TIE
|
| 25 |
+
agent-native solved 0.6271 0.2881 +0.3390 [+0.1864, +0.4915] 59 WIN
|
| 26 |
+
agent-native exact 0.6271 0.2881 +0.3390 [+0.1864, +0.4915] 59 WIN
|
| 27 |
+
agent-native score 0.6271 0.2881 +0.3390 [+0.1864, +0.4915] 59 WIN
|
| 28 |
+
agent-native solved[reserved] 0.5227 0.2727 +0.2500 [+0.0682, +0.4318] 44 WIN
|
| 29 |
+
agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8667] 15 WIN
|
| 30 |
+
toolcall exact 0.9780 0.9780 +0.0000 [-0.0330, +0.0330] 91 TIE
|
| 31 |
+
--- battery DEV vs v2b-final (dev), paired
|
| 32 |
+
paired comparison A=v2c-final (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 33 |
+
task A B diff 95% CI of diff n_shared verdict
|
| 34 |
+
cybermetric 0.8825 0.8800 +0.0025 [-0.0125, +0.0175] 400 TIE
|
| 35 |
+
seceval 0.7450 0.7525 -0.0075 [-0.0275, +0.0125] 400 TIE
|
| 36 |
+
cti-mcq 0.6750 0.6750 +0.0000 [-0.0250, +0.0250] 400 TIE
|
| 37 |
+
cti-rcm 0.7350 0.7500 -0.0150 [-0.0350, +0.0050] 400 TIE
|
| 38 |
+
cti-vsp 0.7644 0.8387 -0.0744 [-0.0900, -0.0588] 400 LOSS
|
| 39 |
+
cti-ate 0.3981 0.3712 +0.0269 [-0.0152, +0.0710] 38 TIE
|
| 40 |
+
cti-taa 0.3235 0.2941 +0.0294 [-0.0882, +0.1471] 34 TIE
|
| 41 |
+
secbench 0.7800 0.7800 +0.0000 [-0.0200, +0.0200] 400 TIE
|
| 42 |
+
secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
|
| 43 |
+
--- agentcy DEV vs v2b-final (dev), paired
|
| 44 |
+
paired comparison A=v2c-final (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 45 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 46 |
+
agent-plain solved 0.6333 0.6833 -0.0500 [-0.1167, +0.0000] 60 TIE
|
| 47 |
+
agent-plain exact 0.6333 0.6833 -0.0500 [-0.1167, +0.0000] 60 TIE
|
| 48 |
+
agent-plain score 0.6333 0.6833 -0.0500 [-0.1167, +0.0000] 60 TIE
|
| 49 |
+
agent-plain solved[reserved] 0.5333 0.5778 -0.0444 [-0.1111, +0.0000] 45 TIE
|
| 50 |
+
agent-plain solved[unseen] 0.9333 1.0000 -0.0667 [-0.2000, +0.0000] 15 TIE
|
| 51 |
+
agent-native solved 0.6167 0.6500 -0.0333 [-0.1167, +0.0500] 60 TIE
|
| 52 |
+
agent-native exact 0.6167 0.6500 -0.0333 [-0.1167, +0.0500] 60 TIE
|
| 53 |
+
agent-native score 0.6167 0.6500 -0.0333 [-0.1167, +0.0500] 60 TIE
|
| 54 |
+
agent-native solved[reserved] 0.5111 0.5556 -0.0444 [-0.1556, +0.0444] 45 TIE
|
| 55 |
+
agent-native solved[unseen] 0.9333 0.9333 +0.0000 [+0.0000, +0.0000] 15 TIE
|
| 56 |
+
toolcall exact 0.9780 0.8462 +0.1319 [+0.0659, +0.2088] 91 WIN
|
| 57 |
+
=== v2c-ck200
|
| 58 |
+
|
| 59 |
+
cyber_composite (tier test): NOT AVAILABLE [not final]
|
| 60 |
+
F1 cybermetric -- incomplete
|
| 61 |
+
F2 seceval -- incomplete
|
| 62 |
+
F3 ctibench -- incomplete
|
| 63 |
+
F4 secbench -- incomplete
|
| 64 |
+
F5 secqa -- incomplete
|
| 65 |
+
F6 cyberbench_jpm -- missing
|
| 66 |
+
F7 vuln2 -- missing
|
| 67 |
+
guards (dev tier, never in the composite):
|
| 68 |
+
mmlu_sec 0.7941 ok vs ref 0.7892: +0.0049
|
| 69 |
+
coding2 0.7657 ok vs ref 0.7615: +0.0042
|
| 70 |
+
agent2 0.7750 ok vs ref 0.6750: +0.1000
|
| 71 |
+
xstest_safe_compliance 0.9894 ok vs ref 0.9947: -0.0053
|
| 72 |
+
--- agentcy DEV (cap 60) vs ab9b-r1, paired
|
| 73 |
+
paired comparison A=v2c-ck200 (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 74 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 75 |
+
agent-plain solved 0.6167 0.5500 +0.0667 [-0.0333, +0.1667] 60 TIE
|
| 76 |
+
agent-plain exact 0.6167 0.5500 +0.0667 [-0.0333, +0.1667] 60 TIE
|
| 77 |
+
agent-plain score 0.6167 0.5500 +0.0667 [-0.0333, +0.1667] 60 TIE
|
| 78 |
+
agent-plain solved[reserved] 0.5111 0.4889 +0.0222 [-0.0889, +0.1333] 45 TIE
|
| 79 |
+
agent-plain solved[unseen] 0.9333 0.7333 +0.2000 [+0.0000, +0.4000] 15 TIE
|
| 80 |
+
agent-native solved 0.6102 0.2881 +0.3220 [+0.1695, +0.4746] 59 WIN
|
| 81 |
+
agent-native exact 0.6102 0.2881 +0.3220 [+0.1695, +0.4746] 59 WIN
|
| 82 |
+
agent-native score 0.6102 0.2881 +0.3220 [+0.1695, +0.4746] 59 WIN
|
| 83 |
+
agent-native solved[reserved] 0.5000 0.2727 +0.2273 [+0.0682, +0.3864] 44 WIN
|
| 84 |
+
agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8667] 15 WIN
|
| 85 |
+
toolcall exact 0.9670 0.9780 -0.0110 [-0.0440, +0.0220] 91 TIE
|
| 86 |
+
--- battery DEV vs v2b-final (dev), paired
|
| 87 |
+
paired comparison A=v2c-ck200 (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 88 |
+
task A B diff 95% CI of diff n_shared verdict
|
| 89 |
+
cybermetric 0.8875 0.8800 +0.0075 [-0.0100, +0.0250] 400 TIE
|
| 90 |
+
seceval 0.7475 0.7525 -0.0050 [-0.0275, +0.0175] 400 TIE
|
| 91 |
+
cti-mcq 0.6800 0.6750 +0.0050 [-0.0200, +0.0300] 400 TIE
|
| 92 |
+
cti-rcm 0.7375 0.7500 -0.0125 [-0.0350, +0.0100] 400 TIE
|
| 93 |
+
cti-vsp 0.7734 0.8387 -0.0653 [-0.0803, -0.0503] 400 LOSS
|
| 94 |
+
cti-ate 0.3938 0.3712 +0.0226 [-0.0229, +0.0646] 38 TIE
|
| 95 |
+
cti-taa 0.3529 0.2941 +0.0588 [-0.0882, +0.2059] 34 TIE
|
| 96 |
+
secbench 0.7825 0.7800 +0.0025 [-0.0175, +0.0225] 400 TIE
|
| 97 |
+
secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
|
| 98 |
+
--- agentcy DEV vs v2b-final (dev), paired
|
| 99 |
+
paired comparison A=v2c-ck200 (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 100 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 101 |
+
agent-plain solved 0.6167 0.6833 -0.0667 [-0.1333, -0.0167] 60 LOSS
|
| 102 |
+
agent-plain exact 0.6167 0.6833 -0.0667 [-0.1333, -0.0167] 60 LOSS
|
| 103 |
+
agent-plain score 0.6167 0.6833 -0.0667 [-0.1333, -0.0167] 60 LOSS
|
| 104 |
+
agent-plain solved[reserved] 0.5111 0.5778 -0.0667 [-0.1333, +0.0000] 45 TIE
|
| 105 |
+
agent-plain solved[unseen] 0.9333 1.0000 -0.0667 [-0.2000, +0.0000] 15 TIE
|
| 106 |
+
agent-native solved 0.6000 0.6500 -0.0500 [-0.1167, +0.0000] 60 TIE
|
| 107 |
+
agent-native exact 0.6000 0.6500 -0.0500 [-0.1167, +0.0000] 60 TIE
|
| 108 |
+
agent-native score 0.6000 0.6500 -0.0500 [-0.1167, +0.0000] 60 TIE
|
| 109 |
+
agent-native solved[reserved] 0.4889 0.5556 -0.0667 [-0.1556, +0.0000] 45 TIE
|
| 110 |
+
agent-native solved[unseen] 0.9333 0.9333 +0.0000 [+0.0000, +0.0000] 15 TIE
|
| 111 |
+
toolcall exact 0.9670 0.8462 +0.1209 [+0.0440, +0.1978] 91 WIN
|
| 112 |
+
=== v2c-ck100
|
| 113 |
+
|
| 114 |
+
cyber_composite (tier test): NOT AVAILABLE [not final]
|
| 115 |
+
F1 cybermetric -- incomplete
|
| 116 |
+
F2 seceval -- incomplete
|
| 117 |
+
F3 ctibench -- incomplete
|
| 118 |
+
F4 secbench -- incomplete
|
| 119 |
+
F5 secqa -- incomplete
|
| 120 |
+
F6 cyberbench_jpm -- missing
|
| 121 |
+
F7 vuln2 -- missing
|
| 122 |
+
guards (dev tier, never in the composite):
|
| 123 |
+
mmlu_sec 0.8039 ok vs ref 0.7892: +0.0147
|
| 124 |
+
coding2 0.7573 ok vs ref 0.7615: -0.0042
|
| 125 |
+
agent2 0.7000 ok vs ref 0.6750: +0.0250
|
| 126 |
+
xstest_safe_compliance 0.9840 ok vs ref 0.9947: -0.0107
|
| 127 |
+
--- agentcy DEV (cap 60) vs ab9b-r1, paired
|
| 128 |
+
paired comparison A=v2c-ck100 (dev) B=ab9b-r1 (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 129 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 130 |
+
agent-plain solved 0.6000 0.5500 +0.0500 [-0.0500, +0.1500] 60 TIE
|
| 131 |
+
agent-plain exact 0.6000 0.5500 +0.0500 [-0.0500, +0.1500] 60 TIE
|
| 132 |
+
agent-plain score 0.6000 0.5500 +0.0500 [-0.0500, +0.1500] 60 TIE
|
| 133 |
+
agent-plain solved[reserved] 0.4889 0.4889 +0.0000 [-0.1111, +0.1111] 45 TIE
|
| 134 |
+
agent-plain solved[unseen] 0.9333 0.7333 +0.2000 [+0.0000, +0.4000] 15 TIE
|
| 135 |
+
agent-native solved 0.5763 0.2881 +0.2881 [+0.1525, +0.4237] 59 WIN
|
| 136 |
+
agent-native exact 0.5763 0.2881 +0.2881 [+0.1525, +0.4237] 59 WIN
|
| 137 |
+
agent-native score 0.5763 0.2881 +0.2881 [+0.1525, +0.4237] 59 WIN
|
| 138 |
+
agent-native solved[reserved] 0.4545 0.2727 +0.1818 [+0.0227, +0.3409] 44 WIN
|
| 139 |
+
agent-native solved[unseen] 0.9333 0.3333 +0.6000 [+0.3333, +0.8000] 15 WIN
|
| 140 |
+
toolcall exact 0.9780 0.9780 +0.0000 [-0.0330, +0.0330] 91 TIE
|
| 141 |
+
--- battery DEV vs v2b-final (dev), paired
|
| 142 |
+
paired comparison A=v2c-ck100 (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B, win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 143 |
+
task A B diff 95% CI of diff n_shared verdict
|
| 144 |
+
cybermetric 0.8950 0.8800 +0.0150 [-0.0025, +0.0325] 400 TIE
|
| 145 |
+
seceval 0.7400 0.7525 -0.0125 [-0.0325, +0.0075] 400 TIE
|
| 146 |
+
cti-mcq 0.6600 0.6750 -0.0150 [-0.0400, +0.0100] 400 TIE
|
| 147 |
+
cti-rcm 0.6950 0.7500 -0.0550 [-0.0825, -0.0300] 400 LOSS
|
| 148 |
+
cti-vsp 0.8456 0.8387 +0.0069 [-0.0038, +0.0178] 400 TIE
|
| 149 |
+
cti-ate 0.3086 0.3712 -0.0627 [-0.1032, -0.0237] 38 LOSS
|
| 150 |
+
cti-taa 0.2353 0.2941 -0.0588 [-0.1765, +0.0588] 34 TIE
|
| 151 |
+
secbench 0.7825 0.7800 +0.0025 [-0.0125, +0.0175] 400 TIE
|
| 152 |
+
secqa 0.9927 0.9927 +0.0000 [+0.0000, +0.0000] 137 TIE
|
| 153 |
+
--- agentcy DEV vs v2b-final (dev), paired
|
| 154 |
+
paired comparison A=v2c-ck100 (dev) B=v2b-final (dev) tier=dev boot=2000 seed=0 (diff = A - B; win if the 95% lower bound > 0, loss if the upper bound < 0)
|
| 155 |
+
task / metric A B diff 95% CI of diff n verdict
|
| 156 |
+
agent-plain solved 0.6000 0.6833 -0.0833 [-0.1500, -0.0167] 60 LOSS
|
| 157 |
+
agent-plain exact 0.6000 0.6833 -0.0833 [-0.1500, -0.0167] 60 LOSS
|
| 158 |
+
agent-plain score 0.6000 0.6833 -0.0833 [-0.1500, -0.0167] 60 LOSS
|
| 159 |
+
agent-plain solved[reserved] 0.4889 0.5778 -0.0889 [-0.1778, -0.0222] 45 LOSS
|
| 160 |
+
agent-plain solved[unseen] 0.9333 1.0000 -0.0667 [-0.2000, +0.0000] 15 TIE
|
| 161 |
+
agent-native solved 0.5667 0.6500 -0.0833 [-0.1500, -0.0167] 60 LOSS
|
| 162 |
+
agent-native exact 0.5667 0.6500 -0.0833 [-0.1500, -0.0167] 60 LOSS
|
| 163 |
+
agent-native score 0.5667 0.6500 -0.0833 [-0.1500, -0.0167] 60 LOSS
|
| 164 |
+
agent-native solved[reserved] 0.4444 0.5556 -0.1111 [-0.2222, -0.0222] 45 LOSS
|
| 165 |
+
agent-native solved[unseen] 0.9333 0.9333 +0.0000 [+0.0000, +0.0000] 15 TIE
|
| 166 |
+
toolcall exact 0.9780 0.8462 +0.1319 [+0.0659, +0.2088] 91 WIN
|
evidence/results/v2c_pick.json
ADDED
|
@@ -0,0 +1,336 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "v2c",
|
| 3 |
+
"ref": "ab9b-r1 (dev)",
|
| 4 |
+
"rule": "eligible = no guard (mmlu_sec, coding2, agent2, xstest_safe_compliance) more than 0.03 below the reference, and agentcy dev tool-call exact not more than 0.03 below it.\nScore = mean( dev composite proxy gain / 0.05 , agentcy dev plain solved gain / 0.15 ); the proxy is the mean over the dev families available for BOTH the candidate and the reference\n(F1-F5 and F7; F6 CyberBench has no dev tier). Highest score among the eligible wins; every eligible candidate within 0.02 of the best counts as tied and the LATEST checkpoint of the tied ones is picked.\nNo eligible candidate -> chosen = null (no automatic TEST scoring).",
|
| 5 |
+
"rows": [
|
| 6 |
+
{
|
| 7 |
+
"tag": "final",
|
| 8 |
+
"battery_tag": "v2c-final (dev)",
|
| 9 |
+
"gguf": "models/gguf/v2c-final-Q4_K_M.gguf",
|
| 10 |
+
"eligible": true,
|
| 11 |
+
"reasons": [],
|
| 12 |
+
"families_used": [
|
| 13 |
+
"ctibench",
|
| 14 |
+
"cybermetric",
|
| 15 |
+
"secbench",
|
| 16 |
+
"seceval",
|
| 17 |
+
"secqa",
|
| 18 |
+
"vuln2"
|
| 19 |
+
],
|
| 20 |
+
"proxy": 0.7581000000000001,
|
| 21 |
+
"ref_proxy": 0.7532666666666668,
|
| 22 |
+
"families": {
|
| 23 |
+
"ctibench": [
|
| 24 |
+
0.5792,
|
| 25 |
+
0.5141
|
| 26 |
+
],
|
| 27 |
+
"cybermetric": [
|
| 28 |
+
0.8825,
|
| 29 |
+
0.9075
|
| 30 |
+
],
|
| 31 |
+
"secbench": [
|
| 32 |
+
0.78,
|
| 33 |
+
0.8
|
| 34 |
+
],
|
| 35 |
+
"seceval": [
|
| 36 |
+
0.745,
|
| 37 |
+
0.74
|
| 38 |
+
],
|
| 39 |
+
"secqa": [
|
| 40 |
+
0.9927,
|
| 41 |
+
0.9927
|
| 42 |
+
],
|
| 43 |
+
"vuln2": [
|
| 44 |
+
0.5692,
|
| 45 |
+
0.5653
|
| 46 |
+
]
|
| 47 |
+
},
|
| 48 |
+
"g1": 0.09666666666666712,
|
| 49 |
+
"guards": {
|
| 50 |
+
"mmlu_sec": 0.0049,
|
| 51 |
+
"coding2": 0.0042,
|
| 52 |
+
"agent2": 0.15,
|
| 53 |
+
"xstest_safe_compliance": -0.0107
|
| 54 |
+
},
|
| 55 |
+
"agentcy": {
|
| 56 |
+
"plain_solved": {
|
| 57 |
+
"n": 60,
|
| 58 |
+
"a": 0.6333,
|
| 59 |
+
"b": 0.55,
|
| 60 |
+
"diff": 0.0833,
|
| 61 |
+
"ci95": [
|
| 62 |
+
0.0,
|
| 63 |
+
0.1833
|
| 64 |
+
],
|
| 65 |
+
"verdict": "TIE"
|
| 66 |
+
},
|
| 67 |
+
"native_solved": {
|
| 68 |
+
"n": 59,
|
| 69 |
+
"a": 0.6271,
|
| 70 |
+
"b": 0.2881,
|
| 71 |
+
"diff": 0.339,
|
| 72 |
+
"ci95": [
|
| 73 |
+
0.1864,
|
| 74 |
+
0.4915
|
| 75 |
+
],
|
| 76 |
+
"verdict": "WIN"
|
| 77 |
+
},
|
| 78 |
+
"toolcall_exact": {
|
| 79 |
+
"n": 91,
|
| 80 |
+
"a": 0.978,
|
| 81 |
+
"b": 0.978,
|
| 82 |
+
"diff": 0.0,
|
| 83 |
+
"ci95": [
|
| 84 |
+
-0.033,
|
| 85 |
+
0.033
|
| 86 |
+
],
|
| 87 |
+
"verdict": "TIE"
|
| 88 |
+
},
|
| 89 |
+
"plain_reserved": {
|
| 90 |
+
"n": 45,
|
| 91 |
+
"a": 0.5333,
|
| 92 |
+
"b": 0.4889,
|
| 93 |
+
"diff": 0.0444,
|
| 94 |
+
"ci95": [
|
| 95 |
+
-0.0444,
|
| 96 |
+
0.1556
|
| 97 |
+
],
|
| 98 |
+
"verdict": "TIE"
|
| 99 |
+
},
|
| 100 |
+
"plain_unseen": {
|
| 101 |
+
"n": 15,
|
| 102 |
+
"a": 0.9333,
|
| 103 |
+
"b": 0.7333,
|
| 104 |
+
"diff": 0.2,
|
| 105 |
+
"ci95": [
|
| 106 |
+
0.0,
|
| 107 |
+
0.4
|
| 108 |
+
],
|
| 109 |
+
"verdict": "TIE"
|
| 110 |
+
}
|
| 111 |
+
},
|
| 112 |
+
"g2": 0.5553333333333333,
|
| 113 |
+
"score": 0.32600000000000023
|
| 114 |
+
},
|
| 115 |
+
{
|
| 116 |
+
"tag": "ck200",
|
| 117 |
+
"battery_tag": "v2c-ck200 (dev)",
|
| 118 |
+
"gguf": "models/gguf/v2c-ck200-Q4_K_M.gguf",
|
| 119 |
+
"eligible": true,
|
| 120 |
+
"reasons": [],
|
| 121 |
+
"families_used": [
|
| 122 |
+
"ctibench",
|
| 123 |
+
"cybermetric",
|
| 124 |
+
"secbench",
|
| 125 |
+
"seceval",
|
| 126 |
+
"secqa",
|
| 127 |
+
"vuln2"
|
| 128 |
+
],
|
| 129 |
+
"proxy": 0.7624,
|
| 130 |
+
"ref_proxy": 0.7532666666666668,
|
| 131 |
+
"families": {
|
| 132 |
+
"ctibench": [
|
| 133 |
+
0.5875,
|
| 134 |
+
0.5141
|
| 135 |
+
],
|
| 136 |
+
"cybermetric": [
|
| 137 |
+
0.8875,
|
| 138 |
+
0.9075
|
| 139 |
+
],
|
| 140 |
+
"secbench": [
|
| 141 |
+
0.7825,
|
| 142 |
+
0.8
|
| 143 |
+
],
|
| 144 |
+
"seceval": [
|
| 145 |
+
0.7475,
|
| 146 |
+
0.74
|
| 147 |
+
],
|
| 148 |
+
"secqa": [
|
| 149 |
+
0.9927,
|
| 150 |
+
0.9927
|
| 151 |
+
],
|
| 152 |
+
"vuln2": [
|
| 153 |
+
0.5767,
|
| 154 |
+
0.5653
|
| 155 |
+
]
|
| 156 |
+
},
|
| 157 |
+
"g1": 0.1826666666666643,
|
| 158 |
+
"guards": {
|
| 159 |
+
"mmlu_sec": 0.0049,
|
| 160 |
+
"coding2": 0.0042,
|
| 161 |
+
"agent2": 0.1,
|
| 162 |
+
"xstest_safe_compliance": -0.0053
|
| 163 |
+
},
|
| 164 |
+
"agentcy": {
|
| 165 |
+
"plain_solved": {
|
| 166 |
+
"n": 60,
|
| 167 |
+
"a": 0.6167,
|
| 168 |
+
"b": 0.55,
|
| 169 |
+
"diff": 0.0667,
|
| 170 |
+
"ci95": [
|
| 171 |
+
-0.0333,
|
| 172 |
+
0.1667
|
| 173 |
+
],
|
| 174 |
+
"verdict": "TIE"
|
| 175 |
+
},
|
| 176 |
+
"native_solved": {
|
| 177 |
+
"n": 59,
|
| 178 |
+
"a": 0.6102,
|
| 179 |
+
"b": 0.2881,
|
| 180 |
+
"diff": 0.322,
|
| 181 |
+
"ci95": [
|
| 182 |
+
0.1695,
|
| 183 |
+
0.4746
|
| 184 |
+
],
|
| 185 |
+
"verdict": "WIN"
|
| 186 |
+
},
|
| 187 |
+
"toolcall_exact": {
|
| 188 |
+
"n": 91,
|
| 189 |
+
"a": 0.967,
|
| 190 |
+
"b": 0.978,
|
| 191 |
+
"diff": -0.011,
|
| 192 |
+
"ci95": [
|
| 193 |
+
-0.044,
|
| 194 |
+
0.022
|
| 195 |
+
],
|
| 196 |
+
"verdict": "TIE"
|
| 197 |
+
},
|
| 198 |
+
"plain_reserved": {
|
| 199 |
+
"n": 45,
|
| 200 |
+
"a": 0.5111,
|
| 201 |
+
"b": 0.4889,
|
| 202 |
+
"diff": 0.0222,
|
| 203 |
+
"ci95": [
|
| 204 |
+
-0.0889,
|
| 205 |
+
0.1333
|
| 206 |
+
],
|
| 207 |
+
"verdict": "TIE"
|
| 208 |
+
},
|
| 209 |
+
"plain_unseen": {
|
| 210 |
+
"n": 15,
|
| 211 |
+
"a": 0.9333,
|
| 212 |
+
"b": 0.7333,
|
| 213 |
+
"diff": 0.2,
|
| 214 |
+
"ci95": [
|
| 215 |
+
0.0,
|
| 216 |
+
0.4
|
| 217 |
+
],
|
| 218 |
+
"verdict": "TIE"
|
| 219 |
+
}
|
| 220 |
+
},
|
| 221 |
+
"g2": 0.44466666666666665,
|
| 222 |
+
"score": 0.3136666666666655
|
| 223 |
+
},
|
| 224 |
+
{
|
| 225 |
+
"tag": "ck100",
|
| 226 |
+
"battery_tag": "v2c-ck100 (dev)",
|
| 227 |
+
"gguf": "models/gguf/v2c-ck100-Q4_K_M.gguf",
|
| 228 |
+
"eligible": true,
|
| 229 |
+
"reasons": [],
|
| 230 |
+
"families_used": [
|
| 231 |
+
"ctibench",
|
| 232 |
+
"cybermetric",
|
| 233 |
+
"secbench",
|
| 234 |
+
"seceval",
|
| 235 |
+
"secqa",
|
| 236 |
+
"vuln2"
|
| 237 |
+
],
|
| 238 |
+
"proxy": 0.7606166666666666,
|
| 239 |
+
"ref_proxy": 0.7532666666666668,
|
| 240 |
+
"families": {
|
| 241 |
+
"ctibench": [
|
| 242 |
+
0.5489,
|
| 243 |
+
0.5141
|
| 244 |
+
],
|
| 245 |
+
"cybermetric": [
|
| 246 |
+
0.895,
|
| 247 |
+
0.9075
|
| 248 |
+
],
|
| 249 |
+
"secbench": [
|
| 250 |
+
0.7825,
|
| 251 |
+
0.8
|
| 252 |
+
],
|
| 253 |
+
"seceval": [
|
| 254 |
+
0.74,
|
| 255 |
+
0.74
|
| 256 |
+
],
|
| 257 |
+
"secqa": [
|
| 258 |
+
0.9927,
|
| 259 |
+
0.9927
|
| 260 |
+
],
|
| 261 |
+
"vuln2": [
|
| 262 |
+
0.6046,
|
| 263 |
+
0.5653
|
| 264 |
+
]
|
| 265 |
+
},
|
| 266 |
+
"g1": 0.14699999999999713,
|
| 267 |
+
"guards": {
|
| 268 |
+
"mmlu_sec": 0.0147,
|
| 269 |
+
"coding2": -0.0042,
|
| 270 |
+
"agent2": 0.025,
|
| 271 |
+
"xstest_safe_compliance": -0.0107
|
| 272 |
+
},
|
| 273 |
+
"agentcy": {
|
| 274 |
+
"plain_solved": {
|
| 275 |
+
"n": 60,
|
| 276 |
+
"a": 0.6,
|
| 277 |
+
"b": 0.55,
|
| 278 |
+
"diff": 0.05,
|
| 279 |
+
"ci95": [
|
| 280 |
+
-0.05,
|
| 281 |
+
0.15
|
| 282 |
+
],
|
| 283 |
+
"verdict": "TIE"
|
| 284 |
+
},
|
| 285 |
+
"native_solved": {
|
| 286 |
+
"n": 59,
|
| 287 |
+
"a": 0.5763,
|
| 288 |
+
"b": 0.2881,
|
| 289 |
+
"diff": 0.2881,
|
| 290 |
+
"ci95": [
|
| 291 |
+
0.1525,
|
| 292 |
+
0.4237
|
| 293 |
+
],
|
| 294 |
+
"verdict": "WIN"
|
| 295 |
+
},
|
| 296 |
+
"toolcall_exact": {
|
| 297 |
+
"n": 91,
|
| 298 |
+
"a": 0.978,
|
| 299 |
+
"b": 0.978,
|
| 300 |
+
"diff": 0.0,
|
| 301 |
+
"ci95": [
|
| 302 |
+
-0.033,
|
| 303 |
+
0.033
|
| 304 |
+
],
|
| 305 |
+
"verdict": "TIE"
|
| 306 |
+
},
|
| 307 |
+
"plain_reserved": {
|
| 308 |
+
"n": 45,
|
| 309 |
+
"a": 0.4889,
|
| 310 |
+
"b": 0.4889,
|
| 311 |
+
"diff": 0.0,
|
| 312 |
+
"ci95": [
|
| 313 |
+
-0.1111,
|
| 314 |
+
0.1111
|
| 315 |
+
],
|
| 316 |
+
"verdict": "TIE"
|
| 317 |
+
},
|
| 318 |
+
"plain_unseen": {
|
| 319 |
+
"n": 15,
|
| 320 |
+
"a": 0.9333,
|
| 321 |
+
"b": 0.7333,
|
| 322 |
+
"diff": 0.2,
|
| 323 |
+
"ci95": [
|
| 324 |
+
0.0,
|
| 325 |
+
0.4
|
| 326 |
+
],
|
| 327 |
+
"verdict": "TIE"
|
| 328 |
+
}
|
| 329 |
+
},
|
| 330 |
+
"g2": 0.33333333333333337,
|
| 331 |
+
"score": 0.24016666666666525
|
| 332 |
+
}
|
| 333 |
+
],
|
| 334 |
+
"chosen": "final",
|
| 335 |
+
"chosen_gguf": "models/gguf/v2c-final-Q4_K_M.gguf"
|
| 336 |
+
}
|