# Security benchmark protocol Protocol: `balanced-letter-ranking-v1`; frozen selection seed: `20261109`. | Suite | Questions | Correct answer positions | | --- | --- | --- | | Held-out Pentesting Explanations | 176 | A: 44, B: 44, C: 44, D: 44 | | SecEval adapted subset | 200 | A: 50, B: 50, C: 50, D: 50 | ## Scoring Each question is shown with four choices using the base model's chat template. The reasoning prefix is closed before scoring. The evaluator compares the next-token logits for A, B, C, and D, and selects the highest. Each label is verified to be a single token in its actual prompt context. **This measures forced direct-answer ranking, with no generated reasoning.** The base is evaluated by disabling the trained LoRA adapter; the fine-tune enables that same adapter. Both use the same NF4 double-quantized weights, FP16 compute, prompt, GPU, and software. Dropout is disabled. Maximum prompt length is 768 tokens. Base/adapter evaluation order alternates. Conditional A–D probabilities and the probability mass of those four tokens within the full vocabulary are recorded. Each suite has a 25% uniform-guessing expectation and 25% majority-label baseline. Reported accuracy includes descriptive Wilson 95% intervals. Paired corrected/regressed counts and topic breakdowns are provided. Topics overlap, and a small or dependent subset does not support broad population claims. ## Selection and overlap checks Pentesting questions come from the held-out split. SecEval selection excludes invalid or multiple-answer cases and uses a seeded round-robin sample across primary topics. Questions with detected position-dependent options, matching training question text/CVE IDs, a training-stem word-4-gram Jaccard similarity of at least 0.8, or overlength prompts are excluded. These lexical checks do not detect every semantic duplicate or base-model pretraining contamination. Choices are reordered to balance labels. [The frozen manifest](./benchmark-plan.json) records source IDs, zero-based original choice indices in their new order, answer labels, topics, prompt lengths, and hashes. It does not redistribute question text. Full question text remains local. Selection counts and exclusions are in the manifest. ## Sources - [SecEval by XuanwuAI](https://huggingface.co/datasets/XuanwuAI/SecEval/tree/205dab7b0888a06f4b53ca7d9c7093e1326683e1): evaluation-only dataset, CC BY-NC-SA. See the [authors' project](https://xuanwuai.github.io/SecEval/). The adapted single-answer subset and this scoring method are **not official SecEval leaderboard results**. The dataset's terms are separate from the model's MIT license. - [Pentesting Explanations by Christopher Dickinson / theelderemo](https://huggingface.co/datasets/theelderemo/pentesting-explanations/tree/7a52367ca4491404603b6d8f2474099d9eb1a4a3): Apache-2.0. Synthetic answers have not been independently fact-checked. ## Execution and reproducibility The evaluator runs automatically after the training process exits, before CPU merge/export, so it does not compete with training for VRAM. Per-question paired checkpoints allow resumption only with identical model, adapter, suite, scorer, and library fingerprints. Results contain adapter SHA-256, pinned base revision, software versions, device, and per-question predictions without question text. Published scripts are under `evaluation/`. To reproduce in the original workspace layout, place them in `deepseek14b/` alongside `model_config.json`, `base/`, `data/train.jsonl`, `data/eval.jsonl`, and `outputs/adapter/`. Fetch the pinned source datasets in the manifest; place SecEval's `questions.json` in `data/benchmark-sources/SecEval/`. Run `benchmark_data.py`, then `run_benchmarks.py --validate-only`, then `run_benchmarks.py` after training. Verify the generated manifest hashes against this release. The corpus hashes are provided for verification; the corpus and SecEval question text are not bundled with these reports. ## Scope Scores apply to the **NF4 runtime**, not to the Q4_K_M GGUF. They are not generated-answer pass@1 or measurements of normal R1 reasoning, tool calling, exploit discovery, or current CVE completeness. No held-out iOS 27 advisory score is available. The additional 10 generated probes are diagnostics, not a validated factual-accuracy benchmark.