deepseek-r1-14b-security / benchmark-protocol.md
Volpidash's picture
Publish security benchmark protocol, measured coverage and available results
8a2e047 verified
|
Raw History Blame Contribute Delete
4.37 kB

Security benchmark protocol

Protocol: balanced-letter-ranking-v1; frozen selection seed: 20261109.

Suite Questions Correct answer positions
Held-out Pentesting Explanations 176 A: 44, B: 44, C: 44, D: 44
SecEval adapted subset 200 A: 50, B: 50, C: 50, D: 50

Scoring

Each question is shown with four choices using the base model's chat template. The reasoning prefix is closed before scoring. The evaluator compares the next-token logits for A, B, C, and D, and selects the highest. Each label is verified to be a single token in its actual prompt context. This measures forced direct-answer ranking, with no generated reasoning.

The base is evaluated by disabling the trained LoRA adapter; the fine-tune enables that same adapter. Both use the same NF4 double-quantized weights, FP16 compute, prompt, GPU, and software. Dropout is disabled. Maximum prompt length is 768 tokens. Base/adapter evaluation order alternates. Conditional A–D probabilities and the probability mass of those four tokens within the full vocabulary are recorded.

Each suite has a 25% uniform-guessing expectation and 25% majority-label baseline. Reported accuracy includes descriptive Wilson 95% intervals. Paired corrected/regressed counts and topic breakdowns are provided. Topics overlap, and a small or dependent subset does not support broad population claims.

Selection and overlap checks

Pentesting questions come from the held-out split. SecEval selection excludes invalid or multiple-answer cases and uses a seeded round-robin sample across primary topics. Questions with detected position-dependent options, matching training question text/CVE IDs, a training-stem word-4-gram Jaccard similarity of at least 0.8, or overlength prompts are excluded. These lexical checks do not detect every semantic duplicate or base-model pretraining contamination.

Choices are reordered to balance labels. The frozen manifest records source IDs, zero-based original choice indices in their new order, answer labels, topics, prompt lengths, and hashes. It does not redistribute question text. Full question text remains local. Selection counts and exclusions are in the manifest.

Sources

Execution and reproducibility

The evaluator runs automatically after the training process exits, before CPU merge/export, so it does not compete with training for VRAM. Per-question paired checkpoints allow resumption only with identical model, adapter, suite, scorer, and library fingerprints. Results contain adapter SHA-256, pinned base revision, software versions, device, and per-question predictions without question text.

Published scripts are under evaluation/. To reproduce in the original workspace layout, place them in deepseek14b/ alongside model_config.json, base/, data/train.jsonl, data/eval.jsonl, and outputs/adapter/. Fetch the pinned source datasets in the manifest; place SecEval's questions.json in data/benchmark-sources/SecEval/. Run benchmark_data.py, then run_benchmarks.py --validate-only, then run_benchmarks.py after training. Verify the generated manifest hashes against this release. The corpus hashes are provided for verification; the corpus and SecEval question text are not bundled with these reports.

Scope

Scores apply to the NF4 runtime, not to the Q4_K_M GGUF. They are not generated-answer pass@1 or measurements of normal R1 reasoning, tool calling, exploit discovery, or current CVE completeness. No held-out iOS 27 advisory score is available. The additional 10 generated probes are diagnostics, not a validated factual-accuracy benchmark.