kev-4b-code-verify-v1

A fine-tune of jaredpalmer/kev-4b (4B, trained on a bf16 backbone on a single L4): a Jev-style decision model that judges whether code written by a cheap coding model is correct, so a router can keep the answer local or escalate it. One forward pass, a calibrated probability, no generated text.

Trained on execution-labelled data: coder attempts at HumanEvalPack (Python) tasks, labelled by running each task's hidden test suite. The model never sees the hidden tests; its input is what a router can compute itself.

How to ask it

The fine-tune binds these exact strings. One noul question (probability that the statement holds):

{"type": "noul", "instructions": "Does the code correctly and completely implement the request, so it would pass a thorough hidden test suite including edge cases? Judge from the request, the code and the checks shown."}

State (an object; Kev renders it as key: value lines):

{"request": "<the task, verbatim>",
  "code": "<the model's code>",
  "checks": {"compiles": true, "defines_requested_function": true, "passes_examples_in_request": true,
             "generated_edge_case_tests": "7 of 8 passed",
             "note": "edge-case tests were written by a small model from the request alone; some may be wrong"}}

generated_edge_case_tests comes from a small model writing ~8 asserts from the request only, then running them. Serve with Kev's runtime (python -m kev.serve --run jtatman/kev-4b-code-verify-v1, a TypeSafe /v1/systemone endpoint) or kev.predictors.LocalPredictor. The checkpoint carries its fitted temperature.

Results

Unseen coder (llama-3.1-8b-instruct, never in training): 488 attempts, pass rate 0.57. "Kept local @X% err" = share of attempts a router can accept, highest score first, with at most X% of them wrong. evidence score = (request examples passed + generated-assert pass rate) / 2.

score	AUROC	Brier	ECE	kept local @2% err	@5%	@10%
baseline kev	0.970	0.060	0.03	0.31	0.51	0.62
fine-tuned kev	0.974	0.049	0.03	0.34	0.54	0.64
evidence score	0.944	0.091	0.09	0.00	0.21	0.62
mean(evidence, baseline)	0.971	0.066	0.06	0.31	0.50	0.63
mean(evidence, fine-tuned)	0.973	0.060	0.06	0.35	0.51	0.64

Development (coders seen in training, tasks not seen):

score	AUROC	Brier	ECE	kept local @2% err	@5%	@10%
baseline kev	0.960	0.052	0.08	0.82	0.87	0.95
fine-tuned kev	0.958	0.032	0.04	0.74	0.90	0.95
evidence score	0.960	0.037	0.07	0.79	0.90	0.92
mean(evidence, baseline)	0.965	0.037	0.08	0.79	0.90	0.95
mean(evidence, fine-tuned)	0.968	0.031	0.05	0.80	0.90	0.95

Numbers are on small sets; see the project repo for task-bootstrap confidence intervals.

Recommended use

Average this model's probability with the execution evidence score, and tune the accept threshold per cheap-tier coder on a few hundred of that coder's execution-labelled attempts: a fixed threshold's error rate depends on how often the coder fails.

Training

• Init: jaredpalmer/kev-4b@139fdd9 (LoRA r=16 + pointer head, base Qwen/Qwen3.5-4B-Base), Kev's own trainer (kev.train, commit f2bb629d).
• Data: 1506 execution-labelled records (task-grouped split; llama-3.1-8b held out) + 2000 public decision-v7 replay records against forgetting.
• One epoch, lr 2e-05, batch 1 x accum 8, gradient checkpointing, bf16 autocast, state limit 1664 tokens, frozen backbone in bf16.
• Temperature fitted on a held-out calibration split (min NLL).
• Hardware: one NVIDIA L4 (Google Colab).

Training data provenance

Code attempts whose correctness this model learned to judge (labels from executing hidden tests):

source	model	license	training rows
or:mistral-small-3.2-24b	mistralai/Mistral-Small-3.2-24B-Instruct-2506	Apache-2.0	297
or:qwen3-235b-a22b	Qwen/Qwen3-235B-A22B-Instruct-2507	Apache-2.0	292
or:qwen3-coder-30b-a3b	Qwen/Qwen3-Coder-30B-A3B-Instruct	Apache-2.0	238
qwen2.5-coder:3b	Qwen/Qwen2.5-Coder-3B-Instruct	Qwen Research License	344
reference-buggy	bigcode/humanevalpack buggy solutions	MIT	115
reference-canonical	bigcode/humanevalpack canonical solutions	MIT	115
ternary-qwen3.8-27b	prism-ml/Ternary-Bonsai-2-27B-gguf (PTQ1_0; base Qwen/Qwen3.8-27B)	Apache-2.0	105

Built with Qwen. Training data includes outputs of Qwen2.5-Coder-3B-Instruct, used under the Qwen Research License Agreement (section 4b). The held-out test coder (Llama 3.1 8B Instruct) was never used for training.

Limitations

• Python function-level tasks only (HumanEvalPack); other languages, repos and multi-file changes are untested.
• The coder pool is 8-10 models of 3B-235B parameters; accept thresholds do not transfer across coders.
• Trained on this question and state layout; other phrasings work less well.

Credits

Kev by Jared Palmer (jaredpalmer/kev, Apache-2.0); Qwen3.5 base (Apache-2.0); HumanEvalPack (bigcode). Project: https://github.com/jtatman/kev-code-verify