reinaldosaraiva/review-v3d-study

review-v3d: training a silent-by-default 8B LoRA code reviewer

A 12-plan, single-author study (September–October 2026) of fine-tuning a LoRA adapter on Qwen3-8B to review single diff hunks in Python and C and emit structured findings, with false positives on clean code as the primary metric.

This repository publishes the study only. No adapter weights and no training data are released (see Why no weights or data).

TL;DR

• The production adapter, review-v3d, detects 47.8 % of defects in a 546-hunk gold set with 2.0 % false positives on clean hunks (1.8 % on a 304-row held-out split). The previous adapter reached 1.2 % recall at 0.7 % FP on the same measurement.
• Getting there took three data fixes, not hyperparameters. The decisive changes were (1) a loss that weighs the "open a finding or not" decision explicitly and pairs buggy/fixed examples, (2) removing the teacher model's 70 % finding prior from the labels, and (3) oversampling subtle boolean mutations.
• The adapter is narrow. It reliably catches the mutation families it was trained on (removed negation, ==→!=, <→<= boundaries, null-check inversion) and little else: 1 of 28 non-seeded real defects.
• Trying to teach real defects from bug-fix commits ("the first line removed by the fix is the defect") raised non-seeded detection from 1/28 to 24/28, but always with 10–17 % false positives. Three retrains that changed the prompt, the explanation templates, and the commit subject did not move the FP rate. The examples themselves caused it. We report this as a negative result.

Task and output

• Unit: one diff hunk, no repository context. Lines prefixed with + are the code under review.
• Output: strict JSON, {"findings": []} when clean, or findings with axis, severity (P0–P3), line, cwe, claim, why. The verdict, ISO/IEC 5055 characteristics and OWASP Top 10:2021 categories are derived from the CWE, never emitted (307-CWE union taxonomy).
• Intended role: a cheap, local layer-1 triage in front of a frontier-model reviewer. It is not a standalone reviewer.

Model and recipe

Component	Value
Base	Qwen3-8B, 4-bit (MLX), dequantized for PyTorch training
Adapter	LoRA r 8, alpha 16, all 7 projections × 36 layers (252 modules)
Optimizer	Adam, lr 2e-4, warmup 10, gradient accumulation 2 with the buggy/fixed pair in the same step
Loss	(L_b + L_a)/2 + 1.0 · softplus(z_a − z_b), with L_x = L_mean + 1.0 · L_dec
Prompt	rubric kept intact, CWE catalog reduced to ids (9,753 → 3,260 tokens)
Checkpoint selection	decision NLL on validation; chosen checkpoint 4,500 of 4,856
Inference	greedy, thinking disabled, float32 (see finding 6)
Hardware	training on one NVIDIA L40 (≈7 h per full run); local inference on Apple Silicon (MLX)

L_dec is the negative log-likelihood of the single decision token right after {"findings, and z_a, z_b are the finding logits on the fixed (a) and buggy (b) member of a contrastive pair.

Data (described, not released)

• Corpus: about 46k hunks. Most are from Linux kernel history, plus FRR, OVS/OVN, Django, FastAPI, Flask, requests, and the author's private repositories.
• Labels: a teacher model (Qwen3-Coder-30B) applied the review rubric. Seeded defects come from a deterministic mutation engine with known line, CWE and severity. Clean negatives come from merged OSS code. Positives from security advisories come from Django CVE fixes, with the diff inverted.
• Contrastive set: buggy/fixed pairs that differ by one seeded mutation, restricted to mutations that change code (not strings or comments).
• Gold (curated): 546 hunks: 139 base, 120 clean OSS, 200 seeded, 87 from advisories. 353 Python, 153 C. Split by commit, never by hunk.

Evaluation gates

Gate	Meaning	Threshold
G5	7 fixed, known-clean controls stay silent	7/7
A3	detection on 20 buggy hunks	≥ 8/20
Paired	buggy scores above its fixed twin (20 pairs; 15 with a real code change)	≥ 12/15
FP after	findings on the fixed member of the pairs	≤ 2/20
FP clean	findings on clean gold hunks	≤ 5.7 %

Trajectory

Phase	What happened	Outcome
P001–P005	Emission diagnostics. The first adapter learned to be silent: perfect on G5 but near-zero recall.	silent baseline
P006–P008	A separate refactor adapter that proposes patches. On a synthetic, line-anchored 100-case held-out, valid patches rose 0 → 26 → 76 %, and to 50 % on 20 historical cases.	experimental only
P009	Threshold/decoding calibration. The reviewer did not discriminate buggy from fixed at all.	FAILED
P010	Trainability study on a 64-pair overfit set. Decision accuracy rose 0.50 → 0.92 with the loss above.	recipe found
P011	Full training with three data fixes (table below).	v3d approved
P012	v3d promoted with rollback. Three retrains to learn real (non-seeded) defects failed the FP gate.	negative result

P011: from teacher prior to a usable reviewer

Adapter	Change	G5	A3	Paired (code)	FP after	FP clean	Recall
v3	P010 recipe, full data	7/7	16/20	—	8/20	20.7 %	77.3 %
v3b	contrastive pairs only where code changes	6/7	15/20	12/15	7/20	21.4 %	67.3 %
v3c	weak teacher findings turned into empty labels	7/7	11/20	10/15	0/20	2.4 %	33.9 %
v3d	boolean-operator family oversampled ×2–×4	7/7	14/20	14/15	1/20	2.0 %	47.8 %

P012: learning real defects from fix commits

Metric	v3d	v3e	v3f	v3g
Change	—	+413 positives from fix commits	neutral subject in prompt, claim cites the code	15 why templates per defect class
FP clean, held-out	1.8 %	13.5 %	10.4 %	16.6 %
Non-seeded defects found (of 28)	1	3	19	24
Exact line on hits	11.4 %	69.9 %	67.0 %	62.5 %
Paired (code, of 15)	14	14	13	11

None of v3e/v3f/v3g was promoted.

Findings

1. Mean token loss hides the decision. The token that decides whether a finding opens got weight 1/77 against 1/12 for the rest. Adding an explicit decision term (L_dec) and a paired margin term was what made the adapter learn to discriminate (0.50 → 0.92 decision accuracy on the overfit set).
2. Long prompts dilute training. 95 % of every example was system prompt. Keeping CWE ids and dropping their names saved 8,053 tokens per review with no loss in decision quality.
3. The teacher's prior becomes the student's prior. 70 % of teacher labels had a finding, and the student reproduced it (21 % FP). The validation split shared the bias, so it never flagged the problem. Turning weak findings into empty labels cut FP to 2.4 %.
4. Without that prior, subtle operators vanish. Boolean mutations (and/or, ==/!=, dropped not) dropped to zero. Oversampling them to the size of the largest family recovered them without reopening FP.
5. Check the test set for no-op mutations. 5 of the 20 paired test cases changed only a string or a comment. For those, not reacting is correct, and gates must account for it.
6. Parity across backends needs float32. In bf16, line-number digits tie exactly and greedy decoding diverged between MLX and PyTorch (44/47), and even between dtypes on one machine (43/47). In float32, 47/47 matched, with |Δ| ≤ 0.00016 nat on the decision margin.
7. Weak fix-commit labels trade recall for false positives, and the trade does not depend on formulation. Our first diagnosis blamed the commit subject ("Fixed …") leaking into the prompt. A leak check refuted it. Our second blamed a dominant why template. Removing it (89 % → 25 % share) made FP worse. The consistent signal: with the 403 fix-commit positives, FP is 10–17 %; without them, 1.8 %. "First removed line = defect" labels code that, isolated in a hunk, is indistinguishable from clean code.
8. Citing the code in claim works. Requiring file:line plus the literal target line removed all copies of the commit subject from the outputs (15 → 0) and made non-seeded detections genuine.

Qualitative probe on unseen code

We ran v3d on code from outside its training distribution and seeded single-line bugs by hand:

Seeded change	Trained family?	Result
if not ok: → if ok: (guard)	yes (removed negation)	flagged, correct line, CWE-480
x == y → x != y in a lookup	yes	flagged, CWE-480
i < n → i <= n in a C loop	yes	flagged, CWE-787 off-by-one
a or not b → a and not b	no (trained direction is and→or)	missed
100 - pct → 100 + pct	no	correct line, wrong explanation (reused the negation template)
split('/')[0] → [1]	partly (index shift)	missed
range(..., len(xs) + 1)	no	missed
clean edits, a real fix commit	—	silent (correct)

The pattern matches the metrics: high precision, recall concentrated on the trained mutation families.

Limitations

• Hunk-only: no cross-file or repository context.
• Python and C only.
• Recall on non-seeded, real-world defects is very low (1/28).
• Line localization is imprecise (11.4 % exact on held-out hits).
• Most positives are synthetic. Synthetic evaluations overestimate real-world performance.
• Single author, single gold set, no human inter-annotator agreement.

Why no weights or data

The training corpus includes code from private repositories and GPL-2.0 code (Linux kernel, FRR). Releasing the data would expose private code, and releasing the weights raises memorization and license questions. A weights release would require retraining on permissively licensed code only. That is possible future work and is not part of this release.

Process

The work ran as frozen-contract sessions: each plan froze gates and thresholds before training, and a failed gate could only be addressed by a written amendment followed by a new run (an "overrun rule" capped at 8 sessions per plan). Every promoted artifact is pinned by SHA-256, and rollback was kept intact throughout. Several of the negative results above exist because this process forbade changing a threshold after seeing the result.

Citation

@misc{saraiva2026reviewv3d,
  author = {Reinaldo Saraiva},
  title  = {review-v3d: training a silent-by-default 8B LoRA code reviewer},
  year   = {2026},
  note   = {Study report, Hugging Face}
}

Text licensed under CC BY 4.0.