Offlin33er/qwen25-coder-7b-solidity-audit

Qwen2.5-Coder-7B — Solidity Audit Adapter (QLoRA, r=32)

A LoRA adapter for Qwen/Qwen2.5-Coder-7B-Instruct fine-tuned to produce structured Solidity security-audit reports: given a contract snippet, it emits ## Finding: <type> with functionality analysis, Description, and Recommendation — or a clear "no vulnerability identified" verdict.

Trained as part of the Solidity Audit Scanner project.

Newer versions available: V2 (canonical 8-class taxonomy, 48.5% type match) and V3 (verified fallback resolution, best detection F1 0.986). This V1 card documents the original run.

Training

Method	TRL SFTTrainer, QLoRA (NF4, double quant, bf16 compute)
LoRA	r=32, alpha=32, dropout 0.05, all attn + MLP projections
Data	Offlin33er/solidity-audit-instruct — 2,224 train / 248 held-out, built from Solodit-audited code
Epochs	2 (278 steps, effective batch 16, seq len 2048)
Optimizer	paged AdamW 8-bit, LR 2e-4, cosine, warmup 10
Hardware	1× Nvidia A10G (24 GB), ~2h16m wall
Tracking	trackio dashboard

Training loss (trackio curve): 1.21 @ step 5 → 0.87 @ step 15 → 0.64 @ step 48 → 0.56 @ step 115 → ~0.52 at the end of the cosine tail; token accuracy 0.71 → 0.86.

Evaluation (measured, n=100 stratified)

Stratified sample of the held-out test split — 70 vulnerable / 30 clean rows, seed 42, greedy decoding, exact training-time system prompt, 512 new tokens. Full per-row records in eval_results.json.

Metric	Result
Vulnerability detection precision	1.00 (70/70 flagged rows true)
Detection recall	1.00 (no vulnerable row missed)
False-positive rate on clean code	0.00 (30/30 clean rows correctly cleared)
Finding-type match vs reference taxonomy	0.30 (21/70, normalized substring/keyword match)

Read this honestly:

• The perfect detection score is largely a dataset artifact. The held-out rows come from real audit reports where vulnerable snippets exhibit strong surface signals (the specific flawed construct the report was about), and clean rows are curated non-vulnerable code. Separating the two is much easier than real-world auditing, where the hard part is finding what nobody reported. Do not read F1 = 1.0 as "production-ready auditor."
• The useful negative result is the 30% type match. The model detects that code is flawed reliably but labels the flaw coarser than the reference taxonomy (e.g., reference "integer overflow" → predicted "arithmetic error"; reference "reentrancy" → predicted "frontrunning"). Improving fine-grained typing needs a larger, normalized-type training set — the dataset's ~200 long-tail types have ≤6 examples each.
• Two prior eval rounds (r2: mismatched system prompt + 256-token truncation → 53% unparseable; r3/r4: corrected protocol) are recorded in the repo history; r4v2 (greedy, deterministic) reproduced r3's counts and persisted per-row records with sample generations.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B-Instruct", dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "Offlin33er/qwen25-coder-7b-solidity-audit")
tok = AutoTokenizer.from_pretrained("Offlin33er/qwen25-coder-7b-solidity-audit")

messages = [
    {"role": "system", "content": "You are a senior smart-contract security auditor. Audit the provided Solidity code. If a vulnerability is present, report it as '## Finding: <type>' followed by the functionality, a Description, and a Recommendation. If none is found, state that no vulnerability was identified and describe what the code does."},
    {"role": "user", "content": "Audit the following Solidity code for security vulnerabilities.\n\n```solidity\n<YOUR CODE>\n```"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Limitations

• Trained on ~2.5K audited snippets — narrow coverage; ~200 vulnerability types appear ≤6 times each.
• Sequence length 2048 truncates some longer descriptions during training.
• Eval was n=100 stratified from the held-out split, not a public benchmark; no cross-benchmark score is claimed.
• It does not replace a professional audit. Findings are candidates for human review.

Defensive security tooling: analyzes code you paste in. Only audit contracts you are authorized to review.