README.md

---
base_model: Qwen/Qwen3-8B
library_name: peft
tags:
  - lora
  - code-correctness
  - livecodebench
  - classifier
---

# LCB correctness classifiers (v2)

Yes/no code-correctness classifiers for LiveCodeBench rollouts, one LoRA per
generator model, all on **Qwen/Qwen3-8B** (thinking disabled). Trained on
rollouts from `samuki-hf/temperature-sweep-data` and `samuki-hf/thinking-rollouts`.

Prompt = problem statement + candidate solution (+ tail of the generator's
reasoning trace for the `-trace` variants) + "Does this candidate solution
correctly solve the problem? Answer Yes or No." Score = log-odds of the
Yes-token family vs No at the first answer position (chat template with
`enable_thinking=False` — the empty `<think></think>` block is part of the
prompt). Training: natural label distribution, problems split 500/60/154
(stratified by difficulty × testtype × pooled pass-rate), ≤100 samples/problem
across all temps, LoRA r16/α32, lr 1e-4, effective batch 128, 1 epoch.

| subfolder (classifier for) | input | AUROC | acc@0 | ECE |
|---|---|---|---|---|
| `qwen25-coder-3b-instruct` | code | 0.968 | 0.926 | 0.036 |
| `qwen25-coder-7b-instruct` | code | 0.921 | 0.853 | 0.085 |
| `qwen3-4b-think` | code | 0.887 | 0.877 | 0.052 |
| `qwen3-14b-think` | code | 0.888 | 0.889 | 0.055 |
| `qwen3-4b-nothink` | code | 0.895 | 0.823 | 0.116 |
| `qwen3-14b-nothink` | code | 0.919 | 0.828 | 0.096 |
| `qwen3-4b-think-trace` | code + trace | (eval pending) | | |
| `qwen3-14b-think-trace` | code + trace | (eval pending) | | |

Metrics: 154 held-out problems, full candidate pools (all samples × temps),
scored with the same prompt path as training.

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(model, "Sheppp/lcb-correctness-classifiers", subfolder="qwen25-coder-7b-instruct")
```