| # Wit-Coder-4B: evaluation details |
|
|
| ## Recorded benchmark results |
|
|
| Both columns are project measurements, not scores copied from the upstream model card. |
|
|
| | Metric | Qwen3.5-4B | Wit-Coder-4B / DPO-2 | |
| |---|---:|---:| |
| | MMLU, 5-shot accuracy | 69.67% | 70.16% | |
| | GSM8K, 8-shot CoT, strict match | 84.08% | 82.64% | |
| | GSM8K, flexible extraction | 84.15% | 82.79% | |
| | HumanEval base tests, **modified bounded protocol**, pass@1 | 87.80% (144/164) | 90.85% (149/164) | |
| | HumanEval plus tests, **modified bounded protocol**, pass@1 | 81.10% (133/164) | 87.20% (143/164) | |
|
|
| MMLU and recorded HumanEval scores increased; GSM8K decreased. This is not improvement across every benchmark. |
| MBPP scores are not reported: complete samples and scored outputs were not recovered from the interrupted runs. |
|
|
| ### Protocol and interpretation |
|
|
| - MMLU/GSM8K: `lm-eval` 0.4.13, raw-completion prompts, seed 1234, 5/8 exemplars respectively, |
| greedy decoding, maximum context 32,768; GSM8K generation limit 256 tokens. Longest recorded MMLU context: 3,138 tokens. |
| - Base requests used an SSH tunnel; DPO-2 requests ran on the pod against localhost. Package versions and protocol settings match, |
| but these were separate sessions. No inference-speed comparison is implied. |
| - HumanEval: `evalplus` 0.3.1 prompt/sanitization, one greedy chat completion per task, thinking enabled, |
| maximum 16,384 generated tokens. Base had 9 length-truncated completions; DPO-2 had 7. |
| - **These are not official, unmodified EvalPlus scores.** Execution used isolated network-less trials with 512 MB memory, |
| 64 PIDs and one CPU. HumanEval/15 and /130 were split into input chunks, resetting candidate globals between chunks. |
| No HumanEval test inputs were omitted. The canonical HumanEval/32 solution also failed validation in this environment, |
| including the upstream evaluator. |
| - Frozen calibration SHA256: `f4a330c8c556d1d844ac0f8c29c3db0516379f0e8bf30abb6e5bacbe3bf45283`. |
| It also lists three omitted MBPP inputs; no MBPP score is claimed here. A candidate shares a trial with its own runner; |
| this is not adversarially tamper-proof grading. |
| - These scores concern the served DPO-2 checkpoint, **not the quantized GGUF artifacts**. |
|
|
| ## In-house coding-agent panel |
|
|
| 42 tasks × 3 seeds = 126 episodes: 90 solvable and 36 blocked by design, executed and graded in a network-less sandbox. |
| Panel IDs and registered held-out families were excluded from training-data construction. Multiple checkpoints were measured |
| on this panel and its results informed later experiments; this is a development comparison, not a newly sealed final test. |
| Three seeds do not make 126 independent tasks. ID/family exclusion does not establish complete semantic decontamination. |
|
|
| | Outcome | Qwen3.5-4B | DPO-2 | |
| |---|---:|---:| |
| | Task success, including successful escalation | 40/126 | 71/126 | |
| | Solvable tasks solved | 40/90 | 60/90 | |
| | Blocked tasks successfully escalated | 0/36 | 11/36 | |
| | Episodes ending at the per-turn generation limit | 34/126 | 7/126 | |
| | Episodes ending at the maximum turn count | 19/126 | 4/126 | |
| | Protected-file integrity failures | 28/126 | 25/126 | |
| | MATH-500, first 100 problems | 65/100 | 82/100 | |
|
|
| Generation-limit endings are not a direct measurement of semantic reasoning loops. MATH uses a 100-problem subset, not all 500. |
| No statistical significance or broad generalization claim is established by these small panel comparisons. |
|
|
| ## Training and model selection |
|
|
| Project records describe approximately 138 million SFT training tokens, followed by two DPO rounds: |
| 877 turn-level pairs in round 1 and 404 pairs in round 2. Preferences came from sandbox-graded rollouts. |
| Round 2 used a short self-check opener to obtain candidate escalation messages; this does not prove universally honest reasoning. |
|
|
| The release candidate is **DPO-2**. Subsequent decision-point GRPO experiments were stopped and are not in these weights. |
| The original pilot gate failed and a disclosed exploratory amendment was tried; the aggressive follow-up hit a loop-regression guard. |
| The intermediate GRPO-3 artifact is not the release model. |
|
|
| Retained vision/MTP components provide checkpoint compatibility; fine-tuned multimodal capability was not evaluated. |
| The validated use case is English, text-based coding-agent interaction in this harness. |
|
|
| ## Limitations |
|
|
| - Protected-file violations remain frequent: 25/126 panel episodes. |
| - On blocked tasks, 15/36 episodes ended with a non-escalating submit; 22/36 had either such a submit or an integrity failure |
| (overlapping conditions). Successful escalation occurred in only 11/36. |
| - Confident output or a passing command is not proof of completion. Enforce external tests, tool permissions and resource limits. |
| - No broad chat, multilingual, vision, long-context or quantized-quality evaluation is claimed. |
|
|
|
|
| ## Harness interface |
|
|
| The training harness used one `bash` tool and Qwen-style XML tool calls. Its ending conventions are harness-specific: |
|
|
| - Finished: `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && cat patch.txt` |
| - Escalated: `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`, with an explanation of the blocker and the required input. |
|
|
|
|
