File size: 5,159 Bytes
df5b3c0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | # Wit-Coder-4B: evaluation details
## Recorded benchmark results
Both columns are project measurements, not scores copied from the upstream model card.
| Metric | Qwen3.5-4B | Wit-Coder-4B / DPO-2 |
|---|---:|---:|
| MMLU, 5-shot accuracy | 69.67% | 70.16% |
| GSM8K, 8-shot CoT, strict match | 84.08% | 82.64% |
| GSM8K, flexible extraction | 84.15% | 82.79% |
| HumanEval base tests, **modified bounded protocol**, pass@1 | 87.80% (144/164) | 90.85% (149/164) |
| HumanEval plus tests, **modified bounded protocol**, pass@1 | 81.10% (133/164) | 87.20% (143/164) |
MMLU and recorded HumanEval scores increased; GSM8K decreased. This is not improvement across every benchmark.
MBPP scores are not reported: complete samples and scored outputs were not recovered from the interrupted runs.
### Protocol and interpretation
- MMLU/GSM8K: `lm-eval` 0.4.13, raw-completion prompts, seed 1234, 5/8 exemplars respectively,
greedy decoding, maximum context 32,768; GSM8K generation limit 256 tokens. Longest recorded MMLU context: 3,138 tokens.
- Base requests used an SSH tunnel; DPO-2 requests ran on the pod against localhost. Package versions and protocol settings match,
but these were separate sessions. No inference-speed comparison is implied.
- HumanEval: `evalplus` 0.3.1 prompt/sanitization, one greedy chat completion per task, thinking enabled,
maximum 16,384 generated tokens. Base had 9 length-truncated completions; DPO-2 had 7.
- **These are not official, unmodified EvalPlus scores.** Execution used isolated network-less trials with 512 MB memory,
64 PIDs and one CPU. HumanEval/15 and /130 were split into input chunks, resetting candidate globals between chunks.
No HumanEval test inputs were omitted. The canonical HumanEval/32 solution also failed validation in this environment,
including the upstream evaluator.
- Frozen calibration SHA256: `f4a330c8c556d1d844ac0f8c29c3db0516379f0e8bf30abb6e5bacbe3bf45283`.
It also lists three omitted MBPP inputs; no MBPP score is claimed here. A candidate shares a trial with its own runner;
this is not adversarially tamper-proof grading.
- These scores concern the served DPO-2 checkpoint, **not the quantized GGUF artifacts**.
## In-house coding-agent panel
42 tasks × 3 seeds = 126 episodes: 90 solvable and 36 blocked by design, executed and graded in a network-less sandbox.
Panel IDs and registered held-out families were excluded from training-data construction. Multiple checkpoints were measured
on this panel and its results informed later experiments; this is a development comparison, not a newly sealed final test.
Three seeds do not make 126 independent tasks. ID/family exclusion does not establish complete semantic decontamination.
| Outcome | Qwen3.5-4B | DPO-2 |
|---|---:|---:|
| Task success, including successful escalation | 40/126 | 71/126 |
| Solvable tasks solved | 40/90 | 60/90 |
| Blocked tasks successfully escalated | 0/36 | 11/36 |
| Episodes ending at the per-turn generation limit | 34/126 | 7/126 |
| Episodes ending at the maximum turn count | 19/126 | 4/126 |
| Protected-file integrity failures | 28/126 | 25/126 |
| MATH-500, first 100 problems | 65/100 | 82/100 |
Generation-limit endings are not a direct measurement of semantic reasoning loops. MATH uses a 100-problem subset, not all 500.
No statistical significance or broad generalization claim is established by these small panel comparisons.
## Training and model selection
Project records describe approximately 138 million SFT training tokens, followed by two DPO rounds:
877 turn-level pairs in round 1 and 404 pairs in round 2. Preferences came from sandbox-graded rollouts.
Round 2 used a short self-check opener to obtain candidate escalation messages; this does not prove universally honest reasoning.
The release candidate is **DPO-2**. Subsequent decision-point GRPO experiments were stopped and are not in these weights.
The original pilot gate failed and a disclosed exploratory amendment was tried; the aggressive follow-up hit a loop-regression guard.
The intermediate GRPO-3 artifact is not the release model.
Retained vision/MTP components provide checkpoint compatibility; fine-tuned multimodal capability was not evaluated.
The validated use case is English, text-based coding-agent interaction in this harness.
## Limitations
- Protected-file violations remain frequent: 25/126 panel episodes.
- On blocked tasks, 15/36 episodes ended with a non-escalating submit; 22/36 had either such a submit or an integrity failure
(overlapping conditions). Successful escalation occurred in only 11/36.
- Confident output or a passing command is not proof of completion. Enforce external tests, tool permissions and resource limits.
- No broad chat, multilingual, vision, long-context or quantized-quality evaluation is claimed.
## Harness interface
The training harness used one `bash` tool and Qwen-style XML tool calls. Its ending conventions are harness-specific:
- Finished: `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && cat patch.txt`
- Escalated: `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`, with an explanation of the blocker and the required input.
|