---
license: apache-2.0
base_model: microsoft/unixcoder-base
tags: [code, agents, verifier, calibration, hindsight]
---

# hindsight-v4

A small, calibrated, non-generative verifier for coding-agent sessions. Given the canonical event
trace of a session (user turns, agent messages, tool calls and results, diffs) it predicts, at every
user-turn boundary, whether the user's next turn will be a correction, and of which kind
(not-done, unrequested, rule-violation, blind-action, drift, communication, damage, quality,
ungrounded). Trained on hindsight labels (what the user actually did next), never on the agent's
own claims. Encoder: UniXCoder-base fine-tuned end to end; level 2: a GRU over the last 24 event
vectors with an attention readout conditioned on the latest user request; heads Platt-calibrated
on a held-out split. Inputs are the event texts and a structural feature vector only: no labelers,
no source id (the unknown-source slot), so the model runs on any harness's trace.

## Held-out AUROC (Platt-scaled, test split by repository and user)

| head | all sources | Claude Code local | Codex local | SWE-chat |
|---|---|---|---|---|
| correction | 0.732 | 0.769 | 0.758 | 0.722 |
| not_done | 0.798 | 0.817 | 0.667 | 0.793 |
| unrequested | 0.794 | 0.885 | 0.653 | 0.811 |
| rule_violation | 0.735 | 0.884 | 0.806 | 0.727 |
| blind_action | 0.675 | 0.297 | 0.842 | 0.662 |
| communication | 0.630 | 0.679 | 0.852 | 0.597 |
| quality | 0.699 | 0.786 | 0.754 | 0.662 |
| ungrounded | 0.652 | 0.566 | 0.773 | 0.657 |

Coverage at 80% precision on the any-correction head is 0.04; use it as a rank or reject signal.
The `other` head is not usable.

## Files

`encoder/` (transformers, safetensors), `model.pt` (level-2 core and heads, keys prefixed `ver.`),
`config.json` (dims, window, source list), `results.json` (metrics, Platt parameters),
`quant_bench.json` (fp32 vs int8 vs ONNX on CPU: keep fp32).

`onnx/` is the deployment export (`gpu/export_onnx.py`): `encoder.onnx` (token ids and mask to a
normalized event vector), `level2.onnx` (a 24-event window to the any-correction logit and the nine
class logits), `tokenizer.json`, `hindsight.json` (feature layout, window, Platt parameters). It is
what `octolib::hindsight` (Rust, ONNX Runtime, CPU) loads; it runs without the tier-1 labelers
(those feature slots are zero) and feeds the unknown-source slot.