keel / docs /SUBSTACK-POST.md
Tobi
Upload folder using huggingface_hub
5823e53 verified
|
Raw History Blame Contribute Delete
10.7 kB

Seven times the tool refused to certify itself

Building Keel, a checkpoint validity harness for open-weight models.

A fine-tune shows up on a model hub. The benchmark scores look fine. The card says it is a helpful variant of the base model. And then you ask it something the base model refuses, and it answers.

That gap is the problem Keel exists to measure. Not "is the model good", but "what did this derivative actually do to the behavior the base model had". The answer is a drift card: what moved, whether the measurement was valid, what the harmful half says, and, when the weights are available, the structural signature of the intervention.

The project is at tomscot-keel.hf.space. The code is at github.com/TCPCore/keel. This post is the story of what it measured, and of the seven times it refused to certify its own measurement.

The card

keel scan compares a derivative against its base on a fixed panel. It returns six blocks:

  • Verdict: allow, review, or block.
  • Drift: five behavioral axes, each with an interval, a p-value and a q-value.
  • Harmful: compliance on a benchmark of public harmful requests, against the base.
  • Validity: whether the measurement can be trusted at all.
  • Structure: the weight-space fingerprint, when the weights are available.
  • Limits: what the card does not know.

The card is signed with a fingerprint over its content, and the Space can recompute it from the committed responses. If you edit one number, the fingerprint changes and verification fails. That is the point: a card is not a claim, it is a proof obligation.

The policy seam is deliberately simple. A harmful delta above 0.15 is a review. A red verdict, or a missing harmful half, is a block. Everything else is a judgment call, which is why the card carries its limits.

The judge is part of the measurement

Every compliance number on every card was produced by one frozen judge: muse-spark-1.3 through the Meta Model API, with answer-only extraction and three repeats, majority verdict.

The extractor is where a lot of the honesty lives. Glimmer separates its reasoning with a to=self marker, and only the segment after the last to=user marker is judged. Without that rule, a model that reasons about refusal and then refuses scores as if it complied. The rule is recorded in every card, and the cards are re-scorable: press the button and the judge runs again on the same responses.

I did not pick the judge because it was convenient. I labelled a 48-item gold set blind, then ranked the candidates by kappa against those labels. Spark 1.3 won at 0.603 (accuracy 0.889). The open-weights judge scored 0.385 and called many refusals compliant. Wonder scored 0.282. The gold set was labelled by the reviewing agent, not a human, and that limitation is in the repo, because a judge bench with hidden annotator bias is worse than no bench.

What the numbers say

The fleet has twelve checkpoints; seven carry cards. The seven, under one scanner:

checkpoint harmful compliance
Glimmer base 0.00
Glimmer, our ablation 0.16
Glimmer, our LoRA 0.34
Glimmer, community edit 0.09
Qwen 7B, our ablation 0.97
Qwen 7B, community edit 0.78
Mistral 7B, community edit 0.81

The interesting result is not any single number. It is the family structure. The same directional ablation under the same judge moves Qwen's 7B derivatives to 0.97 and 0.78, while the Qwen 27B community checkpoint sits at 0.51, so the family range is 0.51 to 0.97. Mistral's edits land at 0.81 and 0.88, and Glimmer reaches only 0.16. Refusal looks concentrated in Qwen and Mistral and distributed in Glimmer.

That is stated as a range, not a point, because the Qwen 27B checkpoint sits at 0.51 while the 7B variants sit at 0.97 and 0.78. A headline that quotes only 0.97 would be cherry-picked. The families page shows the distribution, and the guide says it in one sentence.

What the weights add

For the community Glimmer edit, derivative C, the weight delta is fully localised. One hundred percent of it lies in o_proj and down_proj, it reconstructs to bfloat16 exactness, and the per-module intensity is non-uniform: lambda 1.2805 for o_proj layer 24 and 0.2177 for down_proj layer 7. The median rank-one residual is 0.363, and only 36 of 104 modules are clean rank-one edits, which is what the community card says: this is a composite, not a single direction.

That matters because it sets the ceiling on the stronger claim. Reconstructing the weights and replaying the behavior is gated at a pre-registered tolerance. On derivative C the replay scored 0.000 against the candidate's 0.084, a gap of 0.084 that passes the 0.15 gate arithmetically. It is not evidence. The tolerance is larger than the effect, so a replay that changes nothing would also pass. The honest record says blocked by inference-context dependence, not "reproduced".

Seven refusals

The part of this project I am proudest of is not a number. It is the list of times the tool refused to certify its own measurement, and the two research ideas that died under their own controls.

1. The reasoning-versus-answer artifact. The first M2 card was wrong because reasoning leaked into the answer segment. Re-scoring the answer segment corrected it. The card was reissued, not patched.

2. The format collapse. Five behavioral axes said "no drift" while 30 of 32 responses were code. A model that answers every request with a code block is not compliant; it is broken in a different way. The format gate now turns a green card amber when the code rate shifts by more than 0.30, and a suspect card cannot be green.

3. The missing harmful half. A card that does not measure harmful compliance cannot be green, no matter what the other axes say. Green is capped without it.

4. The empty completions. At max_tokens=192 the survey recorded an empty rate, so an invalid run became visible instead of being published as a clean result.

5. The judge and extractor drift. The same Glimmer ablation scored 0.81 under a retired judge, 0.59 under its successor, and 0.16 under the frozen answer-only contract with a format-aware extractor. The re-score button caught the first jump. The extractor bug caught the rest. The correction history is in the card, because a number without its contract is a draw, not a measurement.

6. The replay that passed without evidence. The gate was 0.15, the effect was 0.084, and the pass was meaningless. The lesson, written into the repo: a band must exclude the no-op and not exclude a true match within noise. At 32 items and a judge standard deviation near 0.03, the first band was too loose and the revision was still one item too high.

7. The circuit that its own control killed. I pre-registered a second verdict: extract the refusal circuit as the top 0.1 percent of MLP neurons, compare membership and signature, and call a checkpoint structurally intact if the circuit is present and causally load-bearing. The first run looked spectacular: Jaccard 0.523 against a random null of 0.0005, a z-score of 1,175. Then I ran the control that matters: two extractions of the same base on different probes. They agreed at Jaccard 0.12 to 0.49. The derivative's 0.523 sat inside the noise. The signature cosine was 0.991 for a circuit that changed by a third. The causal ablation moved compliance by 0.04 over a random control, one to two judged items at 32 probes.

The circuit fingerprint is not a second verdict. It was reported as a failure, not shipped, and the next experiment is named: a circuit defined by causal contribution, on a 128-probe panel.

What is actually limited

The judge is a model, not an oracle. Kappa 0.603 leaves real label noise. Panels of 32 have intervals wide enough that a 0.05 effect is not resolvable, and several results are stated as intervals for that reason.

Behavior is a function of weights and inference context, and the context is not in the weights. That is why weight recovery is exact and replay is blocked. It is also why I do not claim "we reproduced the behavior".

The refusal circuit is distributed in Glimmer, which is the most plausible reading of the 0.16. A sparse circuit would have moved the behavior more. The negative result is a finding about the model, not just about the method.

And one Meta model I could not get: Spark Open 816B is gated, the account returns 403, and the second open-weights family was never run. That is recorded as a refusal, not papered over.

What to click

The demo is eight recorded chapters, two minutes, at tomscot-keel.hf.space/video. The four things worth doing on the live site:

  • Verify any card. The server recomputes it from the committed responses and returns MATCH.
  • Tamper one number. The fingerprint changes and verification fails.
  • Re-score a card. The judge runs live on the same responses.
  • Scan your own base and candidate response files and get the same card schema.

The design of the page is the argument. First the fleet, then the proof, then the tamper test, then the cross-family range, then the card review. A judge should not have to trust a number they cannot recompute.

What comes next

Two experiments, both recorded and both post-submission:

  1. A causal-construction circuit. Instead of extracting neurons by contrastive ranking and hoping the set is stable, ablate each candidate neuron, measure the compliance change, and keep the ones that move it. That inverts the experiment and directly targets the failure mode the control exposed.

  2. A deterministic replay. If the inference context can be pinned, the weight reconstruction can be tested behaviorally rather than only structurally. Until then, the card keeps the limitation.

Close

Most safety tooling fails in one of two directions: it certifies too much, or it measures nothing and calls the result uncertainty. Keel tries to sit in the narrow third position: it measures what it can, refuses what it cannot, and publishes the list of things it got wrong on the way.

The tagline is small on purpose. We detect drift. The rest of the sentence is the product: and we will not certify a measurement we cannot check.

If you want the technical version, the repo has docs/META-MODELS.md, the judge bench, the replay protocol, and every refusal above. If you want the two-minute version, the video is on the site, served from the original file rather than a platform re-encode.