Card accuracy fix: separate independently-verified metrics from session measurements; add independent validation report

README.md

@@ -40,17 +40,25 @@ else:
 
 See `INSERTION.md` for the full deployment spec and `replay_harness.py` for the reference end-to-end evaluation.
 
-## Validated metrics (independent, held-out eval)
+## Validated metrics
+
+Held-out eval (independently verified — re-ran the delivered harness on Linux, reproduced `eval_report.json` bit-exactly):
 
 | Metric | Result |
 |---|---|
 | Must-keep retention | 611/611 = **100%** at τ\*=0.50 (genuinely held-out eval, 0% train overlap) |
 | Downstream fidelity | 100% (50/50 replayed steps) |
 | Tool-output compression | **86.73%** (163,716 → 21,726 tokens) |
-| Real coding sessions (n=12) | **~90%** tool-output compression; **~48% median** total session token savings (range 28–61%) |
 | CPU latency (INT8, per chunk) | p50 ~0.3–0.8s, p99 ~0.9–5.7s depending on hardware/output length |
 
-The eval report (`eval_report.json`) and full validation report are in this repo. The 50% session-savings bar was narrowly missed on the median (48.30%) due to invariant per-turn system prefix dilution — reported as measured, not rounded up.
+Real coding sessions (builder-measured, n=12 — methodology reviewed, not independently re-run):
+
+| Metric | Result |
+|---|---|
+| Tool-output compression | **~90%** |
+| Total session token savings | **~48% median** (range 28–61%; 5/12 sessions ≥50%) |
+
+The eval report (`eval_report.json`), the builder's report (`REPORT.md`), and the independent validation report (`VALIDATION_COMPRESSOR_V2.md`) are all in this repo. The 50% session-savings bar was narrowly missed on the median (48.30%) due to invariant per-turn system prefix dilution — reported as measured, not rounded up.
 
 ## Files
 

VALIDATION_COMPRESSOR_V2.md

@@ -0,0 +1,249 @@
+# Independent Validation — Compressor Reflex v2 Re-delivery
+
+**Validator:** KOS-MOS (independent of the Gemini build loop)
+**Date:** 2026-10-01
+**Delivery:** Drive folder `compressor-reflex` (`1E1HBQQdzab-fqx_6MKHaOGi0LLzlIjt8`), 19 files
+**Local copy:** `~/workspace/compressor-reflex/delivery-v2/`
+**Status:** COMPLETE
+
+## Verdict: SHIP (recalibrated τ\*=0.50 — Eric's rulings recorded 2026-10-01)
+
+**Ruling 1 (G4):** Eric accepts the real-session measurement as the honest
+result — "~90% tool-output compression, ~48% median total session savings
+(n=12 real sessions)" — in place of the 50% bar on the synthetic harness.
+**Ruling 2 (G5):** Eric releases the hardware valve — the <150ms bar is
+dropped; the ~0.5–6s/chunk CPU latency is accepted against the token economics.
+
+| Gate | Bar | Recalibrated (τ\*=0.50, deployment-consistent) | Status |
+|---|---|---|---|
+| G1 must-keep retention | 100% (mandatory) | **611/611 = 100%** (fixed bypass, re-swept) | **PASS** |
+| G2 downstream fidelity | ≥98% | 100% (50/50) | **PASS** |
+| G3 tool compression | — | 86.73% (163,716 → 21,726) | info |
+| G4 realized session reduction | ≥50% (synthetic harness) | **42.90%** harness / 48.30% median real sessions / 86.64% Step-0 projection | **ACCEPTED** — real-session measure binds per Ruling 1 |
+| G5 INT8 p99 latency | <150ms | 901ms Linux / 3688ms Windows (eval); 5.7s p99 real traffic | **VALVE RELEASED** per Ruling 2 |
+| G6 purity / fail-open | pass | pass | **PASS** |
+
+Known residuals carried openly: the 0.5449 anchor (thinnest margin in the set —
+drops at τ≥0.55, kept at τ\*=0.50); training provenance thin (1 epoch, stub
+log); real-traffic p99 latency exceeds eval (deployment should cap very long
+outputs); replay-based measurement cannot capture behavioral effects (true
+live A/B still open if productizing).
+
+## TL;DR
+
+- **Held-out claim: VERIFIED.** 0% verbatim item, intent, and anchor overlap
+  eval↔train and eval↔val, independently recomputed. 220 distinct eval intents
+  (v1 had 14 templates total). The memorization defect is gone.
+- **Deep-line defect: FIXED.** 172/172 anchor lines at line-idx ≥ 50 retained —
+  the exact family v1 failed systematically.
+- **Report honesty: VERIFIED bit-exactly.** The delivered `replay_harness.py`,
+  run on this Linux VM against the delivered INT8 model + eval data, reproduces
+  `eval_report.json` to 13 decimal places (82.0579540179335%,
+  39.67789484226776%). No projection substituted. This is the honesty v1 lacked.
+- **G1 caveat (the one real finding):** the delivered 611/611 depends on the
+  harness's short-item bypass counting *eval elements* (5) instead of *physical
+  lines* (6) for one item with an embedded newline. Under INSERTION.md's
+  deployment spec (bypass iff ≤5 physical lines or ≤64 tokens; this item is
+  6 lines / 160 tokens), the item is scored and one must-keep anchor drops at
+  0.5449 < 0.60 → **610/611 = 99.84%**, deterministic, not noise.
+  Recalibrating τ\* to 0.50 restores 100% under deployment-consistent
+  accounting (verified) — no retraining required.
+- **G4:** 39.68% on the delivered synthetic harness vs the ≥50% bar, honestly
+  marked failed; the 81.98% Step-0 empirical projection is separately labeled,
+  never substituted.
+- **G5:** p99 ≈ 1.3s (Windows) / 2.0s (Linux) vs 150ms — an order of magnitude;
+  the report invokes the hardware release valve explicitly.
+- **Training provenance: WEAK** (documentation gap, not a gate failure).
+
+## A. Held-out integrity (independent re-verification)
+
+| Check | Gemini claim | KOS-MOS measured |
+|---|---|---|
+| verbatim item overlap eval↔train | 0.00% | **0/217 = 0.00%** |
+| verbatim item overlap eval↔val | — | **0/217 = 0.00%** |
+| intent overlap eval↔train | 0.00% | **0/220 = 0.00%** |
+| intent overlap eval↔val | — | **0/220 = 0.00%** |
+| anchor overlap eval↔train | 0.00% | **0/207 = 0.00%** |
+| novel eval lines | 66.74% (5,974/8,951) | **67.18% (5,974/8,892)** — line-count basis differs slightly, same numerator |
+
+Remaining shared lines are tool boilerplate (headers, blank lines, exit summaries),
+each appearing with identical labels — no label-flip traps.
+
+Template diversity vs v1:
+
+| split | distinct items | distinct (project, intent) | projects |
+|---|---|---|---|
+| train | 1,348/2,200 | 601 | 6 |
+| val | 179/250 | 177 | 6 |
+| eval | 217/340 | 220 | 5 |
+
+v1 had 14 templates total; v2's eval alone has 220 distinct intents.
+Train still contains internal duplicates (1,348/2,200 distinct) but none leak
+into eval. Deep-line coverage is real: **172/611 anchor lines sit at line-idx ≥ 50**
+across 43 must-keep eval items (longest item: 78 lines).
+
+## B. Delivery manifest
+
+All 19 files present on Drive. Binary/JSONL files md5-verified against Drive.
+`.json` files differ in bytes from the manifest's SHA-256 because the downloader
+re-serializes JSON (strips `retrieved_at`/`drive_change_recorded_at` injection);
+content verified equivalent. Notable:
+
+- `tokenizer.json`: vocab/merges/added-tokens content-identical to the
+  v1-delivered tokenizer, itself verified against base
+  `heman10x/rlcd-modernbert-151m`. Lineage clean.
+- `serializer.py`, `model.py`, `INSERTION.md`: byte-identical to v1 delivery
+  (same chunking contract, same line-head architecture).
+- `model.onnx` / `model_int8.onnx`: new weights (sha differs from v1).
+  Local INT8 md5 `dfe74457ada35cf36f194810945ffa6e` (both local copies identical).
+- `replay_harness.py`: rewritten (portable). `best_model.pt` / FP32 `model.onnx`
+  were not fully downloaded locally (large-file transfer still running at
+  validation time); all inference used the delivered INT8 model.
+
+## C. Gate honesty (the v1 failure, fixed)
+
+`eval_report.json` marks Gate 4 (39.68% < 50%) and Gate 5 (p99 1320ms > 150ms)
+as **failed**, with the Step-0 81.98% projection separately labeled and the
+latency release valve explicitly invoked. `REPORT.md` presents 39.68% as Gate
+4's primary outcome. No projection substituted for measurement anywhere —
+the exact honesty v1 lacked.
+
+## D. Portability
+
+- `import torch`: absent. Hardcoded Windows paths: 0 in code
+  (`train.log` text contains log-line paths only). `Path(__file__)`-relative: yes.
+- Runs on NumPy + onnxruntime only. Executed clean on this Linux VM.
+
+## E. Training provenance (weak)
+
+- `train.log` (1,038 chars): covers ~7 seconds of data loading only, mislabeled
+  "Training Run (v1)".
+- `training_history.json`: **one epoch** (loss 0.5681, retention 100.0 on 513
+  val targets, compression 86.91%).
+- `checkpoint_records.json`: one epoch, best_epoch 1.
+- Verdict: the "training log" requirement is met in letter but not in spirit —
+  a documentation gap, not a gate failure. (Consistent with a single-epoch CPU
+  run: Epoch 1 took ~6+ hours; 15 epochs was never feasible on the stated host.)
+
+## F. Inference re-measurement
+
+### F.1 Delivered harness reproduces the report bit-exactly (Linux)
+
+Ran the exact delivered `replay_harness.py` on this VM (delivered INT8 model,
+delivered eval set, delivered tokenizer):
+
+```
+Threshold tau = 0.60 -> Must-Keep Retention: 100.00% (611/611)   [all 9 swept thresholds]
+Tool Output Compression Ratio: 82.06% (163,716 -> 29,374 tokens)
+Downstream Next-Action Fidelity: 100.00%
+Harness-Measured Session Reduction: 39.68%
+Per-Chunk CPU Latency: p50 = 579.6ms, p90 = 1437.5ms, p99 = 2039.1ms
+```
+
+`eval_report.json` values reproduced to 13 decimals
+(82.0579540179335%, 39.67789484226776%). The report is an honest measurement
+of the delivered artifacts — on Windows *and* Linux.
+
+### F.2 Independent validator (separate reimplementation)
+
+Reimplemented scoring from the delivered `serializer.py` (NumPy only,
+chunking verified equivalent span-for-span): retention 100% at τ ≤ 0.50,
+**610/611 = 99.84%** at τ = 0.60; deep-line retention 172/172 at all thresholds;
+fidelity 49/50 = 98.00%; realized savings 42.88%; INT8 p99 2090.5ms.
+
+### F.3 The 610-vs-611 discrepancy — root-caused, deterministic
+
+Not noise, not a model defect. One eval item
+(`locate definition location for constant EVAL_ROUTER_FLAG_9854`) has 5
+`lines` *elements* but 6 *physical* lines (one element contains an embedded
+`\n`), 160 tokens. Its must-keep anchor (element 2 → physical line 2) scores
+**0.5449**, deterministically, on the identical model bytes.
+
+- Delivered harness `score_item`: bypasses when `len(item["lines"]) <= 5`
+  (counts **elements** = 5) → returns all-1.0 → anchor "kept".
+- INSERTION.md deployment spec (§38): bypass iff **≤5 physical lines or ≤64
+  tokens** → this item (6 lines, 160 tokens) is **scored** in deployment →
+  anchor dropped at τ\*=0.60.
+- Per-chunk scores verified **bit-identical** between the two scorers;
+  chunking verified span-identical. The entire gap is the bypass counting unit.
+
+Under deployment-consistent accounting, τ\*=0.60 yields **610/611 = 99.84%**,
+not the mandatory 100%. This is a calibration/config defect (threshold set
+against a bypass the deployment doesn't share), not a failure to generalize:
+the model itself is deterministic and the deep-line family is fully repaired.
+
+### F.4 τ\*=0.50 under deployment-consistent accounting
+
+Strict re-run (physical-line bypass per INSERTION.md, element→physical anchor
+mapping, delivered INT8, this VM):
+
+- G1-strict @ τ=0.50: **611/611 = 100.00%** ✓ (the 0.5449 anchor is kept)
+- G3-strict @ τ=0.50: **86.73%** (163,716 → 21,726 tokens)
+- G2-strict @ τ=0.50: **34/34 = 100.0%** (must-keep items in first 50)
+- G4-strict @ τ=0.50: 59.62% on the 340-item horizon; **42.90%** on the
+  apples-to-apples 50-item replay (their exact loop, strict scoring) —
+  vs 39.68% delivered / 42.88% strict at τ=0.60. The τ change barely moves G4
+  (42.88→42.90); the 39.68→42.88 delta is the bypass counting unit.
+  **G4 still misses the 50% bar under every consistent accounting.**
+
+Recalibrating τ\* 0.60 → 0.50 restores the mandatory 100% retention with no
+retraining. Compression at 0.50 (86.73% strict) is *higher* than the delivered
+82.06% because strict physical-line scoring compresses short items the
+element-bypass passed through verbatim.
+
+## G. Remediation — COMPLETE (see §I); rulings recorded 2026-10-01
+
+1. **G1 — recalibrate τ\* 0.60 → 0.50.** ✅ DONE & VERIFIED. Bypass fixed to
+   physical lines per INSERTION.md, sweep re-run, 611/611 = 100% confirmed
+   independently. One-line config change; no retraining.
+2. **G4 — ✅ RULED.** Eric accepts the real-session measurement as the honest
+   result: "~90% tool-output compression, ~48% median total session savings
+   (n=12 real sessions)" in place of the 50% synthetic-harness bar. The
+   publishable claim is the measured one, not "at least 50%."
+3. **G5 — ✅ VALVE RELEASED.** Eric drops the <150ms bar; accepts ~0.5–6s/chunk
+   CPU latency against the token economics. Deployment note carried: very long
+   outputs need chunk caps / graceful handling (real-traffic p99 5.7s).
+
+## H. Standing notes for future rounds
+
+- The short-item bypass **counting unit** (eval elements vs physical lines vs
+  tokens) must be specified identically in the harness, the serializer, and
+  INSERTION.md. Eval `lines` elements may contain embedded newlines.
+- Read the machine artifacts (`eval_report.json`, harness stdout), not the
+  prose report — standing rule, re-confirmed: here both agreed, which is
+  itself a v2 improvement worth noting.
+- JSON downloads get re-serialized by the download tool (metadata scrub);
+  compare content, never raw bytes/SHA, for `.json` — standing rule, no
+  attribution to the builder.
+
+## I. Refreshed artifact verification (2026-10-01 ~21:15 EDT)
+
+Gemini delivered the §G fix: bypass counting unit aligned to INSERTION.md
+(physical lines + ≤64 tokens, 3 sites in `replay_harness.py`), τ\* recalibrated
+to 0.50, `eval_report.json` + `REPORT.md` regenerated. Independently verified:
+
+- **File integrity:** downloaded `replay_harness.py` SHA-256
+  `22afa3cf…5b53e` and `REPORT.md` SHA-256 `af7f2cad…8e3ac` match the
+  delivery report exactly; sizes 13,185 / 10,363 bytes match.
+- **Scope respected:** `eval_frozen.jsonl` untouched; weights, tokenizer,
+  training data unmodified (no retraining).
+- **Bit-exact reproduction:** the refreshed harness run on this Linux VM
+  reproduces the refreshed `eval_report.json` on every non-hardware metric:
+  τ\*=0.50, 611/611 retention, 86.72945833027926% compression,
+  42.89676915071273% session reduction, 86.64272887194898% Step-0 projection,
+  100% fidelity, gates G4/G5 honestly false. (Latency differs by hardware as
+  expected: Linux p99 901.52ms this run vs Windows 3688ms reported — both
+  honestly marked failed.)
+- **Sweep confirms calibration:** 100% at τ ≤ 0.50; 610/611 at 0.55 and 0.60
+  (the 0.5449 anchor) — exactly matching the independent strict reimplementation.
+- **Cross-check:** every refreshed number matches the independent
+  KOS-MOS strict measurements (§F.4): 611/611, 86.73%, 42.90%, 50/50.
+
+## Artifacts
+
+- Static auditor: `~/workspace/compressor-reflex/audit_static_v2.py`
+- Independent validator: `~/workspace/compressor-reflex/validate_compressor_v2.py`
+- Strict τ=0.50 re-run: `/tmp/strict_tau050.py` (scratch)
+- Delivered-harness sandbox: `/tmp/hrun/` (log `/tmp/hrun.log`,
+  report `/tmp/hrun/eval_report.json`)
+- Delivered INT8 md5: `dfe74457ada35cf36f194810945ffa6e`