Thibaut Forest Claude Opus 5 (1M context) commited on
Commit ·
25432c8
1
Parent(s): ab4b0fd
Phase 45: pairwise ranking objective rejected - no new information, demotion artifact exposed; all axes now measured
Browse files- EXPERIMENTS.md +55 -0
EXPERIMENTS.md
CHANGED
|
@@ -2313,3 +2313,58 @@ a claim that nothing is left: J2W demonstrably finds 2.2 deaths per scenario,
|
|
| 2313 |
in well-observed data, that our ordering puts below the cut.
|
| 2314 |
|
| 2315 |
Standing champion remains **3c7b883 at 1661.45**.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2313 |
in well-observed data, that our ordering puts below the cut.
|
| 2314 |
|
| 2315 |
Standing champion remains **3c7b883 at 1661.45**.
|
| 2316 |
+
|
| 2317 |
+
## Phase 45 - the objective axis: pairwise ranking loss REJECTED (last open axis)
|
| 2318 |
+
|
| 2319 |
+
Rationale: log-loss on a 2.3% positive rate spends its capacity on the 97%
|
| 2320 |
+
that obviously survive; the decision is a top-3% ordering, and the entire
|
| 2321 |
+
638-point gap to the leader sits at ranks 12-20. A pairwise loss (RankNet-
|
| 2322 |
+
style, linear scorer, within-origin death-vs-survivor difference vectors, no
|
| 2323 |
+
intercept) optimises exactly that boundary. Two samplers: uniform survivors,
|
| 2324 |
+
and hard negatives (survivors the incumbent ranks highest). Scored on recall
|
| 2325 |
+
at fixed N, both instruments; score drops into rank_score so p42/count/timing
|
| 2326 |
+
would stay byte-identical if shipped.
|
| 2327 |
+
|
| 2328 |
+
| variant | native rec@12 | native tail 13-30 | gate rec@12 | gate tail |
|
| 2329 |
+
|---|---|---|---|---|
|
| 2330 |
+
| logbag (base) | **0.564** | 0.229 | **0.766** | 0.154 |
|
| 2331 |
+
| pair-uniform | 0.509 | 0.264 | 0.743 | 0.154 |
|
| 2332 |
+
| pair-hard | 0.518 | **0.271** | 0.746 | 0.149 |
|
| 2333 |
+
|
| 2334 |
+
Paired at N=12: native -21 (4W/15L), gate -20 (19W/36L). Loses everywhere at
|
| 2335 |
+
the operating point.
|
| 2336 |
+
|
| 2337 |
+
The apparent tail win (0.271 vs 0.229 native) motivated a head/tail hybrid:
|
| 2338 |
+
logbag's top-K byte-identical, pairwise re-orders only below it. Result: the
|
| 2339 |
+
hybrid loses at every K on both instruments (gate K12: -18 at N=16, 8W/24L;
|
| 2340 |
+
native K12: -12 at N=16, 4W/12L). That exposes the tail win as an artifact:
|
| 2341 |
+
**through rank 30 the two objectives catch the same deaths (native rec@30
|
| 2342 |
+
0.793 vs 0.789, gate 0.920 vs 0.894)** - pairwise does not discover new deaths
|
| 2343 |
+
in the tail, it demotes base's head deaths INTO the tail. When the hybrid
|
| 2344 |
+
keeps base's head and asks pairwise to improve the tail with its own picks,
|
| 2345 |
+
there is nothing there.
|
| 2346 |
+
|
| 2347 |
+
**Conclusion: the pairwise scorer extracts no information the log-loss fit
|
| 2348 |
+
does not already have.** Same features, same data, same deaths found, lower
|
| 2349 |
+
positions. Combined with phases 41 (features), 42 (training volume), 44
|
| 2350 |
+
(regularisation, calibration shift), the information ceiling of this feature
|
| 2351 |
+
space is the champion's current ordering. Every axis is now measured:
|
| 2352 |
+
|
| 2353 |
+
| axis | closed by |
|
| 2354 |
+
|---|---|
|
| 2355 |
+
| appetite / count | public (phase 28), break-even arithmetic (44) |
|
| 2356 |
+
| features | fixed-N ablation (41) |
|
| 2357 |
+
| training volume | public pooling (42) |
|
| 2358 |
+
| regularisation | C sweep, flat/negative at N=12 (44) |
|
| 2359 |
+
| calibration robustness | offset cannot reorder a linear model (44) |
|
| 2360 |
+
| objective | this phase |
|
| 2361 |
+
| timing / day placement | lead-time bias, coin flip (43) |
|
| 2362 |
+
| queue pricing | alphabetical ordering damage (43) |
|
| 2363 |
+
| routing-aware inclusion | phases 7/22 |
|
| 2364 |
+
| silent deaths | phases 6-13, 38-39 |
|
| 2365 |
+
|
| 2366 |
+
What would beat 1661 from here is information we do not have locally: either
|
| 2367 |
+
a signal J2W computes that is outside this feature space entirely, or the
|
| 2368 |
+
forward-reading of the split file that phase 43(a) verified and declined.
|
| 2369 |
+
|
| 2370 |
+
Standing champion: **3c7b883 at 1661.45**. Deadline 2026-08-23.
|