Thibaut Forest Claude Opus 5 (1M context) commited on
Commit
25432c8
·
1 Parent(s): ab4b0fd

Phase 45: pairwise ranking objective rejected - no new information, demotion artifact exposed; all axes now measured

Browse files
Files changed (1) hide show
  1. EXPERIMENTS.md +55 -0
EXPERIMENTS.md CHANGED
@@ -2313,3 +2313,58 @@ a claim that nothing is left: J2W demonstrably finds 2.2 deaths per scenario,
2313
  in well-observed data, that our ordering puts below the cut.
2314
 
2315
  Standing champion remains **3c7b883 at 1661.45**.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2313
  in well-observed data, that our ordering puts below the cut.
2314
 
2315
  Standing champion remains **3c7b883 at 1661.45**.
2316
+
2317
+ ## Phase 45 - the objective axis: pairwise ranking loss REJECTED (last open axis)
2318
+
2319
+ Rationale: log-loss on a 2.3% positive rate spends its capacity on the 97%
2320
+ that obviously survive; the decision is a top-3% ordering, and the entire
2321
+ 638-point gap to the leader sits at ranks 12-20. A pairwise loss (RankNet-
2322
+ style, linear scorer, within-origin death-vs-survivor difference vectors, no
2323
+ intercept) optimises exactly that boundary. Two samplers: uniform survivors,
2324
+ and hard negatives (survivors the incumbent ranks highest). Scored on recall
2325
+ at fixed N, both instruments; score drops into rank_score so p42/count/timing
2326
+ would stay byte-identical if shipped.
2327
+
2328
+ | variant | native rec@12 | native tail 13-30 | gate rec@12 | gate tail |
2329
+ |---|---|---|---|---|
2330
+ | logbag (base) | **0.564** | 0.229 | **0.766** | 0.154 |
2331
+ | pair-uniform | 0.509 | 0.264 | 0.743 | 0.154 |
2332
+ | pair-hard | 0.518 | **0.271** | 0.746 | 0.149 |
2333
+
2334
+ Paired at N=12: native -21 (4W/15L), gate -20 (19W/36L). Loses everywhere at
2335
+ the operating point.
2336
+
2337
+ The apparent tail win (0.271 vs 0.229 native) motivated a head/tail hybrid:
2338
+ logbag's top-K byte-identical, pairwise re-orders only below it. Result: the
2339
+ hybrid loses at every K on both instruments (gate K12: -18 at N=16, 8W/24L;
2340
+ native K12: -12 at N=16, 4W/12L). That exposes the tail win as an artifact:
2341
+ **through rank 30 the two objectives catch the same deaths (native rec@30
2342
+ 0.793 vs 0.789, gate 0.920 vs 0.894)** - pairwise does not discover new deaths
2343
+ in the tail, it demotes base's head deaths INTO the tail. When the hybrid
2344
+ keeps base's head and asks pairwise to improve the tail with its own picks,
2345
+ there is nothing there.
2346
+
2347
+ **Conclusion: the pairwise scorer extracts no information the log-loss fit
2348
+ does not already have.** Same features, same data, same deaths found, lower
2349
+ positions. Combined with phases 41 (features), 42 (training volume), 44
2350
+ (regularisation, calibration shift), the information ceiling of this feature
2351
+ space is the champion's current ordering. Every axis is now measured:
2352
+
2353
+ | axis | closed by |
2354
+ |---|---|
2355
+ | appetite / count | public (phase 28), break-even arithmetic (44) |
2356
+ | features | fixed-N ablation (41) |
2357
+ | training volume | public pooling (42) |
2358
+ | regularisation | C sweep, flat/negative at N=12 (44) |
2359
+ | calibration robustness | offset cannot reorder a linear model (44) |
2360
+ | objective | this phase |
2361
+ | timing / day placement | lead-time bias, coin flip (43) |
2362
+ | queue pricing | alphabetical ordering damage (43) |
2363
+ | routing-aware inclusion | phases 7/22 |
2364
+ | silent deaths | phases 6-13, 38-39 |
2365
+
2366
+ What would beat 1661 from here is information we do not have locally: either
2367
+ a signal J2W computes that is outside this feature space entirely, or the
2368
+ forward-reading of the split file that phase 43(a) verified and declined.
2369
+
2370
+ Standing champion: **3c7b883 at 1661.45**. Deadline 2026-08-23.