anhtld commited on
Commit
27ae0d0
·
verified ·
1 Parent(s): bbc6341

Audit K16 LCB dominance diagnostic

Browse files
workspace/reports/draft_audit_ctt.md CHANGED
@@ -141,6 +141,18 @@ diagnostic, but it is still hand-built RGB statistics rather than learned
141
  object-centric CTT geometry. Object-layout-only reaches 0.3125, combined
142
  RGB-stat/object-layout ties 0.3264, and RGB-stat success-weighted reaches
143
  0.3333, so deterministic object-layout features are not yet the missing method.
 
 
 
 
 
 
 
 
 
 
 
 
144
  The nonlinear selector path previously accepted chart-compat feature sets
145
  without loading selector chart maps, making that diagnostic too weak. It now
146
  loads the same deployment-visible selector chart maps as the ridge selector and
@@ -350,6 +362,12 @@ Current artifact evidence:
350
  success selector gap 0.2431, with pairwise causal calibration ECE 0.0150
351
  over 12,434 held-out candidate pairs. It uses train-only target--source
352
  RGB-stat chart compatibility and remains a selector diagnostic.
 
 
 
 
 
 
353
  - `runs/ctt_base_context_obs_nonlinear_dominance_chartcompat_obs_*_envclip_k16_train_to_test`
354
  and `runs/ctt_base_context_obs_nonlinear_dominance_chartcompat_obs_*_reg_envclip_k16_train_to_test`
355
  contain the fixed nonlinear chart-compat diagnostics. They now log
 
141
  object-centric CTT geometry. Object-layout-only reaches 0.3125, combined
142
  RGB-stat/object-layout ties 0.3264, and RGB-stat success-weighted reaches
143
  0.3333, so deterministic object-layout features are not yet the missing method.
144
+ The score-source conformal LCB fallback now has a K=16 train-calibrated Part-G
145
+ artifact as well:
146
+ `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test`. It calibrates
147
+ only on train-split K=16 measured rollout rows and evaluates once on held-out
148
+ test rows. The auto threshold covers 0.1319 of held-out rows and reaches
149
+ 0.2778 selected success, below the 0.2917 base action; the fixed tau0 rerun
150
+ `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test_tau0` falls back
151
+ 0.8889 of the time and only matches the base at 0.2917. Both runs have complete
152
+ action-bound safety labels, unsafe execution rate 0.0000, and pairwise causal
153
+ calibration ECE 0.1633 over 12,434 within-chart measured pairs. Therefore the
154
+ plain score-source LCB rule is a useful safety/reporting artifact, but a
155
+ negative selector result rather than a reliable causal dominance certificate.
156
  The nonlinear selector path previously accepted chart-compat feature sets
157
  without loading selector chart maps, making that diagnostic too weak. It now
158
  loads the same deployment-visible selector chart maps as the ridge selector and
 
362
  success selector gap 0.2431, with pairwise causal calibration ECE 0.0150
363
  over 12,434 held-out candidate pairs. It uses train-only target--source
364
  RGB-stat chart compatibility and remains a selector diagnostic.
365
+ - `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test` and
366
+ `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test_tau0` contain
367
+ the score-source conformal LCB dominance diagnostics for the same K=16 rows.
368
+ They now report pairwise causal calibration ECE 0.1633, complete
369
+ action-bound safety label coverage, unsafe execution rate 0.0000, and
370
+ selected success 0.2778--0.2917, so they are negative selector diagnostics.
371
  - `runs/ctt_base_context_obs_nonlinear_dominance_chartcompat_obs_*_envclip_k16_train_to_test`
372
  and `runs/ctt_base_context_obs_nonlinear_dominance_chartcompat_obs_*_reg_envclip_k16_train_to_test`
373
  contain the fixed nonlinear chart-compat diagnostics. They now log