Audit K16 LCB dominance diagnostic
Browse files
workspace/reports/draft_audit_ctt.md
CHANGED
|
@@ -141,6 +141,18 @@ diagnostic, but it is still hand-built RGB statistics rather than learned
|
|
| 141 |
object-centric CTT geometry. Object-layout-only reaches 0.3125, combined
|
| 142 |
RGB-stat/object-layout ties 0.3264, and RGB-stat success-weighted reaches
|
| 143 |
0.3333, so deterministic object-layout features are not yet the missing method.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
The nonlinear selector path previously accepted chart-compat feature sets
|
| 145 |
without loading selector chart maps, making that diagnostic too weak. It now
|
| 146 |
loads the same deployment-visible selector chart maps as the ridge selector and
|
|
@@ -350,6 +362,12 @@ Current artifact evidence:
|
|
| 350 |
success selector gap 0.2431, with pairwise causal calibration ECE 0.0150
|
| 351 |
over 12,434 held-out candidate pairs. It uses train-only target--source
|
| 352 |
RGB-stat chart compatibility and remains a selector diagnostic.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 353 |
- `runs/ctt_base_context_obs_nonlinear_dominance_chartcompat_obs_*_envclip_k16_train_to_test`
|
| 354 |
and `runs/ctt_base_context_obs_nonlinear_dominance_chartcompat_obs_*_reg_envclip_k16_train_to_test`
|
| 355 |
contain the fixed nonlinear chart-compat diagnostics. They now log
|
|
|
|
| 141 |
object-centric CTT geometry. Object-layout-only reaches 0.3125, combined
|
| 142 |
RGB-stat/object-layout ties 0.3264, and RGB-stat success-weighted reaches
|
| 143 |
0.3333, so deterministic object-layout features are not yet the missing method.
|
| 144 |
+
The score-source conformal LCB fallback now has a K=16 train-calibrated Part-G
|
| 145 |
+
artifact as well:
|
| 146 |
+
`runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test`. It calibrates
|
| 147 |
+
only on train-split K=16 measured rollout rows and evaluates once on held-out
|
| 148 |
+
test rows. The auto threshold covers 0.1319 of held-out rows and reaches
|
| 149 |
+
0.2778 selected success, below the 0.2917 base action; the fixed tau0 rerun
|
| 150 |
+
`runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test_tau0` falls back
|
| 151 |
+
0.8889 of the time and only matches the base at 0.2917. Both runs have complete
|
| 152 |
+
action-bound safety labels, unsafe execution rate 0.0000, and pairwise causal
|
| 153 |
+
calibration ECE 0.1633 over 12,434 within-chart measured pairs. Therefore the
|
| 154 |
+
plain score-source LCB rule is a useful safety/reporting artifact, but a
|
| 155 |
+
negative selector result rather than a reliable causal dominance certificate.
|
| 156 |
The nonlinear selector path previously accepted chart-compat feature sets
|
| 157 |
without loading selector chart maps, making that diagnostic too weak. It now
|
| 158 |
loads the same deployment-visible selector chart maps as the ridge selector and
|
|
|
|
| 362 |
success selector gap 0.2431, with pairwise causal calibration ECE 0.0150
|
| 363 |
over 12,434 held-out candidate pairs. It uses train-only target--source
|
| 364 |
RGB-stat chart compatibility and remains a selector diagnostic.
|
| 365 |
+
- `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test` and
|
| 366 |
+
`runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test_tau0` contain
|
| 367 |
+
the score-source conformal LCB dominance diagnostics for the same K=16 rows.
|
| 368 |
+
They now report pairwise causal calibration ECE 0.1633, complete
|
| 369 |
+
action-bound safety label coverage, unsafe execution rate 0.0000, and
|
| 370 |
+
selected success 0.2778--0.2917, so they are negative selector diagnostics.
|
| 371 |
- `runs/ctt_base_context_obs_nonlinear_dominance_chartcompat_obs_*_envclip_k16_train_to_test`
|
| 372 |
and `runs/ctt_base_context_obs_nonlinear_dominance_chartcompat_obs_*_reg_envclip_k16_train_to_test`
|
| 373 |
contain the fixed nonlinear chart-compat diagnostics. They now log
|