auto-sync 2026-07-03T17:50:39Z workspace (part 4)

workspace/reports/2026-07-03.md

@@ -15,6 +15,9 @@
 - Fixed the CTT rollout loader to avoid repeated `.npz` decompression, and added a CPU sequential fallback because ManiSkill `physx_cpu` does not allow `num_envs > 1`.
 - Updated measured rollout metrics and summaries to report base/selected/proposal-oracle/hidden-oracle success, utility gaps, success support gap, success selector gap, outcome-vector schema, and internal paper-gate pass/fail.
 - Added measured safety-label coverage metrics in `cil/metrics.py`, `scripts/eval_metrics.py`, and `scripts/build_ctt_rollout_comparison.py`. Missing/null `safety_violation` fields are now treated as unknown rather than safe, so current reports show coverage while withholding unsafe-rate numbers when labels are absent.
+- Updated `scripts/eval_ctt_generated_rollout.py` to record action-bound validity labels before optional clipping. Outcome payloads now include `safety_violation_source=action_bounds`, `action_bound_violation`, and `action_clip_max_abs`; the script also supports `--disable-action-clipping` for raw-action replay audits.
+- Updated `scripts/slurm/eval_ctt_generated_rollout.sbatch` with `CLIP_ACTIONS=0` support.
+- Added `scripts/audit_action_bounds.py` to audit whether exported chart actions lie inside the ManiSkill deployment action space and to write a full run artifact under `runs/action_bound_audit_rgb_refs`.
 - Extended `scripts/eval_learned_dominance_selector.py` with logged `--feature-set` and `--target` variants, including success-weighted and tangent-code diagnostics, while keeping the original utility-margin default unchanged.
 - Extended `scripts/eval_learned_dominance_selector.py` again with deployment-visible `context` and `context_tangent` feature sets using target task id, instruction hash, source task id, and source-target task match. These features do not read evaluator-only val/test outcomes.
 - Extended `scripts/eval_dominance_selector.py` with `--score-source checkpoint`, cached chart embeddings/base scores, and single-thread torch scoring so a standalone train-only utility-energy checkpoint can be used for dominance evaluation.
@@ -99,6 +102,10 @@
   - `15116892`: `base_context_obj` seed 2, validation 69 positive-support charts, K=8
   - final scheduler state: all three jobs completed with exit code `0:0`.
   - purpose: object-layout proxy passed the support/safety gate by mean positive distance, so the next evidence must be measured OutcomePTR rather than another proxy claim.
+- Submitted no-clipping visual-stat CTT measured rollout refresh jobs after the action-bound audit found a global action-scale mismatch:
+  - `15117739`, `15117740`, `15117741`: validation `base_context_obs`, seeds 0/1/2, 69 charts, K=8, `CLIP_ACTIONS=0`
+  - `15117742`, `15117743`, `15117744`: test `base_context_obs`, seeds 0/1/2, 48 charts, K=8, `CLIP_ACTIONS=0`
+  - latest scheduler state checked after submission: all six array tasks are running on compute nodes.
 
 ## Jobs Completed
 
@@ -239,6 +246,22 @@
   - OutcomePTR@8 `0.5024`, success support gap `0.2947`, success selector gap `0.1787`
   - generated/base/selected safety-label coverage: `0.0000`; unsafe-rate metrics are not reported because all current `safety_violation` labels are null.
   - interpretation: proxy gate pass did not translate into better measured support. Object-layout-only is worse than RGB-stat on proposal oracle (`0.3816` vs `0.4058`) and selected success (`0.2029` vs `0.2415`), so it is a negative representation diagnostic.
+- Action-bound rollout safety smoke completed:
+  - direct artifact: `runs/ctt_residual_base_context_obs_rollout_safety_smoke_direct_seed0`
+  - Slurm artifact: `runs/ctt_residual_base_context_obs_rollout_safety_smoke_seed0`, job `15117529`, completed `0:0`
+  - setting: `base_context_obs` seed 0, validation, one target chart, K=2, default clipped execution.
+  - result: base and both generated candidates have known action-bound labels and all violate the `[-1,1]^7` action space before clipping (`base_action_clip_max_abs=1.7218`, generated max excess `1.3133` and `1.5043`).
+  - interpretation: the new safety hook is working, and it exposes action-representation mismatch rather than collision/contact safety.
+- No-clipping action-bound smoke completed:
+  - artifact: `runs/ctt_residual_base_context_obs_rollout_safety_noclip_smoke_direct_seed0`
+  - setting: same one target chart and K=2, but with `--disable-action-clipping`.
+  - result: raw replay keeps the same action-bound violations; candidate progress changes from `[0.1141, 0.0396]` under clipping to `[0.1700, 0.0436]` without clipping. This is enough to justify separate full no-clipping validation/test refresh jobs, not enough for a method-success claim.
+- Action-bound audit completed:
+  - artifact: `runs/action_bound_audit_rgb_refs`
+  - train: action violation `0.3075`, base-action violation `1.0000`, max excess `3.6421`
+  - validation: action violation `0.3076`, base-action violation `1.0000`, max excess `3.5995`
+  - test: action violation `0.3079`, base-action violation `1.0000`, max excess `3.4353`
+  - interpretation: every exported base branch in the RGB-ref chart DB exceeds the nominal ManiSkill action space. Existing clipped rollout tables remain useful support/selector diagnostics, but they cannot be final deployment or unsafe-contact evidence until action representation/clipping is explicitly resolved.
 - Base-context-observation CTT proxy diagnostics completed for three seeds:
   - train artifacts: `runs/ctt_residual_base_context_obs_seed{0,1,2}`
   - proxy artifacts: `runs/ctt_residual_base_context_obs_seed{0,1,2}_val_proxy`
@@ -383,6 +406,8 @@ Interpretation: the test proposal oracle clears the internal 50% support target,
 
 Safety metric status: regenerated measured rollout reports now expose `base_safety_label_known`, `generated_safety_label_coverage_at_8`, and `selected_safety_label_known_at_8`. For the visual-stat, object-layout, held-out test, and train-calibration rollout rows, each coverage number is `0.0000`; the legacy validation aggregate without `candidate_outcomes` reports `n/a`. No `generated_unsafe_rate_known` or selected/base unsafe-rate claim is made because the current ManiSkill outcome payloads carry null `safety_violation` labels.
 
+Action-bound audit status: new rollout artifacts now log action-bound safety labels. The first smoke has 100% known labels and 100% action-bound violation for base/generated candidates. The full chart DB audit shows base-action violation rate 1.0000 in train/val/test and branch-level violation rate about 0.308. These are action-space validity labels only; collision/contact safety remains unmeasured.
+
 ## Leakage Audit Status
 
 - `data/cil_charts/train`: outcomes visible, retrieval allowed.
@@ -424,6 +449,7 @@ Safety metric status: regenerated measured rollout reports now expose `base_safe
 - Updated the validation proxy gate discussion to account for object-layout rows now present in `runs/ctt_val_proxy_comparison/table`: RGB-stat remains best on PPTC@0.20, object-layout-only is slightly best on mean positive distance, and measured rollout shows the object-layout row is negative relative to RGB-stat support.
 - Added the measured object-layout validation rollout table via `\input{../runs/ctt_base_context_obj_val_rollout_comparison/table}` and described it as a negative representation diagnostic.
 - Added safety-label coverage semantics to the metrics section and limitations: current measured artifacts log `safety_violation` fields but have 0% known safety labels, so unsafe execution rate is explicitly unmeasured rather than zero.
+- Added action-bound audit table via `\input{../runs/action_bound_audit_rgb_refs/table}` and described the finding as an action-representation validity confound, not a collision/contact safety result.
 - Reframed the title/abstract around `Counterfactual Action Atlas`; CTT is now the current transport generator inside the Atlas, not the whole novelty.
 - Tightened the abstract to avoid overclaiming a successful learned generator: the paper now says the Atlas measures local do-action geometry and audits whether deployment-clean generators reach positive support.
 - Added the train-calibration leakage rule to the split contract: when target charts are from train, retrieval excludes same chart id and same restored-state hash.
@@ -434,11 +460,17 @@ Safety metric status: regenerated measured rollout reports now expose `base_safe
 - Updated limitations: current evidence includes validation/test measured rollout, but the selected action fails the acceptance gate; real-robot recovery, unsafe-contact measurement, and dominance-calibrated deployment remain missing.
 - Added local-atlas/proxy-comparison scripts to the reproducibility artifact list.
 - Added generated-rollout script and Slurm wrapper to the reproducibility artifact list.
+- Verification after the action-bound patch:
+  - `.venv/bin/python -m py_compile scripts/eval_ctt_generated_rollout.py scripts/audit_action_bounds.py tests/test_ctt.py`
+  - `.venv/bin/python -m pytest tests/test_ctt.py tests/test_metrics.py tests/test_cil_metrics.py -q`: 19 passed.
+  - `latexmk -pdf -interaction=nonstopmode main.tex`: PDF already up-to-date.
+  - Overclaim scan for `A*`, `Q1`, `Best Paper`, `SOTA`, and `state-of-the-art` in the main paper/report files returned no matches.
 
 ## Next Actions
 
 1. Stop treating deterministic object-layout as a likely win: it passed proxy but failed measured support relative to RGB-stat.
 2. Replace the deterministic RGB/object descriptors with learned visual-language or task/object tokens that can distinguish target, distractor, gripper/contact region, and task stage.
 3. Train utility/dominance on richer outcome-vector features; the best clean train-calibrated selector still reaches only `0.3125` selected success.
-4. Add actual unsafe/contact labels to rollout outcomes; metric hooks now exist, but current safety-label coverage is `0.0000`.
-5. Rerun validation/test measured rollouts only after a representation or utility model change, not another shallow selector/proxy-only change.
+4. Wait for no-clipping validation/test refresh jobs `15117739`--`15117744`, then aggregate separately from the clipped rollout tables.
+5. Decide whether to fix action scaling at export/decode time or treat action clipping as the deployment constraint; do not mix clipped and raw-action rollout claims.
+6. Add actual collision/contact labels after action-bound validity is resolved.

workspace/reports/draft_audit_ctt.md

@@ -133,6 +133,20 @@ validation aggregate without `candidate_outcomes` reports `n/a`. Therefore the
 draft may say the safety metric hooks exist, but it must not claim an unsafe
 execution rate yet.
 
+A follow-up action-bound safety pass exposes a stronger current blocker.
+`scripts/eval_ctt_generated_rollout.py` now records pre-clipping action-space
+violations as `safety_violation_source=action_bounds`, and
+`scripts/audit_action_bounds.py` writes `runs/action_bound_audit_rgb_refs`.
+That audit finds base-action violation rate 1.0000 in train, validation, and
+test, with about 30.8% branch-level action-bound violations and max excess
+3.64. A one-chart clipped and no-clipping smoke confirms the labels propagate
+into measured rollout rows. This is not a collision/contact safety result; it is
+an action-representation validity confound. Existing clipped rollout tables
+remain support/selector diagnostics, but the paper must not present them as
+final deployment evidence until action scaling or clipping policy is resolved.
+Separate no-clipping validation/test refresh jobs are submitted as
+`15117739`--`15117744`.
+
 Current artifact evidence:
 
 - Phase-0 reproduce bundle exists: `runs/reproduce_v0/`.
@@ -155,6 +169,11 @@ Current artifact evidence:
   `scripts/build_ctt_rollout_comparison.py` now expose measured safety-label
   coverage and known-label unsafe rates. Current coverage is 0% for the
   regenerated rollout rows, so unsafe execution remains unmeasured.
+- `scripts/audit_action_bounds.py` and `runs/action_bound_audit_rgb_refs`
+  expose a global action-space validity issue: every base branch in the
+  RGB-reference chart DB is outside the nominal ManiSkill action bounds before
+  clipping. This must be resolved or explicitly separated before any final
+  deployment-clean rollout claim.
 - `runs/ctt_residual_rollout_val69_seed{0,1,2}` and
   `runs/ctt_val_rollout_comparison` contain the first full validation measured
   rollout results for residual CTT.
@@ -412,6 +431,8 @@ Resolved in this pass:
 Still missing before a full method paper:
 
 - actual unsafe/contact labels and simulator-integrated fallback evaluation
+- action representation/clipping policy for measured rollout; current base
+  chunks violate action bounds before clipping
 - simulator-integrated dominance-calibrated rollout collection
 - visual-language/object-centric chart tokens that make train-only utility
   energy transfer beyond the current negative checkpoint result. The feature
@@ -444,6 +465,9 @@ Current missing:
 
 - Unsafe-contact labels and fallback-rate audit over executed calibrated
   rollouts.
+- Action-bound validity is now audited and failing for base chunks; this is a
+  prerequisite blocker for interpreting clipped rollout results as deployment
+  evidence.
 - Real robot near-miss recovery audit.
 - A dominance model that approaches the selected-success paper gate. The current
   learned ridge row only marginally beats the base, and the full train-only
@@ -480,7 +504,8 @@ Next:
    0.3264 selected success.
 3. Replace the validation-calibrated ridge/context selector with a stronger
    deployment-clean train-only dominance model.
-4. Add actual unsafe/contact labels; safety coverage/known-label metric hooks
-   are now implemented, but current coverage is 0%.
-5. Improve chart tokens from base-action summaries to visual-language and
-   object-centric geometry.
+4. Resolve action scaling/clipping: compare pending no-clipping jobs
+   `15117739`--`15117744` against clipped rollouts, then decide whether the
+   chart export or rollout decoder should normalize actions.
+5. Add actual unsafe/contact labels; action-bound labels now exist, but
+   collision/contact safety remains unmeasured.