Buckets:
| # Pre-registration companion log — deviations and pre-analysis exclusions (append-only; the hashed `docs/newtest_prereg.md` is never edited) | |
| Registered protocol: `docs/newtest_prereg.md`, sha256 `b430096955129e32de09c5aa518592100b499f0db3c58298a89921b08d1616d1` (2026-09-12). Machine-readable twins consumed by the analysis: `results/newtest/deviations_v1.json` (clips), `results/gaze_validation/deviations_v1.json` (gaze windows). Every entry below is script output of `eval/newtest_render.py`; the analysis scripts apply the rule stated per entry and print it in their tables. | |
| | date (UTC) | unit | what | reason (verified) | rule applied in the analysis | effect on the registered design | | |
| |---|---|---|---|---|---| | |
| | 2026-09-13 | S1 clip idx 70074 (engaged, cell A+B, π 0.25, stratum `_pooled` n 1 / N 4) | cannot be rendered by anyone | leaf `Unknown/driver_84/route_1/route_0-1` has **no `qcamera.mp4`** locally or on the canonical remote (dcamera only) → no road video exists | unit excluded; its stratum has no mates → the stratum becomes uncovered: `N_covered_cell` of engaged\|A+B 132 → 128 (coverage 0.88 → 0.853); HT weights of the cell's other strata re-scaled by N_cell / N_covered (registered §3 rule) | S1 160 → 159 rated clips; frame and π of every other unit unchanged | | |
| | 2026-09-13 | S1 clip idx 70149 (manual\|A), S2 clip idx 80027 (engaged\|silent), S3 clip idx 34329 | post-anchor clip shorter than 10 s | route ends 3.45 s / 2.6 s / 8.75 s after the anchor (source `dcamera` + `qcamera` durations 63.65 / 300.0 / — s = remote sizes; genuine route ends, not truncated copies) | **kept**, flagged `post_truncated` + `post_content_s` in `clips_newtest.json`; Task 1 prefix intact; Task 2 / outcome raters see the shorter clip with an on-screen note and may answer never / unknown; the renderer's gate now excludes only clips with < 1 s of post content (none) | none on π; 3 clips carry the flag (analysis lists them) | | |
| | 2026-09-13 | 20 gaze windows: 19 on `VOLKSWAGEN_TIGUAN_MK2/driver_25/route_2/route_0-87` (idx 90011, 90033, 90035, 90047, 90069, 90076, 90084, 90093, 90095, 90102, 90121, 90123, 90128, 90129, 90135, 90155, 90156, 90188, 90193; t 2321–5174 s) + 1 on `KIA_NIRO_PHEV_2022/driver_131/route_1/route_0-17` (idx 90106, t 1076.0 s) | **no cabin frame at t** | TIGUAN: `dcamera.mp4` is byte-identical to the remote (2,915,247,959 B) and ends at **2284.3 s** while `qcamera.mp4` runs to 5280.0 s; **all** 13,899 engaged + 907 manual `face_invalid\|kin_invalid` val rows of that device lie at t > 2284.3 s (no face-valid row does). NIRO: cabin video ends at 1075.9 s, road at 1080.0 s (route-end gap). The automatic state `face_invalid ∧ kin_invalid` there means "no cabin video", not occlusion | windows excluded from human rating (nothing to rate) and reported as their own **frame-level category "no cabin video at t"** with their design weight 1/π (not treated as missing at random); their strata mates keep the original π; the pre-registered "share of face_invalid with occlusion" is estimated on rated windows only (cabin video present) and the "no cabin video" share is reported next to it | cell `face_invalid\|kin_invalid`: 25 drawn → **5 ratable** (engaged 4 / manual 1); HT weight share of the cell that is "no cabin video" **0.803**; all other cells unchanged (180 ratable windows: manual 92 / engaged 88). Gaze calibration window 90093 (excluded) replaced by 90148 (same cell, manual; display flag only, logged by the renderer) | | |
| | 2026-09-13 (before any panel label is analysed; 1 prefix record exists in the store, id `G1`, clip 29271, no label analysed) | **Labelling protocol** | owner's decision: annotation is done on ONE PC by a **five-person committee ([annotator], [annotator], [annotator], [annotator], [annotator])** that discusses each clip; **one operator enters the consensus**. No independent blind raters, no separate Task-2 reviewer ("我不需要你做事件回顾") | registered §4 (two blind raters R(i mod 3), R((i+1) mod 3); Task 2 by R((i+2) mod 3) who has not seen the prefix; adjudication by ADJ*) is **replaced**: tool `--panel G1` serves every clip / window once to the panel id (`G1`, the operator's id; live 2026-09-13); the panel's outcome stage carries the EVENT_FIELDS, stored as a second `stage=event` record (`protocol=committee_consensus`, `panel=G1`, `source_tag=human_h5_panel`; schema h5-v1 unchanged); `h5_to_consensus.py --panel G1` takes the panel record as the label (status `panel_consensus`; the existing `G1` prefix record is native — no alias; the store is never edited); analysis scripts accept `panel_consensus` | **Consequences, stated plainly:** (i) no inter-rater reliability estimate — the pre-registered gates Q4 κ ≥ 0.6, event_type κ ≥ 0.6, onset median \|Δt\| ≤ 1 s are **not applicable and not passed**; every result table prints this; (ii) event timing (Task 2) is **not blinded** to the prefix judgement — the same committee sees the prefix, judges the warrant, then watches the outcome and marks the event onset; the lead-time estimate therefore carries a possible anchoring bias that the blind design would have excluded; (iii) effective n_raters = 1 per clip (a committee decision counts as one label); (iv) calibration session becomes a rubric walk-through, the interim κ checkpoint (09-19) is dropped; (v) paper wording: "labels are the consensus of a five-person committee (H., N., L., J., Y.) entered by one operator"; no κ is reported; the prefix lock (Task 1 judged before the outcome is visible) is unchanged and remains the design's safeguard against outcome bias on W_i. Estimators, π, thresholds, windows and wording rules of §5–§6 are unchanged | | |
| | 2026-09-13 (~01:20 EDT; live before any stage-2 panel label existed: store = 1 prefix record) | **Labelling procedure — stage-2 prefill** (owner asked to simplify stage 2 as far as possible; coordinator implemented) | on entering the outcome stage the tool **prefills** some fields and tags them amber "预填 · 请核对" until touched. Rule `can_rule_v1+own_prefix`: (i) from the post-anchor CAN rows — `vehicle_response_kind`: brake flag or accel < −2 m/s² → braking; speed > 2 → < 0.5 m/s → stop; else steering flag → steering_correction; else gas + accel > 0.5 → acceleration; else none; `vehicle_response_attributed_to`: driver flags → driver; kind none → none; else assist engaged → automation, else unclear; (ii) from the panel's own stage-1 answers — `first_gaze_to_road_s` = anchor when view_region = road_ahead and head_motion ∉ {sustained_turn, unknown}; `first_hand_to_wheel_s` = anchor when both hands on_wheel; `scene_resolution` = demand_resolved when notable_event = none and compatibility = compatible; (iii) `event_type` = [none], `event_onset_s` = null, `ongoing_at_anchor` = no, `severity` = minor **only** when CAN is quiet **and** stage 1 said compatible + not_warranted + notable_event none | labels are taken as submitted (the operator must confirm or edit every field; the panel outcome stage has a watch-to-end gate). **Provenance stored per record**: top-level `prefill: {rule, fields (all prefilled), accepted (still untouched at submit)}` on the outcome record and on the split event record. `h5_to_consensus.py` prints, per field, prefilled-and-accepted vs edited vs no-prefill; the event consensus carries `event_type_prefill_accepted`; `newtest_analysis.py` prints how many E = none labels came from an accepted prefill (and stores `E_none_from_prefill` per clip) | **Consequences:** (i) the vehicle-response fields become CAN-derived by default (they were already meant to follow the log); (ii) `first_gaze` / `first_hand` = anchor and `scene_resolution` = resolved are inherited from the stage-1 judgement — no longer an independent reading of the outcome video; (iii) the **E = none default** under the quiet-CAN + benign-stage-1 condition can hide driver-side events (sustained gaze away, hands off, phone) that leave no CAN trace — exactly the events the event-recall metric is about; the pre-registered estimator is unchanged, but the count of E = none labels that were accepted prefills is printed next to every recall figure so a reviewer can bound the effect (re-derivable from `clips.csv`); (iv) accepted-prefill shares per field are reported descriptively; nothing is silently converted | | |
| | 2026-09-13 (~01:50 EDT) | **Incident — outcome stage invisible to human raters** (tool bug inherited from the round-1 client; found by the coordinator from the owner's live DOM + client beacons) | the page CSS hides the stage-2 container (`#stage2{display:none}`) and the JS "showed" it with `style.display=''`, which only removes the inline override and lets the CSS rule hide it again — the outcome / event fields were rendered (beacon: 4,647 chars, 12 fields) but never visible; the same for the gaze window. Invisible to the stub-DOM harness (no CSS) and to the endpoint smoke suites; round 1 never reached the outcome stage (0 records), so it went unnoticed | fixed live (`style.display='block'` at the four stage toggles; server restarted, panel mode unchanged); static smoke check added: no `''` toggle on a CSS-hidden container | **No label lost**: the store held exactly 1 prefix record (G1, clip 29271) and no outcome could have been submitted before the fix; every outcome / event / gaze record in the store post-dates the fix. Timing pilot of the outcome stage effectively starts at the fix time | | |
| | 2026-09-13 (2026-09-13T21:46:59Z; **written before the draw**) | **Supplementary post-hoc gaze stratum `gaze_supp`** (owner: 如果有需要补抽 … 请你开始运行; IRB approved) | the pre-registered cell `face_invalid\|kin_invalid` kept only 5 ratable windows because 20 of its 25 draws had no cabin frame at t (TIGUAN leaf beyond 2284.3 s + one route end) | **Frame (computed by `eval/gaze_off_validation_sample.py --supplement face_invalid_covered`, script output):** VAL rows of the cell = 15,654; minus the TIGUAN leaf 14,806 (all beyond its cabin video); minus leaves with no dcamera 0 (every remaining leaf has a dcamera locally or in the remote listing); minus rows on leaves with a local dcamera whose t > duration − 0.25 s 21 (KIA NIRO route end 20, Silverado route end 1) → **N = 827** = 775 rows with verified coverage (23 leaves with local dcamera) + 52 rows on 5 leaves without local video (coverage verified at render; a window whose fetched cabin video does not cover t is excluded and logged, π unchanged). Frame composition manual 704 − (not-covered manual) / engaged 144 − (not-covered engaged), exact split printed by the sampler. **Draw:** target n = 20, strata = regime with allocation proportional to N (largest remainder), stratified SRS with the frozen mechanics (device cap 20 within the supplement, 5 s same-route guard), seed 0, idx from 91000, `--expect-n 827` aborts if the frame differs from this entry; π_i = n_regime / N_regime **within the supplementary frame**; own manifest `results/gaze_validation/manifest_supp_v1.csv` + `frame_counts_supp_v1.json` + `sha256_supp_v1.txt` + run json. **Reporting rule:** the supplement is reported (a) alone with its own HT weights, (b) side by side with the original cell's 5 ratable windows (original design weights, plus its 0.803 'no cabin video' share), and (c) in one row explicitly labelled **post hoc pooled (cell rows with cabin video)** — ratio estimators over the union of the two samples, each unit with its own design weight — never merged into the pre-registered cell weight, never presented as pre-registered. Only `share_face_invalid_occluded` and the confusion-table rows of the face_invalid state are affected; the gaze_off precision / recall metrics condition on face-valid rows and are untouched. Windows served to the panel with the same task; the `note` field may carry attendance initials as elsewhere | | |
| | 2026-09-13 (01:08Z) | **Post-hoc committee marker — sensitivity only** | on clip #11 (human driving, ~3 km/h creeping, driver on the phone) the panel submitted stage 1 as not_warranted per the rubric (stopped / creeping → reason 6) and, after the lock, wished it had been a prompt | **The prefix lock stands** (pre-registered: the outcome never revises the prefix; hindsight protection). Additive convention: when the committee disagrees with its own locked Q4 it writes `[事后:提示A]` / `[事后:提示S]` / `[事后:不提示]` (alias `[posthoc:A|S|D]`) into the stage-2 `outcome_note` (or `event_note`). `h5_to_consensus.py` parses it into `posthoc_q4` / `posthoc_flag` / `posthoc_raters`; **the primary label stays the locked stage-1 answer**. `newtest_analysis.py` keeps every primary estimate unchanged and appends one block labelled SENSITIVITY — W_i replaced by the post-hoc verdict on the flagged clips only — with the flagged count; event recall / lead time do not depend on W_i. No server change (notes were already stored) | Why acceptable as a sensitivity: committee mode has no blinding between stages anyway (the same people judge the prefix and watch the outcome), so the marker only makes an existing hindsight opinion explicit and auditable instead of leaving it in free text; it can never enter the primary because the parser writes separate columns. Also a rubric signal: if many markers concern the creeping-with-phone case, the rubric's reason-6 wording is revisited (README addendum, logged) — never retroactively | | |
| Owner decision requested (2026-09-13, morning): a supplementary post-hoc draw of `face_invalid|kin_invalid` windows restricted to rows **with** cabin video — **approved by the owner and pre-registered in the entry above; drawn after that entry.** | |
| ## 2026-09-13 ~01:51 EDT — store reset before the programme proper (owner decision) | |
| All records collected up to this point (5 lines, panel id G1, clip 29271 only: 1 prefix, 1 outcome, 1 event, 2 outcome-skips) were produced during the tool-fixing session (outcome stage invisible until ~01:50 EDT; button / prefill / layout changes in between) and before the rubric clarifications were complete. At the owner's request the store was archived to `cwm2/annotation_human/archive/labels_newtest_reset_20260913T055134Z.jsonl` (read-only, excluded from every analysis) and the live store restarted empty. Effect on the pre-registered design: none of the archived records enters any estimate; the timing pilot (first 5 clips) restarts; the sampling frame, thresholds, π and clip order are unchanged. | |
| | 2026-09-16T07:10:33Z (written before any A′ score is compared with a human label) | **Post-hoc systems A′ and A″ (plan R2 WP2.4; pre-registration row 2026-09-16T06:46:35Z in decisions.md)** | the reviewer point "the best warning head contains no predicted quantity" led to a paired arm that the frozen programme did not have: **A′ = both+dwell+face** = the joint model's predicted-future features (the 144 `is_future` features of A) PLUS the current-state blocks of B (dwell 11 + face 16), HGB, same 979 labels / grid / CV / clean-val temperature as B; score `W_learned_bothhgb_t15_w10_n979dwfc`, **mean over the 3 frozen seeds** of `results/runs/W_I1_all_minus_road2drv_s{0,1,2}_final_dwfcboth/pred_layout_{regime}_{test,val}.npz` (trained 2026-09-16 by `train/train_warning_head.py --feature-sets current,future,both --extra-features dwell,face --w-suffix _dwfcboth`; OOF twin `_oofdwfcboth`). **A″ = placeholder** for the best WP4 predictive read-out (F2′ / F8 future-state features `fs_*`); no layout exists yet | A′ / A″ are scored on the SAME rendered clips at thresholds fitted on the VAL layouts to the registered 10 / h and 20 / h loads (`operating_points.thr_for_rate`, 60 s cooldown) and reported in a block labelled **POST HOC (not pre-registered)** next to A / B / C; the primary A / B / C analysis, π, cells, estimators and wording rules are unchanged; A′ / A″ never enter the S1 union or any inclusion probability (the frame was drawn from A / B / C alerts, so A′-only events are uncovered — stated in every A′ row); no A′ number is compared with a human label before this row exists | none on the registered design; two additional descriptive rows per operating point; the deviations JSON carries the machine-readable twin (`posthoc_systems`) | | |
| ## 2026-09-21 — FINAL pre-registered analysis on the complete store (run 20260921T211937Z; archive `results/newtest/analysis_20260921T211937Z/`) | |
| | date (UTC) | unit | what | reason (verified) | rule applied in the analysis | effect on the registered design | | |
| |---|---|---|---|---|---| | |
| | 20260921T211937Z (store snapshot 20260921T211937Z, 897 records, sha256 0b7da0518bd2…) | **Final analysis on the complete store** | all 299 clips labelled (owner: 我完成了标注); `eval/newtest_analysis.py` (byte-identical copy run in a scratch mirror, outputs redirected; live results/newtest untouched) + `h5_to_consensus.py --panel G1` (prefix / event; copy with the one-line `from collections import Counter` fix) run on **299 of 299 rendered clips** complete with prefix + outcome + event (S3 100/100, S1 159/159, S2 40/40); S3_hard (20, optional) never rendered | 0 clips not yet labelled: | not-yet-labelled clips are **excluded = not labelled** (W = E = NaN; pi, N_cell, N_covered_cell unchanged; no re-weighting); HT ratio estimators unaffected in expectation under MAR within cell; HT totals (FA / h, silent warranted / h) lose the missing weight mass — quantified per cell in `coverage.md` (S1 manual 0.000, S1 engaged 0.000, S2 manual 0.000, S2 engaged 0.000) with an interim within-cell re-weighting sensitivity (not pre-registered); the pre-analysis exclusion 70074 applied as logged | none on pi / thresholds / cells / estimators / wording; this run is the FINAL pre-registered analysis (supersedes the interim 20260921T211305Z run on 295 clips, whose numbers must not be mixed with it); tables carry the run stamp and the prereg sha256 (MATCH) | | |
| | 20260921T211937Z | **MDE column of the analysis script uses the observed discordance** | `newtest_analysis.py` prints `mde_recall = 2.8 * sqrt(disc_observed / n)` whereas prereg §6 fixes the discordance at 0.4 (`2.8 * sqrt(0.4 / n)`; §9 forbids changing the formula); found by the independent re-derivation | verified in `paired.csv` (e.g. manual val20 A-C: observed 0.30 -> 0.34; registered 0.40 at n = 20) | the assembled tables print BOTH columns ('registered' and 'observed'); the 'matches' auxiliary flag uses the registered value; the pre-registered wording verdicts are CI-based and unchanged (24 / 24 agree) | none on any verdict; every paired Δ now carries the registered MDE beside it as §6 requires | | |
| | 20260921T211937Z | **Coverage display after the 70074 exclusion** | the analysis' own S1 coverage table and `cells_enriched.csv` print the frozen engaged|A+B coverage 0.880 (132 / 150) although the weights in force use N_covered 128 (coverage 0.853, as logged 2026-09-13); `coverage_statement.py` also copied the frozen `coverage_cell` | verified: every per-clip weight and HT sum equals an independent derivation with N_covered 128 (max |d| 7e-12) | display only; `coverage.md` / `coverage_cells.csv` and the headline per-cell table recompute coverage = N_covered / N; the analysis script's section is left as generated (byte-identical registered code) with this row as the correction | none on estimators | | |
| | 20260921T211937Z | **Onset precedence and 'censored 0'** | `load_labels` takes the Task-1 prefix onset first, then the Task-2 event onset, and censors at the anchor only when neither exists; 21 E = 1 clips with `ongoing_at_anchor = yes` carry an explicit Task-2 onset (range -3.15 .. +1.15 s around the anchor) and are therefore NOT censored; a stricter reading of the prereg (ongoing without a prefix onset -> censored) would flag them | verified by both independent re-derivations; manual val20 HT recall unchanged under the strict reading (0.235 / 0.088 / 0.000 on the 295-clip snapshot) | precedence as implemented; stated in the headline reading notes | none on the headline numbers | | |
| | 20260921T211937Z | **Task-2 onset stored on 7 E = 0 clips (display only)** | 7 clips with `event_type = none but onset marked` (schema flag) get `onset_source = task2_event` in `clips.csv` although E = 0 (14735, 25408, 33238, 37856, 70026, 70106, 70149); the E = 1-restricted onset split is 53 prefix / 25 event / 0 censored in both derivations | verified | no estimator reads an onset of an E = 0 clip | none | | |
| | 20260921T211937Z | **Independent verification** | two verifier agents re-derived W_i / E_i / onsets / HT weights / alerts / recall / FA / precision / AUROC / wording from the raw store snapshot and the frozen layouts with their own code (not the analysis script): 295-clip interim snapshot CONFIRMED (own device-cluster bootstrap A-C Δ +0.235 [+0.029, +0.533], 1.1 % of resamples at Δ ≤ 0); 299-clip final run CONFIRMED (0 per-clip mismatches on W / E / w / det_*; all 24 wording verdicts agree; every table cell equal to 1e-6) | scripts and logs archived with this run (`verify/`) | — | — | | |
| | 20260921T211937Z | **Fast mode never used** | the prepared fast-mode patch (`response/annotation_fast_mode/ann_fast_mode_v2.patch`, deviation row draft) was **not applied**: every one of the 897 records in the store carries no `mode` tag (modes = {'None': 897}); all fields were collected in the full form | verified from the store snapshot | none; the fast-mode deviation row draft is void unless the owner applies the patch for the remaining clips | none | | |
| | 20260921T211937Z | **Single committee panel G1 (already logged 2026-09-13)** | all 897 records annotator = G1, protocol committee_consensus; `h5_to_consensus.py --panel G1` -> status panel_consensus for every clip; kappa gates n/a; post-hoc `[事后:…]` markers: 0 flagged -> no sensitivity block | verified from the store snapshot and the consensus output | as logged 2026-09-13 | as logged | | |
| | 20260921T211937Z | **Server schema flags kept as submitted** | 20 stored records carry `schema_ok = false`: {'event_onset_s before anchor': 5, 'first_hand_to_wheel_s before anchor': 2, 'event_type none but onset marked': 14}; the consensus scripts do not filter on `schema_ok` (pre-existing behaviour) | verified from the store snapshot | 'event_type none but onset marked' -> E = 0 (onset ignored); 'event_onset_s before anchor' -> the Task-2 onset is used only when no Task-1 prefix onset exists (`load_labels` order) — listed per clip in `consensus_summary_per_clip.csv` | none on estimators; reviewers can re-derive with these clips dropped | | |
| | 20260921T211937Z | **Stage-2 prefill (logged 2026-09-13) — counts at this snapshot** | E = none from an accepted prefill: 76 of 221 'none' labels (299 labelled clips); `event_type` prefilled & accepted share 0.26 (h5_to_consensus output) | script output | printed next to every recall figure by the analysis (`E_none_from_prefill` in clips.csv) | as logged | | |
| | 20260921T211937Z | **gold_extended.py input restricted to S3** | the Task-1 consensus CSV now also holds S1 / S2 clips on TEST devices, which `gold_extended.r2_label_frame` would otherwise pull into `clean_test_r2`; the pre-registered wording is 'clean_test_r2 rows from the S3 clips' | verified in the code path | `gold_extended.py --consensus h5_consensus_S3only.csv` (100 S3 rows; unsure excluded by the script; rows not anchorable on the TEST split dropped by `warning_gold.build_gold`, counts printed) | none | | |
| | 20260921T211937Z | **Post-hoc A' / A'' (logged 2026-09-16)** | A' scored by `tools/posthoc_aprime.py` (val-fitted 10 / 20 per h + test-achieved; same estimators, weights, resamples) in a POST HOC block; A'' has no layout -> not available | script output | FA / h and precision of A' are frame-restricted (A' never entered the S1 union); share of A' TEST alerts within 10 s of an A/B/C frame alert printed per point | none on the registered design | | |
Xet Storage Details
- Size:
- 24 kB
- Xet hash:
- b8636aeaa87fa222edb65ce634e7fdf4b4d01aa8191a61d3b4948a6982c8e366
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.