Buckets:
| """cwm3/data/build_kin_causal_v3.py — hands_on / hand-event label v3 (pre-registered docs/decisions.md 2026-09-16T16:45:31Z). | |
| Imports data/build_kin_causal.py (v2, FROZEN) as a module and reuses its route-level machinery unchanged: pose loading | |
| (raw_from_kpts), channel series (clip_series), the causal expanding median (expanding_median_moving), the head-turn hysteresis | |
| (hysteresis), the 0.2 s-grid onset helper (onsets), route ordering (route_key), constants and the s12 alignment / EM assertions. | |
| WHAT CHANGES (targets only): | |
| wheel reference = causal expanding median of each wrist coordinate (Lwri_x, Lwri_y, Rwri_x, Rwri_y) over MANUAL MOVING frames | |
| <= t: MANUAL = grid10 `engaged` flag False (../s14_corr/cache/grid10/<uid>.npz keys t, G, engaged), MOVING = | |
| the pose npz `moving` flag (= grid10 vEgo > 2 m/s, verified identical). Device pool of earlier routes (route_key | |
| order, manual moving values only, POOL_CAP most recent) + the current route expanding; NaN while n < N_MIN. | |
| grid10 -> pose alignment: integer key k = round(t * 10) on both 10 Hz grids (pose t is float32, cast to float64 | |
| first, as v2 does); a pose frame takes the engaged flag of the grid10 row with the same key; pose frames without | |
| a grid10 row (94 of 5.94 M) are treated as NOT manual (excluded from the reference). | |
| d(t) = min over wrists of hypot(wrist(t) - wheel_ref(t)) (np.fmin: a wrist that is NaN/undefined is ignored; NaN if both) | |
| computed in float32 from the same raw wrist quantities as the channels (d is NOT a stored channel). | |
| hands_on (S) = hysteresis on d: ON if d < HYST_ON = 0.08, OFF if d > HYST_OFF = 0.12, otherwise the previous state is kept; | |
| the state is reset after a gap of more than GAP_RESET = 10 consecutive frames without d (> 1 s, v2 convention); | |
| after a reset (and at the start) the state is undecided (NaN) until the first decisive frame. S_c / S0_c are the | |
| raw causal state at the grid frames (NaN where d is undefined) — no lookahead. | |
| hand_leave / = persistence-confirmed transitions of the 10 Hz state: a transition at frame k (first frame of the new state) | |
| hand_return is an event iff the previous state persisted >= PERSIST_S = 1.0 s (10 frames) before k and the new state persists | |
| >= 1.0 s from k on. Persistence is measured on the state HELD across gaps <= GAP_RESET (a missing frame inside | |
| the run neither breaks nor confirms it); forward confirmation is allowed for TARGETS only. On the 0.2 s target | |
| grid the event is credited to the step h whose interval (t+0.2(h-1), t+0.2h] contains the stamp frame, and | |
| EM_c (label observable) additionally requires that the transition could be confirmed (route not ending < 1 s | |
| after / starting < 1 s before the transition); EM_c otherwise keeps the v2 rule finite(S[h-1]) & finite(S[h]) & M. | |
| head_turned / head_turn_onset : UNCHANGED (v2 reference over all moving frames, v2 thresholds, BK.hysteresis, BK.onsets) — asserted | |
| equal to the v2 files. | |
| INPUT channels : UNCHANGED. K_c / K0_c / Kvalid / Kframe / K0valid / row_ok / route_decoded are recomputed with the v2 code path | |
| (v2 reference = all moving frames) and asserted bit-identical to results/targets_kin_causal_routes_v2; the | |
| kinhist_causal_* files are HARD-LINKED to the v2 files (byte-identical by construction). Neither the regime | |
| flag nor the wheel reference nor d enters any input array. | |
| OUTPUT results/targets_kin_causal_routes_v3/ — same file names / keys / dtypes / row alignment as v2 plus, in kin_causal_*.npz: | |
| wheel_ref_defined bool[N] (manual-only reference defined for BOTH wrists at the anchor frame), dwheel0_c f32[N] (d at the anchor, | |
| causal; target-side diagnostic only), persist_s, hyst_on, hyst_off, gap_reset_frames, label_version="v3", wheel_ref_rule. | |
| route_refs.npz: ref__<uid> [N,5] (channel reference, identical to v2) and wheel_ref__<uid> [N,4] (v3 manual-only wheel reference), | |
| n_manual_moving__<uid>; build_stats_<utc>.json (+ _latest, merged over invocations), run_build_kin_causal_routes_v3_<utc>.json. | |
| """ | |
| import argparse, json, os, shutil, sys, time | |
| from collections import defaultdict | |
| from pathlib import Path | |
| import numpy as np | |
| ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(ROOT / "data")); sys.path.insert(0, str(ROOT / "common")) | |
| import build_kin_causal as BK # noqa: E402 (v2 machinery, frozen file, imported not copied) | |
| from runmeta import write_run_json # noqa: E402 | |
| GRID10 = BK.EXPLORE / "s14_corr" / "cache" / "grid10" | |
| V2_DIR = ROOT / "results" / "targets_kin_causal_routes_v2" | |
| OUT_V3 = ROOT / "results" / "targets_kin_causal_routes_v3" | |
| POSE_PREF = ROOT / "results" / "pose_causal_routes_v2" # v2 build: pose_dir=results/pose_causal_routes_v2, fallback results/pose_causal_routes | |
| HYST_ON, HYST_OFF, PERSIST_S, FPS = 0.08, 0.12, 1.0, 10 | |
| GAP_RESET = BK.GAP_RESET # 10 frames (> 1 s of missing frames resets the state), v2 convention | |
| P_FRAMES = int(round(PERSIST_S * FPS)) # 10 | |
| LABEL_VERSION = "v3" | |
| WHEEL_RULE = (f"causal expanding median of each wrist coordinate over MANUAL (grid10 engaged=False, aligned by round(t*10)) MOVING (pose `moving`) " | |
| f"frames <= t; device pool of earlier routes (route_key order) + current route; POOL_CAP={BK.POOL_CAP} most recent, N_MIN={BK.N_MIN}; " | |
| f"hands_on = hysteresis on min wrist distance (ON < {HYST_ON}, OFF > {HYST_OFF}, reset after > {GAP_RESET} missing frames); " | |
| f"hand events = transitions with >= {PERSIST_S} s persistence before and after, stamped at the first frame of the new state") | |
| # ------------------------------------------------------------------------------------------------------------------ v3 primitives | |
| def engaged_flags_for_pose(uid, t_pose, grid10_dir=GRID10): | |
| """grid10 -> pose alignment. Returns (engaged[N] bool, matched[N] bool): matched False (and engaged False) where no grid10 row shares | |
| the integer key round(t*10). Frames that are not matched are never counted as manual.""" | |
| p = Path(grid10_dir) / f"{uid}.npz"; n = len(t_pose) | |
| if not p.exists(): return np.zeros(n, bool), np.zeros(n, bool) | |
| g = np.load(p); gk = np.round(g["t"].astype(np.float64) * 10).astype(np.int64); eng = g["engaged"].astype(bool) | |
| order = np.argsort(gk, kind="stable"); gk, eng = gk[order], eng[order] | |
| kt = np.round(np.asarray(t_pose, np.float64) * 10).astype(np.int64) | |
| pos = np.clip(np.searchsorted(gk, kt), 0, len(gk) - 1); matched = gk[pos] == kt | |
| return eng[pos] & matched, matched | |
| def wheel_reference(pools, q, manual_moving): | |
| """pools: list of 4 1-D arrays (device's earlier routes, manual moving values); q[N,>=4] raw wrist quantities (Lx,Ly,Rx,Ry); | |
| manual_moving[N] bool -> wheel_ref[N,4] (causal, expanding over manual moving frames <= t; NaN while n < N_MIN).""" | |
| return np.stack([BK.expanding_median_moving(pools[j], q[:, j], manual_moving) for j in range(4)], 1).astype(np.float32) | |
| def wheel_distance(q, wref): | |
| """d[N] = min over wrists of hypot(wrist - wheel_ref); a wrist with NaN position or NaN reference is ignored (np.fmin); NaN if both.""" | |
| dL = np.hypot(q[:, 0] - wref[:, 0], q[:, 1] - wref[:, 1]); dR = np.hypot(q[:, 2] - wref[:, 2], q[:, 3] - wref[:, 3]) | |
| return np.fmin(dL, dR).astype(np.float32) | |
| def hyst_hands(d, on=HYST_ON, off=HYST_OFF, gap_reset=GAP_RESET): | |
| """Causal hysteresis on the wrist–wheel distance. Returns (s_raw, s_held), both float32 in {0, 1, NaN}: | |
| s_raw[k] = state at frame k (NaN where d[k] is not finite, or undecided after a reset while d stays in the dead band); | |
| s_held[k] = s_raw[k] where finite, else the last state carried across a gap of <= gap_reset missing frames (NaN beyond). | |
| ON if d < on, OFF if d > off, else keep; more than gap_reset consecutive missing frames reset the state (undecided until decisive).""" | |
| n = len(d); s = np.full(n, np.nan, np.float32); held = np.full(n, np.nan, np.float32); cur = np.nan; gap = gap_reset + 1 | |
| for i in range(n): | |
| v = d[i] | |
| if not (v == v): # NaN | |
| gap += 1 | |
| if gap <= gap_reset: held[i] = cur | |
| continue | |
| if gap > gap_reset: cur = np.nan | |
| if v < on: cur = 1.0 | |
| elif v > off: cur = 0.0 | |
| gap = 0; s[i] = cur; held[i] = cur | |
| return s, held | |
| def _runs(code): | |
| """code[N] int (-1 = NaN). Returns (run_start[N], run_end_excl[N]) of the maximal run containing each index.""" | |
| n = len(code); chg = np.flatnonzero(code[1:] != code[:-1]) + 1 | |
| starts = np.r_[0, chg]; ends = np.r_[chg, n]; rid = np.repeat(np.arange(len(starts)), ends - starts) | |
| return starts[rid], ends[rid] | |
| def confirmed_transitions(s_raw, s_held, p_frames=P_FRAMES): | |
| """Persistence-confirmed transitions of the 10 Hz state. Returns (ev_return u8[N], ev_leave u8[N], unknown bool[N]): | |
| frame k is a transition iff s_raw[k] is finite and s_held[k-1] is finite and differs (k = first frame of the new state); | |
| it is CONFIRMED iff the held run of the previous state ending at k-1 has >= p_frames frames and the held run of the new state | |
| starting at k has >= p_frames frames. unknown[k] marks transitions that cannot be confirmed only because the series starts | |
| < p_frames before k or ends < p_frames after k (EM is cleared there).""" | |
| n = len(s_raw); ev_r = np.zeros(n, np.uint8); ev_l = np.zeros(n, np.uint8); unk = np.zeros(n, bool) | |
| if n < 2: return ev_r, ev_l, unk | |
| code = np.where(np.isfinite(s_held), np.nan_to_num(s_held, nan=-1.0), -1.0).astype(np.int64) | |
| rs, re = _runs(code) | |
| k = np.arange(1, n); prev = s_held[:-1]; cur = s_raw[1:] | |
| tr = np.isfinite(cur) & np.isfinite(prev) & (cur != prev); kk = k[tr] | |
| if not len(kk): return ev_r, ev_l, unk | |
| run_before = kk - rs[kk - 1]; run_after = re[kk] - kk | |
| ok = (run_before >= p_frames) & (run_after >= p_frames) | |
| truncated = ((run_before < p_frames) & (rs[kk - 1] == 0)) | ((run_after < p_frames) & (re[kk] == n)) | |
| up = cur[tr] > 0.5 | |
| ev_r[kk[ok & up]] = 1; ev_l[kk[ok & ~up]] = 1; unk[kk[~ok & truncated]] = True | |
| return ev_r, ev_l, unk | |
| def hand_events_on_grid(S0, S, ev10_at, unk10_at, Mc): | |
| """S0[N], S[N,H] causal states on the 0.2 s grid (NaN = unknown/censored); ev10_at[N,H,2] = 10 Hz events at the grid frame and the | |
| frame before it (any -> credited to step h); unk10_at[N,H] likewise. Returns (E u8[N,H], EM bool[N,H]) with E <= EM <= Mc.""" | |
| _, em = BK.onsets(S0, S) # finite(S[h-1]) & finite(S[h]) (v2 rule; S already NaN where ~Mc) | |
| EM = em & ~unk10_at & Mc | |
| E = (ev10_at.any(-1) & EM).astype(np.uint8) | |
| return E, EM | |
| # ------------------------------------------------------------------------------------------------------------------ route series | |
| def build_route_series(have, pose_path, head_on, head_off, grid10_dir=GRID10, log=print): | |
| """Per-route 10 Hz series on the dense frame grid k = round(t*10) - k0 (as v2 main_routes), for all routes of the s12 union: | |
| G (channels, v2 reference), V, F, ht (v2 head hysteresis), hon3 raw/held (v3), ev_r/ev_l/unk (v3), wdef (wheel ref defined), d.""" | |
| by_dev = defaultdict(list) | |
| for u in have: by_dev[u.split("__")[1]].append(u) | |
| series, refs, wrefs, nman = {}, {}, {}, {} | |
| agg = dict(n_frames=0, n_grid10_unmatched=0, n_manual_moving=0, n_engaged_moving=0, n_ref_undef_head=0, n_wheel_undef=0, n_d_undef=0, | |
| routes_no_pool_v2=0, routes_no_manual_pool=0, routes_wheel_never_defined=[], routes_without_grid10=[], | |
| n_trans_raw=0, n_conf_return=0, n_conf_leave=0, n_unknown_trans=0) | |
| t_start = t_last = time.time() | |
| for di, (dev, dus) in enumerate(sorted(by_dev.items())): | |
| dus.sort(key=BK.route_key) | |
| pools = [np.zeros(0, np.float32) for _ in range(5)] # v2 channel reference pools (all moving frames) | |
| wpools = [np.zeros(0, np.float32) for _ in range(4)] # v3 wheel pools (manual moving frames) | |
| for u in dus: | |
| z = np.load(pose_path(u)); t = z["t"].astype(np.float64) | |
| F = np.isfinite(z["box"][:, 0]) & np.isfinite(z["kpts"][:, 0, 2].astype(np.float32)); moving = z["moving"].astype(bool) | |
| raw = BK.raw_from_kpts(z["kpts"], F); q = raw["q"] | |
| eng, matched = engaged_flags_for_pose(u, t, grid10_dir) | |
| if not (Path(grid10_dir) / f"{u}.npz").exists(): agg["routes_without_grid10"].append(u) | |
| manual_moving = moving & matched & ~eng | |
| agg["n_frames"] += len(t); agg["n_grid10_unmatched"] += int((~matched).sum()) | |
| agg["n_manual_moving"] += int(manual_moving.sum()); agg["n_engaged_moving"] += int((moving & matched & eng).sum()) | |
| if all(len(p_) == 0 for p_ in pools): agg["routes_no_pool_v2"] += 1 | |
| if all(len(p_) == 0 for p_ in wpools): agg["routes_no_manual_pool"] += 1 | |
| # --- v2 channel reference (unchanged) -> channels, validity, head state | |
| ref = np.stack([BK.expanding_median_moving(pools[j], q[:, j], moving) for j in range(5)], 1) | |
| agg["n_ref_undef_head"] += int(np.isnan(ref[:, 4]).sum()) | |
| G5 = BK.clip_series(raw, ref); V = raw["V"] & np.isfinite(G5[:, [0, 6, 7, 10]]).all(1) | |
| G = G5.astype(np.float16).astype(np.float32) | |
| # --- v3 wheel reference (manual moving frames only) -> distance -> hysteresis -> confirmed transitions | |
| wref = wheel_reference(wpools, q, manual_moving); wdef = np.isfinite(wref).all(1) | |
| d = wheel_distance(q, wref); d[~F] = np.nan | |
| agg["n_wheel_undef"] += int((~wdef).sum()); agg["n_d_undef"] += int(np.isnan(d).sum()) | |
| if not wdef.any(): agg["routes_wheel_never_defined"].append(u) | |
| k = np.round(t * 10).astype(np.int64); k0, k1 = int(k.min()), int(k.max()); M = k1 - k0 + 1 | |
| GG = np.full((M, BK.D), np.nan, np.float32); VV = np.zeros(M, bool); FF = np.zeros(M, bool); DD = np.full(M, np.nan, np.float32); WD = np.zeros(M, bool) | |
| GG[k - k0] = G; VV[k - k0] = V; FF[k - k0] = F; DD[k - k0] = d; WD[k - k0] = wdef | |
| ht = BK.hysteresis(np.abs(GG[:, 7]), head_on, head_off) # unchanged v2 head state | |
| s_raw, s_held = hyst_hands(DD) | |
| ev_r, ev_l, unk = confirmed_transitions(s_raw, s_held) | |
| fin = np.isfinite(s_raw[1:]) & np.isfinite(s_held[:-1]); agg["n_trans_raw"] += int((fin & (s_raw[1:] != s_held[:-1])).sum()) | |
| agg["n_conf_return"] += int(ev_r.sum()); agg["n_conf_leave"] += int(ev_l.sum()); agg["n_unknown_trans"] += int(unk.sum()) | |
| series[u] = dict(k0=k0, G=GG, V=VV, F=FF, ht=ht, hon=s_raw, ev_r=ev_r, ev_l=ev_l, unk=unk, wdef=WD, d=DD) | |
| refs[u] = ref.astype(np.float32); wrefs[u] = wref; nman[u] = int(manual_moving.sum()) | |
| for j in range(5): | |
| v = q[:, j][moving & np.isfinite(q[:, j])]; pools[j] = np.concatenate([pools[j], v.astype(np.float32)])[-BK.POOL_CAP:] | |
| for j in range(4): | |
| v = q[:, j][manual_moving & np.isfinite(q[:, j])]; wpools[j] = np.concatenate([wpools[j], v.astype(np.float32)])[-BK.POOL_CAP:] | |
| if time.time() - t_last > 60: t_last = time.time(); log(f" devices {di+1}/{len(by_dev)} routes {len(series)}/{len(have)} {time.time()-t_start:.0f}s") | |
| return series, refs, wrefs, nman, agg | |
| def _link_or_copy(src, dst): | |
| """Hard-link dst -> src (byte-identical by construction); copy if linking is impossible. Existing dst must be identical.""" | |
| src, dst = Path(src), Path(dst) | |
| if dst.exists(): | |
| same = os.path.samefile(src, dst) or (src.stat().st_size == dst.stat().st_size and src.read_bytes() == dst.read_bytes()) | |
| assert same, f"{dst} exists and differs from {src}" | |
| return "kept" | |
| try: os.link(src, dst); return "hardlink" | |
| except OSError: shutil.copyfile(src, dst); return "copy" | |
| def _event_stats(E, EM, E0, S0, S): | |
| """Per-event hazard / base-rate block of v2's build_stats (same definitions).""" | |
| out = {} | |
| chains = [(S0[:, 0], S[:, :, 0]), (1 - S0[:, 0], 1 - S[:, :, 0]), (S0[:, 1], S[:, :, 1])] | |
| for j, name in enumerate(BK.EVENTS): | |
| z0, z = chains[j]; zp = np.concatenate([z0[:, None], z[:, :-1]], 1); m = EM[:, :, j]; m_tr = m & (zp < 0.5); inact0 = E0[:, j] < 0.5 | |
| rec = dict(n_inactive_at_anchor=int(inact0.sum()), hazard_per_step=float(E[:, :, j][m_tr].mean()) if m_tr.any() else None) | |
| for tau, kk in (("1s", 5), ("3s", 15), ("5s", 25)): | |
| sel = inact0 & m[:, :kk].all(1); y = E[sel, :kk, j].any(1); rec[f"n@{tau}"] = int(sel.sum()); rec[f"pos@{tau}"] = int(y.sum()); rec[f"base_rate@{tau}"] = float(y.mean()) if sel.any() else None | |
| out[name] = rec | |
| return out | |
| # ------------------------------------------------------------------------------------------------------------------ main | |
| def main(argv=None): | |
| ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) | |
| ap.add_argument("--regimes", default="manual,engaged"); ap.add_argument("--splits", default="train,val,test") | |
| ap.add_argument("--out-dir", default=str(OUT_V3)); ap.add_argument("--pose-dir", default=str(POSE_PREF), help="preferred pose dir; routes missing there fall back to results/pose_causal_routes (v2 rule)") | |
| ap.add_argument("--v2-dir", default=str(V2_DIR), help="v2 targets: input-equality check + hard-linked kinhist files") | |
| ap.add_argument("--grid10-dir", default=str(GRID10)) | |
| a = ap.parse_args(argv); t_start = time.time() | |
| out = Path(a.out_dir); out.mkdir(parents=True, exist_ok=True); (out / "logs").mkdir(exist_ok=True) | |
| v2 = Path(a.v2_dir); P2 = Path(a.pose_dir) if a.pose_dir else None | |
| regimes, splits = a.regimes.split(","), a.splits.split(",") | |
| def log(msg): print(msg, flush=True) | |
| old = json.load(open(BK.OLD_STATS)); head_on, head_off = float(old["head_turn_on"]), float(old["head_turn_off"]) | |
| uids = sorted(set().union(*[set(np.unique(np.load(BK.S12R / f"targets_{r}_{s}.npz", allow_pickle=True)["uid"]).tolist()) for r in ("manual", "engaged") for s in BK.S12_SPLITS])) | |
| def pose_path(u): | |
| if P2 is not None and (P2 / f"{u}.npz").exists(): return P2 / f"{u}.npz" | |
| return BK.POSE_R / f"{u}.npz" | |
| have = [u for u in uids if pose_path(u).exists()]; n_from_pref = sum(1 for u in have if P2 is not None and (P2 / f"{u}.npz").exists()) | |
| log(f"v3 build: s12 routes {len(uids)}, with causal route pose {len(have)} ({n_from_pref} from {P2}); head thresholds on={head_on:.4f} off={head_off:.4f}; " | |
| f"hyst on/off {HYST_ON}/{HYST_OFF}, persistence {PERSIST_S} s, gap reset {GAP_RESET} frames") | |
| series, refs, wrefs, nman, agg = build_route_series(have, pose_path, head_on, head_off, Path(a.grid10_dir), log) | |
| np.savez_compressed(out / "route_refs.npz", uids=np.array(list(refs)), ref_quantities=np.array(BK.REFQ), wheel_ref_quantities=np.array(BK.REFQ[:4]), | |
| **{f"ref__{u}": r for u, r in refs.items()}, **{f"wheel_ref__{u}": r for u, r in wrefs.items()}, **{f"n_manual_moving__{u}": np.int64(nman[u]) for u in nman}, | |
| rule=np.array("ref__: " + f"expanding median over MOVING frames <= t, device's earlier routes pool, POOL_CAP={BK.POOL_CAP}, N_MIN={BK.N_MIN} (v2, unchanged); wheel_ref__: " + WHEEL_RULE), | |
| units=np.array(BK.UNITS), label_version=np.array(LABEL_VERSION)) | |
| # v3 reference identical to v2's stored reference? (channels unchanged) — checked on the v2 route_refs when present | |
| n_ref_mismatch = None | |
| if (v2 / "route_refs.npz").exists(): | |
| r2 = np.load(v2 / "route_refs.npz", allow_pickle=True); n_ref_mismatch = sum(1 for u in refs if f"ref__{u}" in r2.files and not np.array_equal(r2[f"ref__{u}"], refs[u], equal_nan=True)) | |
| assert n_ref_mismatch == 0, f"channel reference differs from v2 route_refs for {n_ref_mismatch} routes" | |
| nf = max(agg["n_frames"], 1) | |
| log(f"series for {len(series)} routes in {time.time()-t_start:.0f}s; frames {agg['n_frames']}, grid10-unmatched {agg['n_grid10_unmatched']}, manual moving {agg['n_manual_moving']}, engaged moving {agg['n_engaged_moving']}; " | |
| f"head-ref undefined {agg['n_ref_undef_head']} ({100*agg['n_ref_undef_head']/nf:.2f}%), wheel-ref undefined {agg['n_wheel_undef']} ({100*agg['n_wheel_undef']/nf:.2f}%), d undefined {agg['n_d_undef']} ({100*agg['n_d_undef']/nf:.2f}%); " | |
| f"routes without prior v2 pool {agg['routes_no_pool_v2']}, without prior manual pool {agg['routes_no_manual_pool']}, wheel ref never defined {len(agg['routes_wheel_never_defined'])}; " | |
| f"10 Hz raw transitions {agg['n_trans_raw']} -> confirmed return {agg['n_conf_return']} leave {agg['n_conf_leave']} (unconfirmable at route edges {agg['n_unknown_trans']}); v2 channel-ref mismatches {n_ref_mismatch}") | |
| offs_f = np.arange(1, BK.H + 1) * 2 | |
| stats = dict(source="routes", label_version=LABEL_VERSION, pose_dir=str(P2) if P2 else str(BK.POSE_R), pose_fallback_dir=str(BK.POSE_R), n_routes_from_pose_dir=n_from_pref, grid10_dir=str(a.grid10_dir), v2_dir=str(v2), | |
| head_turn_on=head_on, head_turn_off=head_off, hyst_on=HYST_ON, hyst_off=HYST_OFF, persist_s=PERSIST_S, gap_reset_frames=GAP_RESET, n_routes_s12=len(uids), n_routes_with_pose=len(have), | |
| ref_undefined_frame_frac=agg["n_ref_undef_head"] / nf, wheel_ref_undefined_frame_frac=agg["n_wheel_undef"] / nf, d_undefined_frame_frac=agg["n_d_undef"] / nf, | |
| n_pose_frames=agg["n_frames"], n_grid10_unmatched_frames=agg["n_grid10_unmatched"], n_manual_moving_frames=agg["n_manual_moving"], n_engaged_moving_frames=agg["n_engaged_moving"], | |
| routes_without_prior_pool=agg["routes_no_pool_v2"], routes_without_prior_manual_pool=agg["routes_no_manual_pool"], routes_wheel_ref_never_defined=agg["routes_wheel_never_defined"], routes_without_grid10=agg["routes_without_grid10"], | |
| n_10hz_raw_transitions=agg["n_trans_raw"], n_10hz_confirmed_return=agg["n_conf_return"], n_10hz_confirmed_leave=agg["n_conf_leave"], n_10hz_unconfirmable_transitions=agg["n_unknown_trans"], | |
| v2_channel_ref_mismatch_routes=n_ref_mismatch, pool_cap=BK.POOL_CAP, n_min=BK.N_MIN, units=BK.UNITS, wheel_ref_rule=WHEEL_RULE, splits={}) | |
| for reg in regimes: | |
| for sp in splits: | |
| t_sp = time.time() | |
| d = np.load(BK.S12R / f"targets_{reg}_{sp}.npz", allow_pickle=True); uid, don, t, Mc = d["uid"], d["dongle"], d["t"], d["M"].astype(bool); N = len(t) | |
| K = np.full((N, BK.H, BK.D), np.nan, np.float16); Kvalid = np.zeros((N, BK.H), bool); Kframe = np.zeros((N, BK.H), bool) | |
| K0 = np.full((N, BK.D), np.nan, np.float32); K0valid = np.zeros(N, bool); S = np.full((N, BK.H, 2), np.nan, np.float32); S0 = np.full((N, 2), np.nan, np.float32) | |
| EV = np.zeros((N, BK.H, 2, 2), bool); UNK = np.zeros((N, BK.H), bool); WDEF = np.zeros(N, bool); D0 = np.full(N, np.nan, np.float32) | |
| route_dec = np.zeros(N, bool); row_ok = np.zeros(N, bool) | |
| order = np.argsort(uid, kind="stable"); bounds = np.flatnonzero(np.r_[1, uid[order][1:] != uid[order][:-1], 1]) | |
| for a_, b_ in zip(bounds[:-1], bounds[1:]): | |
| rows = order[a_:b_]; u = uid[rows[0]] | |
| if u not in series: continue | |
| rs = series[u]; route_dec[rows] = True; ka = np.round(t[rows] * 10).astype(np.int64) - rs["k0"]; M = len(rs["G"]) | |
| def gather(A, offs): | |
| idx = ka[:, None] + offs[None, :]; ok = (idx >= 0) & (idx < M); o = A[np.clip(idx, 0, M - 1)] | |
| return np.where(ok[..., None] if o.ndim == 3 else ok, o, np.nan if A.dtype.kind == "f" else False) | |
| K[rows] = gather(rs["G"], offs_f).astype(np.float16); Kvalid[rows] = gather(rs["V"], offs_f); Kframe[rows] = gather(rs["F"], offs_f) | |
| ok0 = (ka >= 0) & (ka < M); kc = np.clip(ka, 0, M - 1) | |
| K0[rows] = np.where(ok0[:, None], rs["G"][kc], np.nan); K0valid[rows] = ok0 & rs["V"][kc]; row_ok[rows] = ok0 & rs["F"][kc] | |
| S[rows, :, 0] = gather(rs["hon"], offs_f); S[rows, :, 1] = gather(rs["ht"], offs_f); S0[rows, 0] = np.where(ok0, rs["hon"][kc], np.nan); S0[rows, 1] = np.where(ok0, rs["ht"][kc], np.nan) | |
| WDEF[rows] = ok0 & rs["wdef"][kc]; D0[rows] = np.where(ok0, rs["d"][kc], np.nan) | |
| for c, key in enumerate(("ev_r", "ev_l")): | |
| EV[rows, :, c, 0] = gather(rs[key].astype(bool), offs_f); EV[rows, :, c, 1] = gather(rs[key].astype(bool), offs_f - 1) | |
| UNK[rows] = gather(rs["unk"], offs_f) | gather(rs["unk"], offs_f - 1) | |
| n_kvalid_raw = int(Kvalid.sum()); Kvalid &= Mc; S[~Mc] = np.nan | |
| E = np.zeros((N, BK.H, 3), np.uint8); EM = np.zeros((N, BK.H, 3), bool); E0 = np.full((N, 3), np.nan, np.float32) | |
| E[:, :, 0], EM[:, :, 0] = hand_events_on_grid(S0[:, 0], S[:, :, 0], EV[:, :, 0], UNK, Mc) | |
| E[:, :, 1], EM[:, :, 1] = hand_events_on_grid(1 - S0[:, 0], 1 - S[:, :, 0], EV[:, :, 1], UNK, Mc) | |
| E[:, :, 2], EM[:, :, 2] = BK.onsets(S0[:, 1], S[:, :, 1]) # head_turn_onset unchanged | |
| E0[:, 0], E0[:, 1], E0[:, 2] = S0[:, 0], 1 - S0[:, 0], S0[:, 1] | |
| assert np.array_equal(uid, d["uid"]) and np.allclose(t, d["t"]) and not (Kvalid & ~Mc).any() and not (EM & ~Mc[:, :, None]).any() | |
| assert not (E.astype(bool) & ~EM).any(), "E_c must be a subset of EM_c" | |
| # ---- inputs identical to v2 (whole split), head channel unchanged | |
| chk = {} | |
| v2k = v2 / f"kin_causal_{reg}_{sp}.npz" | |
| if v2k.exists(): | |
| z2 = np.load(v2k, allow_pickle=True) | |
| for key, arr in (("K_c", K), ("K0_c", K0), ("Kvalid", Kvalid), ("Kframe", Kframe), ("K0valid", K0valid), ("row_ok", row_ok), ("route_decoded", route_dec)): | |
| chk[key] = bool(np.array_equal(z2[key], arr, equal_nan=True) if arr.dtype.kind == "f" else np.array_equal(z2[key], arr)) | |
| chk["S_head"] = bool(np.array_equal(z2["S_c"][:, :, 1], S[:, :, 1], equal_nan=True) and np.array_equal(z2["S0_c"][:, 1], S0[:, 1], equal_nan=True)) | |
| chk["E_head"] = bool(np.array_equal(z2["E_c"][:, :, 2], E[:, :, 2]) and np.array_equal(z2["EM_c"][:, :, 2], EM[:, :, 2])) | |
| bad = [k_ for k_, ok_ in chk.items() if not ok_] | |
| assert not bad, f"{reg}/{sp}: v3 differs from v2 on {bad} (inputs / head channel must be identical)" | |
| hon2 = z2["S_c"][:, :, 0]; hon2_0 = z2["S0_c"][:, 0]; E2 = z2["E_c"]; EM2 = z2["EM_c"] | |
| else: | |
| hon2 = hon2_0 = E2 = EM2 = None | |
| np.savez_compressed(out / f"kin_causal_{reg}_{sp}.npz", uid=uid, dongle=don, t=t, row_ok=row_ok, route_decoded=route_dec, K_c=K, Kvalid=Kvalid, Kframe=Kframe, K0_c=K0, K0valid=K0valid, | |
| S_c=S, S0_c=S0, E_c=E, EM_c=EM, E0_c=E0, channels=np.array(BK.CHANNELS), event_names=np.array(BK.EVENTS), state_names=np.array(BK.STATES), units=np.array(BK.UNITS), source=np.array("routes"), | |
| wheel_ref_defined=WDEF, dwheel0_c=D0, persist_s=np.float32(PERSIST_S), hyst_on=np.float32(HYST_ON), hyst_off=np.float32(HYST_OFF), gap_reset_frames=np.int64(GAP_RESET), | |
| label_version=np.array(LABEL_VERSION), wheel_ref_rule=np.array(WHEEL_RULE)) | |
| v2h = v2 / f"kinhist_causal_{reg}_{sp}.npz"; how = _link_or_copy(v2h, out / f"kinhist_causal_{reg}_{sp}.npz") if v2h.exists() else "missing_v2" | |
| st = dict(N=int(N), routes=int(len(np.unique(uid))), routes_decoded=int(len({u for u in np.unique(uid) if u in series})), n_route_decoded_rows=int(route_dec.sum()), coverage_route_decoded=float(route_dec.mean()), | |
| n_row_ok=int(row_ok.sum()), coverage_row_ok=float(row_ok.mean()), coverage_row_ok_given_decoded=float(row_ok[route_dec].mean()) if route_dec.any() else None, | |
| Kvalid_frac_row_ok=float(Kvalid[row_ok].mean()) if row_ok.any() else None, Kvalid_all25_frac_row_ok=float(Kvalid[row_ok].all(1).mean()) if row_ok.any() else None, Kvalid_dropped_by_s12_M=int(n_kvalid_raw - Kvalid.sum()), | |
| K0valid_frac_row_ok=float(K0valid[row_ok].mean()) if row_ok.any() else None, | |
| wheel_ref_defined_frac_row_ok=float(WDEF[row_ok].mean()) if row_ok.any() else None, S0_hands_finite_frac_row_ok=float(np.isfinite(S0[row_ok, 0]).mean()) if row_ok.any() else None, | |
| S0_hands_finite_frac_row_ok_v2=float(np.isfinite(hon2_0[row_ok]).mean()) if hon2_0 is not None and row_ok.any() else None, | |
| coverage_loss_S0_v2def_v3nan_frac_row_ok=float((np.isfinite(hon2_0[row_ok]) & ~np.isfinite(S0[row_ok, 0])).mean()) if hon2_0 is not None and row_ok.any() else None, | |
| hands_on_prev=float(np.nanmean(S[:, :, 0])) if np.isfinite(S[:, :, 0]).any() else None, hands_on_prev_v2=float(np.nanmean(hon2)) if hon2 is not None and np.isfinite(hon2).any() else None, | |
| hands_on_at_anchor=float(np.nanmean(S0[:, 0])) if np.isfinite(S0[:, 0]).any() else None, head_turned_prev=float(np.nanmean(S[:, :, 1])) if np.isfinite(S[:, :, 1]).any() else None, | |
| n_E_hand_return=int(E[:, :, 0].sum()), n_E_hand_leave=int(E[:, :, 1].sum()), n_E_head_turn=int(E[:, :, 2].sum()), | |
| n_E_hand_return_v2=int(E2[:, :, 0].sum()) if E2 is not None else None, n_E_hand_leave_v2=int(E2[:, :, 1].sum()) if E2 is not None else None, | |
| EM_hand_frac=float(EM[:, :, 0].mean()), EM_hand_frac_v2=float(EM2[:, :, 0].mean()) if EM2 is not None else None, n_EM_cleared_unconfirmable=int((UNK & Mc & BK.onsets(S0[:, 0], S[:, :, 0])[1]).sum()), | |
| channel_nan_frac_row_ok={c: float(np.isnan(K[row_ok][:, :, i].astype(np.float32)).mean()) for i, c in enumerate(BK.CHANNELS)} if row_ok.any() else None, | |
| v2_identity_checks=chk, kinhist_file=how, events=_event_stats(E, EM, E0, S0, S), wall_s=round(time.time() - t_sp, 1)) | |
| stats["splits"][f"{reg}_{sp}"] = st | |
| log(f"{reg}/{sp}: N={N} row_ok={row_ok.sum()} ({row_ok.mean()*100:.2f}%) wheel_ref_defined|row_ok={st['wheel_ref_defined_frac_row_ok']:.4f} S0 finite v3/v2={st['S0_hands_finite_frac_row_ok']:.4f}/{st['S0_hands_finite_frac_row_ok_v2']} " | |
| f"hands_on v3/v2={st['hands_on_prev']:.4f}/{st['hands_on_prev_v2']} E return v3/v2={st['n_E_hand_return']}/{st['n_E_hand_return_v2']} leave v3/v2={st['n_E_hand_leave']}/{st['n_E_hand_leave_v2']} head={st['n_E_head_turn']} " | |
| f"kinhist={how} v2-identity={'OK' if chk and all(chk.values()) else chk} " + " ".join(f"{e}@3s={st['events'][e]['pos@3s']}/{st['events'][e]['n@3s']}" for e in BK.EVENTS) + f" ({st['wall_s']}s)") | |
| stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) | |
| (out / f"build_stats_{stamp}.json").write_text(json.dumps(stats, indent=1)) | |
| latest = out / "build_stats_latest.json"; merged = dict(stats) | |
| if latest.exists(): # merge split blocks over invocations (test -> val -> train) | |
| try: prev = json.load(open(latest)); merged["splits"] = {**prev.get("splits", {}), **stats["splits"]}; merged["merged_from_previous_latest"] = True | |
| except Exception: pass | |
| latest.write_text(json.dumps(merged, indent=1)) | |
| inputs = [BK.S12R / f"targets_{r}_{s}.npz" for r in regimes for s in splits] + [BK.OLD_STATS] + [p for p in [v2 / "route_refs.npz"] if p.exists()] \ | |
| + [p for r in regimes for s in splits for p in (v2 / f"kin_causal_{r}_{s}.npz", v2 / f"kinhist_causal_{r}_{s}.npz") if p.exists()] + [Path(a.grid10_dir) / f"{u}.npz" for u in have if (Path(a.grid10_dir) / f"{u}.npz").exists()] | |
| write_run_json(out, "build_kin_causal_routes_v3", [Path(__file__), Path(BK.__file__)], inputs, None, t_start, | |
| extra=dict(summary={k: v for k, v in stats.items() if k != "splits"}, coverage={k: dict(N=v["N"], route_decoded=v["coverage_route_decoded"], row_ok=v["coverage_row_ok"], wheel_ref_defined=v["wheel_ref_defined_frac_row_ok"], routes=v["routes"]) for k, v in stats["splits"].items()}, | |
| pose_dir=str(P2) if P2 else str(BK.POSE_R), stats_file=f"build_stats_{stamp}.json", regimes=regimes, splits=splits), input_listing=True) | |
| log(f"KIN_CAUSAL_ROUTES_V3_DONE wall {time.time()-t_start:.0f}s") | |
| if __name__ == "__main__": | |
| main() | |
Xet Storage Details
- Size:
- 32.9 kB
- Xet hash:
- a1f3e247db2434f7041082ec47202410a37965bd97e438d23b587a11a1f3aa56
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.