Buckets:
| """cwm3/data/build_kin_causal_v4.py — hands_on / hand-event label v4 (pre-registered docs/decisions.md 2026-09-16T22:18:02Z, last row). | |
| Imports data/build_kin_causal.py (v2, FROZEN) and data/build_kin_causal_v3.py (v3, FROZEN) as modules and reuses their machinery unchanged: | |
| pose loading (BK.raw_from_kpts), channel series (BK.clip_series), the v2 expanding median for the (unchanged) channel reference, the head-turn | |
| hysteresis (BK.hysteresis / BK.onsets), route ordering (BK.route_key), the grid10 regime alignment (V3.engaged_flags_for_pose), the hands | |
| hysteresis with gap reset (V3.hyst_hands), the persistence-confirmed transitions (V3.confirmed_transitions), the grid event helper | |
| (V3.hand_events_on_grid), the kinhist hard link (V3._link_or_copy) and the event-stat block (V3._event_stats). Geometry: data/wheel_ellipse.py. | |
| WHAT CHANGES vs v3 (targets only; the v3 POINT reference becomes an ELLIPSE — the wheel is a ring, not a point): | |
| wheel ellipse = per device, causal robust ellipse fit of the rim locus from BOTH wrists' positions over MANUAL (grid10 engaged=False, | |
| aligned by round(t*10) as v3) MOVING (pose `moving`) frames <= t. Point pool = the most recent POOL_CAP = 4000 manual | |
| moving points PER WRIST over the device's earlier routes (route_key order) + the current route up to t (same device pool | |
| as the v3 wheel reference; the cap now also applies to the current route so that the fit tracks the last ~400 s of manual | |
| driving and every refit is bounded to <= 8000 points). Fit = wheel_ellipse.ransac_ellipse (RANSAC + Fitzgibbon direct | |
| LS ellipse fit on the inliers, MSAC scoring, warm start from the previous fit), REFIT causally every REFIT_FRAMES = 50 | |
| new manual moving frames (5 s) of the current route, plus at the first frame of a route when the earlier-route pool | |
| already holds >= 200 points; between refits the last fit is held (piecewise constant). A fit is DEFINED only if it | |
| passes the pre-registered gate (>= N_MIN_PTS = 200 inliers spanning >= COV_MIN_DEG = 90 deg of arc on the fitted | |
| ellipse), otherwise the ellipse is NaN until the next passing refit. RNG seeded per device (SEED, crc32(device)). | |
| d(t) = min over wrists (np.fmin: a NaN wrist is ignored; NaN if both) of the EXACT SIGNED distance of the wrist to the filled | |
| wheel region (wheel_ellipse.signed_distance: negative inside the ellipse, distance to the rim outside), in the v2/v3 | |
| units (x_norm*1.6, y_norm); NaN where the ellipse is undefined or the frame has no pose. | |
| hands_on (S) = V3.hyst_hands on d with ON if d < HYST_ON = 0.04, OFF if d > HYST_OFF = 0.08 (otherwise keep), reset after a gap of more | |
| than GAP_RESET = 10 consecutive frames without d (> 1 s); undecided (NaN) after a reset until the first decisive frame. | |
| hand_leave / = V3.confirmed_transitions: transitions with >= PERSIST_S = 1.0 s persistence before and after (stamp = first frame of the | |
| hand_return new state; forward confirmation for TARGETS only; EM_c cleared where the transition cannot be confirmed at route edges). | |
| S0_c = the raw causal state at the anchor frame (no lookahead). | |
| head_turned / head_turn_onset : UNCHANGED (asserted equal to the v2 files). | |
| INPUT channels : UNCHANGED. K_c / K0_c / Kvalid / Kframe / K0valid / row_ok / route_decoded are recomputed with the v2 code path and | |
| asserted bit-identical to results/targets_kin_causal_routes_v2; kinhist_causal_* are HARD-LINKED to the v2 files. | |
| OUTPUT results/targets_kin_causal_routes_v4/ — same file names / keys / dtypes / row alignment as v2/v3 plus, in kin_causal_*.npz: | |
| wheel_ref_defined bool[N] (ellipse defined at the anchor; key name kept from v3 for the page tools), dwheel0_c f32[N] (signed d at the | |
| anchor), ellipse0_c f32[N,5] (cx, cy, a, b, theta at the anchor), persist_s, hyst_on, hyst_off, gap_reset_frames, refit_frames, | |
| label_version="v4", wheel_ref_rule. | |
| route_refs.npz: ref__<uid> [N,5] (channel reference, identical to v2), ell__<uid> [N,5] f32 (causal ellipse per pose frame), d4__<uid> f32[N] | |
| (signed d per pose frame), fits__<uid> [n_refit, 12] (columns FIT_COLS: frame index, n_points, n_inlier, coverage_deg, cov_parametric_deg, | |
| cov_central_deg, passed, cx, cy, a, b, theta), n_manual_moving__<uid>; build_stats_<utc>.json (+ _latest, merged), run_*.json. | |
| """ | |
| import argparse, json, sys, time, zlib | |
| from collections import defaultdict | |
| from pathlib import Path | |
| import numpy as np | |
| ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(ROOT / "data")); sys.path.insert(0, str(ROOT / "common")) | |
| import build_kin_causal as BK # noqa: E402 (v2 machinery, frozen file, imported not copied) | |
| import build_kin_causal_v3 as V3 # noqa: E402 (v3 machinery: alignment, hysteresis, persistence, grid events, link) | |
| import wheel_ellipse as WE # noqa: E402 (ellipse fit + exact signed distance) | |
| from runmeta import write_run_json # noqa: E402 | |
| GRID10 = V3.GRID10 | |
| V2_DIR = V3.V2_DIR | |
| OUT_V4 = ROOT / "results" / "targets_kin_causal_routes_v4" | |
| POSE_PREF = V3.POSE_PREF | |
| HYST_ON, HYST_OFF, PERSIST_S, FPS = 0.04, 0.08, 1.0, 10 | |
| GAP_RESET = BK.GAP_RESET # 10 frames (> 1 s of missing frames resets the state) | |
| P_FRAMES = V3.P_FRAMES # 10 | |
| REFIT_S = 5.0; REFIT_FRAMES = int(round(REFIT_S * FPS)) # refit every 50 new manual moving frames | |
| POOL_CAP = BK.POOL_CAP # 4000 most recent points per wrist | |
| SEED = 0 | |
| LABEL_VERSION = "v4" | |
| FIT_COLS = ["frame", "n_points", "n_inlier", "coverage_deg", "cov_parametric_deg", "cov_central_deg", "passed", "cx", "cy", "a", "b", "theta"] | |
| WHEEL_RULE = (f"per-device causal RANSAC + direct-LS ELLIPSE fit of the rim locus from BOTH wrists over MANUAL (grid10 engaged=False, aligned by round(t*10)) MOVING " | |
| f"(pose `moving`) frames <= t; point pool = most recent POOL_CAP={POOL_CAP} points per wrist over earlier routes (route_key order) + current route; " | |
| f"refit every {REFIT_FRAMES} new manual moving frames ({REFIT_S:g} s) and at route start if the pool has >= {WE.N_MIN_PTS} points; defined iff >= {WE.N_MIN_PTS} inliers " | |
| f"(|sampson| <= {WE.TAU_IN}) spanning >= {WE.COV_MIN_DEG:g} deg of arc on the fitted ellipse (min of parametric / central 10-deg-bin coverage), candidates limited to " | |
| f"semi-axes in [{WE.AX_MIN}, {WE.AX_MAX}] and b/a >= {WE.RATIO_MIN}; d = min over wrists of the exact SIGNED distance to the filled ellipse (negative inside); " | |
| f"hands_on = hysteresis (ON < {HYST_ON}, OFF > {HYST_OFF}, reset after > {GAP_RESET} missing frames); hand events = transitions with >= {PERSIST_S} s persistence " | |
| f"before and after, stamped at the first frame of the new state") | |
| # ------------------------------------------------------------------------------------------------------------------ v4 primitives | |
| def device_rng(dev, seed=SEED): | |
| """Deterministic per-device Generator (independent of the processing order of devices).""" | |
| return np.random.default_rng([seed, zlib.crc32(str(dev).encode())]) | |
| def causal_ellipse_series(poolL, poolR, q, manual_moving, rng, refit_frames=REFIT_FRAMES, prev=None, pool_cap=POOL_CAP): | |
| """Causal piecewise-constant ellipse series for one route. | |
| poolL, poolR [n,2]: manual moving wrist points of the device's EARLIER routes (already capped); q[N,>=4] raw wrist quantities (Lx,Ly,Rx,Ry); | |
| manual_moving[N] bool; rng: per-device Generator (consumed once per refit, in frame order); prev: previous geometry (warm start) or None. | |
| Returns (ell f32[N,5] NaN where undefined, fits f64[n_refit, len(FIT_COLS)], poolL_out, poolR_out, prev_out). | |
| Refit frames: k = 0 when len(poolL) + len(poolR) >= N_MIN_PTS, and every frame at which the running count of NEW manual moving frames with | |
| at least one finite wrist reaches a multiple of refit_frames. The fit at frame k uses points of frames <= k only (causal); ell[k:next) = fit | |
| if it passes the gate, else NaN. Prefix property: the series on frames < K does not depend on frames >= K.""" | |
| N = len(q); ell = np.full((N, 5), np.nan, np.float32) | |
| okL = manual_moving & np.isfinite(q[:, 0]) & np.isfinite(q[:, 1]); okR = manual_moving & np.isfinite(q[:, 2]) & np.isfinite(q[:, 3]) | |
| idxL = np.flatnonzero(okL); idxR = np.flatnonzero(okR); ptsL = q[idxL, :2].astype(np.float64); ptsR = q[idxR, 2:4].astype(np.float64) | |
| new = okL | okR; cnt = np.cumsum(new) | |
| trig = np.flatnonzero(new & (cnt % refit_frames == 0)) | |
| if len(poolL) + len(poolR) >= WE.N_MIN_PTS: trig = np.r_[0, trig[trig > 0]] | |
| fits = [] | |
| for i, k in enumerate(trig): | |
| nL = int(np.searchsorted(idxL, k, side="right")); nR = int(np.searchsorted(idxR, k, side="right")) | |
| L = np.concatenate([poolL, ptsL[:nL]])[-pool_cap:]; R = np.concatenate([poolR, ptsR[:nR]])[-pool_cap:] | |
| r = WE.ransac_ellipse(np.concatenate([L, R]), rng, prev=prev) | |
| g = r["geom"] | |
| if g is not None: prev = np.asarray(g, np.float64) | |
| k_end = int(trig[i + 1]) if i + 1 < len(trig) else N | |
| cov_p, cov_c = (np.nan, np.nan) | |
| if g is not None and r["n_inlier"]: | |
| P = np.concatenate([L, R]); P = P[np.isfinite(P).all(1)] | |
| cov_p, cov_c = WE.arc_coverage_deg(g, P[r["inliers"], 0], P[r["inliers"], 1], both=True) | |
| if r["passed"]: ell[int(k):k_end] = np.asarray(g, np.float32) | |
| fits.append([float(k), float(r["n_points"]), float(r["n_inlier"]), float(r["coverage_deg"]), float(cov_p), float(cov_c), float(r["passed"])] | |
| + ([float(v) for v in g] if g is not None else [np.nan] * 5)) | |
| poolL_out = np.concatenate([poolL, ptsL])[-pool_cap:]; poolR_out = np.concatenate([poolR, ptsR])[-pool_cap:] | |
| return ell, np.asarray(fits, np.float64).reshape(-1, len(FIT_COLS)), poolL_out, poolR_out, prev | |
| def wheel_signed_distance(q, ell): | |
| """d[N] = min over wrists of the exact signed distance to the ellipse ell[N,5] (NaN where the ellipse or the wrist is undefined).""" | |
| n = len(q); d = np.full(n, np.nan, np.float32) | |
| ok = np.isfinite(ell).all(1) | |
| if not ok.any(): return d | |
| g = tuple(ell[ok, j].astype(np.float64) for j in range(5)) | |
| dL = WE.signed_distance(g, q[ok, 0], q[ok, 1]); dR = WE.signed_distance(g, q[ok, 2], q[ok, 3]) | |
| d[ok] = np.fmin(dL, dR).astype(np.float32) | |
| return d | |
| def hyst_hands_v4(d): | |
| return V3.hyst_hands(d, on=HYST_ON, off=HYST_OFF, gap_reset=GAP_RESET) | |
| # ------------------------------------------------------------------------------------------------------------------ route series | |
| def build_route_series(have, pose_path, head_on, head_off, grid10_dir=GRID10, refit_frames=REFIT_FRAMES, log=print): | |
| """Per-route 10 Hz series on the dense frame grid k = round(t*10) - k0 (as v2/v3), for all routes of the s12 union.""" | |
| by_dev = defaultdict(list) | |
| for u in have: by_dev[u.split("__")[1]].append(u) | |
| series, refs, ells, d4s, fitlog, nman = {}, {}, {}, {}, {}, {} | |
| agg = dict(n_frames=0, n_grid10_unmatched=0, n_manual_moving=0, n_engaged_moving=0, n_ref_undef_head=0, n_ell_undef=0, n_d_undef=0, | |
| routes_no_pool_v2=0, routes_no_manual_pool=0, routes_ellipse_never_defined=[], routes_without_grid10=[], | |
| n_refits=0, n_refits_passed=0, n_refits_fail_points=0, n_refits_fail_coverage=0, n_refits_no_candidate=0, | |
| n_trans_raw=0, n_conf_return=0, n_conf_leave=0, n_unknown_trans=0, devices={}) | |
| t_start = t_last = time.time(); t_fit = 0.0 | |
| for di, (dev, dus) in enumerate(sorted(by_dev.items())): | |
| dus.sort(key=BK.route_key) | |
| pools = [np.zeros(0, np.float32) for _ in range(5)] # v2 channel reference pools (all moving frames) | |
| poolL = np.zeros((0, 2)); poolR = np.zeros((0, 2)); prev = None; rng = device_rng(dev) | |
| dev_rec = dict(n_routes=len(dus), n_manual_moving=0, n_refits=0, n_refits_passed=0, n_frames=0, n_ell_defined=0, last_passed=None) | |
| for u in dus: | |
| z = np.load(pose_path(u)); t = z["t"].astype(np.float64) | |
| F = np.isfinite(z["box"][:, 0]) & np.isfinite(z["kpts"][:, 0, 2].astype(np.float32)); moving = z["moving"].astype(bool) | |
| raw = BK.raw_from_kpts(z["kpts"], F); q = raw["q"] | |
| eng, matched = V3.engaged_flags_for_pose(u, t, grid10_dir) | |
| if not (Path(grid10_dir) / f"{u}.npz").exists(): agg["routes_without_grid10"].append(u) | |
| manual_moving = moving & matched & ~eng | |
| agg["n_frames"] += len(t); agg["n_grid10_unmatched"] += int((~matched).sum()) | |
| agg["n_manual_moving"] += int(manual_moving.sum()); agg["n_engaged_moving"] += int((moving & matched & eng).sum()) | |
| if all(len(p_) == 0 for p_ in pools): agg["routes_no_pool_v2"] += 1 | |
| if len(poolL) + len(poolR) == 0: agg["routes_no_manual_pool"] += 1 | |
| # --- v2 channel reference (unchanged) -> channels, validity, head state | |
| ref = np.stack([BK.expanding_median_moving(pools[j], q[:, j], moving) for j in range(5)], 1) | |
| agg["n_ref_undef_head"] += int(np.isnan(ref[:, 4]).sum()) | |
| G5 = BK.clip_series(raw, ref); V = raw["V"] & np.isfinite(G5[:, [0, 6, 7, 10]]).all(1) | |
| G = G5.astype(np.float16).astype(np.float32) | |
| # --- v4 causal ellipse (manual moving frames only) -> signed distance -> hysteresis -> confirmed transitions | |
| t0f = time.time() | |
| ell, fits, poolL, poolR, prev = causal_ellipse_series(poolL, poolR, q, manual_moving, rng, refit_frames, prev) | |
| t_fit += time.time() - t0f | |
| edef = np.isfinite(ell).all(1) | |
| d = wheel_signed_distance(q, ell); d[~F] = np.nan | |
| agg["n_ell_undef"] += int((~edef).sum()); agg["n_d_undef"] += int(np.isnan(d).sum()) | |
| if not edef.any(): agg["routes_ellipse_never_defined"].append(u) | |
| if len(fits): | |
| agg["n_refits"] += len(fits); passed = fits[:, 6] > 0.5; agg["n_refits_passed"] += int(passed.sum()) | |
| nocand = np.isnan(fits[:, 7]); agg["n_refits_no_candidate"] += int(nocand.sum()) | |
| agg["n_refits_fail_points"] += int((~passed & ~nocand & (fits[:, 2] < WE.N_MIN_PTS)).sum()) | |
| agg["n_refits_fail_coverage"] += int((~passed & ~nocand & (fits[:, 2] >= WE.N_MIN_PTS) & (fits[:, 3] < WE.COV_MIN_DEG)).sum()) | |
| dev_rec["n_refits"] += len(fits); dev_rec["n_refits_passed"] += int(passed.sum()) | |
| if passed.any(): dev_rec["last_passed"] = dict(zip(FIT_COLS, [float(v) for v in fits[np.flatnonzero(passed)[-1]]]), uid=u) | |
| dev_rec["n_manual_moving"] += int(manual_moving.sum()); dev_rec["n_frames"] += len(t); dev_rec["n_ell_defined"] += int(edef.sum()) | |
| k = np.round(t * 10).astype(np.int64); k0, k1 = int(k.min()), int(k.max()); M = k1 - k0 + 1 | |
| GG = np.full((M, BK.D), np.nan, np.float32); VV = np.zeros(M, bool); FF = np.zeros(M, bool); DD = np.full(M, np.nan, np.float32) | |
| WD = np.zeros(M, bool); EE = np.full((M, 5), np.nan, np.float32) | |
| GG[k - k0] = G; VV[k - k0] = V; FF[k - k0] = F; DD[k - k0] = d; WD[k - k0] = edef; EE[k - k0] = ell | |
| ht = BK.hysteresis(np.abs(GG[:, 7]), head_on, head_off) # unchanged v2 head state | |
| s_raw, s_held = hyst_hands_v4(DD) | |
| ev_r, ev_l, unk = V3.confirmed_transitions(s_raw, s_held) | |
| fin = np.isfinite(s_raw[1:]) & np.isfinite(s_held[:-1]); agg["n_trans_raw"] += int((fin & (s_raw[1:] != s_held[:-1])).sum()) | |
| agg["n_conf_return"] += int(ev_r.sum()); agg["n_conf_leave"] += int(ev_l.sum()); agg["n_unknown_trans"] += int(unk.sum()) | |
| series[u] = dict(k0=k0, G=GG, V=VV, F=FF, ht=ht, hon=s_raw, ev_r=ev_r, ev_l=ev_l, unk=unk, wdef=WD, d=DD, ell=EE) | |
| refs[u] = ref.astype(np.float32); ells[u] = ell; d4s[u] = d; fitlog[u] = fits.astype(np.float32); nman[u] = int(manual_moving.sum()) | |
| for j in range(5): | |
| v = q[:, j][moving & np.isfinite(q[:, j])]; pools[j] = np.concatenate([pools[j], v.astype(np.float32)])[-BK.POOL_CAP:] | |
| agg["devices"][dev] = dev_rec | |
| if time.time() - t_last > 60: t_last = time.time(); log(f" devices {di+1}/{len(by_dev)} routes {len(series)}/{len(have)} {time.time()-t_start:.0f}s (fits {t_fit:.0f}s, refits {agg['n_refits']}, passed {agg['n_refits_passed']})") | |
| agg["fit_wall_s"] = round(t_fit, 1) | |
| return series, refs, ells, d4s, fitlog, nman, agg | |
| # ------------------------------------------------------------------------------------------------------------------ main | |
| def main(argv=None): | |
| ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) | |
| ap.add_argument("--regimes", default="manual,engaged"); ap.add_argument("--splits", default="train,val,test") | |
| ap.add_argument("--out-dir", default=str(OUT_V4)); ap.add_argument("--pose-dir", default=str(POSE_PREF), help="preferred pose dir; routes missing there fall back to results/pose_causal_routes (v2 rule)") | |
| ap.add_argument("--v2-dir", default=str(V2_DIR), help="v2 targets: input-equality check + hard-linked kinhist files") | |
| ap.add_argument("--grid10-dir", default=str(GRID10)) | |
| ap.add_argument("--refit-s", type=float, default=REFIT_S, help="refit cadence in seconds of new manual moving frames (pre-registration: 5 s; fallback 20 s if too slow)") | |
| a = ap.parse_args(argv); t_start = time.time() | |
| refit_frames = int(round(a.refit_s * FPS)) | |
| out = Path(a.out_dir); out.mkdir(parents=True, exist_ok=True); (out / "logs").mkdir(exist_ok=True) | |
| v2 = Path(a.v2_dir); P2 = Path(a.pose_dir) if a.pose_dir else None | |
| regimes, splits = a.regimes.split(","), a.splits.split(",") | |
| def log(msg): print(msg, flush=True) | |
| old = json.load(open(BK.OLD_STATS)); head_on, head_off = float(old["head_turn_on"]), float(old["head_turn_off"]) | |
| uids = sorted(set().union(*[set(np.unique(np.load(BK.S12R / f"targets_{r}_{s}.npz", allow_pickle=True)["uid"]).tolist()) for r in ("manual", "engaged") for s in BK.S12_SPLITS])) | |
| def pose_path(u): | |
| if P2 is not None and (P2 / f"{u}.npz").exists(): return P2 / f"{u}.npz" | |
| return BK.POSE_R / f"{u}.npz" | |
| have = [u for u in uids if pose_path(u).exists()]; n_from_pref = sum(1 for u in have if P2 is not None and (P2 / f"{u}.npz").exists()) | |
| log(f"v4 build: s12 routes {len(uids)}, with causal route pose {len(have)} ({n_from_pref} from {P2}); head thresholds on={head_on:.4f} off={head_off:.4f}; " | |
| f"hyst on/off {HYST_ON}/{HYST_OFF}, persistence {PERSIST_S} s, gap reset {GAP_RESET} frames; ellipse refit every {refit_frames} manual frames, gate >= {WE.N_MIN_PTS} inliers & >= {WE.COV_MIN_DEG:g} deg, " | |
| f"tau_in {WE.TAU_IN}, n_iter {WE.N_ITER}, axes [{WE.AX_MIN}, {WE.AX_MAX}], b/a >= {WE.RATIO_MIN}, seed {SEED}") | |
| series, refs, ells, d4s, fitlog, nman, agg = build_route_series(have, pose_path, head_on, head_off, Path(a.grid10_dir), refit_frames, log) | |
| np.savez_compressed(out / "route_refs.npz", uids=np.array(list(refs)), ref_quantities=np.array(BK.REFQ), ellipse_params=np.array(["cx", "cy", "a", "b", "theta"]), fit_cols=np.array(FIT_COLS), | |
| **{f"ref__{u}": r for u, r in refs.items()}, **{f"ell__{u}": r for u, r in ells.items()}, **{f"d4__{u}": r for u, r in d4s.items()}, **{f"fits__{u}": r for u, r in fitlog.items()}, | |
| **{f"n_manual_moving__{u}": np.int64(nman[u]) for u in nman}, | |
| rule=np.array("ref__: " + f"expanding median over MOVING frames <= t, device's earlier routes pool, POOL_CAP={BK.POOL_CAP}, N_MIN={BK.N_MIN} (v2, unchanged); ell__: " + WHEEL_RULE), | |
| units=np.array(BK.UNITS), label_version=np.array(LABEL_VERSION)) | |
| n_ref_mismatch = None | |
| if (v2 / "route_refs.npz").exists(): | |
| r2 = np.load(v2 / "route_refs.npz", allow_pickle=True); n_ref_mismatch = sum(1 for u in refs if f"ref__{u}" in r2.files and not np.array_equal(r2[f"ref__{u}"], refs[u], equal_nan=True)) | |
| assert n_ref_mismatch == 0, f"channel reference differs from v2 route_refs for {n_ref_mismatch} routes" | |
| nf = max(agg["n_frames"], 1) | |
| log(f"series for {len(series)} routes in {time.time()-t_start:.0f}s (ellipse fits {agg['fit_wall_s']}s); frames {agg['n_frames']}, grid10-unmatched {agg['n_grid10_unmatched']}, manual moving {agg['n_manual_moving']}, engaged moving {agg['n_engaged_moving']}; " | |
| f"head-ref undefined {agg['n_ref_undef_head']} ({100*agg['n_ref_undef_head']/nf:.2f}%), ellipse undefined {agg['n_ell_undef']} ({100*agg['n_ell_undef']/nf:.2f}%), d undefined {agg['n_d_undef']} ({100*agg['n_d_undef']/nf:.2f}%); " | |
| f"refits {agg['n_refits']} passed {agg['n_refits_passed']} (fail <200 inliers {agg['n_refits_fail_points']}, fail coverage {agg['n_refits_fail_coverage']}, no candidate {agg['n_refits_no_candidate']}); " | |
| f"routes without prior v2 pool {agg['routes_no_pool_v2']}, without prior manual pool {agg['routes_no_manual_pool']}, ellipse never defined {len(agg['routes_ellipse_never_defined'])}; " | |
| f"10 Hz raw transitions {agg['n_trans_raw']} -> confirmed return {agg['n_conf_return']} leave {agg['n_conf_leave']} (unconfirmable at route edges {agg['n_unknown_trans']}); v2 channel-ref mismatches {n_ref_mismatch}") | |
| offs_f = np.arange(1, BK.H + 1) * 2 | |
| stats = dict(source="routes", label_version=LABEL_VERSION, pose_dir=str(P2) if P2 else str(BK.POSE_R), pose_fallback_dir=str(BK.POSE_R), n_routes_from_pose_dir=n_from_pref, grid10_dir=str(a.grid10_dir), v2_dir=str(v2), | |
| head_turn_on=head_on, head_turn_off=head_off, hyst_on=HYST_ON, hyst_off=HYST_OFF, persist_s=PERSIST_S, gap_reset_frames=GAP_RESET, refit_frames=refit_frames, refit_s=a.refit_s, seed=SEED, | |
| ellipse=dict(tau_in=WE.TAU_IN, n_iter=WE.N_ITER, n_sample=WE.N_SAMPLE, n_min_pts=WE.N_MIN_PTS, cov_min_deg=WE.COV_MIN_DEG, cov_bin_deg=WE.COV_BIN_DEG, min_per_bin=WE.MIN_PER_BIN, cov_frac=WE.COV_FRAC, | |
| ax_min=WE.AX_MIN, ax_max=WE.AX_MAX, ratio_min=WE.RATIO_MIN, pool_cap_per_wrist=POOL_CAP), | |
| n_routes_s12=len(uids), n_routes_with_pose=len(have), | |
| ref_undefined_frame_frac=agg["n_ref_undef_head"] / nf, ellipse_undefined_frame_frac=agg["n_ell_undef"] / nf, d_undefined_frame_frac=agg["n_d_undef"] / nf, | |
| n_pose_frames=agg["n_frames"], n_grid10_unmatched_frames=agg["n_grid10_unmatched"], n_manual_moving_frames=agg["n_manual_moving"], n_engaged_moving_frames=agg["n_engaged_moving"], | |
| routes_without_prior_pool=agg["routes_no_pool_v2"], routes_without_prior_manual_pool=agg["routes_no_manual_pool"], routes_ellipse_never_defined=agg["routes_ellipse_never_defined"], routes_without_grid10=agg["routes_without_grid10"], | |
| n_refits=agg["n_refits"], n_refits_passed=agg["n_refits_passed"], n_refits_fail_points=agg["n_refits_fail_points"], n_refits_fail_coverage=agg["n_refits_fail_coverage"], n_refits_no_candidate=agg["n_refits_no_candidate"], fit_wall_s=agg["fit_wall_s"], | |
| n_10hz_raw_transitions=agg["n_trans_raw"], n_10hz_confirmed_return=agg["n_conf_return"], n_10hz_confirmed_leave=agg["n_conf_leave"], n_10hz_unconfirmable_transitions=agg["n_unknown_trans"], | |
| v2_channel_ref_mismatch_routes=n_ref_mismatch, pool_cap=BK.POOL_CAP, n_min=BK.N_MIN, units=BK.UNITS, wheel_ref_rule=WHEEL_RULE, devices=agg["devices"], splits={}) | |
| for reg in regimes: | |
| for sp in splits: | |
| t_sp = time.time() | |
| d = np.load(BK.S12R / f"targets_{reg}_{sp}.npz", allow_pickle=True); uid, don, t, Mc = d["uid"], d["dongle"], d["t"], d["M"].astype(bool); N = len(t) | |
| K = np.full((N, BK.H, BK.D), np.nan, np.float16); Kvalid = np.zeros((N, BK.H), bool); Kframe = np.zeros((N, BK.H), bool) | |
| K0 = np.full((N, BK.D), np.nan, np.float32); K0valid = np.zeros(N, bool); S = np.full((N, BK.H, 2), np.nan, np.float32); S0 = np.full((N, 2), np.nan, np.float32) | |
| EV = np.zeros((N, BK.H, 2, 2), bool); UNK = np.zeros((N, BK.H), bool); WDEF = np.zeros(N, bool); D0 = np.full(N, np.nan, np.float32); EL0 = np.full((N, 5), np.nan, np.float32) | |
| route_dec = np.zeros(N, bool); row_ok = np.zeros(N, bool) | |
| order = np.argsort(uid, kind="stable"); bounds = np.flatnonzero(np.r_[1, uid[order][1:] != uid[order][:-1], 1]) | |
| for a_, b_ in zip(bounds[:-1], bounds[1:]): | |
| rows = order[a_:b_]; u = uid[rows[0]] | |
| if u not in series: continue | |
| rs = series[u]; route_dec[rows] = True; ka = np.round(t[rows] * 10).astype(np.int64) - rs["k0"]; M = len(rs["G"]) | |
| def gather(A, offs): | |
| idx = ka[:, None] + offs[None, :]; ok = (idx >= 0) & (idx < M); o = A[np.clip(idx, 0, M - 1)] | |
| return np.where(ok[..., None] if o.ndim == 3 else ok, o, np.nan if A.dtype.kind == "f" else False) | |
| K[rows] = gather(rs["G"], offs_f).astype(np.float16); Kvalid[rows] = gather(rs["V"], offs_f); Kframe[rows] = gather(rs["F"], offs_f) | |
| ok0 = (ka >= 0) & (ka < M); kc = np.clip(ka, 0, M - 1) | |
| K0[rows] = np.where(ok0[:, None], rs["G"][kc], np.nan); K0valid[rows] = ok0 & rs["V"][kc]; row_ok[rows] = ok0 & rs["F"][kc] | |
| S[rows, :, 0] = gather(rs["hon"], offs_f); S[rows, :, 1] = gather(rs["ht"], offs_f); S0[rows, 0] = np.where(ok0, rs["hon"][kc], np.nan); S0[rows, 1] = np.where(ok0, rs["ht"][kc], np.nan) | |
| WDEF[rows] = ok0 & rs["wdef"][kc]; D0[rows] = np.where(ok0, rs["d"][kc], np.nan); EL0[rows] = np.where(ok0[:, None], rs["ell"][kc], np.nan) | |
| for c, key in enumerate(("ev_r", "ev_l")): | |
| EV[rows, :, c, 0] = gather(rs[key].astype(bool), offs_f); EV[rows, :, c, 1] = gather(rs[key].astype(bool), offs_f - 1) | |
| UNK[rows] = gather(rs["unk"], offs_f) | gather(rs["unk"], offs_f - 1) | |
| n_kvalid_raw = int(Kvalid.sum()); Kvalid &= Mc; S[~Mc] = np.nan | |
| E = np.zeros((N, BK.H, 3), np.uint8); EM = np.zeros((N, BK.H, 3), bool); E0 = np.full((N, 3), np.nan, np.float32) | |
| E[:, :, 0], EM[:, :, 0] = V3.hand_events_on_grid(S0[:, 0], S[:, :, 0], EV[:, :, 0], UNK, Mc) | |
| E[:, :, 1], EM[:, :, 1] = V3.hand_events_on_grid(1 - S0[:, 0], 1 - S[:, :, 0], EV[:, :, 1], UNK, Mc) | |
| E[:, :, 2], EM[:, :, 2] = BK.onsets(S0[:, 1], S[:, :, 1]) # head_turn_onset unchanged | |
| E0[:, 0], E0[:, 1], E0[:, 2] = S0[:, 0], 1 - S0[:, 0], S0[:, 1] | |
| assert np.array_equal(uid, d["uid"]) and np.allclose(t, d["t"]) and not (Kvalid & ~Mc).any() and not (EM & ~Mc[:, :, None]).any() | |
| assert not (E.astype(bool) & ~EM).any(), "E_c must be a subset of EM_c" | |
| # ---- inputs identical to v2 (whole split), head channel unchanged | |
| chk = {} | |
| v2k = v2 / f"kin_causal_{reg}_{sp}.npz" | |
| if v2k.exists(): | |
| z2 = np.load(v2k, allow_pickle=True) | |
| for key, arr in (("K_c", K), ("K0_c", K0), ("Kvalid", Kvalid), ("Kframe", Kframe), ("K0valid", K0valid), ("row_ok", row_ok), ("route_decoded", route_dec)): | |
| chk[key] = bool(np.array_equal(z2[key], arr, equal_nan=True) if arr.dtype.kind == "f" else np.array_equal(z2[key], arr)) | |
| chk["S_head"] = bool(np.array_equal(z2["S_c"][:, :, 1], S[:, :, 1], equal_nan=True) and np.array_equal(z2["S0_c"][:, 1], S0[:, 1], equal_nan=True)) | |
| chk["E_head"] = bool(np.array_equal(z2["E_c"][:, :, 2], E[:, :, 2]) and np.array_equal(z2["EM_c"][:, :, 2], EM[:, :, 2])) | |
| bad = [k_ for k_, ok_ in chk.items() if not ok_] | |
| assert not bad, f"{reg}/{sp}: v4 differs from v2 on {bad} (inputs / head channel must be identical)" | |
| hon2 = z2["S_c"][:, :, 0]; hon2_0 = z2["S0_c"][:, 0]; E2 = z2["E_c"]; EM2 = z2["EM_c"] | |
| else: | |
| hon2 = hon2_0 = E2 = EM2 = None | |
| v3k = V3.OUT_V3 / f"kin_causal_{reg}_{sp}.npz"; z3 = np.load(v3k, allow_pickle=True) if v3k.exists() else None | |
| np.savez_compressed(out / f"kin_causal_{reg}_{sp}.npz", uid=uid, dongle=don, t=t, row_ok=row_ok, route_decoded=route_dec, K_c=K, Kvalid=Kvalid, Kframe=Kframe, K0_c=K0, K0valid=K0valid, | |
| S_c=S, S0_c=S0, E_c=E, EM_c=EM, E0_c=E0, channels=np.array(BK.CHANNELS), event_names=np.array(BK.EVENTS), state_names=np.array(BK.STATES), units=np.array(BK.UNITS), source=np.array("routes"), | |
| wheel_ref_defined=WDEF, dwheel0_c=D0, ellipse0_c=EL0, ellipse_params=np.array(["cx", "cy", "a", "b", "theta"]), | |
| persist_s=np.float32(PERSIST_S), hyst_on=np.float32(HYST_ON), hyst_off=np.float32(HYST_OFF), gap_reset_frames=np.int64(GAP_RESET), refit_frames=np.int64(refit_frames), | |
| label_version=np.array(LABEL_VERSION), wheel_ref_rule=np.array(WHEEL_RULE)) | |
| v2h = v2 / f"kinhist_causal_{reg}_{sp}.npz"; how = V3._link_or_copy(v2h, out / f"kinhist_causal_{reg}_{sp}.npz") if v2h.exists() else "missing_v2" | |
| st = dict(N=int(N), routes=int(len(np.unique(uid))), routes_decoded=int(len({u for u in np.unique(uid) if u in series})), n_route_decoded_rows=int(route_dec.sum()), coverage_route_decoded=float(route_dec.mean()), | |
| n_row_ok=int(row_ok.sum()), coverage_row_ok=float(row_ok.mean()), coverage_row_ok_given_decoded=float(row_ok[route_dec].mean()) if route_dec.any() else None, | |
| Kvalid_frac_row_ok=float(Kvalid[row_ok].mean()) if row_ok.any() else None, Kvalid_all25_frac_row_ok=float(Kvalid[row_ok].all(1).mean()) if row_ok.any() else None, Kvalid_dropped_by_s12_M=int(n_kvalid_raw - Kvalid.sum()), | |
| K0valid_frac_row_ok=float(K0valid[row_ok].mean()) if row_ok.any() else None, | |
| wheel_ellipse_defined_frac_row_ok=float(WDEF[row_ok].mean()) if row_ok.any() else None, S0_hands_finite_frac_row_ok=float(np.isfinite(S0[row_ok, 0]).mean()) if row_ok.any() else None, | |
| S0_hands_finite_frac_row_ok_v2=float(np.isfinite(hon2_0[row_ok]).mean()) if hon2_0 is not None and row_ok.any() else None, | |
| S0_hands_finite_frac_row_ok_v3=float(np.isfinite(z3["S0_c"][row_ok, 0]).mean()) if z3 is not None and row_ok.any() else None, | |
| coverage_loss_S0_v2def_v4nan_frac_row_ok=float((np.isfinite(hon2_0[row_ok]) & ~np.isfinite(S0[row_ok, 0])).mean()) if hon2_0 is not None and row_ok.any() else None, | |
| hands_on_prev=float(np.nanmean(S[:, :, 0])) if np.isfinite(S[:, :, 0]).any() else None, hands_on_prev_v2=float(np.nanmean(hon2)) if hon2 is not None and np.isfinite(hon2).any() else None, | |
| hands_on_prev_v3=float(np.nanmean(z3["S_c"][:, :, 0])) if z3 is not None else None, | |
| hands_on_at_anchor=float(np.nanmean(S0[:, 0])) if np.isfinite(S0[:, 0]).any() else None, head_turned_prev=float(np.nanmean(S[:, :, 1])) if np.isfinite(S[:, :, 1]).any() else None, | |
| dwheel0_quantiles_row_ok={f"q{p_}": float(np.nanpercentile(D0[row_ok], p_)) for p_ in (5, 25, 50, 75, 95)} if row_ok.any() and np.isfinite(D0[row_ok]).any() else None, | |
| n_E_hand_return=int(E[:, :, 0].sum()), n_E_hand_leave=int(E[:, :, 1].sum()), n_E_head_turn=int(E[:, :, 2].sum()), | |
| n_E_hand_return_v2=int(E2[:, :, 0].sum()) if E2 is not None else None, n_E_hand_leave_v2=int(E2[:, :, 1].sum()) if E2 is not None else None, | |
| n_E_hand_return_v3=int(z3["E_c"][:, :, 0].sum()) if z3 is not None else None, n_E_hand_leave_v3=int(z3["E_c"][:, :, 1].sum()) if z3 is not None else None, | |
| EM_hand_frac=float(EM[:, :, 0].mean()), EM_hand_frac_v2=float(EM2[:, :, 0].mean()) if EM2 is not None else None, n_EM_cleared_unconfirmable=int((UNK & Mc & BK.onsets(S0[:, 0], S[:, :, 0])[1]).sum()), | |
| channel_nan_frac_row_ok={c: float(np.isnan(K[row_ok][:, :, i].astype(np.float32)).mean()) for i, c in enumerate(BK.CHANNELS)} if row_ok.any() else None, | |
| v2_identity_checks=chk, kinhist_file=how, events=V3._event_stats(E, EM, E0, S0, S), wall_s=round(time.time() - t_sp, 1)) | |
| stats["splits"][f"{reg}_{sp}"] = st | |
| log(f"{reg}/{sp}: N={N} row_ok={row_ok.sum()} ({row_ok.mean()*100:.2f}%) ellipse_defined|row_ok={st['wheel_ellipse_defined_frac_row_ok']:.4f} S0 finite v4/v3/v2={st['S0_hands_finite_frac_row_ok']:.4f}/{st['S0_hands_finite_frac_row_ok_v3']}/{st['S0_hands_finite_frac_row_ok_v2']} " | |
| f"hands_on v4/v3/v2={st['hands_on_prev']:.4f}/{st['hands_on_prev_v3']}/{st['hands_on_prev_v2']} E return v4/v3/v2={st['n_E_hand_return']}/{st['n_E_hand_return_v3']}/{st['n_E_hand_return_v2']} leave v4/v3/v2={st['n_E_hand_leave']}/{st['n_E_hand_leave_v3']}/{st['n_E_hand_leave_v2']} head={st['n_E_head_turn']} " | |
| f"kinhist={how} v2-identity={'OK' if chk and all(chk.values()) else chk} " + " ".join(f"{e}@3s={st['events'][e]['pos@3s']}/{st['events'][e]['n@3s']}" for e in BK.EVENTS) + f" ({st['wall_s']}s)") | |
| stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) | |
| (out / f"build_stats_{stamp}.json").write_text(json.dumps(stats, indent=1)) | |
| latest = out / "build_stats_latest.json"; merged = dict(stats) | |
| if latest.exists(): # merge split blocks over invocations | |
| try: prev = json.load(open(latest)); merged["splits"] = {**prev.get("splits", {}), **stats["splits"]}; merged["merged_from_previous_latest"] = True | |
| except Exception: pass | |
| latest.write_text(json.dumps(merged, indent=1)) | |
| inputs = [BK.S12R / f"targets_{r}_{s}.npz" for r in regimes for s in splits] + [BK.OLD_STATS] + [p for p in [v2 / "route_refs.npz"] if p.exists()] \ | |
| + [p for r in regimes for s in splits for p in (v2 / f"kin_causal_{r}_{s}.npz", v2 / f"kinhist_causal_{r}_{s}.npz") if p.exists()] + [Path(a.grid10_dir) / f"{u}.npz" for u in have if (Path(a.grid10_dir) / f"{u}.npz").exists()] | |
| write_run_json(out, "build_kin_causal_routes_v4", [Path(__file__), Path(BK.__file__), Path(V3.__file__), Path(WE.__file__)], inputs, SEED, t_start, | |
| extra=dict(summary={k: v for k, v in stats.items() if k not in ("splits", "devices")}, coverage={k: dict(N=v["N"], route_decoded=v["coverage_route_decoded"], row_ok=v["coverage_row_ok"], wheel_ellipse_defined=v["wheel_ellipse_defined_frac_row_ok"], routes=v["routes"]) for k, v in stats["splits"].items()}, | |
| pose_dir=str(P2) if P2 else str(BK.POSE_R), stats_file=f"build_stats_{stamp}.json", regimes=regimes, splits=splits, refit_frames=refit_frames), input_listing=True) | |
| log(f"KIN_CAUSAL_ROUTES_V4_DONE wall {time.time()-t_start:.0f}s") | |
| if __name__ == "__main__": | |
| main() | |
Xet Storage Details
- Size:
- 35.2 kB
- Xet hash:
- 450c65923323b8556bdf1f81c3b5005b36ae67508f73242f864e408549b3de0b
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.