Buckets:
| """cwm3/data/extract_pose_causal.py — causal driver pose: RTMW-x-l (COCO-WholeBody 133) on the per-clip CAUSAL person_box | |
| from roi.json (chosen from PREFIX frames only by cwm2/data/extract_frames.driver_roi), re-ordered to HALPE-136. | |
| Time base (verified in cwm2/data/extract_frames.py:118 and cwm2/data/extract_full.py:48, both cams 20 fps, frame-anchored): | |
| t_abs(k) = t0 − 5 + 0.2·k (k = 0..74; k = 25 is the anchor t0), frame_idx = round(t_abs · 20) | |
| which is exactly the `src_idx = round((ws + m/1000)·FPS)` of the frame shards, so the pose grid coincides with the | |
| 5 Hz head/hand crop grid of frames_full_standard (t_ms = 200·k). No container fps is ever used. | |
| Decoding: HEVC 1928×1208 software decode is the bottleneck (no hw decoder on this host), so decoding runs in | |
| `--decode-workers` processes (cv2.VideoCapture seek + grab/retrieve, verified bit-identical to decord.get_batch on 3 windows | |
| of 3 clips on 2026-09-10, max mean|diff| 0.000 while ±1-frame shifts differ by ≥ 3.1) while the main process runs the GPU. | |
| `--verify-decoder N` re-decodes the first N clips with decord and records the max |crop diff| in the benchmark json. | |
| Video source per clip (results/baton_leaves.json): the local leaf dcamera.mp4 if its size equals the remote size | |
| (`local_identical_size`), otherwise the remote leaf's dcamera.mp4 is rsync'd to results/scratch_video/<leaf>/ and deleted | |
| after every clip of that leaf has been processed (disk guard: ≥ 15 GB free + file size before any download). | |
| Pose: detection-free. bbox = person_box expanded by MARGIN (10 % of its width/height per side, clipped to the frame); | |
| rtmlib's own top-down preprocessing (bbox_xyxy2cs padding 1.25, aspect-fixed affine to 192×256, ImageNet mean/std on the | |
| BGR frame exactly as rtmlib.RTMPose does) is applied per frame — on a crop of the frame around the padded box (same | |
| result, avoids copying the 7 MB frame) — the 75 crops of a clip are stacked into ONE onnxruntime CUDA batch (the ONNX has | |
| a dynamic batch axis), SimCC argmax → crop px → native px (rtmlib.RTMPose.postprocess formula with the native center/scale). | |
| 133 → HALPE-136 mapping (COCO-WholeBody slots per explore/opendriver/odlib.py:16-18: body 0–16, feet 17–22, face 23–90, | |
| lhand 91–111, rhand 112–132): | |
| HALPE 0–16 body ← WB 0–16 (same COCO order) | |
| HALPE 17 head_top ← NaN (RTMW has no head-top point) | |
| HALPE 18 neck ← midpoint(WB 5, WB 6), conf = min of the pair | |
| HALPE 19 hip/pelvis ← midpoint(WB 11, WB 12), conf = min of the pair | |
| HALPE 20–25 feet ← BY NAME, not by index shift: 20 LBigToe←17, 21 RBigToe←20, 22 LSmallToe←18, | |
| 23 RSmallToe←21, 24 LHeel←19, 25 RHeel←22 (HALPE interleaves L/R per toe type, COCO-WholeBody | |
| groups per foot; the naive "20–25 = 17–22" shift would swap 4 of the 6 points) | |
| HALPE 26–93 face (68) ← WB 23–90 (same 68-landmark order) | |
| HALPE 94–114 left hand ← WB 91–111 ; HALPE 115–135 right hand ← WB 112–132 (same 21-point hand order) | |
| Output results/pose_causal/<clip_id>.npz: | |
| t_rel[75] f32 (−5..+9.8), t_abs[75] f64, frame_idx[75] i64, frame_ok[75] bool (index inside the video), | |
| kpts[75,136,3] f16 = (x/W, y/H, conf) in NATIVE-frame normalised units (NaN where undefined), | |
| mask[75,136] bool = finite ∧ conf ≥ 0.3, person_box[4] (roi.json, native px), person_box_used[4] (with margin), | |
| roi_source, co_occupant_present, selection_rule, model_id, native_wh[2], uid, dongle, t0, video_source, kpt_order='halpe136'. | |
| GPU use only under `flock /tmp/s13_gpu.lock` (the caller / run_extract_pose_causal.sh does that). Exit code 3 = time/RAM | |
| budget reached with work remaining (resumable: existing npz are skipped); 0 = everything requested is done.""" | |
| import argparse, json, math, os, shutil, subprocess, sys, time | |
| import multiprocessing as mp | |
| from concurrent.futures import ProcessPoolExecutor, wait, FIRST_COMPLETED | |
| from pathlib import Path | |
| import numpy as np | |
| ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(ROOT / "common")) | |
| CLIP_MAP = Path("/home/USER/cwm2/private/clip_map_full.json") | |
| MANIFEST = Path("/home/USER/cwm2/ship/dataset_full/manifest.jsonl") | |
| LEAVES = ROOT / "results" / "baton_leaves.json"; ROI = ROOT / "results" / "roi" | |
| OUT = ROOT / "results" / "pose_causal"; SCRATCH = ROOT / "results" / "scratch_video" | |
| LROOT = Path("/home/USER/BATON"); REMOTE = "USER@REMOTE_HOST"; RROOT = "/data/backup/BATON" | |
| SSH = ["ssh", "-S", "/tmp/baton_bak_ctl", "-o", "BatchMode=yes", "-o", "ConnectTimeout=20"] | |
| ONNX = Path.home() / ".cache/rtmlib/hub/checkpoints/rtmw-dw-x-l_simcc-cocktail14_270e-256x192_20231122.onnx" | |
| MODEL_ID = "rtmw-dw-x-l_simcc-cocktail14_270e-256x192_20231122.onnx (rtmlib 0.0.15 'balanced' pose model, onnxruntime-gpu CUDA EP, detection-free on causal person_box)" | |
| FPS, PRE_S, STEP_MS, NK = 20, 5.0, 200, 75 | |
| IN_W, IN_H = 192, 256; MEAN = np.array((123.675, 116.28, 103.53), np.float32); STD = np.array((58.395, 57.12, 57.375), np.float32) | |
| MARGIN, SCORE_TH, MIN_FREE_GB = 0.10, 0.3, 15.0 | |
| HALPE_FROM_WB = np.full(136, -1, int) # destination -> COCO-WholeBody source; -1 = special (NaN / midpoint) | |
| HALPE_FROM_WB[0:17] = np.arange(0, 17) | |
| HALPE_FROM_WB[[20, 21, 22, 23, 24, 25]] = [17, 20, 18, 21, 19, 22] | |
| HALPE_FROM_WB[26:94] = np.arange(23, 91) | |
| HALPE_FROM_WB[94:115] = np.arange(91, 112) | |
| HALPE_FROM_WB[115:136] = np.arange(112, 133) | |
| def wb133_to_halpe136(k): | |
| """k [N,133,3] native px + conf -> [N,136,3].""" | |
| N = k.shape[0]; h = np.full((N, 136, 3), np.nan, np.float32) | |
| src = HALPE_FROM_WB >= 0 | |
| h[:, src] = k[:, HALPE_FROM_WB[src]] | |
| for dst, (a, b) in ((18, (5, 6)), (19, (11, 12))): | |
| h[:, dst, :2] = (k[:, a, :2] + k[:, b, :2]) / 2 | |
| h[:, dst, 2] = np.minimum(k[:, a, 2], k[:, b, 2]) | |
| bad = ~(np.isfinite(k[:, a, :2]).all(1) & np.isfinite(k[:, b, :2]).all(1)); h[bad, dst] = np.nan | |
| return h | |
| def expand_box(b, W, H, m=MARGIN): | |
| x0, y0, x1, y1 = b; w, h = x1 - x0, y1 - y0 | |
| return [int(max(0, round(x0 - m * w))), int(max(0, round(y0 - m * h))), int(min(W, round(x1 + m * w))), int(min(H, round(y1 + m * h)))] | |
| def frame_grid(t0): | |
| ms = np.arange(NK) * STEP_MS | |
| t_abs = (t0 - PRE_S) + ms / 1000.0 | |
| return ms, t_abs, np.array([int(round(t * FPS)) for t in t_abs], np.int64) | |
| def box_geometry(used): | |
| """rtmlib RTMPose.preprocess geometry: center, aspect-fixed scale (native px).""" | |
| from rtmlib.tools.pose_estimation.pre_processings import bbox_xyxy2cs | |
| center, scale = bbox_xyxy2cs(np.asarray(used, np.float32), padding=1.25) | |
| w, h = float(scale[0]), float(scale[1]); ar = IN_W / IN_H | |
| scale = np.array([w, w / ar], np.float32) if w > h * ar else np.array([h * ar, h], np.float32) | |
| return center.astype(np.float32), scale | |
| def crop_input(fr_bgr, center, scale): | |
| """uint8 [IN_H, IN_W, 3] model input, computed on a crop around the padded box (identical to rtmlib on the full frame).""" | |
| from rtmlib.tools.pose_estimation.pre_processings import top_down_affine | |
| H, W = fr_bgr.shape[:2] | |
| x0 = max(0, int(math.floor(center[0] - scale[0] / 2)) - 2); y0 = max(0, int(math.floor(center[1] - scale[1] / 2)) - 2) | |
| x1 = min(W, int(math.ceil(center[0] + scale[0] / 2)) + 2); y1 = min(H, int(math.ceil(center[1] + scale[1] / 2)) + 2) | |
| crop = fr_bgr[y0:y1, x0:x1] | |
| img, _ = top_down_affine((IN_W, IN_H), scale.copy(), center - np.array([x0, y0], np.float32), crop) | |
| return img | |
| def decode_clip(job): | |
| """Worker: (cid, vpath, t0, n_video_frames, used) -> dict with uint8 crops [75,IN_H,IN_W,3], geometry, timings.""" | |
| import cv2 | |
| cv2.setNumThreads(2) | |
| cid, vpath, t0, n_video, used = job | |
| t = time.time() | |
| ms, t_abs, fidx = frame_grid(t0) | |
| frame_ok = (fidx >= 0) & (fidx < n_video); idx = np.clip(fidx, 0, n_video - 1) | |
| center, scale = box_geometry(used) | |
| crops = np.zeros((NK, IN_H, IN_W, 3), np.uint8); got = np.zeros(NK, bool) | |
| cap = cv2.VideoCapture(vpath) | |
| if not cap.isOpened(): raise RuntimeError(f"cannot open {vpath}") | |
| lo, hi = int(idx.min()), int(idx.max()); want = {} | |
| for k, i in enumerate(idx): want.setdefault(int(i), []).append(k) | |
| cap.set(cv2.CAP_PROP_POS_FRAMES, lo); n_grab = 0 | |
| for i in range(lo, hi + 1): | |
| if not cap.grab(): break | |
| n_grab += 1 | |
| if i in want: | |
| ok, fr = cap.retrieve() | |
| if not ok: break | |
| c = crop_input(fr, center, scale) | |
| for k in want[i]: crops[k] = c; got[k] = True | |
| cap.release() | |
| return dict(cid=cid, crops=crops, got=got, center=center, scale=scale, frame_ok=frame_ok, fidx=fidx, t_abs=t_abs, ms=ms, used=used, | |
| n_grab=n_grab, decode_s=time.time() - t) | |
| def ensure_master(): | |
| if subprocess.run(SSH + ["-O", "check", REMOTE], capture_output=True).returncode == 0: return | |
| subprocess.run(["bash", "/home/USER/cwm2/private/ssh_master.sh"], capture_output=True); time.sleep(2) | |
| if subprocess.run(SSH + ["-O", "check", REMOTE], capture_output=True).returncode != 0: raise RuntimeError("ssh master unavailable") | |
| class Pose: | |
| def __init__(self, gpu_mem_limit=None): | |
| import torch # noqa: F401 -- preloads torch's bundled CUDA 12.8 / cuDNN 9 libs; without it onnxruntime-gpu 1.22 silently falls back to CPU (verified 2026-09-10) | |
| import onnxruntime as ort | |
| from rtmlib.tools.pose_estimation.post_processings import get_simcc_maximum | |
| self.simcc = get_simcc_maximum | |
| so = ort.SessionOptions(); so.log_severity_level = 3 | |
| prov = ["CUDAExecutionProvider"] if gpu_mem_limit is None else [("CUDAExecutionProvider", {"gpu_mem_limit": int(gpu_mem_limit), "arena_extend_strategy": "kSameAsRequested"})] | |
| self.sess = ort.InferenceSession(str(ONNX), so, providers=prov) | |
| assert "CUDAExecutionProvider" in self.sess.get_providers(), self.sess.get_providers() | |
| self.inp = self.sess.get_inputs()[0].name; self.outs = [o.name for o in self.sess.get_outputs()] | |
| self.sess.run(self.outs, {self.inp: np.zeros((8, 3, IN_H, IN_W), np.float32)}) # warm-up | |
| def infer(self, crops_u8, center, scale, chunk=80): | |
| x = np.ascontiguousarray(((crops_u8.astype(np.float32) - MEAN) / STD).transpose(0, 3, 1, 2)) | |
| locs, vals = [], [] | |
| for i in range(0, len(x), chunk): | |
| sx, sy = self.sess.run(self.outs, {self.inp: x[i:i + chunk]}) | |
| l, v = self.simcc(sx, sy); locs.append(l); vals.append(v) | |
| locs = np.concatenate(locs); vals = np.concatenate(vals) | |
| kp = locs / 2.0 / np.array([IN_W, IN_H], np.float32) * scale + center - scale / 2 # rtmlib RTMPose.postprocess | |
| kp[locs[..., 0] < 0] = np.nan | |
| return np.concatenate([kp, vals[..., None]], -1).astype(np.float32) | |
| def leaf_rel(meta): return str(Path(meta["dcam"]).parent.relative_to(LROOT)) | |
| def resolve_video(leaf, leaves, stats): | |
| """-> (path, source) ; downloads the remote leaf dcamera.mp4 into scratch when needed.""" | |
| info = leaves["leaves"].get(leaf); local = LROOT / leaf / "dcamera.mp4" | |
| if info is not None and info["local_status"] == "local_identical_size": return str(local), "local" | |
| if info is None and local.exists(): return str(local), "local_unverified" | |
| ensure_master() | |
| d = SCRATCH / leaf.replace("/", "__"); d.mkdir(parents=True, exist_ok=True); dst = d / "dcamera.mp4" | |
| need = (info or {}).get("dcamera_bytes") or 0 | |
| if dst.exists() and dst.stat().st_size == need: return str(dst), "remote_scratch_cached" | |
| free = shutil.disk_usage(str(ROOT)).free | |
| if free - need < MIN_FREE_GB * 1e9: raise RuntimeError(f"disk guard: free {free/1e9:.1f} GB, need {need/1e9:.1f} GB + {MIN_FREE_GB} GB") | |
| t = time.time() | |
| subprocess.run(["rsync", "-a", "--partial", "--inplace", "-e", " ".join(SSH), f"{REMOTE}:{RROOT}/{leaf}/dcamera.mp4", str(dst)], check=True) | |
| secs = time.time() - t; nb = dst.stat().st_size | |
| if need and nb != need: raise RuntimeError(f"download size mismatch {nb} != {need}") | |
| stats["download_bytes"] += nb; stats["download_secs"] += secs; stats["n_downloads"] += 1 | |
| stats["download_leaves"].append(dict(leaf=leaf, bytes=nb, secs=round(secs, 1), MBps=round(nb / 1e6 / max(secs, 1e-6), 1))) | |
| return str(dst), "remote_download" | |
| def finish_clip(d, meta, roi, pose, video_source): | |
| cid = d["cid"]; W, H = meta["cab"]["w"], meta["cab"]["h"] | |
| t = time.time(); k133 = pose.infer(d["crops"], d["center"], d["scale"]); t_inf = time.time() - t | |
| h = wb133_to_halpe136(k133); h[~(d["frame_ok"] & d["got"])] = np.nan | |
| kn = h.copy(); kn[..., 0] /= W; kn[..., 1] /= H | |
| k16 = kn.astype(np.float16) # mask is derived from the STORED f16 values (recomputable contract) | |
| mask = np.isfinite(k16).all(-1) & (np.nan_to_num(k16[..., 2].astype(np.float32)) >= SCORE_TH) | |
| t = time.time() | |
| np.savez_compressed(OUT / f"{cid}.npz", t_rel=(d["ms"] / 1000.0 - PRE_S).astype(np.float32), t_abs=d["t_abs"], frame_idx=d["fidx"], frame_ok=d["frame_ok"] & d["got"], | |
| kpts=k16, mask=mask, person_box=np.asarray(roi["person_box"], np.int32), person_box_used=np.asarray(d["used"], np.int32), | |
| roi_source=str(roi.get("roi_source")), co_occupant_present=bool(roi.get("co_occupant_present", False)), | |
| selection_rule=str(roi.get("selection_rule")), model_id=MODEL_ID, native_wh=np.array([W, H], np.int32), | |
| uid=str(meta["uid"]), dongle=str(meta["dongle"]), t0=float(meta["t0"]), video_source=video_source, kpt_order="halpe136", | |
| score_th=SCORE_TH, margin=MARGIN, n_video_frames=int(meta["cab"]["n_frames"])) | |
| t_save = time.time() - t | |
| return dict(clip_id=cid, n_frames=NK, n_frame_ok=int((d["frame_ok"] & d["got"]).sum()), n_grab=int(d["n_grab"]), decode_s=round(d["decode_s"], 4), infer_s=round(t_inf, 4), save_s=round(t_save, 4), | |
| det_rate_body=float(mask[:, :17].mean()), mean_conf_body=float(np.nanmean(kn[:, :17, 2])), video_source=video_source) | |
| def verify_with_decord(job): | |
| """Cross-check: decode the same frames with decord.get_batch and compare the model-input crops (uint8).""" | |
| import decord; decord.bridge.set_bridge("native") | |
| cid, vpath, t0, n_video, used = job | |
| _, _, fidx = frame_grid(t0); idx = np.clip(fidx, 0, n_video - 1) | |
| vr = decord.VideoReader(vpath, ctx=decord.cpu(0), num_threads=4) | |
| frames = vr.get_batch(list(idx)).asnumpy()[:, :, :, ::-1] | |
| center, scale = box_geometry(used) | |
| return np.stack([crop_input(np.ascontiguousarray(f), center, scale) for f in frames]) | |
| def main(): | |
| ap = argparse.ArgumentParser() | |
| ap.add_argument("--clips", default="", help="file with clip ids (one per line)"); ap.add_argument("--ids", nargs="*", default=[]) | |
| ap.add_argument("--all", action="store_true", help="every clip in the manifest (manifest order = route-grouped)") | |
| ap.add_argument("--max-minutes", type=float, default=20.0); ap.add_argument("--max-rss-gb", type=float, default=20.0) | |
| ap.add_argument("--decode-workers", type=int, default=6); ap.add_argument("--verify-decoder", type=int, default=0) | |
| ap.add_argument("--bench-out", default="", help="write a benchmark json with per-clip timings + projections") | |
| ap.add_argument("--keep-scratch", action="store_true"); ap.add_argument("--force", action="store_true") | |
| a = ap.parse_args() | |
| from runmeta import write_run_json | |
| import psutil | |
| t_start = time.time(); OUT.mkdir(parents=True, exist_ok=True) | |
| ids = list(a.ids) | |
| if a.clips: ids += [l.strip() for l in open(a.clips) if l.strip()] | |
| if a.all: ids += [json.loads(l)["clip_id"] for l in open(MANIFEST)] | |
| seen = set(); ids = [c for c in ids if not (c in seen or seen.add(c))] | |
| todo = [c for c in ids if a.force or not (OUT / f"{c}.npz").exists()] | |
| cm = json.load(open(CLIP_MAP)); leaves = json.load(open(LEAVES)) | |
| print(f"requested {len(ids)} clips, todo {len(todo)}", flush=True) | |
| groups = {} | |
| for c in todo: | |
| if c in cm: groups.setdefault(leaf_rel(cm[c]), []).append(c) | |
| def is_local(l): return (leaves["leaves"].get(l) or {}).get("local_status") == "local_identical_size" | |
| order = sorted(groups, key=lambda l: (not is_local(l), l)) # local leaves first, remote (download) leaves last | |
| stats = dict(download_bytes=0, download_secs=0.0, n_downloads=0, download_leaves=[]) | |
| recs, fails, checks = [], [], []; budget_hit = False; n_done = 0 | |
| pool = ProcessPoolExecutor(max_workers=a.decode_workers, mp_context=mp.get_context("spawn")) | |
| pose = Pose() | |
| def out_of_budget(): return time.time() - t_start > a.max_minutes * 60 or psutil.Process().memory_info().rss > a.max_rss_gb * 1e9 | |
| def drain(pending, block_all): | |
| nonlocal n_done | |
| while pending and (block_all or len(pending) >= 2 * a.decode_workers): | |
| done, _ = wait(list(pending), return_when=FIRST_COMPLETED) | |
| for f in done: | |
| cid, src, leaf = pending.pop(f) | |
| try: | |
| d = f.result(); recs.append(finish_clip(d, cm[cid], json.load(open(ROI / f"{cid}.json")), pose, src)); n_done += 1 | |
| if n_done % 25 == 0: | |
| r = recs[-25:]; fr = sum(x["n_frames"] for x in r); el = time.time() - t_start | |
| print(f" {n_done}/{len(todo)} infer {1000*sum(x['infer_s'] for x in r)/fr:.2f} ms/f decode(worker) {sum(x['n_grab'] for x in r)/max(sum(x['decode_s'] for x in r),1e-6):.0f} f/s throughput {n_done/el:.2f} clip/s rss {psutil.Process().memory_info().rss/1e9:.1f} GB", flush=True) | |
| except Exception as e: | |
| fails.append(dict(clip_id=cid, leaf=leaf, err=repr(e)[:200])) | |
| pending = {} | |
| for leaf in order: | |
| if out_of_budget(): budget_hit = True; break | |
| cids = groups[leaf] | |
| try: vpath, src = resolve_video(leaf, leaves, stats) | |
| except Exception as e: | |
| for c in cids: fails.append(dict(clip_id=c, leaf=leaf, err=repr(e)[:200])) | |
| continue | |
| for c in cids: | |
| if out_of_budget(): budget_hit = True; break | |
| rp = ROI / f"{c}.json" | |
| if not rp.exists(): fails.append(dict(clip_id=c, leaf=leaf, err="no_roi_json")); continue | |
| meta = cm[c]; roi = json.load(open(rp)); used = expand_box(roi["person_box"], meta["cab"]["w"], meta["cab"]["h"]) | |
| job = (c, vpath, float(meta["t0"]), int(meta["cab"]["n_frames"]), used) | |
| if len(checks) < a.verify_decoder and src == "local": | |
| d1 = decode_clip(job); ref = verify_with_decord(job) | |
| checks.append(dict(clip_id=c, max_abs_diff=int(np.abs(d1["crops"].astype(np.int16) - ref.astype(np.int16)).max()), n_grab=int(d1["n_grab"]))) | |
| pending[pool.submit(decode_clip, job)] = (c, src, leaf) | |
| drain(pending, False) | |
| if src == "remote_download" and not a.keep_scratch: | |
| drain(pending, True) # finish every in-flight decode (incl. this leaf's) before removing its video | |
| shutil.rmtree(Path(vpath).parent, ignore_errors=True) | |
| if budget_hit: break | |
| drain(pending, True); pool.shutdown(wait=True) | |
| remaining = len(todo) - n_done - len(fails) | |
| fr = sum(r["n_frames"] for r in recs); wall = time.time() - t_start | |
| summ = dict(n_requested=len(ids), n_todo=len(todo), n_done=n_done, n_failed=len(fails), n_remaining=max(0, remaining), budget_hit=budget_hit, | |
| frames=fr, wall_s=round(wall, 1), decode_workers=a.decode_workers, decoder_check=checks) | |
| if recs: | |
| inf = sum(r["infer_s"] for r in recs); dec = sum(r["decode_s"] for r in recs); grab = sum(r["n_grab"] for r in recs); sav = sum(r["save_s"] for r in recs) | |
| wall_no_dl = wall - stats["download_secs"]; per_clip_wall = wall_no_dl / len(recs) | |
| by_src = {} | |
| for s in sorted({r["video_source"] for r in recs}): | |
| rr = [r for r in recs if r["video_source"] == s]; f2 = sum(r["n_frames"] for r in rr) | |
| by_src[s] = dict(n_clips=len(rr), infer_ms_per_frame=round(1000 * sum(r["infer_s"] for r in rr) / f2, 3), | |
| worker_decode_fps=round(sum(r["n_grab"] for r in rr) / max(sum(r["decode_s"] for r in rr), 1e-9), 1), | |
| worker_decode_s_per_clip=round(sum(r["decode_s"] for r in rr) / len(rr), 3), | |
| det_rate_body_mean=round(float(np.mean([r["det_rate_body"] for r in rr])), 4), mean_conf_body=round(float(np.mean([r["mean_conf_body"] for r in rr])), 4)) | |
| N_ALL = 26578 | |
| mbps = stats["download_bytes"] / 1e6 / stats["download_secs"] if stats["download_secs"] else None | |
| man_leaves = {leaf_rel(v) for v in cm.values()}; remote_leaves = [l for l in man_leaves if not is_local(l)] | |
| rem_bytes = sum((leaves["leaves"].get(l) or {}).get("dcamera_bytes") or 0 for l in remote_leaves) | |
| all_bytes = sum((leaves["leaves"].get(l) or {}).get("dcamera_bytes") or 0 for l in man_leaves) | |
| dl_h = (rem_bytes / 1e6 / mbps / 3600) if mbps else 0.0 | |
| proj = dict(n_clips=N_ALL, frames=N_ALL * NK, gpu_h_infer_only=round(N_ALL * NK * (inf / fr) / 3600, 2), | |
| cpu_decode_h_single_stream=round(N_ALL * (dec / len(recs)) / 3600, 2), | |
| wall_h_pipeline_no_download=round(N_ALL * per_clip_wall / 3600, 2), | |
| n_remote_only_leaves=len(remote_leaves), remote_only_dcamera_GB=round(rem_bytes / 1e9, 2), | |
| download_h_remote_only=(round(dl_h, 2) if mbps else None), download_h_if_all_487_leaves_remote=(round(all_bytes / 1e6 / mbps / 3600, 2) if mbps else None), | |
| wall_h_total=round(N_ALL * per_clip_wall / 3600 + dl_h, 2), n_20min_chunks=int(math.ceil((N_ALL * per_clip_wall / 3600 + dl_h) * 3))) | |
| summ.update(infer_ms_per_frame=round(1000 * inf / fr, 3), worker_decode_fps_per_worker=round(grab / max(dec, 1e-9), 1), frames_grabbed_per_clip=round(grab / len(recs), 1), | |
| aggregate_decode_fps=round(grab / max(wall_no_dl, 1e-9), 1), save_ms_per_clip=round(1000 * sav / len(recs), 1), | |
| per_clip_wall_s=round(per_clip_wall, 3), clips_per_s=round(len(recs) / max(wall_no_dl, 1e-9), 3), by_video_source=by_src, | |
| download=dict(n_leaves=stats["n_downloads"], bytes=stats["download_bytes"], secs=round(stats["download_secs"], 1), MBps=(round(mbps, 1) if mbps else None), leaves=stats["download_leaves"]), | |
| projection_26578=proj) | |
| if a.bench_out: | |
| Path(a.bench_out).write_text(json.dumps(dict(utc=time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), model_id=MODEL_ID, margin=MARGIN, score_th=SCORE_TH, summary=summ, fails=fails, clips=recs), indent=1)) | |
| write_run_json(OUT, "extract_pose_causal", [__file__], [LEAVES] + ([Path(a.clips)] if a.clips else []), None, t_start, extra={"summary": summ, "n_fails": len(fails), "fails_head": fails[:20]}) | |
| brief = {k: summ[k] for k in ("n_done", "n_failed", "n_remaining", "wall_s")} | |
| if recs: brief.update(infer_ms_per_frame=summ["infer_ms_per_frame"], clips_per_s=summ["clips_per_s"], aggregate_decode_fps=summ["aggregate_decode_fps"]) | |
| print(("BUDGET_EXIT " if budget_hit else "ALL_DONE ") + json.dumps(brief), flush=True) | |
| sys.exit(3 if (budget_hit or remaining > 0) else 0) | |
| if __name__ == "__main__": | |
| main() | |
Xet Storage Details
- Size:
- 23.4 kB
- Xet hash:
- da958062e4dd42aa31d19d4bb706ba26b9e065b72793b4a664d458659cd2ef66
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.