VAR-RFDETR: RF-DETR-Seg large detector with box/segment toggle

Dockerfile

@@ -0,0 +1,53 @@
+# A100 / dedicated-GPU Hugging Face Space (Docker SDK).
+# Models load once at boot and stay warm; pause the Space to stop billing.
+
+FROM pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime
+
+ENV DEBIAN_FRONTEND=noninteractive \
+    PYOPENGL_PLATFORM=egl \
+    PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+    SAM3D_DIR=/app/sam-3d-body \
+    HF_HOME=/app/.cache/huggingface
+
+# System libs for OpenCV / pyrender / video decoding, plus a C++ toolchain.
+# build-essential is required because the runtime base ships no compiler and
+# detectron2 (a1ce2f9) builds its ops from source against the installed torch.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    git ffmpeg libgl1 libglib2.0-0 libosmesa6 libegl1 \
+    build-essential ninja-build \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+# Clone the model repo (added to sys.path by pipeline/gpu.py via SAM3D_DIR)
+RUN git clone https://github.com/facebookresearch/sam-3d-body.git /app/sam-3d-body
+
+# Python deps. detectron2 is pinned to a1ce2f9 and built --no-build-isolation
+# --no-deps so it compiles against the torch already in the base image — the
+# single most fragile step; the pinned base keeps it reproducible.
+COPY requirements.txt /app/requirements.txt
+RUN pip install --no-cache-dir -r /app/requirements.txt \
+    && pip install --no-cache-dir \
+       "git+https://github.com/facebookresearch/detectron2.git@a1ce2f9" --no-build-isolation --no-deps \
+    && pip install --no-cache-dir "git+https://github.com/microsoft/MoGe.git"
+
+# Runtime cache redirection. HF Spaces runs as a non-root user with HOME=/, so
+# anything writing to ~/.cache (torch.hub clones DINOv3 here at model load,
+# matplotlib, fontconfig, ...) hits a read-only /.cache. These are only needed at
+# runtime, so they sit BELOW the pip layer to keep that expensive layer cached.
+ENV HOME=/app \
+    XDG_CACHE_HOME=/app/.cache \
+    TORCH_HOME=/app/.cache/torch \
+    MPLCONFIGDIR=/app/.cache/matplotlib
+
+# App (UI + the pipeline package; GPU code is isolated in pipeline/gpu.py)
+COPY app.py /app/app.py
+COPY pipeline /app/pipeline
+
+# Writable cache + home (HF Spaces runs as a non-root user). torch.hub needs to
+# clone into TORCH_HOME at runtime, so the whole /app tree must be writable.
+RUN mkdir -p /app/.cache/huggingface /app/.cache/torch /app/.cache/matplotlib \
+    && chmod -R 777 /app
+
+EXPOSE 7860
+CMD ["python", "app.py"]

README.md

@@ -1,10 +1,73 @@
 ---
-title: VAR RFDETR
-emoji: 📈
-colorFrom: blue
-colorTo: purple
+title: VAR - RFDETR
+emoji: 🥅
+colorFrom: purple
+colorTo: blue
 sdk: docker
+app_port: 7860
 pinned: false
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# VAR - RFDETR — Offside Visualizer
+
+RF-DETR variant: detection uses Roboflow's **RF-DETR-Seg (large)** with a toggle
+between bounding boxes and segments. Upload a match clip, scrub to the moment the
+ball is played, reconstruct selected players in 3D with SAM 3D Body, and place
+them on a virtual pitch with a draggable
+offside plane.
+
+## Pipeline
+
+1. **Upload** a video clip.
+2. **Scrub** to the offside frame (slider + prev/next, frames seeked on demand).
+3. **Detect** players on that frame — the only GPU step, cached per (video, frame, threshold).
+4. **Select** the players to analyze and mark the defenders (incl. GK).
+5. **Click two goal-parallel lines** (4 points) on the detected frame to fix the offside axis.
+6. **Build** the 3D scene; drag the offside plane and read the OFFSIDE / NO-OFFSIDE verdict.
+
+The GPU runs once per frame (`pipeline/gpu.py`). Scrubbing, line geometry,
+placement, plotting, and the draggable plane are all CPU on the cached result.
+
+## Code layout
+
+```
+app.py            Gradio UI + event wiring (CPU)
+pipeline/
+  video.py        frame seek/probe (CPU)
+  gpu.py          model load + reconstruct_frame  ← the ONLY GPU code
+  geometry.py     vanishing point, ground fit, field frame, scene (CPU)
+  overlay.py      detection boxes + line-click drawing (CPU)
+```
+
+Isolating the GPU in `pipeline/gpu.py` means moving inference to a serverless
+backend (Modal / ZeroGPU) later only touches `reconstruct_frame`.
+
+## Deploy
+
+This is a **Docker SDK** Space for **dedicated GPU hardware** (A100 recommended):
+
+1. Set hardware to an A100 tier.
+2. Add a secret `HF_TOKEN` — a read token for an account with approved access to
+   the gated **`facebook/sam-3d-body-dinov3`**. (Override the repo with the
+   `SAM3D_REPO_ID` env var if you use a different checkpoint.)
+3. First boot builds the image and downloads ~7 GB of weights — give it time.
+   After that, the model stays warm until you pause the Space.
+
+**Cost control:** dedicated GPU bills while the Space is running, with no
+auto-shutoff. Pause the Space from its settings when you are not using it.
+
+### Why not ZeroGPU?
+
+ZeroGPU allocates the GPU per call, caps call duration, enforces a daily quota,
+and cold-loads the ~7 GB model stack on each allocation — a poor fit for an
+interactive video-scrubbing session, and it requires the Gradio SDK (not Docker).
+
+## Notes / limits
+
+- Scale comes from the reconstructed body height, so positions are approximate
+  metres — good for relative offside ordering, **not** sub-10 cm officiating calls.
+  The verdict surfaces a "too close to call" band rather than implying false precision.
+- The offside point currently uses the forward-most body vertex **including arms**;
+  excluding arms (via MHR body-part labels) is planned — see `TODO.md`.
+- "Find the offside moment" is manual scrubbing; automatic pass-instant detection
+  (ball tracking) is future work.

TODO.md

@@ -0,0 +1,21 @@
+# Parked — explicitly out of scope for now (SPEC §7)
+
+These are deliberately NOT built yet. Listed so they aren't lost.
+
+- **Automatic pass-instant detection** — needs ball tracking to replace manual
+  frame scrubbing.
+- **Exclude arms from the forward-most point** — needs MHR body-part vertex
+  labels; today `build_scene`/`offside_plane_x` use the forward-most body vertex
+  including arms.
+- **three.js broadcast frontend** (fog/bloom/camera moves) replacing Plotly.
+- **Team identification by jersey colour** — currently the user labels defenders
+  manually.
+- **Multi-frame tracking.**
+
+# Known nuances carried from the notebook
+
+- `flip up` toggle exists because the ground-normal sign can invert; if median
+  player height < 1.0 m the scene warns to toggle it.
+- The offside line uses the **2nd-last** defender's forward-most point (last is
+  usually the GK). With only one labeled defender it falls back to that defender;
+  with none, to the scene's median X (then drag the plane).

app.py

@@ -0,0 +1,324 @@
+"""
+VAR-style offside visualizer — Gradio app.
+
+Flow:
+  upload -> scrub to the frame -> draw 2 goal-parallel lines on that frame
+  -> Detect players (GPU, once/frame) -> click players to select
+  -> mark defenders -> Build the 3D scene with a draggable offside plane.
+
+The GPU is touched ONLY inside pipeline.gpu (detect_frame / reconstruct_selected);
+every other callback here runs on cached numpy and stays on the CPU.
+"""
+
+import os
+
+# headless GL + CUDA fragmentation hygiene must be set before any heavy import
+os.environ.setdefault("PYOPENGL_PLATFORM", "egl")
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import numpy as np
+import gradio as gr
+
+from pipeline.video import probe_video, grab_frame
+from pipeline.overlay import (annotate_detections, draw_lines, draw_masks,
+                              pick_box, pick_mask)
+from pipeline.autolines import propose_lines
+from pipeline import geometry as G
+
+SEG_STYLE = "Segments"
+
+
+def _render(frame, people, selected, style):
+    """Draw detections in the chosen style (boxes vs segments)."""
+    if style == SEG_STYLE:
+        return draw_masks(frame, people, selected)
+    return annotate_detections(frame, people, selected)
+
+
+def _pick(people, x, y, style):
+    return pick_mask(people, x, y) if style == SEG_STYLE else pick_box(people, x, y)
+
+
+# ============================================================================
+# Stage (a): upload + frame scrubbing  (pure CPU)
+# ============================================================================
+def on_upload(video_path):
+    """New clip: size the slider, show frame 0, reset lines/frame state."""
+    if not video_path:
+        return (gr.update(maximum=1, value=0), None, "Upload a clip to begin.",
+                0, None, [], "")
+    n, fps = probe_video(video_path)
+    n_max = max(n - 1, 0)
+    frame = grab_frame(video_path, 0)
+    return (
+        gr.update(maximum=max(n_max, 1), value=0),
+        frame,
+        f"{n} frames @ {fps:.1f} fps — scrub to the moment the ball is played, "
+        "then auto-detect or click the 2 goal-parallel lines.",
+        n_max,
+        frame,        # st_frame: clean copy for line redraws
+        [],           # st_lines reset
+        "",           # line_status reset
+        [],           # st_families reset
+        0,            # st_fam_idx reset
+    )
+
+
+def on_scrub(video_path, idx):
+    """Show the new frame and reset any half-drawn / proposed lines on it."""
+    if not video_path:
+        return None, None, [], "", [], 0
+    frame = grab_frame(video_path, int(idx))
+    return frame, frame, [], "", [], 0
+
+
+def step_frame(idx, delta, n_max):
+    return max(0, min(int(n_max), int(idx) + delta))
+
+
+# ============================================================================
+# Stage (b): draw 2 goal-parallel lines on the scrubbed frame  (CPU)
+# ============================================================================
+def on_line_click(line_pts, clean_frame, evt: gr.SelectData):
+    """Collect 4 clicks = 2 lines; always redraw from the clean frame (no drift).
+
+    A manual click clears any auto-proposed families (so Flip stops applying).
+    """
+    pts = list(line_pts) if line_pts else []
+    if len(pts) >= 4:
+        pts = []
+    pts.append([float(evt.index[0]), float(evt.index[1])])
+    img = draw_lines(clean_frame, pts)
+    status = {1: "line 1: 1/2", 2: "line 1 set", 3: "line 2: 1/2",
+              4: "both lines set ✓ — now Detect players"}.get(len(pts), "")
+    return pts, img, status, [], 0
+
+
+def on_auto_lines(clean_frame):
+    """Propose the 2 goal-parallel lines from detected pitch lines."""
+    if clean_frame is None:
+        return [], None, "Scrub to a frame first.", [], 0
+    fams = propose_lines(clean_frame)
+    if not fams:
+        return [], gr.update(), ("No clear pitch lines found — draw the 2 "
+                                 "goal-parallel lines by hand."), [], 0
+    pts = fams[0]["pts"]
+    img = draw_lines(clean_frame, pts)
+    status = (f"Proposed goal-parallel lines (direction 1/{len(fams)}). If the "
+              "offside axis looks wrong, click **Flip line direction**, or just "
+              "click the frame to redraw by hand.")
+    return pts, img, status, fams, 0
+
+
+def on_flip_lines(families, fam_idx, clean_frame):
+    """Switch the proposal to the other detected line-family."""
+    if not families:
+        return gr.update(), gr.update(), "Run **Auto-detect lines** first.", gr.update()
+    if len(families) < 2:
+        return (families[0]["pts"], draw_lines(clean_frame, families[0]["pts"]),
+                "Only one line direction was found — redraw by hand if it's wrong.", 0)
+    new_idx = (int(fam_idx) + 1) % len(families)
+    pts = families[new_idx]["pts"]
+    return (pts, draw_lines(clean_frame, pts),
+            f"Line direction {new_idx + 1}/{len(families)}.", new_idx)
+
+
+# ============================================================================
+# Stage (c): detect (GPU, cached) + show boxes  (CPU after the call)
+# ============================================================================
+def on_detect(video_path, idx, conf, style):
+    """Detect step: RF-DETR-Seg only (boxes + masks). No mesh reconstruction here."""
+    from pipeline.gpu import detect_frame
+    if not video_path:
+        return None, [], [], "Upload a clip first.", gr.update(choices=[], value=[])
+    people = detect_frame(video_path, idx, conf)
+    if not people:
+        return (None, [], [], "No players detected — lower the confidence slider.",
+                gr.update(choices=[], value=[]))
+    annotated = _render(grab_frame(video_path, idx), people, [], style)
+    unit = "silhouette" if style == SEG_STYLE else "box"
+    msg = (f"Detected {len(people)} players. Click a player's {unit} to select "
+           "(click again to deselect).")
+    return annotated, people, [], msg, gr.update(choices=[], value=[])
+
+
+def on_restyle(people, selected, clean_frame, style):
+    """Re-render the current detection when the box/segment toggle changes."""
+    if not people or clean_frame is None:
+        return gr.update()
+    return _render(clean_frame, people, selected or [], style)
+
+
+# ============================================================================
+# Stage (d): click players to select  (CPU)
+# ============================================================================
+def on_select_player(people, selected, clean_frame, cur_def, style, evt: gr.SelectData):
+    """Toggle the player clicked; refresh highlight + defender choices (box or segment)."""
+    if not people:
+        return None, selected or [], "Detect players first.", gr.update()
+    hit = _pick(people, evt.index[0], evt.index[1], style)
+    sel = list(selected or [])
+    if hit is not None:
+        sel.remove(hit) if hit in sel else sel.append(hit)
+    sel = sorted(sel)
+    img = _render(clean_frame, people, sel, style)
+    msg = (f"Selected players: {sel}. Mark defenders below, then Build."
+           if sel else "Click a player to select.")
+    keep_def = [d for d in (cur_def or []) if d in sel]
+    return img, sel, msg, gr.update(choices=sel, value=keep_def)
+
+
+# ============================================================================
+# Stage (e)+(f): place players + build the Plotly scene with a draggable plane
+# ============================================================================
+def on_build(video_path, idx, people_det, selected_ids, line_pts, flip_up,
+             attack_dir, defender_ids):
+    from pipeline.gpu import reconstruct_selected, get_faces
+    if not selected_ids:
+        return None, "Click at least one player to select.", gr.update(), None, +1, [], {}
+    if not line_pts or len(line_pts) < 4:
+        return None, "Draw 2 goal-parallel lines (4 points) on the frame first.", \
+               gr.update(), None, +1, [], {}
+
+    # Reconstruct ONLY the selected players' boxes (the heavy GPU step).
+    selected_ids = sorted(int(i) for i in selected_ids)
+    boxes = [people_det[i]["bbox"] for i in selected_ids]
+    recon = reconstruct_selected(video_path, idx, boxes)
+    if not recon:
+        return None, "Reconstruction returned no meshes.", gr.update(), None, +1, [], {}
+    # Key meshes back to their original detection ids (recon order == boxes order).
+    people = {selected_ids[k]: recon[k] for k in range(len(recon))}
+    faces = get_faces()
+    h, w = grab_frame(video_path, idx).shape[:2]
+    focal = people[selected_ids[0]]["focal_length"]
+    gdir = G.goal_dir_from_lines(line_pts, focal, w, h)
+
+    placed = G.place_players(people, selected_ids, gdir, flip_up=flip_up)
+    med = float(np.median([placed[i][:, 2].max() for i in placed]))
+    masks = {i: G.non_arm_mask(people[i]) for i in selected_ids}  # exclude arms/hands
+
+    attack_sign = +1 if attack_dir == "toward +X" else -1
+    dset = [int(d) for d in (defender_ids or [])]
+    plane_x = G.offside_plane_x(placed, attack_sign, dset, masks)
+    fig = G.build_scene(placed, faces, plane_x, attack_sign, dset, masks)
+
+    allX = np.vstack(list(placed.values()))[:, 0]
+    x0, x1 = float(allX.min() - 4), float(allX.max() + 4)
+    plane_update = gr.update(minimum=x0, maximum=x1, value=float(plane_x),
+                             visible=True, label="Drag the offside plane (X, m)")
+
+    warn = "  ⚠ heights look wrong — toggle 'flip up'." if med < 1.0 else ""
+    return (fig, f"Median player height {med:.2f} m (expect ~1.7–1.9).{warn}",
+            plane_update, placed, attack_sign, dset, masks)
+
+
+def on_plane(placed, plane_x, attack_sign, defender_ids, masks):
+    """Re-render the scene at a new plane X — pure CPU on the cached placement."""
+    from pipeline.gpu import get_faces
+    if not placed:
+        return gr.update()
+    return G.build_scene(placed, get_faces(), float(plane_x),
+                         int(attack_sign), defender_ids or [], masks)
+
+
+# ============================================================================
+# UI
+# ============================================================================
+with gr.Blocks(title="VAR Offside Visualizer") as demo:
+    gr.Markdown(
+        "## VAR-style Offside Visualizer\n"
+        "Upload → scrub → **click 2 goal-parallel lines** → Detect → "
+        "**click players to select** → mark defenders → Build.\n\n"
+        "_Scale comes from reconstructed body height, so positions are approximate "
+        "metres — good for relative offside ordering, not sub-10 cm calls._"
+    )
+
+    # session state
+    st_nmax = gr.State(0)        # last valid frame index
+    st_frame = gr.State(None)    # clean RGB of the current frame
+    st_people = gr.State([])     # slim detections
+    st_lines = gr.State([])      # clicked / proposed line points
+    st_families = gr.State([])   # auto-proposed line families
+    st_fam_idx = gr.State(0)     # which proposed family is active
+    st_selected = gr.State([])   # player ids selected by clicking
+    st_placed = gr.State(None)   # placed meshes after build
+    st_attack = gr.State(+1)
+    st_defenders = gr.State([])
+    st_masks = gr.State({})      # per-player non-arm vertex masks
+
+    video = gr.Video(label="1. Upload match clip")
+    status = gr.Markdown()
+
+    with gr.Row():
+        frame_slider = gr.Slider(0, 1, value=0, step=1,
+                                 label="2. Scrub to the offside frame")
+    with gr.Row():
+        prev_btn = gr.Button("◀ prev frame")
+        next_btn = gr.Button("next frame ▶")
+
+    # Stage (b): lines drawn on the scrubbed frame — auto-proposed or by hand
+    frame_view = gr.Image(label="3. Goal-parallel lines: auto-detect or click 4 points",
+                          interactive=True)
+    with gr.Row():
+        auto_lines_btn = gr.Button("✨ Auto-detect lines")
+        flip_lines_btn = gr.Button("↔ Flip line direction")
+    line_status = gr.Markdown()
+
+    # Stage (c)/(d): detect (RF-DETR-Seg), then click to select
+    with gr.Row():
+        thr = gr.Slider(0.0, 0.95, value=0.4, step=0.05, label="Detection confidence")
+        style = gr.Radio(["Bounding boxes", "Segments"], value="Bounding boxes",
+                         label="Detection style")
+        detect_btn = gr.Button("4. Detect players (GPU)", variant="primary")
+    detect_view = gr.Image(label="5. Click a player to select (click again to deselect)",
+                           interactive=True)
+    select_status = gr.Markdown()
+
+    # Stage (e)/(f)
+    defenders = gr.CheckboxGroup(choices=[], label="6. Defenders (incl. GK) — sets the offside line")
+    with gr.Row():
+        flip = gr.Checkbox(False, label="flip up (if players are upside-down)")
+        attack = gr.Radio(["toward +X", "toward -X"], value="toward +X",
+                          label="Attacking direction")
+    build_btn = gr.Button("7. Build 3D scene + offside line", variant="primary")
+
+    scene = gr.Plot(label="3D scene")
+    plane_slider = gr.Slider(-10, 10, value=0, step=0.05, visible=False,
+                             label="Drag the offside plane (X, m)")
+    build_status = gr.Markdown()
+
+    # --- wiring ---
+    video.change(on_upload, [video],
+                 [frame_slider, frame_view, status, st_nmax, st_frame, st_lines,
+                  line_status, st_families, st_fam_idx])
+    frame_slider.change(on_scrub, [video, frame_slider],
+                        [frame_view, st_frame, st_lines, line_status,
+                         st_families, st_fam_idx])
+    prev_btn.click(lambda i, m: step_frame(i, -1, m), [frame_slider, st_nmax], [frame_slider])
+    next_btn.click(lambda i, m: step_frame(i, +1, m), [frame_slider, st_nmax], [frame_slider])
+
+    frame_view.select(on_line_click, [st_lines, st_frame],
+                      [st_lines, frame_view, line_status, st_families, st_fam_idx])
+    auto_lines_btn.click(on_auto_lines, [st_frame],
+                         [st_lines, frame_view, line_status, st_families, st_fam_idx])
+    flip_lines_btn.click(on_flip_lines, [st_families, st_fam_idx, st_frame],
+                         [st_lines, frame_view, line_status, st_fam_idx])
+
+    detect_btn.click(on_detect, [video, frame_slider, thr, style],
+                     [detect_view, st_people, st_selected, select_status, defenders])
+    detect_view.select(on_select_player,
+                       [st_people, st_selected, st_frame, defenders, style],
+                       [detect_view, st_selected, select_status, defenders])
+    style.change(on_restyle, [st_people, st_selected, st_frame, style], [detect_view])
+
+    build_btn.click(
+        on_build,
+        [video, frame_slider, st_people, st_selected, st_lines, flip, attack, defenders],
+        [scene, build_status, plane_slider, st_placed, st_attack, st_defenders, st_masks])
+    plane_slider.change(on_plane,
+                        [st_placed, plane_slider, st_attack, st_defenders, st_masks],
+                        [scene])
+
+
+if __name__ == "__main__":
+    demo.queue().launch(server_name="0.0.0.0", server_port=7860)

pipeline/__init__.py

@@ -0,0 +1,8 @@
+"""VAR offside visualizer pipeline.
+
+Module layout keeps the GPU strictly isolated:
+  video.py     — frame seek/probe (CPU)
+  gpu.py       — model load + reconstruct_frame (the ONLY GPU code)
+  geometry.py  — vanishing point, ground fit, field frame, scene (CPU)
+  overlay.py   — cv2 detection boxes + line-click drawing (CPU)
+"""

pipeline/autolines.py

@@ -0,0 +1,165 @@
+"""
+Auto-PROPOSE the two goal-parallel lines (CPU, OpenCV only).
+
+Fully-automatic goal-direction detection is unreliable on broadcast frames
+(a full pitch homography can score "confident" yet be degenerate), so this only
+*proposes*: it detects the white pitch lines, clusters them by vanishing point
+into the two pitch line-families, and returns the 2 longest lines of each. The UI
+shows one family, lets the user flip to the other, or redraw by hand.
+
+Line detection (field/white masks + Hough merge) is ported from the project's
+backend pitch-registration code, which is already proven on this footage.
+"""
+
+import numpy as np
+import cv2
+
+
+# ----------------------------- masks -------------------------------------- #
+def field_mask(bgr):
+    hsv = cv2.cvtColor(bgr, cv2.COLOR_BGR2HSV)
+    h, s, v = hsv[..., 0], hsv[..., 1], hsv[..., 2]
+    green = ((h > 30) & (h < 95) & (s > 40) & (v > 40)).astype(np.uint8) * 255
+    green = cv2.morphologyEx(green, cv2.MORPH_CLOSE, np.ones((25, 25), np.uint8))
+    green = cv2.morphologyEx(green, cv2.MORPH_OPEN, np.ones((9, 9), np.uint8))
+    cnts, _ = cv2.findContours(green, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+    mask = np.zeros_like(green)
+    if cnts:
+        cv2.drawContours(mask, [max(cnts, key=cv2.contourArea)], -1, 255, -1)
+    return mask
+
+
+def white_mask(bgr, fmask):
+    hsv = cv2.cvtColor(bgr, cv2.COLOR_BGR2HSV)
+    s, v = hsv[..., 1], hsv[..., 2]
+    w = ((v > 150) & (s < 60)).astype(np.uint8) * 255
+    w = cv2.bitwise_and(w, fmask)
+    return cv2.morphologyEx(w, cv2.MORPH_OPEN, np.ones((2, 2), np.uint8))
+
+
+# ----------------------------- line tools --------------------------------- #
+def _seg_angle(seg):
+    x1, y1, x2, y2 = seg
+    return np.degrees(np.arctan2(y2 - y1, x2 - x1)) % 180.0
+
+
+def _line_from_seg(seg):
+    x1, y1, x2, y2 = seg
+    l = np.cross([x1, y1, 1.0], [x2, y2, 1.0])
+    return l / (np.hypot(l[0], l[1]) or 1.0)
+
+
+def _intersect(l1, l2):
+    p = np.cross(l1, l2)
+    return None if abs(p[2]) < 1e-9 else np.array([p[0] / p[2], p[1] / p[2]])
+
+
+def merge_lines(wmask):
+    """Cluster Hough segments into merged pitch lines (angle + signed distance)."""
+    segs = cv2.HoughLinesP(wmask, 1, np.pi / 180, threshold=70,
+                           minLineLength=70, maxLineGap=30)
+    if segs is None:
+        return []
+    segs = segs[:, 0]
+    used = np.zeros(len(segs), bool)
+    merged = []
+    for i in range(len(segs)):
+        if used[i]:
+            continue
+        ai, li = _seg_angle(segs[i]), _line_from_seg(segs[i])
+        group = [segs[i]]; used[i] = True
+        for j in range(i + 1, len(segs)):
+            if used[j]:
+                continue
+            da = abs(ai - _seg_angle(segs[j]))
+            if min(da, 180 - da) > 6:
+                continue
+            mid = np.array([(segs[j][0] + segs[j][2]) / 2,
+                            (segs[j][1] + segs[j][3]) / 2, 1.0])
+            if abs(li @ mid) < 14:
+                group.append(segs[j]); used[j] = True
+        pts = np.array([[s[0], s[1]] for s in group] +
+                       [[s[2], s[3]] for s in group], dtype=np.float32)
+        vx, vy, x0, y0 = cv2.fitLine(pts, cv2.DIST_L2, 0, 0.01, 0.01).ravel()
+        l = _line_from_seg((x0, y0, x0 + vx, y0 + vy))
+        proj = (pts - [x0, y0]) @ np.array([vx, vy])
+        a = np.array([x0, y0]) + proj.min() * np.array([vx, vy])
+        b = np.array([x0, y0]) + proj.max() * np.array([vx, vy])
+        merged.append(dict(line=l, a=a, b=b, length=float(np.hypot(*(b - a)))))
+    return merged
+
+
+# ----------------------- vanishing-point families ------------------------- #
+def _line_dir(l):
+    d = np.array([-l[1], l[0]])
+    return d / (np.linalg.norm(d) + 1e-9)
+
+
+def _vp_inliers(lines, vp, thr_deg=2.5):
+    inl = []
+    for k, m in enumerate(lines):
+        mid = (m["a"] + m["b"]) / 2.0
+        to_vp = vp - mid
+        n = np.linalg.norm(to_vp)
+        if n < 1e-6:
+            inl.append(k); continue
+        cosang = abs(_line_dir(m["line"]) @ (to_vp / n))
+        if np.degrees(np.arccos(min(cosang, 1.0))) < thr_deg:
+            inl.append(k)
+    return inl
+
+
+def _best_family(lines):
+    """VP supported by the most total line-length; returns (inlier_idxs, vp)."""
+    best = None
+    for i in range(len(lines)):
+        for j in range(i + 1, len(lines)):
+            vp = _intersect(lines[i]["line"], lines[j]["line"])
+            if vp is None:
+                continue
+            inl = _vp_inliers(lines, vp)
+            score = sum(lines[k]["length"] for k in inl)
+            if best is None or score > best[0]:
+                best = (score, inl, vp)
+    return (best[1], best[2]) if best else (None, None)
+
+
+def _family_points(fam):
+    """4 points (2 longest lines, 2 endpoints each): [l1a, l1b, l2a, l2b]."""
+    fam = sorted(fam, key=lambda m: -m["length"])[:2]
+    pts = []
+    for m in fam:
+        pts += [[float(m["a"][0]), float(m["a"][1])],
+                [float(m["b"][0]), float(m["b"][1])]]
+    return pts
+
+
+def propose_lines(frame_rgb):
+    """Return up to 2 families, each as {"pts": [[x,y]*4], "vp": [x,y], "n": int}.
+
+    families[0] is the larger (by total length); the goal-parallel one may be
+    either — the UI lets the user flip. Empty list if too few lines are found.
+    """
+    bgr = cv2.cvtColor(frame_rgb, cv2.COLOR_RGB2BGR)
+    merged = merge_lines(white_mask(bgr, field_mask(bgr)))
+    longl = sorted([m for m in merged if m["length"] > 80],
+                   key=lambda m: -m["length"])[:14]
+    if len(longl) < 2:
+        return []
+
+    fams = []
+    inl, vp = _best_family(longl)
+    if inl is None:
+        return []
+    fam1 = [longl[k] for k in inl]
+    fams.append({"pts": _family_points(fam1), "vp": vp.tolist(), "n": len(fam1)})
+
+    rest = [longl[k] for k in range(len(longl)) if k not in inl]
+    if len(rest) >= 2:
+        inl2, vp2 = _best_family(rest)
+        if inl2 is not None:
+            fam2 = [rest[k] for k in inl2]
+            if len(fam2) >= 2:
+                fams.append({"pts": _family_points(fam2),
+                             "vp": vp2.tolist(), "n": len(fam2)})
+    return fams

pipeline/geometry.py

@@ -0,0 +1,182 @@
+"""
+Placement geometry — pure CPU, copied verbatim from the verified Colab notebook
+(sections 4 & the placement cell). Do NOT rewrite this math:
+
+  world_verts = pred_vertices + pred_cam_t            (shared camera frame)
+  goal dir    = back-projected vanishing point of two clicked parallel lines
+  ground      = SVD on the lowest ~3% feet vertices (camera Y is down)
+  field frame = ez=up, ey=goal dir in-plane, ex=ey x ez (the offside axis)
+  per player  : drop each player's own feet to Z=0 (not one global shift)
+  offside     : constant-X plane, parallel to the goal line by construction
+"""
+
+import numpy as np
+import plotly.graph_objects as go
+
+
+def goal_dir_from_lines(line_pts, focal, w, h):
+    """Two clicked goal-parallel lines -> goal-line direction in camera coords."""
+    def homog_line(p, q):
+        return np.cross([p[0], p[1], 1.0], [q[0], q[1], 1.0])
+    lp = np.asarray(line_pts, dtype=float)
+    l1 = homog_line(lp[0], lp[1])
+    l2 = homog_line(lp[2], lp[3])
+    vp = np.cross(l1, l2)
+    K = np.array([[focal, 0, w / 2], [0, focal, h / 2], [0, 0, 1.0]])
+    d_img = (np.array([vp[0], vp[1], 0.0]) if abs(vp[2]) < 1e-9
+             else np.array([vp[0] / vp[2], vp[1] / vp[2], 1.0]))
+    g = np.linalg.inv(K) @ d_img
+    return g / np.linalg.norm(g)
+
+
+def world_verts(p):
+    """Vertices in the shared camera frame = local mesh + per-player cam translation."""
+    return np.asarray(p["pred_vertices"]) + np.asarray(p["pred_cam_t"]).reshape(1, 3)
+
+
+# MHR-70 arm/hand joints: elbows, wrists, and every finger joint. The offside law
+# ignores arms/hands, so vertices nearest these joints are excluded from the
+# forward-most point used for both the line and the verdict.
+_ARM_KP = frozenset({7, 8, 41, 62} | set(range(21, 41)) | set(range(42, 62)))
+
+
+def non_arm_mask(p):
+    """Boolean per-vertex mask, True for body (keep), False for arm/hand vertices.
+
+    Labels each vertex by its nearest keypoint. Falls back to all-True (no
+    exclusion) if keypoints are missing or malformed.
+    """
+    V = np.asarray(p["pred_vertices"])
+    kp = p.get("pred_keypoints_3d")
+    if kp is None:
+        return np.ones(len(V), bool)
+    kp = np.asarray(kp)
+    if kp.ndim != 2 or kp.shape[0] < 15:
+        return np.ones(len(V), bool)
+    kp = kp[:, :3]
+    arm = [i for i in _ARM_KP if i < len(kp)]
+    keep = [i for i in range(len(kp)) if i not in _ARM_KP]
+    if not arm or not keep:
+        return np.ones(len(V), bool)
+    d_arm = np.linalg.norm(V[:, None, :] - kp[arm][None], axis=2).min(1)
+    d_keep = np.linalg.norm(V[:, None, :] - kp[keep][None], axis=2).min(1)
+    return d_keep <= d_arm
+
+
+def _forward_most(Vf, attack_sign, mask=None):
+    """Furthest-forward X (in attack direction) over non-arm vertices."""
+    sel = Vf[mask] if (mask is not None and mask.any()) else Vf
+    return float((attack_sign * sel[:, 0]).max())
+
+
+def place_players(people, selected_ids, goal_dir_cam, flip_up=False):
+    """Place selected meshes on a common field frame: X=offside axis, Y=goal line, Z=up."""
+    def feet_pts(Vc, frac=0.03):
+        thr = np.quantile(Vc[:, 1], 1 - frac)   # camera Y is down -> feet = largest Y
+        return Vc[Vc[:, 1] >= thr]
+
+    def head_pts(Vc, frac=0.03):
+        thr = np.quantile(Vc[:, 1], frac)       # camera Y is down -> head = smallest Y
+        return Vc[Vc[:, 1] <= thr]
+
+    # Up axis = the players' OWN averaged vertical (each player's feet->head),
+    # which is what actually makes them stand upright. A plane fit through the
+    # feet *positions* (the notebook's original approach) tilts whenever feet land
+    # unevenly across a clip, leaving players leaning with one foot pinned at Z=0.
+    ups = []
+    for i in selected_ids:
+        Vc = world_verts(people[i])
+        u = head_pts(Vc).mean(0) - feet_pts(Vc).mean(0)
+        ups.append(u / (np.linalg.norm(u) + 1e-9))
+    n = np.mean(ups, 0)
+    n /= np.linalg.norm(n) + 1e-9
+    if flip_up:
+        n = -n
+
+    feet_all = np.vstack([feet_pts(world_verts(people[i])) for i in selected_ids])
+    o = feet_all.mean(0)                         # origin at the feet centroid
+
+    g = goal_dir_cam - (goal_dir_cam @ n) * n
+    g /= np.linalg.norm(g)
+    ez, ey = n, g
+    ex = np.cross(ey, ez); ex /= np.linalg.norm(ex)
+    ey = np.cross(ez, ex)
+    Rwf = np.stack([ex, ey, ez], axis=1)
+
+    placed = {i: (world_verts(people[i]) - o) @ Rwf for i in selected_ids}
+    for i in placed:                       # drop each player's feet to Z=0 individually
+        placed[i][:, 2] -= placed[i][:, 2].min()
+    return placed
+
+
+def offside_plane_x(placed, attack_sign, defender_ids, masks=None):
+    """X of the offside line = the furthest-forward point among the MARKED defenders.
+
+    Every marked defender is treated as a candidate "second-last defender": the
+    line is drawn at whichever of them reaches furthest forward (in the attack
+    direction), using their furthest body point with arms/hands excluded. Falls
+    back to the scene's median X if no defenders are labeled.
+    """
+    dset = set(int(d) for d in (defender_ids or []))
+    m = masks or {}
+    dfwd = [_forward_most(placed[i], attack_sign, m.get(i)) for i in placed if i in dset]
+    if dfwd:
+        return attack_sign * max(dfwd)
+    return float(np.median(np.vstack(list(placed.values()))[:, 0]))
+
+
+def build_scene(placed, faces, plane_x, attack_sign, defender_ids, masks=None,
+                too_close=0.30):
+    allP = np.vstack(list(placed.values()))
+    x0, x1 = allP[:, 0].min() - 4, allP[:, 0].max() + 4
+    y0, y1 = allP[:, 1].min() - 4, allP[:, 1].max() + 4
+
+    def fwd(x):
+        return attack_sign * x
+
+    m = masks or {}
+    fmost = {i: _forward_most(placed[i], attack_sign, m.get(i)) for i in placed}
+
+    fig = go.Figure()
+    # pitch
+    fig.add_trace(go.Mesh3d(x=[x0, x1, x1, x0], y=[y0, y0, y1, y1], z=[0, 0, 0, 0],
+                            i=[0, 0], j=[1, 2], k=[2, 3], color="seagreen",
+                            opacity=0.5, showlegend=False))
+    # offside plane
+    if plane_x is not None:
+        fig.add_trace(go.Mesh3d(x=[plane_x] * 4, y=[y0, y1, y1, y0], z=[0, 0, 3, 3],
+                                i=[0, 0], j=[1, 2], k=[2, 3], color="red",
+                                opacity=0.3, showlegend=False))
+    # players
+    palette = ["crimson", "royalblue", "gold", "darkorange", "mediumpurple",
+               "deepskyblue", "hotpink", "mediumspringgreen", "tomato", "slateblue"]
+    dset = set(int(d) for d in (defender_ids or []))
+    any_off = False
+    line_fwd = fwd(plane_x) if plane_x is not None else None
+    for n_, i in enumerate(placed):
+        if i in dset:
+            col, lab = "royalblue", "defender"
+        elif line_fwd is None:
+            col, lab = palette[n_ % len(palette)], ""
+        else:
+            m = fmost[i] - line_fwd
+            if m > too_close:
+                col, lab, any_off = "red", f"OFFSIDE +{m:.2f}m", True
+            elif m < -too_close:
+                col, lab = "seagreen", f"onside {m:.2f}m"
+            else:
+                col, lab = "orange", f"close {m:+.2f}m"
+        V = placed[i]
+        fig.add_trace(go.Mesh3d(x=V[:, 0], y=V[:, 1], z=V[:, 2],
+                                i=faces[:, 0], j=faces[:, 1], k=faces[:, 2],
+                                color=col, opacity=1.0, name=f"#{i} {lab}"))
+
+    title = "VAR — OFFSIDE" if any_off else "VAR — NO OFFSIDE"
+    fig.update_layout(
+        title=dict(text=title, x=0.5, font=dict(size=22, color="white")),
+        paper_bgcolor="#0b1f3a", height=640, margin=dict(l=0, r=0, t=40, b=0),
+        scene=dict(aspectmode="data", bgcolor="#0b1f3a",
+                   xaxis_title="X offside axis (m)", yaxis_title="Y goal line (m)",
+                   zaxis_title="Z (m)",
+                   camera=dict(eye=dict(x=0, y=-2.2, z=1.2), up=dict(x=0, y=0, z=1))))
+    return fig

pipeline/gpu.py

@@ -0,0 +1,133 @@
+"""
+The ONLY module that touches the GPU. Two lazily-loaded models, kept separate so
+each stage pays only for what it needs:
+
+  detect_frame()        -> RF-DETR-Seg (large): boxes + per-player masks.
+  reconstruct_selected()-> SAM 3D Body: meshes for ONLY the selected boxes (Build).
+
+This is the RF-DETR variant of the Space: detection uses Roboflow's RF-DETR-Seg
+(default the *large* model), which returns boxes AND instance masks in one pass —
+the UI toggles between showing/selecting by box or by segment. Build still
+reconstructs only the selected players with SAM 3D Body.
+"""
+
+import os
+import sys
+import functools
+
+import numpy as np
+
+# The sam-3d-body repo is cloned here by the Dockerfile and added to sys.path.
+SAM3D_DIR = os.environ.get("SAM3D_DIR", "/app/sam-3d-body")
+if SAM3D_DIR not in sys.path:
+    sys.path.insert(0, SAM3D_DIR)
+
+HF_REPO_ID = os.environ.get("SAM3D_REPO_ID", "facebook/sam-3d-body-dinov3")
+RFDETR_SIZE = os.environ.get("RFDETR_SIZE", "large").lower()
+
+COCO_PERSON = 1   # rfdetr/COCO is 1-indexed: person == 1
+
+_DETECTOR = None
+_ESTIMATOR = None
+_FACES = None
+
+
+# ----------------------------------------------------------------------------
+# Detector — RF-DETR-Seg (boxes + masks). Loaded on the first Detect.
+# ----------------------------------------------------------------------------
+def get_detector():
+    global _DETECTOR
+    if _DETECTOR is None:
+        from rfdetr import (RFDETRSegNano, RFDETRSegSmall,        # type: ignore
+                            RFDETRSegMedium, RFDETRSegLarge)
+        sizes = {"nano": RFDETRSegNano, "small": RFDETRSegSmall,
+                 "medium": RFDETRSegMedium, "large": RFDETRSegLarge}
+        Model = sizes.get(RFDETR_SIZE, RFDETRSegLarge)
+        _DETECTOR = Model()
+        try:
+            _DETECTOR.optimize_for_inference()
+        except Exception:
+            pass
+    return _DETECTOR
+
+
+@functools.lru_cache(maxsize=16)
+def _detect_cached(video_path, idx, conf):
+    """Run RF-DETR-Seg on one frame; keep person boxes + masks. Cached per frame."""
+    from .video import grab_frame
+    frame_rgb = grab_frame(video_path, idx)
+    if frame_rgb is None:
+        return None
+    det = get_detector().predict(frame_rgb, threshold=conf)
+    masks = getattr(det, "mask", None)
+    people = []
+    for k in range(len(det.xyxy)):
+        if int(det.class_id[k]) != COCO_PERSON:
+            continue
+        x1, y1, x2, y2 = [float(v) for v in det.xyxy[k]]
+        people.append({
+            "bbox": np.array([x1, y1, x2, y2], dtype=float),
+            "score": float(det.confidence[k]),
+            "mask": (np.asarray(masks[k], dtype=bool) if masks is not None else None),
+        })
+    # stable left-to-right order
+    people.sort(key=lambda p: (p["bbox"][0], p["bbox"][1]))
+    return people
+
+
+def detect_frame(video_path, idx, conf=0.4):
+    """CPU-cheap wrapper around the cached RF-DETR-Seg detection."""
+    return _detect_cached(str(video_path), int(idx), round(float(conf), 3))
+
+
+# ----------------------------------------------------------------------------
+# Reconstructor — SAM 3D Body. Loaded on the first Build.
+# ----------------------------------------------------------------------------
+def get_estimator():
+    """Lazy-load the SAM 3D Body estimator once; returns (estimator, faces)."""
+    global _ESTIMATOR, _FACES
+    if _ESTIMATOR is None:
+        from huggingface_hub import login
+        token = os.environ.get("HF_TOKEN")
+        if token:
+            login(token=token)
+        from notebook.utils import setup_sam_3d_body
+        _ESTIMATOR = setup_sam_3d_body(hf_repo_id=HF_REPO_ID)
+        _FACES = np.asarray(_ESTIMATOR.faces)
+    return _ESTIMATOR, _FACES
+
+
+def get_faces():
+    return get_estimator()[1]
+
+
+@functools.lru_cache(maxsize=16)
+def _reconstruct_cached(video_path, idx, boxes_key):
+    """Reconstruct ONLY the given boxes. boxes_key is a hashable tuple of int xyxy."""
+    from .video import grab_frame
+    est, _ = get_estimator()
+    frame_rgb = grab_frame(video_path, idx)
+    if frame_rgb is None:
+        return None
+    boxes = np.array(boxes_key, dtype=np.float32).reshape(-1, 4)
+    # Providing bboxes reconstructs exactly these people (in order); the FOV
+    # estimator still runs for focal_length.
+    people = est.process_one_image(frame_rgb, bboxes=boxes)
+    slim = []
+    for p in people:
+        kp = p.get("pred_keypoints_3d")
+        slim.append({
+            "bbox": np.asarray(p["bbox"]).reshape(-1)[:4].astype(float),
+            "pred_vertices": np.asarray(p["pred_vertices"], dtype=np.float32),
+            "pred_cam_t": np.asarray(p["pred_cam_t"], dtype=np.float32).reshape(3),
+            "focal_length": float(np.asarray(p["focal_length"]).reshape(-1)[0]),
+            "pred_keypoints_3d": (None if kp is None
+                                  else np.asarray(kp, dtype=np.float32)),
+        })
+    return slim
+
+
+def reconstruct_selected(video_path, idx, boxes):
+    """Reconstruct meshes for the selected boxes (list of [x1,y1,x2,y2])."""
+    boxes_key = tuple(int(round(v)) for b in boxes for v in b[:4])
+    return _reconstruct_cached(str(video_path), int(idx), boxes_key)

pipeline/overlay.py

@@ -0,0 +1,109 @@
+"""
+cv2 overlays — pure CPU. Draws detection boxes and the two goal-parallel lines
+the user clicks. Works on RGB arrays in, RGB arrays out (Gradio Image is numpy/RGB).
+"""
+
+import cv2
+import numpy as np
+
+
+def annotate_detections(frame_rgb, people, selected_ids=()):
+    """Draw numbered boxes; selected players get a thick yellow box, others thin green."""
+    sel = set(int(i) for i in selected_ids)
+    img = cv2.cvtColor(frame_rgb, cv2.COLOR_RGB2BGR)
+    for i, p in enumerate(people):
+        x1, y1, x2, y2 = [int(v) for v in p["bbox"]]
+        if i in sel:
+            cv2.rectangle(img, (x1, y1), (x2, y2), (0, 230, 255), 3)   # yellow, thick
+        else:
+            cv2.rectangle(img, (x1, y1), (x2, y2), (0, 255, 0), 2)     # green, thin
+        cv2.putText(img, str(i), (x1, max(y1 - 6, 14)),
+                    cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 0, 255), 3)
+    return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+
+
+def pick_box(people, x, y):
+    """Return the index of the smallest box containing (x, y), or None."""
+    hit, best_area = None, None
+    for i, p in enumerate(people):
+        x1, y1, x2, y2 = p["bbox"]
+        if x1 <= x <= x2 and y1 <= y <= y2:
+            area = (x2 - x1) * (y2 - y1)
+            if best_area is None or area < best_area:
+                best_area, hit = area, i
+    return hit
+
+
+# palette for per-player mask tints (BGR)
+_MASK_COLORS = [(60, 60, 220), (220, 140, 60), (60, 200, 220), (200, 80, 200),
+                (80, 200, 120), (220, 200, 80), (140, 100, 240), (90, 180, 60)]
+
+
+def draw_masks(frame_rgb, people, selected_ids=()):
+    """Translucent colored silhouettes; selected players brighter + outlined.
+
+    Falls back to a box for any detection lacking a mask.
+    """
+    sel = set(int(i) for i in selected_ids)
+    img = cv2.cvtColor(frame_rgb, cv2.COLOR_RGB2BGR)
+    for i, p in enumerate(people):
+        m = p.get("mask")
+        col = _MASK_COLORS[i % len(_MASK_COLORS)]
+        is_sel = i in sel
+        if m is not None:
+            alpha = 0.55 if is_sel else 0.30
+            tint = np.zeros_like(img); tint[m] = col
+            img[m] = (img[m] * (1 - alpha) + tint[m] * alpha).astype(np.uint8)
+            if is_sel:  # outline the selected silhouette
+                cnts, _ = cv2.findContours(m.astype(np.uint8), cv2.RETR_EXTERNAL,
+                                           cv2.CHAIN_APPROX_SIMPLE)
+                cv2.drawContours(img, cnts, -1, (0, 230, 255), 3)
+        else:
+            x1, y1, x2, y2 = [int(v) for v in p["bbox"]]
+            cv2.rectangle(img, (x1, y1), (x2, y2),
+                          (0, 230, 255) if is_sel else (0, 255, 0), 3 if is_sel else 2)
+        x1, y1 = int(p["bbox"][0]), int(p["bbox"][1])
+        cv2.putText(img, str(i), (x1, max(y1 - 6, 14)),
+                    cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 255, 255), 3)
+    return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+
+
+def pick_mask(people, x, y):
+    """Index of the player whose mask covers (x, y); ties -> smallest mask.
+
+    Falls back to box hit-testing for detections without a mask.
+    """
+    x, y = int(x), int(y)
+    hit, best_area = None, None
+    for i, p in enumerate(people):
+        m = p.get("mask")
+        if m is not None:
+            if 0 <= y < m.shape[0] and 0 <= x < m.shape[1] and m[y, x]:
+                area = int(m.sum())
+                if best_area is None or area < best_area:
+                    best_area, hit = area, i
+        else:
+            x1, y1, x2, y2 = p["bbox"]
+            if x1 <= x <= x2 and y1 <= y <= y2:
+                area = (x2 - x1) * (y2 - y1)
+                if best_area is None or area < best_area:
+                    best_area, hit = area, i
+    return hit
+
+
+def draw_lines(frame_rgb, pts):
+    """Draw the clicked points and the (up to two) line segments.
+
+    pts is a list of [x, y]; points 0-1 form line 1 (red), 2-3 form line 2 (yellow).
+    """
+    if frame_rgb is None:
+        return None
+    img = cv2.cvtColor(frame_rgb.copy(), cv2.COLOR_RGB2BGR)
+    cols = [(0, 0, 255), (0, 255, 255)]  # BGR: line1 red, line2 yellow
+    for k, (px, py) in enumerate(pts):
+        cv2.circle(img, (int(px), int(py)), 6, cols[k // 2], -1)
+    if len(pts) >= 2:
+        cv2.line(img, tuple(map(int, pts[0])), tuple(map(int, pts[1])), cols[0], 2)
+    if len(pts) >= 4:
+        cv2.line(img, tuple(map(int, pts[2])), tuple(map(int, pts[3])), cols[1], 2)
+    return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

pipeline/video.py

@@ -0,0 +1,26 @@
+"""
+Video helpers — pure CPU. Frames are seeked on demand (no bulk extraction),
+so scrubbing a long clip stays cheap and never touches the GPU.
+"""
+
+import cv2
+
+
+def probe_video(video_path):
+    """Return (frame_count, fps) for a clip. fps falls back to 25 if unknown."""
+    cap = cv2.VideoCapture(video_path)
+    n = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+    fps = cap.get(cv2.CAP_PROP_FPS) or 25.0
+    cap.release()
+    return n, fps
+
+
+def grab_frame(video_path, idx):
+    """Seek to an exact frame index and return it as an RGB array (or None)."""
+    cap = cv2.VideoCapture(video_path)
+    cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx))
+    ok, frame_bgr = cap.read()
+    cap.release()
+    if not ok:
+        return None
+    return cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)

requirements.txt

@@ -0,0 +1,37 @@
+# Pinned to the versions verified during the build (mirrors the Dockerfile's
+# torch pin). Gradio 6 requires Slider minimum < maximum — see app.py.
+gradio==6.19.0
+plotly>=5.20
+opencv-python-headless
+rfdetr>=1.8.1
+supervision>=0.29.0
+numpy
+huggingface_hub
+pytorch-lightning
+pyrender
+yacs
+scikit-image
+einops
+timm
+dill
+pandas
+rich
+hydra-core
+hydra-submitit-launcher
+hydra-colorlog
+pyrootutils
+webdataset
+chump
+networkx==3.2.1
+roma
+joblib
+seaborn
+appdirs
+cython
+jsonlines
+xtcocotools
+loguru
+optree
+fvcore
+pycocotools
+trimesh