test / README.md
Upload source
6916285 verified about 1 month ago
26.2 kB
# Soccer Action Detection

Three-stage action spotting for soccer video, built on the pretrained weights in
`model weights/`.

```
LEVEL 1 — SOCCER KNOWLEDGE      InternVideo2-Soccer
                                players, ball interaction, possession, 15 actions
                                        |
LEVEL 2 — WHEN DID IT HAPPEN?   T-DEED temporal encoder-decoder
                                dense per-frame logits, high-recall candidates
                                        |
LEVEL 3 — SHOULD I TRUST IT?    Verifier: temporal transformer -> XGBoost gate
                                ball/player physics, tracking confidence,
                                hard false positives, frame refinement
```

Ships with two production classes (`pass`, `pass_received`) and scales to all
fifteen by editing one list.

---

## Read this first: the 25 fps timebase

**The annotations are indexed at 25 fps even when the video is not.** The sample
`5.mp4` is 30.0008 fps / 77,391 frames (43.0 min), and its labels run to frame
64,246 — which is 42.8 min at 25 fps, but only 35.7 min at 30 fps. Read as
25 fps, the `goal` at frame 23,562 lands on a goalmouth scramble; read as
30 fps it lands on midfield play.

Everything in this codebase therefore works in **canonical frames** — integer
indices on a 25 fps grid. `soccer_ad/utils/video.py` is the only place that
converts, and `label_fps` is stored per match, so a dataset can mix conventions.
Scanning warns loudly if a file's labels cannot fit the video at any candidate
frame rate.

---

## Quick start

```bash
pip install -r requirements.txt

python run_ui.py                 # dashboard on http://127.0.0.1:8000
```

Then, in the dashboard:

1. **Prepare data**: import a folder containing videos and their annotations,
   choose the cache directory, run tracking, and generate soccer-state labels.
2. **Model jobs**: create a named training job. In its first step, select prepared
   matches and assign whole matches to training, validation, and test. Choose
   the output actions here. Each stage tab contains its own model-source field:
   encoder adaptation accepts the configured InternVideo2 model, an earlier
   job's adapted encoder, or a custom checkpoint; spotting and OOF independently
   select their T-DEED temporal initialization or random initialization; the
   verifier clearly shows that it starts fresh from this job's OOF candidates.
   Configure encoder adaptation, spotting, OOF generation, and verifier training
   using the four stage tabs. Each has its own epochs, batch size, learning rates,
   video settings, freeze schedule, and other advanced settings. Set the OOF fold
   count and optionally enable automatic advancement. Saving setup validates all
   four configurations before any training starts.
   At least two training matches, one validation match,
   and one test match are required. Keep venue/camera families in the same split.
3. Complete the job's steps in order: encoder adaptation, action spotting,
   OOF candidate generation, and verifier training. Only the current step's form
   is shown during training. Each step restores its configuration saved at setup.
   Start the encoder manually; with automatic advancement enabled, the server
   starts each following stage on success, even after closing the browser.
   Automatic steps use their saved configurations, including their own clip and
   batch sizes; edits to one stage do not overwrite another stage's setup.
   Progress, metrics, GPU usage, pause/resume, and stop controls remain available.
4. **Calibrate & infer** appears after all four training steps finish successfully.
   The job's spotting checkpoint, verifier, and candidate file are selected
   automatically. Calibration uses its validation matches; test matches remain
   held out. Select a registered match or browse for a new video to run inference.
   Click an action in the results to inspect its video frame.

Jobs persist under `runs/workflows/<job-id>/`. Each records its configuration,
dataset metadata, fixed splits, selected actions, per-stage model sources,
automation settings, stage status, and artifact references. Stage runs, including
their checkpoints, logs, and OOF fold models, live in
`runs/workflows/<job-id>/runs/<run-id>/`. The adapted encoder is exported into
`runs/workflows/<job-id>/checkpoints/` and passed to later
stages automatically. Changing the preparation data/cache folder does not change
an existing job's dataset. Failed or interrupted stages can be retried; later
stages remain locked. The **Activity** button shows background work.

**Model jobs** opens a separate job selection page. Its **Create job** form is
always visible. Create a named job or select an existing job to open its own
initialization or training workspace. Use **Back to all jobs** to return to the
selection page. Unsaved setup edits are retained while switching jobs in the same browser session.
GPU work queues
on the server. **Pause job** pauses its current training or candidate-generation
stage at a safe boundary and prevents automatic advancement. **Resume job**
continues it. Both controls are available in the job summary and directly above
the training progress bars once a job starts. A live pause retains its worker.
After a server restart or an unexpected failure, **Resume stage** restores the
newest `last.pt` or `paused.pt` checkpoint in the same run directory. Recovery
continues at the next unfinished batch inside an interrupted epoch; a completed
epoch continues at the following epoch. Failures
and cancellation never start the next stage automatically.

OOF generation produces held-out predictions for training matches and separate
spotting predictions for the job's validation matches. Verifier training respects
those explicit splits; the test split is excluded from training and calibration.

Every completed epoch is retained in its run directory as `epoch_0001.pt`,
`epoch_0002.pt`, and so on. `last.pt` points to the newest complete training
state and `best.pt` contains the best validation model. Recovery checkpoints
include the optimizer, scheduler step, AMP scaler, EMA weights when enabled,
configuration, actions, and run state. A paused job can also be stopped cleanly.
Training input is checked before GPU allocation, so missing videos, annotations,
tracking, and states are reported together. CUDA jobs release unused tensors
between stages. With `train.auto_recover_oom: true`, an out-of-memory failure
automatically halves the physical batch, raises gradient accumulation to retain
the effective batch, and resumes from the latest completed epoch. Recovery stops
at batch size 1 after enabling encoder gradient checkpointing; the job then shows
an actionable error and remains resumable.

OOF and ordinary inference use a separate no-gradient micro-batch controller.
It begins at `train.batch_size`, measures real peak VRAM after successful
forwards, grows toward `train.inference_batch_max_size` while retaining the
configured reserve, and retries only the pending clips at the last proven or a
smaller batch after an ordinary allocation failure. A learned batch is reused
only for the same model/input shape and GPU. The same clips and model outputs
are used, so this does not change training or inference semantics. CUDA
allocator/context asserts still require a process restart. Training itself never silently exceeds its
effective batch: set a deliberately larger `batch_size * grad_accum_steps` when
you want the physical training-batch tuner to have more room to grow.

Everything is also available from the CLI:

```bash
python cli.py scan                             # discover matches
python cli.py splits --auto                    # write configs/splits.yaml
python cli.py track --split train              # tracking preprocessing
python cli.py states                           # soccer-state pseudo-labels
python cli.py train encoder                    # stage 1
python cli.py train spotting                   # stage 2
python cli.py oof --k 4                        # out-of-fold candidates
python cli.py train verifier                   # stage 3
python cli.py calibrate --apply                # thresholds vs weighted score
python cli.py infer path/to/video.mp4 --run spotting-cli
python cli.py evaluate --split test            # locked-test evaluation
```

### Headless preprocessing stage

The ball/player tracker and soccer-state pseudo-label extraction can run as a
self-contained first stage, without starting the dashboard or configuring any
model-training settings. Its one JSON file contains every hyperparameter used
by this stage, selected match IDs, cache behaviour, and manifest destination.

```bash
# Start from the complete, editable template.
python preprocess.py --init-config configs/tracking_states.json
# Edit that JSON, then run tracking followed by state extraction.
python preprocess.py --config configs/tracking_states.json
```

[`configs/tracking_states.example.json`](configs/tracking_states.example.json)
is a ready-to-copy production starting point. `matches: null` selects all
videos below `paths.data_root`; use `"matches": ["season_1/match_001"]` to
run a subset. Results are stored as `tracking/<match>.npz`, `states/<match>.npz`,
`preprocessing/features/<match>.npz`, and a JSON manifest under the cache root.
Each feature archive bundles ball tracks, sparse player tracks, state labels,
confidence masks, and metadata for one video. It does not conflict with
`cache/features`, which is reserved for later visual-encoder embeddings.

`skip_existing: true` (the default) skips a video without opening or decoding
it when all its requested outputs already exist. Set it to `false` to rerun
every selected video, or use the individual `overwrite_*` fields to rebuild a
specific cache type. Rebuilding tracking automatically rebuilds its states and
feature archive.

To inspect the result, set `visualization.enabled` to `true` and select one or
more matches. The stage writes an MP4 overlay to
`data/cache/preprocessing/previews/<match>.mp4`; it draws player boxes and
track/team IDs, ball trajectory (orange means interpolated), and the current
state pseudo-labels. By default it exports the first 1,500 canonical frames
(one minute at 25 FPS). This preview reads cached outputs and does not rerun
detection for a skipped video.

The template also includes `tracking.person_onnx`, pointing to the supplied
`model weights/PlayerDetect/person-detection-model.onnx`. The active
`tracking.detector: "rfdetr"` is intentionally the default: it detects both
the ball and people in one model, which this state-extraction stage needs.
`onnx_person` is not currently a selectable standalone detector because it
does not provide the ball detections required to derive possession and ball
state labels.

Any config value can be overridden inline, which makes a sweep a shell loop:

```bash
for n in 50 100 150; do
  python cli.py --set data.clip_len=$n --set name=clip_length_$n train spotting
done
```

Every clip uses **all consecutive canonical frames** during encoder adaptation,
spotting training, validation, and inference. At 25 fps, `clip_len: 100` means
100 input frames spanning four seconds, including both the first and last
frame. There is no separate sampled-frame count or within-clip sampling mode.
Selecting training clip locations still varies across epochs; frames inside
each selected clip remain consecutive. Converting a video's native rate (for
example 30 fps) to the annotation-aligned 25 fps timeline is a separate step.

`model.encoder.temporal_chunk: 8` processes a 100-frame clip in twelve chunks of
eight and one chunk of four, preserving every frame. Larger clips increase GPU
memory and compute; batch size, gradient accumulation, and gradient checkpointing
remain available controls. Old configs containing sampling settings load with
a migration warning and use all frames, so re-evaluate older checkpoints under
the new input policy. Optional feature caches must have stride 1 and one feature
row per canonical video frame.

---

## Data layout

A match is a video with a JSON of the same stem beside it. Sub-folders are
searched recursively and the relative path becomes the match id.

```
data/raw/
    2024-09-01_teamA_vs_teamB.mp4
    2024-09-01_teamA_vs_teamB.json
    batch2/
        5.mp4
        5.json                     -> match id "batch2/5"
```

Annotation format (never modified by the pipeline):

```json
{ "5.mp4": [ {"frame": 111, "action": "pass"},
             {"frame": 139, "action": "pass_received"} ] }
```

A bare list is also accepted, as are `position` (ms) and `seconds` instead of
`frame`. Unknown action names are reported and skipped, not guessed.

---

## The 15 classes

| action | weight | tolerance | | action | weight | tolerance |
|---|---|---|---|---|---|---|
| pass | 1.0 | 1.0 s | | substitution | 4.2 | 2.0 s |
| pass_received | 1.4 | 1.0 s | | block | 4.2 | 2.0 s |
| recovery | 1.5 | 1.5 s | | aerial_duel | 4.3 | 2.0 s |
| tackle | 2.5 | 1.5 s | | shot | 4.7 | 2.0 s |
| interception | 2.8 | 2.0 s | | save | 7.3 | 2.0 s |
| ball_out_of_play | 2.9 | 2.0 s | | foul | 7.7 | 2.5 s |
| clearance | 3.1 | 2.0 s | | goal | 10.9 | 3.0 s |
| take_on | 3.2 | 2.0 s | | | | |

This table lives in `soccer_ad/constants.py` and is the single source of truth:
head widths, target shapes, NMS windows, evaluation tolerances and the weighted
score all derive from it. **Adding a 16th action means adding one row here.**

The active classes only control the *production* head. Stage 1 always trains on
all fifteen, and stage 2 keeps a 15-class auxiliary head, because telling a pass
from a clearance is exactly the discrimination the detector needs.

---

## Headless InternVideo2 + T-DEED training

After tracking/state preprocessing, the first two trainable stages can run as a
single UI-free pipeline. It adapts InternVideo2 using raw video, annotations,
and state pseudo-labels, exports that soccer encoder, then trains the T-DEED
spotter from it. OOF generation and the verifier are not run.

```bash
python train_internvideo_tdeed.py --init-config configs/internvideo_tdeed.json
# Edit dataset IDs, batch sizes, model settings, and Hugging Face repo_id.
# Put a write token in HF_TOKEN; do not store the token in JSON.
python train_internvideo_tdeed.py --config configs/internvideo_tdeed.json
```

The complete template exposes every shared model, data, loss, optimizer,
schedule, AMP, loader, and memory-recovery setting, with per-stage recursive
overrides. The all-frame policy is enforced: every canonical frame appears in a
training clip each epoch, every frame inside each clip is encoded, validation
is exhaustive, and cached visual embeddings cannot replace raw video.

Local checkpoints are written after every completed epoch under
`runs/<run_name>/encoder/` and `runs/<run_name>/spotting/`; each contains
`last.pt`, metric-selected `best.pt`, and retained `epoch_NNNN.pt` files. At
every configured five-epoch boundary, current `best.pt` and `last.pt` are
uploaded to the configured private Hugging Face model repository. The token is
read only from `huggingface.token_env` (default `HF_TOKEN`).

The configured physical batch is used for the first attempt. The trainer reads
peak/free VRAM after each epoch and adjusts the next epoch's physical batch
while preserving effective batch size through gradient accumulation. Initial
or later ordinary CUDA OOMs restart from the newest safe local checkpoint at a
smaller compatible batch; batch 1 falls back to InternVideo gradient
checkpointing before reporting an unrecoverable memory error.

[`configs/internvideo_tdeed.example.json`](configs/internvideo_tdeed.example.json)
is a concise starting configuration. `--init-config` writes the fully expanded
version when every default field should be visible.

---

## Pipeline stages

### 0 · Tracking preprocessing (optional but needed for the verifier)

RF-DETR (SoccerNet-finetuned) detects ball / player / referee / goalkeeper in one
pass. Players go through an IoU tracker with velocity extrapolation, OSNet
embeddings are averaged per track and clustered into two teams, and the ball
trajectory is assembled with a constant-acceleration Kalman filter that gates out
physically implausible detections — which is what rejects the ball on the
adjacent pitch.

Missing data is **never** encoded as `(0, 0)`. Every quantity is stored as
`value + valid mask + confidence`, plus `interpolated` and `frames_since_seen`.

> **Runtime.** Detection dominates. RF-DETR runs ~1.4 fps in fp32 and ~6.5 fps in
> fp16 on a 6 GB RTX 2060. A 45-minute match is ~64,500 canonical frames, so at
> `detect_stride=2` that is ~2.4 h/match on that card — roughly 143 h for 60
> matches. On a 4090/A100 expect several times faster. `detect_stride` divides
> the cost directly; above ~3 the ball trajectory starts losing fidelity while
> player positions tolerate much more. **Stages 1 and 2 do not need tracking.**

### 1 · Encoder soccer adaptation

InternVideo2-B (768-dim, 12 blocks, patch 14, tubelet 1) plus five heads: a
15-class action head and four soccer-state heads (possession, ball state,
player–ball relation, possession transition) trained on tracking-derived
pseudo-labels.

```
L = L_action + 0.25·L_possession + 0.20·L_ball_state
             + 0.20·L_relation   + 0.30·L_transition
```

Every state term is confidence-weighted and masks its `unknown` slot, so a match
with poor tracking still contributes its action supervision in full while
contributing nothing false about possession.

Three phases run as one freeze schedule: head warm-up → upper half → everything
with layer-wise LR decay. Output: `checkpoints/internvideo2_soccer_best.pt`,
the reusable encoder every later stage starts from.

### 2 · Spotting

```
RGB clip -> InternVideo2-Soccer -> temporal adapter -> T-DEED -> dense logits
```

The encoder is pooled **spatially only** — the temporal axis survives to the
heads. With the default tubelet size of 1, the adapter refines all 100 frame
features and projects their width from 768 to T-DEED's 368. The temporal length
stays at 100 throughout; there is no interpolation from sparsely sampled frames.

Each ground truth becomes two targets:

- **broad** — a linear ramp, 1 at the event, 0 at that class's tolerance. Buys
  recall.
- **exact** — a narrow Gaussian (σ ≈ 2–3 frames). Buys frame-exact localisation.

plus a displacement regression that snaps a confident-but-offset peak onto the
event. Per-class sigmoids, not a softmax: actions genuinely co-occur, and each
class needs its own calibrated threshold.

The published 15-action weights are applied whenever `loss.use_class_balance`
is enabled (the default). Stage 1 applies them to all 15 action outputs. Stage 2
applies them to the active production classes and independently to its 15-class
auxiliary head. Thus the current `pass` / `pass_received` head uses a 1.0:1.4
ratio, while `goal` already receives a 10.9:1 ratio relative to `pass` in the
auxiliary objective and will keep that ratio when it is added to the production
class list. Stage 3 also weights each candidate's classification and frame-
refinement losses by its claimed action; its final XGBoost models remain
separate per action.

Validation reports the **weighted competition score on discrete events**, not a
loss — a loss can improve while detection gets worse.

### 3 · Verifier

```
candidate C -> C±30 -> multimodal tokens [61, D]
                          |
              3-layer / 192-D / 4-head temporal transformer
                          |
        +-----------------+------------------+
        v                                    v
  192-D embedding                     61-frame heatmap -> refined frame
        |
  [embedding || T-DEED stats | ball stats | player stats
              | possession stats | tracking-quality stats]
        |
     XGBOOST  ->  P(valid)  ->  class threshold  ->  keep / reject
```

The transformer reads the *shape* of the window; the boosted tree makes the
accept/reject call from that embedding plus ~48 engineered scalars, which is
where trees beat a small network on limited data. Physics is fused into the
visual token through a **reliability gate** driven by tracking quality, so a
failed track degrades to the visual representation instead of poisoning it —
reinforced by deliberately corrupting tracking on ~50% of training samples.

Trained on **out-of-fold** candidates: K models, each predicting the fold it
never saw. A verifier trained on in-fold candidates learns that stage 1 is
trustworthy, which is the opposite of its job.

### 4 · Calibration

Calibration selects the confidence threshold and NMS window for each action
using labeled out-of-fold or validation predictions. The threshold controls
which candidates survive; NMS suppresses nearby duplicate detections. The search
maximizes the **weighted mean of per-class F1** used by this project. Positive
class weights change the contribution of each class to that score; they do not
by themselves justify accepting more false positives within a class.

In **Calibrate & infer**, use **Browse predictions...** to select the candidate JSON.
Their match IDs must map to the registered videos and annotations. Review the
suggested settings, then apply them to the active configuration if desired.
Calibration changes postprocessing parameters and trains no neural weights.
Keep test matches untouched until all training and calibration choices are final.

---

## Maintenance

### Adding matches

```bash
python cli.py scan                     # picks up the new files
python cli.py track                    # only new matches are swept; caches are reused
python cli.py states
python cli.py train spotting           # fine-tune on the expanded set
python cli.py oof --fresh              # regenerate candidates if this is a major retrain
python cli.py train verifier
python cli.py calibrate --apply
```

Nothing needs rebuilding if the tracking models have not changed.

### Adding an action

Edit `actions:` in the config (or tick it in **Prepare data**). The spotting
head grows in place — `SoccerSpottingModel.expand_actions` copies the existing
class rows so a trained pass/pass_received model keeps its pass detectors and
only the new rows are initialised. Then fine-tune stage 2, retrain the verifier
with that class's hard negatives, and recalibrate.

### Replacing the ball/player detector

Re-run tracking → states → verifier. The RGB representation is untouched, so
stages 1 and 2 do not need rebuilding unless the new state labels are
substantially better.

---

## Layout

```
soccer_ad/
  constants.py          15-class taxonomy, weights, tolerances, canonical fps
  config.py             typed layered config (pydantic) + YAML _base_ inheritance
  utils/video.py        canonical-timebase video access  <- the fps conversion
  data/                 ingest, splits, labels, targets, datasets, augmentation
  tracking/             detector, player tracker, ball Kalman, teams, cache
  states/               soccer-state pseudo-labels with confidences
  models/               internvideo2, tdeed, adapter, heads, spotting, verifier
  training/             engine, losses, optim, per-stage trainers, OOF
  eval/                 tolerance-aware metrics, NMS, calibration
  inference/            end-to-end video -> events
  server/               FastAPI + dashboard (static/, no build step)
configs/                default.yaml + presets/
cli.py                  scriptable equivalent of the dashboard
run_ui.py               dashboard entry point
```

### Presets

| preset | for |
|---|---|
| `configs/default.yaml` | baseline; assumes a 24 GB+ GPU |
| `configs/presets/smoke_2060.yaml` | end-to-end sanity check on 6 GB, minutes |
| `configs/presets/production.yaml` | full run on a 4090 / A6000 / A100 |
| `configs/presets/all_15_actions.yaml` | every class at once |

---

## Pretrained weights

| file | what it is | how it is used |
|---|---|---|
| `InternVideo2/pytorch_model.bin` | InternVideo2-B, CLIP-distilled. 768-dim, 12 blocks, patch 14, tubelet 1, pos-embed 1 + 8×256 | vision tower; **160/160 tensors load, 0 missing / 0 unexpected**. The 57 distillation keys (`clip_decoder`, `clip_projector`) are dropped by design |
| `T-DEED/checkpoint_best.pt` | SoccerNet T-DEED: RegNet-Y + SGP/SGPMixer, dim 368, `temp_enc` (100, 368) | **155 temporal tensors load, 0 missing / 0 unexpected**. The RegNet backbone (443 tensors) and the 13/18-class SoccerNet heads are skipped — we supply our own encoder and vocabulary |
| `RF-DETR/checkpoint_best_regular.pth` | rf-detr-large fine-tuned on SoccerNet, 4 classes | ball + players in one pass. Class ids verified empirically: **0=ball, 1=player, 2=referee, 3=goalkeeper** (note: this differs from the reference `miner.py` constants) |
| `TeamDetect/osnet_model.pth.tar-100` | OSNet re-id | per-track embeddings → 2-means team assignment |

T-DEED's `ks=9, k=4` were recovered from the checkpoint: `up_size =
round((ks+1)·k)` forced odd gives the 41-tap `convkw` kernel it actually
contains, and `num_layers=2` is what produces its 5 SGP blocks + 2 mixers.

---

## Notes on this environment

- **PyAV is blocked** by an Application Control policy on this machine, so video
  decoding falls back to OpenCV. The OpenCV path seeks once per clip and then
  uses `grab()`/`retrieve()` to decode the complete canonical clip. If PyAV becomes available, set
  `data.decode_backend: pyav` for a further speedup.
- **onnxruntime is CPU-only** here. Install `onnxruntime-gpu` if you want the
  ONNX player detector path to be fast; the default RF-DETR path uses PyTorch
  and is unaffected.
- The RTX 2060 is Turing, so **`amp_dtype: fp16`** (not bf16) — see the
  smoke preset.