| --- |
| license: cc-by-nc-4.0 |
| library_name: pytorch |
| base_model: facebook/cotracker3 |
| tags: |
| - point-tracking |
| - video-tracking |
| - surgical-video |
| - streaming-inference |
| - distillation |
| datasets: |
| - nct-tso/STIR_pseudo_tracks |
| --- |
| |
| # Verifier-Guided LiteTracker for STIR 2026 |
|
|
| This is the `NCT_TSO` checkpoint for streaming tissue point tracking in the |
| STIR Challenge 2026. It is a CoTracker3-Online-compatible state dictionary and |
| loads directly into the LiteTracker streaming runtime. The model was fine-tuned |
| with confidence-weighted pseudo-labels fused from six teacher trackers. |
|
|
| The implementation is developed from |
| [CoTracker3](https://github.com/facebookresearch/co-tracker), which supplies the |
| student architecture and training loss, and uses LiteTracker for causal |
| streaming inference. The release repository adds the STIR-specific |
| multi-teacher verification, pseudo-label fusion, fine-tuning, and evaluation |
| pipeline; it does not vendor CoTracker3 source. |
|
|
| ## Model details |
|
|
| - **Architecture:** CoTracker3-Online / LiteTracker-compatible, 25,385,700 parameters |
| - **Checkpoint:** `student.pth` |
| - **Input:** RGB video and frame-0 query points in `(x, y)` pixel coordinates |
| - **Output:** per-frame 2D trajectories and visibility estimates |
| - **Window length during training:** 16 |
| - **Training refinement iterations:** 4 |
| - **SHA-256:** `3b964f18793bab959d1537aa58b0411bb8454e2df5f30bf6d66039a215c8455c` |
| - **Code and reproducibility:** https://github.com/danushkv/STIR2026_challenge |
| - **Authors:** Danush Kumar Venkatesh, Peng Liu, Stefanie Speidel |
| |
| ## Intended use |
| |
| The checkpoint is intended for non-commercial research on streaming point |
| tracking in laparoscopic video, reproduction of the STIR 2026 results, and |
| fine-tuning or benchmarking under the documented protocol. It is not a medical |
| device and must not be used for clinical decisions or autonomous intervention. |
| |
| ## Training data and procedure |
| |
| Training used STIROrig together with STIR Challenge 2024 through a combined, |
| non-colliding patient layout. Six offline teachers (`alltracker`, `cotracker3`, |
| `locotrack`, `mft`, `trackon2`, and `trackon_r`) produced trajectories at a |
| temporal stride of 5. A verifier combined endpoint supervision, cycle |
| consistency, and cross-teacher agreement into per-frame confidence weights. |
|
|
| ## Evaluation |
|
|
| Evaluation used 32 clips (234 annotated points) from the STIR 2025 test |
| collection under the LiteTracker streaming runtime. |
|
|
| | track | refinement iterations | metric | score | |
| |---|---:|---|---:| |
| | 2D accuracy | 4 | delta average at 4/8/16/32/64 px | 0.8103 | |
| | 2D latency | 1 | delta average | 0.8085 | |
| | 2D latency | 1 | p95 latency, RTX A5000 | 48.7 ms | |
| | 3D submission-like | 4 | accuracy at 2/4/8/16/32 mm | 0.7031 | |
|
|
| The offline reproduction protocol uses annotation-derived right-view starting |
| points unavailable to the submission harness and therefore reports a higher 3D |
| score. See `docs/CAVEATS.md` in the code repository before comparing numbers. |
|
|
| ## Limitations |
|
|
| - Ground truth exists only at clip endpoints; intermediate trajectory quality |
| and visibility are not directly supervised. |
| - The training run is seeded but not bit-reproducible because deterministic GPU |
| algorithms and DataLoader worker seeding were not enforced. |
| - Training includes STIR Challenge 2024 patients that overlap the 2025 test |
| collection at patient level. |
| - Performance outside laparoscopic tissue video, on unseen devices, or under |
| different image statistics has not been established. |
| - Inference refinement iterations affect accuracy and latency; benchmark 1, 2, |
| and 4 rather than assuming one operating point. |
|
|
| ## Licence |
|
|
| The checkpoint is released under CC BY-NC 4.0 because it descends from |
| CoTracker3/LiteTracker weights. The accompanying code is separately MIT |
| licensed. Users must also follow the licences and terms of all upstream models |
| and datasets. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @software{venkatesh2026stir, |
| author = {Venkatesh, Danush Kumar and Liu, Peng and Speidel, Stefanie}, |
| title = {Verifier-Guided Multi-Teacher Distillation for Streaming Tissue Tracking}, |
| year = {2026}, |
| url = {https://github.com/danushkv/STIR2026_challenge} |
| } |
| ``` |
|
|
| ## Acknowledgements |
|
|
| We thank the authors of CoTracker3, LiteTracker, Track-On/Track-On2/Track-On-R, |
| AllTracker, LocoTrack, MFT, STIRLoader, and STIRMetrics, as well as the STIR |
| Challenge organisers and dataset contributors. The full linked acknowledgement |
| list is maintained in the code repository README. |
|
|
