README.md

---
license: cc-by-nc-4.0
library_name: pytorch
base_model: facebook/cotracker3
tags:
- point-tracking
- video-tracking
- surgical-video
- streaming-inference
- distillation
datasets:
- nct-tso/STIR_pseudo_tracks
---

# Verifier-Guided LiteTracker for STIR 2026

This is the `NCT_TSO` checkpoint for streaming tissue point tracking in the
STIR Challenge 2026. It is a CoTracker3-Online-compatible state dictionary and
loads directly into the LiteTracker streaming runtime. The model was fine-tuned
with confidence-weighted pseudo-labels fused from six teacher trackers.

The implementation is developed from
[CoTracker3](https://github.com/facebookresearch/co-tracker), which supplies the
student architecture and training loss, and uses LiteTracker for causal
streaming inference. The release repository adds the STIR-specific
multi-teacher verification, pseudo-label fusion, fine-tuning, and evaluation
pipeline; it does not vendor CoTracker3 source.

## Model details

- **Architecture:** CoTracker3-Online / LiteTracker-compatible, 25,385,700 parameters
- **Checkpoint:** `student.pth`
- **Input:** RGB video and frame-0 query points in `(x, y)` pixel coordinates
- **Output:** per-frame 2D trajectories and visibility estimates
- **Window length during training:** 16
- **Training refinement iterations:** 4
- **SHA-256:** `3b964f18793bab959d1537aa58b0411bb8454e2df5f30bf6d66039a215c8455c`
- **Code and reproducibility:** https://github.com/danushkv/STIR2026_challenge
- **Authors:** Danush Kumar Venkatesh, Peng Liu, Stefanie Speidel

## Intended use

The checkpoint is intended for non-commercial research on streaming point
tracking in laparoscopic video, reproduction of the STIR 2026 results, and
fine-tuning or benchmarking under the documented protocol. It is not a medical
device and must not be used for clinical decisions or autonomous intervention.

## Training data and procedure

Training used STIROrig together with STIR Challenge 2024 through a combined,
non-colliding patient layout. Six offline teachers (`alltracker`, `cotracker3`,
`locotrack`, `mft`, `trackon2`, and `trackon_r`) produced trajectories at a
temporal stride of 5. A verifier combined endpoint supervision, cycle
consistency, and cross-teacher agreement into per-frame confidence weights.

## Evaluation

Evaluation used 32 clips (234 annotated points) from the STIR 2025 test
collection under the LiteTracker streaming runtime.

| track | refinement iterations | metric | score |
|---|---:|---|---:|
| 2D accuracy | 4 | delta average at 4/8/16/32/64 px | 0.8103 |
| 2D latency | 1 | delta average | 0.8085 |
| 2D latency | 1 | p95 latency, RTX A5000 | 48.7 ms |
| 3D submission-like | 4 | accuracy at 2/4/8/16/32 mm | 0.7031 |

The offline reproduction protocol uses annotation-derived right-view starting
points unavailable to the submission harness and therefore reports a higher 3D
score. See `docs/CAVEATS.md` in the code repository before comparing numbers.

## Limitations

- Ground truth exists only at clip endpoints; intermediate trajectory quality
  and visibility are not directly supervised.
- The training run is seeded but not bit-reproducible because deterministic GPU
  algorithms and DataLoader worker seeding were not enforced.
- Training includes STIR Challenge 2024 patients that overlap the 2025 test
  collection at patient level.
- Performance outside laparoscopic tissue video, on unseen devices, or under
  different image statistics has not been established.
- Inference refinement iterations affect accuracy and latency; benchmark 1, 2,
  and 4 rather than assuming one operating point.

## Licence

The checkpoint is released under CC BY-NC 4.0 because it descends from
CoTracker3/LiteTracker weights. The accompanying code is separately MIT
licensed. Users must also follow the licences and terms of all upstream models
and datasets.

## Citation

```bibtex
@software{venkatesh2026stir,
  author = {Venkatesh, Danush Kumar and Liu, Peng and Speidel, Stefanie},
  title = {Verifier-Guided Multi-Teacher Distillation for Streaming Tissue Tracking},
  year = {2026},
  url = {https://github.com/danushkv/STIR2026_challenge}
}
```

## Acknowledgements

We thank the authors of CoTracker3, LiteTracker, Track-On/Track-On2/Track-On-R,
AllTracker, LocoTrack, MFT, STIRLoader, and STIRMetrics, as well as the STIR
Challenge organisers and dataset contributors. The full linked acknowledgement
list is maintained in the code repository README.