README.md

---
license: cc-by-nc-4.0
tags:
- 3d-tracking
- point-tracking
- video
- arxiv:2609.30222
pipeline_tag: keypoint-detection
---

# TrackEverything

### Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

[Project Page](https://trackeverything.github.io/) · [Paper](https://arxiv.org/abs/2609.30222) · [Code](https://github.com/ayushjain1144/trackeverything)

TrackEverything tracks dense 3D points across long video sequences. Given RGB frames along with
depth and camera poses, it builds and maintains a persistent, de-duplicated 3D scene
representation. Trajectories are refined in 3D across temporal windows, and points are classified
as static or dynamic.

If depth and camera poses are not provided, they are estimated on the fly with feedforward 3D
reconstruction models ([VGGT-Ω](http://vggt-omega.github.io/) or [Pi3](https://github.com/yyfz/Pi3)).

This repo holds the trained checkpoint (`model.pth`). Access is gated: request access above, and
authenticate with `hf auth login` (older `huggingface_hub`: `huggingface-cli login`), or pass
`token=` explicitly, before downloading.

## Installation

Tested on Linux with Python 3.10, PyTorch 2.8, and CUDA 12.9.

```bash
git clone https://github.com/ayushjain1144/trackeverything.git
cd trackeverything

conda create -n trackeverything python=3.10 -y
conda activate trackeverything

pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu129
pip install torch_scatter -f https://data.pyg.org/whl/torch-2.8.0+cu129.html
pip install omegaconf hydra-core ipdb einops rich transformers wandb tqdm tensorboard iopath \
  numpy fsspec torchinfo h5py imageio imageio-ffmpeg matplotlib opencv-python viztracer pillow scipy \
  opt-einsum python-box line-profiler typed-argument-parser plyfile av rerun-sdk accelerate \
  huggingface_hub

# The repository directory is the package `trackeverything`; add its parent to PYTHONPATH:
export PYTHONPATH="$(dirname "$PWD"):$PYTHONPATH"
```

Inference needs nothing further; only training requires the deformable-attention CUDA extension
(see the repository README for training setup).

## Quick Start

```python
from trackeverything.demo_hf import TrackEverything

# Load the checkpoint from the Hub (or pass a local .pth path)
model = TrackEverything.from_pretrained(
    "katefgroup/TrackEverything",
    device="cuda",
    geometry_backend="vggt_omega",  # default; pass "pi3" to switch
)

# Run on a video; depth and camera poses are estimated automatically
out = model.infer(video="path/to/video.mp4")

# Compact scene tracks: one track per distinct scene point, O(T) in sequence length
tracks = out.tracks()
print(tracks.coords.shape)        # (N, T, 3)   world-frame 3D positions
print(tracks.visibs.shape)        # (N, T)      visibility logits
print(tracks.dyn_logits.shape)    # (N, T)      static vs. dynamic logits
print(tracks.src_time.shape)      # (N,)        birth frame of each point
print(tracks.cell_to_token.shape) # (T, H', W') which point each grid cell belongs to
```

Entries before a point is born are set to `-1000`.

To expand back to per-pixel tracks (grows with T squared, so keep clips short):

```python
dense_coords, dense_visibs, dense_dyn = tracks.to_dense()
# (1, T, H', W', T, 3), (1, T, H', W', T), (1, T, H', W', T)
```

## Input Options

```python
out = model.infer(video="path/to/video.mp4")        # video file
out = model.infer(video="path/to/frames/")          # folder of images
out = model.infer(video="https://example.com/v.mp4")# URL
out = model.infer(video=frames_array)               # (T, H, W, 3) uint8 or (T, 3, H, W) float [0,1]

# Temporal cropping / subsampling
out = model.infer(video="video.mp4", start_frame=10, frame_stride=3, max_frames=32)
```

## Providing Your Own Geometry

Pass depths, intrinsics and extrinsics together to skip the geometry backend:

```python
out = model.infer(
    video="path/to/video.mp4",
    depths=depths,          # (T, H, W) float32 numpy or torch
    intrinsics=intrinsics,  # (T, 3, 3)
    extrinsics=extrinsics,  # (T, 4, 4) world-to-camera
    conf_mask=conf_mask,    # optional (T, H, W) in [0, 1]
)
```

## 2D Query Points

Query points are given as `[frame_index, x_pixel, y_pixel]` and unprojected to 3D with the
estimated or supplied geometry. They are never merged with other scene points, so they can
retain higher resolution than the dense tracks.

```python
out = model.infer(
    video="path/to/video.mp4",
    query_points_2d=[
        [0, 100, 200],   # frame 0, pixel (100, 200)
        [5, 150, 300],   # frame 5, pixel (150, 300)
    ],
)

print(out.query_coords.shape)  # (B, T, N, 3)
print(out.query_visibs.shape)  # (B, T, N)
```

## Output

`model.infer()` returns a `TrackEverythingOutput` dataclass. Its native payload is
`prediction`, a set of per-window track snapshots that grows linearly with T; the track views
are built from it on demand.

| Field | Shape | Description |
|-------|-------|-------------|
| `prediction` | — | Per-window snapshots the track views are built from (`None` in sparse mode) |
| `query_coords` | `(B, T, N, 3)` | Query point 3D positions (if query points were given) |
| `query_visibs` | `(B, T, N)` | Query point visibility (if query points were given) |
| `norm_factor` | `float` | Scene normalization factor applied to depths/extrinsics |
| `depths` | `(T, H, W)` | Preprocessed depth maps at inference resolution |
| `conf_mask` | `(T, H, W)` | Geometry confidence in `[0, 1]`, or `None` |
| `intrinsics` | `(T, 3, 3)` | Camera intrinsics at inference resolution |
| `extrinsics` | `(T, 4, 4)` | Extrinsics after cam0-alignment and normalization |
| `rgb` | `(T, 3, H, W)` | Preprocessed RGB, float `[0, 1]` |
| `dense_source_mask` | `(T_src, H', W')` | Dense source validity mask |
| `voxel_size`, `dynamic_threshold` | `float` | Settings the model ran with |

Derived views:

| Accessor | Shape | Cost |
|----------|-------|------|
| `out.tracks()` | `SceneTokens`: `coords` `(N, T, 3)`, `visibs` / `dyn_logits` `(N, T)`, `cell_to_token` `(T, H', W')` | linear in T — preferred |
| `out.coords` | `(B, T_src, H, W, T_tgt, 3)` | built on first access, quadratic in T |
| `out.visibs` | `(B, T_src, H, W, T_tgt)` | built on first access, quadratic in T |
| `out.dynamic_logits` | `(B, T_src, H, W, T_tgt)` | built on first access, quadratic in T |

## License

Released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/): share and adapt
for non-commercial purposes with attribution.

## Citation

```bibtex
@misc{jain2026trackeverythinglonghorizondense,
      title={TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations},
      author={Ayush Jain and Sreeharsha Paruchuri and Ishita Gupta and Fan Zhang and Tanner Schmidt and Jakob Engel and Katerina Fragkiadaki and Adam W. Harley},
      year={2026},
      eprint={2609.30222},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.30222},
}
```