--- library_name: "fastforks" pipeline_tag: "keypoint-detection" license: "mit" tags: - "tracking" - "point-tracking" - "tracking-any-point" - "video" - "cuda" - "pytorch" - "fastforks" - "fastforks-card" base_model: - "aharley/alltracker" --- # AllTracker (fastforks) AllTracker (Harley et al., ICCV 2025): tracking every pixel of a query frame through a video offline, as long-range optical flow. A ConvNeXt (tiny: a residual CNN) encodes every frame at 1/8; 16-frame windows (stride 8) refine the flow and visibility of all anchor pixels to all window frames at once, with a correlation pyramid, 2D ConvNeXt blocks over space and 1D blocks over time, then an 8x convex upsampling. Point queries read the dense result. This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: . Kernels for the `kernels` backend: . Original code: at `e7553135e7b3`. Author weight repos: . Task: tracking. Tier: experimental. Outputs: `tracks`, `visibility`. Backends: `portable`, `kernels`. Input: a video (its frames in order) and query points `(t, x, y)`. Output: the position of every query in every frame and the probability that it is visible there. ## Variants Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/tracking/models/alltracker/card.toml). | variant | weights (author link) | SHA-256 | weights licence | commercial use | |---|---|---|---|---| | `alltracker@1` | [alltracker.pth](https://huggingface.co/aharley/alltracker/resolve/c8ba31225828cab9ec512d0627f0f826e641a877/alltracker.pth) | `ffd9ebcfb6d2` | MIT | yes | | `alltracker-tiny@1` | [alltracker_tiny.pth](https://huggingface.co/aharley/alltracker/resolve/c8ba31225828cab9ec512d0627f0f826e641a877/alltracker_tiny.pth) | `43991b4fcee5` | MIT | yes | ## Licence Original-code strings, recorded as they are (research-use package; check them before any other use). Code: MIT. Licence files: `LICENSE`. Training data: Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), FlyingChairs (Freiburg), Spring (scene flow / optical flow / stereo, University of Stuttgart), HCI HD1K optical flow benchmark, KITTI stereo / optical flow 2012 / 2015, TartanAir, not stated by the authors (Upstream README: stage 1 on Kubric MOVi-F, stage 2 on a mix of point tracking data (Kubric, PointOdyssey, Dynamic Replica and point tracks made from FlyingThings3D, Monkaa, Driving, SPRING) and optical flow data (FlyingChairs, FlyingThings3D, Monkaa, Driving, AutoFlow, SPRING, VIPER, HD1K, KITTI, TartanAir). Kubric, PointOdyssey, Dynamic Replica, AutoFlow and VIPER have no dataset entry here yet ("unstated").). Licence evidence: ## Input size any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `alltracker` 3000x4000 (5.5 GiB peak allocated, 5.9 reserved); `alltracker-tiny` 3000x4000 (5.5 GiB peak allocated, 5.9 reserved) ## Speed and memory ### Tracking, synthetic, 480x854 | model | time (ms): original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `alltracker` | 485 | 391 | 221 | 2.2x | 1.8x | 2100 / 2318 | 993 / 1238 | 1076 / 1288 | | `alltracker-tiny` | 424 | 330 | 217 | 2.0x | 1.5x | 1995 / 2200 | 919 / 1178 | 978 / 1166 | ### Startup and first call | model | load (s): original | portable | kernels | first call (s): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | | `alltracker` | 1.68 | 0.22 | 0.98 | 1.31 | 0.85 | 0.45 | | `alltracker-tiny` | 1.22 | 0.06 | 0.96 | 1.40 | 0.73 | 0.54 | ### torch.compile and TensorRT | model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `alltracker` | 480x854 | 391 ms | 284 ms | planned for 0.2.1 | 221 ms | 105 s | – | 993 / 1238 | 1273 / 1358 | 1076 / 1288 | – | | `alltracker-tiny` | 480x854 | 330 ms | 229 ms | planned for 0.2.1 | 217 ms | 103 s | – | 919 / 1178 | 1198 / 1338 | 978 / 1166 | – | Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB). Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's. ## Accuracy ### On public datasets TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the original code named under Details, same checkpoint and protocol. | variant | dataset | original fp32 | original fp16 | portable | kernels | note | |---|---|---|---|---|---|---| | `alltracker` | TAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA % | 67.45 / 78.66 / 90.82 | 67.41 / 78.65 / 90.69 | 67.42 / 78.71 / 90.70 | 67.38 / 78.63 / 90.65 | | | `alltracker` | TAP-Vid DAVIS (query first), AJ / delta_avg / OA % | 64.17 / 77.00 / 89.49 | | 64.14 / 77.04 / 89.44 | 64.15 / 77.04 / 89.44 | | | `alltracker-tiny` | TAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA % | 65.33 / 76.69 / 88.14 | | 65.36 / 76.66 / 88.14 | 65.34 / 76.66 / 88.10 | | Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13. ### Parity with the original code Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 2.3e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0019 and p99 <= 0.0099 (absolute error of the output `tracks`). ## Limitations - Offline: the features of every frame are kept in memory (about 2.5 MB per frame in float16 at 384 x 512); each query frame costs a forward pass (and a backward pass from frame 0 if it is not the first). - Frames are resized to 384 x 512 (upstream's default evaluation size), whatever their size and aspect ratio. Upstream reports higher accuracy at larger sizes (TAP-Vid DAVIS AJ 65.9 at 448 x 768, 67.2 at 768 x 1024); the size is a card parameter. - Tracks are sampled bilinearly at the query position; upstream's evaluation rounds the query to the nearest pixel. - One video per call; all queries of a call share the video. ## Usage ```python from fastforks import tracking model = tracking.load("alltracker") # mode="auto": the preferred backend from recorded measurements result = model(frames, [[0, x, y]]) # frames: the video's images in order; queries (t, x, y) result.tracks # [T, Q, 2] float32: x, y in input pixels; result.visible [T, Q] bool result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16') ``` Install with `pip install fastforks` (see ). Other variants: `alltracker`, `alltracker-tiny`. ## Kernels backend `tracking.load("alltracker", mode="kernels")` runs our CUDA kernels (`fvm_conv`). It needs a prebuilt kernel for your GPU architecture (downloaded from on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.