---
license: cc-by-nc-4.0
library_name: transformers
base_model: depth-anything/trace-anything
tags:
- trace-anything
- 4d
- trajectory-field
- video
pipeline_tag: video-to-3d
---

# Trace-Anything — transformers format (original code)

Unofficial transformers port of **Trace Anything** (`depth-anything/trace-anything`, `trace_anything.pt`) that runs
the **original** code. Loads with `trust_remote_code`. `model.safetensors` holds the checkpoint's tensors under their
original names (strict load, every tensor checked equal).

> **Not an official release.** Trace Anything was created by Xinhang Liu, Yuxi Xiao, Donny Y. Chen, Jiashi Feng,
> Yu-Wing Tai, Chi-Keung Tang and Bingyi Kang — https://github.com/ByteDance-Seed/TraceAnything, paper *"Trace
> Anything: Representing Any Video in 4D via Trajectory Fields"* (arXiv 2510.13802).

## What it is

A video in, a trajectory field out: for every pixel of every frame, the control points of a B-spline 3D trajectory over
time (and their confidence), in one forward pass. `TraceAnythingModel` holds the official `TraceAnything`'s submodules
(`encoder`: CroCo ViT-L, `decoder`: 24-block trace decoder, `ds_head_time` / `ds_head_track`: DPT heads) and runs the
official `TraceAnything.forward`; `TraceAnythingOutput` stacks its per-view results (`time`, `ctrl_pts3d`,
`ctrl_conf`, `track_pts3d`, `track_conf`) and keeps the original list as `predictions`. Code vendored from
ByteDance-Seed/TraceAnything @ 54677b5e into `ta_layers.py` / `ta_heads.py` / `ta_model.py` (imports only).
`TraceAnythingProcessor` (`processing_trace_anything.py`, `processor_config.json`) wraps `TraceAnythingImageProcessor`,
`scripts/infer.py`'s `_load_images`.

**Local head left out.** `configs/eval.yaml` enables a second ("local") trajectory head, but the released checkpoint
holds no weights for it: `infer.py` loads with `strict=False`, so that head runs with random initial weights and its
`*_local` outputs carry no trained signal. This package sets `head_args.with_local_head: false`; all other outputs are
those of the original.

Verified against the official code (same process, V100; `infer.py`'s model build and `_load_images`): preprocessing
and every output (`time`, control points, confidences, evaluated tracks; also as a tuple) are **bit-identical** on the
repo's example scenes (corgi 2 frames, cabinet 11, the first 8 / 16 frames of elephant and monkeys), with the
original's `pytorch_naive` attention. `flash_attention` (eval.yaml) and `pytorch_auto` have no bf16 kernel on a V100,
so they were not run; the vendored code is the same for all three.

## Usage

```python
import torch
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("Trace-Anything", trust_remote_code=True).eval().cuda()
processor = AutoProcessor.from_pretrained("Trace-Anything", trust_remote_code=True)
inputs = processor(images=frame_paths, return_tensors="pt").to("cuda")   # the frames of one video, in order
outputs = model(**inputs)    # outputs.ctrl_pts3d: (N, 10, H, W, 3); outputs.track_pts3d: (N, N, H, W, 3)
```

- Attention kernel: `config.encoder_args["attn_implementation"]` / `config.decoder_args[...]` — `"flash_attention"`
  (eval.yaml; needs compute capability 8.0+), `"pytorch_auto"` or `"pytorch_naive"` (the original's options). Change
  both on the config before `from_pretrained(..., config=config)` on older GPUs.
- One video per call. `infer.py` keeps every (N // 39)-th frame of scenes with more than 40 frames; do that before
  the processor if needed.
- `return_dict=False` returns a tuple.

Runtime packages beyond torch / transformers: `einops`, `opencv-python`, `torchvision` (as the original; the
original's optional CUDA RoPE extension is not used).

## Licensing

- Weights: **CC-BY-NC-4.0** (non-commercial) — [`LICENSE`](LICENSE), with the upstream notice [`LICENSE_MODEL.txt`](LICENSE_MODEL.txt).
- Code (vendored files and the port): **Apache-2.0** — [`LICENSE.code`](LICENSE.code).
- Attribution, sources and the changes made here: [`NOTICE`](NOTICE).

## Citation

If you use this model, please cite the original work, as its authors ask:

```bibtex
@misc{liu2025traceanythingrepresentingvideo,
      title={Trace Anything: Representing Any Video in 4D via Trajectory Fields}, 
      author={Xinhang Liu and Yuxi Xiao and Donny Y. Chen and Jiashi Feng and Yu-Wing Tai and Chi-Keung Tang and Bingyi Kang},
      year={2025},
      eprint={2510.13802},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.13802}, 
}
```