| --- |
| license: cc-by-nc-4.0 |
| library_name: transformers |
| base_model: depth-anything/trace-anything |
| tags: |
| - trace-anything |
| - 4d |
| - trajectory-field |
| - video |
| pipeline_tag: video-to-3d |
| --- |
| |
| # Trace-Anything — transformers format (original code) |
|
|
| Unofficial transformers port of **Trace Anything** (`depth-anything/trace-anything`, `trace_anything.pt`) that runs |
| the **original** code. Loads with `trust_remote_code`. `model.safetensors` holds the checkpoint's tensors under their |
| original names (strict load, every tensor checked equal). |
|
|
| > **Not an official release.** Trace Anything was created by Xinhang Liu, Yuxi Xiao, Donny Y. Chen, Jiashi Feng, |
| > Yu-Wing Tai, Chi-Keung Tang and Bingyi Kang — https://github.com/ByteDance-Seed/TraceAnything, paper *"Trace |
| > Anything: Representing Any Video in 4D via Trajectory Fields"* (arXiv 2510.13802). |
|
|
| ## What it is |
|
|
| A video in, a trajectory field out: for every pixel of every frame, the control points of a B-spline 3D trajectory over |
| time (and their confidence), in one forward pass. `TraceAnythingModel` holds the official `TraceAnything`'s submodules |
| (`encoder`: CroCo ViT-L, `decoder`: 24-block trace decoder, `ds_head_time` / `ds_head_track`: DPT heads) and runs the |
| official `TraceAnything.forward`; `TraceAnythingOutput` stacks its per-view results (`time`, `ctrl_pts3d`, |
| `ctrl_conf`, `track_pts3d`, `track_conf`) and keeps the original list as `predictions`. Code vendored from |
| ByteDance-Seed/TraceAnything @ 54677b5e into `ta_layers.py` / `ta_heads.py` / `ta_model.py` (imports only). |
| `TraceAnythingProcessor` (`processing_trace_anything.py`, `processor_config.json`) wraps `TraceAnythingImageProcessor`, |
| `scripts/infer.py`'s `_load_images`. |
|
|
| **Local head left out.** `configs/eval.yaml` enables a second ("local") trajectory head, but the released checkpoint |
| holds no weights for it: `infer.py` loads with `strict=False`, so that head runs with random initial weights and its |
| `*_local` outputs carry no trained signal. This package sets `head_args.with_local_head: false`; all other outputs are |
| those of the original. |
|
|
| Verified against the official code (same process, V100; `infer.py`'s model build and `_load_images`): preprocessing |
| and every output (`time`, control points, confidences, evaluated tracks; also as a tuple) are **bit-identical** on the |
| repo's example scenes (corgi 2 frames, cabinet 11, the first 8 / 16 frames of elephant and monkeys), with the |
| original's `pytorch_naive` attention. `flash_attention` (eval.yaml) and `pytorch_auto` have no bf16 kernel on a V100, |
| so they were not run; the vendored code is the same for all three. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoProcessor |
| |
| model = AutoModel.from_pretrained("Trace-Anything", trust_remote_code=True).eval().cuda() |
| processor = AutoProcessor.from_pretrained("Trace-Anything", trust_remote_code=True) |
| inputs = processor(images=frame_paths, return_tensors="pt").to("cuda") # the frames of one video, in order |
| outputs = model(**inputs) # outputs.ctrl_pts3d: (N, 10, H, W, 3); outputs.track_pts3d: (N, N, H, W, 3) |
| ``` |
|
|
| - Attention kernel: `config.encoder_args["attn_implementation"]` / `config.decoder_args[...]` — `"flash_attention"` |
| (eval.yaml; needs compute capability 8.0+), `"pytorch_auto"` or `"pytorch_naive"` (the original's options). Change |
| both on the config before `from_pretrained(..., config=config)` on older GPUs. |
| - One video per call. `infer.py` keeps every (N // 39)-th frame of scenes with more than 40 frames; do that before |
| the processor if needed. |
| - `return_dict=False` returns a tuple. |
|
|
| Runtime packages beyond torch / transformers: `einops`, `opencv-python`, `torchvision` (as the original; the |
| original's optional CUDA RoPE extension is not used). |
|
|
| ## Licensing |
|
|
| - Weights: **CC-BY-NC-4.0** (non-commercial) — [`LICENSE`](LICENSE), with the upstream notice [`LICENSE_MODEL.txt`](LICENSE_MODEL.txt). |
| - Code (vendored files and the port): **Apache-2.0** — [`LICENSE.code`](LICENSE.code). |
| - Attribution, sources and the changes made here: [`NOTICE`](NOTICE). |
|
|
| ## Citation |
|
|
| If you use this model, please cite the original work, as its authors ask: |
|
|
| ```bibtex |
| @misc{liu2025traceanythingrepresentingvideo, |
| title={Trace Anything: Representing Any Video in 4D via Trajectory Fields}, |
| author={Xinhang Liu and Yuxi Xiao and Donny Y. Chen and Jiashi Feng and Yu-Wing Tai and Chi-Keung Tang and Bingyi Kang}, |
| year={2025}, |
| eprint={2510.13802}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2510.13802}, |
| } |
| ``` |
|
|
