---
license: apache-2.0
tags:
- jax
- mashbox
- rigging
- animation
- gltf
---

<p align="center"><img alt="MASHBOX Labs" src="https://huggingface.co/mashbox-labs/Track2Rig/resolve/main/assets/banner-dark.png" width="100%"></p>

# Track2Rig

**Experimental**

Runs with `mashbox run track2rig` in the `mashbox` package, [release 0.1.10](https://pypi.org/project/mashbox/0.1.10/).

### Read the guide: [Track2Rig docs](https://mashbox.gg/docs/models/track2rig)

Our own model: dense 3D point tracks of a moving creature in, a rig out (bones, skin weights, per-frame bone transforms and a skeleton). Runs on a CPU. It does not work well on many clips yet. Newer weights will be published as dated versions. Trained from scratch by MASHBOX Labs. Code and weights: Apache-2.0.

<img alt="Track2Rig preview" width="720" src="https://huggingface.co/mashbox-labs/Track2Rig/resolve/main/assets/hero.gif">

<sub>The quickstart clip (grison_walk.npz, a grison walking, from the synthetic test set) rigged by Track2Rig 0.1.0 (weights track2rig-v2) on a CPU, in a side view and a three-quarter view. Every point is moved only by the predicted skin and bone transforms; lines are the predicted skeleton. Also as an [MP4 video](https://huggingface.co/mashbox-labs/Track2Rig/resolve/main/assets/hero.mp4) (the clip looped).</sub>

## Quickstart

On a CPU, with Python 3.12 (tested on x86-64 Linux). If your `python3` is older, as on a Google Cloud TPU VM, use the uv steps in the [install guide](https://mashbox.gg/docs/models/track2rig#install). The first run downloads `track2rig-v2.safetensors` (11 MB) from this repo at the revision pinned in `models.json` and checks its sha256.

```bash
git clone --branch v0.1.10 https://github.com/mashbox-labs/mashbox && cd mashbox
python3 -m venv .venv && . .venv/bin/activate
pip install ".[cpu,track2rig]"
mashbox run track2rig examples/track2rig/tracks/grison_walk.npz -o grison_rig
```

This writes `grison_rig/rig.json` and `grison_rig/rig.glb` (glTF 2.0: one joint per bone, top-4 skin, one animation).

## From video

Track2Rig starts from 3D point tracks. To make them from a video of one animal we use [TrackEverything](https://github.com/ayushjain1144/trackeverything) (CMU and Meta) at commit `f0c18dd9`, licensed **CC BY-NC 4.0 (non-commercial use only)**, through our wrapper `track2rig-video`. It needs Linux and an NVIDIA GPU. Steps and measured times: [video to tracks to rig](https://mashbox.gg/docs/models/track2rig#from-video). Tracks from any other 3D point tracker work through `track2rig-video align`.

## Results

| Chip | Measure | Result | Baseline |
|---|---|---|---|
| CPU (AMD EPYC 7502P) | Held-out reconstruction error, synthetic test set (32 clips, noisy and partly hidden tracks), % of body size, lower is better | 0.784 | SSDR, our 30-iteration NumPy/SciPy implementation, same bone count: 0.915 |
| CPU (AMD EPYC 7502P) | Joint error, synthetic test set, % of body size, lower is better | 6.24 | SSDR, same implementation and inputs: 7.73 |

## Files

| File | Format | Size | Note |
|---|---|---|---|
| `track2rig-v2.safetensors` | safetensors | 11.1 MB | float32, 55 tensors, 2,764,416 parameters |
| `examples/clips/glass-snake_idle.npz` | npz | 2.9 MB | example |
| `examples/clips/glass-snake_walk.npz` | npz | 2.9 MB | example |
| `examples/clips/grison_idle.npz` | npz | 2.2 MB | example |
| `examples/clips/grison_walk.npz` | npz | 2.1 MB | example |
| `examples/clips/pelican-spider_idle.npz` | npz | 2.5 MB | example |
| `examples/clips/pelican-spider_walk.npz` | npz | 2.7 MB | example |
| `examples/clips/reference.json` | json | 4.1 KB | example |

The weights file `track2rig-v2.safetensors` has sha256 `f8a79263dc21d2af078c123fbf76c2cba85df4694100f6a6c5e8f7608f6094b6`.

## Links

|  |  |
|---|---|
| Code | [mashbox-labs/mashbox](https://github.com/mashbox-labs/mashbox/tree/v0.1.10/src/mashbox/models/track2rig) |
| Guide | [mashbox.gg/docs/models/track2rig](https://mashbox.gg/docs/models/track2rig) |
| Original model | [mashbox-labs/Track2Rig](https://huggingface.co/mashbox-labs/Track2Rig) by MASHBOX Labs |
| Setup from a new Google account | [mashbox.gg/docs/google-cloud](https://mashbox.gg/docs/google-cloud) |
| Companies | hello@mashbox.gg |

## Licence

Code and weights: Apache-2.0 (MASHBOX Labs). The video step's tracker (TrackEverything) and its geometry model (Pi3) keep their own licences and are not in this repo.