MocapAnything-MLX / README.md
Fendercorp's picture
Upload 6 files
fb43850 verified
|
Raw History Blame Contribute Delete
2.15 kB
---
license: mit
library_name: mlx
pipeline_tag: other
tags:
- motion-capture
- mocap
- animation
- retargeting
- mlx
- apple-silicon
base_model:
- kehong/MoCapAnythingV2-weights
- facebook/dinov2-large
- google-t5/t5-base
---
# MocapAnything-MLX
MoCapAnything V2 (`video2pose2rot`) repacked for 3Dify Studio's native MLX path on Apple Silicon.
It turns a monocular video plus a target skeleton into per-joint rotations on that skeleton. No Python
is needed at runtime.
| Folder | Contents | Size |
|---|---|---|
| `mocap/model.safetensors` | video2pose + pose2rot, F32 (38.5M params) | 154 MB |
| `dinov2-large-224/model.safetensors` | DINOv2 ViT-L/14 image encoder, F16, position table pre-interpolated to 224 px (257 tokens) | 607 MB |
| `t5-base-encoder/model.safetensors` | T5-base encoder only, F16, for joint-name embeddings | 219 MB |
| `t5-base-encoder/vocab.json` | SentencePiece unigram vocab as `[piece, score]` rows | 1 MB |
## Conversion
- `mocap/` is the `video2pos2rot_epoch60.pt` state dict saved as safetensors. The key names are unchanged.
- `dinov2-large-224/` is `facebook/dinov2-large` with `embeddings.position_embeddings` replaced by
HF's `interpolate_pos_encoding(·, 224, 224)` output, and `mask_token` dropped.
- `t5-base-encoder/` holds the `encoder.*` and `shared.weight` tensors of `t5-base`, plus
`tokenizer.json` → `model.vocab`.
## Verified (MLX Swift port vs reference)
- T5 joint-name embeddings match the shipped `species_info_dict` vectors (min cosine 1.0000).
- DINOv2 features match HF `transformers` on the same pixels (min token cosine 0.9999).
- Jaguar-Run (y0 reference, y60 input video): MPJPE 0.058 in normalised units, against 0.50 for
holding the reference pose.
## Licences
- MoCapAnything V2 code and weights: MIT, from [phongdaot/MocapAnything](https://github.com/phongdaot/MocapAnything) and
[kehong/MoCapAnythingV2-weights](https://huggingface.co/kehong/MoCapAnythingV2-weights).
- DINOv2: Apache-2.0, from [facebook/dinov2-large](https://huggingface.co/facebook/dinov2-large).
- T5: Apache-2.0, from [google-t5/t5-base](https://huggingface.co/google-t5/t5-base).