--- license: mit library_name: mlx pipeline_tag: other tags: - motion-capture - mocap - animation - retargeting - mlx - apple-silicon base_model: - kehong/MoCapAnythingV2-weights - facebook/dinov2-large - google-t5/t5-base --- # MocapAnything-MLX MoCapAnything V2 (`video2pose2rot`) repacked for 3Dify Studio's native MLX path on Apple Silicon. It turns a monocular video plus a target skeleton into per-joint rotations on that skeleton. No Python is needed at runtime. | Folder | Contents | Size | |---|---|---| | `mocap/model.safetensors` | video2pose + pose2rot, F32 (38.5M params) | 154 MB | | `dinov2-large-224/model.safetensors` | DINOv2 ViT-L/14 image encoder, F16, position table pre-interpolated to 224 px (257 tokens) | 607 MB | | `t5-base-encoder/model.safetensors` | T5-base encoder only, F16, for joint-name embeddings | 219 MB | | `t5-base-encoder/vocab.json` | SentencePiece unigram vocab as `[piece, score]` rows | 1 MB | ## Conversion - `mocap/` is the `video2pos2rot_epoch60.pt` state dict saved as safetensors. The key names are unchanged. - `dinov2-large-224/` is `facebook/dinov2-large` with `embeddings.position_embeddings` replaced by HF's `interpolate_pos_encoding(·, 224, 224)` output, and `mask_token` dropped. - `t5-base-encoder/` holds the `encoder.*` and `shared.weight` tensors of `t5-base`, plus `tokenizer.json` → `model.vocab`. ## Verified (MLX Swift port vs reference) - T5 joint-name embeddings match the shipped `species_info_dict` vectors (min cosine 1.0000). - DINOv2 features match HF `transformers` on the same pixels (min token cosine 0.9999). - Jaguar-Run (y0 reference, y60 input video): MPJPE 0.058 in normalised units, against 0.50 for holding the reference pose. ## Licences - MoCapAnything V2 code and weights: MIT, from [phongdaot/MocapAnything](https://github.com/phongdaot/MocapAnything) and [kehong/MoCapAnythingV2-weights](https://huggingface.co/kehong/MoCapAnythingV2-weights). - DINOv2: Apache-2.0, from [facebook/dinov2-large](https://huggingface.co/facebook/dinov2-large). - T5: Apache-2.0, from [google-t5/t5-base](https://huggingface.co/google-t5/t5-base).