MocapAnything-MLX / README.md
Fendercorp's picture
Upload 6 files
fb43850 verified
|
Raw History Blame Contribute Delete
2.15 kB
metadata
license: mit
library_name: mlx
pipeline_tag: other
tags:
  - motion-capture
  - mocap
  - animation
  - retargeting
  - mlx
  - apple-silicon
base_model:
  - kehong/MoCapAnythingV2-weights
  - facebook/dinov2-large
  - google-t5/t5-base

MocapAnything-MLX

MoCapAnything V2 (video2pose2rot) repacked for 3Dify Studio's native MLX path on Apple Silicon. It turns a monocular video plus a target skeleton into per-joint rotations on that skeleton. No Python is needed at runtime.

Folder Contents Size
mocap/model.safetensors video2pose + pose2rot, F32 (38.5M params) 154 MB
dinov2-large-224/model.safetensors DINOv2 ViT-L/14 image encoder, F16, position table pre-interpolated to 224 px (257 tokens) 607 MB
t5-base-encoder/model.safetensors T5-base encoder only, F16, for joint-name embeddings 219 MB
t5-base-encoder/vocab.json SentencePiece unigram vocab as [piece, score] rows 1 MB

Conversion

  • mocap/ is the video2pos2rot_epoch60.pt state dict saved as safetensors. The key names are unchanged.
  • dinov2-large-224/ is facebook/dinov2-large with embeddings.position_embeddings replaced by HF's interpolate_pos_encoding(·, 224, 224) output, and mask_token dropped.
  • t5-base-encoder/ holds the encoder.* and shared.weight tensors of t5-base, plus tokenizer.json → model.vocab.

Verified (MLX Swift port vs reference)

  • T5 joint-name embeddings match the shipped species_info_dict vectors (min cosine 1.0000).
  • DINOv2 features match HF transformers on the same pixels (min token cosine 0.9999).
  • Jaguar-Run (y0 reference, y60 input video): MPJPE 0.058 in normalised units, against 0.50 for holding the reference pose.

Licences