491-D SMPL-H motion capture from monocular RGB video, with native flow-matching training.
Project / Paper updates · Original GitHub · Base Checkpoint · Latest Checkpoint
Official source: [FlowHMR](https://github.com/flowhmr/flowhmr), revision `f12e6a2d46a63d771a66dbb6b5598a1be65b9eea`. Only the formal 0.46B, 491-D release is supported. The retired HYMotionV2M model is not included. **Tasks:** Monocular Motion CaptureVisual Results · Overview · Quick Start · Evaluation · Motion Representation
## Visual Results ### Supplied-video demo — Base [Input video](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Base/resolve/main/media/demo_input.mp4) · [Download demo](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Base/resolve/main/media/demo_base_mesh.mp4) · [Render provenance](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Base/resolve/main/media/demo_base_mesh.json) · [Other checkpoint on the same video](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Latest#visual-results) **Checkpoint:** [Motius-FlowHMR-Base](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Base), the official Base flow-matching release. This video is an actual Base prediction. - Official checkpoint SHA256: `77b903ba545669bafd6f7709c0f1748f5fa030a4f25f67392c8c082a21f5e672`. - Motius inference safetensors SHA256: `f8dca504bfe1682ab0a8ec6cb99e555b4c93e77220cc5c80636a87f0492ced46`. - 750 frames, 25 seconds, 30 fps; seed 0, 20 Euler steps, CFG 1. - VGGT camera sampling every 100 frames (9 keyframes interpolated to 750). Base and Latest use the exact same cached tracking boxes, SAM features and camera estimates (shared frontend SHA256: `c26b81670919b5b78b3661a022e3e7db294078cfbb464825fdd8246ea5b8419b`). Both use the same seed and sampling settings. A single camera framing fits both predicted sequences, so the display scale and view are identical. The left panel displays the supplied video with the Motius frontend's tracking bbox. The right panel reconstructs SMPL-H vertices from the saved predicted rotations, shape and translation, using Motius's bundled implementation. Three.js provides smooth normals, depth-tested surfaces, lighting and floor shadows. The fixed camera fits the complete sequence; horizontal root motion is centered for display, while vertical motion is preserved. This is a world view, not a mesh reprojection into the source camera. Original audio is retained. No external FlowHMR checkout is needed for inference or mesh reconstruction. Earlier parity-validation clips: [Base, 30 frames](https://github.com/user-attachments/assets/791ec4c4-638d-478c-b79d-7ddb09d3582b) · [Latest, 30 frames](https://github.com/user-attachments/assets/ca588fd7-52c8-49a8-af11-e0def808a715). These use a different input clip. ## Model Overview ### Task APIs | Task | Pipeline API | Evaluation and examples | | --- | --- | --- | | Monocular Motion Capture | `infer_monocular_motion_capture` | [Benchmark and examples](https://huggingface.co/spaces/ZeyuLing/monocular-motion-capture-leaderboard) | ### Frame-Rate Contract | Clock | Rate | | --- | --- | | Training motion | 30 fps, 360-frame windows | | Public preview | 30 fps native | Training FPS is the checkpoint's native temporal clock. Preview FPS only controls media playback; any conversion listed above preserves duration. | Variant | Official checkpoint SHA256 | Use | | --- | --- | --- | | Base | `77b903ba545669bafd6f7709c0f1748f5fa030a4f25f67392c8c082a21f5e672` | Base flow-matching training and inference | | Latest | `b7eb06b119feaf5ad7287bafdb2fff38d42dfc22ea9d256f8cdd2ada26dfc53a` | Inference after upstream PHC+ GRPO post-training | Both artifacts contain inference safetensors, configuration, motion statistics, YOLOX, VGGT-Omega, SAM-3D-Body and MHR frontend weights, source provenance and component licenses. Licensed SMPL-H and the SMPL joint regressor are supplied separately. Network code ships inside Motius; no external checkout is needed. This release implements base training; PHC+ GRPO training is outside its scope. ## Quick Start ```bash pip install -e '.[flowhmr,video-to-motion]' ``` ```python from motius import Pipeline pipe = Pipeline.from_pretrained( "ZeyuLing/Motius-FlowHMR-Latest", # or ZeyuLing/Motius-FlowHMR-Base bundle_kwargs=dict( body_model_path="outputs/checkpoints/flowhmr/body_models/smplh/neutral/model.npz", j_regressor_path="outputs/checkpoints/flowhmr/body_models/smpl_neutral_J_regressor.pt", device="cuda", ), ) result = pipe.infer_monocular_motion_capture( "input.mp4", work_dir="outputs/inference/flowhmr", seeds=[0], ) ``` Provide the official AMASS neutral SMPL-H model (52 joints, 16 shape parameters). `infer_from_feature(tokens, camera_RT)` accepts precomputed `(T,3072)` tokens and OpenCV world-to-camera transforms at 30 fps. `infer_v2m` transcodes video, tracks a person with YOLOX, predicts VGGT cameras and extracts SAM features. `postprocess=True` enables upstream contact/IK cleanup for one seed. The CLI is `tools/infer_flowhmr.py`; local artifact export is `tools/export_flowhmr_hf.py`. No assets are downloaded implicitly during training. ### Training **Data.** The official training corpus is not public. Use your own official-format WebDataset shards: each sample groups `motion.npz`, `camera.npz`, `bbox.npz`, `feature.pt` and metadata as specified in the [upstream data guide](https://github.com/flowhmr/flowhmr/blob/main/docs/resources.md). The adapter uses upstream camera/world conversion, SMPL-H FK, 491-D encoding, 360-frame crop/padding, and 10% feature dropout. It keeps all hand joints. `FlowHMRDataset` also accepts the upstream `roots` list for directory samples. Replacement data constitute a new training run, not an exact dataset reproduction. **Precision.** FP32 (`mixed_precision='no'`), matching the official base launch. **Objective.** Official x1 flow matching with logit-normal time sampling (mean −0.8, std 0.8), SmoothL1 losses for root XZ velocity, root Y position, local joint positions, rotations, shape and contact, plus FK rotation and consistency losses. Weights are 10/10/10/10/1/4/5/5. Loss masks exclude padding. The native runner manages AdamW, gradient clipping (10), cosine LR 1e-4 → 1e-5, and checkpoint state. The public recipe uses 250,000 iterations, equivalent to the official 25 epochs of 10,000 iterations. Eight processes give the official global batch size 64; another process count changes the recipe. ```bash export MOTIUS_SMPLH_MODEL=outputs/checkpoints/flowhmr/body_models/smplh/neutral/model.npz export MOTIUS_J_REGRESSOR=outputs/checkpoints/flowhmr/body_models/smpl_neutral_J_regressor.pt export MOTIUS_FLOWHMR_SHARDS='data/flowhmr/train-{00000..00025}.tar' accelerate launch --num_processes 8 tools/train.py configs/flowhmr/train_flowhmr.py # Resume model, optimizer, scheduler and runner state: accelerate launch --num_processes 8 tools/train.py configs/flowhmr/train_flowhmr.py --auto-resume --load-scope full ``` **Checkpoints.** Saved every 10,000 steps, five retained, plus final state under `outputs/training/flowhmr`. `MOTIUS_PRETRAINED_WEIGHTS` optionally initializes from an official `.ckpt`; it is not a full-state resume. WebDataset is resampled and worker/rank partitioned; resume restores training state, not the exact stream cursor. `bundle.save_pretrained(directory)` exports safetensors, config, statistics, provenance and licenses for `Pipeline.from_pretrained`. This native recipe covers base pretraining. The official GRPO SDE primitives are retained in the vendor, but PHC+ simulator post-training is not advertised as a Motius-native training recipe in this release. ## Evaluation Results > **Canonical metrics.** Public results are tied to the sources below. Motius/uTMR FID always means per-sample L2-normalized embedding-space FID; `—` means the normalized value has not been recomputed. Historical raw-space FID is never substituted. | Task | Canonical result source | Protocol | | --- | --- | --- | | Monocular Motion Capture | [Published results](https://github.com/ZeyuLing/Motius/blob/main/docs/leaderboards/hf_space_monocular_capture/monocular_capture_results.json) | 3DPW camera/world-space capture metrics | ### Source parity The [validation record](https://github.com/ZeyuLing/Motius/blob/main/docs/model_zoo/flowhmr_validation.json) contains checkpoint hashes, Hub revisions, numerical comparison results and the test boundaries. The full 0.46B base network was compared against independently imported pinned upstream classes using the licensed 6890-vertex SMPL-H model. For identical inputs and RNG states, the training loss, every component loss and 304 gradient tensors matched exactly (`rtol=0, atol=0`). Both base and latest four-frame, 20-step ODE outputs matched exactly for all nine returned tensor fields. A 30-frame real-video comparison also matched detection/tracking boxes, VGGT intrinsics/extrinsics, SAM features, base/latest sampling, floor alignment and FK joints exactly. Both Hub artifacts were downloaded into a fresh cache and reloaded through `Pipeline.from_pretrained`; every output matched the local export exactly. These checks establish numerical equivalence for the tested inputs, not every possible input or reproduction of a complete training run. Small-network tests cover padding, strict checkpoint loading, serialization, two-process DDP synchronization and official-format WebDataset preprocessing. The native runner completed two training steps and resumed the full model, optimizer and scheduler state for the third step. ### Upstream reported results | Variant | PHC+ tracking success on Wild-4K | | --- | --- | | Base | 78.79% | | Latest | 82.47% | Source: [pinned upstream Model Zoo](https://github.com/flowhmr/flowhmr/blob/f12e6a2d46a63d771a66dbb6b5598a1be65b9eea/README.md#model-zoo). These are upstream simulation results, not Motius measurements or joint-error scores. The training corpus is not public. No full-scale training or Motius leaderboard result is claimed. ```bash MOTIUS_FLOWHMR_UPSTREAM=/path/to/pinned/flowhmr pytest -q tests/test_flowhmr.py python tools/audit_training_release.py python tools/audit_training_docs.py ``` ## Motion Representation The native 491-D state contains smooth root XZ velocity and Y position, 52 local joint positions, 52 six-dimensional rotations, 16 shape coefficients and four foot-contact channels. Video conditions contain 3072 SAM features and nine camera-rotation values. The native clock is 30 fps and training windows are 360 frames. Hands remain in the native output; the shared capture exposes the standard 22 body joints and preserves all native rotations. The shared result contains Y-up world joints and SMPL-H parameters. Camera-space joints and metric camera-to-motion-world extrinsics are unavailable: VGGT camera translation is not metrically aligned with the independently reconstructed motion world. Output availability explicitly records this boundary. ## Citation and License This implementation originates from [FlowHMR](https://github.com/flowhmr/flowhmr). Follow the upstream project for publication and citation updates; no publication identifier is invented for this release. FlowHMR permits educational, research and non-profit use only. Its terms apply independently of the framework license. See [ATTRIBUTIONS](https://github.com/ZeyuLing/Motius/blob/main/motius/models/flowhmr/ATTRIBUTIONS.md), [FlowHMR license](https://github.com/ZeyuLing/Motius/blob/main/motius/models/flowhmr/vendor/UPSTREAM_LICENSE), [SAM license](https://github.com/ZeyuLing/Motius/blob/main/motius/models/flowhmr/vendor/SAM3D_LICENSE), [VGGT-Omega license](https://github.com/ZeyuLing/Motius/blob/main/motius/models/flowhmr/vendor/VGGT_OMEGA_LICENSE), [DINOv3 license](https://github.com/ZeyuLing/Motius/blob/main/motius/models/flowhmr/vendor/DINOV3_LICENSE.md), [YOLOX license](https://github.com/ZeyuLing/Motius/blob/main/motius/models/flowhmr/vendor/YOLOX_LICENSE) and [MHR license](https://github.com/ZeyuLing/Motius/blob/main/motius/models/flowhmr/vendor/MHR_LICENSE). SMPL-H assets require their separate license and are not redistributed. ---