Update supplied-video FlowHMR mesh and tracking demo
Browse files- .gitattributes +3 -0
- README.md +21 -13
- media/demo_input.mp4 +3 -0
- media/demo_latest_mesh.json +32 -0
- media/demo_latest_mesh.mp4 +3 -0
- media/demo_latest_mesh_preview.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
media/demo_input.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
media/demo_latest_mesh.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
media/demo_latest_mesh_preview.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -9,7 +9,7 @@ tags:
|
|
| 9 |
- flow-matching
|
| 10 |
---
|
| 11 |
|
| 12 |
-
<h1 align="center">FlowHMR Model Card</h1>
|
| 13 |
|
| 14 |
<p align="center"><strong>491-D SMPL-H motion capture from monocular RGB video, with native flow-matching training.</strong></p>
|
| 15 |
|
|
@@ -38,25 +38,33 @@ Official source: [FlowHMR](https://github.com/flowhmr/flowhmr), revision
|
|
| 38 |
|
| 39 |
## Visual Results
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
-
|
| 44 |
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
| Monocular Motion Capture | Monocular RGB video | <video src="https://github.com/user-attachments/assets/791ec4c4-638d-478c-b79d-7ddb09d3582b" controls></video> | [MP4](https://github.com/user-attachments/assets/791ec4c4-638d-478c-b79d-7ddb09d3582b) |
|
| 48 |
|
| 49 |
-
|
|
|
|
| 50 |
|
| 51 |
-
|
|
|
|
|
|
|
|
|
|
| 52 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
| <video src="https://github.com/user-attachments/assets/791ec4c4-638d-478c-b79d-7ddb09d3582b" controls></video> | <video src="https://github.com/user-attachments/assets/ca588fd7-52c8-49a8-af11-e0def808a715" controls></video> |
|
| 57 |
|
| 58 |
-
Real-video previews are generated with the public Motius pipeline, seed 0,
|
| 59 |
-
20 Euler steps, CFG 1 and the official first-frame floor alignment.
|
| 60 |
|
| 61 |
## Model Overview
|
| 62 |
|
|
|
|
| 9 |
- flow-matching
|
| 10 |
---
|
| 11 |
|
| 12 |
+
<h1 align="center">FlowHMR Latest Model Card</h1>
|
| 13 |
|
| 14 |
<p align="center"><strong>491-D SMPL-H motion capture from monocular RGB video, with native flow-matching training.</strong></p>
|
| 15 |
|
|
|
|
| 38 |
|
| 39 |
## Visual Results
|
| 40 |
|
| 41 |
+
### Supplied-video demo — Latest (PHC+ GRPO)
|
| 42 |
|
| 43 |
+
<video src="https://huggingface.co/ZeyuLing/Motius-FlowHMR-Latest/resolve/main/media/demo_latest_mesh.mp4" poster="https://huggingface.co/ZeyuLing/Motius-FlowHMR-Latest/resolve/main/media/demo_latest_mesh_preview.png" controls playsinline preload="metadata" width="100%"></video>
|
| 44 |
|
| 45 |
+
[Input video](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Latest/resolve/main/media/demo_input.mp4) · [Download demo](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Latest/resolve/main/media/demo_latest_mesh.mp4) ·
|
| 46 |
+
[Render provenance](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Latest/resolve/main/media/demo_latest_mesh.json)
|
|
|
|
| 47 |
|
| 48 |
+
**Checkpoint:** [Motius-FlowHMR-Latest](https://huggingface.co/ZeyuLing/Motius-FlowHMR-Latest),
|
| 49 |
+
the official Latest release after PHC+ GRPO post-training.
|
| 50 |
|
| 51 |
+
- Official checkpoint SHA256: `b7eb06b119feaf5ad7287bafdb2fff38d42dfc22ea9d256f8cdd2ada26dfc53a`.
|
| 52 |
+
- Motius inference safetensors SHA256: `f4ae4168a6f576ffe08a3a19b245e0fd65c0d304b8e93bc2c6ab0a46a6f86912`.
|
| 53 |
+
- 750 frames, 25 seconds, 30 fps; seed 0, 20 Euler steps, CFG 1.
|
| 54 |
+
- VGGT camera sampling every 100 frames (9 keyframes interpolated to 750).
|
| 55 |
|
| 56 |
+
The left panel displays the supplied video with the Motius frontend's tracking
|
| 57 |
+
bbox. The right panel reconstructs SMPL-H vertices from the saved predicted
|
| 58 |
+
rotations, shape and translation, using Motius's bundled implementation.
|
| 59 |
+
Three.js provides smooth normals, depth-tested surfaces, lighting and floor
|
| 60 |
+
shadows. The fixed camera fits the complete sequence; horizontal root motion
|
| 61 |
+
is centered for display, while vertical motion is preserved. This is a world
|
| 62 |
+
view, not a mesh reprojection into the source camera. Original audio is retained.
|
| 63 |
+
No external FlowHMR checkout is needed for inference or mesh reconstruction.
|
| 64 |
|
| 65 |
+
Earlier parity-validation clips: [Base, 30 frames](https://github.com/user-attachments/assets/791ec4c4-638d-478c-b79d-7ddb09d3582b) ·
|
| 66 |
+
[Latest, 30 frames](https://github.com/user-attachments/assets/ca588fd7-52c8-49a8-af11-e0def808a715). These use a different input clip.
|
|
|
|
| 67 |
|
|
|
|
|
|
|
| 68 |
|
| 69 |
## Model Overview
|
| 70 |
|
media/demo_input.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8f9e9c1193163940eafb512e9a0621a31fe316406e7a69aa3a357f6deceb17f1
|
| 3 |
+
size 6645208
|
media/demo_latest_mesh.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"renderer": "motius-threejs-smooth-mesh",
|
| 3 |
+
"checkpoint_label": "FlowHMR Latest",
|
| 4 |
+
"checkpoint_sha256": "f4ae4168a6f576ffe08a3a19b245e0fd65c0d304b8e93bc2c6ab0a46a6f86912",
|
| 5 |
+
"source_revision": "f12e6a2d46a63d771a66dbb6b5598a1be65b9eea",
|
| 6 |
+
"frames": 750,
|
| 7 |
+
"fps": 30.0,
|
| 8 |
+
"width": 1600,
|
| 9 |
+
"height": 800,
|
| 10 |
+
"vertices": 6890,
|
| 11 |
+
"faces": 13776,
|
| 12 |
+
"view": "root-centered world; fixed camera; original Y retained",
|
| 13 |
+
"camera": {
|
| 14 |
+
"width": 500,
|
| 15 |
+
"height": 585,
|
| 16 |
+
"frames": 750,
|
| 17 |
+
"vertex_count": 6890,
|
| 18 |
+
"half_height": 1.1323156712833715,
|
| 19 |
+
"camera_target": [
|
| 20 |
+
-0.005707634257960004,
|
| 21 |
+
0.8669544873149713,
|
| 22 |
+
0.1286162645327557
|
| 23 |
+
],
|
| 24 |
+
"camera_position": [
|
| 25 |
+
1.4343384479539978,
|
| 26 |
+
1.730982136642146,
|
| 27 |
+
-5.631568064315076
|
| 28 |
+
]
|
| 29 |
+
},
|
| 30 |
+
"source_sha256": "8f9e9c1193163940eafb512e9a0621a31fe316406e7a69aa3a357f6deceb17f1",
|
| 31 |
+
"video_sha256": "ddfd29ce41206457c9e5150b72bf9d1e7eae52f0725eb049442b7bf21404b60d"
|
| 32 |
+
}
|
media/demo_latest_mesh.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ddfd29ce41206457c9e5150b72bf9d1e7eae52f0725eb049442b7bf21404b60d
|
| 3 |
+
size 7564846
|
media/demo_latest_mesh_preview.png
ADDED
|
Git LFS Details
|