| --- |
| library_name: mmm |
| tags: |
| - motion-matching |
| - character-animation |
| - state-space-model |
| - mamba |
| --- |
| |
| # MMM |
|
|
| Retrieval augmented state space model for real time character animation: a |
| Mamba backbone that takes a joystick request, its own previous pose, and `k` |
| poses retrieved from a motion matching database, and predicts the next pose. |
|
|
| This is a **Phase 1** checkpoint (pre-training on mocap). |
|
|
| ## This model is not usable on its own |
|
|
| The retrieved candidate poses are part of the input, not an optional extra. |
| Without the matching database the model is being fed zeros in 371% |
| of its input width and will not produce correct motion. |
|
|
| The database this checkpoint was trained against: |
|
|
| | | | |
| | --- | --- | |
| | fingerprint | `eb460d0093b66709` | |
| | rows (frames) | 962,891 | |
| | pose width | 151 | |
| | frame time | 0.016666666666666666 s | |
| | clips | 77 (aiming1_subject1, aiming1_subject4, aiming2_subject2, aiming2_subject3...) | |
|
|
| The database is built from LaFAN1 and is **not** distributed here. Rebuild it |
| with `scripts/build_databases.py`; the fingerprint above must match, or the |
| retrieved poses are in a different space from the ones the model learned on. |
|
|
| ## Architecture |
|
|
| | | | |
| | --- | --- | |
| | backbone | MambaBackbone | |
| | n_layers | 4 | |
| | d_model | 256 | |
| | d_state | 16 | |
| | d_conv | 4 | |
| | d_inner | 512 | |
| | head | full | |
| | pose_dim | 151 | |
| | joystick_dim | 12 | |
| | n_candidates | 4 | |
| | input_dim | 163 | |
| | output_dim | 153 | |
| | predict_contacts | True | |
| | parameters | 1,833,625 | |
| | use_retrieval | False | |
|
|
| ## Training |
|
|
| | epoch | train | validation | |
| | --- | --- | --- | |
| | 1 (first) | 1.1842 | 0.5915 | |
| | 55 (best) | 0.2827 | 0.2243 | |
| | 56 (last) | 0.2816 | 0.2262 | |
|
|
| Trained for 56 epochs. |
|
|
| ## Retrieval gain |
|
|
| The number this project is actually judged on. It is the validation loss with |
| the retrieved candidates zeroed, minus the loss with them present, so positive |
| means retrieval is helping. |
|
|
| It is reported in two regimes and **only the free running one is meaningful**. |
| Under teacher forcing the model is handed the true previous pose every frame, |
| and at 60Hz that is a far better predictor than any retrieved candidate -- |
| measured on this database, copying the previous pose scores 2.29 while copying |
| the nearest candidate scores 16.24. A teacher forced model is therefore right |
| to ignore retrieval, and its gain sits at zero whatever the model has learned. |
|
|
| | regime | with retrieval | without | gain | |
| | --- | --- | --- | --- | |
| | teacher forced | 0.2243 | 0.2243 | +0.0000 | |
| | **free running** | 0.7945 | 0.7945 | **+0.0000** | |
|
|
| ## Provenance |
|
|
| - Rotation maths cross validated against GenoViewPython's `quat.py` |
| (MIT, Copyright (c) 2024 Daniel Holden) to 1e-9. |
| - Spring damper after Holden, *Spring-It-On*; inertialization after Bollo, |
| GDC 2018; 6D rotations after Zhou et al., CVPR 2019. |
| - Learned Motion Matching (Holden et al., SIGGRAPH 2020) is background and |
| baseline for this work, not its architecture. |
|
|
| See `NOTICE.md` in the source repository for the full attribution. |
|
|
| ## Format |
|
|
| `safetensors`, with the run's architecture, config and calibrated loss weights |
| JSON encoded in the metadata header. No pickle is involved in loading. |
|
|
| ```python |
| from safetensors import safe_open |
| |
| with safe_open("published.safetensors", framework="pt") as f: |
| print(f.metadata()["architecture"]) |
| ``` |
|
|
