Phase 1 checkpoint, epoch None

phase1_v2/README.md

@@ -0,0 +1,107 @@
+---
+library_name: mmm
+tags:
+- motion-matching
+- character-animation
+- state-space-model
+- mamba
+---
+
+# MMM-versions
+
+Retrieval augmented state space model for real time character animation: a
+Mamba backbone that takes a joystick request, its own previous pose, and `k`
+poses retrieved from a motion matching database, and predicts the next pose.
+
+This is a **Phase 1** checkpoint (pre-training on mocap).
+
+## This model is not usable on its own
+
+The retrieved candidate poses are part of the input, not an optional extra.
+Without the matching database the model is being fed zeros in 371%
+of its input width and will not produce correct motion.
+
+The database this checkpoint was trained against:
+
+| | |
+| --- | --- |
+| fingerprint | `eb460d0093b66709` |
+| rows (frames) | 962,891 |
+| pose width | 151 |
+| frame time | 0.016666666666666666 s |
+| clips | 77 (aiming1_subject1, aiming1_subject4, aiming2_subject2, aiming2_subject3...) |
+
+The database is built from LaFAN1 and is **not** distributed here. Rebuild it
+with `scripts/build_databases.py`; the fingerprint above must match, or the
+retrieved poses are in a different space from the ones the model learned on.
+
+## Architecture
+
+| | |
+| --- | --- |
+| backbone | MambaBackbone |
+| n_layers | 4 |
+| d_model | 256 |
+| d_state | 16 |
+| d_conv | 4 |
+| d_inner | 512 |
+| head | full |
+| pose_dim | 151 |
+| joystick_dim | 12 |
+| n_candidates | 4 |
+| input_dim | 163 |
+| output_dim | 153 |
+| predict_contacts | True |
+| parameters | 1,833,625 |
+| use_retrieval | False |
+
+## Training
+
+| epoch | train | validation |
+| --- | --- | --- |
+| 1 (first) | 1.1842 | 0.5915 |
+| 55 (best) | 0.2827 | 0.2243 |
+| 56 (last) | 0.2816 | 0.2262 |
+
+Trained for 56 epochs.
+
+## Retrieval gain
+
+The number this project is actually judged on. It is the validation loss with
+the retrieved candidates zeroed, minus the loss with them present, so positive
+means retrieval is helping.
+
+It is reported in two regimes and **only the free running one is meaningful**.
+Under teacher forcing the model is handed the true previous pose every frame,
+and at 60Hz that is a far better predictor than any retrieved candidate --
+measured on this database, copying the previous pose scores 2.29 while copying
+the nearest candidate scores 16.24. A teacher forced model is therefore right
+to ignore retrieval, and its gain sits at zero whatever the model has learned.
+
+| regime | with retrieval | without | gain |
+| --- | --- | --- | --- |
+| teacher forced | 0.2243 | 0.2243 | +0.0000 |
+| **free running** | 0.7945 | 0.7945 | **+0.0000** |
+
+## Provenance
+
+- Rotation maths cross validated against GenoViewPython's `quat.py`
+  (MIT, Copyright (c) 2024 Daniel Holden) to 1e-9.
+- Spring damper after Holden, *Spring-It-On*; inertialization after Bollo,
+  GDC 2018; 6D rotations after Zhou et al., CVPR 2019.
+- Learned Motion Matching (Holden et al., SIGGRAPH 2020) is background and
+  baseline for this work, not its architecture.
+
+See `NOTICE.md` in the source repository for the full attribution.
+
+## Format
+
+`safetensors`, with the run's architecture, config and calibrated loss weights
+JSON encoded in the metadata header. No pickle is involved in loading.
+
+```python
+from safetensors import safe_open
+
+with safe_open("published.safetensors", framework="pt") as f:
+    print(f.metadata()["architecture"])
+```