Release joint_mix40 VAE weights, training and GT neighborhood sampling code
Browse files- DATA_FORMAT.md +65 -0
- README.md +158 -0
- SHA256SUMS +22 -0
- augment.py +150 -0
- bimanual_vae.py +32 -0
- config.json +47 -0
- example.py +45 -0
- hand_vae.py +38 -0
- joint_mix40.pt +3 -0
- load_model.py +16 -0
- mano_compat.py +26 -0
- model.safetensors +3 -0
- provenance.json +53 -0
- requirements-training.txt +4 -0
- requirements.txt +4 -0
- train.py +68 -0
- training/README.md +24 -0
- training/history.json +340 -0
- training/original_common.py +14 -0
- training/original_prepare_data.py +54 -0
- training/original_train_pair.py +44 -0
- training_config.json +19 -0
- validation.json +29 -0
DATA_FORMAT.md
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Numeric NPZ schemas
|
| 2 |
+
|
| 3 |
+
All NPZ files are loaded with `allow_pickle=False`. Rotations and positions use
|
| 4 |
+
float32; split, recording, frame, and dataset indices are integers.
|
| 5 |
+
|
| 6 |
+
## Training cache and query poses
|
| 7 |
+
|
| 8 |
+
| Array | Shape | Meaning |
|
| 9 |
+
|---|---|---|
|
| 10 |
+
| `pose_aa` | N × 2 × 15 × 3 | Full native MANO local finger axis angles, radians; left/right |
|
| 11 |
+
| `relative_rotation` | N × 3 × 3 | `R_left.T @ R_right` |
|
| 12 |
+
| `relative_wrist_m` | N × 3 | `R_left.T @ (wrist_right - wrist_left)`, meters |
|
| 13 |
+
| `dataset` | N | 0=ARCTIC, 1=H2O, 2=GRAB in the original cache |
|
| 14 |
+
| `sequence` | N | Recording identifier, consistent within each source; required for training/pool encoding |
|
| 15 |
+
| `split` | N | 0=train, 1=validation, 2=test |
|
| 16 |
+
| `frame_ids` | N | Optional original frame identifier |
|
| 17 |
+
|
| 18 |
+
Only real, same-frame valid hand pairs belong in these inputs. Do not construct a
|
| 19 |
+
pair by combining unrelated single-hand examples. Do not add a hand mean twice.
|
| 20 |
+
The original cache also has `anchor_rotation` and `anchor_wrist_m`; the training
|
| 21 |
+
and portable augmentation scripts do not need them because they model relative
|
| 22 |
+
coordinates. Query files use the same pose fields; `sequence` is not required.
|
| 23 |
+
|
| 24 |
+
The encoder selects only training rows. The sampler uses only train/validation
|
| 25 |
+
queries and finds centers and neighbors in the training pool, restricted to the
|
| 26 |
+
query's source dataset. For an EgoRenderer query based on GRAB motion, use the
|
| 27 |
+
source-motion dataset ID `2`.
|
| 28 |
+
|
| 29 |
+
## Encoded training pool
|
| 30 |
+
|
| 31 |
+
`mu` and `logvar` are N × 40. `source_row` points back into the original training
|
| 32 |
+
cache. `sequence`, `dataset`, `split`, optional `frame_ids`, and the cache
|
| 33 |
+
`source_sha256` preserve provenance. Every pool row must have `split=0`.
|
| 34 |
+
|
| 35 |
+
## Synthetic rendering parameters
|
| 36 |
+
|
| 37 |
+
Let Q be query count, T the number of scales, and D the directions per scale.
|
| 38 |
+
The output has S=Q×T×D samples ordered query, scale, direction.
|
| 39 |
+
|
| 40 |
+
| Array | Shape | Meaning |
|
| 41 |
+
|---|---|---|
|
| 42 |
+
| `z` | S × 40 | Sampled latent |
|
| 43 |
+
| `mano_rotation_matrices` | S × 2 × 16 × 3 × 3 | Wrist root followed by 15 native local finger rotations |
|
| 44 |
+
| `mano_pose_axis_angle` | S × 2 × 16 × 3 | The same full rotations as axis angles in radians |
|
| 45 |
+
| `wrist_positions_m` | S × 2 × 3 | Actual joint-zero positions in meters |
|
| 46 |
+
| `mano_shape` | S × 2 × 10 | Neutral shape, all zeros |
|
| 47 |
+
| `query_index` | S | Query row index |
|
| 48 |
+
| `train_pool_row`, `train_source_row` | S | Center in encoded pool / original cache |
|
| 49 |
+
| `train_sequence`, `train_dataset` | S | Center recording and dataset IDs |
|
| 50 |
+
| `temperature` | S | Interpolation fraction tau, despite the historical field name |
|
| 51 |
+
| `draw` | S | Direction index |
|
| 52 |
+
| `anchor_pool_rows` | Q | Retrieved training centers |
|
| 53 |
+
| `neighbor_pool_rows` | Q × K | Neighbor rows from distinct other training recordings |
|
| 54 |
+
| `endpoint_neighbor_indices` | Q × D × 2 | Pair indices into that query's neighbor list |
|
| 55 |
+
| `endpoint_weights` | Q × D | Convex endpoint weight w |
|
| 56 |
+
|
| 57 |
+
The canonical left wrist is at zero with identity rotation. Right-wrist placement
|
| 58 |
+
comes from the decoded relative pose. Apply the same chosen scene transform to
|
| 59 |
+
both wrists and both root orientations. Local finger rotations remain local.
|
| 60 |
+
|
| 61 |
+
To decode meshes, use MANO with full poses (no additional pose-mean addition),
|
| 62 |
+
then translate each hand so its actual wrist joint matches `wrist_positions_m`.
|
| 63 |
+
Do not pass actual wrist positions directly as MANO `transl` unless your MANO
|
| 64 |
+
implementation explicitly uses a wrist-centered template. Mesh generation is a
|
| 65 |
+
separate operation and is not required to encode/sample the VAE.
|
README.md
ADDED
|
@@ -0,0 +1,158 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: pytorch
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
- zh
|
| 6 |
+
tags:
|
| 7 |
+
- hand-pose
|
| 8 |
+
- bimanual
|
| 9 |
+
- vae
|
| 10 |
+
- mano
|
| 11 |
+
- pose-generation
|
| 12 |
+
- data-augmentation
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# Bimanual Pose VAE — joint_mix40
|
| 16 |
+
|
| 17 |
+
双手姿态 VAE 的代码与权重:学习双手局部姿态、相对腕朝向和相对腕位置,
|
| 18 |
+
可用于姿态先验,以及从训练 GT 的 latent 邻域生成渲染增广候选。
|
| 19 |
+
|
| 20 |
+
This is the **exact joint_mix40 checkpoint** used by the
|
| 21 |
+
[VAE Pose Atlas visualization](https://huggingface.co/spaces/WhynotGPT/vae-pose-atlas).
|
| 22 |
+
The framewise model encodes 30 local finger rotations, one relative wrist rotation,
|
| 23 |
+
and a relative wrist position into a **40-dimensional latent**. It uses a width-384
|
| 24 |
+
MLP and a 189-dimensional input with row-based 6D rotation representations.
|
| 25 |
+
Shared global rotation and translation are removed by expressing the wrist relation
|
| 26 |
+
in the left wrist frame. RGB, camera parameters, hand shape, and temporal dynamics
|
| 27 |
+
are outside the model.
|
| 28 |
+
|
| 29 |
+
## Download and run
|
| 30 |
+
|
| 31 |
+
Python 3.11 is the tested environment. Download the repository, then run from its root:
|
| 32 |
+
|
| 33 |
+
```bash
|
| 34 |
+
pip install huggingface_hub
|
| 35 |
+
python -c "from huggingface_hub import snapshot_download; snapshot_download('WhynotGPT/bimanual-pose-vae', local_dir='bimanual-pose-vae')"
|
| 36 |
+
cd bimanual-pose-vae
|
| 37 |
+
pip install -r requirements.txt
|
| 38 |
+
python example.py --device cpu
|
| 39 |
+
sha256sum -c SHA256SUMS
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
The example uses synthetic inputs to verify loading, reconstruction, and input
|
| 43 |
+
gradients through the frozen VAE. It does not measure accuracy.
|
| 44 |
+
|
| 45 |
+
```python
|
| 46 |
+
import torch
|
| 47 |
+
from load_model import load_model
|
| 48 |
+
from bimanual_vae import pack
|
| 49 |
+
|
| 50 |
+
vae, metadata = load_model(device="cpu") # model.safetensors + config.json
|
| 51 |
+
# local_rot: [B,2,15,3,3]; root_rot: [B,2,3,3]; wrist: [B,2,3]
|
| 52 |
+
with torch.no_grad():
|
| 53 |
+
rot, pos = pack(local_rot, root_rot, wrist)
|
| 54 |
+
mu, logvar = vae.encode(rot, pos)
|
| 55 |
+
reconstructed_rot, reconstructed_pos = vae.decode(mu)
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
Inputs use **left, right** ordering and each hand's native MANO convention. Local
|
| 59 |
+
rotations represent the full pose, including the native hand mean where required
|
| 60 |
+
by the source convention. Wrist positions are actual joint positions in meters,
|
| 61 |
+
not necessarily the MANO `transl` parameter. Both hands must be valid.
|
| 62 |
+
|
| 63 |
+
Decoder rotations have shape `[B,31,3,3]`: left fingers `0:15`, right fingers
|
| 64 |
+
`15:30`, and relative right-wrist orientation `30`. Position `[B,3]` is the right
|
| 65 |
+
wrist in the left wrist frame. To place a pose in a scene, choose a left-wrist
|
| 66 |
+
anchor `(R_L,t_L)`, then set `R_R=R_L @ rot[:,30]` and `t_R=t_L+R_L @ pos`.
|
| 67 |
+
|
| 68 |
+
The original API is also available:
|
| 69 |
+
|
| 70 |
+
```python
|
| 71 |
+
from bimanual_vae import load_pair
|
| 72 |
+
vae, metadata = load_pair("joint_mix40.pt", device="cpu")
|
| 73 |
+
# Preserve gradients through the inputs when using a frozen prior:
|
| 74 |
+
loss_prior = vae.energy(local_rot, root_rot, wrist).mean()
|
| 75 |
+
```
|
| 76 |
+
|
| 77 |
+
## Generate augmentation poses from training GT
|
| 78 |
+
|
| 79 |
+
The portable sampler follows the train-GT neighborhood method used in the Atlas.
|
| 80 |
+
It encodes only `split=0` rows, retrieves the nearest training pose in the same
|
| 81 |
+
dataset as each query, and finds neighbors from 16 distinct other training recordings.
|
| 82 |
+
Queries can be difficult training/validation GT poses; test queries are rejected.
|
| 83 |
+
The caller selects difficult queries, for example using a model's validation PA error.
|
| 84 |
+
|
| 85 |
+
```bash
|
| 86 |
+
python augment.py encode --data /path/to/paired_motion.npz --output train_latents.npz
|
| 87 |
+
python augment.py sample --pool train_latents.npz --queries /path/to/difficult_gt.npz --output render_poses.npz
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
For each retrieved center, the sampler draws eight neighbor pairs with weights
|
| 91 |
+
`w` in `[0.2,0.8]` and reuses those endpoints at `tau = 0.25, 0.5, 1.0`:
|
| 92 |
+
|
| 93 |
+
```text
|
| 94 |
+
endpoint = w * mu_neighbor_a + (1-w) * mu_neighbor_b
|
| 95 |
+
z = (1-tau) * mu_train_center + tau * endpoint
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
Each query yields **24 synthetic bimanual poses**. Output includes full MANO axis
|
| 99 |
+
angles and rotation matrices, actual wrist positions, zero shape coefficients,
|
| 100 |
+
latent values, and training-source indices. The left wrist is at the origin with
|
| 101 |
+
identity orientation. See [DATA_FORMAT.md](DATA_FORMAT.md) for all array conventions.
|
| 102 |
+
|
| 103 |
+
These outputs are pose labels for subsequent rendering. Geometry, scene placement,
|
| 104 |
+
hand-object contact, and collision checks belong to the rendering pipeline. The
|
| 105 |
+
model alone does not generate mesh vertices or guarantee physical plausibility.
|
| 106 |
+
Augmentation gains on a downstream hand estimator have not been established by
|
| 107 |
+
this release. The portable script implements the same sampling rule; it does not
|
| 108 |
+
recreate the Atlas's source-specific query extraction or its exact random draw order.
|
| 109 |
+
|
| 110 |
+
## Training
|
| 111 |
+
|
| 112 |
+
The released checkpoint was selected at **epoch 50** of a 60-epoch run using
|
| 113 |
+
validation reconstruction metrics. Training uses 48 updates per epoch, batch 192,
|
| 114 |
+
AdamW at 0.001, dataset/recording-balanced random sampling, rotation and skeletal
|
| 115 |
+
FK reconstruction, relative-position reconstruction, and KL regularization.
|
| 116 |
+
|
| 117 |
+
| Source | Train paired frames | Validation paired frames | Test paired frames |
|
| 118 |
+
|---|---:|---:|---:|
|
| 119 |
+
| ARCTIC | 32,320 | 4,960 | 4,480 |
|
| 120 |
+
| H2O | 12,686 | 2,560 | 1,760 |
|
| 121 |
+
| GRAB | 104,825 | 10,460 | 11,834 |
|
| 122 |
+
|
| 123 |
+
ARCTIC/H2O use frozen recording splits with the experiment's benchmark recordings
|
| 124 |
+
excluded. GRAB uses subjects s1–s8 / s9 / s10. Preparation retains at most 160
|
| 125 |
+
paired frames per recording; GRAB is first downsampled from 120 fps to 10 fps.
|
| 126 |
+
These are VAE training counts, separate from the main estimator's SFT clip counts.
|
| 127 |
+
|
| 128 |
+
```bash
|
| 129 |
+
pip install -r requirements-training.txt
|
| 130 |
+
python train.py --data /path/to/paired_motion.npz \
|
| 131 |
+
--mano-dir /path/to/mano --output runs/joint_mix40 --device cuda
|
| 132 |
+
```
|
| 133 |
+
|
| 134 |
+
`train.py` preserves the original training algorithm, adding configurable paths,
|
| 135 |
+
device, and run lengths. It needs your prepared motion cache and MANO_LEFT.pkl /
|
| 136 |
+
MANO_RIGHT.pkl for the native skeleton FK loss. Raw data, the motion cache, MANO
|
| 137 |
+
model assets, and their derived mesh buffers are not distributed here. Obtain
|
| 138 |
+
those dependencies under their respective terms. The original experiment scripts
|
| 139 |
+
are archived in [training/](training/README.md); their historical absolute paths
|
| 140 |
+
and external preprocessing dependencies are documented there.
|
| 141 |
+
|
| 142 |
+
The repository includes only the **joint_mix40 trained weights**. Training code
|
| 143 |
+
also exposes the historical joint_ego40 and pose_mix40 variants.
|
| 144 |
+
|
| 145 |
+
## Files and verification
|
| 146 |
+
|
| 147 |
+
- `joint_mix40.pt`: byte-identical original checkpoint, with normalization and metadata.
|
| 148 |
+
- `model.safetensors`, `config.json`: equivalent tensor-only weights and JSON metadata.
|
| 149 |
+
- `bimanual_vae.py`, `hand_vae.py`: unchanged original model definitions and rotation helpers.
|
| 150 |
+
- `load_model.py`, `example.py`: loading and differentiable-prior examples.
|
| 151 |
+
- `train.py`, `training_config.json`, `training/`: training entry point and provenance.
|
| 152 |
+
- `augment.py`, `DATA_FORMAT.md`: train-GT encoding and neighborhood sampling.
|
| 153 |
+
- `validation.json`, `provenance.json`, `SHA256SUMS`: checks and file integrity records.
|
| 154 |
+
|
| 155 |
+
Original checkpoint SHA-256:
|
| 156 |
+
`e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096`.
|
| 157 |
+
The original model and training work is from the MiroHand P3R4 bimanual VAE
|
| 158 |
+
experiment dated 2026-09-27; this package adds portable interfaces for release.
|
SHA256SUMS
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
c3f7f50702d94ae4e2e631818a595723fc417e17f1388f63e228ce725853afb2 DATA_FORMAT.md
|
| 2 |
+
9bb029884f38a9b5d3e80197a03f158eae05e4adfc0273c23dede3ba90901720 README.md
|
| 3 |
+
ea941357aa03afdce5b6608dac2965d44bf6347d9119efb5f832999ebbe89744 augment.py
|
| 4 |
+
9e940527007fc2bf93809eba891bb18b76b75a9e30206a2ab0698704e431f3f6 bimanual_vae.py
|
| 5 |
+
0e486bd7ef71d03f55b2f4de0d4bdb314e198d37380cdb366240789534ce2ed3 config.json
|
| 6 |
+
d81b092510f478f0e0c405553ac8238144239926087f3650ee4c07e48906999d example.py
|
| 7 |
+
5ee21d9a48ce2d8094f28c2ff1ae56a57d5c55244125e18b3c3e50d1a588979c hand_vae.py
|
| 8 |
+
e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096 joint_mix40.pt
|
| 9 |
+
11e94dbc58096fb2ce590af9749f4ab431cf1c27799f6fdeb2341f61453b0338 load_model.py
|
| 10 |
+
c4a0ccd48ebba2e8f03fc295be942e4520385f99521460de4c9915c1776e9da3 mano_compat.py
|
| 11 |
+
6c4fe1b79138e4cd3b7ddccb45780479de2d25270be018e46be6696d78bd05f8 model.safetensors
|
| 12 |
+
f362e83ba1b3798744b94f3d5b79edbb7b0dfdd8c4d0cef328f361abe8cf17a3 provenance.json
|
| 13 |
+
cd3b0fc323d507a9935a38187754f71a8b973f9d3ef32be9bfd29b3f0a521ebb requirements-training.txt
|
| 14 |
+
efba809b61f64bbf0863e65c9c303800b8c2c6a28100c1739e2407964b24e276 requirements.txt
|
| 15 |
+
5b7db63f51871d303c1abad15441c002693df8cdc68cab944905ab8dc146f3db train.py
|
| 16 |
+
89ee9e8f6152ad58d42003119d5c6643d9242e511a35a8a6711a860d57fd082f training/README.md
|
| 17 |
+
a83a1341b55522d329adbaab29937ba5a260f42091975098aacb116490ff9417 training/history.json
|
| 18 |
+
f08a807e45d72cc2fe6de81f65b22db78075d16ac4dcfa390b32572ccc3e185e training/original_common.py
|
| 19 |
+
1c2bb19eeaf58e2f80b76e6ec556af02cb92cbb2ebfeaf580b7ed00fc86bac4d training/original_prepare_data.py
|
| 20 |
+
8c854506d4a084fcea9ce8988c7dc33fe628f22b8380832f7f45a54ffc8a716a training/original_train_pair.py
|
| 21 |
+
e4163ef9cc4f8b7add9d4e4350a853a145c842e51bab851101dcabb3c550c4ff training_config.json
|
| 22 |
+
2e072c6387f5c7765b8a45b66a3217f4b6c11cbc31c0426ed47fece4c3ecf025 validation.json
|
augment.py
ADDED
|
@@ -0,0 +1,150 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Encode training GT and sample distinct-recording latent neighborhoods.
|
| 2 |
+
|
| 3 |
+
Inputs are numeric NPZ arrays, described in DATA_FORMAT.md. Mesh generation is
|
| 4 |
+
external; the output contains full MANO rotations and actual wrist positions.
|
| 5 |
+
"""
|
| 6 |
+
import argparse
|
| 7 |
+
import hashlib
|
| 8 |
+
from pathlib import Path
|
| 9 |
+
|
| 10 |
+
import numpy as np
|
| 11 |
+
import torch
|
| 12 |
+
|
| 13 |
+
from hand_vae import aa_to_matrix, matrix_to_aa
|
| 14 |
+
from load_model import load_model
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
def encode_pose(model, pose, relative_rotation, relative_wrist):
|
| 18 |
+
local = aa_to_matrix(torch.as_tensor(pose, dtype=torch.float32)).reshape(-1, 30, 3, 3)
|
| 19 |
+
rotations = torch.cat([local, torch.as_tensor(relative_rotation, dtype=torch.float32)[:, None]], 1)
|
| 20 |
+
return model.encode(rotations, torch.as_tensor(relative_wrist, dtype=torch.float32))
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
@torch.inference_mode()
|
| 24 |
+
def encode(data_path, output):
|
| 25 |
+
model, _ = load_model()
|
| 26 |
+
with np.load(data_path, allow_pickle=False) as data:
|
| 27 |
+
rows = np.flatnonzero(data["split"] == 0)
|
| 28 |
+
if not len(rows):
|
| 29 |
+
raise ValueError("No split=0 training rows.")
|
| 30 |
+
payload = {name: data[name][rows] for name in ("sequence", "dataset")}
|
| 31 |
+
if "frame_ids" in data:
|
| 32 |
+
payload["frame_ids"] = data["frame_ids"][rows]
|
| 33 |
+
poses = {name: data[name][rows] for name in ("pose_aa", "relative_rotation", "relative_wrist_m")}
|
| 34 |
+
means, logvars = [], []
|
| 35 |
+
for start in range(0, len(rows), 2048):
|
| 36 |
+
ix = slice(start, start + 2048)
|
| 37 |
+
mu, logvar = encode_pose(model, poses["pose_aa"][ix], poses["relative_rotation"][ix], poses["relative_wrist_m"][ix])
|
| 38 |
+
means.append(mu.numpy())
|
| 39 |
+
logvars.append(logvar.numpy())
|
| 40 |
+
with data_path.open("rb") as stream:
|
| 41 |
+
source_hash = hashlib.file_digest(stream, "sha256").hexdigest()
|
| 42 |
+
np.savez_compressed(output, mu=np.concatenate(means), logvar=np.concatenate(logvars),
|
| 43 |
+
source_row=rows, split=np.zeros(len(rows), dtype=np.int8),
|
| 44 |
+
source_sha256=np.array(source_hash), **payload)
|
| 45 |
+
print(f"Encoded {len(rows)} training GT rows into {output}")
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
@torch.inference_mode()
|
| 49 |
+
def sample(pool_path, query_path, output, neighbor_count, directions, scales, seed):
|
| 50 |
+
model, _ = load_model()
|
| 51 |
+
rng = np.random.default_rng(seed)
|
| 52 |
+
with np.load(pool_path, allow_pickle=False) as data:
|
| 53 |
+
pool = {key: data[key] for key in data.files}
|
| 54 |
+
if not np.all(pool["split"] == 0):
|
| 55 |
+
raise ValueError("The candidate pool must contain only training GT.")
|
| 56 |
+
with np.load(query_path, allow_pickle=False) as queries:
|
| 57 |
+
# Queries identify difficult GT regions; held-out test queries are excluded.
|
| 58 |
+
if not np.isin(queries["split"], [0, 1]).all():
|
| 59 |
+
raise ValueError("Queries must use train/validation GT, never test rows.")
|
| 60 |
+
query_datasets = queries["dataset"].copy()
|
| 61 |
+
query_mu, _ = encode_pose(model, queries["pose_aa"], queries["relative_rotation"], queries["relative_wrist_m"])
|
| 62 |
+
query_mu = query_mu.numpy()
|
| 63 |
+
if not len(query_mu):
|
| 64 |
+
raise ValueError("The query file is empty.")
|
| 65 |
+
if not np.isfinite(pool["mu"]).all() or not np.isfinite(query_mu).all():
|
| 66 |
+
raise ValueError("Latents must be finite.")
|
| 67 |
+
if neighbor_count < 1 or directions < 1 or not len(scales) or np.any((scales < 0) | (scales > 1)) or not np.isfinite(scales).all():
|
| 68 |
+
raise ValueError("Use positive counts and finite interpolation scales in [0,1].")
|
| 69 |
+
anchors, neighbors, pairs, weights, latent_batches = [], [], [], [], []
|
| 70 |
+
for query, dataset in zip(query_mu, query_datasets):
|
| 71 |
+
ids = np.flatnonzero(pool["dataset"] == dataset)
|
| 72 |
+
if not len(ids):
|
| 73 |
+
raise ValueError(f"No training rows for dataset {dataset}.")
|
| 74 |
+
anchor = int(ids[np.argmin(np.square(pool["mu"][ids] - query).sum(1))])
|
| 75 |
+
order = np.argsort(np.square(pool["mu"][ids] - pool["mu"][anchor]).sum(1))
|
| 76 |
+
seen = {int(pool["sequence"][anchor])}
|
| 77 |
+
nearest = []
|
| 78 |
+
for row in ids[order]:
|
| 79 |
+
sequence = int(pool["sequence"][row])
|
| 80 |
+
if sequence not in seen:
|
| 81 |
+
nearest.append(int(row))
|
| 82 |
+
seen.add(sequence)
|
| 83 |
+
if len(nearest) == neighbor_count:
|
| 84 |
+
break
|
| 85 |
+
if len(nearest) != neighbor_count:
|
| 86 |
+
raise ValueError(f"Dataset {dataset} needs {neighbor_count + 1} distinct training recordings.")
|
| 87 |
+
nearest = np.asarray(nearest)
|
| 88 |
+
pair = rng.choice(neighbor_count, (directions, 2), replace=True)
|
| 89 |
+
weight = rng.uniform(.2, .8, directions).astype(np.float32)
|
| 90 |
+
endpoint = weight[:, None] * pool["mu"][nearest[pair[:, 0]]] + (1 - weight[:, None]) * pool["mu"][nearest[pair[:, 1]]]
|
| 91 |
+
z = (1 - scales[:, None, None]) * pool["mu"][anchor] + scales[:, None, None] * endpoint[None]
|
| 92 |
+
latent_batches.append(z.reshape(-1, model.latent_dim))
|
| 93 |
+
anchors.append(anchor)
|
| 94 |
+
neighbors.append(nearest)
|
| 95 |
+
pairs.append(pair)
|
| 96 |
+
weights.append(weight)
|
| 97 |
+
z = np.concatenate(latent_batches).astype(np.float32)
|
| 98 |
+
rot, pos = model.decode(torch.from_numpy(z))
|
| 99 |
+
full = torch.zeros(len(z), 2, 16, 3, 3)
|
| 100 |
+
full[:, 0, 0] = torch.eye(3)
|
| 101 |
+
full[:, 1, 0] = rot[:, 30]
|
| 102 |
+
full[:, :, 1:] = rot[:, :30].reshape(-1, 2, 15, 3, 3)
|
| 103 |
+
wrist = torch.zeros(len(z), 2, 3)
|
| 104 |
+
wrist[:, 1] = pos
|
| 105 |
+
anchors = np.asarray(anchors)
|
| 106 |
+
query_index = np.repeat(np.arange(len(anchors)), len(scales) * directions)
|
| 107 |
+
np.savez_compressed(
|
| 108 |
+
output, z=z, mano_rotation_matrices=full.numpy(), mano_pose_axis_angle=matrix_to_aa(full).numpy(),
|
| 109 |
+
wrist_positions_m=wrist.numpy(), mano_shape=np.zeros((len(z), 2, 10), np.float32),
|
| 110 |
+
query_index=query_index, train_pool_row=anchors[query_index],
|
| 111 |
+
train_source_row=pool["source_row"][anchors[query_index]],
|
| 112 |
+
train_sequence=pool["sequence"][anchors[query_index]],
|
| 113 |
+
train_dataset=pool["dataset"][anchors[query_index]],
|
| 114 |
+
temperature=np.tile(np.repeat(scales, directions), len(anchors)),
|
| 115 |
+
draw=np.tile(np.arange(directions), len(anchors) * len(scales)),
|
| 116 |
+
anchor_pool_rows=anchors, neighbor_pool_rows=np.asarray(neighbors),
|
| 117 |
+
endpoint_neighbor_indices=np.asarray(pairs), endpoint_weights=np.asarray(weights),
|
| 118 |
+
seed=np.array(seed), coordinate_system=np.array("left_wrist_local"),
|
| 119 |
+
)
|
| 120 |
+
print(f"Saved {len(z)} synthetic bimanual pose candidates to {output}")
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def main():
|
| 124 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 125 |
+
commands = parser.add_subparsers(dest="command", required=True)
|
| 126 |
+
enc = commands.add_parser("encode")
|
| 127 |
+
enc.add_argument("--data", type=Path, required=True)
|
| 128 |
+
enc.add_argument("--output", type=Path, required=True)
|
| 129 |
+
sam = commands.add_parser("sample")
|
| 130 |
+
sam.add_argument("--pool", type=Path, required=True)
|
| 131 |
+
sam.add_argument("--queries", type=Path, required=True)
|
| 132 |
+
sam.add_argument("--output", type=Path, required=True)
|
| 133 |
+
sam.add_argument("--neighbors", type=int, default=16)
|
| 134 |
+
sam.add_argument("--directions", type=int, default=8)
|
| 135 |
+
sam.add_argument("--scales", type=float, nargs="+", default=[.25, .5, 1.])
|
| 136 |
+
sam.add_argument("--seed", type=int, default=20261110)
|
| 137 |
+
args = parser.parse_args()
|
| 138 |
+
if args.output.exists():
|
| 139 |
+
parser.error("Output already exists; choose a new path.")
|
| 140 |
+
args.output.parent.mkdir(parents=True, exist_ok=True)
|
| 141 |
+
torch.set_num_threads(4)
|
| 142 |
+
if args.command == "encode":
|
| 143 |
+
encode(args.data, args.output)
|
| 144 |
+
else:
|
| 145 |
+
sample(args.pool, args.queries, args.output, args.neighbors, args.directions,
|
| 146 |
+
np.asarray(args.scales, dtype=np.float32), args.seed)
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
if __name__ == "__main__":
|
| 150 |
+
main()
|
bimanual_vae.py
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""P3R4 paired-hand prior. Translation-invariant, common-rotation-invariant.
|
| 2 |
+
Local finger rotations (2 x 15), right wrist orientation and translation in
|
| 3 |
+
left wrist frame. Inference uses camera-coordinate inputs; mocap needs no camera.
|
| 4 |
+
"""
|
| 5 |
+
import torch
|
| 6 |
+
from torch import nn
|
| 7 |
+
from hand_vae import matrix_to_six,six_to_matrix,aa_to_matrix
|
| 8 |
+
class BimanualVAE(nn.Module):
|
| 9 |
+
def __init__(self,latent_dim=40,width=384,pose_only=False):
|
| 10 |
+
super().__init__();self.latent_dim=latent_dim;self.width=width;self.pose_only=pose_only;self.nrot=30 if pose_only else 31;self.dim=self.nrot*6+(0 if pose_only else 3)
|
| 11 |
+
self.register_buffer('input_mean',torch.zeros(self.dim));self.register_buffer('input_std',torch.ones(self.dim))
|
| 12 |
+
self.encoder=nn.Sequential(nn.Linear(self.dim,width),nn.LeakyReLU(.2),nn.Linear(width,width),nn.LeakyReLU(.2));self.mu=nn.Linear(width,latent_dim);self.logvar=nn.Linear(width,latent_dim);nn.init.constant_(self.logvar.bias,-2)
|
| 13 |
+
self.decoder=nn.Sequential(nn.Linear(latent_dim,width),nn.LeakyReLU(.2),nn.Linear(width,width),nn.LeakyReLU(.2),nn.Linear(width,self.dim))
|
| 14 |
+
def features(self,rot,pos):
|
| 15 |
+
x=matrix_to_six(rot[...,:self.nrot,:,:]).flatten(-2)
|
| 16 |
+
return x if self.pose_only else torch.cat([x,pos],-1)
|
| 17 |
+
def encode(self,rot,pos):
|
| 18 |
+
h=self.encoder((self.features(rot,pos)-self.input_mean)/self.input_std);return self.mu(h),self.logvar(h).clamp(-8,4)
|
| 19 |
+
def decode(self,z):
|
| 20 |
+
x=self.decoder(z)*self.input_std+self.input_mean;rot=six_to_matrix(x[...,:self.nrot*6].reshape(*x.shape[:-1],self.nrot,6));pos=torch.zeros((*x.shape[:-1],3),device=x.device,dtype=x.dtype) if self.pose_only else x[...,-3:];return rot,pos
|
| 21 |
+
def forward(self,rot,pos):
|
| 22 |
+
mu,lv=self.encode(rot,pos);z=mu+torch.randn_like(mu)*(.5*lv).exp() if self.training else mu;r,p=self.decode(z);return r,p,mu,lv
|
| 23 |
+
def energy(self,local_rot,root_rot,wrist):
|
| 24 |
+
rot,pos=pack(local_rot,root_rot,wrist);rr,pp,mu,lv=self(rot,pos)
|
| 25 |
+
pose=(rr[:,:30]-rot[:,:30]).square().sum((-1,-2)).mean(-1)
|
| 26 |
+
relation=pose*0 if self.pose_only else (rr[:,30]-rot[:,30]).square().sum((-1,-2))+((pp-pos)/.1).square().sum(-1)
|
| 27 |
+
return pose+relation+.0005*mu.square().mean(-1)
|
| 28 |
+
def pack(local_rot,root_rot,wrist):
|
| 29 |
+
left=root_rot[...,0,:,:];rr=left.transpose(-1,-2)@root_rot[...,1,:,:];pos=(left.transpose(-1,-2)@(wrist[...,1,:]-wrist[...,0,:])[...,None]).squeeze(-1)
|
| 30 |
+
return torch.cat([local_rot.flatten(-4,-3),rr.unsqueeze(-3)],-3),pos
|
| 31 |
+
def load_pair(path,device='cuda'):
|
| 32 |
+
c=torch.load(path,map_location=device,weights_only=False);m=BimanualVAE(**c['architecture']).to(device);m.load_state_dict(c['state_dict'],strict=True);m.eval();m.requires_grad_(False);return m,c
|
config.json
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": {
|
| 3 |
+
"latent_dim": 40,
|
| 4 |
+
"width": 384,
|
| 5 |
+
"pose_only": false
|
| 6 |
+
},
|
| 7 |
+
"config": {
|
| 8 |
+
"variant": "joint_mix40",
|
| 9 |
+
"seed": 20260927,
|
| 10 |
+
"epochs": 60,
|
| 11 |
+
"steps_per_epoch": 48,
|
| 12 |
+
"batch": 192,
|
| 13 |
+
"lr": 0.001,
|
| 14 |
+
"kl_beta": 0.0005,
|
| 15 |
+
"joint_weight": 0.25,
|
| 16 |
+
"position_scale_m": 0.1,
|
| 17 |
+
"balanced_dataset_and_sequence": true,
|
| 18 |
+
"included_datasets": [
|
| 19 |
+
0,
|
| 20 |
+
1,
|
| 21 |
+
2
|
| 22 |
+
],
|
| 23 |
+
"selection": "equal-dataset validation local FK mm + 0.25 relative position mm + relative rotation deg; test not used",
|
| 24 |
+
"data_sha256": "456c2b8011b56552a0806836fdb2eb9fcdd397ca3693f390bb2cbb5f8ba0f06f"
|
| 25 |
+
},
|
| 26 |
+
"epoch": 50,
|
| 27 |
+
"validation": [
|
| 28 |
+
{
|
| 29 |
+
"dataset": "ARCTIC",
|
| 30 |
+
"local_fk_mm": 2.552030563354492,
|
| 31 |
+
"relative_wrist_mm": 15.961709022521973,
|
| 32 |
+
"relative_rotation_deg": 13.042522430419922
|
| 33 |
+
},
|
| 34 |
+
{
|
| 35 |
+
"dataset": "H2O",
|
| 36 |
+
"local_fk_mm": 1.722329020500183,
|
| 37 |
+
"relative_wrist_mm": 7.883533000946045,
|
| 38 |
+
"relative_rotation_deg": 8.399897575378418
|
| 39 |
+
},
|
| 40 |
+
{
|
| 41 |
+
"dataset": "GRAB",
|
| 42 |
+
"local_fk_mm": 2.7820262908935547,
|
| 43 |
+
"relative_wrist_mm": 18.310850143432617,
|
| 44 |
+
"relative_rotation_deg": 17.670961380004883
|
| 45 |
+
}
|
| 46 |
+
]
|
| 47 |
+
}
|
example.py
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""CPU/GPU smoke example. Synthetic inputs only; not an accuracy evaluation."""
|
| 2 |
+
import argparse
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
import torch
|
| 6 |
+
from bimanual_vae import load_pair, pack
|
| 7 |
+
from hand_vae import aa_to_matrix
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
def main():
|
| 11 |
+
parser = argparse.ArgumentParser()
|
| 12 |
+
parser.add_argument("--device", default="cpu")
|
| 13 |
+
args = parser.parse_args()
|
| 14 |
+
torch.manual_seed(0)
|
| 15 |
+
vae, _ = load_pair(str(Path(__file__).resolve().parent / "joint_mix40.pt"), args.device)
|
| 16 |
+
# Synthetic full MANO local axis angles, left hand first, right hand second.
|
| 17 |
+
local_aa = torch.zeros(2, 2, 15, 3, device=args.device, requires_grad=True)
|
| 18 |
+
root_aa = torch.zeros(2, 2, 3, device=args.device, requires_grad=True)
|
| 19 |
+
wrist = torch.tensor([[[0., 0., .5], [.2, 0., .5]],
|
| 20 |
+
[[0., 0., .6], [.2, .02, .6]]],
|
| 21 |
+
device=args.device, requires_grad=True)
|
| 22 |
+
local_rot, root_rot = aa_to_matrix(local_aa), aa_to_matrix(root_aa)
|
| 23 |
+
energy = vae.energy(local_rot, root_rot, wrist)
|
| 24 |
+
loss = 0.1 * energy.mean()
|
| 25 |
+
loss.backward()
|
| 26 |
+
for x in (local_aa, root_aa, wrist):
|
| 27 |
+
assert x.grad is not None and torch.isfinite(x.grad).all()
|
| 28 |
+
assert torch.isfinite(energy).all()
|
| 29 |
+
assert all(not p.requires_grad and p.grad is None for p in vae.parameters())
|
| 30 |
+
with torch.no_grad():
|
| 31 |
+
rot, pos = pack(local_rot, root_rot, wrist)
|
| 32 |
+
rot_rec, pos_rec, mu, logvar = vae(rot, pos)
|
| 33 |
+
assert rot_rec.shape == (2, 31, 3, 3)
|
| 34 |
+
assert pos_rec.shape == (2, 3)
|
| 35 |
+
assert mu.shape == logvar.shape == (2, 40)
|
| 36 |
+
print(json.dumps({"device": args.device, "torch": torch.__version__,
|
| 37 |
+
"synthetic_inputs": True, "energy": energy.detach().cpu().tolist(),
|
| 38 |
+
"rotation_shape": list(rot_rec.shape),
|
| 39 |
+
"position_shape": list(pos_rec.shape),
|
| 40 |
+
"latent_shape": list(mu.shape),
|
| 41 |
+
"finite_input_gradients": True, "frozen_vae": True}, indent=2))
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
if __name__ == "__main__":
|
| 45 |
+
main()
|
hand_vae.py
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Hand-specific, side-conditioned VAE; 15 local rotations, no wrist/global/shape."""
|
| 2 |
+
import torch
|
| 3 |
+
from torch import nn
|
| 4 |
+
from torch.nn import functional as F
|
| 5 |
+
|
| 6 |
+
def aa_to_matrix(x):
|
| 7 |
+
theta=x.norm(dim=-1,keepdim=True);x0,x1,x2=x.unbind(-1);z=torch.zeros_like(x0)
|
| 8 |
+
k=torch.stack([z,-x2,x1,x2,z,-x0,-x1,x0,z],-1).reshape(*x.shape[:-1],3,3)
|
| 9 |
+
a=torch.sinc(theta/torch.pi)[...,None];b=(.5*torch.sinc(theta/(2*torch.pi)).square())[...,None]
|
| 10 |
+
return torch.eye(3,device=x.device,dtype=x.dtype)+a*k+b*(k@k)
|
| 11 |
+
|
| 12 |
+
def matrix_to_aa(m):
|
| 13 |
+
m00,m01,m02=m[...,0,0],m[...,0,1],m[...,0,2];m10,m11,m12=m[...,1,0],m[...,1,1],m[...,1,2];m20,m21,m22=m[...,2,0],m[...,2,1],m[...,2,2]
|
| 14 |
+
qabs=torch.stack([1+m00+m11+m22,1+m00-m11-m22,1-m00+m11-m22,1-m00-m11+m22],-1).clamp_min(1e-8).sqrt()
|
| 15 |
+
candidates=torch.stack([torch.stack([qabs[...,0]**2,m21-m12,m02-m20,m10-m01],-1),torch.stack([m21-m12,qabs[...,1]**2,m10+m01,m02+m20],-1),torch.stack([m02-m20,m10+m01,qabs[...,2]**2,m12+m21],-1),torch.stack([m10-m01,m20+m02,m21+m12,qabs[...,3]**2],-1)],-2)/(2*qabs[...,None].clamp_min(.1))
|
| 16 |
+
ix=qabs.argmax(-1);q=candidates.gather(-2,ix[...,None,None].expand(*ix.shape,1,4)).squeeze(-2);q=F.normalize(q,dim=-1);q=torch.where(q[...,:1]<0,-q,q)
|
| 17 |
+
norm=q[...,1:].norm(dim=-1,keepdim=True);angle=2*torch.atan2(norm,q[...,:1]);scale=torch.where(norm>1e-6,angle/norm.clamp_min(1e-8),2+norm.square()/3)
|
| 18 |
+
return q[...,1:]*scale
|
| 19 |
+
|
| 20 |
+
def matrix_to_six(m):return m[...,:2,:].reshape(*m.shape[:-2],6)
|
| 21 |
+
def six_to_matrix(x):
|
| 22 |
+
a,b=x[...,:3],x[...,3:];r0=F.normalize(a,dim=-1,eps=1e-7);r1=F.normalize(b-(r0*b).sum(-1,keepdim=True)*r0,dim=-1,eps=1e-7);r2=torch.cross(r0,r1,dim=-1);return torch.stack([r0,r1,r2],-2)
|
| 23 |
+
class HandVAE(nn.Module):
|
| 24 |
+
def __init__(self,latent_dim=20,width=256):
|
| 25 |
+
super().__init__();self.latent_dim=latent_dim;self.width=width
|
| 26 |
+
self.register_buffer('input_mean',torch.zeros(90));self.register_buffer('input_std',torch.ones(90))
|
| 27 |
+
self.encoder=nn.Sequential(nn.Linear(92,width),nn.LeakyReLU(.2),nn.Linear(width,width),nn.LeakyReLU(.2))
|
| 28 |
+
self.mu=nn.Linear(width,latent_dim);self.logvar=nn.Linear(width,latent_dim);nn.init.constant_(self.logvar.bias,-2)
|
| 29 |
+
self.decoder=nn.Sequential(nn.Linear(latent_dim+2,width),nn.LeakyReLU(.2),nn.Linear(width,width),nn.LeakyReLU(.2),nn.Linear(width,90))
|
| 30 |
+
def encode(self,rot,side):
|
| 31 |
+
x=(matrix_to_six(rot).flatten(-2)-self.input_mean)/self.input_std;h=self.encoder(torch.cat([x,F.one_hot(side.long(),2).to(x)],-1));return self.mu(h),self.logvar(h).clamp(-8,4)
|
| 32 |
+
def decode(self,z,side):
|
| 33 |
+
x=self.decoder(torch.cat([z,F.one_hot(side.long(),2).to(z)],-1))*self.input_std+self.input_mean;return six_to_matrix(x.reshape(*z.shape[:-1],15,6))
|
| 34 |
+
def forward(self,rot,side):
|
| 35 |
+
mu,lv=self.encode(rot,side);z=mu+torch.randn_like(mu)*(.5*lv).exp() if self.training else mu;return self.decode(z,side),mu,lv
|
| 36 |
+
|
| 37 |
+
def load_vae(path,device='cuda'):
|
| 38 |
+
ck=torch.load(path,map_location=device,weights_only=False);m=HandVAE(**ck['architecture']).to(device);m.load_state_dict(ck['state_dict'],strict=True);m.eval();m.requires_grad_(False);return m,ck
|
joint_mix40.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096
|
| 3 |
+
size 1959670
|
load_model.py
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Load the released tensor-only weights and checkpoint metadata."""
|
| 2 |
+
import json
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
|
| 5 |
+
from safetensors.torch import load_file
|
| 6 |
+
|
| 7 |
+
from bimanual_vae import BimanualVAE
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
def load_model(directory=None, device="cpu"):
|
| 11 |
+
directory = Path(directory) if directory else Path(__file__).resolve().parent
|
| 12 |
+
config = json.loads((directory / "config.json").read_text())
|
| 13 |
+
model = BimanualVAE(**config["architecture"])
|
| 14 |
+
model.load_state_dict(load_file(str(directory / "model.safetensors")), strict=True)
|
| 15 |
+
model.to(device).eval().requires_grad_(False)
|
| 16 |
+
return model, config
|
mano_compat.py
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Compatibility shim for the licensed MANO files and legacy chumpy loader."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import inspect
|
| 6 |
+
|
| 7 |
+
import numpy as np
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
def enable_legacy_mano_loading() -> None:
|
| 11 |
+
"""Restore names removed after the legacy MANO/chumpy release."""
|
| 12 |
+
|
| 13 |
+
if not hasattr(inspect, "getargspec"):
|
| 14 |
+
inspect.getargspec = inspect.getfullargspec # type: ignore[attr-defined]
|
| 15 |
+
aliases = {
|
| 16 |
+
"bool": bool,
|
| 17 |
+
"int": int,
|
| 18 |
+
"float": float,
|
| 19 |
+
"complex": complex,
|
| 20 |
+
"object": object,
|
| 21 |
+
"unicode": str,
|
| 22 |
+
"str": str,
|
| 23 |
+
}
|
| 24 |
+
for name, value in aliases.items():
|
| 25 |
+
if name not in np.__dict__:
|
| 26 |
+
setattr(np, name, value)
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6c4fe1b79138e4cd3b7ddccb45780479de2d25270be018e46be6696d78bd05f8
|
| 3 |
+
size 1954636
|
provenance.json
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"source_experiment": "MiroHand_P3R4_bimanual_VAE_20260927",
|
| 3 |
+
"release_date": "2026-11-15",
|
| 4 |
+
"unchanged_original_files": {
|
| 5 |
+
"joint_mix40.pt": {
|
| 6 |
+
"source_relative_path": "priors/joint_mix40.pt",
|
| 7 |
+
"sha256": "e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096"
|
| 8 |
+
},
|
| 9 |
+
"bimanual_vae.py": {
|
| 10 |
+
"source_relative_path": "code/bimanual_vae.py",
|
| 11 |
+
"sha256": "9e940527007fc2bf93809eba891bb18b76b75a9e30206a2ab0698704e431f3f6"
|
| 12 |
+
},
|
| 13 |
+
"hand_vae.py": {
|
| 14 |
+
"source_relative_path": "code/hand_vae.py",
|
| 15 |
+
"sha256": "5ee21d9a48ce2d8094f28c2ff1ae56a57d5c55244125e18b3c3e50d1a588979c"
|
| 16 |
+
},
|
| 17 |
+
"training_config.json": {
|
| 18 |
+
"source_relative_path": "priors/joint_mix40_config.json",
|
| 19 |
+
"sha256": "e4163ef9cc4f8b7add9d4e4350a853a145c842e51bab851101dcabb3c550c4ff"
|
| 20 |
+
},
|
| 21 |
+
"training/history.json": {
|
| 22 |
+
"source_relative_path": "priors/joint_mix40_history.json",
|
| 23 |
+
"sha256": "a83a1341b55522d329adbaab29937ba5a260f42091975098aacb116490ff9417"
|
| 24 |
+
},
|
| 25 |
+
"training/original_train_pair.py": {
|
| 26 |
+
"source_relative_path": "code/train_pair.py",
|
| 27 |
+
"sha256": "8c854506d4a084fcea9ce8988c7dc33fe628f22b8380832f7f45a54ffc8a716a"
|
| 28 |
+
},
|
| 29 |
+
"training/original_prepare_data.py": {
|
| 30 |
+
"source_relative_path": "code/prepare_data.py",
|
| 31 |
+
"sha256": "1c2bb19eeaf58e2f80b76e6ec556af02cb92cbb2ebfeaf580b7ed00fc86bac4d"
|
| 32 |
+
},
|
| 33 |
+
"training/original_common.py": {
|
| 34 |
+
"source_relative_path": "code/common.py",
|
| 35 |
+
"sha256": "f08a807e45d72cc2fe6de81f65b22db78075d16ac4dcfa390b32572ccc3e185e"
|
| 36 |
+
}
|
| 37 |
+
},
|
| 38 |
+
"converted_weights": {
|
| 39 |
+
"model.safetensors": "All checkpoint state_dict tensors, exactly equal; metadata in config.json"
|
| 40 |
+
},
|
| 41 |
+
"adapted_files": {
|
| 42 |
+
"train.py": "Original algorithm with CLI paths/device/run lengths and standalone compatibility helpers",
|
| 43 |
+
"augment.py": "Portable training-GT neighborhood sampling; query extraction and mesh generation external"
|
| 44 |
+
},
|
| 45 |
+
"trained_variant": "joint_mix40",
|
| 46 |
+
"checkpoint_epoch": 50,
|
| 47 |
+
"excluded_assets": [
|
| 48 |
+
"raw datasets",
|
| 49 |
+
"training motion cache",
|
| 50 |
+
"MANO model files",
|
| 51 |
+
"derived MANO buffers"
|
| 52 |
+
]
|
| 53 |
+
}
|
requirements-training.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
-r requirements.txt
|
| 2 |
+
smplx==0.1.28
|
| 3 |
+
chumpy==0.70
|
| 4 |
+
scipy>=1.10
|
requirements.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
torch>=2.0
|
| 2 |
+
numpy>=1.24
|
| 3 |
+
safetensors>=0.4
|
| 4 |
+
huggingface_hub>=0.25
|
train.py
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Training algorithm preserved from the original 2026-09-27 experiment.
|
| 2 |
+
Only environment/path plumbing and optional smoke-run lengths are adapted.
|
| 3 |
+
"""
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
import json, hashlib
|
| 6 |
+
import numpy as np
|
| 7 |
+
from mano_compat import enable_legacy_mano_loading
|
| 8 |
+
enable_legacy_mano_loading()
|
| 9 |
+
def sha(path):
|
| 10 |
+
with Path(path).open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest()
|
| 11 |
+
def write(path,value):
|
| 12 |
+
Path(path).parent.mkdir(parents=True,exist_ok=True)
|
| 13 |
+
Path(path).write_text(json.dumps(value,indent=2,allow_nan=False)+'\n')
|
| 14 |
+
import torch,time,smplx,argparse
|
| 15 |
+
from bimanual_vae import *
|
| 16 |
+
torch.set_num_threads(2)
|
| 17 |
+
def main(variant, data, mano_dir, output, device, epochs, steps_per_epoch):
|
| 18 |
+
output.mkdir(parents=True,exist_ok=True)
|
| 19 |
+
seed=20260927;torch.manual_seed(seed);rng=np.random.default_rng(seed);a=np.load(data,allow_pickle=False);local=aa_to_matrix(torch.tensor(a['pose_aa'],device=device)).reshape(-1,30,3,3);rot=torch.cat([local,torch.tensor(a['relative_rotation'],device=device)[:,None]],1);pos=torch.tensor(a['relative_wrist_m'],device=device);ds=a['dataset'];split=a['split'];seq=a['sequence'];included=[0,1] if variant=='joint_ego40' else [0,1,2];train=np.flatnonzero((split==0)&np.isin(ds,included));val=np.flatnonzero((split==1)&np.isin(ds,included));model=BimanualVAE(pose_only=variant=='pose_mix40').to(device);x=model.features(rot[train],pos[train]);model.input_mean.copy_(x.mean(0));model.input_std.copy_(x.std(0).clamp_min(.03));opt=torch.optim.AdamW(model.parameters(),lr=.001,weight_decay=1e-5)
|
| 20 |
+
# Balance datasets, then recordings; long recordings cannot dominate.
|
| 21 |
+
groups={d:{int(s):np.flatnonzero((split==0)&(ds==d)&(seq==s)) for s in np.unique(seq[(split==0)&(ds==d)])} for d in included}
|
| 22 |
+
# Simple FK of native 16 MANO skeletal joints, differentiated without rendering meshes.
|
| 23 |
+
offsets=[];parents=None
|
| 24 |
+
for s in range(2):
|
| 25 |
+
m=smplx.MANO(str(mano_dir),is_rhand=s==1,use_pca=False,flat_hand_mean=True);j=m.J_regressor@m.v_template;par=m.parents.numpy();off=j.clone();off[1:]-=j[par[1:]];off[0]=0;offsets.append(off);parents=par
|
| 26 |
+
off=torch.stack(offsets).to(device)
|
| 27 |
+
def fk(q):
|
| 28 |
+
q=q[:,:30].reshape(-1,2,15,3,3);n=len(q);g=[torch.eye(3,device=device).expand(n,2,3,3)];j=[torch.zeros(n,2,3,device=device)]
|
| 29 |
+
for k in range(1,16):
|
| 30 |
+
p=int(parents[k]);j.append(j[p]+(g[p]@off[None,:,k,:,None]).squeeze(-1));g.append(g[p]@q[:,:,k-1])
|
| 31 |
+
return torch.stack(j,2)
|
| 32 |
+
with torch.no_grad():ref=fk(rot)
|
| 33 |
+
cfg=dict(variant=variant,seed=seed,epochs=epochs,steps_per_epoch=steps_per_epoch,batch=192,lr=.001,kl_beta=.0005,joint_weight=.25,position_scale_m=.1,balanced_dataset_and_sequence=True,included_datasets=included,selection='equal-dataset validation local FK mm + 0.25 relative position mm + relative rotation deg; test not used',data_sha256=sha(data))
|
| 34 |
+
write(output/f'{variant}_config.json',cfg);best=float('inf');hist=[];start=time.monotonic()
|
| 35 |
+
for epoch in range(1,cfg['epochs']+1):
|
| 36 |
+
model.train();sums=[]
|
| 37 |
+
for step in range(cfg['steps_per_epoch']):
|
| 38 |
+
ids=[]
|
| 39 |
+
for d in included:
|
| 40 |
+
keys=list(groups[d]);ss=rng.choice(keys,192//len(included));ids.extend(int(rng.choice(groups[d][int(s)])) for s in ss)
|
| 41 |
+
ids=torch.tensor(ids,device=device);rr,pp,mu,lv=model(rot[ids],pos[ids]);rec=(rr-rot[ids,:model.nrot]).square().sum((-1,-2)).mean();jloss=((fk(rr)-ref[ids])/.02).square().sum(-1).mean();ploss=rec*0 if model.pose_only else ((pp-pos[ids])/.1).square().sum(-1).mean();kl=.5*(mu.square()+lv.exp()-lv-1).mean();loss=rec+.25*jloss+ploss+.0005*min(1,epoch/20)*kl;opt.zero_grad();loss.backward();torch.nn.utils.clip_grad_norm_(model.parameters(),5);opt.step();sums.append(float(loss.detach()))
|
| 42 |
+
if epoch==1 or epoch%5==0:
|
| 43 |
+
model.eval();met=[]
|
| 44 |
+
with torch.no_grad():
|
| 45 |
+
for d in included:
|
| 46 |
+
ids=val[ds[val]==d];errs=[];pe=[];re=[]
|
| 47 |
+
for t in range(0,len(ids),1024):
|
| 48 |
+
ix=ids[t:t+1024];rr,pp,mu,lv=model(rot[ix],pos[ix]);errs.append((fk(rr)-ref[ix]).norm(dim=-1).mean((1,2))*1000)
|
| 49 |
+
if not model.pose_only:pe.append((pp-pos[ix]).norm(dim=-1)*1000);rel=rr[:,30]@rot[ix,30].transpose(-1,-2);re.append(torch.acos(((rel.diagonal(dim1=-2,dim2=-1).sum(-1)-1)/2).clamp(-1+1e-7,1-1e-7))*180/torch.pi)
|
| 50 |
+
met.append(dict(dataset=['ARCTIC','H2O','GRAB'][d],local_fk_mm=float(torch.cat(errs).mean()),relative_wrist_mm=float(torch.cat(pe).mean()) if pe else None,relative_rotation_deg=float(torch.cat(re).mean()) if re else None))
|
| 51 |
+
score=float(np.mean([v['local_fk_mm']+.25*(v['relative_wrist_mm'] or 0)+(v['relative_rotation_deg'] or 0) for v in met]));entry=dict(epoch=epoch,train_loss=float(np.mean(sums)),validation=met,selection_score=score,seconds=time.monotonic()-start);hist.append(entry);print(variant,json.dumps(entry),flush=True)
|
| 52 |
+
if score<best:
|
| 53 |
+
best=score;torch.save(dict(architecture=dict(latent_dim=40,width=384,pose_only=model.pose_only),state_dict=model.state_dict(),config=cfg,epoch=epoch,validation=met),output/f'{variant}.pt')
|
| 54 |
+
write(output/f'{variant}_history.json',hist)
|
| 55 |
+
print('COMPLETE',variant,sha(output/f'{variant}.pt'),flush=True)
|
| 56 |
+
if __name__=='__main__':
|
| 57 |
+
p=argparse.ArgumentParser(description='Original paired-hand VAE recipe with portable paths.')
|
| 58 |
+
p.add_argument('--variant',choices=['joint_ego40','joint_mix40','pose_mix40'],default='joint_mix40')
|
| 59 |
+
p.add_argument('--data',type=Path,required=True)
|
| 60 |
+
p.add_argument('--mano-dir',type=Path,required=True)
|
| 61 |
+
p.add_argument('--output',type=Path,required=True)
|
| 62 |
+
p.add_argument('--device',default='cuda')
|
| 63 |
+
p.add_argument('--epochs',type=int,default=60)
|
| 64 |
+
p.add_argument('--steps-per-epoch',type=int,default=48)
|
| 65 |
+
a=p.parse_args()
|
| 66 |
+
if a.epochs<1 or a.steps_per_epoch<1:p.error('Training counts must be positive.')
|
| 67 |
+
if (a.output/(a.variant+'.pt')).exists():p.error('Output checkpoint already exists; choose a new output folder.')
|
| 68 |
+
main(a.variant,a.data,a.mano_dir,a.output,a.device,a.epochs,a.steps_per_epoch)
|
training/README.md
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Original training provenance
|
| 2 |
+
|
| 3 |
+
`original_train_pair.py`, `original_prepare_data.py`, and `original_common.py`
|
| 4 |
+
are unchanged copies from the 2026-09-27 MiroHand P3R4 experiment. They record the
|
| 5 |
+
exact model-training recipe and source-specific data preparation. `history.json`
|
| 6 |
+
is the original joint_mix40 validation history. These archived scripts retain
|
| 7 |
+
the original machine paths and are for reference; use `../train.py` to run
|
| 8 |
+
training with your own paths.
|
| 9 |
+
|
| 10 |
+
The original preparation depended on an earlier filtered ARCTIC/H2O manifest,
|
| 11 |
+
native MANO files, raw GRAB motion, and experiment-specific benchmark exclusion
|
| 12 |
+
lists. Those inputs are not bundled. `../DATA_FORMAT.md` defines the prepared
|
| 13 |
+
cache that the portable trainer consumes. The original training cache SHA-256 is
|
| 14 |
+
recorded in `../training_config.json`.
|
| 15 |
+
|
| 16 |
+
The portable trainer changes only path/device arguments, configurable epoch/step
|
| 17 |
+
counts, local JSON/hash utilities, and the MANO compatibility import. Dataset and
|
| 18 |
+
recording balancing, normalization, losses, optimizer, random seed, validation
|
| 19 |
+
selection, and model code follow the original recipe. CPU smoke checks exercise
|
| 20 |
+
one training update and validation; they do not reproduce the full GPU run.
|
| 21 |
+
|
| 22 |
+
The original experiment trained three variants. Only joint_mix40 weights are
|
| 23 |
+
released in this repository; the other variant names in the scripts are options
|
| 24 |
+
for training new models.
|
training/history.json
ADDED
|
@@ -0,0 +1,340 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"epoch": 1,
|
| 4 |
+
"train_loss": 9.445151468118032,
|
| 5 |
+
"validation": [
|
| 6 |
+
{
|
| 7 |
+
"dataset": "ARCTIC",
|
| 8 |
+
"local_fk_mm": 12.87500286102295,
|
| 9 |
+
"relative_wrist_mm": 163.0471954345703,
|
| 10 |
+
"relative_rotation_deg": 51.87311553955078
|
| 11 |
+
},
|
| 12 |
+
{
|
| 13 |
+
"dataset": "H2O",
|
| 14 |
+
"local_fk_mm": 9.052749633789062,
|
| 15 |
+
"relative_wrist_mm": 117.507080078125,
|
| 16 |
+
"relative_rotation_deg": 38.650142669677734
|
| 17 |
+
},
|
| 18 |
+
{
|
| 19 |
+
"dataset": "GRAB",
|
| 20 |
+
"local_fk_mm": 8.72321891784668,
|
| 21 |
+
"relative_wrist_mm": 189.7671661376953,
|
| 22 |
+
"relative_rotation_deg": 58.388023376464844
|
| 23 |
+
}
|
| 24 |
+
],
|
| 25 |
+
"selection_score": 99.0475378036499,
|
| 26 |
+
"seconds": 0.3951757492031902
|
| 27 |
+
},
|
| 28 |
+
{
|
| 29 |
+
"epoch": 5,
|
| 30 |
+
"train_loss": 0.5073744927843412,
|
| 31 |
+
"validation": [
|
| 32 |
+
{
|
| 33 |
+
"dataset": "ARCTIC",
|
| 34 |
+
"local_fk_mm": 7.015540599822998,
|
| 35 |
+
"relative_wrist_mm": 60.61729431152344,
|
| 36 |
+
"relative_rotation_deg": 38.23624038696289
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"dataset": "H2O",
|
| 40 |
+
"local_fk_mm": 4.575192451477051,
|
| 41 |
+
"relative_wrist_mm": 51.943145751953125,
|
| 42 |
+
"relative_rotation_deg": 29.724584579467773
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"dataset": "GRAB",
|
| 46 |
+
"local_fk_mm": 5.3200531005859375,
|
| 47 |
+
"relative_wrist_mm": 54.36599349975586,
|
| 48 |
+
"relative_rotation_deg": 44.823211669921875
|
| 49 |
+
}
|
| 50 |
+
],
|
| 51 |
+
"selection_score": 57.14214372634888,
|
| 52 |
+
"seconds": 1.4217542971018702
|
| 53 |
+
},
|
| 54 |
+
{
|
| 55 |
+
"epoch": 10,
|
| 56 |
+
"train_loss": 0.2648027865216136,
|
| 57 |
+
"validation": [
|
| 58 |
+
{
|
| 59 |
+
"dataset": "ARCTIC",
|
| 60 |
+
"local_fk_mm": 5.106489181518555,
|
| 61 |
+
"relative_wrist_mm": 33.648101806640625,
|
| 62 |
+
"relative_rotation_deg": 31.729633331298828
|
| 63 |
+
},
|
| 64 |
+
{
|
| 65 |
+
"dataset": "H2O",
|
| 66 |
+
"local_fk_mm": 3.4149880409240723,
|
| 67 |
+
"relative_wrist_mm": 25.74277114868164,
|
| 68 |
+
"relative_rotation_deg": 23.669815063476562
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"dataset": "GRAB",
|
| 72 |
+
"local_fk_mm": 4.339122295379639,
|
| 73 |
+
"relative_wrist_mm": 38.02174758911133,
|
| 74 |
+
"relative_rotation_deg": 43.17218017578125
|
| 75 |
+
}
|
| 76 |
+
],
|
| 77 |
+
"selection_score": 45.26179440816244,
|
| 78 |
+
"seconds": 2.826522510033101
|
| 79 |
+
},
|
| 80 |
+
{
|
| 81 |
+
"epoch": 15,
|
| 82 |
+
"train_loss": 0.16469275082151094,
|
| 83 |
+
"validation": [
|
| 84 |
+
{
|
| 85 |
+
"dataset": "ARCTIC",
|
| 86 |
+
"local_fk_mm": 4.236188888549805,
|
| 87 |
+
"relative_wrist_mm": 30.4024715423584,
|
| 88 |
+
"relative_rotation_deg": 26.648405075073242
|
| 89 |
+
},
|
| 90 |
+
{
|
| 91 |
+
"dataset": "H2O",
|
| 92 |
+
"local_fk_mm": 2.940308094024658,
|
| 93 |
+
"relative_wrist_mm": 34.45616149902344,
|
| 94 |
+
"relative_rotation_deg": 16.92371940612793
|
| 95 |
+
},
|
| 96 |
+
{
|
| 97 |
+
"dataset": "GRAB",
|
| 98 |
+
"local_fk_mm": 3.87188458442688,
|
| 99 |
+
"relative_wrist_mm": 43.58067321777344,
|
| 100 |
+
"relative_rotation_deg": 35.69742202758789
|
| 101 |
+
}
|
| 102 |
+
],
|
| 103 |
+
"selection_score": 39.14258488019308,
|
| 104 |
+
"seconds": 4.37071700906381
|
| 105 |
+
},
|
| 106 |
+
{
|
| 107 |
+
"epoch": 20,
|
| 108 |
+
"train_loss": 0.14342589862644672,
|
| 109 |
+
"validation": [
|
| 110 |
+
{
|
| 111 |
+
"dataset": "ARCTIC",
|
| 112 |
+
"local_fk_mm": 3.692901849746704,
|
| 113 |
+
"relative_wrist_mm": 26.702104568481445,
|
| 114 |
+
"relative_rotation_deg": 21.544221878051758
|
| 115 |
+
},
|
| 116 |
+
{
|
| 117 |
+
"dataset": "H2O",
|
| 118 |
+
"local_fk_mm": 2.599233627319336,
|
| 119 |
+
"relative_wrist_mm": 25.14226722717285,
|
| 120 |
+
"relative_rotation_deg": 13.099417686462402
|
| 121 |
+
},
|
| 122 |
+
{
|
| 123 |
+
"dataset": "GRAB",
|
| 124 |
+
"local_fk_mm": 3.521355390548706,
|
| 125 |
+
"relative_wrist_mm": 24.10993003845215,
|
| 126 |
+
"relative_rotation_deg": 26.255090713500977
|
| 127 |
+
}
|
| 128 |
+
],
|
| 129 |
+
"selection_score": 29.90026553471883,
|
| 130 |
+
"seconds": 5.522418719017878
|
| 131 |
+
},
|
| 132 |
+
{
|
| 133 |
+
"epoch": 25,
|
| 134 |
+
"train_loss": 0.12864494665215412,
|
| 135 |
+
"validation": [
|
| 136 |
+
{
|
| 137 |
+
"dataset": "ARCTIC",
|
| 138 |
+
"local_fk_mm": 3.295466184616089,
|
| 139 |
+
"relative_wrist_mm": 36.14392852783203,
|
| 140 |
+
"relative_rotation_deg": 19.423234939575195
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
"dataset": "H2O",
|
| 144 |
+
"local_fk_mm": 2.3364412784576416,
|
| 145 |
+
"relative_wrist_mm": 32.38401412963867,
|
| 146 |
+
"relative_rotation_deg": 12.863519668579102
|
| 147 |
+
},
|
| 148 |
+
{
|
| 149 |
+
"dataset": "GRAB",
|
| 150 |
+
"local_fk_mm": 3.4695565700531006,
|
| 151 |
+
"relative_wrist_mm": 37.982025146484375,
|
| 152 |
+
"relative_rotation_deg": 25.931699752807617
|
| 153 |
+
}
|
| 154 |
+
],
|
| 155 |
+
"selection_score": 31.315803448359173,
|
| 156 |
+
"seconds": 6.841986288083717
|
| 157 |
+
},
|
| 158 |
+
{
|
| 159 |
+
"epoch": 30,
|
| 160 |
+
"train_loss": 0.08499159834658106,
|
| 161 |
+
"validation": [
|
| 162 |
+
{
|
| 163 |
+
"dataset": "ARCTIC",
|
| 164 |
+
"local_fk_mm": 3.0357227325439453,
|
| 165 |
+
"relative_wrist_mm": 20.172719955444336,
|
| 166 |
+
"relative_rotation_deg": 17.05535125732422
|
| 167 |
+
},
|
| 168 |
+
{
|
| 169 |
+
"dataset": "H2O",
|
| 170 |
+
"local_fk_mm": 2.1297340393066406,
|
| 171 |
+
"relative_wrist_mm": 12.941731452941895,
|
| 172 |
+
"relative_rotation_deg": 10.320504188537598
|
| 173 |
+
},
|
| 174 |
+
{
|
| 175 |
+
"dataset": "GRAB",
|
| 176 |
+
"local_fk_mm": 3.1395211219787598,
|
| 177 |
+
"relative_wrist_mm": 17.7632999420166,
|
| 178 |
+
"relative_rotation_deg": 20.985870361328125
|
| 179 |
+
}
|
| 180 |
+
],
|
| 181 |
+
"selection_score": 23.128713846206665,
|
| 182 |
+
"seconds": 8.110679685138166
|
| 183 |
+
},
|
| 184 |
+
{
|
| 185 |
+
"epoch": 35,
|
| 186 |
+
"train_loss": 0.0695953315589577,
|
| 187 |
+
"validation": [
|
| 188 |
+
{
|
| 189 |
+
"dataset": "ARCTIC",
|
| 190 |
+
"local_fk_mm": 2.890573740005493,
|
| 191 |
+
"relative_wrist_mm": 18.770294189453125,
|
| 192 |
+
"relative_rotation_deg": 16.014467239379883
|
| 193 |
+
},
|
| 194 |
+
{
|
| 195 |
+
"dataset": "H2O",
|
| 196 |
+
"local_fk_mm": 1.9951664209365845,
|
| 197 |
+
"relative_wrist_mm": 11.237444877624512,
|
| 198 |
+
"relative_rotation_deg": 9.762822151184082
|
| 199 |
+
},
|
| 200 |
+
{
|
| 201 |
+
"dataset": "GRAB",
|
| 202 |
+
"local_fk_mm": 2.955854654312134,
|
| 203 |
+
"relative_wrist_mm": 16.69818878173828,
|
| 204 |
+
"relative_rotation_deg": 19.13675308227539
|
| 205 |
+
}
|
| 206 |
+
],
|
| 207 |
+
"selection_score": 21.477373083432514,
|
| 208 |
+
"seconds": 9.64998462004587
|
| 209 |
+
},
|
| 210 |
+
{
|
| 211 |
+
"epoch": 40,
|
| 212 |
+
"train_loss": 0.07276629415961604,
|
| 213 |
+
"validation": [
|
| 214 |
+
{
|
| 215 |
+
"dataset": "ARCTIC",
|
| 216 |
+
"local_fk_mm": 2.755661964416504,
|
| 217 |
+
"relative_wrist_mm": 17.329029083251953,
|
| 218 |
+
"relative_rotation_deg": 14.625399589538574
|
| 219 |
+
},
|
| 220 |
+
{
|
| 221 |
+
"dataset": "H2O",
|
| 222 |
+
"local_fk_mm": 1.9094189405441284,
|
| 223 |
+
"relative_wrist_mm": 9.26237678527832,
|
| 224 |
+
"relative_rotation_deg": 9.038466453552246
|
| 225 |
+
},
|
| 226 |
+
{
|
| 227 |
+
"dataset": "GRAB",
|
| 228 |
+
"local_fk_mm": 2.7325074672698975,
|
| 229 |
+
"relative_wrist_mm": 16.747617721557617,
|
| 230 |
+
"relative_rotation_deg": 20.136404037475586
|
| 231 |
+
}
|
| 232 |
+
],
|
| 233 |
+
"selection_score": 20.67753811677297,
|
| 234 |
+
"seconds": 10.801130139036104
|
| 235 |
+
},
|
| 236 |
+
{
|
| 237 |
+
"epoch": 45,
|
| 238 |
+
"train_loss": 0.05866521876305342,
|
| 239 |
+
"validation": [
|
| 240 |
+
{
|
| 241 |
+
"dataset": "ARCTIC",
|
| 242 |
+
"local_fk_mm": 2.6570005416870117,
|
| 243 |
+
"relative_wrist_mm": 16.533859252929688,
|
| 244 |
+
"relative_rotation_deg": 13.954176902770996
|
| 245 |
+
},
|
| 246 |
+
{
|
| 247 |
+
"dataset": "H2O",
|
| 248 |
+
"local_fk_mm": 1.812648057937622,
|
| 249 |
+
"relative_wrist_mm": 10.88892650604248,
|
| 250 |
+
"relative_rotation_deg": 8.842887878417969
|
| 251 |
+
},
|
| 252 |
+
{
|
| 253 |
+
"dataset": "GRAB",
|
| 254 |
+
"local_fk_mm": 2.5590364933013916,
|
| 255 |
+
"relative_wrist_mm": 19.327693939208984,
|
| 256 |
+
"relative_rotation_deg": 18.18161964416504
|
| 257 |
+
}
|
| 258 |
+
],
|
| 259 |
+
"selection_score": 19.898329814275105,
|
| 260 |
+
"seconds": 12.013499305117875
|
| 261 |
+
},
|
| 262 |
+
{
|
| 263 |
+
"epoch": 50,
|
| 264 |
+
"train_loss": 0.05538660536209742,
|
| 265 |
+
"validation": [
|
| 266 |
+
{
|
| 267 |
+
"dataset": "ARCTIC",
|
| 268 |
+
"local_fk_mm": 2.552030563354492,
|
| 269 |
+
"relative_wrist_mm": 15.961709022521973,
|
| 270 |
+
"relative_rotation_deg": 13.042522430419922
|
| 271 |
+
},
|
| 272 |
+
{
|
| 273 |
+
"dataset": "H2O",
|
| 274 |
+
"local_fk_mm": 1.722329020500183,
|
| 275 |
+
"relative_wrist_mm": 7.883533000946045,
|
| 276 |
+
"relative_rotation_deg": 8.399897575378418
|
| 277 |
+
},
|
| 278 |
+
{
|
| 279 |
+
"dataset": "GRAB",
|
| 280 |
+
"local_fk_mm": 2.7820262908935547,
|
| 281 |
+
"relative_wrist_mm": 18.310850143432617,
|
| 282 |
+
"relative_rotation_deg": 17.670961380004883
|
| 283 |
+
}
|
| 284 |
+
],
|
| 285 |
+
"selection_score": 18.90293010075887,
|
| 286 |
+
"seconds": 13.535741310101002
|
| 287 |
+
},
|
| 288 |
+
{
|
| 289 |
+
"epoch": 55,
|
| 290 |
+
"train_loss": 0.10357473092153668,
|
| 291 |
+
"validation": [
|
| 292 |
+
{
|
| 293 |
+
"dataset": "ARCTIC",
|
| 294 |
+
"local_fk_mm": 2.5025906562805176,
|
| 295 |
+
"relative_wrist_mm": 32.9027214050293,
|
| 296 |
+
"relative_rotation_deg": 12.71257495880127
|
| 297 |
+
},
|
| 298 |
+
{
|
| 299 |
+
"dataset": "H2O",
|
| 300 |
+
"local_fk_mm": 1.7116860151290894,
|
| 301 |
+
"relative_wrist_mm": 24.95145034790039,
|
| 302 |
+
"relative_rotation_deg": 8.395352363586426
|
| 303 |
+
},
|
| 304 |
+
{
|
| 305 |
+
"dataset": "GRAB",
|
| 306 |
+
"local_fk_mm": 2.518946647644043,
|
| 307 |
+
"relative_wrist_mm": 29.448965072631836,
|
| 308 |
+
"relative_rotation_deg": 18.022743225097656
|
| 309 |
+
}
|
| 310 |
+
],
|
| 311 |
+
"selection_score": 22.563226024309795,
|
| 312 |
+
"seconds": 14.875219596084207
|
| 313 |
+
},
|
| 314 |
+
{
|
| 315 |
+
"epoch": 60,
|
| 316 |
+
"train_loss": 0.04643667683315774,
|
| 317 |
+
"validation": [
|
| 318 |
+
{
|
| 319 |
+
"dataset": "ARCTIC",
|
| 320 |
+
"local_fk_mm": 2.3993451595306396,
|
| 321 |
+
"relative_wrist_mm": 22.116172790527344,
|
| 322 |
+
"relative_rotation_deg": 12.27881908416748
|
| 323 |
+
},
|
| 324 |
+
{
|
| 325 |
+
"dataset": "H2O",
|
| 326 |
+
"local_fk_mm": 1.6551059484481812,
|
| 327 |
+
"relative_wrist_mm": 18.894102096557617,
|
| 328 |
+
"relative_rotation_deg": 8.041049003601074
|
| 329 |
+
},
|
| 330 |
+
{
|
| 331 |
+
"dataset": "GRAB",
|
| 332 |
+
"local_fk_mm": 2.4477932453155518,
|
| 333 |
+
"relative_wrist_mm": 16.990732192993164,
|
| 334 |
+
"relative_rotation_deg": 16.812602996826172
|
| 335 |
+
}
|
| 336 |
+
],
|
| 337 |
+
"selection_score": 19.37832240263621,
|
| 338 |
+
"seconds": 16.251781224040315
|
| 339 |
+
}
|
| 340 |
+
]
|
training/original_common.py
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import sys,json,hashlib,inspect
|
| 3 |
+
import numpy as np
|
| 4 |
+
R=Path('/SSD3/yangzijing/MiroHand_P3R4_bimanual_VAE_20260927')
|
| 5 |
+
OLD=Path('/SSD3/yangzijing/FoundationHand-A2_P3R3_VAE_20260922')
|
| 6 |
+
REPO=Path('/SSD2/yangzijing/egoview-v9-foundationhand-s9l10-20260901')
|
| 7 |
+
MANO=Path('/SSD3/yangzijing/shared_dependencies/foundation_hand/pretrained/mano')
|
| 8 |
+
sys.path.insert(0,str(REPO));inspect.getargspec=inspect.getfullargspec
|
| 9 |
+
from pipeline.utils.numpy_compat import ensure_chumpy_numpy_compat
|
| 10 |
+
ensure_chumpy_numpy_compat()
|
| 11 |
+
def read(p):return json.loads(Path(p).read_text())
|
| 12 |
+
def write(p,x):Path(p).parent.mkdir(parents=True,exist_ok=True);Path(p).write_text(json.dumps(x,ensure_ascii=False,indent=2,allow_nan=False)+'\n')
|
| 13 |
+
def sha(p):
|
| 14 |
+
with Path(p).open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest()
|
training/original_prepare_data.py
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from common import *
|
| 2 |
+
import collections,trimesh,smplx,torch,warnings
|
| 3 |
+
from scipy.spatial.transform import Rotation
|
| 4 |
+
warnings.filterwarnings('ignore',category=DeprecationWarning);torch.set_num_threads(2)
|
| 5 |
+
F=Path('/SSD3/yangzijing/ego_full_training_filter_visible_20260925');G=Path('/SSD1/wbr/datasets/GRAB_source/GRAB_unzipped')
|
| 6 |
+
models=[smplx.MANO(str(MANO),is_rhand=s==1,use_pca=True,num_pca_comps=24,flat_hand_mean=False) for s in range(2)]
|
| 7 |
+
ex=set(map(tuple,read(OLD/'priors/excluded_recordings.json')))
|
| 8 |
+
for c in read('/SSD3/yangzijing/MiroHand_A6_benchmark_20260924/selection.json'):ex.add((c['dataset'],c['recording_id']))
|
| 9 |
+
write(R/'audit/excluded_pipeline_recordings.json',sorted(ex))
|
| 10 |
+
seqs=[];refs={};summary=collections.Counter();data=collections.defaultdict(list)
|
| 11 |
+
def add(ds,seq,pose,rot,wrist,fids,source,split=None):
|
| 12 |
+
pose=np.asarray(pose,np.float32).reshape(-1,2,15,3);rot=np.asarray(rot,np.float32).reshape(-1,2,3);wrist=np.asarray(wrist,np.float32);n=len(pose)
|
| 13 |
+
good=np.isfinite(pose).all((1,2,3))&np.isfinite(rot).all((1,2))&np.isfinite(wrist).all((1,2));summary[ds+'_nonfinite_removed']+=int((~good).sum());pose,rot,wrist,fids=pose[good],rot[good],wrist[good],np.asarray(fids)[good]
|
| 14 |
+
if not len(pose):return
|
| 15 |
+
# Each recording contributes at most 160 evenly spaced real paired frames.
|
| 16 |
+
ids=np.unique(np.linspace(0,len(pose)-1,min(160,len(pose))).astype(int));pose,rot,wrist,fids=pose[ids],rot[ids],wrist[ids],fids[ids]
|
| 17 |
+
rr=Rotation.from_rotvec(rot.reshape(-1,3)).as_matrix().reshape(-1,2,3,3);relrot=rr[:,0].transpose(0,2,1)@rr[:,1];relpos=np.einsum('nji,nj->ni',rr[:,0],wrist[:,1]-wrist[:,0]);dist=np.linalg.norm(relpos,axis=-1);assert np.isfinite(relpos).all() and dist.max()<3,(ds,seq,dist.max())
|
| 18 |
+
# Invariant to shared camera/world rigid transform. Keep anchor for exact roundtrip.
|
| 19 |
+
reconstructed=np.einsum('nij,nj->ni',rr[:,0],relpos)+wrist[:,0];assert np.max(abs(reconstructed-wrist[:,1]))<1e-5
|
| 20 |
+
if split is None:
|
| 21 |
+
h=int(hashlib.sha256(('P3R4_20260927:'+ds+':'+seq).encode()).hexdigest()[:8],16)%10;split='test' if h==0 else 'validation' if h==1 else 'train'
|
| 22 |
+
sid=len(seqs);seqs.append(dict(dataset=ds,recording=seq,split=split,frames=len(pose),source=source))
|
| 23 |
+
for k,v in dict(pose_aa=pose,relative_rotation=relrot.astype(np.float32),relative_wrist_m=relpos.astype(np.float32),anchor_rotation=rr[:,0].astype(np.float32),anchor_wrist_m=wrist[:,0],frame_ids=fids,sequence=np.full(len(pose),sid),dataset=np.full(len(pose),['ARCTIC','H2O','GRAB'].index(ds)),split=np.full(len(pose),['train','validation','test'].index(split))).items():data[k].append(v)
|
| 24 |
+
summary[ds+'_'+split]+=len(pose)
|
| 25 |
+
rows=[w for w in map(json.loads,(F/'unified/filtered_manifest.jsonl').open()) if w['dataset'] in ['ARCTIC','H2O'] and len(w['target_sides'])==2]
|
| 26 |
+
groups=collections.defaultdict(list)
|
| 27 |
+
for w in rows:
|
| 28 |
+
rec=w['recording_id'];canonical=rec.removesuffix('/cam4')
|
| 29 |
+
if (w['dataset'],rec) in ex or (w['dataset'],canonical) in ex:summary[w['dataset']+'_excluded_windows']+=1;continue
|
| 30 |
+
groups[(w['dataset'],canonical)].append(w)
|
| 31 |
+
for (ds,seq),ws in sorted(groups.items()):
|
| 32 |
+
ar=collections.defaultdict(list);paths=[]
|
| 33 |
+
for w in sorted(ws,key=lambda x:x['start']):
|
| 34 |
+
p=Path(w['filtered_gt_path']);assert sha(p)==w['gt_sha256'];refs[str(p)]=w['gt_sha256'];z=np.load(p);ok=z['valid'].all(1)
|
| 35 |
+
# Original local_pose includes the native MANO hand mean already.
|
| 36 |
+
for k in ['local_pose_aa_rad','native_global_orient_aa_rad','joints_native_m','frame_ids']:ar[k].append(z[k][ok])
|
| 37 |
+
paths.append(str(p))
|
| 38 |
+
ar={k:np.concatenate(v) for k,v in ar.items()};add(ds,seq,ar['local_pose_aa_rad'],ar['native_global_orient_aa_rad'],ar['joints_native_m'][:,:,0],ar['frame_ids'],paths)
|
| 39 |
+
print('EGO',dict(summary),flush=True)
|
| 40 |
+
rootcache={};convertmax=0.
|
| 41 |
+
for i,p in enumerate(sorted((G/'grab').glob('*/*.npz'))):
|
| 42 |
+
a=np.load(p,allow_pickle=True);pose=[];rot=[];wrist=[];n=int(a['n_frames']);ids=np.arange(0,n,12) # 120fps to 10fps; no interpolation
|
| 43 |
+
for s,key in enumerate(['lhand','rhand']):
|
| 44 |
+
h=a[key].item();q=h['params'];m=models[s];local=q['hand_pose'][ids]@m.hand_components.numpy()+m.hand_mean.numpy();err=float(np.max(abs(local-(q['fullpose'][ids]+m.hand_mean.numpy()))));assert err<2e-5;convertmax=max(convertmax,err)
|
| 45 |
+
template=G/h['vtemp']
|
| 46 |
+
if str(template) not in rootcache:
|
| 47 |
+
vt=np.asarray(trimesh.load(template,process=False).vertices);assert vt.shape==(778,3);rootcache[str(template)]=m.J_regressor[0].numpy()@vt;refs[str(template)]=sha(template)
|
| 48 |
+
pose.append(local);rot.append(q['global_orient'][ids]);wrist.append(q['transl'][ids]+rootcache[str(template)])
|
| 49 |
+
refs[str(p)]=sha(p);sub=p.parent.name;split='test' if sub=='s10' else 'validation' if sub=='s9' else 'train';add('GRAB',str(p.relative_to(G/'grab')).removesuffix('.npz'),np.stack(pose,1),np.stack(rot,1),np.stack(wrist,1),ids,[str(p)],split)
|
| 50 |
+
if i%200==0:print('GRAB',i,dict(summary),flush=True)
|
| 51 |
+
np.savez_compressed(R/'data/paired_motion.npz',**{k:np.concatenate(v) for k,v in data.items()});write(R/'data/sequences.json',seqs);write(R/'audit/source_hashes.json',refs)
|
| 52 |
+
old=read(OLD/'priors/training_manifest_local_only.json');write(R/'audit/previous_single_vae_data.json',dict(counts=old['counts'],data_sha256=old['pose_data_sha256'],note='DexYCB and HO3D single-hand supervision is not converted into artificial paired samples'))
|
| 53 |
+
write(R/'data/summary.json',dict(counts=dict(summary),sequences=len(seqs),paired_frames=sum(x['frames'] for x in seqs),data_sha256=sha(R/'data/paired_motion.npz'),grab_pca_fullpose_max_error_rad=convertmax,coordinate='left wrist MANO local frame; invert using anchor wrist pose in ego camera; no fake camera GT',schema='30 local SO3 rotations + relative wrist SO3 + relative wrist XYZ metres; 189D rotation6D input',sources_missing=['HIMO','Embody3D'],split='GRAB subject s1-s8 train, s9 validation, s10 test; ARCTIC/H2O deterministic sequence split 80/10/10; pipeline benchmark recordings excluded'))
|
| 54 |
+
print('COMPLETE',read(R/'data/summary.json'),flush=True)
|
training/original_train_pair.py
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from common import *
|
| 2 |
+
import torch,time,smplx,argparse
|
| 3 |
+
from bimanual_vae import *
|
| 4 |
+
torch.set_num_threads(2)
|
| 5 |
+
def main(variant):
|
| 6 |
+
seed=20260927;torch.manual_seed(seed);rng=np.random.default_rng(seed);a=np.load(R/'data/paired_motion.npz');device='cuda';local=aa_to_matrix(torch.tensor(a['pose_aa'],device=device)).reshape(-1,30,3,3);rot=torch.cat([local,torch.tensor(a['relative_rotation'],device=device)[:,None]],1);pos=torch.tensor(a['relative_wrist_m'],device=device);ds=a['dataset'];split=a['split'];seq=a['sequence'];included=[0,1] if variant=='joint_ego40' else [0,1,2];train=np.flatnonzero((split==0)&np.isin(ds,included));val=np.flatnonzero((split==1)&np.isin(ds,included));model=BimanualVAE(pose_only=variant=='pose_mix40').to(device);x=model.features(rot[train],pos[train]);model.input_mean.copy_(x.mean(0));model.input_std.copy_(x.std(0).clamp_min(.03));opt=torch.optim.AdamW(model.parameters(),lr=.001,weight_decay=1e-5)
|
| 7 |
+
# Balance datasets, then recordings; long recordings cannot dominate.
|
| 8 |
+
groups={d:{int(s):np.flatnonzero((split==0)&(ds==d)&(seq==s)) for s in np.unique(seq[(split==0)&(ds==d)])} for d in included}
|
| 9 |
+
# Simple FK of native 16 MANO skeletal joints, differentiated without rendering meshes.
|
| 10 |
+
offsets=[];parents=None
|
| 11 |
+
for s in range(2):
|
| 12 |
+
m=smplx.MANO(str(MANO),is_rhand=s==1,use_pca=False,flat_hand_mean=True);j=m.J_regressor@m.v_template;par=m.parents.numpy();off=j.clone();off[1:]-=j[par[1:]];off[0]=0;offsets.append(off);parents=par
|
| 13 |
+
off=torch.stack(offsets).to(device)
|
| 14 |
+
def fk(q):
|
| 15 |
+
q=q[:,:30].reshape(-1,2,15,3,3);n=len(q);g=[torch.eye(3,device=device).expand(n,2,3,3)];j=[torch.zeros(n,2,3,device=device)]
|
| 16 |
+
for k in range(1,16):
|
| 17 |
+
p=int(parents[k]);j.append(j[p]+(g[p]@off[None,:,k,:,None]).squeeze(-1));g.append(g[p]@q[:,:,k-1])
|
| 18 |
+
return torch.stack(j,2)
|
| 19 |
+
with torch.no_grad():ref=fk(rot)
|
| 20 |
+
cfg=dict(variant=variant,seed=seed,epochs=60,steps_per_epoch=48,batch=192,lr=.001,kl_beta=.0005,joint_weight=.25,position_scale_m=.1,balanced_dataset_and_sequence=True,included_datasets=included,selection='equal-dataset validation local FK mm + 0.25 relative position mm + relative rotation deg; test not used',data_sha256=sha(R/'data/paired_motion.npz'))
|
| 21 |
+
write(R/'priors'/f'{variant}_config.json',cfg);best=float('inf');hist=[];start=time.monotonic()
|
| 22 |
+
for epoch in range(1,cfg['epochs']+1):
|
| 23 |
+
model.train();sums=[]
|
| 24 |
+
for step in range(cfg['steps_per_epoch']):
|
| 25 |
+
ids=[]
|
| 26 |
+
for d in included:
|
| 27 |
+
keys=list(groups[d]);ss=rng.choice(keys,192//len(included));ids.extend(int(rng.choice(groups[d][int(s)])) for s in ss)
|
| 28 |
+
ids=torch.tensor(ids,device=device);rr,pp,mu,lv=model(rot[ids],pos[ids]);rec=(rr-rot[ids,:model.nrot]).square().sum((-1,-2)).mean();jloss=((fk(rr)-ref[ids])/.02).square().sum(-1).mean();ploss=rec*0 if model.pose_only else ((pp-pos[ids])/.1).square().sum(-1).mean();kl=.5*(mu.square()+lv.exp()-lv-1).mean();loss=rec+.25*jloss+ploss+.0005*min(1,epoch/20)*kl;opt.zero_grad();loss.backward();torch.nn.utils.clip_grad_norm_(model.parameters(),5);opt.step();sums.append(float(loss.detach()))
|
| 29 |
+
if epoch==1 or epoch%5==0:
|
| 30 |
+
model.eval();met=[]
|
| 31 |
+
with torch.no_grad():
|
| 32 |
+
for d in included:
|
| 33 |
+
ids=val[ds[val]==d];errs=[];pe=[];re=[]
|
| 34 |
+
for t in range(0,len(ids),1024):
|
| 35 |
+
ix=ids[t:t+1024];rr,pp,mu,lv=model(rot[ix],pos[ix]);errs.append((fk(rr)-ref[ix]).norm(dim=-1).mean((1,2))*1000)
|
| 36 |
+
if not model.pose_only:pe.append((pp-pos[ix]).norm(dim=-1)*1000);rel=rr[:,30]@rot[ix,30].transpose(-1,-2);re.append(torch.acos(((rel.diagonal(dim1=-2,dim2=-1).sum(-1)-1)/2).clamp(-1+1e-7,1-1e-7))*180/torch.pi)
|
| 37 |
+
met.append(dict(dataset=['ARCTIC','H2O','GRAB'][d],local_fk_mm=float(torch.cat(errs).mean()),relative_wrist_mm=float(torch.cat(pe).mean()) if pe else None,relative_rotation_deg=float(torch.cat(re).mean()) if re else None))
|
| 38 |
+
score=float(np.mean([v['local_fk_mm']+.25*(v['relative_wrist_mm'] or 0)+(v['relative_rotation_deg'] or 0) for v in met]));entry=dict(epoch=epoch,train_loss=float(np.mean(sums)),validation=met,selection_score=score,seconds=time.monotonic()-start);hist.append(entry);print(variant,json.dumps(entry),flush=True)
|
| 39 |
+
if score<best:
|
| 40 |
+
best=score;torch.save(dict(architecture=dict(latent_dim=40,width=384,pose_only=model.pose_only),state_dict=model.state_dict(),config=cfg,epoch=epoch,validation=met),R/'priors'/f'{variant}.pt')
|
| 41 |
+
write(R/'priors'/f'{variant}_history.json',hist)
|
| 42 |
+
print('COMPLETE',variant,sha(R/'priors'/f'{variant}.pt'),flush=True)
|
| 43 |
+
if __name__=='__main__':
|
| 44 |
+
p=argparse.ArgumentParser();p.add_argument('variant',choices=['joint_ego40','joint_mix40','pose_mix40']);main(p.parse_args().variant)
|
training_config.json
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"variant": "joint_mix40",
|
| 3 |
+
"seed": 20260927,
|
| 4 |
+
"epochs": 60,
|
| 5 |
+
"steps_per_epoch": 48,
|
| 6 |
+
"batch": 192,
|
| 7 |
+
"lr": 0.001,
|
| 8 |
+
"kl_beta": 0.0005,
|
| 9 |
+
"joint_weight": 0.25,
|
| 10 |
+
"position_scale_m": 0.1,
|
| 11 |
+
"balanced_dataset_and_sequence": true,
|
| 12 |
+
"included_datasets": [
|
| 13 |
+
0,
|
| 14 |
+
1,
|
| 15 |
+
2
|
| 16 |
+
],
|
| 17 |
+
"selection": "equal-dataset validation local FK mm + 0.25 relative position mm + relative rotation deg; test not used",
|
| 18 |
+
"data_sha256": "456c2b8011b56552a0806836fdb2eb9fcdd397ca3693f390bb2cbb5f8ba0f06f"
|
| 19 |
+
}
|
validation.json
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"passed": true,
|
| 3 |
+
"torch": "2.6.0+cu124",
|
| 4 |
+
"device": "cpu",
|
| 5 |
+
"original_checkpoint_equals_safetensors": true,
|
| 6 |
+
"original_and_safe_outputs_identical": true,
|
| 7 |
+
"pack_rigid_transform_invariance": true,
|
| 8 |
+
"train_only_encoding_rows": 120,
|
| 9 |
+
"heldout_encoding_rows_excluded": 12,
|
| 10 |
+
"augmentation_samples": 72,
|
| 11 |
+
"distinct_recording_neighbors": 16,
|
| 12 |
+
"latent_convex_formula_verified": true,
|
| 13 |
+
"proper_rotation_matrices": true,
|
| 14 |
+
"axis_angle_roundtrip": true,
|
| 15 |
+
"test_queries_rejected": true,
|
| 16 |
+
"original_cpu_example": {
|
| 17 |
+
"finite_input_gradients": true,
|
| 18 |
+
"frozen_vae": true
|
| 19 |
+
},
|
| 20 |
+
"training_smoke": {
|
| 21 |
+
"epochs": 1,
|
| 22 |
+
"updates": 1,
|
| 23 |
+
"batch_size": 192,
|
| 24 |
+
"validation_completed": true,
|
| 25 |
+
"checkpoint_saved_and_loaded": true,
|
| 26 |
+
"finite_parameters": true,
|
| 27 |
+
"full_retraining_performed": false
|
| 28 |
+
}
|
| 29 |
+
}
|