WhynotGPT commited on
Commit
34988af
·
verified ·
1 Parent(s): 63a625d

Release joint_mix40 VAE weights, training and GT neighborhood sampling code

Browse files
DATA_FORMAT.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Numeric NPZ schemas
2
+
3
+ All NPZ files are loaded with `allow_pickle=False`. Rotations and positions use
4
+ float32; split, recording, frame, and dataset indices are integers.
5
+
6
+ ## Training cache and query poses
7
+
8
+ | Array | Shape | Meaning |
9
+ |---|---|---|
10
+ | `pose_aa` | N × 2 × 15 × 3 | Full native MANO local finger axis angles, radians; left/right |
11
+ | `relative_rotation` | N × 3 × 3 | `R_left.T @ R_right` |
12
+ | `relative_wrist_m` | N × 3 | `R_left.T @ (wrist_right - wrist_left)`, meters |
13
+ | `dataset` | N | 0=ARCTIC, 1=H2O, 2=GRAB in the original cache |
14
+ | `sequence` | N | Recording identifier, consistent within each source; required for training/pool encoding |
15
+ | `split` | N | 0=train, 1=validation, 2=test |
16
+ | `frame_ids` | N | Optional original frame identifier |
17
+
18
+ Only real, same-frame valid hand pairs belong in these inputs. Do not construct a
19
+ pair by combining unrelated single-hand examples. Do not add a hand mean twice.
20
+ The original cache also has `anchor_rotation` and `anchor_wrist_m`; the training
21
+ and portable augmentation scripts do not need them because they model relative
22
+ coordinates. Query files use the same pose fields; `sequence` is not required.
23
+
24
+ The encoder selects only training rows. The sampler uses only train/validation
25
+ queries and finds centers and neighbors in the training pool, restricted to the
26
+ query's source dataset. For an EgoRenderer query based on GRAB motion, use the
27
+ source-motion dataset ID `2`.
28
+
29
+ ## Encoded training pool
30
+
31
+ `mu` and `logvar` are N × 40. `source_row` points back into the original training
32
+ cache. `sequence`, `dataset`, `split`, optional `frame_ids`, and the cache
33
+ `source_sha256` preserve provenance. Every pool row must have `split=0`.
34
+
35
+ ## Synthetic rendering parameters
36
+
37
+ Let Q be query count, T the number of scales, and D the directions per scale.
38
+ The output has S=Q×T×D samples ordered query, scale, direction.
39
+
40
+ | Array | Shape | Meaning |
41
+ |---|---|---|
42
+ | `z` | S × 40 | Sampled latent |
43
+ | `mano_rotation_matrices` | S × 2 × 16 × 3 × 3 | Wrist root followed by 15 native local finger rotations |
44
+ | `mano_pose_axis_angle` | S × 2 × 16 × 3 | The same full rotations as axis angles in radians |
45
+ | `wrist_positions_m` | S × 2 × 3 | Actual joint-zero positions in meters |
46
+ | `mano_shape` | S × 2 × 10 | Neutral shape, all zeros |
47
+ | `query_index` | S | Query row index |
48
+ | `train_pool_row`, `train_source_row` | S | Center in encoded pool / original cache |
49
+ | `train_sequence`, `train_dataset` | S | Center recording and dataset IDs |
50
+ | `temperature` | S | Interpolation fraction tau, despite the historical field name |
51
+ | `draw` | S | Direction index |
52
+ | `anchor_pool_rows` | Q | Retrieved training centers |
53
+ | `neighbor_pool_rows` | Q × K | Neighbor rows from distinct other training recordings |
54
+ | `endpoint_neighbor_indices` | Q × D × 2 | Pair indices into that query's neighbor list |
55
+ | `endpoint_weights` | Q × D | Convex endpoint weight w |
56
+
57
+ The canonical left wrist is at zero with identity rotation. Right-wrist placement
58
+ comes from the decoded relative pose. Apply the same chosen scene transform to
59
+ both wrists and both root orientations. Local finger rotations remain local.
60
+
61
+ To decode meshes, use MANO with full poses (no additional pose-mean addition),
62
+ then translate each hand so its actual wrist joint matches `wrist_positions_m`.
63
+ Do not pass actual wrist positions directly as MANO `transl` unless your MANO
64
+ implementation explicitly uses a wrist-centered template. Mesh generation is a
65
+ separate operation and is not required to encode/sample the VAE.
README.md ADDED
@@ -0,0 +1,158 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: pytorch
3
+ language:
4
+ - en
5
+ - zh
6
+ tags:
7
+ - hand-pose
8
+ - bimanual
9
+ - vae
10
+ - mano
11
+ - pose-generation
12
+ - data-augmentation
13
+ ---
14
+
15
+ # Bimanual Pose VAE — joint_mix40
16
+
17
+ 双手姿态 VAE 的代码与权重:学习双手局部姿态、相对腕朝向和相对腕位置,
18
+ 可用于姿态先验,以及从训练 GT 的 latent 邻域生成渲染增广候选。
19
+
20
+ This is the **exact joint_mix40 checkpoint** used by the
21
+ [VAE Pose Atlas visualization](https://huggingface.co/spaces/WhynotGPT/vae-pose-atlas).
22
+ The framewise model encodes 30 local finger rotations, one relative wrist rotation,
23
+ and a relative wrist position into a **40-dimensional latent**. It uses a width-384
24
+ MLP and a 189-dimensional input with row-based 6D rotation representations.
25
+ Shared global rotation and translation are removed by expressing the wrist relation
26
+ in the left wrist frame. RGB, camera parameters, hand shape, and temporal dynamics
27
+ are outside the model.
28
+
29
+ ## Download and run
30
+
31
+ Python 3.11 is the tested environment. Download the repository, then run from its root:
32
+
33
+ ```bash
34
+ pip install huggingface_hub
35
+ python -c "from huggingface_hub import snapshot_download; snapshot_download('WhynotGPT/bimanual-pose-vae', local_dir='bimanual-pose-vae')"
36
+ cd bimanual-pose-vae
37
+ pip install -r requirements.txt
38
+ python example.py --device cpu
39
+ sha256sum -c SHA256SUMS
40
+ ```
41
+
42
+ The example uses synthetic inputs to verify loading, reconstruction, and input
43
+ gradients through the frozen VAE. It does not measure accuracy.
44
+
45
+ ```python
46
+ import torch
47
+ from load_model import load_model
48
+ from bimanual_vae import pack
49
+
50
+ vae, metadata = load_model(device="cpu") # model.safetensors + config.json
51
+ # local_rot: [B,2,15,3,3]; root_rot: [B,2,3,3]; wrist: [B,2,3]
52
+ with torch.no_grad():
53
+ rot, pos = pack(local_rot, root_rot, wrist)
54
+ mu, logvar = vae.encode(rot, pos)
55
+ reconstructed_rot, reconstructed_pos = vae.decode(mu)
56
+ ```
57
+
58
+ Inputs use **left, right** ordering and each hand's native MANO convention. Local
59
+ rotations represent the full pose, including the native hand mean where required
60
+ by the source convention. Wrist positions are actual joint positions in meters,
61
+ not necessarily the MANO `transl` parameter. Both hands must be valid.
62
+
63
+ Decoder rotations have shape `[B,31,3,3]`: left fingers `0:15`, right fingers
64
+ `15:30`, and relative right-wrist orientation `30`. Position `[B,3]` is the right
65
+ wrist in the left wrist frame. To place a pose in a scene, choose a left-wrist
66
+ anchor `(R_L,t_L)`, then set `R_R=R_L @ rot[:,30]` and `t_R=t_L+R_L @ pos`.
67
+
68
+ The original API is also available:
69
+
70
+ ```python
71
+ from bimanual_vae import load_pair
72
+ vae, metadata = load_pair("joint_mix40.pt", device="cpu")
73
+ # Preserve gradients through the inputs when using a frozen prior:
74
+ loss_prior = vae.energy(local_rot, root_rot, wrist).mean()
75
+ ```
76
+
77
+ ## Generate augmentation poses from training GT
78
+
79
+ The portable sampler follows the train-GT neighborhood method used in the Atlas.
80
+ It encodes only `split=0` rows, retrieves the nearest training pose in the same
81
+ dataset as each query, and finds neighbors from 16 distinct other training recordings.
82
+ Queries can be difficult training/validation GT poses; test queries are rejected.
83
+ The caller selects difficult queries, for example using a model's validation PA error.
84
+
85
+ ```bash
86
+ python augment.py encode --data /path/to/paired_motion.npz --output train_latents.npz
87
+ python augment.py sample --pool train_latents.npz --queries /path/to/difficult_gt.npz --output render_poses.npz
88
+ ```
89
+
90
+ For each retrieved center, the sampler draws eight neighbor pairs with weights
91
+ `w` in `[0.2,0.8]` and reuses those endpoints at `tau = 0.25, 0.5, 1.0`:
92
+
93
+ ```text
94
+ endpoint = w * mu_neighbor_a + (1-w) * mu_neighbor_b
95
+ z = (1-tau) * mu_train_center + tau * endpoint
96
+ ```
97
+
98
+ Each query yields **24 synthetic bimanual poses**. Output includes full MANO axis
99
+ angles and rotation matrices, actual wrist positions, zero shape coefficients,
100
+ latent values, and training-source indices. The left wrist is at the origin with
101
+ identity orientation. See [DATA_FORMAT.md](DATA_FORMAT.md) for all array conventions.
102
+
103
+ These outputs are pose labels for subsequent rendering. Geometry, scene placement,
104
+ hand-object contact, and collision checks belong to the rendering pipeline. The
105
+ model alone does not generate mesh vertices or guarantee physical plausibility.
106
+ Augmentation gains on a downstream hand estimator have not been established by
107
+ this release. The portable script implements the same sampling rule; it does not
108
+ recreate the Atlas's source-specific query extraction or its exact random draw order.
109
+
110
+ ## Training
111
+
112
+ The released checkpoint was selected at **epoch 50** of a 60-epoch run using
113
+ validation reconstruction metrics. Training uses 48 updates per epoch, batch 192,
114
+ AdamW at 0.001, dataset/recording-balanced random sampling, rotation and skeletal
115
+ FK reconstruction, relative-position reconstruction, and KL regularization.
116
+
117
+ | Source | Train paired frames | Validation paired frames | Test paired frames |
118
+ |---|---:|---:|---:|
119
+ | ARCTIC | 32,320 | 4,960 | 4,480 |
120
+ | H2O | 12,686 | 2,560 | 1,760 |
121
+ | GRAB | 104,825 | 10,460 | 11,834 |
122
+
123
+ ARCTIC/H2O use frozen recording splits with the experiment's benchmark recordings
124
+ excluded. GRAB uses subjects s1–s8 / s9 / s10. Preparation retains at most 160
125
+ paired frames per recording; GRAB is first downsampled from 120 fps to 10 fps.
126
+ These are VAE training counts, separate from the main estimator's SFT clip counts.
127
+
128
+ ```bash
129
+ pip install -r requirements-training.txt
130
+ python train.py --data /path/to/paired_motion.npz \
131
+ --mano-dir /path/to/mano --output runs/joint_mix40 --device cuda
132
+ ```
133
+
134
+ `train.py` preserves the original training algorithm, adding configurable paths,
135
+ device, and run lengths. It needs your prepared motion cache and MANO_LEFT.pkl /
136
+ MANO_RIGHT.pkl for the native skeleton FK loss. Raw data, the motion cache, MANO
137
+ model assets, and their derived mesh buffers are not distributed here. Obtain
138
+ those dependencies under their respective terms. The original experiment scripts
139
+ are archived in [training/](training/README.md); their historical absolute paths
140
+ and external preprocessing dependencies are documented there.
141
+
142
+ The repository includes only the **joint_mix40 trained weights**. Training code
143
+ also exposes the historical joint_ego40 and pose_mix40 variants.
144
+
145
+ ## Files and verification
146
+
147
+ - `joint_mix40.pt`: byte-identical original checkpoint, with normalization and metadata.
148
+ - `model.safetensors`, `config.json`: equivalent tensor-only weights and JSON metadata.
149
+ - `bimanual_vae.py`, `hand_vae.py`: unchanged original model definitions and rotation helpers.
150
+ - `load_model.py`, `example.py`: loading and differentiable-prior examples.
151
+ - `train.py`, `training_config.json`, `training/`: training entry point and provenance.
152
+ - `augment.py`, `DATA_FORMAT.md`: train-GT encoding and neighborhood sampling.
153
+ - `validation.json`, `provenance.json`, `SHA256SUMS`: checks and file integrity records.
154
+
155
+ Original checkpoint SHA-256:
156
+ `e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096`.
157
+ The original model and training work is from the MiroHand P3R4 bimanual VAE
158
+ experiment dated 2026-09-27; this package adds portable interfaces for release.
SHA256SUMS ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ c3f7f50702d94ae4e2e631818a595723fc417e17f1388f63e228ce725853afb2 DATA_FORMAT.md
2
+ 9bb029884f38a9b5d3e80197a03f158eae05e4adfc0273c23dede3ba90901720 README.md
3
+ ea941357aa03afdce5b6608dac2965d44bf6347d9119efb5f832999ebbe89744 augment.py
4
+ 9e940527007fc2bf93809eba891bb18b76b75a9e30206a2ab0698704e431f3f6 bimanual_vae.py
5
+ 0e486bd7ef71d03f55b2f4de0d4bdb314e198d37380cdb366240789534ce2ed3 config.json
6
+ d81b092510f478f0e0c405553ac8238144239926087f3650ee4c07e48906999d example.py
7
+ 5ee21d9a48ce2d8094f28c2ff1ae56a57d5c55244125e18b3c3e50d1a588979c hand_vae.py
8
+ e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096 joint_mix40.pt
9
+ 11e94dbc58096fb2ce590af9749f4ab431cf1c27799f6fdeb2341f61453b0338 load_model.py
10
+ c4a0ccd48ebba2e8f03fc295be942e4520385f99521460de4c9915c1776e9da3 mano_compat.py
11
+ 6c4fe1b79138e4cd3b7ddccb45780479de2d25270be018e46be6696d78bd05f8 model.safetensors
12
+ f362e83ba1b3798744b94f3d5b79edbb7b0dfdd8c4d0cef328f361abe8cf17a3 provenance.json
13
+ cd3b0fc323d507a9935a38187754f71a8b973f9d3ef32be9bfd29b3f0a521ebb requirements-training.txt
14
+ efba809b61f64bbf0863e65c9c303800b8c2c6a28100c1739e2407964b24e276 requirements.txt
15
+ 5b7db63f51871d303c1abad15441c002693df8cdc68cab944905ab8dc146f3db train.py
16
+ 89ee9e8f6152ad58d42003119d5c6643d9242e511a35a8a6711a860d57fd082f training/README.md
17
+ a83a1341b55522d329adbaab29937ba5a260f42091975098aacb116490ff9417 training/history.json
18
+ f08a807e45d72cc2fe6de81f65b22db78075d16ac4dcfa390b32572ccc3e185e training/original_common.py
19
+ 1c2bb19eeaf58e2f80b76e6ec556af02cb92cbb2ebfeaf580b7ed00fc86bac4d training/original_prepare_data.py
20
+ 8c854506d4a084fcea9ce8988c7dc33fe628f22b8380832f7f45a54ffc8a716a training/original_train_pair.py
21
+ e4163ef9cc4f8b7add9d4e4350a853a145c842e51bab851101dcabb3c550c4ff training_config.json
22
+ 2e072c6387f5c7765b8a45b66a3217f4b6c11cbc31c0426ed47fece4c3ecf025 validation.json
augment.py ADDED
@@ -0,0 +1,150 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Encode training GT and sample distinct-recording latent neighborhoods.
2
+
3
+ Inputs are numeric NPZ arrays, described in DATA_FORMAT.md. Mesh generation is
4
+ external; the output contains full MANO rotations and actual wrist positions.
5
+ """
6
+ import argparse
7
+ import hashlib
8
+ from pathlib import Path
9
+
10
+ import numpy as np
11
+ import torch
12
+
13
+ from hand_vae import aa_to_matrix, matrix_to_aa
14
+ from load_model import load_model
15
+
16
+
17
+ def encode_pose(model, pose, relative_rotation, relative_wrist):
18
+ local = aa_to_matrix(torch.as_tensor(pose, dtype=torch.float32)).reshape(-1, 30, 3, 3)
19
+ rotations = torch.cat([local, torch.as_tensor(relative_rotation, dtype=torch.float32)[:, None]], 1)
20
+ return model.encode(rotations, torch.as_tensor(relative_wrist, dtype=torch.float32))
21
+
22
+
23
+ @torch.inference_mode()
24
+ def encode(data_path, output):
25
+ model, _ = load_model()
26
+ with np.load(data_path, allow_pickle=False) as data:
27
+ rows = np.flatnonzero(data["split"] == 0)
28
+ if not len(rows):
29
+ raise ValueError("No split=0 training rows.")
30
+ payload = {name: data[name][rows] for name in ("sequence", "dataset")}
31
+ if "frame_ids" in data:
32
+ payload["frame_ids"] = data["frame_ids"][rows]
33
+ poses = {name: data[name][rows] for name in ("pose_aa", "relative_rotation", "relative_wrist_m")}
34
+ means, logvars = [], []
35
+ for start in range(0, len(rows), 2048):
36
+ ix = slice(start, start + 2048)
37
+ mu, logvar = encode_pose(model, poses["pose_aa"][ix], poses["relative_rotation"][ix], poses["relative_wrist_m"][ix])
38
+ means.append(mu.numpy())
39
+ logvars.append(logvar.numpy())
40
+ with data_path.open("rb") as stream:
41
+ source_hash = hashlib.file_digest(stream, "sha256").hexdigest()
42
+ np.savez_compressed(output, mu=np.concatenate(means), logvar=np.concatenate(logvars),
43
+ source_row=rows, split=np.zeros(len(rows), dtype=np.int8),
44
+ source_sha256=np.array(source_hash), **payload)
45
+ print(f"Encoded {len(rows)} training GT rows into {output}")
46
+
47
+
48
+ @torch.inference_mode()
49
+ def sample(pool_path, query_path, output, neighbor_count, directions, scales, seed):
50
+ model, _ = load_model()
51
+ rng = np.random.default_rng(seed)
52
+ with np.load(pool_path, allow_pickle=False) as data:
53
+ pool = {key: data[key] for key in data.files}
54
+ if not np.all(pool["split"] == 0):
55
+ raise ValueError("The candidate pool must contain only training GT.")
56
+ with np.load(query_path, allow_pickle=False) as queries:
57
+ # Queries identify difficult GT regions; held-out test queries are excluded.
58
+ if not np.isin(queries["split"], [0, 1]).all():
59
+ raise ValueError("Queries must use train/validation GT, never test rows.")
60
+ query_datasets = queries["dataset"].copy()
61
+ query_mu, _ = encode_pose(model, queries["pose_aa"], queries["relative_rotation"], queries["relative_wrist_m"])
62
+ query_mu = query_mu.numpy()
63
+ if not len(query_mu):
64
+ raise ValueError("The query file is empty.")
65
+ if not np.isfinite(pool["mu"]).all() or not np.isfinite(query_mu).all():
66
+ raise ValueError("Latents must be finite.")
67
+ if neighbor_count < 1 or directions < 1 or not len(scales) or np.any((scales < 0) | (scales > 1)) or not np.isfinite(scales).all():
68
+ raise ValueError("Use positive counts and finite interpolation scales in [0,1].")
69
+ anchors, neighbors, pairs, weights, latent_batches = [], [], [], [], []
70
+ for query, dataset in zip(query_mu, query_datasets):
71
+ ids = np.flatnonzero(pool["dataset"] == dataset)
72
+ if not len(ids):
73
+ raise ValueError(f"No training rows for dataset {dataset}.")
74
+ anchor = int(ids[np.argmin(np.square(pool["mu"][ids] - query).sum(1))])
75
+ order = np.argsort(np.square(pool["mu"][ids] - pool["mu"][anchor]).sum(1))
76
+ seen = {int(pool["sequence"][anchor])}
77
+ nearest = []
78
+ for row in ids[order]:
79
+ sequence = int(pool["sequence"][row])
80
+ if sequence not in seen:
81
+ nearest.append(int(row))
82
+ seen.add(sequence)
83
+ if len(nearest) == neighbor_count:
84
+ break
85
+ if len(nearest) != neighbor_count:
86
+ raise ValueError(f"Dataset {dataset} needs {neighbor_count + 1} distinct training recordings.")
87
+ nearest = np.asarray(nearest)
88
+ pair = rng.choice(neighbor_count, (directions, 2), replace=True)
89
+ weight = rng.uniform(.2, .8, directions).astype(np.float32)
90
+ endpoint = weight[:, None] * pool["mu"][nearest[pair[:, 0]]] + (1 - weight[:, None]) * pool["mu"][nearest[pair[:, 1]]]
91
+ z = (1 - scales[:, None, None]) * pool["mu"][anchor] + scales[:, None, None] * endpoint[None]
92
+ latent_batches.append(z.reshape(-1, model.latent_dim))
93
+ anchors.append(anchor)
94
+ neighbors.append(nearest)
95
+ pairs.append(pair)
96
+ weights.append(weight)
97
+ z = np.concatenate(latent_batches).astype(np.float32)
98
+ rot, pos = model.decode(torch.from_numpy(z))
99
+ full = torch.zeros(len(z), 2, 16, 3, 3)
100
+ full[:, 0, 0] = torch.eye(3)
101
+ full[:, 1, 0] = rot[:, 30]
102
+ full[:, :, 1:] = rot[:, :30].reshape(-1, 2, 15, 3, 3)
103
+ wrist = torch.zeros(len(z), 2, 3)
104
+ wrist[:, 1] = pos
105
+ anchors = np.asarray(anchors)
106
+ query_index = np.repeat(np.arange(len(anchors)), len(scales) * directions)
107
+ np.savez_compressed(
108
+ output, z=z, mano_rotation_matrices=full.numpy(), mano_pose_axis_angle=matrix_to_aa(full).numpy(),
109
+ wrist_positions_m=wrist.numpy(), mano_shape=np.zeros((len(z), 2, 10), np.float32),
110
+ query_index=query_index, train_pool_row=anchors[query_index],
111
+ train_source_row=pool["source_row"][anchors[query_index]],
112
+ train_sequence=pool["sequence"][anchors[query_index]],
113
+ train_dataset=pool["dataset"][anchors[query_index]],
114
+ temperature=np.tile(np.repeat(scales, directions), len(anchors)),
115
+ draw=np.tile(np.arange(directions), len(anchors) * len(scales)),
116
+ anchor_pool_rows=anchors, neighbor_pool_rows=np.asarray(neighbors),
117
+ endpoint_neighbor_indices=np.asarray(pairs), endpoint_weights=np.asarray(weights),
118
+ seed=np.array(seed), coordinate_system=np.array("left_wrist_local"),
119
+ )
120
+ print(f"Saved {len(z)} synthetic bimanual pose candidates to {output}")
121
+
122
+
123
+ def main():
124
+ parser = argparse.ArgumentParser(description=__doc__)
125
+ commands = parser.add_subparsers(dest="command", required=True)
126
+ enc = commands.add_parser("encode")
127
+ enc.add_argument("--data", type=Path, required=True)
128
+ enc.add_argument("--output", type=Path, required=True)
129
+ sam = commands.add_parser("sample")
130
+ sam.add_argument("--pool", type=Path, required=True)
131
+ sam.add_argument("--queries", type=Path, required=True)
132
+ sam.add_argument("--output", type=Path, required=True)
133
+ sam.add_argument("--neighbors", type=int, default=16)
134
+ sam.add_argument("--directions", type=int, default=8)
135
+ sam.add_argument("--scales", type=float, nargs="+", default=[.25, .5, 1.])
136
+ sam.add_argument("--seed", type=int, default=20261110)
137
+ args = parser.parse_args()
138
+ if args.output.exists():
139
+ parser.error("Output already exists; choose a new path.")
140
+ args.output.parent.mkdir(parents=True, exist_ok=True)
141
+ torch.set_num_threads(4)
142
+ if args.command == "encode":
143
+ encode(args.data, args.output)
144
+ else:
145
+ sample(args.pool, args.queries, args.output, args.neighbors, args.directions,
146
+ np.asarray(args.scales, dtype=np.float32), args.seed)
147
+
148
+
149
+ if __name__ == "__main__":
150
+ main()
bimanual_vae.py ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """P3R4 paired-hand prior. Translation-invariant, common-rotation-invariant.
2
+ Local finger rotations (2 x 15), right wrist orientation and translation in
3
+ left wrist frame. Inference uses camera-coordinate inputs; mocap needs no camera.
4
+ """
5
+ import torch
6
+ from torch import nn
7
+ from hand_vae import matrix_to_six,six_to_matrix,aa_to_matrix
8
+ class BimanualVAE(nn.Module):
9
+ def __init__(self,latent_dim=40,width=384,pose_only=False):
10
+ super().__init__();self.latent_dim=latent_dim;self.width=width;self.pose_only=pose_only;self.nrot=30 if pose_only else 31;self.dim=self.nrot*6+(0 if pose_only else 3)
11
+ self.register_buffer('input_mean',torch.zeros(self.dim));self.register_buffer('input_std',torch.ones(self.dim))
12
+ self.encoder=nn.Sequential(nn.Linear(self.dim,width),nn.LeakyReLU(.2),nn.Linear(width,width),nn.LeakyReLU(.2));self.mu=nn.Linear(width,latent_dim);self.logvar=nn.Linear(width,latent_dim);nn.init.constant_(self.logvar.bias,-2)
13
+ self.decoder=nn.Sequential(nn.Linear(latent_dim,width),nn.LeakyReLU(.2),nn.Linear(width,width),nn.LeakyReLU(.2),nn.Linear(width,self.dim))
14
+ def features(self,rot,pos):
15
+ x=matrix_to_six(rot[...,:self.nrot,:,:]).flatten(-2)
16
+ return x if self.pose_only else torch.cat([x,pos],-1)
17
+ def encode(self,rot,pos):
18
+ h=self.encoder((self.features(rot,pos)-self.input_mean)/self.input_std);return self.mu(h),self.logvar(h).clamp(-8,4)
19
+ def decode(self,z):
20
+ x=self.decoder(z)*self.input_std+self.input_mean;rot=six_to_matrix(x[...,:self.nrot*6].reshape(*x.shape[:-1],self.nrot,6));pos=torch.zeros((*x.shape[:-1],3),device=x.device,dtype=x.dtype) if self.pose_only else x[...,-3:];return rot,pos
21
+ def forward(self,rot,pos):
22
+ mu,lv=self.encode(rot,pos);z=mu+torch.randn_like(mu)*(.5*lv).exp() if self.training else mu;r,p=self.decode(z);return r,p,mu,lv
23
+ def energy(self,local_rot,root_rot,wrist):
24
+ rot,pos=pack(local_rot,root_rot,wrist);rr,pp,mu,lv=self(rot,pos)
25
+ pose=(rr[:,:30]-rot[:,:30]).square().sum((-1,-2)).mean(-1)
26
+ relation=pose*0 if self.pose_only else (rr[:,30]-rot[:,30]).square().sum((-1,-2))+((pp-pos)/.1).square().sum(-1)
27
+ return pose+relation+.0005*mu.square().mean(-1)
28
+ def pack(local_rot,root_rot,wrist):
29
+ left=root_rot[...,0,:,:];rr=left.transpose(-1,-2)@root_rot[...,1,:,:];pos=(left.transpose(-1,-2)@(wrist[...,1,:]-wrist[...,0,:])[...,None]).squeeze(-1)
30
+ return torch.cat([local_rot.flatten(-4,-3),rr.unsqueeze(-3)],-3),pos
31
+ def load_pair(path,device='cuda'):
32
+ c=torch.load(path,map_location=device,weights_only=False);m=BimanualVAE(**c['architecture']).to(device);m.load_state_dict(c['state_dict'],strict=True);m.eval();m.requires_grad_(False);return m,c
config.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architecture": {
3
+ "latent_dim": 40,
4
+ "width": 384,
5
+ "pose_only": false
6
+ },
7
+ "config": {
8
+ "variant": "joint_mix40",
9
+ "seed": 20260927,
10
+ "epochs": 60,
11
+ "steps_per_epoch": 48,
12
+ "batch": 192,
13
+ "lr": 0.001,
14
+ "kl_beta": 0.0005,
15
+ "joint_weight": 0.25,
16
+ "position_scale_m": 0.1,
17
+ "balanced_dataset_and_sequence": true,
18
+ "included_datasets": [
19
+ 0,
20
+ 1,
21
+ 2
22
+ ],
23
+ "selection": "equal-dataset validation local FK mm + 0.25 relative position mm + relative rotation deg; test not used",
24
+ "data_sha256": "456c2b8011b56552a0806836fdb2eb9fcdd397ca3693f390bb2cbb5f8ba0f06f"
25
+ },
26
+ "epoch": 50,
27
+ "validation": [
28
+ {
29
+ "dataset": "ARCTIC",
30
+ "local_fk_mm": 2.552030563354492,
31
+ "relative_wrist_mm": 15.961709022521973,
32
+ "relative_rotation_deg": 13.042522430419922
33
+ },
34
+ {
35
+ "dataset": "H2O",
36
+ "local_fk_mm": 1.722329020500183,
37
+ "relative_wrist_mm": 7.883533000946045,
38
+ "relative_rotation_deg": 8.399897575378418
39
+ },
40
+ {
41
+ "dataset": "GRAB",
42
+ "local_fk_mm": 2.7820262908935547,
43
+ "relative_wrist_mm": 18.310850143432617,
44
+ "relative_rotation_deg": 17.670961380004883
45
+ }
46
+ ]
47
+ }
example.py ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """CPU/GPU smoke example. Synthetic inputs only; not an accuracy evaluation."""
2
+ import argparse
3
+ import json
4
+ from pathlib import Path
5
+ import torch
6
+ from bimanual_vae import load_pair, pack
7
+ from hand_vae import aa_to_matrix
8
+
9
+
10
+ def main():
11
+ parser = argparse.ArgumentParser()
12
+ parser.add_argument("--device", default="cpu")
13
+ args = parser.parse_args()
14
+ torch.manual_seed(0)
15
+ vae, _ = load_pair(str(Path(__file__).resolve().parent / "joint_mix40.pt"), args.device)
16
+ # Synthetic full MANO local axis angles, left hand first, right hand second.
17
+ local_aa = torch.zeros(2, 2, 15, 3, device=args.device, requires_grad=True)
18
+ root_aa = torch.zeros(2, 2, 3, device=args.device, requires_grad=True)
19
+ wrist = torch.tensor([[[0., 0., .5], [.2, 0., .5]],
20
+ [[0., 0., .6], [.2, .02, .6]]],
21
+ device=args.device, requires_grad=True)
22
+ local_rot, root_rot = aa_to_matrix(local_aa), aa_to_matrix(root_aa)
23
+ energy = vae.energy(local_rot, root_rot, wrist)
24
+ loss = 0.1 * energy.mean()
25
+ loss.backward()
26
+ for x in (local_aa, root_aa, wrist):
27
+ assert x.grad is not None and torch.isfinite(x.grad).all()
28
+ assert torch.isfinite(energy).all()
29
+ assert all(not p.requires_grad and p.grad is None for p in vae.parameters())
30
+ with torch.no_grad():
31
+ rot, pos = pack(local_rot, root_rot, wrist)
32
+ rot_rec, pos_rec, mu, logvar = vae(rot, pos)
33
+ assert rot_rec.shape == (2, 31, 3, 3)
34
+ assert pos_rec.shape == (2, 3)
35
+ assert mu.shape == logvar.shape == (2, 40)
36
+ print(json.dumps({"device": args.device, "torch": torch.__version__,
37
+ "synthetic_inputs": True, "energy": energy.detach().cpu().tolist(),
38
+ "rotation_shape": list(rot_rec.shape),
39
+ "position_shape": list(pos_rec.shape),
40
+ "latent_shape": list(mu.shape),
41
+ "finite_input_gradients": True, "frozen_vae": True}, indent=2))
42
+
43
+
44
+ if __name__ == "__main__":
45
+ main()
hand_vae.py ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Hand-specific, side-conditioned VAE; 15 local rotations, no wrist/global/shape."""
2
+ import torch
3
+ from torch import nn
4
+ from torch.nn import functional as F
5
+
6
+ def aa_to_matrix(x):
7
+ theta=x.norm(dim=-1,keepdim=True);x0,x1,x2=x.unbind(-1);z=torch.zeros_like(x0)
8
+ k=torch.stack([z,-x2,x1,x2,z,-x0,-x1,x0,z],-1).reshape(*x.shape[:-1],3,3)
9
+ a=torch.sinc(theta/torch.pi)[...,None];b=(.5*torch.sinc(theta/(2*torch.pi)).square())[...,None]
10
+ return torch.eye(3,device=x.device,dtype=x.dtype)+a*k+b*(k@k)
11
+
12
+ def matrix_to_aa(m):
13
+ m00,m01,m02=m[...,0,0],m[...,0,1],m[...,0,2];m10,m11,m12=m[...,1,0],m[...,1,1],m[...,1,2];m20,m21,m22=m[...,2,0],m[...,2,1],m[...,2,2]
14
+ qabs=torch.stack([1+m00+m11+m22,1+m00-m11-m22,1-m00+m11-m22,1-m00-m11+m22],-1).clamp_min(1e-8).sqrt()
15
+ candidates=torch.stack([torch.stack([qabs[...,0]**2,m21-m12,m02-m20,m10-m01],-1),torch.stack([m21-m12,qabs[...,1]**2,m10+m01,m02+m20],-1),torch.stack([m02-m20,m10+m01,qabs[...,2]**2,m12+m21],-1),torch.stack([m10-m01,m20+m02,m21+m12,qabs[...,3]**2],-1)],-2)/(2*qabs[...,None].clamp_min(.1))
16
+ ix=qabs.argmax(-1);q=candidates.gather(-2,ix[...,None,None].expand(*ix.shape,1,4)).squeeze(-2);q=F.normalize(q,dim=-1);q=torch.where(q[...,:1]<0,-q,q)
17
+ norm=q[...,1:].norm(dim=-1,keepdim=True);angle=2*torch.atan2(norm,q[...,:1]);scale=torch.where(norm>1e-6,angle/norm.clamp_min(1e-8),2+norm.square()/3)
18
+ return q[...,1:]*scale
19
+
20
+ def matrix_to_six(m):return m[...,:2,:].reshape(*m.shape[:-2],6)
21
+ def six_to_matrix(x):
22
+ a,b=x[...,:3],x[...,3:];r0=F.normalize(a,dim=-1,eps=1e-7);r1=F.normalize(b-(r0*b).sum(-1,keepdim=True)*r0,dim=-1,eps=1e-7);r2=torch.cross(r0,r1,dim=-1);return torch.stack([r0,r1,r2],-2)
23
+ class HandVAE(nn.Module):
24
+ def __init__(self,latent_dim=20,width=256):
25
+ super().__init__();self.latent_dim=latent_dim;self.width=width
26
+ self.register_buffer('input_mean',torch.zeros(90));self.register_buffer('input_std',torch.ones(90))
27
+ self.encoder=nn.Sequential(nn.Linear(92,width),nn.LeakyReLU(.2),nn.Linear(width,width),nn.LeakyReLU(.2))
28
+ self.mu=nn.Linear(width,latent_dim);self.logvar=nn.Linear(width,latent_dim);nn.init.constant_(self.logvar.bias,-2)
29
+ self.decoder=nn.Sequential(nn.Linear(latent_dim+2,width),nn.LeakyReLU(.2),nn.Linear(width,width),nn.LeakyReLU(.2),nn.Linear(width,90))
30
+ def encode(self,rot,side):
31
+ x=(matrix_to_six(rot).flatten(-2)-self.input_mean)/self.input_std;h=self.encoder(torch.cat([x,F.one_hot(side.long(),2).to(x)],-1));return self.mu(h),self.logvar(h).clamp(-8,4)
32
+ def decode(self,z,side):
33
+ x=self.decoder(torch.cat([z,F.one_hot(side.long(),2).to(z)],-1))*self.input_std+self.input_mean;return six_to_matrix(x.reshape(*z.shape[:-1],15,6))
34
+ def forward(self,rot,side):
35
+ mu,lv=self.encode(rot,side);z=mu+torch.randn_like(mu)*(.5*lv).exp() if self.training else mu;return self.decode(z,side),mu,lv
36
+
37
+ def load_vae(path,device='cuda'):
38
+ ck=torch.load(path,map_location=device,weights_only=False);m=HandVAE(**ck['architecture']).to(device);m.load_state_dict(ck['state_dict'],strict=True);m.eval();m.requires_grad_(False);return m,ck
joint_mix40.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096
3
+ size 1959670
load_model.py ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Load the released tensor-only weights and checkpoint metadata."""
2
+ import json
3
+ from pathlib import Path
4
+
5
+ from safetensors.torch import load_file
6
+
7
+ from bimanual_vae import BimanualVAE
8
+
9
+
10
+ def load_model(directory=None, device="cpu"):
11
+ directory = Path(directory) if directory else Path(__file__).resolve().parent
12
+ config = json.loads((directory / "config.json").read_text())
13
+ model = BimanualVAE(**config["architecture"])
14
+ model.load_state_dict(load_file(str(directory / "model.safetensors")), strict=True)
15
+ model.to(device).eval().requires_grad_(False)
16
+ return model, config
mano_compat.py ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Compatibility shim for the licensed MANO files and legacy chumpy loader."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import inspect
6
+
7
+ import numpy as np
8
+
9
+
10
+ def enable_legacy_mano_loading() -> None:
11
+ """Restore names removed after the legacy MANO/chumpy release."""
12
+
13
+ if not hasattr(inspect, "getargspec"):
14
+ inspect.getargspec = inspect.getfullargspec # type: ignore[attr-defined]
15
+ aliases = {
16
+ "bool": bool,
17
+ "int": int,
18
+ "float": float,
19
+ "complex": complex,
20
+ "object": object,
21
+ "unicode": str,
22
+ "str": str,
23
+ }
24
+ for name, value in aliases.items():
25
+ if name not in np.__dict__:
26
+ setattr(np, name, value)
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c4fe1b79138e4cd3b7ddccb45780479de2d25270be018e46be6696d78bd05f8
3
+ size 1954636
provenance.json ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "source_experiment": "MiroHand_P3R4_bimanual_VAE_20260927",
3
+ "release_date": "2026-11-15",
4
+ "unchanged_original_files": {
5
+ "joint_mix40.pt": {
6
+ "source_relative_path": "priors/joint_mix40.pt",
7
+ "sha256": "e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096"
8
+ },
9
+ "bimanual_vae.py": {
10
+ "source_relative_path": "code/bimanual_vae.py",
11
+ "sha256": "9e940527007fc2bf93809eba891bb18b76b75a9e30206a2ab0698704e431f3f6"
12
+ },
13
+ "hand_vae.py": {
14
+ "source_relative_path": "code/hand_vae.py",
15
+ "sha256": "5ee21d9a48ce2d8094f28c2ff1ae56a57d5c55244125e18b3c3e50d1a588979c"
16
+ },
17
+ "training_config.json": {
18
+ "source_relative_path": "priors/joint_mix40_config.json",
19
+ "sha256": "e4163ef9cc4f8b7add9d4e4350a853a145c842e51bab851101dcabb3c550c4ff"
20
+ },
21
+ "training/history.json": {
22
+ "source_relative_path": "priors/joint_mix40_history.json",
23
+ "sha256": "a83a1341b55522d329adbaab29937ba5a260f42091975098aacb116490ff9417"
24
+ },
25
+ "training/original_train_pair.py": {
26
+ "source_relative_path": "code/train_pair.py",
27
+ "sha256": "8c854506d4a084fcea9ce8988c7dc33fe628f22b8380832f7f45a54ffc8a716a"
28
+ },
29
+ "training/original_prepare_data.py": {
30
+ "source_relative_path": "code/prepare_data.py",
31
+ "sha256": "1c2bb19eeaf58e2f80b76e6ec556af02cb92cbb2ebfeaf580b7ed00fc86bac4d"
32
+ },
33
+ "training/original_common.py": {
34
+ "source_relative_path": "code/common.py",
35
+ "sha256": "f08a807e45d72cc2fe6de81f65b22db78075d16ac4dcfa390b32572ccc3e185e"
36
+ }
37
+ },
38
+ "converted_weights": {
39
+ "model.safetensors": "All checkpoint state_dict tensors, exactly equal; metadata in config.json"
40
+ },
41
+ "adapted_files": {
42
+ "train.py": "Original algorithm with CLI paths/device/run lengths and standalone compatibility helpers",
43
+ "augment.py": "Portable training-GT neighborhood sampling; query extraction and mesh generation external"
44
+ },
45
+ "trained_variant": "joint_mix40",
46
+ "checkpoint_epoch": 50,
47
+ "excluded_assets": [
48
+ "raw datasets",
49
+ "training motion cache",
50
+ "MANO model files",
51
+ "derived MANO buffers"
52
+ ]
53
+ }
requirements-training.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ -r requirements.txt
2
+ smplx==0.1.28
3
+ chumpy==0.70
4
+ scipy>=1.10
requirements.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ torch>=2.0
2
+ numpy>=1.24
3
+ safetensors>=0.4
4
+ huggingface_hub>=0.25
train.py ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Training algorithm preserved from the original 2026-09-27 experiment.
2
+ Only environment/path plumbing and optional smoke-run lengths are adapted.
3
+ """
4
+ from pathlib import Path
5
+ import json, hashlib
6
+ import numpy as np
7
+ from mano_compat import enable_legacy_mano_loading
8
+ enable_legacy_mano_loading()
9
+ def sha(path):
10
+ with Path(path).open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest()
11
+ def write(path,value):
12
+ Path(path).parent.mkdir(parents=True,exist_ok=True)
13
+ Path(path).write_text(json.dumps(value,indent=2,allow_nan=False)+'\n')
14
+ import torch,time,smplx,argparse
15
+ from bimanual_vae import *
16
+ torch.set_num_threads(2)
17
+ def main(variant, data, mano_dir, output, device, epochs, steps_per_epoch):
18
+ output.mkdir(parents=True,exist_ok=True)
19
+ seed=20260927;torch.manual_seed(seed);rng=np.random.default_rng(seed);a=np.load(data,allow_pickle=False);local=aa_to_matrix(torch.tensor(a['pose_aa'],device=device)).reshape(-1,30,3,3);rot=torch.cat([local,torch.tensor(a['relative_rotation'],device=device)[:,None]],1);pos=torch.tensor(a['relative_wrist_m'],device=device);ds=a['dataset'];split=a['split'];seq=a['sequence'];included=[0,1] if variant=='joint_ego40' else [0,1,2];train=np.flatnonzero((split==0)&np.isin(ds,included));val=np.flatnonzero((split==1)&np.isin(ds,included));model=BimanualVAE(pose_only=variant=='pose_mix40').to(device);x=model.features(rot[train],pos[train]);model.input_mean.copy_(x.mean(0));model.input_std.copy_(x.std(0).clamp_min(.03));opt=torch.optim.AdamW(model.parameters(),lr=.001,weight_decay=1e-5)
20
+ # Balance datasets, then recordings; long recordings cannot dominate.
21
+ groups={d:{int(s):np.flatnonzero((split==0)&(ds==d)&(seq==s)) for s in np.unique(seq[(split==0)&(ds==d)])} for d in included}
22
+ # Simple FK of native 16 MANO skeletal joints, differentiated without rendering meshes.
23
+ offsets=[];parents=None
24
+ for s in range(2):
25
+ m=smplx.MANO(str(mano_dir),is_rhand=s==1,use_pca=False,flat_hand_mean=True);j=m.J_regressor@m.v_template;par=m.parents.numpy();off=j.clone();off[1:]-=j[par[1:]];off[0]=0;offsets.append(off);parents=par
26
+ off=torch.stack(offsets).to(device)
27
+ def fk(q):
28
+ q=q[:,:30].reshape(-1,2,15,3,3);n=len(q);g=[torch.eye(3,device=device).expand(n,2,3,3)];j=[torch.zeros(n,2,3,device=device)]
29
+ for k in range(1,16):
30
+ p=int(parents[k]);j.append(j[p]+(g[p]@off[None,:,k,:,None]).squeeze(-1));g.append(g[p]@q[:,:,k-1])
31
+ return torch.stack(j,2)
32
+ with torch.no_grad():ref=fk(rot)
33
+ cfg=dict(variant=variant,seed=seed,epochs=epochs,steps_per_epoch=steps_per_epoch,batch=192,lr=.001,kl_beta=.0005,joint_weight=.25,position_scale_m=.1,balanced_dataset_and_sequence=True,included_datasets=included,selection='equal-dataset validation local FK mm + 0.25 relative position mm + relative rotation deg; test not used',data_sha256=sha(data))
34
+ write(output/f'{variant}_config.json',cfg);best=float('inf');hist=[];start=time.monotonic()
35
+ for epoch in range(1,cfg['epochs']+1):
36
+ model.train();sums=[]
37
+ for step in range(cfg['steps_per_epoch']):
38
+ ids=[]
39
+ for d in included:
40
+ keys=list(groups[d]);ss=rng.choice(keys,192//len(included));ids.extend(int(rng.choice(groups[d][int(s)])) for s in ss)
41
+ ids=torch.tensor(ids,device=device);rr,pp,mu,lv=model(rot[ids],pos[ids]);rec=(rr-rot[ids,:model.nrot]).square().sum((-1,-2)).mean();jloss=((fk(rr)-ref[ids])/.02).square().sum(-1).mean();ploss=rec*0 if model.pose_only else ((pp-pos[ids])/.1).square().sum(-1).mean();kl=.5*(mu.square()+lv.exp()-lv-1).mean();loss=rec+.25*jloss+ploss+.0005*min(1,epoch/20)*kl;opt.zero_grad();loss.backward();torch.nn.utils.clip_grad_norm_(model.parameters(),5);opt.step();sums.append(float(loss.detach()))
42
+ if epoch==1 or epoch%5==0:
43
+ model.eval();met=[]
44
+ with torch.no_grad():
45
+ for d in included:
46
+ ids=val[ds[val]==d];errs=[];pe=[];re=[]
47
+ for t in range(0,len(ids),1024):
48
+ ix=ids[t:t+1024];rr,pp,mu,lv=model(rot[ix],pos[ix]);errs.append((fk(rr)-ref[ix]).norm(dim=-1).mean((1,2))*1000)
49
+ if not model.pose_only:pe.append((pp-pos[ix]).norm(dim=-1)*1000);rel=rr[:,30]@rot[ix,30].transpose(-1,-2);re.append(torch.acos(((rel.diagonal(dim1=-2,dim2=-1).sum(-1)-1)/2).clamp(-1+1e-7,1-1e-7))*180/torch.pi)
50
+ met.append(dict(dataset=['ARCTIC','H2O','GRAB'][d],local_fk_mm=float(torch.cat(errs).mean()),relative_wrist_mm=float(torch.cat(pe).mean()) if pe else None,relative_rotation_deg=float(torch.cat(re).mean()) if re else None))
51
+ score=float(np.mean([v['local_fk_mm']+.25*(v['relative_wrist_mm'] or 0)+(v['relative_rotation_deg'] or 0) for v in met]));entry=dict(epoch=epoch,train_loss=float(np.mean(sums)),validation=met,selection_score=score,seconds=time.monotonic()-start);hist.append(entry);print(variant,json.dumps(entry),flush=True)
52
+ if score<best:
53
+ best=score;torch.save(dict(architecture=dict(latent_dim=40,width=384,pose_only=model.pose_only),state_dict=model.state_dict(),config=cfg,epoch=epoch,validation=met),output/f'{variant}.pt')
54
+ write(output/f'{variant}_history.json',hist)
55
+ print('COMPLETE',variant,sha(output/f'{variant}.pt'),flush=True)
56
+ if __name__=='__main__':
57
+ p=argparse.ArgumentParser(description='Original paired-hand VAE recipe with portable paths.')
58
+ p.add_argument('--variant',choices=['joint_ego40','joint_mix40','pose_mix40'],default='joint_mix40')
59
+ p.add_argument('--data',type=Path,required=True)
60
+ p.add_argument('--mano-dir',type=Path,required=True)
61
+ p.add_argument('--output',type=Path,required=True)
62
+ p.add_argument('--device',default='cuda')
63
+ p.add_argument('--epochs',type=int,default=60)
64
+ p.add_argument('--steps-per-epoch',type=int,default=48)
65
+ a=p.parse_args()
66
+ if a.epochs<1 or a.steps_per_epoch<1:p.error('Training counts must be positive.')
67
+ if (a.output/(a.variant+'.pt')).exists():p.error('Output checkpoint already exists; choose a new output folder.')
68
+ main(a.variant,a.data,a.mano_dir,a.output,a.device,a.epochs,a.steps_per_epoch)
training/README.md ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Original training provenance
2
+
3
+ `original_train_pair.py`, `original_prepare_data.py`, and `original_common.py`
4
+ are unchanged copies from the 2026-09-27 MiroHand P3R4 experiment. They record the
5
+ exact model-training recipe and source-specific data preparation. `history.json`
6
+ is the original joint_mix40 validation history. These archived scripts retain
7
+ the original machine paths and are for reference; use `../train.py` to run
8
+ training with your own paths.
9
+
10
+ The original preparation depended on an earlier filtered ARCTIC/H2O manifest,
11
+ native MANO files, raw GRAB motion, and experiment-specific benchmark exclusion
12
+ lists. Those inputs are not bundled. `../DATA_FORMAT.md` defines the prepared
13
+ cache that the portable trainer consumes. The original training cache SHA-256 is
14
+ recorded in `../training_config.json`.
15
+
16
+ The portable trainer changes only path/device arguments, configurable epoch/step
17
+ counts, local JSON/hash utilities, and the MANO compatibility import. Dataset and
18
+ recording balancing, normalization, losses, optimizer, random seed, validation
19
+ selection, and model code follow the original recipe. CPU smoke checks exercise
20
+ one training update and validation; they do not reproduce the full GPU run.
21
+
22
+ The original experiment trained three variants. Only joint_mix40 weights are
23
+ released in this repository; the other variant names in the scripts are options
24
+ for training new models.
training/history.json ADDED
@@ -0,0 +1,340 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "epoch": 1,
4
+ "train_loss": 9.445151468118032,
5
+ "validation": [
6
+ {
7
+ "dataset": "ARCTIC",
8
+ "local_fk_mm": 12.87500286102295,
9
+ "relative_wrist_mm": 163.0471954345703,
10
+ "relative_rotation_deg": 51.87311553955078
11
+ },
12
+ {
13
+ "dataset": "H2O",
14
+ "local_fk_mm": 9.052749633789062,
15
+ "relative_wrist_mm": 117.507080078125,
16
+ "relative_rotation_deg": 38.650142669677734
17
+ },
18
+ {
19
+ "dataset": "GRAB",
20
+ "local_fk_mm": 8.72321891784668,
21
+ "relative_wrist_mm": 189.7671661376953,
22
+ "relative_rotation_deg": 58.388023376464844
23
+ }
24
+ ],
25
+ "selection_score": 99.0475378036499,
26
+ "seconds": 0.3951757492031902
27
+ },
28
+ {
29
+ "epoch": 5,
30
+ "train_loss": 0.5073744927843412,
31
+ "validation": [
32
+ {
33
+ "dataset": "ARCTIC",
34
+ "local_fk_mm": 7.015540599822998,
35
+ "relative_wrist_mm": 60.61729431152344,
36
+ "relative_rotation_deg": 38.23624038696289
37
+ },
38
+ {
39
+ "dataset": "H2O",
40
+ "local_fk_mm": 4.575192451477051,
41
+ "relative_wrist_mm": 51.943145751953125,
42
+ "relative_rotation_deg": 29.724584579467773
43
+ },
44
+ {
45
+ "dataset": "GRAB",
46
+ "local_fk_mm": 5.3200531005859375,
47
+ "relative_wrist_mm": 54.36599349975586,
48
+ "relative_rotation_deg": 44.823211669921875
49
+ }
50
+ ],
51
+ "selection_score": 57.14214372634888,
52
+ "seconds": 1.4217542971018702
53
+ },
54
+ {
55
+ "epoch": 10,
56
+ "train_loss": 0.2648027865216136,
57
+ "validation": [
58
+ {
59
+ "dataset": "ARCTIC",
60
+ "local_fk_mm": 5.106489181518555,
61
+ "relative_wrist_mm": 33.648101806640625,
62
+ "relative_rotation_deg": 31.729633331298828
63
+ },
64
+ {
65
+ "dataset": "H2O",
66
+ "local_fk_mm": 3.4149880409240723,
67
+ "relative_wrist_mm": 25.74277114868164,
68
+ "relative_rotation_deg": 23.669815063476562
69
+ },
70
+ {
71
+ "dataset": "GRAB",
72
+ "local_fk_mm": 4.339122295379639,
73
+ "relative_wrist_mm": 38.02174758911133,
74
+ "relative_rotation_deg": 43.17218017578125
75
+ }
76
+ ],
77
+ "selection_score": 45.26179440816244,
78
+ "seconds": 2.826522510033101
79
+ },
80
+ {
81
+ "epoch": 15,
82
+ "train_loss": 0.16469275082151094,
83
+ "validation": [
84
+ {
85
+ "dataset": "ARCTIC",
86
+ "local_fk_mm": 4.236188888549805,
87
+ "relative_wrist_mm": 30.4024715423584,
88
+ "relative_rotation_deg": 26.648405075073242
89
+ },
90
+ {
91
+ "dataset": "H2O",
92
+ "local_fk_mm": 2.940308094024658,
93
+ "relative_wrist_mm": 34.45616149902344,
94
+ "relative_rotation_deg": 16.92371940612793
95
+ },
96
+ {
97
+ "dataset": "GRAB",
98
+ "local_fk_mm": 3.87188458442688,
99
+ "relative_wrist_mm": 43.58067321777344,
100
+ "relative_rotation_deg": 35.69742202758789
101
+ }
102
+ ],
103
+ "selection_score": 39.14258488019308,
104
+ "seconds": 4.37071700906381
105
+ },
106
+ {
107
+ "epoch": 20,
108
+ "train_loss": 0.14342589862644672,
109
+ "validation": [
110
+ {
111
+ "dataset": "ARCTIC",
112
+ "local_fk_mm": 3.692901849746704,
113
+ "relative_wrist_mm": 26.702104568481445,
114
+ "relative_rotation_deg": 21.544221878051758
115
+ },
116
+ {
117
+ "dataset": "H2O",
118
+ "local_fk_mm": 2.599233627319336,
119
+ "relative_wrist_mm": 25.14226722717285,
120
+ "relative_rotation_deg": 13.099417686462402
121
+ },
122
+ {
123
+ "dataset": "GRAB",
124
+ "local_fk_mm": 3.521355390548706,
125
+ "relative_wrist_mm": 24.10993003845215,
126
+ "relative_rotation_deg": 26.255090713500977
127
+ }
128
+ ],
129
+ "selection_score": 29.90026553471883,
130
+ "seconds": 5.522418719017878
131
+ },
132
+ {
133
+ "epoch": 25,
134
+ "train_loss": 0.12864494665215412,
135
+ "validation": [
136
+ {
137
+ "dataset": "ARCTIC",
138
+ "local_fk_mm": 3.295466184616089,
139
+ "relative_wrist_mm": 36.14392852783203,
140
+ "relative_rotation_deg": 19.423234939575195
141
+ },
142
+ {
143
+ "dataset": "H2O",
144
+ "local_fk_mm": 2.3364412784576416,
145
+ "relative_wrist_mm": 32.38401412963867,
146
+ "relative_rotation_deg": 12.863519668579102
147
+ },
148
+ {
149
+ "dataset": "GRAB",
150
+ "local_fk_mm": 3.4695565700531006,
151
+ "relative_wrist_mm": 37.982025146484375,
152
+ "relative_rotation_deg": 25.931699752807617
153
+ }
154
+ ],
155
+ "selection_score": 31.315803448359173,
156
+ "seconds": 6.841986288083717
157
+ },
158
+ {
159
+ "epoch": 30,
160
+ "train_loss": 0.08499159834658106,
161
+ "validation": [
162
+ {
163
+ "dataset": "ARCTIC",
164
+ "local_fk_mm": 3.0357227325439453,
165
+ "relative_wrist_mm": 20.172719955444336,
166
+ "relative_rotation_deg": 17.05535125732422
167
+ },
168
+ {
169
+ "dataset": "H2O",
170
+ "local_fk_mm": 2.1297340393066406,
171
+ "relative_wrist_mm": 12.941731452941895,
172
+ "relative_rotation_deg": 10.320504188537598
173
+ },
174
+ {
175
+ "dataset": "GRAB",
176
+ "local_fk_mm": 3.1395211219787598,
177
+ "relative_wrist_mm": 17.7632999420166,
178
+ "relative_rotation_deg": 20.985870361328125
179
+ }
180
+ ],
181
+ "selection_score": 23.128713846206665,
182
+ "seconds": 8.110679685138166
183
+ },
184
+ {
185
+ "epoch": 35,
186
+ "train_loss": 0.0695953315589577,
187
+ "validation": [
188
+ {
189
+ "dataset": "ARCTIC",
190
+ "local_fk_mm": 2.890573740005493,
191
+ "relative_wrist_mm": 18.770294189453125,
192
+ "relative_rotation_deg": 16.014467239379883
193
+ },
194
+ {
195
+ "dataset": "H2O",
196
+ "local_fk_mm": 1.9951664209365845,
197
+ "relative_wrist_mm": 11.237444877624512,
198
+ "relative_rotation_deg": 9.762822151184082
199
+ },
200
+ {
201
+ "dataset": "GRAB",
202
+ "local_fk_mm": 2.955854654312134,
203
+ "relative_wrist_mm": 16.69818878173828,
204
+ "relative_rotation_deg": 19.13675308227539
205
+ }
206
+ ],
207
+ "selection_score": 21.477373083432514,
208
+ "seconds": 9.64998462004587
209
+ },
210
+ {
211
+ "epoch": 40,
212
+ "train_loss": 0.07276629415961604,
213
+ "validation": [
214
+ {
215
+ "dataset": "ARCTIC",
216
+ "local_fk_mm": 2.755661964416504,
217
+ "relative_wrist_mm": 17.329029083251953,
218
+ "relative_rotation_deg": 14.625399589538574
219
+ },
220
+ {
221
+ "dataset": "H2O",
222
+ "local_fk_mm": 1.9094189405441284,
223
+ "relative_wrist_mm": 9.26237678527832,
224
+ "relative_rotation_deg": 9.038466453552246
225
+ },
226
+ {
227
+ "dataset": "GRAB",
228
+ "local_fk_mm": 2.7325074672698975,
229
+ "relative_wrist_mm": 16.747617721557617,
230
+ "relative_rotation_deg": 20.136404037475586
231
+ }
232
+ ],
233
+ "selection_score": 20.67753811677297,
234
+ "seconds": 10.801130139036104
235
+ },
236
+ {
237
+ "epoch": 45,
238
+ "train_loss": 0.05866521876305342,
239
+ "validation": [
240
+ {
241
+ "dataset": "ARCTIC",
242
+ "local_fk_mm": 2.6570005416870117,
243
+ "relative_wrist_mm": 16.533859252929688,
244
+ "relative_rotation_deg": 13.954176902770996
245
+ },
246
+ {
247
+ "dataset": "H2O",
248
+ "local_fk_mm": 1.812648057937622,
249
+ "relative_wrist_mm": 10.88892650604248,
250
+ "relative_rotation_deg": 8.842887878417969
251
+ },
252
+ {
253
+ "dataset": "GRAB",
254
+ "local_fk_mm": 2.5590364933013916,
255
+ "relative_wrist_mm": 19.327693939208984,
256
+ "relative_rotation_deg": 18.18161964416504
257
+ }
258
+ ],
259
+ "selection_score": 19.898329814275105,
260
+ "seconds": 12.013499305117875
261
+ },
262
+ {
263
+ "epoch": 50,
264
+ "train_loss": 0.05538660536209742,
265
+ "validation": [
266
+ {
267
+ "dataset": "ARCTIC",
268
+ "local_fk_mm": 2.552030563354492,
269
+ "relative_wrist_mm": 15.961709022521973,
270
+ "relative_rotation_deg": 13.042522430419922
271
+ },
272
+ {
273
+ "dataset": "H2O",
274
+ "local_fk_mm": 1.722329020500183,
275
+ "relative_wrist_mm": 7.883533000946045,
276
+ "relative_rotation_deg": 8.399897575378418
277
+ },
278
+ {
279
+ "dataset": "GRAB",
280
+ "local_fk_mm": 2.7820262908935547,
281
+ "relative_wrist_mm": 18.310850143432617,
282
+ "relative_rotation_deg": 17.670961380004883
283
+ }
284
+ ],
285
+ "selection_score": 18.90293010075887,
286
+ "seconds": 13.535741310101002
287
+ },
288
+ {
289
+ "epoch": 55,
290
+ "train_loss": 0.10357473092153668,
291
+ "validation": [
292
+ {
293
+ "dataset": "ARCTIC",
294
+ "local_fk_mm": 2.5025906562805176,
295
+ "relative_wrist_mm": 32.9027214050293,
296
+ "relative_rotation_deg": 12.71257495880127
297
+ },
298
+ {
299
+ "dataset": "H2O",
300
+ "local_fk_mm": 1.7116860151290894,
301
+ "relative_wrist_mm": 24.95145034790039,
302
+ "relative_rotation_deg": 8.395352363586426
303
+ },
304
+ {
305
+ "dataset": "GRAB",
306
+ "local_fk_mm": 2.518946647644043,
307
+ "relative_wrist_mm": 29.448965072631836,
308
+ "relative_rotation_deg": 18.022743225097656
309
+ }
310
+ ],
311
+ "selection_score": 22.563226024309795,
312
+ "seconds": 14.875219596084207
313
+ },
314
+ {
315
+ "epoch": 60,
316
+ "train_loss": 0.04643667683315774,
317
+ "validation": [
318
+ {
319
+ "dataset": "ARCTIC",
320
+ "local_fk_mm": 2.3993451595306396,
321
+ "relative_wrist_mm": 22.116172790527344,
322
+ "relative_rotation_deg": 12.27881908416748
323
+ },
324
+ {
325
+ "dataset": "H2O",
326
+ "local_fk_mm": 1.6551059484481812,
327
+ "relative_wrist_mm": 18.894102096557617,
328
+ "relative_rotation_deg": 8.041049003601074
329
+ },
330
+ {
331
+ "dataset": "GRAB",
332
+ "local_fk_mm": 2.4477932453155518,
333
+ "relative_wrist_mm": 16.990732192993164,
334
+ "relative_rotation_deg": 16.812602996826172
335
+ }
336
+ ],
337
+ "selection_score": 19.37832240263621,
338
+ "seconds": 16.251781224040315
339
+ }
340
+ ]
training/original_common.py ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import sys,json,hashlib,inspect
3
+ import numpy as np
4
+ R=Path('/SSD3/yangzijing/MiroHand_P3R4_bimanual_VAE_20260927')
5
+ OLD=Path('/SSD3/yangzijing/FoundationHand-A2_P3R3_VAE_20260922')
6
+ REPO=Path('/SSD2/yangzijing/egoview-v9-foundationhand-s9l10-20260901')
7
+ MANO=Path('/SSD3/yangzijing/shared_dependencies/foundation_hand/pretrained/mano')
8
+ sys.path.insert(0,str(REPO));inspect.getargspec=inspect.getfullargspec
9
+ from pipeline.utils.numpy_compat import ensure_chumpy_numpy_compat
10
+ ensure_chumpy_numpy_compat()
11
+ def read(p):return json.loads(Path(p).read_text())
12
+ def write(p,x):Path(p).parent.mkdir(parents=True,exist_ok=True);Path(p).write_text(json.dumps(x,ensure_ascii=False,indent=2,allow_nan=False)+'\n')
13
+ def sha(p):
14
+ with Path(p).open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest()
training/original_prepare_data.py ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from common import *
2
+ import collections,trimesh,smplx,torch,warnings
3
+ from scipy.spatial.transform import Rotation
4
+ warnings.filterwarnings('ignore',category=DeprecationWarning);torch.set_num_threads(2)
5
+ F=Path('/SSD3/yangzijing/ego_full_training_filter_visible_20260925');G=Path('/SSD1/wbr/datasets/GRAB_source/GRAB_unzipped')
6
+ models=[smplx.MANO(str(MANO),is_rhand=s==1,use_pca=True,num_pca_comps=24,flat_hand_mean=False) for s in range(2)]
7
+ ex=set(map(tuple,read(OLD/'priors/excluded_recordings.json')))
8
+ for c in read('/SSD3/yangzijing/MiroHand_A6_benchmark_20260924/selection.json'):ex.add((c['dataset'],c['recording_id']))
9
+ write(R/'audit/excluded_pipeline_recordings.json',sorted(ex))
10
+ seqs=[];refs={};summary=collections.Counter();data=collections.defaultdict(list)
11
+ def add(ds,seq,pose,rot,wrist,fids,source,split=None):
12
+ pose=np.asarray(pose,np.float32).reshape(-1,2,15,3);rot=np.asarray(rot,np.float32).reshape(-1,2,3);wrist=np.asarray(wrist,np.float32);n=len(pose)
13
+ good=np.isfinite(pose).all((1,2,3))&np.isfinite(rot).all((1,2))&np.isfinite(wrist).all((1,2));summary[ds+'_nonfinite_removed']+=int((~good).sum());pose,rot,wrist,fids=pose[good],rot[good],wrist[good],np.asarray(fids)[good]
14
+ if not len(pose):return
15
+ # Each recording contributes at most 160 evenly spaced real paired frames.
16
+ ids=np.unique(np.linspace(0,len(pose)-1,min(160,len(pose))).astype(int));pose,rot,wrist,fids=pose[ids],rot[ids],wrist[ids],fids[ids]
17
+ rr=Rotation.from_rotvec(rot.reshape(-1,3)).as_matrix().reshape(-1,2,3,3);relrot=rr[:,0].transpose(0,2,1)@rr[:,1];relpos=np.einsum('nji,nj->ni',rr[:,0],wrist[:,1]-wrist[:,0]);dist=np.linalg.norm(relpos,axis=-1);assert np.isfinite(relpos).all() and dist.max()<3,(ds,seq,dist.max())
18
+ # Invariant to shared camera/world rigid transform. Keep anchor for exact roundtrip.
19
+ reconstructed=np.einsum('nij,nj->ni',rr[:,0],relpos)+wrist[:,0];assert np.max(abs(reconstructed-wrist[:,1]))<1e-5
20
+ if split is None:
21
+ h=int(hashlib.sha256(('P3R4_20260927:'+ds+':'+seq).encode()).hexdigest()[:8],16)%10;split='test' if h==0 else 'validation' if h==1 else 'train'
22
+ sid=len(seqs);seqs.append(dict(dataset=ds,recording=seq,split=split,frames=len(pose),source=source))
23
+ for k,v in dict(pose_aa=pose,relative_rotation=relrot.astype(np.float32),relative_wrist_m=relpos.astype(np.float32),anchor_rotation=rr[:,0].astype(np.float32),anchor_wrist_m=wrist[:,0],frame_ids=fids,sequence=np.full(len(pose),sid),dataset=np.full(len(pose),['ARCTIC','H2O','GRAB'].index(ds)),split=np.full(len(pose),['train','validation','test'].index(split))).items():data[k].append(v)
24
+ summary[ds+'_'+split]+=len(pose)
25
+ rows=[w for w in map(json.loads,(F/'unified/filtered_manifest.jsonl').open()) if w['dataset'] in ['ARCTIC','H2O'] and len(w['target_sides'])==2]
26
+ groups=collections.defaultdict(list)
27
+ for w in rows:
28
+ rec=w['recording_id'];canonical=rec.removesuffix('/cam4')
29
+ if (w['dataset'],rec) in ex or (w['dataset'],canonical) in ex:summary[w['dataset']+'_excluded_windows']+=1;continue
30
+ groups[(w['dataset'],canonical)].append(w)
31
+ for (ds,seq),ws in sorted(groups.items()):
32
+ ar=collections.defaultdict(list);paths=[]
33
+ for w in sorted(ws,key=lambda x:x['start']):
34
+ p=Path(w['filtered_gt_path']);assert sha(p)==w['gt_sha256'];refs[str(p)]=w['gt_sha256'];z=np.load(p);ok=z['valid'].all(1)
35
+ # Original local_pose includes the native MANO hand mean already.
36
+ for k in ['local_pose_aa_rad','native_global_orient_aa_rad','joints_native_m','frame_ids']:ar[k].append(z[k][ok])
37
+ paths.append(str(p))
38
+ ar={k:np.concatenate(v) for k,v in ar.items()};add(ds,seq,ar['local_pose_aa_rad'],ar['native_global_orient_aa_rad'],ar['joints_native_m'][:,:,0],ar['frame_ids'],paths)
39
+ print('EGO',dict(summary),flush=True)
40
+ rootcache={};convertmax=0.
41
+ for i,p in enumerate(sorted((G/'grab').glob('*/*.npz'))):
42
+ a=np.load(p,allow_pickle=True);pose=[];rot=[];wrist=[];n=int(a['n_frames']);ids=np.arange(0,n,12) # 120fps to 10fps; no interpolation
43
+ for s,key in enumerate(['lhand','rhand']):
44
+ h=a[key].item();q=h['params'];m=models[s];local=q['hand_pose'][ids]@m.hand_components.numpy()+m.hand_mean.numpy();err=float(np.max(abs(local-(q['fullpose'][ids]+m.hand_mean.numpy()))));assert err<2e-5;convertmax=max(convertmax,err)
45
+ template=G/h['vtemp']
46
+ if str(template) not in rootcache:
47
+ vt=np.asarray(trimesh.load(template,process=False).vertices);assert vt.shape==(778,3);rootcache[str(template)]=m.J_regressor[0].numpy()@vt;refs[str(template)]=sha(template)
48
+ pose.append(local);rot.append(q['global_orient'][ids]);wrist.append(q['transl'][ids]+rootcache[str(template)])
49
+ refs[str(p)]=sha(p);sub=p.parent.name;split='test' if sub=='s10' else 'validation' if sub=='s9' else 'train';add('GRAB',str(p.relative_to(G/'grab')).removesuffix('.npz'),np.stack(pose,1),np.stack(rot,1),np.stack(wrist,1),ids,[str(p)],split)
50
+ if i%200==0:print('GRAB',i,dict(summary),flush=True)
51
+ np.savez_compressed(R/'data/paired_motion.npz',**{k:np.concatenate(v) for k,v in data.items()});write(R/'data/sequences.json',seqs);write(R/'audit/source_hashes.json',refs)
52
+ old=read(OLD/'priors/training_manifest_local_only.json');write(R/'audit/previous_single_vae_data.json',dict(counts=old['counts'],data_sha256=old['pose_data_sha256'],note='DexYCB and HO3D single-hand supervision is not converted into artificial paired samples'))
53
+ write(R/'data/summary.json',dict(counts=dict(summary),sequences=len(seqs),paired_frames=sum(x['frames'] for x in seqs),data_sha256=sha(R/'data/paired_motion.npz'),grab_pca_fullpose_max_error_rad=convertmax,coordinate='left wrist MANO local frame; invert using anchor wrist pose in ego camera; no fake camera GT',schema='30 local SO3 rotations + relative wrist SO3 + relative wrist XYZ metres; 189D rotation6D input',sources_missing=['HIMO','Embody3D'],split='GRAB subject s1-s8 train, s9 validation, s10 test; ARCTIC/H2O deterministic sequence split 80/10/10; pipeline benchmark recordings excluded'))
54
+ print('COMPLETE',read(R/'data/summary.json'),flush=True)
training/original_train_pair.py ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from common import *
2
+ import torch,time,smplx,argparse
3
+ from bimanual_vae import *
4
+ torch.set_num_threads(2)
5
+ def main(variant):
6
+ seed=20260927;torch.manual_seed(seed);rng=np.random.default_rng(seed);a=np.load(R/'data/paired_motion.npz');device='cuda';local=aa_to_matrix(torch.tensor(a['pose_aa'],device=device)).reshape(-1,30,3,3);rot=torch.cat([local,torch.tensor(a['relative_rotation'],device=device)[:,None]],1);pos=torch.tensor(a['relative_wrist_m'],device=device);ds=a['dataset'];split=a['split'];seq=a['sequence'];included=[0,1] if variant=='joint_ego40' else [0,1,2];train=np.flatnonzero((split==0)&np.isin(ds,included));val=np.flatnonzero((split==1)&np.isin(ds,included));model=BimanualVAE(pose_only=variant=='pose_mix40').to(device);x=model.features(rot[train],pos[train]);model.input_mean.copy_(x.mean(0));model.input_std.copy_(x.std(0).clamp_min(.03));opt=torch.optim.AdamW(model.parameters(),lr=.001,weight_decay=1e-5)
7
+ # Balance datasets, then recordings; long recordings cannot dominate.
8
+ groups={d:{int(s):np.flatnonzero((split==0)&(ds==d)&(seq==s)) for s in np.unique(seq[(split==0)&(ds==d)])} for d in included}
9
+ # Simple FK of native 16 MANO skeletal joints, differentiated without rendering meshes.
10
+ offsets=[];parents=None
11
+ for s in range(2):
12
+ m=smplx.MANO(str(MANO),is_rhand=s==1,use_pca=False,flat_hand_mean=True);j=m.J_regressor@m.v_template;par=m.parents.numpy();off=j.clone();off[1:]-=j[par[1:]];off[0]=0;offsets.append(off);parents=par
13
+ off=torch.stack(offsets).to(device)
14
+ def fk(q):
15
+ q=q[:,:30].reshape(-1,2,15,3,3);n=len(q);g=[torch.eye(3,device=device).expand(n,2,3,3)];j=[torch.zeros(n,2,3,device=device)]
16
+ for k in range(1,16):
17
+ p=int(parents[k]);j.append(j[p]+(g[p]@off[None,:,k,:,None]).squeeze(-1));g.append(g[p]@q[:,:,k-1])
18
+ return torch.stack(j,2)
19
+ with torch.no_grad():ref=fk(rot)
20
+ cfg=dict(variant=variant,seed=seed,epochs=60,steps_per_epoch=48,batch=192,lr=.001,kl_beta=.0005,joint_weight=.25,position_scale_m=.1,balanced_dataset_and_sequence=True,included_datasets=included,selection='equal-dataset validation local FK mm + 0.25 relative position mm + relative rotation deg; test not used',data_sha256=sha(R/'data/paired_motion.npz'))
21
+ write(R/'priors'/f'{variant}_config.json',cfg);best=float('inf');hist=[];start=time.monotonic()
22
+ for epoch in range(1,cfg['epochs']+1):
23
+ model.train();sums=[]
24
+ for step in range(cfg['steps_per_epoch']):
25
+ ids=[]
26
+ for d in included:
27
+ keys=list(groups[d]);ss=rng.choice(keys,192//len(included));ids.extend(int(rng.choice(groups[d][int(s)])) for s in ss)
28
+ ids=torch.tensor(ids,device=device);rr,pp,mu,lv=model(rot[ids],pos[ids]);rec=(rr-rot[ids,:model.nrot]).square().sum((-1,-2)).mean();jloss=((fk(rr)-ref[ids])/.02).square().sum(-1).mean();ploss=rec*0 if model.pose_only else ((pp-pos[ids])/.1).square().sum(-1).mean();kl=.5*(mu.square()+lv.exp()-lv-1).mean();loss=rec+.25*jloss+ploss+.0005*min(1,epoch/20)*kl;opt.zero_grad();loss.backward();torch.nn.utils.clip_grad_norm_(model.parameters(),5);opt.step();sums.append(float(loss.detach()))
29
+ if epoch==1 or epoch%5==0:
30
+ model.eval();met=[]
31
+ with torch.no_grad():
32
+ for d in included:
33
+ ids=val[ds[val]==d];errs=[];pe=[];re=[]
34
+ for t in range(0,len(ids),1024):
35
+ ix=ids[t:t+1024];rr,pp,mu,lv=model(rot[ix],pos[ix]);errs.append((fk(rr)-ref[ix]).norm(dim=-1).mean((1,2))*1000)
36
+ if not model.pose_only:pe.append((pp-pos[ix]).norm(dim=-1)*1000);rel=rr[:,30]@rot[ix,30].transpose(-1,-2);re.append(torch.acos(((rel.diagonal(dim1=-2,dim2=-1).sum(-1)-1)/2).clamp(-1+1e-7,1-1e-7))*180/torch.pi)
37
+ met.append(dict(dataset=['ARCTIC','H2O','GRAB'][d],local_fk_mm=float(torch.cat(errs).mean()),relative_wrist_mm=float(torch.cat(pe).mean()) if pe else None,relative_rotation_deg=float(torch.cat(re).mean()) if re else None))
38
+ score=float(np.mean([v['local_fk_mm']+.25*(v['relative_wrist_mm'] or 0)+(v['relative_rotation_deg'] or 0) for v in met]));entry=dict(epoch=epoch,train_loss=float(np.mean(sums)),validation=met,selection_score=score,seconds=time.monotonic()-start);hist.append(entry);print(variant,json.dumps(entry),flush=True)
39
+ if score<best:
40
+ best=score;torch.save(dict(architecture=dict(latent_dim=40,width=384,pose_only=model.pose_only),state_dict=model.state_dict(),config=cfg,epoch=epoch,validation=met),R/'priors'/f'{variant}.pt')
41
+ write(R/'priors'/f'{variant}_history.json',hist)
42
+ print('COMPLETE',variant,sha(R/'priors'/f'{variant}.pt'),flush=True)
43
+ if __name__=='__main__':
44
+ p=argparse.ArgumentParser();p.add_argument('variant',choices=['joint_ego40','joint_mix40','pose_mix40']);main(p.parse_args().variant)
training_config.json ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "joint_mix40",
3
+ "seed": 20260927,
4
+ "epochs": 60,
5
+ "steps_per_epoch": 48,
6
+ "batch": 192,
7
+ "lr": 0.001,
8
+ "kl_beta": 0.0005,
9
+ "joint_weight": 0.25,
10
+ "position_scale_m": 0.1,
11
+ "balanced_dataset_and_sequence": true,
12
+ "included_datasets": [
13
+ 0,
14
+ 1,
15
+ 2
16
+ ],
17
+ "selection": "equal-dataset validation local FK mm + 0.25 relative position mm + relative rotation deg; test not used",
18
+ "data_sha256": "456c2b8011b56552a0806836fdb2eb9fcdd397ca3693f390bb2cbb5f8ba0f06f"
19
+ }
validation.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "passed": true,
3
+ "torch": "2.6.0+cu124",
4
+ "device": "cpu",
5
+ "original_checkpoint_equals_safetensors": true,
6
+ "original_and_safe_outputs_identical": true,
7
+ "pack_rigid_transform_invariance": true,
8
+ "train_only_encoding_rows": 120,
9
+ "heldout_encoding_rows_excluded": 12,
10
+ "augmentation_samples": 72,
11
+ "distinct_recording_neighbors": 16,
12
+ "latent_convex_formula_verified": true,
13
+ "proper_rotation_matrices": true,
14
+ "axis_angle_roundtrip": true,
15
+ "test_queries_rejected": true,
16
+ "original_cpu_example": {
17
+ "finite_input_gradients": true,
18
+ "frozen_vae": true
19
+ },
20
+ "training_smoke": {
21
+ "epochs": 1,
22
+ "updates": 1,
23
+ "batch_size": 192,
24
+ "validation_completed": true,
25
+ "checkpoint_saved_and_loaded": true,
26
+ "finite_parameters": true,
27
+ "full_retraining_performed": false
28
+ }
29
+ }