--- library_name: pytorch language: - en - zh tags: - hand-pose - bimanual - vae - mano - pose-generation - data-augmentation --- # Bimanual Pose VAE — joint_mix40 双手姿态 VAE 的代码与权重:学习双手局部姿态、相对腕朝向和相对腕位置, 可用于姿态先验,以及从训练 GT 的 latent 邻域生成渲染增广候选。 This is the **exact joint_mix40 checkpoint** used by the [VAE Pose Atlas visualization](https://huggingface.co/spaces/WhynotGPT/vae-pose-atlas). The framewise model encodes 30 local finger rotations, one relative wrist rotation, and a relative wrist position into a **40-dimensional latent**. It uses a width-384 MLP and a 189-dimensional input with row-based 6D rotation representations. Shared global rotation and translation are removed by expressing the wrist relation in the left wrist frame. RGB, camera parameters, hand shape, and temporal dynamics are outside the model. ## Download and run Python 3.11 is the tested environment. Download the repository, then run from its root: ```bash pip install huggingface_hub python -c "from huggingface_hub import snapshot_download; snapshot_download('WhynotGPT/bimanual-pose-vae', local_dir='bimanual-pose-vae')" cd bimanual-pose-vae pip install -r requirements.txt python example.py --device cpu sha256sum -c SHA256SUMS ``` The example uses synthetic inputs to verify loading, reconstruction, and input gradients through the frozen VAE. It does not measure accuracy. ```python import torch from load_model import load_model from bimanual_vae import pack vae, metadata = load_model(device="cpu") # model.safetensors + config.json # local_rot: [B,2,15,3,3]; root_rot: [B,2,3,3]; wrist: [B,2,3] with torch.no_grad(): rot, pos = pack(local_rot, root_rot, wrist) mu, logvar = vae.encode(rot, pos) reconstructed_rot, reconstructed_pos = vae.decode(mu) ``` Inputs use **left, right** ordering and each hand's native MANO convention. Local rotations represent the full pose, including the native hand mean where required by the source convention. Wrist positions are actual joint positions in meters, not necessarily the MANO `transl` parameter. Both hands must be valid. Decoder rotations have shape `[B,31,3,3]`: left fingers `0:15`, right fingers `15:30`, and relative right-wrist orientation `30`. Position `[B,3]` is the right wrist in the left wrist frame. To place a pose in a scene, choose a left-wrist anchor `(R_L,t_L)`, then set `R_R=R_L @ rot[:,30]` and `t_R=t_L+R_L @ pos`. The original API is also available: ```python from bimanual_vae import load_pair vae, metadata = load_pair("joint_mix40.pt", device="cpu") # Preserve gradients through the inputs when using a frozen prior: loss_prior = vae.energy(local_rot, root_rot, wrist).mean() ``` ## Generate augmentation poses from training GT The portable sampler follows the train-GT neighborhood method used in the Atlas. It encodes only `split=0` rows, retrieves the nearest training pose in the same dataset as each query, and finds neighbors from 16 distinct other training recordings. Queries can be difficult training/validation GT poses; test queries are rejected. The caller selects difficult queries, for example using a model's validation PA error. ```bash python augment.py encode --data /path/to/paired_motion.npz --output train_latents.npz python augment.py sample --pool train_latents.npz --queries /path/to/difficult_gt.npz --output render_poses.npz ``` For each retrieved center, the sampler draws eight neighbor pairs with weights `w` in `[0.2,0.8]` and reuses those endpoints at `tau = 0.25, 0.5, 1.0`: ```text endpoint = w * mu_neighbor_a + (1-w) * mu_neighbor_b z = (1-tau) * mu_train_center + tau * endpoint ``` Each query yields **24 synthetic bimanual poses**. Output includes full MANO axis angles and rotation matrices, actual wrist positions, zero shape coefficients, latent values, and training-source indices. The left wrist is at the origin with identity orientation. See [DATA_FORMAT.md](DATA_FORMAT.md) for all array conventions. These outputs are pose labels for subsequent rendering. Geometry, scene placement, hand-object contact, and collision checks belong to the rendering pipeline. The model alone does not generate mesh vertices or guarantee physical plausibility. Augmentation gains on a downstream hand estimator have not been established by this release. The portable script implements the same sampling rule; it does not recreate the Atlas's source-specific query extraction or its exact random draw order. ## Training The released checkpoint was selected at **epoch 50** of a 60-epoch run using validation reconstruction metrics. Training uses 48 updates per epoch, batch 192, AdamW at 0.001, dataset/recording-balanced random sampling, rotation and skeletal FK reconstruction, relative-position reconstruction, and KL regularization. | Source | Train paired frames | Validation paired frames | Test paired frames | |---|---:|---:|---:| | ARCTIC | 32,320 | 4,960 | 4,480 | | H2O | 12,686 | 2,560 | 1,760 | | GRAB | 104,825 | 10,460 | 11,834 | ARCTIC/H2O use frozen recording splits with the experiment's benchmark recordings excluded. GRAB uses subjects s1–s8 / s9 / s10. Preparation retains at most 160 paired frames per recording; GRAB is first downsampled from 120 fps to 10 fps. These are VAE training counts, separate from the main estimator's SFT clip counts. ```bash pip install -r requirements-training.txt python train.py --data /path/to/paired_motion.npz \ --mano-dir /path/to/mano --output runs/joint_mix40 --device cuda ``` `train.py` preserves the original training algorithm, adding configurable paths, device, and run lengths. It needs your prepared motion cache and MANO_LEFT.pkl / MANO_RIGHT.pkl for the native skeleton FK loss. Raw data, the motion cache, MANO model assets, and their derived mesh buffers are not distributed here. Obtain those dependencies under their respective terms. The original experiment scripts are archived in [training/](training/README.md); their historical absolute paths and external preprocessing dependencies are documented there. The repository includes only the **joint_mix40 trained weights**. Training code also exposes the historical joint_ego40 and pose_mix40 variants. ## Files and verification - `joint_mix40.pt`: byte-identical original checkpoint, with normalization and metadata. - `model.safetensors`, `config.json`: equivalent tensor-only weights and JSON metadata. - `bimanual_vae.py`, `hand_vae.py`: unchanged original model definitions and rotation helpers. - `load_model.py`, `example.py`: loading and differentiable-prior examples. - `train.py`, `training_config.json`, `training/`: training entry point and provenance. - `augment.py`, `DATA_FORMAT.md`: train-GT encoding and neighborhood sampling. - `validation.json`, `provenance.json`, `SHA256SUMS`: checks and file integrity records. Original checkpoint SHA-256: `e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096`. The original model and training work is from the MiroHand P3R4 bimanual VAE experiment dated 2026-09-27; this package adds portable interfaces for release.