Release joint Pi3 checkpoint: MANO 110k, Mask 40k, pi32 matching
Browse files- README.md +57 -0
- VALIDATION.json +27 -0
- stereohand_pi3_mano110k_mask40k_matching_pi32.pt +3 -0
- unpack_checkpoint.py +23 -0
README.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
tags:
|
| 5 |
+
- stereohand
|
| 6 |
+
- stereo
|
| 7 |
+
- hand-reconstruction
|
| 8 |
+
- slam
|
| 9 |
+
- pytorch
|
| 10 |
+
---
|
| 11 |
+
# StereoHand
|
| 12 |
+
|
| 13 |
+
[Code](https://github.com/cskrren/StereoHand)
|
| 14 |
+
|
| 15 |
+
Joint Pi3 inference checkpoint combining temporal MANO **110,000 steps**, temporal Mask **40,000 steps**, and the **Pi3 matching `pi32`** decoder/head. Temporal step counts are counted from the start of temporal fine-tuning. The hand Pi3 register token is used for shared inference.
|
| 16 |
+
|
| 17 |
+
## Checkpoint
|
| 18 |
+
|
| 19 |
+
`stereohand_pi3_mano110k_mask40k_matching_pi32.pt` contains full reconstructed model states, including the frozen Pi3 and stereo parameters. Each hand branch is restored from its initial model plus its latest complete training delta. Independently trained Mask and MANO fusion layers are preserved. Identical tensors share storage; values and dtypes are unchanged. Optimizer state is omitted.
|
| 20 |
+
|
| 21 |
+
The file uses the `stereohand-joint-v1` format:
|
| 22 |
+
|
| 23 |
+
```python
|
| 24 |
+
import torch
|
| 25 |
+
checkpoint = torch.load(
|
| 26 |
+
"stereohand_pi3_mano110k_mask40k_matching_pi32.pt",
|
| 27 |
+
map_location="cpu", mmap=True, weights_only=True,
|
| 28 |
+
)
|
| 29 |
+
mask_state = checkpoint["states"]["mask"]
|
| 30 |
+
mano_state = checkpoint["states"]["mano"]
|
| 31 |
+
matching_state = checkpoint["states"]["matching"]
|
| 32 |
+
```
|
| 33 |
+
|
| 34 |
+
## Use with StereoHand
|
| 35 |
+
|
| 36 |
+
Set up the code repository and its runtime assets using the repository instructions. Download the checkpoint and the compatibility utility:
|
| 37 |
+
|
| 38 |
+
```python
|
| 39 |
+
from huggingface_hub import hf_hub_download
|
| 40 |
+
for name in ["stereohand_pi3_mano110k_mask40k_matching_pi32.pt", "unpack_checkpoint.py"]:
|
| 41 |
+
hf_hub_download("cskrren/StereoHand", name, local_dir="weights")
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
The current `SharedHeads` loader expects separate Mask, MANO, and matching paths. Export them from this single-file distribution and update an existing reconstruction config:
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
python weights/unpack_checkpoint.py \
|
| 48 |
+
weights/stereohand_pi3_mano110k_mask40k_matching_pi32.pt \
|
| 49 |
+
--output weights/unpacked \
|
| 50 |
+
--config /path/to/your/reconstruct.json
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
Use the resulting `weights/unpacked/reconstruct.json` with the repository's reconstruction command. Its runtime/source/asset paths come from your supplied config. The complete branch file is used for both initial and checkpoint inputs, which is compatible with the existing strict loader. The output directory must be new.
|
| 54 |
+
|
| 55 |
+
## Validation
|
| 56 |
+
|
| 57 |
+
The merge checks shared Pi3/stereo equality, matching-backbone compatibility (retaining the hand register token), finite tensor values, and exact tensor equality after saving and reloading. `VALIDATION.json` records the result. This validates checkpoint assembly; it is not a new accuracy benchmark for the 110k snapshot.
|
VALIDATION.json
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"complete": true,
|
| 3 |
+
"format": "stereohand-joint-v1",
|
| 4 |
+
"steps": {
|
| 5 |
+
"mask": {
|
| 6 |
+
"step": 40000,
|
| 7 |
+
"global_step": 40000,
|
| 8 |
+
"source_run": "stereohand_temporal_16gpu_20261105/runs/mask",
|
| 9 |
+
"delta_keys": 123
|
| 10 |
+
},
|
| 11 |
+
"mano": {
|
| 12 |
+
"step": 110000,
|
| 13 |
+
"global_step": 110000,
|
| 14 |
+
"source_run": "stereohand_temporal_16gpu_20261105/runs/mano",
|
| 15 |
+
"delta_keys": 605
|
| 16 |
+
}
|
| 17 |
+
},
|
| 18 |
+
"matching": "pi32",
|
| 19 |
+
"register_token_policy": "hand",
|
| 20 |
+
"deduplicated_tensors": 2038,
|
| 21 |
+
"bytes": 12035397828,
|
| 22 |
+
"tensor_equality_after_reload": true,
|
| 23 |
+
"finite_tensors": true,
|
| 24 |
+
"shared_backbone_verified": true,
|
| 25 |
+
"optimizer_included": false,
|
| 26 |
+
"created_at": 1794316894.7070687
|
| 27 |
+
}
|
stereohand_pi3_mano110k_mask40k_matching_pi32.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:88ee0fd5c77870b33ffc398f567566e619f869b38378322a23a875c74c1b90b4
|
| 3 |
+
size 12035397828
|
unpack_checkpoint.py
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Export a joint checkpoint to files accepted by StereoHand SharedHeads."""
|
| 2 |
+
import argparse,json
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
import torch
|
| 5 |
+
|
| 6 |
+
def unpack(checkpoint,output,model_config):
|
| 7 |
+
bundle=torch.load(checkpoint,map_location='cpu',mmap=True,weights_only=True)
|
| 8 |
+
if bundle.get('format')!='stereohand-joint-v1':raise ValueError('Unsupported checkpoint format')
|
| 9 |
+
root=Path(output).resolve();root.mkdir(parents=True,exist_ok=False)
|
| 10 |
+
for branch in ('mask','mano','matching'):
|
| 11 |
+
torch.save({'model':bundle['states'][branch]},root/(branch+'.pt'))
|
| 12 |
+
config=json.loads(Path(model_config).read_text())
|
| 13 |
+
model=config.get('model',config)
|
| 14 |
+
model.pop('inference_cache',None)
|
| 15 |
+
for branch in ('mask','mano'):
|
| 16 |
+
model[branch+'_initial']=model[branch+'_checkpoint']=str(root/(branch+'.pt'))
|
| 17 |
+
model[branch+'_step']=bundle['steps'][branch]['step']
|
| 18 |
+
model['matching_checkpoint']=str(root/'matching.pt')
|
| 19 |
+
target=root/'reconstruct.json';target.write_text(json.dumps(config,indent=2))
|
| 20 |
+
print(target)
|
| 21 |
+
if __name__=='__main__':
|
| 22 |
+
p=argparse.ArgumentParser();p.add_argument('checkpoint');p.add_argument('--output',required=True);p.add_argument('--config',required=True)
|
| 23 |
+
a=p.parse_args();unpack(a.checkpoint,a.output,a.config)
|