File size: 8,072 Bytes
4e4fa44
 
 
 
679eaab
 
156338f
 
 
679eaab
 
4e4fa44
 
7e001e5
 
7a6c582
 
7e001e5
 
 
 
 
 
 
 
 
 
 
 
 
 
5c8a180
7a6c582
7e001e5
156338f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7e001e5
679eaab
 
 
 
 
 
 
 
2de36a8
 
 
 
 
 
679eaab
 
 
 
 
 
741c4e4
679eaab
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7e001e5
85cf5cd
5c8a180
85cf5cd
679eaab
7a6c582
 
 
 
 
156338f
 
 
 
7a6c582
 
156338f
7a6c582
 
 
 
 
156338f
 
 
679eaab
 
 
 
7e001e5
 
 
 
5c8a180
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
---
license: other
license_name: sapiens2
license_link: LICENSE.md
tags:
- pose-estimation
- computer-vision
- whole-body-pose
- hand-pose
- safetensors
- delta-checkpoint
---

# S2HAND

**[Paper](https://arxiv.org/abs/2611.10416) · [Code v0.1.0-rc3](https://github.com/ronggong000/S2HAND/tree/v0.1.0-rc3)**

**Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose
Models**

Singyu Muk (Independent Researcher) · Xuanren Song (Tsinghua University) ·
Kalin Stefanov (Japan Advanced Institute of Science and Technology)

S2HAND specializes the hand predictions of Sapiens2 while preserving its native
308-keypoint whole-body interface. **S2HAND-MT** routes 40 adapted non-wrist
finger heatmaps through a copied pose decoder while preserving the shared
Transformer, auxiliary task heads, and other 268 pose heatmaps. **S2HAND-Pose**
additionally adapts three Fisher-selected Transformer blocks for stronger hand
recovery. The optional **TailGuard** policy conditionally re-runs uncertain hands
with a detected hand crop.

Source code and release documentation are maintained in the public
[`ronggong000/S2HAND`](https://github.com/ronggong000/S2HAND) repository.

**[Source code](https://github.com/ronggong000/S2HAND) ·
[H2SMV-FPV generator](https://huggingface.co/datasets/mage0001/H2SMV-FPV) ·
[Training protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/TRAINING.md) ·
[Evaluation protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/EVALUATION.md)**

<p align="center">
  <img src="assets/s2hand_overview.png" alt="S2HAND method overview" width="100%">
</p>

## Model summary

| Need | Model |
|---|---|
| Exact public routing for 268 non-finger heatmaps and unchanged auxiliary heads | `S2HAND-MT-5B` |
| Best released hand accuracy | `S2HAND-Pose-5B` |
| Smaller whole-body pose model | `S2HAND-Pose-0.4B`, `0.8B`, or `1B` |

Native-pipeline AUC@0.20 (%; TailGuard off):

| Model | NewDays TEST | VISOR TEST | H2O TEST | EPICK VAL | H2SMV-FPV TEST |
|---|---:|---:|---:|---:|---:|
| Public Sapiens2-5B | 61.48 | 56.12 | 69.02 | 55.16 | 47.93 |
| S2HAND-MT-5B | 62.57 | 59.42 | 70.79 | 58.54 | 49.01 |
| **S2HAND-Pose-5B** | **70.04** | **74.88** | **73.96** | **74.35** | **56.14** |

Each row uses its deployable detector/crop pipeline; missed hands count as
failures. EPICK is a validation split and H2SMV-FPV is synthetic. See the source
repository for strict metric definitions and cross-model comparisons.

## Automatic base + delta loading

Five final model checkpoints and their loading script live in this repository.
No GitHub checkout or local research environment is required. The loader downloads
an exact official Sapiens2 pose base, verifies its SHA256, downloads the selected
S2HAND delta, and applies replacement tensors with the final protected decoder route.
Downloads use the Hugging Face cache; models sharing a 5B base reuse that same file.
It does not write another full merged checkpoint.

The repository also contains `training/S2HAND-Training-WarmStart-5B`, a compact
base delta used only to reproduce the common 5B training initialization and
the diagonal-Fisher block probe. It is **not** a deployment model. The final
S2HAND-MT-5B and S2HAND-Pose-5B checkpoints branch independently from this
common warm-start; neither final checkpoint is initialized from the other.

## Quick start

Python 3.12+ and PyTorch 2.7+:

```bash
pip install torch safetensors huggingface_hub
# S2HAND files are public; no login is required to download this repository.
hf download mage0001/S2HAND s2hand.py sapiens2_backbone.py --local-dir S2HAND
cd S2HAND
python s2hand.py --model S2HAND-MT-5B --device cuda
```

The last command downloads, checks and loads the model, then exits if no input is
provided. To download weights without constructing the model, add `--download-only`.
The official backbone source is bundled under the included Sapiens2 License.

## Use in an existing pipeline

```python
import torch
from s2hand import load_model

model = load_model("S2HAND-Pose-0.4B", device="cuda")
# Your existing person-crop pipeline supplies RGB tensors [B,3,1024,768].
# Normalize RGB values in [0,1] with ImageNet mean/std:
# mean = [0.485,0.456,0.406], std = [0.229,0.224,0.225].
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    heatmaps = model(normalized_rgb)  # [B,308,256,192]
```

For a command-line tensor input, save your normalized RGB batch as safetensors
under key `images`, then run:

```bash
python s2hand.py --model S2HAND-Pose-0.4B --device cuda --bf16   --input-tensor crops.safetensors --output heatmaps.safetensors
```

This is the model loader and heatmap forward pass. Detection, person-crop affine
transforms, UDP coordinate decoding, optional flip testing and TailGuard remain
steps in the caller's pipeline. Matching paper scores requires the corresponding
paper preprocessing and evaluation protocol; the loader does not substitute a
new image-resize or detector protocol.

## Models

| Name | Learned Transformer blocks, zero-based | Delta size |
|---|---|---|
| S2HAND-MT-5B | None; complete copied pose decoder | 3.31 MiB |
| S2HAND-Pose-5B | 38,39,40 plus decoder | 1019.04 MiB |
| S2HAND-Pose-1B | 28,29,30 plus decoder | 410.58 MiB |
| S2HAND-Pose-0.8B | 17,18,19 plus decoder | 284.78 MiB |
| S2HAND-Pose-0.4B | 16,17,18 plus decoder | 195.50 MiB |

The smaller Pose models use their own same-size official bases. They are not
compressed versions of the 5B checkpoint. The two 5B variants share one cached
base, which alone is approximately 19.07 GiB; the delta size is not total model
memory. Parameters remain FP32, matching the research runtime; `--bf16` enables
inference autocast. Meta initialization avoids allocating a second random model.

## Exact routing and versions

The public decoder is copied before applying the delta. Only the 40 non-wrist
finger channels use the adapted decoder. The other 268 channels, including wrist
indices 41 and 62, use the frozen public decoder. Pose variants share their adapted
backbone across both decoder routes; exact preservation of protected outputs is
not guaranteed for those variants. MT does not update learned Transformer weights.
The loader preserves the original BF16 RoPE runtime buffer semantics, including
the explicit rebased 5B override. A generic base+delta load without this routing
is insufficient to reproduce the final model interface.

Official base revisions and the S2HAND weight revision are pinned in the script.
Per-model manifests record both SHA256 values and replacement tensor lists. Keep
the downloaded base and delta in the cache to reuse them on subsequent runs.
`--cache-dir` controls the cache location. `--base-path`, `--delta-path`,
`--manifest-path` and `--offline` support existing local files without downloading
another base. For completely offline use, keep `sapiens2_backbone.py` beside the script.

See `VALIDATION.md` for the checks performed before publishing this loader.
The verified standalone loader is included here. Training, evaluation, TailGuard,
and dataset tooling are maintained in the accompanying source repository. The
public source release candidate `v0.1.0-rc3` includes the complete clean HInt
validation replay and end-to-end TailGuard command.

## Use and license

Top-down pose estimation for non-commercial research under the included
[Sapiens2 License](LICENSE.md). Use the documented detector/crop protocol;
TailGuard is an optional external policy.

## Citation

```bibtex
@misc{muk2026s2hand,
  title = {S2HAND: Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models},
  author = {Muk, Singyu and Song, Xuanren and Stefanov, Kalin},
  year = {2026},
  eprint = {2611.10416},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2611.10416}
}
```

## Attribution

Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived
model artifacts are distributed under the included Sapiens2 License.

## Contact

Singyu Muk — `ronggong000@gmail.com`