--- license: other license_name: sapiens2 license_link: LICENSE.md tags: - pose-estimation - computer-vision - whole-body-pose - hand-pose - safetensors - delta-checkpoint --- # S2HAND **[Paper](https://arxiv.org/abs/2611.10416) · [Code v0.1.0-rc3](https://github.com/ronggong000/S2HAND/tree/v0.1.0-rc3)** **Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models** Singyu Muk (Independent Researcher) · Xuanren Song (Tsinghua University) · Kalin Stefanov (Japan Advanced Institute of Science and Technology) S2HAND specializes the hand predictions of Sapiens2 while preserving its native 308-keypoint whole-body interface. **S2HAND-MT** routes 40 adapted non-wrist finger heatmaps through a copied pose decoder while preserving the shared Transformer, auxiliary task heads, and other 268 pose heatmaps. **S2HAND-Pose** additionally adapts three Fisher-selected Transformer blocks for stronger hand recovery. The optional **TailGuard** policy conditionally re-runs uncertain hands with a detected hand crop. Source code and release documentation are maintained in the public [`ronggong000/S2HAND`](https://github.com/ronggong000/S2HAND) repository. **[Source code](https://github.com/ronggong000/S2HAND) · [H2SMV-FPV generator](https://huggingface.co/datasets/mage0001/H2SMV-FPV) · [Training protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/TRAINING.md) · [Evaluation protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/EVALUATION.md)**

S2HAND method overview

## Model summary | Need | Model | |---|---| | Exact public routing for 268 non-finger heatmaps and unchanged auxiliary heads | `S2HAND-MT-5B` | | Best released hand accuracy | `S2HAND-Pose-5B` | | Smaller whole-body pose model | `S2HAND-Pose-0.4B`, `0.8B`, or `1B` | Native-pipeline AUC@0.20 (%; TailGuard off): | Model | NewDays TEST | VISOR TEST | H2O TEST | EPICK VAL | H2SMV-FPV TEST | |---|---:|---:|---:|---:|---:| | Public Sapiens2-5B | 61.48 | 56.12 | 69.02 | 55.16 | 47.93 | | S2HAND-MT-5B | 62.57 | 59.42 | 70.79 | 58.54 | 49.01 | | **S2HAND-Pose-5B** | **70.04** | **74.88** | **73.96** | **74.35** | **56.14** | Each row uses its deployable detector/crop pipeline; missed hands count as failures. EPICK is a validation split and H2SMV-FPV is synthetic. See the source repository for strict metric definitions and cross-model comparisons. ## Automatic base + delta loading Five final model checkpoints and their loading script live in this repository. No GitHub checkout or local research environment is required. The loader downloads an exact official Sapiens2 pose base, verifies its SHA256, downloads the selected S2HAND delta, and applies replacement tensors with the final protected decoder route. Downloads use the Hugging Face cache; models sharing a 5B base reuse that same file. It does not write another full merged checkpoint. The repository also contains `training/S2HAND-Training-WarmStart-5B`, a compact base delta used only to reproduce the common 5B training initialization and the diagonal-Fisher block probe. It is **not** a deployment model. The final S2HAND-MT-5B and S2HAND-Pose-5B checkpoints branch independently from this common warm-start; neither final checkpoint is initialized from the other. ## Quick start Python 3.12+ and PyTorch 2.7+: ```bash pip install torch safetensors huggingface_hub # S2HAND files are public; no login is required to download this repository. hf download mage0001/S2HAND s2hand.py sapiens2_backbone.py --local-dir S2HAND cd S2HAND python s2hand.py --model S2HAND-MT-5B --device cuda ``` The last command downloads, checks and loads the model, then exits if no input is provided. To download weights without constructing the model, add `--download-only`. The official backbone source is bundled under the included Sapiens2 License. ## Use in an existing pipeline ```python import torch from s2hand import load_model model = load_model("S2HAND-Pose-0.4B", device="cuda") # Your existing person-crop pipeline supplies RGB tensors [B,3,1024,768]. # Normalize RGB values in [0,1] with ImageNet mean/std: # mean = [0.485,0.456,0.406], std = [0.229,0.224,0.225]. with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): heatmaps = model(normalized_rgb) # [B,308,256,192] ``` For a command-line tensor input, save your normalized RGB batch as safetensors under key `images`, then run: ```bash python s2hand.py --model S2HAND-Pose-0.4B --device cuda --bf16 --input-tensor crops.safetensors --output heatmaps.safetensors ``` This is the model loader and heatmap forward pass. Detection, person-crop affine transforms, UDP coordinate decoding, optional flip testing and TailGuard remain steps in the caller's pipeline. Matching paper scores requires the corresponding paper preprocessing and evaluation protocol; the loader does not substitute a new image-resize or detector protocol. ## Models | Name | Learned Transformer blocks, zero-based | Delta size | |---|---|---| | S2HAND-MT-5B | None; complete copied pose decoder | 3.31 MiB | | S2HAND-Pose-5B | 38,39,40 plus decoder | 1019.04 MiB | | S2HAND-Pose-1B | 28,29,30 plus decoder | 410.58 MiB | | S2HAND-Pose-0.8B | 17,18,19 plus decoder | 284.78 MiB | | S2HAND-Pose-0.4B | 16,17,18 plus decoder | 195.50 MiB | The smaller Pose models use their own same-size official bases. They are not compressed versions of the 5B checkpoint. The two 5B variants share one cached base, which alone is approximately 19.07 GiB; the delta size is not total model memory. Parameters remain FP32, matching the research runtime; `--bf16` enables inference autocast. Meta initialization avoids allocating a second random model. ## Exact routing and versions The public decoder is copied before applying the delta. Only the 40 non-wrist finger channels use the adapted decoder. The other 268 channels, including wrist indices 41 and 62, use the frozen public decoder. Pose variants share their adapted backbone across both decoder routes; exact preservation of protected outputs is not guaranteed for those variants. MT does not update learned Transformer weights. The loader preserves the original BF16 RoPE runtime buffer semantics, including the explicit rebased 5B override. A generic base+delta load without this routing is insufficient to reproduce the final model interface. Official base revisions and the S2HAND weight revision are pinned in the script. Per-model manifests record both SHA256 values and replacement tensor lists. Keep the downloaded base and delta in the cache to reuse them on subsequent runs. `--cache-dir` controls the cache location. `--base-path`, `--delta-path`, `--manifest-path` and `--offline` support existing local files without downloading another base. For completely offline use, keep `sapiens2_backbone.py` beside the script. See `VALIDATION.md` for the checks performed before publishing this loader. The verified standalone loader is included here. Training, evaluation, TailGuard, and dataset tooling are maintained in the accompanying source repository. The public source release candidate `v0.1.0-rc3` includes the complete clean HInt validation replay and end-to-end TailGuard command. ## Use and license Top-down pose estimation for non-commercial research under the included [Sapiens2 License](LICENSE.md). Use the documented detector/crop protocol; TailGuard is an optional external policy. ## Citation ```bibtex @misc{muk2026s2hand, title = {S2HAND: Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models}, author = {Muk, Singyu and Song, Xuanren and Stefanov, Kalin}, year = {2026}, eprint = {2611.10416}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2611.10416} } ``` ## Attribution Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived model artifacts are distributed under the included Sapiens2 License. ## Contact Singyu Muk — `ronggong000@gmail.com`