|
Download README.md from mage0001/S2HAND: direct link, hf CLI and curl.
- Browser
- Download file 8.07 kB
-
https://huggingface.co/mage0001/S2HAND/resolve/main/README.md
- Command line
-
hf download hf://mage0001/S2HAND/README.md
-
curl -L -o README.md https://huggingface.co/mage0001/S2HAND/resolve/main/README.md
8.07 kB
| license: other | |
| license_name: sapiens2 | |
| license_link: LICENSE.md | |
| tags: | |
| - pose-estimation | |
| - computer-vision | |
| - whole-body-pose | |
| - hand-pose | |
| - safetensors | |
| - delta-checkpoint | |
| # S2HAND | |
| **[Paper](https://arxiv.org/abs/2611.10416) · [Code v0.1.0-rc3](https://github.com/ronggong000/S2HAND/tree/v0.1.0-rc3)** | |
| **Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose | |
| Models** | |
| Singyu Muk (Independent Researcher) · Xuanren Song (Tsinghua University) · | |
| Kalin Stefanov (Japan Advanced Institute of Science and Technology) | |
| S2HAND specializes the hand predictions of Sapiens2 while preserving its native | |
| 308-keypoint whole-body interface. **S2HAND-MT** routes 40 adapted non-wrist | |
| finger heatmaps through a copied pose decoder while preserving the shared | |
| Transformer, auxiliary task heads, and other 268 pose heatmaps. **S2HAND-Pose** | |
| additionally adapts three Fisher-selected Transformer blocks for stronger hand | |
| recovery. The optional **TailGuard** policy conditionally re-runs uncertain hands | |
| with a detected hand crop. | |
| Source code and release documentation are maintained in the public | |
| [`ronggong000/S2HAND`](https://github.com/ronggong000/S2HAND) repository. | |
| **[Source code](https://github.com/ronggong000/S2HAND) · | |
| [H2SMV-FPV generator](https://huggingface.co/datasets/mage0001/H2SMV-FPV) · | |
| [Training protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/TRAINING.md) · | |
| [Evaluation protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/EVALUATION.md)** | |
| <p align="center"> | |
| <img src="assets/s2hand_overview.png" alt="S2HAND method overview" width="100%"> | |
| </p> | |
| ## Model summary | |
| | Need | Model | | |
| |---|---| | |
| | Exact public routing for 268 non-finger heatmaps and unchanged auxiliary heads | `S2HAND-MT-5B` | | |
| | Best released hand accuracy | `S2HAND-Pose-5B` | | |
| | Smaller whole-body pose model | `S2HAND-Pose-0.4B`, `0.8B`, or `1B` | | |
| Native-pipeline AUC@0.20 (%; TailGuard off): | |
| | Model | NewDays TEST | VISOR TEST | H2O TEST | EPICK VAL | H2SMV-FPV TEST | | |
| |---|---:|---:|---:|---:|---:| | |
| | Public Sapiens2-5B | 61.48 | 56.12 | 69.02 | 55.16 | 47.93 | | |
| | S2HAND-MT-5B | 62.57 | 59.42 | 70.79 | 58.54 | 49.01 | | |
| | **S2HAND-Pose-5B** | **70.04** | **74.88** | **73.96** | **74.35** | **56.14** | | |
| Each row uses its deployable detector/crop pipeline; missed hands count as | |
| failures. EPICK is a validation split and H2SMV-FPV is synthetic. See the source | |
| repository for strict metric definitions and cross-model comparisons. | |
| ## Automatic base + delta loading | |
| Five final model checkpoints and their loading script live in this repository. | |
| No GitHub checkout or local research environment is required. The loader downloads | |
| an exact official Sapiens2 pose base, verifies its SHA256, downloads the selected | |
| S2HAND delta, and applies replacement tensors with the final protected decoder route. | |
| Downloads use the Hugging Face cache; models sharing a 5B base reuse that same file. | |
| It does not write another full merged checkpoint. | |
| The repository also contains `training/S2HAND-Training-WarmStart-5B`, a compact | |
| base delta used only to reproduce the common 5B training initialization and | |
| the diagonal-Fisher block probe. It is **not** a deployment model. The final | |
| S2HAND-MT-5B and S2HAND-Pose-5B checkpoints branch independently from this | |
| common warm-start; neither final checkpoint is initialized from the other. | |
| ## Quick start | |
| Python 3.12+ and PyTorch 2.7+: | |
| ```bash | |
| pip install torch safetensors huggingface_hub | |
| # S2HAND files are public; no login is required to download this repository. | |
| hf download mage0001/S2HAND s2hand.py sapiens2_backbone.py --local-dir S2HAND | |
| cd S2HAND | |
| python s2hand.py --model S2HAND-MT-5B --device cuda | |
| ``` | |
| The last command downloads, checks and loads the model, then exits if no input is | |
| provided. To download weights without constructing the model, add `--download-only`. | |
| The official backbone source is bundled under the included Sapiens2 License. | |
| ## Use in an existing pipeline | |
| ```python | |
| import torch | |
| from s2hand import load_model | |
| model = load_model("S2HAND-Pose-0.4B", device="cuda") | |
| # Your existing person-crop pipeline supplies RGB tensors [B,3,1024,768]. | |
| # Normalize RGB values in [0,1] with ImageNet mean/std: | |
| # mean = [0.485,0.456,0.406], std = [0.229,0.224,0.225]. | |
| with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): | |
| heatmaps = model(normalized_rgb) # [B,308,256,192] | |
| ``` | |
| For a command-line tensor input, save your normalized RGB batch as safetensors | |
| under key `images`, then run: | |
| ```bash | |
| python s2hand.py --model S2HAND-Pose-0.4B --device cuda --bf16 --input-tensor crops.safetensors --output heatmaps.safetensors | |
| ``` | |
| This is the model loader and heatmap forward pass. Detection, person-crop affine | |
| transforms, UDP coordinate decoding, optional flip testing and TailGuard remain | |
| steps in the caller's pipeline. Matching paper scores requires the corresponding | |
| paper preprocessing and evaluation protocol; the loader does not substitute a | |
| new image-resize or detector protocol. | |
| ## Models | |
| | Name | Learned Transformer blocks, zero-based | Delta size | | |
| |---|---|---| | |
| | S2HAND-MT-5B | None; complete copied pose decoder | 3.31 MiB | | |
| | S2HAND-Pose-5B | 38,39,40 plus decoder | 1019.04 MiB | | |
| | S2HAND-Pose-1B | 28,29,30 plus decoder | 410.58 MiB | | |
| | S2HAND-Pose-0.8B | 17,18,19 plus decoder | 284.78 MiB | | |
| | S2HAND-Pose-0.4B | 16,17,18 plus decoder | 195.50 MiB | | |
| The smaller Pose models use their own same-size official bases. They are not | |
| compressed versions of the 5B checkpoint. The two 5B variants share one cached | |
| base, which alone is approximately 19.07 GiB; the delta size is not total model | |
| memory. Parameters remain FP32, matching the research runtime; `--bf16` enables | |
| inference autocast. Meta initialization avoids allocating a second random model. | |
| ## Exact routing and versions | |
| The public decoder is copied before applying the delta. Only the 40 non-wrist | |
| finger channels use the adapted decoder. The other 268 channels, including wrist | |
| indices 41 and 62, use the frozen public decoder. Pose variants share their adapted | |
| backbone across both decoder routes; exact preservation of protected outputs is | |
| not guaranteed for those variants. MT does not update learned Transformer weights. | |
| The loader preserves the original BF16 RoPE runtime buffer semantics, including | |
| the explicit rebased 5B override. A generic base+delta load without this routing | |
| is insufficient to reproduce the final model interface. | |
| Official base revisions and the S2HAND weight revision are pinned in the script. | |
| Per-model manifests record both SHA256 values and replacement tensor lists. Keep | |
| the downloaded base and delta in the cache to reuse them on subsequent runs. | |
| `--cache-dir` controls the cache location. `--base-path`, `--delta-path`, | |
| `--manifest-path` and `--offline` support existing local files without downloading | |
| another base. For completely offline use, keep `sapiens2_backbone.py` beside the script. | |
| See `VALIDATION.md` for the checks performed before publishing this loader. | |
| The verified standalone loader is included here. Training, evaluation, TailGuard, | |
| and dataset tooling are maintained in the accompanying source repository. The | |
| public source release candidate `v0.1.0-rc3` includes the complete clean HInt | |
| validation replay and end-to-end TailGuard command. | |
| ## Use and license | |
| Top-down pose estimation for non-commercial research under the included | |
| [Sapiens2 License](LICENSE.md). Use the documented detector/crop protocol; | |
| TailGuard is an optional external policy. | |
| ## Citation | |
| ```bibtex | |
| @misc{muk2026s2hand, | |
| title = {S2HAND: Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models}, | |
| author = {Muk, Singyu and Song, Xuanren and Stefanov, Kalin}, | |
| year = {2026}, | |
| eprint = {2611.10416}, | |
| archivePrefix = {arXiv}, | |
| primaryClass = {cs.CV}, | |
| url = {https://arxiv.org/abs/2611.10416} | |
| } | |
| ``` | |
| ## Attribution | |
| Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived | |
| model artifacts are distributed under the included Sapiens2 License. | |
| ## Contact | |
| Singyu Muk — `ronggong000@gmail.com` | |