Polish model card, project navigation, results, and validation
Browse files- .gitattributes +1 -0
- README.md +60 -1
- VALIDATION.md +57 -17
- assets/s2hand_overview.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
assets/s2hand_overview.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -4,6 +4,9 @@ license_name: sapiens2
|
|
| 4 |
license_link: LICENSE.md
|
| 5 |
tags:
|
| 6 |
- pose-estimation
|
|
|
|
|
|
|
|
|
|
| 7 |
- safetensors
|
| 8 |
- delta-checkpoint
|
| 9 |
---
|
|
@@ -28,6 +31,35 @@ Source code and release documentation are maintained in the private
|
|
| 28 |
[`ronggong000/S2HAND`](https://github.com/ronggong000/S2HAND) repository while
|
| 29 |
the accompanying manuscript is under review.
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
## Automatic base + delta loading
|
| 32 |
|
| 33 |
Five final model checkpoints and their loading script live in this repository.
|
|
@@ -124,9 +156,36 @@ another base. For completely offline use, keep `sapiens2_backbone.py` beside the
|
|
| 124 |
See `VALIDATION.md` for the checks performed before publishing this loader.
|
| 125 |
The verified standalone loader is included here. Training, evaluation, TailGuard,
|
| 126 |
and dataset tooling are maintained in the accompanying source repository. The
|
| 127 |
-
private source release candidate `v0.1.0-
|
| 128 |
validation replay and end-to-end TailGuard command.
|
| 129 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
## Attribution
|
| 131 |
|
| 132 |
Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived
|
|
|
|
| 4 |
license_link: LICENSE.md
|
| 5 |
tags:
|
| 6 |
- pose-estimation
|
| 7 |
+
- computer-vision
|
| 8 |
+
- whole-body-pose
|
| 9 |
+
- hand-pose
|
| 10 |
- safetensors
|
| 11 |
- delta-checkpoint
|
| 12 |
---
|
|
|
|
| 31 |
[`ronggong000/S2HAND`](https://github.com/ronggong000/S2HAND) repository while
|
| 32 |
the accompanying manuscript is under review.
|
| 33 |
|
| 34 |
+
**[Source code](https://github.com/ronggong000/S2HAND) ·
|
| 35 |
+
[H2SMV-FPV generator](https://huggingface.co/datasets/mage0001/H2SMV-FPV) ·
|
| 36 |
+
[Training protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/TRAINING.md) ·
|
| 37 |
+
[Evaluation protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/EVALUATION.md)**
|
| 38 |
+
|
| 39 |
+
<p align="center">
|
| 40 |
+
<img src="assets/s2hand_overview.png" alt="S2HAND method overview" width="100%">
|
| 41 |
+
</p>
|
| 42 |
+
|
| 43 |
+
## Model summary
|
| 44 |
+
|
| 45 |
+
| Need | Model |
|
| 46 |
+
|---|---|
|
| 47 |
+
| Exact public routing for 268 non-finger heatmaps and unchanged auxiliary heads | `S2HAND-MT-5B` |
|
| 48 |
+
| Best released hand accuracy | `S2HAND-Pose-5B` |
|
| 49 |
+
| Smaller whole-body pose model | `S2HAND-Pose-0.4B`, `0.8B`, or `1B` |
|
| 50 |
+
|
| 51 |
+
Native-pipeline AUC@0.20 (%; TailGuard off):
|
| 52 |
+
|
| 53 |
+
| Model | NewDays TEST | VISOR TEST | H2O TEST | EPICK VAL | H2SMV-FPV TEST |
|
| 54 |
+
|---|---:|---:|---:|---:|---:|
|
| 55 |
+
| Public Sapiens2-5B | 61.48 | 56.12 | 69.02 | 55.16 | 47.93 |
|
| 56 |
+
| S2HAND-MT-5B | 62.57 | 59.42 | 70.79 | 58.54 | 49.01 |
|
| 57 |
+
| **S2HAND-Pose-5B** | **70.04** | **74.88** | **73.96** | **74.35** | **56.14** |
|
| 58 |
+
|
| 59 |
+
Each row uses its deployable detector/crop pipeline; missed hands count as
|
| 60 |
+
failures. EPICK is a validation split and H2SMV-FPV is synthetic. See the source
|
| 61 |
+
repository for strict metric definitions and cross-model comparisons.
|
| 62 |
+
|
| 63 |
## Automatic base + delta loading
|
| 64 |
|
| 65 |
Five final model checkpoints and their loading script live in this repository.
|
|
|
|
| 156 |
See `VALIDATION.md` for the checks performed before publishing this loader.
|
| 157 |
The verified standalone loader is included here. Training, evaluation, TailGuard,
|
| 158 |
and dataset tooling are maintained in the accompanying source repository. The
|
| 159 |
+
private source release candidate `v0.1.0-rc3` includes the complete clean HInt
|
| 160 |
validation replay and end-to-end TailGuard command.
|
| 161 |
|
| 162 |
+
## Intended use and limitations
|
| 163 |
+
|
| 164 |
+
- Intended for non-commercial research on whole-body and hand pose estimation,
|
| 165 |
+
subject to the included Sapiens2 license.
|
| 166 |
+
- S2HAND remains a top-down estimator. Native-pipeline accuracy depends on the
|
| 167 |
+
person/hand detector and crop protocol; a checkpoint result is not comparable
|
| 168 |
+
across undeclared crop settings.
|
| 169 |
+
- S2HAND-MT gives an architectural exact-route guarantee for protected pose
|
| 170 |
+
heatmaps and leaves the shared backbone/auxiliary heads unchanged.
|
| 171 |
+
S2HAND-Pose updates shared Transformer blocks and therefore provides measured,
|
| 172 |
+
not exact, non-hand preservation.
|
| 173 |
+
- TailGuard requires an external hand detector and adds conditional inference;
|
| 174 |
+
it is not baked into these weights.
|
| 175 |
+
- The model can fail under severe blur, truncation, occlusion, side ambiguity,
|
| 176 |
+
unusual optics, or detector failure. Do not use it for safety-critical or
|
| 177 |
+
identity-sensitive decisions.
|
| 178 |
+
|
| 179 |
+
## Citation
|
| 180 |
+
|
| 181 |
+
```bibtex
|
| 182 |
+
@misc{muk2027s2hand,
|
| 183 |
+
title = {S2HAND: Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models},
|
| 184 |
+
author = {Muk, Singyu and Song, Xuanren and Stefanov, Kalin},
|
| 185 |
+
year = {2027}
|
| 186 |
+
}
|
| 187 |
+
```
|
| 188 |
+
|
| 189 |
## Attribution
|
| 190 |
|
| 191 |
Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived
|
VALIDATION.md
CHANGED
|
@@ -1,5 +1,14 @@
|
|
| 1 |
# Loader validation
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
Before upload, all five models passed:
|
| 4 |
|
| 5 |
- Exact base/delta SHA256 verification against their manifests.
|
|
@@ -25,20 +34,51 @@ The standalone image pipeline is also covered by unit tests for:
|
|
| 25 |
- RGB/ImageNet input tensor shape and finite normalization; and
|
| 26 |
- 308-channel UDP/DARK decoding back to original-image coordinates.
|
| 27 |
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Loader validation
|
| 2 |
|
| 3 |
+
## Source-package checks (2026-10-30)
|
| 4 |
+
|
| 5 |
+
- All 28 unit tests pass under the release source tree.
|
| 6 |
+
- Metric integration uses `numpy.trapz`, matching the declared NumPy >=1.26
|
| 7 |
+
requirement as well as newer NumPy releases.
|
| 8 |
+
- All local Markdown links resolve.
|
| 9 |
+
- No checkpoint, dataset, or file larger than 10 MiB is tracked by GitHub;
|
| 10 |
+
deployment deltas remain in the Hugging Face model repository.
|
| 11 |
+
|
| 12 |
Before upload, all five models passed:
|
| 13 |
|
| 14 |
- Exact base/delta SHA256 verification against their manifests.
|
|
|
|
| 34 |
- RGB/ImageNet input tensor shape and finite normalization; and
|
| 35 |
- 308-channel UDP/DARK decoding back to original-image coordinates.
|
| 36 |
|
| 37 |
+
The protocol-locked evaluator was also compared with the NewDays paper closure
|
| 38 |
+
aggregate for S2HAND-Pose. Hand count, evaluated-joint count, recall,
|
| 39 |
+
PCK@.05/.10/.15/.20, AUC@.20, severe-tail rate, and finite-hand NME all matched
|
| 40 |
+
exactly.
|
| 41 |
+
|
| 42 |
+
The standalone training adapter's UDP targets were compared element-by-element
|
| 43 |
+
with the original research encoder on randomized in-frame and outside-frame
|
| 44 |
+
points; target weights and all heatmap values were exactly equal. Unit tests
|
| 45 |
+
also cover the 42-channel layout, visibility handling, and protected-route
|
| 46 |
+
gradient stop.
|
| 47 |
+
|
| 48 |
+
On 2026-10-30, the standalone Pose-5B trainer passed a real one-step L40S/BF16
|
| 49 |
+
mechanics test from the pinned training warm-start. It completed canonical
|
| 50 |
+
NewDays/EPICK validation, a forward/backward update of blocks 38--40 and the
|
| 51 |
+
adapted decoder, gradient clipping, best/final checkpoint serialization, and
|
| 52 |
+
manifest hashing. The 1,068,545,694-byte final delta contained exactly the 60
|
| 53 |
+
expected tensors. A separate process then loaded that generated delta over the
|
| 54 |
+
pinned official base with `load_training_checkpoint` and reproduced finite
|
| 55 |
+
canonical validation metrics. The two-hand diagnostic slice is a mechanics
|
| 56 |
+
test only and is not reported as a paper benchmark result.
|
| 57 |
+
|
| 58 |
+
The released S2HAND-MT-5B route was also checked with its verified final delta
|
| 59 |
+
and a real HInt validation image. All 268 protected heatmaps were bitwise equal
|
| 60 |
+
to the frozen public decoder route (maximum absolute difference `0.0`), while
|
| 61 |
+
the 40 adapted finger heatmaps differed from that route. The ordinary model
|
| 62 |
+
forward was bitwise equal to an independently assembled explicit 40/268 route.
|
| 63 |
+
|
| 64 |
+
The clean checkout also replayed the complete 1,175-hand HInt NewDays/EPICK
|
| 65 |
+
validation split for the released S2HAND-Pose-5B delta. The catastrophic-tail
|
| 66 |
+
rate matched the original research record exactly (`0.0144680851`). Selection
|
| 67 |
+
AUC was `0.7656657143`, versus `0.7656003214` in the original record (absolute
|
| 68 |
+
difference `0.0000653929`), and combined NME differed by `0.0000097603`.
|
| 69 |
+
This is a near-exact protocol replay rather than a bitwise claim: the largest
|
| 70 |
+
single metric difference was `0.0019462` for a discontinuous occluded-joint PCK
|
| 71 |
+
threshold count.
|
| 72 |
+
|
| 73 |
+
Finally, after an editable clean-checkout installation, the documented
|
| 74 |
+
inference, hand-evaluation, Fisher-probe, and training CLIs all parsed their
|
| 75 |
+
published command-line interfaces successfully. All relative Markdown links in
|
| 76 |
+
the README, validation record, and `docs/` resolve to existing files.
|
| 77 |
+
|
| 78 |
+
TailGuard was then exercised end to end with the released S2HAND-MT-5B delta,
|
| 79 |
+
real NewDays images, cached person detections, and cached WiLoR-YOLO left/right
|
| 80 |
+
hand detections. Across 8 images (16 hand rows), 11 rows had a matching detector
|
| 81 |
+
box and 4 crossed the frozen confidence threshold. All 12 non-triggered rows
|
| 82 |
+
were bitwise equal to their native predictions; all 4 triggered rows changed
|
| 83 |
+
and used exactly 4 additional pose forwards. This is an implementation smoke
|
| 84 |
+
test, not an additional benchmark result.
|
assets/s2hand_overview.png
ADDED
|
Git LFS Details
|