mage0001 commited on
Commit
156338f
·
verified ·
1 Parent(s): 85cf5cd

Polish model card, project navigation, results, and validation

Browse files
Files changed (4) hide show
  1. .gitattributes +1 -0
  2. README.md +60 -1
  3. VALIDATION.md +57 -17
  4. assets/s2hand_overview.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/s2hand_overview.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -4,6 +4,9 @@ license_name: sapiens2
4
  license_link: LICENSE.md
5
  tags:
6
  - pose-estimation
 
 
 
7
  - safetensors
8
  - delta-checkpoint
9
  ---
@@ -28,6 +31,35 @@ Source code and release documentation are maintained in the private
28
  [`ronggong000/S2HAND`](https://github.com/ronggong000/S2HAND) repository while
29
  the accompanying manuscript is under review.
30
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  ## Automatic base + delta loading
32
 
33
  Five final model checkpoints and their loading script live in this repository.
@@ -124,9 +156,36 @@ another base. For completely offline use, keep `sapiens2_backbone.py` beside the
124
  See `VALIDATION.md` for the checks performed before publishing this loader.
125
  The verified standalone loader is included here. Training, evaluation, TailGuard,
126
  and dataset tooling are maintained in the accompanying source repository. The
127
- private source release candidate `v0.1.0-rc2` includes the complete clean HInt
128
  validation replay and end-to-end TailGuard command.
129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  ## Attribution
131
 
132
  Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived
 
4
  license_link: LICENSE.md
5
  tags:
6
  - pose-estimation
7
+ - computer-vision
8
+ - whole-body-pose
9
+ - hand-pose
10
  - safetensors
11
  - delta-checkpoint
12
  ---
 
31
  [`ronggong000/S2HAND`](https://github.com/ronggong000/S2HAND) repository while
32
  the accompanying manuscript is under review.
33
 
34
+ **[Source code](https://github.com/ronggong000/S2HAND) ·
35
+ [H2SMV-FPV generator](https://huggingface.co/datasets/mage0001/H2SMV-FPV) ·
36
+ [Training protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/TRAINING.md) ·
37
+ [Evaluation protocol](https://github.com/ronggong000/S2HAND/blob/main/docs/EVALUATION.md)**
38
+
39
+ <p align="center">
40
+ <img src="assets/s2hand_overview.png" alt="S2HAND method overview" width="100%">
41
+ </p>
42
+
43
+ ## Model summary
44
+
45
+ | Need | Model |
46
+ |---|---|
47
+ | Exact public routing for 268 non-finger heatmaps and unchanged auxiliary heads | `S2HAND-MT-5B` |
48
+ | Best released hand accuracy | `S2HAND-Pose-5B` |
49
+ | Smaller whole-body pose model | `S2HAND-Pose-0.4B`, `0.8B`, or `1B` |
50
+
51
+ Native-pipeline AUC@0.20 (%; TailGuard off):
52
+
53
+ | Model | NewDays TEST | VISOR TEST | H2O TEST | EPICK VAL | H2SMV-FPV TEST |
54
+ |---|---:|---:|---:|---:|---:|
55
+ | Public Sapiens2-5B | 61.48 | 56.12 | 69.02 | 55.16 | 47.93 |
56
+ | S2HAND-MT-5B | 62.57 | 59.42 | 70.79 | 58.54 | 49.01 |
57
+ | **S2HAND-Pose-5B** | **70.04** | **74.88** | **73.96** | **74.35** | **56.14** |
58
+
59
+ Each row uses its deployable detector/crop pipeline; missed hands count as
60
+ failures. EPICK is a validation split and H2SMV-FPV is synthetic. See the source
61
+ repository for strict metric definitions and cross-model comparisons.
62
+
63
  ## Automatic base + delta loading
64
 
65
  Five final model checkpoints and their loading script live in this repository.
 
156
  See `VALIDATION.md` for the checks performed before publishing this loader.
157
  The verified standalone loader is included here. Training, evaluation, TailGuard,
158
  and dataset tooling are maintained in the accompanying source repository. The
159
+ private source release candidate `v0.1.0-rc3` includes the complete clean HInt
160
  validation replay and end-to-end TailGuard command.
161
 
162
+ ## Intended use and limitations
163
+
164
+ - Intended for non-commercial research on whole-body and hand pose estimation,
165
+ subject to the included Sapiens2 license.
166
+ - S2HAND remains a top-down estimator. Native-pipeline accuracy depends on the
167
+ person/hand detector and crop protocol; a checkpoint result is not comparable
168
+ across undeclared crop settings.
169
+ - S2HAND-MT gives an architectural exact-route guarantee for protected pose
170
+ heatmaps and leaves the shared backbone/auxiliary heads unchanged.
171
+ S2HAND-Pose updates shared Transformer blocks and therefore provides measured,
172
+ not exact, non-hand preservation.
173
+ - TailGuard requires an external hand detector and adds conditional inference;
174
+ it is not baked into these weights.
175
+ - The model can fail under severe blur, truncation, occlusion, side ambiguity,
176
+ unusual optics, or detector failure. Do not use it for safety-critical or
177
+ identity-sensitive decisions.
178
+
179
+ ## Citation
180
+
181
+ ```bibtex
182
+ @misc{muk2027s2hand,
183
+ title = {S2HAND: Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models},
184
+ author = {Muk, Singyu and Song, Xuanren and Stefanov, Kalin},
185
+ year = {2027}
186
+ }
187
+ ```
188
+
189
  ## Attribution
190
 
191
  Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived
VALIDATION.md CHANGED
@@ -1,5 +1,14 @@
1
  # Loader validation
2
 
 
 
 
 
 
 
 
 
 
3
  Before upload, all five models passed:
4
 
5
  - Exact base/delta SHA256 verification against their manifests.
@@ -25,20 +34,51 @@ The standalone image pipeline is also covered by unit tests for:
25
  - RGB/ImageNet input tensor shape and finite normalization; and
26
  - 308-channel UDP/DARK decoding back to original-image coordinates.
27
 
28
- On 2026-10-30, the standalone Pose-5B trainer additionally passed a real
29
- one-step L40S/BF16 optimization and independent checkpoint-reload test. The
30
- released MT-5B route was checked on a real HInt image: all 268 protected
31
- heatmaps were bitwise equal to the public route, while all 40 adapted channels
32
- used the learned route.
33
-
34
- The clean checkout also replayed all 1,175 hands in the HInt NewDays/EPICK
35
- validation split. The catastrophic-tail rate matched exactly
36
- (`0.0144680851`). Selection AUC was `0.7656657143`, versus `0.7656003214` in
37
- the original research record (absolute difference `0.0000653929`), and
38
- combined NME differed by `0.0000097603`. This is a near-exact numerical replay,
39
- not a bitwise metric claim.
40
-
41
- Finally, TailGuard was exercised with the released MT-5B delta, real NewDays
42
- images, person detections, and WiLoR-YOLO hand detections. Non-triggered rows
43
- remained bitwise equal to native predictions, while every triggered row used
44
- the documented coherent 21-joint conditional crop.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  # Loader validation
2
 
3
+ ## Source-package checks (2026-10-30)
4
+
5
+ - All 28 unit tests pass under the release source tree.
6
+ - Metric integration uses `numpy.trapz`, matching the declared NumPy >=1.26
7
+ requirement as well as newer NumPy releases.
8
+ - All local Markdown links resolve.
9
+ - No checkpoint, dataset, or file larger than 10 MiB is tracked by GitHub;
10
+ deployment deltas remain in the Hugging Face model repository.
11
+
12
  Before upload, all five models passed:
13
 
14
  - Exact base/delta SHA256 verification against their manifests.
 
34
  - RGB/ImageNet input tensor shape and finite normalization; and
35
  - 308-channel UDP/DARK decoding back to original-image coordinates.
36
 
37
+ The protocol-locked evaluator was also compared with the NewDays paper closure
38
+ aggregate for S2HAND-Pose. Hand count, evaluated-joint count, recall,
39
+ PCK@.05/.10/.15/.20, AUC@.20, severe-tail rate, and finite-hand NME all matched
40
+ exactly.
41
+
42
+ The standalone training adapter's UDP targets were compared element-by-element
43
+ with the original research encoder on randomized in-frame and outside-frame
44
+ points; target weights and all heatmap values were exactly equal. Unit tests
45
+ also cover the 42-channel layout, visibility handling, and protected-route
46
+ gradient stop.
47
+
48
+ On 2026-10-30, the standalone Pose-5B trainer passed a real one-step L40S/BF16
49
+ mechanics test from the pinned training warm-start. It completed canonical
50
+ NewDays/EPICK validation, a forward/backward update of blocks 38--40 and the
51
+ adapted decoder, gradient clipping, best/final checkpoint serialization, and
52
+ manifest hashing. The 1,068,545,694-byte final delta contained exactly the 60
53
+ expected tensors. A separate process then loaded that generated delta over the
54
+ pinned official base with `load_training_checkpoint` and reproduced finite
55
+ canonical validation metrics. The two-hand diagnostic slice is a mechanics
56
+ test only and is not reported as a paper benchmark result.
57
+
58
+ The released S2HAND-MT-5B route was also checked with its verified final delta
59
+ and a real HInt validation image. All 268 protected heatmaps were bitwise equal
60
+ to the frozen public decoder route (maximum absolute difference `0.0`), while
61
+ the 40 adapted finger heatmaps differed from that route. The ordinary model
62
+ forward was bitwise equal to an independently assembled explicit 40/268 route.
63
+
64
+ The clean checkout also replayed the complete 1,175-hand HInt NewDays/EPICK
65
+ validation split for the released S2HAND-Pose-5B delta. The catastrophic-tail
66
+ rate matched the original research record exactly (`0.0144680851`). Selection
67
+ AUC was `0.7656657143`, versus `0.7656003214` in the original record (absolute
68
+ difference `0.0000653929`), and combined NME differed by `0.0000097603`.
69
+ This is a near-exact protocol replay rather than a bitwise claim: the largest
70
+ single metric difference was `0.0019462` for a discontinuous occluded-joint PCK
71
+ threshold count.
72
+
73
+ Finally, after an editable clean-checkout installation, the documented
74
+ inference, hand-evaluation, Fisher-probe, and training CLIs all parsed their
75
+ published command-line interfaces successfully. All relative Markdown links in
76
+ the README, validation record, and `docs/` resolve to existing files.
77
+
78
+ TailGuard was then exercised end to end with the released S2HAND-MT-5B delta,
79
+ real NewDays images, cached person detections, and cached WiLoR-YOLO left/right
80
+ hand detections. Across 8 images (16 hand rows), 11 rows had a matching detector
81
+ box and 4 crossed the frozen confidence threshold. All 12 non-triggered rows
82
+ were bitwise equal to their native predictions; all 4 triggered rows changed
83
+ and used exactly 4 additional pose forwards. This is an implementation smoke
84
+ test, not an additional benchmark result.
assets/s2hand_overview.png ADDED

Git LFS Details

  • SHA256: 152ad4f0741186a63fb672df68917dd63f1441e37b806b1e3f77ebc338a9d4ff
  • Pointer size: 132 Bytes
  • Size of remote file: 1.31 MB