S2HAND / VALIDATION.md
mage0001's picture
Polish model card, project navigation, results, and validation
156338f verified
|
Raw History Blame Contribute Delete
4.73 kB
# Loader validation
## Source-package checks (2026-10-30)
- All 28 unit tests pass under the release source tree.
- Metric integration uses `numpy.trapz`, matching the declared NumPy >=1.26
requirement as well as newer NumPy releases.
- All local Markdown links resolve.
- No checkpoint, dataset, or file larger than 10 MiB is tracked by GitHub;
deployment deltas remain in the Hugging Face model repository.
Before upload, all five models passed:
- Exact base/delta SHA256 verification against their manifests.
- Strict loading of every official base tensor with no missing/meta parameters.
- Exact replacement tensor values in the original runtime dtypes.
- Bitwise-equal routed-head outputs against the original research head implementation
on deterministic feature tensors, including protection of both wrist channels.
- Correct BF16 RoPE runtime buffer semantics, including the rebased 5B override.
Additionally, the 0.4B model's full backbone-and-head forward pass matched the
original research implementation bitwise on a reduced-resolution 64x48 CPU input.
On 2026-10-29, S2HAND-Pose-0.4B, 0.8B, and 1B each passed a fresh
full-resolution A100/BF16 reconstruction test. Every run verified the official
base and delta hashes, produced finite `1x308x256x192` heatmaps, and was
bitwise-equal across two forwards of the same deterministic spatial-ramp input.
The output hashes are recorded in the per-model manifests.
The standalone image pipeline is also covered by unit tests for:
- 1.25x bbox padding and 3:4 aspect-ratio correction;
- UDP affine mapping at the crop center;
- RGB/ImageNet input tensor shape and finite normalization; and
- 308-channel UDP/DARK decoding back to original-image coordinates.
The protocol-locked evaluator was also compared with the NewDays paper closure
aggregate for S2HAND-Pose. Hand count, evaluated-joint count, recall,
PCK@.05/.10/.15/.20, AUC@.20, severe-tail rate, and finite-hand NME all matched
exactly.
The standalone training adapter's UDP targets were compared element-by-element
with the original research encoder on randomized in-frame and outside-frame
points; target weights and all heatmap values were exactly equal. Unit tests
also cover the 42-channel layout, visibility handling, and protected-route
gradient stop.
On 2026-10-30, the standalone Pose-5B trainer passed a real one-step L40S/BF16
mechanics test from the pinned training warm-start. It completed canonical
NewDays/EPICK validation, a forward/backward update of blocks 38--40 and the
adapted decoder, gradient clipping, best/final checkpoint serialization, and
manifest hashing. The 1,068,545,694-byte final delta contained exactly the 60
expected tensors. A separate process then loaded that generated delta over the
pinned official base with `load_training_checkpoint` and reproduced finite
canonical validation metrics. The two-hand diagnostic slice is a mechanics
test only and is not reported as a paper benchmark result.
The released S2HAND-MT-5B route was also checked with its verified final delta
and a real HInt validation image. All 268 protected heatmaps were bitwise equal
to the frozen public decoder route (maximum absolute difference `0.0`), while
the 40 adapted finger heatmaps differed from that route. The ordinary model
forward was bitwise equal to an independently assembled explicit 40/268 route.
The clean checkout also replayed the complete 1,175-hand HInt NewDays/EPICK
validation split for the released S2HAND-Pose-5B delta. The catastrophic-tail
rate matched the original research record exactly (`0.0144680851`). Selection
AUC was `0.7656657143`, versus `0.7656003214` in the original record (absolute
difference `0.0000653929`), and combined NME differed by `0.0000097603`.
This is a near-exact protocol replay rather than a bitwise claim: the largest
single metric difference was `0.0019462` for a discontinuous occluded-joint PCK
threshold count.
Finally, after an editable clean-checkout installation, the documented
inference, hand-evaluation, Fisher-probe, and training CLIs all parsed their
published command-line interfaces successfully. All relative Markdown links in
the README, validation record, and `docs/` resolve to existing files.
TailGuard was then exercised end to end with the released S2HAND-MT-5B delta,
real NewDays images, cached person detections, and cached WiLoR-YOLO left/right
hand detections. Across 8 images (16 hand rows), 11 rows had a matching detector
box and 4 crossed the frozen confidence threshold. All 12 non-triggered rows
were bitwise equal to their native predictions; all 4 triggered rows changed
and used exactly 4 additional pose forwards. This is an implementation smoke
test, not an additional benchmark result.