vitpose / README.md
Flor's picture
card: update from fastforks 1cbf1671
15450c8 verified
|
Raw History Blame Contribute Delete
10.3 kB
metadata
library_name: fastforks
pipeline_tag: keypoint-detection
license: apache-2.0
tags:
  - pose
  - pose-estimation
  - human-pose-estimation
  - keypoint-detection
  - cuda
  - pytorch
  - fastforks
  - fastforks-card

ViTPose / ViTPose++ (fastforks)

ViTPose (Xu et al. 2022): top-down 2D human pose with a plain ViT encoder and a light deconvolution (or bilinear) decoder; ViTPose++ trains one backbone on six pose datasets with per-dataset MLP experts. Takes person boxes (e.g. from fastforks.detection), crops each at 256 x 192 and returns the 17 COCO keypoints with DARK sub-pixel decoding. The portable module() maps crops to heatmaps.

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/transformers at v4.57.1.

Author weight repos: https://huggingface.co/usyd-community/vitpose-base, https://huggingface.co/usyd-community/vitpose-base-simple, https://huggingface.co/usyd-community/vitpose-plus-base, https://huggingface.co/usyd-community/vitpose-plus-huge, https://huggingface.co/usyd-community/vitpose-plus-large, https://huggingface.co/usyd-community/vitpose-plus-small.

Task: pose. Tier: stable. Outputs: keypoints, keypoint_scores, boxes, scores. Backends: portable, kernels.

Input: one RGB image and a box per person. Output: keypoints with confidence scores for every person.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
vitpose-base@1 vitpose-base-cd9a4e6c.safetensors cd9a4e6cefc3 Apache-2.0 yes
vitpose-base-simple@1 vitpose-base-simple-85375373.safetensors 85375373893d Apache-2.0 yes
vitpose-plus-base@1 vitpose-plus-base-640225e4.safetensors 640225e4a9dd Apache-2.0 yes
vitpose-plus-huge@1 vitpose-plus-huge-0ecb49f1.safetensors 0ecb49f1ab0b Apache-2.0 yes
vitpose-plus-large@1 vitpose-plus-large-8b249f74.safetensors 8b249f74fca7 Apache-2.0 yes
vitpose-plus-small@1 vitpose-plus-small-f7bad8ed.safetensors f7bad8ed09ee Apache-2.0 yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0. Licence files: LICENSE.

Training data: COCO 2017 (train, instance annotations), ImageNet-1k (ILSVRC 2012 classification) (COCO 2017 keypoints; the ViT was initialised from MAE ImageNet-1k pre-training).

Licence evidence: https://huggingface.co/usyd-community/vitpose-base-simple (model card licence)

Input size

any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): vitpose-base 3000x4000 (4.5 GiB peak allocated, 4.8 reserved); vitpose-base-simple 3000x4000 (4.5 GiB peak allocated, 4.8 reserved); vitpose-plus-base 3000x4000 (4.5 GiB peak allocated, 4.8 reserved); vitpose-plus-huge 3000x4000 (5.5 GiB peak allocated, 5.7 reserved); vitpose-plus-large 3000x4000 (4.9 GiB peak allocated, 5.2 reserved); vitpose-plus-small 3000x4000 (4.4 GiB peak allocated, 4.6 reserved)

Speed and memory

Pose, synthetic, 1080x1920

model time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
vitpose-base 139 105 17.8 12.0 8.8x 1.5x 713 / 762 299 / 322 274 / 364
vitpose-base-simple 145 109 20.1 12.1 9.0x 1.7x 828 / 962 590 / 642 295 / 344
vitpose-plus-base 162 113 19.9 11.9 9.5x 1.7x 918 / 1002 597 / 642 274 / 344
vitpose-plus-huge 610 242 104 62.5 3.9x 1.7x 5732 / 5846 1641 / 1704 1351 / 1406
vitpose-plus-large 331 165 53.7 33.2 5.0x 1.6x 2873 / 3004 1014 / 1062 706 / 764
vitpose-plus-small 120 104 7.83 6.05 17.2x 1.3x 312 / 376 164 / 196 145 / 198

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
vitpose-base 0.16 1.16 0.34 0.33 0.35 0.39
vitpose-base-simple 0.15 0.40 1.04 0.17 0.41 0.41
vitpose-plus-base 0.19 0.65 1.10 0.12 0.34 0.34
vitpose-plus-huge 0.65 2.69 2.26 0.25 0.44 0.40
vitpose-plus-large 0.41 1.50 1.63 0.17 0.41 0.41
vitpose-plus-small 0.15 0.81 1.09 0.12 0.46 0.50

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
vitpose-base 1080x1920 17.8 ms 16.4 ms 8.88 ms (network on person crops only) 12.0 ms 10.5 s 35.8 s 299 / 322 597 / 622 274 / 364 473 / 516
vitpose-base-simple 1080x1920 20.1 ms 16.0 ms 9.36 ms (network on person crops only) 12.1 ms 11.8 s 35.0 s 590 / 642 590 / 622 295 / 344 499 / 550
vitpose-plus-base 1080x1920 19.9 ms 16.4 ms 8.87 ms (network on person crops only) 11.9 ms 10.6 s 35.0 s 597 / 642 597 / 622 274 / 344 473 / 516
vitpose-plus-huge 1080x1920 104 ms 92.9 ms 58.7 ms (network on person crops only) 62.5 ms 21.2 s 47.3 s 1641 / 1704 1641 / 1682 1351 / 1406 2595 / 2654
vitpose-plus-large 1080x1920 53.7 ms 46.9 ms 29.3 ms (network on person crops only) 33.2 ms 17.9 s 39.0 s 1014 / 1062 1014 / 1062 706 / 764 1312 / 1338
vitpose-plus-small 1080x1920 7.83 ms 7.54 ms 3.39 ms (network on person crops only) 6.05 ms 10.6 s 37.5 s 164 / 196 472 / 496 145 / 198 214 / 256

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

Top-down on the ground-truth person boxes (non-crowd, at least one labelled keypoint), no flip test; people ranked by the mean of their 17 keypoint scores (not mmpose's rescoring and OKS-NMS), so not the papers' AP; pycocotools OKS AP; whole-body models scored on their 17 body keypoints. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable kernels note
vitpose-base-simple COCO val2017, ground-truth person boxes, keypoint AP 76.3 76.2 76.3 76.3
vitpose-base COCO val2017, ground-truth person boxes, keypoint AP 76.4 76.4 76.4 76.4
vitpose-plus-small COCO val2017, ground-truth person boxes, keypoint AP 76.8 76.8 76.8 76.8
vitpose-plus-base COCO val2017, ground-truth person boxes, keypoint AP 78.6 78.6 78.6 78.6
vitpose-plus-large COCO val2017, ground-truth person boxes, keypoint AP 80.6 80.6 80.5 80.5
vitpose-plus-huge COCO val2017, ground-truth person boxes, keypoint AP 81.0 81.0 80.9 80.9

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 2241b095+uncommitted, 2026-11-09.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.005 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.13 and p99 <= 0.75 (absolute error of the output keypoints).

Usage

from fastforks import pose

model = pose.load("vitpose-base")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg", boxes=[[x0, y0, x1, y1]])  # one box per person, or a fastforks.detection result
result.keypoints, result.keypoint_scores  # [P, K, 2] input pixels, [P, K]
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: vitpose-base, vitpose-base-simple, vitpose-plus-base, vitpose-plus-huge, vitpose-plus-large, vitpose-plus-small.

Kernels backend

pose.load("vitpose-base", mode="kernels") runs our CUDA kernels (fvm_conv). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.