--- library_name: "fastforks" pipeline_tag: "keypoint-detection" license: "apache-2.0" tags: - "pose" - "pose-estimation" - "human-pose-estimation" - "keypoint-detection" - "cuda" - "pytorch" - "fastforks" - "fastforks-card" --- # ViTPose / ViTPose++ (fastforks) ViTPose (Xu et al. 2022): top-down 2D human pose with a plain ViT encoder and a light deconvolution (or bilinear) decoder; ViTPose++ trains one backbone on six pose datasets with per-dataset MLP experts. Takes person boxes (e.g. from `fastforks.detection`), crops each at 256 x 192 and returns the 17 COCO keypoints with DARK sub-pixel decoding. The portable `module()` maps crops to heatmaps. This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: . Kernels for the `kernels` backend: . Original code: at `v4.57.1`. Author weight repos: , , , , , . Task: pose. Tier: stable. Outputs: `keypoints`, `keypoint_scores`, `boxes`, `scores`. Backends: `portable`, `kernels`. Input: one RGB image and a box per person. Output: keypoints with confidence scores for every person. ## Variants Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/pose/models/vitpose/card.toml). | variant | weights (author link) | SHA-256 | weights licence | commercial use | |---|---|---|---|---| | `vitpose-base@1` | [vitpose-base-cd9a4e6c.safetensors](https://huggingface.co/usyd-community/vitpose-base/resolve/95be2991424e646950d656bb7fc15ec9be119700/model.safetensors) | `cd9a4e6cefc3` | Apache-2.0 | yes | | `vitpose-base-simple@1` | [vitpose-base-simple-85375373.safetensors](https://huggingface.co/usyd-community/vitpose-base-simple/resolve/a93ac0c67e0b7e2c55287d21d4c460c8f3c54d45/model.safetensors) | `85375373893d` | Apache-2.0 | yes | | `vitpose-plus-base@1` | [vitpose-plus-base-640225e4.safetensors](https://huggingface.co/usyd-community/vitpose-plus-base/resolve/92be54d7a29e42fad47b6e2ca01dd9e685a61e0d/model.safetensors) | `640225e4a9dd` | Apache-2.0 | yes | | `vitpose-plus-huge@1` | [vitpose-plus-huge-0ecb49f1.safetensors](https://huggingface.co/usyd-community/vitpose-plus-huge/resolve/9f36d7aec1800d23e97f10c2e74393aee92aa53f/model.safetensors) | `0ecb49f1ab0b` | Apache-2.0 | yes | | `vitpose-plus-large@1` | [vitpose-plus-large-8b249f74.safetensors](https://huggingface.co/usyd-community/vitpose-plus-large/resolve/e211df377e49c89dba508ab3e83ddef13c0832b4/model.safetensors) | `8b249f74fca7` | Apache-2.0 | yes | | `vitpose-plus-small@1` | [vitpose-plus-small-f7bad8ed.safetensors](https://huggingface.co/usyd-community/vitpose-plus-small/resolve/0c30b6534bb621af0162b481176742577264e36e/model.safetensors) | `f7bad8ed09ee` | Apache-2.0 | yes | ## Licence Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0. Licence files: `LICENSE`. Training data: COCO 2017 (train, instance annotations), ImageNet-1k (ILSVRC 2012 classification) (COCO 2017 keypoints; the ViT was initialised from MAE ImageNet-1k pre-training). Licence evidence: https://huggingface.co/usyd-community/vitpose-base-simple (model card licence) ## Input size any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `vitpose-base` 3000x4000 (4.5 GiB peak allocated, 4.8 reserved); `vitpose-base-simple` 3000x4000 (4.5 GiB peak allocated, 4.8 reserved); `vitpose-plus-base` 3000x4000 (4.5 GiB peak allocated, 4.8 reserved); `vitpose-plus-huge` 3000x4000 (5.5 GiB peak allocated, 5.7 reserved); `vitpose-plus-large` 3000x4000 (4.9 GiB peak allocated, 5.2 reserved); `vitpose-plus-small` 3000x4000 (4.4 GiB peak allocated, 4.6 reserved) ## Speed and memory ### Pose, synthetic, 1080x1920 | model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `vitpose-base` | 139 | 105 | 17.8 | 12.0 | 8.8x | 1.5x | 713 / 762 | 299 / 322 | 274 / 364 | | `vitpose-base-simple` | 145 | 109 | 20.1 | 12.1 | 9.0x | 1.7x | 828 / 962 | 590 / 642 | 295 / 344 | | `vitpose-plus-base` | 162 | 113 | 19.9 | 11.9 | 9.5x | 1.7x | 918 / 1002 | 597 / 642 | 274 / 344 | | `vitpose-plus-huge` | 610 | 242 | 104 | 62.5 | 3.9x | 1.7x | 5732 / 5846 | 1641 / 1704 | 1351 / 1406 | | `vitpose-plus-large` | 331 | 165 | 53.7 | 33.2 | 5.0x | 1.6x | 2873 / 3004 | 1014 / 1062 | 706 / 764 | | `vitpose-plus-small` | 120 | 104 | 7.83 | 6.05 | 17.2x | 1.3x | 312 / 376 | 164 / 196 | 145 / 198 | ### Startup and first call | model | load (s): original | portable | kernels | first call (s): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | | `vitpose-base` | 0.16 | 1.16 | 0.34 | 0.33 | 0.35 | 0.39 | | `vitpose-base-simple` | 0.15 | 0.40 | 1.04 | 0.17 | 0.41 | 0.41 | | `vitpose-plus-base` | 0.19 | 0.65 | 1.10 | 0.12 | 0.34 | 0.34 | | `vitpose-plus-huge` | 0.65 | 2.69 | 2.26 | 0.25 | 0.44 | 0.40 | | `vitpose-plus-large` | 0.41 | 1.50 | 1.63 | 0.17 | 0.41 | 0.41 | | `vitpose-plus-small` | 0.15 | 0.81 | 1.09 | 0.12 | 0.46 | 0.50 | ### torch.compile and TensorRT | model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `vitpose-base` | 1080x1920 | 17.8 ms | 16.4 ms | 8.88 ms (network on person crops only) | 12.0 ms | 10.5 s | 35.8 s | 299 / 322 | 597 / 622 | 274 / 364 | 473 / 516 | | `vitpose-base-simple` | 1080x1920 | 20.1 ms | 16.0 ms | 9.36 ms (network on person crops only) | 12.1 ms | 11.8 s | 35.0 s | 590 / 642 | 590 / 622 | 295 / 344 | 499 / 550 | | `vitpose-plus-base` | 1080x1920 | 19.9 ms | 16.4 ms | 8.87 ms (network on person crops only) | 11.9 ms | 10.6 s | 35.0 s | 597 / 642 | 597 / 622 | 274 / 344 | 473 / 516 | | `vitpose-plus-huge` | 1080x1920 | 104 ms | 92.9 ms | 58.7 ms (network on person crops only) | 62.5 ms | 21.2 s | 47.3 s | 1641 / 1704 | 1641 / 1682 | 1351 / 1406 | 2595 / 2654 | | `vitpose-plus-large` | 1080x1920 | 53.7 ms | 46.9 ms | 29.3 ms (network on person crops only) | 33.2 ms | 17.9 s | 39.0 s | 1014 / 1062 | 1014 / 1062 | 706 / 764 | 1312 / 1338 | | `vitpose-plus-small` | 1080x1920 | 7.83 ms | 7.54 ms | 3.39 ms (network on person crops only) | 6.05 ms | 10.6 s | 37.5 s | 164 / 196 | 472 / 496 | 145 / 198 | 214 / 256 | Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB). Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's. ## Accuracy ### On public datasets Top-down on the ground-truth person boxes (non-crowd, at least one labelled keypoint), no flip test; people ranked by the mean of their 17 keypoint scores (not mmpose's rescoring and OKS-NMS), so not the papers' AP; pycocotools OKS AP; whole-body models scored on their 17 body keypoints. Original = the original code named under Details, same checkpoint and protocol. | variant | dataset | original fp32 | original fp16 | portable | kernels | note | |---|---|---|---|---|---|---| | `vitpose-base-simple` | COCO val2017, ground-truth person boxes, keypoint AP | 76.3 | 76.2 | 76.3 | 76.3 | | | `vitpose-base` | COCO val2017, ground-truth person boxes, keypoint AP | 76.4 | 76.4 | 76.4 | 76.4 | | | `vitpose-plus-small` | COCO val2017, ground-truth person boxes, keypoint AP | 76.8 | 76.8 | 76.8 | 76.8 | | | `vitpose-plus-base` | COCO val2017, ground-truth person boxes, keypoint AP | 78.6 | 78.6 | 78.6 | 78.6 | | | `vitpose-plus-large` | COCO val2017, ground-truth person boxes, keypoint AP | 80.6 | 80.6 | 80.5 | 80.5 | | | `vitpose-plus-huge` | COCO val2017, ground-truth person boxes, keypoint AP | 81.0 | 81.0 | 80.9 | 80.9 | | Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 2241b095+uncommitted, 2026-11-09. ### Parity with the original code Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.005 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.13 and p99 <= 0.75 (absolute error of the output `keypoints`). ## Usage ```python from fastforks import pose model = pose.load("vitpose-base") # mode="auto": the preferred backend from recorded measurements result = model("photo.jpg", boxes=[[x0, y0, x1, y1]]) # one box per person, or a fastforks.detection result result.keypoints, result.keypoint_scores # [P, K, 2] input pixels, [P, K] result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16') ``` Install with `pip install fastforks` (see ). Other variants: `vitpose-base`, `vitpose-base-simple`, `vitpose-plus-base`, `vitpose-plus-huge`, `vitpose-plus-large`, `vitpose-plus-small`. ## Kernels backend `pose.load("vitpose-base", mode="kernels")` runs our CUDA kernels (`fvm_conv`). It needs a prebuilt kernel for your GPU architecture (downloaded from on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.