anira-project/mediapipe-gesture-recognizer

MediaPipe Gesture Recognizer for anira

MediaPipe's hand tracking and gesture recognition models, converted for real-time inference with anira on LiteRT, ONNX Runtime, LibTorch and ExecuTorch (XNNPACK and Core ML).

The models come from MediaPipe's gesture_recognizer.task bundle (float16): palm detection, 21 hand landmarks, gesture embedding and gesture classification.

Files

Model	Input → output	litert/	onnx/	libtorch/	executorch/	executorch/*_coreml	executorch/*_vulkan
hand_detector	image [1,192,192,3] → boxes + keypoints [1,2016,18], scores [1,2016,1]	✓	✓	✓	✓	✓	✓
hand_landmarks_detector	image [1,224,224,3] → landmarks [1,63], presence [1,1], handedness [1,1], world landmarks [1,63]	✓	✓	✓	✓	✓	✓
gesture_embedder	landmarks [1,21,3], handedness [1,1], world landmarks [1,21,3] → embedding [1,128]	✓	✓	✓	✓	✓	✓
canned_gesture_classifier	embedding [1,128] → 8 gesture scores [1,8]	✓	✓	✓	✓	✓	✓

Images are NHWC RGB in [0, 1]. The eight classes are MediaPipe's canned gestures: None, Closed_Fist, Open_Palm, Pointing_Up, Thumb_Down, Thumb_Up, Victory, ILoveYou.

Export

The ExecuTorch programs (XNNPACK, every CPU platform, runtime 1.3.1) are exported from the ONNX models by export/export_executorch.py, which checks each one against ONNX Runtime (largest deviation 1.5e-4). With --coreml it lowers them to Core ML instead (*_coreml.pte, Apple platforms, float16 for the Neural Engine; within 4.3e-3 of each output's peak):

uv run export/export_executorch.py <folder laid out like this repository> [--coreml]

Formats and checks

Every model here runs on ONNX Runtime, LiteRT and TensorFlow Lite, LibTorch, ExecuTorch (XNNPACK on the CPU, Core ML on Apple, Vulkan on GPUs) and as a native Core ML ML program, with the ONNX models' tensor names. anira foundation models streams every format of every description over twelve consecutive calls with the state fed back and checks that they agree (tools/check_streaming.py). The models are stateless; the deviations are relative to each output's peak.

Format	Files	Max deviation from ONNX Runtime
ExecuTorch, Core ML (Apple, float16 for the Neural Engine)	executorch/*_coreml.pte	hand_detector 4.3e-3, hand_landmarks_detector 1.0e-3, gesture_embedder 7.1e-4, canned_gesture_classifier 0
ExecuTorch, Vulkan (GPUs)	executorch/*_vulkan.pte	exported; not run here (no Vulkan runtime)
Core ML ML program (anira's native engine, coming with v3; macOS 13, iOS 16; float16)	coreml/*.mlpackage	hand_detector 4.3e-3, hand_landmarks_detector 1.0e-3, gesture_embedder 7.1e-4, canned_gesture_classifier 0

uv run export/export_executorch.py <folder laid out like this repository> [--coreml | --vulkan]
uv run export/export_coreml.py <folder laid out like this repository>

This repository holds the weights, the export script and the licence. The anira model descriptions, the contract and the pre- and post-processing live with the code in anira foundation models (models/handtracking), versioned with git. Pin a revision and check each file's SHA256 when you download:

https://huggingface.co/anira-project/mediapipe-gesture-recognizer/resolve/<revision>/onnx/hand_detector.onnx

Licence

The models are Google's, licensed under the Apache License 2.0 as stated in the MediaPipe Hand Tracking model card. The conversions in this repository keep that licence; LICENSE is its text. See the MediaPipe gesture recognizer guide for the original models.