Lyrebird motion encoder

This repository holds the frozen encoder that Lyrebird uses to identify gestures in webcam video.

The encoder reads MediaPipe landmark data and gives one token for each body area in each frame. Lyrebird trains a single linear layer on top of it from about ten examples of each gesture. The encoder itself does not change.

Files

File	Contents
encoder.pt	The encoder parameters and its configuration.
mediapipe/holistic_landmarker.task	The MediaPipe model that finds the landmarks.

The MediaPipe file is a copy of Google's holistic landmarker. Lyrebird keeps a copy here so that it needs only one repository. Google gives its terms and its model card at https://ai.google.dev/edge/mediapipe/solutions/vision/holistic_landmarker

Use

Lyrebird downloads this file for you. To get it directly:

from huggingface_hub import hf_hub_download

from lyrebird.classifier import load_encoder

encoder = load_encoder(hf_hub_download("byadecha/lyrebird", "encoder.pt"))

The model

Property	Value
Parameters	1.70 million
Input	33 body joints, and 21 joints for each hand
Output	4 tokens for each frame, of 128 values each
Frame rate	30 frames each second

Four encoders operate in parallel across the body, each hand, and the movement of the body through the frame. Each one reduces its joints to one token for each frame. It uses graph convolution across the skeleton, and then attention across time. A fusion stage mixes the four tokens in one frame, and then across time. The fusion stage keeps all four tokens. It does not make an average of them.

The encoder is causal. A later frame does not change an earlier frame. Thus the same parameters give a score to a recorded clip and to live video in the same way.

Training data

The encoder is trained on the BABEL annotations of the AMASS motion capture data. The training task is to identify which of 43 motion categories occurs in each frame.

BABEL and AMASS have their own terms of use. This repository makes no statement about what those terms permit for these parameters. Read the terms of both datasets before you use this file:

• AMASS: https://amass.is.tue.mpg.de/
• BABEL: https://babel.is.tue.mpg.de/

Limits

The classifier that Lyrebird trains on this encoder always gives the name of the most probable gesture that it knows. It cannot answer "none of these". Movement that is not like any recorded gesture still gets a name. Record an idle gesture, and record more than one form of it, to reduce this effect.

The gesture vector is a mean across a window of frames. Thus a gesture that fills only a small part of a long window becomes too weak to find. Give a score to windows of about the same length as the recorded clips.