ViBES-Q2M-Evaluator

ViBES paper: ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body (CVPR 2026)

The Question-to-Motion (Q2M) CLIP-style evaluator for ViBES. It learns a shared embedding space between text questions and SMPL-X body-motion sequences, and is used to compute the quantitative motion-generation metrics (FID, R-Precision, MM Dist, Diversity) reported for ViBES body models on the Converse3D eval set.

What this is

• A TM2T-style Text-Motion CLIP model (TextMotionCLIP): a Transformer MotionEncoder + a frozen CLIP text encoder with a trainable projection, trained with symmetric contrastive loss.
• Motion representation: 135-dim = 22 body joints × 6D rotation (132) + root velocity (3).
• This is a clean release of the checkpoint: only the state_dict (+ epoch/step meta) is kept; the Lightning optimizer/callback/loop state was stripped (1.3 GB → 0.78 GB). Fully load-compatible.

Usage

import torch
from omegaconf import OmegaConf
from multimodal_tokenizers.models.text_motion_clip import TextMotionCLIP

cfg = OmegaConf.load("configs/evaluator/question_motion_clip_v3_body.yaml")
model = TextMotionCLIP(cfg)
ckpt = torch.load("question_motion_clip_v3_best.ckpt", map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["state_dict"], strict=True)
model.to("cuda").eval()

Place it at model_files/pretrained_cpt/evaluator/question_motion_clip_v3_best.ckpt in the ViBES repo, then run scripts/eval_vibes_body_q2m.py. See docs/4-evaluation.md in the repo for the full protocol (balanced R-Precision, 20 replications, R_SIZE=32, EMB_SCALE=6.0, etc.).

Citation

If you use this evaluator, please cite ViBES.