fernandotonon/QtMeshEditor-t2m

QtMeshEditor — text-to-motion (t2m)

Text→skeletal-animation models shipped with QtMeshEditor.

QtMesh T2M (v9, text-conditioned) — experimental

QtMeshEditor's own text-to-motion model, offered next to NVIDIA Kimodo in the editor (Animation panel → Model: QtMesh T2M, CLI --backend qtmesh_t2m, MCP generate_motion {backend: "qtmesh_t2m"}). Kimodo stays the default; QtMesh T2M is much smaller and faster, intended for lower-end machines.

It conditions on a sentence embedding of the prompt (Qwen3-Embedding-0.6B), so wording, sides ("with the right hand"), styles and speeds reach the model — unlike the v8 action-label model below.

Kimodo vs QtMesh T2M

Same generated character, same prompt, 3 s, in place, one sample each (seed in the caption).

Jumps vary a lot with the seed: of ten seeds, four bend the knees 70–86° (the one shown), four barely move.

Memory and speed

One 3 s clip on an Apple M-series Mac; the CLI run includes importing the rig and exporting the result.

	NVIDIA Kimodo (native)	QtMesh T2M
Download	~9.2 GB	~740 MB
Text encoding	~1.1 s + loading an 8 GB encoder	~0.2–0.4 s
Motion sampling	~14 s	~2.7 s
Whole CLI run	18–21 s	~5 s
Peak resident memory	7.6–8.2 GB	~0.87 GB

Files (qtmesh_t2m/)

• t2m-text.onnx (98 MB) — 22.6 M-parameter flow-matching transformer with AdaLN-Zero conditioning; the 24-step Euler sampler with classifier-free guidance (2.0) is unrolled in the graph. Inputs text[1,1024] (L2-normalised embedding), seed[1,19800] ~ N(0, 0.5), length[1] frames (≤ 150, 30 fps); output motion[1,150,220]: per canonical role (22 joints) t, quat x y z w, s, world frame, rotation only.
• t2m-text.json — contract: shapes, fps, the embedding instruction, canonical rest directions.
• Qwen3-Embedding-0.6B-Q8_0.gguf — the text encoder (Apache-2.0, LICENSE-Qwen3-Embedding.txt), run through llama.cpp: "Instruct: Given a description of a human body motion, retrieve motions that match it\nQuery:" + prompt, last-token pooling, L2-normalised.
• ATTRIBUTION.md — required credits for the CC-BY-4.0 motion the model was trained on.

Training

• Data: 30,488 samples — 3,400 clips we generated with NVIDIA Kimodo (each also mirrored left↔right, with "left"/"right" swapped in the text), plus the v8 windows (curated template library + CC0/CC-BY corpus) captioned per action; 8,028 distinct texts including paraphrases. Kimodo outputs are not Derivative Models under the NVIDIA Open Model License; Kimodo's Llama 3 text embeddings were never stored or trained on.
• Recipe: scripts/prep-t2m-text.py + scripts/train-t2m-text.py in the QtMeshEditor repo; 200 epochs, batch 128, AdamW 2e-4 cosine, 10 % text dropout for CFG, padded to 150 frames with a length mask. The export pins the never-supervised canonical buttock roles (14/18) to identity.
• Inference details in the editor: long clips start from rest in the Kimodo data, so the editor generates up to 1 s extra and trims the still start; root travel is estimated from the planted foot; the clip is retargeted with the same incompatible-skeleton retargeter as Kimodo.

Limitations

One prompt per clip (no multi-segment timelines), up to 5 s, no root height (jumps stay on the spot), weaker on rare actions; quality varies with the seed — generate a few variations.

---

v8.0 action-label model (previous)

Generative text→skeletal-animation model shipped with QtMeshEditor (qtmesh anim <rig> --generate "<prompt>" --model).

v8.0 — flow-matching DiT, 21.7M params, 30 actions.

Files

• t2m.onnx + t2m-vocab.json — the generative model + its vocab/contract.
• t2m-v61.onnx / t2m-v4.onnx — previous versions, kept for rollback.
• motion-library-v2.json — the curated template clip library (150 clips). It is both the fallback for prompts outside the model vocab and — since v8.0 — the model's training data. This is the exact library v8.0 trained on.
• ATTRIBUTION-v2.md — required credits for the CC-BY clips in that library. It must ship with the data and with anything derived from it.

Dataset

The published library is the full v8.0 training source: 150 clips drawn from 31 source assets — 27 CC-BY-4.0, plus CC0 material (largely Quaternius packs) and 17 clips the author owns outright. Every CC-BY asset is credited in ATTRIBUTION-v2.md; no clip in the library lacks a credit where one is due.

The raw source assets the library was built from — plus per-asset provenance and the build scripts — live in the companion dataset repo fernandotonon/QtMeshEditor-motion-corpus, which also carries a copy of this library under derived/.

To rebuild the training cache from it, see Reproducing below. The cache itself is not published — it is fully regenerable from this library, and it stamps its own buildArgv/buildFlags so any .npz can state how it was built.

What changed in v8.0

Earlier versions trained on the CMU MoCap corpus and inherited a facing problem: generated walks moved convincingly but oriented backwards. Measuring against a real reference walk showed why — CMU "walk" windows sit 1.73–2.07 rad away from it, further than a real punch, dance or jump, and the real-vs-real floor is 0.437. The corpus never carried the correct convention, so the model could not learn it. v6.x models scored 1.89 while their own training data scored 1.73: they were faithfully reproducing bad data.

v8.0 trains on the curated template clips instead (--library-repeat 40), which measure 0.303–0.462 against the real walk. Plus two data gates and one new loss:

• Anatomical ankle gate — rejects clips whose mean knee→foot angle exceeds 70°. Caught march (106°), throw (101°) and hey (90°): the foot folded perpendicular to the shin in 100% of their windows.
• Minimum resolved roles (16/22) — hey resolved only 9, with all four knee/ankle roles invalid, so the model had no leg data and invented legs.
• Leg-chain loss — keeps a walk's bend in the knee rather than the ankle. The v7.x walk bent the knee 25° and the ankle 31° (real: 38°/10°), which renders as a near-straight knee with a false second joint below it.

Measured against a real reference walk

	v7.x	v8.0	real
knee/ankle split	25°/31°	41°/8°	38°/10°
leg-chain penalty	0.359	0.0003	0.014
walk refDist	0.287	0.243	0.437 (real-vs-real floor)
run refDist	0.472	0.324	—

Scored against the training distribution across 20 actions, v8.0 is closer to ground truth on 8, further on 3, and tied on 9. Largest gains: wave 0.710→0.082 (the arm now actually goes overhead), idle, climb, salute, punch, sit.

Vocabulary (30 actions)

attack boxing buildloop cartgiveloop climb cough crouch cut dance death
fall farmloop hit idle jump kick pickup punch run salute shake shoot
sit stonegiveloop strafeleft straferight walk wave woodgiveloop working

march, throw, hey and confession were removed — their source clips are anatomically broken (see the ankle gate above). Those prompts fall back to the curated template library, which for march/throw is strictly better than what the model produced. march/marching resolves to walk.

Quality is uneven across actions: walk, run, wave, cough, death, pickup and attack are the strongest; dance and idle are weak (dance averages five genuinely different dances under one label; idle is nearly static). This is a template-quality ceiling, not an architecture one — more curated clips is the improvement path.

Training data — permissive only

The curated library is CC0/CC-BY content (Sketchfab, OpenGameArt, Quaternius packs) plus clips the author owns; credits are in ATTRIBUTION-v2.md, which must ship with the library and with anything derived from it. Earlier CMU MoCap windows (commercial-OK) remain in the corpus path but no longer dominate training. AMASS / HumanML3D / KIT-ML are excluded (non-commercial). Zombie/fruit/non-humanoid characters are filtered out, as are clips failing the anatomical ankle gate or resolving fewer than 16 of the 22 canonical roles.

Architecture (v8.0)

• 6D-rotation representation (Zhou et al. 2019), column-packed.
• Flow-matching DiT, dim 384 × 8 layers, ~21.7M params — the model predicts a velocity field; the host runs a fixed Euler schedule baked into the graph.
• Auxiliary losses: contralateral gait phase, spine-twist penalty, travel direction, limb amplitude band, jitter band, gait periodicity, and the v8.0 leg-chain term — each gated to the action classes where it is meaningful.
• Exports to ONNX as a single forward pass.

I/O contract

input  "tokens" float32 [1, V]     one-hot over the fixed action vocab (see t2m-vocab.json)
input  "seed"   float32 [1, Z]     latent noise (host samples ~N(0,0.5) and does best-of-N)
output "motion" float32 [1, T, C]  C = 22*10 per-joint [tx,ty,tz, qx,qy,qz,qw, sx,sy,sz]

t2m-vocab.json ships the {vocab, Z, T, C, J, fps, frame, restDir} the host needs. frame: "world" marks the WORLD-frame convention (the retarget takes a world delta); fps: 30, T: 60 (2.0 s).

Reproducing

scripts/prep-t2m-v6.py + scripts/train-t2m-flow-v5.py in the QtMeshEditor repo (one-time, offline dev tools — the app never runs Python):

--corpus is optional and points at the raw CMU-derived corpus, which is NOT published here; with --library-repeat 40 the curated library dominates training either way, so omitting it reproduces the same shape of dataset from the published files alone:

python3 scripts/prep-t2m-v6.py --library motion-library-v2.json \
    --library-repeat 40 --min-roles 16 --out t2m.npz
python3 scripts/train-t2m-flow-v5.py --data t2m.npz --out flow --dim 384 --layers 8 \
    --phase-weight 0.08 --twist-weight 0.5 --travel-weight 0.2 --amp-weight 0.4 \
    --jitter-weight 0.4 --period-weight 0.5 --legchain-weight 0.35 --loco-boost 6.0

The cache stamps its own buildArgv/buildFlags, so any .npz can report how it was built.