| --- |
| license: cc-by-nc-4.0 |
| base_model: Linzhan/UniMate |
| library_name: kaminos |
| tags: |
| - webgpu |
| - browser |
| - motion-generation |
| - text-to-motion |
| - skeletal-animation |
| - unimate |
| --- |
| |
| # UniMate for WebGPU |
|
|
| The [UniMate](https://huggingface.co/Linzhan/UniMate) text-to-motion model and its Flan-T5-base text encoder, packed for generating skeletal animation entirely in the browser with the [Kaminos](https://github.com/lyonsno/kaminos) WebGPU inference kit. Give it a skeleton and a prompt such as "A dog runs forward.", and the visitor's own GPU generates a two-second, 30 fps animation for that body. No server-side compute is involved. |
|
|
| **[Try it in your browser](https://lyonsno.github.io/kaminos/inference-kit/unimate/)** · [Source](https://github.com/lyonsno/kaminos/tree/main/models/unimate) |
|
|
| The page downloads about 370 MB once (the float16 copies) and caches it. On an Apple M4 Max a clip takes about 10 seconds for the 18-joint Go2 robot dog and about 30 seconds for the 51-joint dog, while the page's 3D scene keeps drawing. The browser port matches the reference PyTorch implementation: one network evaluation agrees to a relative error of about 1.5 × 10⁻⁶, and a whole generation from the same starting noise reproduces the reference animation. |
|
|
| ## Contents |
|
|
| | Folder | Component | Format | Size | |
| | --- | --- | --- | --- | |
| | `unimate/` | UniMate v3 motion transformer (UniML3D, 150k steps, EMA weights), 74.5M parameters | float32 | 298 MB | |
| | `t5/` | Flan-T5-base encoder and its SentencePiece vocabulary | float32; JSON | 440 MB | |
| | `unimate-fp16/`, `t5-fp16/` | The same two models with float16 storage, which the page loads. Biases, norms and other small tensors stay float32. Every value is widened back to float32 at load, so a whole generation stays within 0.1% of body size of the float32 result. | float16 / float32 | 149 MB, 219 MB | |
| | `labeler/` | UniMate's rule-based joint labeller (`data_process` modules, MIT), packed for Pyodide. The browser rig importer uses it. | zip | 0.3 MB | |
| | `skeletons/` | Skeleton conditioning for eight bodies: the Unitree Go2 robot dog and seven Truebones animals (cat, dog, horse, T-rex, spider, eagle, parrot) | float32 / int32 | under 0.5 MB each | |
|
|
| Each weight folder has a manifest listing every tensor's name, shape, byte offset and length, and the SHA-256 of the packed file. `skeletons/index.json` lists the bodies with their joint counts and suggested prompts. Each skeleton holds the model's conditioning for that body: rest pose, hierarchy, joint-name embeddings, graph and spectral features, plus the feature normalization statistics. |
|
|
| ## Changes from the original |
|
|
| - The UniMate weights are the released checkpoint with its EMA weights applied (as UniMate's own sampler loads them), copied unchanged into one 256-byte-aligned float32 file. Loaded back into the reference model, the packed file reproduces its outputs bit for bit. |
| - The text encoder is the encoder half of Flan-T5-base. Each layer's query, key and value projections are concatenated into one matrix, as are the two gated feed-forward input projections; no values are changed. |
| - Skeleton conditioning was produced by UniMate's own preprocessing from its released Truebones features and the Go2 example asset. |
|
|
| ## License |
|
|
| The UniMate weights and the skeleton conditioning derived from them are licensed under **CC BY-NC 4.0**, the license of the original checkpoints, for non-commercial use only. See `LICENSE-UniMate`. That license covers only the rights of the UniMate authors; the data UniMate was trained on keeps its own terms. Flan-T5-base is by Google, under the Apache License 2.0. |
|
|
| UniMate is by Linzhan Mou, Jiahui Lei, Zhiyang Dou, Chenyue Cai, Chaoyue Song, Adam Finkelstein and Szymon Rusinkiewicz: [paper](https://arxiv.org/abs/2609.05415), [code](https://github.com/Friedrich-M/UniMate), [project page](https://linzhanmou.com/unimate/). |
|
|
