| --- |
| license: mit |
| library_name: pytorch |
| pipeline_tag: other |
| tags: |
| - camera-trajectory |
| - text-to-camera |
| - camera-control |
| datasets: |
| - yhz-linearalgebra/RealEstate10K-Cap |
| --- |
| |
| # TKCAM checkpoints |
|
|
| Pretrained checkpoints for **TKCAM: Text and Keyframe to Camera Trajectory Generation** |
| (NeurIPS 2026). TKCAM generates a camera trajectory from a text description, optionally conditioned |
| on a first frame or on up to 4 keyframes placed at chosen times. |
|
|
| [Paper (arXiv:2610.11105)](https://arxiv.org/abs/2610.11105) · |
| [Code](https://github.com/linearalgebrayhz/TKCAM) · |
| [Project page](https://linearalgebrayhz.github.io/projects/TKCAM/) · |
| [Dataset](https://huggingface.co/datasets/yhz-linearalgebra/RealEstate10K-Cap) |
|
|
| | mode | conditioning | mask transformer | residual transformer | |
| |---|---|---|---| |
| | `noframe` | text | `mtrans_25kfixed_noframe_r4` (epoch 555) | `rtrans_25kfixed_noframe_r4` (epoch 40) | |
| | `firstframe` | text + first frame | `mtrans_25kfixed_1frame_r4` (epoch 480) | `rtrans_25kfixed_1frame_r4` (epoch 600) | |
| | `xframe` | text + up to 4 keyframes | `mtrans_25kfixed_xframe_r4` (epoch 600) | `rtrans_25kfixed_xframe_r4` (epoch 125) | |
|
|
| All modes share the residual VQ-VAE `rvq_w128_nq4_cb256_25kfixed_60ep` (4 quantizers, 256 codes, |
| window 128; epoch 25). Text is encoded with a frozen pretrained `t5-base` and images with a frozen |
| pretrained CLIP ViT-B/32 vision encoder (`openai/clip-vit-base-patch32`). |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/linearalgebrayhz/TKCAM.git && cd TKCAM |
| pip install -r requirements.txt && pip install -e . # or: conda env create -f environment.yml |
| hf download yhz-linearalgebra/TKCAM --local-dir checkpoints |
| |
| # Text only |
| python scripts/generate.py --mode noframe --duration 6 \ |
| --text "The camera dollies forward slowly, revealing the living room." |
| # Text + first frame |
| python scripts/generate.py --mode firstframe --duration 5 --first_frame room.jpg \ |
| --text "The camera pans left smoothly across the kitchen." |
| # Text + keyframes at chosen times (image@seconds, up to 4) |
| python scripts/generate.py --mode xframe --duration 8 \ |
| --keyframe hall.jpg@0 --keyframe stairs.jpg@4.5 \ |
| --text "The camera moves forward through the hallway, then tilts up toward the stairs." |
| ``` |
|
|
| Each sample is saved as `sample{i}_9d.npy` (`(T, 9)`: `[rot6d(6), x, y, z]`), `sample{i}_c2w.npy` |
| (`(T, 4, 4)`), an MP4 animation and `meta.json`. Trajectories are camera-to-world, relative to the |
| first frame, with OpenGL camera axes (+X right, +Y up, −Z forward), at 30 fps. Durations are clamped |
| to 3–11 s; the training clips are about 4.0–9.3 s. See the code repository for batch inference over a |
| prompts file and for training. |
|
|
| The T5 and CLIP encoder weights are not stored in these files: they are the unmodified public |
| weights and are downloaded from the Hugging Face Hub on first use. |
|
|
| ## Files |
|
|
| The folder mirrors the layout the TKCAM code expects under `checkpoints/`: |
|
|
| ``` |
| realestate10k_rotmat/ |
| rvq_w128_nq4_cb256_25kfixed_60ep/{opt.txt, meta/mean.npy, meta/std.npy, model/net_best_recon.tar} |
| {mtrans,rtrans}_25kfixed_{noframe,1frame,xframe}_r4/{opt.txt, net_best_fid.tar} |
| MANIFEST.json # epochs, source checkpoint, parameter counts, sizes, sha256 |
| ``` |
|
|
| Each `.tar` holds the model `state_dict` and the epoch (no optimizer state). Every transformer |
| file is named `net_best_fid.tar`, the name the TKCAM scripts load by default: it is the checkpoint |
| selected on validation FID (see [Training](#training)); the epochs are in the table above and in |
| `MANIFEST.json`. `opt.txt` is the |
| training configuration; local paths in it were replaced with placeholders. |
|
|
| ## Training |
|
|
| - **Data:** the `train` split of [RealEstate10K-Cap](https://huggingface.co/datasets/yhz-linearalgebra/RealEstate10K-Cap) |
| (21,250 clips), the released version of the dataset. |
| - **Recipe:** MoMask three-stage pipeline (residual VQ-VAE → masked transformer → residual |
| transformer), 600 epochs per transformer, learning rate 5e-5. Visual conditioning uses |
| the first frame (`firstframe`) or 0–4 randomly sampled keyframes per clip (`xframe`). |
| - **Checkpoint selection:** the VQ-VAE by validation reconstruction error. For each transformer, |
| the saved checkpoints of the run (best validation FID during training, best accuracy, best loss, |
| and the last epoch) were re-scored by FID on the 2,500-clip validation split with a CLaTr |
| evaluator whose training data excludes the RealEstate10K-Cap validation and test clips; the best |
| one is released as `net_best_fid.tar`. |
|
|
| Validation split (2,500 clips), CLaTr evaluator; R@k = text-to-trajectory retrieval over the whole |
| split, in %: |
|
|
| | mode | FID ↓ | R@1 | R@3 | R@10 | matching | |
| |---|---|---|---|---|---| |
| | `noframe` | 0.042 | 2.1 | 5.2 | 13.5 | 0.289 | |
| | `firstframe` | 0.045 | 2.5 | 5.6 | 13.6 | 0.279 | |
| | `xframe` | 0.045 | 2.6 | 5.7 | 14.7 | 0.288 | |
|
|
| ## License |
|
|
| MIT, as the [code](https://github.com/linearalgebrayhz/TKCAM). TKCAM builds on |
| [MoMask](https://github.com/EricGuo5513/momask-codes) (MIT). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{yang2026tkcam, |
| title={TKCAM: Text and Keyframe to Camera Trajectory Generation}, |
| author={Yang, Haozhe and Dou, Zhiyang and Gu, Zekai and Lin, Cheng and Wang, Wenping and Liu, Yuan and Komura, Taku}, |
| year={2026}, |
| eprint={2610.11105}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2610.11105} |
| } |
| ``` |
|
|
