---
license: mit
library_name: pytorch
pipeline_tag: other
tags:
- camera-trajectory
- text-to-camera
- camera-control
datasets:
- yhz-linearalgebra/RealEstate10K-Cap
---

# TKCAM checkpoints

Pretrained checkpoints for **TKCAM: Text and Keyframe to Camera Trajectory Generation**
(NeurIPS 2026). TKCAM generates a camera trajectory from a text description, optionally conditioned
on a first frame or on up to 4 keyframes placed at chosen times.

[Paper (arXiv:2610.11105)](https://arxiv.org/abs/2610.11105) ·
[Code](https://github.com/linearalgebrayhz/TKCAM) ·
[Project page](https://linearalgebrayhz.github.io/projects/TKCAM/) ·
[Dataset](https://huggingface.co/datasets/yhz-linearalgebra/RealEstate10K-Cap)

| mode | conditioning | mask transformer | residual transformer |
|---|---|---|---|
| `noframe` | text | `mtrans_25kfixed_noframe_r4` (epoch 555) | `rtrans_25kfixed_noframe_r4` (epoch 40) |
| `firstframe` | text + first frame | `mtrans_25kfixed_1frame_r4` (epoch 480) | `rtrans_25kfixed_1frame_r4` (epoch 600) |
| `xframe` | text + up to 4 keyframes | `mtrans_25kfixed_xframe_r4` (epoch 600) | `rtrans_25kfixed_xframe_r4` (epoch 125) |

All modes share the residual VQ-VAE `rvq_w128_nq4_cb256_25kfixed_60ep` (4 quantizers, 256 codes,
window 128; epoch 25). Text is encoded with a frozen pretrained `t5-base` and images with a frozen
pretrained CLIP ViT-B/32 vision encoder (`openai/clip-vit-base-patch32`).

## Usage

```bash
git clone https://github.com/linearalgebrayhz/TKCAM.git && cd TKCAM
pip install -r requirements.txt && pip install -e .     # or: conda env create -f environment.yml
hf download yhz-linearalgebra/TKCAM --local-dir checkpoints

# Text only
python scripts/generate.py --mode noframe --duration 6 \
  --text "The camera dollies forward slowly, revealing the living room."
# Text + first frame
python scripts/generate.py --mode firstframe --duration 5 --first_frame room.jpg \
  --text "The camera pans left smoothly across the kitchen."
# Text + keyframes at chosen times (image@seconds, up to 4)
python scripts/generate.py --mode xframe --duration 8 \
  --keyframe hall.jpg@0 --keyframe stairs.jpg@4.5 \
  --text "The camera moves forward through the hallway, then tilts up toward the stairs."
```

Each sample is saved as `sample{i}_9d.npy` (`(T, 9)`: `[rot6d(6), x, y, z]`), `sample{i}_c2w.npy`
(`(T, 4, 4)`), an MP4 animation and `meta.json`. Trajectories are camera-to-world, relative to the
first frame, with OpenGL camera axes (+X right, +Y up, −Z forward), at 30 fps. Durations are clamped
to 3–11 s; the training clips are about 4.0–9.3 s. See the code repository for batch inference over a
prompts file and for training.

The T5 and CLIP encoder weights are not stored in these files: they are the unmodified public
weights and are downloaded from the Hugging Face Hub on first use.

## Files

The folder mirrors the layout the TKCAM code expects under `checkpoints/`:

```
realestate10k_rotmat/
  rvq_w128_nq4_cb256_25kfixed_60ep/{opt.txt, meta/mean.npy, meta/std.npy, model/net_best_recon.tar}
  {mtrans,rtrans}_25kfixed_{noframe,1frame,xframe}_r4/{opt.txt, net_best_fid.tar}
MANIFEST.json      # epochs, source checkpoint, parameter counts, sizes, sha256
```

Each `.tar` holds the model `state_dict` and the epoch (no optimizer state). Every transformer
file is named `net_best_fid.tar`, the name the TKCAM scripts load by default: it is the checkpoint
selected on validation FID (see [Training](#training)); the epochs are in the table above and in
`MANIFEST.json`. `opt.txt` is the
training configuration; local paths in it were replaced with placeholders.

## Training

- **Data:** the `train` split of [RealEstate10K-Cap](https://huggingface.co/datasets/yhz-linearalgebra/RealEstate10K-Cap)
  (21,250 clips), the released version of the dataset.
- **Recipe:** MoMask three-stage pipeline (residual VQ-VAE → masked transformer → residual
  transformer), 600 epochs per transformer, learning rate 5e-5. Visual conditioning uses
  the first frame (`firstframe`) or 0–4 randomly sampled keyframes per clip (`xframe`).
- **Checkpoint selection:** the VQ-VAE by validation reconstruction error. For each transformer,
  the saved checkpoints of the run (best validation FID during training, best accuracy, best loss,
  and the last epoch) were re-scored by FID on the 2,500-clip validation split with a CLaTr
  evaluator whose training data excludes the RealEstate10K-Cap validation and test clips; the best
  one is released as `net_best_fid.tar`.

Validation split (2,500 clips), CLaTr evaluator; R@k = text-to-trajectory retrieval over the whole
split, in %:

| mode | FID ↓ | R@1 | R@3 | R@10 | matching |
|---|---|---|---|---|---|
| `noframe` | 0.042 | 2.1 | 5.2 | 13.5 | 0.289 |
| `firstframe` | 0.045 | 2.5 | 5.6 | 13.6 | 0.279 |
| `xframe` | 0.045 | 2.6 | 5.7 | 14.7 | 0.288 |

## License

MIT, as the [code](https://github.com/linearalgebrayhz/TKCAM). TKCAM builds on
[MoMask](https://github.com/EricGuo5513/momask-codes) (MIT).

## Citation

```bibtex
@misc{yang2026tkcam,
  title={TKCAM: Text and Keyframe to Camera Trajectory Generation},
  author={Yang, Haozhe and Dou, Zhiyang and Gu, Zekai and Lin, Cheng and Wang, Wenping and Liu, Yuan and Komura, Taku},
  year={2026},
  eprint={2610.11105},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.11105}
}
```