CRePE / README.md
jinotter3's picture
Upload README.md with huggingface_hub
997b396 verified
|
Raw History Blame Contribute Delete
4.49 kB
---
license: cc-by-nc-sa-4.0
base_model: Wan-AI/Wan2.1-T2V-1.3B
pipeline_tag: text-to-video
library_name: crepe
tags:
- camera-control
- video-generation
- fisheye
- positional-encoding
- wan2.1
---
# CRePE: Curved Ray Expectation Positional Encoding (NeurIPS 2026)
Camera-, lens- and orientation-controlled text-to-video generation with a
lightweight **Geometric Attention Adapter** on a frozen
[Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B).
[Paper](https://arxiv.org/abs/2605.12938) · [Code](https://github.com/jinotter3/CRePE) · [Project page](https://crepe-neurips2026.github.io)
CRePE represents every image token by a depth-aware radial distribution along
its Unified Camera Model (UCM) ray and integrates the expected rotary phasor
along the *curved* path this distribution traces when projected into each query
view. The model follows a target camera trajectory under pinhole, wide-angle and
fisheye lenses (horizontal field of view up to 200°) given only a text prompt,
the trajectory and the lens parameters.
## Files
| file | description |
|---|---|
| `crepe.safetensors` | **CRePE**: camera-, lens- and orientation-controlled text-to-video (the camera-controlled videos of the project page) |
| `crepe_mix.safetensors` | **CRePE-Mix**: also accepts an external radial map (camera + radial-map conditioned generation, motion transfer) |
Each file holds only the adapter weights (36.2M parameters, bf16, ~72 MB); the
configuration is stored in the safetensors metadata. The frozen Wan2.1 weights
are loaded from `Wan-AI/Wan2.1-T2V-1.3B`.
**Note.** The CRePE checkpoint used for the paper's experiments was lost. `crepe.safetensors` comes from a
re-implementation of the method, re-trained with the recipe below, so its numbers differ slightly from the
paper's tables.
## Usage
```bash
git clone https://github.com/jinotter3/CRePE && cd CRePE && pip install -r requirements.txt && pip install -e .
hf download jinotter3/CRePE --local-dir checkpoints
python scripts/inference.py --checkpoint checkpoints/crepe.safetensors \
--input demo/lens_showcase.json --output outputs/lens_showcase --seeds 0
```
```python
from crepe import CRePEPipeline, load_trajectory, save_video
pipe = CRePEPipeline.from_pretrained("checkpoints/crepe.safetensors", device="cuda")
pose = load_trajectory("demo/trajectories/panshot_aQVLCO-OOqM-11-1-no_rot_aug_0.npy") # (81, 3, 4) camera-to-world
frames = pipe("A majestic red fox sitting on a moss-covered rock ...", pose=pose, x_fov=160.0, xi=1.5, seed=0)
save_video(frames, "fox.mp4", fps=16)
```
Output: 81 frames at 832×480, 16 fps. Inference takes about 4.5 minutes per clip on one B200 and about 18 GB of GPU memory.
## Training (`crepe.safetensors`)
* Data: PanShot train split (47,920 clips synthesised from PanFlow panoramic videos, captions from UCPE), 81 frames at 832×480.
* Frozen Wan2.1-T2V-1.3B; one Geometric Attention Adapter per DiT block (width 192), CRePE in blocks 10–19
(0-indexed; K = 5 radial breakpoints), UCPE relative-ray encoding elsewhere, absolute orientation map in every block.
* Loss: flow matching + 1e-3 × confidence-weighted radial-distance loss (α = 0.2) on block 15, with pseudo labels
from [UniK3D](https://github.com/lpiccinelli-eth/UniK3D) (used only during training, not an inference input).
* AdamW, lr 1e-4, 10k steps, bf16, effective batch 16 (2 GPUs × 8). Config: `configs/train_crepe.yaml` in the code repository.
## Limitations
The model inherits the content distribution and the limitations of Wan2.1-T2V-1.3B and of the PanShot training
data. Camera control assumes mostly static scenes; trajectories far outside the training distribution (very fast
motion, unusual scales) are followed less reliably. Generated videos may contain artifacts and should not be
presented as real footage.
## Citation
```bibtex
@inproceedings{jin2026crepe,
title = {CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation},
author = {Jin, Seonghyun and Kim, Youngmin and Park, Sunwoo and Ye, Jong Chul},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}
```
## License
The weights in this repository are released under
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (they were trained with pseudo-labels from
UniK3D, CC BY-NC-SA 4.0). The code on GitHub is Apache-2.0; the base model Wan2.1-T2V-1.3B is Apache-2.0 by its authors.