|
Download README.md from jinotter3/CRePE: direct link, hf CLI and curl.
- Browser
- Download file 4.49 kB
-
https://huggingface.co/jinotter3/CRePE/resolve/main/README.md
- Command line
-
hf download hf://jinotter3/CRePE/README.md
-
curl -L -o README.md https://huggingface.co/jinotter3/CRePE/resolve/main/README.md
4.49 kB
| license: cc-by-nc-sa-4.0 | |
| base_model: Wan-AI/Wan2.1-T2V-1.3B | |
| pipeline_tag: text-to-video | |
| library_name: crepe | |
| tags: | |
| - camera-control | |
| - video-generation | |
| - fisheye | |
| - positional-encoding | |
| - wan2.1 | |
| # CRePE: Curved Ray Expectation Positional Encoding (NeurIPS 2026) | |
| Camera-, lens- and orientation-controlled text-to-video generation with a | |
| lightweight **Geometric Attention Adapter** on a frozen | |
| [Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B). | |
| [Paper](https://arxiv.org/abs/2605.12938) · [Code](https://github.com/jinotter3/CRePE) · [Project page](https://crepe-neurips2026.github.io) | |
| CRePE represents every image token by a depth-aware radial distribution along | |
| its Unified Camera Model (UCM) ray and integrates the expected rotary phasor | |
| along the *curved* path this distribution traces when projected into each query | |
| view. The model follows a target camera trajectory under pinhole, wide-angle and | |
| fisheye lenses (horizontal field of view up to 200°) given only a text prompt, | |
| the trajectory and the lens parameters. | |
| ## Files | |
| | file | description | | |
| |---|---| | |
| | `crepe.safetensors` | **CRePE**: camera-, lens- and orientation-controlled text-to-video (the camera-controlled videos of the project page) | | |
| | `crepe_mix.safetensors` | **CRePE-Mix**: also accepts an external radial map (camera + radial-map conditioned generation, motion transfer) | | |
| Each file holds only the adapter weights (36.2M parameters, bf16, ~72 MB); the | |
| configuration is stored in the safetensors metadata. The frozen Wan2.1 weights | |
| are loaded from `Wan-AI/Wan2.1-T2V-1.3B`. | |
| **Note.** The CRePE checkpoint used for the paper's experiments was lost. `crepe.safetensors` comes from a | |
| re-implementation of the method, re-trained with the recipe below, so its numbers differ slightly from the | |
| paper's tables. | |
| ## Usage | |
| ```bash | |
| git clone https://github.com/jinotter3/CRePE && cd CRePE && pip install -r requirements.txt && pip install -e . | |
| hf download jinotter3/CRePE --local-dir checkpoints | |
| python scripts/inference.py --checkpoint checkpoints/crepe.safetensors \ | |
| --input demo/lens_showcase.json --output outputs/lens_showcase --seeds 0 | |
| ``` | |
| ```python | |
| from crepe import CRePEPipeline, load_trajectory, save_video | |
| pipe = CRePEPipeline.from_pretrained("checkpoints/crepe.safetensors", device="cuda") | |
| pose = load_trajectory("demo/trajectories/panshot_aQVLCO-OOqM-11-1-no_rot_aug_0.npy") # (81, 3, 4) camera-to-world | |
| frames = pipe("A majestic red fox sitting on a moss-covered rock ...", pose=pose, x_fov=160.0, xi=1.5, seed=0) | |
| save_video(frames, "fox.mp4", fps=16) | |
| ``` | |
| Output: 81 frames at 832×480, 16 fps. Inference takes about 4.5 minutes per clip on one B200 and about 18 GB of GPU memory. | |
| ## Training (`crepe.safetensors`) | |
| * Data: PanShot train split (47,920 clips synthesised from PanFlow panoramic videos, captions from UCPE), 81 frames at 832×480. | |
| * Frozen Wan2.1-T2V-1.3B; one Geometric Attention Adapter per DiT block (width 192), CRePE in blocks 10–19 | |
| (0-indexed; K = 5 radial breakpoints), UCPE relative-ray encoding elsewhere, absolute orientation map in every block. | |
| * Loss: flow matching + 1e-3 × confidence-weighted radial-distance loss (α = 0.2) on block 15, with pseudo labels | |
| from [UniK3D](https://github.com/lpiccinelli-eth/UniK3D) (used only during training, not an inference input). | |
| * AdamW, lr 1e-4, 10k steps, bf16, effective batch 16 (2 GPUs × 8). Config: `configs/train_crepe.yaml` in the code repository. | |
| ## Limitations | |
| The model inherits the content distribution and the limitations of Wan2.1-T2V-1.3B and of the PanShot training | |
| data. Camera control assumes mostly static scenes; trajectories far outside the training distribution (very fast | |
| motion, unusual scales) are followed less reliably. Generated videos may contain artifacts and should not be | |
| presented as real footage. | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{jin2026crepe, | |
| title = {CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation}, | |
| author = {Jin, Seonghyun and Kim, Youngmin and Park, Sunwoo and Ye, Jong Chul}, | |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, | |
| year = {2026} | |
| } | |
| ``` | |
| ## License | |
| The weights in this repository are released under | |
| [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (they were trained with pseudo-labels from | |
| UniK3D, CC BY-NC-SA 4.0). The code on GitHub is Apache-2.0; the base model Wan2.1-T2V-1.3B is Apache-2.0 by its authors. | |