CRePE: Curved Ray Expectation Positional Encoding (NeurIPS 2026)

Camera-, lens- and orientation-controlled text-to-video generation with a lightweight Geometric Attention Adapter on a frozen Wan2.1-T2V-1.3B.

Paper · Code · Project page

CRePE represents every image token by a depth-aware radial distribution along its Unified Camera Model (UCM) ray and integrates the expected rotary phasor along the curved path this distribution traces when projected into each query view. The model follows a target camera trajectory under pinhole, wide-angle and fisheye lenses (horizontal field of view up to 200°) given only a text prompt, the trajectory and the lens parameters.

Files

file	description
crepe.safetensors	CRePE: camera-, lens- and orientation-controlled text-to-video (the camera-controlled videos of the project page)
crepe_mix.safetensors	CRePE-Mix: also accepts an external radial map (camera + radial-map conditioned generation, motion transfer)

Each file holds only the adapter weights (36.2M parameters, bf16, ~72 MB); the configuration is stored in the safetensors metadata. The frozen Wan2.1 weights are loaded from Wan-AI/Wan2.1-T2V-1.3B.

Note. The CRePE checkpoint used for the paper's experiments was lost. crepe.safetensors comes from a re-implementation of the method, re-trained with the recipe below, so its numbers differ slightly from the paper's tables.

Usage

git clone https://github.com/jinotter3/CRePE && cd CRePE && pip install -r requirements.txt && pip install -e .
hf download jinotter3/CRePE --local-dir checkpoints
python scripts/inference.py --checkpoint checkpoints/crepe.safetensors \
    --input demo/lens_showcase.json --output outputs/lens_showcase --seeds 0

from crepe import CRePEPipeline, load_trajectory, save_video

pipe = CRePEPipeline.from_pretrained("checkpoints/crepe.safetensors", device="cuda")
pose = load_trajectory("demo/trajectories/panshot_aQVLCO-OOqM-11-1-no_rot_aug_0.npy")   # (81, 3, 4) camera-to-world
frames = pipe("A majestic red fox sitting on a moss-covered rock ...", pose=pose, x_fov=160.0, xi=1.5, seed=0)
save_video(frames, "fox.mp4", fps=16)

Output: 81 frames at 832×480, 16 fps. Inference takes about 4.5 minutes per clip on one B200 and about 18 GB of GPU memory.

Training (crepe.safetensors)

• Data: PanShot train split (47,920 clips synthesised from PanFlow panoramic videos, captions from UCPE), 81 frames at 832×480.
• Frozen Wan2.1-T2V-1.3B; one Geometric Attention Adapter per DiT block (width 192), CRePE in blocks 10–19 (0-indexed; K = 5 radial breakpoints), UCPE relative-ray encoding elsewhere, absolute orientation map in every block.
• Loss: flow matching + 1e-3 × confidence-weighted radial-distance loss (α = 0.2) on block 15, with pseudo labels from UniK3D (used only during training, not an inference input).
• AdamW, lr 1e-4, 10k steps, bf16, effective batch 16 (2 GPUs × 8). Config: configs/train_crepe.yaml in the code repository.

Limitations

The model inherits the content distribution and the limitations of Wan2.1-T2V-1.3B and of the PanShot training data. Camera control assumes mostly static scenes; trajectories far outside the training distribution (very fast motion, unusual scales) are followed less reliably. Generated videos may contain artifacts and should not be presented as real footage.

Citation

@inproceedings{jin2026crepe,
  title     = {CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation},
  author    = {Jin, Seonghyun and Kim, Youngmin and Park, Sunwoo and Ye, Jong Chul},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

License

The weights in this repository are released under CC BY-NC-SA 4.0 (they were trained with pseudo-labels from UniK3D, CC BY-NC-SA 4.0). The code on GitHub is Apache-2.0; the base model Wan2.1-T2V-1.3B is Apache-2.0 by its authors.