README.md

---
license: cc-by-nc-sa-4.0
tags:
- world-model
- autonomous-driving
- video-generation
- autoregressive
- multiview
paper: arxiv.org/abs/2602.20685
---

# [CVPR 2026] RayNova: Scale-Temporal Autoregressive World Modeling in Ray Space

**[Project Page](https://raynova-ai.github.io/) | [Paper](https://arxiv.org/abs/2602.20685) | [Code](https://github.com/Applied-Intuition-Open-Source/RayNova)**

World foundation models aim to simulate the evolution of the real world with physically plausible behavior. Unlike prior methods that handle spatial and temporal correlations separately, RAYNOVA is a geometry-agnostic multiview world model for driving scenarios that employs a dual-causal autoregressive framework. It follows both scale-wise and temporal topological orders in the autoregressive process, and leverages global attention for unified 4D spatio-temporal reasoning. Different from existing works that impose strong 3D geometric priors, RAYNOVA constructs an isotropic spatio-temporal representation across views, frames, and scales based on relative Plücker-ray positional encoding, enabling robust generalization to diverse camera setups and ego motions.

## Model

This release is the **RayNova 2B** model (Infinity-2B backbone, depth 32, width 2048), trained exclusively on **publicly available data** — nuPlan and nuScenes in [ScenarioNet](https://github.com/metadriverse/scenarionet) format. Performance may be slightly lower than our internal model trained on the full dataset.

| File | Description |
|------|-------------|
| `slim-gpt_model_latest.pth` | RayNova transformer weights (plain PyTorch `state_dict`) |
| `config.yaml` | Training configuration (defines model hyperparameters used at inference) |

Companion weights (not included here, from their original releases):

- VAE: [`FoundationVision/Infinity`](https://huggingface.co/FoundationVision/Infinity) `infinity_vae_d32reg.pth`
- Text encoder: [`google/flan-t5-xl`](https://huggingface.co/google/flan-t5-xl)

## Usage

```bash
git clone https://github.com/Applied-Intuition-Open-Source/RayNova
cd RayNova
mkdir -p weights && cd weights

wget https://huggingface.co/FoundationVision/Infinity/resolve/main/infinity_vae_d32reg.pth
git clone https://huggingface.co/google/flan-t5-xl
huggingface-cli download AppliedIntuitionResearch/RayNova --local-dir raynova

cd ..
```

Then follow the inference notebook [`tools/interactive_infer.ipynb`](https://github.com/Applied-Intuition-Open-Source/RayNova/blob/main/tools/interactive_infer.ipynb), setting `model_path` to `weights/raynova/slim-gpt_model_latest.pth` (the accompanying `config.yaml` is picked up from the same directory). Set `enable_model_cache=False` in the notebook args since this is already a slim (inference-only) state dict.

**GPU memory.** Inference at 384×672 (`pn='0.25M'`) needs more than 24 GB of GPU memory (A100/H100-class recommended) — the autoregressive KV cache alone saturates a 24 GB card over a multi-window rollout. On a 24 GB card (e.g. RTX 4090), use 192×336 (`pn='0.06M'`), which runs out of the box with all 8 views (~20 GB peak).

## Citation

```bibtex
@article{xie2026raynova,
  title={RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space},
  author={Xie, Yichen and Peng, Chensheng and Abdelfattah, Mazen and Hu, Yihan and Yang, Jiezhi and Higgins, Eric and Brigden, Ryan and Tomizuka, Masayoshi and Zhan, Wei},
  journal={arXiv preprint arXiv:2602.20685},
  year={2026}
}
```