Image-to-Video
Cosmos
world-model
autonomous-driving
video-prediction
thermal
lidar
adverse-weather
lora
Instructions to use SORA-Lab/TriCosmos-2.5-predict with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use SORA-Lab/TriCosmos-2.5-predict with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from SORA-Lab/TriCosmos-2.5-predict: direct link, hf CLI and curl.
- Browser
- Download file 10.4 kB
-
https://huggingface.co/SORA-Lab/TriCosmos-2.5-predict/resolve/main/README.md
- Command line
-
hf download hf://SORA-Lab/TriCosmos-2.5-predict/README.md
-
curl -L -o README.md https://huggingface.co/SORA-Lab/TriCosmos-2.5-predict/resolve/main/README.md
10.4 kB
| license: other | |
| license_name: nvidia-open-model-license | |
| license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ | |
| base_model: nvidia/Cosmos-Predict2.5-2B | |
| base_model_relation: finetune | |
| library_name: cosmos | |
| pipeline_tag: image-to-video | |
| tags: | |
| - world-model | |
| - autonomous-driving | |
| - video-prediction | |
| - thermal | |
| - lidar | |
| - adverse-weather | |
| - cosmos | |
| - lora | |
| # TriCosmos-2.5-predict | |
| **A driving world model that forecasts RGB, thermal, and LiDAR depth together.** | |
| Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night. | |
| - **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA | |
| - **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR) | |
| - **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ2 grid of sensor tiles | |
| > Paper under review. Full training and evaluation code will be released with the paper. | |
| --- | |
| ## Contents | |
| 1. [Repository files](#repository-files) | |
| 2. [Quick start](#quick-start) | |
| 3. [Input format](#input-format) | |
| 4. [Reading the output](#reading-the-output) | |
| 5. [Model and training details](#model-and-training-details) | |
| 6. [Results](#results) | |
| 7. [Limitations](#limitations) | |
| 8. [License and citation](#license-and-citation) | |
| --- | |
| ## Repository files | |
| | File | Description | | |
| |---|---| | |
| | `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. | | |
| | `TriCosmos_inference.ipynb` | Google Colab notebook: install, download, forecast, and a side-by-side comparison video. | | |
| | `examples/light_snow_example.mp4` | Test window from the DSERT-RoLL validation set (Light Snow, night): 5 observed + 20 real future frames. | | |
| | `examples/light_snow_example.json` | The exact caption used for that window. | | |
| | `config/experiment.json` | Model options used in training. Needed to rebuild the model. | | |
| | `config/export.json` | Checkpoint name and training step. | | |
| | `config/train_tri_lora.yaml` | Full training recipe. | | |
| --- | |
| ## Quick start | |
| ### Requirements | |
| - An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ704. | |
| | GPU | Memory | Works? | | |
| |---|---|---| | |
| | NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | β tested | | |
| | NVIDIA H100 | 80 GB | β tested | | |
| | NVIDIA A100 80 GB | 80 GB | β used for the paper's results | | |
| | Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | β not enough memory | | |
| - A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories. | |
| ### Option 1: Google Colab (easiest) | |
| 1. Download `TriCosmos_inference.ipynb` and open it in Colab (**File β Upload notebook**). | |
| 2. Select **Runtime β Change runtime type β G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory. | |
| 3. Run all cells from top to bottom and paste your Hugging Face token when asked. | |
| The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video: | |
| **input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**. | |
| ### Option 2: Local machine | |
| ```bash | |
| # 1. Cosmos code, exact version used for this model | |
| git clone https://github.com/nvidia-cosmos/cosmos-predict2.5 | |
| cd cosmos-predict2.5 | |
| git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7 | |
| uv sync --extra=cu128 --python 3.10 | |
| uv pip install --python .venv/bin/python "imageio[pyav]" av | |
| # 2. This model | |
| hf auth login | |
| hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos | |
| # 3. Forecast the example window | |
| .venv/bin/python ../tricosmos/tricosmos_infer.py \ | |
| --model-dir ../tricosmos \ | |
| --input ../tricosmos/examples/light_snow_example.mp4 \ | |
| --prompt-json ../tricosmos/examples/light_snow_example.json \ | |
| --out ../forecast_tricosmos | |
| # Optional: the zero-shot base model on the same input | |
| .venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \ | |
| --model-dir ../tricosmos \ | |
| --input ../tricosmos/examples/light_snow_example.mp4 \ | |
| --prompt-json ../tricosmos/examples/light_snow_example.json \ | |
| --out ../forecast_zeroshot | |
| ``` | |
| Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ704. | |
| Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included. | |
| --- | |
| ## Input format | |
| The model expects a video of **1280Γ704** frames at **10 Hz**, each made of four 640Γ352 tiles: | |
| | | left | right | | |
| |---|---|---| | |
| | **top** | RGB camera | thermal camera | | |
| | **bottom** | LiDAR depth | black (unused) | | |
| Only the **first 5 frames** are used as input. | |
| | Tile | How it is made | | |
| |---|---| | |
| | **RGB** | Left RGB camera, 1280Γ704, area-averaged to 640Γ352. | | |
| | **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ704, area-averaged to 640Γ352. Shows the thermal camera's own view (not warped to the RGB image). | | |
| | **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ5 elliptical dilation and a 5Γ5 morphological closing, then max-pooled over 2Γ2 blocks (the nearest depth wins). | | |
| | **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. | | |
| **LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels: | |
| ``` | |
| q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return | |
| ``` | |
| Near objects are bright, far objects dark, and empty pixels black. | |
| **Caption.** The text prompt names the 2Γ2 layout, then the weather and the lighting, in this format (from the Light Snow example): | |
| ``` | |
| driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light | |
| ``` | |
| For other data, change only the weather and lighting at the end. | |
| --- | |
| ## Reading the output | |
| The output has 25 frames: frames 0β4 reproduce the input, and frames 5β24 are the forecast (0.1 s to 2.0 s ahead). | |
| 1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`. | |
| 2. Upsample each tile to 1280Γ704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR. | |
| 3. Decode LiDAR depth in meters: | |
| ```python | |
| depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan) | |
| ``` | |
| --- | |
| ## Model and training details | |
| | Setting | Value | | |
| |---|---| | |
| | Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) | | |
| | Cosmos code | `cosmos-predict2.5`, commit `a2c298b0a3df3778b973fe65e9e58877b292d8a7` | | |
| | LoRA | rank 16, alpha 32, on `q_proj`, `k_proj`, `v_proj`, `output_proj`, `mlp.layer1`, `mlp.layer2` of every transformer block | | |
| | Frozen | base weights, text encoder, video tokenizer | | |
| | Conditioning | 2 latent frames (= 5 observed frames) | | |
| | Optimizer | learning rate 1e-4, weight decay 1e-3 | | |
| | Schedule | 50 linear warm-up steps, then linear decay to 10% | | |
| | Training length | 500 steps, bf16, 93-frame clips | | |
| | Weather balancing | clips sampled with weights β `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition | | |
| | Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** | | |
| --- | |
| ## Results | |
| DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points. | |
| | Method | RGB PSNR β | RGB LPIPS β | Thermal PSNR β | Thermal LPIPS β | LiDAR MAE (m) β | LiDAR Ξ΄1 β | | |
| |---|---|---|---|---|---|---| | |
| | Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 | | |
| | Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 | | |
| | **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** | | |
| - **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error. | |
| - **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS. | |
| --- | |
| ## Limitations | |
| - **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast. | |
| - **Resolution:** each sensor is seen at half resolution (640Γ352) inside the grid. | |
| - **Heavy Snow:** only two validation sequences, so results in this condition are not reliable. | |
| - **Thermal registration:** the thermal tile is not registered to the RGB image. | |
| - **Speed:** about 9 minutes per forecast on an RTX PRO 6000 (model loading included); not for real-time use. | |
| - **Memory:** needs a GPU with at least 48 GB. | |
| - **No action input:** the model is not conditioned on driving actions. | |
| - **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions. | |
| --- | |
| ## License and citation | |
| **License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data. | |
| **Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026). | |
| **Citation.** The paper is under review. A citation will be added after publication. | |
| --- | |
| --- | |
| license: apache-2.0 | |
| --- |