Image-to-Video
Cosmos
world-model
autonomous-driving
video-prediction
thermal
lidar
adverse-weather
lora
Instructions to use SORA-Lab/TriCosmos-2.5-predict with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use SORA-Lab/TriCosmos-2.5-predict with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
File size: 10,378 Bytes
eccdf58 85d3024 eccdf58 688c50c eccdf58 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 eccdf58 f8a3da2 3dd1aff f8a3da2 eccdf58 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c ae71b70 f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c eccdf58 688c50c f8a3da2 688c50c eccdf58 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c eccdf58 f8a3da2 eccdf58 f8a3da2 688c50c f8a3da2 688c50c eccdf58 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 688c50c f8a3da2 eccdf58 688c50c eccdf58 17eeed1 7cc687d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 | ---
license: other
license_name: nvidia-open-model-license
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
base_model: nvidia/Cosmos-Predict2.5-2B
base_model_relation: finetune
library_name: cosmos
pipeline_tag: image-to-video
tags:
- world-model
- autonomous-driving
- video-prediction
- thermal
- lidar
- adverse-weather
- cosmos
- lora
---
# TriCosmos-2.5-predict
**A driving world model that forecasts RGB, thermal, and LiDAR depth together.**
Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night.
- **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA
- **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR)
- **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ2 grid of sensor tiles
> Paper under review. Full training and evaluation code will be released with the paper.
---
## Contents
1. [Repository files](#repository-files)
2. [Quick start](#quick-start)
3. [Input format](#input-format)
4. [Reading the output](#reading-the-output)
5. [Model and training details](#model-and-training-details)
6. [Results](#results)
7. [Limitations](#limitations)
8. [License and citation](#license-and-citation)
---
## Repository files
| File | Description |
|---|---|
| `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
| `TriCosmos_inference.ipynb` | Google Colab notebook: install, download, forecast, and a side-by-side comparison video. |
| `examples/light_snow_example.mp4` | Test window from the DSERT-RoLL validation set (Light Snow, night): 5 observed + 20 real future frames. |
| `examples/light_snow_example.json` | The exact caption used for that window. |
| `config/experiment.json` | Model options used in training. Needed to rebuild the model. |
| `config/export.json` | Checkpoint name and training step. |
| `config/train_tri_lora.yaml` | Full training recipe. |
---
## Quick start
### Requirements
- An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ704.
| GPU | Memory | Works? |
|---|---|---|
| NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | β
tested |
| NVIDIA H100 | 80 GB | β
tested |
| NVIDIA A100 80 GB | 80 GB | β
used for the paper's results |
| Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | β not enough memory |
- A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
### Option 1: Google Colab (easiest)
1. Download `TriCosmos_inference.ipynb` and open it in Colab (**File β Upload notebook**).
2. Select **Runtime β Change runtime type β G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
3. Run all cells from top to bottom and paste your Hugging Face token when asked.
The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video:
**input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**.
### Option 2: Local machine
```bash
# 1. Cosmos code, exact version used for this model
git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
cd cosmos-predict2.5
git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
uv sync --extra=cu128 --python 3.10
uv pip install --python .venv/bin/python "imageio[pyav]" av
# 2. This model
hf auth login
hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
# 3. Forecast the example window
.venv/bin/python ../tricosmos/tricosmos_infer.py \
--model-dir ../tricosmos \
--input ../tricosmos/examples/light_snow_example.mp4 \
--prompt-json ../tricosmos/examples/light_snow_example.json \
--out ../forecast_tricosmos
# Optional: the zero-shot base model on the same input
.venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
--model-dir ../tricosmos \
--input ../tricosmos/examples/light_snow_example.mp4 \
--prompt-json ../tricosmos/examples/light_snow_example.json \
--out ../forecast_zeroshot
```
Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ704.
Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
---
## Input format
The model expects a video of **1280Γ704** frames at **10 Hz**, each made of four 640Γ352 tiles:
| | left | right |
|---|---|---|
| **top** | RGB camera | thermal camera |
| **bottom** | LiDAR depth | black (unused) |
Only the **first 5 frames** are used as input.
| Tile | How it is made |
|---|---|
| **RGB** | Left RGB camera, 1280Γ704, area-averaged to 640Γ352. |
| **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ704, area-averaged to 640Γ352. Shows the thermal camera's own view (not warped to the RGB image). |
| **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ5 elliptical dilation and a 5Γ5 morphological closing, then max-pooled over 2Γ2 blocks (the nearest depth wins). |
| **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
**LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
```
q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
```
Near objects are bright, far objects dark, and empty pixels black.
**Caption.** The text prompt names the 2Γ2 layout, then the weather and the lighting, in this format (from the Light Snow example):
```
driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
```
For other data, change only the weather and lighting at the end.
---
## Reading the output
The output has 25 frames: frames 0β4 reproduce the input, and frames 5β24 are the forecast (0.1 s to 2.0 s ahead).
1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`.
2. Upsample each tile to 1280Γ704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR.
3. Decode LiDAR depth in meters:
```python
depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
```
---
## Model and training details
| Setting | Value |
|---|---|
| Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
| Cosmos code | `cosmos-predict2.5`, commit `a2c298b0a3df3778b973fe65e9e58877b292d8a7` |
| LoRA | rank 16, alpha 32, on `q_proj`, `k_proj`, `v_proj`, `output_proj`, `mlp.layer1`, `mlp.layer2` of every transformer block |
| Frozen | base weights, text encoder, video tokenizer |
| Conditioning | 2 latent frames (= 5 observed frames) |
| Optimizer | learning rate 1e-4, weight decay 1e-3 |
| Schedule | 50 linear warm-up steps, then linear decay to 10% |
| Training length | 500 steps, bf16, 93-frame clips |
| Weather balancing | clips sampled with weights β `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition |
| Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** |
---
## Results
DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
| Method | RGB PSNR β | RGB LPIPS β | Thermal PSNR β | Thermal LPIPS β | LiDAR MAE (m) β | LiDAR Ξ΄1 β |
|---|---|---|---|---|---|---|
| Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
| Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 |
| **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** |
- **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
- **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
---
## Limitations
- **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
- **Resolution:** each sensor is seen at half resolution (640Γ352) inside the grid.
- **Heavy Snow:** only two validation sequences, so results in this condition are not reliable.
- **Thermal registration:** the thermal tile is not registered to the RGB image.
- **Speed:** about 9 minutes per forecast on an RTX PRO 6000 (model loading included); not for real-time use.
- **Memory:** needs a GPU with at least 48 GB.
- **No action input:** the model is not conditioned on driving actions.
- **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions.
---
## License and citation
**License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
**Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026).
**Citation.** The paper is under review. A citation will be added after publication.
---
---
license: apache-2.0
--- |