Image-to-Video
Cosmos
world-model
autonomous-driving
video-prediction
thermal
lidar
adverse-weather
lora
Instructions to use SORA-Lab/TriCosmos-2.5-predict with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use SORA-Lab/TriCosmos-2.5-predict with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -16,21 +16,22 @@ tags:
|
|
| 16 |
- cosmos
|
| 17 |
- lora
|
| 18 |
---
|
| 19 |
-
|
| 20 |
# TriCosmos-2.5-predict
|
| 21 |
-
|
| 22 |
**A driving world model that forecasts RGB, thermal, and LiDAR depth together.**
|
| 23 |
Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night.
|
| 24 |
-
|
| 25 |
- **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA
|
| 26 |
- **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR)
|
| 27 |
- **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ2 grid of sensor tiles
|
|
|
|
| 28 |
> Paper under review. Full training and evaluation code will be released with the paper.
|
| 29 |
-
|
| 30 |
---
|
| 31 |
-
|
| 32 |
## Contents
|
| 33 |
-
|
| 34 |
1. [Repository files](#repository-files)
|
| 35 |
2. [Quick start](#quick-start)
|
| 36 |
3. [Input format](#input-format)
|
|
@@ -39,10 +40,11 @@ Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames
|
|
| 39 |
6. [Results](#results)
|
| 40 |
7. [Limitations](#limitations)
|
| 41 |
8. [License and citation](#license-and-citation)
|
|
|
|
| 42 |
---
|
| 43 |
-
|
| 44 |
## Repository files
|
| 45 |
-
|
| 46 |
| File | Description |
|
| 47 |
|---|---|
|
| 48 |
| `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
|
|
@@ -53,31 +55,35 @@ Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames
|
|
| 53 |
| `config/experiment.json` | Model options used in training. Needed to rebuild the model. |
|
| 54 |
| `config/export.json` | Checkpoint name and training step. |
|
| 55 |
| `config/train_tri_lora.yaml` | Full training recipe. |
|
| 56 |
-
|
| 57 |
---
|
| 58 |
-
|
| 59 |
## Quick start
|
| 60 |
-
|
| 61 |
### Requirements
|
| 62 |
-
|
| 63 |
- An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ704.
|
|
|
|
| 64 |
| GPU | Memory | Works? |
|
| 65 |
|---|---|---|
|
| 66 |
| NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | β
tested |
|
| 67 |
| NVIDIA H100 | 80 GB | β
tested |
|
| 68 |
| NVIDIA A100 80 GB | 80 GB | β
used for the paper's results |
|
| 69 |
| Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | β not enough memory |
|
|
|
|
| 70 |
- A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
|
|
|
|
| 71 |
### Option 1: Google Colab (easiest)
|
| 72 |
-
|
| 73 |
1. Download `notebooks/TriCosmos_inference.ipynb` and open it in Colab (**File β Upload notebook**).
|
| 74 |
2. Select **Runtime β Change runtime type β G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
|
| 75 |
3. Run all cells from top to bottom and paste your Hugging Face token when asked.
|
|
|
|
| 76 |
The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video:
|
| 77 |
**input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**.
|
| 78 |
-
|
| 79 |
### Option 2: Local machine
|
| 80 |
-
|
| 81 |
```bash
|
| 82 |
# 1. Cosmos code, exact version used for this model
|
| 83 |
git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
|
|
@@ -85,18 +91,18 @@ cd cosmos-predict2.5
|
|
| 85 |
git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
|
| 86 |
uv sync --extra=cu128 --python 3.10
|
| 87 |
uv pip install --python .venv/bin/python "imageio[pyav]" av
|
| 88 |
-
|
| 89 |
# 2. This model
|
| 90 |
hf auth login
|
| 91 |
hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
|
| 92 |
-
|
| 93 |
# 3. Forecast the example window
|
| 94 |
.venv/bin/python ../tricosmos/tricosmos_infer.py \
|
| 95 |
--model-dir ../tricosmos \
|
| 96 |
--input ../tricosmos/examples/light_snow_example.mp4 \
|
| 97 |
--prompt-json ../tricosmos/examples/light_snow_example.json \
|
| 98 |
--out ../forecast_tricosmos
|
| 99 |
-
|
| 100 |
# Optional: the zero-shot base model on the same input
|
| 101 |
.venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
|
| 102 |
--model-dir ../tricosmos \
|
|
@@ -104,63 +110,64 @@ hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
|
|
| 104 |
--prompt-json ../tricosmos/examples/light_snow_example.json \
|
| 105 |
--out ../forecast_zeroshot
|
| 106 |
```
|
| 107 |
-
|
| 108 |
Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ704.
|
| 109 |
Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
|
| 110 |
-
|
| 111 |
---
|
| 112 |
-
|
| 113 |
## Input format
|
| 114 |
-
|
| 115 |
The model expects a video of **1280Γ704** frames at **10 Hz**, each made of four 640Γ352 tiles:
|
| 116 |
-
|
| 117 |
| | left | right |
|
| 118 |
|---|---|---|
|
| 119 |
| **top** | RGB camera | thermal camera |
|
| 120 |
| **bottom** | LiDAR depth | black (unused) |
|
| 121 |
-
|
| 122 |
Only the **first 5 frames** are used as input.
|
| 123 |
-
|
| 124 |
| Tile | How it is made |
|
| 125 |
|---|---|
|
| 126 |
| **RGB** | Left RGB camera, 1280Γ704, area-averaged to 640Γ352. |
|
| 127 |
| **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ704, area-averaged to 640Γ352. Shows the thermal camera's own view (not warped to the RGB image). |
|
| 128 |
| **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ5 elliptical dilation and a 5Γ5 morphological closing, then max-pooled over 2Γ2 blocks (the nearest depth wins). |
|
| 129 |
| **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
|
| 130 |
-
|
| 131 |
**LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
|
| 132 |
-
|
| 133 |
```
|
| 134 |
q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
|
| 135 |
```
|
| 136 |
-
|
| 137 |
Near objects are bright, far objects dark, and empty pixels black.
|
| 138 |
-
|
| 139 |
**Caption.** The text prompt names the 2Γ2 layout, then the weather and the lighting, in this format (from the Light Snow example):
|
| 140 |
-
|
| 141 |
```
|
| 142 |
driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
|
| 143 |
```
|
| 144 |
-
|
| 145 |
For other data, change only the weather and lighting at the end.
|
| 146 |
-
|
| 147 |
---
|
| 148 |
-
|
| 149 |
## Reading the output
|
| 150 |
-
|
| 151 |
The output has 25 frames: frames 0β4 reproduce the input, and frames 5β24 are the forecast (0.1 s to 2.0 s ahead).
|
| 152 |
-
|
| 153 |
1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`.
|
| 154 |
2. Upsample each tile to 1280Γ704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR.
|
| 155 |
3. Decode LiDAR depth in meters:
|
|
|
|
| 156 |
```python
|
| 157 |
depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
|
| 158 |
```
|
| 159 |
-
|
| 160 |
---
|
| 161 |
-
|
| 162 |
## Model and training details
|
| 163 |
-
|
| 164 |
| Setting | Value |
|
| 165 |
|---|---|
|
| 166 |
| Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
|
|
@@ -173,25 +180,26 @@ depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
|
|
| 173 |
| Training length | 500 steps, bf16, 93-frame clips |
|
| 174 |
| Weather balancing | clips sampled with weights β `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition |
|
| 175 |
| Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** |
|
| 176 |
-
|
| 177 |
---
|
| 178 |
-
|
| 179 |
## Results
|
| 180 |
-
|
| 181 |
DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
|
| 182 |
-
|
| 183 |
| Method | RGB PSNR β | RGB LPIPS β | Thermal PSNR β | Thermal LPIPS β | LiDAR MAE (m) β | LiDAR Ξ΄1 β |
|
| 184 |
|---|---|---|---|---|---|---|
|
| 185 |
| Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
|
| 186 |
| Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 |
|
| 187 |
| **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** |
|
| 188 |
-
|
| 189 |
- **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
|
| 190 |
- **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
|
|
|
|
| 191 |
---
|
| 192 |
-
|
| 193 |
## Limitations
|
| 194 |
-
|
| 195 |
- **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
|
| 196 |
- **Resolution:** each sensor is seen at half resolution (640Γ352) inside the grid.
|
| 197 |
- **Heavy Snow:** only two validation sequences, so results in this condition are not reliable.
|
|
@@ -200,17 +208,17 @@ DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300
|
|
| 200 |
- **Memory:** needs a GPU with at least 48 GB.
|
| 201 |
- **No action input:** the model is not conditioned on driving actions.
|
| 202 |
- **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions.
|
|
|
|
| 203 |
---
|
| 204 |
-
|
| 205 |
## License and citation
|
| 206 |
-
|
| 207 |
**License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
|
| 208 |
-
|
| 209 |
**Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026).
|
| 210 |
-
|
| 211 |
-
**Citation.** The paper is under review. A citation will be added after publication.
|
| 212 |
-
|
| 213 |
|
|
|
|
|
|
|
| 214 |
|
| 215 |
---
|
| 216 |
license: apache-2.0
|
|
|
|
| 16 |
- cosmos
|
| 17 |
- lora
|
| 18 |
---
|
| 19 |
+
|
| 20 |
# TriCosmos-2.5-predict
|
| 21 |
+
|
| 22 |
**A driving world model that forecasts RGB, thermal, and LiDAR depth together.**
|
| 23 |
Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night.
|
| 24 |
+
|
| 25 |
- **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA
|
| 26 |
- **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR)
|
| 27 |
- **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ2 grid of sensor tiles
|
| 28 |
+
|
| 29 |
> Paper under review. Full training and evaluation code will be released with the paper.
|
| 30 |
+
|
| 31 |
---
|
| 32 |
+
|
| 33 |
## Contents
|
| 34 |
+
|
| 35 |
1. [Repository files](#repository-files)
|
| 36 |
2. [Quick start](#quick-start)
|
| 37 |
3. [Input format](#input-format)
|
|
|
|
| 40 |
6. [Results](#results)
|
| 41 |
7. [Limitations](#limitations)
|
| 42 |
8. [License and citation](#license-and-citation)
|
| 43 |
+
|
| 44 |
---
|
| 45 |
+
|
| 46 |
## Repository files
|
| 47 |
+
|
| 48 |
| File | Description |
|
| 49 |
|---|---|
|
| 50 |
| `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
|
|
|
|
| 55 |
| `config/experiment.json` | Model options used in training. Needed to rebuild the model. |
|
| 56 |
| `config/export.json` | Checkpoint name and training step. |
|
| 57 |
| `config/train_tri_lora.yaml` | Full training recipe. |
|
| 58 |
+
|
| 59 |
---
|
| 60 |
+
|
| 61 |
## Quick start
|
| 62 |
+
|
| 63 |
### Requirements
|
| 64 |
+
|
| 65 |
- An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ704.
|
| 66 |
+
|
| 67 |
| GPU | Memory | Works? |
|
| 68 |
|---|---|---|
|
| 69 |
| NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | β
tested |
|
| 70 |
| NVIDIA H100 | 80 GB | β
tested |
|
| 71 |
| NVIDIA A100 80 GB | 80 GB | β
used for the paper's results |
|
| 72 |
| Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | β not enough memory |
|
| 73 |
+
|
| 74 |
- A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
|
| 75 |
+
|
| 76 |
### Option 1: Google Colab (easiest)
|
| 77 |
+
|
| 78 |
1. Download `notebooks/TriCosmos_inference.ipynb` and open it in Colab (**File β Upload notebook**).
|
| 79 |
2. Select **Runtime β Change runtime type β G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
|
| 80 |
3. Run all cells from top to bottom and paste your Hugging Face token when asked.
|
| 81 |
+
|
| 82 |
The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video:
|
| 83 |
**input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**.
|
| 84 |
+
|
| 85 |
### Option 2: Local machine
|
| 86 |
+
|
| 87 |
```bash
|
| 88 |
# 1. Cosmos code, exact version used for this model
|
| 89 |
git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
|
|
|
|
| 91 |
git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
|
| 92 |
uv sync --extra=cu128 --python 3.10
|
| 93 |
uv pip install --python .venv/bin/python "imageio[pyav]" av
|
| 94 |
+
|
| 95 |
# 2. This model
|
| 96 |
hf auth login
|
| 97 |
hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
|
| 98 |
+
|
| 99 |
# 3. Forecast the example window
|
| 100 |
.venv/bin/python ../tricosmos/tricosmos_infer.py \
|
| 101 |
--model-dir ../tricosmos \
|
| 102 |
--input ../tricosmos/examples/light_snow_example.mp4 \
|
| 103 |
--prompt-json ../tricosmos/examples/light_snow_example.json \
|
| 104 |
--out ../forecast_tricosmos
|
| 105 |
+
|
| 106 |
# Optional: the zero-shot base model on the same input
|
| 107 |
.venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
|
| 108 |
--model-dir ../tricosmos \
|
|
|
|
| 110 |
--prompt-json ../tricosmos/examples/light_snow_example.json \
|
| 111 |
--out ../forecast_zeroshot
|
| 112 |
```
|
| 113 |
+
|
| 114 |
Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ704.
|
| 115 |
Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
|
| 116 |
+
|
| 117 |
---
|
| 118 |
+
|
| 119 |
## Input format
|
| 120 |
+
|
| 121 |
The model expects a video of **1280Γ704** frames at **10 Hz**, each made of four 640Γ352 tiles:
|
| 122 |
+
|
| 123 |
| | left | right |
|
| 124 |
|---|---|---|
|
| 125 |
| **top** | RGB camera | thermal camera |
|
| 126 |
| **bottom** | LiDAR depth | black (unused) |
|
| 127 |
+
|
| 128 |
Only the **first 5 frames** are used as input.
|
| 129 |
+
|
| 130 |
| Tile | How it is made |
|
| 131 |
|---|---|
|
| 132 |
| **RGB** | Left RGB camera, 1280Γ704, area-averaged to 640Γ352. |
|
| 133 |
| **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ704, area-averaged to 640Γ352. Shows the thermal camera's own view (not warped to the RGB image). |
|
| 134 |
| **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ5 elliptical dilation and a 5Γ5 morphological closing, then max-pooled over 2Γ2 blocks (the nearest depth wins). |
|
| 135 |
| **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
|
| 136 |
+
|
| 137 |
**LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
|
| 138 |
+
|
| 139 |
```
|
| 140 |
q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
|
| 141 |
```
|
| 142 |
+
|
| 143 |
Near objects are bright, far objects dark, and empty pixels black.
|
| 144 |
+
|
| 145 |
**Caption.** The text prompt names the 2Γ2 layout, then the weather and the lighting, in this format (from the Light Snow example):
|
| 146 |
+
|
| 147 |
```
|
| 148 |
driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
|
| 149 |
```
|
| 150 |
+
|
| 151 |
For other data, change only the weather and lighting at the end.
|
| 152 |
+
|
| 153 |
---
|
| 154 |
+
|
| 155 |
## Reading the output
|
| 156 |
+
|
| 157 |
The output has 25 frames: frames 0β4 reproduce the input, and frames 5β24 are the forecast (0.1 s to 2.0 s ahead).
|
| 158 |
+
|
| 159 |
1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`.
|
| 160 |
2. Upsample each tile to 1280Γ704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR.
|
| 161 |
3. Decode LiDAR depth in meters:
|
| 162 |
+
|
| 163 |
```python
|
| 164 |
depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
|
| 165 |
```
|
| 166 |
+
|
| 167 |
---
|
| 168 |
+
|
| 169 |
## Model and training details
|
| 170 |
+
|
| 171 |
| Setting | Value |
|
| 172 |
|---|---|
|
| 173 |
| Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
|
|
|
|
| 180 |
| Training length | 500 steps, bf16, 93-frame clips |
|
| 181 |
| Weather balancing | clips sampled with weights β `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition |
|
| 182 |
| Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** |
|
| 183 |
+
|
| 184 |
---
|
| 185 |
+
|
| 186 |
## Results
|
| 187 |
+
|
| 188 |
DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
|
| 189 |
+
|
| 190 |
| Method | RGB PSNR β | RGB LPIPS β | Thermal PSNR β | Thermal LPIPS β | LiDAR MAE (m) β | LiDAR Ξ΄1 β |
|
| 191 |
|---|---|---|---|---|---|---|
|
| 192 |
| Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
|
| 193 |
| Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 |
|
| 194 |
| **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** |
|
| 195 |
+
|
| 196 |
- **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
|
| 197 |
- **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
|
| 198 |
+
|
| 199 |
---
|
| 200 |
+
|
| 201 |
## Limitations
|
| 202 |
+
|
| 203 |
- **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
|
| 204 |
- **Resolution:** each sensor is seen at half resolution (640Γ352) inside the grid.
|
| 205 |
- **Heavy Snow:** only two validation sequences, so results in this condition are not reliable.
|
|
|
|
| 208 |
- **Memory:** needs a GPU with at least 48 GB.
|
| 209 |
- **No action input:** the model is not conditioned on driving actions.
|
| 210 |
- **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions.
|
| 211 |
+
|
| 212 |
---
|
| 213 |
+
|
| 214 |
## License and citation
|
| 215 |
+
|
| 216 |
**License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
|
| 217 |
+
|
| 218 |
**Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026).
|
|
|
|
|
|
|
|
|
|
| 219 |
|
| 220 |
+
**Citation.** The paper is under review. A citation will be added after publication.
|
| 221 |
+
---
|
| 222 |
|
| 223 |
---
|
| 224 |
license: apache-2.0
|