Image-to-Video
Cosmos
world-model
autonomous-driving
video-prediction
thermal
lidar
adverse-weather
lora
Instructions to use SORA-Lab/TriCosmos-2.5-predict with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use SORA-Lab/TriCosmos-2.5-predict with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -16,86 +16,200 @@ tags:
|
|
| 16 |
- cosmos
|
| 17 |
- lora
|
| 18 |
---
|
| 19 |
-
|
| 20 |
# TriCosmos-2.5-predict
|
| 21 |
-
|
| 22 |
-
A world model that forecasts
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|---|---|
|
| 32 |
-
| `tricosmos_ema_bf16.pt` |
|
| 33 |
-
| `
|
| 34 |
-
| `
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
| `config/train_tri_lora.yaml` | Full training recipe. |
|
| 36 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
## Input format
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
| | left | right |
|
| 42 |
|---|---|---|
|
| 43 |
| **top** | RGB camera | thermal camera |
|
| 44 |
| **bottom** | LiDAR depth | black (unused) |
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
| Setting | Value |
|
| 57 |
|---|---|
|
| 58 |
| Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
|
| 59 |
-
| Cosmos code | `cosmos-predict2.5`
|
| 60 |
-
| LoRA | rank 16, alpha 32, on `q_proj, k_proj, v_proj, output_proj, mlp.layer1, mlp.layer2` of every block
|
|
|
|
| 61 |
| Conditioning | 2 latent frames (= 5 observed frames) |
|
| 62 |
-
| Optimizer | learning rate 1e-4, weight decay 1e-3
|
| 63 |
-
|
|
| 64 |
-
|
|
| 65 |
-
|
|
| 66 |
-
|
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
## Results
|
| 71 |
-
|
| 72 |
-
DSERT-RoLL validation set: 58 sequences, six weather conditions, 300 windows. 2 s horizon, scored at tile resolution. Camera metrics
|
| 73 |
-
|
| 74 |
-
| Method | RGB PSNR β | Thermal PSNR β | LiDAR MAE (m) β |
|
| 75 |
-
|---|---|---|---|
|
| 76 |
-
| Persistence (repeat last frame) | 19.26 | 22.90 | 5.71 |
|
| 77 |
-
| Single-sensor LoRA models | 21.79 | 24.86 | 6.81 |
|
| 78 |
-
| **TriCosmos (this model)** | 21.71 | 24.61 | **4.68** |
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
##
|
| 97 |
-
|
| 98 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
|
| 100 |
|
| 101 |
---
|
|
|
|
| 16 |
- cosmos
|
| 17 |
- lora
|
| 18 |
---
|
| 19 |
+
|
| 20 |
# TriCosmos-2.5-predict
|
| 21 |
+
|
| 22 |
+
**A driving world model that forecasts RGB, thermal, and LiDAR depth together.**
|
| 23 |
+
Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night.
|
| 24 |
+
|
| 25 |
+
- **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA
|
| 26 |
+
- **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR)
|
| 27 |
+
- **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ2 grid of sensor tiles
|
| 28 |
+
> Paper under review. Full training and evaluation code will be released with the paper.
|
| 29 |
+
|
| 30 |
+
---
|
| 31 |
+
|
| 32 |
+
## Contents
|
| 33 |
+
|
| 34 |
+
1. [Repository files](#repository-files)
|
| 35 |
+
2. [Quick start](#quick-start)
|
| 36 |
+
3. [Input format](#input-format)
|
| 37 |
+
4. [Reading the output](#reading-the-output)
|
| 38 |
+
5. [Model and training details](#model-and-training-details)
|
| 39 |
+
6. [Results](#results)
|
| 40 |
+
7. [Limitations](#limitations)
|
| 41 |
+
8. [License and citation](#license-and-citation)
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
## Repository files
|
| 45 |
+
|
| 46 |
+
| File | Description |
|
| 47 |
|---|---|
|
| 48 |
+
| `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
|
| 49 |
+
| `tricosmos_infer.py` | Inference script: one forecast from a 2Γ2 grid video; writes the forecast grid and one video per sensor. Also runs the zero-shot base model for comparison. |
|
| 50 |
+
| `notebooks/TriCosmos_inference.ipynb` | Google Colab notebook: install, download, forecast, and a side-by-side comparison video. |
|
| 51 |
+
| `examples/light_snow_example.mp4` | Test window from the DSERT-RoLL validation set (Light Snow, night): 5 observed + 20 real future frames. |
|
| 52 |
+
| `examples/light_snow_example.json` | The exact caption used for that window. |
|
| 53 |
+
| `config/experiment.json` | Model options used in training. Needed to rebuild the model. |
|
| 54 |
+
| `config/export.json` | Checkpoint name and training step. |
|
| 55 |
| `config/train_tri_lora.yaml` | Full training recipe. |
|
| 56 |
+
|
| 57 |
+
---
|
| 58 |
+
|
| 59 |
+
## Quick start
|
| 60 |
+
|
| 61 |
+
### Requirements
|
| 62 |
+
|
| 63 |
+
- An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ704.
|
| 64 |
+
| GPU | Memory | Works? |
|
| 65 |
+
|---|---|---|
|
| 66 |
+
| NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | β
tested |
|
| 67 |
+
| NVIDIA H100 | 80 GB | β
tested |
|
| 68 |
+
| NVIDIA A100 80 GB | 80 GB | β
used for the paper's results |
|
| 69 |
+
| Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | β not enough memory |
|
| 70 |
+
- A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
|
| 71 |
+
### Option 1: Google Colab (easiest)
|
| 72 |
+
|
| 73 |
+
1. Download `notebooks/TriCosmos_inference.ipynb` and open it in Colab (**File β Upload notebook**).
|
| 74 |
+
2. Select **Runtime β Change runtime type β G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
|
| 75 |
+
3. Run all cells from top to bottom and paste your Hugging Face token when asked.
|
| 76 |
+
The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video:
|
| 77 |
+
**input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**.
|
| 78 |
+
|
| 79 |
+
### Option 2: Local machine
|
| 80 |
+
|
| 81 |
+
```bash
|
| 82 |
+
# 1. Cosmos code, exact version used for this model
|
| 83 |
+
git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
|
| 84 |
+
cd cosmos-predict2.5
|
| 85 |
+
git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
|
| 86 |
+
uv sync --extra=cu128 --python 3.10
|
| 87 |
+
uv pip install --python .venv/bin/python "imageio[pyav]" av
|
| 88 |
+
|
| 89 |
+
# 2. This model
|
| 90 |
+
hf auth login
|
| 91 |
+
hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
|
| 92 |
+
|
| 93 |
+
# 3. Forecast the example window
|
| 94 |
+
.venv/bin/python ../tricosmos/tricosmos_infer.py \
|
| 95 |
+
--model-dir ../tricosmos \
|
| 96 |
+
--input ../tricosmos/examples/light_snow_example.mp4 \
|
| 97 |
+
--prompt-json ../tricosmos/examples/light_snow_example.json \
|
| 98 |
+
--out ../forecast_tricosmos
|
| 99 |
+
|
| 100 |
+
# Optional: the zero-shot base model on the same input
|
| 101 |
+
.venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
|
| 102 |
+
--model-dir ../tricosmos \
|
| 103 |
+
--input ../tricosmos/examples/light_snow_example.mp4 \
|
| 104 |
+
--prompt-json ../tricosmos/examples/light_snow_example.json \
|
| 105 |
+
--out ../forecast_zeroshot
|
| 106 |
+
```
|
| 107 |
+
|
| 108 |
+
Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ704.
|
| 109 |
+
Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
|
| 110 |
+
|
| 111 |
+
---
|
| 112 |
+
|
| 113 |
## Input format
|
| 114 |
+
|
| 115 |
+
The model expects a video of **1280Γ704** frames at **10 Hz**, each made of four 640Γ352 tiles:
|
| 116 |
+
|
| 117 |
| | left | right |
|
| 118 |
|---|---|---|
|
| 119 |
| **top** | RGB camera | thermal camera |
|
| 120 |
| **bottom** | LiDAR depth | black (unused) |
|
| 121 |
+
|
| 122 |
+
Only the **first 5 frames** are used as input.
|
| 123 |
+
|
| 124 |
+
| Tile | How it is made |
|
| 125 |
+
|---|---|
|
| 126 |
+
| **RGB** | Left RGB camera, 1280Γ704, area-averaged to 640Γ352. |
|
| 127 |
+
| **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ704, area-averaged to 640Γ352. Shows the thermal camera's own view (not warped to the RGB image). |
|
| 128 |
+
| **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ5 elliptical dilation and a 5Γ5 morphological closing, then max-pooled over 2Γ2 blocks (the nearest depth wins). |
|
| 129 |
+
| **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
|
| 130 |
+
|
| 131 |
+
**LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
|
| 132 |
+
|
| 133 |
+
```
|
| 134 |
+
q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
|
| 135 |
+
```
|
| 136 |
+
|
| 137 |
+
Near objects are bright, far objects dark, and empty pixels black.
|
| 138 |
+
|
| 139 |
+
**Caption.** The text prompt names the 2Γ2 layout, then the weather and the lighting, in this format (from the Light Snow example):
|
| 140 |
+
|
| 141 |
+
```
|
| 142 |
+
driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
|
| 143 |
+
```
|
| 144 |
+
|
| 145 |
+
For other data, change only the weather and lighting at the end.
|
| 146 |
+
|
| 147 |
+
---
|
| 148 |
+
|
| 149 |
+
## Reading the output
|
| 150 |
+
|
| 151 |
+
The output has 25 frames: frames 0β4 reproduce the input, and frames 5β24 are the forecast (0.1 s to 2.0 s ahead).
|
| 152 |
+
|
| 153 |
+
1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`.
|
| 154 |
+
2. Upsample each tile to 1280Γ704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR.
|
| 155 |
+
3. Decode LiDAR depth in meters:
|
| 156 |
+
```python
|
| 157 |
+
depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
|
| 158 |
+
```
|
| 159 |
+
|
| 160 |
+
---
|
| 161 |
+
|
| 162 |
+
## Model and training details
|
| 163 |
+
|
| 164 |
| Setting | Value |
|
| 165 |
|---|---|
|
| 166 |
| Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
|
| 167 |
+
| Cosmos code | `cosmos-predict2.5`, commit `a2c298b0a3df3778b973fe65e9e58877b292d8a7` |
|
| 168 |
+
| LoRA | rank 16, alpha 32, on `q_proj`, `k_proj`, `v_proj`, `output_proj`, `mlp.layer1`, `mlp.layer2` of every transformer block |
|
| 169 |
+
| Frozen | base weights, text encoder, video tokenizer |
|
| 170 |
| Conditioning | 2 latent frames (= 5 observed frames) |
|
| 171 |
+
| Optimizer | learning rate 1e-4, weight decay 1e-3 |
|
| 172 |
+
| Schedule | 50 linear warm-up steps, then linear decay to 10% |
|
| 173 |
+
| Training length | 500 steps, bf16, 93-frame clips |
|
| 174 |
+
| Weather balancing | clips sampled with weights β `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition |
|
| 175 |
+
| Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** |
|
| 176 |
+
|
| 177 |
+
---
|
| 178 |
+
|
| 179 |
## Results
|
| 180 |
+
|
| 181 |
+
DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
|
| 182 |
+
|
| 183 |
+
| Method | RGB PSNR β | RGB LPIPS β | Thermal PSNR β | Thermal LPIPS β | LiDAR MAE (m) β | LiDAR Ξ΄1 β |
|
| 184 |
+
|---|---|---|---|---|---|---|
|
| 185 |
+
| Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
|
| 186 |
+
| Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 |
|
| 187 |
+
| **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** |
|
| 188 |
+
|
| 189 |
+
- **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
|
| 190 |
+
- **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
|
| 191 |
+
---
|
| 192 |
+
|
| 193 |
+
## Limitations
|
| 194 |
+
|
| 195 |
+
- **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
|
| 196 |
+
- **Resolution:** each sensor is seen at half resolution (640Γ352) inside the grid.
|
| 197 |
+
- **Heavy Snow:** only two validation sequences, so results in this condition are not reliable.
|
| 198 |
+
- **Thermal registration:** the thermal tile is not registered to the RGB image.
|
| 199 |
+
- **Speed:** about 9 minutes per forecast on an RTX PRO 6000 (model loading included); not for real-time use.
|
| 200 |
+
- **Memory:** needs a GPU with at least 48 GB.
|
| 201 |
+
- **No action input:** the model is not conditioned on driving actions.
|
| 202 |
+
- **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions.
|
| 203 |
+
---
|
| 204 |
+
|
| 205 |
+
## License and citation
|
| 206 |
+
|
| 207 |
+
**License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
|
| 208 |
+
|
| 209 |
+
**Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026).
|
| 210 |
+
|
| 211 |
+
**Citation.** The paper is under review. A citation will be added after publication.
|
| 212 |
+
|
| 213 |
|
| 214 |
|
| 215 |
---
|