Daishi974 commited on
Commit
688c50c
Β·
verified Β·
1 Parent(s): f8a3da2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +59 -51
README.md CHANGED
@@ -16,21 +16,22 @@ tags:
16
  - cosmos
17
  - lora
18
  ---
19
-
20
  # TriCosmos-2.5-predict
21
-
22
  **A driving world model that forecasts RGB, thermal, and LiDAR depth together.**
23
  Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night.
24
-
25
  - **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA
26
  - **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR)
27
  - **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ—2 grid of sensor tiles
 
28
  > Paper under review. Full training and evaluation code will be released with the paper.
29
-
30
  ---
31
-
32
  ## Contents
33
-
34
  1. [Repository files](#repository-files)
35
  2. [Quick start](#quick-start)
36
  3. [Input format](#input-format)
@@ -39,10 +40,11 @@ Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames
39
  6. [Results](#results)
40
  7. [Limitations](#limitations)
41
  8. [License and citation](#license-and-citation)
 
42
  ---
43
-
44
  ## Repository files
45
-
46
  | File | Description |
47
  |---|---|
48
  | `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
@@ -53,31 +55,35 @@ Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames
53
  | `config/experiment.json` | Model options used in training. Needed to rebuild the model. |
54
  | `config/export.json` | Checkpoint name and training step. |
55
  | `config/train_tri_lora.yaml` | Full training recipe. |
56
-
57
  ---
58
-
59
  ## Quick start
60
-
61
  ### Requirements
62
-
63
  - An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ—704.
 
64
  | GPU | Memory | Works? |
65
  |---|---|---|
66
  | NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | βœ… tested |
67
  | NVIDIA H100 | 80 GB | βœ… tested |
68
  | NVIDIA A100 80 GB | 80 GB | βœ… used for the paper's results |
69
  | Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | ❌ not enough memory |
 
70
  - A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
 
71
  ### Option 1: Google Colab (easiest)
72
-
73
  1. Download `notebooks/TriCosmos_inference.ipynb` and open it in Colab (**File β†’ Upload notebook**).
74
  2. Select **Runtime β†’ Change runtime type β†’ G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
75
  3. Run all cells from top to bottom and paste your Hugging Face token when asked.
 
76
  The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video:
77
  **input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**.
78
-
79
  ### Option 2: Local machine
80
-
81
  ```bash
82
  # 1. Cosmos code, exact version used for this model
83
  git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
@@ -85,18 +91,18 @@ cd cosmos-predict2.5
85
  git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
86
  uv sync --extra=cu128 --python 3.10
87
  uv pip install --python .venv/bin/python "imageio[pyav]" av
88
-
89
  # 2. This model
90
  hf auth login
91
  hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
92
-
93
  # 3. Forecast the example window
94
  .venv/bin/python ../tricosmos/tricosmos_infer.py \
95
  --model-dir ../tricosmos \
96
  --input ../tricosmos/examples/light_snow_example.mp4 \
97
  --prompt-json ../tricosmos/examples/light_snow_example.json \
98
  --out ../forecast_tricosmos
99
-
100
  # Optional: the zero-shot base model on the same input
101
  .venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
102
  --model-dir ../tricosmos \
@@ -104,63 +110,64 @@ hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
104
  --prompt-json ../tricosmos/examples/light_snow_example.json \
105
  --out ../forecast_zeroshot
106
  ```
107
-
108
  Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ—704.
109
  Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
110
-
111
  ---
112
-
113
  ## Input format
114
-
115
  The model expects a video of **1280Γ—704** frames at **10 Hz**, each made of four 640Γ—352 tiles:
116
-
117
  | | left | right |
118
  |---|---|---|
119
  | **top** | RGB camera | thermal camera |
120
  | **bottom** | LiDAR depth | black (unused) |
121
-
122
  Only the **first 5 frames** are used as input.
123
-
124
  | Tile | How it is made |
125
  |---|---|
126
  | **RGB** | Left RGB camera, 1280Γ—704, area-averaged to 640Γ—352. |
127
  | **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ—704, area-averaged to 640Γ—352. Shows the thermal camera's own view (not warped to the RGB image). |
128
  | **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ—5 elliptical dilation and a 5Γ—5 morphological closing, then max-pooled over 2Γ—2 blocks (the nearest depth wins). |
129
  | **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
130
-
131
  **LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
132
-
133
  ```
134
  q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
135
  ```
136
-
137
  Near objects are bright, far objects dark, and empty pixels black.
138
-
139
  **Caption.** The text prompt names the 2Γ—2 layout, then the weather and the lighting, in this format (from the Light Snow example):
140
-
141
  ```
142
  driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
143
  ```
144
-
145
  For other data, change only the weather and lighting at the end.
146
-
147
  ---
148
-
149
  ## Reading the output
150
-
151
  The output has 25 frames: frames 0–4 reproduce the input, and frames 5–24 are the forecast (0.1 s to 2.0 s ahead).
152
-
153
  1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`.
154
  2. Upsample each tile to 1280Γ—704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR.
155
  3. Decode LiDAR depth in meters:
 
156
  ```python
157
  depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
158
  ```
159
-
160
  ---
161
-
162
  ## Model and training details
163
-
164
  | Setting | Value |
165
  |---|---|
166
  | Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
@@ -173,25 +180,26 @@ depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
173
  | Training length | 500 steps, bf16, 93-frame clips |
174
  | Weather balancing | clips sampled with weights ∝ `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition |
175
  | Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** |
176
-
177
  ---
178
-
179
  ## Results
180
-
181
  DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ—352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
182
-
183
  | Method | RGB PSNR ↑ | RGB LPIPS ↓ | Thermal PSNR ↑ | Thermal LPIPS ↓ | LiDAR MAE (m) ↓ | LiDAR Ξ΄1 ↑ |
184
  |---|---|---|---|---|---|---|
185
  | Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
186
  | Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 |
187
  | **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** |
188
-
189
  - **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
190
  - **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
 
191
  ---
192
-
193
  ## Limitations
194
-
195
  - **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
196
  - **Resolution:** each sensor is seen at half resolution (640Γ—352) inside the grid.
197
  - **Heavy Snow:** only two validation sequences, so results in this condition are not reliable.
@@ -200,17 +208,17 @@ DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300
200
  - **Memory:** needs a GPU with at least 48 GB.
201
  - **No action input:** the model is not conditioned on driving actions.
202
  - **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions.
 
203
  ---
204
-
205
  ## License and citation
206
-
207
  **License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
208
-
209
  **Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026).
210
-
211
- **Citation.** The paper is under review. A citation will be added after publication.
212
-
213
 
 
 
214
 
215
  ---
216
  license: apache-2.0
 
16
  - cosmos
17
  - lora
18
  ---
19
+
20
  # TriCosmos-2.5-predict
21
+
22
  **A driving world model that forecasts RGB, thermal, and LiDAR depth together.**
23
  Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night.
24
+
25
  - **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA
26
  - **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR)
27
  - **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ—2 grid of sensor tiles
28
+
29
  > Paper under review. Full training and evaluation code will be released with the paper.
30
+
31
  ---
32
+
33
  ## Contents
34
+
35
  1. [Repository files](#repository-files)
36
  2. [Quick start](#quick-start)
37
  3. [Input format](#input-format)
 
40
  6. [Results](#results)
41
  7. [Limitations](#limitations)
42
  8. [License and citation](#license-and-citation)
43
+
44
  ---
45
+
46
  ## Repository files
47
+
48
  | File | Description |
49
  |---|---|
50
  | `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
 
55
  | `config/experiment.json` | Model options used in training. Needed to rebuild the model. |
56
  | `config/export.json` | Checkpoint name and training step. |
57
  | `config/train_tri_lora.yaml` | Full training recipe. |
58
+
59
  ---
60
+
61
  ## Quick start
62
+
63
  ### Requirements
64
+
65
  - An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ—704.
66
+
67
  | GPU | Memory | Works? |
68
  |---|---|---|
69
  | NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | βœ… tested |
70
  | NVIDIA H100 | 80 GB | βœ… tested |
71
  | NVIDIA A100 80 GB | 80 GB | βœ… used for the paper's results |
72
  | Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | ❌ not enough memory |
73
+
74
  - A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
75
+
76
  ### Option 1: Google Colab (easiest)
77
+
78
  1. Download `notebooks/TriCosmos_inference.ipynb` and open it in Colab (**File β†’ Upload notebook**).
79
  2. Select **Runtime β†’ Change runtime type β†’ G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
80
  3. Run all cells from top to bottom and paste your Hugging Face token when asked.
81
+
82
  The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video:
83
  **input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**.
84
+
85
  ### Option 2: Local machine
86
+
87
  ```bash
88
  # 1. Cosmos code, exact version used for this model
89
  git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
 
91
  git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
92
  uv sync --extra=cu128 --python 3.10
93
  uv pip install --python .venv/bin/python "imageio[pyav]" av
94
+
95
  # 2. This model
96
  hf auth login
97
  hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
98
+
99
  # 3. Forecast the example window
100
  .venv/bin/python ../tricosmos/tricosmos_infer.py \
101
  --model-dir ../tricosmos \
102
  --input ../tricosmos/examples/light_snow_example.mp4 \
103
  --prompt-json ../tricosmos/examples/light_snow_example.json \
104
  --out ../forecast_tricosmos
105
+
106
  # Optional: the zero-shot base model on the same input
107
  .venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
108
  --model-dir ../tricosmos \
 
110
  --prompt-json ../tricosmos/examples/light_snow_example.json \
111
  --out ../forecast_zeroshot
112
  ```
113
+
114
  Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ—704.
115
  Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
116
+
117
  ---
118
+
119
  ## Input format
120
+
121
  The model expects a video of **1280Γ—704** frames at **10 Hz**, each made of four 640Γ—352 tiles:
122
+
123
  | | left | right |
124
  |---|---|---|
125
  | **top** | RGB camera | thermal camera |
126
  | **bottom** | LiDAR depth | black (unused) |
127
+
128
  Only the **first 5 frames** are used as input.
129
+
130
  | Tile | How it is made |
131
  |---|---|
132
  | **RGB** | Left RGB camera, 1280Γ—704, area-averaged to 640Γ—352. |
133
  | **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ—704, area-averaged to 640Γ—352. Shows the thermal camera's own view (not warped to the RGB image). |
134
  | **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ—5 elliptical dilation and a 5Γ—5 morphological closing, then max-pooled over 2Γ—2 blocks (the nearest depth wins). |
135
  | **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
136
+
137
  **LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
138
+
139
  ```
140
  q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
141
  ```
142
+
143
  Near objects are bright, far objects dark, and empty pixels black.
144
+
145
  **Caption.** The text prompt names the 2Γ—2 layout, then the weather and the lighting, in this format (from the Light Snow example):
146
+
147
  ```
148
  driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
149
  ```
150
+
151
  For other data, change only the weather and lighting at the end.
152
+
153
  ---
154
+
155
  ## Reading the output
156
+
157
  The output has 25 frames: frames 0–4 reproduce the input, and frames 5–24 are the forecast (0.1 s to 2.0 s ahead).
158
+
159
  1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`.
160
  2. Upsample each tile to 1280Γ—704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR.
161
  3. Decode LiDAR depth in meters:
162
+
163
  ```python
164
  depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
165
  ```
166
+
167
  ---
168
+
169
  ## Model and training details
170
+
171
  | Setting | Value |
172
  |---|---|
173
  | Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
 
180
  | Training length | 500 steps, bf16, 93-frame clips |
181
  | Weather balancing | clips sampled with weights ∝ `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition |
182
  | Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** |
183
+
184
  ---
185
+
186
  ## Results
187
+
188
  DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ—352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
189
+
190
  | Method | RGB PSNR ↑ | RGB LPIPS ↓ | Thermal PSNR ↑ | Thermal LPIPS ↓ | LiDAR MAE (m) ↓ | LiDAR Ξ΄1 ↑ |
191
  |---|---|---|---|---|---|---|
192
  | Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
193
  | Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 |
194
  | **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** |
195
+
196
  - **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
197
  - **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
198
+
199
  ---
200
+
201
  ## Limitations
202
+
203
  - **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
204
  - **Resolution:** each sensor is seen at half resolution (640Γ—352) inside the grid.
205
  - **Heavy Snow:** only two validation sequences, so results in this condition are not reliable.
 
208
  - **Memory:** needs a GPU with at least 48 GB.
209
  - **No action input:** the model is not conditioned on driving actions.
210
  - **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions.
211
+
212
  ---
213
+
214
  ## License and citation
215
+
216
  **License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
217
+
218
  **Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026).
 
 
 
219
 
220
+ **Citation.** The paper is under review. A citation will be added after publication.
221
+ ---
222
 
223
  ---
224
  license: apache-2.0