Daishi974 commited on
Commit
f8a3da2
Β·
verified Β·
1 Parent(s): 11cb92c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +181 -67
README.md CHANGED
@@ -16,86 +16,200 @@ tags:
16
  - cosmos
17
  - lora
18
  ---
19
-
20
  # TriCosmos-2.5-predict
21
-
22
- A world model that forecasts the next **2 s of RGB, thermal, and LiDAR depth** for a driving vehicle, from **5 observed frames**, in real adverse weather (fog, rain, snow, night).
23
-
24
- It is [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`) with LoRA adapters, trained on the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset. All three sensors are shown to the model as **one video**: a 2Γ—2 grid of sensor tiles.
25
-
26
- > Paper under review. Code will be released with the paper.
27
-
28
- ## Files
29
-
30
- | File | Content |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  |---|---|
32
- | `tricosmos_ema_bf16.pt` | Full model weights (EMA, bf16, 3.9 GB), base model with the LoRA weights, step 500. This is the checkpoint used for all reported results. |
33
- | `config/experiment.json` | Model options used in training (LoRA rank, target layers, conditioning frames). Needed to rebuild the model. |
34
- | `config/export.json` | Name and training step of the checkpoint. |
 
 
 
 
35
  | `config/train_tri_lora.yaml` | Full training recipe. |
36
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37
  ## Input format
38
-
39
- Every frame is a **1280Γ—704 RGB image** made of four 640Γ—352 tiles:
40
-
41
  | | left | right |
42
  |---|---|---|
43
  | **top** | RGB camera | thermal camera |
44
  | **bottom** | LiDAR depth | black (unused) |
45
-
46
- - **Frame rate:** 10 Hz. **Input:** 5 observed frames. **Output:** the next 20 frames (2 s).
47
- - **RGB and thermal tiles:** area-averaged from 1280Γ—704. Thermal is an 8-bit gray image copied to 3 channels. The thermal tile shows the thermal camera's own view; it is not warped to the RGB image.
48
- - **LiDAR tile:** a depth video aligned with the left RGB camera, built from 3 accumulated, ego-motion-compensated sweeps, densified with a 5Γ—5 elliptical dilation and a 5Γ—5 closing, then max-pooled over 2Γ—2 blocks (nearest depth wins). Depth `d` (meters, up to 80 m) is stored as
49
- `q = round(1 + 254 * (1 - d / 80))`, with `q = 0` for pixels without a LiDAR return, copied to 3 channels. Near objects are bright and empty pixels black.
50
- - **Text prompt:** a caption that names the 2Γ—2 layout, the weather, and the lighting.
51
-
52
- To read a forecast, split each output frame back into its tiles and upsample them to 1280Γ—704 (bilinear for RGB and thermal, nearest neighbor for LiDAR). Decode LiDAR depth as `d = 80 * (1 - (q - 1) / 254)` for `q > 0`.
53
-
54
- ## Model and training
55
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  | Setting | Value |
57
  |---|---|
58
  | Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
59
- | Cosmos code | `cosmos-predict2.5` at commit `a2c298b0a3df3778b973fe65e9e58877b292d8a7` |
60
- | LoRA | rank 16, alpha 32, on `q_proj, k_proj, v_proj, output_proj, mlp.layer1, mlp.layer2` of every block; base weights, text encoder, and tokenizer frozen |
 
61
  | Conditioning | 2 latent frames (= 5 observed frames) |
62
- | Optimizer | learning rate 1e-4, weight decay 1e-3, 50 linear warm-up steps, linear decay to 10% |
63
- | Steps | 500, bf16, 93-frame training clips |
64
- | Sampling | clips weighted by `n_w^(-1/2)`, `n_w` = number of training clips of their weather condition |
65
- | Training data | 35 DSERT-RoLL training sequences (Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1). Light Snow was not in training. |
66
- | Generation (as evaluated) | 35 sampling steps, guidance 7 |
67
-
68
- The model options in `config/experiment.json` must be passed when the model is built, for example `model.config.use_lora=true`, `model.config.lora_rank=16`, and `model.config.min_num_conditional_frames=2`.
69
-
70
  ## Results
71
-
72
- DSERT-RoLL validation set: 58 sequences, six weather conditions, 300 windows. 2 s horizon, scored at tile resolution. Camera metrics are on windows where the scene is visible.
73
-
74
- | Method | RGB PSNR ↑ | Thermal PSNR ↑ | LiDAR MAE (m) ↓ |
75
- |---|---|---|---|
76
- | Persistence (repeat last frame) | 19.26 | 22.90 | 5.71 |
77
- | Single-sensor LoRA models | 21.79 | 24.86 | 6.81 |
78
- | **TriCosmos (this model)** | 21.71 | 24.61 | **4.68** |
79
-
80
- Joint training lowers the LiDAR error by 2.1 m compared with a LiDAR-only model, and its camera forecasts stay close to those of the single-sensor models. Known limitations: persistence is still better at near range (< 10 m) for LiDAR, Heavy Snow has only two validation sequences, and the model sees each sensor at half resolution.
81
-
82
- ## Quick check of the weights
83
-
84
- ```python
85
- import torch
86
- sd = torch.load("tricosmos_ema_bf16.pt", map_location="cpu", weights_only=False)
87
- sd = sd.get("model", sd) if isinstance(sd, dict) else sd
88
- lora = [k for k in sd if "lora" in k.lower()]
89
- print(len(sd), "tensors,", len(lora), "LoRA tensors")
90
- ```
91
-
92
- ## License
93
-
94
- The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; see the dataset's own license for use of the data.
95
-
96
- ## Citation
97
-
98
- Paper under review. Citation will be added after publication.
 
 
 
 
 
99
 
100
 
101
  ---
 
16
  - cosmos
17
  - lora
18
  ---
19
+
20
  # TriCosmos-2.5-predict
21
+
22
+ **A driving world model that forecasts RGB, thermal, and LiDAR depth together.**
23
+ Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night.
24
+
25
+ - **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA
26
+ - **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR)
27
+ - **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ—2 grid of sensor tiles
28
+ > Paper under review. Full training and evaluation code will be released with the paper.
29
+
30
+ ---
31
+
32
+ ## Contents
33
+
34
+ 1. [Repository files](#repository-files)
35
+ 2. [Quick start](#quick-start)
36
+ 3. [Input format](#input-format)
37
+ 4. [Reading the output](#reading-the-output)
38
+ 5. [Model and training details](#model-and-training-details)
39
+ 6. [Results](#results)
40
+ 7. [Limitations](#limitations)
41
+ 8. [License and citation](#license-and-citation)
42
+ ---
43
+
44
+ ## Repository files
45
+
46
+ | File | Description |
47
  |---|---|
48
+ | `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
49
+ | `tricosmos_infer.py` | Inference script: one forecast from a 2Γ—2 grid video; writes the forecast grid and one video per sensor. Also runs the zero-shot base model for comparison. |
50
+ | `notebooks/TriCosmos_inference.ipynb` | Google Colab notebook: install, download, forecast, and a side-by-side comparison video. |
51
+ | `examples/light_snow_example.mp4` | Test window from the DSERT-RoLL validation set (Light Snow, night): 5 observed + 20 real future frames. |
52
+ | `examples/light_snow_example.json` | The exact caption used for that window. |
53
+ | `config/experiment.json` | Model options used in training. Needed to rebuild the model. |
54
+ | `config/export.json` | Checkpoint name and training step. |
55
  | `config/train_tri_lora.yaml` | Full training recipe. |
56
+
57
+ ---
58
+
59
+ ## Quick start
60
+
61
+ ### Requirements
62
+
63
+ - An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ—704.
64
+ | GPU | Memory | Works? |
65
+ |---|---|---|
66
+ | NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | βœ… tested |
67
+ | NVIDIA H100 | 80 GB | βœ… tested |
68
+ | NVIDIA A100 80 GB | 80 GB | βœ… used for the paper's results |
69
+ | Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | ❌ not enough memory |
70
+ - A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
71
+ ### Option 1: Google Colab (easiest)
72
+
73
+ 1. Download `notebooks/TriCosmos_inference.ipynb` and open it in Colab (**File β†’ Upload notebook**).
74
+ 2. Select **Runtime β†’ Change runtime type β†’ G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
75
+ 3. Run all cells from top to bottom and paste your Hugging Face token when asked.
76
+ The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video:
77
+ **input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**.
78
+
79
+ ### Option 2: Local machine
80
+
81
+ ```bash
82
+ # 1. Cosmos code, exact version used for this model
83
+ git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
84
+ cd cosmos-predict2.5
85
+ git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
86
+ uv sync --extra=cu128 --python 3.10
87
+ uv pip install --python .venv/bin/python "imageio[pyav]" av
88
+
89
+ # 2. This model
90
+ hf auth login
91
+ hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
92
+
93
+ # 3. Forecast the example window
94
+ .venv/bin/python ../tricosmos/tricosmos_infer.py \
95
+ --model-dir ../tricosmos \
96
+ --input ../tricosmos/examples/light_snow_example.mp4 \
97
+ --prompt-json ../tricosmos/examples/light_snow_example.json \
98
+ --out ../forecast_tricosmos
99
+
100
+ # Optional: the zero-shot base model on the same input
101
+ .venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
102
+ --model-dir ../tricosmos \
103
+ --input ../tricosmos/examples/light_snow_example.mp4 \
104
+ --prompt-json ../tricosmos/examples/light_snow_example.json \
105
+ --out ../forecast_zeroshot
106
+ ```
107
+
108
+ Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ—704.
109
+ Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
110
+
111
+ ---
112
+
113
  ## Input format
114
+
115
+ The model expects a video of **1280Γ—704** frames at **10 Hz**, each made of four 640Γ—352 tiles:
116
+
117
  | | left | right |
118
  |---|---|---|
119
  | **top** | RGB camera | thermal camera |
120
  | **bottom** | LiDAR depth | black (unused) |
121
+
122
+ Only the **first 5 frames** are used as input.
123
+
124
+ | Tile | How it is made |
125
+ |---|---|
126
+ | **RGB** | Left RGB camera, 1280Γ—704, area-averaged to 640Γ—352. |
127
+ | **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ—704, area-averaged to 640Γ—352. Shows the thermal camera's own view (not warped to the RGB image). |
128
+ | **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ—5 elliptical dilation and a 5Γ—5 morphological closing, then max-pooled over 2Γ—2 blocks (the nearest depth wins). |
129
+ | **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
130
+
131
+ **LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
132
+
133
+ ```
134
+ q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
135
+ ```
136
+
137
+ Near objects are bright, far objects dark, and empty pixels black.
138
+
139
+ **Caption.** The text prompt names the 2Γ—2 layout, then the weather and the lighting, in this format (from the Light Snow example):
140
+
141
+ ```
142
+ driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
143
+ ```
144
+
145
+ For other data, change only the weather and lighting at the end.
146
+
147
+ ---
148
+
149
+ ## Reading the output
150
+
151
+ The output has 25 frames: frames 0–4 reproduce the input, and frames 5–24 are the forecast (0.1 s to 2.0 s ahead).
152
+
153
+ 1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`.
154
+ 2. Upsample each tile to 1280Γ—704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR.
155
+ 3. Decode LiDAR depth in meters:
156
+ ```python
157
+ depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
158
+ ```
159
+
160
+ ---
161
+
162
+ ## Model and training details
163
+
164
  | Setting | Value |
165
  |---|---|
166
  | Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
167
+ | Cosmos code | `cosmos-predict2.5`, commit `a2c298b0a3df3778b973fe65e9e58877b292d8a7` |
168
+ | LoRA | rank 16, alpha 32, on `q_proj`, `k_proj`, `v_proj`, `output_proj`, `mlp.layer1`, `mlp.layer2` of every transformer block |
169
+ | Frozen | base weights, text encoder, video tokenizer |
170
  | Conditioning | 2 latent frames (= 5 observed frames) |
171
+ | Optimizer | learning rate 1e-4, weight decay 1e-3 |
172
+ | Schedule | 50 linear warm-up steps, then linear decay to 10% |
173
+ | Training length | 500 steps, bf16, 93-frame clips |
174
+ | Weather balancing | clips sampled with weights ∝ `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition |
175
+ | Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** |
176
+
177
+ ---
178
+
179
  ## Results
180
+
181
+ DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ—352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
182
+
183
+ | Method | RGB PSNR ↑ | RGB LPIPS ↓ | Thermal PSNR ↑ | Thermal LPIPS ↓ | LiDAR MAE (m) ↓ | LiDAR Ξ΄1 ↑ |
184
+ |---|---|---|---|---|---|---|
185
+ | Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
186
+ | Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 |
187
+ | **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** |
188
+
189
+ - **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
190
+ - **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
191
+ ---
192
+
193
+ ## Limitations
194
+
195
+ - **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
196
+ - **Resolution:** each sensor is seen at half resolution (640Γ—352) inside the grid.
197
+ - **Heavy Snow:** only two validation sequences, so results in this condition are not reliable.
198
+ - **Thermal registration:** the thermal tile is not registered to the RGB image.
199
+ - **Speed:** about 9 minutes per forecast on an RTX PRO 6000 (model loading included); not for real-time use.
200
+ - **Memory:** needs a GPU with at least 48 GB.
201
+ - **No action input:** the model is not conditioned on driving actions.
202
+ - **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions.
203
+ ---
204
+
205
+ ## License and citation
206
+
207
+ **License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
208
+
209
+ **Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026).
210
+
211
+ **Citation.** The paper is under review. A citation will be added after publication.
212
+
213
 
214
 
215
  ---