Daishi974's picture
Update README.md
ae71b70 verified
|
Raw History Blame Contribute Delete
10.4 kB
---
license: other
license_name: nvidia-open-model-license
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
base_model: nvidia/Cosmos-Predict2.5-2B
base_model_relation: finetune
library_name: cosmos
pipeline_tag: image-to-video
tags:
- world-model
- autonomous-driving
- video-prediction
- thermal
- lidar
- adverse-weather
- cosmos
- lora
---
# TriCosmos-2.5-predict
**A driving world model that forecasts RGB, thermal, and LiDAR depth together.**
Given **5 observed frames** (0.5 s at 10 Hz), it forecasts the **next 20 frames (2 s)** of all three sensors, in real adverse weather: fog, rain, snow, and night.
- **Base model:** [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`), post-trained with LoRA
- **Training data:** [DSERT-RoLL](https://arxiv.org/abs/2604.03685) (real driving data with synchronized RGB, thermal, and LiDAR)
- **Key idea:** the three sensors are shown to the model as **one video**, a 2Γ—2 grid of sensor tiles
> Paper under review. Full training and evaluation code will be released with the paper.
---
## Contents
1. [Repository files](#repository-files)
2. [Quick start](#quick-start)
3. [Input format](#input-format)
4. [Reading the output](#reading-the-output)
5. [Model and training details](#model-and-training-details)
6. [Results](#results)
7. [Limitations](#limitations)
8. [License and citation](#license-and-citation)
---
## Repository files
| File | Description |
|---|---|
| `tricosmos_ema_bf16.pt` | Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
| `TriCosmos_inference.ipynb` | Google Colab notebook: install, download, forecast, and a side-by-side comparison video. |
| `examples/light_snow_example.mp4` | Test window from the DSERT-RoLL validation set (Light Snow, night): 5 observed + 20 real future frames. |
| `examples/light_snow_example.json` | The exact caption used for that window. |
| `config/experiment.json` | Model options used in training. Needed to rebuild the model. |
| `config/export.json` | Checkpoint name and training step. |
| `config/train_tri_lora.yaml` | Full training recipe. |
---
## Quick start
### Requirements
- An NVIDIA GPU with **at least 48 GB of memory** and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ—704.
| GPU | Memory | Works? |
|---|---|---|
| NVIDIA RTX PRO 6000 Blackwell (Colab runtime **G4**) | 96 GB | βœ… tested |
| NVIDIA H100 | 80 GB | βœ… tested |
| NVIDIA A100 80 GB | 80 GB | βœ… used for the paper's results |
| Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) | < 48 GB | ❌ not enough memory |
- A Hugging Face account that has **accepted the license** of [nvidia/Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B). The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
### Option 1: Google Colab (easiest)
1. Download `TriCosmos_inference.ipynb` and open it in Colab (**File β†’ Upload notebook**).
2. Select **Runtime β†’ Change runtime type β†’ G4** (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
3. Run all cells from top to bottom and paste your Hugging Face token when asked.
The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video:
**input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot**.
### Option 2: Local machine
```bash
# 1. Cosmos code, exact version used for this model
git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
cd cosmos-predict2.5
git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
uv sync --extra=cu128 --python 3.10
uv pip install --python .venv/bin/python "imageio[pyav]" av
# 2. This model
hf auth login
hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
# 3. Forecast the example window
.venv/bin/python ../tricosmos/tricosmos_infer.py \
--model-dir ../tricosmos \
--input ../tricosmos/examples/light_snow_example.mp4 \
--prompt-json ../tricosmos/examples/light_snow_example.json \
--out ../forecast_tricosmos
# Optional: the zero-shot base model on the same input
.venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
--model-dir ../tricosmos \
--input ../tricosmos/examples/light_snow_example.mp4 \
--prompt-json ../tricosmos/examples/light_snow_example.json \
--out ../forecast_zeroshot
```
Each output folder holds `forecast_grid.mp4` and `rgb.mp4`, `thermal.mp4`, `lidar.mp4` at 1280Γ—704.
Settings used in the paper (the script defaults): **35 sampling steps, guidance 7, seed 0**. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
---
## Input format
The model expects a video of **1280Γ—704** frames at **10 Hz**, each made of four 640Γ—352 tiles:
| | left | right |
|---|---|---|
| **top** | RGB camera | thermal camera |
| **bottom** | LiDAR depth | black (unused) |
Only the **first 5 frames** are used as input.
| Tile | How it is made |
|---|---|
| **RGB** | Left RGB camera, 1280Γ—704, area-averaged to 640Γ—352. |
| **Thermal** | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ—704, area-averaged to 640Γ—352. Shows the thermal camera's own view (not warped to the RGB image). |
| **LiDAR depth** | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ—5 elliptical dilation and a 5Γ—5 morphological closing, then max-pooled over 2Γ—2 blocks (the nearest depth wins). |
| **Black** | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
**LiDAR depth encoding.** Depth `d` in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
```
q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
```
Near objects are bright, far objects dark, and empty pixels black.
**Caption.** The text prompt names the 2Γ—2 layout, then the weather and the lighting, in this format (from the Light Snow example):
```
driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
```
For other data, change only the weather and lighting at the end.
---
## Reading the output
The output has 25 frames: frames 0–4 reproduce the input, and frames 5–24 are the forecast (0.1 s to 2.0 s ahead).
1. Split each frame into its tiles: RGB `[:352, :640]`, thermal `[:352, 640:]`, LiDAR `[352:, :640]`.
2. Upsample each tile to 1280Γ—704: **bilinear** for RGB and thermal, **nearest neighbor** for LiDAR.
3. Decode LiDAR depth in meters:
```python
depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
```
---
## Model and training details
| Setting | Value |
|---|---|
| Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
| Cosmos code | `cosmos-predict2.5`, commit `a2c298b0a3df3778b973fe65e9e58877b292d8a7` |
| LoRA | rank 16, alpha 32, on `q_proj`, `k_proj`, `v_proj`, `output_proj`, `mlp.layer1`, `mlp.layer2` of every transformer block |
| Frozen | base weights, text encoder, video tokenizer |
| Conditioning | 2 latent frames (= 5 observed frames) |
| Optimizer | learning rate 1e-4, weight decay 1e-3 |
| Schedule | 50 linear warm-up steps, then linear decay to 10% |
| Training length | 500 steps, bf16, 93-frame clips |
| Weather balancing | clips sampled with weights ∝ `n_w^(-1/2)`, where `n_w` is the number of training clips of the clip's weather condition |
| Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. **Light Snow was not in training.** |
---
## Results
DSERT-RoLL official validation set: **58 sequences, six weather conditions, 300 windows**. 2 s horizon, all models scored at tile resolution (640Γ—352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
| Method | RGB PSNR ↑ | RGB LPIPS ↓ | Thermal PSNR ↑ | Thermal LPIPS ↓ | LiDAR MAE (m) ↓ | LiDAR Ξ΄1 ↑ |
|---|---|---|---|---|---|---|
| Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
| Single-sensor LoRA models | **21.79** | **0.284** | **24.86** | **0.243** | 6.81 | 0.752 |
| **TriCosmos (this model)** | 21.71 | 0.304 | 24.61 | 0.270 | **4.68** | **0.794** |
- **LiDAR:** forecasting the three sensors jointly lowers the LiDAR error by **2.1 m** compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
- **Cameras:** the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
---
## Limitations
- **Near range:** for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
- **Resolution:** each sensor is seen at half resolution (640Γ—352) inside the grid.
- **Heavy Snow:** only two validation sequences, so results in this condition are not reliable.
- **Thermal registration:** the thermal tile is not registered to the RGB image.
- **Speed:** about 9 minutes per forecast on an RTX PRO 6000 (model loading included); not for real-time use.
- **Memory:** needs a GPU with at least 48 GB.
- **No action input:** the model is not conditioned on driving actions.
- **Intended use:** research on world models and sensor forecasting. Not for safety-critical decisions.
---
## License and citation
**License.** The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
**Acknowledgments.** Built on [NVIDIA Cosmos-Predict2.5](https://github.com/nvidia-cosmos/cosmos-predict2.5) and the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset (Cho et al., CVPR 2026).
**Citation.** The paper is under review. A citation will be added after publication.
---
---
license: apache-2.0
---