Image-to-Video
Cosmos
world-model
autonomous-driving
video-prediction
thermal
lidar
adverse-weather
lora
Instructions to use SORA-Lab/TriCosmos-2.5-predict with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use SORA-Lab/TriCosmos-2.5-predict with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: nvidia-open-model-license
|
| 4 |
+
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
|
| 5 |
+
base_model: nvidia/Cosmos-Predict2.5-2B
|
| 6 |
+
pipeline_tag: image-to-video
|
| 7 |
+
tags:
|
| 8 |
+
- world-model
|
| 9 |
+
- autonomous-driving
|
| 10 |
+
- video-prediction
|
| 11 |
+
- thermal
|
| 12 |
+
- lidar
|
| 13 |
+
- adverse-weather
|
| 14 |
+
- cosmos
|
| 15 |
+
- lora
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# TriCosmos-2.5-predict
|
| 19 |
+
|
| 20 |
+
A world model that forecasts the next **2 s of RGB, thermal, and LiDAR depth** for a driving vehicle, from **5 observed frames**, in real adverse weather (fog, rain, snow, night).
|
| 21 |
+
|
| 22 |
+
It is [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`) with LoRA adapters, trained on the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset. All three sensors are shown to the model as **one video**: a 2×2 grid of sensor tiles.
|
| 23 |
+
|
| 24 |
+
> Paper under review. Code will be released with the paper.
|
| 25 |
+
|
| 26 |
+
## Files
|
| 27 |
+
|
| 28 |
+
| File | Content |
|
| 29 |
+
|---|---|
|
| 30 |
+
| `tricosmos_ema_bf16.pt` | Full model weights (EMA, bf16, 3.9 GB), base model with the LoRA weights, step 500. This is the checkpoint used for all reported results. |
|
| 31 |
+
| `config/experiment.json` | Model options used in training (LoRA rank, target layers, conditioning frames). Needed to rebuild the model. |
|
| 32 |
+
| `config/export.json` | Name and training step of the checkpoint. |
|
| 33 |
+
| `config/train_tri_lora.yaml` | Full training recipe. |
|
| 34 |
+
|
| 35 |
+
## Input format
|
| 36 |
+
|
| 37 |
+
Every frame is a **1280×704 RGB image** made of four 640×352 tiles:
|
| 38 |
+
|
| 39 |
+
| | left | right |
|
| 40 |
+
|---|---|---|
|
| 41 |
+
| **top** | RGB camera | thermal camera |
|
| 42 |
+
| **bottom** | LiDAR depth | black (unused) |
|
| 43 |
+
|
| 44 |
+
- **Frame rate:** 10 Hz. **Input:** 5 observed frames. **Output:** the next 20 frames (2 s).
|
| 45 |
+
- **RGB and thermal tiles:** area-averaged from 1280×704. Thermal is an 8-bit gray image copied to 3 channels. The thermal tile shows the thermal camera's own view; it is not warped to the RGB image.
|
| 46 |
+
- **LiDAR tile:** a depth video aligned with the left RGB camera, built from 3 accumulated, ego-motion-compensated sweeps, densified with a 5×5 elliptical dilation and a 5×5 closing, then max-pooled over 2×2 blocks (nearest depth wins). Depth `d` (meters, up to 80 m) is stored as
|
| 47 |
+
`q = round(1 + 254 * (1 - d / 80))`, with `q = 0` for pixels without a LiDAR return, copied to 3 channels. Near objects are bright and empty pixels black.
|
| 48 |
+
- **Text prompt:** a caption that names the 2×2 layout, the weather, and the lighting.
|
| 49 |
+
|
| 50 |
+
To read a forecast, split each output frame back into its tiles and upsample them to 1280×704 (bilinear for RGB and thermal, nearest neighbor for LiDAR). Decode LiDAR depth as `d = 80 * (1 - (q - 1) / 254)` for `q > 0`.
|
| 51 |
+
|
| 52 |
+
## Model and training
|
| 53 |
+
|
| 54 |
+
| Setting | Value |
|
| 55 |
+
|---|---|
|
| 56 |
+
| Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
|
| 57 |
+
| Cosmos code | `cosmos-predict2.5` at commit `a2c298b0a3df3778b973fe65e9e58877b292d8a7` |
|
| 58 |
+
| LoRA | rank 16, alpha 32, on `q_proj, k_proj, v_proj, output_proj, mlp.layer1, mlp.layer2` of every block; base weights, text encoder, and tokenizer frozen |
|
| 59 |
+
| Conditioning | 2 latent frames (= 5 observed frames) |
|
| 60 |
+
| Optimizer | learning rate 1e-4, weight decay 1e-3, 50 linear warm-up steps, linear decay to 10% |
|
| 61 |
+
| Steps | 500, bf16, 93-frame training clips |
|
| 62 |
+
| Sampling | clips weighted by `n_w^(-1/2)`, `n_w` = number of training clips of their weather condition |
|
| 63 |
+
| Training data | 35 DSERT-RoLL training sequences (Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1). Light Snow was not in training. |
|
| 64 |
+
| Generation (as evaluated) | 35 sampling steps, guidance 7 |
|
| 65 |
+
|
| 66 |
+
The model options in `config/experiment.json` must be passed when the model is built, for example `model.config.use_lora=true`, `model.config.lora_rank=16`, and `model.config.min_num_conditional_frames=2`.
|
| 67 |
+
|
| 68 |
+
## Results
|
| 69 |
+
|
| 70 |
+
DSERT-RoLL validation set: 58 sequences, six weather conditions, 300 windows. 2 s horizon, scored at tile resolution. Camera metrics are on windows where the scene is visible.
|
| 71 |
+
|
| 72 |
+
| Method | RGB PSNR ↑ | Thermal PSNR ↑ | LiDAR MAE (m) ↓ |
|
| 73 |
+
|---|---|---|---|
|
| 74 |
+
| Persistence (repeat last frame) | 19.26 | 22.90 | 5.71 |
|
| 75 |
+
| Single-sensor LoRA models | 21.79 | 24.86 | 6.81 |
|
| 76 |
+
| **TriCosmos (this model)** | 21.71 | 24.61 | **4.68** |
|
| 77 |
+
|
| 78 |
+
Joint training lowers the LiDAR error by 2.1 m compared with a LiDAR-only model, and its camera forecasts stay close to those of the single-sensor models. Known limitations: persistence is still better at near range (< 10 m) for LiDAR, Heavy Snow has only two validation sequences, and the model sees each sensor at half resolution.
|
| 79 |
+
|
| 80 |
+
## Quick check of the weights
|
| 81 |
+
|
| 82 |
+
```python
|
| 83 |
+
import torch
|
| 84 |
+
sd = torch.load("tricosmos_ema_bf16.pt", map_location="cpu", weights_only=False)
|
| 85 |
+
sd = sd.get("model", sd) if isinstance(sd, dict) else sd
|
| 86 |
+
lora = [k for k in sd if "lora" in k.lower()]
|
| 87 |
+
print(len(sd), "tensors,", len(lora), "LoRA tensors")
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
## License
|
| 91 |
+
|
| 92 |
+
The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; see the dataset's own license for use of the data.
|
| 93 |
+
|
| 94 |
+
## Citation
|
| 95 |
+
|
| 96 |
+
Paper under review. Citation will be added after publication.
|
| 97 |
+
|
| 98 |
+
|
| 99 |
---
|
| 100 |
license: apache-2.0
|
| 101 |
---
|