Daishi974 commited on
Commit
eccdf58
·
verified ·
1 Parent(s): 6ac1161

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +98 -0
README.md CHANGED
@@ -1,3 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
 
1
+ ---
2
+ license: other
3
+ license_name: nvidia-open-model-license
4
+ license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
5
+ base_model: nvidia/Cosmos-Predict2.5-2B
6
+ pipeline_tag: image-to-video
7
+ tags:
8
+ - world-model
9
+ - autonomous-driving
10
+ - video-prediction
11
+ - thermal
12
+ - lidar
13
+ - adverse-weather
14
+ - cosmos
15
+ - lora
16
+ ---
17
+
18
+ # TriCosmos-2.5-predict
19
+
20
+ A world model that forecasts the next **2 s of RGB, thermal, and LiDAR depth** for a driving vehicle, from **5 observed frames**, in real adverse weather (fog, rain, snow, night).
21
+
22
+ It is [Cosmos-Predict2.5-2B](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (`base/post-trained`) with LoRA adapters, trained on the [DSERT-RoLL](https://arxiv.org/abs/2604.03685) dataset. All three sensors are shown to the model as **one video**: a 2×2 grid of sensor tiles.
23
+
24
+ > Paper under review. Code will be released with the paper.
25
+
26
+ ## Files
27
+
28
+ | File | Content |
29
+ |---|---|
30
+ | `tricosmos_ema_bf16.pt` | Full model weights (EMA, bf16, 3.9 GB), base model with the LoRA weights, step 500. This is the checkpoint used for all reported results. |
31
+ | `config/experiment.json` | Model options used in training (LoRA rank, target layers, conditioning frames). Needed to rebuild the model. |
32
+ | `config/export.json` | Name and training step of the checkpoint. |
33
+ | `config/train_tri_lora.yaml` | Full training recipe. |
34
+
35
+ ## Input format
36
+
37
+ Every frame is a **1280×704 RGB image** made of four 640×352 tiles:
38
+
39
+ | | left | right |
40
+ |---|---|---|
41
+ | **top** | RGB camera | thermal camera |
42
+ | **bottom** | LiDAR depth | black (unused) |
43
+
44
+ - **Frame rate:** 10 Hz. **Input:** 5 observed frames. **Output:** the next 20 frames (2 s).
45
+ - **RGB and thermal tiles:** area-averaged from 1280×704. Thermal is an 8-bit gray image copied to 3 channels. The thermal tile shows the thermal camera's own view; it is not warped to the RGB image.
46
+ - **LiDAR tile:** a depth video aligned with the left RGB camera, built from 3 accumulated, ego-motion-compensated sweeps, densified with a 5×5 elliptical dilation and a 5×5 closing, then max-pooled over 2×2 blocks (nearest depth wins). Depth `d` (meters, up to 80 m) is stored as
47
+ `q = round(1 + 254 * (1 - d / 80))`, with `q = 0` for pixels without a LiDAR return, copied to 3 channels. Near objects are bright and empty pixels black.
48
+ - **Text prompt:** a caption that names the 2×2 layout, the weather, and the lighting.
49
+
50
+ To read a forecast, split each output frame back into its tiles and upsample them to 1280×704 (bilinear for RGB and thermal, nearest neighbor for LiDAR). Decode LiDAR depth as `d = 80 * (1 - (q - 1) / 254)` for `q > 0`.
51
+
52
+ ## Model and training
53
+
54
+ | Setting | Value |
55
+ |---|---|
56
+ | Base model | Cosmos-Predict2.5-2B, `base/post-trained` (video-to-world) |
57
+ | Cosmos code | `cosmos-predict2.5` at commit `a2c298b0a3df3778b973fe65e9e58877b292d8a7` |
58
+ | LoRA | rank 16, alpha 32, on `q_proj, k_proj, v_proj, output_proj, mlp.layer1, mlp.layer2` of every block; base weights, text encoder, and tokenizer frozen |
59
+ | Conditioning | 2 latent frames (= 5 observed frames) |
60
+ | Optimizer | learning rate 1e-4, weight decay 1e-3, 50 linear warm-up steps, linear decay to 10% |
61
+ | Steps | 500, bf16, 93-frame training clips |
62
+ | Sampling | clips weighted by `n_w^(-1/2)`, `n_w` = number of training clips of their weather condition |
63
+ | Training data | 35 DSERT-RoLL training sequences (Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1). Light Snow was not in training. |
64
+ | Generation (as evaluated) | 35 sampling steps, guidance 7 |
65
+
66
+ The model options in `config/experiment.json` must be passed when the model is built, for example `model.config.use_lora=true`, `model.config.lora_rank=16`, and `model.config.min_num_conditional_frames=2`.
67
+
68
+ ## Results
69
+
70
+ DSERT-RoLL validation set: 58 sequences, six weather conditions, 300 windows. 2 s horizon, scored at tile resolution. Camera metrics are on windows where the scene is visible.
71
+
72
+ | Method | RGB PSNR ↑ | Thermal PSNR ↑ | LiDAR MAE (m) ↓ |
73
+ |---|---|---|---|
74
+ | Persistence (repeat last frame) | 19.26 | 22.90 | 5.71 |
75
+ | Single-sensor LoRA models | 21.79 | 24.86 | 6.81 |
76
+ | **TriCosmos (this model)** | 21.71 | 24.61 | **4.68** |
77
+
78
+ Joint training lowers the LiDAR error by 2.1 m compared with a LiDAR-only model, and its camera forecasts stay close to those of the single-sensor models. Known limitations: persistence is still better at near range (< 10 m) for LiDAR, Heavy Snow has only two validation sequences, and the model sees each sensor at half resolution.
79
+
80
+ ## Quick check of the weights
81
+
82
+ ```python
83
+ import torch
84
+ sd = torch.load("tricosmos_ema_bf16.pt", map_location="cpu", weights_only=False)
85
+ sd = sd.get("model", sd) if isinstance(sd, dict) else sd
86
+ lora = [k for k in sd if "lora" in k.lower()]
87
+ print(len(sd), "tensors,", len(lora), "LoRA tensors")
88
+ ```
89
+
90
+ ## License
91
+
92
+ The weights are derived from Cosmos-Predict2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The model was trained on DSERT-RoLL; see the dataset's own license for use of the data.
93
+
94
+ ## Citation
95
+
96
+ Paper under review. Citation will be added after publication.
97
+
98
+
99
  ---
100
  license: apache-2.0
101
  ---