Cosmos
Diffusers
Safetensors
cosmos3_omni
nvidia
cosmos3
omnimodel

Updated Readme.md

#14
by lvaidya - opened
Files changed (1) hide show
  1. README.md +48 -48
README.md CHANGED
@@ -364,35 +364,35 @@ Each per-NFE time is the end-to-end time from the tables below divided by 70 NFE
364
 
365
  | Views | Res | GB200 x1 | GB200 x2 | GB200 x4 | H100 x1 | H100 x2 | H100 x4 | H100 x8 | H20 x1 | H20 x2 | H20 x4 | H20 x8 | H200 x1 | H200 x2 | H200 x4 | H200 x8 |
366
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
367
- | 1 | 480p | 1.2s | 0.6s | 0.4s | 2.4s | 1.3s | 0.8s | 0.6s | | | | | 2.4s | 1.3s | 0.8s | 0.6s |
368
- | | 720p | 5.1s | 2.8s | 1.6s | 10s | 5.6s | 3.1s | 2.0s | | | 13s | 7.5s | 10s | 5.6s | 3.1s | 2.0s |
369
- | 4 | 480p | 5.3s | 2.9s | 1.6s | 11s | 5.7s | 3.3s | 2.3s | | | 13s | 7.5s | 11s | 5.7s | 3.3s | 2.3s |
370
- | | 720p | 22s | 12s | 6.3s | 44s | 24s | 13s | 8.3s | | | 55s | 31s | 46s | 25s | 13s | 8.1s |
371
- | 5 | 480p | 6.7s | 3.7s | 2.1s | 14s | 7.3s | 4.2s | 2.9s | | | | | 14s | 7.2s | 4.3s | 2.9s |
372
- | | 720p | 28s | 15s | 8.1s | 56s | 30s | 17s | 10s | | | | 39s | 58s | 31s | 17s | 10s |
373
- | 7 | 480p | 9.7s | 5.3s | 3.0s | 20s | 12s | 7.2s | 5.3s | | | 26s | 16s | 20s | 12s | 7.0s | 5.1s |
374
- | | 720p | 40s | 21s | 11s | 81s | 44s | 24s | 15s | | | 101s | | 85s | 45s | 24s | 15s |
375
- | 8 | 480p | 11s | 6.0s | 3.5s | 23s | 13s | 8.1s | 6.0s | | | 30s | 18s | 23s | 13s | 8.0s | 5.8s |
376
- | | 720p | 46s | 25s | 13s | 94s | 51s | 28s | 17s | | | 116s | 65s | 97s | 51s | 28s | 17s |
377
- | 11 | 480p | 16s | 8.6s | 4.9s | 33s | 19s | 11s | 8.3s | | | | 26s | 33s | 19s | 11s | 8.1s |
378
- | | 720p | 66s | 35s | 19s | - | 73s | 40s | 24s | | | 166s | 92s | 141s | 75s | 40s | 24s |
379
 
380
  **With LiDAR (per NFE, in seconds)**
381
 
382
  | Views | Res | GB200 x1 | GB200 x2 | GB200 x4 | H100 x1 | H100 x2 | H100 x4 | H100 x8 | H20 x1 | H20 x2 | H20 x4 | H20 x8 | H200 x1 | H200 x2 | H200 x4 | H200 x8 |
383
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
384
- | 1 | 480p | 8.8s | 4.8s | 2.7s | 20s | 11s | 5.9s | 3.4s | | 44s | 23s | 12s | 20s | 11s | 5.8s | 3.4s |
385
- | | 720p | 13s | 6.9s | 3.7s | 28s | 15s | 8.3s | 4.8s | | | 33s | | 29s | 15s | 8.3s | 4.7s |
386
- | 4 | 480p | 13s | 7.2s | 3.9s | 30s | 16s | 9.3s | 6.0s | | | 35s | | 30s | 16s | 9.3s | 5.8s |
387
- | | 720p | 31s | 16s | 8.9s | 67s | 36s | 20s | 12s | | | 77s | 43s | 69s | 36s | 20s | 12s |
388
- | 5 | 480p | 16s | 8.6s | 4.7s | 34s | 19s | 11s | 6.8s | | | 39s | 22s | 34s | 18s | 10s | 6.8s |
389
- | | 720p | 37s | 20s | 11s | 81s | 43s | 24s | 14s | | | 93s | | 81s | 44s | 24s | 14s |
390
- | 7 | 480p | 19s | 10s | 5.7s | 41s | 22s | 13s | 8.5s | | | | | 41s | 22s | 13s | 8.4s |
391
- | | 720p | 50s | 26s | 14s | - | 58s | 32s | 19s | | | 125s | 69s | 112s | 59s | 32s | 19s |
392
- | 8 | 480p | 21s | 11s | 6.4s | 44s | 25s | 14s | 9.3s | | | | 30s | 44s | 24s | 14s | 9.3s |
393
- | | 720p | 56s | 30s | 16s | - | 65s | 36s | 22s | | | 142s | 79s | 125s | 67s | 36s | 21s |
394
- | 11 | 480p | 27s | 14s | 8.1s | 56s | 31s | 18s | 12s | | | | 38s | 56s | 31s | 18s | 12s |
395
- | | 720p | 77s | 41s | 22s | - | - | 50s | 30s | | | | 107s | 171s | 90s | 49s | 29s |
396
 
397
  ### Inference Performance - end2end
398
 
@@ -400,35 +400,35 @@ Each per-NFE time is the end-to-end time from the tables below divided by 70 NFE
400
 
401
  | Views | Res | GB200 x1 | GB200 x2 | GB200 x4 | H100 x1 | H100 x2 | H100 x4 | H100 x8 | H20 x1 | H20 x2 | H20 x4 | H20 x8 | H200 x1 | H200 x2 | H200 x4 | H200 x8 |
402
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
403
- | 1 | 480p | 1m22s | 0m45s | 0m28s | 2m47s | 1m29s | 0m56s | 0m41s | | | | | 2m51s | 1m31s | 0m57s | 0m40s |
404
- | | 720p | 5m54s | 3m13s | 1m55s | 11m41s | 6m32s | 3m40s | 2m21s | | | 15m30s | 8m48s | 11m59s | 6m31s | 3m40s | 2m18s |
405
- | 4 | 480p | 6m11s | 3m22s | 1m53s | 12m32s | 6m39s | 3m52s | 2m42s | | | 15m07s | 8m45s | 12m32s | 6m37s | 3m53s | 2m41s |
406
- | | 720p | 25m22s | 13m28s | 7m23s | 50m58s | 27m54s | 15m21s | 9m38s | | | 64m36s | 36m13s | 53m45s | 28m37s | 15m27s | 9m30s |
407
- | 5 | 480p | 7m49s | 4m18s | 2m27s | 15m49s | 8m28s | 4m57s | 3m25s | | | | | 15m54s | 8m25s | 5m00s | 3m24s |
408
- | | 720p | 32m19s | 17m13s | 9m24s | 65m36s | 35m01s | 19m35s | 12m09s | | | | 45m46s | 68m13s | 36m18s | 19m35s | 11m53s |
409
- | 7 | 480p | 11m20s | 6m08s | 3m29s | 23m28s | 13m47s | 8m24s | 6m08s | | | 30m08s | 18m36s | 23m27s | 13m36s | 8m10s | 5m56s |
410
- | | 720p | 46m21s | 24m36s | 13m24s | 93m58s | 51m19s | 28m20s | 17m21s | | | 117m28s | | 98m43s | 52m33s | 28m18s | 17m04s |
411
- | 8 | 480p | 12m57s | 7m03s | 4m02s | 27m03s | 15m36s | 9m30s | 6m58s | | | 34m46s | 21m15s | 27m05s | 15m40s | 9m23s | 6m48s |
412
- | | 720p | 53m57s | 28m40s | 15m40s | 109m12s | 59m36s | 32m19s | 20m03s | | | 135m49s | 75m32s | 112m56s | 60m02s | 32m46s | 19m43s |
413
- | 11 | 480p | 18m27s | 10m04s | 5m40s | 38m33s | 22m08s | 13m21s | 9m41s | | | | 29m59s | 39m04s | 22m00s | 13m14s | 9m25s |
414
- | | 720p | 77m31s | 41m05s | 22m29s | - | 84m38s | 46m28s | 27m50s | | | 193m47s | 107m25s | 164m35s | 87m16s | 46m49s | 27m40s |
415
 
416
  **With LiDAR (end-to-end inference time)**
417
 
418
  | Views | Res | GB200 x1 | GB200 x2 | GB200 x4 | H100 x1 | H100 x2 | H100 x4 | H100 x8 | H20 x1 | H20 x2 | H20 x4 | H20 x8 | H200 x1 | H200 x2 | H200 x4 | H200 x8 |
419
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
420
- | 1 | 480p | 10m15s | 5m35s | 3m06s | 23m12s | 12m34s | 6m51s | 3m58s | | 51m33s | 26m41s | 14m13s | 23m39s | 12m27s | 6m44s | 3m55s |
421
- | | 720p | 15m07s | 8m00s | 4m18s | 33m08s | 17m41s | 9m42s | 5m34s | | | 38m25s | | 33m40s | 17m50s | 9m40s | 5m32s |
422
- | 4 | 480p | 15m31s | 8m22s | 4m36s | 35m05s | 19m11s | 10m52s | 6m59s | | | 40m56s | | 35m13s | 19m09s | 10m48s | 6m47s |
423
- | | 720p | 35m48s | 19m03s | 10m25s | 78m32s | 42m00s | 22m56s | 13m54s | | | 90m18s | 49m57s | 80m16s | 41m57s | 23m02s | 13m44s |
424
- | 5 | 480p | 18m35s | 10m00s | 5m32s | 39m09s | 21m35s | 12m25s | 7m59s | | | 45m47s | 26m01s | 39m27s | 21m30s | 12m14s | 7m54s |
425
- | | 720p | 42m51s | 22m48s | 12m30s | 94m27s | 50m22s | 27m37s | 16m54s | | | 108m29s | | 94m57s | 50m48s | 27m39s | 16m25s |
426
- | 7 | 480p | 22m35s | 12m02s | 6m40s | 47m16s | 26m12s | 15m01s | 9m57s | | | | | 47m20s | 26m04s | 14m59s | 9m51s |
427
- | | 720p | 57m48s | 30m38s | 16m50s | - | 68m04s | 37m19s | 22m12s | | | 145m47s | 81m01s | 130m18s | 68m28s | 37m14s | 22m07s |
428
- | 8 | 480p | 24m41s | 13m17s | 7m28s | 51m33s | 28m36s | 16m29s | 10m51s | | | | 35m11s | 51m38s | 28m20s | 16m21s | 10m48s |
429
- | | 720p | 65m27s | 34m53s | 19m05s | - | 76m06s | 41m45s | 25m24s | | | 165m19s | 91m47s | 145m15s | 77m40s | 42m09s | 25m04s |
430
- | 11 | 480p | 30m55s | 16m40s | 9m30s | 64m47s | 36m16s | 20m54s | 13m41s | | | | 44m53s | 65m14s | 35m47s | 20m41s | 13m37s |
431
- | | 720p | 89m47s | 47m45s | 25m58s | - | - | 57m47s | 34m26s | | | | 125m07s | 199m14s | 105m01s | 57m42s | 34m06s |
432
 
433
 
434
  ## Usage Example
 
364
 
365
  | Views | Res | GB200 x1 | GB200 x2 | GB200 x4 | H100 x1 | H100 x2 | H100 x4 | H100 x8 | H20 x1 | H20 x2 | H20 x4 | H20 x8 | H200 x1 | H200 x2 | H200 x4 | H200 x8 |
366
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
367
+ | 1 | 480p | 1.2s | 0.6s | 0.4s | 2.4s | 1.3s | 0.8s | 0.6s | 11s | 5.7s | 3.1s | 1.8s | 2.4s | 1.3s | 0.8s | 0.6s |
368
+ | | 720p | 5.1s | 2.8s | 1.6s | 10s | 5.6s | 3.1s | 2.0s | 47s | 25s | 13s | 7.5s | 10s | 5.6s | 3.1s | 2.0s |
369
+ | 4 | 480p | 5.3s | 2.9s | 1.6s | 11s | 5.7s | 3.3s | 2.3s | 46s | 24s | 13s | 7.5s | 11s | 5.7s | 3.3s | 2.3s |
370
+ | | 720p | 22s | 12s | 6.3s | 44s | 24s | 13s | 8.3s | 199s | 104s | 55s | 31s | 46s | 25s | 13s | 8.1s |
371
+ | 5 | 480p | 6.7s | 3.7s | 2.1s | 14s | 7.3s | 4.2s | 2.9s | 59s | 30s | 16 | 9.5s | 14s | 7.2s | 4.3s | 2.9s |
372
+ | | 720p | 28s | 15s | 8.1s | 56s | 30s | 17s | 10s | 253s | 132s | 70s | 39s | 58s | 31s | 17s | 10s |
373
+ | 7 | 480p | 9.7s | 5.3s | 3.0s | 20s | 12s | 7.2s | 5.3s | 86s | 46s | 26s | 16s | 20s | 12s | 7.0s | 5.1s |
374
+ | | 720p | 40s | 21s | 11s | 81s | 44s | 24s | 15s | 364s | 190s | 101s | | 85s | 45s | 24s | 15s |
375
+ | 8 | 480p | 11s | 6.0s | 3.5s | 23s | 13s | 8.1s | 6.0s | 100s | 53s | 30s | 18s | 23s | 13s | 8.0s | 5.8s |
376
+ | | 720p | 46s | 25s | 13s | 94s | 51s | 28s | 17s | 423s | 220s | 116s | 65s | 97s | 51s | 28s | 17s |
377
+ | 11 | 480p | 16s | 8.6s | 4.9s | 33s | 19s | 11s | 8.3s | 141s | 76s | 42s | 26s | 33s | 19s | 11s | 8.1s |
378
+ | | 720p | 66s | 35s | 19s | - | 73s | 40s | 24s | 603s | 315s | 166s | 92s | 141s | 75s | 40s | 24s |
379
 
380
  **With LiDAR (per NFE, in seconds)**
381
 
382
  | Views | Res | GB200 x1 | GB200 x2 | GB200 x4 | H100 x1 | H100 x2 | H100 x4 | H100 x8 | H20 x1 | H20 x2 | H20 x4 | H20 x8 | H200 x1 | H200 x2 | H200 x4 | H200 x8 |
383
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
384
+ | 1 | 480p | 8.8s | 4.8s | 2.7s | 20s | 11s | 5.9s | 3.4s | 86s | 44s | 23s | 12s | 20s | 11s | 5.8s | 3.4s |
385
+ | | 720p | 13s | 6.9s | 3.7s | 28s | 15s | 8.3s | 4.8s | 123s | 63s | 33s | 18s | 29s | 15s | 8.3s | 4.7s |
386
+ | 4 | 480p | 13s | 7.2s | 3.9s | 30s | 16s | 9.3s | 6.0s | 126s | 66s | 35s | 20s | 30s | 16s | 9.3s | 5.8s |
387
+ | | 720p | 31s | 16s | 8.9s | 67s | 36s | 20s | 12s | 283s | 147s | 77s | 43s | 69s | 36s | 20s | 12s |
388
+ | 5 | 480p | 16s | 8.6s | 4.7s | 34s | 19s | 11s | 6.8s | 140s | 73s | 39s | 22s | 34s | 18s | 10s | 6.8s |
389
+ | | 720p | 37s | 20s | 11s | 81s | 43s | 24s | 14s | 338s | 176s | 93s | 52s | 81s | 44s | 24s | 14s |
390
+ | 7 | 480p | 19s | 10s | 5.7s | 41s | 22s | 13s | 8.5s | 169s | 89s | 48s | 27s | 41s | 22s | 13s | 8.4s |
391
+ | | 720p | 50s | 26s | 14s | - | 58s | 32s | 19s | 454s | 236s | 125s | 69s | 112s | 59s | 32s | 19s |
392
+ | 8 | 480p | 21s | 11s | 6.4s | 44s | 25s | 14s | 9.3s | 183s | 96s | 52s | 30s | 44s | 24s | 14s | 9.3s |
393
+ | | 720p | 56s | 30s | 16s | - | 65s | 36s | 22s | 514s | 268s | 142s | 79s | 125s | 67s | 36s | 21s |
394
+ | 11 | 480p | 27s | 14s | 8.1s | 56s | 31s | 18s | 12s | 229s | 121s | 66s | 38s | 56s | 31s | 18s | 12s |
395
+ | | 720p | 77s | 41s | 22s | - | - | 50s | 30s | - | 367s | 194s | 107s | 171s | 90s | 49s | 29s |
396
 
397
  ### Inference Performance - end2end
398
 
 
400
 
401
  | Views | Res | GB200 x1 | GB200 x2 | GB200 x4 | H100 x1 | H100 x2 | H100 x4 | H100 x8 | H20 x1 | H20 x2 | H20 x4 | H20 x8 | H200 x1 | H200 x2 | H200 x4 | H200 x8 |
402
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
403
+ | 1 | 480p | 1m19s | 0m44s | 0m27s | 2m47s | 1m29s | 0m56s | 0m41s | 12m58s | 6m39s | 3m37s | 2m07s | 2m51s | 1m31s | 0m57s | 0m40s |
404
+ | | 720p | 5m55s | 3m15s | 1m59s | 11m41s | 6m32s | 3m40s | 2m21s | 55m21s | 28m57s | 15m30s | 8m48s | 11m59s | 6m31s | 3m40s | 2m18s |
405
+ | 4 | 480p | 6m04s | 3m20s | 2m00s | 12m32s | 6m39s | 3m52s | 2m42s | 54m06s | 27m55s | 15m07s | 8m45s | 12m32s | 6m37s | 3m53s | 2m41s |
406
+ | | 720p | 25m23s | 14m19s | 8m25s | 50m58s | 27m54s | 15m21s | 9m38s | 232m32s | 121m32s | 64m36s | 36m13s | 53m45s | 28m37s | 15m27s | 9m30s |
407
+ | 5 | 480p | 7m43s | 4m17s | 2m29s | 15m49s | 8m28s | 4m57s | 3m25s | 68m27s | 35m27s | 18m56s | 11m04s | 15m54s | 8m25s | 5m00s | 3m24s |
408
+ | | 720p | 32m20s | 17m55s | 10m13s | 65m36s | 35m01s | 19m35s | 12m09s | 295m24s | 154m09s | 81m48s | 45m46s | 68m13s | 36m18s | 19m35s | 11m53s |
409
+ | 7 | 480p | 12m45s | 7m37s | 4m58s | 23m28s | 13m47s | 8m24s | 6m08s | 100m44s | 54m05s | 30m08s | 18m36s | 23m27s | 13m36s | 8m10s | 5m56s |
410
+ | | 720p | 47m03s | 25m50s | 14m57s | 93m58s | 51m19s | 28m20s | 17m21s | 424m38s | 221m37s | 117m29s | 65m25s | 98m43s | 52m33s | 28m18s | 17m04s |
411
+ | 8 | 480p | 14m26s | 9m01s | 5m40s | 27m03s | 15m36s | 9m30s | 6m58s | 116m13s | 62m20s | 34m46s | 21m15s | 27m05s | 15m40s | 9m23s | 6m48s |
412
+ | | 720p | 54m23s | 29m59s | 17m15s | 109m12s | 59m36s | 32m19s | 20m03s | 492m59s | 256m36s | 135m49s | 75m32s | 112m56s | 60m02s | 32m46s | 19m43s |
413
+ | 11 | 480p | 20m21s | 13m46s | 7m42s | 38m33s | 22m08s | 13m21s | 9m41s | 164m52s | 88m25s | 49m04s | 29m59s | 39m04s | 22m00s | 13m14s | 9m25s |
414
+ | | 720p | 76m49s | 42m59s | 24m56s | - | 84m38s | 46m28s | 27m50s | 703m59s | 367m01s | 193m47s | 107m25s | 164m35s | 87m16s | 46m49s | 27m40s |
415
 
416
  **With LiDAR (end-to-end inference time)**
417
 
418
  | Views | Res | GB200 x1 | GB200 x2 | GB200 x4 | H100 x1 | H100 x2 | H100 x4 | H100 x8 | H20 x1 | H20 x2 | H20 x4 | H20 x8 | H200 x1 | H200 x2 | H200 x4 | H200 x8 |
419
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
420
+ | 1 | 480p | 9m54s | 5m40s | 3m09s | 23m12s | 12m34s | 6m51s | 3m58s | 100m42s | 51m33s | 26m41s | 14m13s | 23m39s | 12m27s | 6m44s | 3m55s |
421
+ | | 720p | 15m31s | 8m23s | 4m33s | 33m08s | 17m41s | 9m42s | 5m34s | 143m41s | 74m02s | 38m25s | 20m40s | 33m40s | 17m50s | 9m40s | 5m32s |
422
+ | 4 | 480p | 17m37s | 9m46s | 5m35s | 35m05s | 19m11s | 10m52s | 6m59s | 146m25s | 76m52s | 40m56s | 23m01s | 35m13s | 19m09s | 10m48s | 6m47s |
423
+ | | 720p | 35m03s | 20m17s | 11m28s | 78m32s | 42m00s | 22m56s | 13m54s | 330m20s | 171m22s | 90m18s | 49m57s | 80m16s | 41m57s | 23m02s | 13m44s |
424
+ | 5 | 480p | 18m39s | 10m44s | 6m16s | 39m09s | 21m35s | 12m25s | 7m59s | 163m38s | 85m37s | 45m47s | 26m01s | 39m27s | 21m30s | 12m14s | 7m54s |
425
+ | | 720p | 42m38s | 23m33s | 13m55s | 94m27s | 50m22s | 27m37s | 16m54s | 394m20s | 205m34s | 108m29s | 60m11s | 94m57s | 50m48s | 27m39s | 16m25s |
426
+ | 7 | 480p | 22m42s | 12m55s | 7m52s | 47m16s | 26m12s | 15m01s | 9m57s | 196m55s | 103m21s | 55m38s | 31m56s | 47m20s | 26m04s | 14m59s | 9m51s |
427
+ | | 720p | 58m30s | 32m47s | 18m41s | | 68m04s | 37m19s | 22m12s | 529m15s | 275m54s | 145m47s | 81m01s | 130m18s | 68m28s | 37m14s | 22m07s |
428
+ | 8 | 480p | 25m55s | 14m57s | 8m57s | 51m33s | 28m36s | 16m29s | 10m51s | 213m53s | 112m34s | 60m53s | 35m11s | 51m38s | 28m20s | 16m21s | 10m48s |
429
+ | | 720p | 66m30s | 37m01s | 20m50s | | 76m06s | 41m45s | 25m24s | 599m18s | 313m00s | 165m19s | 91m47s | 145m15s | 77m40s | 42m09s | 25m04s |
430
+ | 11 | 480p | 31m16s | 18m38s | 11m20s | 64m47s | 36m16s | 20m54s | 13m41s | 266m35s | 141m00s | 76m30s | 44m53s | 65m14s | 35m47s | 20m41s | 13m37s |
431
+ | | 720p | 89m35s | 50m50s | 29m14s | - | - | 57m47s | 34m26s | - | 427m57s | 225m51s | 125m07s | 199m14s | 105m01s | 57m42s | 34m06s |
432
 
433
 
434
  ## Usage Example