Consolidate FP8 performance benchmarks by hardware (#2)

- Consolidate FP8 performance benchmarks by hardware (869bc1db266d60fdcbb1e34e961ef698e755b69e)
- Use fastest measured image-to-video backend per GPU (768216b3bd2bb0b24539fd4efe7d08702a54ae6e)
- Add DROID policy throughput for documented 16-action workload (05b6d7e41fd7c97fe85c4f3e893820d9e7b4c20e)


Co-authored-by: Sergiy Fefilatyev <fsergiy@users.noreply.huggingface.co>

README.md

@@ -294,144 +294,32 @@ All models are evaluated on image-to-video generation at **480p, 24 fps**. Throu
 
 The Edge model is a strong initialization for downstream action tasks. For example, post-training it on the DROID dataset produces a policy whose RoboLab success rate is reported in the [Cosmos3-Edge-Policy-DROID model card](https://huggingface.co/nvidia/Cosmos3-Edge-Policy-DROID#robolab).
 
-### PBR (Performance Benchmark Reporting)
+### Performance Benchmarks
 
-The following tables report single-GPU or single-platform inference performance for the Cosmos3-Edge-FP8 **Generator** and **Reasoner** towers.
+The table reports single-GPU inference performance for Cosmos3-Edge-FP8 using FLASH_ATTN unless otherwise noted.
 
-Generator results are measured using end-to-end or generation latency in seconds; lower is better. Reasoner results include serving and token-generation metrics, such as time to first token, request latency, and throughput.
-
-All results were measured using a single GPU and a batch size of 1.
-
-#### Generator
-
-Unless otherwise noted, visual-generation benchmarks use **480p resolution**. Image-to-video benchmarks generate **189 frames**.
-
-##### vLLM-Omni
-
-| GPU or Platform | Image-to-Video | Forward Dynamics | Inverse Dynamics | Policy DROID |
-| --- | ---: | ---: | ---: | ---: |
-| B200 SXM 192 GB | — | 2.44 s | 3.98 s | 0.99 s |
-| H100 SXM 80 GB | 27.64 s | 3.91 s | 5.60 s | 1.41 s |
-| H100 NVL 96 GB | 35.60 s | 4.73 s | 6.39 s | 1.37 s |
-| H20 SXM 96 GB | 108.16 s | 12.77 s | 15.49 s | 3.41 s |
-| RTX PRO 6000 Blackwell Server Edition | 36.29 s | 5.65 s | 7.46 s | 1.87 s |
-| DGX Station | 12.17 s | 4.33 s | 6.34 s | 8.11 s |
-| DGX Spark | 165.96 s | 26.43 s | 30.86 s | 7.66 s |
-| Jetson AGX Thor T5000, 128 GB, MAXN | 137.50 s | 6.05 s | 7.19 s | 6.32 s |
-| Jetson T3000, 32 GB, 1100 MHz | 194.76 s | 8.67 s | 10.25 s | 8.63 s |
-| Jetson T2000, 16 GB, 702 MHz, THOR_NANO | 101.20 s | — | — | — |
-
-##### PyTorch
-
-| GPU or Platform | Image-to-Video | Forward Dynamics | Inverse Dynamics | Policy DROID |
+| Hardware | Generator — 480p (frames/s) | Reasoner — time to first token (ms) | Reasoner — output token throughput (tokens/s) | Action Policy — 480p (actions/s, 16-action chunks) |
 | --- | ---: | ---: | ---: | ---: |
-| H100 SXM 80 GB | 23.92 s | 3.69 s | 3.56 s | 1.25 s |
-| H100 NVL 96 GB | 32.24 s | 4.64 s | 4.52 s | 1.28 s |
-| H20 SXM 96 GB | 97.51 s | 12.78 s | 12.64 s | 2.92 s |
-| RTX PRO 6000 Blackwell Server Edition | 38.98 s | 5.26 s | 5.66 s | 1.32 s |
-| DGX Station | 10.57 s | 2.16 s | 2.26 s | 1.30 s |
-| DGX Spark | 179.80 s | 24.59 s | 26.76 s | 5.44 s |
-| Jetson AGX Thor T5000, 128 GB, MAXN | 153.00 s | — | — | — |
-| Jetson T3000, 32 GB, 1100 MHz | 227.80 s | — | — | — |
-
-#### Reasoner
-
-The following tables report Cosmos3-Edge-FP8 Reasoner performance. Reasoner workloads produce autoregressively generated text and therefore use different metrics from the Generator workloads:
-
-- **Time To First Token (TTFT):** Time from request submission until the first output token is produced. Lower is better.
-- **Request Latency:** End-to-end latency for the complete request. Lower is better.
-- **Request Throughput:** Completed requests per second. Higher is better.
-- **Output Token Throughput:** Generated output tokens per second. Higher is better.
-
-##### vLLM serving benchmarks
-
-These measurements use the `nvidia/Cosmos3-Edge-FP8` checkpoint with one GPU. Metrics were collected at client-side concurrency levels of 1, 64, 128, and 256.
-
-The workload notation is **input sequence length / output sequence length / video FPS**.
-
-###### RTX PRO 4500 Blackwell Server Edition
-
-| Input / Output / Video FPS | Metric | Concurrency 1 | Concurrency 64 | Concurrency 128 | Concurrency 256 |
-| --- | --- | ---: | ---: | ---: | ---: |
-| 50 / 1 / 1 | Time To First Token (ms) ↓ | 165.79 | 8817.33 | 14702.20 | 29482.39 |
-|  | Request Latency (ms) ↓ | 165.79 | 8817.33 | 14702.20 | 29482.39 |
-|  | Request Count (requests) | 50 | 320 | 256 | 512 |
-|  | Request Throughput (req/s) ↑ | 6.00 | 6.55 | 6.55 | 6.52 |
-|  | Output Token Throughput (tok/s) ↑ | 6.00 | 6.55 | 6.55 | 6.52 |
-| 50 / 1 / 2 | Time To First Token (ms) ↓ | 371.67 | 20375.98 | 33812.45 | 68201.55 |
-|  | Request Latency (ms) ↓ | 371.67 | 20375.98 | 33812.45 | 68201.55 |
-|  | Request Count (requests) | 50 | 313 | 249 | 492 |
-|  | Request Throughput (req/s) ↑ | 2.68 | 2.77 | 2.76 | 2.71 |
-|  | Output Token Throughput (tok/s) ↑ | 2.68 | 2.77 | 2.76 | 2.71 |
-| 50 / 100 / 1 | Time To First Token (ms) ↓ | 166.86 | 6900.90 | 19625.83 | 45729.55 |
-|  | Request Latency (ms) ↓ | 764.15 | 16667.01 | 29196.84 | 55749.62 |
-|  | Request Count (requests) | 50 | 320 | 256 | 512 |
-|  | Request Throughput (req/s) ↑ | 1.31 | 3.73 | 3.74 | 3.70 |
-|  | Output Token Throughput (tok/s) ↑ | 130.63 | 372.40 | 373.98 | 369.87 |
-| 50 / 100 / 2 | Time To First Token (ms) ↓ | 374.93 | 23526.65 | 47550.99 | 101553.31 |
-|  | Request Latency (ms) ↓ | 1041.29 | 33712.54 | 57641.53 | 111895.20 |
-|  | Request Count (requests) | 50 | 320 | 256 | 512 |
-|  | Request Throughput (req/s) ↑ | 0.96 | 1.79 | 1.79 | 1.78 |
-|  | Output Token Throughput (tok/s) ↑ | 95.74 | 178.73 | 178.89 | 178.15 |
-
-###### RTX PRO 6000 Blackwell Server Edition
-
-| Input / Output / Video FPS | Metric | Concurrency 1 | Concurrency 64 | Concurrency 128 | Concurrency 256 |
-| --- | --- | ---: | ---: | ---: | ---: |
-| 50 / 1 / 1 | Time To First Token (ms) ↓ | 141.99 | 3213.91 | 5384.51 | 10792.72 |
-|  | Request Latency (ms) ↓ | 141.99 | 3213.91 | 5384.51 | 10792.72 |
-|  | Request Count (requests) | 50 | 320 | 254 | 512 |
-|  | Request Throughput (req/s) ↑ | 6.96 | 18.00 | 17.95 | 17.89 |
-|  | Output Token Throughput (tok/s) ↑ | 6.96 | 18.00 | 17.95 | 17.89 |
-| 50 / 1 / 2 | Time To First Token (ms) ↓ | 239.86 | 7483.22 | 12552.69 | 25259.11 |
-|  | Request Latency (ms) ↓ | 239.86 | 7483.22 | 12552.69 | 25259.11 |
-|  | Request Count (requests) | 49 | 303 | 249 | 491 |
-|  | Request Throughput (req/s) ↑ | 4.06 | 7.28 | 7.49 | 7.34 |
-|  | Output Token Throughput (tok/s) ↑ | 4.06 | 7.28 | 7.49 | 7.34 |
-| 50 / 100 / 1 | Time To First Token (ms) ↓ | 138.74 | 943.46 | 2680.17 | 11599.63 |
-|  | Request Latency (ms) ↓ | 503.44 | 6188.90 | 13022.07 | 26388.89 |
-|  | Request Count (requests) | 50 | 320 | 256 | 512 |
-|  | Request Throughput (req/s) ↑ | 1.98 | 10.27 | 9.57 | 8.95 |
-|  | Output Token Throughput (tok/s) ↑ | 197.75 | 1026.14 | 956.47 | 893.91 |
-| 50 / 100 / 2 | Time To First Token (ms) ↓ | 239.24 | 1798.96 | 11644.84 | 33293.32 |
-|  | Request Latency (ms) ↓ | 638.71 | 13599.89 | 26299.90 | 49165.91 |
-|  | Request Count (requests) | 50 | 320 | 256 | 512 |
-|  | Request Throughput (req/s) ↑ | 1.56 | 4.66 | 4.50 | 4.45 |
-|  | Output Token Throughput (tok/s) ↑ | 155.93 | 465.28 | 449.57 | 444.17 |
-
-##### Embedded-platform eager Transformers benchmarks
-
-These preliminary measurements use raw Hugging Face Transformers in eager mode rather than vLLM. They are presented separately because their runtime, workload, and metric definitions differ from the vLLM serving benchmarks above.
-
-| Board | Specification | Input | Prompt Tokens | Prefill Throughput | Prefill Latency | Decode Throughput | E2E Latency |
-| --- | --- | --- | ---: | ---: | ---: | ---: | ---: |
-| Jetson AGX Thor T5000 | 128 GB / MAXN | Text | 1705 | 8717 tok/s | 0.20 s | 37.3 tok/s | 3.60 s |
-| Jetson AGX Thor T5000 | 128 GB / MAXN | Image | 911 | 4845 tok/s | 0.19 s | 42.6 tok/s | 3.17 s |
-| Jetson AGX Thor T5000 | 128 GB / MAXN | Video | 1263 | 6032 tok/s | 0.21 s | 41.8 tok/s | 3.25 s |
-| Jetson AGX Thor T4000 | 64 GB / MAXN, 1530 MHz | Text | 1705 | 6519 tok/s | 0.26 s | 34.1 tok/s | 3.99 s |
-| Jetson AGX Thor T4000 | 64 GB / MAXN, 1530 MHz | Image | 911 | 3471 tok/s | 0.26 s | 40.3 tok/s | 3.41 s |
-| Jetson AGX Thor T4000 | 64 GB / MAXN, 1530 MHz | Video | 1263 | 4164 tok/s | 0.30 s | 38.1 tok/s | 3.64 s |
-| Jetson Thor T3000 | 32 GB / 1100 MHz | Text | 1705 | 5230 tok/s | 0.33 s | 29.7 tok/s | 4.61 s |
-| Jetson Thor T3000 | 32 GB / 1100 MHz | Image | 911 | 2710 tok/s | 0.34 s | 36.3 tok/s | 3.83 s |
-| Jetson Thor T3000 | 32 GB / 1100 MHz | Video | 1263 | 3388 tok/s | 0.37 s | 33.7 tok/s | 4.14 s |
-| Jetson Thor T2000 | 16 GB / 702 MHz, THOR_NANO | Text | 1705 | 2355 tok/s | 0.72 s | 15.7 tok/s | 8.80 s |
-| Jetson Thor T2000 | 16 GB / 702 MHz, THOR_NANO | Image | 911 | 1233 tok/s | 0.74 s | 19.6 tok/s | 7.21 s |
-| Jetson Thor T2000 | 16 GB / 702 MHz, THOR_NANO | Video | 1263 | 1543 tok/s | 0.82 s | 18.0 tok/s | 7.87 s |
-| Jetson AGX Orin | 64 GB | Text | 1705 | 3260 tok/s | 0.52 s | 12.3 tok/s | 10.83 s |
-| Jetson AGX Orin | 64 GB | Image | 911 | 1840 tok/s | 0.50 s | 12.3 tok/s | 10.81 s |
-| Jetson AGX Orin | 64 GB | Video | 1263 | 2103 tok/s | 0.60 s | 12.2 tok/s | 10.97 s |
-
-**Benchmark notes**
-
-- All Generator measurements use one GPU or one integrated computing platform.
-- Generator values are average end-to-end or generation latency in seconds; lower is better.
-- Unless otherwise specified, visual-generation measurements use **480p resolution**.
-- Image-to-video measurements generate **189 output frames**.
-- Jetson AGX Thor T5000 and Jetson T3000 visual-generation measurements use **832 × 480** resolution.
-- Jetson T2000 visual-generation measurements use **448 × 256** resolution and therefore should not be compared directly with the 480p results. Its image-to-video values are warm-run measurements generating 189 frames.
-- PyTorch Generator values report average generation latency rather than diffusion-only latency.
-- Datacenter and enterprise forward- and inverse-dynamics results use the autonomous-driving (`AV`) configuration.
-- Jetson AGX Thor T5000 and Jetson T3000 forward-dynamics, inverse-dynamics, and policy measurements use the DROID configuration with action chunk `[16, 8]`.
+| H100 | 13.04 | 108.1 | 312.64 | 16.33 |
+| H200 | 12.96 | 103.0 | 341.18 | 16.39 |
+| H20 | 3.30 | 163.6 | 262.42 | 6.05 |
+| DGX Station | - | - | - | - |
+| DGX Spark | - | - | - | - |
+| RTX PRO 6000 | 7.52 | - | - | 6.91 |
+| RTX PRO 5000 | - | - | - | - |
+| Jetson Thor T5000 | - | 205.7 | 76.69 | - |
+| Jetson T3000 | - | 384.3 | 62.16 | - |
+| Jetson T2000 | - | - | - | - |
+| Jetson AGX Orin | - | 817.8 | 41.46 | - |
+
+A dash (-) indicates unavailable benchmark results for the specified configuration.
+
+- **Generator:** FP8 image-to-video (`i2v`) with vLLM-Omni, 832 × 480 resolution, 121 generated frames, 35 denoising steps, concurrency 1, and one warmup request. Each GPU uses its fastest measured backend for this image-to-video workload: FLASH_ATTN for H100 and H200; CUDNN_ATTN for H20 and RTX PRO 6000. Throughput is 121 divided by end-to-end server inference time in seconds and measures inference throughput. The reported times are 9.277 s (H100), 9.339 s (H200), 36.677 s (H20), and 16.085 s (RTX PRO 6000).
+- **Reasoner:** FP8 vLLM with concurrency 1, 50 input text tokens plus a 30-second, 374 × 374 video sampled at FPS 1 (30 frames), and 100 output tokens. Both metrics use this workload and FLASH_ATTN; output throughput copies the reported output-token throughput.
+- **Action Policy:** FP8 vLLM-Omni, Policy Robot (DROID), 480p, one GPU, 16-action chunks, 30 denoising steps, concurrency 1, and one warmup request. Throughput is 16 divided by end-to-end server inference time in seconds, reported in actions per second. Each GPU uses its fastest measured backend for this workload: CUDNN_ATTN for H100, H200, H20, and RTX PRO 6000. The reported times are 0.980 s (H100), 0.976 s (H200), 2.645 s (H20), and 2.315 s (RTX PRO 6000).
+- **Hardware:** Generator measurements use one H100 SXM 80 GB, H200 SXM 141 GB, or H20 SXM 96 GB GPU. The RTX PRO 6000 generator measurement uses one RTX PRO 6000 Blackwell Server Edition GPU.
+- **Embedded hardware:** Reasoner measurements use one integrated GPU: Jetson AGX Thor T5000 (128 GB), a T5000 devkit configured to emulate T3000 (12 SMs, 8 CPUs, 32 GB), and Jetson AGX Orin (64 GB), all in MAXN mode with clocks locked. Orin runs the FP8 checkpoint with weight-only W8A16 compute; Thor uses FP8 compute. T3000 results are emulated measurements. Hardware variants for other rows require matching benchmark results.
+- Throughputs are calculated before rounding to two decimal places; time to first token is displayed to one decimal place. No estimates are included.
 
 ## Usage