tangyue0820 commited on
Commit
be2ea4c
·
verified ·
1 Parent(s): 542b2c8

Sync Quality Benchmark from reviewed MR !13060

Browse files

Sync only the complete Quality Benchmark section from GitLab MR !13060 at eddc7e7a1f221b6f31ec6c107e32e91d8fc31bea. Preserve all other README sections and repository files unchanged.

Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -239,7 +239,7 @@ For more information about the datasets used to train this model, please see the
239
 
240
  ### Reasoner Benchmarks
241
 
242
- We compare **Cosmos3-Edge-FP8** with its BF16 counterpart across general reasoning, robotics, smart infrastructure, and driving benchmarks. Each domain score is the unweighted mean of the included benchmark scores. The benchmark list below defines the evaluation scope. Benchmarks without FP8 results are excluded.
243
 
244
  | Domain | BF16 | FP8 |
245
  | --- | --- | --- |
@@ -255,16 +255,16 @@ We compare **Cosmos3-Edge-FP8** with its BF16 counterpart across general reasoni
255
 
256
  ### Generator Benchmarks
257
 
258
- We compare **Cosmos3-Edge-FP8** with its BF16 counterpart. PAIBench-G evaluates text-to-video and image-to-video generation. RBench assesses task correctness, physical plausibility, and visual fidelity in robotics videos. PhysicsIQ compares video continuations with recorded physical outcomes.
259
 
260
  | Benchmark | BF16 | FP8 |
261
  | --- | --- | --- |
262
  | PAIBench-G I2V ↑ | 81.10 | 80.75 |
263
  | PAIBench-G T2V ↑ | 76.90 | 76.58 |
264
- | RBench ↑ | 55.10 | 56.52 |
265
- | PhysicsIQ ↑ | 31.28 | 31.37 |
266
 
267
- *For quality benchmarks, BF16 results were obtained using PyTorch. FP8 results were obtained using vLLM-Omni.*
268
 
269
  ## Performance Benchmark
270
 
 
239
 
240
  ### Reasoner Benchmarks
241
 
242
+ We compare **Cosmos3-Edge-FP8** with its BF16 counterpart across general reasoning, robotics, smart infrastructure, and driving benchmarks. Each domain score is the unweighted mean of the benchmark scores listed below, using the same benchmark set for BF16 and FP8.
243
 
244
  | Domain | BF16 | FP8 |
245
  | --- | --- | --- |
 
255
 
256
  ### Generator Benchmarks
257
 
258
+ We compare **Cosmos3-Edge-FP8** with its BF16 counterpart. PAIBench-G evaluates text-to-video and image-to-video generation. RBench assesses task correctness, physical plausibility, and visual fidelity in robotics videos. PhysicsIQ evaluates image-to-video generation by comparing predicted video continuations with recorded physical outcomes.
259
 
260
  | Benchmark | BF16 | FP8 |
261
  | --- | --- | --- |
262
  | PAIBench-G I2V ↑ | 81.10 | 80.75 |
263
  | PAIBench-G T2V ↑ | 76.90 | 76.58 |
264
+ | RBench ↑ | 0.55 | 0.57 |
265
+ | PhysicsIQ I2V ↑ | 31.28 | 31.37 |
266
 
267
+ *For quality benchmarks, BF16 results were obtained using PyTorch, and FP8 results were obtained using vLLM-Omni.*
268
 
269
  ## Performance Benchmark
270