Minette Kaunismäki commited on
Commit
9856a54
·
1 Parent(s): 3970466

adding missing models

Browse files
Files changed (3) hide show
  1. README.md +4 -2
  2. app.py +1 -10
  3. ui.py +18 -12
README.md CHANGED
@@ -23,8 +23,10 @@ The live Space is [PrunaAI/P-Bench](https://huggingface.co/spaces/PrunaAI/P-Benc
23
  The current P-Video-2 Pro evaluation snapshots are
24
  `data/t2v-pro.csv` (text-to-video) and
25
  `data/i2v-pro.csv` (image-to-video). Both feed the leaderboards and
26
- price/time Pareto plots with Datapoint and Rapidata Elo. Variants without
27
- either Elo stay in the raw CSVs but do not appear in ranked views. Image-to-video
 
 
28
  sample media is not included in these exports.
29
  Rapidata Elo comes directly from these exports.
30
 
 
23
  The current P-Video-2 Pro evaluation snapshots are
24
  `data/t2v-pro.csv` (text-to-video) and
25
  `data/i2v-pro.csv` (image-to-video). Both feed the leaderboards and
26
+ price/time Pareto plots with Datapoint and Rapidata Elo. All
27
+ models appear in the video-generation leaderboards, including variants with
28
+ missing scores. Missing scores and ranks display as dashes; Pareto plots
29
+ include only models with the required score and price or timing. Image-to-video
30
  sample media is not included in these exports.
31
  Rapidata Elo comes directly from these exports.
32
 
app.py CHANGED
@@ -2431,8 +2431,6 @@ def load_video_editing_dataframe(path):
2431
  def load_video_generation_dataframe(path):
2432
  """Load a text-to-video or image-to-video evaluation snapshot."""
2433
  df = pd.read_csv(path, na_values=["N/A", "n/a", ""])
2434
- model_column = "model_id" if "model_id" in df.columns else "Model"
2435
- df = df[~df[model_column].astype(str).str.casefold().str.startswith("agnes")].copy()
2436
  df = df.rename(
2437
  columns={
2438
  "model_id": "Model",
@@ -2469,13 +2467,6 @@ def load_video_generation_dataframe(path):
2469
  columns=["Execution Time / Output Video Second (s)"],
2470
  errors="ignore",
2471
  )
2472
- elo_columns = [
2473
- column
2474
- for column in ("Datapoint Elo", "Rapidata Elo")
2475
- if column in df.columns
2476
- ]
2477
- if elo_columns:
2478
- df = df.dropna(subset=elo_columns, how="all")
2479
  return df.reset_index(drop=True)
2480
 
2481
 
@@ -2714,7 +2705,7 @@ datasets = [
2714
  },
2715
  {
2716
  "id": "image_to_video",
2717
- "name": "Image-to-Video Benchmark",
2718
  "modality": "image_to_video",
2719
  "data": image_to_video_df,
2720
  "columns": text_to_video_display_columns,
 
2431
  def load_video_generation_dataframe(path):
2432
  """Load a text-to-video or image-to-video evaluation snapshot."""
2433
  df = pd.read_csv(path, na_values=["N/A", "n/a", ""])
 
 
2434
  df = df.rename(
2435
  columns={
2436
  "model_id": "Model",
 
2467
  columns=["Execution Time / Output Video Second (s)"],
2468
  errors="ignore",
2469
  )
 
 
 
 
 
 
 
2470
  return df.reset_index(drop=True)
2471
 
2472
 
 
2705
  },
2706
  {
2707
  "id": "image_to_video",
2708
+ "name": "Vbench-i2V",
2709
  "modality": "image_to_video",
2710
  "data": image_to_video_df,
2711
  "columns": text_to_video_display_columns,
ui.py CHANGED
@@ -112,15 +112,19 @@ prompt suites, so samples are not shown.
112
  ## Current datasets
113
 
114
  ### VBench-2.0 Dataset
115
- VBench-2.0 prompts, comparing P-Video-2 and P-Video-2 Pro variants with other models.
 
 
116
  Quality is Datapoint Elo and Rapidata Elo from pairwise preference. Price
117
  is USD per second of output video. Time per second of video is provider wall
118
  time, except Pruna models which use model execution time. Samples are
119
  available.
120
 
121
- ### Image-to-Video Benchmark
122
- Image-to-video evaluations comparing P-Video-2 Pro cost, speed, and quality modes
123
- with other models, including 480p and 768p variants. Quality is Datapoint
 
 
124
  Elo and Rapidata Elo from pairwise preference. Price is USD per second of
125
  output video. Time per second of video is provider wall time, except Pruna
126
  models which use model execution time. Sample media is not yet available.
@@ -203,8 +207,8 @@ models *within* a Dataset | Metric view.
203
  - **Prompt counts:** OneIG Alignment uses 100 anime, 100 human, and 99 object
204
  prompts (299 total). Qwen Image Dataset uses 100 prompts sampled from the
205
  1,000-prompt pool for roughly even coverage of its fine-grained (L3)
206
- categories. The VBench-2.0 Dataset uses about 90
207
- generations per model. The Pruna Internal Video-Edit Benchmark uses 78
208
  prompts across advertising, e-commerce, real estate, camera, lighting,
209
  text, and related categories. Artificial Analysis and Arena AI use their
210
  own private prompt sets.
@@ -214,11 +218,10 @@ models *within* a Dataset | Metric view.
214
  seed is derived from the prompt, so every model gets the same seed for the
215
  same prompt. Steps, CFG, prompt rewrite, and safety filters follow each
216
  endpoint's default. This does not describe Artificial Analysis or Arena AI.
217
- - **Generation (Text-to-Video):** one clip per prompt per endpoint when the
218
- run exists. The export reports 10 generations for Pro Cost variants and
219
- about 90 for other models.
220
- - **Generation (Image-to-Video):** the export reports 10 generations for
221
- Pro Cost variants and 85–100 for other models, with five-second output clips.
222
  - **Generation (Video-Edit):** one edited clip per prompt per endpoint when
223
  the run exists. Every model sees the same source video for a prompt.
224
  - **Datapoint (Qwen and OneIG):** every model pair is compared on every
@@ -649,7 +652,10 @@ def _assign_leaderboard_ranks(data, overall_column):
649
  overall_column, ascending=False, na_position="last"
650
  )
651
  ranked = ranked.reset_index(drop=True)
652
- ranked.insert(0, "Rank", ranked.index + 1)
 
 
 
653
  return ranked
654
 
655
 
 
112
  ## Current datasets
113
 
114
  ### VBench-2.0 Dataset
115
+ 100 samples from VBench-2.0, comparing P-Video-2 and P-Video-2 Pro variants
116
+ with other models. Categories are sampled in proportion to their prompt
117
+ counts in the original dataset.
118
  Quality is Datapoint Elo and Rapidata Elo from pairwise preference. Price
119
  is USD per second of output video. Time per second of video is provider wall
120
  time, except Pruna models which use model execution time. Samples are
121
  available.
122
 
123
+ ### Vbench-i2V
124
+ 100 image-to-video samples, comparing P-Video-2 Pro cost, speed, and quality
125
+ modes with other models, including 480p and 768p variants. Sampling uses the
126
+ images' `subject` attribute, with each subject represented in proportion
127
+ to its frequency in the original dataset. Quality is Datapoint
128
  Elo and Rapidata Elo from pairwise preference. Price is USD per second of
129
  output video. Time per second of video is provider wall time, except Pruna
130
  models which use model execution time. Sample media is not yet available.
 
207
  - **Prompt counts:** OneIG Alignment uses 100 anime, 100 human, and 99 object
208
  prompts (299 total). Qwen Image Dataset uses 100 prompts sampled from the
209
  1,000-prompt pool for roughly even coverage of its fine-grained (L3)
210
+ categories. VBench-2.0 Dataset and Vbench-i2V each use 100 samples.
211
+ The Pruna Internal Video-Edit Benchmark uses 78
212
  prompts across advertising, e-commerce, real estate, camera, lighting,
213
  text, and related categories. Artificial Analysis and Arena AI use their
214
  own private prompt sets.
 
218
  seed is derived from the prompt, so every model gets the same seed for the
219
  same prompt. Steps, CFG, prompt rewrite, and safety filters follow each
220
  endpoint's default. This does not describe Artificial Analysis or Arena AI.
221
+ - **Generation (Text-to-Video):** 100 samples, with a five-second video
222
+ generated for each sample.
223
+ - **Generation (Image-to-Video):** 100 samples, with a five-second video
224
+ generated for each sample.
 
225
  - **Generation (Video-Edit):** one edited clip per prompt per endpoint when
226
  the run exists. Every model sees the same source video for a prompt.
227
  - **Datapoint (Qwen and OneIG):** every model pair is compared on every
 
652
  overall_column, ascending=False, na_position="last"
653
  )
654
  ranked = ranked.reset_index(drop=True)
655
+ ranks = pd.Series(ranked.index + 1, index=ranked.index, dtype="Int64")
656
+ if overall_column and overall_column in ranked.columns:
657
+ ranks = ranks.where(ranked[overall_column].notna())
658
+ ranked.insert(0, "Rank", ranks)
659
  return ranked
660
 
661