Spaces:
Running
Running
Minette Kaunismäki commited on
Commit ·
9856a54
1
Parent(s): 3970466
adding missing models
Browse files
README.md
CHANGED
|
@@ -23,8 +23,10 @@ The live Space is [PrunaAI/P-Bench](https://huggingface.co/spaces/PrunaAI/P-Benc
|
|
| 23 |
The current P-Video-2 Pro evaluation snapshots are
|
| 24 |
`data/t2v-pro.csv` (text-to-video) and
|
| 25 |
`data/i2v-pro.csv` (image-to-video). Both feed the leaderboards and
|
| 26 |
-
price/time Pareto plots with Datapoint and Rapidata Elo.
|
| 27 |
-
|
|
|
|
|
|
|
| 28 |
sample media is not included in these exports.
|
| 29 |
Rapidata Elo comes directly from these exports.
|
| 30 |
|
|
|
|
| 23 |
The current P-Video-2 Pro evaluation snapshots are
|
| 24 |
`data/t2v-pro.csv` (text-to-video) and
|
| 25 |
`data/i2v-pro.csv` (image-to-video). Both feed the leaderboards and
|
| 26 |
+
price/time Pareto plots with Datapoint and Rapidata Elo. All
|
| 27 |
+
models appear in the video-generation leaderboards, including variants with
|
| 28 |
+
missing scores. Missing scores and ranks display as dashes; Pareto plots
|
| 29 |
+
include only models with the required score and price or timing. Image-to-video
|
| 30 |
sample media is not included in these exports.
|
| 31 |
Rapidata Elo comes directly from these exports.
|
| 32 |
|
app.py
CHANGED
|
@@ -2431,8 +2431,6 @@ def load_video_editing_dataframe(path):
|
|
| 2431 |
def load_video_generation_dataframe(path):
|
| 2432 |
"""Load a text-to-video or image-to-video evaluation snapshot."""
|
| 2433 |
df = pd.read_csv(path, na_values=["N/A", "n/a", ""])
|
| 2434 |
-
model_column = "model_id" if "model_id" in df.columns else "Model"
|
| 2435 |
-
df = df[~df[model_column].astype(str).str.casefold().str.startswith("agnes")].copy()
|
| 2436 |
df = df.rename(
|
| 2437 |
columns={
|
| 2438 |
"model_id": "Model",
|
|
@@ -2469,13 +2467,6 @@ def load_video_generation_dataframe(path):
|
|
| 2469 |
columns=["Execution Time / Output Video Second (s)"],
|
| 2470 |
errors="ignore",
|
| 2471 |
)
|
| 2472 |
-
elo_columns = [
|
| 2473 |
-
column
|
| 2474 |
-
for column in ("Datapoint Elo", "Rapidata Elo")
|
| 2475 |
-
if column in df.columns
|
| 2476 |
-
]
|
| 2477 |
-
if elo_columns:
|
| 2478 |
-
df = df.dropna(subset=elo_columns, how="all")
|
| 2479 |
return df.reset_index(drop=True)
|
| 2480 |
|
| 2481 |
|
|
@@ -2714,7 +2705,7 @@ datasets = [
|
|
| 2714 |
},
|
| 2715 |
{
|
| 2716 |
"id": "image_to_video",
|
| 2717 |
-
"name": "
|
| 2718 |
"modality": "image_to_video",
|
| 2719 |
"data": image_to_video_df,
|
| 2720 |
"columns": text_to_video_display_columns,
|
|
|
|
| 2431 |
def load_video_generation_dataframe(path):
|
| 2432 |
"""Load a text-to-video or image-to-video evaluation snapshot."""
|
| 2433 |
df = pd.read_csv(path, na_values=["N/A", "n/a", ""])
|
|
|
|
|
|
|
| 2434 |
df = df.rename(
|
| 2435 |
columns={
|
| 2436 |
"model_id": "Model",
|
|
|
|
| 2467 |
columns=["Execution Time / Output Video Second (s)"],
|
| 2468 |
errors="ignore",
|
| 2469 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2470 |
return df.reset_index(drop=True)
|
| 2471 |
|
| 2472 |
|
|
|
|
| 2705 |
},
|
| 2706 |
{
|
| 2707 |
"id": "image_to_video",
|
| 2708 |
+
"name": "Vbench-i2V",
|
| 2709 |
"modality": "image_to_video",
|
| 2710 |
"data": image_to_video_df,
|
| 2711 |
"columns": text_to_video_display_columns,
|
ui.py
CHANGED
|
@@ -112,15 +112,19 @@ prompt suites, so samples are not shown.
|
|
| 112 |
## Current datasets
|
| 113 |
|
| 114 |
### VBench-2.0 Dataset
|
| 115 |
-
VBench-2.0
|
|
|
|
|
|
|
| 116 |
Quality is Datapoint Elo and Rapidata Elo from pairwise preference. Price
|
| 117 |
is USD per second of output video. Time per second of video is provider wall
|
| 118 |
time, except Pruna models which use model execution time. Samples are
|
| 119 |
available.
|
| 120 |
|
| 121 |
-
###
|
| 122 |
-
|
| 123 |
-
with other models, including 480p and 768p variants.
|
|
|
|
|
|
|
| 124 |
Elo and Rapidata Elo from pairwise preference. Price is USD per second of
|
| 125 |
output video. Time per second of video is provider wall time, except Pruna
|
| 126 |
models which use model execution time. Sample media is not yet available.
|
|
@@ -203,8 +207,8 @@ models *within* a Dataset | Metric view.
|
|
| 203 |
- **Prompt counts:** OneIG Alignment uses 100 anime, 100 human, and 99 object
|
| 204 |
prompts (299 total). Qwen Image Dataset uses 100 prompts sampled from the
|
| 205 |
1,000-prompt pool for roughly even coverage of its fine-grained (L3)
|
| 206 |
-
categories.
|
| 207 |
-
|
| 208 |
prompts across advertising, e-commerce, real estate, camera, lighting,
|
| 209 |
text, and related categories. Artificial Analysis and Arena AI use their
|
| 210 |
own private prompt sets.
|
|
@@ -214,11 +218,10 @@ models *within* a Dataset | Metric view.
|
|
| 214 |
seed is derived from the prompt, so every model gets the same seed for the
|
| 215 |
same prompt. Steps, CFG, prompt rewrite, and safety filters follow each
|
| 216 |
endpoint's default. This does not describe Artificial Analysis or Arena AI.
|
| 217 |
-
- **Generation (Text-to-Video):**
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
Pro Cost variants and 85–100 for other models, with five-second output clips.
|
| 222 |
- **Generation (Video-Edit):** one edited clip per prompt per endpoint when
|
| 223 |
the run exists. Every model sees the same source video for a prompt.
|
| 224 |
- **Datapoint (Qwen and OneIG):** every model pair is compared on every
|
|
@@ -649,7 +652,10 @@ def _assign_leaderboard_ranks(data, overall_column):
|
|
| 649 |
overall_column, ascending=False, na_position="last"
|
| 650 |
)
|
| 651 |
ranked = ranked.reset_index(drop=True)
|
| 652 |
-
|
|
|
|
|
|
|
|
|
|
| 653 |
return ranked
|
| 654 |
|
| 655 |
|
|
|
|
| 112 |
## Current datasets
|
| 113 |
|
| 114 |
### VBench-2.0 Dataset
|
| 115 |
+
100 samples from VBench-2.0, comparing P-Video-2 and P-Video-2 Pro variants
|
| 116 |
+
with other models. Categories are sampled in proportion to their prompt
|
| 117 |
+
counts in the original dataset.
|
| 118 |
Quality is Datapoint Elo and Rapidata Elo from pairwise preference. Price
|
| 119 |
is USD per second of output video. Time per second of video is provider wall
|
| 120 |
time, except Pruna models which use model execution time. Samples are
|
| 121 |
available.
|
| 122 |
|
| 123 |
+
### Vbench-i2V
|
| 124 |
+
100 image-to-video samples, comparing P-Video-2 Pro cost, speed, and quality
|
| 125 |
+
modes with other models, including 480p and 768p variants. Sampling uses the
|
| 126 |
+
images' `subject` attribute, with each subject represented in proportion
|
| 127 |
+
to its frequency in the original dataset. Quality is Datapoint
|
| 128 |
Elo and Rapidata Elo from pairwise preference. Price is USD per second of
|
| 129 |
output video. Time per second of video is provider wall time, except Pruna
|
| 130 |
models which use model execution time. Sample media is not yet available.
|
|
|
|
| 207 |
- **Prompt counts:** OneIG Alignment uses 100 anime, 100 human, and 99 object
|
| 208 |
prompts (299 total). Qwen Image Dataset uses 100 prompts sampled from the
|
| 209 |
1,000-prompt pool for roughly even coverage of its fine-grained (L3)
|
| 210 |
+
categories. VBench-2.0 Dataset and Vbench-i2V each use 100 samples.
|
| 211 |
+
The Pruna Internal Video-Edit Benchmark uses 78
|
| 212 |
prompts across advertising, e-commerce, real estate, camera, lighting,
|
| 213 |
text, and related categories. Artificial Analysis and Arena AI use their
|
| 214 |
own private prompt sets.
|
|
|
|
| 218 |
seed is derived from the prompt, so every model gets the same seed for the
|
| 219 |
same prompt. Steps, CFG, prompt rewrite, and safety filters follow each
|
| 220 |
endpoint's default. This does not describe Artificial Analysis or Arena AI.
|
| 221 |
+
- **Generation (Text-to-Video):** 100 samples, with a five-second video
|
| 222 |
+
generated for each sample.
|
| 223 |
+
- **Generation (Image-to-Video):** 100 samples, with a five-second video
|
| 224 |
+
generated for each sample.
|
|
|
|
| 225 |
- **Generation (Video-Edit):** one edited clip per prompt per endpoint when
|
| 226 |
the run exists. Every model sees the same source video for a prompt.
|
| 227 |
- **Datapoint (Qwen and OneIG):** every model pair is compared on every
|
|
|
|
| 652 |
overall_column, ascending=False, na_position="last"
|
| 653 |
)
|
| 654 |
ranked = ranked.reset_index(drop=True)
|
| 655 |
+
ranks = pd.Series(ranked.index + 1, index=ranked.index, dtype="Int64")
|
| 656 |
+
if overall_column and overall_column in ranked.columns:
|
| 657 |
+
ranks = ranks.where(ranked[overall_column].notna())
|
| 658 |
+
ranked.insert(0, "Rank", ranks)
|
| 659 |
return ranked
|
| 660 |
|
| 661 |
|