Spaces:
Running
Running
README: license section and dataset citations
#11
by begumcig - opened
README.md
CHANGED
|
@@ -30,6 +30,67 @@ include only models with the required score and price or timing. Image-to-video
|
|
| 30 |
sample media is not included in these exports.
|
| 31 |
Rapidata Elo comes directly from these exports.
|
| 32 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
## Run locally
|
| 34 |
|
| 35 |
From the repo root:
|
|
|
|
| 30 |
sample media is not included in these exports.
|
| 31 |
Rapidata Elo comes directly from these exports.
|
| 32 |
|
| 33 |
+
## License
|
| 34 |
+
|
| 35 |
+
The P-Bench code (the Gradio app, `app.py`, `ui.py`, `model_display.py`) is
|
| 36 |
+
released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
|
| 37 |
+
|
| 38 |
+
The benchmark prompts and third-party data in `data/` are **not** covered by
|
| 39 |
+
that license. They keep the license or terms of their original source:
|
| 40 |
+
|
| 41 |
+
| Data in this Space | Source | License / terms |
|
| 42 |
+
| --- | --- | --- |
|
| 43 |
+
| `data/oneig_combined/prompts.jsonl`: 299 OneIG-Bench prompts (anime, human and object alignment) | [OneIG-Bench](https://huggingface.co/datasets/OneIG-Bench/OneIG-Bench) (Chang et al., 2025) | [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) |
|
| 44 |
+
| `data/qwen_image_bench_combined/prompts.jsonl`: 100 Qwen-Image-Bench prompts | [Qwen-Image-Bench](https://huggingface.co/datasets/Qwen/Qwen-Image-Bench) (Li et al., 2026) | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
|
| 45 |
+
| `data/video_generation_combined/prompts.jsonl`: 90 VBench-2.0 prompts | [VBench-2.0](https://github.com/Vchitect/VBench/tree/master/VBench-2.0) (Zheng et al., 2025) | [Apache 2.0](https://github.com/Vchitect/VBench/blob/master/LICENSE) |
|
| 46 |
+
| `data/arena_ai_text_to_image_leaderboard.csv` | Arena text-to-image leaderboard | the source's terms of use |
|
| 47 |
+
| `data/artificial_analysis_text_to_image_leaderboard.csv` | [Artificial Analysis](https://artificialanalysis.ai/) text-to-image arena | the source's terms of use |
|
| 48 |
+
|
| 49 |
+
The OneIG-Bench, Qwen-Image-Bench and VBench-2.0 prompts are copied verbatim.
|
| 50 |
+
P-Bench uses a selection of each benchmark and adds its own prompt IDs. The
|
| 51 |
+
generated images and videos come from the listed models. Human preference
|
| 52 |
+
scores (Elo) were collected by Pruna AI through Datapoint and Rapidata.
|
| 53 |
+
|
| 54 |
+
Because the OneIG-Bench prompts are licensed for non-commercial use only
|
| 55 |
+
(CC BY-NC 4.0), any reuse of `data/oneig_combined/` must follow that license.
|
| 56 |
+
|
| 57 |
+
## Citation
|
| 58 |
+
|
| 59 |
+
If you use P-Bench, please cite it together with the benchmarks whose prompts it uses:
|
| 60 |
+
|
| 61 |
+
```bibtex
|
| 62 |
+
@misc{P-Bench,
|
| 63 |
+
title={P-Bench: A Leaderboard for Text-to-Image Models},
|
| 64 |
+
author={PrunaAI},
|
| 65 |
+
year={2026},
|
| 66 |
+
howpublished={\url{https://huggingface.co/spaces/PrunaAI/P-Bench}}
|
| 67 |
+
}
|
| 68 |
+
|
| 69 |
+
@article{chang2025oneig,
|
| 70 |
+
title={OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation},
|
| 71 |
+
author={Jingjing Chang and Yixiao Fang and Peng Xing and Shuhan Wu and Wei Cheng and Rui Wang and Xianfang Zeng and Gang Yu and Hai-Bao Chen},
|
| 72 |
+
journal={arXiv preprint arXiv:2506.07977},
|
| 73 |
+
year={2025}
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
@misc{li2026qwenimagebenchgenerationcreationtexttoimage,
|
| 77 |
+
title={Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation},
|
| 78 |
+
author={Niantong Li and Guangzheng Hu and Weixu Qiao and Ying Ba and Qichen Hong and Shijun Shen and Jinlin Wang and Fan Zhou and Jianye Kang and Xin Shang and Ziyi He and Wei Wang and Dalin Li and Jiahao Li and Jie Zhang and Kaiyuan Gao and Kun Yan and Lihan Jiang and Ningyuan Tang and Shengming Yin and Tianhe Wu and Xiao Xu and Xiaoyue Chen and Yuxiang Chen and Yan Shu and Yanran Zhang and Yilei Chen and Yixian Xu and Zekai Zhang and Zhendong Wang and Zihao Liu and Zikai Zhou and Hongzhu Shi and Yi Wang and Bing Zhao and Hu Wei and Lin Qu and Chenfei Wu},
|
| 79 |
+
year={2026},
|
| 80 |
+
eprint={2605.28091},
|
| 81 |
+
archivePrefix={arXiv},
|
| 82 |
+
primaryClass={cs.CV},
|
| 83 |
+
url={https://arxiv.org/abs/2605.28091}
|
| 84 |
+
}
|
| 85 |
+
|
| 86 |
+
@article{zheng2025vbench2,
|
| 87 |
+
title={{VBench-2.0}: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness},
|
| 88 |
+
author={Zheng, Dian and Huang, Ziqi and Liu, Hongbo and Zou, Kai and He, Yinan and Zhang, Fan and Zhang, Yuanhan and He, Jingwen and Zheng, Wei-Shi and Qiao, Yu and Liu, Ziwei},
|
| 89 |
+
journal={arXiv preprint arXiv:2503.21755},
|
| 90 |
+
year={2025}
|
| 91 |
+
}
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
## Run locally
|
| 95 |
|
| 96 |
From the repo root:
|
app.py
CHANGED
|
@@ -1291,6 +1291,7 @@ button.theme-toggle[data-mode="light"] .theme-icon-moon { display: block !import
|
|
| 1291 |
}
|
| 1292 |
|
| 1293 |
.compare-samples-help { color: var(--pruna-text-body); margin-bottom: 8px; }
|
|
|
|
| 1294 |
.compare-controls,
|
| 1295 |
.compare-controls.row,
|
| 1296 |
.compare-controls.column,
|
|
@@ -2622,6 +2623,33 @@ qwen_samples = load_sample_comparison_data(qwen_combined_dir)
|
|
| 2622 |
video_samples = load_sample_comparison_data(video_combined_dir)
|
| 2623 |
text_to_video_samples = load_sample_comparison_data(text_to_video_combined_dir)
|
| 2624 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2625 |
metrics = [
|
| 2626 |
{"id": "datapoint_elo", "column": "Datapoint Elo"},
|
| 2627 |
{"id": "rapidata_elo", "column": "Rapidata Elo"},
|
|
|
|
| 1291 |
}
|
| 1292 |
|
| 1293 |
.compare-samples-help { color: var(--pruna-text-body); margin-bottom: 8px; }
|
| 1294 |
+
.compare-samples-credit { color: var(--pruna-text-muted); font-size: 0.9em; margin: 0 0 12px; }
|
| 1295 |
.compare-controls,
|
| 1296 |
.compare-controls.row,
|
| 1297 |
.compare-controls.column,
|
|
|
|
| 2623 |
video_samples = load_sample_comparison_data(video_combined_dir)
|
| 2624 |
text_to_video_samples = load_sample_comparison_data(text_to_video_combined_dir)
|
| 2625 |
|
| 2626 |
+
# Source credit shown above the sample prompts (full references are in the About tab).
|
| 2627 |
+
SAMPLE_PROMPT_CREDITS = {
|
| 2628 |
+
"oneig": (
|
| 2629 |
+
'Prompts: <a href="https://huggingface.co/datasets/OneIG-Bench/OneIG-Bench" '
|
| 2630 |
+
'target="_blank" rel="noopener noreferrer">OneIG-Bench</a> (Chang et al., 2025), '
|
| 2631 |
+
'licensed <a href="https://creativecommons.org/licenses/by-nc/4.0/" target="_blank" '
|
| 2632 |
+
'rel="noopener noreferrer">CC BY-NC 4.0</a>, non-commercial use only. See About for references.'
|
| 2633 |
+
),
|
| 2634 |
+
"qwen": (
|
| 2635 |
+
'Prompts: <a href="https://huggingface.co/datasets/Qwen/Qwen-Image-Bench" '
|
| 2636 |
+
'target="_blank" rel="noopener noreferrer">Qwen-Image-Bench</a> (Li et al., 2026), '
|
| 2637 |
+
"Apache 2.0. See About for references."
|
| 2638 |
+
),
|
| 2639 |
+
"text_to_video": (
|
| 2640 |
+
'Prompts: <a href="https://github.com/Vchitect/VBench/tree/master/VBench-2.0" '
|
| 2641 |
+
'target="_blank" rel="noopener noreferrer">VBench-2.0</a> (Zheng et al., 2025), '
|
| 2642 |
+
"Apache 2.0. See About for references."
|
| 2643 |
+
),
|
| 2644 |
+
}
|
| 2645 |
+
for _samples, _credit_id in (
|
| 2646 |
+
(oneig_samples, "oneig"),
|
| 2647 |
+
(qwen_samples, "qwen"),
|
| 2648 |
+
(text_to_video_samples, "text_to_video"),
|
| 2649 |
+
):
|
| 2650 |
+
if _samples:
|
| 2651 |
+
_samples["credit"] = SAMPLE_PROMPT_CREDITS[_credit_id]
|
| 2652 |
+
|
| 2653 |
metrics = [
|
| 2654 |
{"id": "datapoint_elo", "column": "Datapoint Elo"},
|
| 2655 |
{"id": "rapidata_elo", "column": "Rapidata Elo"},
|
ui.py
CHANGED
|
@@ -161,6 +161,39 @@ price vs score only.
|
|
| 161 |
External text-to-image Elo (overall and category) from Arena AI. Their
|
| 162 |
prompt set is private, so samples are not shown. Price and generation time
|
| 163 |
are not in this export, so Pareto plots are unavailable.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
"""
|
| 165 |
|
| 166 |
ABOUT_DETAILS_CONTENT = """
|
|
@@ -1383,6 +1416,10 @@ def _build_compare_samples_html(samples, selected_models, num_prompts, seed=0):
|
|
| 1383 |
"""
|
| 1384 |
)
|
| 1385 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1386 |
return "\n".join(blocks)
|
| 1387 |
|
| 1388 |
|
|
|
|
| 161 |
External text-to-image Elo (overall and category) from Arena AI. Their
|
| 162 |
prompt set is private, so samples are not shown. Price and generation time
|
| 163 |
are not in this export, so Pareto plots are unavailable.
|
| 164 |
+
|
| 165 |
+
## Datasets, licenses and references
|
| 166 |
+
|
| 167 |
+
The P-Bench code is released under the Apache License 2.0. The benchmark
|
| 168 |
+
prompts shown in Samples are copied verbatim from their original datasets and
|
| 169 |
+
keep those datasets' licenses; P-Bench uses a selection of each and adds its
|
| 170 |
+
own prompt IDs.
|
| 171 |
+
|
| 172 |
+
- **OneIG Alignment Dataset**: prompts from
|
| 173 |
+
[OneIG-Bench](https://huggingface.co/datasets/OneIG-Bench/OneIG-Bench),
|
| 174 |
+
licensed [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/)
|
| 175 |
+
(non-commercial use only).
|
| 176 |
+
Jingjing Chang, Yixiao Fang, Peng Xing, Shuhan Wu, Wei Cheng, Rui Wang,
|
| 177 |
+
Xianfang Zeng, Gang Yu and Hai-Bao Chen. *OneIG-Bench: Omni-dimensional
|
| 178 |
+
Nuanced Evaluation for Image Generation.* arXiv:2506.07977, 2025.
|
| 179 |
+
- **Qwen Image Dataset**: prompts from
|
| 180 |
+
[Qwen-Image-Bench](https://huggingface.co/datasets/Qwen/Qwen-Image-Bench),
|
| 181 |
+
licensed Apache 2.0.
|
| 182 |
+
Niantong Li et al. *Qwen-Image-Bench: From Generation to Creation in
|
| 183 |
+
Text-to-Image Evaluation.* arXiv:2605.28091, 2026.
|
| 184 |
+
- **VBench-2.0 Dataset**: prompts from
|
| 185 |
+
[VBench-2.0](https://github.com/Vchitect/VBench/tree/master/VBench-2.0),
|
| 186 |
+
licensed Apache 2.0.
|
| 187 |
+
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan
|
| 188 |
+
Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao and Ziwei Liu. *VBench-2.0:
|
| 189 |
+
Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness.*
|
| 190 |
+
arXiv:2503.21755, 2025.
|
| 191 |
+
- **Artificial Analysis Dataset** and **Arena AI Dataset**: Elo (and
|
| 192 |
+
Artificial Analysis price) as published on their leaderboards, under
|
| 193 |
+
their terms of use.
|
| 194 |
+
|
| 195 |
+
BibTeX for all of these is in the
|
| 196 |
+
[Space README](https://huggingface.co/spaces/PrunaAI/P-Bench/blob/main/README.md#citation).
|
| 197 |
"""
|
| 198 |
|
| 199 |
ABOUT_DETAILS_CONTENT = """
|
|
|
|
| 1416 |
"""
|
| 1417 |
)
|
| 1418 |
|
| 1419 |
+
credit = (samples or {}).get("credit")
|
| 1420 |
+
if credit:
|
| 1421 |
+
# Static HTML from SAMPLE_PROMPT_CREDITS in app.py (not user input).
|
| 1422 |
+
blocks.insert(0, f'<p class="compare-samples-credit">{credit}</p>')
|
| 1423 |
return "\n".join(blocks)
|
| 1424 |
|
| 1425 |
|