Spaces:
Running
Running
| title: P-Bench | |
| emoji: 🥇 | |
| colorFrom: green | |
| colorTo: indigo | |
| sdk: gradio | |
| pinned: true | |
| license: apache-2.0 | |
| short_description: A cost/quality/speed Leaderboard for Inference Providers! | |
| sdk_version: 5.19.0 | |
| tags: | |
| - leaderboard | |
| # P-Bench | |
| Compare text-to-image, text-to-video, image-to-video, and video-to-video | |
| models on quality, speed, and price. This repo is the | |
| Gradio dashboard: leaderboards, Pareto plots, and side-by-side samples. | |
| The live Space is [PrunaAI/P-Bench](https://huggingface.co/spaces/PrunaAI/P-Bench). | |
| The current P-Video-2 Pro evaluation snapshots are | |
| `data/t2v-pro.csv` (text-to-video) and | |
| `data/i2v-pro.csv` (image-to-video). Both feed the leaderboards and | |
| price/time Pareto plots with Datapoint and Rapidata Elo. All | |
| models appear in the video-generation leaderboards, including variants with | |
| missing scores. Missing scores and ranks display as dashes; Pareto plots | |
| include only models with the required score and price or timing. Image-to-video | |
| sample media is not included in these exports. | |
| Rapidata Elo comes directly from these exports. | |
| ## License | |
| The P-Bench code (the Gradio app, `app.py`, `ui.py`, `model_display.py`) is | |
| released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). | |
| The data in `data/` is **not** covered by that license. Third-party benchmark | |
| prompts and leaderboard data keep the license or terms of their original | |
| source, and Pruna AI's own data is all rights reserved: | |
| | Data in this Space | Source | License / terms | | |
| | --- | --- | --- | | |
| | `data/oneig_combined/prompts.jsonl`: 299 OneIG-Bench prompts (anime, human and object alignment) | [OneIG-Bench](https://huggingface.co/datasets/OneIG-Bench/OneIG-Bench) (Chang et al., 2025) | [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) | | |
| | `data/qwen_image_bench_combined/prompts.jsonl`: 100 Qwen-Image-Bench prompts | [Qwen-Image-Bench](https://huggingface.co/datasets/Qwen/Qwen-Image-Bench) (Li et al., 2026) | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) | | |
| | `data/video_generation_combined/prompts.jsonl`: 90 VBench-2.0 prompts | [VBench-2.0](https://github.com/Vchitect/VBench/tree/master/VBench-2.0) (Zheng et al., 2025) | [Apache 2.0](https://github.com/Vchitect/VBench/blob/master/LICENSE) | | |
| | `data/video_editing_combined/prompts.jsonl`: 78 video-editing prompts and their source videos | Pruna AI internal video-editing set | © Pruna AI, all rights reserved | | |
| | `data/arena_ai_text_to_image_leaderboard.csv` | [Arena](https://arena.ai/leaderboard/text-to-image/overall) text-to-image leaderboard (formerly LMArena) | [Arena terms of use](https://arena.ai/terms) | | |
| | `data/artificial_analysis_text_to_image_leaderboard.csv` | [Artificial Analysis](https://artificialanalysis.ai/) text-to-image arena | [Artificial Analysis terms of use](https://artificialanalysis.ai/docs/legal/Terms-of-Use.pdf) | | |
| The OneIG-Bench, Qwen-Image-Bench and VBench-2.0 prompts are copied verbatim. | |
| P-Bench uses a selection of each benchmark and adds its own prompt IDs. The | |
| generated images and videos come from the listed models. Human preference | |
| scores (Elo) were collected by Pruna AI through Datapoint and Rapidata. | |
| Because the OneIG-Bench prompts are licensed for non-commercial use only | |
| (CC BY-NC 4.0), any reuse of `data/oneig_combined/` must follow that license. | |
| All other data in `data/` was created by Pruna AI and is © Pruna AI, all | |
| rights reserved. This covers the video-editing prompts and source videos, the | |
| Elo exports (for example `t2v-pro.csv`, `i2v-pro.csv`, | |
| `video-editing-leaderboard.csv`) and the price and generation-time CSVs. | |
| ## Citation | |
| If you use P-Bench, please cite it together with the benchmarks whose prompts it uses: | |
| ```bibtex | |
| @misc{P-Bench, | |
| title={P-Bench: A Leaderboard for Image and Video Generation Models}, | |
| author={PrunaAI}, | |
| year={2026}, | |
| howpublished={\url{https://huggingface.co/spaces/PrunaAI/P-Bench}} | |
| } | |
| @article{chang2025oneig, | |
| title={OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation}, | |
| author={Jingjing Chang and Yixiao Fang and Peng Xing and Shuhan Wu and Wei Cheng and Rui Wang and Xianfang Zeng and Gang Yu and Hai-Bao Chen}, | |
| journal={arXiv preprint arXiv:2506.07977}, | |
| year={2025} | |
| } | |
| @misc{li2026qwenimagebenchgenerationcreationtexttoimage, | |
| title={Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation}, | |
| author={Niantong Li and Guangzheng Hu and Weixu Qiao and Ying Ba and Qichen Hong and Shijun Shen and Jinlin Wang and Fan Zhou and Jianye Kang and Xin Shang and Ziyi He and Wei Wang and Dalin Li and Jiahao Li and Jie Zhang and Kaiyuan Gao and Kun Yan and Lihan Jiang and Ningyuan Tang and Shengming Yin and Tianhe Wu and Xiao Xu and Xiaoyue Chen and Yuxiang Chen and Yan Shu and Yanran Zhang and Yilei Chen and Yixian Xu and Zekai Zhang and Zhendong Wang and Zihao Liu and Zikai Zhou and Hongzhu Shi and Yi Wang and Bing Zhao and Hu Wei and Lin Qu and Chenfei Wu}, | |
| year={2026}, | |
| eprint={2605.28091}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CV}, | |
| url={https://arxiv.org/abs/2605.28091} | |
| } | |
| @article{zheng2025vbench2, | |
| title={{VBench-2.0}: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness}, | |
| author={Zheng, Dian and Huang, Ziqi and Liu, Hongbo and Zou, Kai and He, Yinan and Zhang, Fan and Zhang, Yuanhan and He, Jingwen and Zheng, Wei-Shi and Qiao, Yu and Liu, Ziwei}, | |
| journal={arXiv preprint arXiv:2503.21755}, | |
| year={2025} | |
| } | |
| ``` | |
| ## Run locally | |
| From the repo root: | |
| ``` | |
| python -m venv .venv | |
| source .venv/bin/activate | |
| pip install "gradio==5.19.0" pandas -r requirements.txt | |
| python app.py | |
| ``` | |
| `requirements.txt` lists Plotly. Gradio and pandas are required locally; | |
| Hugging Face Spaces installs Gradio from the YAML `sdk_version` above. | |
| ## Deploy | |
| `origin` is the Space (`https://huggingface.co/spaces/PrunaAI/P-Bench`). | |
| Publish the current branch to the live app with: | |
| ``` | |
| git push origin HEAD:main | |
| ``` | |