| --- |
| title: README |
| emoji: 😻 |
| colorFrom: indigo |
| colorTo: indigo |
| sdk: static |
| pinned: false |
| --- |
| |
| # EvoEval: Evolving Coding Benchmarks via LLM |
|
|
| **EvoEval**<sup>1</sup> is a holistic benchmark suite created by _evolving_ **HumanEval** problems: |
| - 🔥 Contains **828** new problems across **5** 🌠 semantic-altering and **2** ⭐ semantic-preserving benchmarks |
| - 🔮 Allows evaluation/comparison across different **dimensions** and problem **types** (i.e., _Difficult_, _Creative_ or _Tool Use_ problems). See our [**visualization tool**](https://evo-eval.github.io/visualization.html) for ready-to-use comparison |
| - 🏆 Complete with [**leaderboard**](https://evo-eval.github.io/leaderboard.html), **groundtruth solutions**, **robust testcases** and **evaluation scripts** to easily fit into your evaluation pipeline |
| - 🤖 Generated LLM code samples from **>50** different models to save you time in running experiments |
|
|
| <sup>1</sup> coincidentally similar pronunciation with 😈 EvilEval |
|
|
| - GitHub: [evo-eval/evoeval](https://github.com/evo-eval/evoeval) |
| - Webpage: [evo-eval.github.io](https://evo-eval.github.io/) |
| - Leaderboard: [evo-eval.github.io/leaderboard.html](https://evo-eval.github.io/leaderboard.html) |
| - Visualization: [evo-eval.github.io/visualization.html](https://evo-eval.github.io/visualization.html) |
| - Paper: [arXiv](https://arxiv.org/abs/2403.19114) |
| - PyPI: [evoeval](https://pypi.org/project/evoeval/) |
|
|
