| # Evaluating Language Models for Efficient Code Generation (COLM'24) |
|
|
| * [Paper](https://www.arxiv.org/abs/2408.06450) |
| * [Poster](https://jw-liu.xyz/assets/pdf/jiawei-colm-evalperf-poster.pdf) |
|
|
| ## Overview |
|
|
| **Code Efficiency Evaluation** requires: |
|
|
| * **Performance-exercising tasks**: |
| * Computationally non-trivial *task* |
| * Computationally intensive *test input* |
| * **Meaningful compound metric**: |
| * We need to evaluate on multiple tasks to get statistical sense of LLM's code efficiency |
| * Yet, commonly used avg. speedup is biased towards tasks with larger efficiency gaps. |
|
|
| Using **Differential Performance Evaluation**, we curate the EvalPerf dataset -- current version (`20240328`) includes: |
|
|
| * 118 performance-exercising tasks |
| * Each task is equipped with a computationally challenging test input generated by the SaS generator |
| * Differential performance score (DPS) that brings conclusions like "Your submission can outperform 80% of LLM solutions..." |
| * Pairwise comparison of LLMs' code efficiency over commonly passing tasks to ablate correctness impact |
|
|
| ## Running EvalPerf |
|
|
| ```bash |
| evalplus.evalperf --model {model_name} --backend [vllm|hf|openai|google|anthropic] |
| # model_name can be hugginface path such as `ise-uiuc/Magicoder-DS-6.7B` |
| ``` |
|
|
| This script overall performs four steps: |
|
|
| * **Step 1**: We sample 100 solutions (`n_samples`) from each LLM to evaluate |
| * **Step 2**: For tasks with at least 10 passing samples (`min_correct`), we perform efficiency evaluation |
| * **Step 3**: Produce a `{model_name}_evalperf_v{VERSION}.jsonl` file where each line includes: |
| * `task_id` (str) |
| * `results` (`List[Dict]`) |
| * `solution` (str) |
| * `pass` (bool) |
| * `profiled` (bool) |
| * `matching_cluster_idx` (`Optional[int]`) |
| * `_num_cpu_instructions` (`Optional[int]`) |
| * `dps` (`Optional[float]`) |
| * `dps_norm` (`Optional[float]`) |
| * `ref` (`List[Dict]`) |
| * `solution` (str) |
| * `score` (float; 100 based) |
| * `_num_cpu_instructions` (`Optional[int]`) |
| * `dps` (`Optional[float]`) |
| * `dps_norm` (`Optional[float]`) |
| * `pass@1` (float; 100 based) |
| * `n_profiled` (`Optional[int]`) |
| * **Step 4**: Compute the differential performance score |
|
|
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{liu2024evaluating, |
| title = {Evaluating Language Models for Efficient Code Generation}, |
| author = {Liu, Jiawei and Xie, Songrun and Wang, Junhao and Wei, Yuxiang and Ding, Yifeng and Zhang, Lingming}, |
| booktitle = {First Conference on Language Modeling}, |
| year = {2024}, |
| url = {https://openreview.net/forum?id=IBCBMeAhmC}, |
| } |
| ``` |
|
|
