| --- |
| title: codebleu |
| tags: |
| - evaluate |
| - metric |
| description: "CodeBLEU" |
| sdk: gradio |
| sdk_version: 3.0.2 |
| app_file: app.py |
| pinned: false |
| --- |
| |
| # Metric Card for CodeBLEU |
|
|
| ## Metric Description |
|
|
| CodeBLEU from [CodeXGLUE](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/code-to-code-trans/evaluator) |
| and from article [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://arxiv.org/abs/2009.10297) |
|
|
| NOTE: currently works on Linux machines only due to dependency from languages .so |
|
|
| ## How to Use |
|
|
| ```python |
| module = evaluate.load("dvitel/codebleu") |
| src = 'class AcidicSwampOoze(MinionCard):§ def __init__(self):§ super().__init__("Acidic Swamp Ooze", 2, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Destroy(), WeaponSelector(EnemyPlayer())))§§ def create_minion(self, player):§ return Minion(3, 2)§' |
| tgt = 'class AcidSwampOoze(MinionCard):§ def __init__(self):§ super().__init__("Acidic Swamp Ooze", 2, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Destroy(), WeaponSelector(EnemyPlayer())))§§ def create_minion(self, player):§ return Minion(3, 2)§' |
| src = src.replace("§","\n") |
| tgt = tgt.replace("§","\n") |
| res = module.compute(predictions = [tgt], references = [[src]]) |
| print(res) |
| #{'CodeBLEU': 0.9473264567644872, 'ngram_match_score': 0.8915993127600096, 'weighted_ngram_match_score': 0.8977065142979394, 'syntax_match_score': 1.0, 'dataflow_match_score': 1.0} |
| ``` |
|
|
| ### Inputs |
| - **predictions** (`list` of `str`s): Translations to score. |
| - **references** (`list` of `list`s of `str`s): references for each translation. |
| - **lang** programming language in ['java','js','c_sharp','php','go','python','ruby'] |
| - **tokenizer**: approach used for standardizing `predictions` and `references`. |
| The default tokenizer is `tokenizer_13a`, a relatively minimal tokenization approach that is however equivalent to `mteval-v13a`, used by WMT. |
| This can be replaced by another tokenizer from a source such as [SacreBLEU](https://github.com/mjpost/sacrebleu/tree/master/sacrebleu/tokenizers). |
| - **params**: str, weights for averaging(see CodeBLEU paper). |
| Defaults to equal weights "0.25,0.25,0.25,0.25". |
| |
| ### Output Values |
|
|
| - CodeBLEU: resulting score, |
| - ngram_match_score: See paper CodeBLEU, |
| - weighted_ngram_match_score: See paper CodeBLEU, |
| - syntax_match_score: See paper CodeBLEU, |
| - dataflow_match_score: See paper CodeBLEU, |
| |
| #### Values from Popular Papers |
| *Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.* |
| |
| ### Examples |
| *Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.* |
| |
| ## Limitations and Bias |
| Linux OS only. See above a set of programming languages supported. |
| |
| ## Citation |
| ```bibtex |
| @InProceedings{huggingface:module, |
| title = {CodeBLEU: A Metric for Evaluating Code Generation}, |
| authors={Sedykh, Ivan}, |
| year={2022} |
| } |
| ``` |
| |
| ## Further References |
| *Add any useful further references.* |
| |
