CodeBLEU

Unofficial CodeBLEU implementation with Linux and MacOS supports available with PyPI and HF HUB.

Based on original CodeXGLUE/CodeBLEU code -- refactored, build for macos, tested and fixed multiple crutches to make it more usable.

Metric Description

An ideal evaluation metric should consider the grammatical correctness and the logic correctness. We propose weighted n-gram match and syntactic AST match to measure grammatical correctness, and introduce semantic data-flow match to calculate logic correctness.
(from CodeXGLUE repo)

In a nutshell, CodeBLEU is a weighted combination of n-gram match (BLEU), weighted n-gram match (BLEU-weighted), AST match and data-flow match scores.

The metric has shown higher correlation with human evaluation than BLEU and accuracy metrics.

Usage

from codebleu import calc_codebleu

prediction = "def add ( a , b ) :\n return a + b"
reference = "def sum ( first , second ) :\n return second + first"

result = calc_codebleu([reference], [prediction], lang="python", weights=(0.25, 0.25, 0.25, 0.25), tokenizer=None)
print(result)
# {
#   'codebleu': 0.5537, 
#   'ngram_match_score': 0.1041, 
#   'weighted_ngram_match_score': 0.1109, 
#   'syntax_match_score': 1.0, 
#   'dataflow_match_score': 1.0
# }

where calc_codebleu takes the following arguments:

• refarences (list[str] or list[list[str]]): reference code
• predictions (list[str]) predicted code
• lang (str): code language, see codebleu.AVAILABLE_LANGS for available languages (python, c_sharp, java at the moment)
• weights (tuple[float,float,float,float]): weights of the ngram_match, weighted_ngram_match, syntax_match, and dataflow_match respectively, defaults to (0.25, 0.25, 0.25, 0.25)
• tokenizer (callable): to split code string to tokens, defaults to s.split()

and outputs the dict[str, float] with following fields:

• codebleu: the final CodeBLEU score
• ngram_match_score: ngram_match score (BLEU)
• weighted_ngram_match_score: weighted_ngram_match score (BLEU-weighted)
• syntax_match_score: syntax_match score (AST match)
• dataflow_match_score: dataflow_match score

Alternatively, you can use k4black/codebleu from HuggingFace Spaces:

import evaluate
metric = evaluate.load("dvitel/codebleu")

result = metric.compute([reference], [prediction], lang="python", weights=(0.25, 0.25, 0.25, 0.25))

Feel free to check the HF Space with online example: k4black/codebleu

Installation

Requires Python 3.8+

The metrics can be installed with pip and used as indicated above:

pip install codebleu

alternatively the metric is available as k4black/codebleu in evaluate (lib installation required):

import evaluate
metric = evaluate.load("dvitel/codebleu")

Citation

Official CodeBLEU paper can be cited as follows:

@misc{ren2020codebleu,
      title={CodeBLEU: a Method for Automatic Evaluation of Code Synthesis}, 
      author={Shuo Ren and Daya Guo and Shuai Lu and Long Zhou and Shujie Liu and Duyu Tang and Neel Sundaresan and Ming Zhou and Ambrosio Blanco and Shuai Ma},
      year={2020},
      eprint={2009.10297},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}