Unofficial CodeBLEU implementation that supports Linux and MacOS available on PyPI.
Project description
CodeBLEU
This repository contains an unofficial CodeBLEU
implementation that supports Linux and MacOS. It is available through PyPI
and the evaluate
library.
The code is based on the original CodeXGLUE/CodeBLEU and updated version by XLCoST/CodeBLEU. It has been refactored, tested, built for macOS, and multiple improvements have been made to enhance usability
Available for: Python
, C
, C#
, C++
, Java
, JavaScript
, PHP
.
Metric Description
An ideal evaluation metric should consider the grammatical correctness and the logic correctness. We propose weighted n-gram match and syntactic AST match to measure grammatical correctness, and introduce semantic data-flow match to calculate logic correctness.
(from CodeXGLUE repo)
In a nutshell, CodeBLEU
is a weighted combination of n-gram match (BLEU)
, weighted n-gram match (BLEU-weighted)
, AST match
and data-flow match
scores.
The metric has shown higher correlation with human evaluation than BLEU
and accuracy
metrics.
Installation
As this library require so
file compilation it is platform dependent.
Currently available for Linux
(manylinux) and MacOS
with Python 3.8+.
The metrics is available as pip package and can be installed as indicated above:
pip install codebleu
or directly from git repo:
pip install git+https://github.com/k4black/codebleu.git
Usage
from codebleu import calc_codebleu
prediction = "def add ( a , b ) :\n return a + b"
reference = "def sum ( first , second ) :\n return second + first"
result = calc_codebleu([reference], [prediction], lang="python", weights=(0.25, 0.25, 0.25, 0.25), tokenizer=None)
print(result)
# {
# 'codebleu': 0.5537,
# 'ngram_match_score': 0.1041,
# 'weighted_ngram_match_score': 0.1109,
# 'syntax_match_score': 1.0,
# 'dataflow_match_score': 1.0
# }
where calc_codebleu
takes the following arguments:
refarences
(list[str]
orlist[list[str]]
): reference codepredictions
(list[str]
) predicted codelang
(str
): code language, seecodebleu.AVAILABLE_LANGS
for available languages (python, c_sharp c, cpp, javascript, java, php at the moment)weights
(tuple[float,float,float,float]
): weights of thengram_match
,weighted_ngram_match
,syntax_match
, anddataflow_match
respectively, defaults to(0.25, 0.25, 0.25, 0.25)
tokenizer
(callable
): to split code string to tokens, defaults tos.split()
and outputs the dict[str, float]
with following fields:
codebleu
: the finalCodeBLEU
scorengram_match_score
:ngram_match
score (BLEU)weighted_ngram_match_score
:weighted_ngram_match
score (BLEU-weighted)syntax_match_score
:syntax_match
score (AST match)dataflow_match_score
:dataflow_match
score
Alternatively, you can use k4black/codebleu
from HuggingFace Spaces (codebleu
package required):
import evaluate
metric = evaluate.load("dvitel/codebleu")
prediction = "def add ( a , b ) :\n return a + b"
reference = "def sum ( first , second ) :\n return second + first"
result = metric.compute([reference], [prediction], lang="python", weights=(0.25, 0.25, 0.25, 0.25))
Feel free to check the HF Space with online example: k4black/codebleu
Contributing
Contributions are welcome!
If you have any questions, suggestions, or bug reports, please open an issue on GitHub.
License
This project is licensed under the terms of the MIT license.
Citation
Official CodeBLEU paper can be cited as follows:
@misc{ren2020codebleu,
title={CodeBLEU: a Method for Automatic Evaluation of Code Synthesis},
author={Shuo Ren and Daya Guo and Shuai Lu and Long Zhou and Shujie Liu and Duyu Tang and Neel Sundaresan and Ming Zhou and Ambrosio Blanco and Shuai Ma},
year={2020},
eprint={2009.10297},
archivePrefix={arXiv},
primaryClass={cs.SE}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for codebleu-0.3.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e739012fc4ca76a6bbae7d12e0a1d5d3827825ea77f1c4bf104ab542927c015 |
|
MD5 | d1e7cf7a8bdb9631f0ea3e6b508b9e31 |
|
BLAKE2b-256 | 7218ecb136fc7a5c5f59293e8f2b7f448fcfda3680fcc4af05b733d07a2fc707 |
Hashes for codebleu-0.3.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ed8f56965a8e0c66a54ab60588d170dc8112214b563ce393aaafbc2bd7791de |
|
MD5 | 92e7507b8c345aa7ce0d6a32a5cb1801 |
|
BLAKE2b-256 | 1e62d4448f36b5f3496db9ceacebf1da024446bc00b6adf56b5c7d8ece842bd3 |
Hashes for codebleu-0.3.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33d541cc980f24c4047c72a2c00f8a0a49058b0a244794e1c0d4d23f271cde27 |
|
MD5 | 62304b4f18ea161d599248bde0928960 |
|
BLAKE2b-256 | 8cf0c5c148aad2734565d6b1e65c562506dd2ff5168b1b38a95e1d61c3dd3cd2 |
Hashes for codebleu-0.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 641dd19f771490b77b19337187e80263232d02b983d1a93bfe4c92ebc0abb177 |
|
MD5 | 313924dfc50005cf8fc0d781802dab4d |
|
BLAKE2b-256 | 99428b2f4d5647c417b8f3045056f8cd45950213ed4eb9f5bd8dc0735e54ecdf |
Hashes for codebleu-0.3.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 033008413f46e60928bdba88111cdf87fdf60798498781e79d9f348d13d23e7c |
|
MD5 | 641b4e740164fea272418c0eb12c3893 |
|
BLAKE2b-256 | 7989b0b97b5791ce440be9a33bd2d6c5998cee8daab2c08684d7b398e4e0ebed |
Hashes for codebleu-0.3.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 776657e57eb797bceb3596ab947cd8087e5ee623367892a9653822a07ec3edf5 |
|
MD5 | eeb329756a52ef8ab7be74c288eb27bb |
|
BLAKE2b-256 | 5eb9a29a7048ce3b57980d214b6d40427ccdf7b86ed47c66af796c13e1ff51de |
Hashes for codebleu-0.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21de9472eb95e6a4d9c71d7f6e17cfb49afb1ccf5233b76fa66956b9f940c72c |
|
MD5 | 85b9caacde225932ba97f41f02b1ceb2 |
|
BLAKE2b-256 | f124bde66b155933f90088bdefcd5946fd6300a5751460acfe024f8a3c3901a2 |
Hashes for codebleu-0.3.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e80715e5069c723156b1747648b760bbe32157cf1dee6d795f1b8e724e9ae70 |
|
MD5 | 4ecd6c3bb97478d3855a1bdce84d126c |
|
BLAKE2b-256 | a4c91a2bc0b120144c5a83539c0634eb78e952e0d45662d66a10654de4244a91 |
Hashes for codebleu-0.3.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2c620b8c01b1851984a4c3e58c72eb3542ae98a6e1f8d98065b58a0750ef48c2 |
|
MD5 | deb5b83599bc2addac0ea94d934865ff |
|
BLAKE2b-256 | 268a5624705a0e7142763d29e21cbb9b54b5f6d9b6e32a8c56658aed20875551 |
Hashes for codebleu-0.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b5305365b3732724ff53567cf5c296b8e8eae965f44010524bacda9973774de6 |
|
MD5 | 670fce2dd6f344cb58afea6d4120415d |
|
BLAKE2b-256 | 44575ebf0b309118704eaef1931d47aa3a0ccad929c69c3368f0f471213a419e |
Hashes for codebleu-0.3.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 91d2defb605b905079a7bb23a36fcea64fa8fee96c51e42234c8913edd858531 |
|
MD5 | 2b7b820a0ecc3d9c2720b8b4ce1e7d37 |
|
BLAKE2b-256 | ce878c43d3997c7ce5abd2617a6d6af3906097debf5edc1cdf3da5ed8c77e009 |
Hashes for codebleu-0.3.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6978ec00c95b1131300e9d0fa2e038e66e2ea0267e247a265025bffe11c8272 |
|
MD5 | b08289946602de30c5e02861d0fe7a01 |
|
BLAKE2b-256 | ae8fd759bb2fc08e8c9e411a8345cd7d8792f001f2f6d089419a487ce5b090f2 |
Hashes for codebleu-0.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26b56ec6ee0f639f9c7b5eebecc8566e4a6487dec6c657bc3144fa9921e05852 |
|
MD5 | a7020f731de3d1a0f35aee0373ac9a1a |
|
BLAKE2b-256 | 329e895bb3cafcca1dd528c170430a97bfb6304580aba69da885e6fab89a5bb7 |
Hashes for codebleu-0.3.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c4ce67cdda7e06f0350a84c03266285e3a4f4818f263fe2ddfd0c3c40e7eb7ac |
|
MD5 | f0a731d5d3c7c8e3882847ebecac40ef |
|
BLAKE2b-256 | c35dfb91c3385d2498b56d92f5947f6e48deb74b6d0369a52a7883a42a155a1b |
Hashes for codebleu-0.3.0-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b8ecb32cfad1631634990b7e9ab1697bdf617da522756b18c8588bb0fbf70579 |
|
MD5 | f7dddfe9160fe5c65c7a02cb1d0ae377 |
|
BLAKE2b-256 | 87d541ec7c7cb115d2d094c5861b9b211f5ddce08298b8da0acedcd41860ca0a |