CodeBLEU
This repository contains an unofficial CodeBLEU implementation that supports Linux, MacOS (incl. M-series) and Windows. It is available through PyPI and the evaluate library.
Available for: Python, C, C#, C++, Java, JavaScript, PHP, Go, Ruby, Rust, TypeScript, TSX, and Fortran.
The code is based on the original CodeXGLUE/CodeBLEU and updated version by XLCoST/CodeBLEU. It has been refactored, tested, built for macOS and Windows, and multiple improvements have been made to enhance usability.
Metric Description
An ideal evaluation metric should consider the grammatical correctness and the logic correctness. We propose weighted n-gram match and syntactic AST match to measure grammatical correctness, and introduce semantic data-flow match to calculate logic correctness.
[from CodeXGLUE repo]
In a nutshell, CodeBLEU is a weighted combination of n-gram match (BLEU), weighted n-gram match (BLEU-weighted), AST match and data-flow match scores.
The metric has shown higher correlation with human evaluation than BLEU and accuracy metrics.
Installation
CodeBLEU supports Python 3.11 through 3.15 on Linux, macOS, and Windows. CI tests both ARM and Intel macOS runners. Tree-sitter 0.26.1 or later in the 0.26 series is required. Older Python versions are no longer supported.
The metrics is available as pip package and can be installed as indicated above:
pip install codebleu
or directly from git repo (require internet connection to download tree-sitter):
pip install git+https://github.com/k4black/codebleu.git
Also you have to install tree-sitter language you need (e.g. python, rust, etc):
pip install tree-sitter-python
Or you can install all languages:
pip install codebleu[all]
The all extra installs compatible grammar releases from PyPI.
Individual grammars must match the versions declared in pyproject.toml.
Pip builds packages from source if a compatible wheel is unavailable.
A source build requires a C compiler.
Grammar updates can change AST shapes and metric scores. Record the CodeBLEU, Tree-sitter, and grammar versions with evaluation results. See metric definitions and score changes before comparing results with earlier releases.
Use lang="typescript" for TypeScript and lang="tsx" for TSX.
Both grammars use the JavaScript data-flow extractor.
Fortran support uses the free-form grammar and treats variable names without case distinctions.
The extractors approximate dependencies. They do not implement complete compiler semantics.
Usage
from codebleu import calc_codebleu
prediction = "def add ( a , b ) :\n return a + b"
reference = "def sum ( first , second ) :\n return second + first"
result = calc_codebleu([reference], [prediction], lang="python", weights=(0.25, 0.25, 0.25, 0.25), tokenizer=None)
print(result)
# {
# 'codebleu': 0.7135,
# 'ngram_match_score': 0.1042,
# 'weighted_ngram_match_score': 0.7500,
# 'syntax_match_score': 1.0,
# 'dataflow_match_score': 1.0
# }
where calc_codebleu takes the following arguments:
references(list[str]orlist[list[str]]): reference codepredictions(list[str]) predicted codelang(str): code language, seecodebleu.AVAILABLE_LANGSfor available languages (python, c_sharp, c, cpp, javascript, java, php, go, ruby, rust, typescript, tsx, and fortran)weights(tuple[float,float,float,float]): weights of thengram_match,weighted_ngram_match,syntax_match, anddataflow_matchrespectively, defaults to(0.25, 0.25, 0.25, 0.25)tokenizer(callable): to split code string to tokens, defaults tos.split()
and outputs the dict[str, float] with following fields:
codebleu: the finalCodeBLEUscorengram_match_score:ngram_matchscore (BLEU)weighted_ngram_match_score:weighted_ngram_matchscore (BLEU-weighted)syntax_match_score:syntax_matchscore (AST match)dataflow_match_score:dataflow_matchscore
Alternatively, you can use k4black/codebleu from HuggingFace Spaces (codebleu package required):
import evaluate
metric = evaluate.load("dvitel/codebleu")
prediction = "def add ( a , b ) :\n return a + b"
reference = "def sum ( first , second ) :\n return second + first"
result = metric.compute([reference], [prediction], lang="python", weights=(0.25, 0.25, 0.25, 0.25))
Feel free to check the HF Space with online example: k4black/codebleu
Contributing
Contributions are welcome!
For questions, suggestions, and non-security bugs, open a GitHub issue.
Report vulnerabilities through private vulnerability reporting.
Make your own fork and clone it:
git clone https://github.com/k4black/codebleu
For development, create a virtual environment and install the language and test extras:
python -m venv .venv
# Activate the virtual environment for your shell.
python -m pip install -e ".[all,test]"
For testing just run pytest:
python -m pytest
To perform a style check, run:
python -m ruff check codebleu tests
python -m ruff format codebleu tests --check
python -m ty check codebleu
Release Please opens a PR that updates VERSION and CHANGELOG.md from Conventional Commits.
Merge the release PR after CI passes.
The automation creates the GitHub release and publishes its tagged source to PyPI.
See release setup for the GitHub App and trusted publisher configuration.
License
This project is licensed under the terms of the MIT license.
Citation
Official CodeBLEU paper can be cited as follows:
@misc{ren2020codebleu,
title={CodeBLEU: a Method for Automatic Evaluation of Code Synthesis},
author={Shuo Ren and Daya Guo and Shuai Lu and Long Zhou and Shujie Liu and Duyu Tang and Neel Sundaresan and Ming Zhou and Ambrosio Blanco and Shuai Ma},
year={2020},
eprint={2009.10297},
archivePrefix={arXiv},
primaryClass={cs.SE}
}
Metadata
Release files for codebleu 0.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| codebleu-0.8.0.tar.gz | 179.1 kB | With provenance | Details for codebleu-0.8.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| codebleu-0.8.0-py3-none-any.whl | Python 3 | none | any | Details for codebleu-0.8.0-py3-none-any.whl |
Total release size: 209.5 kB
Release files / codebleu-0.8.0.tar.gz
| Download URL | codebleu-0.8.0.tar.gz |
|---|---|
| Size | 179.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
238650a051458842bac08765c3d15ab38635a475154addd5258bf74f5d91f151
|
|
BLAKE2b-256 checksum How to use checksums |
2eacd4eb2d9e957c325e0bbfb714aaa84f8d0131d05cf8abae589d49b696ecba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logRelease files / codebleu-0.8.0-py3-none-any.whl
| Download URL | codebleu-0.8.0-py3-none-any.whl |
|---|---|
| Size | 30.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
985b6b41e822c2346705c88308ecbdd99d585b46cb7b9ecc105638bc85c68dd2
|
|
BLAKE2b-256 checksum How to use checksums |
5a0e9ae3beb13a0b01870acd0cab68ada0dd5a6da02f0d39a7cd7becf0d4dc86
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logSecurity for codebleu 0.8.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for codebleu-0.8.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.