Skip to main content

CodeBLEU

Publish Test codecov PyPI version

This repository contains an unofficial CodeBLEU implementation that supports Linux, MacOS (incl. M-series) and Windows. It is available through PyPI and the evaluate library.

Available for: Python, C, C#, C++, Java, JavaScript, PHP, Go, Ruby, Rust, TypeScript, TSX, and Fortran.


The code is based on the original CodeXGLUE/CodeBLEU and updated version by XLCoST/CodeBLEU. It has been refactored, tested, built for macOS and Windows, and multiple improvements have been made to enhance usability.

Metric Description

An ideal evaluation metric should consider the grammatical correctness and the logic correctness. We propose weighted n-gram match and syntactic AST match to measure grammatical correctness, and introduce semantic data-flow match to calculate logic correctness. CodeBLEU
[from CodeXGLUE repo]

In a nutshell, CodeBLEU is a weighted combination of n-gram match (BLEU), weighted n-gram match (BLEU-weighted), AST match and data-flow match scores.

The metric has shown higher correlation with human evaluation than BLEU and accuracy metrics.

Installation

CodeBLEU supports Python 3.11 through 3.15 on Linux, macOS, and Windows. CI tests both ARM and Intel macOS runners. Tree-sitter 0.26.1 or later in the 0.26 series is required. Older Python versions are no longer supported.

The metrics is available as pip package and can be installed as indicated above:

pip install codebleu

or directly from git repo (require internet connection to download tree-sitter):

pip install git+https://github.com/k4black/codebleu.git

Also you have to install tree-sitter language you need (e.g. python, rust, etc):

pip install tree-sitter-python

Or you can install all languages:

pip install codebleu[all]

The all extra installs compatible grammar releases from PyPI. Individual grammars must match the versions declared in pyproject.toml. Pip builds packages from source if a compatible wheel is unavailable. A source build requires a C compiler.

Grammar updates can change AST shapes and metric scores. Record the CodeBLEU, Tree-sitter, and grammar versions with evaluation results. See metric definitions and score changes before comparing results with earlier releases.

Use lang="typescript" for TypeScript and lang="tsx" for TSX. Both grammars use the JavaScript data-flow extractor. Fortran support uses the free-form grammar and treats variable names without case distinctions. The extractors approximate dependencies. They do not implement complete compiler semantics.

Usage

from codebleu import calc_codebleu

prediction = "def add ( a , b ) :\n return a + b"
reference = "def sum ( first , second ) :\n return second + first"

result = calc_codebleu([reference], [prediction], lang="python", weights=(0.25, 0.25, 0.25, 0.25), tokenizer=None)
print(result)
# {
#   'codebleu': 0.7135,
#   'ngram_match_score': 0.1042,
#   'weighted_ngram_match_score': 0.7500,
#   'syntax_match_score': 1.0, 
#   'dataflow_match_score': 1.0
# }

where calc_codebleu takes the following arguments:

  • references (list[str] or list[list[str]]): reference code
  • predictions (list[str]) predicted code
  • lang (str): code language, see codebleu.AVAILABLE_LANGS for available languages (python, c_sharp, c, cpp, javascript, java, php, go, ruby, rust, typescript, tsx, and fortran)
  • weights (tuple[float,float,float,float]): weights of the ngram_match, weighted_ngram_match, syntax_match, and dataflow_match respectively, defaults to (0.25, 0.25, 0.25, 0.25)
  • tokenizer (callable): to split code string to tokens, defaults to s.split()

and outputs the dict[str, float] with following fields:

  • codebleu: the final CodeBLEU score
  • ngram_match_score: ngram_match score (BLEU)
  • weighted_ngram_match_score: weighted_ngram_match score (BLEU-weighted)
  • syntax_match_score: syntax_match score (AST match)
  • dataflow_match_score: dataflow_match score

Alternatively, you can use k4black/codebleu from HuggingFace Spaces (codebleu package required):

import evaluate
metric = evaluate.load("dvitel/codebleu")

prediction = "def add ( a , b ) :\n return a + b"
reference = "def sum ( first , second ) :\n return second + first"

result = metric.compute([reference], [prediction], lang="python", weights=(0.25, 0.25, 0.25, 0.25))

Feel free to check the HF Space with online example: k4black/codebleu

Contributing

Contributions are welcome!
For questions, suggestions, and non-security bugs, open a GitHub issue. Report vulnerabilities through private vulnerability reporting.

Make your own fork and clone it:

git clone https://github.com/k4black/codebleu

For development, create a virtual environment and install the language and test extras:

python -m venv .venv
# Activate the virtual environment for your shell.
python -m pip install -e ".[all,test]"

For testing just run pytest:

python -m pytest

To perform a style check, run:

python -m ruff check codebleu tests
python -m ruff format codebleu tests --check
python -m ty check codebleu

Release Please opens a PR that updates VERSION and CHANGELOG.md from Conventional Commits. Merge the release PR after CI passes. The automation creates the GitHub release and publishes its tagged source to PyPI. See release setup for the GitHub App and trusted publisher configuration.

License

This project is licensed under the terms of the MIT license.

Citation

Official CodeBLEU paper can be cited as follows:

@misc{ren2020codebleu,
      title={CodeBLEU: a Method for Automatic Evaluation of Code Synthesis}, 
      author={Shuo Ren and Daya Guo and Shuai Lu and Long Zhou and Shujie Liu and Duyu Tang and Neel Sundaresan and Ming Zhou and Ambrosio Blanco and Shuai Ma},
      year={2020},
      eprint={2009.10297},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}

Metadata

Release files for codebleu 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for codebleu 0.8.0
File Size Uploaded Provenance Actions
codebleu-0.8.0.tar.gz 179.1 kB With provenance Details for codebleu-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for codebleu 0.8.0
File Interpreter ABI Platform Actions
codebleu-0.8.0-py3-none-any.whl Python 3 none any Details for codebleu-0.8.0-py3-none-any.whl

Total release size: 209.5 kB

Release files / codebleu-0.8.0.tar.gz

Download URL codebleu-0.8.0.tar.gz
Size 179.1 kB
Tags Source
SHA-256 checksum
How to use checksums
238650a051458842bac08765c3d15ab38635a475154addd5258bf74f5d91f151
BLAKE2b-256 checksum
How to use checksums
2eacd4eb2d9e957c325e0bbfb714aaa84f8d0131d05cf8abae589d49b696ecba
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / codebleu-0.8.0-py3-none-any.whl

Download URL codebleu-0.8.0-py3-none-any.whl
Size 30.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
985b6b41e822c2346705c88308ecbdd99d585b46cb7b9ecc105638bc85c68dd2
BLAKE2b-256 checksum
How to use checksums
5a0e9ae3beb13a0b01870acd0cab68ada0dd5a6da02f0d39a7cd7becf0d4dc86
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for codebleu 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for codebleu-0.8.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page