Skip to main content

A Python Wrapper to calculate standard BLEU scores for NLP

Project description

bleu (Python Package)

Pypi Downloads Month_Downloads

A Python Wrapper for the standard BLEU evaluation for Natural Language Generation (NLG).

Installation

Requirement: Python 3

Option 1: Install pip package

pip install --upgrade bleu

Option 2: Build from source

pip install --upgrade git+git://github.com/zhijing-jin/bleu.git

How to Run

The most standard way to calculate BLEU is by Moses' script for detokenized BLEU. This package provides easy calls to it.

Function 1: Calculate the BLEU for lists

If you want to check only one hypothesis (a list of sentences):

>>> from bleu import list_bleu
>>> ref = ['it is a white cat .',
             'wow , this dog is huge .']
>>> ref1 = ['This cat is white .',
             'wow , this is a huge dog .']
>>> hyp = ['it is a white kitten .',
            'wowww , the dog is huge !']
>>> hyp1 = ["it 's a white kitten .",
             'wow , this dog is huge !']
>>> list_bleu([ref], hyp)
34.99
>>> list_bleu([ref, ref1], hyp1)
57.91

If you want to check multiple hypothesis (several lists of sentences):

>>> from bleu import multi_list_bleu
>>> multi_list_bleu([ref, ref1], [hyp, hyp1])
[34.99, 57.91]

detok=False: It is not advisable to use tokenized bleu (by multi-bleu.perl), but if you want to call it, just use detok=False:

>>> list_bleu([ref], hyp, detok=False)
39.76
# or if you want to test multiple hypotheses
>>> multi_list_bleu([ref, ref1], [hyp, hyp1], detok=False)
[39.76, 47.47]

verbose=True: If there are unexpected errors, you might want to check the intermediate steps by verbose=True.

Function 2: Calculate the BLEU for files

If you want to check only one hypothesis file:

# if you already have the following files
>>> from bleu import file_bleu
>>> hyp_file = 'data/hyp0.txt'
>>> ref_files = ['data/ref0.txt', 'data/ref1.txt']
>>> file_bleu(ref_files, hyp_file)
34.99

If you want to check multiple hypothesis files:

>>> from bleu import multi_file_bleu
>>> hyp_file1 = 'data/hyp1.txt'
>>> bleus = multi_file_bleu(ref_files, [hyp_file, hyp_file1])
[34.99, 57.91]

detok=True: Set it if you want to calculate the (not recommended) tokenized bleu.

verbose=True: Set it if you want to inspect how the bleu calculations are made:

>>> bleu = file_bleu(ref_files, hyp_file, verbose=True)
[Info] Valid Reference Files: ['data/ref0.txt', 'data/ref1.txt']
[Info] Valid Hypothesis Files: ['data/hyp0.txt']
[Info] #lines in each file: 2
[cmd] perl detokenizer.perl -l en < data/ref0.txt > data/ref0.detok.txt 2>/dev/null
[cmd] perl detokenizer.perl -l en < data/ref1.txt > data/ref1.detok.txt 2>/dev/null
[cmd] perl detokenizer.perl -l en < data/hyp0.txt > data/hyp0.detok.txt 2>/dev/null
[cmd] perl multi-bleu-detok.perl data/ref0.detok.txt data/ref1.detok.txt < data/hyp0.detok.txt
2-ref bleu for data/hyp0.detok.txt: 34.99
>>> bleu
34.99

Option 3: Detokenize files

>>> from bleu import detok_files
>>> detok_ref_files = detok_files(ref_files, tmp_dir='./data', file_prefix='ref_dtk', verbose=True)
[cmd] perl ./TMP_DIR/detokenizer.perl -l en < data/ref0.txt > data/ref_dtk0.txt 2>/dev/null
[cmd] perl ./TMP_DIR/detokenizer.perl -l en < data/ref1.txt > data/ref_dtk1.txt 2>/dev/null
>>> detok_ref_files
['data/ref_dtk0.txt', 'data/ref_dtk1.txt']

In Case of Unexpected Outputs

Check the python file bleu.py and adapt it.

Contact

If you have more questions, feel free to check out the common Q&A, or raise a new GitHub issue.

In case of really urgent needs, contact the author Zhijing Jin (Miss).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bleu-0.3.tar.gz (5.2 kB view details)

Uploaded Source

File details

Details for the file bleu-0.3.tar.gz.

File metadata

  • Download URL: bleu-0.3.tar.gz
  • Upload date:
  • Size: 5.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.4.2 requests/2.20.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.34.0 CPython/3.6.6

File hashes

Hashes for bleu-0.3.tar.gz
Algorithm Hash digest
SHA256 6edae73a588fe8b72678d3233da3be9e901a29d8313858347998910dc28baf80
MD5 b005e08282effc650705ae4eb0a2bf28
BLAKE2b-256 aedf4fd9bfe6dc240a1760f8e95ca41aa31382e328e31de45145f528dab5c7f8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page