mt-telescope 0.0.1rc1

A visual platform for contrastive evaluation of machine translation systems

MT-Telescope is a toolkit for comparative analysis of MT systems that provides a number of tools that add rigor and depth to MT evaluation. With this package we endeavour to make it easier for researchers and industry practitioners to compare MT systems by giving you easy access to:

1. SOTA MT evaluation metrics such as COMET (rei, et al 2020).
2. Statistical tests such as bootstrap resampling (Koehn, et al 2004).
3. Dynamic Filters to select parts of your testset with specific phenomena
4. Visual interface/plots to compare systems side-by-side segment-by-segment.

We highly recommend reading the following papers to learn more about how to perform better MT-Evaluation:

• Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers
• To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

Install:

Via pip:

pip install mt-telescope==0.0.1rc1

Note: This is a pre-release currently.

Locally:

Create a virtual environment and make sure you have poetry installed.

Finally run:

git clone https://github.com/Unbabel/MT-Telescope
cd MT-Telescope
poetry install

Scoring:

To get the system level scores for a particular MT simply run telescope score.

telescope score -s {path/to/sources} -h {path/to/translations} -r {path/to/references} -l {target_language} -m COMET -m chrF

Comparing two systems:

For comparison between two systems you can run telescope using:

1. The command line interface
2. A web browser

Command Line Interface (CLI):

For running system comparisons with CLI you should use the telescope compare command.

Usage: telescope compare [OPTIONS]

Options:
  -s, --source FILENAME           Source segments.  [required]
  -x, --system_x FILENAME         System X MT outputs.  [required]
  -y, --system_y FILENAME         System Y MT outputs.  [required]
  -r, --reference FILENAME        Reference segments.  [required]
  -l, --language TEXT             Language of the evaluated text.  [required]
  -m, --metric [COMET|sacreBLEU|chrF|ZeroEdit|BLEURT|BERTScore|TER|Prism|GLEU]
                                  MT metric to run.  [required]
  -f, --filter [named-entities|duplicates]
                                  MT metric to run.
  --seg_metric [COMET|ZeroEdit|BLEURT|BERTScore|Prism|GLEU]
                                  Segment-level metric to use for segment-
                                  level analysis.

  -o, --output_folder TEXT        Folder you wish to use to save plots.
  --bootstrap
  --num_splits INTEGER            Number of random partitions used in
                                  Bootstrap resampling.

  --sample_ratio FLOAT            Folder you wish to use to save plots.
  --help                          Show this message and exit.

Example 1: Running several metrics

Running BLEU, chrF BERTScore and COMET to compare two systems:

telescope compare \
  -s path/to/src/file.txt \
  -x path/to/system-x/file.txt \
  -y path/to/system-y \
  -r path/to/ref/file.txt \
  -l en \
  -m BLEU -m chrF -m BERTScore -m COMET

Example 2: Saving a comparison report

telescope compare \
  -s path/to/src/file.txt \
  -x path/to/system-x/file.txt \
  -y path/to/system-y \
  -r path/to/ref/file.txt \
  -l en \
  -m COMET \
  --output_folder FOLDER-PATH

Web Interface

To run a web interface simply run:

streamlit run app.py

Cite:

@inproceedings{rei-etal-2021-mt, author = {Rei, Ricardo and Stewart, Craig and Farinha, Ana C and Lavie, Alon}, title = "{MT}-{T}elescope: {A}n interactive platform for contrastive evaluation of {MT} systems", booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Demonstrations)", month = aug, year = "2021", address = "Online", publisher = "Association for Computational Linguistics", }