formeval 1.0.4

language form evaluation in python

Language Form Evaluation

This package includes efficient pure python implementation of the following metrics:

• BLEU

  • current implementation is a wrapper of nltk.translate.bleu_score
    • known issue:
      • drop in performance when evaluating multi-candidates vs multi-references

• ROUGE (in progress)

• METEOR (in progress)

• CIDEr/CIDEr-D

  • original implementation: https://github.com/vrama91/cider
    • with the same tokenizer
      • error: < 1 %
      • speed: + 81 %
    • with different tokenizers (FormEval use Regexp by default)
      • error: ~ 15 %
      • speed: + 332 %

• SPICE (in progress)

*All stats shown above are estimations

Dependencies

• python 3.6 +
• nltk 3.5+

Setup

pip install formeval

optional setup

python3 -c 'import nltk; nltk.download("wordnet")'

Example Data

• PASCAL VOC

Usage

See run_examples.py