formeval 1.0.5

language form evaluation in python

Language Form Evaluation

This package includes efficient (almost) pure python implementation of the following metrics:

• BLEU
  • reference implementation: nltk.translate.bleu_score
    • error: < 1%
    • speed: + 189%
• ROUGE (in progress)
• METEOR (in progress)
• CIDEr/CIDEr-D
  • reference implementation: https://github.com/vrama91/cider
    • with the same tokenizer
      • error: < 1 %
      • speed: + 81 %
    • with different tokenizers (FormEval use Regexp by default)
      • error: ~ 15 %
      • speed: + 332 %
• SPICE
  • placeholder wrapper of reference implementation: https://github.com/tylin/coco-caption/tree/master/pycocoevalcap/spice
    • TODO: python scene graph parser

*All stats shown above are estimations

Dependencies

• python 3.6 +
• nltk 3.5+

Setup

pip install formeval

optional setups:

• spice dependencies

• wordnet lemmatizer

  python3 -c 'from formeval.setup import setup_everything; setup_everything()'

Example Data

• PASCAL VOC

Usage

See run_examples.py