formeval 1.0.2

language form evaluation in python

Language Form Evaluation

This package includes efficient pure python implementation of the following metrics:

• BLEU (in progress)

• ROUGE (in progress)

• CIDEr/CIDEr-D

  • compared to original implementation: https://github.com/vrama91/cider
    • With the same tokenizer
      • error: < 1 %
      • speed: + 181 %
    • With different tokenizer (formeval cider use Regex from nltk)
      • error: ~15 %
      • speed: + 432 %

• SPICE (in progress)

All stats represented above are rough estimation

Dependencies

• python 3.6 +
• nltk 3.5+

Setup

pip install formeval

optional setup

python3 -c 'import nltk; nltk.download("wordnet")'

Example Data

• PASCAL VOC

Comparison

CIDEr/CIDEr-D (https://github.com/vrama91/cider):

• difference < 1%
• efficiency

Usage

See run_examples.py