ConsistencyBench 0.1.3

Tools and Techniques for Consistency Benchmarking

ConsistencyBench

Setup

First install the package from PyPI.

pip install consistencybench

Additionally, if you'd like to use the NER metric (consistencybench.metrics.AgreementNER), run the following first.

python -m spacy download en_core_web_sm

To start generation and scoring

Set the arguments from run_eval.py

python run_eval.py --openai_api_key <OPENAI_API_KEY>

Example Output file: consistencybench/result_gpt-3.5-turbo_paraphrasing.csv

Example Jupyter Notebook: example.ipynb