Tools and Techniques for Consistency Benchmarking
Project description
ConsistencyBench
Setup
First install the package from PyPI.
pip install consistencybench
Additionally, if you'd like to use the NER metric (consistencybench.metrics.AgreementNER), run the following first.
python -m spacy download en_core_web_sm
To start generation and scoring
Set the arguments from run_eval.py
python run_eval.py --openai_api_key <OPENAI_API_KEY>
Example Output file: consistencybench/result_gpt-3.5-turbo_paraphrasing.csv
Example Jupyter Notebook: example.ipynb
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
consistencybench-0.1.tar.gz
(14.9 kB
view details)
Built Distribution
File details
Details for the file consistencybench-0.1.tar.gz.
File metadata
- Download URL: consistencybench-0.1.tar.gz
- Upload date:
- Size: 14.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.2 CPython/3.11.5 Linux/6.6.10-76060610-generic
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 37568e8b86094843db13a533f1dbd1af93bb23e908200fe4158468fa52567c66 |
|
| MD5 | e94f66053919899e6fbfc60c44a36b98 |
|
| BLAKE2b-256 | 1c7b859495a50ec9fad1c8a58e302cf5769d8c515310187dd382f36dd8983ab1 |
File details
Details for the file consistencybench-0.1-py3-none-any.whl.
File metadata
- Download URL: consistencybench-0.1-py3-none-any.whl
- Upload date:
- Size: 18.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.2 CPython/3.11.5 Linux/6.6.10-76060610-generic
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | ca0f7c54d3fb797e806ae89022021a3297e99757d29f3e57547f4056ab1b8f84 |
|
| MD5 | 064b29fac4801b9efc76112f21212aba |
|
| BLAKE2b-256 | 3c8d9e68bf9e634386bf73d66a94b3c325f056791645f3695026c5ac4c521478 |
