Evaluation of Generative AI Models
Project description
GENAI EVALUATION
GenAI Evaluation is a library which contains methods to evaluate differences in Real & Synthetic Data.
Functions
- multivariate_ecdf: Computes joint or multivariate ECDF in contrast to the univariate capabilities provided by packages like statsmodels
- ks_statistic: Calculates the KS Statistic for two multivariate ECDFs
Authors
- Dr. Vincent Granville - Research
- Rajiv Iyer - Development/Maintenance
Installation
The package can be installed with
pip install genai_evaluation
Tests
The test can be run by cloning the repo and running:
pytest tests
In case of any issues running the tests, please run them after installing the package locally:
pip install -e .
Usage
Start by importing the class
from genai_evaluation import multivariate_ecdf, ks_statistic
Assuming we have two pandas dataframes (Real & Synthetic) and only numerical columns, we pass them to the multivariate_ecdf function which returns the computed multivariate ECDFs of both.
query_str, ecdf_real, ecdf_synth = multivariate_ecdf(real_data, synthetic_data, n_nodes = 1000, verbose = True)
We then calculate the multivariate KS Distance between the ECDFs
ks_stat = ks_statistic(ecdf_real, ecdf_synth)
Motivation
The motivation for this package comes from Dr. Vincent Granville's paper Generative AI Technology Break-through: Spectacular Performance of New Synthesizer
If you have any tips or suggestions, please contact us on email.
History
0.1.0 (2023-09-11)
- First release on PyPI.
0.1.1 (2023-09-11)
Corrected
- Function name from compute_ecdf to multivariate_ecdf
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for genai_evaluation-0.1.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1321e9c63e4ac7b760962e8bc993306ec2dac747946ce3f16754f199a1a0eae0 |
|
MD5 | 40a6b4d4b0e3e99afa3ed5b9c73c580b |
|
BLAKE2b-256 | 567c9373130262499eeec454ddde0e2d5a03b074c1eb8bcd09fb80d2d81aa493 |