A framework to benchmark the performance of synthetic data generators for non-temporal tabular data
An open source project from Data to AI Lab at MIT.
- License: MIT
- Development Status: Pre-Alpha
- Homepage: https://github.com/sdv-dev/SDGym
Synthetic Data Gym (SDGym) is a framework to benchmark the performance of synthetic data generators for tabular data. SDGym is a project of the Data to AI Laboratory at MIT.
What is a Synthetic Data Generator?
A Synthetic Data Generator is a Python function (or class method) that takes as input some data, which we call the real data, learns a model from it, and outputs new synthetic data that has similar mathematical properties as the real one.
Please refer to the synthesizers documentation for instructions about how to implement your own Synthetic Data Generator and integrate with SDGym. You can also read about how to use the ones included in SDGym and see the current leaderboard.
SDGym evaluates the performance of Synthetic Data Generators using datasets that are in three families:
- Simulated data generated using Gaussian Mixtures
- Simulated data generated using Bayesian Networks
- Real world datasets
Further details about how these datasets were generated can be found in the Modeling Tabular data using Conditional GAN paper and in the datasets documentation.
This is a summary of the current SDGym leaderboard, showing the number of datasets in which each Synthesizer obtained the best score.
The complete scores table can be found in the synthesizers document and it can also be downloaded as a CSV file form here: sdgym/leaderboard.csv
Detailed leaderboard results for all the releases are available in this Google Docs Spreadsheet.
Gaussian Mixture Simulated Data
Bayesian Networks Simulated Data
Real World Datasets
SDGym has been developed and tested on Python 3.6, 3.7 and 3.8
Also, although it is not strictly required, the usage of a virtualenv is highly recommended in order to avoid interfering with other software installed in the system where SDGym is run.
Install with pip
The easiest and recommended way to install SDGym is using pip:
pip install sdgym
This will pull and install the latest stable release from PyPi.
If you want to install it from source or contribute to the project please read the Contributing Guide for more details about how to do it.
Benchmarking your own synthesizer
All you need to do in order to use the SDGym Benchmark, is to import
sdgym and call its
run function passing it your synthesizer function and the settings that you want to use
for the evaluation.
For example, if we want to evaluate a simple synthesizer function in the
we can execute:
import numpy as np import sdgym def my_synthesizer_function(real_data, categorical_columns, ordinal_columns): """dummy synthesizer that just returns a permutation of the real data.""" return np.random.permutation(real_data) scores = sdgym.run(synthesizers=my_synthesizer_function, datasets=['adult'])
- You can learn how to create your own synthesizer function here.
- You can learn about different arguments for
The output of the
sdgym.run function will be a
pd.DataFrame containing the results obtained
by your synthesizer on each dataset, as well as the results obtained previously by the SDGym
adult/accuracy adult/f1 ... ring/test_likelihood IndependentSynthesizer 0.56530 0.134593 ... -1.958888 UniformSynthesizer 0.39695 0.273753 ... -2.519416 IdentitySynthesizer 0.82440 0.659250 ... -1.705487 ... ... ... ... ... my_synthesizer_function 0.64865 0.210103 ... -1.964966
Benchmarking the SDGym Synthesizers
If you want to run the SDGym benchmark on the SDGym Synthesizers you can directly pass the
corresponding class, or a list of classes, to the
For example, if you want to run the complete benchmark suite to evaluate all the existing synthesizers you can run (this will take a lot of time to run!):
from sdgym.synthesizers import ( CLBNSynthesizer, CTGANSynthesizer, IdentitySynthesizer, IndependentSynthesizer, MedganSynthesizer, PrivBNSynthesizer, TableganSynthesizer, TVAESynthesizer, UniformSynthesizer, VEEGANSynthesizer) all_synthesizers = [ CLBNSynthesizer, IdentitySynthesizer, IndependentSynthesizer, MedganSynthesizer, PrivBNSynthesizer, TableganSynthesizer, CTGANSynthesizer, TVAESynthesizer, UniformSynthesizer, VEEGANSynthesizer, ] scores = sdgym.run(synthesizers=all_synthesizers)
For further details about all the arguments and possibilities that the
benchmark function offers
please refer to the benchmark documentation
- Datasets used in SDGym are detailed here.
- How to write a synthesizer is detailed here.
- How to use benchmark function is detailed here.
- Detailed leaderboard results for all the releases are available here.
SDV, for Synthetic Data Vault, is the end-user library for synthesizing data in development under the HDI Project. SDV allows you to easily model and sample relational datasets using Copulas through a simple API. Other features include anonymization of Personal Identifiable Information (PII) and preserving relational integrity on sampled records.
CTGAN is the GAN based model for synthesizing tabular data presented in the Modeling Tabular data using Conditional GAN paper. It's also developed by the MIT's Data to AI Lab and is under active development.
TGAN is another GAN based model for synthesizing tabular data. It's also developed by the MIT's Data to AI Lab and is under active development.
v0.2.2 - 2020-10-17
This version adds a rework of the the benchmark function and a few new synthetsizers.
- New CLI with
- Parallel execution via Dask or Multiprocessing
- Download datasets without executing the benchmark
- Support for python from 3.6 to 3.8
v0.2.1 - 2020-05-12
New updated leaderboard and minor improvements.
- Add parameters for PrivBNSynthesizer - Issue #37 by @csala
v0.2.0 - 2020-04-10
New Becnhmark API and lots of improved documentation.
- The benchmark function now returns a complete leaderboard instead of only one score
- Class Synthesizers can be directly passed to the benchmark function
- One hot encoding errors in the Independent, VEEGAN and Medgan Synthesizers.
- Proper usage of the
evalmode during sampling.
- Fix improperly configured datasets.
v0.1.0 - 2019-08-07
First release to PyPi
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for sdgym-0.2.2-py2.py3-none-any.whl