sdgym 0.7.0

Benchmark tabular synthetic data generators using a variety of datasets

This repository is part of The Synthetic Data Vault Project, a project from DataCebo.

Overview

The Synthetic Data Gym (SDGym) is a benchmarking framework for modeling and generating synthetic data. Measure performance and memory usage across different synthetic data modeling techniques – classical statistics, deep learning and more!

The SDGym library integrates with the Synthetic Data Vault ecosystem. You can use any of its synthesizers, datasets or metrics for benchmarking. You can also customize the process to include your own work.

• Datasets: Select any of the publicly available datasets from the SDV project, or input your own data.
• Synthesizers: Choose from any of the SDV synthesizers and baselines. Or write your own custom machine learning model.
• Evaluation: In addition to performance and memory usage, you can also measure synthetic data quality and privacy through a variety of metrics.

Install

Install SDGym using pip or conda. We recommend using a virtual environment to avoid conflicts with other software on your device.

pip install sdgym

conda install -c pytorch -c conda-forge sdgym

For more information about using SDGym, visit the SDGym Documentation.

Usage

Let's benchmark synthetic data generation for single tables. First, let's define which modeling techniques we want to use. Let's choose a few synthesizers from the SDV library and a few others to use as baselines.

# these synthesizers come from the SDV library
# each one uses different modeling techniques
sdv_synthesizers = ['GaussianCopulaSynthesizer', 'CTGANSynthesizer']

# these basic synthesizers are available in SDGym
# as baselines
baseline_synthesizers = ['UniformSynthesizer']

Now, we can benchmark the different techniques:

import sdgym

sdgym.benchmark_single_table(
    synthesizers=(sdv_synthesizers + baseline_synthesizers)
)

The result is a detailed performance, memory and quality evaluation across the synthesizers on a variety of publicly available datasets.

Supplying a custom synthesizer

Benchmark your own synthetic data generation techniques. Define your synthesizer by specifying the training logic (using machine learning) and the sampling logic.

def my_training_logic(data, metadata):
    # create an object to represent your synthesizer
    # train it using the data
    return synthesizer

def my_sampling_logic(trained_synthesizer, num_rows):
    # use the trained synthesizer to create
    # num_rows of synthetic data
    return synthetic_data

Learn more in the Custom Synthesizers Guide.

Customizing your datasets

The SDGym library includes many publicly available datasets that you can include right away. List these using the get_available_datasets feature.

sdgym.get_available_datasets()

dataset_name   size_MB     num_tables
KRK_v1         0.072128    1
adult          3.907448    1
alarm          4.520128    1
asia           1.280128    1
...

You can also include any custom, private datasets that are stored on your computer on an Amazon S3 bucket.

my_datasets_folder = 's3://my-datasets-bucket'

For more information, see the docs for Customized Datasets.

What's next?

Visit the SDGym Documentation to learn more!

---

The Synthetic Data Vault Project was first created at MIT's Data to AI Lab in 2016. After 4 years of research and traction with enterprise, we created DataCebo in 2020 with the goal of growing the project. Today, DataCebo is the proud developer of SDV, the largest ecosystem for synthetic data generation & evaluation. It is home to multiple libraries that support synthetic data, including:

• 🔄 Data discovery & transformation. Reverse the transforms to reproduce realistic data.
• 🧠 Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular, multi table and time series data.
• 📊 Measuring quality and privacy of synthetic data, and comparing different synthetic data generation models.

Get started using the SDV package -- a fully integrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries for specific needs.

History

v0.7.0 - 2023-06-13

This release adds support for SDV 1.0 and PyTorch 2.0!

New Features

• Add functions to top level import - Issue #229 by @fealho
• Cleanup SDGym to the new SDV 1.0 metadata and synthesizers - Issue #212 by @fealho

Bugs Fixed

• limit_dataset_size causes sdgym to crash - Issue #231 by @fealho
• benchmark_single_table crashes with metadata dict - Issue #232 by @fealho
• Passing None as synthesizers runs all of them - Issue #233 by @fealho
• timeout parameter causes sdgym to crash - Issue #234 by @pvk-developer
• SDGym is not working with latest torch - Issue #210 by @amontanez24
• Fix sdgym --help - Issue #206 by @katxiao

Internal

• Increase code style lint - Issue #123 by @fealho
• Remove code support for synthesizers that are not strings/classes - PR #236 by @fealho
• Code Refactoring - Issue #215 by @fealho

Maintenance

• Remove pomegranate - Issue #230 by @amontanez24

v0.6.0 - 2023-02-01

This release introduces methods for benchmarking single table data and creating custom synthesizers, which can be based on existing SDGym-defined synthesizers or on user-defined functions. This release also adds support for Python 3.10 and drops support for Python 3.6.

New Features

• Benchmarking progress bar should update on one line - Issue #204 by @katxiao
• Support local additional datasets folder with zip files - Issue #186 by @katxiao
• Enforce that each synthesizer is unique in benchmark_single_table - Issue #190 by @katxiao
• Simplify the file names inside the detailed_results_folder - Issue #191 by @katxiao
• Use SDMetrics silent report generation - Issue #179 by @katxiao
• Remove arguments in get_available_datasets - Issue #197 by @katxiao
• Accept metadata.json as valid metadata file - Issue #194 by @katxiao
• Check if file or folder exists before writing benchmarking results - Issue #196 by @katxiao
• Rename benchmarking argument "evaluate_quality" to "compute_quality_score" - Issue #195 by @katxiao
• Add option to disable sdmetrics in benchmarking - Issue #182 by @katxiao
• Prefix remote bucket with 's3' - Issue #183 by @katxiao
• Benchmarking error handling - Issue #177 by @katxiao
• Allow users to specify custom synthesizers' display names - Issue #174 by @katxiao
• Update benchmarking results columns - Issue #172 by @katxiao
• Allow custom datasets - Issue #166 by @katxiao
• Use new datasets s3 bucket - Issue #161 by @katxiao
• Create benchmark_single_table method - Issue #151 by @katxiao
• Update summary metrics - Issue #134 by @katxiao
• Benchmark individual methods - Issue #159 by @katxiao
• Add method to create a sdv variant synthesizer - Issue #152 by @katxiao
• Add method to generate a multi table synthesizer - Issue #149 by @katxiao
• Add method to create single table synthesizers - Issue #148 by @katxiao
• Updating existing synthesizers to new API - Issue #154 by @katxiao

Bug Fixes

• Pip encounters dependency issues with ipython - Issue #187 by @katxiao
• IndependentSynthesizer is printing out ConvergeWarning too many times - Issue #192 by @katxiao
• Size values in benchmarking results seems inaccurate - Issue #184 by @katxiao
• Import error in the example for benchmarking the synthesizers - Issue #139 by @katxiao
• Updates and bugfixes - Issue #132 by @csala

Maintenance

• Update README - Issue #203 by @katxiao
• Support Python Versions >=3.7 and <3.11 - Issue #170 by @katxiao
• SDGym Package Maintenance Updates documentation - Issue #163 by @katxiao
• Remove YData - Issue #168 by @katxiao
• Update to newest SDV - Issue #157 by @katxiao
• Update slack invite link. - Issue #144 by @pvk-developer
• updating workflows to work with windows - Issue #136 by @amontanez24
• Update conda dependencies - Issue #130 by @katxiao

v0.5.0 - 2021-12-13

This release adds support for Python 3.9, and updates dependencies to accept the latest versions when possible.

Issues closed

• Add support for Python 3.9 - Issue #127 by @katxiao
• Add pip check worflow - Issue #124 by @pvk-developer
• Fix meta.yaml dependencies - PR #119 by @fealho
• Upgrade dependency ranges - Issue #118 by @katxiao

v0.4.1 - 2021-08-20

This release fixed a bug where passing a json file as configuration for a multi-table synthesizer crashed the model. It also adds a number of fixes and enhancements, including: (1) a function and CLI command to list the available synthesizer names, (2) a curate set of dependencies and making Gretel into an optional dependency, (3) updating Gretel to use temp directories, (4) using nvidia-smi to get the number of gpus and (5) multiple dockerfile updates to improve functionality.

Issues closed

• Bug when using JSON configuration for multiple multi-table evaluation - Issue #115 by @pvk-developer
• Use nvidia-smi to get number of gpus - PR #113 by @katxiao
• List synthesizer names - Issue #82 by @fealho
• Use nvidia base for dockerfile - PR #108 by @katxiao
• Add Makefile target to install gretel and ydata - PR #107 by @katxiao
• Curate dependencies and make Gretel optional - PR #106 by @csala
• Update gretel checkpoints to use temp directory - PR #105 by @katxiao
• Initialize variable before reference - PR #104 by @katxiao

v0.4.0 - 2021-06-17

This release adds new synthesizers for Gretel and ydata, and creates a Docker image for SDGym. It also includes enhancements to the accepted SDGym arguments, adds a summary command to aggregate metrics, and adds the normalized score to the benchmark results.

New Features

• Add normalized score to benchmark results - Issue #102 by @katxiao
• Add max rows and max columns args - Issue #96 by @katxiao
• Automatically detect number of workers - Issue #97 by @katxiao
• Add summary function and command - Issue #92 by @amontanez24
• Allow jobs list/JSON to be passed - Issue #93 by @fealho
• Add ydata to sdgym - Issue #90 by @fealho
• Add dockerfile for sdgym - Issue #88 by @katxiao
• Add Gretel to SDGym synthesizer - Issue #87 by @amontanez24

v0.3.1 - 2021-05-20

This release adds new features to store results and cache contents into an S3 bucket as well as a script to collect results from a cache dir and compile a single results CSV file.

Issues closed

• Collect cached results from s3 bucket - Issue #85 by @katxiao
• Store cache contents into an S3 bucket - Issue #81 by @katxiao
• Store SDGym results into an S3 bucket - Issue #80 by @katxiao
• Add a way to collect cached results - Issue #79 by @katxiao
• Allow reading datasets from private s3 bucket - Issue #74 by @katxiao
• Typos in the sdgym.run function docstring documentation - Issue #69 by @sbrugman

v0.3.0 - 2021-01-27

Major rework of the SDGym functionality to support a collection of new features:

• Add relational and timeseries model benchmarking.
• Use SDMetrics for model scoring.
• Update datasets format to match SDV metadata based storage format.
• Centralize default datasets collection in the sdv-datasets S3 bucket.
• Add options to download and use datasets from different S3 buckets.
• Rename synthesizers to baselines and adapt to the new metadata format.
• Add model execution and metric computation time logging.
• Add optional synthetic data and error traceback caching.

v0.2.2 - 2020-10-17

This version adds a rework of the benchmark function and a few new synthesizers.

New Features

• New CLI with run, make-leaderboard and make-summary commands
• Parallel execution via Dask or Multiprocessing
• Download datasets without executing the benchmark
• Support for python from 3.6 to 3.8

New Synthesizers

• sdv.tabular.CTGAN
• sdv.tabular.CopulaGAN
• sdv.tabular.GaussianCopulaOneHot
• sdv.tabular.GaussianCopulaCategorical
• sdv.tabular.GaussianCopulaCategoricalFuzzy

v0.2.1 - 2020-05-12

New updated leaderboard and minor improvements.

New Features

• Add parameters for PrivBNSynthesizer - Issue #37 by @csala

v0.2.0 - 2020-04-10

New Becnhmark API and lots of improved documentation.

New Features

• The benchmark function now returns a complete leaderboard instead of only one score
• Class Synthesizers can be directly passed to the benchmark function

Bug Fixes

• One hot encoding errors in the Independent, VEEGAN and Medgan Synthesizers.
• Proper usage of the eval mode during sampling.
• Fix improperly configured datasets.

v0.1.0 - 2019-08-07

First release to PyPi