SyntheRela - Synthetic Relational Data Generation Benchmark

About SyntheRela

SyntheRela is a comprehensive benchmark designed to evaluate and compare synthetic relational database generation methods. It provides a standardized framework for assessing both the fidelity and utility of synthetic data across multiple real-world databases. The benchmark includes novel evaluation metrics, particularly for relational data, and supports various open-source and commercial synthetic data generation methods.

SyntheRela is highly extensible, allowing users to benchmark on their own custom datasets and implement new evaluation metrics to suit specific use cases.

Our research on SyntheRela is presented in the paper "SyntheRela: A Benchmark For Synthetic Relational Database Generation" at the ICLR 2025 Workshop "Will Synthetic Data Finally Solve the Data Access Problem?", available on OpenReview.

We maintain a public leaderboard on Hugging Face where you can compare the performance of different synthetic data generation methods.

Installation

To install only the benchmark package, run the following command:

pip install syntherela

Replicating the paper's results

For detailed instructions on how to replicate the paper's results, please refer to docs/REPLICATING_RESULTS.md.

Adding a new metric

The documentation for adding a new metric can be found in docs/ADDING_A_METRIC.md.

Synthetic Data Methods

Open Source Methods

• SDV: The Synthetic Data Vault
• RCTGAN: Row Conditional-TGAN for Generating Synthetic Relational Databases
• REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers
• ClavaDDPM: Multi-relational Data Synthesis with Cluster-guided Diffusion Models
• IRG: Generating Synthetic Relational Databases using GANs
• RGCLD: Relational Data Generation with Graph Neural Networks and Latent Diffusion Models
• Generating Realistic Synthetic Relational Data through Graph Variational Autoencoders*
• Generative Modeling of Complex Data*
• BayesM2M & NeuralM2M: Synthetic Data Generation of Many-to-Many Datasets via Random Graph Generation*

* Denotes the method does not have a public implementation available.

Commercial Providers

A list of commercial synthetic relational data providers is available in docs/SYNTHETIC_DATA_TOOLS.md.

Conflicts of Interest

The authors declare no conflict of interest and are not associated with any of the evaluated commercial synthetic data providers.

Citation

If you use SyntheRela in your work, please cite our paper:

@inproceedings{
    iclrsyntheticdata2025syntherela,
    title={SyntheRela: A Benchmark For Synthetic Relational Database Generation},
    author={Martin Jurkovic and Valter Hudovernik and Erik {\v{S}}trumbelj},
    booktitle={Will Synthetic Data Finally Solve the Data Access Problem?},
    year={2025},
    url={https://openreview.net/forum?id=ZfQofWYn6n}
}

License

This project is licensed under the MIT License. See the LICENSE file for details.