Synthetic Data Quality Assurance 🔎

license GitHub Release PyPI - Python Version

Documentation | Sample Reports | Technical White Paper

Assess the fidelity and novelty of synthetic samples with respect to original samples:

calculate a rich set of accuracy, similarity and distance metrics
visualize statistics for easy comparison to training and holdout samples
generate a standalone, easy-to-share, easy-to-read HTML summary report

...all with a few lines of Python code 💥.

https://github.com/user-attachments/assets/b27e270a-f19c-4059-b4f2-ed209c9a26b9

Installation

The latest release of mostlyai-qa can be installed via pip:

pip install -U mostlyai-qa

On Linux, one can explicitly install the CPU-only variant of torch together with mostlyai-qa:

pip install -U torch==2.13.0+cpu torchvision==0.28.0+cpu mostlyai-qa --extra-index-url https://download.pytorch.org/whl/cpu

Quick Start

import pandas as pd
import webbrowser
from mostlyai import qa

# initialize logging to stdout
qa.init_logging()

# fetch original + synthetic data
base_url = "https://github.com/mostly-ai/mostlyai-qa/raw/refs/heads/main/examples/quick-start"
syn = pd.read_csv(f"{base_url}/census2k-syn_mostly.csv.gz")
# syn = pd.read_csv(f'{base_url}/census2k-syn_flip30.csv.gz') # a 30% perturbation of trn
trn = pd.read_csv(f"{base_url}/census2k-trn.csv.gz")
hol = pd.read_csv(f"{base_url}/census2k-hol.csv.gz")

# calculate metrics
report_path, metrics = qa.report(
    syn_tgt_data=syn,
    trn_tgt_data=trn,
    hol_tgt_data=hol,
)

# pretty print metrics
print(metrics.model_dump_json(indent=4))

# open up HTML report in new browser window
webbrowser.open(f"file://{report_path.absolute()}")

Basic Usage

from mostlyai import qa

# initialize logging to stdout
qa.init_logging()

# analyze single-table data
report_path, metrics = qa.report(
    syn_tgt_data = synthetic_df,
    trn_tgt_data = training_df,
    hol_tgt_data = holdout_df,  # optional
)

# analyze sequential data
report_path, metrics = qa.report(
    syn_tgt_data = synthetic_df,
    trn_tgt_data = training_df,
    hol_tgt_data = holdout_df,  # optional
    tgt_context_key = "user_id",
)

# analyze sequential data with context
report_path, metrics = qa.report(
    syn_tgt_data = synthetic_df,
    trn_tgt_data = training_df,
    hol_tgt_data = holdout_df,  # optional
    syn_ctx_data = synthetic_context_df,
    trn_ctx_data = training_context_df,
    hol_ctx_data = holdout_context_df,  # optional
    ctx_primary_key = "id",
    tgt_context_key = "user_id",
)

Sample Reports

Baseball Players (Flat Data)
Baseball Seasons (Sequential Data)

Citation

Please consider citing our project if you find it useful:

@misc{mostlyai-qa,
      title={Benchmarking Synthetic Tabular Data: A Multi-Dimensional Evaluation Framework},
      author={Andrey Sidorenko and Michael Platzer and Mario Scriminaci and Paul Tiwald},
      year={2025},
      eprint={2504.01908},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2504.01908},
}

Metadata

Release files for mostlyai-qa 1.10.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mostlyai-qa 1.10.9
File	Size	Uploaded
mostlyai_qa-1.10.9.tar.gz	30.6 MB	Sep 23, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for mostlyai-qa 1.10.9
File	Interpreter	ABI	Platform	Reset
mostlyai_qa-1.10.9-py3-none-any.whl 30.7 MB Sep 23, 2026	Python 3	none	any	Details

Total release size: 61.3 MB

Release files / mostlyai_qa-1.10.9.tar.gz

Download URL	mostlyai_qa-1.10.9.tar.gz
Size	30.6 MB
Tags	Source
SHA-256 checksum How to use checksums	`e2d0f2dce544caf51bcf1fb464da0c364f97cf3fe197c6dc870d8abfc390f102`
BLAKE2b-256 checksum How to use checksums	`498bcafafea6c5517e5fbc168ceb97d9e32a75350b88fe3a3882fcf1bf789a71`
Upload date	Sep 23, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/mostly-ai/mostlyai-qa
Publishing commit: github.com/mostly-ai/mostlyai-qa/tree/cbcb41e996d00e4c8b877d933a1e9464877c7fde

Workflow

Publishing configuration: github.com/mostly-ai/mostlyai-qa/blob/cbcb41e996d00e4c8b877d933a1e9464877c7fde/.github/workflows/release-2-publish.yml
Publishing logs: github.com/mostly-ai/mostlyai-qa/actions/runs/35926975001/attempts/1

Artifact

Subject: mostlyai_qa-1.10.9.tar.gz
SHA-256 checksum: e2d0f2dce544caf51bcf1fb464da0c364f97cf3fe197c6dc870d8abfc390f102 Verifying attestations

Release files / mostlyai_qa-1.10.9-py3-none-any.whl

Download URL	mostlyai_qa-1.10.9-py3-none-any.whl
Size	30.7 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`ca147710c37b492f2fa5392970a52bda776f7ec2c32a938d3571066ff2e8b10a`
BLAKE2b-256 checksum How to use checksums	`7d7330fc2f38d3ac7524c7991f2c49e9db0802440f8a2fa0bb00a8af7d655ce4`
Upload date	Sep 23, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/mostly-ai/mostlyai-qa
Publishing commit: github.com/mostly-ai/mostlyai-qa/tree/cbcb41e996d00e4c8b877d933a1e9464877c7fde

Workflow

Publishing configuration: github.com/mostly-ai/mostlyai-qa/blob/cbcb41e996d00e4c8b877d933a1e9464877c7fde/.github/workflows/release-2-publish.yml
Publishing logs: github.com/mostly-ai/mostlyai-qa/actions/runs/35926975001/attempts/1

Artifact

Subject: mostlyai_qa-1.10.9-py3-none-any.whl
SHA-256 checksum: ca147710c37b492f2fa5392970a52bda776f7ec2c32a938d3571066ff2e8b10a Verifying attestations