synthex 0.4.2

Generate high-quality, large-scale synthetic datasets 📊🧪

Learn more · Documentation · Demo

🚀 Generate any structured dataset from scratch • 👤 Anonymize your dataset and be GDPR-compliant

🔥 Highlights

• Want to train a AI model, but don't have a training dataset? Describe it to Synthex and it will generate it for you.
• Need to anonymize your dataset to be GDPR-compliant? Synthex can replicate your dataset and remove Personally Identifiable Information (PII).
• Need to test your software, but don't yet have customer data? Synthex will generate a dataset for you.

⏩ Demo

Don't have time to go through this README? We got you covered. Just click on the button below for an interactive demo.

Introduction

Synthex is a Python library for generating large-scale, high-quality synthetic datasets. It is helpful in cases in which real data is too limited, unbalanced, completely absent or contains Personal Identifiable Information (PII).

Features & Use Cases

Key Features

• Generate any dataset from scratch: you don't need any data at all; Synthex will generate any structured dataset, even if it has no sample dataset to copy from.
• Natural language description: describe your requirements and any constraint the dataset may have in natural language.
• Specify fields and types: Synthex allows you to specify your dataset schema, including field names and types.

Use Cases:

• AI Model Training: lacking training data is no longer an issue; generate any training dataset for AI models with Synthex.
• Dataset anonymization: need a dataset without Personally Identifiable Information to be GDPR-compliant? Synthex is anonymous by default.
• Software testing: we all know testing your new software without customer data is not easy. Synthex will generate a testing dataset for you, so you can populate your DB.

🚀 Quickstart Guide

Installation

Install the library with

pip install synthex

Basic Usage

To create a new dataset, use Synthex.jobs.generate_data(). For this method's full documentation, see the Tanaos Docs.

from synthex import Synthex

client = Synthex()

client.jobs.generate_data(
    schema_definition = {
        "surface": {"type": "float"},
        "number_of_rooms": {"type": "integer"},
        "construction_year": {"type": "integer"},
        "city": {"type": "string"},
        "market_price": {"type": "float"}
    },
    examples = [
        {
            "surface": 104.00,
            "number_of_rooms": 3,
            "construction_year": 1985,
            "city": "Nashville",
            "market_price": 218000.00
        },
        {
            "surface": 98.00,
            "number_of_rooms": 2,
            "construction_year": 1999,
            "city": "Springfield",
            "market_price": 177000.00
        },
        {
            "surface": 52.00,
            "number_of_rooms": 1,
            "construction_year": 2014,
            "city": "Denver",
            "market_price": 230000.00
        }
    ],
    requirements = [
        "The 'market price' field should be realistic and should depend on the characteristics of the property.",
        "The 'city' field should specify cities in the USA, and the USA only"
    ],
    output_path = "output_data/output.csv",
    number_of_samples = 100,
    output_type = "csv"
)

where the parameters are as follows:

• schema_definition: A dict which specifies the output dataset's schema. It must have the following format:

  {
      "<name_of_column_1>": {"type": "<datatype_of_column_1>"},
      "<name_of_column_2>": {"type": "<datatype_of_column_2>"},
      ...
      "<name_of_column_n>": {"type": "<datatype_of_column_n>"}
  }
  

  the possible values of "type" are "string", "integer" and "float".

• examples: A List[dict], which specifies a few (3 to 5 are enough) sample datapoints that will help the data generation model understand what the output data should look like. They must have the same schema as the one specified in the schema_definition parameter, or an exception will be raised.

• requirements: a List[str], where each string specifies a requirement or constraint for the job. It can be an empty list if no specific requirements are present.

• output_path: a str which specifies the path where the output dataset will be generated. It does not need to contain a file name, as this will be added automatically if one is not provided. If output_path does contain a file name, its extension must be consistent with the output_type parameter. If this is the case, the provided output_path is used in its entirety. Otherwise, the provided extension is replaced with one that is consistent with output_type. For example:

• number_of_samples: an int which specifies the number of datapoints that the model should generate. Keep in mind that the maximum number of datapoints you can generate with a single job depends on whether you are on a free or paid plan. For example:

• output_type: a str which specifies the format of the output dataset. Only "csv" (meaning a .csv file will be generated) is supported at this time, but we will soon add more options.

🔗 Sample Datasets & Demos

• Demo - Create your first Synthetic Dataset: an interactive tutorial on how to use Synthex to create your first synthetic dataset (Live Demo)
• Online Store Chatbot Guardrail Training Dataset: a synthetic dataset, created with Synthex, used for training a Guardrail Model for an online store chatbot (See Dataset on HuggingFace).

🔑 Plans

Free plan: each user enjoys 1500 datapoints per month and 500 datapoints per job for free.

Pay-as-you-go: for additional usage beyond the free tier:

1. create an account on our platform
2. add credits to it
3. create an Api Key and pass it to Synthex at initialization:

   from synthex import Synthex
   
   client = Synthex(
       api_key="<your-api-key>"
   )
   

   The pay-as-you-go pricing is 1$ per 100 datapoints. Once you finish your credits, if you have not exceeded the monthly limit, you will be automatically switched to the free plan.

📚 Documentation & Support

• Full documentation: https://docs.tanaos.com/synthex
• Contact: [e-mail]