goodcrap 0.1.2

goodcrap generates controlled random data.

goodcrap

goodcrap is a python package that generates tables, databases or csv files and fill them with random, or fake, data.

Motivation

This software enables data engineers to replicate the database schemas at their organisations, and then generate fake data that resemble a random sample of the actual data in their organisation.

While public datasets, such as those hosted at Google or Kaggle, is a common starting point for people interested in learning data analytics and machine learning, many of these datasets require extensive data cleaning so that they can be usable in analytics pipelines. This makes the use of these datasets difficult for AI learners and practitioners.

Public datasets are also utilized by data engineers who are interested in testing their ETL/ELT pipelines. Those folks are particularly interested in data quantity, more than quality. Most public datasets are limited in quantity, which make them not so useful for testing pipelines or for benchmarking query execution times.

Nowadays, generating random data is increasingly a requirement for data teams. It is a better alternative to using public datasets that require cleaning.

goodcrap was developed to enable:

• AI learners to generate their own custom datasets
• AI practitioners to benchmark the scalability of their models and methods
• data engineers to test and benchmark their ETL/ELT data pipelines
• data engineers to benchmark query execution times against custom-made huge datasets

The data generated by goodcrap is, after all, crap. But it's good crap because:

• data values are configured based on json configuration files
• data can be generated to fill tables, databases, data warehouses and data lakes
• data values can be made totally random, or fulfill a certain distribution
• goodcrap server can generate time series data

Installation

You can install goodcrap using the pip command as follows:

pip install goodcrap

Basic usage

The simplest use-case scenario is generating a csv file with random data. goodcrap ships with a number of template tables that you can use. For example, let's generate 10,000 records in the customers table, using the random seed 3:

goodcrap --size 10000 --seed 3 --template_table customers --to_csv

The file customers.csv will be generated.

goodcrap populate databases with crap, in addition to filling csv files. You can set goodcrap to connect to your database via a database configuration file, the name of which is passed to goodcrap via the command line argument --database_config. This is a json file that looks like this (for a MySQL database):

{
    "db_type": "mysql",
    "host":"localhost",
    "port":"3306",
    "user":"root",
    "passwd":"",
    "database":"goodcrap"
}

Here is an example command to create and fill the customers table in the database:

goodcrap --size 1000 --seed 3 --database_config mysql_config --template_table customers --to_csv

where mysql_config is the name of the configuration json file (mysql_config.json).

For every table mytable you want to fill with random values, you must provide either:

• a file mytable.crap_labels.json: this file tells goodcrap what sort of random values to generate for each record
• a sample database table or csv file with matching structure and with some values: goodcrap will learn how to generate new random values based on the sample values

Template data structures

goodcrap has a number of template tables and databases that you can use. They are in the templates/ directory.

The crap_labels.json settings file

For every table you want to generate, you have to provide the crap_labels.json file. If you are using the python library, then you can pass the crap_label dictionary instead - as is explained below.

The dictionary in crap_labels.json tells goodcrap how to fill each column in the table with random values. You can either use any of the faker providers there, or you can use the ones in crappers.

How random data is made: faker and crappers

in progress

Python library

The class GoodCrap is your goodcrap interface. You instantiate it with the key settings, and then generate the data by using the member functions write_csv(), get_dataframe() or run().

• write_csv(): writes a csv file
• get_dataframe(): returns a pandas DataFrame object populated with the random data
• run(): that's the more generic function that can generate tables and databases and populate them

And example usage for the goodcrap library is as follows:

gc = GoodCrap(size=10000,seed=123)
craplabels = {
    "customer_number": "ssn",
    "first_name": "first_name",
    "last_name": "last_name",
    "phone": "phone_number",
    "address_line": "street_address",
    "city": "city",
    "state": "state",
    "postalcode": "postalcode",
    "country": "current_country",
    "date_of_birth": "date",
    "credit_limit": {
        "type": "random_int",
        "min": 0,
        "max": 1000,
        "multiplier": 10
    },
    "income": {
        "type": "random_int",
        "min": 0,
        "max": 10000,
        "multiplier": 10
    }
}
df = gc.get_dataframe('customers',craplabels)

How data for a foreign key is generated

goodcrap will detect whether a column in a table is related to another table, and will fill that column with random selections of the related column. To demonstrate, run this command:

goodcrap --size 1000 --seed 3 --database_config examples\mysql_config --template_database customers_orders

This command will use the database settings in examples\mysql_config.json to generate the template database customers_orders and fill the tables with 1000 rows each. There are two tables here: customers and orders, and they are related: orders has a column customer_number that is tied to customers via the foreign key customers.customer_number. Therefore, that column is filled with random selections from customers.customer_number.

goodcrap with Mage

goodcrap can be used in Mage as a Data Loader. You can generate as much data as you want from multiple random goodcrap Data Loaders into your pipelines for testing, such as for testing the convergence of data into a data warehouse. You can also schedule the generation of data from goodcrap sources to simulate time series data traffic. A typical test-case scenario here is to run an SQL query at the data destination while data is being continuously loaded.

Here is an example goodcrap source in Mage:

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test

from goodcrap import GoodCrap

@data_loader
def load_data(*args, **kwargs):
    gc = GoodCrap(size=10000,seed=123)
    craplabels = {
        "customer_number": "ssn",
        "first_name": "first_name",
        "last_name": "last_name",
        "phone": "phone_number",
        "address_line": "street_address",
        "city": "city",
        "state": "state",
        "postalcode": "postalcode",
        "country": "current_country",
        "date_of_birth": "date",
        "credit_limit": {
            "type": "random_int",
            "min": 0,
            "max": 1000,
            "multiplier": 10
        },
        "income": {
            "type": "random_int",
            "min": 0,
            "max": 10000,
            "multiplier": 10
        }
    }
    return gc.get_dataframe('customers',craplabels)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'

Guessing the crap_labels.json settings

in progress

Learning the values from a data sample

in progress

Contrinuting to goodcrap

That would be much appreciated. Check here.

License

goodcrap is licensed under the GPL3 license.