Skip to main content

Generate fake data conforming to a Table Schema

Project description

Generate tabular fake data conforming to a Table Schema

Usage

Installation

$ pip3 install tsfaker

Simple usage

Generate 3 rows of fake data from a single table schema file.

$ tsfaker https://gitlab.com/healthdatahub/tsfaker/raw/master/tests/schemas/implemented_types.json  --nrows 3 --pretty
  boolean         string            number      integer        date              datetime  year yearmonth
0       1  haHoKysholbSI    9780230269.512  -7061309068  1914-10-03  1902-04-11T11:21:11Z  1939    196405
1       0      rLugGhNek    990894536.8945   2529879443  2026-09-08  2015-11-27T16:21:54Z  1932    192909
2       1         ipqVXm  -4371053960.8987   -529880373  1994-09-27  1937-01-12T18:40:15Z  2021    193303

Advanced usage

Show help message.

$ tsfaker --help
Usage: tsfaker [OPTIONS] [SCHEMA_DESCRIPTORS]...
...

Download examples schemas from project schema-snds.

$ git clone  https://gitlab.com/healthdatahub/schema-snds && cd schema-snds

Generate fake data for all schemas in a schemas folder using csv files in nomenclatures folder, and write them to fake_data folder.

$ mkdir fake_data
$ tsfaker schemas -o fake_data -r nomenclatures
2019-01-01 00:00:00 :: INFO :: Data generated from descriptor 'schemas/PMSI/PMSI MCO/T_MCOaa_nnE.json' will be written on 'fake_data/PMSI/PMSI MCO/T_MCOaa_nnE.csv'
2019-01-01 00:00:00 :: INFO :: Data generated from descriptor 'schemas/PMSI/PMSI MCO/T_MCOaa_nnFASTC.json' will be written on 'fake_data/PMSI/PMSI MCO/T_MCOaa_nnFASTC.csv'
2019-01-01 00:00:00 :: INFO :: Data generated from descriptor 'schemas/PMSI/PMSI SSR/T_SSRaa_nnE.json' will be written on 'fake_data/PMSI/PMSI SSR/T_SSRaa_nnE.csv'
...

Goals

We aim to generate fake data conforming to a schema.

We do not aim to generate realistic data with statistical information (see related work).

Implementation steps

  • Generate data conforming to types
  • Generate data conforming to formats and constraints, such as min/max, enum, missing values, unique, length, and regex
  • Generate multiple tables conforming to foreign key references, with optional tables’ data provided through csv

API

  • We want to provide both a Python API and a command line API

Development methodology

We will conform to Test Driven Development methodology, hence writing test before writing implementation.

We want generated data to be valid when using goodtables.

We could go by conforming to more and more content checks, which are included in table-schema specification.

Release notes

Since version 0.10

  • boolean type is implemented, default values for this type are 0 for False and 1 for True

Since version 0.11

  • yearmonth type does not follow ISO 8601 format ‘YYYY-MM’ and is now generated without a dash ‘YYYYMM’

Since version 0.12

  • It is possible to specify trueValues and falseValues for boolean type (according to TableSchema standard)
  • Only one item is accepted in trueValues and falseValues arrays
  • It is possible to specify a format for types : date and datetime

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for tsfaker, version 0.12
Filename, size File type Python version Upload date Hashes
Filename, size tsfaker-0.12-py3-none-any.whl (22.3 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size tsfaker-0.12.tar.gz (16.5 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page