Skip to main content

Python implementation of the R package synthpop

Project description

synthpop

Python implementation of the R package synthpop.

The R implementation of synthpop is a tool for producing synthetic versions of microdata containing confidential information so that they are safe to be released to users for exploratory analysis. The key objective of generating synthetic data is to replace sensitive original values with synthetic ones causing minimal distortion of the statistical information contained in the dataset. Variables, which can be categorical or continuous, are synthesised one-by-one using sequential modelling. Replacements are generated by drawing from conditional distributions fitted to the original data using parametric or classification and regression trees models.

This is a reimplementation in Python which allows synthetic data to be generated via the method .generate() after the algorithm had been fit to the original data via the method .fit(). The process can be largely automated, if default settings are used, or with methods defined by the user. Optional parameters can be used to influence the disclosure risk and the analytical quality of the synthetic data.

Development status and roadmap

This project is in Alpha status and the roadmap can be found here.

Installation from sources

git clone git@github.com:hazy/synthpop.git
cd synthpop
pip install -r requirements.txt
python setup.py install

Examples

Adult dataset

We will use the US adult census dataset, which is a freely available open dataset extracted from the US census bureau database. The dataset is initially designed for a binary classification problem and the task is to predict whether a person earns over $50,000 a year. The dataset is a mixture of discrete and continuous features, including age, working status (workclass), education, marital status, race, sex, relationship and hours worked per week.

In [1]: from datasets.adult import df

In [2]: df.head()
Out[2]:
   age          workclass  fnlwgt   education  educational-num       marital-status          occupation    relationship    race   gender  capital-gain  capital-loss  hours-per-week  native-country  income
0   39          State-gov   77516   Bachelors               13        Never-married        Adm-clerical   Not-in-family   White     Male          2174             0              40   United-States   <=50K
1   50   Self-emp-not-inc   83311   Bachelors               13   Married-civ-spouse     Exec-managerial         Husband   White     Male             0             0              13   United-States   <=50K
2   38            Private  215646     HS-grad                9             Divorced   Handlers-cleaners   Not-in-family   White     Male             0             0              40   United-States   <=50K
3   53            Private  234721        11th                7   Married-civ-spouse   Handlers-cleaners         Husband   Black     Male             0             0              40   United-States   <=50K
4   28            Private  338409   Bachelors               13   Married-civ-spouse      Prof-specialty            Wife   Black   Female             0             0              40            Cuba   <=50K

synthpop

  1. Use default parameters for the Adult dataset:
In [1]: from synthpop import Synthpop

In [2]: from datasets.adult import df, dtypes

In [3]: spop = Synthpop()

In [4]: spop.fit(df, dtypes)
train_age
train_workclass
train_fnlwgt
train_education
train_educational-num
train_marital-status
train_occupation
train_relationship
train_race
train_gender
train_capital-gain
train_capital-loss
train_hours-per-week
train_native-country
train_income

In [5]: synth_df = spop.generate(len(df))
generate_age
generate_workclass
generate_fnlwgt
generate_education
generate_educational-num
generate_marital-status
generate_occupation
generate_relationship
generate_race
generate_gender
generate_capital-gain
generate_capital-loss
generate_hours-per-week
generate_native-country
generate_income

In [6]: synth_df.head()
Out[6]:
   age   workclass  fnlwgt education  educational-num       marital-status      occupation    relationship    race   gender  capital-gain  capital-loss  hours-per-week  native-country  income
0   21           ?  213055      11th                7        Never-married               ?   Not-in-family   Other   Female             0             0              30   United-States   <=50K
1   23     Private  150683   HS-grad                9        Never-married    Adm-clerical   Not-in-family   White   Female             0             0              40   United-States   <=50K
2   61     Private  191417      10th                6              Widowed           Sales   Not-in-family   Black   Female             0             0              32   United-States   <=50K
3   50     Private  190762   HS-grad                9             Divorced           Sales   Not-in-family   White     Male             0             0              60   United-States   <=50K
4   42   Local-gov  255675   HS-grad                9   Married-civ-spouse   Other-service         Husband   Black     Male             0             0              40   United-States   <=50K

In [7]: spop.method
Out[7]:
age                sample
workclass            cart
fnlwgt               cart
education            cart
educational-num      cart
marital-status       cart
occupation           cart
relationship         cart
race                 cart
gender               cart
capital-gain         cart
capital-loss         cart
hours-per-week       cart
native-country       cart
income               cart
dtype: object

In [8]: spop.visit_sequence
Out[8]:
age                 0
workclass           1
fnlwgt              2
education           3
educational-num     4
marital-status      5
occupation          6
relationship        7
race                8
gender              9
capital-gain       10
capital-loss       11
hours-per-week     12
native-country     13
income             14
dtype: int64

In [9]: spop.predictor_matrix
Out[9]:
                 age  workclass  fnlwgt  education  educational-num  marital-status  occupation  relationship  race  gender  capital-gain  capital-loss  hours-per-week  native-country  income
age                0          0       0          0                0               0           0             0     0       0             0             0               0               0       0
workclass          1          0       0          0                0               0           0             0     0       0             0             0               0               0       0
fnlwgt             1          1       0          0                0               0           0             0     0       0             0             0               0               0       0
education          1          1       1          0                0               0           0             0     0       0             0             0               0               0       0
educational-num    1          1       1          1                0               0           0             0     0       0             0             0               0               0       0
marital-status     1          1       1          1                1               0           0             0     0       0             0             0               0               0       0
occupation         1          1       1          1                1               1           0             0     0       0             0             0               0               0       0
relationship       1          1       1          1                1               1           1             0     0       0             0             0               0               0       0
race               1          1       1          1                1               1           1             1     0       0             0             0               0               0       0
gender             1          1       1          1                1               1           1             1     1       0             0             0               0               0       0
capital-gain       1          1       1          1                1               1           1             1     1       1             0             0               0               0       0
capital-loss       1          1       1          1                1               1           1             1     1       1             1             0               0               0       0
hours-per-week     1          1       1          1                1               1           1             1     1       1             1             1               0               0       0
native-country     1          1       1          1                1               1           1             1     1       1             1             1               1               0       0
income             1          1       1          1                1               1           1             1     1       1             1             1               1               1       0
  1. Define the visit sequence for the Adult dataset:
In [1]: from synthpop import Synthpop

In [2]: from datasets.adult import df, dtypes

In [3]: spop = Synthpop(visit_sequence=[0, 1, 5, 3, 2])

In [4]: spop.fit(df, dtypes)
train_age
train_workclass
train_marital-status
train_education
train_fnlwgt

In [5]: synth_df = spop.generate(len(df))
generate_age
generate_workclass
generate_marital-status
generate_education
generate_fnlwgt

In [6]: synth_df.head()
Out[6]:
   age          workclass  fnlwgt      education       marital-status
0   57   Self-emp-not-inc  327901    Prof-school   Married-civ-spouse
1   24            Private   34568      Assoc-voc        Never-married
2   50            Private  256861        HS-grad   Married-civ-spouse
3   28            Private  186239   Some-college        Never-married
4   38            Private  216129      Bachelors             Divorced

In [7]: spop.method
Out[7]:
age                sample
workclass            cart
fnlwgt               cart
education            cart
educational-num      cart
marital-status       cart
occupation           cart
relationship         cart
race                 cart
gender               cart
capital-gain         cart
capital-loss         cart
hours-per-week       cart
native-country       cart
income               cart
dtype: object

In [8]: spop.visit_sequence
Out[8]:
age               0
workclass         1
fnlwgt            4
education         3
marital-status    2
dtype: int64

In [9]: spop.predictor_matrix
Out[9]:
                age  workclass  fnlwgt  education  marital-status
age               0          0       0          0               0
workclass         1          0       0          0               0
fnlwgt            1          1       0          1               1
education         1          1       0          0               1
marital-status    1          1       0          0               0

License

This project is being developed at Hazy Limited and is released under MIT license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

py_synthpop-0.1.1-py3-none-any.whl (19.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page