Skip to main content

Library for creating BQ tables with fake sensible data

Project description


Library for creating BQ tables with fake pii data.

The drive and use case to create this library, was when you need a lot of data to validate if your org complies with regulations like CCPA, HIPAA, GDPR.

PyPi License Issues

1. Environment setup

1.1. Get the code

git clone https://.../bq_fake_pii_table_creator.git
cd bq_fake_pii_table_creator

1.2. Auth credentials

1.2.1. Create a service account and grant it below roles

The Service Account authenticated must have administrator privileges for Cloud Storage and BigQuery.

1.2.2. Download a JSON key and save it as
  • <YOUR-CREDENTIALS_FILES_FOLDER>/bq_fake_pii_table_creator-credentials.json

Please notice this folder and file will be required in next steps.

1.3. Virtualenv

Using virtualenv is optional, but strongly recommended unless you use Docker or a PEX file.

1.3.1. Install Python 3.6+
1.3.2. Create and activate a virtualenv
pip install --upgrade virtualenv
python3 -m virtualenv --python python3 env
source ./env/bin/activate
1.3.3. Install the dependencies
pip install --editable .
1.3.4. Set environment variables

Replace below values according to your environment:

export GOOGLE_APPLICATION_CREDENTIALS=credentials_file_path

1.4. Docker

See instructions below.

2. Sample application entry point

2.1. Run

  • Virtualenv

Only the project-id argument is required.

python --project-id your_project --bq-dataset-name your_dataset --bq-table-name your_table --num-rows 5000 --num-cols 10 --obfuscate-col-names true

2.2. Or using Docker

docker build -t bq_fake_pii_table_creator .
docker run --rm --tty -v CREDENTIALS_FILES_FOLDER:/data \
bq_fake_pii_table_creator \
 --project-id your_project


0.0.1 (2020-04-30)

  • First release on PyPI.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bq-fake-pii-table-creator, version 0.0.4
Filename, size File type Python version Upload date Hashes
Filename, size bq_fake_pii_table_creator-0.0.4-py3-none-any.whl (10.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size bq-fake-pii-table-creator-0.0.4.tar.gz (7.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page