Library for creating BQ tables with fake sensible data
Project description
bq_fake_pii_table_creator
Library for creating BQ tables with fake pii data.
The drive and use case to create this library, was when you need a lot of data to validate if your org complies with regulations like CCPA, HIPAA, GDPR.
1. Environment setup
1.1. Get the code
git clone https://.../bq_fake_pii_table_creator.git
cd bq_fake_pii_table_creator
1.2. Auth credentials
1.2.1. Create a service account and grant it below roles
The Service Account authenticated must have administrator privileges for Cloud Storage and BigQuery.
1.2.2. Download a JSON key and save it as
<YOUR-CREDENTIALS_FILES_FOLDER>/bq_fake_pii_table_creator-credentials.json
Please notice this folder and file will be required in next steps.
1.3. Virtualenv
Using virtualenv is optional, but strongly recommended unless you use Docker or a PEX file.
1.3.1. Install Python 3.6+
1.3.2. Create and activate a virtualenv
pip install --upgrade virtualenv
python3 -m virtualenv --python python3 env
source ./env/bin/activate
1.3.3. Install the dependencies
pip install --editable .
1.3.4. Set environment variables
Replace below values according to your environment:
export GOOGLE_APPLICATION_CREDENTIALS=credentials_file_path
1.4. Docker
See instructions below.
2. Sample application entry point
2.1. Run main.py
- Virtualenv
Only the project-id argument is required.
python main.py --project-id your_project --bq-dataset-name your_dataset --bq-table-name your_table --num-rows 5000 --num-cols 10 --obfuscate-col-names true
2.2. Or using Docker
docker build -t bq_fake_pii_table_creator . docker run --rm --tty -v CREDENTIALS_FILES_FOLDER:/data \ bq_fake_pii_table_creator \ --project-id your_project
History
0.0.1 (2020-04-30)
- First release on PyPI.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size bq_fake_pii_table_creator-0.0.4-py3-none-any.whl (10.3 kB) | File type Wheel | Python version py3 | Upload date | Hashes View |
Filename, size bq-fake-pii-table-creator-0.0.4.tar.gz (7.5 kB) | File type Source | Python version None | Upload date | Hashes View |
Hashes for bq_fake_pii_table_creator-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea4033b38eb44050e6fae166c1dffa93e5859ef69bfc50414a2de1e75153c113 |
|
MD5 | 58d03106e29a45ca25724f8e90491660 |
|
BLAKE2-256 | d656d4b484c5538d4248af3a4bf7e78b18d64c787236ec60c834488fabbbbbec |
Hashes for bq-fake-pii-table-creator-0.0.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9174523fd375db7bc57b62fc9c3bb8c714111741f147e0217ed846a460f3f60b |
|
MD5 | 8a6902f1936ead68c7efedb8a4fc2cfd |
|
BLAKE2-256 | 57504fe99cad1259874208451885c00c1ffce57968b970d08380bb249a727f99 |