useful package for pipelines testing and data mocking for new generation data warehouse
Project description
py-test-utility
Contains a collection of class and functions which aim to help developers implement and test data pipelines against the new generation data warehouses and storing system i.e. BigQuery and GCS
Installation
py-test-utility can be installed via pip
pip install py-test-utility
mockdata - class csv_mock(csv,schema)
Extract the equivalent json from csv with nested and repeated records structures
Args
- csv
- path and file name of the csv
- mandatory
- nested fields shall be separated by a dot "." (i.e. item.id, item.quantity)
order | item.id | item.quantity | delivery.address | delivey.postcode |
---|---|---|---|---|
A0001 | item1 | 5 | address1 | e13bp |
item2 | 1 | |||
item3 | 3 | |||
A0002 | item4 | 4 | address4 | e13bp |
item1 | 4 | |||
item3 | 2 |
- schema
- path and schema file name of the table schema
- required if the CSV contain nested and repeated records
- json format i.e.
[ { "mode": "NULLABLE", "name": "order", "type": "STRING" }, { "fields": [ { "mode": "NULLABLE", "name": "id", "type": "STRING" }, { "mode": "NULLABLE", "name": "quantity", "type": "STRING" } ], "mode": "REPEATED", "name": "item", "type": "RECORD" }, { "fields": [ { "mode": "NULLABLE", "name": "address", "type": "STRING" }, { "mode": "NULLABLE", "name": "postcode", "type": "STRING" } ], "mode": "NULLABLE", "name": "delivery", "type": "RECORD" } ]
Methods
- to_json()
- if successfuls return the json extracted from the csv
Usage
>>> from mockdata import mockdata as md >>> mockdata_csv = md.csv_mock( ... csv="mockdata/test/data/csv/repeated_records.csv", ... schema="mockdata/test/schema/repeated_records_schema.json") # initialise the object >>> mockdata_json = mockdata_csv.to_json() # return the equivalent json
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size py_test_utility-0.0.2-py3-none-any.whl (5.0 kB) | File type Wheel | Python version py3 | Upload date | Hashes View |
Filename, size py-test-utility-0.0.2.tar.gz (3.8 kB) | File type Source | Python version None | Upload date | Hashes View |
Close
Hashes for py_test_utility-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba654f64c3f279e39dd4c587975b72182bc34b6c49da57c03d96019366a5a924 |
|
MD5 | d448cd689d35248d0037d416330506bb |
|
BLAKE2-256 | fcfef94eb1c03c40794ea5a9df80e7c2f5d6c81e195ae0d282cab49ddd408879 |