useful package for pipelines testing and data mocking for new generation data warehouse
Project description
py-test-utility
Contains a collection of class and functions which aim to help developers implement and test data pipelines against the new generation data warehouses and storing system i.e. BigQuery and GCS
Installation
py-test-utility can be installed via pip
pip install py-test-utility
mockdata - class csv_mock(csv,schema)
Extract the equivalent json from csv with nested and repeated records structures
Args
- csv
- path and file name of the csv
- mandatory
- nested fields shall be separated by a dot "." (i.e. item.id, item.quantity)
order | item.id | item.quantity | delivery.address | delivey.postcode |
---|---|---|---|---|
A0001 | item1 | 5 | address1 | e13bp |
item2 | 1 | |||
item3 | 3 | |||
A0002 | item4 | 4 | address4 | e13bp |
item1 | 4 | |||
item3 | 2 |
- schema
- path and schema file name of the table schema
- required if the CSV contain nested and repeated records
- json format i.e.
[
{
"mode": "NULLABLE",
"name": "order",
"type": "STRING"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "quantity",
"type": "STRING"
}
],
"mode": "REPEATED",
"name": "item",
"type": "RECORD"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "address",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "postcode",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "delivery",
"type": "RECORD"
}
]
Methods
- to_json()
- if successfuls return the json extracted from the csv
Usage
>>> from mockdata import mockdata as md
>>> mockdata_csv = md.csv_mock(
... csv="mockdata/test/data/csv/repeated_records.csv",
... schema="mockdata/test/schema/repeated_records_schema.json") # initialise the object
>>> mockdata_json = mockdata_csv.to_json() # return the equivalent json
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
py-test-utility-0.0.2.tar.gz
(3.8 kB
view hashes)
Built Distribution
Close
Hashes for py_test_utility-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba654f64c3f279e39dd4c587975b72182bc34b6c49da57c03d96019366a5a924 |
|
MD5 | d448cd689d35248d0037d416330506bb |
|
BLAKE2b-256 | fcfef94eb1c03c40794ea5a9df80e7c2f5d6c81e195ae0d282cab49ddd408879 |