useful package for pipelines testing and data mocking for new generation data warehouse
Project description
py-test-utility
Contains a collection of class and functions which aim to help developers implement and test data pipelines against the new generation data warehouses and storing system i.e. BigQuery and GCS
Installation
py-test-utility can be installed via pip
pip install py-test-utility
mockdata - class csv_mock(csv,schema)
Extract the equivalent json from csv with nested and repeated records structures
Args
- csv
- path and file name of the csv
- mandatory
- nested fields shall be separated by a dot "." (i.e. item.id, item.quantity)
order | item.id | item.quantity | delivery.address | delivey.postcode |
---|---|---|---|---|
A0001 | item1 | 5 | address1 | e13bp |
item2 | 1 | |||
item3 | 3 | |||
A0002 | item4 | 4 | address4 | e13bp |
item1 | 4 | |||
item3 | 2 |
- schema
- path and schema file name of the table schema
- required if the CSV contain nested and repeated records
- json format i.e.
[
{
"mode": "NULLABLE",
"name": "order",
"type": "STRING"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "quantity",
"type": "STRING"
}
],
"mode": "REPEATED",
"name": "item",
"type": "RECORD"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "address",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "postcode",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "delivery",
"type": "RECORD"
}
]
Methods
- to_json()
- if successfuls return the json extracted from the csv
Usage
>>> from mockdata import mockdata as md
>>> mockdata_csv = md.csv_mock(
... csv="mockdata/test/data/csv/repeated_records.csv",
... schema="mockdata/test/schema/repeated_records_schema.json") # initialise the object
>>> mockdata_json = mockdata_csv.to_json() # return the equivalent json
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
py-test-utility-0.0.1.tar.gz
(3.7 kB
view hashes)
Built Distribution
Close
Hashes for py_test_utility-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 924321fde012526ae8fc9d77caedf5fb316431f8f501377bb7f26207d42a7710 |
|
MD5 | 4df218712087c5a2a83575055a855be3 |
|
BLAKE2b-256 | 510d9b95adc19c0bb977f402d8771e419c874fd7db593674373485eb4556aec2 |