Skip to main content

useful package for pipelines testing and data mocking for new generation data warehouse

Project description

py-test-utility

Contains a collection of class and functions which aim to help developers implement and test data pipelines against the new generation data warehouses and storing system i.e. BigQuery and GCS

Installation

py-test-utility can be installed via pip

pip install py-test-utility

mockdata - class csv_mock(csv,schema)

Extract the equivalent json from csv with nested and repeated records structures

Args

  • csv
    • path and file name of the csv
    • mandatory
    • nested fields shall be separated by a dot "." (i.e. item.id, item.quantity)
order item.id item.quantity delivery.address delivey.postcode
A0001 item1 5 address1 e13bp
item2 1
item3 3
A0002 item4 4 address4 e13bp
item1 4
item3 2
  • schema
    • path and schema file name of the table schema
    • required if the CSV contain nested and repeated records
    • json format i.e.
[  
    {
      "mode": "NULLABLE", 
      "name": "order", 
      "type": "STRING"
    },  
    {
      "fields": [
        {
          "mode": "NULLABLE", 
          "name": "id", 
          "type": "STRING"
        },
        {
          "mode": "NULLABLE", 
          "name": "quantity", 
          "type": "STRING"
        }
      ], 
      "mode": "REPEATED", 
      "name": "item", 
      "type": "RECORD"
    }, 
    {
      "fields": [
        {
          "mode": "NULLABLE", 
          "name": "address", 
          "type": "STRING"
        }, 
        {
          "mode": "NULLABLE", 
          "name": "postcode", 
          "type": "STRING"
        }
      ], 
      "mode": "NULLABLE", 
      "name": "delivery", 
      "type": "RECORD"
    }
  ]

Methods

  • to_json()
    • if successfuls return the json extracted from the csv

Usage

>>> from mockdata import mockdata as md
>>> mockdata_csv = md.csv_mock(
...     csv="mockdata/test/data/csv/repeated_records.csv", 
...     schema="mockdata/test/schema/repeated_records_schema.json") # initialise the object
>>> mockdata_json = mockdata_csv.to_json() # return the equivalent json

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

py-test-utility-0.0.2.tar.gz (3.8 kB view hashes)

Uploaded source

Built Distribution

py_test_utility-0.0.2-py3-none-any.whl (5.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page