Skip to main content

Helper for extract query from dag file and validate query

Project description

kueri

A query toolkit

Description

There are two module in kueri package:

  1. Query extractor from DAG file : Used for extract query from each task in a dag file
  2. Query validator : Validate query using dry run api bigquery. This module use standard sql dialect

How to run unit test

Need to install berfore run unit test

  • pytest (pip install pytest)
  • pytest-cov (pip install pytest-cov)
  • rootpath (pip install rootpath)
  • mock
  • sqlparse
  • rootpath
  • apache-airflow[gcp]==1.10.5
  • requests

Command to run unit test

  • pytest (output only pass or failed testing)
  • pytest --cov-report term --cov=kueri tests/ (Give percentage coverage of unit testing)
  • pytest --cov-report term-missing --cov=kueri tests/ (give line of code function that not covered in unit test)

Installation

pip3 install kueri

Note
Before you install kueri, please make sure that you have installed all module that's your dag need

Usage

Query validator

kueri cmd='query_validator' query='{query need to validate}' project_id='{project id of your query}'

example:

kueri cmd='query_validator' query='select * from `mapan-data-prd.mapan_data_prod_dv.hub_arisan_chairmen`' project_id='mapan-data-prd'

Query extractor from DAG file

kueri cmd="dag_bq_extractor" path_to_folder_file='{path to folder folder file DAG}' file_name='{name of file}'

example:

kueri cmd="dag_bq_extractor" path_to_folder_file='/Users/mapan_1004/Documents/workspace/list_of_dag/' file_name='test_retrieve_bigquery_param_bql_dag.py'

Note
Please use ' (single quote) instead of " (double quotes) in parameter.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kueri-0.5.3.tar.gz (5.9 kB view details)

Uploaded Source

File details

Details for the file kueri-0.5.3.tar.gz.

File metadata

  • Download URL: kueri-0.5.3.tar.gz
  • Upload date:
  • Size: 5.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.9

File hashes

Hashes for kueri-0.5.3.tar.gz
Algorithm Hash digest
SHA256 ddd80a67ca9d561cb3503a2ba27077fa2e9dc01b024ddcf2ce5cccf51c7a5e49
MD5 507ad38b2bc8ca19801f9055ca7c7fb6
BLAKE2b-256 dc0e931ea32c32e5d74ed03e9711737fc376b8db485a08a7627387c2a7b494ea

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page