Quollio Data Profiler
Project description
quollio-data-profiler
Description
This system collects advanced metadata like table to table lineage or anomaly record and ingests them to QDC.
Prerequisite
Before you begin to use this, you need to do the following.
- Add your assets to QDC with metadata agent.
- Issue External API client id and client secret on QDC.
Install
Install with the following command.
pip install quollio-data-profiler
Usage
Here is an example of creating a view for snowflake lineage. Please enter any values for <>.
from quollio_data_profiler.repository.qdc import QDCExternalAPIClient
from quollio_data_profiler.repository.snowflake import SnowflakeConnectionConfig
from quollio_data_profiler.snowflake_lineage_profiler import execute
def view_build_only():
company_id = "<company id issued by quollio.>"
build_view_connection = SnowflakeConnectionConfig(
account_id="<Snowflake account id. Please use the same id of metadata agent.>",
account_role="<Role necessary for creating view in your account>",
account_user="<user name>",
account_password="<password>",
account_warehouse="<compute warehouse>",
)
qdc_client = QDCExternalAPIClient(
client_id="<client id issued on QDC.>",
client_secret="<client secret>",
base_url="<base endpoint>",
)
execute(
company_id=company_id,
sf_build_view_connections=build_view_connection,
qdc_client=qdc_client,
is_view_build_only=True,
)
if __name__ == "__main__":
view_build_only()
Please refer to the scripts in ./examples
for other usages.
Development
How to test
Unittest
- Run
make test
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for quollio_data_profiler-0.2.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14c9c9fa2b8113bb38fa586c95cb1f9b1b0c9f65e406164ba3e5cb8fc549f1e4 |
|
MD5 | 208c95b70214c44a528433653c26cf51 |
|
BLAKE2b-256 | b30c484e1851594a54cd3c6f9d21cd6713f044a4cb79ca4682dd204a8866d931 |
Close
Hashes for quollio_data_profiler-0.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dfbab9a45bff16fb5cf0b46c7c7ecb38d816824af2b2891aea48af76ebd78578 |
|
MD5 | e813fcab5696cf12bb6c2182477741c8 |
|
BLAKE2b-256 | fa718bae0c6eea7a904b9a30b85b28519d10c074a511d61c5cd08aa1ee6f2929 |