odp
Project description
odp
- Announcement Post: https://github.com/open-data-products/odp/blob/main/docs/announcement.md
Open Data Products (ODP) is a toolkit that helps data practitioners and data leaders better understand the value of their data. It can help teams understand what data is in use and who to talk to in order to understand which data is driving business value and why. For example, the detect-unused command can help find tables or columns that are unused within a certain time range (e.g. 60 days).
pip install odp
odp detect-unused --dialect=snowflake --grain=table --since-days=60
Read 63 queries from SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
Read 116 information schema rows from ANALYTICS_PROD.INFORMATION_SCHEMA.COLUMNS
Most common tables (20):
('CATALOG', 'PUBLIC', 'AGG_DAILY'): 30
('CATALOG', 'PUBLIC', 'BRAND'): 27
...
Unused tables (27):
('CATALOG', 'PUBLIC', 'AD_CLICK')
('CATALOG', 'PUBLIC', 'AD_COMBINED')
('CATALOG', 'PUBLIC', 'AD_CONVERSION')
...
- Github repository: https://github.com/open-data-products/odp/
Dev Usage
poetry install
poetry run python -m odp detect-unused --schema-file=examples/snowflake/info-schema.csv --queries-file=examples/snowflake/query-history.csv
or, run with snowflake env:
cat <<EOF > .env
ODP_SNOWFLAKE_ACCOUNT=your_account
ODP_SNOWFLAKE_USERNAME=your_user
ODP_SNOWFLAKE_PASSWORD=your_password
ODP_SNOWFLAKE_DATABASE=your_database
ODP_SNOWFLAKE_WAREHOUSE=your_warehouse # optional
ODP_SNOWFLAKE_ROLE=your_role # optional
EOF
poetry run python -m odp detect-unused
or, run with bigquery env:
cat <<EOF > .env
ODP_GOOGLE_APPLICATION_CREDENTIALS=your_credential_file
ODP_GOOGLE_PROJECT=your_project
EOF
poetry run python -m odp detect-unused --dialect=bigquery
Maintainers
Made with :heart: by
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
odp-0.3.1.tar.gz
(9.3 kB
view details)
Built Distribution
odp-0.3.1-py3-none-any.whl
(10.3 kB
view details)
File details
Details for the file odp-0.3.1.tar.gz
.
File metadata
- Download URL: odp-0.3.1.tar.gz
- Upload date:
- Size: 9.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.3 CPython/3.11.5 Darwin/23.5.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 454143a9335c8d70ae011d0ac66f9cbe948dfab2379d8c552abc7da7991f7d37 |
|
MD5 | e88a4c12839138f9c7618c804243491c |
|
BLAKE2b-256 | 64da43ffee18e349d574e74e4b394c9d1d3f953f2f160de063a11d1a1bf7242d |
File details
Details for the file odp-0.3.1-py3-none-any.whl
.
File metadata
- Download URL: odp-0.3.1-py3-none-any.whl
- Upload date:
- Size: 10.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.3 CPython/3.11.5 Darwin/23.5.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ffd67adaea22add814fd99e9e662c437a9702179ad4f7808fb888afef78dba0 |
|
MD5 | a8242a18a4b56540139f3e291678e092 |
|
BLAKE2b-256 | 84c0a488965f801029bdd563f392fc28b2991bde98dd5eb85403798d39d0f764 |