Skip to main content

odp

Project description

odp

Open Data Products (ODP) is a toolkit that helps data practitioners and data leaders better understand the value of their data. It can help teams understand what data is in use and who to talk to in order to understand which data is driving business value and why. For example, the detect-unused command can help find tables or columns that are unused within a certain time range (e.g. 60 days).

pip install odp
odp detect-unused --dialect=snowflake --grain=table --since-days=60
Read 63 queries from SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
Read 116 information schema rows from ANALYTICS_PROD.INFORMATION_SCHEMA.COLUMNS
Most common tables (20):
('CATALOG', 'PUBLIC', 'AGG_DAILY'): 30
('CATALOG', 'PUBLIC', 'BRAND'): 27
...
Unused tables (27):
('CATALOG', 'PUBLIC', 'AD_CLICK')
('CATALOG', 'PUBLIC', 'AD_COMBINED')
('CATALOG', 'PUBLIC', 'AD_CONVERSION')
...

Dev Usage

poetry install

poetry run python -m odp detect-unused --schema-file=examples/snowflake/info-schema.csv --queries-file=examples/snowflake/query-history.csv

or, run with snowflake env:

cat <<EOF > .env
ODP_SNOWFLAKE_ACCOUNT=your_account
ODP_SNOWFLAKE_USERNAME=your_user
ODP_SNOWFLAKE_PASSWORD=your_password
ODP_SNOWFLAKE_DATABASE=your_database
ODP_SNOWFLAKE_WAREHOUSE=your_warehouse # optional
ODP_SNOWFLAKE_ROLE=your_role           # optional
EOF

poetry run python -m odp detect-unused

or, run with bigquery env:

cat <<EOF > .env
ODP_GOOGLE_APPLICATION_CREDENTIALS=your_credential_file
ODP_GOOGLE_PROJECT=your_project
EOF

poetry run python -m odp detect-unused --dialect=bigquery

Maintainers

Made with :heart: by

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

odp-0.3.1.tar.gz (9.3 kB view details)

Uploaded Source

Built Distribution

odp-0.3.1-py3-none-any.whl (10.3 kB view details)

Uploaded Python 3

File details

Details for the file odp-0.3.1.tar.gz.

File metadata

  • Download URL: odp-0.3.1.tar.gz
  • Upload date:
  • Size: 9.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.11.5 Darwin/23.5.0

File hashes

Hashes for odp-0.3.1.tar.gz
Algorithm Hash digest
SHA256 454143a9335c8d70ae011d0ac66f9cbe948dfab2379d8c552abc7da7991f7d37
MD5 e88a4c12839138f9c7618c804243491c
BLAKE2b-256 64da43ffee18e349d574e74e4b394c9d1d3f953f2f160de063a11d1a1bf7242d

See more details on using hashes here.

File details

Details for the file odp-0.3.1-py3-none-any.whl.

File metadata

  • Download URL: odp-0.3.1-py3-none-any.whl
  • Upload date:
  • Size: 10.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.11.5 Darwin/23.5.0

File hashes

Hashes for odp-0.3.1-py3-none-any.whl
Algorithm Hash digest
SHA256 3ffd67adaea22add814fd99e9e662c437a9702179ad4f7808fb888afef78dba0
MD5 a8242a18a4b56540139f3e291678e092
BLAKE2b-256 84c0a488965f801029bdd563f392fc28b2991bde98dd5eb85403798d39d0f764

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page