OpenDataDiscovery dbt tests metadata collecting

CLI tool helps run and ingest dbt test to platform.

It can be used as separated CLI tool or within ODD CLI package which provides some useful additional features for working with OpenDataDiscovery.

Supported adapters

Adapter	version
Snowflake	^1.6
Postgres	^1.6

Profiles inside the file looks different for each type of data source.

Snowflake host_settings value is created from field account. Field value should be <account_identifier> For example the URL for an account uses the following format: <account_identifier>.snowflakecomputing.com Example Snowflake account identifier hj1234.eu-central-1.

Supported tests types

Generic tests
Singular tests. Currently Singular tests are not supported.

Installation

pip install odd-dbt

To see all available commands

odd_dbt_test --help

Example

For each command that involves sending information to OpenDataDiscovery platform exists set of env variables:

ODD_PLATFORM_HOST - Where you platform is
ODD_PLATFORM_TOKEN - Token for ingesting data to platform (How to create token?)
DBT_DATA_SOURCE_ODDRN - Unique oddrn string describes dbt source, i.e '//dbt/host/localhost'

It is recommended to add them as ENV variables or provide as flags to each command

export ODD_PLATFORM_HOST=http://localhost:8080
export ODD_PLATFORM_TOKEN=token***
export DBT_DATA_SOURCE_ODDRN=//dbt/host/localhost

Commands

create-datasource - helps to register dbt as data source at OpenDataDiscovery platform. Used later for ingesting metadata. Despite in the logs you can see something like: export DBT_DATA_SOURCE_ODDRN=//dbt/host/http://localhost:8080 it doesn't mean that script have exported it for you in terminal, so don't forget to do this command manually.

odd_dbt_test create-datasource --name=my_local_dbt --dbt-host=localhost

ingest-test - Read results_run file under the target folder to parse and ingest metadata.

odd_dbt_test ingest-test --profile=my_profile

If you are not in the directory that is a targeted dbt_project you should specify absolute paths for parameters: --project-dir and --profiles-dir, like this:

odd_dbt_test ingest-test --project-dir=absolute_path_for_dbt_project --profiles-dir=absolute_path_for_dbt_profiles  --profile=my_profile

ingest-lineage - Builds and ingest a lineage for tests into platform. Executing style is simular to ingest-test command:

odd_dbt_test ingest-lineage- --project-dir=absolute_path_for_dbt_project --profiles-dir=absolute_path_for_dbt_profiles  --profile=my_profile

test - Proxy command to dbt test, then reads results_run file under the target folder to parse and ingest metadata.

odd_dbt_test test --profile=my_profile

Run commands programmatically

You could run that scrip to read, parse and ingest test results to the platform.

# ingest_test_result.py

from odd_dbt import config
from odd_dbt.domain.cli_args import CliArgs
from odd_dbt.libs.dbt import get_context
from odd_dbt.libs.odd import create_dbt_generator_from_oddrn
from odd_dbt.service.odd import ingest_entities
from odd_dbt.mapper.test_results import DbtTestMapper
from odd_dbt.mapper.lineage import DbtLineageMapper

cfg = config.Config()  # All fields can be set manually or read from ENV variables
client = config.create_odd_client(host=cfg.odd_platform_host, token=cfg.odd_platform_token)
generator = create_dbt_generator_from_oddrn(oddrn=cfg.dbt_data_source_oddrn)

cli_args = CliArgs.default()
context = get_context(cli_args=cli_args)

# Ingest lineage
data_entities = DbtLineageMapper(context=context, generator=generator).map()
ingest_entities(data_entities, client)

# Or ingest test results
data_entities = DbtTestMapper(context=context, generator=generator).map()
ingest_entities(data_entities, client)

Metadata

Release files for odd-dbt 0.2.13

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for odd-dbt 0.2.13
File	Size	Uploaded	Provenance
odd_dbt-0.2.13.tar.gz	18.8 kB	Jun 21, 2024	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for odd-dbt 0.2.13
File	Interpreter	ABI	Platform	Reset
odd_dbt-0.2.13-py3-none-any.whl 24.4 kB Jun 21, 2024 Without provenance	Python 3	none	any	Details

Total release size: 43.2 kB

Release files / odd_dbt-0.2.13.tar.gz

Download URL	odd_dbt-0.2.13.tar.gz
Size	18.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`c5dbfb29f5fc07dbee0f61173889f019d70a1b8c236150430f505232616c5323`
BLAKE2b-256 checksum How to use checksums	`9b97829b3ff67ddff81a6b25c0c01f67ab41d4b0a278b1dea72af59efb4c0a4f`
Upload date	Jun 21, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.3.1 CPython/3.9.1 Linux/6.5.0-1022-azure`

Release files / odd_dbt-0.2.13-py3-none-any.whl

Download URL	odd_dbt-0.2.13-py3-none-any.whl
Size	24.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`5aeb9ce9645600f5fffedb6993fdff7fa72993743f172210bb0d9a7de9cccc3f`
BLAKE2b-256 checksum How to use checksums	`988bb0261341b5dd00141032491b0fe2dff3176aaa0bb010dc2ffeb72a938fea`
Upload date	Jun 21, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.3.1 CPython/3.9.1 Linux/6.5.0-1022-azure`

Security for odd-dbt 0.2.13

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.