dbt extractor

demo app

This repository contains a tool that processes the most common jinja value templates in dbt model files. The tool depends on tree-sitter and the tree-sitter-jinja2 library.

Strategy

The current strategy is for this processor to be 100% certain when it can accurately extract values from a given model file. Anything less than 100% certainty returns an exception so that the model can be rendered with python Jinja instead.

There are two cases we want to avoid because they would risk correctness to user's projects:

Confidently extracting values that would not be extracted by python jinja (false positives)
Confidently extracting a set of values that are missing values that python jinja would have extracted. (misses)

If we instead error when we could have confidently extracted values, there is no correctness risk to the user. Only an opportunity to expand the rules to encompass this class of cases as well.

Even though jinja in dbt is not a typed language, the type checker statically determines whether or not the current implementation can confidently extract values without relying on python jinja rendering, which is when these errors would otherwise surface. This type checker will become more permissive over time as this tool expands to include more dbt and jinja features.

Architecture

This architecture is optimized for value extraction and for future flexibility. This architecture is expected to change, and is coded in fp-style stages to make those changes easier for the future.

This processor is composed of several stages:

parser
type checker
extractor

Additionally, the following tools utilize the above processor:

browser-based demo of dbt extraction as you type

The tree-sitter parser is located in the tree-sitter-jinja2 library. The rust bindings are used to traverse the concrete syntax tree that tree-sitter creates in order to create a typed abstract syntax tree in the type checking stage. The errors in the type checking stage are not raised to the user, and are instead used by developers to debug tests.

The parser is solely responsible for turning text into recognized values, while the type checker does arity checking, and enforces argument list types (e.g. nested function calls like {{ config(my_ref=ref('table')) }} will parse but not type check even though it is valid dbt syntax. The tool at this time doesn't have an agreed serialization to communicate refs as config values, but could in the future.)

The extractor uses the typed abstract syntax tree to easily identify all the refs, sources, and configs present and extract them.

Running The Demo App

To see the full implementation extract dbt values live as you type in a browser, run:

make demo

It may take a moment for the demo to compile an optimized version of itself.

Kill the server with ctrl+c to end the demo.

Testing The Project

make test

Future Work

Refactor the tree-sitter jinja parser into its own repository to potentially open source and engage with the community on implementing improvements.
Remove ref, source, and config type checking as hard coded rules and instead read these function types from external function definition statements.
Create input path for a manifest file so it can be run on any project without additional pre-processing

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt_extractor-0.4.1.tar.gz (264.6 kB view details)

Uploaded Mar 17, 2022 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

dbt_extractor-0.4.1-cp36-abi3-win_amd64.whl (245.4 kB view details)

Uploaded Mar 17, 2022 CPython 3.6+Windows x86-64

dbt_extractor-0.4.1-cp36-abi3-win32.whl (228.1 kB view details)

Uploaded Mar 17, 2022 CPython 3.6+Windows x86

dbt_extractor-0.4.1-cp36-abi3-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+musllinux: musl 1.2+ x86-64

dbt_extractor-0.4.1-cp36-abi3-musllinux_1_2_i686.whl (1.3 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+musllinux: musl 1.2+ i686

dbt_extractor-0.4.1-cp36-abi3-musllinux_1_2_armv7l.whl (1.5 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+musllinux: musl 1.2+ ARMv7l

dbt_extractor-0.4.1-cp36-abi3-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+musllinux: musl 1.2+ ARM64

dbt_extractor-0.4.1-cp36-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+manylinux: glibc 2.17+ s390x

dbt_extractor-0.4.1-cp36-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+manylinux: glibc 2.17+ ppc64le

dbt_extractor-0.4.1-cp36-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (1.2 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+manylinux: glibc 2.17+ ppc64

dbt_extractor-0.4.1-cp36-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.1 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+manylinux: glibc 2.17+ ARMv7l

dbt_extractor-0.4.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+manylinux: glibc 2.17+ ARM64

dbt_extractor-0.4.1-cp36-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+manylinux: glibc 2.5+ x86-64

dbt_extractor-0.4.1-cp36-abi3-manylinux_2_5_i686.manylinux1_i686.whl (1.1 MB view details)

Uploaded Mar 17, 2022 CPython 3.6+manylinux: glibc 2.5+ i686

dbt_extractor-0.4.1-cp36-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (691.1 kB view details)

Uploaded Mar 17, 2022 CPython 3.6+macOS 10.9+ universal2 (ARM64, x86-64)macOS 10.9+ x86-64macOS 11.0+ ARM64

dbt_extractor-0.4.1-cp36-abi3-macosx_10_7_x86_64.whl (354.2 kB view details)

Uploaded Mar 17, 2022 CPython 3.6+macOS 10.7+ x86-64

File details

Details for the file dbt_extractor-0.4.1.tar.gz.

File metadata

Download URL: dbt_extractor-0.4.1.tar.gz
Upload date: Mar 17, 2022
Size: 264.6 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2

File hashes

Hashes for dbt_extractor-0.4.1.tar.gz
Algorithm	Hash digest
SHA256	`75b1c665699ec0f1ffce1ba3d776f7dfce802156f22e70a7b9c8f0b4d7e80f42`
MD5	`48e71fbb44be0add5bdaa7de58acbf53`
BLAKE2b-256	`af2ea110b40212480fd02bff567ff84effea8b9937ccd6ebfad0f10a382183d2`

Algorithm	Hash digest
SHA256	`35265a0ae0a250623b0c2e3308b2738dc8212e40e0aa88407849e9ea090bb312`
MD5	`be09280a8cd0a99855b053abd010bafd`
BLAKE2b-256	`978f344cddf0cb94c9c8d19966cfc389056208d8d7703e8920ed3da02e7b598c`

Algorithm	Hash digest
SHA256	`3fe8d8e28a7bd3e0884896147269ca0202ca432d8733113386bdc84c824561bf`
MD5	`ac5b53daca6a3e401386e040c6da47d0`
BLAKE2b-256	`d16770549d920ae5528aed090fc87e268aa41eceb1796763add6a2a7016b4d35`

Algorithm	Hash digest
SHA256	`037907a7c7ae0391045d81338ca77ddaef899a91d80f09958f09fe374594e19b`
MD5	`4b4441a24fa8d7aa5d21d10406c0be09`
BLAKE2b-256	`1cbdd14a9011a2d023f00154af4c61819b1927a94b4bdadda3fd81e5a4c7436c`

Algorithm	Hash digest
SHA256	`7d7c47774dc051b8c18690281a55e2e3d3320e823b17e04b06bc3ff81b1874ba`
MD5	`0635887cfefa364d371dc28db1180482`
BLAKE2b-256	`414b00cded27a10581a4c44365b1843735139d3b1330f5879f90d17ae4b54f32`

Algorithm	Hash digest
SHA256	`9da211869a1220ea55c5552c1567a3ea5233a6c52fa89ca87a22465481c37bc9`
MD5	`f11a1f8cb14c42de57bc3ee0ba6440fe`
BLAKE2b-256	`d3b80617c3a66aacc8b97958f686d4279f2d1934bdb756121d24fc14ed249f66`

Algorithm	Hash digest
SHA256	`34783d788b133f223844e280e37b3f5244f2fb60acc457aa75c2667e418d5442`
MD5	`15a9e5197eb952e504a4daa88d55c5d5`
BLAKE2b-256	`a53491373ef44a9bed8d85a51b070877c957837066a4aa54c54227c6bed06be7`

Algorithm	Hash digest
SHA256	`554d27741a54599c39e5c0b7dbcab77400d83f908caba284a3e960db812e5814`
MD5	`89e402839d550e97c43bb4371e4c86ce`
BLAKE2b-256	`21d59d4c3fcf21c015a8d56e05a39f42d3084930ec31f79bb431cdf8fede9e0a`

Algorithm	Hash digest
SHA256	`822b1e911db230e1b9701c99896578e711232001027b518c44c32f79a46fa3f9`
MD5	`38ffbc11c0f71d6358946121965169a7`
BLAKE2b-256	`54340350983b2b72268c587c9c7bde278008e8370cacce3b15bef2c490b6fed0`

Algorithm	Hash digest
SHA256	`7c291f9f483eae4f60dd5859097d7ba51d5cb6c4725f08973ebd18cdea89d758`
MD5	`9df9766c4845994bfc8ee07fc7747f5f`
BLAKE2b-256	`fde71f26ec7ffd95d9892a0a4184650682eb91c121e36ea48dfa29ca3db4b0d3`

Algorithm	Hash digest
SHA256	`81435841610be1b07806d72cd89b1956c6e2a84c360b9ceb3f949c62a546d569`
MD5	`680f1a38f7d8a4158778aa40472f6073`
BLAKE2b-256	`adeb8120ff2b9736aa9cc913d77b103cd5d18a1fcab8926b90f324f76742481c`

Algorithm	Hash digest
SHA256	`76872cdee659075d6ce2df92dc62e59a74ba571be62acab2e297ca478b49d766`
MD5	`9728373a8c4d9b5d9eb78b25237b1dc4`
BLAKE2b-256	`58c1699c60d4149379e97c5a0ded03e73fa410cdb146df2e1421f531e1648803`

Algorithm	Hash digest
SHA256	`cad90ddc708cb4182dc16fe2c87b1f088a1679877b93e641af068eb68a25d582`
MD5	`3567e99b68ff7e909515160fd41196b7`
BLAKE2b-256	`a4d28b8ab86a5d40ed4d8489eaca062fc15e1902822051daa9b411e05f809374`

Algorithm	Hash digest
SHA256	`a805d51a25317f53cbff951c79b9cf75421cf48e4b3e1dfb3e9e8de6d824b76c`
MD5	`6566af46e0a1cb18793197c0c04cf89a`
BLAKE2b-256	`6219ed940b6331a517607eca310fb2900c944bf7f9824cc1a237659c78b8f790`

Algorithm	Hash digest
SHA256	`bc9e0050e3a2f4ea9fe58e8794bc808e6709a0c688ed710fc7c5b6ef3e5623ec`
MD5	`9111ad0ad5fe4d2b0c5afbb6b128a3ab`
BLAKE2b-256	`d64b00a5bd94051bb83ca78a4937c6be61cd09aa1038379d9ed98eadab992ba8`

Algorithm	Hash digest
SHA256	`4dc715bd740e418d8dc1dd418fea508e79208a24cf5ab110b0092a3cbe96bf71`
MD5	`cb5ddf853fa3c24f0a5e046dcc811bc7`
BLAKE2b-256	`d1ee78064a293f065a8a6db24b64fa312db7e9cbf117f8911d68258f1643c65f`

dbt-extractor 0.4.1

dbt extractor

Strategy

Architecture

Running The Demo App

Testing The Project

Future Work

Project links

Key dates

1 maintainer

Credits

License

Requires

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

Release history Release notifications | RSS feed