dbt extractor

demo app

This repository contains a tool that processes the most common jinja value templates in dbt model files. The tool depends on tree-sitter and the tree-sitter-jinja2 library.

Strategy

The current strategy is for this processor to be 100% certain when it can accurately extract values from a given model file. Anything less than 100% certainty returns an exception so that the model can be rendered with python Jinja instead.

There are two cases we want to avoid because they would risk correctness to user's projects:

  1. Confidently extracting values that would not be extracted by python jinja (false positives)
  2. Confidently extracting a set of values that are missing values that python jinja would have extracted. (misses)

If we instead error when we could have confidently extracted values, there is no correctness risk to the user. Only an opportunity to expand the rules to encompass this class of cases as well.

Even though jinja in dbt is not a typed language, the type checker statically determines whether or not the current implementation can confidently extract values without relying on python jinja rendering, which is when these errors would otherwise surface. This type checker will become more permissive over time as this tool expands to include more dbt and jinja features.

Architecture

This architecture is optimized for value extraction and for future flexibility. This architecture is expected to change, and is coded in fp-style stages to make those changes easier for the future.

This processor is composed of several stages:

  1. parser
  2. type checker
  3. extractor

Additionally, the following tools utilize the above processor:

  1. browser-based demo of dbt extraction as you type

The tree-sitter parser is located in the tree-sitter-jinja2 library. The rust bindings are used to traverse the concrete syntax tree that tree-sitter creates in order to create a typed abstract syntax tree in the type checking stage. The errors in the type checking stage are not raised to the user, and are instead used by developers to debug tests.

The parser is solely responsible for turning text into recognized values, while the type checker does arity checking, and enforces argument list types (e.g. nested function calls like {{ config(my_ref=ref('table')) }} will parse but not type check even though it is valid dbt syntax. The tool at this time doesn't have an agreed serialization to communicate refs as config values, but could in the future.)

The extractor uses the typed abstract syntax tree to easily identify all the refs, sources, and configs present and extract them.

Running The Demo App

To see the full implementation extract dbt values live as you type in a browser, run:

make demo

It may take a moment for the demo to compile an optimized version of itself.

Kill the server with ctrl+c to end the demo.

Testing The Project

make test

Future Work

  • Refactor the tree-sitter jinja parser into its own repository to potentially open source and engage with the community on implementing improvements.
  • Remove ref, source, and config type checking as hard coded rules and instead read these function types from external function definition statements.
  • Create input path for a manifest file so it can be run on any project without additional pre-processing

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.6.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for fishtown-analytics from gravatar.com fishtown-analytics

Credits

Author: dbt Labs, Nathaniel May, Peter Webb

License

Apache Software License (Apache-2.0)

Requires

Python >=3.6.1

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dbt_extractor-0.5.0-cp38-abi3-win_amd64.whl (284.4 kB view details)

Uploaded CPython 3.8+Windows x86-64

dbt_extractor-0.5.0-cp38-abi3-win32.whl (263.8 kB view details)

Uploaded CPython 3.8+Windows x86

dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_x86_64.whl (1.5 MB view details)

Uploaded CPython 3.8+musllinux: musl 1.2+ x86-64

dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_i686.whl (1.5 MB view details)

Uploaded CPython 3.8+musllinux: musl 1.2+ i686

dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_armv7l.whl (1.6 MB view details)

Uploaded CPython 3.8+musllinux: musl 1.2+ ARMv7l

dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_aarch64.whl (1.5 MB view details)

Uploaded CPython 3.8+musllinux: musl 1.2+ ARM64

dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.5 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ s390x

dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.4 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ppc64le

dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (1.5 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ppc64

dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.3 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ARMv7l

dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ARM64

dbt_extractor-0.5.0-cp38-abi3-manylinux_2_12_i686.manylinux2010_i686.whl (1.3 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.12+ i686

dbt_extractor-0.5.0-cp38-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (868.2 kB view details)

Uploaded CPython 3.8+macOS 10.9+ universal2 (ARM64, x86-64)macOS 10.9+ x86-64macOS 11.0+ ARM64

dbt_extractor-0.5.0-cp38-abi3-macosx_10_7_x86_64.whl (444.8 kB view details)

Uploaded CPython 3.8+macOS 10.7+ x86-64

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-win_amd64.whl.

File metadata

  • Download URL: dbt_extractor-0.5.0-cp38-abi3-win_amd64.whl
  • Upload date:
  • Size: 284.4 kB
  • Tags: CPython 3.8+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 0dc2879068f4b325beb2d866842b923d5df5ad953c122588b17151b32d9cd33b
MD5 57903bff68bb1adae5a024f99b1d10b9
BLAKE2b-256 679e633a15f4a9302b36de479d3ee5f61ab7c5668d2e29a9b1c304afeee24b12

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-win32.whl.

File metadata

  • Download URL: dbt_extractor-0.5.0-cp38-abi3-win32.whl
  • Upload date:
  • Size: 263.8 kB
  • Tags: CPython 3.8+, Windows x86
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-win32.whl
Algorithm Hash digest
SHA256 4314011c99f91a4041f048c6a4e2b9d24180f5e5935febca68dd28db5ef55177
MD5 6916a6cbc4efdac9c76284d83319305d
BLAKE2b-256 1f6bbaa3d866b2c84d1ff3135bd27353f1748cc5f4b7b814b289bfa0f20b8561

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 7199d4fccd21889b59d873fd7b453a59672e9cdf445b938cf280d0fb1d805afc
MD5 9a7c6fcb5ed9e9b5bece7ebeb2164a67
BLAKE2b-256 f1fd6d5438aab6a3c65d6e2e03293c8ae3b1eb93303dafbfb98c25e35d2e8f5e

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 fcce461dc21e737e3c9788413bb3142dc50bf5c64a35035ffe1f8441bc86d133
MD5 f491ffd9a645f8be36834b19aababf18
BLAKE2b-256 17bad9ac8608359e5bd41d366908f9f5dfdcc3427cef177911a04ced95094ce0

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_armv7l.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_armv7l.whl
Algorithm Hash digest
SHA256 01216bd1636a6e0f80a72f1baaefe2a9ec95e16dc86e4c4c9474bf11307c0847
MD5 bdb0aeff4a2cef3637ea06e6dfdf5ea4
BLAKE2b-256 cbcc5e11098624c1da9b2001cb33f64a15773700c4a49bbde1bafbab82265eef

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 b61a14f273d94b9279510e3cfab0559ca4f8bff83a3e1ea8b367d17a34bccf65
MD5 0930fb0bcf00e0b6331db336b45fd6c1
BLAKE2b-256 15de6b78834606248789c7813a6ce20c7fa3bf08c6973617a8b189c78a540263

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2a56a24cc0b52d004274c9a4eadb342fddc36afae178f59da400da5a214061a4
MD5 e7de1c319d98c8a15b8d861e22fb6691
BLAKE2b-256 9b5150458dc1e1fa985a46ebb350cb61d4fc0d0f3fe00ae55c7e614f25801865

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 6ca976dd4d73de8be2eddde1285007ee7f2f826218a7975bd00e71830b499eac
MD5 0c9af6b894b38ffce5d7b5a473c47266
BLAKE2b-256 547e9c4ce08a59b40e4456252888c0ab13deee1dcbb957180fc39429b8ace8c4

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 856719ad009cf3aa29d8f0abbf72eff0286ba0f2351c58c0c4eb87303aabf476
MD5 6323de38e690ce648df6dfaf678ab9a8
BLAKE2b-256 100dca9020c487375d20e70ca235cceb1f395a82a0af1ef79431ae56ca70d4a4

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl
Algorithm Hash digest
SHA256 c1a4dcf3571da55671dc3d96df64ea0ec3649588bf5a58677cad5b207edd9e93
MD5 cf0321889f751cf7042e368e34f6b156
BLAKE2b-256 738f89700f7d8a2956bfafe92be5b664308189fafc63d1dc6f2035aedce8b16c

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm Hash digest
SHA256 3249ce65b6add2c0bef756d8e7fe19a838451c5a63c08e52de7b4a4c5aad40de
MD5 50b64c5ceeec6e9f40aef030cb7eafe1
BLAKE2b-256 f68b4c4ac4394bc63e209b50adb93fe1ac87d57ecd288dd82de09a857e983731

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 166bad7ee26487ad22b4d7d3c98f168b0c450004aaaa7babd9ccf3c7e36a48f8
MD5 79a40f5e1320195ecfd967ddda15f7da
BLAKE2b-256 3208cf1d804b6150598b89ae6918b89651f9ac1770e3c44f3346db525a96f2e6

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-manylinux_2_12_i686.manylinux2010_i686.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm Hash digest
SHA256 a980467342e3a9f9f79eb8eab2dd0106f8fd3c13913b7af2f4ad96279ad3df25
MD5 ce37cb8fe4ef07f3a128024c9991c629
BLAKE2b-256 69cf6074bf968eac73fdd6bc6713049c028798a5032e6cb709310203900f731f

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 111e4766736a00f1a4b76d0fe2016d70015b833b791b0bf20c2328e3d27e35f6
MD5 5a568107979c4d0e10801ebfed976b20
BLAKE2b-256 59389908b2b11978febbe7cf22e1088de8cabec1ec9b2256c1c61d959856de4b

See more details on using hashes here.

File details

Details for the file dbt_extractor-0.5.0-cp38-abi3-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.5.0-cp38-abi3-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 f333af362bdb95fa6e52b88e1df3575755680190b4867344c808d438a98b6e71
MD5 3d34666c17cbf9a2c899d3c86448abc4
BLAKE2b-256 903c3572e10db3fc6aa5505752c685214526000b23b8f7fc98b4a44ea2a781a8

See more details on using hashes here.

Release history Release notifications | RSS feed

0.6.0

16 files

0.5.1

16 files

This release

0.5.0 This release

15 files

0.4.1

16 files

0.4.0

53 files

0.3.0

44 files

0.2.0

44 files

0.1.0

44 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page