Skip to main content

A tool to analyze and extract information from Jinja used in dbt projects.

Project description

dbt extractor

Understanding dbt-extractor

This repository contains a tool that processes the most common jinja value templates in dbt model files. The tool depends on tree-sitter and the tree-sitter-jinja2 library.

demo app

Getting started

Strategy

The current strategy is for this processor to be 100% certain when it can accurately extract values from a given model file. Anything less than 100% certainty returns an exception so that the model can be rendered with python Jinja instead.

There are two cases we want to avoid because they would risk correctness to user's projects:

  1. Confidently extracting values that would not be extracted by python jinja (false positives)
  2. Confidently extracting a set of values that are missing values that python jinja would have extracted. (misses)

If we instead error when we could have confidently extracted values, there is no correctness risk to the user. Only an opportunity to expand the rules to encompass this class of cases as well.

Even though jinja in dbt is not a typed language, the type checker statically determines whether or not the current implementation can confidently extract values without relying on python jinja rendering, which is when these errors would otherwise surface. This type checker will become more permissive over time as this tool expands to include more dbt and jinja features.

Architecture

This architecture is optimized for value extraction and for future flexibility. This architecture is expected to change, and is coded in fp-style stages to make those changes easier for the future.

This processor is composed of several stages:

  1. parser
  2. type checker
  3. extractor

Additionally, the following tools utilize the above processor:

  1. browser-based demo of dbt extraction as you type

The tree-sitter parser is located in the tree-sitter-jinja2 library. The rust bindings are used to traverse the concrete syntax tree that tree-sitter creates in order to create a typed abstract syntax tree in the type checking stage. The errors in the type checking stage are not raised to the user, and are instead used by developers to debug tests.

The parser is solely responsible for turning text into recognized values, while the type checker does arity checking, and enforces argument list types (e.g. nested function calls like {{ config(my_ref=ref('table')) }} will parse but not type check even though it is valid dbt syntax. The tool at this time doesn't have an agreed serialization to communicate refs as config values, but could in the future.)

The extractor uses the typed abstract syntax tree to easily identify all the refs, sources, and configs present and extract them.

Join the dbt Community

Reporting bugs and contributing code

Code of Conduct

Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the dbt Code of Conduct.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt_extractor-0.6.0.tar.gz (270.5 kB view details)

Uploaded Source

Built Distributions

dbt_extractor-0.6.0-cp39-abi3-win_amd64.whl (277.1 kB view details)

Uploaded CPython 3.9+Windows x86-64

dbt_extractor-0.6.0-cp39-abi3-win32.whl (252.7 kB view details)

Uploaded CPython 3.9+Windows x86

dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl (613.0 kB view details)

Uploaded CPython 3.9+musllinux: musl 1.2+ x86-64

dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_i686.whl (644.4 kB view details)

Uploaded CPython 3.9+musllinux: musl 1.2+ i686

dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_armv7l.whl (697.3 kB view details)

Uploaded CPython 3.9+musllinux: musl 1.2+ ARMv7l

dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl (618.0 kB view details)

Uploaded CPython 3.9+musllinux: musl 1.2+ ARM64

dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442.7 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (494.9 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ s390x

dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (518.0 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ppc64le

dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (524.3 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ppc64

dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (470.1 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ i686

dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (434.6 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARMv7l

dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (435.1 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl (404.4 kB view details)

Uploaded CPython 3.9+macOS 10.12+ x86-64

dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl (790.2 kB view details)

Uploaded CPython 3.9+macOS 10.12+ universal2 (ARM64, x86-64)macOS 10.12+ x86-64macOS 11.0+ ARM64

File details

Details for the file dbt_extractor-0.6.0.tar.gz.

File metadata

  • Download URL: dbt_extractor-0.6.0.tar.gz
  • Upload date:
  • Size: 270.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for dbt_extractor-0.6.0.tar.gz
Algorithm Hash digest
SHA256 d6cf08ec793b8bc2bd6e260ef818230ae68a4f71436fa489f08d7db1a52e2ffe
MD5 2014bf941a760886130e594c1daae874
BLAKE2b-256 f1061f7b5d277af4bd7c3ab5065f79407c46a73950f0879fac69e51067c87649

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0.tar.gz:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 a79a570fdcb672505ac2bdc12360a2a7aec622ef604d8c607225854ff862518c
MD5 973833b4181e67b50b1cc863c18a79e3
BLAKE2b-256 6036f5b1c4159fa911607f3a49fcbc535e4783870fd887bc0a1b3ad42587cb73

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-win_amd64.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-win32.whl.

File metadata

  • Download URL: dbt_extractor-0.6.0-cp39-abi3-win32.whl
  • Upload date:
  • Size: 252.7 kB
  • Tags: CPython 3.9+, Windows x86
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-win32.whl
Algorithm Hash digest
SHA256 369dcc3499f160256756585783f1308868076d5a65d0a051348d22da8b90e67d
MD5 e731904c01b0512b7931ff3909bb466a
BLAKE2b-256 c4ded1492ab6beaf0a18aee17c7a9562592ac2981e962b4058262f5eb6dabfc5

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-win32.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 caeaba8d8c813f8e32d586c12615c0c7d6b99bee4f1be845312e80ef731de164
MD5 fb7c70d2d5923c5f1cd5aa451730651a
BLAKE2b-256 3b02b111856273e414ac80ef58d2103c9b7c6a5b29b1ec248999d3d5873ada00

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 1b9ed7b15df983a735f87773f6765db8458680c02fcebbf89df4e238503c0e08
MD5 691ed355b7df5d9b951aed36ef7d238b
BLAKE2b-256 946d46bdb9a809c66784fcc19b853311568cfd3041c075f0a578cb7116686841

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_i686.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_armv7l.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_armv7l.whl
Algorithm Hash digest
SHA256 080fd1edf123926ed97929c65a75874d0fea687ccd5d3ebbc9e81b339f099604
MD5 f81dcff5fdc59dc591d7fa0a5c07e17f
BLAKE2b-256 18c6cdaf1ac8959d571b5cb3587b8afef9e5fe60b99fe59aca94560808501d8b

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_armv7l.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 a5cb810edc60c0486f78cc29739ebda70c81b10a1686861e78addc9f91fcd7de
MD5 be4b4c8497a4e95d28f7f399d009f0a1
BLAKE2b-256 9d8249068ee2b9f38aa34d0f3196bb7b71d11af86630d5ed5cb6626108c97cd6

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 aecfa43f7e6f139e76d47e4e1d7b189655ae19a8cf697686230bacb89a94ae74
MD5 4cc8aec00c7abf7f3e26e26679f1c51f
BLAKE2b-256 df5efad01e18d68ffd09c0f39cdedeed8fcaaea74a8b46d1a944472b5f95b72b

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 311f0d3a4994751c541a4fa303d205727ba90e90c85286c03d3d9284e2bf0bd4
MD5 0ec7713b4c4b2098f6462386d8b9e254
BLAKE2b-256 a44f6994cdfb51c5652fad0c8f9cf5b3ec1816cb10e99ed145eb27e6a9bcc16b

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 c1fd2b083a75e80b13e9874dc9699bfdfddf3baa9b6a8dea48de06d51a082733
MD5 1da82b1dbc71e631de18d44f9eeaebb2
BLAKE2b-256 9ead9dd410d4d95e336ae6b10c53c939bf1ff8e9991e1adb5ea4aefc4a87c445

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl
Algorithm Hash digest
SHA256 868af715a6328d7317ce6e4db238f850f660fef13fb36b7ab4cf9163ed5f54ff
MD5 b34886fe0117437546dd7c548ae8a750
BLAKE2b-256 b218611dceb2fa7ea668471f290f34fec55fa3283e3ee9d0475d964e6ffaff97

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 71b3f8897138cc6698d313b9a3d0450fd021937ff5463269ee18ed415541781b
MD5 7778de006f6ecf3666203836da4a9073
BLAKE2b-256 405a468a2855181aaee5402efbf9ef757d074cd306eec22bbcd267cdd0edbe94

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm Hash digest
SHA256 05bcfab7ebd70296ceb31742e8333ba66a2c939de44e61a7088bebafa939aaf6
MD5 00b1a53fd0d783892da1c156693ed7dc
BLAKE2b-256 582e1ef1cd2b36973bea0a6823a7b7cd1b3db29b61ddebb015ceaea88b9e9347

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 af451633390ac19669d3bde6c79822e657d32f5d903b3388bb00d56333fd52d5
MD5 6e014be12bca72ca5f2651bde5029984
BLAKE2b-256 c92ae14c13b9a437780c5712525ce537915b531bba45481fc7102deb4492ff83

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 dcf14ed245de8df269815ff4c4f555fa72d2621f4fff37c023b8c99d0e421b4f
MD5 187191ecd45604baf94b64359d324456
BLAKE2b-256 035f233f326336aa21fbd9e7268f239a8464af145abd398a360d894c3286699d

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl.

File metadata

File hashes

Hashes for dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Algorithm Hash digest
SHA256 4b6b1e70dde78cb904ca7a8958c2c803e77779b6ce108f4ea7ac479f5700db89
MD5 2cb46da35a4e2d8fbdd42ed04db4f3b3
BLAKE2b-256 9fddec8f9e48e7dd5a52a69cca7907681d1779cf1cc8b02f2aa2acb6a2bf8bb4

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl:

Publisher: release.yml on dbt-labs/dbt-extractor

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page