dbt extractor
Understanding dbt-extractor
This repository contains a tool that processes the most common jinja value templates in dbt model files. The tool depends on tree-sitter and the tree-sitter-jinja2 library.
Getting started
- Read the introduction and viewpoint of dbt
Strategy
The current strategy is for this processor to be 100% certain when it can accurately extract values from a given model file. Anything less than 100% certainty returns an exception so that the model can be rendered with python Jinja instead.
There are two cases we want to avoid because they would risk correctness to user's projects:
- Confidently extracting values that would not be extracted by python jinja (false positives)
- Confidently extracting a set of values that are missing values that python jinja would have extracted. (misses)
If we instead error when we could have confidently extracted values, there is no correctness risk to the user. Only an opportunity to expand the rules to encompass this class of cases as well.
Even though jinja in dbt is not a typed language, the type checker statically determines whether or not the current implementation can confidently extract values without relying on python jinja rendering, which is when these errors would otherwise surface. This type checker will become more permissive over time as this tool expands to include more dbt and jinja features.
Architecture
This architecture is optimized for value extraction and for future flexibility. This architecture is expected to change, and is coded in fp-style stages to make those changes easier for the future.
This processor is composed of several stages:
- parser
- type checker
- extractor
Additionally, the following tools utilize the above processor:
- browser-based demo of dbt extraction as you type
The tree-sitter parser is located in the tree-sitter-jinja2 library. The rust bindings are used to traverse the concrete syntax tree that tree-sitter creates in order to create a typed abstract syntax tree in the type checking stage. The errors in the type checking stage are not raised to the user, and are instead used by developers to debug tests.
The parser is solely responsible for turning text into recognized values, while the type checker does arity checking, and enforces argument list types (e.g. nested function calls like {{ config(my_ref=ref('table')) }} will parse but not type check even though it is valid dbt syntax. The tool at this time doesn't have an agreed serialization to communicate refs as config values, but could in the future.)
The extractor uses the typed abstract syntax tree to easily identify all the refs, sources, and configs present and extract them.
Join the dbt Community
- Be part of the conversation in the dbt Community Slack
- Read more on the dbt Community Discourse
Reporting bugs and contributing code
- Want to report a bug or request a feature? Let us know on Slack, or open an issue
- Want to help us build
dbt-extractor? Check out the Contributing Guide
Code of Conduct
Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the dbt Code of Conduct.
1 maintainer
Release files for dbt-extractor 0.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dbt_extractor-0.6.0.tar.gz | 270.5 kB | Details |
Built distributions (wheels)
Total release size: 7.9 MB
Release files / dbt_extractor-0.6.0.tar.gz
| Download URL | dbt_extractor-0.6.0.tar.gz |
|---|---|
| Size | 270.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d6cf08ec793b8bc2bd6e260ef818230ae68a4f71436fa489f08d7db1a52e2ffe
|
|
BLAKE2b-256 checksum How to use checksums |
f1061f7b5d277af4bd7c3ab5065f79407c46a73950f0879fac69e51067c87649
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-win_amd64.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-win_amd64.whl |
|---|---|
| Size | 277.1 kB |
| Tags | CPython 3.9 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
a79a570fdcb672505ac2bdc12360a2a7aec622ef604d8c607225854ff862518c
|
|
BLAKE2b-256 checksum How to use checksums |
6036f5b1c4159fa911607f3a49fcbc535e4783870fd887bc0a1b3ad42587cb73
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-win32.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-win32.whl |
|---|---|
| Size | 252.7 kB |
| Tags | CPython 3.9 Windows x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
369dcc3499f160256756585783f1308868076d5a65d0a051348d22da8b90e67d
|
|
BLAKE2b-256 checksum How to use checksums |
c4ded1492ab6beaf0a18aee17c7a9562592ac2981e962b4058262f5eb6dabfc5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 613.0 kB |
| Tags | CPython 3.9 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
caeaba8d8c813f8e32d586c12615c0c7d6b99bee4f1be845312e80ef731de164
|
|
BLAKE2b-256 checksum How to use checksums |
3b02b111856273e414ac80ef58d2103c9b7c6a5b29b1ec248999d3d5873ada00
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_i686.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_i686.whl |
|---|---|
| Size | 644.4 kB |
| Tags | CPython 3.9 Linux musl 1.2+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
1b9ed7b15df983a735f87773f6765db8458680c02fcebbf89df4e238503c0e08
|
|
BLAKE2b-256 checksum How to use checksums |
946d46bdb9a809c66784fcc19b853311568cfd3041c075f0a578cb7116686841
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_armv7l.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_armv7l.whl |
|---|---|
| Size | 697.3 kB |
| Tags | CPython 3.9 Linux musl 1.2+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
080fd1edf123926ed97929c65a75874d0fea687ccd5d3ebbc9e81b339f099604
|
|
BLAKE2b-256 checksum How to use checksums |
18c6cdaf1ac8959d571b5cb3587b8afef9e5fe60b99fe59aca94560808501d8b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 618.0 kB |
| Tags | CPython 3.9 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
a5cb810edc60c0486f78cc29739ebda70c81b10a1686861e78addc9f91fcd7de
|
|
BLAKE2b-256 checksum How to use checksums |
9d8249068ee2b9f38aa34d0f3196bb7b71d11af86630d5ed5cb6626108c97cd6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 442.7 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
aecfa43f7e6f139e76d47e4e1d7b189655ae19a8cf697686230bacb89a94ae74
|
|
BLAKE2b-256 checksum How to use checksums |
df5efad01e18d68ffd09c0f39cdedeed8fcaaea74a8b46d1a944472b5f95b72b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl |
|---|---|
| Size | 494.9 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ IBM System/390x abi3 |
|
SHA-256 checksum How to use checksums |
311f0d3a4994751c541a4fa303d205727ba90e90c85286c03d3d9284e2bf0bd4
|
|
BLAKE2b-256 checksum How to use checksums |
a44f6994cdfb51c5652fad0c8f9cf5b3ec1816cb10e99ed145eb27e6a9bcc16b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl |
|---|---|
| Size | 518.0 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ PowerPC 64-le abi3 |
|
SHA-256 checksum How to use checksums |
c1fd2b083a75e80b13e9874dc9699bfdfddf3baa9b6a8dea48de06d51a082733
|
|
BLAKE2b-256 checksum How to use checksums |
9ead9dd410d4d95e336ae6b10c53c939bf1ff8e9991e1adb5ea4aefc4a87c445
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl |
|---|---|
| Size | 524.3 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ PowerPC 64-be abi3 |
|
SHA-256 checksum How to use checksums |
868af715a6328d7317ce6e4db238f850f660fef13fb36b7ab4cf9163ed5f54ff
|
|
BLAKE2b-256 checksum How to use checksums |
b218611dceb2fa7ea668471f290f34fec55fa3283e3ee9d0475d964e6ffaff97
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl |
|---|---|
| Size | 470.1 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
71b3f8897138cc6698d313b9a3d0450fd021937ff5463269ee18ed415541781b
|
|
BLAKE2b-256 checksum How to use checksums |
405a468a2855181aaee5402efbf9ef757d074cd306eec22bbcd267cdd0edbe94
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl |
|---|---|
| Size | 434.6 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
05bcfab7ebd70296ceb31742e8333ba66a2c939de44e61a7088bebafa939aaf6
|
|
BLAKE2b-256 checksum How to use checksums |
582e1ef1cd2b36973bea0a6823a7b7cd1b3db29b61ddebb015ceaea88b9e9347
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 435.1 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
af451633390ac19669d3bde6c79822e657d32f5d903b3388bb00d56333fd52d5
|
|
BLAKE2b-256 checksum How to use checksums |
c92ae14c13b9a437780c5712525ce537915b531bba45481fc7102deb4492ff83
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 404.4 kB |
| Tags | CPython 3.9 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
dcf14ed245de8df269815ff4c4f555fa72d2621f4fff37c023b8c99d0e421b4f
|
|
BLAKE2b-256 checksum How to use checksums |
035f233f326336aa21fbd9e7268f239a8464af145abd398a360d894c3286699d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency logRelease files / dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
| Download URL | dbt_extractor-0.6.0-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl |
|---|---|
| Size | 790.2 kB |
| Tags | CPython 3.9 abi3 macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
4b6b1e70dde78cb904ca7a8958c2c803e77779b6ce108f4ea7ac479f5700db89
|
|
BLAKE2b-256 checksum How to use checksums |
9fddec8f9e48e7dd5a52a69cca7907681d1779cf1cc8b02f2aa2acb6a2bf8bb4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 7, 2025.
Transparency log
