dbt-polars

dbt-polars integrates the flexibility and performance of Polars with the data governance and lineage capabilities of dbt.

Try the quick start and run your first dbt-polars project in a few minutes, on your own laptop.

Python and SQL, side by side

Choose Python or SQL per model, whichever fits the transformation best. Both run on the same Polars engine and can freely reference each other. Unlike in dbt-core, Python models are first class and can do anything a SQL model can, even be ephemeral.

import polars as pl


def model(dbt, session):
    return (
        dbt.ref("orders")
        .group_by("country")
        .agg(revenue=pl.sum("amount"))
    )

select
    country,
    sum(amount) as revenue
from {{ ref('orders') }}
group by country

The flexibility of Python and Polars

Run anywhere. Polars executes your models in the same process as dbt. You don't need an expensive data warehouse to run your transformations. dbt-polars runs anywhere you can run Python.

Store data where you want. dbt-polars connects to popular platforms through catalogs. Setting one up only takes a few lines in your profile.

Extensible to any platform. Implement a custom catalog to tailor dbt-polars to your platform. A custom catalog controls authentication, where data is written and how metadata is registered. The dbt-polars engine still performs the writes.

The familiarity of dbt

If you know dbt, you already know how to use dbt-polars. Introduce dbt-polars to your Python projects to benefit from key dbt features, such as lineage, documentation and tests. See dbt compatibility for the full list of supported dbt features.

Built for efficiency

dbt-polars builds on the full power of the Polars engine. Models run lazily, so Polars optimizes the whole query before running it. Execution and writes are streamed in batches to keep memory use low.

Get started

Install dbt-polars and create a new project:

pip install dbt-polars
dbt init

The quick start walks you through the rest. It runs locally and writes tables to your disk, so you don't need a cloud account. Afterwards, the demo project builds a weather data pipeline that uses many of dbt-polars' features along the way.

Limitations

• Views are not supported, as there is no database to hold them. Use table or ephemeral instead.
• The hash macro is not supported. Use a Python model for hashing instead.

Documentation

The full documentation is available at datamindedbe.github.io/dbt-polars.