dbt-polars

dbt-polars integrates the flexibility and performance of Polars with the data governance and lineage capabilities of dbt.

Why dbt-polars

Combine SQL and Python models

• Freely mix SQL and Python models, even when they are ephemeral.
• Minimize data transfer by delaying model execution until write.
• Python models provide an escape hatch for complex SQL logic.

SQL and Python models both compile to a Polars LazyFrame, so they can ref() each other freely. The same model, written both ways:

SQL: models/revenue_per_country.sql	Python: models/revenue_per_country.py
select country, sum(amount) as revenue from {{ ref('orders') }} where status = 'paid' group by country	import polars as pl def model(dbt, session): orders = dbt.ref("orders") paid = orders.filter( pl.col("status") == "paid" ) return paid.group_by("country").agg( revenue=pl.col("amount").sum() )

Python ephemeral models

Unlike most dbt adapters, dbt-polars supports ephemeral Python models thanks to Polars' lazy frames. This is how it works:

1. Polars runs inside the dbt process, so Python models need no separate compute.
2. ref() returns a LazyFrame: a reference to storage. No data is read yet.
3. The ephemeral Python model returns a LazyFrame, which the downstream SQL or Python model uses like any other table. Both add to one query plan.
4. On write, Polars optimizes the full plan, reads only the data it needs, and streams the result to storage.

Supported catalogs

Catalog extensions define where dbt-polars reads and stores data.

To support another platform, you can write your own catalog. See Custom catalogs.

Limitations

• Views are not supported, as there is no database to hold them. Use table or ephemeral instead.
• The hash macro is not supported. Use a Python model for hashing instead.

See dbt compatibility for the full overview, including supported macros.

Getting started

The first steps are below. For the full walkthrough, see the Getting started guide.

Install dbt-polars and create a new dbt project:

pip install dbt-polars
dbt init

When prompted, choose polars as the database and local as the catalog type. This creates a profile that writes tables to your local disk:

my_project:
  target: dev
  outputs:
    dev:
      type: polars
      schema: main
      catalog_type: local
      root: ./data

dbt-polars does not support views, so replace the models: block in dbt_project.yml with:

models:
  my_project:
    +materialized: table

Then build the example models:

dbt run

The Getting started guide continues from here with other catalogs, seeds, ephemeral models, docs, tests and incremental ingestion.

Documentation

The full documentation is available at datamindedbe.github.io/incubator-dbt-polars.