dpyr

dplyr for Python. The tidyverse's verbs — filter, mutate, group_by, summarize, joins, across, tidyselect — as Python method chains, executing on polars or duckdb, with real IDE autocompletion and semantics verified against dplyr itself.

pip install dpyr        # or: uv add dpyr

from dpyr import read, col, n, desc

starwars = read("starwars.parquet")   # read() takes anything tabular:
                                      # .parquet/.csv/.arrow/.db paths, dicts,
                                      # lists of records, polars/pandas frames,
                                      # arrow tables, Hugging Face datasets,
                                      # numpy/torch/jax

(
    starwars
    .filter(col.height > 180, col.mass < 100)
    .mutate(bmi = col.mass / (col.height / 100) ** 2)
    .group_by(col.species)
    .summarize(
        n = n(),
        mean_bmi = col.bmi.mean(),
    )
    .arrange(desc(col.mean_bmi))
)

Evaluate that in a notebook and you see rows immediately. Typo a column name and you get the error on that line, with a did-you-mean suggestion. Wrap the same code in a pipeline and only .collect() at the end, and the whole chain runs as one fused query with predicate pushdown. That combination — schema-eager, data-lazy, display-eager — is the core design.

Two backends, one semantics

import duckdb
from dpyr import read

df  = read({"x": [1, 2, 3], "g": ["a", "a", "b"]})   # polars engine
con = duckdb.connect("warehouse.db")
tbl = read(con, "events")                            # SQL pushdown

Identical chains produce identical results on both engines — enforced by a Hypothesis fuzzer that runs random verb chains on both and compares bit-for-bit, and by differential tests against real dplyr: every spec in tests/specs/ is executed by dplyr (via oracle/run_specs.R) to produce a committed golden parquet, then replayed through dpyr on both backends. Where R and the engines genuinely disagree, the decision is documented in docs/SEMANTICS.md, not left to chance.

The dplyr you know

Grouped mutate/filter are windowed per group, summarize peels one grouping level, joins use .x/.y suffixes and match NAs by default — the dplyr behaviors, deliberately.

Autocompletion that actually works

• df.c.height — frame-bound proxy: column names complete from the live schema, and the returned expression is typed (.mean() on numerics, .str_detect() on strings; calling .mean() on a string column raises immediately, at build time).
• df.filter(lambda c: c.height > 180) — lambda style for the same effect.
• dpyr stubgen data/*.parquet -o schemas.py — generates typed schema modules so completion and type-checking work statically in any IDE.

The database is a destination, not just a source

db = read("warehouse.db")                 # catalog object: db.tables, db.orders
gold = db.orders.group_by(col.region).summarize(rev = col.amount.sum())
gold.to_table("gold_revenue")             # CREATE TABLE AS <sql>, fully in-engine
gold.to_view("gold_live")                 # the lazy plan as a named view
gold.write("gold.parquet")                # in-engine COPY (extension dispatch)
mem = read({"region": ["east"], "target": [1000.0]})
gold.inner_join(mem, on = col.region)     # in-memory frames bridge into duckdb
                                          # automatically (arrow, zero-copy)

Your own functions, row by row

from dpyr import vectorize

@vectorize(threads=8)
def classify(text: str, labels: str) -> str: ...      # any Python: a parser, an API, a model

reviews.mutate(topic = classify(col.text, labels="price,quality"))

Typed from the annotation, one call per distinct input, results remembered for the session, and a displayed dataframe only runs the rows it shows (see the row functions guide).

Interactive by default, lazy when you need it

df.persist()           # checkpoint: materialize now (duckdb: temp table)
df.lazy()              # this frame never executes implicitly
dpyr.options.interactive = False   # global opt-out for production pipelines

Results are cached by plan hash, so re-displaying a frame in a notebook never recomputes it.

Documentation

Full guides at maximerivest.github.io/dpyr — get started, grouped data, joins, window functions, column-wise operations, reshaping, expressions & autocompletion, and the backends guide (connecting and operating polars and duckdb).

Project documents

License

MIT © Maxime Rivest