narwhals-datafusion

Apache DataFusion backend for Narwhals, registered through the narwhals.plugins entry point.

pip install narwhals-datafusion                    # core
pip install "narwhals-datafusion[extra-functions]" # + mode, skew, kurtosis

Usage

import narwhals as nw
import pyarrow as pa
from datafusion import SessionContext

ctx = SessionContext()
df = ctx.from_arrow(pa.table({"a": [1, 2, 3], "b": ["x", "y", "x"]}))

lf = nw.from_native(df)  # nw.LazyFrame on this backend
result = (
    lf.group_by("b")
    .agg(nw.col("a").sum())
    .sort("b")
    .collect(backend="pyarrow")
)

Everything is lazy until .collect(): expressions become datafusion.Expr, frame verbs become datafusion.DataFrame methods, and DataFusion executes the plan. Dtypes are pyarrow end-to-end.

Architecture

A subclass of narwhals' SQL layer (narwhals._sql, narwhals._compliant), the layer DuckDB, Ibis and Spark also use. Tested against the narwhals release vendored as the narwhals/ submodule (2.25) and datafusion 54, the major pyproject.toml pins.

mode, skew, kurtosis

These aggregates live in the Rust-only datafusion-extra-functions crate. The extra-functions extra installs datafusion-extra-functions-ffi, a prebuilt wheel exposing them through datafusion-python's __datafusion_aggregate_udf__ capsule protocol. Without it the three methods raise NotImplementedError. The wheel pins the datafusion major it was built for.

API coverage

narwhals 2.25 on datafusion 54. ⚠️ entries work with the caveat in parentheses; see Known limitations.

Not listed: methods narwhals does not support on any lazy backend (Expr.filter, Expr.drop_nulls, Expr.unique, Expr.map_batches, Expr.ewm_mean, LazyFrame.tail, LazyFrame.gather_every).

Known limitations (as of datafusion 54)

• join_asof, exact quantile, cum_prod, list.sum/mean/median, dt.total_*, dt.offset_by, dt.timestamp, str.replace, Enum casts: no engine support, raise NotImplementedError.
• mode, skew, kurtosis: need the extra-functions extra.
• n_unique().over(...) raises: DataFusion ignores DISTINCT inside window aggregates, which would return wrong results. It also drops ORDER BY and IGNORE NULLS there; this backend moves those onto the window itself.
• fill_null(strategy=..., limit=n) raises: bounded frames with first_value/last_value need retract_batch, not implemented engine-side.
• replace_time_zone supports None and "UTC" only; use convert_time_zone for instant-preserving conversions.
• str.to_datetime/to_date require an explicit format.
• replace_strict requires an explicit default.
• nw.scan_csv/nw.scan_parquet cannot dispatch to a plugin backend yet (narwhals gap); read with a SessionContext and pass the frame to nw.from_native.
• str.to_titlecase uses initcap, which does not break words on digits.
• Row order is guaranteed only after sort: concat may interleave inputs and backward fill_null may reorder rows.

Development

git submodule update --init                       # narwhals at the tested tag
uv sync --group tests --extra extra-functions
uv run --group tests pytest tests                 # this package's tests
uv run --group tests python run_tests.py          # narwhals' suite, known failures deselected

See CONTRIBUTING.md.