narwhals-datafusion

Apache DataFusion backend for Narwhals, registered through the narwhals.plugins entry point.

pip install narwhals-datafusion                    # core
pip install "narwhals-datafusion[extra-functions]" # + mode, skew, kurtosis

Usage

import narwhals as nw
import pyarrow as pa
from datafusion import SessionContext

ctx = SessionContext()
df = ctx.from_arrow(pa.table({"a": [1, 2, 3], "b": ["x", "y", "x"]}))

lf = nw.from_native(df)  # nw.LazyFrame on this backend
result = (
    lf.group_by("b")
    .agg(nw.col("a").sum())
    .sort("b")
    .collect(backend="pyarrow")
)

Everything is lazy until .collect(): expressions become datafusion.Expr, frame verbs become datafusion.DataFrame methods, and DataFusion executes the plan. Dtypes are pyarrow end-to-end. A longer tour, extra included: docs/playground.ipynb.

Used by

• pandera: pip install 'pandera[datafusion]' validates datafusion.DataFrame through this plugin. tests/pandera/ pins the narwhals.stable.v1 calls it makes.

Compatibility

• Python >=3.10; tested on 3.12.
• datafusion>=54,<55: capped at the tested major; a release moves the cap.
• narwhals>=2.27: floor only; tested on 2.27, the narwhals/ submodule.
• datafusion-extra-functions-ffi>=0.1 (extra): each shim minor pins its datafusion major.
• Weekly, compat.yml (the badge): both suites on the newest narwhals release inside the datafusion cap, extra included.

Architecture

A subclass of narwhals' SQL layer (narwhals._sql, narwhals._compliant), the layer DuckDB, Ibis and Spark also use.

API coverage

narwhals 2.27 on datafusion 54. ⚠️ entries work with the caveat in parentheses.

Known limitations

• mode, skew and kurtosis need the extra-functions extra. It installs datafusion-extra-functions-ffi, a prebuilt wheel of the datafusion-extra-functions crate. Without it the three raise NotImplementedError.
• Not in the table: Expr.ewm_mean, Expr.map_batches, LazyFrame.tail, LazyFrame.gather_every: narwhals does not support them on lazy frames. Expr.filter, Expr.drop_nulls, Expr.unique: narwhals' SQL layer does not support them.
• Row order is guaranteed only after sort; concat may interleave inputs.
• nw.scan_csv/nw.scan_parquet cannot dispatch to a plugin yet (narwhals gap); read with a SessionContext and pass the frame to nw.from_native.