narwhals-datafusion

Apache DataFusion backend for Narwhals, implemented as an out-of-tree plugin via the narwhals.plugins entry-point system.

Usage

import narwhals as nw
import pyarrow as pa
from datafusion import SessionContext

ctx = SessionContext()
df = ctx.from_arrow(pa.table({"a": [1, 2, 3], "b": ["x", "y", "x"]}))

lf = nw.from_native(df)  # -> nw.LazyFrame, dispatched to this plugin
result = (
    lf.group_by("b")
    .agg(nw.col("a").sum())
    .sort("b")
    .collect(backend="pyarrow")
)

Everything stays lazy until .collect(): narwhals expressions are translated to datafusion.Expr, frame verbs to datafusion.DataFrame methods, and the plan executes in DataFusion's Rust engine. Dtypes are pyarrow end-to-end.

Architecture

The backend sits on narwhals' shared SQL layer (narwhals._sql), the same abstraction DuckDB, Ibis, and Spark use. It subclasses narwhals' private internals (narwhals._sql, narwhals._compliant), so it is tested against one exact release, vendored as the narwhals/ git submodule (currently 2.25), and the dependency pins are floors only. The plugin provides:

mode, skew, kurtosis via datafusion-extra-functions

DataFusion core deliberately keeps its function library lean, so aggregates like mode/skewness/kurtosis live in the contrib datafusion-extra-functions crate (Rust-only, no wheel on PyPI). The datafusion-extra-functions-ffi package — a prebuilt wheel, installed through the extra-functions extra — exposes its aggregate UDFs to datafusion-python via the __datafusion_aggregate_udf__ PyCapsule protocol; they back Expr.mode, Expr.skew, and Expr.kurtosis:

pip install "narwhals-datafusion[extra-functions]"

Without the extra those three methods raise NotImplementedError and everything else works. The shim's FFI ABI is tied to the datafusion major it was compiled against (currently 54), which the wheel's own dependency pin enforces.

API coverage

Status of the narwhals public API on this backend (narwhals==2.25, datafusion==54). ⚠️ entries work with the caveat in parentheses; details in Known limitations.

Not listed: methods narwhals itself doesn't support on any lazy/SQL backend (Expr.filter, Expr.drop_nulls, Expr.unique, Expr.map_batches, Expr.ewm_mean, LazyFrame.tail, LazyFrame.gather_every).

Known limitations (as of datafusion 54)

• join_asof, exact quantile, cum_prod, list.sum/mean/median, dt.total_*, dt.offset_by, dt.timestamp, str.replace, Enum casts — no engine support; raise NotImplementedError.
• mode, skew, kurtosis need the extra-functions extra (see above); without it they raise NotImplementedError naming it.
• n_unique().over(...) raises: DataFusion silently ignores DISTINCT inside window aggregates, which would return wrong results. (The same engine quirk drops ORDER BY/IGNORE NULLS declared inside window aggregates — this backend moves those modifiers onto the window itself.)
• fill_null(strategy=..., limit=n) raises: bounded window frames with first_value/last_value need retract_batch, unimplemented engine-side.
• replace_time_zone supports None (strip) and "UTC" only; use convert_time_zone for instant-preserving conversions.
• str.to_datetime/to_date require an explicit format.
• replace_strict requires an explicit default.
• nw.scan_csv/nw.scan_parquet cannot dispatch to a plugin backend yet (narwhals gap); read with a DataFusion SessionContext and pass the frame to nw.from_native.
• str.to_titlecase uses initcap, which doesn't break words on digits.
• No row-order guarantees except after sort (standard for SQL engines): concat may interleave its inputs and backward fill_null may reorder rows. Sort explicitly when order matters.

Development

git submodule update --init      # narwhals, pinned to the targeted release
uv sync --group tests --extra extra-functions

See CONTRIBUTING.md for the full workflow.

Running narwhals' own test suite against this backend

The narwhals repo is vendored as a git submodule pinned to the targeted release:

git submodule update --init
uv run --group tests python run_tests.py    # known failures deselected

For the full, unfiltered run:

uv run --group tests pytest narwhals/tests -c narwhals/pyproject.toml \
    -p narwhals_datafusion.testing -p env --use-external-constructor

Regenerate the deselect list after fixing tests or bumping the submodule with uv run --group tests python update_run_tests.py.