narwhals-datafusion

Apache DataFusion backend for Narwhals, implemented as an out-of-tree plugin via the narwhals.plugins entry-point system.

Usage

import narwhals as nw
import pyarrow as pa
from datafusion import SessionContext

ctx = SessionContext()
df = ctx.from_arrow(pa.table({"a": [1, 2, 3], "b": ["x", "y", "x"]}))

lf = nw.from_native(df)  # -> nw.LazyFrame, dispatched to this plugin
result = (
    lf.group_by("b")
    .agg(nw.col("a").sum())
    .sort("b")
    .collect(backend="pyarrow")
)

Everything stays lazy until .collect(): narwhals expressions are translated to datafusion.Expr, frame verbs to datafusion.DataFrame methods, and the plan executes in DataFusion's Rust engine. Dtypes are pyarrow end-to-end.

Architecture

The backend sits on narwhals' shared SQL layer (narwhals._sql), the same abstraction DuckDB, Ibis, and Spark use. It targets narwhals==2.25 internals (narwhals._sql, narwhals._compliant), vendored as the narwhals/ git submodule pinned to that release. The plugin provides:

mode, skew, kurtosis via datafusion-extra-functions

DataFusion core deliberately keeps its function library lean, so aggregates like mode/skewness/kurtosis live in the contrib datafusion-extra-functions crate (Rust-only, no wheel on PyPI). The datafusion-extra-functions-ffi package — a required dependency, installed as a prebuilt wheel — exposes its aggregate UDFs to datafusion-python via the __datafusion_aggregate_udf__ PyCapsule protocol; they back Expr.mode, Expr.skew, and Expr.kurtosis. Its FFI ABI is tied to the datafusion major it was compiled against (currently 54), which the wheel's own dependency pin enforces.

API coverage

Status of the narwhals public API on this backend (narwhals==2.25, datafusion==54). ⚠️ entries work with the caveat in parentheses; details in Known limitations.

Not listed: methods narwhals itself doesn't support on any lazy/SQL backend (Expr.filter, Expr.drop_nulls, Expr.unique, Expr.map_batches, Expr.ewm_mean, LazyFrame.tail, LazyFrame.gather_every).

Known limitations (as of datafusion 54)

• join_asof, exact quantile, cum_prod, list.sum/mean/median, dt.total_*, dt.offset_by, dt.timestamp, str.replace, Enum casts — no engine support; raise NotImplementedError.
• n_unique().over(...) raises: DataFusion silently ignores DISTINCT inside window aggregates, which would return wrong results. (The same engine quirk drops ORDER BY/IGNORE NULLS declared inside window aggregates — this backend moves those modifiers onto the window itself.)
• fill_null(strategy=..., limit=n) raises: bounded window frames with first_value/last_value need retract_batch, unimplemented engine-side.
• replace_time_zone supports None (strip) and "UTC" only; use convert_time_zone for instant-preserving conversions.
• str.to_datetime/to_date require an explicit format; str.to_time parses "HH:MM:SS"-style strings via Arrow's cast, ignoring custom formats.
• replace_strict requires an explicit default.
• str.to_titlecase uses initcap, which doesn't break words on digits.
• No row-order guarantees except after sort (standard for SQL engines); backward fill_null may reorder rows.

Development

git submodule update --init      # narwhals, pinned to the targeted release
uv sync --group tests

Running narwhals' own test suite against this backend

The narwhals repo is vendored as a git submodule pinned to the targeted release:

git submodule update --init
uv run --group tests python run_tests.py    # known failures deselected

For the full, unfiltered run:

uv run --group tests pytest narwhals/tests -c narwhals/pyproject.toml \
    -p narwhals_datafusion.testing -p env --use-external-constructor

Regenerate the deselect list after fixing tests or bumping the submodule with uv run --group tests python update_run_tests.py.