Skip to main content

DataFusion in Python

Python test Python Release Build

This is a Python library that binds to Apache Arrow in-memory query engine DataFusion.

DataFusion's Python bindings can be used as an end-user tool as well as providing a foundation for building new systems.

Features

  • Execute queries using SQL or DataFrames against CSV, Parquet, and JSON data sources
  • Queries are optimized using DataFusion's query optimizer
  • Execute user-defined Python code from SQL
  • Exchange data with Pandas and other DataFrame libraries that support PyArrow
  • Serialize and deserialize query plans in Substrait format
  • Experimental support for executing SQL queries against Polars, Pandas and cuDF

Comparison with other projects

Here is a comparison with similar projects that may help understand when DataFusion might be suitable and unsuitable for your needs:

  • DuckDB is an open source, in-process analytic database. Like DataFusion, it supports very fast execution, both from its custom file format and directly from Parquet files. Unlike DataFusion, it is written in C/C++ and it is primarily used directly by users as a serverless database and query system rather than as a library for building such database systems.

  • Polars is one of the fastest DataFrame libraries at the time of writing. Like DataFusion, it is also written in Rust and uses the Apache Arrow memory model, but unlike DataFusion it does not provide full SQL support, nor as many extension points.

Example Usage

The following example demonstrates running a SQL query against a Parquet file using DataFusion, storing the results in a Pandas DataFrame, and then plotting a chart.

The Parquet file used in this example can be downloaded from the following page:

from datafusion import SessionContext

# Create a DataFusion context
ctx = SessionContext()

# Register table with context
ctx.register_parquet('taxi', 'yellow_tripdata_2021-01.parquet')

# Execute SQL
df = ctx.sql("select passenger_count, count(*) "
             "from taxi "
             "where passenger_count is not null "
             "group by passenger_count "
             "order by passenger_count")

# convert to Pandas
pandas_df = df.to_pandas()

# create a chart
fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()
fig.savefig('chart.png')

This produces the following chart:

Chart

More Examples

See examples for more information.

Executing Queries with DataFusion

Running User-Defined Python Code

Substrait Support

Executing SQL against DataFrame Libraries (Experimental)

How to install (from pip)

Pip

pip install datafusion
# or
python -m pip install datafusion

Conda

conda install -c conda-forge datafusion

You can verify the installation by running:

>>> import datafusion
>>> datafusion.__version__
'0.6.0'

How to develop

This assumes that you have rust and cargo installed. We use the workflow recommended by pyo3 and maturin.

The Maturin tools used in this workflow can be installed either via Conda or Pip. Both approaches should offer the same experience. Multiple approaches are only offered to appease developer preference. Bootstrapping for both Conda and Pip are as follows.

Bootstrap (Conda):

# fetch this repo
git clone git@github.com:apache/arrow-datafusion-python.git
# create the conda environment for dev
conda env create -f ./conda/environments/datafusion-dev.yaml -n datafusion-dev
# activate the conda environment
conda activate datafusion-dev

Bootstrap (Pip):

# fetch this repo
git clone git@github.com:apache/arrow-datafusion-python.git
# prepare development environment (used to build wheel / install in development)
python3 -m venv venv
# activate the venv
source venv/bin/activate
# update pip itself if necessary
python -m pip install -U pip
# install dependencies (for Python 3.8+)
python -m pip install -r requirements-310.txt

The tests rely on test data in git submodules.

git submodule init
git submodule update

Whenever rust code changes (your changes or via git pull):

# make sure you activate the venv using "source venv/bin/activate" first
maturin develop
python -m pytest

How to update dependencies

To change test dependencies, change the requirements.in and run

# install pip-tools (this can be done only once), also consider running in venv
python -m pip install pip-tools
python -m piptools compile --generate-hashes -o requirements-310.txt

To update dependencies, run with -U

python -m piptools compile -U --generate-hashes -o requirements-310.txt

More details here

Metadata

Release files for datafusion 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datafusion 0.8.0
File Size Uploaded Provenance Actions
datafusion-0.8.0.tar.gz 4.7 MB Without provenance Details for datafusion-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datafusion 0.8.0
File Interpreter ABI Platform Actions
datafusion-0.8.0-cp37-abi3-manylinux_2_31_x86_64.whl CPython 3.7 abi3 Linux glibc 2.31+ x86-64 Details for datafusion-0.8.0-cp37-abi3-manylinux_2_31_x86_64.whl

Total release size: 17.4 MB

Release files / datafusion-0.8.0.tar.gz

Download URL datafusion-0.8.0.tar.gz
Size 4.7 MB
Tags Source
SHA-256 checksum
How to use checksums
e4c94745a30276b7a3d0c3b18a37c8da6667f397fcf3bf5bfd874cb75419c5b7
BLAKE2b-256 checksum
How to use checksums
b4be8973bb99d71f9edfebdc8097e8cf0533fd289ec16733c8c700f85c8074e5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/0.14.10

Release files / datafusion-0.8.0-cp37-abi3-manylinux_2_31_x86_64.whl

Download URL datafusion-0.8.0-cp37-abi3-manylinux_2_31_x86_64.whl
Size 12.7 MB
Tags CPython 3.7 Linux glibc 2.31+ x86-64 abi3
SHA-256 checksum
How to use checksums
0917abd71a782849784e8533915491da70039163ac7360978f572900b5023fce
BLAKE2b-256 checksum
How to use checksums
f4fda377e4d998e07816227979a427733a541291ceae9628eb5ec2089a0f70f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/0.14.10

Security for datafusion 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.8.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page