Skip to main content

DataFusion in Python

Python test Python Release Build

This is a Python library that binds to Apache Arrow in-memory query engine DataFusion.

DataFusion's Python bindings can be used as an end-user tool as well as providing a foundation for building new systems.

Features

  • Execute queries using SQL or DataFrames against CSV, Parquet, and JSON data sources.
  • Queries are optimized using DataFusion's query optimizer.
  • Execute user-defined Python code from SQL.
  • Exchange data with Pandas and other DataFrame libraries that support PyArrow.
  • Serialize and deserialize query plans in Substrait format.
  • Experimental support for transpiling SQL queries to DataFrame calls with Polars, Pandas, and cuDF.

Comparison with other projects

Here is a comparison with similar projects that may help understand when DataFusion might be suitable and unsuitable for your needs:

  • DuckDB is an open source, in-process analytic database. Like DataFusion, it supports very fast execution, both from its custom file format and directly from Parquet files. Unlike DataFusion, it is written in C/C++ and it is primarily used directly by users as a serverless database and query system rather than as a library for building such database systems.

  • Polars is one of the fastest DataFrame libraries at the time of writing. Like DataFusion, it is also written in Rust and uses the Apache Arrow memory model, but unlike DataFusion it does not provide full SQL support, nor as many extension points.

Example Usage

The following example demonstrates running a SQL query against a Parquet file using DataFusion, storing the results in a Pandas DataFrame, and then plotting a chart.

The Parquet file used in this example can be downloaded from the following page:

from datafusion import SessionContext

# Create a DataFusion context
ctx = SessionContext()

# Register table with context
ctx.register_parquet('taxi', 'yellow_tripdata_2021-01.parquet')

# Execute SQL
df = ctx.sql("select passenger_count, count(*) "
             "from taxi "
             "where passenger_count is not null "
             "group by passenger_count "
             "order by passenger_count")

# convert to Pandas
pandas_df = df.to_pandas()

# create a chart
fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()
fig.savefig('chart.png')

This produces the following chart:

Chart

Configuration

It is possible to configure runtime (memory and disk settings) and configuration settings when creating a context.

runtime = (
    RuntimeConfig()
    .with_disk_manager_os()
    .with_fair_spill_pool(10000000)
)
config = (
    SessionConfig()
    .with_create_default_catalog_and_schema(True)
    .with_default_catalog_and_schema("foo", "bar")
    .with_target_partitions(8)
    .with_information_schema(True)
    .with_repartition_joins(False)
    .with_repartition_aggregations(False)
    .with_repartition_windows(False)
    .with_parquet_pruning(False)
    .set("datafusion.execution.parquet.pushdown_filters", "true")
)
ctx = SessionContext(config, runtime)

Refer to the API documentation for more information.

Printing the context will show the current configuration settings.

print(ctx)

More Examples

See examples for more information.

Executing Queries with DataFusion

Running User-Defined Python Code

Substrait Support

Executing SQL against DataFrame Libraries (Experimental)

How to install (from pip)

Pip

pip install datafusion
# or
python -m pip install datafusion

Conda

conda install -c conda-forge datafusion

You can verify the installation by running:

>>> import datafusion
>>> datafusion.__version__
'0.6.0'

How to develop

This assumes that you have rust and cargo installed. We use the workflow recommended by pyo3 and maturin.

The Maturin tools used in this workflow can be installed either via Conda or Pip. Both approaches should offer the same experience. Multiple approaches are only offered to appease developer preference. Bootstrapping for both Conda and Pip are as follows.

Bootstrap (Conda):

# fetch this repo
git clone git@github.com:apache/arrow-datafusion-python.git
# create the conda environment for dev
conda env create -f ./conda/environments/datafusion-dev.yaml -n datafusion-dev
# activate the conda environment
conda activate datafusion-dev

Bootstrap (Pip):

# fetch this repo
git clone git@github.com:apache/arrow-datafusion-python.git
# prepare development environment (used to build wheel / install in development)
python3 -m venv venv
# activate the venv
source venv/bin/activate
# update pip itself if necessary
python -m pip install -U pip
# install dependencies (for Python 3.8+)
python -m pip install -r requirements-310.txt

The tests rely on test data in git submodules.

git submodule init
git submodule update

Whenever rust code changes (your changes or via git pull):

# make sure you activate the venv using "source venv/bin/activate" first
maturin develop
python -m pytest

Running & Installing pre-commit hooks

arrow-datafusion-python takes advantage of pre-commit to assist developers with code linting to help reduce the number of commits that ultimately fail in CI due to linter errors. Using the pre-commit hooks is optional for the developer but certainly helpful for keeping PRs clean and concise.

Our pre-commit hooks can be installed by running pre-commit install, which will install the configurations in your ARROW_DATAFUSION_PYTHON_ROOT/.github directory and run each time you perform a commit, failing to complete the commit if an offending lint is found allowing you to make changes locally before pushing.

The pre-commit hooks can also be run adhoc without installing them by simply running pre-commit run --all-files

How to update dependencies

To change test dependencies, change the requirements.in and run

# install pip-tools (this can be done only once), also consider running in venv
python -m pip install pip-tools
python -m piptools compile --generate-hashes -o requirements-310.txt

To update dependencies, run with -U

python -m piptools compile -U --generate-hashes -o requirements-310.txt

More details here

Metadata

Release files for datafusion 35.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datafusion 35.0.0
File Size Uploaded Provenance Actions
datafusion-35.0.0.tar.gz 107.7 kB Without provenance Details for datafusion-35.0.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for datafusion 35.0.0
File Actions
datafusion-35.0.0-cp38-abi3-win_amd64.whl CPython 3.8 abi3 Windows x86-64 Details for datafusion-35.0.0-cp38-abi3-win_amd64.whl
datafusion-35.0.0-cp38-abi3-manylinux_2_28_aarch64.whl CPython 3.8 abi3 Linux glibc 2.28+ ARM64 Details for datafusion-35.0.0-cp38-abi3-manylinux_2_28_aarch64.whl
datafusion-35.0.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.8 abi3 Linux glibc 2.17+ x86-64 Details for datafusion-35.0.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
datafusion-35.0.0-cp38-abi3-macosx_11_0_arm64.whl CPython 3.8 abi3 macOS 11.0+ ARM64 Details for datafusion-35.0.0-cp38-abi3-macosx_11_0_arm64.whl
datafusion-35.0.0-cp38-abi3-macosx_10_12_x86_64.whl CPython 3.8 abi3 macOS 10.12+ x86-64 Details for datafusion-35.0.0-cp38-abi3-macosx_10_12_x86_64.whl

Total release size: 80.1 MB

Release files / datafusion-35.0.0.tar.gz

Download URL datafusion-35.0.0.tar.gz
Size 107.7 kB
Tags Source
SHA-256 checksum
How to use checksums
467f8d8f963256a321031f405cc553c8aa27e3cab6402389eb10ddb2db55a40d
BLAKE2b-256 checksum
How to use checksums
a601a1ad24f37c3b5a1fd4f076197bd438dd91318f6932c286a7a9a4607345b0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.12

Release files / datafusion-35.0.0-cp38-abi3-win_amd64.whl

Download URL datafusion-35.0.0-cp38-abi3-win_amd64.whl
Size 16.1 MB
Tags CPython 3.8 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
9c66e2b258b705adbfd611b492698eb52b42f2bd123ff838e5cbe7353e54bd3d
BLAKE2b-256 checksum
How to use checksums
886353502c334c02a04b6cc24a39ccefe13445d5114bede7e9d70c0b980134e3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.12

Release files / datafusion-35.0.0-cp38-abi3-manylinux_2_28_aarch64.whl

Download URL datafusion-35.0.0-cp38-abi3-manylinux_2_28_aarch64.whl
Size 16.8 MB
Tags CPython 3.8 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
fb299b7550b5466f84463c3715605153c356f78fdd0c67ea630feb67631789d2
BLAKE2b-256 checksum
How to use checksums
ed6357f4e6aec694f324b64a1d0b6cc7dec5873e3493f09199629f84e0f8e532
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.12

Release files / datafusion-35.0.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL datafusion-35.0.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 17.7 MB
Tags CPython 3.8 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
ca943440ad61ac3f9cd894e5bd2266986809132076f951687ea40642b1bd1103
BLAKE2b-256 checksum
How to use checksums
fe13a3f561b56ff17eea0a6c1c2e75ecbc77b45d66567679137dbd08fc122b52
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.12

Release files / datafusion-35.0.0-cp38-abi3-macosx_11_0_arm64.whl

Download URL datafusion-35.0.0-cp38-abi3-macosx_11_0_arm64.whl
Size 14.1 MB
Tags CPython 3.8 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
e817d61c82d93fc0f64334b86045e1fcdae39f9a84b2afb82c5b66d7f6b9b85d
BLAKE2b-256 checksum
How to use checksums
0d5da1916dbe695961fa15829f01f2d4dfc95a3da741123613146d18d656758c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.12

Release files / datafusion-35.0.0-cp38-abi3-macosx_10_12_x86_64.whl

Download URL datafusion-35.0.0-cp38-abi3-macosx_10_12_x86_64.whl
Size 15.2 MB
Tags CPython 3.8 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
8e460dff1f6588632273629145d53d77878b2969b6fbeccec034a9404bed8dbe
BLAKE2b-256 checksum
How to use checksums
92c5baba4f321c75901c6ef3680c633389c6012b0d1967bc133e14ed40c26a95
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.12

Security for datafusion 35.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

35.0.0 This release

6 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page