Skip to main content

Native Delta Lake Python binding based on delta-rs with Pandas integration

Project description

Deltalake-python

PyPI userdoc apidoc

Native Delta Lake Python binding based on delta-rs with Pandas integration.

Example

from deltalake import DeltaTable
dt = DeltaTable("../rust/tests/data/delta-0.2.0")
dt.version()
3
dt.files()
['part-00000-cb6b150b-30b8-4662-ad28-ff32ddab96d2-c000.snappy.parquet',
 'part-00000-7c2deba3-1994-4fb8-bc07-d46c948aa415-c000.snappy.parquet',
 'part-00001-c373a5bd-85f0-4758-815e-7eb62007a15c-c000.snappy.parquet']

See the user guide for more examples.

Installation

pip install deltalake

NOTE: official binary wheels are linked against openssl statically for remote objection store communication. Please file Github issue to request for critical openssl upgrade.

Build custom wheels

Sometimes you may wish to build custom wheels. Maybe you want to try out some unreleased features. Or maybe you want to tweak the optimization of the Rust code.

To compile the package, you will need the Rust compiler and maturin:

curl https://sh.rustup.rs -sSf | sh -s
pip install maturin

Then you can build wheels for your own platform like so:

maturin build --release --out wheels

For a build that is optimized for the system you are on (but sacrificing portability):

RUSTFLAGS="-C target-cpu=native" maturin build --release --out wheels

Cross compilation

The above command only works for your current platform. To create wheels for other platforms, you'll need to cross compile. Cross compilation requires installing two additional components: to cross compile Rust code, you will need to install the target with rustup; to cross compile the Python bindings, you will need to install ziglang.

The following example is for manylinux2014. Other targets will require different Rust target and Python compatibility tags.

rustup target add x86_64-unknown-linux-gnu
pip install ziglang

Then you can build the wheel with:

maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux2014 \
    --out wheels

If you expect to only run on more modern system, you can set a newer target-cpu flag to Rust and use a newer compatibility tag for Linux. For example, here we set compatibility with CPUs newer than Haswell (2013) and Linux OS with glibc version of at least 2.24:

RUSTFLAGS="-C target-cpu=haswell" maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux_2_24 \
    --out wheels

See note about RUSTFLAGS from the arrow-rs readme.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

deltalake-0.9.0.tar.gz (4.5 MB view hashes)

Uploaded Source

Built Distributions

deltalake-0.9.0-cp37-abi3-win_amd64.whl (19.9 MB view hashes)

Uploaded CPython 3.7+ Windows x86-64

deltalake-0.9.0-cp37-abi3-macosx_11_0_arm64.whl (17.7 MB view hashes)

Uploaded CPython 3.7+ macOS 11.0+ ARM64

deltalake-0.9.0-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (37.2 MB view hashes)

Uploaded CPython 3.7+ macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

deltalake-0.9.0-cp37-abi3-macosx_10_7_x86_64.whl (19.6 MB view hashes)

Uploaded CPython 3.7+ macOS 10.7+ x86-64

deltalake-0.9.0-1-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (29.8 MB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ x86-64

deltalake-0.9.0-1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (29.2 MB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ARM64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page