Skip to main content

Native Delta Lake Python binding based on delta-rs with Pandas integration

Project description

Deltalake-python

PyPI userdoc apidoc

Native Delta Lake Python binding based on delta-rs with Pandas integration.

Example

from deltalake import DeltaTable
dt = DeltaTable("../rust/tests/data/delta-0.2.0")
dt.version()
3
dt.files()
['part-00000-cb6b150b-30b8-4662-ad28-ff32ddab96d2-c000.snappy.parquet',
 'part-00000-7c2deba3-1994-4fb8-bc07-d46c948aa415-c000.snappy.parquet',
 'part-00001-c373a5bd-85f0-4758-815e-7eb62007a15c-c000.snappy.parquet']

See the user guide for more examples.

Installation

pip install deltalake

NOTE: official binary wheels are linked against openssl statically for remote objection store communication. Please file Github issue to request for critical openssl upgrade.

Build custom wheels

Sometimes you may wish to build custom wheels. Maybe you want to try out some unreleased features. Or maybe you want to tweak the optimization of the Rust code.

To compile the package, you will need the Rust compiler and maturin:

curl https://sh.rustup.rs -sSf | sh -s
pip install maturin

Then you can build wheels for your own platform like so:

maturin build --release --out wheels

For a build that is optimized for the system you are on (but sacrificing portability):

RUSTFLAGS="-C target-cpu=native" maturin build --release --out wheels

Cross compilation

The above command only works for your current platform. To create wheels for other platforms, you'll need to cross compile. Cross compilation requires installing two additional components: to cross compile Rust code, you will need to install the target with rustup; to cross compile the Python bindings, you will need to install ziglang.

The following example is for manylinux2014. Other targets will require different Rust target and Python compatibility tags.

rustup target add x86_64-unknown-linux-gnu
pip install ziglang

Then you can build the wheel with:

maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux2014 \
    --out wheels

If you expect to only run on more modern system, you can set a newer target-cpu flag to Rust and use a newer compatibility tag for Linux. For example, here we set compatibility with CPUs newer than Haswell (2013) and Linux OS with glibc version of at least 2.24:

RUSTFLAGS="-C target-cpu=haswell" maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux_2_24 \
    --out wheels

See note about RUSTFLAGS from the arrow-rs readme.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

deltalake-0.18.1.tar.gz (4.8 MB view details)

Uploaded Source

Built Distributions

deltalake-0.18.1-cp38-abi3-win_amd64.whl (26.2 MB view details)

Uploaded CPython 3.8+Windows x86-64

deltalake-0.18.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28.4 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

deltalake-0.18.1-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (26.5 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ARM64

deltalake-0.18.1-cp38-abi3-macosx_11_0_arm64.whl (23.3 MB view details)

Uploaded CPython 3.8+macOS 11.0+ ARM64

deltalake-0.18.1-cp38-abi3-macosx_10_12_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.8+macOS 10.12+ x86-64

File details

Details for the file deltalake-0.18.1.tar.gz.

File metadata

  • Download URL: deltalake-0.18.1.tar.gz
  • Upload date:
  • Size: 4.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.6.0

File hashes

Hashes for deltalake-0.18.1.tar.gz
Algorithm Hash digest
SHA256 aa49822a4d559d138aee5bae3e529b231dd2dc2f1fcc6272f32853c99597c867
MD5 39418ef3f9365d266eb317782ba8c84e
BLAKE2b-256 6a89ee26102ca424575642871cef243286500b7fac8895b3975f0d92c615ad6c

See more details on using hashes here.

File details

Details for the file deltalake-0.18.1-cp38-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for deltalake-0.18.1-cp38-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 b8d7648fac6b4f0165173b516b3f43c0ae550bce6d6492d6016c3700c52dcb31
MD5 b04c55ff7884a8c98f2da1196d6bebd6
BLAKE2b-256 e53b7b18aa671e07b970838e72aeb49e3afee76e9762f0872f2419ee28fafcf3

See more details on using hashes here.

File details

Details for the file deltalake-0.18.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deltalake-0.18.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 aa049c74979e7905c6631d9df6d419cbaf37cd07de3f0b3e3733d3f835930f5f
MD5 4994dd183ced8426aa76cb3629177a90
BLAKE2b-256 e501a91fe938d1b0fa3e096f2bf643a73be03e69ac4ac16bf552d6cdf9778cdd

See more details on using hashes here.

File details

Details for the file deltalake-0.18.1-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deltalake-0.18.1-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 52c636137e77172b6ace4c916857678e2e88a4782cfac7a21e7fc4f0f45519b1
MD5 9527648c549f39d07b90ff9611cb9308
BLAKE2b-256 6baffc020bc745b03c9276069c72b056cda2e86029b4f1762e8f28b99ecd4ed5

See more details on using hashes here.

File details

Details for the file deltalake-0.18.1-cp38-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deltalake-0.18.1-cp38-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 74b89e3b8cdfdd23d0f17465ab008b9f19882dc7497c6a8ea3d78d8be2bf893f
MD5 64ac20aef88386e4d3475aedc2a48f8a
BLAKE2b-256 96965383f2a229516e83d374f02e4cc705f0589718b48bdca6de159f301b819c

See more details on using hashes here.

File details

Details for the file deltalake-0.18.1-cp38-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for deltalake-0.18.1-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 eea574f2f755a8378d75d33461d30ae562aad9d511f567aec6000904f67d13b4
MD5 df9119ebe3f9af7d1764cac978cc648f
BLAKE2b-256 2f035c3269bd6dec2cbdfcfee0f1534839966041efe595ef3a0906efcc39e9ba

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page