Skip to main content

Hops-deltalake-python

PyPI userdoc apidoc

A HopsFS supported version of native Delta Lake Python binding based on delta-rs with Pandas integration.

Example

from deltalake import DeltaTable
dt = DeltaTable("../rust/tests/data/delta-0.2.0")
dt.version()
3
dt.file_uris()
['s3://bucket/table/part-00000-cb6b150b-30b8-4662-ad28-ff32ddab96d2-c000.snappy.parquet',
 's3://bucket/table/part-00000-7c2deba3-1994-4fb8-bc07-d46c948aa415-c000.snappy.parquet',
 's3://bucket/table/part-00001-c373a5bd-85f0-4758-815e-7eb62007a15c-c000.snappy.parquet']

See the user guide for more examples.

Compatibility Matrix

hops-deltalake hops-object-store HopsFS
1.1.2-post2 1.0.3 3.2.0.18
1.4.0 1.1.0 3.2.0.18-EE-RC1
1.4.0-post1 1.1.1 >= 3.2.0.18-EE-RC1 < 3.4.4.0-EE-RC0

Installation

# with pip
pip install hops-deltalake

NOTE: official binary wheels are linked against openssl statically for remote objection store communication. Please file Github issue to request for critical openssl upgrade.

Tracing and Observability

Delta-rs supports OpenTelemetry tracing for performance analysis and debugging.

Basic Example

import os
import deltalake
from deltalake import write_deltalake, DeltaTable

# Enable logging to see trace output in stdout
os.environ["RUST_LOG"] = "deltalake=debug"

# Initialize tracing (uses default HTTP endpoint or OTEL_EXPORTER_OTLP_ENDPOINT env var)
# For authentication, set OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key"
# The HTTP exporter automatically reads OTEL_EXPORTER_OTLP_HEADERS for API keys
deltalake.init_tracing()

# All Delta operations are now traced
write_deltalake("my_table", data)
dt = DeltaTable("my_table")
df = dt.to_pandas()

When you run this code, you'll see trace information in stdout showing operation timings and execution flow.

Build custom wheels

Sometimes you may wish to build custom wheels. Maybe you want to try out some unreleased features. Or maybe you want to tweak the optimization of the Rust code.

To compile the package, you will need the Rust compiler and maturin:

curl https://sh.rustup.rs -sSf | sh -s

Then you can build wheels for your own platform like so:

```sh
uvx maturin build --release --out wheels

Note:

  • uvx invokes a tool without installing it.
  • if you plan to often use maturin, you can install the "tool" with uv tool install maturin.

For a build that is optimized for the system you are on (but sacrificing portability):

RUSTFLAGS="-C target-cpu=native" uvx maturin build --release --out wheels

Cross compilation

The above command only works for your current platform. To create wheels for other platforms, you'll need to cross compile. Cross compilation requires installing two additional components: to cross compile Rust code, you will need to install the target with rustup; to cross compile the Python bindings, you will need to install ziglang.

The following example is for manylinux2014. Other targets will require different Rust target and Python compatibility tags.

rustup target add x86_64-unknown-linux-gnu

Then you can build wheels for the target platform like so:

uvx --from 'maturin[zig]' maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux2014 \
    --out wheels

If you expect to only run on more modern system, you can set a newer target-cpu flag to Rust and use a newer compatibility tag for Linux. For example, here we set compatibility with CPUs newer than Haswell (2013) and Linux OS with glibc version of at least 2.24:

RUSTFLAGS="-C target-cpu=haswell" uvx --from 'maturin[zig]' maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux_2_24 \
    --out wheels

See note about RUSTFLAGS from the arrow-rs readme.

Metadata

Release files for hops-deltalake 1.4.0.post122

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hops-deltalake 1.4.0.post122
File Size Uploaded Provenance Actions
hops_deltalake-1.4.0.post122.tar.gz 5.2 MB Without provenance Details for hops_deltalake-1.4.0.post122.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for hops-deltalake 1.4.0.post122
File Interpreter ABI Platform Actions
hops_deltalake-1.4.0.post122-cp310-abi3-manylinux_2_28_x86_64.whl CPython 3.10 abi3 Linux glibc 2.28+ x86-64 Details for hops_deltalake-1.4.0.post122-cp310-abi3-manylinux_2_28_x86_64.whl
hops_deltalake-1.4.0.post122-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for hops_deltalake-1.4.0.post122-cp310-abi3-macosx_11_0_arm64.whl

Total release size: 92.0 MB

Release files / hops_deltalake-1.4.0.post122.tar.gz

Download URL hops_deltalake-1.4.0.post122.tar.gz
Size 5.2 MB
Tags Source
SHA-256 checksum
How to use checksums
700d53a8fc7bcf8399f29d83d90b31067d3f507fa423c896020a518af726cdda
BLAKE2b-256 checksum
How to use checksums
ba828e7c74e3ba0d8e7e8ca9d2894fe2d2aedab2ce71cc901394653f933453bb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.6.0

Release files / hops_deltalake-1.4.0.post122-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL hops_deltalake-1.4.0.post122-cp310-abi3-manylinux_2_28_x86_64.whl
Size 47.4 MB
Tags CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
2c1697f810db986bca86d02307eb872d79621a7f2c0c63eb338a6835fec04470
BLAKE2b-256 checksum
How to use checksums
9f047e43784531da97740465adc05fbc9214204179a75763b17bf9142193903e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.6.0

Release files / hops_deltalake-1.4.0.post122-cp310-abi3-macosx_11_0_arm64.whl

Download URL hops_deltalake-1.4.0.post122-cp310-abi3-macosx_11_0_arm64.whl
Size 39.4 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
32e3d0ce7cd11e43b58f68bb21f0c59f5ba2e4f5e404eeaa96edff7bc016ebfa
BLAKE2b-256 checksum
How to use checksums
90bb2c48de4d2e40e906db90dde74d74eab57196fb8782414274a914009f9888
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.16.0

Security for hops-deltalake 1.4.0.post122

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page