Skip to main content

No project description provided

Project description

Deltalake2DB

This is a simple project that uses Metadata from deltalake package to provide methods to read Delta Lake Tables to either Polars or DuckDB with better Protocol Support as the main deltalake package.

Use with Duckdb

Install deltalake2db and duckdb using pip/poetry/whatever you use.

Then you can do like this:

from deltalake2db import get_sql_for_delta,

with duckdb.connect() as con:
    dt = DeltaTable("tests/data/faker2")
    sql = get_sql_for_delta(dt, duck_con=con) # get select statement
    print(sql)
    duckdb_create_view_for_delta(con, dt, "delta_table") # or let it create a view for you. will point to the data at this point in time

    con.execute("select * from delta_table").fetch_all()

If you'd like to manipulate you can use get_sql_for_delta_expr which returns a SqlGlot Object

Use with Polars

Install deltalake2db and polars using pip/poetry/whatever you use.

dt = DeltaTable("tests/data/faker2")
from deltalake2db import polars_scan_delta
lazy_df = polars_scan_delta(dt)
df = lazy_df.collect()

Protocol Support

  • Column Mapping
  • Almost Data Types, including Structs/Lists, Map yet to be done
  • Test data types, including datetime
  • Deletion Vectors

In case there is an unsupported DeltaLake Feature, this will just throw DeltaProtocolError as does delta-rs

Cloud Support

For now, only az:// Url's for Azure are tested and supported in DuckDB. For polars it's a lot easier, since polars just uses object_store create, so it should just work.

Looking for something different? :)

We also have the following projects around deltalake:

Or projects from other people:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

deltalake2db-0.2.1.tar.gz (8.0 kB view details)

Uploaded Source

Built Distribution

deltalake2db-0.2.1-py3-none-any.whl (8.9 kB view details)

Uploaded Python 3

File details

Details for the file deltalake2db-0.2.1.tar.gz.

File metadata

  • Download URL: deltalake2db-0.2.1.tar.gz
  • Upload date:
  • Size: 8.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for deltalake2db-0.2.1.tar.gz
Algorithm Hash digest
SHA256 51f6a3c00146aee43e50adc13b5e4b3548d62e60a86f4bb2d009d62e86150fe5
MD5 8a124948233bd44ebeaefb20729f3b9d
BLAKE2b-256 d8d9834b9810b97bedb48df36069888291a8230e675867109ab85cd9b78894d4

See more details on using hashes here.

File details

Details for the file deltalake2db-0.2.1-py3-none-any.whl.

File metadata

  • Download URL: deltalake2db-0.2.1-py3-none-any.whl
  • Upload date:
  • Size: 8.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for deltalake2db-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 713ecaa5bafa2e54388af6db5a9dc08b13c0eafa0a41fe15cabed89f412e9bb0
MD5 daf3ffafaffd44298ac1e3e822129c6c
BLAKE2b-256 a5d8589c31a025f191a52f9332487eff77ac6310b74f3634d45d1be02b99da0f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page