No project description provided
Project description
Deltalake2DB
This is a simple project that uses Metadata from deltalake
package to provide methods to read Delta Lake Tables
to either Polars or DuckDB with better Protocol Support as the main deltalake
package.
Use with Duckdb
Install deltalake2db
and duckdb
using pip/poetry/whatever you use.
Then you can do like this:
from deltalake2db import get_sql_for_delta,
with duckdb.connect() as con:
dt = DeltaTable("tests/data/faker2")
sql = get_sql_for_delta(dt, duck_con=con) # get select statement
print(sql)
duckdb_create_view_for_delta(con, dt, "delta_table") # or let it create a view for you. will point to the data at this point in time
con.execute("select * from delta_table").fetch_all()
If you'd like to manipulate you can use get_sql_for_delta_expr
which returns a SqlGlot Object
Use with Polars
Install deltalake2db
and polars
using pip/poetry/whatever you use.
dt = DeltaTable("tests/data/faker2")
from deltalake2db import polars_scan_delta
lazy_df = polars_scan_delta(dt)
df = lazy_df.collect()
Protocol Support
- Column Mapping
- Almost Data Types, including Structs/Lists, Map yet to be done
- Test data types, including datetime
- Deletion Vectors
In case there is an unsupported DeltaLake Feature, this will just throw DeltaProtocolError
as does delta-rs
Cloud Support
For now, only az:// Url's for Azure are tested and supported in DuckDB. For polars it's a lot easier, since polars just uses object_store
create, so it should just work.
The package does some work to make DuckDB's "Azure Storage Options" work in Polars, to be able to use the same options.
This means you can:
- pass an absolute DuckDB-style Path to Polars, meaning something like
abfss://⟨my_storage_account⟩.dfs.core.windows.net/⟨my_filesystem⟩/⟨path⟩
- pass "chain" as option, which will act like DuckDB's Credential Chain. This requires
azure-identity
Package
Looking for something different? :)
We also have the following projects around deltalake:
- LakeAPI for providing deltalake Tables
- Odbc2deltalake to load MS SQL Server/ODBC Tables to Deltalake
Or projects from other people:
- polars-deltalake An experimental native polars deltalake reader
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file deltalake2db-0.5.2.tar.gz
.
File metadata
- Download URL: deltalake2db-0.5.2.tar.gz
- Upload date:
- Size: 11.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a85eef28a3b78e52d523f1095b8584f7b251df75db1719b65b09ff0ec61dac3 |
|
MD5 | 4f4e2b25f4fa159238d3ae5781d6ab97 |
|
BLAKE2b-256 | 91e188f93c43f627ef90239eab39de15582f8ab417de20ea93510bbfeb5f29f7 |
File details
Details for the file deltalake2db-0.5.2-py3-none-any.whl
.
File metadata
- Download URL: deltalake2db-0.5.2-py3-none-any.whl
- Upload date:
- Size: 14.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ea12e5a9a8c89686caf5b99b866c2a439407e3aa370482c6f664e43424ff4db |
|
MD5 | 42f6fdc4331841d71fbe0ece5e80ce8d |
|
BLAKE2b-256 | f4d4392de15ebdcb690a8643855d6a7f5d094f14e1eb9215aab61615af458779 |