Delta Funnel

Surreal banner showing Delta Lake data flowing through a Rust-orange funnel into a database barrel.

Fast, lightweight Delta Lake to SQL Server loads without Spark or ODBC.

A lightweight Rust and Python toolkit for reading Delta Lake tables,
transforming them with DataFusion SQL, and writing through native TDS bulk loads.

Built in Rust. Python API included.
Read the Delta Funnel documentation.

Rust docs crates.io PyPI Python 3.10+

[!NOTE] Delta Funnel is early project code. The Rust crate is available on crates.io, and the Python package is available on PyPI.

When To Use It

Use Delta Funnel when you need to:

  • Read Delta Lake tables from local paths or object-store URIs.
  • Transform rows with DataFusion SQL.
  • Load one or more results into Microsoft SQL Server.
  • Use native TDS bulk writes designed to be significantly faster than ODBC-based loads.
  • Run the workflow from Rust or from a PyO3 native extension module in Python.
  • Avoid standing up Spark for a focused Delta Lake to SQL Server pipeline.

Install Or Build

For Rust, add the delta-funnel crate:

cargo add delta-funnel

For Python, add the deltafunnel package:

uv add deltafunnel

Python Quickstart

from deltafunnel import Session

connection_string = (
    "server=tcp:localhost,1433;"
    "database=warehouse;"
    "user id=etl_user;"
    "password=REPLACE_ME;"
    "encrypt=true;"
    "TrustServerCertificate=yes"
)

session = Session(default_mssql_connection_string=connection_string)
orders = session.delta_lake("file:///path/to/orders-delta", name="orders")

daily_orders = session.table_from_sql("""
    select customer_id, order_date, total_amount
    from orders
    where order_date >= date '2026-01-01'
""")

report = daily_orders.write_to_mssql(
    schema="dbo",
    table="daily_orders",
    load_mode="create_and_load",
)

Use replace only for existing targets. DeltaFunnel writes a staging table, validates it, then swaps it into the final target name. Table metadata such as indexes, constraints, triggers, permissions, and extended properties is not preserved.

session.delta_lake(..., name="orders") registers a Delta source immediately. session.delta_lake(...) without name returns a pending source; call .alias("orders") before SQL references it.

For private S3 Delta sources in Python, see the docs-site/docs/python-api-walkthrough.md guide. The current S3 path expects explicit storage_options credentials for local shell usage and can behave differently from deltalake if the two libraries resolve AWS credentials differently.

Reports are plain Python dict values converted from Rust report types. Report formatting is designed to avoid exposing connection strings, credentials, and raw row values. See docs/failure-reports-and-tracing.md for the failure-report and tracing rules.

Dry Runs

Use dry_run=True on the same write methods to validate the plan without writing rows:

dry_run_report = daily_orders.write_to_mssql(
    schema="dbo",
    table="daily_orders",
    load_mode="create_and_load",
    dry_run=True,
)

There are no public Python dry_run_* methods.

Multi-output Writes

Table.to_mssql(...) creates an output spec without writing. Session.write_all writes the specs in one workflow.

active_orders = session.table_from_sql("""
    select *
    from orders
    where status = 'active'
""").alias("active_orders")

west = session.table_from_sql("""
    select * from active_orders where region = 'west'
""")
east = session.table_from_sql("""
    select * from active_orders where region = 'east'
""")

outputs = [
    west.to_mssql(
        schema="dbo",
        table="active_orders_west",
        load_mode="append_existing",
        name="west_active_orders",
    ),
    east.to_mssql(
        schema="dbo",
        table="active_orders_east",
        load_mode="append_existing",
        name="east_active_orders",
    ),
]

dry_run_report = session.write_all(outputs, dry_run=True)
report = session.write_all(outputs, options={"cache_mode": "disabled"})

options={"cache_mode": "auto"} is the default execute behavior. It may cache shared lazy SQL aliases during one write_all call. Use options={"cache_mode": "disabled"} to force the baseline path.

[!IMPORTANT] options is only accepted for execute write_all calls, not dry runs.

The first Python surface does not include persistent cache, persist, or materialize APIs.

Rust API

The Rust crate owns the workflow implementation and public report types. A minimal dry-run example is available at crates/delta-funnel/examples/query_load_dry_run.rs.

Run it with a local Delta table path:

DELTA_FUNNEL_EXAMPLE_ORDERS_DELTA=/path/to/orders \
  cargo run -p delta-funnel --example query_load_dry_run

Core Rust entry points include:

  • DeltaFunnelSession for source registration and session state.
  • DeltaFunnelRuntime for lazy SQL planning, dry runs, and writes.
  • OutputWritePlan and MssqlOutputTarget for output planning.
  • WriteAllOptions and WriteAllCacheMode for multi-output execution.

Build And Test

cargo fmt --all --check
cargo check --workspace
cargo test --workspace
cargo clippy --workspace --all-targets --all-features -- -D warnings

SQL Server integration tests are opt-in:

cargo xtask sqlserver-test

The xtask runner can start a local SQL Server container, run Rust and Python write tests, and remove the container when it exits. See docs/mssql-integration-tests.md.

Project links

Data verified by PyPI on Jul 8, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.6.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mag1cfrog from gravatar.com mag1cfrog

GitHub Statistics

Data verified by PyPI on Jul 8, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.10

Tags

delta-lake sql-server datafusion tds arrow

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

deltafunnel-0.2.1-cp310-abi3-win_amd64.whl (41.3 MB view details)

Uploaded CPython 3.10+Windows x86-64

deltafunnel-0.2.1-cp310-abi3-manylinux_2_28_x86_64.whl (49.0 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ x86-64

deltafunnel-0.2.1-cp310-abi3-macosx_11_0_arm64.whl (42.0 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

deltafunnel-0.2.1-cp310-abi3-macosx_10_12_x86_64.whl (44.1 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file deltafunnel-0.2.1-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: deltafunnel-0.2.1-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 41.3 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for deltafunnel-0.2.1-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 16271663caaa3437820aaf58ab633dfae9aa5554ecba694dc8ffb80dd3789ffc
MD5 d1eb4584436fc6d4068072afaf22b2f4
BLAKE2b-256 1dcf92eb3174b7bd94413df95cb5e45845c6b731dc28678d5a53a93b9334e6fc

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltafunnel-0.2.1-cp310-abi3-win_amd64.whl:

Publisher: pypi.yml on mag1cfrog/delta-funnel

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file deltafunnel-0.2.1-cp310-abi3-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for deltafunnel-0.2.1-cp310-abi3-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 07c0a6f288d96d04a64a1ce2ec33a1512b28b0356972fdeeae80a01f3605ceaa
MD5 a32710584b1fad20d17cf9ecabac885c
BLAKE2b-256 9fd6be19dfb5cc7ebdfcccbf5c6acf3809bef2cbea65c84f1416eb004fdaf806

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltafunnel-0.2.1-cp310-abi3-manylinux_2_28_x86_64.whl:

Publisher: pypi.yml on mag1cfrog/delta-funnel

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file deltafunnel-0.2.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deltafunnel-0.2.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 5abea29082ae05f30564c1b30ddf10c5e003012c3f2bb496c54828fef1ae8297
MD5 d51b1b3b07b554d036073eb1d48d9809
BLAKE2b-256 9e5704fda6ab9595da37bb8cfe1fce93a3502ddb680f448fc193e8bb9686dc3f

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltafunnel-0.2.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: pypi.yml on mag1cfrog/delta-funnel

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file deltafunnel-0.2.1-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for deltafunnel-0.2.1-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 04860e063dfc24ced3e4fa4fb06c858505cbe4973b3d0a9cb6fe5c204f4f17a1
MD5 a87cd84211c58e671524eff0be585e27
BLAKE2b-256 2c50d8f51a31f07bd5f5dd6c9489ac9fc5efe5a2f0a30df0a79d0eaad35c2801

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltafunnel-0.2.1-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: pypi.yml on mag1cfrog/delta-funnel

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.6.0

4 files

0.5.0

4 files

0.4.1

4 files

0.4.0

4 files

0.3.3

4 files

0.3.2

4 files

0.3.1

4 files

0.3.0

4 files

This release

0.2.1 This release

4 files

0.2.0

4 files

0.1.6

4 files

0.1.5

4 files

0.1.4

4 files

0.1.3

4 files

0.1.2

4 files

0.1.1

4 files

0.1.0

4 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page