Delta Funnel

Surreal banner showing Delta Lake data flowing through a Rust-orange funnel into a database barrel.

Delta Lake to SQL Server. No Spark. No JDBC/ODBC bottleneck.

DataFusion SQL in.
Native TDS bulk load out.

Observed: 13.4M rows in ~14 minutes vs. a ~2 hour Spark/JDBC path.
Read the Delta Funnel documentation.

Rust docs crates.io PyPI Python 3.10+

[!NOTE] Delta Funnel is early project code. The Rust crate is available on crates.io, and the Python package is available on PyPI.

When To Use It

Use Delta Funnel when:

  • SQL Server writes are the bottleneck.
  • Spark is too much machinery for a focused export.
  • You want SQL transforms over Delta Lake without a cluster.
  • You want Rust or Python orchestration with reports and tracing.

Install

For Rust, add the delta-funnel crate:

cargo add delta-funnel

For Python, add the deltafunnel package:

uv add deltafunnel

Python Quickstart

from deltafunnel import Session

ado_connection_string = (
    "server=tcp:localhost,1433;"
    "database=warehouse;"
    "User ID=etl_user;"
    "Password=REPLACE_ME;"
    "encrypt=true;"
    "TrustServerCertificate=yes"
)

session = Session(default_mssql_connection_string=ado_connection_string)

# Register the Delta table as "orders" so SQL can reference it.
orders = session.delta_lake("file:///path/to/orders-delta", name="orders")

# Build a lazy DataFusion SQL query. No rows are read yet.
daily_orders = session.table_from_sql("""
    select customer_id, order_date, total_amount
    from orders
    where order_date >= date '2026-01-01'
""")

# Preview executes the DataFusion query with a limit; notebooks render it as a table.
daily_orders.preview(limit=20)

Synthetic Delta Funnel table preview showing customer_id, order_date, and total_amount rows.

# Write executes the query and loads the result into SQL Server.
report = daily_orders.write_to_mssql(
    schema="dbo",
    table="daily_orders",
    load_mode="create_and_load",  # "replace" also supports a missing target
    # dry_run=True,  # validate the load plan without writing rows
)

For private S3 sources, SQL Server load modes, dry runs, and reports, see the private S3 sources, SQL Server guide, and dry runs and reports.

Multi-output Writes

For workflows that write several related tables in one run, use Table.to_mssql(...) to create output specs and Session.write_all(...) to execute them together. Shared lazy SQL dependencies can be cached during the workflow so common upstream work is not repeated for each output.

See the multiple outputs and shared caching guide for setup, dry runs, cache options, and failure behavior.

Rust Quickstart

use delta_funnel::{
    DeltaFunnelRuntime, DeltaFunnelSession, DeltaSourceConfig, LoadMode,
    MssqlConnectionConfig, MssqlOutputTarget, MssqlTargetConfig,
    MssqlTargetTable, OutputWritePlan, RunMode, SessionOptions,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let ado_connection_string = concat!(
        "server=tcp:localhost,1433;",
        "database=warehouse;",
        "User ID=etl_user;",
        "Password=REPLACE_ME;",
        "encrypt=true;",
        "TrustServerCertificate=yes",
    );

    let default_connection = MssqlConnectionConfig::new(ado_connection_string)?
        .with_display_label("warehouse");
    let mut session = DeltaFunnelSession::new(
        SessionOptions::new().with_default_mssql_connection(default_connection),
    )?;
    let runtime = DeltaFunnelRuntime::new()?;

    // Register the Delta table as "orders" so SQL can reference it.
    let _orders = session.delta_lake(DeltaSourceConfig::new(
        "orders",
        "file:///path/to/orders-delta",
    ))?;

    // Build a lazy DataFusion SQL query. No rows are read yet.
    let daily_orders = runtime.table_from_sql(
        &mut session,
        r#"
        select customer_id, order_date, total_amount
        from orders
        where order_date >= date '2026-01-01'
        "#,
    )?;

    // Preview executes the DataFusion query with a limit.
    let preview = runtime.preview_table(&session, &daily_orders, 20)?;
    println!("{}", preview.text());

    // Write executes the query and loads the result into SQL Server.
    let target = MssqlTargetConfig::new(MssqlTargetTable::new("dbo", "daily_orders")?)
        .with_load_mode(LoadMode::CreateAndLoad);
    let output = OutputWritePlan::new(
        daily_orders,
        MssqlOutputTarget::new("daily_orders", target, RunMode::Execute),
    );
    let report = runtime.write_to_mssql(&session, &output)?;

    println!("wrote output {}", report.output_name());
    Ok(())
}

For a guided version of this workflow, see the Rust quickstart. For the full Rust API, see docs.rs/delta-funnel and the query_load_dry_run example.

Development

For local builds and test setup, see the local development guide.

Project links

Data verified by PyPI on Jul 28, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.6.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mag1cfrog from gravatar.com mag1cfrog

GitHub Statistics

Data verified by PyPI on Jul 28, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.10

Tags

delta-lake sql-server datafusion tds arrow

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

deltafunnel-0.4.1-cp310-abi3-win_amd64.whl (41.7 MB view details)

Uploaded CPython 3.10+Windows x86-64

deltafunnel-0.4.1-cp310-abi3-manylinux_2_28_x86_64.whl (49.4 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ x86-64

deltafunnel-0.4.1-cp310-abi3-macosx_11_0_arm64.whl (42.4 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

deltafunnel-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl (44.5 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file deltafunnel-0.4.1-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: deltafunnel-0.4.1-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 41.7 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for deltafunnel-0.4.1-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 68219097f619871d423dc3ca605a1dbf80b326f0c80abc39b73c8fb72a12596a
MD5 90b82df9bfb94d5948ee416dd171bb1a
BLAKE2b-256 8920ffd718963e2fd9bfef2ddd48e96479736616773335df038ee7c696cbf052

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltafunnel-0.4.1-cp310-abi3-win_amd64.whl:

Publisher: pypi.yml on mag1cfrog/delta-funnel

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file deltafunnel-0.4.1-cp310-abi3-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for deltafunnel-0.4.1-cp310-abi3-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 49d38ec31a0294a7546cfaec40a8f5be4da2abc8a98445ee46e00a22b1fdafb1
MD5 4f77b9194a9107b434647c5ad27aa0b0
BLAKE2b-256 29d77e33dd358a29369b561b255da015ab91ad3ba73d2a0bd7cf947953a333c7

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltafunnel-0.4.1-cp310-abi3-manylinux_2_28_x86_64.whl:

Publisher: pypi.yml on mag1cfrog/delta-funnel

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file deltafunnel-0.4.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deltafunnel-0.4.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 5f08fa6d314958f7c0f0118cae8bd5d2f1d53ef935517e874b34210fce782044
MD5 4db6cf0031d722d8f73c0b9dda4c09c8
BLAKE2b-256 ee2a24392b6a7d10f1783b1c7e5e9efd32d7ccf6ac70d0815f8ebb67c6d31a77

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltafunnel-0.4.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: pypi.yml on mag1cfrog/delta-funnel

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file deltafunnel-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for deltafunnel-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 6b6de6d30f0e51fc0d8be18f9be36cbf80fee6aeb3234569f1fa008afcc6f335
MD5 b5ebfac720070d95f5df27f14765c25b
BLAKE2b-256 b705ab418586d739d77d58c96f4aa9d8f7305810d9d7482ab4efa1ea13b4742f

See more details on using hashes here.

Provenance

The following attestation bundles were made for deltafunnel-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: pypi.yml on mag1cfrog/delta-funnel

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.6.0

4 files

0.5.0

4 files

This release

0.4.1 This release

4 files

0.4.0

4 files

0.3.3

4 files

0.3.2

4 files

0.3.1

4 files

0.3.0

4 files

0.2.1

4 files

0.2.0

4 files

0.1.6

4 files

0.1.5

4 files

0.1.4

4 files

0.1.3

4 files

0.1.2

4 files

0.1.1

4 files

0.1.0

4 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page