Skip to main content
polars-io-tools logo, an igloo made of blocks in different blue shades

Custom parsing extensions for lazy polars

Build Status codecov License PyPI

Overview

polars-io-tools extends Polars lazy execution with custom I/O sources that push filters and column projections all the way down into the systems that hold your data — SQL databases, ClickHouse, Datadog, and Delta Lake — instead of loading everything and filtering in memory. It also adds lazy-friendly operations (joins, multi-source composition, time-series windows, caching, distributed execution) that keep predicate pushdown working where vanilla Polars would otherwise give up and materialize the whole frame.

Everything is exposed through the piot LazyFrame namespace and a handful of top-level scan_* / sink_* functions, so it composes naturally with the Polars API you already use.

Who is this for

Reach for polars-io-tools when you want Polars' lazy API over data that lives in an external store, and you care about not fetching rows or columns you will immediately throw away. It is most valuable for large, partitioned, or remote datasets where a filter on a date or key column should translate into a smaller query against the source. If your data already fits comfortably in memory or lives in local Parquet/CSV, plain Polars is the simpler choice.

Installation

pip install polars-io-tools

polars-io-tools requires Python 3.11 or newer. See the Installation guide for conda and source builds.

Quickstart

Importing the package registers the piot namespace on every Polars LazyFrame:

import polars as pl
import polars_io_tools  # registers the .piot namespace

left = pl.LazyFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
right = pl.LazyFrame({"x": [-1, -2, 3], "z": [7, 8, 9]})

# An inner join where the keys present on the left are pushed down as a
# filter on the right frame *before* the join runs.
result = left.piot.filtered_join(right, on="x").collect()
print(result)
# shape: (1, 3)
# ┌─────┬─────┬─────┐
# │ x   ┆ y   ┆ z   │
# │ --- ┆ --- ┆ --- │
# │ i64 ┆ i64 ┆ i64 │
# ╞═════╪═════╪═════╡
# │ 3   ┆ 6   ┆ 9   │
# └─────┴─────┴─────┘

For a guided walkthrough, start with the Getting Started tutorial.

What's included

  • Lazy I/O sources with predicate & projection pushdown — scan_db (any ODBC database), scan_clickhouse, scan_datadog, scan_delta, and from_narwhals. Filters on the resulting LazyFrame are translated into the source's own query language (SQL WHERE, Datadog time ranges, Delta partition pruning) so only the matching rows and columns are fetched.
  • Lazy writers — sink_delta and sink_clickhouse write a LazyFrame directly to Delta Lake or ClickHouse, including streaming/chunked writes and transparent handling of types the target store cannot represent natively.
  • Pushdown-preserving query building — filtered_join, filtered_join_asof, join_between, pushdown_combine, concat_named, and ts_with_columns express joins, multi-source composition, and rolling/lookback time-series logic without blocking the filter pushdown that those operations normally defeat.
  • Caching — cache keeps an in-memory, column- and partition-level cache for iterative work; cache_parquet materializes date-partitioned Parquet on local disk or S3, fetching only the partitions a query needs.
  • Distributed execution — execute_on_ray splits a LazyFrame across an existing Ray cluster, one task per partition. Build partitions with by_time (calendar windows), by_value (discrete keys), by_range (numeric buckets), by_key (enumerated keys), or an explicit ReadPartition list. (For multi-stage distributed pipelines, Polars Cloud is the more strategic option.)
  • Ergonomics — iter_rows for memory-efficient row iteration, debug to inspect what Polars pushes into a source, and disable_optimizations to compare against plain Polars.

Documentation

Full documentation lives in the project wiki:

Contributing

Contributions are welcome. See the Contributing guide and Local Development Setup to get started.

License

polars-io-tools is licensed under the Apache 2.0 license.

Metadata

Release files for polars-io-tools 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for polars-io-tools 0.3.0
File Size Uploaded Provenance Actions
polars_io_tools-0.3.0.tar.gz 496.8 kB Without provenance Details for polars_io_tools-0.3.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for polars-io-tools 0.3.0
File Actions
polars_io_tools-0.3.0-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for polars_io_tools-0.3.0-cp311-abi3-win_amd64.whl
polars_io_tools-0.3.0-cp311-abi3-manylinux_2_28_x86_64.whl CPython 3.11 abi3 Linux glibc 2.28+ x86-64 Details for polars_io_tools-0.3.0-cp311-abi3-manylinux_2_28_x86_64.whl
polars_io_tools-0.3.0-cp311-abi3-manylinux_2_28_aarch64.whl CPython 3.11 abi3 Linux glibc 2.28+ ARM64 Details for polars_io_tools-0.3.0-cp311-abi3-manylinux_2_28_aarch64.whl
polars_io_tools-0.3.0-cp311-abi3-macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64 Details for polars_io_tools-0.3.0-cp311-abi3-macosx_11_0_arm64.whl

Total release size: 3.4 MB

Release files / polars_io_tools-0.3.0.tar.gz

Download URL polars_io_tools-0.3.0.tar.gz
Size 496.8 kB
Tags Source
SHA-256 checksum
How to use checksums
e0d2cc7ccb55f70d7ddb39cde31abbbc0754d902f5ab719d06f3f21014d0d48e
BLAKE2b-256 checksum
How to use checksums
1c04501b45cd12adff47b6a660de395f994fad04fe4e414f4ede3bc02c4c3a84
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / polars_io_tools-0.3.0-cp311-abi3-win_amd64.whl

Download URL polars_io_tools-0.3.0-cp311-abi3-win_amd64.whl
Size 652.3 kB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
959822ab91e298625fa4e4fb8a87df6686127640a69a82b4d63f8568a2d35282
BLAKE2b-256 checksum
How to use checksums
7e25b879e3c92fc8c26a65f7c425ba75b1ea24a326ae4ca4194ecfe35440246d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / polars_io_tools-0.3.0-cp311-abi3-manylinux_2_28_x86_64.whl

Download URL polars_io_tools-0.3.0-cp311-abi3-manylinux_2_28_x86_64.whl
Size 763.3 kB
Tags CPython 3.11 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
829198b053e9727f64ffdcaf8e0c62686d87ac909e14a3e77e10e501b4701724
BLAKE2b-256 checksum
How to use checksums
b1a1799d1915c5f4df126e986242c01f4a1fa7107ab95fa70f8c29db4a89b640
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / polars_io_tools-0.3.0-cp311-abi3-manylinux_2_28_aarch64.whl

Download URL polars_io_tools-0.3.0-cp311-abi3-manylinux_2_28_aarch64.whl
Size 762.2 kB
Tags CPython 3.11 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
6b701acf777b374d2045ea9ea9ada0f726faa2d1b7987547d13a681ca0110bf3
BLAKE2b-256 checksum
How to use checksums
139aea673357f78e910b0c33a48e63837a8b494d0ac34a02aae38f9d1230cbc1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / polars_io_tools-0.3.0-cp311-abi3-macosx_11_0_arm64.whl

Download URL polars_io_tools-0.3.0-cp311-abi3-macosx_11_0_arm64.whl
Size 744.0 kB
Tags CPython 3.11 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
2eb548c8d0205bb75763d954dca6557da9c539e320f3c352496f398c353f8612
BLAKE2b-256 checksum
How to use checksums
ae7a8b637482c81a2f2f11528daac378f625fc4e849d3034f4527f8d26fa705e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for polars-io-tools 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page