polars_cache
A lightweight, lazy, disc-based cache for Polars LazyFrames.
Usage
import polars as pl
import polars_cache as pc
lf = pl.LazyFrame({"x" : range(100)})
def very_expensive(col: str):
pl.col(col).pow(2).exp().sqrt()
query = (
lf
.with_columns(very_expensive("x"))
.pipe(pc.cache_to_disc, max_age=120) # set up cache
)
df1 = query.collect() # populate the cache
df2 = query.collect() # second invocation will be much faster!
# do some downstream computation
another_query = query.with_columns(y = pl.col("x") + 7)
df3 = another_query.collect() # this will use the cache!
Updating a source will cause the cache to refresh:
import os
query_from_a_file = (
pl.scan_parquet("data.parquet")
.group_by("age", "sex")
.agg(pl.len())
.pipe(pc.cache_to_disc, check_sources=True)
)
_ = query_from_a_file.collect() # populate cache
result = query_from_a_file.collect() # load from cache
os.utime("data.parquet") # update source timestamp
new_result = query_from_a_file.collect() # cache is invalid -- will refresh
⚠️ Warning ⚠️
This function is opaque to the Polars optimizer and will split your query into two chunks: one before the cache statment and one after. Each query will be independently optimzed by Polars, but optimizations (e.g. projection and predicate pushdown) will NOT be able to cross the cache barrier. Use with caution.
1 maintainer
Metadata
1 maintainer
Release files for polars-cache 1.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| polars_cache-1.0.5.tar.gz | 20.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| polars_cache-1.0.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 24.1 kB
Release files / polars_cache-1.0.5.tar.gz
| Download URL | polars_cache-1.0.5.tar.gz |
|---|---|
| Size | 20.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5999acd98a04f6a392aae28cb1085cc3aa1e22156ca640ad74bbec0c655c1e0e
|
|
BLAKE2b-256 checksum How to use checksums |
cab46f791898872e2a10f9cac17e0b2a0fdf98449c79fe09745c1f1bcb142de4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.8.3
|
Release files / polars_cache-1.0.5-py3-none-any.whl
| Download URL | polars_cache-1.0.5-py3-none-any.whl |
|---|---|
| Size | 3.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1b7b530eaaf918d7689864a8c89d85f126286f36396e2cb34228a6bf8da7195f
|
|
BLAKE2b-256 checksum How to use checksums |
0385ee6e874719ee1198d5a8d1323a149335f2cd4498a2b39c858e944370025a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.8.3
|
