freshet
A lightweight caching framework for Python data pipelines.
Decorate your pipeline functions with @flow and @source, then instantiate Freshet with your pipeline module. Execution and cache state are scoped to that instance.
Installation
pip install freshet
Quick start
from freshet import Freshet
from my_project import my_pipeline
f = Freshet(my_pipeline)
result = f.tap("final_output") # computes + caches upstream DAG
result = f.tap("final_output") # cache hit
Base data sources
Use @source to declare external data sources as the roots of your pipeline. The function body returns a File or Directory descriptor:
from freshet import source, File
@source
def raw_data():
return File("data/raw.parquet")
File-mode outputs
For flows that produce files rather than Python objects, call flow_output() to allocate a cache path, write to it, and return the File:
import types
from freshet import Freshet, flow, flow_output, File
@flow
def plot_chart() -> File:
out = flow_output(".png")
save_plot([1, 2, 3], out.path)
return out
pipeline = types.ModuleType("my_pipeline")
pipeline.plot_chart = plot_chart
f = Freshet(pipeline, cache_dir=".freshet")
result = f.tap("plot_chart") # returns a File pointing to the cached file
Auto-bridging
If a flow expects an in-memory type (e.g. pl.DataFrame) but receives a File, freshet auto-loads it based on the file extension:
import polars as pl
from freshet import source, flow, File
@source
def raw_trades():
return File("data/raw/trades.parquet")
@flow
def cleaned(raw_trades: pl.DataFrame) -> pl.DataFrame:
# raw_trades is auto-bridged: File → pl.read_parquet
return raw_trades.filter(pl.col("price") > 0)
DAG introspection
freshet infers a dependency graph from function argument names. If a @flow function takes a parameter named raw_data, and there's a registered artifact called raw_data, freshet records that edge:
artifacts = f.artifacts() # all registered artifacts
edges = f.edges() # list of (upstream, downstream) tuples
bases = f.bases() # only @source artifacts
Custom serializers
By default, Polars DataFrames are serialized as Parquet and everything else uses pickle. You can register your own:
from freshet import Serializer, register_serializer
class MySerializer:
key = "my_format"
extension = "bin"
def can_handle(self, value) -> bool: ...
def save(self, value, path) -> None: ...
def load(self, path): ...
register_serializer(MySerializer())
Configuration
f = Freshet(my_pipeline, cache_dir="/path/to/cache")
By default the cache lives at .freshet/ in the current working directory.
Cache management
f.clear_cache("my_function") # clear one function's cache
f.clear_cache() # clear everything
Visualization
widget = f.chart() # anywidget-compatible DAG visualization
Security
freshet uses pickle as a general-purpose serializer fallback. Treat cache directories as trusted input only.
To reduce accidental code execution risk, f.chart() does not unpickle cached artifacts for graph details/preview by default. You can opt in with:
FRESHET_UNSAFE_PICKLE_INSPECT=1
Metadata
Release files for freshet 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| freshet-0.1.0.tar.gz | 123.3 kB | Without provenance | Details for freshet-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| freshet-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for freshet-0.1.0-py3-none-any.whl |
Total release size: 251.2 kB
Release files / freshet-0.1.0.tar.gz
| Download URL | freshet-0.1.0.tar.gz |
|---|---|
| Size | 123.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d0b0f371dda1465d731d36c2da87bfe0923f09dac817e6da5fb9b84e594ea132
|
|
BLAKE2b-256 checksum How to use checksums |
2d629a29b4f6d39413030991e0eb5cd6e01b43867e5e919b97539c43d397e061
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.7
|
Release files / freshet-0.1.0-py3-none-any.whl
| Download URL | freshet-0.1.0-py3-none-any.whl |
|---|---|
| Size | 128.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0a0bfbf9c37fe50031b4c7d7d1066a7c3b4500ea2c251057b6a9f88b86a40ec5
|
|
BLAKE2b-256 checksum How to use checksums |
55817b9588aee192cef2d6f0b5fd3d22d398cfcb4cc3ac01e2fdce9aead7fc3e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.7
|
Security for freshet 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.