PySpark Dubber
A compatibility layer for the pyspark API, allowing you to run pyspark code on backends
such as DuckDB and Polars without porting your code.
Why
Lately, SQL engines and DataFrame libraries such as DuckDB and Polars have become popular, offering great performance for non-distributed analytical workflows up to relatively large datasets (tens of GBs). For these sizes and below, Spark adds a lot of overhead and its startup time is relatively slow, making it not very cost- and time-efficient.
However, Spark is still the most mature and widely used data processing framework, meaning that many people and organizations have large codebases relying on its APIs.
pyspark-dubber is a library that allows you to run pyspark code on many backends, such as
DuckDB and Polars (actually any backend supported by ibis at this time),
making it possible to migrate old code to a new backend with minimal changes.
The aspiration of pyspark-dubber is be bug-for-bug compatible with pyspark.
Documentation
You can find API documentation and more information about the project in our documentation page.
1 maintainer
Release files for pyspark-dubber 0.2.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pyspark_dubber-0.2.4.tar.gz | 7.1 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pyspark_dubber-0.2.4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 7.2 MB
Release files / pyspark_dubber-0.2.4.tar.gz
| Download URL | pyspark_dubber-0.2.4.tar.gz |
|---|---|
| Size | 7.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c07f2672a9e8263eead08dec77eb4336a8860b755f0adc0fc7713258de2d73cc
|
|
BLAKE2b-256 checksum How to use checksums |
773fe8d2d190c3e2479807926292e34ab4edf0b12d4a574d11dd2486f8bb74ea
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 8, 2026.
Transparency logRelease files / pyspark_dubber-0.2.4-py3-none-any.whl
| Download URL | pyspark_dubber-0.2.4-py3-none-any.whl |
|---|---|
| Size | 36.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6dd4ad61c6d69b0432cde73c49b4bf0da26cd74dcd27f647c4e73a35d539e3f7
|
|
BLAKE2b-256 checksum How to use checksums |
e2c76fd585cc639ea32de786c951faa8ddea8ba721480aa7d17a3746ec7cbad1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 8, 2026.
Transparency log
