Sparkless — PySpark-compatible DataFrames without the JVM
Install: pip install "sparkless>=4,<5"
Use Sparkless to run PySpark-style unit tests and local pipelines 10–100× faster in CI. Powered by the open-source Rust engine robin-sparkless (Polars execution).
Not a full cluster replacement. See Before you adopt for UDF, parity, and production caveats.
Choose your path
| I want to… | Start here |
|---|---|
| Use Sparkless in Python (tests, local pipelines) | python/README.md · pip install "sparkless>=4,<5" |
| Embed the Rust engine | docs/QUICKSTART.md · robin-sparkless = "4" on crates.io |
| Contribute | CONTRIBUTING.md · make check-full |
Full documentation: Read the Docs
Quick start (Python)
# Swap the import—everything else stays the same.
from sparkless.sql import SparkSession, functions as F
spark = SparkSession.builder.app_name("demo").get_or_create()
df = spark.createDataFrame([{"x": 1}, {"x": 2}])
df.filter(F.col("x") > 1).show()
pip install "sparkless>=4,<5"
More: Python getting started · Testing guide · FAQ
Why Sparkless (Python)?
- Familiar API —
SparkSession,DataFrame,Column, and PySpark-like functions so you can reuse patterns without the JVM. - Fast local execution — Runs natively (no JVM) and uses Polars for IO, expressions, and aggregations.
- Test the same suite two ways — Use
sparkless.testingto run tests with Sparkless (fast) or real PySpark (parity checks). - Optional “Spark-like” features — SQL, temp/global temp views,
saveAsTable, Delta, and JDBC (see python/README.md).
Features (Python surface)
| Area | What’s included |
|---|---|
| Core | SparkSession, DataFrame, Column, functions |
| IO | CSV, Parquet, JSON, Delta |
| Expressions | col, lit, when/otherwise, casts, null handling |
| Aggregates | count, sum, avg, min, max, groupBy().agg() |
| Window | row_number, rank, dense_rank, lag, lead, first_value, last_value via .over() |
| Arrays, strings, JSON | Common PySpark functions (explode, regexp_*, get_json_object, from_json, to_json, …) |
| SQL + views | spark.sql, temp/global temp views, saveAsTable, catalog().listTables() |
| JDBC | Read/write via spark.read.jdbc(...) / df.write.jdbc(...) |
Parity: 200+ fixtures validated against PySpark. Before you adopt · PySpark differences · Parity status
Installation
Python (sparkless v4) — recommended
pip install "sparkless>=4,<5"
Contributors: pip install ./python or cd python && maturin develop — see CONTRIBUTING.md.
Rust engine (optional)
Most users should use the Python package above. To embed the engine in Rust:
[dependencies]
robin-sparkless = "4"
Optional features: sql, delta, jdbc, sqlite, jdbc_mysql. See docs/QUICKSTART.md.
Development
Prerequisites: Rust (see rust-toolchain.toml), Python 3.8+, maturin. Java only for SPARKLESS_TEST_MODE=pyspark.
See CONTRIBUTING.md for setup, make check-full, pytest, and maturin workflow.
| Command | Description |
|---|---|
make check-full |
Full CI-equivalent check (Rust + Python) |
pytest tests/ -v |
Python tests (sparkless backend) |
SPARKLESS_TEST_MODE=pyspark pytest tests/ -v |
Same tests against real PySpark |
make check |
Rust: format, clippy, audit, tests |
Documentation
| Resource | Description |
|---|---|
| Python package | Install, quick start, platform matrix, API overview |
| Read the Docs | Getting started, testing, migration, FAQ |
| Before you adopt | UDF limits, parity caveats, production notes |
| CONTRIBUTING | Dev setup and PR checklist |
| docs.rs | Rust API reference |
| Testing Guide | Dual-mode testing with sparkless.testing |
| PySpark Differences | Known divergences |
| RELEASING | Publishing to crates.io and PyPI |
See CHANGELOG.md for version history.
License
MIT
1 maintainer
Metadata
1 maintainer
Release files for sparkless 4.14.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| sparkless-4.14.1.tar.gz | 694.0 kB | Without provenance | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| sparkless-4.14.1-cp38-abi3-win_arm64.whl | CPython 3.8 | abi3 | Windows ARM64 | Details |
| sparkless-4.14.1-cp38-abi3-win_amd64.whl | CPython 3.8 | abi3 | Windows x86-64 | Details |
| sparkless-4.14.1-cp38-abi3-musllinux_1_2_x86_64.whl | CPython 3.8 | abi3 | Linux musl 1.2+ x86-64 | Details |
| sparkless-4.14.1-cp38-abi3-musllinux_1_2_aarch64.whl | CPython 3.8 | abi3 | Linux musl 1.2+ ARM64 | Details |
| sparkless-4.14.1-cp38-abi3-manylinux_2_28_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.28+ ARM64 | Details |
| sparkless-4.14.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| sparkless-4.14.1-cp38-abi3-macosx_11_0_arm64.whl | CPython 3.8 | abi3 | macOS 11.0+ ARM64 | Details |
| sparkless-4.14.1-cp38-abi3-macosx_10_12_x86_64.whl | CPython 3.8 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 321.3 MB
Release files / sparkless-4.14.1.tar.gz
| Download URL | sparkless-4.14.1.tar.gz |
|---|---|
| Size | 694.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
42ed99c84b2c1de4bd8683119dd25d26ac4024a2a28863aef9b6ffa47f14e172
|
|
BLAKE2b-256 checksum How to use checksums |
8d07030e333aad3b2d9822bfb5e5709af25cb22757cc44b29431890f5168b54c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Release files / sparkless-4.14.1-cp38-abi3-win_arm64.whl
| Download URL | sparkless-4.14.1-cp38-abi3-win_arm64.whl |
|---|---|
| Size | 39.8 MB |
| Tags | CPython 3.8 Windows ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
9d629c3f3d1995ade9c4358da8b9838b42e4c65fd1a9675e1a688652d757340a
|
|
BLAKE2b-256 checksum How to use checksums |
0bcc56fd4a2d9bbda6e4b21318e5d5be254bf79f0cef30da0ca00a77784dea54
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Release files / sparkless-4.14.1-cp38-abi3-win_amd64.whl
| Download URL | sparkless-4.14.1-cp38-abi3-win_amd64.whl |
|---|---|
| Size | 43.5 MB |
| Tags | CPython 3.8 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
b1eec4a46c602e7d36c0754208cdaf5c398dab2c73fd572e38f640733c43e383
|
|
BLAKE2b-256 checksum How to use checksums |
075c7ff755d1fd40f4f1ec7d5b20c65ac25441b1c3d290c5496a3426ae97d72e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Release files / sparkless-4.14.1-cp38-abi3-musllinux_1_2_x86_64.whl
| Download URL | sparkless-4.14.1-cp38-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 41.5 MB |
| Tags | CPython 3.8 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
711f14abff9e52dc19d3931af8dbcf56eca2ebbf39b12b2abb9fbea21a0d13cf
|
|
BLAKE2b-256 checksum How to use checksums |
466b52e9c85b95465a4e0df56cacc0543c59dda01e51e236b3776e117c1143b2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Release files / sparkless-4.14.1-cp38-abi3-musllinux_1_2_aarch64.whl
| Download URL | sparkless-4.14.1-cp38-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 38.7 MB |
| Tags | CPython 3.8 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
817fad5d34a0b6c33ddad636ccc48309a55d266438e32e27b67b498966a65081
|
|
BLAKE2b-256 checksum How to use checksums |
4bd6f603e5d26d68a0bddb8bc89121ffd81d1d9a2344aa658396ca9adcf7908a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Release files / sparkless-4.14.1-cp38-abi3-manylinux_2_28_aarch64.whl
| Download URL | sparkless-4.14.1-cp38-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 38.7 MB |
| Tags | CPython 3.8 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
b5d924c61600b3a3820411be951dc4fe99027d6038ee9210b0a37acc5f92a945
|
|
BLAKE2b-256 checksum How to use checksums |
c2852d39255650490fb0736ca47768486a2ceb0b1d6d7348fcf0d94b130acd09
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Release files / sparkless-4.14.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | sparkless-4.14.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 41.5 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
87a99ae73e7046a3f82a7ad1a7e753717f56bbf6c0f2014d464f307f53f2a7da
|
|
BLAKE2b-256 checksum How to use checksums |
6cf89a80cd050dffe16cfd92a38e40252b07b98103629e21e05ba9752d2a28a0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Release files / sparkless-4.14.1-cp38-abi3-macosx_11_0_arm64.whl
| Download URL | sparkless-4.14.1-cp38-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 37.1 MB |
| Tags | CPython 3.8 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
801160abb681dd71e7cf5ce534173da39890e0caa8866d9c99c55e8ee3229893
|
|
BLAKE2b-256 checksum How to use checksums |
cce3496aaec9c0e7ad264549773855015e71135e6d1580f1238a8726ee06aa6d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Release files / sparkless-4.14.1-cp38-abi3-macosx_10_12_x86_64.whl
| Download URL | sparkless-4.14.1-cp38-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 39.9 MB |
| Tags | CPython 3.8 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
e1cdb1b8e0c9e027d8a673e3b804554dc4b004177f76002ac89e897b605bbb57
|
|
BLAKE2b-256 checksum How to use checksums |
2970547785f51bb5cb11041dee7939bec1a63b4dcedf521f45744e63f055b103
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.15.0
|
Security for sparkless 4.14.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
