Sparkless — PySpark-compatible DataFrames without the JVM

Install: pip install "sparkless>=4,<5"

Use Sparkless to run PySpark-style unit tests and local pipelines 10–100× faster in CI. Powered by the open-source Rust engine robin-sparkless (Polars execution).

Not a full cluster replacement. See Before you adopt for UDF, parity, and production caveats.

CI PyPI crates.io docs.rs Documentation License: MIT

Choose your path

I want to… Start here
Use Sparkless in Python (tests, local pipelines) python/README.md · pip install "sparkless>=4,<5"
Embed the Rust engine docs/QUICKSTART.md · robin-sparkless = "4" on crates.io
Contribute CONTRIBUTING.md · make check-full

Full documentation: Read the Docs

Quick start (Python)

# Swap the import—everything else stays the same.
from sparkless.sql import SparkSession, functions as F

spark = SparkSession.builder.app_name("demo").get_or_create()
df = spark.createDataFrame([{"x": 1}, {"x": 2}])
df.filter(F.col("x") > 1).show()

pip install "sparkless>=4,<5"

More: Python getting started · Testing guide · FAQ

Why Sparkless (Python)?

  • Familiar API — SparkSession, DataFrame, Column, and PySpark-like functions so you can reuse patterns without the JVM.
  • Fast local execution — Runs natively (no JVM) and uses Polars for IO, expressions, and aggregations.
  • Test the same suite two ways — Use sparkless.testing to run tests with Sparkless (fast) or real PySpark (parity checks).
  • Optional “Spark-like” features — SQL, temp/global temp views, saveAsTable, Delta, and JDBC (see python/README.md).

Features (Python surface)

Area What’s included
Core SparkSession, DataFrame, Column, functions
IO CSV, Parquet, JSON, Delta
Expressions col, lit, when/otherwise, casts, null handling
Aggregates count, sum, avg, min, max, groupBy().agg()
Window row_number, rank, dense_rank, lag, lead, first_value, last_value via .over()
Arrays, strings, JSON Common PySpark functions (explode, regexp_*, get_json_object, from_json, to_json, …)
SQL + views spark.sql, temp/global temp views, saveAsTable, catalog().listTables()
JDBC Read/write via spark.read.jdbc(...) / df.write.jdbc(...)

Parity: 200+ fixtures validated against PySpark. Before you adopt · PySpark differences · Parity status

Installation

Python (sparkless v4) — recommended

pip install "sparkless>=4,<5"

Contributors: pip install ./python or cd python && maturin develop — see CONTRIBUTING.md.

Rust engine (optional)

Most users should use the Python package above. To embed the engine in Rust:

[dependencies]
robin-sparkless = "4"

Optional features: sql, delta, jdbc, sqlite, jdbc_mysql. See docs/QUICKSTART.md.

Development

Prerequisites: Rust (see rust-toolchain.toml), Python 3.8+, maturin. Java only for SPARKLESS_TEST_MODE=pyspark.

See CONTRIBUTING.md for setup, make check-full, pytest, and maturin workflow.

Command Description
make check-full Full CI-equivalent check (Rust + Python)
pytest tests/ -v Python tests (sparkless backend)
SPARKLESS_TEST_MODE=pyspark pytest tests/ -v Same tests against real PySpark
make check Rust: format, clippy, audit, tests

Documentation

Resource Description
Python package Install, quick start, platform matrix, API overview
Read the Docs Getting started, testing, migration, FAQ
Before you adopt UDF limits, parity caveats, production notes
CONTRIBUTING Dev setup and PR checklist
docs.rs Rust API reference
Testing Guide Dual-mode testing with sparkless.testing
PySpark Differences Known divergences
RELEASING Publishing to crates.io and PyPI

See CHANGELOG.md for version history.

License

MIT

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for eddiethedean from gravatar.com eddiethedean

License

MIT License (MIT)

Requires

Python >=3.8

Tags

pyspark polars dataframe spark etl

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparkless-4.13.2.tar.gz (582.3 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

sparkless-4.13.2-cp38-abi3-win_arm64.whl (35.2 MB view details)

Uploaded CPython 3.8+Windows ARM64

sparkless-4.13.2-cp38-abi3-win_amd64.whl (38.3 MB view details)

Uploaded CPython 3.8+Windows x86-64

sparkless-4.13.2-cp38-abi3-musllinux_1_2_x86_64.whl (36.6 MB view details)

Uploaded CPython 3.8+musllinux: musl 1.2+ x86-64

sparkless-4.13.2-cp38-abi3-musllinux_1_2_aarch64.whl (34.2 MB view details)

Uploaded CPython 3.8+musllinux: musl 1.2+ ARM64

sparkless-4.13.2-cp38-abi3-manylinux_2_28_aarch64.whl (34.2 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.28+ ARM64

sparkless-4.13.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.6 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

sparkless-4.13.2-cp38-abi3-macosx_11_0_arm64.whl (32.7 MB view details)

Uploaded CPython 3.8+macOS 11.0+ ARM64

sparkless-4.13.2-cp38-abi3-macosx_10_12_x86_64.whl (35.1 MB view details)

Uploaded CPython 3.8+macOS 10.12+ x86-64

File details

Details for the file sparkless-4.13.2.tar.gz.

File metadata

  • Download URL: sparkless-4.13.2.tar.gz
  • Upload date:
  • Size: 582.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.14.1

File hashes

Hashes for sparkless-4.13.2.tar.gz
Algorithm Hash digest
SHA256 201daa00306256a974d8a2752ced2e25ba9fd74fe325c2484c058312444b1398
MD5 9b8228c28769eb6a42d02f2ab6ae1b32
BLAKE2b-256 ef96a40a3c98eec4ac48bf2a888318cdce61b5aa7abaca0f0be207ac02f143e0

See more details on using hashes here.

File details

Details for the file sparkless-4.13.2-cp38-abi3-win_arm64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.2-cp38-abi3-win_arm64.whl
Algorithm Hash digest
SHA256 cc6198d0620de61c578e3a6c2b9e79d0d86e51286ea4247ebb4acea57f460ff5
MD5 ecb16af2eff20393ec3230a386adfdc2
BLAKE2b-256 05ed79c82a46ea3141104e44c306e9cc1850e47007326851fdb1726250733587

See more details on using hashes here.

File details

Details for the file sparkless-4.13.2-cp38-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.2-cp38-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 d5cd48aea4a5b6bb67f8da76552f1b105ecc59e6958f7a2be51012a402258693
MD5 02a279a7880b4f31a9aee4339bd7946a
BLAKE2b-256 4d121bad19aa739a13c99512c7e4ea2bd8199775a13ba8d254fbaecd1686ef10

See more details on using hashes here.

File details

Details for the file sparkless-4.13.2-cp38-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.2-cp38-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 221bfac19fed7e098e5f3ce65bc3c4752cb949725f3d1eb6a47f9770e2967a0f
MD5 0e6069d3761fba8d54e547eab930009e
BLAKE2b-256 e97a8947b8bdb8ffcc120dcc10fd5df6409b701fcfd8bd287c738f434ebaa24a

See more details on using hashes here.

File details

Details for the file sparkless-4.13.2-cp38-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.2-cp38-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 eff3c97d15ae7d48800c909513a7b9b3f4d1a22600c0b9b5a973ee7be585686c
MD5 3c48aa10a1ececc27c1ac8ab4deff025
BLAKE2b-256 b85973a1341c0bccb33c6e99a8228a80b2a5bc5ff1000f2ae965e17d18cc9968

See more details on using hashes here.

File details

Details for the file sparkless-4.13.2-cp38-abi3-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.2-cp38-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 36163e7a90af6eec4fda39af6599610886a4fb58ed197fc92aa1d46fa7dd1a18
MD5 73dd6c007d130425c873c76a63891641
BLAKE2b-256 a2b67cafa8b7e77dd0596c1e6532b7d8cc02562772f305877f571a1d38a7a449

See more details on using hashes here.

File details

Details for the file sparkless-4.13.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c9edc2ae525ab3ed9eabd0d730c069afda599c196bb276bdbe73e782dcd42a1c
MD5 03a067c03bd2985c1ef1b952d35bef05
BLAKE2b-256 c966297079a11d921ce72793dbbd69429651422474f94e6989d99c0d0bde8bae

See more details on using hashes here.

File details

Details for the file sparkless-4.13.2-cp38-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.2-cp38-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 22f0dffd77bd8ea78d10cf90aa20e5b56aef6c6d950a8adc55ebbade1a14c754
MD5 38df5df621d019b2e7595a6b1195f813
BLAKE2b-256 259ac86f896d15db78315195715ba1d09b50b85d7d6945ec7397ac026760bdb0

See more details on using hashes here.

File details

Details for the file sparkless-4.13.2-cp38-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.2-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 e7d4ed2d7c93a00b4acb7f86ac6fd21a325b75e8ef7e43bc5aa6f5ad43100984
MD5 4a374fd851793367ff4910bd96cef63e
BLAKE2b-256 6d605fe8c24ba58acd8756287accf1cd21e8156f0d1c377a6fb400cbef1c90bd

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

4.13.2 This release

9 files

4.13.1

9 files

4.13.0

9 files

4.12.1

9 files

4.11.0

9 files

4.10.0

9 files

4.9.0

9 files

4.8.0

9 files

4.7.0

9 files

4.6.0

9 files

4.5.7

9 files

4.5.6

9 files

4.5.5

9 files

4.5.4

9 files

4.5.3

9 files

4.5.2

9 files

4.5.1

9 files

4.5.0

9 files

4.4.0

9 files

4.3.0

9 files

4.2.1

9 files

4.2.0

9 files

4.1.0

9 files

4.0.1

9 files

3.31.0

2 files

3.30.0

2 files

3.29.0

2 files

3.28.0

2 files

3.27.1

2 files

3.27.0

2 files

3.26.1

2 files

3.26.0

2 files

3.25.0

2 files

3.24.0

2 files

3.23.0

2 files

3.22.0

2 files

3.21.0

2 files

3.20.0

2 files

3.19.1

2 files

3.19.0

2 files

3.18.7

2 files

3.18.6

2 files

3.18.5

2 files

3.18.4

2 files

3.18.3

2 files

3.18.2

2 files

3.18.1

2 files

3.18.0

2 files

3.17.11

2 files

3.17.10

2 files

3.17.9

2 files

3.17.8

2 files

3.17.7

2 files

3.17.6

2 files

3.17.5

2 files

3.17.4

2 files

3.17.3

2 files

3.17.2

2 files

3.17.1

2 files

3.17.0

2 files

3.16.0

2 files

3.15.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page