Sparkless — PySpark-compatible DataFrames without the JVM

Install: pip install "sparkless>=4,<5"

Use Sparkless to run PySpark-style unit tests and local pipelines 10–100× faster in CI. Powered by the open-source Rust engine robin-sparkless (Polars execution).

Not a full cluster replacement. See Before you adopt for UDF, parity, and production caveats.

CI PyPI crates.io docs.rs Documentation License: MIT

Choose your path

I want to… Start here
Use Sparkless in Python (tests, local pipelines) python/README.md · pip install "sparkless>=4,<5"
Embed the Rust engine docs/QUICKSTART.md · robin-sparkless = "4" on crates.io
Contribute CONTRIBUTING.md · make check-full

Full documentation: Read the Docs

Quick start (Python)

# Swap the import—everything else stays the same.
from sparkless.sql import SparkSession, functions as F

spark = SparkSession.builder.app_name("demo").get_or_create()
df = spark.createDataFrame([{"x": 1}, {"x": 2}])
df.filter(F.col("x") > 1).show()

pip install "sparkless>=4,<5"

More: Python getting started · Testing guide · FAQ

Why Sparkless (Python)?

  • Familiar API — SparkSession, DataFrame, Column, and PySpark-like functions so you can reuse patterns without the JVM.
  • Fast local execution — Runs natively (no JVM) and uses Polars for IO, expressions, and aggregations.
  • Test the same suite two ways — Use sparkless.testing to run tests with Sparkless (fast) or real PySpark (parity checks).
  • Optional “Spark-like” features — SQL, temp/global temp views, saveAsTable, Delta, and JDBC (see python/README.md).

Features (Python surface)

Area What’s included
Core SparkSession, DataFrame, Column, functions
IO CSV, Parquet, JSON, Delta
Expressions col, lit, when/otherwise, casts, null handling
Aggregates count, sum, avg, min, max, groupBy().agg()
Window row_number, rank, dense_rank, lag, lead, first_value, last_value via .over()
Arrays, strings, JSON Common PySpark functions (explode, regexp_*, get_json_object, from_json, to_json, …)
SQL + views spark.sql, temp/global temp views, saveAsTable, catalog().listTables()
JDBC Read/write via spark.read.jdbc(...) / df.write.jdbc(...)

Parity: 200+ fixtures validated against PySpark. Before you adopt · PySpark differences · Parity status

Installation

Python (sparkless v4) — recommended

pip install "sparkless>=4,<5"

Contributors: pip install ./python or cd python && maturin develop — see CONTRIBUTING.md.

Rust engine (optional)

Most users should use the Python package above. To embed the engine in Rust:

[dependencies]
robin-sparkless = "4"

Optional features: sql, delta, jdbc, sqlite, jdbc_mysql. See docs/QUICKSTART.md.

Development

Prerequisites: Rust (see rust-toolchain.toml), Python 3.8+, maturin. Java only for SPARKLESS_TEST_MODE=pyspark.

See CONTRIBUTING.md for setup, make check-full, pytest, and maturin workflow.

Command Description
make check-full Full CI-equivalent check (Rust + Python)
pytest tests/ -v Python tests (sparkless backend)
SPARKLESS_TEST_MODE=pyspark pytest tests/ -v Same tests against real PySpark
make check Rust: format, clippy, audit, tests

Documentation

Resource Description
Python package Install, quick start, platform matrix, API overview
Read the Docs Getting started, testing, migration, FAQ
Before you adopt UDF limits, parity caveats, production notes
CONTRIBUTING Dev setup and PR checklist
docs.rs Rust API reference
Testing Guide Dual-mode testing with sparkless.testing
PySpark Differences Known divergences
RELEASING Publishing to crates.io and PyPI

See CHANGELOG.md for version history.

License

MIT

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (4.13.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for eddiethedean from gravatar.com eddiethedean

License

MIT License (MIT)

Requires

Python >=3.8

Tags

pyspark polars dataframe spark etl

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparkless-4.13.1.tar.gz (20.8 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

sparkless-4.13.1-cp38-abi3-win_arm64.whl (35.2 MB view details)

Uploaded CPython 3.8+Windows ARM64

sparkless-4.13.1-cp38-abi3-win_amd64.whl (38.2 MB view details)

Uploaded CPython 3.8+Windows x86-64

sparkless-4.13.1-cp38-abi3-musllinux_1_2_x86_64.whl (36.6 MB view details)

Uploaded CPython 3.8+musllinux: musl 1.2+ x86-64

sparkless-4.13.1-cp38-abi3-musllinux_1_2_aarch64.whl (34.2 MB view details)

Uploaded CPython 3.8+musllinux: musl 1.2+ ARM64

sparkless-4.13.1-cp38-abi3-manylinux_2_28_aarch64.whl (34.2 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.28+ ARM64

sparkless-4.13.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.5 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

sparkless-4.13.1-cp38-abi3-macosx_11_0_arm64.whl (32.7 MB view details)

Uploaded CPython 3.8+macOS 11.0+ ARM64

sparkless-4.13.1-cp38-abi3-macosx_10_12_x86_64.whl (35.1 MB view details)

Uploaded CPython 3.8+macOS 10.12+ x86-64

File details

Details for the file sparkless-4.13.1.tar.gz.

File metadata

  • Download URL: sparkless-4.13.1.tar.gz
  • Upload date:
  • Size: 20.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.14.1

File hashes

Hashes for sparkless-4.13.1.tar.gz
Algorithm Hash digest
SHA256 72d83b0cc097806f231d73c841a7dcfd04e69342bc66159dc4d932a02dc32f29
MD5 b433ea690c0e52fdacc9c7c19d19f2c5
BLAKE2b-256 02d0da47001c4ce30879db8bf8c53ec5b19dbafda89d9de69ff6b26160550b98

See more details on using hashes here.

File details

Details for the file sparkless-4.13.1-cp38-abi3-win_arm64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.1-cp38-abi3-win_arm64.whl
Algorithm Hash digest
SHA256 22b985e92a6512c3821ff75ddd9e3bfdfe580668069bbdbdcccc7079e0c77cc2
MD5 dfed1a5bf314a320a80a28bb5ca5d98e
BLAKE2b-256 f1b165bc28cf98a49dbb4c9179016d74f669622eb873a1d9c6410bc6afa27924

See more details on using hashes here.

File details

Details for the file sparkless-4.13.1-cp38-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.1-cp38-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 e77040fd4801072ec7b3c7215696c3e2795e046e75d6419c4814469f8efc7c99
MD5 00bb9108d64269d78787bd92bd1a0875
BLAKE2b-256 a731ec1e8595c0b9116e64606c918d8caaaca3c2470e968cc1d0cd734b829f73

See more details on using hashes here.

File details

Details for the file sparkless-4.13.1-cp38-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.1-cp38-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 95e2cce8a221fa7cbebef06ef6d798aaeb0baf86748d761a2f3ad7fabc609eda
MD5 13cb4fe219fa82f33853288e7ec07f79
BLAKE2b-256 dfba91f19d540dc358ba0a7b7cf30a77374324b27d269988b8b50688d5286b25

See more details on using hashes here.

File details

Details for the file sparkless-4.13.1-cp38-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.1-cp38-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 097e7dd89aec1e992283a887c96339ae013428bbdbdae9c1bcffcd3f21e938f0
MD5 0899b34e38b8cfffb261cc64d144564e
BLAKE2b-256 938a47d8e0cf9353790021c0ce5a8773d04ce5a22fc684f2c2eda45bf5984e1f

See more details on using hashes here.

File details

Details for the file sparkless-4.13.1-cp38-abi3-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.1-cp38-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 e8d21a7fd8d0925670b429bc83cc156e8fe786d9675a0005dbf95243ec0e5756
MD5 eb181bd29de403ce17622af9c5dc168c
BLAKE2b-256 ac558a54aa9932a05ec1de8d0065d3ef060f5fb7178ed70e51e6c0fa2314746d

See more details on using hashes here.

File details

Details for the file sparkless-4.13.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ab0d19cf5682672cda57a2809760d54cc8ecf237b60a1c141d08ea03aeb8214e
MD5 3b9b488119ae7df803c5d21075f6067b
BLAKE2b-256 3a9ad4930e0723ba78857690f2e12b7b018a1745eea2459b7d76e500559b9314

See more details on using hashes here.

File details

Details for the file sparkless-4.13.1-cp38-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.1-cp38-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 3bfba0eaff6d2205273c0bc3042857aac6af281184d6c1061e49d789b5db82ce
MD5 b6ec5b2343af808e8324c4809a9e7dd0
BLAKE2b-256 11d7d92cc68f380b0ba24b939b08ea32dfa2c7e3574a491703cc58cec4c2b02f

See more details on using hashes here.

File details

Details for the file sparkless-4.13.1-cp38-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for sparkless-4.13.1-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 3ba241026c9bf5434ce284fc3f26090ecd0e9d8076199b1214249cdd3332011e
MD5 63fa17aa2bad829ff83d8a95cdb02ac9
BLAKE2b-256 739a13fdd1804b8be224c470a05804754e2aa0f065b5f210020e7aeeac046450

See more details on using hashes here.

Release history Release notifications | RSS feed

4.13.2

9 files

This release

4.13.1 This release

9 files

4.13.0

9 files

4.12.1

9 files

4.11.0

9 files

4.10.0

9 files

4.9.0

9 files

4.8.0

9 files

4.7.0

9 files

4.6.0

9 files

4.5.7

9 files

4.5.6

9 files

4.5.5

9 files

4.5.4

9 files

4.5.3

9 files

4.5.2

9 files

4.5.1

9 files

4.5.0

9 files

4.4.0

9 files

4.3.0

9 files

4.2.1

9 files

4.2.0

9 files

4.1.0

9 files

4.0.1

9 files

3.31.0

2 files

3.30.0

2 files

3.29.0

2 files

3.28.0

2 files

3.27.1

2 files

3.27.0

2 files

3.26.1

2 files

3.26.0

2 files

3.25.0

2 files

3.24.0

2 files

3.23.0

2 files

3.22.0

2 files

3.21.0

2 files

3.20.0

2 files

3.19.1

2 files

3.19.0

2 files

3.18.7

2 files

3.18.6

2 files

3.18.5

2 files

3.18.4

2 files

3.18.3

2 files

3.18.2

2 files

3.18.1

2 files

3.18.0

2 files

3.17.11

2 files

3.17.10

2 files

3.17.9

2 files

3.17.8

2 files

3.17.7

2 files

3.17.6

2 files

3.17.5

2 files

3.17.4

2 files

3.17.3

2 files

3.17.2

2 files

3.17.1

2 files

3.17.0

2 files

3.16.0

2 files

3.15.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page