sparkless 4.13.0

PySpark-like DataFrame API in Python—no JVM. Uses robin-sparkless (Rust/Polars) as the execution engine.

Sparkless — PySpark-compatible DataFrames without the JVM

Install: pip install "sparkless>=4,<5"

Use Sparkless to run PySpark-style unit tests and local pipelines 10–100× faster in CI. Powered by the open-source Rust engine robin-sparkless (Polars execution).

Not a full cluster replacement. See Before you adopt for UDF, parity, and production caveats.

---

Choose your path

I want to…	Start here
Use Sparkless in Python (tests, local pipelines)	python/README.md · pip install "sparkless>=4,<5"
Embed the Rust engine	docs/QUICKSTART.md · robin-sparkless = "4" on crates.io
Contribute	CONTRIBUTING.md · make check-full

Full documentation: Read the Docs

---

Quick start (Python)

# Swap the import—everything else stays the same.
from sparkless.sql import SparkSession, functions as F

spark = SparkSession.builder.app_name("demo").get_or_create()
df = spark.createDataFrame([{"x": 1}, {"x": 2}])
df.filter(F.col("x") > 1).show()

pip install "sparkless>=4,<5"

More: Python getting started · Testing guide · FAQ

---

Why Sparkless (Python)?

• Familiar API — SparkSession, DataFrame, Column, and PySpark-like functions so you can reuse patterns without the JVM.
• Fast local execution — Runs natively (no JVM) and uses Polars for IO, expressions, and aggregations.
• Test the same suite two ways — Use sparkless.testing to run tests with Sparkless (fast) or real PySpark (parity checks).
• Optional “Spark-like” features — SQL, temp/global temp views, saveAsTable, Delta, and JDBC (see python/README.md).

---

Features (Python surface)

Area	What’s included
Core	SparkSession, DataFrame, Column, functions
IO	CSV, Parquet, JSON, Delta
Expressions	col, lit, when/otherwise, casts, null handling
Aggregates	count, sum, avg, min, max, groupBy().agg()
Window	row_number, rank, dense_rank, lag, lead, first_value, last_value via .over()
Arrays, strings, JSON	Common PySpark functions (explode, regexp_*, get_json_object, from_json, to_json, …)
SQL + views	spark.sql, temp/global temp views, saveAsTable, catalog().listTables()
JDBC	Read/write via spark.read.jdbc(...) / df.write.jdbc(...)

Parity: 200+ fixtures validated against PySpark. Before you adopt · PySpark differences · Parity status

---

Installation

Python (sparkless v4) — recommended

pip install "sparkless>=4,<5"

Contributors: pip install ./python or cd python && maturin develop — see CONTRIBUTING.md.

Rust engine (optional)

Most users should use the Python package above. To embed the engine in Rust:

[dependencies]
robin-sparkless = "4"

Optional features: sql, delta, jdbc, sqlite, jdbc_mysql. See docs/QUICKSTART.md.

---

Development

Prerequisites: Rust (see rust-toolchain.toml), Python 3.8+, maturin. Java only for SPARKLESS_TEST_MODE=pyspark.

See CONTRIBUTING.md for setup, make check-full, pytest, and maturin workflow.

Command	Description
make check-full	Full CI-equivalent check (Rust + Python)
pytest tests/ -v	Python tests (sparkless backend)
SPARKLESS_TEST_MODE=pyspark pytest tests/ -v	Same tests against real PySpark
make check	Rust: format, clippy, audit, tests

---

Documentation

Resource	Description
Python package	Install, quick start, platform matrix, API overview
Read the Docs	Getting started, testing, migration, FAQ
Before you adopt	UDF limits, parity caveats, production notes
CONTRIBUTING	Dev setup and PR checklist
docs.rs	Rust API reference
Testing Guide	Dual-mode testing with sparkless.testing
PySpark Differences	Known divergences
RELEASING	Publishing to crates.io and PyPI

See CHANGELOG.md for version history.

---

License

MIT