Sparkless

PySpark-compatible DataFrames in Python—no JVM, no cluster.
Drop-in replacement for local development and testing. Same API, 10–100× faster feedback.

Before you adopt: Read caveats — UDF limits, parity gaps, not a Spark cluster replacement.

• Python 3.8+ · PySpark 3.2–3.5 compatible · MIT

In 30 seconds

# Swap the import—everything else stays the same.
from sparkless.sql import SparkSession, functions as F

spark = SparkSession.builder.app_name("demo").get_or_create()
df = spark.createDataFrame([{"x": 1}, {"x": 2}])
df.filter(F.col("x") > 1).show()

pip install "sparkless>=4,<5"

No Java, no Spark install, no 30-second session startup. Ideal for unit tests and CI.

Why Sparkless?

v4 uses the robin-sparkless Rust engine (Polars under the hood). No Polars Python dependency at runtime—just a small native extension.

Installation

From PyPI (recommended)

pip install "sparkless>=4,<5"

Prebuilt wheels are provided for common platforms (Linux, macOS, Windows).
For development from the robin-sparkless repo:

# From repo root
pip install ./python

Contributors should use the uv setup and native build workflow.

Optional extras

Supported platforms

Prebuilt wheels are published for CI targets on each release (see PyPI files):

Python: 3.8, 3.9, 3.10, 3.11, 3.12 (cp38–cp312 ABI tags on wheels).

If no wheel matches your platform, follow the source build instructions.

Quick start

Basic usage

from sparkless.sql import SparkSession, functions as F

spark = SparkSession.builder.app_name("MyApp").get_or_create()
data = [{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]
df = spark.createDataFrame(data)

df.filter(F.col("age") > 25).select("name").show()
# +----+
# |name|
# +----+
# | Bob|
# +----+

result = df.filter(F.col("age") > 25).select("name").collect()
# [Row(name='Bob')]

In tests (pytest)

import pytest
from sparkless.sql import SparkSession, functions as F

def test_high_scores():
    spark = SparkSession.builder.app_name("test").get_or_create()
    df = spark.createDataFrame([{"score": 95}, {"score": 87}, {"score": 92}])
    high = df.filter(F.col("score") > 90)
    assert high.count() == 2
    assert high.agg(F.avg("score")).collect()[0][0] == 93.5
    spark.stop()

SQL and temp views

df = spark.createDataFrame([{"name": "Alice", "salary": 50000}, {"name": "Bob", "salary": 60000}])
df.createOrReplaceTempView("employees")

spark.sql("SELECT name, salary FROM employees WHERE salary > 50000").show()

Bounded Arrow export

toArrow() still returns a materialized PyArrow Table. Use toArrowBatches() to stream RecordBatches from the query plan with row and byte limits:

with df.toArrowBatches(max_rows=1024, max_bytes=8 * 1024 * 1024) as batches:
    for batch in batches:
        consume(batch)  # batch is a pyarrow.RecordBatch
        if should_stop():
            break

Each batch stays within both values (max_bytes is checked against RecordBatch.nbytes); if a single row is larger than max_bytes, iteration raises ValueError. The defaults are 10,000 rows and 64 MiB. Close the iterator (or use it as a context manager) after an early stop so the query stops producing batches.

JDBC (PostgreSQL, SQLite, MySQL, etc.)

# Read
url = "jdbc:postgresql://localhost:5432/mydb"
df = spark.read.jdbc(url=url, table="users", properties={"user": "u", "password": "p"})

# Write
df.write.jdbc(url=url, table="users_backup", properties={"user": "u", "password": "p"}, mode="overwrite")

Supported: PostgreSQL, SQLite, MySQL, MariaDB, SQL Server, Oracle, DB2. Same options as PySpark (sessionInitStatement, batchsize, truncate, etc.).

API at a glance

Detailed parity and differences: PYSPARK_DIFFERENCES.md, PARITY_STATUS.md.

Dual-mode testing (sparkless + PySpark)

Use sparkless.testing to run the same tests with the sparkless backend (fast) or real PySpark (parity check).

1. Register the plugin (e.g. in conftest.py):

pytest_plugins = ["sparkless.testing"]

2. Write tests using the spark and optional spark_imports fixtures:

def test_filter(spark):
    df = spark.createDataFrame([{"id": 1}, {"id": 2}])
    assert df.filter(df.id > 1).count() == 1

def test_upper(spark, spark_imports):
    F = spark_imports.F
    df = spark.createDataFrame([{"name": "alice"}])
    assert df.select(F.upper("name")).collect()[0][0] == "ALICE"

3. Run

# Default: sparkless backend (no JVM)
pytest tests/ -v

# Parallel
pytest tests/ -n 12 -v

# Against real PySpark (requires Java + pip install pyspark)
SPARKLESS_TEST_MODE=pyspark pytest tests/ -v

Fixtures: spark, spark_mode, spark_imports, isolated_session, table_prefix
Markers: @pytest.mark.sparkless_only, @pytest.mark.pyspark_only, @pytest.mark.delta
Helpers: assert_dataframes_equal(), compare_dataframes()

Full details: TESTING_GUIDE.md.

Requirements

• Python 3.8+
• Sparkless mode: No Java or Spark required.
• PySpark mode (optional): Java and pip install pyspark for SPARKLESS_TEST_MODE=pyspark.

Development

Follow CONTRIBUTING.md to create the root .venv, install development dependencies with uv, and build the native extension. From the repository root:

# Rebuild after changing Rust or PyO3 code
(
  cd python
  uv run --no-project --python ../.venv/bin/python \
    python -m maturin develop --uv --locked --release
)

# Run a focused test against the rebuilt extension
uv run --no-project --python .venv/bin/python \
  python -m pytest tests/unit/test_external_catalog_provider.py -q

For wheels and PySpark parity runs, see the contributor guide. Rust engine, UDFs, embedding: see the main README and docs.

Limitations

• UDFs: Use built-in functions and expressions; Python UDFs are not exposed. Rust UDFs are supported in the engine (UDF_GUIDE).
• Compatibility: Some APIs or behaviors differ from PySpark; see PYSPARK_DIFFERENCES.md.

Optional security hardening

For production or multi-tenant use, set SPARKLESS_HARDENED=1 and/or:

• SPARKLESS_JDBC_ALLOW_ARBITRARY_SQL=false — disallow arbitrary JDBC SQL; use dbtable only
• SPARKLESS_FILES_BASE=/path/to/sandbox — confine read/write file paths to that directory

See PYSPARK_DIFFERENCES.md — Security hardening for resource limits and multiprocessing notes.

Version history: v3 vs v4

Same from sparkless.sql import SparkSession in both. v4 is the current major release.

Links

• This repo: github.com/eddiethedean/robin-sparkless — Rust engine + Python package
• Docs: robin-sparkless.readthedocs.io
• Sparkless v3: github.com/eddiethedean/sparkless
• License: MIT (LICENSE in main repo)