mockpyspark

A drop-in mock for Apache PySpark 4.x for unit and integration testing. Runs entirely in Python — no JVM, no Hadoop, no cluster — using PyArrow as the compute engine and DuckDB for SQL.

pip install mock-pyspark

mockpyspark (the PyPI distribution is mock-pyspark; you still import mockpyspark) installs alongside real pyspark; there is no namespace collision. Tests opt in via a one-liner so that from pyspark.sql import ... in your production code transparently resolves to the mock.

Quick start

# conftest.py
import mockpyspark
mockpyspark.activate()

# tests/test_my_pipeline.py
from pyspark.sql import SparkSession, functions as F

def test_normalize(spark=None):
    spark = SparkSession.builder.getOrCreate()
    df = spark.createDataFrame([
        {"id": 1, "name": "  alice ", "score": 10},
        {"id": 2, "name": "BOB",      "score": 20},
    ])
    out = (
        df.withColumn("name", F.initcap(F.trim(F.col("name"))))
          .filter(F.col("score") > 15)
          .select("id", "name", "score")
          .collect()
    )
    assert out == [(2, "Bob", 20)]

No changes to your Spark logic. The alias just makes pyspark.* imports resolve to mockpyspark.* for the test process.

Version targeting

mockpyspark supports both PySpark 4.0.x and 4.1.x. APIs that only exist in 4.1 (e.g. current_time, TimeType) raise AttributeError when running in 4.0 mode.

It picks a target version automatically, in order:

1. PYSPARK_VERSION env var
2. The pyspark pin in pyproject.toml, requirements*.txt, poetry.lock, uv.lock, Pipfile.lock, or setup.cfg (walking up from the working directory)
3. SparkSession.builder.config("spark.mock.targetVersion", "4.0").getOrCreate() — per-session override
4. If nothing is found, defaults to the latest supported version and emits a UserWarning. Set MOCKPYSPARK_STRICT_VERSION=1 to raise instead.

So if your project pins pyspark==4.0.1 in pyproject.toml, the mock will refuse 4.1-only APIs without any configuration on your part.

Usage modes

The pytest plugin also accepts --no-mock-pyspark to bypass activation for a single invocation, useful for running the same suite against real PySpark for integration coverage.

See docs/usage.md for the full pattern, including switching between mock and real PySpark in the same suite, and known limitations.

Detecting the mock at runtime

import pyspark
if getattr(pyspark, "__mock__", False):
    print("running against mockpyspark")

Contributing

Run unit tests (no Java, no Docker):

git clone https://github.com/PeterDowdy/mock-pyspark.git && cd mock-pyspark
pip install pyarrow duckdb pandas numpy pytest
PYTHONPATH=. pytest -q -m unit

Run integration tests against real PySpark (requires Java 17):

# Via Docker (easiest)
./start.sh 4.1 --integration-test

# Or locally with a PySpark venv
python -m venv /opt/spark-venv && /opt/spark-venv/bin/pip install pyspark==4.1.0
PYTHONPATH=. PYSPARK_VERSION=4.1 PYSPARK_PYTHON=/opt/spark-venv/bin/python pytest -q -m integration

See docs/contributing.md for the full setup guide.

API coverage

Supports PySpark 4.0.x and 4.1.x — the complete PySpark 4.1 pyspark.sql.functions surface, full DataFrame/Column/Window/Catalog API, CSV/JSON/Parquet IO, Delta Lake (the delta.tables API plus Delta SQL: MERGE/UPDATE/OPTIMIZE/VACUUM/RESTORE/CLONE/time travel, on a built-in in-memory engine — no extra dependency — whose version follows the target PySpark: delta-spark 4.0.1 or 4.1.0) and Unity-Catalog-style three-level namespaces (catalogs, schemas, volumes, grants, information_schema).

Delta tables live in memory. The default Delta backend keeps each table's transaction log and data in the process: it is fast and needs no files, but a table's directory holds no _delta_log and no Parquet. Code that must read the files of a Delta table (with deltalake, pandas, or a real Spark) can select delta-rs instead, with pip install "mock-pyspark[delta]" and either MOCKPYSPARK_DELTA_BACKEND=deltalake or SparkSession.builder.config("spark.mock.delta.backend", "deltalake"). A Delta table that real Delta wrote to disk is read and written through delta-rs on either backend.

• STATUS-4.1.md — PySpark 4.1 coverage
• STATUS-4.0.md — PySpark 4.0 coverage