mockpyspark

A drop-in mock for Apache PySpark 4.x for unit and integration testing. Runs entirely in Python — no JVM, no Hadoop, no cluster — using PyArrow as the compute engine and DuckDB for SQL.

pip install mock-pyspark

mockpyspark (the PyPI distribution is mock-pyspark; you still import mockpyspark) installs alongside real pyspark; there is no namespace collision. Tests opt in via a one-liner so that from pyspark.sql import ... in your production code transparently resolves to the mock.

Quick start

# conftest.py
import mockpyspark
mockpyspark.activate()

# tests/test_my_pipeline.py
from pyspark.sql import SparkSession, functions as F

def test_normalize(spark=None):
    spark = SparkSession.builder.getOrCreate()
    df = spark.createDataFrame([
        {"id": 1, "name": "  alice ", "score": 10},
        {"id": 2, "name": "BOB",      "score": 20},
    ])
    out = (
        df.withColumn("name", F.initcap(F.trim(F.col("name"))))
          .filter(F.col("score") > 15)
          .select("id", "name", "score")
          .collect()
    )
    assert out == [(2, "Bob", 20)]

No changes to your Spark logic. The alias just makes pyspark.* imports resolve to mockpyspark.* for the test process.

Version targeting

mockpyspark supports both PySpark 4.0.x and 4.1.x. APIs that only exist in 4.1 (e.g. current_time, TimeType) raise AttributeError when running in 4.0 mode.

It picks a target version automatically, in order:

1. PYSPARK_VERSION env var
2. The pyspark pin in pyproject.toml, requirements*.txt, poetry.lock, uv.lock, Pipfile.lock, or setup.cfg (walking up from the working directory)
3. SparkSession.builder.config("spark.mock.targetVersion", "4.0").getOrCreate() — per-session override
4. If nothing is found, defaults to the latest supported version and emits a UserWarning. Set MOCKPYSPARK_STRICT_VERSION=1 to raise instead.

So if your project pins pyspark==4.0.1 in pyproject.toml, the mock will refuse 4.1-only APIs without any configuration on your part.

Usage modes

The pytest plugin also accepts --no-mock-pyspark to bypass activation for a single invocation, useful for running the same suite against real PySpark for integration coverage.

See docs/usage.md for the full pattern, including switching between mock and real PySpark in the same suite, and known limitations.

Detecting the mock at runtime

import pyspark
if getattr(pyspark, "__mock__", False):
    print("running against mockpyspark")

Contributing

Run unit tests (no Java, no Docker):

git clone https://github.com/PeterDowdy/mock-pyspark.git && cd mock-pyspark
pip install pyarrow "duckdb>=2,<3" pandas numpy pytest
PYTHONPATH=. pytest -q -m unit

Run integration tests against real PySpark (requires Java 17):

# Via Docker (easiest)
./start.sh 4.1 --integration-test

# Or locally with a PySpark venv
python -m venv /opt/spark-venv && /opt/spark-venv/bin/pip install pyspark==4.1.0
PYTHONPATH=. PYSPARK_VERSION=4.1 PYSPARK_PYTHON=/opt/spark-venv/bin/python pytest -q -m integration

See docs/contributing.md for the full setup guide.

API coverage

Supports PySpark 4.0.x and 4.1.x — the complete PySpark 4.1 pyspark.sql.functions surface, full DataFrame/Column/Window/Catalog API, CSV/JSON/Parquet IO, Delta Lake (the delta.tables API plus Delta SQL: MERGE/UPDATE/OPTIMIZE/VACUUM/RESTORE/CLONE/time travel, on a built-in in-memory engine — no extra dependency — whose version follows the target PySpark: delta-spark 4.0.1 or 4.1.0) and Unity-Catalog-style three-level namespaces (catalogs, schemas, volumes, grants, information_schema).

Delta tables live in memory. The default Delta backend keeps each table's transaction log and data in the process: it is fast and needs no files, but a table's directory holds no _delta_log and no Parquet. Code that must read the files of a Delta table (with deltalake, pandas, or a real Spark) can select delta-rs instead, with pip install "mock-pyspark[delta]" and either MOCKPYSPARK_DELTA_BACKEND=deltalake or SparkSession.builder.config("spark.mock.delta.backend", "deltalake"). A Delta table that real Delta wrote to disk is read and written through delta-rs on either backend.

S3. s3://, s3a:// and s3n:// paths work for the file formats (CSV, JSON, Parquet, text) and for Delta tables, through boto3 (pip install "mock-pyspark[s3]"), configured as in Spark: spark.hadoop.fs.s3a.endpoint, .access.key, .secret.key, .path.style.access, .connection.ssl.enabled, .endpoint.region (also per bucket, fs.s3a.bucket.<name>.*, and through sc._jsc.hadoopConfiguration()). That reaches a local moto server or MinIO; with no endpoint set, boto3's own credential chain applies, which is what an in-process moto.mock_aws() provides.

Nothing is written to S3 (or moto) unless you ask. By default what is written to an s3:// path stays in the process - a file is read back from there, with the modes and partition layout of a write to a bucket, and a Delta table lives in memory - so tests pay nothing for S3. Set spark.mock.s3.writes.enabled to true (SparkSession.builder.config(...), spark.conf.set(...), or the environment variable MOCKPYSPARK_S3_WRITES=true) to have writes reach the bucket: files are uploaded, and a Delta table there is a real one (_delta_log/*.json and Parquet files that real Spark can read) whose commits by other writers are followed. Reading what a bucket already holds needs no flag. With the flag on and a bucket that cannot be reached at all (no credentials, no server) a Delta table stays in memory; a session that names an endpoint which does not answer gets the error.

A streaming query reads s3:// paths too (readStream.text("s3://bucket/"), .json("s3://bucket/prefix"): the objects are listed on each trigger, so new ones are picked up), and its checkpoint can be at an s3:// path: it is kept in the process without the flag, and in the bucket with it, where deleting the checkpoint's objects restarts the stream from the beginning, as with real Spark.

• STATUS-4.1.md — PySpark 4.1 coverage
• STATUS-4.0.md — PySpark 4.0 coverage