Spark Extension

This project provides extensions to the Apache Spark project in Scala and Python:

Diff: A diff transformation and application for Datasets that computes the differences between two datasets, i.e. which rows to add, delete or change to get from one dataset to the other.

Histogram: A histogram transformation that computes the histogram DataFrame for a value column.

Global Row Number: A withRowNumbers transformation that provides the global row number w.r.t. the current order of the Dataset, or any given order. In contrast to the existing SQL function row_number, which requires a window spec, this transformation provides the row number across the entire Dataset without scaling problems.

Inspect Parquet files: The structure of Parquet files (the metadata, not the data stored in Parquet) can be inspected similar to parquet-tools or parquet-cli by reading from a simple Spark data source. This simplifies identifying why some Parquet files cannot be split by Spark into scalable partitions.

Install Python packages into PySpark job: Install Python dependencies via PIP or Poetry programatically into your running PySpark job (PySpark ≥ 3.1.0):

# noinspection PyUnresolvedReferences
from gresearch.spark import *

# using PIP
spark.install_pip_package("pandas==1.4.3", "pyarrow")
spark.install_pip_package("-r", "requirements.txt")

# using Poetry
spark.install_poetry_project("../my-poetry-project/", poetry_python="../venv-poetry/bin/python")

Count null values: count_null(e: Column): an aggregation function like count that counts null values in column e. This is equivalent to calling count(when(e.isNull, lit(1))).

.Net DateTime.Ticks: Convert .Net (C#, F#, Visual Basic) DateTime.Ticks into Spark timestamps, seconds and nanoseconds.

Available methods:

dotnet_ticks_to_timestamp(column_or_name)         # returns timestamp as TimestampType
dotnet_ticks_to_unix_epoch(column_or_name)        # returns Unix epoch seconds as DecimalType
dotnet_ticks_to_unix_epoch_nanos(column_or_name)  # returns Unix epoch nanoseconds as LongType

The reverse is provided by (all return LongType .Net ticks):

timestamp_to_dotnet_ticks(column_or_name)
unix_epoch_to_dotnet_ticks(column_or_name)
unix_epoch_nanos_to_dotnet_ticks(column_or_name)

Spark temporary directory: Create a temporary directory that will be removed on Spark application shutdown.

Example:

# noinspection PyUnresolvedReferences
from gresearch.spark import *

dir = spark.create_temporary_dir("prefix")

Spark job description: Set Spark job description for all Spark jobs within a context.

Example:

from gresearch.spark import job_description, append_job_description

with job_description("parquet file"):
    df = spark.read.parquet("data.parquet")
    with append_job_description("count"):
        count = df.count
    with append_job_description("write"):
        df.write.csv("data.csv")

For details, see the README.md at the project homepage.

Using Spark Extension

PyPi package (local Spark cluster only)

You may want to install the pyspark-extension python package from PyPi into your development environment. This provides you code completion, typing and test capabilities during your development phase.

Running your Python application on a Spark cluster will still require one of the ways below to add the Scala package to the Spark environment.

pip install pyspark-extension==2.15.1.3.4

Note: Pick the right Spark version (here 3.4) depending on your PySpark version.

PySpark API

Start a PySpark session with the Spark Extension dependency (version ≥1.1.0) as follows:

from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .config("spark.jars.packages", "uk.co.gresearch.spark:spark-extension_2.12:2.15.1-3.4") \
    .getOrCreate()

Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your PySpark version.

PySpark REPL

Launch the Python Spark REPL with the Spark Extension dependency (version ≥1.1.0) as follows:

pyspark --packages uk.co.gresearch.spark:spark-extension_2.12:2.15.1-3.4

Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your PySpark version.

PySpark `spark-submit`

Run your Python scripts that use PySpark via spark-submit:

spark-submit --packages uk.co.gresearch.spark:spark-extension_2.12:2.15.1-3.4 [script.py]

Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your Spark version.

Your favorite Data Science notebook

There are plenty of Data Science notebooks around. To use this library, add a jar dependency to your notebook using these Maven coordinates:

uk.co.gresearch.spark:spark-extension_2.12:2.15.1-3.4

Or download the jar and place it on a filesystem where it is accessible by the notebook, and reference that jar file directly.

Check the documentation of your favorite notebook to learn how to add jars to your Spark environment.

Metadata

Release files for pyspark-extension 2.15.1.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyspark-extension 2.15.1.4.2
File	Size	Uploaded
pyspark_extension-2.15.1.4.2.tar.gz	366.9 kB	Sep 10, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for pyspark-extension 2.15.1.4.2
File	Interpreter	ABI	Platform	Reset
pyspark_extension-2.15.1.4.2-py3-none-any.whl 351.6 kB Sep 10, 2026	Python 3	none	any	Details

Total release size: 718.5 kB

Release files / pyspark_extension-2.15.1.4.2.tar.gz

Download URL	pyspark_extension-2.15.1.4.2.tar.gz
Size	366.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`06f1f48b2c6449c31af684702b575530169c966a93cfb7fa7e615432a69f2544`
BLAKE2b-256 checksum How to use checksums	`5bee358d8ef76ba0ce1bb24f4f7b4898be683c5c392feb8e65c75e8135364f9c`
Upload date	Sep 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.13.13`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/G-Research/spark-extension
Publishing commit: github.com/G-Research/spark-extension/tree/6fac23ec65193aadc918aa24fe9cb6d0bb760acc

Workflow

Publishing configuration: github.com/G-Research/spark-extension/blob/6fac23ec65193aadc918aa24fe9cb6d0bb760acc/.github/workflows/publish-release.yml
Publishing logs: github.com/G-Research/spark-extension/actions/runs/34508391621/attempts/1

Artifact

Subject: pyspark_extension-2.15.1.4.2.tar.gz
SHA-256 checksum: 06f1f48b2c6449c31af684702b575530169c966a93cfb7fa7e615432a69f2544 Verifying attestations

Release files / pyspark_extension-2.15.1.4.2-py3-none-any.whl

Download URL	pyspark_extension-2.15.1.4.2-py3-none-any.whl
Size	351.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`f539f13f18585de7faae80bc560bed30dc8893a36993e8757da1e8fee844055e`
BLAKE2b-256 checksum How to use checksums	`c9f0ed95827e524a78cdaedbe88c0659c292417571f73409e670f7369c39d84e`
Upload date	Sep 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.13.13`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/G-Research/spark-extension
Publishing commit: github.com/G-Research/spark-extension/tree/6fac23ec65193aadc918aa24fe9cb6d0bb760acc

Workflow

Publishing configuration: github.com/G-Research/spark-extension/blob/6fac23ec65193aadc918aa24fe9cb6d0bb760acc/.github/workflows/publish-release.yml
Publishing logs: github.com/G-Research/spark-extension/actions/runs/34508391621/attempts/1

Artifact

Subject: pyspark_extension-2.15.1.4.2-py3-none-any.whl
SHA-256 checksum: f539f13f18585de7faae80bc560bed30dc8893a36993e8757da1e8fee844055e Verifying attestations