Skip to main content

Spark acceleration for Scikit-Learn

This project is a major re-write of the spark-sklearn project, which seems to no longer be under development. It focuses specifically on the acceleration of Scikit-Learn's cross validation functionality using PySpark.

Improvements over spark-sklearn

scikit-spark supports scikit-learn versions past 0.19, spark-sklearn have stated that they are probably not going to support newer versions.

The functionality in scikit-spark is based on sklearn.model_selection module rather than the deprecated and soon to be removed sklearn.grid_search. The new model_selection versions contain several nicer features and scikit-spark maintains full compatibility.

Installation

The package can be installed through pip:

pip install scikit-spark

# or to also install with pyspark
pip install scikit-spark[spark]

It is currently tested with Spark 3.3.2 and 3.4.0, but should work with older versions (e.g. Spark 2.4.x). Python versions 3.10 and up are also currently tested, though 3.11 is only supported with PySpark 3.4.0.

Supported scikit-learn versions

  • 1.2.2

Older versions of scikit-learn

The following pre-1.0 releases of scikit-learn are supported by scikit-spark==0.4.0:

  • 0.19
  • 0.20
  • 0.21 (Python 3 only)
  • 0.22 (Python 3 only)

Usage

The functionality here is meant to as closely resemble using Scikit-Learn as possible. By default (with spark=True) the SparkSession is obtained internally by calling SparkSession.builder.getOrCreate(), so the instantiation and calling of the functions is the same (You will preferably have already created a SparkSession).

This example is adapted from the Scikit-Learn documentation. It instantiates a local SparkSession, and distributes the cross validation folds and iterations using this. In actual use, to get the benefit of this package it should be used distributed across several machines with Spark as running it locally is slower than the Scikit-Learn parallelisation implementation.

from sklearn import svm, datasets
from pyspark.sql import SparkSession

iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[0.01, 0.1, 1, 10, 100]}
svc = svm.SVC()

spark = SparkSession.builder\
    .master("local[*]")\
    .appName("skspark-grid-search-doctests")\
    .getOrCreate()

# How to run grid search
from skspark.model_selection import GridSearchCV

gs = GridSearchCV(svc, parameters, spark=spark)
gs.fit(iris.data, iris.target)

# How to run random search
from skspark.model_selection import RandomizedSearchCV

rs = RandomizedSearchCV(svc, parameters, spark=spark)
rs.fit(iris.data, iris.target)

Current and upcoming functionality

  • Current
    • model_selection.RandomizedSearchCV
    • model_selection.GridSearchCV
  • Upcoming
    • model_selection.cross_val_predict
    • model_selection.cross_val_score

The docstrings are modifications of the Scikit-Learn ones and are still being converted to specifically refer to this project.

Performance optimisations

Reducing RAM usage

Coming soon

Metadata

Release files for scikit-spark 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scikit-spark 1.0.0
File Size Uploaded Provenance Actions
scikit_spark-1.0.0.tar.gz 96.1 kB With provenance Details for scikit_spark-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scikit-spark 1.0.0
File Interpreter ABI Platform Actions
scikit_spark-1.0.0-py3-none-any.whl Python 3 none any Details for scikit_spark-1.0.0-py3-none-any.whl

Total release size: 198.9 kB

Release files / scikit_spark-1.0.0.tar.gz

Download URL scikit_spark-1.0.0.tar.gz
Size 96.1 kB
Tags Source
SHA-256 checksum
How to use checksums
c8008ad72d5a847cf44ba01bf6f70e4057e919a52d53e7bb154b445ff6af353d
BLAKE2b-256 checksum
How to use checksums
8d9dd1419b84663b54e5686d9353ec29753bcd03304939d4530cf23a7f9f111e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 10, 2026.

Transparency log

Release files / scikit_spark-1.0.0-py3-none-any.whl

Download URL scikit_spark-1.0.0-py3-none-any.whl
Size 102.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
581277f275ead7d6d56908a3cf67a6ea459afaa04e8b321a26c5a4bd9c70c54f
BLAKE2b-256 checksum
How to use checksums
ed94331e2f7d703d7e772c8c7ba78ffe53382effa61f412f100b20786a1929f4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 10, 2026.

Transparency log

Security for scikit-spark 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for scikit_spark-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page