Spark acceleration for Scikit-Learn
This project is a major re-write of the spark-sklearn project, which seems to no longer be under development. It focuses specifically on the acceleration of Scikit-Learn's cross validation functionality using PySpark.
Improvements over spark-sklearn
scikit-spark supports scikit-learn versions past 0.19, spark-sklearn have stated that they are probably not
going to support newer versions.
The functionality in scikit-spark is based on sklearn.model_selection module rather than the
deprecated and soon to be removed sklearn.grid_search. The new model_selection versions
contain several nicer features and scikit-spark maintains full compatibility.
Installation
The package can be installed through pip:
pip install scikit-spark
# or to also install with pyspark
pip install scikit-spark[spark]
It is currently tested with Spark 3.3.2 and 3.4.0, but should work with older versions (e.g. Spark 2.4.x). Python versions 3.10 and up are also currently tested, though 3.11 is only supported with PySpark 3.4.0.
Supported scikit-learn versions
- 1.2.2
Older versions of scikit-learn
The following pre-1.0 releases of scikit-learn are supported by scikit-spark==0.4.0:
- 0.19
- 0.20
- 0.21 (Python 3 only)
- 0.22 (Python 3 only)
Usage
The functionality here is meant to as closely resemble using Scikit-Learn as
possible. By default (with spark=True) the SparkSession is obtained
internally by calling SparkSession.builder.getOrCreate(), so the instantiation
and calling of the functions is the same (You will preferably have already
created a SparkSession).
This example is adapted from the Scikit-Learn documentation. It instantiates
a local SparkSession, and distributes the cross validation folds and
iterations using this. In actual use, to get the benefit of this package it
should be used distributed across several machines with Spark as running it
locally is slower than the Scikit-Learn parallelisation implementation.
from sklearn import svm, datasets
from pyspark.sql import SparkSession
iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[0.01, 0.1, 1, 10, 100]}
svc = svm.SVC()
spark = SparkSession.builder\
.master("local[*]")\
.appName("skspark-grid-search-doctests")\
.getOrCreate()
# How to run grid search
from skspark.model_selection import GridSearchCV
gs = GridSearchCV(svc, parameters, spark=spark)
gs.fit(iris.data, iris.target)
# How to run random search
from skspark.model_selection import RandomizedSearchCV
rs = RandomizedSearchCV(svc, parameters, spark=spark)
rs.fit(iris.data, iris.target)
Current and upcoming functionality
- Current
- model_selection.RandomizedSearchCV
- model_selection.GridSearchCV
- Upcoming
- model_selection.cross_val_predict
- model_selection.cross_val_score
The docstrings are modifications of the Scikit-Learn ones and are still being converted to specifically refer to this project.
Performance optimisations
Reducing RAM usage
Coming soon
Metadata
Release files for scikit-spark 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scikit_spark-1.0.0.tar.gz | 96.1 kB | With provenance | Details for scikit_spark-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| scikit_spark-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for scikit_spark-1.0.0-py3-none-any.whl |
Total release size: 198.9 kB
Release files / scikit_spark-1.0.0.tar.gz
| Download URL | scikit_spark-1.0.0.tar.gz |
|---|---|
| Size | 96.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c8008ad72d5a847cf44ba01bf6f70e4057e919a52d53e7bb154b445ff6af353d
|
|
BLAKE2b-256 checksum How to use checksums |
8d9dd1419b84663b54e5686d9353ec29753bcd03304939d4530cf23a7f9f111e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 10, 2026.
Transparency logRelease files / scikit_spark-1.0.0-py3-none-any.whl
| Download URL | scikit_spark-1.0.0-py3-none-any.whl |
|---|---|
| Size | 102.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
581277f275ead7d6d56908a3cf67a6ea459afaa04e8b321a26c5a4bd9c70c54f
|
|
BLAKE2b-256 checksum How to use checksums |
ed94331e2f7d703d7e772c8c7ba78ffe53382effa61f412f100b20786a1929f4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 10, 2026.
Transparency logSecurity for scikit-spark 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for scikit_spark-1.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.