SparkConnector
Helper to connect to CERN's Spark Clusters
This extension is built as a Python module named sparkconnector, which simplifies the connection to Spark clusters.
It installs:
- an nbclassic-extension
- a Jupyterlab extension
- an iPython extension
Requirements
- JupyterLab >= 4.0.0
- pyspark (not installed by default)
Install
To install the extension, execute:
pip install sparkconnector
jupyter nbclassic-extension install sparkconnector --py
jupyter nbclassic-extension enable sparkconnector --py
It is also necessary to enable the iPython code. Append the following code to the config file (usually in ~/.ipython/profile_default/ipython_kernel_config.py, check here):
c.InteractiveShellApp.extensions.append('sparkconnector.connector')
Uninstall
To remove the extension, execute:
pip uninstall sparkconnector
Contributing
Development install
Note: You will need NodeJS to build the extension package.
The jlpm command is JupyterLab's pinned version of
yarn that is installed with JupyterLab. You may use
yarn or npm in lieu of jlpm below.
# Clone the repo to your local environment
# Change directory to the sparkconnector directory
# Install package in development mode
pip install -e "."
# Link your development version of the extension with JupyterLab
jupyter labextension develop . --overwrite
# Rebuild extension Typescript source after making changes
jlpm build
You can watch the source directory and run JupyterLab at the same time in different terminals to watch for changes in the extension's source and automatically rebuild the extension.
# Watch the source directory in one terminal, automatically rebuilding when needed
jlpm watch
# Run JupyterLab in another terminal
jupyter lab
With the watch command running, every saved change will immediately be built locally and available in your running JupyterLab. Refresh JupyterLab to load the change in your browser (you may need to wait several seconds for the extension to be rebuilt).
By default, the jlpm build command generates the source maps for this extension to make it easier to debug using the browser dev tools. To also generate source maps for the JupyterLab core extensions, you can run the following command:
jupyter lab build --minimize=False
Development uninstall
pip uninstall sparkconnector
In development mode, you will also need to remove the symlink created by jupyter labextension develop
command. To find its location, you can run jupyter labextension list to figure out where the labextensions
folder is located. Then you can remove the symlink named @swan-cern/sparkconnector within that folder.
Packaging the extension
See RELEASE
1 maintainer
Metadata
1 maintainer
Release files for sparkconnector 3.0.18
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sparkconnector-3.0.18.tar.gz | 386.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sparkconnector-3.0.18-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 789.3 kB
Release files / sparkconnector-3.0.18.tar.gz
| Download URL | sparkconnector-3.0.18.tar.gz |
|---|---|
| Size | 386.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b2bef7474dffcaecc0ef45276a0240b420c73a274fc01704f66da9b9b32500ad
|
|
BLAKE2b-256 checksum How to use checksums |
77169a233aa2b6b7ca097925efa18a6902822bb6d17fdbf5726e14323a383db4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency logRelease files / sparkconnector-3.0.18-py3-none-any.whl
| Download URL | sparkconnector-3.0.18-py3-none-any.whl |
|---|---|
| Size | 402.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
40e846675670cf6f051db834ec7b5a4f62e2989887c54542ffcbbb4a2f3907e7
|
|
BLAKE2b-256 checksum How to use checksums |
62e1f362c5870fb457f5e204560158e73dfb7c918889b6fcef7aaa5d8baf2188
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency log
