Helper to connect to CERN's Spark Clusters
Project description
SparkMonitor
Prerequisites
- PySpark on Apache Spark version 2.1.1 or higher
- Jupyter Notebook version 4.4.0 or higher
Build from Source
Building the extension involves three parts:
- Bundle and minify the JavaScript
- Compile the Scala listener into a JAR file.
- Package and install the python package.
git clone https://github.com/swan-cern/jupyter-extensions
cd SparkMonitor
and then
#Build Javascript
yarn install
yarn run webpack
#Build SparkListener Scala jar
cd scalalistener/
sbt package
or
make
Install
- Install the python package. The python package contains the JavaScript resources and the listener jar file.
pip install .
- The frontend extension is symlinked (
--symlink
) into the jupyter configuration directory byjupyter nbextension
command. The second line configures the frontend extension to load on notebook startup.
jupyter nbextension install --py sparkmonitor --user --symlink
jupyter nbextension enable sparkmonitor --user --py
- Configure the server extension to load when the notebook server starts
jupyter serverextension enable --py --user sparkmonitor
- Create the default profile configuration files (Skip if config file already exists)
ipython profile create
- Configure the kernel to load the extension on startup. This is added to the configuration files in users home directory
echo "c.InteractiveShellApp.extensions.append('sparkmonitor.kernelextension')" >> $(ipython profile locate default)/ipython_kernel_config.py
Configuration
By default the Spark Web UI runs on localhost:4040
. If this is not the case, setting the environment variable SPARKMONITOR_UI_HOST
and SPARKMONITOR_UI_PORT
overrides the default Spark UI hostname localhost
and port 4040 used by the Spark UI proxy.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
sparkmonitor-1.0.0.tar.gz
(3.2 MB
view hashes)
Built Distribution
Close
Hashes for sparkmonitor-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 156d2e9ed7370b3bc91ab2ec6d341868eba9b5c845793bc941148af4915a99e3 |
|
MD5 | 33c14e09fd8d9af122de889329865019 |
|
BLAKE2b-256 | 4dca5b86b14fcb87c400db4d1a7adbbaa960b70af9b148f909ec6058facdcaf7 |