A pyspark management framework
Project description
Spark Management Consolidated
=============================
A small module that will load as a singleton class object to manage Spark
related things.
Installation
------------
Directly via ``pip`` on the command line, in a `virtualenv`:
.. code:: shell
pip install https://github.com/matz-e/sparkmanager/tarball/master
or for the current user:
.. code:: shell
pip install --user https://github.com/matz-e/sparkmanager/tarball/master
Usage
-----
The module itself acts as a mediator to Spark:
.. code:: python
import sparkmanager as sm
# Create a new application
sm.create("My fancy name",
[("spark.executor.cores", 4), ("spark.executor.memory", "8g")])
data = sm.spark.range(5)
# Will show up in the UI with the name "broadcasting some data"
with sm.jobgroup("broadcasting some data"):
data = sm.broadcast(data.collect())
The Spark session can be accessed via ``sm.spark``, the Spark context via
``sm.sc``. Both attributes are instantiated once the ``create`` method is
called, with the option to call unambiguous methods from both directly via
the :py:class:`SparkManager` object:
.. code:: python
# The following two calls are equivalent
c = sm.parallelize(range(5))
d = sm.sc.parallelize(range(5))
assert c.collect() == d.collect()
Cluster support scripts
-----------------------
.. note::
Scripts to run on the cluster are still somewhat experimental and should
be used with caution!
Environment setup
~~~~~~~~~~~~~~~~~
To create a self-contained Spark environment, the script provided in
``examples/env.sh`` can be used. It is currently tuned to the requirements of
the `bbpviz` cluster. A usage example:
.. code:: shell
SPARK_ROOT=/path/to/my/spark/installation SM_WORKDIR=/path/to/a/work/directory examples/env.sh
The working directory will contain:
* A Python virtual environment
* A basic Spark configuration pointing to directories within the working
directory
* An environment script to establish the setup
To use the resulting working environment:
.. code:: shell
. /path/to/a/work/directory/env.sh
Spark deployment on allocations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Within a cluster allocation, the script ``sm_cluster`` can be used to start
a Spark cluster. The script will be automatically installed by `pip`. To
use it, pass either a working directory containing an environment or
specify them separately:
.. code:: shell
sm_cluster startup $WORKDIR
sm_cluster startup $WORKDIR /path/to/some/env.sh
Similar, to stop a cluster (not necessary with slurm):
.. code:: shell
sm_cluster shutdown $WORKDIR
sm_cluster shutdown $WORKDIR /path/to/some/env.sh
Spark applications then can connect to a master found via:
.. code:: shell
cat $WORKDIR/spark_master
TL;DR on BlueBrain 5
~~~~~~~~~~~~~~~~~~~~
Setup a Spark environment in your current shell, and point `WORKDIR` to a
shared directory. `SPARK_HOME` needs to be in your environment and point to
your Spark installation. By default, only a file with the Spark master and
the cluster launch script will be copied to `WORKDIR`. Then submit a
cluster with:
.. code:: shell
sbatch -A proj16 -t 24:00:00 -N4 --exclusive -C nvme $(which sm_cluster) startup $WORKDIR
=============================
A small module that will load as a singleton class object to manage Spark
related things.
Installation
------------
Directly via ``pip`` on the command line, in a `virtualenv`:
.. code:: shell
pip install https://github.com/matz-e/sparkmanager/tarball/master
or for the current user:
.. code:: shell
pip install --user https://github.com/matz-e/sparkmanager/tarball/master
Usage
-----
The module itself acts as a mediator to Spark:
.. code:: python
import sparkmanager as sm
# Create a new application
sm.create("My fancy name",
[("spark.executor.cores", 4), ("spark.executor.memory", "8g")])
data = sm.spark.range(5)
# Will show up in the UI with the name "broadcasting some data"
with sm.jobgroup("broadcasting some data"):
data = sm.broadcast(data.collect())
The Spark session can be accessed via ``sm.spark``, the Spark context via
``sm.sc``. Both attributes are instantiated once the ``create`` method is
called, with the option to call unambiguous methods from both directly via
the :py:class:`SparkManager` object:
.. code:: python
# The following two calls are equivalent
c = sm.parallelize(range(5))
d = sm.sc.parallelize(range(5))
assert c.collect() == d.collect()
Cluster support scripts
-----------------------
.. note::
Scripts to run on the cluster are still somewhat experimental and should
be used with caution!
Environment setup
~~~~~~~~~~~~~~~~~
To create a self-contained Spark environment, the script provided in
``examples/env.sh`` can be used. It is currently tuned to the requirements of
the `bbpviz` cluster. A usage example:
.. code:: shell
SPARK_ROOT=/path/to/my/spark/installation SM_WORKDIR=/path/to/a/work/directory examples/env.sh
The working directory will contain:
* A Python virtual environment
* A basic Spark configuration pointing to directories within the working
directory
* An environment script to establish the setup
To use the resulting working environment:
.. code:: shell
. /path/to/a/work/directory/env.sh
Spark deployment on allocations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Within a cluster allocation, the script ``sm_cluster`` can be used to start
a Spark cluster. The script will be automatically installed by `pip`. To
use it, pass either a working directory containing an environment or
specify them separately:
.. code:: shell
sm_cluster startup $WORKDIR
sm_cluster startup $WORKDIR /path/to/some/env.sh
Similar, to stop a cluster (not necessary with slurm):
.. code:: shell
sm_cluster shutdown $WORKDIR
sm_cluster shutdown $WORKDIR /path/to/some/env.sh
Spark applications then can connect to a master found via:
.. code:: shell
cat $WORKDIR/spark_master
TL;DR on BlueBrain 5
~~~~~~~~~~~~~~~~~~~~
Setup a Spark environment in your current shell, and point `WORKDIR` to a
shared directory. `SPARK_HOME` needs to be in your environment and point to
your Spark installation. By default, only a file with the Spark master and
the cluster launch script will be copied to `WORKDIR`. Then submit a
cluster with:
.. code:: shell
sbatch -A proj16 -t 24:00:00 -N4 --exclusive -C nvme $(which sm_cluster) startup $WORKDIR
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
sparkmanager-0.5.8.tar.gz
(12.5 kB
view hashes)
Built Distribution
Close
Hashes for sparkmanager-0.5.8-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 096f9bbae2e27e7ff6b462c0cfbce89e590634c2bf7ae58ae568e682b30fcb5e |
|
MD5 | 8fde3acefe0406abae979f92bf9f610b |
|
BLAKE2b-256 | 8f2f7a331d2b5f922a374fbf11b66aa014c7e084fbb2065688a2f2400d1c9c33 |