Jupyter metakernel for apache spark and scala
This is an extremely early proof of concept for using the metakernel in combination with py4j to make a simpler kernel for scala.
On python 3.5+
pip install .
Installing the jupyter kernel
To install the jupyter kernel install it using
python -m spylon_kernel install
Using the kernel
The scala spark metakernl prodived a scala kernel by default. At the first scala cell that is run a spark session will be constructed so that a user can interact with the interpreter.
Customizing the spark context
The launch arguments can be customized using the %%init_spark magic as follows
%%init_spark launcher.jars = ["file://some/jar.jar"] launcher.master = "local" launcher.conf.spark.executor.cores = 8
Since this makes use of metakernel you can evaluate normal python code using the %%python magic. In addition once the spark context has been created the spark variable will be added to your python ernvironment.
%%python df = spark.read.json("examples/src/main/resources/people.json")
To get completions for python, make sure that you have installed jedi
Using as a magic
Spylon-kernel can be used as a magic in an existing ipykernel. This is the recommended solution when you want to write relatively small blocks of scala.
from spylon_kernel import register_ipython_magics register_ipython_magics()
%%scala val x = 8 x
Using as a library
If you just want to send a string of scala code to the interpreter and evaluate it you can do that too.
from spylon_kernel import get_scala_interpreter interp = get_scala_interpreter() # Evaluate the result of a scala code block. interp.interpret(""" val x = 8 x """) interp.last_result()
Release history Release notifications | RSS feed
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size spylon-kernel-0.1.1.tar.gz (30.9 kB)||File type Source||Python version None||Upload date||Hashes View|