Skip to main content

Jupyter metakernel for apache spark and scala

Project description

# spylon-kernel [![Build Status](https://travis-ci.org/mariusvniekerk/spylon-kernel.svg?branch=master)](https://travis-ci.org/mariusvniekerk/spylon-kernel) [![codecov](https://codecov.io/gh/mariusvniekerk/spylon-kernel/branch/master/graph/badge.svg)](https://codecov.io/gh/mariusvniekerk/spylon-kernel)

This is an extremely early proof of concept for using the metakernel in combination with py4j to make a simpler kernel for scala.

## Installation

On python 3.5+

`bash pip install . `

## Installing the jupyter kernel

To install the jupyter kernel install it using

` python -m spylon_kernel install `

## Using the kernel

The scala spark metakernl prodived a scala kernel by default. At the first scala cell that is run a spark session will be constructed so that a user can interact with the interpreter.

## Customizing the spark context

The launch arguments can be customized using the %%init_spark magic as follows

`python %%init_spark launcher.jars = ["file://some/jar.jar"] launcher.master = "local[4]" launcher.conf.spark.executor.cores = 8 `

## Other languages

Since this makes use of metakernel you can evaluate normal python code using the %%python magic. In addition once the spark context has been created the spark variable will be added to your python ernvironment.

`python %%python df = spark.read.json("examples/src/main/resources/people.json") `

To get completions for python, make sure that you have installed jedi

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for spylon-kernel, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size spylon_kernel-0.0.1-py2.py3-none-any.whl (13.3 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size spylon-kernel-0.0.1.tar.gz (11.2 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page