dapla-ipython-kernels

Python module for use within Jupyter notebooks. It contains kernel extensions for integrating with Apache Spark, Google Cloud Storage and custom dapla services.

Getting Started

Install the module from pip:

# pip
pip install dapla-ipython-kernels

Now the module is ready to use with a single import:

import dapla as dp

This module is targeted to python kernels in Jupyter, but it may work in any IPython environment. It also depends on a number of custom services, e.g. the custom auth service

To test, simply create any Pandas dataframe. This can be stored in Google Cloud Storage at a specific path:

import pandas as pd
import dapla as dp

data = {
    'apples': [3, 2, 0, 1], 
    'oranges': [0, 3, 7, 2]
}
# Create pandas DataFrame
purchases = pd.DataFrame(data, index=['June', 'Robert', 'Lily', 'David'])

# Write pandas DataFrame to parquet
dp.write_pandas(purchases, '/testfolder/python/purchases', valuation='INTERNAL', state= 'INPUT')

Conversely, parquet files can be read from a path directly into a pandas DataFrame.

import dapla as dp
# Read path into pandas dataframe 
purchases = dp.read_pandas('/testfolder/python/purchases')

Other functions

Since the python module integrates with Google Cloud Storage and custom dapla services, some other functions exist as well:

import dapla as dp

# List path by prefix
dp.show('/testfolder/python')

import dapla as dp

# Show file details
dp.details('/testfolder/python/purchases')

See also the example notebook written for Jupyter.

Deploy to SSB jupyter

Release version pypi

Make sure you have a clean master branch.
run make bump-version-patch - this will update version and commit to git.
run git push --tags origin master - important to have --tags to make it auto deploy to pypi

If everything was ok we should see a new release her: https://pypi.org/project/ssb-ipython-kernels/

Update jupyter image on staging

• Bump ssb-ipython-kernels in dapla-gcp-jupyter Dockerfile
  
  • Example of previous update
    
• Check new tag from build on azure piplines
• update platform dev jupyter-kubespawner-config with tag
  • Example

For now, we have to delete the running jupyer hub instance to make it use this new config