dapla-ipython-kernels

Python module for use within Jupyter notebooks. It contains kernel extensions for integrating with Apache Spark, Google Cloud Storage and custom dapla services.

Getting Started

Install the module from pip:

# pip
pip install dapla-ipython-kernels

Now the module is ready to use with a single import:

import dapla as dp

This module is targeted to python kernels in Jupyter, but it may work in any IPython environment. It also depends on a number of custom services, e.g. the custom auth service

To test, simply create any Pandas dataframe. This can be stored in Google Cloud Storage at a specific path:

import pandas as pd
import dapla as dp

data = {
    'apples': [3, 2, 0, 1], 
    'oranges': [0, 3, 7, 2]
}
# Create pandas DataFrame
purchases = pd.DataFrame(data, index=['June', 'Robert', 'Lily', 'David'])

# Write pandas DataFrame to parquet
dp.write_pandas(purchases, '/testfolder/python/purchases', valuation='INTERNAL', state= 'INPUT')

Conversely, parquet files can be read from a path directly into a pandas DataFrame.

import dapla as dp
# Read path into pandas dataframe 
purchases = dp.read_pandas('/testfolder/python/purchases')

Other functions

Since the python module integrates with Google Cloud Storage and custom dapla services, some other functions exist as well:

import dapla as dp

# List path by prefix
dp.show('/testfolder/python')

Path	Timestamp
/testfolder/python/purchases	1593120298095
/testfolder/python/other	1593157667793

import dapla as dp

# Show file details
dp.details('/testfolder/python/purchases')

Size	Name
2908	42331105444c9ca0ce049ef6de7160.parquet

See also the example notebook written for Jupyter.

Deploy to SSB jupyter

Release version pypi

Make sure you have a clean master branch.
run make bump-version-patch - this will update version and commit to git.
run git push --tags origin master - important to have --tags to make it auto deploy to pypi

If everything was ok we should see a new release her: https://pypi.org/project/ssb-ipython-kernels/

Update jupyter image on staging

Bump ssb-ipython-kernels in dapla-gcp-jupyter Dockerfile
- Example of previous update
Check new tag from build on azure piplines
update platform dev jupyter-kubespawner-config with tag
- Example

For now, we have to delete the running jupyer hub instance to make it use this new config

Release files for ssb-ipython-kernels 0.3.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ssb-ipython-kernels 0.3.3
File	Size	Uploaded
ssb-ipython-kernels-0.3.3.tar.gz	25.3 kB	Dec 1, 2021	Details

Built distribution (wheel)

Table of built distributions (wheels) for ssb-ipython-kernels 0.3.3
File	Interpreter	ABI	Platform	Reset
ssb_ipython_kernels-0.3.3-py3-none-any.whl 29.4 kB Dec 1, 2021	Python 3	none	any	Details

Total release size:54.7 kB

Release files / ssb-ipython-kernels-0.3.3.tar.gz

Download URL	ssb-ipython-kernels-0.3.3.tar.gz
Size	25.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`5fde0db08be21e8082cd70eaeeb9a832db54e9ef77467f2874bb12ee5f97be25`
BLAKE2b-256 checksum How to use checksums	`c1500dbd3944013356007b66c31ca1102cd26dbe060e7fd02b7699fa4bd1146d`
Upload date	Dec 1, 2021
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.6.0 importlib_metadata/4.8.2 pkginfo/1.8.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12`

Release files / ssb_ipython_kernels-0.3.3-py3-none-any.whl

Download URL	ssb_ipython_kernels-0.3.3-py3-none-any.whl
Size	29.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`15e84d565a0eeaaddef09cd28f54fe4fac92f3aba791aecf41b2991aa1153c83`
BLAKE2b-256 checksum How to use checksums	`13f7c67b79f7877844be3ec9b637cef1b2f64dbf1466c6fecbd12be9d76bab9b`
Upload date	Dec 1, 2021
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.6.0 importlib_metadata/4.8.2 pkginfo/1.8.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12`