kbatch-papermill 0.1.0a1

KBatch Papermill: Run papermill in kbatch

kbatch-papermill

Currently designed for Destination-Earth GFTS, it runs notebooks with papermill on Kubernetes via kbatch. It does not currently target general use because the following assumptions, specific to GFTS deployments, are made:

1. Default AWS credentials are set up via environment variables, and work.
2. Jobs should always run with the same $JUPYTER_IMAGE as the submitting environment.
3. $JUPYTER_IMAGE has papermill.
4. We have read/write access to S3 for both the code input directory and the output directory (completed job results).

Note that we do not use the ConfigMap approach to pass the code directory, because of the size limit on config maps. So we essentially replicate the code directory functionality of kbatch, but store in S3 instead.

We also add some generic functionality to make a nicer Python API for kbatch, which should perhaps be upstreamed. See _kbatch.py for most of that.