Skip to main content

Run your workflow on JupyterHub

Project description

JupyterFlow

Run your workflow on JupyterHub!

What is JupyterFlow?

Run Argo Workflow on JupyterHub with single command.

  • No Kubernetes knowledge (YAML) needed to run.
  • No container image build & push or deploy.
  • Just simply run your workflow with single command jupyterflow.

jupyterflow is a single command that helps user utilize Argo workflow engine without making any YAML files or building containers on JupyterHub. The following jupyterflow command will make sequence workflow for you. That's it!

jupyterflow run -c "python hello.py >> python world.py"

To make parallel workflow, write your own workflow.yaml file.

Problem to solve

  • I wanted to train multiple ML models efficiently.
  • Using Kubernetes was a good idea, since
    • it is easy to make distributed jobs.
    • it is easy to schedule ML jobs on multiple training server.
    • it has native resource management mechanism.
    • it has good monitoring system.
  • But there were some drawbacks.
    • I needed to re-build & re-push image everytime I updated my model. This was painful.
    • People who were not familiar with k8s had a hard time using this method.

JupyterFlow aims to solve this problem. Run your workflow on JupyterHub with single command without Kubernetes & container troublesome task.

Limitation

JupyterFlow only works on JupyterHub deployed on Kubernetes.

Therefore, although using JupyterFlow does not require Kubernetes knowledge, setting up JupyterFlow requires Kubernetes understandings(YAML, helm, Service). If you're familiar with Kubernetes, it will not be too hard.

Getting Started

To set up jupyterflow and start running your first workflow, follow the Getting Started guide.

How does it work

To learn how it works, go to How it works guide.

Examples

For examples how to use, please see Examples page.

Configuration

To find out more configuration, take a look at Configuration page.

CLI Reference

For more detail usage of jupyterflow command line interface, find out more at CLI Reference page.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

jupyterflow-0.0.4.tar.gz (9.3 kB view details)

Uploaded Source

Built Distribution

jupyterflow-0.0.4-py3-none-any.whl (23.6 kB view details)

Uploaded Python 3

File details

Details for the file jupyterflow-0.0.4.tar.gz.

File metadata

  • Download URL: jupyterflow-0.0.4.tar.gz
  • Upload date:
  • Size: 9.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.23.0 setuptools/46.4.0.post20200518 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.8.3

File hashes

Hashes for jupyterflow-0.0.4.tar.gz
Algorithm Hash digest
SHA256 625393cbf151f8083c5bcf62e6390772e08dd5aaa414d059626011dc659a42e9
MD5 94eae57b71f80a5e597c94b2cd7a78ea
BLAKE2b-256 beaa5e15ea1b9da92622624a1df4761a0daa732a438c55d501e8e1c699b8d3b0

See more details on using hashes here.

File details

Details for the file jupyterflow-0.0.4-py3-none-any.whl.

File metadata

  • Download URL: jupyterflow-0.0.4-py3-none-any.whl
  • Upload date:
  • Size: 23.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.23.0 setuptools/46.4.0.post20200518 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.8.3

File hashes

Hashes for jupyterflow-0.0.4-py3-none-any.whl
Algorithm Hash digest
SHA256 eb7fb64b717e95d550ee4d434a24426b8bc330c9462a5631102918d6a8f17f35
MD5 d88dd723f8ef3787a0cc41c3b3075e3a
BLAKE2b-256 d441e77361118ef172b0747b721340b8e1bea3136b84a6cd7580b4754c922940

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page