Run your workflow on JupyterHub
Project description
JupyterFlow
Run your workflow on JupyterHub!
What is JupyterFlow?
Run Argo Workflow on JupyterHub with single command.
- No Kubernetes knowledge (YAML) needed to run.
- No container image build & push or deploy.
- Just simply run your workflow with single command
jupyterflow
.
jupyterflow
is a single command that helps user utilize Argo workflow engine without making any YAML files or building containers on JupyterHub. The following jupyterflow
command will make sequence workflow for you. That's it!
jupyterflow run -c "python hello.py >> python world.py"
To make parallel workflow, write your own workflow.yaml
file.
Problem to solve
- I wanted to train multiple ML models efficiently.
- Using Kubernetes was a good idea, since
- it is easy to make distributed jobs.
- it is easy to schedule ML jobs on multiple training server.
- it has native resource management mechanism.
- it has good monitoring system.
- But there were some drawbacks.
- I needed to re-build & re-push image everytime I updated my model. This was painful.
- People who were not familiar with k8s had a hard time using this method.
JupyterFlow aims to solve this problem. Run your workflow on JupyterHub with single command without Kubernetes & container troublesome task.
Limitation
JupyterFlow only works on JupyterHub deployed on Kubernetes.
Therefore, although using JupyterFlow does not require Kubernetes knowledge, setting up JupyterFlow requires Kubernetes understandings(YAML, helm
, Service
). If you're familiar with Kubernetes, it will not be too hard.
Getting Started
To set up jupyterflow
and start running your first workflow, follow the Getting Started guide.
How does it work
To learn how it works, go to How it works guide.
Examples
For examples how to use, please see Examples page.
Configuration
To find out more configuration, take a look at Configuration page.
CLI Reference
For more detail usage of jupyterflow
command line interface, find out more at CLI Reference page.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file jupyterflow-0.0.4.tar.gz
.
File metadata
- Download URL: jupyterflow-0.0.4.tar.gz
- Upload date:
- Size: 9.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.23.0 setuptools/46.4.0.post20200518 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.8.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 625393cbf151f8083c5bcf62e6390772e08dd5aaa414d059626011dc659a42e9 |
|
MD5 | 94eae57b71f80a5e597c94b2cd7a78ea |
|
BLAKE2b-256 | beaa5e15ea1b9da92622624a1df4761a0daa732a438c55d501e8e1c699b8d3b0 |
File details
Details for the file jupyterflow-0.0.4-py3-none-any.whl
.
File metadata
- Download URL: jupyterflow-0.0.4-py3-none-any.whl
- Upload date:
- Size: 23.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.23.0 setuptools/46.4.0.post20200518 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.8.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb7fb64b717e95d550ee4d434a24426b8bc330c9462a5631102918d6a8f17f35 |
|
MD5 | d88dd723f8ef3787a0cc41c3b3075e3a |
|
BLAKE2b-256 | d441e77361118ef172b0747b721340b8e1bea3136b84a6cd7580b4754c922940 |