Execute several jupyter cells simultaneously
Project description
igogo 🐎🏎️
Execute several jupyter cells at the same time
Have you ever just sited and watched a long-running jupyter cell? Now, you can continue to work in the same notebook freely
Use cases
- You have a long-running cell, and you need to check something.
You can just start the second cell without interrupting a long-running cell.
Example: you run a machine learning train loop and want to immediately save the model's weights or check metrics. With
igogo
you can do so without interrupting the training. - If you need to compare the score of some function with different parameters, you can run several
functions at the same time and monitor results.
Example: you have several sets of hyperparameters and want to compare them. You can start training two models, monitoring two loss graphs at the same time.
- Process data in chunks. Check processed data for validity
Example: you do data processing in steps. With
igogo
you can execute several steps at the same time and process data from the first processing step in the second processing step in chunks. Also, you can quickly check that the first step produces the correct results
Install
Igogo is available through PyPi:
pip install igogo
Wait, isn't it just a background job? No.
- No multithreading, no data races, no locks. You can freely operate with your notebook variables without the risk of corrupting them.
- Beautiful output. When several cells execute in parallel, all printed data is displayed in the corresponding cell's output. No more twisted and messed out concurrent outputs.
- Easily cancel jobs, wait for completion, and start the new ones.
- Control execution of jobs through widgets.
Usage
At the core of igogo is collaborative execution. Jobs need to explicitly allow other jobs to execute through igogo.yielder()
. Mind that regular cells also represent a job.
To start an igogo job, you can use %%igogo
cell magic or function decorator.
import igogo
@igogo.job
def hello_world(name):
for i in range(3):
print("Hello, world from", name)
# allows other jobs to run while asleep
# also can be `igogo.yielder()`
igogo.sleep(1)
return name
Call function as usual to start a job:
hello_world('igogo'), hello_world('other igogo');
Configure jobs
Decorator @igogo.job
has several useful parameters.
kind
Allows to set how to render output. Possible options:text
,markdown
,html
Default:text
displays
As igogo job modify already executed cell, it needs to have spare placeholders for rich output. This parameter specifies how many spare displays to spawn. Default:10
name
User-friendly name of igogo job.warn_rewrite
Should warn rewriting older displays? Default:True
Markdown example:
Display additional data
You can use igogo.display
inside a job to display additional content.
For example, you can show pyplot figures.
import numpy as np
import matplotlib.pyplot as plt
import igogo
def experiment(name, f, i):
x = np.linspace(0.01, i, 1000)
fig, ax = plt.subplots(figsize=(5, 1))
ax.plot(x, f(x), c='r')
ax.set_title(name)
# display figure to job's output.
# if called from outside job, falls back to
# IPython.display.display
igogo.display(fig)
# close pyplot so that it does not show content
# automatically
plt.close()
As noted in "Configure jobs" section, igogo
jobs have limited number of displays.
If you try to display more objects than job has, warning will be shown and the oldest displays will be overwritten.
Cell magic
The same way with %%igogo
:
%load_ext igogo
%%igogo
name = 'igogo'
for i in range(3):
print("Hello, world from", name)
igogo.sleep(1)
Widgets
All executed igogo
jobs spawn a widget that allows to kill them. Jobs are not affected by KeyboardInterrupt
Killing jobs
Apart from killing through widgets, igogo
jobs can be killed programmatically.
igogo.stop()
Can be called insideigogo
job to kill itself.igogo.stop_all()
Stops all runningigogo
jobsigogo.stop_latest()
Stops the latestigogo
job. Can be executed several times.igogo.stop_by_cell_id(cell_id)
Kills all jobs that were launched in cell withcell_id
(aka [5], cell_id=5).
Also, you can stop jobs of one specific function.
hello_world.stop_all()
Stops alligogo
jobs created byhello_world()
Supported clients
Currently, igogo
runs correctly on:
- Jupyter Lab
- Jupyter
Does not run on:
- VSCode. For some reason it does not update display data. Therefore, no output is produced.
- DataSpell. It displays
[object Object]
and not output.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.