Computer science experiment manager - experimental plans, job scheduling, parameter handling
Project description
Experimaestro is a computer science experiment manager whose goals are:
- To decompose experiments into a set of parameterizable tasks
- Schedule tasks and handle dependencies between tasks
- Avoids to re-run the same task two times by computing unique task IDs dependending on the parameters
- Handle experimental parameters through tags
Full documentation can be found in https://experimaestro.github.io/experimaestro-python/
Install
From binary wheel
Binary wheels are available with a recent version of pip.
From source
Experimaestro depends on external libraries Poco and libssh. You can install them using
- on Mac or Linux (user)
brew install poco libssh
- on Debian/Ubuntu Linux (root)
apt-get install libpoco-dev libssh-dev
You can then install the package using pip install experimaestro
Example
This very simple example shows how to submit two tasks that add two numbers. Under the curtain,
- A directory is created for each task (in
workdir/jobs/helloworld.add
) based on a unique ID computed from the parameters - Two processes are launched (there are no dependencies, so they will be run in parallel)
- A tag
y
is created for each task - tags are experimental parameters you vary in your experiments, so you can easily keep track of them
# --- Task and types definitions
import logging
from experimaestro import *
from experimaestro.click import cli, TASK_PREFIX
import click
import time
# --- Just to be able to monitor the tasks
def slowdown(N: int):
for i in range(N):
time.sleep(2)
progress((i+1)/N)
# --- Define the tasks
hw = Typename("helloworld")
@Argument("word", type=str, required=True, help="Word to generate")
@Task(hw.say, prefix_args=TASK_PREFIX)
class Say:
def execute(self):
slowdown(len(self.word))
print(self.word.upper(),)
@Argument("strings", type=Array(Say), help="Strings to concat")
@Task(hw.concat, prefix_args=TASK_PREFIX)
class Concat:
def execute(self):
# We access the file where standard output was stored
says = []
slowdown(len(self.strings))
for string in self.strings:
with open(string._stdout()) as fp:
says.append(fp.read().strip())
print(" ".join(says))
# --- Defines the experiment
@click.option("--port", type=int, default=12345, help="Port for monitoring")
@click.argument("workdir", type=str)
@cli.command()
def xp(port, workdir):
"""Runs an experiment"""
# Sets the working directory and the name of the xp
ws = experiment(workdir, "helloworld")
ws.server(port)
# Submit the tasks
hello = Say(word="hello").submit()
world = Say(word="world").submit()
# Concat will depend on the two first tasks
Concat(strings=[hello, world]).submit()
if __name__ == "__main__":
cli()
which can be launched with python test.py xp /tmp/helloworld-workdir
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
experimaestro-0.3.3.tar.gz
(8.5 MB
view hashes)
Built Distributions
Close
Hashes for experimaestro-0.3.3-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4051e7e46ae1322dccd0e41903d85f4a85f26052ab98acec9c0cf66cc7bfb99c |
|
MD5 | 601595e742420643e4b42d95d0dab53d |
|
BLAKE2b-256 | a11064d79dee256ab0ae53b2095cc2af5e5170321f1a1b6daac8ec9a75dbde3f |
Close
Hashes for experimaestro-0.3.3-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e02bd6120c3f1b3f410727e2704905325b82aa24fb4c0337cc129b3ec167602f |
|
MD5 | 267474e4ec771ff40d76002d31b513a4 |
|
BLAKE2b-256 | a95103a25837427c7c0a6930d42d9167949e836d960b6040d4cc702c7a7d6394 |
Close
Hashes for experimaestro-0.3.3-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30c8fad27a3df7f0de353f61bc1285bc225077c276a73d0fa9a660788e37a1e8 |
|
MD5 | 085307d3f06869050c28b4372189814c |
|
BLAKE2b-256 | 3282d930ea065206e838fea59944acadf4d88aa4ccbede57c7388fc6db98fcd0 |