Computer science experiment manager - experimental plans, job scheduling, parameter handling
Project description
Experimaestro is a computer science experiment manager whose goals are:
- To decompose experiments into a set of parameterizable tasks
- Schedule tasks and handle dependencies between tasks
- Avoids to re-run the same task two times by computing unique task IDs dependending on the parameters
- Handle experimental parameters through tags
Full documentation can be found in https://experimaestro.github.io/experimaestro-python/
Install
Experimaestro depends on external libraries Poco and libssh. You can install them using
- on Mac or Linux (user)
brew install poco libssh
- on Debian/Ubuntu Linux (root)
apt-get install libpoco-dev libssh-dev
You can then install the package using pip install experimaestro
Example
This very simple example shows how to submit two tasks that add two numbers. Under the curtain,
- A directory is created for each task (in
workdir/jobs/helloworld.add
) based on a unique ID computed from the parameters - Two processes are launched (there are no dependencies, so they will be run in parallel)
- A tag
y
is created for each task - tags are experimental parameters you vary in your experiments, so you can easily keep track of them
# --- Task and types definitions
import logging
from experimaestro import *
from experimaestro.click import cli, TASK_PREFIX
import click
# --- Define the tasks
hw = Typename("helloworld")
@Type("word", type=str, required=True, help="Word to generate")
@RegisterTask(hw.say, prefix_args=TASK_PREFIX)
class Say:
def execute(self):
print(self.word.upper(),)
@Type("strings", type=ArrayOf(Say), help="Strings to concat")
@RegisterTask(hw.concat, prefix_args=TASK_PREFIX)
class Concat:
def execute(self):
# We access the file where standard output was stored
says = []
for string in self.strings:
with open(string._stdout()) as fp:
says.append(fp.read().strip())
print(" ".join(says))
# --- Defines the experiment
@click.argument("workdir", type=str)
@cli.command()
def xp(workdir):
# Sets the working directory and the name of the xp
experiment(workdir, "helloworld")
# Submit the tasks
hello = Say(word="hello").submit()
world = Say(word="world").submit()
# Concat will depend on the two first tasks
Concat(strings=[hello, world]).submit()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
Close
Hashes for experimaestro-0.3.2-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f522950c9ab6d36127029686113df18b108d76fbed1565b43669c50e565897a |
|
MD5 | 26d9ad2a20dceb4d83c69802ac3f97a0 |
|
BLAKE2b-256 | c0050d8cac420affc2a4440c8ac488df7f571416a19e7f9e69d09c0403950712 |