Skip to main content

smallab (Small Lab) is an experiment framework designed to be easy to use with your experiment

Project description

Documentation Status

smallab

:small_blue_diamond: :microscope: :small_blue_diamond:

smallab (Small Lab) is an experiment framework designed to be easy to use with your experiment.

The code in this repo should be understandable as a whole in at most 10 minutes.

Features

  • Easy to understand and simple
  • Easy parallelization of experiments
  • Only runs not previously completed experiments
  • Hooks allow monitoring batch progress
  • All parameters to methods use the typing module
  • Dashboard for monitoring experiment progress

Installation

pip install smallab

or clone here from source

Documentation

https://smallab.readthedocs.io/en/latest/

Usage

Check out the examples folder or demo.py (Copied here)

import typing
import random
import os
import dill

#What you need to import from smallab
from smallab.experiment_types.experiment import Experiment
from smallab.runner.runner import ExperimentRunner


#Write a simple experiment
class SimpleExperiment(Experiment):
    #Need to implement this method, will be passed the specification
    #Return a dictionary of results
    def main(self, specification: typing.Dict) -> typing.Dict:
        random.seed(specification["seed"])
        for i in range(specification["num_calls"]): #Advance the random number generator some amount
           random.random()
        if "fail" in specification and specification["fail"]:
            raise Exception()
        return {"number":random.random()}

runner = ExperimentRunner()


#Optional: Email yourself when the whole batch is done
#Read https://stackoverflow.com/questions/5619914/sendmail-errno61-connection-refused about how to start an stmp serevr
from smallab.utilities.email_hooks import EmailCallbackBatchOnly

runner.attach_callbacks([EmailCallbackBatchOnly("test@test.com",40)])
#Take it back off since we don't actually want to bother Mr. Test
runner.attach_callbacks([])

#Set the specifications for our experiments, the author reccomends reading this from a json file!
specifications = [{"seed": 1,"num_calls":1}, {"seed":2,"num_calls":1}]

#Fire off the experiment
runner.run("random_number",specifications,SimpleExperiment())

#Read back our results
for root,_,files in os.walk(runner.get_save_directory("random_number")):
    for fname in files:
        if ".pkl" in fname:
            with open(os.path.join(root, fname), "rb") as f:
                results = dill.load(f)
                print(results["specification"]["seed"])
                print(results["result"]["number"])


from smallab.specification_generator import SpecificationGenerator
#If you want to run a lot of experiments but not manual write out each one, use the specification generator.
#Note: This is also JSON serializable, so you could store this in a json file
generation_specification = {"seed":[1,2,3,4,5,6,7,8],"num_calls":[1,2,3]}

#Call the generate method. Will create the cross product.
specifications = SpecificationGenerator().generate(generation_specification)
print(specifications)

runner.run("random_number_from_generator",specifications,SimpleExperiment(),continue_from_last_run=True)

#Read back our results
for root,_,files in os.walk(runner.get_save_directory("random_number_from_generator")):
    for fname in files:
        if ".pkl" in fname:
            with open(os.path.join(root, fname), "rb") as f:
                results = dill.load(f)
                print(results["specification"]["seed"])
                print(results["result"]["number"])

#If you have an experiment you want run on a lot of computers you can use the MultiComputerGenerator
#You assign each computer a number from 0..number_of_computers-1 and it gives each computer every number_of_computerth specification
from smallab.specification_generator import MultiComputerGenerator
all_specifications = SpecificationGenerator().from_json_file('test.json')

g1 = MultiComputerGenerator(0,2)
g2 = MultiComputerGenerator(1,2)
specifications_1 = g1.from_json_file("test.json")
specifications_2 = g2.from_json_file("test.json")


assert len(specifications_1) + len(specifications_2) == len(all_specifications)

#Need to freeze the sets in order to do set manipulation on dictionaries
specifications_1 = set([frozenset(sorted(x.items())) for x in specifications_1])
specifications_2 = set([frozenset(sorted(x.items())) for x in specifications_2])
all_specifications = set([frozenset(sorted(x.items())) for x in all_specifications])

#This will generate two disjoint sets of specifications
assert specifications_1.isdisjoint(specifications_2)
#That together make the whole specification
assert specifications_1.union(specifications_2) == all_specifications



#You can use the provided logging callbacks to log completion and failure of specific specifcations
from smallab.utilities import logger_callbacks
runner.attach_callbacks([LoggingCallback()])
runner.run('with_logging',SpecificationGenerator().from_json_file("test.json"),SimpleExperiment(),continue_from_last_run=True)

How it works

The ExperimentRunner class is passed a list of dictionaries of specifications. These dictionaries need to be json serializable.

The ExperimentRunner looks at the completed.json in the folder for the batch name (the name parameter of the .run method) and computes which experiments need to be run. The experiments that need to run are the specifications not in the completed.json.

The ExperimentRunner begins runnning the batch either in parallel or single threaded. If the parallel implementation is used each specification is joblib's threaded backend.

Once all experiments are either completed or failed (they threw an exception) the results are saved as a pickle file. The results are saved in a dictionary that looks like

{
    "specification": <the specification the experiment was passed>,
    "result": <what the experiment .main returned>
}

The return value of the experiment .main function must be pickle serializable.

Callbacks

The runner has several hooks which are called at different times.

  • on_specification_complete called whenever a specification completes running (Ususually a single experiment)
  • on_specification_falure called whenver a specification fails running (Throws an exception)
  • on_batch_complete called after runner .run has finished running, passed all the succesfully completed specifications
  • on_batch_failure called after runner .run has finished running, passed all the failed specifications

Folder Structure

Each experiment is saved in the following structure

experiment_runs/
  <name>/                        # The name you provide to runner.run
    <specification_hash>/        # A hash of the dictionary you provide as the specification
      specification.json         # The specification.json
        <specification_hash>.pkl # The results dictionary

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

smallab-2.3.1.tar.gz (30.6 kB view details)

Uploaded Source

Built Distribution

smallab-2.3.1-py3-none-any.whl (42.8 kB view details)

Uploaded Python 3

File details

Details for the file smallab-2.3.1.tar.gz.

File metadata

  • Download URL: smallab-2.3.1.tar.gz
  • Upload date:
  • Size: 30.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.1

File hashes

Hashes for smallab-2.3.1.tar.gz
Algorithm Hash digest
SHA256 a294e885b7cdb55f49253abb2ebf4c9f9d9d688ec0b5c925e893c1a3657db1cb
MD5 6a41220dfb1173506384ad498734662d
BLAKE2b-256 61048ea66cc6b0726ff33a7b7e00541601c73cb5631904ec7f13e122dccf45d1

See more details on using hashes here.

File details

Details for the file smallab-2.3.1-py3-none-any.whl.

File metadata

  • Download URL: smallab-2.3.1-py3-none-any.whl
  • Upload date:
  • Size: 42.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.1

File hashes

Hashes for smallab-2.3.1-py3-none-any.whl
Algorithm Hash digest
SHA256 8d1a04d438ed79f5018c9b3bef68e8b14f3648a40d09e8331d9d74a184c1b9c3
MD5 8a587f26cfeda883667f363715ce67e1
BLAKE2b-256 525521daed8dfbf4a1bac437781a46187aee71a6fa59759d5764e5311b2cc4c5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page