The Meeshkan Client for interactive machine learning
Project description
Meeshkan client
Getting started
Sign up at app.meeshkan.com and you will get your client secret via email.
Install the client in your Python environment with pip install meeshkan
. You can then either run meeshkan setup
to set things up, or manually add the folder .meeshkan
in your home directory and, inside the folder, add a file named credentials
with the following format:
[meeshkan]
token=my-client-secret
Installation
pip install meeshkan
Command-line interface
To list available commands, execute meeshkan
or meeshkan help
:
Usage: meeshkan [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help Show this message and exit.
--version Show the version and exit.
Commands:
clean Alias for `meeshkan clear`
clear Clears Meeshkan log and job directories in ~/.meeshkan.
help Show this message and exit.
list Lists the job queue and status for each job.
logs Retrieves the logs for a given job.
notifications Retrieves notification history for a given job.
report Returns latest scalar from given job identifier
setup Configures the Meeshkan client.
sorry Send error logs to Meeshkan HQ.
start Starts Meeshkan service daemon.
status Checks and returns the service daemon status.
stop Stops the service daemon.
submit Submits a new job to the service daemon.
In all instances used, a JOB_IDENTIFIER can be either the job's UUID, the job number, or a pattern to match against the job's name. In the latter case, the first match is returned.
Start service daemon
meeshkan start
If you get Unauthorized
error, please check your credentials. If the problem persists, please contact Meeshkan support.
Check service status
meeshkan status
You should get the message Service is up and running
.
Submit a Python script for execution
Submit the example script hello_world.py for execution:
meeshkan submit [--name job_name] examples/hello_world.py
List submitted jobs
meeshkan list
Retrieve stdout and stderr for a job
meeshkan logs JOB_IDENTIFIER
You will get a complete output of stderr and stdout for the given job, and it's output path for any additional files.
Review job notification history
meeshkan notifications JOB_IDENTIFIER
Review latest scalar reports
meeshkan report JOB_IDENTIFER
Canceling a job
meeshkan cancel JOB_IDENTIFIER
If the job is currently running, you will be prompted to verify you want to abruptly cancel a running job.
Stop service
meeshkan stop
Python API
General
The purpose of the Python API is to be as intuitive, minimal and least invasive as possible.
Once the service daemon is running (using meeshkan start
), you can communicate with it through the Python library.
To begin, import meeshkan
.
Reporting scalars
You can report scalars from within a given script using meeshkan.report_scalar(name1, value1, name2, value2, ...)
.
The command allows reporting multiple values at once, and giving them self-documenting names (you may use anything, as
long as it's a string!).
Some examples include (assume the mentioned variables exist, and are updated in some loop, e.g. a training/evaluation
loop), you may:
meeshkan.report_scalar("train loss", train_loss) # Adds a single value for this process
# Or add multiple scalars simulatenously
meeshkan.report_scalar("train loss", train_loss, "evaluation loss", eval_loss, "evaluation accuracy", eval_acc)
# Or possibly combine them for a new metric
meeshkan.report_scalar("F1", 2*precision*recall/(precision+recall))
Adding conditions
The service daemon only notifies you on either a scheduled notification (using the --report-interval/-r
flag, e.g.
hourly notifications), or when a certain criteria has been met.
You can define these criteria anywhere in your code (but outside of loops, these conditions only need to be set once!),
even before your scalars have been registered. When a scalar is used before it is registered, a default value of 1 is
used in its place.
Consider the following examples:
# Notify when evaluation loss and train loss are significantly different.
meeshkan.add_condition("train loss", "evaluation loss", lambda train_loss, eval_loss: abs(train_loss - eval_loss) > 0.2)
# Notify when the F1 score is suspiciously low
meeshkan.add_condition("F1", lambda f1: f1 < 0.1)
Reporting scalars from PyTorch
Get started
Start the service first, then run
meeshkan submit --report-interval 10 examples/report.py
By default, the submit
command will run your code without time-based notifications.
When presented with the -r/--report-interval
flag, the service will notify you with recent updates every time the
report interval has elapsed. The report interval is measured in seconds.
The default argument (if none are provided) is 3600 seconds (i.e. hourly notifications).
PyTorch
See examples/pytorch_mnist.py for an example script. To run the script, first ensure that PyTorch is installed in your Python environment. Then submit the example as
meeshkan submit -r 10 examples/pytorch_mnist.py
# OR using the "long" option:
meeshkan submit --report-interval 10 examples/pytorch_mnist.py
Meeshkan reports the training and test loss values to you every 10 seconds.
Development
Installation
For users:
python setup.py install
For developers:
pip install -e .[dev]
Running tests
pytest
# OR
python setup.py test
Running lint
pylint -f msvs meeshkan
Building the documentation
python setup.py docs
# OR (the long way...)
cd docs
sphinx-apidoc -f -e -o source/ ../meeshkan/
make html
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for meeshkan-0.1.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a85b23eb244e8abc8665fc6ad4c0bc63713807845279117817b893185de514a5 |
|
MD5 | e688850cd021676f5989099711509b69 |
|
BLAKE2b-256 | 4bc1bf0ed2abf757cdd1f4d66478027caf861212c9c184faadcc2403daca7ab0 |