The Meeshkan Client for interactive machine learning
This repository contains Meeshkan client-side code.
Table of contents
How it works
The Meeshkan agent
The agent is a daemonized process running in the background. Agent is responsible for scheduling jobs (Python scripts) and interacting with them. Agent is responsible for, e.g.,
- sending job notifications so that you know how your jobs are doing and
- listening to instructions from the server. If you remotely execute the command to, for example, stop a job, the agent gets the instruction to stop the job from the server and stops the job.
The agent is managed using the command-line interface (CLI).
The Meeshkan Python library
The Python library imported with
import meeshkan is used in scripts to
control the notifications you get. For example, by including a command such as
import meeshkan meeshkan.report_scalar("Train loss", loss)
in your Python script you specify that notifications you get should contain the value for
meeshkan.add_condition can be used to send notifications for arbitrary events.
For detailed documentation of the library usage, see below.
Note that using
meeshkan in your Python scripts is optional (though recommended). If you do
not specify reported scalars, you will only get notifications for when jobs start or finish.
We recommend running all command-line commands below in a Python virtual environment.
Sign up at meeshkan.com and you will get your client secret via email.
meeshkan with pip:
$ pip install meeshkan
If install fails, your Python version may be too old. Please try again with Python >= 3.6.2.
Setup your credentials:
$ meeshkan setup
You are prompted for your client secret that you should have received when signing up, so fill that in.
The command creates the folder
.meeshkan in your home directory. The folder contains your credentials, agent logs
and outputs from your submitted jobs.
$ wget https://raw.githubusercontent.com/Meeshkan/meeshkan-client/dev/examples/report.py
Start the agent:
$ meeshkan start
If starting the agent fails, check that your credentials are properly setup. Also check known issues.
Submit the example job with 10 second reporting interval:
$ meeshkan submit --name report-example --report-interval 10 report.py
If you setup Slack integration in meeshkan.com, you should receive a notification for job being started. You should get notifications every ten seconds. The script runs for 20 seconds, so you should get one notification containing scalar values.
Other helpful commands
List running jobs:
$ meeshkan list
Retrieve logs for the job named
$ meeshkan logs report-example
Stop the agent:
$ meeshkan stop
You can use Meeshkan with any Python machine learning framework. As an example, let us use PyTorch to train a convolution neural network on MNIST.
$ pip install torch torchvision
Then download our PyTorch example:
$ wget https://raw.githubusercontent.com/Meeshkan/meeshkan-client/dev/examples/pytorch_mnist.py
Ensure that the agent is running:
$ meeshkan start
Submit the PyTorch example with a one-minute report interval:
$ meeshkan submit --name pytorch-example --report-interval 60 pytorch_mnist.py
To list available commands, execute
Usage: meeshkan [OPTIONS] COMMAND [ARGS]... Options: -h, --help Show this message and exit. --version Show the version and exit. Commands: clean Alias for `meeshkan clear` clear Clears Meeshkan log and job directories in ~/.meeshkan. help Show this message and exit. list Lists the job queue and status for each job. logs Retrieves the logs for a given job. notifications Retrieves notification history for a given job. report Returns latest scalar from given job identifier setup Configures the Meeshkan client. sorry Send error logs to Meeshkan HQ. start Starts Meeshkan service daemon. status Checks and returns the service daemon status. stop Stops the service daemon. submit Submits a new job to the service daemon.
In all instances used, a JOB_IDENTIFIER can be either the job's UUID, the job number, or a pattern to match against the job's name. In the latter case, the first match is returned.
Start the agent
starts the agent as daemonized service.
If you get
Unauthorized error, please check your credentials. Also check known issues. If the problem persists, please contact Meeshkan support.
Submit a script for execution
Submit a Python script as job:
meeshkan submit [--name job_name] [--report-interval 60] examples/hello_world.py
The agent runs submitted jobs sequentially in first-in-first-out order. By default, the
submit command will run your code
without time-based notifications (see below). When presented with the
-r/--report-interval flag, the service will notify you with recent updates every time
the report interval has elapsed. The report interval is measured in seconds.
The default argument (if none are provided) is 3600 seconds (i.e. hourly notifications).
List submitted jobs
Retrieve stdout and stderr for a job
meeshkan logs JOB_IDENTIFIER
Here JOB_IDENTIFIER can be either the job's UUID, the job number, or a pattern to match against the job's name.
You will get a complete output of stderr and stdout for the given job, and it's output path for any additional files.
Review job notification history
meeshkan notifications JOB_IDENTIFIER
Review latest scalar reports
meeshkan report JOB_IDENTIFER
Canceling a job
meeshkan cancel JOB_IDENTIFIER
If the job is currently running, you will be prompted to verify you want to abruptly cancel a running job.
Usage as Python library
The purpose of the Python API is to be as intuitive, minimal and least invasive as possible.
Once the agent has been started using
meeshkan start, you can communicate with it from your
Python scripts through
You can report scalars from within a given script using
meeshkan.report_scalar(name1, value1, name2, value2, ...).
The command allows reporting multiple values at once, and giving them self-documenting names (you may use anything, as
long as it's a string!).
Some examples include (assume the mentioned variables exist, and are updated in some loop, e.g. a training/evaluation loop):
meeshkan.report_scalar("train loss", train_loss) # Adds a single value for this process # Or add multiple scalars simulatenously meeshkan.report_scalar("train loss", train_loss, "evaluation loss", eval_loss, "evaluation accuracy", eval_acc) # Or possibly combine them for a new metric meeshkan.report_scalar("F1", 2*precision*recall/(precision+recall))
The agent only notifies you on either a scheduled notification (using the
--report-interval/-r flag, e.g.
hourly notifications), or when a certain criteria has been met.
You can define these criteria anywhere in your code (but outside of loops, these conditions only need to be set once!),
even before your scalars have been registered. When a scalar is used before it is registered, a default value of 1 is
used in its place.
Consider the following examples:
# Notify when evaluation loss and train loss are significantly different. meeshkan.add_condition("train loss", "evaluation loss", lambda train_loss, eval_loss: abs(train_loss - eval_loss) > 0.2) # Notify when the F1 score is suspiciously low meeshkan.add_condition("F1", lambda f1: f1 < 0.1)
Start occasionally fails in macOS
In macOS, running
meeshkan start may fail with
objc: +[NSValue initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
This happens because of threading restrictions introduced in macOS High Sierra. You can read more about it here, here, here, and here. We hope to find a permanent fix for this, but in the meanwhile you can run
$ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
before starting the client and then try again. If this does not fix the issue, please contact email@example.com.
We welcome contributions!
Install from the source
First clone this repository. Then install dependencies excluding test dependencies:
pip install -e .
Include test dependencies:
pip install -e .[dev]
pylint -f msvs meeshkan
Building the documentation
python setup.py docs # OR (the long way...) cd docs sphinx-apidoc -f -e -o source/ ../meeshkan/ make html
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size & hash SHA256 hash help||File type||Python version||Upload date|
|meeshkan-0.1.3-py2.py3-none-any.whl (41.7 kB) Copy SHA256 hash SHA256||Wheel||py2.py3|
|meeshkan-0.1.3.tar.gz (37.9 kB) Copy SHA256 hash SHA256||Source||None|