running-mate 0.0.4

Lightweight MLOps framework.

Running Mate

Version and monitor your models, record inferences, and send alerts without any additional infrastructure. Designed for small data science or machine learning teams lacking a full MLOps solution.

Training:

1. Create local SQLite database
2. Version model
3. Generate baseline stats

Inference:

1. Check features against baseline stats
2. Create and send alerts

Getting Started

Install:

$ pip install running-mate

Running Mate stores data in a local SQLite database. Start by creating the database, creating a new Mate (which represent your AI/ML model), and generate the baseline statistics:

from mate.db import create_db, version_or_create_mate
from mate.generators import generate_baseline_stats


create_db()
mate = version_or_create_mate("mate-name")
generate_baseline_stats(your_dataframe, "mate-name", mate.version)

This ideally happens at training time.

Then, in your serving environment, define the alert targets, get the current Mate version, load the model, wrap you model prediction in the mate context manager:

from mate.alerts import TerminalAlertTarget
from mate.db import get_current_mate
from mate.run import RunningMate


alert_targets = [
    TerminalAlertTarget(),
]

version = get_current_mate("mate-name").version

model = load(f"models/mate-name-{version}.joblib")

with RunningMate("mate-name", version, your_dataframe, alert_targets):
    output = model.predict(enc.transform(your_dataframe))

Example

The example project uses TerminalAlertTarget, for outputting alerts to the terminal, and AlertWebhookTarget, for sending alerts to http://localhost:5000/hook.

Create/activate a virtual environment and install the dependencies:

$ cd example

$ python3 -m venv venv
$ source venv/bin/activate

$ pip install -r requirements.txt

In one terminal window, run the web server:

$ cd web

$ FLASK_DEBUG=1 python -m flask run

Then, train the model and run an inference through it in a different window:

$ python train.py
$ python infer.py

Navigate to http://localhost:5000 to see the alert.

Development

After cloning down the repo, create/activate a virtual environment and install the dependencies:

$ python3 -m venv venv
$ source venv/bin/activate

$ pip install -r requirements-dev.txt

Install as local package:

$ pip install -e .

Run tests:

$ pytest .

Lint, format code, and type check:

$ flake8 --ignore=E501,W503 mate tests example

$ black mate tests example

$ isort --profile black mate tests example

$ mypy mate tests example

TODO

1. add github actions
2. optionally send runtime stats (like latency)
3. add more tests
4. document how to pass in custom, user-defined stats