Running Mate

Version and monitor your models, record inferences, and send alerts without any additional infrastructure. Designed for small data science or machine learning teams lacking a full MLOps solution.

Training:

1. Create local SQLite database
2. Version model
3. Generate baseline stats

Inference:

1. Check features against baseline stats
2. Create and send alerts

Getting Started

Install:

$ pip install running-mate

Running Mate stores data in a local SQLite database. Start by creating the database, creating a new Mate (which represent your AI/ML model), and generate the baseline statistics:

from mate.db import connect_db, version_or_create_mate
from mate.generators import generate_baseline_stats


connect_db()
mate = version_or_create_mate("mate-name")
generate_baseline_stats(your_dataframe, "mate-name")

This ideally happens at training time.

Then, in your serving environment, define the alert targets, get the current Mate version, load the model, wrap you model prediction in the mate context manager:

from mate.alerts import TerminalAlertTarget
from mate.db import connect_db, get_current_mate
from mate.run import RunningMate


alert_targets = [
    TerminalAlertTarget(),
]

connect_db()
version = get_current_mate("mate-name").version

model = load(f"models/mate-name-{version}.joblib")

with RunningMate("mate-name", version, your_dataframe, alert_targets):
    model.predict(enc.transform(your_dataframe))

By default, not all feature values are recorded. To record all values, set should_save_all_feature_values to True:

with RunningMate("mate-name", version, your_dataframe, alert_targets, should_save_all_feature_values=True):
    model.predict(enc.transform(df))

You can generate a summary stats report like so:

from mate.generators import generate_baseline_stats

print(generate_feature_stats_summary_report("mate-name"))

Examples

The basic example project, found in "examples/basic", uses TerminalAlertTarget, for outputting alerts to the terminal, and AlertWebhookTarget, for sending alerts to http://localhost:5000/hook.

Create/activate a virtual environment and install the dependencies:

$ cd example/basic

$ python3 -m venv venv
$ source venv/bin/activate

$ pip install -r requirements.txt

In one terminal window, run the web server:

$ cd web

$ FLASK_DEBUG=1 python -m flask run

Then, train the model and run an inference through it in a different window:

$ python train.py
$ python infer.py

Navigate to http://localhost:5000 to see the alert.

There's also a drift example in examples/drift.

Development

After cloning down the repo, create/activate a virtual environment and install the dependencies:

$ python3 -m venv venv
$ source venv/bin/activate

(venv)$ pip install -r requirements-dev.txt

Install as local package:

(venv)$ pip install -e .

Run tests:

(venv)$ python -m pytest .

Lint, format code, and type check:

(venv)$ python -m flake8 --ignore=E501,W503 mate tests examples

(venv)$ python -m black mate tests examples

(venv)$ python -m isort --profile black mate tests examples

(venv)$ python -m mypy mate tests examples