Lightweight MLOps framework.
Project description
Running Mate
Version and monitor your models, record inferences, and send alerts without any additional infrastructure. Designed for small data science or machine learning teams lacking a full MLOps solution.
Training:
- Create local SQLite database
- Version model
- Generate baseline stats
Inference:
- Check features against baseline stats
- Create and send alerts
Getting Started
Install:
$ pip install running-mate
Running Mate stores data in a local SQLite database. Start by creating the database, creating a new Mate (which represent your AI/ML model), and generate the baseline statistics:
from mate.db import connect_db, version_or_create_mate
from mate.generators import generate_baseline_stats
connect_db()
mate = version_or_create_mate("mate-name")
generate_baseline_stats(your_dataframe, "mate-name")
This ideally happens at training time.
Then, in your serving environment, define the alert targets, get the current Mate version, load the model, wrap you model prediction in the mate
context manager:
from mate.alerts import TerminalAlertTarget
from mate.db import connect_db, get_current_mate
from mate.run import RunningMate
alert_targets = [
TerminalAlertTarget(),
]
connect_db()
version = get_current_mate("mate-name").version
model = load(f"models/mate-name-{version}.joblib")
with RunningMate("mate-name", version, your_dataframe, alert_targets):
model.predict(enc.transform(your_dataframe))
By default, not all feature values are recorded. To record all values, set should_save_all_feature_values
to True
:
with RunningMate(MATE_NAME, version, your_dataframe, alert_targets, should_save_all_feature_values=True):
model.predict(enc.transform(df))
Example
The example project uses TerminalAlertTarget
, for outputting alerts to the terminal, and AlertWebhookTarget
, for sending alerts to http://localhost:5000/hook
.
Create/activate a virtual environment and install the dependencies:
$ cd example
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt
In one terminal window, run the web server:
$ cd web
$ FLASK_DEBUG=1 python -m flask run
Then, train the model and run an inference through it in a different window:
$ python train.py
$ python infer.py
Navigate to http://localhost:5000 to see the alert.
Development
After cloning down the repo, create/activate a virtual environment and install the dependencies:
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements-dev.txt
Install as local package:
$ pip install -e .
Run tests:
$ pytest .
Lint, format code, and type check:
$ flake8 --ignore=E501,W503 mate tests example
$ black mate tests example
$ isort --profile black mate tests example
$ mypy mate tests example
TODO
- add github actions
- optionally send runtime stats (like latency)
- document how to pass in custom, user-defined stats
- add prometheus and grafana example
- create basic report
Potential drift example:
from alibi_detect.cd import KSDrift
cd = KSDrift(X, p_val=0.05, preprocess_x_ref=enc) # save at training time
preds = cd.predict(df, drift_type='feature', return_p_val=True, return_distance=True)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for running_mate-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f890c8fd932eccdbec7e8f8a195a4f92166e4400f63832148a199a7077f19fc3 |
|
MD5 | 8ea57cc4d7a4bf2bc80913da424895b1 |
|
BLAKE2b-256 | 507476b8cc7f9fa600809066fe1a0b747a9f4c900fce6d0f15470774c6277847 |