mylearn 0.0.3

mylearn: my Machine Learning framework

mylearn: my Machine Learning framework

---

mylearn is a Machine Learning framework based on Airflow and MLflow for designing machine learning systems in a production perspective.

Work in progress... Stay tuned!

Index

1. Recommended prerequisites
2. Installation & Setup
3. Usage

Recommended prerequisites

Git

sudo apt-get install git

pyenv

Install binary dependencies and build tools:

sudo apt update
sudo apt install build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev curl libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Install pyenv:

curl https://pyenv.run | bash
echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
source ~/.bashrc

Install a Python version and set it as default:

pyenv install 3.11.2
pyenv global 3.11.2

poetry

curl -sSL https://install.python-poetry.org | python3 -
echo 'export PATH="~/.local/bin:$PATH"' >> ~/.bashrc

Installation & Setup

mylearn leverages poetry and poethepoet to make its installation and setup surprisingly simple.

Installation

It is recommended to install requirements within a virtualenv located at the project root level, although not required.

poetry config virtualenvs.in-project true

Installation is run with:

poetry install

Should you install from the requirements.txt file instead of the poetry.lock file:

pyenv shell 3.11.2
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Airflow Setup

Airflow setup is initialized via a poe command

poe airflow-init

Airflow Scheduler & Webserver can be run with

poe airflow-scheduler
poe airflow-webserver

Airflow UI can be opened at localhost (port 8080), and you can login with username and password admin.

If you want to clean your Airflow setup before rerunning poe airflow-init, you need to kill Airflow Scheduler & Webserver and run

poe airflow-clean

MLflow Setup

MLflow UI can be opened at localhost (port 5000) after execution of the following command:

poe mlflow-ui

Usage

MLflow Pipelines Regression Template

The mlflow-template pipeline, based on the MLflow Pipelines Regression Template, can be run independently with

poe mlflow-run

or via an Airflow Directed Acyclic Graph (DAG) by triggering the mlflow-template DAG via Airflow UI or with

TO BE COMPLETED

Other examples

Work in progress... Stay tuned!