mylearn: my Machine Learning framework
Project description
mylearn: my Machine Learning framework
mylearn is a Machine Learning framework based on Airflow and MLflow for designing machine learning systems in a production perspective.
Work in progress... Stay tuned!
Index
Recommended prerequisites
Git
sudo apt-get install git
pyenv
Install binary dependencies and build tools:
sudo apt update
sudo apt install build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev curl libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
Install pyenv:
curl https://pyenv.run | bash
echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
source ~/.bashrc
Install a Python version and set it as default:
pyenv install 3.11.2
pyenv global 3.11.2
poetry
curl -sSL https://install.python-poetry.org | python3 -
echo 'export PATH="~/.local/bin:$PATH"' >> ~/.bashrc
Installation & Setup
mylearn leverages poetry and poethepoet to make its installation and setup surprisingly simple.
Installation
It is recommended to install requirements within a virtualenv
located at the project root level, although not required.
poetry config virtualenvs.in-project true
Installation is run with:
poetry install
Should you install from the requirements.txt
file instead of the poetry.lock
file:
pyenv shell 3.11.2
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Airflow Setup
Airflow setup is initialized via a poe
command
poe airflow-init
Airflow Scheduler & Webserver can be run with
poe airflow-scheduler
poe airflow-webserver
Airflow UI can be opened at localhost (port 8080), and you can login with username and password admin
.
If you want to clean your Airflow setup before rerunning poe airflow-init
, you need to kill Airflow Scheduler &
Webserver and run
poe airflow-clean
MLflow Setup
MLflow UI can be opened at localhost (port 5000) after execution of the following command:
poe mlflow-ui
Usage
MLflow Pipelines Regression Template
The mlflow-template pipeline, based on the MLflow Pipelines Regression Template, can be run independently with
poe mlflow-run
or via an Airflow Directed Acyclic Graph (DAG) by triggering the mlflow-template DAG via Airflow UI or with
TO BE COMPLETED
Other examples
Work in progress... Stay tuned!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.