Kaze's MLOps stack

This is an evolving stack of tool I uses for my ML workflow. It is going to be very opinionated, not necessarily always up to date (considering the pace of the field, I think this is fair), and not always the best choice for your use case. I choose this stack because it fits my philosophy and my workflow.

The main purpose of this repo is to serve as a reference to retrace my steps whenever I need, instead of a template which I just copy and deploy to the next project. I have no intention of making this a full fledge library.

Stack

• Machine learning framework: Jax (with Flax for neural networks)
• Hyperparameter tuning : Optax (for optimizers) + Optuna (for hyperparameter tuning)
• Object storage: MinIO (for storing data and models)
• Database: PostgreSQL (for storing metadata and results)
• Experiment tracking: MLflow (for tracking experiments and models)
• Data versioning: DVC (for data versioning and pipelines)
• Orchestration: Dagster (for orchestrating the pipeline)
• Deployment: BentoML (for deploying the model as a service)
• Monitoring: Prometheus (for monitoring the service) + Grafana (for visualizing the metrics)