PipeLab — ML Pipeline Management Platform

A Python platform for managing end-to-end Machine Learning pipelines, backed by MLflow for experiment tracking and model registry, with a FastAPI backend and React + TailwindCSS frontend.

---

Getting Started

Quick start (local)

./run.sh

This sets up the virtual environment (if needed), starts the MLflow UI at http://127.0.0.1:5000 and the pipelab server at http://localhost:8000, using the bundled ./workdir (contains project-01). Options: --workdir DIR, --port PORT, --mlflow-port PORT, --reload.

Prerequisites

• Python ≥ 3.10
• Node.js ≥ 20 (only for frontend development)

1. Create and activate a virtual environment

cd pipelab
python -m venv .venv
source .venv/bin/activate   # Linux / macOS
# .venv\Scripts\activate    # Windows

2. Install pipelab in editable mode

pip install -e .

This installs all dependencies (FastAPI, MLflow, scikit-learn, pandas, etc.).

3. Set up your working directory

pipelab discovers projects from a workdir — a directory where each subdirectory containing an pipelab.yaml file is treated as a project.

# Create a workdir (or use any existing directory)
mkdir -p ~/pipelab-workdir
cd ~/pipelab-workdir

# Initialize your first project
pipelab init my-project --description "My first ML project"

This generates the following structure:

~/pipelab-workdir/
└── my-project/
    ├── pipelab.yaml      # Project configuration
    ├── pipeline/       # Place your pipeline service classes here
    ├── notebooks/      # Jupyter notebooks - for your organization
    └── mlruns/         # Local MLflow tracking store

4. Start MLflow (optional — for the tracking UI)

pipelab uses MLflow under the hood. You can optionally start the MLflow UI to inspect experiments directly:

# From within your workdir (runs on port 5000 by default)
mlflow ui --backend-store-uri my-project/mlruns

Open http://localhost:5000 to view the MLflow dashboard.

5. Start the pipelab server

# Serve from the workdir so pipelab discovers your projects
pipelab serve --workdir ~/pipelab-workdir --port 8000

Open http://localhost:8000 in your browser.

Tip: If you run pipelab serve without --workdir, it defaults to the current directory.

---

Features

• Project Workspaces — Organize experiments by project with pipelab.yaml config
• Dataset Versioning — Track dataset versions with metadata
• Data Split Management — K-fold CV, train/test, custom splits
• Experiment Tracking — Register experiments and runs via MLflow
• Benchmark & Comparison — Compare runs side-by-side with charts
• Model Registry — Register, version, and manage model lifecycle stages
• Pipeline Engine — Define and execute sequential ML pipelines (data → preparation → experiment → deploy)
• Model Deployment — Deploy models as REST endpoints
• Monitoring & Alerts — Track prediction stats and drift indicators
• Plugin Architecture — Extend with custom modules
• CLI — pipelab serve, pipelab init, pipelab run-pipeline

---

Architecture

pipelab/
├── entities/        # Domain dataclasses + pipeline ABCs
├── services/        # Abstract interfaces + MLflow implementations
├── infrastructure/  # MLflow client wrapper
├── pipelines/       # Pipeline execution engine
├── plugins/         # Plugin base class
├── api/             # FastAPI app + REST routers
│   └── routers/     # projects, datasets, experiments, models, ...
├── static/          # Built React frontend (served by FastAPI)
└── cli.py           # Typer CLI commands

Clean Architecture Layers

Layer	Location	Purpose
Domain	entities/	Dataclasses, pipeline ABCs — the core data model
Services	services/interfaces.py	Abstract contracts (ABCs)
Infrastructure	services/mlflow_*.py	Concrete MLflow implementations
API	api/	HTTP interface (FastAPI routers + Pydantic schemas)
UI	frontend/ → static/	React + TailwindCSS SPA

---

CLI Commands

# Start the web server (defaults to current directory as workdir)
pipelab serve --host 0.0.0.0 --port 8000 --workdir ~/pipelab-workdir

# Initialize a new project
pipelab init my-project --description "My ML Project" --workdir ~/pipelab-workdir

# Run all discovered pipeline services for a project
pipelab run-pipeline ~/pipelab-workdir/my-project

---

Writing Pipeline Services

Create Python files in your project's pipeline/ directory. pipelab autodiscovers subclasses of the pipeline ABCs:

# ~/pipelab-workdir/my-project/pipeline/data.py
from pipelab.pipeline import DatasetService, PipelineContext

class IrisLoader(DatasetService):
    """Load the Iris dataset."""
    def execute(self, context: PipelineContext):
        from sklearn.datasets import load_iris
        data = load_iris()
        context.results["X"] = data.data
        context.results["y"] = data.target
        return {"samples": len(data.data)}

# ~/pipelab-workdir/my-project/pipeline/train.py
from pipelab.pipeline import ExperimentService, PipelineContext

class RandomForestTrainer(ExperimentService):
    """Train a Random Forest classifier."""
    def execute(self, context: PipelineContext):
        from sklearn.ensemble import RandomForestClassifier
        from sklearn.metrics import accuracy_score
        import mlflow

        X_train = context.data.X_train
        y_train = context.data.y_train
        X_test = context.data.X_test
        y_test = context.data.y_test

        clf = RandomForestClassifier(n_estimators=100, random_state=42)
        clf.fit(X_train, y_train)
        acc = accuracy_score(y_test, clf.predict(X_test))

        with mlflow.start_run(experiment_id=context.experiment_name):
            mlflow.log_metric("accuracy", acc)
            mlflow.sklearn.log_model(clf, "model")

        return {"accuracy": acc}

Configure preparation in pipelab.yaml:

name: my-project
datasets:
  - name: iris
    data_service: IrisLoader
    preparation_method: TrainTestSplit
    preparation_params:
      test_size: 0.2
      random_state: 42

Then run from the web UI or CLI:

pipelab run-pipeline ~/pipelab-workdir/my-project

---

Extending the Framework

Custom Services

Implement any abstract interface from pipelab.services.interfaces:

from pipelab.services.interfaces import DatasetService

class MyDatasetService(DatasetService):
    def list_datasets(self, project_id=None):
        ...

Plugins

from pipelab.plugins import PipelabPlugin
from fastapi import APIRouter

class MyPlugin(PipelabPlugin):
    @property
    def name(self):
        return "my-plugin"

    def get_routes(self):
        router = APIRouter()
        @router.get("/hello")
        async def hello():
            return {"msg": "Hello from plugin!"}
        return [router]

---

Frontend Development

cd pipelab/frontend
npm install
npm run dev     # Dev server with API proxy to :8000
npm run build   # Build to ../static/

---

API Endpoints

Endpoint	Methods	Description
/api/v1/projects/	GET, POST, PUT	Project management
/api/v1/datasets/	GET, POST, DELETE	Dataset CRUD + versions + splits
/api/v1/experiments/	GET, POST, DELETE	Experiment management
/api/v1/experiments/{name}/runs	GET	List experiment runs
/api/v1/experiments/runs/compare	POST	Compare multiple runs
/api/v1/models/	GET, POST, DELETE	Model registry
/api/v1/models/register	POST	Register a model from a run
/api/v1/pipelines/services	GET	List autodiscovered services
/api/v1/pipelines/run	POST	Execute a pipeline
/api/v1/pipelines/dataset-configs	GET, POST, DELETE	Dataset configurations
/api/v1/pipelines/eda/load	POST	Load dataset for EDA
/api/v1/deployments/	GET, POST, DELETE	Model deployments
/api/v1/monitoring/	GET, POST	Prediction stats + alerts
/api/v1/settings/	GET, PUT	Configuration

---

Requirements

All Python dependencies are managed via pyproject.toml and installed automatically:

Package	Purpose
fastapi	REST API framework
uvicorn[standard]	ASGI server
mlflow	Experiment tracking & model registry
scikit-learn	ML utilities & default preparation methods
pandas	Data manipulation
pyyaml	Project configuration parsing
typer	CLI framework
pydantic	Request/response validation

---

Building & Publishing to PyPI

1. Build the frontend (bundles into pipelab/static/)

cd pipelab/frontend
npm install
npm run build
cd ../..

2. Build the Python package

pip install build
python -m build

This creates dist/pipelab-X.Y.Z.tar.gz and dist/pipelab-X.Y.Z-py3-none-any.whl.

3. Upload to PyPI

pip install twine
twine upload dist/*

Tip: For test uploads, use TestPyPI first:

twine upload --repository testpypi dist/*
pip install --index-url https://test.pypi.org/simple/ pipelab

4. Version bumps

Update the version in pyproject.toml:

[project]
version = "0.2.0"

Then rebuild and upload.

---

License

MIT