MLForge

A reproducible Python toolkit for training, evaluating, selecting, and fitting tabular machine-learning models with leakage-safe preprocessing and deterministic cross-validation.

MLForge is a local, library-first workflow built on pandas and scikit-learn. It turns a CSV and an explicit target into inspectable evidence: validated data, fair model comparisons, immutable JSON manifests, trusted-local model artifacts, and schema-checked predictions. It is deliberately small, single-process, and honest about what it does not implement.

Quick start

Install the published distribution (the import package and command are both named mlforge):

python -m pip install hivmind-mlforge
mlforge --version

From a repository clone, run the bundled classification benchmark:

mlforge benchmark examples/customer_churn.csv --target churn --metric balanced_accuracy --cross-validation-folds 3 --benchmarks-dir mlbenchmarks

Representative output from the bundled eight-row dataset is shown below. Scores are captured from the real command; run IDs, filesystem paths, and timing are omitted because they vary. The tiny dataset demonstrates the workflow, not estimator quality.

Protocol: 3-fold stratified cross-validation (shuffle seed=42)
Primary metric: balanced_accuracy
Leaderboard:
  1. logistic-regression:        0.833333 +/- 0.236
  2. random-forest-classifier:   0.833333 +/- 0.236
  3. dummy-classifier:           0.500000 +/- 0.000

Best observed mean: logistic-regression ranked first.
Note: cross-validation selects an estimator; it does not fit a final deployment model.

Final fitting is a separate, explicit decision. Copy the benchmark UUID from that command and run:

mlforge finalize examples/customer_churn.csv --target churn --benchmark-id BENCHMARK_ID --benchmarks-dir mlbenchmarks --final-models-dir mlfinalmodels --artifacts-dir artifacts

MLForge verifies the persisted selection and exact dataset, refits the rank-one estimator on all rows, and creates a new immutable final-model manifest plus trusted-local artifact. The recorded cross-validation score remains selection evidence; it is not relabeled as post-selection test performance.

Continue with the complete local workflow tutorial to train, inspect, save, load, and predict with a fitted artifact.

Why MLForge

Model comparison is easy to make convincing and surprisingly hard to make fair. Preprocessing the full dataset leaks validation information. Giving estimators different partitions makes scores incomparable. Unstable tie-breaking makes repeated runs hard to explain. Recording only the winning number discards the evidence needed to audit it later.

MLForge makes those concerns explicit:

The result is not an experiment-tracking platform. It is a focused reference workflow whose data, modeling, persistence, and trust boundaries are small enough to understand and test.

Core capabilities

Training

• Strict local CSV validation and deterministic profiling.
• Explicit classification or regression task selection.
• Deterministic train/validation splitting and leakage-safe preprocessing.
• Five baseline estimators with task-appropriate held-out metrics.
• Versioned, create-only run manifests with failure evidence.

Artifacts and prediction

• Fitted preprocessing and estimator saved together in a versioned .mlforge archive.
• Structure, schema, environment, size, lineage hash, and payload checksum inspection without deserializing the model.
• Explicit trusted loading for pickle-based artifacts.
• Exact feature-schema validation, column-order restoration, batch prediction, and atomic CSV output.

Benchmarking

• Task-aware defaults: three classification baselines or Ridge and random-forest regression.
• One shared holdout split, selectable primary metric, deterministic ranking, and fitted in-memory winner.
• A strict aggregate manifest that references every underlying run.

Cross-validation

• Deterministic 2-10 fold benchmarking: stratified folds for classification and ordinary shuffled K-folds for regression.
• A fresh estimator and fold-local preprocessing pipeline for every training fold.
• Identical ordered fold fingerprints for every estimator.
• Per-fold classification or regression metrics, arithmetic means, population standard deviations, direction-aware ranking, warnings, timing, and failure location.
• Selection evidence only: no nested-tuning or untouched post-selection performance claim.

Explicit final-model fitting

• Accepts only a persisted successful cross-validation result and its exact selected dataset.
• Reconstructs the recorded preprocessing, feature-role, seed, estimator, and parameter contract.
• Fits one new preprocessing/model pipeline on every selected row without inventing new metrics.
• Writes a create-only final-model manifest and a version-2 artifact lineage record.
• Reuses safe inspection, explicit trusted loading, schema validation, and batch prediction.

Reliability guarantees

• Fit after split: no data-derived transformer state is learned from validation rows.
• Comparable evidence: dataset bytes, target, configuration, seed, and exact partitions are recorded and checked before comparison.
• Immutable local history: run, benchmark, cross-validation, and final-model manifests are atomically created and never silently overwritten.
• Bounded input handling: CSV and artifact readers validate structure and enforce documented size limits.
• Fail-closed artifact loading: untrusted, corrupt, incompatible, or structurally invalid artifacts are rejected.
• Reproducible randomness: supported splits and randomized estimators use recorded seeds; random forests use one process.

Numerical results can still change when dependency versions change. Manifests record the exact Python, MLForge, pandas, NumPy, SciPy, and scikit-learn versions so the environment can be reconstructed and interpreted.

Architecture

flowchart LR
    A["CSV + explicit target"] --> B["Validation + fingerprint"]
    B --> C["Profile"]
    B --> D["Holdout split"]
    B --> E["Shared task-aware folds"]
    D --> F["Leakage-safe pipeline fit"]
    E --> G["Fresh fold-local pipeline fits"]
    F --> H["Metrics + run manifest"]
    G --> I["Aggregates + leaderboard"]
    I --> J["Cross-validation manifest"]
    J --> M["Explicit all-row final fit"]
    M --> K
    H --> K["Trusted-local artifact"]
    K --> L["Schema validation + prediction"]

The CLI is a thin adapter over importable domain APIs. Dataset, pipeline, training, benchmark, run, artifact, and inference modules own their behavior; none depends on a web service, database, or worker. See the architecture document for responsibilities, dependency direction, extension points, and security boundaries.

Python API

The same cross-validation workflow is available as a typed Python API:

from pathlib import Path

from mlforge.benchmarks import (
    CrossValidationConfig,
    LocalCrossValidationStore,
    cross_validate_benchmark,
)
from mlforge.artifacts import LocalArtifactStore
from mlforge.datasets import load_csv
from mlforge.final_models import LocalFinalModelStore, fit_selected_model
from mlforge.pipelines import CrossValidationSplitConfig

dataset = load_csv(Path("examples/customer_churn.csv"), target="churn")
result = cross_validate_benchmark(
    dataset,
    CrossValidationConfig(
        primary_metric="balanced_accuracy",
        split=CrossValidationSplitConfig(fold_count=3, random_seed=42),
    ),
    store=LocalCrossValidationStore(Path("mlbenchmarks/cross-validation")),
)

print(result.manifest.winner)
print(result.manifest.to_json())

final_model = fit_selected_model(
    dataset,
    result,
    final_model_store=LocalFinalModelStore(Path("mlfinalmodels")),
    artifact_store=LocalArtifactStore(Path("artifacts")),
)
print(final_model.manifest.to_json())
print(final_model.artifact_path)

Cross-validation deliberately returns an immutable selection record. fit_selected_model is the separate all-row refit-and-save step and never reports training-set metrics as evaluation. For the complete workflow, see examples/finalize_customer_churn.py and the Python API reference.

CLI

Every command provides --help; user-facing workflows also support --json where structured output is useful.

Use --trust-artifact only after verifying the artifact's source and custody. Inspection and a matching checksum do not make hostile pickle content safe.

Installation

MLForge supports Python 3.11 and 3.12. Create an isolated environment before installation:

python -m venv .venv
python -m pip install hivmind-mlforge

Activate with .venv\Scripts\Activate.ps1 on Windows PowerShell or source .venv/bin/activate on macOS/Linux. For a development checkout, install the development extra instead:

python -m pip install -e ".[dev]"

Local web application

The local single-user web interface supports the application shell, dashboard, CSV upload, core-backed validation, explicit target selection, a real data overview, and persisted classification or regression comparison configuration, execution, core-backed experiment results, and explicit rank-one model finalization. It also provides a Models screen for reviewing completed local models, their source evidence, input schema, and recorded runtime. Finalized local models can run schema-validated prediction CSVs, preview the first 20 results, and download the complete output. The Experiments screen lists saved configurations and durable execution states, with links to the existing dataset metadata, benchmark evidence, failure details, and finalized model information. The interface uses a responsive modal navigation on smaller screens, visible keyboard focus, named scrollable table regions, associated form guidance, and text-based status labels. Important routes distinguish restrained loading, true empty, retryable server-error, validation-error, success, and partial-result states; the dashboard reflects persisted experiment history rather than sample data. The Settings screen provides browser-local appearance, validated defaults for future comparison forms, workspace counts and usage, runtime diagnostics, and a create-only backup download. Runtime workspace and upload-limit controls remain environment-backed and require an API restart. Install its optional adapter and start the API from the repository root:

python -m pip install -e ".[dev,web]"
python -m mlforge.web

In another terminal, start the frontend:

cd frontend
npm install
npm run dev

Open http://localhost:3000. Local upload bytes and SQLite metadata are written under the ignored .mlforge-web/ workspace. Use Predictions to select a finalized model and submit a matching CSV. The web workspace is intentionally separate from CLI directories such as mlruns/ and artifacts/; existing CLI history is not imported automatically. Retrain or explicitly reproduce the workflow when an older artifact records a different MLForge or dependency environment.

The Python wheel contains the library, CLI, and FastAPI adapter. The Next.js frontend and private container profile are distributed through the repository and source archive because they are built with Node.js and Docker rather than installed into Python site-packages.

Private deployment profile

The repository includes a provider-neutral two-container profile for one trusted operator. It keeps the API on an internal network, binds the browser-facing port to host loopback, persists the complete workspace in one Docker volume, and supplies liveness/readiness probes. It must be reached through an SSH tunnel or a reviewed private access gateway; it is not safe for direct public exposure.

See private single-user deployment for the architecture, startup, backup, upgrade, rollback, and security boundaries.

Validation evidence

The v0.6.0 release candidate has 258 passing behavioral tests at 85.36% statement coverage, including regression comparison, finalization, and web prediction coverage. It enforces a conservative 80% floor. CI covers Ubuntu on Python 3.11/3.12 and Windows on Python 3.12, with Ruff, formatting, strict mypy, pytest, package builds, pip check, and installed-wheel smoke tests. Offline real-data tests exercise scikit-learn's breast cancer and diabetes datasets; separate release validation covers Iris, Wine, and breast cancer cross-validation workflows.

See release validation for the dataset and clean-package boundaries.

Documentation

• Complete local workflow tutorial
• Python API reference
• Architecture and design boundaries
• Compatibility and versioning policy
• Artifact trust and secure-use guidance
• Release validation
• Maintainer release procedure
• Roadmap and changelog

Development

Run the same quality gate used by CI:

ruff check .
ruff format --check .
mypy src tests
python -m pytest
python -m build
python scripts/check_source_archive.py dist
python -m twine check --strict dist/*
cd frontend
npm run lint
npm run build
npx playwright install chromium
npm run test:e2e

python -m pytest includes pytest-cov and fails below 80% statement coverage. The CI build then installs the wheel into a separate environment and executes scripts/wheel_smoke.py outside the source tree. The Playwright test launches both local servers against an isolated temporary workspace and verifies the complete browser workflow through prediction CSV download.

Project structure

mlforge/
|- src/mlforge/          # Importable production package
|  |- datasets/          # Strict ingestion and deterministic profiles
|  |- pipelines/         # Splits, folds, feature roles, and preprocessing
|  |- training/          # Baseline fitting and evaluation
|  |- benchmarks/        # Holdout/CV orchestration, ranking, and manifests
|  |- final_models/       # Verified selection lineage and explicit all-row fitting
|  |- runs/              # Immutable experiment records and comparison
|  |- artifacts/         # Trusted-local model persistence
|  `- web/               # Thin local FastAPI adapter over public core APIs
|- frontend/             # Next.js single-user web interface
|- tests/                # Unit, integration, HTTP, CLI, edge-case, and real-data tests
|- examples/             # Runnable source-checkout workflows and small CSVs
|- scripts/              # Release-tag and installed-wheel validation
|- docs/                 # Tutorial, API, architecture, security, and release guidance
`- .github/workflows/    # Cross-platform CI and trusted release publishing

Project status and current limits

MLForge v0.6.0 combines the feature-complete local Python core with a supported single-user web workflow, bounded operator settings, workspace backup, and a private deployment profile. The web adapter reuses the core ML algorithms and evidence model rather than implementing a second training system.

MLForge currently supports local, single-process tabular classification/regression. Cross-validation and selection-driven final fitting support both tasks. The web application has a local HTTP dataset boundary, dashboard, upload flow, data review, supervised experiment configuration, a one-worker comparison/finalization runner with persisted job-level status, detailed cross-validation results, safe final-model artifact metadata, and a Models screen backed by verified local lineage. It also exposes a schema-validated prediction submission workflow for finalized local models, bounded result previews, and complete CSV downloads. MLForge does not provide hyperparameter tuning, nested evaluation, an untouched post-selection test estimate, shared storage, distributed execution, request authentication, authenticated public deployment, or monitoring.

The repository does include a private, loopback-bound single-operator deployment profile. It does not provide authenticated public deployment, multi-user isolation, public online model serving, or production monitoring.

The separate shared-service infrastructure milestone remains conditional on real multi-user requirements; it is not active development. The roadmap records these boundaries so planned work is not presented as shipped functionality.

Contributing, security, and license

Read CONTRIBUTING.md for setup, required checks, and pull-request expectations. Report vulnerabilities through the private process in SECURITY.md, and review the artifact security model before loading or sharing model files.

MLForge is available under the Apache License 2.0, including its explicit patent grant.