MaatML

MaatML is where a small, task-specific model is built, weighed, and packaged, across text, vision, and vision-language. One declarative model.yml drives prepare → train → evaluate → export → serve. The runtime is where the model answers. Licensed under Apache-2.0.

What makes it different: correctness is checked outside the model by validators. The same validator gates your synthetic data and your evaluation, and can guard your live inference: maatml serve runs it per request on /predict?validate=1, and on every response under --enforce, where a failing output is rejected with HTTP 422. So a MaatML model ships with a contract, not just weights. That validator-gated data → eval → serving loop, now across modalities, is what general fine-tuning tools leave out.

Site: maatml.pages.dev · PyPI: maatml · Source: github.com/Moralfish-Labs/maatml

Implementation and release status

The package version is 0.14.0. This release includes rollout/DAgger collection, train-only dataset rows, tool-call chat training and multi-turn masking fixes. Upgrade older installations to use these features; see CHANGELOG.md for release notes and ROADMAP.md for remaining work.

MaatML owns the model lifecycle and its evidence. Model folders own task contracts and recipes; an external orchestrator owns remote machines, scheduling, storage transfers, monitoring, and recovery policy. The orchestration guide describes how to keep those responsibilities separate while carrying checkpoints and run records between machines.

Installation

python -m venv .venv
source .venv/bin/activate

# Library + CLI (no torch)
pip install maatml

# Training / evaluation stack
pip install "maatml[ml]"

# Optional extras
pip install "maatml[ml,cuda]"    # QLoRA on NVIDIA CUDA (bitsandbytes)
pip install "maatml[ml,pref]"    # DPO / ORPO (TRL)
pip install "maatml[ml,vision]"  # torchvision + ONNX (examples/vision)
pip install "maatml[vllm]"       # Linux-only vLLM serving (examples/vision-vlm)
pip install "maatml[teacher]"    # OpenAI-compatible teacher for datagen
pip install "maatml[docs]"       # site tooling; also requires a source checkout

Then:

maatml --help
maatml scaffold ~/models/my-task --architecture causal_sft --name my-task
maatml validate ~/models/my-task

For contributing to this repository (editable install), see CONTRIBUTING.md.

Example models

Four reference models share the identical folder layout and CLI, from a one-command support-ticket triage to a vLLM-servable vision-language model:

Any directory with a valid model.yml works the same way: install maatml from PyPI and point the CLI at the folder. Scaffold a new model folder with maatml scaffold.

Where MaatML fits

MaatML builds on Hugging Face transformers / peft / trl and does the one thing those building blocks leave to you: it wraps them in an opinionated, validator-gated lifecycle for small task-specific models you can train on a laptop and deploy to the edge or vLLM.

• Complements general fine-tuning tools (Axolotl, LLaMA-Factory, Unsloth, TRL) rather than competing on scale. Reach for those for large models, multi-node training, RL, or broad model coverage.
• Runs its own fixed lifecycle (prepare → train → evaluate → export → verify / serve), as one command: maatml run, which skips steps that are already fresh and stops non-zero at the first failure. It is not a general-purpose workflow scheduler: no triggers, no arbitrary shell/Python steps, no remote executors. An external orchestrator can invoke its CLI on local machines, notebook GPUs, or GPU VMs; see orchestration and recovery.
• Its niche: local-first, multimodal, structured-output models with correctness gated outside the model, from data generation through serving.

Requirements

• Python 3.10+ (developed against 3.13)
• OS macOS and Linux; Windows has a torch-free CLI/test CI job, not a demonstrated GPU training path
• Disk / memory depend on the selected extras, base model, context length and batch size. Reserve space for downloads, checkpoints and exports; use a smoke run to check the recipe on your hardware.

CLI overview

Most commands take a model folder (containing model.yml) as their first argument. Outputs land under <model-folder>/output/ (gitignored). Run maatml <command> --help for the full flag list. Errors in your input (a missing file, an unparseable model.yml, an unregistered plugin) print one line; maatml --debug <command> prints the traceback.

Multi-GPU (CUDA): accelerate launch -m maatml.cli train <model-dir>/ or torchrun --nproc_per_node=N -m maatml.cli train <model-dir>/.

QLoRA (CUDA + [cuda]): set training.quantization.load_in_4bit: true in model.yml. Preference data: dataset.format: preference_jsonl with {prompt, chosen, rejected} rows; scaffold with --architecture dpo.

Export defaults to a safetensors bundle + manifest.json. GGUF/MLX need external tooling (llama.cpp convert / mlx_lm). Pin base-model revisions with training.model_revision.

The causal SFT path also preserves per-row tool definitions, assistant tool calls and tool results. It supervises every assistant turn and masks the other roles; see tool-call chat training.

Docs: maatml.pages.dev · Roadmap: ROADMAP.md · In-repo docs: docs/ (from the repository root, pip install -e ".[docs]" then mkdocs serve). See the CLI reference and configuration reference.

End-to-end example (Support Ticket Triage)

The quickest model to run: a LoRA fine-tune of Qwen3-0.6B that turns a raw support ticket into {priority, category, team, summary} JSON, gated by a schema validator plus a category → team routing contract enforced outside the model. Every reference model now registers a validator and declares evaluation.gates.

git clone https://github.com/Moralfish-Labs/maatml.git
cd maatml
pip install -e ".[ml]"   # match the framework to the checked-out examples

maatml prepare  examples/support-ticket-triage/
maatml train    examples/support-ticket-triage/ --smoke   # fast pipeline check
maatml train    examples/support-ticket-triage/
maatml evaluate examples/support-ticket-triage/ --gate    # enforce eval gates
maatml serve    examples/support-ticket-triage/           # JSON inference API

For a multimodal walkthrough (image → description, servable by vLLM) see examples/vision-vlm/.

From a passed gate to a claim

A green evaluate --gate is the start of the evidence, not the end of it. The same CLI derives the floors, chooses the operating point, names the populations, and carries the run home:

maatml gates derive <model-dir> --run RUN --write      # Wilson floors, derivation beside each
maatml operating-point derive <model-dir> --run RUN --write --confirm-on-test
maatml ship-check <model-dir> CANDIDATE BASELINE        # absolute + delta + population
maatml evaluate <model-dir> --blind                     # once per frozen candidate
maatml runs <model-dir> --pack RUN                      # → --adopt on the machine that exports
maatml report <model-dir>                               # everything above, from the records alone

model.yml declares the rest: dataset.isolation / pins / blind_samples for populations, dataset.attribution for the licence table every source must be signed in, training.select_by for checkpoint selection on val. docs/evidence.md walks through it.

Batch scripts

# Deterministic seed corpora (no API calls)
python examples/support-ticket-triage/scripts/build_seeds.py
python examples/vision/scripts/build_seeds.py
python examples/vision-vlm/scripts/build_seeds.py
python examples/vision-describer/scripts/build_seeds.py

# Train / evaluate example models
python scripts/train_all.py --smoke
python scripts/train_all.py
python scripts/evaluate_all.py

Apple Silicon / MPS notes

• Default precision is bf16 autocast with fp32 master weights.
• The MPS device profile disables mid-training evaluation; setting eval_steps does not override that profile.
• grad_checkpointing defaults to false; dataloader_num_workers=0 everywhere (multi-worker + MPS can deadlock via fork pickling).
• PYTORCH_ENABLE_MPS_FALLBACK=1 is set by the CLI for unsupported ops.

Repository layout

examples/                   # reference task models (plugins + data)
  support-ticket-triage/    # causal LoRA SFT
  vision/                   # multitask vision
  vision-vlm/               # vision-language LoRA SFT
  vision-describer/         # seq2seq captioning

src/maatml/                 # core framework (architectures, CLI, harnesses)
scripts/                    # batch train/eval/validate
tests/                      # core unit tests

Trust boundary

A model folder is executable code, not just data. Any plugins: entry in model.yml is imported as Python the moment the folder is loaded, and every command that reads model.yml, including maatml validate and maatml plan, loads the folder. Running any maatml command against a folder therefore runs that folder's code with your privileges. Only run maatml on model folders you trust, the same way you would only run a script you trust. Use maatml validate --no-plugins to check the schema and paths without importing plugin code.

Development

See CONTRIBUTING.md for setup, PR expectations, DCO sign-off, and versioning policy. AI coding agents: AGENTS.md.

Community: CODE_OF_CONDUCT.md · Security: SECURITY.md · Changes: CHANGELOG.md

Licensing

• maatml is licensed under the Apache License 2.0.
• This repository does not redistribute base-model weights, only Hugging Face Hub IDs. Your fine-tuned checkpoints inherit the base model's license terms.
• Seed corpora are fully synthetic, produced by deterministic builders under examples/*/scripts/, with no proprietary source data shipped.