MaatML
Build task-specific models with checks you can inspect. MaatML prepares
data, trains models, measures their outputs against task contracts, and packages
them for inference across text, vision, and vision-language. One declarative
model.yml drives prepare → train → evaluate → export → serve.
What makes it different: correctness is checked outside the model by
validators. The same validator gates your synthetic data and your
evaluation, and can guard your live inference: maatml serve runs it
per request on /predict?validate=1, and on every response under --enforce,
where a failing output is rejected with HTTP 422. The contract defines what is
checked; passing it establishes those checks, not correctness beyond them.
Site: maatml.pages.dev ·
PyPI: maatml ·
Source: github.com/Moralfish-Labs/maatml
Implementation and release status
The package version is 0.19.0. This version adds a served-model parity
predictor, reasoning-span loss masking for SFT, free-text rollouts, the
filtered_imitation objective and an environment-verified GRPO trainer. It also
includes regular-expression LoRA targets, the adapter export format with a
layout check, a refusal of over-long SFT rows instead of truncation,
processor-aware SFT trainers, frozen-component checks, multimodal GGUF
companions, maatml snapshot checkpoint
backups, per-prediction evidence, streaming preparation, dataset.train_samples for corrective examples, stricter
population checks, rollout/DAgger collection, tool-call chat training, and
multi-turn masking. See CHANGELOG.md and ROADMAP.md.
MaatML owns the model lifecycle and its evidence. Model folders own task contracts and recipes; an external orchestrator owns remote machines, scheduling, storage transfers, monitoring, and recovery policy. The orchestration guide describes how to keep those responsibilities separate while carrying checkpoints and run records between machines.
Installation
python -m venv .venv
source .venv/bin/activate
# Library + CLI (no torch)
pip install maatml
# Training / evaluation stack
pip install "maatml[ml]"
# Optional extras
pip install "maatml[ml,cuda]" # QLoRA on NVIDIA CUDA (bitsandbytes)
pip install "maatml[ml,pref]" # DPO / ORPO (TRL)
pip install "maatml[ml,vision]" # torchvision + ONNX (examples/vision)
pip install "maatml[vllm]" # Linux-only vLLM serving (examples/vision-vlm)
pip install "maatml[teacher]" # OpenAI-compatible teacher for datagen
pip install "maatml[docs]" # site tooling; also requires a source checkout
Then:
maatml --help
maatml scaffold ~/models/my-task --architecture causal_sft --name my-task
maatml validate ~/models/my-task
For contributing to this repository (editable install), see CONTRIBUTING.md.
Example models
Four reference models share the identical folder layout and CLI, from a one-command support-ticket triage to a vLLM-servable vision-language model:
| Model | Task | Architecture | Base |
|---|---|---|---|
| Support Ticket Triage | triage → JSON | causal_sft (LoRA) |
Qwen3-0.6B |
| Vision VLM | describe a scene image | vlm_sft (vLLM-servable) |
SmolVLM-256M-Instruct |
| Vision | scene + detect + pose | vision_multitask |
MobileNetV3-Large |
| Vision Describer | caption from vision JSON | seq2seq |
flan-t5-small |
Any directory with a valid model.yml works the same way: install maatml from
PyPI and point the CLI at the folder. Scaffold a new model folder with
maatml scaffold.
Where MaatML fits
MaatML builds on Hugging Face transformers / peft / trl and does the
one thing those building blocks leave to you: it wraps them in an opinionated,
validator-gated lifecycle for small task-specific models. Train on a suitable
local device or cloud GPU, then export for the runtime your model supports.
- Complements general fine-tuning tools (Axolotl, LLaMA-Factory, Unsloth, TRL) rather than competing on scale. Reach for those for large models, multi-node training, RL, or broad model coverage.
- Runs its own fixed lifecycle (
prepare → train → evaluate → export → verify/serve), as one command:maatml run, which skips steps that are already fresh and stops non-zero at the first failure. It is not a general-purpose workflow scheduler: no triggers, no arbitrary shell/Python steps, no remote executors. An external orchestrator can invoke its CLI on local machines, notebook GPUs, or GPU VMs; see orchestration and recovery. - Its niche: local-first, multimodal, structured-output models with correctness gated outside the model, from data generation through serving.
Requirements
- Python 3.10+ (developed against 3.13)
- OS macOS and Linux; Windows has a torch-free CLI/test CI job, not a demonstrated GPU training path
- Disk / memory depend on the selected extras, base model, context length and batch size. Reserve space for downloads, checkpoints and exports; use a smoke run to check the recipe on your hardware.
CLI overview
Most commands take a model folder (containing model.yml) as their first
argument. Outputs land under <model-folder>/output/ (gitignored). Run
maatml <command> --help for the full flag list. Errors in your input (a
missing file, an unparseable model.yml, an unregistered plugin) print one
line; maatml --debug <command> prints the traceback.
| Command | What it does |
|---|---|
run |
The whole lifecycle in one command: prepare, train, evaluate (gated), export, verify. Skips steps that are already fresh |
prepare |
Build train/val/test splits from the seed corpus; enforces dataset.isolation / pins, records the benchmark version and the corpus lock, checks source sign-offs when dataset.attribution is declared |
train |
Fine-tune the model (--smoke, --resume auto|PATH, --set K=V, --seeds N); training.select_by picks the checkpoint on val |
sweep |
Offline grid HPO over --param K=a,b |
evaluate |
Score a checkpoint; --gate exits non-zero on a gate miss, --cache keeps per-row predictions, --out DIR separates reports from historical results, --batch-size N uses a predictor's predict_batch when available, --set K=V overrides configuration (never with --gate), --blind spends the blind manifest once, --strict-population refuses mismatched gate populations before inference. The token budget defaults to packaging.max_input_tokens |
gates derive |
Floors from a run's report: Wilson 95 % at each metric's own denominator, cluster bootstrap from a cache, --seed-study; --write rewrites evaluation.gates with the derivation beside each floor |
ship-check |
CANDIDATE BASELINE: absolute, delta and population verdict in one; --replay re-evaluates both over the current test split |
operating-point derive |
Sweep the predictor's rescore over a val cache under a budget; --write the cut, --confirm-on-test spends test once |
export |
Deployable bundle + manifest.json (--format, --parity) |
compile |
Dispatch an export to a registered target compiler; --require-gated rejects missing, failed or smoke-tier gate evidence |
manifest amend |
Add files inside an export to its checksummed manifest without changing gate evidence |
verify |
Recompute sha256 of an export against its manifest.json |
serve |
JSON inference API; --enforce (422), --max-retries, --auth-token, --capture |
datagen |
Validator-gated seed generation (--teacher, --allow-ungated) |
distill |
Validator-gated teacher labels over a prompt pool (--replay offline) |
mint |
Preference pairs (chosen/rejected) from validator-scored candidates |
ingest |
Import external samples (--map field=col, --sanitize tag) |
runs |
List recorded training runs (--compare tabulates their metrics); --pack RUN / --adopt BUNDLE carry a run, its evidence and its record between machines |
report |
Runs, floors with their derivation, slices, pathologies, seed statistics and spends, regenerated from output/ alone (--format md|csv) |
plan |
Show which lifecycle steps are stale (alias for run --dry-run) |
plugins |
List registered trainers, validators, metrics, formats, generators, exporters, compilers, servers and environments |
audit |
Check the environment, plugins, and a model folder; exits 1 on problems |
rollout |
Run a checkpoint in a verifying environment; --label collects teacher labels for DAgger, and --teacher-mix mixes teacher decisions |
scaffold |
Create a new model folder (--architecture, --plugin, --force) |
validate |
Check model.yml and paths (--no-plugins skips plugin code) |
Multi-GPU (CUDA): accelerate launch -m maatml.cli train <model-dir>/ or
torchrun --nproc_per_node=N -m maatml.cli train <model-dir>/.
QLoRA (CUDA + [cuda]): set training.quantization.load_in_4bit: true in
model.yml. Preference data: dataset.format: preference_jsonl with
{prompt, chosen, rejected} rows; scaffold with --architecture dpo.
Export defaults to a safetensors bundle + manifest.json. GGUF/MLX need
external tooling (llama.cpp convert / mlx_lm). Pin base-model revisions
with training.model_revision.
The causal SFT path also preserves per-row tool definitions, assistant tool calls and tool results. It supervises every assistant turn and masks the other roles; see tool-call chat training.
Docs: maatml.pages.dev · Roadmap: ROADMAP.md ·
In-repo docs: docs/ (from the repository root, pip install -e ".[docs]"
then mkdocs serve). See the CLI reference and
configuration reference.
End-to-end example (Support Ticket Triage)
The quickest model to run: a LoRA fine-tune of Qwen3-0.6B that turns a raw
support ticket into {priority, category, team, summary} JSON, gated by a
schema validator plus a category → team routing contract enforced outside
the model. Every reference model now registers a validator and declares
evaluation.gates.
git clone https://github.com/Moralfish-Labs/maatml.git
cd maatml
pip install -e ".[ml]" # match the framework to the checked-out examples
maatml prepare examples/support-ticket-triage/
maatml train examples/support-ticket-triage/ --smoke # fast pipeline check
maatml train examples/support-ticket-triage/
maatml evaluate examples/support-ticket-triage/ --gate # enforce eval gates
maatml serve examples/support-ticket-triage/ # JSON inference API
For a multimodal walkthrough (image → description, servable by vLLM) see examples/vision-vlm/.
From a passed gate to a claim
A green evaluate --gate is the start of the evidence, not the end of it.
The same CLI derives the floors, chooses the operating point, names the
populations, and carries the run home:
maatml gates derive <model-dir> --run RUN --write # Wilson floors, derivation beside each
maatml operating-point derive <model-dir> --run RUN --write --confirm-on-test
maatml ship-check <model-dir> CANDIDATE BASELINE # absolute + delta + population
maatml evaluate <model-dir> --blind # once per frozen candidate
maatml runs <model-dir> --pack RUN # → --adopt on the machine that exports
maatml report <model-dir> # everything above, from the records alone
model.yml declares the rest: dataset.isolation / pins / blind_samples
for populations, dataset.attribution for the licence table every source
must be signed in, training.select_by for checkpoint selection on val.
docs/evidence.md walks through it.
Keep acceptance floors fixed while measuring a corrective candidate. Deriving
new floors from that candidate is a separate calibration decision, not a way
to turn a failed experiment into a pass. ship-check rejects failed aggregate
gate evidence, missing or mismatched comparison populations, and a candidate
population that differs from a declared evaluation.gates_benchmark.
--replay must produce reports over the same rows; it does not waive those checks.
Corrective data without changing the test population
For the jsonl_seed format, add reviewed corrective examples through
dataset.train_samples instead of mixing them into the seed population:
dataset:
format: jsonl_seed
seed_samples: datasets/samples/seed_samples.jsonl
train_samples: datasets/samples/corrections.jsonl
benchmark_samples: datasets/samples/benchmark.jsonl
Train-only rows enter after the seed split. Preparation checks held-out group
collisions, applies configured isolation and attribution checks, and records
the extra input in corpus.lock.json. This preserves the seed split assignment;
keep the seed data, grouping, split settings, and benchmark fixed when comparing
experiments. Source attribution records provenance, not model quality.
Cloud training, progress, and recovery
Use the same model folder and CLI on the GPU runtime. Transfer the prepared data, plugin, recipe, and any local base checkpoint before running:
maatml validate <model-dir>
maatml train <model-dir> --device cuda \
--set training.logging_steps=1 --set training.disable_tqdm=true
# After an interruption, resume from a saved training checkpoint:
maatml train <model-dir> --device cuda --resume auto \
--set training.logging_steps=1 --set training.disable_tqdm=true
maatml evaluate <model-dir> --checkpoint RUN --cache --out <report-dir>
maatml runs <model-dir> --pack RUN
disable_tqdm replaces the SFT progress bar with ordinary Trainer log records;
logging_steps controls their frequency. Evaluation prints row progress and
estimated time remaining. Set PYTHONUNBUFFERED=1 when an external orchestrator
streams Python output. Resume requires a saved checkpoint; work since the last
save may need to run again.
The orchestrator provisions and stops the GPU, streams logs, and transfers
checkpoints. On the receiving machine, maatml runs <model-dir> --adopt BUNDLE
imports the completed run and its evidence. A completed training run does not
by itself establish an acceptance or release pass.
Batch scripts
# Deterministic seed corpora (no API calls)
python examples/support-ticket-triage/scripts/build_seeds.py
python examples/vision/scripts/build_seeds.py
python examples/vision-vlm/scripts/build_seeds.py
python examples/vision-describer/scripts/build_seeds.py
# Train / evaluate example models
python scripts/train_all.py --smoke
python scripts/train_all.py
python scripts/evaluate_all.py
Apple Silicon / MPS notes
- Default precision is bf16 autocast with fp32 master weights.
- The MPS device profile disables mid-training evaluation; setting
eval_stepsdoes not override that profile. grad_checkpointingdefaults tofalse;dataloader_num_workers=0everywhere (multi-worker + MPS can deadlock via fork pickling).PYTORCH_ENABLE_MPS_FALLBACK=1is set by the CLI for unsupported ops.
Repository layout
examples/ # reference task models (plugins + data)
support-ticket-triage/ # causal LoRA SFT
vision/ # multitask vision
vision-vlm/ # vision-language LoRA SFT
vision-describer/ # seq2seq captioning
src/maatml/ # core framework (architectures, CLI, harnesses)
scripts/ # batch train/eval/validate
tests/ # core unit tests
Trust boundary
A model folder is executable code, not just data. Any plugins: entry in
model.yml is imported as Python the moment the folder is loaded, and every
command that reads model.yml, including maatml validate and maatml plan,
loads the folder. Running any maatml command against a folder therefore runs
that folder's code with your privileges. Only run maatml on model folders you
trust, the same way you would only run a script you trust. Use
maatml validate --no-plugins to check the schema and paths without importing
plugin code.
Development
See CONTRIBUTING.md for setup, PR expectations, DCO sign-off, and versioning policy. AI coding agents: AGENTS.md.
Community: CODE_OF_CONDUCT.md · Security: SECURITY.md · Changes: CHANGELOG.md
Licensing
- maatml is licensed under the Apache License 2.0.
- This repository does not redistribute base-model weights, only Hugging Face Hub IDs. Your fine-tuned checkpoints inherit the base model's license terms.
- Seed corpora are fully synthetic, produced by deterministic builders under
examples/*/scripts/, with no proprietary source data shipped.
Metadata
Release files for maatml 0.19.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| maatml-0.19.0.tar.gz | 321.0 kB | With provenance | Details for maatml-0.19.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| maatml-0.19.0-py3-none-any.whl | Python 3 | none | any | Details for maatml-0.19.0-py3-none-any.whl |
Total release size: 649.9 kB
Release files / maatml-0.19.0.tar.gz
| Download URL | maatml-0.19.0.tar.gz |
|---|---|
| Size | 321.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
85e697fe06106c84e36fae876b04c2c5d2a194146be3dca990a42baad392cdad
|
|
BLAKE2b-256 checksum How to use checksums |
fdc49b268be94c93669896e37e1c08575701f2cd2e08e5314e6d140b03e3a3d6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / maatml-0.19.0-py3-none-any.whl
| Download URL | maatml-0.19.0-py3-none-any.whl |
|---|---|
| Size | 328.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e21675a01ea1a47a7f7f2eecaa0fe24239e7891b4524d13c8ef487ee3acf169a
|
|
BLAKE2b-256 checksum How to use checksums |
91b295ee6c796b0d2c6bf172c31997d8afb2c9dff0532b75b1839e940700a1c8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logSecurity for maatml 0.19.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for maatml-0.19.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.