Skip to main content

MaatML

PyPI License Python CI

Build task-specific models with checks you can inspect. MaatML prepares data, trains models, measures their outputs against task contracts, and packages them for inference across text, vision, and vision-language. One declarative model.yml drives prepare → train → evaluate → export → serve.

What makes it different: correctness is checked outside the model by validators. The same validator gates your synthetic data and your evaluation, and can guard your live inference: maatml serve runs it per request on /predict?validate=1, and on every response under --enforce, where a failing output is rejected with HTTP 422. The contract defines what is checked; passing it establishes those checks, not correctness beyond them.

Site: maatml.pages.dev · PyPI: maatml · Source: github.com/Moralfish-Labs/maatml

Implementation and release status

The package version is 0.19.1. The 0.19 line adds a served-model parity predictor, reasoning-span loss masking for SFT, free-text rollouts, the filtered_imitation objective and an environment-verified GRPO trainer. It also includes regular-expression LoRA targets, the adapter export format with a layout check, a refusal of over-long SFT rows instead of truncation, processor-aware SFT trainers, frozen-component checks, multimodal GGUF companions, maatml snapshot checkpoint backups, per-prediction evidence, streaming preparation, dataset.train_samples for corrective examples, stricter population checks, rollout/DAgger collection, tool-call chat training, and multi-turn masking. See CHANGELOG.md and ROADMAP.md.

MaatML owns the model lifecycle and its evidence. Model folders own task contracts and recipes; an external orchestrator owns remote machines, scheduling, storage transfers, monitoring, and recovery policy. The orchestration guide describes how to keep those responsibilities separate while carrying checkpoints and run records between machines.

Installation

python -m venv .venv
source .venv/bin/activate

# Library + CLI (no torch)
pip install maatml

# Training / evaluation stack
pip install "maatml[ml]"

# Optional extras
pip install "maatml[ml,cuda]"    # QLoRA on NVIDIA CUDA (bitsandbytes)
pip install "maatml[ml,pref]"    # DPO / ORPO (TRL >= 0.9)
pip install "maatml[ml,vision]"  # torchvision + ONNX (examples/vision)
pip install "maatml[vllm]"       # Linux-only vLLM serving (examples/vision-vlm)
pip install "maatml[teacher]"    # OpenAI-compatible teacher for datagen / distill
pip install "maatml[docs]"       # site tooling; also requires a source checkout

Then:

maatml --help
maatml scaffold ~/models/my-task --architecture causal_sft --name my-task
maatml validate ~/models/my-task

For contributing to this repository (editable install), see CONTRIBUTING.md.

Example models

Four reference models share the identical folder layout and CLI, from a one-command support-ticket triage to a vLLM-servable vision-language model:

Model Task Architecture Base
Support Ticket Triage triage → JSON causal_sft (LoRA) Qwen3-0.6B
Vision VLM describe a scene image vlm_sft (vLLM-servable) SmolVLM-256M-Instruct
Vision scene + detect + pose vision_multitask MobileNetV3-Large
Vision Describer caption from vision JSON seq2seq flan-t5-small

Any directory with a valid model.yml works the same way: install maatml from PyPI and point the CLI at the folder. Scaffold a new model folder with maatml scaffold.

Where MaatML fits

MaatML builds on Hugging Face transformers / peft / trl and does the one thing those building blocks leave to you: it wraps them in an opinionated, validator-gated lifecycle for small task-specific models. Train on a suitable local device or cloud GPU, then export for the runtime your model supports.

  • Complements general fine-tuning tools (Axolotl, LLaMA-Factory, Unsloth, TRL) rather than competing on scale. Reach for those for large models, multi-node training, large-scale RL, or broad model coverage. MaatML includes supervised fine-tuning, DPO/ORPO, filtered_imitation, and environment-verified grpo scoped to small task-specific models.
  • Runs its own fixed lifecycle (prepare → train → evaluate → export → verify / serve), as one command: maatml run, which skips steps that are already fresh and stops non-zero at the first failure. It is not a general-purpose workflow scheduler: no triggers, no arbitrary shell/Python steps, no remote executors. An external orchestrator can invoke its CLI on local machines, notebook GPUs, or GPU VMs; see orchestration and recovery.
  • Its niche: local-first, multimodal, structured-output models with correctness gated outside the model, from data generation through serving.

Requirements

  • Python 3.10+ (developed against 3.13)
  • OS macOS and Linux; Windows has a torch-free CLI/test CI job, not a demonstrated GPU training path
  • Disk / memory depend on the selected extras, base model, context length and batch size. Reserve space for downloads, checkpoints and exports; use a smoke run to check the recipe on your hardware.

CLI overview

Most commands take a model folder (containing model.yml) as their first argument. Outputs land under <model-folder>/output/ (gitignored). Run maatml <command> --help for the full flag list. Errors in your input (a missing file, an unparseable model.yml, an unregistered plugin) print one line; maatml --debug <command> prints the traceback.

Command What it does
run The whole lifecycle in one command: prepare, train, evaluate (gated), export, verify. Skips fresh steps; accepts --resume auto|PATH and --progress-file PATH
prepare Build train/val/test splits from the seed corpus (plus optional dataset.train_samples); enforces dataset.isolation / pins, records the benchmark version and corpus lock (including supporting assets), and checks source sign-offs when dataset.attribution is declared
train Fine-tune the model (causal_sft, seq2seq, multi_head_classifier, dpo, orpo, diffusion_lora, filtered_imitation, grpo, or plugin architectures); supports --smoke, --resume auto|PATH, --progress-file PATH, --set K=V, --seeds N; training.select_by picks the checkpoint on val
sweep Offline grid HPO over --param K=a,b
evaluate Score a local checkpoint or live OpenAI-compatible endpoint (served_http / serve_parity); --gate exits non-zero on a gate miss, --cache keeps per-row predictions and prediction_context, --out DIR separates reports from historical results, --batch-size N uses a predictor's predict_batch when available, --set K=V overrides configuration (never with --gate), --blind spends the blind manifest once, --strict-population refuses mismatched gate populations before loading the predictor. The token budget defaults to packaging.max_input_tokens
gates derive Floors from a run's report: Wilson 95 % at each metric's own denominator, cluster bootstrap from a cache, --seed-study; --write rewrites evaluation.gates with the derivation beside each floor
ship-check CANDIDATE BASELINE: absolute, delta and population verdict in one; --replay re-evaluates both over the current test split
operating-point derive Sweep the predictor's rescore over a val cache under a budget; --write the cut, --confirm-on-test spends test once
export Deployable bundle + manifest.json (--format safetensors|gguf|mlx|adapter|onnx, --parity); adapter checks extensions.adapter.spec, gguf supports extensions.gguf.mmproj multimodal companions
compile Dispatch an export to a registered target compiler; --require-gated rejects missing, failed or smoke-tier gate evidence
manifest amend Add files inside an export to its checksummed manifest without changing gate evidence
verify Recompute sha256 of listed manifest files and flag unlisted files under the export directory
serve JSON inference API (http default, or anthropic / plugin backends); --enforce (422), --max-retries, --auth-token, --capture
snapshot Wait for a complete policy or SFT checkpoint (>= --min-step) and write a gzip-compressed portable backup bundle without stopping training
datagen Validator-gated seed generation (--teacher, --allow-ungated)
distill Validator-gated teacher labels over a prompt pool (--replay offline)
mint Preference pairs (chosen/rejected) from validator-scored candidates
ingest Import external samples (--map field=col, --sanitize tag, --video FILE)
runs List recorded training runs (--compare tabulates their metrics); --pack RUN / --adopt BUNDLE carry a run, its evidence and its record between machines
report Runs, floors with their derivation, slices, pathologies, seed statistics and spends, regenerated from output/ alone (--format md|csv)
plan Show which lifecycle steps are stale (alias for run --dry-run)
plugins List registered trainers, validators, metrics, predictors, formats, generators, exporters, compilers, servers and environments
audit Check the environment, plugins, and a model folder; exits 1 on problems
rollout Run a checkpoint in a verifying environment (candidate-ranking line_world or free-text free_text / arithmetic_world); --label collects teacher labels for DAgger, --teacher-mix mixes teacher decisions
scaffold Create a new model folder (--architecture, --plugin, --force)
validate Check model.yml and paths (--no-plugins skips plugin code)

Multi-GPU (CUDA): accelerate launch -m maatml.cli train <model-dir>/ or torchrun --nproc_per_node=N -m maatml.cli train <model-dir>/.

QLoRA (CUDA + [cuda]): set training.quantization.load_in_4bit: true in model.yml. Preference data: dataset.format: preference_jsonl with {prompt, chosen, rejected} rows; scaffold with --architecture dpo. Verified-output post-training: use architecture: filtered_imitation for environment-verified expert iteration, or architecture: grpo for environment-verified GRPO with a frozen anchor checkpoint KL penalty (see Rollouts and Lifecycle).

Export defaults to a safetensors bundle + manifest.json. adapter exports adapters/<name>/ (adapter_config.json + adapter_model.safetensors) with header-only rank and optional extensions.adapter.spec shape checks. GGUF/MLX need external tooling (llama.cpp convert / mlx_lm), and extensions.gguf.mmproj: true emits an f16 mmproj-<name>.gguf multimodal companion. Pin base-model revisions with training.model_revision, confine LoRA to one tower with a regex training.lora.target_modules, and guard frozen submodules with training.frozen_module_prefixes.

The causal SFT path preserves per-row tool definitions, assistant tool calls (ChatML <tool_call> and Gemma 4 <|tool_call>), and tool results. It refuses rows exceeding training.max_input_tokens (RowTooLongError) instead of silently truncating prompts, supervises every assistant turn while masking other roles, and supports selective reasoning-span loss masking via training.reasoning_loss: include | mask | reasoning_only; see tool-call chat training and reasoning-span loss masking.

Docs: maatml.pages.dev · Roadmap: ROADMAP.md · In-repo docs: docs/ (from the repository root, pip install -e ".[docs]" then mkdocs serve). See the CLI reference and configuration reference.

End-to-end example (Support Ticket Triage)

The quickest model to run: a LoRA fine-tune of Qwen3-0.6B that turns a raw support ticket into {priority, category, team, summary} JSON, gated by a schema validator plus a category → team routing contract enforced outside the model. Every reference model now registers a validator and declares evaluation.gates.

git clone https://github.com/Moralfish-Labs/maatml.git
cd maatml
pip install -e ".[ml]"   # match the framework to the checked-out examples

maatml prepare  examples/support-ticket-triage/
maatml train    examples/support-ticket-triage/ --smoke   # fast pipeline check
maatml train    examples/support-ticket-triage/
maatml evaluate examples/support-ticket-triage/ --gate    # enforce eval gates
maatml serve    examples/support-ticket-triage/           # JSON inference API

For a multimodal walkthrough (image → description, servable by vLLM) see examples/vision-vlm/.

From a passed gate to a claim

A green evaluate --gate is the start of the evidence, not the end of it. The same CLI derives the floors, chooses the operating point, names the populations, and carries the run home:

maatml gates derive <model-dir> --run RUN --write      # Wilson floors, derivation beside each
maatml operating-point derive <model-dir> --run RUN --write --confirm-on-test
maatml ship-check <model-dir> CANDIDATE BASELINE        # absolute + delta + population
maatml evaluate <model-dir> --blind                     # once per frozen candidate
maatml runs <model-dir> --pack RUN                      # → --adopt on the machine that exports
maatml report <model-dir>                               # everything above, from the records alone

model.yml declares the rest: dataset.isolation / pins / blind_samples for populations, dataset.attribution for the licence table every source must be signed in, training.select_by for checkpoint selection on val. docs/evidence.md walks through it.

Keep acceptance floors fixed while measuring a corrective candidate. Deriving new floors from that candidate is a separate calibration decision, not a way to turn a failed experiment into a pass. ship-check rejects failed aggregate gate evidence, missing or mismatched comparison populations, and a candidate population that differs from a declared evaluation.gates_benchmark. --replay must produce reports over the same rows; it does not waive those checks.

Corrective data without changing the test population

For the jsonl_seed format, add reviewed corrective examples through dataset.train_samples instead of mixing them into the seed population:

dataset:
  format: jsonl_seed
  seed_samples: datasets/samples/seed_samples.jsonl
  train_samples: datasets/samples/corrections.jsonl
  benchmark_samples: datasets/samples/benchmark.jsonl

Train-only rows enter after the seed split. Preparation checks held-out group collisions, applies configured isolation and attribution checks, and records the extra input in corpus.lock.json. This preserves the seed split assignment; keep the seed data, grouping, split settings, and benchmark fixed when comparing experiments. Source attribution records provenance, not model quality.

Cloud training, progress, and recovery

Use the same model folder and CLI on the GPU runtime. Transfer the prepared data, plugin, recipe, and any local base checkpoint before running:

maatml validate <model-dir>
maatml train <model-dir> --device cuda \
  --progress-file /tmp/progress.json \
  --set training.logging_steps=1 --set training.disable_tqdm=true \
  --set training.checkpoint_snapshots=true
# Optionally back up a completed mid-run checkpoint while training continues:
maatml snapshot <model-dir> --started-after /tmp/launch.marker \
  --writer-lock /tmp/train.lock --min-step 200 --out /tmp/ckpt-200.tar.gz
# After an interruption (SIGTERM/SIGINT flushes a checkpoint and records status="interrupted"):
maatml run <model-dir> --device cuda --resume auto \
  --progress-file /tmp/progress.json \
  --set training.logging_steps=1 --set training.disable_tqdm=true
maatml evaluate <model-dir> --checkpoint RUN --cache --out <report-dir>
maatml runs <model-dir> --pack RUN

--progress-file (or FLOW_JOB_PROGRESS_FILE) writes atomic JobProgress JSON snapshots (heartbeatAt, stage, epoch, step, runId, metricName, metricValue, checkpointAgeSecs) from rank 0 so supervisors can monitor progress without scraping logs. disable_tqdm replaces the SFT progress bar with ordinary Trainer log records; logging_steps controls their frequency. On SIGTERM or SIGINT inside the training loop, MaatML flushes a checkpoint at the step boundary and records run status interrupted in runs.jsonl. training.checkpoint_snapshots: true publishes a hash-bound completion marker beside each saved single-process SFT checkpoint (single-file or sharded) so maatml snapshot captures only complete checkpoints at gzip level 1. Evaluation prints row progress and estimated time remaining. Set PYTHONUNBUFFERED=1 when an external orchestrator streams Python output. Resume requires a saved checkpoint; work since the last save may need to run again.

The orchestrator provisions and stops the GPU, streams logs, and transfers checkpoints. On the receiving machine, maatml runs <model-dir> --adopt BUNDLE imports the completed run and its evidence. A completed training run does not by itself establish an acceptance or release pass. See orchestration and recovery.

Batch scripts

# Deterministic seed corpora (no API calls)
python examples/support-ticket-triage/scripts/build_seeds.py
python examples/vision/scripts/build_seeds.py
python examples/vision-vlm/scripts/build_seeds.py
python examples/vision-describer/scripts/build_seeds.py

# Train / evaluate example models
python scripts/train_all.py --smoke
python scripts/train_all.py
python scripts/evaluate_all.py

Apple Silicon / MPS notes

  • Default precision is bf16 autocast with fp32 master weights.
  • The MPS device profile disables mid-training evaluation; setting eval_steps does not override that profile.
  • grad_checkpointing defaults to false; dataloader_num_workers=0 everywhere (multi-worker + MPS can deadlock via fork pickling).
  • PYTORCH_ENABLE_MPS_FALLBACK=1 is set by the CLI for unsupported ops.

Repository layout

examples/                   # reference task models (plugins + data)
  support-ticket-triage/    # causal LoRA SFT
  vision/                   # multitask vision
  vision-vlm/               # vision-language LoRA SFT
  vision-describer/         # seq2seq captioning

src/maatml/                 # core framework (architectures, CLI, harnesses)
scripts/                    # batch train/eval/validate
tests/                      # core unit tests

Trust boundary

A model folder is executable code, not just data. Any plugins: entry in model.yml is imported as Python the moment the folder is loaded, and every command that reads model.yml, including maatml validate and maatml plan, loads the folder. Running any maatml command against a folder therefore runs that folder's code with your privileges. Only run maatml on model folders you trust, the same way you would only run a script you trust. Use maatml validate --no-plugins to check the schema and paths without importing plugin code.

Development

See CONTRIBUTING.md for setup, PR expectations, DCO sign-off, and versioning policy. AI coding agents: AGENTS.md.

Community: CODE_OF_CONDUCT.md · Security: SECURITY.md · Changes: CHANGELOG.md

Licensing

  • maatml is licensed under the Apache License 2.0.
  • This repository does not redistribute base-model weights, only Hugging Face Hub IDs. Your fine-tuned checkpoints inherit the base model's license terms.
  • Seed corpora are fully synthetic, produced by deterministic builders under examples/*/scripts/, with no proprietary source data shipped.

Metadata

Release files for maatml 0.19.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for maatml 0.19.1
File Size Uploaded Provenance Actions
maatml-0.19.1.tar.gz 324.6 kB With provenance Details for maatml-0.19.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for maatml 0.19.1
File Interpreter ABI Platform Actions
maatml-0.19.1-py3-none-any.whl Python 3 none any Details for maatml-0.19.1-py3-none-any.whl

Total release size: 654.7 kB

Release files / maatml-0.19.1.tar.gz

Download URL maatml-0.19.1.tar.gz
Size 324.6 kB
Tags Source
SHA-256 checksum
How to use checksums
c65721a937ae3451b78c791e250355cf3c77302634d2d54030ec5130e8f09729
BLAKE2b-256 checksum
How to use checksums
e3bfd572d739e5855d0eae36cd336144633f06f959529ea7e45446beea29784a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / maatml-0.19.1-py3-none-any.whl

Download URL maatml-0.19.1-py3-none-any.whl
Size 330.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bec91bc6aaae21d066549354f1554848df40e80b480f4a541a7ee772768fbc08
BLAKE2b-256 checksum
How to use checksums
e39364b608a3c05291771537e36eb128031679c20480eccf95de5b599577a844
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Security for maatml 0.19.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for maatml-0.19.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page