CheckMAITE

CheckMAITE is an integration point for AI testing and evaluation tooling for computer vision workflows.

CheckMAITE provides API and UI workflows that make model evaluation and dataset analysis straightforward and reproducible. It integrates JATIC tools through common MAITE-compatible interfaces so users can run model evaluation, robustness testing, and dataset analysis workflows from a shared application layer.

CheckMAITE is based on the maite protocols (see maite documentation).

Description

What is CheckMAITE? CheckMAITE is a Python API which makes testing and evaluation of models and datasets straightforward and reproducible. It can be used to run a wide variety of Model Evaluation and Dataset Analysis investigations for both Object Detection and Image Classification computer vision problems.

It was built as an integration point for all CDAO JATIC tools and has since expanded focus. Our goal is to make T&E easier for analysts!

To learn more please visit our published documentation.

Supported environment

• OS: Linux x86_64 (CI-tested). macOS x86_64 and arm64 are supported but not CI-tested.
• Python: 3.11, 3.12, 3.13, and 3.14 with uv or pip. conda supports 3.11, 3.12, and 3.13; the committed conda lock is for 3.13. checkmaite-plugins is <3.12 on all paths.
• GPU: CPU is the supported baseline. CUDA is optional for PyTorch.
• Hardware, architectures, and internet access: see System requirements.
• Container: a batch container is built from the repository Dockerfile for Linux AMD64, with cpu and NVIDIA cuda targets. See Run CheckMAITE in a container.

Limitations and prerequisites

CheckMAITE evaluates models and datasets you supply.

• Built-in loaders cover COCO, YOLO, and VisDrone object detection, and YOLO image classification.
• Models and datasets expose an index2label map as dict[int, str]. Default torchvision weights build it from the weights. Custom torchvision weights and ONNX models need an index2label in their JSON config (default key index2label, override with index2label_key), given as a list or a dict; both are normalized to dict[int, str]. num_classes defaults to len(index2label). For object detection, set it when the model head has a different class count (e.g. a background slot). For classification it must equal len(index2label).
• Where a model's and a dataset's index2label overlap, a shared index must name the same class and a shared class name must use the same index. CheckMAITE does not validate this; mismatches produce silently wrong per-class results.
• ONNX models need the onnx (or onnx-cuda) extra, and VisDrone models need the visdrone extra; neither is in the base install. ONNX wrappers accept only uint8 images or finite float images in [0, 1] (cast to float32); other integer dtypes raise TypeError. Non-finite or out-of-[0, 1] float images raise ValueError.
• Object-detection boxes are float32 xyxy arrays of shape (N, 4). See Conventions.
• This package provides the Python API and the batch container. The web UI is the separate CheckMAITE application (checkmaite-frontend). PDF export needs the reporting extra.
• CPU is the supported baseline. Machine, OS, and Python bounds are in the supported environment.

Installation

CheckMAITE is published on PyPI. Install it with any of:

python -m pip install checkmaite   # pip
uv add checkmaite                  # uv project (or: uv pip install checkmaite)
poetry add checkmaite              # Poetry project

With conda, create an environment and install from PyPI (CheckMAITE is not on conda-forge):

conda create -n checkmaite python=3.12 pip
conda activate checkmaite
python -m pip install checkmaite

On macOS, brew install libomp is required (dataeval loads LightGBM, which needs libomp.dylib).

To install from source, optional extras (reporting, onnx, cloud, ...), and the locked uv and conda environments, see the Setup Guide. To run CheckMAITE as a batch container instead of installing the package, see Run CheckMAITE in a container.

Usage

This example builds a tiny image classification dataset, checks it for duplicates with DataEval, and evaluates a two-class torchvision model on it with MAITE. It runs on a CPU in under a minute and needs no downloads. The model has untrained weights, so its accuracy is meaningless.

import json
from pathlib import Path

# Import checkmaite before torch so it can configure PyTorch on macOS.
from checkmaite.core.image_classification import DataevalCleaning, MaiteEvaluation
from checkmaite.core.image_classification.dataset_loaders import load_yolo_classification_dataset
from checkmaite.core.image_classification.metrics import accuracy_multiclass_torch_metric_factory
from checkmaite.core.image_classification.models import TorchvisionICModel

import torch
import torchvision
from PIL import Image

# A tiny YOLO classification dataset: <root>/<split>/<class>/<image>.
# Each "dog" image is a copy of a "cat" image, so the dataset has duplicates.
root = Path("example_data")
for label in ["cat", "dog"]:
    (root / "test" / label).mkdir(parents=True, exist_ok=True)
    for i in range(4):
        Image.new("RGB", (64, 64), color=(60 * i, 0, 0)).save(root / "test" / label / f"{i}.jpg")
dataset = load_yolo_classification_dataset(root_dir=root, dataset_id="example", split="test")

# A two-class torchvision model (untrained weights) and its index2label config.
torch.save(torchvision.models.resnext50_32x4d(num_classes=2).state_dict(), root / "model.pt")
(root / "model.json").write_text(json.dumps({"index2label": {"0": "cat", "1": "dog"}}))
model = TorchvisionICModel(
    model_name="resnext50_32x4d", weights_path=root / "model.pt", config_path=root / "model.json"
)
metric = accuracy_multiclass_torch_metric_factory(num_classes=2)

# Check the dataset with DataEval, then evaluate the model with MAITE.
cleaning = DataevalCleaning().run(datasets=[dataset])
print(cleaning.outputs.duplicates)

evaluation = MaiteEvaluation().run(datasets=[dataset], models=[model], metrics=[metric])
print(evaluation.outputs.metrics)

The output lists the duplicate images (exact=[[0, 4], [1, 5], [2, 6], [3, 7]]) and the accuracy result. Every capability follows the same pattern: wrap your model, dataset, and metric, then call run on a capability.

To learn how to get started using CheckMAITE, please visit our quick start guide.

There are also How-to Guides & Tutorials that will help get new users up to speed on the available functionality.

Roadmap

Planned features and releases for the next six months are in ROADMAP.md.

Contributing

The CheckMAITE team welcomes contributions of all forms - questions, documentation, and code contributions. Our contribution strategy is in CONTRIBUTING.md.

Reporting bugs and requesting features

Report bugs and request features as issues, using the bug report or feature request template:

• JATIC users: the GitLab issue board (bug report, feature request).
• Everyone else: the GitHub issue tracker (bug report, feature request).

CONTRIBUTING.md describes what to include and how issues are managed.

Security

Report security vulnerabilities privately, as described in SECURITY.md. Do not open an ordinary issue.

Authors and acknowledgment

This project was created for CDAO JATIC and is maintained by OpenTeams with collaborative community support.

CDAO Funding Acknowledgment

This material is based upon work supported by the Chief Digital and Artificial Intelligence Office under Contract No. W519TC-25-9-2041. The views and conclusions contained herein are those of the author(s) and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government.