CheckMAITE

CheckMAITE is an integration point for AI T&E tooling for computer vision that is based on the maite protocols (see maite documentation).

Description

What is CheckMAITE? CheckMAITE is a Python API which makes testing and evaluation of models and datasets straightforward and reproducible. It can be used to run a wide variety of Model Evaluation and Dataset Analysis investigations for both Object Detection and Image Classification computer vision problems.

It was built as an integration point for all CDAO JATIC tools and has since expanded focus. Our goal is to make T&E easier for analysts!

To learn more please visit our published documentation.

Supported environment

• OS: Linux x86_64 (CI-tested). macOS x86_64 and arm64 are supported but not CI-tested.
• Python: 3.10, 3.11, 3.12, and 3.13 with uv or pip. conda supports 3.11, 3.12, and 3.13; the committed conda lock is for 3.13. checkmaite-plugins is <3.12 on all paths.
• GPU: CPU is the supported baseline. CUDA is optional for PyTorch.
• Hardware, architectures, and internet access: see System requirements.
• Container: a batch container is built from the repository Dockerfile for Linux AMD64, with cpu and NVIDIA cuda targets. See Run CheckMAITE in a container.

Limitations and prerequisites

CheckMAITE evaluates models and datasets you supply.

• Built-in loaders cover COCO, YOLO, and VisDrone object detection, and YOLO image classification.
• Models and datasets expose an index2label map as dict[int, str]. Default torchvision weights build it from the weights. Custom torchvision weights and ONNX models need an index2label in their JSON config (default key index2label, override with index2label_key), given as a list or a dict; both are normalized to dict[int, str]. num_classes defaults to len(index2label). For object detection, set it when the model head has a different class count (e.g. a background slot). For classification it must equal len(index2label).
• Where a model's and a dataset's index2label overlap, a shared index must name the same class and a shared class name must use the same index. CheckMAITE does not validate this; mismatches produce silently wrong per-class results.
• ONNX models need the onnx (or onnx-cuda) extra. ONNX wrappers accept only uint8 images or finite float images in [0, 1] (cast to float32); other integer dtypes raise TypeError. Non-finite or out-of-[0, 1] float images raise ValueError.
• Object-detection boxes are float32 xyxy arrays of shape (N, 4). See Conventions.
• This package provides the Python API and the batch container. The web UI is the separate CheckMAITE application (checkmaite-frontend). PDF export needs the reporting extra.
• CPU is the supported baseline. Machine, OS, and Python bounds are in the supported environment.

Installation

For detailed installation instructions please refer to Setup Guide. To run CheckMAITE as a batch container instead of installing the package, see Run CheckMAITE in a container.

On macOS, brew install libomp is required (dataeval loads LightGBM, which needs libomp.dylib).

Usage

To learn how to get started using CheckMAITE, please visit our quick start guide.

There are also How-to Guides & Tutorials that will help get new users up to speed on the available functionality.

Contributing

The CheckMAITE team welcomes contributions of all forms - questions, documentation, and code contributions. Our contribution strategy is in CONTRIBUTING.md.

Reporting bugs and requesting features

Report bugs and request features as issues, using the bug report or feature request template:

• JATIC users: the GitLab issue board (bug report, feature request).
• Everyone else: the GitHub issue tracker (bug report, feature request).

CONTRIBUTING.md describes what to include and how issues are managed.

Security

Report security vulnerabilities privately, as described in SECURITY.md. Do not open an ordinary issue.

Authors and acknowledgment

This project was created for CDAO JATIC and is maintained by OpenTeams with collaborative community support.

CDAO Funding Acknowledgment

This material is based upon work supported by the Chief Digital and Artificial Intelligence Office under Contract No. W519TC-25-9-2041. The views and conclusions contained herein are those of the author(s) and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government.