DataEval Flow

DataEval Flow provides workflow orchestration for DataEval evaluators, packaging data cleaning, dataset analysis, coverage assessment, drift monitoring, OOD detection, splitting, prioritization, and parameter-sweep pipelines behind a single declarative configuration format and both headless and interactive CLIs.

📖 Documentation: https://dataeval-flow.readthedocs.io/

What DataEval Flow is for

DataEval Flow lets T&E engineers compose and run multi-stage data evaluation pipelines without writing Python glue code. Pipelines are described in YAML or JSON, executed locally or in a CUDA-enabled container, and produce both human-readable reports and machine-readable result envelopes that satisfy JATIC interoperability requirements. It builds directly on the DataEval library, so the underlying evaluators — outlier and duplicate detection, coverage and gap analysis, drift and OOD monitoring, dataset splitting, prioritization, and statistical analysis — are the same algorithms DataEval exposes, wrapped in a reproducible orchestration layer.

Target Audience

DataEval Flow is intended for data scientists, ML engineers, and T&E engineers who want to run automated data-evaluation pipelines against their image datasets and models. It is part of the JATIC suite of tools: DataEval Flow has native interoperability when using MAITE-compliant datasets and models, so its outputs compose with other MAITE-conforming JATIC tools.

Limitations and requirements for use

• Computer-vision image datasets only. DataEval Flow operates on image classification and object-detection datasets; it does not handle NLP or tabular data.
• MAITE for native interoperability. Non-MAITE sources are consumed through the built-in adapters (HuggingFace, COCO, YOLO, TorchVision, ImageFolder); native JATIC interoperability requires MAITE-compliant datasets/models.
• Some workflows need metadata. Bias, parity, and metadata-insight analyses require per-sample metadata factors to be present in the dataset.
• Some workflows need a model or embeddings. Embedding-space drift, OOD detection, and prioritization require a feature extractor (ONNX/PyTorch) or precomputed embeddings.
• Drift and OOD need a representative reference. Detection baselines are only as good as the reference dataset they are fit on.
• Batch container, not a service. The container runs a pipeline to completion and exits; it is not a long-running web service (no health-check endpoint).

System Requirements

The guidance below applies to both the container and the Python-library forms.

Tested platforms

DataEval Flow is developed and tested on Linux (Ubuntu 22.04 and 24.04, including WSL2). The Python package supports Python 3.10–3.14; the CI test matrix runs all five. The container images are all built on Ubuntu 24.04 (cpu, cu126, cu130). macOS and Windows are supported only through Docker or WSL2 and are not part of the CI test matrix — if you hit an issue on those hosts, the OS/hardware may be the root cause.

Architecture

All container images and the dependency stack target linux/amd64 (x86-64). arm64 / Apple Silicon is not built or tested; on those hosts run the CPU image under emulation or install the library from source. The dataeval_flow package ships no compiled extensions of its own, so the library form runs anywhere its dependencies (PyTorch, NumPy, SciPy) provide x86-64 wheels.

Recommended minimum hardware

A GPU is never required. When deploying the container under Kubernetes, request at least the minimum CPU/memory above; size memory to your largest dataset.

Internet access

• Installation needs network access to PyPI and the PyTorch wheel index (or, for the container, to the base image and the Harbor registry).
• First run downloads any datasets referenced from the HuggingFace Hub (and, in the tutorials, sample datasets such as MNIST / CPPE-5). Model weights referenced by URL are likewise fetched on first use.
• Offline / air-gapped operation is supported once the image, datasets, and models are staged locally: point the config at on-disk dataset/model paths and set HF_HUB_OFFLINE=1 (and HF_DATASETS_OFFLINE=1). With local inputs the batch container makes no outbound network calls of its own at run time.

Quick Start

# 1. Build CUDA 12.6 container
docker build -f docker/Dockerfile.cu126 -t dataeval:cu126 .

# 2. Show help
docker run dataeval:cu126

# 3. Run with data and output
docker run --gpus all \
  --mount type=bind,source=/path/to/data,target=/dataeval,readonly \
  --mount type=bind,source=/path/to/output,target=/output \
  dataeval:cu126

Pulling pre-built images

Pre-built, cosign-signed images are published to Harbor for every merge to main and every release tag. Pull one of these instead of building from source if you don't need to modify the code.

Rolling channel — tracks the latest commit on main. The tag is overwritten on every merge.

docker pull harbor.jatic.net/aria/dataeval:cu126   # cpu / cu126 / cu130

Pinned release channel — immutable, version-tagged images cut from v* git tags. Use these for reproducible workloads.

docker pull harbor.jatic.net/aria/dataeval:0.2.1-cu126

Verifying the signature — every published image is signed with cosign. The public key is committed at docker/cosign.pub.

cosign verify --key docker/cosign.pub harbor.jatic.net/aria/dataeval:cu126

Then drop the dataeval:cu126 reference in the Quick Start docker run commands above with the fully-qualified harbor.jatic.net/aria/dataeval:cu126 (or pinned version) and skip step 1.

Note on feature branches. Containers are only built and published from main and release tags — no image is produced for MRs or topic branches. If you want to run a feature branch as a container, check it out and follow the Quick Start to build locally; the resulting image will pick up the branch's version via git describe.

Requirements

The CUDA runtime libraries ship inside the image via PyTorch and onnxruntime-gpu, so the host needs only the NVIDIA driver and the NVIDIA Container Toolkit — no host CUDA install. The driver floors above are the CUDA major versions' minimums: any driver from the 525 series up runs a CUDA 12.x image under minor-version compatibility, and CUDA 13.0 requires a 580-series or newer driver.

Verify GPU Access

docker run --rm --gpus all nvidia/cuda:12.6.3-base-ubuntu24.04 nvidia-smi

Volume Mounts

File Permissions

The container runs as a non-root user (dataeval, UID 1000). Mounted directories for /output and /cache must be writable by the container process. There are two approaches:

Option 1: Pass your host UID (recommended)

Use --user to run the container as your host user, so mounted directories are naturally writable:

docker run --gpus all \
  --user "$(id -u):$(id -g)" \
  --mount type=bind,source=/path/to/data,target=/dataeval,readonly \
  --mount type=bind,source=/path/to/output,target=/output \
  dataeval:cu126

Option 2: Open directory permissions

Make the output and cache directories world-writable on the host:

chmod 777 /path/to/output /path/to/cache

Then run without --user. This is simpler but less secure.

Custom Data Root

The data root path can be overridden via the DATAEVAL_DATA environment variable:

docker run --gpus all \
  -e DATAEVAL_DATA=/data \
  --mount type=bind,source=/path/to/data,target=/data,readonly \
  --mount type=bind,source=/path/to/output,target=/output \
  dataeval:cu126

Environment Variables

All runtime environment variables are optional; command-line options take precedence over them (see Input Precedence below).

No secret mounts or credentials are required — DataEval Flow uses no API keys, tokens, or passwords. (DATAEVAL_FLOW_VERSION and DATAEVAL_NOX_UV_EXTRAS_OVERRIDE are build-time only and are not read at run time. UV_EXTRAS_OVERRIDE and CONTAINER_MODE are baked into the image by the build and read at run time by the entrypoint, purely to label the help text and decide whether to run the GPU check — neither is meant to be set by the caller.)

Input Precedence

For any input, the resolution order is:

1. Command-line option — --config, --data, --output, --cache
2. Environment variable — DATAEVAL_DATA, DATAEVAL_OUTPUT, DATAEVAL_CACHE
3. Built-in default — the container mount paths above (or the current directory outside the container)

Dataset and model paths inside a config file are resolved relative to the data root; a relative path not found directly is also looked up under the conventional data/ (datasets) and models/ (models) subfolders of the data root.

Interface Documentation

The container prints its full interface — mounts, environment variables, CLI options, precedence, and examples — via its help command, which is also the default when the container runs with no pipeline arguments:

docker run dataeval:cu126 --help

The library form exposes the same options via python -m dataeval_flow --help. The published container reference documents every input, default, and configuration dependency.

Configuration

Config files (YAML or JSON) can be placed anywhere in your data directory. By default, all YAML/JSON files at the root of the data mount are auto-discovered and merged.

To specify a config path explicitly:

# Config folder within data directory
docker run --gpus all \
  --mount type=bind,source=/path/to/data,target=/dataeval,readonly \
  --mount type=bind,source=/path/to/output,target=/output \
  dataeval:cu126 --config config/

# Single config file
docker run --gpus all \
  --mount type=bind,source=/path/to/data,target=/dataeval,readonly \
  --mount type=bind,source=/path/to/output,target=/output \
  dataeval:cu126 --config params.yaml

Dataset and model paths in config files are resolved relative to the data root (/dataeval by default).

Dataset Formats

A datasets entry selects its loader with format:

Both single-split datasets and multi-split dataset dicts are supported. In-memory MAITE and TorchVision datasets are also accepted programmatically via DatasetProtocolConfig, which is not serializable and so cannot be named in a config file. See the Run workflows in containers guide for the full field list of each format.

CPU Fallback

For machines without NVIDIA GPU:

docker build -f docker/Dockerfile.cpu -t dataeval:cpu .
docker run dataeval:cpu  # Shows help
docker run \
  --mount type=bind,source=/path/to/data,target=/dataeval,readonly \
  --mount type=bind,source=/path/to/output,target=/output \
  dataeval:cpu

CLI Modes

DataEval Flow has four modes:

dataeval-flow --version reports the installed build — useful for pinning down which image produced a result.

Headless execution

# Every enabled task in the config
dataeval-flow --config params.yaml --data . --output ./results

# One task by name, whether or not the config marks it enabled
dataeval-flow --config params.yaml --task clean_my_data

# Several, in the order given
dataeval-flow --config params.yaml --task clean_my_data --task analyze_my_data

# Stop the pipeline when a finding breaches its health threshold
dataeval-flow --config params.yaml --output ./results --fail-on-warning

Exit codes: 0 when every task succeeded, 1 when any task failed. Health warnings are reported but are not fatal unless you pass --fail-on-warning, which turns them into an exit code of 1 as well. Every run's result.json carries a health block per task (status, warnings, findings) so a pipeline can gate on findings without parsing the text report.

Workflow Discovery (workflows)

For images that ship without the TUI extra:

# What can this build run?
dataeval-flow workflows
dataeval-flow workflows --json

# What does a given workflow type accept?
dataeval-flow workflows data-cleaning

Interactive TUI (app)

Installation:

uv sync --extra app          # or: pip install dataeval-flow[app]

Usage:

# Launch with a blank config
python -m dataeval_flow app

# Load an existing config for editing
python -m dataeval_flow app --config /path/to/params.yaml

The TUI provides a three-pane dashboard for config editing, task execution, and result viewing. It auto-discovers available torchvision transforms, dataeval selection classes, and workflow types, generating dynamic parameter forms from their schemas.

Simple CLI Config Builder (config)

For environments without the TUI dependency:

python -m dataeval_flow config
python -m dataeval_flow config --config /path/to/params.yaml

Configs can be saved as YAML or JSON.

Encoding Descriptor (encoding)

Extract the metadata encoding descriptor from an archived result.json and write it where it can be reviewed and committed, so a later dataset is cut the same way:

python -m dataeval_flow encoding output/results/result.json -o policy/factor_bins.json
python -m dataeval_flow encoding output/results/result.json --task clean_my_data

Reference the committed descriptor from a metadata policy's encoding field. A run with -o already writes results/encoding.json beside its results; this command recovers one from a result archived earlier. See Configure metadata binning.

Dependencies

Declared runtime dependencies:

• dataeval - Core evaluation library
• datamaite - MAITE protocol dataset loaders (which brings in maite itself)
• pydantic - Structural typing and schema validation
• click - Interactive prompts for the simple CLI config builder
• pyyaml - YAML config parsing
• numpy - Array handling

PyTorch arrives transitively through dataeval. Optional extras (onnx, opencv, app, ontology, and the CUDA variants) are listed under Running Without Container.

Troubleshooting

Build appears stuck at uv sync

The Docker build may appear frozen during the uv sync step:

=> [builder 7/7] RUN uv sync --frozen --no-dev --no-install-project    1139.3s

This is normal. The step downloads ~2GB of dependencies (PyTorch, scipy, etc.) with no progress indicator.

Tip: First build is slow; subsequent builds use Docker cache and complete in seconds.

Running Without Container

The dataeval_flow package can be used standalone without Docker.

Installation:

See the Installation guide for the full matrix. In short:

pip from PyPI (no source checkout). PyTorch arrives transitively via dataeval. To choose a variant, install torch from that variant's index first, then install DataEval Flow — it accepts the build already present (omit step 1 and you'll get the CUDA-bundled manylinux build of torch from PyPI, which is much larger):

# 1. Pick your PyTorch build (cpu / cu126 / cu130)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130

# 2. Install DataEval Flow
pip install dataeval-flow

Use --index-url, not --extra-index-url, to pick a CUDA build. --extra-index-url adds an index instead of replacing PyPI, and pip then takes the highest version across both. The CUDA indexes lag the latest PyTorch release, so PyPI usually wins and you silently get the default CUDA-bundled build — the install succeeds with no warning. --index-url replaces the index outright, so it is reliable. (For CPU only, pip install dataeval-flow --extra-index-url https://download.pytorch.org/whl/cpu does work, because the CPU index tracks the latest release.)

torchvision is not installed by default — it is imported lazily and is only needed for preprocessing pipelines, the torchvision dataset adapter, and the TUI's transform discovery. Install it together with torch in step 1 so both come from the same index. Feature extras (onnx, onnx-cu126, onnx-cu130, opencv, opencv-fips, app, ontology) work normally under pip and are independent of the PyTorch variant:

pip install "dataeval-flow[onnx,opencv,app]"

The cpu / cu126 / cu130 extras do not select a PyTorch variant under pip. All three declare the same requirements (torch, torchvision); what distinguishes them is [tool.uv.sources], which routes those packages to the right wheel index. That is project metadata applied by uv when resolving from source — it is not part of the published wheel. Under pip their only real effect is pulling in torchvision. Select the variant with --index-url under pip, --torch-backend under uv pip, and use the extras only for source installs.

uv from PyPI:

uv pip install dataeval-flow --torch-backend cpu     # or cu126 / cu130 / auto

uv from source (default toolchain; uses committed uv.lock) — extras apply here:

git clone https://github.com/aria-ml/dataeval-flow.git
cd dataeval-flow
uv sync --extra cpu      # or cu126 / cu130; add --extra onnx --extra opencv --extra app as needed

poetry (source checkout; uses committed poetry.lock):

git clone https://github.com/aria-ml/dataeval-flow.git
cd dataeval-flow
poetry install

conda / mamba (source checkout; uses committed environment.yml):

git clone https://github.com/aria-ml/dataeval-flow.git
cd dataeval-flow
conda env create -f environment.yml
conda activate dataeval-flow
pip install -e .         # install the package itself; conda manages deps

Notes:

• PyTorch is installed from PyPI/download.pytorch.org in every path (it is no longer maintained on conda-forge).
• GPU variants (cu126, cu130) are only wired through uv and pip today; the Poetry/conda paths install the CPU build of PyTorch.

CLI Usage:

python -m dataeval_flow --config /path/to/config --output /path/to/output
python -m dataeval_flow --data /path/to/data --output /path/to/output

Python API Usage:

from pathlib import Path
from dataeval_flow import load_config, run_tasks

config = load_config(Path("/path/to/data/config.yaml"))
results = run_tasks(config, data_dir=Path("/path/to/data"))
print(results[0].report())

Development:

uvx --with nox-uv nox -s dev     # build .venv with the dev toolchain
source .venv/bin/activate
nox                              # run the default sessions

nox -s dev prompts for the Python version and device variant, records the choice in .cuda-version so the other sessions match it, and installs the onnx and app extras alongside. See Development Setup for the arguments it takes.

Versioning

The package version is derived from git tags — there is no hardcoded version anywhere in the source tree. hatch-vcs reads git describe --tags at build/install time and writes the resolved version to a generated src/dataeval_flow/_version.py (gitignored), which dataeval_flow.__init__ imports at runtime.

Release flow for container images:

1. Push a semver tag (e.g. v0.2.0) — this is the single source of truth for the release version.
2. The push:docker CI job runs git describe --tags --always --dirty | sed 's/^v//' to resolve ${VERSION}, then passes --build-arg DATAEVAL_FLOW_VERSION="${VERSION}" to docker buildx build for both the test and prod stages.
3. The prod stage in docker/Dockerfile.j2 redeclares the ARG and:
   • Bakes the resolved version into /app/src/dataeval_flow/_version.py so dataeval_flow.__version__ matches the wheel version at runtime.
   • Stamps the OCI org.opencontainers.image.version label with the same value.
4. The image is pushed to Harbor and cosign-signed.

The ARG DATAEVAL_FLOW_VERSION="…" default rendered into each committed docker/Dockerfile.<variant> by docker/generate.py is only used for local docker build invocations that don't pass --build-arg. Release builds always override it, so the committed default is allowed to drift from the latest tag and does not need to be regenerated at release time.

License

MIT — see LICENSE.

Contributing

See CONTRIBUTING.md for bug reports and contribution guidelines, and BRANCHING.md for the branching and release strategy.

Contact

For questions or feedback, reach out to dataeval-flow@ariacoustics.com.

Acknowledgement

CDAO Funding Acknowledgement

This material is based upon work supported by the Chief Digital and Artificial Intelligence Office under Contract No. W519TC-23-9-2033. The views and conclusions contained herein are those of the author(s) and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government.