GATE: Data Drift Detection for Machine Learning Pipelines

GATE is a Python module that detects drift in partitions of data. GATE computes partition summaries, which are then fed into an anomaly detection algorithm to detect whether a new partition is anomalous. This minimizes false positive alerts when detecting drift in machine learning (ML) pipelines, where there may be many features and prediction columns.

Support for Embeddings

We now support drift detection on embeddings, in addition to structured data. GATE considers both the structured data and the embeddings when computing partition summaries and detecting drift. Check out the embeddings page for a walkthrough of how to use GATE with embeddings.

Installation

GATE is available on PyPI and can be installed with pip:

pip install gate-drift

Note that GATE requires Python 3.8 or higher.

Usage

GATE is designed to be used with Pandas dataframes. Check out the documentation for a walkthrough of how to use GATE.

Research Contributions

GATE was developed and is maintained by researchers at the UC Berkeley EPIC Lab.

An initial version of GATE was developed as part of a collaboration with Meta, and the research paper, "Moving Fast With Broken Data" by Shankar et al., is available on arXiv. This module slightly differs from the original implementation, but the core ideas around partition summaries and anomaly detection are the same.