dedrift 0.0.3

Statistically rigorous behavioral drift detection for AI agents. Agents don't throw errors when they degrade — dedrift catches it.

dedrift

Agents don't throw errors when they degrade. They keep confidently producing worse outputs. dedrift catches it.

dedrift is an open-source Python package that detects silent behavioral drift in AI agents. It logs agent interactions, runs a frozen canary suite repeatedly, extracts behavioral signatures, and applies statistically rigorous drift detection with config-change attribution ("behavior shifted within 6h of the model version change").

Why dedrift

Model updates, prompt edits, tool-schema changes, RAG index refreshes, and provider-side silent updates all shift agent behavior without throwing a single error. Existing monitoring catches crashes, not character changes.

dedrift's differentiation is statistical correctness:

• Every detector controls its documented false-alarm rate — validated by simulation tests in CI.
• All alerting passes through FDR control (Benjamini–Hochberg). No raw per-test p-values dressed up as alerts.
• Every alert requires both statistical significance and a configurable effect-size (materiality) threshold. Fewer, higher-confidence alerts.
• LLM outputs are stochastic: canaries run N times per cycle and we compare distributions, never single outputs.
• Dual baselines: every check runs against a rolling recent window (sudden breaks) and a frozen golden baseline (boiling-frog drift).
• Honest about power: small N means low detection power, and the docs show you the math instead of hiding it.

Status

Pre-alpha, under active development. Working today: logging schema + store, canary runner (N repetitions per cycle), Tier-1 structural signatures, the full detector battery (KS/AD/Welch/Levene/bootstrap-P95/two-proportion z, PSI, Page–Hinkley) with BH-FDR and materiality gating, dual baselines, config-change attribution, and deterministic markdown reports — all with calibration and power tests enforced in CI. Coming for v0.1.0: embedding signatures (MMD), the 10-minute README demo, and a PyPI release.

Install

pip install dedrift              # core: zero ML dependencies
pip install "dedrift[embeddings]"  # + semantic signatures (sentence-transformers)
pip install "dedrift[judge]"       # + LLM-judge tier

For development: pip install -e ".[dev]".

Quickstart (target v0 walkthrough)

dedrift init                 # create a project
# ... log agent interactions, run canaries ...
dedrift check                # drift detection with FDR + materiality gating
dedrift report               # deterministic markdown report with attribution

A full simulated demo (synthetic agent, mid-log model swap, detection + attribution) ships with v0.1.0.

dedrift Pro

Anytime-valid sequential inference (e-processes), conditional production-traffic drift, and importance weighting are part of a separate commercial tier and are not in this repository.

License

AGPL-3.0-only. See LICENSE.