ragdrift

5-dimensional drift detection for production RAG systems. Rust core, Python frontend.

The problem

You have a RAG system in production. It worked when you launched. Then, quietly, it stopped working as well. By the time the support tickets arrive and you finally get an eval-set rerun on the calendar, the regression has been live for three weeks.

The reason is that retrieval quality drifts in five different places, and none of them produce a loud signal on their own. The embedding model gets re-trained and now maps the same input to a slightly different region. The corpus gets reindexed and the document distribution shifts. Users find your product and start asking different questions. The re-ranker score distribution drifts because of one of the above. Each of these can degrade answer quality without breaking a single test.

ragdrift watches all five and alerts on the one that moves.

Install

pip install ragdrift                    # core
pip install 'ragdrift[opensearch,aws]'  # OpenSearch adapter + CloudWatch exporter
pip install 'ragdrift[dev]'             # plus pytest, mypy, ruff, maturin

30-second quickstart

import numpy as np
import ragdrift

rng = np.random.default_rng(0)

# Baseline: a frozen sample from when retrieval quality was known-good.
baseline = ragdrift.BaselineSnapshot(
    embeddings=rng.standard_normal((4096, 384)).astype(np.float32),
    confidence_scores=rng.uniform(0.85, 0.99, size=4096).astype(np.float64),
)

# Current: today's window. In production, pull from your vector store.
current_emb = rng.standard_normal((4096, 384)).astype(np.float32) + 1.5
current_conf = rng.uniform(0.55, 0.75, size=4096).astype(np.float64)

monitor = ragdrift.RagDriftMonitor(baseline)
report = monitor.check(embeddings=current_emb, confidence_scores=current_conf)

if report.any_exceeded():
    for s in report.scores:
        print(f"[{'DRIFT' if s.exceeded else 'ok':5s}] {s.dimension:11s} "
              f"score={s.score:.4f} method={s.method}")

Run python examples/quickstart.py for a synthetic end-to-end demo.

Why not X

• Arize Phoenix / Phoenix Arize. Excellent eval and tracing. Less focused on multi-dimensional batch drift; their drift surface is embeddings-only and tied to the Phoenix collector.
• Evidently. Great at tabular data drift, broad statistical coverage, no native treatment of embedding or query drift in the RAG sense.
• WhyLabs / whylogs. Excellent profiling primitive. The whylog format is optimized for compaction; getting an embedding distribution test out of it is more work than calling MMD directly.
• NannyML. Strong for supervised tabular drift with a holdout label. Not a fit for the unlabeled embedding/query side of RAG.

ragdrift is the smaller-scope option: five concrete dimensions, one batch function call per dimension, fast Rust core, no service to run.

Architecture

+----------------+     +------------------+     +-------------------+
| python facade  | --> | ragdrift._native | --> | ragdrift-core     |
| (typed)        |     | (PyO3 bindings)  |     | (Rust statistics) |
+----------------+     +------------------+     +-------------------+
       |                                                 ^
       v                                                 |
+----------------+                                       |
| adapters/      |  pull baseline + current  ------------+
|   opensearch   |  embeddings, features, etc.
|   pgvector
|   pinecone     |
+----------------+
       |
       v
+----------------+
| exporters/     |  publish DriftReport
|   cloudwatch   |
|   prometheus   |
|   datadog      |
+----------------+

All numerics live in Rust. The Python layer is a thin typed facade plus adapters and exporters. One abi3-py310 wheel covers Python 3.10–3.13.

Status

0.x. The public API may break between minor versions while the library finds its shape; semver is respected within 0.x.y patch releases.

Contributing

See CONTRIBUTING.md. The local quality gates are:

cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
maturin develop
pytest -v
mypy --strict python/ragdrift
ruff check . && ruff format --check .

License

Dual-licensed under MIT and Apache-2.0. Pick whichever is friendlier to your downstream. See LICENSE-MIT and LICENSE-APACHE.