context-drift-detector-py 0.1.0

Detect topic drift between user intent, retrieved context, and AI answers. Python port of @mukundakatta/context-drift-detector.

context-drift-detector-py

Detect topic drift between user intent, retrieved context, and AI answers. A fast, lexical Jaccard-overlap heuristic for "did the model wander off?" -- useful as a cheap first-pass guardrail before reaching for an embedding-based check. Zero runtime dependencies.

Python port of @mukundakatta/context-drift-detector.

Install

pip install context-drift-detector-py

Usage

from context_drift_detector import detect

intent  = "What is the capital of France?"
context = ["Paris is the capital of France. It sits on the Seine."]
answer  = "Paris is the capital of France."

report = detect(intent, context, answer)
report.drift              # False
report.drift_score        # 0.0 - 1.0 (higher = more drift)
report.signals            # dict of jaccard overlaps
report.signals["answer_to_context"]  # 0.0 - 1.0

When drift is real:

report = detect(
    intent="What is the capital of France?",
    context_chunks=["Paris is the capital of France."],
    answer="Cats love tuna and naps.",
)
report.drift           # True
report.drift_score     # high (e.g. > 0.65)

API

detect(
    intent: str,
    context_chunks: str | Sequence[str],
    answer: str,
    *,
    threshold: float = 0.65,
    min_term_len: int = 3,
) -> DriftReport

DriftReport fields:

Field	Meaning
drift	True iff drift_score > threshold.
drift_score	0.0-1.0; weighted blend of answer-to-context (60%) and answer-to-intent (40%) overlap, inverted.
signals.intent_to_context	Jaccard overlap between intent and retrieved context.
signals.answer_to_context	Jaccard overlap between answer and retrieved context.
signals.answer_to_intent	Jaccard overlap between answer and intent.
intent_terms / context_terms / answer_terms	Frozensets of the terms used.

detect_context_drift(...) is exported as a JS-aligned alias.

How it works

Tokenizes each input into the lowercase set of min_term_len-char alphanumeric runs, then computes pairwise Jaccard overlaps. Empty inputs short-circuit to drift-free; this is intentional so a totally absent retrieval doesn't get flagged as drift on its own.

This is a cheap heuristic -- it doesn't catch paraphrases, synonyms, or semantically grounded contradictions. Use it as a fast first filter, then invest in an embedding/LLM-as-judge check for borderline cases.

See the JS sibling's README for the full design notes.