llm-driftscope 0.1.0

Local-first semantic drift detection for LLM outputs.

DriftScope

DriftScope is a local-first Python library for detecting semantic drift in LLM outputs. It wraps your model calls, stores prompts and responses in SQLite, embeds outputs locally, calibrates a baseline from your own traffic, and reports when later outputs move outside the learned normal range.

No hosted dashboard. No external data pipeline. No cloud account required.

Install

pip install llm-driftscope

For sentence-transformer embeddings:

pip install "llm-driftscope[embeddings]"

Quick Start

from driftscope import monitor

@monitor(session="support-bot")
def call_llm(prompt: str) -> str:
    return my_llm_client.chat(prompt)

answer = call_llm("How do I reset my password?")

By default, events are stored at .driftscope/driftscope.sqlite3.

Generate a local report:

driftscope report --session support-bot

Try the included drift simulation:

python examples/simulate_drift.py

Sample output:

DriftScope report

Session: demo
Samples: 9
Status: drift
Latest score: 1.0015
Latest distance: 1.0202
Threshold: 0.2229
Topic: warming up; topic drift needs two windows
Changepoint: warming up; changepoint detection needs more samples
Drift events: 3

Why DriftScope Exists

Infrastructure metrics can tell you that a service is healthy. They cannot tell you that an LLM has quietly started answering in a different semantic region than it used to. Hosted observability platforms can help, but many developers need a small embeddable library that works inside an existing codebase and keeps data local.

DriftScope focuses on that gap.

Methodology

Fixed distance thresholds are brittle. A customer support bot should usually be consistent. A creative writing assistant naturally has wider semantic variation. DriftScope avoids hardcoded assumptions by calibrating against each application's own warmup data.

During warmup, DriftScope stores output embeddings and estimates the empirical distribution of distances from the baseline centroid. Once enough samples exist, it sets the alert threshold from a high percentile of that null distribution. New outputs are compared against the centroid, smoothed with EWMA, and flagged when the smoothed drift score crosses the calibrated threshold.

The v1 detector reports:

• raw semantic distance from the baseline
• EWMA-smoothed drift score
• calibrated threshold
• drift status
• warmup progress
• topic distribution drift
• changepoint detection over semantic distance

API

from driftscope import DriftConfig, DriftScope

config = DriftConfig(
    warmup_samples=50,
    threshold_percentile=99,
    ewma_alpha=0.3,
)

scope = DriftScope(session="my-app", config=config)
result = scope.record(prompt="hello", output="world")

if result.drifted:
    print(result.score, result.threshold)

Decorator form:

from driftscope import monitor

@monitor(session="my-app")
def generate(prompt: str) -> str:
    return "model output"

CLI

driftscope report
driftscope report --session my-app
driftscope report --db .driftscope/driftscope.sqlite3
driftscope report --json
driftscope sessions
driftscope export --format csv --out events.csv

Optional sidecar API:

pip install "llm-driftscope[server]"
driftscope serve

The sidecar exposes GET /health, POST /events, and GET /report.

Optional Extras

pip install "llm-driftscope[topics]"
pip install "llm-driftscope[changepoints]"
pip install "llm-driftscope[server]"
pip install "llm-driftscope[full]"

Development

pip install -e ".[dev]"
python -m pytest
python -m build