agentdrift 0.1.0

Detect when your AI agents get dumber. Open-source quality drift monitoring.

AgentDrift

Detect when your AI agents get dumber.

Open-source quality drift monitoring for AI agents. Track agent outputs over time, detect quality degradation, and get alerts before your agents go off the rails.

The Problem

AI agents degrade over time. Context pollution, error accumulation, prompt drift, and model updates silently erode output quality. By the time you notice, your users already have.

Research shows 42% of long-running agents experience quality drift, causing a 3.2x increase in human intervention.

Quick Start

pip install agentdrift

from agentdrift import track_agent

@track_agent("my-agent")
def my_agent(query: str) -> str:
    return llm.call(query)

Check drift status:

agentdrift status

┌──────────────────────────────────────┐
│         Agent Drift Status           │
├──────────┬──────────┬───────┬───────┤
│ Agent    │ Status   │ Score │ Change│
├──────────┼──────────┼───────┼───────┤
│ my-agent │ STABLE   │ 0.847 │ +2.1% │
│ qa-bot   │ DRIFTING │ 0.612 │-18.3% │
└──────────┴──────────┴───────┴───────┘

Features

• 1-line integration - Decorator or manual logging
• Quality scoring - LLM-as-Judge auto-scoring (Anthropic / OpenAI)
• Drift detection - Baseline vs recent quality comparison
• CLI dashboard - Quick status checks from terminal
• Golden tests - Regression testing for agents (coming soon)
• Alerts - Slack/Discord/Email webhooks (coming soon)

How It Works

Agent Call → Log Trace → Score Quality → Detect Drift → Alert

1. Track: Log every agent input/output with the SDK
2. Score: LLM-as-Judge rates output quality (0.0-1.0)
3. Compare: Recent scores vs baseline period
4. Alert: Notify when quality drops below threshold

Manual Logging

from agentdrift import AgentTracker

tracker = AgentTracker("my-agent")

result = my_agent(query)
tracker.log(
    input=query,
    output=result,
    tokens_in=100,
    tokens_out=50,
    latency_ms=1200,
)

Drift Detection

from agentdrift import DriftDetector

detector = DriftDetector(
    baseline_window=50,   # first N traces as baseline
    recent_window=20,     # last N traces to compare
    threshold_pct=10.0,   # alert if quality drops 10%+
)

report = detector.check("my-agent")
print(report.message)
# "DRIFT DETECTED: my-agent quality dropped 18.3% (baseline: 0.847 -> current: 0.612)"

Examples

See the examples/ directory:

• Basic tracking & drift detection
• LangChain integration
• Quality scoring with LLM-as-Judge

Roadmap

• Agent call tracking (SDK)
• Quality scoring (LLM-as-Judge)
• Drift detection
• CLI dashboard
• Web dashboard
• Golden test suite (agentdrift test init / agentdrift test run)
• Alert webhooks (Slack, Discord, Email)
• Multi-agent interaction tracing
• OpenTelemetry export
• Managed cloud service

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

MIT