llm-eval-exporter 0.2.0

A lightweight Prometheus exporter for LLM eval metrics: faithfulness, semantic drift, cost, and CI/CD regression gating.

llm-eval-exporter

A lightweight Prometheus exporter for LLM eval metrics — hallucination risk (via LLM-as-judge faithfulness scoring), semantic drift across model/prompt versions, latency, and token cost — designed to plug into observability stacks teams already run, instead of requiring adoption of a new platform.

Why this exists

Most LLM observability tools (Langfuse, Braintrust, Arize Phoenix, etc.) are full platforms: their own UI, their own storage, their own workflow. That's the right call for teams building observability from scratch. But plenty of teams already run Prometheus + Grafana for every other service they operate, and would rather their LLM calls show up as three more panels on an existing dashboard than as a separate tool with a separate login.

llm-eval-exporter wraps your LLM calls, computes eval scores, and exposes them as standard Prometheus metrics. It's a library, not a platform — a few lines of Python, one /metrics endpoint, no new UI to learn.

What it tracks

Metric	What it tells you
llm_request_latency_seconds	p50/p95/p99 latency per model/provider
llm_request_total	Request volume and error rate
llm_tokens_total	Prompt/completion token usage (cost proxy)
llm_faithfulness_score	LLM-judge score (0-1) for how well a response sticks to its provided context — a hallucination proxy
llm_semantic_drift	Cosine distance between a response and its recorded baseline for the same prompt — catches silent behavior changes across model upgrades or prompt edits
llm_eval_failures_total	Count of responses that failed a faithfulness or drift threshold

Quickstart

pip install -r requirements.txt
python examples/demo.py
# metrics now live at http://localhost:9464/metrics

To run against the real Anthropic API instead of mocks:

export ANTHROPIC_API_KEY=sk-ant-...
python examples/anthropic_example.py

This traces a real claude-sonnet-5 call and judges its faithfulness with a cheaper claude-haiku-4-5 call — the standard cost-effective pattern for LLM-as-judge in production (you don't need your most expensive model just to check "did this answer stick to the context").

Then, optionally, bring up Prometheus + Grafana pointed at it:

docker compose -f examples/docker-compose.yml up
# Grafana at http://localhost:3000 (anonymous admin access, demo only)

Usage in your own code

from llm_eval_exporter import LLMTracker, start_metrics_server

start_metrics_server(port=9464)  # call once at app startup

tracker = LLMTracker(
    judge_fn=my_judge_fn,   # any callable: str prompt -> str response
    embed_fn=my_embed_fn,   # optional: defaults to sentence-transformers locally
)

with tracker.track(
    model="gpt-4o",
    provider="openai",
    prompt_id="refund_policy_qa",  # stable id -> enables drift tracking
    context=retrieved_context,     # enables faithfulness scoring
) as call:
    response = your_llm_client.call(prompt)
    call.set_response(
        response.text,
        prompt_tokens=response.usage.prompt_tokens,
        completion_tokens=response.usage.completion_tokens,
    )

judge_fn and embed_fn are intentionally pluggable — bring your own LLM client (Anthropic, OpenAI, a local model) rather than depending on one vendor's SDK.

Design notes

• SQLite by default for baseline embeddings and eval logs — a single-file, zero-ops dependency, matching this project's "drop it into an existing stack" philosophy. Swap for Postgres if it needs to scale past one instance.
• Faithfulness scoring is a lightweight, single-call analogue of RAGAS-style faithfulness — not a claim to replace a dedicated eval framework, just enough signal to catch regressions cheaply.
• Drift detection needs a prompt_id you assign to prompts you care about tracking over time; the first call for a given id just records a baseline, so drift shows up starting from the second call.

Status

Early-stage side project — built to explore a gap in the current LLM observability tooling landscape (see docs/market-notes.md for the reasoning), not a production-ready alternative to the platforms above. Contributions and issues welcome.

Running tests

pip install -r requirements.txt
pytest tests/ -v