replysignal

Catch broken LLM generations in production — without ground-truth labels or an LLM judge.

Your model returns 200 OK. Latency is flat. Error rate is zero.

And it's been looping the same sentence for six hours, or refusing every third request, or answering in JSON wrapped in "Sure! Here's the JSON:".

from replysignal import Monitor

monitor = Monitor(labels={"model": "gpt-4o"})

health = monitor.observe(reply_text)

if not health.ok:
    for report in health.degraded:
        log.warning("%s: %s", report.signal, report.reason)

repetition: 63% of 3-grams are repeats; the model is looping
format_violation: JSON was requested but the reply does not parse, starting 'Sure! Here's the JSON:'

Why this exists

Generation quality is measured offline, against labels, or by paying for a second LLM call to judge the first one. Both are fine for evaluating a model change. Neither tells you a production model started looping at 2am.

ragas, deepeval, and trulens-eval are excellent at what they do — judging correctness and faithfulness with labels or a judge model. They are built for offline evaluation and batch runs. replysignal is the other half: online, per-response, and judge-free. Use both — this is what runs on every single response; save the judge for a sample.

Install

pip install replysignal                  # zero dependencies
pip install "replysignal[prometheus]"    # metrics export

Python 3.9+. The core has no runtime dependencies.

The idea

You cannot know a reply is correct without a judge. You can know a great deal about whether it's shaped like the broken ones:

None of these prove an answer is wrong. They prove it's shaped like the ones that were — which is exactly what a paging threshold is for.

Every signal is honest about what it needs

Only Truncation, Repetition, Emptiness, and Refusal run by default — they need nothing but the text itself. PromptEcho, Ungrounded, and FormatViolation need extra context and SKIP cleanly without it, rather than guessing:

from replysignal import Reply, Ungrounded, check

reply = Reply(
    text=answer_text,
    prompt=user_question,  # unlocks PromptEcho
    sources=retrieved_chunks,  # unlocks Ungrounded
    expected_format="json",  # unlocks FormatViolation
    finish_reason=api_response.choices[0].finish_reason,
)

health = check(reply, signals=[Ungrounded(warn_below=0.3)])

"I couldn't measure that" is not "that's fine"

A signal that cannot run reports SKIPPED, never OK. A reply nobody could judge does not quietly count as healthy, and a skipped signal is never exported as a number — a fabricated value on a dashboard is worse than a gap in it.

health.signal("ungrounded").severity  # Severity.SKIPPED, no sources given
health.ok  # unaffected by skips

Truncation trusts the provider first

If the API tells you finish_reason="length", that's a fact, not a heuristic — Truncation reports it as CRITICAL immediately. Only without that does it fall back to checking whether the text ends on terminal punctuation, and even then only warns, since a bare list item or code block legitimately lacks a full stop.

Refusal is a rate, not an incident

One refusal is often the right call. A rising rate of them across your fleet is the interesting signal — a prompt regression, a model swap, an over-tuned filter — which is why this reports WARNING, not CRITICAL: the value is the trend line, not the individual response.

Prometheus

monitor = Monitor(labels={"model": "gpt-4o", "endpoint": "chat"})

# Page when repetition spikes for five minutes.
rate(replysignal_degraded_total{signal="repetition",severity="critical"}[5m]) > 0

# Watch the refusal rate trend, not any single refusal.
rate(replysignal_degraded_total{signal="refusal"}[1h])

prometheus_client is optional. Without it, Monitor still runs every signal and still returns a Health — it just does not export. A monitoring library must never be the reason a request fails, and that includes failing on an import.

Guarantees

Verified by property-based tests across hundreds of generated replies (tests/test_properties.py):

• Evaluating a reply never raises for any well-formed input, including adversarial unicode, empty strings, and control characters.
• Evaluation never mutates the reply it is given.
• Identical inputs produce identical verdicts.
• Every signal returns exactly one report — nothing vanishes unrecorded.
• A SKIPPED signal never carries a value, and never exports one.
• The bounded signals (truncation, repetition, format violation, echo, grounding, refusal) always land in 0..1 — they measure a fraction of the reply itself, which cannot be over-delivered the way a page of search results can.

Because checking is a pure function returning a plain value, you can assert on it:

def test_empty_reply_is_critical():
    health = check("")
    assert health.signal("emptiness").severity is Severity.CRITICAL

Development

git clone https://github.com/mathewOracle/replysignal && cd replysignal
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest && ruff check . && ruff format --check . && mypy

Contributions welcome — see CONTRIBUTING.md. New signals are especially welcome and shouldn't require touching anything but your own class.

License

MIT — see LICENSE.