ranksignal

Catch ranking and retrieval quality failures in production — without ground-truth labels.

Your search, RAG, or recommender pipeline returns 200 OK. Latency is flat. Error rate is zero. Every dashboard is green.

And the results have been garbage for six hours.

from ranksignal import Monitor, Ranking

monitor = Monitor(labels={"model": "v3"})

health = monitor.observe(Ranking(results, query=query_id, requested=20))

if not health.ok:
    for report in health.degraded:
        log.warning("%s: %s", report.signal, report.reason)

starvation: returned 3 of 20 requested (15%), at or below the critical fill rate
score_collapse: scores are nearly identical (variation 0.0004); the ordering is effectively arbitrary

Why this exists

Ranking quality is measured offline, with labels, in a batch job. That is a good way to compare two models, and no way at all to find out that the one in production broke at 2am.

ranx, pytrec_eval and ir-measures are excellent at what they do — computing nDCG, MAP and friends against qrels. They are offline, batch, and label-dependent by design. ranksignal is the other half: online, per-request, and label-free. Use both.

Install

pip install ranksignal                  # zero dependencies
pip install "ranksignal[prometheus]"    # metrics export

Python 3.9+. The core has no runtime dependencies.

The idea

You cannot know whether a result is relevant without a label. You can know a great deal about whether a ranking is shaped like the broken ones:

None of these prove your results are irrelevant. They prove the ranking looks like the ones that were — which is exactly what a paging threshold is for.

Signals are scale-free where they can be

ScoreCollapse measures the coefficient of variation, not raw spread, so cosine similarities in 0..1 and BM25 scores in the hundreds are judged on the same axis. Starvation measures fill rate, not a raw count, so one threshold works for a page of 10 and a page of 100.

ScoreFloor is the deliberate exception: "a good score" is entirely ranker-specific, so it ships with no default thresholds. A plausible-looking wrong number is worse than making you choose.

from ranksignal import Monitor, ScoreFloor, Starvation

monitor = Monitor(
    signals=[
        Starvation(warn_below=0.8, critical_below=0.4),
        ScoreFloor(warn_below=0.35, critical_below=0.20),  # tuned to *your* scores
    ]
)

"I couldn't measure that" is not "that's fine"

A signal that cannot run reports SKIPPED, never OK. An unscored ranking does not quietly count as healthy, and a skipped signal is never exported as a number — a fabricated value on a dashboard is worse than a gap in it.

health.signal("score_collapse").severity  # Severity.SKIPPED
health.ok  # unaffected by skips

Detecting a bad deploy

Churn is the one signal that needs history. Give it what the query returned before — yesterday, or from the model you are replacing:

from ranksignal import Churn, Ranking, check

health = check(
    Ranking(new_results, requested=10),
    signals=[Churn(baseline=yesterday_ids, k=10, warn_below=0.5)],
)

Measured as Jaccard overlap of the top-k id sets, because a document sliding from rank 2 to rank 3 is noise, and a document vanishing entirely is not.

Prometheus

monitor = Monitor(labels={"model": "v3", "index": "docs-v2"})

# Page when retrieval starves for five minutes.
rate(ranksignal_degraded_total{signal="starvation",severity="critical"}[5m]) > 0

# Warn when fill rate sags, before anyone files a ticket.
ranksignal_value{signal="starvation"} < 0.5

Values are gauges because these are quality levels, and the natural alert is "below 0.5 for five minutes". A histogram would answer a question nobody asks about a bounded ratio.

prometheus_client is optional. Without it, Monitor still runs every signal and still returns a Health — it just does not export. A monitoring library must never be the reason a request fails, and that includes failing on an import.

Guarantees

Verified by property-based tests across hundreds of generated rankings (tests/test_properties.py):

• Evaluating a ranking never raises for any well-formed input.
• Evaluation never mutates the ranking it is given.
• Identical inputs produce identical verdicts.
• Every signal returns exactly one report — nothing vanishes unrecorded.
• A SKIPPED signal never carries a value, and never exports one.
• Duplicate fraction and churn overlap always land in 0..1. Fill rate is deliberately not capped above — returning 20 when 10 were asked for is a fill rate of 2.0, and clamping it would hide a real over-delivery bug.
• ScoreCollapse is scale-invariant: rescaling every score cannot change the verdict, which is what lets one threshold survive a swap from cosine similarity to BM25.

Because checking is a pure function returning a plain value, you can assert on it:

def test_empty_retrieval_is_critical():
    health = check(Ranking([], requested=10))
    assert health.signal("starvation").severity is Severity.CRITICAL

Development

git clone https://github.com/mathewOracle/ranksignal && cd ranksignal
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest && ruff check . && ruff format --check . && mypy

Contributions welcome — see CONTRIBUTING.md. New signals are especially welcome and shouldn't require touching anything but your own class.

License

MIT — see LICENSE.