llmranker

LLM-based ranking and reasoning algorithms for search and recommendation. Currently includes pointwise, pairwise, listwise, setwise, and tournament-style (TourRank) ranking, with more strategies planned (see ROADMAP.md), implemented on top of LiteLLM so the same code runs against OpenAI, Gemini, Anthropic, Azure, Bedrock, local Ollama models, or any of the 100+ providers LiteLLM supports.

from llmranker import Candidate, LLMConfig, SetwiseRanker

ranker = SetwiseRanker(LLMConfig(model="gpt-4o-mini"), num_child=4, k=5)

candidates = [
    Candidate(id="1", text="A budget hostel in the city center."),
    Candidate(id="2", text="A five-star beachfront resort with a spa."),
    Candidate(id="3", text="A family-run guesthouse near the old town, kid-friendly."),
]

result = ranker.rank(query="affordable, family friendly, near historical sites", candidates=candidates)
print([c.id for c in result])  # ['3', '1', '2']

What this is

Before you fine-tune a ranking model or build an embedding index, you can often just ask an LLM which candidate is more relevant to a query. This package is a toolkit of strategies for doing exactly that: scoring, comparing, sorting, or tournament-ranking a list of candidates with any LLM, no training required. Each strategy is grounded in published IR/NLP research, cited per-strategy below and in full under Citing the underlying research:

Strategy	How it works	LLM calls	Notes
Pointwise	Score each candidate independently (0-10)	O(n)	Cheapest, but ignores relative preference between candidates
Pairwise	Repeatedly ask "A or B?", sort via heapsort/bubblesort/allpairs	O(n log n) to O(n²)	Simple, robust comparisons; optional self-consistency bias-checking
Setwise	Ask "which of these k is best?", sort via k-ary heapsort/bubblesort/insertion	O(n log n / log k)	Fewer calls than pairwise for the same sort, longer prompts
Listwise	Ask the LLM to output a full ranking of a sliding window at once	O(n / step)	Fewest calls, but degrades as window size grows
TourRank	Group candidates like a sports tournament, LLM picks winners per group, repeat over several stages and tournament runs, sum points	More calls, ensembled over multiple runs	Most robust to candidate input order; see TourRank paper

All five are zero-shot: no training data, no fine-tuning, no embeddings. You give it a query and a list of candidates, it gives you a ranked list.

Why LLM-based ranking

• No training data. New inventory, a new market, or a one-off internal tool rarely comes with click/purchase logs to train a ranker on.
• Captures compositional, natural-language preference. "Family friendly, near historic sites, not on the beach" is a conjunction of soft constraints that keyword search can't express and embedding search tends to blur together.
• Cheap at the scale that matters for reranking. You're not ranking your whole catalog with an LLM; you're reranking the top-k (dozens, not millions) that a cheap first-pass retrieval already narrowed down.

See examples/hotel_recommendation/ for the full worked example this README's numbers come from, and examples/ for RAG document reranking, product search, and multi-provider comparisons.

Install

pip install llmranker

Set whichever provider's API key you're using as an environment variable: LiteLLM reads the standard ones automatically (OPENAI_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY, ...). See LiteLLM's provider docs for the full list, including self-hosted/local options that need no key at all.

Quickstart

Prefer an interactive walkthrough? Open examples/quickstart.ipynb in Colab.

from llmranker import Candidate, LLMConfig, PairwiseRanker

ranker = PairwiseRanker(LLMConfig(model="gpt-4o-mini"), strategy="heapsort", k=10)

candidates = [Candidate(id=str(i), text=doc) for i, doc in enumerate(my_documents)]
result = ranker.rank(query="my search query", candidates=candidates)

for c in result:
    print(c.id, c.score)

Swap providers in one line

Every ranker takes an LLMConfig, whose model field is a LiteLLM model string. Nothing else about your code changes:

LLMConfig(model="gpt-4o-mini")                     # OpenAI
LLMConfig(model="gemini/gemini-1.5-flash")         # Google Gemini
LLMConfig(model="claude-3-5-sonnet-20241022")      # Anthropic
LLMConfig(model="azure/my-deployment-name")        # Azure OpenAI
LLMConfig(model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0")  # AWS Bedrock
LLMConfig(model="ollama/llama3")                   # local, via Ollama

See examples/multi_provider_swap.py.

Choosing a strategy

Rough guidance, in order of what to reach for first:

• Start with setwise (num_child=4-8, strategy="heapsort"), the best cost/quality tradeoff for most use cases.
• If you want the simplest possible mental model (and don't mind more LLM calls), use pairwise.
• If latency matters more than call count and your candidate list is small (fits in one window), use listwise.
• Use pointwise when you need a standalone relevance score per candidate (e.g. for thresholding "is this even relevant at all") rather than just a ranking, or when n is large and you can't afford comparisons at all.
• Use TourRank when the order candidates arrives in is unreliable (or you don't have one) and you want a result that doesn't depend on it. It's more expensive than setwise, but explicitly designed to be robust to input order, unlike listwise's sliding window.
• If cost is the constraint and your candidate list is long, don't run an expensive strategy over everything: cascade a cheap ranker (pointwise) to narrow the field, then an expensive one (setwise) to carefully re-rank just the survivors. See Cascading.

Concurrency

Every ranker takes a max_concurrency param (default 5) that controls how many LLM calls run at once via a thread pool. Calls are parallel by default, and max_concurrency=1 forces fully sequential behavior.

It only speeds up strategies whose calls don't depend on each other's results:

Strategy	Parallelized by max_concurrency?
PointwiseRanker	Yes: every candidate is scored independently
PairwiseRanker(strategy="allpairs")	Yes: every comparison is independent
PairwiseRanker(strategy="heapsort"/"bubblesort")	No: each comparison's outcome determines the next one
SetwiseRanker (any strategy, incl. "insertion")	No: same reason, n-ary
ListwiseRanker	No: each window's input is the previous window's output
TourRankRanker	Yes, within a stage: every group's LLM call is independent of the others; stages and tournament runs themselves stay sequential

For the non-parallelizable strategies, max_concurrency is accepted for constructor-signature consistency but genuinely does nothing; that's documented on each class rather than silently ignored.

# fast: dispatches all scoring calls in parallel, up to 5 at once
PointwiseRanker(LLMConfig(model="gpt-4o-mini"))

# more parallel, if your provider/plan can take it
PointwiseRanker(LLMConfig(model="gpt-4o-mini"), max_concurrency=15)

# fully sequential, useful on a strict rate limit (e.g. a free tier)
PointwiseRanker(LLMConfig(model="gpt-4o-mini"), max_concurrency=1)

If you're hitting rate limits (429s) on a free or low tier, lower max_concurrency rather than relying on retries alone. The built-in retry/backoff in llmranker.llm.call_llm handles occasional transient errors, but it won't save you from a provider that's rejecting bursts of concurrent requests outright.

Quality: reasoning, self-consistency, structured output

How hard a ranker works to get a reliable judgment is controlled by three params on every ranker, kept separate from the LLMConfig that controls which model it's talking to: reasoning, num_samples, and structured_output.

ranker = SetwiseRanker(LLMConfig(model="gpt-4o-mini"), reasoning=True)

Reasoning

reasoning=True asks the model to think step by step before giving its final answer, shown to help across a 2025 wave of reasoning-reranker papers (Rank1, Rank-R1, and others). This is a prompting technique, not a switch to a dedicated reasoning model; it works with any chat model. (If you want to route to an actual reasoning-capable model instead, that's just a model string, e.g. LLMConfig(model="o1-mini"), orthogonal to this flag.) It doesn't change how many calls are made, only prompt and completion content: expect longer, more expensive completions. A low default max_tokens on some providers can truncate a reasoning chain before it reaches the final answer; raise it via LLMConfig(extra_kwargs={"max_tokens": ...}) if you see that happen.

Reducing position bias with num_samples

LLMs have a documented bias toward whichever candidate happens to be listed first (or second, model-dependent) in a pairwise/setwise prompt, independent of actual content. num_samples repeats each judgment that many times and combines the results (mean for pointwise scores, majority vote for pairwise/setwise choices, a Borda-style merge for listwise rankings) instead of trusting a single call. On PairwiseRanker and SetwiseRanker, each sample also randomly reassigns which candidate lands on which label before asking, which cancels position bias as a side effect: it's no longer tied to a fixed slot, just noise the majority vote averages out.

ranker = PairwiseRanker(LLMConfig(model="gpt-4o-mini"), num_samples=5)

This costs num_samples calls per judgment instead of 1, dispatched in parallel via the same max_concurrency every ranker already uses, so it adds spend rather than wall-clock time. num_samples only helps at LLMConfig(temperature=...) above 0: at the default temperature=0.0 every repeat returns the same answer, so PointwiseRanker logs a warning if you raise num_samples without also raising temperature. TourRankRanker has its own repeated-sampling mechanism (num_tournaments) and ignores num_samples.

Structured output

structured_output=True uses LiteLLM's normalized JSON-schema response_format instead of regex-parsing free text, for providers that support it:

ranker = SetwiseRanker(LLMConfig(model="gpt-4o-mini"), structured_output=True)

If a model still returns malformed JSON despite the schema, parsing falls back to the same regex parser used when structured_output is off, rather than raising. reasoning and structured_output can't both be enabled at once: reasoning needs free text ending in a final-answer marker, while strict JSON-schema mode needs the entire completion to be the JSON payload, leaving no room for reasoning text.

Multi-criteria scoring

PointwiseRanker can score named sub-criteria separately instead of one holistic judgment, then combine them — useful for compositional queries ("family friendly, near historical sites, affordable") where you want to know why a candidate scored the way it did, or want explicit control over which constraint matters most, rather than leaving that blend to whatever the model implicitly does with a single score. Pass a criteria dict or "auto":

# weighted sum: you name the criteria and their relative weight
# (weights don't need to sum to 1, they're normalized internally)
ranker = PointwiseRanker(
    LLMConfig(model="gpt-4o-mini"),
    criteria={"price_fit": 0.5, "location_fit": 0.3, "family_friendly": 0.2},
)

# priority-hierarchical: "high" mathematically dominates any possible
# combination of "medium"/"low", so a candidate can't compensate for
# failing a high-priority criterion by scoring well on lower ones
ranker = PointwiseRanker(
    LLMConfig(model="gpt-4o-mini"),
    criteria={"family_friendly": "high", "price_fit": "medium", "location_fit": "low"},
)

# auto: the model extracts the criteria from the query itself, combined
# with equal weight, so there are no criteria names to maintain per
# domain, at the cost of not choosing them yourself
ranker = PointwiseRanker(LLMConfig(model="gpt-4o-mini"), criteria="auto")

Off by default (criteria=None), identical behavior to plain holistic scoring. Costs exactly the same as holistic scoring — one call per candidate either way, since every named criterion is scored together in a single response — except "auto" mode, which adds exactly one extra call per rank() (not per candidate) to extract the criteria first; if extraction produces nothing parseable, it falls back to holistic scoring for that call rather than raising. rank()'s output candidates carry the breakdown in Candidate.metadata["criteria_scores"] (merged with any metadata already on the input candidate), so you can see per-criterion scores, not just the combined one; score() keeps returning a plain float and re-extracts on every call in "auto" mode, so prefer rank() when scoring multiple candidates against the same query that way.

Composes normally with reasoning and num_samples; the existing reasoning+structured_output restriction still applies, inherited rather than a new rule.

Cascading (cheap-then-expensive)

CascadeRanker composes two already-configured rankers instead of being a new ranking algorithm itself: a cheap one narrows a long candidate list down, then an expensive one carefully re-ranks just the survivors (FrugalGPT-style cascading). Each stage keeps its own model and reasoning/num_samples/structured_output settings, exactly as if it were used standalone; CascadeRanker only owns how many survive the first stage:

from llmranker import CascadeRanker, LLMConfig, PointwiseRanker, SetwiseRanker

ranker = CascadeRanker(
    narrow=PointwiseRanker(LLMConfig(model="gpt-4o-mini")),
    refine=SetwiseRanker(LLMConfig(model="gpt-4o"), num_child=4),
    narrow_to=10,
)
result = ranker.rank(query="my search query", candidates=candidates)

ranker.total_calls / total_prompt_tokens / total_completion_tokens sum both stages, and ranker.config reports the refine stage's config (whichever model actually produced the final ranking) — it plugs into compare_rankers (see Evaluation & benchmarking) just like any other ranker.

Use case: hotel recommendation

The flagship example lives in examples/hotel_recommendation/. It reranks 7 hotels against natural-language guest preferences like "family friendly hotel with kids, close to historical places, not right on the beach." This is exactly the kind of compositional, subjective query that trips up keyword and embedding search but an LLM reading full descriptions handles naturally.

cd examples/hotel_recommendation
python run.py

It runs all five strategies against the same query and candidates and prints a side-by-side comparison of ranking quality, LLM calls, tokens, estimated cost, and latency using llmranker.compare_rankers.

More use cases

• examples/rag_document_reranking.py: rerank RAG retrieval results before they go into a prompt, so context budget goes to passages that actually answer the question instead of merely-related near-duplicates.
• examples/product_search_reranking.py: e-commerce search reranking against multi-constraint natural-language intent (price, fit, use case).
• Other good fits: job/candidate matching, content and media recommendation, support ticket triage, lead scoring: anywhere you have a short list of candidates and a query or profile to rank them against.

Customizing for your domain

Every ranker accepts an item_label (used in the default prompts, e.g. "hotel", "product", "document", ...) and an optional system_prompt override if you want full control over the wording:

ranker = SetwiseRanker(
    LLMConfig(model="gpt-4o-mini"),
    item_label="job candidate",
    system_prompt="You are a hiring assistant ranking candidates against a job description...",
)

Evaluation & benchmarking

from llmranker import RankingMetrics, compare_rankers

metrics = RankingMetrics()
metrics.get_metrics(true_ranking=["b", "a", "c"], predicted_ranking=["a", "b", "c"])
# {'ndcg': ..., 'mrr': ..., 'mae': ..., 'spearman': ..., 'kendall_tau': ...}

report = compare_rankers([ranker_a, ranker_b], query, candidates, true_ranking)
# pandas DataFrame: ranking quality + LLM calls/tokens/cost/latency, side by side

true_ranking is a ground-truth ordering of candidate ids (best to worst), if you have one, e.g. from human labels or a held-out click log.

API reference

Module	Contents
llmranker.types	Candidate(id, text, score, metadata), Ranker (structural protocol)
llmranker.llm	LLMConfig, call_llm, truncate_to_tokens, estimate_cost
llmranker.rankers	PointwiseRanker, PairwiseRanker, SetwiseRanker, ListwiseRanker, TourRankRanker, CascadeRanker; each takes reasoning, num_samples, structured_output
llmranker.metrics	RankingMetrics (NDCG, MRR, MAE, Spearman, Kendall's Tau)
llmranker.benchmark	compare_rankers
llmranker.prompts	Default prompt templates, plus extract_final_answer/reasoning_suffix for the reasoning flag
llmranker.structured	JSON-schema builders/parsers backing structured_output
llmranker.criteria	resolve_weights and text parsers backing PointwiseRanker's criteria param

Every ranker implements rank(query, candidates) -> list[Candidate] and tracks total_calls / total_prompt_tokens / total_completion_tokens after each call.

Contributing

Issues and PRs welcome. Run tests with:

pip install -e ".[dev]"
pytest

Tests run entirely offline against a fake LiteLLM backend, so no API key is needed to contribute.

See ROADMAP.md for what's researched but not built yet, and why.

Citing this package

If llmranker itself is useful to you, please cite the repository:

@misc{Rajaa_llmranker,
author = {Rajaa, Shangeth},
title = {{llmranker: LLM-based ranking and reasoning algorithms for search and recommendation}},
url = {https://github.com/shangeth/llmranker}
}

For a citation pinned to the exact version/commit you used, use GitHub's "Cite this repository" button in the sidebar (APA or BibTeX) instead of the snippet above: it reads CITATION.cff live off whatever's checked out, so it's always accurate without anyone needing to hand-update a version number in this README.

Citing the underlying research

If you use one of the ranking strategies implemented here, please also cite the paper(s) behind it:

@article{zhuang2023setwise,
  title={A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models},
  author={Zhuang, Shengyao and Zhuang, Honglei and Koopman, Bevan and Zuccon, Guido},
  journal={arXiv preprint arXiv:2310.09497},
  year={2023}
}

@inproceedings{podolak2025setwiseinsertion,
  title={Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency with Setwise Insertion},
  author={Podolak, Jakub and Peri{\'c}, Leon and Jani{\'c}ijevi{\'c}, Mina and Petcu, Roxana},
  booktitle={Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year={2025}
}

@inproceedings{chen2025tourrank,
  title={TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy},
  author={Chen, Yiqun and Liu, Qi and Zhang, Yi and Sun, Weiwei and Ma, Xinyu and Yang, Wei and Shi, Daiting and Mao, Jiaxin and Yin, Dawei},
  booktitle={Proceedings of the ACM Web Conference 2025},
  year={2025}
}

License

MIT, see LICENSE.