rag-consistency-scorer

MMD-based semantic consistency scoring for RAG pipeline outputs.

Measures whether a RAG pipeline produces semantically stable answers when asked the same question multiple times. Uses Maximum Mean Discrepancy (MMD) over atomic-fact embeddings — a statistically principled distributional test — rather than naive string similarity.

---

Installation

pip install rag-consistency-scorer

Optional extras:

pip install "rag-consistency-scorer[pca]"   # enables PCA dimensionality reduction
pip install "rag-consistency-scorer[dev]"   # testing dependencies

---

Quick start

from rag_consistency_scorer import ConsistencyScorer

data = [
    {
        "query": "What is the boiling point of water?",
        "responses": [
            "Water boils at 100 degrees Celsius at sea level.",
            "The boiling point of water is 100°C at standard pressure.",
            "At 1 atm pressure, water reaches its boiling point at 100°C.",
            "Water transitions to steam at 100 degrees Celsius under normal conditions.",
            "The boiling point of water is 100°C, equivalent to 212°F.",
            "Under standard atmospheric conditions, water boils at 100°C.",
            "Water boils at 373 Kelvin, i.e. 100°C at sea level.",
            "Standard boiling point of water is 100°C at atmospheric pressure.",
        ],
    }
]

scorer = ConsistencyScorer()
results = scorer.score(data)

for r in results:
    print(r.query_id, r.stability_score)
    # → q_abc12345  0.921

No API key needed with default settings. The package uses a lightweight TF-IDF embedding fallback when no embedding_config is provided.

---

Input format

scorer.score() accepts a list of dicts, each with:

Key	Type	Required	Description
query	str	✓	The prompt / question sent to the RAG pipeline
responses	List[str]	✓	One answer string per run. Minimum 2, recommended 8–15.
query_id	str	–	Optional identifier. Auto-generated if omitted.

data = [
    {
        "query": "Who invented the telephone?",
        "query_id": "telephone_q1",
        "responses": [
            "Alexander Graham Bell invented the telephone in 1876.",
            # ... at least 7 more responses
        ]
    }
]

---

Output format

Each call to .score() returns a List[QueryScore] — one per input query.

result = results[0]

result.stability_score   # float in [0, 1]. 1 = perfectly stable.
result.mmd2_mean         # mean pairwise MMD² across all run pairs
result.mmd2_min          # min pairwise MMD²
result.mmd2_max          # max pairwise MMD²
result.sigma_used        # kernel bandwidth selected for this query
result.tau_used          # τ used in exp(-MMD²/τ) mapping
result.n_runs            # number of responses scored
result.query_id          # identifier
result.query             # original query string

# Per-pair statistical diagnostics
for pair in result.pairs:
    pair.run_i, pair.run_j   # which two runs
    pair.mmd2                # raw MMD² (may be slightly negative)
    pair.p_value             # permutation p-value (Phipson-Smyth corrected)
    pair.p_value_bh          # BH-FDR adjusted p-value
    pair.significant         # True if p_value_bh ≤ alpha

# Per-run outlier diagnostics (sorted most → least anomalous)
for run in result.runs:
    run.run_index         # index into original responses list
    run.outlier_score     # mean MMD² to all other runs (higher = more anomalous)
    run.z_score           # bootstrap z-score (> 2 is suspicious)
    run.n_facts           # number of atomic facts extracted
    run.extraction_source # "llm" or "heuristic"

# Serialise
import json
json.dumps(result.to_dict())

Interpreting stability_score

Score	Interpretation
≥ 0.80	Stable — answers are semantically consistent
0.60–0.79	Watch — mild drift; monitor over time
0.40–0.59	Investigate — notable inconsistency
< 0.40	Unstable — likely retrieval or generation failure

Thresholds depend on your tau setting. Run the calibration utilities to set domain-specific thresholds.

---

Configuration

from rag_consistency_scorer import ConsistencyScorer, ScorerConfig

scorer = ConsistencyScorer(
    config=ScorerConfig(
        # --- Extraction ---
        extraction_mode="heuristic",   # or "llm"
        llm_config=None,               # required if extraction_mode="llm"

        # --- Embedding ---
        embedding_config=None,         # uses TF-IDF fallback if not set

        # --- Statistical ---
        tau=None,           # None = auto-calibrate from data (recommended)
        n_permutations=500, # higher → more accurate p-values, slower
        alpha=0.05,         # BH-FDR significance level
        bandwidth_candidates=10,
        seed=42,
    )
)

LLM-backed extraction (OpenAI)

scorer = ConsistencyScorer(
    config=ScorerConfig(
        extraction_mode="llm",
        llm_config={
            "api_key": "sk-...",
            "model": "gpt-4o-mini",
            "max_facts": 40,
            "heuristic_fallback": True,   # fall back silently on LLM error
        },
        embedding_config={
            "api_key": "sk-...",
            "model": "text-embedding-3-small",
        },
    )
)

Azure OpenAI

scorer = ConsistencyScorer(
    config=ScorerConfig(
        extraction_mode="llm",
        llm_config={
            "api_key": "your-azure-key",
            "model": "gpt-4o-mini",                         # deployment name
            "api_base": "https://your-resource.openai.azure.com/",
            "api_version": "2024-02-01",
        },
        embedding_config={
            "api_key": "your-azure-key",
            "model": "text-embedding-ada-002",              # deployment name
            "api_base": "https://your-resource.openai.azure.com/",
            "api_version": "2024-02-01",
        },
    )
)

---

How it works

For each query the scorer:

1. Extracts atomic facts from each response (heuristic regex or LLM-based).
2. Embeds each fact into a vector space and L2-normalises.
3. Selects σ (kernel bandwidth) via power-guided selection on a log-uniform grid, replacing the naive median heuristic.
4. Computes pairwise MMD² (unbiased U-statistic) between every pair of runs.
5. Runs permutation tests with Phipson-Smyth corrected p-values.
6. Applies BH-FDR correction across all pairs.
7. Detects outlier runs via bootstrap z-scores.
8. Maps to [0,1]: stability = exp(-max(0, MMD²_mean) / τ), with τ auto-calibrated from the data.

---

Running tests

pip install pytest
pytest tests/ -v

---

What this does NOT measure

• Correctness — a pipeline can be stably wrong.
• Faithfulness to retrieved context.
• Logical contradiction detection.

Use alongside correctness/faithfulness metrics (RAGAS, Giskard, etc.) for a complete RAG evaluation suite.

---

License

MIT