sfs-mcp-toolkit 0.1.0

Spatial Faithfulness Score (SFS) — a domain-agnostic MCP toolkit for measuring context faithfulness in RAG systems

SFS MCP Toolkit

Spatial Faithfulness Score (SFS) — a domain-agnostic metric and MCP server for measuring whether LLM-generated reasoning is grounded in retrieved context, not hallucinated.

The Problem

Retrieval-Augmented Generation (RAG) systems retrieve evidence and feed it to an LLM to produce grounded answers. But how do you measure whether the LLM actually used that evidence faithfully, or fabricated plausible-sounding numbers?

Existing faithfulness metrics (RAGAS, DeepEval, etc.) rely on an LLM-as-judge approach — asking another LLM whether the output is faithful. This creates a circularity problem: you're using the same class of system to audit itself.

The Solution: SFS

SFS takes a deterministic, evidence-grounded approach:

SFS = verified_claims / total_claims

1. Extract — Scan the LLM output for numerical assertions (prices, percentages, distances, counts, dosages, ratios, etc.)
2. Verify — For each claim, check whether the stated value falls within a configurable tolerance of any value in the evidence pool
3. Score — The ratio of verified claims to total claims gives a 0–1 faithfulness score

Why This Works for Any RAG System

The key insight is that any RAG system that produces numerical claims can be audited this way, regardless of domain:

Domain	Claim Types	Evidence Source
Property Valuation	Prices, gradients, distances	Comparable sales DB, spatial statistics
Medical RAG	Dosages, lab values, durations	Clinical databases, drug references
Legal RAG	Monetary amounts, durations, counts	Case law, statutory databases
Financial Analysis	Prices, returns, ratios	Market data, financial statements

The SFS formula remains identical. Only the extraction patterns and tolerance thresholds change per domain — and these are fully configurable via DomainConfig.

Advantages Over LLM-as-Judge

Property	LLM-as-Judge	SFS
Deterministic	No — varies across runs	Yes — same input = same score
Auditable	Opaque reasoning	Every claim traced to evidence
Cost	Requires additional LLM calls	Zero LLM cost (regex + arithmetic)
Circular bias	Yes — LLM evaluating LLM	No — purely evidence-based
Reproducible	Low inter-run agreement	Perfect reproducibility

Validated Calibration

SFS was rigorously validated in the property valuation domain:

• Inter-rater reliability: Cohen's κ = 0.687 (substantial agreement) between automated SFS and expert human raters
• Rounding tolerance: 15% threshold correctly distinguishes acceptable rounding (68% vs 68.5%) from hallucination (50% vs 68.5%)
• Tested across 5 prompting strategies (zero-shot, CoT, S-CoT, few-shot, self-reflection) with consistent behaviour

Installation

pip install sfs-mcp-toolkit
# or from source:
cd sfs-mcp-toolkit && pip install -e .

MCP Server Setup

Claude Desktop / Claude Code

Add to your MCP configuration:

{
  "mcpServers": {
    "sfs": {
      "command": "sfs-mcp",
      "description": "SFS — RAG faithfulness metric"
    }
  }
}

Available Tools

Tool	Description
sfs_compute	Primary tool — end-to-end: extract claims + verify + score
sfs_extract_claims	Extract numerical claims from LLM output
sfs_verify_claims	Verify claims against an evidence pool
sfs_verify_single	Verify a single claim
sfs_list_presets	List available domain presets
sfs_get_preset	Get full config for a preset

Quick Start

End-to-End Scoring (via MCP tool)

Tool: sfs_compute
text: "The median house price in this suburb is $850,000, located 12.5 km
       from the CBD. With 190 comparable sales showing a 3.2% gradient
       north, the estimated value is $875,000."
evidence_json: [
    {"key": "median_price",    "value": 842000, "unit": "AUD"},
    {"key": "cbd_distance",    "value": 12.3,   "unit": "km"},
    {"key": "comparable_count","value": 188,     "unit": "count"},
    {"key": "north_gradient",  "value": 3.5,     "unit": "%"},
    {"key": "ml_prediction",   "value": 880000,  "unit": "AUD"}
]
preset: "property_valuation"

Output:

{
  "sfs_score": 1.0,
  "total_claims": 5,
  "verified_claims": 5,
  "failed_claims": 0,
  "type_breakdown": {
    "currency":   {"total": 2, "verified": 2, "failed": 0},
    "distance":   {"total": 1, "verified": 1, "failed": 0},
    "count":      {"total": 1, "verified": 1, "failed": 0},
    "percentage": {"total": 1, "verified": 1, "failed": 0}
  }
}

Python API (Direct Usage)

from sfs_toolkit.extractor import extract_claims
from sfs_toolkit.verifier import compute_sfs
from sfs_toolkit.models import EvidencePool, EvidenceItem
from sfs_toolkit.presets import get_preset

# Configure for your domain
config = get_preset("medical")

# Your LLM output
text = "The patient's HbA1c level of 7.2% indicates moderate control. \
        With 45 patients in the cohort showing similar patterns..."

# Your evidence pool (from RAG retrieval)
evidence = EvidencePool(items=[
    EvidenceItem(key="hba1c", value=7.1, unit="%", source="lab_results"),
    EvidenceItem(key="cohort_size", value=44, unit="count", source="study_db"),
])

# Extract and verify
claims = extract_claims(text, config=config)
report = compute_sfs(claims, evidence, config=config)

print(f"SFS: {report.sfs_score:.0%}")  # SFS: 100%
print(f"Verified: {report.verified_claims}/{report.total_claims}")

Custom Domain Configuration

from sfs_toolkit.models import DomainConfig, ExtractionPattern

config = DomainConfig(
    name="Climate Science RAG",
    tolerance=0.10,  # 10% tolerance
    type_tolerances={"temperature": 0.05},  # tighter for temperature
    extraction_patterns=[
        ExtractionPattern(
            claim_type="temperature",
            pattern=r'([\d.]+)\s*°[CF]',
            unit="°C",
            min_value=-100,
            max_value=200,
        ),
        ExtractionPattern(
            claim_type="concentration",
            pattern=r'([\d.]+)\s*ppm',
            unit="ppm",
            min_value=0.1,
        ),
    ],
)

Architecture

┌─────────────────────────────────────────────────────────┐
│                    MCP Client                           │
│  (Claude Desktop, Claude Code, any MCP-compatible app)  │
└──────────────────────┬──────────────────────────────────┘
                       │ MCP Protocol (stdio)
┌──────────────────────▼──────────────────────────────────┐
│                  SFS MCP Server                         │
│                                                         │
│  ┌─────────────┐  ┌──────────────┐  ┌───────────────┐  │
│  │  Extractor   │→│   Verifier    │→│  SFS Report    │  │
│  │ (regex-based)│  │ (tolerance-  │  │ (score +      │  │
│  │              │  │  based)      │  │  breakdown)   │  │
│  └─────────────┘  └──────────────┘  └───────────────┘  │
│         ↑                                               │
│  ┌─────────────┐                                        │
│  │   Presets    │  property_valuation | medical |        │
│  │             │  legal | financial | custom             │
│  └─────────────┘                                        │
└─────────────────────────────────────────────────────────┘

Domain Presets

Preset	Tolerance	Claim Types	Use Case
property_valuation	15%	currency, percentage, distance, count	Real estate RAG (GeoScribe)
medical	10% (5% for dosages)	dosage, lab_value, percentage, count, duration	Clinical decision support
legal	10% (5% for currency)	currency, percentage, duration, count	Legal research RAG
financial	10% (5% for ratios)	currency, percentage, ratio	Financial analysis

Thesis Framing

SFS was developed as part of the GeoScribe research project (FYP, IIT) for spatial property valuation. However, its design is intentionally domain-agnostic:

1. The metric itself (verified/total claims) is universal
2. The verification logic (relative error within tolerance) applies to any numerical assertion
3. The extraction layer is configurable via domain presets
4. The calibration methodology (inter-rater reliability) is reproducible in any domain

This positions SFS as a general contribution to RAG evaluation methodology, not just a property-valuation artifact. The GeoScribe implementation serves as a validated reference implementation, while the MCP toolkit enables adoption across domains.

License

MIT