hallx

Lightweight hallucination-risk scoring for production LLM pipelines

What Is Hallx

Hallx is a practical guardrail layer that evaluates LLM responses before they are trusted in downstream systems.

It scores responses using:

• schema validity
• consistency across repeated generations
• grounding against provided context

It returns:

• confidence
• risk_level
• issues
• recommendation

For agentic and retrieval flows, hallx adds three deeper guards:

• Claim-level grounding — splits the response into individual claims and verifies each one against its best evidence snippet, surfacing exactly which sentences are supported, weak, or unsupported.
• Pluggable verifiers — swap in local NLI (LocalNLIChecker, hallx[nli]) or an LLM-as-judge (GroundingJudge) to score claim–evidence entailment.
• Tool-call validation — catch hallucinated tool names and malformed arguments in agentic LLM output before a handler executes them.

Quick Start

pip install hallx

from hallx import Hallx

checker = Hallx(profile="balanced")
result = checker.check(prompt="p", response="r", context=["c"])
print(result.confidence, result.risk_level, result.recommendation)

Claim-Level Grounding

Detect which sentence in the response has evidence, and which is invented:

from hallx.attribution import check_claim_grounding

response = "The Eiffel Tower is in Paris. The Eiffel Tower is also in Berlin."
context = ["The Eiffel Tower is located in Paris, France."]

result = check_claim_grounding(response, context)

for claim in result.claims:
    if claim.status != "filtered":
        print(f"[{claim.status}] {claim.text}")

[supported] The Eiffel Tower is in Paris.
[unsupported] The Eiffel Tower is also in Berlin.

The result exposes score, supported_count, weak_count, unsupported_count, and per-claim similarity, evidence_index, and evidence_snippet. Use hallx.extract_claims alone to get the typed Claim spans.

Semantically grounded with a verifier

By default clashes are scored with fuzzy text similarity — fast and dependency-free. For real NLI, pass a FaithfulnessVerifier:

from hallx.faithfulness import LocalNLIChecker   # needs: pip install 'hallx[nli]'

verifier = LocalNLIChecker()
result = check_claim_grounding(response, context, verifier=verifier)

Or use any hallx LLM adapter as an LLM-as-judge (no extra dependency):

from hallx import OpenAIAdapter
from hallx.judge import GroundingJudge

judge = GroundingJudge(llm_adapter=OpenAIAdapter("gpt-4o-mini", api_key="..."))
result = check_claim_grounding(response, context, verifier=judge)

Claim checks inside the main check

Enable claim-level evidence on every Hallx.check call and the verdict lands on result.evidence:

from hallx import Hallx, OpenAIAdapter
from hallx.judge import GroundingJudge

checker = Hallx()
judge = GroundingJudge(llm_adapter=OpenAIAdapter("gpt-4o-mini", api_key="..."))

result = checker.check(
    prompt="Summarize the refund policy",
    response="Refunds are allowed within 30 days.",
    context=["Refunds are allowed within 30 days of purchase."],
    claims=True,
    verifier=judge,
)

print(result.claim_grounding_score)      # mean claim grounding
print(result.claims_supported)           # count of supported claims
print(result.unsupported_claims)         # the hallucinated bits, if any

Both check and check_async accept claims=True and verifier=.... Embedding-backed scoring works too: pass embedding_callable and optionally context_embeddings.

Tool-Call Validation

Guard agentic pipelines against hallucinated tool calls before execution:

from hallx import Hallx

tools = {
    "get_weather": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"],
        "additionalProperties": False,
    },
    "send_email": {"parameters": {"type": "object", "properties": {"to": {"type": "string"}}, "required": ["to"]}},
}

checker = Hallx()
agent_output = [
    {"function": {"name": "get_weather", "arguments": '{"city": "Paris"}'}},
    {"name": "rm -rf", "arguments": "{}"},          # hallucinated tool name
]

result = checker.check_tool_call(agent_output, tools)
print(result.score)                                  # 0.25
print(result.verdicts)                               # ok / unknown_tool
if result.recommendation["action"] == "block":
    print("Regenerate before invoking any tool.")

Accepts raw (name, arguments) pairs, dicts, OpenAI-style {function: ...} payloads, and typed ToolCall instances. The standalone check_tool_call / score_tool_calls helpers are exported too. Verdict statuses: ok, invalid_arguments, malformed, invalid_definition, unknown_tool.

Workflow

Essential Links

Languages

• English
• Tamil
• Malayalam
• Hindi
• Kannada
• German
• Japanese
• Chinese

Community

• Contributing
• GitHub Sponsors
• Buy Me a Coffee
• Monthly Downloads (PyPIStats)