litmus-ai 0.2.1

A Python SDK for Hallucination Detection — powered by Litmus AI

Litmus Hallucination Detector

Litmus Hallucination Detector is an SDK for detecting hallucinations in Large Language Model (LLM) outputs. It uses geometric algebra on embeddings to verify outputs against a provided context.

Installation

pip install litmus-ai

Quick Start

1. Context Grounding (Type I - Default)

Checks if claims are supported by the provided context.

from litmus_ai import LitmusDetector

# Initialize grounding detector (default)
detector = LitmusDetector(type='context-grounding')

context = "The capital of France is Paris."
output = "Paris is the capital of France."

result = detector.check(context=context, output=output)
print(f"Outcome: {'Supported' if not result.is_hallucination else 'Hallucination'}")
print(f"Support Score: {result.score}")

Output:

Outcome: Supported
Support Score: 1.0

2. Context Contradiction (Type II)

Checks if claims actively contradict the provided context.

# Initialize contradiction detector
detector = LitmusDetector(type='context-contradiction')

context = ["Revenue increased 15%."]
output = ["Revenue decreased 15%."]

result = detector.check(context=context, output=output)
print(f"Is Hallucination: {result.is_hallucination}")
print(f"Contradiction Score: {result.score}")

Output:

Is Hallucination: True
Contradiction Score: 0.9998

3. Relational Inversion (Type III)

Detects if the logical relationship between entities is flipped or if there's a direct contradiction/negation.

# Initialize relational inversion detector
detector = LitmusDetector(type='relational-inversion')

context = ["Alice hired Bob."]
output = ["Bob hired Alice."] # Relational Inversion

result = detector.check(context=context, output=output)
print(f"Status: {result.details['claim_scores'][0]['type']}")

Output:

Status: Relational Inversion

4. Instruction Drift (Type IV)

Detects when an answer drifts away from the query-context relationship, even if it uses similar keywords. Requires a query parameter.

# Initialize instruction drift detector
detector = LitmusDetector(type='instruction-drift')

query = "Why did revenue decline?"
context = "Revenue declined due to supply chain issues."
output = "Revenue declined amid a challenging environment."

result = detector.check(query=query, context=context, output=output)
print(f"Drift Score: {result.score:.4f}")
print(f"Is Hallucination: {result.is_hallucination}")

# Override threshold at runtime (must be 0.0 to 1.0)
result = detector.check(query=query, context=context, output=output, threshold=0.5)

Output:

Drift Score: 0.45
Is Hallucination: True

Threshold Override

Every detector.check() call supports an optional threshold parameter to override the default for that call:

# Uses default threshold
result = detector.check(context=context, output=output)

# Override threshold (0.0 to 1.0 inclusive)
result = detector.check(context=context, output=output, threshold=0.9)

# Invalid — raises ValueError
result = detector.check(context=context, output=output, threshold=1.5)

Detection Types

Type Value	Classification	Model	Default Threshold	Description
context-grounding	Type I	all-MiniLM-L6-v2	0.7	Checks if claims are supported by evidence.
context-contradiction	Type II	nli-deberta-v3-base	0.3	Checks if claims contradict evidence.
relational-inversion	Type III	nli-deberta-v3-base	0.5	Checks for entity inversions or contradictions.
instruction-drift	Type IV	all-MiniLM-L6-v2	0.3	Checks if answers drift from query-context alignment.

How it works

Litmus uses mathematical projections to verify claims:

• Type I projects embeddings onto the subspace spanned by context facts.
• Type II projects cross-encoded pairs onto a contradiction discriminant.
• Type III compares projection magnitudes on specialized Inversion vs. Contradiction subspaces.
• Type IV compares bivector plane alignment between (Query ∧ Context) and (Query ∧ Answer).

License

This project is licensed under the MIT License