veralith 0.1.0

Hallucination diagnosis for RAG systems — Sufficiency, Faithfulness, Completeness verdicts plus rule-based remediation.

Veralith

Hallucination diagnosis for RAG systems. Wrap one line around your retrieval-augmented pipeline and get structured reports on what failed and how to fix it — not just a single yes/no hallucination flag.

Veralith decomposes every (query, context, response) trace into atomic sub-questions and claims, runs three LLM-as-judge metrics over them (Sufficiency, Faithfulness, Completeness), and classifies the trace into one of six diagnostic cells with a concrete remediation suggestion.

Status: alpha (0.1.x). Public API is stable; expect additions, not breaking changes.

---

Why Veralith

A monolithic "is this response hallucinated?" judge is a smoke alarm — it can tell you something is wrong but not what or where. Veralith is a diagnostic dashboard:

• Sufficiency — was the retrieval adequate for each part of the query?
• Faithfulness — is each claim in the response grounded in the retrieved context?
• Completeness — does the response actually answer every part of the query?

Cross-tabulating these gives you a named failure mode (retrieval gap, intrinsic hallucination, padded answer, etc.) plus actionable fixes (lower temperature, bump retrieval-K, tighten generator prompt, ...) for every trace.

---

Install

pip install veralith

Optional extras:

pip install "veralith[langchain]"   # LangChain auto-tracing adapter
pip install "veralith[dev]"          # pytest, ruff, build, twine (contributors)

Set your OpenAI key:

export OPENAI_API_KEY=sk-...

---

30-second quickstart

import veralith

result = veralith.evaluate(
    query="What is a P/E ratio and what was Apple's P/E in 2023?",
    context=[
        "The price-to-earnings (P/E) ratio is computed by dividing a company's "
        "share price by its earnings per share."
    ],
    response=(
        "A P/E ratio divides share price by earnings per share. "
        "Apple's P/E in 2023 was 42.7."
    ),
    persist=False,
)

print(result.diagnosis.failure_cell.value)   # 'b_ungrounded'
print(result.suggestion.title)               # 'Worst-case failure'
for action in result.suggestion.actions:
    print(" -", action)

You get back a typed EvaluationResult with per-claim verdicts, per-Qi sufficiency, a failure-cell diagnosis, and a concrete suggestion. Optionally persisted to a local SQLite database for later analysis.

---

Integration patterns

1. Explicit one-liner — works with any RAG stack

import veralith

def answer(query: str) -> str:
    chunks = my_retriever(query)
    response = my_generator(query, chunks)

    veralith.log(query=query, context=chunks, response=response)   # background eval
    return response

2. Decorator — zero code reshape

import veralith

@veralith.trace
def my_rag(query: str):
    chunks = my_retriever(query)
    response = my_generator(query, chunks)
    return response, chunks   # the decorator captures (response, context)

3. Synchronous eval — full result inline

result = veralith.evaluate(query, context, response, persist=False)
if result.diagnosis and result.diagnosis.failure_cell.value.endswith("ungrounded"):
    handle_hallucination(result.faithfulness)

4. LangChain — zero-code auto-tracing

import veralith.adapters.langchain as adapter
adapter.install()

# every RetrievalQA.invoke() now auto-traces to Veralith

---

What Veralith detects

Each evaluated trace lands in one of six cells from the cross-tab of Completeness × Faithfulness:

	Grounded (F=✓)	Ungrounded (F=✗)
Complete (a)	a_grounded — healthy	a_ungrounded — intrinsic hallucination
Incomplete (b)	b_grounded — retrieval gap or generator-skipped	b_ungrounded — worst case
Extra (c)	c_grounded — padded answer	c_ungrounded — extrinsic hallucination

Plus a per-trace Sufficiency level (HIGH/LOW), learned per knowledge base from the distribution of healthy traces. Together they drive a rule-based suggester that maps every diagnosis to a concrete remediation (lower temperature / bump K / tighten generator prompt / etc.).

---

Configuration

Defaults work out of the box. Tunable via environment variables or veralith.config.settings:

Variable	Default	Purpose
OPENAI_API_KEY	—	Required
VERALITH_JUDGE_MODEL	gpt-4o	Model for S/F/C judges
VERALITH_DECOMPOSER_MODEL	gpt-4o-mini	Model for query / response decomposition
VERALITH_DB_PATH	veralith.db	SQLite persistence path

Each evaluation costs roughly 5 LLM calls (3 batched judges + 2 decomposition) — about $0.005 per trace on the default models. Cost is tracked per call via veralith.observability.cost.

---

The result object

class EvaluationResult:
    trace_id: int
    query: str
    sub_questions: list[SubQuestion]           # decomposed Q
    claims: list[Claim]                         # decomposed R
    sufficiency: list[SufficiencyJudgment]      # per-Qi verdicts
    faithfulness: list[FaithfulnessJudgment]    # per-Ri verdicts + grounding chunks
    completeness: CompletenessJudgment | None   # Ri ↔ Qi alignment
    diagnosis: Diagnosis | None                 # failure_cell + sufficiency level + counts
    suggestion: Suggestion                      # title + body + actionable steps
    created_at: datetime
    errors: dict[str, str]                       # any per-metric failures (D3)
    latency_ms: dict[str, float]                 # per-phase wall-clock timing

Every field is a typed Pydantic model.

---

Roadmap

What's in 0.1:

• Three judges (Sufficiency, Faithfulness, Completeness) with batched LLM calls.
• Diagnostic classifier and rule-based suggester.
• Outcome-based threshold calibration per knowledge base.
• SDK: log(), @trace, LangChain adapter, background eval worker.
• SQLite persistence with self-healing migrations.
• Cost tracking with per-trace budget guard.
• CLI entry point.

On the roadmap:

• LLM-enriched trace-specific suggestions (Suggestion.detailed_body).
• Cross-trace pattern detection ("you keep hallucinating on time-sensitive queries").
• Additional judges (reasoning validity, temporal validity).
• More framework adapters (LlamaIndex, raw OpenAI tools).
• Hosted dashboard with multi-tenant projects.

---

Authors

Srijan Shekhar and Kaustav Das Gupta.

License

MIT — see LICENSE.

Links

• Source: https://github.com/SrijanShekhar21/VeralithAI
• Issues: https://github.com/SrijanShekhar21/VeralithAI/issues