Veralith

Hallucination diagnosis for RAG agents. For every (query, context, response) trace, get a typed failure diagnosis and a concrete fix — evaluated on Veralith's hosted API and streamed to your dashboard at app.veralithai.com, or run fully offline.

Python 3.10+ · hosted or offline evaluation · non-blocking, fail-safe SDK.

Installation

pip install veralith

Quick start

export VERALITH_API_KEY=vk_live_...   # app.veralithai.com → project → API keys

import veralith

def answer(query: str) -> str:
    chunks = my_retriever(query)
    response = my_generator(query, chunks)
    veralith.log(query=query, context=chunks, response=response)
    return response

log() enqueues the trace and returns immediately; evaluation runs on Veralith's servers.

API

veralith.log(query, context, response, latency_ms=None) -> None

Enqueue a trace for server-side evaluation. Non-blocking.

Requires VERALITH_API_KEY. If it is unset, log() is a no-op (warns once per process) so the same code is safe to run in tests and local dev.

@veralith.trace

Decorator alternative — capture (response, context) from the return value:

import veralith

@veralith.trace
def rag(query: str):
    chunks = my_retriever(query)
    response = my_generator(query, chunks)
    return response, chunks          # (response, context)

Callers of rag(query) receive just response: the decorator strips the tuple, logs the trace, and returns the response. The query is read from the first positional argument or a query= keyword. async functions are supported.

When returning a bare tuple is awkward, return a TraceReturn:

from veralith import trace, TraceReturn

@trace
def rag(user_question: str):
    ...
    return TraceReturn(response=answer, context=chunks)

LangChain adapter

Auto-trace RetrievalQA chains with no call-site changes:

import veralith.adapters.langchain as adapter
adapter.install()
# every RetrievalQA.invoke(...) now logs a trace

veralith.evaluate(query, context, response, persist=False) -> EvaluationResult

Run the full evaluation locally — no account, no traffic to Veralith. Uses your OPENAI_API_KEY. Intended for CI, prompt tuning, and air-gapped use.

result = veralith.evaluate(
    query="What is a P/E ratio?",
    context=["Price-to-earnings ratio is share price / earnings per share."],
    response="A P/E ratio is share price divided by earnings per share.",
    persist=False,
)
print(result.diagnosis.failure_cell.value)   # 'complete_grounded'

Use log() in production and evaluate() in tests.

veralith.shutdown(wait=True) -> None

Flush and stop the background worker. Registered via atexit, so long-running apps rarely call it; use it in short scripts or tests to join pending traces cleanly.

Behavior

The SDK is designed to sit in a hot request path without risk:

• Non-blocking — log() / @trace enqueue onto an in-process ThreadPoolExecutor (VERALITH_WORKER_CONCURRENCY, default 4) and return immediately.
• Fail-safe — a Veralith outage, network error, or malformed response is swallowed and warned, never raised into your call path.
• No-op without a key — with VERALITH_API_KEY unset the SDK does nothing (warns once). No feature flags, nothing to strip before prod.
• Backpressure — if the worker pool is saturated the trace is dropped (warned once) rather than blocking your app.
• Clean shutdown — an atexit hook flushes queued traces on exit.

Set VERALITH_DEFAULT_SYNC=1 to send synchronously (blocking) instead — useful in serverless runtimes where background threads may not flush before freeze.

Failure cells

Each evaluated trace lands in one cell of Completeness × Faithfulness. Cell names read <completeness>_<faithfulness>:

complete_grounded is healthy; incomplete_ungrounded (missed part of the query and invented a claim) is the worst case. Each cell maps to a concrete suggestion — lower temperature, raise retrieval-K, tighten the generator prompt, fix a chunk boundary, etc. A per-trace sufficiency level (HIGH/LOW) is calibrated per knowledge base from the distribution of healthy traces.

Self-heal (via MCP, not this package)

Diagnosis is where this SDK stops — the fix loop lives on the platform. When failing traces cluster into a recurring pattern, Veralith opens a heal card. Point a coding agent (Claude Code, Codex, or Cursor) at Veralith's MCP server and it reads the diagnosis plus your actual RAG code and opens a pull request:

claude mcp add --transport http veralith \
  https://api.veralithai.com/mcp/http \
  --header "Authorization: Bearer vk_live_..."

This runs through your agent over MCP; the veralith pip package itself only handles instrumentation and evaluation. See docs.veralithai.com for the full loop.

The result object

evaluate() returns a typed EvaluationResult (all Pydantic models):

class EvaluationResult:
    trace_id: int
    query: str
    sub_questions: list[SubQuestion]         # decomposed query
    claims: list[Claim]                      # decomposed response
    sufficiency: list[SufficiencyJudgment]   # per sub-question
    faithfulness: list[FaithfulnessJudgment] # per claim (+ grounding chunks)
    completeness: CompletenessJudgment | None
    diagnosis: Diagnosis | None              # failure_cell + sufficiency level + counts
    suggestion: Suggestion                   # title + body + steps
    latency_ms: dict[str, float]             # per-phase wall-clock timing
    errors: dict[str, str]                   # per-metric failures, if any
    created_at: datetime

Configuration

Hosted log() evaluation runs on Veralith's own model keys and counts against your project's monthly trace quota. Offline evaluate() runs on your own OPENAI_API_KEY.

Links

• Dashboard — https://app.veralithai.com
• Docs — https://docs.veralithai.com
• Source — https://github.com/SrijanShekhar21/VeralithAI
• Issues — https://github.com/SrijanShekhar21/VeralithAI/issues

License

MIT — see LICENSE.