pip install auditlm
agentaudit init
agentaudit dashboard

---

Why AgentAudit?

32% of AI teams cite quality as their #1 barrier to production. Most teams ship prompts blindly or debug with print statements.

AgentAudit gives you a proper evaluation layer:

• Hallucination detection — did the model make up something not in the source?
• Faithfulness scoring — did the model contradict the retrieved context?
• Cost tracking — how much does each query cost in USD and INR?
• Latency logging — how slow is each step?
• Regression testing — did my last prompt change make things better or worse?
• Agent trace viewer — which tools did the agent call, in what order?

---

Quick Start

1. Install

pip install auditlm

2. Setup (one time)

agentaudit init

Walks you through picking a provider (Gemini or OpenAI), pasting your API key, and choosing a judge model. Config saved locally at ~/.agentaudit/.env.

3. Add to your pipeline

Before:

def answer_question(query):
    docs = retriever.search(query)
    return llm.generate(query, docs)

After — 3 lines added:

from agentaudit import audit, set_context          # line 1

@audit(name="my_pipeline")                          # line 2
def answer_question(query):
    docs = retriever.search(query)
    set_context(docs)                                # line 3
    return llm.generate(query, docs)

Your code works exactly the same. AgentAudit silently evaluates every run.

4. See results

agentaudit dashboard

Opens a local dashboard at localhost:8501 with four pages: Overview · Run Details · Trends · Regression

---

Zero-code starter

agentaudit example

Creates my_pipeline.py with clear TODO markers. Replace two lines, run it, done.

---

CLI

Command	What it does
agentaudit init	First-time setup
agentaudit dashboard	Open Streamlit dashboard
agentaudit compare --pipeline NAME	Compare last two runs
agentaudit config --show	Show current config
agentaudit config --reset	Update API key or model
agentaudit example	Generate starter file

---

How it works

AgentAudit uses LLM-as-judge — sends your pipeline's output and context to a judge model (Gemini Flash by default) and asks:

"Extract every claim. For each, is it supported by the context?"

The judge returns claims with verdicts (SUPPORTED / UNSUPPORTED / CONTRADICTED) and AgentAudit calculates:

hallucination_score = unsupported claims / total claims

Everything saved to local SQLite. Nothing leaves your machine.

---

Exact cost tracking (optional)

By default, AgentAudit estimates tokens from text length. For exact cost, add one line:

from agentaudit import set_tokens

set_tokens(
    input=response.usage.prompt_tokens,
    output=response.usage.completion_tokens,
)

Dashboard labels costs as estimated or exact.

---

Agent tracing

from agentaudit import AgentTracer

with AgentTracer("my_agent") as tracer:
    result = search_tool(query)
    tracer.log_tool_call("search", input=query, output=result)
    tracer.log_response(answer)

---

Supported judge models

Google Gemini (free tier available): gemini-flash-latest · gemini-3.5-flash · gemini-3-flash · gemini-3.1-pro

OpenAI: gpt-4o-mini · gpt-4o

Custom models: add_model_pricing("model", input_per_1k, output_per_1k)

---

Tech stack

Python · SQLite · Streamlit · LLM-as-judge · pyproject.toml

---

Contributing

PRs welcome. Open an issue for bugs or feature requests.

License

MIT