auditkit 1.0.0

Evaluate any model on any dataset and any task.
One library for benchmark, judge, code, red-team, security, and performance
evaluation — zero required deps.

Documentation (internal) · Quickstart · Examples · Contributing

TL;DR

Why AuditKIT

Evaluating AI models is fragmented. Academic benchmarks (MMLU, GSM8K) use one tool. LLM-as-judge evaluations use another. Red-teaming and performance profiling each have their own frameworks. There is no single library that does all of them with a consistent API, zero required dependencies, and a provenance-first data model.

AuditKIT is that library. It provides a unified evaluation spine that supports every technique, so you can compare results across benchmarks, judge evaluations, red-team probes, and performance profiles — all from a single ak.evaluate() call.

Key features

• Metric families. Benchmark (exact match, F1, BLEU, ROUGE, ChrF), LLM-as-judge (GEval, rubric items), code/deterministic (contains, regex, JSON validation), RAG (lexical groundedness, context overlap/coverage), embedding similarity, hallucination detection, toxicity/bias, pairwise/preference (win rate, Elo, preference accuracy), security (DEFCON grade), performance (latency, throughput).
• Zero required deps. Core runs on the Python standard library alone. Heavy backends (BERTScore, vLLM, LiteLLM, OpenAI, Anthropic, HuggingFace) are optional extras.
• Any model backend. echo for testing, openai:, anthropic:, hf:, lexsi:, vllm:, litellm: (which also reaches Ollama, e.g. litellm:ollama/llama3.1), api:, groq:, openrouter:, or any callable. Auto-resolved via AutoModel.resolve().
• Many datasets, one model. evaluate_many() runs one model across several datasets in a single call, returning one RunResult per dataset.
• Red teaming. Built-in adversarial probes (prompt injection, jailbreak, encoding, over-refusal) and detectors (keyword, refusal, injection success, system prompt leak) via RedTeamRunner.
• Experiment tracking. Named experiments with ExperimentDB, MLflow logging, cross-run comparison with bootstrap significance tests.
• Model comparison. compare_models() runs the same dataset against multiple models and produces side-by-side results with pairwise significance.
• CLI with YAML config. auditkit eval, init, list, redteam, compare subcommands. Define evaluations in YAML files with prompts:, model config, tags, and split strategies.
• Provenance by default. Every run writes a RunResult with a stable fingerprint (sha256 over model+seed+tasks+config). Results are cacheable and comparable by fingerprint.

How it works

flowchart LR
    A["Dataset<br/>Samples · Scenario · CSV"] --> B["Adapter<br/>generation · chat · instruction"]
    B --> C["Model<br/>echo · openai · hf · vllm"]
    C --> D{"Metrics"}
    D -->|per sample| E["Score + Prediction"]
    E --> F["Aggregate<br/>Stats · Headline"]
    F --> G["RunResult<br/>fingerprint · cache"]

Install

pip install auditkit                     # core (zero deps)
pip install "auditkit[openai]"           # OpenAI backend
pip install "auditkit[all]"              # all backends + heavy metrics

Requires Python 3.10+ on Linux, macOS, or Windows.

pip install "auditkit[anthropic]"            # Anthropic backend
pip install "auditkit[litellm]"              # LiteLLM (Ollama, etc.)
pip install "auditkit[vllm]"                 # vLLM backend
pip install "auditkit[transformers]"         # HuggingFace + hallucination + embedding similarity
pip install "auditkit[bert-score]"           # BERTScore
pip install "auditkit[mlflow]"               # MLflow experiment tracking

# From source
pip install "auditkit @ git+https://github.com/Lexsi-Labs/AuditKIT.git"

Quickstart

import auditkit as ak

samples = [
    ak.Sample(input="What is 2+2?", target="4"),
    ak.Sample(input="What is 3+3?", target="6"),
]

result = ak.evaluate(samples, model=lambda prompts: prompts)
print(result.summary())

From the CLI:

auditkit eval --model hf:gpt2 <<< "What is 2+2?"

With a YAML config:

model: openai:gpt-4o-mini
temperature: 0.0
prompts:
  - "What is the capital of France?"
output: results.json

auditkit eval --config auditkit.yaml

What it evaluates

Output

Every evaluate() returns a RunResult:

Examples

Colab-ready notebooks with real models and real datasets — see examples/README.md for the full, current list.

Applications

Real-world scenarios answered end to end, not feature tours — see examples/applications/README.md. Only 01 uses the lm-evaluation-harness integration (ak.run_lmeval(), needs auditkit[lmeval]), as an authoritative cross-check alongside the native evaluation; none of the others do.

Repository map

Developed by Lexsi Labs

Created by the team at Lexsi Labs, AuditKIT provides a unified evaluation spine for AI models across every technique and modality.

License

This project is released under the Lexsi Labs Source Available License (LSAL) v1.2 — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.

Join Community / Contribute

• Issues and discussions are welcomed on the GitHub issue tracker.
• See the Contributing section for contribution standards, code reviews, and documentation tips.