auditkit 1.1.1

Evaluate any model on any dataset and any task.
One library for benchmark, judge, code, red-team, security, and performance
evaluation — zero required deps.

Documentation · Quickstart · Examples · Contributing

TL;DR

Why AuditKIT

Evaluating AI models is fragmented. Academic benchmarks (MMLU, GSM8K) use one tool. LLM-as-judge evaluations use another. Red-teaming and performance profiling each have their own frameworks. There is no single library that does all of them with a consistent API, zero required dependencies, and a provenance-first data model.

AuditKIT is that library. It provides a unified evaluation spine that supports every technique, so you can compare results across benchmarks, judge evaluations, red-team probes, and performance profiles — all from a single ak.evaluate() call.

Key features

• Metric families. Benchmark (exact match, F1, BLEU, ROUGE, ChrF), LLM-as-judge (GEval, rubric items), code/deterministic (contains, regex, JSON validation), RAG (lexical groundedness, context overlap/coverage), embedding similarity, hallucination detection, toxicity/bias, pairwise/preference (win rate, Elo, preference accuracy), security (DEFCON grade), performance (latency, throughput).
• Zero required deps. Core runs on the Python standard library alone. Heavy backends (BERTScore, vLLM, LiteLLM, OpenAI, Anthropic, HuggingFace) are optional extras.
• Any model backend. echo for testing, openai:, anthropic:, hf:, lexsi:, vllm:, litellm: (which also reaches Ollama, e.g. litellm:ollama/llama3.1), api:, groq:, openrouter:, agent: (a deployed agent over HTTP), or any callable. Auto-resolved via AutoModel.resolve().
• Many datasets, one model. evaluate_many() runs one model across several datasets in a single call, returning one RunResult per dataset.
• Red teaming. Built-in adversarial probes (prompt injection, jailbreak, encoding, over-refusal) and detectors (keyword, refusal, injection success, system prompt leak) via RedTeamRunner.
• Agent and RAG evaluation. Score tool calls, parallel/dependency behavior and retrieval on any model. The auditkit.agent_eval layer evaluates a whole task episode: a verified outcome (final state, artifact, answer, or a custom predicate) as the headline, with tool, trace and judge scores as diagnostics. It reads recorded runs (AgentTune, OpenAI messages, a Sample trace) or calls a deployed agent: endpoint, and works with any framework that emits OpenAI messages or serves over HTTP.
• Experiment tracking. Named experiments with ExperimentDB, MLflow logging, cross-run comparison with bootstrap significance tests.
• Model comparison. compare_models() runs the same dataset against multiple models and produces side-by-side results with pairwise significance.
• CLI with YAML config. auditkit eval, init, list, redteam, compare, agent subcommands. Define evaluations in YAML files with prompts:, model config, tags, and split strategies.
• Provenance by default. Every run writes a RunResult with a stable fingerprint (sha256 over model+seed+tasks+config). Results are cacheable and comparable by fingerprint.

How it works

flowchart LR
    A["Dataset<br/>Samples · Scenario · CSV"] --> B["Adapter<br/>generation · chat · instruction"]
    B --> C["Model<br/>echo · openai · hf · vllm"]
    C --> D{"Metrics"}
    D -->|per sample| E["Score + Prediction"]
    E --> F["Aggregate<br/>Stats · Headline"]
    F --> G["RunResult<br/>fingerprint · cache"]

Install

pip install auditkit                     # core (zero deps)
pip install "auditkit[openai]"           # OpenAI backend
pip install "auditkit[all]"              # all backends + heavy metrics

Requires Python 3.10+ on Linux, macOS, or Windows.

pip install "auditkit[anthropic]"            # Anthropic backend
pip install "auditkit[litellm]"              # LiteLLM (Ollama, etc.)
pip install "auditkit[vllm]"                 # vLLM backend
pip install "auditkit[transformers]"         # HuggingFace + hallucination + embedding similarity
pip install "auditkit[bert-score]"           # BERTScore
pip install "auditkit[mlflow]"               # MLflow experiment tracking

# From source
pip install "auditkit @ git+https://github.com/Lexsi-Labs/AuditKIT.git"

Quickstart

import auditkit as ak

samples = [
    ak.Sample(input="What is 2+2?", target="4"),
    ak.Sample(input="What is 3+3?", target="6"),
]

result = ak.evaluate(samples, model=lambda prompts: prompts)
print(result.summary())

From the CLI:

auditkit eval --model hf:gpt2 <<< "What is 2+2?"

With a YAML config:

model: openai:gpt-4o-mini
temperature: 0.0
prompts:
  - "What is the capital of France?"
output: results.json

auditkit eval --config auditkit.yaml

Cohere / Aya models

Aya Expanse, Tiny Aya, Aya Vision and North Micro Vision run on the hf: backend (pip install "auditkit[transformers,vision]", transformers 5.15 or newer). Score a model before and after fine-tuning:

import auditkit as ak

samples = [ak.Sample(input="Translate to French: good morning", target="bonjour")]
metrics = ["exact_match", "f1_score"]

base = ak.evaluate(samples, "hf:CohereLabs/aya-expanse-8b", metrics)
tuned = ak.evaluate(samples, "hf:<you>/aya-expanse-8b-tuned", metrics)
print(ak.compare(base, tuned).summary())

For the vision models (CohereLabs/aya-vision-8b, CohereLabs/aya-vision-32b, CohereLabs/North-Micro-Vision-Instruct), attach images to the sample; on the hf: backend they go through the model's processor:

from PIL import Image

sample = ak.Sample(input="What does this chart show?", target="sales by month",
                   images=[Image.open("chart.png")])
result = ak.evaluate([sample], "hf:CohereLabs/aya-vision-8b", ["f1_score"])

Images travel on hf: and on api: in chat mode, whichever host api: points at: a self-hosted vLLM or SGLang server, or a hosted provider. api: raw-prompt mode raises rather than drop them. The offline vllm: backend doesn't read Sample.images yet and drops them without an error: for a vision eval on vLLM, vllm serve the model and use api:.

Aya Vision does work text-only on hf: — measured, vision tower unused, 5/5 on a five-language probe, and identical to Aya Expanse 8B on the same probe. For text work prefer Expanse: same quality or better, and it runs natively on vLLM and SGLang, where Aya Vision needs each one's Transformers backend.

The Cohere models run on hf:, vllm: and SGLang (all nine measured on each), but not all natively. Aya Vision has no native server class (vLLM dropped it after 0.24.0, SGLang never had it), so both servers run it through their Transformers backends: automatically on vLLM, with compat.py --patch-sglang on SGLang. North Micro Vision is native on vLLM; on SGLang it needs transformers >= 5.15 in the SGLang env plus --patch-sglang. Tiny Aya and Aya Expanse are native on both. The full model-by-runtime table is in docs/model_backends.md.

Tiny Aya (CohereLabs/tiny-aya-global and siblings fire, water, earth), Aya Expanse and Aya Vision are gated on the Hub: accept the licence on the model page, then set HF_TOKEN (or pass token= to auditkit.model.hf_gen.HFGenModel). North Micro Vision is not gated.

What it evaluates

Output

Every evaluate() returns a RunResult:

Examples

Colab-ready notebooks with real models and real datasets — see examples/README.md for the full, current list.

Applications

Real-world scenarios answered end to end, not feature tours — see examples/applications/README.md. Only 01 uses the lm-evaluation-harness integration (ak.run_lmeval(), needs auditkit[lmeval]), as an authoritative cross-check alongside the native evaluation; none of the others do.

Repository map

Developed by Lexsi Labs

Created by the team at Lexsi Labs, AuditKIT provides a unified evaluation spine for AI models across every technique and modality.

Shreeyans Arora · Utsav Avaiya · Zera Lyngkhoi · Ram Mohan Rao Kadiyala · Hem Gosalia · Vinay Kumar Sankarapu · Pratinav Seth

Citation

If you use AuditKIT in research, please cite it:

@software{auditkit2026,
  title  = {AuditKIT: one library to evaluate any model on any dataset and any task},
  author = {Arora, Shreeyans and Avaiya, Utsav and Lyngkhoi, Zera and
            Kadiyala, Ram Mohan Rao and Gosalia, Hem and Sankarapu, Vinay Kumar and
            Seth, Pratinav},
  year   = {2026},
  version = {1.2.0},
  url    = {https://github.com/Lexsi-Labs/AuditKIT}
}

CITATION.cff carries the same author list for GitHub's citation widget.

License

This project is released under the Lexsi Labs Source Available License (LSAL) v1.2 — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.

Join Community / Contribute

• Issues and discussions are welcomed on the GitHub issue tracker.
• See Contributing for the development setup, code standards and the pull request process.