RAI Check Kit

RAI = Responsible AI. A Python package suite for evidence-grade audits of responsible, secure, and trustworthy AI systems.

Run fairness, data quality, robustness, compliance, image, medical imaging, LLM safety, RAG security, and agent trace checks. Export HTML, Markdown, or JSON reports and gate CI pipelines on risk thresholds.

Author: Sai Teja Erukude | License: MIT

Why this exists

AI teams often run fairness, robustness, RAG, and agent security checks separately. RAI Check Kit brings them into one evidence and reporting workflow, so teams can review findings consistently, preserve audit artifacts, and apply the same CI gates across model types.

What it looks like

HTML audit report	Model card export
LLM and RAG audit output	Agent trace finding

Packages

Install

pip install rai-check-kit          # core + tabular ML
pip install "rai-check-kit[all]"   # all modules (dl + genai)

Quick start

rai-check ml run --help

Inspect installed packages, plugin load failures, and missing optional dependencies:

rai-check doctor
rai-check doctor --json

Diagnostics include installation or repair commands. Missing optional features are informational; installation errors return exit code 1.

For repeatable audit workflows, generate and run a YAML configuration:

rai-check init --project loan-model
rai-check run --config audit.yaml

rai-check init writes a starter predictions.csv beside audit.yaml; replace it with captured model predictions, or update audit.data to point at your CSV. Classification configs expect y_true and y_pred columns by default.

Configured runs write report artifacts and an evidence manifest with input, environment, source-revision, and artifact hashes.

from rai_audit.ml import ClassificationAudit

report = ClassificationAudit(
    y_true=y_true,
    y_pred=y_pred,
    sensitive_features=sensitive_df,
).run()

report.to_html("audit_report.html")

Examples

• Fairness audit walkthrough
• Batch drift monitoring
• MLflow and Airflow templates
• Captured-response LLM and RAG audit suite
• Scientific image robustness audit
• Agent trace with a webpage prompt-injection attempt

Development

pip install uv
uv sync
uv run pytest

See CONTRIBUTING.md for monorepo layout and release workflow.