raki 0.5.1

Retrieval Assessment for Knowledge Impact — evaluate agentic RAG quality

RAKI — Retrieval Assessment for Knowledge Impact

A CLI tool that evaluates agentic RAG quality from session transcripts.

Report Preview

Features

• Operational metrics — verify rate, rework cycles, severity distribution, cost analysis (no LLM required)
• Ragas retrieval metrics — context precision/recall, faithfulness, answer relevancy
• HTML reports — interactive reports with session-level detail and color-coded thresholds
• Pluggable adapters — bring any session format; built-in support for session-schema and Alcove
• Ground truth matching — curated YAML entries matched by domain for retrieval evaluation

Quick Start

uv pip install raki --extra html
uv run raki validate --manifest raki.yaml
uv run raki run --manifest raki.yaml --no-llm

The --no-llm flag runs operational metrics only (no API keys needed). To include Ragas retrieval metrics, omit the flag and configure an LLM provider.

Usage

# Run all metrics (requires LLM provider)
uv run raki run --manifest raki.yaml

# Run operational metrics only
uv run raki run --manifest raki.yaml --no-llm

# Validate manifest and session data
uv run raki validate --manifest raki.yaml

# List available adapters
uv run raki adapters

Documentation

• Getting Started — install, run, and understand your first report
• Interpretation Reference — what each metric means and when to act
• Ground Truth Curation Guide — write effective ground truth entries
• Adapter Guide — integrate custom session formats
• Session Schema Reference — field definitions for session-schema format

Development

uv sync --python 3.14 --all-extras
uv run pytest tests/ -v
uv run ruff check src/ tests/
uv run ruff format src/ tests/
uv run ty check src/raki/

See CONTRIBUTING.md for the full contribution workflow.

License

Apache 2.0 — see LICENSE for details.