raki 0.6.0

Retrieval Assessment for Knowledge Impact — evaluate agentic RAG quality

RAKI — Retrieval Assessment for Knowledge Impact

A CLI tool that evaluates agentic RAG quality from session transcripts.

Report Preview

Features

• Operational metrics — verify rate, rework cycles, severity distribution, cost analysis (no LLM required)
• Ragas retrieval metrics — context precision/recall, faithfulness, answer relevancy
• HTML reports — interactive reports with session-level detail and color-coded thresholds
• Pluggable adapters — bring any session format; built-in support for session-schema and Alcove
• Ground truth matching — curated YAML entries matched by domain for retrieval evaluation

Quick Start

uv pip install raki --extra html
uv run raki validate --manifest raki.yaml
uv run raki run --manifest raki.yaml --no-llm

The --no-llm flag runs operational metrics only (no API keys needed). To include Ragas retrieval metrics, omit the flag and configure an LLM provider. Use --judge-provider anthropic for direct Anthropic API access, or --judge-provider vertex-anthropic (default) for Vertex AI.

Usage

# Run all metrics (requires LLM provider -- Vertex AI Anthropic by default)
uv run raki run --manifest raki.yaml

# Run all metrics with direct Anthropic API
uv run raki run --manifest raki.yaml --judge-provider anthropic

# Run operational metrics only
uv run raki run --manifest raki.yaml --no-llm

# Run specific metrics only
uv run raki run --manifest raki.yaml --no-llm --metrics cost_efficiency,rework_cycles

# Validate manifest and session data
uv run raki validate --manifest raki.yaml

# List available adapters
uv run raki adapters

# List available metrics (name, display name, LLM requirement)
uv run raki metrics

# Machine-readable metric list
uv run raki metrics --json

Documentation

• Getting Started — install, run, and understand your first report
• Interpretation Reference — what each metric means and when to act
• Ground Truth Curation Guide — write effective ground truth entries
• Adapter Guide — integrate custom session formats
• Session Schema Reference — field definitions for session-schema format
• CI Quality Gate Example — GitHub Actions workflow for automated RAG evaluation

Development

uv sync --python 3.12 --all-extras
uv run pytest tests/ -v
uv run ruff check src/ tests/
uv run ruff format src/ tests/
uv run ty check src/raki/

See CONTRIBUTING.md for the full contribution workflow.

License

Apache 2.0 — see LICENSE for details.